Steamout/RaR_forcasting_distilled_qwen3-4b

RaR Forecasting — SFT-distilled Qwen3-4B

Qwen3-4B supervised-fine-tuned on reasoning traces from openai/gpt-oss-120b for binary-event forecasting: given a question and a set of retrieved news articles, emit a calibrated probability that the event resolves YES.

This is the SFT-distillation baseline for a Rubric-as-Reward / RLVR comparison. It exists so that "RL beats SFT" can be claimed against a properly tuned SFT baseline on the same base model, the same data, and the same eval harness — rather than against no baseline at all.

Checkpoint: step 946 (end of epoch 2), the final training step.

Results

Test split, 500 questions, all rows at full coverage. Frozen eval settings (below). UNC = π(1−π) = 0.2275 at base rate π = 0.35.

Per-question view (the k=8 samples averaged into one forecast):

model	Brier ↓	REL ↓	RES ↑	ECE ↓	AUROC ↑	gen tokens
always 0.5	0.2500	—	0.0000	—	—	—
constant base rate	0.2275	0.0000	0.0000	—	—	—
Qwen3-4B (base)	0.2567	0.0574	0.0282	0.2098	0.6833	1,293
this model	0.1973	0.0106	0.0413	0.0825	0.7315	5,195
gpt-oss-120b (teacher)	0.1892	0.0082	0.0470	0.0736	0.7512	4,204

BS = REL − RES + UNC, 10 equal-width bins (Murphy decomposition).

Paired bootstrap over sample_id, n=500 (negative = this model is better):

comparison	Δ Brier	95% CI	significant
vs Qwen3-4B base	−0.0594	[−0.0791, −0.0400]	yes
vs Qwen3.5-27B	−0.0221	[−0.0378, −0.0065]	yes
vs gpt-oss-120b teacher	+0.0081	[−0.0002, +0.0164]	no

The 4B student is not statistically distinguishable from its 120B teacher on Brier. The CI straddles zero by a hair — read that as "not distinguishable", not "equal".

If you need to compare against numbers reported per-generation (500×8 = 4,000 rows rather than 500 questions), this model scores 0.2083; base scores 0.2735 and the teacher 0.1970. Note that per-generation confidence intervals are roughly √8 too narrow, because the k samples of one question are not independent.

Honest caveats

Read these before quoting the headline.

• Most of the gain is calibration, not new knowledge. Decomposing the −0.0594 improvement over base: +0.0468 comes from Reliability, +0.0131 from Resolution — about 78% calibration / 22% discrimination. That said, Resolution did rise 0.0282 → 0.0413 (+46% relative), and Resolution is exactly the term a distribution-fitting model cannot have.
• All of the skill requires the retrieved news. With the news stripped and only the question text given, Brier degrades 0.1973 → 0.2532, which is worse than the constant base-rate predictor (0.2275). This model is not a standalone world-knowledge forecaster.
• A mild question-text prior was picked up. Closed-book AUROC is 0.6062 (base: 0.5563). It is not usable skill — closed-book Brier is still worse than the base rate — but it is worth re-checking on any successor.
• Base Qwen3-4B has negative skill here (0.2567 vs 0.2275 base rate), so "beats base" understates what happened: base was not forecasting in the aggregate sense at all.
• Training saw 4,396 of 5,265 train questions. The 50/50 positive/negative mixture cannot consume every positive trace (the teacher is correct-side only ~67% of the time), so ~800 easy questions drop out entirely. This is inherent to that mixture ratio, not a defect.

Usage

The prompt format is load-bearing. Training and evaluation prompts are byte-identical, and the model was never trained on any other rendering.

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "Steamout/RaR_forcasting_distilled_qwen3-4b"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto", device_map="auto")

# `question_and_news` is the dataset's `prompt` column flattened to one string.
content = question_and_news.rstrip() + "\n\n/think"      # Qwen3 soft switch
prompt = tok.apply_chat_template(
    [{"role": "user", "content": content}],              # ONE user message, no system turn
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True,
)

Generate with temperature=0.6, top_p=1.0, top_k=-1. The model emits <think>…</think> followed by <answer>0.XX</answer>. Budget generously: mean completion is ~5,200 tokens.

Parsing the answer — important

Split on the last </think> before parsing, and read only the text after it. The model states a probability inside its reasoning block ~90% of the time, and that in-think number is not always the final answer.

region = text.rsplit("</think>", 1)[-1]     # do this first
m = re.search(r"<answer>\s*([0-9]*\.?[0-9]+)\s*</answer>", region)

This habit is inherited from the teacher (which does it 96.5% of the time); base Qwen3-4B never does it. Any downstream reward function or scorer that regexes the raw completion without the </think> split will silently read the wrong number.

Training

Teacher selection. gpt-oss-120b was chosen over two 2026-era Qwen3.5 candidates after a gate that required: significant paired-bootstrap improvement over the student base, higher Resolution, a clean closed-book contamination probe, strict parse rate ≥95%, truncation <2%, and a near-zero no-reasoning rate. Its 2024-06 knowledge cutoff predates this dataset's 2025 resolution window, so it cannot be answering from memory. One 122B candidate was disqualified as contaminated — it beat the base rate closed-book with no retrieved news at all (AUROC 0.687), which is the memorised-outcome signature.

Trace extraction. Train split, K=4 samples per question, temperature=1.0 (deliberately higher than eval — at 0.6 the K samples are near-duplicates and no positive/negative split exists to filter on), max_tokens=16384, teacher's own native chat template with no format coercion. Yield: 5,265/5,265 questions, 21,060 generations, parse rate 100%, truncation 0.33%, no-reasoning 0.0%.

Dataset construction. Filters, in order: parse failure → non-strict parse rule → truncated → too-short reasoning → over max sequence length → max 2 traces per question → positive/negative mixture. Quality drops totalled 0.43%. Truncated traces are dropped, never cut — a cut trace teaches a cut answer.

A trace is "positive" when the teacher's probability landed on the correct side of the outcome (Brier ≤ 0.25). The mixture is held at 0.500, matching the positive-50 convention of the RL run it is compared against. Positive-only filtering is deliberately not used: in forecasting the label is a single Bernoulli draw, so a question with true probability 0.7 that resolved NO has "positive" traces that argued for a low probability — reasoning that was wrong, selected by luck.

Result: 7,708 examples, 4,396 unique questions, 28.4M supervised tokens.

Decontamination. Exact sample_id and exact question-text overlap between train and test are both zero on this dataset, and both checks are insufficient. 71 train questions (1.35%) are reworded near-duplicates of a test question at content-word Jaccard ≥ 0.6, and 50 of those carry the same label. They were removed before training. Example of what exact matching misses:

TEST   AfD receive more than 20.0% of Zweitstimmen ...   (y=1)
TRAIN  AfD receive at least 20%   of Zweitstimmen ...   (y=1)   J=0.882

The leak is not via retrieved news — train prediction dates all precede test resolution dates — it is via the label: a same-deadline twin resolves the same way, and positive filtering selects exactly the traces that argued toward it.

Hyperparameters.

epochs	2 (946 steps)
learning rate	1e-5, cosine_with_min_lr, min ratio 0.1
warmup	28 steps (3%)
effective batch	16
optimizer	AdamW (fused), β (0.9, 0.999), wd 0.0, grad clip 1.0
precision	bf16 compute, fp32 master weights
max sequence length	20,480
loss	completion-only (prompt masked to -100)
parallelism	plain DDP, gradient checkpointing

Eval loss: 1.569 → 1.539 → 1.530 → 1.525; train loss 1.512.

Two implementation notes that silently corrupt this kind of run if missed:

1. The SFT target is assembled by hand as <think>\n{reasoning}\n</think>\n\n{final}<|im_end|> and never re-passed through a chat template. Qwen3 templates can strip <think> blocks out of assistant turns, which deletes the entire trace being taught while the loss curve still looks healthy.
2. The teacher emits reasoning inside its own control-token channels. Decoding with the default skip_special_tokens=True deletes those markers, the reasoning/final split then finds nothing, and the no-reasoning rate hits 100% while parse rate, Brier and truncation all look fine.

No length collapse. Mean completion rose 1,293 → 5,195 tokens. The usual trace-SFT failure mode did not occur. Checkpoints also improved monotonically to the final step — step 946 significantly beats both step 472 and step 708 — so the common "peaks before end of epoch 2" expectation did not reproduce here.

Evaluation protocol

Frozen; changing any of these invalidates comparison with the numbers above.

split=test  n=8  temperature=0.6  top_p=1.0  top_k=-1  seed=1
max_tokens=24000  max_model_len=30000  n_bins=10
parse failure scored as Brier 1.0

Diagnostics for this checkpoint: parse rate 100%, strict parse rate 99.8%, truncation 0.5%, no-reasoning rate 0.5%.

Weights

bfloat16, 8.04 GB, single shard, tied embeddings. Training stored fp32 master weights; the released weights are an exact round-to-nearest cast of those, which is also the precision the evaluation itself ran at — so these weights reproduce the reported numbers rather than merely approximating them.

Provenance

• Base model: Qwen/Qwen3-4B (Apache-2.0)
• Teacher: openai/gpt-oss-120b (Apache-2.0)
• Dataset: LightningRodLabs/future-as-label-paper-training-dataset