aestudio/Olmo-3-7B-Instruct-DPO-GRPO-math-lora-gen270

Olmo-3-7B-Instruct-DPO + GRPO on DeepMath (LoRA adapter, generation 270)

A LoRA adapter over allenai/Olmo-3-7B-Instruct-DPO, trained with GRPO and a verifiable correctness reward on DeepMath-103K.

On a 1,500-problem held-out DeepMath test it reaches parity with Ai2's released RLVR checkpoint (allenai/Olmo-3-7B-Instruct): pass@1 0.699 against RLVR's 0.702, a paired difference of −0.003 [−0.012, +0.006], closing 98.5 % of the gap from the DPO checkpoint it starts from. Parity, not a lead. Its training data is fully enumerated — every prompt it was ever differentiated on is listed, per generation — and the test excludes all of it.

This supersedes aestudio/Olmo-3-7B-Instruct-DPO-GRPO-math-lora, the generation-150 checkpoint of the same run (78 % of the gap), which stays up for reproducibility.

This is an adapter, not a merged model. You need the base model above.

---

⚠️ Requires transformers >= 5.13

On transformers 5.0–5.12 this adapter silently evaluates as a different model. Those versions apply Olmo 3's YaRN rope scaling to all 32 layers, while vLLM, OLMo-core and the Olmo 3 paper apply it only to the 8 full-attention layers. The HF forward pass then sits 1.17 nats/token away from the one vLLM serves — 32 % of tokens off by more than a nat, and the error grows with position.

This is not a subtle numerical difference; a model trained or evaluated through it is a different model. Upstream issue huggingface/transformers#39847, fixed by #46911 in 5.13.0.

If you are stuck on an older version, apply per-layer rope scaling yourself: sliding-window layers should get plain RoPE (same θ, scaling factor 1.0) while the full-attention layers keep YaRN. Verify before trusting any number — see "Checking your setup" below.

Stack this was trained and evaluated on: vllm 0.28.0 · transformers 5.17.0 · trl 1.13.0 · peft 0.20.0 · torch 2.13.0+cu130.

---

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("allenai/Olmo-3-7B-Instruct-DPO", dtype="bfloat16")
model = PeftModel.from_pretrained(base, "aestudio/Olmo-3-7B-Instruct-DPO-GRPO-math-lora-gen270")
tok = AutoTokenizer.from_pretrained("allenai/Olmo-3-7B-Instruct-DPO")

Prompt exactly as it was trained and evaluated: a single user turn containing the problem, followed by

Put your final answer within \boxed{}.

No system prompt.

Results

Held-out test: 1,500 DeepMath problems (tl_rl_test, built by TL-23 from the same source with every prompt this adapter trained on or was selected on removed along with near-duplicates), n = 16 at temperature 0.6, paired on identical problems with a bootstrap CI:

	pass@1	paired Δ vs base	paired Δ vs RLVR
Olmo-3-7B-Instruct-SFT	0.376
Olmo-3-7B-Instruct-DPO (base)	0.492	—
this adapter (generation 270)	0.699	+0.207 [+0.194, +0.219]	−0.003 [−0.012, +0.006]
Olmo-3-7B-Instruct (RLVR, for reference)	0.702	+0.210 [+0.198, +0.224]	—

Parity with the released RLVR checkpoint, not a lead: 98.5 % of the DPO→RLVR gap, and the two are indistinguishable on this set. The checkpoint was chosen on a separate 1,000-problem validation set (where it read 0.716 against RLVR's 0.709); the test number above is the one to quote.

Validation (checkpoint selection), 1,000 problems, n = 4 at temperature 0.6: base 0.485 · this adapter 0.716 · RLVR 0.709; paired Δ vs base +0.231 [+0.210, +0.251].

The curve, same set, same pairing

generation	20	50	90	120	150	200	250	270	300
pass@1	0.537	0.560	0.606	0.639	0.660	0.693	0.695	0.716	0.716
% of gap	23	34	54	69	78	93	94	103	103

Generations 270 and 300 tie on validation (paired difference +0.000 [−0.014, +0.015]); 270 is published because the policy's entropy began rising over generations 260–300 while validation stayed flat. On the held-out test, generations 250 / 260 / 270 / 300 score 0.686 / 0.699 / 0.699 / 0.708; generation 300 is +0.009 [+0.004, +0.014] above this checkpoint there. The choice was fixed before the test was scored and was not revisited on it; generation 300's general-capability retention has not been measured.

General capability was not traded away

IFEval (541 prompts, strict prompt-level) and ARC-Challenge (1,172 items), graded programmatically (no LLM judge), n = 4 at temperature 0.6, paired against the base on identical items. The regression threshold was fixed before any arm was scored: a regression counts only if the paired Δ's CI upper bound falls below −0.02.

	IFEval (strict)	paired Δ	ARC-Challenge	paired Δ
base (DPO)	0.7731	—	0.6259	—
RLVR (reference)	0.8198	+0.0467 [+0.0287, +0.0642]	0.6517	+0.0258 [+0.0070, +0.0437]
generation 150 (previous release)	0.7671	−0.0060 [−0.0185, +0.0069]	0.6681	+0.0422 [+0.0256, +0.0580]
this adapter (generation 270)	0.7791	+0.0060 [−0.0079, +0.0203]	0.6890	+0.0631 [+0.0461, +0.0794]

Instruction following is unchanged (and the small dip seen at generation 150 is gone); non-math multiple-choice reasoning improved by more than the released RLVR checkpoint's own gain. \boxed{} leakage onto non-math prompts is 0.002 on IFEval and 0.004 on ARC (base: 0.003 / 0.006), the degeneration rate is 0, and responses are 0.96× the base's length on IFEval and 1.24× on ARC (RLVR: 0.79× / 1.26×). The expected failure mode for a reward that pays only for boxed answers — format and verbosity lock-in — did not appear at 270 generations either.

These are n = 4 at temperature 0.6, not IFEval's usual greedy n = 1, so they are not comparable to published leaderboard scores. They compare these checkpoints to each other.

Checking your setup

Score the base model and this adapter on any shared math set. If the adapter is not clearly ahead, suspect the transformers version before the weights. A direct check: an HF forward pass should agree with a vLLM ≥ 0.28 sampler to about 0.01 nats/token on the same tokens; ~1.2 means the rope problem above.

Training

Two stages of one recipe. Stage 2 continues stage 1's adapter weights with a fresh optimiser and a fresh data order, on the pool minus 14,500 prompts reserved for follow-up work.

algorithm	GRPO (TRL 1.13 with a colocated vLLM 0.28 engine), DAPO token-level loss, β = 0, ε 0.2 / 0.28
adapter	LoRA r = 128, α = 256, dropout 0, all-linear, bf16 (~1.2 GB)
optimiser	constant lr 1e-5, unscaled advantages
rollouts	64 prompts × 8 samples = 512 per generation, temperature 1.0, top_p 1.0, 8,192-token cap
stage 1	generations 1–150 on the full 65,626-prompt pool; 600 optimiser steps, 15.4 h on 8×H200
stage 2	generations 151–270 from stage 1's weights; 480 optimiser steps, ~12.6 h on 8×H200
total	270 generations = 138,240 rollouts, 1,080 optimiser steps
reward	1.0 if the final \boxed{} answer matches the gold under math_verify, else 0.0. No format shaping, no length term; truncated completions are unmasked and score 0
data	DeepMath-103K, decontaminated against 22 public pretraining/instruction corpora; 16,146 distinct problems seen (9,600 in stage 1, 7,680 in stage 2, 1,134 in both), each once per stage

The recipe follows Ai2's published open-instruct RL script for this checkpoint, with LoRA in place of a full fine-tune. Trainer-and-sampler agreement was gated throughout and held at ≤ 0.013 nats/token across all 270 generations. Every prompt the adapter trained on is recorded per generation, so seen/unseen splits of DeepMath against this model can be drawn exactly.

Limitations

• Parity with RLVR, not a lead. −0.003 [−0.012, +0.006] on the held-out test. One seed, one recipe.
• Retention covers two axes. Instruction following and multiple-choice reasoning were measured; nothing else was.
• Math only. No open-ended quality, multilingual, coding, or safety evaluation.
• Evaluated on DeepMath's own distribution. Competition-style or out-of-distribution math was not the target of these measurements.
• Adapter, not a model. Requires the base weights, and the transformers constraint above is not optional.

License

The adapter follows the base model's license (Apache-2.0, allenai/Olmo-3-7B-Instruct-DPO). DeepMath-103K is MIT.