aestudio/Olmo-3-7B-Instruct-DPO-GRPO-math-lora

Olmo-3-7B-Instruct-DPO + GRPO on DeepMath (LoRA adapter)

A LoRA adapter over allenai/Olmo-3-7B-Instruct-DPO, trained with GRPO and a verifiable correctness reward on DeepMath-103K.

It recovers 78 % of the gap between the DPO checkpoint it starts from and Ai2's released RLVR checkpoint (allenai/Olmo-3-7B-Instruct), measured on 1,000 held-out DeepMath problems, without measurable cost to instruction following or non-math reasoning.

This is an adapter, not a merged model. You need the base model above.

---

⚠️ Requires transformers >= 5.13

On transformers 5.0–5.12 this adapter silently evaluates as a different model. Those versions apply Olmo 3's YaRN rope scaling to all 32 layers, while vLLM, OLMo-core and the Olmo 3 paper apply it only to the 8 full-attention layers. The HF forward pass then sits 1.17 nats/token away from the one vLLM serves — 32 % of tokens off by more than a nat, and the error grows with position.

This is not a subtle numerical difference; a model trained or evaluated through it is a different model. Upstream issue huggingface/transformers#39847, fixed by #46911 in 5.13.0.

If you are stuck on an older version, apply per-layer rope scaling yourself: sliding-window layers should get plain RoPE (same θ, scaling factor 1.0) while the full-attention layers keep YaRN. Verify before trusting any number — see "Checking your setup" below.

Stack this was trained and evaluated on: vllm 0.28.0 · transformers 5.17.0 · trl 1.13.0 · peft 0.20.0 · torch 2.13.0+cu130.

---

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("allenai/Olmo-3-7B-Instruct-DPO", dtype="bfloat16")
model = PeftModel.from_pretrained(base, "<this-repo>")
tok = AutoTokenizer.from_pretrained("allenai/Olmo-3-7B-Instruct-DPO")

Prompt exactly as it was trained and evaluated: a single user turn containing the problem, followed by

Put your final answer within \boxed{}.

No system prompt.

Results

1,000 held-out DeepMath problems, drawn from a reserve the model never trained on, n = 4 at temperature 0.6, paired on identical problems with a bootstrap CI:

	pass@1	paired Δ vs base	% of the DPO→RLVR gap
Olmo-3-7B-Instruct-DPO (base)	0.4850	—	0 %
this adapter	0.6597	+0.1747 [+0.1560, +0.1935]	78.1 %
Olmo-3-7B-Instruct (RLVR, for reference)	0.7087	+0.2237 [+0.2025, +0.2443]	100 %

It remains −0.0490 [−0.0653, −0.0317] short of the released RLVR checkpoint. The curve was still rising when training stopped — generation 150 vs 120: +0.0210 [+0.0062, +0.0355] — but decelerating, at 0.51 → 0.49 → 0.31 points of gap per generation over the last three windows.

General capability was not traded away

Five arms, graded programmatically (no LLM judge), paired against the base model. The threshold was fixed before any arm was scored: a regression counts only if the paired Δ's CI upper bound falls below −0.02.

	IFEval (strict)	paired Δ	ARC-Challenge	paired Δ
base (DPO)	0.7731	—	0.6259	—
RLVR (reference)	0.8198	+0.0467	0.6517	+0.0258
this adapter	0.7671	−0.0060 [−0.0185, +0.0069]	0.6681	+0.0422 [+0.0256, +0.0580]

Instruction following is unchanged; non-math multiple-choice reasoning improved. \boxed{} leakage onto non-math prompts is 0.002 (base: 0.003), the response-length ratio is 1.01×, and the degeneration rate is 0. The expected failure mode for a reward that pays only for boxed answers — format and verbosity lock-in — did not appear.

These are n = 4 at temperature 0.6, not IFEval's usual greedy n = 1, so they are not comparable to published leaderboard scores. They compare these checkpoints to each other.

Checking your setup

Score the base model and this adapter on any shared math set. If the adapter is not clearly ahead, suspect the transformers version before the weights. A direct check: an HF forward pass should agree with a vLLM ≥ 0.28 sampler to about 0.01 nats/token on the same tokens; ~1.2 means the rope problem above.

Training

algorithm	GRPO (TRL 1.13 with a colocated vLLM 0.28 engine), DAPO token-level loss, β = 0, ε 0.2 / 0.28
adapter	LoRA r = 128, α = 256, dropout 0, all-linear, bf16 (~1.2 GB)
optimiser	constant lr 1e-5, unscaled advantages
rollouts	64 prompts × 8 samples = 512 per generation, temperature 1.0, top_p 1.0, 8,192-token cap
length	150 generations = 76,800 rollouts, 600 optimiser steps, 15.4 h on 8×H200
reward	1.0 if the final \boxed{} answer matches the gold under math_verify, else 0.0. No format shaping, no length term; truncated completions are unmasked and score 0
data	DeepMath-103K, decontaminated against 22 public pretraining/instruction corpora; 9,600 problems seen, each exactly once

The recipe follows Ai2's published open-instruct RL script for this checkpoint, with LoRA in place of a full fine-tune. Trainer-and-sampler agreement was gated throughout and held at 0.011 nats/token across all 150 generations.

Limitations

• 78 % of the gap, not 100 %. It does not match the released RLVR checkpoint and is not a drop-in replacement for it.
• Math only. The retention evidence covers instruction following and multiple-choice reasoning. There is no open-ended quality, multilingual, coding, or safety evaluation.
• Evaluated on DeepMath's own distribution. Competition-style or out-of-distribution math was not the target of these measurements.
• Adapter, not a model. Requires the base weights, and the transformers constraint above is not optional.

License

The adapter follows the base model's license (Apache-2.0, allenai/Olmo-3-7B-Instruct-DPO). DeepMath-103K is MIT.