Shamima/dpo-implicit-reward

Exploring DPO's Implicit Reward — checkpoints

Research checkpoints for a study asking whether DPO's implicit reward

r̂(x, y) = β · log [ π_θ(y|x) / π_ref(y|x) ]

generalizes as a reward model as well as an explicitly trained Bradley–Terry (BT) reward model, under matched data and compute, in-distribution and under distribution shift.

• H1: explicit RM > implicit on held-out in-distribution pairs.
• H2: the gap widens under distribution shift.
• A null result (implicit ≈ explicit) would empirically support DPO's claim at this scale.

Code: github.com/silvererudite/dpo-implicit-reward

⚠️ Status: preliminary

These are work-in-progress research artifacts, not production models. At upload time the evaluation had not finished, so no H1/H2 accuracy numbers are attached yet. See the caveat on compute matching below before drawing comparisons between the two scorers.

Contents

Path	What it is
sft_merged/	π_ref — the SFT reference policy, LoRA merged into the weights. Full model.
sft/	The SFT LoRA adapter alone (applies to Qwen/Qwen2.5-0.5B).
dpo_beta0.1_8k/	DPO policy π_θ, β=0.1, 8k preference pairs. LoRA adapter over sft_merged.
rm_8k/	Explicit Bradley–Terry reward model, 8k pairs. SEQ_CLS LoRA + scalar head over sft_merged.

Each directory carries a budget.json recording steps, epochs, effective batch, LR, seed and wall-clock, so the compute-matching claim is auditable rather than asserted.

π_ref must be the merged checkpoint. Loading base + SFT adapter and letting TRL derive the reference by disabling the adapter yields the raw base model, which is a different quantity and silently changes every implicit-reward number.

Training setup

Base model Qwen/Qwen2.5-0.5B, LoRA r=16 / α=32 / dropout=0.05 on all attention and MLP projections, bf16 on a single NVIDIA A10G, TRL 0.15.2. DPO and the RM both initialize from sft_merged so neither scorer starts ahead of the other.

Stage	Steps	Epochs	Effective batch	LR	Wall-clock
SFT → π_ref (32k pairs)	1998	1.0	16	2e-4	1h 13m
DPO β=0.1 (8k pairs)	500	1.0	16	5e-5	42m 21s
BT reward model (8k pairs)	470	1.0	16	5e-5	23m 19s

Known caveat on the compute match

The RM completed 470 steps (7,520 pairs) against DPO's 500 (8,000) at the same nominal 8k budget and identical effective batch. The cause is a difference between the TRL trainers: RewardTrainer drops examples exceeding max_length while DPOTrainer truncates them. The two conditions therefore do not see byte-identical data, which weakens the "same pairs" half of the matched-compute definition. Pre-filtering both to a common length-eligible subset and re-running is pending; treat any implicit-vs-explicit comparison from these exact checkpoints as provisional.

Intended use and limitations

Intended for research on reward-model generalization, reproducing this study, or as small reward-model baselines. At 0.5B, accuracy sits near chance on harder RewardBench subsets, and single-seed numbers are noisy — the study design calls for ≥3 seeds with confidence intervals before any headline claim.

Both scorers can proxy response length rather than quality; non-neural baselines on this data show pick-longer reaching 0.804 on RewardBench-Chat while falling to 0.294 on the adversarial Chat-Hard subset. Report length-controlled accuracy alongside raw accuracy.

These models inherit the biases of Qwen2.5-0.5B and of the preference data (UltraFeedback, HH-RLHF). They are not safety-aligned and should not be used to moderate or rank content in any deployed setting.

License

Apache 2.0, following the Qwen/Qwen2.5-0.5B base model.