DPO-4B-LiteOS-ecb96fbe6

Offline trajectory-level DPO on HaoranLiu/SFT-4B-LiteOS using HaoranLiu/DPO-Qwen3-LiteOS-ecb96fbe6 (382 deduplicated preference pairs over 109 Lite.OSWorld tasks), trained with cua-lite + slime.

This checkpoint is iter_191 — the end of epoch 2, not the end of training. Epoch 3 overfits; see below.

Results — Lite.OSWorld eval split

332 tasks (the 369-task split minus exclude_reason), greedy (temperature=0), max_steps: 30, group_size=1. Every row was measured under this one protocol in the same project, so they are directly comparable.

model	steps	mean episode_return	pass (=1.0)
SFT-4B-LiteOS (init)	—	0.3220	103
DPO-4B-LiteOS (older 898-pair data)	224	0.3511	113
this data, iter_95 (epoch 1)	96	0.3579	117
this checkpoint, iter_191 (epoch 2)	192	0.3801	123
this data, iter_284 (epoch 3, final)	285	0.3571	116

Epoch 3 overfits — why this is the epoch-2 checkpoint

Per task: iter_191 beats iter_95 on 14 / loses 5. Then iter_284 loses to iter_191 on 14 and wins only 6, with six suites sliding together (thunderbird −0.071, vlc −0.067, vs_code −0.056, multi_apps −0.029, chrome −0.023, libreoffice_impress −0.021).

The training signal predicts it:

	median dpo_loss	median dpo_weight	median grad_norm	saturated (<1e-3)
epoch 1	1.355	1.21e-1	67.2	5.3%
epoch 2	1.48e-4	1.48e-5	0.014	28.4%
epoch 3	8.46e-5	8.45e-6	0.008	28.4%

The pairs are already separated by the end of epoch 1 — the loss falls four orders of magnitude and w = β·sigmoid(−β·Δ) collapses with it — so epoch 3 spends 95 steps nudging probability mass on already-solved pairs, with --kl-coef 0 leaving nothing to anchor the policy to the SFT init.

Use 1–2 epochs on a set this size. Do not carry epoch counts over from SFT runs on the same environment; SFT and DPO overfit differently.

Scope of the comparison (read before citing the +0.029)

This run existed to test whether the older DPO set taught the model which model wrote the trajectory rather than task quality — in that set, 100% of pairs were teacher-chosen vs student-rejected.

This dataset only partly removes that confound: 48/382 pairs (12.6%) are student-chosen vs student-rejected; the other 334 keep the teacher/student split. The format asymmetry is softened but present (Action: prefix in 90.8% of chosen steps vs 100% of rejected; the older set was 80.2% vs 100%), and the length asymmetry is unchanged (rejected 1.57× longer). iter_191 does beat the older 224-step run by +0.029 at a comparable step count, but deduplication (0% duplicates here) and step counts also differ, so the gain cannot be attributed to the source confound. A clean test needs pairs where chosen and rejected come from the same model.

Full provenance in run_info.txt.

Usage

uv run python scripts/rollout.py \
  --model-id Qwen/Qwen3-VL-4B-Instruct \
  --model-path <path to this checkpoint> \
  --env-id lite.osworld --splits eval \
  --filter "lambda m: not m.others.get('exclude_reason')" \
  --config-path scripts/configs/qwen3_vl/default/lite.osworld.yaml

The qwen3_vl adapter/config is required — that rendering is frozen into the training data.