DPO-4B-MobileGym

Offline trajectory-level DPO on HaoranLiu/SFT-4B-MobileGym using HaoranLiu/DPO-Qwen3-MobileGym, trained with cua-lite + slime.

This checkpoint is iter_137 — the end of epoch 2 of 3.

Results — MobileGym eval split

All 256 tasks (no filter), greedy (temperature=0), max_steps: 45, group_size=1. Every row was measured in this project under this one protocol.

model	mean_return	success (=1.0)	success_rate	partial
Qwen3-VL-4B base	0.2140	36	14.06%	45
SFT-4B-MobileGym (init)	0.3656	67	26.17%	62
iter_68 (epoch 1)	0.3758	72	28.12%	59
iter_137 (epoch 2, this)	0.3865	71	27.73%	65
iter_206 (epoch 3)	0.3829	71	27.73%	65

The two metrics pick different checkpoints

MobileGym's judges emit a continuous progress rate in [0,1], so mean_return rewards partial progress while success_rate counts only fully solved tasks.

ep1 -> ep2   mean_return  0.3758 -> 0.3865   (+0.011)
             success         72   ->   71    (-1)
             partial         59   ->   65    (+6)

Epoch 2's mean gain comes entirely from six more tasks earning partial credit — it solves one fewer task outright than epoch 1, and epoch 3 changes neither count. Pick accordingly:

• fully-completed tasks → iter_68 (epoch 1, and one third of the training cost)
• average progress → this checkpoint
• iter_206 has no argument in its favour

Data handling worth repeating

The dataset card flags ~18% exact-duplicate pairs (identical chosen and rejected action sequences, an artifact of pairing on source-row reference rather than content). Training on the raw file gives those pairs double gradient weight, so this run deduplicated by content hash first: 336 → 276 pairs.

Why epoch 2 still improves here (and does not everywhere)

The rejected trajectories in this dataset come from Qwen3-VL-4B-Instruct — the same model family being trained, i.e. on-policy negatives. In a companion experiment on lite.osworld, two runs from an identical SFT init differed only in the model that produced the rejected side:

rejected from	ep1	ep2	ep3
Qwen3-VL-4B (on-policy)	0.3579	0.3801	0.3571
Qwen3-VL-2B (off-policy)	0.3581	0.3491	—

They are indistinguishable after one epoch (0.3579 vs 0.3581) and diverge only in epoch 2. The training signal explains it: with 2B negatives the pairs separate much faster (epoch-2 median dpo_loss 4.37e-5 vs 1.48e-4; epoch-3 saturated steps 36.3% vs 28.4%), so later epochs spend their budget suppressing errors the 4B policy would never make.

Rule of thumb from these runs: on-policy negatives → 2 epochs; off-policy → 1.

Reproducibility

An earlier standalone 1-epoch run on the same data and parameters produced an iter_68 scoring 0.3793 / 72 successes — this run's epoch-1 checkpoint scored 0.3758 / 72, the same success count and mean within 0.0035. Run-to-run variance is small enough that the epoch-to-epoch differences above are not noise.

Full provenance in run_info.txt.

Usage

uv run python scripts/rollout.py \
  --model-id Qwen/Qwen3-VL-4B-Instruct \
  --model-path <path to this checkpoint> \
  --env-id mobilegym --splits eval \
  --config-path scripts/configs/qwen3_vl/default/mobilegym.yaml

The qwen3_vl adapter/config is required — that rendering is frozen into the training data.