DPO-4B-MobileGym-fork

Trajectory-level DPO on fork-point pairs, from HaoranLiu/SFT-4B-MobileGym. Epoch-2 checkpoint, 24 steps.

Task-id pairing yields zero pairs on this source (160 tasks, 1 sample each), so every pair comes from forking a failed trajectory at the judged first mistake.

• data: HaoranLiu/DPO-fork-Qwen3-MobileGym, 51 of 58 pairs (7 with a quality-tagged chosen side dropped)
• chosen: gpt-5.5 fork continuation · rejected: Qwen/Qwen3-VL-4B-Instruct, return 0.0

Hyperparameters

objective	trajectory-level DPO, summed per-action log-probs (no length normalization)
dpo_beta	0.1
kl_coef	0.0
optimizer	Adam, beta1 0.9, beta2 0.95, eps 1e-8
lr	5e-7, cosine decay to 0, warmup fraction 0.1
weight_decay	0.1
clip_grad	1.0
global_batch_size	4 pairs/step
micro_batch_size	1
seq_length	4096
precision	bf16
seed	1234
parallelism	TP=2, PP=1, CP=1, DP=1 (2xH100), sequence_parallel
optimizer sharding	distributed (ZeRO-1) + CPU offload, precision-aware
recompute	full, uniform, num_layers 1
loss / advantage hooks	lite.train.dpo.dpo_loss_function / lite.train.dpo.compute_dpo_pair_weights

Results

mobilegym eval, all 256 tasks, greedy, max_steps 45, group_size 1.

model	steps	mean	success	success_rate
Qwen3-VL-4B-Instruct (base)	—	0.2140	36	14.06%
SFT-4B-MobileGym iter_129	—	0.3120	57	22.27%
SFT-4B-MobileGym (init)	—	0.3656	67	26.17%
fork ep1	12	0.3704	68	26.56%
this ckpt — fork ep2	24	0.3757	70	27.34%
fork ep3	36	0.3605	66	25.78%
naive pairs ep1 (276 pairs)	69	0.3758	72	28.12%
naive pairs ep2	138	0.3865	71	27.73%
naive pairs ep3	207	0.3829	71	27.73%