DPO-4B-LiteOS-fork596

Trajectory-level DPO on fork-point (prefix-aligned) pairs, from HaoranLiu/SFT-4B-LiteOS. Full fork_turn > 0 set, 1 epoch, 149 steps.

• data: HaoranLiu/DPO-fork-Qwen, all 596 pairs with fork_turn > 0
• chosen: gpt-5.5 fork continuation · rejected: Qwen/Qwen3-VL-4B-Instruct

Hyperparameters

objective	trajectory-level DPO, summed per-action log-probs (no length normalization)
dpo_beta	0.1
kl_coef	0.0
optimizer	Adam, beta1 0.9, beta2 0.95, eps 1e-8
lr	5e-7, cosine decay to 0, warmup fraction 0.1
weight_decay	0.1
clip_grad	1.0
global_batch_size	4 pairs/step
micro_batch_size	1
seq_length	4096
precision	bf16
seed	1234
parallelism	TP=2, PP=1, CP=1, DP=1 (2xH100), sequence_parallel
optimizer sharding	distributed (ZeRO-1) + CPU offload, precision-aware
recompute	full, uniform, num_layers 1
loss / advantage hooks	lite.train.dpo.dpo_loss_function / lite.train.dpo.compute_dpo_pair_weights

Results

lite.osworld eval, 332 tasks, greedy, max_steps 30, group_size 1. Mean over all 332.

model	steps	mean	success	success_rate
SFT-4B-LiteOS (init)	—	0.3220	103	31.02%
this ckpt — fork596 (1 ep)	149	0.3557	116	34.94%
fork350 ep1	88	0.3628	116	34.94%
fork350 ep2	176	0.3571	115	34.64%
fork350 ep3	261	0.3406	110	33.13%
naive pairs ep1	96	0.3579	117	35.24%
naive pairs ep2	192	0.3801	123	37.05%
naive pairs ep3	285	0.3571	116	34.94%
2B-negatives ep1	102	0.3581	117	35.24%
32B-negatives ep1	84	0.3466	111	33.43%
mixed 2B+4B+32B ep2/3	188	0.3746	122	36.75%
first-diverge (loss on fork turn only) ep2	388	0.2718	87	26.20%