SPPO Qwen3-4B (UltraChat50k)
Self-Play Preference Optimization on Qwen3-4B: SFT on UltraChat200k, then 3 self-play iterations on UltraChat50k.
There is no iter0. Iteration 1 starts from model_hub/Qwen3-4B-ultrachat200k.
Final policy: iter3/.
from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained("HoangTran223/SPPO_Qwen3-4B", subfolder="iter3")
| Path | Contents |
|---|---|
iterN/ |
Final HF weights after 2 training epochs |
iterN/synthetic_data/ |
Preference parquet used by the trainer |
iterN/generated/ |
Combined generations, PairRM probabilities.json, raw responses_*.json |
iterN/ranking/ |
PairRM rank matrix (0_0.npy) |
iterN/logs/ |
vLLM generate, PairRM rank, trainer stdout |
logs/wandb/ |
Offline wandb runs |
configs/ |
Trainer yaml for each iteration |
Training: loss_type=sppo, max_length=1024, max_prompt_length=512, batch 1 × grad accum 2, RMSProp, LR 5e-7, 2 epochs.
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support