Model Card for SAIGE-dpo-v5

DPO fine-tune of Qwen/Qwen2.5-3B-Instruct on the 16 high-contrast Right-Speech preference pairs from M1ztyk/SAIGE-right-speech-dpo-v4 (dpo_pairs_v4.jsonl). This is run5 of the SAIGE DPO experiment series — the data-contrast test: run4's hyperparameters held fixed, only the dataset changed (delta ≥ 3 rejects vs the old delta-median-1 rejects).

An equivalent-config run of the same experiment exists at M1ztyk/SAIGE-dpo-v4-run5; both produced identical ablation numbers.

Data

• 16 pairs (rs 6 / generic 5 / none 5) generated and judged by Qwen/Qwen2.5-72B-Instruct-AWQ under the verbatim SAIGE rubric; kept only if judge score_delta ≥ 3 (11 at delta 3, 3 at delta 4, 2 at delta 6).
• Judge calibration: re-scoring the original 62 pairs under the same judge gave mean delta 0.42 (near-symmetric — no inflation).
• Regeneration pipeline: generate_rejects_v4.py in the dataset repo.

Training (run4 config, unchanged)

LoRA r=16 α=32 all-linear on 4-bit NF4 QLoRA · lr 5e-6 · beta 0.1 · loss ["sigmoid","sft"] weights [1,1] · effective batch 4 · fp16 compute with fp32 trainable weights · 52 optimizer steps (13 epochs × 4 steps, matched to run4's 51 so run5 vs run4 isolates data contrast).

Observed training metrics: rewards/margins 0.006 → 2.49 (run4: 0.042 → 0.105; run3: 0.171 → 0.429); rewards/chosen 0.002 → +1.21 (rising) with rewards/rejected 0.002 → −1.28 (falling) — proper preference separation, unlike run4 where both rose together; train rewards/accuracies 1.0 from epoch 2.

Ablation result (the important caveat)

On the 11 held-out eval records from run4's grouped split (seed 42), run5 is indistinguishable from base (margins in nats, from ablation_results_v2.json):

State	RS prompt	Generic	None
base	−12.09	−15.62	−14.61
run5	−12.21	−15.56	−15.30

Interpretation: 16 pairs bought deep memorization (train margins 2.49), zero generalization. Contrast determines whether learning happens; quantity determines whether it transfers. Next step: the 9×9 prompt/persona expansion (~45 pairs) at the same delta ≥ 3 gate, then retrain and re-ablate. The rs/generic gap stayed flat for every adapter across all ablations — the prompt-independent (experiential) mechanism is real; what has been missing is volume.

Quick start

from transformers import pipeline

generator = pipeline("text-generation", model="M1ztyk/SAIGE-dpo-v5", device="cuda")
output = generator([{"role": "user", "content": "I'm struggling with a difficult coworker."}],
                   max_new_tokens=256, return_full_text=False)[0]
print(output["generated_text"])

Note: the adapter changes generation only mildly on unseen prompts (see ablation above) — treat it as an experiment artifact, not a production model.

Training procedure

Trained with DPO (Direct Preference Optimization) via TRL.

Framework versions

• TRL: 1.13.0
• Transformers: 5.17.0
• Pytorch: 2.14.0
• Datasets: 5.0.1
• Tokenizers: 0.23.2

Citations

@inproceedings{rafailov2023direct,
  title        = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},
  author       = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn},
  year         = 2023,
  booktitle    = {Advances in Neural Information Processing Systems 36 (NeurIPS 2023)},
}

@software{vonwerra2020trl,
  title   = {{TRL: Transformers Reinforcement Learning}},
  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Kashif Rasul and Gallouédec, Quentin},
  license = {Apache-2.0},
  url     = {https://github.com/huggingface/trl},
  year    = {2020}
}