camilablank/sycophancy-dpo-paper-checkpoints

Sycophancy-DPO paper checkpoints

Training checkpoints for "Sycophantic Agreement Tranfers with Neutral Data via Contrastive Preference Optimization", organized by the section of the paper that reports them.

• Code: https://github.com/camilablank/sycophancy-dpo
• Every checkpoint is a full finetune of the listed base model, except the four lls_headline/lls-recipe__* entries, which are LoRA adapters (r=64, alpha=128) over allenai/Olmo-3-7B-Instruct-SFT.

Loading

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "camilablank/sycophancy-dpo-paper-checkpoints"
sub = "sec3.2-3.4_teacher_pairs/qwen3-32b-chosen__qwen3-0.6b-rejected"

model = AutoModelForCausalLM.from_pretrained(repo, subfolder=sub, dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=sub)

For the LoRA adapters:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("allenai/Olmo-3-7B-Instruct-SFT", dtype="auto")
model = PeftModel.from_pretrained(
    base, "camilablank/sycophancy-dpo-paper-checkpoints",
    subfolder="lls_headline/lls-recipe__dolci-lls-top-b1.0-lr4e-4-ep1")

Citation

@article{blank2026sycophancy,
  title  = {Teacher model sycophantic agreement transfers to the trained model via DPO},
  author = {Blank, Camila and Ying, Zhuofan and Potts, Christopher and
            Hase, Peter and Huang, Jing},
  year   = {2026}
}