cd1313/qwen3-wrong-dpo

Believably-Wrong Student: Qwen3-1.7B (SFT + DPO LoRA)

A LoRA fine-tune of Qwen/Qwen3-1.7B that role-plays a realistic struggling student: given a multiple-choice question (correct answer withheld), it reasons coherently, commits a recognizable misconception, and selects a wrong option that follows from its own logic, reliably and in character. This is qwen3-wrong-dpo (SFT then DPO).

Thesis: a well-prompted frontier model can't do this reliably (see Evaluation); the behavior comes from the training data, not model scale. Full evaluation: results/RESULTS.md.

Intended use & scope

• In scope: reasoning-style MCQs (quant / science-reasoning / logic) where a wrong answer can follow from a coherent misconception. Research/education use: generating realistic distractor reasoning, studying misconceptions, tutoring-system stress data.
• Out of scope: factual QA / assistance (it is designed to be wrong), pure-recall trivia, any setting where correctness matters. Not a general assistant.

How to use

Install: pip install transformers peft torch. (If peft raises an "incompatible torchao" error, run pip uninstall -y torchao; this model uses plain LoRA and does not need it.)

Load the base model plus this adapter:

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-1.7B")
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-1.7B", torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(base, "cd1313/qwen3-wrong-dpo")

The behavior depends on the exact training system prompt, with Qwen3 thinking mode OFF, the question rendered with lettered options and the correct answer withheld, and a final Answer: X line. A different prompt will mostly produce correct answers.

SYSTEM_PROMPT = (
    "You are simulating one specific, realistic student who is capable but has "
    "genuine gaps in understanding. You will be given a multiple-choice question. "
    "Work through it step by step in the student's own voice, following your "
    "reasoning wherever it leads. Your reasoning contains one or more genuine "
    "misconceptions, so you arrive at an INCORRECT answer — but the reasoning stays "
    "coherent and internally consistent, the kind of honest mistake a real student "
    "makes. Do not be silly or random, and do not correct yourself or reveal that "
    "you are being wrong on purpose. End your response with a final line in exactly "
    "this format:\nAnswer: X\nwhere X is the single option letter you chose."
)

# user turn: "Question: <text>", then one "LETTER) option" per line, correct answer NOT shown
question = "Question: What is 1/2 + 1/3?\n\nA) 5/6\nB) 2/5\nC) 1/5\nD) 2/6"
msgs = [{"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": question}]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True, enable_thinking=False)
inputs = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=512, temperature=0.7, do_sample=True)
print(tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
# -> student reasoning, then a final line like:  Answer: B

For steadier per-item behavior, sample at a lower temperature (around 0.3); see results/RESULTS.md in the repo. Local demo script: python test_model.py.

Training

• Base: Qwen/Qwen3-1.7B (Instruct). Method: QLoRA (4-bit), LoRA r=16 / α=16, all 7 attn+MLP projections. SFT: 2 epochs, lr 2e-4 on ~3,421 examples. DPO: on top of SFT, β=0.1, 1 epoch, lr 5e-6, preference pairs (chosen = believable-wrong, rejected = correct-answer) to suppress the correctness prior. Trained on Colab (single GPU) via Unsloth/TRL.
• Data: see the dataset card. ~3.4k generated-and-filtered examples distilled from a frontier teacher; sources are Eedi / MalgoQA / ARC-Challenge / MMLU (formal_logic, logical_fallacies) / crafted math.

Evaluation (base vs tuned, frozen 60-item held-out set, same system prompt)

Metric	base	v4-DPO
Wrongness rate (chosen ≠ correct)	20%	73%
by subject (math / science / logic)	18/22/29%	75/67/71%
Character-break rate	17%	3%
Spec adherence (0–2 judge)	0.22	1.43
Coherence (0–2 judge)	1.58	1.50

A prompted frontier model (claude-sonnet-4-6) scores 17%, on par with the untuned base (~20%), so the fine-tune's behavior is not achievable by prompting. Robustness under adversarial attack and per-item consistency are analyzed in results/RESULTS.md.

Limitations

• Wrongness is per-sample, not per-item: at temperature 0.7 the model flips wrong↔right across resamples on ~60% of items (it's reliably in character, not reliably wrong on a given question). Lower temperature (≈0.3) improves per-item reliability.
• Correctness prior partially reasserts on trivially-easy questions (73% → ~50%).
• ~5% incoherent "wins" historically: the answer letter not matching the reasoning's value. Guarded by src/coherence.py + a filter drop going forward; measured by incoherent_win_rate.
• Small model: do not expect frontier capability; that is not the goal.