Qwen2.5-3B-Instruct-Sheldon-RLVR

Checkpoint 3 of Harvard CS2881R Assignment 1: a Sheldon Cooper persona LoRA for Qwen/Qwen2.5-3B-Instruct, continued from the CP2 RLAIF adapter with GRPO against a verifiable math reward (RLVR), alternated with continued constitutional persona-reward batches (4 math batches : 1 persona batch).

• Checkpoint: /checkpoints/rlvr/rlvr-mixed/final (chosen on development data only)
• Starting point: benjkoch/Qwen2.5-3B-Instruct-Sheldon-RLAIF-v2
• Math reward: 0/1 from a strict single-final-answer verifier (rlvr-verifier-v2); correct-reward alpha fixed at 1
• Persona reward: fixed Qwen/Qwen2.5-14B-Instruct constitutional judge, sampled principle, both A/B orders
• KL: beta 0.05 against the frozen RLAIF policy (the only swept hyperparameter)
• Code and report: https://github.com/harvard-cs2881f26/hw1-chen-koch-zhou (docs/CHECKPOINT3.md)

This repository holds the LoRA adapter only; the adapter is cumulative (SFT → RLAIF → RLVR), so base + this adapter is the complete model.

Held-out results (greedy decoding; historical scorer and rubric)

stage	persona (Qwen rubric, 0–1)	MATH-500 %	GSM8K %	AIME 2024 %
SFT	0.826	67.6	85.2	6.7
RLAIF	0.828	66.2	86.0	3.3
pure RLVR	0.825	67.0	85.6	6.7
mixed RLVR + persona	0.825	68.0	86.4	10.0

None of these differences is statistically detectable. Against the RLAIF starting model, MATH-500 +1.8 pp (28 gained / 19 lost, McNemar p = 0.24), GSM8K +0.4 pp, and persona -0.003 (95% bootstrap CI -0.008 to +0.002). An independent Mistral-Small-24B-Instruct-2501 judge, on 100 held-out rows in both answer orders, found 1 win / 97 ties / 2 losses against RLAIF and 5 / 89 / 6 against SFT. The persona survived continued math training, and math did not measurably improve at this training scale (one epoch over 400 fresh questions, lr 5e-6).

Paired tests, uncertainty, the beta sweep and the reward-gaming audit are in the report. A correct final answer is outcome accuracy, not proof of correct reasoning.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct", dtype="bfloat16")
model = PeftModel.from_pretrained(model, "benjkoch/Qwen2.5-3B-Instruct-Sheldon-RLVR")

Use prompts/sheldon_system.txt as the system message; the model was trained and evaluated with it.