Sheppp/lcb-correctness-classifiers

LCB correctness classifiers (v2)

Yes/no code-correctness classifiers for LiveCodeBench rollouts, one LoRA per generator model, all on Qwen/Qwen3-8B (thinking disabled). Trained on rollouts from samuki-hf/temperature-sweep-data and samuki-hf/thinking-rollouts.

Prompt = problem statement + candidate solution (+ tail of the generator's reasoning trace for the -trace variants) + "Does this candidate solution correctly solve the problem? Answer Yes or No." Score = log-odds of the Yes-token family vs No at the first answer position (chat template with enable_thinking=False — the empty <think></think> block is part of the prompt). Training: natural label distribution, problems split 500/60/154 (stratified by difficulty × testtype × pooled pass-rate), ≤100 samples/problem across all temps, LoRA r16/α32, lr 1e-4, effective batch 128, 1 epoch.

subfolder (classifier for)	input	AUROC	acc@0	ECE
qwen25-coder-3b-instruct	code	0.968	0.926	0.036
qwen25-coder-7b-instruct	code	0.921	0.853	0.085
qwen3-4b-think	code	0.887	0.877	0.052
qwen3-14b-think	code	0.888	0.889	0.055
qwen3-4b-nothink	code	0.895	0.823	0.116
qwen3-14b-nothink	code	0.919	0.828	0.096
qwen3-4b-think-trace	code + trace	(eval pending)
qwen3-14b-think-trace	code + trace	(eval pending)

Metrics: 154 held-out problems, full candidate pools (all samples × temps), scored with the same prompt path as training.

from peft import PeftModel
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B")
model = PeftModel.from_pretrained(model, "Sheppp/lcb-correctness-classifiers", subfolder="qwen25-coder-7b-instruct")