Raiff1982/codette-newton-star-hard

newton-star-hard — ⚠️ Research Artifact: Below-Baseline Result

Second arm of a controlled STaR study. Trained on 350 keep-correct reasoning chains from difficulty-matched MMLU-Pro STEM (61% keep yield — the data genuinely challenged the model). Scored 28.0% on GPQA-main vs the untrained baseline's 34.0%: harder data attenuated the degradation seen with easy data (25.0%) but did not eliminate it.

Notable provenance: trained free on a Kaggle T4 after working through six documented environment issues (all fixes committed to the repo), and one literal household power failure. Where there's a will.

The STaR Study (GPQA-main, reason mode, n=100 per arm)

Adapter	Training data	GPQA	Verdict
newton (untrained baseline)	—	34.0%	reproduced to the decimal, 4 days apart
newton-star-r	350 keep-correct + 180 rationalized	benchmark in progress	complete STaR method
newton-star-hard	350 MMLU-Pro STEM keep-correct	28.0%	attenuated the harm, below baseline
newton-star	500 easy-science keep-correct	25.0%	regressed to chance

Finding: keep-correct self-taught reasoning consolidates existing ability rather than extending it — training data difficulty orders the outcome perfectly, but no keep-correct arm matched the untrained baseline. Full methodology, controls, and changelogs: Codette-Reasoning (see docs/CHANGELOG_2026-07-09.md, docs/CHANGELOG_2026-07-11.md).

Created by Jonathan Harrison (Raiff1982) · Raiff's Bits LLC