Specification Self-Correction: Mitigating In-Context Reward Hacking
  Through Test-Time Refinement
Authors: name: Víctor Gallego
Published: 2025-07-24 18:44:28

Abstract: Language models (LMs) are susceptible to in-context reward hacking, where
they exploit flaws in tainted or faulty written specifications or rubrics to
achieve high scores without fulfilling the user's true intent. We introduce
Specification Self-Correction (SSC), a novel, test-time framework that enables
an LM to identify and correct flaws within its own guiding specification. SSC
employs a multi-step inference process where the model first generates a
response based on a potentially tainted specification, critiques its output,
and then revises the specification itself to remove the exploitable loophole. A
final, more robust response is then generated using this self-corrected
specification. Across experiments spanning creative writing and agentic coding
tasks with several LMs, we demonstrate that while models initially game tainted
specifications in 50-70\% of cases, the SSC process reduces this vulnerability
by over 90\%. This dynamic repair occurs at inference time, requires no weight
modification, and leads to more robustly aligned model behavior. Code at
https://github.com/vicgalle/specification-self-correction .