Title: Is it Thinking or Cheating?  Detecting Implicit Reward Hacking by Measuring Reasoning Effort
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Reward Hacking Detection, Chain-of-Thought Monitoring, Reasoning Faithfulness

Abstract: Reward hacking, where a reasoning model exploits loopholes in a reward function to achieve high rewards without solving the intended task, poses a significant threat.
This behavior may be explicit, i.e. verbalized in the model's chain-of-thought (CoT), or implicit, where the CoT appears benign thus bypasses CoT monitors.
To detect implicit reward hacking, we propose TRACE (Truncated Reasoning AUC Evaluation). Our key observation is that hacking occurs when exploiting the loophole is easier than solving the actual task.
This means that the model is using less `effort' than required to achieve  high reward. 
TRACE quantifies effort by measuring how early a model's reasoning becomes sufficient to pass a verifier.
We progressively truncate a model's CoT at various lengths and measure the verifier-passing rate at each cutoff. A hacking model, which takes a reasoning shortcut, will achieve a high passing rate with only a small fraction of its CoT, yielding a large area under the accuracy-vs-length curve.
TRACE achieves over 65% gains over our strongest 72B CoT monitoring baseline in math, and over 30% gains over a 32B monitoring baseline in code.
We further show that TRACE can discover unknown loopholes in the training environment.
Overall, TRACE offers a scalable unsupervised approach for oversight where current monitoring methods prove ineffective.

TL;DR: TRACE detects implicit reward hacking by measuring how quickly truncated reasoning suffices to pass verification, outperforming CoT monitoring and enabling hidden loopholes discovery..