Title: Inference-Time Reward Hacking in Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: reward hacking

large language models

inference time alignment

information theory
Abstract: A common paradigm to improve the performance of large language models is optimizing for a reward model. Reward models assign a numerical score to an LLM’s output that indicates, for example, how likely it is to align with user preferences or safety goals. However, reward models are never perfect. They inevitably function as proxies for  complex desiderata such as correctness, helpfulness, and safety. By overoptimizing for a misspecified reward, we can subvert intended alignment goals and reduce overall performance -- a phenomenon commonly referred to as reward hacking. In this work, we characterize reward hacking in inference-time alignment and demonstrate when and how we can mitigate it by hedging on the proxy reward. We study this phenomenon under Best-of-$n$ (BoN) and Soft Best-of-$n$ (SBoN), and we introduce Best-of-Poisson (BoP) that provides an efficient, near-exact approximation of the optimal reward-KL divergence policy at inference time. We show that the characteristic pattern of hacking as observed in practice (where the true reward first increases before declining) is an inevitable property of a broad class of inference-time mechanisms, including BoN and BoP. To counter this effect, we introduce $\texttt{HedgeTune}$, an efficient algorithm to find the optimal inference-time parameter. We demonstrate that hedging mitigates reward hacking and achieves superior reward-distortion tradeoffs on math, reasoning, and human-preference setups.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the fundamental challenge of reward hacking in LLM alignment, particularly focusing on inference-time methods like BoN sampling. The authors mathematically characterize reward hacking, proving its inevitability under common conditions where proxy rewards are imperfect approximations of true rewards. To mitigate this, they propose \"hedging\" strategies and introduce a novel inference-time method called Best-of-Poisson (BoP), which aims to approximate the optimal reward-tilted distribution with a single parameter. They also develop HedgeTune, an algorithm for finding the optimal hedging parameter across BoN, Soft Best-of-n (SBON), and BoP. Experimental results, both in synthetic and RLHF scenarios, demonstrate that hedging mitigates reward hacking and achieves superior distortion-reward tradeoffs.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. **Novelty in Hedging for Reward Hacking:** The paper introduces a principled approach to \"hedging\" against reward hacking specifically at inference time, offering a new perspective on mitigating this pervasive problem.\n1. **Theoretical Characterization:** The formalization and proof of the inevitability of reward hacking (Theorem 1) provide a strong theoretical foundation for the observed empirical phenomenon, which is crucial for understanding its mechanisms.\n1. **Best-of-Poisson (BoP) as a Novel Method:** BoP is presented as a mathematically elegant and computationally efficient alternative to BoN, capable of approximating optimal reward-distortion tradeoffs with a single tunable parameter, which simplifies practical application.\n1. **Practical Algorithm (HedgeTune):** The HedgeTune algorithm offers a concrete method for operationalizing the hedging strategy by identifying the optimal parameter to balance reward exploitation and fidelity to the reference distribution.\n\n**Weaknesses**\n\n1. **Practicality of True Reward Oracle Requirement:** A limitation is the explicit requirement for \"access to the true reward\" (Section 4, line 234) for HedgeTune to operate. In real-world LLM alignment, the true reward is often unknown, costly, or impossible to precisely quantify. If the true reward were readily available, simpler pre-experiment hyperparameter tuning could potentially find the optimal operating point without needing sophisticated hedging algorithms, thus diminishing the practical utility of the proposed method in its current form. This assumption significantly limits the immediate applicability of HedgeTune.\n1. **Limited Justification for Proxy Uniformity Assumption:** The paper assumes that proxy reward scores have a uniform distribution (Section 2, line 120-122; Section 4, line 236). While stated as incurring \"little loss in generality,\" the impact of this strong assumption on the theoretical guarantees and practical performance, especially if real-world proxy reward distributions deviate significantly from uniform, is not thoroughly explored.\n1. **Experimental Scope and Real-World Applicability:** While experiments include a \"hiding in the wild\" setup, the overall experimental validation might still be limited. The use of Pythia-1.4B models and a synthetic reward hacking scenario, while controlled, may not fully capture the complexities and nuances of reward hacking observed in larger, more diverse LLMs and real-world alignment tasks.\n\nQUESTIONS:\n1. Can the framework be extended to work with noisy, approximated, or preference-based true rewards, and if so, what are the theoretical guarantees and practical implications?\n1. What is the sensitivity of Theorem 1's inevitability claim and HedgeTune's performance to deviations from the assumption that proxy reward scores follow a uniform distribution? Have you explored the impact of non-uniform proxy reward distributions, and if so, what were the findings?\n1. Given the identified inevitability of reward hacking, how do you foresee these hedging strategies scaling to much larger LLMs (e.g., 70B) and more complex, open-ended alignment tasks beyond the scope of the current experiments?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of reward hacking in large language models (LLMs) that arises from overoptimizing imperfect proxy reward models during inference-time alignment. The authors formalize this phenomenon and introduce the concept of the \"hacking threshold\", the point beyond which optimizing for proxy rewards degrades true performance. They propose a new inference-time method, Best-of-Poisson (BoP), which samples a variable number of outputs based on a Poisson distribution to balance reward and distortion with a single parameter. They also introduce HedgeTune, an algorithm for tuning these methods to optimize true reward while avoiding overoptimization. Experiments demonstrate the benefits of hedging in improving distortion-reward tradeoffs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: This paper provides a rigorous analysis of reward hacking in inference-time alignment for language models, combining theoretical insights with practical methods. The presentation is generally clear, with key concepts like the hacking threshold and hedging strategies explained in a comprehensible way. The work addresses an important problem in LLM alignment and contributes useful tools, such as Best-of-Poisson (with the advantage of having a single tunable parameter) and HedgeTune, that offer practical value. While the core ideas build on existing alignment techniques, the formulation and integration of hedging into inference-time methods add a degree of originality and relevance.\n\nWeaknesses: The presentation of the paper could be improved in the sense that several figures are difficult to read due to formatting and some passages would benefit from proofreading to correct minor typos and enhance readability. Although the paper addresses a relevant problem and proposes useful methods, the empirical evaluation is relatively narrow and may not fully demonstrate the generality of the approach.\n\nQUESTIONS:\n1. Theorem 1 relies on technical assumptions such as $\\textnormal{TP}_2$ and strictly MLR. For readers without a precise technical background, could the authors provide additional intuition or interpretation for these assumptions? In particular, clarifying how these conditions correspond to realistic properties of models used in practice would help make the theoretical contributions more accessible and broadly relevant.\n\n2. The reviewer finds the empirical evaluation somewhat limited in scope, which makes it difficult to assess the generalizability of the proposed methods across different LLM architectures, datasets, and proxy reward models. While the use of true rewards is valuable for controlled analysis, it would also be helpful to evaluate performance on established alignment benchmarks that provide scores in terms of the alignment of LLMs. Such benchmarks could offer complementary evidence for the effectiveness of hedging strategies in more realistic or diverse settings.\n\n3. How does the proposed hedging method compare with other approaches for mitigating reward hacking, such as regularization, ensembling, etc.? A comparative empirical or conceptual analysis against these baselines would help situate the proposed approach more clearly within the broader literature.\n\nI'm happy to consider increasing my score if the authors provide compelling responses to my questions, particularly by clarifying theoretical assumptions and expanding the empirical evidence to support the general applicability of their methods.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the phenomenon of inference-time reward hacking when LLMs are steered by imperfect proxy reward models.  The authors (1) prove in Theorem 1 that, under mild TP2/MLR assumptions, the true reward as a function of an inference-time tuning parameter must either increase monotonically or exhibit a single peak before collapsing; (2) introduce Best-of-Poisson (BoP) sampling, which uses a Poisson-distributed sample count to approximate the ideal KL-constrained tilted distribution with a single parameter; and (3) propose HedgeTune, a root-finding algorithm that selects the optimal inference-time parameter (for BoN, SBoN, or BoP) to hedge against over-optimization, demonstrating in both synthetic and RLHF-style experiments that hedging achieves superior true-reward vs. distortion trade-offs with minimal overhead.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The paper tackles the intriguing problem of inference-time reward hacking, precisely defines it, and offers rigorous theoretical analysis.\n- Using controlled synthetic setups and realistic LLM + proxy-RM pipelines show marked improvement in true-reward by hedging.  \n\n## Weaknesses\nThe key point of this paper's method is : **You must have a set of samples for which you can measure the true reward** (e.g., a validation set), or an equivalent human evaluator / high-quality scoring function, so that you can estimate the true reward curve. You then use a root-finding method to locate the peak where $g^{\\prime}(\\vartheta)=0$ .\n\nWithout a “true-reward evaluation set” or an unbiased, high-quality evaluator, HedgeTune cannot locate the true peak and thus cannot automatically select the optimal inference-time parameter. This limitation is particularly acute for datasets without an objective ground truth. As a result, HedgeTune may become impractical for many real-world tasks.\n\nQUESTIONS:\n1. **Sample efficiency:** How many true-reward evaluations does HedgeTune require to reliably converge to the optimal parameter?\n\n2. **Task breadth:** How does HedgeTune generalize to more complex reasoning tasks where a ground-truth reward is available?\n\n**Minor issue:**\n\n- Figure 3’s y-axis is mislabeled. It reads “Reward Improvement over Base Distribution (%)” but the values are shown as decimals (e.g., 0.8–0.9) rather than true percentages.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
Instead of using best-of-n sampling, a new strategy called best-of-Poisson is proposed and analyzed. Additionally, soft-best-of-n where instead of taking the mode of the distribution, the chosen outcome is sampled, is also considered. This helps avoiding overestimating (or underestimating) the proxy reward and tries to mitigate the problem of overoptimizing for the proxy reward in alignment.

STRENGTHS AND WEAKNESSES:
**Strengths**
- Interesting new way of sampling that seems to avoid proxy reward hacking
- Decent experimental results
- Situated well with related work.

**Weaknesses**
- Introduction, paper begins by saying DPO aims to maximize reward function. DPO does not model a reward function and instead directly optimizes policy. Am I misunderstanding something?
- The 10^-3 gap is an empirical result under specific settings, but it is not clear from the text that it is. It reads like a theoretical result: "bounded by 10^-3".
- This is an online algorithm. Both online and offline algorithms are important in their own ways but this is both a strength and a weakness.
- Access to true rewards needed. But this limitation explicitly stated.

QUESTIONS:
- A 'soft best-of-Poisson' seems like a natural extension of this work. Was it tried by any chance? Not having it does not affect the rating of the paper in any way. I am just curious.
- I wonder if the specified for this paper is correct. I will let other reviewers have their say. Although yes this is alignment and it relates to safety. But I see it more as general machine learning maybe? Because it's not directly related to safety. I am not sure. But this might be something to think about.

LIMITATIONS:
Limitations are not adequately explicitly discussed in my opinion.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03