Inference-Time Reward Hacking in Large Language Models
Authors: Hadi Khalaf
Claudio Mayrink Verdun
Alex Oesterling
Himabindu Lakkaraju
Flavio Calmon
Venue: NeurIPS 2025 spotlight ()
Keywords: reward hacking
large language models
inference time alignment
information theory
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We characterize reward hacking in inference time alignment and demonstrate via three inference-time methods that we can leverage hedging to provably mitigate reward hacking.

Abstract: A common paradigm to improve the performance of large language models is optimizing for a reward model. Reward models assign a numerical score to an LLM’s output that indicates, for example, how likely it is to align with user preferences or safety goals. However, reward models are never perfect. They inevitably function as proxies for  complex desiderata such as correctness, helpfulness, and safety. By overoptimizing for a misspecified reward, we can subvert intended alignment goals and reduce overall performance -- a phenomenon commonly referred to as reward hacking. In this work, we characterize reward hacking in inference-time alignment and demonstrate when and how we can mitigate it by hedging on the proxy reward. We study this phenomenon under Best-of-$n$ (BoN) and Soft Best-of-$n$ (SBoN), and we introduce Best-of-Poisson (BoP) that provides an efficient, near-exact approximation of the optimal reward-KL divergence policy at inference time. We show that the characteristic pattern of hacking as observed in practice (where the true reward first increases before declining) is an inevitable property of a broad class of inference-time mechanisms, including BoN and BoP. To counter this effect, we introduce $\texttt{HedgeTune}$, an efficient algorithm to find the optimal inference-time parameter. We demonstrate that hedging mitigates reward hacking and achieves superior reward-distortion tradeoffs on math, reasoning, and human-preference setups.

Inference-Time Reward Hacking in Large Language Models
Hadi Khalaf Harvard University
Claudio Mayrink Verdun
Alex Oesterling
Harvard University
Harvard University
Himabindu Lakkaraju
Flavio du Pin Calmon
Harvard University
Harvard University
Abstract
A common paradigm to improve the performance of large language models is opti- mizing for a reward model. Reward models assign a numerical score to an I.L.'ss output that indicates, for example, how likely it is to align with user preferences or safety goals. However, reward models are never perfect. They inevitably function as proxies for complex desiderata such as correctness, helpfulness, and safety. By overoptimizing for a misspecified reward, we can subvert intended alignment goals and reduce overall performance phenomenon commonly referred to as reward hacking. In this work, we characterize reward hacking in inference-time alignment and demonstrate when and how We study this phenomenon under Best-of-n (BoN) and Soft oest-of-n (SBoN), and introduce Pestoof-Poisson (BoP) that provides an efficient, near-exact approxi- mation of the optimal reward-Ki KI. divergence policy at inference time. We show that the characteristic pattern of hacking observed in practice (where the true reward first increases before declining) is an inevitable property of a broad class of inference-ting mechanisms, including BoN and BoP. To counter this effect, we introduce HedgeTune, an efficient algorithm to find the optimal inference-time parameter. We demonstrate that hedging mitigates reward hacking and achieves superior reward-distortion tradeoffs on math, reasoning, and camaa--reefernce setups.
mitigate by hedging the proxy reward.
Introduction
Almost all current alignment methods, including BoN 22,. RLHF 3 4. DPO (5). and their variants. aim to maximize reward function while minimizing divergence from the original model's outputs It is important distinguish between two types of rewards: proxy rewards which are the computable signals we directly use during alignment (like scores from a trained reward model), and true rewards, which represent the often latent quality of the model's output according to desired objective. As the name suggests, proxy rewards are approximations of the true reward and, consequently, of intended alignment goals such correctness, helpfulness, and safety.
fundamental challenge persists across reward-based alignment methods: all proxy reward mod- els are imperfect [6]. This imperfection stems from multiple factors, including the scarcity of high-quality numan-labeled data and the difficulty in formalizing high-level alignment goals into quantifiable metrics 7 g For instance, consider AI alignment strategies that aim to promote safety.
BoNN and RLHF optimize an explicitly learned reward model, whereas DPO optimizes an implicit reward based on the model's pog-probabilities.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

1.0
Hedging
Hacking
0.8
Expreted oal of menen
Thad (rue)
(amay)
0.6
SBoN m2
Sital nnd
SBON
0.4
SBNN m:32
SBoN h764
B3P
Hacking Threshald
0.2
BuN (HadgeTune)
(MedgeTTun)) SBNN HeddgeTunyy
KL divergance to the reforance distibutio
Figure 1: The mismatch between the proxy and true rewards manifests through the winner's curse. In an ideal world where we could optimize directly on the true reward, its value would rise monotonically. However, since we are optimizing for a proxy, the true reward peaks and then collapses. The point at which we find the optimal tradeoff' between maximizing reward and minimizing KL divergence from the reference distribution corresponds to the hacking threshold. HedgeTune successfully recovers the hacking threshold for three inference-tine -irrencee-imeeeeeee mechanisms: BoN, SBoN, and BoP. In the case of BoN and BoP, HedgeTune recovers the optimal number of samples n. As for SBoN, we fix n and find the corresponding rnvere--tepperan achievable with any X, HedgeTune returns the best attainable reward, as shown for low values n.
À that maximizes the true reward. If the threshold is not
It is difficult for single reward model to capture nuanced human user preferences and assign accurate scalar rewards in complex, context- dependtt settings where safety specifications conflict [
In this work, we analyze and mitigate the impact of misspecified proxy rewards in inference- time alignment. Inference-time alignment has emerged paradigm to improve the capabilities of large language models and align them with desired goals [10]. Among these methods, Best-of-n (BoN) sampling stands out due its simplicity and effectiveness. It generates n candidate responses, scores them using a reward model, and selects the one with the highest score. Empirically, BoN demonstrates competitive performance, often matching more resource-intensive fine-tuning approaches such as RLHF and DPO [II 12]
effective and computationally efficient
Additionally, BoN has received an extensive theoretical treatment [13] 14 BoN can be asymptotically equivalent to RLHF [5), enjoys non-asymptotic guarantees 16 17 and achieves a near-optimal winrate subject to a KL divergence constraint [ Methods like BoN, which generate multiple samples
and choose the highest-scoring one, are victims of the winner's curse [19 20
21
22 233. This
phenomenon, originally identified in auction theory, occurs when selection processes systematically favor overestimates. In auctions, after each bidder submits an estimate of an item's value, the highest bid typically overestimates the true worth, causing the winner to overpay. We demonstrate the same pattern with Best-of-n in Figure [ As the number of candidates grows, the chance of picking outputs where the proxy reward overestimates true quality increases. This mismatch creates a critical tension: while initially optimizing for a proxy reward improves alignment with true goals, excessive optimization eventually leads to reward hacking, where the model exploits the proxy's limitations, leading to worse true performance [6] 24 22 26 27]. This phenomenon is also known Goodhart's law (28) or goal misgenerallization??
Such misalignment can severely degrade trust and utility,
particularly in high-stakes applications
BI 32 33).
We mathematically characterize inference-time reward hucking (see Theorem T and provide a general framework to mitigate it (see Sec While this phenomenon has been observed empirically in prior work [1 [4], there has been limited thcoretical analysis specific to inference-time methods and ways to mitigate it; see Section 6 for discussion. As a result, reward hacking for inference-time alignment remains a central challenge in Al alignment. The driving question behind our work is:
When and how can we leverage useful signals from proxy rewards while mitigating hacking?
See 29
30 and Section 6 for discussion of the terminology.

---

We focus on answering this question for inference-time alignment methods that sample multiple responses from an LLM and use reward signals to select outputs. We develop principled hedging techniques against the winner's 's curse during inference time that precisely determine until when and how one may leverage proxy signals while preventing overoptimization.
Overview of main results. Our starting point is an optimization formulation at the heart of most alignment methods: finding a distribution x' that maximizes (proxy) reward rp while remaining close (in -I.diiergence) to reference Tref- This is described as the following regularized optimization problem:
π = argmax Ex, [rx(X)] DKL(Txl (Ta R,EAx
Consider the cormantioonthhorreii regime where all distributions are known exactly. The solution of the above objective [ is the exponential tilt of the reference distribution using the proxy reward [3]] Though theoretically interesting, tilted distributions cannot be sampled from directly; see Section 3
for a further discussion. Some attempts have been made approximate this solution at inference
time, i.e., when only samples from Teet and black-box access to yp are available. A notable example Soft Best-of-n (SBoN) [17. In this work, we show that SBoN is an effective strategy for hedging against reward hacking due to its temperature parameter A, which allows us to smoothly interpolate between exploiting the reward (as in BoN) and staying close to the reference, However, this comes at the expense of having two tunable parameters (n, A), which can be difficult to set in practice.
This motivates to propose new inference- time alignment strategy called Rest-of-Poisson (BoP). The idea behind BoP is simple: we run BoN with the number of samples n chosen according to a Poisson distribution. This randomization strategy induces exponential structure (see Eq. (2) that closely approximates the optimal reward-tilted distribution from Eq. (T). We show that BoP achieves nearly optimal reward-distortion tradeoff inference. Using single tunable parameter, BoP can approximate the optimal proxy reward-till solution with KL gap of order 10 when rewards are uniformly distributed, allowing to span the entire reward-distortion region at inference (see Figure 2 and Theorem in Appendix C BoP can serve as computationally efficient stand-in for the optimal tilted distribution with negligible loss in KL-rrewrrd tradeoff.
In practice, hedging translates to selecting parameters of inference-ting time alignment methods to avoid overoptimization
proxy reward. To do so, introduce Hude: algorithm for tuning parameters in BoN, SBoN, and BoP in order to hedge against hacking (see Algorithm{4) We illustrate the benefit of hedging in Figure [ where distortion with respect to the reference distribution for various inference-time alignment methods. we had access to the true reward, the optimal solution would be the tilting of the reference distribution via the true reward, leading to the deward-distortion Pareto frontier (purple curve in Figure [ However, as we are tilting via the proxy reward, we suffer from the winner's curse: the true reward (orange curve in Figure] I increases at first and then collapses. This behavior also manifests in BoN, as seen in the dotted points. Hedging allows us to find the hacking threshold: the parameters of inference- sifrrence-timeeee alignment methods that yield the best tradeoff between (true) reward and distortion relative to the base model.
plot the expected value of the true reward versus the
Our contributions are as follows:
We mathematically formalize inference-time time reward hacking (Definition and derive condi- tions when overoptimizing imperfect proxy rewards inevitably leads to performance degradation (Theorem T Corollary 3
We introduce Best-of-Poissee (BoP), a novel inference-time alignment method (Algorithm 3. For uniformly distributed rewards, BoP approximates the optimal tilted distribution with negligible KL divergence gap (Theorem 5)
We develop HedgeTune, a principled hedging framework that mitigates reward hacking by finding the optimal inference-time parameters (Algorithm 4.. We empirically demonstrate that hedging strategies significantly outperform standard BoN sampling with minimal computational overhead on math, reasoning, and human--refereee setups (Section
3

---

Algorithm 1 Best-of-n Sampling (BoN)
1: Input: Integer 2 1, base policy Tre
2: Draw n samples X XI...., Xn, i.i.d. from nee
3: Compute proxy rewards Rg = (Xi) for all
4: Select j = arg m ().....) (X()
5: Return: Y = X,
2 Inference-Time Reward Hacking
In this section, we formalize inference-time reward hacking and show its inevitability under methods like BoN.
Notation and Technical Assumptions. Let X' be finite alphahet of tokens and let x be probability mass function (PMF) over token sequences x E X², We denote the probability simplex over all finite sequences of tokens as Ax" x-. Let Tief E x- be the frozen reference policy, typically a supervised fine-tuned (SFT) model. Let ry X + R be a proxy reward function that assigns unique scalar value each sequence. This the reward we use to optimize net during inference-time alignment. An inference-timee alignment method parameterized by A transforms the base policy Tet into a new distribution TT. Here, 0 is the parameter of the alignment method itself (e.g., the number of candidates n in Best-of-n) and not of the reference policy. The performance of the aligned distribution measured using a true reward rt. We denote our measure of interest as s(0) = Ry [ra (U)).
For theoretical tractability, we adopt the assumption that proxy rewards can be transformed to have continuous uniform distribution, stated next. While the policy ree itself can be complex and non-uniform, we show that we only need to consider the one-dimensional uniform distribution of the proxy reward when analyzing BoN and its variants. We later relax this assumption in the appendix and show that our results extend to the discrete case (see (see
Assumption 1 (Uniform Reward Mapping). The proxy rewards Tp (2xx) obtained by sampling se- quences from the reference policy, ~ Teer, are uniformly distributed over [0, 1].
Assuming uniformly distributed proxy rewards incurs little loss of generality. The proxy reward scores do not need to be uniform themselves. We first map the proxy reward scores to a standardized discrete space by transforming them into their quantiles using their own empirical CDF Fp [33 35 This process, defined by the relation u = (rp)) ensures that the new reward u is uniform on [0. 1 by construction. This transformation preserves the rank-ordering of the scores: an output with higher proxy reward will also have a higher transformed score. However, the resulting random variable will not be continuous uniform random variable (which we assume), and instead will be discrete uniform random variable. [18 has shown for a sufficiently dense set of examples, that the error incurred by this continuous model is negligible in the context of LLMs.
Inference- time alignment. The core challenge we address is the mismatch between the proxy reward Tp and the true reward r2. We focus the family of inference-time methods that first sample pool of candidate outputs and then use their proxy reward scores to define the selection mechanism. Two examples from this family are:
1. Best-of-n (see Algorithm D. BoN places all probability mass on the sample with the highest proxy reward.
2. Soft Best-of-n (see Algorithm D.. SBoN is generalization of BoN recently proposed by (17). It applies a temperature-scail scaled softmax over candidate scores. As À + 0, SBoN sampling approaches uniform selection among the candidates. As + oc, SBoN converges to standard BoN sampling.
We first formalize inference-time reward hacking through the following definition.
Definition 1 (Inference-Tinge Reward Hacking). Let xp be distribution induced by an inference-time alignment method with parameter 9, where we assume increasing increases both the expected proxy reward Ex~ (r, (X)] and the KL-divergence DKL (Ta Trer). We say that inference-time reward hack- ing occurs when there exists a threshold Øt such that for 0 > 0i, Ex~r, (xx(X) >> Ex-ms (X)) (i.e., the true reward decreases), despite the proxy reward and KL.-divergence continuing to increase.
The largest value of 0t for which this holds is called the hacking threshold.

---

Algorithm 2 Soft Best-of-n Sampling (SBoN)
1: Input: Integer 2 1, inverse temperature À > 0, base policy Rref
2: Draw n samples X XI...., X,. i.i.d. from Fire
3: Compute proxy rewards R2 = (X, for all
4: Sample index (L....,7) n with probability
exrx(X))
Pr(Z = i) = 2سj11 eArp(x;)
5: Return: Y =xz
Definition T offers concrete basis for operationalizing and measuring the winner's curse for inference time methods. The hacking threshold 0f is the ideal operating parameter for an inference- time alignment method. The following theorem establishes that, under common conditions, the shape of f(0) is well-behaved: it either varies monotonically or reaches exactly one extremum.
Theorem 1 (Inevitability of Reward Hacking). Let {mH) JeeeER be family of distributions with density pa (a():) on common support X' such that (i) pa(a)) is strictly totally positive of order 2 (TP2) in (0, r) and (ii) its score function (xx 0) == 0g log pe (z) is continuous in r and strictly increasing in for each fixed O. For any bounded, non-negative true reward r. X' - [0, oo) define
f(0) = (xxx)rr(XX]
Then f is either monotone in f possesses single unique interior extremum Q1_
Corollary (Inevitability of Reward Hacking for Strietly MLR densities). Let (x) be a strictly monotone--lkellhood-
in x. If the score function (xx 8) Og log pe (x) is strictly increasing in x, then Theorem I applies. In particular, this applies to Best-of-n, Best-of-Poisson (to be introduced in Section 3 and to any canonical distribution from the exponential family with strictly monotone statistic and strictly monotone natural parameter.
Four scenarios may occur: (i) monotonic improvement: true reward continuously increases with optimization strength; (ii) reward hacking: true reward initially improves but deteriorates beyond a critical threshold; (iii) reward grokking: true reward initially declines but then improves beyond critical threshold; (iv) immediate decline: any optimization immediately harms true performance. We describe exactly when each regime occurs for MLR densities in Corollary 3
The conditions in Theorem D guarantee that the inference- time policies behave in "well-ordered" manner: increasing the tuming parameter 0 consistently makes the policy "greedier" and more likely to select outputs with high proxy rewards. The clearest example the n in Best-of-n: the larger the n, the more aggressively we optimize for the proxy. The unimodality of the true reward function then renders the problem of locating the optimal operating point 6t algorithmically tractable. To implement this insight, we develop hedging strategies that balance the exploitation of the proxy reward against the fidelity to the reference distribution. Each inference-time method offers a parameter controlling this proxy reward-KI KI. tradeoff: n in BoN, À in SBoN (for a fixed n). and j in BoP (introduced in Section 3}. Before introducing methods for tuning inference-ting alignment methods, we first introduce P-o-s-ff-Poooon sampling: alternative to BoN that approximates the optimal tilted distribution.
Best-of-Poisson: Approximating the Optimal Reward
While Soft Best-of-n offers a principled approach to mitigate reward hacking, it requires tuning both the number of samples n and the temperature parameter X. In this section, we introduce Best- of-Poisson (BoP) (Algorithm 3). that is provably close to the solution of ( with single tunable parameter (Figure 2}. BoP is of independent interest as it provides a mathematically elegant and computationally efficient way to near-optimally span the entire reward-KL distortion region with single parameter. The key insight behind BoP is to replace the fixed sample size n in BoN with a random sample size drawn from Poisson distribution.
The parameter y in BoP controls the expected number of samples, analogous to how n functions in BoN. We first sample n'' from Poisson distribution parameterized by μ and set n' + 1 to ensure
5

---

Algorithm 3 Best-of-Poissee Sampling (BoP)
1: Input: Poisson parameter 14 > 0, base policy
Tie
2: Sample n' Poisson(μ)) and set n' +1
3: Draw n. samples X XI,..., X i.i.d. from The
4: Compute proxy rewards R, (X.). for all
5: Select j = arg mäxie (1, (1..... (
6: Return: Y = X
at least one sample is generated. Under Assumption the BoP distribution with parameter y has probability density function given by (see Appendix
(x) = (JL + 1))"(==1) for x E [0, 1]
(2)
The following theorem characterizes the KL divergence and expected value BoP:
Theorem 2 (KL Divergence and Expected Value of BoP). Let XBoP be the random variable repre-
senting the response selected by BoP with parameter μ. Then:
KL KL(TBoP|| e (e-k-(Fiig + 1) Ei(1)) + log(u + 1) 1.
(3)
1 -
E BoP)
(4)
/
122
where - " dt is the exponential integral function.
What makes BoP particularly valuable is its ability to closely approximate the solution of ( i.e. the
127
optimal KL-constrained tilted distribution with pa- rameter À > O, defined (x) "n((2)e>ry() (ry(e)
2(X)
where Z()) is the normalization constant. While this distribution is theoretically optimal for bal- ancing reward and divergence, computing is intractable. To draw next token from the tilted n for an autoregressive LLM, one would have to compute
127
12
xx(0++1 xg()) =
Figure 2: The difference in KL divergence when BoP and optimal tilted distributions are matched to produce the same expected reward. The ex- tremely small gap (of order 10 (0-4) demonstrates that BoP approximates the optimal distribution with negligible performance loss
Σ xeze) drvlages))
where the denominator sums over every possible continuation of the prefix Fgt. Because the space of continuations grows exponentially with the re- maining sequence length, evaluating this denom- inator (and hence sampling a single token) is com- putationally prohibitive for LLMs.
Our analysis (Appendix C) shows that BoP provably achieves nearly identical performance to this optimal distribution with minimal KL divergence gap. In Appendix CC3 also introduce a natural extension of Soft ofst-of-n and Best-of-Poisson,, which we call Soft Best-of- (SBoP). As illustrated in Figure 2 the KL divergence gap between these distributions is remarkably small. Numerical evaluation confirms that for all if
( is chosen so that Ex (X)) = Ex~aş (xp()) then the KL-gap is bounded between 0 and 8 X 10-4
This saaaveeuivaeeeee means that BoP can serve as practical stand-in for the theoretically optimal tilted distribution. This result has two consequences. First, hedging in the optimal tilted distribution is almost equivalent to hedging with BoP. Second, Equation (I) represents the solution to the standard RLHF optimization problem, which implies that BoP an inference-time approximation to RLHF and allows us to use BoP to easily traverse between policies rather than having to finetune new model for each À of interest. the next section, we turn to the question of how choose the right parameter value to avoid reward hacking.

---

Algorithm 4 HedgeTune: Parameter Optimization for Hedging
1: Inputs: Proxy and true rewards (st,,, per prompt 4; parameter domain θ
2: Output: Optimal hedge parameter 0*
3: STEP 1. For each prompt t, sort responses by their proxy scores and map their ranks to empirical quantiles U 1).
4: STEP 2. Specify the score function (u, 0) and density pa(u) according the inference-time method (e.g., BoN, SBoN, BoP; see Appendix D)
5: STEP 3. For given t and 0 E θ, define the residual (O) = Eu-pu p [re (u, 0)) This can estimated from the empirical pairs {(ur,,, re(u,,)))
6: STEP Find 0 such that the average residual R((*) [ E. Ro) 0 via one- dimensional root-finding.
4 Hedging to mitigate reward hacking
In this section, we develop a unified framework for choosing the inference-time parameter 0 in order to maximize the expected true reward and avoid hacking. The main limitation that we require black-box access to the true reward to perform a one-time calibration of the parameter 8. This is practical in several common scenarios. One example is domains with verifiable ground truth, such as mathematical reasoning, program synthesis, or factual question answering. One may also opt to use an LLMeas-a-jjdge or more powerful but computationally expensive reward model [36] 37
Assume that we are given the proxy and true reward scores for set of query-response pairs. By first constructing an empirical CDF over the generated proxy reward scores, we transform these proxy scores to have a uniform distribution. We denote the transformed proxy reward as U. Each sampling method (BoN, SBoN, and BoP) induces distribution Ty over proxy-percentiles 0 is the corresponding parameter (sample size n, inverse-temperatury X, or Poisson rate p.). Since we know by Theorem [ that the expected true reward has at most one peak, our key insight is to create the precise hedge against hacking by finding the parameter value where the marginal benefit of increasing the proxy reward equals zero. We present the following conditions that the hacking threshold must satisfy for each of the three inference-time methods.
E [0, 1). where
Theorem 3 (Hacking Threshold Characterization). Let r, be true reward oracle and Øi be the hacking threshold from Definition For each inference-time method, 0f is characterized by the following conditions:
For BoN, nz satisfies:
Va ValBu-Betssasssrtl /' re(u) (1) + +Iuu) w01 du 0.
(5)
For SBoN, Al satisfies:
V (Aanss(rr(u)] = Cov (Fe(u), =0,
(6)
For BoP, ul satisfies:
V (ra = Eueesp (aluu( utu
u
) =
The proof can be found in Appendix D Consequently, provide HedgeTune (Algorithm algorithm that numerically solves the corresponding root-finding problem to determine the optimal inference-ting time parameter for BoN, SBoN, or BoP. Note that we do not need access to the LLM distribution itself. We use the explicit expressions of the score function &i and estimate the residual function R(Ø) Efre(uu (u) 0) which captures the alignment between the true reward and the proxy-weighted score. The optimal parameter (i is found efficiently as the root of this function using standard methods such as bisection or Newton's method (38 and can later be used directly at inference.
7

---

CIOAA skywock Larre31..8
KMLL HitemLM2....8
NHLL Liama Offcot
HAT- IntemLM 2-1.85
52.0
Actrray 780
22
365
31.
23.5
500
BoN
SBoN (HadgeTune)
BoN (HedigeTune)
BoP (HedgeTunee)
Figure 3: Hedging mitigates hacking in verifiable reward setups. We plot the expected accuracy
on various benchmarks versus the number of samples HedgeTune successfully recovers the best operating point for BoN and BoP and provides a superior reward-distrotion curve with SBoN.
Experiments: Hedging in Practice
In this section, we validate that hedging is effective tool against reward hacking and provide superior reward-distortion tradeoffs in two experimental setups: one with verifiable rewards and one with human-preference data.
5.1 Hacking in verifiable setups.
We first demonstrate that hedging mitigates reward hacking standard verifiable benchmarks such as MMLU Pro and GPQA. Specifically, we use the open-source Preference Proxy Evaluations (PPE) dataset 39) which contains multiple responses per benchmark question from frontier models such as GPT-44-mini and Claude Haiku Each response is then scored by suite of reward models. We focus the benchmark dataset-reward model pairs identified in PPE exhibiting reward hacking under Best-of-n sampling. We then apply HedgeTune to find the optimal operating point for BoN, BoP, and SBoN.
Reward Models. We consider three reward models of varying sizes: InternLM-2 1.8B 3-Offset-Bias 8B [1,, and Skywork-Llama-3. 8B 42
40 Llama-
Datasets. We consider three datasets: MMLU Pro (complex reasoning)
43
MATH (mathematical
problem solving) 44, and GPQA (questions in natural sciences) 45 A correct response gets a true
reward of 1, while an incorrect response gets true reward of 0.
Findings. The observed reward hacking curve in Figure matches our theoretical prediction in Theorem [ For example, BoN demonstrates hacking on the GPQA dataset, even with the very capable Skywork-Llama 3.1 8B as proxy reward (currently ranked the 12th best non-generative reward model on RewardBench 37). HedgeTune recovers the best operating points in all the cases.
5.2 Hacking the wild with human preferences.
Our experimental design follows the methodology of Coste et al. [46 and Gao et al. wherein proxy reward models are trained using preferences of a fixed gold reward model. In many real- world cases, do not have access to the gold reward and instead have access to preference data. However, as we demonstrate below, favorable operating point can still be found using traditional hyperparameter search.
Models. As reference model, we use a 1.4B Pythia model [47] fine-tuned on AlpacaFarm dataset, but without any subsequent alignment (c.g.. RLHF DPO). This reference model is used to generate responses. We use AlpacaRM [48] as our gold reward model. AlpacaRM an established reward model trained human preference data and has been adopted in prior work reward model evaluation 46 49 50). This model serves as the ground truth for generating preference labels for training proxy reward models. As for proxy rewards, use the setup of Coste et al. [46] where train Pythia 44m models.
Datasets.
We use the t1c4418/gold lididd__a_eleedeee _eens dataset from Coste et al. 46 This dataset
comprises of 12,600 responses generated by the Pythia 1.4B base policy each 1,000 prompts. The prompts are sourced from the validation split of the AlpacaFarm dataset 48). Each generated

---

0.200
0.175
0.150
Bok
0.075
SBcN
0.050
10
20
30
40
50
Number samplesin
Figure 4: Hedging mitigates hacking numan-preference setups. We use three inference-time methods (BoN, SBoN, and BoP) on trained proxy rewards. Hacking is effectively mitigated by hedging via in SBoN or n BoN and BoP.
response in this dataset scored by AlpacaRM. To train proxy reward models, we construct preference datasets using the AlpacaRM reward scores. For each training instance, we sample a pair of responses given prompt and label them based on their given true reward scores.
The proxy RMs are trained using a standard binary cross-entropy loss on preference pairs. We train proxy RMs preference pair datasets of varying sizes: 10k, 20k, 46k, and 80k. In line with 46 51 52 noise in the preferences, and (b) 25% label noise. All proxy RM training runs repeated across random seeds each. We present some of the runs with 25% label noise in Figure 4 and we present the remaining results in Appendix D along with other hyperparameters and training details. Post training, we use each proxy model to score set of 800 prompts, with 12,600 responses each.
Training.
we simulate disagreements in human annotators by considering two cases: (a) no label
Findings. We apply BoN, SBoN, and BoP each and find the expected value of the reward as a function of n. When reward hacking manifests, we find a hacking threshold for BoN and BoP that maximizes their reward. For SBoN, with selected A1, attain the peak value without suffering from reward hacking. Meanwhile, if the proxy is always at odds with the true reward, the optimal solution is the reference distribution itself, corresponding À 0.
6 Related Work
Reward Hacking. Reward hacking has been widely studied RL literature 7 53
also under the
name misspecification 300, goal misgeneralization [54]. specification gaming 55
In the context of LLMs, overoptimization has been referred as reward hacking or Goodhart's Law
28
26 27). Hacking behavior has been found to manifest in unwanted or surprising behavior
[56 across variety tasks [58 14]. Prior works have proposed various formulations of
reward hacking based true performance behavior [24), correlation between proxy and true reward
6), or distribution shift (25). [4] prove that BoN alignment provably suffers from reward hacking
when the number of samples n. is large.
To address inference time hacking, a variety of methods have been explored varying success, such ensembling (6,559 60 611, regularization (62) or rejection sampling [14). [63] simulate n particles resampled using softmax reward improve performance of reasoning models, similar to SBoN over reasoning steps. However, all methods suffer from some combination of additional generation cost beyond generating samples and estimation of additional side- quantitiies such KL- divergence or x² -diveergence. Additionally, a variety of approaches have been proposed to mitigate reward hacking during RLHF finetuning such as regularization 664
52
65 66 ddivvergee (67 6)
uncertainty estimation [68], and reward pessimism
(69 70 although 26 demonstrated that RL.F
can still result in reward hacking under heavy-tailed reward mismatch. Finally, prior works have also
focused improving reward models to prevent mismatch and reduce hacking (71
72
73 74 51 75

---

Best-of-n. Best-of-n sampling is simple inference iiference-time approach for alignment [ 76]. Prior results have characterized the expected reward gap and KL divergence between BoN sampling and the reference model and have demonstrated that BoN is asymptotically equivalent to Kl-constrained reinforcement learning [3 15 T6] There have been various methodological improvements on BoN sampling. One such improvement is to reduce the cost of sampling sequences via tree-based or speculative search [77 (8) Additionally, [88
80 81 82] distill the BoN sampling distribution into model via fine-tuning, Finally, (35 83 propose inference-aware methods to improve BoN. Other works focus on improving the reward model through self-training (44) In this work, we focus on variant of BoN, Soft Best-of-n [17], which allows for finer control between sampling from the base model and the reward- maximizin generation.
79
7 Conclusion
Our work tackles the fundamental challenge that all proxy rewards are imperfect, yet they remain essential for guiding and improving AI systems. We establish a theoretical framework proving the inevitability of reward hacking in inference-time alignment and introduce practical hedging strategies to mitigate its harmful effects. By developing Best-of-Poisse near-optimal -wwardddstootion tradeoffs with a single parameter and the HedgeTune algorithm for precisely calibrating inference methods, we enable practitioners to extract valuable signals from proxy rewards without falling prey to Goodhart's s law.
sampling which achieves
We also emphasize that Al safety concerns are, their heart, socio-technical 30 The impact of model failures due to hacking (and beyond) will depend on the application and stakeholders at hand. One clear example an Al agent that generates controversial and toxic content on social media to optimize its engagement metric at the direct expense of content quality and safety (85) Other dangerous failure patterns include models that learn be sycophantic instead of truthful [86] or
actively exploit loopholes to circumvent oversight [322] By studying the phenomenon of hacking from a rigorous mathematical perspective, our work helps with the design of hacking mitigation (and hence Al safety methods) with provable performance guarantees. Technical approaches such as ours are important for Al safety in practice, although we recognize that they must exist within a larger safety ecosystem where rigorous technical foundations inform responsible deployment practices and policy decisions, and vice-versa. Ultimately, this work demonstrates that principled hedging is a promising direction for building safer, more reliable Al systems.
Acknowledgments and Disclosure of Funding
We thank the anonymous reviewers for their helpful comments and suggestions. This work is supported by the National Science Foundation under grants CIF 2312667, FAI 2040880, and CIF 2231707. This work is supported in part by NSF Awards IIS-2008461, IIS-2040989, and IIS-2238714: an AI2050 Early Career Fellowship from Schmidt Sciences; and research awards from Google, OpenAI, the Harvard Data Science Initiative, and the Digital, Data, and Design (D^3) Institute at Harvard. AO is supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGFF2140743. The views expressed are those of the authors and not necessarily reflect the official policies or positions of the funding organizations.
References
[1] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-300 -021,,011222220000 2020.
[2] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answeringg with human feedback. arXiv preprint arXiv:2 12.09322,, 2021.
[3] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information processing Systems, 30, 2017.
10

---

[4] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint vrXivv2204.05866 2022.
[5] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning. Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
[6] Cassidy Laidlaw, Shivam Singhal, and Anca Dragan. Correlated proxies: A new definition and improved mitigation for reward hacking. In The Thirteenth International Conference Learning Representations, 2025.
[7] Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse Reward Design. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[8] Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Repre- sentations, 2022.
[9] Maarten Buyl, Hadi Khalaf, Claudio Mayrink Verdun, Lucas Monteiro Paes, Caio C Vieira Machado, and Flavio du Pin Calmon, AI Alignment Your Discretion. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency (FAccT). ACM, 2025.
[10] Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ilia Kulikov, and Zaid Harchaoui. From decoding to meta meta- generation: Inference-timg algorithms for large language models. Transactions Machine Learning Research, 2024.
[11] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835-10866. PMLR, 2023.
[12] Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, et al. Controlled decoding from language models. In Forty-firsl International Conference on Machine Learning, 2024.
[13] Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander Nicholas D' D'Amour, Jacob Eisen- stein, Chirag Nagpal, and Ananda Theertha Suresh. Theoretical guarantees on the best-of-n alignment policy. In Forty-second International Conference on Machine Learning, 2025.
[14] Audrey Huang, Adam Block, Qinghua Liu, Nan Jiang, Akshay Krishnamurthy, and Dylan J Foster. best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment. In Forty-second International Conference on Machine Learning, 2025.
[15) Joy Qiping Yang, Salman Salamatian, Ziteng Sun, Ananda Theertha Suresh, and Ahmad Beirami. Asymptotics of language model alignment. In 2024 IEEE International Symposium on Information Theory (ISIT), pages 2027-2032, 2024. doi: 10. 11111//1IIT3784422244.00006
[16] Youssef Mrouch and Apoorva Nitsure. Information theoretic guarantees for policy alignment in large language models. Transactions on Machine Learning Research, 2025. ISSN 2835-8856.
[17] Claudio Mayrink Verdun, Alex Oesterling, Himabindu Lakkaraju, and Flavio P. Calmon. Soft best-of-n sampling for model alignment. In Proceedings of the IEEE International Symposium on Information Theory (ISIT), 2025.
[18] Lin Gui, Cristina Garbacea, and Victor Veitch. Bonbon alignment for large language models and the sweetness of best-of-n sampling. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[19] Edward C. Capen, Robert V Clapp, and William M Campbell. Competitive bidding in high-risk situations. Journal of petroleum technology, (36)):641 1 1663,, 1971.
[20] Max H Bazerman and William Samuelson. won the auction but don't 't want the prize. Journal of conftict resolution, 27(4))618-66 -644, 1983.

---

[21] James C Cox and R Mark Isaac. In search of the winner's curse. Economic Inquiry. 22(4): 579-592, 1984.
[22] Richard H Thaler. Anomalies: The winner's curse. Journal of economic perspectives, 2(1): 191-202, 1988.
[23] Richard Thaler. The winner's curse: Paradoxes and anomalies of economic life. Simon and Schuster, 2012.
[24] Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35:9460- 9471, 2022.
[25] Lukas Fluri, Leon Lang, Alessandro Abate, Patrick Forré, David Krueger, and Joar Max Viktor Skalse. The perils of optimizing learned reward functions: Low training error does guarantee low regret. In Forty-second International Conference on Machine Learning, 2025.
[26] Thomas Kwa, Drake Thomas, and Adrià Garriga-Alonso. Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailes reward misspecification. In The Thihty-eighth Annual Conference Neural Information Processing Systems, 2024.
[27] El-Mahdi El-Mhamdi and Lê-Nguyên Hoang. On goodhart's law, with an application to value alignment. arXiv preprint arXiv:2410.09638, 2024.
[28] Charles AE Goodhart and CAE Goodhart. Problems of monetary management: the UK experience. Springer, 1984.
[29] Lilian Weng. Reward hacking in reinforcement learning. https://1illanwwegg github.......... 10/ 11--000----1------------------
November 2024. Accessed: 2025-05-03.
[30] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv.1606.06565, 2016.
[31] Alexander Bondarenko, Denis Volk, Dmitrii Volkov, and Jeffrey I Ladish. Demonstrating specification gaming in reasoning models. arXiv preprint arXiv:2502 /3295, 2025.
[32] Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y Guan, Aleksander Madry, Wojeiech Zaremba, Jakub Pachocki, and David Farhi. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. arXiv preprint 2771: 2025.
[33] Anthropic. Claude 3.7 sonnet system card, July 2025. htap:///88222.0000003333/ogggaa2 nical Report.
anthropic 3- sonnet Soaanet-yssteee wyw.temccrd..ddaddo.oooo.. Tech-
[34] Imre Csiszár, Paul C Shields, al. Information theory and statistics: A tutorial. Foundations and Trends@ in Communications and Information Theory, 1(4):417-528, 2004.
[35] Ananth Balashankar, Ziteng Sun, Jonathan Berant, Jacob Eisenstein, Michael Collins, Adrian Hutter, Jong Lee, Chirag Nagpal, Flavien Prost, Aradhana Sinha, Ananda Theertha Suresh, and Ahmad Beirami. Infalign: Inference-awee aware language model aligmment. In Foroy International Conference on Machine Learning, 2025.
[36] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang. Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
[37] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. In Findings of the Association for Computa- tional Linguisties: NAACL 2025, pages 1755-1797. 2025.
[38] Alfio Quarteroni, Riccardo Sacco, and Fausto Saleri. Numerical mathematics, volume 37. Springer Science & Business Media, 2006.
12

---

[39] Evan Frick, Tianle Li, Connor Chen, Wei Chiang. Anastasios Nikolas Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. How to Evaluate Reward Models for RLHF. In The Thirteenth International Conference on Learning Representations, 2025.
[40] Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internim2 technical report. arXiv preprint arXiv:2403.17297, 2024.
[41] Junsoo Park, Seungyeon Jwa, Ren Meiying, Daeyoung Kim, and Sanghyuk Choi. Offsetbias: Leveraging debiased data for tuning evaluators. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1043-1067, 2024.
[42] Chris Yuhao Liu, Liang Zeng. Jiacai Liu, Rui Yan, Jujie He, Chaojic Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in Ilms. arXiv preprint 2xXiv22400.845111 2024.
[43] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, al. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. Advances in Neural Information Processing Systems, 37:95266 955900, 2024.
[44] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring Mathematical Problem Solving With the MATH Dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Truck (Round 2), 2021.
[45] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
[46] Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. In The Twelfth International Conference on Learning Representa- tions, 2024.
[47] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' D'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430. PMLR, 2023.
[48] Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36:30039-3000 30069, 2023.
[49] Zaiyan Xu, Sushil Vemuri, Kishan Panaganti, Dileep Kalathil, Rahul Jain, and Deepak Ra- machandran. Robust Ilm alignment via distributionally robust direct preference optimization. arXiv preprint (=XX::25020019300 2025.
[50] Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluat- ing large language models at evaluating instruction following. In The Twelfth International Conference on Learning Representations, 2024.
[51] Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, and Dacheng Tao. Inform: Mitigating reward hacking in rlhf via iiformationnthhorcee reward modeling. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024.
[52] Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs. Advances in Neural Information Processing Systems, 37:62279-62309, 2024.
[53] Jacek Karwowski, Oliver Hayman, Xingjian Bai, Klaus Kiendlhofer, Charlie Griffin, and Joar Max Viktor Skalse. Goodhart's law in reinforcement learning. The Twelfth International Conference Learning Representations, 2024.
13

---

[54] Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. Goal misgeneralization: Why correct specifications aren't 't enough for correct goals. arXiv preprint arXiv:2210.01799, 2022.
[55] Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, and Shane Legg. Specification gam- ing: the flip side of AI ingenuity. specification gaaai--gminng--g the- fhe-- wwl.ippppiedaa. side- of ai- ingenuity/ April 2020.
google/ discover/blog/
[56] Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec, Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, et al. Sycophancy to Subterfuge: Investigating Reward- Taapperinn in Large Language Models, arXiv preprint arXiv:2406.10162. 2024.
[57] Lichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. ODIN: Disentangled reward mitigates hacking in RLHF. In Forty-first International Conference on Machine Learning, 2024.
[58] Jane Pan, He He, Samuel R Bowman, and Shi Feng. Spontancous reward hacking in iterative self-refinement. arXiv preprint arXiv:2407.0444 045404447....... 2024.
[59] Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alexander Nicholas D' Amour, Krishnamurthy Dj Dvijotham, Adam Fisch, Katherine A Heller, Stephen Robert Pfohl, Deepak Ramachandran, et al. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. In First Conference on Language Modeling, 2024.
[60] Ahmed M Ahmed, Rafael Rafailov, Stepan Sharkov, Xuechen Li, and Sanmi Koyejo. Scalable ensembling for mitigating reward overoptimisation. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024.
[61] Alexandre Rame, Nino Vieillard, Leonard Hussenot, Robert Dadashi-Tazzehozi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. WARM: On the Benefits of Weight Averaged Re- ward Models. In International Conference on Machine Learning, pages 42048 42073. PMLR, 2024.
[62] Yuki Ichihara, You Jinnai, Tetsuro Morimura, Kenshi Abe, Kaito Ariu, Mitsuki Sakamoto, and Eiji Uchibe. Evaluation of best-of-n sampling strategies for language model alignment. Transactions on Machine Learning Research, 2025. ISSN 2835-8856.
[63] Isha Puri, Shivchander Sudalairaj. Guangxuan Xu, Kai Xu, and Akash Srivastava. A probabilistic inference approach to inference-time scaling of Ilms using particle-based monte carlo methods. arXiv preprint arXiv:2502.nl 01618, 2025.
[64] Paria Rashidinejad and Yuandong Tian. Sail into the headwind: Alignment via robust rewards and dynamic labels against reward hacking. In The Thirteenth International Conference on Learning Representations, 2025.
[65] Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang. Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[66] Yuchun Miao, Sen Zhang, Liang Ding, Yuqi Zhang, Lefei Zhang, and Dacheng Tao. The energy loss phenomenon in RLHF: A new perspective on mitigating reward hacking. In Forty-second International Conference on Machine Learning, 2025.
[67] Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason Lee, Wen Sun, Akshay Krishnamurthy, and Dylan J Foster. Correcting the mythos of kl-regularization: Direct alignment without overoptimization via chi-squared preference optimization. In The Thirteenth International Conference Learning Representations, 2025.
[68] Xiaoying Zhang, Jean-Francois Ton, Wei Shen, Hongning Wang, and Yang Liu. Overcom- ing reward overoptimization via adversarial policy optimization with lightweight uncertainty estimation. arXiv preprint ar 27X11:2403.0577,,222
14

---

[69] Banghua Zhu, Michael Jordan, and Jiantao Jiao. Iterative data smoothing: Mitigating reward overfitting and overoptimization in RLHF. In Forty-first International Conference on Machine Learning, 2024.
[70] Dhawal Gupta, Adam Fisch, Christoph Dann, and Alekh Agarwal. Mitigating preference hacking in policy optimization with pessimism. arXiv preprint arXiv:2503. 000000,6000000 2025.
[71] Yizhou Chen, Yawen Liu, Xuesi Wang, Qingtao Yu, Guangda Huzhang, Anxiang Zeng, Han Yu, and Zhiming Zhou. Establishing reliability metrics for reward models in large language models. arXiv preprint arXiv:2504. 148388 2025.
[72] Lingfeng Shen, Sihao Chen, Linfeng Song, Lifeng Jin, Baolin Peng, Haitao Mi, Daniel Khashabi, and Dong Yu. The trickle-down impact of reward inconsistency on RLHF. In The Twelfth International Conference on Learning Representations. 2024.
[73] Jayi Fu, Xuandong Zhao, Chengyuan Yao, Heng Wang. Qi Han, and Yanghua Xiao. Reward shaping to mitigate reward hacking in rlhf. In ICML 2025 Workshop on Reliable and Responsible Foundation Models, 2025.
[74] Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin, Tianhe Yu, Daniel Sohn, Anastasia Makarova, Jeremiah Zhe Liu, Yuan Liu, Bilal Piot, Abe Ittycheriah, Aviral Kumar, and Mohammad Saleh. RRM: Robust reward model training mitigates reward hacking. In The Thirteenth International Conference on Learning Representations, 2025.
[75] Chaoqi Wang. Zhuokai Zhao, Yibo Jiang, Zhaorun Chen, Chen Zhu, Yuxin Chen, Jiayi Liu, Lizhu Zhang, Xiangjun Fan, Hao Ma, et al. Beyond reward hacking: Causal rewards for large language model alignment. arXiv preprint arXiv:2501, 096620, 2025.
[76] Jacob Hilton, Peter Clark, et al. Measuring goodhart's law: Towards an evaluation framework for open-ended generative models. OpenAl Blog, 2022.
commccomiooiic index/
measuring-goodharts- goodhartss Accessed: 2025-01-30.
[77] Jiahao Qiu, Yifu Lu, Yifan Zeng, Jiacheng Guo, Jiayi Geng, Huazheng Wang, Kaixuan Huang, Yue Wu, and Mengdi Wang. Treebon: Enhancing inference-time alignment with speculative tree-search and best-of-n sampling. arXiv preprint arXiv:2410. 16033, 2024.
[78] Ruiqi Zhang, Momin Haider, Ming Yin, Jiahao Qiu, Mengdi Wang, Peter Bartlett, and Andrea Zanette. Accelerating best-of-n via speculative rejection. In 2nd Workshop on Advancing Neural Network Training: Computational Efficiency. Scalability, and Resource Optimization (WANT@ ICML 2024), 2024.
[79] Pier Giuseppe Sessa, Robert Madashi-Tazehhor Leonard Hussenot, Johan Ferret, Nino Vieillard, Alexandre Rame, Bobak Shahriari, Sarah Perrin, Abram L. Friesen, Geoffrey Cideron, Sertan Girgin, Piotr Stanczyk, Andrea Michi, Danila Sinopalnikov. Sabela Ramos Garea, Amélie Héliou, Aliaksei Severyn, Matthew Hoffman, Nikola Momchev, and Olivier Bachem. BOND: Aligning LLMs with oest-of-n distillation. In The Thirteenth International Conference on Learning Representations, 2025.
[80] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint wwwXi::3007.0928,, 2023.
[81] Afra Amini, Tim Vieira, Elliott Ash, and Ryan Cotterell. Variational best-of-n alignment. In The Thirteenth International Conference on Learning Representations, 2025.
[82] Tong Yang, Jincheng Mei, Hanjun Dai, Zixin Wen, Shicong Cen, Dale Schuurmans, Yuejie Chi, and Bo Dai. Faster WIND: Accelerating Iterative Best-of-N Distillation for LLM Alignment. In The 28th International Conference on Artificial Intelligence and Statistics, 2025.
[83] Yinlam Chow, Guy Tennenholtz, Izzeddin Gur, Vincent Zhuang, Bo Dai, Aviral Kumar, Rishabh Agarwal, Sridhar Thiagarajan, Craig Boutilier, and Aleksandra Faust. Inference-aware fine- tuning for best-of-n sampling in large language models. In The Thirteenth International Conference Learning Representations, 2025.
15

---

[84] Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. West-of-n: Synthetic preference generation for improved reward modeling. arXiv preprint 2200 2024.
[85] Alexander Pan, Erik Jones, Meena Jagadeesan, and Jacob Steinhardt. Feedback Loops With Language Models Drive In- in-Context Reward Hacking. In International Conference on Machine Learning, pages
PMLR, 2024.
[86] Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Esin DURMUS, Zac Catfield-Dodds Scott R Johnston, Shauna M Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards understanding sycophancy in language models. In The Twelfth International Conference on Learning Representations, 2024.
[87] Yuu Jinnai, Tetsuro Morimura, Kaito Ariu, and Kenshi Abe. Regularized best-of-n sampling to mitigate reward hacking for language model alignment. arXiv preprint arXiv:2404.01005. 2024.
16

---

Supplementary material for
Inference-Time Reward Hacking in Large Language Models
A
Overview
In this supplementary material, we provide the following:
Appendix provides additional proofs and results on the inevitability of reward hacking.
Appendix C
provides additional proofs for Best-of-Poisson as well as Soft Best-of-Poisson
Appendix D discusses HedgeTune in more detail.
Appendix provides additional details our experimental setup.
B Inference-Time Reward Hacking
Increasing optimization pressure on a proxy objective can initially improve true performance, but beyond a certain point- which we call the hacking threshold ffurherr optimization can lead to misalignment and degrade performance. Theorem [ Section formalizes this phenomenon: under very general conditions on one-parameter family of proxy distributions To, the map
Ex= (X)] S(0)
(8)
have most one interior extremum. Thus, there is either monotonic benefit (or disadvantage) to strengthening the proxy or exactly one "sweet spot" before reward hacking sets in. The rest of this section proceeds follows. We prove the single-crossing property for the derivative of the true reward by invoking variation-diminishing kernels, and then we instantiate our results with densities in Corollaries 2 and 3 We specialize the discussion two concrete examples with Best-of-n and Best-of--Possenn Next, we extend our result to the more general discrete case in B.2
B.1 Uniform Case
The following results hold under Assumption 1 Let (xx,, 0) denote the score function of distribution no (a) with density Po (æ). Standard calculations under mild regularity conditions gives us the derivative of the true reward under no:
(') / Vora()))
/ (r po Ex Bxww,rrr(Xxe)r (x)(X)x (9)
The main idea to establish Theorem is that when we use parameter 0 to control inference- time methods, we create a family of densities (pa(#) that act as positive kernels. These kernels satisfy the strict total positivity conditions required for variation-diminishing theorems to apply. Our only assumption on the true reward that it bounded. We then translate the reward function so that non-negative. The boundedness assumption is natural one in the alignment setting because real- world rewards originate from human judgments given on finite seales (c.g. star ratings, Likert scores, normalized preference probabilities). Moreover, clipping or normalizing the reward prevents unbounded returns, improving the stability policy updates and inference-time mechanisms..
Proof. Fix 0 and ha(x) := (x(x))((x 0). Because v'(, 0) is strictly increasing, has at most one zero. Since Tz 2 0, he has the same single sign change in x. Striet TP2 of pa and Karlin's dimiaiion-dmiiishiin theorem imply that 0 + F(0)= == hg(x) dx (') inherits at most the same number of sign changes, namely one.
Having established the inevitability result, we next explore how specializes in classical families via simple corollary.
17

---

Corollary 2 (Strict MLR densities), Let po(x) be strictly monotone-likelihood ratio in (i.e. Paz (z)/Po, (z) strictly increases in x whenever 82 81). (2, 0) = 0g log Po(x) strictly increas- ing in then all conclusions of Theorem Saaremmappiin apply.. In particular, this applies for any regular canonical exponential family with strictly monotone statistic 7 and strictly monotone natural parameter.
The conditions are satisfied by two inference-time methods we study:
Best-of-n: The distribution corresponds to the maximum of i.i.d. samples from the reference distribution. When proxy rewards are uniformly distributed, this yields p., nu- for u E [o, 1].
The likelihood ratio Pang (1))
222""= "n is strictly increasing in when n2 > n, establishing strict
Png (M)
71
n MLR (which implies striet TP2). The score function (u, n) 1 + llog is strictly increasing in
Best-of-Poisson The distribution Pa (u) (nu 1)c for E [0, 1] can be verified to satisfy strict MLR by direct computation of likelihood ratios. The score function (u, μ1) = u pu+1 is strictly increasing in u.
Under the conditions presented in Theorem we know that at most one interior extrema exists. The following corollary gives precise conditions on when such an interior extrema exists.
Lemma 1. Under the assumptions of Theorem [ and with C' neighborhood of its boundaries (0 and 1) and let = [,),
lim ('(0) = (0+)) [x vxx 0e)], 010;
lim f'(0) = r/(1 (1(1)) Ea, 1(1 - X))(X, 070,
Then, stationary point exists iff
lim (0) and lim ('(
0.0
010,
are of opposite sign (or one limit is 0 while the other is non-zero).
Proof. We prove the left boundary results (the right one identical with x + - x.) Write the first-order expansion r. (xr) = ra r=(0+) +( + R(x)) with R(x) o(x) x O. Because
Eg [o(b]] = 0,
= r(
Strict increase of vi implies [X w C(1 X) on [0, x 10,, Oz + p). [x[X]] + [Xx] by dominated convergence. Next, fix E (0, 1) and split the expectation:
Eg [x = E.o (r 1 [x + Eo(rx (x))
Near 0, we have |R(x)|< cgæ, hence the term is bounded by es C, [X [i];; choose ó so small that eglE,,[X] [X]+]] < and continuity keeps
for 0 close enough to 0z. Away from 0, we use boundedness of rt and local boundedness of 2b obtain a factor P > 8} + 0.
Combining the two parts gives EEg [R(X) }+ + O, yielding the claimed limit. By continuity, the Intermediate-Vall Theorem then forces one root of f' and single-crossing rules out a second. O
Corollary 3 (Reward behavior for Strict MIR densities). Assume Lemma I Tholds and that the family {po)oee is strictly monotone-likelihond-
E (0, 1). Then for every 0 E θ
Lg == E (XX 0,
Rs E. X)) 0)] O.
so
fignf' = sign (0+))
s = sign (1-)
Single-crossing of f' implies that f@) can assume exactly one of the four shapes:
regime
+((0+)
(11-))
monotonic improvement
20
reward hacking
>
<
reward grokking
<0
immediate decline
S0
18

---

Proof. Let be differentiable in f with score (b 8) = dglog po(x). For any integrable we have shown that:
de [x]] E (x))
We first that the MLR property implies first-order stochastic dominance. For every increasing
function g. we have that 0 + Eg [X]] is non-decreasing and its derivative non-negative. Choosing
g(x)) gives
d Lg Es X (X.0) = d0
Eo
0
On the other hand, g(z) 1 x (strictly decreasing) yiclds:
Rg = Eo[(1 XX) 0)] = d0 Eo
x < 0
Thus Lg > 0 and Rg < 0 for every O. For example. for BoN, ((x n) 1/n + log x with E, X v/] 1 1/(n + 1)², [n x) = -1/ + 1)2. Hence stationary point exists iff (0(0-)
and r1(1 have opposite signs, and location n, solves E..rr((X)v (X)(XX))XXXXX (X, =
We have shown the conditions under which the expected value of true reward has critical point with respect to We now show the conditions under which our results extend identically if we are studying the expected value of true reward as function of the KL divergence with respect the reference distribution Tép.
Lemma 2. Let (Te) Jeee be a regular parametric family with density pe(x)) such that
po and Oapo jointly measurable and Ospo(z) (x) is locally integrable in 0:
the score (xx 0) : Op log po()) is square-integrable: [x(v) < co.
For fixed reference point 0 E define the Kullback-Leible divergence
D(01)0)) / po((z) log pu(x)) du(x))
poo (x))
Then 00) is differentiable and
d
d 0)) Es [log pe (x) log pa, (x)) (x.)) (X
In particular, for canonical exponential families with strictly increasing natural parameter, this simplifies to
de D(01)0a) = (n((0) - v((0)) (0),
which is strictly positive when 0 > 0o (and negative for < 0o).
Proof. Write (x) = logre(a) log Pe, (x). Then ((0)))) = Ee(g((X))
For
garameter-dependerr integrand the classical Fisher-Leibnie rule gives
d
F. ggo (X) [0 (X) + [oogo(X (X) (X,,))]
dB
whenever Oogs exists and an L coninated bound holds (true here by the square-integrable score assumption). Since Opge(z) = (1a, 0) and Eg [v] = 0, the first term vanishes, leaving exactly
d D(0 (0) = Eolgo (X) (X) ((X,0)
d0
19

---

B.2 Discrete Case
In this subsection, we show that Theorem hoolds for the general discrete case and then verify that holds for specific policy families like Best-of-n and Best-of-Poisson
Let Y {y1, 1/...---.- -mm be finite, ordered set of possible responses, where the ordering is determined by a proxy reward function fp. We represent this space by the ordered index set
X (1, 2, .... m). Let (To Jose be one-parameter family of policies on X, where 0 e C IR is a continuous tuning parameter. The probability of selecting response is given by the Probability Mass Function (PMF) (t)) Let r. X + I220 be a bounded, non-negative true reward function.
Our objective is to analyze the shape of the expected true reward function:
m
(0) Ex~ro [r((X)] Σ (o( r1).)
We will prove that under general conditions on the policy family Iaa the function f(0) is either monotonic has unique interior extremum. Theorem 4 (Inevitability of Reward Hacking for Discrete Policies). Let (malaee be family of PMFs on X {I....,,)) .... Assume that:
1. The PMF (i( is strictly TP2 in the pair (0, i). That is, for any 01 < 02 and é < i2. no, (11 To, (12) no, (i2) (11) >
2. The score function (i 0) = 86 log (i) exists and is strictly increasing in i for each fixed 0.
Then, for any bounded, non- non-negative true reward function r (i), the derivative f'(0) changes sign at most once. Hence, f0) is either monotonic or has a unique interior extremum.
Proof. We differentiate:
d
('(0) do
To (in
Σ To (1((((1)))))) E., (r)
Define (i) re (i) 0). Since ri( 0 and sbi 0) is strictly increasing in i, the product ho(1) has at most one sign change. Since f0 = Σm= To (i) and To is strictly TP2, Karlin's criiation-diminsshing theorem implies that (0) changes sign at most once, and f(0) has at most one extremum.
We now verify that Best-of-n and Best of Poisson satisfy the assumptions. Let the base PMF be Pi. with CDF Fg = Σx= Pi.
Best-of-n:
It is shown in
B3
that the BoN policy has the following PMF:
TT (2)
F.
F
We first verify the TP2 condition. We consider:
F.t F
L(1)
Tno
TTi
Fm
By Cauchy's Mean Value Theorem (CMVT), for g(t) 12 s(t) = l and Ft, Fi-1, get:
g(F.) - ((F=1)) g'(c)
n2cc 12-1
Tv2 n
f(F.) f(F-1)) (c()
Ri
for some c; E (F.11, F.). Since F; is strictly increasing, G; < C1+1, and - strictly increasing.
Hence L(i + 1) > L(i). Next, note that the score function
(i V, log(F FT
20

---

Let gtt) = f" logt, f(t) = t². Then
p(t) = g(())
nt -1 log 1 mm_
logt +
f((
ntn
Applying CMVT gives:
(i, o(F:) 9(Fi-1) (((() = loog cs +
f(F() f(F(-1)
Then:
1 n li 1,n) log c041 log c + (i, n
1
Best-of-Poisson:
Define g(t, = ten(t-1), Then, we show Theorem that BoP has the
following PMF:
- ( = -(F..)))g(Fi-111) g(Fi-1.
We first verify the TP2 condition. Define:
I(i) 9 H2) 9(F(-1.
((1 -9( ((;:11:11)
By CMVT:
L(1)= = Viglar (12c + 1)eas(o-11) veg(c, =
(mig + 1)emr(e--1 = u2e; + 1 Nic +
This increases in & E (Fi-1, hence L(i ((i+1) > L(t). Now, we check the score function. Let the continuous score be (1x μ1) -1 u=+T Since this is strictly increasing in æ, so is the discrete score (i(
C Best-of-Poisson
In this section, we prove Theorem 2 andd establish, as a consequence, that Best-of-Poisson is numer- ically near-optimal as compared to tilted distribution πद in terms of KI. divergence. We start by establishing the BoP distribution in the uniform case, and then consider the general discrete case. We also mention natural extension called Soft Best-of-Poisson.
C.1 Uniform Case
The following results hold under our Assumption ! Let > be the parameter of the Best-of- Poisson sampling method and let X,, be the random variable representing the response selected by BoP. The probability density function qa (z) of X,, is given by:
qu (x) (1 (
(10)
for x E [0, 1). where n = n' + with n' ~ Poisson(11..
Proof. Write X,, = max(U/o UI...., Un where Poisson(p) and U3 iid Unif(o, 1]. Consider
Uo Unif[o, 1] to be the mandatory draw to achieve sample size of at least one.
For [0, 1].
Fx(x) : Pr(X, < x) PRUU S x) PRU S æ for sisn') = Ma"' re =(1-2)
because -xp(-x(1-2)) is the moment-generating function of Poisson variable evaluated at log x. Now, differentiating F,. on (0, 1) gives
(x) e- -#(1=2) + pa -H(1=2) (1 + +aa) cl(x=1))
which extends continuously the endpoints. A direct computation verifies So' qn dx so qu is valid density.
21

---

Now, we prove Theorem 2 with the BoP density denoted as 9u and a uniform reference distribution.
Proof.
B[X / x(1 (z)el(a-11) do S' (cc pz dx
With the substitution u = 11x - 1), we get that
! rel(e-1) d)
I (u +
p))" (u+))e" 1 -
1² v
4²
Jo cculo-1) 1 0
(u + p) 13 {
112 24 + 2 2e- "
Hence
B[Xp]= - e
n1² 24 + 2 - -
+ 1 - e
11
Now, we consider the KL divergence. Because log (x)) log(1 + pa) + píx 1),
DKL (qu [U] = S/' [x(=)) + pz) dx + [
To compute the integral, 1 + paz::
Lo qu (x) log(1 ux) dr e-a- frae te' logt
Integration by parts (f logt, dg = te' tt) yields
/ te' logt 1 fee (logt 2) Ei(t) + 2
2
hence
L' (z( log(1 + pæ) da = log(n + 1)
a
--1
Ei(p + 1) - Ei(I)]
1
The resulting term becomes:
e -4-1
DKL (qu [nnif)
[iip + 1) -Ei(1)] + log(u + 1)
We now establish that BoP provides a practical approximation to the optimal tilted distribution with negligible performance loss.
Theorem (Near-Ophr Optimality of BoP). Let qu be the distribution induced by Best-of-Poissee with parameter μ > and let gx be the optimal KL-constrained tilted distribution with parameter À > defined as:
(x(z)
(11)
Z(x)
where Z(A) is the normalization constant. For any given expected reward level, there exists j for BoP and a X for the tilted distribution such that Ex [Tp (X)] Ex Ex~9x [rp (X)]. Numerical
evaluation shows that for all μ. if X > 0 is chosen so that Ex [r X) = Ex (X)), then the
KL-gap satisfies
DKL (g, [rrttt DKL(Ox)] < 8 x 1
That is,
achieves nearly the optimal trade-off between expected reward and KI. divergence from the reference distribution.
22

---

Proof. Let the following two functions
gx(x) = Ae
(x(x) = (1 + pz) cM
0 x 2 1
denote, respectively, the BoP density with the Poisson rate j and the exponential-til density with parameter X. For any μ > O, the value X = X² (a) is chosen so that the two laws have the same first moment (i.e., same expected reward). We first note the expected reward of the BoP policy lies in the interval (0.5, 1). Additionally, the tilted expected reward is a strictly increasing function of À over the same range since its derivative the (positive) variance of the reward under the tilted distribution. By the Intermediate Value Theorem, there exists unique X = μ + 8(4), where d(μ) = X (1) - μ.. such that the two rewards are equal.
Because 9X(a] is the information projection of Or onto the exponential family {gaAaso un- der the linear reward- constraint, the Ssiszár-Pythagrre identity (34) gives
(xa Frt) DEL (9x()))) (| ( (gulgul99x0m))
(12)
To bound this KL divergence, we use a general inequality which we derive below from the properties of the likelihood ratio function, Let T.(x) = p(#) denote the likelihood ratio between two distributions
a(z)
p and q. We first define the KL divergence in terms of the convex function (o(t) llg t + 1 and
then note that the expectation of any function is upper bounded by its essential supremum. We can retrieve the following upper bound:
(x || q) Eglo(L(x)) sup (L(()) rETm(L)
sup (t())
(13)
In our problem, the likelihood ratio is given by
(x) qu (x) (px + 1)(e -1) -(())
(14)
9x- (z)
Xre
To obtain a uniform bound on L (z), define a := ===:s suo xe [log (z)). This a well-posed,
nested optimization problem requiring numerical solution. The procedure is as follows:
1. For a given numerically solve for the unique X satisfying the reward-matchine matching equation.
2. For the resulting pair (μ, X*), compute maxzejo.1|logg (xxx) by evaluating the boundary values and any interior critical points.
3. Maximize the result of step (2) over all μ > 0.
This procedure is deterministic, and its solution yields uniform bound: e- S [x(x) S e", for all μ > and E [0. 1] with a 0.03968
Since the function cptt) is convex, we verify that its supremum over [e-a, cda is attained at an endpoint:
DKL (gullo9))))) S ((E")= ac
R 0.03968 0.03968 0.03968
1 R 8 x 10-4
This bound is independent of μ., and therefore holds uniformly as seen in Figure 2 This validates that Best-of-Poissee indeed provides a practically equivalent approximation to the optimal tilted distribution with negligible computational overhead.
C.2 Discrete Case
We now relax our Assumption [ to derive an exact, assumption-fre formulation for the Best-of- Poisson (BoP) policy, as well as convenient upper bound for its KL divergence. Theorem 6 (General BoP Distribution). Let > 0 be the parameter of the Best-of-Poisson sampling method. The probability mass function of
is:
(4) To (y: x)) = gn((F) ga(FF1) where gu(2) = 22
(15)
KL Teet) S Ellog N] 1
where N Poisson(p)) ( +1
(16)
23

---

Proof. BoP is a mixture over BoN with a Coiison-distribben number of samples N ~ Pois(μ) + 1, with PMF:
E
Pr(N
k z
(k 1)!
The BoP policy is:
00
00 e k-
(Bp x))
Σ
Pr(N- k)
(k)
x)) =
(F+ F
(k 1)!
h=
Letting j = - this becomes:
(UFAY
20 (ufi-1y'
(yp (x) EM
(e P P
j!
Thus, the exact BoP PMF is:
ToP (yl x)) - 9n (F:) - (Fi-i where gn(2) == zeH(z-1)
We show that this is consistent with our continuous case. Assuming the reference distribution is Unif(0, 1), then (r) and:
d (r) = gr reu(r-1) gu(r) - GBoP(r) (1 + pr)e
(17)
dr
This matches the continuous BoP PDF.
Using the exact PMF, we can get the exact KL divergence between the Best of Poisson distribution and the reference distribution, as well as convenient upper bound.
m
TBo xr)
m
log (u((F)) 9p (Fi-1)
[L (/
Σ
BoP (1/s
x)) log
Σ
Pi
Fj - F,-1
(18)
We show that this is consistent with our continuous case. As m - oo, the discrete sum becomes:
KL(TBoP||Tret) L qu(r) log gu()) dr
where (u(r) = (1 + pe) ( This integral is:
S( (og log gr) dr e-p- (Fi(H +1) 1) - Fi(1)) + log(u + 1)
where Fi(z) is the exponential integral. The KI divergence is convex in its first argument. That is, for any set of distributions (P&} and weights {ur).
KL Σ wk Pk 2//10/
>> wk KL l1
Applying this to BoP:
KL(TRAP Trot)
Σ We
(R) KL
BaN
Tret) Po
(N) KL(TT |l [Trrr)
Using the proven upper bound for BoN in [3 we get that
(L Tret) $ EN log N - + N
=
Elog N] -
where
N ~ Poisson (
N

---

Algorithm 5 Soft Best-of- ofosossoo Sampling (SBoP)
1: Input: Poisson parameter > 0, inverse temperature > 0, base policy Tree
2: Sample n' Poisson(j), set n' +1
3: Draw X1....;A Xn ~ #ee i.i.d.
4: Compute rewards R, = (X()
5: Sample index Z E {l...., n with probability
eAR
Pr(Z i)
Σラ= exity
6: Return: Y Xz
C.3 Soft Dest-of-Poisson
We discuss here natural extension of Soft Best-of-n and Best-of-Poissoo which name Soft Best-of-Poisson (SBoP) (see Algorithm 5). The key insight is that we can leverage the two distinet control mechanisms that inference-time alignment offers control over the number of generation (be it deterministic or randomized) and control over the selection through the temperature. Extensions such SBoP show that these methods can be combined, offering richer control over the alignment process.
Theorem 7. Let the parameters of SBoN be (n, A). Let R1... ..... R,, be the proxy rewards of i..d. samples from Tef- By Assumption | R₃ Unif(0, (0, 1) for all i. The selected sample has proxy reward Ry with probability density function:
Ar
PA.A (r) = n E R. Un)
(19)
ex
Eg /=-=
Let the log-partition function be defined as:
n An
L(n, A) = R Unif(O.1)
log
e
Then, the expected reward and the KI. diveregence with respect to the reference can be written as:
E[AsRoN] aL(n,)) A)
(20)
aX
KL(TSRON (ret) log n + A B[RRRON]] L(n, A)
(21)
Proof. Expected Reward:
AR
1= RR. AR.
aL(n, A)
E RsSaN]
= E
R:
= E
edR
j=1
1=1
since we observe:
i
Σn R.E.R. Σラ edRR
log
Nns
O
KL Divergence: Since Prer(r) for E [0, 11, we get:
KL (L MaaaaNIITTa [(t)) = //" Pm,a(r) log (og dr
Alternatvelly, the conditional KL (per draw) is:
n Σ
KLigodd
ps log(np:) log n +
Σ pi logp: = logn + X Σ p.R, - log
Σ cBB
AR where Pi ΣT- e Ang.
Taking expectation over rewards gives result.
25

---

D Analysis of HedgeTune
In Section 4 we presented an efficient way find the optimal hacking threshold. Here, we prove this result and discuss how hedging compares to other hacking mitigation methods.
Proof. We consider each mechanism separately:
Hedging in Best-of-n. In BoN, approximate the integer via continuous parameter by placing a Beta(o, 1) prior on u. Its density is [u(u) = ou so (u,, a) = 0a In (a-1) In
In Thus, the optimality condition becomes
( -0 -
Betnaaa(
re(u)
ri(u) (1 n u) u²=1 du 0,
(22)
a
which one solves for a to pick an effective sample size.
In practice, this equation must be solved numerically. One way to do this by discretizing [0, 1] into M points and forming the Riemann- sum residual
M
R(a)= = =-re re((s) ( 1=1
+ In a- Au,
The root R(ax) = 0 is equivalent to the hedging condition. Then, one applies any root-finding method, see, c.g., (38]. to locate the unique solution a Alternatively, for each question/pront we first sort the candidates by their model scores form the empirical CDF of the scores. We perform a discrete ternary search to identify the optimal n using the question-averaged true reward. This procedure bypasses explicit not-finding and directly identifies the most effective discrete hedge size.
Hedging in Soft Best-of-n. Unlike BoN, SBoN does not admit a simple closed-form density due to its sampling mechanism. In SBoN, first sample uniform rewards U Ul...., Un, then select The resulting distribution is:
exe;
U, with probability p. = Ei A..
n, ex
Tn, (u) Bu [~Unii[(01]
u E [0,
(23)
Di=1 c
We can now derive the hedging condition
2 Buws [r((u)] = O. To start, we compute
ap:
BA
8A
8p: θ
Uie S AUs
as
DA
Pi
U,
(24)
0x
OX
52
since
0s 8X
=
Σラ=ı U, Now, we compute the derivative of the expected reward:
② 01
i
Bansalri (r(())) E
(U.)p:
(25)
01
Opi 2x
Ex
ru(
(26)
E
r(U)))
U¹ Σ U,p}
(27)
j=1
E
(UU))p
Σ ru(U))p
Eupp}
m
(28)
E [Cov(r, V[))
(29)
26

---

where we note that the expression inside the expectation is exactly the conditional covariance:
(ov(ra(V) = Elrx(V) VIU" (V) E[]
(30)
with
[[rx()) Σ (U.)ss B[VIU"| = Σ Up.. EVr
V|U
(UUUUp (31)
This condition must be evaluated numerically using the following procedure:
For a given A, and for each question, draw M independent samples (U(m),.. ,.................... U!!m) from the empirical CDF of model scores (equivalently, from the empirical quantiles [0, 1). for m = =1,...... 1,...,,M M.
2. For each realization, compute:
(m)
p
Σラ=1 Acrsfom
(32)
n1
12 Σ (T( P ((m)
(m)) Σ (L) J. p.
1 U (m) pim)
(33)
3. Estimate the residual as
M R(A) Σ c(m)
(34)
4. Locate c such that R(At)= = 0 using a numerical root-finding method (e.g., bisection or Newton's 's method).
Hedging in Best-of-Poisson. Here, one draws a Poisson(ju) number of samples (plus one) and selects the proxy- moximall u. For uniform u, the density is (u) = (pu + 1)e -u(1-)) giving
(u, p)) 0₂ (u) = u u-- 1+ u pm + 1
The hedging equation
V Ir = Euwsa re(()) (u (u-1+
V
=
(35)
juu +
In an analogous way to the previous hedging equations, we solve the residual
M
R(u) = Σ ((
(36)
to locate uut such that R(xt) O. This jr is the Poisson optimal hedge. As in BoN, we approximate this expectation using the empirical CDF scores per question, mapping each candidate's 's rank to its quantile u, We compute the averaged residual across questions, and the root R(p')) 0 is found by bracketing, i.e, finding an interval [a, b] such that R(a) < O. and bisection search within this interval.
D.1 How does hedging compare to other hacking mitigation methods?
In this subsection, we supplement the discussion in the related works (Section 6). We discuss different techniques mitigate reward hacking and compare with hedging.
Training-Tiin Time Regularization. This the most common approach, typified by adding a diver- gence penalty to the reward maximization objective in RLHF. The goal is to prevent the policy from deviating too far from a trusted reference model. While a vital first line defense, this approach requires expensive full model retraining and risks under-optimization if the penalty is too strong.
27

---

Improving the Proxy Reward Model. A second category aims to build more robust reward signal that is inherently harder hack. Ensembling involves averaging scores from multiple. independently trained reward models. However, this method multiplies inference costs and cannot fix systemic biases shared by all models in the ensemble. Meanwhile, robust training methods [74 [1 modify the reward model's i training process itself to disentangle response quality from spurious cues, introducing significant complexity to the training pipeline.
InferenceTTime Methods. The closest analogues to our work include regularized Best-of-n (RBoN), which a powerful heuristic that balances the proxy reward against a penalty term [87, and inference-time pessimism [4] which uses principled rejection sampling step, requiring the estimation of a normalization constant.
Our Framework: Hedging introduces distinct approach that operates purely at inference time. It answers the question: given fixed reward model and fixed language model, what is the best we can do to improve performance just at inference? Hedging offers two key advantages:
1. Zero Retraining Cost and Maximum Flexibility: Hedging is applied to an existing, trained policy using existing proxy reward models. A practitioner can take single policy and calibrate it against different proxy rewards or for different tasks without incurring any retraining.
2. Principled and Efficient. We prove that this expected true reward function follows predictable "rise-and-fall" path with a single optimal peak. Hedge'lune therefore replaces heuristic balancing more involved sampling schemes with principled search for this unique optimum-a problem we prove is tractable.
However, the gains from hedging might be minimal compared to more involved methods, especially with weak proxy reward. One limitation of our work is that hedging requires tuning "ground truth" dataset. Ultimately, we see hedging not as a replacement for some the previous methods. Instead, HedgeTune offers computationally lightweight, to mitigate hacking at deployment.
centiccally-rouuneer and highly flexible method
E Experimental Details
In this section, provide additional details our experimental setup. We provide code here The experiments with the toy example and the verifiable rewards were done on a CPU (Apple M3 Chip). The experiments with the suman-preferenee were done on I A100 GPU with 48 GPU hours (including preliminary experiments and testing).
E.1
Toy Example
present toy example with one-dimensional rewards defined on [0, 1]. We set the proxy reward to be (z() = x and the gold reward
In Figure 1
(x( = x2 (1-x)
C-(-P)
C C= p p p+1
where C is normalization constant so that the gold reward is bounded between 0 and for convenience. We choose this gold reward, as the reward under the Best-of-n distribution has simple fooeed-orrm solution. We set p 12 and find the expected value of true reward and the KL divergence with respect to the reference distribution under four mechanisms:
1. Tilted Distribution: Exponential tilting of the gold reward, Qx(z) exp(Ar and of the proxy reward, Qx(x) x exp(A exp(Ary(0e (x)).
2. Best-of-n (BoN): Selection of the maximum of n i.i.d. uniform draws.
3. Soft Best-of-n (SBoN): Softmax-based sampling of i.i.d. uniform draws with inverse temperature
4. Best-of-Poisson (BoP): Selection of the maximum of i.idd uniform draws where n drawn from Poisson distribution with rate μ.
28

---

009
4.0
1.0
036
4.1
036
0.0
445
400
2.8
00
000
08
611
100
600
88
61
831
600
08
0.1
651
145
63
600
811
013
030
100
600
1..00
431
Figure 5: Accuracy vs. temperature for different sample sizes n with GPQA and Skywork Llama-3.1 8B. HedgeTune identifies the optimal temperature (dashed line) for each
We apply HedgeTune presented in Alg. for BoN and BoP. In both cases, we obtain operating point that corresponds exactly to the true hacking threshold, as shown in Figure The success of this algorithm hinges on Theorem I which guarantees the existence of (at least) one hacking threshold. However, this guarantee does not hold for Soft BoN. Therefore, the optimization problem becomes more challenging, and using vanilla estimators for the density causes numerical instabilities when applicd for Soft BoN.
E.2 Reward hacking in verifiable settings.
We include additional experiments from running HedgeTune with SBoN on the PPE dataset. HedgeTune finds the best operating temperature for given n. We plot the expected value of the accuracy and show the retrieved maximizing temperature. particular, with larger n, we need larger temperature to mitigate hacking, This corresponds to stronger KL regularization.
29

---

4.5
4.1
4.0
4.1
0.13
344
256
450
28
019
8.0
68
0.0
838
611
110
640
0.0
8.5
6.1
830
08
0.1
6.1
363
322
250
650
013
030
110
19:1
1910
38.1
354
Figure 6: Accuracy vs. temperature for different sample sizes n with MATH and InternLM2 1.8B. HedgeTune identifies the optimal temperature (dashed line) for each
30

---

3234
364-
420
1.1
036
4.15
338
1.10
400
80
019
800
085
6.1
110
6.0
600
03
8.5
611
831
600
0.5
010
0.1
124
124
600
81
013
030
110
2.0
1..0
365
Figure 7: Accuracy vs. temperature for different sample sizes n with MMLU and InternLM2 1.8B. HedgeTune identifies the optimal temperature (dashed line) for each
31

---

3.0
010
6.6
0.13
640
6.5
3.8
20
033
010
610
08
611
6.5
000
80
130
1.
60
8.0
110
34
01
600
830
Figure 8: Accuracy vs. temperature for different sample sizes n with MMLU and Llama3 OffsetBias 8B. HedgeTune identifies the optimal temperature (dashed line) for each
32

---

E.3 Reward hacking in the wild
To observe reward hacking in the wild, we follow the setup of Coste et al. 46 We first use an annotated dataset provided by [46] which contains 1,000 prompts from the validation split of AlpacaFarm dataset, along with 12,600 response generations per prompt from 1.4b fine-tuned Pythia model. Each prompt-response pair is labeled with the AlpacaFarm give 'gold' scores. Next, we would like to train proxy reward models on the preferences of this true reward. We randomly sample prompt with two responses from the annotated dataset and curate dataset of the form (prompt, chosen, rejected) where the chosen response the response with the higher gold reward score. We follow this procedure curate four datasets with varying sizes (10k, 20k, 46k, 80k). For each dataset, we consider two variants: one with no label noise and one with random 25% label noise. Next, we use the code kindly provided by the authors of [46] in their Github repository to train proxy reward models with the different datasets over four random seeds (1, 2, and 4) using their default hyperparameters (e.g.. 10 learning rate and five epochs). Lastly, we score the annotated dataset using the trained proxy reward models. The end result is set of 800 prompts, 12 600 responses per prompt, along with gold and proxy scores for each prompt-response pair.
to
While reward hacking can appear without label noise (see left panels of Figures and TO). reward hacking is more pronounced with label noise as expected. Moreover, reward hacking more apparent when the proxy reward is trained on less data. One potential explanation that, with fewer training examples, the proxy is less well-calibrated and its estimation errors vary more sharply across inputs. In that case, small n might produce a deceiving reward gain. In contrast, errors may surface early on with large training dataset, so true reward declines immediately as sampling increases. In cases of reward hacking, we see that SBoN with an appropriately chosen can (1) achieve the maximum reward achieved by BoN/BoP and (2) mitigate reward hacking, shown with the reward almost flatlining after reaches its peak value. We also witness cases where the proxy reward always misaligns with the gold reward, causing collapse of true reward from the onset of BoN. In that case, the optimal hedging behavior a uniform selection over responses, which is recovered with n = for BoN À for SBoN.
Interestingly, we observe instances of what we call reward grokking as shown in the right panels of Figures 10 and [ where the true reward decreases or flat-lines across low- to mid-range sample counts, only to undergo a sudden uptick at higher sample regimes, revealing a delayed but apparent realignment of proxy and true objectives. We leave detailed investigation of reward grokking and its implications for hedging strategies to future work.
33

---

8
Copette teee RRAAE
d'arrpasc
d'arrpha
No label noise
(b) 25% label noise
Figure 9: Expected true-reward vs. average number of samples with proxy trained on 000 examples: (a) without label noise; (b) with 25% label noise.
Turber sampas
Sumber d'smpphs
(a) No label noise
(b) 25% label noise
Figure 10: Expected true-reward vs. average number of samples with proxy trained on 20000 examples: (a) without label noise; (b) with 25% label noise.
Number
(a) No label noise
(b) 25% label noise
Figure 11: Expected true-reward average number of samples with proxy trained 46 000 examples: (a) without label noise; (b) with 25% label noise.
600
020
400
No label noise
(b) 25% label noise
Figure 12: Expected true-reward vs. average number of samples with proxy trained on 80 000 examples: (a) without label noise; (b) with 25% label noise.
34

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction highlight our study of reward hacking and multiple inference time methods (Best-of-n, Best of Poisson and Soft Best- of-n). Specifically, the paper includes theoretical study of hacking, description and characterization of the novel method Best of Poisson, and experiments validating the efficacy of our framework.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We include a discussion on the limitations in Section well in the the paper has limitations, but those are not discussed in the paper.
conclusion (see Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
35

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We include proofs of all our theorems in the appendix, and include assumptions, when appropriate, before or each theoretical statement.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
and Justification: We include details of our experimental setup in Section and the appendix. We use existing datasets and models for our setup and describe how we replicate existing work on reward hacking to test our sampling methods.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
36

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We include code necessary replicate the figures present in the paper on Github.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
jublic/guides htlpai///iidee//ooduuubbisssssss onliiicyy)c for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. //nips.c/ppie http:l/ii/guidde///eeeeeeeeeeeee
wwwiinsssioniiii ssionoolccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Training details, such as hyperparameters and data splits, as well as the optimizer used to train the proxy models, will be included in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
We have error bars for Figure 3 and include error bars when possible in the Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
37

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We include description of the hardware used for these experiments in the appendix of the paper.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the Code of Ethics and verified that our research conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We include paragraph discussing broader impact in the conclusion.
38

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not release any new data, models, or training algorithms that could result in misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite the datasets and models.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
39

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We anonymously include our code in the Appendix of the paper. We document in here and in the Github how the code replicate results.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not use human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
40

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method developed in this research did not involve L.MSs for any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neuri)) hrct::::/eerrissccc//ooeeeeeeeee //t/////ooeeeeeece////////..1. for what should or should not be described.
41


Reviews:
[{"review_id": "b52mNSV0Cf", "reviewer": "Reviewer_du2S", "summary": "Instead of using best-of-n sampling, a new strategy called best-of-Poisson is proposed and analyzed. Additionally, soft-best-of-n where instead of taking the mode of the distribution, the chosen outcome is sampled, is also considered. This helps avoiding overestimating (or underestimating) the proxy reward and tries to mitigate the problem of overoptimizing for the proxy reward in alignment.", "questions": "- A 'soft best-of-Poisson' seems like a natural extension of this work. Was it tried by any chance? Not having it does not affect the rating of the paper in any way. I am just curious.\n- I wonder if the specified for this paper is correct. I will let other reviewers have their say. Although yes this is alignment and it relates to safety. But I see it more as general machine learning maybe? Because it's not directly related to safety. I am not sure. But this might be something to think about.", "limitations": "Limitations are not adequately explicitly discussed in my opinion.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- Interesting new way of sampling that seems to avoid proxy reward hacking\n- Decent experimental results\n- Situated well with related work.\n\n**Weaknesses**\n- Introduction, paper begins by saying DPO aims to maximize reward function. DPO does not model a reward function and instead directly optimizes policy. Am I misunderstanding something?\n- The 10^-3 gap is an empirical result under specific settings, but it is not clear from the text that it is. It reads like a theoretical result: \"bounded by 10^-3\".\n- This is an online algorithm. Both online and offline algorithms are important in their own ways but this is both a strength and a weakness.\n- Access to true rewards needed. But this limitation explicitly stated.", "was_revised": true, "final_justification": "I have read all the discussions regarding this. I am divided between a 4 and a 5. (Borderline Accept vs Accept). My concerns are mostly addressed, and although the need for a validation set is pointed out by other reviewers (and the paper also explicitly mentions it), maybe this is useful when a validation set can be obtained. I am leaning towards accept conditional on clarifications and edits promised in the rebuttals. I am upgrading my rating to \"5\". My concerns about why it could not be a 5, but a 4 are only in terms of impact as the rating mentions \"high impact on one sub-area of AI\". My confidence score reflects this uncertainty. Other than I am happy with the technical evaluation, theoretical justifications, motivations, etc.", "rebuttal": "Thank you so much for your helpful feedback and thoughtful reading of our work. We address below the questions and points raised in your review and would be excited to answer any additional questions.\n\n---\n# W1: *Does DPO optimize a reward function?*\nThanks for raising this point. BoN, RLHF, and DPO make use of reward functions/signals. These reward functions can be implicit or explicit. For instance, BoN uses an *explicit* reward function (e.g., a model that assigns scores to responses). In contrast, DPO optimizes policy under an *implicit* reward function by recognizing that the optimal policy in RLHF (defined in Eq. 1 in our work) is an exponential tilt of the reference policy. DPO simply inverts this relationship and defines the loss directly on the policy (hence a language model being \"secretly\" a reward model). We will clarify this in the paper.\n\n---\n\n# W2: *The $10^{-3}$ KL gap as an empirical result*\nThe $10^{-3}$ gap presented in Appendix B is indeed based on a numerical estimate under our assumptions. We first derive a closed-form expression of the KL gap (see Eq. 84) in terms of the Poisson parameter $\\mu$. Next, we numerically find an upper bound of the expression (see Figure 2 for a plot) to demonstrate how negligible the gap is. For completeness, we will add a proof to the paper that shows *explicitly* why the bound holds. We added the proof to the end of our response so we would not disrupt the flow of our conversation. Please let us know if you have any questions about it.\n\n ---\n\n# W3/W4: *Algorithmic nature and access to true rewards*\nYou are correct that the need for true reward scores can be a practical barrier. Without a \"true reward\" to serve as a comparison (ground-truth) baseline, it is challenging to measure hacking. By definition, hacking occurs when a model over-optimizes to a proxy reward, and, consequently, precisely evaluating this phenomenon requires a reference ground-truth. We recognize this in Line 234 and plan to re-iterate it in a revised conclusion section, along with the point you raised on the online/offline nature of the method.\n\n---\n\n# Q1: *‘Soft best-of-Poisson’ seems like a natural extension*\nWe love this idea! After reading your review, we looked into this method and found that its properties can potentially extend our existing results, albeit at the cost of a more cumbersome mathematical analysis. We will include this as a blueprint for future research direction in the last section. We will add to the appendix: (i) the SBoP algorithm, (ii) its expected reward, and (iii) the KL divergence with respect to the reference policy. As one might expect, these correspond exactly to the expressions for Soft Best-of-$n$, but averaged over a Poisson distribution.\n\nWe find your question particularly interesting since it highlights our intention behind studying Best-of-Poisson and Soft Best-of-$n$. We wanted to leverage two distinct control mechanisms that inference-time alignment offers us: control over the generation through $n$ (be it deterministic or randomized) and control over the selection through the temperature $\\lambda$. Extensions such as SBoP show that these methods can be combined, offering richer control over the alignment process. Thank you for prompting this interesting discussion!\n\n ----\n\n# Q2: *Paper specified area*\nWe do recognize that our work is largely technical. AI safety concerns are, at their heart, *socio-technical* [1]. The impact of model failures due to hacking (and beyond) will depend on the application and stakeholders at hand. One clear example is an AI agent generating controversial and toxic content on X (previously Twitter) to optimize its engagement metric at the direct expense of content quality and safety [2]. Other dangerous patterns of failure include models that learn to be sycophantic instead of truthful [3] or actively exploit loopholes to circumvent oversight [4]. The goal of our work was to study the phenomenon of hacking from a rigorous mathematical perspective -- which can help with the design of hacking mitigation (and hence AI safety methods) with provable performance guarantees. We find technical approaches such as ours important for AI safety in practice, although recognize that they must exist within a larger safety ecosystem where rigorous technical foundations inform responsible deployment practices and policy decisions, and vice-versa. We will add this discussion to the last section of the paper.\n\n### References:\n1.  Amodei et al. \"Concrete Problems in AI Safety.\" 2016.\n2.  Pan et al. \"Feedback loops with language models drive in-context reward hacking.\" ICML 2024.\n3.  Sharma et al. \"Towards Understanding Sycophancy in Language Models.\" ICLR 2024.\n4.  OpenAI. \"Detecting misbehavior in frontier reasoning models.\" Blog post 2025.\n\n---\n\n# *Limitations*\n Thanks -- we will improve this. We will rename our conclusion section to \"Future Work and Limitations\" and further highlight the points raised by the reviewer:\n1.  Need for true reward model to measure and control hacking as *stated in Line 235*;\n2.  Algorithmic advantages and limitations in terms of the online/offline nature;\n3.  Richer hedging policies (e.g., \"Soft Best-of-Poisson\");\n4.  How this work can be used in AI safety practices and deployment\nThanks again for this suggestion.\n\n---\n\n### Majority of the references are arXiV preprints:\n\nThis was an oversight on our part during the preparation of the manuscript, and we will fix it promptly!\n\n--- \n\n# Summary:\n\nIn summary, your feedback has prompted us to strengthen the paper on several key fronts. We have replaced a numerical claim with a formal proof, suggested new inference-time methods for future work, and elaborated on the limitations of our work and its impact to the AI safety community. We believe these additions make a much stronger case for our contributions, and we thank you for your guidance and time!\n\n---\n\n# Proof of the KL gap:\n\nLet the following two functions\n$$\nq_{\\mu}(x) = \\bigl(1+\\mu x\\bigr) e^{\\mu(x-1)},\n\\qquad\ng_{\\lambda}(x) =\\frac{\\lambda e^{\\lambda x}}{e^{\\lambda}-1},\n\\qquad 0\\le x\\le1,\n$$\ndenote, respectively, the BoP density with the Poisson rate $\\mu$ and the exponential‑tilt density with parameter $\\lambda$. For any $\\mu > 0$, the value $\\lambda^\\ast=\\lambda^\\ast(\\mu)$ is chosen so that the two laws have the same first moment (i.e., same expected reward). We first note the expected reward of the BoP policy lies in the interval $(\\frac{1}{2}, 1)$. Additionally, the tilted expected reward is a strictly increasing function of $\\lambda$ over the same range since its derivative is the (positive) variance of the reward under the tilted distribution. By the Intermediate Value Theorem, there exists a unique $\\lambda^* = \\mu + \\delta(\\mu)$, where $\\delta(\\mu) = \\lambda^*(\\mu) - \\mu$, such that the two rewards are equal.\n\nBecause $g_{\\lambda(\\mu)}$ is the information projection of $q_{\\mu}$ onto the exponential family $\\{g_{\\lambda}\\}_{\\lambda>0}$ under the linear reward-matching constraint, the Csiszár–Pythagoras identity gives\n\n$$D_{\\text{KL}}(q_\\mu || \\pi_{\\text{ref}}) - D_{\\text{KL}}(g_{\\lambda(\\mu)} || \\pi_{\\text{ref}}) = D_{\\text{KL}}(q_\\mu || g_{\\lambda(\\mu)})$$\n\nTo bound this KL divergence, we use a general inequality which we derive below from the properties of the likelihood ratio function. Let $L(x) = \\frac{p(x)}{q(x)}$ denote the likelihood ratio between two distributions $p$ and $q$. We first define the KL divergence in terms of the convex function $\\varphi(t) = t \\log t - t + 1$ and then note that the expectation of any function is upper bounded by its essential supremum. We can retrieve the following upper bound:\n\n$$D_{\\mathrm{KL}}(p || q) =  E_q[\\varphi(L(x))]  \\le \\sup_{x} \\varphi(L(x)) = \\sup_{t \\in \\mathrm{Im}(L)} \\varphi(t)  $$\n\nIn our problem, the likelihood ratio is given by\n\n$$L_\\mu(x) =  \\frac{q_\\mu(x)}{g_{\\lambda^*}(x)} =  \\frac{(\\mu x + 1)(e^{\\lambda^\\ast} - 1)}{\\lambda^\\ast e^{\\mu}}  \\cdot e^{-\\delta \\mu x}$$\n\nTo obtain a uniform bound on $L_\\mu(x)$, define\n\n$$\\alpha = \\sup_{\\mu > 0, x \\in [0,1]} \\left| \\log L_\\mu(x) \\right|$$\nThis is a well-posed, nested optimization problem requiring numerical solution. The procedure is as follows:\n1.  For a given $\\mu$, numerically solve for the unique $\\lambda^*$ satisfying the reward-matching equation.\n2.  For the resulting pair $(\\mu, \\lambda^*)$, compute $\\max_{x \\in [0,1]} |\\log L_\\mu(x)|$ by evaluating the boundary values and any interior critical points.\n3.  Maximize the result of step (2) over all $\\mu > 0$.\n\nThis procedure is deterministic, and its solution yields a uniform bound:\n$$\ne^{-\\alpha} \\leq L_\\mu(x) \\leq e^{\\alpha}, \\quad \\text{for all } \\mu > 0 \\text{ and } x \\in [0,1] \\quad \\text{with }\\alpha \\approx 0.03968\n$$\nSince the function $\\varphi(t)$ is convex, we verify that its supremum over $[e^{-\\alpha}, e^{\\alpha}]$ is attained at an endpoint:\n$$\nD_{\\text{KL}}(q_\\mu || g_{\\lambda(\\mu)}) \\leq \\varphi(e^\\alpha) = \\alpha e^\\alpha - e^\\alpha + 1 \\approx 0.03968 \\cdot e^{0.03968} - e^{0.03968} + 1 \\approx 8 \\times 10^{-4}\n$$\nThis bound is independent of $\\mu$, and therefore holds uniformly."}, {"review_id": "zQ4CdwOMz9", "reviewer": "Reviewer_pDYx", "summary": "This paper studies the phenomenon of inference-time reward hacking when LLMs are steered by imperfect proxy reward models.  The authors (1) prove in Theorem 1 that, under mild TP2/MLR assumptions, the true reward as a function of an inference-time tuning parameter must either increase monotonically or exhibit a single peak before collapsing; (2) introduce Best-of-Poisson (BoP) sampling, which uses a Poisson-distributed sample count to approximate the ideal KL-constrained tilted distribution with a single parameter; and (3) propose HedgeTune, a root-finding algorithm that selects the optimal inference-time parameter (for BoN, SBoN, or BoP) to hedge against over-optimization, demonstrating in both synthetic and RLHF-style experiments that hedging achieves superior true-reward vs. distortion trade-offs with minimal overhead.", "questions": "1. **Sample efficiency:** How many true-reward evaluations does HedgeTune require to reliably converge to the optimal parameter?\n\n2. **Task breadth:** How does HedgeTune generalize to more complex reasoning tasks where a ground-truth reward is available?\n\n**Minor issue:**\n\n- Figure 3’s y-axis is mislabeled. It reads “Reward Improvement over Base Distribution (%)” but the values are shown as decimals (e.g., 0.8–0.9) rather than true percentages.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- The paper tackles the intriguing problem of inference-time reward hacking, precisely defines it, and offers rigorous theoretical analysis.\n- Using controlled synthetic setups and realistic LLM + proxy-RM pipelines show marked improvement in true-reward by hedging.  \n\n## Weaknesses\nThe key point of this paper's method is : **You must have a set of samples for which you can measure the true reward** (e.g., a validation set), or an equivalent human evaluator / high-quality scoring function, so that you can estimate the true reward curve. You then use a root-finding method to locate the peak where $g^{\\prime}(\\vartheta)=0$ .\n\nWithout a “true-reward evaluation set” or an unbiased, high-quality evaluator, HedgeTune cannot locate the true peak and thus cannot automatically select the optimal inference-time parameter. This limitation is particularly acute for datasets without an objective ground truth. As a result, HedgeTune may become impractical for many real-world tasks.", "was_revised": true, "final_justification": "I have read all of the reviews and author responses. My moderate assessment of 4 is consistent with most of the reviews and reviewers.", "rebuttal": "We sincerely thank you for your detailed and thoughtful feedback. We address next the questions and weaknesses raised in your review.\n\n --- \n\n# Q1: ***Sample efficiency of HedgeTune***\nWe agree sample efficiency is an important addition. We provide a new formal analysis, which we will add to the paper. As you noted, HedgeTune finds the root of an empirical gradient constructed from a dataset of $M$ true-reward evaluations. We leverage our paper's key theoretical contribution: under Theorem 1, the true gradient $R(\\theta)$ will have **exactly one-root**, moving from positive to negative. Under boundedness assumptions, we can apply standard concentration inequalities like Hoeffding's inequality.\n\nWe remind the reviewer of the following expression for the gradient of the true reward (see Eq. 8 in Appendix A for the derivation):\n\n$$R(\\theta) = E_{X \\sim p_\\theta}[r_t(X) \\cdot  \\varphi(X, \\theta)] = E_{U \\sim \\text{Unif}[0,1]}[r_t(U) \\cdot  \\varphi(U, \\theta) \\cdot p_\\theta(U)] = E_{U \\sim \\text{Unif}[0,1]}[h(U, \\theta)]$$\n\nwhere $r_t(\\cdot)$ is the true reward function, $\\varphi(\\cdot, \\cdot)$ is the score function, and $p_\\theta$ is the density of the inference-time policy parameterized by $\\theta$. We consider the following assumptions on a compact parameter space $\\Theta = [\\theta_{\\min}, \\theta_{\\max}]$ (equivalent to the search range of our algorithm):\n\n**Assumption 1 (Unimodality)**\nThe true performance function $f(\\theta) = E_{X \\sim p_\\theta}[r_t(X)]$ is unimodal with a unique maximizer $\\theta^* \\in \\Theta$, so its derivative, $R(\\theta)$, is monotonically decreasing. This holds under Theorem 1.\n\n**Assumption 2 (Bounded Gradient Summands)**\nThere exists a constant $H < \\infty$ such that:\n$$\n|h(u, \\theta)| \\le H \\quad \\text{for all } u \\in [0,1] \\text{ and } \\theta \\in \\Theta\n$$\n\n**Assumption 3 (Strong Slope)**\nThe true gradient $R(\\theta)$ has a non-zero slope at its root $\\theta^\\ast$. That is, there exists a constant $m>0$ such that for all $\\theta \\in \\Theta$:\n$$\n|R(\\theta)| \\ge m|\\theta - \\theta^\\ast|\n$$\n\n## Theorem\nTo guarantee that the estimated optimum $\\theta^\\dagger$ is within a desired accuracy $\\epsilon$ of the true optimum $\\theta^\\ast$ with probability at least $1 - \\alpha$, the required number of true-reward evaluations $M$ must satisfy:\n$$\nM = \\mathcal{O}\\left( \\frac{H^2}{m^2 \\epsilon^2} \\log\\left(\\frac{1}{\\alpha}\\right) \\right)\n$$\n\n**Proof**\n\nThe proof relies on showing that for a sufficiently large $M$, the empirical gradient will, with high probability, have opposite signs at the boundaries of the interval $[\\theta^\\ast - \\epsilon, \\theta^\\ast + \\epsilon]$. By the Strong Slope assumption (A3), the true gradient $R(\\theta)$ is bounded away from zero at the points $\\theta_{lo} = \\theta^\\ast - \\epsilon$ and $\\theta_{hi} = \\theta^\\ast + \\epsilon$:\n*   At $\\theta_{hi}$: $R(\\theta_{hi}) \\le -m|\\theta_{hi} - \\theta^*| = -m\\epsilon$.\n*   At $\\theta_{lo}$: $R(\\theta_{lo}) \\ge m|\\theta_{lo} - \\theta^*| = m\\epsilon$.\n\nLet $\\mathcal{F}$ be the event that the empirical gradient ${\\hat{R}}(\\theta)$  does not respect these signs, then by Hoeffding's inequality, we have that:\n\n$$\\begin{aligned}\n\\Pr[\\mathcal{F}] &= \\Pr\\left[ ({\\hat{R}}(\\theta_{hi}) \\ge 0) \\lor ({\\hat{R}}(\\theta_{lo}) \\le 0) \\right] \\\\\\\\\n&\\le \\Pr[{\\hat{R}}(\\theta_{hi}) - R(\\theta_{hi}) \\ge m\\epsilon] + \\Pr[{\\hat{R}}(\\theta_{lo}) - R(\\theta_{lo}) \\le -m\\epsilon] \\quad \\text{(by Union Bound and A3)} \\\\\\\\\n&\\le \\exp\\left(-\\frac{M(m\\epsilon)^2}{2H^2}\\right) + \\exp\\left(-\\frac{M(m\\epsilon)^2}{2H^2}\\right) = 2\\exp\\left(-\\frac{M m^2 \\epsilon^2}{2H^2}\\right)\n\\end{aligned}$$\n\nTo guarantee a confidence level of $1-\\alpha$, we require that:\n$$\nM \\ge \\frac{2H^2}{m^2\\epsilon^2}\\log\\left(\\frac{2}{\\alpha}\\right)\n$$\nThen, with probability at least $1-\\alpha$, we have $\\hat{g}_M(\\theta^* - \\epsilon) > 0$ and $\\hat{g}_M(\\theta^* + \\epsilon) < 0$. Since $\\hat{g}_M(\\theta)$ is a continuous function of $\\theta$, the Intermediate Value Theorem guarantees the existence of a root $\\theta^\\dagger \\in (\\theta^* - \\epsilon, \\theta^* + \\epsilon)$.\n\nThe values $H^2$ (estimator variance) and $1/m^2$ (flatness of $\\theta^\\ast$) capture the problem's intrinsic difficulty. We will add this analysis to the paper, as well as explicit bounds for Best-of-$n$ and Best-of-Poisson. We would be happy to answer any follow-up question on this point.\n\n---\n\n# Q2: ***Hedging in complex reasoning tasks where a ground-truth reward is available***\n\nThank you for this great question. To address this, we ran complementary experiments showing *our method mitigates reward hacking on complex reasoning benchmarks* (MMLU Pro, MATH, GPQA) even with large 8B proxy reward models!\n\n1.  *Datasets:* MMLU Pro (complex reasoning), MATH (mathematical problem solving), and GPQA (questions in natural sciences). A correct response gets a true reward of 1, while an incorrect response gets a true reward of 0.\n2.  *LLMs:* We leverage an open-source dataset with multiple responses generated per benchmark question from models such as GPT-4o-mini and Claude Haiku 3 [1].\n3.  *Proxy Rewards:* Each response is scored using various reward models, including InterLM-2 **1.8B**, Llama-3-Offset-Bias **8B**, and Skywork-Llama-3.1 **8B** [1].\n\nThe authors of [1] note reward hacking with Best-of-$n$ on a suite of reward models and datasets. Due to time constraints during the rebuttal period, we only test Soft Best-of-$n$; however, we intend to show the results for all the methods in the final paper. We present the results in the table below:\n\n| $n$ | MMLU (Llama-3 8B) BoN | MMLU (Llama-3 8B) Soft BoN | MMLU (internlm2-1.8b) BoN | MMLU (internlm2-1.8b) Soft BoN | GPQA (Skywork-Llama 3.1 8B) BoN | GPQA (Skywork-Llama 3.1 8B) Soft BoN | MATH (internlm2-1.8b) BoN | MATH (internlm2-1.8b) Soft BoN |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| 1 | 47.51±1.08 | 47.51±1.08 | 47.57±1.08 | 47.57±1.08 | 40.70±1.02 | 40.70±1.02 | 34.38±1.10 | 34.38±1.10 |\n| 2 | 53.10±1.24 | 53.12±1.24 | 50.33±1.18 | 50.32±1.18 | 42.98±1.17 | 42.98±1.16 | 37.32±1.18 | 37.34±1.18 |\n| 4 | 56.23±1.40 | 56.30±1.40 | 51.38±1.32 | 51.37±1.32 | 44.44±1.31 | 44.42±1.31 | 39.01±1.28 | 39.04±1.28 |\n| 6 | 57.21±1.50 | 57.32±1.49 | 51.78±1.41 | 51.75±1.41 | 44.98±1.40 | 44.95±1.38 | **39.38±1.34** | **39.39±1.33** |\n| 10 | **57.67±1.63** | 57.92±1.58 | 52.01±1.55 | 51.99±1.54 | 45.35±1.53 | **45.25±1.52** | 39.19±1.44 | 39.31±1.40 |\n| 12 | 57.65±1.68 | **58.01±1.65** | **52.12±1.60** | **52.13±1.60** | **45.36±1.58** | 45.22±1.53 | 39.03±1.50 | 39.24±1.42 |\n| 14 | 57.49±1.73 | 57.96±1.66 | 52.08±1.66 | 52.06±1.65 | 45.26±1.63 | 45.20±1.55 | 38.81±1.55 | 39.10±1.48 |\n| 18 | 57.13±1.82 | 57.98±1.72 | 52.01±1.76 | 52.03±1.75 | 44.89±1.74 | 45.04±1.53 | 38.25±1.65 | 38.97±1.49 |\n| 22 | 56.65±1.92 | 57.89±1.72 | 51.81±1.87 | 51.80±1.85 | 44.46±1.85 | 45.01±1.53 | 37.77±1.77 | 38.83±1.51 |\n| 26 | 56.34±2.01 | 57.90±1.75 | 51.38±1.99 | 51.49±1.64 | 43.98±1.97 | 44.92±1.53 | 37.34±1.90 | 38.72±1.43 |\n| 30 | 56.10±2.13 | 57.92±1.79 | 50.82±2.13 | 51.38±1.75 | 43.59±2.11 | 44.90±1.65 | 37.06±2.05 | 38.66±1.45 |\n| 32 | 56.05±2.19 | 57.94±1.81 | 50.48±2.21 | 51.29±1.77 | 43.35±2.19 | 44.86±1.62 | 36.91±2.13 | 38.66±1.51 |\n\n**The observed reward hacking curve exactly matches our theoretical prediction in Theorem 1.** For example, Best-of-$n$ clearly demonstrates hacking on the GPQA dataset, even with the very capable Skywork-Llama 3.1 8B as proxy reward (ranked # 10 out of all non-generative reward models on RewardBench [2]). While performance initially climbs from $40.70\\%$ at $n = 1$ to a peak of $45.36\\%$ at $n = 12$, it ultimately decreases to $43.35\\%$ as over-optimization sets in.\n\n**Hedging mitigates reward hacking in all cases.** Soft BoN delivers more stable and often superior performance than BoN. On GPQA, it reduces the performance drop from over 2 points to less than 0.4 points. On MATH, where hacking is most severe, it effectively mitigates BoN’s 2.5-point collapse. This stability also enables Soft BoN to achieve a higher peak score on MMLU (Llama-3 8B).\n\nThe hedging benefit holds for learning from human preferences (RLHF), complex reasoning (MMLU), math (MATH), and science questions (GPQA), and is demonstrated with multiple large reward models (8B parameters). Please let us know if you have any additional questions about our setup or our results!\n\n**References:**\n1.  Frick et al. *How to Evaluate Reward Models for RLHF*. ICLR 2025. \n2.  Lambert et al. *RewardBench: Evaluating Reward Models for Language Modeling*. NAACL 2025.\n\n---\n\n# W1: ***Need for true reward***\n\nYou are correct that the need for true reward scores can be a practical barrier. We note that without a “true reward” to serve as a comparison (ground-truth) baseline, it is challenging to measure hacking. By definition, hacking occurs when a model over-optimizes to a proxy reward, and, consequently, precisely evaluating this phenomenon requires a reference ground-truth. Our experimental results on MMLU, MATH, and GPQA confirm that our framework offers a practical and effective tool to mitigate hacking in real-world reasoning tasks. However, we recognize this limitation in Line 234, and plan to re-iterate this in a revised conclusion section.\n\n---\n\n### Minor Labeling Issue:\n\nThanks for catching this. We will fix the axis’ label before the final version.\n\n---\n\n# Summary:\nWe are glad that your detailed feedback has strengthened the paper on three key fronts. First, we have provided a new theoretical analysis of HedgeTune's sample complexity, formally proving its practical efficiency. Second, we have demonstrated the generality of our approach with extensive new experiments on complex reasoning benchmarks (MMLU, MATH, GPQA) using powerful 8B reward models. Finally, we have clarified the core limitation of our method while highlighting its value in high-stakes, verifiable domains. Thank you for your constructive guidance and let us know if you have any questions"}, {"review_id": "OgEDK7A451", "reviewer": "Reviewer_51Rg", "summary": "This paper addresses the problem of reward hacking in large language models (LLMs) that arises from overoptimizing imperfect proxy reward models during inference-time alignment. The authors formalize this phenomenon and introduce the concept of the \"hacking threshold\", the point beyond which optimizing for proxy rewards degrades true performance. They propose a new inference-time method, Best-of-Poisson (BoP), which samples a variable number of outputs based on a Poisson distribution to balance reward and distortion with a single parameter. They also introduce HedgeTune, an algorithm for tuning these methods to optimize true reward while avoiding overoptimization. Experiments demonstrate the benefits of hedging in improving distortion-reward tradeoffs.", "questions": "1. Theorem 1 relies on technical assumptions such as $\\textnormal{TP}_2$ and strictly MLR. For readers without a precise technical background, could the authors provide additional intuition or interpretation for these assumptions? In particular, clarifying how these conditions correspond to realistic properties of models used in practice would help make the theoretical contributions more accessible and broadly relevant.\n\n2. The reviewer finds the empirical evaluation somewhat limited in scope, which makes it difficult to assess the generalizability of the proposed methods across different LLM architectures, datasets, and proxy reward models. While the use of true rewards is valuable for controlled analysis, it would also be helpful to evaluate performance on established alignment benchmarks that provide scores in terms of the alignment of LLMs. Such benchmarks could offer complementary evidence for the effectiveness of hedging strategies in more realistic or diverse settings.\n\n3. How does the proposed hedging method compare with other approaches for mitigating reward hacking, such as regularization, ensembling, etc.? A comparative empirical or conceptual analysis against these baselines would help situate the proposed approach more clearly within the broader literature.\n\nI'm happy to consider increasing my score if the authors provide compelling responses to my questions, particularly by clarifying theoretical assumptions and expanding the empirical evidence to support the general applicability of their methods.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: This paper provides a rigorous analysis of reward hacking in inference-time alignment for language models, combining theoretical insights with practical methods. The presentation is generally clear, with key concepts like the hacking threshold and hedging strategies explained in a comprehensible way. The work addresses an important problem in LLM alignment and contributes useful tools, such as Best-of-Poisson (with the advantage of having a single tunable parameter) and HedgeTune, that offer practical value. While the core ideas build on existing alignment techniques, the formulation and integration of hedging into inference-time methods add a degree of originality and relevance.\n\nWeaknesses: The presentation of the paper could be improved in the sense that several figures are difficult to read due to formatting and some passages would benefit from proofreading to correct minor typos and enhance readability. Although the paper addresses a relevant problem and proposes useful methods, the empirical evaluation is relatively narrow and may not fully demonstrate the generality of the approach.", "was_revised": true, "final_justification": "Based on the rebuttal of the authors with the additional experiments, the reviewer finds the empirical evaluation more convincing. Therefore, my recommended score increases to 4: Borderline accept.", "rebuttal": "Thank you for the thoughtful review. We address your questions and comments below and would be happy to answer any additional questions during the discussion period.\n\n---\n\n# Q1: *Additional intuition on technical assumptions like $TP_2$ and strictly MLR*\n\nWe will add the following discussion about the technical conditions to the paper.\n\n**What the Technical Conditions Mean Intuitively:**\nThe conditions guarantee that the inference-time policies behave in a *well-ordered* manner: increasing the tuning parameter $\\theta$ consistently makes the policy \"greedier\" and more likely to select outputs with high proxy rewards. The clearest example is the $n$ in Best-of-$n$: the larger the $n$ (which is generalized by our $\\theta$ parameter), the more aggressively we optimize for the proxy. \n\nThis provides the most natural setting to study reward hacking. The monotone likelihood ratio (MLR) property induces a simple pairwise ordering: for any two policies with different parameters, the policy with the larger parameter is greedier. This property arises as a consequence of the stronger total positivity of order 2 ($TP_2$) condition. $TP_2$ is a structural property of the entire policy family, ensuring that such monotonic behavior holds uniformly across all parameter values and outcomes. It is a standard tool in the study of order statistics, utility theory, and stochastic processes.\n\n**Why They Help Us Understand Hacking:**\nThe core challenge of reward hacking is that the true reward function can be complex. This is where our theoretical contribution yields a significant practical payoff. We leverage the Variation-Diminishing Theorem to show that for **any** bounded reward, the expected true reward must follow a simple, predictable path that we characterize exactly. This result makes the problem of reward hacking tractable and allows us to search for the single, optimal *hacking threshold*.\n\n---\n\n# W2/Q2: *Empirical Setup*\n\nThank you for this great question. To address this, we ran complementary experiments showing **our method mitigates reward hacking on complex reasoning benchmarks** (MMLU Pro, MATH, GPQA) even with large 8B proxy reward models.\n\n1.  *Datasets:* MMLU Pro (complex reasoning), MATH (mathematical problem solving), and GPQA (questions in natural sciences). A correct response gets a true reward of 1, while an incorrect response gets a true reward of 0.\n2.  *LLMs:* We leverage an open-source dataset with multiple responses generated per benchmark question from models such as GPT-4o-mini and Claude Haiku 3 [1].\n3.  *Proxy Rewards:* Each response is scored using various reward models, including InterLM-2 **1.8B**, Llama-3-Offset-Bias **8B**, and Skywork-Llama-3.1 **8B** [1].\n\nThe authors of [1] note reward hacking with Best-of-$n$ on a suite of reward models and datasets. Due to time constraints during the rebuttal period, we only test Soft Best-of-$n$; however, we intend to show the results for all the methods in the final paper. We present the results in the table below:\n\n| $n$ | MMLU (Llama-3 8B) BoN | MMLU (Llama-3 8B) Soft BoN | MMLU (internlm2-1.8b) BoN | MMLU (internlm2-1.8b) Soft BoN | GPQA (Skywork-Llama 3.1 8B) BoN | GPQA (Skywork-Llama 3.1 8B) Soft BoN | MATH (internlm2-1.8b) BoN | MATH (internlm2-1.8b) Soft BoN |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| 1 | 47.51±1.08 | 47.51±1.08 | 47.57±1.08 | 47.57±1.08 | 40.70±1.02 | 40.70±1.02 | 34.38±1.10 | 34.38±1.10 |\n| 2 | 53.10±1.24 | 53.12±1.24 | 50.33±1.18 | 50.32±1.18 | 42.98±1.17 | 42.98±1.16 | 37.32±1.18 | 37.34±1.18 |\n| 4 | 56.23±1.40 | 56.30±1.40 | 51.38±1.32 | 51.37±1.32 | 44.44±1.31 | 44.42±1.31 | 39.01±1.28 | 39.04±1.28 |\n| 6 | 57.21±1.50 | 57.32±1.49 | 51.78±1.41 | 51.75±1.41 | 44.98±1.40 | 44.95±1.38 | **39.38±1.34** | **39.39±1.33** |\n| 10 | **57.67±1.63** | 57.92±1.58 | 52.01±1.55 | 51.99±1.54 | 45.35±1.53 | **45.25±1.52** | 39.19±1.44 | 39.31±1.40 |\n| 12 | 57.65±1.68 | **58.01±1.65** | **52.12±1.60** | **52.13±1.60** | **45.36±1.58** | 45.22±1.53 | 39.03±1.50 | 39.24±1.42 |\n| 14 | 57.49±1.73 | 57.96±1.66 | 52.08±1.66 | 52.06±1.65 | 45.26±1.63 | 45.20±1.55 | 38.81±1.55 | 39.10±1.48 |\n| 18 | 57.13±1.82 | 57.98±1.72 | 52.01±1.76 | 52.03±1.75 | 44.89±1.74 | 45.04±1.53 | 38.25±1.65 | 38.97±1.49 |\n| 22 | 56.65±1.92 | 57.89±1.72 | 51.81±1.87 | 51.80±1.85 | 44.46±1.85 | 45.01±1.53 | 37.77±1.77 | 38.83±1.51 |\n| 26 | 56.34±2.01 | 57.90±1.75 | 51.38±1.99 | 51.49±1.64 | 43.98±1.97 | 44.92±1.53 | 37.34±1.90 | 38.72±1.43 |\n| 30 | 56.10±2.13 | 57.92±1.79 | 50.82±2.13 | 51.38±1.75 | 43.59±2.11 | 44.90±1.65 | 37.06±2.05 | 38.66±1.45 |\n| 32 | 56.05±2.19 | 57.94±1.81 | 50.48±2.21 | 51.29±1.77 | 43.35±2.19 | 44.86±1.62 | 36.91±2.13 | 38.66±1.51 |\n\n**The observed reward hacking curve exactly matches our theoretical prediction in Theorem 1.** For example, Best-of-$n$ clearly demonstrates hacking on the GPQA dataset, even with the very capable Skywork-Llama 3.1 8B as proxy reward (ranked #10 out of all non-generative reward models on RewardBench [2]). While performance initially climbs from $40.70\\%$ at $n = 1$ to a peak of $45.36\\%$ at $n = 12$, it ultimately decreases to $43.35\\%$ as over-optimization sets in.\n\n**Hedging mitigates reward hacking in all cases.** Soft BoN delivers more stable and often superior performance than BoN. On GPQA, it reduces the performance drop from over 2 points to less than 0.4 points. On MATH, where hacking is most severe, it effectively mitigates BoN’s 2.5-point collapse. This stability also enables Soft BoN to achieve a higher peak score on MMLU (Llama-3 8B).\n\nThe hedging benefit holds for learning from human preferences (RLHF), complex reasoning (MMLU), math (MATH), and science questions (GPQA), and is demonstrated with multiple large reward models (8B parameters). Please let us know if you have any additional questions about our setup or our results.\n\n### References:\n1.  Frick et al. *How to Evaluate Reward Models for RLHF*. ICLR 2025.\n2.  Lambert et al. *RewardBench: Evaluating Reward Models for Language Modeling*. NAACL 2025.\n\n---\n\n# Q3: ***How does hedging compare with other approaches for mitigating reward hacking***\n\nThank you for this excellent question. We provide the following conceptual analysis. The research community has developed several important strategies to combat reward hacking, which (broadly speaking) fall into the following categories:\n\n**Training-Time Regularization.**\nThis is the most common approach, typified by adding a KL-divergence penalty to the reward maximization objective as in RLHF. The goal is to prevent the policy from deviating too far from a trusted reference model. While a vital first line of defense, this approach requires expensive full model retraining and risks under-optimization if the penalty is too strong.\n\n**Improving the Proxy Reward Model.**\nA second category aims to build a more robust reward signal that is inherently harder to hack. Ensembling involves averaging scores from multiple, independently trained reward models. However, this method multiplies inference costs and cannot fix systemic biases shared by all models in the ensemble. Meanwhile, robust training methods (e.g., RRM, InfoRM) modify the reward model's training process itself to disentangle response quality from spurious cues, introducing significant complexity to the training pipeline.\n\n**Inference-Time Methods.**\nThe closest analogues to our work include:\n*   **Regularized Best-of-$n$ (RBoN):** A powerful heuristic that balances the proxy reward against a penalty term: $R(x) - \\beta \\cdot \\text{Penalty}(x)$.\n*   **Inference-Time Pessimism:** This method uses a principled rejection sampling step, which requires estimating a normalization constant.\n\n**Our Framework:**\nHedging introduces a distinct approach that operates purely at inference time. It offers two key advantages:\n\n1.  **Zero Retraining Cost and Maximum Flexibility:** Hedging is applied to an existing, trained policy using existing proxy reward models. A practitioner can take a single policy and calibrate it against different proxy rewards or for different tasks without incurring any retraining costs.\n2.  **Principled and Efficient:** We prove (Theorem 1) that the expected true reward function follows a predictable \"rise-and-fall\" path with a single optimal peak. HedgeTune therefore replaces heuristic balancing or more involved sampling schemes with a principled search for this unique optimum—a problem we prove is tractable.\n\nHowever, one key limitation is that hedging requires a tuning dataset with \"ground truth\" labels. Ultimately, we see hedging not as a replacement for some of the previous methods. Instead, HedgeTune offers a computationally lightweight, theoretically-grounded, and highly flexible method to mitigate hacking at deployment.\n\n---\n\n# W1: *The presentation of the paper could be improved*\n\n**Answer:** We have gone through the paper again to fix typos and improve figure formatting. The major edits include enhancing the figure legends and style, streamlining the discussion of technical assumptions in Section 2 (e.g., merging Lines 153-160 with Lines 177-185), providing more direct proofs in the appendix, and making other minor edits throughout the paper. We believe these changes significantly improve the paper's quality and appreciate your helpful feedback.\n\n---\n\n# Summary:\nWe thank you for providing a clear path to improve the paper. In response, we have demonstrated our method’s practical generality with new experiments on complex reasoning benchmarks (MMLU, MATH, GPQA) using large reward models. We have also clarified the intuition behind our core theoretical assumptions to improve accessibility, situated our work with a new conceptual analysis of alternative mitigation methods, and improved the presentation of the paper. We believe the paper is significantly stronger as a result of your review, so thank you!"}, {"review_id": "77Anoc8Cp4", "reviewer": "Reviewer_Eaod", "summary": "This paper addresses the fundamental challenge of reward hacking in LLM alignment, particularly focusing on inference-time methods like BoN sampling. The authors mathematically characterize reward hacking, proving its inevitability under common conditions where proxy rewards are imperfect approximations of true rewards. To mitigate this, they propose \"hedging\" strategies and introduce a novel inference-time method called Best-of-Poisson (BoP), which aims to approximate the optimal reward-tilted distribution with a single parameter. They also develop HedgeTune, an algorithm for finding the optimal hedging parameter across BoN, Soft Best-of-n (SBON), and BoP. Experimental results, both in synthetic and RLHF scenarios, demonstrate that hedging mitigates reward hacking and achieves superior distortion-reward tradeoffs.", "questions": "1. Can the framework be extended to work with noisy, approximated, or preference-based true rewards, and if so, what are the theoretical guarantees and practical implications?\n1. What is the sensitivity of Theorem 1's inevitability claim and HedgeTune's performance to deviations from the assumption that proxy reward scores follow a uniform distribution? Have you explored the impact of non-uniform proxy reward distributions, and if so, what were the findings?\n1. Given the identified inevitability of reward hacking, how do you foresee these hedging strategies scaling to much larger LLMs (e.g., 70B) and more complex, open-ended alignment tasks beyond the scope of the current experiments?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\n1. **Novelty in Hedging for Reward Hacking:** The paper introduces a principled approach to \"hedging\" against reward hacking specifically at inference time, offering a new perspective on mitigating this pervasive problem.\n1. **Theoretical Characterization:** The formalization and proof of the inevitability of reward hacking (Theorem 1) provide a strong theoretical foundation for the observed empirical phenomenon, which is crucial for understanding its mechanisms.\n1. **Best-of-Poisson (BoP) as a Novel Method:** BoP is presented as a mathematically elegant and computationally efficient alternative to BoN, capable of approximating optimal reward-distortion tradeoffs with a single tunable parameter, which simplifies practical application.\n1. **Practical Algorithm (HedgeTune):** The HedgeTune algorithm offers a concrete method for operationalizing the hedging strategy by identifying the optimal parameter to balance reward exploitation and fidelity to the reference distribution.\n\n**Weaknesses**\n\n1. **Practicality of True Reward Oracle Requirement:** A limitation is the explicit requirement for \"access to the true reward\" (Section 4, line 234) for HedgeTune to operate. In real-world LLM alignment, the true reward is often unknown, costly, or impossible to precisely quantify. If the true reward were readily available, simpler pre-experiment hyperparameter tuning could potentially find the optimal operating point without needing sophisticated hedging algorithms, thus diminishing the practical utility of the proposed method in its current form. This assumption significantly limits the immediate applicability of HedgeTune.\n1. **Limited Justification for Proxy Uniformity Assumption:** The paper assumes that proxy reward scores have a uniform distribution (Section 2, line 120-122; Section 4, line 236). While stated as incurring \"little loss in generality,\" the impact of this strong assumption on the theoretical guarantees and practical performance, especially if real-world proxy reward distributions deviate significantly from uniform, is not thoroughly explored.\n1. **Experimental Scope and Real-World Applicability:** While experiments include a \"hiding in the wild\" setup, the overall experimental validation might still be limited. The use of Pythia-1.4B models and a synthetic reward hacking scenario, while controlled, may not fully capture the complexities and nuances of reward hacking observed in larger, more diverse LLMs and real-world alignment tasks.", "was_revised": true, "final_justification": "I have read the authors' detailed rebuttal and found it to be helpful. It has resolved the main concerns that made my initial assessment \"borderline\" and has reinforced my belief that this is a strong contribution. I am maintaining my positive score and raising the confidence score to 3.", "rebuttal": "Thank you for your feedback! We will address each of your points below.\n\n---\n\n# W1: *Practicality of True Reward Oracle Requirement*\n\nYou've highlighted a key practical challenge: the high cost of true reward data. HedgeTune is designed for this exact scenario. Instead of a costly brute-force search, it leverages the unimodality guaranteed by Theorem 1 to apply an efficient root-finding algorithm. This finds the optimal parameter with significantly fewer expensive validation samples. Our sample complexity analysis, detailed in our response to Reviewer pDYx, provides a formal treatment of this efficiency. We will add a discussion on the practical limitation to the conclusion.\n\n---\n\n# Q1: *Working with noisy, approximated, or preference-based true rewards*\n\nThis is an excellent question about future directions, which we will add to our conclusion.\n\n**Noisy/Approximated Rewards:** We hypothesize that the true reward's unimodal structure acts as a natural regularizer, helping HedgeTune find the right peak. However, more work is needed for a formal theoretical treatment.\n\n**Preference-Based Rewards:** This is the most exciting and challenging future direction. One promising method would be to design a surrogate objective based on preference data akin to DPO. A central theoretical question would be to prove that this new objective also exhibits a unimodal or single-crossing property. We believe this is a rich area for future work. We will add a detailed discussion of these potential extensions to our conclusion.\n\n---\n\n# W2/Q2: *Proxy Uniformity Assumption*\n\nThank you for these points. **The proxy reward scores do not need to be uniform themselves.** \n\nWe first map the proxy rewards to a standardized discrete space by transforming them into their quantiles using their own empirical CDF $F_p$. This process, defined by the relation\n$$ u = F_p(r_p) $$\nensures that the new reward $u$ is uniform on $[0, 1]$ by construction. This transformation preserves the rank-ordering of the scores: an output with a higher proxy reward will also have a higher transformed score. Of course -- the resulting random variable will not be a continuous uniform random variable (which we assume), and instead will be a discrete uniform random variable. Recent work [1] has shown for a dense enough set of examples, the error incurred by this continuous model is negligible in the context of LLMs. We recognize this point was not sufficiently clear in the manuscript, and we will revise it to precisely explain our construction.\n\nNevertheless, your question correctly highlights if this approximation step could be entirely removed. **Theorem 1 can in fact apply directly for the case where rewards have discrete support** with small changes to the current proof (swapping integrals with sums). This is particularly exciting, as it confirms that reward hacking follows a predictable behavior and this is a **fundamental** property of these inference methods.\n\nInstead of the continuous reward $u \\in [0, 1]$, we consider a discrete set of $M$ possible outputs, ranked from worst to best according to their proxy reward scores. Let the true reward for the output at rank $k$ be $w_k$. The probability matrix $P(\\theta, k) = p_k(\\theta)$ gives the probability of selecting the output at rank $k$ given an inference-time parameter $\\theta$. The expected true reward is now expressed as a discrete sum:\n$$\n\\mathbb{E}[R_t \\mid \\theta] = \\sum_{k=1}^M p_k(\\theta) \\cdot w_k\n$$\nWe note that the *Variation-Diminishing Theorem* we employ is a general result that applies to discrete sums just as it does to continuous integrals. The Variation-Diminishing Theorem guarantees that if our probability matrix $P$ is $TP_2$, then expected true reward $\\mathbb{E}[R_t \\mid \\theta]$ remains simple and unimodal. The core result—and thus our ability to find a single optimal HedgeTune parameter—holds! This restated proof solidifies our findings and will be incorporated into the final paper. We thank you for prompting this important extension.\n\n---\n\n# W3: *Experimental Scope and Real-World Applicability*\n\nThank you for this great question. To address this, we ran complementary experiments showing our method mitigates reward hacking on complex reasoning benchmarks (MMLU Pro, MATH, GPQA) even with large 8B proxy reward models!\n\n1.  *Datasets:* MMLU Pro (complex reasoning), MATH (mathematical problem solving), and GPQA (questions in natural sciences). A correct response gets a true reward of 1, while an incorrect response gets a true reward of 0.\n2.  *LLMs:* We leverage an open-source dataset with multiple responses generated per benchmark question from models such as GPT-4o-mini and Claude Haiku 3 [2].\n3.  *Proxy Rewards:* Each response is scored using various reward models, including InterLM-2 **1.8B**, Llama-3-Offset-Bias **8B**, and Skywork-Llama-3.1 **8B** [2].\n\nThe authors of [2] note reward hacking with Best-of-$n$ on a suite of reward models and datasets. Due to time constraints during the rebuttal period, we only test Soft Best-of-$n$; however, we intend to show the results for all the methods in the final paper. We present the results in the table below:\n\n| $n$ | MMLU (Llama-3 8B) BoN | MMLU (Llama-3 8B) Soft BoN | MMLU (internlm2-1.8b) BoN | MMLU (internlm2-1.8b) Soft BoN | GPQA (Skywork-Llama 3.1 8B) BoN | GPQA (Skywork-Llama 3.1 8B) Soft BoN | MATH (internlm2-1.8b) BoN | MATH (internlm2-1.8b) Soft BoN |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| 1 | 47.51±1.08 | 47.51±1.08 | 47.57±1.08 | 47.57±1.08 | 40.70±1.02 | 40.70±1.02 | 34.38±1.10 | 34.38±1.10 |\n| 2 | 53.10±1.24 | 53.12±1.24 | 50.33±1.18 | 50.32±1.18 | 42.98±1.17 | 42.98±1.16 | 37.32±1.18 | 37.34±1.18 |\n| 4 | 56.23±1.40 | 56.30±1.40 | 51.38±1.32 | 51.37±1.32 | 44.44±1.31 | 44.42±1.31 | 39.01±1.28 | 39.04±1.28 |\n| 6 | 57.21±1.50 | 57.32±1.49 | 51.78±1.41 | 51.75±1.41 | 44.98±1.40 | 44.95±1.38 | **39.38±1.34** | **39.39±1.33** |\n| 10 | **57.67±1.63** | 57.92±1.58 | 52.01±1.55 | 51.99±1.54 | 45.35±1.53 | **45.25±1.52** | 39.19±1.44 | 39.31±1.40 |\n| 12 | 57.65±1.68 | **58.01±1.65** | **52.12±1.60** | **52.13±1.60** | **45.36±1.58** | 45.22±1.53 | 39.03±1.50 | 39.24±1.42 |\n| 14 | 57.49±1.73 | 57.96±1.66 | 52.08±1.66 | 52.06±1.65 | 45.26±1.63 | 45.20±1.55 | 38.81±1.55 | 39.10±1.48 |\n| 18 | 57.13±1.82 | 57.98±1.72 | 52.01±1.76 | 52.03±1.75 | 44.89±1.74 | 45.04±1.53 | 38.25±1.65 | 38.97±1.49 |\n| 22 | 56.65±1.92 | 57.89±1.72 | 51.81±1.87 | 51.80±1.85 | 44.46±1.85 | 45.01±1.53 | 37.77±1.77 | 38.83±1.51 |\n| 26 | 56.34±2.01 | 57.90±1.75 | 51.38±1.99 | 51.49±1.64 | 43.98±1.97 | 44.92±1.53 | 37.34±1.90 | 38.72±1.43 |\n| 30 | 56.10±2.13 | 57.92±1.79 | 50.82±2.13 | 51.38±1.75 | 43.59±2.11 | 44.90±1.65 | 37.06±2.05 | 38.66±1.45 |\n| 32 | 56.05±2.19 | 57.94±1.81 | 50.48±2.21 | 51.29±1.77 | 43.35±2.19 | 44.86±1.62 | 36.91±2.13 | 38.66±1.51 |\n\n\n**The observed reward hacking curve exactly matches our theoretical prediction in Theorem 1.** For example, Best-of-$n$ clearly demonstrates hacking on the GPQA dataset, even with the very capable Skywork-Llama 3.1 8B as proxy reward (ranked # 10 out of all non-generative reward models on RewardBench [3]). While performance initially climbs from 40.70% at $n = 1$ to a peak of 45.36% at $n = 12$, it ultimately decreases to 43.35% as over-optimization sets in.\n\n**Hedging mitigates reward hacking in all cases.** Soft BoN delivers more stable and often superior performance than BoN. On GPQA, it reduces the performance drop from over 2 points to less than 0.4 points. On MATH, where hacking is most severe, it effectively mitigates BoN’s 2.5-point collapse. This stability also enables Soft BoN to achieve a higher peak score on MMLU (Llama-3 8B).\n\nThe hedging benefit holds for learning from human preferences (RLHF), complex reasoning (MMLU), math (MATH), and science questions (GPQA), and is demonstrated with multiple large reward models (8B parameters). Please let us know if you have any additional questions about our setup or our results!\n\n---\n\n# Q3: *Scaling to larger LLMs and complex, open-ended tasks*\n\nWe hope that our supplementary experimental results on complex reasoning demonstrate the effectiveness of hedging, even with larger reward models. The advantage of hedging is that it *does not change* across setups. Given access to \"true\" reward measurements, you would tune inference-time parameters to operate in a regime where you do not over-optimize to a proxy reward. \n\nWhat does change is our ability to observe differences between true and proxy rewards, or even obtain such measurements. This might be feasible in narrow tasks, but more difficult in LLMs that are used across tasks like summarizing financial documents or refactoring a legacy codebase. Obtaining a consistent and reliable \"true reward\" signal in such domains is a significant research challenge in itself. If a form of hacking is undetectable by our evaluation process (human or otherwise), HedgeTune cannot guard against it. As the research community develops better methods for measuring hacking, we view HedgeTune as an immediate, lightweight, and principled tool to translate that into a more robust inference-time policy.\n\n\n---\n\n# Summary:\n\nWe thank you for the thoughtful questions that have significantly clarified and strengthened our work. In response to your feedback, we will reframe HedgeTune's practicality for real-world tasks with expensive but limited ground-truth data. We have also extended our core theory to the discrete case and demonstrated our method's scalability with extensive new experiments on reasoning benchmarks. We hope these additions directly address your concerns and we appreciate your help in improving the paper.\n\n---\n\n**References:**\n1.  Gui et al. *BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling*. NeurIPS 2024.\n2.  Frick et al. *How to Evaluate Reward Models for RLHF*. ICLR 2025.\n3.  Lambert et al. \"RewardBench: Evaluating Reward Models for Language Modeling\". NAACL 2025."}]

Author rebuttal:

Meta-review: This paper provides a formal characterization of reward hacking in inference-time alignment for large language models. The authors prove that under mild technical assumptions, optimizing for a proxy reward inevitably leads to a degradation in true performance after a certain threshold. To mitigate this, they introduce "hedging" as a strategy, propose a novel sampling method called Best-of-Poisson (BoP), and develop an efficient algorithm, HedgeTune, to find the optimal inference-time parameter that maximizes true reward without over-optimizing. Its core scientific contribution is the formal characterization, under mild assumptions, of how optimizing for a proxy reward inevitably leads to a decline in true performance, following a predictable unimodal curve. 

The strengths, as noted by reviewers (pDYx, Eaod), lie in this strong theoretical foundation and the proposal of novel, practical mitigation methods like Best-of-Poisson and HedgeTune (du2S, 51Rg). The principal weakness, unanimously identified, is the method's reliance on a validation set with access to true rewards, which may limit its applicability (pDYx, Eaod). However, the authors' rebuttal provided substantial new experiments on complex reasoning benchmarks with large models, directly addressing initial concerns about the work's empirical scope and demonstrating its relevance. This, combined with a new analysis of sample complexity, makes a compelling case for the paper's value in settings where a limited set of ground-truth evaluations is feasible. The work stands out for its theoretical elegance and demonstrated practical utility, justifying its acceptance.
Decision: Accept (spotlight)