From Curiosity to Caution: Mitigating Reward Hacking for Best-of-$N$ with Pessimism
Authors: Zhuohao Yu
Steven Wu
Adam Block
Venue: ICLR 2026 Poster (2026)
Keywords: Reward Hacking
Reward Models
Pessimism
Inference-time Scaling
Large Language Models
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Inference-time compute scaling has emerged as a powerful paradigm for improving language model performance on a wide range of tasks, but the question of how best to use the additional compute remains open.  A popular approach is *Best-of-$N$* (BoN) sampling, where $N$ candidate responses are generated, scored according to a reward model, and the highest-scoring response is selected.  While this approach can improve performance, it is vulnerable to *reward hacking*, where performance degrades as $N$ increases due to the selection of responses that exploit imperfections in the reward model instead of genuinely improving generation quality. Prior attempts to mitigate reward hacking---via stronger reward models or heavy-handed distributional regularization---either fail to fully address over-optimization or are too conservative to exploit additional compute.  In this work, we explore the principle of *pessimism* in reinforcement learning (RL), which uses lower confidence bounds on value estimates to avoid out-of-distribution (OOD) actions with uncertain reward estimates. Our approach, termed as *caution*, can be seen as the *reverse* of *curiosity*: where curiosity (e.g., via Random Network Distillation, RND) rewards prediction error as a signal of novelty, caution penalizes prediction error as a signal of distributional uncertainty. Practically, caution trains an error model on typical responses and uses its prediction error to lower reward estimates for atypical ones. Our extensive empirical evaluation demonstrates that caution is a simple, computationally efficient approach that substantially mitigates reward hacking in BoN sampling.  We also provide a theoretical analysis in a simplified linear setting, which shows that caution provably improves over the standard BoN approach.  Together, our results not only establish caution as a practical solution to reward hacking, but also provide evidence that curiosity-based approaches can be a general OOD detection technique in LLM settings.

Published as a conference paper at ICLR 2026
FROM CURIOSITY TO CAUTION: MITIGATING REWARD HACKING FOR BEST-OF-N WITH PESSIMISM
Zhuohao Yu', Zhiwei Steven Wu', Adam Block²
'Carnegie Mellon University "Columbia University
yu. ww.zhuohao@icloud wow.. com, zstevenwu@cmu. edu, adam. block@co lumbia. edu
ABSTRACT
Inference-time compute scaling has emerged as a powerful paradigm for improv- ing language model performance on a wide range of tasks, but the question of how best use the additional compute remains open. A popular approach is Best-of-N (BoN) sampling, where N candidate responses are generated, scored according to a reward model, and the highest-scoring response is selected. While this approach can improve performance, it is vulnerable reward hacking, where performance degrades as N increases due to the selection of responses that ex- ploit imperfections in the reward model instead of genuinely improving generation quality. Prior attempts to mitigate reward hacking- stronger reward models heavy-handed distributional regularization- either fail fully address over- optimization or are too conservative to exploit additional compute. In this work, we explore the principle of pessimism reinforcement learning (RL), which uses lower confidence bounds on value estimates to avoid out-of -iit----dittitttinn (OOD) actions with uncertain reward estimates. Our approach, termed as caution, can be seen as the reverse of curiosity: where curiosity (e.g., via Random Network Distillation, RND) rewards prediction error as signal of novelty, caution penal- izes prediction eπrоr signal of distributional uncertainty. Practically, caution trains an error model on typical responses and uses its prediction error to lower re- ward estimates for atypical ones. Our extensive empirical evaluation demonstrates that caution is a simple, computationally efficient approach that substantially mit- igates reward hacking in BoN sampling. We also provide theoretical analysis in a simplified linear setting, which shows that caution provably improves over the standard BoN approach. Together, our results not only establish caution as prac- tical solution to reward hacking, but also provide evidence that curiosity- based approaches can be a general OOD detection technique in LLM settings.
INTRODUCTION
Inference-time scaling has emerged as a transformative paradigm for enhancing language model performance, enabling significant improvements across wide range of reasoning tasks without in- creasing model size (Brown et al., 2024; Guo et al., 2025; Jaech et al., 2024). This success motivates the question of how best leverage additional inference-time compute maximize performance. A particularly popular and effective approach is Best-of-N (BoN) sampling (Stiennon et al., 2020; Nakano al., 2021; Wang al., 2022: Li et al., 2022: Huang al., 2025a), where multiple can- didate responses are generated for a given prompt, scored according to a reward model A. and the gighest-scoring response is selected. This approach capitalizes on the intuition that generating more candidates should increase the probability of finding higher-quality solutions, allowing the model to effectively 'explore" a larger portion of the response space than it could with only single response.
While BoN is simple and competitive baseline that is capable of astonishing gains in many settings (Brown et al., 2024), its success is fundamentally limited by the quality of the reward model used to score and select responses. Indeed, a common phenomenon often occurs with BoN, where performance initially improves as N increases, but then hits an inflection point after which larger N lead to increasingly worse outcomes (Gao et al., 2023; Huang et al., 2025b; Khalaf et al., 2025); an example of this phenomenon can be seen in Figure 1, where we plot the performance of BoN on the mathematical reasoning task GSM8K for different N' scored by several different reward models.

---

Published as a conference paper at ICLR 2026
85.0
80.0
Avg. Accuracy %
77.5
75.0
72.5
70.0
67.5
Owen3 Reward Model
Reward Model
LLMM33.2 Reward Model
OASST Reward Model Pessi mism (Ours)
65.0
2-0
2-1
2n6
2
N samples per problem
Figure 1: Average Accuracy with different sampling budgets for Best-of-N on the GSM8k dataset. We see that standard Best-of-N sampling (blue, red, and gold) suffers from reward hacking, exhibiting the characteristic rise-and-fall pattern N increases. In contrast, caution (our approach, green) consistently improves with larger N, effectively mitigating reward hacking.
This counterintuitive phenomenon, whereby increasing leads worse performance, occurs due to reward hacking, the process by which BoN selects responses that exploit f opposed to the ground truth reward y* we actually care about; while & may be reasonable approximation to r* on 'typical" generations, as N increases, BoN selects more atypical responses lying outside the training distribution of f that achieve high f scores through spurious correlations rather than genuine quality improvements. In other words, reward hacking is manifestation of Goodhart's Law: whenever metric (in this case f) becomes an optimization target, it ceases to be a reliable measure of quality (Goodhart, 1984; Weng, 2024); for example, reward models known favor certain formatting preferences and surface-leved patterns learned during training that do not necessarily correspond to improved reasoning or correctness (Liu et al., 2024b; Yu et al., 2025; Bukharin et al.. 2025).
With the intuition that reward hacking is caused by out-of-distribution (OOD) generations from the perspective of the reward model, several works have proposed mitigation attempts by either improving the reward model (Liu et al., 2025; Yu et al., 2025) or by regularizing the selection pro- cess to favor in-distribution responses (Huang et al., 2025b). Unfortunately, the former approach fundamentally limited by the asymmetric difficulty of the problem: while it is relatively straight- forward to obtain representative examples of high-quality responses through careful curation and human annotation, exhaustively characterizing all possible reward hacking strategies is intractable. Representative of the latter work Huang et al. (2025b), who propose to regularize the selection process ensure that the distribution of selected responses does not drift too far from the distribu- tion of responses seen during reward model training in a strong information theoretic sense. The authors provide strong theoretical guarantees on the efficacy, monotonicity, and optimality of their approach, and demonstrate that it can be efficiently implemented in practice through a simple rejec- tion sampling scheme (Block & Polyanskiy, 2023). However, this approach overly conservative in practice, preventing the selection of genuinely better responses that are slightly out-of-distribution and thus fails to fully leverage the benefits of inference- time scaling. The problem arises because the regularization is distributional and thus simultaneously ensures all possible OOD responses are penalized equally, regardless of whether or not they are likely to arise from imperfections in é. The starting point for this paper is thus to ask: Can design BoN sampling scheme that is both robust to reward hacking and still able to leverage the full benefits of inference-time scaling?
Contributions. We answer this question in the affirmative by introducing caution, inference-time instantiation of the pessimism principle from Reinforcement Learning (RL) (Jin et al., 2021; Guo et al., 2022). Pessimism relies lower confidence bounds to avoid selecting OOD actions with uncertain rewards. While Huang et al. (2025b) implements pessimism at the distribution level by constraining the sampling distribution to remain close the base policy, instead apply it the
2

---

Published as a conference paper at ICLR 2026
Sorted by rlag)
lo anderaad how a00
Problem Prompt write 3-page different friends rotcr week paga dues he wrise year?
Rollout
Brased Made Co
x xx([xx)
22:y
Hiaved, earily hached
Sare, to help
The antwrr im
Best-of-N Sampling with Reward Model
Training
2 Inference
Targer Network Toxy) RM Wingin
RM
+1
Promues
Rollout Once
argming Ty(8.399)
(x, y)
#2:
A small subset
Preficur Nerwoek 1e6, St Randon Loiddedd
rybo, yol)
*3
Sorted by r(s.y)- nes..y
Pessimistic Penalty
(p(2,3n00 P(z, 36 0) T(a 2lll
Figure 2:
Overview. Predictor is trained to match RM features on typical responses; at inference, select the candidate with the highest pessimistic reward, down-weighting OOD ones.
reward level: we penalize OOD responses by subtracting per-response uncertainty estimates from the scores assigned by A, and then select the response with the highest pessimistic score.
Conceptually, caution is the dual of curiosity. a principle used to drive optimistic exploration in deep RL (Pathak et al., 2017: Burda et al., 2018). As in curiosity, measure uncertainty by fixing sim- ple learning target (e.g., a frozen feature embedding). training a student model to predict this target in-distribution data, and using the prediction ever, our setting is fully offline, so no continual student training is required-maker nificantly simpler and more practical. Through extensive empirical evaluation, show that caution is computationally efficient and effectively mitigates reward hacking in BoN sampling. We further provide a theoretical analysis in simplified linear setting, proving that caution strictly improves over standard BoN. Taken together, these results demonstrate that caution is both powerful prac- tical solution to reward hacking and compelling evidence for the broader efficacy of curiosity-style uncertainty signals in OOD detection and pessimistic policy learning for language models.
uncertainty signal. Unlike curiosity, how-
the method sig-
METHODOLOGY
2.1 PROBLEM FORMULATION
Consider trained language model (LM) n, where X + A()) is map from prompts X' to distributions over responses y E y. While the LM is typically an autoregressive model, generating one token at time before feeding the generated tokens back in as context, abstract this process away and instead consider prompts and responses as single entities, possibly consisting of the concatenation of many individual tokens. Suppose we have access to a learned reward model A X' y + IR that assigns score to pronpt-response pair (x, y) intended to reflect the quality of the response y to the prompt z. While the learner has access to f, true goal is to output a response that approximately maximizes some ground-truth reward r" X' x ) + R, which is unknown to the learner. We focus on reward-hacking behavior that seeks to maximize at the expense r*
Given a prompt x, the learner samples N candidate responses .I,..., VN T(-[æ) independently from the LM and then selects one of these responses y; with i E [N] to output. Perhaps the simplest strategy is the Best-of-N (BoN) strategy, which simply selects the response with the highest reward model score: i = yx = argmax;giNi f(z, yt). Unfortunately, this strategy vulnerable to reward hacking when we only have i r" on typical samples from æ. As N grows large, the distribution of i can differ substantially from that of typical samples from म, leading to poor performance with respect to r* as shown in (Huang et al., 2025b). While that work proposes address this issue by constraining the BoN sampling distribution क to be close to म, we instead propose to directly apply regularization to the reward estimates themselves. Thus, we will design uncertainty estimate a : X' x y + Rgo that quantifies how uncertain we are about the reward model's estimate f(x, y) for given prompt-response pair (x, y) and then define ÅL.B (x. f(a, v) Aa(æ, y) for some
3

---

Published as a conference paper at ICLR 2026
GSMBK (Trained)
MATH-500
BigBench-Hard
25
82.5
14-
20
30.0
15
57.5
10
275.0
472.
IN DOMAIN
8
OUT OF DISTRIBUTION
5-
OUT OF DO
70.0
2
2
2
2
2
24
2
2
2
2 24
2a 24 2
2
2
2
2
2
2
2
2
samples per problem Passinism
Model Pessiiim
Figure 3: Scaling over N across distributions and domains. Best-of-N sampling on GSM8K, MATH-500, and BigBench-Hard. Curves compare selection by Reward Model, Pessimism, and Reward Model + Pessimism. Note that the pessimism function is trained only on GSM8K; thus, MATH-500 represents an out-of-distribution setting, while BigBench-Hare represents a fully out- of-domain setting.
to be a pessimistic variant of the reward model that penalizes uncertain responses (Jin et al., 2021). As long as a(x. v) is small for typical samples from <--1x; and large for 'atypical' samples, this approach will successfully penalize reward-haad hacking responses by ensuring that FLCB S r* and this inequality is approximately tight for actually good quality responses. The final algorithm will thus be return = argmax;e [N) FLCB (2, y:) and the question becomes what choice of uncertainty estimate a appropriately captures OOD responses in a reward-awan manner?
2.2 CURIOSITY TO CAUTION: INSTANTIATING PESSIMISM
Our approach introduces the principle of caution, a phenomenon dual to the well-known technique of curiosity used in online RL to incentivize exploration (Pathak et al., 2017; Burda et al., 2018). While online RL, OOD states are desirable (as they represent good exploration targets), in our offline setting, OOD responses are to be avoided, as we have no reliable way to estimate their true reward given that f RS r* only on typical responses from æ. We draw inspiration from curiosity (Pathak et al., 2017) () and Random Network Distillation (RND) (Burda et al., 2018 8) in order to ensure that selected responses remain close to the distribution π on which is reliable.
In curiosity and RND, the core idea is to continually train predictor network to match the outputs of a fixed target that is easily evaluated by the learner; the supervised learning error then becomes proxy for OOD detection, with the intuition being that the predictor will be accurate states similar to those seen during training and inaccurate otherwise. While empirically successful as an 'intrinsic reward' for optimistic exploration, this method be challenging implement due the continued training of the predictor during online RL, which can lead to nontrivial memory and time overheads practical RL pipelines. Our method uses the same core idea of using the supervised learning error as a proxy for OOD detection, but is trained fully offline, which greatly increases practicality and ease of implementation.
Implementing "Caution" We employ two neural networks operating on the internal representa- tions of the reward model R(x, y): (1) a fixed larget network T T(x,, y) h (ar, y). defined as the hidden state extracted from layer L of the frozen reward model, and (2) a trainable predictor network Po(a, y) that learns to predict this hidden state. The first L layers of the reward model serve as our target feature extractor, with their output hff (x, y) the prediction target. The predictor network Po(a, y) is a separate trainable network with parameters 0 that can use various architectural choices (shared embeddings, simplified encoders, projection layers) but is designed be lightweight to ensure efficient training and inference.
Training Process. We train our predictor Po on dataset D Durin of prompt-response pairs constructed directly from the benchmark train split using mean squared error loss: L(A) E(a.y))Don (o), u) - T(r, ]) where the norm is Euclidean. We construct responses ~ (-12z) by sampling from æ as an efficient empirical proxy for the training distribution of the re- ward model. By training this supervision-free distribution, the predictor achieves low error
4

---

Published as a conference paper at ICLR 2026
Table 1: Scaling Performance Across Datasets with Reward Hacking Mitigation. Peak Acc: highest accuracy achieved across all N values. Final Ace: accuracy at N = 512 where reward hacking most severe. Degradation: performance drop from peak final (lower is better). Results show mean with 95% confidence intervals as subscripts across 3 bootstrap runs. Best results are in bold and second-ber best are underlined.
GSM8K (2021)
MATH-500 (2023)
BBH-Hard (2022)
Method
Peak
Final
Degr.
Peak
Final
Degr.
Peak
Final
Degr.
Reward Model
19.
71.5
7.7
11.540.3
8,5
3.0
17.3
1.7g0.1
15.6
Pessimism Only
81.3
80.3
1.0
13.640.7
11.9
1.7
22.9
22.1
0.9
RM Pessimism
82.6
81.1
1.5
12.3+1.0
10.1
2.3
18.5
11.0
7.5
in-distribution patterns while preserving high error distributional outliers. We emphasize that the data requirements for this step are restricted to prompts, which are often much cheaper to collect than the high-quality labelled data used train reward models; a gold standard would be reuse the same prompts used to train the reward model itself, but in practice we find that our method's OOD detection is relatively robust across different prompt distributions (ef. Section 3.1).
Pessimistic Reward Estimation at Inference Time. Once we have trained our predictor Po. we can use it to define our uncertainty estimate a(x, y) = (z, (.,z).))) y) - (ax y) the prediction error of the predictor on the target network's features. We then plug this into pessimistic reward f(x, A ||Po y) T(x, y)ll". Note that this score is very easy to compute at inference time, requiring only two forward passes (one through Po and one through 7) in addition to the forward pass through the reward model compute i;; note that all of these passes can be fully parallelized and the cost of evaluating a(x, y) is on the same order as that of evaluating f due the reuse of features from The parameter controls the strength of the pessimism penalty, with À = 0 recovering standard BoN sampling. The final selection becomes = argmax,e[N] ÅLCB (2, Vs). i.e., choosing the response with the highest pessimistic reward estimate. This procedure summarized Figure 2.
estimate (LCB(E, y) f(z:, y) Aa(x.
2.3 THEORETICAL INTERPRETATION
In order to provide further motivation for our approach, we analyze in Section C simple theoret- ical setting in which our approach provably improves upon BoN. While we defer the details to the appendix, summarize the main theorem here:
Theorem (Informal version of Theorem 3). Let y1,..., UN E Rd be i.i.d. samples from model t and r"(y) be a linear reward function. Let i* argmax r" (ys) ) be the optimal response and let argmax,e(N F(y)) be the response selected by BoN using a learned reward model î. "pess (e[mmax;[[N] (LCB(y:), where FLCB is caution-regularzees reward estimate, then under suitable conditions the target network, r², f, n, and the predictor network, it holds that
EE [r
(Jipos)) 0.
(Viiper)))) r" (v:) Z Vlo((N),
and
lim No00
E (y++))
While the assumptions and conditions of Theorem 1 are necessarily somewhat strong in order to facilitate the analysis, the theorem provides a proof-of-concepl that our approach can provably out- perform BoN in stylized setting as well as, to the best of our knowledge, the first theoretical guarantee on the success of curiosity- and RND-style methods for OOD detection.
3 EXPERIMENTS
We now empirically evaluate our proposed approach, with a focus on answering the following three questions: (1) How well does caution mitigate reward hacking the number of candidates N in- creases? (2) What design decisions (e.g. architecture and training hyperparameters) contribute most to the overall success of method? (3) In which scenarios does method outperform standard
5

---

Published as a conference paper at ICLR 2026
Table 2: Weight Ablation and Design Comparison on GSM8K. We compare different mixing weights between uncertainty score and f score, contrasting our approach (distilling reward model features) against traditional RND (distilling random network features). Peak Ace: best accuracy across all N. Final Ace: accuracy at N 512. Results demonstrate that distilling reward model representations substantially outperforms random network distillation across weight combinations.
Caution (Ours)
Traditional RND
Pessimism strength (A)
Peak Ace
Final Acc
Peak Acc
Final Acc
0.0 (RM Only)
78.9
71.2
78.9
71.2
0.2
79.5
72.5
78.7
71.3
0.4
80.3
76.0
78.5
72.1
0.6
81.5
80.2
78.4
73.1
0.8
82.1
81.0
78.1
74.9
1.0 (Caution Only)
81.3
79.8
77.0
72.9
BoN sampling, and what factors drive its success? We briefly describe our empirical setup (with full details deferred to Section D) before addressing each of these questions in turn.
Experimental Setup. We use Llama-3. m.mna-3.2.. 3.2-38 3-3-IIIsructtt (Dubey et al., 2024) as our language model π due to its strong reasoning capabilities and open-source availability.' We consider three prompt distributions coming from reasoning datasets of varying difficulty: GSMBK (Cobbe al., 2021), MATH-500 (Hendrycks et al., 2021), and BigBench-Hare Our ground truth reward r" is binary, with reward given if and only if response provides the correct answer to the given prompt. While we consider several reward models A, we focus primarily on OASST (Köpf et al., 2023), which Figure demonstrates provides better perfor- mance than comparably sized reward models like Skywork weework--wwwreeee rd- V2 1122-wen3--e 0. 6B and Skywork -ewardd- -aaeaardd------ee V2-LLama 3. 2- 1B (Liu et al., 2025), making it strong baseline for demon- strating reward hacking phenomena. For each prompt and N, we use vLLM (Kwon et al., 2023) to generate N independent responses from the base model (and boostrap this process 3 times to generate confidence intervals), before scoring each response with & and measuring the performance of the selected response according to r". To train our predictor network Pe. we use an independent dataset of responses generated by the base model т itself on the training split of GSMBK, and evaluate the performance of the uncertainty estimates both in-distribution (prompts from the test set of GSM8K) and out-of- -fi-tii-iiiitttion (prompts from other reasoning datasets). In order to maintain consistency, we normalize all rewards to be centred with unit variance using an independent set of responses and we do the same for the uncertainty estimates from Po.
(Suzgun al., 2022).
3.1 CAUTION MITIGATES REWARD HACKING
Our main results demonstrate that our proposed mechanism of caution mitigates reward-hacking and leads to improved performance of BoN sampling as increases both for in- and out- of-distribution prompts. In Figure 1, we exhibit the performance of BoN sampling on GSM8K for several choices of reward models and observe that they all exhibit reward hacking, with performance degrading for large N. We also see that using our pessimistic approach. this reward- -aakkig is substantially mitigated, with performance improving monotonically with N for all considered N. Moreover, we get substantial performance boost of 4.2% over peak accuracy for the reward model alone and an astonishing 15.5% boost over the final accuracy of the reward model alone. This trend compares favorably with that of Huang et al. (2025b), who observe monotonicity of performance in N, but struggle to outperform BoN sampling for optimally tuned N for most choices of æ and f.
The above results involve evaluating caution on the same distribution of prompts used to train the predictor network Pg. which is likely beneficial to its performance. In reality, we expect to use caution in scenarios where the prompts are potentially OOD and hope that the uncertainty estimates remain valid. To evaluate the extent to which this holds, we used the same uncertainty estimates to
'ooe other ween-weight models are thought to be contaminated with henchmark datasets ( (Wu et al., 2025), which could skew our results, further motivating our choice in model.

---

Published as a conference paper at ICLR 2026
Table 3: Ablation study of predictor architecture. Training Loss measures how well the predictor fits reward model representations. Peak Acc shows best performance across all N. Final Acc shows performance largest N, where reward hacking is most severe.
Predictor Configuration
Reconstruction Loss
Peak Ace (%)
Final Ace ( (%)
Simplified Architecture Variants Lightweight + Trainable Emb.
0.242
82.2
82.2
Lightweight Frozen Emb.
0.246
81.5
81.3
Lightweight + Separate Emb.
0.255
82.7
81.8
Lightweight w/o Projection
0.281
81.8
81.3
Full Architecture Variants
Full + Trainable Emb.
0.127
80.3
80.2
Full Frozen Emb.
0.127
80.4
78.4
produce pessimistic BoN sampling two significantly harder reasoning datasets: MATH-500 and HaaBench-Hard These datasets comprise prompts that are significantly different from those in GSMBK, with the latter even coming from different, non-mathematical domain.
We report the results of these OOD prompt experiments in Figure 3 and Table 1, comparing the performance of the in-distribution prompts (left) with the two other tasks (centre and right). As in Figure I, we sweep over logarithmic grid N from 1 to 512 and compare three approaches: BoN using only A, BoN using only the uncertainty estimates from Pe. and BoN using caution. As in Figure I, we see that caution successfully mitigates reward hacking while preserving the benefits of reward-guided selection. The results MATH 500 reveal different pattern: pessimism-onfe out- performs the combined approach, achieving 13.6% peak accuracy with only 1.7 points degradation compared to 2.3 points for the combined method. This counterintuitive finding reflects the increased problem tifficulty-whll MATH-500 shares the mathematical reasoning domain with GSMBK, the problems are significantly more complex and multi-step, making the task of more difficult. this regime, i becomes less reliable at distinguishing genuine quality improvements, making its contribu- tion less beneficial. However, caution still effectively prevents the severe degradation seen with BoN (3.0 points). The most challenging task we consider is BigBench HagBench-Hard, where the reward model A fails catastrophically. Indeed, we observe the counterintuitive phenomenon that using the uncer- tainty penalty alone is the most performant of all three methods, doing even better than combining uncertainty with f. This surprising result due to the fact that when reward models encounter prob- lems substantially harder than their training distribution, their learned spurious correlations dominate their judgments, leading to systematic selection of responses that superficially mimic quality patterns without genuine correctness. Caution, while also operating outside its training distribution, avoids being misled by these hacking features and maintains stable performance by preferring responses that match familiar distributional patterns rather than chasing unreliable reward signals.
3.2 ABLATION STUDIES
In this section, we dissect the design and implementation of our proposed caution mechanism to understand which components contribute most to its success. Further ablation studies, including de- tailed inspections of uncertainty scores for individual responses and the effect they have on and r* scores be found in Section F, while additional details the ablations can be found in Section D.
We evaluate three critical design decisions: (1) the precise architectural choice of the predictor network; (2) the use of pre-trained embeddings in the target network; (3) the use of a projector layer between the predictor and target networks. We also conduct two additional ablations to validate our core design choices: (a) the choice of mixing weights between and the uncertainty estimates; (b) the choice of distilling reward model features versus random network features.
Architectural Ablations. We first examine the extent to which the complexity of Pe and its re- lationship to the target network T affects performance, in particular whether Po needs to share an architecture with the teacher T'. The results are shown in Table 3: while greater flexibility in Po leads to a better reconstruction loss, this does not transfer to better performance when used to instantiate caution. This occurs because improving the reconstruction loss actually hurt the ability of P
7

---

Published as a conference paper at ICLR 2026
Systcm Procupe Solve the Collowng problem stop by step. Gire your Ginal nusoccical susvet at the cnd wiihe #hEe [NUM] Quesation: Rastmond does hall much luunlly Sanah, and Sandl does 4 times much looddry 1aeid caleskree the difference the or landry Rlymndd and Disid do
Question: petslopp had cages rodents of the erges have 10 hemsers and the other guines pige many
Sanah dors pounds lundey.
pershop
Responwe: Let' beeak down the information giem: Samah does 4 times much leandry David Sand 400 poande of luniey
Response: To find the soad number of rodenss, noed to find gaines plas The namiee hamniers is
euprrsen Darke's laundry thas Swalts laundry
aumbser The 13, we number of hameters and grinaa rige to the tora number of radime Thei modenes 30 15 Toral midkene - 45 natt 45
sumber
400 4 much laundry Sandh we repeesent Reymondss landdy y. then y (1/2s. ym (1/2) to0 y
pounds lunddrs Reytond sobre fir
up oqution
Su, Dovid duos
number gilses pigs
Raymond doer 50 pornds hondry 4 The difieencc in amoure of kundey Raymond and David do Difference- Raemonif's luundry Dasid'' laundry = 50 100 -50 Since werre looking for the difference, we take the abookute valae of te afference the
hunday Raymond DavkE
pounds.
Answer: X Reward 0.15 (91 / 1000), Reward Pessimism: 0.43 (19996 / 1009)
Auswen:
Rewaedi 0.31 (1976/10000) Rewaed Pessimism: 1.72 (6) 1000)
Figure 4: Contrasting Selection Behaviors: Reward Hacking vs. Format Compliance. Two rep- resentative examples showing how reward models favor verbose responses regardless of correctness, while our curiosity-driven pessimism prioritizes format compliance and distributional familiarity. RM assigns high scores to detailed responses regardless of correctness, while pessimism detects distributional deviation from training patterns and prefers correctly formatted solutions.
to detect distributional novelty, as it becomes too good at reconstructing even features. We observe a similar trend when considering whether or not to tune the embeddings or share them between Po and T:: while sharing embeddings leads to better reconstruction, it can hurt the final performance, which suggests that forcing predictors to learn representations from scratch creates more sensitive distributional boundaries than inheriting potentially overly simplified features from pre-trained models. Finally, we see that adding projection layer between the predictor and target networks provides consistent but modest benefits, indicating that information bottlenecks can help prevent pure memorization while preserving signal quality. Together, these findings establish a key empirical finding for instantiating caution: effective detection requires a careful balance between reconstruction accuracy and novelty sensitivity.
Strength of regularization with Caution. To validate our design choices, systematically vary the mixing weights between the uncertainty score and that of f to identify the optimal balance for reward hacking mitigation and understand the robustness of our results this choice. Note that due to the normalization of f and uncertainty estimates described above, the strength À can be viewed as a direct measurement of the influence of the pessimism relative to f. We report our findings in Table 2 and Appendix F. We observe that our approach is relatively robust to the choice of X, with moderate to high weights (0.6-0.8) achieving optimal performance. While the precise optimal weight will vary by task and choice of r. with larger À being required in situations where i is less reliable, the results suggest that our approach does not require delicate tuning to be effective.
Comparing Caution to RND. Finally, we compare our approach of distilling reward model fea- tures (motivated by curiosity in Pathak al. (2017)) against the RND (Burda et al., 2018), which takes the teacher 7' to be a randomly initialized network (possibly on top of pre-trained embeddings), testing whether our hypothesis about distributional familiarity requires semantic grounding in the re- ward model's 's representations. We again consider a range of mixing weights À and report the results in Table 2. The results reveal stark contrast between caution and RND, with the latter exhibiting dramatically inferior performance across all choices of X. This observation provides evidence for the hypothesis that effective distributional regularization requires semantic grounding: randomly initialized features cannot provide meaningful distributional boundaries, while reward model fea- tures capture task-relevant patterns that enable robust novelty detection. The results establish that curiosity- driven pessimism succeeds not merely because of prediction error signals, but specifically because these signals are computed relative to the reward model's 's learned task representations.
3.3 WHY IT WORKS: A CASE STUDY
We now turn to case study to illustrate the mechanism by which caution mitigates reward hack- ing and improves performance. In Figure 4, we present two representative examples of questions- response pairs. On the left, we an instance that f scores highly (99.8th percentile of all scored responses) despite being incorrect; this response is verbose and contains multiple mathematical rea- soning steps, but ultimately fails to provide the correct answer and does not follow the required
8

---

Published as a conference paper at ICLR 2026
formatting, instead demonstrating exactly the type of reward hacking method targets: super- ficial mimicry of quality patterns (detailed explanations, step-by-step structure) without genuine correctness adherence to specified requirements. Note that the caution score for this response appropriately identifies it as OOD, likely due to its failure to follow the formatting requirements that common in high-quality responses in the training data.
By contrast, on the right of Figure 4, we see a concise, accurate response following the formatting requirements. While does not rank this response well as the first, the pessimism recognizes it as familiar, matching the patterns of high-quality responses in the training data that consistently follow formatting specifications and provide direct, correct solutions. This juxtaposition reveals that reward models can conflate verbosity with quality, particularly at larger N where more elaborate responses become available. Our method effectively distinguishes between responses that appear sophisticated (verbose explanations) and those that are actually correct and compliant with task specifications.
4 DISCUSSION
In this work, we investigated instantiating pessimistic reward estimation with the principle of cau- tion, an approach adapted from curiosity-driven exploration in RL that uses a supervised learning error as measure of distributional uncertainty and penalizes the estimated reward of uncertain, out-of-distribution (OOD) inputs. While our focus was on mitigating reward hacking in Best-of-N sampling, particular inference-time scaling technique, our results demonstrate that, when properly applied on top of pre-trained features, caution can effectively detect OOD samples and instantiate pessimistic policies in general. While we defer more complete survey of related work to Section B, we now provide a brief summary thereof as well as discuss future directions.
Related Work. Best-of-N sampling was introduced in Stiennon al. (2020) and has been empir- ically investigated in many reasoning settings (Cobbe et al., 2021; Lightman et al., 2023; Li et al., 2022; Brown et al., 2024). While often effective, when the scoring rule is learned (as opposed to oracular), the approach has long been shown to be vulnerable to swward-hacking (Amodei et al., 2016; Skalse et al., 2022: Gao et al., 2023). While many attempted mitigations of reward- haakkng have been explored for RL finetuning, relatively few works have focused on the inference-ting time set- ring. Of particular note is Huang et al. (2025b), which proposes a distributional regularization approach that samples according motivated and empirically effective at ensuring monotonicity in N, the approach is overly conser- vative in practice. Another approach is that of Jinnai et al. (2024), who apply distributional regular- ization with respect to a Wasserstein distance on some embedding space; while potentially effective, the computation required grows quadratically in N, making it impractical for even moderate values of N. In contradistinction to these works, our approach applies pessimism directly to the estimated rewards in way that naturally leverages the beyond- worst-case errors present in estimated reward models in way that is impossible for these distributional regularization approaches.
x² regulaarid BoN procedure. While theoretically well-
heyond-worst-case
Our key technique of caution is built on top of the foundational curiosity (Pathak et al., 2017) and Random Network Distillation (RND) (Burda et al., 2018) from classical deep RL. While the such techniques have been very popular in aiding exploration, the extent to which they can be used for OOD detection and pessimistic learning has been a matter of some debate, with Rezacifar et al. (2022) claiming negative results and Ciosek et al. (2019); Nikulin et al. (2023) demonstrating some positive results. While these works are (i) in classical RL supervised learning settings and (ii) do not use pre-trained features, our results morally align with the latter camp, demonstrating that such approaches can be effective for OOD detection and pessimistic learning in language models.
Future Directions. Our work provides strong evidence that caution, when correctly applied on top of pre-trained features, can be an effective detector of OOD text. While we instantiate this approach for pessimistic reward estimation, it is natural to wonder if curiosity can be used as an explicit reward signal to encourage exploration of novel behaviors either purely during inference or during RL post- training of reasoning models. While some preliminary work like Gao et al. (2025) has explored this idea, we believe that their mixed results stem from the fact that the curiosity module was trained from scratch rather than top of pre-trained features, which we find to be so effective for OOD detection. Another interesting direction is to explore the extent to which our proposed approach can help ensure continued AI alignment in the face of adversarial prompting distributional shift
9

---

Published as a conference paper at ICLR 2026
due inference-time scaling. While the tasks consider in this work related to reasoning, we expect that our results would carry over to safety and aligmment tasks mutatis mutandis, which would represent promising approach to ensuring robust alignment of language models.
ACKNOWLEDGMENT
Both AB and ZSW were supported part by research grant from Schmidt Sciences.
ETHICS STATEMENT
We affirm adherence to the ICLR Code of Ethics. This work only involve publicly available bench- mark datasets under their respective licenses; we do not collect new human-subject data and process no personally identifiable information.
REPRODUCIBILITY STATEMENT
To ensure reproducibility and transparency of the results within this paper, document relevant hyperparameters and implementation details in the appendix. We will open-source our codebase along with detailed instructions and scripts.
REFERENCES
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 techni- report. arXiv preprint arXiv:4412.08055, 2024.
Afra Amini, Tim Vieira, Elliott Ash, and Ryan Cotterell. Variational best-of-n alignment. arXiv preprint arXiv:2407. 06055, 2024.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in ai safety. arXiv preprint arXiv: 160iv/6606.000 1606.0666, 2016.
Yoshua Bengio and Yann LeCun. Scaling learning algorithms towards AI. In Large Scale Kernel Machines. MIT Press, 2007.
Adam Block and Yury Polyanskiy. The sample complexity of approximate rejection sampling with applications smoothed online learning. In The Thirty Sixth Annual Conference on Learning Theory, pp. 228-273. PMLR, 2023.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprini arXiv:2407.217877 2024.
Alexander Bukharin, Haifeng Qian, Shengyang Sun, Adithya Renduchintala, Soumye Singhal, Zhilin Wang, Oleksii Kuchaiev, Olivier Delalleau, and Tuo Zhao. Adversarial training of reward models. arXiv preprint arXiv:2504.06144, 2025.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint
12894 2018.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.152111 152 2023.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- fems, 30, 2017.
10

---

Published as a conference paper at ICLR 2026
Kamil Ciosek. Vincent Fortuin, Ryota Tomioka, Katja Hofmann, and Richard Tumnr. Conserva- tive uncertainty estimation by fitting prior networks. In International Conference on Learning Representations, 2019.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichrr Nakano, et al. Training verifiers to solve math word problems. arXiv preprint wxXv:2110.14110 2021.
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv:2310.02743, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D' Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herd- ing? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.09244 2023.
Jingtong Gao, Ling Pan, Yejing Wang, Rui Zhong, Chi Lu, Qingpeng Cai, Peng Jiang, and Xi- angyu Zhao. Navigate the unknown: Enhancing IIm reasoning with intrinsic motivation guided exploration. arXiv preprint 2505::2500..7221, 2025.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Charles Goodhart. Monetary theory and practice: the uk experience. 1984.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501.12948. 2025.
Kaiyang Guo, Shao Yunfeng, and Yanhui Geng. Model-based offline reinforcement learning with pessinisn-modulated dynamics belief. Advances in Neural Information Processing Systems, 35: 449 2022.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 03874, 2021.
Geoffrey F. Hinton, Simon Osindero, and Yee Whye Teh. fast learning algorithm for deep belief nets. Neural Computation, 18:1527-1554, 2006.
Sam Houliston, Alizée Pace, Alexander Immer, and Gunnar Rätsch. preference optimization. arXiv preprint arXiv:4410.20187 2024.
direct
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. Advances in neural information processing sys- tems, 29, 2016.
Audrey Huang, Adam Block, Dylan J Foster, Dhruv Rohatgi, Cyril Zhang, Max Simchowitz, Jor- dan T Ash, and Akshay Krishnamurthy. Self-improvement in language models: The sharpening mechanism. The Thirteenth International Conference on Learning Representations, 2025a.
Audrey Huang, Adam Block, Qinghua Liu, Nan Jiang, Akshay Krishnamurthy, and Dylan J Foster. Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment. arXiv preprint ar arxiv:2503.2 20..5503.2877.. 1878, 2025b.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang. Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 12186, 2024.
11

---

Published as a conference paper at ICLR 2026
Yuki Ichihara, Yuu Jinnai, Tetsuro Morimura, Kaito Ariu, Kenshi Abe, Mitsuki Sakamoto, and Eiji Uchibe. Evaluation best-of-n sampling strategies for language model alignment. arXiv preprint arXiv:2502. 126668 2025.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemel: Convergence and gen- eralization in neural networks. Advances in neural information processing systems, 31, 2018.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint wrXiv:2412.167200 2024.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline ri? In International conference on machine learning, pp. 5084-5096. PMLR, 2021.
Yuu Jinnai, Tetsuro Morimura, Kaito Ariu, and Kenshi Abe. Regularized best-of-n sampling with minimum bayes risk objective for language model alignment. arXiv preprint arXiv:2404.0105: 2024.
Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback. 2024.
Hadi Khalaf, Claudio Mayrink Verdun, Alex Oesterling, Himabindu Lakkaraju, and Flavio du Pin Calmon. Inference-tinne reward hacking in large language models. arXiv preprint arXiv:2506. 192488 2025.
Andreas Köpf, Yannic Kilcher, Dimitri Von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen, Oliver Stanley, Richard Nagyfi, et al. Openassistant conversations demooraionns large language model alignment. Advances neural information processing systems, 36:47669-47681, 2023.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Cassidy Laidlaw, Shivam Singhal, and Anca Dragan. Correlated proxies: new definition and improved mitigation for reward hacking. arXiv preprint 2rXiv:2403.03183 2024.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.1378. 2024.
Jiong Li and Pratik Gajane. -uriosity-driven exploration in sparse-reward multi-agent reinforcement learning. arXiv preprint arXiv:2302. 10825, 2023.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, al. Competition-Ievel code generation with alphacode. Science, (7866624):0092-1097) 2022.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skwwork-reward: Bag of tricks for reward modeling in llms. arXiv preprint 2rX1v:2410.18451, 2024a.
Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, et al. Skywork-reward-- Scaling preference data curation via human-ai synergy. arXiv preprint arXiv:2507.01352 2025.
Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. Rm-bench: Benchmarking reward models language models with subtlety and style. arXiv preprint arXiv:2410.1618. 16184, 2024b.
12

---

Published as a conference paper at ICLR 2026
Odelia Melamed, Gilad Yehudai, and Gal Vardi. Adversarial examples exist in two-layer relu net- works for low dimensional linear subspaces. Advances in Neural Information Processing Systems, 36:5028-5044 2023.
Antonio Valerio Miceli-Baron Alexandra Birch, and Rico Sennrich. Distributionally robust recur- rent decoders with random network distillation. arXiv preprint arXiv:2110. 13229, 2021.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim. Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint
2021.
Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov. Anti-exploration by random network distillation. In International Conference on Machine Learning, pp. 26228- 26244. PMLR, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. arXiv preprint arXiv:2201.035: 0335444, 2022.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-2787. PMLR, 2017.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 11.5372--53744 2023.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits of weight averaged reward models. arXiv preprint 2rXiv:2401.12188 2024.
Patrik Reizinger and Márton Szemenyei. Attention-based curiosity- driven exploration in deep rein- forcement learning. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 3542-3546. IEFE, 2020.
Shideh Rezaeifar, Robert Dadashi, Nino Vieillard, Léonard Hussenot, Olivier Bachem, Olivier Pietquin, and Matthieu Geist. Offline reinforcement learning as anti-exploration. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8106-8114, 2022.
Wenlei Shi and Xing Jin. Heimdall: test-time scaling the generative verification. arXiv preprint orXiv:2504.1033: 2025.
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and character- izing reward gaming. Advances in Neural Information Processing Systems, 35:9460-9471, 2022.
Mei Song, Andrea Montanari, and P Nguyen. A mean field view of the landscape two-layers neural networks. Proceedings of the National Academy of Sciences, 115(33)E77665-7777) 2018.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 03:3008-3021, 2020.
Haoran Sun, Yekun Chai, Shuohuan Wang, Yu Sun, Hua Wu. and Haifeng Wang. Curiosity-driven reinforcement learning from human feedback. arXiv preprint arXiv:2501.1 /1463, 2025.
13

---

Published as a conference paper at ICLR 2026
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le. Ed H Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether chain-of-thought can solve them. arXiv preprint arXtv:2210.02661, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz. Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47, Cambridge university press, 2018.
Martin Wainwright. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cam- bridge university press, 2019.
Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of Iims for diverse user preferences: Directional preference alignment with multi-objective rewards. In ACL, 2024a.
Haoxiang Wang. Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of mixtuur-of-eexprts. In EMNLP, 2024b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le. Ed Chi, Sharan Narang, Aakanksha Chowdh- cry, and Denny Zhou, Self-consistency improves chain of thought reasoning in language models. arXiv preprint 2rXiv:2203.11177 2022.
Yidong Wang. Zhuohao Yu, Zhengran Zeng, Linyi Yang. Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. Pandalm: An automatic evaluation benchmark for 11m instruction tuming optimization. arXiv preprint arXiv:2306. 230000077700 2023.
Lilian Weng. Reward hacking in language models. lil'log. 2024. URL https://Lilianwett github. io/ 20224 06 19- reward- harrddh.
Yixuan Weng. Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with self-verification. arXiv preprint arXiv:2212.009560 2022.
Mingqi Wu, Zhihao Zhang, Qiaole Dong. Zhiheng Xi, Jun Zhao, Senjie Jin, Xiaoran Fan, Yuhao Zhou, Huijie Lv, Ming Zhang, et al. Reasoning or memorization? unreliable results of reinforce- ment learning due to data contamination. arXiv preprint
2025.
Yuzi Yan, Xingzhou Lou, Jialian Li, Yiping Zhang, Jian Xie, Chao Yu, Yu Wang, Dong Yan, and Yuan Shen. Reward-robust rihf in llms. arXiv preprint arXiv:2409.15360, 2024.
Gilad Yehudai and Ohad Shamir. On the power and limitations random features for understanding neural networks. Advances in neural information processing systems, 32, 2019.
Zhuohao Yu, Chang Gao, Wenjin Yao, Yidong Wang, Zhengran Zeng, Wei Ye, Jindong Wang, Yue Zhang, and Shikun Zhang. Freeeval: A modular framework for trustworthy and efficient evaluation of large language models. arXiv preprint arXiv:2404.00 06003, 2024a.
Zhuohao Yu, Weizheng Gu, Yidong Wang, Xingru Jiang, Zhengran Zeng, Jindong Wang, Wei Ye, and Shikun Zhang. Reasoning through execution: Unifying process and outcome rewards for code generation. arXiv preprint arXiv:2412.15118. 2024b.
Zhuohao Yu, Jiali Zeng, Weizheng Gu, Yidong Wang, Jindong Wang, Fandong Meng, Jie Zhou, Yue Zhang, Shikun Zhang, and Wei Ye. Rewardanything: Generalizable principle-follot following reward models. arXiv preprint arXiv:2506.03637, 2025.
Yuanzhao Zhai, Han Zhang, Yu Lei, Yue Yu, Kele Xu. Dawei Feng, Bo Ding, and Huaimin Wang. Uncertainty-penall reinforcement learning from human feedback with diverse reward lora en- sembles. arXiv preprint arXiv:2401.0024: 2023.
14

---

Published as a conference paper at ICLR 2026
Jianfei Zhang, Jun Bai, Bei Li, Yanmeng Wang, Rumei Li, Chenghua Lin, and Wenge Rong. Dis- entangling preference representation and text generation for efficient individual preference align- ment. arXiv preprint arXiv:2412.208..2 2024a.
Xiaoying Zhang, Jean-Francois Ton, Wei Shen, Hongning Wang, and Yang Liu. Overcoming re- ward overoptimization via adversarial policy optimization with lightweight uncertainty estima- tion. arXiv preprint arXiv:2403.05/7,, 2024b.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint ar arXiv:1909.085900 2019.
A THE USE OF LARGE LANGUAGE MODELS (LLMs)
LLMs used assist refining the writing of this paper, including grammar correction, wording refinement, and formatting adjustments. We also use LLM agents to help with finding relevant work and implementing parts of code. The use of AI tools does affect the originality of the work or the authors' responsibility for the content.
B ADDITIONAL RELATED WORK
We now give more detailed discussion of how the present paper relates to prior work. Our approach situated at the intersection three key areas of research: Best-of-N sampling with reward models, reward hacking and its mitigation attempts, and curiosity-driven exploration techniques.
Best-of-N Sampling and Reward Models. Best-of-N (BoN) sampling generates N can- didate responses and selects the highest-scoring according to a reward model: y" arg maxye>> r(x,y) y) (Stiennon et al., 2020). This technique has proven effective for mathemati- cal reasoning (Cobbe et al., 2021; Lightman et al., 2023) and competitive programming (Li et al., 2022) and the version of BoN with oracular rewards (called 'pass@k) (k) is a standard evaluation metric for reasoning models (Dubey al., 2024; Ouyang al., 2022; Abdin et al., 2024; Hui et al., 2024; Lambert et al., 2024). Reward models are typically trained on tuples of preference data (xr, Vchosen; Brejected) consisting of prompt x well as preferred and dispreferred responses Jchosen and Prejected using ranking losses. While a powerful paradigm, this training procedure intro- duces vulnerabilities: models must infer complex reasons for preferences, often leading them to rely on spurious correlations like response length, formatting patterns, or stylistic preferences (Liu al., 2024b). High-quality reward models are also computationally expensive train, requiring sources similar to base language models. This cost makes ensemble approaches, a common strategy for reducing vulnerabilities in ML systems, impractical, as organizations typically train only single reward model per domain (Gao et al., 2023; Jinnai et al., 2024). Our work focuses mitigating reward hacking BoN sampling with a single reward model, without retraining or modifying the reward model itself.
Reward Hacking and Mitigation Attempts. Reward hacking occurs when optimizing against imperfect proxy rewards leads to high-scoring but low-quality outcomes manifestation of Good- hart's Law (Amodei al., 2016; Skalse al., 2022). Gao al. (2023) observed a common trend in BoN sampling, where as N increases, performance first rises then falls. As demonstrated theoreti- cally in Huang et (2025b), these two phases correspond to an initial phase (when N is small) and the learned reward ft is effective proxy for the true reward r", leading to BoN succeeding, and second phase where N grows so large so as to produce atypical responses on which f is no longer effective due to spurious correlations and poor coverage during training (Eisenstein et al., 2023; Liu et al., 2024b; Yu et al., 2025). Broadly, there have been two main approaches to mitigating this reward-hacke problem: training-ting -iiee and inference -imee approaches.
Training- time ensemble methods attempt to mitigate this by combining multiple reward mod- els (Coste al., 2023; Zhai al., 2023; Ramé et al., 2024; Yan al., 2024). However, Fisenstein
15

---

'worst-case' and Published as a conference paper at ICLR 2026
et al. (2023) demonstrate that ensembles reduce but do not eliminate reward hacking. More criti- cally, ensemble methods require multiple expensive reward sources, creating prohibitive costs for practical upplications.
enference-time approaches work with existing reward models and intervene in the sampling proce- dure itself. Of note, Jinnai et al. (2024) proposed an approach that involves regularizing the BoN selection with respect to a Wasserstein distance in some embedding space and demonstrated em- pirical improvement over BoN as N grows. Unfortunately, this approach requires computation to scale quadratically in N, making it somewhat impractical for larger N. Of greatest relevance to the present work is that of Huang et al. (2025b), who propose instead to use an ceermaaiio-thhoorteee divergence, the x² xd-divergence, to regularize the BoN selection. They demonstrate that this approach statistically and computationally optimal under the assumption that the learned reward model is close to the true reward in expected squared error loss for responses sampled from the base model. While theoretically well-motivated and empirically effective at ensuring monotonicity in N, the ap- proach is overly conservative in practice, leading to suboptimal performance when N is moderate. Another inference-time approach is that of Khalaf et al. (2025), who consider a KL-regularized BoN procedure well computationally efficient approximation using the Poisson distribution. Un- fortunately, their method exhibits limited improvement over standard BoN, likely due to the fact that BoN is already effectively KL-regularized unless N is growing exponentially. Unlike these works, our approach does not seek to provide distributional regularization but instead instantiates pessimism through the reward estimates themselves. Thus, our approach is able to leverage the fact that reward models may be imperfect in ways that are not dnformmaiionthhorretie adapt accordingly.
Curiosity-Driver Exploration and Random Network Distillation. Curiosity exploration in reinforcement learning uses prediction error to quantify novelty of given states (Pathak et al., 2017: Sun et al., 2025; Li Gajane, 2023). The Intrinsic Curiosity Module (ICM) (Pathak et al., 2017) uses prediction error (10()))) <(s+41)ll as a curiosity signal, while Random Network Distillation (RND) (Burda al., 2018) employs simpler approach with two networks: fixed target network Tss) and a trainable predictor P(s) that minimizes [[ T(s) In both cases, the supervised learning error serves proxy for OOD detection, with high error indicating states that are far outside the distribution (and thus worth exploring from the perspective of online RL). While these methods were originally developed for exploration, they can instead be instantiated to prevent a policy from moving to far out of the distribution of observed states. While there has been some debate as to the efficacy of these approaches for OOD detection and pessimistic learning (Rezaeifar et al., 2022; Ciosek et al., 2019; Nikulin et al., 2023), our results suggest that when properly implemented, such an approach can be effective in language modeling.
C THEORY
In this section, we give formal statements for and prove our theoretical results. We begin in Sec- tion C. for formalizing the setting we consider which pessimism can help mitigate reward hack- ing in Best-of-N sampling. We emphasize that this setting is a simplified abstraction intended to cleanly showcase the benefits of pessimism and help develop intuition for our approach and is not intended to represent realistic model of language or reward modeling. We then continue in Sec- tion C.2 to prove general results on the performance of BoN and pessimistic selection in our setting. We proceed in Section C.3 to introduce simplified model of our proposed caution approach, moti- vated by Random Network Distillation (RND) (Burda et al., 2018). We prove in simplified linear and two-layer ReLU settings that this approach can be used to instantiate pessimism under an idealized optimization model. Finally, we combine these results in Section C.4 to prove our main theorem that our proposed approach improves over BoN sampling in the model we consider.
C.I FORMAL SETTING
In order to formalize the setting we consider, we suppose that learner is given access to a language model that maps prompt to distribution over responses y. We will further identify these prompt-response pairs with their embeddings in some linear feature space Id and, for the sake of simplicity, consider results per-prompt basis. Thus we will assume fixed prompt and write
16

---

Published as a conference paper at ICLR 2026
for ((x). We will further suppose that there exists ground truth reward function that is linear in the embedded features, i.e.
r" (v) = (0*, .))
Such an abstraction is partially justified by the fact that many modern reward models are linear layers on top of pre-trained Language Models (Lambert et al., 2024; Liu et al., 2024a; Wang et al., 2024a;b) and we thus simply directly associate the concatenation of prompt and response with its embedding in the final layer of the LM.
The goal of the learning problem is similar to that in Huang et al. (2025b); given access to U1.-... V/ ~ sampled independently, as well as some imperfect proxy reward function F: R - R (e.g., a learned reward model), we wish to select response y; with E [n]] such that r" (1;) is as large as possible. Note that if were in fact a perfect reward model, so that f = r", then the optimal strategy would be to simply select y; with = argmax;g(v] f(ye), which the popular Best-of-NN (BoN) strategy. However, as discussed in the main text, this strategy can fail when is poor approximation to r" due to reward-hacking.
Unlike Huang et (2025b), wherein the authors suppose that & and r* are close in expected squared error under we will instead suppose that f and r* agree on a low dimensional subspace of R may arbitrarily different on the orthogonal complement. More precisely, suppose that there exists linear subspace V C IRd such that dim(V) = k < d and r" r") = F(y) for all y V.
We will further make the assumption that r* and f are linear functions, i.e., f(y) = (0,y) v) and
("()) = (0", y) for some 0, 0* Rd. To aid our analysis, we will make the simplifying assumption that 0 E i.e. projy1 0" = O, representing the fact that the true reward is a function only of certain linear features; the error between the learned reward & and the groundtruth r* thus restricted to OOD effects, conceptualized as the orthogonal complement V of V.
Finally, in order make the analysis tractable, we will assume that æ is a centred Gaussian distri- bution with covariance Σ E Rdxd i.e., = (0, Σ). Note that Σ may have full rank d and thus may have full support on TRd, which ensures that & r* on V-, the orthogonal complement of V.
C.2 PESSIMISTIC SELECTION AND BEST-OF-N SAMPLING
In this section provide two key bounds the performance of Best-of-N sampling and pes- simistic selection, which will demonstrate the benefits of pessimism in our setting. Before doing so, state and prove simple bound on the best possible performance that can be achieved by any selection strategy.
Proposition 1. Let ,,...., YN ~ be independent samples from æ. Then it holds that
E max r" (u=))0"
|=
MN,
FE(N]
where MN E [max]e[N] Z:] and Zi....; ZN are i.d.d. N(001) random variables. Moreover, holds that
log((N - o(1) s MN S 2llo((N)
Proof. The second claim a classical fact about the maximum of Gaussians (Wainwright, 2019; Vershynin, 2018), thus it suffices to prove the first statement. Note that
(v) (0" y) ~ NN(0, 12/00-13
Thus r" (v) - /a00'1 Z for Z /O, 1). The result then follows by positive homogeneity of the maximum and expectation.
L
We now provide lower bound on the performance of BoN in the setting we consider
17

---

Published as a conference paper at ICLR 2026
Proposition 2. Let y...... UN be independent samples from a and let i argmax f(16) for fly) = (0.1) 2/ Let argnax r" (y:) for r* (y) = (0*,") v). Then holds that
E
- -rr))-r((u -u"(v)000 - E (mx-)) 1
projj 0112
11
2 12
MN-
||2 81 8102 1
|1/20* ||
Proof. As we have already computed the expectation of r* (1)= ) in Proposition 1, it suffices to compute the conditional expectation of rU;)). Note that because, conditioned on d and 0", f(y) and r" (y) are jointly Gaussian, it holds that
E = Cov(f(y),r f(y)
Var(f(y))
(8,, m②)
+(())
Σ
1-/-000
-f(y)
E
|| |/00/0000 11
2 -(y)
2
projj
EVeprojy
/
12
EVeprojv f(()).
ΣV20 1| +
By the same computation in Proposition 1, it holds that
E f(v:)10r 0| 0 .M V EV 12+
Σ /Vpprovv 6|p² -MN.
The result follows by plugging in and rearranging.
We now show that with pessimism instantiated correctly, we can strictly improve on the performance of BoN.
Proposition Let yı..... UN n be independent samples from a and let a Rd - R4 be
function satisfying
(1 c) ||projy -ll S a(v)) < ||projv vll
te
for some e E (0, 1) and E > Let
projy
pess argmax f(;)) X- aa(i))
A
SE[N]
Then it holds that
2
I (1/1) ) - -r" (r"(((paa)) 5 \
Tr (Eprojy + 28
18

---

Published as a conference paper at ICLR 2026
Proof. Let
(LCB(V) = -(y) X. a(y).
We claim that with the assumption on and a, it holds that rLCB(U) S (r) + A for all y E Rd. Indeed, note that
r" (v)
(1)
0- 0.u) + A +A.a(y) a(()))
projv 0.9) +d.a(v)
projy 0/| |lprojv= v/l + ((1 cc) ||projy E)
where the first inequality is by Souchy-Schwarc
Now observe that
r" ("(r)) - r" (Vignna = =r (w:-) - fr.cn(y:)) )] + [fr.cn(w=)) - fI. ]] + (VVppee) 1" (sipane)
(y-( - (LCB(U--) + XE,
where the second group of terms is non-positive by definition of ipess and the last group of terms is non-positive by the claim above. To conclude, we observe that
(yi-) FLCB = (yi-) f(y)-) ) aly:-)
= <projy 0, yc- + (Iprrov= +
We now observe that as random variables, i is independent of the set {proiv- wild [N]] by Gaus- sian orthogonality and the fact that projv 0" = O. Thus it holds first that
E { projvv 0, (.. 10, 0
E (projv 0, v.) 00,00 fir =dt] Prlii = =10,, 0*1
=
2=1
E
(projy- 0 v. 0- Prli =10,
0,
where the last equality follows from the fact that y is centred. For the second term, observe similarly that
[projyy y/- 10,0010000 0+ =
IIprojy u:ll fir [i==
Pr[i* =10, 0*]
E [Iproiv 1illivv:lll0 1011
Prli" = il0, 0r1
E |rrojy v1ll 10,0] =
n "Pojoojy I|.
The result follows by combining the above.
Critically, the bound in Proposition 3 does depend on N and so, as long as r" r"(()) grows with N, the pessimistic algorithm will eventually outperform BoN. We now show that guarantees on can be obtained through Random Network Distillation.
C.3 ACHIEVING OOD DETECTION WITH CAUTION
Above isolated the key role that |projy 1 yll plays in the failure of the greedy algorithm. While in the linear setting, projection to V results in the optimal algorithm, is impractical in general settings, where V is unknown. Here demonstrate that two simplified models of caution, both inspired by
19

---

Published as a conference paper at ICLR 2026
Random Network Distillation (RND) (Burda et al., 2018) can be used to approximate projection to V and thus yield pessimistic algorithm that avoids the failure modes of the greedy algorithm. Recall that RND involves training student network fa R R" to predict a teacher network fw : IRd - Th on samples from the the explored distribution, which in this case supported on V. The error (1 - fu- (v)ll is then used as a measure of how far out of distribution y is. We will consider two simplified models of RND. The first linear model, where fw (v) Wy for
w E IR The second is a two-layer ReLU network of the form
((1) T
Σ Swa (v),
where
(w(y) UR)LU(Wy)) ReLU(u) = max 0),
(1)
and W (Wi,.... Wm)) Rmxd U _mx" and ReLU is applied coordinate wise. In both cases, we will suppose that the teacher network is fixed with a randomly initialized parameter and the student network is trained to minimize squared error on samples from V. We abstract the mini- mization in the following definition.
Definition 1. Given a distribution D on Rd and an initial parameter W we say that W is trained with gradient based methods with samples from D if W W is in the linear span of WWwww hesupp(DD)
This definition abstracts the precise optimization method used to train the student network, but captures the key property that the final parameter is obtained by following gradients of the squared error loss samples from D. Note that this includes gradient descent and its variants well as more sophisticated approaches involving preconditioning, momentum, and continuous time limits.
We begin with the simpler linear case.
Proposition 4. Suppose that W*, E <mdx are Gaussian random matrices with independent (0, 1/m) entries. Let W denote the minimizer of the expected squared error on samples from distribution with support V attained through gradient based methods from samples y supported in V initialized at Wo in the sense of Definition 1. Then with probability at least 1 -5 over the choice of W* and Wo, it holds that for all y E Rdd,
(1 (1-CV/ h: + log(1/6)
[|provvlvl' S (1) Sw- [w)(1)]| S (
1 1 1
k + log(1/6)
2
!|projvivl?*
In particular, if as long as m > k then (15 - fw- (w)ll is good approximation to |projv
Proof. We first observe that Wprojy = Woprojv since the training data is supported on V. Indeed, the gradient of the squared error loss on sample y E V is given by
Vw IWy - W* y/l = 2(Wy - w)y""" = 2(W (projjyyy)) projv,
where we used the fact that y = projv y for all y E V. Thus Vw |[y w projy 0 for all E V and the claim follows. Moreover, it is immediate that W Wprojv- w "'projy since the loss is minimized at W by strong convexity of the loss function. Thus it holds that
(v) - fw- (w)ll = | - W')'' |
"') || = II( - W projir v
Letting Z = Wo - W+, we see that Z is a Gaussian random matrix with independent (0, 2/m) entries. We may now apply standard results the singular values of Gaussian random matrices (cf. e.g. Vershynin (2018, Theorem 4.6.1)) to observe that there is some constant C' such that with probability least 1- 5,
1 1 1-CV / /k log(1/8)
k log(1/8)
< Amiu (2) S Amax (2) +
This suffices prove the first statement. The second statement follows immediately.
20

---

Published as a conference paper at ICLR 2026
Note that the above result shows that the RND enror is good approximation to llprojv yl uni- formly over all y E Rd with high probability as long as the embedding dimension is sufficiently large relative the intrinsic dimension of the explored distribution. While this is nice first step, the lack of flexibility of linear functions is a significant limitation. We now proceed to the hidden layer ReLU network case.
Instead of assuming that (w( is a linear function, we now suppose that fw (y) is a two-layer ReLU network of the form given in (I). For the sake of simplicity, we assume that only the weights W are trained and that the second layer weights u; ~ (O, 1/m) are fixed. Note that, while practically unrealistic, the assumption that only a single layer is trained common in the study of deep learning (ef. e.g. (Jacot et al., 2018; Yehudai & Shamir, 2019; Song et al., 2018) and the references therein) and is motivated by the utility of random features (Rahimi Recht, 2007). Our analysis is inspired by that of Melamed et al. (2023), who consider similar model of neural networks, but for the very different aim of investigating adversarial robustness. We show that as long as the number of hidden features m is sufficiently large relative to ||y||, then the RND error is again good approximation to I vll uniformly over all y Ied with bounded norm with high probability.
Theorem 2. Let W W E 1m be Gaussian random matrices with independent N N 1) entries with rows W and wi respectively. Let u², uo ~ (0, 1) and let fw- and fwo be the corresponding ReLU networks as in (1). Suppose (i) that W is obtained through gradient-basee
method from
a distribution with support V initialized at W as in Definition I and (ūi that fw((y) (y) = Jw- (v)
for all V. Then it holds with probability at least 1 5 over the choice of W Wo Ug u4 that simultaneously for all y E Rdd,
d dllo(dm//)) C I|yll d dlog(dm//)) < (1) Sw- (v)l
e ellprojv-vl
C Cluull
Tm
Tm
and
[(T(()) - Sw- (w)ll < ||proivivl| + C [Ivll? V dlog(dm/6) Tm O |lvll d dloo(dm/))
Tm
In particular, for T' > ||yll* and up to constants, probability.
[1p(() fw- (n)l(
llprojv all with high
Proof. We first prove the result for T = 1. This step rests four lemmata. We first show in Lemma the key property that, due the training data being supported on V and the fact that we are only training the first layer weights, it holds that W projy = W projy i.c., weights in the orthogonal complement of V are never changed during training. By the assumption that we have trained to convergence, then, it holds that W proojy W Wroojv. We then use prove in Lemma a decompo- sition of the RND error that rests on the precise funcitonal form of the ReLU network. Using this decomposition, we are able to provide bounds on the mean and concentration of the RND error in Lemma 3 and Lemma 4 in the special case that |/y/| = 1. The result then follows by combining these lemmata with the positive homogeneity property of ReLU networks: for any r > 0, it holds that fw (rv) rfw (v) for all y E Rd and all W Ri
Now that the result is proved for T = 1, the general case follows by tensorization across the inde- pendent We.
Lemma 1. Let fw., fwo, and fw be as in Theorem Then it holds that Woprojy: W Wprojyn.
Proof. By Definition suffices to show that for any y E V. (www))))) projv 0. To see this, observe that
1 m
Vwfw(y)
Σ j=1
utyll ( w) > 0] y.
Because yprojyı 0 for all y V, it follows that Fw/w(y))pmg 0. The result follows. n
21

---

Published as a conference paper at ICLR 2026
Lemma 2. Let fws, fwo, and fip be in Theorem 2. Then with probability at least choice of W W U4, up. it holds that for all y I2
over the
1
m
(ugeug projv 1 v) / [projjw projv wº, projvy + (projvi)) 0]
(fw (v) fw- (y))
m
m
(uj,wt projv- v) / ((Kuf.projj + (projv v) > 0
m
Proof. By the fundamental theorem of calculus, we have that
(v) - fi (proiiy v)), = f(' VVyfw (projv + t(proiv y p)oj - projo v) dt
L. (projvv
(projvy + lproiv u)s, projv y) dl.
For any W, it holds that
m (y) Σ u ( v) > 0].
Thus by the linearity of the integral and Lemma 1, it holds that
m
(u projv v\ (. [(0s, proivy + (projv 0] dt
mLF (v) fw
Σ (us:ro) projvi v)) J [projywt projy wi projvy (projy 0] di,
where we used the fact that W W "projy projvi by Lemma 1. A similar, but simpler, argument applies to fw.. We now use the fact that Wprojy = W "Prrooyy to observe that
fw (v) fw- (v) = (w) Iw (proive) (fw- (y) fw (projvy))
and the result follows.
Lemma 3. Let sw,, fwo, and fw be as in Theorem Then it holds that
projv yl1 S || (v) fw. [w]( s ||prjv-
Proof. Let
g(w, v) = (wc, por.proj--y S.' I((,, projvy + (projv > 0) dt
(2)
and
(w, w.. v) = (w', wwr.iooiv y\ / [(projyw + projv w', projvy tprojy v) 0] dt. (3)
By Lemma it holds that
27 Σ Ci wo v) u5,0(uf y)
E
(F (v) - fw. (2));²
E m²
E [g(w]
22

---

Published as a conference paper at ICLR 2026
because the uf; and uj, are independent and have variance and g'(w,, wp, y) and g(w v) independent of ul; and uj, and identically distributed. Note now that
gw y)² s {projv-y; projy-y)
and thus has expectation at most l - ylla. For the lower bound, observe that
!'' /'" (i) projv w)² 2 [ projvy y + tprojv y) 0] [[(w] proiv + sprojv 8) > 0] dsdt
g(u .)2
!/"
/ (w), proiv w)² [ projve) 00][[(((] projv- y) > 0] dsdl
(w) projv 8)²
Thus
[Iprojy-bl' vll (wi projy Mroiv--i y)² S TE (W (v) fw- (w))3
(w7 projv- |lrojv
4m
4m
Summing over j gives the result.
Lemma 4. Let fizz fwo, and fw- be as in Theorem 2. Then with probability at least - over the choice of W*, ,Wo, Ube up it holds that uniformly in for llyll =
(1
dllogddmg (dm/s)
d (lmg()
[1 (v)l E (1 fw- (v)) cc
m
Proof. Begin by noting that by positive homogeneity, it suffices to set |yyl 1. Continuing with the notation introduced in (2) and (3) we see that, conditional on w w. it holds that (fir(y) Sw (v)); are independent centred Gaussians with varianese given by
m
Var(U/ip()) - fw- (w)) /wo, w!)
Σ g'o wl u) + g(wl, u2)?
m
i=1
Thus by standard bounds on the concentration of norm of Gaussian vectors, it holds that with prob- ability at least 1 8,
1
m
| (v) fw- (v)ll
(w: wi y) + glw 3)
m
C.
o (w wp.y)² y)² g(w7, 2)2
log(?/6)
m
C
(w w. y)² + g(wt, log(1/))
m²
Letting
1 m
G(w", w..
m Σ g (wtt wi wp,v) y) ((w),y)
we see that
( [og(1/0) m
log(1/))
(1) fw- (v)] G(w=, w 8) s C- CCGG""" ...y))
v)
(4)
We now demonstrate that G(w", w y) concentrates around its mean for fixed y. To do this, will first observe that g(w, y) and g'(w, w', y) are identically distributed and thus it suffices to show concentration for g(w,, y) and apply a union bound. Indeed, we have that
g( v) < (wo projv v)²
23

---

Published as a conference paper at ICLR 2026
and thus has Orlicz 1/1 norm at most C' ||projv vl for some constant C. Thus by standard concen- tration results for sums of independent subexponential random variables (cf. e.g. Vershynin (2018); Wainwright (2019)), it holds that with probability at least 1 5.
log(//5)
Σ gww.. u)² E [g(w,,
= C Ilprojy vll
Combining this argument with the triangle inequality and (4) along with the fact that ||ll| 1 and Lemma 3 gives that with probability at least 5,
[1 - fw- [w)( (
(1 fw- ())) S sc
(V
log(4/6)
m
m
We now observe that by standard high probability bounds on the operator norms of Gaussian random matrices (cf. e.g. Vershynin (2018, Theorem 4.6.1).. it holds that with probability at least 1 - 6, that fw and fw C-Lipschitz as is E [ø(w, y)² in for llvll =. Thus by standard covering argument on the unit sphere and a union bound, the result follows.
C.4
MAIN RESULT
We now state our main result, which combines the analysis of pessimism in Section C.2 with the analysis of caution in Section C.3.
Theorem 3. Let V be a k-dimensional subspace of IRd and let E be a positive semidefinite matrix such that Tr (r(pprojy)) O. Let {y: be i.i.d. samples from (0, Σ) and let (V) (0*, y)
for some lr E V. Suppose that â IRd such that projy 0* and let f(y) -0. Let
(L) f(y) A- a(y), for a<(y) = (v) fw- (y)l with fw(y) being either the linear model considered in Proposition 4 or the one hidden layer ReLU network considered in Theorem 2.
Let i argmax FLCB(U) and i" [rgmax,e[N] r* (3:). In the case that fw is linear, as long
k(d- k) || projy 0|1
and X = p[[proov - 6 f|). it holds that log(N)
as m 2
(yipmen) - r (1r) Z Vlo((N).
In the case that fw is one hidden layer ReLU network, as long as T'm 2 d(d d(d-k) pl 61 log(N)
the
same holds with À = provv 0
Moreover, in both of these cases it holds that
E
[r* (y--) r (Vlpare)) 1 (ys+
E |r - r (v:)
lim N10o
lim Nooc
E [r+"())))
Proof. The result follows immediately by combining Proposition 3 with Proposition 4 and Theo- rem 2 and observing that [|projy yll & v(d - k) (18) with probability at least 1 8 by standard concentration results for chi-squared random variables (cf. e.g. Wainwright (2019)).
D EXPERIMENTAL DETAILS
Hyperparameter Configuration. Table provides comprehensive overview of all hyperparam- eters used throughout our experimental evaluation. The configuration represents a careful balance between computational efficiency and model expressiveness, with key design choices motivated by theoretical analysis and empirical ablations.
The RND architecture employs 10 layers for both target and predictor networks, significantly deeper than the default 4 layers, which we found provides better representation quality for uncertainty es- timation. The RND weight = 0.2 represents a moderate pessimism strength that effectively mitigates reward hacking while preserving the benefits of reward- reward-guided selection. Training hyper- parameters including the reduced learning rate (1 x 10-5) and extended training duration (5 epochs) ensure stable convergence of the predictor network the GSM8K training distribution.
24

---

Published as a conference paper at ICLR 2026
Architectural Ablations. For the architectural ablation study, we design different levels of net- work complexity and check their impact on the training objective, the reconstruction loss and also the final accuracies. Table 5 contains comprehensive introduction of settings involved in our abla- tion studies.
Implementation Framework. Our experimental setup utilizes PyTorch 2.3.1 as the primary deep learning framework, with model inference accelerated through vLLM 0.10.0 and Transformers 4.55.1 for efficient large-scale language model deployment. For RND training, we extract repre- sentations from the first 10 layers of both predictor and target networks, employing learning rate of 1 x 10-5 with 50 warmup steps across 5 training epochs. The RND models are trained on out- puts generated by the backbone model using the GSM8K training set, and subsequently evaluated against the validation set of GSM8K as well as Math-500 and BigBench-Har benchmarks to assess generalization capabilities.
Inference Configuration. All inference experiments maintain consistent hyperparameters with temperature set to 1.0 and maximum token limits of 500 for GSM8K and 1024 for Math-500 and BigBench-Hard evaluations. The increased token limit for harder datasets is necessary because these benchmarks require significantly more reasoning tokens avoid truncation before reaching a con- clusion. Response generation uses vLLM for efficient parallel sampling across multiple candidates in Best-of-N evaluation.
E ADDITIONAL CASE STUDY
To gain deeper insights into the mechanisms underlying reward hacking and caution-based mit- igation, we analyze the correlation patterns between reward model scores and pessimism scores through detailed scatter plot visualizations. Each plot displays z-normalized scores for all responses to individual GSM8K problems, with green points representing correct responses and red points representing incorrect responses.
The scatter plots in Figure 5 reveal two critical failure modes of reward models that our approach successfully identifies. In the high reward, low pessimism region (upper-left quadrant), we ob- serve responses that exemplify systematic reward hacking. These responses achieve high reward scores not through genuine correctness or adherence to task requirements, but by exploiting spu- rious correlations that the reward model learned during training. Crucially, these responses often ignore fundamental formatting requirements of the mathematical reasoning task, such as provid- ing the final answer in the required format, yet still receive high rewards because the reward model prioritizes superficial indicators like verbosity, step-by-step presentation, or mathematical terminol- ogy over actual task compliance. This reveals that reward models can be systematically misled by responses that mimic the surface patterns of high-quality reasoning without delivering the essential components of a correct solution.
Conversely, the low reward, high pessimism region (lower-right quadrant) contains responses that follow proper formatting conventions and adhere elosely to the expected task structure, yet receive low reward scores. This pattern illuminates fundamental limitation of reward models: they func- tion primarily as distributional fitness measures rather than objective quality assessors. These well-formatted responses are penalized not because they lack correctness or clarity, but because they deviate from the specific stylistic preferences and response patterns that dominated the re- ward model's training distribution. The reward model essentially measures how closely response matches its learned notion of "preferred" responses rather than evaluating genuine task performance or adherence to explicit instructions.
This analysis demonstrates that our caution mechanism successfully identifies both forms reward model failure: it flags spurious high-reward responses that exploit correlational biases while rec- ognizing genuinely task-compliant responses that happen to fall outside the reward model's 's narrow preference distribution. The results underscore that effective reward hacking mitigation requires moving beyond simple score-basee based selection toward distributional awareness that can distinguish between genuine quality and superficial pattern matching.
25

---

Published as a conference paper at ICLR 2026
Question Mondy Ponocier $100, They agood momblyy 2%, Mandy was to pay sller mondss, how much give Beseloor
3-
Answer: The munebly intoeer is $100 2/100 --c11000/100000--0 100-20022 After mombs, the tond imercc smouze to 92x 5eo253-600 Therefore Maraly skoddd give Benedat toaal u1000 - Ss 1ec100060 1050-116 105, #hri 105
Reward (z)
Respunac 2:
To sovve this pusllens need to ciletate the imerese for montss, The Laseeet Prixipal Rae
1-
fonndla fox 0.02 docimal), Time mombs Fint, Here, Priccpal in $100, monthe years,
the time years.
soned, mmoothh 1/12 years Now, we ov calottoe the imoeer lmoree $100 0C0 Intreet - 53 Su. Maady owes Besedlo in wilkiond $5 intrrss In woul Mands dooul par Slik) (srigical ansun)) + 55 (ercnod - $1065 to Beredis.
interes
(Figh Reward, Low Peas.]
-1
Respumee 2: To find oun how much Mandy should give m Benofier, we reed to
caloshne the soad isnoos for 3 moethe Calealate the monthly insera 20 $100 (2/100 - $100 $2 mumes Calaaare the roal intror fir 3 monthe $2 $1.0 5105, Mandy
-2
odgiial shoudd give Deseda St0k, #inett (108)
Add the sed inssest
-2.5
-2.0
-1.
-1.0
0.5
1.5
[Higg Pess, Reward]
Pessimism (2)
Question: Mack warss to count things and dleided to count how mony hirds there are the sky over the neu week Ou days one and two, Madk spy a total cky three be sew unse, On cass Jout and live, Le sox 120 lucs. On day sis he sow 20 and or day seven be sew 90 On avergee how many lands dil Mack see in doy?
laids Oa
3.0
2.5
days seven, 120-20-000 20-0000--0000-- linds, the whole week be one 250- 50- ex200 50-230000380 hinds De acorgg Mack 200/7-xxx 7- 405-0-0 hinds day. Hestes 40
Answen
one theee,
2.0
Response 1: To find the werrgg namrees of back Madk son yes day, we noal to caloulate the notal number of hinds he saw and diside it the number of days Eat's add up the number of hinds Micck sas on each dap Day Ii s0 bank Day 2 s0 bind Day is birds Day 4 120 binde Day 5: 170 bink Day birck Day 120
Reward (z)
1.5
1.0
bink Toral num ber sinds Mack counted
50-+
50 Tool number of bads 450
days, dirik
mimber hizda by cumber daye Avenge
0.5
numbec per linds per day - 450/ 7 Averge mimber binck per day 64.29 So, Mack sau aypnosimendy 64.20 hinds per day on averag.
Total number hinds / Number of days Avengs number
0.0
(High Reword, Law Pos]]
0.5
the cotal number linds ore and disidett it by the won mimber of doys calesher the roial number of linds Mack ere- Doye
Response
everge cumber of land
in
cakoskee
50 birts- Day 3:0 birbs Days & 120 hinde- Day & 20 bich- Day 75 50 hinds Tora number of hinds-
-..0
a d 50 280 New, se noal to fidd the soai number of days Mark counted bink over days. Toua number of days = To find the avergs, diside the und mumber of hinc by the namber days Average nutber of badk yer Youd nunker kols Total dayslam
-1.0
-0.5
0.5
Pessimism (2)
HHggh Pess, Low Reward]
Figure 5: Pessimism-Rewart visualization on GSM8K. Each row shows one problem: a scatter plot of z-normalized pessimism (x-axis) and z-normalized reward (y-axis), with green points for correct responses and red for incorrect. Upper-left points (high reward, low pessimism) illustrate re- ward reepoosssssssosss that score well despite low distributional support. Lower-right points (low reward, high pessimism) are well-formed, instruction foodioinn responses that the reward model un- dervalues; our caution mechanism up- weights these relative to reward-only selection.
F DETAILED RESULTS FOR ABLATION STUDIES
This section complements Table 2 with full scaling curves. We sweep pessimism strength X E (0.0, 0.2, 0.8, 1.0) and vary the Best-of-N budget over N (1,2, 4, 8, 16, 32, 64, 128, 256, 512]. Figure shows our caution variant that uses reward-model features; Figure shows traditional RND with random targets.
Caution (RM features). Moderate-highe À (about 0.6-0.8) maintains improves accuracy N grows, preventing the reward-hacking drop seen at X = 0 (RM-only). À = 1.0 (pessimism-only) is competitive but slightly conservative at small N. this is because for easier problems, most responses contain the correct answer but not all of them strictly follows the specified answering format, and applying pessimism only would at least filter out the incorrectly formatted responses. Overall, À e [0.6, 0.8] delivers the best trade-off across most N.
Traditional RND (random targets). Accuracy remains flat or declines with N for all X, and rarely exceeds the RM-only baseline. Sweeping offers little benefit, indicating that random targets lack the semantic grounding needed for useful uncertainty estimates.
26

---

Published as a conference paper at ICLR 2026
82
80
Aug. Accuracy
78
76
14
72
Only
N samples problem
Figure 6: Caution (RND-on-RM-features) scaling with A. Best-of-N accuracy on GSM8K versus samples per problem (X-axis) for Pessimism strengths À E {0.0, 0.2, 0.4, 0.6, 0.8, 1.0). The predic- tor is trained against frozen target network built from reward- model features. Larger À increases pessimism strength; À = 0 reduces to Reward-Model-only Model-only selection, and À to pessimism-only Moderate-high high weights (roughly 0.6-0.8) preserve scaling while curbing reward hacking, outper- forming both the RM-only and RND-only extremes across most N:
Table 6: Layer depth ablation for target network and predictor network.
Number of Layers (L). In Table we study how the selection of L would impact performance, on smaller subset of GSMSK with 100 problems and 100 responses for each problem. Generally we observe higher peak ac- curacies and less degradation when given more layers, al- though at the cost of compute budget.
Peak Acc.
Final Acc.
2
79
77
83
78
84
81
10
84
84
G ADDITIONAL BASELINE COMPARISONS
To further validate the effectiveness of our pessimism approach against reward hacking, we con- duct comprehensive comparisons with additional baseline methods from the literature. Specifically, compare ours against x2 regularized sampling Huang al. (2025b), softmax best-of-n (which applies temperature-based softening to the selection distribution), and Best-of-Poisson sampling (which draws a Poisson-distributed number of samples before selection). We also evaluate com- binations of pessimism with these alternative sampling strategies to assess whether our approach provides complementary benefits. All methods are evaluated the same experimental setup using GSM8K.
Figure 8 presents the scaling curves for all methods. The results demonstrate that pessimism approach exhibits superior robustness against reward hacking degradation. While the standard Best- of-N baseline shows severe performance degradation. Notably, combinations of pessimism with other sampling strategies (e.g., Pessimism + x², Pessimism Poisson, Pessimism + Softmax) show comparable stability to pessimism alone.
27

---

Published as a conference paper at ICLR 2026
80
78
Avg. Acouracy
76
74
Rewand Model Only
Erflle RND (MHM
0. R.2-RaaRND-6 grAM
B.RRaaRRDDO 0. 2MMM
Ofly
2
211
2-8
samples per problem
Figure 7: Traditional RND (random targets) baseline. Best-of-N N GSM8K accuracy when us- ing classical RND with a randomly initialized target network (no reward-mode features), sweeping A E (0.0, 0.2. 0.4, 0.6, 0.8, 1.0}. Unlike our caution variant, this baseline shows little to no scal- ing benefit and generally does not surpass the neward-Model-oog curve, indicating that semantic grounding from reward-mon model features is crucial for effective distributional regularization.
Polsson
Sclie
Seveenr Aaseaneee (00
12
Number of Candidates (M)
Figure 8: Comparison of pessimism-bann based sampling with alternative baseline methods.
28

---

Published as a conference paper at ICLR 2026
Table 4: Comprehensive Hyperparameter Configuration. All hyperparameters used in training and evaluation of the Random Network Distillation (RND) approach for mitigating reward hacking. The table is organized by component: RND architecture, training process, inference settings, and evaluation configurations.
Component
Parameler
Value
Description
Target layers
Number of layers extracted from reward model for target net- work
Prodictor layers
Number of layers in predictor network architecture
RND Architecture
RND weight (A)
0.8
Strength of pessimism penalty in combined scoring
Exact unchitecture
False
Whether predictor cupies exact reward model archilecture
Embodding strategy
shared_trainabl
How embeddingg are handled: shared, trainable. hared_frezeer separate
Use projection
True
Whether to add projection layer between predictor and target
Batch
8
Training
for RND predictor
Learning rale
le-5
Learning rale for predictor network
Training
Number epochs
5
Trsining epechs for prodictor network
Process
Warmap steps
50
Learning rate scheduler warmup steps
Max examples
5000
Maximum training examples from GSM8K split Minimum VRAM requirement for GPU
VRAM usage
24GB
Temperature
1.0
Sampling temperature response generation
Inference
Max tokens (GSM8K)
500
Maxinum tokens for GSMBK responses
Settings
Max tokens (MATH/BBH)
1024
Maximum tokens for harder reasoning tasks
Number of samples (N)
I-512
Range of Best-of-N sampling candidates
Backbone model
Llana.3 Instruct
Base langunge model response generation
Reward model
OASST DeBERTa
Primary reward model for scoring responses
Training datasct
GSM8K
Datasct training RND predictor
Evaluation Configuration Bootstrap runs
Test datasets
GSMBK, MATH-500, BBH
Evaluation benchmarks (in-domain and OOD) Number of boststrap runs for confidence intervals Normalization method for reward and RND scores Method for selecting best response from candidates
3
Score normalization
Z-score
Selection strategy
highest_reward
RND weight range
0,0-1.0
Range of values tested in weight ablation
Ablation
Architecture variants
4 types
Fall, simplified, embedding strategies, projection ablations Standard Best-of-N and possimism-only baselines
Study
Comparison basclines
BoN, RND-only
Table 5: Summary of ablations. Each row defines one setting and what it means in practice.
Ablation Setting
What it means/ / Rationale
Predictor architecture
Same as Target
Predictor uses the same overall structure as the target network (e.g., same block type and connectivity), matching width and depth. Isolates training dynamies from architectural mismatch.
Simplitied
Predictor keeps the same hidden size and number of layers as the tar- get but replaces specialized target blocks with vanilla Transformer encoder blocks. This deliberately reduces architectural complexity while preserving depth/width, aiming for better generalization and lower overfittigg risk.
Embedding strategy Shared, trainable
Predictor shares the target's 's token embeddings and updates them during train- ing. Pros: reuse target's ss pretrained semantic representations and potentially quicker convergence. Cons: tighter coupling may leak target-specific biases into the predictor.
Shared, frozen
Predictor reuses the target's token embeddings but keeps them frozen. Pros: stable token mapping and clean isolation of predictor encoder learning. Cons: less flexibility to adapt embeddings to the predictor's 's simplified blocks. Predictor creates its own embedding layer with random initialization (initial- ized via model's s standard weight init). Pros: full decoupling from the target, potentially better regularization. Cons: longer warm-up and higher op- timization burden to reach alignment.
Separate, randomly initialized
Projection head No projection head
The predictor's 's final hidden states are directly mapped to the output space used for matching the target's 's features. Minimal additional parameters; simplest path that reduces opportunities for overfitting.
I Linear projection head
Adds a single linear layer after the predictor's 's hidden states and before the output. Acts as a light adapter/bottleneck to better match target fcature ge- ometry; can improve fit at small cost in extra parameters, but may introduce overfitting.
29


Reviews:
[{"review_id": "03Z49enjft", "reviewer": "Reviewer_4Xb3", "summary": "The paper addresses the problem of reward hacking in Best-of-N sampling, where due to biases of the reward model sub optimal preferences might be made. The authors propose \"caution,\" an inference-time technique that instantiates the principle of pessimism from reinforcement learning. It works by training a lightweight \"predictor\" network to predict the internal feature representations of the frozen reward model on a dataset of typical responses. The prediction error of this network is then used as an uncertainty score. This score is subtracted from the reward model's output to create a \"pessimistic\" reward, which penalizes responses that are out-of-distribution (OOD) from the perspective of the reward model. The authors demonstrate this approach with significant gains over GSM8K, MATH-500, BigBench-Hard reasoning tasks.", "questions": "- The main comparison is to standard BoN. How might \"caution\" compare to other classes of OOD/uncertainty detection methods applied at inference time, such as using Monte Carlo dropout on the RM's final layers or calculating Mahalanobis distance in the feature space of the RM to penalize outliers?\n- The predictor Pθ is trained on typical responses from one policy (Llama-3.2-3B). How well would this pre-trained predictor generalize if used to score responses from a different, more capable base model?\n- he case study highlights the detection of formatting errors. Can you provide examples of more semantic or stylistic deviations that the method successfully identifies and penalizes?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper addresses an important and persistent problem in reward hacking that exists despite a large amount of papers on the subject.\n- The experimental evaluation is comprehensive and well-designed. The testing across different distributions and domains (GSM8K, MATH-500, BBH) convincingly demonstrates the robustness of the approach.\n- The ablation studies in Section 3.2 are thorough and successfully validate the key design decisions, especially the critical importance of using reward model features over random ones.\n- The proposed method is computationally efficient and practical. The predictor network is trained fully offline, and at inference time, its forward pass can be parallelized with the reward model's, adding minimal overhead. This makes it a readily applicable solution for practitioners already using BoN sampling.", "weaknesses": "- Weak theoretical justification, It is unclear how the insights from this linear-Gaussian model translate to the complex, high-dimensional geometry of transformer feature spaces.\n- Stronger baselines. The paper could be strengthened by comparing \"caution\" to other plausible inference-time uncertainty estimation or OOD detection techniques. There has also been a lot of work in reward hacking mitigation, so a comparison with more techniques would be appreciated.\n- Qualitative examples - the paper would benefit from more qualitative study of the hacking mitigations\n- Human alignment - more human evaluation would strengthen the usefulness of the approach for most tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w3At8xsgmX", "reviewer": "Reviewer_sWFV", "summary": "* The paper aims to improve the degradation in performance in the Best-of-N sampling scheme for large values of $N$ (\"reward hacking\") by constructing a weighted sampling scheme which combines reward and uncertainty scores.\n* In the model, a language model stochastically maps a prompt $x$ to a response $y$, a reward model $r$ is a mapping from prompt-response pairs $(x,y)$ to scores. Given reward model $r$, uncertainty estimate $u$ and hyper-parameter $\\\\lambda$, the proposed approach samples candidates according to the “pessimistic” reward $r_\\\\mathrm{LCB}=r-\\\\lambda u$. The uncertainty estimate is inspired by random network distillation, aiming to detect OOD datapoints. \n* A theoretical analysis is briefly described in the body of the paper, and presents a simplified setting in which pessimistic sampling converges towards the optimal-reward response but the absolute gap between naive and pessimistic sampling grows with $N$.\n* The empirical analysis evaluates performance on three datasets, showing favorable performance compared to the Best-of-$N$ baseline.", "questions": "* Is it possible to elaborate on the intuitive assumptions and limitations underlying the theoretical analysis?\n* What are the general limitations of the proposed approach, and when is it expected to fail? \n* How does the method handle cases in which the ground truth reward has “true\" label noise? (e.g., topic in which annotators have typically diverse preferences).\n* Is it possible to formulate the relation the proposed method as a \"soft distance constraint\" between the training set and candidate response distributions? For example, if I understand correctly, it seems that only \"in-distribution\" candidates will be sampled when $\\\\lambda \\\\to \\\\infty$ and $u$ is a perfect OOD detector, because any OOD candidates will be penalized heavily. Does this intuition hold? And if so, what can be said about intermediate values of $\\\\lambda$?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* Addresses a well-motivated and timely issue.\n* Proposed method seems relatively simple to implement, suggesting potential practical applicability.\n* The presentation is generally clear and easy to follow.", "weaknesses": "* Despite the focus on empirical analysis, the analysis code doesn’t seem to be attached.\n* While related prior work is mentioned (e.g., Huang et al. 2025, Jinaai et al. 2024), empirical evaluation doesn’t seem to include it as a possible baseline.\n* The theoretical analysis underlying section 2.3 seems to be relatively intricate, but the body of the paper presents it only briefly. In particular, key assumptions, their limitations, implications of the analysis, and proof techniques are do not seem to be discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fd8F7EPmqW", "reviewer": "Reviewer_7MEf", "summary": "In this work, the authors mitigate reward hacking by adding a pessimism or caution term to a reward model. Specifically, they focus on Best-of-N sampling and mitigate the selection of high proxy-reward, low true-reward generations by penalizing out-of-distribution samples where the proxy-reward estimate is uncertain and inaccurate. They do so by training a second network to reconstruct the proxy-reward model's latent representation. Then, on future data, they use the error of this reconstruction as a measure of the certainty of the reward model. They regularize their reward with this certainty term, penalizing OOD samples which result in low reconstruction while having abnormally high reward.", "questions": "* In Table 2, it seems you use $\\lambda$ as a convex combination of reward terms, $\\lambda r(x) + (1-\\lambda) u(x)$. This is different than your regularization term introduced in line 236. Can you standardize this?\n * Intuitively why do we expect any performance gains when using caution only? We aren't using any reward signal at all, and are just picking completions that are more in distribution. What are the costs in terms of coherency or diversity in completion?\n * In Table 3, for Lightweight + Separate Emb.: How can Peak Acc be lower than Final Acc?\n * In your proofs, you condition on $\\theta$, but isn't $\\theta$ not a random variable? Why are you conditioning on it?\n * Can you explain in Prop. 2 where you use the $E[r^*|\\hat{r}]$?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* The method is intuitive and interesting in its own right: quantifying OOD-ness of data using a second model trained with reconstruction loss\n * The paper is clear and well-written and provides a thorough study of their method and demonstrates its efficacy.\n * The empirical results demonstrate hacking mitigation on mathematical reasoning tasks.", "weaknesses": "* The authors say they see \"monotonic\" performance improvements for all N, yet in my opinion the curves look more like a plateauing reward. Indeed, past work on hacking often frames the approach as avoiding over-optimization and recovering an \"optimal\" reward rather than expecting gains as N increases forever [1]. I don't think this claim is necessary for the impact of the paper, and I think some may disagree with it.\n * Notationally, it was a little confusing introducing $T$ as separate from $r$. I understand that you take some intermediate layer from $r$, but I might define that more explicitly in the paragraph on line 202 or streamline notation. \n\n[1] Khalaf, H., Verdun, C. M., Oesterling, A., Lakkaraju, H., & Calmon, F. D. P. (2025). Inference-Time Reward Hacking in Large Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "45lyib57xv", "reviewer": "Reviewer_JSWD", "summary": "This paper addresses reward hacking in Best-of-N (BoN) sampling. The authors introduce an elegant, lightweight, and practical solution called \"caution.\" \n\nThe method works by training a lightweight predictor network to mimic the internal features of a given reward model on in-distribution data. At inference time, the predictor's error on a new candidate response serves as an uncertainty penalty, which is subtracted from the original reward score. Extensive experiments demonstrate that this approach effectively mitigates reward hacking, leading to improved performance with N.", "questions": "- How do you expect the \"caution\" mechanism to perform if the base reward model is systematically biased? \n\n- The paper's discussion of Huang et al. (2025b) is very clear. A direct empirical comparison on the same experimental setup would be appreciated, as it represents the most relevant alternative approach.\n\n- Why do you not consider calibrated reward as done in InfAlign (Balashankar et al., 2024)? Since Best of n only cares about the ranking of the reward scores, such a calibration is theoretically motivated. Instead, you opt for a z-score transformation. Can you elaborate on lines 300-302? In particular, did you perform the z-score transformation per prompt or across prompts?\n\n- You may consider citing [2]. They use the reward model's learned representation of the prompt response pair to model its uncertainty. They argue when a data point is (almost) in distribution, there is less uncertainty.\n\n[2] Zhang, X., Ton, J. F., Shen, W., Wang, H., & Liu, Y. (2024). Overcoming reward overoptimization via adversarial policy optimization with lightweight uncertainty estimation. arXiv preprint arXiv:2403.05171.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- A core strength is the paper's excellent presentation and framing of the problem. Drawing a parallel between curiosity-driven exploration in RL and \"caution\" for avoiding OOD reward exploitation is an insightful and generally useful contribution.\n- The proposed solution is compelling in its simplicity and practicality. Training the auxiliary predictor network is a one-time offline cost (with unlabeled data), and the inference-time overhead is minimal as it leverages the same internal representations as the reward model. \n- The experiments are extensive and well-designed.", "weaknesses": "W1. This is both a perceived weakness and a question. Please correct me if I misunderstood the setup. The \"caution\" mechanism defines \"in-distribution\" by learning the typical feature patterns of the base reward model. This is effective at penalizing novel responses that exploit weird loopholes, but it may fail to mitigate, and could even reinforce, the reward model's own systemic biases which you mention in Lines 81-83 and in Figure 4. For instance, if a reward model was trained on data where humans consistently preferred verbose answers, its internal features would encode verbosity as a \"normal\" characteristic of high-quality responses. The caution predictor would learn this pattern and would consequently fail to penalize a verbose, incorrect, reward-hacking response. It might even penalize a concise, correct response for being stylistically \"out-of-distribution.\" In essence, the method regularizes generations toward the central tendencies of the reward model, and if those tendencies are themselves flawed, the method risks preserving those flaws. If doing so is indeed shown to mitigate hacking, why do you think hacking happens to begin with?\n\nW2. What is the most fundamental way to combine a reward signal and an uncertainty signal with BoN? Subtracting the uncertainty from the reward feels very natural from an RL / training point of view. However, BoN only cares about the ranking of the responses according to their rewards, making it insensitive to any shifts or rescaling. Is there a way we can introduce uncertainty in the ranking itself and how would you justify subtraction for inference-time alignment? I think this aspect is worth discussing in the paper and makes your decision of running a z-score transformation followed by using a global hyperparameter to balance the subtraction of the reward with the uncertainty more convincing.\n\nOther work considered, for example, a multiplicative exponential term to model uncertainty in reward scoring and they motivate it well for policy optimization [1].\n\n[1] Houliston, Sam, et al. \"Uncertainty-penalized direct preference optimization.\" arXiv preprint arXiv:2410.20187 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uuPtdvB26m", "reviewer": "Reviewer_nUU2", "summary": "The paper proposes Best-of-N with pessimism to mitigate potential reward hacking issues in test-time scaling. Specifically, they capture \"caution\" by training a lightweight predictor network to reconstruct reward model features, and the pessimistic score is larger when the prediction error is lower (the reward model is more familiar with the response pattern). Experiment results on GSM8K, Math-500 and BBH-hard demonstrate that their methods are stronger than the Best-of-N baseline.", "questions": "See weaknesses above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed pessimistic score is reasonable, which accounts for the reward model's familiarity with the input. The reward design thus becomes the original reward minus the prediction error, and is clear and easy to implement.\n- Experiment results demonstrate that the proposed pessimistic score is effective when applied to Best-of-N method. What's more, it is also useful for OOD domain test-time scaling.", "weaknesses": "- The authors did not compare with other strong test-time scaling methods, such as self-consistency. So it is unsure whether their method is widely applicable to other test-time scaling methods, as well as state-of-the-art inference time scaling methods.\n- The proposed method is not as lightweight as the authors claimed. In fact, it requires extra training for each reward model. That means for every reward model on a specific task, they need to train a predictor network again based on the specific reward model.\n- The layer number L is a hyperparameter, and it is unclear how they select it. The authors only mentioned in appendix that they use L=10. It is unsure whether this hyperparameter can be applied to other settings / domains / reward models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)