Why is Your Language Model a Poor Implicit Reward Model?
Authors: Noam Razin
Yong Lin
Jiarui Yao
Sanjeev Arora
Venue: ICLR 2026 Poster (2026)
Keywords: Reward Models
Language Models
Generalization
Distribution Shifts
Primary area: foundation or frontier models, including LLMs

Abstract: Reward models are key to language model post-training and inference pipelines. Conveniently, recent work showed that every language model defines an implicit reward model (IM-RM), without requiring any architectural changes. However, such IM-RMs tend to generalize worse, especially out-of-distribution, compared to explicit reward models (EX-RMs) that apply a dedicated linear head over the hidden representations of a language model. The existence of a generalization gap is puzzling, as EX-RMs and IM-RMs are nearly identical. They can be trained using the same data, loss function, and language model, and differ only in how the reward is computed. Toward a fundamental understanding of the implicit biases underlying different reward model types, we investigate the root cause of this gap. Our main finding, backed by theory and experiments, is that IM-RMs rely more heavily on superficial token-level cues. Consequently, they often generalize worse than EX-RMs under token-level distribution shifts, as well as in-distribution. Furthermore, we provide evidence against alternative hypotheses for the generalization gap. Most notably, we challenge the claim that IM-RMs struggle in tasks where generation is harder than verification because they can operate both as a verifier and a generator. Overall, our results highlight that seemingly minor design choices can substantially impact the generalization behavior of reward models.

Published as a conference paper at ICLR 2026
WHY IS YOUR LANGUAGE MODEL A POOR IMPLICIT REWARD MODEL?
Noam Razin!, Yong Lin Jiarui Yaoł, Sanjeev Arora
f Princeton Language and Intelligence, Princeton University
1 University of Illinois Otbana-Champaiger
ABSTRACT
Reward models are key to language model post-training and inference pipelines. Conveniently, recent work showed that every language model defines an implicit reward model (IM-RM), without requiring any architectural changes. However, such IM-RMs tend to generalize worse, especially out-of-distribution, compared to explicit reward models (EX-RMs) that apply dedicated linear head over the hidden representations of a language model. The existence of a generalization gap is puzzling, as EX-RMs and IM MM-RMs nearly identical. They be trained us- ing the same data, loss function, and language model, and differ only in how the reward is computed. Toward fundamental understanding of the implicit biases underlying different reward model types, we investigate the root cause of this gap. Our main finding, backed by theory and experiments, is that IM-RMs rely more heavily on superficial token-level cues. Consequently, they often generalize worse than EX- EX-RMs under token-level distribution shifts, as well as in-distribution. Fur- thermore, we provide evidence against alternative hypotheses for the generaliza- tion gap. Most notably, we challenge the claim that IM-RMs struggle in tasks where generation is harder than verification because they can operate both as a verifier and a generator. Overall, our results highlight that scemingly minor design choices can substantially impact the generalization behavior of reward models.
INTRODUCTION
Language model post-training and inference pipelines often rely on reward models to assess the quality of generated responses (Cobbe al., 2021; Achiam et al., 2023; Dubey et al., 2024; Team al., 2024; Qwen et al., 2024; Snell et al., 2025). Yet, little known about the relative advantages and disadvantages of different reward model types. Two prevalent, nearly identical types are explicir reward models (EX-RMS RMs) (Ouyang et al., 2022) and implicit reward models (IM-RMs) (Rafailov al., 2023). EX-RMs and IM-RMs be trained based on the same language model mo, using the same data and loss function. They differ only in how the reward is computed: EX-RMs apply a linear head over the hidden representation that Te produces for prompt-response pair (x, y), while the reward of an IM-RM is implicitly defined by ιa through In Th (y) x) see Figure 1.
Explicit Reward Model (EX-RM)
Implicit Reward Model (IM-RM)
Linear head over hidden representation of yx
T x] (x)
Language Model T
Language Model T
Prompt x
Response y
Prompt x
Response y
Figure 1: Explicit vs implicit reward models. To compute the reward for prompt-response pair (x, y). an EX-RM applics lincar head to the hidden representation that the language model πa produces for (x, y). In contrast, the reward of IM-RM is implicitly defined by through In. *a(r(x) where & E Ryo is fixed coefficient Tee a reference distribution (ef. Rafailov et al. (2023)).
<x)()/)))

---

Published as a conference paper at ICLR 2026
100%
3.7%
9.3%
20.4%
Training Data: UltraFeedback
In Distriibuuiin
Token-Level Shift
Domain Shift
EX-RM Win
Tle
UltraFondaack
Paraphrased Translater aled UltraFoodback Variants
Math & Cade
IM-RM Win
16.6%
63.0%
87.0%
Figure 2: IM-RMs less robust than EX-RMs token-level distribution shifts, but perform compara- bly or better under domain shifts. trained EX-RMs and IM-RMs on UltaFeedback (Cui al., 2024), using the same initial language models, and evaluated their accuracy in-distribution (UltraFeedback test set), under token-level shifts (three UltraFeedback variants, in which responses were either paraphrased or translated another language), and under domain shifts (two math and one code datascts). Reported are the win-rates, which either the EX-RM or M-RM achieved higher accuracy. If the
Le., the percentage of evaluations accuracics were within 1% of cach other, we considered it a tic. The experiment included three random seeds per configuration and six language models: Gemma-2-2B-IT (Team et al., 2024), Swen-2.5--1.5B-mmms Qwen-2.5-3B-Insee (Qwen et al., 2024), Llama-3,22 2- IB--nruucttt Llama-3.2-3 3.33.-.-nnnt,,,,cc,,,,--- and Llama-3.1--0 Instruct (Duhey al., 2024). See Section 5.2 for additional details.
Despite the vast similarity of EX-RMs and IM-RMs, prior work (Lin et al., 2024; Lambert et al., 2025; Swamy et al., 2025) observed empirically that IM-RMs tend to generalize worse, especially out-of-distribution,
measured by accuracy in ranking candidate responses. The existence of generalization gap is puzzling. Why would a seemingly minor difference in how the reward is computed substantially affect the resulting accuracy of reward model?
Toward fundamental understanding of the implicit biases underlying different reward model types, investigate the root cause for the generalization gap between EX-RMs and IM-RMs. Our main finding, established through theory and experiments, is that IM-RMs rely more heavily on superficial token-level cues. As result, IM-RMs typically generalize worse than EX-RMs token-level distribution shifts (i.e., to responses that are semantically similar to in-distribution responses, but have different surface forms), as well as in-distribution. On the other hand, when subject to domain shifts, IM-RMs can perform comparably to or better than EX-RMs see Figures 2 and 5.
Before arriving at this conclusion, we first consider an alternative hypothesis for the generalization gap, alluded to in the literature (cf. Dong et al. (2024); Singhal et al. (2024)): IM-RMs are harder to gap due to their dual role as verifier and generator
learn in tasks with a (Section 3). Specifically, in tasks where responses can be categorized into correct and incorrect, an IM-MM is trained not only to assign a high reward to correct responses, but also to generate them via its underlying language model. If generating correct responses is harder than verifying their correctness, then the (verification) accuracy of IM-RMs should intuitively lag behind that of EX-RMs, which need only verify responses. However, we challenge this intuitive argument by proving that learning to verify with IM-RMs does not require learning to generate. Experiments on a Hamiltonian cycle verification task corroborate our theory.
Then, to identify what drives the generalization gap between EX-RMs and IM-RMs, we theoreti- cally characterize their learning dynamics, i.e., the evolution of rewards during gradient-based train- ing (Section 4.1). Our analysis reveals that the learning dynamics of EX- RMS depends on responses primarily through their hidden representations, whereas IM-RMs are more sensitive to the specific tokens appearing in the responses. In particular, for IM-RMs, increasing the reward of response may not affect, or even decrease, the reward of a semantically similar response that consists of dif- ferent tokens. This leads to our conclusion that IM-RMs often underperform EX-RMs since they rely more strongly on superficial token-level cues. We further substantiate this claim: (i) thcoret- ically, by providing settings in which IM-RMs provably fail to generalize to unseen tokens, while EX-RMs generalize successfully when hidden representations are well-structured (Appendix B), and (ii) empirically across controlled and real-world settings (Section 5).
Overall, our results highlight that seemingly minor design choices can have an outsized effect on how reward models generalize. We hope insights from this work will spur further research into the implicit biases of different reward model types and facilitate enhancing their robustness.
Related work. We discuss related work throughout and defer extended account to Appendix C.
2

---

Published as a conference paper at ICLR 2026
2 PRELIMINARIES
Let V be a finite vocabulary of tokens and V denote the set of all finite-length token sequences. Language models can be decomposed into two parts. First, neural network backbone that in- takes a sequence of tokens v E V and produces a hidden representation hy E R (e.g., Trans- former (Vaswani et al., 2017)). Second, an umembedding matrix U E RM that converts the hidden representation into logits for the next-token distribution. Given a prompt V*, language model Ta assigns probabilities to responses y E V autoregressively:
< = y
To (v III by softmax (Uhx.ycx)y"
where l stands for the language model's parameters (i.e., includes the parameters of the neural network backbone and the unembedding matrix), ve and yı denote the first k - tokens and kth token of respectively, and softmax (z). == cxp(z=)/ Ewey exp(z.) for 2 2 ERI.
2.1 REWARD MODELS
Reward models are typically initialized from a preexisting language model vo and trained predict scalar reward that indicates the quality of response y to prompt x. Two prevalent reward model types are explicit reward models (EX-RMs) (Ouyang et al., 2022) and implicit reward models (IM- RMs) (Rafailov et al., 2023). As detailed below, EX-RMs and IM-RMs are almost identical. They are trained using the same data, loss function, and language model Ty. and differ only how the reward is computed based Ty. This work is devoted to understanding why, despite these vast similarities, EX-RMs and IM-RMs generalize differently.
Explicit reward model (EX-RM), To compute the reward for prompt-response pair (x, y), an EX-RM applies a linear head u E TR. over the hidden representation hx.y that To produces:
reax yx (u, hx,y)
(I)
where OEX stands for the trainable parameters of the EX-RM (i.e., it includes the parameters of the neural network backbone and the linear head).
Implicit reward model (IM-RM). shown in Rafailov et al. (2023), every language model ma defines an IM-RM through the log probabilities that it assigns to responses:
To (x, y) = 31n Tes (y|x)
Teef (ylx)
where OIM 0 denotes the trainable parameters of the IM-RM, 3 E R20 is fixed coefficient, and the reference distribution Feet is canonically the language model from which the IM-RM was initial- ized. Note that besides assigning rewards, an IM-RM can generate responses via TeMM- Moreover, increasing the reward of a response entails increasing its probability under TeiM
Training objective. Let Dj be training set containing preferences (x, yT), where is prompt, y is chosen response to x, and y is a rejected response to x. EX-RMs and IM-RMs usually trained by minimizing Bradley- log- ligeliiod loss (Bradley and Terry, 1952):
C(r)
D Eisy's y JeD, In o (r(x, y r(x,y")
where r : V* x V + F. can be either Toex or Tomx and a: RR + [0, 1] denotes the sigmoid function.
Outcome vs process rewards. In certain domains, such as math and reasoning, reward models have been used for providing feedback on intermediate steps of a response (Uesato et al., 2022; Lightmnn evaluate partial responses, the former
al., 2024). Both EX-RMs and IM-RMs can be adapted by applying the linear head over the hidden representation of each intermediate step and the latter by using the conditional log probabilities of these steps. For conciseness, we focus on settings where the reward is assigned to complete responses.
Generative reward model. A nascent reward model variant, which we refer to as explicit generative reward model (EX-GRM) (Zhang et al., 2025), directly asks the language model Te to verify whether y is a good response to x. Then, the probability assigned to the token Yes is taken as the reward. For brevity, focus RM-RMs and IM-RMs in the main text and defer an extension of our theoretical and empirical analyses for EX-GRMs to Appendices D and F, respectively. Notably, we find that the main conclusions stated for EX-RMs hold for EX-GRMs as well.
3

---

Published as a conference paper at ICLR 2026
2.2 MEASURING GENERALIZATION VIA ACCURACY
In accordance with Lin et al. (2024); Lambert et al. (2025); Zhou et al. (2025); Frick et al. (2025); Liu et al. (2025), measure generalization via the accuracy of reward model in ranking responses over preference data unseen in training.
Definition 1. For finite set S containing preferences yx,y*, y+, y where x is prompt, y is chosen response, and y is a rejected response, the accuracy of r V x V + R over is:
accs [8] Eix
yx JES 1 [rxx,x**) r(x,y>] + [x(,y*)) = r yx,,y
where 1[:] is an indicator function. Note that the maximal accuracy is one and the minimal is zero
3 ARE IM-RMS HARDER TO LEARN IN TASKS WITH A GENERATION- VERIFICATION GAP?
A potential explanation for why IM-RMs often underperform EX-RMs, alluded to in the literature (cf. Dong et al. (2024); Singhal et al. (2024)), is that IM-RMs are harder to learn in tasks with a generationn-verifcaatio gap. Namely, in tasks where responses can be categorized into correct and incorrect, an IM-RM is trained not only to assign a high reward to correct responses, but also to generate them via its underlying language model. Thus, if generating correct responses is harder than verifying their correctness in given task, then the accuracy of IM-RMs should intuitively fall below that of EX-RMs, which need only verify responses.
We prove that this intuitive explanation is flawed learning to verify with IM-RMs does not require learning to generate (Section 3.1). Experiments on Hamiltonian cycle verification task, which is widely believed to exhibit ceneration-verificaatio gap (Arora and Barak, 2009), demonstrate that IM-RMs learn to accurately verify such cycles without being able to generate them (Section 3.2).
3.1 THEORY: LEARNING TO VERIFY DOES NOT REQUIRE LEARNING TO GENERATE
Consider task defined by set of valid prompts X' C V and a function C that maps every prompt E X' to a set of correct responses C(x) S V". Concretely, X' can consist of math problems, with C(x) containing the correct solutions of problem x. A prompt x can also describe the input to some algorithmic task. For example, if the task is to find Hamiltonian cycles graph, each describes a graph and each response in C(x) encodes Hamiltonian cycle (Section 3.2 presents experiments over this task).
In this context, it is natural to say that reward model is verifier for the task (X, if it assigns non-negligibly higher rewards to correct responses relative to incorrect ones.
Definition 2. A reward model r V V - R is verifier with margin ō E R20 for the task
(X' C) if for all x E X. y E C(x), and y E V* C(x):
r(x,y+) 2 r(x,
Note that if r is a verifier for (X' (), then it achieves perfect accuracy (Definition 1) over all evalu- ation sets that contain preferences (x, y y),, where x E y+ E C(x), and y E V \ C(x).
Theorem below establishes that, for an IM-RM to be a verifier, the probability that its underlying language model assigns correct responses needs grow by at most a constant multiplicative factor relative to the initial reference distribution Tref. In particular, if Nee assigns low probability to correct responses, then an IM-RM can accurately verify correct responses even if unable to generate them. Thus, the hypothesis that IM-RMs struggle because they need to learn to generate, as opposed to just verify, does not explain the generalization gap between EX-RMs and IM-RMs.
We further formalize this argument through the notion of an efficient generator (Definition 3). A distribution π is an efficient generator if the probability that assigns to correct responses decays at most polynomially with the prompt length |x|. The rationale behind this definition is that obtaining a correct response from an efficient generator requires, with high probability, only a number of samples polynomial in |x], which often corresponds to task complexity (e.g., the size of a graph in the task of finding Hamiltonian cycles). Corollary shows that, if Rref is not an efficient generator, then IM-RM does not need to be efficient generator in order be verifier.
4

---

Published as a conference paper at ICLR 2026
Hamiltonian Cycle Verification
Hamiltonian Cycle Generation
Prompt X
y
EX-RM
IM-RM
Train
1
Accuracy
IM-RM
Test
0.980
0.993
Accuracy
3 + 5
5
Correct
0
Generations
+1 Yes/No o Is (x) > yxx,y")?
Is Hamiltonian cycle? +1 Yes/No 0
Figure 3: Learning to verify with IM-RMs does not require learning to generate. trained EX-RMs and IM-RMs to solve a Hamiltonian cycle verification task, based on the Pythia- language model. Each prompt in the dataset describes an undirected graph and the chosen and rejected responses are permutations of vertices. The chosen responses form Hamiltonian cycles in their respective graphs. while the rejected responses do not (see Appendix G.I for further dctails). accordance with our theory (Section 3.1), although IM-RMs are unable to generate even a single correct Hamiltonian eycle for graphs in the training or test sets, they accurately distinguish between chosen and rejected responses, slightly outperforming EX-RMs. Values in the table are means across three random seeds (slandard deviation was under 0.008 in all cases).
Theorem 1. Let TIM be the IM-RM induced by a distribution over token sequences, i.e., (x, (In (y(xx) - ( ()) for x,y E V*, 3 Ry0, and reference distribution Tref. Then, TIM can be verifier with margin ó E for the task (X, C) (Definition 2) even if for all prompts X
π (C(x)|xx) S (Cex((()) exp(6/3).
That is, for all prompis, the probability of v generating a correct response is greater than that of Teel by most constant multiplicatiiv factor.
Proof sketch (full proof in Appendix E.I). The proof is by construction. We define a distribution m such that the IM-RM it induces is verifier for the task (X,C) with an exact margin of 6. Then, directly upper bound the probability that π assigns to correct responses.
Definition 3. We say that distribution w token sequences is efficient generator for the task (X,C) C) if there exist
and a E E>o such that π (C(x)lx) 2 a -|x for all x E X.
Corollary 1. Under the notation of Theorem suppose that Teef not an efficient generator for the task (X, C) (Definition 3). Then, for any 8 E R>s- the IM-RU DM-RM rIM can be verifier with margin 8 for (X, C) (Definition 2) even if the underlying distribution n is not an efficient generator for (X, (X.)).
Proving Corollary based on Theorem is straightforward see Appendix E.2.
3.2 EXPERIMENTS: HAMILTONIAN CYCLE VERIFICATION
We corroborate the analysis of Section 3.1 by empirically demonstrating that, in tasks where gen- eration is harder than verification, IM-RMs can learn to verify comparably or better than EX-RMs, without learning to generate. To avoid confounding factors, we focus a synthetic Hamiltonian cycle verification task which, unless P P NP, exhibits a generation--vrrifcattion verification gap.
Setting. We created a preference dataset in which every prompt describes an undirected graph that contains at least one Hamiltonian cycle and the chosen and rejected responses are permutations of vertices. Chosen responses form Hamiltonian eycles in their respective graphs, whereas the re- jected responses do not. We then trained EX-RMs and IM-RMs based on the Pythia- 1B language model (Biderman et al., 2023) and evaluated their accuracy. We also evaluated the ability of IM-RMs to generate Hamiltonian cycles. See Appendix G.1 for additional details.
Results. Figure illustrates the experimental setup and reports the results. Confirming our theory (Section 3.1), IM-RMs are able to accurately verify responses, i.e., achieve perfect accuracy on the training set and near- searfeerfect on the test set, while being unable to generate even a single correct Hamiltonian cycle for graphs in the training test sets. This showcases that the lower accuracy that IM-RMs often achieve compared to EX-RMs (Lin et al., 2024; Lambert et al., 2025; Swamy et al., 2025) does not stem from IM-RMs needing learn generate in order to verify.
5

---

Published as a conference paper at ICLR 2026
4 THEORY: IM-RMS RELY ON TOKEN-LEVEL CUES WHILE EX-RMS
GENERALIZE VIA HIDDEN REPRESENTATIONS
To identify what causes the generalization gap between EX-RMs and IM-RMs, we analyze their learning dynamics. Specifically, we characterize how the reward assigned to prompt-response pair evolves during gradient- training (Section 4.1). The characterization suggests that IM-RMs often generalize worse than EX-RMs since they rely more heavily on superficial token-level cues. We further support this claim: (i) theoretically, by providing a (simplified) setting in which IM- RMs provably fail to generalize to unseen tokens, whereas EX-RMs can generalize when hidden representations are well-structured (Appendix B), and (ii) empirically, by showing that IM-RMs are less robust to token-level shifts, but perform comparably or better under domain shifts (Section 5).
4.1 LEARNING DYNAMICS
We examine how performing a gradient update on the training example (x,y+ ,y") E Df influ- ences the reward assigned to an unseen prompt-response pair (x, i.e.:
Aro == To Wown-ooe...tt........o. ( 8) re (x.
where E Ry0 is a learning rate, fo(x. loss over (x, y y"). and @ stands for either Bgx or 9IM. We note that analogous approaches have been valuable for studying the effects of language model post-training (Im and Li, 2025; Razin et al., 2025a; Ren and Sutherland, 2025). By a Taylor approximation of ra reward can be expressed
-
ro yx.y denotes the
(x( (x, around 0, the change in
Are (x, -1 ( (rr((x y), Vlo(x,yt, y O(1)).
Thus, up to second order terms in the learning rate n, which is commonly small for reward model training (Liu et al., 2024; Malik et al., 2025), the change in reward is determined by the inner product of the reward and loss gradients. Below, characterize this inner product for EX-RMs and IM-RMs. Motivated by the fact that reward models have achieved competitive performance when fixing the backbone that produces hidden representations (Wang et al., 2024a), we assume that hidden representations are not updated during training." Yet, as Section 5 verifies empirically, the implications of our analysis apply also when all reward model parameters are learned (we do not fix the hidden representations in our experiments).
Assumption 1. Hidden representations are fixed during training: only the linear head u for EX-RMs and unembedding matrix U for IM-MMs are updated (i.e., 0gx and lIM = U).
EX-RM dynamics. For EX-RMs, the change in reward is given by (derivation in Appendix E.3):
Aroex (8, (hx.y: hxyy hx.y-) (0xx)
where g(0xx) (TBEx (x,y ) - (x, )) > O. As Equation (4) shows, TREx (x increases when hx,y is more closely aligned with hy.y- than with hx.y- In particular, the change in reward depends on responses only through their hidden representations. Consequently, the extent to which an EX-RM generalizes to unseen prompt-respons. pairs is largely determined by the structure of the hidden representations, which are produced by pretrained (and sometimes also post-trained) language model. Since these representations are known to encode semantics (Zou et al., 2023; Park et al., 2024). this suggests that EX-RMs can generalize to unseen responses even if they consist of entirely different tokens from responses in the training set.
'An analogous learning dynamics analysis be conducted without assuming fixed hidden representations (Assumption 1). In that case, the resulting dynamics remain the same, up additive terms introduced by the update to hidden representations. These terms are less interpretable because they depend on the specific neural network architecture used for producing hidden representations. Nonetheless. the close match between our thcoretical predictions and the experiments in Section 5, where hidden representations are not fixed, indicates that the effect of these additional terms does not counteract the mechanisms identify under Assumption 1.

---

Published as a conference paper at ICLR 2026
IM-RM dynamics. For IM-RMs, the change in reward is more complex and is given by (derivation in Appendix E.4; adapted from Theorem of Razin et al. (2025a)):
191 lyt
ly
ArbM (x,
Paalyy) (bxgee h Axyt
Pmaly))
(bx bxved
1=1
(0 + (1)²)
(5)
where g(0mM) == ( (x, r0is (x, y+) and the coefficient (v.(() E [- 2). for v E {y y ). determined by the tokens yk. VI, and corresponding next-token distributions:
(v) 15x+ vi] Thp (y&|x, v=I) This (vi y<k) + (Te (-1 y<R), ToiM (-|x, v=())
In contrast to EX-RMs, the change in reward for IM-RMs depends on the specific tokens that appear in y. yt, and y- opposed just their hidden representations. This dependence is introduced by the coefficients (y+ and px, ). which can be positive or negative. Focusing on (x. (the analysis for (y is analogous), we distinguish between two cases. If yı: yi' : then (x+)(y+) is positive since it can be written as (y+) = (1 Tesa (Fn|x, y 21) - Tena (y 8x,))))) + (< (1|x, In this case, the term Pk. +) ( hxy ) has an effect analogous to (xx.) ) from the dynamics of EX-RMs (Equation (4)): it increases the reward of (x.5) if the hidden representation of (X.5) is aligned with that of (x, y+). How- be negative. In this case the effect is oppo- site: the corresponding term decreases the reward of (x. if its hidden representation is aligned with that of yx, Notably, when yR A yt the coefficient (y+ consists of three terms: (No (-|&, y <k),, To (-1x, (()))) Tas (y& |x, y+t). and -oo (yr |x, Fck). The first term is pos- itive and measures the agreement between the next-token distributions corresponding to the contexts of Ya and yit The latter two terms contribute negatively, and their magnitude large when yı is probable under the context of yi and vice versa. As a result, Pk. (y+) is likely to be negative when Yk and y are tokens that appear in similar contexts.
EvEV((Pe) Thys y Y
ever, if y& o yi. then the coefficient px,
Since hidden representations often encode semantics, the above implies that the learning dynamics of an IM-RM may inadvertently decrease the reward of responses that are semantically similar to chosen responses in the training set, and increase the reward of those similar to rejected responses, their tokens have little overlap. This suggests that the generalization gap between EX-RMr and IM-RMs may stem from the latter being less robust to superficial token-level shifts. We support this prospect theoretically in Appendix B, by providing a concrete (simplified) setting in which IM-RMs provably generalize worse than EX-RMs, and empirically in Section 5.
Relation to prior work. The learning dynamics of IM-RMs was previously analyzed in Razin et al. (2025a); Im and Li (2025), but for other purposes. Specifically, our work focuses generalization to unseen responses, whereas Razin et al. (2025a) studied an optimization issue and Im and Li (2025) considered generalization across prompts when responses seen in training and evaluation the same. See Appendix C for further details.
EMPIRICAL DEMONSTRATION
Our theory (Section 4) indicates that IM-RMs are more prone than EX-RMs to overfitting superfi- cial token-level cues. In this section, we verify that this conclusion bears out in practice. Namely, in both controlled (Section 5. and real-world (Section 5.2) settings, we show that IM-RMs gen- eralize worse than EX-RMs under token-level distribution shifts (e.g., paraphrasing), and often in- distribution, yet perform comparably or better under domain shifts. The experiments are based on language models of up to 8B scale from different families: Pythia (Biderman et al., 2023), Gemma-2 (Team et al., 2024), Qwen-2.5 (Qwen et al., 2024), and Llama-3 (Dubey et al., 2024). For brevity, we defer to Appendices and G some experiments and implementation details, respectively.
5.1 CONTROLLED EXPERIMENTS: TOKEN- SHIFT
Setting, For our controlled experiments, we considered prompts from the Persona dataset (Perez et al., 2022), which ask language model whether it agrees or disagrees with given statement.
7

---

Published as a conference paper at ICLR 2026
Prompt
following statement something you would say?
Accuracy
Responses
Prompts
EX-RM
IM-RM
Training: Original Respunses
Evaluation: Paraphrased Responses
y
y
Original
Train
1
Test
1
truthful reply yes
response would certainly
Train
0.022
Paraphrased
Test
0.019
Considering the statement, say
lean toward not really
Figure 4: IM-RMs fail to generalize simple token-level distribution shift, while EX-RMs generalize perfectly. We trained EX-RMs and IM-RMs on prompts from the Persona dataset (Perez et al., 2022). Chosen responses expressed agreement with the prompts, whereas rejected responses expressed disagreement. During evaluation, we included paraphrased versions of the original responses (figure includes exemplar responses). In line with our analysis (Section 4), IM-RMs
extremely inaccurate over paraphrased responses, whereas EX-RMs achieve perfect accuracy. The experiments were based on four language models: Pythia-IB, Qwen- 2.5-1.5B-Imm 158-Instruc,, Llama-3.2-1P IB, and Llama-3.2- 3.---..-21111trreee IB-Instruct. Values the table are means across the models and three random seeds (standard deviation was below 0.04 in all cases).
We manually wrote four chosen responses that express agreement and four rejected responses that express disagreement. We then trained EX-RMs and IM-RMs, using the same initial language mod- els (Pythia-1B, Qwen-2.5-1 1..--2..5115BBInsttrre their accuracy over the original responses and paraphrased versions of them, i.e., responses that similar in meaning but consist of different tokens. See Appendix G.2 for further details.
-lama-3.2-1B, and Llama-3.2-1 -atmaa..-----nnttttee IB-Instruct), and evaluated
Results: IM-RMs fail to generalize to paraphrased responses. Figure 4 illustrates the experi- mental setup and reports the results. As our theory suggests (Section 4), despite achieving perfect accuracy over the original responses. IM-RMs achieve near-zero accuracy over the paraphrased sponses. This reveals that, for IM-RMs, maximizing reward difference between chosen and rejected responses can inadvertently have an opposite effect on paraphrased responses. In contrast, EX-RMs generalize perfectly to the paraphrased responses.
5.2
REAL-WORLD EXPERIMENTS: TOKEN-LEVER AND DOMAIN SHIFTS
5.2. SETTING
We compared the generalization of EX-RMs and IM-RMs in real-world settings by evaluating their accuracy in-distribution, under token-level shifts, and under domain shifts. We ran experiments in two settings general chat and math using six language models ranging in scale from 1B to 8B: Qwen-2.5-3B-Inn
Llama-3.2- 1i---..22---Intrueee Llama- I.2tuet, and Llama-3.1-80 --St-tnsttuct. As specified below, the two settings differ in which dataset was used for training and the categorization of evaluation datasets into in-iitributton, token- level shift, and domain shift. See Appendix G.3 for further details.
Gemma-222B-IT,
General chat. We trained EX-RMs and IM-RMs over UltraFeedback (Cui et al., 2024), based on each language model specified above. In-distribution evaluation was performed over the UltraFeed- back test set. For evaluating robustness to teken-level distribution shifts, we created three variants of the UltraFeedback test set by either paraphrasing, translating to French, or translating to Spanish all responses (via 1PT-4.1). For domain shifts, we used the math and code subsets of Reward- Bench (Lambert al., 2025) and the RewardMATH dataset (Kim et al., 2024).
Math. We used RewardMATH for training and evaluated in-distribution performance a held- out test set. In this setting, the math subset of RewardBench poses a token-level shift while the UltraFeedback variants and code subset of RewardBench pose domain shift.
5.2.2 RESULTS
For the general chat and math settings, respectively, Figures 2 and 5 present the percentage of evalua- tions in which either the EX-RM or the IM-RM achieved a higher accuracy, where we only compare pairs of reward models that were trained from the same initial language model. Furthermore, Table
8

---

Published as a conference paper at ICLR 2026
100%
16.7%
16.7%
40.0%
45.6%
Training Data: RewardMATH
In-Distribution
Token-Level
Domain Shift
EX-RM Win
Tie
RewardMATH
ReweardBenchh Marhh
UltaFeedhuck Variants Code
IM-RM Win
66.6%
14.4%
Figure 5: IM-RMs are less robust than EX-RMs to token-level distribution shifts, but perform compa- rably better under domain shifts. This figure presents the results of an experiment identical that of Figure 2, except that the reward models were trained on the RewardMATH dataset instead of Uttaaeedhacck, Accordingly, the math subset of RewardBench poses a token-level shift while UtraFeedback variants and the code subset of RewardBench pose domain shift. Note that, in this setting, EX-RMs and IM-RMs perform similarly in-distribution since both reach near mear-maximal accuracy (see Table 1).
This table supplements Figures 2 and 5 by reporting the accuracy and absolute (normalized) reward margin over the different evaluation categories. In each row, bold font marks the highest accuracy and absolute reward margin (unless the values are within 0.01 of each other, after taking into account standard deviations). For each reward model and evaluation dataset separately, the absolute reward margin is normalized by the standard deviation of rewards to account for arbitrary differences in scale. Notice that EX-RMs consistently induce higher reward margin, which was shown in Razin et al. (2025h) he beneficial for optimization via reinforcement learning. Values the table are means across the models (six in total) and evaluation datasets, with standard deviation computed based on three random seeds. See Tables 3 and 5 in Appendix F for a per evaluation dataset breakdown of the results.
Table I:
Accuracy
Absolute Reward Margin
Training Data
Evaluation
EX-RM
IM-RM
EX- RX-RM
IM-RM
In- Di-ittttbbutinn
0.752 +: 0.009
0.646 1 0.006
1.014 1 0.023
0.813 :: 0.003
UltraFeedback
Token-Level Shift
0.665 0.005
0.602 1: 0.003
0.976 0.008
0.763 0.003
Domain Shift
0.621 + 0.012
0.720 + O. 0.004
0.807 + 0.008
0.726 = 0.001
in-Diitribution
0.971 + 0.003
0.972 + 0.002
1.602 = 0.011
1.377 0.007
RewardMATH
Token-Level Shift
0.988 0.003
0.515 + 0.007
1.667 = 0.017
1.035 0.011
Domain Shift
0.505 + 0.012
0.517 + 0.001
0.755 = 0.008
0.604 0.001
reports the accuracy and absolute reward margin of reward models for each evaluation category. See Tables 2, 3, 4, and 5 and Figures and 10 in Appendix F for a per evaluation dataset and language model breakdown of the results.
IM-RMs are less robust than EX-RMs token-level distribution shifts. Recall, our theoretical analysis (Section 4) indicates that IM-RMs are more sensitive than EX-RMs to superficial token- level cues. If this is indeed the case, then one would expect IM-RMs to underperform EX-RMs when subject to token-level distribution shifts. On the other hand, EX-RMs should not enjoy distinct advantage under domain shifts. The empirical results match these expectations. 2 Moreover, the in-distribution accuracy of IM-RMs in the general chat setting is consistently lower than that of EX-RMs. We attribute this to in-distribution evaluation being closer to a token-level shift than to a domain shift. Namely, in in-distribution test examples share semantic structure with training examples but take on different surface forms.
EX-RMs induce higher reward margin. Table highlights additional benefit of EX-RM- over IM-RMs: EX-RMs induce a higher absolute reward margin. This was recently shown to yield better optimization landscape for reinforcement learning (Razin et al., 2024b; 2025b).
Although EX-RM RMs are more robust than IM-RMs loken-level distribution shifts, they do suffer drop in accuracy under the general chat setting (as similarly observed in Liu al. (2025); Wu et al. (2025h)).
'Specifically, for a reward model prompt x. and responses y', the absolute reward margin is defined by yr(x,y)= r(x, y') y')l. Razin et al. (2024b; 2025b) proved that low reward variance, which is equivalent to the expected squared reward margin, leads to flat objective landscape that hinders policy gradient optimization.
9

---

Published as a conference paper at ICLR 2026
Evidence against alternative hypotheses. Finally, provide evidence against two alternative can- didate sources for the generalization gap between EX- RMMMs and IM-RMs, aside from the one already ruled out in Section 3. First, the reward of an EX-RM is based on the hidden representation of the whole response, whereas IM-RMs depend also on the hidden representations of intermediate tokens in the response. Intuitively, the hidden representations of intermediate tokens may be misleading since they do not capture the full meaning of the response. Second, the reward of an IM-RM is shifted by the log probability of a reference distribution, which not the case for EX-RMs. Fig- ure 7 in Appendix F.3 demonstrates that these differences do not explain the generalization gap by considering EX-RM EX-RMs trained over the hidden representations of all intermediate tokens and IM-RMs without a reference distribution.
CONCLUSION
Reward models are a key component in language model post-training and inference pipelines. Yet, the comparative advantages and disadvantages of different reward model types are poorly under- stood. In this work, we mostly focused on two prevalent reward model types: explicit reward models (EX-RMs) and implicil reward models (IM-RMs). Through theory and experiments, we established that IM-RMs rely more strongly on superficial token-level cues. Consequently, they typically gen- eralize worse than EX-RMs under token-level distribution shifts, as well as in-distribution. This corroborates and provides an explanation for existing empirical findings on the relative benefits of EX-RMs (Singhal et al., 2024; Lin et al., 2024; Lambert et al., 2025; Swamy et al., 2025). We also provided evidence against an alternative hypothesis, by which the generalization gap between EX-RMs and IM-RMs stems from IM-RMs needing to learn to both generate and verify the quality responses, whereas EX-RMs only need to learn to verify. Overall, our work highlights that seem- ingly minor design choices can substantially impact how reward models generalize. As elaborated below, we hope its insights will motivate research into the implicit biases of different reward model types and facilitate enhancing their robustness.
6.1 LIMITATIONS AND FUTURE WORK
Theoretical analysis. Section included couple of simplifying assumptions. Namely, we assumed that hidden representations are fixed and Theorem 2 (in Appendix B) also required responses to be of length one. Although Section 5 empirically demonstrated that the conclusions of our theory apply when all reward model parameters are trained and responses are of arbitrary length, alleviating these restrictions may yield further insights into how reward models generalize.
Factors influencing generalization. We highlighted one cause for the difference in generalization between EX-RMs and IM-RMs- - a stronger reliance of IM-RMs on token-level cues. However, there are likely additional factors that affect their generalization. In particular, while EX-RMs more robust to token-level shifts, our experiments show that IM-RMs can generalize better under other types of distribution shifts. Investigating whether there are cases in which IM-RMs consis- tently outperform EX-RMs and why is left to future work.
Beyond accuracy. As customary, we primarily measured reward model generalization via accuracy (cf. Lin et al. (2024); Lambert et al. (2025)). While accuracy is an important measure, it is not the only quantity that determines the effectiveness of reward model (Chen al., 2024; Wen et al., 2025; Razin et al., 2025b). Exploring how different reward model types compare across a broader set of evaluation criteria remains a valuable direction for future work.
Reward model types. Our work covers three common reward model types: EX-RMs, IM-RMs, and a generative reward model variant (Appendix D; of. Zhang et al. (2025)). We hope that will encourage studying the implicit biases introduced by additional types, e,g., reward models that provide rewards on intermediate steps of a response (Uesato et al., 2022; Lightman et al., 2024).
ACKNOWLEDGEMENTS
We thank Eshbal Hezroni for aid in preparing illustrative figures and Peter Henderson for providing feedback the manuscript. NR is supported by Princeton Language and Intelligence (PI.I) and the
10

---

Published as a conference paper at ICLR 2026
Zuckerman STEM Leadership Program. SA acknowledges funding from ONR, Schmidt Science, and OpenAl.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 tochnical report. arXiv preprint arXiv:2303.08/7 08774, 2023.
Bo Adler. Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. Nemotron-4 340b technical report. arXiv preprint arXiv:2406./ //.406//7704, /704, 2024.
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang. Niklas Muen- nighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey on data selection for language models. arXiv preprint (xii::2402.168221 2024.
Sanjeev Arora and Boaz Barak. Computational complexity: a modemn approack. Cambridge University Press, 2009.
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. In Advances in Neural Information Processing Systems, 2019.
Ananth Balashankar, Ziteng Sun, Jonathan Berant, Jacob Fisenstein, Michael Collins, Adrian Hutter, Jong Lee, Chirag Nagpal, Flavien Prost, Aradhana Sinha, al. Infalign: Inference-aware language model alignment. In Internatinna Conference on Machine Learning. 2025.
Raphaël Berthier. Incremental learning in diagonal linear networks. Journal of Machine Learning Research, 24(171):1-26, 2023.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' Brien, Eric Hallahan, Mohammad Atlah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning. PMIR, 2023.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(344)322 345, 1952.
Yanjun Chen, Dawei Zhu, Yirong Sun, Xinghao Chen, Wei Zhang, and Xiaoyu Shen. The accuracy paradox in rlhf: When better reward models don't 't yield better language models. arXiv preprint 4410::2410.06554, 2024.
Hlsgg-Huu Chou, Carsten Gieshoff, Johannes Maly, and Holger Rauhut. Gradient descent for deep matrix factorization: Dynamics and implicit bias towards low rank. Applied and Computational Harmonic Analysis, 6815019,,, 2024.
Yinlam Chow, Guy Tennenholtz. Izzeddin Gur, Vincent Zhuang, Bo Dai, Sridhar Thiagarajan. Craig Boutilier, Rishabh Agarwal, Aviral Kumar, and Aleksandra Faust. Inference-awand fine-tuning for oest-of-n sampling in large language models. International Conference on Learning Representations, 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plap- pert, Jerry Tworek, Jacob Hilton, Reichhiro Nakano, al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 11.11.11.1...... 2021.
Edo Cohen-Karlik, Itamar Menuhin-Gruman, Raja Giryes, Nadav Cohen, and Amir Globerson. Learning low dimensional state spaces with overparameterized recurrent neural nets. In International Conference on Learning Representations, 2023.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ulrafeedback: Boosting language models with high-quality feedback. In International Conference on Machine Learning, 2024.
Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano, and Sayak Ray Chowdhury. Active preference optimiza- tion for sample efficient rlhf. arXiv preprint arXiv:2402. 10500, 2024.
Hanze Dong. Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304. 06767, 2023.
11

---

Published as a conference paper at ICLR 2026
Hanze Dong. Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang. Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF workflow: From reward modeling to online RLHF, Transactions on Machine Learning Research, 2024.
Yihan Du, Anna Winnicki, Gal Dalal, Shic Mannor, and R Srikant. Exploration-driven policy optimization in rlhf: Theoretical insights on efficient data utilization. International Conference on Machine Learning. 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:240.221788 2024.
Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios N Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. How to evaluate reward models for rihf. In International Conference on Learning Representations, 2025.
Mudit Gaur, Amrit Singh Bedi, Raghu Pasupathy, and Vaneet Aggarwal. On the global convergence of online rhhf with neural parametrization. arXiv preprint arXiv:2410.15610. 2024.
Gauthier Gidel, Francis Bach, and Simon Lacoste-Julien. Implicit regularization of discrete gradient dynamics in linear neural networks. In Advances in Neural Information Processing Systems, pages 3196 3206, 2019.
Sebastian Goldt, Madhu Advani, Andrew M Saxe, Florent Krzakala, and Lenka Zdeborová. Dynamics of stochastic gradient deseent for two-layer neural networks in the teacher-student sctup. In Advances in Neural Information Processing Systems, pages 6979- 6989, 2019.
Caglar Gulcehre, Tom Le Paine. Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahem, Miaosen Wang, Chenjie Gu, al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv: 08998, 2023.
Suriya Gunasekar, Blake Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization matrix factorization. Advances in neural information processing systems, 2017.
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. In International Conference on Machine Learning 2018.
Jiwoo Hong, Noah Lee, Funki Kim, Guijin Son, Woojin Chung, Aman Gupta, Shao Tang, and James Thome On the robustness of reward models for language model alignment. International Conference on Mackine Learning, 2025.
Jiawei Huang, Bingcong Li, Christoph Dann, and Niao He. Can rlhf be more efficient with imperfect reward models? a policy coverage perspective. arXiv preprint 2rXiv:2502.1955. 2025.
Shawn and Yixuan Li. Can dpo learn diverse human values? theoretical scaling law, Advances Neural Information Processing Systems, 2025.
Kaixuan Ji, Jiafan Hc, and Quanquan Gu. Reinforcement learning from human feedback with active queries. arXiv preprint arXiv:2402.090 2244000,,990,2033
Xiang Ji, Huazheng Wang, Minshuo Chen, Tuo Zhao, and Mengdi Wang. Provable benefits of policy learning from human preferences in contextual bandit problems. arXiv preprint arXiv:2307. 12975, 2023.
Sunghwan Kim, Dongjin Kang. Taeyoon Kwon, Hyungjoo Chae, Jungsoo Won, Dongha Lee, and Jinyoung Yeo. Evaluating robustness of reward models for mathematical reasoning, arXiv preprint arXiv:2410.01729, 2024.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. In Findings of the Association for Computational Linguistics: NAACL, 2025.
Andrew K Lampinen and Surya Ganguli. An analytic thcory of generalization dynamics and transfer lcarning in deep lincar networks. International Conference on Learning Representations, 2019.
Zihao Li, Xiang Ji. Minshuo Chen, and Mengdi Wang. Policy evaluation for reinforcement learning from human feedback: A sample complexity analysis. In International Conference on Artificial Intelligence and Statistics, pages 2737-2745. PML.R, 2024.
12

---

Published as a conference paper at ICLR 2026
Hunter Lightman. Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe, Let's verify step by step. In International Conference on Learning Representations. 2024.
Yong Lin, Skyler Seto, Maartje Ter Hoeve, Katherine Metcalf, Barry-John Theobald, Xuan Wang. Yizhe Zhang. Chen Huang, and Tong Zhang. On the limited gencralization capability of the implicit reward model induced by direct preference optimization. In Findings of the Association for Computational Linguisties: EMNLP 2024. Association for Computational Linguisties, 2024.
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujje He, Chanjje Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. srywork-rewart Bag tricks for reward modeling in llms, arXiv preprint arXiv:2410. 18451, 2024.
Yantao Liu, Zajun Yao, Rui Min, Yixin Cao. Lei Hou, and Juanzi Li. Rm-hench: Renchmarking reward models of language models with subtlety and style. International Conference on Learning Representations, 2025.
Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannanch Hajishirzi. and Nathan Lambert. Rewardbench Advancing reward model evaluation. arXiv preprint arXiv:2506.01957, 2025.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Advances Neural Information Processing Systems, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almoida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances Neural Information Processing Systems, 2022.
Aldo Pacchiano, Aadirupa Saha, and Jonathan Lee. Dueling rl: Reinforcement learning with trajectory prefer- enees. In International conference on artificial intelligence and statistics, pages 6263-6289. PMLR, 2023.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In International Conference on Machine Learning, 2024.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS- W. 2017.
Ethan Perez. Sam Ringer, Kamilè Lukošiūté, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Discovering language model behaviors with model-written
Olsson, Sandipan Kundu, Saurav Kadavath, evaluations. arXiv preprint arXiv:22 22..0921,, 2022.
Garima Pruthi. Frederick Liu. Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 2020.
Tianyi Qiu, Fanzhi Zeng, Jiaming Ji, Dong Yan, Kaile Wang, Jiayi Zhou, Yang Han, Josef Dai. Xuehai Pan, and Yaodong Yang. Reward generalization in rihf: A topological perspective. arXiv preprint arXi::4402.10884, 2024.
Qwen. ;, An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Day- iheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin. Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li. Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xu- ancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv: 151 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Infor- mation Processing Systems, 2023.
Noam Razin and Nadav Cohen. Implicit regularization deep learning may not be explainable by norms. In Advances in Neural Information Processing Systems, 2020.
Noam Razin, Asaf Maman, and Nadav Cohen. Implicit regularization in tensor factorization. International
Conference on Machine Learning, 2021.
Noam Razin, Asaf Maman, and Naday Cohen. Implicit regularization in hicrarchical tensor factorization and
deep convolutional neural networks. International Conference on Machine Learning, 2022.
Noam Razin, Yotam Alexander, Edo Cohen- Raja Giryes, Amir Globerson, and Nadav Cohen. Implicit bias of policy gradient in linear quadratic control: Extrapolation to unseen initial states. In International Conference on Machine Learning, 2024a.
13

---

Published as a conference paper at ICLR 2026
Noam Razin, Hattie Zhou, Omid Saremi, Vimal Thilak, Arwen Bradley, Preetum Nakkiran, Joshua M. Susskind, and Etai Littwin. Vanishing gradients reinforcement finetuning of language models. In termational Conference on Learning Representations, 2024b.
Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. Unintentional unalignment: Likelihood displacement direct preference optimization. In International Conference on Learning Reprexentations. 2025a.
Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason Lee, and Sanjeev Arora. What makes reward model a good teacher? an optimization perspective. In Advances in Neural Information Processing Systems, 2025b.
Yi Ren and Danica J Sutherland. Learning dynamics of Ilm finetuning. In International Conference on Learning Representations. 2025.
Yi Ren, Shangmin Guo, Wonho Bae, and Danica J Sutherland. How prepare your task head for finetuning. International Conference on Learning Representations, 2023.
Andrew M Saxe, James I. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep lincar neural networks. International Conference on Learning Representations, 2014.
Antoine Scheid, Etienne Boursier, Alain Durmus, Michael I Jordan, Pierre Ménard, Eric Moulines, and Michal Valko. Optimal design for reward modeling in rihf. arXiv preprint arXiv:2410. 17055, 2024.
Ruizhe Shi, Minhak Song, Runlong Zhou, Zihan Zhang, Maryam Fazel, and Simon S. Du. Understanding the performance gap in preference learning: A dichotomy of rihf and dpo. arXiv preprint arXiv:2505.19700 2025.
Prasann Singhal, Nathan Lambert, Scott Niekum, Tanya Goyal, and Greg Durrett, D2po: Discriminator-guian guidddd dpo with response evaluation models. In Conference on Language Modeling, 2024.
Yonatan Slutzky, Yotam Alexander, Noam Razin, and Nadav Cohen. The implicit bias of structured state space models can be poisoned with clean labels. In Advances in Neural Information Processing Systems, 2025.
Charlic Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test test-time compute optimally can be more effective than scaling model parameters. In International Conference on Learning Representations, 2025.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1), 2018.
Hao Sun, Yunyi Shen, and Jean-Francois Ton. Rethinking reward modeling in preference- large language model alignment. In International Conference on Learning Representations, 2025.
Gokul Swamy, Sanjiban Choudhury, Wen Sun, Zhiwei Steven Wu, and Andrew Bagnell. roads lead to likelihood: The value of reinforcement learning in fine-tuning, arXiv preprint arXiv:2503.0106, 2025.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma Improving open language models at practical size. arXiv preprint 2408::2408.001 18, 2024.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Sicgel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- outcome- based foedback. arXiv preprint arXiv:2211, 14275, 2022.
Gal Vardi. On the implicit bias in deep-learning algorithms. Communications of the ACM, 66(6):86-93, 2023.
Bhavya Vasudeva, Jung Whan Lee, Vatsal Sharan, and Mahdi Soltanolkotabi. The rich and the simple: On the implicit bias of adam and sgd. arXiv preprint arXiv:2505. 24022, 2025.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances neural information processing systems, 30, 2017.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang- Interpretable preferences via multi- objective reward modeling and mixture-of-experts. In Findings of the Association for Computational Lin- guistics: EMNLP 2024. Association for Computational Linguistics, 2024a.
Yuanhao Wang, Qinghua Liu, and Chi Jin. Is rlhf more difficult than standard rl? theoretical perspective. Advances in Neural Information Processing Systems, 2023.
14

---

Published as a conference paper at ICLR 2026
Zihao Wang, Chirag Nagpal, Jonathan Berant, Jacob Eisenstein, Alex D' Amour, Sanmi Koyejo, and Victor Veitch. Transforming and combining rewards for aligning large language models, International Confer- ence on Machine Learning, 2024b.
Xueru Wen. Lou, Yaojie Lu, Hongyu Lin. Xing Yu, Xinyu Lu. Ben He, Xianpei Han, Debing Zhang, and Le Sun. Rethinking reward model evaluation: Are we barking up the wrong tree? In International Conference on Learning Representations, 2025.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cis- tac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: Mtae-offthe-art natural language processing. arXiv preprint 191.00771, 2019.
Runzhe Wu and Wen Sun. Making ri with preference-based feedback efficient via randomization. In Interna- tional Conference Learning Representations, 2024.
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An em- pirical analysis of compute-optimal inference for problem-solving with language models. In International Conference on Learning Representations, 2025a.
Zhaofeng Wu, Michihiro Yasunaga, Andrew Cohen, Yoon Kim, Asli Celikyilmaz, and Marjan Ghazvininejad. rewordhench: Benchmarking and improving the robustness of reward models with transformed inputs. arXiv preprint arXiv:2503. 1751, 2025b.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influ- cntial data for targeted instruction tuning. International Conference on Machine Learning, 2024.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang. Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In International Conference on Machine Learning, 2024.
Wenhao Zhan, Masatoshi Uchara, Nathan Kallus, Jason D Lec, and Wen Sun. Provable offline preference-baie based reinforcement learning. arXiv preprint arXiv:23005. 14866 2023.
Wenhao Zhan, Masatoshi Uchara, Wen Sun, and Jason Lec. Provable reward-agnostic preference-based reinforcement learning. In International Conference on Learning Representations, 2024.
Chenyang Zhang, Difan Zou, and Yuan Cao. The implicit bias of adam on separable data. Advances in neural information processing systems, 2024.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mchran Kazemi, Aviral Kumar, and Rishabh Agarwal. Genera- tive verifiers: Reward modeling as next-token prediction. In International Conference on Learning Repre- sentations, 2025.
Enyu Zhou, Guodong Zheng, Binghai Wang, Zhiheng Xi, Shihan Dou, Rong Bao, Wei Shen, Limao Xiong, Jessica Fan, Yurong Mou, al. Rmb: Comprchensively benchmarking reward models in Ilm alignment. In International Conference on Learning Reprexentations, 2025.
Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedback from pairwise k-wise comparisons. International Conference Machine Learning, pages 43037-43067. PMIRR 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 000p:/999.0993 2019.
Andy Zou, Long Phan, Sarah Chen, James Camphell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Amn- Aun-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprini arXiv:2310.01405 2023.
15

---

Published as a conference paper at ICLR 2026
A REPRODUCIBILITY STATEMENT
Our theoretical results (Sections 3.1 and 4 and Appendices B and D) are accompanied by complete proof' in Appendix E. All information required to reproduce the experiments is provided in Sec- tions 3.2 and and Appendix G. Our code implementation is available https:: /tttp::/gittubb.. com/ princeton- p1 /eee/exvvvv/ va- mrm.
B GENERALIZATION GAP BETWEEN EX-RMS AND IM-RMS
The goal of this appendix is to provide concrete setting in which IM-RMs provably generalize worse than EX-RMs, due to the stronger reliance on token-level cues (identified in Section 4.1). Alongside assuming that the hidden representations are fixed (Assumption 1), consider the case where responses in the training set are length onc. Furthermore, to ensure a fair comparison between EX-RMs and IM-RMs, require that both are able perfectly fit the training set.
Assumption 2. Responses in the training set Df are of length one.
Assumption 3. There exist OEX and IIM such that the corresponding EX-RM and IM-RM achieve perfect accuracy over the training set Dy, i.e., accDy (rowx) ) accp, (ro.u) = 1.
Under these conditions, Theorem 2 establishes that an IM-RM trained via gradient descent does not generalize to unseen tokens - achieves trivial accuracy over any evaluation set containing responses that did not appear in the training set 'D. This inability to generalize occurs regardless of the structure of hidden representations or the initial unembedding matrix. By contrast, an EX-RM generalizes successfully unseen tokens if the hidden representations are well-structured. Namely, let u* E RD be the following max- max-margin separator over hidden representations in Dy:
argmingeRo ||u|| s.t. V (x,y*, .y E DT (u, hx.y+ - -x,y 2 1.
(6)
The EX-RM will rank correctly any pair of responses that u² ranks correctly.
Theorem 2. Suppose we train an EX-RM and an IM-RM via gradient descent over the training
set DJ with learning rate 1) < 2B-2 min 1}, where B the maximal hidden representation norm in DT. i.e., B == m VEDD,,E(xx.xx....... (xx. }) llllhil Denote by O(t + := 0(t)
N ((ra()) the gradient descent iterales, for / 0, where O stands for either OEX 0yM,
IM-RM reference distribution is Tref = (0)a((0) and the loss L is defined in Equation (3). Then, and 3, for all initializations θgx (0), (14((0) and finite evaluation sets De
under Assumptions
that contain preferences (x,y+, y ), in which x E V and yt yy V are responses that do not appear Dr,, the following hold.
Both the EX-RM and IM-RM perfectly fit the training set: That is, limm L(TTEx(1)) =
lime-+00 C(roma(t) and lime-+00 acCD, (TDxx(()) limm aceD, aCCD,(Tonaat (Tona(t)
The IM-RM fails to generalize to unseen tokens: aceD, (Tonaatt) 0.5 all t 0.
The EX-RM can generalize via hidden representations: Let the max-margin separator defined in Equation (6). Then, there exists time to 2 O such that for all ' lo:
1(xxyt,y E De :e::("", (u", bxy (u", bx }
accD, (reEx(0))
iDel
Proof sketch (full proof in Appendix E.6). With fixed hidden representations, the loss of an EX-RM and the loss of an IM-RM can be framed as logistic regression problems over different input spaces. Fitting of the training set thus follows by standard convex optimization results. We then specialize the learning dynamics of an IM-RM (Equation (5)) to the case of single-token responses and show that the difference between the rewards of two unseen tokens is constant through training. This implies that accp, (Fo)g((t) remains at its initial trivial value of 0.5. Lastly, by applying the seminal result of Soudry et al. (2018), we get that the linear head of the EX-RM converges in direction to u*. This yields the guarantee accD, (F))x(t)).
16

---

Published as a conference paper at ICLR 2026
C
RELATED WORK
Reward models for language model post-training and inference. In real-world applications, it is rarely feasible to evaluate the quality generated responses via rule-based rewards. As result, reward models have been extensively used in the language model ecosystem for training via reinforcement learning (Ziegler et al., 2019; Ouyang et al., 2022; Achiam al., 2023; Dubey al., 2024; Qwen et al., 2024; Team et al., 2024), labeling preferences in direct alignment algorithms (Dong et al., 2024; Meng et al., 2024; Adler et al., 2024), rejection sampling (Gulcehre et al., 2023; Dong et al., 2023), data filtering (Dubey et al., 2024; Qwen et al., 2024; Albalak et al., 2024), and inference-time scaling (Cobbe et al., 2021; Wu et al., 2025a; Snell et al., 2025).
Analyses of reward models. Prior analyses mostly bounded the sample complexity for estimating a ground truth reward, under various technical conditions (Pacchiano et al., 2023; Zhu et al., 2023; Wang et al., 2023; Zhan et al., 2023; Ji et al., 2023; Du et al., 2024; Xiong et al., 2024; Qiu et al., 2024; Das et al., 2024; Ji et al., 2024; Scheid et al., 2024; Gaur et al., 2024; Zhan et al., 2024; Wu and Sun, 2024; Li et al., 2024; Huang al., 2025; Sun al., 2025). An additional line of research considered properties of a reward model that benefit robustness (Wang et al., 2024b; Hong et al., 2025), compatibility with a given inference procedure (Chow et al., 2025; Balashankar al., 2025), or the optimization landscape for reinforcement learning (Razin et al., 2024b; 2025b). However, the works mentioned above do not account for the difference between reward model types or the effect of their particular parameterizations on generalization, which is the goal of this study.
Most relevant in our context are Im and Li (2025); Razin et al. (2025a); Shi et al. (2025). Similarly to Section 4.1, Im and Li (2025) and Razin et al. (2025a) analyzed the learning dynamics of IM- RMs, but for other purposes. Specifically, our work focuses on generalization, whereas Razin et al. (2025a) addressed an optimization issue that causes the reward assigned decrease. Regarding Im and Li (2025), under conditions similar to those of Theorem 2, they proved that IM-RMs can generalize well to unseen prompts if the responses used for training and evaluation are the same. In contrast, Theorem 2 establishes that IM-RMs fail to generalize when the evaluation responses do not appear in the training set more realistic scenario. Lastly, Shi et al. (2025) constructed a setting in which EX-RMs enjoy a better sample complexity than IM-RMs (Section 4 therein). Though, their result requires nonstandard reward model parameterizations and a reward estimation method tailored to specific ground truth reward. While our analysis also operates under simplifying assumptions, it identifies a cause for the generalization gap observed in practice between EX-RMs and IM-RMs, as we extensively verify empirically (Section 5).
chosen responses to
DPO vs RLHE The question of why IM-RMs often generalize worse than EX-RMs is closely re- lated to, yet distinct from, comparisons of DPO (Rafailov et al., 2023) and RLHF (Ouyang et al., 2022). Specifically,, DPO corresponds to the language model underlying an IM-RM and RLHF refers to first training EX-RM and then optimizing a language model based on it. Swamy et al. (2025) argued that DPO usually underperforms RLHF due to the presence of ceneration-verificatioo gaps in many practical scenarios. Our results (Section 3) show that, while such gaps may underlie performance differences between language models trained with DPO and RLHF, they do not ex- plain the difference in generalization between IM-RMs and EX-RMs (as measured by accuracy). Furthermore, the analysis in Section 4 can be interpreted as suggesting another cause for the per- formance difference between DPO and RLHF: the former suffers from a reliance on superficial token-level cues.
Learning dynamies of neural networks. In Section 4.1, characterized how the reward assigned to prompt-response pairs changes due to a gradient update. Analogous approaches have been valu- able both in theory, for studying the effect of language model post-training (Im and Li, 2025; Razin et al., 2025a; Ren and Sutherland, 2025), and in practice, for identifying mislabeled examples (Pruthi et al., 2020) and developing data selection algorithms (Xia et al., 2024). More broadly, analyzing the trajectory of gradient-base training is fundamental tool in the vast implicit bias literature. There, the focus is typically on understanding why overparameterized neural networks tend to generalize well, despite the existence of parameter assignments that do not (Saxe et al., 2014; Gunasekar et al., 2017: 2018; Soudry et al., 2018; Arora et al., 2019; Gidel et al., 2019; Goldt et al., 2019; Lampinen and Ganguli, 2019; Razin and Cohen, 2020; Razin et al., 2021; 2022; Berthier, 2023; Cohen- Karlik et al., 2023; Ren et al., 2023; Razin et al., 2024a; Chou et al., 2024; Zhang et al., 2024; Slutzky et al.,
2025; Vasudeva et al., 2025). We refer to Vardi (2023) for survey of the field.
17

---

Published as a conference paper at ICLR 2026
D EXPLICIT GENERATIVE REWARD MODELS
A nascent reward model variant, proposed in Zhang et al. (2025), rewards responses by asking a language model Ty to assess their quality. We refer this type of reward models as explicir generative reward models (EX-GRMs). Specifically, for prompt-response pair (x, y), EX-GRMs receive as input / [|x y] E V*, which some textual format that requests the model to verify whether y is good response to x. For example, Zhang et al. (2025) concatenate to (x, y) the suffix "Is the answer correct (Yes/No)?". Then, the reward for (x.y) is taken to be the probability that the underlying language model assigns to the token Yes, Le.:
Too (x. y) = The (Yes) [x, yl).
where 8G 0 denotes the trainable parameters of the EX-GRM.
Instead of the Bradley-Terry log-likelihood loss (Equation (3)), Zhang et al. (2025) suggested an alternative loss for EX-GRMs:"
(EDD - -In Tea C (rea) == [DT -e
Téc (Nol/[x,y I). (7)
In Appendix D.I, extend the analysis of Section 4.1 EX-GRMs. We show that, similarly to EX-RMs, the learning dynamics of EX-GRMs depends on responses primarily through their hidden representations. This suggests that EX-GRMs should also be more robust than MMRMss to token- level distribution shifts. We corroborate this hypothesis empirically in Appendix F.
D.1 LEARNING DYNAMICS
We characterize how performing a gradient update on the training example (x. yt, E DT influences the reward that an EX-GRM assigns to an unseen prompt-response pair (x, y), ie::
Area () == Vea (xyty (x, rea 8).
where 7) E Ryo a learning rate and
Oe yx,y y y ) = In T&G [xx) In Tea (Nol ( I)
denotes the EX-GRM loss over .t,ytty)) By a Taylor approximation of rea (x, 8) around Oc. we may write the change in reward as:
Areo y) = (Vrog (x, (), VCG yx ,y 0(7")
As in Section 4.1, assume that hidden representations are fixed during training (Assumption 1), in which case the trainable parameters of the EX-GRM are 0g = U. where U is the unembedding matrix of Tea. Under this assumption, the change in reward for EX-GRMs is given by (derivation in Appendix E.5):
Areg (x. 9) = Toa (Yes|[[x,))) (z(x*)- (huss,s), b)x +n(y") (hirs.)) hy
(8)
017?),
where the coefficients 7(y*) E [0, 2] and 7(y E [-2, 1] are defined as:
+) T0a (Yes / [xx y Toe (Ycs / x, I1) + (TaG (-) y)), Too (-17[x, y 1))
< = - Tea [No|/
x y)) Tea (oec(Yes (Yes x
(mea [-| y)), Toc (: [[xx, 1))
with Tea (-1z) denoting the vector of probabilities that Je. assigns tokens conditioned on z.
Similarly to EX-EEM- RMs (Equation (4)), and in contrast to IM-RMs (Equation (5)), the change in reward for EX-GRMs depends on y. y+, and y primarily through the hidden representations of the corre- sponding inputs (i.e., h/ hı[x.--), and hx(x.)))). Notably, since 7 f) Z 0, the contribution of
"The input 7 |xx y] can optionally include chain-of-thought tokens. $7hang al. (2025) include additional -X In wag (y+)x) loss term, with À 0, that encourages the model retain its response generation capabilities.
18

---

Published as a conference paper at ICLR 2026
(b((x.)) hr)x)))) mirrors that of (hx.y: bxyy)) in the EX-RM dynamics: it increases the reward of (x, y) when the hidden representations corresponding to (x, y) and (x, y are aligned. The con- tribution of the term involving y(y-) may differ from the analogous term in the EX-RM dynamics since ((y") ) can be positive. Nonetheless, EX-GRMs, like EX-RMs, expected be more robust than IM-RMs to superficial token-level distribution shifts. Appendix F empirically demonstrates that this indeed the case.
E DEFERRED PROOFS
E. PROOF OF THEOREM
For a prompt x E X, we define x(-[x) by:
Z(x) Z(x) (y)
exp(6/3) exxp)00)))
E C(x)
Ty|x))
[(x) Tea (ylx)
y E V C(xx)
where
7(x) = Ey
EC(x) EC(() mr (x) exp(6/3) Σ,
y EV*((()) Troef (y (x)
a normalization constant that ensures n (.|xx) is a valid distribution. The probability that æ assigns to any other sequence of tokens be defined arbitrarily (as long it is consistent with the prob- abilities defined above). Since (x, y) = ((nnTyy|x) - Tref -e (y|x))
for all x E N, E V*,
where F'M is the IM-RM induced by n, we have that:
5 31 ((x)
y VECK C(x)
(x)
1-
y E V" (((x)
Clearly, rIM is verifier with margin for C) since FIM (x, y+) = rIM((x, 8 all e X, y+ C(x), and y E V* C((x).
Now, notice that 7(x) > since it a sum over the probabilities Teee Ty (y|x), for y E V* up to terms corresponding to E C(x) being multiplied by exp(6/3) > 1. As result, for any x X and y E C(x) holds that:
1 Z(x)
T =
Tret (y(x) exp(6/3) S <re((y|x) exp(6/3).
Summing over responses in C(x), we conclude:
=CC())lx) I (xx(xx)))))xxx(///
E.2 PROOF OF COROLLARY 1
By Theorem 1, there exist distribution æ and corresponding IM-RM rIM such that FIM is verifier with margin d I> for (X,C) C). although for all X' holds that:
exp(-8/33 (C(x) I Tref [C()))())
We show that, since Reef is efficient generator for (X,C), the distribution is not efficient generator for (X' C) either. Assume by way of contradiction that this is not the case, i.e., that π is an efficient generator for (X,C), Let & E N and E I be such that (C(x) x) 2 a a- for all x E X. Defining y = a exp(8/33 B), follows that for all x E X:
Trer > expp-pp0))) (C(x))x) exp(-8/3) a -|||- = 7 1x
Le., Teef is efficient generator for (X, C) contradiction.
19

---

Published as a conference paper at ICLR 2026
E.3 DERIVATION EXPLICIT REWARD MODEL LEARNING DYNAMICS (EQUATION (4))
Under Assumption I, the trainable parameters of the EX-RM are 0gx = u. Thus, the loss gradient for (x, yt, y D with respect to OFx is given by:
vleex yx, yy -g((0xx) (Vroxx(x, Vreex (x,
=-9(0xx) (bx,y+ - lx.y-):
where g((eex) follows by:
Oxx (x, = ofreex (x, raxx (x, > 0. Equation (4) then
Arpex (2, Toux VVVE (x.y (8,8) TBEx (8,
uu 1) 1) (x, y y 1xx,) > - (u, hx.9)
(bx.y. hx.y+ hx ng((Fx))
E.4 DERIVATION IMPLICIT REWARD MODEL LEARNING DYNAMICS (EQUATION (5))
Equation (5) follows by steps similar to those used for proving Theorem in Razin et al. (2025a),
where the difference stems from the hidden representations being fixed in case (Assumption 1). In particular, under Assumption 1, the trainable parameters of the IM-RM are lIM = U. Thus, the loss gradient for yx,y+ y DT with respect to flM is given by:
Vone (x. (x,yt, yy) g((ma) (Vrend (x.
Vr y
-g(0mm)) (v3ln
Ters (y+
v3 Tref (y
Tous (y xx)
Trat (y+1x)
V Th (y 1x)) g((0m)) (V In Teis (y (xx)
where ((0M)) = - =- (x, reward gradient for (8 is:
> 0. Furthermore, the ofFeia (x,y Tona (x, y))
Tiss (y x
Vreua () V
3 V Taps (8/x).
Trat (y
Now, for any prompt x' E V and response y' E V":
In TTnA (y' (x) = lv'l
In Tomm (y'cx' y'cr)
(Uy) be In Eeey exp (Un
ly'l Aml (yy - (x1x' xe.
where e, E RI denotes the standard basis vector corresponding to v V and Tona (-xx' y'en) the vector of probabilities that Thm assigns tokens conditioned on (x', y22x)- Plugging this gradient expression into the expressions for Vrews (x. and Vlom (x,y y yields:
(Vrnn (x. y), V (x, yt, .
19
lyt1
Σ
E9 Tonc (- (-1x, (x ycn
eyy
Tome (-1x, y (-xx. b g(fIM))
Oy. - Tom (-1x, Fck) (x
Ox - (1x, x.xi. g(Ona))
191
y
181
ly
pk.ly (beyeer bx ) g(Ona))
().(y"):
(besser bxyk
20

---

Published as a conference paper at ICLR 2026
where for all (y+, . .. k E (I...., 1.1., and I (l,...,/v]) Pk.((v) == (eyx Topa ( (-1888 <<),, ev: Tona (-1x, v<l))
= 1[yx = vı] - This (yx|x, Ver) Tess (vi x, 8 8ck) + (Tees (-1,, Tonc ver))
Equation (5) then follows by the above and the fact that:
Arona (x,9) = 7 (Vress (), Vlana (x, ,y yy Or)))
Lastly, to see that (v.((v) resides within |-2, 2] for all v E V", k E {I...., /v],, notice that:
and E
[pk,(v)]
= (og, - Toin (- F<k), - (001,11x,, v<1)
$ llyy -Tera (-12, 8 &<cn</l lvv Thmn To (-1x, Veiilloc
< 2
2,
where ||-II; and I1- denote the l and too norms, respectively.
E.5 DERIVATION OF EXPLICIT GENERATIVE REWARD MODEL LEARNING DYNAMICS (EQUATION (8))
Under Assumption I, the trainable parameters of the EX-GRM are 0g = U. Recall (Appendix D.1) that the EX-GRM loss over yx,y+.y ,y+ E DT defined as:
Se yx. ,y) == - In Tea (oea(Yeee (Yes| (x, yx,,' in Te [xx I).
The loss gradient for yx, y-) DT with respect to Og is therefore given by:
V G tg (x, yt y = v In Tee (Yes y - V In Tao y
- -evrss Tec [|x II) hr|x.y+ (eNo Tea (-11[x, D) h)
where eYee E R and eNo E RII are the standard basis vectors corresponding to the tokens Yes and No, respectively, and Nea (-I[[x, y and nea T (-14(x. 1) are the vectors of probabilities that Tee assigns to tokens conditioned / /xxy" +1 and 1 [x, y respectivvel
Furthermore, the reward gradient for (x, is:
Vrea (x,
VIOG (Yes [xx vl) Top (Yes [ y)) V In Tos (Yes|
[Yes) yl) (eyes (- h
Thus:
(Vroa (x. f), -VG (x,y+, .y .y-))
= = = (Ves|/(xx)))
(ev ( (eYes - To yx]][xx+++]]) Txxxt)))))) + naa (Yes| [x) f)) ((eves - (-1[1, 81)) (1)( (eNo Too (-1[[x, ()]) h
= [Yes|/ })) (y+)) (hITss) 1)(x) + 7(y") (hiss)) hIx 1
where the coefficients 7 and (y are given by:
+) ((eYes- TTc (-14 {8, y)), eYos - TBa (x1
Tea (Yes / x, y Tue (Ycs x, t1) + + (TeO (-11 (1 y y1))
>(y ) = (eeess - -Tec (-1 |x, y eNo Toc (-1((x,y
= - (No//[x, Tea (Yes| + (Toa (-17(x,))) &)), (-T(xxx
21

---

Published as a conference paper at ICLR 2026
Equation (8) then follows by the above and the fact that:
Arog (x y) -1 (Vrag (x,)), F), Vege (x,y*, y )) + 01122)
Lastly, to see that 7(y*) E [0.2] and 7(y 7) 1-2, 1], notice that:
7ly = Tea (Yes) 1|x ((),)) - mec ( "1))
Σ (Yes) Tea yT) (x)//xx,
Since (1 Toa YYss) 81) Toa (Yes) y+1) }]] E [0, and
00 Σ (Yes) 40a [v| y) TO (1) / |x, y y+) S Σ Evev The (vl [x) 1,
it follows that 7(y+)= E [0, 2]. Turning our attention to rly"), ), it can be written as:
7ly = - (Nol y)) Tea [x.
Σ
mea (1l
(v|[[x,y]]
NEV
Since the first two terms on the right-hand side are bounded within |-1, 0] and
0 s Σ VEV Tea (vl [[x (1/ S Σney Tua (u]/[1,)))) 81) = 1,
we get that 7(y E 1-
E.6 PROOF OF THEOREM 2
We begin by expressing the loss L (Equation (3)) for T0Ex and reix as logistic regression problems over different input spaces. This is possible since only the linear head OEx u and unembedding matrix IIM are trained (Assumption 1). Let E(a) = Ino((a) In(1 + exp(-a)) be the logistic loss, for I, and define for all (x.y* y Ey:
(x == hx.y! hx E ERD
ΦIM (x, y 3 (ey+h By- hx E RR
where By E R denotes the standard basis vector corresponding to y E V. We can write the loss for EX-RM as:
((TeEx) =
1
E (x.+tyy)eet
(1TExx(x, - reex (x,
[DT
(10)
1
E(xyty-eeD (E, yx ))))
[Dr
This describes a logistic regression problem with respect to OEx and inputs dEx yt, y for (x, ttyy y) DT, whose labels are all positive. On the other hand, for an IM-RM we have:
((rom)) |Drl Σ
y JED, ((ros) (x,+ - -0 (x,y
|DT (xytt JEDy e (Bln ThiM (y (x)
This (y (x)
3h Tref (y xx)
Tref (y+
By Assumption 2, responses in the training set Dyr are of length one. Meaning, y y V for all (x.y*,y y E Dr. Notice that for any y V:
In Tomm (y|x) (Uy, hx) - In Σ Vey exp((U,, hx)),
where Uv denotes the row of corresponding to v. Along with Teef = To (0) this leads to:
L(rosx))
Eieyty (U hx) B (U, (0) Uy hx), [DT Eveyty JED, ((U, xt. (U(0), (I,(x y)))..
DD
(11)
22

---

Published as a conference paper at ICLR 2026
Up constant bias terms, of the form (U()), (I, y yy),) this describes logistic regression problem with respect to fIM = and inputs OIM (x, y yy). for (x,y", y DT, whose labels are all positive.
With Equations (10) and (11) in place, Lemma 2 shows that both losses [(r and ((Ton)) are B² max 1 >msmooth. Furthermore, based Assumption Lemma 3 proves that the corre- sponding logistic regression problems are over linearly separable data. That is, there exist a linear head u E IR" and unembedding matrix Ü E IR
MD such that for all (x.y".y E Dy:
(ú, (Ex y () 00
(U, (IM(X .y > 0.
This implies that inføgx ((rOEx) = inf (((0())) = 0 as one can reduce the loss to be arbitrarily close to zero by scaling up the norms of the linear separators and U (notice that {(a) 2 0 for all a E IR and f(a) + O when a + cc). Next, we rely on these observations to establish the three parts of Theorem 2.
Both the EX-RM and IM-RM perfectly fit the training set. As shown above, C(roex) and [(r0nx) can be formulated logistic regression problems over linearly separable data (Equa- tion (10), Equation (11), and Lemma 3). Since the losses are convex and B² max ( 2 } }sssooot (Lemma 2), standard arguments from the convex optimization literature imply that gradient descent with learning rate < 2B min
minimizes them to their infimal value of zero (e.g., see Lemma 1 in Soudry et (2018)). That is, lim;-+oe [(reex(t) = lim,- (()) = =0.
The fact that the training loss converges to zero directly implies that the accuracy of the EX-RM and M-RM over Dy converges to one. To see it is so, notice that there exists time t' 2 0 such that for all & > 1' it holds that ((reex(1)) < [DTl (x, yt y) E Dy:
In2 and (((8ma(t)
< In2 Hence, for any training example [DT
(x, - TeEx(1) (x,y )) 2,
((ronc(e)(xx,) - To) (x, y )) < 1n
This holds and only (t)x(t) (x,y yt) ))+ > (0) (x,y (x. for all (x,yt, )) E Dy. Thus, accD, (copx((oex(t)) = (Ce = for all t 2 tt, iee, lom aceD, (T)) = lim aceD, (rorv(t)) =.
and (o (x,y+) > Tomalt) (x.y")
The IM-RM fails to generalize to unseen tokens. For any OIM U, the gradient of L(rona) is given by:
v((rn))
[DT Σ (xyy y JEDr (' ((rons(x, - (x, (x, (
|Drl Σ (xy+ty))DD 3E (ronse (x, - -rews (x,y (ey+ -gy h
Notice that for any token y V that does not appear to Uy - the row corresponding to y in U - is zero. This implies that (t) = (0) for any such y E V and all & 2 0. Hence, for all (x, yt yx,++ y responses yt, y - E V do not appear in Dy, we have that:
response in Dy, r. the gradient with respect
De and
because the evaluation
(x, (x, = In Foi (y (x)
- -33 .1 To (y (x)
Keef (y+
fy(xx [x)
Tuae (o (y- [x)
= 3 (Uy (t) - Uy- (t(t), - 3 3(U y (0) Uy-(0), hx)
= =0
from which it follows that (e (x, = (e (-) and aCCD, (Tona(()) = 0.5.
The EX-RM can generalize via hidden representations. By Theorem 3 of Soudry et al. (2018), in logistic regression problems with linearly separable data, gradient descent converges in direction
to the max-margin separator. Invoking Theorem 3 of Soudry et al. (2018) for the EX-RM loss implies that u(t) converges in direction to u" defined in Equation (6), i.e., lim, a(t)/|[((t)) = u" [|| Note that the requirements the learning rate satisfied: it needs to be smaller than
23

---

Published as a conference paper at ICLR 2026
2/a, with denoting the smoothness coefficient of the loss, which in case is B m
(Lemma 2). Thus, for all yx,y",'' ,y E De we get that:
lim 1-000
u(t) bx.y+ bxy )
||u+|| bxyt bx.y>
u
||u(t)]
If (u", bxy++
bx.y-) then there exists a time t' such that for all t::
||(OI|l xx,y+ hxxx ) >>>0 0,
u(l)
and so:
(u(t), bx.y+ bx,y- (t) (x,y") -00x((t) (x.y 0. By defining to to be the maximal such over all (x,y*,y yy)) DD for which
(u", hx hx.y-) 0,
we arrive at the desired conclusion (note that it is possible to take the maximum over such times /' since De is finite). Namely, for all t to the EX-RM rosx(t) accurately ranks at least the responses that u ranks correctly, i.e.:
acCDe (F))x(1)) 2 1(x,yt,y E Dg (u", hx (u", bx )
[Del
E.6.1 AUXILIARY LEMMAS
Lemma 1. Let e: Rz0 denote the logistic loss, i.e., {(a) == - -Ino(a) for a E R, where
a: R + [0, is the sigmoid function. Then, for all E R:
(e" - 4
= Proof. For all a E R. the derivative of the sigmoid satisfies d'(a) = aa))a(-a) Thus, a straight-
forward differentiation of l gives (a) - and:
(a) = (a)a
Noticing that (--a) (a) and that the maximal value of p(1 p) for [0,1] is 1/4, conclude that !e" (a) < 1/4.
Lemma 2. Under the setting of Theorem 2, both the loss L(roux) ) with respect to 0gX and the loss L(rons) with respect to 6iM U are B² max{ 1 -mmoohhh i.e. the spectral norm of their Hessians is bounded by B max{83
Proof. Starting with L(rBEx), by Equation (10) and straightforward computations we can write its Hessian as:
V V
Σ
Φxx(xxy* . )oEx(x,y*, y
[DTl
(x,+t y>et,
Since dEX (x,yt yt y bxyy bx,-. by the triangle inequality (x, .y )ll < 2B for
all (x, y y Dr. Furthermore, by Lemma have that ("(,, . }}} < 1/4.
Thus:
[(ronx)) 4 Eisyty |x y y= < B²
where 11.1|2 denotes the spectral norm and ||-|| denotes the Euclidean norm. Multiplying B by max (,22, 1) can only increase it. Thus, L(roex) is B² max smooohh.
24

---

Published as a conference paper at ICLR 2026
For ((ross) analogous derivation leads to:
1112 4|DTl Eixt JED, yx y y//
Since ΦIM (x,y .y- )- = 3. (cy+hh - -y- )x by the triangle inequality 1 yt, yy v 2B3 for all (x,y*, ,y) E Dr. We therefore have that [(rons) is B max , }-smooth as well:
[ S B² -
Lemma 3. Under the setting of Theorem 2, there exist a linear head E IRD and unembedding matrix U E RDVxD such that for all (x, y+,y) E Dr:
(ū, 11. yt, y) bxy+ hx.y)
(U, .y )) (Uyt -Uy- ,,)) >
where ΦEX and eιd are defined in Equation (9) and U. denotes the row of U corresponding to v, for any token v
Proof. Starting with the EX-RM, by Assumption 3 we know that there exists 0gX u E IRD such that for all (x. y . Dr:
0 < Topx y - Tonx (x,y = (u, bx.+ bxy-) = (xxxx
Thus, = u satisfies the requirement of the lemma.
For the IM-RM, by Assumption 3 we know that there exists OIM = U E RI such that for all
(x. yt, y Dy:
To (y+ xx)
T0 (y x))
0 To Tx (x,y+) TBma (x.y ===
-B1n
Teef (y+(x) (x)
Teet (y (x)
(Uy - Uy-, hx) - B ((U (0) UU (0), hx)
UU - U(0), (x, y
Thus, U := U U()) satisfies the requirement of the lemma.
F ADDITIONAL EXPERIMENTS
F.I HAMILTONIAN CYCLE VERIFICATION (SECTION 3.2)
In the experiments of Figure 3, we used a learning rate of 1e-6 and set 3 to 0.01 for IM-RMs. We additionally considered lower and higher values for both hyperparameters (namely, learning rates 5e-7 and 5e-6 and 3 coefficients 0.005, 0.05, and 0.1). The results were analogous. That is, both the EX-RMs and IM-RMs were able to accurately distinguish between valid and invalid Hamiltonian cycles, although the IM-RMs were unable to generate even a single Hamiltonian cycle.
F.2 CONTROLLED EXPERIMENTS: TOKEN-LEVEL SHIFT (SECTION 5.1)
In the experiments of Figure 4, we used a learning rate of 1e-6 and set 3 to 0.01 for IM-RMs. We additionally considered lower and higher values for both hyperparameters (namely, learning rates 5e-7 and 5e-6 and 3 coefficients 0.005, 0.05, and 0. The results were analogous. That is, IM- RMs were extremely inaccurate over paraphrased responses, whereas EX-RMs achieved perfect accuracy (for both training and test prompts).
Furthermore, ran the same experiments with explicit generative reward models (EX-GRMs; Appendix D) in place of EX EX. We found that EX-GRMs exhibit similar trends to EX- EX-RMs (i.e., generalize perfectly to paraphrased responses).
25

---

Published as a conference paper at ICLR 2026
F.3
EXPERIMENTS: LEVEL AND DOMAIN SHIFTS (SECTION 5.2)
EX-RMs vs IM-RMs. Listed below are additional results, omitted from Section 5.2, comparing the generalization of EX-RMs and IM-RMs.
Tables 2 and 4 provide per evaluation dataset breakdown of the results in Figures and 5, respectively.
Tables 3 and provide a per evaluation dataset breakdown of the results in Table the general chat and math settings, respectively.
Figure 6 demonstrates that the results in Section 5.2 are robust to different learning rates and B coefficients for IM-RMs.
Figures 9 and supplement Figures 2 and 5, respectively, by including the accuracy of reward models per initial language model and evaluation dataset.
Evidence against alternative hypotheses. Recall, our analysis (Section 4) indicates that IM-RMs are more sensitive superficial token-level cues, and thus often generalize worse than EX-RMs. We provide evidence against two alternative potential sources for the generalization gap between EX-RMs and IM-RMs. First, the reward of EX-RM based on the hidden representation of the whole response, while IM-RMs depend also on the hidden representations of intermediate tokens the response. Intuitively, the hidden representations of intermediate tokens may be misleading since they do not capture the full meaning of the response. Second, the reward of an IM-RM is shifted by the log probability demonstrates that these differences do not explain the generalization gap, the gap remains when considering EX-RMs trained over the hidden representations of all intermediate tokens and IM-RMs without reference distribution."
reference distribution, which not the case for EX-RMs. Figure
EX-GRMs vs IM-RMs. Listed below are additional experiments, omitted from Section 5.2, com- paring the generalization of explicit generative reward models (EX-GRMs Appendix D) and IM-RMs. Notably, we find that the results of EX-GRMs are analogous to those of EX-RMs: they robust token-level shifts than IM-RMs, as anticipated by analysis (Appendix D.1).
Figure 8 presents the results of an experiment identical to that of Figures 2 and 5, except that it compares EX-GRMs (instead EX-RMs) to IM-RMs.
Table 6 supplements Figure by reporting the accuracy and absolute (normalized) reward margin of EX-GRMs and IM-RMs over the different evaluation categories.
Tables 7 and provide per evaluation dataset breakdown of the results in Figure 8.
Tables 8 and 10 provide a per evaluation dataset breakdown of the results in Table 6 for the general chat and math settings, respectively.
Figures 9 and 10 supplement Figure 8 by including the accuracy, per initial language model and evaluation dataset, for the general chat and math settings of Section 5.2, respectively.
G ADDITIONAL IMPLEMENTATION DETAILS
In this appendix, we provide implementation details omitted from Section 3.2, Section 5, and Ap- pendix F. Code for reproducing our results, based on the PyTorch (Paszke et al., 2017) and Hugging Face (Wolf et al., 2019) frameworks, can be found at ht httpp::/ggtt //tpttgbt////tt/////// com/ pttppprtnnneepppii.. exrm-vs mmm.
In all experiments, for each prompt and response, used the following chat template (unless the model already had a chat template, in which case we used the original chat template):
[USER] [prompt|| [ASSIItAAAASSSSSATTATAAA [response [EOS]
where [USER], [ASSISTANT], and [EOS] are defined as special tokens.
"These results do not preclude the possibility that, in certain settings where the last token can completely change the meaning of a response, the IM-RM's reliance on hidden representations of intermediate tokens may lead to generalization than EX-RMs.
26

---

Published as a conference paper at ICLR 2026
G.1 HAMILTONIAN CYCLE VERIFICATION (SECTION 3.2)
Data. Each example the dataset consisted of: (i) a prompt that describes an undirected graph with N N vertices, (ii) chosen response, which describes permutation of vertices that forms a Hamiltonian cycle in the graph, and (iii) a rejected response, which describes a permutation of vertices that does not form Hamiltonian cycle in the graph. Below
examples for the prompt and response formats, where vertices are denoted by integers from 0 to N - 1, the token [sep] is used separate vertices and edges, and the token [edge_sep] is used to separate the two vertices of an edge. The examples are for graphs with N = 10 vertices.
Hamiltonian cycle preference dataset: prompt example
Vertices: http://pppllaaoo//////////////et
Edges: httpsh/www......i...o/////////pa
[[edge_seplil Sondgeesep//lxxpp0lede..... http:///cccee..peoooe///eeeeeeee Selps///deesppppSSsepeeeeegeee sep]3
[sep]5[edge sep]7|sep]óledge
beep:l/cdge.ssp//spp/lcee..xx sep] sep}B[sep]2ledge sep]3[sep]ß[edge sep]6
Hamiltonian cycle preference dataset: response example
We randomly generated training and test sets with 1000 and 200 examples, respectively. To ensure that each graph has least one Hamiltonian cycle, we first added the necessary edges for random permutation of vertices. Then, for each remaining possible edge independently, we added it to the graph with probability p E [0, 1]. For the experiments of Figure 3, we created graphs with N 10 vertices and chose p = 0.2. Experiments with additional configurations (e.g.. N = 8, 12 and p 0.1, 0.3) led to similar outcomes.
Training. We minimized the Bradley- log-likelihood loss (Equation (3)) via the Adam opti- mizer (Kingma and Ba, 2015) for 15 epochs with learning rate le-6 and batch size 32 (emulated via two gradient accumulation steps of size 16). For IM-RMs, we set 99 to 0.01. As stated in Ap- pendix F.1, experiments with both lower and higher learning rates and B coefficients led to similar results. Note that all parameters of reward model were trained, including the neural network back- bone and the linear head for EX-RMs or the unembedding matrix for IM-RMs.
Generation. When evaluating the ability of IM-RMs generate Hamiltonian cycles, we used temperature of and sampled one output for each prompt the training and test sets.
Hardware. Experiments ran on a single Nvidia H100 GPU with 80GB of memory.
G.2 CONTROLLED EXPERIMENTS: TOKEN-LEVEL SHIFT (SECTION 5.1)
Data. We took from the Persona dataset (Perez al., 2022) prompts that match the "agreeableness" behavioral trait, of which there are 500, and created training and test splits by randomly sampling 80 and prompts, respectively. For the chosen and rejected responses, we randomly assigned each prompt one of four manually written options (see below). Note that chosen responses express agree- ment with the prompt and rejected responses express disagreement, During evaluation, alongside the original responses, we measured the accuracy of the reward models over paraphrased versions of the original responses (see below).
Original chosen responses
A truthful reply is yes
A truthful reply is sure
In my view, the answer yes
In my view, the answer sure
27

---

Published as a conference paper at ICLR 2026
Original rejected responses
Considering statement, say no
Considering the statement, say never
My judgement: no
My judgement: never
Paraphrased chosen responses
My response would be certainly
My response would be absolutely
Paraphrased rejected responses
lean toward not really
lean toward nope
Training. We minimized the Bradley- Terry log-likelihood loss (Equation (3)) via the Adam opti- mizer for 50 epochs with learning rate le-6 and batch size 32 (emulated via four gradient accumu- lation steps of size 8). For IM-RMs, we 3 to 0.01. As stated in Appendix F.2, experiments with both lower and higher learning rates and 3 coefficients led to similar results. Note that all parame- ters of reward model were trained, including the neural network backbone and the linear head for EX-RMs or the unembedding matrix for IM-RMs.
EX-GRMs. In accordance with Zhang et al. (2025), we trained EX-GRMs by minimizing the loss in Equation (7), using the same hyperparameters as for EX-RMs. Inputs to EX-GRMs were formatted the following template. For simplicity, did not use chain-of- ohothhttht tokens.
EX-GRM input format
Question:
[prompt] AnAnswer: [response] Verificatii: Is the answer correct (Yes/No)?
Hardware. Experiments a single Nvidia H100 GPU with 80GB of memory.
G.3
REAL-WORLD EXPERIMENTS: TOKEN-LEVEL AND DOMAIN SHIFTS (SECTION 5.2)
Data. The experiments involved two training datasets - one for the general chat setting and another for the math setting and seven evaluation test sets that were shared among the settings.
Training sets.
UltraFeedback. For the general chat setting, took the training set of the binarized Ultra- Feedback dataset" (Cui et al., 2024) and filtered out examples in which either the prompt or one of the responses exceeded 512 tokens according to the Llama-3.2- 1B tokenizer. We fur- ther removed examples in which the prompt contained the words "translate" or "translation", which may lead to nonsensical examples when translating the responses to different languages (as done for creating evaluation sets with token-level shifts; see details below). Then, we randomly sampled 2000 examples from the remaining examples.
RewardMATH, For the math setting, we used the pairwise preferences version of the Reward- the chat setting, we filtered out examples in which
MATH d (Kim et al., 2024). either the prompt or one of the responses exceeded 512 tokens according to the Llama-3.2- IB tokenizer. Then, we created a training set 1000 randomly sampled examples (note that RewardMATH does not contain predefined training and test splits).
Phstps://hugginee ee/ se/dat datasets Buggingg scedd wtwea//eeeeeeeeaaaacee.... binariiiniiiee Prttpa:// /huggh aagaae, sarface.ooo aset RowardMAT /Rewa YdMATH _a.
28

---

Published as a conference paper at ICLR 2026
Evaluation sets.
Ultraeeedback, We processed the test set of UltraFeedback in the same way as the training set, and randomly sampled 200 examples.
Ultrafeedback: Paraphrased. We took the UltraFeedback test set and paraphrased both the chosen and rejected responses via GPT-4.1 (version gpt-4.1-0225-04-114
Prompt to GPT-4.1 for paraphrasing UltraFeedback responses
I will provide text. Please rewrite it so that the meaning remains the same, but the wording overlaps with the original text as little as possible. Aim to minimize word and phrase overlap while preserving all key information and nuance. Output only the rewritten text and nothing else.) nlere is the original text))n [response] |nRewritten version:),n
To assess the quality of paraphrasing, manually examined random subset of 40 para- phrased responses. We found 39 of the paraphrases to be valid, in the sense that they preserve the meaning of the original response. The single invalid paraphrasing occurred because the corresponding prompt required a unique correct response (e.g., the name try). Moreover, used GPT-5 mini (version 2o--5-mini-2025--00 00-077)77 verify the full set of 400 responses (200 chosen and 200 rejected responses). Only 9 out of the 400 (2.25%) were marked potentially invalid.
specific coun-
Prompt to GPT-5 mini for validating paraphrased responses
Below are two responses. Determine whether the second response is a paraphrased version of the first one. That is, do they have similar meaning if they different words convey it?',n Please include a brief explanation and then output final line in striet JSON format. paraphrased version of the first, respond with: ("paraphrased", true)
the second response (n otherwise, respond with:),n n ("paraphrased"" false) A'nnn Response 1: [response.1] |n)nn'n Response 2: [response.2]
French, We took the UltraFeedback test set and translated both the chosen and rejected responses to French via GPT-4.1 (version gpt-4. -2225-1-022504-- 04---444444
UltraFeedbcck:
Prompt to GPT-4.1 for translating Ultraeeedback responses to French
I will provide text in English. Please translate it French while ensuring that the ing remains the same. Output only the translated text and nothing else. (ns.ee is the original text:),n [response] wwwespposseennanaaaaaaaaaaaa
\nTraanslated version:),n
Spanish. We took the UltraFeedback test and translated both the chosen and rejected responses to Spanish via GPT-4. (version apt-4.1-2025-04-11
UltraFeedback:
Prompt to GPT-4.1 for translating UltraFeedback responses to Spanish
I will provide in English. Please translate Spanish while ensuring that the mean- ing remains the same. Output only the translated text and nothing else. else.),nHed. \nHre is the original text:),n [response] inTranslated version:),n
RewardBench: Math. We randomly sampled 200 examples from the math subset of Reward- Bench" (Lambert et al., 2025) (i.e., examples whose subset field is mmath-prm"), "), after filtering out examples in which either the prompt or one of the responses exceeded 512 tokens accord- ing to the Llama-3.2-IB tokenizer.
RewardBench: Code, We randomly sampled 200 examples from the code subset of Reward- Bench (i.e., examples whose subset field starts with "hep"), after filtering out examples in
Prttps: hhttpa://huggiii agfa oo/datasets/ a/a /reward-bench
29

---

Published as a conference paper at ICLR 2026
which either the prompt one of the responses exceeded 512 tokens according to the Llama- 3,2-1B tokenizer.
RewardMATH. When creating the RewardMATH training set, we also designated 200 ran- domiy sampled examples as test examples.
Training. We minimized the Bradley-Terry log-likelihood loss (Fquation (3)) via the Adam op- timizer for 5 epochs with learning rate 1e-6 and batch size 32 (emulated via eight gradient accu- mulation steps). For IM-RMs, we set 3 to 0.01. As demonstrated by Figure in Appendix F.3, experiments with both lower and higher learning rates and / coefficients led to similar results. Note that all parameters of a reward model were trained, including the neural network backbone and the linear head for EX-RMs or the unembedding matrix for IM-RMs. To ensure fair compari- son between EX-RMs and IM-RMs, verified that their training loss and accuracy were roughly same. Specifically, their training loss was below 0.04 in the general chat setting and 0.005 in the math setting. The accuracy above 0.993 in both settings (values means across initial language models and random seeds).
EX-GRMs. See EX-GRMs paragraph in Appendix G.2.
Absolute reward margin computation. For each reward model r and evaluation set separately, to measure the absolute (normalized) reward margin, first computed the standard deviation of rewards over all responses (chosen and rejected). Denoting this standard deviation by s, for each example (x,y.. y in the evaluation set, the absolute (normalized) reward margin is given by:
1 (r(x,yt)-((x,,
We report the mean of this quantity over the evaluation set. Note that the normalization is intended account for arbitrary differences in reward scale between reward models.
Hardware. Experiments based on Llama-3.2-1 Instruct ran on a single Nvidia H100 GPU with 80GB of memory. For experiments with the remaining language models (of scales ranging from 1.5B to 8B), we used four such GPUs per run.
30

---

Published as a conference paper at ICLR 2026
Table 2: Per evaluation datasct breakdown of the win-rates reported in Figure (I.e, for the general chat setting of Section 5.2). We abbreviate UltraFeedback UF and RewardBench as RB.
Win-Rate (%)
Evaluation
Dataset
EX-RM
Tic
IM-RM
In Di-Distribution
UF
100
0
Paraphrased
100
0
Token-Level Shift
UF: French
72.2
16.7
11.1
UF: Spanish
RB: Math
22.2
77.8
Domain Shift
RewardMATH
38.9
22.2
38.9
RB: Code
27.8
72.2
Table 3: Per evaluation dataset breakdown of the accuracy and absolute (normalized) reward margin values reported in Table 1, for the general chat setting of Section 5.2 (i.e. for the rows corresponding to UltraFeedback training data). We abbreviate UltraFedbback UF and RewardBench RB.
Accuracy
Absolute Reward Margin
Evaluation
Dataset
EX-RM
IM-RM
EX-RM
IM-RM
In- Di-Ditriibutin
UF
0.752 0.009
0.646 + 0.006
1.014 + 0.023
0.813 + 0.003
UF: Paraphrased
0.687 0.005
0.379 0.002
0.954 0.010
0.730
Token-Level Shift
UF: French
0.645 0. 0.004
0.616 = 0.001
0.991 = 0.008
0.785 + 0.004
UF: Spanish
0.662 0.010
0.612 0.002
0.984 0.007
0.774 0.004
RB: Math
0.513 0.041
0.737 + 0.008
1.092 0.024
0.002
Domain Shift
RewardMATH
0.594 + 0.022
0.593 = 0.007
0.922 0.021
0.802 0.001
RB: Code
0.754 0.014
+ 0.002
0.409 = 0.003
0.000
Table 4: evaluation dataset breakdown of the win-rates reported Figure 5 (i.e., the math setting Section 5.2). We abbreviate UltraFeedback as UF and RewardBench as RB.
Win-Ram (%)
Evaluation
Dataset
RX-RM
IM-RM
In-Distribution
RewardMATH
16.7
66.6
16.7
Token-Level Shift
RB: Math
100
0
UF
38.9
11.1
50.0
UF: Paraphrased
55.5
16.7
27.8
Domain Shift
UF: French
38.9
22.2
38.9
UF: Spanish
55.6
22.2
22.2
RB: Code
11.1
88.9
31

---

Published as a conference paper at ICLR 2026
Table 5: Per evaluation dataset breakdown of the accuracy and absolute (normalized) reward margin values reported in Table I, for the math setting Section 5.2 (i.e, for the rows corresponding to RewardMATH training data). We abbreviate UltraFedbbcc as UF and RewardBench as RB.
Accuracy
Absolute Reward Margin
Evaluation
Dataset
EX-RM
IM-RM
EX-RM
IM-RM
In- nn-Diitrrbution
RewardMATH
0.971 + 0.003
0.972 = 0.002
1.602 = 0.011
1.377 0.007
Token-Level Shift
RB: Math
0.988 0.003
0.515 = 0.007
1.667 = 0.017
1.035 #
UF
0.487 0.018
0.475 + 0.005
0.881 0.013
0.697 0.003
UF: Paraphrased
0.467 0.0117
0.433 + 0.002
0.872 : 0.021
0.703 :: 0.003
Domain Shift
UF: French
0.484 0.018
0.475 1 0.005
0.891
0.002
0.698 0.005
UF: Spanish
0.485 0.004
0.462 1 0.002
0.883 1 0.006
0.693 0.006
RB: Code
0.604 0.008
0.740 1 0.008
0.247
0.003
022 0.003
10
Random Guess EX-RM
Accuracy
Co-
EX-RMOr-SS 061
DX-RMQr5Se 071
Ce.
IM RM0r-55 061
IM RMMM0--0
Distribution
Paraphrased
Spanish
Bolken-Level
10
Guass
0.-
Accuracy
EX RM
Co-
IM-RM (3 0051
IM-RM (3-0
02-
IM-RM (S-0 1)
Distiibuution
UF: Paraphrased
Urench
UP: Soanish
Math
RB: RewardMATH
Tolven-Levll Shift
Domaln Shift
Figure 6: The results of Section 5.2 are robust to different learning rates (top) and 3 coefficients for IM-RMs (bottom). In the general chat setting of Section 5.2, we compare the accuracy of EX-RMs and IM-RMs trained using additional learning rates and coefficients (for IM-RMs). We consider both lower and higher values than the default ones (as specified in Appendix G. the default learning rate le-6 and default coefficient 0.01). All reward models were trained on Ulrraeeddback, starting from the model. In the figure, we abbreviate UltraFeedback as UF and RewardBench as RB. Error bars mark standard deviation across three random seeds. For the range of hyperparameters considered, the trends remain the same in Figure 2. Namely, IM-RMs are less robust token-level shifts than EX-RMs, yet perform comparably better under domain shifts.
language
Table 6: This table supplements Figure 8 by reporting accuracy and absolute (normalized) reward margin over the different evaluation categories. In each row, bold font marks the highest accuracy and absolute reward margin (unless the values are within 0.01 of each other, after taking into account standard deviations). For each reward model and evaluation dataset separately, the absolute reward margin is normalized by the standard deviation of rewards to account for arbitrary differences scale. Values in the table means across the models (six in total) and evaluation datasets, with standard deviation computed based on three random seeds. See Tables 8 and 10 for a per evaluation dataset breakdown of the results.
Accuracy
Absolute Reward Margin
Training Data
Evaluation
EX-GRM
IM-RM
EX-GRM
IM-RM
In- Distribution
0.714 + 0..0
0.646 + 0.006
1.075 + 0.000
0.813 = 0.000
UltraFeedback
Toven-Level Shift
0.666 00.002
0.602 = 0.003
0.915 0.012
0.763 0.003
Domain Shift
0.616 i 0.004
0.720 1 0.004
0.707 1 0.004
0.726 1 0.001
In- Distribution
0.979 + 0.003
0.972 + 0.002
1.724 + 0.014
1.377 = 0.007
RewardMATH
Token-Leve Shift
0.918 0.006
0.515 + 0.007
1.339 1 0.032
1.035
Domain Shift
0.563 1 0.005
0.517 + 0.001
0.251 +: 0.004
0.604 + 0.004
32

---

Published as a conference paper at ICLR 2026
10- ca
Random Gures
Accuracy
RM
EX RM (all repr?
Ca-
C2.
IN RM ino refl
Paraphraed
Math
Takan-LLvedd Shift
Domain Shift
Figure 7: Evidence against alternative hypotheses on generalization gap between EX-RMs and IM-RMs. the general chat setting Section 5.2, we compare the accuracy four reward model types: (i) a standard EX-RM, which applies a linear head to the last hidden representation of prompt-response pair yx,y), ie., (
to hx.y (Equation (I)), (ii) an EX-RM that applies linear head to the mean of all hidden representations of the response ("all repr"), ie., to ly| Σ hx..y (iit) standard IM-RM (Equation (2)), and (iv) IM- RM without reference distribution ("no ref"), Le., for a prompt-response pair (x, y) it assigns the reward In Toma (y x) instead of (1nn Tona (y)x) In (y/x)) All reward models were trained on Uttraccedback, starting from the Llama-3.1-1 1-BBBBBBBBBBBB. --Istuuttt language model. In the figure, we abbreviate UltraFeddback and RewardBench RM and IM-RM variants exhibit similar trends
UF RB. Error bars mark standard deviation across three random seeds. Notice that the EX- original ones. Namely, both IM-RMs are less robust to poken-level shifts than the EX-RMs, yet perform comparably or better under domain shifts. This suggests that the generalization between EX-RMs and IM-RMs is not caused by the IM-RMs' reliance on hidden representations of intermediate tokens in response reference distribution.
100% -
13.0%
14.8%
Tranning Data: Feeeeeebaaae
FltraFee
11.1%
In- Distribution D
Token -ekennLLvvelsee Shift
Domain Shiff
EX GRM Win
Paraphrassdd & Transiine Ultral eedbock Vooiinng
Tie
Code
IM-RM Win
74.1%
87.0%
11.2%
100%
44.4% 49%
17.8%
Trainine Data:
EX-GRM Wm
In-Distribution
Leven Shift
Domain Shift U Utrafeedback Variants S Cade
Tie
RewordMATH
RewwardBernt Math
RMWI Win
75.5%
B 8: IM-RMs are less robust than EX-GRMs (Appendix D) to token-level distribution shifts, but per- form comparably better under domain shifts. This figure presents the results of experiment identical that Figures and 5, except it compares EX- GRMMs (instead of EX (X-Mss) to IM RMs.
Figure
33

---

Published as a conference paper at ICLR 2026
Table 7: Per evaluation dataset breakdown of the win-rates reported in Figure 8 for the general chat setting of Section 5.2 (i.e., for the row corresponding to UltraFeedback training data in Figure 8). We abbreviate UltraFeedback UF and RewardBench as RB.
Win-Rate (%)
Evaluation
Dataset
EX-GRM
Tie
IM-RM
In-Distribution
UF
100
0
UF: Paraphrased
100
0
Token-Level Shift
UF: French
72.2
27.8
0
UF: Spanish
88.9
11.1
0
RB: Math
100
Domain Shift
RewardMATH
33.3
11.1
55.6
RB: Code
11.1
22.2
66.7
Table 8: evaluation dataset breakdown accuracy and absolute (normalized) reward margin values reported in Table 6, for the general chat setting of Section 5.2 (i.e. for the rows corresponding to UtraFeedback training data). We abbreviate UltraFeedback as UF and RewardBench as RB.
Accuracy
Absolute Reward Margin
Evaluation
Dataset
EX-GRM
IM-RM
EX-GRM
IM-RM
In-Distrihution
UF
0.714 +: 0.005
0.646 0.006
1.075 0.007
0.813 1 0.003
UF: Paraphrased
0.667 + 0.004
0.579 0.002
0.923 0.010
0.730 + 0.008
Token-Level Shift
UF: French
0,660 + 0.004
0.616 0.001
0.909 0.006
0.785 = 0.004
UF: Spanish
0.672 1: 0.001
0.612
0.914
0.774 0.004
RB: Math
0.497 + 0.015
0.737 + 0.008
0.844 0.012
1.056 + 0.002
Domain Shift
RewardMATH
0.565 0.004
0.593 0.007
0.882 0.010
0.802 0.001
RB: Code
0.786 :: 0.008
0.830 0.002
0.395 + 0.003
0.319 1 0.005
Table 9: Per evaluation dataset breakdown of the win-rates reported in Figure 8 for the math setting of Sec- tion 5.2 (i.e., for the row corresponding to RewardMATH training data in Figure 8). abbreviate UltraFeed- back as UF and RewardBench as RB.
Win-Rate (%)
Evaluation
Dataset
EX-GRM
IM-RM
In Distribution
RewardMATH
44.4
44.4
11.2
Token-Level Shift
RB: Math
100
0
UF
83.3
11.1
UF: Paraphrased
83.3
16.7
0
Domain Shift
UF: French
88.9
11.1
0
UF: Spanish
100
0
RB: Code
22.2
77.8
34

---

Published as a conference paper at ICLR 2026
Table Per evaluation dataset breakdown of the accuracy and absolute (normalized) reward margin values reported in Table 6, for the math setting of Section 5.2 (i.e, for the rows corresponding to RewardMATH training data). We abbreviate UltraFeedback as UF and RewardBench as RB.
Accuracy
Absolute Reward Margin
Evaluation
Dataset
EX-GRM
IM-RM
EX-GRM
IM-RM
In-Distrihution
RewardMATH
0.979 0.002
0.972 0.002
1.724 0.014
1.377 0.007
Token-Level Shift
RB: Math
0.918 + 0.006
0.515 0.007
1.339 0.032
1.035 + 0.011
UF
0.540 + 0.004
0.475 0.005
0.325 + 0.002
0.697 + 0.003
UF: Paraphrased
0.530 0.001
0.433
0.315 0.007
0.703 + 0.003
Domain Shift
UF: French
0.552 + 0.003
0.475 = 0..05
0.214 0.006
0.698 + 0.005
UF: Spanish
0.548 0.001
0.462 0.002
0.228 0.007
0.693 : 0.006
RB: Code
0.645 :: 0.018
0.740 0.008
0.174 +: 0.008
0.228 1: 0.003
35

---

Published as a conference paper at ICLR 2026
Training Data: traaeedback (UF),
Model: 1.lama-3.2-1B-nmstt
10:
Accuracy
EX GRM
Ca-
IM-RM
Token Shift
Domain Shift
Training Data: U traFeedback (UF),
Model: Llama-3.2 1arma-3.2-3B--n-ee 3.2-3B-Inttruct
Accuraco aoV
Random Guess
Ca-
EX-GRM
IM RM
0.0
Dietttr bution
UF: Paraphrased
Spanish
RB:
R8: RewardMATH
Token-Level Shift
Domain Shift
Training Data: traFeedback (UF), Model: Llama-3.1 1.1aa-3.-1----
Accuracy
68-
Rondorm Guess
Ca-
EX-GRM
EX RM
0.
b-Distribution
UF: Paraphaadddd
Ur:French
UT: Sponish
RB:: Math
R0:Coree
RewardMATII
Token-Level Shift
Domain Shift
Training Data: UltraFoedback (UF), Model: Gemma 2-mma--2----- 2-2B-IT
08
Accuraty
Rondom Guess
EX-GAM
EX-RM
69-
n-Distributioo
UF: Parappreeed
UF:
UF:: Spontsh
RB: Math
RB: Corle
RewardMATH
Token-Level Shift
Domain Shift
Training Data: UltraFeedback (UF), Model: Qwen 2.5 1.5B Instruct
10
Accuracy
EX GRM
Ce.
IM RM
Parapheedd
Shift
Domain
Training Data: UltraFeedback (UF),
Model: (wwen-2.5-3B-Instruu
Accuracy
Rondom Guess
Co
LX-GRM
IM RM
60
I-Distrbuttonn
Parppprprreeedd
UF:French
UP: Spontsh
RB: Moth
RB: Code
RewordMATH
Token-Level Shift
Domain Shift
Figure 9: This figure supplements Figures and 8 by including the accuracy, per initial language model and evaluation dataset, of the EX-RMs. IM-RMs, EX-GRMs trained on UltraFeedback as part the general chat setting experiments Section 5.2. In the figure, we abbreviate UltraFeedback as UF and RewardBench as RB. Error bars mark standard deviation across three random seeds.
36

---

Published as a conference paper at ICLR 2026
Training Data: RewardMATH, Model: Saama-3.2-BBlnnttrer
Accuracy
Random
66-
EX GAM
IM-RM
00
n-Distriution
Math
UF: Parsphrazed
UF:French
UF: Sponish
RB: Code
Training Data: RewardMATH, Model: Llama-3.2-36 2a----.-----sstuucc
10-
Accuracy
Random
EX-GAM
04-
IM-4IM
Diverr
Paraphriood
UFFFranch
Speriih
Training Data: RewardMATH, Model: Llama-3. 1eeee3..--1--trreeee
10
Accuracy
Co-
EX GRM
C4
IM-RM
Token-Lavel Paraphrased
Domain
Training Data: RewardMATH, Model: Gemma-2-22-N
10
CB.
Accuracy
EX GRM
C4-
EX-RM
IM RM
Parsphrased
Tokion Level
Domain
Training Data: RewardMATH, Model: Qwen-2.5-11. 1.55-1...B-nnstrrctt
Accuracy
CB
Random
EX-GRM
6.-
IM-RM
Distr buttonnttoon
PaPaapaphasddd
UF:Frinch
UF: Sperish
Token-Level Shift
Domain Shift
Training Data: RewardMATH, Model: Diven-2.5-3--Instt
0.8
Accuracy
Rondom
EX-GRM
EX-RM
02:
Distribution
UF:French
UF: Sparish
Token-Level Shift
Domain Shift
Figure 10: This figure supplements Figures and 8 by including the accuracy, per initial language model and evaluation dataset, of the EX RMs. IM-RMs, and EX EX-GRMs trained on RewardMATH as part of the math setting experiments of Section 5.2. In the figure, abbreviate UlraFeedback UF and RewardBench as RB. three random seeds.
Error hars mark standard deviation
37


Reviews:
[{"review_id": "x5Rg6CZw5z", "reviewer": "Reviewer_D62L", "summary": "The paper contrasts implicit reward models (IM-RMs; rewards via log-likelihood ratios) with explicit reward models (EX-RMs; linear head over hidden states) and asks why IM-RMs underperform despite near-identical training data and losses. The core claim is that IM-RMs lean on surface token cues, so they break under token-level distribution shifts (paraphrases/translations) even when they look fine in-distribution; meanwhile they can be competitive under domain shifts. The authors (i) prove that an IM-RM can verify without being able to generate correct responses (so the gen-vs-verify gap is not the culprit), (ii) analyze learning dynamics to show explicit token dependence in IM-RMs, and (iii) back this up with controlled Persona paraphrase tests and broader UltraFeedback/RewardMATH/RewardBench experiments across 1B–8B backbones.", "questions": "Paraphrase hygiene. How do you ensure paraphrases preserve label semantics (beyond model intuition)? Any human spot-checks or automatic entailment filters, and what are their pass rates?\n\nCheap mitigations. Did you try (a) paraphrase-contrastive pairs during IM-RM training, (b) freezing the unembedding or adding token-dropout on the reward path, or (c) sharing a representation head but decoupling logits for reward vs. generation?\n\nReward margin & RL. You note EX-RMs yield larger normalized reward margins. Can you show an RL-phase consequence (e.g., better PPO/DPO-RFT optimization curves) on the same base policy to quantify end-to-end impact?\n\nDecoding sensitivity. Do IM-RM failures persist when evaluation decoding changes (nucleus vs. greedy; different temps) or when rewards are computed over masked tokens only (e.g., function words stripped)?\n\nCross-lingual asymmetry. In translation shifts, are failures symmetric (EN→FR vs FR→EN)? Any sign that subword overlap (BPE sharing) modulates the IM-RM gap?\n\nWhen would you still pick IM-RM? Beyond domain-shift scenarios in your figures, are there resource or deployment regimes (e.g., no extra head, log-prob reuse for TTS) where IM-RM’s simplicity outweighs its fragility?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Sharp negative result against a popular hypothesis: verification ≠ generation for IM-RMs; the Hamiltonian task illustrates this cleanly.\n\nMechanistic story that matches data. The gradient-level analysis predicts exactly where IM-RMs fail (paraphrases/translations), and the controlled Persona setup nails the failure mode.\n\nBreadth and consistency. Multiple model families (1B–8B) and both general-chat and math settings; token-shift brittleness shows up systematically.\n\nPractical takeaway. If your evals include paraphrases or multilingual restatements, EX-RMs are the safer default; IM-RMs can be fine under domain shift but need help on surface variation.", "weaknesses": "Paraphrase pipeline dependence. Robustness claims rest heavily on how paraphrases/translations were produced. I’d like BLEU/chrF ranges, style diversity checks, and a sanity control where paraphrases are fed back through the teacher to confirm semantic equivalence.\n\nLimited exploration of mitigations. The paper diagnoses token-level sensitivity but doesn’t dig into cheap fixes (representation freezing, unembedding regularization, token-dropout on reward paths, contrastive paraphrase augmentation).\n\nCalibration/uncertainty left on the table. If IM-RMs rely on surface cues, do they also become miscalibrated on paraphrases (reward margin vs. accuracy)? A short calibration slice (ECE/Brier) would make the case stronger.\n\nHead-vs-backbone entanglement. Theory assumes fixed representations; experiments train full models but don’t isolate the contribution of updating the unembedding vs. earlier layers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cvWDxxFvwl", "reviewer": "Reviewer_fVe7", "summary": "This paper investigates the generalization gap between EX-RMs and IM-RMs, two nearly identical reward model architectures. This work's main contribution is identifying that IM-RMs rely more heavily on superficial, token-level cues, whereas EX-RMs leverage semantic information encoded in hidden representations. Through theoretical analysis of their learning dynamics and extensive empirical evaluation, the paper demonstrates that this reliance causes IM-RMs to perform poorly under token-level distribution shifts. The work also provides evidence against the alternative hypothesis that IM-RMs struggle due to a \"generation-verification gap.\" The experiments show that while EX-RMs are more robust to token-level shifts, IM-RMs can perform comparably or better under larger domain shifts.", "questions": "- Could this work elaborate on the intuition for why the theoretical conclusions derived under the fixed hidden representation assumption still seem to hold empirically when all parameters are fine-tuned?  Is it because the updates to the backbone are minimal? Or add a comparison experiment with a fixed backbone?\n- The results in Figures 2 and 5 consistently show IM-RMs performing well, and sometimes better than EX-RMs, under domain shifts. Does this work have a hypothesis to explain this?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses a well-defined and important problem: understanding the performance discrepancy between EX-RMs and IM-RMs, which are structurally very similar yet exhibit different generalization behaviors.\n- The paper effectively challenges the \"generation-verification gap\" hypothesis. It proves theoretically that verification with an IM-RM does not require generation and demonstrates this empirically on a synthetic Hamiltonian cycle task.\n- The paper’s claims are substantiated by a comprehensive set of experiments that systematically test the core hypothesis under various conditions.", "weaknesses": "- The primary theoretical analysis in Section 4 and Appendix B relies on simplifying assumptions that are violated in the main experiments, potentially limiting the direct applicability of the theory.  1) Assumption 1 posits that hidden representations are fixed during training. However, the empirical results are generated by training all reward model parameters. The paper notes that its conclusions still hold empirically, but does not fully bridge the gap to explain why the dynamics under the simplifying assumption are still predictive. 2) The theoretical result in Theorem 2 is derived under the highly restrictive case of single-token responses. This is far from the practical scenario of long-form text generation.\n- While the mathematical analysis is a core strength, the presentation of some key concepts could be improved for better readability and intuition. For example, the coefficient ρ_k,l(v) in Equation (5)is central to the argument about IM-RMs' token-level dependency. A more intuitive explanation in the main text of how token identity (the indicator function) and distributional similarity (the inner product of next-token distributions) contribute to its value would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FAExLGlI6x", "reviewer": "Reviewer_haqk", "summary": "This work explores reward models, specifically how implicit reward models generalize worse than explicit reward models. The authors present a theoretical analysis highlighting that implicit reward models rely more heavily on superficial token-level queues, which makes them less robust to token level distribution shifts.\n\nThe contributions of the work are as follows. First, the authors present a theoretical formulation of the gap between EX-RMs and IM-RMs. This is then empirically supported by a set of experiments. These experiments show that IM-RMs fail to generalize to paraphrased responses, and that IMs are less robust than EX-RMs to token-level distribution shifts. In real-world experiments, they train both EX-RMs and IM-RMs on a series of datasets in reasoning and general task domains.\n\nThe strengths of the work are as follows. First, the problem is relevant and training strong reward models are important for LLM training. Second, there is a strong theoretical component, showing that EX-RMs and IMs have a gap. Third, there is a strong empirical second, showing an identification of the root cause.\n\nThe weaknesses of the work are as follows. First, there is not a downstream performance analysis, and reward model performance does not always correlate with downstream performance. The work could benefit from a stronger empirical analysis that includes downstream model performance, although this is computationally challenging.", "questions": "Are there any concrete recommendations for when EX-RMs or IM-RMs should be used? Are there hybrid approaches?\nAre there any evaluation metrics considered beyond accuracy?\nWill code and trained models be released?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of the work are as follows.\n- First, the problem is relevant and training strong reward models are important for LLM training.\n- Second, there is a strong theoretical component, showing that EX-RMs and IMs have a gap.\n- Third, there is a strong empirical second, showing an identification of the root cause.", "weaknesses": "The weaknesses of the work are as follows. First, there is not a downstream performance analysis, and reward model performance does not always correlate with downstream performance. The work could benefit from a stronger empirical analysis that includes downstream model performance, although this is computationally challenging.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Aa5XPjSCRH", "reviewer": "Reviewer_zVNe", "summary": "**Disclaimer**: I am not an expert in theoretical analysis, so my evaluation of the technical proofs should be taken with limited weight. My review primarily reflects my assessment of the paper’s motivation, empirical validation, and broader relevance to the reward modeling community.\n\n**Summary**:  \nThis paper provides a timely and insightful investigation into the generalization gap between explicit reward models (EX-RMs)—typically linear heads on top of frozen or fine-tuned LMs—and implicit reward models (IM-RMs), as used in DPO. Through a combination of theoretical analysis and carefully designed experiments, the authors convincingly argue that IM-RMs suffer from over-reliance on superficial token-level cues, which harms their robustness under token-level distribution shifts (e.g., paraphrasing or translation). In contrast, EX-RMs generalize better in such settings by leveraging semantic structure encoded in hidden representations.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper tackles a practically important and currently underexplored question: *why* do EX-RMs often outperform IM-RMs despite near-identical training setups? The dual theoretical–empirical approach strengthens the validity of the conclusions.  \n2. The authors effectively refute a commonly held hypothesis—the “generation-verification gap”—by both proving that verification does not require strong generation capability and demonstrating this empirically on a Hamiltonian cycle task. This clarification is valuable for the field’s understanding of IM-RM limitations.", "weaknesses": "While the focus on EX-RM vs. IM-RM is well-motivated, I would have appreciated some discussion or preliminary results involving **generative reward models (GenRMs)**—particularly those that instruct variant or thinking model variant. \nRecent empirical work suggests such GenRMs can be more robust to reward hacking and exhibit better out-of-distribution generalization than both scalar EX-RMs and DPO-style IM-RMs. Extending the analysis to this emerging class of reward models would significantly broaden the paper’s impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)