On the Robustness of Reward Models for Language Model Alignment
Authors: Jiwoo Hong
Noah Lee
Eunki Kim
Guijin Son
Woojin Chung
Aman Gupta
Shao Tang
James Thorne
Venue: ICML 2025 poster (2025)
Keywords: RLHF
Reward modeling
Over-optimization
Primary area: deep_learning->large_language_models
TL;DR: We analyze the Bradley-Terry model as a source of reward model over-optimization and propose batch-wise sum-to-zero (BatchSum) regularization,

Abstract: The Bradley-Terry (BT) model is widely practiced in reward modeling for reinforcement learning with human feedback (RLHF). Despite its effectiveness, reward models (RMs) trained with BT model loss as one-way classifiers are prone to over-optimization, losing generalizability to unseen inputs. In this paper, we study the cause of over-optimization and its downstream effects on the RLHF procedure, highlighting the importance of robustness in RMs. First, we show that the excessive dispersion of hidden state norms is the main source of over-optimization. Correspondingly, we propose batch-wise sum-to-zero regularization (BSR) that enforces reward sum for each batch to be zero-centered, constraining the rewards with abnormally large magnitudes. We assess the impact of BSR in improving robustness in RMs through four scenarios of over-optimization, where BSR consistently manifests better robustness on unseen inputs. Then, we compare the plain BT model and BSR on RLHF training and empirically show that robust RMs better align the policy to the gold preference model. Finally, we apply BSR to high-quality data and models, which surpasses state-of-the-art RMs in the 8B scale by adding more than 5\% in complex preference prediction tasks. By conducting RLOO training with 8B RMs, AlpacaEval 2.0, with reducing generation length by 40\% while adding a 7\% increase in win rate, further highlights that robustness in RMs induces robustness in RLHF training.

On the Robustness of Reward Models for Language Model Alignment
Jiwoo Hong'
Noah Lee'
Eunki Kim
Guijin Son?
Woojin Chung'
Aman Gupta i Shao Tang²
James Thorne'
Abstract
1. Introduction
Reward models (RMs) are crucial components in reinforce- ment learning with human feedback (RLHF), being proxies for human preference for aligning large language models (LLMs) (Christiano al., 2017; Ziegler et al., 2020; Casper et al., 2023; Wang et al., 2024a). RMs map text sequences to scalar score, and are incorporated into the RLHF pipeline by directly using the scores (Ziegler et al., 2020; Ahma- dian et al., 2024), or labeling pairwise preferences (Rafailov et al., 2023; Hong et al., 2024; Meng et al., 2024).
The Bradley-Terry (BT) model is widely prac- ticed in reward modeling for reinforcement learn- ing with human feedback (RLHF). Despite its effectiveness, reward models (RMs) trained with BT model loss are prone to over-optimizationn. losing generalizability to unseen input distribu- tions. In this paper, we study the cause of over-optimization RM training and its down- stream effects on the RLHF procedure, accen- tuating the importance of distributional robust- ness of RMs in unseen data. First, we show that the excessive dispersion of hidden state norms is the main source of over-optimization. Then, we propose batch-wise sum-to-zert regulariza- tion (BSR) to enforce zero-centered reward sum per batch, constraining the rewards with extreme magnitudes. We assess the impact of BSR in improving robustness in RMs through four sce- narios of over-optimization, where BSR consis- tently manifests better robustness. Subsequently, we compare the plain BT model and BSR on RLHF training and empirically show that robust RMs better align the policy the gold preference model. Finally, we apply BSR to high-quality data and models, which surpasses state-of-the- art RMs in the 8B scale by adding more than 5% in complex preference prediction tasks. By conducting RLOO training with 8B RMs, Al- pacaEval 2.0 reduces generation length by 40% while adding a 7% increase in win rate, further highlighting that robustness in RMs induces ro- bustness in RLHF training. We release the code, data, and models: https: wit.ub.coom.mmmm...... LinkedIn- Bww.C///M-Rooustenee
The Bradley-Terry (BT) model (Bradley & Terry, 1952) formulation is widely adopted to train RMs using tu- ple of a prompt and two corresponding responses with a pre-annotated preference relation (e.g., chosen and re- jected), maximizing the margin of rewards between the responses. Recent works proposed additive approaches to the BT model to further adjust to the neural language model context (Yuan et al., 2024; Coste al., 2024; Yang et al., 2024b). However, these do not adjust the learning objective of the BT model at fundamental level.
Reward model over-optimization, where RMs overfit to the train set and eventually losing the alignment capability to the true preference distribution, is a typical limitation of neural RMs (Gao et al., 2023; Coste et al., 2024). Efforts to curate high-quality pairwise preference datasets has contributed to enhancing RMs (Yuan et al., 2024; Liu et al., 2024a; Cui et al., 2025), assessed through pairwise preference bench- marks (Kim et al.. 2024: Lambert et al., 2025; Liu et al., 2025b). However, using LLMS either as preference data generators or annotators often induces inherent biases like verbosity bias and self-enhancement biases (Saito al., 2023; Zhang et al., 2024b; Chen al., 2024a) and obstructs understanding of underlying RM over-optimization.
In this paper, we show that reward modeling with the BT model induces excessive hidden state norm dispersion, lead- ing to the over-optimization problem. We propose batch- wise sum-to-zero regularization (BSR) as a straightforward solution control the dispersion by penalizing abnormal reward outliers, eventually reducing the dispersion in hidden state norms. We categorize four generalization scenarios for RMs with respect to prompt and response space, allowing
'KAIST AI "OneLineAl 'LinkedIn Corporation. Correspon- dence to: Jiwoo Hong- < <jiwoo wwwggoooon@@kaee
kr>,
Proceedings of the 4dnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

On the Robustness of Reward Models for Language Model Alignment
fine-grained analysis of over-optimization and robustness. Based on these scenarios, we propose regularized reward modeling objective CET-BSR. excelling baseline methods in all generalization scenarios and improving accuracy in com- plex tasks over 5% in RMM-Bechh compared to simple BT model. Further analysis of the propagation of robustness in RMs to RLHF training shows BSR yields 40% decrease in generation length with 7% gain in AlpacaEval 2.0.
where S6 (Xi -) represents the scoring function specific to prompt zi. However, the reward model ro parameterized by the language model unifies these prompt-specific BT models by learning a single parameterization @ that maps the entire space of prompts and responses X' x y to preference scores.
Reward model parameterization Let M = M Mr be set of language models where each Mr defines a con- ditional distribution over responses given a prompt. For any prompt in the prompt set X' N each model Mj induces a prompt-specific response space (x)
2. Background
2.1. Preliminaries
aupp(Ma(11a))
Reward modeling with language models Reward mod- els (RMs) for language model alignment use the Bradley- Terry (BT) model (Bradley & Terry, 1952) for human pref- erence between chosen and rejected responses yoe and yı given a prompt x as:
y(+) = U )a()),
(5)
where V(x) represents all possible responses for prompt x across all models. Thus, rg can be represented as:
(x(
ro: x (xx) R. R. =
(6)
P (1v > /11z)= cr(2.0w) + cr(z.n))
(1)
where = (z(..)) E.
(7)
Typically, they are implemented as classifiers with projec- tion head W, (Ziegler et al., 2020; Ouyang et al., 2022):
This shared parameterization enables learning preference patterns that generalize across the prompt space while main- taining the BT structure within each prompt context.
r(z, v) = W M(,,)), W, W
(2)
2.2. Problem setup: robustness of reward models
h(z y) refers to the last hidden state from the backbone language model with the hidden dimension of H. given the prompt and response pair (z. y). In practice, both encoder- only (Jiang et al., 2023b) and decoder-only language models (Yuan et al., 2024; Liu et al., 2024a) are used as a backbone model that returns h(x, y). And W, is the projection head which the weights are randomly initialized by (0, (H + 1)-1) (Stiennon et al., 2020; Huang al., 2024).
Motivated by how RMs parameterize multiple prompt-level BT models into a single parameter 0, we expand the issue of reward model over-optimization (Gao et al., 2023) by categorizing generalization scenarios based on prompt dis- jointness and response disjointness. By doing so, we study how each generalization scenario of RM is affected by over-optimization and how to improve the robustness.
RMs are fine-tuned to maximize the margin between the chosen and rejected responses' scores (Christiano et al., 2017: Stiennon et al., 2020):
We follow how Gao et al. (2023) sets the synthetic gold RM instead of human annotations for controlled assessment. Let there exist a true preference model y* : X x (zz) + R. Let Musin C M and Myalid C M be the sets of models used for generating responses in the training and validation sets respectively, The corresponding response spaces are:
CBT -E -B( log (r(()))) + en(z.e))
where D is the set of triplets comprising chosen and rejected responses Yu and y1 given the fixed prompt z, and r(æ, y) E IR is score assigned to (x. v)-
Vinain (xx) = U (aa(())
(8)
Mr M.m.
(x)
U [l(z).
M.E.Mais
Preference modeling with BT model BT model is de- fined for directly comparable components (Bradley & Terry, 1952; Davidson, 1970; Huang et al., 2004), leaving the re- sponses Vi.j ~ pu[- (xz)) given the fixed prompt z; to be com- parable. For each prompt 24, there exists a BT model with prompt-specific parameters @; that defines the preference:
Thus, the train set Duran for training To can be defined as:
D = (22, Bhes B1) (1w,, v1) ~ Virain (x), x E X
(10)
First, prompt disjointness is defined as two prompt sets Xrain and X being disjoint. And response disjointness is defined as two response spaces Virain and Yeaid having
co, (2.181,))
P(y.1 > Vi.2/2; (b:) e (RiNica) ) + (2..N)))
(4)
disjoint response model sets Mmain and Mvaidd- From this context, we categorize RM generalization scenarios into:
2

---

On the Robustness of Reward Models for Language Model Alignment
1. In-domain (DI):: DID shares the same prompts with Diran and responses sampled from the same response model set M
varying sizes. We select Llama-3.2-IBB 3B, and Llama-3.1- 8B base models from the Llama-3, along with Qwen2.5- 1.5B, 3B, and 7B base models from the Qwen2.5. We use UltraChat (Ding et al., 2023, UC) to conduct supervised fine- tuning (SFT) for every model. Detailed SFT and reward modeling training configurations are in Appendix A.
((r, yoe 31 (y yt) ~ Virain()) (r), x E X
(11)
2. Prompt-disjoint (D~-Prompt): re being generalizable to the unseen prompt set Xaatia but with the same response model set M
Datasets We adopt UltraFeedback (Cui et al., 2025, UF), which harnesses 17 different models with varying model families to sample four responses per prompt, to set one train set and four validation sets as described in Section 2.2.
{2x, yws yt) (Ywr 31) ~ Virai(()) (x), x E Xealia)
(12)
3. Response-disjoint (D~ (e re being generalizable to the seen prompt set Krain but with the unseen response model set Mvalid:
We set the original 17 models as Musin and select four dis- joint models as Muatia: Gemma-2-22-- (Team et al., 2024), 1on02-7B-Instruu((OLMM) et al., 2025), SmoILM2-1.77 1.7B- Instruct (Allal et al., 2025), and DiistralIInstrutt. v0.2 (Jiang et al., 2023a). We excluded the Llama and Qwen families to avoid contamination with the training models and Musinn After generating four more responses per prompt with them, we prepare train set Duain with 51,200 rows and four val- idation sets: DID. Promppi, DD Respansee and DD We report detailed preprocessing procedures in Appendix B.
{2x, ywn 91) (Vw, 31) ~ (valia()); x E Xuainn
(13)
4. Mutual-disjoint (D~Mutaal:: ro being generalizable to both unseen prompts Avalid and response models Mvalid:
{Z,, yw, (1 y1) ~ (r), E Meatia) (14)
From this context, we define RM over-optimization as RM's accuracy on Duam and DId increasing, while its performance on Prommoppe D-RResposee: and D.uMutual stagnate or degrades. Intuitively, equivalent to rg losing its alignment with r* in the general cases after training with limited samples.
3.2. Part 2: Propagation of RM robustness in RLHF
We simulate the impact of robustness in RM during re- inforcement learning with human feedback (RLHF) with RLOO (Ahmadian et al., 2024). We test if the trained model with each RM can align with the gold preference model r*.
2.3. Post-analysis: Reinforcement learning with human feedback
We then analyze the propagation of over-optimizztion in the downstream RLHF applications. By fine tuning the lan- guage model N with reinforcement learning (RL. algorithms to maximize the reward with respect to as human prefer- ence proxies, the policy æ is trained to maximize the average reward for its response y ~ T (-1xx) given the prompt æ ~ D:
Models and dataset We set the Qwen2.5-115B base model with supervised fine-tuning (SFT) applied using UC as an initial policy. We employ Qwen2.5-3B based RMs trained with LeT and LET.SR from Section 3.1, they demon- strated the highest overall performance in Figure 5 for both methods. We use Xvalid as the seed prompts for experiments. We report the additional training configurations in Table 4.
max (ywn()))) [ro( v)] 3l.. (#(w):r)) mer x))
(15)
To understand the impact of over-optimization in RLHF, we assess the gold reward scores of interim π during the training. Doing so, we verify if maximizing r(z, y) is aligned with maximizing r" (x.
3.3. Part 3: Real-world impact of robustness in RM
Finally, we extend our experiments in Sections 3.1 and 3.2 to the 8B model and high-quality synthetic preference data to demonstrate the scalability and effectiveness of the proposed method.
3. Experiments
Models and datasets We use TULU3 SFT mixture (Lam- bert et al., 2024) to conduct SFT on Qwen2.5-1.5B, Then, we employ Llama-3.1-88 based RMs trained with LBT and LAT.SRR on Skywork- high-quality synthetic preference dataset (Liu et al., 2024a; 2025b). We use the same Xvalid as the seed prompts for RLOO. We report the additional training configurations in Table 4.
We set ArmoRM (Wang et al., 2024b) as the gold preference model r", a setup similar to that of Gao et al. (2023). We selected ArmoRM as it utilizes a wide range of representa- tive synthetic preference data and is reported to be robust to various biases (Wang et al., 2024b; Meng et al., 2024).
3.1. Part 1: Reward model over-optimization
Models We use two different model families. Llama-3 (Dubey et al., 2024) and Qwen2.5 (Yang et al., 2024a), with
httpa://hugginggaace wtt.:e/sseaaaaaaa//aeaeaaaeeeeee Skyworkk/ Boyyork-Rewarre www.eeewwwweeeeeeeeeeeeeee 80K-v0.2
3

---

On the Robustness of Reward Models for Language Model Alignment
40
1.005
W
35
10
1.000
[|nxx -nx yill
[[ ntx, nix.yvill
20
100 dp
[W,1
0.995
25
30
20
15
0.985
10
0.980
Qwen2.5 (1.58) Qwen2.5 (3B) Llama-3.2 (1B) Lamm-3322 (3B)
0.2
0.4
0.6 Epach
0.8
1.0
0.2
0.4
0.6 Epech
0.8
1.0
Figure 1. | distribution after reward modeling for four seeds
each. || generally stays around one after the training.
(a) Qwen2.5 (3B)
(b) Llama- 3.2 (3B)
Figure 2. Growth of [[f(z, Mw) h(2, yr throughout reward modeling with Car. The variance of the hidden state difference grows incrementally with the right-skewed distribution. The width of the colored area indicates the standard deviation.
4. Robustness and Over-optimization in RMs
In this section, we propose excessive dispersion the hid- den states of the reward model (RM) a cause over- optimization. We support our point by analyzing the train- ing dynamics of Lut in Section 4.1. Then, we propose batch-wise sum-to-zero regularization (BSR) as a method to mitigate such dispersion, enhancing the robustness of RMs.
the gradient updates for W, can be written as:
θCgT - (1-oo(rr)
AAr (19)
aw,
aw,
hh1, h(x,)1),
(20)
4.1. Hypothesis
awp
Given the prompt and response pair (z, y), the final score of RM ra as a dot product between the projection head W, and the hidden state h(x, y) can be decomposed into:
where oxx) (1 + exp (-x)) is the sigmoid function. As (- = 1 - (rr)
0LBT
aw, - -o(-Arr) (h(x, ym) (xx (21)
re(a, v) = |W 114(2, v)ll cos v,
(16)
Then, the gradient norm for W is:
where cos v represents the cosine similarity between the two vectors W, E "xx1 and (ax y) E R Typically, the norm of two vectors ||W ||/(2, v)ll largely contributes to maximize the softmax value and cause over-confidence issue (Wei et al., 2022), especially in the context of large language models (LLMs) that have large hidden sizes (Yang et al., 2024a; Dubey et al., 2024). Inspired by Wei et al. (2022), we hypothesize that excessive growth of hidden state norm dispersion is major cause of RM over-optimization.
dLBT
= or-Ar) [1((,, Mhw - 1(2, will.
(22)
ow,
Having (r-Ar) and |||(x, Moo) - h(r, will as main com- ponents, the gradient norm of W, saturates as CBT is min- imized by maximizing Ar. Despite Ar in the initial phase of training, ||h(,,, ynu) h(r,, yr)l minimal the beginning the output hidden states of large language mod- els (LLMs) (i.e., backbone LM of RMs) have low effective rank, being concentrated to certain regions (Biś et al., 2021; Wei et al., 2024).
Chosen and rejected responses share projection head Within three components in Equation (16), we first analyze the contribution of W, to over-optimization. Let Ar the reward margin r(z, Vu.) r(x, 21) for the prompt x and chosen and rejected responses yo and yr:
Thus, ||W,l| is expected to have marginal difference with its initial value, which E [|]l 1 by 100,1/(H + 1)) initialization (Stiennon al., 2020). We support this through Figure 1, average ||W,ll WI across four seeds for each model after training with Lat actually stays near 1.
Ar WT (M(r, (w) M((,,
(17)
We Norm variance inflates in hidden states with LeT
continue by analyzing the impact of [|h(,, 3)ll over- optimization. ming LBT can be achieved through maximiz- ing Ar. the inner term of sigmoid function:
As Equation (3) is equivalent to:
Ler -E DDD (logo(Ar)) o (Ar))],
(18) Aro W |1 h yv)) ho ut)ll cos tha, (23)

---

On the Robustness of Reward Models for Language Model Alignment
RM
RM
RMar
RMar-asit
Desh
Dunn
Do
D- Fromet
Fromet
Rassoms
Rapoms
D-Mutul
D
174
176
178
180
182
184
186
90.75
91.00
91.25
91.50
91.75
92.00
92.25
92.50 92.75
|]htt,y011
(a) Qwen2.5 (3B)
(b) Llama 3.2 (3B)
Figure 3. Hidden state norm dispersion comparison between RMET and RSR Batch-wise sum-to-zero regularization (BSR) alleviates hidden state norm dispersion and demonstrates consistent range of norms across different generalization scenarios Section 2.2. Reducing the variability of hidden state norms, Lasx improves the robustness of RMs for unseen data.
when cos 1/ is the cosine similarity between Wp and ho(2, yo:) ha( W1.. Having ||W stays near its initial value of one, CET encourages 0 to increase | (z, Bw) (x, 3t)l and cos A.
skewed by enforcing the reward sum to zero, constraining the norm dispersion and outliers in Figure 2.
In detail, the gradient for fa(z, yw) and h(x, 31)) are symmet- ric, having same magnitude but opposite directions:
Over-confidence and over-optimization in RMs In multi-class classification task, Wei al. (2022) points out that excessive growth of logit magnitude in classifiers causes overfitting in the training set: i.e., over-confidence issue. This is closely connected to the over-optimization problem as the reward models are simply classifiers opti- mized with two-class classification objective LBT- View- ing ro (2, y) (16) as a logit in a classifier, our analysis nar- rows the cause of growth in [|ro(ar, y)ll the growth of (1, vw) - (x yt)l as ||W remains near one.
θCBT Oh( ywo
-0 (-(Arr)
(26)
δCBT
ah ah(z, y1) =o(- Wr.
(27)
By CBT. || vo.) h(r, yt) incrementally increases as Figure 2. Mcantime, Casa penalizes outliers with large mag- nitude toward either positive or negative directions, shown through its gradient:
@CBSR Öh(2,, ve.))
(FE (ort. Wp,
In Figure 2, we track the growth of [ yw)) - while fine-tuning RM with Lar- Compared to that of hid- den states from the SFT model (Esooch 0), we observe a consistent increase the average norm and its variance, es- pecially strengthening the right-skewness. This aligns with how the over-confidence issue happens due to growing logit magnitude in typical classification tasks (Wei et al., 2022).
- ha 31)l
(28)
for prompt and response pair (1)) with i {1, {1...., |B]} and j E (w,1), preventing excessive growth as the gradient is proportional to r(a, This is straight- forward solution for norm variance inflation in Figure 2, as r(z, y) and its dispersion are largely influenced by [[h y)ll as discussed in Section 4.1, especially as ro( Vhe) and ro (2,31) y1) are positive and negative values.
4.2. Method: Batch sum-to-zero regularization (BSR)
From this context, we propose regularizing the reward sum of batch to zero to control the inflation in hidden state norm variance, namely batch sum-to-zero regularization (BSR):
We support this point through Figure 3, RM trained with LETSSR having significantly lower dispersion in ||,(z,)) y)1l- Comparing the variability of [|h(,, y)ll on four differ- ent generalization scenarios in Section 2.2, reward model trained with LAT.SR (RMBT.RRR) is robust to unseen prompt and responses, while the norm range differs with reward model trained with LBT (RMgT).
CBT-BRR = LBT + X LBSR
(24)
2
18|
LISR
r(I Vo.s)
(25)
2
JE(
4.3. Baselines
where À is the weight hyperparameter and B refers to the batch of (z, yoe v1). LASR penalizes r(x, y) from being
Based on Section 4.1, we adopt three algorithmic mitiga- tions as baseline methods to fulfill Equation (23).
5

---

On the Robustness of Reward Models for Language Model Alignment
0.84
Accuracy
Kendall's
0.630
0.925
0.020
0.78
0.575
Llama-3 11B)
Uama-3 138)
Qwin2.5 (2.58)
Qwer2.5 (38)
Llama-3 (18)
Liama-3 (38)
Qwer2:5 (3. 56)
Qwen2. (38)
(a) In-Domain (Dm))
(b) Prompt-disjoint (D>.Prompi)
0.825
Kandall's
0.500
Kendall's
0.25
0.525
0500
0.520
0.450
0.475
Llama-3.2 (18)
Uaama-322 138)
Qwen2 5 (1.58)
Qwen2.5 (38)
Uama-3 2 (181
Liama-3.2 (38)
Qwer (). (8)
Qwen2.5 (38)
(c) Response-disjoint (e>Resoose))
(d) Mutual-disjoint (D~Mutua))
Figure Assessing the rohustness each reward modeling ohjective on four generalization scenarios Section 2.2. By applying Chtbsr, downstream RMs are more robust to unseen prompt sets response generation model sets by being more aligned the gold preference model r", measured through accuracy and Kendall's 7 against the preference annotations r". Full results are reported Table
First, we can explicitly normalize rewards with their norm. Logit normalization (Wei et al., 2022) can be applied to LBT as it isolates classification objective to the directions of logits, cos vi, without being affected by [[h(a, v) by normalizing with the of rewards:
5. Results and Analysis
Through three steps, we analyze the significance of robust- ness in reward models (RMs) in reinforcement learning with human feedback (RLHF). Beginning with how each method affects robustness in RMs through over-optimization as- sessment in Section 5.1, we study how the robustness in RMs as proxies can boost the aligmment toward the true preference in Section 5.2. Finally, expand scope state-of-the-art data and models in Section 5.3.
r(z, - r( 31) Vr(u, Bwr + r(x, )1
LBT-Norm = - log
(29)
Second, we can set the hardbound of Ar to explicitly pre- vent the divergence in Equation (23). We add the hinge loss (Cortes, 1995) that provides hardbound m:
5.1. Part 1: Reward model over-optimization
In Figure 4, assess the robustness of reward models (RMs) trained with each method in Section 4.3 on four generalization scenarios discussed in Section 2.2. The full results are reported in Appendix C.
LBTe max (0, m (r(x, r(x, ())) (30)
Third, we test the extreme case of boosting the margin in Equation (23), which is expected to reinforce the over- confidence issue. We find the loss function in Yuan al. (2024) additionally boosting the margin with separate logis- tie losses for chosen and rejected responses, respectively:
RMs are prone to unseen response styles Comparing Figures 4b and 4d, most of the RMs were prone unseen response styles. While both D~Prompt and D~Manull shares the same prompt set unseen during the training, adding an unseen response model set Mvalid triggers more than 10% loss in Kendall's values. For instance, RMgT experienced 16.4% decrease, from T of 0.705 to 0.587. This implies that harnessing diverse LLMs in synthetic preference datasets is crucial for the general use RMs.
LATDR LBT - log a (ro(a, )) log o--ra v1)).
(31)
Based on our analysis in Section 4.1, we hypothesize that LETDD will underperform compared to Cet in all generaliza- tion scenarios by booting the hidden state norm dispersion.
6

---

On the Robustness of Reward Models for Language Model Alignment
115
145
RMy
RM/grist
RMlre
0.11
Goid EMA Itoeart (Ncemad
2.0
4.30
032
AM90
Nanord
023
8,00
048
A880
A2 0. 00 020
19 Lis 15 125
Ado 25
4.5
30 2.5
25
2.5 50 425
Dusin/lo-))
Eheccpp
mo
(a)
(b)
(c)
(d)
Figure 5. Reward to KL divergence plots (Figures 5a and 5b) and alignment to gold reward model r" (Figures 5c and 5d) while fine-tuning
Qwen2.5-1 1.55B SFT model as wa with RLOO using RMa and RMEraSR, respectively. We measure the alignment with the gold preference model r" to verify if reward maximization with each RM as a proxy is leading to maximizing r" scores in Figures Sc and 5d. Using
RMRTasR consistently improved the reward assessed by r", while RMgr stagnated in the last half of the training.
BSR enhances the robustness of RMs In Figure 4, RMBT-BSR best aligns to r" across all generalization scenar- ios. Meantime, the accuracy of RMET-DR, which is expected to have even stronger norm dispersion, is consistently lower than RMRr, evidently supporting our hypothesis. With Fig- ure 3, we inspect that [1, y)ll remaining stable with un- seen data is a crucial condition for robustness in RMs.
Table I. Effective rank analysis for RMAT and RMRE-SS by col- lecting the hidden states of each model. While RMBT structures the train data into a lower rank, the effective rank significantly increases on the unseen evaluation dataset from RM-Bench.
erankmai
erann
A (1)
RMET
23.34
33.76
+10.42
Also, RMBT-Hinge occasionally exceeded RMgr in Figures 4c and 4d. Recall that Lin. sets hardbound to restrict the divergence in Equation (23); such result supports that the growth of [1, w)ll is major cause of over-optimization as discussed. However, explicit logit normalization with Nom underperformed in every scenario. This implies that [[, y)ll shouldn't be excluded in reward prediction, highlighting the necessity of soft regularization.
R.M.B
33.45
33.59
+0.14
sensitive to the unseen data.
BSR and stable reward maximization Through Figures Sa and 5b, compare the stability in reward maximization by tracking the divergence of Ty. With RMBT. initial reward maximization required a leap in KL divergence, as shown in the 0.0 to 7.5 range in Figure 5a. On the other hand, RMBT-RSR induced continuous exponential growth in reward while Thter diverged. Along with steady increase in r* score, this implies that RMgT-BSR yield stable training.
Finally, we observe that the performance gap between RMBT and RMgT-BSR is amplified the model size increases in all four plots, comparing within the same model families. We further study this in Section 5.3 with Llama-3.1-88
5.2. Part 2: Propagation of RM robustness in RLHF
5.3. Part 3: Real-world impact of robustness in RM
We study how robust RMs improve RLHF tasks by using RMET and RMET-SSK for RLOO training in Figure 5. We re- fer to TBT and Bsr to be the two RI Ri-Oined policies with Qwen2.5-3B based RMgT and RMBT-BSR.
We apply our method and insights to the state-of-the preference dataset and the most widely practiced base model, Skywork- wrework-Preeeeenees--eee v0.2 (SKP) and 1lama-3.1-8B, where we do not have access to the true preference model. For RMgr, we use the official checkpoint of Liu et al. (2024a)?.
Robustness in RMs lead to better alignment of ι with x* In Figures Sc and 5d, we observe wBT.bSR being better aligned the gold preference model r* (i.e., ArmoRM). While твт and TBT-BSR both incrementally maximized the reward in terms of RMST and RMETESSK Figure 5d depicts that the scores stagnate in the last half for RLOO training for тBT. This could imply RM8T providing an imperfect reward signal in the high-reward region, where [xx y)ll tends to be relatively large. This aligns with the observa- tion Figure& where the average [[,((,,) y) of RMBT was
Parametric robustness analysis for RMET and R.ET.BRRR Along with RM-bench, we employ the effective rank (Roy & Vetterli, 2007: Wei et al., 2024, erank) as an parametric analysis of robustness. The large discrepancy between the erank of the train and evaluation sets indicates that the model fails to capture information that does not exist in the train
sk Skywork SkyxorkkRewardd Reward Llama- 1. 88

---

On the Robustness of Reward Models for Language Model Alignment
Table 2. RM-Bench comparison between RMs trained with RMaT and BTEEess with varying À E {10-2, 1003 10-"). By setting À = 0.
BTREES equivalent to RMET. As gets larger, accuracy hard tasks with subtle differences increases, consistently exceeding RM
RM-Bench
Chat
Math
Code
Safety
Hard Acc
Normal Acc
Easy Acc
Overall
RMBT (A = 0)
69.2
62.1
53.4
95.9
47.8
74.0
88.4
70.1
RMBT-BSR (A = 10 10-4)
70.0
61.4
52.9
95.1
47.9
74.3
87.4
69.9
RMgT-aSR (A = 10 10-3) 3)
70.1
61.2
53.9
95.4
52.1
73.5
85.0
70.2
RMBT.RRR (A- 10
70.0
60.1
53.1
96.0
55.7
72.5
81.9
70.0
set but in the general corpus: i.e., overfitted to the train set. Given a dataset 'D with N rows, we collect set of hidden states H E RN* To compute the effective rank of H, let Q = min(N, N, H). and denote the singular values of H by 01 , QQ we have:
Table 3. AlpacaEval 2.0 average response length and length con- trolled win ate for the checkpoints throughout the RLHF process us- ing Qwen2.5-1.55 with SFT on TULU3 mixture. Using RM5T+aSR significantly reduces verboseness while being best aligned.
Qwen2.5-1.5B
Length
LC AE2.0 (%)
SFT
2247
2.59
erank( exp
log
RLOO (RMBT)
2180
8.41
1=1
04
si=l 03
+ +RLOO (RMETHUsk)
1337
9.02
(32) In Table 1, we compute the erank of RMgT and RMBT-BSR by collecting their hidden states for SKP (erankmii and
RM- RMM-Bnnch (erankwat). While RMET-SSR retains erankqain in the evaluation set, RMBTESR experiences around 40% increase. Thus, RMBT is prone to over-optimization, as previous analyses show. Furthermore, this makes RMs even more vulnerable dataset biases, such as verbosity bias (Saito et al., 2023; Zhang al., 2024b; Chen et al., 2024a).
et al., 2024) is used as LLM-as-a-judge. In Table 3, TBT-SR demonstrates the highest L.C win rate with a significantly shorter response than the SFT model and TBT.
As preferring verbose response is chronic problem in gen- erative evaluators (Zheng et al., 2023; Dubois et al., 2024), reducing generation length by 40% compared to SFT model while achieving the best win rate in Table 3 highlights how RMgT-BSR could be advantagcous for preventing policies falling into local minima while maximizing the reward: i.e., reward gaming (Skalse et al., 2022; Pang et al., 2023; Chen et al., 2024b). Comprehensively. results underscore the effectiveness of LET-BSR for the overall RLHF pipeline.
BSR captures subtle preference factors When r* is not available, we evaluate RMs through RM-Bench (Liu et al., 2025b), which provides preference pairs with subtle dif- ferences and achieves higher correlation against actual use cases than RewardBench (Lambert al., 2025).
In Table 2, we observe RM's accuracy consistently increas- ing in the response pairs with subtle differences ("Hard Acc") as À gets larger. Liu et al. (2025b) reports that hard accuracy of RM in RM-Bench has the highest correlation with the downstream policy's s actual performance, while the correlation was near zero for easy accuracy. Along with the fact that the over-confidence issue hinders accurate classifi- cation in hard tasks (Wei et al., 2022), inspect that RM3T fails to capture subtle differences in their representations, also supported by significant increase in erank for unseen data in Table 1.
6. Related Works
Over-optimization of reward models As formally out- lined by Gao et al., 2023, reward ver-ooptiiization refers to the phenomenon where the reward model (RM) fails to generalize to the gold objective due to excessive training, generalizability some works
To enhance have proposed ensembling RMs (Gleave & Irving, 2022; Zhai et al., 2023; Coste et al., 2024), meta-learning RMs on the shifted target distribution (Wang et al., 2024a), aug- menting preference data for RMs (Liu al., 2025a), jointly learning the text-generation loss in reward modeling (Yang et al., 2024b), constraining the preference optimization process itself (Moskovitz et al., 2024; Zhang et al., 2024a; Liu et al., 2024b; Gupta et al., 2025). Interestingly, Rafailov et al., 2024 outlined how a similar phenomenon can be seen in Direct Alignment Algorithms (DAAs), where the implicit rewards of the policy replace RM. On the other hand, some
For final RLHF training, we select R.M.ERRRRRR with À = 10 in Table 2, regarding a balance between the hard and easy tasks along with the overall scores. We report the reward logs during the training for both in Appendix D.
R.E..SR boosts AlpacaEval without verboseness We evaluate тEт and TETeSR through length-controlles (LC) Alpacaeval 2.0 (Dubois et al., 2024), which GPT-4 (OpenAl
8

---

On the Robustness of Reward Models for Language Model Alignment
applications on multiliguality (Wu al., 2024; Hong al., 2025) have suggested how RMs can possess cross-lingual transfer to languages unseen during reward modeling.
00398115, Technology research ensure authenticity and consistency of results generated by AI) and (RS-2019- I1190075, Artificial Intelligence Graduate School Program (KAIST)).
Evaluating reward models Parting from assessing the accuracy of the validation sets of popular preference datasets (Stiennon et al., 2020; Bai et al., 2022), Lambert et al., 2025 formalized a reproducible toolkit to evaluate reward models for enhanced explainability in diverse preference domains and tasks. Subsequent works have addressed improvements in cases of handling more sensitive, subtle cases (Liu al., 2025b) and in wider coverage of rorldworld scenarios (Zhou et al., 2025) or specializations in tasks such as mathematical reasoning (Kim et al., 2024). Frick et al., 2024 introduced an evaluation pipeline directly targeted at assessing the RM in its role in the reinforcement learning with human feedback (RLHF) pipeline.
References
Ahmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., and Hooker, S. Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computaaionnal Linguistics (Volume 1: Long Papers), pp. 12248-12267, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https: /aalaaaaaaannooooooaooaaaa org/ 2024. acl-long 662.
Allal, L. B., Lozhkov, A., Bakouch, E.,, Blázquez, G. M., Penedo, G., Tunstall, L., Marafioti, A., Kydliček, H., Lajarín, A. P., Srivastav, V., Lochner, J., Fahlgren, C., Nguyen, X.-S., Fourrier, C., Burtenshaw, B., Larcher, H., Zhao, H., Zakka, C., Morlon, M., Raffel, C., von Werra, I.., and Wolf, Smollm2: When smol goes big data- centric training of small language model, 2025. URL https: /atp:////aaivvaa/a org/ abs///as///// /tw.bs//2522000 02737.
Conclusion
This study outlines why reward models (RMs) trained with the Bradley-Tene Terry (BT) model loss can be vulnerable to over- optimization issues, losing generalizability to unseen tasks. We highlight the dispersion in the norm of hidden states in RMs as primary source of such issue with theoretical anal- ysis and empirical demonstrations across model families and sizes. Then, we propose batch-wise sum-to-zero regu- larization (BSR), an add-on to the BT model that penalizes the rewards for having an abnormally large magnitude. We present threefold experiments throughout the overall RLHF pipeline, starting by assessing the robustness of BSR and four baseline methods through the alignment against the synthetic gold preference model. Then, we RLHF training with RLOO using RMs trained with BT model and BSR, re- spectively. We observe a stronger alignment of the resulting policy after RLOO against the gold preference model. Even- tually, we expand the experiments in 8B size model with high-quality preference data, where RM with BSR surpasses the ttate-of-the-an RM in 8B size with 7% improvements.
Almazrouei, E., Alobeidli, H., Alshamsi, A., Cappelli, A., Cojocaru, R., Debbah, M., Étienne Goffinet, Hesslow, D., Launay, J., Malartic, Q., Mazzotta, D., Noune, B., Pannier, B., and Penedo, G. The falcon series of open language models, 2023. URL https: htttps://arxine http///xxvvv.rr.//// abs 22311116186
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.0586. 2022.
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O'Bri,,, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for ana- lyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Impact Statement
This paper aims to identify the causes of reward model over-optimization and suggests a simple method to enhance robustness in reward models. As reward models are proxies for human preferences in language model alignment, the proposed method has the potential to impact our society, none of which we feel must be specifically highlighted here.
Bis, D., Podkorytov, M., and Liu, X. Too much in common: Shifting of embeddings in transformer language models and its implications. In Toutanova, K., Rumshisky, A.. Zettlemoyer, L., Hakkani-Tur D.,, Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., and Zhou, Y. (eds.), Proceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Lin- guistics: Human Language Technologies, pp. 5117- 5130, Online, June 2021. Association for Computa-
Acknowledgement
This work was supported by Institute for Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT)) (RS-2024-
9

---

On the Robustness of Reward Models for Language Model Alignment
tional Linguistics. doi: 10.18653/ 403. URL https: htt///alaaan /ttaa//aaaaooollooooo////// naacl main. meiin.4033 403/.
Coste, T., Anwar, U., Kirk, R., and Krueger, D. Reward model ensembles help mitigate overoptimization. In The Twelfth International Conference on Learning Represen- tations, 2024. URL https: oppnrevieeeeeeeeeeeeeeeeee / net/ forum? d=deiiddddjjjtpp..s.
Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952. ISSN 00063444. URL http://wwe hww.:jstor... jstor. stable/2334023
Cui, G., Yuan, I.., Ding, N., Yao, G., He, B., Zhu, W. Ni, Y., Xie, G., Xie, R., Lin, Y., Liu, Z., and Sun, M. Ultrafeedback: boosting language models with scaled ai feedback. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org. 2025.
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J.. Rando, J.. Freedman, R., Korbak, T., Lindner, D., Freire, P., Wang, T. T., Marks, S., Segerie, C.-R., Car- roll, M., Peng, A., Christoffersen, P. J., Damani, M., Slocum, S., Anwar, U., Siththaranjan, A., Nadeau, M., Michaud, E. J., Pfau, J., Krasheninnikov, D., Chen, X., Langosco, L., Hase, P., Biyik, E., Dragan, A., Krueger, D., Sadigh, D., and Hadfield-Menell, D. Open problems and fundamental limitations of reinforcement learning from human feedback. Transactions on Machine Learn- ing Research, 2023. ISSN 0835-8856 URL https: / open ew net/ forum? d=bx2 4 KpJ4Eb. Sur- vey Certification, Featured Certification.
Davidson, R. R. On extending the bradley-terry model accommodate ties in paired comparison experiments. Journal of the American Statistical Association, 65(329): 317-328, 1970. ISSN 01621459, 1537274X. URL http: //www. wwww.jstor.. jstor. org/ st sttbae///eeeeeee
Ding, N., Chen, Y., Xu, B., Qin, Y., Hu, S., Liu, Z., Sun, M., and Zhou, B. Enhancing chat language mod- els by scaling high-quality instructional conversations. In Bouamor, H., Pino, J., and Bali, K. (eds.), Pro- ceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 3029-3051, Singapore, December 2023. Association for Computa- tional Linguisties. doi: 10.18653/v 183. URL https: http::ss//aaaaathhlloooo www.ttthhhooo..oo.o....o.. 2023. emnlp mmnlp-main... 183/.
Chen, G. H., Chen, S., Liu, Z., Jiang, F., and Wang, B. Hu- mans or LLMs as the judge? study on judgement bias. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, pp. 8301-8327, Miami, Florida, USA, November 2024a. Association for Computational Linguistics. doi: 10.18653/w1/2024. emnlp-main.474 URL https:: /aaaaaaaanholloggoo/o org/2024 emnlp main. 474/.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravanku- mar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roziere, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C.,, Keller, Touret, C., Wu, C., Wong, C., Ferrer, C. C., Nikolaidis, C., Al- lonsius, D., Song, D., Pintz, D., Livshits, D., Esiobu, D., Choudhary, D., Mahajan, D., Garcia-Olano, D., Perino, D., Hupkes, D., Lakomkin, E., AlBudawy, E., Lobanova, E., Dinan, E., Smith, F. M., Radenovic, F., Zhang, F, Synnaeve, G., Lee, G., Anderson, G. L., Nail, G., Mi- alon, G., Pang, G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, I. A., and et al, I. K. The llama 3 herd of models, 2024. URL https: / http://axvvvvv//// org/ abs 24..22177......... 21783.
Chen, L., Zhu, C., Chen, J., Soselia, D., Zhou, T., Goldstein, T., Huang, H., Shoeybi, M., and Catanzaro, B. ODIN: Disentangled reward mitigates hacking in RLHF In Forty- first International Conference on Machine Learning, 2024b. URL https: / hap.nrpeeeeveeeeee net/ forum? id id=zcIV8OQEVE.
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:/ hemps:///mmae //tm:///mmmsmm// org/or/g//ooo... /blog/ 1023-03-30-vit cinna/
Dubois, Y., Liang, P., and Hashimoto, T. Length-controllld controlled alpacaeval: simple debiasing of automatic evalu- ators. In First Conference Language Modeling, 2024. URL https: httpp/pppwweeeeee/o//p//. i id=CybBmZWBXO.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
hwwp:nrwwwwwieeeeeeeeeeeeeeeeeee net forum?
Cortes, C. Support- networks. Machine Learning, 1995.
Frick, E., Li, T., Chen, C., Chiang, W.-L., Angelopoulos, N., Jiao, J., Zhu, B., Gonzalez, J. E., and Stoica, I.
10

---

On the Robustness of Reward Models for Language Model Alignment
How to evaluate reward models for rlhf, arXiv preprint arXiv:2410. 148742 2024.
2024, URL https:/ /ttpa////////eeeee////// net/ id=kHo2ZTaBe3.
forum? fer.iirmmmmee/mmmmm
Gao, L., Schulman, J.. and Hilton, J. Scaling laws for reward model overoptimization. In Krause, A., Brunskill, E., Cho. K., Engelhardt. B., Sabato, S., and Scarlett, (eds.), Proceedings of the 40th International Conference on Ma- chine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 2023. URL http::ttt//t// http//peoednngssseeeeeppppppssss ngs. mlr. press// v202/gao23h html.
Huang, T.-k., Lin, C.-j, and Weng, R. A general- ized bradley-terry model: From group competition to individual skill. In Saul, L., Weiss, Y., and Bottou, L. (eds.), Advances in Neural Informa- tion Processing Systems, volume 17. MIT Press, 2004. URL https://proceeddngg.. neurips.. cc /cc/spper_fiilesppperr/22///// 22ffccdf00390077caffed3333333333 pdf.
PMLR, 23-29 Jul
Paper.
Gleave, A. and Irving, G. Uncertainty estimation for lan- guage reward models. arXiv preprint arXiv:2203.07472, 2022.
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C.,
Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-
Gupta, A., Tang, S., Song, Q. Zhu, S., Hong, J., Saha, A., Gupta, V., Lee, N., Kim, E., Zhu, J., Pillai, N., and Keerthi, S. S. AlphaPO reward shape matters for LLM alignment. In Forty-second International Conference on Machine Learning, 2025. URL https://aaxii: hrg/abs/2ee ///aabb///111 2501. 03884.
A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023a.
Jiang, D., Ren, X., and Lin, B. Y. LLM-blender: Ensembling large language models with pairwise ranking and genera- tive fusion. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (cds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 14165-14178, Toronto, Canada, July 2023b. Association for Computational Linguistics. URL https:
Hong, J., Lee, N., and Thore,, J. ORPO: Monolithic pref- erence optimization without reference model. In Al- Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceed- ings of the 2024 Conference on Empirical Methods in Natural Laguage Processing, pp. 11170-11 189, Miami, Florida, USA, November 2024. Association for Compu- tational Linguistics. URL https:ttp////...... /ttpss///ccll /a//a/aaaannoooooo.o.oooo org/ 2024 sannp-mminn.... 626.
doi: 10. 10.18653/1//00233 792. //aclantho wwgp:/oooorrg//222333 wttp://222..ccl11e... acl- awl.acll..lo11111 792/.
Kendall, M. Rank Correlation Methods. Griffin books on statistics. Hafner Publishing Company, 1962.
Hong. J., Lee, N., Martínez-Castaño. R., Rodríguez, C., and Thorne, J. Cross-lingul lingual transfer of reward models in mul- tilingual alignment. In Chiruzzo, L., Ritter, A., and Wang. L. (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technolo- gies (Volume 2: Short Papers), pp. 82-94, Albuquerque, New Mexico, April 2025, Association for Computational Linguistics. ISBN 979-8-89176-190-2 URL https:// acl aataholoyyyy......ooooooolloooot wtt. 2225. naacl short. 8/.
URL https:/ //tokk///okoooo/ooo http://ooooo..ogll.cooo id=1whKAAAAMAA
wwwgle.oooooooooooooooooo co. kr/books?
Kim, S., Kang, D., Kwon, T., Chae, H., Won, J., Lee, D., and Yeo, J. Evaluating robustness of reward models for mathematical reasoning, 2024. URL https::////ee //arxiv.. org/ abs/ / 2210/411.0000 01729.
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Effi- cient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP '23, pp. 611-626, New York, NY, USA, 2023. Associa- tion for Computing Machinery. ISBN 9798400702297. doi: 10.114453600000666611666 URL ht https: /doii org / 10, 114 14411 0ee.111///600000000 361000000006616....
Hsu, P.-L., Dai, Y., Kothapalli, V., Song, Q. Tang, S., Zhu, S., Shimizu, S., Sahni. S., Ning, H., and Chen, Y. Liger kernel: Efficient triton kernels for Ilm training, 2024. URL https: /ttp///axxvvvv//v///////eeeeeaaa org/ abs 2410.4210.0 10989.
Hu, J., Wu, x., Zhu, Z. Xianyu, Wang, W., Zhang, D., and Cao, Y. Openrlhf: An easy-to-use, scalable and high-performance rihf framework. arXiv preprint arXiv:2405.11 1143, 2024.
Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, J. V., Liu, A., Dziri, N., Lyu, S., Gu, Y., Malik, S., Graf, V., Hwang, J. D., Yang, J., Bras, R. L., Tafjord, O., Wilhelm, C., Soldaini, L., Smith, N. A., Wang, Y., Dasigi, P., and Hajishirzi, H. Tulu 3: Pushing frontiers in open language model post-training, 2024. URL wwtph:tpps/......... //tp//axvvvv//// www...iwr///...e..... wwwgg///////////// 1512..
Huang, S., Noukhovitch, M., Hosseini, A., Rasul, K., Wang, W., and Tunstall, L. The nt implementation details of RLHF with PPO: case study on TL.DR summa- rization. In First Conference on Language Modeling,

---

G., Parish, J., Parparita, E., Passos, A., Pavlov, M., Peng, On the Robustness of Reward Models for Language Model Alignment
Lambert, N., Pyarkin, V., Morrison, J., Miranda, L. J. V., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N. A., and Hajishirzi, H. RewardBench: Evaluating reward models for language modeling. In Chiruzzo, I.., Ritter, A., and Wang, I.. (eds.), Findings of the Association for Computational Linguistics: NAACL 2025, pp. 1755-1797, Albuquerque, New Mexico, April 2025. Association for Computational Linguisties. ISBN 779-8-89176-195-77 httpttss:: /tap:ss///cllaaaae //a//allaatttt /atp/aaaaaannn....../ org/2025 f.25.ffnnn www.ffinnnng.... aaaaacc.....
Representations, 2025. URL https: /opp://////eereeee.ee.mom
OLMo, T., Walsh, P., Soldaini, L.., Groeneveld, D., Lo, K., Arora, S., Bhagia, A., Gu, Y., Huang, S., Jordan, M.. Lambert, N., Schwenk, D., Tafjord, O., Anderson, T., Atkinson, D., Brahman, F,, Clark, C., Dasigi, P., Dziri, N., Guerquin, M., Ivison, H., Koh, P. W. Liu, J., Malik, S., Merrill, W. Miranda, L. J. V., Morrison, J., Murray, T., Nam, C.. Pyatkin, V., Rangapur, A., Schmitz, M., Skjonsberg, S., Wadden, D., Withelm, C., Wilson, M., Zettlemoer, L.,, Farhadi, A., Smith, N. A., and Hajishirzi, H. 2 olmo furious, 2025. URL https://arx:: org/ abs 22001. 00656.
Liu, C. Y., Zeng, L., Liu, J., Yan. R., He, J., Wang, C., Yan, S., Liu, Y., and Zhou, Y. Skywork-Reward, Bag of Tricks for Reward Modeling in LLMs, Octo- ber 2024a. URL http: / arxiv. wrw. awwpaa//..1....... 18451. arXiv:2410.1845 [es].
OpenAL, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L.., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Bal- aji. S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M.. Belgum, J., Bello, I., Berdine, J., Bernadett-Shapin G., Berner, C., Bogdonoff, L., Boiko, O., Boyd, M., Brakman, A.-L.,, Brockman, G., Brooks, T., Brundage, M., Button, K., Cai, T., Campbell, R., Cann, A., Carey, B., Carlson, C., Carmichael, R., Chan, B., Chang, C., Chantzis, F., Chen, D., Chen, S., Chen, R., Chen, J., Chen, M., Chess, B., Cho, C., Chu, C., Chung, H. W., Cummings, D., Cur- rier, J., Dai, Y., Decareaux, C., Degry, T., Deutsch, N., Deville, D., Dhar, A., Dohan, D., Dowling, S., Dunning, S., Ecoffet, A., Eleti, A., Eloundou, T., Farhi, D., Fedus, L., Felix, N., Fishman, S. P., Forte, J., Fulford, I., Gao, L., Georges, E., Gibson, C., Goel, V., Gogineni. T., Goh, G., GontijoLoppss R., Gordon, J., Grafstein, M.. Gray, S., Greene, R., Gross, J., Gu, S. S., Guo, Y., Hallacy, C.,, Han, J., Harris, J., He, Y., Heaton, M., Heidecke, J., Hesse. C., Hickey, A., Hickey, W., Hoeschele, P., Houghton, B., Hsu, K., Hu, S., Hu, X.,, Huizinga, J., Jain, S., Jain, S., Jang, J., Jiang, A., Jiang, R., Jin, H., Jin, D., Jomoto, S., Jonn, B., Jun H., Kaftan, T., Lukasz Kaiser, Kamali, A., Kanitscheider, I., Keskar, N. S., Khan, T., Kilpatrick, L., Kim, J. W., Kim, C., Kim, Y., Kirchner, J. H., Kiros, J., Knight, M.. Kokotajlo, D., Lukasz Kondraciuk, Kondrich A., Konstantinidis, A., Kosic, K., Krueger, G., Kuo, V., Lampe, M., Lan, I., Lee, T., Leike, J., Leung, J., Levy, D., Li, C. M., Lim, R., Lin, M., Lin, S., Litwin, M., Lopez, T., Lowe, R., Lue, P., Makanju, A., Malfacini, K., Manning, S., Markov, T., Markovski, Y., Martin, B., Mayer, K., Mayne, A., McGrew, B., McKinney, M., McLeavey, C., McMillan, P., McNeil, J., Medina, D., Mchta, A.. Menick, J., Metz, L.. Mishchenko, A., Mishkin, P., Monaco, V., Morikawa, E., Mossing, D., Mu, T., Murati, M., Murk, O., Mély, D., Nair, A., Nakano, R., Nayak, R., Neelakantan, A., Ngo, R., Noh, H., Ouyang, L., Oeeefe C., Pachocki,
Liu, T., Xiong, W., Ren, J., Chen, L., Wu, J., Joshi, R., Gao, Y., Shen, J., Qin, Z., Yu, T., Sohn, D., Makarova, A., Liu, J. Z., Liu, Y., Piot, B., Itrycheriah, A., Kumar, A., and Saleh, M. RRM: Robust reward model training mitigates reward hacking. In The Thirteenth International Confer- ence on Learning Representations. 2025a. URL https: //openrevier n forum? i 8= 8ASMMQ
Liu, Y. Yao, Z., Min, R., Cao, Y., Hou, I.,, and Li, RM-bench: Benchmarking reward models of language models with subtltyy and style. In The Thirteenth In- ternational Conference on Learning Representations, n forum?
2025b. httpttps: /ttttps://pperreviii id=QEHrm@PBdd.
Liu, Z., Lu, M., Zhang, S., Liu, B., Guo, H., Yang, Y., Blanchet, J., and Wang, Z. Provably mitigating overop- timization in RLHF: Your SFT loss is implicitly an adversarial regularizer. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b. URL btttps::/opeereeeeee wwwprrvvvieieneeeoteeeeeee net/forum? id=2c031Phked
Meng, Y., Xia, M., and Chen, D. SimPO: Simple preference optimization with a reference-free reward. In The Thirty- eighth Annual Conference on Neural Information Pro- cessing Systems, 2024. URL https::////// http:://////peeeeeee/e// met/forumm2 dtt.r:mmmmiitmooooooooooo 1LKb.
Moskovitz, T., Singh, A. K., Strouse, D., Sandholm, T., Salakhutdinov, R., Dragan, A., and McAleer, S. M. Confronting reward model overoptimization with con- strained RLHF. In The Twelfth International Confer- ence on Learning Representationss, 2024. URL https: //openreview.oo net net wtt.://ormmdddooe hor.
Noukhovitch, M., Huang, S., Xhonneux, S., Hosseini, A., Agarwal, R., and Courville, A. Faster, more efficient RLHF through off pollcy asynchronous learning. In The Thirteenth International Conference on Learning
J., Paino, A., Palermo, J., Pantuliano, A., Parascandolo,
A., Perelman, A., de Avila Belbute Peres, F., Petrov, M.,
12

---

On the Robustness of Reward Models for Language Model Alignment
de Oliveira Pinto, H. P., Michael, Pokorny, Pokrass, M.,, Pong, V. H., Powell, T., Power, A., Power, B., Proehl, E.,, Puri, R., Radford, A., Rae, J., Ramesh, A., Raymond, C., Real, F., Rimbach, K., Ross, C., Rotsted, B., Roussez, H., Ryder, N., Saltarelli, M., Sanders, T., Santrrkrr, S., Sastry, G., Schmidt, H., Schnurr, D., Schulman, J., Selsam, D., Sheppard, K., Sherbakov, T., Shieh, J.. Shoker, S., Shyam, P. Sidor, S., Sigler, E., Simens, M., Sitkin, J., Slama, K., Sohl, I., Sokolowsky, B., Song, Y., Staudacher, N., Such, F. P.. Summers, N., Sutskever, I., Tang, J., Tezak, N., Thompson, M. B., Tillet, P., Tootoonchian, A., Tseng, E.,, Tuggle, P.. Turley, N., Tworek, J., Uribe, J. F. C., Vallone, A., Vijayvergiya, A., Voss, C., Wainwright, C., Wang, J.J Wang, A., Wang, B., Ward, J., Wei, J., Weinmann, C., Welihinda, A., Welinder, P., Weng, J.. Weng, L., Wi- ethoff, M.. Willner, D., Winter, C., Wolrich, S., Wong, H., Workman, L., Wu, S., Wu, J., Wu, M., Xiao, K., Xu, T., Yoo, S., Yu, K., Yuan, Q. Zaremba, W., Zellers, R., Zhang, C., Zhang, M., Zhao, S., Zheng, T., Zhuang, J., Zhuk, W., and Zoph, B. Gpt-4 technical report, 2024. URL https:://///mm /t////axxx arxiv. www.i.trg./.....aaa.........w.tt
Storage and Analysis, SC '20. IEEE Press, 2020. ISBN 9781728199988
Roy, O. and Vetterli, M. The effective rank: A measure of effective dimensionality. In 2007 15th European Signal Prucessing Conference, pp. 606-610, 2007.
Saito, K., Wachi, A., Wataoka, K., and Akimoto, Y. Ver- bosity bias in preference labeling by large language mod- els. In NeurIPS 2023 Workshop on Instruction Tun- ing and Instruction Following, 2023. URL https: / /popnrevvl ew net/ forum? dagaaiggaaaggggggg
Skalse, J. M. V., Howe, N. H. R., Krasheninnikov, D., and Krueger, D. Defining and characterizing reward gam- ing. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https: / openreview net/ Torum?1d=yb33o0003.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008-300 -3021, 2020.
08774.
Ouyang, I.., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 55:27730-277444 2022.
Taori, R. Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-followiing llama model.
Pang, R. Y., Padmakumar, V., Sellam, T. Parikh, A., and He, H. Reward gaming in conditional text gener- ation. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 4746-4763, Toronto, Canada, July 2023. Association for Computational Linguistics. URL https:
https://githutt www/tatsu-laeetteeecceac stanford_alpaces 2023.
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, I.,, Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C. L.., Jerome, S., Tsit- sulin, A., Vieillard, N., Stanczyk, P., Girgin, S., Momchev, N., Hoffman, M., Thakoor, S., Grill, J.-B., Neyshabur, B., Bachem, O., Walton, A., Severyn, A., Parrish, A., Ahmad, A., Hutchison, A., Abdagic, A., Carl, A., Shen, A., Brock, A., Coenen, A., Laforge, A., Paterson, A., Bastian, B., Piot, B., Wu, B.. Royal, B., Chen, C., Kumar, C., Perry, C., Welty, C., Chooueete- Choo, C. A., Sinopalnikov, D., Weinberger, D., Vijaykumar, D., Rogozińska, D., Her- bison, D., Bandy, E., Wang, E., Noland, E., Moreira, E., Senter, E., Eltyshev. E., Visin, F., Rasskin, G., Wei, G., Cameron, G., Martins, G., Hashemi, H., Klimezak- Plucińska, H., Batra, H., Dhand, H., Nardini, I., Mein, J., Zhou, J., Svensson, J., Stanway, J., Chan, J., Zhou, J. P., Carrasqueira, J., Iljazi J., Becker, J., Fernandez, J.. van Amersfoort, J., Gordon, J., Lipschultz, J., Newlan, J., yeong Ji, J., Mohamed, K., Badola, K., Black, K., Milli- can, K., McDonell, K. Nguyen, K., Sodhia, K., Greene, K., Sjoesund, L. L., Usui, L., Sifre, L.., Heuermann, L., Lago, L., McNealus, L., Soares, L. B., Kilpatrick, L., Dixon, I.,, Martins, L.., Reid, M., Singh, M., Iverson,
doi: 10. 10.18655/ww12222..accloogg //aclanthology wttp://ggg.rrg///....//3 acl- 1ww.......20 262/.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. In Thirty- seventh Conference on Neural Information Processing Systems, 2023. URL https: http::: https////peeeeee....oom Forum?i wis.:idd-HPPSSSSSaaa
Rafailov, R., Chittepu, Y., Park, R., Sikchi, H., Hejna, J., Knox, W. B., Finn, C., and Niekum, S. Scaling laws for reward model overoptimization in direct alignment algo- rithms. In The eiihty-eighth Annual Conference on Neu- ral Information Processing Systems, 2024. URL https: //openreview net/ fotr//ooummmmm/ wiiiidddffffouyy.
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: memory optimizations toward training trillion parame- ter models. In Proceedings of the International Con- ference for High Performance Computing, Networking,
13

---

On the Robustness of Reward Models for Language Model Alignment
M., Görner, M., Velloso, M., Wirth, M., Davidow, M., Miller, M., Rahtz, M., Watson, M., Risdal, M., Kazemi, M., Moynihan, M., Zhang, M., Kahng, M. Park, M., Rahman, M., Khatwani, M.. Dao, N., Bardoliwalla, N., Devanathan, N., Dumai, N., Chauhan, N., Wahltinez, O., Botarda, P., Barnes, P., Barham, P., Michel, P., Jin, P., Georgiev, P., Culliton, P., Kuppala, P., Comanescu, R., Merhej, R., Jana, R., Rokni, R. A., Agarwal, R., Mullins, R., Saadat, S., Carthy, S. M., Perrin, S., Arnold, S. M. R., Krause, S., Dai, S., Garg, S., Sheth, S., Ronstrom, S., Chan, S., Jordan, T., Yu, T., Eccles, T., Hennigan, T., Kocisky, T., Doshi, T., Jain, V., Yadav, V., Meshram, V., Dharmadhikari, V., Barkley, W., Wei, W., Ye, W., Han, W., Kwon, W. Xu, X., Shen, Z., Gong, Z., Wei, Z., Cotruta, V., Kirk, P., Rao, A., Giang, M., Peran, L., Warkentin, T., Collins, E., Barral, J., Ghahramani, Z., Hadsell, R., Sculley, D., Banks, J.. Dragan, A., Petrov, S., Vinyals, O., Dean, J., Hassabis, D., Kavukcuoglu, K., Farabet, C., Buchatskayy, E., Borgeaud, S., Fiedel, N., Joulin, A., Kenealy, K., Dadashi, R., and Andreev, A. Gemma 2: Improving open language models at a practical size, 2024. URL https: /tt//aaxvvvv.////////e//e/aaevvt orgg abs /200 00118.
von Werra, L.,, Belkada, Y., Tunstall, L.,, Beeching, E., Thrush, TT, Lambert, N., Huang. S., Rasul, K., and Gallouédec, Q. Trl: Transformer reinforcement learn- ing. https: https://github. http///gttttb.teee 2020.
Wang, B., Zheng, R., Chen, I.., Liu, Y., Dou, S., Huang, C., Shen, W., Jin, S., Zhou, E., Shi, C., Gao, S., Xu, N., Zhou, Y., Fan, X., Xi, Z.., Zhao, J., Wang, X., Ji, T., Yan, H., Shen, L.., Chen, Z., Gui, T., Zhang, Q. Qiu, X., Huang, X., Wu, Z., and Jiang. Y.GG. Secrets of rlhf in large language models part ii: Reward modeling, 2024a. URL https: arxiv. org/ abs 24011 06080.
Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang, T. Interpretable preferences via multi-objective re- ward modeling and mixture-of-experts In Al-Onaizan, Y.. Bansal, M., and Chen, Y.-N. (eds.), Findings of the Association for Computational Linguisties: EMNLP 2024, pp. 10582-10592, Miami, Florida, USA, November 2024b. Association for Computational Linguisties. doi:
620. URL https: / http://aaaaaathhhc f Lindings-emnlag 620/.
org/ 2024.
Team, M. N. Introducing mpt-30b: Raising the bar for open-source foundation models. Blog, 2023.
Wei, H., Xie, R., Cheng, H., Feng, L., An, B., and Li, Y. Mitigating neural network overconfidence with logit normalization. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Pro- ceedings of the 39th International Conference on Ma- chine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 23631 23644. PMLR, 17-23 Jul 2022. URL https: /ttp:p::/pprrceeee htppooeeeinggs.... ml r. press/ v162 1t62/wei2// weid html.
www.mosaiiell.com/bbgmmmgg
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel. D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshom, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I, Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A.. Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models, 2023.
Wei, L., Tan, Z., Li, C., Wang, J., and Huang, W. Diff-erank: A novel rank-based metrie for evaluating large language models. In The Tiirty- Annual Conference Neu- ral Information Processing Systems, 2024. URL https: / openrevieww net/ forum' den.nddnnnccsaam
Wu, Z., Balashankar, A., Kim, Y., Eisenstein, J., and Beirami, A. Reuse your rewards: Reward model transfer for zero-shot cross-lingual alignment. In Al- Onaizan, Y., Bansal, M., and Chen, Y.N. (eds.), Proceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1332-1353, Miami, Florida, USA, November 2024. Association for Compu- tational Linguistics. doi: 10. 10.18663//1122224emmll main. 79. URL https: htsss//aalaatthhoooo org/ 2024. emn 1mmlp-main.7?
Tunstall, L., Lambert, N., Rajani, N., Beeching, E., Le Scao, T., von Werra, L., Han, S., Schmid, P., and Rush, A. Creating a coding assistant with starcoder. Hugging Face Blog, 2023. http::/huugingffcc..ooooggggo/// co.bloo/staarcaaat..tt..ooo
Tunstall, L., Beeching, E. E.,, Lambert, N., Rajani, N., Rasul, K., Belkada, Y., Huang, S., Werra, L. V., Four- rier, C., Habib, N., Sarrazin, N., Sanseviero, O., Rush, A. M., and Wolf, T. Zephyr: Direct distillation of LM alignment. In First Conference on Language Modeling, 2024. URL https: pettts::/open.ee http///pppeee.e.c//e/ee/ee net/ Forum? kk=akkAWZ.6JV
Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., Lin, Q., and Jiang, D. WizardL Empower- ing large tre-trained language models to follow com- plex instructions. In The Twelfth International Confer-
14

---

On the Robustness of Reward Models for Language Model Alignment
ence on Learning Reprexentations, 2024, URL https: htwpereeiieweeeeeeeeeeeeeeeeeeee
Gonzalez, J. E., and Stoica, I. Judging LLM-as-a-judge with MT-bench and chathot arena. In Thirry-evennth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: / // //openrevii ew net/ / forum? d=uccHPGD
/ //openreview..o/// net. forum? wttp:/mmmmmmcccffcnn3333..
Yang, A., Yang, B., Hui, B.. Zheng. B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang. F., Dong, G., Wei, H., Lin, H., Tang, J., Wang. J. Yang. J., Tu, J., Zhang, J., Ma, J., Yang, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J. Dang, K., Lu, K., Chen, K., Yang, K., Li, M., Xue, M., Ni, N., Zhang, P. Wang, P., Peng, R., Men, R., Gao, R., Lin, R., Wang, S., Bai, S., Tan, S., Zhu, T., Li, T., Liu, T. Ge, W., Deng, X., Zhou, X., Ren, X., Zhang, X., Wei, X., Ren, X., Liu, X., Fan, Y., Yao, Y., Zhang,
Zhou, E.,, Zheng, G., Wang, B., Xi, Z., Dou, S., Bao, R., Shen, W., Xiong, L., Fan, J., Mou, Y., Zheng, R., Gui, T., Zhang, Q., and Huang, X. RMB: Comprchensively benchmarking reward models in LLM alignment. In The Thirteenth International Conference on Learning Representations, 2025. URL https: petttss:/oppnneee //pp////epe....... net/ fort/formm2an 2tc.
Y. Wan, Y.. Chu, Y., Liu, Y., Cui, Z., Zhang, Z., Guo, Z., and Fan, Z. Qwen2 technical report, 2024a. URL https: //a:://araa //tp/aaa.vv//..//////.../...a... org/ abs//bb////// 2200240000.00 10671.
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences, 2020.
Yang, R., Ding, R., Lin, Y., Zhang, H., and Zhang, T. Reg- ularizing hidden states enables learning generalizable reward model for LLMs. In The eighth Annual Conference on Neural Information Processing Systems, 2024b. URL https: id= jwh 9M..
htttps://ope.reviee wptp//wineeeeeeeeee/ee/ee
Yuan, L., Cui, G., Wang, H., Ding, N., Wang, X., Deng. J., Shan, B., Chen, H., Xie, R., Lin, Y., Liu, Z., Zhou, B., Peng, H., Liu, Z., and Sun, M. Advancing LLM reasoning generalists with preference trees. In Al for Math Workshop @ ICML 2024, 2024. URL https: // //openreview. net. fnet/fooumit forum? 1ididddd11111 iididddd2111ppy
Zhai, Y., Zhang, H., Lei, Y., Yu, Y., Xu, K., Feng, D., Ding, B., and Wang, H. eneertainty-penalize reinforcement learning from human feedback with diverse reward lora ensembles. arXiv preprint arXiv:2401. 22240,002433 2023.
Zhang, X., Ton, J.-.,, Shen, W., Wang, H., and Liu, Y. Mitigating reward overoptimization via lightweight uncertainty estimation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. URL https: h/tpp//ee///eeeeiiie... net/ forum? 1d=kYio3xH6eb.
Zhang, X., Xiong, W., Chen, L.,, Zhou, T., Huang, H., and Zhang, T. From lists to emojis: How format bias af- feets model aligmment, 2024b. URL https: //t:://axxiv..v org/ abs///b///// 2409. 11704.
Zhao, Y., Gu, A., Varma, R., Luo, I.., Huang, C.-C., Xu, M., Wright, L., Shojanazeri, H., Ott, M., Shleifer, S., Desmai- son, A., Balioglu, C., Damania, P., Nguyen, B., Chauhan, G., Hao, Y., Mathews, A., and Li, Pytorch fsdp: Expe- riences on scaling fully sharded data parallel. Proc. VLDB Endow, 16(12)3348-3360, 2023. URL https: /ww.. vldb. org. wwwpoogggpvvv...oeoo
htt//vllda. wwwlvl/bvvlllll //046//633888888 p3848-huang huang. pdf.
Zheng, L., Chiang, W.-L., Sheng. Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H.,
15

---

On the Robustness of Reward Models for Language Model Alignment
A. Training Configurations
For supervised fine-tuning (SFT) and reward modeling, we use Liger-Kernel Kernel (Hsu et al., 2024) with DeepSpeed ZeRO-3 (Rajbhandari et al.. 2020) and FSDP (Zhao al., 2023) for efficient training. Including reinforcement learning with human feedback (RLHF) phase, we utilize the TRL library (von Werra et al., 2020) to adjust our usage. We used NVIDIA A100 and A6000 GPUs throughout the experiments.
A.1. Supervised fine-tuning
We train every model on UltraChat for single epoch with global batch size of 512 following Tunstall et (2024). We set learning rate of 10-5 with 10% warmup and cosine decay.
A.2. Reward modeling
We train reward models top of the SFT models above with four different seeds. We fix the global batch size of 128 across the models and methods. We set learning rate of 10-6 for Llama-3.2-1B and Qwen2.5-1.5B models, 3 10-6 for 3B models, and 2 X 10-6 for Llama-3.1-8 and Qwen2.5-7B models. 5% warmup and linear decay were applied following Lambert et al. (2024). We use FSDP for distributed training in reward modeling. We set A = 10-3 in Section 5.1 and ablate different À in Section 5.3.
A.3. Reinforcement learning with human feedback
We leverage Asyne RLHF (Noukhovitch et al., 2025) with vLLM (Kwon et al., 2023) backend to reduce the bottleneck in on-policy generations. To test reward models with varying sizes under controlled training configurations (e.g., batch size), we separately deploy the reward models following OpenRLHF (Hu et al., 2024). We use batch size of for every reward model prevent potential distortions in reward scores with padding applied.
Table RLOO training configuration details for each section. We train Qwen2.5-1.5h with SFT using corresponding reward models for each section using 4 A6000 GPUs, excluding the GPUs assigned for the reward model and vLLM engine for on on-policy generation.
Category
Section 3.2
Section 3.3
Learning Rate
2 x 10-6
10 10-6 6
0.05
0.05
Number of responses (k)
2
2
Global Batch (Effective)
128
128
Learning Rate Scheduler
Linear Decay
Linear Decay
Warmup Ratio
0.03
0.03
Training Epochs
16

---

On the Robustness of Reward Models for Language Model Alignment
B. Data Preprocessing
We adopt UltraFeedback (Cui et al., 2025) for our experiments". Before splitting the train and validation sets, we filtered items with duplicated prompts and responses, We first removed the items with identical prompts and removed the rows when at least two responses out of four were identical. As result, we used 62,282 rows after filtration from 63,966.
In detail, 17 models comprising Musin for UF are: GPT-3.5-Turbo GPT-4 (OpenAl et al., 2024), Bard', Llama-2-7B-Char Alama-2-13B-Chaal Llama-2-70B-Chh (Touvron et al., 2023), WizardI. MM-7B, WizardI MM-13B, WizardI (Xu et al., 2024), Vicuna-33B (Chiang et al., 2023), Alpaca-7B (Taori et al., 2023), Falcon-40B-Instrue (Almazrouci et al., 2023), MPT-30B-Char (Team, 2023), StarChat- beta (Tunstall et al., 2023), and Pythia-123 12B (Biderman et al., 2023).
Since large proportion of the models in Musan are Llama-2 based models, we selected distinct model families for M including OLMO2-7B-Instruc (OLMo al., 2025), SmnlLLM2I..7B-Instee (Allal al., 2025), Mistral--nntruct-v0..2 (Jiang et al., 2023a), and Gemma-2-2BB (Team al., 2024).
Train set (D train) First, select random set of 51,200 samples from UF as the train set. Then, choose random responses out of four for each prompt in the train set. Thereby, we have 51,200 triplets comprising prompt and corresponding chosen and rejected responses, according to ArmoRM.
Validation 1 in-domain (Dw) Then, we use the remaining two responses in 51,200 prompts in the train set as the in-domain validation set to evaluate if the trained reward models can generalize in same prompt and response spaces. Since we have two responses per prompt, we use binary accuracy as an evaluation metric.
Validation 2 prompt out-of-domain (PPrompt-ODD) We set the remaining 12,800 instances as a prompt out-of-domain (Prompt OOD) validation set, having different prompt space but same response space. Since we have four responses per prompt, use Kendall's J ranking correlation (Kendall, 1962) an evaluation metric.
Validation 3 response out-of-domain (PResponse-OoD) For the prompts in the train set, we additionally generate four Innstr---
different responses from the new models: Gemma-2-2B- (Team et al., 2024), SmoLL.M2-1.7B-nnstnee (Allal et al., 2025), and
et al., 2025),
(Jiang et al., 2023a). By having response out-of- domain (Response OOD) validation set, we assess reward models when prompt space stays the same, but the response space differs. Since we have four responses per prompt, we use Kendall's 7 as an evaluation metric.
Validation mutual out-of-domain (DMutualooo)) Using the same models in the Response OOD set, we generate the responses for the prompts from the Prompt OOD set, having mutual out-of-domain (Mutual OOD) validation set. Here, we test the model's robustness when both the prompt and response spaces are distinct. Since we have four responses per prompt, use Kendall's T as an evaluation metric.
'https: / / httpa//awa...at.aattt//////////a
*The final dataset can found in: /ttppp////uugggggss.....////ooos
Insttrrutt
17

---

On the Robustness of Reward Models for Language Model Alignment
C. Robustness Assessment with Gold Preference Model
report the full results visualizing Figure Table
Table 5, Preference prediction accuracy over different types of validation sets.
In-Domain (Accuracy 1)
Prompt OOD
Response OOD
Mutual OOD (Kendall's f)
Model
Method
(Kendall's 's r f)
(Kendall's 7 f)
Cet (Bradley & Terry, 1952)
0,8132
0.6157
0,5115
0.5065
LaT.Hinge (Cortes, 1995)
0.8111
0.6102
0.5168
0,5131
Llama-3.2 (IB)
CaT-Noe (Wei et al., 2022)
0.7792
0,5580
0,4503
0.4488
CBT-DR (Yuan et al., 2024)
0,8029
0.6015
0.493
0.4937
CHESSK (Ours)
0.813
0.6198
0,5246
0,5206
Cer (Bradley & Terry, 1952)
0.8415
0.6835
0.5601
0.5566
Livetiinge (Cortes, 1995)
0.8388
0.6756
0,5592
0.5561
Qwen2.5 (1,5B)
LET.Nom (Wei et al., 2022)
0.8177
0,6403
0,4502
0.4507
CaTOR (Yuan et al., 2024)
0.8326
0.6625
0,5187
0.5201
CHLBSR (Ours)
0.8423
0.6849
0,5618
0,5573
Lut (Bradley & Terry, 1952)
0,8478
0.679
0,6195
0.609
Lunstingee (Cortes, 1995)
0.8458
0.6829
0,6210
0.61
Llama-3.2 (3B)
LaTNorm (Wei et al., 2022)
0,8161
0.6272
0,5542
0.5509
LETDR (Yuan et al., 2024)
0.8375
0.6638
0,5916
0,5849
LurASa (Ours)
0.8491
0.702
0.6402
0.6306
Cst (Bradley & Terry, 1952)
0.8496
0.705
0.5917
0.587
Lurttinge (Cortes, 1995)
0,8488
0,6984
0,5784
0.5685
Qwen2.5 (3B)
Lyrssco (Wei et al., 2022)
0.8272
0.6605
0,5295
0,5282
LBT-DR (Yuan et al., 2024)
0.8446
0.6855
0,5602
0.5564
LITBSR (Ours)
0,8541
0.7202
0.5991
0.6106
D. Training logs for Section 5.3
We report additional training logs for RLOO training with Ekywork-Reward-Lllame 3..-----..111--.-22 (RMgT) and RMBT-BSR () 10 3) based on Llama-3. 1.mma-3.1-80 3-BB in Table 2.
RMgr
RM5T
1.0
-7-
Reward
Reward
0.0
111
-1.0
20
Training Steps (10 Batches)
Training Steps (10 Batches)
(a) RLOO with Skywork-Rewwe Reward-Llama- Llama-3. 3.1-8B.--BBB... 8B-2 (RMot)
(b) RLOO with Llama-3. 1-18 RMOTaSR(A 10 10-3)
Figure 6. Reward maximization trajectories with Slywoark-eearr--Laam 1--0----00 (RMpr) and Llama-3. 1-8B RMBTESR.
18


Reviews:
[{"review_id": "oMHlO6ed4B", "reviewer": "Reviewer_rCnJ", "summary": "The paper investigates the cause of reward model over-optimization in RLHF and finds that it stems from the increasing variance of the final-layer outputs (hidden states) in the reward model (RM). The authors propose Batch-wise Sum-to-Zero Regularization (BSR) for RM training, which penalizes the second moment of rewards at the batch level. Experiments show that BSR improves RM stability and enhances LLM fine-tuning via RLOO.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s detailed comments. Below, we address each concern:\n\n\n1. **Claims/Evidence #1 – Notations in Section 4.2**:  We regret the confusion from the ambiguous explanations in lines 261–271 and acknowledge that lines 265–269 apply only to a specific case. In the final version, we will include a concise analysis of  $\\frac{\\partial \\mathcal{L}\\_\\mathrm{BSR}}{\\partial h(x,y)}$, showing that it introduces a linear penalty on the growing reward scale, which indirectly limits the expansion of $\\|h(x,y)\\|$. Thus, $\\mathcal{L}\\_\\mathrm{BSR}$ properly complements the BT loss with regularization.\n\n2. **Claims/Evidence #2 – Is BSR properly regularizing the hidden state norms?**:  Logit normalization (i.e., $\\mathcal{L}\\_\\text{BT-Norm}$ in Section 4.3) decouples the reward margin from the hidden state norm by dividing the reward by its L2 norm, making the loss scale-invariant [1]. Our analysis in Section 4.1 shows that the reward scale is largely driven by the hidden state norm since $||W_p|| \\simeq 1$. However, Figure 4 demonstrates that the strictly normalized loss (RM$_\\text{BT-Norm}$) performs substantially worse, suggesting that completely discarding magnitude information removes a valuable discriminative signal for out-of-distribution generalization. In contrast, our BSR method softly regularizes the norm, mitigating over-optimization while preserving useful information. Moreover, Table A shows that even with BSR, the projection head’s norm remains around 1, confirming that the BSR penalty is effectively propagated without fully suppressing the norms.\n\n3. **Theoretical Claims #1 – Ambiguity in “unconstrained norm dispersion”**: We acknowledge that “unconstrained” may not be ideal, given the sigmoid function’s gradient mechanism. Nonetheless, without a proper regularizer like BSR, norm dispersion can be amplified, especially under certain hyperparameter choices (e.g., learning rate), and we have fully demonstrated this throughout the paper. In the final version, we will revise the term to “excessive norm dispersion” or similar.\n\n4. **Strengths and Weaknesses #1 – Generalizability of the Method**:  We emphasize that the ultimate evaluation of reward models (RMs) is based on their performance in the RLHF stage. As shown in [2], downstream performance varies significantly with the RM used. Despite marginal improvements in RM benchmarks in some cases, BT-BSR outperforms in the RLHF stage while mitigating verbosity bias [3]—as evidenced by shorter responses and a higher win rate (Table 3). Combined with the improvements seen in RM benchmarks (Figure 4 and Table 2), our regularization method is both generalizable and practically valuable.\n\n5. **Comments or Suggestions #1 – Further Validation for Figures 1 and 2 with BT-BSR**: Table A confirms that $\\|W_p\\| \\simeq 1$ for BT-BSR checkpoints (as shown in Figure 1). Figure 2 empirically supports that the BT loss triggers norm dispersion, while the comparison between Figures 3(a) and 3(b) confirms that BSR effectively mitigates this phenomenon.\n\n6. **Comments or Suggestions #2 – Questions on Notations**: Since $\\mathbb{D}\\_\\mathrm{KL}$ is conditioned on the prompt $x$, both the reward and the KL penalty should be under $x \\sim \\mathcal{D}$. Additionally, lines 184–188 bridge the over-confidence issue in classic multi-class classifiers to RMs with shared projection heads. (For further details, please refer to our comment on “Q2” for reviewer YT72.) We will clarify these notations in the final version.\n\n7. **Question #1 – Drop in Easy Tasks**: Appendix O of RM-Bench [4] shows that “hard accuracy” has the highest correlation with policy performance ($r=0.45$) while “easy accuracy” is near 0 ($r=0.07$). Although the drop in “easy accuracy” may stem from a compressed representation space due to BSR, it is crucial that “hard accuracy” aligns with actual policy performance after RLHF. Our experiments in Figure 5 and Table 3 confirm that BT-BSR improves outcomes in both controlled and practical settings.\n\n&nbsp;\n\n|   | **$\\| W_p \\|$** |\n|:--:|:--:|\n| **Qwen2.5 (1.5B)** |  0.9884 |\n|  **Qwen2.5 (3B)**  |  0.9933 |\n| **Llama-3.2 (1B)** |  0.9896 |\n| **Llama-3.2 (3B)** |  0.9905 |\n\n> **Table A**: The norm of the projection head for four different models after reward modeling with BT-BSR.\n\n&nbsp;\n\n**Reference**\n\n[1] Wei, Hongxin, et al. \"Mitigating neural network overconfidence with logit normalization.\" International Conference on Machine Learning. PMLR, 2022.\n\n[2] Meng, Yu, Mengzhou Xia, and Danqi Chen. \"Simpo: Simple preference optimization with a reference-free reward.\" Advances in Neural Information Processing Systems 37 (2024): 124198–124235.\n\n[3] Yann Dubois, et al. “Length-controlled alpacaeval: A simple way to debias automatic evaluators.” First Conference on Language Modeling.\n\n[4] Liu, Yantao, et al. \"RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style.\" The Thirteenth International Conference on Learning Representations.", "claims_and_evidence": "The claims are generally well-supported, but some concerns remain:  \n\n* The theoretical discussion of BSR in Section 4.2 is unclear. For example, in lines 265–269, the notation $\\prec$ is ambiguous. Does it mean $\\leq$? If so, considering cases where $r$ is large, should the inequality direction be reversed?  \n\n* A misalignment between the hypothesis in Section 4.1 and the proposed method in Section 4.2 appears to exist. Section 4.1 argues that, considering the final-layer computation of $r(x,y) = W h(x,y)$ in the reward model, reward over-optimization occurs due to the increasing variance of $\\| h(x,y) \\|$ (or $\\| h(x,y_w) - h(x,y_l) \\|$) while the size of the projection head $W$ remains approximately constant throughout RM training. However, BSR directly penalizes the reward magnitude, which might unnecessarily reduce $W$, contradicting the initial hypothesis. To validate this hypothesis, an additional experiment regularizing $\\| h(x,y) \\|$ or $\\| h(x,y_w) - h(x,y_l) \\|$ should be included. Alternatively, if the focus is on BSR itself, Section 4.1 should analyze $r(x,y)$ directly instead of decomposing it into $W$ and $h(x,y)$.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria are appropriate for the problem. The use of RM-Bench and Length-Controlled AlpacaEval 2.0 effectively assesses the impact of BSR on reward model stability and RLHF fine-tuning.", "theoretical_claims": "The paper does not present a formal theoretical analysis, but its mathematical arguments raise concerns about ambiguity. For instance, the claim of \"unconstrained hidden state norm dispersion\" may be overstated. In ordinary models (excluding tabular models), the sigmoid function term in the gradient of the BT model should reduce the gradient magnitude as the hidden state norm dispersion increases. Moreover, Figure 2 suggests that the hidden state norm dispersion saturates early.", "experimental_designs_or_analyses": "The experimental design is well-structured, and the dataset construction process is appropriate for evaluating the proposed hypothesis and the effect of BSR.", "relation_to_broader_scientific_literature": "This paper is related to the broader literature on LLM alignment, particularly in addressing reward hacking and reward model over-optimization in RLHF.", "essential_references_not_discussed": "To the best of my knowledge, there are no essential references missing.", "other_strengths_and_weaknesses": "One concern is that the performance improvement from the proposed method appears marginal in some evaluations, such as Gold RM evaluation (Figure 5) and RM-Bench (Table 2). The results seem sensitive to hyperparameters and minor experimental settings, raising the possibility that the observed gains could be reversed under slightly different conditions.", "other_comments_or_suggestions": "- To better validate the role of BSR in addressing the hypothesis, it would be helpful to show how Figures 1 and 2 change with and without BSR.  \n- Line 135: Should $D_{\\text{KL}}$ be marginalized over $x$?  \n- Line 185: Should \"softmax value\" be \"reward value\"?", "questions_for_authors": "Table 2 shows a noticeable drop in Easy Acc in RM-Bench when applying BSR. How should this be interpreted? Is there an underlying reason why BSR negatively impacts performance on easier tasks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0JC1xcu1mS", "reviewer": "Reviewer_Ajeh", "summary": "This paper explores the challenges of over-optimization in reward models used in RLHF of LLMs. It identifies the dispersion of hidden state norms as a primary cause of over-optimization and proposes batch-wise sum-to-zero regularization (BSR) to address this by penalizing outliers and controlling reward dispersion. The study demonstrates that BSR not only improves the robustness of reward models but also enhances their performance on complex preference prediction tasks across different datasets.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comment and would like to further discuss the addressed points.\n\n&nbsp;\n\n**W1 - Limited Experimental Results**: Our experimental design can be streamlined into threefold: (1) assessing alignment between proxy RMs and gold RMs with different learning objectives, (2) propagation of over-optimization in RMs to their downstream usage for RLHF training, and (3) understanding real-world impact of such propagation with state-of-the-art model and data. This line of analysis is fully supported with Qwen2.5 and Llama3.2 series with 1B and 3B scale on carefully curated UltraFeedback, followed by Llama-3.1-8B model and 80k Skywork-Preferences dataset. While the clarity in writing could be a valid concern if the logical flow is not fully conveyed, we believe the paper encompasses a large range of models and datasets, especially aligning with practical usage in the research community (e.g., the majority of RMs in RewardBench [1] are based on Llama-3.1 or Llama-3.2 series and Skywork-Preferences dataset).\n\nTo ensure that the concerns on the width of experiments, we extend the experiment of Figure 5 (Section 5.2) by scaling the model to Qwen2.5-3B in Table A. This result further validates that the trend shown in Qwen2.5-1.5B on the propagation of RM robustness in the RLHF stage on a larger scale. If the concerns about the clarity of writing can be specified, we would happily provide further explanation and an update in the final version of the paper.\n\n&nbsp;\n\n\n**W2 - Novelty of Batch-wise Sum-to-Zero Constraint in Reward Modeling**: We appreciate the concern on methodological novelty, but there seems to be a major confusion on the zero-centering aspect of BSR. While [1] is cited as the core reference to point out that the idea of centering rewards to zero is not novel, we do not find it to be connected to our approach:\n\n> (Excerpt from Discussion section of RewardBench according to reviewer’s pointer)...Few RMs are Gaussian in their scores across the REWARDBENCH datasets, fewer RMs are centered around 0 reward, and none we tested centered Gaussians. Future work should identify a preferred RM output distribution for downstream RL training.\n\nThis is the only part that we find in [1] that mentions centering. However, they intend that RMs are rarely centered through post-hoc analysis. To the best of our knowledge, [2] is the only work that adopts zero-centering *per-prompt*. However, their core motivation lies in the identifiability of RMs, which is distant from ours. We build a solid reason for adopting a *batch-wise* sum-to-zero constraint regarding the mitigation of reward model over-optimization. We would happily incorporate the discussion on [2] for our final version of the paper. However, we emphasize that our method remains novel from its theoretical background and empirical support with varying model sizes, types, and datasets.\n\n&nbsp;\n\n\n| Qwen2.5-3B | Gold Reward Mean | Gold Reward Std |\n|:---:|:---:|:--:|\n| RM$\\_\\text{BT}$     | 0.120  | 0.045  |\n| RM$\\_\\text{BT-BSR}$ | **0.123**   |  0.045  |\n> **Table A**. Qwen2.5-3B-SFT trained with RLOO with each reward model for 5 epochs, evaluated by the gold reward model (ArmoRM). The reward scores are NOT normalized, unlike Figure 5. Unnormalized score of ArmoRM typically ranges around 0.1-0.2 [3]\n\n\n&nbsp;\n\n**Reference**\n\n[1] Lambert, Nathan, et al. \"Rewardbench: Evaluating reward models for language modeling.\" arXiv preprint arXiv:2403.13787 (2024).\n\n[2] Eisenstein, Jacob, et al. \"Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking.\" First Conference on Language Modeling.\n\n[3] Wang, Haoxiang, et al. \"Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts.\" Findings of the Association for Computational Linguistics: EMNLP 2024. 2024.", "claims_and_evidence": "The preliminary theoretical derivation makes sense. However, the claims for different scenarios might need more comprehensive experiments to be robust and convincing.", "methods_and_evaluation_criteria": "The use of batch-wise sum-to-zero regularization (BSR) in the paper is a sensible approach to addressing over-optimization in reward models by controlling reward dispersion. Nonetheless, the paper’s discussions and experiments lack of focus and the paper is quite hard to read. Various scenarios are considered but for each of them, the experiments are not comprehensive enough (for example consider more models and datasets) to make the results convincing.\nFurthermore, it’s important to note that the idea of centering rewards at zero is not entirely novel, as similar concepts have already been explored in existing models (for example see the discussion in [1].\n \n[1] Lambert, N., Pyatkin, V., Morrison, J., Miranda, L. J., Lin, B. Y., Chandu, K., ... & Hajishirzi, H. (2024). Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787.", "theoretical_claims": "There are some preliminary theoretical derivations which looks correct. There are no technical theoretical claims or proofs.", "experimental_designs_or_analyses": "As mentioned above, the experiments are not comprehensive enough and I have concerns about the practical utility of most of the designed evaluation metrics in the discussion.", "relation_to_broader_scientific_literature": "The key contribution is on the sum-to-zero regularization, but this idea might lack novelty.", "essential_references_not_discussed": "See above.", "other_strengths_and_weaknesses": "The exploration of various scenarios and the provision of hypotheses and insights are strengths of the paper.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k5VHLDV2lp", "reviewer": "Reviewer_hrWq", "summary": "This paper investigates the issue of over-optimization in reward models (RMs) within RLHF, identifying excessive hidden state norm dispersion as a key factor. To address this, the authors introduce batch-wise sum-to-zero regularization (BSR), which constrains reward magnitudes by ensuring batch-level zero-centering. They further categorize four generalization scenarios in reward modeling to analyze robustness and over-optimization. Their proposed method outperforms baseline methods and achieves promising experimental results.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments and will add the suggested paper to the related works.\n\n&nbsp;\n\n**Q1 - Normalization techniques to implicit rewards of DAAs**: DAAs use the language model as an implicit reward model, differing from our classifier-based reward models in that they use a sum of log ratios rather than a linear projection. As our normalization, BT-BSR, starts with the solid background on the over-confidence issue that classifiers are prone to, it may not be intuitive to directly apply batch-wise sum-to-zero constraint on implicit RMs. Applying a batch-wise sum-to-zero constraint directly to implicit rewards, which often exhibit negative scales [1], would be counterintuitive.\n\nNevertheless, over-optimization can also occur with implicit rewards [2], as evidenced by their divergence to negative infinity [3]. As such, divergence is similar to hidden state norm dispersion in our scenario, the straightforward solution could be Z-loss formulation [4,5]:\n\n$$\n\\mathcal{L}\\_{IRM-BSR} = - \\lambda \\log^2 \\sum\\_i^{N} \\left( \\beta \\log \\frac{\\pi\\_\\theta(y\\_{i, w}|x)}{\\pi\\_\\mathrm{ref}(y\\_{i, w}|x)} + \\beta \\log \\frac{\\pi\\_\\theta(y\\_{i, l}|x)}{\\pi\\_\\mathrm{ref}(y\\_{i, l}|x)}  \\right).\n$$ \n\nZ-loss is a well-established method for stabilizing large-scale language model pre-training by penalizing divergence in logits. Here, $\\mathcal{L}\\_\\mathrm{IRM-BSR}$ prevents the implicit reward (log-ratio) from diverging to negative values, mirroring how Z-loss regulates the softmax normalizing constant.\n\nWe conducted a minimal experiment on Qwen2.5-1.5B trained with TULU3-SFT data [6] comparing plain DPO with DPO plus the $\\mathcal{L}\\_\\mathrm{IRM-BSR}$ regularizer. As shown in Tables A, B, and C, the Z-loss-based regularizer substantially mitigates negative divergence in log-likelihood while preserving training preference accuracy.\n\n&nbsp;\n\n|        | 0   | 5   | 10  | 15  | 20  | 25  | 30  | 35  | 40  | 45  | 50  | 55  | 60  | 65  | 70  | 75  | 80  | 85  | 90  |\n|--------|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|\n| DPO    | -302| -242| -274| -306| -378| -322| -314| -334| -460| -434| -448| -426| -528| -414| -440| -376| -474| -466| -424|\n| DPO+Reg| -249| -260| -270| -292| -340| -296| -251| -304| -366| -384| -364| -344| -378| -320| -356| -306| -392| -388| -324|\n> **Table A**: Log-likelihood of chosen responses during training.\n\n|        | 0   | 5   | 10  | 15  | 20  | 25  | 30  | 35  | 40  | 45  | 50  | 55  | 60  | 65  | 70  | 75  | 80  | 85  | 90  |\n|---|---|---|---|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|\n| DPO    | -340| -296| -366| -430| -492| -432| -484| -476| -488| -612| -572| -632| -740| -596| -652| -564| -656| -604| -620|\n| DPO+Reg| -314| -318| -338| -422| -478| -434| -384| -458| -474| -524| -470| -544| -572| -508| -498| -450| -532| -506| -520|\n> **Table B**: Log-likelihood of rejected responses during training.\n\n|        | 0     | 5     | 10    | 15    | 20    | 25    | 30    | 35    | 40    | 45    | 50    | 55    | 60    | 65    | 70    | 75    | 80    | 85    | 90    |\n|---|---|----|----|---|---|--|---|--|--|--|---|---|-|--|-|---|--|---|-|\n| DPO    | 0.2125 | 0.375  | 0.5125 | 0.625  | 0.6    | 0.625  | 0.675  | 0.675  | 0.6875 | 0.7125 | 0.6875 | 0.7625 | 0.6999 | 0.7125 | 0.75   | 0.6999 | 0.6625 | 0.75   | 0.675  |\n| DPO+Reg| 0.1875 | 0.4    | 0.5375 | 0.6062 | 0.6000 | 0.6687 | 0.6875 | 0.6812 | 0.7437 | 0.6937 | 0.6875 | 0.7312 | 0.8125 | 0.6875 | 0.7437 | 0.6437 | 0.7624 | 0.7812 | 0.7312 |\n> **Table C**: Preference accuracy for each batch during training.\n\n&nbsp;\n\n\n\n&nbsp;\n\n\n**Reference**\n\n[1] Rafailov, Rafael, et al. \"From $ r $ to $ Q^* $: Your Language Model is Secretly a Q-Function.\" First Conference on Language Modeling.\n\n[2] Rafailov, Rafael, et al. \"Scaling laws for reward model overoptimization in direct alignment algorithms.\" Advances in Neural Information Processing Systems 37 (2024): 126207-126242.\n\n[3] Shi, Zhengyan, et al. \"Understanding likelihood over-optimisation in direct alignment algorithms.\" arXiv preprint arXiv:2410.11677 (2024).\n\n[4] Chowdhery, Aakanksha, et al. \"Palm: Scaling language modeling with pathways.\" Journal of Machine Learning Research 24.240 (2023): 1-113.\n\n[5] Wortsman, Mitchell, et al. \"Small-scale proxies for large-scale Transformer training instabilities.\" The Twelfth International Conference on Learning Representations.\n\n[6] Lambert, Nathan, et al. Tulu 3: Pushing frontiers in open language model post-training.\" arXiv preprint arXiv:2411.15124 (2024).", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "The evaluation settings follow (Gao, 2023) to set the synthetic gold RM instead of real datasets with human annotations. This makes sense to do controlled experiments on reward modeling.", "theoretical_claims": "No theoretical contents.", "experimental_designs_or_analyses": "Yes I checked the evaluation on in-domain, prompt-disjoint, response-disjoint and mutual-disjoint cases. Also I checked the experiments that validates the hypothesis that over-optimization is related to the unconstrained growth of the h(x,y). The design seems reasonable.", "relation_to_broader_scientific_literature": "The paper addresses reward hacking problem by adding proper normalization during training, which is orthogonal to prior work that introduces other loss functions (such as penalty in output length [1]).\n\n[1] Chen, Lichang, et al. \"ODIN: Disentangled Reward Mitigates Hacking in RLHF.\" International Conference on Machine Learning. PMLR, 2024.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is very well-written, with each message clearly conveyed in the experimental section.\n2. The topic is relevant since reward hacking is a long-standing and unsolved question in the LLM space. \n3. The evaluation on in-domain, prompt-disjoint, response-disjoint and mutual-disjoint cases is very useful information.", "other_comments_or_suggestions": "None", "questions_for_authors": "Does the proposed normalization techniques also work for DPO-style methods, where there is no explicit training of a separate reward model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dt4svSZS1v", "reviewer": "Reviewer_YT72", "summary": "The paper provides a new theoretical analysis framework to understand the robustness of RM in LLMs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your suggestions on the typo and the section title. We will make sure to address them in the final version of our paper.\n\n&nbsp;\n\n**Q1 - Quantifying disjointness between datasets:**  We appreciate the reviewer’s question for disjointness quantification as it would make the splitting criteria clearer. We originally planned on splitting the datasets based on the cosine distances of the embeddings of the prompt and responses. However, due to the diverse split settings outlined in Section 2.2, we were unable to provide a strict quantification rule of thumb for disjointness quantifications.\n\n**Q3 - Over-confidence vs over-optimization:**  For clarity, we address the third question before the second one. We view the over-confidence issue, which is originally studied for the conventional multi-class classifiers, as a core cause of the reward model over-optimization problem in the context of reward modeling formulation. [1] shows that an overly growing magnitude of logits triggers classifiers to be over-confident due to the nature of the softmax function. In this paper, we carefully analyze how the backbone model’s hidden state norm is a core cause of the growth in reward magnitude (i.e., which is equivalent to the logit magnitude) as they are a unique type of classifier with two classes but with a single projection head, unlike conventional multi-class classifiers. We continue how over-confidence analysis in RMs connects to over-optimization in Q2.\n\n**Q2 - Clarifications on lines 184-187:** This explanation intends to describe RMs’ formulation as a special case of conventional classifier models, where [1] studies the over-confidence problem, and builds how this over-confidence issue in RM formulation is connected to the over-optimization problem. Reward modeling loss $\\mathcal{L}\\_\\mathrm{BT}$ can be written as a two-class softmax classification. While the over-confidence problem originates from growing logit magnitude in conventional classifiers [1], we are analyzing this problem in the RM context. As the magnitude of the reward (i.e., logit) comes largely from $||h(x, y_||$ by $||W_p|| \\simeq 1$ even after training, as we showed, the over-confidence problem in RMs stems from the dispersion of hidden states. As the hidden dimension grows (which is common in larger language models), the norm tends to increase, thereby potentially exacerbating over-optimization issues. This is why these lines of explanation bridge the important connection between the over-confidence and over-optimization issues in RMs.hmm (briefly talked about utilizing embedding model + cosine distance for quantification)\n\n**Q4 - Clarification in Figures 5(c) and 5(d):**  Appendix A.3 explains that we used a linear decay scheduler during RLOO training. Consequently, the later phase of training operates with a lower learning rate, which leads to slower parameter updates and a tendency for the model to overfit the prompt over multiple epochs. Moreover, note that the y-axis scales differ between Figures 5(c) and 5(d): Figure 5(c) covers a range of [0.0, 1.0], while Figure 5(d) focuses on [0.85, 1.0] to allow a more fine-grained comparison between the policies trained with $RM_{\\text{BT}}$ and $RM_{\\text{BT-BSR}}$. In this context, the result can be interpreted as $RM_{\\text{BT}}$ stagnating in the later phase of training, reaching about 95% of the maximum reward, while $RM_{\\text{BT-BSR}}$ reaches that level by epoch 5. The transient dip and subsequent boost in the curve likely reflect the combined effects of the decaying learning rate and overfitting dynamics, resulting in non-monotonic updates to the proxy reward signal.\n\n\n&nbsp;\n\n**Reference**\n\n[1] Wei, Hongxin, et al. \"Mitigating neural network overconfidence with logit normalization.\" International conference on machine learning. PMLR, 2022.", "claims_and_evidence": "Yes, I think most of the claims made in the submission clear and convincing.", "methods_and_evaluation_criteria": "See Questions For Authors part.", "theoretical_claims": "I didn't check all the proofs in detail but the theorems provides seem to be reasonable and sounding.", "experimental_designs_or_analyses": "Most of the experimental designs are sound and valid. However, there are small points that need to be clarified. See Questions For Authors.", "relation_to_broader_scientific_literature": "The paper provides a valuable vision for the current understanding of RM/RLHF robustness.", "essential_references_not_discussed": "Potentially related works missing:\n1. Yan Y, Lou X, Li J, et al. Reward-robust rlhf in llms[J]. arXiv preprint arXiv:2409.15360, 2024.", "other_strengths_and_weaknesses": "See Questions For Authors.", "other_comments_or_suggestions": "1.  line 909: ]\n\n2.  Section 3 should be named as \"Experiment Settings\" as no results are shown in this section.", "questions_for_authors": "1.  Is there any quantification method to quantify the disjointness between each datasets mentioned in Section3?&#x20;\n\n2.  Line 184, left column, \"Typically, the norm of two vectors ||Wp|| · ||h(x, y)|| largely contributes to maximize the softmax value and cause over-confidence issue, especially in the context of large language models (LLMs) that have large hidden sizes.\" Can you explain the sentence in detail? What's more, in RM training process, there is no softmax (the language modeling head is replaced by value head), so what is the relationship between this sentence and the context?\n\n3.  Is over-confidence the same as overoptimization?\n\n4.  For the result in Figure 5(c) and (d), why is the change in the two curves discontinuous despite being caused by the same loss? For example, in RM\\_BT, during epoch 2.25-2.5, the curve starts to decline, but during epoch 2.5-2.75, the curve again boosts. Why?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers generally ranged from weakly in favor of acceptance to outright rejection, which upon reading the paper I struggle to fully agree with, but nor do I wish to ignore them.

The two primary concerns (novelty and significance of performance gains) are certainly reasonable concerns. BSR *looks* like many other methods, at least at a cursory level. Normalization of rewards and advantages is fairly common (at the policy learning stage), and regularizing everything from weights to activations with a squared loss is a reliable technique. However, the author(s) do more than simply throw that reliable technique against a new wall. It is well motivated and those motivations empirically investigated, and alternatives evaluated. In terms of the significance of the improvements on reward modeling and eventual policy learning, again I agree there is reason for concern or at least caution.  However, the empirical work shows a modest but generally consistent benefit.

As such I see these two concerns as less worrisome than perhaps some of the reviewers, and when coupled with a (to my reading) clear overall narrative, organization and experimental design make this a paper with contributions worth sharing.
Decision: Accept (poster)