Rethinking Reward Modeling in Preference-based Large Language Model Alignment
Authors: Hao Sun
Yunyi Shen
Jean-Francois Ton
Venue: ICLR 2025 Oral (2025)
Keywords: Bradley-Terry Model
Reward Modeling
Large Language Models
Primary area: foundation or frontier models, including LLMs

Abstract: The Bradley-Terry (BT) model is a common and successful practice in reward modeling for Large Language Model (LLM) alignment. However, it remains unclear *why* this model --- originally developed for multi-player stochastic game matching --- can be adopted to convert pairwise response comparisons to reward values and make predictions. Especially given the fact that only a limited number of prompt-response pairs are sparsely compared with others. 
In this paper, we first establish the convergence rate of BT reward models based on deep neural networks using embeddings, providing a theoretical foundation for their use.
Despite theoretically sound, we argue that the BT model is not a necessary choice from the perspective of downstream optimization, this is because a reward model only needs to preserve the correct ranking predictions through a monotonic transformation of the true reward. 
We highlight the critical concept of *order consistency* in reward modeling and demonstrate that the BT model possesses this property.
Moreover, we propose a simple and straightforward upper-bound algorithm, compatible with off-the-shelf binary classifiers, as an alternative order-consistent reward modeling objective. 
To offer practical insights, we empirically evaluate the performance of these different reward modeling approaches across more than 12,000 experimental setups, using $6$ base LLMs, $2$ datasets, and diverse annotation designs that vary in quantity, quality, and pairing choices in preference annotations.

Published as a conference paper at ICLR 2025
RETHINKING REWARD MODELING IN PREFERENCE- BASED LARGE LANGUAGE MODEL ALIGNMENT
Hao Sun; Yunyi Shen", Jean-Francois Ton
University of Cambridge, Massachusetts Institute of Technology, ByteDance Research
ABSTRACT
The Bradley-Terry (BT) model is a common and successful practice in reward modeling for Large Language Model (LLM) alignment. However, remains un- clear why this model - originally developed for multi-player stochastic game matching - can be adopted to convert pairwise response comparisons to reward values and make predictions. Especially given the fact that only a limited num- ber of prompt-response pairs are sparsely compared with others. In this paper, we first establish the convergence rate of BT reward models based on deep neural net- works using embeddings, providing a theoretical foundation for their use. Despite theoretically sound, argue that the BT model is not necessary choice from the perspective of downstream optimization, this is because a reward model only needs preserve the correct ranking predictions through a monotonic transforma- tion of the true reward. We highlight the critical concept of order consistency in reward modeling, propose a simple and straightforward upper-bound algorithm, compatible with off-the-shelf' binary classifiers, as an alternative order-consistent reward modeling objective. To offer practical insights, we empirically evaluate the performance of these different reward modeling approaches across more than 12,000 experimental setups, using base LLMs, datasets, and diverse annota- tion designs that vary in quantity, quality, and pairing choices in preference anno- tations. [Code Data] [A full version is available at this link]
INTRODUCTION
The alignment of Large Language Models (LLMs) is crucial for their safe and effective deploy- ment across various applications. Current research on reinforcement learning from human feedback
(RLHF) (Christiano al. 2017 Stiennon et al. 2020 Ouyang et al., 2022; Bai et al. 2022a has largely focused on utilizing baaeed annotations provided by humans or general-purpose
LLMSs (Ba et al. (2022b); Lee et al.] (2023); Guo et al. (0224). In general, there are two primary
approaches to RLHF, namely the direct policy optimization (Rafailov et al.) [2224] Zhao al. 2023
Azar et al. 2023) that aligns LLMS with supervised learning objectives, and the alternate method
that constructs reward model to guide the LLM optimization via either supervised learning or
reinforcement learning (Sun) 2023 Yuan et al. 2023 Munos et al. 2023 Li et al. 2023).
Among these strategies, the Bradley-Terry (BT) model (Bradley & Terry 1952) is commonly em- ployed to convert pairwise comparisons into scores and has demonstrated its success in large-scale alignment systems (Ouyang et al. 2022 Touvron al. 2023). Despite its empirical success, the theoretical justification for using the BT model in this context remains underexplored, particularly when dealing with sparse comparisons on limited number of prompt-response pairs. Furthermore, it is unclear how necessary it is to use the BT models, and what data format is preferred in anno- tation. Our work aims to address these key questions by reflecting on reward modeling in LLM alignment. The subsequent sections of this paper are structured to answer those questions:
Question 1: When the number of players is greater than the number of comparisons (as often the case in LLM alignment settings), is the of the BT model theoretically sound, and what factors contribute to its success in practice? Section 2 we first review the usage of BT from a statistics perspective. We show that the
"HS and YS contributed equally. hs789 cam.ac.uk, sahen99@@it..cnn, Aataar/@@@miiicc.aaaaaaaaaaoaaao jeanfraaccoisseeyycdaeee..cc

---

Published as a conference paper at ICLR 2025
classical BT model can be seamlessly applied the LLM arena cases, but falls short in reward modeling due to the extremely sparse comparison and the requirement of making predictions. We explore the regression variants of BT models (c.g., SSpringall 1973; Bockenholt (988) and their neural network extensions, providing theoretical results on their ability to approximate true reward up an additive constant in the context of L.LM alignment.
Question What are alternative approaches to reward modeling other than the BT model? Since the primary objective of reward modeling is to optimize LLMs outputs by identifying the good response in inferences, learning reward function up to any monotonic transformation should be sufficient for such an objective. This fact motivates us to propose simple objective based on order consistency in Section B] We show both the BT model and Classification models fall in this objective class. Corresponding empirical studies are conducted in Section 4
Question 3: The conventional applications of the BT model assume randomized pairwise com- parisons (e.g., randomized game matches among players). Would cross-prompt comparisons lead to more effective reward modeling?
Our theoretical analysis emphasizes the necessity of using regression on the embedding space to predict rewards, which hints at the possibility of different types of comparisons. Specifically, find no inherent advantages in restricting comparisons to responses from the same prompt, as is commonly done in the literature. In Section 4 empirically investigate the impact of cross- prompt comparisons in LLM alignment and provide insights into their potential benefits.
We defer related work discussions to Appendix A To summarize our contributions:
1. Formally, we provide a comprehensive analysis of the application of the BT model in LLM alignment, contrasting its traditional use in multi-player arenas with the unique challenges posed in this context. We analyze the underlying rationale and offer thorough justification for applying the BT model to LLM reward modeling.
Theoretically, we introduce the first asymptotic theory for neural network-based BT regression in preference- based reward modeling. Our work establishes the first risk bound for BT model reward estimation the context of LLM alignment.
3. Practically, we propose order consistency as a core objective of reward modeling, demonstrating how this principle can derive both the BT model and alternative classification-bal approach. This alternative offers greater flexibility compared to the BT model, broadening its applicability. Empirically, we conduct extensive experiments - covering 6 base LLMs, 2 datasets, response sampling methods, 6 annotation noise levels, 3 reward model implementations, 4 annotation availability scenarios, and 5 random seeds - resulting in over 12,000 runs. These experiments demonstrate the statistical efficacy of the dassification-based reward models and compare them with the BT model across diverse settings.
2 RETHINKING THE USAGE OF BT MODELS IN LLM ALIGNMENT
2.1 TWO DIFFERENT BT MODELS PARAMETER ESTIMATION AND PREDICTION
There are at least two distinct applications of the BT model in LLM settings: (1) LLM chatbot arenas (Chiang al., 2024) and (2) reward modeling. While both utilize pairwise comparisons, their objectives and the volume of comparisons differ significantly. In the chatbot arena, multiple LLMs compete against one another based on human feedback through paired comparisons. Here, each LLM functions as a player, and the annotatee preferences represent the outcome of these matches. The goal is to assign a single performance score to each LLM player. CCiang et al. (2024) 130 models were compared across more than 1.7 million comparisons, with each model participating in over 26, 000 matches on average.
In contrast, reward modeling seeks to assign single reward value for each prompt-response pair, with the added challenge of predicting rewards for unseen pairs. Typically, any given prompt- response is compared to another one only once, resulting in far fewer comparisons than in the arena setting. To understand how a single model would handle these seemingly different tasks, it is bene- ficial to have overview of the BT model and their variants.
The original BT model (Bradley & Terry 1952), aligned with the Luce-Shephard choice rule (Luce. 1959 Shepard 1957), posits that in a simplified two-option scenario, the probability of selecting
2

---

Published as a conference paper at ICLR 2025
option i from set i.j is proportional to the utility u(+) assigned to that option. Formally, this can be expressed as a softmax output of the log utilities r()) Bockenholt 1988)
m(() P(i 1) u11) m(j) exp(r(i)) + exp(r())) softmax(r(0)
exp(r(i))
(I)
In the LLM arena setting, estimating the values of r(-) is sufficient to achieve the primary goal of evaluating each chatbot's performance. This aligns closely with the original objective of BT
model in ranking sports teams (Bradley & 'eerry 1952). Previous work has shown that, with enough pairwise competition, one can estimate these ability scores well (Ford Jrl 1957, Han et al. [020] Wu et al.) (022) up to a constant additive factor. It is shown that to estimate N scores via random pairwise comparisons, the theoretical lower bound on the number of comparisons is O(N log( [N)
while the best-known methods require Oo
(N)) comparisons (Han et al. 2020).
In contrast, the application of the BT model to reward modeling is not as straightforward. First, the implications of using the BT model in this context are not well-defined in the literature. For instance, if each prompt-response pair is treated as a player, how do we characterize the stochastic nature of human annotations as the game results? What assumptions need to be made? The answers to these questions were not elaborated in the literature (Christiano al., 2017 [Stiennon al. 2020) Ouyang et al., 2022 Bai et al. 2022a). In Appendix B we provide an analysis of the underlying assumptions applying the BT model preference-basee annotation as a game.
Additionally, estimating a separate r(-) for each prompt-response pair is impractical. In typical LLM alignment scenarios, we often have only N/2 comparisons for I pairs, far below the theoretical lower bound for consistent estimation (Han et al. 2020). Furthermore, unlike the arena setting, there no clear way to predict the score for new, unseen pair. However, this challenge is not unique to LLM alignment; sports analysts also need to estimate a team's 'ss ability before many competitions or predict the ability of a new team. A common approach in such cases is to use features or covariates, such as team composition or funding status, to predict scores. For LLM, one could have sentence embeddings as such covariates.
These extensions of the BT model were explored shortly after its original introduction: Springall 1973) assumed that r(-) could be expressed as a linear combination of covariates. In this scenario, the problem reduces to classic logistic regression on the difference between two sets of covariates. This allows us to predict the score for a new team or prompt-rexponse pair based on its covariates before any comparisons are made. More complex nonlinear models such as spline models have also been explored (De Soete & Winsberg 1993). ) For a more comprehensive review from a statistical perspective, refer readers to Cattelan 2012).
In practice, reward modeling in LLM alignment often employs neural networks, with multilayer per- ceptrons (MLPs) being common choice to map embeddings scores. However, there is currently no theoretical justification for why this particular choice of model and loss function is effective for learning reward functions. From theoretical standpoint, this model variant can be viewed as a non- parametric logistic regression problem (Bockenholi 1988] SSchiddt-Hibber 2020) with additional structural assumptions on the network, and our analysis builds on this framework. In the following section, we establish the asymptotic theory for learning reward models using MIPs and BT loss.
2.2 ASYMPTOTIC THEORY ON MLP-BASED BT REGRESSION REWARD MODELING
In preference-based based LLM alignment, we work with the dataset under the form of Dpref ((ri, ((ra,,1,,, y1,i,
v2 he)) where each tuple consists of the prompt Ij, the corresponding responses Vi.; and V2.; sampled from the LLM to be aligned v1,1; ((xx), and the human-annotated preference h; being 1 if yı,; is preferred and - otherwise.
Assume we have a known embedding function ((.,)) : X y + 1]d, such that there exists an unknown reward function RR + R. and can be expressed as (( y)) for all x, y. Without loss of generality, we assume the embeddings are scaled within the range [0, otherwise, we can scale the embeddings into this range. Under this framework, reward modeling reduces to learning the function r. Note that under this formalism there is no need for a comparison to have the same
Inttuitinn hehind this bound can come from the fact that, given the true underlying scores, quicksort algorithm requires O(N N)) comparisons on average.
3

---

Published as a conference paper at ICLR 2025
prompt. We will empirically explore the effects of using cross-prompt and same-prompt in experiment section.
Denote our reward model fee. parameterized by 0, when there is no confusion, will abbreviate it as f. We denote the vector of two rewards as and the class probability is then softmax(()).
Thus, training this reward model reduces to training classifier with a cross-entropy loss, where the predicted conditional class probabilities are computed as softmax (F( (V V1 (1)). F(V (2) (z2.32)) y2))).
Our theoretical analysis follows closely the work of [Bss & Schmidt-Hieber (2022) and [Schmidt- Hieber (2020). In this setting, we consider a special case of model that preserves anti-symmetry: we exchange the roles of 21, 91 with Z2, y2, the reward difference becomes negative.
We demonstrate that an MLP can approximate the probability that the first pair is preferred over the second, and subsequently show that this approach enables the model learn the underlying reward function effectively. For notational simplicity, let (1)) and T represent the embeddings of the i-th pair, where ,, n. Without loss of generality, we assume vp( E [0. 1J Let
the true preference probability for the i-th pair be Po (6) and the predicted probability be p(9) =
(a(AA y L ( 1 a(A ( I ))( softmax((" f( q )))")) The preference label (() equals (1. 0) if the first response pair is preferred, and (0, 1) otherwise. In this way, the BT model can be reduced to pairwise classification problem, where the likelihood is given by:
EcE(P) -
- E(hh (1) log(p!)), p arg mig LCE(p) 2=1
-
(2)
peF,
unrealistic to assume we can find an NN that actually attends the global minimum, we denote the difference between the fitted NN and the global minimum as
Au (Po: p) = (p)
min ΣCE (p)
PEto
We consider the truncated KL risk following Bos & Schmidt-Hicbbr (2022) overcome the diver- gence problem of KL risk. Definition 2.1 (Truncated KI. risk (Bos & Schmidt-Hieber 2022
The B ttuunccteed KI. risk for
probability estimator P
RB RB p) E Pb
T min
B, log
Po p
(4)
Our main theorem establishes that, with regularity assumptions on the true reward function, an MLP reward model can accurately predict preference probabilities, measured by truncated KI. risk.
Theorem 2.2 (Truncated KL risk bound, informal). Suppose the true utility function induced prob- ability of preference satisfies smoothness and regularity assumptions detailed in Assumption C.3 with smoothness characterised by constant B and regularity characterised by constant a with di- mension of embedding being d. Let p be an estimator from the family of MLP satisfying regularity (1+018
(1t))8-(3t03) (140)a4d For
assumptions detailed in Assumption C.2 with depth L. Define on := 2 (140)8-a
sufficiently large there exists constants C²,CO C" such that when (D. S Log log2(n) (n)
then
Ra P) S C' Lon (n)) where a S b means there exists some constant C' s.t. S cb abaand a x b means a S b and b S a.
(5)
A detailed formal statement and proof are given Theorem C.11 in the appendix.
Corollary 2.3 (Connecting probability to reward). Given that (Va- 2 (a b)2 /////V/a/ vb)2 we can apply Lemma C.12 to demonstrate that in a large subset of the embedding space,
(Po( 12) p 22 S VPo
VolVo. I...loo(n)
(6)
VPo + v
br ( r( (f(
VonTLog(m
p(1-p)
where P is probability between Po and p. the second line is due to mean value theorem. This indicates that the comparison should be between those pairs that are relatively close in reward to avoid diverging behavior of the logit function.
Formal proofs and detailed theoretical analyses are provided in Appendix C
4

---

Published as a conference paper at ICLR 2025
3 RETHINKING REWARD MODELING OBJECTIVES IN LLM ALIGNMENT
Practical implementation of the BT model poses several requirements including the paired data, the specially designed anti-symmetric model structure, and the inherent assumptions of the BT model itself. This leads us to question whether we can have alternative approaches to reward modeling. To address this, it is helpful to pause and reflect the essential requirements of reward model in LLM aligmment. Our data consists of binary preferences, representing the relative ranking between two prompt-response pairs. We can view this form of binary classification, with the ultimate goal being to learn a continuous score for optimization. While the Bradley-Terry model serves this purpose through its log-likelihood loss and anti-symmetric structure, now consider whether these formal elements are indispensable and explore possible simplifications.
3.1 THE UNIFIED TARGET OF ORDER CONSISTENCY
In basic binary classification, we prioritize accuracy over modeling output probabilities precisely. For example, neural classifiers, despite being overconfident (Guo et al. 2017). are widely used for their accuracy. Similarly, we don't require the BT model to predict comparison probabilities accurately, but rather to provide a reliable signal for ranking LLM outputs at inference.
Since our goal is response optimization using a reward proxy, it is sufficient learn the reward function up to a monotonic transformation. While this might alter preference probabilities, it won't t affect optimization results. To this end, the learned reward function & only needs to satisfy the following condition: for any two distinct prompt-response pairs (x1, v1) and (22, 22). we require that (f(z1, v1) - f(r22 y22 (r(x), y)) - rx22, 322) > O. In other words, the learned reward function must preserve the ordering as the true reward function.
This condition implics the existence of a strictly monotonic increasing function such that f(.) h(r(-)). Such an equivalence is sufficient for optimizing the reward in settings such as sampling- based optimization, and contextual bandits (Agarwal et al., 2014; Lattimore & Szepesvári 2020) Ideally, if we have access to the ground truth ordering, r 82)). If we can (L) construct a model H X y x X + {+1,- -1 that predicts the correct ordering with high accuracy (i.e., is order consistent), and (2) map this ordering into continuous value, then we can meet the requirements for downstream optimization.
can define h sign(r(#1, v1)
However, this observed ordering is often subject to noise from human annotators. Drawing on insights from the psychological bottleneck literature (Stewart et al., 2005 Guest et al. (016)), it is reasonable to assume that when the true underlying scores of two responses are similar, it becomes more difficult for annotators to distinguish between them. Formally, we have
Assumption 3.1 (Imperfect Preference Annotation in Approximating True Scores). Denote the true utility difference Ar := |r(11, -r(2, y2)), and the annotator function l1 I2, y1, 31,32) provides feedback that probabilistically aligns with the oracle utility r(x, y). We will assume it is harder for them to assign correctly when the reward difference between two pairs is Ar according to:
P h(m1,, 22, vn, y2) (r(m,, 31) r(22, > Ar E(Ar),
(8)
where r is the unknown oracle utility function, and {-) any monotonic increasing function to [0.5, 1].
With those noisy annotations, the best we can consider is an order consistent with the noisy ordering:
Definition 3.2 (Order Consistency). We consider the loss over an ordering model A
Loc = Te h fr
That is, the probability that a reward model ordering agrees with annotation.
We show with the following proposition that minimizing this loss would help us achieve order consistency with true reward function:
Proposition 3.3 (Lower bound on population level order consistency ). Suppose a learned model i achieves objective equation up to dc error for some small 0 < and < 3/20, iee,
E Exr.wa.v..e... h H 2 1 - 8c
(10)
5

---

Published as a conference paper at ICLR 2025
Then, with probability at least 1 - 8 over Ar, for any given Ar the order consistency of f with respect to the oracle utility is bounded below by:
Buy Eu 22 1 (H [r(21,31) v1) r(2,, vz)) 0) Ar 2 (1 -) -2 + (1- (11)
Further if we assume that with probability at least Ki that E(Ar) 2 Ve² + - 3e + e, we have
be) [ (it (r(11.81) - r (x22, y2)] > 0)] 2 1
(12)
In classical classification problems, accuracy is often optimized indirectly using losses like cross- entropy. This hints us to apply techniques for improving prediction accuracy to improve reward modeling. While the BT model uses cross-entropy loss and an antisymmetric structure, in the fol- lowing, we show alternative choice that could lead to a simple classification-basee algorithm.
3.2 THE BT MODEL AS A CHOICE
The BT model is designed to enforce order consistency the following way: it models the proba- bility that h = 1 using (f 3/1 ( U2)), where is the sigmoid function. This allows training with binary cross-entropy loss:
V1) (Fr(22,99) + 1= (f v1) f y2)))] (13) This structure guarantees that flipping the comparison order will also flip the prediction.
LET E 11
3.3 RELAXING THE ANTI-SYMMETRY CONSTRAINT: A CLASSIFCCATIO METHOD
BT's difference-in-rewail structure inherently enforces antisymmetry. To better understand this, consider an order model A that outputs a preference vector for both prompt-response pairs, i.e., f = , H2). where H1, H2 : X' x y (1, } and ideally align with (h, -h). The BT model imposes hard constraint such that H, - -H2. With sufficient data, instead of explicitly enforcing this constraint, the structure could be learned implicitly by ensuring order consistency, i.e., Hi Re and Hg ~ hh. Consider single model e.g., neural network or tree Har. Under this construction, the order consistency could be written as foe := E(h Har(11, y1) A -h Har(x2, 82)). we could train model targeting on predicting these two parts separately, and if the model predicts well, it should satisfy the antisymmetry constraint. A union bound of order consistency is
(14) instead of directly enforcing order consistency, we can use the classification accuracy of each prompt-response pair as surrogate. In practice, this means training classifier by treating the annotations and prompt-response pairs independently. Then the logit can be used as a proxy for the reward model. For alternative perspective: instead of learning the joint probability P(i j) that depends on both players i and j. we focus on learning the marginal probability Pli wins). These two are related via Jensen's 's inequality, with further details provided in Proposition D.I
Loc S Let == E(h Har((1, + E(-h
(arl(x2.33))
In both BT and classification- based reward modeling, there is no theoretical requirement to limit comparisons to the same prompts. For classification models, this is intuitive, they do not rely on paired data. Similarly, in traditional BT applications, random pairwise comparisons among players common. This further motivates our investigation into how random comparisons across different prompts affect reward modeling performance.
4 EXPERIMENTS
Objectives of Experiments In this section, we present empirical results to validate our key in- sights and methodological contributions. We begin by elaborating high-level design motiva- tions for experiments. We aim to address the following questions:
1. How effective are different learning objectives given by the order consistency framework? Specifically, how does the performance of a classification-bar model compare to the widely used BT model in reward modeling? (Section[4.1
2. How do various reward modeling methods perform as annotation quality and availability changes? (Section
3. How good is the cross-prompt comparison in annotations? What is essential in determining whether this annotation design is beneficial or not practice when is available? (Section

---

Published as a conference paper at ICLR 2025
Reproducibility, Controllability, and Computational Efficiency Our experiments are designed with three key desiderata in mind: high reproducibility, controllability to enable diverse ablation studies and computational efficiency. We prioritized these principles in all aspects of our experimental setup:
Key Design Principles:
Base Models. We conducted experiments using three open-source LLMs at different scales:
Gemma2b, Gemma7b, and LLaMA3-8b (Team et al. 2024 Meta 2024). alongside their SFT- ed versions (Gemma2b-SFT, Gemma7b-SFT,, LLaMA3-8b-SFT), following the setup in Stiennon et al. 2020). Details are provided in Appendix
Annotations. To ensure controllability in generation and annotation, and closely simulate human annotation processes, we followed approaches from Gao et al. (2023): Liu et al. (2023); TTran &
Chris Glaze ) Dong et al. (2024) to utilize open-source golden reward models as annotators to maintain affordability for the community.
datasets (Bai ct al. 2022a), as these are extensively studied in the context of reward modeling, and open-source
Datasets. We used the Anthropi c-Ha ess and Metthropi--Helppeee
golden reward models are available (Yang et al.) 2024b Dong et al. 2023 2024).
Evaluation of Learned Reward Models. We evaluated the effectiveness of different reward
models using Best-of-N (BoN) sampling following Gao et al. (2023). This choice was driven by
three key considerations:
(1) Performance: Empirical studies show BoN achieves better performance than PPO (Dong
et al. 2023 Yuan et al. [203 Gao et al. 2023 Coste et al. 2023).
(2) Stability and Reduced Engineering Overhead: BoN requires no hyperparameter tuning and
is more stable than PPO, leading to more consistent and interpretable results (Ivison et al. 2024
Xu et al. (202)).
- (3) Computational Efficiency and Reproducibility: BoN's reusability across generations dur- ing test time makes it more computationally efficient compared to policy-gradieen optimiza- tions (Li et al. 2023; Stiennon et al. 2020). In contrast, using PPO (Schulman et al.) 2017) for our 12,000 experimental setups would be computationally prohibitive since each setup requires distinct LLM fine-tuning.
For more details on experiment setup, please refer to Appendix E
4.1 COMPARING THE BRADLEY-TERRY AND CLASSIFICATION OBJECTIVES
Experiment Setups. In this section, we compare reward models trained with different learning ob- jectives - the BT model and the Classification model - as well as their different implementations. For the BT model, the Siamese structure (Bromley et al. (993) is required, making MI.P the only viable backhone implementation (denoted in the plots as BT-MLPP). For a fair and clear comparison to BT and to isolate the source of gains from implementation, we implement the classification model with both MLP (denoted in the plots as CLF-MLP) and the LightGBM (Ke et al. (017)) (denoted in the plots as CLF-LGB) given its wide success in machine learning applications [Bentéjac et al, (021) and especially its successful application in embedding based reward modeling (Sun et al. 2023). We evaluate those reward models using BoN with N=500, reporting improvements over base models to provide clear comparison of their performance.
Harmiess
Helpful
Golden Rewerd value Imprevmmen
H1 MEL
Golden Reward Value Improvement
6--
30
1.5
01-
09
40
(LAMA3 So
(CAMA So
Base Models
Base Models
Comparison between BT and Classification (CLF) reward models. In general, the classification reward models achieve better performance than the BT reward models, with the added flexibility of using off- the-shelf classifiers beyond MLPs. Error bars given by runs with different seeds.
Figure 1:
7

---

Published as a conference paper at ICLR 2025
Results and Takeaways. Figure I presents the results on both datasets. The x-axis shows different base models, and the y-axis shows the improvement on golden reward values on test prompts using BoN (i.e., the relative performance gains achieved through the reward models). The results indicate that classification- based reward models not only perform better than the BT reward models but also offer greater flexibility by allowing the of diverse off-the-shelf machine learning algorithms. This makes them a competent alternative to BT models in reward modeling.
4.2 How ANNOTATION QUALITIES AND QUANTITIES AFFECT PERFORMANCE
Gemmazb
Gemma2b-SFT
Comma7b-SFT
LLMMA-8B
Coiden Reveard Villaee
1.75
1.0
2.2
1.00
3.09
1.00
3.00
2.50
$10-
07
Annotation Emor Rate
Annotation Error Rate
Annotation Ertor Rate
Annotation Error Rate
Annotation Ertor Rate
Annotation Frror Rate
Gemma2b
Gemmazb
Gemma
Gemma76
LLaMA3
LLaMA3
Colden Boward Velues
25
Annotation Error Rate
Annotation Error Rate
Annotation trror Rate
Annotation Error Rate
Annotation Enror Rate
Annotation Error Hate
Figure 2: Changing the annotation quality. Top row: Harmless; Bottom row: Helpful. Results with 5 seeds.
Gemma2b
Gemmazb-SP SFT
Gemma7b
Gemma7b SFT
LLaMA3-B0 9b
LLaMA3 SFT
125
3.55
1.5
159
3.59
1.0
1.8
Coliten Rewward
1.00
168
142
200
1.0
0:50
024
059
655
3:0
0.5
000
6.0
1300 zin jadea ecice |raining Annotationn
J30CE axs 2800 483
20899 Juict 2ax0 4300
14000 2850 Joice 1o00
LOOK 20000 39383 scice
Juice 2006 30000 4933
Traiiing Annotation ons.
Training Annetations
Traning Annohations
Training Annotations
Training Annotations
Gemmazo
Gemmazb SFT
Gomma7b
Gemma SFT
LLaNA3 8b
LLaM33 Bb SFT
Coiden Baroord Walues
2091 p000 exi0 4930
L0bi0 24830 id90 siDeo # Training Annotations
1000E 20000 0190 6193
JURE 1000 # Trsiming Arnotations
ca00 2930 TTaining Annotations
2009 2000 1980 4983
Tranng Annotation
TTssning Annotations
-raining Annotations
Figure 3: Changing annotation availability. Top Harmless; Bottom
Helpful. Results with seeds.
Experiment Setups. In this section, we systematically vary both the quality and quantity of anno- tations empirically assess the performance of different reward models under diverse conditions. For annotation quality, we use the sigmoid instantiation of equation 8 i.e., ((Ar)= = (BAr), and vary the annotation quality parameter over the range [0.5, 0.7, 1.0, 3.0, 5.0, 10.0], using a fixed set of 40000 annotations for training. These 3 values correspond to annotation error rates ranging from 5% to 38%, which aligns with realistic human annotations (Zheng et al. 2023 Coste et al. 2023 Dubois et al. 2024). To explore the impact of annotation quantity, we experiment with datasets con- taining 5000, 10000, 20000, 40000) annotations, while holding 3 constant. Additional results cross-sweeping those two set-ups can be found in Appendix
Results and Takeaways. Figure presents the results of varying annotation quality. From the plots, it is evident that as annotation error rates increase, the classification models exhibit greater robustness compared to the BT models, experiencing smaller performance drops. On the other hand, the BT models outperform classification models when annotation quality is high (i.e., less than 10% wrong labels). Figure shows the results of varying annotation quantities. We can conclude from the results that the classification models consistently outperform BT models, not only delivering superior performance with the same number of annotations but also demonstrating more consistent improvements as the number of annotations increases.
8

---

Published as a conference paper at ICLR 2025
4.3 LLM ALIGNMENT WITH PREFERENCE ANNOTATIONS BETWEEN DIFFERENT PROMPTS
Experiment Setups. In this section, we examine how cross-prompt comparisons, as opposed to annotations on the same prompt, affect the performance of different reward models. Specifically, for each training prompt, two responses are randomly generated by the LLMs and then presented to annotators (the golden reward model) for preference labeling. In the standard reward modeling setup, the annotators label response pairs generated from the same prompt. In the cross cross-prompl annotation setup (denoted by the postfix X-Prompt in the legends), we randomly select two prompt- response pairs from the dataset for comparison and annotation. We present the results with / = in the preference annotation processes and 40000 pairs of annotations in our main text as the most common setting. We provide full results over other experimental configurations in Appendix F
Harmlass: Random Comparison
Helpful: Random Camparison
Roward Value Imprevement
Golden Reward Valuue Improvement
HEREP CLE C
0.-
CLE-U
MUP Phompt
24-
O.F4G8 24hronpe
0.F4GG
-44
VA 0000000
-
0.2
Gemmazb
Gemmazb SFT
Gemma7o
Germaao SFT
LLaMA3- do
LLMMA3-B Bb
Gemmalb
Gemmazb SFT
Gemma7o
Gemma7o SFT
LLaMA3388 80
UaMA3 ab
Models
Base Models
Figure 4: Results comparing cross-prompt comparison based annotations. Preference annotations on cross- prompt comparisons outperform same-prompt comparisons. Error hars are from 5 runs with different seeds.
Results and Takeaways. Figure shows the results across 2 datasets, 6 base models, and 3 dif- ferent reward modeling methods. Shaded plots represent results from cross-prompt comparisons. The y-axis measures golden reward value improvements achieved through BoN sampling using the respective reward models. From these results, is clear that cross-promp comparisons outperform same-prompt comparisons, offering substantial improvements in reward model performance.
Harmlest: Simllar Comparisan
Helpful: Similar Comparisan
Goldlen Reward Valuee Improvement
Goldden neward Valuee Improvement
0000000
STA
www.k www.
a 00000000000000000000000000000000 00000000
CLE
CUP HLF X.hrorpt
CUP Poooo
T2
CUF siirar
1.5
CLF-LGR
200000
500000 p000000
3.
3.
Gemmaže
Gemma2h Gemmaa7b
Gemma75 SET
LLaMA3-00 8b
LLaMA3 8b
Gemmazo
Gemma26 SFT
Gemmaab
Gemma7b-SFF
LLaMA3-60 88
LLaMA3-88 8b SFF
Base Models
Base Models
Härmless: Diversified Comparison
Helpful: Diversified Camparisan
Gelden Reward Value improvement
Golden Reward Value Imprroeen improvement
HIME Hornot CLEME
10
CUAACE
CL Coeeet
300 CLP.0R-Chuee
02-
0.2
Gemma2b
Gemmazb-Si SFT
Gemma 70
Gemma70-SFF LLMMA3-80
LLaMA3 LO-FT
SD-
Gemmuz0
Gemmaz6-SB SFFT
Gemma7 To
Germa7o--S LLMMA3-00
ULMMA 9D-SFT
Base Modals
Base Models
Figure 5: Results comparing cross-prompp comparison- based annotations on syuthetically generated similar or diversified comparison pairs. Cross modeling with same- prompt response pairs lacking diversity. Error bars are from 5 runs with different seeds.
Cross-prompt prompt comparison significantly improves the performance of reward
Further Investigation. Intuitively, the cross prossp comparison can improve the quality of anno- tation since it increases the diversity in generation. To further explore this, we introduce two synthetic

---

Published as a conference paper at ICLR 2025
setups designed to analyze the source of gains. In the Similar Comparison setup, response pairs generated for a single prompt exhibit similar quality and lack diversity. the Diversified Com- parison setup, response pairs for single prompt are of different qualities. Specifically, we use golden reward model to score 10 generated responses per prompt and select the two middle-ranked responses to simulate the Similar Comparison setup. Whereas in the Diversified Comparison set- ting, we use the highest and lowest-scored responses to construct the response pair for each prompt.
Results and Takeaways. Figure 5 shows the results of those two synthetic setups. We can con- clude that cross-prompt comparisons are essential when responses for a single prompt lack diversity. In the Similar Comparison setting, the conventional same- prompp comparisons often fail to pro- duce informative reward models that can improve golden reward values. In contrast, cross-prompli annotations substantially enhance performance in such cases. In the Diversified Comparison set- ting, the need for cross-prompt comparisons diminishes as response diversity increases, though they do not have evident negative impact reward modeling performance. Comparing results across both settings, as well as results in Figure 4 we find the superiority of cross cross-prompt comparison is reflected in its general capability regardless of response diversities the performance achieved by cross-prompt annotations is more stable and has lower dependence of the response diversities.
Harmless
Helpful
Golden Reward Value Improvement
8T-MLP, Cor::
Golden Reward Value Improvement
MLP. Cor::
0.30
CLF-MLP, Corn: 0.811
CLF-MLP, Con:: 0.742
CLF-LGB, Corr: 0.809
CLF-LGB, Cor:: 0.865
0.15
0.10
0.05
Random Similar Diverse
Random
0.00
-05
Similar
Diverse
0.05
-1.0
-0.
0.0
05
1.0
-1.0
-08
-066
-044
-0.
0.0
0.
0.4
n.
Averaged Absolute Reward Difference
Averaged Absolute Reward Difference
Figure 6: Comparing the averaged absolute difference in in pairwise annotations (x-axis) and improve- ments achieved by using cruss-prompt annotations (y-axis). The two variables are highly correlated.
Finally, we examine the relationship between the average absolute score differences in pairwise annotations (x-axis) across three setups (i.e., Random for randomly select two responses for each pair, Similar and Diverse for the two synthetic settings) and the corresponding performance im- provements from cross-prompt annotations (y-axis). Scatter plots with linear fits are shown in Figure [6] The strong correlation indicates that cross-prompt annotations are most beneficial when same-prompt responses lack diversity. Importantly, pairwise data in the synthetic setting of Similar cases and the Random settings are similar, this im- plies that practice, when randomly selecting two responses for a single prompt to be annotated, those pairs are likely facing the challenge of high response similarity. And this is the case when cross-prompl annotation can be applied improve performance.
find the averaged reward differences between
Conclusive Remark on Experiments Our experimental results highlight the advantages of classification-based reward models over traditional BT models, particularly in terms of flexibility and robustness to annotation quality and quantity. While BT models perform better under high- quality annotations, classification models demonstrate superior overall performance and resilience to increasing annotation eтror rates. Additionally, our empirical studies on cross-pre prompt compar- isons show it significantly improves reward modeling, especially when responses the same prompt lack diversity. Through synthetic experiments, we further demonstrate that the challenge of limited diversity is likely occur practice, providing additional justification for exploring this annotation method in future research.
It is worth noting that none of those two synthetic setups is realizable in practice without reward model. And the randomly sampled pairs are the only realistic setups for annotations.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
We sincerely appreciate the efforts of our anonymous reviewers, ACs, and SACs for their valuable feedback and constructive suggestions, which have made our paper complete. We thank Ziping Xu and Jalaj Bhandari for their carly-stage discussions and encouragement during the 1st RL Con- ference, which was crucial in motivating this paper. Additionally, we are grateful Rui Yang and Ruocheng Guo for their insightful discussions that contributed to the development of this research.
REFERENCES
Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In International conference on machine learning, pp. 1638-1646. PMLR, 2014.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. general theoretical paradigm to understand learning from human preferences. arXiv preprini arXiv:2310. 12006, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08073. 2022b.
Candice Bentéjac, Anna Csörgö, and Gonzalo Martínez- A comparative analysis of gradient boosting algorithms. Artificial Intelligence Review, 14:7-1999967 2021.
U Bockenholt. logistic representation of multivariate gaired-oomprrison models. Journal of
mathematical psychology, 32(1):44 1988.
Thijs Bos and Johannes Schmidt-Hieber. Convergence rates of deep relu networks for multiclass classification. Electronic Journal of Statistics, 16(1):722--2773, 2022.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345 1952.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verifi- cation using a" siamese" time delay neural network. Advances in neural information processing systems, 6, 1993.
Manuela Cattelan. Models for paired comparison data: A review with emphasis dependent data. Statistical Science, 2012.
Xiaohong Chen and Demian Pouzo. Estimation of nonparametric conditional moment models with possibly nonsmooth generalized residuals. Econometrica, 80(1):277-321, 2012.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132, 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- tems, 30, 2017.
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv:2310.02743 2023.
Geert De Soete and Suzanne Winsberg, A thurstonian pairwise choice model with univariate and multivariate spline transformations. Psychometrika, 58(2):233-266, 1993.

---

Published as a conference paper at ICLR 2025
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint 0rXhv:2304.06677, 2023.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.00 240.02535.07 2024.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024.
Lester Ford Jr. Solution of ranking problem from binary comparisons. The American Mathe- matical Monthly, 64(8P2):28-33, 1957.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Duncan Guest, James S Adelman, and Christopher Kent. Relative judgement relatively difficult: Evidence against the role of relative judgement in absolute identification. Psychonomic Bulletin & Review, 23:922-931, 2016.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Shangmin Guo. Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint 2oXiv:2402.04 04792, 2024.
Ruijian Han, Rougang Ye, Chunxi Tan, and Kani Chen. Asymptotic theory of sparse bradley-terry model. The Annals of Applied Probability, 30(5):2491-255) 2020.
Hamish Ivison, Yizhong Wang. Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A Smith, Yejin Choi, and Hannaneh Hajishirzi. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. arXiv preprint arXiv22406.09279 2024.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang. Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: highly efficient gradient boosting decision tree. Advances neural information processing systems, 30, 2017.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Tor Lattimore and Csaba Szepesvári. Bandil algorithms. Cambridge University Press, 2020.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00261 2023.
Ziniu Li, Tian Xu, Yushun Zhang, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effective, and efficient method for aligning large language models. arXiv preprint arXiv:2310.10500 2023.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. arXiv preprint (XXiv:3309.066577 2023.
12

---

Published as a conference paper at ICLR 2025
Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adver- sarial regularizer. arXiv preprint TrX1v:2405.164366 2024.
Duncan Luce. Individual choice behavior, volume Wiley New York, 1959.
AI Meta. Introducing meta llama 3: The most capable openly available Ilm to date. Meta AI, 2024.
Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang. Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. arXiv preprint arXiv: 2ttp::3/2.000880 2023.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 17730-277441 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with relu activation function. Annals of Statistics, 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Roger N Shepard. Stimulus and response generalization: stochastic model relating generalization to distance in psychological space. Psychometrika, 22(4):325-345, 1957.
Gordon Simons and Yi-Ching Yao. Asymptotics when the number of parameters tends to infinity in the bradley-terry model for paired comparisons. The Annals of Statistics, 27(3): 1999.
A Springall. Response surface fitting using a generalization of the bradley-terry paired comparison model. Journal of the Royal Statistical Society Series C: Applied Statistics, 22(1):59-68, 1973.
Neil Stewart, Gordon DA Brown, and Nick Chater. Absolute identification by relative judgment. Psychological review, 1 118(4):881, 2005.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Hao Sun. Reinforcement learning in the era of 1lms: What is essential? what is needed? an perspective on rlhf, prompting, and beyond. arXiv preprint arXiv:23100061177 2023.
Hao Sun and Mihaela der Schaar. Inverse-rlignment: Inverse reinforcement learning from demonstrations for Ilm alignment. arXiv preprint 2tt.:2005./5224 2024.
Hao Sun, Alihan Hüyük, and Mihaela van der Schaar. -uery-dependent prompt evaluation and optimization with offline inverse rl. In The Twelfth International Conference on Learning Repre- sentations, 2023.
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Row- land, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Gen- eralized preference optimization: A unified approach to offline alignment. arXiv preprint wwXiv:2402.05749, 2024.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière. Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295 2024.
I.I Thurstone. A law of comparative jundgrenivejudgmmen... Psychological Review, 34(4), 1927.
13

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXtv:2337709228. 2023.
Hoang Tran and BH Chris Glaze. Snorkel-mistral-paarrr--
2024.
Amos Tversky and Daniel Kahneman. Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainnt, 5:297-323, 1992.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. Trl: Transformer reinforcement learning. lttps:://åthub.
com/hugg i in aace/ /trl, 2020.
Weichen Wu, Brian W Junker, and Nynke Niezink. Asymptotic comparison of identifying con- straints for bradley-terry models. arXiv preprint arXiv:2205.0434.) 2022.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for IIm alignment? a comprehensive study. arXiv preprint arXiv:2404. 10719, 2024.
Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. Regularizing hidden states enables learning generalizable reward model for llms. arXiv preprint arXiv:2406. 10216, 2024a
Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards- in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. arXiv preprint arXiv:2402. 2024b.
Atsutomo Yara and Yoshikazu Terada. Nonparametric logistic regression with deep learning. arXiv preprint arXiv:2401. 12482, 2024.
Yueqin Yin, Zhendong Wang, Yi Gu, Hai Huang, Weizhu Chen, and Mingyuan Zhou. Relative preference optimization: Enhancing IIm alignment through contrasting responses across identical and diverse prompts. arXiv preprint arxiv:2402. 100558 2024.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses align language models with human feedback without tears. arXiv preprint arXiv:2304.05302, 2023.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408.15240, 2024.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprini arXiv:2305. 10425, 2023.
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, al. Secrets of rlhf in large language models part Ppo. arXiv preprint arXiv:2307.04964, 2023.
Rui Zheng, Hongyi Guo, Zhihan Liu, Xiaoying Zhang, Yuanshun Yao, Xiaojun Xu, Zhaoran Wang, Zhiheng Xi, Tao Gui, Qi Zhang, et al. Toward optimal I1m alignments using two-player games. arXiv preprint arXiv:2406. 10977, 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 190.:199900090999 2019.
14

---

Published as a conference paper at ICLR 2025
A RELATED WORK
The Bradley-Terry Model in RLHF Hark back to the seminnal paper on RLHF (Christiano et al., 2017), the motivation for using the Bradley-Terry-opp model is to understand it as a specialization of the Luce-Shephard choice rule [Luce 1959; [Shepard 1957). In simplified version of the two- option setting, the Luce-Shephard choice rule says the probability of choosing a particular option from set of alternatives is proportional to the utility assigned to that option. 'The application of such a rule is well aligned with the Bradley-Terry model Bradley & Terry] (1952) where pairwise comparisons between players can be used to estimate their ability (utility) scores, where game is inherently stochastic while the ability of a player is fixed. In later work of alignment from human feedback Stiennon et al. (2020); Ouyang et al. (2022); Rafailov et al. (2024) such model is directly applied and has achieved great success in improving the quality of LLMs in various tasks when direct evaluation is impossible.
The usage of the Bradley- Terry model in RLHF has been challenged in the literature from different perspectives: Munos et al. (2023) points that the wradley-Terry model can not capture non-transitive preferences, and maximizing the corresponding Elo score can be a different objective as compared to the objective of optimizing preferences. |Aarr et al. (2023) points out that using the Bradley- Terry modelization in the direct preference optimization methods (Rafailov et al. 2024) could lead to problematic overfitting when sampled preferences are deterministic.
Bradley-Terry Model for Parameter Estimation and Prediction The most classic Bradley- Terry model suitable for arena setting has been extensively studied theoretically started by [radley & Teeryy (1522) themselves. FFord Jr (1957) established identifiability and asymptotic theory. More recently Wu et al. (2022) compared asymptotic theory under different identifiability assumptions. Simons & Yao 1999) studied Bradley-Terry when number of players diverge under the assumption number of competitions each players played also diverge. In contrast Han et al. (2020) studied the setting when comparisons are sparse and proposed a consistent procedure needs only O(N log"(N)) comparisons, On prediction side, using features to predict ability was explored soon after Bradley & Terry (1952), Springal] (1933) assumed ability was a linear combination of features while De Socte & Winsberg (1933) used spline functions. [Bockenhol] (9888) cast Bradley-Tere as a special logistic regression problem and opened investigation from this len. Chen & Pouzo (2012) developed theory for a general class of nonparametric models including logistic regression. On deep learning side
Schmidt-Hibber (2020) studied asymptotic theory deep neural network for nonparametric regres- sion and a follow up paper Bos & Schmidt-Hieber (2022) studied nonparametric logistic regression using deep neural net. However these theory cannot be directly applied Bradley-Tery model because we need a unique reward model for all prompt-response pairs and the probability is not arbitrary but soft max from passing two pairs single network. Studies like Bos & Schmidt-Hieee (2022) could be seen as two pairs passing two arbitrary neural networks rather than one.
Non-pairwise data (Ethayarajh 2024) and cross prosspproppt comparisons (Yin et al.) 2024) were explored in the literature. The Ethayarajj et al. 2024) is rooted in prospect theory (Tversky & Kahneman 1992) that risk- averse human prospective theories assign different values to losses and gains. And RPO (Yin et al.) proposed to reflect the complex nature of human learning in alignment through involving response comparisons among both identical and similar questions. On one hand, RPO supports our insight into making cross-prompt comparisons in practice by drawing inspiration from human learning. On the other hand, the motivations and implementations between our work and RPO both different: in our work, the cross-prompt annotation practice is motivated by a "why nor" ob- servation - given the fact that there is no specialty of considering single-prompt comparisons in our order--onsi theory and the BT model for predictions. Besides different motivations, there are several significant differences between RPO and our cross-prompt comparison: RPO does not study the annotation efficiency. or what is the optimal way of doing annotation under budget, yet our focus is on studying the cross-prompt annotation as an alternative way of doing annotations. Moreover, RPO considers strategic re-weighting method based on embedding space similarity among responses, leading to a direct alignment method. All of those works are direct alignment methods (Zheng al., 2024 Liu et al.) 22224] Zhao et al.) 2023 Azar et al. 2023 Ethayarajh al.) 2024). which do not explicitly build reward models as their intermediate steps. One potential draw- back of those methods without explicit reward models is that will be challenging or impossible to
Data and Cross-Prompe Comparisons in RLHF
et al.
KTO
2024
15

---

Published as a conference paper at ICLR 2025
conduct enference-time optimization (Liu al. 2023
especially when compared with em- bedding based liggt-weighte reward models. And recent work has demonstrated the superiority of two stage methods over direct alignment methods (Ivison et al. 2024
Representation Learning in Reward Modeling In our work, we separate the reward model learn- ing from the representation learning task and focus only on the reward modeling part to better isolate the source of gains. Recent advance in generative reward models highlight it is possible to use gener- ation tasks to regularize the learned embeddings and improve reward modeling performance (Yang er al., 2024a). Also in /Zhang et al. (024)) the generative verifiers construct reward predictions through next-token prediction to maximally leverage the ability of L.LM as token generators to im- prove their evaluation ability. While our research the reward models is orthogonal to this line of research, future exploration on combining either embedding learning or generative ability of LLMs with different order consistency objectives could be highly promising directions to enjoy the im- provements from both sides.
B UNDERLYING ASSUMPTIONS OF APPLYING THE BRADLEY-TERRY MODEL PREFERENCE ANNOTATIONS
Different from those prior works that challenge the practical application of the BT model Azar et al. 2023 ; Munos et al. 2023) ; Tang et al. 2202)). we revisit the basic assumptions of modeling human preference annotations with the BT model, and answer the following questions:
What are the underlying assumptions when we assume the BT model can be used to model the preference annotations?
The canonical interpretation of how to apply equation T iin preference-based learning is that: when randomly sampling human annotator from the population, the human annotator's choice of the preferred response is proportional to the response's utility value. Formally, we use Φ, U1 3/2 to denote the prompt and responses, the above interpretation implies the following assumptions:
Assumption B.1 (Existence of Deterministic Oracle Utility Value). Given I, the utility value of any response y exists and is deterministie. i.e., Tz,y, Va, y deterministic real value.
Definition B.2 (Individual Annotation under Bias). For annotator A, the annotation result is de- terministic and depends on the comparison of their biased evaluation of the utility values of both responses yı and y2, according
PA(MI v2/=) = ((-(4) 2,91
(A) 1(rmm - -aaa > 6(x, V1, A) - b(x, y2, A))
(15)
explicitly write the assumption of applying the BT Assumption B.3 (Logistic Difference Assumption). The b(z, v1. b(xz 92. A) is sampled iid from a standard logistic distribution for all x, U, A:
Based on the above definitions, model to this randomized annotation process:
P(b(x, U1 A) b(z, U2. St) I+e-
(16)
Remark B.4. (Transitive property of difference) reader might be (rightfully) worried if this as- sumption is consistent with transitive property of difference, e.g., when considering multiple com- parisons we have to have b(x, y/, A) b(x,, 1/3 A) = b(x, y1, A) - b(x,, y2, A) + b(xr, 1/, A) b(z, Va. A) while knowing sum of (independent) logistic distributions is not logistic. One should be aware that the two terms being summed are not independent and the assumption can be achieved by assuming all annotator biases are independently Gumbel distributed with the same scale parameter.
With this assumption, we arrive at the BT-rype model Proposition B.5 (BT--rppe Model on Annotation under Logistic Difference Assumption).
P(y > V2(æ) P(r Tz,02 b(x, y1. A) b(,, V2. A))
(17)
(rem Taagg
In comparison, alternatively have the following assumption and have another model:
16

---

Published as a conference paper at ICLR 2025
Assumption 8.6 (Gaussian Difference Assumption). The b(x, y1, A) b(x, y2, A) is sampled from a standard Gaussian distribution:
v1 A) - b(r, y2, A) 1))
(18)
Proposition B.7 (Model on Annotation under Gaussian Difference Assumption).
Piy > y2 x²) P(rz.g, T2zy2 > b(x, V1. A) b(x, 92. A)) (T Tnya),
(19)
where &e is the CDF of the Gaussian distribution.
The next paragraph elaborates implications of those different assumptions.
Understanding the BT Model from Gaussian Assumption on Players' Performance in Game Technically, the BT model assumes that a player's ability in specific game can be rep- resented by score. To better understand the rationale behind the BT model, we can take step back and consider a Gaussian assumption for game performances. Specifically, we assume that in each two-player game, player's performance can be modeled Gaussian distribution centered on their score, with the variance of these distributions capturing the stochastic nature of the game and the variability in the players' performances.
For instance, when player A with a score of SA and variance A competes against player B with a score of Sg and variance dg. the probability that A defeats B in game (denoted A > B), given the Gaussian assumption on performance, is
P(A> B) = P (2. 2 23lTa ~ o3) ~ N(SB. ()) = 2 erf ( SA SA-Sp (22 + o
(20)
This is known to be the Thurstonian model (Thurstone) 1927). In practice, if we further assume that the stochasticity characterized by performance variance game-dependent rather than player- dependent, the denominator in equation 20 becomes player-agnostic constant, which can be ab- sorbed into the players' scores. In other words, the absolute values in the scoring system are scaled by the inherent variability of the game.
In the BT model, the tanh(-) rather than the error function erf(-) used, for the sake of better empirical fit and mathematical convenience Formally, the BT model is
P(A - B) = 1 T e-AA-S))
(21)
C ASYMPTOTIC THEORY ON BT REWARD MODELING
Recall that have known embedding function (): x' [0, 1] such that there exists unknown function g : Rdd + R and the true utility function r(y.*) = g Tr x)) for all y. x. We assume the embedding to have a range of [0, 1], the pretrained model do not have this range, we
can scale it to be. Then reward modeling reduce to learn the function Observe that under this formalism there is no need for a comparison have the same prompt.
C.I ANALYZING BT REWARD MODELING AS NON-PARAMETRIC LOGISTIC REGRESSION WITH ANTI-SYMMETRIC STRUCTURE
Denote our reward model as fg. parameterized by 0, when there is no confusion, we will ab- breviate it as The reward difference between two pairs of prompts and responses (æ1, 31) (Iz, 82) is then م ( (#1. V1), (T 82)) := f I1 = T2. i.c., having matched prompt but is not necessary. And the predicted probability that (I1, V1) better than (I2, is then afA (1 v1), (I2, 82))) with a being sigmoid func- tion. This the same as having a softmax over two rewards, i.e., of (x1.1)) ( y2)))
V (( f1)) )) - (( B2)). We could have
So equivalently, the BT model assumes the players' performance differences are sampled from logistic distribution rather than Gaussian distribution.
17

---

Published as a conference paper at ICLR 2025
softmax(f(W(aa)) v1)), F( (2 v2))). The second viewpoint is easier in applying theoretical analy- sis techniques developed in literature (Schmidt-Hicbee
2020; Bos & Schmidt- Hieberr (0022) To that end, denote the vector of two rewards as & and the class probability is then softmax Training such reward model reduce to train a classifier with cross entropy loss whose predicted conditional class probability being softmaa (f( (11, 91 )): F(V (V (22, 12))). Our theoretical development gener- ally follow Bos & symmetry such that if we exchange the role of æ1, V1 and 22, y2 get a negative difference, By showing this particular logit-Siamese architecture could approximate the true class probability, we can deduce that a BT reward model could approximate true reward.
(2022). This is a special case of an MLP that could preserve
For notational convenience, we denote the embedding of the ith pair as L 2 for i 1,-..,9. n,, without lose of generality we assume ( E [0, 1]d. Denote the true preference prob- ability as Pú and model predicted as p(+) = (a (T)() (0) )),1 a(P ()) = softmax ( (6 1 )) denote the preference vector h! equals (1,0) if the first response pair is prefered and (0, 1) otherwise. The BT model is reduced to a (pairwise) classification problem such that the likelihood given by
1 n "(o(o))oo( log(p()),
ECE(P)
arg arg min LcE (p)
PeFe
is unrealistic to assume we can find an NN that actually attends the global minimum, we denote the difference between the fitted NN and the global minimum as
An (po. p) E (CF(D)
min (FF(
pefa
We consider truncated KI. risk, similar to Bos & Schmidt-Hieber (022) to overcome the divergence problem of KL risk.
Definition C.1 (Truncated KL risk (Bos & Slhmidt-Hicber 2022 The B-truncated KL risk for a probability estimator p
Pe
RB D) = E po min
B, log
(22)
We consider MLP with ReLU activations for Fo. depends on depth and width vector m = (mo,... m i.e.,
F(L.m) {f me + RR f(x) WL WL-1... Wide, Woz}
where the max x v, 0) is the ReLU activation with bias
Assumption C.2 (MLP reward model). We will further assume the network parameters having
norm restriction and sparsity, common assumption studying MLP for classification problems
Yara & Terada 2024: Bos & Schmidt-Hieber 2022). That is, in this work we consider networks
from the family
F(L, m) = fif E F:fEF(L,m) max max Wyllo [0yl00) S 1, Σ (IWW, luylo) S
j=.....
1=0
(23)
Another useful function class in theory is the softmax output version of the reward model, i.e..
consider
FL = (p( 1 M2 pp (W))
F(L,m.s)
Next assume the probability of preference is not too close 0 1, in the form of small value bound
Definition C.3 (Small value bound by Bos & Sehmidt-Hiebee (2022)). Let 2 0 and H is function class we say H. is a-smmll value bounded (a (G-SVB) if there exists constant C' > 0 s.t. for all probability estimated p (po, P1) H it holds that
PP(pe( V22 S <() Ct for all & (0, 1] and all 0,
(24)
18

---

Published as a conference paper at ICLR 2025
which indices that our reward function and design of comparisons should have tail behavior that we do not tend to compare pairs having very different reward function. We denote this family as
S(x, C) == {g= IP ,,,, v (g(V)) و 2 S a--aaaa (4) I Ct for all E (0, 10.1]}
(25)
Definition C.4 (Hölder smooth function). For B > 0 and D C Rd, the ball of Hölder functions with radius Q is defined as
8 f(x) - 87 f(y)]
(D, = {: f:D+R:
1100l + +lllbb
Σ
sup 7:bll EFEED
E
||
1:0 [e]
S Q
(26)
Assumption C.5 (Class of true utility function). We assume that the true reward functions Hölder and the induced probability by softmax is a SVB. Le., we consider the function class
Ga ( (D,
(27)
Note that this is nonempty since constant satisfy the requirement for any B > 0 > 0 with
Q > 1/2 and C 2 20.
Theorem C.6 (Theorem 5 of Schhiitt- (020), approximating Hölder smooth functions). For every function f E (D, Q) and every M max((3 + 1)0, (Q + 1)0/de2) there exist 3)) m
neural network H F(L, m, s) with ( + 1)
(d,6(d [BT] [M488] J...., 1) and S 423(d + B + 1)3+d /d/ log, (M) + 1) such that
H - fl Cq
M
Remark C.7. Note that since softmax with two output Lipchetz with constant have Loe distance between the softmax output being bounded by 2.Q M by applying the above theorem.
Theorem C.8 (Oracle inequality, theorem 3.5 Bos & Schmidt- Hieber (222 Let F be a class of conditional class probubilities and p be any estimator taking value in F. If B 2 and Nn loog( -B //n), then
N (8, log(F),d (; 2 3fr T
RB(Po, S (1 c))
An(Po, + 35 PEF
inf
(28)
(1 + 2) 68B log(N,,) + 272B + 3C, (log(n/Cn)) B)
n
for all 8, (0, and 0 On Sn/e.
Lemma C.9 (Adapted lemma 3.8 of (Bos & Schmidt-Hieber then for every 8 > 0
2022 Let [+t (me + +1),
log m, s)) 11b0) S (88 (L+
(29)
and
log([ (8, Substitute to the first bound and take log yield the second line.
log,N(8,
m, 8)), 11x0) < (s + 1 1og 10
(L 1)d
(30)
largely Note that, although the proof largely carries follow follow/Bos& apply lemma 3.8 of [oos & Schmidt-Hiebeer (2022) in our setting because before softmax layer the two one-dimensional scores came from the same MLP with identical activation. This is only a subset of the family Bos & Hhlmidt-Hiebery (2022) considered in their work.
Sehmidt-Hieher (022) we cannot directly
Proof. For E log(F, (L, m, s), there exists an fo E F(L, s) such that g() = log(softmax(fp (x1), (x2)))
By Lemma of Schhmiit-HHiebbe 2020 we have
((664,, m, m),), || 11 < (88-1 1)V2)
(31)
Let > 0, denote ( the centroid of a minimum 8/4 cover of F(L, m, s), thus for each
fg E F(T., m),). s); there exists f s.t., fj's are interior of a 5 5/2-cover of F(T,, m. s) for g
19

---

Published as a conference paper at ICLR 2025
log( (L, sL, by covering property there exists a s.t. l15s fu1l2 5/2, by proposition C.6
of Bos & Schmidt-Hieber (002)) we will abbrivate (x() as
|gg log(ssftmaxx( ())
fz)) = ||loo (fg. log(softmax(f),
(32)
=2110 f2] 18) fillo S 2 max |15 fj1l ~
Since 9 arbitrary, we have log(softmax (f3, ff)) is an internal å cover of log(Fa m, 8). hence
(á, log(Fo m, 8)), I1 [m0) s (, m, s), l1 - 85 (L + 1)V2
(33)
The second bound is by having mo d. M 1 (since we have scalar reward) and removing inactive nodes we have by proposition A of Bos & Schmidt-Hieber (2022) V S d=
Remark C.10. Readers might suspect this is direct result Bos & Schmidt-Hieher (2022) by first
concatenating two embeddings and training an MLP with this joint embedding to predict softmax score. While this proposal will satisfy the requirements for the theory, it does not provide way to generate a reward for one embedding that does not depend on another embedding and might not be antisymmetric. Theoretically, the rate depends on embdding dimension d rather than the concatenated dimension 2d as directly using results from Bos & Schmidt-Hiebeer (2022).
Theorem C.11 (Truncated KL risk bound). Suppose the true utility function induced probability 2340)04(3+0) (140)84a
(140)8
of preference in 9. (3, Q, C)) for a E [0. 1] and Φn estimator from family Fo (L, m, s) satisfying 1) A(d, logg(n) S L nón, for some suitable constant A(d, 3), 2) min; my Z nio and 3) 8 x nỘ, log(n). For sufficiently large n, there exists constants C¹, C such that when (n) S (" (og((n) (n) then
n. (140)8-2, Let P be an
RB(Po. p) S C² Lon (n) where a Z b means there exists some constant C s.t. a S cb and a x means a S b and b S a
(34)
Proof. We apply oracle bound Theorem C.8 Take 8 n1 1 and
Cn = 1, using the fact that d,
is upper bounded by sup-norm. Then apply Lemma we have
3 ) inf RB p) An Po. p n 68B(s + 1) log 22 t 88-88-11 + 1)d3 + 272B + 3(log(n) B) + 4
Rg(p.P)
2
p
(35)
n1
(2-o)a 8
We pick M = [c2 (1+0)++4 n (1+0)0+4 for small small with large enough n, we apply Theo- m, s) with L and maximim width bounded by M4/3 n/3 and similarly we have s S M// (M) cli Bli (M). Whenever we have A(d, 3) log, (n) S I. S nºn is 2 né, and x né, log(n). Observe that the softmax output network satisfy Theorem 3.2 4(4 2Cq.8.d). The 2 before Cq.s,d. difterent to the exact statement of Theorem 3.2 of Bos & Schmidt- (2022) is because our Cq.3,a before softmax layer and since softmax layer is Lipchetz of constant 2. We further have Ci C1+15 4(5 + 2CQ.s,a), we have
rem C.6 its softmax transformed version denote as p is in Fo (L,m,,
have the maximum width
of Bos & Schmidt-Hicbed (2022). we have with C
inf R(po, p) S 8023+a (5 3+22Q (1+ lasi, pET.
+ log(a)) S on log(n)
MI+a
1-
Together with oracle bound Equation 35 and
no, log(n). the statement follows.
Lemma C.12 (Lemma 3.4 of Bos & Schmidt-Hieber (2022)), For any B > 2, P. Q being proba- bility measure on the same measure space, we have
(P,)Q) (L)())Q)
(L
(36)
where for discrete probabilities
=²(P:Q)= I
20

---

Published as a conference paper at ICLR 2025
Remark C.13 (Connecting probability reward). Since we have (vā vb)² (00-b///V/a vb)² use Lemma C.12 indicates that in large subset of the embedding space
po 1 22 pl 1.
22 x VPo - v lo
VVDo + vpl p(1 p)
(( ( (f( w F( 22 x
VonLlog(n)
where ý is a probability between Po and c. the second line is due to mean value theorem. This indicates that the comparison should be between those pairs that relatively close in reward avoid diverging behavior of logit fuucttion
D ANALYZING ORDER CONSISTENCY
Proposition 3.3 (Lower bound on population level order consistency ). Suppose a learned model H achieves objective equation up òe error for some small
and < 3/20, i.e., (10)
Ex Na. h=
òc
Then, with probability at least - over Ar, for any given Ar the order consistency of f with respect to the oracle utility is bounded below by:
(it (r(11, r(,2 0) Ar 2 21 - (((r)))Ar)))))))))))) +e (1- ((Ar())) (11)
Bay Bay
Further if we assume that with probability at least K, that E(Ar) 2 ve - 36 + e, we have
(At ( [F(01, 91) rxx2, 82)1 > 0)] 2 4e K
(12)
Proof. The idea the proof is to first use Markov's inequality to bound probability that for a given distance Ar the preference model not well approximate the annotator and under the event that the preference model approximate the annotator well, we bound the total error combined by preference model and annotator.
By assumption we have the (marginal) error probability averaging over all distances r
1 A + h)
P b.y P.y..vill N2.h
( Ar)
(37)
EAr
P
<dc
Denote the random variable
11 = P (B#hAr))
(38)
by Markov's inequality
PP, (II, 2 (II,
=
(39)
In the event (Ar: : P A + Ar e}. with probability 1 1-6 8 we bound the error rate function of Ar. Condition on År, define the following probabilities:
Pannotator = <(Ar) is the probability that the annotator h is correct (i.e., agrees with the oracle utility) given the oracle distance.
Pannotator = 1 - ((Ar) is the probability that the annotator Hg is incorrect given the oracle distance.
21

---

Published as a conference paper at ICLR 2025
Given the bounded difference between Hg and Ha:
Correct Case: When the annotator correct, the learned model agrees with the annotator with probability at least 1 e. Thus:
Peotrct 2 c)- -((r).
(40)
Incorrect Case: When the annotator is incorrect, the learned model agrees with the annota- tor with probability at most e. Thus:
Pincorret S E(Ar))
(41)
The order consistency of the learned model Ho- with the oracle utility can expressed as:
(m) [ (f((()).)) r(v2, 2 0)
Ar
Peomect
Pammolater
+ Pincorrect (1 Pannotator
(42)
Substituting the bounds and simplifying, we have
En.y. ver I ((((())))) - r(u2, x)) 2 20 Ar 2 (1-e)
(43)
Second part of the proof is by observing that E(Ar) 2 2 + - 3e + implies (1 (1- (Ar) + e. (1 -((rr)² 2 1 4e when 3/20, consider this conditional bound
-rr((,,)))r((,a=))))) Ea. 1 (f((())))) r(v2, x)) 2 0) Ar 2 21-18 4E
(44)
the stated bound fail either 1) II, > e, with probability at most or 2) ((Ar) < + + with probability most K, thus the stated bound is true with probability at least 1 due to union bound on failure modes.
Then by definition of conditional probability, the bound in theorem true.
Proposition D.1 (Classification reward). Suppose data actually coming from BT model Equa- tion T and the score logit P(i wins) is connected to BT reward that for constant C does not depends on
-
Proof. We condition on which j that competed with and apply Jensen's 's inequality
Pii wins) E, [P ob5) Ej
Ui 21 + u
2,
EE (
With some straightforward algebra, we have that
P(i wins)
1 Pii wins) B(ug 2 mz]
Take log each side and substitute u, = exp(r;) then rearrange
== logit P(i wins) 2 - log E[exp(r,)]
have log B[exp(r,)) )] constant.
E EXPERIMENT DETAILS
To enhance the reproducibility of our work, all code, datasets (demonstrations)) fine-tuned LLMs, generated training and test responses, annotations of those responses, and their embeddings will he made publicly available.
22

---

Published as a conference paper at ICLR 2025
Computational Efficient Experiment Design and Reproducibility Our experiments conducted on a cluster having 128 Intel (R) Xeon (R) Platinum B336C CPOs e2. 30GHz with NVIDIA V100 32GB NVIDIA A100 BOG GPU nodes. We use
vllm (Koo et al. 2023) to accelerate the LLM generation process.
To reproduce 12,000 experiment results (since will release the embeddings of the generated responses), only CPUs are needed, and to reproduce all experiments, 6000h CPU-core hours are needed machine with 128 CPU cores, it will take 50 hours to reproduce all of our 12000 experiments (This includes our 5 repeated runs using different random seeds. Running with only seed will take less than 10 hours). Each set-up take less than 30 min CPU-core time usage less than minutes to finish on 128-core server.
Supervised Fine Tuning Stage of Base Models Following Stiennon et al. (2020;; Bai et al. (2022a); Sun van der Schaar (2024), we use held-out demonstration datasets generated by GPT4 on the two tasks to conduct SFT on the three base models we use (Gemma2b, Gemma7b, ILAMA33 8b). Such an SFT procedure generates three SFT-ed checkpoints as additional base LLMs, the Gemma2b- Gemma7b-Sti -FFT, LLMM3--B--FF The SFT takes less than 10 hours (4 hours for the 2b models) using A1 00 GPUs and the TRL framework (yon Werra et al. 2020).
Training and Test Data (Responses) Creation The Harml ess dataset contains 41876 training prompts and 2273 test prompts. The Helpful dataset contains 42846 training prompts, and 2292 test prompts. In our experiment, for each of the 6 base LLMs, we create 10 responses on each training prompt candidates for reward model training. For each of the prompts, create 500 responses and annotate their golden utilities using the golden reward models for testing.
Creating Embeddings We use Gemma2b (Team et al. 2024 to generate embeddings for reward modeling in our experiments. Since we have 6 base LLMs to generate responses and datasets, creating the embeddings those generations requires GPUs. We use V100 GPUs with 32GB memory for embedding generation. Each setting (40000 prompts with generations and 2000 prompts with 500 generations) takes around 16 GPU hours to finish.
Simulated Preference Annotation with Golden Reward Models To simulate the imperfect notation process of human labors, we consider label noises in our experiments following the liter- ature (Ziegler al. 2019: Dubois al. 2024 Coste et al., 2023). However, instead of randomly inject noise to the labeling process, we consider a more realistic annotation simulation using the cognitive bottleneck models studied in psychology (Stewart al. 2005 Guest al. (016): the comparisons made between responses that has similar scores will have a higher possibility of being mis-labeled, formally, have (equation
h(x, yr. (2) ()) r(x, 02)) > 0 (r)r) = E(Ar),
We instantiate E(Ar) (BAr), the sigmoid function in determining the probability of getting correct lable. The 3 parameter here controls the annotation quality: when 3 = O, annotations are purely random, while when 3 + oo, annotations are perfect. In experiments, the default setting of 3 is unless explicitly specified otherwise (as in the secton of experiments studing performance under different noise levels in annotations.)
Hyper- Parper-Paramtters of the LGB models and MLPs To maximally isolate the source of gains, and separate the contribution of methods from their implementations, we use the identical hyper- parameter set up for all experiments (a single default hyper-paramette setup for LightGBM, same MLP configurations for BT and Classification).
For LGB models, we use the default hyper-parameter setting of
hyper-param-Igh
'objective's 'binary', netriccs 'bilory_logloss!
23

---

Published as a conference paper at ICLR 2025
For MLPs, we use minimalism thrre-layer-fecd-forwaaa ferwrforward structure of
hyper-param-mip
'activation' 'ReLU', 'nnits : (1024, 512, 1) "loss'; 'BCELOSs', 'optimizer': 'Adam', 'ir': 10.001', "early_stop patience': '3%, mmxx_epoch's '30'
6
While further sweeping on hyper-parametert for each set up will very likely be able to further im- prove the performance, those engineering efforts irrelevant of the scope of our research focus investigating different methods nor informative in drawing conclusions to answer the above ques- tions.
F ADDITIONAL RESULTS AND DISCUSSION
Figures reporting results on changing annotation qualities. Figure 7
Figures reporting results on changing annotation qualities under different annotation avail- ability. Figure and Figure
Figures reporting results on changing annotation availability under different annotation qual- ities. Figure 10 and Figure 11
Figures reporting cross-prompt annotation results on changing annotation availability and annotation quality. Figure 12 to Figure
Post-hoc explanation of the superiority of the classification- model over the BT model
()). Analytically, the BT model can be used estimate the winning rate each player when matched with another. On the other hand, the dussifiication-base reward model estimates the marginal probability of winning. As we showed in Section both options achieve order consistency, therefore, they both satisfied the minimal requirement for inference time optimization, despite the classification-based score estimation having less information due to the marginalization.
However, given the same dataset, a more informative objective may not be as easy to learn com- pared the less informative objective.
In reward modeling settings, we will have to estimate/learn these logits from noisy (binary) annota- tions. The other side of the same coin (of having less information) in this marginalizationnaveraee that it makes the marginal probability and classification-bant -baed reward score easier to learn from data. i.., we are trading some information that might not necessary for inference time optimization for simpler target during the learning/reward modeling stage. We can see this by evaluating the variance these two targets and one can see that the classification-bal reward score has lower variance, and therefore, an easier-to-lean objective.
Vari, (P(i wins jli, Var,(E, (P(i wins jli,j)ld)) + EVar,, wins jlb,))l))
(45)
=Var, wins (i) + (Va,rr,(P(( wins jli,))l))
The first term RHS the classification-bass target's variance and the second term is negative. So the classification-buue method targets an objective with less noise in it.
(2) Empirically, such an insight can be observed experiments. In Figure 7 we report the results in barplots on changing the annotation quality. In the top 3 panels, the annotation noises are high (having around or above 30 percent error rate), and in those setups, we find the classification-baeed reward models in general outperform the BT reward models. In the bottom 3 panels, the annotation noises are low (less than 15 percent error rate), and in those setups, we find the BT models achieve better or on-par performance as compared with the classification-bar reward models.
24

---

Published as a conference paper at ICLR 2025
Harmiess beta 0.5 (Anritaatonn Error Nate 0,371
Helpful buta 0.5 (Atnotation Enor Rate 0,55)
07:47
Coldenn Doa Vill
Lanu
5as2 Model
Base Model
Flemmless bets 0.7 (Ammtettion Bate 6,33)
Helpful beta 0.7 (Armotitioo Emor Rate
07
Lano
Dase Models
Moduls
Paerriaana bale (Aenstation Rate 0.20)
Helofdd beta (Areotiation Empr faae 0.5)
D1 ML
07:47
Se
Base Models
Baga Modds
Formless (denstation 0.14)
Helpfid Mknnotation Rate 0.14)
SHTT Po Sowe Colda
Sarl 50
Base Model
Base Models
Paeriimus beke (Aenstation Rate 0.0%)
Helplid beta Mknnstation Emmr Rate 0.09)
Base Models
Base Moduls
Harmless (Annstation
5.051
Helpfll (Ainctation Error Rate
Golden Rowsrdvvslues
Gomasb.l
St
of
LOMG St
Nodwls
Bana Models
Figure 7: Experiment results in barplots on changing the annotation quality. Error bars are given by 5 runs by changing seeds.
25

---

Published as a conference paper at ICLR 2025
Gemmalb
Germaa2bSSF
Gemma7b
CommasbbsFT
LLaMA3-8b
1L-MA3-4b-SFT
Coldten Rewaard
2.2
575
872
6.5
675
675
675
675
6.00
2:30
CEP
(a) 5000 annotations
Deemmad
Cemmu2b-SFT
Gemma7b
Comma7b-SFT
L1 ILaMA3-8b
11 L-MAA-
1.59
3.59
6.35
675
6.75
875
6.35-
Dolden
8.%%
875
2.3%
83
62
02
Annotation Error
Annotation Rate
Annotation Rate
Annotation
10000 annotations
Gemmazb
SemmazZb-FT
Gemma7b
Gemma7b-SFT
LLSMA3 db
LLaMA3 ub
Colden Vewerr.aaaaee
3.59
150
1.2
2.20
5.5
5.30
375
5.70
5.30
A25
8.25
A1
02
01
23
F2
01
02
92
A1
20000 annotations
Gemma2b
Gemmazb SFT
Gemma76
Gemma7b SFT
LLaMA3 8b
LLa/A3 ab SFT
Coldee Besourd
1.28
3.25
3.2%
3.25
6.55
575
6.75
875
6.55
625
81
02
03
23 42
01
03
31
92
03
24
01.
63
Annotation Error
Annotation
(d) 40000 annotations
Figure 8: Harmless Dataset: additional results on changing annotation quality under different anno- tation availability, Error bars are given by runs by changing seeds.
26

---

Published as a conference paper at ICLR 2025
GemmaZb-SFT
LLaMA3-Ab-SFT
Colden
Annotation Rate
Annotation
Annotation
Annotation
Annotation
Annotation
(a) 5000 annotations
Gemma20
Gemma2b-SFT
Gemma 7b
Gomma7 b SFT
LLaMA3-8B 8b
LLaMA3-8b-5R
Coolten Whuns
Hlg
41
42
Annotation Rate
Annotation
Annotstion Rate
Rate
10000 annotations
Gemma2b
GemmaZb-SFT
Gemma 7b
Gemms/b-SFT
LLaMA3-88
LLaNA3-UB-SFT
Rowward
31
62
63
24
6. Anmotation Rate
23
34
62
62 Rate
C3
61
94
63
Annotation Rabe
Anrotation Rate
Annolation
Annotaaioon
(c) 20000 annotations
Gemma2b
GemmaZt-SFT
Gamma 7b
Gamma7
LLaMA3-Bb
LLaMA3-Rb-SFT
H
Colden Reooant
X1
83 Annotation Error Rate
C2
05
81
02
63
Annotation Error Rate
Annotation Error Rate
Annotation Error Rate
Annotation Error Rate
Annotation Error Rate
(d) 40000 annotations
Figure 9: Helpful Dataset: additional results changing annotation quality under different annota- tion availability. Error bars are given by runs by changing seeds.
27

---

Published as a conference paper at ICLR 2025
Gemmazb
Gemmazb-SF1
Gemma7b
Golden Rewward Vakuel
225
2.5
177-
1.2
125
123
3.2
1.0
072
335
035
6.5
3.5
075
925
035
6.5
122
Triining Antatations
Taining Annotation
Training Aniotaations
Training Annotations
Training Annolations
Training Annotations
(a) B 0.5
Gemmalb
Gemmalb-SFT
GammaTh
Gemma7bbSFF
LLaMA3-8b
LLaNAB-8b-SFT
Colden Rewaard Velues
159
159
2.2
125
120
125
2.00
3.0
115
635
8.55
KT C2440
938
625
835
JI0C
2009 4399
10890
Training Annotations
Annotation
Annotations
Annistations
Annotations
Annotations
(b)
0.7
Gemma2b
Gemma2b-SF
Gemma76
Gomma7b.55 SFF
LLaMA3 8b
LLaMA3-- 8b SFT
Doldeen Rewarrd Weltes
2.30
156
159
1.0
1.00-
1.5
1:00
1.0
139
389
0.00
3.50
-I- 8140
1990 2839 1889 IEKE Training Annotstions
Jac8 2910 183 4BR Training Annotations
2003 ioice eae 1991 Trainin Annotations
12000 2853 IDice sEb Training Annotations
I0KE 2091 1983 reme Training Annotations
i888 zon e00r 498) Training Annotations
(c)
1.0
Gemmazb
Gemmazb-SF1
Gemma/b
Gemms/b-SFF
LLaMA3-Bb
1.5
3.50
154
159
359
1.5
1.2
Rewward
4.30
168
149
140
100
8:50
036
059
3.0
000
18
B3X 2i9 18d4 4600
J00CE aw 2800 6X0
20099 200t 2000 4929
16000 2869 J004 1000 Traiiing Annolaaiion
L000 2000 3333 4c84
Juice 2000 30090 433
Taining Annolations
Taining Annotations ons
Training Anroiattion
Training Annolations
Training Annolations
(d) β 3.0
Gemmadb
GemmaZh-SFT
Gamma7b
CemmeibbSFF
LLaMA3-Bb
L1 LLNAA3-b--FF
Coldenn Retoart Vekues
3.19
1.2
422
125
120
12.0
143
1.0
1.0
180
052
052
3:0
1000 2009 1869 4034
1908 30000 9999 490
20099 2000 3000 4999
3000 4000 Training Annotatioons
30800 2909
1000 2909 3939 aciC
J008 2000 3909 000 Training Annotations
Training Annotations
"Faining Annotations
Training Annotaation
Training Annotations
(e)
5.0
Gemma2b
Gemma2b-S5
Gomma76
Gomms7b-SFT
LLaMA3.6 8b
LLaMA3-BB
Oolden Howward Wilues
1.75
135
135
3.36
1%%
1.7
1:2
125
125
325
1.2
1.2
8.75
0.5
0.00
355
020
1889 Training Annotations
20009
Taining Annotation
Trainnng Annotations
Training Annotations
Traininn Annotations
Training Annotations
(f) 0 = 10.0
Figure 10: Harmless Dataset: additional results on changing annotation availability under different annotation quality. Error bars are given by 5 runs by changing seeds.
28

---

Published as a conference paper at ICLR 2025
Gemmazo
GeMmaZb-SFT
Gemma7b
Cerma 7b-FF
LLaMAB-80
LLaMA3-Bb-SFT
Geiden Reveard Values
07:4
2891 vdea
vaes aréen * Training Annotations
Inére gis
yeta
Taining Annotations
# Training Annotations
* Tra ning Arnctations
Training Annotations
Tranning Annotations
(a)
0.5
Colden Rowandd Velues
Trainin Annotation
training Annotations
trsining Annotations
training Arnotations
training Annotations
Training Annotations
(b)
0.7
Gemmazo
Gemmazb SFT
Gomma7b
Gomma 7b-SFT
LLaMA3-86
LLaMA3-80 BD-SFFF
Colitee Howarrd Wiles
2043
07-000
2591 zites sxea 1980 Training Annotations
Dee 2839 edax sim Training Annotations
iotee zue cem Et3)
3900 2900 HE8 mil
3008e 2000 1989 eE Training Annotations
Trsining Annotations
ming Arnotations
Training Annotations
(c) 3 1.0
Gemmaze
GemmaZb-SFT
Gemms/b
Gemma 7b-5IT
LLMMA3-86
3-1---1
Coloiee Rowward Valtes
200 sidea 3000 4900
U890 2800 X00 saces
usis 2009 2090 419
1538 2ik4 2000 4900
1200 1909 J6Ka 4684
20099 20000 3900 4XX
Training Annotations
Training Aniotattions
Training Annolations
Tra Training Arnolations
Training Annotalioonnnni innss
Tainiing Annotalions
(d) 3.0
GemmaZo
GemmaZb-SFT
Camma7b
Gamma
LLaNA3-8b
LLaMA3-RD-SFT
Goldeen Benoart Velues
0/44
2093 3000 3000 4900
1000 2909 x003 40000
1066 2000 30800 4000
3099 2006 300 4900
5909 2900 3000 icice Training Annotations
3009 2000 3909 490
Traiiing Annotations
# Training Annotatioos
# Training Annotations
# Training Arnctations
Taining Annotations
(e)
5.0
GemmaZo
GemmaZb
Gomma7b
LLaMA3-88
LLaMA3
Golden Rlwaard Wines
0V.M
2291 xdea
zid0 adda mdm # Training Annotations
1niee ats
s0ta Training Annotations
Taining Annotations
# Training Annotations
* Tra ming Arnotations
Training Annotations
(f) 0 = 10.0
Figure II: Helpful Dataset: additional results on changing annotation availability under different annotation quality. Error bars are given by 5 runs by changing seeds.
29

---

Published as a conference paper at ICLR 2025
Harmless: #Annstations5000 5000
Helpful: Random MAmostation=5000
Golden Rowerd velue imorovement
CLANEE
04
(F-00
A 0 8
Coldlen
Gemmaa2
Gemmaazh:57
Gemmnalb
Camma/Do557
(LaMA3 80
GenmazZ
Gemmaüto
Genma70
Gemmatb-SF
LEAMAZ 8e
UaHHA3-B -B-33-bb-FFT
Hormless: Similar Comparison, #Arnotation-5000
Helpful: Simiar Comparison, #Arnotation=500
Golcen Roward value improvement
(U-- shore:
E1F Eharel
Golden Value
0000000
0000
R
Gemma76
Gemmazb-SrT Gemmaa7
Gemma7B-59 76-5/T
LLMMA3-90
LLaMA3 80-5P7
Gomma7b
Gemma26-5p7
Gemma7b
Gemma76-5p7
LUaMA3 8b-577
Base Models
Base Models
Hormiess: Divarsifeed Comparisan, #Amotttion-5000
Helpful: Diversified Comporisan, #4nnotation5000
Imprroreeeet
Colden Rowerd Value Improvemest Covmmmmr
0.3
Gemmalb
GommaZo SF7
Gemmaa
Gomma70 SFT
LLaAHA3 8b
1LMMA3-80-SF7 SPT
Gemmazo
Cemmazb-Stt Gemma7o
Gemmar7-SSF SFT
LLOMA2-80
11aMA3-8b SFT
Models
Models
Figure 12: Results on cross-promp prompt comparisons, with 5000 annotations, = 1. Error bars are given by 5 runs by changing seeds.
30

---

Published as a conference paper at ICLR 2025
Harmless: #Annstations5000 5000
Helpful: Random MAmostation=500
Colden Rowere Volue morrsvmmnt
CLANEE
GF-00
2.
010000
8
0.2
GemmaaZ
Gemmaazh-55
Gemmnalb
Camma/o-557
(LaMA3 80
M.MMA--0--SF7
Gemma220
Gemmaat
Genma37
Cemmarl-SFF
LEAMAT 8e
LLaHA3-b--SFT
Harmless: Comperisan, #Arrstationm0000
Helpfl: Simiar Comparison, #Arnotation=500
Golcen Roward Value improvement
Coldenn Inproveeet
BTMJ
Ahroppt
fur Bib. BE BIS
(U-R shore:
OLFLSE Jereaat
E3 chane!
03
muinn in eoe
0000 1000-
02
0.2
Gartmw2b
Gemma26 Commaab
Gemma7o LLMMA3-B
LLAMAB 86
Gomma7b
Gemma2o-5p Gemma7b
Gemma76-5p7
HLMMAt-8b
LLaMAB 8b-577
Base Models
Base Models
Harmless: Dioessffied Comperioon, #Armatation-SPOn 5000
Helpful: Diversified Comper #Ammstation5000
Coldon Reward Valuc aoprrvommnnttt
02
CLANE
carte
8 88888
04-
03-
W W0
0.7
Gemma20
GommaZo SF7
Germmal
Gomma70 SFT
LLaAHA3-B
LLaMA3-80 SPT
Gemmazo
Gemmaabb-9 SRT
Gemma70
Gemmanb SPT
LLAMA2-B0
[1aMA3-8B 8- SFT
Models
Figure 13: Results on cross-prompp comparisons, with 5000 annotations, = 10. Error bars are given by 5 runs by changing seeds.
31

---

Published as a conference paper at ICLR 2025
Harmiess: Randem MAmnotationn-0000
Helpful: #Amnstation==0000
Coldeh Rowerd value morrsvmmnt
CLANEE
& 8
(U-40)
04
8
Gemmaa2
Germmazh-57
Gemmnalb
Uamma/D-5F7
(LaMA3 80
M.MMA3-00-SF7
Gemma220
Garmaut
Genma37
Gemmall-SFF
LEAMAT 8e
L1aHA3- 8b-3-
Harmless: Similer Comperistn, #Armctttioomoooo
Helpful: Similar Comparison, #Arnotationn
BTMJ
Golcon Roward Value improvement
Airoppt
Eu EU-- shore:
CLFLSE Jureaat
EI--0 chonge
00000000
Golden (ewward)
01000000000
0.-
Gemma26 Gommasb
Gemma7o-5h LLMMA--8
LLAMAB 86
Gomma7b
Gemma26-5p7
Gemma7b
Gomma76-5p7
HLMMAt-8b
LUIMAS 6b-577
Base Models
Base Models
Hermless: Diversified Comperisors #frmotation 140000
Helpful: Diversified Commerisann @Amrotationn-- Conrootttioo-00000
06.
Coldon Reware U0 improvement
CLANE
23
8 8888
CUF-225
02
Gemmal0
GommaZo SF7
Germaal
Gomma70 SFT
(LaAHA3-80
LLaMA3-88-53 SPT
Gemmazo
Gemmanbb SRT
Gemma70
Gemmatb SFT
LLAMA2-B0
[1aMA3-9B SFT
Models
Figure 14: Results on cross-prompl comparisons, with 40000 annotations, 3 1. Error bars are given by 5 runs by changing seeds.
32

---

Published as a conference paper at ICLR 2025
Harmiess: Randem MAmnotationn
Helpful: #Amnstation=4000
Colden Rowere value moprsvmmnt
CLANEE
8 8
GF-400
1.4
GemmaaZ
Gemmaazh-57
Gemmnalb
Camma/o-5F7
(LaMA3 80
L.MMA3-00-SF7
Gemma220
Garmaat Genma77
Gemmarl-SF7
LEAMAT 8e
LLaMA3-ab-SFT
Harmless: Similer Comperisnn #Armotatioom4oooo
Helpful: Simiarr Comperison, #Arnctation=-0000
Golcon Roward Value imprevement
BTMJ
Colddnn Value Inproveeeen
shroppt
12
(U- shore:
CLFLSE Jereaat
C3-400 chone!
buiin iuii main
02
8
Gemma26
Gemma70
LLMMA3-8b
LLAMAB 86
Gomma7b
Gemma2b-5p
Gemma7b
Gemma76-5p7
HaMMA7-8b
LLaMAB 8b-577
Base Models
Base Models
Hermless: Diversified Comparivon, #Armotation 140000
Helpful: Diversified Commerisoni Momrotttion-o0OO
06-
CLANE
Coldon Reware S0 Improveni
8 8888
Gemma20
GommaZo SF7
Germmal
Gomma70 SFT
LLaAH33-8
LLaMA3-80-5P SPT
Gemmazo
Gemmaabb-9 SFT
Gemma70
Gemmanb SFT
LLAMA2-B0
[1aMA3-9B SFT
Models
Figure 15: Results on cross prompt comparisons, with 40000 annotations, B 10. Error bars are given by 5 runs by changing seeds.
33

---

Published as a conference paper at ICLR 2025
G THEORETICAL JUSTIFICATIONS FOR CROSS-PROMPT COMPARISONS
In both BT and classification- based reward modeling, there is no theoretical requirement to limit comparisons to the same prompts. For classification models, this is straightforward, they do not rely on paired data at all. Similarly, in traditional BT applications, random pairwise comparisons among players are common. This further motivates our investigation into how randomized compar- isons across different prompts affect reward modeling performance.
To further motivate the usage of cross-prompp comparison, we introduce the following notation on annotation quality and analysis as case study under a Gaussian assumption on score distributions. In equation [ we consider special case of & in equation & to be E(-) = (B.), that the annotators' ability is characterized by B. When 3 = 0, we have random annotations:
h(21, 22, V1, (r vn) - -ar---r(0 ((22.2)) > 0 Ar) o(BAr) 8-20 0.5,
(46)
and when 8 + co, have perfect annotations:
P h(11, T2, vn, yr, (2) vn) r(22, U2)) > 0 Ar) (BArr
8 8-000 1.
(47)
In a nutshell, the annotator's abilities and the differences among prompt-response pairs together determine how much preference is correctly labeled in the annotation. In the following, show special case when two responses prompt x are randomly sampled from a single LLM
Example 1 (Annotation Quality under Gaussian Score). When data for pairwise annotation is generated through random sampling of two responses y1. V/2 ~ =((z), we further assume the utility of those two responses are sampled from Gaussian distribution with variance o² i.e., y ~ f(x), r(x,), v) ~ NPpz; Then the annotation quality (x) on such prompt can be defined as the averaged annotation order consistency:
Qpair (x) Ey |
Eyr.m (8)r(( vi) ((x.y2)))
(48)
where Tu = =B)r(ax, v1) rxx, 32)l is a random variable (over y1, U2) and the probability density function of Tz is
(log (
1
(t, V=8803
exp
(49)
1,32
{(1
Proof. To get the PDF of f(r), denote a(pl)) where we use p(z) ((r (2)]=))= r (12 (z), we have p(x) ~ (0, 2,300² follows the normal distribution, and [p(x)] follows folded normal distribution, its cumulative distribution function, mean and variance are given by
2,0
= 2
(1 = 0, =23203 =erf
Mll
(50)
2,0
V
respectively.
To find the PDF of 7 alp)), we use the change of variables formula. If = g(X) and g is monotonic function, the PDF of Y. (v), can be obtained by:
(v = fx(g- (2))
9 (v)
(51)
dy
For the sigmoid function, the inverse is given by:
y/
g (v) = log 1 y
(52)
The derivative of the inverse sigmoid function is:
d
(53)
dy
v(1 - v)
34

---

Published as a conference paper at ICLR 2025
Plugging these into the change of variables formula, we get:
(log 1-r
1 exp
1 8(1 ((1-t)
50(1 = VTT300?
(54)
4
In this special case, it is easy to get some numerical results: when 8 = 1, we have Qpair R 0.6749, e.g., roughly 67.5% of data are correctly labelled by annotators. Similarly, when f300² 2, Qpair 0.7251; when o suggests that the effect of better annotators and the effect of response utility diversity are always coupled: in order improve data quality, we may either improve the ability of annotators further diversify the generation utilities as both of those parameters control the annotation quality. Next, we show that cross-prs prosspprompt comparison in this example can be an effective practice to increase response utility diversity. And show that cross-prompp annotation can improve annotation quality.
4, Qpair 0.7781; when 8 10, Spair 0.8428; This
Pross-Prompt Annotation Improves Quality under Gaussian Score When considering multi- ple prompts T, i = 1, 2, ..., NN, we denote the corresponding responses y/. and scores r(x,, M&) ~ N N(H,, of). In the following, we show that cross- prosspprompt annotation can improve annotation quality.
Proposition G.1 (Cross-Prompp Comparisons Increase Utility Diversity). When data for pairwise annotation is generated through random sampling of two responses y1s 1/2 ~ l(x), and the utility of those two responses are sampled from Gaussian distribution with variance oz i.e, y ~ f(xr), Ta,y ~ N N(P o²),, when there are multiple prompts x, we have
Ey [1aa. T2.vall S Eu aa E By 121l2 (Pam Txaawa
(55)
Proof. Let Fik N(kx; of), and Ø51 ~ NN4, of), with h + then
Fik - ズzt
N(H Mc, of off
(56)
The expectation of jaisk 1zil is given by
E [laak 2221] of of /2 2 exp (- (1x - 11)2 ) 2( + of)
-11)
+ 1p4e -elerf (1mx md
(57)
of + of
as its special case, let lygk N N of),
then
Fikk Tzkk ~ NO 20
(58)
E 1 - 23 1] = -20
(59)
we consider the special case of μ4 MC:
E laik xzal||
Laak Tzs1 2 1,
(60)
the equality holds only if of = of. This is because 2(1+2)2
(1++)² reaches its only minimum when
and we can let é 2x. Since v exp(- x²) + [[=e)f([=]] is a monotonically increasing function at
di 0, equation 60 also holds for μ4 * pa.
This gives us an intuitive result that, in expectation, the reward differences between cross-prompt comparisons are larger than the reward differences between prompt-response pairs sharing the same prompt.
More generally, pross-prompt comparisons improve data quality when the utility distribution of dif- ferent randomly sampled responses given a single prompt is unimodal and symmetric (c.g., Gaus- sian).
35

---

Published as a conference paper at ICLR 2025
Theorem G.2 Prosst Annotation Improves Annotation Quality). When data for pairwise annotation is generated through random sampling of two responses y1 Y2 ~ E(x), and the utility of those two responses are sampled from location-scale scale family with probability density function Ia (x( = f((x - oxx) for I being unimodal and symmetric to O. For any & RR+ - [1/2, 1). first order differentiable, monotone increasing and concave, we have
Ex Qupaiii (x)] = Ey walk 15(1 - -xz.gall D)
(61)
< 2n Ey yn (an valma [5(I T22.22 D] == 12 promppt (21, x2)]]
In the above equation, Ceross prompt (41, .22) is defined slightly different from Qpair (x) which only takes one prompt as its input. We can understand Q (ar) as a special case of 2eross-prompt prompt (4), 22)
2cross-prompt
when I = x. Theorem G.2 highlights that cross-prompl comparisons improve annotation quality for broad class of utility distributions and ç.
Proof. Theorem G.2 follows directly the combination of Lemma Lemma G.5
Section Summarization In this section, we highlighted the theoretical superiority of using cross- prompt comparisons in preference annotations. In expectation, cross-prompl comparisons can im- prove annotation quality since they increase the expected differences between prompt-response pairs.
Lemma G.3. Suppose & : R4 [1/2, 11, first order differentiable, monotone increasing and with f being density symmetric to 0 and unimodal. We have for all μ
B(E(12 + Al) 2 B(E([=)))
(62)
Proof. Without loss of generality, we assume μ 2 0. Suppose the results hold for Ja 2 0, it has to hold for μ S O. To see that, we observe that = ~ P due to symmetry, and apply the result for positive j so that EE{([= + nl)) B(5(1 - =l)) 2 B(E(1 - =\)) = B(E(I=))).
It thus suffices to show the result for nonnegative μ.. To do so, we prove that this expectation, as function of μ. is monotone increasing by taking the derivative.
d E(E(|= + nl)) E djc
E(I= + mi))
Op
d
<(1= + n4)) sign(z + m))
TF
d/2 + 11
d
<(12 + (1)++)))gg sign(( + p)f(2)dz
d|2 + 11
d
/ d/z + ul|
d
5(1z + #1)f(2)dz
E(1= + (2)
d/z + ml
1 d
S d|z
E(1=)]((= - -)dd=
[((=)){(= =1)dz
J
d|=
d
d
<(1=))((=
- -)dz
<(1=)){(= + m)dz
Jn d|z
d|z
2 0
((2=-))= The last line due to unimodality and we must have for all 2 E o0) [(=- 1)) 2 [(z+ for 2 0 while d मंΣ {(1=)) symmetric to 0 and bounded.
Lemma G.4. Suppose I1, 02 iid from a unimodal symmetric location scale family density grs iee. the density of æ1, 12 can be written as (x< = [(xx===)oo)))
being unimodal and symmetric to O. Further, suppose y1, 4/2 ~ By lid with density (y) = f((y ( Hty) ygy for the same f. we have
for a function& & 18+ + [1/2, first order differentiable. monotone increasing and concave, that
1 E(E(101 - x22)) + B((((1s - v2 < E(E(12) - -y11))
(63)
2
36

---

Published as a conference paper at ICLR 2025
Proof. By assumption 1/. V2, 21, 222222 are from the same location-scale family. There exists whose density is such that y1 has the same distribution as 0y2 + My and y1 y22 having the same distribution as v20,z, distribution with scale V02 + og and location Me MMy We first find an upper bound of the left-hand side using Jensen's is inequality as & concave
x2 having same distribution as /2 and œ1 - having the same
1 マ/ B(((( x2))) _ B(E((v1 v2l)) BBE( (B(((20.|=2)) + _ ((E((22,)=1))
(64)
2
= 20 |z1) + Σ 20,1=11)
(65)
S B 1
/2 /20l=11 + 1220y ==]
(66)
(xx + 0z)?
E
|2|
(67)
2
(68)
The righthand side
B(E((11 - (111) E (Ius My + v o03 + +0321)
(69)
2 R $ 00 + 03|21)
(70)
by Lemma G.3
We observe that for all on By that
x + o,)3
S oo
(71)
2
Thus for all |a|, we have & (V03 + of]21)
28
( (V(ostoo)11) and in turn by taking expectation
[el)
the result in the statement is true.
Lemma G.S. We have that for random variables described above and suppose az By are iid, Pas Py are tid
Eor.py Bulor (5(121 - x22 1) S Bos. Ba.ggaa. (E(121--))))))0) -11l))
(72)
Proof. Since ow ou Pa By iid we can rewrite the left hand side as
1 2
Eog.ne Eulonse (E(1=1 x2l))
[E. me Eelor (E(1x1 x22)) + Eo, My Byloumn (5(1g1 - 2/2 92)
(73)
1
Emloaan (5(12)) x2l) + 2 Buloo (E(19) v221)
(74)
and the statement reduces to the two-pair case we showed before.
37

---

Published as a conference paper at ICLR 2025
H DIRECT EVALUATION METRICS FOR REWARD MODELS
Explicitly evaluating reward models' accuracies would be useful in understanding the performance of different reward models. Below, we present the results of two evaluation metrics: we first show the success rate of choosing a higher-quality response as compared with the population median score of responses (higher better) in Figure 16 In Figure 17 we present the performance percentile of the reward model selected responses in the population (higher is better). We find different imple- mentations of the classification-bass reward models perform similarly and are in general better than using the BT models. And those discoveries are well aligned with the BoN evaluations.
Harmless
Helpful
Rowward Model Prediction Accuracy
Dewward Model Prediction Accuracy
ST-ALP
CLT4GS
LL0043 56
Ganetate
Base Models
Baso Models
Figure 16: Success rate of choosing a higher-scored response than the population median the Harmless (left) and Helpful (right) datasets. The results are from seeds.
Harmless
Heloful
STHLLP C1P-LEM
BoN Nefommance Bercentile
BoN Rerforancc Percentile
0.3
CLT-GG
Gerezate
Uu0043 db
Garorate
Base Models
Base Models
Figure 17: BoN performance percentile on the Harmless (left) and Helpful (right) datasets. The results are from 5 seeds.
38


Reviews:
[{"review_id": "pBa0FohXq0", "reviewer": "Reviewer_Ce8Z", "summary": "This paper investigates the theoretical validity of employing the BT model in the reward modeling process. It also explores alternative reward modeling approaches and compares them with the BT model. Additionally, this work tests the feasibility and effectiveness of using a cross-prompt comparison method for reward modeling, thereby moving beyond the current reward data paradigm of utilizing the same prompt with different responses.", "questions": "1. In the ablation study of Annotation Error Rate, CLF method almost not decrease as Annotation Error Rate arises. This is not common for a  classification model. Can you give more  explanation of this unexpected robustness to annotation errors? And could you also discuss potential implications of this finding for practical applications?\n2. Have you considered comparing your methods on existing benchmarks? And how the results in paper might compare to or complement existing benchmark results?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "As BT model has been widely used as a  default paradigm in reward modeling, there are few work disscuss the validity for this pattern. This work gives a detailed theoretical analysis of using BT model in reward modeling and offers other alternative methods for reward modeling and  conducts rich ablation studies on them. The theoretical analysis reveals some very basic logics of reward modeling and BT model, which offers a deep explanation of how reward modeling and BT model works.", "weaknesses": "1. As illustrated in paper, the label of classifation is simply to  assign (1, 0) to the prefered answer. It is seemingly to be a very rough way for building classifation labels since there are pairs may both be, we say like, 'helpful'. One is labeled as 'chosen' in raw dataset just becase it is more helpful than the rejected one. But it does not mean the rejected one should be  classified in to 'not helpful'.  For example, chosen answer may get a 10/10 score in helpful, and rejected answer may get 9/10.  Can you disscuss this limitation and its potential impact on their results? And have you ever considered alternative labeling schemes that might better capture nuanced differences between responses?\n2. The  experiment model size only ranges from 2B to 8B. It remains unknown weather CLF method still more effective when applied to larger models. Do you have any hypotheses about how these results would be if scale to larger models? Can you discuss this limitation and propose how future work could address it?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zvXd8l0qW9", "reviewer": "Reviewer_wGfE", "summary": "This paper investigates the theoretical and practical foundations of the BT model in LLM alignment, examining its common use in transforming pairwise comparisons into reward scores despite sparse data. The authors establish a convergence rate for BT reward models based on neural network embeddings, ensuring its theoretical validity, but argue that BT is not essential for downstream optimization, as any monotonic transformation of true reward suffices. They propose an order-consistent alternative using a binary classifier framework that simplifies reward modeling. Furthermore, the authors propose to utilize cross-prompt comparisons for improving the reward modeling performance.", "questions": "- **Q1:** The results showing that $\\\\mathcal{L}_{\\\\textrm{clf}}$ outperforms the BT model in Figure 1 are somewhat counterintuitive to me, as the classification objective learns marginal probability while the BT model learns joint probability (line 305). The classification objective seems to use less information than the BT model, as the correspondence between $i$ and $j$ is missing. Could you elaborate on why the classification objective not only works but also performs better than the BT model?\n\n- **Q2:** The idea of utilizing cross-prompt comparisons is interesting and shows significant improvements in experiments. Could you discuss any restrictions on using cross-prompt comparisons and describe scenarios where cross-prompt comparisons are particularly effective?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Provides a thorough analysis of the BT model's application in LLM alignment, identifying unique challenges and offering theoretical justification, including the introduction of asymptotic theory for BT regression in preference-based reward modeling.  \n2. Proposes `order consistency` as a core principle in reward modeling, leading to an alternative to the BT model. Additionally, the use of cross-prompt comparisons enhances training by expanding available comparisons, which further contributing to performance improvements.  \n3. The paper is presented in a clear and coherent manner. The authors introduce the purpose behind each step before detailing the results, providing a helpful summary of expected outcomes. Despite the dense content, the writing remains accessible and easy to follow.\n4. The experiment is thoughtfully designed to validate the proposed methods and offer valuable insights to readers.", "weaknesses": "I have no major concerns. I have only one minor suggestion: Figures 2 and 3 should include dataset names in each row for added clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bFZeCOv5nm", "reviewer": "Reviewer_omr2", "summary": "The paper addresses the prevalent use of the Bradley-Terry (BT) model for reward modeling in LLM alignment. It introduces an asymptotic theory for BT regression in LLM reward modeling, setting the first risk bound for BT model in LLM alignment, and propose order consistency as the central goal of reward modeling, which can inform both the BT model and an alternative classification-based approach. This approach is shown to be more flexible and efficient than the BT model, especially when dealing with varying levels of annotation quality and quantity. The paper supports its claims with extensive testing across different LLMs and annotation strategies. \n\nThe proposed theoretical framework includes: \n- Argues that a reward model does not need to provide exact probabilities of preference; instead, it must maintain the correct order of preferences between prompt-response pairs.\n- Establishes a bound on the prediction error of a reward model based on MLPs. This bound ensures that, under specific conditions, the model will predict preference probabilities with a controlled error margin.\n- Shows that the predicted rewards will be close to the true rewards as long as the preference probabilities are well approximated.\n- Provides a theoretical guarantee that the model will remain order-consistent even when dealing with noisy annotations.", "questions": "- In RLHF, preferences may be non-transitive—such that if A > B and B > C, it doesn’t always imply A > C. Could this be consider as annotation errors? Or should this also be invested (as it might also reflect genuine complexities in human judgments, influenced by context or subjective factors)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper provides a theoretical framework to justify the use of order consistency. \n- The authors conduct extensive experiments with diverse setups, support their claims about the classification-based model’s advantages over BT.\n- The paper is well written.", "weaknesses": "- The paper evaluates reward model performance primarily through BoN sampling improvements. While BoN results provide insights into relative ranking, they do not directly reflect the reward model’s prediction accuracy or error, which is central to the theoretical claims in Theorem 2.2. Including accuracy or error metrics in the evaluation could provide a more comprehensive validation of the theoretical bounds on prediction error, demonstrating that the model’s predictions align with the theoretical guarantees.\n- In RL training (e.g., using PPO), the RM’s calibration is crucial: a well-calibrated RM should assign progressively higher rewards to better trajectories. Theorem 2.2 focuses on minimizing prediction error, which may not ensure adequate calibration for RL. To fully evaluate the proposed approach, an analysis of the reward model’s calibration or additional experiments using PPO (or other algorithms) would help determine if the model can effectively support RL tasks where calibration quality directly impacts learning stability and performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)