Reward Modeling with Ordinal Feedback: Wisdom of the Crowd
Authors: Shang Liu
Yu Pan
Guanting Chen
Xiaocheng Li
Venue: ICML 2025 poster (2025)
Keywords: reward modeling
ordinal feedback
human preference learning
Primary area: deep_learning->large_language_models
TL;DR: How to deal with different levels of preferences in RLHF: use quantitative labels.

Abstract: The canonical setup of learning a reward model (RM) from human preferences with binary feedback discards potentially useful samples (such as "tied" between the two responses) and loses fine-grained information  (such as "slightly better'"). This paper proposes a framework for learning RMs under *ordinal feedback*, generalizing the binary feedback to arbitrary granularity. We first identify a marginal unbiasedness condition, which generalizes the existing assumption of the binary feedback. The condition is validated via the sociological concept called "wisdom of the crowd". Under this condition, we develop a natural probability model and prove the benefits of fine-grained feedback in terms of reducing the Rademacher complexity, which may be of independent interest to another problem: the bias-variance trade-off in knowledge distillation. The framework also sheds light on designing guidelines for human annotators. Our numerical experiments validate that: (1) fine-grained feedback leads to better RM learning for both in- and out-of-distribution settings; (2) incorporating a certain proportion of tied samples boosts RM learning.

Reward Modeling with Ordinal Feedback: Wisdom of the Crowd
Shang Liu" Yu Pan *2
Guanting Chen 3 Xiaocheng Li'
Abstract
However, there is inconsistency between current ways of collecting human preference data and the training of reward models. For example, the Llama team collects fine- grained human feedback: the annotations are made to 4 levels named "significantly better", "better", "slightly bet- ter", and "marginally better" (Llama Team, 2024), while the post-training of Llama treats "significantly better" and "better" as the same and discard all the others. Such a process wastes the potentially useful samples that cost the human an- notators additional time and discards the useful information hidden in the preference level.
The canonical setup of learning a reward model (RM) from human preferences with binary feed- back discards potentially useful samples (such as "tied" between the two responses) and loses fine- grained information (such as "slightly better"). This paper proposes a framework for learning RMs under ordinal feedback, generalizing the bi- nary feedback to arbitrary granularity. We first identify a marginal unbiasedness condition, which generalizes the existing assumption of the binary feedback. The condition is validated via the so- ciological concept called "wisdom of the crowd". Under this condition, we develop a natural proba- bility model and prove the benefits of fine-grained feedback in terms of reducing the Rademacher complexity. which may be of independent interest to another problem: the bias- trade-off in knowledge distillation. The framework also sheds light on designing guidelines for human annota- tors. Our numerical experiments validate that: (1) fine-grained feedback leads better RM learning for both in- and incorporating certain proportion of tied samples boosts RM learning.
In this paper, we study the problem of reward modeling un- der ordinal feedback. We relate the annotator's preference feedback with the probability that one response better than the other on population level. We introduce a marginal unbiasedness condition that validates the probability setup of the ordinal feedback system. The assumption is justified in two ways: (1) its restricted version in the binary feedback case is already widely adopted; (2) the assumption corre- sponds to the "wisdom of the crowd" concept in sociology. Under the assumption, we build a probability model of or- dinal feedback. We also propose learning objective as a generalized version of the binary feedback by, for example, replacing the binary-valued empirical probability with more fine -ineggrained values in the cross-entropy loss. Theoretically, we establish the advantage of ordinal feedback, which draws an interesting connection with the literature on soft labeling and knowledge distillation.
settings; (2)
1. Introduction
Reinforcement learning from human feedback (RLHF) (Christiano et al., 2017; Ziegler et al., 2019; Askell et al., 2021; Ouyang et al., 2022) is vital to aligning large lan- guage models (L.LMs) with human preferences. The RLHF involves either explicitly training a reward model (RM) from human preferences data (Ouyang et al., 2022) or im- plicitly using the LLM itself one (Rafailov al., 2024).
Our paper is organized follows:
In Section 2, we model the feedback by the probability that one response is better than the other on population level.
The binary feedback (Z = {0, 1}) is extended to the general ordinal feedback (Z = { for 0 S 21 <.... < 2m S 1), providing a way to transform the qualitative label into quantitative ones.
'qqual contribution 'Imperial College Business School, Im- perial College London, UK 'The University of Sydney Business School, The University of Sydney, Australia 'Department of Statis- tics and Operations Research, UNC Chapel Hill, USA. Corre- spondence to: Guanting Chen <guanting wwuneuuucccsss Xiaocheng Li <xiaochhng @ imperial.ac.uk.
In Section 3, we build up the probability model of ordinal feedback. We present the assumption (Assumption 3.1) that the annotators are labeling the human preference data in a marginally unbiased sense to the population Such an assumption (which we call "wisdom of the crowd") is only generalization of the Bernoulli feedback. this light, we suggest revising the annotation guideline by pro- viding a direct quantitative description of the qualitative
Zgganntng@unee
oracle.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Reward Modeling with Ordinal Feedback
opinions.
Definition 2.1 (Ordinal Feedback). Suppose the feedback Z
takes values in 2 == {21,..., _ where 0 S 21 < Rm S I, we call Z an ordinal feedback and Z the ordinal feedback set.
In Section 4, we prove the statistical benefits of the ordi-
nal feedback. For example, the Rademacher complexity is reduced if the cross-entrope loss is adopted. The result is proved via a special coupling argument that we call hierar- chical expectation, which also provides a new bias- vars-nariance trade-off in knowledge distillation and soft labeling.
The binary feedback is a special case of the ordinal feed- back by letting m 2, 21 = 0, and 22 = 1. The motivation to introduce the ordinal feedback definition is to capture the richer options in practice, where the annotator is al- lowed choose from Ziex = (better/worse than, same as} or Zient = (better/worse than, slightly better/worse) de- scribe the preference between responses y/1 and y/. We defer to the next section the questions of how to match Zes with Z.
In Section 5, we conduct two numerical experiments. The first experiment sets up four different ordinal feedback sys- tems (oracle, 5-level, 3-1evel, and binary) and validates the theoretical findings that fine-grained ordinal feedback achieves higher accuracies in both in- and dittof-distribution settings. The second experiment mixes the training data with different proportions of tied and untied samples. With the same number of training samples, we find out that a certain level of tied samples boosts RM learning.
For now, suppose we have the dataset Dz taking values in 2 translated from Z We propose to learn the reward model by minimizing the following cross-entropy loss (for other loss functions, see Appendix C.2)
We defer more related works to Appendix A.
2. Problem Setup
min 8
Σ
Zi log (a (ro(zs, V1.1) - ro (2i, 11.2)))
1=1
Consider the task of reward modeling based on the pair- wise preference data. Each sample consists of tuple (z, V1, 2), where X' denotes prompt, y1, 3/2 E y are two candidate responses to the prompt and Z is a random variable (taking values in 2) that denotes the feed- back (by either human annotators or advanced AI models) indicating the preference between y1 and y2. The feedback Z is proxy of the probability that 1/1 is better than y2 for the prompt x. denoted by IP(11 > 1/2 lr).
- (1- - Z,) log (a (re(z,, - ne (3 Vi vi,1))),
(2)
where (-) is the sigmoid function such that (x)
exp(x)/(1+ (1+ exp(x)). exp(())
When the feedback is binary Z, E {0, 1}, the above objec- tive function reduces to exactly how people learn the reward models under the Bradley-Tene ize the binary feedback to avoid: (1) discarding potentially useful samples (for example, neglecting "same as"); or (2) losing subtle information by merging different options (for example, treating "better" and "significantly better" as the same).
assumption. We general-
Reward modeling is the learning of reward function To (z. Xxx + R with 0 E θ from an annotated dataset D2= = {(T), V6,1. 34,2 H= and approximates the prefer- ence accordingly. For example, the prevalent Bradley-Terry model (Bradley& Terry, 1952) says:
This generalization raises three questions: (1) How to trans- late between qualitative Ziett and quantitative Z? (2) How to relate the quantities in Z to P (v1 > y2 v²/z;)? (3) How does this setup differ from the canonical binary feedback? The first two questions are addressed in Section 3, and the last is addressed theoretically in Section 4 and numerically in Section 5.
exp (ro(z, y1))
Po(y > y21z) =
exp (ro(z, exp (ro y2)) (r)(
where the probabilities are approximated by the softmax reward values.
Binary feedback: In the canonical setup ((ai et al., 2022: Ouyang et al., 2022) among others), the feedback Z takes binary values, i.e., 2 = {0, 1}. Here, one assumes 74 is Bernoulli random variable such that
3. Probability Model of Ordinal Feedback
For a population of human preferences, we first define the oracle feedback as
P(Z, = 1) = 1 PiZ = 0) = IP (11,1 21,2
(1)
Zoracte (z, V1. 32) = P (31 > y21z).
This assumption is the backbone of the training of many reward models.
This is the preference model that one aims to learn, regard- less of whatever model is assumed (for example, BT model or others). Here we should think of a human annotator as a random draw from the population.
Ordinal feedback: In this paper, we consider the setting of ordinal feedback, which gives a richer feedback structure than the binary feedback above and is defined as follows.
2

---

Reward Modeling with Ordinal Feedback
better than same as worse than
(0
Assumption 3.1 (Ordinal feedback probability model wis- dom of the crowd). We assume the ordinal feedback Z defined in Definition 2.1 satisfies
Ziext
+ 2 =
0.5
For a more fine-grained 5-level feedback setting, it is less clear how one can convert as follows.
E 2 (xx, y1, [2)] = Zoracle (25, 11, 3/2) for any (I, y2) ) E X' x y²
better than slightly better same as slightly worse worse than
As shown later, this assumption alone is sufficient to define the probability model of the ordinal feedback setting and validate the learning of the reward model. To interpret the assumption, we note that it is not stricter than the existing assumption (I) that people impose for the binary feedback model. Under the binary feedback model where Z takes values in Z {0, 1}, Assumption 3.1 is equivalent to (1),
7%
Ziext
0.5
1-
In this light, our ordinal feedback model and Assumption 3.1 provide the following insights in guiding the annotations. We suggest that rather than providing vague wording of "better than" or "sightly better", one can write the following in the guidance to the human annotators:
To see another example, consider the set = {0, (0, } where the labels 0 and 1 denote "better" and "worse" respec- tively, and the label 0.5 denotes "same as". The assumption requires that with Z taking values in this new 2 with an ad- ditional 0.5 option, its expectation matches the oracle value ooralee on the sample (at, V1, 1/2.. Under the general ordinal feedback setting, human annotators label their preferences on different scales, i.e.,
Example annotation guideline. The label "slightly better" represents that 75% of the population think re- sponse y1 is better than response y/2 The label "slightly worse" represents that 25% of the population think response y1 is better than response 3/-
2 = 10, (0, 0.5, 1}, or {0, 0.25, 0.5, 0.75, 1}.
Such additional guidance endows the narrative labels in Z with numerical meaning. The corresponding numerical values can be directly used in the learning objective (2).
The assumption says that the change of granularity does not introduce bias on the population level.
Sociological interpretation: We name the assumption by "wisdom of the crowd";; the concept was first coined by the article Vox Populi (Galton, 1907) for a social experiment under the title "the voice of the people". The social exper- iment is about weight-judgee judging competition conducted in England for random people to guess the weight of an ox. The average of all 787 guesses was 1,197 pounds, while the actual weight was 1,198 pounds, as shown in Figure 1. Each individual's guess can be far off the target, yet the population average tends to be very accurate. For the context of preference annotation, Assumption 3.1 and the current practice of human annotation exercise the wisdom of the crowd in two folds: First, each individual annotator has access to the population preference Zuracles but their annotation can be viewed as an unbiased random realization of Zoracle. Second, such unbiasedness does not change if a difference annotation scale Z (feedback set) is used.
3.2. Existence of feedback probability model
In this subsection, we detour from the discussions of reward modeling and show the existence of probability models that satisfy Assumption 3.1, which ensures the assumption is well-defined. The following theorem states that for any ordi- nal feedback set Z and any oracle model Zoracle, there exists a probability model satisfying Assumption 3.1; furthermore, the model is uniquely determined by the oracle Zeracle (up to convex combinations).
Theorem 3.2. For any ordinal feedback set Z {Z1,...; bm} and any oracle model zoracle (I. 21, 12). if Zoracte (I. y1, v1. E [zy,, for some j,k E [ml) then one can set the marginal probability measure (,,,(2))= ())= == P(Z =1(2, 91, 32)) to be
(Ex Zoructe) / (2k 22))
if Ejo
3.1. Implications on annotation guidance
(5,((2) =
(Zonacle 2z) =z),
if = =ks otherwise.
0,
In practice, annotators label in the set Ziex (c.g. (better, same as, worse)), and this results in a gap towards the set Z { km used in the learning of the reward model (2). For the binary or 3-level feedback setting, the following conversion is natural and reflects the thinking process of the annotator:
The ordinal feedback Z under Pij.e fulfills Assumption 3. (where the dependence (x, 21, W2) is omitted when there is no confusion).
On the other hand, for any ordinal feedback Z with marginal probability measure u(z) == P(z = 2|(2, y1. (2)) satis- fying Assumption 3.1, there must exist non-negative real
Z = (better than, worse than) + 2 (0,1),
3

---

Reward Modeling with Ordinal Feedback
What the weight of the ox?
What is the oracle of the preference pair?
Response Windows intuitive and fast.. Response 2: Windows is cluttered and slow...
1083
1249
Inaccurate Individuals
True: 1198
1197
x
0.64
True (aracle): 0.60
Estinated 0.62
0
0.5
Predictive Crowd
Figure 1. Wisdom of the crowd. Left: Each individual guess can be far off the target for an ox-weight-guessing social experiment, but the average tends to be very accurate. Each human annotator has no access to the population oracle preference model Reeicle, but their annotation constitutes an unbiased realization of Zoraclo-
numbers Σj. O'jk = where the summattin is made for all (j, k) pairs such that Zoracle E [2j, 2x).
1 such that Jt = Ej Orj Piak,
dition of the loss function that is satisfied by the cross- entropy loss (2) (and generalized hinge loss in Appendix B.2, for example).
The theorem says that all such distributions should be (con- vex combinations of) some two-point distributions, where the weights are assigned to interpolate the linear system
Ez [>]](=] (B( for any E [0, 1].
(3)
Introducing the affinity condition (3) enables us to com- pare two ordinal feedback systems (say, Z and Z') if their conditional expectation is the same under Assumption 3.1. Specifically, if l satisfies (3), then the population loss of Z and Z' are the same for arbitrary same h (can be taken as the reward function r, for example),
(g)(zz)) 24 + (z)(2x)) 2k = Zoracle: (1,((z)) + (z.((zz) =1.
For example, consider reward model with ties where (0, 0.5, If the oracle feedback is, say, 0.8, then we can construct feedback Z with probability masses (00.5) 0.4 and s41) = 0.6 such that the unbiased assumption is fulfilled. In this way, if we are (in an ideal world) given an oracle model Zoracle, we can generate different unbiased ordinal feedback problem models accordingly, which is applied in our numerical experiments (Section 5).
Ez,g.2 (ZZ h(x, V1. Mel) Ez.g.22' [((z), h(x, ((x. 82))
However, (4) only states that two population losses are the same under the same function h; what if we get two different learned reward models and h from and Z' correspond- ingly? In the following, we shall see that from the perspec- tive of the generalization bound (specifically, Rademacher complexity), more fine- fine-grained feedback system induces smaller population loss.
4. Statistical Benefits of Ordinal Feedback
In this section, we provide theoretical explanation of the benefits of ordinal feedback. The theory not only captures the benefits of the ordinal feedback model but also provides insights into the technique of saft labeling in knowledge dis- tillation (Ba & Caruana, 2014: Hinton, 2015; Müller al., 2019; Phuong & Lampert, 2019; Yuan et al., 2020; Zhou et al., 2021), which can be of independent interest. In short, the theoretical result says that any feedback model satisfy- ing Assumption 3.1 reduces the Rademacher complexity compared to the canonical binary feedback model.
4.1. Finite-sample benefits
We first introduce the concepts of coupling and hierarchical expectation.
Definition 4.1 (Coupling). For any two random variables g and E if there exist two random variables and Ç" over one probability space such that Ç has the same distribution as { and Ç' the same as E', we call them coupling of É and g'. Definition 4.2 (Hierarchical Expectation). For any two or- dinal feedback systems Z and Z' taking values in 2 and 2'
Before we proceed, we introduce the following affinity con-
4

---

Reward Modeling with Ordinal Feedback
over the same probability space (Q, F,,,PP) P), we say that Z is hierarchical expectation of Z' if there exists combina- tion of random variables (W, W") over probability space (2, P) such that: (a) (W, W") forms a coupling between 7 and 2 and (b) W = [[ holds almost surely.
where e i's are independent random variables all taking val- ues in {+1, -1) with equal chances. By assuming that (Ii Vi.1, 36.2; Z,''s are i..d. and taking expectations over the entire distribution P, we have the Rademacher complex- ity as
The concept of hicrarchical expectation defines the relative granularity of the feedback system. In general, if Z is a hierarchical expectation of Z', then we say Z is more fine- grained than Z' since Z allows annotators to give more subtle responses.
(n H) == ED Raddy, (e 70).
Rademacher complexity popular tool for describing generalizing behavior. The following theorem says that more fine-graing linegrained feedback system leads to a smaller (which is better) Rademacher complexity for any hypothesis class H. where setting H = fro @ E )} yields results in RM learning. We include how the Rademacher complexity consequently affects the generalization bound in Appendix B. 10 for completeness.
Proposition 4.3 (Existence of Hierarchical Expectation). For any two ordinal feedback systems Z and Z' taking val- ues in 2 and Z' over the same probability space F,, PP,, suppose the marginal distribution of Z is of measure μ = Eziez axSz, and that of Z' is p' = = o2zzz where 5()) is the Dirac delta distribution. If there exist real num- bers 8y E [0, 1] such that
Theorem 4.6. Suppose the loss function E satisfies the affin- ity to feedback condition (3) and H. is a hypothesis class of real-valued functions over X x y². For any two ordinal feedback systems Z and Z' taking values in Z and Z' such that Z is a hierarchical expectation of Z', we have
(a) 2j is a convex combination of 2 's with coefficients 3z,'s. 's. That is, and Em.LEED 3r 2j for
En.AEE 33
any 2j E 2;
Radz,n HH) S Radz', H).
(b) Es B5,00 = o'2 for any 2 E 2';
As a consequence, the following corollary states that any feedback system that satisfies Assumption 3.1 can be viewed as in the middle of the binary feedback system and the ideal oracle feedback system Zoracle. It is impossible to access the population preference model Zeracle through human annota- tors in practice, but any ordinal feedback system provides more fine-grained information and leads us towards Serucler
Then Z must be a hierarchical expectation of LZ, On the other hand, if Z is hierarchical expectation of Z' accord- ing to coupling (W, W') on Fi,,,PP) then there must exist real numbers Bj. E [0, 1] satissyyig the above requirements by setting 85 P(W" </- 2,)-
The following corollary gives some concrete examples (per- haps the most concerning ones).
Corollary 4.7 (Ordinal feedback better than binary). Sup- pose Z is an ordinal feedback taking values in Z and Z' is binary feedback in Z' {0, 1} over the same probabil- ity space. If they both satisfy Assumption 3.1 and the loss function E satissies condition (3), then
Corollary 4.4. Suppose Z and Z' are two ordinal feedback sets satisfying Assumption& 3.1. Then Z is always a hier- archical expectation of Z' if any one of the following two conditions holds:
Rad Rad a.d (lo H) < Raddzn H) S Radz H).
(a) Z is the oracle model such that Z Zoraclei
(b) Z' is the binary feedback such that Z' = {0, 1}.
The above discussions show that a more fine-grained feed- back system can help reduce learning difficulties. The problem of how large the benefit is has been discussed in Appendix 1. In short, the reduction of Radmacher complexity results from Jensen's inequality gap; the gap is approximately the same order as the reduction of variance.
With the above definitions, we are ready to characterize the finite sample properties of different ordinal feedback systems using Rademacher complexity.
Definition 4.5 (Rademacher Complexity). Let Dz ((r,,
(111, 11,1, 24 7 be dataset with Z taking values in Z. Consider a hypothesis class H of real-valued functions over X' x 2d2 and a loss function& f. Then, the empirical Rademacher complexity is defined as
4.2. Implications on soft labeling
As noted earlier, the results developed above also have im- plications for soft labeling/knowledge distillation. We will show how the analysis can be applied to the context of soft labeling and induce a novel variance trade-off for knowledge distillation. For general k-nary classification
Radpz(Eof)) = 1 EE
Σ E (o, M(T (.1.
sup NEH im1
5

---

Reward Modeling with Ordinal Feedback
problems, the standard feedback (labeling of the target vari- able) is k-dimensional one-hot vector. However, these all- zero-but-one labels might make the model overfit. Knowl- edge distillation (Hinton, 2015) is a well-known technique to regularize the model from fitting the noise. The original data is used to train a teacher model whose predictions are named soft labels. Then, a student model is trained to mimic the predictions of the teacher model, that is, minimizing the training loss against the soft labels generated by the teacher model rather than the original ones.
the following assumption that the marginal output of the teacher model is unbiased:
Assumption 4.8 (Soft-labeling version of Assumption 3.1), We assume that the teacher model is marginally unbiased. That is,
Ey BTlyT(a [a] = (ara
Here T is the teacher model trained from the original labels y j.s. The randomness of T comes from the randomness of y's I's and the training procedure (e.g. random seeds).
Existing theoretical works (Phuong & Lampert, 2019; Zhou et al., 2021; Pareek et al., 2024) are developed to under- stand the benefits of knowledge distillation and soft label- ing. Our theoretical perspective in the preceding subsection provides new perspective on that problem: the trained teacher model could be viewed as one (possibly biased) oracle feedback, and learning from the oracle eases the overfitting via reducing the labeling variance (and hence smaller Rademacher complexity). Consider the following four learning paradigms:
In general, the learned teacher model outputs yT's are bi- ased estimations of yorrale s. and the biases are towards the original labels y's (in the extreme case where T interpolates all the labels, we have yT = y). We note that the assump- tion only requires an unbiasedness in a marginal sense, that those biases cancel out each other in the marginal sense. We have the following result (analogous to Theorem 4.6).
Theorem 4.9. Under Assumption 4.8 and with the cross- entropy loss, we have
Reduced variance == Rady (lo H) RRdyy (6 H) 2 0.
(a) Oracle(ideal): learn from oracle labeled samples (de-
noted as (z, Moracte)ss 's in this subsection). Moracie Ely|zr]
In this light, the notion of hierarchical expectation and the re- duced Rademacher complexity in Theorem 4.9 render a new bias-variance tradeoff' for the knowledge distillation meth- ods. Compared to that of Zhou et al. (2021), our approach theoretically shows that the variance is always reduced by introducing the soft labels, while Zhou et al. (2021) makes the reduction an assumption and verifies it empirically.
(b) Original: learn from the original samples (denoted (z, y)'s 's in this subsection), where y Voracle is random one-hot vector.
(c) Knowledge distillation: learn from the teacher model T's output (denoted as (I, y in this subsection). The la- bels yT's are random vectors where the randomness comes from the teacher model T.
(d) Sampling from teacher: learn from the teacher model, but not directly from yT: instead, we use sampled label
5. Numerical Experiments
yT V/T
We perform numerical experiments to answer two questions: (1) How do different granularities of the feedback model affect the learning of the reward model? (2) Does the inclu- sion of these ordinal feedback data with the objective (2) benefit the learning of the reward model?
What is the "bias-variance" tradeoff in the learning paradigm (c) compared to (b)? First, Voracle shares the same conditional expectation with y, and so do J/T and Y/T, where their pop- ulation cross-entropy losses are the same due to (4). The "bias" comes from yr as an imperfect estimation of foracle. while training according to (c) or (d) introduces an addi- tional loss in the original population loss. That is the "bias" term
5.1. Experiment Settings
Datasets. In the following numerical experiments, we lever- age the Bertoo---eeaar-PPefeeeee -00 dataset (Liu et al., 2024a) as our base training dataset. We perform mul- tiple runs and report the average performance (along with the confidence intervals); for each run, we randomly sample a 1 1024-sized subset as the hold-out evaluation dataset. In addition, we use the RewardBench dataset (Lambert et al., 2024) for the out-of-distributio evaluation task to compre- hensively assess the performance of different trained models. More details can be found in Appendix C.3.'
Bias = ET Eaws (('l,h")(z)))
h+)(())]] -Ea. [el' h" (+))]
2 0,
where we set y to be i.i.d. as to prevent the dependence of T on y, and
h7 = arg min Ex, (((T,M(x)))
h" = arg min E.,y h(z)), bER
Base Models. Our base models for the following experi- ments include
which are the corresponding hypotheses minimizing popu- lation losses.
(Llama Team, 2024), gemma-2-28 2-2b-i (Gemma Team, 2024), and qwen2.5-1.5b. instruct (Yang et al., 2024). All models are trained under
To explicitly show that the "variance" is reduced, we make
6

---

Reward Modeling with Ordinal Feedback
under hull-parameter fine-tuning. For the training parame- ters and other details, we refer Appendix C.4.
Table 1 (and Figure 2 in Appendix C.I) summarizes the experiment results, which are aligned with the findings in the previous sections. Three take-away messages are: First, more fine-grained feedback structure leads to better reward learning for both in-distribution (ID) and out-of- (OOD) performance. Second, though we do not have access to the oracle model in practice, the 5-level feedback system provides a good proxy for that. Third, the learning objective (2), generalization of the canonical cross-entropy loss for binary feedback, is effective one to handle the ordinal feedback data.
Ordinal Feedback. The original Seyoork-Reward Preference-80K-vv0) dataset only contains a binary feed- back for each prompt I and a response pair y1 and y2- To generate feedback labels with different levels of granu- larities, we adopt a well-trained reward model, Skywork- -aatrd---mmm-2-277----a
(Liu et al., 2024a), as the or- acle scoring model Foracle X' x y - R in this case. We chose this model because (1) it was exclusively trained on the to-be-labeled base training dataset, hence there is hardly& a risk of ditto--distribution mislabeling; (2) the model ranks first on the RewardBench online leaderboard up to the time of this paper, making its output oracle scores more reliable. Accordingly, the induced oracle model be- ing Soracie (zt y1, 122 = a (Frorace (z, y1) F'rrute (zz) (222) where T is temperature parameter and o (-) denotes the sigmoid function.
5.3. Tied data benefits
Now we restrict our attention to the 3-1evel feedback setting and investigate the effect of the proportion of the tied data (samples with labels of y1 "same as" y2). Specifically, we limit the training samples to 32,768 and consider 5 different proportions of the tied data:
We consider the following four types of feedback systems:
0%--ied: All the data samples are binary-labeled.
Oracle: Soracle is directly used as the feedback label and Zoracle C [0, 1].
25%, 50%, 75% of the data samples are tied.
100%-tied: All the data samples are tied.
Binary: Zoinary ~ Bernoulli( (eerace)) and Z2 = {0, 1}.
More details of data generation are deferred C.3.
3-1evel: the label is sampled the process in Theorem 3.2 considering only the smallest interval containing
Table (and Figure 3 in Appendix C.1) summarizes the ex- periment results. We make the following observations. First, the 100%-tied setting fails in that it results in significantly worse performance than the other settings. This is natural as it leads to a reward collapse, as also observed in other semi- supervised learning algorithms; to see this, if we are given only the tied data, one way to learn the reward model is to have all the rewards equal to constant. Second, mixing a proportion of the tied data and using the learning objective function (2) leads to a better performance than the case of 0%-tied data. One subtle point here is that, in practice, if we do not employ the learning objective (2) and simply drop the tied samples, this will result in a smaller sample size for learning the reward model, and an even worse performance than the 0%-tied setting here.
ooracle and Z3 {0, 0.5, 17.
5-level: the label is sampled as the process in Theorem 3.2 considering only the smallest interval containing Zuracle and 2;- (0, 0.2, 0.5, 0.8,
We provide a label histogram in Appendix C.3 and adopt the cross-entropy objective function (2) to train the model.
5.2. Fine-grained feedback benefits
As discussed earlier, a more fine-grained feedback system should lead to better reward learning. For the four feedback models listed above, they should have the following orders in terms of performance:
Discussions other ways to deal with tied data. Re- cent works (Chen et al., 2024; Liu et al., 2024b) have no- ticed the importance of incorporating tied samples and em- ployed the Rao-Kupper model (Rao & Kupper, 1967), or the Bradley-Terry Terry model with Ties (BTT) abbreviated by Liu et al. (2024b), for preference modeling and to explore the benefits of leveraging ties. The BTT model represents pref- erence probabilities as follows: P(y) > y2 x)) eittez'
Oracle " 2 5-leeel "2 " 3-level " "2 Binary
where " > represents an advantage in model performance.
Here we perform numerical experiments verify such in- tuitions and for each combination of the reward model and feedback system, we conduct 5 independent training runs and report the average results. For the setting of learning with oracle feedback, we set Z, = P (yI,1 > (12)11)) = Soracte (I;, 1 2i.2) in the learning objective (2). For more fine-gre aine-griiedd ordinal feedback, we sample the feedback ac- cording to Section 5.1.
P/y ~ 1/2 x) = Ter (32-1) are, respectively, abbreviations for exp (ro(2, v1)) and (ro(x, y1 ~ y/2 denotes a tie between y1 and y/2
+ea where el and ez
7

---

Reward Modeling with Ordinal Feedback
Table Model convergence statistics under different feedback models. ID stands for in-distribution. OOD stands for out-of-distribution. The ID and OOD datasets are in Section 5.1. The oracle CE loss is computed by adopting Eorck as Z, regardless of the feedback type.
Oracle CE
Loss
ID Accuracy
OOD Accuracy
Model
Feedback
Mean
Std
Mean
Std
Mean
Std
Oracle
0.5711
0.0020
0.9382
0.0037
0.8193
0.0016
5-level
0.5714
0.0019
0,9372
0.0040
0.8100
0,0013
Llama
3-level
0.5715
0.0021
0,9359
0.0044
0.8016
0.0034
Binary
0.5736
0.0024
0.9329
0.0044
0.7667
0.0012
Oracle
0.5698
0.0018
0.9401
0.0031
0.8697
0.0072
5-level
0,5704
0.0016
0.9371
0.0082
0.8584
0.0107
Gemma
3-level
0.5704
0.0018
0,9381
0.0083
0.8580
0,0016
Binary
0.5709
0.0021
0,9368
0.0074
0.8237
0.0101
Oracle
0.5740
0.0026
0.9411
0.0031
0.8543
0.0074
5-level
0.5746
0.0028
0.9394
0.0019
0.8422
0.0080
Qwen
3-level
0.5749
0.0025
0.9388
0.0039
0.8352
0.0092
Binary 0.5756
0.0026
0.9378
0.0034
0.8232
0.0102
Table 2. Model convergence statistics under different tied data ratios. The evaluation dataset remains fixed across all ratio settings and is directly sampled from original dataset, ensuring distribution closely matches that of whole dataset.
Oracle CE Loss
ID Accuracy
OOD Accuracy
Model
Tied Ratio
Mean
Std
Mean
Std
Mean
Std
0%
1.0421
0.0363
0.9224
0.0080
0.7661
0.0182
25%
0.3327
0.0051
0.9341
0.0173
0.7672
0.0093
Llama
50%
0.4187
0.0043
0.9336
0.0014
0.7545
0.0082
75%
0.5339
0.0052
0.9268
0.0180
0.7749
0.0008
100%
0.6931
0.0017
0.3428
0.0677
0.4424
0.0393
0%
6.4762
0.3392
0.9355
0.0041
0.8319
0.0080
25%
0.6031
0.0019
0.9467
0.0117
0.8487
0.0100
Gemma
50%
0.5775
0.0001
0.9526
0.0075
0.8277
0.0006
75%
0.6122
0.0006
0.9521
0.0069
0.8236
0.0084
100%
0.6931
0,0001
0.4814
0.0055
0.4928
0.0158
0%
4.4457
0.1296
0,9313
0.0010
0.8215
0.0068
25%
0.6110
0.0037
0.9339
0.0060
0.8354
0.0046
Qwen
50%
0.5796
0.0031
0.9336
0.0017
0.8267
0.0032
75%
0.6136
0,0030
0.9228
0.0093
0.8193
0.0092
100%
0.6933
0.0007
0.4342
0.0256
0.4616
0.0356
and À 2 1 is a threshold hyperparameter that controls the likelihood of assigning ties. Notably, when À = the BTT model reduces the canonical BT model.
tional tied probability, which is useful when ties have real- world implications. However, the ultimate goal of the RM training is to provide reference feedback function for the following RLHF step rather than separately predicting each probability. Second, even if the BTT model is suited for the 3-level feedback case, it is not enough for the richer feedback system that has already been adopted by those LLM companies. Generalizing the BTT model to the more
The BTT model may not suit preference learning perfectly as it introduces unnecessary complexity. First, the BTT model introduces an additional hyperparameter À designed for other purposes (for example, predicting a football match). The BTT model introduces a tunable À to output an addi-
8

---

Reward Modeling with Ordinal Feedback
fine-grained case is challenging if the only solution is introduce more hyperparameters.
Impact Statement
This paper contributes to advancing machine learning by improving reward modeling with ordinal feedback. A key societal implication is the potential mitigate bias in Al systems. Traditional binary feedback often oversimplifies human preferences, leading to biased or unrepresentative learning outcomes. By incorporating fine-grained ordinal feedback, our approach captures more nuanced human judg- ments, promoting fairer and more equitable Al decision- making. While our method enhances preference modeling, its effectiveness depends on the diversity and representa- tiveness of annotators, aspect that warrants further study. Beyond this, work may have additional societal impacts, though none require specific emphasis at this time.
We defer more numerical experiments to Appendix C.
6. Conclusion
In this paper, we propose reward modeling with ordinal feed- back as generalization of binary feedback. Such frame- work fully utilizes the potentially useful samples and the fine-grained information discarded by the binary feedback practice. We generalize the assumption of the BT model to the general marginal unbiasedness assumption named by a sociological concept "wisdom of the crowd". Under the assumption, we build a natural probability model for ordinal feedback. We also show that the Rademacher com- plexity is reduced by adopting ordinal feedback. Numerical results validate the theoretical findings. Further experiments imply that mixing some tied preference samples benefits RM learning, which may be worth future exploration. Our results suggest that the annotation guideline should encour- age the quantitative description (for example, 70%) of the qualitative option (for example, "slightly better"). Our theo- retical analysis based on hierarchical expectation may be of independent interest to the field of knowledge distillation, providing novel variance trade-off perspective.
References
Akrour, R., Schoenauer, M., and Sebag, M. Preference- based policy learning, In Machine Learning and Knowl- edge Discovery in Databases: European Conference, ECML PKDD 2011, Athens, Greece, September 5-9, 2011. Proceedings, Part 11, .p.pp. 12-27. Springer, 2011.
Amini, A., Vieira, T., and Cotterell, Direct pref- erence optimization with an offset. arXiv preprint arXh::242.110571 2024.
Askell, A., Bai, Y., Chen, A.. Drain, D., Ganguli, D., Henighan, T., Jones, A., A., Joseph, N., Mann, B., DasSarma, N., et al. A general language assistant laboratory for alignment. arXiv preprint
2021.
Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. International Conference on Artificial Intelligence and Statistics, pp.
PMLR, 2024.
Ba, J. and Caruana, R. Do deep nets really need to be deep? Advances neural information processing systems, 27, 2014.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.05862. 2022.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
Burges, C., Ragno, R., and Le, Q. Learning to rank with non- smooth cost functions. Advances in neural information processing systems, 19, 2006.

---

Reward Modeling with Ordinal Feedback
Chakraborty, S., Qiu, J., Yuan, H., Koppel, A., Huang, E,, Manocha, D., Bedi, A. S., and Wang, M. Maxmin- RLHF: Towards equitable alignment of large language models with diverse human preferences. arXiv preprint arXiv:2402.08925, 2024.
Hinton, G. Distilling the knowledge in neural network.
arXiv preprint arXiv:/503.0253: 2015.
Ji, J.. Qiu, T. Chen, B., Zhang, B., Lou, H., Wang, K., Duan,
Y., He, Z., Zhou, J., Zhang, Z., et al. Ai alignment: comprehensive survey, arXiv preprint arKiv:2310. 19852. 2023.
Chen, J., Yang. G., Lin, W., Mei, J., and Byrne B. On extending direct preference optimization accommodate ties. arXiv preprini arXiv:2409. 17431, 2024.
Kaufmann, T., Weng, P., Bengs, V., and Hüilermeier, E. A survey of reinforcement learning from human feedback. arXiv preprint arXiv:2312. 14925, 2023.
Cheng, W., Fürnkrazz J., Hüllermeier, E., and Park, S.-H. Leveraging preference learning for reinforcement learning. In Machine Learning and Knowledge Discovery in Databases: European Con- ference, ECML.PKDD 1, , Athens, Greece, September 5-9, 2011. Proceedings, Part 11, pp. 312-327. Springer, 2011.
Preference-basee policy iteration:
Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N. A., and Hajishirzi, H. Rewardbench: Evalu- ating reward models for language modeling, 2024.
Liu, C. Y., Zeng, L.,, Liu, J., Yan, R., He, J., Wang, C., Yan, S., Liu, Y., and Zhou, Skywork-reward: Bag of tricks for reward modeling in LLMs. arXiv preprint WrXiv:240.184555 2024a.
Choshen, L., Fox, L., Aizenbud, Z., and Abend, O. On the weaknesses of reinforcement learning for neural machine 2019.
translation. arXiv preprint
Liu, J., Ge, D., and Zhu, R. Reward learning from preference with ties. arXiv preprint arXiv:2410.05328, 2024b.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
Liu, T., Zhao, Y., Joshi, R., Khalman, M., Saleh, M., Liu, P. J., and Liu, J. Statistical rejection pling improves preference optimization. arXiv preprint 2r399
Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G., Liu, Z., and Sun, M. Ultrafeedback: Boosting lan- guage models with high-quality feedback. arXiv preprint arXiv:2310.01377, 2023.
2023.
Liu, T. Qin, Z., Wu, J., Shen, J., Khalman, M., Joshi, R., Zhao Y., Saleh, M., Baumgartner, S., Liu, J., et al. Lipo: Listwise preference optimization through learning- to-rank. arXiv preprint arXiv:2402.01888 2024c.
Davidson, On extending the Bradley-Terry model to accommodate ties in paired comparison experiments. Journal of the American Statistical Association, 65(329): 317-328, 1970.
Llama Team, A. M. The llama 3 herd of models. arXiv preprint arXi: 2024.
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T. Raft: Re- ward ranked finetuning for generative foundation model alignment. arXiv preprint 3XXi:2334400666) 2023a.
Müller, R., Kornblith, S., and Hinton, G. E. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., al. Training language models follow instructions with human feedback. Advances in neural information processing systems, 35:27730-2775 2022.
Dong, Z., Yuan, Y., Hao, J., Ni, F., Mu, Y., Zheng, Y., Hu, Y., Lv, T., Fan, C., and Hu, Z. Aligndiff: Aligning diverse human preferences via behavior-customisabbe diffusion model. arXiv preprint arXiv:2310.02054, 2023b.
Ethayarajh, K., Xu, W. Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01300, 2024.
Pareek, D., Du, S. S., and Oh, S. Understanding the gains from repeated self-distillation. arXiv preprint arXiv:2407.0440 2024.
Galton, F. Vox populi. Nature, 75(1949):450-4551 1907.
Phuong, M. and Lampert, C. Towards understanding knowl- edge distillation. In International conference machine learning, 5142-5151 PMLR, 2019.
URL https::/p/// / /////dd./ /doiooooo org/ 110.010.0000 1000/0075800000
Gemma Team, G. D. Gemma 2: Improving open lan- guage models at a practical size. arXiv preprint arXiv:2408. 00/ 18, 2024.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Er- mon, S., and Finn, C. Direct preference optimization:
10

---

Reward Modeling with Ordinal Feedback
Your language model is secretly reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Zeng, D., Dai, Y., Cheng, P., Wang, L.,, Hu, T., Chen, W., Du, N., and Xu, Z. On diversified preferences of large language model alignment. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 9194- 9210, 2024a.
Rao, P. and Kupper, L. Ties in paired-comparison ex- periments: generalization of the Bradley-Terry model. Journal of the American Statistical Association, 62(317):
Zeng, Y., Liu, G., Ma, W. Yang, N., Zhang, H., and Wang, J. Token-level direct preference optimization. arXiv preprint arXiv:2404. 1999, 2024b.
194-204, 1967.
Schölkopf, B., Tsuda, K., and Vert, J.-P. Kernel methods in computational biology. MIT press, 2004.
Zhao, Y., Joshi, R., Liu, T., Khalman, M., Salch, M., and
Liu, P. J. SLiC-HF: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425, 2023.
Song. F., Yu, B., Li, M., Yu, H., Huang, F., Li, Y., and Wang, H. Preference ranking optimization for human alignment. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38(17), pp. 18990-18998, 2024.
Zhou, H., Song, L.,, Chen, J., Zhou, Y., Wang. G., Yuan, J., and Zhang, Q. Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. arXiv preprint arXiv:2102.006500 2021.
Wang, C., Jiang, Y., Yang, C., Liu, H., and Chen, Y. Be- yond reverse kl: Generalizing direct preference optimiza- tion with diverse divergence constraints. arXiv preprint arXiv:2309. 16240, 2023.
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences, arXiv preprint 2019.
Wang, H., Lin, Y., Xiong, W., Yang, R., Diao, S., Qiu, S., Zhao, H., and Zhang, T. Arithmetic control of LLMs for diverse user preferences: Directional pref- erence alignment with multi-objective rewards. arXiv preprint A00.::2402. 18571, 2024a.
Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang, T. Inter- pretable preferences via multi-objective reward modeling and miixture-of-expeee arXiv preprint arXiv:2406.122 12845, 2024b.
Wang, 7., Dong, Y., Delalleau, O., Zeng, J., Shen, G., Egert, D., Zhang, J. J., Sreedhar, M. N., and Kuchaiev, O. Help- steer2: Open- dataset for training top-tertie perfformnii reward models. arXiv preprint #xXiv2446.006673 2024c.
Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Van Durme, B., Murray, K., and Kim, J. Contrastive preference op- timization: Pushing the boundaries of LLM performance in machine translation. arXiv preprint arXiv:2401.0844 2024.
Yang. A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024.
Yuan, H., Yuan, Z., Tan, C., Wang, W., Huang, S., and Huang, F. Rrhf: Rank responses to align language models with human feedback. Advances Neural Information Processing Systems, 36, 2024.
Yuan, L., Tay, F. E., Li, G., Wang, T., and Feng. J. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3903-3911, 2020.

---

Reward Modeling with Ordinal Feedback
A. Related Works
Reinforcement learning from human feedback (RLHF) originates from the idea of preference- reinforcement learning (Cheng et al., 2011; Akrour et al., 2011). The term RLHF is proposed by the large language model (LLM) community and has been a mainstream framework for aligning LLMs with human preferences (Askell et al., 2021; Ouyang al., 2022). For more detailed survey of the history of RLHF, we refer Kaufmann et al. (2023). While many people have incorporated the supervised fine-tuning (SFT) stage as a part of the RLHF pipeline (Ziegler et al., 2019; Ouyang et al., 2022; Ji et al., 2023), our discussion would focus on the reward modeling and policy optimization via reinforcement learning (RL), which takes a supervised fine-tuned model as the starting point.
Although proven effective for aligning LLMs with human preferences, the canonical RLHF suffers in several aspects, including complicated implementation, difficult hyperparameter tuning. low sampling efficiency (Choshen et al., 2019), and computational overhead (Yuan et al., 2024), which promotes the studies to optimize with the relative preferences without depending RL. One major alternative to the RLHF is the direct policy optimization (DPO) proposed by Rafailov et al. (2024), which directly trains the language model to increase the probability of the preferred response and decrease the other. Wang et al. (2023) discusses the influence of f-divergence as the constraint term in PPO and proposes f-DPO. Azar et al. (2024) considers a general learning objective under pair-wise preferences, i.e., POOO, points out the potential overfitting issue in RLHF and PPO, and further mitigates the problem with a specific instance of L/PO, i.c., IPO. Zeng et al. (2024b) investigates the optimization at a finer level and employs forward KL. divergence constraints for each token to improve the alignment. There are also some methods (Xu et al., 2024; Ethayarajh et al., 2024) that try to skip the SFT stage lower the costs and mitigate the issues while directly imitating the reference data. Liu et al. (2023) uses statistical rejection sampling to address the mismatch between the training data and optimal policy data, hence enhancing the preference data collection. Amini et al. (2024) proposes DPO with an offset (ODPO) that takes the preference strength into consideration, which forces the language model to separate the probabilities of two responses in the training dataset by an offset. Compared to our work, the authors do not theoretically prove the benefits of considering the reference strength, and their proposal requires tuning the offset.
Parallel previously mentioned efforts to overcome the dependence on RI., researchers also directly optimize the policy under different loss functions with different types of preference data. We list some efforts non-exhaustively here, including RAFT (Dong et al., 2023a), SLiC (Zhao et al., 2023), LiPO (Liu et al., 2024c), RRHF (Yuan et al., 2024), and PRO (Song et al., 2024). For pairwise comparison data, Zhao et al. (2023) proposes a hinge-type loss to encourage the LLM to output the chosen sequence more likely than the rejected one. For the list data that compares many different responses, Dong et al. (2023a) selects the best response and fine-tunes the LLM on the best-of-K data. Yuan et al. (2024) adopts the ranking loss in the same spirit of increasing the preferred sequence's 's likelihood and summing up all the pairwise comparisons. Song et al. (2024) replaces the pairwise comparison with the preferred one against the remaining responses and recursively iterates from the most preferred one to the second-least preferred. All those ranking- based methods are purely based on the relative positions without considering the preference strength, and forcing a rank among nearly tied responses introduces additional noise. To mitigate this issue, Liu et al. (2024c) applies the LambdaLoss (Burges et al., 2006) to build a LiPO-A objective that takes the preference strength into the weights of the ranking loss. However, the proposed method has two drawbacks. First, the loss is heuristically defined and lacks theoretical guarantees. Second, the requirement of labeling each response with quantitative reward is not easy for human annotators: on the contrary, the initial motivation behind the RL.FF method to train reward model by pure comparison data to avoid asking human annotators to quantify reward precisely. In contrast, our ordinal feedback only requires the human annotators to calibrate the qualitative comparison with quantity.
With the rapid development of optimization frameworks, some scholars notice the issue of diverse preferences in reward modeling, not only in the field of LLM (Dong et al., 2023b). Zeng et al. (2024a) proposes a multi-objective reward learning method (MORE) to calibrate the reward models with the shared preferences and enhance alignment performance. Chakraborty ct al. (2024) introduces a MaxMin alignment objective to learn a mixture of diverse preference mixtures and greatly improve the overall performance. Wang et al. (2024a) employs the multi-objective reward modeling and models the preferences of users, implementing better objective control. Wang et al. (2024b) enhances the interpretability and performance of reward models by combining multi-objective reward modeling and mixture-of-expert techniques. Different from these works, our method considers a more practical scenario without requiring multiple labels for preference pairs and constructs the probability model for the single-objective preference learning.
Two recent works (Chen et al., 2024; Liu et al., 2024b) incorporate the tied samples into the learning of the reward model by considering some generalized versions (Rao & Kupper, 1967; Davidson, 1970) of the Bradley-Terry model (Bradley
12

---

Reward Modeling with Ordinal Feedback
& Terry, 1952). By introducing more complicated model, they enable the reward model to predict three probabilities: better, worse, and tied. The authors directly the cross-entropy loss for the trinary classification for the RM learning- As comparison, our ordinal feedback is more general and not limited to the Bradley-Terry model and the 3-level feedback. We also avoid introducing additional hyperparameters and keep the original training paradigm.
B. Proofs and Theoretical Discussions
B.1. Proof of Theorem 3.2
Proof. We first prove that the constructed ordinal feedback Mi..t satisfies Assumption 3.1, where
(2k Zoracle) / (Zoracle 22) (=x 2g);
/ (2k 10--- 2j);
if 2 2j+
(5,((2)=
if z k/ otherwise.
0,
If coracle 2; for any Z, then the constructed measure is the Dirac measure for Zoracles which automatically fulfills the requirement.
We consider the cases where Zoracle E (Zj, zx.) for some 2j, E 2. Then the expectation of Z w.r.t. jijk is
Ez [Z] (z)((zz) 2j + (2)((2z)) 2/
(2)21; 2, Zoracle 2y2 (Ex - 2z)
Zoracle
For the second part of Theorem 3.2, we prove the conclusion case by case. Suppose we have an ordinal feedback Z: 02 + Z satisfying Assumption 3.1. If Soracle E Z, for example, Zoracle Zio then
Mi
Mio.k 8zng"
implying set those coefficients Oj 's and A 's to be arbitrary non-negative real numbers such that E3 aj + Σx & p(2).) without affecting the measure the other points.
Therefore, we can (without loss of generality) assume Zoracle & Z. Assume < ... < 2mm < Sorucle 2mi+1 < ... <
Case (1): when there is only one point on each side of Zoracte. that is, my = 1 and m1 + 1 = m.
Then by Assumption 3.1, we have
((21)= 21 + (((mm) 2m
Zoracle
Combining it with the constraint such that
m(21) + p(²m)=
we have
((21) (Zm Zoracte) (zm 21), M(2m) (Boracke 22m 21);
which is exactly the same as N1,m-
Case (2): when there is only one point larger than Zoncle, that is, mi 1 Then we will prove that there exist non-negative real numbers ajjm such that
1 Σ
ajjm Mjmm-
In fact, we can construct (for any + m)
Oj.m == u1(z))
Mj.m (2z)
13

---

Reward Modeling with Ordinal Feedback
Then each jjm is non-negative. Furthermore, by Assumption 3.1,
Σ [1(2g)
Σ
ajm.Hjjm (Zm)
Hyjm (Zm))
M3.m(zz)
5pm
jmm
Σ jpmm (((zg)
Zoracle Esfm (b(2g) 2j
2m Zoracle
p((zm))
Zoracle
( lo( 2mm) Zm
2m Zoracle
((2m);
indicating that = E jm Ojjmpij.- By the property of probability measures, we can easily see that
Σ
a.m = 1.
fm
Case (3): when there is only one point smaller than Foracles that is, 1. This case can be proved similarly to Case (2).
Case (4): general cases where there are (possibly) multiple points on each side Zoracle- We prove it by induction. Suppose there are a elements Z smaller than Romele and b elements larger than Zoracle- Denote the case by (a, b). Suppose that the conclusion has been proved for all the cases (a, b) if m or b m m.. Now prove it for the case (m1, m m1). We first define the corresponding index (if there multiple elements in tie, select arbitrarily)
i == arg min | - Zoractel ((z.).
(5)
Without loss of generality, we assume 2i, Zoraele- Then we have m. > 1 due to Assumption 3.1. We now construct coefficient m, such that
((zz) Pi (Zix)
ai
By the definition (5), we have
p(zm++1) 2 0ig.m111 Pig.m +1 (Zm1+1).
Hence we can construct new measure
if i1;
pl(z)) (u(2mi+1)= 0b1.m111 Mi (2mm(((m+))))) /11 a.,m))))) (1(z))((1 - -ii,
if m. + 1; otherwise.
This measure can be easily verified as a probability measure with m. elements smaller than Zarnele- By induction hypothesis, we can construct nen-negative real numbers aj,4's 's summing up such that
1/ = Σ dy Pj.k-
J.K
Then we have
Σ
j.kk
= mi
Pii.
+
Pj.k.
m1+1
3.k
of which the coefficients are non-negative and sum to
14

---

Reward Modeling with Ordinal Feedback
B.2. Validating (3)
Cross-entropy loss:
The cross-entropy loss is
(Z, - [og + (1 2) log(1 - =)],
where [0, 1), which is the probability under the BT model. The cross-entropy loss is affine to Z:
Ez (2, =
Σ
P(Z = =2) [2g log(z) + (1 - 2z) log(1 -2)]
P(Z = =2) 22)) -log(2) + (1 Σ
P(Z = 2,) -))))))))) ::)) loog(1 2)
22)
JiRzEZ
[B|Z] log(z) + (1- -[]] log(1 -2)]
Leo (1(
Generalized hinge loss:
The hinge loss sets the loss be
Cninge (Z, 1[2 1} max (C (C-22, z, 0) + 1(2 03 max (C + 2, 2) where I, which is the difference of the reward functions (3( 91) (0, 1/2). We can generalize the hinge loss
Chingo (Z, = Z max (C 2, (1 Z) max (C + 2, 0)
The (generalized) hinge loss is affine to Z by a similar argument the cross-entropy loss:
Ez (ninge (Z, =)]
Σ P(Z zz) [zz max (C+ + 2, + (1 =z)) max (C z, 2,0]] E
Σ PiZ- = 2))) mmax (C 0) + (1- Σ
Píz = =z²) 23))22)) 2))
-max (C 2, 0)
JAZEE
jRzEE
E[Z] max (C + 2. + (1- AL1-Elg B[Z)) max (C - 2, 2.
= Eninge (B]
B.3. Validating (4)
We can prove a stronger conclusion such that for any (r. y1,
E X' x y12
Ez (Z, Mxx yn. uz)) (x: 3n, U2)] Ez [((z h(x, yr, 92)) (z. (1.
Since both Z and Z' satisfy Assumption 3.1, we have for any (I, yı, v22) E X' x y22
E|Z (x, y1, v2)l = Foracto (I,, Vl, U2) = 1E|Z ( (z, U1, V2]]
Then by the affinity condition (3), we have for any / E H.
E2 [ZZ h(x, U1, 322) (z. 1/1 82)]
[E|Z (z. yn, U2)], M(,, y1, U2) el Zorrete (x, 3r, 12). ha, U1, Uz))
The same arguments also lead to that which concludes the proof.
Ez (z' 1(x, vn, wel) (s: yi, 22)] = El (Zoracte (x, U/, M2), Ma,, 31. y2)),
15

---

Reward Modeling with Ordinal Feedback
B.4. Proof of Proposition 4.3
Proof. We construct (W, w") as follows: set W to be identically distributed as Z, and
P(W = W = =z) = 3z.e-
Then by property (a) of Bi,N's ss we have
WW = zz] دزン
indicating
W E[W[[W]
By property (b) of Bj.a* s, we have that the constructed W has marginal distribution identical to Z'. Hence, (W, W") is coupling satisfying the hierarchical expectation requirements.
On the other hand, if we have a coupling (W, W") satisfying the hierarchical expectation condition, we can easily verify that the conditional probabilities satisfy the requirements in Proposition 4.3.
B.5. Proof Corollary 4.4
Proof. For case (a) such that Z Zomele =: 2 almost surely, we have μ 8zantt By Assumption 3.1, the probability measure ye of Z' satisfies
Σ
(1(e)
2
Foracle
almost surely. Setting 3I.x == (zk) fulfills the properties in Proposition 4.3.
For case (b) such that 2' = {z] == 0, 2/2 = 13, we can construct Bj.x 's as
B₁ Zjz B;. = Zjz
for any 2j E Z. Then one can easily verify that the construction satisfies the requirement (a) in Proposition 4.3, For part (b),
by Assumption 3.1, we have
Σ
0z2j Zoracte:
J.aEZ
and
02 = Roracle-
Combining the above two equalities, have
Σ ajB3, =az
jRgEZ
On the other hand,
Σ
a,(1 2j) = Zoracle:
JRREZ
and
01 = Zoracley
which implies that
Σ a1B3,1 = =
16

---

Reward Modeling with Ordinal Feedback
B.6. Proof of Theorem 4.6
Lemma B.1. Any affine function is also convex.
Lemma B.2. The pointwise supremum of a family of convex functions still convex. In other words, for any family of convex fonctioonn fa(.) where 8 & S, we have supses f=(.) still being convex.
We provide those two lemmas without proof since the proof can be found in any convex analysis texthook. corollary is the following.
Lemma B.3. If the loss function satisfies the affinity condition (3), then for any hypothesis class H. the the following function
sup Σ NEH
is forvex any realization of taking values in {+1,
Proof of Lemma B.3. The argument breaks up into three pieces: first, any linear combination of affine functions still affine (which is straightlorward), hence
edl eell; (11,
is affine as long condition (3) holds.
Second, any affine function is also convex (Lemma B.1), therefore,
ael(s MC,, v1.))
is also convex for any h E H.
Third, the supremum of any class of convex functions is still convex (Lemma B.2), which means taking the supremum over the hypothesis class H suffices.
Proof of Theorem 4.6. We denote the hierarchical expectation coupling of Z and Z' by (W, WT). By direct inspection, we have
R.. (eo H)- = Er..y.
sup WEH i=1
ell 7%, (1,, v.1,
eil (W (1 Vi.1 M B.l
Em ,
sup NCH im1
Er .y.
Ew,w
sup he7 i=1
e e Mo,,, Vc.1, v.1.
= Err.y. M.s
Ew
Ew
sup WEH 1=1
e
W
>
- 1E M.z
Ew
sup NEW
(LE
Er. .
Ew
(W,
hEH i=1
E M.M.
Ez
E Ege(Z h2(hhha0 V1.1, V1.2))
heH
4m1
17

---

Reward Modeling with Ordinal Feedback
R.
sup nEW 1=1
e, e, V4.1, V1.2))
= RRddz,n H.),
where the first equality is by definition, the second equality is because the coupling's marginal distribution on W is equal that of Z', the third equality is due the exchangeability of the order of integration (by Fubini's Theorem), the fourth equality is because of the tower property of the conditional expectation, the first inequality is the result of Lemma B.3 and Jensen's 's inequality, the fifth equality is owing to the definition of the hierarchical expectation (Definition 4.2), the sixth is on account of the property of the coupling again, the seventh thanks to Fubini's Theorem again, and the last equality the definition of Rademacher complexity again.
B.7. Proof of Corollary 4.7
Proof. A straightforward conclusion of Theorem 4.6 and Corollary 4.4.
B.8. Proof of Theorem 4.9
Proof. Denote the one-hot vector at dimension j as 6j. The following arguments are made for any E X, and we omit the dependence on a for notation simplicity.
We only need to show that BT is a hierarchical expectation of and the theorem is the result Theorem 4.6. We construct the coupling (w, w') as follows:
== VT:
and
P(w' = eslw) == w/y
where wy denotes the j-th entry of w. We now verify that (w, w²) is coupling of (VT. v). The fact that w and yr have the same distribution is easy. For w/ and y, have
IP(w' = =e)) [ = =E[IP(w = esww)]
-R[wy]
E s1
Yoracle.
IP =G),
where ØT.j (or j)racl..) denotes the j-th entry of yr (or yoracte,, and the dependence on x has been omitted. Here, the first equality the tower property of the conditional expectation, the second due the definition of w', the third because the construction of w. the fourth account of Assumption 4.8, and the last is the definition of yornlle.
Hence (w, w') is coupling of (UT.V) y)-
We combine that conclusion with the fact that
E Elw
[w]
w,
leading to the conclusion that yr is hierarchical expectation of y.
B.9. Generalizations to DPO
Our results naturally extend to the direct policy optimization (DPO) (Rafailov et al., 2024) case. We first give quick introduction to the DPO training objective. Suppose the ground-truth reward function for any prompt-response pair (x. y) is (*( y). Then, under the reinforcement learning objective of maximizing the reward (with a Kullback- divergence regularization of strength 8 from the original policy mer)). the optimal policy under the ground truth reward function should be
1 ((.0)))))
x (y(x) x (uar()))))))))))) exp
18

---

Reward Modeling with Ordinal Feedback
Under the Bradley-Terry model, have
T (V11x) log P(y > 32/x)) = =0
(y) x+) Trer (322
(yr (x)
where the ((-) is the sigmoid function o (æ) = exp(x)/(1 + exp(x))) Then the DPO training objective is to minimize the
cross-entropy loss under the binary feedback setting
Zi log a 8 log To (35,
min
[ve((Ui,112))
To (1(.22
3 log Trerl (1r, 2x:g)
To (V(.2)))))
(1 - Z,) log
Blog (1c 24,)
To To (T2)
-log
(6)
Teer (io (1r((Vi,1(x4))
By considering a richer feedback than the binary case 2 ordinal feedback. The affinity condition (3) is fulfilled since the loss is still the cross-entropy loss. Thus, applying Theorem 4.6 for 11 ( 0 } yields similar result. We present here without repeating the proof.
{0, 1}. we can use (6) to train the LLM ma directly under the
Corollary B.4. Consider the corresponding loss function in the training objective (6) and policy class II { E θ). For any ordinal feedback systems Z and Z' taking values in Z and Z' such that Z is hierarchical expectation of Z', have
Ra MI S Radz (n (e
B.10. Generalization bound under Rademacher complexity
The following proposition is a well-known generalization bound. We present here only for the completeness of our argument as the proof can be found in any statistical learning lecture notes.
Proposition B.5 (Generalization Bound). Suppose we have Dz as dataset consisting of n i.i.d. samples. For any hypothesis class H. we have with probability at least 1 - that for every function E H.
B B[e(Z,, (.1. < EDअ [( ((,,, + 2Radz,n (f H. + log(1/8)
B.11. Lower bounds reduction of Rademacher complexity
We take the cross-entropy loss an example. Suppose we have two ordinal feedbacks Z and Z', taking values in 2 and Z', respectively, Suppose Z is a hierarchical expectation of Z', of which the coupling is formulated (W, W'). We make further assumptions here besides the "wisdom of the crowd" assumption (Assumption 3.1).
Assumption B.6. Assume that each Z; (or Z/) is independent. Furthermore, assume that each zj is bounded away from 0 and by
7< < 2 < <17-
The above assumption is not restrictive: the independence naturally holds if each annotation is independent. The boundedness can be satisfied by clipping the feedback options, and is assumed to prevent those infinite values attained in the supremum step due to logarithms (if = +1) while making sure those maxima can be obtained if s = -1. Assumption B.7. Assume that the function class H. is rich enough (w.r.tt 7) with high probability (2 1 1 36 2 0, s.t.
Ó).). 6). That is,
E (br 1- >]" E (h((1) 11,1 v1,2)... ... Al( Vn vn, E H}}
holds with probability at least 8.
19

---

Reward Modeling with Ordinal Feedback
We denote the number of positive Rademacher variables (that is, those e, +1) by (which is random variable itself). From the definition of т, we know
n E | = 2
Without loss of generality. we assume that E1 -... = +1, and =... = -1. Due to the assumption that each (I,, 1t,1 Vi,2; Z₁) is i.idd (or Z),, such 7 summarizes all the dependence on E. The two Rademacher complexities are (omitting the dependence on æ, 91, y/2 for notational simplicity and denoting h; == h (24 96.1 Mi.2))
1
Radz,, (L H) -E,,w
sup WEH
Wi (o-(ogg(hs)) - (1 - W.) log(1 he)
n
i=1
Σ
W; log(h:) W.) log(1 nx))
1=
and
Rad (f H)) -R,,,
Ew
sup hEH
W log(h:) (1 - W() log(1 - h;)
1=1
+ Σ 1=
Wi log(h;) + (1 W3) loog1 - hss) ww
Lemma B.8. Under Assumptions 4.8 and B.6, if E holds, then the following holds (here, qs 's are real numbers independent of h4ys s):
sup NEW
W. log(h;) W.) log(1 - h.) +
Σ
W; (og(og(hh) + (1 - W:) -oog1 h4)
x
sup KCH 1=1
W; log(h:) - (1 W.) log(1 - h;) +
sup
Σ
W (o-log(a:) + +(1 W.) -log(1 9.),
11h,1111
im++1
and
sup bEH
W/. log(h;) - (1- W/) log(1 h.) +
Σ
W/ log(h;) (1- W() llog(1 hi)
n
sup heW
Σ
W log(h+) (1 Wi) log(1 h₂)
sup
Σ
W log(g:) + (1 W?) log(1 q:).
w.h.
Proof. From the fact that {(h1..... ...1.....,...................... q,) H} c [(h E H} under E, know that
sup NEW
Σ
Wi log(h:) (1 W.) loggl1 -h+) + Σ
W, log(h) + (1-W:) log(1 h2)
sup
W log(h:) (1 W.) log(1 h;)
Σ
W. log(g;) + (1 W.) log(1 9.)
he 1.7.111/1110
i=++1
n
sup kEW
Σ
WW log(h:) (1 -W.). -log11 h:) sup
Σ
W log(g:) + +(1 -W,) -log(1 kc),
1
and
sup
Σ 1=1
W log(h:) - (1 - W!) log(1 - h₂) + Σ
W log(h;) + (1 - W;) log(1 - hz)
bER
i=++
20

---

Reward Modeling with Ordinal Feedback
sup e
Σ W log(h:) (1 Wi) log(1 h2)
W log(g:) (1 W/) log(1 q:)
i
sup heH
Wi log(h:) (1 W() log(1 h₂) sup
W log(g:) + (1 W() log(1 -q))
weh,1111
imr+1
From Assumption B.6, we know that Vh E H.
W₁ logh:) + W.) loog1 - h.)) s
Σ 1=+1
-En)(W;)= = sup
Σ
W (g.log(()) + (1 (1-W)) -log(1 - -;),,
4. 1=
1=++1
which implies that
sup
Σ
Wi log(h,) - -(1 (1- W.) log(1 - h1) +
Σ
Wi Wi-loge (og())(h)) + (1-W) loog(1 h4)
NCH
1=++1
Sup NEW
W; log(h:) - (1 W;) log(1 h12) + sup
Σ
W; (g.log(ht))) + (1 W2) loggl - -t:)
NEH
1=1
i=++1
n
sup KCH
Wi log(h,) - (1 W.) log(1 - h.) + sup
Σ
We-log(g)) + + (1, -log(1 q.).
1=1
i
Combining the above, conclude the proof.
Lemma B.9. Denote the supremum over the cross-entropy
f(z) = sup
z- =-log(g) + (1-2) 2) -log(1 - q).
9E(9,1->l
Then
Radzz,n H) - Radzzn (1 H) 2 Er,w
Σ l1++1
Ew [s(W))
E.,w
Σ s(W,) lir+1
n
Proof. Due the property of supremum, the supremum taken over the entire (gr+1..... In) E [y, 1 2 T is equivalent to the supremum taken step by step (that is, by taking each q;' 's supremum one by one). Therefore,
W. Wl-log(::) + (1- log(1 (p-
W; Woglog(4) 1-)
sup
+ +1 (1- log([ - 9:)
Σ J(W.).
4.E|,.1-1)
imT+1
Similar conclusions also hold for Wi's. We conclude the proof by combining the facts with Lemma B.8.
Lemma B.10. Suppose w E [7, 1 - 7] and w' E [0, 1]. Denote the clipped wt by
w/, = max (,, minfw', 773..
Then
f(w') 2 f(w) + dʃ (w' - w) + 2(ww/g w²)²
dz
Proof. For those w' E Eh,11 - -7], w/, w/. The function f of w' is the entropy, of which the second-order derivative is
d
w0(1 (1-www w/) 2 4.
d
2=w"
Then by Taylor's formula, we know that 39 E [0, 1], s.t.
S(w') = f(w) + df
d (w' (a'-ww) + d=2
(w' ww)
dz 1=
21

---

Reward Modeling with Ordinal Feedback
df dz (w -w) + 2(w'" w)²
2 (w) +
For those w' > 1 7, w/ = 1 - 7. The function f is linear in the interval [] 7, 1J, of which the first-order derivative is da 1z=17 (easy to check f is continuously differentiable 7). Thus,
(00)= + df
dz == (w' - w/y
2 f(w()) + df
(w' - -w'))
dæ
f(w) + df
(w' - w¹y + w²y -w) + (w', w)²
dz
which verifies the proof.
For those w' < 7, the conclusion holds similarly.
Proposition B.11. Under Assumptions 3.1, B.6, and B.7, we have
Rad H) Radz.n (f H) 2 (Bw [Var(W"[]]]
Proof. Denote the conditional distribution of W/f on observing W; by Pw:jw.- From Lemmas B.9 and B.10, we obtain that
Radæ (e H) - Radz, (& H)
2 Er,w
Σ
Ew [S(W;) Wa]
Er,,
Σ s(wi)
i
l=++1
1
/ S(W)) S(W)) dPwww..
- Er,w
i
1 Er,w w
Σ S(W:) + df
(W7 W.) + 2(W,, W.) -(W,) dPw::w.
11
dz 2=
- Er,w
Σ
2(W w.) d.w:qw.
n
By Assumption B.6, we know that
(W7, W2)
(W7 W3)2 = 9(1),
which implies that
f 2(W -W,, dPwiw,, 2 2(1) /(W -W,) dPwwww. ?(Var(W:)lW.).
wo)
Since E(7] = 3. we know that
Rad (e(o H) - RRd (e H) = ((VVa(WW][W.))
22

---

Reward Modeling with Ordinal Feedback
C. Complementary Numerical Results
this section, we present some more experimental results that are not listed in the main paper due to space limitations.
C.1. Complementary figures
In-distribution accuracy
Oracle cross-entropy loss
0.94
Oracle
0.61
5-level
0.92
0.60
3-level
0.90
Binary
Oracle
0.59
0.88
5-level
0.86
3-level
0.58
0.84
Binary
0.57
200
400
600
800
1000
0
200
400
600
800
1000
Steps
Steps
(a) in-distribution (ID) accuracy/lamaa models
(b) Oracle CE loss/Ilama models
In-distribution accuracy
Oracle cross-entropy loss
Oracle
0.94
0.60
5-level
3-level
0.92
0.59
Binary
Oracle
0.90
5-level
0.58
3-level
0.88
Binary
0.57
200
400
600
800
1000
200
400
600
800
1000
Steps
Steps
(c) In-distribution (ID) accuracy/gemma models
Oracle CE loss/gemma models
in-distribution accuracy
Oracle cross-entropy loss
0.950
Oracle
0.925
5-level
0.62
3-level
0.900
Binary
0.875
Oracle
0.60
5-level
0.850
3-level
0.825
Binary
0.58
0
200
400
600
800
1000
200
400
600
800
1000
Steps
Steps
(e) In-distribution (ID) accuracy/qwen models
Oracle CE loss/qwen models
The evaluation dynamics of the three base models for different ordinal feedback labels. The horizontal axis "Steps" represents the number of training samples; the rightward direction means larger training set. The vertical axis represents the "accuracy" or the "loss" separately. For the accuracies, the larger, the better. For the losses, the smaller, the better. We can see from the figure that more fine- grained feedback leads to better RM learning.
Figure 2.
23

---

Reward Modeling with Ordinal Feedback
In-distribution accuracy
Oracle cross-entropy loss
tied ratio 0%
0.94
1.0
tied ratio=25%
tied ratio=50
0.92
0.8
tied r
0.90
tied ratio=0%
0.6
0.88
tied 25t
tied 50t
0.86
tied r5tio=75%
0.4
100
200
300
400
100
200
300
400
Steps
Steps
ID accuracy/llama models
Oracle CE loss/llama models
In-distribution accuracy
Oracle cross-entropy loss
tied ratio=0%
tied ratio=25%
tied ratio=50%
0.94
ratio= 75%
4
0.92
tied ratio=0%
tied ratio=25%
tied ratio=50%
2
0.90
tied ratio-759) 75%
100
200
300
400
100
200
300
400
Steps
Steps
(c) ID accuracy/gemma models
(d) Oracle CE loss/gemma models
in-distribution accuracy
Oracle cross- entropy loss
0.94
tied ratio=0%
0.92
tied ratio=25%
tied ratio=50%
0.90
tied ratio 75%
0.88
tied latio=0% 0%
0.86
tied 25tio=25%
2
0.84
tied 55t
tied ratio= 75%
0.82
100
200
300
400
100
200
300
400
Steps
Steps
(e) ID accuracylqwen models
(I) Oracle CE loss/qwen models
three models for different data ratios. horizontal axis "Steps" represents the number of training samples; the rightward direction means larger training set. The vertical axis represents the "accuracy" or the "loss" separately, For the accuracies, the larger, the better. For losses, the smaller, the better. The 100%-tied case is not plotted as would detract from the clarity and readability of the plot due to its failure. We can from the figure that mixing certain ratio of tied data benefits the RM learning compared to 0% performance.
Figure evaluation dynamics
C.2. Reward modeling with ordinal feedback under hinge loss
In the main paper, we focus on reward modeling with ordinal feedback under the cross cross-entropy loss. Here we extend the analysis to the case of hinge loss. Recall that the learning objective (2) the main paper can be interpreted an induction
24

---

Reward Modeling with Ordinal Feedback
of the cross-entropy loss based on the Bradley-Terry model. Given the probabilistic model of ordinal feedback, the learning objective naturally be extended to incorporate other types of loss functions. Among these, hinge loss (Schölkopf et al., 2004) is one of the most widely used, particularly in classification tasks, alongside cross-entropy loss. Hinge loss is commonly associated with Support Vector Machines (SVM) and is characterized by its core principle of enforcing a margin between distinct classes. Building on this observation, we define the learning objective under hinge loss as follows:
min Z. [max (0, C
re ro (2.. (1 2,) |max (0, C (re (344, V1.2) re re Mta))))
where C is the margin hyperparameter that controls the separation between preference classes. When the feedback is binary, i.e. Z4 {0, }). the above objective simplifies to the hinge loss commonly used in reward modeling (Liu et al., 2024a). In our experiments, the margin parameter is tuned with grid search, where the search space is (0.5, 1, 1, 4), and we select C C=1. 1.
To further validate the conclusions presented in Section 5, we conduct analogous experiments under the learning objective
(8). The feedback in the oracle feedback setting is still kept as
Z: = (yt,1 > y1,2/ra) = Zaracle (2i Mi.1 (/,2)
just as the cross-entropy setting. The feedback in the Jevee//r-yy settings is sampled as the process in Theorem 3.2 by considering only the smallest interval [3j, 2j+1] 3 Suracle- We then replicate the experimental setup from Section 5.2, employing
our base model. The experiment results are reported in Figure 4 and Table 3.
Experiments illustrate the same three conclusions in Section 5.2: (1) more fine fine-grained feedback structures result in better learning for both ID and OOD performance; (2) the fine-grained feedback (e.g. 5-level) may be good proxy for the oracle; (3) the generalized hinge loss handle the feedback richer than binary well.
Apart from the observations analogous to Section 5.2, we observe that the hinge objective performs weaker than the cross-entropy objective. We attribute this outcome to two key factors. First, margin hyperparameter C' significantly impacts the model's convergence speed and overall performance, which we only with a coarse grid due computational constraints. Second, the inherent nature of the hinge objective means that only specific subset of data points influences the final decision boundary. In contrast, the cross-entropy objective leverages the entire dataset during optimization. Given the complexity of language modeling and the intricacies of semantic space, we hypothesize that preference data embeddings are distributed in a noisy and overlapping manner such that the decision boundary may not be effectively established under the hinge objective.
0.74
In-distribution accuracy
0.94
Oracle
0.72
5-level
0.92
Oracle hinge loss
3-level
0.70
Binary
0.90
0.68
0.88
0.66
0.86
Oracle
0.64
5-level
3-level
0.62
0.84
Binary
0.60
200
400
600
800
1000
200
400
600
800
1000
Steps
Steps
(a) accuracy
(b) Oracle GH loss
Figure 4. The evaluation dynamics of laama models for different ordinal feedback labels under generalized hinge loss. The horizontal axis "Steps" represents the number training samples; the rightward direction means a larger training set. The vertical axis represents "accuracy" or the "loss" separately. For the accuracies, the larger, the better. For the losses, the smaller, the better.
25

---

Reward Modeling with Ordinal Feedback
Table Model convergence statistics for llama under generalized hinge loss.
Oracle CE
Loss
ID Accuracy
OOD Accuracy
Model
Feedback
Mean
Std
Mean
Std
Mean
Std
Oracle
0.6030
0.004
0.9359
0.0036
0.7798
0.0076
5-level
0.6046
0.0003
0.9345
0.0076
0.7660
0.0068
Llama
3-level
0.6037
0.0003
0.9326
0.0068
0.7617
0.0163
Binary
0.6072
0.0005
0.9322
0.0020
0.7580
0.0040
Algorithm 1 3-level Sampling Algorithm
Input: Oracle label Foracle E [0, 1]
Output: Sampled label 2z (0, 0..,
if Soracle <, 0.5 then
Sample ~ Bernoulli Aomate)
2 6 0.5
else Zoracle > 0.5 then
Sample y Bernuulli Fonale Fommle-0.5)
0.5
0.5 y 0.5
else
+- 0.5 end if
return z
C.3. Dataset details
(SRP) dataset is curated subset of publicly available preference data, spanning wide range of knowledge domains. Reward models trained on this dataset have achieved top performance in the Reward Bench benchmark. The released version contains 77,016 samples, with approximately 5,000 overlapping samples removed compared to v0.1. our experiments, we used Skyywork-Rewand Gemma-2-27B-v0.2 annotate data pairs with oracle scores.
Skywork-Reward SreearddPreferencc--8K-----
The -eatt--eeaa-----eeeeeeeee
Rationale For Scaling, Instead of direetly using the sigmoid values of the oracle model's score differences, introduce scaling parameter 7 because the raw output scores of the oracle model are highly concentrated, as shown in Figure 6a. However, it is generally understood that people often hold diverse opinions on preference samples, meaning real-world preference distributions should not be heavily concentrated near a probability of 1. To investigate, we consider two commonly used preference datasets with fine- line-grained preference scores, UltraFeedback (Cui et al., 2023) and HelpSteer2 (Wang et al., 2024c). visualization of their preference ratings provided in Figure 5, where many samples show no strong preference but only weak agreement. Based these observations, we carefully adjusted the scaling parameter T and chose T ensuring it produces peak within the slight agreement interval (approximately 0.6-0.7), shown in Figure 6b. This choice is further justified in the explanation of the tied sample experiments discussed later.
26

---

Reward Modeling with Ordinal Feedback
3000
60000
2500
50000
# Samples
# Samples
2000
40000
1500
30000
20000
1000
10000
500
0.0 0.5
2.5 3.0 3.5 4.0 Preference strength
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Preference strength
(a) Preference strength distribution in UltraFeedback
(b) Preference strength distribution in HelpSteer2
Figure 5. Distributions of preference strengths in the two datasets. For the UltraFeedback dataset, we compare the chosen and rejected scores pairwisely and their differences preference strengths. For the HelpSteer2 dataset, its latest version provides preference strength label and we directly adopt it.
50000
2000
40000
1750
1500
# Samples
# Samples
30000
1250
1000
20000
750
500
10000
250
0
0
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.8
0.9
1.0
Oracle score
Oracle score
(a) Distribution of raw oracle
(b) Distribution of scaled oracle
Figure 6. Distribution of oracle lahels before and after scaling with T Note that the oracle score is always recorded for the chasen response relative to rejected response, hence, all oracle scores are no less 0.5.
Tied Samples. As presented in Section 5.1, we refer to preference pairs with a label = 0.5 under the 3-level feedback system Z3 = {0, 0.5, 1} as tied data or tied samples. We use the term "tied" because the labels are generated based on the oracle label Soracle using an interpolation paradigm, as described in Theorem 3.2. These labels represent preference samples where people perceive (almost) equal advantages. The detailed sampling process is summarized Algorithm 1. And can be easily extended any ordinal feedback system.
Intuitively, the closer the oracle label of preference pair to 0.5, the more likely it is to be assigned label of 0.5. An important observation is that the sampled label distribution is dominated by Zoracle- To ensure sufficient binary and tied data for the tied ratio experiments, we aim for the binary and tied data to each constitute half of the dataset. After tuning, we 3 not only simulates a real-world preference data distribution but also satisfies the tied ratio experiment requirements. This further justifies the choice T.
found that selecting 7' 20
Rench Evaluation. The RewardBench evaluation dataset combines multiple datasets across four categories: Chat,
27

---

Reward Modeling with Ordinal Feedback
Chat Hard, Safety, and Reasoning. The scoring follows standard reward modeling paradigm, where success is defined as the chosen response having a higher score than the rejected response for given prompt. The evaluation score is computed as weighted average across all prompts in the selected subset.
28

---

Reward Modeling with Ordinal Feedback
C.4. Training details
Table Hyperparameter Search Space
Hyperparameter
Search Range/Values
Learning Rate
[le-5, 5e-6, 2c-5] [64, 128]
Batch Size
Warm-up Ratio
[0.03, 0.05, 0. 0.10]
Table 5. Shared Hyperparameters
Hyperparameter
Value
Batch Size
128
Optimizer
paged paged_adame Ladamw.aww.... .32bit le-3
Weight Decay
Epochs Scheduler
Linear Warm-up + Cosine Decay
Table 6. Model-specified Hyperparameters
Hyperparameter
Llama-3.2-lb
Gemma-2-2b
Qwen-2.5-1.5b
Learning Rate
le-5
5e-6
5e-6
Warm-up Ratio
0.1
0.05
0.05
We choose some key training hyperparameters based on a grid search. The performance is assessed by in-distribution evaluation (CE) loss under oracle label settings. The grid search space is shown in Table 4.
The base models share most of the training parameters, as given Table 5. The different parameters are listed Table
29


Reviews:
[{"review_id": "hmJWH4aQ7w", "reviewer": "Reviewer_rnHP", "summary": "This paper proposes Reward Modeling with Ordinal Feedback as an alternative to traditional binary feedback in reward modeling. The authors argue that binary preference data discards valuable information, such as subtle distinctions between choices and tied responses. They introduce an ordinal feedback framework that incorporates a marginal unbiasedness condition, justified by the sociological concept of wisdom of the crowd. The paper provides a theoretical foundation demonstrating that ordinal feedback reduces Rademacher complexity, improving generalization. Empirical experiments validate that fine-grained feedback enhances reward model performance in both in-distribution (ID) and out-of-distribution (OOD) settings, and incorporating a certain proportion of tied responses further improves model training.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the kind comments on our work. Here are our responses to the reviewer’s concerns and questions.\n\n1. Experiments only contain 2 base models, which appear not to be very solid for the LLM community.\n\nThanks for pointing out that issue. We have conducted some further experiments on Qwen 2.5 base models, and the updated results can be found in this anonymous link: https://anonymous.4open.science/r/icml25-anonymous-4r32. We apologize for not including further foundation models due to the limitations of time and computational resources. We are glad to add more results to our future versions.\n\n2. The literature on the “intransitivity” risk of the BT model during RLHF is not mentioned in this paper.\n\nThanks for the reviewer’s notice. The term “intransitivity” describes cases with cyclic relations in the pairwise preference comparisons. The literature focusing on the intransitivity risk of RLHF (e.g., the paper the reviewer mentioned) mainly focuses on providing alternative ways to model the preference probabilities (e.g., the Blade-Chest model) rather than a scalar reward score in the BT model. However, our paper aims to provide a plug-in solution to fully use the fine-grained feedback under the same framework of the scalar reward function. In that sense, the intransitivity of RLHF is somewhat “orthogonal” to our work. But we are glad to include more discussions about the intransitivity case of preference learning in our future versions, and thank the reviewer for pointing them out.\n\n3. Consider RM is a core composite in RLHF, what is the ultimate goal for generalizing the RMs to cover “hard cases” and improving the expressiveness of “rewards”?\n\nThanks for the question. As we understand, we guess the “hard cases” represent those samples where $(x, y_1, y_2)$ has a preference probability close to 0.5. From our interpretation, those “hard cases” are discarded even though options such as “tied” are introduced by (some of) the current practice, while those cases do have a positive chance to appear in the following RLHF step. For example, LLMs’ responses to some prompts may be intrinsically hard to distinguish, making almost all the pairwise comparisons for those prompts impossible to learn. Discarding those “hard cases” might cause the reward function landscape to be unpredictable and undesirable on those prompts, deteriorating the RLHF quality (e.g., over-optimization issue). For the second part (improving the expressiveness of “rewards”), we believe it’s a promising direction for future LLM alignment studies, considering the subtlety and diversity of human preferences. Researchers have made several attempts to handle the preference beyond a scalar reward function, including multi-dimensional scores (e.g., HelpSteer2 includes “helpfulness”, “correctness”, and so on) and the intransitive model mentioned by the reviewer. We believe the ultimate goal of such research is to train LLMs to better align with subtle and diverse human values.\n\n4. How does the theoretical work make wider impact to the optimization algorithms and human-in-the-loop in operation, e.g., in an active learning setting?\n\nIt’s an interesting question to see how this work is linked to the active learning algorithms. For example, the uncertainty sampling algorithm originates from the spirit that “those samples close to the decision boundary are more valuable in learning the decision boundary”. In that sense, when training RMs, uncertainty sampling would require the learner to query more samples “near the boundary”, that is, preference probabilities close to 0.5, and expect a better outcome. However, that might not be the case in RM training, as we have shown that as the tied ratio increases to 100%, the training result becomes worse (e.g., Table 2 and Figure 3). Another interesting bunch of active learning algorithms is the representation-based algorithms (e.g., CoreSet or MaxHerding), which require the queried samples to cover the feature space as much as possible. In that sense, our results imply that covering some samples close to the boundary (those “tied” ones) would benefit the learning (Table 2 and Figure 3) compared to the current practice of simply discarding these samples. We believe there are also other possibilities to improve the learning of RMs once all the fine-grained labels are included.\n\nWe are grateful for the questions and suggestions. We are glad to discuss with you in the following week if there are any further questions/comments.", "claims_and_evidence": "The below major claims are supported through theory and experiments.\n- Ordinal feedback improves reward model learning: this is supported by theoretical results showing reduced Rademacher complexity and empirical experiments demonstrating better accuracy than binary feedback.\n- Ordinal feedback connects to knowledge distillation: a bias-variance trade-off analysis links the benefits of soft labeling to ordinal feedback.", "methods_and_evaluation_criteria": "The methods and evaluations are solid, but comparisons to other preference modeling techniques (e.g., DPO variants) would strengthen the findings.", "theoretical_claims": "The proofs appear solid, leveraging convex analysis and coupling arguments.\n- The generalization issue is justified by Rademacher complexity.\n- The connection to knowledge distillation is justified by a bias-variance trade-off analysis.", "experimental_designs_or_analyses": "The authors conduct ablation studies with 2 base models and 4 feedback types (binary, 3-level, 5-level, oracle). It has sufficient evidence and analysis supporting that finer-grained labels enhance learning.\nHowever, the only 2 base model appears to be not very solid for the LLM community where more types of foundation models exist.", "relation_to_broader_scientific_literature": "The paper extends Bradley-Terry models and connects to RLHF reward modeling.\nIt (re)-addresses the issues in BT models and the importance of ordinal feedbacks, soft labeling and knowledge distillation techniques in the proposed framework. \n\nTo my knowledge, there is another line of work searching for generalized RM modeos. In a broader landscape, RLHF is exposed to 'intransitivity' risk, because it rely on Bradley-Terry model as the preference function, where all preferences are transitive by assumption. \n- Certain literatures have shown that such 'transitive' relationship between preference annotations may not always hold and some techniques are explored but not mentioned in this paper. \n- https://arxiv.org/abs/2409.19325 (Duan et al, 2017) presented some evidence and can be of interest for future work.", "essential_references_not_discussed": "This paper covered its essential references.", "other_strengths_and_weaknesses": "See discussed above.", "other_comments_or_suggestions": "See discussed above.", "questions_for_authors": "- Consider RM is a core composite in RLHF, what is the ultimate goal for generalizing the RMs to cover 'hard cases' and improving the expressiveness of 'rewards'? \n- How does the theoretical work make wider impact to the optimization algorithms and human-in-the-loop in operation, e.g., in an active learning setting?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UVFErdt2AQ", "reviewer": "Reviewer_ynWp", "summary": "This paper proposes a new framework for reward learning with ordinal feedback, which uses more fine-grained preference pairs generalized from binary preferences. Using a generalized probability model, the authors prove that the ordinal feedback framework reduces the Rademacher Complexity. The authors also use this theoretical argument to render a new bias-variance tradeoff for soft labels in knowledge distillation. Empirically it is shown that fine-grained feedback improves reward model’s accuracy both in distribution and out of distribution.\n\n## update after rebuttal\nThanks the authors for their detailed responses. My questions and concerns are mostly addressed, and I will keep my original rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating our work and the interesting questions.\n\n1. Despite in the domain of RLHF, what is the difference between your framework and training upon soft labels in knowledge distillation?\n\nThe major difference between our work and the knowledge distillation of trained RMs is the character of those RMs. In our work, we use the trained RMs’ soft labels to *simulate* the underlying belief of a population of annotators. Ideally, we would like the data to be collected from real human annotators, but current open-sourced human preference datasets do not include multiple levels of feedback simultaneously, which makes it hard to show the benefits of more fine-grained feedback numerically. We have to resort to the oracle RMs to simulate those multiple levels of feedback; in practice, for LLM companies, such ordinal feedback should be collected from real human annotators. Using RMs to simulate human annotators’ feedback, perhaps, isn’t the best way to convince the audience, yet it is a probably the only feasible method and has been applied in many papers (e.g., “Scaling Laws for Reward Model Overoptimization”). We are not hoping to improve over those trained RMs via knowledge distillation. Instead, we view those RMs as the *gold-standard* models (“oracle”) to value the benefits of fine-grained feedback. On the contrary, the knowledge distillation hopes to obtain a better student model via the soft labels from the teacher model. Such a process, although closely related to our theoretical results (Theorem 4.9), has a different goal from our work. Our work aims to show that (1) fine-grained ordinal feedback can be taken a good advantage of by endowing it a numerical meaning rather than just throwing it away, and (2) by providing the population with ordinal feedback options, there are some provable benefits (under the assumption of the “wisdom of the crowd”).\n\n2. How does upsampling the conflicting labels in preference dataset relate to this ordinal method?\n\nWe thank the reviewer for this question. In our opinion, the samples with conflicting labels are those with stronger noises or larger variances (that is, the preference probability close to 0.5 in our framework). If we treat the preference learning problem as a binary classification problem (to decide which one is preferred), the noisy parts are those close to the underlying decision boundary. Upsampling that part is a method to improve the learning on those “hard cases”, reducing the noise by averaging. In that sense, our ordinal feedback method shares the same spirit of decreasing the noise but through a different approach. Our goal is to prevent the noises caused by the annotation system itself. For example, if the annotation system only contains binary options, and annotators are faced with a sample $(x, y_1, y_2)$ with very similar $y_1$ and $y_2$. Then, the probability of each response being chosen is one-half, leading to possible conflicting labels in the dataset. But many of the conflicts can be avoided if a third option, “tied”, were provided. Instead of upsampling those “hard cases” under the *same* feedback system, we try to point out that the learning could also be improved by designing a *better* feedback system.\n\nWe thank the reviewer again for the questions and suggestions. Also, we have corrected those typos. We are happy to have further discussions in the following week.", "claims_and_evidence": "Yes. The paper provides detailed explanation on the problem formulation, theoretical statements, and numerical experiments.", "methods_and_evaluation_criteria": "Using fine-grained feedback pairs for reward learning is a reasonable approach to the problem.", "theoretical_claims": "Yes, I checked the proofs for Theorem 3.2 and Theorem 4.6 and did not find any major issues.", "experimental_designs_or_analyses": "Yes, the experiments are done with the standard reward learning pipeline and seem fine to me.", "relation_to_broader_scientific_literature": "Improving granularity of feedback pairs seems a reasonable approach for addressing the shortcomings of current reward learning frameworks. The authors provide both theoretical and empirical justifications for the benefits of their ordinal feedback framework.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "About weaknesses of the paper, one problematic aspect is about the way of collecting data. The ordinal feedbacks are supposed to be obtained from human labeling, but in empirical simulations are in fact generated from a teacher reward model. In this case, it is hard to distinguish the proposed framework from knowledge distillation.", "other_comments_or_suggestions": "Minor typos: \n\nLine 223: uncompiled latex control sequence Omega\n\nLine 366: “tide” should be “tied”.", "questions_for_authors": "1. Despite in the domain of RLHF, what is the difference between your framework and training upon soft labels in knowledge distillation?\n2. How does upsampling the conflicting labels in preference dataset relate to this ordinal method?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "efurGHTvwo", "reviewer": "Reviewer_Tq3d", "summary": "The paper examines reward learning in scenarios where annotators provide ordinal feedback. Specifically, annotators select an option from a set $\\mathcal{Z}$, which, in the common case of binary feedback, could be $\\{0,1\\}$. The authors first offer a statistical justification for the benefits of this extension, arguing that compared to binary feedback, more granular feedback can reduce the Rademacher complexity when using the cross-entropy loss. They then support this claim with empirical evidence, using simulated fine-grained feedback to demonstrate the advantage.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable questions. Before we dive into the rebuttal, we’d like to summarize the reviewer’s major concerns.\n\n1. The construction of ordinal feedback via Theorem 3.2 is not justified. There are possible reasonable options (e.g., the Luce-Shepard model) that have not been considered.\n\n2. Assumption 3.1 alone is insufficient to guarantee an impactful expansion on the feedback set $\\mathcal{Z}$ by introducing some vague options.\n\n3. Theorem 4.6 and Corollary 4.7 prove that ordinal feedback may bring up some benefits, but they do not fully characterize how large the benefits are.\n\n4. The linear system of $\\beta$’s is underdetermined: there are $m \\times n$ variables yet only $2m + n - 2$ independent linear constraints, which makes Proposition 4.3 seemingly trivial.\n\nFor Point 1, the answer is that the Luce-Shepard model is not considered because it cannot induce a natural reward function (RM). Algorithms like RLHF need a reward function for every prompt-response pair for the following policy gradient steps. That reward function is obtained via pair-wise comparison in the BT model: two reward scores, two probabilities. But if multiple levels of feedback are considered, those probabilities do not naturally correspond to *two* reward scores. For example, the Luce-Shepard model formulates those probabilities with multiple scores: $p_i = \\frac{\\exp(s_i)}{\\sum_j \\exp(s_j)}$. How to obtain two scores with three $s_i$’s remains unclear, as the additional “tied” term cannot be neglected. Our work is to construct a simple *plug-in* solution to those multiple-level feedback systems while keeping only two score functions. Due to practical limitations, we have to turn to RMs rather than real human annotators to simulate the labeling process; Theorem 3.2 is only to show how this simulation should be conducted rather than modeling a single annotator’s behavior (e.g., whether s/he chooses the closest $z_i$ or not). We admit that the simulation process may not fully prove the effectiveness of our method in the real world, but that’s the best one can do without really hiring annotators.\n\nPoints 2 and 3 are about the same question: How large is the benefit of ordinal feedback? We thank the reviewer for encouraging us to think deeper. The reduction of the Rademacher complexity (Theorem 4.6 and Corollary 4.7) is a direct result of Jensen’s inequality. The answer to the question is: the reduced quantity is approximately of the same order as the conditional variance in the hierarchical coupling. For example, suppose there are two ordinal feedbacks $Z$ and $Z^\\prime$, $Z$ is a hierarchical expectation of $Z^\\prime$, and the coupling is $(W, W^\\prime)$. Then Jensen’s gap would be of the same order of $\\mathbb{E}[\\mathrm{Var}(W^\\prime | W)]$ (see https://arxiv.org/pdf/1707.08644 for example), by assuming the supremum in the Rademacher complexity as a function of feedback $Z$ (or $Z^\\prime$) is twice differentiable, $c_1$-strongly convex, and $c_2$-smooth (which can be satisfied in some non-restrictive cases). We are glad to include more in our future versions. The quantity $\\mathbb{E}[\\mathrm{Var}(W^\\prime | W)]$ plays a critical role in the reduction of noises: if the hierarchical expectation is constructed by introducing some options that would never be chosen, then this conditional variance would be zero (since $W^\\prime = W$ almost surely). But it’s very unlikely that all the annotators would avoid a “tied” option for all the samples. And hence the conditional variance would always be non-zero, indicating a strict benefit of ordinal feedback.\n\nThe last question’s answer is: there are limitations on the range of $\\beta$’s. The feasible region of each $\\beta$ is $[0, 1]$, as it stands for the conditional probability. The underdetermined linear system’s solutions’ existence may be trivial in $\\mathbb{R}^{m \\times n}$, but it is possible to be infeasible in $[0, 1]^{m \\times n}$. For example, a binary feedback with $\\mu(0) = \\mu(1) = 0.5$ can never be a hierarchical expectation of a 3-level feedback with $\\mu(0.5) = 1$.\n\nFor the minor point about the significance of the improvement, an important point is that we are comparing different levels of feedback in the *same* amount of data. However, many practices are still throwing away all the “tied” samples (e.g., Llama 2/Llama3). We believe those thrown-away “tied” samples would benefit the learning process even within the same amount of data (Table 2), not to mention that including them would further increase the data volume.\n\nWe thank the reviewer again for the suggestions and for carefully reading our manuscripts. We look forward to further discussions in the following week.", "claims_and_evidence": "I'm particularly concerned about the following claims:  \n\n- The authors assert that Assumption 3.1 (wisdom of the crowd) is sufficient for obtaining meaningful comparisons. This is certainly not the case. I'll elaborate further when discussing the theoretical claims, but intuitively, one can always expand the set of possible feedback $\\mathcal{Z}$ by introducing new options that the user never selects. This expansion has no impact on anything and still satisfies Assumption 3.1. Similarly, I can construct a larger set of options that contain less information and still meet Assumption 3.1.  \n\n- Theorem 3.2 is initially presented to establish that there exist $\\mathcal{Z}$ and $z_{oracle}$ that satisfy Assumption 3.1. While this is true (and fairly straightforward), the authors later use the construction in this theorem to simulate ordinal feedback. I found no justification for this step, which I will further discuss under experimental design.  \n\n- A minor point, but I believe the authors are overinterpreting Theorem 4.6 and Corollary 4.7. What these results suggest is a possible reduction in Rademacher complexity when transitioning from binary to ordinal feedback. However, the extent of this reduction remains unknown; it could even be zero. I think it has never been in doubt that more granular feedback can make learning easier; the key question is by how much, and these theoretical results do not provide an answer.", "methods_and_evaluation_criteria": "I believe the choice of datasets, models, and evaluation criteria was appropriate.", "theoretical_claims": "I checked the main theoretical results, including Proposition 4.3, Corollary 4.4, Theorem 4.6, and Corollary 4.7. While the proofs appear sound, I have doubts about their usefulness.\n\nIn particular, the argument for hierarchical expectations in Proposition 4.3 raises concerns. To establish the existence of hierarchical expectations, we need to specify $m \\times n$ beta parameters. Condition (a) provides $2m$ equations, while Condition (b) contributes $(n-1)$ equations (since the $\\alpha'_k$ terms sum to one, and each row of $\\beta$ also sums to one, making one equation redundant). This results in at most $2m + n - 1$ equations. However, at least one of these equations is dependent—combining (a) and (b), we see that the expectation remains the same for both $Z$ and $Z'$. Consequently, there are at most $2m + n - 2$ independent linear constraints for $m \\times n$ parameters, implying that a suitable $\\beta$ can always be found for any $n \\geq 2$. I might be missing something here, but this suggests the result is somewhat trivial.\n\nA more fundamental issue with the theoretical results is that we can arbitrarily expand $\\mathcal{Z}$ by introducing new options that are chosen with zero probability. Such an expansion preserves Assumption 3.1 and implies that the size of $\\mathcal{Z}$ should not inherently matter in comparisons. Given this, I do not see how the authors justify their claim in Section 5.2 that 5-level feedback is necessarily better than 3-level feedback.", "experimental_designs_or_analyses": "My primary concern with the experimental results is the construction of ordinal feedback. As mentioned earlier, Theorem 3.2 establishes the existence of a feedback system, but I see no justification for using it to simulate ordinal feedback. In fact, I believe this is a poor choice, as users almost deterministically select one of the closest $z_i$s.\n\nA more reasonable approach would be to use an established ordinal feedback model. For instance, the Luce-Shepard model, which generalizes the Bradley-Terry model mentioned in the paper, seems like a natural alternative. It would be helpful to understand why the authors did not consider such a model for their simulations.\n\nI'm also a bit concerned that the results do not show a significant difference in domain. Maybe having more fine-grained feedback is not always as helpful as we thought.", "relation_to_broader_scientific_literature": "I believe the paper closely relates to soft labeling, and the authors have effectively clarified that connection.", "essential_references_not_discussed": "The authors have covered related work to the best of my knowledge.\n\nAs a minor point regarding references, I wouldn't call DPO and direct alignment methods RLHF, as the introduction calls them.", "other_strengths_and_weaknesses": "The question of whether more fine-grained feedback is necessary, and to what extent, has been on my mind for some time. I believe this paper addresses an important problem. However, I would have liked to see stronger justifications or a clearer quantification of the value of ordinal feedback, particularly through a more well-grounded model of human annotators.", "other_comments_or_suggestions": "Section 4.2 can be removed without changing the story.\n\nThere are multiple typos in the paper like Omega instead of $Omega$ or using $z$ instead of $Z$ when it is a random variable. I encourage the authors to do another pass on the writing.", "questions_for_authors": "Please discuss the weaknesses I mentioned. I'll update my score if I find your response convincing.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9guz6jZYY8", "reviewer": "Reviewer_AqRB", "summary": "The paper extends the canonical setup of binary feed-back RM to ordinal feedback.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for all the comments. Here are our responses to the questions.\n\n1. “I think more experimental results is needed to prove the superiority of the proposed method. For example, the authors could leverage the method in [1], setting up a seperated RM as the oracle and test the performance of the proposed method by evaluating the accuracy.”\n\nThanks for the reviewer’s kind reminder. Actually, that is exactly what we have done in our work: setting up a trained RM as the gold-standard model to simulate human preference data. We apologize for the confusion caused, and we’ll improve our presentation by emphasizing that point in our future versions.\n\n2. “The theory seems to be fascinating. However, the empirical superiority of the proposed method against the binary baseline seems not to be obvious due to Table 2 and Figure 2.”\n\nWe’re happy that the reviewer appreciates our theory. For the empirical effectiveness of our method, we want to make some clarifications. The motivation behind our work is that (some of) the current practice of RM training simply discards all those fine-grained labeled samples (e.g., “slightly better”), even though many companies have already launched fine-grained feedback systems for human annotators. This waste of precious human-annotated data motivates us to think about whether any better ways can be applied to use that part of the data than simply discarding it. As for Figure 2, our comparisons of different levels of feedback (binary, 3-level, 5-level, and oracle) are made under the *same* amount of data. However, we can use *more* data compared to the current throwing-away solutions. For example, if there are 80% of binary labels and 20% of fine-grained labels (e.g., “tied”), the current practice uses only those 80% of data, while our approach uses all of them. We further show that incorporating some “tied” samples doesn’t harm (and even benefits) the RM learning via Table 2. In a word, our approach makes improvements in two ways: decreasing the noise and increasing the effective data sample size.\n\nWe thank the reviewer again for the time, and we’re glad to refine our draft to those suggestions. Please let us know if there is any further questions arise in the following week; we will get back to you timely.", "claims_and_evidence": "See Other Strengths And Weaknesses.", "methods_and_evaluation_criteria": "I think more experimental results is needed to prove the superiority of the proposed method. For example, the authors could leverage the method in [1], setting up a seperated RM as the oracle and test the performance of the proposed method by evaluating the accuracy.\n\n[1] Gao L, Schulman J, Hilton J. Scaling laws for reward model overoptimization[C]//International Conference on Machine Learning. PMLR, 2023: 10835-10866.", "theoretical_claims": "I didn't check all the proofs in detail but the theorems provides seem to be reasonable and sounding.", "experimental_designs_or_analyses": "See Methods And Evaluation Criteria", "relation_to_broader_scientific_literature": "The paper provides a good vision for the current RM training.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The theory seems to be fascinating. However, the empirical superiority of the proposed method against the binary baseline seems not to be obvious due to Table 2 and Figure 2.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a practical, theoretically grounded, and easy-to-integrate framework for improving reward modeling by using ordinal feedback. While the experimental setup relies on simulation due to current data limitations, the justification is reasonable, and the results are clear.
Given its relevance to RLHF and LLM alignment, along with a well-supported rebuttal, this paper provides valuable insights for both theory and practice.
Decision: Accept (poster)