Beyond Binary Preferences: A Principled Framework for Reward Modeling with Ordinal Feedback
Authors: Amirhossein Afsharrad
Ruida Zhou
Luca Viano
Sanjay Lall
Mohammad Ghavamzadeh
Venue: ICLR 2026 Poster (2026)
Keywords: reward modeling
ordinal regression
Likert scale
preference learning
human feedback
RLHF
discrete ordinal regression
Bradley-Terry model
ordinal preferences
large language models
alignment
preference data
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We replace ad-hoc heuristics in reward modeling with a principled ordinal regression framework that properly models Likert scale preference data.

Abstract: Reward modeling is crucial for aligning large language models with human preferences, yet current approaches lack a principled mathematical framework for leveraging ordinal preference data. When human annotators provide graded preferences on a Likert scale (e.g., significantly better, better, slightly better, negligibly better), existing methods typically apply ad-hoc heuristics, such as margin terms or scaling factors, to loss functions derived from binary preference models like Bradley-Terry. These approaches lack an underlying mathematical model for how ordinal preference data is generated. We present a theoretically grounded framework that formulates reward modeling with Likert scale preferences as a discrete ordinal regression problem. We derive two loss functions from this formulation: a negative log-likelihood loss and an all-threshold loss, both of which learn threshold parameters that naturally capture the ordinal structure of preferences. Unlike existing heuristic methods that manually specify fixed margins or scaling weights, our approach learns these parameters directly from data within a coherent probabilistic framework. Experimental results on multiple benchmarks demonstrate that our ordinal regression approach consistently achieves competitive or superior performance compared to existing heuristic methods across diverse evaluation categories including chat, reasoning, and safety tasks. Our work provides the first principled mathematical framework for incorporating Likert scale preferences into reward model training, moving beyond ad-hoc modifications of binary preference models to enable more effective utilization of fine-grained human feedback.

Published as a conference paper at ICLR 2026
BEYOND BINARY PREFERENCES: A PRINCIPLED FRAMEWORK FOR REWARD MODELING WITH ORDINAL FEEDBACK
Amirhossein Afsharrad Ruida Zhou², Luca Viano', Sanjay Lall', Mohammad Ghavamzadeh* Sttanford University, 2 Amazon AGI, PEPFL, *Qualcomm Al Research, AAtuus Al
ABSTRACT
Reward modeling is erucial for aligning large language models with human pref- erences, yet current approaches lack a principled mathematical framework for leveraging ordinal preference data. When human annotators provide graded pref- erences on a Likert scale (e.g., significantly better, better, slightly better, negligibly better), existing methods typically apply ad-hoc heuristics, such as margin terms scaling factors, to loss functions derived from binary preference models like Bradley-Terry. These approaches lack an underlying mathematical model for how ordinal preference data is generated. We present theoretically grounded frame- work that formulates reward modeling with Likert scale preferences as a discrete ordinal regression problem. We derive two loss functions from this formulation: a negative log-likelihood loss and an all-threshold loss, both of which learn thresh- old parameters that naturally capture the ordinal structure of preferences. Unlike existing heuristic methods that manually specify fixed margins or scaling weights, our approach learns these parameters directly from data within a coherent prob- abilistic framework. Experimental results on multiple benchmarks demonstrate that our ordinal regression approach consistently achieves competitive or supe- rior performance compared to existing heuristic methods across diverse evaluation categories including chat, reasoning, and safety tasks. Our work provides the first principled mathematical framework for incorporating Likert scale preferences into reward model training, moving beyond ad-hoc modifications of binary preference models enable more effective utilization of fine-grained human feedback.
INTRODUCTION
Human feedback drives modern language model alignment. Through reinforcement learning from human feedback (RLHF) (Christiano al. Zielll et al.1
Stiennon al. 22020 Ouyang et al., 2022) and Direct Preference Optimization (DPO) style algorithms (Rafailov Azar et al. 2023 Ethayarajh et al. 2024 Xu et al. 2024 Hong et al., 2024 Meng 2024. models learn to generate outputs that humans prefer. Both approaches fundamentally
al., | Bai
2022
al.
2023:
et al.
rely on the Bradley-Terry model (Bradley and Terry 1952). which connects to human preferences through reward eunction-explicill in RLHF and implicitly in DPO. This model assumes that the probability of preferring one response over another increases monotonically with the difference between their reward values. Crucially, Bradley- Terry was designed for binary comparisons: sponse A is either preferred or dispreferred to response B. This binary foundation means that when preference data contains richer information, there is no systematic way to leverage it within these frameworks. Yet increasingly, preference datasets capture more than binary choices. When annota- tors compare responses, they often specify not just which one is better but by how muuhh-assignin ordinal ratings like "slightly better." "moderately better," or "significantly better" on Likert scales Wang et al., 2024c 2025). This ordinal structure contains substantially more signal than binary labels, but current methods, constrained by their binary foundations. largely ignore it.
"Corresponding author: a @fsharrad@stanforda

---

Published as a conference paper at ICLR 2026
Current attempts to leverage ordinal preferences mainly rely heuristics, such as adding margins between preference levels (Touvron et al. 2023), scaling losses by preference strength (Wang et al, 2024a), treating ordinal levels soft probability labels (Gunter al.) 2024 Liu al.] 2024a) These approaches share two fundamental problems. First, they lack any mathematical model of how humans assign ordinal preferences- they simply modify the binary Bradley-Term loss with ad-hoc terms chosen through intuition rather than principle. Second, they require practitioners to manually tune ayper-parameters that have no clear interpretation: what margin should separate "slightly bet- ter" from "significantly better"? Should strong preference contribute twice or thrice the gradient of weak one? Moreover, these hyper-paan parametrssssssss must be retuned whenever the number or definition of preference levels changes, making the approaches brittle and dataset-specific.
We reframe reward modeling with ordinal feedback as a discrete ordinal regression problem well-established statistical framework for ordered categorical data (McCullagh 1980; Winship and
Mare 1984). The machine learning community has developed extensive methodology for such problems (Chu and Keerthi 2005 2007 Shashua and Levin 2002: Rennie and Srebro 2005 Srebro ct al. 2004). yet this machinery has not been applied to preference learning. Our approach models the relationship between reward differences and preference levels through learned thresholds that partition the continuous reward space. Rather than manually specifying margins or scaling factors, derive principled loss functions both probabilistic based cumulative link models and margin- based inspired by structured prediction- that learn all parameters directly from data.
Our work demonstrates that treating preferences inherently ordinal yields consistent improve- ments across benchmarks and architectures. Beyond standard binary accuracy metrics, our meth- ods accurately predict preference strength, achieving approximately 85% accuracy within one ordinal level on held-out data. The learned threshold structures provide interpretable insights into how annotators distinguish between preference levels, revealing patterns that manual hyper- parameter tuning could never discover. As preference collection evolves toward richer anno- tation schemes-confideen scores, multi-aspect ratings, and other forms of structured feed- back-principled frameworks become essential. Our ordinal regression approach provides a mathe- matical foundation that naturally extends to handle these diverse forms of feedback. The core insight is simple: human preferences often carry ordinal information, and it would be much better if our methods leverage this structure rather than forcing it through binary models with heuristic patches.
2 PRELIMINARIES
Notations. For any positive integer K, we denote the set ......,,, by [K] and the set {-K, + 1,...,..1 by [-K]. Throughout this paper, we use a(t) I+e-r to denote the sigmoid (logistic) function.
2.1 DISCRETE ORDINAL REGRESSION
Ordinal regression addresses prediction problems where the target variable takes values from a dis- crete ordered set. Unlike standard classification, where categories unrelated, or regression, where outputs are continuous, ordinal regression must respect the inherent ordering among categories while maintaining their discrete nature. Common applications include rating systems (1-5 stars), survey responses (strongly disagree to strongly agree), and severity levels (mild, moderate, severe).
Given a training dataset an ordinal label, the goal is to learn a model that predicts the correct ordinal level for new inputs. Most ordinal regression methods model the ordinal structure through a latent continuous variable that is subsequently discretized. Specifically, these methods learn scoring function 85 Rd + R parameterized by Φ, which maps inputs to a oinedimmnsional latent space, along with a set of ordered thresholds x Co Ç1 intervals. The thresholds create a correspondence between latent scores and ordinal levels: higher scores correspond to higher ordinal categories. This framework reduces to binary classification when K = 2, with a single threshold (typically zero) separating the two classes.
=1)) where x, E Rd is feature vector and 2 [K] is
(K-1 (K too that partition the latent space into K
Given this latent variable framework, there are two fundamentally different approaches to learning the parameters (o. {): probabilistic modeling via generalized linear models and direct loss mini- mization through margin- based methods.
2

---

Published as a conference paper at ICLR 2026
Probabilistic Modeling via Generalized Linear Models (GLMs). The probabilistic approach explicitly models the conditional distribution P(z|z) through cumulative probabilities. The ordinal regression model assumes
P(2 S k x)) = I(Sx - =x())),
(1)
where IR - [0, 1] is cumulative distribution function. The most common choice is f(1) ((l).
yielding the ordered logit model. An alternative is f(1) = (t) (the standard normal CDF), yielding
the ordered probit model. This cumulative model naturally ensures that P(z S k) 2 P[ 1)
for all k. Thus, the probability of observing exactly level k is obtained by
P(z k x) f(A sa(x)) f( -g(2))
For the ordered logit model with f = maximum likelihood estimation leads to minimizing the negative oog-likelihood
LNLIL S: log [o(S= - sa(()) -(--1-))(-) sx))))]
-
where we have simplified the notation slightly by dropping the explicit dependence on the loss function. This loss penalizes the model for assigning low probability mass to the interval containing the true label. Note that this probabilistic formulation is specific to the GLM approach and provides complete model for P(z[xr).
Margin-Based Direct Loss Minimization. As an alternative to probabilistic modeling, Rennie and Srebro (2005) proposed methods that directly penalize threshold violations without explicitly modeling probabilities. These approaches, inspired by large-margin methods in machine learning, focus on ensuring the latent score falls within the correct interval. The framework uses margin penalty function
IR + R4 where g()) measures the cost of violating a margin constraint by amount u. Common choices of penalty functions include logistic g(u)
log (u), hinge g(u)
max(0, 0, - u), and exponential g(u) Immediate-Threthh (IT) Loss only considers the boundaries of the target interval,
Two loss functions emerge from this framework. The
LT ( Ci = ((5((x)) (=-1) + g(C= - -8((2))
(4)
This loss encourages 8 to within (C--1, 1, Cz) but treats all misclassifications equally, regardless of severity. The All-Threshold (AT) Loss accumulates penalties from all threshold violations,
K E g(v((=)) (Gi so()))), 1=1
(5)
where 2(1;2)= =1 < and v((;))= +1if z. This formulation ensures that we penalize (z(x) Gr when the true label and penalize (x) > & when < 1. The AT loss increases with the degree of misclassification, making it more sensitive to large errors than the IT loss. For the logistic margin penalty g(u) log (u), these losses can be written as
Lir(so,(i2) log o (x) Cc-1) - log (Se sz(x)),
(6)
K-1 LAT(So,Gi2) Σ
log a (u(l:2) (Ci (GI- (x()))
The key distinction between these two approaches is that the probabilistic method models the full conditional distribution [P(2[=) and uses maximum likelihood estimation, while the margin-based methods focus correct classification through direct optimization without probabilistic interpreta- tion. The margin-bant based losses are often computationally simpler, particularly for large-scale prob- lems, noted by Rennie and Srebro (2005) who observed that "even just evaluating the likelihood of a predictor is not straight-forwadd in ordered logit and probit models.
In subsequent sections, will employ both the NLL loss from the probabilistic approach (Eq, 3) and the AT loss from the margin- based approach (Eq. 7). We exclude the IT loss since the empirical results in Rennie and Srebro (005) demonstrate its inferior performance compared to the AT loss.
3

---

Published as a conference paper at ICLR 2026
2.2 REWARD MODELING
Reward modeling followed by policy optimization using deep reinforcement learning algorithm is main approach post-training LLMs, often referred to RLHE, A reward model (RM) trained on dataset of comparisons between two responses (y. :/) to the same input prompt x. Learning an RM consists of training a binary classifier to discriminate between the preferred and dispreferred responses using a logistic regression loss. A popular choice for the classifier is the Bradley- Terry (BT) model. The BT model stipulates that the human preference distribution p* can be written as
p" (y s'(x)= a (r"" (x,y) (x, ((),))))
where r" (ar, y) the (latent) reward function of the annotator for prompt x and response y-
Under the BT preference model and given training set 'D = {xa, Uur. Ut, } h= in which yue and y denote preferred and dispreferred responses for the prompt x, one can learn the RM by minimizing the negative log-likelihood loss
Lum(r D) = - (m [log log a (1
yw) (w)
This loss encourages the reward model to assign higher scores to preferred responses compared to dispreferred ones. The learned reward model ra can then be used provide reward signals for RL algorithms that optimize the language model policy. While we focus on reward modeling in this work, the techniques we develop can be adapted to Direct Preference Optimization (DPO) (Rafailov et al. 2023) style algorithms, as we will discuss in Appendix
3
ORDINAL REGRESSION FOR REWARD MODELING
In this section, we first formulate the problem of reward modeling using ordinal preference data. We then provide brief overview of the existing heuristic approaches to this problem. Finally, present our approach which is base on the ordinal regression framework and present our algorithms for learning reward model in this setting.
3.1 PROBLEM FORMULATION
We consider the problem of learning reward models (RMs) from preference data with ordinal feed- back. Unlike standard preference learning where annotators provide binary comparisons, we con- sider the more informative setting where annotators indicate not only which response is preferred but also the degree of their preference. More specifically, for each input prompt and two candidate responses y and y', the annotator provides feedback on Likert scale indicating how much better one response compared
the other. We denote y k y' to indicate that y is compared to y at level k, where & E [-K] U {0) U [K]. When l E [K]. this means y is preferred to y' at level k, with higher values indicating stronger preference. When
[-K], this means y dispreferred relative to y' at level |k]. The case y ro indicates that the annotator finds the two responses approximately equal or is unsure which one is better. This results a dataset of the form D {(x,, Yr, y/ z, H where each tpple contains prompt, two responses, and the ordinal preference such that yi >,, V/. In practice (e.g., Wang et al. (024b)), annotators might use labels such as "sig- nificantly better", "better", "slightly better", and "negligibly better", which correspond to different values of |z depending on which response is preferred.
(O)
The goal is D and learn reward model ",( y) parameterized by that can score individual responses in a way that respects the ordinal preference structure. Following the BT model, the natural predictor for this problem takes the form of reward differences.
s0 The challenge is learn Ta such that the reward difference ss(x, y, v') not only has the correct sign (indicating which response is preferred) but also the correct magnitude (reflecting the degree of preference expressed by z).
ra(x y) (=(
(10)
3.2 EXISTING HEURISTIC A APPROACHES AND THEIR LIMITATIONS
Current approaches to learning from ordinal preference data modify the standard BT loss in various ad-hoc ways incorporate the preference strength information.
4

---

Published as a conference paper at ICLR 2026
Margin Brarlyy-Terry. Proposed in Llama-2 (Tovrron et al. 2023), this approach adds margin term to the standard BT loss,
L (T6) U. V 2) log (ro(2, y) ro (x, v/) m()))
(11)
where the margin m {- K....,KK For instance, TTourron al. (2023) experimented with m(k) (3, 2, 1,0 for k (significantly better, better, slightly better, negligibly better).
K} + IR is a manually specified function of the preference rating.
Scaled Bradley-Terry. In HelpSteer2, Wang et al. (0024)) proposed scaling the loss itself by the preference strength,
LSBr(ra) 2, U',,', = mm()) log o (ro(1, -ro(z))))))
(12)
where m(k) 23) for different preference levels.
Soft Label.
Gunter et al. 2024 interpreted ordinal feedback as soft labels,
LSL (SL = p(2) log o (ro(2, y) (x. (1 - p(2)) logg a (rø(z, v/) y)
(13)
where p(=) represents an arbitrary probability assignment, such as p(k) = (0.95, 0.85, 0.75, 0.65)
for different preference levels.
These approaches share two fundamental limitations. First, they lack a coherent mathematical model of ordinal preferences. While the BT model provides principled probabilistic framework for binary preferences, i.e., p(y > y ()) (r(2, y) -r(x, v)), these extensions have no corresponding model that explains how humans assign ordinal labels. The modifications the loss function are arbitrary and it is unclear what assumptions about human preference behavior would lead to these particular formulations. Second, they require manual specification of their hyper-pare prper-parameters m(-) or p(-) with no principled method for choosing their values learning them from data. In contrast, ordinal regression provides a well-established statistical framework where the loss functions arise naturally from explicit modeling assumptions and all parameters are learned from data.
3.3 A PRINCIPLED ORDINAL REGRESSION FRAMEWORK
We propose formulating reward modeling with ordinal feedback as discrete ordinal regression problem. Unlike the heuristic approaches, ordinal regression provides principled methods where the relationship between reward differences and preference levels is learned from data rather than manually specified. To extend ordinal regression to preference learning, we introduce 2K thresholds S-K...., 51, ÇK that partition the real line into 2K + segments. For convenience, denote (-(++1) = -o and (K+1 = +00. Note that we deliberately exclude Ço: the segment between 5-1 and Ç1 corresponds the preference level 0 (approximately equal responses).
Given that we are modeling preferences, the predictor values for our ordinal regression problem take the form described in (10). These thresholds serve different roles depending the model- ing paradigm we adopt. As we describe in the following sections, the probabilistic approach uses thresholds define cumulative probabilities for different preference levels, while the margin-based approach uses them as boundaries that the predictor should respect. Both approaches learn the thresholds from data, eliminating the arbitrary hyper- syper-parameter choices required by the heuristic methods. The key insight is that ordinal regression provides established frameworks-bott statisti- cal and optimization based for learning from ordered categorical data. We adapt these frameworks to the specific structure of preference learning, where the ordinal variable represents the degree and direction of preference between two responses.
3.3.1 Loss FUNCTIONS
We adapt two approaches from discrete ordinal regression our preference learning setting, each representing a different paradigm for utilizing ordinal information.
Probabilistic Approach: Negative Log- Log-Likelihood. The probabilistic approach assumes that hu- man annotators follow ordered logit model when assigning preference levels. Given reward dif-
5

---

Published as a conference paper at ICLR 2026
ference s1 v.). the probability of observing preference level is modeled
[((2 -,(0, 1
if [-K], if [K],
ply - 1z)=
11 o({ -
(14)
((5 - (x,,,/)())) (00 y. y,
if =0
This formulation is a direct application of the ordered logit model from 2 to our preference learning setting, where the predictor value the reward difference 80(r, y, The model ensures that probabilities sum to one across all preference levels.
Maximizing the likelihood of the observed preferences leads the negative log-likelihood loss,
CNLL 3, = - log > y [z),
(15)
where ply > y/ (x) is given by (4) This loss has a clear interpretation: it penalizes the model for assigning low probability to the observed preference level.
Margin- -aaed Approach (All-Threshold Loss). The margin-based approach does not assume specific probabilistic model of human behavior. Instead, inspired by large-margin methods in machine learning, directly penalizes violations of the ordinal structure. The AT loss accumulates penalties for the reward difference being on the wrong side of each threshold,
Cxr(ra Siz, Vi
Σ
log (v(l;2)- (Ci =s(=,,,)1)))
(16)
LEI [Ei-K]-[K]
where =(1;2) = ifl < 2 and v(l; 2) +1 if 2 2. This formulation encourages s s2(2, y, y²) to be greater than all thresholds & with 1 < and less than all thresholds G with 2 2, with logarithmic penalties for violations.
Regularization for Stable Optimization. The simultaneous learning of reward parameters & and thresholds introduces fundamental challenge in optimization. As we formalize below, without regularization, the optimization problem admits unbounded solutions.
Theorem 3.1 (Unbounded Solution Set). Consider either the NLL loss LNIs, or the AT loss LAT without regularization. Suppose there exists solution (r);() such that all training examples are correctly ordered, i.e., st E (1 for all (zi, yes V/s 2.) D. where st = (3 Ws) (z,, v/). Then for any e >>> 1, the scaled solution o(") achieves strictly lower loss, and thus, Tim E; ((rr", e: ; Di, V/y 24) = O. Consequently, the unregularized optimization problem has no finite optimal solution.
The proof, reported in Appendix B shows that correctly classified examples contribute vanishing loss as the scale increases, while misclassified examples incur linearly growing penalties. This creates optimization pathologies: gradient descent can drive thresholds arbitrarily large values while maintaining or reducing the loss, leading to numerical instability and poor model calibration.
To address this issue, we add an Iæ-regularization term on the thresholds, yielding the following regularized objective:
min OCEEC
Σ
Clro,Ci Siz, *
(17)
where C= {C E R2K, S-K <... <...<<<.... <... < CK) and can be either ENLL or CAT- The regularization term prevents unbounded growth of thresholds and ensures the existence of a finite optimal solution.
3.3.2 SYMMETRIC AND ASYMMETRIC MODELS
An important modeling choice in framework concerns whether impose symmetry constraints on the thresholds. This choice is orthogonal to the selection of loss function and can be applicd with either the probabilistic margin- based approach.

---

Published as a conference paper at ICLR 2026
Symmetric Model. A natural assumption is that the strength of preference should be symmetric: preferring y over y at level should correspond to the same strength as preferring y over y at level kk. Formally, this translates to imposing the constraint Ç-A - Vk E [K]. Under the probabilistic model, this symmetry constraint has the following theoretical justification.
Theorem 3.2 (Threshold Symmetry under Ordered Logit), Consider the ordered logit model in (14)
If the preference data satisfies the symmetry property that for all E [K] and all reward differences E R,
P(y > s/ s0(2, v. =) P(y' - y so .9 9.9)) = =T),
(18)
then the thresholds must satisfy -c all & [K].
The proof, reported in Appendix C shows that the symmetry in human preferences necessarily im- plies symmetry in the thresholds. While this theorem provides theoretical justification under the probabilistic model, the symmetry constraint can also be imposed with the margin-based approach a reasonable structural assumption. The symmetric model reduces the number of threshold pa- rameters from 2K to K, improving computational efficiency and reducing overfitting risk.
Asymmetric Model. Human preferences often exhibit asymmetries due various cognitive bi- ases. For instance, annotators might be more conservative when strongly endorsing one response over another (high positive k) than when strongly criticizing (high negative [k]). Psychological literature has documented numerous such asymmetries, including loss aversion, negativity bias, and framing effects, all of which can manifest in preference annotations. The asymmetric model accom- modates such patterns by learning all 2 thresholds independently from data, without imposing any relationship between Ç-x and Ça- This provides maximum flexibility to capture the actual structure of human preferences reflected in the training data.
The choice between symmetric and asymmetric models represents bias-variance trade-off: the symmetric model has fewer parameters and stronger inductive bias, while the asymmetric model has more flexibility to capture nuanced patterns in human annotations. our experiments, we evaluate both variants and find that the symmetry assumption is reasonable in practice- the symmetric model often outperforms the asymmetric variant.
3.3.3 OPTIMIZATION VIA REPARAMETERIZAA
To handle the ordering constraints in 17 reparameterize the thresholds using monotonic transformation. For the asymmetric model, we parameterize
Ç-K ap and
Ç4 Ça-1 + exp(ax), vk e [-K + (
(19) The exponential mapping ensures strict positivity of
where ag R and ag E R for all k increments, maintaining Ç& > S by construction. For the symmetric model, we only parameterize the positive thresholds starting from Çi = exp(a1), with negative thresholds determined by Ç- = -Sa, reducing the number of parameters from 2K to K.
This reparameterization transforms the constrained optimization into an unconstrained problem over (ó, ar), solvable via standard gradient-based methods. In our implementation, we update reward and threshold parameters simultaneously at each iteration (see Appendix for the detailed training algo- rithm). Alternative optimization strategies, including projected gradient descent and asynchronous updates, are discussed in Appendix
EXPERIMENTS
We evaluate ordinal regression framework real preference datasets with Likert-scale anno- tations, comparing against heuristic baselines across multiple model architectures and examining trade-offs between symmetric and asymmetric threshold formulations.
4.1 EXPERIMENTAL SETUP
Base Models. We evaluate our approach on three instruction-tume language models: Llama-3.11 8B Meta AI 2024). Mistral-7B (Jang et al.) 2023), and Zephyr-7B (Tunstall et al., (023). Each model uses its supervised fine-tuned checkpoint as initialization, with the language modeling head replaced by single-layer reward head outputting scalar rewards.
7

---

Published as a conference paper at ICLR 2026
Datasets. We train HelpSteer2 Waag al. 2024c) and HelpSteer3 (Wag et al. 2025 which provide pairwise comparisons with 7-level preference strength labels from (significanlly worse) to +3 (significantly better). Both datasets contain fine-grained human annotations suitable for eval- uating ordinal regression methods.
Methods. We evaluate three instantiations our ordinal regression framework from Section 3 (1) NLL-Smmmetri, using the negative log-likelihood loss (Eq. [5) with symmetric threshold con- straints Ç-x = -Sa: (2) NLL-Asymmetric, using the same NLL loss but learning all 2K thresh- olds independently; and (3) All-Threshold, using the margin-based all-threshold loss (Eq. [6) with asymmetric thresholds. We compare against three baseline heuristics: Margin BT, Scaled BT, and Soft Label. For ordinal methods, jointly optimize reward and threshold parameters with 1.2 regularization on thresholds to ensure convergence (see Appendix for empirical validation).
Training and Evaluation. All models are trained for epochs with effective batch size 64 using Fully Sharded Data Parallel across 8 Us (H100 or H200). We evaluate RewardBench (Lambert) ct al. 2024) and RM- R.M-eench (Liu et al. 2024b). Comprehensive hyperparameter search details and infrastructure specifications are provided in Appendix
4.2 MAIN RESULTS
Tables and 2 present our evaluation results. Across both benchmarks, the performance of our ordi- nal regression methods consistently match or exceed the baselines, with NLL-Symmetric emerging as our most effective approach. This method achieves the highest average scores in the majority of configurations, outperforming all baselines by 2-5% on average. The consistent superiority of NLL- Symmetric over NIL.-Asymmetric validates our theoretical analysis in Theorem 3.2, suggesting that human preferences indeed exhibit the symmetry property assumed in our formulation.
The advantage of principled threshold learning becomes apparent when examining convergence be- havior. As shown in Appendix G our learned thresholds converge to stable values under appropriate regularization, with the choice of offering a trade-off between convergence speed and flexibility. Without such regularization, thresholds exhibit unbounded growth as predicted by Theorem highlighting the importance of our theoretical framework in ensuring stable optimization.
Beyond binary preference accuracy, our ordinal methods demonstrate genuine understanding of pref- erence strength. Appendix H shows that NLL-Symmetric achieves approximately 55% exact accu- racy and 85% accuracy within ordinal level validation data, confirming that approach learns meaningful ordinal structure rather than merely preserving rankings. While All-Threshold generally performs below the NLL variants, it still outperforms baselines in several cases, indicat- ing that both probabilistic and margin-based formulations improve upon ad-hoc heuristics. In the following section, we investigate the benefits of our ordinal framework beyond standard ranking ac- curacy, examining error severity, the necessity of joint training, and robustness to annotation noise.
4.3 ANALYSIS BEYOND BINARY ACCURACY
Standard binary accuracy treats all errors equally regardless of severity and correct predictions equally regardless of calibration. We now examine three aspects of model quality that this metric misses.
Error Severity. When a reward model errs, the magnitude of the misranking matters: model that is narrowly wrong on ambiguous cases is far preferable to one that confidently assigns high reward to dispreferred responses. We compare error margin distributions of standard Bradley-Terry (Simple BT) and NLL-Symmetric, both trained on HelpSteer2 with Llama-3. 1-BB and evaluated on RewardBench. The ordinal approach reduces the number of errors by 35% (282 vs. 433), but the improvement in error severity is far more dramatic: the mean error margin drops from 3.827 to 0,501, a reduction of 87%. Simple BT produces errors with margins as large as 20, while NLL Ordinal Symmetric produces none exceeding 2.5. This means that when our model errs, it does so on genuinely ambiguous cases with low confidence, rather than making bigh-confidence mis- rankings- critical property for downstream RL, where confidently incorrect rewards can severely misguide policy optimization. See Appendix [forr the full distributional analysis.
8

---

Published as a conference paper at ICLR 2026
Table I: RM-Bench evaluation results. Best results within each dataset model group and column are in bold, second best are underlined.
Dataset
Model
Method
Chat
Math
Code
Safety
Hard
Normal
Easy
Total
Margin BT
0.662
0.563
0.521
0.885
0.363
0.701
0.908
0.657
Scaled BT
0.636
0.552
0.497
0.798
0.389
0.640
0.832
0.621
Soft Label
0.571
0,550
0.505
0.727
0.225
0.638
0,902
0.588
Llama
NLL--ym 0.638
0.550
0.508
0.907
0.487
0.668
0.798
0.651
NLL-Asym 0,574
0.544
0.523
0.823
0.397
0.631
0.819
0.616
All-Thresh 0.688
0.566
0.519
0.860
0.421
0.681
0.874
0.658
HelpSteer2
Margin BT
0.634
0.539
0.496
0.760
0.317
0.643
0.862
0.607
Scaled BT
0.613
0.548
0,520
0.837
0,439
0.654
0.795
0,630
Soft Label
0.525
0.501
0.491
0.623
0.265
0.537
0,803
0.535
Mistral
NLL-Sym 0.654
0.538
0.540
0.858
0.406
0.673
0.864
0.647
ALym-Asym 0.584
0.539
0.483
0.716
0,280
0.617
0.845
0.580
All-Thresh 0.619
0.519
0.493
0.775
0.315
0.630
0.859
0.601
Margin BT
0.615
0.519
0.511
0.856
0,392
0.642
0.841
0.625
Scaled BT
0.621
0.534
0.501
0.762
0.327
0.637
0.850
0.605
Soft Label
0.635
0,531
0,504
0.840
0,349
0.658
0.875
0.627
Zephyr
NLL-Sym 0.661
0.533
0.519
0.835
0.399
0.667
0.846
0,637
NsL-Asym 0.604
0.530
0.509
0.650
0.230
0.617
0,873
0,573
All-Thresh
0.628
0.534
0.498
0.796
0.329
0.638
0.875
0.614
Margin BT
0.717
0.587
0.532
0.765
0.390
0,681
0,880
0.650
Scaled BT
0.742
0.592
0.534
0.823
0.476
0.697
0.844
0.673
Soft Label
0.708
0.578
0.536
0.862
0.421
0.698
0.894
0.671
Llama
NLL-Sym 0.695
0.586
0.541
0.881
0.413
0.713
0,802
0.676
NLL-Asym 0.684
0.587
0.569
0.815
0.539
0.683
0.776
0.666
All-Thresh 0.663
0.573
0.540
0.815
0.526
0.663
0.755
0.648
HelpSteer3
Margin BT
0.700
0.555
0.506
0.718
0.414
0.636
0.809
0.620
Scaled BT
0.692
0.551
0.501
0.593
0.361
0.597
0.794
0.584
Soft Label
0.643
0.556
0.494
0.638
0.474
0.589
0.686
0.583
Mistral
NLL-Sym 0.638
0.535
0.504
0.725
0.294
0.631
0.878
0.601
NLL-Asym 0.622
0.580
0.522
0.706
0.392
0.633
0.798
0.608
All-Thresh 0.621
0.560
0.523
0.555
0.255
0.599
0.841
0.565
Margin BT
0.668
0.559
0.532
0.678
0.386
0.629
0.813
0.609
Scaled BT
0.694
0.551
0.492
0.693
0.355
0.631
0.837
0.607
Soft Label 0.653
0.564
0.525
0.605
0.318
0.618
0.825
0.587
Zephyr
NLL-Sym 0.684
0.548
0.502
0.727
0.357
0.640
0.847
0.615
NLL-Asym 0.643
0.554
0.336
0.519
0.303
0.569
0.817
0.563
All-Thresh 0.628
0.552
0.521
0.513
0.297
0.570
0.794
0.553
Table 2: RewardBench evaluation results for models trained on HelpSteer2 (left) and HelpSteer3 (right). Best results per metric are in bold, second best are underlined. HelpSteer2
HelpSteer3
Model
Safety
Reason
Avg
Model
Method
Safety
Reason
Avg
Margin BT
0.961
0.660
0.885
0.703
0.802
Margin BT
0.947
0.700
0.780
0.656
0.771
Scaled
0.927
0.638
Scaled BT
0.944
0.793
0.638
0.78L
Liama
Soft Label
0.939
0.581
0.689
0.680
0.722
Soft Label
0.922
0.728
0.823
0.747
0.805
NLL-Sym
0.94]
0.728
0.897
0.804
0.843
NLL-Sym
0.947
0.765
0.808
0.707
0.807
NLL-Asym
0.911
0.695
NLL-Asym
0.936
0.785
0.611
0.765
All-Thresh
0.922
0.689
0.872
0.798
0.820
All-Thresh
0.872
0.728
0,780
0.676
0.764
Margin
0.933
0.725
Margin
0.700
0.588
0.727
Mistral
Soft Label
Scaled BT
0.905
0.930
0.625
0.660
0.569
0.824
0.776
0.787
0.694
0.800
Mistal
Soñt Label
Scaled BT
0.791
0.908
0.700
0.724
0.653
0.687
0.654
0.704
0.756
0.699
NLL-Sym
0.939
0.684
NLL-Sym
0.927
0.695
0.668
0.749
NLL-Asym
0.919
0.629
0.819
0.699
0.767
NLL-Asym
0.858
0.627
0.685
0,616
0.704
Alr-Thresh 0.897
0.680
0,730
0.827
0.783
All-
0.925
0.568
0.603
0.651
0.687
Margin BT
0.922
0.700
0.857
0.593
0.768
Margin BT
0.911
0.660
0.643
0.612
0.707
Zephyr
Soft Label
Scaled BT
0.935
0.944
0.660)
0.665
0.819
0.727
0.786
0.815
0,807
0.781
Zephyr
Suñt Label
Scaled BT
0.908
0.858
0.685
0.634
0.642
0.723
0.673
0.566
0.700
0.730
NLJ-Sym
0.953
0.603
0.761
0.756
0.768
NLL-Syy
0.913
0.723
0.753
0.668
0.764
NLL-Asym
0.911
0.662
0.680
0.856
0.777
NLL-Asym
0.916
0.640
0.574
0.611
0.685
All-Thresh
0.933
0.662
0.751
0.821
0.792
Ahr-Thresh
0.925
0.600
0.557
0.615
0.674
9

---

Published as a conference paper at ICLR 2026
Table 3:
Ordinal prediction HelpSteer2 test set (448 examples). Joint training substantially
outperforms post-hoc calibration of baseline reward models across all metrics.
Method
MAE
Acc@0
Acc@1 1
Acc@2
Scaled BT (post-hoc)
2.667
10.9%
37.1%
44.0%
Margin BT (post-hoc)
2.181
16.1%
48.7%
59.6%
Soft Label (post-hoc)
1.725
12.9%
47.8%
78.1%
ALmmmeymttric (joint)
1.060
29.7%
72.5%
92.9%
Joint Training vs. Post-Hoc Calibration. A natural question is whether our framework's bene- fits could be replicated by simply fitting thresholds standard reward model after training. We train baseline models (Margin BT, Scaled BT, Soft Label) on HelpSteer2 with Llama-3.1-88 then perform post-hoc calibration by learning thresholds on frozen reward differences using the same or- dinal NLL objectíve. Table
these against jointly trained NLL-Symmetric on the held-out test set. Joint training achieves 38% lower MAE (1.060 vs. 1.725) and nearly doubles exact ordinal accuracy (29.7% vs. 1)). The Acc@1 gap is especially notable: our method predicts prefer- ence strength within one level for 72.5% of examples, versus fewer than half for the best baseline. Post-hoc calibration cannot recover the fine- fine-grained structure that joint optimization discovers. See Appendix H for the full procedure and training-set results.
Robustness to Label Noise. We evaluate robustness under two noise models corruption rates from 25% to 100%: systematic shift noise, where labels are perturbed by one ordinal level (modeling calibration errors), and random label noise, where labels are replaced uniformly random (model- ing catastrophic failures). Under systematic shift noise, performance is essentially unchanged even 100% corruption (RewardBench average: 0.808 0.846 vs. 0.843 baseline), as the learned thresh- olds absorb systematic biases. Under random noise, degradation is graceful: at 25% corruption performance matches the clean baseline, and meaningful learning persists through 50% corruption. Full results including per-category breakdowns and training dynamics are in Appendix
CONCLUSIONS
We presented the first principled framework for reward modeling with ordinal preference data, mov- ing beyond ad-hoc modifications of binary preference models. By formulating the problem as dis- crete ordinal regression, we derived theoretically grounded loss functions that learn threshold pa- rameters directly from data, climinating the need for manually specified margins or scaling factors. Our theoretical analysis revealed fundamental properties of the optimization landscape, including the necessity of regularization for bounded solutions and proof that symmetric thresholds arise naturally when human preferences exhibit symmetry. Empirical results across multiple benchmarks and model architectures demonstrate that our approach consistently achieves competitive or superior performance compared to heuristic baselines, with the symmetric NLL formulation proving partic- ularly effcetive-vvaidattiie both theoretical framework and suggesting that human preferences indeed exhibit the symmetry property. Beyond ranking accuracy, our ordinal framework yields substantially hetter-calibrated models, reducing Bradley-Tere Terry, and these benefits cannot be replicated by post-hoc threshold fitting. The frame- work also exhibits strong robustness realistic annotation noise. Future work could extend this framework to DPO as outlined in Appendix A handle more complex preference structures such as multi-aspect ratings or pairwise comparisons with uncertainty estimates, and adapt to evolving data collection methodologies. As the field progresses toward more sophisticated annotation schemes that capture nuanced human feedback beyond binary preferences, principled methods for leveraging these richer signals will become increasingly critical for effective language model alignment.
enror severity by 87% compared to standard
10

---

Published as a conference paper at ICLR 2026
Reproducibility Statement. We have made extensive efforts to ensure the reproducibility of our work. Complete proofs for all theoretical results (Theorems 3.1 and 3.2) are provided in Appen- dices and C with clear statements of assumptions and detailed derivations. The full training algorithm for our ordinal regression approach is detailed Appendix D with alternative optimiza- tion strategies discussed in Appendix [ Comprehensive experimental details provided in Ap- pendix including: complete model and data configurations (Section F.I. exhaustive hyperparam- eter search spaces and final selections (Section F.3). and infrastructure specifications including GPU requirements and implementation details (Section G.1 All experiments use fixed random seeds and consistent preprocessing pipelines as described in Appendix Data processing steps, including tokenization limits, padding strategies, and prompt truncation procedures, are fully specified in Sec- tion F The convergence behavior our learned thresholds empirically validated in Appendix G and additional ordinal performance metrics are reported in Appendix H Error margin analysis and noise robustness experiments are detailed in Appendices [ and respectively. Our implementation builds on the open-source TRL library with extensions for ordinal regression methods. As stated in Section G.I all model checkpoints, configuration files, and code will be released upon publication to facilitate reproduction and extension of our work.
REFERENCES
Amirabbas Afzali, Amirhossein Afsharrad, Seyed Shahabeddin Mousavi, and Sanjay Lall. One goal, many challenges: Robust preference optimization amid content-aware, multi-source noise. Reinforcement Learning Journal, 6:2485-25011 2025.
Shayan Alipour, Shruti Phadke, Seyed Shahabeddin Mousavi, Amirhossein Afsharrad, Morteza Zi- hayat, and Mattia Samory. The gray area: Characterizing moderator disagreement on reddit. arXiv preprint w:Xiv:2601.01600 2026.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310.1203: 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Paul FF Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing sys- tems, 30, 2017.
Wei Chu and S Sathiya Keerthi. New approaches support vector ordinal regression. Proceed- ings of the 22nd international conference on Machine learning, pages 145-152, 2005.
Wei Chu and S Sathiya Keerthi. Support vector ordinal regression. Neural computation, 19(3): 792-815, 2007.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024.
Tom Gunter, Zirui Wang, Chong Wang, Ruoming Pang, Andy Narayanan, Aonan Zhang, Bowen Zhang, Chen Chen, Chung-Cheng Cheng Chiu, David Qiu, et al. Apple intelligence foundation language models. arxiv preprint arXiv:240..21075. 2024.
Jiwoo Hong, Noah Lee, and James Thore. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07661 2024.
Shawn Im and Sharon Li. How well can preference optimization generalize under noisy feedback? arXiv preprint 2rXiv225/00044 2025.
11

---

Published as a conference paper at ICLR 2026
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825) 2023.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A Smith, and Hannaneh Hajishirzi. Re- wardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403. 13787, 2024.
Shang Liu, Yu Pan, Guanting Chen, and Xiaocheng Li. Reward modeling with ordinal feedback: Wisdom of the crowd. arXiv preprint arXiv:2411.128.. 2024a.
Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. Rm-bench: Benchmarking reward models of language models with subtlety and style. arXiv preprint arXiv:2410.16184 2024b.
Peter McCullagh. Regression models for ordinal data. Journal of the Royal Statistical Society: Series B (Methodological), 42(2):109-127, 1980.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Advances in Neural Information Processing Systems, 111124198-1222555 2024.
Meta AL. The llama herd of models. arXiv preprint arXiv:2407.21783 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730 27744, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
J. Rennie and N. Srebro. Loss functions for preference levels: Regression with discrete ordered labels. In Proceedings the Nineteenth International Joint Conference on Artificial Intelligence, 2005.
Amnon Shashua and Anat Levin. Ranking with large margin principle: Two approaches. Advances in neural information processing systems, 15, 2002.
Nathan Srebro, Jason Rennie, and Tommi Jaakkola. Maximum-mane margin matrix factorization. Ad- vances in neural information processing systems, 17, 2004.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint wxXiv22307.09288 2023.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310.16944 2023.
Z. Wang, A. Bukharin, O. Delalleau, D. Egert, G. Shen, J. Zeng, O. Kuchaiev, and Y. Dong. Helpsteer2-prefereeee preference: Complementing ratings with preferences, 2024a.
Zhilin Wang, Yi Dong. Olivier Delalleau, Jiaqi Zeng. Gerald Shen, Daniel Egert, Jimmy Zhang,
Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer 2: Open-source dataset for training top pep-eerformig reward models. Advances in Neural Information Processing Systems, 37:1474- 1501, 2024b.
12

---

Published as a conference paper at ICLR 2026
Zhilin Wang. Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint arXhv:2406.00667. 2024c.
Zhilin Wang, Jiaqi Zeng, Olivier Delalleau, al. Helpsteer3: Open human-annotated preference data across diverse tasks and languages. arXiv preprint arXiv:2505. 1475, 2025.
Christopher Winship and Robert D Mare. Regression models with ordinal variables. American
sociological review, pages 512-525, 1984.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation. arXiv preprint arXiv:2401.08417, 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 08593, 2019.
13

---

Published as a conference paper at ICLR 2026
A EXTENSION TO DIRECT POLICY OPTIMIZATION
A.1 DIRECT POLICY OPTIMIZATION PRELIMINARIES
Unlike the standard RLHF approach that requires training reward model followed by RL-based policy optimization, Direct Policy Optimization (DPO) (Rafailov al. (023) directly optimizes the policy without explicitly learning a reward model. Despite this conceptual difference, DPO relies on the same Bradley- Terry preference model and uses the same type of preference dataset as reward modeling.
{ Mu Mt with preferred and dispreferred responses, DPO mini- mizes the following loss as function of a parameterized policy no.
Given a training D
(T((uuliz) )
To (1 (x: Teat (3/ (z) )
(oo D) -E (o
log
[Blog
-3 Blog
(20)
(ver(Buolz))
where "re a reference policy (typically the SFT checkpoint) and is a parameter that controls the deviation from Tel--
Following the convention in Rafailov (2023), we can define the pseudo-reward of DPO
Te (v(l)))) (T((())
fo y) = 3 log
(21)
With this definition, we note that both the reward modeling loss and the DPO loss the same form of log-sigmoid of reward differences, with parameterized rewards rg and ñe. respec- tively. This structural similarity implies that the ordinal regression framework we develop for reward modeling the main paper can naturally extended DPO by replacing re y) with ( y).
(00 take
A.2 ORDINAL DPO
Given the structural similarity between reward modeling and DPO, we can extend the ordinal re- gression framework from Section 3 to direct policy optimization. For DPO with ordinal feedback, the predictor value becomes the pscudo-reward difference,
[ Tree (T) (T(''(x)) (x) /roe (V/(z))
Ba(x.y./ fz y) fa y²) B log
(22)
Using the same threshold structure with 2K thresholds 1-I......-I:661 SK. adapt both loss functions from Section 3.3.1
Probabilistic Approach for DPO.
The NL.L loss for ordinal DPO becomes
DNLL-DIO Ci V. = logp(y > v/ xx,,
(23)
where the probability is computed using S0(x, y, y') in place of s2 2/) in equation (14
Margin-Based Approach for DPO.
The all-threshold loss for ordinal DPO is
Σ
log oogg((((12)) (Gr S0 v. v)))),
(24)
(EK)(0)
where 2)): is defined as in the reward modeling case.
The optimization problem for ordinal DPO becomes
min O.CEC
Σ
L(TO, (,,, + AlKI
(25)
where the regularization term on thresholds is necessary for the same reasons discussed in Theorem
14

---

Published as a conference paper at ICLR 2026
Symmetric and Asymmetric Models. The choice between symmetric and asymmetric models applies equally to DPO. Under the symmetric model, we impose Ç-x - while the asymmetric model learns all thresholds independently. The same reparameterizzaton strategy from Section 3.3.3 can be used to handle the ordering constraints.
Implementation Considerations. The key difference in implementing ordinal DPO compared to ordinal reward modeling lies in computing the pseudo-rewards. At each optimization step, must:
1. Compute the policy probabilities To (yllz) and (o(y'lav
2. Compute the reference probabilities (er(y)a)) and Trer(y/( (z:)
3. Calculate the pseudo-reward difference 80 (x, v.
4. Apply the chosen ordinal loss function
The gradient computation involves backpropagating through both the policy network and the thresh- old parameters, similar the reward modeling case but with the additional complexity of the log- ratio terms.
A.3 THEORETICAL CONSIDERATIONS
While we leave empirical evaluation of ordinal DPO for future work, we note that the theoretical properties established for ordinal reward modeling largely transfer the DPO setting. In particular:
1. The unbounded solution set problem (Theorem 3.1 applies equally to DPO, necessitating threshold regularization.
2. The symmetry theorem (Theorem 3.2) can be adapted to DPO under appropriate assump- tions about the symmetry of human preferences in the policy optimization context.
3. The convergence guarantees of standard DPO (Rafailov et al. 2023) extend to the ordinal when the thresholds are properly regularized.
B PROOF OF THEOREM 3.11
We prove Theorem 3.1 by analyzing how the losses behave under scaling. Consider a solution (ra, (*) and its scaled version (cr c(*) for > 0.
All-Threshold Loss. For a training example (x, y. v 2) with reward difference ro(x, y) (a(z,y') the all-threshold loss under scaling becomes
LAT(Cs, cC)
Σ - llogo(c. ((li)) (G - s)).
(26)
LEKY10}
Consider each term in the sum. If [11 2) (G x) > 0 (the threshold is correctly satisfied), then as c + 0O, we have acc v(1;2)- (G - s)) + 1, so the term contributes - logg(1) = 0 to the loss.
If ((;; (Gi s) < 0 (threshold violation), then the argument the sigmoid is large and negative. Using the approximation (u) e" for large negative u, we obtain
log ((go(c-((12)) (G s)) x log(e>((ss>((-0)))
(27)
= c v(4;2) (Gi -
(28)
clci - ss
(29)
which grows linearly with c.
For correct classification with > 0, we need s E (S--1. (2). This ensures that > Çi for all | (making p(4; 2) (Gi s) 0) and < Gi for all 2 (also making 2(1; 2)- -Ci ((1- s) > 0). Therefore, all terms in the sum approach zero as c > cc. The case < is analogous, and for 2 = 0, correct classification means (5-1: (1), which similarly ensures all terms vanish.
15

---

Published as a conference paper at ICLR 2026
Negative Log-Likelihood Loss. For the NLL loss with 0 (other cases analogous),
(ess cL) log|o(c((== s)) a(c(s=11 -)]]]-
(30)
If (--1 S2 (conrect classification), then as - oo, have S= 0, which implies (c((; - 8)) + and - - < 0, which implies a (c(((211-))) + 0. Therefore, LNLL o() - log(1 0) 0.
If < 5--1 (misclassification), both (S= - s) and (G=-1 - s) are positive, so both sigmoids approach
1. Using the approximation o(u) R 1 e for large positive u, we obtain
o(c(S= s)) acc(S--1 s)) = [] - - -1-- c
(31)
ddc 1- - -cc--
(32)
2 -
(33)
where the last approximation holds because - -) < e --dls-1- for large c. Thus,
LNLL c() x log(e (((---((()-1-))) = c(5=-1 s).
(34)
which grows linearly with
Similarly, if > Çz (misclassificationn) both (S= - 8) and (S=-1 8) are negative, so both sigmoids approach O. Using the approximation (u)) e" for large negative u, the difference becomes approximately (<(s==>) leading LNLL e() c(s (2), which again grows linearly with
C PROOF OF THEOREM 3.2
We prove that symmetrie preference data implies symmetric thresholds under the ordered logit model.
Consider the ordered logit model where the probability of observing preference level & given reward difference s2( (2, y/ 3/) is
(x(r) = a(Ca -) o((s-11 r).
(35)
Similarly, for preference level -k < 0:
(-x(r) 0(- -)) -1) 0(5- - -r).
(36)
The data symmetry assumption requires pk (r) (- for all r E R and all k E {1....,K)
We begin with the boundary case l: = K. Since {K = 0o, we have
(r( = ((2)) - a (CCK -) = - o(GK-1-1 - r))
(37)
Similarly, since (-(++1) = S-K-1 -0, we have
(-x(r)= 0 o(5-K41 r) - o(-o0) o - r))
(38)
The symmetry condition (r(r) = P- x x-rr gives
1 - a(KKI1 r) o + +r).
(39)
Using a(-u)= =1= =--(uu) rewrite the left side a0-(o((--) + r). Therefore,
o(-Sk-1tP) o(5-k+1+
(40)
Since the sigmoid function is strictly monotonic, this equality for all implies
-(x-1 C-K+1
Ç-K+1 -KK-1-
(41)
16

---

Published as a conference paper at ICLR 2026
Now, for k K L, using the symmetry condition and the fact that we ve established Ç-K+1
-KK-1
PK-1(r) o((----- p- P-(k- = o(5-K( -((-k++++ = = + - o(-6K-1 +r).
-
(42)
(43)
(44)
Using a--u)=1 =1 ouu) on the second term:
p P-(K-1) o(- (
(45)
Setting P-- and simplifying:
o(SK-1 r) - o (CK = -00====-ttt))td((---------- -1
(46)
This reduces to
- -() (((
(47)
which, by the same argument as before, implies Ç-K+2 -K
Proceeding inductively, we can show that Ç-k = -Sa for all & E (I...., K}. The case t = 0 is automatically satisfied since we excluded Ço from our parameterization, with the interval (C-1. (1) corresponding to preference level z = 0.
D TRAINING ALGORITHM
Algorithm provides the complete training procedure for our ordinal regression approach to reward modeling. The algorithm jointly optimizes the reward model parameters and threshold parameters using gradient descent, with the thresholds reparameterized through exponential mappings to maintain ordering constraints. The symmetric variant reduces the number of threshold parameters by enforcing Ç- = Ck. while the asymmetric variant learns all thresholds independently.
E ALTERNATIVE OPTIMIZATION STRATEGIES
While we use unconstrained optimization via reparameterization in our main experiments, we ex- plored several alternative approaches for handling the threshold ordering constraints.
E. PROJECTED GRADIENT DESCENT
An alternative to reparameterization is to directly optimize the thresholds while projecting them onto (-1 < Çi <
the constraint set at each iteration. Since the constraint set = (C Ç-K
{K) }o is open, we cannot directly project onto it. Instead, we define a closed approximation,
C. = {C E R2K C4+1 2 So + c for consecutive k}.
(48)
where is small constant ensuring minimum separation between thresholds.
The projected gradient descent algorithm proceeds as follows:
Algorithm Projected Gradient Descent for Ordinal Regression
Require: Learning rate minimum gap initial parameters do. So E C.
Ensure: Optimized parameters ø², <*
1: Initialize with uniform spacing in C.
2: for = 0, 2....
...... do 3: Compute gradients V&L and VcL
4: Update Φt+1 = di VoL
5: Update 54+1 = - V
6: Project G++1 Projc, (G++1)
7: end for
17

---

Published as a conference paper at ICLR 2026
Algorithm Ordinal Regression for Reward Modeling
Require: Dataset D { yt, 2/ 2:) )= loss type {LNLL, CAT). symmetry flag
Require: Learning rates 7h T/a, regularization weight X, epochs E
Ensure: Trained reward model r and thresholds Ç
1: Initialize reward model parameters from SFT checkpoint
2: if symmetric then
3: Initialize threshold parameters [on ..l....,,.0.. -. 0Kk IRK
4: else
5: Initialize threshold parameters a = 01. ( E R2Kk
6: end if
7: Initialize Opt, with cosine schedule, Opt, with constant schedule
8: for epoch = to E do
9: for batch (x, y, v/, 22) in
10:
Compute reward difference: uo(,,,,1/ ro(2,1) - r 1/)
symmetric then
12:
(1 =1=00 (a1 for to K do
13:
14:
C (--1 exp(ax)
15:
end for
for k to K do
17:
C- = -c
end for
19:
else
Ç-K 0i
21:
for j = - K + to -1 do
Sj = (j-1 + exp(a)+k))
23:
end for
24:
Ci = (-1 + exp(ak)
25:
for j = 2 to K do
26:
C = Sj-1 + exp(a)+k-1)
27:
end for
28:
end if
29:
Compute loss: I, = L( v/) >15112 Update model: o + a N Opt (oclecp(/ooo VoL)
30:
31:
Update thresholds: + na Opr, step V VaL)
end for
33:
end for
The projection onto C, can be formulated as quadratic program:
Proje, (5) argmin ||ş - 2112
(49)
SEC.
which can be solved efficiently using specialized QP solvers, particularly when K is small (as is typical in practice).
E.2 ASYNCHRONOUS PARAMETER UPDATES
During experimentation, observed that updating reward parameters & and threshold parameters Ç at different frequencies can improve optimization stability. This approach is motivated by the observation that thresholds typically converge more slowly than reward parameters and may benefit from less frequent but more stable updates.
The asynchronous update scheme works follows:
18

---

Published as a conference paper at ICLR 2026
Algorithm 3 Asynchronous Parameter Updates
Require: Update frequency N for thresholds, learning rates n/. 1/a
1: for t == 0, 1, 2. do
2: Compute gradient Vaf C and update @++1 = Φ: no V
3: if mod N 0 then
4:
Compute gradient v.L L using accumulated statistics Update a++1 = 1/a V VaL
5:
6: else
7:
Keep 01+1 Cz
8: end if
9:
end for
In experiments, tested N {10, 50, 100} and found that while this approach stabilize training in some scenarios, particularly when the reward model is prone to large gradient updates, the simultancous update strategy (equivalent to N = 1) generally performed comparably or better with appropriate learning rate scheduling. The asynchronous approach may he beneficial when:
1. The reward model architecture particularly deep or unstable
2. The dataset contains significant label noise
3. Computational resources require less frequent threshold updates
E.3 COMPARISON OF OPTIMIZATION STRATEGIES
Each optimization strategy has distinct trade-offs:
Reparameterization (our choice): Transforms the problem into unconstrained optimization, en- abling the use of standard optimizers without modification. The exponential mapping naturally maintains constraints but can occasionally lead to numerical issues a values become very nega- tive (causing near-zero increments).
Projected Gradient Descent:
Maintains thresholds their natural space, making interpretation easier. However, it requires solving projection problem at each iteration and choosing appro- priate value.
Asynchronous Updates: Can improve stability but requires tuning the update frequency N and may slow convergence thresholds are updated too infrequently.
In experiments, found reparameterization be the most robust across different datasets and model architectures, which motivated our choice for the main implementation.
F DETAILED EXPERIMENTAL SETUP
F.1 MODEL AND DATA CONFIGURATION
Base Models. We conduct experiments using three pre-trained language mod- els obtained from HuggingFace Hub: (L18ma-3.1-8BB,
(Mistral-7B). and (Zephyr-7B). Each model initializes from its supervised fine-tuned checkpoint. The language modeling head is replaced with a single-layer reward head that outputs scalar rewards. All experiments employ mixed precision training with 6 for computational efficiency.
httgg/iggaaettt/mmsssssla-a-aa-. miinneenthhandboo// petoo////ppy-----stt-ffllll -tt
Data Processing. For both HelpSteer2 and HelpSteer3 datasets, we apply consistent preprocess- ing. Sequences are limited to 2048 tokens with prompts truncated 1600 tokens when necessary. Tokenization uses model-specific tokenizers with padding tokens set to EOS when undefined. Dur- ing training, prompt tokens masked with - -100 to exclude them from loss computation. Each response is ensured to terminate with exactly one EOS token, with any embedded EOS tokens re- moved during preprocessing.
19

---

Published as a conference paper at ICLR 2026
F.2 HYPERPARAMETER SEARCH
We conducted comprehensive hyperparameter optimization to identify robust training configura- tions. The search space and final selections are detailed below.
Learning Rates and Scheduling. For model parameters, we evaluated learning rates in {5 x 10-7,1x 10 2,2x x 10 5 10-6) with both cosine and linear schedulers. The optimal configura- tion uses 1 10 with cosine scheduling and 10% warmup steps. For ordinal threshold parameters, 2), finding 1x 10-3 with constant
we tested rates in {1 x 10 x5100 10x10=+,,00 *,1x 10 9,5x 10
scheduling work best.
Regularization Strategies. We explored L2 regularization weights À E (0,0.1, 10) for threshold parameters in ordinal methods. Lower values (A < 0.1) led to optimization instabilities as predicted by Theorem 3.1] while very high values (A 10) overly constrained threshold learning. The optimal range was À E 50, with specific values varying by model and dataset. We also tested weight decay model parameters but found no improvement over unregularized training.
Training Dynamics. We investigated updating threshold parameters at different frequencies rela- tive to model updates, testing intervals of {1,5, 10, 15) steps. With properly tuned hyperparameters. synchronous updates (interval=1) achieved performance matching that of asynchronous schemes while being simpler
implement. Details of the asynchronous update strategy are provided in Appendix H Batch size experiments ranged from 32 to 128 (effective batch size after gradient ac- cumulation), with 64 providing the best stability-efficienny trade-off. All models are trained for epochs, which determined sufficient for convergence through preliminary experiments.
F.3 INFRASTRUCTURE AND IMPLEMENTATION
Computational Resources. All experiments utilize 8 GPUs (either NVIDIA H100 with 80GB HBM3 or H200 with 141GB HBM3e) with Fully Sharded Data Parallel (FSDP) for distributed training. We employ automatic transformer layer wrapping with full sharding strategy to mini- mize memory footprint while maintaining training efficiency. Gradient accumulation over 4 steps achieves an effective batch size of 64 while fitting within GPU memory constraints.
Model Selection. We select the best checkpoint based on reward accuracy on the validation set, where reward accuracy measures the frequency of assigning higher rewards to preferred responses in preference pairs. To ensure model quality, we exclude checkpoints with anomalously high vali- dation loss (typically occurring in very early training) and, for ordinal methods, those from training transition phases where thresholds undergo rapid adjustments (see Appendix Gor detailed analysis of threshold convergence behavior).
Reproducibility. All experiments fixed random seed 0 for weight initialization, data shuffling, and dropout. Training configurations are managed through Hydra framework with all hyperparam- eters logged to Weights & Biases for full reproducibility. Model checkpoints and configuration files will be released upon publication. Our implementation builds on the Transformer Reinforcement Learning (TRL) library with extensions for ordinal regression methods.
G THRESHOLD CONVERGENCE ANALYSIS
G.1 EFFECT OF REGULARIZATION ON THRESHOLD STABILITY
Figure [ illustrates the evolution of threshold parameters during training for NLL-Symmetric
Llama with HelpSteer3, comparing three L2 regularization weights: À E {0, 0.1, The plots reveal distinct convergence behaviors that represent different trade-offs between stability and flexi- bility.
Without regularization (A O, red curves), the thresholds exhibit continuous growth through- out training, with outer thresholds (zeta and zeta 5, corresponding to <-3 and (3) showing the most dramatic increases. This unhounded behavior empirically validates Theorem 3.1. which
20

---

Published as a conference paper at ICLR 2026
ondrud/cete__ rowudd 13 lem onde endr appi000) rugh
ordindizetaa cosed fud Jama syreedrs and rog
rowul 183, Aotie rop
24
1300% 2004
ortrokizer.2 maidd Fot rdir spenere aplinm
onforoliee
ordralizett_0 0
selel mooot Fox Lasa addad Sytwers aplaent
mead hot, bino miial oyvernn apiin
422
64
Figure 1: Evolution of symmetric threshold parameters during training with different L2 regulariza- tion weights. The plots show zet through zeta which correspond the ordinal thresholds S-3 through Ç3 in our formulation (excluding So which is not parameterized). Due to symmetry constraints, zeta_0 = C3., zeta_1 -2,, and zeta_2 - the plots. Without regularization (A = 0. red), thresholds exhibit unbounded growth. Moderate regularization (A 0.1, blue) allows gradual convergence with more flexibility, while stronger regularization (A = 1, green) enforces rapid convergence to stable values.
proves that the unregularized optimization problem may admit no finite optimal solution. The model achieves arbitrarily low loss by scaling both rewards and thresholds simultaneously, leading to nu- merical instability despite potentially maintaining correct relative orderings.
With moderate regularization (A = 0.1, blue curves), thresholds ultimately converge but maintain more flexibility during training, allowing the model explore a wider range of threshold config- urations before settling into stable values. This gradual convergence can be beneficial when the optimal threshold structure not immediately apparent from the data. contrast, stronger regu- larization (A = 1, green curves) enforces rapid convergence within the first 100k steps, providing greater stability at the potential cost of reduced adaptability. The choice between these settings rep- resents a trade-off: A = 0.1 offers more flexibility for learning complex preference patterns, while = provides robustness and faster convergence. Both configurations successfully prevent the unbounded growth observed without regularization, making them viable choices depending the specific requirements of the task.
The convergence patterns also reveal that inner thresholds (zeta 2 and zeta corresponding to Ç-1 and (1) stabilize more quickly than outer ones across all regularization settings, suggesting that distinguishing between moderate preference levels is learned before extreme preferences. The symmetric constraint is maintained throughout training, with negative thresholds mirroring their positive counterparts.
G.2 TRAINING PHASES AND TRANSITIONS
When regularization moderate (as with À = 0.1), we observe a distinct multi-phase training behavior where thresholds evolve through discrete stages rather than converging smoothly. Figure illustrates this phenomenon more clearly for a representative training run.
The training proceeds through several distinct phases, each characterized by relatively stable thresh- old values, with rapid transitions between them. This behavior particularly pronounced for the outer thresholds (zeta_0 and zet a.5, corresponding to Ç-3 and (s), which undergo several dis- crete jumps before reaching their final values. The inner thresholds exhibit similar but less dramatic phase transitions.
21

---

Published as a conference paper at ICLR 2026
andiaaizens_
ortinal
andinal
-ow Joo. www.kaaw..oieeeooeeot.t.... www..e/www.oorro...oooeeeee.eee.
rohadd www.
0.
06
06
2906
150c
ardineliseta_2
ordinattenaa.
ordinal
-w00-100. 1,00
rewund www.arrs.rrdiii...o http://swwta...mmcccccccccc wwt.
raverr, rovert_ www.://oooaaaaoooooom www.aaaaaanonnnaoooooooooo www.syyrmmenccc..o updittic segle
0.3
0.3
0.4
Figure 2: Multi-phase threshold evolution during training with moderate regularization (A 0.1). The thresholds exhibit step-like progression with distinct stable phases separated by rapid transition periods. Outer thresholds (zeta.0 and zeta.5) show the most pronounced phase transitions.
This multi-phase behavior likely reflects the model's progressive refinement of its preference un- derstanding. early phases, the model learns coarse distinctions between preference levels, es- tablishing rough boundaries. As training progresses and the model encounters more challenging examples that fall near existing boundaries, it triggers threshold adjustments to better capture the nuanced preference structure in the data. Each phase transition represents the model discovering better configuration for separating preference levels, with the step-like nature suggesting that these improvements require coordinated changes across multiple thresholds rather than gradual local ad- justments.
Importantly, during these transition phases-the periods of rapid threshold adjustment between sta- ble phases-both the threshold values and the associated model weights exhibit increased instability. The reward predictions become less reliable as the model reconfigures its internal representation of preference levels. This instability makes checkpoints from transition phases unsuitable for evalua- tion, as they represent incomplete adaptations new threshold configurations. In our experimental protocol, we therefore exclude checkpoints from these transition periods when selecting models for evaluation, focusing instead on checkpoints from stable phases where the thresholds have settled into consistent values. This filtering ensures that our reported results reflect the model's 's perfor- mance when operating with a coherent and stable preference structure rather than during periods of internal reorganization.
G.3 REWARD VALUE STABILITY UNDER REGULARIZATION
Theorem 3.1
establishes that without regularization, rewards and thresholds can scale jointly to achieve arbitrarily low loss. Our regularization strategy addresses this by bounding threshold growth, which necessarily anchors the reward scale as well, since the two parameter sets are cou- pled through the loss function. Having demonstrated threshold convergence Section G.I we now
empirically verify that reward values also remain stable under regularization.
Figures 3 and show the evolution of reward values assigned to chosen and rejected responses during training and evaluation for the same NLL-Symmetric models analyzed in Section G.1 The empirical results confirm our theoretical predictions. Without regularization (A = 0, red curves), reward values exhibit the unbounded growth predicted by Theorem 3.1 On the training set, chosen responses receive increasingly positive rewards while rejected ones receive increasingly negative rewards, with the separation growing continuously. On evaluation data, this divergence is more pronounced, with chosen rewards growing increasingly positive and rejected rewards growing increasingly negative without bound, demonstrating the joint scaling of rewards and thresholds.
22

---

Published as a conference paper at ICLR 2026
@ewards_trin/chhie
rewards_train/rejecee
reward hs3 symimmtric syptttttttttcuptttttll... podate@1__eggee
reward hs3 Ilama,
_al__llaaaooaaaaa__aaaaa symirntissmmmmiicc..mmmmmm updatee11111111111 reggee-
reward Lr___llla_orddd "aama_ordinlL_ymmeed (egmmetric_uddate011.. update01_reee
reward wtt.oordninasmmmmeeee update01_reg
reward aa__lllaa_ordiin soydmnilsymmetrccc. syymmerric_udaate11 update01_rep regg
reward_hst laallllaaaoodda cacimaaordina_agee saainnssssmmmm_cuuuu _ptpiitp__________11 regl
2
0
-4
-6
-2
-10
Step
-12
Step
50k
100k
150k
200k
250k
300k
50k
100k
150k
200k
250k
300k
Figure 3: Training reward values for chosen and rejected responses across different regularization settings. Without regularization (A 0, red), rewards divenge significantly with chosen responses receiving increasingly large positive rewards and rejected ones receiving increasingly negative re- wards. With regularization (A 0.1 in blue, À in green), reward values remain bounded and stable throughout training.
rewards_ serrs_ssevl/choeen
rewards_eeve eval/rejjcced
rwward h63, Jamaaamdaaaa laamaaadiiaaaaaaeoaaaaaa symmentissmmtiiccc...mm updase01_ regle-1
roward_ hs3, Jaamaaaaaaaaaaaaaa ondinnal wymetrkk_pdatton regle-1
rewerd llams, ordinal symmetric spdateti),
Is3. llama ordinal symmetric, updatetl,
reward haamaaassaa_____..._oo Malam_ordinal__p0 ardinal_symmerrt upddite01__ee
homa_ordinal, hatpulssmmmtttneeeeeeee
-1
-5
-6
Step
Step
200k
100k
Figure 4: Evaluation reward values for chosen (left) and rejected (right) responses across different regularization settings. Without regularization (red), both chosen and rejected rewards diverge to extreme values. With regularization (blue and green), both reward types remain bounded and stable.
With threshold regularization (A = 0.1 in blue and À = in green), reward values remain stable as theoretically expected. Training rewards converge responses receiving rewards around () to 2 and -2 to -4, respectively. On evaluation data, chosen while rejected rewards stabilize around to -2, remaining constant
bounded values, with chosen and rejected
rewards stabilize around after initial convergence. The stronger regularization (A 1) produces slightly more conservative reward values compared to the moderate regularization (A 0.1), which is consistent with the faster threshold convergence observed in Figure
These results validate that regularizing thresholds effectively anchors the reward scale through the coupling in the loss function. By preventing threshold growth, simultaneously prevent rewards from scaling independently, ensuring convergence to a well-defined solution with stable and inter- pretable parameters for both rewards and thresholds.
23

---

Published as a conference paper at ICLR 2026
H ORDINAL PERFORMANCE METRICS
While our main evaluation follows standard practice in reward modeling by focusing on binary pref- erence accuracy-Whether the preferred response receives a higher reward-this metric provides limited insight into the effectiveness of ordinal regression. Binary accuracy treats all correct pre- dictions equally, failing to distinguish between model that correctly predicts preference strength versus strength 3, as long as the ranking is preserved. To assess whether our ordinal regression methods genuinely capture fine-grained preference structure, we introduce additional metries that evaluate the accuracy of predicted preference strengths.
This appendix is organized into three parts. We first define the ordinal performance metrics used throughout our evaluation (.H.1)) We then examine training dynamics, showing how these metrics evolve as our ordinal model learns (sH.2). Finally, we compare our joint training approach against post-hoc calibration of baseline methods, demonstrating that learning thresholds alongside reward parameters essential for capturing ordinal structure ($H.3)
H.1 METRIC DEFINITIONS
evaluate ordinal performance using complementary metrics that measure different aspects of prediction quality:
Mean Absolute Error (MAE). The average absolute difference between predicted and actual preference strength levels
MAE
pred
2
where z denotes the discrete ordinal level. Lower values indicate hetter performance, with perfect prediction yielding MAE = O.
Accuracy Within A: We measure prediction accuracy allowing for different tolerance levels. For-
mally, Accuracy Within defined as
n
Acci
Σ
1 [1
= S k)),
where [[]] is the indicator function. This metric captures the proportion of predictions that fall within tk levels of the true preference strength.
For A: 0, this reduces to exact accuracy, the most stringent metric requiring perfect ordinal pre- diction beyond mere ranking preservation. For k = and h = 2, we obtain near accuracy metrics that allow for minor deviations, recognizing that adjacent preference levels may have overlapping boundaries in human judgment. These metrics form natural hierarchy Acc; AccR+1, provid- ing increasingly relaxed measures of ordinal prediction quality. In our evaluation, we report MAE alongside Accą for k (0, 1,
H.2 TRAINING AND EVALUATION DYNAMICS
Figures and 6 show the evolution of these metrics during training for NLL-Symmetric on Llama with HelpSteer3.
The training metrics show rapid improvement in the first 50k steps, with MAE dropping from above 2 to below 0.5 and exact accuracy rising from near to above 60%. This corresponds the period when thresholds are establishing their initial configuration (compare with Figure The subsequent gradual improvement reflects fine-tuning of both thresholds and model weights.
On validation data, the model achieves approximately 55% exact accuracy and 85% accuracy within one level, with MAE stabilizing around 0.6. While exact accuracy is lower than on train- ing data eata-expected given the challenge of generalizing fine-graie mme-grained distinctions- the high near- accuracy demonstrates that the model successfully learns the ordinal structure rather than merely memorizing training examples. The fact that over 95% of predictions fall within two levels of the
24

---

Published as a conference paper at ICLR 2026
perfomanerroomaccee. train//ean_eessllle _error reward ham_ ordinal symmerric, uodate91_ rogle
pedinel_eeformmaaee train//ccuraacy.cooooooooo within
reward hit llama ordinal symmetrc updated] rgle
Slep
50c
100k
190k
2004
250k
120x
55k
100k
150k
200k
250k
300k
ordinal_pertt performance train//accuaccy. within__1 reward hst Jamz aranal symmetric, urdate01 recle1
vuddina__eerfrmmt train//ccuuacyy within_o reward hs3 Ilama, ordinal, symmetr e. uported1 mate
0.6
044
300p
Step
30K
100k
130k
2004
2004
Figure 5: Ordinal performance metrics on the training set. MAE decreases rapidly in early training, stabilizing around 0.2. Exact accuracy reaches approximately 85%, while near-perfect accuracy (Within approaches 95%, indicating strong ordinal prediction training data.
ordinall_prfeee performeeeforemcee... eval/meen, absolute, prrorr error reward, comnoordinl__yymmeee updatetl,
ordinal performance, evaljaccuracy, within 2 revard hsl, Homa ordinal, c_updatee]__p.. reple-]
15
1.6
0.
12
0.85
0.8
55k
100k
1534
200k
250k
50k
100k
1504
2003
2504
ordinall_ performaeerfreeeee.... eval/accurryy within
ordinal performance pvaljaccuracy,
reward..111 713, comm_.odinnL.ymmmm semtin/_syymmmruuuuuuuuee undaoes
revard_hst Jomm hooond_h__lllmmmooom wwwns_ordiollsyymeee c_spdstett_ regle-]
0.8
0.5
02
0.6
Step
Stop
55k
100k
150k
200k
250k
506
1003
155k
200k
250k
Figure 6: Ordinal performance metrics on the validation set. The model achieves approximately 55% exact accuracy and 85% accuracy within 1 level, demonstrating meaningful generalization of ordinal structure despite the more challenging validation distribution.
true value confirms that our ordinal regression approach captures meaningful preference strength information beyond what binary metrics reveal.
25

---

Published as a conference paper at ICLR 2026
These results validate that our ordinal regression methods do than preserve rankings; they learn to predict the degree of preference with reasonable accuracy, justifying the additional complexity of the ordinal framework over simpler binary approaches.
H.3 COMPARISON WITH POST-HoP CALIBRATION
To validate that joint threshold learning is essential-rather than convenience that could be re- placed by post-hoc calibration -we compare our approach against baseline methods augmented with learned thresholds, Although baseline approaches (Margin BT, Scaled BT, Soft Label) do not natively produce ordinal predictions, we can enable ordinal evaluation through post-hoc calibration: after training each baseline model, we learn thresholds } that map reward differences to ordinal levels by optimizing the same ordinal negative log- log-likelihood objective used in our framework. This allows us isolate the effect of threshold learning strategy: (1) models that learn thresholds jointly with reward parameters (our approach), versus (2) models that learn thresholds separately after re- ward model training (post-hoc calibration). If joint training offers no advantage, post-hoe calibration should achieve comparable ordinal performance.
H.3.1 POST-Hoc CALIBRATION PROCEDURE
After training a baseline reward model (Scaled BT, Margin BT, or Soft Label) for the standard epochs on HelpSteer2, we freeze the model weights and learn thresholds (S-3, S-2. S-1: S1, S2, Sa} that map reward differences to ordinal predictions.
Threshold Learning. Given trained reward model (with frozen parameters f), compute
ward differences Ar; = re (v²") ro (v{) for all training examples. Let z; E (-3, -2, 1, 1,2,3) denote the true ordinal strength for example i. We learn thresholds {<-3. C-2, (-1, <2, (3) (with
S-1 = -oo and Ç1 +oo, and no So since the interval (5-1- corresponds to 2 = 0) by mini- mizing the ordinal negative log-likelihood
Cealib =
E log p(y." >,, va x4),
(50)
where the probability ply > V | 2)) is given by the ordered logit model (following Equation 14 from the main text)
if -. if E (1,2,3), o((1 Ar) 11-1-AA1 =
aol6= Ar) - o(s--1 Ar)
ply ~= 1/ x)
ol6e+1 - Ar) ols Ar)
To ensure monotonicity (5-3 (-2 < S-1 Si positive increments and optimize using gradient descent (100 epochs, learning rate 0.01).
(3). parameterize thresholds using
Ordinal Prediction. After learning thresholds, predict ordinal strength for new example by finding which interval (Ga, (6+1) 1) the reward difference Ar falls into (treating the interval (S-1,C)) as corresponding 0). Equivalently, can compute
c = argmax p(y > 3/ xx) zE (-3......3)
(51)
where p(u >= y (x) is given by the ordered logit model above.
We explored two additional calibration methods methods-quantile-ee quantile-based (setting thresholds as empirical quantiles of reward differences) and classification-base (directly optimizing threshold positions to minimize bAE)-but found that ordinal NLL calibration consistently outperformed both. There- fore, we report only ordinal NLL calibration results for the baselines.
H.3.2 EXPERIMENTAL SETUP
We evaluate all methods using the Llama-3.1 T.luu---- model trained on HelpSteer2 for 5 epochs. For our ordinal regression method INLL-Symmetric), thresholds are learned jointly during training. For baselines, we apply post-hoc calibration the same training data, then evaluate the held-out
26

---

Published as a conference paper at ICLR 2026
test set. All models identical architectures, training hyperparameters, and data splits, ensuring that differences in ordinal metrics reflect only the threshold learning strategy (joint versus post-hoc). We use seed 0 for reproducibility.
H.3.3 RESULTS
Tables 1 and present ordinal performance metrics for all methods on training and test sets, respec- tively.
Table Ordinal performance metries on HelpSteer2 training set (8,677 examples). NLL-Symmettr learns thresholds jointly during training, while baselines use post-hoc calibration with frozen reward models.
Method
MAE
Acc@0
Acc@1
Acc@2
Scaled BT (post-hoc)
1,320
27.2%
69.3%
77.5%
Margin BT (post-hoc)
0.994
29.2%
77.8%
95.7%
Soft Label (post-hoc)
1.609
14.7%
51.6%
81.1%
NLL-Symmetric (joint)
0.443
60.2% %
95.7%
99.8%
Table 5: Ordinal performance metrics on HelpSteer2 test set (448 examples). Joint training (NLL-
Symmetric) substantially outperforms post-hoc calibration across all metrics.
Method
MAE +
Acc@0
Acc@ 1
Acc@2 f
Scaled BT (post-hoc)
2.667
10.9%
37.1 %
44.0%
Margin BT (post-hoc)
2.181
16.1%
48.7%
59.6%
Soft Label (post-hoc)
1.725
12.9%
47.8%
78.1%
NLL-Symmetric (joint)
1.060
29.7%
72.5%
92.9%
Joint Training Advantage. On the training set, our jointly-trained ordinal model achieves sub- stantially lower MAE (0.443) compared the best post-hoe baseline (Margin BT: 0.994), repre- senting 55% reduction error. More strikingly, exact accuracy (Acc@0) reaches 60.2% versus 29.2% for Margin BT, demonstrating that joint optimization enables the model learn fine-grained preference distinctions that post-hoc calibration cannot recover.
The advantage persists and even amplifies the test set. NyL--ymmetric achieves MAE of 1.060 compared to 1.725 for the best baseline (Soft Label), and exact accuracy of 29.7% versus 16.1% (Margin BT). The gap in Acc@ is particularly notable: 72.5% for our method versus 48.7% for Margin BT, indicating that our model predicts preference strength within one level for nearly three- quarters of examples, while the best baseline achieves this for fewer than half.
Baseline Comparison. Among post-hoe calibrated baselines, Margin BT generally performs best on the training set (MAE: 0.994, Acc@1 1: 77.8%), likely because its margin-aware loss provides a better foundation for threshold learning. However, on the test set, Soft Label achieves low- est MAE (1.725) and highest Acc@2 (78.1%), suggesting it generalizes differently than Margin BT. Scaled BT consistently underperforms, indicating that scaling rewards alone does not provide sufficient structure for accurate ordinal prediction.
H.3.4 SUMMARY
These results validate that joint ordinal training provides substantial benefits over post-hoc calibra- tion. The 38% reduction in test MAE (1.060 versus 1.725) and the doubled exact accuracy (29.7% versus 16. 16) demonstrate that learning thresholds alongside reward parameters is essential for cap- turing fine-graine preference structure. Post-hoc calibration, despite using the same probabilistic objective, cannot recover the ordinal information that emerges from joint optimization. This con- firms that our ordinal regression framework does more than preserve rankings; learns predict
27

---

Published as a conference paper at ICLR 2026
the degree of preference with meaningful accuracy that cannot be achieved through threshold fitting alone.
I ERROR MARGIN ANALYSIS
Beyond standard accuracy metrics, we analyze the severity of errors made by different models. When reward model incorrectly ranks responses, assigning higher reward to the dispreferred sponse, the magnitude of this ranking violation provides insight into model calibration and confi- dence. Models that make large-margin errors may be particularly problematic in deployment, as they indicate strong but incorrect preferences.
I.1 ERROR MARGIN DEFINITION
For each example in RewardBench consisting prompt x, chosen response ye. and rejected re- sponse Vr, define error occurring when r= yr) ra(x, ve). The error margin for such misclassified examples is computed as
Error Margin r Ur) rá ye))
(52)
Larger values indicate that the model not only made an incorrect ranking decision but did with high confidence, assigning substantially higher reward to the dispreferred response. This metric re- veals whether model's errors tend to be borderline cases (small margins) or egregious misrankings (large margins).
1.2 EXPERIMENTAL COMPARISON
We compare the error margin distributions two methods trained on HelpSteer2 using Llama-3.1 8B the base model:
1. Simple BT: Standard Bradley-Terry loss treating all preferences binary, completely ig- noring ordinal strength information the training data.
2. NLL Ordinal Symmetric: Our proposed negative log likelihood ordinal regression ap- proach with symmetric threshold constraints.
Both models are evaluated on the full RewardBench test set, and we analyze only the incorrectly ranked examples.
1.3 RESULTS AND ANALYSIS
Figure 7 presents the error margin distributions for both methods. The contrast is striking across multiple dimensions:
Error Frequency. Simple BT produces 433 errors RewardBench, while NLL Ordinal Symmet- ric makes only 282 errors, showing a 35% reduction in total misclassifications This improvement alone demonstrates the value of leveraging ordinal information.
Error Severity. The mean error magnitude reveals an even more dramatic difference: Simple averages 3.827 error margin, while NLL Ordinal Symmetric averages just 0.501. This is reduction of over 87%. This indicates that our ordinal approach not only makes fewer mistakes but also makes less severe errors when it makes a mistake.
Distribution Shape. The histograms reveal fundamentally different error characteristics:
Simple BT (top panel) exhibits heavy-tailed distribution with errors extending to margins as large as 20. The distribution shows substantial mass at moderate-to-larg eтror margins (2-10), indicating frequent high-confidence misrankings. Even after the initial peak near zero, the distribution maintains considerable density across a wide range, suggesting sys- tematic calibration issues.
28

---

Published as a conference paper at ICLR 2026
Simple BT RewardBench Error Margin Distribution
Total Errors: 433 Mean Error Magnitude: 3.827
60
50
Count
20
10
0.0
7.5
12.5
Ordinal Symmetric RewardBench Error Margin Distribution
Total Errors: 282
Mean Error Magnitude: 0.501
140
120
100
Count
60
0.0
2.5
7.5
10.0
12.5
15.0
17.5
Error Magnitude (/Scoremesen Scorenjectad)
Figure 7: Error margin distributions comparing Simple BT (top) and NLL Ordinal Symmetric (bot- tom) on RewardBench. The ordinal method reduces both the frequency and severity of errors, with no errors exceeding margin 2.5 compared to Simple BT's errors extending to margin 20. The dra- matic difference in mean error magnitude (3.827 0.501) and distribution shape demonstrates that leveraging ordinal information improves not only ranking accuracy but also model calibration.
NLL Ordinal Symmetric (bottom panel) displays sharply concentrated distribution with nearly all enrors confined margins below 2.5. The distribution drops off rapidly after the peak near zero, with virtually no errors exceeding margin 3. This tight concentration indicates that when the ordinal model errs, does with low confidence. The reward dif- ferences are small, suggesting genuine ambiguity rather than systematic misunderstanding.
Maximum Error Magnitude. Perhaps most significantly, NLL Ordinal Symmetric produces no with margins up
errors with margins exceeding approximately 2.5, while Simple BT generates to 20. These large-margin errors in Simple BT represent cases where the model is highly confident in an incorrect preference, potentially the most dangerous failure mode in practice.
1.4 IMPLICATIONS
These results demonstrate that ordinal regression provides benefits beyond simple accuracy improve- ments. By learning from fine-grained preference strength information, the model develops better calibration and becomes more hesitant to make strong predictions when preferences are subtle or ambiguous. The dramatic reduction in harge-margin errors suggests that ordinal methods produce more reliable reward signals for downstream applications like reinforcement learning, where high- confidence incorrect rewards could severely misguide policy optimization.
29

---

Published as a conference paper at ICLR 2026
J ROBUSTNESS TO LABEL NOISE
Real-world preference datasets inevitably contain annotation errors due to annotator disagreement, subjective judgment, and cognitive biases. Recent empirical work has documented the prevalence and complexity of such disagreement: for instance, instance,,liipurr et al. (2026) show that even experienced human moderators exhibit substantial disagreement when labeling the same content, highlighting inevitability of noisy labels in human annotation pipelines. Understanding how reward models be- have under label noise is crucial for practical deployment. While binary preference models face noise simple flips (preferred ++ dispreferred), ordinal regression confronts a richer noise structure where labels can be corrupted in varying degrees. Recent work has highlighted the importance of noise robustness in preference learning: Afzali el. (2025) address content-dependent multi-souree source noise in preference optimization through multi-objective optimization, while Im and Li (2025) pro- vide theoretical generalization guarantees under noisy feedback, showing that moderate noise rates need not prevent effective learning if the data is well-structured. Here we investigate whether our ordinal regression framework exhibits robustness different patterns of label corruption.
J.1 NOISE MODELS
We evaluate robustness under the following two noise paradigms that represent different real-world annotation failure modes:
This models annotators who consistently apply incorrect thresholds when assigning preference strength. For instance, an annotator might systematically overestimate or un- derestimate preference intensity. Given a true ordinal label E [-k,.... K}, we corrupt it by shifting one level up or down with equal probability, i.c.,
Systematic Shift Noise.
min(= + 1, k) max(2 - 1,
with probability 0.5,
nooss
(53)
with probability 0.5.
This shifts labels by one ordinal level while respecting the boundaries of the ordinal scale. The noise preserves the general magnitude of preference strong preference remains relatively strong), making it representative of calibration errors rather than complete confusion.
Random Label Noise. This models annotators who occasionally assign a completely arbitrary label, perhaps due to fatigue, misunderstanding lack of attention. We replace the true label with a uniformly random draw from K,.... .K), i.e.,
moiss
Uniform(
(- ((- - K - 1,
(54)
This represents catastrophic annotation failures where the assigned label bears no relationship to the true preference, making it much more challenging noise setting.
For each noise type, we evaluate at corruption rates of 25% 50%, 75%, and 100%, representing increasingly severe label corruption. At each comupption rate, the specified percentage of training examples has their labels replaced according to the noise model. The extreme case of 100% corrup- tion provides insight into the theoretical limits of model robustness.
J.2 EXPERIMENTAL CONFIGURATION
We train NLm-SSmmetric models (our best-performing method from Section 4) on Llama-3.1 1-B8B with HelpSteer2 under each noise condition. All hyperparameters remain identical to the noise-free setting, including learning rates and regularization strength. Models are trained for 5 epochs and evaluated on the clean (noise-free) validation set to measure their ability to recover true preferences despite training on corrupted labels.
J.3 BENCHMARK PERFORMANCE UNDER NOISE
Tables and present comprehensive evaluation results on RewardBench and RM-Bench, respec- tively.
30

---

Published as a conference paper at ICLR 2026
Table 6: RewardBench performance under different noise conditions.
All models NLL--
Symmetric with Llama-3.1 trained on HelpSteer2.
Noise Setting
Chat
Chat Hard
Safety
Reasoning
Average
No Noise
0.941
0.728
0.897
0.804
0.843
Shift 25%
0.953
0.697
0,892
0.771
0.828
Shift 50%
0.930
0.728
0.895
0.764
0.829
Shift 75%
0.930
0.726
0,850
0.726
0.808
Shift 100%
0.955
0.693
0.853
0.881
0.846
Random 25%
0.930
0.678
0.885
0.722
0.804
Random 50%
0.894
0.636
0.878
0.619
0.757
Random 75%
0.872
0.548
0.743
0.724
0.722
Random 100%
0.542
0.441
0.480
0.668
0.532
Table 7: RM-Bench performance under different noise conditions. All models use NLL-Symmetric with Llama-3. 1-8B trained on HelpSteer2.
Noise Setting
Chat
Math
Code
Safety
Hard
Normal
Easy
Average
No Noise
0.669
0.558
0.515
0.873
0.418
0.681
0.862
0.654
Shift 25%
0.718
0.575
0.518
0.882
0.446
0.702
0.872
0.673
Shift 50%
0.683
0.557
0.517
0.919
0.427
0.698
0.883
0.669
Shift 75%
0.661
0.569
0.514
0.878
0.418
0.684
0.865
0.655
Shift 100%
0.628
0.556
0.527
0.878
0.371
0.684
0.886
0.647
Random 25%
0.706
0.556
0.518
0.839
0.430
0.677
0.857
0.655
Random 50%
0.633
0,557
0.500
0.826
0.346
0.655
0.886
0.629
Random 75%
0.554
0.531
0.489
0.815
0.405
0.619
0.768
0.597
Random 100%
0.426
0.485
0.498
0.398
0.371
0.442
0.542
0.452
The benchmark results reveal striking differences in robustness between the two noise types. Per- formance remains remarkably stable under systematic shift noise across both benchmarks. On Re- wardBench, the average accuracy ranges from 0.808 to 0.846 across all shift noise levels, staying within the noise level or slightly below the average accuracy of the no-noise baseline of 0.843. Sim- ilarly, RM-Bench, performance varies only from 0.647 to 0.673. These small variations are not meaningfully different from the baseline, demonstrating strong robustness to systematic calibration errors.
Random label noise presents a more challenging scenario with graceful degradation. At 25% cor- ruption, performance remains competitive with the no-noise baseline (0.804 on RewardBench, 0.655 on RM-Bench), indicating strong robustness to moderate random noise. At 50% corruption, where half the training labels are completely arbitrary, the model achieves 0.757 and 0.629 respectively, representing moderate degradation from baseline. The degradation accelerates at 75% noise and becomes severe at 100% corruption. Notably, since 50% accuracy represents chance-level perfor- mance on these benchmarks, the 100% random noise results (0.532 and 0.452) confirm that the model learns essentially nothing when trained entirely on random labels, as expected.
Certain evaluation categories exhibit differential sensitivity to noise. The Safety category maintains high scores even under heavy noise (0. 74.7430-.99 on RewardBench across all conditions except 100% random), suggesting that safety-related preferences have clearer signal that survives label corruption. Conversely, Chat Hard and Reasoning categories show more pronounced degradation under random noise, indicating these tasks may benefit more from clean, high-quality annotations.
J.4 TRAINING AND EVALUATION DYNAMICS
Figures g through provide detailed insight into how label noise affects learning dynamics and generalization.
31

---

Published as a conference paper at ICLR 2026
Vidina_performaeee wtw..u/innnnnetbooooooeeooo wwwolltteeereoooooooo error ndiiziit
ordinaL_eeformance performance. trainiaccuacc...tllii within
ordinal,
retssiion robnitiaiii rebultanm retustotin
0.
2.5
0.8
0.4
022
pedimal_erfommanceeeooo train//cccuracyy.ooooooooo within
performance. trainjaccuracy. within_
10. 10.99110 redic2ldakl
nasettalel
0.8
0.2
Stop
S10p
33% 43%
20k
30%
Figure 8: Training dynamics under systematic shift noise (left: mean absolute error, right: accuracy metrics). Training performance degrades with increasing noise levels, indicating the model does not overfit to corrupted labels.
Systematic Shift Noise. Comparing Figures[8and Freveal a crucial pattern: training performance degrades with increasing shift noise levels, while evaluation performance remains nearly constant across all noise conditions. This contrast demonstrates that the model is not overfitting to the noise. If the model were memorizing the corrupted training labels, we would observe high training accu- racy accompanied by poor validation performance. Instead, the degraded training curves indicate the model resists fitting to the shifted labels, while the stable evaluation curves show it successfully recovers the true preference structure. This behavior confirms genuine robustness rather than mere memorization, the learned thresholds effectively compensate for systematic biases in the training labels.
Random Label Noise. Figures 10 and 11 show systematic degradation as noise level increases, with both training and evaluation performance declining progressively. At 100% corruption, the model fails to learn anything meaningful, as evidenced by training metrics that barely improve and evaluation performance at chance level. This type of noise is fundamentally more severe than sys- tematic shifts because destroys the ordinal structure entirely rather than merely miscalibrating it. The parallel degradation training and evaluation metrics confirms that the model cannot extract useful signal from purely random labels.
The training and evaluation dynamics support the findings in the benchmark tables, demonstrating that ordinal regression exhibits strong robustness under reasonable noise conditions. The framework handles systematic calibration emror with minimal performance impact, while degrading gracefully under moderate random corruption. These properties make ordinal regression well-suited for prac- tical deployment where annotation quality may vary but is unlikely be entirely random.
32

---

Published as a conference paper at ICLR 2026
ordinal pww.prrrancceccccmo eval/mean, absoluteeeeee emer
ordinal pwrformance...cc ovaljaccuracy. within
14
13
0.75
10k
203
10k
10k
50k 45k
podinal_erformance wwaljaccuracyy..ooooo weallaccurcccwitthiii. within
ordinal performaee htt.:p/aaaammmmmlccrrrreeee wwwljaccuraccy.v.... within_
reboliibn rebatilash rebsison rolusti
rebstiit rebaitiahhi
rebuisann miuutedi
63
0.83
0.55
0.2
35
Figure 9: Evaluation performance under systematic shift noise on clean validation data (left: mean absolute error, right: accuracy metrics). All noise levels achieve similar validation performance, demonstrating robustness to systematic label shifts.
indina__erformmane trwin//mannnnna maaanmmmann..mm aaboooute_erro
ordinal, www.orraaccceccccpmp www.n/jcccurccy.yitttotoot
re.nddee relestineceen rebulSorndom Xoandorenddrr
re.//die rebictneccn reautStrndom
35
Step
30k
4dk
30k
40k
peddonatprrormmmmcc...o wtwp://cttttnnncurrrrr.ooeore within
prdinat_peffrmmnce.noo http://mww..ttttooccuroooooiiiie
rebubCunncen
Figure 10: Training dynamics under random label noise (left: mean absolute error, right: accuracy metrics). Training performance degrades systematically with noise level, with 100% corruption preventing meaningful learning.
33

---

Published as a conference paper at ICLR 2026
redina_performmnncc.ooo wwll/men_abboluute.cmm www.aabbe_eerreeeoo rabrSStunsbre
ordinal www.ormancceecc.om withrr
-1disShasdat
3.55
365
See
10k
25k
30%
13k
10k
25k
30k
40k
Mndinal__erffrmman
http://pwwmmmeeeelccccooo...oee evallaccuracy_wtthe
ordinsi_perfomance performance...mmmmoo perfomance_evllaccureeee wwwp://ccccccccyoooooooooo within
ordinal
netes3Crando
03
0.5
0.2
25k
10k
303
Figure 11: Evaluation performance under random label noise on clean validation data (left: mean absolute error, right: accuracy metrics). Performance degrades progressively with noise level, with 100% corruption resulting in near- rearoma performance.
34


Reviews:
[{"review_id": "LkWZvcrddv", "reviewer": "Reviewer_xrft", "summary": "This paper proposes a framework for learning reward models from ordinal (Likert-scale) preference data, replacing ad‑hoc extensions of Bradley-Terry with discrete ordinal regression. The threshold between different levels are now also learnable parameters. Two families of losses are considered from the literature of ordinal regression: (i) a probabilistic ordered logit negative log-likelihood (NLL) over learned thresholds that partition reward differences, and (ii) an all‑threshold (AT) margin-based loss. The $L_2$ regularizer is introduced by considering the limit behavior of scaling going to infinity. Symmetric and asymmetric derivations are discussed. An optimization reparameterization is adopted to guarantee ordering. Experiments on HelpSteer2/3 training data with 7B-8B backbones evaluate on RM‑Bench and RewardBench, suggesting the NLL-Symmetric variant generally performs the best.", "questions": "1. Have you considered per-annotator (or random-effects) thresholds to capture personalized/individualized ordinal scales?\n\n2. Can you show performance vs. fraction of ordinal data and under controlled label noise?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Modeling of ordinal feedback from the perspective of discrete ordinal regression is natural. This enables the authors to adopt the literature already exists and makes more sense than simply heuristic choices.\n\n2. Theory is insightful. Theorem 3.1 justifies the choice of regularizer. Theorem 3.2 characterizes one sufficient condition of symmetry.\n\n3. The paper is clearly written and well organized.\n\n4. Empirical experiments show gains on several datasets/models. The NLL‑Symmetric variant is often best or competitive. The additional ordinal metrics go beyond binary accuracy and are also valuable points.", "weaknesses": "1. Joint learning of threshold and reward is still challenging. Scale identifiability remains under‑addressed. Regularizing thresholds cures the unbounded loss, but the joint scaling of reward head and thresholds can still be weakly identifiable. Anchoring strategies (e.g., fixing one threshold gap, adding mild L2 on reward head, or a temperature/variance parameter) and a short calibration section would improve interpretability.\n\n2. Positioning requires more contrast. The claim of being the first principled ordinal framework for reward modeling should be more carefully justified against existing works that already explore ordinal feedback for RMs (the paper itself cites several heuristic and non‑heuristic directions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AkXnTo0giU", "reviewer": "Reviewer_6o57", "summary": "The paper argues that treating human feedback as binary throws away signal when labels come in strengths (e.g., “slightly” vs. “much” better). It reframes reward modeling as ordinal regression on the reward difference, learning a set of thresholds that separate preference levels instead of hand-tuning margins or weights. The paper introduces two principled objectives—an ordered-logit NLL and an all-threshold (margin) loss—that jointly fit the reward and these thresholds, with L2 regularization to keep training stable and interpretable. Empirically, the paper shows that across different base model families and HelpSteer-style data, the approach trains RM with competitive performance as existing methods measured by RM/Rewardbench.", "questions": "1. Regarding weakness 1: could authors provide more motivations on why this method is better if it doesn't produce a better performance?\n2. Regarding weakness 2: could authors provide more theoretical support of why we see a experimental result like that among the 3 proposed objectives?\n3. See weakness 3:", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "originality: The paper introduces a new theoretical framework that uses **ordinal regression** on the reward difference to do reward model training. It introduces 2 major variations of such objective – ordered-logit NLL and an all-threshold (margin) loss, and also provide the practical recipe for training with these objectives stably.\n\nquality & clarity: The theoretical analyses and proofs are quite thorough and the experiment part comprehensively include different model backbones and benchmarks. The table presentation is clear and readable.\n\nsignificance: I find the idea of leveraging the difference in rated score as a proxy of strength is interesting. The paper also provides different potential methods on how this signal can be leveraged. It’d have more significance if the experimental result shows more obvious and consistent gain.", "weaknesses": "1. Overall weakness: It would be helpful if the authors provide more explanation of the method's motivation. From the experimental result itself (e.g. Table 2), the proposed method doesn't produce better benchmark results than other cited methods for quite a lot of the combinations tested. If that's the case, the it indicates that learning the ordinal relationship doesn't buy much gain in the performance? \n2. Overall weakness: The paper introduces 3 different objective designs. Although in the theory part, the authors provide some design principles for each objective, I think the paper lacks a general link between the theoretical proposal and empirical results. E.g. the experimental results indicate that symmetric NLL is better but I don't see a strong theoretical illustration of why that should be better. \n3. Overall weakness: Both datasets used in the paper are Helpsteer series of work. I wonder how common is the exact score rating in popular preference dataset. If the method needs to first assume that preference dataset contains the exact score, this may significantly limit where the method can be applied.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z6gJ3sE239", "reviewer": "Reviewer_5KEL", "summary": "The paper makes the case that human preferences are inherently ordinal and current reward modeling approaches do not incorporate this in a principled manner (these rely on naive margins or scaling terms). The authors introduce two different methods for this (a) the first uses the CDF over the rating levels to ensure that preference ratings learned are well calibrated with annotations in existing datasets, and (b) the second ensures that model score differences fall within the desired intervals (of levels). These are formulated as loss functions that are compared against the more common Bradley-Terry-based formulations with 3 LLMs (Llama 3.1 8B, Zephr 7B, Mistral 7B) on two datasets (HelpSteer 2 and 3) that come with rating levels for preferences. In general, ordinal reward modeling outperforms the baselines to demonstrate the validity of the proposed technique on real data and confirm the theory behind the design choices (asymmetric rewards, etc.).", "questions": "1) Are humans well calibrated in providing Likert-style preference judgments on pairs of examples? I imagine the proposed method is less tolerant to this noise than a more aggregated binary preference judgement. This is independent of your method, but I think the premise is that this is a natural way we make judgments (which I agree). It's just that I suspect it leads to more disagreement when we create datasets. \n\n2) Does the performance of the method vary with the granularity of levels provided i.e. K=3 vs 5 vs 7? Is this purely empirical/dependent on the model/dataset?\n\n3) Does HelpSteer filter out examples with high disagreement? I wonder if this works in favour of your model in that many examples might be high-margin? \n\n4) It's not a 'weakness', but there isn't a clearly preferred loss formulation among the variants proposed. It is not clear to me if this could be due to variance in optimization error or something like noise in the training data. Do you have a clear recommendation to practitioners using your technique?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1) It's very intuitive to incorporate a measure of 'strength' of preference into the reward modeling. The authors do a good job of creating a mathematical formulation to support this, backed by empirical results. \n\n2) The experiments are very well done with care taken for reproducibility and robustness of takeaways. \n\n3) The formulation reduces back down to something very much like DPO when you set two levels of preference, which has a nice 'backward compatibility' to existing methods.", "weaknesses": "1) The method assumes a consensus among annotators about different levels of ratings. Human preference datasets are already noisy, and this creates an additional source of noise (calibration among annotators), so while we do better at benchmarks, it is unclear if these reward models successfully capture the true distribution of user preferences well.\n\n2) I think some fine-grained error analysis and qualitative examples would really help provide some context for some of the results. For example, do the ordinal methods make fewer 'large-margin' errors (option was strongly preferred but scored wrongly) than the Bradley-Terry models? Can you plot something like a calibration curve for 'strength of preference' vs accuracy? Are there any examples where the labels are consistently flipped from incorrect to correct, or vice versa, by the ordinal ranking methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NE7i5LH8Ho", "reviewer": "Reviewer_oAHx", "summary": "This paper proposes a principled mathematical framework for training reward models from ordinal preference data, such as 7-point Likert scales. The authors argue that existing methods for handling such data—like adding margin terms or scaling factors to the standard binary Bradley-Terry loss—are ad-hoc heuristics.\n\nThe primary contribution is to reframe reward modeling with ordinal feedback as a discrete ordinal regression problem. From this established statistical framework, the authors derive two loss functions:\n1.  A probabilistic Negative Log-Likelihood loss based on the ordered logit model.\n2.  A margin-based All-Threshold loss inspired by large-margin classifiers.\n\nA key advantage of this framework is that the thresholds partitioning the reward-difference space are learned directly from the data, rather than being manually specified. The paper also provides two theoretical results: Theorem 3.1, which proves the necessity of regularization to ensure a finite optimal solution when learning thresholds, and Theorem 3.2, which provides a theoretical justification for using symmetric thresholds.", "questions": "See the weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.  **Principled Formulation:** The paper's core strength is its reframing the problem as discrete ordinal regression connects LLM alignment to a mature field of statistics and provides a solid theoretical foundation, moving beyond the intuitive but \"ad-hoc\" heuristics currently in use.\n2.  **Learned Thresholds:** The shift from manually-specified margins to data-driven, learned thresholds is a significant conceptual improvement. This makes the method more general and removes a key hyperparameter tuning burden from practitioners.\n3.  **Generalizability:** The provided extension to DPO in Appendix A also demonstrates the generalizability of the framework.", "weaknesses": "1.  **Notational Inconsistency in Formulation:** There is a critical notational contradiction in the paper's core formulation. Section 3.3 explicitly states that the threshold $\\zeta_0$ is *excluded*. However, the probabilistic model in Equation (14) implicitly requires $\\zeta_0$ to compute the probabilities for $z=-1$ (which uses $\\zeta_{z+1} = \\zeta_0$). As written, this makes the central NLL loss function ill-defined.\n2.  **Missing Ordinal Evaluation of Baselines:** The paper's entire motivation is that its method *better leverages ordinal information*. However, the main experimental results (Tables 1 & 2) only report binary preference accuracy, which fails to measure this. The paper introduces excellent ordinal metrics (MAE and Accuracy-within-k) in Appendix H, but reports them only for its own method. To substantiate the paper's core claim, it is essential to compare these ordinal metrics against the baselines.\n3.  **Potentially Unfair Baseline Comparison:** The baseline methods (Margin BT, Scaled BT) are highly sensitive to their specific hyperparameters (the margin values $m(z)$ or scaling/probability values $p(z)$). While Appendix F details the hyperparameter search for the proposed methods (e.g., learning rates, regularization $\\lambda$), it is not stated that the crucial $m(z)$ or $p(z)$ hyperparameters for the baselines were also tuned. If the authors used fixed, non-optimized values from prior work for the baselines while carefully tuning their own model, the comparison is not a fair one.\n4.  **Lack of Experimental Robustness:** All results in Tables 1 and 2 are reported from a single experimental run (using \"fixed random seed 0\"). Given the known high variance in LLM fine-tuning, single-seed results are insufficient to make strong claims of superiority, especially when many of the reported gains are small (1-3%). Reporting the mean and standard deviation over multiple seeds is necessary to demonstrate robust and statistically significant improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)