Learning Ordinal Probabilistic Reward from Preferences
Authors: Longze Chen
Lu Wang
Renke Shan
Ze Gong
Run Luo
Jiaming Li
Jing Luo
Qiyao Wang
Min Yang
Venue: ICLR 2026 Poster (2026)
Keywords: Reward Modeling
Large Language Models
RLHF
Primary area: foundation or frontier models, including LLMs

Abstract: Reward models are crucial for aligning large language models (LLMs) with human values and intentions.
Existing approaches follow either Generative (GRMs) or Discriminative (DRMs) paradigms, yet both suffer from limitations:
GRMs typically demand costly point-wise supervision, while DRMs produce uncalibrated relative scores that lack probabilistic interpretation.
To address these challenges, we introduce a novel reward modeling paradigm: Probabilistic Reward Model (PRM).
Instead of modeling reward as a deterministic scalar, our approach treats it as a random variable, learning a full probability distribution for the quality of each response.
To make this paradigm practical, we present its closed-form, discrete realization: the **Ordinal Probabilistic Reward Model** (OPRM), which discretizes the quality score into a finite set of ordinal ratings.
Building on OPRM, we propose a data-efficient training strategy called **Region Flooding Tuning** (RgFT).
It enables rewards to better reflect absolute text quality by incorporating quality-level annotations, which guide the model to concentrate the probability mass within corresponding rating sub-regions.
Experiments on various reward model benchmarks show that our method improves accuracy by **2.9% ～ 7.4%** compared to prior reward models, demonstrating strong performance and data efficiency.
Analysis of the score distribution provides evidence that our method captures not only relative rankings but also absolute quality.

Published as a conference paper at ICLR 2026
LEARNING ORDINAL PROBABILISTIC REWARD FROM PREFERENCES
Longze Chen 1:2
Lu Wang Renke Shan
Ze Gong
Run Luo
Jiaming L
Jing Luo- Qiyao Wang!.2 Min Yang!.
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences
2 University of Chinese Academy of Sciences Ritzz-AI
4 Shenzhen University of Advanced Technology
{12 , chen2, ze gong, min. yang @siat ac. cn
ABSTRACT
Reward models are crucial for aligning large language models (LLMs) with human values and intentions. Existing approaches follow either Generative (GRMs) Discriminative (DRMs) paradigms, yet both suffer from limitations: GRMs typically demand costly point-wise supervision, while DRMs produce uncalibrated relative scores that lack probabilistic interpretation. To address these challenges, we introduce a novel reward modeling paradigm: Probabilistic Reward Model (PRM). Instead of modeling reward deterministic scalar, our approach treats it as a random variable, learning a full probability distribution for the quality of each response. To make this paradigm practical, we present its closed-form, discrete realization: the Ordinal Probabilistic Reward Model (OPRM), which discretizes the quality score into a finite set of ordinal ratings. Building on OPRM, we propose data-efficient training strategy called Region Flooding Tuning (RgFT). It enables rewards to better reflect absolute text quality by incorporating quality- level annotations, which guide the model to concentrate the probability mass within corresponding rating sub-regions. Experiments on various reward model benchmarks show that method improves accuracy by to prior reward models, demonstrating strong performance and data efficiency. Analysis of the score distribution provides evidence that our method captures not only relative rankings but also absolute quality.
compared
INTRODUCTION
Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal technique for aligning Large Language Models (LLMs) with human values and intentions (Achiam et al., 2023; Ouyang et al., 2022). As critical component of the RLHF process (Bai et al., 2022), the reward model is trained to assign scores that quantify the degree of alignment between the model's 's outputs and human preferences. Recent advances (Guo al., 2025a; Lightman et al., 2024) have shown that well-designed reward signals, whether applied during training or inference, can significantly enhance LLM performance across diverse domains (Shao al., 2024: Luo et al., 2025; Jin al., 2025; Wang et al., 2025b; Li et al., 2025). However, learning a reward model that can accurately capture human preference signals remains significant challenge (Gao et al., 2023; Sun et al., 2025a; Zhong et al., 2025). Most recent efforts typically follow either the generative or discriminative paradigm, yet both approaches exhibit inherent limitations that hinder their effectiveness practice.
Discriminative Reward Models (DRMs), which append an MIP- value head to base model, are commonly optimized with the Bradley-Terry Terry objective to output a scalar reward (Liu et al., 2024b; Cai et al., 2024; Lou et al., 2024). A key limitation of this paradigm is that its reward scores reflect only relative preferences, not intrinsic quality. It indicates that one response is preferred but fails to explain why, making it difficult to establish trusted acceptance threshold to discern high-quality responses from low quality ones. In response, Generative Reward Models (GRMs) have emerged (Mahan
Corresponding authors.

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of OPRM with baseline reward models across multiple dimensions. Margin Sensitivity (whether distinguish samples with subtle preference differences), Require Training (whether requires training on preference data), Value Head Free (whether eliminates the need for additional value head), and Input Flexibility (whether supports rating single and multiple responses).
Basetines
Input Format
Output Format
Margin Sensitivity
Require Training
Head
Value
Input Flexible
Bradley-Temt (Bradley & Terry, 1952)
Single Response
Continuous Score
PairRM (Jiang al., 2023)
Response Pairs
Contimnous Score
Cloud (Ankner al., 2024)
Single Response
Critiuee Continuous Score Discrete Score
L.M-as-aJJddee (Zheng al., 2023)
Response Pairs
K
Pointwise GRM (Liu et al., 2025h)
Single Response
Critique + Discrete Score Continuous Score
OPRM (Ours)
Single Response
et al., 2024; Zhang et al., 2024). These models leverage the native generative capabilities of LLMs to produce Chain-of-Thought critiques before rendering a preference judgment, conceptually aligning with the LLMMaa--aJJdge paradigm (Zheng et al., 2023). While GRMs offer superior interpretability through their critique generation, they often rely on rigid pairwise input formats that limit flexibility in Best-of-N (BoN) scenarios. Moreover, achieving performance comparable to DRMs frequently requires costly pointwise supervision for calibration, substantially increasing the annotation burden. Consequently, the field faces a critical trade-off: choosing between the efficiency of DRMs and the interpretabilityy of GRMs, with neither approach offering a complete solution.
To transcend this trade-off, we introduce novel reward modeling paradigm: Probabilistic Reward Model (PRM). Instead of approximating rewards with a deterministic scalar value like the Bradley- Terry model (Bradley & Terry, 1952), PRM reframes the task as learning full probability distribution over the reward space. Since learning this continuous distribution is computationally intractable, we translate it into discrete realization: Ordinal Probabilistic Reward Model (OPRM). Specifically, OPRM discretizes the reward space into a finite set of ordinal ratings (Liu et al., 2025a; Wang et al., 2025a), thereby replacing the intractable integration with a closed-form summation that makes paradigm more practical. Thus, OPRM resolves the core trade-off. By providing a full reward distribution, it unlocks richer interpretability and uncertainty estimation than DRMs, while its flexible input format enables efficient scoring of single or multiple responses, making it better suited than GRMs for modern evaluations like Best-of-N (BoN). Table summarizes the advantages of OPRM over existing reward modeling baselines.
Building upon the OPRM paradigm, we further propose Region Flooding Tuning (RgFT), a novel training strategy designed to calibrate the reward distribution to reflect absolute textual quality. The core principle of RgFT is leverage quality-level annotations (i.e., good, normal, and bad) on preference data. Rather than optimizing over the full distribution of ordinal ratings, RgFT guides the model to concentrate probability mass within rating sub-regions corresponding to the quality-level labels. While simply restricting scores to specific quality intervals can lead to optimization stagnation due to constant gradients, RgFT floods these rigid constraints into a triangular probability landscape. This restores the gradient incentives, guiding the model to not only locate the correct quality region but also maximize the preference margin by pushing scores towards the extremes of their respective ranges. Critically, RTFT facilitates semi-supervised training by jointly leveraging mixture of quality-labeled and preference-only data, obviating the need for costly large-scale annotation.
Before delving into details, we summarize our contributions as follows:
We propose novel reward modeling paradigm, the Ordinal Probabilistic Reward Model. By learning a full probability distribution for a response's 's quality, OPRM mitigates the core trade-off between the efficiency of DRMs and the interpretability of GRMs.
We design a data-efficient training strategy, Region Flooding Tuning, which grounds the reward distribution in an absolute quality scale by guiding the model to concentrate probability mass within correct rating sub-regions.
We conduct extensive experiments four benchmarks covering over tasks, demonstrating the effectiveness of OPRM in precise confirm that RgFT significantly improves the accuracy, robustness, and interpretability of OPRM.
reward modeling across diverse scenarios. Additional studies
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Discriminative Reward Model. Discriminative reward model typically consists of a base model and MLP-based reward head (classifier) that outputs scalar score for given input. These models are commonly trained using the Bradley-Terry (BT) (Bradley & Terry, 1952) loss to maximize the reward margin between chosen and rejected responses. While the core BT loss remains standard component, considerable research has focused on enhancing data quality and refining the modeling framework (e.g. Skywork-reward (Liu al., 2024b), IntermLM2-rewan (Cai et al., 2024), Helpsteer2- preference (Wang et al., 2024b), QRM (Dorka, 2024), URM (Lou et al., 2024), CLoud (Ankner al., 2024). ArmoRM (Wang et al., 2024a). and PURM (Sun et al., 2025b)), further boosting DRM performance. Nonetheless, these methods are limited to learning a pairwise ranking, yielding scores that are unbounded and difficult to interpret. In contrast, our approach learns probabilistic distribution over scores, which enables more reliable and calibrated outputs.
Generative Reward Model. Generative reward models directly leverage LLM-generated outputs
to evaluate preference, which is aligned with the LLM-as-a-Judge paradigm (Zheng et al., 2023). These models output of-tin-of-though reasoning (critiques) before generating preference judgments (e.g., Critic-RM (Yu al., 2024), PROMETHEUS (Kim al., 2023), CLoud (Ankner et al., 2024), GenRM (Zhang et al., 2024; Mahan et al., 2024), Synthetic Critique (Ye et al., 2024), and RISE (Yu al., 2025)), enhancing the interpretability of the reward signals. Recent advances have employed reinforcement learning to construct reasoning-based reward models (SPCT (Liu et al., 2025b), RM- RI (Chen al., 2025b), (Whitehouse al., 2025), RRM (Guo et al., 2025b), and JudgeLRM (Chen et al.. 2025a)), demonstrating promising scalability in inference- time computation. However, these approaches often struggle outperform DRMs under computational constraints. Conversely, our approach achieves comparable efficiency and performance while preserving interpretability.
Ordinal Regression and Distribution Learning. Deep ordinal regression has evolved from simple continuous discretization (Fu et al., 2018; Rothe et al., 2018) to distribution ordering learning (Wang et al., 2025a). Prominent methods like SORD (Diaz & Marathe, 2019), ALDL (Li et al., 2022), and POE (Li et al., 2021) model label distributions or latent uncertainty capture ordinal relationships effectively. While OPRM draws inspiration from these probabilistic frameworks to construct reward distributions the LLM vocabulary, it is conceptually distinct from RLHF approaches utilizing ordinal feedback (Liu et al., 2025a). Prior RLHF works typically focus on refining the granularity of input supervision (e.g.. "significantly better") for continuous regressors. In contrast, OPRM enforces ordinality within the output representation space, treating rewards as discrete variables on statistically ordinal scale anchored to semantic quality, thereby bridging distributional ordinal regression with pairwise preference optimization.
3 PRELIMINARIES
Preference data annotation. To annotate the preference data, the SFT model π SFT is given prompts two distinet outputs (y), 1/22 SfST x)) These output pairs are then presented to human labelers, who express their preference for one output. This preference can be denoted as ye > Yr x, where Ve and y, represent the chosen and rejected outputs, respectively, from the pair (V1, y2).
Standard Bradley-Terry Reward Modeling, Following the Bradley-Terry model (Bradley & Terry, 1952), we model the probability of preferring response Ve over y/+ based on their underlying scalar rewards, which are provided by reward function rw (2 v). This preference distribution is formulated as follows:
exp (r Wc))
(w > Dr ax)
exp (r, Mc)) exp (ro(E, V+)) (r (z, ye) To (2, y.)),
which is the logistic function. Treating the problem a binary classification task yields the negative log-likelihood loss function:
C(rv) = - -I [log Pv (Me > dr x)].
(2)
= - E Me. D.m. (log (r ye) - T (x (x 1r))
3

---

Published as a conference paper at ICLR 2026
dli MAR
LM Head
LLM
Figure The architectures of Ordinal Probabilistic Reward Model. Given problem and pair of responses, designated as chosen and rejected, the OPRM utilizes its language model (LM) head to obtain the ordinal rating probabilities for each response. A joint probability matrix is then constructed by computing the Cartesian product of these two sets of probabilities for optimization.
where dataset is composed of comparisons denoted as Drm = (-(+)) of LLMs, the network (2 y) is often initialized using the SFT model x STI (v x). It then incorporates an additional linear layer on the final transformer layer generate singular scalar prediction representing the reward value.
( D1 1==1 In the realm
4 ORDINAL PROBABILISTIC REWARD MODEL
In this section, we introduce Ordinal Probabilistic Reward Model, novel reward modeling paradigm that learns probability distribution over response quality. We begin by outlining the continuous form of our reward modeling optimization paradigm, the Probabilistic Reward Modeling ($ 4.1.. We then discretize this formulation into a tractable form, termed OPRM ($ 4.2) and conclude by presenting the complete training and inference pipeline for OPRM ($ 4.3).
4.1 PROBABILISTIC REWARD MODELING
Departing from the conventional Bradley-Terry reward model (Section 3), which estimates a single scalar value for each response, we propose reward modeling objective derived from Random Utility Model theory (Manski, 1977; Cascetta, 2009). Our objective enables the model to learn probability distribution over the quality of each response. Coneretely, we model the quality score of response for a given input as continuous random variable S. This random variable is supported on bounded interval [a, b] C R. Our reward model, parameterized by b learns the conditional probability density function (PDF) (s xr y) of this variable, where is realization of S, This density must satisfy S. Po x, 1) ds = 1.
Given preference pair (ye, V/)) with chosen and a rejected response, we model their quality scores as two independent random variables, S. and S,. Their scores are drawn from the distributions defined by their respective conditional PDFs: 8c (- x, Nc) and 8, ~ p+ xr, yr). The probability of the preference Be > Yr is then modeled as the probability that the score of the chosen response exceeds that the rejected one:
Polye > Vr x)) Es..s, [1(se sc)
where (-)) denotes the indicator function. Expanding the expectation in integral form over the bounded interval [a, 6]. we obtain:
x) - 1/
[[ 1((e-
(u > yr 1z)
8x) (u Be) P P ye) ds, dsc
This expression corresponds computing the probability that random score sampled from the chosen response exceeds a random score from the rejected response, integrating over their joint distribution constrained the bounded interval. Since ( > s,) only when se > S, and otherwise, so it effectively truncates the integral domain to (a, 8e) for (sr Jr Ur). Thus, we can equivalently restructure the double integral as follows:
Pel (ye > 12) = 1 (s I., (e) (S/² (o x,, (, ds.
(5)
4

---

Published as a conference paper at ICLR 2026
Atacted
<Good, Badr
<Dood Normal>
<Normal, Bed>
Figure 2: Region Flooding Tuning. To ensure the correctness of the reward modeling, region flooding is applied to each of the three partition combinations, resulting in lower triangular form.
Finally, we can simply optimize the Eq. (5) by minimizing the negative log-likelihood loss. Notably, the Bradley-Terry model is a special case of the Probabilistic Reward Modeling framework, arising when the quality score distribution is constrained to unimodal Gumbel distribution with fixed shape parameters (see Appendix for a detailed proof). However, this objective lacks a closed-fom analytical solution and requires estimation through Monte Carlo sampling. This computational challenge motivates our transition from the continuous formulation to a more tractable discrete one.
4.2 FROM CONTINUOUS TO DISCRETE
To obtain a closed-form analytical solution, we adapt the continuous formulation in Eq. (5) by modeling the scores as discrete random variables over a finite set of ordinal ratings {a, a + I,...,b.. This yields the following closed- expression:
Po (ye > yr 1 x)) = Σ polse xr Me)
Σ prp x, Vr) Sy
(6)
As observed in Eq. (2), the Bradley-Terry model maximizes the score gap between chosen (ye) and rejected (y:) responses, creating a steep reward landscape with pronounced gradients beneficial for RL optimization. Similarly, our optimization objective in Eq. (6) inherits and generalizes this desirable property. By operating over full reward distributions instead of single scalars, our objective naturally shifts probability mass upward for chosen responses and downward for rejected ones, thereby widening their separation.
This intuition is formally supported by the gradient dynamics of the probability mass functions (PMFs). Specifically, the sensitivity of the objective J a Pu Vr x) with respect to the mass at score & is given by Eq. (7).
JJ = P(s < k), (p.(k)
aJ
= P(se > k). Op, (k)
These derivatives imply that increasing the probability mass for ye at score & is incentivized whenever Ye is likely be lower than k. Consequently, shifting mass from lower score & to higher score ++ 1 for the chosen response yields a strictly non- negativee gain proportional to (kk). This creates a consistent optimization pressure driving the distribution of ye. towards the maximum score b and y, towards the minimum score a. A detailed proof via gradient analysis can be found in Appendix C.
Ordinal Probabilistic Reward Modeling presents two key advantages: (1) Quantifying Uncertainty, the variance of the output distribution serves as a measure of model confidence- wide distributions for ambiguous comparisons indicate uncertainty, while sharp, peaked distributions reflect clear preferences, enhancing interpretability. Our method thus explicitly captures the inherent uncertainty in human preference judgments, a crucial aspect often overlooked by discriminative reward models. (2) Handling Annotation Disagreement, our method can represent multimodal score distributions (e.g.. Mixture of Gaussians), enabling it to capture disagreements among annotators. By explicitly capturing conflicting signals within the score distribution, our model becomes robust to the performance degradation often caused by inconsistent preference data (Sun et al., 2025a). This contrasts sharply with traditional methods like the Bradley-Terry model, which are restricted to unimodal preferences.
4.3 PIPELINE
Our training pipeline, illustrated in Figure 1, begins by formatting the preference data pairs (x, ye, Vr) (see Section into structured input using prompt template. The details of this template and criteria
5

---

Published as a conference paper at ICLR 2026
Table Overall results of different methods and models on four RM benchmarks. bold numbers indicate the best performance. Underlined numbers indicate the second best. The Overall* score is the average performance excluding Reward Bench due to known data contamination issues.
Model
Reward Bench
PPE-P
PPE-C
RMB
Overall
Overall
Skywork- wewwrt--eward-GGemm--2222- Reported Results of Public Models
93.8
56.6
56.6
60.2
66.8
57.8
81.5
62.8
58.5
65.7
67.1
62.3
Gemini-1.5-
86.8
66.1
59.8
56.5
67.3
60.8
ArmoRM-SB-vO.1
61.2
62.1
IntemL.M2-20--Rowaa
90.2
61.0
63.0
62.9
69.3
62.3
84.1
65.3
59.2
68.9
69.4
64.5
Claude-3.5-sonne-
84.2
65.3
58.8
70.6
69.7
63.2
Nemotron--43300-RRowae
92.0
59.3
60.8
69.9
70.5
63.3
GPT-4o
86.7
67.1
57.6
Reproduced Results of Baseline Methods From DeepSeek
LLM-as-a-Judge
83.4
58.8
67.8
62.6
DeepSeck-BTRRM-778
BTRM-227B
81.7
68.3
66.7
57.9
68.6
64.3
-Loud-Gemma-222777
82.0
67.1
62.4
63.4
68.7
64.3
SeepSeck-PairRMM-222
87.1
65.8
64.8
58.2
69.0
62.9
DeepScek-GRMM-27B-RT7
84.5
64.1
59.6
67.0
68.8
63.6
DeepSeck-GRRM-278
86.9
64.7
59.8
69.0
69.9
Results Our Method
OPRM-Qwen2.5-7B
87.8
61.1
61.3
OPRM-Qwen2.5-144
89.3
63.0
64.3
73.8
72.6
67.0
OPRM-QWen2.5-322
91.3
63.9
66.1
75.6
74.2
68.5
OPRM-QQwe2..-72BB
89.3
65.1
64.3
73.5
73.1
67.6
Results of Our Method (w/ Region Flooding Tuning)
OPRM-RgFT-QQwen2.5...
86.2
62.3
62.4
70.1
70.3
64.9(10.3)
OPRM-RgFT--Qwen255511
87.3
63.4
65.6
72.8
72.3(103)
67.3<11.3)
OPRMM-KgF7T-Qwen225--28
88.9
64.6
67.3
73.9(10.)
68.9(10.0)
OPRMM-RgFT-Qwwn2255-22
89.1
65.3
66.4
74.2
73.8/10.)
68.6(1100)
are provided in Appendix H. As the next step in the pipeline, following the parameter-free technique from prior work (Cui al., 2023), compute the distribution over quality score (a, (a.a+1,.... b) (where 6 2) by directly repurposing the LM head's 's vocabulary probabilities, thus obviating the need for a separate prediction head and avoiding any new parameters. In our implementation, set the quality score range from to 9 (i.e., 1, 9) The score distribution is then formed by directly extracting the vocabulary probabilities of the corresponding numeric tokens (i.e., to '9') This approach allows the model to directly leverage its inherent ordinal knowledge of numbers.
In summary, both the chosen and rejected inputs are fed into the LLM backbone and LM head, yielding the post-softmax vocabulary probability distributions p(w. ær Me) and p(w, | xr, Ur) at the last token position. The probabilitics of all numeric tokens are then normalized to form the distribution for our ordinal probabilistic reward modeling:
p(we- = 'i' [..e))
Pels: x. ) = p(u, = 'i' a.
Po(se = = Σ p(w. = '3" xx..))
Σ p(u; 'j' Mr)
Finally, we substitute the obtained (sc = x. Vc) and (s = Uz) into Eq. (6) and maximize Py y/ x) using the negative likeeeiiddood loss. See Appendix L for detailed computational overhead analysis.
During the inference stage, we simply input response y given prompt æ to obtain a quality score distribution (s(( x, v). We can derive scalar reward score through either argmax or weighted averaging. For discussion of other possible decoding strategies, see Appendix K.2. our subsequent experiments, we adopt the straightforward weighted averaging approach compute the reward score: ry (2, y) -' (s(s xr, y). avoiding the tie-prone argmax method.

---

Published as a conference paper at ICLR 2026
Table 3: Detailed results of different methods on the PPE Correctness benchmark. Bold numbers indicate the best performance. Underlined numbers indicate the second best.
Model
MMLU-Pro
MATH
GPQA
MBPP-Plus
IFEval
PPE Correctness
Results of Our Method
OPRM-Qwent -wen2.5-77
65.2
70.1
56.3
59.0
56.1
61.3
OPRM- OPRM-Qwen2.5-1 Qwen2.5-142
66.7
70.7
57.1
67.4
59.5
64.3
OPRM-Qwen2.5-322
71.2
73.2
57.9
66.2
62.2
66.1
OPRM-QWen2.5-72B
73.4
75.9
58.6
54.1
59.5
64.3
Results of Our Method (w/ Region Flooding Tuming)
OPRM-RgFF-QQwen2.555
64.8
71.6
55.9
63.0
56.8
62.4(11.)
WPRM--RgFT-Qwwe2.5-5
69.5
74.0
57.3
67.0
60.0
65.6/11.)
OPRM-RRFF--Qwen2.5-52
73.3
76.8
58.5
67.2
60.6
67.3
OPRM-RRFFT-QQwe2255-777
72.8
77.1
59.0
62.0
61.2
66.4(12.1
REGION FLOODING TUNING
While OPRM effectively captures relative preferences, precisely aligning its scoring distribution with absolute quality judgments presents a further challenge. To address this, we introduce Region Tuning (RgT), cost-effective method that enhances the model's fidelity absolute quality scores using minimal annotations ($ 5.1). Subsequently, we refine RgT to preserve the desirable properties (as detailed in Appendix C), culminating in final method: Region Flooding Tuning (RgFT) ($ 5.2).
5.1 REGION TUNING
Building upon the OPRM optimization objective from Eq. (6), which employs the finite set of ordinal ratings {1, .,, for all data, we introduce more fine-grained partitioning based on the absolute quality of each response, technique we term Region Tuning (RgT).
Specifically, we further partition the finite set into three quality levels, guiding the model to concentrate the probability mass within corresponding rating sub-region: Sbad {1, 21,2,3) Snormat {4, 5, 6). and Syood= {7,5 8, 9}. Consequently, for single preference data point consisting a chosen and a rejected response. there are six possible combinations of quality levels. These include pairs from dif- ferent levels, as well as pairs where both responses fall into the same level, denoted as <lehoven <rejected>: <good, normal>, <good, bad>, <normal, bad>, <good, good>, <normal, normals, <bad, bad>
This partitioning allows us to redefine the preference probability by conditioning on the quality levels of the chosen and rejected responses. Thus, the optimization objective is formulated as:
(ul > lehosen; Irejected)
(se x. Vc)
Σ
(0r
F. (r.:+)1 (8c > 8c)
St
El
(9)
Details on the partition of the semantic regions (Stud, Snomma- Sgood) are provided Appendix
5.2 FROM REGION TUNING TO REGION FLOODING TUNING
As shown in Figure 2. when letessen bejected, Eq. (9) optimizes equare-shaper joint probability region, resulting in constant partial derivatives ap objective no longer shifts the probability mass of the chosen response upwards and the rejected response downwards to increase their separation. This leads to the loss of desirable property of
7p.(k] and ap (p-(AJ In this case, the optimization
OPRM, as mentioned in Section 4.2 (see Appendix C and for a formal proof and analysis).
As shown in Figure 2, we propose region flooding to the optimized joint probability region, expanding into a lower triangular shape to preserve the desired property. As its expansion process closely resembles breadth-first search algorithm, term Region Flooding Tuning (RgFT). RgFT provides three key advantages: (1) Interpretabiliry, RgFT constrains the model to concentrate probability mass within the regions correspond to pre-defined quality levels, enabling reward scores to more accurately reflect the absolute quality responses. (2) Semi- semi-supervised Learning, RgFT supports semi-supervised training by combining quality-labeled data with preference-only data. (3)
7

---

Published as a conference paper at ICLR 2026
Table Detailed results of Qwen2.5-7B with different methods on the Role Play benchmark. Bold numbers indicate the best performance. Underlined numbers indicate the second best.
Method
Pair-Accuracy
Best-of-N
Best-of-N-plus
Worst-of-N
Overall
Random Baseline
50.0
25.5
31.3
68.7
43.9
Training on Role Play Data Only
BT Model
70.4
48.6
51.3
83.6
63.5
BT Model w/ Margin
71.0
49.3
52.3
84.2
64.2
OPRM (ours)
71.3
49.4
52.5
84.
64.3
OPRM-ReFT (ours)
72.1(-0.8)
50.7(113)
53.6(11.D)
85. 85.1(t10)
65.4(+1.)
Training Mixed Role Play and General- Domaii Data
BT Model
73.8
51.2
54.3
86.0
66.3
BT Model w/ Margin
75.3
53,4
55.7
87.2
67.9 9
OPRM (ours)
74.4
54.1
56.
87.8
68.1
OPRM-RgFT (ours)
75.8(11.4)
15.8
59.3(13.2)
89.9(12.1)
70.2
Customizaailltyy, RgFT allows for the flexible tailoring of rating sub-regions to their corresponding quality levels, making the strategy adaptable to diverse application requirements (see Appendix K.1).
6 EXPERIMENTS
6.1 EXPERIMENTAL SETUP
In our experiments, we curate a dataset of 130k samples for reward model training, drawn primarily from publicly available open-source datasets: Skywork Reward Preference 80K (Liu et al., 2024b) and UltraFeedback Binarized Preferences (Cui et al., 2023). We employ the @ series of models (7B, 14B, 32B, and 72B) (Team, 2024) the backbone for training the OPRM. We compare OPRM to different categories of baselines: Discriminative RMs, Generative RMs and DeepSeek-RM RM. Following prior work, evaluate the performance of different methods various RM benchmarks: Reward Bench (Lambert et al., 2024). PPE-Preference, PPE-Correctness (Frick al., 2024), and RMB (Zhou et al., 2024). We use the standard pair accuracy and Best-of-N evaluation metrics for each benchmark. Detailed information on the training preference data, baselines, benchmarks, and evaluation metrics provided in Appendix D.
6.2
MAIN RESULTS
As shown in Table 2, we compares the overall results of OPRM with different baseline reward models on RM benchmarks. We present the performance of OPRM with the reported results of public models and the reproduced results of baseline methods from DeepSeck. We observe that OPRM outperforms the baseline methods in overall performance, and achieves competitive results against. strong public RMs, such as 72B models surpass all prior leading reward models, improving upon the previous best result by 1.3%, 2.9%, and 188, respectively, despite being significantly smaller in scale. Moreover, the most significant performance enhancement observed on the RMB and PPE-Correctness benchmarks, which utilize Best-of-N evaluation to better reflect practical effectiveness on downstream tasks. We attribute the 32B model's superior performance over the 72B model to the exceptional zero-shot capability of Qwen2.5-32B. 5-32B. This enables it to outperform larger models on the RM benchmark without any fine-tuning. The more detailed numbers on RewardBench, PPE Correctness, and RMB are in Table 8, Table 9, and Table 10 in Appendix I.
and GPT-4o. Notably, the 14B, 32B, and
6.3 THE IMPACT OF REGION FLOODING TUNING
Building upon OPRM, incorporate RgFT for further experimentation. We train the OPRM- RgFT series of models using the preference data, further enriched with three defined quality level annotations (good, normal, and bad). Consistent with the advantages of R&FT descrihed in Section 5.
8

---

Published as a conference paper at ICLR 2026
6.3.1 FEWER ANNOTATIONS, BETTER RESULTS
As presented in Table and Table 3, our evaluation OPRM-RgFT on four RM benchmarks reveals notable performance divergence. On one hand, RgFT consistently improves performance across all model sizes on the PPE benchmarks. Notably, SPRM-RgFT--33 67.3% on the Correccrssssssss benchmark, surpassing all prior leading reward models. On the other hand, its performance on other benchmarks is inconsistent. We hypothesize that this discrepancy stems from biases introduced by our annotation strategy for general data (see Appendix J). This process, involving coarse Al annotation with simple manual correction, is effective for verifiable tasks with explicit correctness labels like PPE-Corress but likely introduces label noise for other tasks, Further supporting this claim, our subsequent experiments show that incorporating fine-grained manual annotations leads to consistent performance improvements.
achieves SOTA accuracy of
6.3.2 BEYOND ACCURACY, RELIABLE CALIBRATION
While high accuracy is desirable, reliable re- ward model must also provide calibrated confi- dence, particularly for probabilistic approaches. To strictly quantify this, we measure the Ex- pected Calibration Error (ECE) the Reward- Bench dataset. Given the inherent noise in fine- grained ordinal ratings, aggregate scores into semantic tiers (bad, normal, good) to compute robust ECE against ground truth labels verified by GPT-4o and humans. As shown in Table OPRM-32B significantly outperforms the baseline, reducing ECE by over 60%. Crucially, OPRM- RgFT-32B achieves a minimal ECE of 5.18%, an 80.6% relative reduction compared to the baseline. This demonstrates that RgFT not only boosts ranking performance but also effectively regularizes probability estimates, preventing overconfidence and ensuring alignment with empirical correctness.
Table 5: Comparison of accuracy and calibration (ECE-10) on the RewardBench.
Model
Acc (%)
ECE- 10 (%, J)
Qwen-2.5-32B
69.56
26.72
OPRM-32B
81.04
10.62
OPRM-RgFT-32B
90.90
5.18
6.3.3 TOWARDS HUMAN-ALIGNED SCORE DISTRIBUTIONS.
To evaluate the impact of Region Flooding Tuning abso- lute quality assessment, we analyze the score distributions produced by our models. Specifically, tinct datasets for this analysis: an Absolute-Good Set with 100 high-quality prompt-response pairs and an Absolute- Bad Set with 100 poor-quality pairs. These pairs are manually selected by experts based on multi-faceted evaluation across dimensions such as instruction follow- ing, factual accuracy, and helpfulness. We then score both datasets using three models: the baseline BTRM-32B, our base model OPRM-32B, and its RgFT-enhanced version. As illustrated in Figure the base OPRM-32B already exhibits a basic capacity for absolute quality assessment: within its [1, 9] scoring range, it generally assigns scores above 5 to good responses and below to bad ones. Cru- cially, OPRM-RgFT-322 significantly enhances this pability. The RgFT-enhanced model polarizes the score distributions, pushing scores for high-quality responses into the [7,9] range while confining low- quality ones to 3]. This increased separation makes the score itself more reliable and interpretable indicator of absolute quality. Case studies in Appendix M provide detailed scoring examples that further corroborate these findings and demonstrate the improved reliability of RgFT scores.
0.6
STRV
curated dis-
Density
0.2
STR9W-20
0P104-225
Density
-10
-0
10
Reward Score
Figure 3: Comparison of score distribu- tions for responses of high-quality (Top) and low-quality (Bottom).
6.3.4 SEMI-SUPERVISE DOMAIN ADAPTATION
To simulate practical applications and reduce annotation costs, we investigate RgFT's effectiveness in semi-supervised domain adaptation setting. Specifically, curated training set of 31K role-playing instances with quality- labels (see Appendix for annotation details) and a mixed dataset by combining these with equal volume of unlabeled general-domain preference data.

---

Published as a conference paper at ICLR 2026
OPRM-328
OPRM-328
OPRM-Rg
BTRM-32B
OFFEM--7BBAArggaaDDeeodd
OPRM-RgT-322 (wio Flooding)
100
100
80
90
90
75
700
65
60
60
80
80
50
55
50.8
RowardBeschh PPE-P
FPE-C
RMB
RewardBeachh PPE-P
PPE.C
RMB
Mar
cmaa
Mom
(a)
(b)
(c)
Figure 4: Ablation Study: (a) Assessing the superiority of OPRM over the BT Model. (b) Evaluating the efficacy Weighted Average Decoding. (c) Validating the necessity Region Flooding.
For evaluation, we build a test set of 500 questions, each with 5-10 responses, and designed three new metrics: Best-of-N (top-scoring good-level), Worst-of-N (bottom-scoring is bad-level), and Desttof-N-plus(top-ssees scoring is not bad-level). As shown in Table 4, we benchmark our models against BT and Mariiin baselines (see Appendix F for detailed formulas) under two settings: training on role-play data only, and the mixed dataset. In both settings, OPRM surpasses the baselines, and OPRM-RgFT further improves upon OPRM. Crucially, incorporating unlabeled general-domain data significantly boosts the performance of OPRM and OPRM- RgFT from 64.3% to 68.1% and 65.4% to 70.2%, respectively. This demonstrates that RgFT can effectively leverage unlabeled preference data in semi-super manner, offering cost-effective path for domain adaptation.
6.4 ABLATION STUDY
We perform an ablation study to isolate and evaluate the key components of our OPRM and RgFT frameworks. As depicted in Figure 4, the removal of any single component detrimentally affects the overall performance.
Effectiveness of OPRM Loss. Replacing the proposed OPRM loss with the standard Bradley-Terry (BT) loss on an identical Qwen-32B model incurs consistent performance penalty of 1.1% to 2.1% across all benchmarks (Figure 4(a)). This result empirically substantiates the advantage of modeling the reward as an ordinal variable rather than strict pairwise preference.
Impact Decoding Method. We contrast our weighted averaging decoding strategy with a naive Argmax baseline, which greedily selects the highest-probabine token. As shown in Figure 4(b), Argmax leads substantial 7% to 17.5% performance drop. This is primarily because Argmax fails to capture fine-gra fine-grained quality differences, thereby generating excessive number of ties.
Necessity the Flooding Mechanism. The flooding mechanism designed to create desirable lower triangular score regions (see Appendix C). Removing it resulted in 1.0% to 2.9% performance drop on the PPE Correctness benchmark (see Figure 4(c)). The degradation is most pronounced when distinguishing between marginally different responses, confirming the mechanism's 's critical role.
7 CONCLUSION
In this paper, we propose Ordinal Probabilistic Reward Model, novel paradigm that learns full probability distribution over an ordinal reward space. To better anchor these rewards to absolute quality, we further proposed Region Flooding Tuning, a training strategy that leverages quality-level annotations to calibrate the model's probability distribution. Extensive experiments on four diverse reward modeling benchmarks show that our approach consistently improves performance by 2.9% to 7.4%. Furthermore, detailed analysis reveals that OPRM superior to the conventional Haudeey-Teryy model and that RgFT is crucial for discerning fne-grained quality differences. We believe OPRM with RgFT offer a powerful framework for developing more accurate and reliable reward models, a critical step towards building more capable and aligned large language models.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Min Yang was supported by National Key Research and Development Program of China (2044FF09082000, National Natural Science Foundation of China (Grant No. 62376262), and Natu- ral Science Foundation of Guangdong Province of China (2024A1515030166 2025B1515020033),
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.065: 08774, 2023. URL https: /ttp//aavivv//.////////eee/.aava org/ abs 2203. 08774.
Zachary Ankner, Mansheej Paul, Brandon Cui. Jonathan D Chang, and Prithviraj Ammanabrolu. Critique-out-loud reward models. arXiv preprint vxXiv:2408.11791 2024. URL https://
arxiv. org/ abs 2408 11791.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05882 2022. URL
org/ alss 2204 05862.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internm22 technical report. arXiv preprint arXiv:2403. 2024. URL https: //wpi//aavvvv//aa//eee/eaeeawwva org/ als /2003443.00000 17297.
Ennio Cascetta. Random utility theory. In Transportation systems analysis: models and applications, pp. 89-167. Springer, 2009.
Nuo Chen, Zhiyuan Hu, Qingyun Zou, Jiaying Wu, Qian Wang, Bryan Hooi, and Bingsheng He. Judgelrm: Large reasoning models as a judge. arXiv preprint 2500:25540000500, 2025a. URL
https: //aaxi/aaivvv////aa//eaa org/ abs// /bs/2504 2504. 00050.
Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, et al. Rm-rl: Reward modeling as reasoning. arXiv preprint 02Ii:22505.02887, 2025b. URL https: /ttp:v//axxiii oww.o/////aooo.
02387.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedbak:: Boosting language models with high-quality feedback. arXiv preprint ar arXiv:2310.01377 2023. URL https: arxiv org/obgg/// 2210. 01377.
Raul Diaz and Amit Marathe. Soft labels for ordinal regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4738-4777 2019.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint 1tti:23355.123333 2023. URL https: /ttp///aaavvv//e///e//eeeeeaaele org /bs/ 2303.
Nicolai Dorka. Quantile regression for distributional reward models in rlhf. arXiv preprint arXiv:2409. 101644 2024. URL http:tttt://aaet http://www....///a///...........
Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios N Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. How to evaluate reward models for rlhf. arXiv preprint arXiv:2410. /4872, 2024. URL https: //arxivv..v/////////a///eer/aaar org//bogss......ooo 2410 14872.
Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2002-2011, 2018
11

---

Published as a conference paper at ICLR 2026
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023. URL https:
//proceedings mlr. presssppes////eee v202//22222//// gaa./h/ao3..... html.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. URL https: /atxiv//aiiiv/v/ia org aabs///////////// /400 21783.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint urXiv:2501.12945, 2025a. URL https:////// //apxiv./ erq/abs/2501 12948.
Jiaxin Guo, Zewen Chi, Li Dong, Qingxiu Dong. Xun Wu, Shaohan Huang, and Furu Wei. Reward reasoning model. arXiv preprint arKiv:2505. 14674, 2025b. URL https:s/////x org/ /bb// 2505. 14674.
Shengding Hu, Yifan Luo, Huadong Wang, Xingyi Cheng, Zhiyuan Liu, and Maosong Sun. Won't get fooled again: Answering questions with false premises. arXiv preprint arXiv:2307.02394 2023. URL https: //axxvv org abs 2200 02394.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Adítya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. URL https: / aww.i. org/ abs 2417.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLM-blender: Ensembling large language mod- els with pairwise ranking and generative fusion. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Anmual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 14165-1411 14178, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 1/265/////222..cllllll 792. URL https: / /t///alaann//t http///caannthooooyooooooo owt.org/////// ww..022..clloonn... 792/-
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Serean Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training iims to reason and leverage search engines with reinforcement learning. arXiv preprint arXhv:2503.09516 2025. URL https: www.ii.orrrr.ooo abs /2503. 09516.
Seungone Kim. Jamin Shin. Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Scongjin Shin, Sungdong Kim, James Thorn,, et al. Prometheus: Inducing fine-grained eval- uation capability in language models. In The Twelfth International Conference on Learning Representations, 2023. URL https: https://////eeeeeeeeeee /wwpnreeeeeeeeeee//o/ee net/ fne:/forummid-BBddataae
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint 2rXiv:2403.13788 2024. URL https: http://axxvvv.oooo abs/ 0tts/2403.133 13787.
Jiaming Li, Longze Chen, 7e Gong, Yukun Chen, Lu Wang, Wanwei He, Run Luo, and Min Yang. Implicit actor critic coupling via a supervised learning framework for rlvr. arXiv preprint arXiv:2509.02522 2025, URL https: http://aaxvvvv.////ee
02522.............
Qiang Li, Jingjing Wang, Zhaoliang Yao, Yachun Li, Pengju Yang, Jingwei Yan, Chunmao Wang. and Shiliang Pu. Unimedaal-onceetrrated loss: Fully adaptive label distribution learning for ordinal regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20513-20522, 2022.
Wanhua Li, Xiaoke Huang, Jiwen Lu. Jianjiang Feng, and Jie Zhou. Learning probabilistic ordinal embeddings for uncertainty-awee aware regression. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 13896-13905 2021.
12

---

Published as a conference paper at ICLR 2026
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2024. URL httpt:tt/pppeeeeeeeeeeee
? i d= 1006000...
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021. URL httph:t//txxiiiiie /220099 /2109.
07958.
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture- of-experts language model. arXiv preprint ar arXiv:2405.0444. 04444444, 2024a. URL org/aba 2405. 04434.
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling llms. arXiv preprint 12Xi::441011855/1 2024b. URL httpt: //axxiv./// org/ abs / 2410 18451.
Shang Liu, Yu Pan. Guanting Chen, and Xiaocheng Li. Reward modeling with ordinal feedback: Wisdom of the crowd. In Pruceedings of the 42nd International Conference on Machine Learning, 2025a. URL https::/prroeeeingg/
mlr. press/v26777 /at.67/////appp html..
Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling. arXiv preprint urXiv:2504.02495. 2025b.
URL wwthhhtps////ae /ttpsv////aavvv/// org/ abs/2504 02495.
Xingzhou Lou, Dong Yan, Wei Shen, Yuzi Yan, Jian Xie, and Junge Zhang. Uncertainty-awart reward model: Teaching reward models to know what is unknown. arXiv preprint arXiv:2410.00847
2024. URL https::////t //arxii.a.iiiiaa/aa
org.oo/goaoooooo hatg///b//// haa//abb/221000 /2100. 00847.
Run Luo, Lu Wang, Wanwei He, Longze Chen, Jiaming Li, and Xiaobo Xia. Gui-rl: A generalist r1-style vision-language action model for gui agents. arXiv preprint arXiv:2504. 10458, 2025.
URL https::///eett /ttppss//aaxv.ooo//o/
org/ abs/ 10458.
Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv preprint
2IXi::2410.12832 2024. URL https: / arxiv org/ http//b/./11.......
Charles F Manski. The structure of random utility models. Theory and decision, 8(3):229, 1977.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- http://oooeeeooo/ooooooccccocccc 27744, 2022. URL https: / ngs neuri wesrrsss.......... ec wwwpr/apppppp/eee fi secerrrffllspeeeeeee /aper// 2000////00000E010111101000000000
html.
Rasmus Rothe, Radu Timofte, and Luc Van Gool. Deep expectation of real and apparent age from single image without facial landmarks. International Journal of Computer Vision, 126(2):144-157) 2018.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models. arXiv preprint Atl::2402.03300 2024. URL
attts:://ariii: org/ /orgg//bs/2222 03300.
Hao Sun, Yunyi Shen, and Jean-Francoii Ton. Rethinking reward modeling in preference- large language model alignment. In The Thirteenth International Conference on Learning Representa- tions, 2025a. URL https:///// // openreviiew net/ forum? ddbf 1 E1 0qm.
Wangtao Sun, Xiang Cheng, Xing Yu. Haotian Xu, Zhao Yang, Shizhu He, Jun Zhao, and Kang Liu. Probabilistic uncertain reward model. arXiv preprint arXiv:2503.224480 2025b. URL
ittps://arxiv org/ 2200.22280........ 22480.
13

---

Published as a conference paper at ICLR 2026
Gemini Team, Petko Georgiev, Ving lan Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang. et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXhv:2403.05530, 2024. URL
htttss://art www.riivvo..o...o..ooooooooooort alss 2.....055555 05530.
Qwen Team. Qwen2 technical report. arXiv preprint arXiv:2407.10671 2024. URL
attpss//ietaaostaai..oo---oooooe wwt.aaaaaaaaaaacceeeeeteeeteteee www.:aaaaaaaa----- -twpe//eeeeoallll-------------a-
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 10582-10592. Miami, Florida, USA, November 2024a. Association for Computational Linguistics. doi: 10.18653/v1/2024 findinns-nediinnnnn emnlp.620. URL https://aclantho.t.t org/2024. f nd iindinns-eennnn
620/.
Jinhong Wang. Jintai Chen, Jian Liu, Dongqi Tang. Danny Z. Chen, and Jian Wu. survey on ordinal regression: Applications, advances and prospects. arXiv preprint (2Xiv:2503.00953 2025a. URL https::/aate org/ abs 00952.
Minzheng Wang, Yongbin Li, Haobo Wang, Xinghua Zhang, Nan Xu, Bingli Wu, Fei Huang, Haiyang Yu, and Wenji Mao. Adaptive thinking via mode policy optimization for social language agents. arXiv preprint 20xh:22505.02/56. 2025b. URL https: arwiv.iiiivo/oo org/ abs / 2505. 02 156..
Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. Helpsteer2-preferenn Complementing ratings with preferences. arXiv preprint arXiv:2410.01257, 2024b. URL httpt:tttt//ttttteee org abs 2410.01257.
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint arXiv:2406.08673, 2024c. URL https:// arxiv. www.orgggaa....... abs / 2406. 68373.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. arXiv preprint
arX.v:2109.01652, 2021. URL
abss/109s 01652.
Chenxi Whitehouse, Tianlu Wang, Ping Yu, Xian Li, Jason Weston, Ilia Kulikov, and Swarnadeep Saha. J1: Incentivizing thinking in llm-as-a-jjude via reinforcement learning. arXiv preprint arXiv:2505.10320, 2025. URL
wrg./ii.oo...ooooooo.......o.... abs / 2505 10320.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. Wizardlm: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations, 2024. URL
httpss//ppenreviieee http///ppeeeee.ccccoee/e. net/ forumrrrmo
Minghao Yang, Chao Qu, and Xiaoyu Tan.
Inf-orm-llmmas 1- 1-70B] (https:
2024.
URL
/ hhttps://buggingface co/ cwt.cclc/////////c/////c...lllat -RMMIIIN---O-LM---
ORM- amama 3. 107))
Zihuiwen Ye, Fraser Greenlee-Scott, Max Bartolo, Phil Blunsom, Jon Ander Campos, and Matthias Gallé. Improving reward models with synthetic critiques. arXiv preprint wrXiv:2405.20850. 2024.
URL attpp:://zxiiss org/ abs/ /4002 20850.
Jiachen Yu, Shaoning Sun, Xiaohui Hu, Jiaxu Yan, Kaidong Yu, and Xuelong Li. Improve llm- as-a-judge ability as a general ability. arXiv preprint arXiv:2502.1 16689, 2025. URL https:
Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan. Chao Zhang, al. Self-generated critiques boost reward modeling for language models. arXiv preprint arXiv:2411.16646, 2024. URL https: //arxiv wwtp:iiwwrggab///aeeooo 16646.
14

---

Published as a conference paper at ICLR 2026
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408. 15240, 2024. URL https: http///xxivv.rr/./ abs 22408.15240
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023. URL https: /ttpoe/ig///occiingggs.. wttp://sssssruri.cce...eco hash/91 91 f1 1a1.1a1111888888 A www.:eetee--tttttt-aaaaeae-e Benchmarks. html.
http:///pproeeedos..ne....oooo.. cc
/iless/appr//222
Jialun Zhong, Wei Shen, Yanzeng Li. Songyang Gao, Hua Lu, Yicheng Chen, Yang Zhang, Wei Zhou, Jinjic Gu, and Lei Zou. A comprchensive survey of reward models: Taxonomy, applications, challenges, and future. arXiv preprint arXiv:2504. 12328, 2025. abs/2504.122 123228.
Enyu Zhou, Guodong Zheng, Binghai Wang, Zhiheng Xi, Shihan Dou, Rong Bao, Wei Shen, Limao Xiong, Jessica Fan, Yurong Mou, al. Rmb: Comprehensively benchmarking reward models in Ilm alignment. arXiv preprint arXiv:24 2024. URL httttp:://axxxi.or///aa. 2410. 09893.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
A
THE USE OF LARGE LANGUAGE MODELS (LLMS)
We utilize Large Language Models (LLMs) to aid in the writing and polishing of this manuscript. Specifically, LLMs are employed to correct grammatical errors, improve sentence structure, and enhance the clarity and conciseness of the text. This process primarily applied the Introduction, Related Work, and Appendix sections. All scientific contributions, methodologies, and conclusions presented in this paper are the original work the authors. The LLMs serve solely as a writing- enhancement tool.
B BRADLEY-TERRY AS A SPECIAL CASE OF PROBABILISTIC REWARD MODELING
In this section, we demonstrate that the Bradley-Terry model for pairwise preferences can be derived from more general probabilistic reward modeling framework under a specific set of distributional assumptions.
Let (s x. y) denote the probability density function of a score s assigned a response y given context I, where the scoring mechanism is parameterized by vb. Consider two responses for the same context x: a chosen response Ye and rejected response yr- Let 85 and 8, be the random variables for their respective scores, with distributions Pip (8c. x; Vc) and pvp (8 xr, Ur). We assume and are conditionally independent given æ, yes 3r-
The probability that Ye is preferred over y/. denoted (e > Mr x), is the probability that the score of the chosen response is greater than that of the rejected one, i.e., P(se > 8,). This can be expressed generally as:
Pa > Dr (x) = f (8c v. ye) ( (o I ar, U1) Wr) dsc.
(10)
We show that this general formulation reduces the Bradley-Teny model under specific assumptions. For clarity, we first establish the functional form of the Bradley-Tery model in terms of the sigmoid function.
Lemma B.1 (Bradley- Terry Model in Sigmoid Form). The Bradley-Terry (BT) model, which defines the preference probability based underlying quality scores T (2, Ve) and r (z. y.) as
exp(r Mc))
Per(uc > 1x) exp(r=(x, ve)) + exp(r=(a,
(11)
equivalent to the sigmoid function of the difference in scores:
Per Y: |x) o(rs(x, ye) - r (z, yc)),
(12)
where (22) = 1/(11 +-= the standard logistic sigmoid function.
Proof. We start from the standard definition of the BT model and manipulate it algebraically. By
dividing the numerator and the denominator of Eq. (equation 11) by exp(ro(a,, ur)), obtain:
(xp(ry(z.v.)) expp:rrr))))) (exp(ro(E, + +xp): -epp::))))xxxx(((ee(xr exp(
Po D1 xx)
exp(ro(x, (e) - Ur))
exp(r - (r)(2)))))) +1
exp(ro(x, - re (x)) exp(ra(x, (a (I,
16

---

Published as a conference paper at ICLR 2026
To bring this into the form of the sigmoid function (2)), we can divide the numerator and denominator by exp(ry(a; Vc) (2 y+)):
PaT > yr x)
1+exp(ru ra exp(re oo (2.
cxp(-(ra Dc) (1) + (r Me) (pl(,.1))..
This completes the proof of the lemma.
With this lemma, we can prove the main proposition.
Proposition B.2. The general preference probability Po(Ue > Y/ x) defined in Eq. (10) is equivalent to the Bradley- Terry model if the following assumptions hold:
1. The score difference A8 - -
follows a logistic distribution.
2. The mean of this logistic distribution is the difference of deterministic underlying quality scores: f (z, Bc) (z, Wr).
3. The scale parameter of the logistic distribution is unity 1).
Proof. The preference probability is the probability that the score of the chosen response exceeds that of the rejected one. This can be expressed in terms of the score difference random variable Asp Sc Sy:
PolNo yr x)) P(se > 8,) P(As# > 2).
(13)
Assumption 1 states that Ase follows a logistic distribution. The cumulative distribution func- tion (CDF) of logistic random variable Z with mean y and scale is given by Fz(2) compute the preference probability
(1 + (a--/a--)/a--- Therefore,
P(As > 0) = - PPAss S 50)
FAFE (0)
1 e- 1e--(0-w//- (0-1)/5
1 1 eu/s
(1 + (0/8)-11
eH/e
1+ 4//8
1+el eP/s
(14)
1 1-/8"
This final expression is precisely the sigmoid function, (μ/S).
Then, we apply the remaining assumptions. Assumption posits that the mean of the distribution y rw (2, vo) Tw (a,, y.). Assumption 3 sets the scale parameter to unity, s 1. Substituting these into our result from Eq. (14) yields:
Polye Dr x) (ros a(ro(1,. (x)
(15)
From Lemma 1. we know that the Bradley-Tere model also simplifies to (r (x, Be) ny (æ, yr)) Since the probabilistic reward model under the specified assumptions and the Bradley-Terry model both yield the identical functional form, we have shown that the latter is a special case of the former.
17

---

Published as a conference paper at ICLR 2026
C GRADIENT ANALYSIS OF THE PREFERENCE PROBABILITY
In this section, we conduct a formal gradient- analysis to demonstrate that maximizing the preference probability, Pulyee 1/ x), incentivizes the underlying probabilistic model to maximally separate the score distributions of the chosen and rejected responses.
Proposition C.1 (Optimization Incentive of Preference Maximization). Let the scores for responses be drawn from a discrete set {a, a+........ Let pe (K) & (s = x, yc) and p.(k) & pulss = k a. Me) be the respective probability muss functions (PMFs), Maximizing the preference probability P( c > 8+) with respect to the variables (Pc(k)) and {pr (k)) under the constraints P&Po(k)
and Ex Pr (k) 1 creates the following incentives:
1. For the chosen response Yes shifting probability mass from any score k to a higher score k + will increase or maintain the objective value.
2. For the rejected response Yr, shifting probability mass from any score k + a lower score k will increase or maintain the objective value.
This implies that the optimization process drives the PMF of ye towards the maximum score and the PMF of yr towards the minimum score a.
Proof. The preference probability & Polye > yv ()) for discrete scores is given by:
b P pc())((s, < = Σ (c( (ERLOO)
(16)
We analyze the gradient of with respect the probability mass at each score for ye and yr separately.
Part 1: Incentive for the Chosen Response Score (ye). We first compute the partial derivative of P with respect to (k) for some score & E (a....,b) bb. From Eq. (16), only the term where i k depends on Pe (k)),, so:
OP
Pe(k) Σ (+)
~T( P(s, k).
(17)
Öpe (1:)
(k)
f=a
This derivative represents the sensitivity of the objective to an increase in probability mass at score k. To understand the incentive for shifting mass, consider moving an infinitesimal probability mass from a score & to higher score k + 1. This corresponds to a change in the PMF: (kk - (kk))
and (k + 1) + (k + 1) + The resulting change in P. denoted AP, can be approximated by the first-order Taylor expansion (which is exact since P is linear in Pc):
OP
0P
AP R3
Opo(k + 1) (k)
(P(srr < <k+1) P(st k)) (using Eq, equation 17)
P(s, k)
6. (k).
(18)
Since probabilities are non-negane non-negative, (k) 2 0, and we defined > 0, it follows that AP > 0. This demonstrates that any shift of probability mass to a higher score for Be is guaranteed to be non-decreasing change the objective function. This creates a persistent optimization pressure to move the entire distribution Pe towards the maximum score b.
Part 2: Incentive for the Rejected Response Score (yz). To analyze the effect of (k)), it is more convenient rewrite Eq. (16) by swapping the order of summation:
b-1 P P=
b
Σ
po(i)
(19)
==a
1=
18

---

Published as a conference paper at ICLR 2026
The partial derivative of P with respect to (r( for is fa....,b- 1 is:
0P
a
b
(r)
Σ (i( = Σ pe(i) = P(sc > k). i=++1
(20)
Öp, kk) Op, (k)
k=++1
Now, consider shifting an infinitesimal probability mass e 0 from a score is + lower score k.
This corresponds to the change: (r(k) + (k)) + and pr 1) + Pr + 1) - c. The resulting change P is:
OP
OP
AP E (k.(k) Op.(k +1
(P(se > k) P(se > h 1))
(using Eq. equation 20)
= P(oe- = =k+1)
Po + 1).
(21)
Since (e + 1) 2 0 and E O, we have AP 2 O. This shows that shifting probability mass to lower score for is always a non-decreasing change. This creates a consistent optimization pressure to move the distribution pr towards the minimum score a.
Combining both parts, we have formally shown that maximizing the preference probability P(sc > 8c) drives the model to separate the score distributions by pushing the mass of Pe towards the highest possible score and the mass of py towards the lowest possible score.
D DETAILED EXPERIMENTAL SETUP
Training Preference Data. We curate dataset of 130k samples for reward model training, drawn primarily from publicly available open-source datasets: Skywork Reward Preference 80K (Liu al., 2024b) high-quality, pairwise preference dataset that spans multiple domains, including chat, safety, mathematics, and code. It employs advanced data filtering techniques ensure the reliability of preferences across different tasks. UltraFeedback Binarized Preferences (Cui et al., 2023) is large-scale, fine- -iaiinr, and diverse preference dataset designed for training powerful reward and critic models. It comprises approximately 64k prompts from various sources, including UltraChat (Ding et al., 2023), ShareGPT, Evol-Instruct (Xu al., 2024), TruthfulQA (Lin al., 2021), FalseQA (Hu et al., 2023), and FLANN (Wei et al., 2021). Each prompt is used to query multiple LLMs generate four distinct responses, resulting in a total of 256k samples.
- series of models (7B, 14B, 32B, and 72B) (Team, 2024) as the backbone for training the OPRM. We compare OPRM to different categories of baselines: (1) Discriminative RMs, including Skywork-Reward (Liu et al., 2024b), ArmoRM (Wang et al., 2024a), Internl.
Baselines. In our main experiments, we employ the Qwen2.5 Instruction
(Cai et al., 2024), and Nemotron-43340B Reward (Wang et al., 2024c). (2) Generative RMs, including DeepSeek-Ve V2.5 (Liu et al., 2024a), Gemini-L.5-Pro (Team et al., 2024), LLaMA-3. (Grattafiori et al., 2024), Claude-3.5 3.5-de-..5-sonet. and GPT-40 (Hurst et al., 2024). (3) DeepSeek-RM, a collection of baselines re-implemented by DeepSeek, including LLM-as-A-Judge (Zheng et al., 2023), DeepSeek-BTRe BTRM (Bradley & Terry, 1952), DeepSeek-PairRl (Jiang et al., 2023), CLoud-Gemma-2 (Ankner et al., 2024) and DeepSeek- GRM (Liu et al., 2025b).
Benchmarks and Evaluation Metrics. Following prior work, we evaluate the performance of different methods various RM benchmarks of different domains: Reward Bench (Lambert et al., 2024), PPE--reference, OPE-Correetness (Frick et al., 2024), RMB (Zhou et al., 2024). We use the standard Best-of-N evaluation metrics for each benchmark: the accuracy of picking the best response from a set of responses. Specifically, Reward Bench and PPE Preference involve pairwise comparisons, with each prompt featuring two candidate responses. In contrast, PPE Correctness is designed for a large-scale Best-of-N evaluation, presenting 32 responses for each prompt. RMB is a hybrid, incorporating both pairwise comparison tasks and Best-of-5 selection format.
19

---

Published as a conference paper at ICLR 2026
E SENSITIVITY ANALYSIS ON BOUNDARY AND BIN CONFIGURATIONS
In this section, we investigate the robustness of OPRM-RgFT to variations in ordinal bin definitions and boundary placements. Specifically, we aim verify that the method's performance is primar- ily driven by the underlying probabilistic framework rather than specific hyperparameter choices regarding the ordinal scale.
To this end, we trained a variant of OPRM-RgFT-32B using an expanded scale S' = (o..... 9 with irregular boundaries: bad (0, 1, 2,3), normal (4, 5), and good (6, 8, 9}. This setup introduces asymmetry and changes the cardinality of the semantic sets, contrasting with our default uniform configuration which employs a symmetric partition of the scale S = {I....,9) (bad {1, 3).. normal {4, 5, 6). good (7)8.))).
As presented in Table 6, the performance deviation between the default uniform setting and the irregular variant is negligible, with the difference in the Overall score being less than 0.1% This empirical evidence demonstrates that OPRM-RgFT is robust to boundary shifts and does not rely on uniform partitions to achieve high performance.
Table 6: Performance comparison between the default uniform configuration and an irregular bound- ary variant. The results indicate that the method is highly robust to binning strategies.
Configuration
RewardBench
PPE-P
PPE-C
RMB
Overall
Irregular Variant (Scale 0...9) ... bad {0, 1,2,3), normal (4, good {6, 7,89
89.1
64.1
67.7
74.4
73.8
Default Uniform (Scale ...9) bud (1,2,3) 3). normal (4, 5, good [7, (7,
88.9
64.6
67.3
74.8
73.9
Despite the demonstrated robustness to irregular boundaries, adhere to the default uniform config- uration for two principled reasons related to efficiency and priors. First, regarding computational efficiency, utilize the range [1, 9] to maximize granularity while ensuring each ordinal score maps to a single token. Mainstream LLM tokenizers typically treat digits (0..... 9} as individual tokens, whereas values > 10 decomposed into multiple tokens. Restricting the support set single tokens allows OPRM to compute the full probability distribution in a single forward pass, avoiding the prohibitive computational costs associated with computing joint probabilities over multi-token sequences. Second, concerning geometric simplicity, we employ a uniform partition (3 x 3 regions) as a neutral prior minimize inductive bias. This uniformity aligns with the theoretical design of Region Flooding Tuning, allowing probability mass to flood into a symmetric lower triangular geometry of consistent size. In the absence of domain-specific knowledge suggesting that good samples require finer granularity than bad ones, symmetric division simplifies hyperparameter selection and ensures balanced gradient pressure across different quality levels.
F BRADLEY-TERRY LOss WITH MARGIN
Inspired by INF-ORM (Yang et al., 2024), which employs GPT-4o to evaluate the preference margin between chosen and rejected responses, we annotate each pair in our dataset with a margin label. The original evaluation in INF-ORM follows these rules: (1) If the chosen answer is much better than rejected answer, set margin to 10; (2) If the chosen answer is better than the rejected answer, set margin to 3; (3) If the chosen answer is slightly better than rejected answer, set margin to 1.
Analogously, we define margins based on the combination of quality annotations. Specifically, pairs with the same quality level, such as <good, good>, <normal, normal>, and <bad, bad>, are assigned margin of Pairs with adjacent quality levels, namely <good, normal> and <normal, bad>, are assigned a margin of 3. Finally, margin of 10 is assigned to pairs with distant quality levels, like <good, bad>
After that, the Bradley-Terry Loss with Margin is defined as:
((rx) -E (m ye. Ut) log (1, r2( v+))],
(22)
20

---

Published as a conference paper at ICLR 2026
Here, (2; Ves yr) stands for the margin value between chosen and rejected responses. This formula helps the model to better understand which responses are preferred over others, based on the scores gave them.
G DISENTANGLING THE IMPACT REGION FLOODING FROM LABEL AUGMENTATION
central hypothesis this work is that the Region Flooding Tuning (RgFT) framework provides methodological benefits beyond the simple inclusion of absolute quality labels. To verify whether the observed performance gains stem from the architecture itself or merely from data augmentation, we conducted controlled experiments comparing OPRM- OPRM-RgFT against standard classification paradigms trained on identical data (comprising both preference pairs and semantic quality labels).
We formulate two baseline approaches to isolate the contribution of the modeling strategy:
Baseline A (Hard Classification): A standard multi-class classification model optimizing for three discrete quality tiers (good, normal, bad). Inference performed via maximum posteriori estimation, selecting the class with the highest probability.
Baseline B Sccalar-Weighted Classification): A regression-oriented approach designed to provide finer granularity than hard classification. Here, the reward score is computed as the expected value over class probabilities: E[s] = Eee eg (groodnormmll.bdd P(c) Vo. where V. represents the centroid of the corresponding semantic region (e.g., values mapped to 2, 5, and 8).
Table 7: Comparative analysis of OPRM-RgFT against standard classification baselines utilizing identical supervision signals. The results demonstrate that the proposed region flooding mechanism yields significant gains over pure classification approaches.
Method
RewardBench
PPE-P
PPE-C
RMB
Overall
Baseline A (Hard Classification)
71.0
43.9
46.6
55.7
54.3
Baseline B Weiihted Classification)
85.4
61.5
64.3
71.3
70.6
OPRM-ReP R-3322B (Ours)
88.9
64.6
67.3
74.8
73.9
The empirical results, detailed in Table 7, indicate a clear performance hierarchy. Hard Classification significantly outperforms Scalar- Scalar-Weighted Classification, underscoring the necessity of continuous score representations for ranking tasks. However, it consistently underperforms OPRM-RgFT across all benchmarks (e.g., 3.3% deficit in the Overall score). This performance gap substantiates that the efficacy of our method is not solely attributable to the availability of absolute labels. Rather, the Region Flooding strategy effectively harmonizes discriminative pairwise ranking with absolute semantic constraints. Unlike standard classification, which treats labels as independent categories, RgFT anchors the preference distribution within a continuous ordinal space, thereby achieving superior calibration and ranking fidelity.
H OPRM'S PROMPT TEMPLATES
As shown in Prompt H.1, we present the prompt templates used during the OPRM training process. Furthermore, we designed distinct scoring criteria for general-domain and domain nomain-specific data. Specifically, the general-domain preference data is evaluated using the OpenAl 3H Criteria, as detailed Prompt H.2. In contrast, the role-play data used in our experiments is assessed against our custom Role Play Criteria, presented in Prompt H.2.
21

---

Published as a conference paper at ICLR 2026
H.1 PROMPT
Ordinal Distributional Reward Modeling Prompt
## Evaluation Guidelines
You are an expert text quality evaluator. Your task is to carefully assess and score the AT-generated candidate response based on the specific criteria provided below.
## Scoring Criteria
(criteria)
## Context of Interaction
The following represents a historical conversation between a user and AI assistant:
(prompt)
# Response to Evaluate
Below the candidate respouse generated by the Al that needs to be evaluated:
(response)
## Evaluation Instructions
Please score the response according to the criteria scale from to 9, where:
1-3: Bad performance
4-6: Normal performance
7-9: Good performance
Use the following format for your final assessment: "Score: <total points>".
Score:
H.2 CRITERIA
Role Play Criteria
Responses should be irrelevant. Responses should be coherent and reasonable within the given context, without containing logical flaws.
Relevant:
Responses should be credible, meet the prompt requirements and provide accurate information. In role-playing or storytelling, responses should be consistent with the given character and background.
Credible:
Interesting: Responses should engaging and able to capture the user's attention. Responses that are self-aware, thought- or captivaling will be more interesting.
Openai 3H Criteria
The content generated should aim to be of assistance to the user. It should provide accurate information, answer questions to the best of the model's knowledge, and offer solutions that are practical and relevant to the user's queries. The model should not generate content that is intentionally misleading or unhelpful.
Helpfulness:
The model should strive to provide information that is truthful and based the most accurate data available up its knowledge cutoff. It should not fabricate information present opinions facts. When uncertain, the model should acknowledge the limitations of its knowledge.
Truthfulness:
The content should not cause harm to individuals groups. It should avoid generating content that is offensive, discriminatory, or promotes harmful behaviors. The model should be programmed to refuse to generate content that could be used for malicious purposes, including but not limited to generating false information, engaging in deception, or promoting illegal activities.
Harmlessness:
22

---

Published as a conference paper at ICLR 2026
I DETAILED EXPERIMENT RESULTS
We report the detailed per-subset experiment results on RewardBench 8, PPE Correctness 9, and RMB 10. The results for the baseline methods sourced from their original papers.
Table 8: Detailed results different methods on the Reward Bench benchmark.
Method
Chat
Chat Hard
Safety
Reasoning
Reward Bench
Reported Results of Public Models
Skywok-RRward.iiemma22277
95.8
91.4
91.9
96.1
93.8
DeepSeck-V2.5-090
81.5
Gemini-1.5-PP
94.1
77.0
85.8
90.2
86.8
ArmoRM-SB-vO..
96.9
76.8
90.5
97.3
90.4
MnternL.M-22B--Reen
98.9
76.5
89.5
95.8
90.2
1LaMA-3.1-70b-Img
97.2
70.2
82.8
86.0
84.1
Blaude-3.5-somme
96.4
74.0
81.6
84.7
84.2
Mimbtrron--340B-Reee
95.8
87.1
91.5
93.6
92.0
GPT-4o
96.1
76.1
88.1
86.6
86.7
Reproduced Results of Baseline Methods From DeepSeek
LLM-as-a-uudge
96.7
69.3
83.5
84.3
83.4
DeepSee--BTRM-277
96.7
86.2
75.7
89.8
81.7
"Loud-Gemma-22
96.7
69.3
83.5
84.3
82.0
DeepSeck-PairRM--275
95.5
86.8
52.3
92.0
87.1
DeepSee--GRM-27B-B.R
94.7
77.2
87.0
79.2
84.5
DeepSeek-GRM-278
94.1
78.3
88.0
83.8
86.0
Results of Our Method
OPRM-QNwen2.5-77
96.4
76.3
86.2
92.2
87.8
OPRM-Qwen2.5-1448
78.1
86.1
89.3
OPRM-Qwen2.5-32
96.9
81.8
89.6
96.7
91.3
OPRM-Qwen2.5-72B
96.4
79.6
88.1
93.0
89.3
Results of Our Method (w/ Region Flooding Tuning)
OPRM-RgFTT-QWen255-7
95.5
76.5
86.4
86.5
86.2
OPRN-RggFT-QQwen255-111
96.9
79.4
88.1
84.6
87.3
OPRM-RRFTT-QWee2255-222
95.3
82.7
89.2
88.4
88.9
OPR.N-RFFT-QQwen2.5-722
96.9
82.7
89.7
87.1
89.1
J QUALITY-LEVEL ANNOTATION
J.1 GENERAL-DOMAIN DATA
For the acquisition of quality-level annotations, follow the methodology of UltraFeedback (Cui et al., 2023). The process involves two main steps. First, we employ the gpt-4o model (Hurst et al., 2024) to annotate each prompt-response pair with fine-grained scores across multiple dimensions. such as instruction-fie cilruction-folooiingg truthfulness, and helpfulness. Second, these scores are averaged, and the resulting value is mapped to one our three predefined quality levels goodd normal, or bad-based specific score intervals.
Following this automatic annotation, we perform a manual verification step. For verifiable tasks, such mathematics and coding, check the responses against the ground truth. If response is found to be incorrect, its quality level is manually downgraded to bad. We acknowledge that for more subjective tasks, this per-instance verification is not always feasible, which may introduce some annotation noise.
Finally, to ensure logical consistency, filter out all pairs where the chosen response is not strictly better than the rejected one. This includes invalid combinations of <dgbusen <nejestdd> such as <normal, good>, <bad, normalz, and <had, good>. The remaining data constitutes final training set.
23

---

Published as a conference paper at ICLR 2026
Table 9: Detailed results of different methods the PPE Correctness benchmark.
Method
MMLU-Pro
MATH
GPQA MBPP-Plus
IFEval
PPE Correctness
Reporied Result of Public Models
Ayyook-Reward-Gemmee
54.0
63.0
53.0
59.0
54.0
DeepSeek-V2.5-00
Gemini-1.5-Pro
ArmoRM-8B-v0.1
66.0
71.0
57.0
54.0
58.0
61.2
IntemL.M2-20B-Rewe
68.0
70.0
57.0
58.0
62.0
63.0
1.aMMA-.--70h-Tnssru
59.2
Blaude-3.5-sonme
66.0
63.0
56.0
52.0
57.0
58.8
Remotroon--340B-Reeeee
70.0
65.0
57.0
49.0
63.0
60.8
GPT-4o
57.6
Reproduced Results of Baseline Methods From DeepSeek
LLM--s-a-Juude
66.0
68.0
52.8
50.2
56.8
58.8
DeepSeck-BTRM--1
68.8
73.2
56.8
68.5 8
66.0
66.7
CLuud-Gemmma22227
68.7
68.8
53.5
59.0
62.0
62.4
DespScck-PairRM--27
68.3
74.7
55.0
63.1
62.9
64.8
DeepSeck-GRM-27B5
64.8
68.7
55.5
49.0
60.2
59.6
DespSeek-GRM-27B
64.8
68.8
55.6
50.1
59.8
59.8
Result of Our Method
OPRM-Qwen2.5-778
65.2
70.1
56.3
59.0
56.1
61.3
OPRM-QQwen..5-14B
66.7
70.7
57.1
67.4
59.5
64.3
OPRM-QWen2.5-222
71.2
73.2
57.9
66.2
62.2
OPRM-QQwe22.5-72B
73.4
75.9
58.6
54.1
59.5
Results of Our Method (w/ Region Flooding Tuning)
OPRM-RREFT-Qwen255577
64.8
71.6
55.9
63.0
56.8
62.4
OPRM-RRFFT:0wen2225511
69.5
74.0
57.3
67.0
60.0
65.6
FPRM-RRFT-QQee22555-00
73.3
76.8
58.5
60.6
DPRM-RgFF-QQwen25-777
72.8
77.1
59.0
61.2
Table 10: Detailed results of different methods on the RMB benchmark.
Method
Helpfulness BoN
Helpfulness Pair Harmlessness BoN
Harmlessness Pair
RMB
Reported Results of Public Models
Skywork Skywork-Reward-GGnn Bewaar--Gemma-22777
47.2
65.3
56.1
72.1
60.2
DeepSeek V2.5-0905
Gemini-1.5 1.5-Pmo
53.6
76.3
66.1
56.5
AsmoRM-8B--o --.v01
63.6
78.7
49.7
66.3
64.6
M2-2008 Reward
58.5
LLaMA-3.1-706 --t-------ob-nsttteeet
64.8
81.1
55.8
73.9
68.9
Claudo-3.5-soman
70.5
83.8
51.8
76.4
70.6
Nemotron---340BBB
GPT-4o
63.9
81.5
68.2
81.4
73.8
Reproduced Results of Baseline Methods From DeepSeck
LLM-as-a-Judge
55.8
78.5
BTRM-27B BTRM
64.0
83.0
33.6
51.0
57.9
CLoud-Gemma-2-278
64.7
81.1
41.7
66.1
63.4
PeepSeck-PairRM-22
DeopSee-GRMM-27B-RFT
58.4
79.3
54.2
76.0
67.0
DeepSeek GRM 27B
62.3
80.5
57.0
76.1
69.0
Results of Our Method
OPRM-Qwen2,5-78
63.1
78.4 4
65.7
78.8
71.5
OPRM-Qwee2.5-1-BB
65.8
80.7
68.2
80.5
73.8
OPRM-Qwen2.5-328
69.2
82.1
OPRM-Qwee2.5-72BB
68.7
82.4
64.2
78.5
73.5
Results of Our Method (w/ Region Flooding Tuning)
FORM-RRFT-QWen2.555 FTRM-REFT-Qwen2.5.5
63.4
79.0
62.4
70.1
66.3
81.3
65.3
78.2
72.8
OPRM-RgFT-Qweenn T-Qwen2.5-32B
67.6
81.4
69.1
81.2
74.8
OPRM-ReFT Feai-RFFT-Qeeeee Fww--FFFF-Qwe225-7221
67.9
J.2 ROLE-PLAY DATA
For the domain-specific data, we employe a team of six human experts to perform accurate quality- level annotation. The data consists of Role-Play Dialogues, for which the experts assessed each prompt-response pair against four core dimensions: Core Role-Playing Consistency, Interactivity & Narrative Progression, Fundamental Linguistic Quality, and Immersion. Based this multi-
24

---

Published as a conference paper at ICLR 2026
dimensional evaluation, they directly assigned quality level of good, normal, or bad to each prompt-response pair. To ensure high annotation quality and consistency, a label was only accepted if at least two experts reached consensus. We consider this high-fidelity annotation process to be crucial for fully leveraging the capabilities of our Region Flooding Tuning method.
J.3 DISTINGUISHING ANNOTATION ERROR FROM DISAGREEMENT
Our methodology emphasizes robustness to inconsistent preference data and the ability to handle annotation disagreement. To provide clarity guish between two fundamentally different types of label conflicts: Annotation Errors (Logical Inconsistency) and Annotation Disagreements (Subjective Ambiguity).
data processing pipeline, it is crucial distin-
Annotation Errors (Logical Inconsistency). Our training paradigm relies preference pairs (t)) where the chosen response Ye is preferred over the rejected response y,- A semantic label configuration such as <normal, good> (where the chosen response has lower semantic quality than the rejected one) constitutes a direct violation of the preference premise (ye > Dr). We classify such cases as logical inconsistencies annotation errors rather than subjective disagreements. Empirical analysis of our dataset reveals that these contradictions are extremely rare, accounting for less than 0.1% of the total samples. Consequently, our decision to filter these instances represents a standard data preprocessing aimed at eliminating verifiable noise, rather than an evasion of challenges.
Annotation Disagreements (Subjective Ambiguity). In contrast, the disagreement that our prob- abilistic framework aims model refers valid variations in semantic judgment where the core preference relationship remains intact. For instance, given a valid preference pair (ye > 3r), one annotator might assign the labels <good, normal>, while another might assign <good, bad>. Both annotations respect the ordinal constraint (ye is superior to yr) but differ in the perceived margin of quality. This type of variation reflects genuine subjective ambiguity in reward assessment. Unlike point-estimate models that are forced to regress to single mean value, OPRM is specifically designed to capture this aleatoric uncertainty by learning full probability distribution over the reward space.
In summary, the filtering step mentioned in prior sections is strictly limited to removing logical contradictions that violate the definition of a preference pair. It does not compromise our claim of robustness; rather, it ensures that the model focuses on learning meaningful distributional patterns from legitimate subjective variations.
K FUTURE WORK
In this section, we outline several promising directions for future research that build upon the OPRM framework, such as customized Region Flooding Tuning method and customized decoding method.
K.1 CUSTOMIZED REGION FLOODING TUNING
Figure 5: Annotation Region Flooding Tuning. As annotation ambiguity increases. the target optimization region "floods" to encompass a wider set of plausible outcomes. A more uncertain annotation results in larger target region than a more certain one.
As discussed in Section 5.2, a core advantage of Region Flooding Tuning is its customizability. In this section, we demonstrate a novel application of this feature. Annotator inconsistency is a well-known challenge in preference data collection, and leads to increased label ambiguity. To address this, we propose method that explicitly models this ambiguity by permitting a single proppt-response pair to be associated with multiple potential
issue that is exacerbated at finer annotation granularities
25

---

Published as a conference paper at ICLR 2026
quality levels. For instance, response might be considered both good and normal, even all three levels in cases of extreme uncertainty. As illustrated in Figure 5, our approach handles this by optimizing expanded set of joint probabilities, corresponding to all plausible quality-level assignments for a given pair.
K.2
CUSTOMIZED DECODING METHOD
The full probability distribution Pe x. x. produced by OPRM allows us go beyond a simple expected score. To fully leverage this rich distributional information, we introduce Uncertainty- aware Decoding. This method adjusts the expected score by penalizing predictive uncertainty, thereby favoring responses that are predicted to be high-quality with high confidence. The final reward score Tw (z, y) is calculated as:
Cib ( v)
Σ p x.,))
x A. A.((z,))
(23)
Uncertainly Term
Expocted Score
where the first term is the standard expected score. The second term, u(x; y), is uncertainty measure the distribution, such as its Shannon entropy or variance. The hyperparameter 2 controls the strength of the uncertainty penalty.
L COMPUTATIONAL OVERHEAD ANALYSIS
A key consideration for advanced reward modeling frameworks is the potential trade-off between performance gains and computational costs. To rigorously evaluate this, we conducted a systematic benchmark comparing the computational overhead of OPRM against a standard Bradley-Terry model under identical experimental conditions. Both models employ the Qwen2.5-32B architecture as the executed using
H200 GPUs for training and 4x H200 GPUs for inference, with results reported as the average wall-clock time over three independent runs.
backbone. Benchmarks
Table 11: Computational efficiency comparison between BTRM and OPRM.
Method
Training Time
Inference Time
Hardware (Train)
Hardware (Infer)
BTRM (Standard DRM)
7.3h
5.5 min
32 x H200
4 x H200
OPRM (Ours)
6.9h
2.1 min
32 x H200
4 x H200
As summarized in Table 11, OPRM exhibits training efficiency comparable to, and slightly superior to, the BTRM baseline (6.9 hours vs. 7.3 hours). This result empirically confirms that our ordinal preference modeling approach captures richer distributional information without imposing additional computational burdens during the training phase.
More notably, OPRM demonstrates substantial advantage in inference latency, achieving 2.6x speedup over BTRM (2.1 min vs. 5.5 min). This efficiency gain stems directly from OPRM's architectural design: by eliminating the need for an external scalar value head, OPRM remains strictly within the standard vocabulary space the LLM. Consequently, with highly optimized inference engines (c.g., vLLM, SGLang) without requiring custom kernel modifications or specialized implementations. This characteristic renders OPRM particularly suitable for high-thue throughput applications, such online RLHF and large-scale preference ranking tasks like Best-of-N sampling, where inference speed is a critical bottleneck.
natively compatible
M CASE STUDY
We present several case studies in Table 12, Table 13, and Table 14, along with the scoring results from BTRM, OPRM, and OgRM-RgFT for comparative analysis. A key finding is that by being grounded absolute quality assessment, OPRM and OPRM- exhibit a reduced susceptibility to length bias, known issue in purely relative models like BTRM. Moreover, for tasks with verifiable
26

---

Published as a conference paper at ICLR 2026
answers, our models consistently prioritize factual correctness, whereas BTRM can be misled by other stylistic attributes.
Table 12: Case
Example
Question:
You highly skilled mathematician tasked with solving complex math problems. Your goal provide clear, step-by sepppy-step solutions that can be easily parsed and evaluated.
Here is the math problem you need to solve: <problem> The length of rectangle is twice its width. Given the length of the diagonal 5 find the area of the rectangle. </problem>
Box your final answer using LaTeX, for example: $x = \boxed [ Yourr final numerical or algebraic answer)) Now, please solve the given math problem and provide your solution in the specified format.
Chosen Response: Here's s my solution to the problem:
Let the width of the rectangle be Since the length is twice the width, the length of the rectangle is 2w.
We given that the length of the diagonal is v5. Using the Pythagorean theorem, write: (5v5)²
2² + (2w)²
Simplifying the equation, we get: Combine like terms: 5 5w.
w² 4w
Divide both sides by 5: 25 = w²
Take square root of both sides: w 25
Now that we have the width, we can find the length: 2w = 2. 10
width and length: = w Therefore, the area of the rectangle is 50.
The area the rectangle is product
Rejected Response: To solve this problem, we can start by using the Pythagorean theorem to relate the length and width of the rectangle to the length of the diagonal: d === 12 w²
where d the length of the diagonal, is the length of the rectangle, and w is the width of the rectangle. Since the length of the rectangle twice width,
can write: = 2
Substituting this into the Pythagorcan theorem, we get: de = (2w)² + Simplifying this expression, we get: df 5w²
Since given that the length of the diagonal 5v5,, can up the equation: 5
Substituting this into the equation above, we get: 5v5² 5w Simplifying ingg this expression, we get: 25 5 5w? Simplifying further, we get: 125 = u²²
Taking the square root both sides, we get: w V225 5/5 Now that we have found the width, we can find the length by substituting this value back into the equation: I 2w 2(5 10v5
The area of the rectangle is given by: Iw (5v5)) 50.5"" 50 Final Answer: The final answer is 250.
250
BTRM-Qwen2.5-322
Chosen Score: 0.03
Rejected Score: -2.70
OPRM--Qen22.5-33 (Ours)
Chosen Score: 7.04
Rejected Score: 3.63
OPRM-RgFTT Ftw.--ggF----e22--3222- (Ours)
Chosen Score: 8.06
Rejected Score: 3.13
27

---

Published as a conference paper at ICLR 2026
Table 13: Case
Example
Question: (x (x²
Chosen Response: Yes, the equation (x*)*
yM indeed true, and a direct consequence of the
properties of exponents in mathematics.
To see why this is true, let's consider the equation on both sides:
Left side: (x:))²
Right side: (ar))
First, we simplify the left side by applying the power of power rule, which states that (a"")" a (mn))
Left side: (x*)* = (x44)
Now, let's simplify the right side using the same power of power rule: Right side; (x (*
Now we can see that both sides of the equation are equal:
(x22) (20)
Since the bases (x) are the same and the exponents (yz) and (æy) are also the same, we can conclude that the equation (x*)* = (x²)*
) is true. This property holds for any real numbers y, and 2, as long as x * and
and non-zcro.
Rejected Response: A classic question about exponentiation!
Unfortunately, the answer is no, (==)** is not necessarily equal (x*)* in general.
To see why, let's break down:
(a*)² means "raise x to the power of y. and then raise the result to the power of ". This can be written as: (x*)* ((2==)
On the other hand, (x*)² means "raise x to the power of z, and then raise the result to the power of y". This can he written as:
(x="" = zz (=*b)
As you see, the expressions are equal in general, because y is necessarily equal y. For example, œ 2, y and z 4, then:
(2²)² 2(3-) 2 4096
while
(2*)3 2(+33) 12 4096
But in this case, it just so happens that * = 2 * so the two expressions are equal. However, this is not always the case.
For example,
2, 2. and then:
(22)3- = 2(++3 = 26 64
while
(2*)² 2(+2 26 64
Again, it just so happens that two expressions are equal case, this is not general rule.
So, to summarize: (I=*)* 3 is not necessarily equal to (x*)", unless and commute, ie., y z
BTRM-Qwen2.5-3322
Chosen Score: -0.96
Rejected Score: -0.39
OPRM-QWee22.5-328 (Ours)
Chosen Score: 7.85
Rejected Score: 3.84
OPRM-Rg-FT-Qwnn22 (Ours)
Chosen Score: 8.03
Rejected Score: 2.05
28

---

The correct option is E. "has sent." Published as a conference paper at ICLR 2026
Table 14: Case
Example
Question: Anita some application letters to company but they never her.
A.has send
B.have sent
C.have send
D.have sending
E. sent
Chosen Response:
The correct answer "has sent".
Explanation:
In this sentence, we need to use the present perfect tense, which is formed by "has/have + past participle". The past participle of the verb "send" is "sent".
A. "has send" is incorrect because "send" is the base form of the verb, not the past participle.
B. "have sent" is incorrect because the subject "Anita" is singular, so we should use "has" instead of "have".
C. "have send" is incorrect for two reasons: "have" used with plural subject, and "send" is not the past participle.
D. "have sending" is incorrect because "sending" is the present participle, not the past participle.
E. "has sent" is the correct form, using "has" for the singular subject "Anita" and the past participle "sent". Therefore, correct sentence should be: "Anita has sent some application letters to the company, but they never call her."
Rejected Response:
The sentence should read: "Anita has sent some application letters to the company but they never call her." The verb "has sent" is used because the subject of the sentence, "Anita," is singular and the action of sending the letters happened the past. The present perfect past, but its effects are still being felt in the present. Therefore, "has sent" is the appropriate form of the verb to use in this sentence.
used indicate that the action was completed in the
BTTRM-Qwen2.5-
Chosen Score: 0.24
Rejected Score: -1.12
OPRM-Qwen2.5-322 (Ours)
Chosen Score: 8.08
Rejected Score: 6.76
SPRM--RFFT-Qwen2.555 (Ours)
Chosen Score: 8.81
Rejected Score: 8.14
29


Reviews:
[{"review_id": "xIXdeyaKFT", "reviewer": "Reviewer_Tq6c", "summary": "The paper proposes a Probabilistic Reward Modeling (PRM) that treats a response’s quality as an absolute reward value rather than relative value. This absolute value is modeled as a random variable and the trained model learns its distribution. Then the author bin the continuous values into 9 discrete bins, where the ordinal quality distribution ranges from 1 to 9, and optimizes the probability that the chosen response’s score exceeds the rejected one. A training strategy, Region Flooding Tuning (RgFT), further uses good/normal/bad quality labels to improve the training. The paper reports empirical results on RewardBench, PPE, and RMB by improving over current baselines.", "questions": "1. I'm curious about what the definition of \"ordinal\" is in your sense. For example, there is another previous paper *Reward Modeling with Ordinal Feedback: Wisdom of the Crowd*. The term \"ordinal\" of yours is a bit different from theirs, and you should discuss about why your discrete absolute scores are one kind of \"ordinal\" label.\n\n2. Please see Weaknesses 1. Calibration measurement should be included, or the authors should delete the claim \"calibrated\" in their text.\n\n3. Please see Weaknesses 2 and 3. RgFT is the core part. The authors should include more discussions on RgFT, including formal definitions and intuitions.\n\n4. Please see Weaknesses 4. The authors should include other distributional RMs in the comparison.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. Rating the response by an integer is not novel, but the paper proposes a good way of learning the integer's distribution by contrasting the overall score distribution of chosen and rejected respones. Empirical results show the efficiency.\n\n2. RgFT training controls the shift of probability mass more precisely by using some additional information of quality (\"good/normal/bad\"). It makes the training more efficient.\n\n3. Reusing the LM head probabilities over numeric tokens is easy to implement and simple yet effective.", "weaknesses": "1. Calibration not measured. A primary motivation is \"calibrated, interpretable distributions\", yet there are no metrics (e.g., ECE) in regards of calibration reported. This is a big question, as the authors claim that they are modeling the whole distribution. Accuracy isn't enough.\n\n2. I'm not very satisfied with the presentation of this paper. Two major concerns (but not limited to them): (1) Important model/algorithms such as RgFT are not formally defined, which is confusing. Also, gradient updates of the loss can be derived in the main text as this is an important intuition. (2) The authors spend many efforts in describing how effective their method is in the abstract and the introduction, but they do not describe their model or algorithm ideas briefly. That prevents readers from acquiring the useful information. From the perspective of readers, most submissions to NeurIPS/ICML/ICLR are SOTA, but the key difference is how and why.\n\n3. Where do the gains come from? RgFT mixes labeled and unlabeled preference data and changes the optimized region. It would help to quantify how much improvement is due to (i) access to absolute labels vs. (ii) the flooding geometry itself vs. (iii) more total training signal.\n\n4. Limited comparisons to other distributional RMs. The related‑work section cites distributional/uncertainty‑aware RMs (e.g., PURM, quantile approaches), but comparisons in the main tables primarily cover BT‑style and generative baselines. Including at least one representative distributional RM would better show OPRM’s advantages.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cGT2Ph4IoL", "reviewer": "Reviewer_axbh", "summary": "The paper addresses limitations of traditional generative and discriminative reward models in RLHF. The authors propose a probabilistic reward model (PRM) training paradigm that discretizes the quality score into a finite set of ordinal ratings and trains the reward model as a classifier over these regions. They further introduce a “region flooding tuning” training strategy to improve coverage and robustness around decision boundaries. Experiments across multiple benchmarks indicate improved effectiveness over baselines.", "questions": "1.\tCan you explain Eq. 8 and Figure 2 in detail?\n\n2.\tHow sensitive are results to the number of bins and boundary placement?\n\n3.\tCan PRM incorporate external verifiable rewards as additional ordinal classes or as targets?\n\n4.\tWhat is the computational overhead of training?\n\n5.\tDoes the method improve robustness to reward hacking?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Presents a novel training paradigm that requires minimal architectural changes to existing neural reward models.\n2. The proposed method may improve calibration and robustness compared to traditional Bradley-Terry reward models\n3. Extensive experimental evaluation across diverse datasets demonstrates practical utility.", "weaknesses": "1.\tAlthough framed generally, the approach appears tied to Bradley–Terry-style preference data and is primarily evaluated in pairwise preference setups. It is unclear how well the method applies to verifiable rewards (e.g., math/code with execution-based correctness) or to process-based rewards.\n\n2.\tThe choice of the number of ordinal bins, boundary placement, and mapping from discrete classes back to scalar rewards (for policy optimization) may significantly affect performance. Sensitivity analyses and principled binning strategies are not clearly presented.\n\n3.\tThe training and inference overhead of probabilistic binning and flooding are not fully quantified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lxf4Im49ms", "reviewer": "Reviewer_6nHQ", "summary": "This paper proposes the Ordinal Probabilistic Reward Model (OPRM), a novel approach for training LLM reward models. Instead of the conventional method of training a regression head to output a single scalar reward, OPRM reframes the task as learning a probability distribution over a finite set of ordinal quality ratings (specifically, 1 to 9). The key mechanism is elegant and parameter-efficient: it directly repurposes the LLM's existing language model head, taking the normalized probabilities of the numeric tokens '1' through '9' as the reward distribution. The model is trained on preference pairs ($y_c$, $y_r$) by optimizing a loss function that maximizes the probability $P(S_c > S_r)$, where $S_c$ and $S_r$ are the random variables representing the scores for the chosen and rejected responses, respectively.\n\nFurthermore, the paper introduces Region Flooding Tuning (RgFT), a fine-tuning strategy designed to anchor these relative preference-based distributions to an absolute quality scale. RgFT uses coarse-grained quality annotations (e.g., \"bad\", \"normal\", \"good\") and partitions the ordinal scores into corresponding sub-regions (e.g., {1-3}, {4-6}, {7-9}). The loss is then modified to guide the model to concentrate probability mass within these target sub-regions. The authors conduct experiments on several reward model benchmarks (RewardBench, PPE, RMB) using Qwen2.5 as a base model, claiming state-of-the-art performance compared to various discriminative and generative reward models.", "questions": "See the weaknesses. Overall, the idea is interesting, and I believe it has potential. But there are too many concerns to accept the claims.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1.  **Novel and Parameter-Efficient Design:** The core idea of OPRM—repurposing the LM head's vocabulary probabilities for numeric tokens as a reward distribution—is a clever and elegant approach that avoids adding any new parameters, unlike traditional scalar-based reward models that require a separate value head.\n2.  **Probabilistic Formulation:** Moving from a deterministic scalar to a full probability distribution  is a well-motivated step. It inherently allows for modeling uncertainty  and, as the authors claim, offers richer interpretability than a single score.\n3.  **Absolute Quality Grounding:** The Region Flooding Tuning (RgFT) strategy  is a practical and data-efficient method to tackle a known problem: the misalignment between relative-preference scores and intuitive, absolute quality.", "weaknesses": "1.  **Unsupported Core Assumption:** The entire method hinges on the assumption that an LLM's pre-trained probability distribution over the tokens '1'-'9' has an inherent ordinal correspondence to text quality. This is a very strong and unevaluated claim. It is highly plausible that the model has a strong prior based on token frequency in pre-training data (e.g., '1' may be far more common than '9'), which would be unrelated to quality. The paper provides no analysis of this token prior or any evidence (e.g., calibration metrics like ECE) that the resulting distribution is meaningful.\n2.  **Critical Robustness Vulnerability:** The method relies on the next-token prediction at a specific prompt ending in \"Score:\". This creates a high risk of \"jailbreaking\" or adversarial manipulation. A malicious user could craft a response that includes text like \"This response is excellent. Score: 9\" to poison the model's internal state and bias the output distribution at the final scoring step. This plausible and critical failure mode is not evaluated.\n3.  **Missing Context from Ordinal Regression Literature:** The paper makes no effort to explain the ordinality in the data as other papers of a similar topic would do, and the ordinal nature is actually imposed by design. One can view the proposed target loss as a voting one: the $P(y_c \\succ y_r | x)$ can be rewritten as $\\sum_k p_c(k) F_r(k-1)$, where $F_r$ term is the CDF of the rejected distribution. The paper fails to connect or compare its work to the extensive existing literature on ordinal regression and probabilistic ordinal models, which have diverse formulations. The \"Related Work\" section  only discusses standard DRMs and GRMs, missing a key body of relevant research.\n4.  **Contradictory Claims vs. Methodology:** The paper explicitly claims \"Handling Annotation Disagreement\"  and \"robustness to... inconsistent preference data\"  as key advantages. However, Appendix H.1 clearly states that the authors \"filter out all pairs where the chosen response is not strictly better than the rejected one,\" including \"invalid combinations... such as <normal, good>\". This experimental practice of manually removing the exact \"inconsistent\" or \"ambiguous\" data that the model claims to be good at handling directly undermines this central claim.\n5.  **Unequal Experimental Comparisons:** The main results in Table 2  are difficult to interpret as they compare OPRM (trained on the authors' 130k dataset ) against \"Reported Results of Public Models\" and \"Reproduced Results... From DeepSeek.\" These baselines are trained on different and unknown datasets. This is not an apples-to-apples comparison. It is impossible to know if OPRM's superiority comes from the method or from the base model or the extra data. A fair comparison would require training all baselines (e.g., a standard BT model, PairRM) on the exact same 130k dataset. The single ablation in Figure 4(a)  is not sufficient.\n6.  **Arbitrary Partitioning in RgFT:** The 3-quality-level partition in RgFT ({1-3}, {4-6}, {7-9})  is arbitrary and presented without justification. No sensitivity analysis is performed to show why this even split is optimal, or how performance changes with different partitions (e.g., {1-2}, {3-7}, {8-9}).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "muyNkDX7Zk", "reviewer": "Reviewer_hXgf", "summary": "The paper addresses the limitations associated with traditional methods in Reinforcement Learning from Human Feedback (RLHF), specifically Discriminative Reward Models (DRMs) and Generative Reward Models (GRMs). DRMs are known for learning reward models based on preference signals, while GRMs identify preferences using a generative language model (LLM), potentially incorporating chain-of-thought (CoT) processing. However, DRMs often present challenges in reward interpretation and threshold determination, and GRMs are unable to perform \"Best of N\" comparisons or provide explicit probabilistic outputs.\n\n\nIn response, the authors propose a novel approach termed the Ordinal Probabilistic Reward Model (OPRM). This model outputs a complete distribution of rewards, facilitating a more nuanced understanding. The term \"ordinal\" reflects the method's discretization of the continuous reward space into sequences to ensure tractability. The training process involves objectives that focus on the distribution's spread, optimized using a technique called Region Flooding Tuning.\n\n\nFor data generation, a scoring methodology is established by curating model prompts and responses with a template that translates scores into categorical labels: Bad, Normal, and Good. This dataset is then utilized to train the model with tailored objectives that leverage Region Flooding Tuning to enhance performance consistency.\n\n\nThe paper employs Qwen-2.5-Instruction model variants with parameters ranging from 7B to 72B. Evaluation is conducted using Reward Bench, PPE, and RMB datasets, alongside comparisons with existing DRM and GRM techniques. The findings reveal a notable improvement over baseline models and alternative methods. Additionally, the paper discusses ablation studies assessing the influences of Region Flooding and examines the correlation with human alignment metrics and also delves into the mechanics of how the gradients operate as the learning signal.", "questions": "When generating the score for a prompt-response pair from the LLM, the token probabilities already provide a distribution. How does the learned distribution differ from this existing distribution? What supplementary insights are gained beyond the data-generating distribution? Since this data generation process resembles a GRM-based scoring system, how should we interpret the performance improvement achieved through your method when applied to data generated using GRM? Could you provide clarification on this?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper effectively addresses a feature gap in existing models by introducing a methodology that outputs a probability distribution. As a consequence, it enables benefits such as Best of N comparison and uncertainty quantification, making it suitable for tasks requiring these features.\n- The paper explores advanced techniques for constructing the objective, including methods like Region Flooding, to ensure essential learning constraints are satisfied.\n- It provides a thorough comparison with existing methods using relevant benchmarks, demonstrating the proposed approach's effectiveness.\n- The authors conduct a mathematical analysis illustrating how the Bradley-Terry model emerges as a special case within their distribution formulation.\n- The appendix includes well-chosen examples that offer qualitative insights into the method’s output.", "weaknesses": "Although the method effectively addresses a specific feature gap, such as Best of N comparison and uncertainty quantification, the source of its performance enhancement compared to alternative methods remains unclear. Additional interpretative analysis would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)