ResponseRank: Data-Efficient Reward Modeling through Preference Strength Learning
Authors: Timo Kaufmann
Yannick Metz
Daniel A. Keim
Eyke Hüllermeier
Venue: NeurIPS 2025 poster ()
Keywords: Reward Modeling
Response Time
RLHF
Preference Learning
Preference Strength
Learning from Rankings
Data Efficiency
Primary area: general_machine_learning
TL;DR: ResponseRank enables data-efficient learning of distance-aware reward models through stratified comparison strength rankings.

Abstract: Binary choices, as often used for reinforcement learning from human feedback (RLHF), convey only the *direction* of a preference. A person may choose apples over oranges and bananas over grapes, but *which preference is stronger*? Strength is crucial for decision-making under uncertainty and generalization of preference models, but hard to measure reliably. Metadata such as response times and inter-annotator agreement can serve as proxies for strength, but are often noisy and confounded. We propose ResponseRank to address the challenge of learning from noisy strength signals. Our method uses relative differences in proxy signals to *rank responses to pairwise comparisons by their inferred preference strength*. To control for systemic variation, we compare signals only locally within carefully constructed strata. This enables robust learning of utility differences consistent with strength-derived rankings while making minimal assumptions about the strength signal. Our contributions are threefold: (1) ResponseRank, a novel method that robustly learns preference strength by leveraging locally valid relative strength signals; (2) empirical evidence of improved sample efficiency and robustness across diverse tasks: synthetic preference learning (with simulated response times), language modeling (with annotator agreement), and RL control tasks (with simulated episode returns); and (3) the *Pearson Distance Correlation (PDC)*, a novel metric that isolates cardinal utility learning from ordinal accuracy.

Response Rank: Data-Efficient Reward Modeling through Preference Strength Learning
Timo Kaufmann LMU Munich, MCML
Yannick Metz University of Konstanz
timo. kaufmann@ifi. 1mu de
yanni ik. -olduuni--oottaa.....
Daniel Keim
Eyke Hüllermeier
University of Konstanz keimšuni wwonskan..nn.................aaa
LMU Munich, MCML, DFKI
eyke&lmu.de
Abstract
Binary choices, as often used for reinforcement learning from human feedback (RLHF), convey only the direction of preference. A person may choose apples over oranges and bananas over grapes, but which preference is stronger? Strength is crucial for decision- decision-making under uncertainty and generalization of preference models, but hard to measure reliably. Metadata such as response times and inter- annotator agreement can serve as proxies for strength, but are often noisy and confounded. We propose ResponseRank address the challenge of learning from noisy strength signals. Our method uses relative differences in proxy signals to rank responses to pairwise comparisons by their inferred preference strength. control for systemic variation, compare signals only locally within carefully constructed strata. This enables robust learning utility differences consistent with strength- derived rankings while making minimal assumptions about the strength signal. Our contributions are threefold: (1) ResponseRank, novel method that robustly learns preference strength by leveraging locally valid relative strength signals; (2) empirical evidence of improved sample efficiency and robustness across diverse tasks: synthetic preference learning (with simulated response times), language modeling (with annotator agreement), and RL control tasks (with simulated episode returns); and (3) the Pearson Distance Correlation (PDC), novel metric that isolates cardinal utility learning from ordinal accuracy.
Introduction
Consider an Al system choosing between a risky action (usually good, occasionally catastrophic) and safe one (consistently mediocre). Optimal poccsinn-making in such uncertain scenarios requires knowing not just which outcome is preferred, but by how much the preference strength. A slight preference for the good outcome over the mediocre might favor safety, while strong preference could justify the risk. Knowing preference strength is essential for optimal decision-making when outcomes uncertain [1]. However, standard reinforcement learning from human feedback (RLHF) typically only collects binary preferences, identifying the preference order but not this crucial strength information the distance between utilities.
In addition to the benefits for decision making under uncertainty, learning preference strength can significantly improve reward model generalization [2, 3]. providing benefits even in deterministic tasks where the optimal policy only depends on the order of outcome returns. When assuming that preferences adhere to a latent utility function, utility differences provide additional structural information that aids learning this utility [4]. For example, if A is strongly preferred to but only
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Stratum
Preference Sirength Predictor
Udilt
normaliac
By, by 7,,
A
Stratum 2
W
Utiley Up
stratification
Ru 9,7
by
Rer b,5
Plackett Luce Loss
Target Hanking Construction
17
Sp, by, b, 5,0L,
bp. b,
Du, 5 Rgg
augment
R1 by
b2
R4 h2 9
7a
Figure 1: ResponseRank learns strength-award preferences from rankings over comparisons derived from implicit strength signals. We start from dataset of comparisons, consisting of pair of objects (a;, b,), a preference (p; E {a, 6)), strength signal (T;, here scalar response times), and metadata (m;). (1) Comparisons ensuring local validity of the signal-strength relationship. The metadata can be discarded after this step. (2) Within each stratum, instances (a,, b, Pt, T.) are normalized (w,, 4, T3) format based on the preference label p;, where w; is the preferred item and I, is the dispreferred item. (3) Rank construction then sorts these normalized tuples (e.g.. by ascending response time T;) and appends virtual anchor element with a fixed score of 0 to create target rankings. (4) A preference predictor models these target rankings by predicting signed utility differences so (w,, 42) = ug (w:) up (4)). (5) The values (and the anchor's s 0) act as latent strengths for a Plackett- model; NLL loss minimization yields parameters the utility function (or reward model) ug.
stratified using metadata m, (e.g.. by annotator ID),
weakly preferred to B. we can infer B > C' without observing that comparison directly. Moreover, strength signals can provide an implicit regularization effect, emphasizing confident preferences and downweighting noisy ones.
Lacking explicit strength signal, standard RLHF has rely entirely the model's generalization between comparisons to learn preference strength. Collecting such explicit signals requires additional annotation effort [5], however, and is susceptible to metacognitive challenges [6]. As alternative explicit collection, response metadata can often convey strength information. A motivating example of such implicit strength signals response time (RT), the time taken by annotators to provide their feedback. Faster responses often indicate stronger preferences [7, 8J. While promising, signals such as response times can be difficult learn from as they are noisy and confounded by factors such as individual annotator speed and task complexity [8, 9]. Consequently, the inverse RT-strength relationship is most reliably observed locally within homogeneous contexts, motivating methods that can robustly exploit such signals.
Contributions. (1) We introduce ResponseRank (Figure 1), novel RLHF method that extracts preference strength from any locally-valid relative strength signal by leveraging the relative order of strength within homogeneous strata (e.g., per-annotator). This local approach can mitigate systemic noise through stratification and provides a proxy for preference strength designed to improve reward model generalization and decision-making under uncertainty. (2) We empirically demonstrate ResponseRank improved accuracy and generalization in multiple domains.
(3) We introduce the Pearson Distance Correlation (PDC), novel metric designed to quantify how well model captures preference strength.
Learning preference strength from local rankings
introduce ResponseRank, novel method for learning strength-aware preference models from rankings of comparisons. Our method builds on the principle that auxiliary signals can indicate preference strength are often noisy and confounded. We learn utility differences by leveraging the relative order of these signals within homogeneous partitions (strata).

---

2.1 Problem setting and utility framework
We aim to learn a reward model from human feedback, incorporating strength signals to improve strength modeling and generalization. The input consists of a dataset D = { bú, Pt, Tú, m i- Each data point consists of two items ag, E X being compared, preference label p; indicating the preferred item, the strength signal T; (e.g., response time), and optional metadata m, (e.g,, annotator ID, session timestamp).
Our approach is grounded in utility learning, a preference learning approach seeking to learn utilityy function u X' - R that assigns scalar value each item, such that y O u(x) 2 u(y). 1 The magnitude of the utility difference, u(x) u(y), quantitatively represents the preference strength. In reinforcement learning from human feedback (RLHF), such learned utility functions often serve as reward models guiding policy optimization [10, 11]. The ResponseRank method, detailed next, learns utility function ug by leveraging both the choice p; and the strength signal TÚ to infer not just which item is preferred, but by how much.
2.2 The ResponseRank method
ResponseRank starts by constructing strength-ranker comparisons within homogeneous strata (step 1 in Figure 1). It then learns preferences and strength jointly from these rankings using a joint ranking loss (steps 2-5 in Figure 1). We detail this process in the following subsections and discuss ResponseRank's 's relation to the conventional choice-only Bradley-Terry approach.
2.2.1 Constructing strength-ranker comparisons within homogeneous strata
Stratification. A key challenge is that strength signals like response times are often confounded globally but reliable locally. Consider response times across different annotators: baseline speed differences make global comparisons misleading, but stratifyyng by annotator isolates the within- annotator strength relationship. More generally, any factor believed to systematically influence the strength signal can serve stratification criterion. enabling flexible encoding of domain knowledge. Examples include task attributes such as text complexity clusters, labeling sessions, or other contextual features.
Within each stratum, the treengt--iignn relationship is more homogeneous. Rather than explicitly modeling the signal-generation process, we exploit this local validity by leveraging ordinal (relative) information within each stratum. This ordinal-only approach is inherently robust to outliers and signal magnitude variations, allowing any locally valid strength signal to serve as proxy for preference strength only a monotonic relationship within each stratum is needed.
Partitioning and batch packing, Once strata are formed, ResponseRank optionally applies additional partitioning within each stratum, 2 driven by practical constraints the Plackett-Luce training objective: (1) Plackett-Luce requires rankings to be co-located within training batch, necessitating careful packing and potential splitting of rankings into batches; (2) Strength signals may contain ties; partitioning can ensure tie-free rankings for cleaner training signals. (3) Smaller rankings sometimes improve robustness to noisy signals.
We detail algorithms for batch packing as well as size-constrained and tie-avoidant partitioning in Appendix B. Once stratification and partitioning are complete, the final step is sorting comparisons within each partition by the strength signal, yielding the target rankings used in the learning step.
2.2.2 Learning preferences and strength jointly from comparison rankings
To learn from the strength rankings constructed in the previous section, we need model that captures both preference direction and strength. We adopt Plackett-uce [12, 13). a probabilistic ranking model that connects latent scores (strengths here) to observed rankings."
'ee use the term utility function: in RLHF literature, this is commonly called the reward function. Globally valid signals may even only require partitioning and no data-driven stratification.
$ 'Leamning from such rankings of comparisons, a form of learning from recond-orde preferences, provides a strong signal for utility differences. Under reasonable assumptions, this is sufficient to identify the utility function up a linear transformation [14]. A brief theoretical discussion can be found in Appendix F.4.
3

---

Table 1: Target ranking of single stratum using illustrative comparisons with response times as the strength signal. Original comparisons (g:) are first normalized to (winner, loser) format (g)), then sorted by ascending response time Tr (faster RTs indicate stronger preferences and are ranked higher). A virtual anchor element (latent score 0) is appended to this list for the Plackeut-Luce model. The model learns positive utility differences so(g") uo (winner) =oo(er))) ordinally consistent with this ranking (illustrative latent s(g') shown).
Comparison (g:)
Preference (p:)
Normalized (o/)
RT (Tt)
Exemplary s(q')
(a1, b1)
a > b
(a) b1)
1.2s
1.7
(a2, b2)
a2 > b2
(a2, bg)
1.7s
1.0
(az, b3)
b3 > 03
(b3, 03)
1.7s
0.9
(a1, b1)
by >
(ba, a4)
2.3s
0.5
Virtual Anchor Element
N/A
0 (fixed)
The PL model defines the probability distribution of ranking æ (q1 92.-..- 9t)) of size is as the product of sequentially choosing each comparison. The probability of qj being chosen next from the set of not-yet-ranked comparisons C* is (g from C³ ()) = exp(so(g;))) Eyees exp(so(g))).
In our case, the latent utility values to be learned are preference strengths (by construction of the ranking). To capture direction alongside strength, we require the learned strengths se (q) to be signed, with the sign indicating the preference direction. We achieve this by introducing a virtual anchor element with fixed score () and add it to each ranking. We normalize all comparisons to q (w;, 4) format (preferred item first) and place the anchor the end, resulting in all- all-positive target values. The Plackett-Luce log likelihood loss then naturally penalizes negative predictions, as they would make the target ranking, which each comparison o' is ranked above the O-valued anchor by construction, less likely. Our goal is then to learn utility differences sp so(w,, 4) uw)))) ul4)) that are positive (to match the normalized format) and ordinally consistent with the strength rankings.
Table I illustrates this process with concrete example, showing how comparisons from a single partition are normalized, ranked by strength, and augmented with the virtual anchor. The rightmost column shows illustrative predicted (g') values that are ordinally consistent with the ranking.
We optimize 9 via maximum likelihood estimation, maximizing the joint probability of the observed rankings under the PL model. Concretely, we model the utility function u as a neural network up that maps items scalar utilities (or rewards in RLHF), giving rise utility differences so so(a;, bx) uo ua(a)) - no (b:). During training, for each ranking, we predict so (q") ) for all comparisons and fit sp to the target ranking via negative log-likelihood loss minimization.
2.2.3 Relationship to Bradley-Terry
A key property of ResponseRank, as formalized in Theorem 1, is that it gracefully reduces to the standard Bradley-Terry loss in the degenerate case of single comparison per stratum (proof in Appendix G). This reduction places ResponseRank as a generalization of Bradley-Teryy and allows for seamless mixing of strength- ranked and choice-only data: For any datapoint missing strength signal, we can place it in a stratum of size 1, resulting in the standard Bradley-Term loss for that comparison.
Theorem (ResponseRank reduces to BT for a single comparison). When applying the Respon- seRank method to a siraium containing only a single normalized comparison of = (w, the target ranking for the Pluckett-Luce model is (g's do), where Ao the virtual anchor. Minimizing the NLL for this ranking, given the Plackett-Laee Luce scores so(g") uo(w)- uo(l) and s s(>o) = 0, is equivalent to minimizing the binary cross-ent cross-entropy loss of a Bradley- Terry model for the preference w > with item utilities un ua(w) and u))
Intuitively, for multiple comparisons per stratum, each comparison beating the anchor contributes BT preference term, while the ordering among comparisons encourages stronger preferences have larger utility differences. See Appendix A.4 for details.

---

Original
Transtormed
Original
Transfomeed (2u 2u++5)
1.0
03 (oos
1.0 (Basi)
Fraction of Signs Fipped
08
02
0.0 15:4
00 (oos
0.0
0.5
1.0 0.0
0.5 1.0
0.0
0.5
100 3.0
0.3
1.0
Magnitudes (roe)
Figure 2: PDC robustly isolates distance learning, unlike TCE. Heatmaps compare PDC and TCE on synthetic data under varying levels of ordinal error (fsian y-axis, flipping signs) and distance information loss (fmag. x-axis, shuffling magnitudes), for original and affine-scaled (2u + 5) utilities. PDC remains optimal (blue) when only ordinal information is degraded (Smag 0), decreases systematically as distance information is lost (mma > 0), and is consistent across utility scalings and ordinal accuracy, unlike TCE which conflates these aspects.
The Pearson Distance Correlation
To effectively quantify how well a model learns the strength distance between utilities beyond just their ordinal ranking, we introduce the Pearson Distance Correlation (PDC). While some existing metrics assess aspects of preference strength, PDC specifically designed to robustly isolate the learning of cardinal utility distances by satisfying key desirable properties. Further details, comparison with other metrics, and discussion of these properties are provided in Appendix H.
The core idea of PDC, formalized in Definition 1, is to measure the linear correlation between the absolute magnitudes of true utility differences and predicted utility differences. This focus absolute differences aims to decouple the assessment of distance learning from the correctness of ordinal preference, property term ordinal independence. Figure visually contrasts PDC with calibration eгror (TCE, I distance between predicted confidences and true likelihoods), illustrating PDC's robustness and key characteristics like ordinal independence and affine invariance.
Definition 1 (Pearson Distance Correlation). Let X, X' widd Pdata be independent items sampled from the data distribution pdsta and let u, û: X' - TR denote the true and predicted utility functions, respectively, Define the true absolute utility difference as (U(X.X))
lu(X) u(X')] and the
predicted absolute utility difference as SÚ(X,X)) = |((X)) - a(X'). The PDC the Pearson correlation coefficient between these random variables AU and AÙ;
(PDC(u, ú; Prina) = Corr AÚ) Cov(AU,
ViAU V[AŪ]
Variances and covariances are taken over the joint distribution of (X,X') where X' miid Pdata-
PDC is designed to satisfy several crucial properties for evaluating learned utility distances, as stated Proposition (details and proof Appendix H.3).
Proposition (PDC Properties). Given
utility function and predicted utility function the PDC satisfies: (i) Affine Invariance: Unchanged by positive affine utility transformations. (ii)
Distance Sensitivity: Monotonically increases with accuracy of predicted difference magnitudes. (iii
Known Baseline and Scale: Values of 0 (no distance learning) and (ideal distance learning). (iv)
Ordinal Independence: Baseline for no distance learning is independent of ordinal accuracy.
A PDC significantly greater than 0 implies non-trivial learning of utility difference magnitudes. This allows more precise statements about if. and how well, model has learned relative preference strengths (see Appendix A for an example). PDC is designed to complement standard ordinal metrics, not replace them. Due to ordinal independence, high PDC can occur even with some ordinal mispredictions, provided the magnitudes of these (potentially erroneous) differences still correlate with true magnitudes. Note that real-world datasets generally not have true utilities, making exact computation impossible. Appendix H.l discusses this problem and proposes approximation.
5

---

Determlobete
Stochortie
Drift diftuionn
77.4%
04
1.49
16
Trining Sek Slas (3)
Training Sut Siz (9)
Training Set Sloo (%)
Figure 3: Synthetic results with RT variability across strata. Comparison of PDC (top) and accuracy (bottom) on Deterministic, Drift-Diffusion (DDM), and Stochastic datasets as a function of dataset size (100 runs; shading indicates 95% CI assuming normality). BT baseline performance with the full dataset is shown as a dashed red line; ResponseRank's s interpolated breakeven point is marked by an arrow. Higher PDC indicates better learned preference strength (utility distances); higher accuracy indicates better ordinal preference predictions. ResponseRank matches BT performance with only 61-83% of the data, demonstrating the efficiency of the method. The permutation baseline KeepponseRank-Pert confirms that performance improvements stem specifically from informative strength signals rather than modeling artifacts.
4 Synthetic preference learning experiments
In this section, evaluate ResponseRank synthetic pairwise comparison datasets with known ground-truth utility differences and simulated response times. We compare against baselines on met- rics measuring preference strength learning (PDC) and ordinal accuracy under controlled conditions.
4.1 Experimental setup
Datasets. Synthetic datasets are generated by generating item pairs with random attributes, assigning utilities with a known function, and then deriving choices and strength signals in the form of synthetic RTs via human choice model. To test robustness to mismatched assumptions, we consider two distinct human choice models: (1) stochastic Bradley-Tere (BT) model extended with custom link function mapping utility differences response times, and (2) a joint (DDM) simultaneously generating stochastic choices and response times. We set model parameters to yield similar error rates (approximately 8%) in both scenarios, although the DDM exhibits higher noise in the response time generation. A 'Deterninistic" setting uses model (1) without choice/RT noise. Data is partitioned into artificial strata (simulating, c.g., different annotators) and we apply random RT multiplier to each stratum (simulating, e.g., annotator speed differences). The utility function is modeled as a small feed-forwer neural network with a single hidden layer (64 neurons, ReLU activation) and optimized with Adam W. Full details and further results are in Appendix C. The source code for our experiments is available at https: /// htt///gttttubbt com/ www.//mmmmeeeeeeeeeeeeeeeeeeeeee timokaum www.aaaeeesppssreaeaeeeee
model
Baselines. We compare ResponseRank against: (1) a Bradley-Terry (BT) baseline, the fa-facto standard for learning utilities from pairwise preferences which considers only preference labels without response time information, (2) RtRegression: models RTs as inversely proportional to utility differences via specific link function; this link function mirrors the one used in generating data for the deterministic and stochastic conditions, but does not use the stratum RT multipliers. (3) two

---

permutation control baselines (ResponseRank-Pemi ReRegression-Perm)) which train the respective methods on permuted response times to assess robustness and verify response-time informativeness. (4) ResponseRank-Pond an ablation that omits stratification in favor of single global stratum. Detailed information on baselines and modeling assumptions is described in Appendix I.
4.2 Research questions and evaluation criteria
We investigate three specific research questions: (Q1) Preference strength learning: Does ResponseR- ank better recover underlying utility differences compared to baselines? (Q2) Robustness across data generating processes: Does ResponseRank maintain performance when trained with datasets gener- ated from different assumptions? (Q3) Choice prediction accuracy: Does incorporating response-time information via ResponseRank improve ordinal preference prediction accuracy relative to baselines?
We quantitatively evaluate these aspects using two complementary metries: (1) Pearson Distance Correlation (PDC): measuring correlation between predicted and true utility differences (Section 3). High PDC indicates superior recovery of cardinal utility distances. (2) Choice Accuracy: predicting held-out preference choices; higher accuracy represents a better ordinal utility model.
4.3 Results and discussion
Figure 3 shows the performance of ResponseRank and the BT baseline as a function of dataset size. Results demonstrate consistent improvements across most settings, indicating the robustness and effectiveness of the ResponseRank approach.
Preference strength recovery (Q1). ResponseRank achieves significantly higher median PDC than BT, demonstrating improved recovery of utility difference magnitudes. The ResponseRank Perm control performs notably worse than ResponseRank, confirming improvements stem from informative RT signals, not just artifacts of the training procedure. Nonetheless ResponseRank- still demonstrates reasonable performance, highlighting our method's robustness noise in response times and bad stratification.
Robustness to generatinn model assumptions (Q2). ResponseRank consistently outperforms BT and permutation baselines across all aata-eennrttion paradigms (Deterministic, DDM, Stochastic) in median PDC and accuracy (Figure 8). This indicates ResponseRank robustly exploits relative RT signals even when data-generating mechanisms vary. The advantage is largest when the strength signal is cleanest (Deterministic) and decreases with signal noise (Stochastic, then DDM).
Improving ordinal predictions via response time (Q3). Incorporating RTs via ResponseRank significantly improves ordinal choice prediction accuracy across scenarios. This improvement, despite RTs not directly adding ordinal labels, highlights that learning better cardinal models through RTs enhances ordinal generalization. This suggests that ResponseRank could also benefit deterministic scenarios or domains less obviously requiring cardinal utilities, like language- model fine-tuning from pairwise human preferences, by learning more accurate and generalizable preference models.
4.4 Comparison to regression-based modeling and robustness analysis
In Figure 3, RtRegression underperforms both ResponseRank and the BT baseline. This is because RiRegression does not model inter-stratum variability: If there were no variability and the link between the strength signal and utility differences was perfectly known, RtRegression would viable alternative. Detailed results for this setting are available in Appendix C. Reesonneeank-PPon similarly does not take this variability into account, but its performance remains slightly above the baseline. This indicates that the ResponseRank approach is more robust than RiRegression, likely because of its much weaker assumptions about the RT-strength relationship. This highlights the practical robustness of the proposed ranking-basee based approach, which avoids rigid link-function assumptions entirely.
Permuting RTs (ResponseRank-Peness significantly degrades ResponseRank's performance, confirm- ing RTs' role in the method. However, accuracy remains above chance and PDC remains above zero, indicating robustness to even extreme amounts of noise in the raw timing data. ResponseRank's effectiveness even on the drift diffusion dataset with noisy response times (Figure 3) demonstrates its robustness to noisy, yet informative, response time signals.
7

---

MultiPref Test Accuracy
RewardBench 2 Score
0.8
0.6
0.7
0.5
0.6
0.4
ET
6T82
RR- RR-Randont
RR-RT
RR Stated
RR. RR-Agree
BT
BT@2
RR Randon
RR-RT
RR-Stated
RR. Agree
Figure 4: Response time approach comparison. We compare BT baseline (3 epochs), BT@2 (2 epochs), RR-Random (ablation: using random strength rankings size 2). RR-RT (using length buckets with size-2 constraint), RR-Stated, and RR-Agree on both MultiPref test accuracy and RewardBench performance. We report mean and 95% CI (assuming normality) across 30 seeds with distinct train/test splits.
Our experiments support ResponseRank's effectiveness and robustness. The empirical evaluation confirms considerable advantages over baselines in both cardinal and ordinal preference modeling, effective across diverse assumptions about human choice processes and providing tangible benefits even for purely ordinal prediction accuracy. These results indicate promising potential for broader applications, such as training language models, by improving model generalization through the incorporation of implicit cardinal information.
Language modeling experiments on MultiPref
To validate our method's applicability to real-world language modeling data, we conducted exper- iments the MultiPref preference dataset [15]. This dataset provides text preferences with rich metadata. It spans 227 distinct crowdworkers, 5,323 unique prompts, 10,461 distinct responses, and four annotations per instance from workers of different expertise. It includes rich data annotator expertise. preference strength (clear or slight), per-category preferences (helpful, truthful, harmless), and justifications for the preferences.
Experimental setup. We compare the BT baseline, not using any strength information, with several variants of ResponseRank using distinct strength signals: RR-Random ablates the impact of strength by using random strength rankings of size 2, RR-RT uses response time with stratification by annotator, RR-Stated uses binary stated strength of the preference (slightclear) given by the annotator with stratification by annotator identity, and RR-Agree uses the inter-annotator agreement of the annotations for each comparison in the MultiPref dataset without stratification.
Our experimental design aims to validate that incorporating preference strength information improves reward model performance. In addition the MultiPref binarized test we evaluate the cut-of-distribution RewardBench dataset, which correlates strongly with downstream performance [16].
Results. Figure shows that, given a relatively clean strength signal such as inter- aneettnnntttor agreement (RR-Agree), ResponseRank improves both in-distributton accuracy and out-of-distributaa Reward- baseline. Experiments
Bench performunce (+3.0 pp improvement over BT) compared to with smaller subsets of the training data (Figure 10 in Appendix D) further show that RR-Agree matches final BT performance in RewardBench with approximately 30% fewer preference pairs, demonstrating its data efficiency.

---

RR-RT performs poorly, indicating that response time is not useful signal of preference strength in this dataset. This may be because MultiPref's annotation protocol includes additional annotations (strength, categories, justifications) that extend response times and obscure the RT-strength relation- ship. Future work could investigate whether RT provides useful strength signals in datasets with simpler annotation protocols, or explore whether improved filtering and stratification could extract meaningful signals from MultiPref.
RR-Stated, which uses explicit strength annotations (slight vs. clear), outperforms RR-Random and RR-RT, confirming that ResponseRank benefits from informative signals. However, gains over BT are modest (+0.66 pp on RewardBench. not statistically significant, p = 0.414), likely because rankings require responses with different strength labels (limited by label imbalance) and self-reported strength annotations are inherently noisy.
Training robustness. We observe that training with the BT loss for epochs is better for MultiPref performance while three epochs (our primary baseline) improve RewardBench performance. Analysis of RewardBench subcategories (Figure 11 in Appendix D) shows that the epoch gains concentrate in the Focus category (measuring prompt adherence). We hypothesize that continued training simultaneously learns transferable skills (off-topic detection) useful for the RewardBench Focus category while also overfitting to other patterns gains benefit Focus, but costs dominate on MultiPref test. ResponseRank variants avoid this tradeoff, demonstrating a regularizing effect of the ranking loss. Even ER-Random despite fitting to entirely random strength signals, exceeds three-epoch BT performance on MultiPref test; this does not generalize to RewardBench, however, where informative strength signals are necessary.
RL control experiments
While our language modeling results demonstrate improved reward model generalization on out-of- distribution benchmarks, remaining question is whether learned preference strength translates to other domains and improved downstream policy performance. To assess this, we conducted additional experiments in the control domain using simulated episode returns strength signals. We train reward models on preference data generated using ground-truie truuh enviromment rewards and use them to optimize policies via reinforcement learning, then evaluate the resulting policies on the actual task. This allows us to test whether capturing preference strength genuinely improves the policies that agents learn, rather than merely improving reward model statistics.
Experimental setup and strength signal. Following the methodology of Metz al. [17], we tested three MuJoCo environments (HalfCheetah, Swimmer, Walker2d) and Highway merge-vO, comparing the BT baseline with ResponseRank using 5000 queries (4000 training/1000 validation) and averaging results over 5 runs per configuration. As return differences are globally valid, we used random partitions size 16 to construct the rankings.
Results. Results detailed in Appendix E show that ResponseRank-rewarr models generally increase reward model accuracy, which translates to downstream policy improvements across environments. While this general trend is clear, the experiments in this domain are small scale and the differ- ences often fall within the confidence interval. We encourage future work on applying and tuning ResponseRank to control domains at a larger scale.
Related work
Our work builds on rich literature in preference learning, auxiliary strength signals, and human feedback. In this section, we concisely highlight works most relevant to learning preference strength and the evaluation thereof. Appendix expands on this and further discusses ranking- based utility learning, explicit strength specification, and adaptive losses. We give particular attention to response times as strength signal in this discussion, as we consider them particularly promising due to their ready availability and their well-established relationship with preference strength.
Learning from choices. Learning latent utility functions from observed choices is a common approach in preference learning, widely used in RLHF [10, 18, 19]. Standard models like the Bradley-Terry model [20], when combined with proper scoring rules, can implicitly learn some aspects of utility differences, reflecting preference strength in choice probabilities. However, robustly

---

inferring preference strength solely from choices is challenging when comparisons are seen only once, as is typical in large-scale RLHF. This limitation motivates the use of additional signals, such as RTs, inter-annotato agreement, or categorical strength ratings, to capture preference strength more directly.
Response time as a strength signal. Response times as a readily available signal with an established relationship with preference strength are key motivator for ResponseRank. Prior methods that inte- grate RT' often employ explicit cognitive process models like the Drift Diffusion Model (DDM) [e.g.. 21, 2] or make strong assumptions about a direct, global relationship between RT magnitude and preference strength (e.g., 22].* ResponseRank differs fundamentally: it assumes correlations (e.g.. RT-strength) hold only locally within strata (e.g.. per-annotator) and leverages only their monotonic ordinal ranking rather than modeling the full relationship. This local, relative-ranking approach enhances robustness against confounders, avoids the need for explicit DDM modeling or its associated complexities, and bypasses restrictive data requirements (e.g., repeated queries or fixed item sets [2, 22)), rendering it suitable for typical large-scale RLHF data.
Evaluating Preference Strength. While empirical evidence suggests that using signals such as RTs can improve overall model performance and generalization [3, 2. 21J, robustly quantifying how well preference strength itself is learned remains an open challenge. Standard metrics like choice prediction accuracy [e.g., used by 2] or general calibration measures like the Brier score [e.g.. 21] only indirectly reflect success in capturing strength, not specifically isolating learned utility distances or satisfying key invariances desirable for this particular evaluative task. To address this, we propose the Pearson Distance Correlation (PDC) (Section 3) as novel metric tailored for evaluating learned preference strength by directly addressing utility distances and satisfying key invariances.
8 Discussion and conclusion
We introduced ResponseRank, a novel method to learn preference strength from locally valid relative signals of preference strength, enhancing RLHF without extra annotation.
Limitations and Future Work. In this work, we demonstrated ResponseRank's effectiveness across synthetic datasets, real- language modeling, and RL control tasks. While response times proved challenging in MultiPref - likely due to the complex annotation protocol - inter- annotator agreement demonstrated strong performance, validating the approach with informative signals. Future work should explore datasets with simpler annotation protocols where RT may prove effective, improved noise-handlin techniques (such refined stratification) to extract weak strength signals, and alternative low-cost strength proxies. Our approach also discards tied preferences, though these could be incorporated as additional indicators of small preference differences. We validated ResponseRank-trainee reward models on RewardBench (which correlates with downstream performance) and demonstrated improved policies in RL control tasks; validation in large-scale RLHF for language model policies remains important future work.
Broader applications. The core principle of ResponseRank - -inferring signal strength from localized, relative rankings of noisy auxiliary data- extends beyond RTs in RLHF It could improve human- AI alignment by capturing preference intensity from various implicit signals (e.g., hesitation) in applications like recommender systems. This approach could also generally enhance supervised learning by estimating label confidence or informativeness from contextual cues, aiding active learning or calibration.
Conclusion. ResponseRank robustly extracts cardinal preference information from noisy strength signals, and our proposed Pearson Distance Correlation (PDC) metric specifically quantifies this. Our experiments show ResponseRank significantly improves preference strength recovery (via PDC) and ordinal accuracy over choice-only basclines, indicating better sample efficiency and generalization. Key future work remains identifying cheaply available sources of strength signal or further investigating the use of response times. By enabling a more granular understanding of human preferences, ResponseRank contributes to developing AI systems that better align with the nuanced spectrum of human values, crucial for building more reliable and beneficial AI.
'Theoretical foundations of the RT- Rtrentent relationship and detailed comparison with prior methods using RT's provided in Appendix F and Appendix J.
10

---

Acknowledgments and Disclosure of Funding
The authors acknowledge support and computational resources from the Munich Center of Machine Learning (MCML), Additional computational resources were provided by several computing centers. For the CPU-based synthetic experiments, the authors gratefully acknowledge the computational and data resources provided by the Leibniz Supercomputing Centre (www.Irz.de)) For the GPU- based experiments, the authors gratefully acknowledge the scientific support and resources of the Al service infrastructure LRZ AI Systems provided by the Leibniz Supercomputing Centre (LRZ) of the Bavarian Academy of Sciences and Humanities (BAdW). funded by Bayerisches Staatsministerium für Wissenschaft und Kunst (StMWK). The authors further acknowledge support by the state of Baden-Wiine Beden-Wüütteembee Württemberg through bwHPC providing compute and GPU resources.
This work has also received funding from the European Union's Horizon Europe research and innovation programme under the Marie Sklodowska-Curt grant agreement No 101073307. The work at the University of Konstanz was partially funded by the Deutsche Forschungsgemeiisscattt (DFG, German Research Foundation) Project-ID 251654672 TRR 161.
References
[1] Ariel Rubinstein. Lecture 2. Utility. In Lecture Notes in Microecooonic Theory. Princeton University Press, 2006. URI. https::/aasstttt wwwpaastee..rrsss..eeee.. prr.ett..ereess...tttooo.. wwtp://wbattteeeeeeeettteeeeeeae
pdf.
[2] Shen Li, Yuyang Zhang, Zhaolin Ren, Claire Liang. Na Li, and Julie A. Shah. Enhancing Preference- based Linear Bandits via Human Response Time. In Advances in Neural Information Processing Systems (NeurIPS), 2024. URL http:htt//oocceede/eeeeeeeo neurriiniiinnnnnnn..eeeee.ee wwtpaper_fiilieeeeeee saaper_files/paaaee haales//paerr2224//aaaa. /e2 aoff
Conffeeeceeoeceeceeeeeeeeeeeeeee html.
[3] John A. Clithero. Improving out- sample predictions using response times and a model of the decision Journal of Economic Behavior & Organization, 148:344 375, 2018. doi: 10.101 (/b6//..ebo.2000 02.007.
[4] Ziwei Xu and Mohan Kankanhalli. Strong Preferences Affect the Robustness of Preference Models and Value Alignment. In Proceedings of the International Conference on Learning Representations (ICLR), 2025. URL
[5] Nils Wilde. Erdem Biyik, Dorsa Sadigh, and Stephen L. Smith. Learning Reward Functions from Scale Feedback. In Proceedings of the Conference on Robot Learning (CoRL). PMLR. 2021. URL https://proceediing mlr, wwtpp//eee//aipeeeeeeeeeeeeeeeee hwt.s////wlee222.. html.
[6] Stephen M. Fleming and Hakwan C. Lau. How to measure metacognition. Frontiers in Human Newro- science, 8, 2014. doi: 10.3389/fnhum.201444 2014.00443.
[7] Milica Milosavljevic, Jonathan Malmaud, Alexander Huth, Christof Koch, and Antonio Rangel. The Drift Diffusion Model can account for the accuracy and reaction time of value-based choices under high and low time pressure. Judgment and Decision Making, 5(6):437-449, 2010. doi: 1017788990027550000222 285.
[8] Peter G. Moffatt. Stochastic Choice and the Allocation of Cognitive Effort. Experimental Economics, 8(4); 369-388, 2005. doi: 10. 1007//5068300055577777
[9] Benjamin Goecke, Florian Schmitz, and Oliver Wilhelm. Binding Costs in Processing Efficiency as Deter- minants of Cognitive Ability, Journal of Intelligence, 9(2):18, 2021, doi: 10.330/jieelliemcc00001111
[10] Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforce- ment Learning from Human Preferences. Advances Neural Information Processing Systems (NIPS). Curran Associates, Inc., 2017. URL https: : /ttp:::::://reeee http://prceeeeeenee///ee/// newriiseusppp.. ct/ ct/pappr//011//1 haa.////aah////eeeeeeeee d5e2c 0adad503:91f91.d htw.:999.....4400000000000 -ateeeee----bb-rarteeeee Abstract html.
[11] Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. Survey of Reinforcement Learning from Human Feedback. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL
https://openrevicet
[12] R. Plackett. The Analysis of Permutations. Journal of the Royal Statistical Society. Series C (Applied Statistics), 2432):1 1975. doi: 10.2307/2346567.
[13] R. Duncan Luce. Individual Choice Behavior: Theoretical Analysis. John Wiley, 1959.
[14] Patrick Suppes and Muriel Winet. An Axiomatization of Utility Based on the Notion of Utility Differences. Management Science, 1(3-4):259-21) 270, 1955, doi: 10. 1.37/mmscc11.34.255

---

[15] Lester James Validad Miranda, Yizhong Wang, Yanai Elazar, Sachin Kumar, Valentina Pyatkin, Faeze Brahman, Noah A. Smith, Hannanch Hajishirzi, and Pradeep Dasigi. Hybrid Preferences: Learning Route Instances Human vs. Feedback. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL), Association for Computational Linguisties, 2025. doi: 10. 122 acl-Iong.355.
[16] Saumya Malik, Valentina Pyatkin. Sander Land, Jacob Morrison, Noah A. Smith, Hannaneh Hajishirzi, and Nathan Lambert. RewardBench 2: Advancing Reward Model Evaluation, 2025. URL http: //arxiv. //aaxiv.org/aaattt wwwg/ba////0/g00 01937. preprint.
[17] Yannick Metz, Andras Geiszl, Raphaël Baur, and Mennatallah El-Assady, Reward Learning from Multiple Feedback Types. In Proceedings of the International Conference on Learning Representations (ICLR), 2025. URL
[18] Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NewrIPSS) 2022. URL h http:///pcooeee..e//////// www.:eedddnnnnnn..iiieeee /blefe658ee334441111144488668883
Cow.rracctt-oofeeee...... html.
[19] Nisan Stiemnon, Long Ouyang. Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems (NeurIPS). Curran Associates, Inc., 2020. URL https: / //poceeedncs. neurips. wttp://cccpppppccaaepppppeee paappee//200aaaaaaa8aeeaeeee2220 /tt.://8888500099990000000066600
www.taat.tttaaaee html.
[20] Ralph Allan Bradley and Milton E. Terry. Rank Analysis of Incomplete Block Designs: The Method of Paired Comparisons. Biometrika, 39(3/4)3324 345, 1952. doi: 10.2307/2334029.
[21] Michael Shvartsman, Benjamin Letham, Eytan Bakshy, and Stephen Keeley. Response Time Improves Gaussian Process Models for Perception and Preferences. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI). PMLR, 2024. URL attpss:/ppooeeddiii mlr ppress/v24// shvar tsman24a. html.
[22] C. Jansen, H. Blocher, T. Augustin, and G. Schollmeyer. Information efficient learning of complexly structured preferences: Elicitation procedures and their application to decision making under uncertainty. International Journal of Approximate Reasoning, 144:69-91, 2022.
doi: (...001/j.jjar.222222
[23] Tilmann Gneiting and Adrian E. Raftery. Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, 102(477) 2007. doi: 191800661140000000000
[24] Kenneth E. Train. Discrete Choice Methods with Simulation. Cambridge University Press, 2 edition, 2009. doi: 10.1017/CB0978051 1805271.
[25] Chuan Guo. Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On Calibration of Modern Neural Networks. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2017. URL mlr, ppess//70/guu11/2 html.
https: https://proceeddnngg
[26] Telmo Silva Filho, Hao Song, Miquel Perello-Nieto, Raul Santos- Rodriguez, Santos-Rodriguez, Meelis Kull, and Peter Flach. Classifier calibration: A survey on how assess and improve predicted class probabilities. Machine Learning, 112(9)3211-322) 2023, doi: 10. 10000/009944-023 023-0033---33
[27] L. R. Ford. Solution of a Ranking Problem from Binary Comparisons. The American Mathematical Monthly, 64(8):28-33, 1957. doi: 10.2307/2308513.
[28] Heejong Bong and Alessandro Rinaldo. Generalized Results for the Existence and Consistency of the ML.F Luce in the Bradley-Terry -Lrre-Luee Model. In Proceedings of the International Conference on Machine Learning
(ICML). PMLR, 2022. URL https: //poce/e//oocooossssoooo mlr, wwwss/pesssiiieeee /etes///vv/////// 2ea///////g2aa.... html.
[29] David R. Hunter. MM algorithms for generalized Bradley-Terry models. The Annals of Statistics, 32(1): 486400, 2004. doi: 10.121 2/2//////0999//0011
[30] Hao Sun, Yunyi Shen, and Jcan-Francois Ton. Rethinking Bradley-Terry Models Preference-Basee Reward Modeling: Foundations, Theory, and Alternatives, 2025. URL .04991. preprint.
2b1//411
[31] Ruijian Han and Yiming Xu. A unified analysis of likelihood based estimators in the Plackett- Luce model. The Annals of Statistics. 53(5):2077-2102 2025. doi: 10.1214/55-A022330
12

---

[32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems (NIPS). Curran Associates, Inc., 2017. URL https: : httpp/aaprsa.//////// wwwppa.aapps...................p pctps/caapp/p/1///aaaateee Abstract.
[33] Roger Ratcliff and Gail McKoon. The Diffusion Decision Model: Theory and Data for Two-Choice Decision Tasks. Neural computation, 20(4))873 922292,73- 2008. doi: 10.1 1116////cco220882200066640
[34] Yadolah Dodge, Randomized Block Design. In The Concise Encyclopedia of Statistics, pages 447-448. Springer, New York, NY, 2008. doi: 10.1007/978.03337-923333 1344.
[35] AI @ Meta Llama Team. The Llama Herd of Models, 2024. URL http: hww.:htttp//....ooooo/ httppp///aaii..orrraa///00111
1783. preprint.
[36] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannanch Hajishirzi. RewardBench: Evaluating Reward Models for Language Modeling. In Findings of the Association for Compudational Lin- guistics: NAACL 2025. Association for Computational Linguistics, 2025. doi: 10.18653/ 1/2025. findings- naacl.96.
[37] Emanuel Todorov, Tom Erez, and Yuval Tassa. MulaCo: A physics engine model-based control. In Proceedings of the JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2012. doi: 201100000..2022...388
[38] Edouard Leurent. An Environment for Autonomous Driving Decision-Making. https: bettps///githn htttp://githhub.om/// com/e leurent/hs wwwpreettthhhway-eeee gewaiyhwwwyyy.. 2018.
[39] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms, 2017. URL http:://rxxxi. wwtp://////////////////// /atg///bb//77...///// .03447. preprint.
[40] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Gable-Basclincs'. Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research, 22(268):1-8, 2021. ISSN 1533-7928. URL http://jmlz: arggpapers/v222 Saapers/v22/20. 22222221136222- html.
[41] Daniel S. Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations. In Proceedings the International Conference on Machine Learning (ICMI.). PMIR, 2019. URL http:/ httpp://prooeeeding mlr. prese/v9 7/brown19a, htm...
[42] Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In Proceedings of the International Conference on Learning Reprexentations (ICLR), 2019. URL https: httpps:///penrvveev..o net / fforumPdd=BgggRRCQQV.
[43] Vael Gates, Frederick Callaway, Mark K. inferences about others' preferences based on response times. Cognition, 217:104885, 2021. doi: 10. 10.1016/j.oogniiion20000 wttp:aiiioon222..1.1 2021..0488....
Ho, and Thomas L., Griffiths. A rational model of people's
[44] Cary Frydman and Ian Krajbich. Using Response Times to Infer Others' Private Information: An Application to Information Cascades. Management Science, 68(4):2970-2986, 2022. doi: 10. 1217/mmss.22211.3992
[45] Arkady Konovalov and Ian Krajbich. Decision Times Reveal Private Information in Strategic Set- tings: Evidence from Bargaining Experiments. The Economic Journal, 1336656):3007-3333 2023. doi: 10.1093/ej/uead055
[46] Sophie Bavard, Erik Stuchly, Arkady Konovalov, and Sebastian Gluth. Humans can infer social preferences from decision speed alone. PLoS biology, 2206)230 2024. doi: 10. wtwp///allppooo....o pbi0.3002686.
[47] Carlos Alós-Femerr, Frnst Fehr, and Nick Netzer. Time Will Tell: Recovering Preferences When Choices Are Noisy. Journal of Political Economy, 12986))1828-1877 2021. doi: 10.6//3773333.
[48] Roger Ratcliff. A theory of memory retrieval. Psychological Review, 85(2):59-108, 1978. doi: 10. 00.11000033-295X.0 255...
[49] Danny Campbell, Morten Raun Mørkbak, and Søren Bøye Olsen. The link between response time and preference, variance and processing heterogeneity in stated choice experiments. Journal of Environmental Economics and Monagement, 88:18 34, 2018. doi: 10. 1..1166f.eem.220110000333
13

---

[50] Eladio Montero-Poreee Porras, Tom Lenaerts, Riccardo Gallotti, and Jelena Grujic. Fast deliberation is related to unconditional behaviour in iterated Prisoners' Dilemma experiments. Scientific Reports, 12(1):20287.
2022. doi: 10.1038/s4 -98-022-24449-44
[51] Jerome R. Busemeyer. Decision making under uncertainty: A comparison of simple scalability, fixed- sample, and sequential-samping models. Journal of Experimenal Psychology: Learning, Memory, and Cognition, 11(3):388-564 1985. doi: 10.103770278-7333.1...
[52] Christian Szegedy. Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the Inception Architecture for Computer Vision. In /EEE Conference on Computer Vision and Puttern Recognition (CVPR), 2016. doi: 109/C
[53] Rafael Müller, Simon Komblith, and Geoffre E Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems (NeurIPS). Curran Associates, Inc., 2019. URL htttps://ppocee dings. .ongs.menn meurips. wwtpaper//00 2019 htw.///aaaaa//aaaeeaaeeeaeeeeeaa fetaahff17488e TTATA466000000449999751100011110 wtwpreabaeett--baaaaataaaaa html.
[54] Julian Lienen and Eyke Hüllermeier. From Label Smoothing to Label Relaxation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021. doi: 10. 10.1609aaaii.... (35:10.17041)
[55] Johannes Fürnkranz, Eyke Hüllermeier, Eneldo Loza Mencía, and Klaus Brinker. Multilahel classification via calibrated label ranking. Machine Learning, 73(2):133-153, 2008. doi: 10990077/00994-008-- 000-5----8.
[56] Rebecca Roelofs, Nicholas Cain, Jonathon Shlens, and Michael C. Mozer. Mitigating Bias in Calibration Error Estimation. In Proceedings the International Conference on Artificial Intelligence and Statistics (AISTATS). PMLR, 2022. URL https: httpee//ppcopeeeeeeeeeeeee mlr ppwsss/vii////p /wtt://ooo//oo2..a..mmmm html.
[57] Ananya Kumar, Perey S Liang, and Tengyu Ma. Verified Uncertainty Calibration. In Advances in Neural Information Processing Systems (NeurlPSS). Curran Associates, Inc., 2019. URL https: h/tpr//porceeee... neurips.cc/ wwwpccppppppppaeeeeecccecee.eepe petapper/01//// htt0///////////00066666350311130
Abstractt-aaattt........cc.aac.. html.
[58] Adam Gleave, Michael D. Dennis, Shane Legg, Stuart Russell, and Jan Leike. Quantifying Differences in Reward Functions. In Procediigss of the International Conference on Learning Representations (ICLR), /3448.
2021. URL. https: / //tlrr.clrr////// cc/ wwirtuaa/221/////
[59] Joseph Lee Rodgers and W. Alan and Nicewander. Thirteen Ways Look the Correlation Coefficient. The American Staristician, 42(1):59 (95666, 1988. doi: 10. 1080/0003110: 3005. 1988. 10475524.
[60] Yadolah Dodge. Coefficient of Determination. In The Concise Encyclopedia of Statistics, pages 88 91. Springer, New York, NY, 2008 doi: 10000/98---387-3288800
[61] Yadolah Dodge. Correlation Coefficient. In The Concise Encyclopedia of Statistics, pages 115- 119. Springer, New York, NY, 2008. doi: 10.
[62] Robert V. Hogg, Joseph W. McKean, and Allen T. Craig. Introduction to Mathematical Statistics. Pearson Education, 8 edition, 2019. ISBN 978-0-13 17780-11--400 13-66699--8
[63] Minyoung Hwang, Gunmin Lee, Hogun Kee, Chan Woo Kim, Kyungjae Lee, and Songhwai Oh. Sequential Preference Ranking for Efficient Reinforcement Learning from Human Feedback. In Advances in Neural Information Processing Systems (NewrIPS), 2023. URL /ttpttts::////oeoeet nww.ridnnnninnnnnnnnneo.o ec/paper_f 1les/paper/2023 htwp//2///aaahaaa httphhhee99666cca866eeeeeeeeeee0 http://ccee.esee.22aaaaaaaaaeeee htwp::/eecccbcbcbbbbabbaeaeeeeee Abstract- Conf erence. html.
[64] Heewoong Choi, Sangwon Jung, Hongjoon Ahn, and Taesup Moon. Listwise Reward Estimation for Offline Preference- preference-based Reinforeement Learning In Proceedings of the International Conference on Machine Learning (ICML), 2024. URL https: //pprr/viee/eeeeeeee
[65] Xuening Feng. Zhaohui Jiang, Timo Kaufmann, Eyke Hüllermeier, Paul Weng. and Yifei Zhu. Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries. In Proceedings of the International Conference on Machine Learning (ICML), 2025. URL https: / //tp://////////i///////// net/t orum? idd SeeePdddCCfgsWrrre
[66] Dimitris Papadimitriou and Daniel S. Brown. Bayesian Constraint Inference from User Demonstrations Based on Margin- Respecting Preference Models. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2024, doi: N0.1109/CCRA57147.2222 10144.00611000 1095.
[67] Ilgee Hong, Zichong Li, Alexander Bukharin, Yixiao Li, Haoming Jiang, Tianbao Yang, and Tuo Zhao. Adaptive Preference Scaling for Reinforcement Learning with Human Feedback. In Advances in Neural Information Processing Systems (NeuriPS), 2024. URL https: http://aappessstsee www.iiii.cppppppooo cc/paper___ saaperffllespp hataaaaaea1111aa5544444444eeeee0 htt.://00000a2200tttttttttttttt -wwetraatttCoooooeeeeeeeee html.
[68] Eric-Jan Wagenmakers, Han L. J. Van Der Maas, and Raoul P. P. P. Grasman. An EZ-diffusion model for response time and accuracy. Psychonomic Bulletin & Review, 14(1):3 22, 2007. doi: D0.7758/BF03194022
14

---

Appendix
A Implicit learning of preference strength in standard choice models
Standard pairwise preference labels (a > b) used in reinforcement learning from human feedback (RLHF) primarily yield ordinal information (i.e., u(a) 2 u(b)). However, common RLHF method- ologies often implicitly capture cardinal preference strength (the magnitude of u(a) u(b)) through several interconnected mechanisms. This implicit capture of information is suggested by the empirical success of RLHF methods (e.g., 10]. If utility differences were learned entirely arbitrarily, preserving only preference order but not magnitude, these methods would likely struggle to learn effective and generalizable policies.
A.1 Mechanisms of implicit strength learning
Several factors contribute this implicit learning of preference strength. First, probabilistic choice models like the Bradley-Terry (BT) model [20] or Plackett-Luce models [12] inherently link choice probabilities latent utility differences. In such models, for example the BT model, the probability p(a > b) can be expressed as a function of the utility difference u(a) u(b), such as a(u(a) u(b)). This means that larger utility differences yield choice probabilities closer 0 or thereby encoding preference strength within the choice probability distribution itself. Second, training these models typically involves minimizing proper scoring rule, such as cross-entropy loss [23]. This objective incentivizes the model to accurately predict the true choice probabilities. Since these probabilities are dependent on utility differences, the training process implicitly pushes the model to learn the underlying differences necessary to match the observed choice patterns [24]. Crucially, while the absolute utility values u(x) from such models are typically identifiable" only up to an arbitrary additive shift, the resulting learned utility differences u(a) u(b) are shift-invariant and become meaningful representations of preference strength and choice probability [24).
Furthermore, inductive biases from the modeling approach assist in learning strength. The core assumption of a consistent utility function enforces transitivity. The use of smooth function ap- proximators, such as neural networks, encourages generalization and interpolation of relative utility differences across similar item pairings. For example, if u(a) > u(b) and b zs c, smoothness in the utility approximator promotes learning that u(aa > u(c). This capability is crucial, as proper scoring rules alone may not recover true probabilities when only ever observing unique training instances with hard, binary labels.
The Bradley-Terry model serves clear illustration of these principles in action. It is a popular model for pairwise comparisons, where the probability of preferring item i over item j is typically defined as:
exp(u:)
p(i; ui, Uz) exp(u:) + exp(u;)
where i, and uy represent the utilities of items and respectively. This expression is akin pairwise logistic regression model, where the utility difference effectively serves as a feature. In many practical RLHF scenarios, we only observe a single outcome for each comparison (e.g.. i is chosen over j), meaning the training labels are degenerate (p; E {0, 13). In principle, fitting a model to such degenerate labels without any regularization could theoretically push the learned absolute utility differences towards infinity, as the model continually seeks to improve the likelihood for the observed certain outcome, This issue parallels the challenge of fitting a classifier to perfectly separable hard labels, which can lead to overconfident and poorly calibrated models that assign extreme probabilities [25, 26]. In both preference learning and classification, the goal extends beyond mere accuracy in the primary prediction (the choice or the class label) to also obtaining meaningful secondary information, such as preference strength or a well-calibrated probability. The aforementioned inductive biases, particularly smoothness from neural network approximators, help to regularize the learning process and yield finite, meaningful utility differences.
'See Appendix A.3 for limitations of this identifiability in the RLHF setting.
15

---

A.2 Empirical evidence and limitations
Our own experimental results provide empirical evidence for this implicit learning of distance by standard models. As demonstrated in Figure 3 in the main paper, the Bradley-Terry baseline consistently achieves a Pearson Distance Correlation (PDC) that is significantly and robustly above zero across various synthetic datasets and conditions. As established in Appendix H, PDC value greater than zero robustly indicates that the model has learned non-trivial information about the true underlying utility differences.
Despite these mechanisms allowing for implicit strength capture, it is crucial to recognize their limitations. The preference strength information learned in this manner is indirect and can be weak, noisy, or highly contingent on the specific validity of model assumptions, the quality of the data, and the chosen architectural details. The inherent indirectness and potential unreliability of this implicitly learned strength motivate the development of methods like ResponseRank. ResponseRank aims to leverage more direct and potentially richer signals of preference strength, such as human response times, for more robust reward modeling. Indeed, while our results confirm that standard BT models do implicitly learn preference strength to some degree, these same results (Figure 3 in Appendix C) also clearly demonstrate that ResponseRank consistently achieves superior performance in capturing these cardinal utility differences, highlighting the benefits of its explicit, RT-informed approach.
A.3 Identifiability
We briefly discuss identifiability of utilities in choice models and how ResponseRank compares to Bradley-Terry.
Classical results. In the Bradley-Terry model with a fixed finite set of items, the MILE of the item utilities is identifiable up to an additive constant under the condition that the comparison graph is strongly connected [27], and is consistent given sufficient data [28].
ResponseRank and antisymmetry of strength. ResponseRank builds on the Plackett-Luce model, a generalization of BT to rankings. Plackett- Luce item strengths are identifiable up to additive shift under similar conditions as BT [29]. For ResponseRank, we further constrain this model as we rank comparisons and prescribe that item strengths (each comparison being an item) take the form $(a.6) u(a) u(b). This introduces antisymmetry (#(a.)) -(bba)) which prevents additive shift of the strength, making item strengths exactly identifiable. Strength implies utility up to shift (e.g.. fixing any reference item I, we have u(a)= = ((a,x) + uxx)). Thus, ResponseRank has the same identifiability properties for item utilities and preference strength as BT. Note, however, that in the general case the identified utilities are not the same as those BT would identify from the pairwise
comparisons alone, as ResponseRank uses more information to better estimate strength from finite data.
Structured setting (RLHF). The above identifiability results primarily consider the unstructured setting, where we estimate individual item utilities from comparisons among fixed set of items. In RLHF, items are typically unique prompt-respone function approximator instead of individual item utilities. This utility function makes use of structured items with feature vectors (e.g., embeddings of prompt-response pairs). This setting is called BT- Regression by Sun et al. [30]. Classical identifiability results still inform what can be learned: utilities only up to shift, and meaningful comparisons only between items or features connected by similar training comparisons. If the dataset contains only within- -omiinn comparisons (e.g., poems vs. poems, code vs. code, but never across), cross-domain utility comparison not grounded. ResponseRank, like BT, inherits these limitations, though the strength-based ranking provides additional signal that may help constrain learned utilities under limited data.
pairs, and we estimate parameters of a utility
A.4 Structural interpretation via rank-breaking
To build intuition for what the ResponseRank loss captures beyond the single-comparises case (Theorem 1), it is helpful to analyze its full rank breaking, the decomposition of each ranking into all implied pairwise comparisons. This analysis is meaningful because the quasi-MLE obtained by applying Bradley-Terry to a full rank-brean bbe-biea targets the same parameters as the full PL. MLE [31, Proposition 3.2].
16

---

The full breaking of a ResponseRank ranking consists of two types of pairs: (1) pairs between comparison and the anchor element, and (2) pairs between two non-anchor comparisons. Each pair of type (1) has the same structure as the single-comparison case in Theorem 1 and therefore contributes a standard BT loss term for the corresponding comparison's is preference direction, Pairs of type (2) further constrain the relative magnitudes of preference strengths within each stratum: if normalized comparison of is ranked above qf. the loss encourages se (g(() > sg(g)). Thus, ResponseRank can be viewed Bradley-Tery on the preference directions, augmented with additional pairwise constraints on the order of strengths.
B Ranking construction details
While conceptually the primary means of constructing rankings is stratification, ensuring that the strength signal is comparable within each stratum, practical considerations often necessitate additional constraints: (1) Strata may have fied strength signals (e.g., two comparisons with identical response time from the same annotator), requiring partitioning into smaller rankings to avoid ties. (2) Strata may be large, while smaller rankings may be preferable in some cases (see Appendix D.6 for discussion). (3) Each ranking needs to be entirely co-located in a single on-device batch during training compute the ranking loss, imposing memory constraints on ranking size. We address the first two considerations with a fie- tie-breaking partitioning strategy within each stratum and the third with batch packing algorithm.
a
B.1 Tie-breaking partitioning strategy
The ranking loss requires preferences within each partition to have distinct strength values. Addi- tionally. it often desirable to split these strata further, they may contain ties strength values and smaller rankings may prove more robust to noisy signals (see Appendix D.6). Therefore, after stratification (if applicable), we further partition the data within each stratum into tie-free partitions with an optional maximum size s. Given stratum of size n we compute the number of partitions m, where m is the size of the largest tie group, otherwise. We group preferences by strength value, then distribute each group across the i partitions round-robin fashion. As long as is at least the size of the largest tie group, this guarantees that no two preferences within a partition have the same strength value. This maintains balanced partition sizes while maximizing the spread of strength values within each partition.
[n/s] if a maximum size s is specified
B.2 Batch packing
The Plackett-Luce loss computation requires entire rankings to be co-located within single on-device training batches, as the probability calculation depends on the full set of comparisons within each stratum. Although the resulting batches are not i..dd, as they contain comparisons from the same stratum (partially mediated by packing strata randomly into batches), this has precedent in techniques like length-basee based grouping for efficient GPU utilization [32] and has little performance impact in practice (see Figure 10 in Appendix D). The batch size additionally constrains maximum ranking length, which is further analyzed in Appendix D.5.
We designed greedy 'largest first best fit' approach to pack the partitioned rankings into fixed-size batches for training:
Initialize sufficient empty batches of fixed size to hold all comparisons. All batches will be filled to capacity, except possibly the last one.
Process partition fragments in descending order of size (implemented using a max-heap). Shuffle each fragment's comparisons randomize split points.
Place the fragment in the batch with the least remaining space that can still accommodate it, if one exists. Otherwise place the largest prefix that fits and reinsert the remaining suffix into the heap.
While this leads to non-i.i.d. batches, we observe no adverse effects on training performance
'Validated by comparing BT with random vs. RR-Agree batching, with RR- Agree batching showing minor improvement in performance: MultiPref test +0.9 pp (p 0.001), RewardBench 2 +0.4 pp 0.536).
17

---

Table 2: Parameters for the synthetic data generation models.
Model
Parameter
Value
Cradley-Terry & Log-Normal
Choice temperature Response time SD
0.2
0.4
Minimum response time
0
Maximum response time
10
Partition RT variability
3.0
Decision threshold
1.2
Non-decision time
0.3
Drift rate multiplier
1.0
Noise standard deviation
0.4
Drift rate variability
0.0
Starting point variability
0.0
Non-decision time variability
0.0
Time increment
0.001
Number of partitions
Partition RT variability
3.0
C
Supplemental details on synthetic experiments
In this appendix we first provide additional details on the synthetic experimental setup, including details the synthetic dataset and training procedure. We then provide additional plots and numeric results for reproducibility.
C.1 Detailed experimental setup
We use three synthetic datasets in our experiments: a deterministic dataset where the choice always favors the item with the higher utility, a stochastic dataset where the choice is made according to Bradley-Terry model parameterized by the true utility, and drift-diffusion dataset where choice and response times are generated jointly with a drift-diffusion model parameterized by the utility difference the drift rate. The parameters for both models are shown in Table 2.
We generate synthetic dataset of pairwise comparisons annotated with choice label and a response time in two steps: query generation and labeling.
C.1.1 Query generation
This process generates pairs of objects for comparison. It be further subdivided into object generation and pairing. We start by generating random objects, which are represented as vectors of 20 attributes each sampled from a uniform distribution over [0, 1]. The number of features and the dimensionality of the feature space can be adjusted to control the complexity of the dataset. We then generate queries from these objects by randomly sampling pairs of objects (a;, bz).
C.1.2 Labeling
We start by generating a random utility function u, modelled as neural network with random weights. The random utility function simulates complex but learnable relationship between object attributes and utility and uses two hidden layers (one more than the reward predictor) with 64 neurons each and ReLU nonlinearity. It takes an object x as input and outputs scalar utility value u(x) that is function of the object's 's attributes and is guaranteed to be learnable by neural network. The architecture of the utility function can be adjusted to control the complexity the dataset. We then synthesize preference labels and response times based on the utility function. To test the robustness of ResponseRank to different preference and response time models, we consider two different models for generating the preference labels and response times, one generating preferences and response times independently, and one generating them jointly. We discuss the models more detail below.
18

---

(99.200 2913
70L (S77))
(0.0%)
(46.1R
(5.88)
(46.7:5
Optin
Optine
Class Optans
Oprikz
3003
Cyrkka
(a) Deterministic
(b) Stochastic
(c) DDM
Figure 5: Confusion matrices of the synthetic datasets.
Independent labeling with Bradley-Terry and Log-Normal. We generate preference labels p; based on Bradley-Terry model and response times тi from a log-normal distribution with hyperbolic link function. Concretely, we sample p; from Bernoulli distribution with parameter p = ((((a,) u(b)) (l)))/0), where a is the sigmoid function and the choice temperature. For response times, we first compute the mean using a hyperbolic link function (matching Equation (14) and repeated here for convenience)
rtmax rtmin
link Au; remin Ilmax) = rtmin +
Aul +
with rtmin = 0 and Itmax 10. Then, we sample T; from a lor-normal distribution with this mean and fixed standard deviation parameter. The log-normas distribution ensures that response times are always positive while allowing for right-skewed distribution commonly observed in human response time data. Note that in this model, the response time depends only on the absolute utility difference, so 'correct' and 'incorrect' responses are not differentiated in their response times.
Partition-basse Variability. To better simulate real-world data collection scenarios, introduce an additional source of systematic variability in response times through partition-based approach. Each comparison is randomly assigned one of partitions, representing different data sources, annotators, or experimental conditions.
Within each partition, all response times scaled by a partition-speciffe multiplier. These multipliers are sampled from normal distribution with mean 1.0 and standard deviation 3.0, then converted to absolute values ensure they remain positive. This creates systematic variation in response times between partitions while maintaining the same underlying preference patterns, simulating scenarios where different data sources might exhibit different response time characteristics.
Joint labeling with Drift-Diffusion Diffusion. We jointly generate preference labels p; and response times based the drift-diffusion model. We derive the drift rate from the utility difference, i.e., choice experiment comparing a; and b, modeled as a drift-diffusion process with drift rate u(a:) u(b)). Since all items generated from the same distribution and passed through the same utility function, the drift rate has an expected value of Additionally, we normalize the standard deviation to 1. All other parameters, such the decision threshold, non-decision time, and noise standard deviation, are fixed to constant values as shown in Table 2. We assume symmetric decision thresholds, which intuitively control how cautious the decision-maker is. We chose these parameters to ensure a reasonable resulting joint distribution of response times and preference labels given the synthetic utilities and normalized drift rates.
These datasets can be parameterized, e.g., by re-scaling utilities and their differences, resulting in different levels and types of noise. The parameters were manually set to values resulting in reasonable noise levels, which are reported in this section.
C.1.3 Dataset analysis
To characterize the synthetic datasets, we evaluated the noise levels affecting both ordinal preferences and response time signals (Figures 5 and 6). The confusion matrices (Figure 5) confirm the expected ordinal preference accuracy: perfect accuracy for the deterministic dataset, and approximately 8%
19

---

Dececmboistie
Storbwate
Drih-- Dil---iioo
Tropannc
(a) No variability Stochustic
Detecministic
Drif-DiEEuio
120
Fvepuaar
Kendall Correlation
(b) With variability
Figure Histograms the Kendall-tau correlation between the strength ranking implied by response times and the ground-trute strength ranking on synthetic datasets.
symmetric error rates for both the Stochastic and DDM datasets due to their probabilistic choice mechanisms.
Figure assesses the fidelity of response time as an indicator of preference strength. It displays the distribution of Kendall's tau correlations (across 100 datasets) between the ranking derived from simulated RTs and the ranking derived from true utility differences. Kendall's tau is used here as quantifies the correlation of ranks (the order), which the specific information ResponseRank aims to leverage from RTs. The results highlight several points: (1) Under ideal (deterministic, no variability) conditions, the RT ranking perfectly matches the true strength ranking (by design). (2) Introducing inter-stratum variability weakens this correlation, demonstrating the challenge of using raw RTs globally, assuming such variability is present in real-world datasets. (3) The inherent noise in the Stochastic and DDM generation processes further reduces the correlation, with the Stochastic process preserving the rank order more reliably than the DDM. (4) Inter-stratum variability consistently degrades the RT-strength signal across dataset types.
C.1.4 Assumptions
Our method makes minimal assumptions about human behavior. requires only that (1) preferences approximately follow some utility function (standard Bradlcy-Terry assumption) and (2) we can approximately rank preference groups by strength. The only assumption about response times is an inverse monotonic relationship with preference strength. Our synthetic data, while far from perfectly realistic, reflects these properties: preferences generated from utility functions (extensively validated through BT model usage) and response times following inverse monotonic relationships (supported by psychology literature [7]). Going beyond this, one of our settings follows the drift-diffusion model, which has been extensively validated in psychological research [33], providing an approximation of response time behavior that goes beyond the monotonic relationship. We choose base utility functions randomly, which while not aligned with specific human preferences, reasonable since the model should learn any utility function matching two key assumptions.
20

---

Determinietic no warishilitt
Sochaetic no warabillity
Drin dimision to variability
67.4%
10.14
7%
O8.1 35
Training Set Slae (34)
Training Sut Sin (9))
Teatning Sot Stoo (%)
Figure 7: Synthetic results without RT variability across strata. Comparison of PDC (top) and accuracy (bottom) on Deterministic, Drift-Diffusion (DDM), and Stochastic datasets as a function of dataset size (100 runs; shading indicates 95% CI assuming normality). Here ablate the effect of inter-stratum variability by omitting stratum-RT multipliers. BT baseline performance with the full dataset is a dashed red line; ResponseRank's interpolated breakeven point is marked by an arrow. Higher PDC indicates better learned preference strength (utility distances); higher accuracy indicates better ordinal preference predictions.
C.2 Training
We rum 100 trials per condition. Each trial uses independently generated dataset consisting of 50 training examples, 200 test examples, and 20 features per example. We optimize a small feed-forward neural network with single hidden layer of size 64 and ReLU activation. Due the small dataset size, we use full-batch gradient descent, processing all training examples each update. Each model trained for 200 gradient steps (AdamW optimizer, no early stopping, learning rate 0.001, weight decay 0.01). This avoids batch packing considerations described in Appendix B.2, which apply only larger-scale settings. We optimize the standard cross-entropy loss for the BT baseline, MSE for RtRegression, and negative log- likeeiioo for ResponseRank. Each loss is normalized by dividing by the batch size (i.e., the full training set size).
C,3 Additional results
Here we present additional experiments supporting the ones in the main body as well as additional detail and visualizations for the main experiments.
Results without inter-stratum RT variability. In the main experiments, we multiply response times in each stratum with stratum-specific random multiplier to simulate effects such individual annotator speeds. Here we ablate this choice by presenting results without this variability multiplier. This simulates setting where all annotators display the exact same response time patterns. We include these results to demonstrate RtRegression's 's behavior under idealized conditions and the cost of unnecessary stratification on ResponseRank.
Figure 7 shows that, when inter-stratum RT variability is absent, stratification offers no benefit compared to an ResponseRank variant using single global stratum (ResponseRank--Pon Pool). This is expected, as strata have no difference in this setting and smaller rankings offer less grounding. Both variants continue to outperform the BT baseline.
21

---

Determanistic
Drits dirusion
Stochastic
0.2
8
108
400
Figure 8: Performance comparison on synthetic datasets when inter-stratum RT variability is introduced. Boxplots show PDC (top) and accuracy (bottom) across 100 runs for deterministic, drift- diffusion, and stochastic data. This condition simulates diverse annotator speeds or task complexities affecting RT' across different strata. Higher PDC indicates better learned preference strength (utility distances); higher accuracy indicates better ordinal preference predictions.
Table 3: PDC performance across datasets with response time variability over 100 trials. Format: mean + std [= 95% CI half-width). Best values per column are bolded.
Datzser/ Learner
50%
80%
70%
80%
90%
100%
Deterministi
DT
0.34.10.11 0.15
0.00)
0.39 0.14
0.03]
0.41 1 0.13
0.03]
0.48 ( 0.02)
0.51 1 0.11
0.021
0.53 1. 0 0.02]
RiRegresiion
2:132:00.17
033
0.36
I138 18 0.24]
0.37 0.17
1 0.51
0.20 0.17 0301
1122 2 0.38
D04
0.28 2: 0.19
[a 0.14
RiRejression Pemn
0.08 2:112
0.21
0.0 1112
0.021
0.00 00022013
1 0.21
0.10
0.32
1111 2 0.35
003
0.12 2: 0.14 [1: 0.13]
XespotseKank:
0.47:10.11 0.14
0.03
0.54
1..1411.11
0.031
0.58 0.14
11 0.01
0.02 0.13 0:30
0.65 0.13
0031
0.67 1: 0.12
0.021
KesponseRank Pool
0.30
0.17
0.00]
0.32
1.15 11: 0.04]
0.34 0.31 0.18
0.04]
0.37
1 11
0.50 0.15
0.0
0.42 0.10 0.04]
RoguusseRank-Pem
6:21
0.14
(= 0303
0122
11.33
0.03
0.20 0.16
0.05]
0.21
0.133
0.26
D03
0.27 0.14 [ 0.133
Drift diffusion
BT
0:35 0.13
0.00)
0.58
0.12 0.02]
0.42 0.11
0.03]
0.41 1 0.15 0.00)
0.47 0.13
0.00
0.50 1 0.13 0.03]
XiRegresiio
6.26 0.10
0303
1127
[1.55
18 0.001
0.21 0.16
0.51
0.33 a 0.14
0:01
35 0.35
0.37 4 0.16
la 0.13
RiRegression Penn
6.16 0.12
0.21
1120
I1.4
0.001
0.21 2 0.11
0.51
0.22 0.14
0.0
11
0.14
0.25 2: 0.15 la: 0.13)
KespotseKank:
0.38
0.18 14
0.03
0.42
[1.3 [á 0.031
0.47 2 0.13
0.05
0.52 a 0.12
11
0.53
0.10
003
0.56 1: 0.12
0.021
RogumscHank-Rol
6.27 0.14
003)
0.26
(1.1 0.03
0.28 = 0.16
0.03
0.51 + 0.16 0.00
0.52 = 0.15
008
0.33 0. (+ 0.13]
RoguissHanik-eem
0:21::00.11 C303
(1.2
[1.1
0.03
0.23 0.16
0.0
0.25 2
0.25
008
0.23 0.15 (+ 0.133
Stochastic
BT
6.36-2-0.13 (= CA3)
(14)
[1.1
0.031
0.41 0.0]
0.69 6.12
0.50
002)
0.53 0.12 0.121
RiRegresion
2:132:00.13
0333
I136 I115 0.001
0.182:00
[a: ousl
0.21 a 0.17 0:00
I1.23 2 0.38
0.24 0.10 la 0.14
RiRegression Penn
0.09 (019--00.12 0.12
0322
I108 I1133
0.221
0.112::11
[a 0.51
0.11 a 0.14
0.01
11122 2 0.15
0031
0.13 0.14 [1 0.13
ResputseKank
0.48 0.11 0.14
0.03
0.54 I1.3 11:311.00 0.001
0.59 0.9-:012
0.02
0.63 1 0.10 0.32]
0.65 2 0.1
0021
0.68 0.00 [1 0.021
RogunscRank-PPol
6.20 1:00--000 0.16
CA3)
015) 1.16 0.03)
0.33 0:16
0.55
0.96 0.177 0.01
0.57
008
0.38 1. + 0.13]
RoganscHani--een
6.23 6.25+:0..14 0.14
C.03]
(1.22 [1.33 0.03
0.26 = 013 0.5
0.27 = 0.3)
1133 0.15
003)
0.26 03 0.13]
22

---

Table 4: Accuracy performance across datasets with response time variability over 100 trials. Format:
mean + std [ 95% CI half-width). Best values per column are bolded.
Dataset Learner
50%
60%
70%
80%
9076
100%
Deterministic
BT
70.8 14:
71.3 5.0%
1.0
742 4.4%
0.91
70.7 4: 4.4%
12 0.91
77.4 4.%
081
77.7
4.25
0.5
RrRegresien
60.4+
61.1 01.++90 9.0%
[+ 2.00
02.2 0.7%
1:9
62.8 10.0%
2.1
63.3
11 050
2.3]
04:24 11.08
2.2]
#RRegecion-Pomm
5...+...%%
2.0)
56.4 9.0%
1.8]
65.2+ 10.6%
2.3]
56.0
10.1%
2.0
56.2
11 05
2.3
66.5+
10.95
2.3]
RogunscRakk
76.2+-0.5%
1.0
77.5 188
t= 1.0
78.9 1,0%
1.00
80.5 1.0%
[.9
61.5
1
[+0
82.1 4.35
0.91
JoguuscRank-Pon
69.5
71. 7.0%
1.4]
71. 7.0%
7
1.4]
72.64 7.4%
1.5
7331 7.35
1.4]
RosuussRank-Poor
6.19.16..%
67 5.85
(-
07.9
66.2 5.0%
1.0
5.0% 68.0
5.45 09.0
1.1]
Drift diffusion
B7
70.94 14: 110
72.2 4.0%
0.9
736 5.0%
1.01
74B 4.4%
12:
70.6
4.4%
14
0.9
76. 4.7%
0.91
RrRegresion
07.1++6.07
68.0 5.85
1.21
08.8 0.15
1.2]
00.5 6.0%
1.2
70.0 5
1+
T1.4 5.95
1.2]
#RRegeciin-Pemm
63.7++6.9%
1.3)
66.5
6.2%
64. 0.0%
1.3
10.5++6.18
1.2
66.5 5.7%
07.1 5.88
1.3]
RogurscRakk
7..2
73.5
1.0%
74.8 4.8%
1.0)
76.1 +.7%%
0.9)
76.9 1.3%
0.9
77.7 4.5%
0.91
RogunacRank-PWa
67.8.t-0.0%
08.1 06.1+:00 6.1%
1.2]
07.9 6.3%
1.3]
68.9 0.8%
1.11
08.1 5.0%
00. 0.85
1=3
RospunsRankk-eee Penn
06.3:t.5.07%
00.5
062
67.8 5.7%%
1.1)
67.4
07.2 5.8%
1.2]
Stochastic
BT
71.64:5.4%
1.11
71.0 5.4%
730 1.2%
1.01
70.3
4.7%
0.91
70.84 44%
0.1
70.0
4.2%
0.5
RrRegresiin
60:3++9:11%
61.3 9.55
1.8)
02.6 8.7%
1.7]
05.7 9.0%
[+ 188
61.1 +
9
04. 0.05
1.9
Rr #iRegrocionPoom
54.3++9.55%
51.8 9.8
56.5 10.05
2.0
56.9
10.4%
57.
2.]
57.9 11
2.3
Rogu
74.9 +.0---..77 [+009]
76.8 1.3%
0.
78.3 1.0%
0.8]
79.1 +.3.%
80.0 3
(+07)
80.8 38%
0.7
RoqunseRank-Pon
67.9.2-6.2%
08.8 2 6.2%
09.8 1.%
1.3]
70. 2 6.0%
1.3)
70.7 6.6
71.0 6.25
1.2
HoopossoRankPeeee
Penn
6.39.1.6.5
00.5 5.05
067 1.1
1.0)
67.9
1.0
67.3
67.3 5
1.2]
Determiniatic varlability
Dritt diffasion no variability
Stochaste variability
07
011
400
x/6
255
36
606
415
Figure 9: Performance comparison on synthetic datasets when inter-stratum RT variability is absent. Boxplots show PDC (top) and accuracy (bottom) across 100 runs for deterministic, drift- diffusion, and stochastic data. this condition, all strata have identical RT patterns (no inter-stratum variability). Higher PDC indicates better learned preference strength (utility distances); higher accuracy indicates better ordinal preference predictions.
23

---

Table 5: PDC performance across datasets without response time variability. Values reported as
mean + standard deviation over 100 trials.
Datiser Learrer
30%
60%
70%
N0%
50%
1001S
Deterministic no variabiiity
B1
0.36 2: 0.14
0,03]
11.00
11:34
0.031
0.44 0:14
[4: 0.051
0.86 2 0.14
0.31
0.50 = 0.34
11 0003
0.52 = 0.18
0.131
RuRegression
0.54 1: 0.15
0.033
0.50
0.13
0.031
0.63 a
11 0.051
0.06 1 0.12
0.02
0.69 2 0.32
12 0002
0.70 1: 0.12
0.022
RdRegroscon-Pom
0.15
0.05]
0.26
0.34
[+
0.08
0:30=0.11
+ 0.55
0.32 0.15
0.33
0.35 0.34
003
0.10 + 0.15
0.033
Respirseaank
0.48++0.14
0.05
0.54
0.34
0.08
0.58 0.18
+ 0.51
0.61
+ 0.15
0.31
0.56 + 0.32
802
0.66 0
0.112]
Acspntsekank-Pod
0.49 0.15
0.55
0.54
11.35 0.00
0.58 = 0.14
+ 0.51
0.61
+ 2.0.4
0.33
0.58 = 0.38
8003
0.65 = 0.18
0.113
Resporseaank- Perm
0.25 0.15
0.031
0.25
11.34
12 0.031
0.21
0:14
[ 0.051
0.25 0.16
0.33
0.23 2 0.35
8003
0.30
2: 0.14
0.113]
Drift difftssoon uo variability
BT
0.31 0.144
0,03]
0.36 12
0.03
0
[4 0.51
0.44 2.4.20000 2.11
0.47 = 0.11
1= 0021
0.50 a: 0.12 0.112
RtRegression
0.36 0.14
0.033
0.39
0134
0.031
0:22 0:14
[ 0051
0.46 a 0.12
0.00
0.48 a 0.12
002
0.50 a: 0.11
0.022
RIR RiRegroscinnPem
0.24 0..14 0.14
0.05)
0.26 0.34
0.08
0:32-011 + 0.5]
0.55-+0.1151
0.00
0.35 + 0.35
004
0.38 0:
0.03
Resporseaank
0.87 0.11
[= 0.05
0.42 0.34
++ 0.08
0.47 = 0.10
+ 0.5
0.50 0.10 0.3
0.54 + 0.32
802
0.56 0
0.12
Roggrtsefank-Pot Pool
0.35 0.15
== 0.05]
0.30
11.35
[- 0.08
0.48 = 0.16
+ 0.5)
0.86 0.14
0.28
0.48 = 0.35
== 0088
0.50 + 0.14
RegponsoRank-Pen Penn
0:20 0.20-+-0:15
[+ 0.03)
0.22
0 0.13 0.03]
0.25 - 0.14
[ 0.05]
0.22-+00.14 0.003
0.20 + 0.14
[- 0033
0.26 + 0.15 0.03)
Stcchstic no varishility
BT
0.82
0.03]
0137 = 0.33
0.33
0.41 = 0.1 [ aus]
0.47 = 0.32 0021
0.50 a: 0.11 [# 0.121
RV RiRegrossion
0.588 1.68++0.115
0.055
0.59 + 0.131
0.3
0.62 0.12
[+ 0.02
0.05 101-+00.12++00 0.02]
0.67 + 0.11
[+ 0.2
0.69 0 0.02]
RIR
RiRegroscinn-oom
6.21 +.11
0.05
0.25 0.34
0.00
0.2 -013
+ 0.0
0.51 +00.1 0.20]
0.32 + 0.34
809
0.37 0 0.03
RospisseHank
0.466+00.16
[ 0.05]
0.52 0.34
0.08
0.57 011
+ 0.0
0.61 + +.1.2 0.22
0.5 = 0.11
802
0.66 + 0.10 0.122
Rogontsefani-Paa Pool
0.00
0.15 0.05]
0.55
0..38 0.08
0.58 = 0.18
+ 0.85
0.61 + 0.12 0.22
0.5 = 0.32
== 002)
0.64 + 0.12 0.ll
RegponsoRank- Penn
0.22 9: 0.15 0.033
0.23
0.14 ( 0.03]
0.25 120000000
[+ 0.05]
0.21 0.14 0.003
0.31 + 0.14
[- 0.03
0.28 + 0 [+ 0.03)
Table 6: Accuracy performance across datasets without response time variability, Values are reported
mean + standard deviation over 100 trials.
Dutreet/ Leamer
50%
60%
100%
Dederminitti variahility
71.5
5.4%
5.25
1.0]
ReRegression
70:3
60.6
81.7
0.8]
RiRegression-Per Perm
06.5
68.1 5.7%
68.7
700
1.21
1.31
ResponseRakk
75.1
5.0%
5.5%
79.9
81.5
0.9
RespomeRink-Paal
76.1 4.0%5
776 6.6%
79.1 5.05
79.7 6.1%
1.0
1.00
81.2 4.7%
0:91
RospomsRinl-Pan
67.3 5.695
1.1]
5.0% 07.5
66.5 6.2%
1.2)
07.9 6.8%%
1.3)
6:256
1.2)
00:3 6.0%
1.31
Drift diffusion no variability
BT
60.8 5.85
1.13
71.7
1.33%
1.00
73.0
5.0%
1.0)
74.3 4.85L
1.0)
7 4.55%
0.0
75.8
4.3%
0.9
ReRegression
71.3
5.35
1.13
72.9 5.181.515
73.0
1.0
75.0 5.5
1.0)
70.0
1.0
76.3
4.7%
0.9
Rikegression Pern
07.4 5.295
1.0
68.8 5.6%
0.9
71.1 4: 5.38%
1.01
71.4 2
+: 1.00
72:3
4.8%
12: 1.01
ResponseRank
71.6 6.6%%
1.3)
73.7 a. 5.0%
1.21
75.0
5.7%
1.11
76.3 1. 5.25
1.00
77.4
4.0%
11 1.00
78.0
4.5%
0.9
RespomeRtnn-Puoll
70.9
6.255
++ 1.2)
72:2 + 0%
73.0 5.7%
1.1)
74.3 + 6.2%
1.00
76.5
+ 1.00
76.0
5.0%
[+ 1.00
Respomeeeak-Peem
65.1 = 6.355
1.3)
96.0 + 5.0%
1.3]
6.1% 66.5
++ 1.2]
0AL9 + 5.8%%
1.00
67.7 5.65
(+ 1.1)
5.4% 08.4
1.31
Stochastie no variability
BT
70.111550PP 5.005
[1: 1.0)
11.1 1.44%
E1 L.I
72.7
49%
[1 1.0]
73.9 1 4.8%
L 1.00
75.1 4.35
0.008
76.0 4.3%
0.90
ReRegrassion
T6.4 4.5%
[t:09)
77.81-47%
699
59.
[1 0.8
80.5 1 4.18
[ 0.8)
81.1
1
1.25
[.006)
81.8
4.0%
[+:0.]]
RiRegression Perm
65.1 6.7%
12 1.3)
65.1 6.3%
121
68.3
5.1%
L1 1.0
69.3 1 5.7%
1.31
71.3
5.0%
1.0
T1.4 5.3%
1.0
ResponseRank
74.1 4.8%%
[1
76.0 1 4.8%
1.0
77.8
4.5%
[1 0.9
79.1 4.2%
L 0.8
80.0
4.3%
1.0
50.7 4.0%
[1 0.55
RespomeeRnn-PPool
70.4= 4.6%5
0.)
77.1 + 1.3%
0.9
76.2
[ 0.91
7922 + 4.3%
0.99
80.0
4.555
++ 0.9)
90.6
4.2%
[+ 0.55
ResponeRaal-Peem
66.3 0.255
1.0)
66.5 5.3%
1.0
66.8
5.0%
1.11
07.1 2 5.4%
1.31
67.0
0
1.2)
08.3 0.3%
1.31
We further find that in this setting, where RtRegression assumes a perfectly accurate model of the strength-RT relationship in the deterministic and stochastic datasets, it performs comparably to ResponseRank. This is to be expected, but not reflective of realistic settings.
We further include a boxplot of performances with the full dataset (Figures 8 and 9) and numerical results for the experiments presented in the main body (Tables 3 to 6). Boxplots (Figures 8 and 9), visualize median (center linc), interquartile ranges (IQR; boxes), 95% confidence intervals of the median (notches), and outliers (dots).
D Supplemental details on MultiPref experiments
In this section we share additional results and details our experiments with the MultiPref dataset, initially discussed in Section 5.
D.1 Data processing
MultiPref is dataset of 10,461 instances (5,323 unique prompts), each consisting of a prompt and two completions [15]. For each instance, four distinct preference annotations are collected: two from expert annotators (defined as having a graduate degree in the relevant domain) and two from normal crowdworkers, drawn from a pool of 227 total annotators. Each annotation includes extensive metadata: sub-preferences for helpfulness, truthfulness, and harmlessness; explicit preference strength (clear vs. slight); and time spent annotating.
24

---

Filtering and splits. We preprocess the dataset by filtering samples where one of the compared texts exceeds the maximum sequence length of 1,024 tokens resulting final dataset of 9,846 samples. We shuffle this filtered dataset and then split 2,000 samples off as test set, resulting in distinct splits for different random seeds. We report all results across 30 random seeds.
Annotation sampling. We randomly sample one of the four annotations per comparison to simu- late typical RLHF data collection for the training set. For evaluation, we use MultiPref's official human_ caaao_overllll binarized oiirrellbinnarizee labels, which aggregates annotations by majority vote. Tied prefer- ences are ignored for both training and evaluation, resulting in smaller effective train and test set size varying slightly depending on the sampled annotations determined by the seed (26.32% of annotations the full length- length-filtered filtered dataset are ties).
Fractional training sizes. To simulate different training set sizes (reported as fractions in the line charts), we select random subset of the training set, keeping the test set unchanged. This does not affect results reported in bar charts, which all use the full training set (fraction 1.0). Uniformly sampling individual comparisons could result in unrealistic annotation patterns with very few or even only one comparison per annotator. This would undermine stratification by annotator identity (Appendix D.2), which relies on sufficient within-annotator data. We therefore perform annotator- aware sampling: after annotation sampling, we group comparisons by annotator, shuffle these groups, and sequentially include groups until reaching the target fraction. All but the last group are included completely.
D.2 Strength signals and stratification
Stratification groups comparisons into homogeneous subsets where the relationship between the strength signal and preference strength is locally valid, mitigating confounds such as individual annotator speed and task complexity. This is analogous to blocking in experimental design [34), where units are grouped to reduce unexplained variance. We consider three distinct strength signals from the MultiPref dataset:
Response time. We derive strength from the total time spent annotating each comparison, assuming shorter times indicate stronger preferences. Since annotation time may depend on individual annotator speed and reading time, we stratify by annotator identity and text length. We first discretize length by grouping instances into 8 global length buckets. Strata then formed by the Cartesian product of annotator identity and length bucket. After stratification, we form rankings of size < 2 within each stratum using the partitioning strategy (Appendix B. minimize the effect of noisy response times. Some resulting strata may have size one (2.0% of comparisons on average); those reduce to Bradley-Terry (Theorem 1). We refer to this variant as RR-RT. As discussed in Section 5, the response time signal captures not only the preference decision time, but also time spent on additional annotations and justifications, likely confounding its relationship with preference strength.
Stated strength. We use the explicit binary preference strength annotation (slight vs. clear) as the strength signal. Since this produces only two distinct strength values, most rankings are limited to size 2 (pairs of preferences with different stated strengths). As stated-strength judgments may be annotator-specine
stratify by annotator identity before forming rankings. Within each stratum, we pair comparisons with different stated strengths using the partitioning strategy (Appendix B.1) As strengths are imbalanced (more 'slight" than 'clear'), ) many comparisons cannot be paired and remain singletons (33.1% of comparisons), which reduce to Bradley-Terry (Theorem 1). We refer to this variant as RR-Stated.
Inter-annotator agreement. We compute weighted inter-annotator agreement as our primary strength signal. Each of the four annotations per instance indicates either a clear or slight preference for one of the two responses. Given N annotations in total, including n, ties, nē strong and n." slight preferences for the chosen response, and n. clear and n, slight preferences for the rejected response, we compute the agreement as
Agreement (1.0- (n n. + 0.5 (n. n. /NN
We do not use any stratification for inter-annotator agreement. As MultiPref has = 4 annotations per instance, the agreement score can only take a few discrete values, however, requiring tie splitling through the partitioning strategy (Appendix 1. This results rankings of size 4.58 on average. We refer to this variant as RR-Agree.
25

---

Table 7: Hyperparameters and training configuration for MultiPref experiments. We use the same hyperparameters for BT and all RR variants.
Hyperparameter
Value
Learning Rate
15c-6
LR scheduler
Linear with 0.05 warmup ratio
Gradient Clipping
1.0
Weight Decay
0.1
AdamW W optimizer settings (31, $2, E)
0.999, lc-08)
On-Device Batch Size
16
Gradient Accumulation Steps
4
Max Sequence Length (tokens)
1024
Dropout
Precision
bfloat16
Epochs
D.3 Detailed experimental setup
[35] with single classification head that extracts scalar reward prediction from the last hidden state, fine-tuned full using both BT and ResponseRank losses (negative log- loglliklihhood. For both losses, we normalize by the batch size (number of comparisons). We use equal hyperparameters (Table 7) for both methods, supported by the hyperparameter sensitivity analysis in Appendix D.8.
Model and hyperparameters. Reward models are initialized from 1lama-3.1-88
Evaluation. We evaluate reward models on both in-distribution and out-of-distribution test data. The in-distribution test data is held-out portion of the MultiPref dataset (2,000 samples) using aggre- gated labels overall binarized), providing an in-distribution evaluation with respect to prompts and responses. For outoo-ddistribution evaluation, we primarily rely on RewardBench 2 as it correlates strongly with downstream performance [16]. In Tables 8 to 10, additionally report results for the original RewardBench benchmark [36]. As we did for the MultiPref dataset, we filter RewardBench for samples where both responses within the 1024 token limit.
D.4 Details main experiments
Performance vs. training set size. Matching the synthetic experiments Section 4, we investigate performance function of training set size to determine to which degree the best-per best-performing strength-aware methods (primarily RR-Agree, with RR-Stated shown for comparison) improve sample efficiency. Figure 10 shows that RR-Agree outperforms BT by 3.0 percentage points RewardBench 2. Additionally, RR-Agree matches final BT performance on that benchmark with approximately 30% fewer preference pairs, demonstrating its high potential for data-efficient preference learning.
Numerical results. To support reproducibility, we report numerical results for experiments the main text as well as the training data size experiments in Tables 8 to 10. We also report results for the original RewardBench benchmark (version 1) for additional validation. Note that reported confidence intervals assume normality and are not corrected for multiple comparisons.
RewardBench category breakdown. The RewardBench 2 benehmark consists of seven distinct categories, each measuring different aspects of reward model performance [ [16]. Figure 11 shows the performance of all the variants studied in the main text broken down by category as a function of training set size. It shows that variants that make use of an effective strength signal (agreement and, to a lesser degree, stated strength) outperform the BT baseline primarily in the Safety, Ties, and Math categories while the Focus category is most impacted by BT training duration (2 vs. 3 epochs). Performance increases on the particularly difficult Ties category highlight that agreement signals can help learn calterecal reward models.
26

---

BT (baseline)
RR-Stated
RR-Agree (ours)
MultiPree Test Accuracy
RewardBench Score
67.7%
+4.6
+3.0
70
50
60
40
50
30
10
20
30
40
50
60
70
80
90
100
10
20
30
40
50
60
70
80
90
100
Training Set Size (%)
Figure 10: Performance vs. training set size. As function of training set size (percent of entire training set), we show: MultiPref test set accuracy (in iin-iitributtion). RewardBench score (out-of- distribution). 30 runs; shading indicates 95% CI assuming normality.
Table MultiPref test set accuracy. Format: mean column are bolded.
[=t: 95% CI half-width), Best values per
Learnr
10%
20%
30%
40%
50%
[ 0.8]
[+ 0.7]
71.6
[+
1.7% 0.6]
71.2
BT@2
71.8 1.7%
1= 0.6]
73.5 1.6% [+ 0.6
73.6 1.7%
[= 0.6]
73.6 [ 0.6
74.2 16%
0.6
RR-Random
RR RT
1.7%
[ 0.6]
71.9 1.9%
[1 0.7]
1 0.6]
72.7
1.9%
(1 0.7]
72.5 1.6%
0.6
RR RR-Stated
72.141.755
0.6]
73.4
2.1%
[+ 088
73.9 1.5%
++ 0.6]
73.9
1.5%
[+ 0.5]
74.3 1.4%
0.5
RR-Agree
73.2 1.5%
74.5
1.3%
75.0 1.3%
0.5]
74.9
111%
75-4
13%
0.5]
Leamnr
60%
70%
80%
90%
100%
BT
71.2+1.7%
[= 0.6]
71.0 1.4%
[+ 0.5]
71.0 ++ 0.4]
71.1 + 1.4%
[+ 0.5]
70.9 +
1.4%
[+ 0.51
BT@2
74.3 1.2%
742 1.3% +
74.5 1.3
[=
74. 1.1%
71.2 +
13%
0.5]
RR-Random
72.2
13%
0.5
72.5
[= 0.6]
72.6
[1
72.1 [1
72.1
72.4
RR. Stated
74.4+145%
[+ 0.5]
73.8
1.6%
[+ 0.6
73.9 1.5%
1+ 0.6]
73.9 0.4]
73.9 +
1.4%
1+ 0.5]
RR Agree
75.3 1.3%
0.5)
75.4 1.3%
0.5]
75.0 1.0%
[1 0.4)
75.3 1.3% [1 0.5]
75.4
1.2%
(1: 0.5]
27

---

Table 9: RewardBench version score. Format: mean + std [ 95% CI half-widthh. Best values per
column are bolded.
Leamer
10%
20%
30%
40%
50%
BT
66.2 5.1%
[= 1.9
09.1 = 5.3%
+ 2.0]
70.4 + 1.9
7288 [ 1.5]
71.0 [ 1.5]
BT@2
61.9 4.5% [= 1.7]
61.
5.2%
+ 1.9
09.0 4.9% = 1.8]
6.01 [ 2.2)
71.9 4.8%
1.8]
RR Randon
RR RT
61.1 61.1+44.5%
[+ 1.6]
64.1
4.5%
[+ 1.7]
4.77+
1+ 1.7]
6.26 [+ 1.6]
71.2+-5.9
1+ 1.9
RR-Stated
64.0 5.3%
2.0
68.3 4.8%
+ 188
70.8 4.6%
4.1%
3.8%
1.4]
RR Agree
62.4 +: 5.1%
[+ 1.9]
69.5 4.5%
( 1.7]
71.3 5.0%
(+ 1.9
74.4 4.6% (+ 1.7]
77.0 2.8%
1.0)
Learnrr
60%
80%
90%
100%
BT
3.0% [= 1.11 53.5
76.3 [+ 1.1]
3.2% [=
3.29% [
78.4 + 2.3
0.91
BT@2
[= 2.0]
74.3 1.3% [ 1.7)
3.15% [= 1.2]
3.7% [ 1.4)
77.8 1 3.7
1.4]
Random
76.3
2.9%
111
RR-RT
7.54 77.0
[= 1.31
72.1 1.1%
( 1.5)
33.43.66
1 1.3]
72.8 128441%
+ 1.5]
710 5.2%
1.91
RR Stated
1.2]
75.6 3.1% 1.2
7
[ 0.8]
78.3
2.5%
[1 0.9)
79.2
2.2%
0.8]
RR ARrgge
78.0 2.6%
1.0]
79.1
2.3%
78.6
2.8%
79.2 1.9%
80.0
2.0%
0.7]
Table 10: RewardBench 2 score. Format: mean + std [ 95% CI balf-widthh. Best values per column are bolded.
Learner
10%
20%
30%:
40%
50%
BT
33.2 + 4.8% [+ 1.8]
37.7 + 5.4%
[+ 2.0]
40.8 4.7%
[+ 1.8]
43.4 + 4.6% [+ 1.7]
46.2 + 5.4%
[+ 2.0
BT@2
29.9 3.6% [= 1.3]
32.9
5.3%
+ 2.0]
36.8 4.7% [= 1.7]
6.0% 2.2]
41.6 5.2%
1.9]
RR- RR-Random
RR-RT
2..6+ [= 1.3]
32.0 4.3%
[+ 1.6]
34.7 1 1.8]
35.8 4.1% [+ 1.5]
40.2 4.9%
[+ 1.8]
31.1 RR-Statted
1.5)
36.2
1.2%
16
39.8 4.5%
42.7 48%
1.8
16.2 4.0%
1.5]
RR-Agree
30.0 - 4.2%
[= 1.6]
4.4% 37.7
[ 1.6
41 1
4.9%
[ 18
45.2 5.2%
+ 1.9)
49.5 4 3.2%
1.2)
Learner
BT
48.6+44.5% [+ 1.7]
49.0 3.9%
[+ 1.5]
3.1% |+ 1.2]
52.7 3.1% [+ 1.1]
52.6 4 2.7%
1.0]
BT@2
5.9% [= 2.2]
46.9 = 4.5% [ 1.7)
18.814499 [ 1.6)
49.9 5.7% (1: 2.1]
51.1 5.5%
2.0
Random
4.5%
RR-RT
4.52
[= 1.7]
43.2 4.6% + 1.7]
45.2 ++ 1.5]
140 [+ 1.9]
47.2
5.2%
2.0
RR-Stated
4..1
1.5)
18.7 + 18
42 1.6
518
1.2
53.2 - 3.2%
1.2]
RR-Agree
50.9
1.3)
53.1 3.1%
1.2)
53.3 1 3.3%
1.2
542 3.1%
1.1
55.6
1 2.3%
[ 0.91
D.5
Validation experiments
Here we present additional experiments and ablations validate choices. All experiments report mean performance across seeds and indicate the 95% CI (assuming normality) in the error bars.
Impact of ranking size. As ties are prevalent in agreement scores. tie splitting and batch packing (Appendix B) result in rankings of at most size (average size 4.58). To investigate the effect of ranking size on RR-Agree, we additionally test variants that limit ranking sizes to at most 2, 3, and 4 (resulting in average sizes of 2.00, 2.67, and 4.00 after batch packing'). Figure 12 shows performance increases with ranking size, demonstrating that larger rankings convey more information about preference strength. All variants, even with minimal rankings of size 2, outperform the BT baseline with the jump performance between no strength signal (BT) and small rankings being particularly pronounced.
Agreement signal with directional information. Agreement scores are by design independent of the chosen preference, only reflecting the degree of consensus among annotators. This choice was made simulate realistic settings where only single annotator's choice is available, with agreement serving as proof of concept for strong strength signal. As consequence of this choice, cases may arise where the sampled annotation disagrees with the majority opinion, yet the agreement score thheagereeeee remains high (e.g., when 3 out of 4 annotators prefer the response rejected by the chosen annotation). To assess the impact this design choice, we additionally evaluate a variant using signed agreement as the strength signals, where agreement is multiplied with +1 if the chosen response aligns with the mean opinion (where slight preferences are weighted as 0.5) and -1 otherwise. This is equivalent to using the mean preference the strength signal, and call this variant RR-MeanPref. Figure 13
As 3 does not divide the batch size, this leads to increased fragmentation during batch packing.
28

---

BT (baseline)
BT@2 RR-Stated
RR-Random
RR-RT
RR-Agree (ours)
60
40
20
Safety
Focus
Score
20
Factuality
Math
40
20
Precisc if
Tics
10
20
30
40
50
60
70
80
90
100
10
20
30
40
50
60
70
80
90
100
Training Set Size (%)
Figure 11: RewardBench category breakdown. As a function of training set size (percent of entire training set), we show RewardBench performance for each category composing the benchmark. 30 runs; shading indicates 95% CI assuming normality.
compares RR-Agree with RR-MeanPref. As MeanPref can take more distinct values than Agreement, leads larger average ranking sizes (average 7.03 compared to 4.58 for RR-Agree). We therefore again compare several different configurations with limited ranking sizes for RR- RR-MeenPref (effective sizes: 2.00 for size 4.00 for size 4.80 for size 6). We observe that, while RR-MeanPref slightly increases performance on the in-distribution MultiPref test set, it underperforms RR-Agree on the out-of-distribution RewardBench benchmark, indicating potential benefits of slightly noisy rank signals for generalization. The effects in both directions are small, however, and should not be over-interpreted Note that agreement and McanPref differ rarely as the majority opinion is sampled most of the time. We again observe increasing performance with increasing ranking size, though this effect plateaus at size (the difference between size and full size is not statistically significant: +0.3 pp. p 0.519 on RewardBench, -0.1 pp. p 0.369).
BT with agreement scores as soft labels. As agreement scores are a particularly clean and infor- mative strength signal that gives clear cardinal target, it would be natural to use them directly as soft labels in the BT loss. We therefore evaluate two alternatives: BT-MeanPref. which uses mean opinion as the target preference probability like RR-MeanPref; and BT-Agree, which uses p" 0.5 + Agreement/? 2 the choice probability for the chosen response. BT-MeanPref therefore uses stronger signal than agreement, which is stronger than the binary preference observed by

---

MultiPref Test Accuracy
RewardBench 2 Score
0.8
0.6
0.7
0.5
0.6
0.4
BT
Size
Size
Size
Full Size
BT
Size
Size
Size
Full Size
Ranking
Size
Ranking Size
Figure 12: Impact of ranking size for RR-Agree. Effective partition sizes after batch packing: 2.00
(size 2), 2.67 (size 3), 4.00 (size 4), 4.58 (full size).
MultiPref Test Accuracy
RewardBench 2 Score
0.8
0.6
0.7
0.5
BT
RR- Agrae
RR. MRaMeeePeef-2
RR Meantr-4
RR-MeanPref-6
RR RR-MeamPref (full)
BT
RR: Agroe
RR: PR-MManPeef-2
RR Pran
RR MeanPraf-6
RR-MeanPred (full)
Figure 13: RR-Agree vs. RR-MeanPref ranking comparison. We compare RR-Agree with different
ranking size configurations of RR-MeanPref. The dashed baseline represents the BT model.
standard BT. Figure 14 compares these approaches, showing modest improvement of BT-Agree over BT in both MultiPref test (+0.2 pp. = 0.179) and RewardBench (+0.5 pp. = 0.314). While BT, p 0.037), it under- performs the BT baseline on RewardBench (-0.4 pp. p = 0.694), indicating potential overfitting to the training distribution. All these effects small and not statistically significant. They highlight, however, how difficult soft labels can be to use effectively.
BT-MeanPref leads to further improvement MultiPref test (+0.3 pp
D.6 Robustness analysis
Robustness noisy strength signals. To assess the robustness of RR-Agree noisy preference data, we conducted experiments with varying levels of data corruption using partial shuffle noise. This method randomly selects specified percentage of the preference annotations from the training dataset (prior to stratification) and shuffles their strength values among themselves." 0% represents clean data and 100% represents completely randomized preferences. Figure 15 shows performance of
'While more realistic ranking noise models could be considered, partial shuffle noise provides simple and interpretable way to control the noise level. More realistic alternatives such Mallows would operate
30

---

MultiPref Test Accuracy
RewardBench 2 Score
0.8
0.6
0.7
0.5
0.6
0.4
BT
BT-Agree
BT Pref
RR Agroe
BT
BT-Agree
BT MeanPref
RR Ar-Agge
Figure 14: BT with agreement scores as soft labels. We compare BT baseline, BTT-Agree (BT loss with agreement scores as largets), BT-MeanPref (BT loss with mean preference as targets), and RR-Agree (ResponseRank with agreement-base ranking).
MultiPref Test Accuracy
RewardBench Score
0.8
0.6
0.7
0.6
0.4
0%
25%
50%
75%
100%
0%
25%
%
75%
100%
0%
25%
50%
75%
100%
0%
2
50%
75%
100%
Size
Full Size
Size
Full Size
Figure 15: Robustness to noisy strength signals (RR-Agree). Comparison of two variants of
RR-Agree, one with full ranking size and one limited to size 2, under varying levels of partial shuffle noise.
RR-Agree and the "size 2' variant which uses the same strength signal with smaller rankings across various noise levels. It shows that RR-Agree falls below baseline performance with 50% noise, while the smaller rankings are much more robust to noise. This motivates our choice of ranking size 2 for the response time experiments with assumed noisy strength signals.
further confirm the effect of ranking size on noise sensitivity in Figure 16, where we compare RR-Random configurations (random, uninformative strength information) with rankings of different sizes. We observe that performance degrades sharply with increasing ranking size, demonstrating that larger rankings make the ResponseRank loss more sensitive to noise in the strength signal.
Robustness to missing strength data. To evaluate the robustness of RR-Agree missing rank data, we conducted experiments with different filter fractions applied to the training data. We systematically varied the fraction of ranks retained during training. shown in Figure 17, RR-A RR-Agree performance
rank level (after stratification) instead of on value level, making comparison between different ranking sizes challenging, and have less interpretable parameters.
31

---

MultiPref Test Accuracy
RewardBench 2 Score
0.8
0.6
I
0.6
0.4
Size
Size
Size
Size
Figure 16: Robustness to noisy strength signals (RR-Randone Comparison of RR-Random with different ranking sizes (2 vs 4) using completely random strength signals.
MultiPref Test Accuracy
RewardBench Score
0.8
0.6
I
0.7
0.5
0.6
0.4
+
BT
15
BT
Filter Strength
Filter Strength
Figure 17: Robustness to missing strength data. Filter robustness analysis comparing RR-Agree variants with different filter fractions applied to the training data.
degrades gracefully to BT performance. While very little strength information is sufficient for performance increases the MultiPref test set, performance regresses much faster on RewardBench 2 and reaches baseline performance at around 75% missing data.
D.7 Signal-specific stratification analysis
Impact of stratification on response time signal. As individual annotators are likely have different baseline speeds, we stratify by annotator identity in our response time experiments (RR-RT). Since longer texts naturally require more time read and evaluate, stratification to ensure fair comparisons: examples are grouped by text length into buckets, and response times are only compared within each bucket. Here we ablate both choices. Figure 18 shows results for RR-RT with different levels of length stratification (fewer buckets means coarser stratification). shows very little impact of stratification on RR-RT performance, further supporting that the response time signal is weak in this dataset, independent of stratification choice. This may also question the assumption that longer texts necessarily require more time to evaluate, as annotators may skim skip parts of longer texts. The MultiPref evaluation shows slight benefit of stratification
additionally apply length
32

---

MultiPref Test Accuracy
RewardBench 2 Score
0.8
0.6
0.7
0.5
0.6
0.4
Global
No buckets
buckets
buckets
buckets
16 buckets
32 buckets
Global
No buckets
buckets
buckets
8 buckets
16 buckets
32 buckets
Figure 18: Impact of stratification on response time signal (length buckets). Comparison of
different stratification approaches (no stratification, annotator-oni, annotator 2/4/8/16/32 length buckets) for the RT-based approach. Reference lines show BT baseline (red) and RR Rendam (gray).
(4 bucket is better than annotator-only which is better than no stratification), but this not reflected in RewardBench performance. It is notable that the no- no-stratification condition underperforms the random baseline (RR-Random) on MultiPref test. One possible explanation is that systematically fast annotators provide low-quality annotations (classified as strong by the RT signal) or longer texts (with longer RTs) being systematically classified as weak. We choose length buckets for our main experiments as a compromise between granularity and sufficient data per bucket, but the figure indicates that annotator-only stratification or even no stratification would perform similarly.
To put these empirical results into context, we analyze correlations in the dataset to understand the relationship between response time, preference strength, and text length. Here we treat 'consensus' which has empirically proven a reliable strength signal, as a proxy for the unknown true preference strength. We find that RT strongly correlated with length (within-annotate median p +0.47), supporting the intuition that longer texts take longer to evaluate. The overall correlation between RT and consensus weak (global p = -0.05; negative values indicate that lower RT corresponds to higher strength). Stratifying by annotator improves the correlation slightly (within-annotator median 0.08), demonstrating that controlling for annotator-specific effects helps. However, when additionally stratifying by length (8 buckets), the correlation weakens to p -0.05, showing that length stratification does not help in this datasct. This is because the already- weak RT signal operates largely through text length: Length itself has weak correlation with consensus (within-annotatte annottaoo length-consensus median p = -0.07), indicating that longer texts tend to have weaker preferences. Since RT and length strongly correlated, controlling for length removes part of the predictive power that RT has. This analysis shows that RT is a weak strength signal in this dataset and, when used, annotator-only stratification is preferable. Note, however, that RT variance is not fully explained by length, and datasets with simpler annotation protocols (e.g., forced binary choices with immediate feedback) may exhibit stronger RT-strength correlations.
large part of the difference of performance across different length stratification choices may be explained by changes in ranking structure: Stratification increases singleton fractions from 0.0001 (no stratification) to 0.02 (annotator-only), then progressively to 0.04 (2 buckets), 0.08 (4), 0.14 (8), 0.26 (16), and 0.42 (32).
Impact of stratification on stated strength. We evaluate RR-Stated, an alternative approach that uses explicitly stated preference strength from annotators rather than inter-annotator agreement. We compare two variants: the standard RR-Stated approach with local stratification, RR-Stated (global) which uses global ranking across all examples, and RR-Stated (partialshuffle 100) which applies full shuffling noise to test robustness. As shown Figure 19, the stated strength approach provides useful signal but underperforms compared to the agreement-basee method.
33

---

MultiPref Test Accuracy
RewardBench 2 Score
0.8
0.6
0.7
0.5
0.6
0.4
Stated
Stated- Global
Statedl Shuf
BT
RR
Stated
Stated Global
Stated Shuf
BT
RR
Figure 19: Stated preference strength ablation. Comparison of RR-Stated variants that use explicit annotator-stated preference strength, compared to BT baseline and RR-Agree which uses inter-annotator agreement. The stated strength approach provides signal but underperforms the agreement-basee method.
MultiPref Test Accuracy
RewardBench Score
0.8
0.6
I
I
0.7
0.6
0.4
525
550
S75
BT
525
550
575
BT
Slight Weight
Slight Weight
Figure 20: Weighting of slight preferences. Comparison of Rg-egree variants with varying weight
for slight preferences, showing robustness to different choices.
Weighting of slight preferences. In our main experiments we assign slight preferences weight 0.5 and clear preferences weight 1.0 for the agreement computation. This choice somewhat arbitrary and depends on each annotator's individual calibration of 'slight'. Here we compare two alternative variants where assign slight preferences weight 0.25 and 0.75 respectively. Figure 20 shows that performance varies slightly across these variants, but largely robust to this choice.
D.8 Hyperparameter sensitivity
To evaluate the robustness of both BT and RR-Agree across different training configurations, we conducted hyperparameter sensitivity analysis. We systematically vary learning rate (Figure 21), gradient accumulation steps (Figure 22), gradient clipping (Figure 23), number of training epochs (Figure 24), warmup ratio (Figure 25), and weight decay (Figure 26). The values used our main experiments are bolded. While some hyperparameters have meaningful impact (particularly
34

---

MultiPref Test Accuracy
RewardBench 2 Score
0.8
0.6
0.7
0.5
0.6
0.4
16-05
5e-06
26-05
1e-05
5e-06
26-05
1e-05
Sc- 06
2e-05
Ie-05
5e-06
2e-05
BT variants
RR variants
BT variants
RR variants
Figure 21: Learning rate sensitivity analysis.
MultiPref Test Accuracy
RewardBench Score
0.8
0.6
0.7
0.5
0.6
0.4
BT variants
RR variants
BT variants
RR variants
Figure 22: Gradient accumulation steps sensitivity analysis.
those related to the amount of optimization, e.g., epochs and learning rate), both methods perform reasonably across range of values.
The main hyperparameters using RewardBench performance as feedback. While this could lead to optimistic absolute performance estimates, the scope for such overfitting is limited: several hyperparameters have minimal impact, and the tuning was small-scale exploratory rather than systematic optimization. Identical settings used for BT and all ResponseRank variants, and the sensitivity analysis confirms that these are favorable for both methods, ensuring fair comparison.
selected through exploratory manual tuning during development,
E Supplemental details on RL control experiments
Our experiments show that ResponseRank improves reward model accuracy across synthetic datasets and human preference datasets. In additional series of experiments, investigate the effectiveness of our approach for downstream reinforcement learning performance. We investigate this in a control experiment with synthetic preference labels and preference strength information.
Setup. We are interested in the performance of agents trained with learned reward models based on the ResponseRank loss. We evaluate downstream performance in MuJoCo [37], continuous
35

---

MultiPref Test: Accuracy
RewardBench 2 Score
0.8
0.7
0.5
1.0
BT variants
RR variants
BT variants
RR variants
Figure 23: Gradient clipping sensitivity analysis.
MultiPref Test Accuracy
RewardBench 2 Score
0.8
0.6
0.7
0.5
0.6
0.4
BT variants
RR variants
BT variants
RR variants
Figure Training epochs sensitivity analysis.
MultiPref Test Accuracy
RewardBench 2 Score
0.8
0.6
0.7
0.5
0.6
0.4
0.01
0.05
0.01
0.05
0.1
0.01
0.05
0.1
0.01
0.05
0.1
BT variants
RR variants
BT variants
RR variants
Figure 25: Warmup ratio sensitivity analysis.
36

---

MultiPref Test Accuracy
RewardBench 2 Score
0.8
0.6
0.7
0.5
0.6
0.4
0.001
0.1
0.001
0.1
0.001
0.1
0.001
0.1
BT variants
RR variants
BT variants
RR variants
Figure 26: Weight decay sensitivity analysis.
control task, and highway-env (38), benchmark environment with discrete action space. Both are well-established environments in RI. research. As the RI. training algorithm, we use PPO [39] from the Stable-Baselingg implementation [40]. For downstream RL evaluation, we use the same hyperparameters from Stable Baselines3 Zoo across all experimental conditions.
Generating synthetic preference and strength labels. We follow methodology from Metz et al. [17]. We collect dataset of trajectories based on expert models (trained via standard RL on ground-truth rewards) that generate trajectories sampled across multiple checkpoints, ensuring diverse policy coverage and skill levels for our offline dataset, similar to [41]. Then we randomly sample 5000 segment pairs of length 50 (truncated at episode boundaries) for reward model training. Out of these 5000 pairs, 4000 are used as the reward model training set, and 1000 samples are used as the validation set. We generate synthetic preference and preference-strengti labels from ground truth-rewards To a) for state-action pairs in the environment. Given an annotated trajectory pair (a,, b.) and discounted return (seg) Eien.50 (st, synthesize the preference using
a; > b, + Ra,(a,) 2 (b;) and the strength as (a, b,) = (a.) -
Clearer preference differences correspond to lower simulated response times. Note that since only the order of pseudo RTs matters, negative values are allowed, and any preference strength estimate could substitute for response time. The discount factor is specific to environment and fixed across all training configurations. We then randomly construct rankings of size 16 (matching our on-device batch size, avoiding the need for batch packing) for reward model training using the ResponseRank loss.
(b:). R
Noise To test robustness to reward noise, we also experimented with perturbed rewards. Here, we sample rewards from truncated Gaussian distribution with mean equal to the retrieved environment reward and scale a = & or, where is the noise factor (from tables) and a,, is the dataset reward standard deviation. Truncation ensures samples stay within the observed reward range, following prior work [17], though it's not essential to our approach. Higher noise levels reduce both preference accuracy and strength reliability.
Training As reward models, we use 6-layer MLPs with 256 hidden units, processing concatenated state and action vectors (one-hot encoded for discrete action spaces). We optimize with AdamW [42] with a learning rate of le 5, weight decay enabled, batch size of 16, and early stopping on a validation holdout set (p We train the reward model on the pre-collected data, and do not use further online training of the reward model during RL training.
Reward Model Training In Table 11, we report reward model accuracies, i.e., prediction accuracy for identifying the preferred segment. We find that ResponseRank outperforms the BT-baseline in most configurations but that accuracies only differ slightly. We hypothesize that the improved downstream RI. performance (see Figure 27), due to the learned reward models predicting better shaped reward function compared to the Tradley-Terry bascline.
37

---

Table 11: Validation accuracies of reward models for different environments and noise levels. ResponseRank reward models exhibit higher reward model accuracy, but differences are minor and generally fall within the confidence interval.
Environment noise
RR
BT
HalfCheetah-v3/ noise=0
0.998 0.001 [1] 0.001]
0.995 1: 0.005 [1 0.006]
HalfCheetah--5/ noise=0.1
0.982 1 0.004 [1: 0.006]
0.980
0.004 [1 0.006]
HalfCheetah-v55 noise=0.25
0.932 1 0.004 [1] 0.005]
0.930 1: 0.005 [1 0.007]
HalfChoctah-v5// / noise=0.5
0.864 = 0.003 [ 0.003)
0.857 = 0.011 [= 0.016]
Swimmer-v5 noise=0
0.979 = 0.012 = 0.016]
0.948 + 0.009 [ 0.013)
Swimmer-v5 / noise=0.1
0.920 # 0.018 + 0.026]
0.901 +: 0.020 [= 0.028)
Swimmer-v5/ n0osee-0.55
0.812 + 0.028 + 0.039]
0.799 + 0.027 [+ 0.037]
Swimmer-v5 0.0
0.716 +: 0.033 + 0.046]
0.714 + 0.029 [+: 0.041]
Walker2d-v5 /noise=
1.000 = 0.001 [ 0.001]
0.996 0.002 [ 0.003]
Walker2d-v5 1
0.990 + 0.002 + 0.002]
0.988 + 0.001 [ 0.002]
Walker2d-v5/mappe m
0.955 + 0.008 + 0.011]
0.932 + 0.008 [+ 0.011]
Walker2ddwv5/noise-el
0.884 + 0.005 + 0.007]
0.883 0.004 [+ 0.005]
merge-vO noise=0
0.843 1 0.012 = 0.017]
0.845 1 0.010 [1010011000000000000000000000000 0.014]
merge-vO noise=0.1
0.829 1 0.013 [ 0.018]
0.828 1 0.013 [1 0.018]
merge-vO noisc=0.25
0.798 = 0.020 = 0.028|
0.789 + 0.018 0.025]
merge-v0 noise=0.5
0.714 = 0.010 = 0.014]
0.712 = 0.010
0.013)
Table 12: RL control with reward noise (final reward).
Environment noise
BT
RR
HalfCheettahv5// noise=0
621.0 862.1]
6 433.0]
HaliCheetaah-5/ noise=0.1 I
5014.9 = 595.3 [
4946.7 : 493.5 [ 685.2]
HalfCheetah-v5 noise=0.25
4847.7 567.5 787.9]
5448.2 526.5 (30.9) 730.9]
HalfCheetaah-v/ noise=0.5
4491.9 533.8 [ 741.1]
1890.7 1065.3]
Swimmer-v5 noise=0
21.2 23.8 [ 33.0]
167.7 145.2 [ 201.6]
Swimmer-v5 noise=0.1
32.2 2222 12.9 [= 17.9]
81.8 : 107.1 = 148.7]
Swimmer-v5 noise=0.25
9.8 27.1 + 37.7]
83.4 = 122.8 = 170.4]
Swimmer-v5 / noise=0.5
58.5 51.8 [ 71.9)
81.1 + 128.7 + 178.7]
Walker2d-v5 noise=0
2531.5 = 570.8 [ 792.4]
2363.9 1183.3 + 1642.7]
Walker2d-v5 noise=0.1
2526.2 + 730.5 + 1014.1]
2742.2 + 1164.8 1617.0]
Walker2d-v5 noise=0.25
2978.8 539.8 ( 749.4]
2042.0 + 1295.0 [ 1797.7]
Walker2d-v5 noise=0.5
n.0
2391.6 + 13400.
1636.8 + 758.4 [= 1052.8)
merge-v0/ noise=0
0.3
12.1 0.2 [ 0.3]
merge-v0/ noise=0.1
11.2
12.0 - 0.4 0.6]
merge-v0/ noise=0.25 merge-v0/ noise=0.5
11.4 0.5
12.1 0.6
[1 0.9
11.2 + 0.6 0.8]
12.4 0.3
[1 0.4]
Downstream RL Results Figure 27 shows that ResponseRank generally outperforms BT models. For merge-v0, ResponseRank consistently outperforms despite similar reward model accuracy, suggesting better reward shaping. all environments, the maximum reward achieved during training is higher for the RL agent with access to the ResponseRank reward model. For all but one environment, also the final reward is higher.
Noise robustness. The results above were measured with perfect environment rewards. To measure robustness noise, we also benchmarked with different noise levels (following Metz al. [17]), which perturbs the underlying ground- -trut reward and in turn extracted response times and prefer- ences. Figure 27 (numerical results Tables 12 and 13) shows that ResponseRank outperforms BT-model in low-noise scenarios, and only underperforms when overall RL performance deteriorates significantly. In environments that are generally robust to noise (such as merge-v0 HalfCheetah), ResponseRank also stays stable.
38

---

BT
RR
HalfCheetah-v5
Swimmer-v5
Walker2d-v5
merge-vO
Max Reward
6000
4000
5000
2000
0
4000
0.25
0.5
0.1
0.25
0.5
0
0.1
0.25
0.5
0.1 0.25
0.5
Final Reward
6000
4000
200
5000
2000
0
4000
0.1 0.25
0.5
0.1
0.25
0.5
0 0.1 0.25
0.5
0 0.1 0.25
0.5
Noise
Noise
Noise
Noise
Figure 27: RL control with reward noise. Showing final and maximum reward (mean and 95% CI)
across 5 seeds as a function of noise.
Table 13: RL control with reward noise (max reward).
Environment noise
BT
RR
HalfCheetah-v5 noise=0
685.6 951.8)
942..6
755.4]
HalfCheetah-v5/ noise=0.1
6227.7+230.00
319.3]
5203.7 433.8]
HalfCheetah-v5/ noise=0.25
195144611.66
849.1]
701.3 973.6]
HalfCheetah-v55 /arp:aahh-///oii---///// noise=0.5
588.5 [ 816.9]
4905.3 756.5 1050.2]
Swimmer-v5 /ninae/--////oie=1)))) noise=0
8.2 [+ 11.3]
194.2]
Swimmer-v5 noise=0.1
17.0 [ 23.7]
94.6 100.3 [ 139.2]
Swimmer-v5 noise=0.25
8.4 [ 11.6]
106.0 = 164.6]
Swimmer-v5 mooisee-- noise=0.5
68.6 [ 66.3]
106.2 = 163.5]
Walker2d-v5 noise=0
2961.1 = 582.9]
3741.9 1024.6 1422.3] 3126.1
Walker2d-v5 / noise=0.1
2828.0 1090.1]
1390.3]
Walker2d-v5 noise=0.25
3486.3 5
2831.4 993.7 [ 1379.4]
Walker2d-v5 noise=0.5
2844.1 + 1020.5 [ 1416.6]
1918.2 + 764.5 [ 1061.3]
merge-v0 noise=0
12.1 0.1 [ 0.1]
12.4 [ 0.4]
merge-v0 / noise=0.1
12.2 0.4 0.4 0.6]
12.3 - 0.5]
merge-v0 / noise=0.25
11.9 1 0.5]
12.5 0.5]
merge-vO noise=0.5
12.0 - 0.4 0.4 0.6]
12.6 0.3]
Training Curves In Figure 28 to Figure 31, we report the training curves for the RL agent training with both the BT-baseline and ResponseRank variants. The ResponseRank reward models improve learning performance most cases. However, the results for the Swimmer-v5 environment show very high variance for the ResponseRank reward models. Note that shaded areas in the training curve plots indicate standard deviation, not 95% CI as in most other plots. This indicates training stability and complements the 95% Cls used in Figure 27.
Discussion RL Results Our results provide a strong first indication that the improved reward modeling performance translates to downstream RI. performance. However, we acknowledge that this investigation is limited so far and requires additional experiments, ideally across more scenarios and environments. Larger scale validation in the control domain remains important future work.
39

---

RL Training Curves for HalfChentah-vS:: RT vs Non-RT Comparison
Nobe:
Now
Gobal Soure
X10
Clobal Stops
X
Nole: 0.25
No
14.00
27.0
100
11.00
Guiaa
Stepe
Figure 28: Training curves for HalfCheetah--v5 across different noise levels. Showing mean and std across 5 seeds.
RL Training Curves for Walker2d-v5: RT Non-RT Comparison
Neie:: 0
No me 0.1
30:0
1000
1000
Nobe: 0.25
No 0.5
4000
M
2010
104
10:00
1.3
3.5
3.5
3.5
Gobdl
Figure 29: Training curves for Walker2d-v5 across different noise levels. Showing mean and std across 5 seeds.
F Background on response times and preference learning
This section discusses the foundational models and empirical evidence that support using response times (KTs) signal for preference strength. It also addresses the complexities inherent in this relationship and draws conceptual parallels to learning strength in other contexts.
F1 Foundational support for RT's as preference signals
The premise that RTs offer valuable signals for inferring preference strength is supported by body of thcoretical work, cognitive modeling, and empirical evidence. Humans demonstrably infer others' preference strength from RTs in various contexts, from simple inference tasks [43, 44] to strategic interactions like bargaining games [45].
key theoretical justification is that decision time can reveal latent aspects of preferences beyond the choice outcome itself [46, 47]. Alós-Ferrer et al. [47], for instance, provide formal grounding, demonstrating through analysis of RT distributions that preferences (u(x) 2 u(y)) can be learned with weaker noise assumptions than choice-only data typically require. Their findings indicate that RTs can help recover basic preferences, generalize preference order under symmetric noise assumptions, and even generalize relative choice probabilities under stricter Fechnerian noise assumptions. This
40

---

RL Training Curves for Swimmer-uS: RT vs Non-RT Comparison
Nekic
Vols:
Nav-RT
Newlt
222
Gobal
X30
Gotal Store
X00
Noke:: 0.25
Noise: 0.5
-20
150
-00
-20
Gobal
X30
Gobal
Figure 30: Training curves for Swimmer-v5 across different noise levels. Showing mean and std across 5 seeds.
RL Training Curves for mDrge-vD: RT vs Non-RT Comparison
New
Man Theced
Ulbbal Step
Unbbal Stor
Note: 0.5
MAx
Figure 31: Training curves for merge-v0 across different noise levels. Showing mean and std across 5 seeds.
informational content of RTs is further highlighted by empirical findings showing that RTs often correlate with latent utility differences learned by models trained only on binary comparisons (implicitly learning strength, as discussed Appendix A), suggesting RT's reflect these underlying differences [21] and may help learn them more data-efficiently While ResponseRank does not estimate full RT distributions, this fundamental insight that RTs inherently contain information about underlying utility differences supports their use signal for preference strength.
The drift-diffusion model (DDM) [48, 33] provides a well-supported cognitive model explaining this link. It describes the decision process as a stochastic accumulation of evidence over time until a decision threshold is reached. The DDM posits that stronger preferences lead faster evidence accumulation (higher drift rate) and thus quicker decisions. This inverse relationship is observed empirically; for example, faster responses frequently indicate stronger preferences in tasks like food choice [7], while more cognitive effort (Ionger RT')) is common when decision-maken are close to indifference in decisions under risk [8]. In value-based DDM applications [7], the drift rate u, indicating evidence accumulation speed and direction, is often modeled as linear function of the utility difference between two options, v = ((,, us). Here, u, and u, are item utilities, and /3 is a sensitivity parameter. This model predicts that larger utility differences (stronger preferences) result in higher drift rates and, consequently, faster RT's, providing a theoretical justification for using RTss as proxy for preference strength.
41

---

Moreover, the ability to learn preference strength, as opposed to just ordinal preferences, has been shown enhance model generalization and robustness. From a theoretical point of view, assuming the existence of item utilities allows for generalization to unseen comparisons. Utility differences, often implied by RTs, provide additional structural information about the underlying utility function that aids learning beyond simple preference order. For instance, incorporating RT's to better model utility differences can lead to improved out-of-sample prediction in choice tasks [3] and better performance in preference-based bandit settings [2]. Humans similarly use RT-implied utility differences to infer outcomes of unseen comparisons beyond simple transitivity [43]. Understanding preference strength is also critical for robust value alignment, particularly when choice probabilities near degenerate values (0 or 1), where outcomes become highly sensitive to small perturbations in learned utilities [4]. These benefits underscore the value of methods capable of extracting cardinal preference information.
F.2 Complexities and confounding factors in using RTs
Despite foundational support, the relationship between RT and preference strength is complex and subject to numerous confounding influences. Within the DDM framework itself, observed RTs are affected not only by drift rate (utility difference) but also by parameters like decision threshold (caution), non-decision time (e.g., stimulus encoding). and evidence accumulation starting point. These parameters can vary significantly across individuals (e.g., differing annotator speeds) and tasks.
External contextual influences further complicate the picture. User fatigue, experience and boredom, or variations in financial incentives can impact RTs [8]. Task complexity is a major factor, for instance, RTs can be dominated by the reading time required for lengthy texts or by the inherent difficulty of discriminating between complex stimuli [9]. This overall variability makes absolute RTs globally noisy indicators of preference strength.
The nature of the RT-strength relationship can also be nuanced. Factors like decision difficulty, uncertainty, and cognitive load can mean slower RTs sometimes correlate with stronger deliberation over highly valued but conflicting options [49]. Camphell et al. [49] highlights that RT is linked not only to utility differences but also to error variance and cognitive processing strategies. The use of RTs extends to various contexts for inferring cognitive processes; for example, Montero-Porras al. [50] show that in some settings of the iterated Prisoner's Dilemma, extended deliberation times (slower RTs) are associated with higher cooperation rates, indicating RT's can signal underlying strategy intent - further showing the complex nature of the signal.
Furthermore, the typical inverse RT-strength link may not hold universally. Under certain high- variance experimental conditions, increased deliberation time might not lead to improved (or even worsen) decision accuracy [51]. In strategic settings, individuals might consciously alter their RT' to influence an observer's perception, irrespective of their true preference strength [45]. Even under ideal conditions, the marginal information that RTs provide beyond choice consistency varies with preference strength: RT's are most informative when preferences are strong and choices alone near-degenerate, but contribute less when preferences are weak and choices already vary informatively [2].
The design of ResponseRank is fundamentally motivated by the need to navigate these complexities. By using relative RTs within carefully constructed, homogeneous strata (c.g., per-annotator per- session), ResponseRank aims to control for many of these DDM-internal and external confounders. The assumption is that within such local contexts, confounding influences are more likely to be constant or vary less, allowing relative RT differences to more reliably reflect preference strength. This approach enables ResponseRank to learn utility differences robustly without explicitly modeling the full DDM or the exact, complex relationship between absolute RTs and preference strength across diverse settings and individuals. Moreover, Gates al. [43] suggest that relative utility differences can often be inferred from RTs without needing strict assumptions about the decision model.
F33 Learning strength in classification-lik settings
Learning utility functions with choice models like the Bradley-Terry model effectively reduces preference learning to a classification task, where preference strength can be reflected in the classifier's confidence. Techniques in supervised learning that address label certainty or aim to modulate learning based on kind of 'strength' can offer conceptual parallels to learning preference strength.
42

---

Label smoothing is a common technique to prevent overfitting in classification by relaxing hard labels to target distribution, typically mixing the ground-truth label with a uniform distribution [52]. While this can mitigate overconfidence in predictions, its fixed and static nature implies that the model is encouraged to predict this smoothed target distribution regardless of the input, which might be suboptimal in some cases [53].
Label relaxation provides an alternative by relaxing the classification target to a range of permissible values, rather than single fixed distribution [54]. This approach allows the model to predict labels based on its own learned confidence for given input, relying on its inductive biases to distinguish between easy (high confidence) and hard (low confidence) examples. The model is not forced to predict a specific distribution but can learn to assign higher confidence (analogous stronger preference) to certain outcomes based on the data.
While these techniques are not directly about learning from RT's, they address the challenge of moving beyond binary outcomes to incorporate a notion of strength or confidence in a learning process. This shares a conceptual similarity with ResponseRank's goal of extracting preference strength information from relative RTs, rather than solely relying on binary preference choices.
F.4 Second-order preference learning
ResponseRank's approach to learning preference strength is grounded in the concept of second-orde preferences. A second-order prefereee relation is preference over comparisons themselves, rather than just preference over individual items. For instance, the preference for ai over by might be judged stronger than the preference for 02 over b2- Assuming an underlying utility function u(x), this implies an ordering over utility differences: [u(a1) u(b,)l > (u(a2) 1l)-wdd u(b))l-
In ResponseRank, we leverage response times (RTs) to induce a ranking over a set of pairwise comparisons. This RT-derived ranking (see Section 2 and Table 1) serves as an empirical proxy for such a second-order preference relation, with faster RTs (for given choice direction) indicating stronger preference.
The theoretical basis for learning from second-order preferences is well-established. Suppes and Winet [14] demonstrate that, under certain assumptions concerning second-order preference relation (which they term a difference relation), information about the ordering of utility differences is sufficient to identify
underlying utility function u(x) up positive linear transformation. ResponseRank aims to recover this cardinal utility function. Presupposing the existence of u(r) (a common starting point in RLHF), these assumptions from Suppes and Winet [14] translate two main categories of conditions for our context:
1. Reliable RT Proxy: Human-generated RTs must consistently reflect the ordering of true latent utility differences. Since RTs are real numbers, any quantitative strength measure derived from them will be numerically comparable and transitive. The core assumption is that this derived order is a faithful proxy for the true order of utility differences.
2. Structural Integrity of True Latent Preferences: The underlying true utility function u(x) and its differences are assumed to possess structural properties ensuring they behave like measures on a continuous, well-ordered scale. Key among these (following Suppes and Winet [14]'s Axioms A6, A9, A10, All) are that:
The set of items K and the utility function u are sufficiently rich (e.g., to allow for approximate midpoints and continuity).
Utility differences combine additively and consistently.
All non-zero utility differences are Archimedean (i.c., commensurable, without in- finitely small or large differences relative to others). While RT-derived strengths are inherently Archimedean (due to RT's being real numbers), the foundational assumption is that this reflects an Archimedean nature of the true underlying preferences.
In essence, these structural assumptions ensure that subjective 'preference strength' corresponds to a well-behaved cardinal utility scale. ResponseRank, by learning from RT-derived rankings, operates on the premise that RT' are a sufficiently faithful proxy to allow recovery of a utility model un (a) that reflects these underlying principles of u(z).
43

---

A second-order preference relation subsumes first-order one by Σ1 + ab Ee aa. ResponseRank realizes this by: (1) using the first-order preference P: normalize comparisons into a (w;.4;) format, and (2) ranking the resultant utility differences so(w;, 12) against a virtual anchor. This anchor serves as the conceptual aa element, with so(aa) = 0 (or more precisely, its score in the Plackett-Luce list is 0). Note that this element does not really refer to any object being compared: the score is entirely virtual and not related to any comparison. An anchor element serving comparable role appears in the multiclass classification approach of Fürnkranz et al. [55], where an 'artificial calibration label' separates relevant from irrelevant labels. Ranking (wi, 4:) pairs above this zero-score anchor directly ensures sø( so(w;; 4) > thereby establishing the correct ordinal preference up(w;) > ug (4). The predictor's inherent antisymmetry (sp(2; xt 2) = O, sg(z, y) = sy (y, 2)) then correctly orients s0(4, w,) automatically. Thus, single Plackett-Luce loss, operating these RT-ordered, correctly-signed utility differences so(w;, 44) and the anchor, learns hoth preference order and strength simultaneously.
G Proof of reduction to BT
Proof for Theorem The theorem considers the special case where an ResponseRank stratum con- tains only single comparison. Let this comparison be q. where item is preferred to item b Following the ResponseRank methodology (as described Section 2), this comparison q is normal- ized to (a, b) (winner first, loser second). Its RT-derived ranking places it above the virtual anchor element, Ag- Thus, the target ranking for the Plackett- (PL) model is the ordered list [g. Aol-
The PL. model assigns scores to the elements being ranked.
The score for the comparison q = (a, b) (representing its preference strength) the predicted utility difference sg(g) = uo(a) - un (b). For brevity, let's denote this as Sab-
The score for the virtual anchor Åo is fixed at s(Ao) 0.
The probability of observing the target ranking [g, Ao] under the Plackett Luce Luce model is calculated as the product of probabilities of selecting each item in order from the remaining set:
exp(Sat)
exp(s(Ao))
Pni. ((g, Aol) exp(sab) + exp()(An))
exp(s (s()))
Prob. of chosen first from (o.
Prob. of Ao chosen next from {
exp($ab)
(4)
exp(8at) + exp(0) exp(sab)
(5)
exp(sab) +1
Now, substituting B0b = (a(a) - (b) (the predicted utility difference):
PeL (1g, Aol) = exp(us(a) - up (b))
(6)
+ exp(ue (a) us (b))
exp(ue / (a)) exp( (b))
(7)
1 + (exp(us exp(uu(b))) (b)) exp(ug(a))
(8)
exp(us(b)) + (xp(us(a)
This final expression (8) is precisely the probability that item is preferred to item b (denoted a > b) under the BT model, (ar(a > b).
Optimizing the parameters to maximize likelihood is equivalent to maximizing the log- (LL). For single observed preference, y {0, 1). where y 1 if a and y 0 > a, and
p = (ar(a > b), the negative log-likelihood is:
NLL = - -iy log(p) + (1 3) log(1 - p)].
(9)
44

---

This NLL is, by definition, the binary cross entropy between the true (observed) preference distribu- tion (represented by y) and the model's predicted probability distribution (represented by p). Thus, maximizing Prt. (lg, Aol) is equivalent to minimizing the binary cross-entropy loss for (B > b). This completes the proof.
H Extended discussion of the Pearson distance correlation
This appendix provides more detailed discussion of the Pearson Distance Correlation (PDC) metric, complementing Section 3 in the main text. We elaborate on its motivation, desirable properties, its validation via synthetic experiments, estimation from finite data, formal proof's of its properties, and its limitations.
H.1 Reliance on true utilities
practical challenge for PDC is its reliance on true utilities u (yielding true differences AU), which are often unavailable in real-world datasets. When a large dataset is accessible, we propose an approximation: train a BT model on its entiret,, using the derived utility function uBT as a proxy ground-truth. This uBT then allows computing ResponseRank) trained on smaller data subsets. This approach justified because BT utilities are identified up to constant additive shift in log space [24]; thus, with sufficient data, the absolute utility differences will closely approximate the true AU (further discussed in Appendix A). For model training it would be preferable to directly use the model trained on more data, but this method enables PDC-based evaluation of leaaning on real-world data during method development. This setup specifically tests how much preference strength can be extracted from (in this case artificially) limited data using informative signals.
approximate PDC for other models (e.g..
H.2 Motivation for a dedicated preference strength metric
Common metrics like calibration error (c.g., True Calibration Error, TCE, based on 4 distance between predicted confidences and true likelihoods [25, 56, 57J) might seem relevant, especially when evaluating models like Bradley-Terry that link utility differences to choice probabilities (akin to classification confidence). However, such metrics are often insufficient for the specific purpose of isolating how well model has learned cardinal distance information (i.e., preference strength). They typically lack key properties desirable for this task:
They often do not possess affine invariance. Scaling utilities directly changes choice prob- abilities derived from them, thereby affecting calibration metries. However, since only relative preference distances (up to a positive scaling factor) influence expected utility maxi- mization, the evaluation of learned distance should be robust to such utility transformations. Intuitively, if we rank policies by expected total reward, affine transformations of these preserve ordering and therefore do not change the optimal policy an RLHF context.
They usually lack ordinal independence, as they tend to conflate ordinal accuracy (which item is preferred) with the accuracy probability estimates (which are tied to utility differences). A model with perfect ordinal accuracy but poor probability estimation (due to miscalibrated distances) would be penalized, making it hard to isolate the quality of distance learning.
They typically do not provide a clear baseline for zero distance learning that is independent of ordinal performance.
To address these limitations, a metric for preference strength should ideally satisfy several key properties. The Pearson Distance Correlation (PDC), introduced in Section 3, is designed with these in mind:
1. Distance Sensitivity. The metric should increase monotonically as the model's predictions more accurately capture information about the true differences (distances) in utility. should specifically reflect the quality of the learned magnitudes of these differences.
2. Affine Invariance. The metric should be unchanged by positive affine transformations of the utility function (i.c., ut az(x) au(z) + b for > 0). This is important because such
45

---

transformations preserve the underlying preference structure (and thus relative distances), and only these relative distances are relevant for expected utility maximization, the ultimate goal of RLHF. Hence, a metric should not penalize a different, but equally good, scaling of utilities. Further, scaling alters choice probahilities derived from the utilities, as this deriva- tion generally involves arbitrary scaling factors (e.g., temperature in a softmax function), factors that should not influence the assessment of learned distance.
3. Known Baseline and Scale. The metrie must have well-defined values corresponding to no learning of utility distances (c.g., O, indicating predicted magnitudes are uncorrelated with true ones) and perfect learning (e.g.. 1), respectively. This enables hypothesis testing about the presence of distance learning and standardized comparison across models, independent of their ordinal accuracy.
4. Ordinal Independence. The metric's baseline value (indicating no distance learning) should be achievable even if the model perfectly predicts ordinal preferences (i.e., correctly identifies which item preferred) but contains information about the magnitude of these differences. This property is critical for distinguishing models that genuinely capture utility distances from those that merely predict accurate ordinal rankings.
PDC's design allows it to reveal insights, such as a Bradley-Terry model implicitly learning significant distance information from ordinal feedback, which would be difficult to ascertain robustly with standard calibration metrics due to their conflation of ordinal and cardinal performance.
Related approaches. Gleave et al. [58] similarly leverage Pearson correlation' affine invariance to compare reward functions, correlating either canonically shaped rewards (EPIC) or episode returns (ERC). While sharing much of the motivation, PDC differs by correlating absolute utility differences, explicitly discarding ordinal information to achieve ordinal independence and supplement existing accuracy metrics.
H.. Formal properties and proofs
This section provides more formal statements and outlines the proof' for the properties of PDC, elaborating on Proposition from the main text.
Theorem 2 (PDC Properties- Formal Statement). Given true utility function u and predicted utility function à, let X, X' be independent items sampled ii.dd from a data distribution Pasta- Define the true absolute utility differences as AU |u(X) u(X') and the predicted absolute utility differences as AÜ [X) (x')). Assuming V[AU]= > 0 and V[AU] > the Pearson Distance Correlation (PDC), pPdc(u, ú) = Corr(AU, AU), satisfies:
1. Affine Invariance: For any constants C1, C2 E R and 81, 82 > 0: Let u'() =ju(X)= + ci and ü' (X) 82ū(X) + 02. Then AU" AU and AÚ" S2. follows that PPDC (u', a') = pPDo(u, u.)
2. Distance Sensitivity: Let ûn and ú2 be two predicted utility functions, yielding AÙ; ân (X) - ûn (X') and AÙ, = [ύ₂(X) ( If AÜ, provides a "better" positive linear approximation to AU than AÚ2 does, then PPDc(u, ûn) 2 PeDc(u, u2). "Better" here means that the variance of the residuals from an optimal linear regression of AU, on AU (i.e., V'A (a + bx)) for optimal og >> 0, by) is smaller for k = 1 than for k- =2 2, relative to V(AU, and V[AU2] respectively.
3. Known Baseline and Scale: PPDo(u, a) = O if AU and AÙ are uncorrelated. PPDC (u, û) = 1 if AÚ is a perfect positive linear transformation of AU (i.e., AÙ = aAU + b with a > 0). PrDe(u, úi) = -1 1 if AÚ is perfect negative linear transformation of AU (i.e., AÚ aAU + b with < 0).
4. Ordinal Independence: Let ūn and ú2 be two predicted utility functions. If. for any pair (X, X') drawn from Paata x Pasta, it holds that ( (x( - âj (X') = [x( (x())) (i.e., the magnitude predictions are identical), then PPDC(u, ūz) (PDCC k2)
5. Range -1 S pPDc(u, a) S 1.
46

---

Outline of Proofs for Theorem 2.
The PDC is defined as Corr( AU, AU). Its properties thus largely follow directly from standard properties of the Pearson correlation coefficient.
1. Affine Invariance:
Given w'(X) = $1 + c1 and (x) = 82 (x + C2 with 81, 82 0. The transformed absolute differences
AU" = (X) u (x()) |1 (u() - =(X)) $1 AU (since $1 > 0), and similarly AÙ" szAU. The Pear- son correlation coefficient Corr(A, B) is invariant to positive linear scaling of its arg-
ments, i.e., Corr(s: $ A, s2 = Corr(A, A, B3) for $1 $2 [cf. 59]. Thus, Proc(u', at) Corr(s)AU, 22A = Corr (AU AÚ) = PPDC [u, ū).
2. Distance Sensitivity: The PDC, being Corr(AU, AU&). directly reflects the goodness of the linear fit between these two variables. The square of the Pearson correlation is the coefficient of determination (R²), which for a regression of AÜ, on AU is [[residuals] W[Aü The coefficient of determination quantifies the proportion of the variance in variable that is predictable from the other variable in a linear regression, thus reflecting the goodness of linear fit [60, 61]. A "better" positive linear approximation defined in Theorem (i.e.. a smaller relative residual variance, with assumed regression coefficient ag > 0) implies a higher R Consequently, this leads to a higher PPDC ûg) value.
3. Known Baseline and Scale: These are direct consequences of standard Pearson correlation properties: Corr(A, B) = 0 if A and B are uncorrelated. Corr(A, B) = if B = aA + b with 0, and Corr A, B) = -1 B aA + with a < [62]. PDC inherits these directly with = AU and B AÙ.
4. Ordinal Independence: If [u (X) - d1 (X') = 12 (X) úz (X') for all pairs (X,x') then the random variables for predicted absolute differences, Ab; and A02 are identical. By definition, PPDC(u, ún) = Corr(AU, At1) and PrDc(u, ú2) = Corr(AU, At2). Since
A-1 = AU2, it immediately follows that PPDC (u, 01) PPDC (u. m2).
5. Range: The Pearson correlation coefficient Corr(A, (A, B) always lies in the interval [-1, [62]. As PDC is defined as such a correlation, it directly inherits this range.
Corollary 1 (PDC baseline for no distance learning). If predicted utility difference magnitudes [((X)= ú(X') are statistically independent of true utility difference magnitudes [u(X) u(X') (even if signs of the utility differences are preserved), then the PDC satisfies:
PPDC(u, (u, û, Pdata) 0.
(10)
This establishes a baseline of 0 when no meaningful distance information learned, providing basis for hypothesis tests.
H.4 Synthetic experiment validating PDC properties
Figure 2 in the main text (Section 3) compares PDC and TCE under simulated prediction degradation to validate PDC's properties. The experiment starts with perfect predictions (á = 2) based on synthetic utility data (NV 1000 items, random utilities with normal distribution). Degradation is applied by:
1. Randomly flipping the sign of the predicted utility difference for fraction faia of item pairs (introducing ordinal errors, y-axis of the heatmaps).
2. Shuffling the magnitudes (absolute values) of the predicted utility differences for a fraction Smay of item pairs (destroying distance information, x-axis of the heatmaps).
This process is performed for both the original utilities u and affine-transformed utilities w' = 2u + 5.
The figure demonstrates PDC's properties as follows:
Ordinal Independence: Observed along the left edge (fmag = 0) of the PDC heatmaps. Even faigo increases (moving up the y-axis, introducing more ordinal errors), PDC remains
47

---

at its maximum value (blue), indicating perfect correlation of magnitudes. This demonstrates that PDC's assessment of distance learning unaffected by purely ordinal errors when true distance information is perfectly preserved. This contrasts sharply with TCE, which worsens (increases) with Sign.
Distance Sensitivity: Observed as one moves from left (Smag = 0) to right (Smag 1) across the PDC heatmaps. PDC systematically decreases (plots become less blue/more red) as faag increases, reflecting its sensitivity to the progressive loss of true magnitude information caused by shuffling.
Demonstrated by comparing the left PDC panel (original u) with the right PDC panel (affine-scaled = 2u + 5). The heatmaps are nearly identical, confirming that PDC's evaluation is robust to positive affine transformations of the underlying utility function. TCE, conversely, shows different patterns for u and u', highlighting its lack of affine invariance. Note that technically the desirable property is invariance affine transformations of the utility distance. Scaling of utilities directly translates to scaling of distances, however, shifting utilities cancels out, and shifting of distances is not meaningful due to the anchoring at O.
Affine Invariance:
optimal (blue, approxi- mately 1) when faug (perfect distance information, irrespective of sagn), and baseline (red, approximately 0) when fmug = 1 (all distance information destroyed, irrespective of faga). This aligns with the theoretical properties.
Known Baseline and Scale: The PDC heatmaps show a clear scale:
The TCE plots in Figure 2 clearly lack these desirable properties. TCE increases (worsens) with both sign flips (Jaign 0) and magnitude shuffling (Smag >> 0), thereby conflating ordinal and distance errors. Furthermore, its sensitivity to affine scaling (evident from the differing left and right TCE panels) and its lack of a clear, ordinally- independent baseline for distance learning make interpretations of TCE regarding Icarned strength ambiguous.
Contrast with TCE:
This empirical validation highlights the PDC's ability to specifically and robustly quantify Iearned utility distance information, fulfilling the design goals outlined in Appendix H.2 and Section
H.S Estimation of PDC from sample data
In practical scenarios, the PDC (Definition 1) is estimated from a finite dataset of pairwise compar- isons. Let Paairs- = (ai, bi) denote N item-pairs sampled i.i.d. from Pdata Pdata- The sample PDC is computed as the sample Pearson correlation coefficient:
ΣN (AU; AU) (AU, AÚ)
(PDC d; Ppairs)
(11)
V EiE (AU; AU) ΣN (AU; AÚ
where AU; = (u((;)- and aÜ, = ((a;))((b))) are the observed true and predicted absolute
utility differences for the i-th pair, and AU, AÙ denote their respective sample means. While the sample PDC, OPDC. is not generally an unbiased estimator of the population PDC, PPDC, it is consistent estimator under standard assumptions [62, Section 9.7].
I Baselines
This appendix details the baseline and control methods used for comparison against ResponseRank in the experiments (Section 4).
Bradley-Terry (BT). This baseline represents standard RLHF approaches that learn utility functions solely from preference labels, without considering response times. As a response-time tnaware baseline, we use the Bradley-Terry model [20] to learn the utility function from the preference labels p; alone. The Bradley-Terry model is special case of the Plackett-Luce model discussed in Section for two alternatives. Unlike ResponseRank, which ranks entire comparisons, the BT model as used here ranks individual items (c.g., a; b.). The model learns utility function u that maps
48

---

each item to scalar utility value. It assumes that the probability of choosing item a; over item b, is given by the softmax function
e<(ma)
p > b,) (a((a)) + (a(b))"
(12)
This baseline helps establish the performance level achievable using only the ordinal preference information inherent in choices.
Response Time Regression (RtRegression). The RtRegression baseline represents methods that directly model RTs by assuming a specific, known, and global functional relationship between RT and preference strength. It is a regression-basee approach that directly models the response times т as a function of the utility differences. It assumes т; = (in((((a)) - u(bz)), where link is a specific function mapping utility differences response times. It further requires link to be known and invertible, which is a strong assumption often not met in practice. Given this assumption, the response times Ti give access to the unsigned utility differences via (u(a:) u(b;)] = link (r,). To model cardinal preference strength and direction jointly, preference label pt. We define sign(pt) = +1 if oz is preferred (e.g., p: 0), and sign(p;) -1 b; is preferred (e.g.. pt 1). The regression target is then:
convert this signed difference using the
Au; = sign(p;) link-1(1))
(13)
This signed difference Aú; forms the regression target. A neural network uy is trained with an MSE loss to predict these signed differences. The network's prediction for pair (a,, by) is the strength predictor sy (a;, b,) = ug (a:) - ub.b;). Minimizing (s,((a, b;) - Á yields the parameters for our utility function ug. In experiments, we assume hyperbolic link function (Equation (14)) that exactly matches the data-generating process in some synthetic datasets.
link (in Itmin Itmax) = rtmin rbmax rbmin
(14)
Au +1
This specific link function is invertible (if the sign of Au is known), inversely proportional Au), and bounded between rtmin and rtmax. While informed by the general observation of an inverse RT-strength relationship [43], the specific choice of Equation (14) is illustrative and by no means unique: the requirement to assume some explicit link function is a primary weakness of this baseline approach.
Critically, in contrast to ResponseRank, this RtRegression approach relies on the absolute RT values (processed via the assumed inverse link function). It makes strong global assumptions about the exact functional form relating RT and preference strength. ResponseRank avoids these assumptions by using only the relative order of RTs within carefully constructed strata (Section 2). This reliance on relative ranks within strata provides robustness when the exact global RT-strength relationship is unknown or varies significantly across contexts or individuals.
and RdRegression-Perm) These controls verify that performance gains stem from informative RT signals, rather than from the structure of the loss or model itself. The core mechanism involves randomly permuting the response times т; across the dataset to get (()) where и is random permutation. This isolates the contribution of RT magnitude information while preserving the ordinal preference direction. We include two such controls based on our primary RT-aware methods:
Permutation Controls ReeponnseRank-Pee
GesponseRankk-Penn Applies the ResponseRank ranking method but uses the permuted times Tm(() when constructing the magnitude component of the sort keys k,. The preference label pt still determines the key's ss directional component.
MuRegression-Perm Applies the RtRegression method but uses the permuted times Tx))) as input to the inverse link function, i.e., the regression target becomes Aú; = sign(p:) link (Tx(e)) Similarly here the sign of the inferred utility strength remains untouched by the permutation.
A significant drop in performance (especially in PDC, Section 3) for these controls compared to their con-permuted versions (ResponseRank and RtRegression) confirms that the original methods successfully leveraged meaningful information encoded in the timing of human responses.
49

---

Stratification ablation (ResponseRank-Pool)) This ablation investigates the specific benefit of stratification within the ResponseRank framework. We consider ResponseRank-Pood ResponseRank that ablates the stratification step. This variant omits stratification based on meta- data m, (Section 2). Instead, comparisons are grouped into random batches of size b for ranking. All other aspects of the ResponseRank method (e.g., learning utility differences via Plackett- Luce loss on comparison ranks) remain the same. By comparing the performance of ResponseRank to ResponseRank-Pon we can isolate the performance impact specifically attributable to the strate- gic use of stratification to control for confounders. If ResponseRank significantly outperforms ResponseRank-Pont it validates the effectiveness of the stratification strategy.
variant of
J Extended discussion of related work
This appendix provides further context and detailed comparisons related to the main paper's discussion of related work, covering foundational perspectives on response times and prior computational methods leveraging RTs other strength signals.
J.1 Detailed comparison with prior RT modeling approaches
Section 7 briefly introduced prior methods integrating response times (RTs) into preference learning. Here, we provide more detailed comparison highlighting the technical distinctions between these methods and our ResponseRank approach, particularly concerning their underlying assumptions and applicability to large-scale RLHI.
Prior methods that integrate RTs often employ explicit cognitive process models like the Drift Diffusion Model (DDM) [e.g.. 21, 2] or make strong assumptions about a direct, global relationship between RT magnitude and preference strength [e.g., 22].
J.2 Ranking-based utility learning
Methods like SeqR [63] and LiRE [64] also leverage ranking structures in RLHF. Both construct preference rankings of items from pairwise feedback and derive overlapping preference pairs (e.g.. A B, B > C, A > C) to train with Bradley-Terry loss. This imposes soft constraints on utilities and thereby conveys implicit strength information [64].
ResponseRank differs in two key aspects: (1) it ranks comparisons of items rather than items themselves, and (2) it uses external strength signals rather than preference feedback to structure the rankings. This reflects different objectives: SeqRank and LiRE improve feedhack efficiency through querying strategies and data augmentation, requiring control over the queries posed to the annotator, while ResponseRank focuses on loss formulation for leveraging auxiliary strength information.
J.3 Explicit strength specification
While ResponseRank leverages implicit signals of preference strength, other works take the approach of explicitly eliciting strength information from users through modified feedback mechanisms.
Several approaches directly request strength annotation during preference collection. Wilde et al. [5] proposes scale feedback, where users provide slider-based response &i E [- for each comparison, where the magnitude encodes preference strength proportional to reward difference. Similarly, Jansen et al. [22] propose a strength elicitation method that requires users to explicitly assign preference strength labels during comparisons.
Other approaches ask users specify the relative strength of preferences between multiple compar- isons. DistQ [65] requires users to select which of two comparisons is easier to distinguish, then judge that pair. While conceptually related to ResponseRank, DistQ requires explicit meta-cognition about comparison difficulty and uses a composite pairwise loss instead of our joint ranking loss. Future work could explore combining ResponseRank's loss with DistQ's s explicit distinguishability queries or vice versa. Another approach, taken by Papadimitriou and Brown [66], asks users to group items into preference tiers (e.g., good, acceptable, dangerous) and then explicitly specify the relative margins between these tiers (c.g., acceptable dangerous is twice as large good acceptable).
50

---

All these approaches recognize the importance of preference strength, but they all require explicit user annotation of strength. ResponseRank instead focuses on leveraging implicit response metadata without modifications to the feedback collection process, making it broadly applicable and able to reuse existing datasets.
J.4 Adaptive losses
While ResponseRank leverages explicit external strength signals, complementary work explores using adaptive losses to allow the learner to more robustly learn preference strength implicitly. Hong et al. [67] propose Ada-Pref, which learns instance-specion via distributionally robust optimization. Pairs with ambiguous preferences receive smaller scaling factors, allowing the reward model to learn smaller reward differences; pairs with clear preferences receive larger factors, enabling larger learned differences. This adaptivity emerges implicitly, similar to implicit strength learning in non-adaptive BT discussed in Appendix A.
loss scaling factors for each preference pair
Ada-Pref and ResponseRank address complementary scenarios. ResponseRank excels when domain- specific strength signals are readily available and locally valid. Ada-Pref suits settings where no reliable strength signal exists, but preference data contains implicit variation in confidence or difficulty. Future work may explore combining these two approaches, though ResponseRank's ranking construction may already capture much of the implicit learning Ada-Pret achieves.
J.5 Approaches based on explicit cognitive models
A significant line of research incorporates RT' by employing explicit cognitive process models, primarily variants of the DDM. These models aim to provide a mechanistic account of how latent utility differences jointly influence generated choices and RTs, enabling learning from both signals.
Shvartsman et al. [21], for example, propose integrating DDM within Gaussian Process (GP) framework for preference learning, primarily targeting the low-data regime and reporting significant improvements sample efficiency. Their method links the GP's latent utility difference to the DDM's drift rate. A key challenge is that the joint choice-RT likelihood of the DDM is analytically intractable, and common numerical approximations are non-differentiable. They tackle this challenge by approximating the DDM using moment- moment-matching matching with family of parametric skewed distributions, which allows them to optimize the GP hyperparameters by minimizing the approximate negative log marginal likelihood of both choices and RT's within a DDM-inspired likelihood framework.
Similarly grounded in cognitive modeling, Li et al. [2] use the simplified, computationally tractable dEZDM variant [68] to incorporate RT's into preference-base best-arm identification. Their core mechanism relies on an analytical property of the dEZDM linking the ratio of expected choice and expected decision time directly to the scaled utility difference, where the utility function is assumed linear within their bandit framework. They estimate these scaled linear parameters via linear regression, using empirical averages of choices (C..) and decision times (f.) computed over multiple identical query repetitions as the target value Cz/E,, and integrate this into the Generalized Successive Elimination (GSE) algorithm. This approach requires averaging over repeated identical queries to estimate these expectations (Cz, Eg) and assumes utility linearity within a bandit setting, using Cz/ [r, as the target for linear regression to estimate scaled linear parameters.
linear bandits, focusing on fixed-budget
ResponseRank contrasts significantly with these DDM-based approaches. Firstly, it entirely avoids specifying or fitting an explicit cognitive model of the RT generation process. This sidesteps the complexities associated with DDM likelihood calculations, their approximations, and the potential fragility of assuming specific cognitive process accurately describes diverse human behavior across different tasks and contexts. Secondly, the RT signal used is fundamentally different. ResponseRank relies solely on the relative rank order of total observed RTs within specific strata as noisy, non- parametric indicator of preference strength rank. It does not attempt to link RT magnitudes directly to model parameter like drift rate, nor does it need to isolate a theoretical "decision time" from the measured total RT. Consequently, ResponseRank circumvents the restrictive data requirements seen in some prior work, such as the need for repeated identical queries or assumptions of utility linearity [2], making it directly applicable to standard RI datasets typically consisting of unique comparisons.
51

---

J.6 Other RT integration strategies
Other methods avoid fitting full cognitive models directly or integrate RT's in different ways.
Shvartsman et al. [21], in addition to their DDM-based approach, also propose prediction stacking- This is a two-stage approach where a separate model is first trained to predict RTs based on item features. These predicted RTs then serve as additional input features for choice model (a final GP model that predicts choice probabilities), potentially informing it about decision difficulty without relying the DDM's generative assumptions for the choice model itself. RT predictions might act as a form of regularization, modulating the influence of choices based on predicted difficulty. ResponseRank differs as it uses observed (not predicted) RT's and integrates their rank information directly into the training objective via the ranking loss, rather than using RT's as input features.
Aiming to elicit complex preference systems (IA, R1; R2), with R2 representing preference strength, over a fixed item set A, Jansen al. [22] propose time elicitation. After collecting ordinal preferences (R2) via pairwise comparisons, they measure the user's 's response time for each pair. Crucially, the method then deterministically constructs the cardinal relation (Bz) representing relative preference strength. This construction is based directly the RT magnitudes, assuming these perfectly repre- sent preference exchange strength. This direct construction of a preference system from assumed clean response times over fixed set A differs fundamentally from learning utility function from potentially noisy, total response times over structured objects in an open world, as is typical in RLHF settings. Nonetheless, their direct construction of R2 partially motivates our construction of the target ranking relation in ResponseRank.
J.7 Core distinctions
While the methods discussed above offer diverse ways leverage RTs, ResponseRank is distinet in its minimal and localized assumptions. Primarily, ResponseRank diverges in the assumed scope of the RT-strength correlation. Unlike prior works that often implicitly or explicitly assume global correlation (faster RTs consistently indicate stronger preferences across the dataset), ResponseRank posits a weaker, local assumption. This local view holds that the inverse RT-strength correlation is reliable only within specific, carefully constructed strata (e.g., per-annotator or per-session), which are designed to control for confounding factors like annotator effects or task complexity. This local approach is crucial for robustness, enabling effective learning from pooled data across diverse annotators and contexts where global RT comparisons would be unreliable.
Furthermore, ResponseRank does not specify or fit an explicit cognitive model (like DDM) to describe the joint generation of choices and RTs. This design choice avoids the complexities of DDM likelihood calculations and their approximations (cf. 21, 2). and the potential fragility of assuming single cogmitive process accurately describes diverse human behavior. Consequently, rather than using RT magnitudes directly as model inputs, as features (e.g., in prediction stacking; 21), or for deterministic rule construction from raw RTs (e.g., in time elicitation; 22), ResponseRank utilizes the relative rank order of total observed RTs within these strata. This ordinal information from RTs serves as noisy, non-parametric proxy for the rank order of preference strength, without attempting to model precise RT values or link them to specific model parameters like a DDM's drift rate.
These characteristics enable ResponseRank to circumvent many restrictive data requirements common in prior work, such as the need for repeated identical queries [2], assumptions of utility linearity [2], operations on fixed item set [22], or assumptions about 'clean' RT magnitudes [22]. By leveraging relative RT ranks within strata via a ranking loss (like Plackett- ResponseRank is designed to bypass these requirements. This offers robustness and direct applicability to typical large-scale RLHF data, which often involves unique comparisons, non-linear utilities, and noisy total response times from diverse sources. The method aims to filter out systemic variations and noise from confounding factors (individual speed, fatigue, task complexity) without explicit modeling of these factors or relying on pristine RT magnitudes.
K Compute resources
We conducted the synthetic experiments on a single compute node with CPU cores and 32 GB of RAM. They take approximately 17 hours to complete in parallel this hardware. The MultiPref
52

---

experiments were run on A100 and H100 GPUs, with a full-size (full training set fraction) run taking approximately 1.5h on a single H100 GPU.
The RI experiments were conducted on a compute cluster. The control experiments had four separate stages: (1) Training of the baseline RI models for collecting checkpoints, (2) Generating feedback datasets based on rollouts from checkpoints, (3) Training of reward models with feedback datasets, and (4) training of downstream RL models with reward models. Steps 1,2, and were conducted on CPU nodes, with each individual run being allocated 2 CPU cores and 16GB RAM. Training of RL agents takes around hour, collection of feedback around 30 minutes. Reward model training was performed on nodes with access to A 100 and H100 GPU shared across four parallel runs. Reward model training took approximately 30 minutes. In total. 20 baseline runs and 20 feedback datasets were generated. To reproduce the results, 240 reward model trainings and RL down-streand training runs are required (with shared GPUs) this results in approx. 30 GPU hours and 480 CPU hours.
Software and data assets
This work uses the following external assets:
MultiPref dataset (version 1.0) [15]: http:://wwggiggaae.e..oo/aaataat multipref, licensed under ODC-BY.
Llama-3.--B-Innttrutt [35]] https : htt////uggnngaaeeeeeeee hawgginnfaae...o/eteellaaaa
B- Instruct, licensed under the Llama 3.1 Community License.
MuJoCo v5 enviromments (HalfCheetah, Swimmer, Walker2d)
om/ htm/googlle-ddeemmind/mmujmoooo licensed under Apache 2.0.
highway-env merge-v0 nv, licensed under MIT.
We release the following assets hLtppe///gghaaaccco//tttttteepps documented in the repository README:
Experiment source code, licensed under MIT.
Numerical results for synthetic, RL, and LLM experiments.
53

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims match theoretical and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses limitations in the limitations subsection of the discussion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
54

---

Justification:
Full proofs of the theoretical results are provided in the appendix. The assumptions are clearly stated the main paper or the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The experiments are described the main paper (Sections 4.1, 5 and 6) and expanded on in the appendix (Appendices C to E).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution primarily new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (c.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
55

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will provide access to the code, which enables synthetic data generation and model training.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines ttttps://nips...
DJic/guuiees/ooeeuuubmiiess http://ooeeummmmiiinnnnee onPolicy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //ipp.ccppppiiiiiomm wwtppcccccc/////cccccccccccc..lt wttpc//gg//eeeeeeeceeeeeeesseees https//ooSSSuuiiseeeeee soabbbmsssionlli ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental setups are described in Sections 4.1, , 5 and 6 and expanded on in the appendix (Appendices C to E), with remaining details in the provided source code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All plots include error bars, with the captions indicating the type of error bars used.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
56

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute resources are described in Appendix K.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https /ttp:r////////p////eee cccpublic//ettcecececccece hawbllic/chiccsuddeeeeees
Answer: [Yes]
Justification: This research does not involve human subjects and otherwise adheres to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses potential broader impact in the discussion, but societal impact in general is not beyond that of a typical ML paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
57

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not involve releasing data or models that have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets are cited and listed with URLs and licenses in Appendix L.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
58

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode thcode. com aamdatasees has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We release source code and numerical results; see Appendix L..
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
59

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The paper does not involve LLMs as an important, original, or non-standard component of the core methods.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
60


Reviews:
[{"review_id": "anuqBvireC", "reviewer": "Reviewer_7tw8", "summary": "This paper considers the degree of preference strength in preference learning. Specifically, the article uses the time intervals during data annotation as an indicator of preference strength, assuming that stronger preferences lead to shorter reaction times. After grouping the annotated data accordingly, they model it using Plackett-Luce. To measure how well reward models capture this degree information, the authors propose the Pearson Distance Correlation (PDC) metric. On synthetic datasets, their approach outperforms the baseline (BT model).", "questions": "1. What does the preference pair normalization mentioned in Figure 1 refer to, why is normalization necessary, and how should real data be processed?\n2. There are many possible scenarios in real annotation settings. Has the author considered real-world scenarios, such as how to screen for outliers, including more practical data cleaning issues like different comprehension times for different questions?", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n1. The authors use annotation time intervals as an indicator of preference strength, which allows incorporating preference degrees into the scoring information and should theoretically yield better results. \n2. Meanwhile, the authors focus on addressing the lack of metrics for evaluating how well preference models capture preference degrees, introducing a measure based on the correlation coefficient between true scores and trained scores.\n\nWeakness：\n1. Lacks training and testing results in real-world scenarios. The experimental section of the paper is based on synthetic data, which inherently involves strong assumptions. The actual effectiveness in real preference learning scenarios remains unknown.\n2. The PDC metric is difficult to utilize. The use of the PDC metric in real-world scenarios seems to conflict with the method itself. Since we use BT as an approximation for the golden truth in real scenarios, this implies the assumption that BT performs best on large-scale data. However, the paper's method shows better performance than the BT model on synthetic data, which contradicts this assumption.", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "qQc2C41T58", "reviewer": "Reviewer_jccL", "summary": "This paper utilizes the response time of the human labeler to leverage the strength of the preferences. The authors say that the short response time indicates the stronger preference, and using such preference strength, the utility model is trained to quantify the preference strength. Furthermore, by introducing the Pearson Distance Correlation to evaluate how the model learns the preference strength, in the experiment, the proposed method shows higher PDC and accuracy than other baselines.", "questions": "Already mentioned in the Weaknesses section.", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. This paper uses additional information, response time, to quantify the strength of the user preference, and show that the effectiveness of proposed method compared to the standard Bradley-Terry model.\n\n\nWeaknesses\n\n1. First, the reason for the effectiveness of leveraging the response time to quantify the preference strength is not clear. Why the response time can indicate the preference strength? Is there any toy experiments on showing the effectiveness of use response?\n\n2. There is no reinforcement learning experiment using the proposed method. Since the high accuracy or PDC do not directly reflect the better performance of RL agent, it would be better to show the performance of RL agent using the proposed reward model.\n\n3. More comparison should be performed with other state-of-the-art baselines such as sequential pair-wise comparison methods [1] or list-wise comparison methods [2]. The authors compared the proposed method only with standard models.\n\n[1] Hwang et. al., Sequential Preference Ranking for Efficient Reinforcement Learning from Human Feedback, NeurIPS, 2023\n\n[2] Choi et. al., Listwise Reward Estimation for Offline Preference-based Reinforcement Learning, ICML, 2024", "was_revised": true, "final_justification": "Since the authors addressed my concern, I increase my score to 4.", "rebuttal": "Thank you for your review and your insightful comments, which will help us improve the paper. We appreciate that you recognize the strength of our method, but also appreciate the weaknesses identified which we will discuss in the following.\n\n> **1. First, the reason for the effectiveness of leveraging the response time to quantify the preference strength is not clear. Why the response time can indicate the preference strength? Is there any toy experiments on showing the effectiveness of use response?**\n\n**Foundational support for RT.** Psychology research demonstrates that response time is inversely correlated with preference strength [1]. When options are similar in utility, humans require more time to analyze differences and estimate each option's worth accurately. This principle underlies the drift-diffusion model, which has extensive empirical support. We discuss this thoroughly in Appendix A.1 (third paragraph) and Section 5 (\"Learning from RTs\"), with additional details in Appendix G.1.\n\n**Caveats of RT.** Response times require clean data and careful stratification. We initially planned experiments using MultiPref dataset response times but found them unusable due to confounding time taken for metadata annotation. Unfortunately, we lack access to clean real-world RT datasets. However, our MultiPref experiments using agreement-based strength rankings provide strong support for real-world applicability when preference strength rankings are available.\n\n**RtRank beyond RT.** Crucially, our core contribution, the rtrank loss, is not restricted to response times. It can incorporate any secondary information about preference strength, requiring only ordinal and local comparisons (no globally comparable values). Beyond response times, we could use mouse movement, gaze information, domain knowledge, or directly stated strength information commonly collected but underutilized in preference datasets.\n\n> **2. There is no reinforcement learning experiment using the proposed method. Since the high accuracy or PDC do not directly reflect the better performance of RL agent, it would be better to show the performance of RL agent using the proposed reward model.**\n\n**RL experiments.** We conducted additional control domain experiments using established ground-truth reward functions to synthesize preferences and strength rankings. Following recent methodology [2], we tested three Mujoco environments (HalfCheetah, Swimmer, Walker2d) and Highway merge-v0, comparing Bradley-Terry baseline with RtRank using 5000 queries per training run (5 seeds each). Here, we report both max. and final scores (in brackets the comparison with the BT baseline):\n\n\n| Environment | Max Reward | Final Reward |\n|------------:|-----------:|-------------:|\n| HalfCheetah-v5 | 5866.9 **(+352.3)** | 5215.2 (-219.3) |\n| Swimmer-v5 | 110.0 **(+66.1)** | 98.3 **(+77.1)** |\n| Walker2d-v5 | 3905.1 **(+1088.1)** | 2679.7 **(+313.2)** |\n| merge-v0 | 12.2 **(+1.2)** | 11.4 **(+1.0)** |\n\nRtRank generally outperforms BT models. For the Mujoco environments, we find that RtRank reward models also achieve higher accuracy, in line with improved downstream performance. For merge-v0, RtRank consistently outperforms despite similar reward model accuracy, suggesting better reward shaping.\n\n**Noise robustness.** The results above were measured with perfect environment rewards. To measure robustness to noise, we also benchmarked with different noise levels (according to the methodology outlined in [2]), which perturbs the underlying ground-truth reward and in turn extracted response times and preferences.\nWe find that RtRank outperforms the BT-model in low-noise scenarios, and only underperforms when overall RL performance deteriorates significantly. In environments that are generally robust to noise (such as merge-v0 or HalfCheetah), RtRank also stays stable.\n\n| Environment | Max Reward | Final Reward |\n|------------:|------------:|------------:|\n| HalfCheetah-v5 noise=0.0 | 5866.9 **(+352.3)** | 5215.2 (-219.3) |\n| HalfCheetah-v5 noise=0.1 | 5208.3 (-124.3) | 4922.7 (-387.7) |\n| HalfCheetah-v5 noise=0.25 | 5850.7 **(+418.2)** | 5481.4 **(+201.4)** |\n| HalfCheetah-v5 noise=0.5 | 5433.3 **(+155.5)** | 5433.3 **(+439.5)** |\n| Swimmer-v5 noise=0.0 | 110.0 **(+66.1)** | 98.3 **(+77.1)** |\n| Swimmer-v5 noise=0.1 | 94.6 **(+48.9)** | 81.8 **(+49.6)** |\n| Swimmer-v5 noise=0.25 | 36.7 (-4.8) | 9.5 (-12.6) |\n| Swimmer-v5 noise=0.5 | 47.4 (-29.1) | 17.0 (-49.7) |\n| Walker2d-v5 noise=0.0 | 3905.1 **(+1088.1)** | 2679.7 **(+313.2)** |\n| Walker2d-v5 noise=0.1 | 3126.1 **(+298.1)** | 2742.2 **(+216.0)** |\n| Walker2d-v5 noise=0.25 | 2567.7 (-1344.8) | 1919.1 (-1616.6) |\n| Walker2d-v5 noise=0.5 | 1754.5 (-1387.6) | 1621.9 (-1037.9) |\n| merge-v0 noise=0.0 | 12.2 **(+1.2)** | 11.4 **(+1.0)** |\n| merge-v0 noise=0.1 | 12.1 **(+0.9)** | 11.4 **(+0.9)** |\n| merge-v0 noise=0.25 | 11.8 **(+0.5)** | 11.7 **(+1.1)** |\n| merge-v0 noise=0.5 | 12.4 **(+0.8)** | 11.5 **(+0.3)** |\n\n\n**MultiPref experiments.** To demonstrate real-world applicability, we trained language reward models on MultiPref using inter-annotator agreement for strength rankings (lacking reliable RT data). While not a perfect test for RtRank with response times, this serves as proof of concept for our method's applicability to language modeling settings. Results across training sizes:\n\n| Train samples (fraction) | Test set accuracy | RewardBench v1  | RewardBench v2  |\n|-------------------------:|------------------:|----------------:|----------------:|\n| 3271 (0.4)                | 74.7 **(+3.0)** | 77.4 **(+5.4)** | 49.0 **(+3.9)** |\n| 4907 (0.6)                | 74.6 **(+3.2)** | 77.1 **(+0.8)** | 54.4 **(+4.8)** |\n| 6540 (0.8)                | 74.1 **(+2.5)** | 79.3 **(+1.6)** | 56.1 **(+3.8)** |\n| 7359 (0.9)                | 74.2 **(+3.6)** | 76.0 (-1.8)     | 52.3 (-0.3)     |\n| 8179 (1.0)                | 74.7 **(+3.7)** | 78.1 (-1.4)     | 56.1 **(+1.3)** |\n\nWe fine-tuned from Llama-3.1-8B-Instruct with optimal hyperparameters found for both variants. Strength rankings used weighted annotator agreement (clear preference: weight 1.0, slight: 0.5). Results demonstrate consistent improvements across most settings.\n\nAll sizes are subsets of the total MultiPref dataset after withholding a test set. We filter both sets for texts exceeding 1024 tokens, resulting in ~1400 test samples. We shuffle the dataset with different seeds and report mean performance across 10 seeds. Columns show performance on in-distribution (MultiPref) test set and two out-of-distribution sets: RewardBench v1 and v2 [4,5] (measured per official methodology). Both RtRank and BT use the same optimal hyperparameters: 3 epochs, effective batch size 64 (16 on-device, 4 accumulation steps), learning rate 15e-6, linear LR scheduler with 0.05 warmup ratio, gradient clipping at 1.0, weight decay 0.1, Adam optimizer (0.9/0.999/1e-08). For RtRank, we compute strength rank by weighted annotator agreement, then construct balanced strata with no ties (most strata size 4).\n\nWe will include detailed descriptions of these experiments in the paper.\n\nThese preliminary results highlight RtRank's potential for downstream tasks. Particularly RewardBench v2 has been shown to correlate strongly with downstream performance [4], further validating our approach.\n\n> **3. More comparison should be performed with other state-of-the-art baselines such as sequential pair-wise comparison methods [1] or list-wise comparison methods [2]. The authors compared the proposed method only with standard models.**\n\nThank you for highlighting these relevant works! Both Choi et al. and Hwang et al. use transitivity to construct item rankings from pairwise comparisons, then apply Bradley-Terry loss to generated pairs. Their rankings do communicate strength information similar to ours.\n\nHowever, they differ in two crucial aspects: (1) They construct rankings over *items* while we rank *comparisons*. (2) They **require specialized sampling procedures** while rtrank works with any preference data given local strength judgments (e.g., response times).\n\nThis makes direct comparison difficult since their contribution lies primarily in querying procedures and data augmentation, while ours focuses on novel loss formulation. We cannot apply their methods unchanged to our dataset, and implementing their querying strategy would make approaches quite similar. Importantly, the methods are **inherently complementary**.\n\nWe will **add discussion** of these approaches, their impact on preference strength learning, and potential complementary usage to our related work section.\n\n---\n\n*[1] Milosavljevic et al., \"The Drift Diffusion Model can account for the accuracy and reaction time of value-based choices under high and low time pressure.\" Judgment and Decision Making, 2010.*\n\n*[2] Metz et al., Reward Learning from Multiple Feedback Types, ICLR2025*\n\n*[3] Lambert et al., \"RewardBench: Evaluating Reward Models for Language Modeling\", NAACL findings, 2025.*\n\n*[4] Malik et al., \"RewardBench 2: Advancing Reward Model Evaluation\", arXiv preprint, 2025.*"}, {"review_id": "gF8izvz3gF", "reviewer": "Reviewer_LCxR", "summary": "This paper seeks to use response times during RLHF to learn a better utility model. The main idea is that response times can be ranked to provide a ranking loss that gives a signal about a user's strength of preference. The paper proposes RtRank that looks at relative response times within different groups (strata) of preference annotations to better model preference strength. Results show that the proposed approach outperforms methods that do not use response time or stratified modeling.", "questions": "Missing related work on learning based on preference strength: Papadimitriou et al.. \"Bayesian Constraint Inference from User Demonstrations Based on Margin-Respecting Preference Models.\" ICRA 2024.\n\nLine 92: How do you ensure that task difficulty is relatively constant? It seems that for any user, the task difficulty will vary by query. \n\nIs it possible to also include a measure of fatigue when learning from response times? For example, what if the metadata for each preference contained the order it was answered or how long the annotator had been working before they supplied a preference label.\n\nIt is not clear from the text why you need an anchor element. Would the proposed approach work as long as the learned utility differences satisfy the relative ranking? \n\nFig 2: How is PCE unaffected by ordinal error? Shouldn't it perform worse with more noisy data\n\nHow do you know the learned rewards are better? Prior work (e.g., Tien et al. \"Causal Confusion and Reward Misidentification in Preference-Based Reward Learning.\" ICLR. 2023.) has shown that accuracy on a held out set is not always indicative of downstream RL performance.", "limitations": "I would have liked to see actual human experiments. It seems that the focus is very much human factors-oriented (focusing on response times, strata, etc) but then all the experiments are synthetic making the merit of the work unclear.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n+Good motivation for learning from response times\n\n+good synthetic experiments that compare with baselines and test ablations, robustness, and sensitivity\n\n+well written paper with nice figures and tables\n\nWeaknesses:\n\n-This paper focuses on human response times but contains no actual human data. \n\n-No justification for the synthetic data creation or why it is a good model of human data\n\n-No results showing down-stream performance using the learned reward models.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns.  I still think doing actual experiments with real human response times is needed to understand the true benefits of this approach, but I agree that the simulated results are promising and I appreciate the new results described in the authors rebuttal. The paper studies an interesting topic and has nice simulation results which is common for NeurIPS so I will keep my score.", "rebuttal": ""}, {"review_id": "VVnwiiUF19", "reviewer": "Reviewer_WFhw", "summary": "This paper introduces RtRank, a method for learning *preference strength* from response times (RTs) in the context of reinforcement learning from human feedback (RLHF). Traditional RLHF relies on binary preferences: user specifies which item is preferred, not by how much. RtRank proposes to incorporate such an additional signal without requiring further annotation (a binary preference is still elicited) by leveraging response time (RT) of the preference elicitation process. The core hypothesis is that low(er) RT corresponds to a stronger preference; because RTs are typically noisy, the authors propose leveraging *relative* RTs within homogeneous strata (e.g., defined by annotator or session) to infer the strength of preferences. The method uses a Plackett–Luce-based model to learn utility differences and introduces a new metric, Pearson Distance Correlation (PDC), designed to measure learned utility magnitude correctness, independent of the preference-prediction correctness. The paper presents empirical results on synthetic datasets under various noise and generation conditions.", "questions": "1. Can you comment on the feasibility and challenges of applying RtRank to real-world datasets? Are you planning or conducting such evaluations? What pitfalls do you expect when scaling RtRank to real-world data?\n2. How sensitive is the method to poor stratification or suboptimal strata sizes? Could automated or adaptive stratification be a viable extension? The backbone of the work is an assumption that correlation between RT and preference-strenght is high in “optimal” stratas; I believe it would strengthen the work if such discussion is included.\n3. Beyond RTs, have you though about other implicit indicators of preference strength? Would it make sense to extend RtRank to take into account multiple indicators (assuming they exist in the dataset)?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- The idea of stratified RT-based ranking for capturing preference strength is well-motivated and novel in this context (as per my current knowledge of the relevant literature).\n- The proposed PDC metric is compelling and, when used in conjunction with additional metrics, may provide more granular insight into the learned models.\n- Preliminary results on synthetic datasets are likewise compelling.\n- Paper is clearly written and easy to follow.\n\n### Weaknesses\n\n- It would be great to see real-world evaluations, though it is understandable that the paper presents a first proof-of-concept.\n- The method’s performance depends on strata quality; deeper analysis of stratification choices would strengthen the work (see also my question below).", "was_revised": true, "final_justification": "The authors have addressed my general questions and concerns with clarifications and new experiments. While these additions improved my understanding of the method, I believe further validation on real-world data is needed to fully establish its impact. I will therefore maintain my score.", "rebuttal": ""}]

Author rebuttal:

Meta-review: This work proposes a new way that takes response time into account to model preference strength. To handle the noisy response time data, this paper introduces RtRank. RtRank leverages relative RT differences to rank pairwise comparisons by their inferred preference strength. The paper provides empirical validations for the proposed method.

The method offers a new perspective for evaluating the strength of a response, which is critical in preference alignment. The idea of using response time is reasonable and the authors also notice the noises therein. The reviewers initially concerns about the lack of experimental results with real-world human response time data, but in the rebuttal, the authors manage to provide some proxy results as a response. 

Meanwhile, the current experiment results have room for improvement as public datasets usually do not perfectly capture response time. Overall, I believe this work brings an interesting direction for future study and it has the potential to further improve preference alignment.
Decision: Accept (poster)