Preference Learning with Response Time: Robust Losses and Guarantees
Authors: Ayush Sawarni
Sahasrajit Sarmasarkar
Vasilis Syrgkanis
Venue: NeurIPS 2025 poster ()
Keywords: Preference learning
Response Time
Evidence Accumulation Drift Diffusion
Neyman Orthogonality
Statistical Efficiency
Primary area: general_machine_learning
TL;DR: We offer statistically robust methods for preference learning that leverage response time in the estimation of rewards to yield large improvements in statistical efficiency.

Abstract: This paper investigates the integration of response time data into human preference learning frameworks for more effective reward model elicitation. While binary preference data has become fundamental in fine-tuning foundation models, generative AI systems, and other large-scale models, the valuable temporal information inherent in user decision-making remains largely unexploited. We propose novel methodologies to incorporate response time information alongside binary choice data, leveraging the Evidence Accumulation Drift Diffusion (EZ) model, under which response time is informative of the preference strength. We develop Neyman-orthogonal loss functions that achieve oracle convergence rates for reward model learning, matching the theoretical optimal rates that would be attained if the expected response times for each query were known a priori. Our theoretical analysis demonstrates that for linear reward functions, conventional preference learning suffers from error rates that scale exponentially with reward magnitude. In contrast, our response time-augmented approach reduces this to polynomial scaling, representing a significant improvement in sample efficiency. We extend these guarantees to non-parametric reward function spaces, establishing convergence properties for more complex, realistic reward models. Our extensive experiments validate our theoretical findings in the context of preference learning over images.

Preference Learning with Response Time: Robust Losses and Guarantees
Ayush Sawarni" Stanford University ayushsaw@stanford.. edu
Sahasrajit Sarmasarkar' Stanford University sahasras@stanfore edu
Vasilis Syrgkanis Stanford University
vsyrgk0stanford edu
Abstract
This paper investigates the integration of response time data into human prefer- ence learning frameworks for more effective reward model elicitation. While binary preference data has become fundamental fine-tuning foundation mod- els, generative AI systems, and other large-scale models, the valuable temporal information inherent user decision-making remains largely unexploited. We propose novel methodologies to incorporate response time information alongside binary choice data, leveraging the Evidence Accumulation Drift Diffusion (EZ) model, under which response time is informative of the preference strength. We develop syyman-orthogona loss functions that achieve oracle convergence rates for reward model learning, matching the theoretical optimal rates that would be attained if the expected response times for each query were known a priori. Our theoretical analysis demonstrates that for linear reward functions, conventional preference learning suffers from error rates that scale exponentially with reward magnitude. contrast, our response augeene approach reduces this polynomial scaling, representing significant improvement in sample efficiency. We extend these guarantees to non-parametric reward function spaces, establishing convergence properties for more complex, realistic reward models. Our extensive set of experiments validate our theoretical findings the context of preference learning over images.
Introduction
Human preference feedback has emerged as crucial resource for training and aligning machine learning models with human values and intentions. Human preference learning
in domains from recommender systems to robotics and natural language processing-typicee solicit binary comparisons between two options and use the chosen option infer user's under- lying utility function [BT52]. Such binary feedback is popular because it is simple, intuitive, and imposes low cognitive load on users - This paradigm underpins wide range of applica- tions, including tuning recommendation engines [XAY23], teaching robots personalized objectives HIS 22) fine-tuning large language models via reinforcement learning from human feed-
back (RLHF) [ RSM 24 ZSW+ 19 22b), and vision model (WDR #24 WSZ+
However, a single sample of binary choice conveys very limited information-iin tells which option is preferred but not how strongly it is preferred (KK19). As result, learning reward functions or preference models from pairwise choices can sample-inefficiae to accurately capture nuanced human preferences. This issue is exacerbated in scenarios where one outcome consistently dominates, yielding nearly deterministic choices and providing minimal insight into the degree of preference [KK19 CCliiaa Clil&b). In practice, reward models often learned either implicitly by integrating the reward estimation directly into the policy optimization, as in
often requiring many queries
Equual contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

[RSM 24), which minimizes a reward-learning objective by plugging in a closed-form expression of the reward in terms of the policy-or or explicitly- by first fitting a separate reward function to preference data and then using that function to fine-tune the policy [K WKH24 BJIN
Direct Preference Optimization (DPO)
TSS+24 24]]. Researchers have explored augmenting binary feedback with more expressive inputs like numerical ratings confidence scores [BJN 22 [BSS22]) but such explicit feedback increases user effort and interface complexity [KA22] JCB 24
One promising implicit signal is the time a human takes to make each choice. Response times are essentially free to collect and do not disrupt the user's experience [C1i18b] AFFN2I] Moreover, a rich literature in psychology and neuroscience suggests strong link between decision response times and the strength of underlying preferences [KK19 Tye In particular. faster decisions tend to indicate clearer or stronger preferences. whereas slower decisions often suggest the person found the options nearly equally preferable [KK19 AFFN2T] This inverse relationship between decision time and preference strength has been documented in cognitive experiments and is quantitatively modeled by drift-diffusion models (DDMs) of decision making [WvdMG07a] PHSO5] DDMs interpret binary choices as the result of an evidence accumulation process: when one option has much higher subjective value, evidence accumulates quickly toward that option, leading to a fast and confident choice; conversely, if the options are nearly tied, the accumulation is slow, resulting in longer deliberation time [WvdMG07al BKM 23 PHS05] However, since DDMs do not admit a tractable differentiable solution for inference, researchers have proposed differentiable approximations to the response time likelihood to make them suitable for Gaussian process regression [SBK24 KLS 23)
Recent work in preference-based linear bandits has shown that integrating response times with choices-using using lightweight EZ-diffusion model from cognitive psychology -leads to substantially more data-efficien learning compared to choice-only approaches [LZR+ 1.ZR+244. These studies confirm that response times can serve as a powerful additional feedback signal. providing valuable information that boosts sample efficiency in inferring human preferences. While the prior works have broken important new ground, their scope has been somewhat limited. [LZR focuses on an active wreference-based linear bandit in which the algorithm controls the query distribution and assumes linear reward function. This setup does not cover more general human-in -n--tee--ttoo settings such as training large generative models- where preference data are often collected passively and the true reward function can be highly complex. In many state-of-the- LLMs alignment of diffusion models). feedback is gathered on model outputs drawn from broad distribution, rather than by actively choosing each query, and the reward model is usually nonlinear, implemented deep neural network.
applications (c.g., RLHF for
Our technical innovation centers on a Neyman-orthogonal loss function that integrates binary choice outcomes with response time observations. Neyman orthogonality ensures that small errors in estimating the response-time model do not bias the reward learning, yielding fast convergence rates. We prove that method achieves the same asymptotic rate oracle that knows the true expected response time for every query, and we demonstrate in experiments on image-preferee benchmarks that it significantly outperforms preference-only basclines.
Our Contributions:
We propose a Neyman-ortiinn orthogonal loss that jointly leverages response-time signals (via the EZ diffusion model) and binary preference data to estimate reward functions. This con- struction (i) enables integration of cognitive DDM insights into standard machine-learning- from-human-fees feeddhack frameworks, and (ii) yields significant empirical and theoretical improvements over classical MLE-based log-loss estimator that uses only preference data.
For linear reward models, we derive asymptotic variance bounds showing that the estimation error of the log-loss estimator scales exponentially with the norm of the true parameter, whereas our orthogonal estimator's s variance grows only linearly. Moreover, unlike the active-query method of [Z)+ which requires carefully designed query selection, our estimator achieves better variance scaling under passive, i.i.d. queries drawn from an unknown distribution.
We extend our analysis to nonparumetric reward classes (e.g. RKHS and neural networks), proving finite-sample convergence bounds in which errors in the response-time estimation enter only as second-order terms.
2

---

We validate our framework on synthetic and semi- synthetic benchmarks. Experiments show that our orthogonal loss consistently outperforms both the log-loss and oonhortho alternatives in estimating linear reward functions, three-layer neural network models, and an image-based preference learning task.
Other Related Work In preference-based learning, the dueling-bandit is common paradigm, where an agent selects a pair of actions each round and observes a binary preference signal [BBKJ12 YIII 22 BBFEMPH21], Treating each action pair as a composite arm yields a closely related logistic-bandit formulation, which has been studied extensively for cumulative regret guarantees [AAC222 7S23 DFET8, SDBS24)
FACF20 AFC21
Our setting departs from these formulations along axes: (i) sampling- we analyze a supervised setting with i.i.d. queries rather than adaptive selection; and (ii) feedback leverage response times addition noisy pairwise comparisons. While [LZR also incorporates response-time feedback, their focus is best-arm identification (BAI), which is typically easier than the supervised reward-estimation problem we study. Moreover, substituting our orthogonal loss into their sequential- elimination algorithm improves the probability of best arm identification (see Appendixx D.I.
GP prior rewards and develops variational approximate Bayesian inference approach using moment matching to refine the posterior to incorporate response time information. In particular, the paper suffers from the standard issues with GP regression and does not scale to high-dimensional (curse of dimensionality in GP regression [BCF17 BW22 GRSH22) and is not suitable for large-scale machine learning models. Our setting is motivated by large-scale "learning from human feedback" pipelines-e.g fine-tuning large language or vision models-where inputs live in very high dimensions and downstream tasks require a pointwise reward estimate. We completely bypass the use of variational approximations and moment-matching techniques as our method avoids modeling the full likelihood altogether.
Recent work by [SLBK24]
2 Notations and Preliminaries
Preference-Larninng Setup We adopt the standard preference-learning framework. On each trial, user is presented with two alternatives, X and X drawn i.i.d. from unknown distribution D. The user then selects one option, which we encode as a binary preference Y E {+1, -1]. and we record the response time T. We further assume that both the preference Y and the response time T' are governed by an underlying reward function r. The learner's 's objective is to learn from the observed data {XX,, x?, Yı, T:))==
Numerous prior works |AA 17 RSM+24 employ the Bradley-Terry model BT52 to link the binary preference Y to the latent reward function r. In this formulation, the probability of selecting alternative X is proportional to exp(r(X') To capture both choice accuracy and response-time variability, we adopt the EZ diffusion model, which extends this log-odds specification by modeling response-time distributions alongside choice probabilities. By design, the EZ diffusion model reduces to the Bradley-Terry model when only choice probabilities are considered. As is common in current learning from human feedback literature [OWJ+222 2112 KWBH24), we assume homogeneity in the samples, i.e., we assume that the reward function uniform across samples and the feature vectors encapsulate any user heterogeneity.
EZ diffusion model treats decision making as a drift-diffusion process with drift r(X²) -x and noise B(T) ~ (O T). After an initial encoding delay fnd. evidence accumulates E'() = μ + (T)) until first reaches one of two symmetric absorbing barriers at +a or -a. The response time T and the preference Y is given by
Given query X X22 the EZ diffusion model [WVDMG07b BKM 23
if E(T a, {-, -1, E(T) =-a.
[1,
T= min(r > 0: E(T) {ba}}
Y Y=
In most applications, one observes the total response time Ind + T, where lnd, captures the non- decision time required to perceive and encode the query. In vision-based preference tasks, 4nd is often treated as a constant [CliT8al YK23], whereas language or more complex cognitive tasks it may depend on the properties of the text [BSH24 VT16). The reward difference r(X') r(x2)
3

---

reflects the strength of preference for a given query. while the barrier a governs the conservativeness inherent to both the task and individual characteristics [VT16]. For simplicity in notation, we use X
to denote the pair (X*, X²) Further, we overload and say r(X) r ( rrx²2) to denote the difference of the rewards from choices X and X². The EZ-diffusion model implies the following
key expressions as computed in [PHS05]:
(tanb(ar(X)) r(X)
if rf + 0, otherwise
P(Y = 1|X)= 1+ exp( - 2a r(X))
E
(1)
a
In applications to machine learning, both and and a may be informed by extensive psychology and economics literature [vR009] CliT8a, W VDDGG07b, BSH24 FNSS220 XCSWC24). For clarity, we treat these parameters as known, fixing a 1, the main text and defer their estimation and uncertainty analysis to Appendix A. In that appendix, we show that when a is unknown one may equivalently learn the scaled reward function r(X)/u with identical guarantees, and that our proposed loss is only second-order sensitive misspecification of Ind-
Note that if we ignore response times and consider only the preferences Y, the EZ-diffusion model reduces exactly to the Bradley-Ters model. Further, combining the choice and timing expressions in (I) (and setting 1) gives convenient identity
X
(X)
T
X
Additional Notations: Now that the model defined, we introduce notation to distinguish true functions from their estimators: write (X( for the true reward-difference and f(X) for its estimate, and similarly (X) == E[T X] with estimate f(X). We further define norms Il (P)) and (lll, (D) as / (Ex~DIf(X) and (x~rllf(X)] respectively. We also define the random variable as the tuple = (X,Y
Preference-Only Learning Estimating the reward function r(-) from binary preferences
{-1,+1) reduces to logistic regression, since P(Y = 1 X) = (1 + compute the maximum-likelibee estimate, or equivalently minimize the logistic loss:
One can
Clogloss (r) = E [log(1 + exp(-2Y rxx)))
(3)
3 Incorporating Response Time
Using the identity in ② natural starting point is the "naive" squared-error loss
quen-ortho (r; t) = (Y-r(X)((X))}
X]. However, this formulation suffers from few serious drawbacks. Estimation of r. is highly sensitive to any error in estimating 4,, Moreover the function often hard estimate even when the reward function r. linear. Second, even if ro is linear, the mapping to (X) sanh(ra(X)) is highly nonlinear and the response time T' does not admit simple ro(X)
where 1(X) estimates (X( E[
closed-form density in terms of r(X). Moreover, 7 is also sensitive to the End assumed the model. For these reasons, the loss in 4 is generally impractical.
The function t(-) acts as a nuisance parameter in the naive loss (4) and any error t(-) directly contaminates the estimate of r.).). To address this, we design modified loss that is Neyman- orthooonal to i, eliminating its first-order effect on the gradient with respect to r. In the next section, review the seyman--rthooonalln conditions and present our orthogonal loss.
3.1 Neymant-rthhoonnality and Orthogonal Statistical learning.
We consider a population loss L(0,g) E[((0. g: Z)], where A is the target parameter and g is nuisance parameter. The loss is said to be Neyman Neyman--rrthoonaal at the true pair (0o, go) if its mixed

---

directional derivative I vanishes:
Dg Do (000.g9)(h, k]
for all directions and k.
(5)
This condition ensures that errors 9o have zero first-order impact on the estimator of 0, so that any error from ! influences estimation of 0, only a higher order (e.g.. quadratic).
3.2 Orthogonal Loss for Preference learning
To prevent errors estimating the decision-tinn iitiioonnteeeee function from biasing the reward estimate r, we define the orthogonalized loss
L Cortho = -X
(6)
where t a preliminary estimator of the true reward function r-(-). In Lemma 3.1 we prove that Cortho satisfies
with respect to the nuisance pair g (t, Crucially, need only be a rough initial estimate (e.g. via logistic loss), since first-order errors in are automatically corrected in Larthe As shown in [Section 5 this yields final estimator for that is robust to substantial nuisance-estimation error.
Lemma 3.1. The population loss Corthe is Neyman-ortoogona with respect to muisance g = (v, i.e. C,rt,oortooog 9r)(r To, g - 9ol =0 Vr E R Vgg E G.
Proof. Let ((.) be the pointwise evaluation of Cortho data point Z (X,Y.T). direct calculation gives
Dg (rr goi X, X,Y,T) - To: g -=)==
-rx)))
rx(())
+2(T16(X) - Lo (x))) - rx)) (r(X) - r (x)))).
E Y X
Because rx(
we have E + T Tro( X] = 0. Similarly, by definition of 4.
lo( X)
E (X)
X
= 0. Taking expectations of the directional derivative, therefore yields
- To. 9 -9ol] 0, E ((r,
which establishes the claimed Neyman-orthogonallin
We present Meta-Algorithm estimate the reward model using nuisance functions e(-) and ((-).
Input: S = ((X!), x T Goal: Estimate reward model f(-)
1: Compute nuisance functions è and as an initial estimate of reward model and response time.
2: Now use these functions (f, f) nuisance to minimize the orthogonalized loss function ootho
Meta-Algorithm 1: Estimate Reward Model via Orthogonal Loss
Different implementations of the Algorrtlg vary in how the nuisance functions t and t are estimated, following [FS23 CNRI& [KSM21]. In data-splitting, the data split into two halves: nuisances are fitted one half, and the orthogonal loss is minimized on the other. Cross-fitting generalizes this to K folds, training nuisances out-of-fold and evaluating on each held-out fold before aggregating. Data-reuse fits both nuisance and target models on the full dataset. In the subsequent sections, we specify which variant used for each theoretical guarantee and empirical experiment.
Furthermore, since the EZ diffusion model implies identities in ( we may plug in t(-) tanb(r(-))
+(-)
directly into Lotho and the loss remains Neyman-orthogons. This plug-in strategy offers further flexibility: one can first train the reward model r (e.g. by minimizing the logistic loss on preference data), then uses the fitted as the nuisance in Cortho to exploit response-time information T for faster
'The directional derivative of - IR at in direction defined by FJ)[[ 4 FF + ( For bivariate functional L(0, 9). write D and D,L indicate differentiation w.r.t. each argument.
5

---

convergence. While our work focuses on reward estimation, this framework also supports DPO-style objectives (4) discussed in Appendix A. Treating the estimation of as black box, our theoretical guarantees hold with only mild second-order corrections; see Appendix C for details.
In the
sections, present theoretical guarantees for Meta-Algorithm 1 In Section 4 focus on linear reward models and state the results that show our orthogonal estimator achieves am exponential improvement estimation error-as function of the reward magnitude--sriclly outperforming the asymptotic rates of the preference-only estimator. In InSecciion 5) we derive finite- sample bounds for general non-linear reward classes, including non-parametric estimators. These bounds essentially recover the oracle rates that is, the rates one would attain if the true average response-tine time function (.) were known and the naive loss _non (non-orthe ortho were used.
4 Asymptotic Rates for Linear Reward Function
We now restrict to the linear class R = {a + (x, 0) : 0 E IR"), that ro(X) = =0,,) and the true reward, (X) = (fo. X). Our goal is to estimate 6..
Preference-only estimator. Let d1og minimize the empirical version the logistic loss in (3). Under the condition that E (-((-2000- X)) (2 (0) x)) xx is invertible, standard argument such as the in [K855 (see Appendix B for derivation) yield
vn (flog - 0o) S s(1,, [4E x)) (2)(0. x)) x)) x
Orthogonal estimator. Let Oortho denote the resulting estimator from Meta- Alggo-Algoritmm (either with cross-fitting data-splittig)..
Theorem 4.1. Let fortho minimize the orthogonal loss Cortho (f) is invertible, then
Vn (Portho 0o) S. 25 (0,E)
where E (am) 2 XXT (
-2
{ (tanb((0..
E
(0o.X)
XXT
Theorem 4. is proven in Appendix B using techniques developed in [CNR18) for asymptotic statis- ties for debiased estimators. Furthermore, the asymptotic variance of fortho is point-wise smaller than that of f2og. and this continues hold for any barrier height This follows from the fact 4 a (2x) < (Lanh(z)) for all æ 2 0 (see Appendix B). Because tauh(z) decays polynomi- ally with |x|| while (=))(-xx) decays exponentially, the variance of the orthogonal estimator is exponentially smaller than the log-loss estimator.
Our asymptotic guarantee differs fundamentally from L.ZR Theorem 3. Li al. establish point-wise convergence for fixed query z the number of observations of that specific query approaches infinity, whereas our result is framed in terms of the overall sample size n accumulated by the learner. Basing the limit rather than per-query counts better reflects how data are gathered practical preference-leaminn scenarios.
Moreover, while this guarantee also covers the setting described in L.R+ +24], the guarantees are sig-
nificantly tighter. In particular the variance of their estimator decays with min xEXummpl
tanh((0 (0,,z)
where Ksample denotes the set of distinct pairs of queries x. In contrast, in our case, this term is inside expectation and thus results in stronger guarantees.
One may naturally ask how our rates depend on the accuracy of the nuisance estimates E(.) and i(-). We prove that the asymptotic guarantees hold, provided
[ - to toll L.()) = (P),
B 2
|| T TolL o(n Rz),
B2
2
3
(8)
For linear rewards, one can achieve the required "slow" convergence rates for estimating by applying kernel ridge regression with Sobolev kernel associated with the RKHS W [AF03] Wai19]

---

Alternatively, one may first via logistic-loss minimization and then set i(X) tauh ( (+(x))
F(X)
Linearity of r and the Lipschitz continuity of the mapping x + tanh(x) then ensure the required slow-rate bounds (see Appendix C).
5 Finite Sample Guarantees for General Reward Functions
We denote the joint nuisance pair as (r.t). In this section we bound the estimation error
|| roll (D)) in terms of the population pseudo-excess risk defined as Cortho (f: (f: - Lortho (ro. 9) plus higher-order terms depending nuisance estimation error. We assumee тo E R and to E T, and
let 9 Rxx denote the joint nuisance class (and 9 E 9). Further let star {X, R x'} == {(1 +
E N &e E [0, 1]}. Further, we adopt the standard Lp norm for functions i.e.
(D) -V (xx)D[((X))],
Il sup 1f(z))
Next we define a pre-norm (R, (need not satisfy triangle inequality) on the nuisance function g(-) to first present a general guarantee in term of this norm for any function class, and then further bound with (D) norn, attaining different rates for different function classes such as RKHS balls and finite- VC subclasses following the general theorem and in Appendix C. [[(())(()(())))))
| sup FER
+
(9)
||
||||| ||
While Neyman orthogonality ensures higher-order dependence on nuisance error, defining the pre- norm this way gives additional robustness with respect to errors in t. The product T(-)t(-) couples the estimation errors of and even if t is poorly estimated- under large reward magnitudes-4 sufficiently accurate estimate t can keep the product of errors small, yielding low nuisance error in the (R. a:) norm.
Theorem 5.1. Suppose i minimizes the orthogonal population loss Cot and satisfies
gortho (f: g) Cortho (ro; () < c(F,g))
Let S be absolute bound on Tor Then the two-stage Meta-Algorithm guarantees
| ro (L)) I
482 e(f, 9) + 48 l1 90
(R.o)'
The proof of the theorem follows by instantiating FS23 Theorem 1]. The details can be found in Appendix C. Now we instantiate the above theorem for specific function classes.
5.1 Nuisance estimation error || goll(R,,)
Let |||læ denote the RKHS norm of r. First, if E R lies in an RKHS ball of bounded radius and with 1-a
a kernel whose cigenvalues decay j j-1/a, then by INNI we have ||r]|oo I C' [r|l ( which in turn implies
||g| (R.)) C /VVlV (lltt(((())l
||(.)² (D))
Second, if we have [ (el))
< C2-11 and |elleyery < C2-44 then setting a = 0 and applying (elleg())
Cauchy-Schwarz gives
| (R))) < C2 [It|] ||]|2z(()) + IT:1l2,
Under these cases, any bound Ç on || (2()) and ||e roll shows up as ( प in reward estimation error.
Remark 5.2. The finite-sample excess-risk bounds in Corollaries 5.3 and 5.4 are obtained via critical-radius argument and Talagrand's concentration inequality, both which require the loss to be
uniformly bounded piint-wise. However, our original (pointwise) orthogonal loss: gortho Z)
(Y - (T (()))(()) -rX)(((()))) ((x))² can be unbounded when the decision time T is unbounded, violating these assumptions.
7

---

To remedy this, in Appendix C we introduce a simple modification where we cap the contribution of any single decision time T at large constant B. This puts additional bias on the &2 error in Theorem 5.1 decaying fast with threshold B. Moreover, in any real-world use case, all the recorded response times would always be bounded; this modeling choice aligns with real-world application of the framework. The effect of capped-loss construction unite-sample rates appear in Appendix C
5.2 Bounding excess-risk e(f.
We consider two nuisance estimation schemes for Meta-Algorithm data-splitting (independent nuisance and target estimation) and data-reuse (joint estimation). We start by defining critical radius of function class and provide the guarantees in terms of the critical radius. Further, let gorho denote the point-wise loss version of population loss Lonbe
Critical radius. Following [ai19], define the localized Rademacher complexity of function class F by
Rad,, (F,0) == Be
sup JEF 1f
Σ fatl)))
The critical radius 6, is the smallest 5 > 0 satisfying (F,5) S 8².
We now present the rates in terms of moment function (C) which an upper bound on moment of E[T< X] assuming that r(X) I S for every Ç <> 1. From [WY08], we know that every çth moment exists. One can choose Ç to get the tightest bound.
Corollary 5.3 (Data-splitting). Let 8n be the critical radius of the star-shaped class
star{ r - ro: r E R,
and define 8th max Eng vc/n} for some constant 0. Then under data-splitting, with probability least - c1 exp exp(-c2n(8#") Meta-Algorithm satisfies
() s 98 Bool IIf rollca(D) + 10 10 (5
for universal constants C1, 02 > O. Consequently for every ( 1,
2 ( M(1)
4
|| | I poly(S) B (80)))++
+ Isa l1@ go (
1+0
(Bs--)
holds with the same probability.
For the case of data-reuse we have, Corollary 5.4 (Data-reuse). Let 8n be the critical radius of
starf gorchorr, gi -) (ortho gi -) E R, gEG),
and define ddr = max{ on. Ve/n} for some constant > Then under data- data-reuse, with probabil- ity at least e1 exp( exp(-e2n(0p) (2 ()))))) Meta-Algorithhm satisfies
c(f, g) I 9 8 ||F TollLa(D) + 10 (6dr))
for universal constants C1, 02 > O. Consequently for every Ç 2 1,
|| ( S poly(S) (odr)2 +
M(S)
4. 4.
l1 9o
4 1+a
CBS-
(R.))
holds with the same probability.
The full proof of the critical-radius bounds and further discussion appear in Appendix d Our analysis follows [Wai19] Theorem 14.20), leveraging uniform law for Lipschitz loss functions.
The critical radius differs between data-splitting and data-reuse: with data reuse, each observation Z (X, Y) influences both the reward model r(-) and the nuisance g(-), creating conditional dependence that increases the critical radius and slows convergence, whereas under data splitting the two estimates remain independent, yielding a smaller radius and faster rates.
Above bounds in Corollary and Corollary 5.3 are loose in S and can be tightened via careful analysis.
8

---

Error vs. Scaling (d 5, n 8000)
Error vs. Scaling = 10, n 8000)
Log Loss
Log Loss
Ortho Loss
Ortho Loss
Non-ortho Loss
Non-ortho Loss
Aug Emor
12
Reward Scale |18,112
Reward Scale ||0,11z
Figure 1: Performance the linear-reward model as the true parameter magnitude ||0oll varies. Left: d = 5; right: d = 10.
Experiments
We evaluate three settings: (a) linear reward models, (b) neural network- parworr-parrmmttriieedd rewards, and (c) semi-synthetic text-to-image preference dataset. In addition, (Appendix 1.. applies our orthogonal loss within the sequential-clin agquental-elimmiation algorithm of (LZR 24 for best-arm identification, where it outperforms their algorithms/
I Linear rewards. We evaluate synthetic data where each query pair (X1 X²) is drawn uniformly Preferences Y
from the unit-radius sphere and the true reward ro(X) = (0,, X) with ||0/]|2 and response times T are generated via the EZ. diffusion model. For each B, we draw 0, at random, generate independent datasets, and by minimizing the logistic loss, non-orthogonal loss and orthogonal loss. Further to estimate the nuisance parameter r(-) and t(.) for orthogonal and non-orthogonal loss, we use logistic regression and 3 layered neural network respectively. We report the average error ||0 - 0,/12 as we vary in Figure 1 Full experimental details are provided in Appendix D. We observe that the estimation error under the logistic loss grows rapidly with B and the orthogonal loss Cortho consistently outperforms the other two losses.
Non-linear rewards-neut neural networks We generate synthetic data from random three-layer neural networks with sigmoid activations, fixed input dimension, and hidden-layer widths. For each training size N, sample new network (details in Appendix D) the true reward model and draw N query pairs X,, X2 uniformly from the unit sphere. We evaluate all three losses-logistic, non-orthogonal, and orthogonal-ann and for the orthogonal loss compare both simple data-split implementation and data-reuse implementation. Figure reeeporss the mean squared error (and + standard error) of the estimated reward under each of the three loss functions and the corresponding policy regret after thresholding into binary decision. Regret measures the gap between the learned policy and the optimal binary policy. Denoting by X; the option selected by our policy for query i, the regret over M new queries is
M Regret (max
X
f(X)) 2 f(X3),
max max((ro(X)) (xxx ro(
x,
X? otherwise.
Text-to-image preference learning. We evaluate approach
real-world text-to-image
preference dataset Pick-a-pick [3)S= which contains an approx 500k text-to-image dataset generated from several diffusion models. Furthermore, we use the PickScore model [KPS+2] as an oracle reward function, we simulate binary preferences Y E [+1,-17 and response times T via the EZ-diffusion process conditioned on the PickScore difference between each image-test pair. To learn the reward model we extract J024-dimenlional embeddings from both the text prompt and the generated image using the CLIP model 2 On top of these embeddings, a 4-layered feed-forward neural network with hidden layers of sizes 1024, 252, under three training objectives: proposed orthogonal loss, a non-orthogonal response-time loss, and the standard log-loss on binary preferences. Further experiment details are available in Appendix D. As
train
'The experiment code is available in
Preference- Lew.eerreeee---Lennnnn with- Recch--esposse-2111

---

Mean Squared Error vs Training Size
Regret vs Training Size
1800
Loss
Mean Squared Enror
30-
Citho Loss with spitting
orth
1600
Non-ortho Loss
with Salkting
Regret
1.400
12-
1000
8-
800
2500
5000
7500
12500 15000 17500 Training Size
10000
20000
2500
5000
7500
10000
12500 15000 17500 20000
Training Size
Figure 2: Left: mqaa-e error (+ standard error); right: cumulative regret (+ standard error) over M 3000 new queries randomly sampled non-linear (neural network) reward models, both plotted against training-set size N.
MSE Training Size
Regret Training Size
log
3.5
Ortho Loos
1600
Mean Squared Error
ortho
Non-orthe
3.0
1400
2.5
1200
2.5
1000
0.5
000
200000 300030 Number training samples
100000
300000
400000
500000
Number training samples
Figure 3: Left: mean-squared error (+ standard deviation); right: cumulative regret (+ standard deviation) over M 10000 new queries on the Pick-a-Pic text-to-image task, both plotted against training-set size N.
shown in Figure 3] the orthogonal loss consistently achieves significantly lower Mean squared error and regret compared to the non-orthogonal and log-loss basclines.
7 Conclusion and future work
Our work proposes a Neyman-orthogonal loss function that jointly learns over the preferences and response time to estimate reward functions. We show that it better estimates the reward model theoretically and empirically in semi-synthetic setups. Possible future directions might include:
Dueling bandits model an online setting where the learner queries arm pairs and observes a noisy binary preference. Extending this framework to incorporate response time as an auxiliary feedback signal-and adapting our supervised loss such adaptive querying for regret minimisation -peesents an interesting direction for future work.
Extension to bandit setup
Experiments with true response time data : Our experiments assume a homogeneous reward model and synthetically generated response times. Evaluating real-world response time data, which may be noisy and population-depender (with varying barriers a and true rewards r-).)), could provide valuable insights into model robustness.
DPO-style loss [RSM 24
Beyond reward estimation, our framework can be extended to direct policy learning via a DPO-style objective by replacing the reward model with a policy parameteriza- tion. Adapting the orthogonal loss Corth for this purpose is promising avenue (see Appendix A.6).
10

---

Acknowledgments
Vasilis Syrgkanis is supported by NSF Award IIS-2337916. Ayush Sawarni partially supported by NSF Award IIS-2337916.
References
[AF03] Robert A. Adams and John J. Fournier. Sobolev Spaces, volume 140 of Pure and Applied Mathematics (Amsterdam). Elsevier/Academic Press, Amsterdam, second edition, 2003.
[AFC21] Marc Abeille, Louis Faury, and Clement Calauzenes. Instance-wise minimax- optimal algorithms for logistic bandits. In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 3691-3699. PMLR, 13-15 Apr 2021.
[AFFN21] Carlos Alós-Ferrer, Ernst Fehr, and Noam Netzer. Time will tell: Recovering preferences when choices are noisy. Journal of Political Economy, 129(6):182)) 1828- 1877, 2021.
[BBFEMPH21] Viktor Bengs, Róbert Busa-Fekete, Adil EI Hesaoudi-Paul and Eyke Hüllermeier. Preference-basee online learning with dueling bandits: a survey. J. Mach. Learn. Res., 22(1), January 2021.
[BJN+22] Harrison Bai, Andy Jones, Juliana Ndousse, Amanda Askell, Yuntao Chen, and et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.08 05862, 2022.
[BKM+23] Renato Berlinghieri, Ian Krajbich, Fabio Maccheroni, Massimo Marinacci, and Marco Pirazzini. Measuring utility with diffusion models. Science Advances, 9(34):ead11665, 2023.
[BSH24] Aline Bompas, Petroc Sumner, and Craig Hedge. Non-decision time: The higgs boson of decision. Psychological Review, 2024.
[BT52] Ralph A. Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4)3324 345, 1952.
[BW22] Mickaël Binois and Nathan Wycoff. A survey on high-dimensional gaussian process modeling with application to bayesian optimization. ACM Trans. Evol. Learn. Optim., 2(2), aug 2022.
[CLB+17] Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Proceedings of the 31 International Conference on Neural Information Processing Systems, NIPS' 17, page
Red Hook, NY, USA, 2017. Curran Associates Inc.
[Cli18a] John A. Clithero. Improving out-of-sample predictions using response times and a model of the decision process. Journal of Economic Behavior & Organization, 148:344-3755 2018.
[Clil8b] John A. Clithero. Response times in economics: Looking through the lens of sequential sampling models. Journal of Economic Psychology, 69:61-86, 2018.
[CNR18] Victor Chemozzukov, Whitney K Newey, and James Robins. Double/de-biased machine learning using regularized riesz representers. Technical report, cemmap working paper, 2018.
(CNSS24) Victor Chernozhukov, Whitney Newey, Rahul Singh, and Vasilis Syrgkanis. Adver- sarial estimation of riesz representers, 2024.

---

[DFE18] Bianca Dumitrascu, Karen Feng, and Barbara E. Engelhardt. Pg-ts: improved thompson sampling for logistic contextual bandits. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, page 6629-46335 Red Hook, NY, USA, 2018. Curran Associates Inc.
[DKSM21] Tri Dao, Govinda M Kamath, Vasilis Syrgkanis, and Lester Mackey, Knowledge distillation as semiparametric inference. arXiv preprint (rXxx:2104.09732, 2021.
[FACF20] Louis Faury, Mare Abeille, Clement Calauzenes, and Olivier Fercoq. Improved optimistic algorithms for logistic bandits. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5052-3060. PMLR, 13-18 Jul 2020.
[FAJC22]
Louis Faury, Mare Abeille, Kwang-Sung Jun, and Clement Calauzenes. Jointly efficient and optimal algorithms for logistic bandits. In Gustau Camps-Valls, Fran- cisco J. R. Ruiz, and Isabel Valera, editors, Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 546-580 PMLR, 28-30 30 Mar 2022.
[FK85] Ludwig Fahrmeir and Heinz Kaufmann. Consistency and asymptotic normality of the maximum likelihood estimator in generalized linear models. The Annals of Statistics, 13(1):342-366 1985.
[FNSS20] Drew Fudenberg, Whitney Newey, Philipp Strack, and Tomasz Strzalecki. Test- ing the drift-diffusion model. Proceedings of the National Academy of Sciences, 117652)33141-33144 2020.
[FS23] Dylan Foster and Vasilis Syrgkanis. Orthogonal statistical learning. The Annals of Statistics, 51(3):879-908, 2023.
[GADGP+24] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm understand learning from human preferences. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pages 4447-4455. PMLR, 02-04 May 2024.
[GRSH22] Matteo Giordano, Kolyan Ray, and Johannes Schmidt- Hieber. On the inability of gaussian process regression to optimally learn compositional functions. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22 Red Hook, NY, USA, 2022. Curran Associates Inc.
[Ham74] Frank R. Hampel. The influence curve and its role in robust estimation. Journal of the American Statistical Association, 69(34)):383-39,, 1974.
[HIS23] Donald Joseph Hejna III and Dorsa Sadigh. Few-shot preference learning for human-in-the-loop rl. In Conference Robot Learning, pages 2014-2025, PMLR, 2023.
[Hubll] Peter J. Huber. Robust statistics. In Miodrag Lovric, editor, International Encyclo- pedia of Statistical Science, pages 1251.251. Springer Berlin Heidelberg, Berlin, Heidelberg, 2011.
[JCB+24] Ruili Jiang, Kehai Chen, Xuefeng Bai, Zhixuan He, Juntao Li, Muyun Yang. Tiejun Zhao, Liqiang Nie, and Min Zhang. A survey on human preference learning for large language models. arXiv preprint arXiv:2406.1199) 2024.
[KAS21] Pallavi Koppol, Henny Admoni, and Reid Simmons. Interaction considerations in learning from humans. In IJCAI, pages 283-291, 2021.
12

---

[KBCF17] Karl Krauth, Edwin V. Bonilla, Kurt Cutajar, and Maurizio Filippone. Autogp: Ex- ploring the capabillities and limitations of gaussian process models. Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence (UAI 2017), page- AUAI Press, 2017.
[KK19] Arkady Konovalov and Ian Krajbich. Revealed strength of preference: Inference from response times. Judgment and Decision making, (4(4):381-394, 2019.
[3] Stephen Keeley, Benjamin Letham, Craig Sanders, Chase Tymms, and Michael Shvartsman. A semi-parametrie model for decision making in high-dimensional sensory discrimination tasks. In Proceedings of the Thirty-Sevnnth AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in 23/
Artificial Intelligence,
AAAI Press, 2023.
[KPS+23] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-imaae generation. Advances in Neural Information Processing Systems, 36:36652-36663, 2023.
[KBBH24] Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback, 2024.
[4] Shen Li, Yuyang Zhang, Zhaolin Ren, Claire Liang, Na Li, and Julie A. Shah. Enhancing preference-based linear bandits via human response time. In Advances in Neural Information Processing Systems (NeurIPS), 2024. Preprint.
[MN10] Shahar Mendelson and Joseph Neeman. Regularization in kernel learning. The Annals of Statistics, 38(1), February 2010.
[Mo666] Jürgen Moser. A rapidly convergent iteration method and non-linear partial differen- tial equations Annali della Scuola Normale Superiore di Pisa Scienze Fisiche e Matematiche, 20(2):265-3 -35, 1966.
[2a]+22a] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schul- man, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc
[OW]=22b] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744 2022.
[PHSO5] John Palmer, Alexander C. Huk, and Michael N. Shadlen. The effect of stimulus strength the speed and accuracy of perceptual decision. Journal of Vision, 1-5) 05 2005.
[RKH*21] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sand- hini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Interna- tional conference on machine learning, pages
PmLR, 2021.
[RSM+24] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model, 2024.
[SDBS24] Ayush Sawarni, Nirjhar Das, Siddharth Barman, and Gaurav Sinha. Generalized linear bandits with limited adaptivity. In A. Globerson, L. Mackey, D. Belgrave,
13

---

A. Fan, U. Paquet. J. Tomczak, and C. Zhang, editors, Advances in Neural Infor- mation Processing Systems, volume 37, pages 0369.8369. Curran Associates, Inc., 2024.
[SK18] Stephanie M Smith and lan Krajbich. Attention and choice across domains. Journal of Experimental Psychology: General, 147(12))1810, 2018.
[SLBK24] Michael Shvartsman, Benjamin Letham, Eytan Bakshy, and Stephen Keeley. Re- sponse time improves gaussian process models for perception and preferences. In Proceedings of the Fortieth Conference on Uncertainty Artificial Intelligence, UAI '24 JMIR.org. 2024.
[TSS+24) Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine- tuning of llms should leverage suboptimal, on-policy data, 2024.
[Tye79] Tyzoon T. Tyebjee. Response latency: A new measure for scaling brand preference. Journal of Marketing Research, 16(1):96-101, 1979.
[vRO09] Don van Ravenzwaaij and Klaus Oberauer. How to use the diffusion model: Param- eter recovery of three methods: Ez, fast-dm, and dmat, Journal of Matheraatiaal Psychology, 53(6) 2009.
[VT16] Stijn Verdonck and Francis Tuerlinekx. Factoring out nondecision time in choice reaction time data. Psychological Review, 123(2):208 227, 2016.
[Wai19] Martin J. Wainwright. High-Dimensional Statistics: Non-Asymptotic Viewpoint. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge Univer- sity Press, 2019.
[WBSS22] Nils Wilde, Erdem Biyik, Dorsa Sadigh, and Stephen L. Smith. Learning reward functions from scale feedback. In Conference on Robot Learning, pages 353-362. PMLR, 2022.
[WDR+24] Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2238, 2024.
[WLH+22] Xiaofei Wang, Kimin Lee, Kourosh Hakhamaneshi, Pieter Abbeel, and Michael Laskin. Skill preferences: Learning to extract and execute robotic skills from human feedback. In Conference on Robot Learning, pages 1259-1268. PMLR, 2022.
[WSZ+23] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao and Hongsheng Li. Human preference score: Better aligning text-to-image models with human preference. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2096-2105, 2023.
[WvdMG07a] Eric-Jan Wagenmakers, Han van der Maas, and Raoul Grasman. EZ-diffusion model for response time and accuracy. Psychonomic Bulletin & Review, 14(1):3-24, 2007.
[WVDMG07b] Eric-Jan Wagenmakers, Han L. J. Van Der Maas, and Raoul P. P.P. Grasman. An EZ- diffusion model for response time and accuracy. Psychonomic Bulletin & Review, 14(1):3-22, February 2007.
[WY08] Huiqing Wang and Chuancun Yin. Moments of the first passage time of one- dimensional diffusion with two-sided barriers. Statistics and Probability Letters, 78(18):3373-333 2008
[XAY23] Wanqi Xue, An, and Shuicheng Yan. Prefrec: Recommender systems with human preferences. In KDD, 2023.
14

---

[XCSWC24] Khai Xiang Chiong, Matthew Shum, Ryan Webb, and Richard Chen. Combining choice and response time data: A drift-diffusion model of mobile advertisements. Management Science, 228112
2 2024.
[YBKJ12] Yisong Yue, Josef Broder, Robert Kleinberg, and Thorsten Joachims. The k-armed dueling bandits problem. Journal of Computer and System Sciences, 78(5):1538 1556, 2012. JCSS Special Issue: Cloud Computing 2011.
[YJ1]] Yisong Yue and Thorsten Joachims. Beat the mean bandit. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML'11, 11, page 241-248, Madison, WI, USA, 2011. Omnipress.
[YK23] Xiaozhi Yang and Ian Krajbich. dynamic computational model of gaze and choice in multi-attribute decisions. Psychological Review, 130(1):52, 2023.
[2 Xinyi Yan, Chengxi Luo, Charles I., A. Clarke, Nick Craswell, Ellen M. Voorhees, and Pablo Castells. Human preferences as dueling bandits. In Proceedings of the 45th International ACM SIGIR Conference Research and Development in Information Retrieval, SIGIR '22, page 567-577, New York, NY, USA, 2022. Association for Computing Machinery.
[2JJ24] Banghua Zhu, Jiantao Jiao, and Michael I. Jordan. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons, 2024.
[ZS23] Yu-Jie Zhang and Masashi Sugiyama. Online (multinomial) logistic bandit: Im- proved regret and constant computation cost. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 29741-29782. Curran Associates, Inc., 2023.
[7SW+19] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint
2019.
15

---

A Discussion on Barrier a and Non-Decision Time End
A.1 Properties of the EZ Diffusion Model
Recall the EZ diffusion model for decision between two options X1, with drift = r
r(X2) and symmetric absorbing barriers at =a. Writing X (X1, X2) and overloading r(X) =
r(X1) (X2), the choice and response-time moments are given by [WvdMG07a]:
P(Y = 1 xx = 1 --r())))))
E XX] = tanh (ar())r())), Var(Y |X) = 1 t--aannn(o0 (ar(x))
a(etan(X)) -1
-aaa(()))2aa -
a tanh (ann(ar(())
r(X) * 0,
E x]
r(X)
r(x) #0, r(X) = 0,
Var (x) =
r(X)3 (c2ar(x) + 1)
a²,
2a
r(X) = 0.
3
A.2 Loss Functions for General Barrier
When a is known, a natural non-orthogonal baseline loss is
nnnn artho (r: () E. {<(( x(
where t(X) ve B[T X]. This also results in the following orthogonal loss
Cortho (r; t, () E (Y (T (()) [(x)
x(X)
(10)
a
where r(.) crude estimate of the reward function r-) (which can be calculated by minimizing the loss using the log loss) specified below
Clogloas (r) E log({1 + exp( (xrr())))
(11)
A.3 Reward Learning When Is Unknown
If the barrier a is unknown, one can still estimate the scaled reward r(-)/a with identical guarantees. Note that this suffices for any standard machine learning tasks involving learning from human feedback. Indeed, minimizing the log-loss (TT) yields an estimate of a rr(()), and hence of ro(.) up to scale. However, the loss (10) cannot be applied directly because it requires the nuisance v(-)/a, which is not identifiable without knowing a.
Instead, introduce new nuisance pair (y.t), where y(X) approximates E[Y | X]. We also define the notation yo (X) = TE X]. One can then define an alternative orthogonal loss,
ortt 2 (r: y, t) = E (V (T t(X))
x(x) [x)
+(x))(X) )
(12)
Equation (12) is also Neyman-orthogonal with respect of (y. () as we show in the lemma below 3.1 Moreover, the error rate guarantees for and 12 are identical for the linear case, as shown in Appendix B
Lemma A.1. The population loss aatth is Neyman with respect to muisance g = (y, t) Lee. Dg Dy cortho-- go)(r To:9 9 go]
Vr E R Vg EG
Proof. Let ((-) be the pointwise evaluation of ootth 2 at a data point Z = (X,Y,T). A direct calculation gives
-3
-2
(Y + vx()) -2 2- rx(X)) (X)) (t(x) (xx)))((()) ro(X))
(.D,,((rg.:..))) ro: 9 go]
2
+ (T- - tx(x)) - no (r(X) - - (X)).
16

---

Because [a(X] B|- |X]
have E Y +y,(X) - 2 Lo (X) x 0. Similarly, by
(X)
definition of o. E.T - lo(( X] 0. Taking expectations of the directional derivative, therefore yields
E
9 -9ol 0,
which establishes the claimed beyman-orthogomme
A.4
Experiment varying
We conduct experiments for varying barrier levels for the case where the barrier a is unknown. Similar the synthetic neural network experiment the main paper, we generate synthetic data from random three-layer neural networks with sigmoid activations, fixed input dimension (= 10), and hidden-layer widths (32, 16). We sample random neural network (details Appendix D) as true reward model and draw 2000 query pairs X1, X2 sampled from spherical guassian distribution. We evaluate all three losses logistic, orthooohog,nl,, and orthogonal Larthe for different values of barrier a. We find that the mean squared error of the estimated reward under each of the three loss functions and the corresponding policy regret (after thresholding into a binary decision) is consistently better for Cortho-2
Table Mean squared error for different losses across barrier values
Log-loss Clogloss
Non-orthogonal Lnon orthe
Orthogonal Cortho-2
Barrier
0.5
13.1047
10.3015
9,2564
0.7
14.8284
10.8872
9,3922
0.9
16. 1897
11.2413
9,2729
1.1
17.0966
11,7575
9.5093
1.3
17.8099
11.8773
9.6774
1.5
18.4114
12.3013
9.7517
1.7
18.9334
12.4945
9,8271
1.9
19.2903
12.6231
9,8571
2.1
19.6200
12.8721
9,8152
2.3
19.8787
13.1132
9.9202
2.5
20. 1948
13.1526
10.1428
A.5 Second order dependence in errors Ind
Theorem 5.1 directly implies the following corollary when lnd is misspecified Corollary A.2. Let End be any estimate of the non-decision time satisfying
End - tnd
and nuisance {(c) estimated using mis- sii--ppe decision times by
T Trotal - -Tnd
and let ple) be the reward estimator obtained by replacing T. with T in the orthogonal loss. Then under the same conditions as Theorem 5.1
|p(e)
L2(D) = Of(etin + on()),
where (1) the estimation-error from Theorem Theorem 5.1 and in that theorem.
*Recall that the regret over M new queries is
M
X2 f(X?) 2 F(X?),
Regret
max(ro(XX), ro(x3)) ro(xi)).
X,
x? otherwise.
17

---

Table 2: Regret for M 2000 queries for different losses across different barrier values
Log-loss Clogloas
Non-orthogonal Lnon arthe
Orthogonal Cortho
Barrier
0.5
0.2980
0.2537
0.2538
0.7
0.3437
0,2987
0.2971
0.9
0.3635
0,3000
0.3016
1.1
0.3732
0,3152
0.3044
1.3
0.3686
0.3142
0.3120
0.3807 1.5
0.3148
0.3115
1.7
0.3787
0.3155
0.3105
0.3751 1,9
0.3216
0.3037
2.1
0.3750
0.3198
0,3082
2.3
0.3772
0.3203
0.3028
2.5
0.3763
0.3171
0.3085
Proof. Apply Theorem
5.1
with the nuisance pair 9 (t, 80)). Observe that:
1. The misspecification And - Lnal induces at most O(e) error in the estimated nuisance function t(X).
2. By Neyman-orthogonalliy, the orthogonal loss incurs only higher-order dependence any nuisance error; in particular, perturbation of order in i contributes O(E*) to the final reward-estimation error.
3. The debiasing term (T- t(X)) unaffected by the shift in non-decision time, since fad cancels in the difference.
Moreover, can write
|eTOO)L,, |||||
|ا (1-ool(RR.)) (1) +C sup FR
(13)
1-a L2
where depends only on the uniform bound S ||r| Combining these observations with the (1) estimation rate from Theorem 5.1 yields the stated bound.
L
A.6 Estimating the nuisance / via plug-in of
convenient way to estimate 1(.) to exploit the EZ-diffusion identity
((X) tanh( (t())
t(X)
Substituting this directly into the orthogonal loss ortho preserves Seyyman-orthogonalee Concretely, one proceeds in two stages:
1. Fit a preliminary reward model i (for example by minimizing the logistic loss on preference data).
2. Define the plug-in nuisance
tanh(t(XX)
f(x)
E(X)
and minimize the orthogonalized squared loss with è as the nuisance
(x Tx(X) + tanl(((())) - r(X) (anh(t(X))
27
(r:
(14)
r(X)
18

---

This plug-in in strategy retains the first-order robustness to errors in and yields faster convergence by leveraging response-time information in the second stage.
Hence, one can extend this idea to directly learn a policy on preference data. In Direct Preference Optimization (DPO) [RSM 22), one obtains the reward function in closed form from learned policy π.
- - c
=c(108 log
log (x²²) ( (Te((X2)
T(X))
where Fref is the reference policy and c is a constant. We can embed these into the two-stage procedure: 1) Train क by minimizing the logistic-loss (DPO-loss) on the observed preferences,
a(X) yielding r(X) = log [x] 2) Plug into (14). Since r itself is a known function of π, this directly learns new policy that incorporates response time, Adapting Corth for directly learning policy from preference data can be an interesting direction for future work.
19

---

B Asymptotic guarantee for the linear reward model classes
B.1 Guarantees for Clogloss
We now give asymptotic guarantees on the convergence of the choice-only estimator using the idea of influence functions [Hub[]] H4)7 For the choice-only estimator, the logistic regression loss function E(O, X, Y) = log (a(2a) X))) as described in (11) The informal lemma shows asymptotic results on the estimator @ minimize the loss function Y.
iid dataset. Further, we assume that this dataset is generated using a model parametrized by fu- i.e. 20 ((0: X, Y)] = 0. Further, we often overload |1.1|2 with (D) norm when operated on function formally, ||6||2 Bx [((()))]].
Lemma B.1. Define the IF(O, YY == (B 18 (0,, 100,, Let the estima-
for in minimize the loss function E(.) over the dataset {X; Yi) Under standard regularity assumptions,
vū((0n - 06) 4 NO0 Var(IF(0n, X, X)))))
We now compute the influence function for logistic losses E(0, X,,) log (a (2aY (0, X))). a
Now, observe that 80 ((0,X,Y) = 2o (0, X)) XY. Further, 802 {10, X, Y)
4a"o (2aY (0, X)) (-2aY (0, X)) XX*. This argument is fairly standard and we restate the derivation below for expository purposes.
Since
E
Dg ((o, X)
0, the variance of 86
((0, X,Y)
equals
E
l(0o. X, Y) 800
2
8 ((oo. X, 1)) 800
Now,
Var
E(oo. (1fo,X,Y)) x,))
4a ABa (-22 X)) XXx1
78
2a ( 20 (0) x)) (2a/00, X))
(15)
+ a (2a x)) (2a x)) σ 20, x)) xxx]
(16)
4a [ (-2a X)) σ (2a X))xx"
(17)
(a) follows via conditioning X and the fact that P from the fact that afr) + +--r) = Using similar arguments, one show that
=1 1) o (2a8 (0., X)) (b) follows
82 ((0g, x, x,y) 4a"E [or (-2a(0o. ()) (2),
(18)
0g2
Since Var(IF X)))) = (E
82
e (00x -2 Var
E(0n, x, x.y)). applying Lemma B. 300
a
dzz
we have
Lemma B.2.
"1) where the estimator êch denotes the estimator obtained from logistic regression on the preference data.
Vn(lCcH - 0o) S NO, (4a2E lo (- x)) o (2,, X))xX"1)
Since (-r) scales asymptotically as exp(-Irl), one can above that the variance term can scale exponentially in S asymptotically for larger norms of 6g where S denotes the &z bound on 0g.
'The constant 2a follows from the probability P(Y XX) in
20

---

B.2 Computing Rates for Orthogonal Loss
6 and 12
We first restate the notations from Appendix
this section, we consider the linear class
{z + (x, 8) 0 [R so that re (X)) (0. X) and the true reward model ro(.X) (f.. X) and the goal is to estimate 8. We further assume that the lz norm of queries satisfies ||ا|| S Further recall that Z denotes the tuple (XYYT))
Recall from Section 4 that the asymptotic rates for linear function classes holds under cross-fitting and data-splitiing We now restate cross-fitting from [CNRI8] for expository purposes which involves training nuisances out-of-fold and evaluating on each held-out fold before aggregating. Further denote gurtho and gortho-2 as the point-wise versions of orthogonal losses Cortho and Lortho-
While state cross- fooss-fitting under the orthogonal loss _ortho loss gatt with the only difference in the nuisance functions computed.
easily state it for the orthogonal
Input: S ((x.)) x!2), YE, T.))
Goal: Estimate reward model r(-)
1: Partition the training data into B equally sized folds (call it {Sp}p=))
2: For each fold S,, estimate the nuisance reward function ((-)()) and time function ((-)(p) using the out of fold data points.
3:
Use t(-) and f(-) as nuisance to estimate the reward model r(-) using an orthogonal loss by minimising Ep= Eies, E(r; (()); ; Z,) over observed data points Z; = (X,, T,) for
orthogonal loss gortho
Meta-Algorithe 2: Estimate Reward Model via Orthogonal Loss and cross-fitting
Under data-reuse the nuisance is fitted on one half of the data and the orthogonalized loss is minimized on the other.
Before we present the asymptotic rates for linear reward classes for orthogonal losses. we first state the following theorem from [CNRI8 Section 3.2]. Recall that g(-) the nuisance function, given by the tuple (v()). t(-)) under the orthogonal loss Lortho-2 and by the tuple (r(-), f(-)) under the orthogonal loss Cortho
B.2.1 Moment function and Neyman orthogonality setup from [CNR18) Section 3]
Further, following the notation in Section 5 we define the nuisance function class by G and we also assume that the nuisance estimates also belong to this class.
Define the sample moment function
m(0.g.2) j(g Z)0 + o(g.2).
and the associated population moments
M(0,g) = (m((0,g,))
J(g) = E[j(g, 2)],
V(g) = Beu(g, 2)].
Let fg be the (unique) parameter solving the population moment condition
M(fo. 0,
where 9o denotes the true nuisance function.
We refer to [g. 2) as the sample Jacobian matrix and to its expectation ((g) as the population Jacobian matrix.
To map to our set up where we minimize the pointwise loss gott (0. g. Z) or gortho-2 20, g. Z), we compute the function m(.) by taking the gradient with respect the first component of the loss function.
We now state assumption 3.1 from [NR18
Because we focus on the moment function m(-). Neyman orthogonality is defined solely with respect to the nuisance function, whereas for loss function É(. it is defined with respect to both the nuisance function and the target parameter.
21

---

Assumption B.3 (Neyman Orthogonality and continuity). The directional derivative D (M (9 gol equals zero (satisfying Neymann orthogonality) and further, gg M g)gg 9ol is continuous in small neighborhood of Jo- Further, the eigen values of the Jacobian matrix J (go) lie between constants 6₁ and C2.
We now let cn eı 0,8 and q > 2 be some finite constants such that co C1; and let 6, be some sequences of positive constants converging to zero such that 6n 2 n 1/11 Recall that 9 denotes the nuisance parameters estimated from the first stage.
Assumption B.4 (Score Regularity and Quality of Nuisance Estimates). The following moment conditions hold:
The 4th order moment conditions hold i.e. suPgeg [B[]m((0..222)111) and
supgeg (Elliia, 2)1 are bounded by constant c.
The Jacobian J-) satisfies [|](ğ) J(go)) S òn where 11-1 denotes the operator norm.
The moment function satisfies (1)m((0: 0, 7) m(00. Po: 2) 2) 1/2 8n-
The second order directional derivative in direction of the nuisance error converges zero faster than n- 1/2 1/2 i.e. M Ø) go] (1), Vg Tğ 7)9o- [0, 1].
The variance [m( 9o. (). Io. _ is lower bounded.
Under these Assumption B.3 and Assumption B.4 the following lemma holds on the estimate Ô under data-splitting and cross fitting
Lemma B.5. Under these Assumption B.3 and Assumption B.4 the estimate â is asymptotically linear under data-spii spliitiiiiiing and cross-fitting.:
vn (0- ( 00)
1 (1o) ( 90. 2) + op(1),
This implies that the following convergence holds distribution vn( 1o) 4
N(0, -(o 2 9o: (00 90: T)
With this setup, we now compute the guarantees for orthogonal loss Lortho and 2ortho-2
B.3 Asymptotic Rates for orthogonal loss Cortho
In this notation É(.) refers to the pointwise version of orthogonalized loss Cortho Recall that the nuisance function g(-) denotes the tuple of nuisance functions (r, t) with go(-) (r.. to) denoting their true values. Further the nuisance function 9 (t, f) denotes an estimate of true nuisance function (.).) = (ro. to) after the first stage.
We adopt the following notation. Recall that O, Rd denoted the true parameter vector. We define the moment function by
m(6 y. U. = 1
which, after algebraic manipulation, can be written as
m(0, t, ( x()) +(X)(()) (T
x.
addition, the two auxiliary functions can be defined as:
j(t: X)= ((x)² a²
xx" and 2(t, X, =1/a) (-ta)()t((()++++++) (1)0()))))))))) r(X)((X) (T-((X))) (19)
22

---

1 B1. We also compute the expectation based mappings:
(8, = B m(....t....
J(t) = a aB [(x)* xx*
(20)
t(c,t) = = ((to((((tx)))))))<)) ((6(xx)(((x))) =(x)x(x) 2 (x)) (21)
A straightforward calculation shows that
M(0, t) = J(t) 0 ((r,t).
Claim B.6 (Orthogonality of the loss and continuity (assumption B.3 and Dg (0. go] continuous
Dg Mo so)lā gol = =0
Proof.
Dym(fo: 90, X,Y, 90] =2a 2 (x) a X(Y a (To)) (o(X)) + (xxx)))))-)(
22 2n- -xx)) -t(x)
Since, (X( (X,0.) and the fact that ax.(X) vo(X))(XX), we obtain that E [D 9o. X, & gol X] =0
The continuity of the second order functional derivative naturally follows.
Recall that we have the following assumption of invertibility of (to)) E (X) Theorem 4.1
from
Assumption B.7 (Invertibility the Jacobian). We assume that the Jacobian matrix satisfies
JJ - (1x))
<C
for some constant C > 0. This claim is justified under mild conditions on the boundedness of the reward and eigen values of the data matrix E[X X).
We now prove the following claim on nuisance estimation.
Claim B.8 (Nuisance Estimation Rates). There exists black-box learner such that its root mean squared error (RMSE) satisfies
|1 to (L2() =o ( with 31 2 4'
1
and
||e - rol =o
n3 with 32
These conditions ensure that the nuisance estimates converge sufficiently fast the sample size increases.
Proof. We show that these slow rates can be achieved under both conditions of plug-in where f(-) taub with t(-) estimated via log-loss. Alternately, one could separately estimate t(-) as well to obtain these slow rates. A discussion is presented in in/ppeddix B.5
+(.)
Furthermore, we assert the following claim regarding the boundedness of the nuisance function (X). Claim B.9.
tx( = Lanh(a X))
(0,, X) S a.
23

---

A brief inspection using the properties of the hyperbolic tangent function (notably, that tanh(z) S for x 2 0) confirms this bound.
We now show that assumptions in Assumption B.4 hold for our loss function using the following three claims and lemmas namely Claim B.1oj B.1 Lemma B.12 and Claim B.13
Claim B.10. The jacobian J() satisfies J(i) [(t)) o(1) and the tthh order moment conditions
hold i.e. i.e. yup (E [|]]/(((...22)191)))) and supyes 2111919))1))) is bounded.
Proof. Write
((t)) = (1(x)2 xxx
Since the operator norm is bounded by the Frobenius norm, we have [| - J -
Using the Frobenius norm, we estimate
[t(t) - J(to)llF= = (11
(x()) xx"]]]
Using the boundedness of X (i.e. || $ 1). we have
= || <1
Notice that
|1 - (X)) (XX))
Both and t, uniformly bounded (lt (it())),, (X)) - a²), then (t(X) tx(x < 2. Hence,
[((x)2 - tx()² < 2a² [( - to (X)) Thus, we obtain
- - ((.))lF S 2a 2 [(X) - (x() = 2a 2E [()) - (x)]]
Finally, applying Jensen's 's inequality (or noting that E|(((X) - (x)Il S ||I 1. (D))) yields
[[](t) J(po)loop S 2a 2llt toll
This concludes the proof the nuisance estimators consistent i.e 1 go (0))) 0p(1)
We now check the moment condition. This naturally follows from the fact that E[7a X] is bounded for every a 2 1 and the fact that rest all random variables functions are bounded.
Claim B.11. Let g be a funce defined as
[t(x)]
g(x)
[(xx)
Then, for any g Tÿ + (1 T)g. with E [0, 1]. we have
/n D gg M (f.. 9) و go] = (1).
Proof. We wish to control the second-order term in the expansion of M (6., g) around go. By
Taylor's theorem in the direction 9 Io. we have
0² M (0. g + s -bo)
D M (0.. g gol 0g²
This term decomposes into two parts:
M M (0.,9) 96 gol = ds²
(J ([ 1.)) ((-))(())))))))
( (o),
+
8² O5²
Ets (t- ((-rr)) ro))
(1)
(22)
24

---

First Term (I): For [t) -E(((())XX" we have
の² ds
+((x))ss((()))) - (((x))+((((3)22xx -xx())))) -2(1())- - Lx(x))²
Thus,
([ s (f - to)))a)) ==0 2a - B (f(((() - (x))² xx" Po.
Since ||XX*|| dlop is bounded (using ||x||| - 1) and 0, is fixed, it follows that
2 (((X) (x)) x 0, = o (18 to113)
Under Claim B.8 we have ||e 1o.13 = of vn so that
Vn 0 |1i 1oll d1).
Second Term (II): For the function V(I, y), we have
82
Os² ((+ + (f- -to), 0 (Ú yo)) 1 a a (2T (((X))(E(X) -o (x()))((()) (x)) x]
2 (F(00 - (()))=x]
FE ((,(x) - 2((X)) - -r()))())))-(())())))))) - (()))
2 B (+X))(((-- (X
4
(23)
The last equality follows via conditioning the first term. Using the boundedness of X, ((X) and inequality, we obtain
t( (X) and applying the
1 ((())) - (x))) (B(X) - rx()) x
<0|10- [ell8--ollaae (oll rollea(D),
(24)
E (x(X)(((x)) (xx))) scllf to ttolL2
(25)
for some constant C' > 0. By Claim B.8, the product /|č - tollez(D) ||e - rolles is o and || (o)li)(D) is o(1/n). Therefore,
vū E (x (E(X) (Fx x o(1)
Combining the two terms, we conclude that
/n D (0 - go! = (p())
This completes the proof.
Lemma B.12. The The following holds
t0, r,t; Y, mn0,, ro, to: XX -t toll? 22(1) + +||t-1u rol 2oll))()))
Given that ||E - tolle,(D)
o(1) and ||e - -rollc(()) = o(1), we have
El]]m(0..0 9. Z) m(0.. go: [] o(1)
Proof.
We have
t, X, Y,T) to:
(26)
=a ((
(27)
25

---

({())(X) -1777177777110711077777111111133
-(x(X)(X)) -,C
=I(X)(((X) -
Using the fact that ||||| - 1, we have
m, - 1: = vda--1 (1000
-
+ +x(x)(((x) - (00 (x(x(()))) -C0 ((x)) + + + -(x)))
(28)
The desired result is immediate via the the identity that (a+b+c+d+e) + except for the following terms
E - -1
-E - |X] - C|st toll??.
for some constant C.
One can similarly argue that
E -r(0))))(
s Clr roll
Finally this gives us
B -1 50 toll- + (48 + 1)1l= (.())
Claim B.13. Each component of the matrix jto, X) satisfies
E Bijna(tos x)p = 0(1)
Furthermore, the components of the moment function satisfy
mm,(o,, tn)² = 0(1)
Proof. The first statement follows directly from the boundedness of (X) and the boundedness of X. For the second statement, note that
e m/ Mos (,)²] S = (Effo(x))]
+ +E(a "(X, 0,) (xx)) -[xx(()((
Since the functions ra and lo are bounded and the variance of T is also bounded, it follows that
E my (fo. (ooo = 0
Invoking Lemma B.5
we conclude that the estimator Ó obtained via minimising orthogonal loss Cortho is asymptotically linear. In particular,
n
V 0.) =
IF X,, Y,) o,,
(29)
26

---

where the influence function is given by (F For to: X, T) = Elaa (xxx= For to: X.Y.T
Note that by the definition of 0, we have E [m((0. (on X, X,YTT) Now
To, to; X,Y,7) = (X,0) 0) to tx()²² vo - (((()) - vto())) x
(30)
(Vn(X) tt (x)) tx) ( 3o(()))
(31)
Thus using the fact that ETT X] (x(x), EELY X] (x) and Var(Y (x) M)()))
get
Covar(m((0, loi X, X)), -ي2 [(
(1(x))(1=xx(x)))) vx(x))) + Var(TIX) (ar(T))))6(xx))))x)
The last step follows from the fact that CCaim B.24). Thus, the covariance of the influence function is given by
Covar(IP) = (X) -X
In particular, this concludes the proof of T'eerrm 4.1 We restate it below for clarity.
Theorem. Let fortho minimize the orthogonal loss Lortho If (X)) is invertible, then
VnilBortoo - -0,) 4 N ON) (0
where
(atanh(a(O0) x1) {
[ (o)
Σ
XXT
E
XXT
(0o, X)
(0,, X)
From claim Claim B.25 and the fact that tanh than the variance computed in Lemma B.2
< 1, one can observe that this variance Σ is smaller
B.4 Guarantees for Orthogonal Loss -
In this section, we prove the following theorem which the analog of Theerem 4.T while minimising the orthogonal loss Corthe-2 We obtain identical asymptotic rates as obtained in Cottho
Theorem B.14. Let dortho minimize the orthogonal loss Cartho-2, [[E[((())=xx]] is invertible, then
Vn(lortho -0,) 10.40000
where
Σ (< (0o, X)
2 (1)
-2
(atanh(aa00: X1) XXT (0o. X)
T
E
Further in this setup, we use E(-) to denote the pointwise version orthogonal loss Coreho 2 Recall that the nuisance function g()) denotes the tuple of nuisance functions (y, t) with (-(-) = (yo, to) denoting their true values. Further the nuisance function ý = (ý, !) denotes an estimate of true nuisance function (.) = (yo, (o) after
We now define the moment functions as defined in Appendix B.2.
the first stage. Let 0, E Rd denote the true parameter vector. We define the moment function by
m00, y. X,Y, 1 rx
27

---

which, after algebraic manipulation, can be written as
m(0, t; = = (xx.0) x(x)* ((X=+xxx(= - -
addition, we introduce the auxiliary functions:
j(4 x(x) xx" and =V
(32)
We also define the expectation-based mappings:
mapiiia
M(O. v. E H, (.X.T.
J(1) = [
V(v. = a E (Fa_'E(((((o_ ((o(()) (XX)) (2)))1(0)--((0000))
straightforward calculation shows that
M y, J(t) Vyy,
Claim B.15 (Orthogonality of the loss and continuity (assumption B.3). and 9g (0, g)[g go] continuous
Dy M go -9ol = 0
Proof.
Dom 90. X.Y, - gol =2a -2 2 (xx - -xxx))) - to()))
a (X(((X - to(x)) + +a a - to(x)) b0()
(f 00>l0 90l m[ (fa. to yo. x, X,Y to 0 voll
(1/a) E [(2a (X, to) (X - ()( xx(x))]x]]
+ E|E((T to (o( - [X]
Note E[(T - (X)))(((( - 2o (X)) = 0 since [] to(X).
Also, observe that E22(X, (X) - (X Y| = 0 since (X, = axa(X) 1o(X)
and E|y|X =
(X).
Thus, we can show that Dym(Bo 90 X., -o) = 0
The continuity of the second order functional derivative naturally follows.
The following assumption comes from the assumption in Theorem B.14
Assumption B.16 (Invertibilityy of the Jacobian). We assume that the Jacobian matrix satisfies
(J(t..)- 1ppp ( xxT
SC
O. This claim is justified under mild conditions on the boundedness of the reward and eigen values of the data matrix EEXX"* X
for some constant C >
Furthermore, we assert the following claim regarding the boundedness the nuisance function (X).
We now state our assumptions on the convergence rates of the nuisance function estimators. Claim B.17 (Nuisance Estimation Rates). There exists a black-bo box learner such that its root mean squared error (RMSE) satisfies
|| to toolLo(D)) = ( (11 with 3 2 4'
and
1 1 3 with 32 2
- 1--Vollla(D) =
2
These conditions ensure that the nuisance estimates converge sufficiently fast as the sample size n increases.
28

---

A discussion on how these slow rates can be attained in provided in Appendix B.5
Claim B.18.
tanh(((.: X)) (X)
(0o, X) S 1.
A brief inspection using the properties of the hyperbolic tangent function (notably, that tanh(x) S for 2 confirms this bound.
We now show that assumptions in Assumption B. hold for our loss function using the following four claims and lemmas namely Claim B.19 Claim B.20 Lemma B.21 and Claim B.22
Claim B.19. The jacobian J-) satisfies J(f) J(to) o(1) and the tth order moment conditions hold Le. iee. supgeg [[]m((0.. [1)1110 and supges (B(liig, 2)119///)))) is bounded.
Proof. Write
xx7]. Since the operator norm is bounded by the Frobenius norm, we have
J(1) - J(1)) =a -
[|[(t) - J(c))lopp s (1)(() - ((.))le.
Using the Frobenius norm, we estimate
[J(() J(o)]l (1tt))l===aaa) ( x
Using the boundedness of X (i.e. || S 1), we have
||X | 1 11
Notice that
| [(x)2 - 1o (xx) - (x0)
4xX))-
Both and t, are uniformly bounded (l((X),, [X)] S 1). then ((X) + to(X) S 2. Hence,
[(X)² (x)² < ((() - (X)]. Thus, we obtaín
[ - ( S 2)(xx - tx( = 2a - 2 ((x) - (x)]]
Finally, applying Jensen's inequality (or noting that El|((X) - tx)ill S tollol))a( yields
[[)(t) - J(to)l - 2a P|l toll tol1
This concludes the proof of the first part the nuisance estimators consistent i.e || gollL,(2) (1)
We now check the moment condition. This naturally follows from the fact that E X] is bounded for every a 2 1 and the fact that rest all random variables or functions are bounded.
Claim B.20. Let g be a vector-valued function defined as
[(xx)
g(x)
y(())
Then, for any g = Tg + (1 T)g. with E [0, 1). we have
/n D og M (0.. 9) [g gol (1).
29

---

Proof. We wish to control the second-order term in the expansion of M (6., g) around Jo- By Taylor's theorem in the direction g 9o. we have
8² M fo g+ s - (o) swm)
ggg (0.. g) - gol
(0o.
Og²
This term decomposes into two parts:
0²
02 +
Dog ().99)6 - go]
()(t - s - to))
v si+ to)s + s sw - Bol)
ds²
O5²
1--
(
(33)
First Term (I): For J(t) = a -=xx we have
8²
(((X)) + - ((( =2(t(x)-to(X)
0s²
Thus,
(J(T+ s - (tt)ttttt))))) a=0 = 2B (t(x) (x))²² xx" 1..
25²
Since ||XX*| Npp is bounded (using || S 1) and 0, is fixed, follows that
2 (t(x) tx xxT 0 = o (1t toll2)
Under Claim B.17 we have ||e - 1o.12 = o (121 so that
vn 0 [|t 101l2 o(1)
Second Term (II): For the function ((t, v), we have
8 f++ V s(t to), (y vo)) - a [ txxx)
tx))) (2(x) -x,())))) x)
0s2
Using the boundedness of X and applying the ineuuuann we obtain
B (u(X)) - (x))) (x(X) vo(x)) x C' ||t tollza(2) |l Vollea(D))
0. By Claim B.17 the product ||t tollex(D) l1y voll is of (t)).
for some constant Therefore,
(1()) - tx(x)) (v(X) - 3o(()) = o(1).
Combining the two terms, conclude that
vn D (f.. (B) gol = (1).
This completes the proof.
Lemma B.21. The The following holds
Elllm(0., u, X,Y,T) mm(0: Yos loi X,,,))111 -1111- -611² L2(D) + +11y- -vol vool).()))
Given that || - hollez(D) = o(1) and ||e rollea(D) = o(1). we have Ellim(0n: 2) m(00, g0; =)][===ff)
Proof. We have
yo. to;
=a- (a (0,, X)((X) - (x)3) + (v(X) (x(X) (X))((()) v(X)(X) Y(((X) -(xxx))((x))
30

---

We can further write the term
(o()too(X) - u(X)x(X)
Using the fact that ||||| I 1, we have
F(m(0o: - No, XX: =Vāa
Vdaa -1 (X))))/)(()---)))))))))++++0+
-x()) + ((x) - ((x) + (((o(x))((x))
- vo
The desired result is immediate via the the identity that ++ < 1(a2 + + d except for the following term
((v(X) (I
27 = E
((v(x) - (x | x
E (u(x) - 3o(X))* ( (oT-)
X
[vv(X) (() X)] S E [v((X) (x(())
The last inequality follows since Var(T X) I 1. Finally this gives us
m(0.. v. (,Y,Y, m0(0. No. lo: X, ) s (45 + 1)]l - 2011212200 (2((D) + (48 1)1ly yoll" -0112 (2(
Claim B.22. Each component of the matrix ilo; X) satisfies
[jpg(too x}}| O(1)
Furthermore, the components of the moment function satisfy
E mm Bor to) = O())
Proof. The first statement follows directly from the boundedness of (X) and the boundedness of X. For the second statement, note that
E m0 yo, Non to)²] < 20 (B] + B "([a-"(X, (x) to)))) - (T)())((-=
Since the functions yo and to bounded and the variance of I is also bounded. it follows that No: (o) = 0().
E
Lemma B.5 obtained via minimising orthogonal loss 2ertho-2 is asymptotically linear. In particular,
Invoking
Vn (fortho 00) =
IF yor to: X,, Y,, Ti) op))
(34)
where the influence function is given by
[F Vo, to: X,, Ti) = Ela to (x)2xx m(f., No. to: Xz. Y,, Ti)
Note that by the definition of 0,, we have E [m((,, Nor Loi X, X,Y,T) = 0. Now
m(0.. to No: X, Y,T) (X,0) (x) + (X) - (x)()) - rto(X))x
(35)
(vo(X) - to - - -v(()))
(36)
31

---

Thus using the fact that B X] to(X), x| = (x) and Var(Y (X) 1 (o(X), we
get
Covar(m((o;, Nos to: XX,T), - 1
- 2o( ((xxx)((1x((x))))) Var(T]X) 2o(x)) xXX
a²
(37)
(38)
The last step follows from the fact that Claim B.24). Thus, the covariance of the influence function is given by
Covar(IP) Elt (X) "X Xx
This completes the proof of theorem B.14
B.5 Estimation nuisance functions in first stage
Plug-in loss:
In this setup, one first estimate r(-) by minimizing the log-loss described in Appendix B and thus obtain |]É(-) r(-)lL2(()) - Now plug in this to estimate {(.)
x(.h
can get
<(-)
|| (D) 0 using Lipschitzness.
For orthogonal loss Corthe- similar plug-in would give |1 yolle,(D) O
This provides the slow rates that we desire in Claim B.8 and Claim B.17 for orthogonal losses Cat and Lortho respectively.
Separate nuisance estimation:
In this setup, we separately estimate the nuisance function t(-) and r(.). The reward nuisance function r(-) can be estimated by minimizing the log-loss as described above obtain ||ê(-) r(-)llra(P)) =
To estimate time, get first argue that the Waa Sobolev norm [FF) (with > 1) of to is bounded using the composition theorem [Mo666]. More specifically, one can show that W.,2 Sobolev norm of t-)-) is given by O (Ss!) as the 8th order derivative of tanh æ scales as s!. Now applying kernel ridge regression with a Sobolev kernel associated with the RKHS W/, AF03 Wai19 to get the
desired slow rate i.c. || - toll O ss.Sn" = One can choose the value of appropriately
to obtain the desired rates. Choosing sufficiently high can give us faster decay with number of samples but the pre-muliptlier s! would be higher.
For Cortho-2 one can bound || yoll 0 via plug-in.
We thus get the desired slow rates to satisfy Claim B.8 and Claim B.17
B.6 Inequalities used in asymptotic results
We now prove two inequalities that have been used in the main paper. However, we restrict ourselves the case where and fnnndo 0.
Claim B.23. Var(T' X) < (E[T)) X])²
Proof. Now consider the expression of Var(T) X) from Appendix A.I have
Now denoting r(X) by r.
Aar 2ar
(2 1
Var [T X) = a
and (EE[T) X))²
(39)
(e + 1)2
r2 ( + 1)
32

---

tx Observe that it is sufficient to consider r 2 0 as both functions are symmetric. Thus, it is sufficient to show that ar 1)² 2 eaar 2are Yr 2 Now define ra and we now define the function (() as follows and argue that it is non-negative.
f(v) vle2" 1)2 - 4uc (v 11)ctt -2ve2v + +0+1
(40)
(v(e² +1) - -1)- -1) 20 +1)
(41)
Since 22v +1 > 0, it suffices to show
g(v) v0c²" + (c - 1) 2 0.
Differentiating w.r.t. e gives
g'(v) = (2u - 1)e2 20 + +1, g0(v) = 20 2 0.
Hence g'(v) is monotonically increasing, so
g((1) 2 (() = 0,
which in turn implies
g(v) 2 g(0) =
Thus the desired result follows.
Claim B.24, Define (X) = BET X] and (x( = =F[7 X].
We then have
(X)3
((X)((1 (x)3) Var(T (x)=
This follows from the expressions in Appendix A. proof.
Proof.
For brevity, we refer r(X) by in this
Now consider
to (X) - (o()) + Var(T') X))o(X)² a²
(42)
p (exp(2ar) +1 -1) (
[exp(2ar)
4 exxp( (2ar)
(exp(4ar) - 1 4ar exp(2ar) (exp(2ar) + 1)2
[exp (exp(2ar) (2ar) - exp (2ar)
(exp(2ar) 1)2
(43)
(exp(4ar)= (exp((2ar) 1)² (exp(2ar) 1)4
(44)
a²
The following claim would be useful to lower bound 4a² (2)r.(X)) by tx
a²
Claim B.25. -2ax) < function.
(tanli(ar) a2
2 for every x, a 2 where (.) denotes the sigmoid
Proof. Since both functions are symmetric, is sufficient to consider the case where > 0. Observe that
2ax
(tanh( (ax) 2
2ax
1
(22x))o(2200)) (c2az + 1)2
and
(45)
2 c2av
Now substitute y = a.r and thus, to prove the desired result, sufficient show that e²y 1 > 2 < et - 22-2y 2y 2 O. Now define g(v) = ed ee -
v 2y and thus, g'(y) e te v 2 from AM-GM inequality. This implies g 2 g(0) = 0 proving the desired result.
33

---

C Finite sample rates for General reward functions
Given set of samples {a:]i= each drawn i.i.d from D, consider the empirical distribution (x(x)) == + Ei= sz, (x) which places point mass J at each sample. Thus, the emperical mean is denoted by P.) E and the population mean P/ Ex->l/(X)),
As noted in Section 5 our critical-radius argument for bounding the excess risk <(f; 9) requires the per-sample loss to be uniformly bounded. In the EZ-diffusion model, however, the response time T's in principle unbounded unbounded-despite having finite moments [WY08]-4 we replace the original loss by a truncated version. This truncation both restores the boundedness needed for the theory and reflects the fact that, in practice, decision times are effectively bounded.
By Markov's inequality and the boundedness of higher moments, for any Ç 2 1 we have
B[TS]
M(()
P(T B (
(46)
BC
BC
Here M() = ETT] denotes the Ç-th moment of the decision time T (Y00). One selects / to meet a desired error tolerance, ensuring the tail probability in (46) is sufficiently small. Finally, define the truncated response time T = min D} and substitute T for T in the loss function Lortho
C.1 Bounding the loss function Cartho with bounded decision time
We now introduce some new notations with respect to the decision time. Let B denote the bound at which the decision time is capped. Let (X) = E[["" X]. Further, let f denote the nuisance function corresponding the the capped decision time and let denote an estimate of t. We further overwrite Z to denote the tuple of random variables (X, Y, Y2)". The joint nuisance pair is denoted by = (t. i) and let = (t, Io). Now, assume that L,2 E T and G denotes the joint nuisance class R. x T with E G. Before, we start the analysis we state the following bound on to f. To bound, we use the following result i.e. E[X] J2 P(X 2 x dz for non-negative random variable X.
Recall that S denotes an absolute bound on the reward function r(-) which we assume to be larger than 4.
00 /
M(C)
llto - tollex =
P(T 2)d2 - s=
(47)
CBS-1
The first equality follows as T T - Thus the orthogonalized loss function from (6) is redefined as
implies T B + z for every positive s.
-fX)(xxx (x-- -/
Northo (r, - ( (y-
(48)
Next, we verify that each of the four assumptions in HS23 Section 3] holds for the loss Cortho
Observe that the first assumption of neyman-orthogonaal
satisfied approximately here with a bias decaying with the threshold β that is characterized by (-))()) - [I (qqattion (47)).
Lemma C.1 (Approximately Orthogonal loss). For all r R and G. we have DD,D,oortho go) - To: g - goll S 52111o - Tollx 11C Eolle,())llr rolle.(P) (49)
Proof. Consider
D₂ D,D, Northo
(ro, Fo, -99 gml
(4)
2F (T - (xx(((())6())((( (ro)))(()))()(((((())))))) -ro(())(((())) -r())) 2 [Y- -CX0))EE)))0000000000))---------
- - -110x)--XXXXX -6(x))(((x) - -x)()]]
< - Toll 161.1
34

---

Observe that the first term in (a) goes to zero via conditioning on X as E(T" x] = (X)). The second term in (a) is simplified using Cauchy-Schware
Lemma C.2 (First order condition).
(ros rol 251
roll
Proof. Now consider the following functional derivative
D, jottho (To (o) rol = 2E (Y - (7- (.w.://w....--......////./------
5 28E (tx( - (x)))) -ro))))))))))))) rx(x))]]
s 2 (o)lL,(D)) || rol L2(()
(a) follows via conditioning on X and the fact that E| X] to(X)ro(XX..
We now prove the smoothness condition from FS23 from (9) (denoted by (R., a)).
Assumption 3]. this setup, I|-Ile is defined
Lemma C.3 (Higher-Order Smoothness of Cortho), Let B1 and B3 4S. Then gortho satisfies:
1. Second-order smoothness w.r.t target. For all E R and all F E star(R,
D& Fortho go - To: r rol s 31 ||r rolla
2. Higher-order smoothness. There exists E [0, 1] such that for all r E Star((RTo), g E G. and star(9, 9o).
|DGD, Cortho (ro, Or
90, g0]l < B2 -ellr-rollienn roll 1g -
Proof. In this proof, E(-) denotes the pointwise version of the orthogonal loss gortho
First, observe that D. Ios X, (r To, r ral (rr(X)(r(X) rx))) Thus, [D Io, X, Y, (r To, roll S 2 0.11 where ro(x) (0., for every 1 The last inequality follows from the fact that to (X,, X2) 1.
E
go.9- - goll
=E - rx())) - 4t (x)(()(((()) - tto()))(((() ((()))((()))((X(6(((())))) rx(X))
(50)
<4S1|g - yo 1I - ra 11
(51)
(52)
The last statement follows from the definition.
Lemma C.4 (Strong Convexity of (ortho), The population risk is strongly convex w.r.t. the target
parameter. There exist constant
<
(anbi(S) such that for all R, g E g and all
i E star(R,
(r r rol 2 ||| - par-roll2ant roll2,.())
where [0, [0, is as in Assumption C.3
35

---

To, r Proof. First observe that D2 - 2f
- Thus,
E. [D 9, X, () Ya, - ro]] (X) ( -ra(x))
(53)
Now considering a lower bound of tanh(S) every function in T, prove the lemma.
With this, we now prove the main theorem below using FS23 Theorem 1]. Theorem C.5. Suppose f minimizes the orthogonal loss Cotho and satisfies
fo (f.g) g) Lortho (ro: 9) I (f.g).
With S denoting an absolute bound Then the two stage meta- aheta-algoritmm with orthogonal loss fortho guarantees
F - rolli S CCS (c(f, llto Foll2..)
4ST 11 +
1+o gol
(54)
(R.a)
Further, the error term |14o lo can be bounded in (47 in terms of bound B. One choose moment ( 2 1 to get the largest bound.
Proof. Applying
FS23
observe that
||F rollliz ( <45² (c(f, 0) + 25 - + 5 - Tolle. A/l E rol (o
+ 4 وا Jo (R.a)
ITo
(55)
Applying the AM-GM inequality, we have for some constant C
|| - fr-rooleace rollEa(D) +f) - Eo (1+ E -oo 4S 110 - go
(R.a)
(56)
<CS² (e(f.g) + llt. - to ) + 45 وا (R.0)
(57)
We now prove the following corollaries from Section As we discussed above, we cannot instantiate Theorem 5.1 directly obtain rates using critical radius the critical radius crucially invokes Talagrand's inequality which requires the loss functions to be point-wise bounded. We thus invoke Theorem C.5 with bounded loss function fortho by capping the decision time T by a threshold B.
C.2
Proof Corollaayy 5.3 and Corollary 5.4
Corollary 5.3 (Data-splitting). Let 8n be the critical radius of the star-shaped class
star{ Toi r. E R, R,0).
and define 8dh = maxx 8n, v /enn for some constant e > 0. Then under data-splittngg with probability at least 1 - expl (--2n (5))))))))) Meta-Algorithhn 1 satisfies
c(f,g) I 9S Bod I| rollia(D) + 10 s0 (5
for universal constants C, > O. Consequently for every ( 1,
|| | & poly(() ( (8:0) 2 M(C) 2
+ 4 4T+ ||g go (R)))
I+a
CBC-1
holds with the same probability.
Although FS233 assumes exact orthogonality of the loss, any remaining bias contributes only an additive term which account for here. This immediately follows from the proof of [FS23 Theorem
36

---

Proof. Let (.)tho denote the population loss and let (.) be the empirical loss evaluated on the sample S [Z ) Further, write fortho for the pointwise version of the loss Cottho suffices to show that for every r E R.
Egtto (r,
((ortho(r, Lortho (ro: 9))
S99 B gdh ||r - FollL2(D) + 10 B (d
(58)
Once 58) holds, the fact that f minimizes the empirical loss (so Lg (f,g) L Loreho (ro: immediately yields the desired corollary.
The proof of this analysis follows standard techniques Wail9 Theorem 14.20]. Let R. star(fr To : E RJ, 0) be the star-convex hull. Further, let
(C)
sup [|r-rolle SC
Pr fortho (r, -portho 0.)) TP( (Pooootoo (r, 1 -orruo (ros 9: -11--
(59)
We now define two events as function of nuisance g(.). Let
E. (Z (5%) 2 10/ (5%h)² 3
and
= (3r
Pn (ortho (r.g. jortho
Pn
-ootho (eobboroo....)) -P ((roorthorrr (r. -orrtho (ro.
-ootthorro 9.
295 B 8olo ||| rolle,(D) and ||| - rollec))) 2 2
(60)
the bound in (58) does not hold, then either event & or event & must be true. The probability of the event & can be bounded using same peeling arguments from [Wai19] Theorem 14.1).
To bound the probability of &., we first employ Talagrand's concentration for empirical processes obtain for every nuisance estimate 9
C2n22
P (()
+w) u S exp (-?² u
(61)
We shall now crucially use the fact that nuisance function g(.) is conditionally independent of the data-set Zı... Zn.
E Iz (c)
(1)
<E
sup
1
Σ (cortho (r, 2,) - gortho (roch 9, Zi))
(62)
rER:: ( Tull ()) $C
(5))
- 21
sup
1
Σ SBca
r0) -ro)
(63)
P n
(68) I CSBRad, (Ro, 4SBro valid for all ( 2 8ds
(64)
The step (i) follows from symmetrization argument, step (ii) follows from the independence of 9 with respect to data-set Zı..... Zn and bound B on decision time I and step (iii) follows from our choice of 5ds.
Now apply the Talagrand's concentration lemma to bound the probability of event &, and conclude the proof.
The corollary statement follows by applying the bound e(f,g) to Theorem C.5 and bounding the rate
!|!, - Toll using Equation (47)
37

---

Next, we prove the corollary for data- data-reuse reuse for Cortho Observe that the critical radius is computed over a bigger class for the case of data-reuse as the independence of nuisance estimate 9 and the data-set Z1.... Zn cannot be assumed.
Corollary 5.4 (Data-reuse). Lel on be the critical radius of
star{ earcho (r.g:) gortho (ros 9: : R. gEg).
and define our max(on, Ve/n} for some constant > O. Then under data-reuse, with probabil- ity at least 1 - (2nn ))), Meta-Algorrthmm satisfies
cff.9 - 958 8h ||F - Tollez(D) + 10 (
for universal constants C12 02 > O. Consequently for every 221,
||- ( & poly(S) (0th)²
(()
+ 4SIta 11
4
I+a
(BS-1
(R,a)*
holds with the same probability.
Proof. We denote Portho (.) as the sample loss evaluated on a set of n data points with S = (Z....., }. J. Further, denote jortho as the point-wise loss version of gortho
Observe that it is sufficient show that the following bound is satisfied. This proves the corollary as minimizing the empirical loss ensures Lg the (f,g) Ég tho (ro:g)) 2 for some g E g.
(Egrtho (r. Lortho (rosg))= Northo (r.g) Cortoo (To.g))
I -654 /ortbo (r; (r:g)
Cortho g)l + 10(5 Vr E R and g E G.
(65)
The proof this analysis follows standard techniques [Wal19] Theorem 14.20). Let I denote the star convex hull defined in the corollary:
star (Z - jortho (r:g, 2) fortbo foxtbo(roig (raig, 7): Vr E R.
(66)
In the notation below, f denotes any element in F.
7n(r)= sup ||
[[Pa()) P(OII
Now we define two events
Ep [Z, (8 2 9 (5thr)
and
ε = & [P())) - P()) z 106 || fl( and [[]]] 2 6dr
If that the bound in 65 does not hold true, either event & or event & must hold true. The probability of & can be bounded by same peeling arguments as done WWil9 Theorem 14.1].
To bound the probability of Eg, we first employ Talagrand's concentration for empirical processes to obtain
(- C2nu²
P (Z, 2 (C) + 5 5 exp
(67)
In particular, the expectation can be bounded as follows.
(
Σ (af(z))
(2x(()) <2E
sup SE ||
(68)
4Rad, (F, < 4 valid for all ( 2 8dr
(69)
38

---

The step (i) follows from symmetrization argument, step (ii) follows from our choice of 8% Thus, get the desired bound on the event & which completes the proof.
C.3 Instantiating Critical Radius Guarantees for Data-Splitting case
We now provide standard critical radius rates dde for some standard function classes R. For RKHS classes with eigen vales of kernel IC decaying as j (eg. Sobolev spaces), the critical radius can be bounded O(n )) assuming a bound of unity on the RKHS norm [Wi19]
For VC sub-classes R (star-shaped at ro) with an absolute bound on the reward model class, we can bound the critical radius 6n Wai19] CNSS24).
V /dlog where d denotes the VC-subgraph dimension
C.4 Estimation of nuisance parameters
C.4.1 Plug-in estimates for preference and time
In this case, since the functions tanh(x)/2 is 1-Lipschitz, one can argue that ||č 17.-t, toll ||i rall where / = tanh(E)
One can thus estimate r(-) by minimizing the log-loss B and since log-loss strongly convex with parameter - [Wai19] Example 14.1 18] we argue that rate || rolltz(P) = (e where 5, is
the critical radius of the function class R. Via plugging in i.e. estimating f tanh r(-)
argue from Lipschitzness that the same rate holds for || - to (D)) However, the estimate || dell would suffer from small bias bounded by ||to tolle.. which decays with the bound / as discussed in (47).
c(.)
C.4.2 Separate estimation of preference and time
One can get bounds on the rate I| - tall based on the critical radius 6, of the class T post centering at Tg. We can get standard rates for RKHS and VC sub-classes. Estimation of nuisance r(-) can be performed using log-loss.
C,5 Proof of Theorem 5.1
We can prove Theorem 5.1 very similar to the proof of Theorem C.5 by instantiating 1] to our case.
LES2
Theorem
Assumption (Neymann orthogonality) in [F223 Section 3.2] for the loss function Crth has been already proven in Lemma 3.1 Assumptions 3 (Higher-order smoothness) and 4 (strong-convexity)
from FS23 can be proven very similar to the proof in Lemma C.3 and Lemma C.4 L
D Experimentation Details
Below we provide the experiment details
Linear Rewards We work with a linear reward model defined by (x-) = (0, X and ro (x²)
X2) for each query pair (X1, X2), and denote the ground-truth parameter by 0., All queries restricted to lie on the unit-radiue shell: we sample each coordinate independently and then rescale the resulting vector so that || la 1. Preferences Y and response times T' are generated synthetically according to the EZ diffusion model, producing the dataset (X!, X?. Y. T: ? Experiments are performed for various values of the norm |10.112.
The anonymized code can be found at]
39

---

The ground-truie parameter 8, R is itself drawn at random, sampling each coordinate indepen- dently; every draw therefore yields a new dataset. Our aim is recover t, using the three losses introduced earlier: the logistic loss [2og the non-orthogonal loss _ and the orthogonal loss Cortho defined in Equations (3) (4) and (6) For each loss we compute the lg estimation error
|10 |10- 000112 as a function of the true-parameter norm B = |10o|12. averaged over 10 datasets generated from different random draws of 8.- In the orthogonal and non-orthogonal settings the nuisance functions- the reward model r-(.) and the time model i(.) are learned on a held-out split and then plugged into the second-stage optimization. Because E, x] is non-linear, we approximate it with a three-layer neural network. For the logistic baseline, the entire dataset is instead used to fit the reward model via standard logistic regression.
Non-linear rewards-neural networks. We generate synthetic data from random three-layer neural networks with sigmoid activations in the two hidden layers (widths 64 and 32) and final linear output layer, fixed input dimension d = 10. For each training size N, we sample three independent "true" reward networks by drawing each hidden-layer weight matrix and the final-layer vector i.i.d. from (00, 1), then for each network generate N training pairs and 3000 test pairs of queries (X1, as i.i.d. Gaussian vectors; each reward model is trained four times to assess variability. We evaluate all three losses logistic, non-orthogonal, and orthogonal and for the orthogonal loss compare both a simple data-split implementation and a data-reuse implementation. Using this synthetic data, we first learn the nuisance by minimizing the logistic loss with three-layer network of widths (10, 32, 16, 1), and learn the (-nuisance by minimizing squared error on T with a three-layer network of widths (20, 32, 16, 1) taking (X1, X2) concatenated as input. Finally, for each repetition we fit the reward model (same architecture as t) by minimizing each candidate loss. Figure 2 reports the mean squared error of the estimated reward under each loss and the corresponding policy regret after thresholding f into a binary decision.
Text-to-image preference learning. We evaluate our approach on a real-world text-to-image preference dataset- Pick-a-pick (KPS= which contains an approx 500k text-to-image dataset generated from several diffusion models. Furthermore, we use the PickScore model [KPS' 23] as an oracle reward function, we simulate binary preferences Y E {+1, and response times T via the EZ-diffusion process conditioned on the PickScore difference between each image-test pair. To learn the reward model we extract d024-dime embeddings from both the text prompt and the generated image using the CLIP model [KKH+21] 21). On top of these embeddings, we train a 4- layered feed-forware neural network with hidden layers of sizes 1024, 512, 256, under three training objectives: our proposed orthogonal loss, a oothoorhogonal response-ting loss, and the standard log-loss on binary preferences. The time nuisance model i uses the same four-layer architecture, and the initial reward nuisance t matches the architecture of r. For each training size N, we draw N random image-text pairs for training and an additional 10000 for testing (from the remaining dataset). For each N we repeat the training process 3 times with different seeds.
D.1 Additional experiments comparing with LZR+24
Our primary focus is supervised reward estimation under passive (i.i.d)) queries, which is often more challenging than adaptive best-arm identification (BAI). To demonstrate improved performance in the adaptive setting as well, we embed our estimator into the BAI pipeline of 1.ZR+24 Algorithm 1] and compare directly.
Setup. We use the real-world food proe-erffeeence dataset SSK88 w which provides pairwise choices and response times (RT) from 42 participants. Following [LR 24]. we construct bandit tasks for each participant and adhere to their sequential-climinatin protocol (Algorithm 1 in [LZR 24)).
Protocol. We replace their RT estimator with our orthogonal- estimator while keeping all other components unchanged. For each participant, we run 70 independent trials at total sample budgets 500, 1000, and 1500. As in Fig. 4 of [LZR+ across participants by reporting the 25th percentile (Q1), median, and 75th percentile (Q3).
summarize the probability of correct selection
Findings. Across budgets, our estimator achieves higher probability of correctly identifying the best arm, with the largest gains appearing at smaller budgets; the gap narrows as the budget increases. Full quartile summaries for each budget are reported in Table 3
40

---

Budget 500
Budget = 1000
Method
Q1
Median
Q3
Method
Q1
Median
Q3
Preference Only
0.50
0.67
0.83
Preference Only
0.37
0.56
0.64
LZR+24
0.25
0.43
0.57
LZR+24
0.03
0.10
0.37
Ours
0.03
0.27
0.39
Ours
0.00
0.10
0.26
Budget 1500
Method
Q1
Median
Q3
Preference Only
0.21
0.39
0.50
1.ZR+24
0.00
0.06
Ours
0.00
0.05
0.36
Table 3: Probability of correctly incorrectly identifying the best arm summarized across participants (Q1/Median/(3) for the three total sample budgets.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect paper's contributions and scope?
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
41

---

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
42

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
mublic/guides
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
43

---

Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emoor bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
44

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work is primarily theoretical in nature and has societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not existing assets.
45

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the datasets..paerssttt paperswi thcode. com/datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: [NA]
46

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question:
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy for what should or should not be described.
47


Reviews:
[{"review_id": "75ETuZWrRL", "reviewer": "Reviewer_rfdo", "summary": "This paper considers using the EZ-diffusion model — a model of the binary choice process that models response times — for learning from preferences. The advantage of this model is that the observed response time for a given choice reveals additional information about the reward difference between the two options. The downside is that because both the expected response time and the reward difference must now be estimated, there is a risk that a biased estimate of the expected response time will also bias the estimated reward difference. This paper addresses this by introducing a new Neyman-orthogonal loss function for this setting, thus largely avoiding response time mis-estimates biasing the reward estimate.\n\nThe paper then proves a number of asymptotic and finite-sample bounds on the reward function inference error. Empirical experiments (with synthetic choice generated from the EZ-diffusion model) show that accounting for response times yields better reward inference than a logistic model (which does not model response times), and that the proposed orthogonal loss yields a decent further improvement in inference quality.", "questions": "On line 96: does $Y = +1$ signify a choice for $X^1$ or $X^2$? This is not actually stated.", "limitations": "No specific section on the limitations of the proposed method is present. There are obvious limitations of the work, for example the asymptotic guarantees being limited to linear reward functions, which should have been listed. There are also limitations inherent in using the EZ-diffusion model for choice modeling (such as needing to record response times), which are of course shared with prior work, but should have been listed nonetheless.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The proposed orthogonal loss yields (provable!) improvements in reward inferences from human preferences when response times are available. The loss itself requires little additional work compared to baselines and is therefore an important and valuable contribution to the field.\n\nThe theoretical results help to show (more so than empirical evaluation) the benefits of the proposed loss. The asymptotic analysis only applies to linear reward functions but this is nevertheless an important class of reward functions which is widely used in practice. As someone who works with preference learning and human choice modeling in a more practical capacity, the finite sample analysis in section 5 was a bit beyond my mathematical background, but I do appreciate the headline conclusions here.\n\nThough I have no doubt that the theoretical work done in this paper yield a sufficiently large contribution for acceptance, I do find the empirical experiments to be lacking. The motivation for this paper is to improve learning from human preferences, and data sets of human preference with response times exist (see e.g. the data used in [LZR+24]), so why was that data not used in the experiments here?\n\nFinally, I do think the authors should address the lack of a conclusion (highly unusual) or a discussion on limitations (absolutely necessary).", "was_revised": true, "final_justification": "As stated in my original review, the paper is technically solid and provides valuable theoretical insights. My initial concern was mostly that the method had not been tested on human data, but this has now been addressed by the authors.", "rebuttal": "We thank the reviewer for the feedback. We address the comments and questions below:\n\n1. **Additional experiments based on [LZR+24]**.  \nWe tackle a more general—and inherently harder—supervised learning problem without adaptive query selection, yet even in the simpler adaptive best-arm identification setting, our method outperforms the estimator from [LZR⁺24].  We evaluate our orthogonal-loss estimator on bandit tasks derived from the real-world food-preference dataset [1], which contains choice and response-time data from 42 participants. Following their sequential-elimination protocol (Algorithm 1 in [LZR⁺24]), we substitute our orthogonal estimator for their response time estimator, run 70 independent trials per participant, and then summarize the probability of correctly identifying the best arm across participants (Figure 4 in [LZR+24])  via the 25th percentile (Q1), median, and 75th percentile (Q3) for each total sample budget. Results at budgets 500, 1000, and 1500 are shown below. We observe that our estimator clearly outperforms their proposed estimator in [LZR+24] for smaller budget values and the improvement reduces as the budget increases.\n  ---\n*Budget = 500*\n\n| Method          | Q1   | Median | Q3   |\n|:----------------|:-----|:-------|:-----|\n| Preference Only | 0.50 |  0.67  | 0.83 |\n| LZR+24          | 0.25 |  0.43  | 0.57 |\n| Ours            | 0.03 |  0.27  | 0.39 |\n\n\n\n---\n\n*Budget = 1000*\n\n| Method          | Q1   | Median | Q3   |\n|:----------------|:-----|:-------|:-----|\n| Preference Only | 0.37 |  0.56  | 0.64 |\n| LZR+24          | 0.03 |  0.10  | 0.37 |\n| Ours            | 0.00 |  0.10  | 0.26 |\n\n---\n\n*Budget = 1500*\n\n| Method          | Q1   | Median | Q3   |\n|:----------------|:-----|:-------|:-----|\n| Preference Only | 0.21 |  0.39  | 0.50 |\n| LZR+24          | 0.00 |  0.06  | 0.21 |\n| Ours            | 0.00 |  0.05  | 0.36 |\n---\n\n\n2. **Limitations of our work**.  \nOur method presumes that the drift‐diffusion model (EZ-DM) provides an accurate account of human decision‐making in the target application. When this holds, our orthogonal‐loss framework is highly effective for reward estimation. We recognize that DDMs may not capture every nuance of response‐time variability [4,5]. In particular, the EZ diffusion model assumes a constant drift rate and fixed decision boundaries, yet real‐world fluctuations in attention and focus can violate these assumptions [6]. In scenarios where response‐time data depart substantially from the EZ‐DM’s stationary‐drift or boundary‐separation assumptions, our approach may be less appropriate. There exist extensions that relax these constraints [5], and adapting our loss formulation to such generalized diffusion frameworks is an interesting direction for future work.\n\n3. **Regarding asymptotic guarantees**.  \nOur non-asymptotic bounds in Section 5 already imply finite-sample convergence rates for linear reward models. We further derive asymptotic results in the linear setting to emphasize the exponential pointwise improvement over the preference-only estimator. We will clarify this distinction in the updated version.\n\n4. **\"Does $Y=1$ signify a choice for $X^1$ or $X^2$?\"**  \nThank you for highlighting this. We will revise the text to state explicitly that $Y=1$ indicates that $X^1$ was chosen. \n\n[1] S. M. Smith and I. Krajbich. Attention and choice across domains. Journal of Experimental Psychology.  \n[4] Ratcliff, R., Smith, P. L., Brown, S. D., & McKoon, G. (2016). Diffusion decision model: Current issues and history. Trends in cognitive sciences.  \n[5] Fudenberg, D., Newey, W., Strack, P., & Strzalecki, T. (2020). Testing the drift-diffusion model. Proceedings of the National Academy of Science.  \n[6] Myers, C. E., Interian, A., & Moustafa, A. A. (2022). A practical introduction to using the drift diffusion model of decision-making in cognitive psychology, neuroscience, and health sciences. Frontiers in Psychology."}, {"review_id": "fnj6DBPtsR", "reviewer": "Reviewer_j9qd", "summary": "This paper introduces a new method for learning human reward models by incorporating user response time into the preference learning process. The central idea is that the time a person takes to choose between two options reveals the strength of their preference, a concept modeled by the Evidence Accumulation Drift Diffusion (EZ) model. While traditional methods rely only on the binary choice (which option was preferred), this work leverages the temporal data of the decision to improve the accuracy and efficiency of reward model learning.", "questions": "1. In the orthogonal Loss meta algorithm 1, since they are estimating two more estimators, what is the computational complexity for this algorithm compared to the original loss? Can the authors provide theoretical or experimental evidence?", "limitations": "See weaknesses.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strenghts: This work novelly proposes a new preference model by considering response time, and provide rigorous theoretical proof to derive convergence guarantees. Moreover, they proposes an orthogonal loss function and successfully turn from exponentail dependence on the reward range to logarithmic.\n\nWeaknesses: \n1. The intuition and the connection between their proposed preference learning with response time and RLHF are not so obvious. Since in RLHF, a standard formulation is bt model. In the author's setting, what is the distribution of the response time and why would the probability of one response is preferred than another grows with the response time. The authors should give RLHF examples to justify the reasonableness of their setting.\n\n2. Besides, their experiments are on bandits, but their theory also lacks a connection to the bandit problem. I understand that the novel regression is their contribution, but since their setting is in RLHF. It would be better if they provide bandit regret bounds in the theoretical part. Also, it is interesting if the response time and be combined with the online decision making process.", "was_revised": true, "final_justification": "The authors basically answer my questions, so I will keep the score.", "rebuttal": "We thank the reviewer for the feedback. We address the comments and questions below:\n\n1. **Regarding intuition and connections to the Bradley-Terry model and example application**.  \nNote that the EZ diffusion model naturally incorporates the Bradley Terry model for preferences (see equation 1 in the paper). Crucially, the EZ diffusion model allows to exploit the response time information in learning the reward model. In practical settings, the response time can be cheaply recorded along with preferences and are indicative of preference strength (see the discussion on lines 43-54 for a detailed discussion). A key application is fine-tuning diffusion models with human preference data [c], which motivates our text- and image-based experiments in Figure 3.\n\n2. **Regarding the distribution of response time**.  \nOne of the core contributions of our paper is handling the intractable likelihood of the response time. The likelihood of the response time is reduced to an infinite sum [a], and prior works approximate with either a parametrized heavy-tailed distribution [b] or series truncation [a]. In contrast, our Neyman-orthogonal loss entirely obviates likelihood computation, relying only on the closed-form conditional expectation. \n\n3. **Regarding bandit regret bounds**.  \nOur framework operates in a standard supervised-learning setting rather than an adaptive bandit environment. In our experiments, we evaluate our method by the MSE of reward estimation, and—as a proxy for downstream tasks—compare the value of the naïve policy that always picks the action with the highest estimated reward.  \nThis crucially differentiates us from prior work in [LZR+24] and addresses a major drawback of the prior approach. We tackle a more general—and inherently harder—supervised learning problem without adaptive query selection. The queries are assumed to be coming from an unknown distribution. While one could extend our approach to active query selection (e.g., optimal experiment design) and recover a bandit‐style analysis in the linear case—yielding guarantees similar to [LZR+24]—this is beyond the paper’s current scope. In practice, even for the case of the best arm selection problem, we observe that our method generally has a superior performance to [LZR+24]. For a more detailed discussion, please see our response to reviewer rfdo.\n\n4. **Regarding computational complexity**.  \nOur meta‐algorithm supports any function class and gradient‐based optimizer. Its runtime scales with the chosen optimizer, the model class, and the dataset size, but remains asymptotically identical to the preference‐only estimator. In particular, when instantiated with a linear reward model, the optimization is convex and thus solvable in polynomial time. Empirically, for neural network experiments, nuisance estimation and orthogonal-loss minimization require similar runtimes.\n\n[a] Navarro, D. J., & Fuss, I. G. (2009). Fast and accurate calculations for first-passage times in Wiener diffusion models. Journal of mathematical psychology.  \n[b] Shvartsman, M., Letham, B., & Keeley, S. (2023). Response Time Improves Choice Prediction and Function Estimation for Gaussian Process Models of Perception and Preferences.  \n[c] Wu, X., Sun, K., Zhu, F., Zhao, R., & Li, H. (2023). Human preference score: Better aligning text-to-image models with human preference. In Proceedings of the IEEE/CVF International Conference on Computer Vision."}, {"review_id": "AgHTU6V3oC", "reviewer": "Reviewer_iuYD", "summary": "This paper considers a new modeling approach for preference learning by adding a response time feature into the Bradley-Terry framework. Leveraging EZ diffusion, a Neyman-orthogonal loss function is proposed, and theoretical convergence is derived under both linear reward and general reward scenarios given good nuisance estimation. Experiments on linear, neural-network-based, and synthetic text-to-image reward models show lower MSE/regret with the proposed loss.", "questions": "Please refer to weaknesses.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "*Strengths.*\nThe theoretical response-time incorporation is well-motivated and supported by the EZ diffusion framework.\nThe Neyman-orthogonal loss reduces the error during nuisance estimation by zeroing out the first-order impact during parameter estimation and also demonstrates better convergence rates than the logistic loss in theory.\nTheoretical results are strong and hold for both linear cases and nonparametric reward classes with either asymptotic or finite-sample guarantees.\n\n*Weaknesses.* \nWhether the theoretical framework can be used to describe real-world preferences is not validated, and thus questions the scope of the methodology for application. A plug-in strategy is described in the paper but not validated in realistic setups.\nWhen the theoretical framework (EZ diffusion) fits, in theory, the convergence guarantee for both linear/general reward functions requires accurate nuisance estimation as reflected in Eq. (8) and the (R, $\\alpha$) norm in Eq. (9). This questions the degree of practical benefits of the proposal due to the strong reliance of a good preliminary estimator of the true reward function. Furthermore, this makes the comparison between the ortho-estimator and log-loss estimator unfair for both theory and experiments, since sometimes the former uses the latter as an initialization, as discussed in Sections 4 and 6.", "was_revised": true, "final_justification": "I have looked over the responses to my own questions and identified weaknesses, as well as the other reviews and their responses, which have partially adressed my concerns. I am thus raising my score over the accept line to 4.", "rebuttal": "We thank the reviewer for the feedback. We address the comments and questions below:\n\n1. **Regarding accurate estimation of Nuisance functions**.  \nOne of the core advantages of our Neyman‑orthogonal loss is that it is first‑order insensitive to errors in both nuisance functions (the preliminary reward estimate and the response‑time model). In fact, the estimate of the initial reward function can be arbitrarily poor if the estimate of $t_o(X)$ is $\\sqrt{n}$ consistent. \nFurthermore, Theorem 5.1, implies that we only need slow rates for the nuisance functions to achieve the oracle rates; nuisance estimators can converge at rates orders of magnitude slower than the oracle rate.  For example, for certain RKHS settings, one still obtains a $\\frac{1}{\\sqrt{n}}$ convergence rate even when the preliminary reward estimate and the response‑time estimate both converge at a much slower rate of $\\frac{1}{n^{1/4}}$.\nSimilarly, in equation (8) a $\\frac{1}{n^{1/4}}$ rate suffices and can be easily achieved [7] (see the discussion on lines 220-224).\n\n2. **Regarding fair comparison of log-loss estimator and ortho-estimator**.  \nOur goal is *not* to argue that the log‑loss estimator is inferior—when only binary choices are available, log loss is indeed the minimax‐optimal approach. Rather, our focus is on *incorporating response‑time data*, which the log‑loss objective cannot accommodate. We include the log‑loss estimator purely as a baseline to quantify the gain from response‑time augmentation. Initializing our orthogonal estimator with the log‑loss solution speeds up convergence but confers no statistical advantage: all performance improvements stem from the additional information in T.\n\n3. **Regarding scope of the methodology**.  \nOur work targets large‑scale  machine learning —characterized by high dimensional input and large datasets. To emulate real‑world conditions at this scale, we use semi‑synthetic data targeting this regime and work with image and text datasets. Fully real data experiments in this regime would be prohibitively expensive.  \nAt its core, our work is driven by the practical challenge of integrating drift‑diffusion models into modern, large‑scale machine‑learning pipelines. Whereas classical preference learners (e.g. Bradley–Terry) rely on a tractable likelihood and plug directly into gradient‑based training via maximum‑likelihood estimation, the DDM’s full likelihood is intractable—making it unclear how to exploit response time at scale. Our paper addresses this important gap.  \n  Our method presumes that the drift‐diffusion model (EZ-DM) provides an accurate account of human decision‐making in the target application. When this holds, our orthogonal‐loss framework is highly effective for reward estimation. We recognize that DDMs may not capture every nuance of response‐time variability [4,5]. In particular, the EZ diffusion model assumes a constant drift rate and fixed decision boundaries, yet real‐world fluctuations in attention and focus can violate these assumptions [6]. In scenarios where response‐time data depart substantially from the EZ‐DM’s stationary‐drift or boundary‐separation assumptions, our approach may be less appropriate. There exist extensions that relax these constraints [5], and adapting our loss formulation to such generalized diffusion frameworks is an interesting direction for future work.\n \n\n[4] Ratcliff, R., Smith, P. L., Brown, S. D., & McKoon, G. (2016). Diffusion decision model: Current issues and history. Trends in cognitive sciences.  \n[5] Fudenberg, D., Newey, W., Strack, P., & Strzalecki, T. (2020). Testing the drift-diffusion model. Proceedings of the National Academy of Science.  \n[6] Myers, C. E., Interian, A., & Moustafa, A. A. (2022). A practical introduction to using the drift diffusion model of decision-making in cognitive psychology, neuroscience, and health sciences. Frontiers in Psychology.  \n[7] Ji, Ziwei, and Matus Telgarsky. \"Risk and parameter convergence of logistic regression.\" arXiv preprint arXiv:1803.07300 (2018)."}, {"review_id": "bKHwzoWVUK", "reviewer": "Reviewer_iNrS", "summary": "This paper studies an interesting topic where the response time is included for preference learning. It is modeled by the Evidence Accumulation Drift Diffusion, where the response time inversely reflects the preference strength. Based on this model, the authors develop Neyman-orthogonal loss functions and prove that it enjoys oracle convergence rates for linear reward models. The proposed loss function also empirically outperforms log-loss and non-orthogonal loss on different settings: linear reward model and MLP reward model on synthetic dataset; MLP reward model on a text-to-image dataset.", "questions": "please see above", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "strengths:\n\n- this paper is well-motivated: response time may encode preference strength and we should leverage that during preference learning\n- the theoretical analysis on the proposed orthogonal loss and comparison to non-ortho loss is clear and sound\n- the empirical experiment design is clean, and the results support the authors' claim\n\nweakness (more like questions):\n\n- since the orthogonal loss requires a preliminary estimator of the true reward function, its estimation error also matters. around line 237, when the preliminary estimator is poorly estimated, you then need an accurate estimate of t to make the product of errors small? i'm wondering how largely this assumption can be true?\n- the experiments mostly use simple functions (linear or MLP) and only tracks the loss. in many real world application (e.g., LLM preference learning), we use large transformers for reward modeling. I am wondering if the proposed method can be applied there and how much improvement we can expect over the traditional BT model, especially for the down-stream performance that we really care - the model behavior after preference-tuning?", "was_revised": true, "final_justification": "The authors' rebuttal answers my first question. I still have concerns on the scalability of the method given the presented experiments in the paper and authors' rebuttal. I hence maintain my weak accept decision.", "rebuttal": "We thank the reviewer for the feedback. We address the comments and questions below:\n\n1. **Regarding estimation of nuisance function and robustness of preliminary reward error**.  \nNote that, since our Neyman‑orthogonal loss is first‑order insensitive to errors in both nuisance functions (the preliminary reward estimate and the response‑time model), one can tolerate significantly slower convergence in the preliminary estimator. For example, an $n^{-1/4}$ rate suffices to achieve $n^{-1/2}$ error rate in reward estimation for various function classes (such as RKHS or parametric models).  \nMoreover, as the reviewer points out, the loss is in fact robust to estimation of the preliminary reward function.  In particular, because the preliminary‑reward error only enters multiplied by the error in $t_o$, one may have arbitrarily slow convergence for the preliminary reward estimator so long as the response‑time estimator achieves an $n^{−1/2}$ (or the oracle convergence rate). For example, see the case of linear reward (line 219 (8)). \n\n2. **Regarding performance in downstream tasks**.  \nIn our experiments, in addition to MSE loss, we also measure reward‐estimation quality in terms of policy value (see lines 288–289). Although this policy simply selects the option with the higher estimated reward, it serves as a proxy for how a large‑scale policy (e.g., a generative model) would behave after preference tuning. Figures 2 and 3 present these results, showing a clear improvement in policy value."}, {"review_id": "0gaOX1dpmW", "reviewer": "Reviewer_DNMn", "summary": "The authors provide a method to incorporate response time for binary human preference learning, focusing on using basic drift diffusion dynamics to better learn a latent preference function. The EZ drift diffusion model provides expectations of binary responses as a function of stimulus input and reward ($\\mathbb{E}[Y|r(X)]$) as well as expectations of response time given input ($\\mathbb{E}[T|X]$), allowing for the incorporation of these moments into a preference learning framework. The authors introduce a new neyman-orthogonal loss function to efficiently estimate the latent preference function given response times. They demonstrate better estimates of true underlying reward function using their response-time informed model with their orthogonal loss function using a variety of simulated and semi-simulated datasets. They provide good theoretic properties when reward functions r(X) are linear as well as sampling bounds in general for their estimator. Their theoretical guarantees are impressive and provide an important contribution to this research space.", "questions": "My primary question with this work is how it adapts to a GP-based non-parametric logistic settings and in particular the response time GP model mentioned above.\n\nEvaluation on real human response-time data would also increase the impact of this paper. The results of the paper (including the semi-simulated datasets evaluation) implicitly rest on the EZ diffusion time being an accurate model of human response times, but it may not be, especially in certain types of preference learning settings. However, given the large theoretical contribution of their simple assumptions here, is in my opinion not absolutely necessary to evaluate on real response-time data to warrant publication.\n\nOne smaller question I have is to what extent the data-splitting and cross-fitting approaches effect the model outcomes. Are the results shown in the figures in the main paper data-splitting or cross-fitting? Some clarity/elaboration here would be helpful.", "limitations": "Some more discussion about the limitations of the method, especially with respect to types of datasets or failure modes of optimization would be helpful. Are there regimes where the overall estimation of r(X) fails? What are the limitations of the EZ diffusion model in capturing real human response times?", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is clear and addresses an important set of questions in human preference learning. The analysis is thorough and the main text and especially supplement provides many mathematical properties of their Meta algorithm under data-splitting and cross-fitting approaches. The results on simulated data are compelling and demonstrate the utility of the Neyman-Orthogonal objective, which is an interesting application that is highly applicable in this setting.\n\nHowever, two of their core contributions, using the DDM to integrate with binary responses and extension to non-parametric reward functions have been previously implemented in a binary preference learning with response time setting (See Shvartsman et al.*) In this paper, the authors very similarly use the properties of the drift diffusion model which has closed-form expressions of moments of the response time distribution as a function of the drift parameter to better estimate a latent preference function. This work was borne out of GP models for human preference learning that did not use response times **, ***. Lack of discussion and comparison with this work as well as the context of this literature severely limits it's impact, as the works are highly related. The primary important distinction is the authors in shvartsman et al only consider non-parametric (GP based) preference functions, but they additionally consider pairwise comparisons with a specific pairwise GP kernel. Comparing their approach the one outlined here is therefore important. Also, the title of this paper \"Preference Learning with Response Times\" I believe should be changed in light of the fact that this has been done before in Shvartsman et al. The Neyman orthogonal objective and assumption of EZ diffusion process allows for a greater amount of mathematical developman than in Shvartsman et al, and this should be emphasized.\n\nThere is no evaluation on real human reaction time data. This makes it difficult to know how reasonable it is to assume the EZ diffusion model holds in a variety of real world preference learning settings. \n\n*Response Time Improves Gaussian Process Models for Perception and Preferences  - Shvartsman, et al. UAI 2024. \n** A semi-parametric model for decision making in high-dimensional sensory discrimination tasks. Keeley et. al AAAI 2023\n*** Bayesian Active Model Selection with an Application to Automated Audiometry - Gardner et al. NIPS 2015", "was_revised": true, "final_justification": "With the proposed edits I have adjusted my score accordingly. I thank the authors for their thorough response to my concerns. \n\nThe authors have included a comparison with the relevant manuscript, so I have further adjusted my score.", "rebuttal": "We thank the reviewer for highlighting the important work of Shvartsman et al. While our contributions are distinctly different, we agree that a direct comparison belongs in our work, and we will update our paper with a dedicated comparison with their work. In light of this prior work, we will update the title of the paper to \"Robust Losses for Preference Learning with Response Time\" in the final version. \n\nWhile both papers leverage closed‑form DDM moments, our contributions and intended applications are complementary. Specifically:\n\n\n1. Scalability to high-dimensional inputs.  \nOur setting is motivated by large-scale “learning from human feedback” pipelines—e.g., fine‑tuning large language or vision models—where inputs live in very high dimensions and downstream tasks require a pointwise reward estimate. Shvartsman et al. use a GP prior on rewards and develop a variational approximate Bayesian inference approach using moment matching to refine the posterior so as to incorporate response time information. In particular, the paper suffers from the standard issues with GP regression and does not scale to high-dimensional (curse of dimensionality in GP regression [1,2,3]) and is not suitable for large-scale machine learning models.  \nWe completely bypass the use of variational approximations and moment-matching techniques as our method avoids modeling the full likelihood altogether.   We develop a statistical learning theory approach with a Neyman orthogonal (aka locally robust) loss function that allows for flexible function approximation that can scale large input dimensions and function spaces. \n\n2. Theoretical guarantees for reward learning.  \nWe are the first to show that incorporating response times can provably reduce estimation error in reward models in a supervised learning setting. In doing so, we also bridge a gap in prior DDM parameter‑estimation literature and modern learning from human feedback settings by deriving a loss function that admits finite‑sample error bounds for a wide range of function classes.\n\n3. Function‑class‑agnostic method.   \nOur orthogonal loss wraps around any reward learner—neural nets, kernels, or even Bayesian models—allowing practitioners to plug in any approximator and optimize via standard gradient methods.  This agnosticism also paves the way to online learning (e.g., logistic bandits) or implicit reward‑learning (direct policy optimization as described in Appendix A) as future directions.\n\n\n**Regarding adaptability to the response time model in Shvartsman et al**.  \n  \nAside from an initial‑state offset $x_o$ of the diffusion process, the DDM in Shvartsman et al. is mathematically equivalent to ours. We assume that the queries are symmetric, that is, swapping the queries $X^1$ and $X^2$ does not affect the model.  In fact, enforcing symmetry—$\\text{Pr}(Y=1│X_1=a, X_2 = b) =1–\\text{Pr}(Y=1│X_1= b, X_2 =a)$—forces $x_o=0$, yielding exactly the DDM formulation (EZ- diffusion model) in our work. Although our current work does not require it, our orthogonal-loss construction can be extended to accommodate a nonzero initial-state offset.\n\n**Regarding different approaches for nuisance estimation: data-splitting, cross-fitting, data reuse**.  \nIn the experiments for linear functions, we use data splitting (highlighted in the linear paragraph section in Appendix D) in the main paper. For the case of linear rewards, our preliminary results using data-reuse and cross-fitting show that data-reuse consistently outperforms both cross-fitting and standard data-splitting. In the paper, to demonstrate the benefit of orthogonal loss over preference only estimator for the linear case, we stuck to the more conservative approach of data splitting. We will include a more detailed discussion in the appendix of the experimental section. We ran our neural network experiments with data reuse and data splitting. We consistently observed that data reuse outperformed the data splitting estimator.\n\n\n\n**Regarding methodology scope and limitations**.  \nOur method presumes that the drift‐diffusion model (DDM) provides an accurate account of human decision‐making in the target application. When this holds, our orthogonal‐loss framework is highly effective for reward estimation—especially at modern, large scale—but it is important to recognize that DDMs may not capture every nuance of response‐time variability [4,5]. In particular, the EZ diffusion model assumes a constant drift rate and fixed decision boundaries, yet real‐world fluctuations in attention and focus can violate these assumptions [6]. In scenarios where response‐time data depart substantially from the EZ‐DM’s stationary‐drift or boundary‐separation assumptions, our approach may be less appropriate. There exist extensions that relax these constraints [5], and adapting our loss formulation to such generalized diffusion frameworks is an interesting direction for future work.\n\n[1] Giordano, M., Ray, K., & Schmidt-Hieber, J. (2022). On the inability of Gaussian process regression to optimally learn compositional functions. Advances in Neural Information Processing Systems.  \n[2] Binois, M., & Wycoff, N. (2022). A survey on high-dimensional Gaussian process modeling with application to Bayesian optimization. ACM Transactions on Evolutionary Learning and Optimization.  \n[3] Krauth, K., Bonilla, E. V., Cutajar, K., & Filippone, M. (2016). AutoGP: Exploring the capabilities and limitations of Gaussian process models.  \n[4] Ratcliff, R., Smith, P. L., Brown, S. D., & McKoon, G. (2016). Diffusion decision model: Current issues and history. Trends in cognitive sciences.  \n[5] Fudenberg, D., Newey, W., Strack, P., & Strzalecki, T. (2020). Testing the drift-diffusion model. Proceedings of the National Academy of Science.  \n[6] Myers, C. E., Interian, A., & Moustafa, A. A. (2022). A practical introduction to using the drift diffusion model of decision-making in cognitive psychology, neuroscience, and health sciences. Frontiers in Psychology."}]

Author rebuttal:

Meta-review: The reviewers agreed that the paper studies an interesting problem of modeling the response time along with binary preference within frameworks of learning from human preferences. The paper introduces a novel methodology to incorporate response time, and demonstrates its effectiveness in eliciting a more effective reward model through theoretical analysis and experiments. However, the reviewers also raised several concerns and questions in their initial reviews, particularly regarding the limited experimental evaluation. We want to thank the authors for their responses and active engagement during the discussion phase. The reviewers appreciated the responses, which helped in answering their key questions. The reviewers have an overall positive assessment of the paper, and there is a consensus for acceptance. The reviewers have provided detailed feedback, and we strongly encourage the authors to incorporate this feedback when preparing the final version of the paper.
Decision: Accept (poster)