Doubly Robust Alignment for Large Language Models
Authors: Erhan Xu
Kai Ye
Hongyi Zhou
Luhan Zhu
Francesco Quinzan
Chengchun Shi
Venue: NeurIPS 2025 poster ()
Keywords: Large language models
Doubly robust methods
Reinforcement learning from human feedback
Semiparametric efficiency
Primary area: general_machine_learning

Abstract: This paper studies reinforcement learning from human feedback (RLHF) for aligning large language models with human preferences. While RLHF has demonstrated promising results, many algorithms are highly sensitive to misspecifications in the underlying preference model (e.g., the Bradley-Terry model), the reference policy, or the reward function, resulting in undesirable fine-tuning. To address model misspecification, we propose a doubly robust preference optimization algorithm that remains consistent when either the preference model or the reference policy is correctly specified (without requiring both). Our proposal demonstrates superior and more robust performance than state-of-the-art algorithms, both in theory and in practice. The code is available at https://github.com/DRPO4LLM/DRPO4LLM

Doubly Robust Alignment for Large Language Models
Erhan Xu" Department of Statistics LSE London, UK
Kai Ye" Department of Statisties LSE London, UK
Hongyi Zhou' Department of Mathematics Tsinghua University Beijing, China
Luhan Zhu
Francesco Quinzan
Chengchun Shit Department of Statistics LSE London, UK
School of Design LCC, UAL
Department of Engineering Science University of Oxford Oxford, UK
London, UK
Abstract
This paper studies reinforcement learning from human feedback (RLHF) for align- ing large language models with human preferences. While RLHF has demonstrated promising results, many algorithms are highly sensitive misspecifications in the underlying preference model (e.g., the Bradley-Terry model), the reference policy, the reward function, resulting in undesirable fine-tuning. To address model misspecification, we propose doubly robust preference optimization algorithm that remains consistent when either the preference model the reference policy is correctly specified (without requiring both). Our proposal demonstrates superior and more robust performance than state-of-the-an algorithms, both in theory and practice. The code is available at https: //::::/gitttu github. com COtp/DRPOOLLM/DRREE
Introduction
Recent advances in large language models (LLMs) have revolutionized various natural language processing tasks, ranging from text generation to human-Al conversation and more complex reasoning tasks [TH]]. These models are typically trained in stages. In the pre-training stage, LLMs learn general linguistic patterns and commonsense knowledge from vast, unlabeled text data through autoregressive next-token prediction. However, pretrained models face a critical objective mismatch: while they are optimized for token prediction, real-world deployment requires aligmment with complex human values such as helpfulness, honesty and harmlessness 44. This mismatch calls for an additional post-traie must-training stage, aiming at better aligning these pre-trained models with human preference.
The paper studies reinforcement learning from human feedback (RLHF). a post-training paradigm that adapts pre-trained models through reinforcement learning [RI.5]] The RL.FF literature has rapidly expanded in recent years, where existing algorithms or preference-based (Section for review). While demonstrating remarkable success in domains including roboties control, video games, and LLMs fine-tuning [see e.g., 6-10), they often suffer from various model misspecifications (see also Table T for a summary):
be broadly categorized reward-baid based
1. Preference model misspecification. Most reward-based algorithms rely on the Bradley-Terry (BT, preference model (see [quation[]] However, this model entails various unrealistic assumptions
* 'Erhan Xu, Kai Ye, and Hongyi Zhou contributed equally to this paper and listed in alphabetical order. $ Francesco Quinzan and Chengchun Shi are joint senior contributors and are listed in alphabetical order. Address for correspondence: Francesco Quinzan, Ph.D., francesco quinzan@eng, ox ac. uk: Chengchun Shi, Ph.D., c. shw7@1se.e.....
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

fire
Optimal Policy
Query
Seturnace Delloy
Y
Suboptimal Polley
Figure A visualization our proposed preference optimization algorithm. jet denotes the specified reference policy whereas g denotes the specified preference model. Our proposal is doubly robust that it requires correct specification of either the reference policy, or the preference model.
on human preference, including transitivity, context-independenne and perfect relationality, which are likely violated based on empirical evidence [2 While some preference- based algorithms impose more general preference model (GPM) assumptions [see e.g., 19], their effectiveness still depends on correct model specification.
Reward model misspecification. Under the BT model assumption, classical reward-based algorithms first estimate the reward function from human preference data and then apply RL algorithms such as the proximal policy optimization [PPO, 20) to derive the optimal policy. However, policy learning through RL is highly sensitive to the estimated reward. Misspecifying the reward can lead to reward hacking (2 22 and misguide policy learning 25]
Reference policy misspecification. To alleviate misspecification of the reward, recent algorithms based on direct preference optimization [DPO,26] propose to express the reward in closed form using the reference policy for policy learning. However, these algorithms are sensitive to the specification of reference policy 27 29
Drawing from doubly robust estimation methods in econometrics and RL (see Section for literature review), we introduce a novel RLHF algorithm that is robust to model misspecification and statistically efficient; Figure for visualization of our algorithm. Our major contributions summarized as follows:
We propose robust and efficient estimator for preference evaluation, i.c., evaluating the probability of target policy being preferred over the reference policy. The proposed preference estimator achieves two desirable properties: double robustness (Corollary 3 it converges to the true preference probability when either the preference model or the reference policy is correctly specified, and (ii) semi-paramedd
efficiency (Corollary
attains the smallest mean squared error (MSE)
among all regular and asymptotically linear estimators [1].
Leveraging this preference estimator, we further develop preference optimization algorithm for LLM fine-tuning. The proposed algorithm maintains double robustness (Corollary (6) and remains consistent even when the BT model assumption is violated (Theorem Meanwhile, when the BT model assumption holds, its suboptimality gap likely smaller than that of PPO- or DPO-based algorithms (Theorem
Related Works
Our work is closely related reward- and preference-based RLHF algorithms, well as doubly robust (DR) methods. We discuss these related works below.
Reward- RLHE Reward-basee algorithms assume the existence of a latent utility reward function that determines human preferences, estimate the reward function from the data and apply RI. for policy learning. Recent research has focused addressing practical challenges such reward

---

These issues arise due to PPO's sensitivity to reward specification, gradient clipping thresholds, and the tuning parameter controlling Kullback-Leibler diverriieeeeecc Existing approaches to these challenges fall into three categories: (i) The first category focuses on improving the reward learning algorithm to obtain more accurate reward functions [4]40]. (ii) The second category develops better policy learning algorithms using the estimated reward function [41447]. (ii) The third category is apO-based, which bypasses reward learning entirely and directly optimizes policies under the BT model assumption [48)52). Recent studies have developed robust variants of DPO to handle pairwise noise where preference labels in the training data may be flipped [53 55].
hacking and
regularization [32 24 33
Unlike many of these algorithms,, our proposal does not rely on the BT model assumption, and is more robust to the misspecification of reward or reference policy when the BT model holds.
Preference-basee RLHF, Preference-based algorithms do not assume the existence of latent reward function at all; instead, they search the optimal policy that maximizes the alignment with human preferences [see e.g., [56]. In particular, there is a growing line of research that adopts the Nash learning from human feedback INLHF, (5] framework, which formulates the alignment problem as a two-player constant-sum game and solves for policies that achieve the Nash equilibrium (58-64). Beyond NLHF, [65] develops a Bayesian approach for aligmment, whereas [66] and 19 propose energy- -ased and general preference models to relax the BT model assumption.
Our proposal belongs to this class of preference-baie based methods. In particular, the proposed algorithm is most closely related the identity preference optimization (IPO) algorithm proposed by Azar et al. [56]. as both maximizing the same objective function in the population level (see Section 4 for the objective). However, unlike IPO, our proposed method is robust to misspecifications of the reference policy. Similarly, compared to 19), the proposed algorithm is more robust to the misspecification of the preference model. Finally, our work differs from NLHF in its primary focus: we study robust and statistically efficient preference estimation from data, rather than developing computationally efficient algorithms solve the Nash equilibrium.
Doubly robust methods. DR has been extensively studied in statistics, econometrics and machine learning. These methods originate from the missing data and causal inference literature [see e.g..
67 68). To illustrate these methods, consider the fundamental causal inference problem of estimating the average treatment effect (ATH) the difference in the mean outcome between newly-developed treatment policy and a baseline policy for a given patient population. DR first estimates two models from the data: (i) propensity score model (similar to the reference policy in LLMs) that characterizes the treatment assigmment mechanism and (ii) an outcome regression model (similar to the reward function) that specifies the conditional mean function of a patient's outcome. It then employs both models to construct the ATE estimator, whose consistency requires only one of the models to be correct. Furthermore, when both models are correct, the resulting estimator semiparamettriallly efficient [69]. These methods' favorable statistical properties have led to extensive follow-up research [see e.g., 70. A seminal extension appears in Chernozhukov et al. 933. which proposes to learn both the propensity score and outcome regression models using machine learning methods to deal with complex data structures with high-dimensional covariates, texts or images.
Beyond treatment effect estimation in causal inference, doubly robust methods have been widely applied broad range of other problems, including the estimation and evaluation of optimal (dynamic) treatment regimes [94|]005, conditional independence testing [106-111], offline policy learning 117 and off-policy evaluation [OPE, 113
However, none of the aforemennioned works considers the application of fine-tuning LLMs - a gap aim to bridge by connecting these two vibrant research areas. One exception is Lin et al. [140]. who considered settings with observed numerical outcome for each answer, rather than the standard RLHF paradigm in which a preference is given comparing two completions.
3 RLHF Preliminaries: Data, Modeling and Baseline Algorithms
Data generating process. Assume we are given dataset D, consisting of n i.i.d. tuples of the form (X, (1), (2), 2). Each of these tuples is generated as follows: Given a prompt X. two independent responses (1) (2)) are generated under a reference policy Ter such that (L), (()) ~
Tiet [ ()). These data (x, (1), (2) are then shown to a human expert, who provides binary
3

---

Table 1: Robustness of different algorithms to model misspecification. Our algorithm is denoted by DRPO, short for doubly robust preference optimization.
Robust misspecified:
preference model
reward model
reference policy
PPO-based DPO-based
X
Reward-based
RLHF
IPO [56]
Preference-based
GPM 19
DRPO
preference = [( y(2) where Y (()) indicates that the first response is preferred, and I(+) denotes the indicator function. Additionally, let g" denote the preference function such that (X, 2()) ((Y > (x) determines the probability of Y ( being favored over (2)
conditional on X.
We remark that the reference policy Ter is not always known. For instance, the responses might be generated by an LLM different from the target model that we wish fine-tune [8. Furthermore, the responses might be produced by a heterogeneous set of models rather than single model [14] 143
BT model. As commented in Section 2 most existing reward-based RLHF algorithms impose the BT model assumption, which requires the preference function g to take the following form,
(x,y (2)) = (rr
r" (y(2),x))
where r* denotes some underlying reward function that measures how well response answers given prompt, and a denotes the sigmoid function. As commented in the introduction, this assumption likely violated due to the inherent intransitivity, inconsistency and stochasticity in human preference.
Assuming TT) holds, the goal learn an optimal policy n² that maximizes the expected reward
J(x) ( xxy""((("">""">>" (y, X)),
among all policies Here, the outer expectation is taken with respect the prompt distribution, whereas the inner expectation taken with respect to the response generated by a given policy m.
We next introduce two types of baseline algorithms- - PPOO-ased and DPO-based - for learning T². Both approaches operate under Assumption (I)
PPO-based approaches. PPO-based algorithms proceed in two steps. In the first step, they compute an estimated reward function f using maximum likelihood estimation empirical risk minimization. In the second step, they learn x:" by maximizing
Ex~D, (x)()))) [Fly, X)] 3 DKL [x X) Trer(y/ X)),
over π II (e.g., a transformer-based policy class), where the expectation is taken over prompts X from the empirical data distribution and responses y from a target policy DkL, denotes the KL divergence measure between the target and reference policies, and the tuning parameter > controls the degree to which π is allowed to deviate from Nee. The KL regularization term in encourages the learned policy to stay close to Tut. in order to mitigate ever-fitting and prevent the leamed policy from collapsing to narrow set of high-reward responses 24
DPO-based approaches. DPO-based algorithms are motivated by the fact that the argmax to (3) (denoted by 7)) can be represented in closed-form using the estimated reward F. This in turn yields the following closed-form expression for F,
7(v|=) TTe (z)))
Fly, x) = log
C(x),
(4)
for some perponse-indepeddeen function C(x) that will cancel out in pairwise comparisons. As such, instead of solving â in two steps, DPO-based approaches directly parameterize the reward via Equation (4) and compute
a single step for example, by maximizing the likelihood of the human preference data under the BT model.
To conclude this section, we note that, as shown in Equation 3 the optimal policy computed by PPO can be highly sensitive to the estimated reward function P. While DPO-based approaches

---

eliminate this dependence. Equation reveals that their optimization relies on the specification of the reference policy rer- Due these sensitivities, even under the idealized setting where the BT model holds, both PPO- and DPO-based algorithms can underperform our proposed algorithm, which inherently more robust to misspecification both P' and Tret- We provide theoretical justification in Section and empirical validation in Section 6
4 Double Robust Preference Evaluation and Optimization
This section introduces the proposed doubly robust approach; see Figure for a visualization. Different from these reward-basce algorithms discussed in Section[3 we adopt preference-basee approach that searches the optimal policy by maximizing its total preference. Specifically, given target policy n, its total preference the reference policy [56] is defined by
p (T) == P( > ( = XXXp)=r((\X))x//>amaa
(X, 9/)
where we recall that g" denotes the preference function Ply > y((X), and the outer expectation is taken with respect to the prompt distribution. As both (()) and ( are generated under The we have
p" (a( - - Σ (( (X,,,
(S)
For preference evaluation, goal is to accurately estimate p" (π) for given target policy from the dataset D. In the following, we first introduce two baseline estimators: direct method (DM)
estimator and importance sampling (IS) estimator, where the names borrowed from the OPE
literature [see e.g., [44]. We next introduce our proposed DR estimator, which combines both DM
and IS for efficient and robust preference evaluation.
DM estimator. The direct method estimator is motivated by (5). It proceeds by first estimating g" and then plugging the estimated g" (denoted by g) into (5) construct the estimator,
PDM( 1 ( (X. y()) T(X, y, Y(2)))
(6)
where X is drawn from the empirical data distribution, y drawn from ı and the expectation be approximated using Monte Carlo sampling.
When an external preference model is available, it can be used directly G. as in (57 Otherwise, g can be estimated from the data D. For instance, under the BT model assumption, one can estimate reward function r* and plug the estimator into (1 to derive y. Alternatively, one can employ more general preference models that do not rely on the BT model.
IS estimator. The second baseline estimator is the IS estimator, which is motivated by the following lemma that expresses p" (ir) using the IS ratio w(y, x) = π (y]x))/ /ee x.)
Lemma 1. Assume wly x) < oo for any x, y. Then p² (T) BB (1))) X2 + (2))((2))))(( Z)].
The proof Lemma T is straightforward. follows directly from the symmetry of pairwise comparisons where the preference can be equivalently expressed using either ("(. y, v/) or g" (X(,, v), and an application of the change-of measure theorem (see Appendix A.I
Based on this identity, we define the following IS estimator:
1 (T)( - Bxxyr
((Y 7 ((Y (X) Weer (Y()))
The (Y (1 2
where से denotes an estimated reference policy. If Tef is known, we can directly use the oracle reference policy. Otherwise, for some external datasets [e.g..8). well-trained reference models are available and can be used as fe.- Finally, when no such external model is available and Tie is unknown, we estimate it from the observed data tuples (X, ( (()) using supervised fine-tuning (SFT).
DR estimator. A closer look Equations 6 and 7 reveals that the DM and estimators' consistencies depend crucially on the correct specification of the preference function and the reference
5

---

Doubly Robust Preference Evaluation
Doubly Robust Preference Optimization
3 3
Theorem 2
Corollary 4
Corollary
Theorem 5
Theorem
Doubly evaluation
Non-asymptotic MSE bound
Semi-parametrio efficiency
Doubly Non-asymptotic performance bound
suboptimalitty gops
optimization
Figure 2: visualization of our theoretical findings.
policy. We next introduce our proposed DR estimator, which is more robust to misspecifications in these models. relies on the following estimating function (X, y()), ((2), Zym, n, meef, 9). defined as
1 E 2
TY X) Nel (Y X
Y())
IZ o(X, y()), y())
(8)
By definition, this estimating function contains two terms: (i) the first term is essentially the estimating function of the DM estimator in (6). and (ii) the second term is augmentation term, which is similar to IS in 7 but with the observed preference Z replaced by its residual Z 0(X, ( y()). The purpose introducing the additional augmentation term to correct for the bias introduced by misspecification of the preference model in the DM estimator. This leads to our DR estimator,
(T) Ex Y
,(,, 2)), (1), ( 20, TT Aet
(9)
p is reduced to the IS estimator when setting T to zero, and the DM estimator when setting the IS ratio ///ree to zero. However, as shown in (8) key different from those bandit estimators is that in pairwise comparison setting, each data tuple is used twice as (X,Y Y1), ( Z) and (X, ( ( Z) in constructing the estimating function. This effectively reduces the variance of the resulting estimator. As a result, we will formally show in Section that DR estimator is cami-paramttricllly efficient. Additionally, we will establish the consistency of when either δ or Tee is correctly specified.
Similar to the DR estimator in the bandit setting 112
Preference optimization. For preference optimization, our goal is to identify the optimal policy that maximizes the average total preference p (x). Under the BT model assumption, immediate to that the argmax is equivalent to n² defined in (2). Given the proposed DR estimator, we estimate the optimal policy by solving
न argmax max (T) XExxDDRI.I (m( X) mor (+ xX]]
(10)
We refer to 10 as DRPO, short for doubly robust preference optimization. Theoretically, we will show Section that our estimated policy â achieves a smaller suboptimality gap bound than PPO- and DPO-based algorithms when the BT assumption holds. Practically, we implement three refinements to stabilize the training: (i) clipping the IS ratio avoid extremely large ratio; (ii) designing pseudo objective function to enable Monte Carlo sampling from the target policy during optimization; (iii) adopting the KL divergence mcasure from the group relative policy optimization 43 for variance reduction. Details relegated to Appendix B to save space.
5
Theoretical Analysis
We begin with a summary our theories; Figure 2 outlines the roadmap. Our theories are concerned with (i) the MSE of our preference evaluation estimator (DR(TT) (see (9). and (ii) the performance gap bounds of से (see (10) computed by the proposed preference optimization algorithm. Specifically, 2 provides finite sample upper bound for the MSE of PIRR which in turn yields its double robustness (Corollary 3} and semi-parametric efficiency (Corollary 4 Meanwhile, Theorem uppper bounds the difference total preference between the optimal in-class policy and स. without assuming the BT model holds. reveals the double robustness property of our preference optimization algorithm (Corollary 6). When the BT model holds, Theorem further upper bounds the suboptimal gap of т, demonstrating that it general achieves smaller gaps than PPO- and DPO-based algorithms.
Theorem
We next introduce some technical conditions.

---

Assumption 1 (Coverage). T/ ner and T g/ are upper bounded by some constant > 0. Assumption 2 (Boundedness). When the BT model holds, both the oracle reward function r* and its estimator are bounded functions.
Assumption (Realizability). When the BT model holds, x² that maximizes the expected reward (see (2) belongs to the parameterized policy class II in 10).
Assumption 4 (Model complexity). II belongs to the Definition 2.1] with finite VC index >> 0.
(VC) type class [145]
We remark that similar coverage, boundedness and realizability assumptions are commonly imposed in the OPE and RL literature [see e.g.. [46] 147 144). The VC-class condition is also frequently assumed in statistics and machine learning [see e.g.. 148
149].
MSE of PDR(TT) We next study the statistical properties of the proposed preference estimator POR (T)). Without loss of generality, we also assume both कee and 9 (or f, in the case where the BT model holds) are obtained from external models independent of 'D. This condition is mild. Even when such external models are available and नाe and !/ are learned internally from D, independence can be preserved using sample-splitting and fross-fitting [93].
with n data tuples, the semi-parametric efficiency bound (SEB) for estimating p" (TT) is given by n VVr((v(XX ( ((2), Z; Trefs )). Additionally, the MSE of our PDR( equals
Theorem (MSE). Under Assumption /
SEB + O -
(11-9"1) +
9'll
(
- 111
+0
(1
Aret
112 10 10---)1))
(11)
Teef
Ree
where / Tree 111 and Il g'll denote the root mean squared errors of Tret/ Tiet and g: see Appendix 3 their definitions.
The first part of Theorem 2 establishes the SEB the mmaleest-possibll MSE that one can hope for estimating p" (TT. The second part upper bounds the excess MSE of our estimator over SEB. Specifically, this excess MSE consists of three parts: the first two are excess variance terms arising from estimation errors in the reference policy and the preference model, while the third is a bias term introduced by these estimation errors. Notably, (i) it can be shown that SEB scales as O(n (ii) the two variance terms decrease to zero as the sample size n approaches infinity; (iii) the bias term is product of the MSEs of सee and g.
Consequently, when either सित or I is correctly specified, the MSE of PDR converges to zero as n approaches to infinity. This establishes the double robustness property of our estimator, which we state below.
Corollary 3 (Doubly robust evaluation). Under Assumption 7 when either मee or 9 is correctly specified, the MSE of (T)( decays to zero as n. approaches to infinity.
We next consider the case where both मe and ğ are "approximately" correct in that both root MSEs || Teet - 1|| and وا 9-1l decay to zero as n - co. Since SEB is of the order O(n (1) the first two variance terms in (TT) decay to zero at a even faster rate than SEB. Mcanwhile, when the product || 1|l 96 = o 11/2), the last bias term in IT] becomes negligible compared to SEB as well. Together, these conditions imply that the MSE of PDR asymptotically matches the SEB, which establishes the semi-parametric efficiency of our estimator. We also remark that || Weet/Teee
conditions similar to
1 1 g"ll = o(n o12) are widely assumed in the literature [see
e.g.. 93 103 82 150].
Corollary 4 (Semi-
efficiency). Under Assumption ! when both || decay to zero as n + oo, and their product is onn then (USE + 1 as n + cc.
Eat 1|| and Ig g"ll Rga
Regret of f.. Next. we derive the statistical properties of the proposed policy स. When the BT model assumption is violated, we measure the performance gap of given policy π using the gap between the total preference of the best in-class policy and that of æ, i.c., Gap( (T)) sup p" (x') p² (TT) By definition, a smaller performance gap indicates a better policy.
Theorem 5 (Performance gap). Under Assumptions (assuming it holds for any π E IT) and then
Gap(f) = o(3 (3+
2 +
+
v
R Tret - 1|11l0 -''ll..
(12)
7

---

It can be seen from (3) that the performance gap depends on several factors: (i) it decays with the sample size n: (ii) increases with the regularization parameter /3 in the KL divergence penalty; (iii) it increases with v, which measures the complexity of the policy class; (iv) it decreases with the estimating error of the reference policy and the preference model. Crucially, the last dependence appears as the product | fee 1| I 9 which enables us to establish the double robustness property in the context of preference optimization.
Corollary 6 (Doubly robust optimization). Suppose /3 - n. - O. Under the conditions in Theorem when either सe or 9 is correctly specified, Gap(ft) decays to zero as n1 + co.
Finally, we restrict attention to the ideal setting where the BT model holds and upper bound the suboptimality gap, defined as the difference in the expected reward between the optimal policy R¹ and our म. i.e., ( (
Theorem 7 (Suboptimality gap). Suppose the BT model assumption in ① holds. Under Assumptions 5 the suboptimality gap of ß is upper bounded by
and the conditions in Theorem
o(3
0
1 Tree 1111 (1ll--r']) r')).
(13)
n
n
Meanwhile, for PPO-based algorithms, their suboptimality gaps are bounded by
o(3 +
(14)
Finally, DPO-based algorithms, their suboptimality gaps are bounded by
o( (xxp(=BB")+ + 1| Feet 11))
(15)
for some constant ē > 0, under conditions specified in Appendix A.7
According to 13 and that, by using a sufficiently small B, the suboptimality gaps of PPO-based and our algorithms are of the order (n-11/2 + || --(l) I]) and O(n + |/ttt/ F respectively. As for DPO-based algorithms, setting / = 2---Coog log n for some constant >> makes the first term in (15) of order Onn which can be made arbitrarily small with sufficiently large logarithmic factor, yielding overall
C. The second term remains of order 1(n-1/2) up suboptimality gap of (n-11/2 log + ||Teet/ / I]). Consequently, our algorithm's 's suboptimality gap is more robust to estimation errors in the reference policy and preference model, as these errors influence our bound only through their product. To the contrary, for PPO- and DPO-based algorithms, these errors affect their suboptimality bounds in the first order. In particular, when these errors converge to zero at rate of ((n for some 0 < 1/2, our algorithm achieves strictly smaller suboptimality bounds than both DPO- and PaO-b algorithms.
To conclude this section, we make two remarks. First, a key novelty of our analysis lies in the derivation of DPO's sub-optimality bounds without relying on linearity assumptions. While there is extensive literature on DPO-based algorithms, their sub-optimality gaps are relatively underexplored. Some recent works derive such bounds under strong linear assumptions, which simplify the analysis by allowing the sub-optimality gap to be expressed directly in terms of parameter estimation error [15]]. contrast, our analysis proceeds without such linear assumptions, which makes the derivation much more challenging. Second, Theorem establishes upper bounds on the sub-optimality gaps, and we discuss the tightness of these bounds in Appendix A.7
Experiments
In this section, we first use the IMDb dataset 152
empirically validate the double robustness
property of our preference estimator POR (Equation 9 established in Corollary 3 We next compare the proposed preference optimization algorithm (Equation I0) against baseline approaches on the Too Long; Didn't Read (TL;DR, [53] and Anthropic Helpful and Harmless [HH, datasets.
These datasets are particularly suited for studying preference and/or reference model misspecification. Specifically: (i) TL;DR illustrates reference model misspecification we use the SFT model trained by CleanRL [154]. which was learned on filtered subset of the data, leading to misspecified
8

---

reference policy; (ii) HH illustrates preference model misspecification, as prior works suggest this dataset contains unmodeled pairwise noise beyond BT 54]; (iii) IMDb illustrates both types of misspecification, since it is synthetic dataset where we have access to the ground-truth preference and reference models.
Preference Evaluation. We consider the controlled sentiment generation task which aims to produce pos- itive movie reviews using the IMDb dataset. We
Rew 1.
1.0
Rew v,
first apply SFT to the EleutherAI base model [155].
Hew K, 9
which serves as the reference policy for response gen- eration. The generated responses are then annotated using pre-trained sentiment classifier to produce preference labels. Using these synthetic data, we train an optimal policy via DPO. Our objective in this section is to evaluate the total preference of this DPO-trained policy over the SFT-based reference policy, Its oracle value, computed via Monte Carlo, is 0.681. Additional details on data generation and model training are provided in Appendix C.I
Aew X, 9
0.6
0.4
800
800
1000
1200
1400
Sample Size
Figure 3: MSEs of different preference evalu- ation estimators on the IMDb dataset. Shaded areas visualize the 95% confidence bands.
To empirically assess the double robustness property, we evaluate four variants of our preference estimator. each with either the preference model and/or the reference policy correctly specified or misspecified. To misspecify the preference model, we set ÿ to a uniformly random value in 00, 1]. To misspecify the reference policy, we use the unfine-tuned El eleuherAA base model. Figure displays displass the MSEs (solid lines on left panel) and their associated 95% confidence intervals (shaded areas) of the four estimators across different sample sizes, averaged over 500 simulations. It can be seen that the estimator with both models misspecified (red line) exhibits a significantly larger MSE than the other three and shows minimal improvement beyond 800 samples. To the contrary, when either the preference model or the reference policy is correctly specified (yellow and green lines), the MSE is substantially reduced with moderately large sample size. This aligns with the double robustness property. Meanwhile, the estimator with both correctly specified models (blue line) achieves the lowest MSE (being very close with 1500 data tuples), supporting its semiparametric efficiency.
Preference Optimization. This section considers two tasks: summarization and human dialogue. First, for summarization, we use the TL:DR dataset, where lengthy Reddit posts serve prompts and preference annotations are from Stiennon et al. [141], to fine-tune LLMs for concise, informative summaries. Both SFT and reward models for this task are obtained from CleanRL. Second, for human dialogue, the HH dataset (human queries as prompts) used to align LLMs for helpful responses. Since the original SFT and reward models unavailable, we train them ourselves using the TRL framework [156]. For each task, a reward- based BT preference model (using the same reward model for PPO training) and general preference model [19 are adopted to serve as ğ (donating as DRPO-BT and DRPO-GPM). Refer to more details of the implementation and baseline model training in Appendix
Pairwise Win Rate Matrix (TL;DR)
Pairwise Win Rate Matrix (HH)
1.0
0.70
0.515
DRPO
0.4//
0.6
0.8
0.60
0.485
0.575
0 0.848
0.6
pepo
0.502
0.523
0.630
0.55
0.50
0.38b
0.425
0./63
0.4
0.023
0.///
0.623
0.45
0.2
0.152
PPO
0.577
0.35
0.0
0.30
DRPO
DPO
070
Figure 4: Pairwise win rate matrices between different methods across two datasets. Left: TL:DR. Right: HH. Each entry indicates how often the row method outperforms the column method.

---

Table 2: Win rates of DRPO (using BT as the preference model) compared to various base- line algorithms on TL:DR. Higher win rates indicate better performance of DRPO over the baseline algorithm.
Table 3: Win rates of different algorithms com- pared to SFT on HH. 'LC Win Rate" denotes the Singth-contoolle win rate. DRPO uses GPM as preference model.
Model
LC Win Rate (%)
Win Rate (%)
Baseline Model
Win Rate (%)
Dr. DPO
92.16
90.93
DRPO vs Dr. DPO
72.5
iDPO
86.89
85.71
DRPO vs rDPO
65.0
cDPO
85,05
84.28
DRPO vs cDPO
63.5
CPO
73.59
71.28
DRPO vs CPO
90.0
ORPO
75.92
53.91
DRPO vs ORPO
57.5
IPO
78.29
78.88
DRPO IPO
98.5
RSO
80.62
79.50
DRPO RSO
69.5
DRPO
86.38
84.84
We compare our DRPO against nine baseline fine-tuning algorithms. including the standard PPO and DPO, and seven variants of DPO: (i) Dr. DPO [54]: (ii) rDPO [53]: (iii) cDPO [157]: (iv) CPO 158) : (v) ORPO [59]: (vi) IPO (6); (vii) RSO 48). Given the absence of ground-truth preference and reward models, we adopt two evaluation strategies. The first strategy uses on-iistribution data. Specifically, for both TL;DR and HH, one portion of the dataset is used to fine-tune the LLMs, while the remaining portion used to generate responses for evaluation. Following prior works 26 41 40
we employ GPT-4o-based annotator to compare the quality of responses produced by two LLMSs (details in Appendix C.2p. Win rates - the percentage of cases in which one LLM's response is preferred over another-
reported at the default temperature of 1.0 in Figure 4 and Table with results at other temperatures provided in Appendix D The second strategy uses the out-of-distribution data provided via the AlpacaEval 2.0 benchmark [600, which covers broad instructions designed for general-
collection of conducted using GPT-4-Turbo Turr--TTubb-bbseed annotator. Since summarization is domain-speciff task, we apply the out----distrbbution evaluation only to human dialogue (Table
tasks. Pairwise comparisons
In summarization, both DRPO-BT and DRPO-GPM substantially outperform PPO, DPO (see the left panel of Figure and DPO's variants (Table As mentioned earlier, the reference policy in this dataset is misspecified, likely contributing to the weaker performance of DPO and its variants. Despite trained on the same misspecified reference policy, the superior performance of DRPO highlights its robustness to such misspecification. In human dialogue, DRPO-GPM demonstrates the best in-distribution performance, whereas DRPO-BT outperforms PPO and achieves comparable performance to DPO (see the right panel of Figure 4.. The poor performance of PPO partly supports the potential misspecification achieves win rate of 57% against PPO, demonstrating its robustness. As for out-of-distribution evaluation, DRPO performs comparably to robust DPO variants (cDPO, rDPO and Dr. DPO) while attaining higher win rates than other variants (Table 3). As discussed earlier, the HH dataset likely contains pairwise noise, which these robust variants are explicitly designed to handle, whereas DRPÒ employs a preference model that does not account for such noise. If DRPO were to adopt the same noise-aware preference model used in these methods, its performance would likely improve further.
BT this task. Despite using the same preference model, DRPO-BT
7 Discussion
This work introduces a novel doubly robust preference optimization (DRPO) for LLM fine-tuning. Our approach enables accurate preference evaluation and policy optimization, providing robustness against misspecifications in both the reference policy and the preference model. We formally establish that our preference evaluation estimator is both doubly robust (Corollary 3 and semiparametrically efficient (Corollary 4 and demonstrate that our optimization procedure yields policies with a small performance gap (Theorem 5). and lower suboptimality bound than DPO and PPO (Theorem 7 Our empirical results reinforce the theoretical advantages, demonstrating DRPO's robustness to pieerecce-policy misspecification (Table 2 Figure 4 left) and preference-mode misspecification (Table 3) Figure 4 right).
10

---

Acknowledgments
Hongyi Zhou's research was partially supported by NSFC 12271286 & 11931001 and the China Schol- arship Council. Shi and Zhu's ss research was partially supported by the EPSRC grant EP/W01497 1/7 Francesco acknowledges funding from ELSA: European Lighthouse on Secure and Safe Al project (grant agreement No. 101070617 under UK guarantee). The authors thank the anonymous referees and the area chair for their insightful and constructive comments, which have led to a significantly improved version of the paper.
References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-
1901, 2020.
[2] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 2023.
[3] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint 25t.:::5501..24455 2025.
[4] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as laboratory for alignment. arXiv preprint arXiv:21 2021.
[5] Richard Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction. MIT press Cambridge, 2018
[6] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[7] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 1909.089999 2019.
[8] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. 2022.
[9] Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Campbell- Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, et al. Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Information Processing Systems, 35:38176-333 77-38889, 2022.
[10] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models follow instructions with human feedback. Advances neural information processing systems, 01-27730-277444 2022.
[11] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4)3324-355, 1952.
[12] Kenneth O. May. Intransitivity, utility, and the aggregation of preference patterns. Economet- rica, 22:1, 1954. UwLLattp://pp...ennttcccoooo www.oollaa..or/ooouoooooe 156169619
[13] Amos Tversky. Intransitivity of preferences. Psychological Review, 76:31 1969. URL https://api. www.etiiimcooolaaae.o.e.e.oa http://ch....aaooooooooooroaaaaa
11

---

[14] M Gardner. Mathematical games, the paradox of the nontransitive dice and the elusive princi- ple of indifference. Dec 1970. URL http://www.cceet..iicccmaacaaaaa com/ article/ mathematical- -aaees----e---e
- 1970-12/
[15] Marina Agranov and Pietro Ortoleva. Stochastic choice and preferences for randomization. Journal of Political Economy, 125:40- 68, 2015. URL httpss//p www.ppp..ssmttnnccoll....ee
org/ Corpus 11440933
[16] Eric J Michaud, Adam Gleave, and Stuart Russell. Understanding learned reward functions. arXiv preprint arXiv:2012.05802 2020.
[17] Silvia Milano, Mariarosaria Taddeo, and Luciano Floridi. Ethical aspects of noelee recommendation systems. The information society, 37(1):35 45, 2021.
[18] David Lindner and Mennatallah El-Assady. Humans are not boltzmann distributions: Chal- lenges and opportunities for modelling human feedback and interaction reinforcement learning. arXiv preprint arXiv:2206. 13316, 2022.
[19] Yifan Zhang, Ge Zhang, Yue Wu, Kangping Xu, and Quanquan Gu. General preference modeling with preference representations for aligning language models. arXiv preprint arXiv:2410.02197, 2024.
[20] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint TrXXiv/707.06344 2017.
[21] Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35: 9460-9471, 2022.
[22] Cassidy Laidlaw, Shivam Singhal, and Anca Dragan. Correlated proxies: new definition and improved mitigation for reward hacking. arXiv preprini arXiv:2403.0 185, 2024.
[23] Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback. arXiv preprint 1IXXi:2311214925: 10, 2023.
[24] Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. Secrets of rlhf in large language models part i: Ppo. arXiv preprint arXiv:2307.00 04964,, 2023.
[25] Yanjun Chen, Dawei Zhu, Yirong Sun, Xinghao Chen, Wei Zhang, and Xiaoyu Shen. The accuracy paradox in rlhf: When better reward models don't yield better language models. arXiv preprint arXiv:2410.06555 2024.
[26] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36:53728-53741, 2023.
[27] Yixin Liu, Pengfei Liu, and Arman Cohan. Understanding reference policies in direct prefer- ence optimization. arXiv preprint wxXiv:2407.13709 2024.
[28] Alexey Gorbatovski, Boris Shaposhnikov, Alexey Malakhov, Nikita Surnachev, Yaroslav Aksenov, Ian Maksimov, Nikita Balagansky, and Daniil Gavrilov. Learn your reference model for real good alignment. arXiv preprint 244i::24040066560 2024.
[29] Wenda Xu, Jiachen Li, William Yang Wang, and Lei L.. Bpo: Staying close to the behavior Ilm creates better online IIm alignment. arXiv preprint arXiv:2406.12 2024.
[30] Whitney K Newey. Semiparametric efficiency bounds. Journal of applied econometrics, 5(2): 99-135, 1990.
[31] Anastasios A. Tsiatis. Semiparametric Theory and Missing Data. Springer, 2006.
[32] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters deep rl: case study on ppo and trpo. In International conference on learning representations, 2019.
12

---

[33] Jiancong Xiao, Ziniu Li, Xingyu Xie. Emily Getzen, Cong Fang, Qi Long, and Weijie J Su. On the algorithmic bias of aligning large language models with rlhf: Preference collapse and matching regularization. arXiv preprint arXiv:2405. / 16455, 2024.
[34] Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi- thi-Quan Luo. Remax: simple, effective, and efficient reinforcement learning method for aligning large language models. arXiv preprint arXiv:23 10.22300.110 10.0000, 2023.
[35] Alex J Chan, Hao Sun, Samuel Holt, and Mihacla Van Der Schaar. Dense reward for free in reinforcement learning from human feedback. arXiv preprint arXiv:2402.0077 2024.
[36] Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. On designing effective rl reward at training time for Ilm reasoning. arXiv preprint arXiv:2410.15115. 2024.
[37] Pangpang Liu, Chengchun Shi, and Will Wei Sun. Dual active learning for reinforcement learning from human feedback. arXiv preprint arXiv:2410.02504 2024.
[38] Jiayi Fu, Xuandong Zhao, Chengyuan Yao, Heng Wang, Qi Han, and Yanghua Xiao. Reward shaping to mitigate reward hacking in rlhf. arXiv preprint 25x2: 2025.
[39] Teng Xiao, Yige Yuan, Mingxiao Li, Zhengyu Chen, and Vasant G Honavar. On connection between imitation learning and rlhf. arXiv preprint arXiw:250300507, 2025.
[40] Kai Ye, Hongyi Zhou, Jin Zhu, Francesco Quinzan, and Chengehun Shi. Robust reinforce- ment learning from human feedback for large language models fine-tuning. arXiv preprint arXiv:2504.0374 2025.
[41] Tianhao Wu, Banghua Zhu, Ruoyu Zhang, Zhaojin Wen, Kannan Ramchandran, and Jiantao Jiao. Pairwise proximal policy optimization: Language model alignment with comparative rl. In First Conference on Language Modeling, 2024.
[42] Han Zhang, Yu Lei, Lin Gui, Min Yang. Yulan He, Hui Wang, and Ruifeng Xu. Cppo: Contin- ual learning for reinforcement learning with human feedback. In The Twelfth International Conference on Learning Representations, 2024.
[43] Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 10X:::2402.033000, 2024.
[44] Jian Hu, Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.0322 2025.
[45] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.2072 2025.
[46] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
[47] Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.055 18, 2025.
[48] Yao Zhao, Misha Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. Calibrating sequence likelihood improves conditional language generation. arXiv preprint arXiv:2210.0004, 2022.
[49] Chaoqi Wang. Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints. arXiv preprint arXiv:2309.1622 2023.
[50] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint 2rX0v:2402.01336, 2024.
13

---

[51] Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 18990-18998, 2024.
[52] Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749) 2024.
[53] Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust dpo: Aligning language models with noisy feedback. In International Conference on Machine Learning. pages 42258 42274. PMLR, 2024.
[54] Junkang Wu, Yuexiang Xie, Zhengyi Yang. Jiancan Wu, Jiawei Chen, Jinyang Gao. Bolin Ding, Xiang Wang, and Xiangnan He. Towards robust alignment of language models: Distributionally robustifying direct preference optimization. arXiv preprint arXiv:247.007880 2024.
[55] Xize Liang. Chao Chen, Shuang Qiu, Jie Wang. Yue Wu, Zhihang Fu, Zhihao Shi, Feng Wu, and Jieping Ye. Ropo: Robust preference optimization for large language models. arXiv preprint arxivv2404.04100 2024.
[56] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference Artificial Intelligence and Statistics, pages 4447-4455. PMLR, 2024.
[57] Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Row- land, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. arXiv preprint arXiv:2312.00888 18, 2023.
[58] Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, et al. Human alignment of large language models through online preference optimisation. arXiv preprint arXiv:2403.cm 2403::2003.88355, 2024.
[59] Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03715 2024.
[60] Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. A minimaximalist approach to reinforcement learning from human feedback. arXiv preprint 1.Xiv:2401.04056 2024.
[61] Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675, 2024.
[62] Chenlu Ye, Wei Xiong, Yuheng Zhang, Hanze Dong, Nan Jiang, and Tong Zhang. Online iter- ative reinforcement learning from human feedback with general preference model. Advances in Neural Information Processing Systems,
-1807, 2024.
[63] Yuheng Zhang, Dian Yu, Baolin Peng, Linfeng Song. Ye Tian, Mingyue Huo, Nan Jiang, Haitao Mi, and Dong Yu. Iterative nash policy optimization: Aligning 1lms with general preferences via no-regret learning. arXiv preprint arXiv:2407.00617, 2024
[64] Kaizhao Liu, Qi Long, Zhekun Shi, Weijie Su, and Jiancong Xiao. Statistical impossibility and possibility of aligning Ilms with human preferences: From condorcet paradox to nash equilibrium. arXiv preprint arXiv:2503. 2025.
[65] Jiashuo Wang, Haozhao Wang, Shichao Sun, and Wenjie Li. Aligning language models with human preferences via a bayesian approach. Advances in Neural Information Processing Systems, 36:49113-499122 2023.
14

---

[66] Yuzhong Hong, Hanshan Zhang, Junwei Bao, Hongfei Jiang, and Yang Song. Energy- preference model offers better offline alignment than the bradley-terry preference model. arXiv preprint arXiv:2412. 13862, 2024.
[67] James M. Robins, Andrea Rotnitzky, and Lue Ping Zhao. Estimation of regression coefficients when some regressors are not always observed. Journal of the American Statistical Association, 89(427):846 866, 1994.
[68] Daniel O Scharfstein, Andrea Rotnitzky, and James M Robins. Adjusting for nonignorable drop-out using semiparametric nonresponse models. Journal of the American Statistical Association, 94(448):109--1222 120. 1999.
[69] Heejung Bang and James M Robins. Doubly robust estimation in missing data and causal inference models. Biometrics, 61(4):962-973, 2005.
[70] Zhiqiang Tan. Bounded, efficient and doubly robust estimation with inverse weighting. Biometrika, 97(3):661-682 2010.
[71] Anastasios A Tsiatis, Marie Davidian, and Weihua Cao. Improved doubly robust estimation when data are monotonely coarsened, with application longitudinal studies with dropout. Biometrics, 67(2):336-555, 2011.
[72] Kosuke Imai and Mare Ratkovic. Covariate balancing propensity score. Journal of the Royal Statistical Society Series B: Statistical Methodology, 76(1): (6611:22332233 2014.
[73] Karel Vermeulen and Stijn Vansteelandt. Bias-reduced doubly robust estimation. Journal of the American Statistical Association, 10(551)):022-100333 2015.
[74] Edward H Kennedy, Zongming Ma, Matthew D McHugh, and Dylan S Small. Non-parametric methods for doubly robust estimation of continuous treatment effects. Journal of the Royal Statistical Sociery Series B: Statistical Methodology, 79(4): 1245, 2017.
[75] James M Robins, Lingling Li, Rajarshi Mukherjee, Eric Tchetgen Tchetgen, and Aad van der Vaart. Minimax estimation of functional on structured high-dimensional model. THE ANNALS of STATISTICS, pages 1951-1987, 2017.
[76] Stefan Wager and Susan Athey. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 1 113(523): 1242, 2018.
[77] Linbo Wang and Eric Tchetgen Tchetgen. Bounded, efficient and multiply robust estimation of average treatment effects using instrumental variables. Journal of the Royal Statistical Society Series B: Statistical Methodology, 80(3):531-550, 2018
[78] Sören R Künzel, Jasjeet S Sekhon, Peter Bickel, and Bin Yu. Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the national academy of sciences, 16((10)4156-4165. 2019.
[79] Miruna Oprescu, Vasilis Syrgkanis, and Zhiwei Steven Wu. Orthogonal random forest for causal inference. International Conference on Machine Learning. pages 4932 4941. PMLR, 2019.
[80] Claudia Shi, David Blei, and Victor Veitch. Adapting neural networks for the estimation of treatment effects. Advances in neural information processing systems, 32, 2019.
[81] Isabel R Fulcher, Ilya Shpitser, Stella Marealle, and Eric Tchetgen Tchetgen. Robust inference on population indirect causal effects: the generalized front door criterion. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82(1):199-214, 2020.
[82] Max H Farrell, Tengyuan Liang, and Sanjog Misra. Deep neural networks for estimation and inference. Econometrica, 89(1):181-213 2021.
[83] Xinkun Nie and Stefan Wager. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika, 108()):299-31))) 2021.
15

---

[84] Haoxuan Li, Chunyuan Zheng, and Peng Wu. Stabledr: Stabilized doubly robust learning for recommendation on data missing not random. arXiv preprint arXiv:2205.4470,, 2022.
[85] Yifan Cui, Michael R Kosorok, Erik Sverdrup, Stefan Wager, and Ruoqing Zhu. Estimating heterogeneous treatment effects with right-censored data via causal survival forests. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85(2):179 221, 2023.
[86] Larry Han, Zhu Shen, and Jose Zubizarreta. Multiply robust federated estimation of targeted average treatment effects. Advances in Neural Information Processing Systems, 36:70453- 70482, 2023.
[87] Edward H Kennedy. Towards optimal doubly robust estimation of heterogeneous causal effects. Electronic Journal of Statistics, 17(2):3008-3049 2023.
[88] Xinyu Li, Wang Miao, Fang Lu, and Xiao-Hua Zhou, Improving efficiency of inference in clinical trials with external control data. Biometrics, 79(1):394 403, 2023.
[89] Yifan Cui, Hongming Pu, Xu Shi, Wang Miao, and Eric Tchetgen Tchetgen. Semiparametric proximal causal inference. Journal of the American Statistical Association, 119(546):1348- 1359, 2024.
[90] Yuhao Wang and Rajen D Shah. Debiased inverse propensity score weighting for estimation of average treatment effects with high-ddmenniioal confounders. The Annals of Statistics, 52 (5):1978-2000) 2024.
[91] Lu Wang and Peisong Han. Multiply robust estimation for average treatment effect among treated. Statistical Theory and Related Fields, 8(1):29-39, 2024.
[92] Jin Zhu, Jingyi Li, Hongyi Zhou, Yinan Lin, Zhenhua Lin, and Chengchun Shi. Balancing interference and correlation in spatial experimental designs: A causal graph cut approach. In Forty-second International Conference on Machine Learning. 2025.
[93] Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1-C66 2018.
[94] James M Robins. Optimal structural nested models for optimal sequential decisions. In Proceedings of the Second Seattle Symposium in Biostatistics: analysis of correlated data, pages 189-326. Springer. 2004.
[95] Baqun Zhang, Anastasios A Tsiatis, Eric Laber, and Marie Davidian. A robust method for estimating optimal treatment regimes. Biometrics, 68(4): 1010-1011,, 2012.
[96] Baqun Zhang, Anastasios A Tsiatis, Eric B Laber, and Marie Davidian. Robust estimation of optimal dynamic treatment regimes for sequential treatment decisions. Biometrika, 100(3): 681-694, 2013.
[97] Phillip J Schulte, Anastasios A Tsiatis, Eric Laber, and Marie Davidian. Q-and a-learning methods for estimating optimal dynamic treatment regimes. Statistical science: a review journal of the Institute of Mathematical Statistics, 29(4)6640, 2015.
[98] Alexander R Luedtke and Mark Van Der Laan. Statistical inference for the mean outcome under possibly non-unique optimal treatment strategy. Annals of statistics, 44(2):713, 2016.
[99] Caiyun Fan, Wenbin Lu, Rui Song. and Yong Zhou Concordance-assisted learning for estimating optimal individualized treatment regimes. Journal of the Royal Statistical Society Series B: Statistical Methodology, 79(5):1565-15 1582, 2017.
[100] Runchao Jiang, Wenbin Lu, Rui Song, and Marie Davidian. On estimation of optimal treatment regimes for maximizing t-year survival probability. Journal of the Royal Statistical Society Series B: Statistical Methodology, 79(4): 166--11811111111 2017.
[101] Rui Song, Shikai Luo, Donglin Zeng. Hao Helen Zhang, Wenbin Lu, and Zhiguo Li. Semipara- metric single-index model for estimating optimal individualized treatment strategy. Electronic journal of statistics, 11(1):364, 2017.
16

---

[102] Chengchun Shi, Alin Fan, Rui Song, and Wenbin Lu. High- siseh-dmeensioaal a-learning for optimal dynamic treatment regimes. Annals of statistics, 46(3):925, 2018.
[103] Chengehun Shi, Wenbin Lu, and Rui Song. Breaking the curse of nonregularity with subagging inference of the mean outcome under optimal treatment regimes. Journal of Machine Learning Research, 21(176) 2020.
[104] Chengchun Shi, Wenbin Lu, and Rui Song. sparse random projection praseetion-bbsed test for overall qualitative treatment effects. Journal of the American Statistical Association, 2020.
[105] Chengchun Shi, R Song, and W Lu. Concordance and value information criteria for optimal treatment decision. Annals of Statistics, 49(1):49-75, 2021.
[106] Hao Zhang, Shuigeng Zhou, Jihong Guan, and Jun Huan. Measuring conditional independence by independent residuals for causal discovery. ACM Transactions on Intelligent Systems and Technology (TIST), 10(5):1-19, 2019.
[107] Rajen D Shah and Jonas Peters. The hardness of conditional independence testing and the generalised covariance measure. The Annals of Statistics, 48(3):1514- 2020.
[108] Chengchun Shi, Tianlin Xu, Wicher Bergsma, and Lexin Li. Double generative adversarial networks for conditional independence testing. Journal of Machine Learning Research, 22 (285):1-32, 2021.
[109] Francesco Quinzan, Ashkan Soleymani, Patrick Jaillet, Cristian Rojas, and Stefan Bauer. Drefs: Doubly robust causal feature selection. In International Conference on Machine Learning, pages 28468 28491. PMLR, 2023.
[110] Chengchun Shi, Yunzhe Zhou, and Lexin Li. Testing directed acyclic graph via structural, supervised and generative adversarial learning. Journal of the American Statistical Association, 119(547):
2024.
[111] Yi Zhang, Linjun Huang. Yun Yang, and Xiaofeng Shao. Doubly robust conditional indepen- dence testing with generative neural networks. arXiv preprint arXiv:2407.176% 2024.
[112] Miroslav Dudík, Dumitru Erhan, John Langford, and Lihong Li. Doubly Robust Policy Evaluation and Optimization. Statistical Science, 29(4):485 511, 2014. doi: 10.1214/ 14-STS500.
[113] Nathan Kallus and Masatoshi Uchara. Statistically efficient off-policy policy gradients. In International Conference on Machine Learning, pages 5089-5100. PMLR, 2020.
[114] Masatoshi Uehara, Masahiro Kato, and Shota Yasui. Off-policy evaluation and learning for external validity under covariate shift. Advances in Neural Information Processing Systems, 33:49 61, 2020.
[115] Peng Liao, Zhengling Qi, Runzhe Wan, Predrag Klasnja, and Susan A Murphy. Batch policy learning in average reward markov decision processes. Annals of statistics, 50(6):3364, 2022.
[116] Chengchun Shi, Shikai Luo, Yuan Le, Hongtu Zhu, and Rui Song. Statistically efficient advantage learning for offline reinforcement learning in infinite horizons. Journal of the American Statistical Association, 119(545):232 245, 2024.
[117] Chengchun Shi, Zhengling Qi, Jianing Wang, and Fan Zhou. Value enhancement of rein- forcement learning via efficient and robust trust region optimization. Journal of the American Statistical Association, 1196547)2011-2223 2024.
[118] Nan Jiang and Lihong Li. Doubly robust off oof-po evaluation for reinforcement learning, In International Conference on Machine Learning, pages 652-661. PMLR, 2016.
[119] Philip S. Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for rein- forcement learning. In International Conference on Machine Learning, pages 2139-2148. PMLR, 2016.
17

---

[120] Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off- off-policy evaluation. International Conference on Machine Learning, pages 1447-1456, 2018.
[121] Nathan Kallus and Angela Zhou. Policy evaluation and optimization with continuous treat- ments. In International conference artificial intelligence and statistics, pages 1243-1251. PMLR, 2018
[122] Aurelien Bibaut, Ivana Malenica, Nikos Vlassis, and Mark Van Der Laan. More efficient off-policy evaluation through regularized targeted learning. In International Conference on Machine Learning, pages 654-663, PMLR, 2019.
[123] Ziyang Tang, Yihao Feng, Lihong Li, Dengyong Zhou, and Qiang Liu. Doubly robust bias reduction in infinite horizon off-policy estimation. In International Conference on Learning Representations, 2020.
[124) Nathan Kallus and Masatoshi Uehara. Double reinforcement learning for efficient off-policy evaluation in markov decision processes. Journal of Machine Learning Research, 21:1-63, 2020.
[125] Yi Su, Maria Dimakopoulou, Akshay Krishnamurthy, and Miroslav Dudík. Doubly robust off- policy evaluation with shrinkage. In International Conference on Machine Learning, pages 9167-9176. PMLR, 2020.
[126] Masatoshi Uehara, Jiawei Huang, and Nan Jiang. Minimax weight and q-function learning for off-policy evaluation. Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 9659-9668. PMLR, 13-18 Jul 2020.
[127] Hengrui Cai, Chengehun Shi, Rui Song, and Wenbin Lu. Deep jump learning for off-policy evaluation in continuous treatment settings. Advances Neural Information Processing Systems, 34:15285-15300 2021.
[128] Chengchun Shi, Runzhe Wan, Victor Chernozhukov, and Rui Song. Deeply-debiased off- policy interval estimation. In International conference on machine learning, pages 9580-9591. PMLR, 2021.
[129] Chengehun Shi, Masatoshi Uehara, Jiawei Huang. and Nan Jiang. A minimax learning ap- proach to off-policy evaluation in confounded partially observable markov decision processes. In International Conference Machine Leurning, pages 20057-20094. PMLR, 2022.
[130] Yang Xu, Chengehun Shi, Shikai Luo, Lan Wang, and Rui Song. Quantile off off-policy evaluation via deep conditional generative learning. arXiv preprint 2rXiv:2212.14466, 2022.
[131] Ting Li, Chengchun Shi, Jianing Wang, Fan Zhou, et al. Optimal treatment allocation for efficient policy evaluation in sequential decision making. Advances in Neural Information Processing Systems, 36:48890-48900 2023.
[132] Chengchun Shi, Runzhe Wan, Ge Song, Shikai Luo, Hongtu Zhu, and Rui Song. A multiagent reinforcement learning framework for off-policy evaluation two-sided markets. The Annals of Applied Statistics, 17(4):2701-273) 2023.
[133] Chuhan Xie, Wenhao Yang, and Zhihua Zhang. Semiparametrically efficient off-policy evaluation in linear markov decision processes. In International Conference on Machine Learning, pages 38227-38257. PMLR, 2023.
[134] Yang Xu, Jin Zhu, Chengchun Shi, Shikai Luo, and Rui Song. An instrumental variable approach to confounded off-policy evaluation. In International Conference on Machine Learning, pages 38848-38880. PMLR, 2023.
[135] Defu Cao and Angela Zhou. Orthogonalized estimation of difference of q-functions. arXiv preprint arXiv:2406. 08697, 2024.
18

---

[136] Ting Li, Chengchun Shi, Qianglin Wen, Yang Sui, Yongli Qin, Chunbo Lai, and Hongtu Zhu. Combining experimental and historical data for policy evaluation. International Conference Machine Learning, pages 28630-28656. PMLR, 2024.
[137] Ye Shen, Hengrui Cai, and Rui Song. Doubly robust interval estimation for optimal policy evaluation in online learning. Journal of the American Statistical Association, 119(548): 2811-2821, 2024.
[138] Chengchun Shi, Jin Zhu, Ye Shen, Shikai Luo, Hongtu Zhu, and Rui Song. Off-policy confidence interval estimation with confounded markov decision process. Journal of the American Statistical Association, 119(545):73-2888 2024.
[139] Haoyu Wei. Characterization of efficient influence function for off-policy evaluation under optimal policies. arXiv preprint arXiv:2505..3889 2025.
[140] Victoria Lin, Eli Ben-Michacl, and Louis Louis-Philippe' Morency. Optimizing language models for human preferences is causal inference problem. arXiv preprint wrXv:22402.1499, 2024.
[141] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning summarize with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 3008-3021, 2020.
[142] Huiying Zhong, Zhun Deng, Weijie J Su, Zhiwei Steven Wu, and Linjun Zhang. Provable multi- party reinforcement learning with diverse human feedback. arXiv preprint arXiv:2403.0500 0500000 2024.
[143] Gholamali Aminian, Amir R Asadi, Idan Shenfeld, and Youssef Mroueh. Theoretical analysis of kl-regularized rlhf with multiple reference models. arXiv preprint arXiv:2502.01203, 2025.
[144] Masatoshi Uehara, Chengchun Shi, and Nathan Kallus. A review of off-policy evaluation in reinforcement learning. arXiv preprint arXiv:2212.06335, 2022.
[145] Victor Chernozhukov, Denis Chetverikov, and Kengo Kato. Gaussian approximation of suprema of empirical processes. The Annals of Statistics, pages 1564-1597, 2014.
[146) Jinglin Chen and Nan Jiang. Information-theone considerations in batch reinforcement learning. In International Conference on Machine Learning, pages 1042-1051. PMLR, 2019.
[147] Jianqing Fan, Zhaoran Wang, Yuchen Xie, and Zhuoran Yang. A theoretical analysis of deep q-learning. In Learning for dynamics and control, pages 486-489. PMLR, 2020.
[148] Aad W Van Der Vaart, Jon A Wellner, Aad W der Vaart, and Jon A Wellner. Weak convergence. Springer, 1996.
[149] Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
[150] Nathan Kallus and Masatoshi Uchara. Efficiently breaking the curse of horizon in off-policy evaluation with double reinforcement learning. Oper. Res., 7066):3282-3302, November 2022.
ISSN 0030-364X.
[151] Andi Nika, Debmalya Mandal, Parameswaran Kamalaruban, George Tzannetos, Goran Radanović, and Adish Singla. Reward model learning vs. direct policy optimization: A comparative analysis of learning from human preferences. In 41st International Conference on Machine Learning, pages 38145-3818 MLR Press, 2024.
[152] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christo- pher Potts. Learning word vectors for sentiment analysis. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Languuge Technologies, pages 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http:///www aclweb. www./anthooogg//11-----1
19

---

[153] Michael Völske, Maxime Peyrard, Janek Bevendorff. Martin Potthast, and Benno Stein. TL:DR: Mining reddit to learn automatic summarization. In Proceedings of the Workshop New Frontiers in Summarization (EMNIPP), pages 59-63. 63. Association for Computational Linguistics, 2017.
[154] Shengyi Huang, Michael Noukhovitch, Arian Hosseini, Kashif Rasul, Weixun Wang, and Lewis Tunstall. The n+ implementation details of rlhf with ppo: A case study on tl; dr summarization. arXiv preprint arXiv:2403.1703/ 2024.
[155] Sid Black, Gao Leo, Phil Wang, Connor Leaby, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorfuue March 2021. URL https //doi. 10. 5281/ 5297715 If you use this software, please cite it using these metadata.
[156] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https:// pestps:/ggithe github. com http//hggingfac/////////
2020.
[157] Eric Mitchell. A note on DPO with noisy preferences and relationship to IPO. https //ericmitchell.aat af/ccpo pdf 2023. Accessed: 2025-10-18.
[158] Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation. arXiv preprint trXiv:2401.08417 2024.
[159] Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, 2024.
[160] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlle alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04445. 2024.
[161] Peter J. Bickel, Chris J. Klaassen, Ya' acov Ritov, and Jon A. Wellner. Efficient and Adaptive Estimation for Semiparametric Models. Springer, New York, 1998.
[162] Aaron K Han. Non-parametric analysis of generalized regression model: the maximum rank correlation estimator. Journal of Econometrics, 35(2-3):303-316, 1987.
[163] Robert P Sherman. The limiting distribution of the maximum rank correlation estimator.
Econometrica: Journal of the Econometric Society, pages 123-137, 1993.
[164] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: Sfate-of-the- art natural language processing. In Proceedings of the 2020 Conference
Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguistics. URL httphhttt////t.t. //www aclweb. wrrgeb.org///oo..o..oooooooo...t anthology/
2020. -annl--dmos...
[165] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL
https://arxiv. /tgg///wiwiirriaaa///o/// abs//711.05m
[166] Jochen Hartmann, Mark Heitmann, Christian Siebert, and Christina Schamp. More than feeling: Accuracy and application of sentiment analysis. International Journal of Research
URL in Marketingg 40(1):05 87, 2023. doi: http:///woo/aataacc/o//////////0
http://wwwwiiinwwcece.ccmcc com www.///ccccee//ooooooooooooceeee
[167] Owen Team. Qwen2.5: A party of foundation models, September 2024. URL https
lqwenlm.github... ho/blog/ htt.loggg/ennn.////
[168] Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances Neural Information Processing Systems, 36:30039 -00099 2023.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We confirm the claims in the abstract and introduction reflect the paper's contributions and scope, and match our methodology, theory, and experiments in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Limitations and potential extensions are discussed in Section 7 Appendix E
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

Justification: All the assumptions are clearly illustrated in Section 5 and all the proofs are attached into Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The details of the practical implementation are provided in Appendix B and
the details of the experimental setting are presented in Section 6 and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
22

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The source code is provided in the supplemental material with usage instruc- tions. The dataset used in our experiments publicly accessible, as detailed in Section 6 and Appendix C
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips..om cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
y /nips. /nips.ccc/ppp wttps/cccpcccgguuddeeseeeeeeee/e htwllliccggiiee/CCoeeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Experimental setting/details are provided in Section 6 and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Confidence intervals are reported alongside the results, as illustrated in Fig- ure
3
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provided the compute resources in Appendix
C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: :tttlpp:///euuriie hww.nnuu...../. http://rrp.....ceeeee/eetteeeeae cc/ppbblic/Bthhcccccueellllll
Answer: [Yes]
Justification:
The research presented in this paper fully complies with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts discussed in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not release a dataset or model with such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: They are provided in Appendix
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
25

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The new asset includes the implementation and source code of the methods introduced in the paper. has been provided as part of the supplemental material.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
26

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs are used to judge the responses of trained models and evaluate the performance of different algorithms, which is a common methodology. Besides, LLMs are used for writing, editing, or formatting purposes. general, LLMs do not impact the core methodology of the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, ntn-standard components.
Please refer to our LLM policy hatt:e///ourrpssscc///oofereeer/ for what should or should not be described.
27

---

Appendix
A Technical Proof
In this section, we present the regularity conditions and proof's for all the lemmas and theorems. By nature, the vocabulary size is finite; as such, all random variables - including the prompts X and the responses Y are discrete. Let the upper bound in Assumption D be E where and fixed constant. Consequently, it is omitted from the error bound in the main text. However, in the proof of Theorems, for completeness, we will explicitly highlight how the leading terms of the error bounds depend on e.
i bounded
A.1 Proof of Lemma
By direct calculation, it follows that
B T((())))
foall Cttllaa
Bo Siayy x yel)
E (100())))))
TTe((V(())))
(oOA) ( yie) lale fcx
E
o6
= E
Σ T(U x(y1X)9
vv. vi 1
( (v,Y(e),
where the first equality is derived by the law of total expectation, the second equality follows from the definition of the preference function g", and the third equality follows from the change-or of-egsuuree measure
theorem (e.g., Radon Radon-Nikodym theorem).
Following a similar argument and using the fact that - Z = (( > y()),, we obtain
E ( x) (1 2)} ( (v. y(u) x))
(1), Consequently, p² (T)) = (X) + (((())))))( 2)}, which finishes the proof of the lemma.
A.2 Auxiliary lemma for proving Theorem
Before proceeding to the proof of Theorem we first introduce an auxiliary lemma.
Lemma 8. Under Assumption I with n independent data tuple W; (X,, (1) y 2.),
1.... n., the efficient function [see e.g., 31) for the detailed definition] for p² (T) is given by 4E (X,, ((2) y(22 Ze T. Ficts p" (T)) with tp defined in equation
(I), Y Z; n. neet g"). Let M denote the model that generates these data triplets, which are i.i.d. copies of W (Z, ()())((2)) X). This model involves three types of parameters:
Proof of Lemma 8 To simplify notation, we denote (W) = (())
(i) those to model the prob- ability mass function fx (+) of the prompt X (denoted by 7); (ii) those to model the reference policy which generates response (1), (2) independently conditional on the prompt X (denoted by 6) and (iii) those to model the preference probability g" which characterize the probability of (1) is prefemed than (2) given X (denoted by 1)). Then the likelihood function for data tuple W given by
I(W:7,b, = (X()) (1,, y(2), x)² - g, (2) (1))
(16)
Additionally, let (70, bo, 1o) denote the true parameters in the model so that fno fx, Tho Ref and 9n g"
28

---

The proof follows from standard techniques in semi semi-paramett parametric statistic; see e.g. Chapters 2 & in Bickel al. [[6]] and Theorem 3.5 in Tsiatis [31] See also the proof of Theorem in [124] Specifically:
1. For any given policy и, we first prove that E[()((W) p (T)) V log I(W: bo. 70)] a valid derivative of p" (œ) with respect the parameters (7o, bo, 110), where V denotes the gradient operator.
2. We next prove that (WW) p² (T) lies in the tangent space of the data generating process model M (denoted by TM), that is, (W) p" (TT) E TM-
Step 1: (((((W) p" [T)]Vlog ((W; 7o, bo, no)] is valid derivative of p*(() with respect to (70. bo: 170).
Noted that the ligliikel has zero mean. Therefore, in order to prove step 1, we only need to verify the following three equations hold.
(i) E (W) by log l(W; Yo, bo: m0) = 8y ( h=70
(ii) E (W) & log I(W; 70, bo, no)} p (T)) li=bo
(iii) E (W) 8 on log I(W; 7o, bo, bo, mo) By (ʃ m=
By definition, p² (x)) can be represented
p² (T)
I. By P(un > 12|
(ww. 2.31 322
Σ
Let w = (x, y1, U2, 2)
denote the realization of W = (x, (x, ( (2), 2). follows from equation
16 that
log (wwlw,,,b,, 7,b,))
log (o) + (v) + log (322zz)
+2 9n (3/1 82 x) + (1 - 2) log(1 - 9, (1,1882.))) (17)
With some calculations, we obtain
8 log (ww; Yo. bo, no)
1 0
(x)
0y
fx(x) oy
7=
0 log ((w; 70, bo. no)
1
06
= (er (x) 06
To
b=bu
+
Teet
(vr (x) Ob To (222 x))
b=bu"
0
log (ww; 70s bu. no)
9y (31 122 x
Orj
(11, #2, 9" (U1, 1/2
Or
1=
In the following proof, we omit 1y bb= and In to ease notation.
For equation (i): Let Ber(p) denote the Bernoulli distribution with success probability p. The seft-hand-side (LHS) of equation (i) can be represented by
E (W)
log I(W; :70 bo, bo.noo)) no)
Oy
1
T(v (x) Trer( (z))
T (z) Treer Trer (x) (= -0" -v"(
Σ
Ber(s" (1 (21))
2
②
x (1 xt) m
f((z))
gr
1
(x) 2 a (z())
Σ
(g"(v", V1. x) g"" 1/2 x)) m ( (ar) (
29

---

Using the fact that Bz-Berts (un vxax)) f2 (y 32, x)} = O, the first term on the right-hande hind-side (RHS) of the above equation vanishes. Therefore,
θ E (WW)) By
a en (w)(1)
log (W; To: dn, 70)
Σ 3.91-0"
g' g'(v" yn,
Σ #.M2.0'"
y2, (2, (2) TTe 3' By fr (x)
Σ 9" (y" y, (x) Oy 5m())
2.3.3"
(T).
For equation (ii): Notice that the LHS of equation (ii) can be represented
a
E
(W) log 1W; bo. 70)
0b
1
{
<(V1la)) (Tre(U1
<(U2lz) )
Σ
E Berls' (w1
Tver(1/1 (x:) bb >o(unk))) +
2
(tr x) 0b molosle) x (z - g" g211 y2, (
0
2
(0' 21 9 (v" V22 x2, < |x) 3221.
Σ
(11
mL, (x2l=))fx(x)
0b
Follows similar argument in proving equation (i), the first term on the RHS equals zero. The second term can be further represented by
a
Σ
(g" (v" """.".. 21, + (v' y2, x)) (v" ( (T) (v2 [x) (xx)
0b
0
Σ g" (v'
ab
2.
+
Σ g"
U2, 12, (z) fx)x))
2 06
F.y" M2
Σ 9 (y' y. x) x)
ab t fx))))
2.9.9"
26 (T).
(TT)
This finishes the proof of equation (ii).
For equation (iii): Its LHS can be represented as
E (w) log Uww yo, bu, no)
Om
T(v2lz)) ) (2 g0(,1,
Σ E. Eer (on (on.ppaa))
V2. x))
2
[Te((1112))
Tee
a. :922
1-2
T (1)
gy gn( y2 x)) fx(())
(211112222
g" (y) (2)
Σ
Egn-Berl (oi))anr))
(0" 2) + gy"""" U2. x)) (t (x) (z)
2
X (x2(())))
gn (111 12, x).
g (21,
g" (yl 1/2
On
30

---

The second term is equal to zero due to the fact that
1 1 2
Berlo (on v2.))
0.
(y1, 12,
9" (1/1, V2, 82,
On the other hand, since
1
Ber (11., (2 (2- 9 0y011 V2, x))
9r (111,1222
9" (111 U2:
(V1, 82, 2) x (1 (2, x))
V2.
9" (y/1, x2)
-1
+(1 - 9" 92, x)) (-g" (v), 122 x)) g" (11,
the LHS in equation (iii) can be further represented by
(vvlar)
Ter( ( 8 (22()22
(V212z)
Σ
Ön gn(uir y2, 02,
2
1
Σ
(V22 (1
Ory In
(n (v1 V2, x) (z)
2
2,01.322
Σ
x(yIlr) x)) On pr y2, )2, (xx(())
0
= On (T)
(18)
where the second-to- equality follows from the fact 9n 32,2) 9, a). This finishes the proof of equation (iii).
Thus, with equation (i) - (iii) verified, Step I proven.
Step 2: (W) p" (π) lies in the tangent space TM-
By definition, the tangent space TM is the linear closure of the set of score functions of the all one-dimensional submodels regarding M that pass through true parameter, see Definition 2 in [124] Based on the likelihood function in equation (17), we can explicitly calculate the tangent space of the data generating process model M. In fact, the tangent space TM is product space, which can represented as T, + T + Typ with Ty, Txx T, being the sets of score functions of all one- dimensional submodels passing through the marginal distribution (z), conditional distribution Tee and preference probability g". Take the calculation of Ty as an example. Consider a one-dimensional submodel {fe (x)), defined
fe(x)= - fx fx + =q(x));
where (xx) satisfies E Eg (xx))(()) oc. Since we require fe to be valid probability mass function,
it must satisfy En fe(x) = 1, which indicates Eg(X) = Then the score function with respect to is given by
log log fc(x) = ((x).
de
Therefore, the tangent space for the marginal distribution function f(x) can be represented
T (xx) : :[q(() = 0, Σ (t))((())<))))))))))))))))))))))
Meanwhile, consider a one-dimensional submodel
Te = TT((y)=) + Eq(y.
where g(y, .)) satisfies Σg (x, y) ( < co. Since we require ne (y((ar)) be a valid conditional
probability mass function, it must satisfy Σ, (e(l))) = for any x, which indicates ( x)) for all x. Then the score function with respect to given by
d
log ===0
q(vi. x) + 9(y2, x).
31

---

Therefore, the tangent space for the reference policy Jee can be represented as
Ta - -==((t ((v1.)) + g(v2, x) Ba (g(v, x) XX= = x] = 0, Σ T m
Following similar arguments, we can obtain 9" (Ur, V2. x) g(v1, V2.
(2 y1, f2) (2)))))) o0
T 9r -g)
Σ 2-811-322
50).
To verify vWW) - 1² (i) lies in the tangent space, consider the following three functions:
x ( x)
444 (w)
(= g (11, y2, x:) (
9" Tvre (x)
g¹(1 9r) 9" 5²) (T ( ((02l1) fx (z),
(2 32,22 = En {o(v" ,,, v) + g(v" 12, x)) 2
8¹ (')
(b)(x) = 2Bymmatt (olv" ,,,"")))))) v) - (m²) y (((x()))))
It is easy to verify that 1/1 (W) Tg. the (Y()), y X) Ta and 4 E Ty. Therefore,
(W) (w)-p((a>) (x) 2 (W) + (V())) y x) + vx)()) E TM-
This finishes the proof of Step 2.
With Step and Step 2 verified, together with the fact that (e(W) = (x)) we obtain that (WW) is an efficient influence function.
A.3 Proof of Theorem
Let En denote the empirical average over the n tuples (X, y()), ( 2) in the dataset D. Accord- ingly, our estimator for p" (r) can be represented by [E.[/b(w; n,, mer,, 7)
We further define the following norms:
g((()) y(0), x) gr rr,), x00,,,)))) 1/2
g"ll =
1 2 1/2
11
nee (X(I)) Tee [x)
E
Tyef
In the proof of this theorem, we assume these norms are bounded. Such boundedness assumption is automatically satisfied for ||@ - g" since both g and ğ are probabilities. These assumptions are to simplify our sample error bound by omitting some higher-order remainder terms, which can be more heavily dependent on the aforementioned norms.
With some calculations, we can show that
E, (hrk(w; ww; n, neet 9 En bn n. Tet. g¹) + I++ II + III,
where
x(V x
T(((a)
(X)
En
(x,
( (Y(o) (X)
2
II
1
E,
(1)z)
[x(X, y, y(o)) 66
(X X ii
2
1
T (
x (())
- ((2))-
lllee 'i 1t /
+
En
2
Thetl (Y x)
2
(1)) y(2)) - /6 (x, (1),, (le))
III
En
(X(())))
(X 1 Tace
2
1=1
32

---

From Lemma 8 we know that E, (w;; n, Tet. 9²) is unbiased estimator for p" (T)) with variance equal SEB. Since both uet and g are obtained from external models independent of D, analogous to the proof of Lemma [ we know that the first term and the second term IT have zero means. The third term ITT is the bias term. Therefore, we obtain the following bias-variance decomposition for MSE(PDR):
(SD (T)) Var(EEn v, Teet 9¹) + + +1 III) + (E(III)²
(19)
Since 9 is bounded by under the coverage assumption (Assumption)
we obtain that
1 Var( (b5; n, Tref, 9²) Var(ib(w; Tiref
0 -B
π (YIA)
9"") Trefs
(Y
(20)
0 m
x² (v) (v)X))
O
1
Tret X))
me
y
Similarly, we have
X
x 22
E12t
<-E
Tier (Ye((
2 (Y
2)
E
(21)
22
1ot (Y))
Teet (YI))
Teel
1
1 1
n
Tref
and
EIT2
( nc²
19 0²
EII = (
1
1 12
(22)
ne2
By Cauchy-Sehwer inequality, we have for any random variables U and that [Cov(U,V) V /Va(UU)aa(VV) It follows that
1
1)
Cov (B: Tret, 9") I, III) =0
323/2
Tref
Cov (Enns11; Tref, 9 11), =0
3/2 |10 - g²1l
(23)
Cov (I III, II) =0
l
gell
1
nc?
Since e a constant, the high-order terms Var(T), Var(T) and Var(III) are dominated by the first two
terms in (3 Combining equations (20).
(1
21.00
22 with
23
yields
11]
=1) (24)
Var(En (w; n, Tiets g²) 1+ 11 III)
SEB
3/3 1
+
3/2
Tree
Finally, using Cauchy-Schiie Schwarz inequality again, we obtain that
(y 1/2
1/2 E E || O (ā (x)²(X, (1), (()))
Tee (Y|X)
1
72 (YIX)
T (YX))
1 | g-l
0
|| /re 11)
Combining with 25 we obtain that
MSE(pDR (T)
E (E, (w; n. mo D - -pp (TT)
SEB
3/3 1 - 9 + +0
ne3/2 |lee गee - 11)
+0
A Trat 11|2 1 g" g²1(2]
This finishes the proof of Theorem
33

---

A.4
Proofs of Corollaries and
The proofs of Corollaries and follow directly from the assertion of Theorem
A.5 Proof of Theorem 5
Let n² denote the maximizer of p" (œ) in the policy class 11. Throughout the proof, for any policies Ti and 1T2,
shorthand and write Ex-D DKL (T X) n2( X)] (L ||=2). Since î
a maximizer (T)( - 3KL((T ( we have
DDR (T) (KL((] (DR(=T) - (KL((m" (
It directly follows that
p'(TT) - p" (T)
p" p'(x") PDR (T') PDR (T) (*) + ((L((T(K KL (())
EB (='p'xx)) (x²"
PDRR (x + EE PDR (R) p (f) 0 (3 log t
2E sup Ip" (TT) PDR (T)) (3 log c)
(25)
#ell
where the second inequality follows from the coverage assumption, which entails that
π (v)x) O(log_+ e).
[L[T|| Ex Eynr((l log
Teet X)
Additionally, following the proof of Theorem 2 the bias of the proposed preference evaluation estimator can be upper bounded by
Tre sup (w;; n, Tet 9") p² (+)] + (116 110-- 9¹1l well
sup [p (T) ( nlll
11]
Nref
(26)
It remains to upper bound the empirical process term E sup [., (obb(ur n, Teer 9²) Toward that end, we employ Corollary 5.1 in [45]. To invoke this corollary, notice that
(TT)l--
1. According to Assumption 4 11 is policy class with VC dimension v. Under Assumption I follows from Lemma A.6 145 that the function class { π. मee., 0) E II} also has
VC dimension of v.
Using the coverage assumption again, the function class F is uniformly bounded by O(1/e).
The variance sup Var(f(W)) uniformly bounded by O(1/e²).
Consequently, an application of Corollary 5.1 in 145] yields that
E sup (PR((T) E E'PDRR(T) (m)] o log
22 log (2
rell
ne
log
vlog llg--1
n
nE
Combining equations (25
(26 and
27
we obtain for any E II that
1 =V
v log
+ v log
Are
p" p²(**) - -²)) log
c
9"ll 11
- 111
n
ne
Trel
This completes the proof of Theorem
5
A.6 Proof of Corollary
The proof of Corollary follows directly from the assertion of Theorem Before proving Theorem 7 we discuss the tightness of the suboptimality upper bounds derived therein for PPO, DPO, and DRPO. For each algorithm, its gap contains three components:
34

---

A bias term induced by KL-regularization, which is proportional to B.
A statistical complexity term of the form Vu/n, which depends the sample size and the complexity measure v of the policy class.
A reward/reference policy estimation error term.
Since the first term can be made arbitrarily small by choosing a sufficiently small 3, we discuss the tightness of the second and the third terms below. For the second term, our upper bounds for PPO and DPO match the lower bounds developed in [15]]. indicating their tightness, Finally, our theoretical investigation reveals that under certain settings e.g., when the reward is linear and the prompt distribution is multivariate Gaussian the suboptimality gap of PPO depends linearly on the estimation error of the regression coefficient, which itself is proportional to the reward model estimation error. Meanwhile, for DPO, when there is a constant gap between the specified and oracle reference policy, the algorithm suffers from a constant suboptimality gap that will not converge to zero. This demonstrates the tightness of the third term.
A.7 Proof of Theorem 7
Suboptimality gap for DRPO: If the BT assumption holds, we have g" (V/ y2, y2, x) ((" (f1..)
r" (V2.2)) where (ax 1/(11 + e-*) is the sigmoid function. Since the sigmoid function is mono- tonically increasing, under the realizability assumption, which maximizes J(x) also maximizes
(*x). This follows from the classical results on the maximum rank correlation estimator that has been widely studied in the econometrics literature [see 162 163]. Therefore,
(#)
Byrnun a. (V" ,V, 9" v. (E) [(r"(y" - r ry(,))) (r" r x), Eyrwar a~a (E)(()) xz) - (7. x))}
Eyg
2 Co(J(=))-J(A)
where Co is some positive constant and some real number between r" (y",x) r (y, x) and r" ""G æ² - ry" ()) Here, the second equality follows from mean value theorem. The last equality follows from the identity that d'(x) (I) (x)). which is bounded away from zeroo under Assumption
2 that the reward is bounded by some constant. Thus, we obtain J(T") J(
O(Gap( (t) and the suboptimality gap for DRPO follows directly from the assertion Theorem
Suboptimality gap for PPO-based algorithm: We begin with some notations. For a given estimated reward F, define
[(x) = EE [E X() 3KL(T (T [|TT))) (T) E,, Eyy Fly, X)= (KL[T||TTt))
ãे arg max e (TT)
* ãे = arg ma I, (T))
Using the fact that ((7) 2 1(xT*) and (.) 2 (7), we obtain the following upper bound:
J(x²) - J(
< E ([[[m*)-(((+) [ - (
- ((#)) + (( ((7))
< ([[[["))-((+=)) E (((()) (()))))(()()))) sup (. (27)
nell
For the first term, we have
E [{[J(=))- - [(=*)]]
= Eggar F(y, X) r" (3, + (KL |Tt))
T" (yy
E Enywsree
TTeer (y) X) Fly, X) rv, x31 + OB1log
= o - + + log E),
(28)
where the last equation follows from Sluchh-Schwarze inequality. Using a similar argument, we obtain that E ((((F) - (())) O 11 -r -Blo
35

---

Finally, under assumption the function class F= E, Fly, E =1 is bounded by constant. Using similar arguments to the proof of Theorem 5, we can employ Corollary 5.1 in [45]] to show that
E sup {1((T) #ell
4. o 0
V
v +
+033lg e).
(29)
Combining equations
28 and
29
we obtain that
J(x') - ((ā) = gog
r'1l
Ve
Suboptimality gap for DPO-based algorithm: We need some additional technical conditions to prove the suboptimality gap for DPO-based algorithms. Recall that when BT-model holds, there exists a one-on-one correspondence between the policy and reward model [26). We further assume
Assumption 5 (Realizabiliity. The oracle reward r* lies in the bounded reward function class R = (3 log क ( + 3Z(x) ग E [T} induced by the policy class
Assumption (Coverage). Both "ie and Tiet are lower bounded by some constant >
Assumption 7 (Suboptimality gap for oracle reward). Let y/ arg max, r" (ylzr and Vs arg maX r" (ylz). There exists a positive constant é such that for any x,
(ya x) - r" (Te x) 2 C.
Notice that both the realizabilityy and the coverage in Assumptions and differ from those in the main text. Specifically, Assumption iimposes the realizability assumption on the oracle reward rather than the optimal policy whereas Assumption is stronger than that in the main text by requiring the denominators of the IS ratios to strictly positive.
We also redefine the norm /Tf Tee - 1|l as
Fhee (Y Tea (YOI))) "]?
E max
Teet (Y()) Re (Y(I))))
Notice that this norm is no smaller than the one used in the proposed algorithm.
We next introduce some notations. For given estimated reference policy कeer any policy induce reward function
<(y|>)) ())())
(y. x) = , log
+ (zx
(30)
Let 1(T)) be the log-likelihood function induced by reward nd and 1" ('()) (T) be its variant with कe in the denominator of (30) replaced by the ground truth Ref- Denote गे arg max, (E,((T and
f arg max, (T)) ). It follows that
E,,(() - E., [I( (I))))
El(R) El( (7)
< El(R) EI" (EI"()) EI" (Tt) BI(T)
BC log te ()
< -C,E (F(11 2(y2,2) r" (v), +r +B
Tiet Tier (Y
s -C + B, // नIतर 1112.
(31)
where a² = E (((),, F(y2, x) (y), .)) + r (y2,
x:) 11 and both C1 and C2 are positive constants because the Hessian matrix is bounded away from and infinity, which follows from the boundedness assumption on the reward. Additionally, the last inequality is due to that æ S exp(x) - for any 2 which entails
E log n ((())))
Fio (Y(I)) (Y)) (
xx)
E
log max
Teet (Y(IX))
Tref (YUI))) (r((())))))))
Fi (1)) Thee (x
(Y Aeed (X)) - -1]
max
36

---

Moreover, according to Corollary 5.1 in [145], using similar arguments to the proof of Theorem and PPO-based algorithms, we have
(,, (E),(() (l + (T)
2B sup
((TT)
(T))
=ell
2 0
+ =
(32)
n
n
This together with equation (1) yields that Ci(0 2/o/n)2 & ēv/n + C2 some constant > and hence
/
/Te Tee
1
for
o (V
+ / Teet 1/12
(33)
Recall that n² is the true optimal policy, and मे in this part of the proof denotes DPO's estimated optimal policy. We further define मt as softmax optimal policy based on the oracle reward function
T (vlax exp( frr (v,x))
(y(x) Ey H (v/(2)) exp( 2 (1 (V², 2))
With some calculations, it follows that J(xT) - ( Ey r" (g, X) Eynar X)] -BBBayrr r" (H, X) - Ey-z. r (y, X)) + E(By r Xy. x - Eywar"(y ()),
(34)
where the outer expectations are taken with respect to the prompt distribution.
Recall that 3 denotes the optimal response to the prompt æ. The first term El
Ey-g- X)] can be upper bounded by
Σ r" (y. (y) exp 1 (r, x))
Er Er" (Vx, X) B =
Er -
Ey Teen (X) exp F/rm (U. X))
(2r
r"xxx X x) (X) exp
(tr' (uX (u'' x))
s Er Er' x),xx)=4
E, (v)( exp
1 3 3r (2, x)
= o exp
2 [
where the last equality is due to that under Assumptions
and
7 the difference between 1 and the
ratio Rar (X) ( XX) is of the order O 3, Rarlal X) exp( (r,
exp
almost surely.
Using mean value theorem, the second term can be bounded by
1 E max -
E Σ [+] (v)x))
B |F r" 111,
(35)
where the last inequality follows from the fact that
|| mp-r"h r=1 = B[(f - r2)
1/2
E
Σ Trer
1/2
VCB Σ (F(ylx) -
-
(36)
37

---

To complete the proof, it remains to upper bound - r'112 1 using o². Recall that o² [F((Y())))) f() - r- (Y X) + r* x))l2= Since y(2) independent of y
given X and that Ter is lower bounded by > 0. it follows that
+ (vo. (r, 1 2 o² 2 FE - F(yo. X) r" (x)
for fixed Voo Notice that the RHS corresponds to the mean squared error between F and r*, up to baseline term that is independent of y()) Without loss of generality, we can assume this baseline term r" (No, X) - F(yo, X) equal to zero without affecting the validity of the proof. This because the true reward can be redefined as r" (("xX)) X), since it is equivalent up to function independent of the response. Similarly, the estimated optimal policy e( (x) computed by DPO can be represented using the difference f( x) F(yo, z), and we can replace f in (35) using this difference.
Consequently, we obtain that o² 2 =|lī - r"1 and hence
| - rl1 = 0
1/1
1/2
+
BE 1/2 || | vief 11/2)
Combining this together with equations
33 and (34).
we obtain that the regret is upper bounded by
exp( 1ap(-E8-1)
1
+ =l Teet 111
Be
n
The proof is hence completed.
B DRPO Algorithm Details and Practical Implementation
This section details our proposed algorithm. Notably, the reference model काe and the preference model I are pre-trained independently prior to policy optimization. The proposed objective function is defined as
I (TO; Foee Fy. D) (T) 3Ex X) Ror ( [X]]
(37)
The gradient of [(TO) is given by:
(oT(TO)
1 Fxr
2
R yrut (o)x) 0(X.0. v(a)) log (oy x)]
2
(Z gx ( ((men }
V o (Y
a-1
0(X,
न (Y (a) X)
a=1
v DKL (To
X) R X)]
(38)
Intuitively, the gradient operates as follows: The first term guides the policy to favor responses preferred by the preference model ğ. When y()) > y(2), which means Z 1, the second term enhances the likelihood of (1) while diminishing the likelihood of ( and vice versa.
The empirical loss function is constructed such that its negative gradient corresponds to VoJ (TT) in Equation 38 The metho term is approximated using Monte Carlo sampling by drawing several new responses Di := (Y* Y ~ To( X)} from the current policy my for a given prompt X each policy update. A k3-type empirical KI. divergence is utilized, following 43
2 Σ g(Y", y(a), X) log To
LOROO
E Exyr
By BY-wDX
2
(X (-1) a- 1))--1
(Z g(X, (1), ((2)))
+
Troo ( (X)
8=1
Txet (
X) कe ( X))
+BEy +. XNDD
+
log
(39)
(Y [X)
πe Y ())
38

---

Maximization of (TT) is achieved by minimizing the loss function. In practice, the original offline dataset is augmented D D by including swapped pairs (i.e. for (X,Y(), y Z), add (X, Y(,,,), (-)1))))) to D, simplifying the empirical loss function (39p. Furthermore, the importance sampling ratio is clipped, and its calculation is decoupled from the gradient computation. This is achieved by stopping auto-differentiation for the ratio and multiplying the importance sampling term by log Te, which shrinks (rather than eliminates) gradients in small ref regions while maintaining approximate arithmetic equivalence. Consequently, the loss function reformulated as:
1 LDRPO
Xx Ey-nD' ,(Y y(2), x) log (Y (
term
Te (Y (
+
clip
Tref (1)) - E1, 1 (2 ( x(2),)))
log To (Y ())
Trof (Y* [X]
TY X) log nof (Y* XX To(Y= x)
+ +3 Y D XND
(40)
where sg( denotes stop-gradient operation, clip(e, a, signifies clipping values to the interval [a, b). and é1, are hyperparameters defining the clipping range. See full details in Algorithm
C Experiments Implementation details
For the baseline models training, we follow the framework of TRL: Transformer Reinforcement Learn- ing [156) and Transformers Stute-of-the-Arl Natural Language Processing [164] For the general preference model, we follow the framework of general-preferencce model proposed by Zhang et al. [19]. All models were trained with default hyperparameter configu- rations unless otherwise specified.
The Preference Evaluation experiments are conducted on machine equipped with an NVIDIA RTX 6000 Ada GPU and AMD Ryzen Threadripper PRO 7945WX 12-core CPU. The Preference Optimization experiments are performed on a system with an H20 NVLink GPU and 20 vCPU Intel(R) Xeon(R) Platinum 8457C processor. AdamW [165] are used default optimizer.
C.1 Preference Evaluation Experiment IMDb
Oracle Preference Model. Since the IMDb dataset does not contain human preference labels, we adopt the known sentiment classifier siebert/ ground-truth reward-based labeler. This classifier will give score s(X, Y) = p(positive X, Y), which we convert into reward signal using the log-odds transformation:
wwweimeeeerrrrtaaeeee lalaaa-ngggiieee [166], as
s(X
(X, Y) log (i-s(X,))
Using the Bradley-Terry (BT) model, we then compute the ground-truth preference probability between two completions as:
() = aox"(xx, (x,
where o (()) is the sigmoid function.
Data Generation and Policy Training Process. We begin by fine-tuning supervised fine-tuning (SFT) models initialized from two base architectures of different scales: the EleutherAI /ggttt neo-125m 155] and the Oven/Quen2.cr 5-7B [167]. Both models are trained for three epochs on 25,000 samples from IMDb training set. Prompts are constructed by extracting 5-word prefixes from movie reviews. Using the fine-tuned SFT model as the reference policy, we generate pairs of completions for each prompt. Next, we use the oracle preference model to estimate the preference probabilities between each pair of completions. Based on these probabilities, we sample binary
39

---

Algorithm Double Robust Preference Optimization
Require:
reference policy Tref= preference model g. offline dataset D (x,, y y/21, 7 clipping range [e1, E2], regularization parameter 9, and other hyperparameters, effective batch size [B], learning rate and the optimizer, number of Monte Carlo samples |
Ensure: trained policy Te
1: Initialize policy (0), total train steps
For brevity let the number of training epochs 88
N 1.
2: for t 1, ..... do
3: for i in B, = {(t .).., B do
4:
Sample >x, (Y Yj T
(t-
xe)
5:
Esitmate term I:
T [Dx, Y-EXx;
Σ
I(Y y(2) Y,",,,") X;) log T
(1-2) (y-M)
6: Estimate term II:
(1-1) 1) (X, (xi)
(x)1))
ñi, clip
cι, 61,146. 62 (z ggx 10,0) xi))
,x)))))))
Teef (x:)
7: Estimate KI. divergence:
Frot (Y" X)
Frot (Y' X)
DKL. =
log
[Dx
To (Y* X)
TT ( X)
Y+ER%
8: Compute the empirical loss function the batch:
1
Σ
1 i, sg( (1.) log T (x, Xx.) + +BDKL.
L
Br
2
iEB.
9: end for
update g()) with gradient descent and get T (t)
10:
((a) (1--1) 1) VoL
11: end for
preference labels indicating which response is preferred. This synthetic preference dataset is then used to train a target policy using the Direct Preference Optimization (DPO) algorithm over an additional epochs. To quantify the relative preference for the target policy over the reference policy, we adopt a Monte Carlo estimation approach. Specifically, for each of the 25,000 prefixes in the IMDb test set, both the target and reference policies generate single completion. The oracle preference model is then used to compute the preference probability between the two completions. Aggregating these results, we estimate the overall probability, which 0.681, that the target policy's ouiputs are preferred over those of the reference policy.
versions of the reference policy estimator meet a correctly specified version, where eee corresponds to the SFT model, and a misspecified version, where कee corresponds to the untrained base model. Similarly, we consider two versions of the preference estimator ğ: correctly specified version, which uses the oracle preference model, and misspecified version, where ğ is drawn uniformly random from [0, 1]. By taking all pairwise combinations of माा and g. we construct four distinct variants of the preference evaluation framework. For the Direct Method (DM) estimator Equation [6) we apply a Monte Carlo approach by sampling 8 responses from the target policy for each prompt. For the Importance Sampling (IS) estimator Equation 7
Preference Evaluation Process.
We consider
we use a clipping ratio of 100 when मंन is correctly specified and 40 when it misspecified. In contrast to the clipping ratio used during preference optimization, a larger ratio adopted here better demonstrate the double robustness property of our preference evaluation framework. The results based on the EleutherAI gepggggtteeooooeee -ew- 125m model [155] presented in
40

---

as both the reference Figure 3 in Section 6 while those based on the Qwen/Qwen2. 5- model [16 are summarized in Table
Table 4: MSE of the proposed preference estimator with a 7B base model. The preference model and reference policy can be misspecified or correctly specified.
Sample size
500
1000
2000
3000
Both correct
0.002212
0.001160
0.000702
0.000390
Wrong preference model
0.024942
0.018757
0.016763
0.016594
Wrong reference model
0.066897
0.021389
0.013358
0.008383
Both wrong
0.265155
0.069340
0.043276
0.045954
C.2 Preference Optimization Experiment on Real Data
Baseline models training. For the summarization task, we adopt models from a group of Hugging Face, cleanrl, known for their validated and quality-assured implementations [14.] Specifi- cally, we use cleant1/Eleuther/ www.aaerr-_ttaaaa-------- wtwpbiddddduddd_ss______eee
and initial policy model. This SFT policy is trained via token-level supervised fine-tuning on human- written summaries from a filtered TL;DR Reddit dataset [154]. The associated reward model is wtwpiiddddddd___eeeeeeee _eardd____wrd______________d____ _tldr. For PPO training, we search the hyperparameter over the KL coefficient B E {0.05, 0.2) and select 3 0.05 based on em- pirical performance. Notably.we observe that PPO training can experience policy collapse under low-precision, as the value function fails to fit accurately; thus, PPO models are trained under full precision (FP32). In contrast, all models are trained using 6 (BF16) for improved computational efficiency. To ensure a fair comparison, we set the maximum response length to 128 for all models, providing a consistent basis for assessing summarization quality. For DPO and its variants, we use default hyperparameter setting in TRL with BF16 precision. Notably, DP. had no official TRL implementation, so adapt the loss function in DPOTrainer with Dr.DPO's reweighting strategy and use the suggested hyperparameters in 54
_ldr.
For human dialogue, the SFT model is trained from the base model Qwen/Qwen2.55 2-...2..5-1..8 [67 to better align with the Helpfulness and Harmlessness (HH) dataset. Unlike the summarization SFT model, this version leverages both the preferred (chosen) and non-preferred (rejected) responses from the HH preference dataset. It is trained for 3 epochs. We also train three versions of the reward model, all from the same base model (Qwen/Qwen2) 111e2.551.. 5-1. to avoid additional information, corresponding to epochs 1, 2, and 3, as we observe that PPO training in this setting is highly sensitive to the reward model. When the reward model overfits or becomes overly confident, the KL penalty becomes ineffective, and PPO tends to suffer from policy collapse, hacking the reward model by repeating high-reward tokens. To mitigate this issue, we select the reward model from epoch 1, which achieves an evaluation accuracy of 72.1 %. We further conduct a hyperparameter search over KL coefficients 3 E {0.05, 0.1, 0.2) and learning rates {1e-7, 1c-6, 3e-6). We select KL coefficient of 0.05 combined with a learning rate of 1c-7 as it yields the most stable and effective PPO training performance. Similar those in summarization, DPO and its variants are trained with default setting.
DRPO Implementation DRPO implementation inherits transf ormers. Trainer class. For DRPO- BT, we compute the rewards for two candidate responses and output the preference probability under the BT framework as g. For DRPO-GPM we directly compute the preference probability using the corresponding general preference model [19. Although our proposed algorithm allows the use of a more powerful general preference model for estimating T. as in [57]. we ensure fairness by training all preference models using the same base model and dataset. This avoids introducing any additional information that could bias the comparison. For both tasks, we set the clipping range to [0.04, 2.5]. a fairly casual (and wide) specification only to force the IS ratio to not deviate far from I and thus not inject too much variance into our estimation. The regularization parameter 3 is set to 0.04, the same as that in the default trl implementation for GRPO 43), which also uses k3-type empirical KL. divergence. The number of Monte Carlo samples |D" is set to 3 (TL;DR) or 2 (HH). Although more samples may mitigate bias, the effect of adding samples is marginally decreasing (since the convergence rate O(n")) -)). As such, it proper to choose a parsimonious volume of samples and thus incurring little extra computational cost compared to PPO. Other not-mentioned
41

---

hyperparameters are simply set to default values. For further details, please refer to the examples in the codebase.
Evaluation For in-distribution evaluation, we compare DRPO with DPO and PPO using GPT-4o- mini evaluate the quality of generated response of each task. Specifically, for the language model fine-tuned by either baseline or our method, we can sample a response at certain temperature after receives a prompt. With the responses of two methods (say and B), we feed them with query asking GPT to judge which is more aligned with certain demands. The query template used for TL;DR is shown in Table [ which tries avoid GPT's favor of lengthy responses following [40]. The query template used for HH is shown in Table standard template that is widely adopted by e.g. [26] 40]. It noteworthy that we randomly shuffle the order of the responses for each query to eliminate the potential bias from the order of the responses.
Here, temperature is the scaler of logits before softmax, which can be used adjust the output distribution of a certain policy. In general, a temperature less than tends to make kurtosis of the distribution larger (thus more greedy when generating responses), and temperature larger than generate even more random responses. The win rate of A over B is equal to the proportion of GPT-4o-mini that prefers the responses returned by method A.
For out-of of-distributton evaluation in HH dataset, we evaluate our models using the AlpacaEval 2.0 benchmark [160), an LLM-based automatic evaluator designed to assess models' general performance. The prompt set in AlpacaEval 2.0 is derived from AlpacaFarm [[68], which contains a broad collection of human-written instructions covering wide range of general-purpose tasks beyond the Helpful-Harmless (HH) domain. By default, AlpacaEval 2.0 compares each model-generated response against reference response produced by GPT-4-Turbo, and GPT-4-urbo-bbase annotator determines which of the two is preferred. However, we observed that all fine-tuning algorithms achieved consistently low win rates when evaluated against GPT-4-Turbo references, likely due the substantial capability gap between GPT-4-Turbo and the fine-tuned models. To ensure a fairer and more interpretable comparison, therefore replace the reference responses with those generated by the SFT model, allowing AlpacaEval 2.0 to compute the win rate of each fine-tuning algorithm relative to the SFT baseline.
Table 5: Query template for the summarization task.
Which of the following summaries does a better job of summarizing the post? Strictly follow two criteria when selecting the best summary: 1. Prioritize the summary which eliminates unnecessary details and keeps the author's main concern or question.
2. Prioritize the shorter summary as long as it remains clear and preserves the main idea.
Post: <post>
Response A: <response_a>
Response B: <response_b>
FIRST provide one nee-sentecce comparison the two summaries, explain- ing which
you prefer and why. SECOND, on new line, state only "A" or to indicate your choice. Your response should use the format:
Comparison: <one <one-sentence comparison and explanation>
Preferred: <'A" or 'B">
42

---

Table 6: Query template for the human dialogue task.
For the following query to chatbot, which response is more helpful?
Query: <user_query>
Response A: <response_a>
Response B: <response_b>
FIRST provide a one-sentence comparison of the two responses and explain which you feel is helpful. SECOND, new 1ine, state only
"A" or "B" to indicate which response is helpful. Your response should use the format:
<one-sentence comparison and explanation> More helpful: <"A"" or "B">
Comparison:
TL:DR Pairwise Win Rale Matrix Ntemp-0.75)
TL,DR Pairwise Win Rate Malrix (temp--.
DRPO
0.8
-08
DRPO
0.44
0.76
0.38
0.78
Win Rale
0.29
0.27
0.56
0.4
PRO
0.15
0.22
0.28
-04
0.22
0.24
0.44
DPD
0.8
0.41
0.72
-00
-00
DRPC
DRPO
RPO
DPO
DRPC
DRPC
PPO
DPO
Figure 5: Pairwise Win Rates on TL;DR Dataset under different sampling temperatures (left: 0.75;
right: 0.25)
D Additional Empirical Results
In this section, we first provide pairwise win rates on the TL;DR dataset with other sampling tempera-
tures (see Figure 5) Our method consistently dominates across all temperatures. DPO's performance improves when temperature gets lower, which in line with results [26]. PPO's performance deteriorates in decreasing temperature. likely due to PPO is trained with default temperature 1.0.
Next, we present pairwise win rates on HH dataset with other sampling temperatures (see Figure (6). The results are consistent with that of temperature 1.0. In general, DRPO-GPM- ORPO-BT = DPO > PPO, showcasing the robustness our algorithm.
Additionally, we present some of the sampled responses of our method and baselines and how got-4o-min mini judges the quality of the completions. See Table 10 for TL:DR examples and
Table
12
3 14 for HH examples.
E
Limitation and Broader Impact
A potential limitation of our methodology is its reliance on IS ratios for preference evaluation, which can result in high variance when the target and behavior policies differ substantially. While we apply clipping to the IS ratios to partially mitigate this issue, the issue may still remain concern particularly when the reference policy differs substantially from the target policy. Additionally, although our experiments on training large language models with real-world datasets demonstrate the effectiveness of our approach, we did not evaluate on substantially larger-scale models due to hardware constraints. This is potential limitation of our experimental validation.
43

---

HH Pairwise Win Rate Mafrix (temp=0.75 75)
HH Pairwise Win Rate Mafrix (temp=0.29)
0.70
0.70
DRPO
3-11
0.50
0.60
0:85
0.48
0.47
0.50
0.5
0.60
0.80
DEPR
0:58
0:58
0.68
0.52
0.48
0.64
0.55
20
0.50
0:50
3.44
0.69
0:45
DP0
0.53
0.54
0.70
0.45
0:40
0.52
0.35
PPO
0.35
0.30
0.30
ST
GPM
BT
GPM
Figure 6: Pairwise Win Rates on HH Dataset under different sampling temperatures (left: 0.75; right: 0.25)
Our work contributes to the development of doubly robust approach to preference evaluation and optimization, which aims to improve the alignment of large language models (LLMs) with human preferences. This may improve models' ability, contributing to safer and more controllable LLM behavior. However, improved alignment methods may be misused, such as aligning models with the preferences of specific group will disadvantage others. Furthermore, if the training data contains preferences for harmful content, the model may learn and reproduce such harmful behaviors. The alignment algorithm itself does not produce harmful content; such outcomes arise only when the model is optimized to align with harmful preferences. Therefore, it is important to carefully manage dataset to prevent large language models from giving harmful responses.
44

---

Table 7: Sample Responses of a TL;DR Prompt and Judgement
Prompt
SUBREDDIT:
r/relationships
TITLE: [18/F] My friend [18/F) has naked picture of my boyfriend [17/M]
POST:
This is the weirdest situation ever so please bear with me. My friend was over earlier tonight and she was using my cell phone while was making food.
My cell phone has nudes of my boyfriend on it, I know it's stupid to do that so I don't need lesson about it please. We'v ve been together for over two years and we fully trust eachother and I'm never doing anything like it again. had gave her permission use my phone, she didn't 't just take it. But hadn't thought about the pictures, she was just using it to go online on though so didn't t think much of it. I came back in the room and got my phone back and we just went on with our evening. But about an hour after she leaves I got some email notifications on my phone, that was weird because don't 't have my email set up on my phone in the email app. So I check it and she had left her email signed in, I saw messages from herself herself, of pictures that was sent around the time she wouldve been using my phone. know it's kind of snooping but I clicked on it, and she literally sent pictures of my boyfriend on my phone, to her own email. So she probably has them saved on her computer or something now. have no idea how to confront her about because it's the most awkward situation ever, but have to say something. And I want to make sure she deletes those pictures. don't It even want to continue friendship anymore if she doesn't 't. feel completely betrayed and I think that was disrespectful my boyfriend, me, our relationship, and our friendship. haven't 't told my boyfriend yet as I' I'm sure he'll 'lll be mad at me, but I'll tell him after I resolve this. What can I do?! :( TL:DR:
DPO
My friend made sure to send naked pictures to me around the time she was using my phone just to "go online" apparently, there's ' obviously something suspicious since my boyfriend will be angry at me and how can I prove she doesn't 't actually have those pictures on her computer now?
DRPO-BT
my friend has nude pictures of my boyfriend on her cell phone. was over earlier tonight and she was using my phone go online. We didn't think much of it because we trust eachother. I got email notifications on my phone later that night that were of my boyfriend sending pictures of her to her own email. need advice on how to deal with this situation! I don't 't even want to continue our friendship if it is that bad.
Judgment
prefer Response A [DRPO-BT] because it concisely captures the main issue of the friend's betrayal regarding the nude pictures, while eliminating unnecessary details and maintaining clarity about the author's 's concerns.
45

---

Table 8: Sample Responses of a TL;DR Prompt and Judgement
Prompt
SUBREDDIT:
r/tifu
TITLE: TIFU by terrifying freshie for weeks
POST: So, this is about 4 years ago. I was sophomore in highschool. It was in the first week of school that me and my friend group heard the name of this freshman.. He had an awesome name. Idk how can make something comparable.. We'll call him Barry Powers. wad like that, a superhero type name. So anyway, we heard the name Barry Powers during freshman roll call and we were like "damn! What a sick name!" And made our ultimate mission to find this guy. And we asked, "do you know Barry Powers?" And other questions like "what does he look like?", "where did you last see him?", and "where can find him?". We really wanted to find this kid and congratulate him on his name. So, after weeks of this walking around and searching and asking, about Mr. Powers and their answer is "oh, he's right there." we get freakin hyped and beeline the dude, who looks us with fear in his eyes. Thinking back it, we had the poor kid, who was eating lunch on his own, in a corner surround by 4 of us. asked, "hey man. You Barry Powers?" "y-yeah.." He managed to respond. "oh, well, you have a sweet name!" said. "ok.." Mummledd Powers. Now, the next moment was like the end of an always sunny episode; my friends and I looked around and each other and decided, "oh cool we did it, moving on now." was only little bit later that we heard that this poor guy thought this was some massive hazing ritual the whole time. And since we had asked so many people, most of the freshman class thought here was some crazy hazing where we chose one random dude, hunted him down, and did something to him. Oops! TL:DR:
searched. It took weeks. Whenever
met other freshmen,
ask someone
DPO
Had the creepy freshies name Barry Powers searched for about four years and finally found extremely freshie who thought they had some hazing done it's name.
DRPO-GPM
TIFU by terrifying a freshie for 4 weeks for the name Barry Powers. The dude who was eating lunch by himself thought it was some huge hazing ritual and was devastated.
Judgment
prefer Response [DRPO-GPM] because clearly conveys the main concern of the post while eliminating unnecessary details, and it does so in a concise manner.
46

---

Table 9: Sample Responses of a TL;DR Prompt and Judgement
Prompt
SUBREDDIT: r/relationships
TITLE:
me (23f) lying to my boyfriend (27) about smoking
POST: So, I've been smoker since was 14 years old, I realize at this point how horrible it is for my health and after many failed attempts to quit seem to be having an extremely hard time. can't 't take chantix as I have epilepsy and anything with nicotine in it usually leads me back to smoking again. I'll get it in my head that im going to stop and then the addict in my brain finds some excuse to make it seem okay. My boyfriend of 3 years also used to be smoker but quit a few months back, he's been trying to make an honest attempt to get healthier and create more positive habits for himself and the last thing want to do is pull him away from that. He's caught me lying few times the past couple weeks. I feel guilty every time pick one up, not only because I' m lying but because it's just so bad for me and don't 't want to continue such a nasty habit. I realize that he probably doesn't t feel can trust me, that if it's easy for me to justify smoking and lying about it, what else might there be? He says he' dd prefer for me to just be honest with him and DGAF but it would absolutely kill me if he picked up smoking again because of my lack of impulse control which have always struggled with when it comes to doing the difficult thing. I'm afraid of failing and know that's exactly what im doing by not trying. I need to quit smoking. need to work on healthier habits and I'm having extremely difficult time working through the mental gymnastics that is impulse control and gaining motivation. Any advice on working towards becoming healthier and dealing with this situation and furthermore quitting would be awesome. TL;DR:
PPO
I've been smoker since I was 14 years old, realize at this point how horrible for my health and after many failed attempts to quit seem to be having an extremely hard time. I can't take chantix nicotine it usually leads me back to smoking again. need to quit smoking, I need to work healthier habits and I' I'm having extremely difficuult time working through the mental gy gymnastics that is impulse control and gaining motivation.
have epilepsy and anything with
DRPO-BT
I'm a smoker and I'm lying to my boyfriend about smoking and it's causing so much stress in our relationship, need help quitting and working through impulse control.
Judgment
prefer Response B DRPO-BT because it succinctly captures the main issues of lying about smoking and the associated stress in the relationship, while still conveying the need for help with quitting and impulse control, making it clearer and more concise summary of the author's 's main concern.
47

---

Table 10: Sample Responses of a TL,DR Prompt and Judgement
Prompt
SUBREDDIT: r'relationships
Found out my (28f) ex (32m) having kid, and now all want is to sleep around and m very depressed.
TITLE:
POST: I dated this guy last year. He impregnated 20f pretty quickly after broke up and when found out, I got extremely sad and jealous, because I wish on my way to having family. We broke up because of infidelity. Now though, all that want to do is out and get plowed by pretty much any guy, don't t want relationship. I just want to go out and have revenge sex. Why that? I think I' I'm just so upset, and angry. don't know what to do, or really how handle this.
TL:DR:
PPO
Found my (28f) ex (32m) is having kid, and now all I want is sleep around and I'm very depressed. Why is that? think I' I'm just so upset, and angry. don't know what do, really how handle this.
DRPO-GPM
Found out ex having kid, all want to do is have revenge sex, I'm very upset and angry. How do I handle this?
Judgment
prefer Response B [DRPO-GPM] because it eliminates unnecessary details while clearly conveying the author's main emotional state and question, leading to a more concise summary.
Table 11: Sample Responses of a HH Prompt and Judgement
Prompt
Human: I woke up with a really stiff neck that hurts when try turn my head, Can you help me manage the pain?
Assistant: You might have "stinger" neck, which common injury caused by sleeping in an awkward position.
Human:
think that's What can do for the pain?
Assistant:
DPO
For minor neck pain, over-the-counter pain relievers like ibuprofen or ac- etaminophen can help reduce the pain and swelling. You can apply warm towel to the area to promote relaxation of the muscles, and if the pain persists you can see doctor. Are you experiencing any other symptoms? For example, does your neck hurt when you move your head to the side?
DRPO-BT
You could use a heat or cold pack on the neck to reduce the pain and swelling. A heat pack is better for improving blood flow to the area, and a cold pack will help with reducing swelling and pain. You might feel better in 10-20 minutes. If the pain is severe, you could also take over-the-counter pain relievers like acetaminophen ibuprofen. You could also warm up your neck with some gentle movements, but be careful not force any movements that cause pain.
Judgment
prefer Response B [DRPO-BT] because provides clearer, step-by-step ap- proach to managing neck pain with both heat and cold options, as well as gentle movement suggestions, making it more comprchensive.
48

---

Table 12: Sample Responses of a HH Prompt and Judgement
Prompt
Human: What are some ways that I can get smoother skin?
Assistant: Here some links about few of them:
Try using rosewater to clean your face
Try a face mask on occasion
Try drinking lot of water
Try putting a mild facial peel on your face (A glycolic peel, or a salicylic peel, fruit acid peel)
Human: What is rosewater?
Assistant: Rosewater is special distilled form of rose petals. It's used for beauty products. Some people use it for watering the garden or for washing dishes, or to soothe the skin.
Human: How much water should I try drinking for smoother skin?
Assistant:
DPO
It depends on you, but generally you want to drink as much water as you can.
Drinking enough water helps your body flush out toxins.
You can drink water throughout the day, but it's probably best to drink at least eight 8-ounce glasses a day.
DRPO-GPM
Drinking lots of water has benefits for the health and appearance of your skin, as well other bodily functions. The US Centers for Disease Control recommends drinking 8-12 cups day, and most people's 's body needs vary based on their size, age, gender, and other factors.
Judgment
prefer Response [DPO-GPM] because it provides a clearer recommendation based guidelines from reputable source (CDC) and acknowledges individual variability in hydration needs, making more informative and supportive.
49

---

Table 13: Sample Responses of HH Prompt and Judgement
Prompt
Human: How do groom a cat?
Assistant: Grooming a cat involves combing their fur with a brush. It keeps the fur healthy, and also strengthens the bond between you and your cat.
Human:
That sounds good. Are there special brushes for cats?
Assistant:
PPO
Yes, there are many types of brushes for cats, including ones that have soft bristles for easy grooming and ones that have firmer, more efficient design for cleaning and brushing. Many cat owners like to use a finger brush, which can be especially helpful for removing tangles and mats.
DRPO-BT
Yes, there are different kinds of brushes, or grooming tools, as they're 'r often called. They come in variety of materials, like plastic or nylon, and they also come in different sizes. A cat brush for small cat will be very different than brush for large cat. You might also want to consider a "comb" or "slicker", which works on cat's fur from behind, and more effective for removing a lot of loose hair. slicker is specifically designed to remove hair, while comb is used to detangle or brush. Some of the brushes used for grooming have a long, flexible brush handle with a tiny comb or slicker, which you could use to groom your cat from the front or the back.
Judgment
prefer Response A [DRPO-BT]] because it provides more detailed explanation of different types of brushes, their materials, and their specific uses, offering the user broader understanding of grooming tools for their cat.
Table 14: Sample Responses of a HH Prompt and Judgement
Prompt
Human:
snowboarding hard to learn?
Assistant:
PPO
Snowboarding is challenging and demanding sport, and it takes practice and dedication to master the skills required for it. However, with proper instruction, training, and practice, it is possible for anyone learn how snowboard.
DRPO-GPM
It can be! You need good flexibility, balance, strength, and must be confident in your abilities. That doesn't mean you can't 't learn though. Many people can definitely learn from coaching and some even try to practice on their own, often in combination with lessons. Your best bet is to start somewhere, find class or someone to go with, and then figure it out as you go. You should be able to give snowboarding shot in few weeks.
Judgment
prefer Summary A [DRPO-GPM] because it provides more detailed and practical approach to learning snowboarding, highlighting the importance of flexibility, balance, and the value of coaching.
50


Reviews:
[{"review_id": "u8XdiLOAe9", "reviewer": "Reviewer_8Qi6", "summary": "The paper leverages the characteristic that preference learning consistency is maintained when either preference modeling or reference policy is correctly specified, proposing a novel preference alignment algorithm, DRPO, which uses a doubly robust learning approach that considers both simultaneously. Experiments were conducted on IMDb, TL;DR, and Anthropic Helpful and Harmless datasets, achieving superior performance compared to DPO.", "questions": "1. question whether this approach scales effectively to larger models and commonly used preference learning datasets.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. Decomposing the misspecification issue into preference modeling and reference policy problems and tackling them jointly is a novel approach.\n2. The core idea underlying this research's methodology, which consistency is maintained when either preference modeling or reference policy is correctly specified, has been verified both mathematically and through toy-scale experiments, providing evidence for the validity of this approach.\n3. The ablation studies on the main objective functions used in the experiments are well-conducted.\n\nWeaknesses\n1. I question whether the experimental setup is appropriate for demonstrating preference model misspecification and reference model misspecification. The IMDb sentiment, TLDR Reddit, and HH datasets have very low preference complexity, with pos-neg pair compositions that are easily distinguishable. The results obtained from such setups are inadequate for demonstrating the resolution of preference misspecifications, which is the main motivation of this research. This paper should be experience on datasets containing diverse human preferences like UltraFeedback, or in setups with mixed preferences within datasets as in the Dr. DPO paper (Which research target to robust preference alignement).\n2. The experiments were conducted on models under 1B parameters with relatively low-difficulty datasets. It is questionable whether the method will scale well to commonly used models around 7B parameters and widely adopted preference learning datasets like UltraFeedback.\n3. Given that this work targets robust alignment, there should be relevant baselines for comparison, but they are absent. Comparisons with methods like IPO or Dr. DPO should be included.", "was_revised": true, "final_justification": "My primary concern was the absence of a baseline, which the reviewer addressed by conducting seven additional baseline experiments.\nAdditionally, experiments on the robustness of model size resolved my related doubts.\nRegarding concerns about the experimental data, I determined it was not a significant issue, as the data is widely used in alignment research.\n\nAs a result, I have increased the score by 1 point.", "rebuttal": "We wholeheartedly appreciate your insightful comments. During the rebuttal, we have conducted extensive experiments to strengthen our empirical evaluation and address your concerns. In summary, we have: (i) scaled up the model size; (ii) included **seven** more recent baseline algorithms for comparison; and (iii) adopted more comprehensive benchmark protocols for evaluation. We next elaborate on these points. Finally, we discuss the appropriateness of our experimental setup for demonstrating preference model misspecification and reference model misspecification. \n\n> Weakness 3: Given that this work targets robust alignment, there should be relevant baselines for comparison, but they are absent. Comparisons with methods like IPO or Dr. DPO should be included. \n\nWe appreciate you pointing out these robust alignment algorithms such as Dr. DPO, and we fully agree that they are relevant to our work and have compared them empirically during the rebuttal. Before elaborating on our experiments, we first clarify our proposed DRPO and robust alignment methods like Dr. DPO address different forms of robustness. Specifically, algorithms such as Dr. DPO are **robust to the discrepancies between training and test distributions**, such as pairwise noise in the training data. In contrast, our DRPO is **robust to the model misspecification** (misspecification of the preference model or reference policy), **assuming the training and test data share the same distribution**. Our experiments, detailed below, also highlights this difference. \n\nTo address your comment, we compared against **seven** baseline algorithms during the rebuttal. These include the robust DPO methods such as **Dr.DPO** that you mentioned, as well as **cDPO** (Mitchell, 2023) and **rDPO** (Chowdhury et al., 2024). We also included more advanced DPO-type algorithms such as **RSO** (Zhao et al., 2023) and **CPO** (Xu et al., 2024). Finally, we included **IPO** that you mentioned, and **ORPO** (Hong et al., 2024), another recent preference optimization algorithm. Most algorithms were implemented using Hugging Face’s `trl` library. An exception is Dr. DPO, which we trained using a modified version of the DPO trainer, as its official `trl` implementation is not available.\n\nTable A1 below reports the win rate and length-controlled win rate of different algorithms relative to the SFT policy on the HH dataset. Both win rates are evaluated using AlpacaEval 2.0 (following the suggestion from Reviewer ZHXd). Table A2 reports the win rate of the proposed DRPO method compared to the seven baseline algorithms on the TL;DR dataset. \n\nOn the HH dataset, DRPO achieves higher win rates than DPO-style algorithms (**RSO**, **CPO**) as well as **IPO** and **ORPO**, performs comparably to robust DPO methods **rDPO** and **cDPO**, and underperforms **Dr. DPO**. On the TL;DR dataset, however, DRPO consistently achieves win rates above 50% across all comparisons—often exceeding **60\\%** and in some cases **90\\%**. \n\nThis performance difference likely arises from the presence (or absence) of pairwise noise in the datasets. DRPO is specifically designed to handle model misspecification rather than noisy labels. Its strong performance on TL;DR suggests that this dataset contains minimal pairwise noise—making DRPO more effective than methods tailored to robustness under label noise. In both the rDPO and Dr. DPO papers, the authors found that setting the flip probability to $\\epsilon = 0.1$ yields the best performance on the HH dataset, suggesting that HH contains certain levels of pairwise noise. We adopt the same $\\epsilon$ in our experiments to ensure these algorithms attain their best performance. These pairwise-noise-robust methods thus outperform or match DRPO on HH, as expected. Meanwhile, it is also possible to integrate these robust alignment algorithms with our DRPO to enhance its performance in handling pairwise noise: for instance, one could incorporate reward functions learned via robust DPO methods into our total preference estimator for preference optimization, or apply the weighting function $w$ from DR. DPO to reweight each observation, in order to handle pairwise noise.\n\n**Table A1**. Win rate and length-controlled win rate of different algorithms relative to the SFT policy on the HH dataset. Higher win rates indicate better performance.\n\n| Model    | LC Win Rate (%) | Win Rate (%) |\n|----------|------------------|---------------|\n| DPO      | 83.90            | 84.09         |\n| Dr. DPO  | 92.16            | 90.93         |\n| rDPO     | 86.89            | 85.71         |\n| cDPO     | 85.05            | 84.28         |\n| CPO      | 73.59            | 71.28         |\n| ORPO     | 75.92            | 53.91         |\n| IPO      | 78.29            | 78.88         |\n| RSO      | 80.62            | 79.50         |\n| DRPO     | 86.38            | 84.84         |\n\n**Table A2**. Win rate of the proposed DRPO method compared to various baseline algorithms on the TL;DR dataset. Higher win rates indicate better performance of DRPO over the baseline algorithm.\n\n| Additional Baseline Model | Win Rate (%) |\n|---------------------------|--------------|\n| DRPO vs Dr. DPO           | 72.5         |\n| DRPO vs rDPO              | 65.0         |\n| DRPO vs cDPO              | 63.5         |\n| DRPO vs CPO               | 90.0         |\n| DRPO vs ORPO              | 57.5         |\n| DRPO vs IPO               | 98.5         |\n| DRPO vs RSO               | 69.5         |\n\n> Weakness 2 and Question 1: The experiments were conducted on models under 1B parameters. I question whether this approach scales effectively to larger models. \n\nWe apologize for any confusion. First, in the main paper, we clarify that only the **preference evaluation** experiments in Section 6.1 were conducted with models with less than 1B parameters. For the **preference optimization** experiments on the HH and TL;DR datasets, we use larger base models -- each with billions of parameters -- as discussed in Appendix C.2.\n\nSecond, to directly address your concern, we have conducted an additional experiment during the rebuttal period using a **7B base model** for the preference evaluation. The results summarized in Table A3 below again validate the double robustness property of our estimator. We are happy to replace the original results with these updated results shall our paper be accepted.\n\n**Table A3**: MSE of the proposed preference estimator with a 7B base model. The preference model and reference policy can be misspecified or correctly specified. \n\nSample size | 500  | 1000 | 2000 | 3000  \n-|-|-|-|-\nBoth correct               | 0.002212 | 0.001160 | 0.000702 | 0.000390 \nWrong preference model  |0.024942 | 0.018757 | 0.016763 | 0.016594\nWrong reference model   |0.066897 | 0.021389 | 0.013358 | 0.008383\nBoth wrong                      | 0.265155 |0.069340 | 0.043276 | 0.045954\n\n\n> Weakness 1: Whether the experimental setup is appropriate for demonstrating preference model misspecification and reference model misspecification.\n\nWe demonstrate the appropriateness of our employed experimental datasets for investigating preference model misspecification and reference model misspecification below.\n\nFirst, **the HH dataset is well-suited for illustrating preference model misspecification**. As mentioned earlier, both the rDPO and Dr. DPO papers noted that this dataset likely contains unknown pairwise noise that is not captured by standard preference \n(e.g., BT) -- thus violating the assumed preference model. Indeed, the rDPO paper deliberately avoids flipping preferences in this dataset due to the suspected presence of inherent noise, and finds that setting the flip probability to $\\epsilon = 0.1$ yields optimal performance. The Dr. DPO paper similarly reports improved results when applying a flip rate of 0.1, even when the dataset itself is not explicitly flipped. These observations indicate that the HH dataset contains mixed preferences -- a setting you recommended to conducted experiments -- making it appropriate for testing the impact of preference model misspecification.\n\nSecond, **the TL;DR dataset is well-suited for illustrating reference model misspecification**. As detailed in Appendix C.2, we use the public model `cleanrl/EleutherAI_pythia-1b-deduped__sft__tldr` as the reference policy. However, this SFT model was trained on a filtered version of the dataset, which differs from the full TL;DR dataset used in our experiments (see Huang et al., 2024). As a result, the reference policy may be misspecified for our setting. Our experimental findings support this: we observe that DPO, which requires a correctly specified reference policy for consistency, underperforms PPO, suggesting potential reference model misspecification.\n\nFinally, **the IMDb dataset serves as a useful illustrative example for investigating both types of misspecification**. As it is a synthetic dataset, we have access to the ground-truth preference and reference models, allowing us to precisely control model misspecification. Notably, both the rDPO and Dr. DPO papers also use this dataset. For instance, rDPO introduces pairwise noise to simulate preference model misspecification, while in our setup, we intensify this by randomly assigning preferences (each response being preferred with 50% probability). Consequently, our simulated data has even larger preference complexity than in rDPO.\n\n--- \n\nWe hope these address your concerns, and we would be happy to discuss further if you have any follow-up questions."}, {"review_id": "g8uEs5kprS", "reviewer": "Reviewer_ZHXd", "summary": "The paper proposes a doubly robust preference optimization algorithm for aligning large language models with human preferences through reinforcement learning from human feedback. The algorithm remains consistent when either the preference model or the reference policy is correctly specified, addressing model misspecification issues in existing methods. It introduces a robust and efficient preference evaluation estimator and develops a corresponding optimization algorithm for LLM fine-tuning, demonstrating superior performance in both theory and practice.", "questions": "- Expand Related Work: Provide a detailed comparison with recent robust DPO methods.\n- Enhance Baseline Comparisons: Include more advanced and similar methods in the experimental evaluations.\n- Expand Experimental Scope: Conduct experiments on additional datasets and benchmarks to validate generalizability and robustness.", "limitations": "The authors have discussed the limitations in their manuscript.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. **Innovative Algorithmic Contribution**: The paper introduces a novel doubly robust preference optimization algorithm that addresses a significant challenge in RLHF—model misspecification. This algorithm is designed to remain consistent when either the preference model or the reference policy is correctly specified, which is a substantial advancement over existing methods that often fail under such misspecifications.\n2. **Theoretical Rigor and Empirical Validation**: The authors provide a thorough theoretical analysis of their proposed algorithm, establishing its double robustness and semi-parametric efficiency. \n\n### Weaknesses\n1. **Insufficient Related Work Discussion**: The paper's discussion of related work is somewhat limited, particularly in the context of robust DPO methods. Many recent advancements in robust DPO, such as \"Provably Robust DPO: Aligning Language Models with Noisy Feedback\" (ICML 2024), \"Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization\" (ICLR 2025), and \"ROPO: Robust Preference Optimization for Large Language Models\" (ICLR 2025), are not adequately compared or analyzed. This omission makes it difficult to fully assess the novelty and comparative advantages of the proposed method.\n2. **Weak Baseline Comparisons**: The choice of baselines in the experiments is relatively weak. The paper could benefit from comparing against more similar and advanced methods, such as the aforementioned robust DPO techniques. This would provide a more comprehensive evaluation of the proposed algorithm's performance and robustness in the context of existing state-of-the-art solutions.\n3. **Limited Experimental Scope**: The experimental evaluations, while demonstrating the algorithm's effectiveness, are somewhat limited in scope. The paper could be strengthened by including more comprehensive benchmark analyses, such as using datasets like AlpacaEval2 or Arena-Hard. These additional benchmarks would provide a more thorough assessment of the algorithm's performance across a wider range of scenarios and challenges, thereby enhancing the paper's overall impact and significance.", "was_revised": true, "final_justification": "The authors effectively addressed the core concerns through expanded related work (7 additional DPO algorithms), enhanced experimental evaluation (AlpacaEval 2.0), and clear theoretical positioning. While minor clarity issues remain regarding method scope and baseline positioning, the substantial improvements in technical rigor and comprehensive comparisons outweigh these concerns, warranting a 1-point score increase.", "rebuttal": "We wholeheartedly appreciate your insightful comments, which raise three concerns about (i) insufficient related work discussion (Weakness 1 \\& Question 1); (ii) weak baseline comparisons, including the lack of robust DPO algorithms and other advanced or closely related algorithms (Weakness 2 \\& Question 2) and (iii) limited experimental evaluations. In response, during the rebuttal, we have: \n\n* Provided a detailed discussion of the connections and distinctions between our doubly robust algorithm and existing robust DPO methods. We plan to incorporate this discussion into the paper shall it be accepted.\n* Added comparisons with **seven** additional DPO-style algorithms to further demonstrate the competitiveness of our proposed DRPO.\n* Employed AlpacaEval 2 to strengthen our evaluation by benchmarking across different algorithms.\n\nWe next elaborate on these points below.\n\n**Insufficient related work discussion**. We appreciate you pointing out these references on robust DPO, and we fully agree that they are relevant to our work and worthwhile to cite. We have also compared against these algorithms empirically (refer to our next response). Shall our paper be accepted, we plan to use the extra page to expand the discussion section to further elaborate on the connections and differences between our work and these approaches, as follows:\n\nRecent studies have developed robust variants of DPO (Chowdhury et al., 2024; Liang et al., 2024; Wu et al., 2025), focusing on distributional robustness -- that is, robustness to discrepancies between training and test distributions. In particular, these methods studied pairwise noise, where preference labels may be flipped in the training data. In contrast, our algorithm does not address distributional shift or pairwise noise. Instead, it focuses on model misspecification and is designed to remain consistent under misspecification of the preference model or reference policy, assuming the training and test data share the same distribution.\n\nThus, the two lines of work are robust in different senses. Our newly conducted numerical study further highlights this distinction (see our response to your next comment). Specifically, robust DPO methods tend to perform better or comparably when the level of pairwise noise or distributional shift is high, whereas our method outperforms them when such issues are mild or absent. \n\nThat said, our DRPO framework is general and can be extended to handle pairwise noise: for instance, one could incorporate reward functions learned via robust DPO methods into our total preference estimator for preference optimization, or apply the weighting function $w$ from Dr. DPO to reweight each observation, in order to handle pairwise noise. \n\nIf the reviewer has additional related works to suggest, we would be glad to compare and discuss them shall our paper be accepted.\n\n**Experiments**. Following your suggestion, we compared against **seven** baseline algorithms during the rebuttal. These include the robust DPO methods **rDPO** and **Dr. DPO** that you mentioned, as well as **cDPO** (Mitchell, 2023). We also included more advanced DPO-type algorithms such as **RSO** (Zhao et al., 2023) and **CPO** (Xu et al., 2024). Finally, we included **IPO** (Azar et al., 2024)   and **ORPO** (Hong et al., 2024), which are two closely related algorithms to ours. Most algorithms were implemented using Hugging Face’s `trl` library. An exception is DR-DPO, which we trained using a modified version of the DPO trainer based on the authors’ released code, as its official `trl` implementation is not available. Unfortunately, due to the lack of publicly available code for **ROPO**, we were unable to include it in our experimental comparison at this time.\n\nTable B1 below reports the win rate and length-controlled win rate of different algorithms relative to the SFT policy on the HH dataset. Both win rates are evaluated using **AlpacaEval 2.0** you suggested. Table B2 reports the win rate of the proposed DRPO method compared to the seven baseline algorithms on the TL;DR dataset. \n\nOn the HH dataset, DRPO achieves higher win rates than DPO-style algorithms (**RSO**, **CPO**) as well as **IPO** and **ORPO**, performs comparably to robust DPO methods **rDPO** and **cDPO**, and underperforms **Dr. DPO**. On the TL;DR dataset, however, DRPO consistently achieves win rates above 50% across all comparisons—often exceeding **60\\%** and in some cases **90\\%**. \n\nThis performance difference likely arises from the presence (or absence) of pairwise noise in the datasets. DRPO is specifically designed to handle model misspecification rather than noisy labels. Its strong performance on TL;DR suggests that this dataset contains minimal pairwise noise—making DRPO more effective than methods tailored to robustness under label noise. In both the rDPO and Dr. DPO papers, the authors found that setting the flip probability to $\\epsilon = 0.1$ yields the best performance on the HH dataset, suggesting that HH contains certain levels of pairwise noise. We adopt the same $\\epsilon$ in our experiments to ensure these algorithms attain their best performance. These pairwise-noise-robust methods thus outperform or match DRPO on HH, as expected. \n\nFinally, we note that TL;DR is a domain-specific summarization task. General-purpose benchmarks such as AlpacaEval 2.0 may not be appropriate for the evaluation on this dataset. Therefore, we follow the evaluation protocol from DPO and P3O to assess the quality of the generated responses from each algorithm (Note that rDPO and Dr. DPO did not report results on this dataset).  \n\n**Table B1**. Win rate and length-controlled win rate of different algorithms relative to the SFT policy on the HH dataset. Higher win rates indicate better performance.\n\n| Model    | LC Win Rate (%) | Win Rate (%) |\n|----------|------------------|---------------|\n| DPO      | 83.90            | 84.09         |\n| Dr. DPO  | 92.16            | 90.93         |\n| rDPO     | 86.89            | 85.71         |\n| cDPO     | 85.05            | 84.28         |\n| CPO      | 73.59            | 71.28         |\n| ORPO     | 75.92            | 53.91         |\n| IPO      | 78.29            | 78.88         |\n| RSO      | 80.62            | 79.50         |\n| DRPO     | 86.38            | 84.84         |\n\n**Table B2**. Win rate of the proposed DRPO method compared to various baseline algorithms on the TL;DR dataset. Higher win rates indicate better performance of DRPO over the baseline algorithm.\n\n| Additional Baseline Model | Win Rate (%) |\n|---------------------------|--------------|\n| DRPO vs Dr. DPO           | 72.5         |\n| DRPO vs rDPO              | 65.0         |\n| DRPO vs cDPO              | 63.5         |\n| DRPO vs CPO               | 90.0         |\n| DRPO vs ORPO              | 57.5         |\n| DRPO vs IPO               | 98.5         |\n| DRPO vs RSO               | 69.5         |\n\n--- \n\nWe hope these address your concerns, and we would be happy to discuss further if you have any follow-up questions."}, {"review_id": "L4NEPvkgJj", "reviewer": "Reviewer_Q4tG", "summary": "The authors set to improve the RLHF paradigm by recognising the fact that preferences, reward models and reference policies can be specified. To that end they introduce DRPO, which introduces doubly-robust methods to RLHF. The proposed method converges to the true preference probability when either the preference model or the reference policy is correctly specified. The method is also semi-parametric, meaning it attains the smallest MSE among all regular and asymptotically linear estimators.  \n\nThe method evaluates preferences over policies and is built from a direct estimator and an importance sampling estimator. The direct estimator is the preference model using an existing reward model or DPO-type estimator (which can be generalized beyond BT models). The importance estimator which evaluates the ratios of completion pairs drawn from the trained policy and reference policy. These two estimators are combined to form the DR estimator. The main contribution is the second term in eq 8, which corrects for bias introduced by misspecification of the preference model. \n\nThe authors prove a number of theoretical results, including a regret bound (which is more robust to optimization than PPO and DPO alone). The authors also prove semi-parametric efficiency and double-robustness. \n\nFinally, the authors perform a number of experiments. The first shows that their method works by ablating the reference policy and preference specification. The second runs a head-to-head comparison of DRPO  against PPO and DPO on standard preference optimization tasks.", "questions": "- Can the authors please clarify in eq 8, which part of the equation addresses preference model misspecification, reward model misspecification and reference policy misspecification?\n- Can the authors expand on how this method can be generalised beyond Bradley-Terry?\n- Would the authors be willing to open-source the code?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n- I find the paper to be interesting and novel. Introducing the importance sampling estimator is a simple yet elegant solution towards making preference optimization more robust. \n-These claims are theoretically supported by the analysis on double-robustness, semi-parametric efficiency and regret analysis. \n- I find the regret analysis quite convincing since the authors show that their regret is bounded through the only the product of reference/preference policy error. \n- The first experiment shows how the DRPO is robust to either type of misspecification. I appreciate the use of multiple seeds and confidence intervals, which are often missing in LM research. \n- I was convinced by the second experiment, which shows a practical experiment where the reference policy is trained on a different dataset. \n- I find the problem setting, theoretical results and empirical results quite convincing. \n\n# Weaknesses\n- While the paper is generally well written, I got lost in the details of section 4. I was confused because the authors mention that eq 6 and 7 require correct preference and reference policies. Then, through eq 8, the authors mention that the correction term addresses misspecification of the preference model. It was unclear to me how misspecification of the reference policy is handled. I think it would be helpful at the end of section 4 to show how each part of eq 8 supports the main claims of the paper. \n- no code provided.", "was_revised": true, "final_justification": "I found the problem, theoretical analysis and experiments quite convincing. Even if the novelty might be marginal, I still think the research is useful for the community, and the analysis is thorough enough to justify acceptance. The authors addressed my concerns during the rebuttal; I stand by my score and increase my confidence.", "rebuttal": "We wholeheartedly appreciate your insightful comments, which we will carefully incorporate should the paper be accepted. Below, we respond to your comments and questions. \n\n> No code provided. Would the authors be willing to open-source the code?\n\nWe would like to clarify that we did include our code in the Supplementary Materials. \n\nOf course, we are happy to open-source our code. Indeed, we have already published our code in an anonymous GitHub repository (Due to NeurIPS policy, we are unable to include the link).  \n\n> While the paper is generally well written, I got lost in the details of section 4. I was confused because the authors mention that eq 6 and 7 require correct preference and reference policies. Then, through eq 8, the authors mention that the correction term addresses misspecification of the preference model. It was unclear to me how misspecification of the reference policy is handled. I think it would be helpful at the end of section 4 to show how each part of eq 8 supports the main claims of the paper.\n\n> Can the authors please clarify in eq 8, which part of the equation addresses preference model misspecification, reward model misspecification and reference policy misspecification?\n\nWe greatly appreciate the above two thoughtful comments. Following your suggestion, we plan to include a discussion at the end of Section 4 to clarify how each type of model misspecification is handled by Eq (8).\n\nTo begin with, we clarify that Eq (6) and (7) correspond to the DM and IS estimators, respectively. As you pointed out, their consistency depends on the correct specification of the preference model (for DM) and the reference policy (for IS). \n\nThe DR estimator combines the virtue of DM and IS by involving both the preference model and reference policy in Eq (8). Each type of misspecification is addressed as follows:\n\n* **Reference policy misspecification**. In this case, the preference model is correct (i.e., $\\widehat{g}=g^\\*$). The reference policy misspecification is addressed by the inclusion of the residual $Z-g^\\*$ in the second term of Eq (8). Specifically, since $\\mathbb{E}[Z-g^\\*(X,Y^{(1)},Y^{(2)})|Y^{(1)},Y^{(2)},X]=0$, by the law of total expectation, the second term in Equation (8) -- although it involves an incorrect IS ratio -- has zero expectation by the law of total expectation. As a result, the expectation of Eq (8) reduces to that of the first term, which corresponds to the DM estimator and remains consistent under correct specification of the preference model.\n\n* **Preference model misspecification**. In this case, we have $\\widehat{\\pi}\\_{\\textrm{ref}}=\\pi\\_{\\textrm{ref}}$, and we can rewrite Eq (8) into\n        $$\\textrm{Eq}~(7)\n        +\\Big[\\frac{1}{2}\\sum_{a=1}^2\\mathbb{E}\\_{y\\sim \\pi(\\bullet|X)} [\\widehat{g}(X,y,Y^{(a)})]-\\frac{\\pi(Y^{(1)}|X)}{2\\pi_\\text{ref}(Y^{(1)}|X)}\\widehat{g}(X, Y^{(1)}, Y^{(2)})- \\frac{\\pi(Y^{(2)}|X)}{2\\pi_\\text{ref}(Y^{(2)}|X)}[1-\\widehat{g}(X, Y^{(1)}, Y^{(2)})]\\Big]$$\n    Here, the misspecification of the preference model is addressed by the inclusion of the IS ratio in the second term within the square brackets. Specifically, with a correctly specified reference policy, the second term is of mean zero. As such, the expectation of Eq (8) is the same to that of Eq (7), which corresponds to the IS estimator and remains consistent under correct specification of the reference policy.\n\n* **Reward model misspecification**. Under the BT model assumption, one can estimate the reward and plug in this estimator to estimate the preference function. If the reward model is misspecified, the resulting preference model $\\widehat{g}$ is also misspecified. Nonetheless, as in the previous case, the inclusion of the IS ratio in Eq (8) corrects this error. Thus, the same logic used to address preference model misspecification applies here as well.\n\n> Can the authors expand on how this method can be generalised beyond Bradley-Terry?\n\nAbsolutely. Our proposed doubly robust algorithm relies on estimating two nuisance functions: (i) the reference policy, and (ii) the preference function -- which defines the probability that one response is preferred over another, given the prompt. When the BT model assumption is violated, this preference probability can no longer be expressed as a sigmoid function of the reward difference between two responses. However, the preference probability itself remains well-defined and can still be directly estimated from data.\n\nFor example, general preference models (e.g., Zhang et al., 2024) model preferences without assuming the existence of an underlying scalar reward function. Other models such as PairRM also provide valid estimates of pairwise preference probabilities without relying on the BT assumption. These models can be used to estimate the preference probability, which can be plugged into our total preference estimator for the subsequent preference optimization.\n\nFrom a theoretical perspective, without the BT model assumption, a scalar reward function that measures the goodness of individual responses may not exist, rendering the policy value undefined. To address this, we use the gap in total preference as the measure to gauge the theoretical performance of our algorithm (see Theorem 5 \\& Corollary 6). If a reward function does exist but the preference probabilities follow a general (non-sigmoid) link function, our Theorem 7 can be extended to derive similar suboptimality bounds in terms of policy values. We elaborate on this in our Response \\#2 to Reviewer YVEF (see **Beyond BT model** paragraph). \n\n--- \nWe hope this clarifies your question, and we would be happy to elaborate further if you have any follow-up questions."}, {"review_id": "WShvVjU5E9", "reviewer": "Reviewer_b762", "summary": "This paper adopts doubly robust estimator for the offline policy evaluation literature and apply it in the context of large language model fine-tuning and alignment. This method is useful when the either reference model or reward model is misspecified. The double robustness ensures that unless the product of misspecification rate is converging root-n rate, the estimator is consistent. Besides theoretical justifications, this paper also demonstrates the effectiveness of the method empirically.", "questions": "- In Line 110, it says that Y1 and Y2 are sampled from the reference policy. But there are cases where the two responses are not generated from the reference policy like Anthropic’s HH dataset? Do your results, in particular, double robustness, rely on that assumption? How to deal with the case when the reference policy is unknown?\n- Typo in line 195-196? (9) should be reduced to IS estimator when setting g_hat to zero and the DM estimator when setting the IS ratio to zero?\n- Assumption 4 is unlikely to be relevant for LLM where the model capacity is really large and VC dim is thus big too. While it seems fine asymptotically, because one can have bounded and fixed VC dimension and simply increase sample size. But typically, as one increases the sample size, one would use a larger model and thus larger VC dimension as well?\n- I am kind of surprised my the relative poor performance of PPO here even compared to PPO. Are you using standard hyper-parameter? Or do you do the hyperparmater search for PPO yourself?\n- In line 265, when do you mean by “when the BT model assumption is violated”? In that case, how does one define preference probability?", "limitations": "Experiment setting are weak. See the weakness section.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- Strengthes\n    - The paper is well-organized and clearly-written.\n    - The idea is novel. I personally have thought about this before as well and it is nice to see it executed here.\n- Weaknesses\n    - The experiment result might be week. The base model only uses 125m parameters. That’s way below standard.", "was_revised": true, "final_justification": "I am voting for acceptance because i think this is a good paper.", "rebuttal": "We wholeheartedly appreciate your insightful comments, which we will carefully incorporate should the paper be accepted. Below, we first respond to your concern on our experiments. We next respond to your questions.  \n\n> The experiment result might be weak. The base model only uses 125m parameters. That's way below standard.\n\nWe apologize for any confusion and are grateful for the chance to clarify. First, the 125M model was used only for the **preference evaluation**  in Section 6.1, where the goal is to evaluate the MSE of our proposed doubly robust preference estimator. For the **preference optimization** experiments on the HH and TL;DR datasets, we use much larger base models -- each with billions of parameters -- as discussed in Appendix C.2.\n\nSecond, in response to your concern, we have conducted an additional experiment during the rebuttal period using a **7B base model** instead of 125M for the preference evaluation. The results summarized in Table A1 below again validate the double robustness property of our estimator. We are happy to replace the original 125M results with these updated results if needed.\n\nThird, we have substantially expanded our numerical study during the rebuttal by: \n\n* adding **7** new baseline algorithms for comparison;\n* using new benchmarks for evaluation.\n\nRefer to our responses to Reviewers 8Qi6 and ZHXd for details. \n\n**Table A1**: MSE of the proposed preference estimator with a 7B base model. The preference model and reference policy can be misspecified or correctly specified. \n\nsample size | 500  | 1000 | 2000 | 3000  \n-|-|-|-|-\nBoth correct               | 0.002212 | 0.001160 | 0.000702 | 0.000390 \nWrong preference model  |0.024942 | 0.018757 | 0.016763 | 0.016594\nWrong reference model   |0.066897 | 0.021389 | 0.013358 | 0.008383\nBoth wrong                      | 0.265155 |0.069340 | 0.043276 | 0.045954\n\n> In Line 110, it says that Y1 and Y2 are sampled from the reference policy. But there are cases where the two responses are not generated from the reference policy like Anthropic’s HH dataset? Do your results, in particular, double robustness, rely on that assumption? How to deal with the case when the reference policy is unknown?\n\nWe make some clarifications. First, $Y_1$, $Y_2$ are the compared responses in the data, rather than those simulated based on our algorithms. We define the reference policy as the distribution of these responses given the prompt. \n\nSecond, our proposal accommodates unknown reference policies, as is the case with the HH dataset. It does **not** require access to the oracle reference policy. In such cases, we can estimate the reference policy via SFT and plug in this estimator for preference optimization. Indeed, the double robustness property is particularly relevant in these cases: it guarantees consistency as long as either the reference policy or the reward (or preference) model is correctly estimated—not necessarily both.\n\nWhen estimation errors exist, our algorithm's performance gaps in Theorems 5 \\& 7 depend explicitly on these estimation errors. However, due to the double robustness property, these errors enter our suboptimality bound in a **product (second-order)** form, whereas in PPO and DPO, they appear **linearly (first-order)**. As a result, when both estimation errors converge to zero -- thanks to the strong approximation capabilities of transformer-based architectures -- our algorithm’s suboptimality gap converges to zero much faster than that of PPO and DPO.\n\n> Typo in line 195-196? (9) should be reduced to IS estimator when setting $\\hat{g}$ to zero and the DM estimator when setting the IS ratio to zero?\n\nThanks for pointing it out. You are right. We will correct the typo. \n\n> Assumption 4 is unlikely to be relevant for LLM where the model capacity is really large and VC dim is thus big too. While it seems fine asymptotically, because one can have bounded and fixed VC dimension and simply increase sample size. But typically, as one increases the sample size, one would use a larger model and thus larger VC dimension as well?\n\nExcellent comment -- thank you for raising it. Before clarifying the VC dimension, we note that Assumption 4 is only required in Theorems 5, 7 and Corollary 6. It is not needed in Theorem 2 and Corollaries 3 \\& 4. \n\nSecond, the VC dimension in Assumption 4 can be restricted to the complexity of the policy class $\\Pi$ used during fine-tuning, rather than to the entire function class represented by the underlying LLM. While LLMs do have high capacity, fine-tuning typically restricts optimization to a much smaller and more structured subset of the parameter space -- for instance, For example, fine-tuning is often performed for only a few epochs, updates may employ low-rank adaptations such as LoRA (Hu et al., 2021), and KL regularization keeps the fine-tuned policy close to the reference policy. As a result, the effective policy class can have a much smaller VC dimension than the full model class, making this assumption more practical.\n\n> I am kind of surprised my the relative poor performance of PPO here even compared to PPO. Are you using standard hyper-parameter? Or do you do the hyperparmater search for PPO yourself?\n\nAre you referring to the relatively poor performance of PPO compared to DPO, rather than PPO itself? We employed two datasets in our main paper: TL;DR and HH. Interestingly, **PPO outperforms DPO on TL;DR**, but **performs worse than DPO on HH**.\n\nTo explain this, as detailed in Appendix C.2 of our paper:\n\n* For the HH dataset, we train the reference  policy via SFT using the HH dataset itself. This ensures that the reference policy is well-aligned with the data and likely well-specified.\n* For the TL;DR dataset, we use the public model `cleanrl/EleutherAI_pythia-1b-deduped__sft__tldr` as the reference policy. However, this SFT model was trained on a filtered version of the dataset, which differs from the original dataset used in our experiments (see Huang et al., 2024). As a result, this reference policy may be misspecified for TL;DR.\n\nThis explains the difference in PPO's performance compared to DPO: DPO performs better on HH (where the reference policy is correctly specified), whereas PPO performs better on TL;DR (where it is not). It also aligns with our theoretical analysis, which shows that DPO's suboptimality gap depends on the estimation error of the reference policy. \n\n**Regarding hyperparameters**, we conduct our own hyperparameter search for PPO on the HH dataset.\nSpecifically, as we mentioned in Appendix C.2, training PPO on the HH dataset is highly sensitive to the reward model. We experimented with different numbers of training epochs for the reward model and found that when the reward model overfits, PPO tends to suffer from policy collapse, hacking the reward model by repeating high-reward tokens. To mitigate this, we limit the reward model training to just one epoch to avoid overfitting. We then conduct a hyperparameter search over KL coefficients $\\beta \\in (0.05, 0.1, 0.2)$ and learning rates $\\alpha\\in (1\\text{e-}7, 1\\text{e-}6, 3\\text{e-}6)$. Finally, we set $\\beta=0.05$ and $\\alpha=1\\text{e-}7$ as it yields the most stable training performance.\n\n> In line 265, when do you mean by “when the BT model assumption is violated”? In that case, how does one define preference probability?\n\nViolation of the BT model assumption refers to settings where human preference does not follow the BT model -- the probability of preferring one response over another cannot be expressed as a sigmoid function of the difference between their scalar rewards. Moreover, in these settings, a scalar reward function that measures the goodness of a response may not even exist.\n\nNonetheless, the preference probability remains well-defined as a function of two responses and the prompt, even though it is no longer a function of reward differences. Additionally, these preference functions can be modeled using more general preference functions (see e.g., Zhang et al., 2024), without assuming an underlying scalar reward function.\n\nSince the reward function may not exist in these settings, the value of a policy is not well-defined. As such, we use the gap in total preference as the performance measure for our algorithm (see Theorem 5 and Corollary 6).\n\n--- \nWe hope this clarifies your question, and we would be happy to elaborate further if you have any follow-up questions."}, {"review_id": "5Ec5ZIJP3J", "reviewer": "Reviewer_YVEF", "summary": "This paper comes up with a doubly robust estimation for RLHF. By bringing the doubly robust framework into the RLHF setting, the authors proposed the DRPO algorithm. Theoretically, they showed that the algorithm is consistent whenever the estimation of $\\pi_{\\mathrm{ref}}$ is consistent or the estimation of the model is consistent. Additionally, they carried experiments over IMDb dataset to show that the doubly robust property indeed appears in practice.", "questions": "1. When $\\hat{\\pi}_{\\mathrm{ref}} = \\pi_{\\mathrm{ref}}$, does the algorithm has theoretically improvement over algorithm like PPO or DPO?\n2. Can you provide any form of lower bound to demonstrate that the terms in the upper bound are tight or necessary?\n3. Do you have any suggestions on improving the algorithm when the coverage assumption, i.e. $\\pi/\\pi_{ref}\\le \\epsilon^{-1}$ is not satisfied?\n4. When the reward model extends beyond the Bradley-Terry framework, does your algorithm still retain a meaningful upper bound or performance guarantee? If so, under what conditions?\n5. This paper explains the ‘doubly robust property’ in Theorem 2 (or Theorem 5) in terms of $\\|\\hat{\\pi}/\\pi_{ref}-1\\|^2*\\|g^*-\\hat{g}\\|^2$, as it goes to zero if any one of the above terms goes to zero. I understand that DPO/PPO are two type of algorithms which let the above two terms goes to zero respectively, but by simply from this formula it’s not clear how do we do better than any of these two algorithms. It would be better if the authors can explain an algorithm (not only a framework) which can theoretically better than PPO and DPO algorithm. I will increase my score accordingly if I am convinced that such algorithm exists.", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. This paper is well-written. The proofs are correct, and the numerical experiments seem promising.\n2. The problem of double robustness in RLHF is of great interest in the community, since the number of samples required by the RLHF algorithm is usually very large, hence any kind of reduction to the number of samples is useful.\n3. The algorithm DRPO also enjoys the 'doubly robust' property, which is useful in designing more general algorithms for RLHF, as it can improve the upper bounds guarantee.\n\nWeaknesses:\n1. From theoretical perspective, this paper merely join together two previously studied problem: the doubly-robust algorithms for RL, and also preference-based algorithms for RL. The semi-parametric analysis is also common in doubly robust RL literatures, e.g. [1]. Hence I question the novelty of this work. I would open to discussion and improving the score if the authors could properly justify the novelty of this paper.\n2. No lower bound is provided in addition to the upper bound of the DRPO algorithm. The absence of a matching or complementary lower bound leaves open the question of optimality.\n3. In the description of the results, the authors hide some factors that might be large, e.g. the dependence on the bound on the reward models. It will be better if these terms can be made explicitly in the upper bound.\n4. The term 'regret' seems a little bit confusing here, as regret usually refers to the culmulative performance difference terms, and the setting in this paper is not a sequential setting. It will be better if the authors can change this term into 'regret' into something like 'performance gap'.\n\n[1] Kallus, Nathan, and Masatoshi Uehara. \"Double reinforcement learning for efficient off-policy evaluation in markov decision processes.\" Journal of Machine Learning Research 21.167 (2020): 1-63.", "was_revised": true, "final_justification": "I think this paper has some nontrivial results in it. But the problem formulation still lacks some novelty, and no significant technical contribution is presented in this paper. Hence I deem that this paper is slightly below the bar of acceptance.", "rebuttal": "We wholeheartedly appreciate your insightful comments, which we will carefully incorporate should the paper be accepted. During the rebuttal, we have conducted extensive theoretical and empirical studies to address the concerns raised by you and other reviewers. Below, we summarize your major comments, followed by our responses.\n\n> **Technical novelty** (Weakness 1).\n\nWe understand your concern regarding novelty, given that both doubly robust (DR) methods and preference-based RLHF have been well-studied individually. Before discussing our technical novelty, we would like to highlight that the connection our work establishes between these two areas itself is novel and, we believe, highly valuable. While there is extensive literature on LLM fine-tuning, no prior work has approached it from the perspective of DR or semiparametrically efficient estimation. In that sense, our proposal not only introduces a novel methodology but also opens the door for integrating other DR and more general OPE methodologies (e.g., marginalized IS, Liu et al., 2018; conditional IS, Rowland et al., 2020; more robust DR, Farajtabar et al., 2018; to mention a few) into LLM fine-tuning. \n\nSecond, regarding technical novelty, even if one aims to apply DR to RLHF, this is far from straightforward. The primary challenge stems from the difference in settings: in RLHF, we do not observe scalar reward signals but instead receive pairwise human preference. Consequently, extending standard DR -- which are designed for scalar rewards -- to reward-based RLHF (see Section 2) for learning doubly robust policy values poses significant challenges. Our approach to this challenge is to shift the focus from learning policy values to directly learning preferences. By adopting the preference-based framework, we manage to apply DR to LLM fine-tuning.\n\nFinally, we highlight a unique aspect of our DR estimator that sets it apart from existing estimators. Due to our focus on pairwise preference feedback, we are able to leverage the inherent symmetry in the data: we can treat the second response as a reference and estimate the probability that the first response is preferred using a DR estimator, or alternatively, treat the first response as the reference and estimate the probability that the second response is preferred. Our analysis shows that neither estimator alone achieves semiparametric efficiency. However, averaging the two -- taking advantage of the symmetry in pairwise preferences -- does yield a semiparametrically efficient estimator. This insight explains why our proposed estimator (see Equation (8)) is constructed as the average of two estimators. This structure arises uniquely from the nature of pairwise comparison and does not appear in prior DR or efficient RL estimators.\n\n> **Improvement over PPO/DPO** (Questions 1, 4 \\& 5)\n\nThank you for raising this important point. We appreciate the opportunity to clarify how our algorithm improves upon both DPO and PPO. We structure our response as follows, discussing our improvements in three settings: (i) when the BT model holds; (ii)  when BT does not hold; (iii) where the reference policy is correctly specified. Notice that (ii) and (iii) directly address your Questions \\#4 and \\#1, respectively.\n\n* **When BT model holds**. Theoretically, our proposal enjoys two key properties: (i) double robustness; (ii) (semiparametric) efficiency. The first, as you noted, ensures robustness over DPO/PPO -- specifically, the error vanishes if either of two terms converges to zero, as is required by PPO or DPO individually. To quantitatively characterize our improvement over PPO and DPO, the second property -- efficiency -- comes into play.\n\n    Specifically, semiparametric efficiency means that, in the context of preference evaluation, the proposed estimator achieves the smallest possible asymptotic MSE (see Theorem 2 and Corollary 4). As for preference optimization, this efficiency is precisely characterized in Theorem 7, where we show that the suboptimality gap of our algorithms scales as the product $\\|\\|\\widehat{\\pi}\\_{\\textrm{ref}}/\\pi\\_{\\textrm{ref}} -1\\|\\| \\|\\|\\widehat{r}-r^*\\|\\|$, whereas for PPO and DPO, their gaps scale linearly with one of these terms alone: \n   \n    - PPO: $\\|\\| \\widehat{r}-r^*\\|\\|$\n    - DPO: $\\|\\| \\widehat{\\pi}\\_{\\textrm{ref}}/\\pi\\_{\\textrm{ref}} -1\\|\\|$\n\n    The product structure means that the impact of each estimation error on our algorithm is **second-order**, while in PPO and DPO it is **first-order**. Consequently, when both errors converge to zero (thanks to the approximation capabilities of transformer-based architectures), our algorithm's suboptimality converges to zero at a much faster rate than that of PPO and DPO. This provides a clear quantitative theoretical improvement over both PPO and DPO. \n\n    Additionally, our concrete algorithm is detailed on Page 12 of the Appendix. It is not merely a conceptual framework.\n\n* **Beyond BT model**. Yes, our algorithm retains meaningful performance guarantees beyond the BT framework. In fact, Theorems 2 and 5, as well as Corollaries 3, 4, and 6, do not rely on the BT model assumption. When the BT model does not hold, a scalar reward function that measures the goodness of each response may not even exist, rendering its value function undefined. In this case,  we use the total preference as the performance measure. For instance, Theorem 5 upper bounds the gap in total preference between our estimated policy and the one that maximizes the total preference. These guarantees hold under the coverage (Assumption 1) and model complexity assumptions (Assumption 4).\n\n    Next, suppose the scalar reward function does exist, but human preference does not satisfy BT. For instance, $\\mathbb{P}(y_1 \\succ y_2|x) = g(r(y_1,x) - r(y_2,x))$ for some monotonic link function $g$ that is not a sigmoid. In this case, the BT model is violated, and DPO/PPO no longer retain their theoretical guarantees. However, under Assumptions 2 and 3, we can extend our Theorem 7 to this setting and derive a similar suboptimality bound for our algorithm. This highlights another advantage of our approach: PPO and DPO require to correctly specify the form of the link function $g$, whereas our algorithm does not. \n\n* **When $\\widehat{\\pi}\\_{\\textrm{ref}}=\\pi\\_{\\textrm{ref}}$**. We focus on the case where the BT model does hold, since without this assumption, DPO and PPO no longer retain their theoretical guarantees. As discussed in our earlier response, the reward estimation error does not affect the suboptimality gap of our algorithm -- this term becomes zero. In contrast, PPO’s suboptimality still depends linearly on the reward estimation error. \n\nWe hope this response clarifies our theoretical improvements over DPO/PPO. Please let us know if you have any follow-up questions.\n\n> Lower bound (Weakness 2 \\& Question 2) \n\nWe focus on Theorem 7 which provides the suboptimality gaps for PPO, DPO and DRPO. For each algorithm, its gap contains three components:\n   \n* A bias term induced by KL-regularization (proportional to $\\beta$);\n* A statistical complexity term of the form $\\sqrt{v/n}$, which depends on the sample size $n$ and the complexity measure $v$ of the policy class;\n* A reward/reference policy estimation error term.\n   \nSince the first term can be made arbitrarily small by choosing a sufficiently small $\\beta$, we discuss the tightness of the second and third terms below. For the second term, our upper bounds for PPO/DPO match the lower bounds developed in Nika et al. (2024, arXiv:2403.01857), indicating their tightness. Finally, our theoretical investigation during the rebuttal reveals that  under certain settings -- e.g., when the reward is linear and the prompt distribution is multivariate Gaussian -- the suboptimality gap of PPO depends linearly on the estimation error of the regression coefficient, which itself is proportional to the reward model estimation error. Meanwhile, for DPO, when there is a constant gap between the specified and oracle reference policy, the algorithm suffers from a constant suboptimality gap that will not converge to zero. This demonstrates the tightness of the third term.\n\nGiven the tightness of the established bounds, we are more confident that our DRPO's suboptimality gap -- being second-order in the estimation error -- can be much smaller than those for PPO/DPO. \n\n> Coverage assumption (Question 3)\n\nGood point. First, we can clip the IS ratio (as implemented in our experiments) to improve the algorithm's stability. Second recent works such as Huang et al. (2025, arXiv:2407.13399) propose to replace the KL divergence with a chi-square divergence. This relaxes the uniform coverage assumption to a weaker partial coverage condition -- namely, coverage is only required to hold for the optimal policy rather than for all policies. We will discuss these approaches shall our paper be accepted. \n\n> Weaknesses 3&4\n\nWe will adopt the term performance gap and revise the bound to make its dependence on the magnitude of the reward function explicit.\n\n--- \n\nWe hope these address your concerns, and we would be happy to discuss further if you have any follow-up questions."}]

Author rebuttal:

Meta-review: This paper adapts the classical doubly robust estimator technique to the problem of offline alignment in RLHF. The main result is a new RLHF algorithm that achieves double robustness with respect to the preference model and the reference/data collection policy. The authors provide a sample complexity analysis for the algorithm, as well as an empirical validation. 

Reviewers initially had some reservations, but were positive after the discussion period, particularly because the authors were able to include more extensive experimental results. Overall, the paper is competently executed and well written, and is likely to be of interest to the RLHF community
Decision: Accept (poster)