Design Considerations in Offline Preference-based RL
Authors: Alekh Agarwal
Christoph Dann
Teodor Vanislavov Marinov
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning from Human Feedback
RLHF
DPO
Primary area: theory->learning_theory
TL;DR: We study offline preference-based RLHF algorithms like DPO, and provide theoretical guarantees on the quality of the learned policy for a large class of methods.

Abstract: Offline algorithms for Reinforcement Learning from Human Preferences (RLHF), which use only a fixed dataset of sampled responses given an input, and preference feedback among these responses, have gained increasing prominence in the literature on aligning language models. In this paper, we study how the different design choices made in methods such as DPO, IPO, SLiC and many variants influence the quality of the learned policy, from a theoretical perspective. Our treatment yields insights into the choices of loss function, the policy which is used to normalize log-likelihoods, and also the role of the data sampling policy. Notably, our results do not rely on the standard reparameterization-style arguments used to motivate some of the algorithms in this family, which allows us to give a unified treatment to a broad class of methods. We also conduct a small empirical study to verify some of the theoretical findings on a standard summarization benchmark.

Design Considerations in Offline Preference-based RL
Alekh Agarwal' Christoph Dann'
Teodor V. Marinov
Abstract
good policy from the preference dataset, with Direct Prefer- ence Optimization (DPO) (Rafailov et al., 2024b) being the pioneering approach in this line of work. These approaches are referred to as offline or direct alignment, since they do not draw any fresh samples from the learned policy during training, and only use the responses observed in the pref- erence dataset. This paper focuses on this latter family of algorithms, and studies how the properties of the data and the learning objective affect the quality of the learned policy.
Offline algorithms for Reinforeement Learning from Human Preferences (RLHF), which use only fixed dataset of sampled responses given an input, and preference feedback among these re- sponses, have gained increasing prominence in the literature on aligning language models. In this paper, we study how the different design choices made in methods such as DPO, IPO, SLiC and many variants influence the quality of the learned policy, from theoretical perspective. Our treat- ment yields insights into the choices of loss func- tion, the policy which is used to normalize log- likelihoods, and also the role of the data sampling policy. Notably, our results do not rely on the standard reparameterizaaioo-styy arguments used to motivate some of the algorithms in this family, which allows to give unified treatment to a broad class of methods. We also conduct small empirical study to verify some of the theoretical findings on a standard summarization benchmark.
Offline methods for RLHF such as DPO (Rafailov et al., 2024b). IPO (Azar et al., 2024), SLiC (Zhao et al., 2022; 2023) and KTO (Ethayarajh et al., 2024), along with a grow- ing number of variants have received a significant attention in the academic literature owing properties. First, the removal of an explicit reward learning step simplifies the number of modeling choices and steps required for the RLHF pipeline, along with reducing the demand on computational resources. Furthermore, the re- quirement to only evaluate a policy's 's likelihood on a fixed set of responses in the preference dataset, as opposed to gen- erating responses from the learned policy as in online RI, adds further resource efficiency. However, the growth of this literature has also spurred equally large body of work now detailing the various deficiencies of these techniques, such as the tendency of the algorithms to shift the proba- bility mass outside the support of the observed responses in the preference dataset, significant preference hacking be- haviors and notorious collapses in the learning dynamics with continued training (Pal al., 2024; Park al., 2024; Rafailov et al., 2024a; Fisch et al., 2024).
number of attractive
1. Introduction
The now substantial literature on Reinforcement Learning with Human Preferences (RLHF) can be broadly catego- rized into two families of methods. Given a dataset of human preferences, the first class of online algorithms are based on learning a reward function that assigns numerical scores to a response y given some input x, such that the high-scoring responses are preferred over the low-scoring ones in our preference dataset. These methods subsequently maximize this reward function using an online RL algo- rithm like PPO (Christiano al., 2017; Stiennon et al., 2020; Ouyang et al., 2022) or Reinforce (Ahmadian et al., 2024). A different approach forgoes the reward learning step and uses a reparameterization trick to directly learn a
While some of the issues raised above have received theo- retical treatment for specific approaches, the literature still lacks a comprehensive theoretical foundation underlying these offline RLHF techniques. Most of these techniques have been motivated by some variant of the original repa- rameterization argument the DPO paper, but the argument hinges unformalized assumptions regarding the coverage of the data with respect to the learned policy, and does not capture all the algorithmic variants which have subsequently been developed in the literature.
Eqqaa contribution 'Google Research. Correspondence to: Alekh Agarwal Kalekhagarwall www.:eawwoooooooooomo.oo Christoph Dann <chrisdann @ google.com>, Teodor Marinov <tvmari- nov@ moo@goolc.com...
In this paper, we instead adopt the perspective of offline RLHF as just solving a loss minimization problem on a preference dataset, and study when the optimal solution
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Offline Learning Preference-basee RL
Method
Base policy p(yla)
Loss l
Constrain/Reegularie
DPO (Rafailov et al., 2024b)
[ver(y/ar)
((2) log I+exp(-Pay
IPO (Azar et al., 2024)
((2) = (2 - 7)²
Slic-HF (Zhao et al., 2023)
1
{(2) max 2, 2,03
CE T
GPO (Tang et al., 2024)
(2) f(32)
CPO (Xu et al., 2024)
1
((z) log
CE CB(TTe TTe T
I+exp(-
R-DPO (Park et al., 2024)
(u)(() exp (gl01)
((=) = log
1 (+exp(-A2z)
ODPO (Amini et al., 2024)
{(z) = - log
I+exp( -8z4T)
SimPO² (Meng et al., 2024)
((z) = log
[++exp--P8 (83)
Table 1. A collection of methods for offline RLHI from preference feedback, along with the instantiation of the different design choices that make them special case of our general framework. For ODPO and SimPO, they are not included in the full generality in our framework as discussed in the footnotes.
of this loss minimization yields a desirable policy. A key
(- v/). We use the shorthand (z, y. V :w) ~ Dayu to succinctly denote samples from this generative process. The label w captures our preference for the response y over y' for the input æ. We study offline preference- RL methods, which are parameterized by some loss function I-R, Rj + [0, oo). In particular, we study methods which take as input policy class II S {X' - A())). and find policy which minimizes the following loss, given samples:
challenge in undertaking such study is the identification of a benchmark policy which is both desirable in terms of the responses it generates, and is attainable under reasonable assumptions using offine RLHF. With this background, our paper makes the following contributions.
1. We identify a benchmark policy to measure the perfor- mance of offline RLHF against. Prior work (Swamy et al., 2024) shows that the class of offline techniques considered here cannot attain the optimal policy for on- line RLHF in general, and we instead develop a weaker benchmark under reasonable assumptions on the data generating process and the learning setup. For DPO, this benchmark does correspond to the optimal softmax pol- icy when the preference data follows the Bradley-Terry Luce (Bradley& & Terry, 1952; Luce, 2012) model.
n # = argmin Ee
5e(-wi
log T(H(xx))
log (16/ (r:)) [ x4)) 1v (x;)
=ell Y=1
i=w?
Here j:: X -> TR refers to an arbitrary base policy, which does not need be normalized. Typically é is chosen so that incentivizes w,w,"" F' to be positive. This means that tends to increase the probability of producing yi compared with V/ relative the base probabilities under μt, when w, 1. In the particularly simple case when μ is just the uniform policy, we see that से results in larger probability of producing yi over y/, when wi = 1, which captures the basic intuition behind offline preference-basee RL.
2. We provide a bound on the sub-optimality of the learned policy to the yardstick identified above in learning framework that encapsulates most existing offline RLHF variants. The bound depends on the curvature of the loss function, as well as the coverage of the offline dataset.
3. We corroborate some of our theoretical findings using an empirical study
summarization task, and find that squared loss of IPO outperforms the logistic loss of DPO due to its nicer curvature properties, while dropping the normalization with reference policy likelihood causes a small, but consistent deterioration in the quality of the learned policy, as predicted by our theory.
Choice of policy class. A common policy parameteriza- tion is to use softmax policies To To(y & some parameter set O, where ff is some fixed network architecture. While we could explicitly define the policy class II this way, we intentionally leave II general this
with
2. Problem Setup
'ODPO the general case allows a margin of the form If (score(rr, y) -score(x, v/ )), which is not admissible in our setup as specified, though it can be incorporated in our analysis with some additional work. When f is the identity function, simply define μوy) x)) x =
Preference-based learning. We consider the setting of learning from offline preference data. In this setting, we are typically given a dataset of samples where each sample consists of an input x X ~ Dzs two responses y, v/ E y y Lid. Dy(-(z)) and binary label w E {-1,1}
y). a special case. "SiMPO in the general case has an additional length nor- malization, leading to the loss log(usigmoid log (3) /z)//vl B log ( [ y 7). This length normalization is currently not included in our theoretical setup and analysis.
2

---

Offline Learning Preference-based RL
point, to allow our framework to further include:
Existing offline RLHF methods. The formalization of meference-basee learning above captures a wide range of existing offline RLHF methods through appropriate choices of the base policy μ and loss f. Table 1 presents selection of methods that fit the setup and which our formulation cap- tures. We note that the methods predominantly vary along three axes: the loss function E, the choice of the base policy I and the choice of the constraint or regularizer to guide the policy optimization. In the next section, we present theoretical framework and our main technical results on the quality of the learned policy म. with an emphasis on understanding the effects of these choices. We note here that the GPO paper of Tang et al. (2024) considers an almost identical set of design choices (other than the flexibility in I and the choice of II) as our work, but their emphasis on empirical evaluation while we seek to understand the design space in theory.
Additional constraints (I): Several works (e.g. Zhao et al. (2023); Xu et al. (2024)) regularize or constrain the preference objective of Equation with a cross-entropy term based on CE(TT,, r) == Egg Da..0-o(--l))) In (y]az) where To is some pol- icy of reasonable quality. This is done by adding aCB( it) to the objective or constraint that CE( T S X. The addition keeps the loss optimiza- tion from degenerating when the distribution D under- lying our data has a limited support, and the optimal policy से might produce responses outside the support of Day. where we do not have preference feedback. We capture such additional constraints or regularizers as optimization with a reduced policy class II.
Early stopping: different way to constrain the extent of optimization is directly in the parameter space. Most often the preferred optimizer is first-order method such as Gradient Descent, AdaGrad, Adam or AdaFac- tor. Early stopping optimization with first-order meth- ods directly corresponds to a constraint in the param- eter space e (Yao et al., 2007: Raskutti et al., 2014; Neu & Rosasco, 2018; Suggala et al., 2018; Vaskevi- cius et al., 2019; Sonthalia al., 2024). For example, if GD is used as the optimizer, early stopping would correspond to an &g distance bound in parameter space between the parameters underlying и and the initial policy To at the start of optimization. Early-stopping or limited training in typical fine-tuning setups is a common practice that we again capture through an appropriate choice for II, e.g. with additional le con- straints the policy parameters.
While our setup is general and captures many existing RLHF methods, there are some approaches which are closely re- lated but do not quite fit our framework. S-DPO, (Chen et al., 2024), samples multiple responses with single response that is preferred to all others. Ramesh et al. (2024) study version of the problem where multiple pairs of responses are sampled from multiple groups and the objective is to minimize the worst-case expected pref- erence loss across groups. SimPO (Meng et al., 2024), nearly matches the DPO objective, however, the authors include additional length normalization, leading to the loss (3 log ((yll=//yll-5p log (t (//l=))//////>>)) There is no straightforward way incorporate the length normalization in our analysis. One option is to incorporate the normalization into the reward class, however, this also affects the policy class definition. We note that the other main modification in SimPO, which is to remove the nor- malization by the log- lig-likelshoods of the reference policy is indeed covered by our analysis through the choice of μ.
s(ylz) We assume for simplicity that the policy class II and the
base policy μ are chosen such that log M(ylz) log s(y (m))
m(y"(n) e
[-R,31 for x, y/ drawn according to the generative pro-
cess described above, with probability 1. Consequently, the
3. Analysis Framework and Main Results
loss on each sample is also bounded by some B, and we expect that the empirical loss minimizer ã from (1) has also small population loss:
In this section, we set up a framework for analyzing of- fline preference learning algorithms which optimize (1), and present our main results. We begin with a discussion to set up the performance criterion.
Lu( Dzuw) - minL,((T; Dayw) S Ens with (2) #ell
(T;; Dayu) = B( Dey 2,, (T,WW,,,,..)) = flw (ww-wwTThe will (I. (I.
3.1. Analysis Framework
Performance criterion. How to measure the efficacy of a preference- learning technique? As described above, based on our choices of f, μ and II, we get a guarantee on the expected loss of the resulting policy. But we want to measure how well the policy Ωे does in terms of producing highly preferred outputs y, given inputs x. It is not clear that a policy which has a small loss also produces good outputs. For instance, suppose that the learned â is such that
The error bound en typically scales as O( / Bdz/ /n), /n), with d, being the statistical complexity of such as In [II for finite classes, log-covering number for the infinite case, or other complexity measures like Rademacher or Gaussian complexities. We abstract such treatment into a general error term cn, as this analysis of the empirical loss minimizer is standard and not key focus of our study.
3

---

Offline Learning Preference-based RL
E[w|r, y, y RH (2, y. 3/) 0 for any x; y, in the sup- port of our training distribution, and the base policy je is uni- form. In this case, we can conclude that (y)lz) whenever [[w|a, V. > O. But this does not preclude the two probabilities from being extremely close, though correctly ordered, and more generally रे might still place a non-trivial probability on the least desirable outputs for many imputs z.
that for all x E &,y E V.1/ E
fty"l (yy/
w # (1,
argmin Ep, [M (rl u)la,,,11 vel-St.5]
We discuss in Appendix B.1 how this assumption can be relaxed to an approximate form of realizability, and still imply similar performance guarantees with a slightly dif- ferent structural assumption on the loss function é. We focus on exact realizability in the main text for the clean- est analysis. A necessary condition for Assumption 3.2 to hold is that there is fixed policy x which minimizes the loss f, in pointwise manner for all x. y/s U.y', when we take conditional expectation only over the preference labels The assumption further requires that within the range of Rx which parameterizes π E I1 we have that xt is the pointwise minimizer of E(w v). This makes the optimal policy x* independent of the distributions D and Dy over x, y, y'. To further understand why it is helpful to have such an optimal policy π², we make standard calibration assumption on the loss function E in Equation 1.
Ideally, would like say that स places most of the mass on the most desirable outputs y. Since the conditional prob- abilities D (r, y. B") can be interpreted as preference function Ply > l/) one notion of an optimal policy is provided by the Nash equilibrium policy for the two player- game encoded by this preference function, as considered in prior works (Wang et al., 2023; Munos et al., 2023; Swamy et al., 2024). However, as Swamy et al. (2024) show, this optimal solution cannot be attained by minimizers of the objective (1) in general, where they provide lower bound for the special case of DPO. Consequently, we need a dif- ferent yardstick to measure our performance for the setup of offline preference- gin with introducing some useful notation and our formal assumptions needed define the benchmark policy.
RL,, which we do next. We be-
Assumption 3.3 (Proper loss), We assume that the loss function é is a proper loss for class probability estima- tion (Reid & Williamson, 2010). That is, there is function ge which depends only on L, such that for all r e [0, [0; argmin, ne(v) + (1 ((-())) = ge(m).
Given the form of the loss in Equation 1, we have to reason about the log probabilities as the main object of interest. It is therefore convenient to denote for each policy a its log probabilities by R& with (x, y) = log π Ty]2z) and conversely by TR the policy associated with R. We will also refer to such R as a reward function since it measures the quality of the output y for input x, and the policy nn ascribes higher probability to outputs with high rewards under R. For the policy class II we can then define the ac- companying reward class as R = { R : π E IT]. Note that this is only nomenclature and is not a modeling assumption such as a reward-based Bradley-Teree Meelley-Tery-uuee (BTL) model of preferences in the data generating process.
That is, when we take conditional expectation over the bi- nary label according to probability 17, then the minimizer of the loss correctly recovers some loss-dependent function of n.. This condition is satisfied by most commonly used differ- entiable losses for binary classification such as the logistic loss, squared loss, squared hinge loss etc. For instance, the function is given by 9:g (1)((1) = 2n 1 for squared loss and (1) = In(n/(11 - -)) for the logistic loss.
Realizability, proper losses, and optimal policies. When we use proper losses, the realizability condition takes a particularly intuitive form when the data generating process and the gr function underlying the loss l agree with each other. For instance, suppose use the logistic loss and the preferences are generated according to a BTL model: P(w = 1 1|2, y, 3/) = 1/(1 + exp( (z, - R (z, y))) for some R' E R. Then under Assumptions 3.2 and 3.3, we have that x = TR for R such that for any x. y, y
Modeling assumptions. We start by assuming that the log-probabbe prebaabbiiities of all policies and the base policy μ are bounded, which ensures that the inputs of the loss function are in a bounded range. Additionally, we assume that also the loss outputs are bounded which holds for all practical loss functions, given the bounded domain [-R, +29..
Assumption 3.1. For all z. and all E II,
we have |R~ (I. v)1 < at (T; < B. 4 |ogg)())))
< 4
2t
and
R(z. g) R(x,V)) R (².2z) v) R (x,z')
(3)
+ +Inu(y(x)) - In uu (x).
That is, R is given by R' + In n. up to an x dependent offset, within the support of the data. A similar conclu- sion holds for the squared loss, and P(w = 1/2, y.8')= = 0.5 (R" (z. y) - R (z, v/)),2, /2. A more detailed discussion of how modeling Da, as part of the exponential family leads
Given the policy class II, we now make realizability as- sumption on the data generating mechanism with respect to this class.
Assumption 3.2 (Realizability). There exists n' E II such
4

---

Offline Learning Preference-based RL
to proper losses, f, can be found in Appendix A. We see that in these cases, the policy underlying realizability learns ground-truth reward function which underlies our data gen- eration process, In such scenario, where there is a reward function Rt underlying the observed preferences, a natural benchmark is the KL-regularized reward maximizing policy, T (y)a) xc (o(y)zz) (xp(3 (x, v)), where mo is some base policy with (such as the SFT policy), with respect to which the KL divergence is defined. When j Wo. then we see that the policy n¹ exactly corresponds to this optimal policy for an appropriately chosen loss function.
A sufficient condition for Assumption 3.5 is to instead have the stronger condition that for any u, 8 [- Rl, we have
{(u) 2 - (()) (u- u) + CH (u u)?.
2
This condition holds for Cpa = 2 for the squared loss: l(u,,) = (u - v²² and with an R-dependent constant for many other losses that are induced by log-likelihoodd of exponential families, which includes the logistic loss and the probit loss. Assumption 3.5 weakens this condition by requiring curvature on the expected loss L only around the optimal policy x², rather than pointwise on E.
Performance criterion under realizability. Based on these insights, we adopt the policy x² as our performance yardstick, and seek policy и to minimize (T,(T"I| [|x). Ap- proximately minimizing Ly may not be sufficient to derive meaningful bounds [I" however, as there still needs to be alignment between the data-generating distribu- tion Dy y and x² Otherwise, if there is no good coverage of the support of x²' (-1x)) by (-(r)) there is no guarantee that t will be able to distinguish good responses, y, according to x² from highly sub-optimal ones. This necessitates making the following coverage assumption.
3.2. Main Results
With our main modeling assumptions set up, we now give the main theoretical result on the KL divergence between an approximate minimizer of the population loss L D Dayw) and the benchmark n².
Theorem 3.6. For any E II such that [T, Dayu) [u( Days) S c, where the corresponding loss to Lp given by is proper, and under Assumptions 3-1-3.5, il holds that
Assumption 3.4 (Coverage for optimal policy). Let R(z,3) 3) = R(x, y) ED, [R(z, [r] denote the y-centered reward and let R denote the parameterization of n² Fur- ther, let A Af(2, y) = R(x, y) - R (xt, y). We assume that there exists constant C s.t. for R E R it holds that
[ Cp
Ce
a Ce
Ez
S
+
Cp
2
Cp
Ex Ex (112) LARr, 3) S C Ex D(, 3)
In Section B show version of Theorem 3.6 which only uses approximate realizability, that is, the version utilizes a relaxed version of Assumption 3.2.
The coverage condition is akin to generalized coverage con- ditions used in the offline RL literature (Xie et al., 2021; Jiang& & Xie, 2024). A sufficient condition to ensure this x" (ylar)
Remark 3.7 (Choice of loss function). Our bound scales inversely with the curvature constant, meaning that losses with high curvature will lead to more favourable bounds in Theorem 3.6. The squared loss satisfies Assumption 3.5 with c₂ = for any range of w ,P as it is strongly convex, while the squared hinge loss only satisfics Assumption 3.5
holds is to have s (,())=) I C, but the generalized no- tion also holds when the class R- = {wT o(z, y) ||w]|2 < 1} and we have that Amaax (E, 1/2 E,, x1/2) S C. Here we denote Σ = 1 (..nDDD (-yx>>-la))e)eeeee (0< v)0(z, 8) J. and abbre- viate Σ, = Ep,- Clearly, this second condition can be much weaker than the density ratio assumption, and indeed has underpinned several methods that effectively handle coverage issues in offline RL with large function spaces and nigh-dimensional data, motivating our definition here.
in (-oc, 1). finally the logistic loss satisfies Assumption 3.5 with a range dependent Cμ as the loss becomes less curved as the wt .M approaches co. Our theorem suggests that opti- mizing the squared loss is ideal in terms of the final bound as c₁ is constant and bounded away from 0 across the full range of the loss. Squared loss is also a proper loss, so that the main assumption which might fail is realizability. We do note that the probabilistic model corresponding to squared loss naturally is less realistic than say, the BTL model, corresponding to the logistic loss. Nevertheless, the benefits of squared loss are verified by our experiments in Section 5. We note that a related discussion of the curva- ture properties comparing the squared and logistic cases can also be found in Azar et al. (2024), who correctly identify similar issues with the curvature of the logistic loss, and use this to motivate the IPO algorithm. However, there is no
Before stating our main result we need somewhat standard curvature assumption on the loss.
Assumption 3.5 (Curvature around optimum). For any policy क II and functions R, R E R such that x = nR, x² nR. there is constant Cp > such that
(T; Drgw)) - (TT Dorw)) 2 E [wo (wwo" * " ((x,y./)) (w0T (I, 1²) "n " ((x v/)]
Cu E (W (w.""(,,,))))) - (x.","")))""
5

---

Offline Learning Preference-based RL
explicit quantitative analysis of the role of the link function beyond simple example, since the focus is more ob- taining a practical algorithm for the identity link function case. We instead derive precise convergence guarantees, and also highlight the role of offline data coverage which is not captured in the IPO paper.
degeneracies.
4. Analysis
The high-level reasoning to prove Theorem 3.6 is the fol- lowing. We first use Assumption 3.5 to establish that any e-minimizer of L(TR; Dzyw) admits a bound on the ex- pected error in the centered rewards (( [AR(z, y) We then invoke the coverage condition of Assumption 3.4 to translate this error bound to be under the benchmark policy n.. Subsequently, we relate the KI. divergence between π and x in terms of expectation of A AR(z, y)² under n², using a careful anaylsis of the log- partition function.
Remark 3.8 (Choice of base policy). While the choice of base policy does not appear directly in the KL bound, it in- fluences the realizability assumption. As already discussed, under a proper loss such as the logistic loss and a corre- sponding reward model such as BTL,, realizability becomes equivalent to having the reward model plus log μ term be part of the reward space R. Further, the choice of μ can change x², and as we discuss in Section 3.1, the choice of Teet naturally yields a desirable n.. In our experiments we use two commonly studied choices of ps, the uniform policy which puts equal probability on all responses, and a SFT policy Tre--
We start with the first step in the sketch above.
Lemma 4.1. Under Assumptions 3.2 and 3.5, any policy
a E IT with T. (x; Dayw) TTt( Payw) S c, satisfies En [A)(x,)) y)
Remark 3.9 (Effect of constraints). Recall that we constrain the optimization problem to a policy class II which captures any constraints that we incorporate such as CE to a SFT policy, To. or the implicit constraints induced by the choice of optimizer and early stopping. We note that the choice of II determines if the benchmark, n², which satisfies the realizability and curvature assumptions has to be part of II. In this context, a cross-entropy regularization to Tee essen- tially makes an assumption that n² lies in the vicinity of Tee. Biasing the reference policy in cross-entropy towards preferred responses such as in CPO (Xu et al., 2024) can be further beneficial in ensuring the feasibility of n* Remark 3.10 (Connections with prior results). As men- tioned earlier, there now a substantial literature on the degeneracies of DPO in particular, due its popularity, with primarily empirical (Park et al., 2024; Rafailov et al., 2024a; Fisch et al., 2024), but also some theoretical results in these works that demonstrate that DPO tends to shift mass away from the support of the preference data, with probabilities of both preferred and dispreferred responses in the data rapidly degrading to zero. The reader might wonder how to reconcile these negative observations with our positive result on the loss minimizers of DPO-style losses. However, there are few caveats which apply to the specific case of DPO. First, as we remarked earlier, the curvature constant cy for DPO degrades exponentially fast with R. While this scaling is not ideal we note that it is typical and appears in prior and concurrent works as well (Zhao et al., 2024; Zhu et al., 2023; Xiong et al.. 2023). Further, since DPO does not control the lig-likelihood ratios through regularization terms, the quantity Ai can rapidly grow large empirically, as pointed out in multiple papers, and as we corroborate in the next section. In fact, some works on incorporating pessimistic reasoning in DPO (Fisch et al., 2024; Liu et al., 2024: Cen et al., 2024: Huang et al., 2024) result in adding regularization terms which partly mitigate some of these
2x
=Exgy>ol(AR(r v)- A
CH
Proof. The first condition of the lemma is equivalent to
Bie (ww U. v./l)) 2 Eee (Wot "(x'""(xx,, "(x,,.0'))
+ Elwll (ww" (ww*"""(z,u) ((z,v.'')) ((w x (x, wn " "(x (x,,)'"))
+ E Eu (w - R(,, v') R (2, v) + R (x,,,')))
2
Now, Assumption 3.2 implies that for any E II.
Eujoww [6' (ww* "" (ww""""x." "(x.y.8))
(w(w= (...V..)) w w v. u')) 2 0,
where we have used the fact that e² = w= "(x, y, v/) is the minimizer of la [f(W o*]] together with first order optimality so that Eu (C(("" )w(( -²)] 2 0 for any and in particular for any, n, such that wt (x, y, v²). This, together with the second assumption of the lemma imply that
e 2 Ele (ww TM ((x, (x, y. v. s/)) - BE(E (w " v. "(,,
Eu (((w(R(a,)))
R(a, v/) R (x, + R (2, v/ ())))
2
where we have used the parametrizatioo of and n².
Next we show how to bound the KL by AR and the frac- tion of log- pagrtiatiionn functions 2,, and Z., where (x(r)= = Σ, exp R (I, (I).
Lemma 4.2. Under Assumption 3.4, for any π E II such that π x exp( R(x, v)), the expected KL. divergence Ez [KL( (11=)=()))]]](((((())))))
is bounded by
7 (z) v 2C 22C E.., / R(2, 2) E, log Z, (x)
6

---

Offline Learning Preference-based RL
Proof.
tinuing with our simplification, we get the bound
E B (-1u) log x (v(ax)
(1)
[,]KL(("'< (1+)) s Ex
Ex log Z. (z)
Sv B E Byys (-1x) AR(x, y)²
(1lx)
(vlar)
nt' (ular)
log (x(x)
2
Eg E Eywr(()))) (12) A AR AR(2.))² g)²
$ Eg / 2 (-Ix)
log
2
T (z)
(x)
Ce ch
Ce
+ Eg / 2/ (11x) (log Z (x)) - log (x:))
$ Cy 2 Cp
V
(x))
< V 20 Be Ee. x) + Eae log (z)
The first inequality above follows by triangle inequality to the sum inside absolute value, followed by Jensen's 's inequal- ity on the first term. Finally, we invoke Assumption 3.4 and Lemma 4.1 to control the last term, which completes the proof of the lemma.
where in second inequality follows from Jensen's 's inequality, and the third inequality follows from a combination of (z +
v)² I 2x.² + 2y² and v サリ I v + VD. Vz, y 2 0, and the third inequality uses Jensen to push the expectation
Combining Lemma 4.2 and Lemma 4.3 finishes the proof of Theorem 3.6.
inside the square root, along with Assumption 3.4.
0
The next lemma bounds the ratio of log-partitions.
5. Experiments
Lemma 4.3. For any such that [(x;; Daygs) L(TT; Dayw) S e, it holds that
We evaluate the impact of different design choices for offline RLHF methods on the standard TL;DR summarization task (Völske et al., 2017: Stiemnon et al., 2020), where the task is to provide short summaries of articles. For our policy, we use a large T5 model (Raffel et al., 2020) with 770M parameters, which has been fine-tuned to maximize the log- likelihood of the human responses in the TL;DR dataset. This ensures that the policy is initialized with parameters that have reasonable likelihoods for the responses observed in our data. We experiment with two different losses, é:
(x)
Ce at Ce C/
Ex log
S
[z(
2
Cy.
Proof: Using the definition of 7T we have
2 (x)
exp Rog (x, v))
E log
E.p log
Z. (x)
Zz(x))
y
((x) = log( (1 + cxp(-3x)) ((x) (3z - 1)?:
(Logistic loss) (Squared loss)
exp( Re (x, v))
= Eg log
(z)z)
exp R. (z. y))
where 3> 0 is a hyper-parameter, governing the strength of how much each preference in the dataset should in the policy. We pair each of losses with two possible choices for the base policy μc: The uniform base policy μ = 1, which results in the following optimization
Here the second equation rearranges the definition x² (glz) = exp(R. (x, (x). Proceeding further
(x)
E log 7.(x)) Ex, In E Eynrr( exp(AR((xg)) y))l
min Σ TELL 1=1
E( - ws (log π (w.(2;)) - log x (s<lx)))
< E2 In 1+ 11 AR(z,y)
and the second choice uses μ Tee. the policy obtained after fine-tuning on TL.DR which our optimization is ini- tialized with. This corresponds to the objective is:
+ 2
2 (1)r)(1)))) A u)²)
< E5 E Ey))(())) Al((, v)
min Σe( e(--us wwwaaaooog (log
=x)](aa)) llog mn 2.) xeat (x(1))))))
(1/(1z)) ))-
+ 20 Eg Byran (12) AR(z, y)²
2
We recall that when μ = Tee. using the logistic loss from (Logistic loss) corresponds to the DPO algorithm (Rafailov et al., 2024b) and the squared loss from (Squared loss) cor- responds to IPO (Azar et al., 2024).
Here the first inequality uses that e" < 1 + x + e 2/2, Vr & A. The second inequality uses In(1 + x) S x. Con-
7

---

Offline Learning Preference-based RL
Comparison with base polley Rly VS. uniform
Comparison against
Loss:
0.6
Loss: logistte equarreed
0.8
log
Esslntion Preference
0.5
0.7
Preferencs
0.4
uniiorm
0.6
0.3
02
0.3
0.3
0.4
5000
10000
15000
20000
0
5000
10000
15000
20000
Training steps
Troining steps
Figure 1. Left panel shows the preference of the learned policy's 's summaries against those from the initial policy Twi, as evaluated by a prompted Gemini 1.0 Ultra model. Shaded regions represent 95% error bands. Both the logistic loss variants quickly improve in terms of the preference scores initially, but then suffer a catastrophic collapse. Squared loss improves at a similar rate initially, and remains stable throughout the training regime. Right panel shows a direct comparison between the variants of logistic loss using l uniform and = Ter (DPO) at regular intervals in the training process. Interestingly, the uniform variant is preferred in the early stages of training, but the training collapses around the training step 5K. the her variant starts to improve. Nevertheless, the absolute performance of both variants reaches its peak earlier in the training and rapidly worsens after 5K steps, suggesting that the preference for THe over uniform in this region might not be particularly significant. See text for more nuanced discussion.
For each variant, we tuned the 3 parameter in the loss in the interval {0.1, 0.5, 1.0]. For logistic loss, the best results obtained = 0.1, while we did not see a significant difference across these choices for the squared loss, and show the results at 3 = 0.5. See Appendix C for details.
To further understand the training dynamics, we plot the log- prg-poobabilitis of the preferred and dispreferred responses from the preference data for all the variants in Figure 2. For the logistic loss, these plots demonstrate that while both the likelihoods rapidly decrease to zero, the rate is faster for the dispreferred responses. In terms of our analysis setup, this corresponds to ~ being large, which in turn leads to a small value of the curvature constant C₁ and a large value of R. This makes the bound in Theorem 3.6 drastically worse, confirming our theoretical findings empirically, In the case of μ Tee. corresponding DPO, a similar collapse of log probabilities was also observed in prior works (Fisch et al., 2024; Rafailov et al., 2024b). While the log-likelihoods also decline for the squared loss, the decrease is milder, which in turn means that the magnitude of wh and the necessary value of R remain adequately bounded. This, addition with the better curvature constant of the squared loss makes the findings to be consistent with the theory.
We evaluate the different variants in two ways. First we compare the policies generated by each method at regular intervals the training process, with the initial policy Tee by comparing the generated summaries in terms of their quality and conciseness by a prompted Gemini 1.0 Ultra model (Team et al., 2023). Figure I (left) shows that squared loss variants perform significantly better than those with logistic loss, which reach peak preference at 2k steps and suffer from a dramatic collapse afterwards. In the case of " = Tet. this collapse is consistent with prior findings on the DPO algorithm in the literature (Fisch et al., 2024; Rafailov et al., 2024a). In comparison, both squared loss variants reach peak performance after similar number of steps, but maintain that performance more stably afterwards.
In Table 2 we report approximations to C₁ and St for all the different loss variants, as measured empirically. We approximate R by the value of TT averaged over the last mini- bati-batchon of training. Further, we approximate eμ by the curvature of by for the values of R and Th. obtained this way. We note that -HT decreases monotonically during training and so the approximation to Ri and cp is tightest at the end of training. The reported values of c, and R are consistent with our theory and the empirical observations in Figuresl and 2.
Notably, using = गee is consistently better than = 1, when combined with the squared loss. In case of logistic loss, the comparison against Ter in Figure (left) suggests that the base policy choice does not affect performance. However, when we compare the policies produced by the two variants with μ and μ nee directly in Figure (left), we do observe strong impact. Initially, the uniform variant is slightly preferred with a strong preference after- wards in the other direction, after roughly 5K steps. How- ever, at this point the preference of each variants against Rie has already collapsed, indicating perhaps less meaning- ful comparison between two sets of bad responses in this region.
8

---

Offline Learning Preference-based RL
Winning lopprebbbiiity
Losing log probsbility
Loss:
Losde
logistic
Winiing lag prggebbbbyttttt
-20
squared
Base Polley:
Looling log probabiiitt
-25
-50
Dase Follyy
squared
Ref
Reet
uniforas
-75
uniffrm
-100
-50
-125
-150
-175
5000
30003 15000
20000
5000
10000 15000
20000
Training steps
Training steps
Figure 2. Evolution of the log-likelihoods of the preferred response (left) and dispreferred response (right) from the preference dataset across the training process. Both variants the squared loss decrease the log-likelihoods both the responses during training, but the decrcase is relatively mild. The logistic loss, the other hand, sends these log-likelihoods crashing sharply, even though the dispreferred responses have significantly lower values, so the difference of log- oogliielihoods remains highly negative, driving the loss to zero. We suspect that this degeneration of log-like le-likeiihhods is responsible for the eventual collapse observed for the logistic loss in Figure 1.
Loss
Base policy
R
C₁
((2) log
1 ref 73.203
0.00066
Teexp( B52
(z) log
72.949
0.00068
1+exp1- (113
((2) = (32 - 1)²
J = Nel
4.678
2.0
[() = (3z - 1)²
--=1
4.924
2.0
Table 2. We approximate R by " averaged over the last mini-batch and approximate eu by computing the curvature of ép at "i..
6. Discussion
References
This works adopts a different line of reasoning than the typical reparameterization arguments to motivate the cor- respondence between offline and online RLHF techniques, with a goal of understanding the impact of design choices in the offline methods, many of which do not fit cleanly in the arguments for equivalence of online and offline methods. Our theory and experiments collectively indicate that per- haps the limited coverage of offline data, and the propensity of log-likelihoone drop in certain methods are the key obstacles to reliable learning in this setup. Our findings suggest that using losses which do not decay to zero at a slow rate, like the logis- tic loss, and using experimental design techniques for data collection before offline RLHF can be fruitful avenues for addressing the concerns uncovered here.
Ahmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., and Hooker, S. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint 240.::2402.147400. 2024.
Amini, A., Vieira, T., and Cotterell, R. Direct pref- erence optimization with an offset. arXiv preprint 240. 2024.
of the preference data to precipitously
Azar, M. G., Guo, Z. D.. Piot. B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Bradley, R. A. and Terry, M. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-344 1952.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Cen, S., Mei, J.,, Goshvadi, K., Dai, H., Yang. T., Yang, S., Schuurmans, D., Chi, Y., and Dai, B. inteetiicnt preference optimization: A unified approach to online and offline rlhf. arXiv preprint arXiv:2405. 19320, 2024.
Chen, Y., Tan, J., Zhang, A., Yang, Z., Sheng, L., Zhang, E., Wang, X., and Chua, T.-S. On softmax direct prefer- ence optimization for recommendation. arXiv preprint arXiv:2406.0922 2024.
9

---

Offline Learning Preference-based RL
Christiano, P. F,, Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
Park, R., Rafailov, R., Ermon, S., and Finn, C. Disentan- gling length from quality in direct preference optimiza- tion. arXiv preprint arXiv:2403. 19159, 2024.
Rafailov, R., Chittepu, Y., Park, R., Sikchi, H., Hejna, J., Knox, B., Finn, C., and Niekum, S. Scaling laws for reward model overoptimization in direct alignment algo- rithms. arXiv preprint arXiv:2406.0222 02900, 2024a.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kicla, D. KTO: Model aligmment as prospect theoretic optimization. arXiv preprint arXiv:2402.01.30 2024.
Fisch, A., Eisenstein, J., Zayats, V., Agarwal, A., Beirami, A., Nagpal, C.,, Shaw, P., and Berant, J. Robust preference optimization through reward model distillation. arXiv preprint arXiv:2405. 0e0224055.93311 19316, 2024.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024b.
Huang, A., Zhan, W., Xie, T., Lee, J. D., Sun, W., Krish- namurthy, A., and Foster, D. Correcting the mythos of ke-regularizatton: Direct alignment without overopti- mization via chi-squared preference optimization. arXiv preprint arXiv:2407. 13399, 2024.
Raffel, C.,, Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, Exploring the limits of transfer learning with unified text-to-text transformer. Jourmal of machine learning research, 21 (140):1-67, 2020.
Jiang, N. and Xie, T. Offline reinforcement learning in large state spaces: Algorithms and guarantees. Statistical Science, 2024.
Ramesh, S. S., Hu, Y., Chaimalas, Mchta, V., Sessa, P. G., Bou Ammar, H., and Bogunovic, I. Group robust preference optimization in reward-free rlhf. Advances in Neural Information Processing Systems, 37:37100- 37137, 2024.
Liu, Z., Lu, M., Zhang, S., Liu, B., Guo, H., Yang, Y., Blanchet, J., and Wang, Z. Provably mitigating overopti- mization in rlhf: Your sft loss is implicitly adversarial regularizer. arXiv preprint arXiv:2405. 2024.
Raskutti, G., Wainwright, M. and Yu, B. Early stop- ping and con-parametric regression: optimal data- dependent stopping rule. The Journal of Machine Learn- ing Research, 15(1)3335-366, 2014.
Luce, R. Individual Choice Behavior: A Theoretical Analy- sis. Dover Books on Mathematics. Dover Publications, 2012. ISBN 9780486153391, URL https: h/ttttss:/bbos..... google, com
Reid, M. D. and Williamson, R. C. Composite binary losses. The Journal of Machine Learning Research, 11:2387- 2422, 2010.
Meng, Y., Xia, M., and Chen, D. SimPO: Simple preference optimization with reference-free reward. arXiv preprinl arXiv:2405.17734, 2024.
Sonthalia, R., Lok, J., and Rebrova, On regularization via early stopping for least squares regression, arXiv preprint arXiv:2406.0444 04425, 2024.
Munos, R., Valko, M., Calandriello, D., Azar, M. G., Row- land, M., Guo, Z. D., Tang, Y., Geist, M., Mesnard, T., Nash learning from human feedback. arXiv preprint arXiv:23122 23/.:223 2023.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. Learning to summarize with human feedback. Ad- vances in Neural Information Processing Systems, 33: 3008-3021, 2020.
Michi, A.,
Neu, G. and Rosasco, L. Iterate averaging as regulariza- tion for stochastic gradient descent. In Conference On Learning Theory, pp. 3222-3242. PMLR, 2018
Suggala, A., Prasad, A., and Ravikumar, Connect- ing optimization and regularization paths. Advances in Neural Information Processing Systems, 31, 2018.
Ouyang. L., WU, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang. C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 15:27730-277444 2022.
Swamy, G., Dann, C., Kidambi, R., Wu, Z. S., and Agarwal, minimaximalist approach to reinforcement learning from human feedback. arXiv preprint arXiv:2401.0400 04056, 2024.
Pal, A., Karkhanis, D., Dooley, S., Roberts, M.. Naidu, S., and White, C. Smaug: Fixing failure modes of pref- erence optimisation with dpo-positive. arXiv preprint arXiv:2402. / 13228, 2024.
Tang, Y., Guo, Z. D., Zheng, Z., Calandriello, D., Munos, R., Rowland, M., Richemond, P. H., Valko, M., Pires, A., and Piot, B. Generalized preference optimization:
10

---

Offline Learning Preference-based RL
A unified approach offline alignment. In Forty-first International Conference Machine Learning, 2024.
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., al. Gemini: a family of highly capable multimodal models.
arXiv preprint ar.iv:2312. /805, 2023.
Vaskevicius, T., Kanade, V., and Rebeschini, Implicit regularization for optimal sparse recovery. Advances in Neural Information Processing Systems, 32, 2019.
Völske, M., Potthast, M., Syed, S., and Stein, B. TI; dr: Min- ing reddit to learn automatic summarization. In Proceed- ings of the Workshop on New Frontiers Summarization, pp. 59-63, 2017.
Wang. Y., Liu, Q., and Jin, C. Is rlhf more difficult than standard r!? arXiv preprint arXiv:2306. 14/11, 20233 2023.
Xie, T., Cheng, C.-A., Jiang, N., Mineiro, P., and Agarwal, A. Bellman-consister pessimism for offline reinforce- ment learning. Advances in neural information process- ing systems, 34:6683 66944, 2021.
Xiong, W. Dong, H., Ye, C., Wang, Z., Zhong, H., Ji, H., Jiang, N., and Zhang, T. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. arXiv preprini arXiv:2312. 1456, 2023.
Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L.., Van Durme, B., Murray, K., and Kim, Y. J. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation. arXiv preprint arXiv:2401.055 2024.
Yao, Y., Rosasco, I.,, and Caponnetto, A. On early stopping in gradient descent learning. Constructive Approximation, 26(2):289-311 -15, 2007.
Zhao, H., Ye, C., Gu, Q., and Zhang, T. Sharp analysis for kl-regularized contextual bandits and rlhf. arXiv preprint arXiv:441.004625 2024.
Zhao, Y., Khalman, M., Joshi, R., Narayan, S., Saleh, M. and Liu, J. Calibrating sequence likelihood im- proves conditional language generation. arXiv preprint arXiv:2210.00045. 2022.
Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, J. SLiC-HF: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023.
Zhu, B., Jordan, M., and Jiao, J. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning, pp. 43037-43067. PMLR, 2023.

---

Offline Learning Preference-based RL
A. From preference model to proper loss
natural family for the reward generating distributions is the exponential family, particular we are going to assume that D., is in the exponential family parametrized by some E* := (" y, 3/), that is P(w|z, y, y f/)= = exp(wv" d(v")), where is some strictly convex function. Considering the exponential family naturally leads an objective function for learning the unknown parameter v, that is to maximize the log- oogeliidl which will recover ut in the following way. First take the derivative of the log-likelihood with respect
V Eu [lo exp(we <(v))] = Eu| - Vo(u)] Vo(u*) Vo(w))
Setting the derivative and using the strict convexity to invert VΦ((v) shows that V W)). summarize the above, preference model which follows exponential family distribution gives rise to natural loss function given by
min <E 99.9
(o) we),
and has closed form solution v* VØ (Ew (w]).
To make this discussion concrete we focus on a BTL model, where we set v( v.v') R(x,v) R(x, y'). Then - R(x, y))) and the corresponding loss function is then ((o(2.M.1/) wo(x, v..,.,,3/ log(exp( (R(x,)) R(x, 1/) exp( -(x,)))))))
6(v(,, v')) = log(exp(R((z,)) - R( v/) +
log(exp(u(R(z.)) 2(xx./))))
log(1 + cxp(-w(((x.)))) R(a,y')))),
which is precisely the logistic loss. The above derivation already shows that any loss derived from the exponential family with link function V is going to be proper as e is strictly convex and hence Vø is invertable with ge in Assumption 3.3 satisfying Je = v We can further check that for a minimizer, v, of the logistic loss we must have
V. (1) log(1 + exp(-v)) (1 17) log(1 1 + exp(x))) nexp(-v) (1 - 1) exp(v) 0, exp(-u)
1 exp(u)
where P(w = 1|, y. v.). The above implies = log T-my' Equivalently, we could have computed the derivative of the convex conjugate of o. Vø", which is precisely V Finally, establish the claimed connection between the parametrization of TI to the BTL parametrization with v* we have the following. The fact that the logistic loss is proper implies
R(x, u) R(x, v/) - log m(ylz)) wn''(,,,.
1(''(z))
exp (t" (I, v) 12" (x, log 7(x) RR* (x. y) - R (x,,))))
= log 1-
which further simplifying gives Equation 3.
similar line of reasoning shows that when {(v) = (1 - -) the link function satisfies r 1to"
and so the resulting
2
reward model is P(w|z, y, v/) 1+R' (2. R" (2,8/)
B. Approximate realizability
We now relax Assumption 3.2 to the following. Assumption B.1 (Approximate Realizability). There exists nr Il such that for all X, V.v/ eV:
Be_TE(w "
min vE[-
[fw (l),,,,)
To recover a version of Theorem 3.6 we need to modify Lemma 4.1 and its proof as that the only place where realizability utilized. We will also need to use the stronger version of Assumption 3.5.
12

---

Offline Learning Preference-based RL
Assumption B.2. For any u, u E [- R). we have
f(u) 2 ((v)) +())+ (u- u) + Gp (u -v)
2
Assumption B.2 posits that E is -trSnnyy convex.
Lemma B.3. Under Assumptions B.1 and B.2, any policy E Il with (T; Dayw) (T=; Dx Dayyw S satisfies
B AR(x, [)
B...n''-(AAR( - AR(a, v/) 4(26' +c)
Cp.
Proof. Using the strong convexity part of Assumption B.2 we have that for π and n
E (w(R(z,)) v) - R(x, v² )) S Ele (ww " (x. 1)] - Ele (wn*)]
cu
2
T (T;; Dayuw) [ Dayw) B[e(w nn "(I, (x,,,,))) ([()w)
<e +
Cu (w((1t (I,, v) 12" (xx11/ x²)) 2
E
El (ww HH (x. U. v'))] E(e (ww*)] s
where o* argmin ED_ [((w-v) v) Jr, y, n'] and we have used first order optimality, together with the strong convexity of l. Using the above two inequalities we have
cu B((([[(r,)) v) - R(x, - R² (2" v) + R (x,2/))))
4
c B[[[([(E v) A(x,8') v² + +0² u Rd (z, v) + R (x.
4
s 5 2 (w (R((, v) - A(x,8') uz) )²)) + 5 c
(w (x.
(z. v) - R (x,))))v1))} s 2 te.
2
The remainder of the proof of Theorem 3.6 remains unchanged except for replacing e by 2 + And so we have Theorem B.4. For any π E II such that (T; Dz Lp(TTi Dzyw) S where the corresponding loss to Ly given by é is proper, and under Assumption 3.1, Assumption 3.4, Assumption B. and Assumption B.2, it holds that
(c+ (c+26))
C(c+
αt C(c+2e')
Eg KKL (+)))) <
Cp
2
Cμ
C. Experiment details
We evaluate the different variants on the TL:DR dataset (Völske et al., 2017), where the task is to summarize posts on reddit forums. The dataset consists of an original reddit posts, along with pair of responses which are rated by human judges to provide the groumd-truth preference annotations (Stienmon et al., 2020). Our experiments use a T5 large model (Raffel et al., 2020) with 770M parameters, which is further fine-tuned to maximize the log-likelihood of the winning responses in the TL:DR dataset. We train for 20000 iterations, with a batch size of 32. A KL regularizer is used to the reference Net checkpoint with coefficient equal to 0.005. The optimizer used is Adafactor with learning rate that is constant with linear warm-up for 2000 steps and a base rate of le 4.
We used the following text to prompt the Gemini evaluator used for our experiments:
You are an expert summary rater who prefers very short and high quality summaries. Given a document and two candidate summaries, say 1 if SUMMARY the better summary, or if SUMMARY2 is the better summary. Give a short
13

---

Offline Learning Preference-based RL
reasoning for your answer.
ARTICLE: <article-here
SUMMARY1: <summary-by-n>
SUMMARY2: <surnary-by-mwp>
14


Reviews:
[{"review_id": "KOqSRasf6J", "reviewer": "Reviewer_ShLH", "summary": "The paper investigates offline reinforcement learning methods that use a fixed dataset of responses and human preference feedback to align language models. It analyzes how various design choices in methods like DPO, IPO, and SLiC affect the quality of the learned policy. The study provides a unified theoretical framework that bypasses traditional reparameterization arguments and includes an empirical validation on a standard summarization benchmark.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the encouraging feedback. We will add the suggested references [1, 2] to our discussion.", "claims_and_evidence": "The claims about bound on the sub-optimality and other theoretical insights are well supported.", "methods_and_evaluation_criteria": "The benchmark DPO methods are appropriate and diverse enough. Multiple loss functions are also studied. Given the loss function and the data coverage assumption, the main bounds are derived.", "theoretical_claims": "Section 4 analysis contains accurate derivation of the proof of Theorem 3.6 which is intuitive and accurate.", "experimental_designs_or_analyses": "The experimental results might be limited in terms of various datasets and backbone models. However, the major ablation sufficiently demonstrates the theoretical point regarding the assumptions.", "relation_to_broader_scientific_literature": "Related to major DPO methods [1,2] where this paper focuses on various design choices and how they affect the learning of an optimal policy.\n\n[1] Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024b.\n\n[2] Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447–4455. PMLR, 2024.", "essential_references_not_discussed": "Multi-negative DPO [1] could be further discussed. In addition, group-based regularization [2] could also be discussed.\n\n[1] Chen, Yuxin, Junfei Tan, An Zhang, Zhengyi Yang, Leheng Sheng, Enzhi Zhang, Xiang Wang, and Tat-Seng Chua. \"On softmax direct preference optimization for recommendation.\" arXiv preprint arXiv:2406.09215 (2024).\n\n[2] Ramesh, Shyam Sundhar, Yifan Hu, Iason Chaimalas, Viraj Mehta, Pier Giuseppe Sessa, Haitham Bou Ammar, and Ilija Bogunovic. \"Group robust preference optimization in reward-free rlhf.\" Advances in Neural Information Processing Systems 37 (2024): 37100-37137.", "other_strengths_and_weaknesses": "The theoretical bound of sub-optimality is intuitive and useful. \n\nEmpirically, the setting could use more ablation study including various backbone models, datasets, etc.", "other_comments_or_suggestions": "See Strengths and Weaknesses", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "rm5Am502Z3", "reviewer": "Reviewer_L7N1", "summary": "The paper offers a theoretical analysis of offline preference-based RL algorithms such as DPO. Specifically, the authors investigate an empirical observation: offline preference-based RL is often worse than online RL, and faces some degeneracies during optimization.\n\nTo set up the problem, the authors first set up a general policy class to include different regularization methods and then make 5 assumptions: bounded log prob and loss; realizability ;proper loss; coverage and curvature.\n\nThe main conclusion is Thm 3.6, which is an error bound decided by $c_\\mu$, $C$ and $\\epsilon$. As a result, losses with a high curvature will have a larger $c_\\mu$, therefore better error bounds. Moreover, base policy and constraints also play a role: different base policies may change the $\\pi^*$, and different constraints make different assumptions on $\\pi^*$. Lastly, the  Authors show that DPO may have large R and small $c_\\mu$, causing degeneracies observed in previous works.\n\nExperiments show that square loss is superior to logistic loss and $\\pi_{ref}$ performs better than uniform distribution.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your encouraging and thoughtful comments. \n\n**Severity of restrictions due to Assumption 3.5**: You are correct in noting that Assumption 3.5 places some restrictions on the loss function, but we believe that these restrictions are relatively mild and standard as we discuss below:\n\n1. Please note that the curvature condition is not pointwise (i.e. per x, y, y’ triple), but only in expectation, and not uniform, but only local around the optimal policy $\\pi^\\star$. So the required assumption is local curvature of the expected loss around the optimal solution, which is a very common assumption in optimization and convergence theory.\n2. The curvature assumption, while not explicitly stated, is responsible for the e^B term in almost all prior works on analysis within the Bradley-Terry-Luce model with the logistic loss, such as [1], [2] and [3]. Please also see Remark 4.9 in [1] for an explicit mention of this aspect. In fact, even in simpler settings such as analysis of generalized linear models, a similar curvature assumption can be seen in [4], Assumption 1.\n\nWhile we agree that weakening such conditions further would be highly desirable, addressing this goes well beyond the study of preference-based learning methods, and would be a significant undertaking to be carried out in a separate work. \n\n**Differences to analysis in IPO paper**: We apologize for missing the discussion of differences from the analysis in the IPO paper in the submission. The authors in IPO indeed note the issue of the curvature in the link function, and use this to motivate the IPO algorithm. However, there is no explicit quantitative analysis of the role of the link function beyond a simple example, since the focus is more on obtaining a practical algorithm for the identity link function case. We instead derive precise convergence guarantees, and also highlight the role of offline data coverage which is not captured in the IPO paper. \n\nWe thank you again for the insightful comments, and we will update the final submission with the discussions above, as well as address your other suggestions.\n\nReferences:\n* [1] Sharp Analysis for KL-Regularized Contextual Bandits and RLHF, arXiv 2411.04625.\n* [2] Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons, ICML 2023.\n* [3] Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint, ICML 2024.\n* [4] Parametric Bandits: The Generalized Linear Case, NeurIPS 2010.", "claims_and_evidence": "### Square loss is better than logistic loss\nEvidence:\n* The $c_\\mu$ of square loss is ½, yielding a reasonable bound\n* The experiment shows that square loss is better than logistic loss\n\t\nQuestion:\n* The $c_\\mu$ should be 2 instead of ½?\n\n### Base policy matters, and $\\pi_{ref}$ is a relative good choices\nEvidence:\n* Jointly supported by Section 3.1 and experiment\n\n### Different constraints play a role in the $\\pi^*$ assumption, which may affect results\nEvidence:\n* Remark 3.9.\nQuestion:\n* Is it possible to draw a plot of CPO to justify this assumption?\n\n### DPO faces degeneracies because of large R and low $c_\\mu$\nEvidence:\n* Remark 3.10", "methods_and_evaluation_criteria": "The methods and evaluation make sense.", "theoretical_claims": "Strengths:\n* I have a rough glance at Section 4 and do not find obvious mistakes.\n\nWeakness:\n* The Assumption 3.5 may be too strong and prevent analysis on many possible losses, for example, sigmoid.", "experimental_designs_or_analyses": "Strengths:\n* Experiments and analyses make sense. \n\nWeakness:\n* As I mentioned earlier in “Claims and Evidence”, it would be better to add experiment such as CPO to justify the claim on constraints.", "relation_to_broader_scientific_literature": "Strengths:\n* Different from other papers such as GPO, which gives empirical analysis, this paper gives a unified theoretical analysis and understanding on the offline preference-based RL.\n\nWeakness:\n* The assumption 3.5 restricts the theoretical analysis to a limited class of losses. Specifically, the authors analyze square loss and show its superiority to logistic loss. However, similar analyses and conclusions have been found in prior works such as IPO.", "essential_references_not_discussed": "It may be beneficial to discuss the difference between IPO’s theory analysis and this paper’s.", "other_strengths_and_weaknesses": "Other Strengths\n* It is good to see that authors report error bars in the experiments, which should be encouraged\n* I personally think the paper is clearly written overall.\n\nOther Weakness\n* See above", "other_comments_or_suggestions": "* Line 366 right side: “consistently” -> “consistently better”\n* Line 373 right side: “Figure 1(left)” -> “Figure 1 (right)”\n\n## update after rebuttal\n\nI keep my score.", "questions_for_authors": "See above. My main concern is that Assumption 3.5 seems a bit strong to prevent analysis on other losses, and I will be happy to raise my score if authors can address these concerns.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3V1SdVB0Ap", "reviewer": "Reviewer_GPNE", "summary": "This paper explores the theoretical aspects of offline preference-based reinforcement learning (PBRL). It examines a broad range of offline PBRL methods, including DPO and IPO, and establishes theoretical bounds on the sub-optimality of the policies learned by these methods. The analysis is based on specific assumptions regarding the loss functions and base policies, with the derived bounds also depending on certain quantities determined by the loss functions. Consequently, the theoretical findings offer some insights into selecting appropriate loss functions and base policies. Lastly, the paper presents experiments to validate the theoretical results.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thoughtful feedback on our work. You are right in noting that the main contributions of our work are theoretical, and not in proposing new methods. We address your main concerns with our theoretical results below.\n\n**Squared loss and realizability**:. You are correct that it is not the case that the squared loss is a clear win across all the theoretical criteria, and makes a less natural realizability assumption. We note that we can use standard arguments to extend our analysis to the case where the realizability assumption holds uniformly, up to an error $\\epsilon$. We did not include this for clarity of exposition in the submission, but can easily add this to the final version. More generally, our view is that realizability is an unverifiable assumption in practice, but is perhaps more likely to hold for rich model classes such as LLMs. At the same time, the curvature of the loss is something we can control better. So an approach that primarily favors the squared loss as a default objective based on the curvature properties is reasonable practically, since it will yield good results, unless realizability is violated, in which case a practitioner can switch to other losses, such as logistic. This tradeoff between the intuitive modeling niceness versus optimization and learning tractability is often encountered in the literature on generalized linear models as well, and we will expand on this discussion in the final submission.\n\n**Choice of loss function affects benchmark policy**: You are also correct that the choice of the base policy affects the benchmark policy. As we note on line 161 (right), the most natural benchmark, which would be independent of these choices, cannot be recovered by the class of offline methods including DPO and variants which are studied here. That said, the base policy is a part of the loss function. It is quite standard even in the literature on simpler classification and regression setups that when the loss changes, the convergence point changes accordingly, and we can only analyze convergence to the optimal solution according to the chosen loss. Indeed, as we notice in Remark 3.8, the choice of base policy has no explicit role in the convergence bound, but only plays a more subtle role in influencing the realizability assumption and the benchmark policy. We think it is still useful to uncover the different points of convergence that different choices used in practice correspond to, so that we can prefer methods which correspond to desirable points of convergence.\n\n**Gap between the performance measures in theory and experiments**: We agree that it would have been great if we could just have a similar KL bound or an easy surrogate in our experiments as well. Unfortunately, without access to the unknown benchmark policy, this is an impossible task. Consequently, we measure the most natural quantity that we could think of empirically, and that is also consistent with the broader literature. While not perfect, we believe this is still insightful. If the reviewer has additional recommendations for performance measures that are measurable, and that would narrow the theory-practice gap, we would be happy to include them in the final submission.\n\n**Notation**: We apologize for the inconsistency regarding the notation for loss, we will fix it. The notation $\\Pi_{\\mathcal{R}}$ is a typo from an earlier version. It should just be $\\Pi$.\n\nThanks again for your insightful comments. We hope that our responses address some of your concerns, and we would be happy to discuss further if you have additional feedback.", "claims_and_evidence": "## \n\nSome of the key claims made in this paper are not well supported by clear and convincing evidence.\n\n1. The paper claims that its theoretical results provide insights into design choices such as the selection of loss functions and base policies for offline RLHF. However, the provided evidence does not fully support this claim. Regarding the loss function, Remark 3.7 suggests that the squared loss is preferable due to its curvature properties but fails to satisfy the realizability assumption. However, the paper does not extend its theoretical analysis to scenarios where this assumption is violated. As a result, it remains unclear when one should choose a particular loss function in practice. Similarly, regarding the base policy, Remark 3.8 states that its selection affects the benchmark policy. This implies that different variants with distinct base policies are not evaluated against a common standard, reducing the meaningfulness of theoretical comparisons.\n2. Additionally, there is a significant gap between the theoretical analysis and experimental validation. The primary discrepancy lies in the performance metrics used. The theoretical results rely on KL divergence with respect to the benchmark policy, whereas the experiments evaluate performance using evaluation preference and the log probability of winning or losing. Since the theoretical and empirical evaluations employ fundamentally different measures, the experimental results do not convincingly support the proposed theory.", "methods_and_evaluation_criteria": "This paper primarily analyzes existing methods rather than proposing a new one. However, it introduces a benchmark policy specifically for the theoretical analysis of offline RLHF methods. This benchmark policy is defined as the policy that attains the pointwise minimum of the loss function. Under the Bradley-Terry (BT) model assumption, this benchmark policy coincides with the standard KL-regularized reward-maximizing policy. Given this alignment, I find the proposed benchmark policy to be a reasonable choice for studying offline RLHF.", "theoretical_claims": "I only checked the proof sketch in Section 4 and did not check the detailed proof. I feel the results are correct.", "experimental_designs_or_analyses": "Please see my comments in the above Claims And Evidence part.", "relation_to_broader_scientific_literature": "This paper studies offline RLHF for LLM alignment and may inspire advancements toward training better LLMs.", "essential_references_not_discussed": "To the best of my knowledge, this paper provides a sufficiently thorough discussion of all closely related works.", "other_strengths_and_weaknesses": "Please see my comments in the above Claims And Evidence part.", "other_comments_or_suggestions": "1. The notation $\\ell$ is used inconsistently. Before line 255, the loss function $\\ell$ is defined as taking a single variable as input. However, in line 255, $\\ell$ is instead expressed as a function of two variables. This inconsistency may cause confusion and should be clarified.", "questions_for_authors": "1. What is the meaning of the notation $\\Pi_{\\mathcal{R}}$ under Assumption 3.1?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WFYyvSWExq", "reviewer": "Reviewer_8if4", "summary": "The paper provides a theoretical study of offline learning methods from human preferences. The authors first establish a unified framework and relevant assumptions which fit most all of preference-based learning losses. Then, they propose a policy benchmark which is used to measure the quality of a policy output from optimizing empirical losses on preference data. Theorem 3.6 provides bounds on the closeness of these two policies. The findings suggest that using the squared loss is the best choice in that its optimizer is the closest to the benchmark \"ideal\" policy among other losses. These findings are also supported by some simple experimental results on language models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your encouraging feedback on our work. We address your key questions below.\n\n**1. Relaxing Assumption 3.1**: You are correct in noting that having an exponential dependence on the size of log-ratios is not ideal. But this limitation is shared by a line of prior works [1], [2], [3]. While it would be desirable to weaken this dependence, we also note that the degradation of learning quality as the log probability ratios grow large during learning is quite clear in our experiments. As we see in Figures 1 and 2, the drop of log probabilities under the learned policy coincides with degradation in the performance of DPO. Since the reference policy has well-behaved log probabilities, the probability ratios look nearly identical to the learned policy’s log probabilities here. So we also think that the bounds capture a phenomenon actually realized in the experiments qualitatively, even though we might be able to improve the precise functional forms from being milder than exponential in future work.\n\n**2. Generating model underlying squared loss**: We thank you for the insightful question about the generating model underlying the squared loss. We refer you to Appendix A, which provides this discussion for general link functions, as well as gives the specific preference generation model for squared loss in lines 591-592: $P(\\omega=1|x, y, y’) = \\frac{1}{2} + \\frac{R^\\star(x, y) - R^\\star(x, y’)}{2}$.\n\nWe thank you again for your thoughtful feedback, and will add these discussions to the final version of the paper.", "claims_and_evidence": "The claims made in the submission are supported by rigorous theoretical proof and experimental results.", "methods_and_evaluation_criteria": "The proposed models, dataset and losses are all relevant to the considered setting.", "theoretical_claims": "I checked the correctness of the proof of Theorem 3.6 and, to the best of my understanding, it is clear and concise.", "experimental_designs_or_analyses": "I did not replicate the experimental results of the paper.", "relation_to_broader_scientific_literature": "The key contribution of the paper is the theoretical insight of the benefits of using squared losses when directly optimizing over a given preference dataset. These findings are definitely relevant to the literature of learning from human preferences.", "essential_references_not_discussed": "There are no essential references missing, although the paper could benefit from a more comprehensive related work section in the broader theoretical RLHF literature (as opposed to the more narrow offline PbRL literature which is mentioned).", "other_strengths_and_weaknesses": "The paper is clear, concise and easy to follow. The added contribution is  indeed original as it unifies previously proposed loss function by considering the necessary assumptions which all of them seem to satisfy, and providing a general upper bound that depends on parameters of interest. The implications of Theorem 3.6 point out the seeming superiority of IPO in terms of producing a policy that is closer to the benchmark policy. This makes for an interesting finding.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. I understand that in practice, the reference policies and the policies to be optimized are often non-degenerate. However, they can have negligible entries in the sense that $\\mu(y|x) \\approx 0$ or $\\pi(y|x) \\approx 0$ for some $x,y$. That makes the bounds on $\\log\\pi(y|x)$ or $\\log\\pi(y|x) / \\log\\mu(y|x)$ extremely large, especially when you consider it over the data generating distribution, not only the data itself. Moreover, the presence of such a bound as an exponent in the upper bounds on Theorem 3.6 seems concerning for such border cases. Can Assumption 3.1 be relaxed? If it doesn't hold, is there a way to get meaningful bounds?\n\n2. Although the squared loss seems to be superior to the other losses under the assumptions made, it is not particularly clear how realistic it is. Specifically, the logisitc loss, for instance, is not an arbitrary choice, but results from the synthesis of the RLHF pipeline into one step. Thus there is an inherent intuition behind its definition. Is there a practical intuition behind the squared loss? Under what preference generation model, if any, does it make sense?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a unified theoretical framework for RLHF algorithms. It provides several interesting and intuitive theoretical insights (e.g., square vs. logistic loss, early stopping, etc.) and validates them in experiments. Although this paper is concisely written, it is quite informative and brings clarity to the community. Therefore, I would recommend acceptance for this paper.
Decision: Accept (poster)