The Importance of Online Data: Understanding Preference Fine-tuning via Coverage
Authors: Yuda Song
Gokul Swamy
Aarti Singh
Drew Bagnell
Wen Sun
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning from Human Feedback
LLM finetuning
Primary area: reinforcement_learning

Abstract: Learning from human preference data has emerged as the dominant paradigm for fine-tuning large language models (LLMs). The two most common families of techniques -- online reinforcement learning (RL) such as Proximal Policy Optimization (PPO) and offline contrastive methods such as Direct Preference Optimization (DPO) -- were positioned as equivalent in prior work due to the fact that both have to start from the same offline preference dataset. To further expand our theoretical understanding of the similarities and differences between online and offline techniques for preference fine-tuning, we conduct a rigorous analysis through the lens of *dataset coverage*, a concept that captures how the training data covers the test distribution and is widely used in RL. We prove that a global coverage condition is both necessary and sufficient for offline contrastive methods to converge to the optimal policy, but a weaker partial coverage condition suffices for online RL methods. This separation provides one explanation of why online RL methods can perform better than offline methods, especially when the offline preference data is not diverse enough. Finally, motivated by our preceding theoretical observations, we derive a hybrid preference optimization (HyPO) algorithm that uses offline data for contrastive-based preference optimization and online unlabeled data for KL regularization. Theoretically and empirically, we demonstrate that HyPO is more performant than its pure offline counterpart DPO, while still preserving its computation and memory efficiency.

The Importance of Online Data: Understanding Preference Fine-tuning via Coverage
Yuda Song Carnegie Mellon University yudasêcs. cmu. edu
Gokul Swamy
Aarti Singh
Carnegie Mellon University gswamy@cs. cmu. edu
Carnegie Mellon University aarti@cs, cmw.cmu....oomo edu
J. Andrew Bagnell
Wen Sun
Aurora Innovation, Carnegie Mellon University dbagnell@aurora.ter tech
Comell University ws455@corne11.co edu
Abstract
Learning from human preference data has emerged as the dominant paradigm for fine-tuning large language models (LLMs). The two most common families of techniques online reinforcement learning (RL) such Optimization (PPO) and offline contrastive methods such as Direct Preference positioned equivalent in prior work due to the
Proximal Policy
Optimization (DPO) fact that both have to start from the same offline preference dataset. To further expand theoretical understanding of the similarities and differences between online and offline techniques for preference fine-tuning, we conduct a rigorous analysis through the lens of dalaset coverage, concept that captures how the training data covers the test distribution and is widely used in RL. We prove that global coverage condition both necessary and sufficient for offline contrastive methods to converge to the optimal policy, but a weaker partial coverage condition suffices for online RL methods. This separation provides one explanation of why online RL methods can perform better than offline methods, especially when the offline preference data is not diverse enough. Finally, motivated by our preceding theoretical observations, we derive hybrid preference optimization (HyPO) algorithm that uses offline data for contrastive-based preference optimization and online unlabeled data for KI. regularization. Theoretically and empirically, we demonstrate that HyPO is more performant than pure offline counterpart DPO, while still preserving its computation and memory efficiency.
Introduction
Due to the difficulty of manually specifying reward functions for complex tasks [7], preference- learning has emerged as critical component in the fine-tuning procedure for large language models (LLMs) [40, 30, 45, 44]. There are two predominant flavors of preference learning for LLMs: online reinforcement learning (RL) methods such PPO [11, 30] and offline contrastive methods like Direct Preference Optimization (DPO) [33] and Identity Preference Optimization (IPO) [3].
Online RL methods usually follow the two-stage procedure prescribed in [30]: one first trains reward model (classifier) on fixed offline preference dataset before using it to provide reward labels for on-policy generations, which are then fed to downstream RL algorithm like Proximal Policy Optimization (PPO) [36]. Since the reward model learned from static offline preference data, to avoid over-optimizing the reward model [17], typically adds reverse KL penalty to encourage the model to stay close to some reference policy. We will refer to this procedure as reinforcement learning from human feedback (RLHF) in this paper. While empirically performant, RLHF requires repeated querying of the reward model (which is often itself an LLM) as well as sampling from the current policy. In response the computational expense and relatively complex nature of this
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

procedure, purely offline methods like DPO [33] and IPO [3] have been proposed as alternative methods for preference fine-tuning. These methods do not need to fit separate reward models, instead opting to simply train the policy directly on the offline preference dataset via a ranking loss.
Offline contrastive methods like DPO are usually derived via applying reparameterization trick to the closed-form solution of the minimum relative entropy problem [63] that RLHF techniques attempt to approximate. Thus, several authors have described these methods as equivalent (at least in theory) to the standard RLHF procedure [33, 3]. However, recent (mostly empirical) work has contradicted this perspective: [43] find that online methods out out-perform offline methods and attribute this fundamentally to on on-policy sampling, [55] argues that the online RL methods produce an often desirable subset of the possible DPO loss minimizers, and [42] provide empirical support for the claim that online and contrastive training provide orthogonal benefits. However, a rigorous theoretical separation still lacking the pre-existing literature, which motivates our key questions:
What is the statistical separation between the online RLHF method and offline contrastive methods? What causes this separation and what does it imply?
To answer these questions, we focus on the coverage of the preference dataset, a key concept that widely used in RL [22, 28, 58] for analyzing the impact of offline or exploratory data distributions. Through the lens of coverage of the offline preference dataset, we make the following contributions:
We prove that the global coverage condition [28]. the strongest possible coverage condition in RL, is necessary for offline contrastive algorithms like DPO to converge to the optimal policy. In contrast, we identify a weaker local coverage condition that is sufficient for online RLHF algorithms, thus provably separating the two types of algorithms. The separation is due to the difference in reward modeling and on/offline regularization - in short, there is no free lunch from bypassing explicit reward learning and online rollouts. As global coverage might sometimes be violated in practice, our separation result can perhaps explain why RLHF works better than offline methods [42, 43, 56).
Although offline contrastive methods are derived from reverse- KL. objective, we prove that the policies trained via offline methods can still have infinite reverse-KL in the partial coverage setting. In contrast, we show that RLHF can always control the reverse KL via directly optimizing reverse KL using online samples. This means that on realistic problems, RLHF has stronger guarantees for remaining close to the reference policy than offline contrastive methods.
We propose Hybrid Preference Optimization (HyPO) to address the deficiencies of offline contrastive methods while maintaining some of their computational simplicity. HyPO a hybrid RL algorithm [51, 39] where offline data is used for the DPO objective while online samples are used to explicitly control the reverse KI. divergence to the reference policy. We empirically demonstrate that HyPO outperforms DPO, on the TI.DR summarization task [40] on all metrics including both the GPT4 win-rate and the reverse KL. divergence to the reference policy, and on general chat benchmarks such as AlpacaEvall 2.0 [15], trained with the UltraFeedback dataset [14]. In addition, HyPO also mitigates the overfitting issues observed in the offline constrastive based methods [43].
We provide an explanation of why RLHF and offline contrastive methods decrease the probability of both preferred and rejected responses during training. In particular, under our function approximation baase global coverage condition, show that such behavior is actually desirable for DPO and RLHF policies to extrapolate and generalize to optimal actions that do not appear in the training dataset. However, without function approximation, algorithms like DPO can mistakenly increase the likelihood of sub-optimal actions. This establishes the importance of function approximation for the success of the algorithms such DPO.
Taken together, our results establish the critical role coverage plays in terms of convergence properties of preference learning algorithms as well as in the design of new, performant empirical approaches.
2 Related Work
Preference Fine-Tuning (PFT). As discussed in the introduction of our work, there are two major paradigms for preference fine-tuning of LLMs. The first one, online RL methods [30], proposes to first train reward model (classifier) to predict human preferences, followed by running an RL method to optimize this learned reward function. While PPO [36] is the most popular RL algorithm used in the online RLHF framework by far [40, 30, 45), more recent work by [1] shows that simpler online RL algorithms like REINFORCE [48] also work well. The second class of methods, offline contrastive techniques [60. 33, 3), avoid explicit reward modeling and directly optimize their objective the offline preference dataset. Recently there are hybrid methods that combine offline
2

---

preference data with online preference labels [52, 19, 35, we leave extending our analysis to this setting to future work. Throughout our paper, we assume for simplicity of analysis that preferences are generated by an underlying utility function and therefore contain no intransitivitties [29, 41].
Understanding PFT. Prior work has studied different parts of the standard RLHF recipe [17, 23, 38, 16] and the impact of preference data quality [37]. In our work, we instead take converge-basee perspective on the relationship between online RL methods and offline contrastive methods. Although derived from the same minimum relative entropy objective [63] and perceived as equivalent by some early work [33, 3], more recent work has started to unravel the distinctions between these two classes of methods. [43] repeatedly observe better performance from online rather than offfine methods and after rigorously validating a variety of hypotheses. conclude that on on-policy sampling is indispensable for ensuring a high quality policy. [42] perform in-depth study of the effects of preference data, contrastive losses, and on-policy sampling and conclude that a combination of contrastive losses and interactive training is most preferable in practice. [55] also observe better performance from online PPO than from offine DPO and argue this is because the former is able to eliminate a larger set of policies that undesirable from the perspective of the later. We supplement these mostly empirical observations with a rigorous theoretical explanation for the observed behavior through the lens of dataset coverage, well as designing algorithm that addresses the key weaknesses of offline contrastive approaches.
We defer additional related works to Appendix A.
3 Preliminaries
Following wide range of recent works [33, 3], we consider the RLHF problem in the contextual bandit formulation [25]. This is a reasonable simplification, as one can consider the generated sequence of tokens one single action, due the fact that the states the generated tokens, and the dynamics are deterministie. We denote the context (prompt) space as X, and the action (response) space y/ Note that due the finiteness of the possible tokens, the action space is finite but combinatorially large. We use p E ( to denote the distribution of the prompts, and X' A()) policies (LLMs) that map prompts to distribution of responses. We also consider the reward function class R : X x y + R. which assigns reward to each context-response pair.
We assume access to reference policy Tref. which is usually referred to the policy learned using supervised data when training the LLM, that needs to be further fine-tuned to align with human values. An offline preference dataset is collected in the format of D (ary" y" "t+ y triplets: given context x ~ p. the preference policy samples two responses y' yy² ~ pa- |)), where is the offline response distribution. Previous works assume either μ to be the same distribution
Tref [33] or different offline distribution [3, 35, 18]. Then, y² is labelled as y (thus y² as y with probability p"(y" > y² wr), where p" is defined by the Bradley-Terry model [6]:
(x,(r"(x,
p" p'(y² > 122 x))= exp(r" + exp(r* (z,z2))*
where r" is the human's implicit reward function. Note that this rules out intransitive preferences [41, 29J. Throughout the paper, we will make the following assumption on the reward function:
Assumption 3.1 (Boundedness of the reward). ||r* S R.
In many previous works, this formulation has been the canonical way to model the preference data in the RLHF literature [11, 33, 3]. The goal is to learn a policy и to maximize the objective J((T), where
(I) i.., we want to both maximize the human implicit reward, and not deviate too much from the reference policy. We denote the optimal policy π E argmax+em [(x). Here we call KL(T(- x)] (- x)) reverse KL because ग the policy be optimized, appears first. We will
(TT) = Ex-p [Ey- Ir" (xx v)1 - 3KL(T(-) | mrr [))]
call (L(TTer() x) [T[ æ)) forward KI. By the definition of KI., we have
Definition of reverse KL: KL(T(- |= xx)) = Ey-- (n(
(2)
Note that the expectation reverse KI. under æ, indicating that evaluating and optimizing reverse KL requires drawing online samples from In contrast, evaluating forward KL only requires offline
3

---

samples drawn from re As we will show, this key difference between reverse KL and forward KL plays an important role of separating online RLHF and offline contrastive methods such as DPO. In this paper, we consider two types of algorithms: online RL.-based algorithms, and offline contrastive-brt based algorithms.
Online RLHF Algorithms. We consider algorithms such as [11, 1] as the online RL based methods. We abstract these algorithms the following procedure: the algorithm performs the following two- stage procedure: one first trains reward model F thht minimizes the Bradley-Terr loss
exp(r(a,y*))
E argmax E Bg. DD D
log exp(r (z, exp(r(z, y
(3)
rER
and perform policy optimization (such as PPO [36]) to optimize the policy with the reward model
Trihf E argmax EawD (1y) [( w)] 3KL(TT( |x) [x] [ [))]-
[
However, this policy optimization step requires extensive online sampling, and possibly training an additional critic model (e.g., PPO), addition to the reward model and policy.
Offline Contrastive Algorithms. To circumvent the mentte-ee computational burden, several purely offline contrastive-base methods (i.e., without RL) have been proposed. In this paper, we focus the following two most representative methods. The first is Direct Preference Optimization (DPO) [33], where the objective Tdpo E argmax, fdpo with
x(y [2)
exp [Bog
(T) BZ. o wD WD
log
exp B
log
T [x) That (9) (x)
+ exp ( log
x Teet(ar (x)
Another offline contrastive method we will discuss in our paper is Identity Preference Optimization [3], but will defer its technical details to the appendix.
4 Offline Contrastive Methods Require Stronger Coverage Condition than Online RL Methods
We start by introducing the mathematical formulation of the coverage framework. The strongest coverage condition is the following global coverage condition [28]: we say any offline distribution =(yla) < Cgto- Throughout this section, we will adopt the covers a policy ı if we have maXxx)=(=)>0 plvl
setting where ref [33]. Formally, assume the following condition:
Assumption 4.1 (Global Coverage). For all n, we have
T xt) x.y:p(a)>0 Tref x))
max
S Cglo.
For the coverage terms, we always adopt the convention that O = O. Note that one sufficient condition for this assumption is that, for any prompt x, and any token sequence y, we have (e ()) 2 1/Cgl-
As has been recognized in the offline RL literature, global coverage is a strong assumption, and efforts have been made cireumvent this assumption with more relaxed coverage conditions [46, 10, 58]. In this paper, we will consider the following partial coverage assumption that is weaker than Assumption 4.1:
Assumption 4.2 (Local KL-ball Coverage). For all EKi < oo and all policy a such that Eg (KL( x=) [ree( =))] & Eki. have Tγy x:)
max #.yp(=)>0 (e (x))
< Cay-
Note that Coy depends on Eыg- This coverage notion is relatively new in the RL literature and only appeared in previous analysis of RLHF algorithms [9]. We call this local coverage condition since only requires Tref to cover the policies that is within some KI--divergence ball centered at mee The intuition of this assumption is, for any algorithm that can control the reverse KL of the output policy, we can leverage the coverage condition to relate the error under the output policy to its error under the offline distribution, and thus guarantee its performance. Finally, note that since the policies with bounded KL is subset of all policies, for fixed Tref. we always have Cz., S Cglo-
Wee use E to denote the empirical expectation over the dataset.

---

Remark 4.1. Taking a closer look at Assumption 4.2, we can see that this assumption is always a(yla)
true in the sense that for any policy with Eki oc, max (ad(a/a) < oc, i.e., Cen co, for any Eki. However, while being bounded, Ceu can be large. Indeed simple calculation can show x
This can be undesirable because this suggests bounded reverse KL itself is not enough to guarantee optimality: the error can have an exponential amplification when switching from ref æ. Thus this motivates Assumption 4.2, which assumes that Ce. is reasonably small.
that (:p())=0 [er()) can be as large max maxx..em(yls)>> exp
x(ylz)
In what follows, we will show that the global coverage assumption (Assumption 4. is necessary for offline contrastive-baset algorithms such as DPO and IPO, but partial coverage assumption such as Assumption 4.2 is sufficient for online RL based algorithms. This establishes a separation between the two types of algorithms. We emphasize this theoretical separation explains why in practice online methods is less prone to problems such as reward hacking and producing out-of-distributin responses that are due to dataset with insufficient coverage.
4.1 Global Coverage is Necessary for Offline Contrastive Algorithms
Failure of DPO Under Partial Coverage. Now we show that if the strong coverage Assumption 4.1 breaks, then DPO can not guarantee any performance with respect to the objective function Eq.
(I). The intuition is based on a rather common observation of the DPO algorithm: the DPO policy ddp may generate out of distribution responses, while in contrast, RLHF does not generate responses outside of the support of re due to online reverse- KL. constraint. For example, [55] provides a construction where dpp chooses response where RLHF policy assigns mass, thus proving that RLHF policies are a subset of DPO policies.
However, such construction assumes that the reward learning procedure of DPO makes arbitrarily large errors. Also, previous constructions assume deterministic preference, which only true if the underlying reward function is unbounded. This violates the natural assumption of Assumption 3.1. In the following, we relax these constraints and thus show that DPO fails to guarantee any performance in rather strong sense. Concretely, DPO constructs the following implicit reward class with the policy =(ylz)
π ]1) where ((x) is partition function that maps con- text to a real number and is independent y. Plugging this formulation into the BT loss (Eq. (3)) re- covers exactly the DPO loss (Eq. (4)) as the partition functions are canceled. Now we can characterize the returned policy by DPO as exactly whose corresponding reward function accurate distribution:
class II: Rdpo = { log ())
We assume the DPO policy ddpo satisfies that: Assumption 4.3 (In Distribution Reward Learning).
( log
27
Typo x)) x)2 -- --(((x))))
Ex Exsyo-oo
< Edpo
Note that this is rather strong assumption for BT loss - by Lemma B.2, best one can only hope: for any learned reward function F, for each context x, there exists a constant c(x) such that
E Ex.gr-porat (f(z, y) - (z. c(x))² Se,
(5)
i.e. the reward model predicts the human reward up to gap that is independent of y. This is due to the fact that BT loss only requires the reward function to capture the relative difference, or in other word, any constant shift (with respect to context) in the reward will be canceled in the BT loss. However, for the rest of the section, we will make the stronger learning assumption that the gap c(x) = 0 (such as in the case of Assumption 4.3). Previous counterexamples analysis violates this assumption, but we will show that even under this strong assumption, DPO still can not guarantee any performance.
Proposition 4.1. Denote Nee as any reference policy such that Assumption 4.1 breaks. Let Io be the set of DPO returned policies such that Assumption 4.3 holds. Then there exists policy E I1 such that J (T) -C.
Proof sketch. Without loss of generality, we consider a promptless setting, and assume that the response space y = {y), 3/2 J. Again without loss generality, we assume Tee only covers yı and y2, and thus Assumption 4.1 breaks. We assume partition function Z = 1 for all but we will be
5

---

rigorous in the formal proof. Then consider the following policy π such that
T T(1) r (31) (01)) Vddpo and
) π x12)
Blog
3 llg
--r((>)) V
Tret TTer
One can check satisfies Assumption 4.3. Now consider the optimal policy x² *"()) (Br) exp (vi)) for i E [1, and n" (ys) = 0. Since n" (vi) + x (22) = L, combining ev- crything we get T (ya) > 0, which implies ((eet is unbounded, thus we complete the proof. [
One can first relate the above construction to the parital coverage assumption Assumption 4.2: since the policy æ considered in the proof has unbounded reverse KL with respect to Tref, thus it is not in the KL-ball Eki around Feef which implies that Assumption 4.2 is not sufficient for DPO, Next we show that global coverage is necessary for the IPO algorithm.
Failure of IPO Under Partial Coverage. To show that the global coverage is necessary for IPO, we can even assume stronger in-distribution learning guarantee, that is, the returned policy achieves the smallest error on its population loss in distribution.
Proposition 4.2 (Informal). Denote Te as any reference policy such that Assumption 4.1 breaks. Let po be the set of IPO returned policies such that it is the minimizer of in-distribution error on its population loss. Then there exists policy π E Il such that J(T) -00.
We defer the detailed setup and formal version to Appendix D, but the construction for the above proof's share the same intuition: the reverse KL term in the objective function can be unbounded. For offline contrastive- beass algorithms, the KL regularization is only enforced under the data distribution, and thus the algorithm can not guarantee bounded reverse KL if the reference policy does not cover the response space well. Although that the same intuition holds for other offline contrastive-bast this point would be: how about the forward KI.? Not surprisingly, the forward KL for DPO (but conjecture for other offline constructive-based methods as well) is vacuously large, and we formalize this result in Appendix C.2.
only showed counterexamples for DPO and IPO, we conjecture algorithms. One natural question at
Remark 4.2. The folklore that DPO is equivalent to RLHF is often based some assumption that much stronger than Assumption 4.3: requires that the learned policy has a point-wise accuracy guarantee (dppe (y/ (xz) ref (y(=z))
r* (æ, y) for all z, y. Such point-wise guarantee is unrealistic in reality and does not hold in general in the supervised learning sense. The in-distribution style guarantee in Assumption 4.3 is the best could hope for from a supervised learning algorithm.
4.2 Global Coverage is Sufficient for Offline Contrastive Algorithms
After showing that global coverage is necessary for DPO to guarantee any performance, we now show that it is sufficient for the performance guarantee.
Theorem 4.1. Let Reef be any reference policy such that Assumption 4.1 holds. For any policy Tapo such that the event in Assumption 4.3 holds, have that
J(T") [(Tpp)) = (CCglo
Proof. By Lemma 1. we have
J(x") (TTpp) S (y [rr (z, w²) Edpo (2, w²) r" (I, (Ix + ropo y²)]
< EampByt ~ (1a)) (r (x, rdpo 111) - rr (2 + Tdpa (Ir, ))
C2
C2. Bumpllyy mm) (rr y-) (z. y²) 1rt (x, y2) Edpo (z, 02))²].
and we can complete the proof by plugging in the error guarantee from Assumption 4.3.
Note that as the proof suggests, the result holds with the more general reward learning guarantee as in
Lemma B.2 one only needs to be accurate in predicting the relative rewards between response pairs.

---

4.3 Online RL Method Under Partial Coverage
Finally, we contrast the previous negative results in Section 4.1 for offline contrastive-buse based algorithms to positive result for online RL.-based algorithms, under the partial coverage setting. We will show that in general global coverage is not necessary for RLHE, i.e., it can guarantee performance under partial coverage. In fact, one might still be able to show an impossibility result for RLHF under partial coverage, by reusing the same counterexample as in the previous section (c.r., Proposition 4.1). Concretely, as long as the learned reward f(ya) + oc, Tane (1s) will be and thus the reverse KL will be unbounded. However, this is rather unrealistic scenario, as the construction requires a reward model (e.g., a neural network) to output an unbounded value. Thus this motivates the following assumption:
Assumption 4.4. For all learned reward model F from the reward model class, we have that [| - R'.
At this point, one might ask why similar assumption is missing for the offline contrastive- based anal- ysis, since in Remark 4.2 we argued that a point-wise learning guarantee is unrealistic but Assumption 4.4 is indeed also a point-wise boundedness assumption. The reason lies in the different construction of the model class F for those algorithms: for DPO and IPO, the reward model is constructed as
dapo = 3 log Tiat- and there is no natural function class for π such that point-wise assumptions such as the one in Remark 4.2 or Assumption 4.4 holds. In contrast, post-processing such clipping, offline normalization and on-the-fly normalization of rewards is standard in practice, which means the policy will always witness bounded rewards [8, 9, 18, 1] during online RL training (e.g.. PPO). As we will show in the following, the difference in the reward function (which is tied to the offline vs. online nature of the algorithms) can explain the different coverage requirement of the algorithms. Note that we use the same in-distribution reward learning assumption for both types of methods.
To relate to Assumption 4.2, we first show that the reverse KI. divergence of the RLHF policy is always bounded under Assumption 4.4.
Lemma 4.1. Suppose that Assumption 4.4 holds. Then for any RLHF policy Thhtt we have that
Trinf x))
x)) )]] <
2R'
(L(TTiel||TT (Tret) = Exoa E (yynaa(-11))
log
Teef
B
Then we can show that the RLHF algorithm can guarantee performance under partial coverage:
Theorem 4.2. Suppose that Assumption 4.4 holds. Then for any reference policy Tee for which 2R' and any RLHF policy weht with F such that (c.r. Assumption
Assumption 4.2 holds with Exi
B,
4.3) En poadd (r" - ffxz, ")}² S Erewards we have
J(m²) J(Tent) O vrewardd)
Conditioned on Lemma 4.1, the proof of this theorem is similar to that of Theorem 4.1 so we defer it to Appendix D. Similar Theorem 4.1, we note that Theorem 4.2 holds under weaker reward learning guarantee as in Lemma B.2. We also remark that long
is finite, Con is finite, so the bound never vacuous. Since Cen S Cgtofor all ext, indicates the regrel bound of RLHF is never worse and can be much better than the regrel bound of DPO. Combining Theorem 4.1 and Theorem 4.2, we complete the separation result between offline contrastive methods and online RL methods.
A natural question at this point could be: can we further relax the local KL-ball cover- age condition in Assumption 4.2
single-policy coverage condition, i.e., just assuming maxx.y x (z)]z)) Tree (v|æ)) < C? Prior work [59] shows that with explicit pessimism, it is pos- sible. However, using pessimism makes the algorithm from [59] not computationally tractable and hard to scale to LLM experiments. Our conjecture is that for the RLHF policy Thhf. it is not possible to achieve meaningful regret under the single policy coverage condition, due KL not being strong enough to induce pessimism (i.e., bounded KL between and ree can still imply exponentially large density ratio x/ /. Developing lower bound for Tht under single policy coverage in this case can be an interesting future work.
7

---

Algorithm Hybrid Preference Optimization (HyPO)
require Pretrained LLM T0 reference policy Tef, offline data D, learning rate a, KL coefficient X.
I: for = l...., do
2: Sample minibatch of offline data Doff == (x. y y D.
The (x)))) Blog [Te(U-(=)
The-1 (U [x] Neae (V [2]
3: Compute DPO loss fdpo == Exrwl v EDon log Blog
4: Sample (unlabeled) online data Don == {x, where a ~ D2 1T0 (x).
5: Compute lui :- D., log(TO, (1]2z)) ss loog ( (g/x))
6: Update 02 = 0,-1 + a Vo. (Edpo Mlu).
return T..
5 Hybrid Preference Optimization: Regularizing Offline Learning with Unlabeled Online Samples
In this section, we will provide a practical algorithm that bridges the gap between the offline contrastive-based algorithms and the online RL-based algorithms. As in the previous sections, the difference between the two types of algorithms their reward model parametrization, and whether perform online rollouts. In the following, we will show that these two properties are in fact tightly intervened with each other.
Here will focus the DPO algorithm. One way fix the issue of the unbounded reverse KL of DPO (which is caused by the unbounded reward model class) to consider the following ideal procedure: at the beginning of the algorithm, filter out all the policies such that (L(T||/Tee) z 2R', where R' is the boundedness of the reward function class for RLHF. Now applying the same analysis of Theorem 4.2, we can show that this revised DPO algorithm can guarantee performance under the partial coverage assumption, because now the Lemma 4.1, sufficient condition for Theorem 4.2, is explicitly enforced by the constraints. We defer the detailed statement and analysis to Appendix F.I.
first go through the policy class II, and then
However, such filtering procedure is not possible in practice, but can instead consider the follow- ing constrained optimization problem: we call the definition of DPO loss in Eq. (4), we want to solve
2R'
max Eppo
s.t.
KL(T [rar) S
(6)
8
using the KKT conditions, we can show that the following Lagrangian form is equivalent to Eq. (6):
max Capo XKL(T] (re))
where À is the Lagrange multiplier. However, in reality, since we do not know the exact value of R', we can consider setting À to be a hyperparameter. We present the pseudocode Algorithm 1. Note that due to the reverse KL term, the Hybrid Preference Optimization (HyPO) algorithm optimizes Eq. (7) via both offline and online samples where the offline samples are used for constructing and optimizing Edpo (here a denotes the sigmoid function), and the online samples y TT- (r) are for KL (i.e., (2)) Note that regularizing with reverse KL via online samples is widely used in online RLHF PPO [40], APA [62], REBEL [18]). Here sg refers to the stop gradient operation, which is common practice in optimizing reverse KL in the LLM fine-tuning setting [30, 47]. Finally, previous iterative RLHF methods [53] can be interpreted hybrid methods as well. but they require labeling online samples from additional reward model while HyPO only requires unlabeled online samples.
PPO
Summarization. Our first experiment is on the TL:DR dataset [40]. Our experiment setup mostly follows [18]: we use a maximum context length of 512 and a maximum generation length of 53. We use Pythia 1.4B and Pythia 2.8B [5] the pre-trained model. For the supervised fine-tuning (SFT) model, we train it over 1 epoch of the dataset with human reference responses as labels. We train the reward model on top of the SFT over epoch of preference data. Both HyPO and DPO are trained over I epoch of preference data with Low-rank Adaptation (LoRA) [21]. We defer more experiment details in Appendix F.
8

---

Table 1: Results on TL;DR dataset. Winrate is evaluated by GPT4 and RM score is from the trained re- ward model. Experiments are repeated for 3 random seeds. Mean and standard deviation are reported.
Model size
Algorithm
Winrate (t)
RM score (t)
KL(=I| (4)
DPO
42.17% (2.5%)
0.16 (0.05)
44.90 (1.29)
1.4B
HyPO
46.44% (2.39% %)
0.37 (0.05)
27.07 (2.34)
DPO
44.39% (0.4%)
2.43 (0.10)
68.95 (3.08)
2.8B
HyPO
50.50% (1.9%)
2.51 (0.13)
48.98 (4.23)
Table 2: Results on general chat benchmarks. We evaluate the base model (Meta-Llama---88- Instruct), DPO-fine-tuned model, and HyPO-fine-tuned model.
MT-Bench 2nd Turn
AlpacaEval 2.0
Model
1st Turn
Average
LC Win Rate
Win Rate
Meta-Llamas Llama-3-8B-Bl 3-88ttnntttett [27]
8.31
7.89
8,10
26.0
25.3
DPO-Llama-3
8,08
7.41
7.75
28.6 .4
30,9
HyPO-I.lmma-
8.43
7.75
8.09
30.7
32.2
We summarize the results in Table 1: HyPO outperforms DPO in terms of GPT4 win-rage and reverse KI.. Particularly, the significant reduction in reverse KL implies the impact of including a reverse KL term explicitly into the DPO objective. While comparing with PPO (e.g., Table 1 in [18]), HyPO's performance is still lower in winrate, HyPO does preserve the key advantages of DPO over PPO: we avoid training additional reward model and value network.
Mean Valldaatonn 10
Hyfo
150.
Nonn Score
2003 se65 250 Fice pncer 200000000
Timestop
General Chat. In the general chat setting, the model is required to produce response y given user instruction æ. We again follow the experiment setup in [18]. where we finetune the Meta-Llama-3- 8B-Instruct [27] model on the ultrafeedback dataset [14]. Due to the computation constrain, we follow the setup in [18] where we only train the last 4 layers of the network for both HyPO and DPO.
Figure I: Mean valida- tion reverse KL the ref- erence policy when DPO and HyPO are trained for epoch on the TL.DR dataset. We repeat the experiment for 3 random seeds and plot the median and the shaded areas denote the min and max over the 3 repetitions.
For evaluation, we use the common metrics including AlpacaEval 2.0
[15], MT-bench [61] and Open L.M leaderboard tasks: MMLU [20], GSMSK [13], Arc [12]. TruthfulQA [26] and HellaSwag [57]. We
provide the results for AlpacaEval and MTT-enchh in Table 2, and the the results of the remaining tasks can be found in Table 3.
HyPO Mitigates Overfitting in Contrastive Methods. Since the offline contrastive based methods only work with a static offline dataset, the overfitting issue has been observed [43]. In our last experiment, we show that HyPO can effectively address the overfitting issue by leveraging the unlabeled online data. We follow the setup of the summarization task with Pythia-2.8B base model. We train DPO and HyPO for 5 epochs respectively, and evaluate on the first 300 data in the validation dataset. We plot the validation KL in Figure 1: we observe that HyPO is better at preventing the deviation from the reference policy caused by overfitting from training on excessive epochs, even though the methods theoretically both have KL regularization to the reference policy.
6 Function Approximation Coverage: Can Fine-tuned Policies Extrapolate?
Our final result is a theoretical explanation of the extrapolation behavior of preference fine-tuning algorithms under the global coverage assumption in the function approximation (FA) setting. The extrapolation behavior refers to the phenomenon of RLHF algorithms (e.g., DPO) can improve SFT models despite the fact that during training the policies assign decreasing likelihood to both the preferred and rejected responses (i.e., they must increase the likelihood of responses outside of the training data) [31].
A previous attempt [32] to explain this behavior is based on the assumption that the responses from the reference policy have the same distribution as the preferred responses from the dataset, i.c..
y 1 y Tref. However, mentioned in Section 3, more realistically, one should assume that

---

Table 3: Results on Open LLM leaderboard. We evaluate the base model (Meta-Llama-33- ---lmma--------e --a---Llamm---B----ttuu))))
DPO-fine-tuned model, and HyPO-fine-tunee model.
MMLU (5-shot)
GSM8K (5-shot)
Are (25-shot)
TruthfulQA (0-shot)
HellaSwag (10-shot)
Model
Average
Meta-Lllaaa--8B-Innnttn
65.68
74,91
62.12
43,88
78.76
65.07
[27]
DPO-IJama-3
65.82
73.62
63.14
45.02
79.1
65.34
HyPO-IJama-3 65.74
73.84
62.71
45.55
79.74
65.51
y ~ 11 y Tee since it is more natural to use the reference policy to generate pairs of responses to collect labels; or even more generally by considering Supp(D) C supp(TTee). The latter is common in practice, for example, the dataset is often precollected, the reference policy might have a small mass on some responses, so with a high probability they are not sampled during the data collection process.
In the following,
illustrate this behavior using linear function approximation:
Definition 6.1 (Linear function approximation). Consider the promptless setting with response space y. For all y E V, the ground truth reward r- r"( = =w <(y), where E IE is universal linear weight vector and o y + Ed is a d-dimensional feature map. In addition, all policies are parametrized as softmax linear policies, i.e., (V) x exp(wa σly)) The feature map & is known to the learner, and w is unknown. Finally, let y² = argmax r" ry) be the optimal action.
In general, we should not expect the offline dataset contain the optimal action y* under all situations. We show that thanks to the linear function approximation and the dataset coverage, DPO has hope to extrapolate correctly, i.e., it can increase the model's likelihood of the optimal action while decreasing the likelihood of both the preferred and rejected actions from the offline data:
Propesition 6.1. Under linear function approximation (Definition 6.1), there exists dataset collected from distribution μu that does not cover y". i.e., p(y*) = O, but has global coverage in the linear function approximation setting [54]: let Σn = Ey ((y) T, then all n, Eyy=ald(y)) s S
Cz. Then DPO will return a policy Tapo such that (y)) > O, but Tepa S Tref (y) for all y in the offline data support, i.e., (y() > O.
We defer the proof to Appendix D.3. The above result shows when the training dataset together with the function approximation allow the learned function to generalize (c.g.. learn a function that can predict well test examples beyond the training data a property supervised learning can have), algorithms like DPO can extrapolate correctly, i.e., they can push up the likelihood of the optimal responses outside of the training data while pushing down the likelihood of all the responses in the training data. Although it is not possible to show that extrapolation is guaranteed to always happen: suppose in the offline dataset, there is action y' whose feature ó(y') is almost identical p(v"), ), then in the finite sample case, with small learning error the final policy might increase the probability of y' instead of extrapolate to y", but in this case the policy is still near optimal. That said, our construction in the proof is general enough to explain non-edge-case under function approximation.
To validate theory result, in Appendix E we perform synthetic experiment global coverage with linear function approximation.
7 Discussion
There are few limitations of our work: 1) our theoretical analysis only considers the statistical perspective of each algorithm, but we believe our result is complementary to the other work that considers the optimization perspectives [42]. 2) we only conduct experiments on limited models and benchmarks. 3) The experiment result shows that HyPO's performance is still below the one of online RLHF: this might suggest that our theory does not fully explain the benefit of all the components of online RLHF. For example, one hypothesis is that the learn reward function may have better generalization ability. 4) It is not clear that the KLlbaal coverage necessary for online RL-based methods. However, as we discussed, since bounded reverse KL might still induce exponential error amplification, we conjecture that at least the single policy coverage [58] is not sufficient for online based methods that use reverse KL. We believe these limitations lead to several interesting further directions. Finally, our method may not explicitly address the potential hallucinations or toxic behavior of LLMs, which is a common shortcoming of ceneral-purposs fine-tuning algorithms.
10

---

References
[1] A. Ahmadian, C. Cremer, M. Gallé, M. Fadace, J. Kreutzer, A. Üstün, and S. Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14744 14740, 2024.
[2] Amortila, D. Foster, N. Jiang. Sekhari, and T. Xie. Harnessing density ratios for online reinforcement learning, In The Twelfth International Conference on Learning Representations, 2024.
[3] M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference Artificial Intelligence and Statisticx, pages 4447-4455. PMLR, 2024.
[4] Bagnell, S. M. Kakade, J. Schneider, and A. Ng. Policy search by dynamic programming. Advances in neural information processing systems, 16, 2003.
[5] S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430. PMLR, 2023.
[6] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
[7] S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freire, al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307. 152/7, 2023.
[8] J. D. Chang, K. Brantley, R. Ramamurthy, D. Misra, and W. Sun. Learning to generate better than your Ilm. arXiv preprint vrXiv:2306.11816 2023.
[9] J. D. Chang, W. Shan, O. Oertell, K. Brantley, D. Misra, J. D. Lee, and W. Sun, Dataset reset policy optimization for rlhf. arXiv preprint arXiv:2404.08499 2024.
[10] J. Chen and N. Jiang. Offline reinforcement learning under value and density-ratio realizability: the power of gaps. In Uncertainty in Artificial Intelligence, pages 378-388, PMLR, 2022.
[11] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances neural information processing systems, 30, 2017.
[12] Clark, Cowhey, O. Etzioni, T. Khot, Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 18Xi 05457, 2018.
[13] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 2021.
arXiv:2110.14I0
[14] G. Cui, L. Yuan, N. Ding, G. Yao, W. Zhu, Ni, G. Xie, Z. Liu, and M. Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377, 2023.
[15] Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint XXiv:2444.044755 2024.
[16] J. Eisenstein, C. Nagpal, A. Agarwal, A. Beirami, A. D' Amour, D. Dvijotham, A. Fisch, K. Heller, Pfohl, D. Ramachandran, et al. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:23
2023.
[17] I.. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. In International Conference Machine Learning, pages 10835-10866. PMLR, 2023.

---

[18] Z. Gao, J. D. Chang, W. Zhan, O. Oertell, G. Swamy, K. Brantley, T. Joachims, J. A. Bagnell, J. D. Lee, and W. Sun. Rebel: Reinforcement learning via regressing relative rewards. arXiv preprini arXiv:2404. 16767, 2024.
[19] S. Guo, B. Zhang, T. Liu, T. Liu, M. Khalman, F. Llinares, A. Rame, T. Mesnard, Y. Zhao, B. Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.0 04792, 2024.
[20] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009 203,:000000030,,, 2020.
[21] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang. and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint wrXiv:2106.09685 2021.
[22] S. Kakade and J. Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pages 267-274, 2002.
[23] R. Kirk, I. Mediratta, C. Nalmpantis, J. Luketina, E. Hambro, E. Grefenstette, and R. Raileanu. Understanding the effects of rihf on Ilm generalisation and diversity. arXiv preprint arXiv:2310.06495 2023.
[24] W. Kool, H. van Hoof, and M. Welling, Buy 4 REINFORCE samples, get baseline for free! 2019.
[25] J. Langford and T. Zhang, The epoch- algorithm for multi-armed bandits with side information. In Advances neural information processing systems, pages 817-824, 2008.
[26] Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprini arXiv:2109.07958, 2021.
[27] Meta. Introducing meta llama 3: The most capable openly available Ilm to date, 2024. URL https://ai. meta. com/blog/meta-llama 2024.
[28] R. Munos and C. Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9(5), 2008
[29] R. Munos, M. Valko, D. Calandriello, M. G. Azar, M. Rowland, Z. D. Guo, Y. Tang, M. Geist, T. Mesnard, A. Michi, et al. Nash learning from human feedback. arXiv preprint arXiv:2312.00886 2023.
[30] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[31] A. Pal, D. Karkhanis, S. Dooley, M. Roberts, S. Naidu, and C. White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint 242. 2024.
[32] R. Rafailov, J. Hejna, R. Park, and C. Finn. From r to q*: Your language model is secretly a q-function. arXiv preprint arXiv:2404. 12358, 2024.
[33] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2023.
[34] S. Ross and J. A. Bagnell. Agnostic system identification for model-based reinforcement 1007, 2012.
learning. arXiv preprint
[35] C. Rosset, C.-A. Cheng, A. Mitra, M. Santaeroce, A. Awadallah, and T. Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404. 03715, 2024.
[36] Schulman, Wolski, P. Dhariwal, A. Radford, and Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06333 2017.
12

---

[37] A. Sharma, S. Keh, E. Mitchell, C. Finn, K. Arora, and T. Kollar. A critical evaluation of ai feedback for aligning large language models. arXiv preprint arXiv:2402.12366, 2024.
[38] Singhal, T. Goyal, Xu, and G. Durreu. long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.0377 2023.
[39] Y. Song, Y. Zhou, A. Sekhari, J. A. Bagnell, A. Krishnamurthy, and W. Sun. Hybrid rl: Using both offline online data can make efficient. arXiv preprint arXiv:2210.0677 2022.
[40] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-300 3021, 2020.
[41] G. Swamy, C. Dann, R. Kidambi, Z. S. Wu, and A. Agarwal. A minimaximalist approach to reinforcement learning from human feedback. arXiv preprint arXiv:2401.0400 2024.
[42] Tajwar, A. Singh, A. Sharma, R. Rafailov, Schneider, T. Xie, Ermon, C. Finn, and
A. Kumar. Preference fine-tuning of lims should leverage suboptimal, on-policy data. arXiv preprint arXiv:2404. 14367, 2024.
[43] Y. Tang, D. Z. Guo, 7. Zheng, D. Calandriello, Y. Cao, F. Tarassov, R. Munos, B. Á. Pires, M. Valko, Y. Cheng, et al. Understanding the performance gap between online and offline alignment algorithms. arXiv preprint arXiv:2405.084/ 2024.
[44] G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac. J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, al. Gemini: family of highly capable multimodal models. arXiv preprini arXiv:2312. 1805, 2023.
[45] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.00 0007809 2023.
[46] M. Uehara and W. Sun. Pessimistic model-based offline reinforcement learning under partial coverage. arXiv preprint arXiv:2107.00
2021.
[47] L. von Werra, Y. Belkada, L. Tunstall, E. Beeching, T. Thrush, N. Lambert, and S. Huang. Trl: Transformer reinforcement learning. https://g/t thub. com com/hugginggace. tr1, 2020.
[48] R. J. Williams. Simple statistical
algorithms for connectionist reinforcement renarctcoeettrrriinfeeeeeee learning. Machine learning, 8:229 256, 1992.
[49] T. Xie, C.-A. Cheng, N. Jiang, P. Mineiro, and A. Agarwal. offline reinforcement learning. Advances in neural information processing systems, 34:6683- 6694, 2021.
pessimism for
[50] T. Xie, D. J. Foster, Y. Bai, N. Jiang, and S. M. Kakade. The role of coverage in online reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.
[51] T. Xie, N. Jiang, H. Wang, C. Xiong, and Y. Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems, 34:27395-27407, 2021.
[52] W. Xiong, H. Dong, C. Ye, Z. Wang, H. Zhong, H. Ji, N. Jiang, and Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023.
[53] W. Xiong, H. Dong, C. Ye, Z. Wang, H. Zhong. H. Ji, N. Jiang, and T. Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Conference on Machine Learning, 2024.
[54] W. Xiong, H. Zhong, C. Shi, C. Shen, L. Wang, and T. Zhang. Nearly minimax optimal offline reinforcement learning with linear function approximation: Single-agent mdp and markov game. arXiv preprint arXiv:2005. 15512, 2022.
13

---

[55] S. Xu, W. Fu. J. Gao, W. Ye, W. Liu, Z. Mei, G. Wang, C. Yu, and Y. Wu. Is dpo superior to ppo for Ilm alignment? comprehensive study, arXiv preprint arXiv:2404.10719 2024.
[56] L. Yuan, G. Cui, H. Wang, N. Ding, X. Wang, J. Deng, B. Shan, H. Chen, R. Xie, Lin, et al. Advancing IIm reasoning generalists with preference trees. arXiv preprint arXiv:2404.02078, 2024.
[57] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv: 078300 2019.
[58] W. Zhan, B. Huang, A. Huang, N. Jiang. and J. Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory, pages 2730-2775. PMLR, 2022.
[59] W. Zhan, M. Uehara, Kallus, D. Lee, and W. Sun. Provable offline preference-based reinforcement learning, In The Twelfth International Conference on Learning Representations, 2023.
[60] Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and J. Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint 20X1::2305.10425, 2023.
[61] L.. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-asa-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
[62] B. Zhu, H. Sharma, F. V. Frujeri, Dong. C. Zhu, M. I. Jordan, and J. Jiao. Fine-tuning language models with advantage-inducee policy alignment. arXiv preprint 2rXiv22306.02231, 2023.
[63] B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey, et al. Maximum entropy inverse reinforce- ment learning, In Aaai, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.
14

---

A Additional Related Works
Extrapolation Behavior of PFT. Recent work [56, 31, 32] has observed an interesting effect of the DPO procedure: a simultancous decrease in the likelihood of both preferred and rejected responses. This behavior is surprising first glance because one would expect that DPO will increase the likelihood of preferred responses and decrease the likelihood of rejected responses. We provide rigorous statistical explanation of this behavior and show that this behavior natural when the offline preference data only contains sub-optimat responses but the function approximation allows DPO to extrapolate and generalize to correct optimal responses. This highlights the role of function approximation in the success of offline contrastive based methods.
Coverage. We analyze online RLHF and offline contrastive- -aaese methods via the concept of coverage. Coverage measures how well an offline (data) distribution covers the support of the policy of interest, which has been the key technical tool in offline RL [28, 49, 46, 58], offline-onlin RL [34, 51, 39, 2] and online RL [22, 4, 50J. The data coverage plays an important role in our analysis since both online RLHF and offline contrastive-based methods rely on an offline preference dataset for learning.
B Auxiliary Lemmas
Lemma B.1 (Objective decomposition). Let ((T be the objective function defined in (1), and for reward function f, we let
f E argmax En-p (Ey- (f() v)1 - (KL((((- | [TT [x)]]
(8)
x
then have
J(TT) J(+) S Exupl
(r f(x,,()))))))) r (2, f(r, x²]]]-
Proof. We have
((xT) J(a)
=Exup By (:100 (r" (ar, v)] BKL( [ meet [])]] Ex-p ( [r [2) (KL((( [x) Tee [))]
Exwpp BByna (: (r" (a. v)] ( [eet
[x)]] (Ezpp Eyy= (x, (KL( ()) m [=))))
+ Ex- B (- [f
3KL(A(- [ [r
x)) (Be~pl By [f ( [x) [Trer [x))])
-Byro Byynar (:10) (r" (z, v)1 SKL((* (: [
( [x))] - (Ezmp
Ey [fx, v)1 _SK((T [)]l x))])
+ Exup By [fx,, v)1 - (KL((( x) [re x)) - (Eznpl By fr v)] - SKL(A(- [)]l [x)))
-Eaup By (-1x) (r" (xr w) f(zz, w)] En-p [By+ (xr, v) fxx v)]]
where the inequality due to Eq. (8). To complete the proof, note that
Expp B (-1)) (a, 3) f(z, 8)] Ex-plyy- [r (I, y) - f(x, a)]]
EEwp (11),,) (r (z, y²) - f(z,v")] Ex~p (1)) (a)" f(x,) y²]]] Exwp [By' (-1 - [r' (x, v²) f(I, - r" (a, 2²) + f(z,
Lemma B.2 (Lemma C.2 from [9]). Assume that r' is bounded, let R. be the reward function class, and Let
exp(r(x,y*)) argmin En.y .y- -D log exp(r(x,yt)) +exp(((xvx)) ).
rER
then we have with probability at least - 8 that
En Bacgt-a--ppoo poooo (r" - - ((x, + flx, 13)) ck
N
where & measures the non-linearity of the function, and e is a constant, N == [D] the size of the offline dataset.
15

---

C Additional Results
C.1 Results for IPO
In this section we give detailed technical details for IPO, and the negative results for IPO under partial coverage. Recall that the empirical objective of IPO is is Tipo E argmin, lipo (TT. where
[100E
8-1 27
( (T) = Exy,y v wa/|
T TT Teeee x))
Ter
xx)
2
The empirical objective derived from the following population loss
(T) = Ex,n² spmportee
(hr (lut.0") ((,t,)))18)
(9)
where
"TV (Trer))ret((2)) ( (yn
h 1/2 log
and I(y', 12²) is a Bernouli random variable with parameter p = p" (y1 > y2). where here p" can be any underlying human preference (that is not necessarily parametrized by the Bradley Tenry model). To show the negative result, we can make the following learning assumption:
Assumption C.1 (In distribution guarantee for IPO). We assume that the returned policy Tipo satisfies that
Tipo argmin (io
=e
Le., the returned policy Tipo induces the smallest possible in-distribution error its population loss.
With the setup, we can state and prove the formal version of the result:
Proposition C.1 (Formal version of of Proposition 4.2). Denote neet as any reference policy such that Assumption 4. breaks. Let To be the set of IPO returned policies such that Assumption C.I holds. Then there exists policy π E 1o such that ( -00.
Proof. Without loss of generality, we consider a promptless setting, and assume that the response space is y = {81, V2. Vs Again without loss of generality, we assume only covers y1 and y2. and thus Assumption 4.1 breaks. Specifically, let Tree Tea (y2) = 1/2. Then we have
Tipo = argmin Ey yaatee (Log (z²2)
T(y²)
(v+, (u'.82)18) 27
xell
which gives
Tipo(11) ) Tipo (U22) = p= p0( h > 82)/3,
log
and thus we have the relation that
(1po = (122 (p" (v1 > 9))
Let Tipo (322 = a E (0, 1]. then for any a such that Tipo = 1 - (1 + exp(p" (y1 v2)/3))a > we will have that KL (Tipol |Tef) is unbounded, and thus complete the proof.
C.2 DPO Has Vacuous Forward KL
In this section, we show that in the worst case, the forward KI. of DPO is vacuously large. We first how
relate the forward KL divergence of the DPO policy with the reward learning guarantee. Consider any DPO policy dapo and its corresponding reward model rdpo- By construction of the DPO algorithm, have, for any x, y pair that is covered in the dataset, Ay æ))
16

---

Torly ue) exp(FFiia (x.))())))) 2(xr) divergence is
where ((x) = Ey Trat(1l / x)) exp(rdpo v)88). Then the forward KL
Tepp xx) } Ea..grpoo log
Teet (y 2))
(x)) (Tpo w)/B)
)
Pápo (2, y)
Exyyrpor
log
E. gryppoo
log 2 ((x))
Although the first term be easily related the reward learning guarantee, the second term (Eg~a (oog (x))) can unfortunately be vacuous without further assumptions. We formalize in the following result:
Proposition C.2. There exist dip such that Assumption 4.3 holds, but (L(T Tdpa) is arbitrarily large.
Proof. First without loss of generality let consider that
Now suppose there exists
such that Frof (ū x) = म for all v, where will be determined soon. Now suppose that for all æ, idpo (z, 17 r- (x( 9) and (T, y) (x, y) for all y + ÿ. Now can check that
E Byyna((
(Fāpo (z, v) r (ar, v))
1
n²
which is diminishing if we take be big enough. We can also check that
B Edpo (2, y)
1
R
B
3
5
and thus the first term will have little impact on the final bound. However, the second term can be lower bounded as follows:
(", v) + F(xx, v) r (xr, y)
log Σ Feaf (y x)) exp((((x, v)/B3
= log
Σ Tref (y/
| x)) exp
B
F(x, 1) - r (2x v)
2 log
Σ Trer(y) x)) exp
F(x, () r² (x., ②
log
Teet a:) exp
3
3 4log(n).
Putting everything together, we have
R
KL(Trer TTdpo) 2 4log(n)
n
4B
and since we can take n arbitrarily big we complete the proof.
D Omitted Proofs
D.1 Proof of Proposition 4.1
Proposition D.1 (Restatement of Proposition 4.1). Denote Fee as any reference policy such that Assumption 4.1 breaks. Let Ioo be the set of DPO returned policies such that Assumption 4.3 holds. Then there exists policy π E Il such that ((r) = -c.
Proof. Again in the proof sketch, without loss of generality, consider promptless setting, and assume that the response space is y = {y1, 1/2 ya Again without loss of generality, we assume mee only covers y1 and y/, and thus Assumption 4.1 breaks. Now consider the optimal policy
n (2) Traf x)) (x)())
Vy v
(1)
17

---

where Z" = Eyey (y 1) (v)/8), note that by construction T² T1²ss) = 0.
Then consider the following policy π such that
(311) TT (vi)
(82)
B log
Z = r (v1) v Edoo and
Blog
= r (12) V
T (U2) Z
Then have
Eny (Allee
ddpo Weof (y) x)) "=
r" (z, (1)
21
Edpo
thus и satisfies Assumption 4.3. Rearranging we can see that (1/1) < T5" (v1) and (1/2) < (1/). Now since π 0, we have
n" (y1) + π (8/2) =
and combine we get (1s) > O, which implies (L(T||(TTet is unbounded, since Thee 0.
D.2 Proof of Theorem 4.2
In this section we prove Theorem 4.2:
Theorem D.1 (Restatement of Theorem 4.2). Suppose that Assumption 4.4 holds. Then for any
reference policy Tee such that Assumption 4.2 holds with EKi 21L' for any RLHF policy Teit with such that (c.r. Assumption 4.3),
Ez gopprt (1 (2, F(x, w)) S
Freward
or more generally, the event Lemma B.2 holds for F, we have
( (TTan) S O(Ceu Ereward).
To prove this we first prove the following lemma so we can leverage Assumption 4.2: Lemma D.1 (Restatement of Lemma 4.1). Suppose that Assumption 4.4 holds. Then for any RLHF policy Telff we have that
Trihf (yr x:)
2R'
KL (T(nrelll (Tee)) ==
(-1x)
log
<
Trof (y
B
Proof. since we have that Teinf a:) = exp(((z2))))) for all x E supp(p), E V. we have 2(x)
(xp(F(xx))) BZ
)}
f(x, v) log 1og(2(1))
KL(Tarl (Trer) = By log
Ey
B
Plugging in the definition Z(x) we get
f(x, v) B
)1)
f(a, w)
log(Z(z)) log (yynn
exp
E By (1)
due to Jensen's inequality. Thus we have
f(x, y) B
f(x, 2)
21'
KL(Tmell (Tee)) & By
En E By--mar(-1o)
<
8
8
Now with Lemma 4.1, we can prove Theorem 4.2:
18

---

Proof. By Lemma B.1, we have
J(T)) - (FThht)
< ExmpBy'' (1))
(x, w²) =(x,,")))))))) - -xx,, + (/)
(BexpBy' (1x)),1 (r* (1, f(z, 3²) -" 3²) f( 82))
Ggio Bump ya (aae)))rrr( (r)(z))") F(z, v2) r (x, F( 1()
(Lemma 4.1 and Assumption 4.2)
S C rewardd
(Lemma B.2)
D.3 Proof of Proposition 6.1
Proposition D.2 (Restatement of Proposition 6.1). Under linear function approximation (Defini- tion 6, there exists dataset collected from distribution i that does not cover y". i.e., p(y") = 0, but has global coverage in the linear function approximation setting [54]: let Dn Eyrn ((y)()) then for all π, EEy=ell (l I Cx Then DPO will return policy Tdpo such that dpo (v" 0, but ddpo (v) S Tree(() (y),for for all y in the offline data support, i.e., ply) > O.
Proof. Consider a response space y = {yı 1/2 ya). with o(y1) = [1,0], ó(y)) =
[1/2, 1/2), Φ(us) = [0. 1] Let Whef [1, 1), then we have (re((Ui) 1/3, Vi [1,2.3}. Let the ground truth reward function r"(y) = [10, 1] (vy), and suppose supp(u) = {yı, 12 i.e. the data only covers y/ and y2. and (4yaa) 0. And as always, the preference is based on the ground truth reward function under the Bradley-Terry model.
We can first check that the data distribution indeed has global coverage: for all i we have Ey> By=x)1(( I Cx. If parameterize f(y) @ <(y) (or in case of DPO, can still
check and see that (y)) = Wdpo dly) because of the softmax linear parametrization of the policies), for either direct reward learning DPO, we can have the learned reward function F(y) = [10, I] o(v) + where is the constant reward shift (c.r. Eq. (5)). Then a simple calculation (by (v) x Tref exp(f(y)/ shows that, as long as small enough, the policies will decrease the likelihood of y1 and y/2 and increase the likelihood of ys.
E Synthetic experiment for extrapolation
E.1 Extrapolation with function approximation
We first describe our experiment setup. We consider linear function approximation setting where have 100 responses (D/1 = 100). We consider 16-dimensional feature vector Φ : y - R16 and we generate Φ(v) by simply sampling 99 random 16-dimensional vectors where the l norm of each
vector is We add one final p0y) = [1, 0, ......
We construct the implicit human reward r" r"(() = w² (yy), where = [5, and the rest of the entries are sampled from 2ni2(-
We parametrize the policies as softmax linear policies, i.e., parametrize each policy with TH
p(y) R 16 such that ( Evey wi o(v) One can check in this formulation the implicit reward in DPO (Fdpo) is linear in Φ.
We generate 10000 preference pairs, according to the BT model under r", for the first 50 responses. We checked that the first responses indeed span IR 16 Thus the offline data has global coverage in linear function approximation setting.
For on-pol RI. methods, we first train a reward model. Then we simply perform gradient descent on the KL-regularized bandit loss (we assume nee is uniform). For DPO, we simply perform SGD
19

---

Online RUHF with Linear FA
DPO with Linear FA
-2-
Log Probabiitty
Log Probobittny
Preferred Probabiltyy Best Probability
-6-
-10
800
1000
1000
2000
3000
4000
5000
iteration
Iteration
Figure 2: Extrapolation behavior of Online RL method and DPO under linear function approximation. We plot the mean log probability of the preferred responses and the log probability of the best response, which is unseen in the training data. We see that both algorithms correctly assigns increasing probability the best response.
DPO without FA
0.00400
0.00375
0.00350
Probabiity
0.00325
0.00300
0.00275
0.00250
0.00225
0.00200
Out Distribution Probability
1000
3000
Figure 3: Extrapolation behavior of DPO without function approximation. We plot the average probability -fisttriiitiitt responses along the training and DPO assigns increasing probability to out-of cut-of-distribution responses.
on the offline preference dataset. We track two qualities over the training: the mean log probability of random subset of preferred responses, and the log probability of best response @(v) = [1, 0, ..
We plot the results in Figure 2. We observe that both methods have the extrapolation behavior - the probability of preferred responses decays but the probability of the optimal response goes up.
E.2 Extrapolation without function approximation
Now we describe the setting where function approximation fails, and this reduces Multi-arm bandit setting. We set | = 500, and the offline data only covers the first half of the responses. The r"(y) by sampling from Unif(-10.100 10), and generate 10000 offline samples by uniformly sample pairs of responses from the first half of the response space, and then label them with BT model under r". We train DPO with 5000 iterations, and plot the mean probability of the responses outside of the data support in Figure 3: we observe that the mean probability of the out-of distribbttii responses are increasing, however, this could be an undesirable behavior because the reward of the dut-of-distribution responses could be arbitrarily bad.
F Details of Section 5
F1
Theoretical guarantee
In this section, we consider the constrained optimization version of HyPO (Eq. (6)). Note that the n(gla)
reward function class is identical DPO, i.e., Rhypo g log
II where ((x)
the partition function. Then for each output policy Thypo denote its implicit reward function
20

---

(1, v) == B [y)=)
[ue(()=))-((]) and similarly to Theorem 4.2, we can obtain the following guarantee
in the partial coverage condition:
Theorem F.1. For any reference policy Te such that Assumption 4.2 holds with HyPO policy Thypo such that the event in Lemma B.2 holds, i.e.,
28'
3 for any
Ex 2 papto (r" (x,y²) - r- (x,8²) - TTypo y + Thypo (I, y x² S Ehypo
we have
J(x)) (TTypoo) 3 O
/hhypo
Proof. The proof mostly follows the proof of Theorem 4.2, It remains to show the following two properties:
1) Note that Theorem 4.2 requires Assumption 4.4, which does not hold for Thypo (note that Thypo only bounded under p. but not for all r)) but we only use it to prove the sufficient condition in Lemma 4.1, which is satisfied by the constraint of HyPO.
2) We need to check that the premise of Lemma B.1 holds, i.e.
Thypo argmax BegmaxEx<pl () (E) Thypo (z, y)] BKL(T(- Tee(( x))]
note that with the reparametrization between Thypo and Thypo Thypo is always among the minimizer of the unconstrained policy set, so we can still invoke Lemma B.2. The rest the proof now follows the proof of Theorem 4.2 so we omit the details.
Finally, we remark the connection to the negative result of DPO, i.e, Proposition 4.1: note that given (u Ted (g)lz)
KL (Thypol [rett S oo, we have that for all such that p(x) > O, we have for all y. B log
00, (again with the convention that 0 = 0), which breaks the construction of Proposition 4.1.
0
F.2
Experiment details
F.211
Summarization
In this section, we provide more details of our summarization experiment. We use the Pythia 1.4B and 2.8B model [5] with hugging face model cards: The TL;DR dataset is available at https: //g thub. cwm/cccm///////eee opmna// summar commarize-io from- feedback. The human reference dataset contains 117k training. 6.45K validation and 6.55K testing data. The preference dataset contains 92.9K training and 83.8K validation data. The reward evaluation and KL computation is performed on the whole validation data of the reference dataset. The GPT winrate is computed on subset of 600 samples from the validation data. The GPT API checkpoint we use gpt-4-0613. We follow the standard prompt for the winrate evaluation (e.g.. see Appendix D.3 of [18]). Below we provide the hyperparameter for HyPO and DPO. Note that to optmize the online KL, we use Reinforce with Leave One Out (RLOO) [24] with two generations per prompt (A 2) and optimize trajectory-leve KL.
and
For our experiment, we run on a cluster mixture of Nvidia A6000 and L40 GPUs with 48 GB VRAM. We use 4 GPUs in parallel for training, and for DPO the experiment time varies from 1 hour hours finish, and for HyPO the time varies between 4 hours to 5 hours.
Table 4: RM/SFT hyperparameters.
Table 5: DPO hyperparameters.
Learning rate Batch size
3e-6
Learning rate
3e-6
64
Batch size
64
Learning rate scheduler
cosine
Learning rate scheduler
cosine
Optimizer
Adamw
Optimizer
Adamw
LoRA
False
B
0.05
21

---

Table 6: HyPO hyperparameters.
Learning rate
3e-6
Table 7: Lora configurations.
Batch size
64
Learning rate scheduler
cosine
1024
Optimizer
Adamw
2048
B
0.05
Dropout 0
0.0001
RLOO k
2
F.2.2
General Chat
For the base model of general chat experiments, we use Llama3-8B-Instruct [27] with hugging face model card: -er---amma/ee---L---------------
The dataset card of the Ultrafeedback dataset [14] is #uggiingfacell/fuutttreeeeddddd banarrrrreeeeebbccceiaaiieedd task add additional length penalty, and the online penalty of generation with context becomes log
addition the KL penalty, the general chat
<(ulz)
alyl- We summarize the hyperparameter of each baseline below.
We the general chat experiment node of 8 Nvidia A100 80GB GPUs. DPO takes 3 hours to train one epoch while HyPO takes 18 hours to train one epoch.
Table 8: HyPO hyperparameters.
Table 9: DPO hyperparameters.
Learning rate
3
Batch size
8
Learning rate
3e-6
Learning rate scheduler
linear
Batch size Learning rate scheduler
8
Optimizer
Adamw
linear
B
0.05
Optimizer
Adamw
A
0.0002
B
0.05
RLOO k
2
0.02
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have theoretical and empirical results supporting the claims made in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contri- butions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We provide diseussion about the limitations of our work in the discussion
section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specificationn, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., the approach was only tested on few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low images are taken low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We clearly state all our assumptions all theorem statements, and the proofs can be found in the appendix.
23

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide pseudocode, hyparameter table and code this submission.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of model checkpoint, other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
We use public data on hugging face, and we submit the code for this submis- sion.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
24

---

Please see the NeurlPS code and data submission guidelines (https://nips.c/publii cttpbppicccpppbccccccc deides/ ssionPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips. cctppblcpclccccccccc cuubicc/on wc//giidess Suuide//Coeeuumi is lonoliccy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only subset of experiments are reproducible, they should state omitted from the script and why. At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
which
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide all the training details and hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We repeat all our experiments over 3 seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, the hypothesis Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates). If error bars are reported in tables or plots, The authors should explain in the text how they were caleulated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
25

---

For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provided run time and type GPUs we used our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or
cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http://eeeriiicoce htt.ee/wwww...c.poooocc//tttttee
Answer: [Yes]
Justification:
We respect the code of conduct.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impact in the discussion section.
Guidelines:
The answer NA means that there is societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
26

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release data or models in this work
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cited all the dataset and models used in this work.
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (c.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g.. website), the copyright and terms of service that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswil thcode. com/dataseeeeeeeeeaaaoacesss datassts has curated licenses for some datasets. Their licensing guide can help determine the license of dataset. For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
We do not release any new asset.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/moodl part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
27

---

Justification:
We do not have human subject.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No human subject.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "aTPJFJEJa6", "reviewer": "Reviewer_YCrR", "summary": "This paper studies learning from human preference feedback for aligning large language models (LLMs). Many existing works share the same observation that offline alignment methods such as DPO underperforms their online counterpart such as PPO. But this phenomenon has not been well understood. This paper studies this difference through the lens of coverage. It unveils that the KL constrain in offline alignment methods can be violated due to incomplete coverage and thus offline methods may find bad solutions. Inspired by this insight, the authors propose a new method, hybrid preference optimization (HyPO), which combines the offline preference update with the online KL regularization. Experiments on the summarization task demonstrate that HyPO is better than DPO. This paper also sheds light on a counterintuitive observation that DPO often decreases the likelihood of preferred responses through an illustrative example.", "questions": "In the TL;DR experiment, I am curious to know what happens if the authors also train HyPO for 4 epochs. Would it perform better than PPO?", "limitations": "This paper includes a fairly thorough discussion on the limitations at the end. I would like to add one point: the analysis in this work only focuses on the solution space. It explains why offline methods may find bad solutions, but doesn't explain why they often find the bad ones in practice. As far as I know, people often use a very low learning rate for DPO in practice. For example, the original paper used 1E-6. One possible reason for such low learning rates is that it compensates the ineffective KL regularization. Despite such efforts, DPO still finds bad solutions. It will be insightful if we get a better understanding of why bad solutions seem inevitable.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This work addresses an important matter and provides insights on a widely discussed question - why do offline alignment methods often underperform online methods? The paper is well written and easy to follow. The theoretical claims are sensible and easy to understand. I believe the community will benefit a lot from this work.", "weaknesses": "The computational cost of the proposed HyPO algorithm is not sufficiently discussed. Usually the most computational cost in online alignment methods comes from the sampling process of LLMs. In practice, the computational cost plays an important role in comparing algorithms. Thus it will provide useful guidance if the computational cost is well discussed in the paper.\n\nThe discussion in Section 6 is bit handwavy. While Example 6.1 provides good intuition, a more rigorous analysis is needed to justify the claim.\n\nOne minor suggestion on presentation: It will be good to provide the PPO numbers in Table 1 to help the readers compare HyPO to PPO more easily.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and we address the reviewer's comments below:\n\n> Computational cost analysis. \n\nWe thank the reviewer for pointing this out. Indeed the computational analysis is important in comparing finetuning methods. We had a short discussion at the end of Appendix E.2 in the submission but the runtime comparison is performed across a mixed types of computational resource. Towards a rigorous analysis of the computational comparison, in the supplemental material for the rebuttal we include the results of the computational comparison in Fig. 1 – we fix the computational resource on the same node with 8 A6000 GPUs, and we run each algorithm for a batch of 512 prompts, averaged over 50 batches. We use the reported hyperparameter for each algorithm (DPO and HyPO’s hyperparameter can be found in Table 3 and 4 in the submission, and PPO’s hyperparameter is in Table 5 of the supplementary material). \n\n> The discussion in Section 6 is bit handwavy. While Example 6.1 provides good intuition, a more rigorous analysis is needed to justify the claim.\n\nThank you for the suggestion. Yes indeed example 6.1 can be turned into a rigorous proof which states (in high level): under linear function approximation, as long as the offline data has global coverage in terms of relative condition number (instead of density ratio), and the optimal response is not in the offline dataset, then the extrapolation behavior of preference learning algorithms will provably happen. We will update this in the final version. \n\n>  Adding PPO numbers. \n\nWe adopt the PPO result from [1], which uses the https://github.com/vwxyzjn/summarize_from_feedback_details codebase. We used the same codebase to report DPO performance and implement HyPO. We include the PPO performance in Table.1 in the supplementary material of the rebuttal. The hyperparameter for PPO we use in the computation analysis is the same as the ones used to report in Table 1.  \n\n> Multiple epochs training.\n\nWe want to clarify that the current statement in the submission about PPO epoch number is actually imprecise: typically PPO does not train for 4 overall epochs, but the 4 epochs refer to the fact that PPO requires 4 epochs RL update for each online minibatch. We will fix the statement in our final version. PPO’s additional computation cost is indicated in the new computation comparison analysis. Since HyPO uses REINFORCE/RLOO for online kl optimization, we observe that taking 1 gradient step per minibatch suffices and multiple epochs per minibatch does not improve performance (however, multiple batches of PPO update seems necessary). \n\n\n> The analysis in this work only focuses on the solution space.\n\nWe thank the reviewer for pointing out the additional limitation that our paper only focused on the solution space. We agree that analyzing the more fine-grained dynamics of the algorithms is a very important future direction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e5CEuAzfSS", "reviewer": "Reviewer_ewpX", "summary": "The paper focuses on the paradigm of fine-tuning large language models (LLMs) using human preference data. It delves into two primary techniques: online reinforcement learning (RL) and offline contrastive methods. The authors challenge the previous notion of these techniques being equivalent by conducting a theoretical analysis through the lens of dataset coverage. They introduce the concepts of global and partial coverage conditions, proving that the former is necessary for offline methods using forward KL like DPO to converge to the optimal policy, while the latter is sufficient for online RL methods using reverse KL. The paper proposes a hybrid preference optimization (HyPO) algorithm, demonstrating its empirical superiority over DPO while maintaining some computational efficiency. The authors also provide a coverage-based explanation for why RL and offline contrastive methods might decrease the probability of preferred responses.", "questions": "See the Weaknesses section.", "limitations": "The authors discussed the limitations well in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper provides a rigorous mathematical analysis of the different conditions under which offline and online methods have provable performance guarantees, contributing to the theoretical foundation of preference learning in RL.\n\n2. The authors introduce the HyPO algorithm and support their theoretical findings with empirical results, demonstrating the effectiveness of HyPO on the TL;DR dataset.", "weaknesses": "1. The experiment is not sufficient. Since it is done with Pythia 1.4B and the TL;DR dataset, it is unclear if the proposed method is still valid on larger models and other datasets. While theoretically interesting, it is unclear if using reverse KL instead of forward KL can lead to a significant performance gain in practice. \n\n2. The proposed HyPO method only uses online samples to calculate the KL loss rather than to collect new preference feedback. While simpler, it may fail to fully leverage the benefits of online methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer for the helpful comments and we address them below: \n\n> Insufficient experiments.\n\nWe agree that experimenting on larger model sizes and additional dataset is important to demonstrate the effectiveness of HyPO and validate our theory. In the supplementary material for the rebuttal, we include the experiment for finetuning Llama 3 8B-instruct model on UltraFeedback dataset, and provide comparison with DPO on MT-Bench, AlpacaEval, and Open LLM Leaderboard (Table 2 and 3), the standard large-scale finetuning setup. We refer the reviewer to the common response section for more details for these results.  \n\n> The proposed HyPO method only uses online samples to calculate the KL loss rather than to collect new preference feedback. While simpler, it may fail to fully leverage the benefits of online methods.\n\nWe believe that it is beneficial to obtain additional online information such as reward information when the resource permits. However, gathering online reward information either requires additional human labellers or another reward model, which the latter reduces to online RLHF which requires additional computational resources. We performed a controlled experiment on the computational comparison between DPO, HyPO and PPO in Fig 1 of the supplementary material, which demonstrates the computational advantage of not using a reward model. On the other hand, if the reward model is trained on the same offline dataset, then theoretically one does not require the online reward since it does not introduce new information. (However, in practice people also observe that using reward model indeed improves performance even though it does not introduce new information, which as we pointed out in the limitation section is an interesting future theory direction to pursue).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qG5L3YvEVr", "reviewer": "Reviewer_yufq", "summary": "This work considers the statistical separation between contrastive algorithms (DPO and IPO) and RLHF. It proves that DPO/IPO requires global convergence assumption which is in general a very strong assumption while on the other hand RLHF only requires local coverage. This separation stems from the explicit KL constraint in the objective of RLHF while the implicit reward in contrastive algorithms can results in unbounded KL. Given the observation, a hybrid (offline + online) algorithm is proposed: the preference optimization remains offline DPO however it draws online samples to enforce KL constraints on DPO policy, recovering the statistical guarantee of RLHF while attains the practical computational strengths (w/o value and reward models) of DPO.", "questions": "- I might have overlooked, what is the intuition behind online KL evaluation? Is it possible to enforce such KL constraint purely in offline setting?", "limitations": "Overall, I am convinced this is a good submission with good technical quality. However, as I did not extensively follow the theoretical results on preference optimization, I am not able to comment on the novelty of this work, hence I give confidence score of 3. \n\nIt would also be beneficial if the authors could create a table summarizing previous works and highlighting their contributions in comparison to prior research.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Well written and easy to follow; understanding DPO and RLHF is important towards better preference optimization algorithms.\n\n- Good technical qualities and offer viable insights into DPO.\n\n- The proposed hybrid algorithm recovers the statistical guarantee of RLHF while keeps the practical computational strengths of DPO (w/o using value and reward models).", "weaknesses": "I only have a couple of minor comments:\n\n- Second sentence of L562 should be appended to L231 to make the proof sketch immediately clear.\n\n- Theorem E.1 could be relocated to Section 5 to be more self-contained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the postive feedback and we hope our rebuttal can be helpful to further demonstrate our contribution:\n\n> Second sentence of L562 should be appended to L231 to make the proof sketch immediately clear. Theorem E.1 could be relocated to Section 5 to be more self-contained.\n\nThank you for your suggestion for improving the writing of the paper. We will incorporate these changes in the final version which allows additional pages. \n\n> What is the intuition behind online KL evaluation? Is it possible to enforce such KL constraint purely in offline setting?\n\nThe reason for the additional online KL is exactly because purely offline methods can not enforce the KL constraints with only the offline data (Proposition 4.1). The intuition behind Proposition 4.1 is that, since DPO is modeling reward in the form of log ratio between policies, even though it is accurate over the offline data, the behavior of the policy outside the offline data is uncontrolled, which can not enforce the KL constraint, because in reverse KL is measured under the trained policy. In order to prove that offline data alone suffices to control the KL, one has to assume a condition similar to that the offline data covers all possible sequences of tokens (Theorem 4.1), which is unrealistic in most situations. \n\n> Comparison to prior research.\n\nThank you for your suggestions and we hope our rebuttal can help clarify our theoretical contributions. We will provide a Table comparing with the relevant previous theoretical RLHF works that we mentioned in the related work section in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2MED8971Zv", "reviewer": "Reviewer_MDLu", "summary": "This paper focuses on the optimization and learning methods for ``online'' RLHF and contrastive offline methods (DPO, IPO). The authors aim to understand the separation between these two type of methods, where they are different in terms of whether new responses can be sampled or not. The authors state that the difference in the reward parameterization is key to the separation and propose global coverage and local coverage to capture such a difference. \n\nThe theoretical insights also motivate a novel approach called Hybrid Preference Optimization (HyPO) that combines offline data with online samples, where the online samples are used to control the KL divergence. Empirical results are provided to verify the effectiveness of the proposed methods.", "questions": "see weakness part", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors study the theory of RLHF under the KL-regularized target, which is more close to the practice, as compared to many previous works using the non-regularized reward;\n- The notion of local coverage condition is novel in the literature and is very natural in the analysis of KL-regularized target. I appreciate the paper writing at the beginning of section 4, which is easy to follow and informative.  \n- Building open the notions of local convergence and global convergence, the authors show a clear separation between the offline algorithms like DPO and online RL-based algorithms. This aligns with the recent observations that the online algorithms outperform their offline counterparts with a large margin. \n- This paper takes a step further to study the differences between the DPO and RLHF. While the original DPO paper states that the learning of DPO is equivalent to the RLHF, the empirical results do not align with this. The discussion related to the parametrization of reward for assumption 4.4 explains such a difference in practice. \n- Overall, I feel that the story of this paper is complete. Lemma 4.1 and the discussion  around assumption 4.4 clearly show that the offline algorithms can search for the policy with a large KL (possibly due to the parameterization of reward). This not only aligns with the separation between global coverage condition and the local coverage condition, but also motivates the practical algorithmic design to explicitly control the KL divergence.", "weaknesses": "This is not a weakness but some clarification on the terminology. In the literature of RL theory, particularly for the preference learning paper, the learning (online exploration) is mentioned with querying the human preference so as to learn the $r^*$. In the setup of this paper, the online data is only used to compute the KL loss without querying the human feedback. Therefore, it is more related to an intermediate setting that we use the offline preference dataset to construct a proxy reward function and we are allowed to query the model to get new responses but not the human feedback. See the RSO [1], some discussions in [2] for proposing multi-step RSO, and some discussions in [3]. \n\n[1] Statistical rejection sampling improves preference optimization.\n[2] Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint\n[3] Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data.\n\nThen, it would be interesting to see the empirical results in terms of the ONLINE setting where we can query the new response and the new human preference signals because a lot recent works show that even with the DPO, the online framework outperforms the original one with a large margin. Moreover, this is standard in the PPO literature like Chat-GPT, Claude, and LLaMA2.\n\n[4] Training language models to follow instructions with human feedback\n[5] Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback\n[6] Llama 2: Open Foundation and Fine-Tuned Chat Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments and we address the reviewer's comment below:\n\n> Discussion on online data.\n\nThank you for pointing out the subtlety of the terminologies. We agree that HyPO indeed does not query any additional information from $r^\\ast$, which is different from the papers that the reviewer brought up and some related works that we discussed in the related work section. We agree with the reviewer that HyPO is different from the other online RLHF methods because we only have unlabeled online samples (i.e., the data is not labeled with a reward model). One can see it as a monte-carlo estimation of the reverse KL. We will make this distinction clear with additional discussion with the works mentioned by the reviewer. \n\nWe also believe that investigating the standard online setting has important empirical values. In our setting, if we use the reward model that is trained on the same offline dataset to label the online data, then theoretically this is equivalent to HyPO which avoids storing and querying the extra reward model. In Fig.1 in the supplemental material of the rebuttal, we compare the memory cost between DPO, HyPO and PPO (which requires the additional reward model), and we show that HyPO is almost as memory efficient as DPO, where PPO requires more than twice memory than DPO and HyPO (although it also requires to store a value function, but that still indicates the additional memory overhead of the reward model). That said, in the case where one has additional GPU memory, many recent and contemporary works [1,2 and many others] showed that empirically, such an online DPO (or iterative DPO) method indeed greatly improves over the offline DPO results, indicating the effectiveness of online reward information. We will make sure to add additional discussion on this topic in our final version. \n\n[1] RLHF Workflow: From Reward Modeling to Online RLHF. 2024\n\n[2] Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF. 2024\n\n\nWe also want to mention that in the general response section, we performed additional large-scale experiment on HyPO on finetuning Llama 3 8B instruct model, and we hope this additional empirical result can further demonstrate the effectiveness of online unlabeled data alone.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### General responses

We thank all reviewers for their positive and constructive feedback. In the general response we provide some additional empirical results, which will be incorperated in our final version of the paper. 

1. **Large-scale experiments:**
In response to Reviewer ewpX’s suggestion on experiments on larger model and additional dataset, we finetune Llama 3 8B instruct model on the Ultrafeedback dataset [1] and evaluate on AlpacaEval [2], MT-Bench [3] and Open LLM Leaderboard, the standard large-scale experiment setup for empirical RLHF papers [4,5]. Following [4], we only finetune the last 4 layers (same for the DPO baseline), and we trained on 8 A100 GPUs. For this experiment, we use RLOO [6] to optimize the online trajectory level KL with $k=2$ (number of repeated online generation). We summarize the results in Table 2 and 3, and we provide the hyperparameter for HyPO with RLOO in Table 4 of the rebuttal supplemental material. 

2. **Comparison with PPO and computational analysis:**
In response to Reviewer YCrR’s suggestion we updated our TL;DR result with comparison with PPO: the updated Table is in Table 1 of the supplementary material. Note that as we mentioned in the submission, there is still a gap between HyPO and PPO, and providing a more in-depth theoretical analysis of such gap is an interesting future direction. In addition, we performed a controlled experiment on the computational cost of each method. Following the setup of [4], we fix the computational resource on the same node with 8 A6000 GPUs, and we run each algorithm for a batch of 512 prompts of the TL;DR dataset, averaged over 50 batches. We present the result in Figure 1 of the supplemental material: although HyPO introduces more generation cost, but the time and memory cost of HyPO is still significantly lower than PPO. The hyperparameter of PPO in this experiment is recorded in Table 5 of the supplemental material.

We believe the additional empirical results can further validate our theoretical results, and the large-scale experiment can also have independent empirical contribution to the community. We appreciate the reviewers for their suggestions and we look forward to further feedbacks during the discussion period.

[1] Ultrafeedback: Boosting language models with high-quality feedback. 2023

[2] Length-controlled alpacaeval: A simple way to debias automatic evaluators. 2024

[3] Judging llm-as-a-judge with mt-bench and chatbot arena. 2024

[4]  Rebel: Reinforcement learning via regressing relative rewards. 2024

[5] SimPO: Simple Preference Optimization with a Reference-Free Reward. 2024

[6] Buy 4 REINFORCE samples, get a baseline for free! 2019

Meta-review: This paper studies the performance gaps between online and offline preference alignment methods, and uses the notions of strong and weak dataset coverage to provide explanations. A Hybrid PO algorithm is introduced to use online samples for calculating KL divergence, which gains some performance improvement over original DPO. All reviewers agreed that the story is complete and the contributions are solid, and I concur.
Decision: Accept (poster)