The Importance of Online Data: Understanding Preference Fine-tuning via Coverage

Authors: yuda  song
gokul  swamy
aarti  singh
j andrew bagnell
wen  sun
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 05:17:27

Abstract: Learning from human preference data has emerged as the dominant paradigm for fine-tuning large language models (LLMs). The two most common families of techniques -- online reinforcement learning (RL) such as Proximal Policy Optimization (PPO) and offline contrastive methods such as Direct Preference Optimization (DPO) -- were positioned as equivalent in prior work due to the fact that both have to start from the same offline preference dataset. To further expand our theoretical understanding of the similarities and differences between online and offline techniques for preference fine-tuning, we conduct a rigorous analysis through the lens of *dataset coverage*, a concept that captures how the training data covers the test distribution and is widely used in RL. We prove that a global coverage condition is both necessary and sufficient for offline contrastive methods to converge to the optimal policy, but a weaker partial coverage condition suffices for online RL methods. This separation provides one explanation of why online RL methods can perform better than offline methods, especially when the offline preference data is not diverse enough. Finally, motivated by our preceding theoretical observations, we derive a hybrid preference optimization (HyPO) algorithm that uses offline data for contrastive-based preference optimization and online unlabeled data for KL regularization. Theoretically and empirically, we demonstrate that HyPO is more performant than its pure offline counterpart DPO, while still preserving its computation and memory efficiency.

Main paper: 
 introduction: 
 Due to the difficulty of manually specifying reward functions for complex tasks [7] , preference-based learning has emerged as a critical component in the fine-tuning procedure for large language models (LLMs) [40, 30, 45, 44] . There are two predominant flavors of preference learning for LLMs: online reinforcement learning (RL) methods such as PPO [11, 30] and offline contrastive methods like Direct Preference Optimization (DPO) [33] and Identity Preference Optimization (IPO) [3] .Online RL methods usually follow the two-stage procedure prescribed in [30] : one first trains a reward model (classifier) on a fixed offline preference dataset before using it to provide reward labels for on-policy generations, which are then fed to a downstream RL algorithm like Proximal Policy Optimization (PPO) [36] . Since the reward model is learned from static offline preference data, to avoid over-optimizing the reward model [17] , one typically adds a reverse KL penalty to encourage the model to stay close to some reference policy. We will refer to this procedure as reinforcement learning from human feedback (RLHF) in this paper. While empirically performant, RLHF requires repeated querying of the reward model (which is often itself an LLM) as well as sampling from the current policy. In response to the computational expense and relatively complex nature of this procedure, purely offline methods like DPO [33] and IPO [3] have been proposed as alternative methods for preference fine-tuning. These methods do not need to fit separate reward models, instead opting to simply train the policy directly on the offline preference dataset via a ranking loss.Offline contrastive methods like DPO are usually derived via applying a reparameterization trick to the closed-form solution of the minimum relative entropy problem [63] that RLHF techniques attempt to approximate. Thus, several authors have described these methods as equivalent (at least in theory) to the standard RLHF procedure [33, 3] . However, recent (mostly empirical) work has contradicted this perspective: [43] find that online methods out-perform offline methods and attribute this fundamentally to on-policy sampling, [55] argues that the online RL methods produce an often desirable subset of the possible DPO loss minimizers, and [42] provide empirical support for the claim that online and contrastive training provide orthogonal benefits. However, a rigorous theoretical separation is still lacking in the pre-existing literature, which motivates our key questions:What is the statistical separation between the online RLHF method and offline contrastive methods? What causes this separation and what does it imply?To answer these questions, we focus on the coverage of the preference dataset, a key concept that is widely used in RL [22, 28, 58] for analyzing the impact of offline or exploratory data distributions.Through the lens of coverage of the offline preference dataset, we make the following contributions:• We prove that the global coverage condition [28] , the strongest possible coverage condition in RL, is necessary for offline contrastive algorithms like DPO to converge to the optimal policy. In contrast, we identify a weaker local coverage condition that is sufficient for online RLHF algorithms, thus provably separating the two types of algorithms. The separation is due to the difference in reward modeling and on/offline regularization -in short, there is no free lunch from bypassing explicit reward learning and online rollouts. As global coverage might sometimes be violated in practice, our separation result can perhaps explain why RLHF works better than offline methods [42, 43, 56 ]. • Although offline contrastive methods are derived from a reverse-KL objective, we prove that the policies trained via offline methods can still have infinite reverse-KL in the partial coverage setting.In contrast, we show that RLHF can always control the reverse KL via directly optimizing reverse KL using online samples. This means that on realistic problems, RLHF has stronger guarantees for remaining close to the reference policy than offline contrastive methods. • We propose Hybrid Preference Optimization (HyPO) to address the deficiencies of offline contrastive methods while maintaining some of their computational simplicity. HyPO is a hybrid RL algorithm [51, 39] where offline data is used for the DPO objective while online samples are used to explicitly control the reverse KL divergence to the reference policy. We empirically demonstrate that HyPO outperforms DPO, on the TL;DR summarization task [40] on all metrics including both the GPT4 win-rate and the reverse KL divergence to the reference policy, and on general chat benchmarks such as AlpacaEval 2.0 [15] , trained with the UltraFeedback dataset [14] . In addition, HyPO also mitigates the overfitting issues observed in the offline constrastive based methods [43] . • We provide an explanation of why RLHF and offline contrastive methods decrease the probability of both preferred and rejected responses during training. In particular, under our function approximation-based global coverage condition, we show that such behavior is actually desirable for DPO and RLHF policies to extrapolate and generalize to optimal actions that do not appear in the training dataset. However, without function approximation, algorithms like DPO can mistakenly increase the likelihood of sub-optimal actions. This establishes the importance of function approximation for the success of the algorithms such as DPO.Taken together, our results establish the critical role coverage plays in terms of convergence properties of preference learning algorithms as well as in the design of new, performant empirical approaches. 
related work: 
 Preference Fine-Tuning (PFT). As discussed in the introduction of our work, there are two major paradigms for preference fine-tuning of LLMs. The first one, online RL methods [30] , proposes to first train a reward model (classifier) to predict human preferences, followed by running an RL method to optimize this learned reward function. While PPO [36] is the most popular RL algorithm used in the online RLHF framework by far [40, 30, 45] , more recent work by [1] shows that simpler online RL algorithms like REINFORCE [48] also work well. The second class of methods, offline contrastive techniques [60, 33, 3] , avoid explicit reward modeling and directly optimize their objective on the offline preference dataset. Recently there are hybrid methods that combine offline preference data with online preference labels [52, 19, 35, 3] -we leave extending our analysis to this setting to future work. Throughout our paper, we assume for simplicity of analysis that preferences are generated by an underlying utility function and therefore contain no intransitivities [29, 41] .Understanding PFT. Prior work has studied different parts of the standard RLHF recipe [17, 23, 38, 16] and the impact of preference data quality [37] . In our work, we instead take a converge-based perspective on the relationship between online RL methods and offline contrastive methods. Although derived from the same minimum relative entropy objective [63] and perceived as equivalent by some early work [33, 3] , more recent work has started to unravel the distinctions between these two classes of methods. [43] repeatedly observe better performance from online rather than offline methods and after rigorously validating a variety of hypotheses, conclude that on-policy sampling is indispensable for ensuring a high quality policy. [42] perform an in-depth study of the effects of preference data, contrastive losses, and on-policy sampling and conclude that a combination of contrastive losses and interactive training is most preferable in practice. [55] also observe better performance from online PPO than from offline DPO and argue this is because the former is able to eliminate a larger set of policies that are undesirable from the perspective of the later. We supplement these mostly empirical observations with a rigorous theoretical explanation for the observed behavior through the lens of dataset coverage, as well as designing an algorithm that addresses the key weaknesses of offline contrastive approaches.We defer additional related works to Appendix A. 
preliminaries: 
 Following a wide range of recent works [33, 3] , we consider the RLHF problem in the contextual bandit formulation [25] . This is a reasonable simplification, as one can consider the generated sequence of tokens as one single action, due to the fact that the states are the generated tokens, and the dynamics are deterministic. We denote the context (prompt) space as X , and the action (response) space as Y. Note that due to the finiteness of the possible tokens, the action space is finite but combinatorially large. We use ρ ∈ ∆(X ) to denote the distribution of the prompts, and π : X → ∆(Y) as policies (LLMs) that map prompts to a distribution of responses. We also consider the reward function class R : X × Y → R, which assigns a reward to each context-response pair.We assume access to a reference policy π ref , which is usually referred to as the policy learned using supervised data when training the LLM, that needs to be further fine-tuned to align with human values. An offline preference dataset is collected in the format of D = {x, y + , y -} triplets: given context x ∼ ρ, the preference policy samples two responses y 1 , y 2 ∼ µ(• | x), where µ is the offline response distribution. Previous works assume either µ to be the same distribution as π ref [33] or different offline distribution [3, 35, 18] . Then, y 1 is labelled as y + (thus y 2 as y -) with probability p * (y 1 ≻ y 2 | x), where p * is defined by the Bradley-Terry model [6] :p * (y 1 ≻ y 2 | x) = exp(r * (x, y 1 )) exp(r * (x, y 1 )) + exp(r * (x, y 2 )),where r * is the human's implicit reward function. Note that this rules out intransitive preferences [41, 29] . Throughout the paper, we will make the following assumption on the reward function:Assumption 3.1 (Boundedness of the reward). ∥r * ∥ ∞ ≤ R.In many previous works, this formulation has been the canonical way to model the preference data in the RLHF literature [11, 33, 3] . The goal is to learn a policy π to maximize the objective J(π), whereEQUATIONi.e., we want to both maximize the human implicit reward, and not deviate too much from the reference policy. We denote the optimal policy π * ∈ argmax π∈Π J(π). Here we callKL(π(• | x)||π ref (• | x))reverse KL because π -the policy to be optimized, appears first. We will callKL(π ref (• | x)||π(• | x)) forward KL.By the definition of KL, we have Definition of reverse KL:EQUATIONNote that the expectation in reverse KL is under π, indicating that evaluating and optimizing reverse KL requires drawing online samples from π. In contrast, evaluating forward KL only requires offline samples drawn from π ref . As we will show, this key difference between reverse KL and forward KL plays an important role of separating online RLHF and offline contrastive methods such as DPO. In this paper, we consider two types of algorithms: online RL-based algorithms, and offline contrastive-based algorithms.Online RLHF Algorithms. We consider algorithms such as [11, 1] as the online RL based methods. We abstract these algorithms as the following procedure: the algorithm performs the following twostage procedure: one first trains a reward model r that minimizes the Bradley-Terry loss1 EQUATIONand perform policy optimization (such as PPO [36] ) to optimize the policy with the reward model r:π rlhf ∈ argmax π E x∼D E y∼π(•|x) [ r(x, y)] -βKL(π(• | x)||π ref (• | x)) .However, this policy optimization step requires extensive online sampling, and possibly training an additional critic model (e.g., PPO), in addition to the reward model and policy.Offline Contrastive Algorithms. To circumvent the above-mentioned computational burden, several purely offline contrastive-based methods (i.e., without RL) have been proposed. In this paper, we focus on the following two most representative methods. The first is Direct Preference Optimization (DPO) [33] , where the objective is π dpo ∈ argmax π ℓ dpo (π) withEQUATION)Another offline contrastive method we will discuss in our paper is Identity Preference Optimization [3] , but we will defer its technical details to the appendix.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: We clearly state all our assumptions in all theorem statements, and the proofs can be found in the appendix.Guidelines: • The answer NA means that the paper does not include theoretical results. • All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
offline contrastive methods require a stronger coverage condition than online rl methods: 
 We start by introducing the mathematical formulation of the coverage framework. The strongest coverage condition is the following global coverage condition [28] : we say any offline distribution µ covers a policy π if we have max x,y:ρ(x)>0 π(y|x) µ(y|x) ≤ C glo . Throughout this section, we will adopt the setting where µ = π ref [33] . Formally, we assume the following condition: Assumption 4.1 (Global Coverage). For all π, we havemax x,y:ρ(x)>0 π(y | x) π ref (y | x) ≤ C glo .For the coverage terms, we always adopt the convention that 0 0 = 0. Note that one sufficient condition for this assumption is that, for any prompt x, and any token sequence y, we haveπ ref (y | x) ≥ 1/C glo .As has been recognized in the offline RL literature, global coverage is a strong assumption, and efforts have been made to circumvent this assumption with more relaxed coverage conditions [46, 10, 58] . In this paper, we will consider the following partial coverage assumption that is weaker than Assumption 4.1: Assumption 4.2 (Local KL-ball Coverage). For all ε kl < ∞ and all policy π such thatE x∼ρ [KL(π(• | x)||π ref (• | x))] ≤ ε kl , we have max x,y:ρ(x)>0 π(y | x) π ref (y | x) ≤ C ε kl .Note that C ε kl depends on ε kl . This coverage notion is relatively new in the RL literature and only appeared in previous analysis of RLHF algorithms [9] . We call this local coverage condition since it only requires π ref to cover the policies that is within some KL-divergence ball centered at π ref .The intuition of this assumption is, for any algorithm that can control the reverse KL of the output policy, we can leverage the coverage condition to relate the error under the output policy to its error under the offline distribution, and thus guarantee its performance. Finally, we note that since the policies with bounded KL is a subset of all policies, for a fixed π ref , we always have C ε kl ≤ C glo .Remark 4.1. Taking a closer look at Assumption 4.2, we can see that this assumption is always true in the sense that for any policy with ε kl < ∞, max x,y:ρ(x)>0 π(y|x) π ref (y|x) < ∞, i.e., C ε kl < ∞, for any ε kl . However, while being bounded, C ε kl can be large. Indeed a simple calculation can show that max x,y:ρ(x)>0 π(y|x) π ref (y|x) can be as large as max x,y:π(y|x)>0 exp ε kl π(y|x) . This can be undesirable because this suggests bounded reverse KL itself is not enough to guarantee optimality: the error can have an exponential amplification when switching from π ref to π. Thus this motivates Assumption 4.2, which assumes that C ε kl is reasonably small.In what follows, we will show that the global coverage assumption (Assumption 4.1) is necessary for offline contrastive-based algorithms such as DPO and IPO, but partial coverage assumption such as Assumption 4.2 is sufficient for online RL based algorithms. This establishes a separation between the two types of algorithms. We emphasize this theoretical separation explains why in practice online methods is less prone to problems such as reward hacking and producing out-of-distribution responses that are due to dataset with insufficient coverage.Failure of DPO Under Partial Coverage. Now we show that if the strong coverage Assumption 4.1 breaks, then DPO can not guarantee any performance with respect to the objective function Eq. ( 1). The intuition is based on a rather common observation of the DPO algorithm: the DPO policy π dpo may generate out of distribution responses, while in contrast, RLHF does not generate responses outside of the support of π ref due to online reverse-KL constraint. For example, [55] provides a construction where π dpo chooses a response where RLHF policy assigns 0 mass, thus proving that RLHF policies are a subset of DPO policies.However, such construction assumes that the reward learning procedure of DPO makes arbitrarily large errors. Also, previous constructions assume deterministic preference, which is only true if the underlying reward function is unbounded. This violates the natural assumption of Assumption 3.1. In the following, we relax these constraints and thus show that DPO fails to guarantee any performance in a rather strong sense. Concretely, DPO constructs the following implicit reward class with the policy class Π: R dpo = β log π(y|x) π ref (y|x)Z(x) | π ∈ Π , where Z(x) is a partition function that maps context to a real number and is independent of y. Plugging this formulation into the BT loss (Eq. ( 3)) recovers exactly the DPO loss (Eq. ( 4)) as the partition functions are canceled. Now we can characterize the returned policy by DPO as exactly whose corresponding reward function is accurate in distribution: Assumption 4.3 (In Distribution Reward Learning). We assume the DPO policy π dpo satisfies that:E x,y∼ρ•π ref β log π dpo (y | x) π ref (y | x)Z(x) -r * (x, y) 2 ≤ ε dpo .Note that this is a rather strong assumption for BT loss -by Lemma B.2, at best one can only hope: for any learned reward function r, for each context x, there exists a constant c(x) such thatEQUATIONi.e., the reward model predicts the human reward up to a gap that is independent of y. This is due to the fact that BT loss only requires the reward function to capture the relative difference, or in other word, any constant shift (with respect to context) in the reward will be canceled in the BT loss. However, for the rest of the section, we will make the stronger learning assumption that the gap c(x) = 0 (such as in the case of Assumption 4.3). Previous counterexamples analysis violates this assumption, but we will show that even under this strong assumption, DPO still can not guarantee any performance. Proposition 4.1. Denote π ref as any reference policy such that Assumption 4.1 breaks. Let Π dpo be the set of DPO returned policies such that Assumption 4.3 holds. Then there exists policy π ∈ Π dpo such that J(π) = -∞.Proof sketch. Without loss of generality, we consider a promptless setting, and assume that the response space is Y = {y 1 , y 2 , y 3 }. Again without loss of generality, we assume π ref only covers y 1 and y 2 , and thus Assumption 4.1 breaks. We assume partition function Z = 1 for all π but we will be rigorous in the formal proof. Then consider the following policy π such thatβ log π(y 1 ) π ref (y 1 ) = r * (y 1 ) - √ ε dpo , and β log π(y 2 ) π ref (y 2 ) = r * (y 2 ) - √ ε dpo ,One can check π satisfies Assumption 4.3. Now consider the optimal policy π * (y i ) = π ref (y i ) exp 1 β r * (y i ) , for i ∈ {1, 2}, and π * (y 3 ) = 0. Since π * (y 1 ) + π * (y 2 ) = 1, combining everything we get π(y 3 ) > 0, which implies KL(π||π ref ) is unbounded, thus we complete the proof.One can first relate the above construction to the parital coverage assumption Assumption 4.2: since the policy π considered in the proof has unbounded reverse KL with respect to π ref , thus it is not in the KL-ball of ε kl around π ref , which implies that Assumption 4.2 is not sufficient for DPO. Next we show that global coverage is necessary for the IPO algorithm.Failure of IPO Under Partial Coverage. To show that the global coverage is necessary for IPO, we can even assume a stronger in-distribution learning guarantee, that is, the returned policy achieves the smallest error on its population loss in distribution. Proposition 4.2 (Informal). Denote π ref as any reference policy such that Assumption 4.1 breaks. Let Π ipo be the set of IPO returned policies such that it is the minimizer of in-distribution error on its population loss. Then there exists policy π ∈ Π ipo such that J(π) = -∞.We defer the detailed setup and formal version to Appendix D, but the construction for the above proofs share the same intuition: the reverse KL term in the objective function can be unbounded. For offline contrastive-based algorithms, the KL regularization is only enforced under the data distribution, and thus the algorithm can not guarantee bounded reverse KL if the reference policy does not cover the response space well. Although we only showed counterexamples for DPO and IPO, we conjecture that the same intuition holds for other offline contrastive-based algorithms. One natural question at this point would be: how about the forward KL? Not surprisingly, the forward KL for DPO (but we conjecture for other offline constructive-based methods as well) is vacuously large, and we formalize this result in Appendix C.2. Remark 4.2. The folklore that DPO is equivalent to RLHF is often based on some assumption that is much stronger than Assumption 4.3: it requires that the learned policy has a point-wise accuracy guarantee β ln(π dpo (y|x)/π ref (y|x)) = r * (x, y) for all x, y. Such a point-wise guarantee is unrealistic in reality and does not hold in general in the supervised learning sense. The in-distribution style guarantee in Assumption 4.3 is the best one could hope for from a supervised learning algorithm.After showing that global coverage is necessary for DPO to guarantee any performance, we now show that it is sufficient for the performance guarantee. Theorem 4.1. Let π ref be any reference policy such that Assumption 4.1 holds. For any policy π dpo such that the event in Assumption 4.3 holds, we have thatJ(π * ) -J(π dpo ) = O(C glo √ ε dpo ).Proof. By Lemma B.1, we haveJ(π * ) -J(π dpo ) ≤ E x∼ρ E y 1 ∼π * (•|x),y 2 ∼π dpo (•|x) r * (x, y 1 ) -r dpo (x, y 1 ) -r * (x, y 2 ) + r dpo (x, y 2 ) ≤ E x∼ρ E y 1 ∼π * (•|x),y 2 ∼π dpo (•|x) (r * (x, y 1 ) -r dpo (x, y 1 ) -r * (x, y 2 ) + r dpo (x, y 2 )) 2 ≤ C 2 glo E x∼ρ E y 1 ,y 2 ∼π ref (•|x) (r * (x, y 1 ) -r dpo (x, y 1 ) -r * (x, y 2 ) + r dpo (x, y 2 )) 2 ,and we can complete the proof by plugging in the error guarantee from Assumption 4.3.Note that as the proof suggests, the result holds with the more general reward learning guarantee as in Lemma B.2 -one only needs to be accurate in predicting the relative rewards between response pairs.Finally, we contrast the previous negative results in Section 4.1 for offline contrastive-based algorithms to a positive result for online RL-based algorithms, under the partial coverage setting. We will show that in general global coverage is not necessary for RLHF, i.e., it can guarantee performance under partial coverage. In fact, one might still be able to show an impossibility result for RLHF under partial coverage, by reusing the same counterexample as in the previous section (c.r., Proposition 4.1). Concretely, as long as the learned reward r(y 3 ) → ∞, π rlhf (y 3 ) will be 1 and thus the reverse KL will be unbounded. However, this is a rather unrealistic scenario, as the construction requires a reward model (e.g., a neural network) to output an unbounded value. Thus this motivates the following assumption:Assumption 4.4. For all learned reward model r from the reward model class, we have that∥ r∥ ∞ ≤ R ′ .At this point, one might ask why a similar assumption is missing for the offline contrastive-based analysis, since in Remark 4.2 we argued that a point-wise learning guarantee is unrealistic but Assumption 4.4 is indeed also a point-wise boundedness assumption. The reason lies in the different construction of the model class r for those algorithms: for DPO and IPO, the reward model is constructed asr dpo = β log π π ref •Z, and there is no natural function class for π such that point-wise assumptions such as the one in Remark 4.2 or Assumption 4.4 holds. In contrast, post-processing such as clipping, offline normalization and on-the-fly normalization of rewards is standard in practice, which means the policy will always witness bounded rewards [8, 9, 18, 1] during online RL training (e.g., PPO). As we will show in the following, the difference in the reward function (which is tied to the offline vs. online nature of the algorithms) can explain the different coverage requirement of the algorithms. Note that we use the same in-distribution reward learning assumption for both types of methods.To relate to Assumption 4.2, we first show that the reverse KL divergence of the RLHF policy is always bounded under Assumption 4.4.Lemma 4.1. Suppose that Assumption 4.4 holds. Then for any RLHF policy π rlhf , we have thatKL(π rlhf ||π ref ) := E x∼ρ E y∼π rlhf (•|x) log π rlhf (y | x) π ref (y | x) ≤ 2R ′ β .Then we can show that the RLHF algorithm can guarantee performance under partial coverage:Theorem 4.2. Suppose that Assumption 4.4 holds. Then for any reference policy π ref for which Assumption 4.2 holds with ε kl = 2R ′ β , and any RLHF policy π rlhf with r such that (c.r. Assumption 4.3) E x,y∼ρ•π ref (r * (x, y) -r(x, y))2 ≤ ε reward , we haveJ(π * ) -J(π rlhf ) ≤ O(C ε kl √ ε reward ).Conditioned on Lemma 4.1, the proof of this theorem is similar to that of Theorem 4.1 so we defer it to Appendix D. Similar to Theorem 4.1, we note that Theorem 4.2 holds under a weaker reward learning guarantee as in Lemma B.2. We also remark that as long as ε kl is finite, C ε kl is finite, so the bound is never vacuous. Since C ε kl ≤ C glo for all ε kl , it indicates the regret bound of RLHF is never worse and can be much better than the regret bound of DPO. Combining Theorem 4.1 and Theorem 4.2, we complete the separation result between offline contrastive methods and online RL methods.A natural question at this point could be: can we further relax the local KL-ball coverage condition in Assumption 4.2 to a single-policy coverage condition, i.e., just assuming max x,y π * (y|x)/π ref (y|x) ≤ C? Prior work [59] shows that with explicit pessimism, it is possible. However, using pessimism makes the algorithm from [59] Sample a minibatch of offline data D off := {x, y + , y -} ∼ D.Compute DPO loss ℓ dpo := x,y + ,y -∈D off log σ β logπ θ t-1 (y + |x) π ref (y + |x) -β log π θ t-1 (y -|x) π ref (y -|x) . 4:Sample (unlabeled) online data D on := {x, y} where x ∼ D, y ∼ π θt-1 (x).Compute ℓ kl := x,y∈Don log(π θt-1 (y|x)) • sg log(π θ t-1 (y|x)) (π ref (y|x)).6:Update θ t = θ t-1 + α • ∇ θt-1 (ℓ dpo -λℓ kl ). return π T .Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We provide pseudocode, hyparameter table and code in this submission. Guidelines: • The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
hybrid preference optimization: regularizing offline learning with unlabeled online samples: 
 In this section, we will provide a practical algorithm that bridges the gap between the offline contrastive-based algorithms and the online RL-based algorithms. As we see in the previous sections, the difference between the two types of algorithms is their reward model parametrization, and whether to perform online rollouts. In the following, we will show that these two properties are in fact tightly intervened with each other.Here we will focus on the DPO algorithm. One way to fix the issue of the unbounded reverse KL of DPO (which is caused by the unbounded reward model class) is to consider the following ideal procedure: at the beginning of the algorithm, we first go through the policy class Π, and then we filter out all the policies such that KL(π||π ref ) ≥ 2R ′ β , where R ′ is the boundedness of the reward function class for RLHF. Now applying the same analysis of Theorem 4.2, we can show that this revised DPO algorithm can guarantee performance under the partial coverage assumption, because now the Lemma 4.1, a sufficient condition for Theorem 4.2, is explicitly enforced by the constraints. We defer the detailed statement and analysis to Appendix F.1.However, such a filtering procedure is not possible in practice, but we can instead consider the following constrained optimization problem: we call the definition of DPO loss in Eq. ( 4), we want to solveEQUATIONusing the KKT conditions, we can show that the following Lagrangian form is equivalent to Eq. ( 6):EQUATIONwhere λ is the Lagrange multiplier. However, in reality, since we do not know the exact value of R ′ , we can consider setting λ to be a hyperparameter. We present the pseudocode in Algorithm 1. Note that due to the reverse KL term, the Hybrid Preference Optimization (HyPO) algorithm optimizes Eq. ( 7) via both offline and online samples where the offline samples are used for constructing and optimizing ℓ dpo (here σ denotes the sigmoid function), and the online samples y ∼ π(• | x) are for KL (i.e., ℓ kl ). Note that regularizing with reverse KL via online samples is widely used in online RLHF (e.g., PPO [40] , APA [62] , REBEL [18] ). Here sg refers to the stop gradient operation, which is a common practice in optimizing reverse KL in the LLM fine-tuning setting [30, 47] . Finally, previous iterative RLHF methods [53] can be interpreted as hybrid methods as well, but they require labeling online samples from an additional reward model while HyPO only requires unlabeled online samples.Summarization. Our first experiment is on the TL;DR dataset [40] . Our experiment setup mostly follows [18] : we use a maximum context length of 512 and a maximum generation length of 53.We use Pythia 1.4B and Pythia 2.8B [5] as the pre-trained model. For the supervised fine-tuning (SFT) model, we train it over 1 epoch of the dataset with human reference responses as labels. We train the reward model on top of the SFT over 1 epoch of preference data. Both HyPO and DPO are trained over 1 epoch of preference data with Low-rank Adaptation (LoRA) [21] . We defer more experiment details in Appendix F. We repeat the experiment for 3 random seeds and plot the median and the shaded areas denote the min and max over the 3 repetitions.We summarize the results in Table 1 : HyPO outperforms DPO in terms of GPT4 win-rage and reverse KL. Particularly, the significant reduction in reverse KL implies the impact of including a reverse KL term explicitly into the DPO objective. While comparing with PPO (e.g., Table 1 in [18] ), HyPO's performance is still lower in winrate, HyPO does preserve the key advantages of DPO over PPO: we avoid training additional reward model and a value network.General Chat. In the general chat setting, the model is required to produce a response y given user instruction x. We again follow the experiment setup in [18] , where we finetune the Meta-Llama-3-8B-Instruct [27] model on the ultrafeedback dataset [14] . Due to the computation constrain, we follow the setup in [18] where we only train the last 4 layers of the network for both HyPO and DPO.For evaluation, we use the common metrics including AlpacaEval 2.0 [15] , MT-bench [61] and Open LLM leaderboard tasks: MMLU [20] , GSM8K [13] , Arc [12] , TruthfulQA [26] and HellaSwag [57] . We provide the results for AlpacaEval and MT-bench in Table 2 , and the the results of the remaining tasks can be found in Table 3 . HyPO Mitigates Overfitting in Contrastive Methods. Since the offline contrastive based methods only work with a static offline dataset, the overfitting issue has been observed [43] . In our last experiment, we show that HyPO can effectively address the overfitting issue by leveraging the unlabeled online data. We follow the setup of the summarization task with Pythia-2.8B base model. We train DPO and HyPO for 5 epochs respectively, and evaluate on the first 300 data in the validation dataset. We plot the validation KL in Figure 1 : we observe that HyPO is better at preventing the deviation from the reference policy caused by overfitting from training on excessive epochs, even though the methods theoretically both have KL regularization to the reference policy.6 Function Approximation Coverage: Can Fine-tuned Policies Extrapolate?Our final result is a theoretical explanation of the extrapolation behavior of preference fine-tuning algorithms under the global coverage assumption in the function approximation (FA) setting. The extrapolation behavior refers to the phenomenon of RLHF algorithms (e.g., DPO) can improve SFT models despite the fact that during training the policies assign decreasing likelihood to both the preferred and rejected responses (i.e., they must increase the likelihood of responses outside of the training data) [31] .A previous attempt [32] to explain this behavior is based on the assumption that the responses from the reference policy have the same distribution as the preferred responses from the dataset, i.e., y+ ∼ µ d = y ∼ π ref .However, as mentioned in Section 3, more realistically, one should assume that y ∼ µ d = y ∼ π ref sinceit is more natural to use the reference policy to generate pairs of responses to collect labels; or even more generally by considering supp(D) ⊂ supp(π ref ). The latter is common in practice, for example, the dataset is often precollected, or the reference policy might have a small mass on some responses, so with a high probability they are not sampled during the data collection process.In the following, we illustrate this behavior using linear function approximation: Definition 6.1 (Linear function approximation). Consider the promptless setting with response space Y. For all y ∈ Y, the ground truth reward r * (y) = w ⊤ ϕ(y), where w ∈ R d is a universal linear weight vector and ϕ : Y → R d is a d-dimensional feature map. In addition, all policies are parametrized as softmax linear policies, i.e., π(y) ∝ exp(w ⊤ π ϕ(y)). The feature map ϕ is known to the learner, and w is unknown. Finally, let y * = argmax y∈Y r * (y) be the optimal action.In general, we should not expect the offline dataset to contain the optimal action y * under all situations. We show that thanks to the linear function approximation and the dataset coverage, DPO has hope to extrapolate correctly, i.e., it can increase the model's likelihood of the optimal action while decreasing the likelihood of both the preferred and rejected actions from the offline data: Proposition 6.1. Under linear function approximation (Definition 6.1), there exists dataset collected from distribution µ that does not cover y * , i.e., µ(y * ) = 0, but has global coverage in the linear function approximation setting [54] : let Σ µ = E y∼µ ϕ(y)ϕ(y) ⊤ , then for all π, E y∼π ∥ϕ(y)∥ 2 Σ -1 µ ≤ C π . Then DPO will return a policy π dpo such that π dpo (y * ) > 0, but π dpo (y) ≤ π ref (y) for all y in the offline data support, i.e., µ(y) > 0.We defer the proof to Appendix D.3. The above result shows when the training dataset together with the function approximation allow the learned function to generalize (e.g., learn a function that can predict well on test examples beyond the training data -a property supervised learning can have), algorithms like DPO can extrapolate correctly, i.e., they can push up the likelihood of the optimal responses outside of the training data while pushing down the likelihood of all the responses in the training data. Although it is not possible to show that extrapolation is guaranteed to always happen: suppose in the offline dataset, there is an action y ′ whose feature ϕ(y ′ ) is almost identical to ϕ(y * ), then in the finite sample case, with small learning error the final policy might increase the probability of y ′ instead of extrapolate to y * , but in this case the policy is still near optimal. That said, our construction in the proof is general enough to explain non-edge-cases under function approximation.To validate our theory result, in Appendix E we perform a synthetic experiment on global coverage with linear function approximation.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We use public data on hugging face, and we submit the code for this submission.Guidelines: • The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We provide all the training details and hyperparameters.Guidelines: • The answer NA means that the paper does not include experiments. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
discussion: 
 There are a few limitations of our work: 1) our theoretical analysis only considers the statistical perspective of each algorithm, but we believe our result is complementary to the other work that considers the optimization perspectives [42] . 2) we only conduct experiments on limited models and benchmarks. 3) The experiment result shows that HyPO's performance is still below the one of online RLHF: this might suggest that our theory does not fully explain the benefit of all the components of online RLHF. For example, one hypothesis is that the learn reward function may have better generalization ability. 4) It is not clear that the KL-ball coverage is necessary for online RL-based methods. However, as we discussed, since a bounded reverse KL might still induce exponential error amplification, we conjecture that at least the single policy coverage [58] is not sufficient for online RLHF-based methods that use reverse KL. We believe these limitations lead to several interesting further directions. Finally, our method may not explicitly address the potential hallucinations or toxic behavior of LLMs, which is a common shortcoming of general-purpose fine-tuning algorithms.Extrapolation Behavior of PFT. Recent work [56, 31, 32] has observed an interesting effect of the DPO procedure: a simultaneous decrease in the likelihood of both preferred and rejected responses. This behavior is surprising at first glance because one would expect that DPO will increase the likelihood of preferred responses and decrease the likelihood of rejected responses. We provide a rigorous statistical explanation of this behavior and show that this behavior is natural when the offline preference data only contains sub-optimal responses but the function approximation allows DPO to extrapolate and generalize to the correct optimal responses. This highlights the role of function approximation in the success of offline contrastive based methods.Coverage. We analyze online RLHF and offline contrastive-based methods via the concept of coverage. Coverage measures how well an offline (data) distribution covers the support of the policy of interest, which has been the key technical tool in offline RL [28, 49, 46, 58] , offline-online RL [34, 51, 39, 2] and online RL [22, 4, 50] . The data coverage plays an important role in our analysis since both online RLHF and offline contrastive-based methods rely on an offline preference dataset for learning.Lemma B.1 (Objective decomposition). Let J(π) be the objective function defined in (1), and for reward function r, we letEQUATIONthen we haveJ(π * ) -J(π) ≤ E x∼ρ E y 1 ∼π * (•|x),y 2 ∼π(•|x) r * (x, y 1 ) -r(x, y 1 ) -r * (x, y 2 ) + r(x, y 2 ) .Proof. We haveJ(π * ) -J(π) =E x∼ρ E y∼π * (•|x) [r * (x, y)] -βKL(π * (• | x)||π ref (• | x)) -E x∼ρ E y∼π(•|x) [r * (x, y)] + βKL(π(• | x)||π ref (• | x)) =E x∼ρ E y∼π * (•|x) [r * (x, y)] -βKL(π * (• | x)||π ref (• | x)) -E x∼ρ E y∼π(•|x) [r * (x, y)] -βKL(π(• | x)||π ref (• | x)) + E x∼ρ E y∼π(•|x) [r(x, y)] -βKL(π(• | x)||π ref (• | x)) -E x∼ρ E y∼π(•|x) [r(x, y)] -βKL(π(• | x)||π ref (• | x)) ≤E x∼ρ E y∼π * (•|x) [r * (x, y)] -βKL(π * (• | x)||π ref (• | x)) -E x∼ρ E y∼π * (•|x) [r(x, y)] -βKL(π * (• | x)||π ref (• | x)) + E x∼ρ E y∼π(•|x) [r(x, y)] -βKL(π(• | x)||π ref (• | x)) -E x∼ρ E y∼π(•|x) [r * (x, y)] -βKL(π(• | x)||π ref (• | x)) =E x∼ρ E y∼π * (•|x) [r * (x, y) -r(x, y)] -E x∼ρ E y∼π(•|x) [r * (x, y) -r(x, y)] ,where the inequality is due to Eq. ( 8). To complete the proof, note that [9] ). Assume that r * is bounded, let R be the reward function class, and Let r = argmin r∈R Êx,y + ,y -∼D log exp(r(x, y + )) exp(r(x, y + )) + exp(r(x, y -)) , then we have with probability at least 1 -δ thatE x∼ρ E y∼π * (•|x) [r * (x, y) -r(x, y)] -E x∼ρ E y∼π(•|x) [r * (x, y) -r(x, y)] =E x∼ρ E y 1 ∼π * (•|x),y 2 ∼π(•|x) [r * (x, y 1 ) -r(x, y 1 )] -E x∼ρ E y 1 ∼π * (•|x),y 2 ∼π(•|x) [r * (x, y 2 ) -r(x, y 2 )] =E x∼ρ E y 1 ∼π * (•|x),y 2 ∼π(•|x) r * (x, y 1 ) -r(x, y 1 ) -r * (x, y 2 ) + r(x, y 2 ) . Lemma B.2 (Lemma C.2 fromE x,y 1 ,y 2 ∼µ•π ref r * (x, y 1 ) -r * (x, y 2 ) -r(x, y 1 ) + r(x, y 2 ) 2 ≤ cκ 2 log(|R|/δ) N ,where κ measures the non-linearity of the link function, and c is a constant, N := |D| is the size of the offline dataset.In this section we give detailed technical details for IPO, and the negative results for IPO under partial coverage. Recall that the empirical objective of IPO is is π ipo ∈ argmin π ℓ ipo (π), whereℓ ipo (π) = E x,y + ,y -∼D log π(y + | x)π ref (y -| x) π(y -| x)π ref (y + | x) - β -1 2 2 .The empirical objective is derived from the following population lossEQUATIONwhereh π (y 1 , y 2 ) = log π(y 1 )π ref (y 2 ) π(y 2 )π ref (y 1 ) ,and I(y 1 , y 2 ) is a Bernoulli random variable with parameter p = p * (y 1 ≻ y 2 ), where here p * can be any underlying human preference (that is not necessarily parametrized by the Bradley Terry model).To show the negative result, we can make the following learning assumption: Assumption C.1 (In distribution guarantee for IPO). We assume that the returned policy π ipo satisfies thatπ ipo = argmin π∈Π ℓ ipo (π),i.e., the returned policy π ipo induces the smallest possible in-distribution error on its population loss.With the setup, we can state and prove the formal version of the result: Proposition C.log π ipo (y 1 ) π ipo (y 2 ) = p * (y 1 ≻ y 2 )/β,and thus we have the relation thatπ ipo (y 1 ) = π ipo (y 2 ) • exp(p * (y 1 ≻ y 2 )/β).Let π ipo (y 2 ) = α ∈ (0, 1], then for any α such that π ipo (y 3 ) = 1 -(1 + exp(p * (y 1 ≻ y 2 )/β))α > 0, we will have that KL(π ipo ||π ref ) is unbounded, and thus we complete the proof.In this section, we show that in the worst case, the forward KL of DPO is vacuously large. We first see how we can relate the forward KL divergence of the DPO policy with the reward learning guarantee. Consider any DPO policy π dpo and its corresponding reward model r dpo . By construction of the DPO algorithm, we have, for any x, y pair that is covered in the dataset, π dpo (y| x) = π ref (y|x) exp( r dpo (x,y)/β) Z(x), where Z(x) = y π ref (y | x) exp( r dpo (x, y)/β). Then the forward KL divergence isEx,y∼ρ•π ref log π ref (y | x) π dpo (y | x) = Ex,y∼ρ•π ref log Z(x) exp( r dpo (x, y)/β) = Ex,y∼ρ•π ref - r dpo (x, y) β + log(Z(x)) .Although the first term can be easily related to the reward learning guarantee, the second term (E x∼ρ [log(Z(x))]) can unfortunately be vacuous without further assumptions. We formalize in the following result: Proposition C.2. There exist π dpo such that Assumption 4.3 holds, but KL(π ref ||π dpo ) is arbitrarily large.Proof. First without loss of generality let us consider that r * > 0. Now suppose there existsỹ such that π ref (ỹ | x) = 1n 4 for all x, where n will be determined soon. Now suppose that for all x, r dpo (x, ỹ) -r * (x, ỹ) = n and r dpo (x, y) = r * (x, y) for all y ̸ = ỹ. Now we can check thatE x∼ρ E y∼π ref (•|x) ( r dpo (x, y) -r * (x, y)) 2 = 1 n 2 ,which is diminishing if we take n to be big enough. We can also check thatE x∼ρ E y∼π ref (•|x) - r dpo (x, y) β ≥ - 1 n 3 β - R n 4 βand thus the first term will have little impact on the final bound. However, the second term can be lower bounded as follows:log y π ref (y | x) exp( r(x, y)/β) = log y π ref (y | x) exp r * (x, y) + r(x, y) -r * (x, y) β ≥ log y π ref (y | x) exp r(x, y) -r * (x, y) β = log π ref (ỹ | x) exp r(x, ỹ) -r * (x, ỹ) β = n β -4 log(n).Putting everything together, we haveKL(π ref ||π dpo ) ≥ n β -4 log(n) - 1 n 3 β - R n 4 βand since we can take n arbitrarily big we complete the proof. Then there exists policy π ∈ Π dpo such that J(π) = -∞.Proof. Again as in the proof sketch, without loss of generality, we consider a promptless setting, and assume that the response space is Y = {y 1 , y 2 , y 3 }. Again without loss of generality, we assume π ref only covers y 1 and y 2 , and thus Assumption 4.1 breaks. Now consider the optimal policyπ * (y) = π ref (y | x) exp(r * (y)/β) Z * (t) , ∀y ∈ Y,where Z * = y∈Y π ref (y | x) exp(r * (y)/β), note that by construction π * (y 3 ) = 0.Then consider the following policy π such thatβ log π(y 1 ) π ref (y 1 ) • Z * = r * (y 1 ) - √ ε dpo , and β log π(y 2 ) π ref (y 2 ) • Z * = r * (y 2 ) - √ ε dpo ,Then we haveE y∼π ref β log π dpo (y) π ref (y | x) • Z * -r * (x, y) 2 = ε dpo ,thus π satisfies Assumption 4.3. Rearranging we can see that π(y 1 ) < π * (y 1 ) and π(y 2 ) < π * (y 2 ). Now since π * = 0, we haveπ * (y 1 ) + π * (y 2 ) = 1,and combine we get π(y 3 ) > 0, which implies KL(π||π ref ) is unbounded, since π ref (y 3 ) = 0.In this section we prove Theorem 4.2:Theorem D.1 (Restatement of Theorem 4.2). Suppose that Assumption 4.4 holds. Then for any reference policy π ref such that Assumption 4.2 holds with ε kl = 2R ′ β , for any RLHF policy π rlhf with r such that (c.r. Assumption 4.3),E x,y∼ρ•π ref (r * (x, y) -r(x, y)) 2 ≤ ε reward ,or more generally, the event in Lemma B.2 holds for r, we haveJ(π * ) -J(π rlhf ) ≤ O(C ε kl √ ε reward ).To prove this we first prove the following lemma so we can leverage Assumption 4.2:Lemma D.1 (Restatement of Lemma 4.1). Suppose that Assumption 4.4 holds. Then for any RLHF policy π rlhf , we have thatKL(π rlhf ||π ref ) := E x∼ρ E y∼π rlhf (•|x) log π rlhf (y | x) π ref (y | x) ≤ 2R ′ β .Proof. since we have that π rlhf (y| x) = π ref (y|x) exp(r(x,y)/β) Z(x)for all x ∈ supp(ρ), y ∈ Y, we haveKL(π rlhf ||π ref ) = E x∼ρ E y∼π rlhf (•|x) log exp(r(x, y)) βZ(x) = E x∼ρ E y∼π rlhf (•|x) r(x, y) β -log(Z(x)) .Plugging in the definition of Z(x) we getlog(Z(x)) = log E y∼π ref (•|x) exp r(x, y) β ≥ E y∼π ref (•|x) r(x, y) βdue to Jensen's inequality. Thus we haveKL(π rlhf ||π ref ) ≤ E x∼ρ E y∼π rlhf (•|x) r(x, y) β -E x∼ρ E y∼π rlhf (•|x) r(x, y) β ≤ 2R ′ β .Now with Lemma 4.1, we can prove Theorem 4.2:Proof. By Lemma B.1, we have Under linear function approximation (Definition 6.1), there exists dataset collected from distribution µ that does not cover y * , i.e., µ(y * ) = 0, but has global coverage in the linear function approximation setting [54] : letJ(π * ) -J(π rlhf ) ≤ E x∼ρ E y 1 ∼π * (•|x),y 2 ∼π rlhf (•|x) r * (x, y 1 ) -r(x, y 1 ) -r * (x, y 2 ) + r(x, y 2 ) ≤ E x∼ρ E y 1 ∼π * (•|x),y 2 ∼π rlhf (•|x) (r * (x, y 1 ) -r(x, y 1 ) -r * (x, y 2 ) + r(x, y 2 )) 2 ≤ C 2 glo E x∼ρ E y 1 ,y 2 ∼π ref (•|x) (r * (x, y 1 ) -r(x, y 1 ) -r * (x, y 2 ) + r(x, y 2 ))Σ µ = E y∼µ ϕ(y)ϕ(y) ⊤ , then for all π, E y∼π ∥ϕ(y)∥ 2 Σ -1 µ ≤ C π .Then DPO will return a policy π dpo such that π dpo (y * ) > 0, but π dpo (y) ≤ π ref (y) for all y in the offline data support, i.e., µ(y) > 0.Proof. Consider a response space Y = {y 1 , y 2 , y 3 }, with ϕ(y1 ) = [1, 0], ϕ(y 2 ) = [1/2, 1/2], ϕ(y 3 ) = [0, 1] Let w ref = [1, 1], then we have π ref (y i ) = 1/3, ∀i ∈ {1, 2, 3}.Let the ground truth reward function r * (y) = [10, 1] ⊤ ϕ(y), and suppose supp(µ) = {y 1 , y 2 }, i.e., the data only covers y 1 and y 2 , and µ(y 3 ) = 0. And as always, the preference is based on the ground truth reward function under the Bradley-Terry model.We can first check that the data distribution indeed has global coverage: for all π we haveE y∼π ∥ϕ(y)∥ 2 Σ -1 µ ≤ C π .If we parameterize r(y) = w ⊤ ϕ(y) (or in case of DPO, we can still check and see that r dpo (y) = w dpo ⊤ ϕ(y) because of the softmax linear parametrization of the policies), for either direct reward learning or DPO, we can have the learned reward function r(y) = [10, 1] ⊤ ϕ(y) + c, where c is the constant reward shift (c.r. Eq. ( 5)). Then a simple calculation (by π(y) ∝ π ref (y) exp( r(y)/β)) shows that, as long as c is small enough, the policies will decrease the likelihood of y 1 and y 2 and increase the likelihood of y 3 .We first describe our experiment setup. We consider linear function approximation setting where we have 100 responses (|Y| = 100). We consider a 16-dimensional feature vector ϕ : Y → R 16 , and we generate ϕ(y) by simply sampling 99 random 16-dimensional vectors where the ℓ 1 norm of each vector is 1. We add one final ϕ(y) = [1, 0, 0, . . . ].We construct the implicit human reward r * (y) = w * ⊤ ϕ(y), where w * = [5, ...], and the rest of the entries are sampled from Unif(-2,2).We parametrize the policies as softmax linear policies, i.e., we parametrize each policy π with w π ∈ R 16 such that π(y) = w π ⊤ ϕ(y) y∈Y w π ⊤ ϕ(y) . One can check in this formulation the implicit reward in DPO ( r dpo ) is linear in ϕ.We generate 10000 preference pairs, according to the BT model under r * , for the first 50 responses. We checked that the first responses indeed span R 16 . Thus the offline data has global coverage in linear function approximation setting.For on-policy RL methods, we first train a reward model. Then we simply perform gradient descent on the KL-regularized bandit loss (we assume π ref is uniform). For DPO, we simply perform SGD We plot the mean log probability of the preferred responses and the log probability of the best response, which is unseen in the training data. We see that both algorithms correctly assigns increasing probability to the best response. on the offline preference dataset. We track two qualities over the training: the mean log probability of a random subset of preferred responses, and the log probability of best response ϕ(y) = [1, 0, 0, . . . ]. We plot the results in Figure 2 . We observe that both methods have the extrapolation behavior -the probability of preferred responses decays but the probability of the optimal response goes up.Now we describe the setting where function approximation fails, and this reduces to a Multi-arm bandit setting. We set |Y| = 500, and the offline data only covers the first half of the responses. The r * (y) is set by sampling from Unif(-10,10), and we generate 10000 offline samples by uniformly sample pairs of responses from the first half of the response space, and then label them with BT model under r * . We train DPO with 5000 iterations, and plot the mean probability of the responses outside of the data support in Figure 3 : we observe that the mean probability of the out-of-distribution responses are increasing, however, this could be an undesirable behavior because the reward of the out-of-distribution responses could be arbitrarily bad.In this section, we consider the constrained optimization version of HyPO (Eq. ( 6)). Note that the reward function class is identical to DPO, i.e., R hypo = β log π(y|x) π ref (y|x)Z(x) | π ∈ Π , where Z(x) is the partition function. Then for each output policy π hypo , we can denote its implicit reward function For the base model of general chat experiments, we use Llama3-8B-Instruct [27] with hugging face model card: meta-llama/Meta-Llama-3-8B-Instruct. The dataset card of the Ultrafeedback dataset [14] is HuggingFaceH4/ultrafeedback_binarized. In addition to the KL penalty, in the general chat task we add an additional length penalty, and the online penalty of a generation y with context x becomes log π(y|x) π ref (y|x) + α|y|. We summarize the hyperparameter of each baseline below. We run the general chat experiment on a node of 8 Nvidia A100 80GB GPUs. DPO takes 3 hours to train one epoch while HyPO takes 18 hours to train one epoch. Justification: We provide a discussion about the limitations of our work in the discussion section.Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We repeat all our experiments over 3 seeds.Guidelines: • The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss the broader impact in the discussion section. Guidelines: • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 11. Safeguards38th Conference on Neural Information Processing Systems (NeurIPS 2024).We use E to denote the empirical expectation over the dataset. 
 
 Appendix:  


Decision: This paper studies the performance gaps between online and offline preference alignment methods, and uses the notions of strong and weak dataset coverage to provide explanations. A Hybrid PO algorithm is introduced to use online samples for calculating KL divergence, which gains some performance improvement over original DPO. All reviewers agreed that the story is complete and the contributions are solid, and I concur.

Reviews: review: paper_summary: paper_summary: This paper studies learning from human preference feedback for aligning large language models (LLMs). Many existing works share the same observation that offline alignment methods such as DPO underperforms their online counterpart such as PPO. But this phenomenon has not been well understood. This paper studies this difference through the lens of coverage. It unveils that the KL constrain in offline alignment methods can be violated due to incomplete coverage and thus offline methods may find bad solutions. Inspired by this insight, the authors propose a new method, hybrid preference optimization (HyPO), which combines the offline preference update with the online KL regularization. Experiments on the summarization task demonstrate that HyPO is better than DPO. This paper also sheds light on a counterintuitive observation that DPO often decreases the likelihood of preferred responses through an illustrative example.
strength_weakness: strength_weakness: The computational cost of the proposed HyPO algorithm is not sufficiently discussed. Usually the most computational cost in online alignment methods comes from the sampling process of LLMs. In practice, the computational cost plays an important role in comparing algorithms. Thus it will provide useful guidance if the computational cost is well discussed in the paper.

The discussion in Section 6 is bit handwavy. While Example 6.1 provides good intuition, a more rigorous analysis is needed to justify the claim.

One minor suggestion on presentation: It will be good to provide the PPO numbers in Table 1 to help the readers compare HyPO to PPO more easily.
questions: questions: In the TL;DR experiment, I am curious to know what happens if the authors also train HyPO for 4 epochs. Would it perform better than PPO?
limitations: limitations: This paper includes a fairly thorough discussion on the limitations at the end. I would like to add one point: the analysis in this work only focuses on the solution space. It explains why offline methods may find bad solutions, but doesn't explain why they often find the bad ones in practice. As far as I know, people often use a very low learning rate for DPO in practice. For example, the original paper used 1E-6. One possible reason for such low learning rates is that it compensates the ineffective KL regularization. Despite such efforts, DPO still finds bad solutions. It will be insightful if we get a better understanding of why bad solutions seem inevitable.
paper_summary: paper_summary: The paper focuses on the paradigm of fine-tuning large language models (LLMs) using human preference data. It delves into two primary techniques: online reinforcement learning (RL) and offline contrastive methods. The authors challenge the previous notion of these techniques being equivalent by conducting a theoretical analysis through the lens of dataset coverage. They introduce the concepts of global and partial coverage conditions, proving that the former is necessary for offline methods using forward KL like DPO to converge to the optimal policy, while the latter is sufficient for online RL methods using reverse KL. The paper proposes a hybrid preference optimization (HyPO) algorithm, demonstrating its empirical superiority over DPO while maintaining some computational efficiency. The authors also provide a coverage-based explanation for why RL and offline contrastive methods might decrease the probability of preferred responses.
strength_weakness: strength_weakness: 1. The experiment is not sufficient. Since it is done with Pythia 1.4B and the TL;DR dataset, it is unclear if the proposed method is still valid on larger models and other datasets. While theoretically interesting, it is unclear if using reverse KL instead of forward KL can lead to a significant performance gain in practice. 

2. The proposed HyPO method only uses online samples to calculate the KL loss rather than to collect new preference feedback. While simpler, it may fail to fully leverage the benefits of online methods.
questions: questions: See the Weaknesses section.
limitations: limitations: The authors discussed the limitations well in the paper.
paper_summary: paper_summary: This work considers the statistical separation between contrastive algorithms (DPO and IPO) and RLHF. It proves that DPO/IPO requires global convergence assumption which is in general a very strong assumption while on the other hand RLHF only requires local coverage. This separation stems from the explicit KL constraint in the objective of RLHF while the implicit reward in contrastive algorithms can results in unbounded KL. Given the observation, a hybrid (offline + online) algorithm is proposed: the preference optimization remains offline DPO however it draws online samples to enforce KL constraints on DPO policy, recovering the statistical guarantee of RLHF while attains the practical computational strengths (w/o value and reward models) of DPO.
strength_weakness: strength_weakness: I only have a couple of minor comments:

- Second sentence of L562 should be appended to L231 to make the proof sketch immediately clear.

- Theorem E.1 could be relocated to Section 5 to be more self-contained.
questions: questions: - I might have overlooked, what is the intuition behind online KL evaluation? Is it possible to enforce such KL constraint purely in offline setting?
limitations: limitations: Overall, I am convinced this is a good submission with good technical quality. However, as I did not extensively follow the theoretical results on preference optimization, I am not able to comment on the novelty of this work, hence I give confidence score of 3. 

It would also be beneficial if the authors could create a table summarizing previous works and highlighting their contributions in comparison to prior research.
paper_summary: paper_summary: This paper focuses on the optimization and learning methods for ``online'' RLHF and contrastive offline methods (DPO, IPO). The authors aim to understand the separation between these two type of methods, where they are different in terms of whether new responses can be sampled or not. The authors state that the difference in the reward parameterization is key to the separation and propose global coverage and local coverage to capture such a difference. 

The theoretical insights also motivate a novel approach called Hybrid Preference Optimization (HyPO) that combines offline data with online samples, where the online samples are used to control the KL divergence. Empirical results are provided to verify the effectiveness of the proposed methods.
strength_weakness: strength_weakness: This is not a weakness but some clarification on the terminology. In the literature of RL theory, particularly for the preference learning paper, the learning (online exploration) is mentioned with querying the human preference so as to learn the $r^*$. In the setup of this paper, the online data is only used to compute the KL loss without querying the human feedback. Therefore, it is more related to an intermediate setting that we use the offline preference dataset to construct a proxy reward function and we are allowed to query the model to get new responses but not the human feedback. See the RSO [1], some discussions in [2] for proposing multi-step RSO, and some discussions in [3]. 

[1] Statistical rejection sampling improves preference optimization.
[2] Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint
[3] Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data.

Then, it would be interesting to see the empirical results in terms of the ONLINE setting where we can query the new response and the new human preference signals because a lot recent works show that even with the DPO, the online framework outperforms the original one with a large margin. Moreover, this is standard in the PPO literature like Chat-GPT, Claude, and LLaMA2.

[4] Training language models to follow instructions with human feedback
[5] Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
[6] Llama 2: Open Foundation and Fine-Tuned Chat Models
questions: questions: see weakness part
limitations: limitations: yes

Comments: title: Thank you for the response!
Reply to reviewer MDLu
Thanks for the responses
comment: I would like to thank the authors for their response. After reading the authors' response and other reviewers' comments, I believe my initial evaluation is appropriate and thus will maintain my rating.
We thank the reviewer for their insightful comments. We agree with the reviewer that if we have a reward model which is trained on a more diverse preference dataset (which has a better coverage condition), then it will be beneficial for hypo to query labeled online samples which will provably improve upon our current setting where we do not acquire additional reward information with the unlabelled online samples. As the reviewer points out, a refined study on such iterative/online methods is indeed an important future direction. We are happy to see that the reviewer agrees that the coverage condition is hard to satisfy in the LLMs setting and thus in this work we argue that algorithms that require weaker coverage condition can have better empirical performance, and thus one of the advantage of the iterative methods is to further relax the coverage condition through online exploration. Again we will make sure to distinguish the iterative/online setting from the setting from the current paper in the revised version, with additional discussion with the iterative/online methods including but not limited to the works mentioned in our discussion.
For DPO-like algorithm, the memory efficiency is not very important because we do not train a value network. The bottleneck of the PPO algorithm is from the challenges of training policy model and value model simultaneously. In particular, the reward model, and reference model to compute the KL divergence can be served with a remote server. 

Personally, I believe that the coverage condition is hard to satisfy in the context of LLMs and online exploration and online iterative training are standard in making the state-of-the-art models like llama3.1, gemini, Claude, and gpt according to their technical report. In these settings, the reward models are typically trained on a much larger and more diverse preference dataset (compared to the training set of DPO), and is further combined with some human-written rules, and llm-as-a-judge, which are a proxy of the r^* or P^* (see llama 3.1 report or [1] RLHF Workflow: From Reward Modeling to Online RLHF. 2024 for examples). This setting is different from the online setting considered in this paper. I believe that further study in this setting can be promising as future work.