Title: The Importance of Online Data: Understanding Preference Fine-tuning via Coverage
Venue: NeurIPS 2024
Accepted: yes
Keywords: Reinforcement Learning from Human Feedback

LLM finetuning
Abstract: Learning from human preference data has emerged as the dominant paradigm for fine-tuning large language models (LLMs). The two most common families of techniques -- online reinforcement learning (RL) such as Proximal Policy Optimization (PPO) and offline contrastive methods such as Direct Preference Optimization (DPO) -- were positioned as equivalent in prior work due to the fact that both have to start from the same offline preference dataset. To further expand our theoretical understanding of the similarities and differences between online and offline techniques for preference fine-tuning, we conduct a rigorous analysis through the lens of *dataset coverage*, a concept that captures how the training data covers the test distribution and is widely used in RL. We prove that a global coverage condition is both necessary and sufficient for offline contrastive methods to converge to the optimal policy, but a weaker partial coverage condition suffices for online RL methods. This separation provides one explanation of why online RL methods can perform better than offline methods, especially when the offline preference data is not diverse enough. Finally, motivated by our preceding theoretical observations, we derive a hybrid preference optimization (HyPO) algorithm that uses offline data for contrastive-based preference optimization and online unlabeled data for KL regularization. Theoretically and empirically, we demonstrate that HyPO is more performant than its pure offline counterpart DPO, while still preserving its computation and memory efficiency.

The Importance of Online Data: Understanding Preference Fine-tuning via Coverage

Abstract

Learning from human preference data has emerged as the dominant paradigm for fine-tuning large language models (LLMs). The two most common families of techniques - online reinforcement learning (RL) such as Proximal Policy Optimization (PPO) and offline contrastive methods such as Direct Preference Optimization (DPO) - were positioned as equivalent in prior work due to the fact that both have to start from the same offline preference dataset. To further expand our theoretical understanding of the similarities and differences between online and offline techniques for preference fine-tuning, we conduct a rigorous analysis through the lens of dataset coverage , a concept that captures how the training data covers the test distribution and is widely used in RL. We prove that a global coverage condition is both necessary and sufficient for offline contrastive methods to converge to the optimal policy, but a weaker partial coverage condition suffices for online RL methods. This separation provides one explanation of why online RL methods can perform better than offline methods, especially when the offline preference data is not diverse enough. Finally, motivated by our preceding theoretical observations, we derive a hybrid preference optimization (HyPO) algorithm that uses offline data for contrastive-based preference optimization and online unlabeled data for KL regularization. Theoretically and empirically, we demonstrate that HyPO is more performant than its pure offline counterpart DPO, while still preserving its computation and memory efficiency.

1 Introduction

Due to the difficulty of manually specifying reward functions for complex tasks [7], preference-based learning has emerged as a critical component in the fine-tuning procedure for large language models (LLMs) [40, 30, 45, 44]. There are two predominant flavors of preference learning for LLMs: online reinforcement learning (RL) methods such as PPO [11, 30] and offline contrastive methods like Direct Preference Optimization (DPO) [33] and Identity Preference Optimization (IPO) [3].

Online RL methods usually follow the two-stage procedure prescribed in [30]: one first trains a reward model (classifier) on a fixed offline preference dataset before using it to provide reward labels for on-policy generations, which are then fed to a downstream RL algorithm like Proximal Policy Optimization (PPO) [36]. Since the reward model is learned from static offline preference data, to avoid over-optimizing the reward model [17], one typically adds a reverse KL penalty to encourage the model to stay close to some reference policy. We will refer to this procedure as reinforcement learning from human feedback (RLHF) in this paper. While empirically performant, RLHF requires repeated querying of the reward model (which is often itself an LLM) as well as sampling from the current policy. In response to the computational expense and relatively complex nature of this

Aarti Singh

Wen Sun

procedure, purely offline methods like DPO [33] and IPO [3] have been proposed as alternative methods for preference fine-tuning. These methods do not need to fit separate reward models, instead opting to simply train the policy directly on the offline preference dataset via a ranking loss.

Offline contrastive methods like DPO are usually derived via applying a reparameterization trick to the closed-form solution of the minimum relative entropy problem [63] that RLHF techniques attempt to approximate. Thus, several authors have described these methods as equivalent (at least in theory) to the standard RLHF procedure [33, 3]. However, recent (mostly empirical) work has contradicted this perspective: [43] find that online methods out-perform offline methods and attribute this fundamentally to on-policy sampling, [55] argues that the online RL methods produce an often desirable subset of the possible DPO loss minimizers, and [42] provide empirical support for the claim that online and contrastive training provide orthogonal benefits. However, a rigorous theoretical separation is still lacking in the pre-existing literature, which motivates our key questions:

What is the statistical separation between the online RLHF method and offline contrastive methods? What causes this separation and what does it imply?

To answer these questions, we focus on the coverage of the preference dataset, a key concept that is widely used in RL [22, 28, 58] for analyzing the impact of offline or exploratory data distributions. Through the lens of coverage of the offline preference dataset, we make the following contributions:

We prove that the global coverage condition [28], the strongest possible coverage condition in RL, is necessary for offline contrastive algorithms like DPO to converge to the optimal policy. In contrast, we identify a weaker local coverage condition that is sufficient for online RLHF algorithms, thus provably separating the two types of algorithms. The separation is due to the difference in reward modeling and on/offline regularization - in short, there is no free lunch from bypassing explicit reward learning and online rollouts . As global coverage might sometimes be violated in practice, our separation result can perhaps explain why RLHF works better than offline methods [42, 43, 56].

Although offline contrastive methods are derived from a reverse-KL objective, we prove that the policies trained via offline methods can still have infinite reverse-KL in the partial coverage setting. In contrast, we show that RLHF can always control the reverse KL via directly optimizing reverse KL using online samples. This means that on realistic problems, RLHF has stronger guarantees for remaining close to the reference policy than offline contrastive methods.

We propose Hybrid Preference Optimization (HyPO) to address the deficiencies of offline contrastive methods while maintaining some of their computational simplicity. HyPO is a hybrid RL algorithm [51, 39] where offline data is used for the DPO objective while online samples are used to explicitly control the reverse KL divergence to the reference policy. We empirically demonstrate that HyPO outperforms DPO, on the TL;DR summarization task [40] on all metrics including both the GPT4 win-rate and the reverse KL divergence to the reference policy, and on general chat benchmarks such as AlpacaEval 2.0 [15], trained with the UltraFeedback dataset [14]. In addition, HyPO also mitigates the overfitting issues observed in the offline constrastive based methods [43].

We provide an explanation of why RLHF and offline contrastive methods decrease the probability of both preferred and rejected responses during training. In particular, under our function approximation-based global coverage condition, we show that such behavior is actually desirable for DPO and RLHF policies to extrapolate and generalize to optimal actions that do not appear in the training dataset. However, without function approximation, algorithms like DPO can mistakenly increase the likelihood of sub-optimal actions. This establishes the importance of function approximation for the success of the algorithms such as DPO.

Taken together, our results establish the critical role coverage plays in terms of convergence properties of preference learning algorithms as well as in the design of new, performant empirical approaches.

2 Related Work

Preference Fine-Tuning (PFT). As discussed in the introduction of our work, there are two major paradigms for preference fine-tuning of LLMs. The first one, online RL methods [30], proposes to first train a reward model (classifier) to predict human preferences, followed by running an RL method to optimize this learned reward function. While PPO [36] is the most popular RL algorithm used in the online RLHF framework by far [40, 30, 45], more recent work by [1] shows that simpler online RL algorithms like REINFORCE [48] also work well. The second class of methods, offline contrastive techniques [60, 33, 3], avoid explicit reward modeling and directly optimize their objective on the offline preference dataset. Recently there are hybrid methods that combine offline

preference data with online preference labels [52, 19, 35, 3] - we leave extending our analysis to this setting to future work. Throughout our paper, we assume for simplicity of analysis that preferences are generated by an underlying utility function and therefore contain no intransitivities [29, 41].

Understanding PFT. Prior work has studied different parts of the standard RLHF recipe [17, 23, 38, 16] and the impact of preference data quality [37]. In our work, we instead take a converge-based perspective on the relationship between online RL methods and offline contrastive methods. Although derived from the same minimum relative entropy objective [63] and perceived as equivalent by some early work [33, 3], more recent work has started to unravel the distinctions between these two classes of methods. [43] repeatedly observe better performance from online rather than offline methods and after rigorously validating a variety of hypotheses, conclude that on-policy sampling is indispensable for ensuring a high quality policy. [42] perform an in-depth study of the effects of preference data, contrastive losses, and on-policy sampling and conclude that a combination of contrastive losses and interactive training is most preferable in practice. [55] also observe better performance from online PPO than from offline DPO and argue this is because the former is able to eliminate a larger set of policies that are undesirable from the perspective of the later. We supplement these mostly empirical observations with a rigorous theoretical explanation for the observed behavior through the lens of dataset coverage, as well as designing an algorithm that addresses the key weaknesses of offline contrastive approaches.

We defer additional related works to Appendix A.

3 Preliminaries

Following a wide range of recent works [33, 3], we consider the RLHF problem in the contextual bandit formulation [25]. This is a reasonable simplification, as one can consider the generated sequence of tokens as one single action, due to the fact that the states are the generated tokens, and the dynamics are deterministic. We denote the context (prompt) space as X , and the action (response) space as Y . Note that due to the finiteness of the possible tokens, the action space is finite but combinatorially large. We use ρ ∈ ∆( X ) to denote the distribution of the prompts, and π : X → ∆( Y ) as policies (LLMs) that map prompts to a distribution of responses. We also consider the reward function class R : X × Y → R , which assigns a reward to each context-response pair.

We assume access to a reference policy π ref , which is usually referred to as the policy learned using supervised data when training the LLM, that needs to be further fine-tuned to align with human values. An offline preference dataset is collected in the format of D = { x, y + , y -} triplets: given context x ∼ ρ , the preference policy samples two responses y 1 , y 2 ∼ µ ( · | x ) , where µ is the offline response distribution. Previous works assume either µ to be the same distribution as π ref [33] or different offline distribution [3, 35, 18]. Then, y 1 is labelled as y + (thus y 2 as y -) with probability p ∗ ( y 1 ≻ y 2 | x ) , where p ∗ is defined by the Bradley-Terry model [6]:

where r ∗ is the human's implicit reward function. Note that this rules out intransitive preferences [41, 29]. Throughout the paper, we will make the following assumption on the reward function:

Assumption 3.1 (Boundedness of the reward) . ∥ r ∗ ∥ ∞ ≤ R.

In many previous works, this formulation has been the canonical way to model the preference data in the RLHF literature [11, 33, 3]. The goal is to learn a policy π to maximize the objective J ( π ) , where

i.e., we want to both maximize the human implicit reward, and not deviate too much from the reference policy. We denote the optimal policy π ∗ ∈ argmax π ∈ Π J ( π ) . Here we call KL ( π ( · | x ) || π ref ( · | x )) reverse KL because π - the policy to be optimized, appears first. We will call KL ( π ref ( · | x ) || π ( · | x )) forward KL. By the definition of KL, we have

Note that the expectation in reverse KL is under π , indicating that evaluating and optimizing reverse KL requires drawing online samples from π . In contrast, evaluating forward KL only requires offline

samples drawn from π ref . As we will show, this key difference between reverse KL and forward KL plays an important role of separating online RLHF and offline contrastive methods such as DPO. In this paper, we consider two types of algorithms: online RL-based algorithms, and offline contrastive-based algorithms.

Online RLHF Algorithms. We consider algorithms such as [11, 1] as the online RL based methods. We abstract these algorithms as the following procedure: the algorithm performs the following twostage procedure: one first trains a reward model ̂ r that minimizes the Bradley-Terry loss 1

and perform policy optimization (such as PPO [36]) to optimize the policy with the reward model ̂ r :

However, this policy optimization step requires extensive online sampling, and possibly training an additional critic model (e.g., PPO), in addition to the reward model and policy.

Offline Contrastive Algorithms. To circumvent the above-mentioned computational burden, several purely offline contrastive-based methods (i.e., without RL) have been proposed. In this paper, we focus on the following two most representative methods. The first is Direct Preference Optimization (DPO) [33], where the objective is π dpo ∈ argmax π ℓ dpo ( π ) with

Another offline contrastive method we will discuss in our paper is Identity Preference Optimization [3], but we will defer its technical details to the appendix.

4 Offline Contrastive Methods Require a Stronger Coverage Condition than Online RL Methods

We start by introducing the mathematical formulation of the coverage framework. The strongest coverage condition is the following global coverage condition [28]: we say any offline distribution µ covers a policy π if we have max x,y : ρ ( x ) > 0 π ( y | x ) µ ( y | x ) ≤ C glo . Throughout this section, we will adopt the setting where µ = π ref [33]. Formally, we assume the following condition:

Assumption 4.1 (Global Coverage) . For all π , we have

For the coverage terms, we always adopt the convention that 0 0 = 0 . Note that one sufficient condition for this assumption is that, for any prompt x , and any token sequence y , we have π ref ( y | x ) ≥ 1 /C glo .

As has been recognized in the offline RL literature, global coverage is a strong assumption, and efforts have been made to circumvent this assumption with more relaxed coverage conditions [46, 10, 58]. In this paper, we will consider the following partial coverage assumption that is weaker than Assumption 4.1:

Assumption 4.2 (Local KL-ball Coverage) . For all ε kl < ∞ and all policy π such that E x ∼ ρ [ KL ( π ( · | x ) || π ref ( · | x ))] ≤ ε kl , we have

Note that C ε kl depends on ε kl . This coverage notion is relatively new in the RL literature and only appeared in previous analysis of RLHF algorithms [9]. We call this local coverage condition since it only requires π ref to cover the policies that is within some KL-divergence ball centered at π ref . The intuition of this assumption is, for any algorithm that can control the reverse KL of the output policy, we can leverage the coverage condition to relate the error under the output policy to its error under the offline distribution, and thus guarantee its performance. Finally, we note that since the policies with bounded KL is a subset of all policies, for a fixed π ref , we always have C ε kl ≤ C glo .

Remark 4.1. Taking a closer look at Assumption 4.2, we can see that this assumption is always true in the sense that for any policy with ε kl < ∞ , max x,y : ρ ( x ) > 0 π ( y | x ) π ref ( y | x ) < ∞ , i.e., C ε kl < ∞ , for any ε kl . However, while being bounded, C ε kl can be large. Indeed a simple calculation can show that max x,y : ρ ( x ) > 0 π ( y | x ) π ref ( y | x ) can be as large as max x,y : π ( y | x ) > 0 exp ( ε kl π ( y | x ) ) . This can be undesirable because this suggests bounded reverse KL itself is not enough to guarantee optimality: the error can have an exponential amplification when switching from π ref to π . Thus this motivates Assumption 4.2, which assumes that C ε kl is reasonably small.

In what follows, we will show that the global coverage assumption (Assumption 4.1) is necessary for offline contrastive-based algorithms such as DPO and IPO, but partial coverage assumption such as Assumption 4.2 is sufficient for online RL based algorithms. This establishes a separation between the two types of algorithms. We emphasize this theoretical separation explains why in practice online methods is less prone to problems such as reward hacking and producing out-of-distribution responses that are due to dataset with insufficient coverage.

4.1 Global Coverage is Necessary for Offline Contrastive Algorithms

Failure of DPO Under Partial Coverage. Now we show that if the strong coverage Assumption 4.1 breaks, then DPO can not guarantee any performance with respect to the objective function Eq. (1). The intuition is based on a rather common observation of the DPO algorithm: the DPO policy π dpo may generate out of distribution responses, while in contrast, RLHF does not generate responses outside of the support of π ref due to online reverse-KL constraint. For example, [55] provides a construction where π dpo chooses a response where RLHF policy assigns 0 mass, thus proving that RLHF policies are a subset of DPO policies.

However, such construction assumes that the reward learning procedure of DPO makes arbitrarily large errors. Also, previous constructions assume deterministic preference, which is only true if the underlying reward function is unbounded. This violates the natural assumption of Assumption 3.1. In the following, we relax these constraints and thus show that DPO fails to guarantee any performance in a rather strong sense. Concretely, DPO constructs the following implicit reward class with the policy class Π : R dpo = { β log ( π ( y | x ) π ref ( y | x ) Z ( x ) ) | π ∈ Π } , where Z ( x ) is a partition function that maps context to a real number and is independent of y . Plugging this formulation into the BT loss (Eq. (3)) recovers exactly the DPO loss (Eq. (4)) as the partition functions are canceled. Now we can characterize the returned policy by DPO as exactly whose corresponding reward function is accurate in distribution :

Assumption 4.3 (In Distribution Reward Learning) . We assume the DPO policy π dpo satisfies that:

Note that this is a rather strong assumption for BT loss - by Lemma B.2, at best one can only hope: for any learned reward function ̂ r , for each context x , there exists a constant c ( x ) such that

i.e., the reward model predicts the human reward up to a gap that is independent of y . This is due to the fact that BT loss only requires the reward function to capture the relative difference, or in other word, any constant shift (with respect to context) in the reward will be canceled in the BT loss. However, for the rest of the section, we will make the stronger learning assumption that the gap c ( x ) = 0 (such as in the case of Assumption 4.3). Previous counterexamples analysis violates this assumption, but we will show that even under this strong assumption, DPO still can not guarantee any performance.

Proposition 4.1. Denote π ref as any reference policy such that Assumption 4.1 breaks. Let Π dpo be the set of DPO returned policies such that Assumption 4.3 holds. Then there exists policy π ∈ Π dpo such that J ( π ) = -∞ .

Proof sketch. Without loss of generality, we consider a promptless setting, and assume that the response space is Y = { y 1 , y 2 , y 3 } . Again without loss of generality, we assume π ref only covers y 1 and y 2 , and thus Assumption 4.1 breaks. We assume partition function Z = 1 for all π but we will be

rigorous in the formal proof. Then consider the following policy π such that

One can check π satisfies Assumption 4.3. Now consider the optimal policy π ∗ ( y i ) = π ref ( y i ) exp ( 1 β r ∗ ( y i ) ) , for i ∈ { 1 , 2 } , and π ∗ ( y 3 ) = 0 . Since π ∗ ( y 1 ) + π ∗ ( y 2 ) = 1 , combining everything we get π ( y 3 ) > 0 , which implies KL ( π || π ref ) is unbounded, thus we complete the proof.

One can first relate the above construction to the parital coverage assumption Assumption 4.2: since the policy π considered in the proof has unbounded reverse KL with respect to π ref , thus it is not in the KL-ball of ε kl around π ref , which implies that Assumption 4.2 is not sufficient for DPO. Next we show that global coverage is necessary for the IPO algorithm.

Failure of IPO Under Partial Coverage. To show that the global coverage is necessary for IPO, we can even assume a stronger in-distribution learning guarantee, that is, the returned policy achieves the smallest error on its population loss in distribution.

Proposition 4.2 (Informal) . Denote π ref as any reference policy such that Assumption 4.1 breaks. Let Π ipo be the set of IPO returned policies such that it is the minimizer of in-distribution error on its population loss. Then there exists policy π ∈ Π ipo such that J ( π ) = -∞ .

We defer the detailed setup and formal version to Appendix D, but the construction for the above proofs share the same intuition: the reverse KL term in the objective function can be unbounded. For offline contrastive-based algorithms, the KL regularization is only enforced under the data distribution, and thus the algorithm can not guarantee bounded reverse KL if the reference policy does not cover the response space well. Although we only showed counterexamples for DPO and IPO, we conjecture that the same intuition holds for other offline contrastive-based algorithms. One natural question at this point would be: how about the forward KL? Not surprisingly, the forward KL for DPO (but we conjecture for other offline constructive-based methods as well) is vacuously large, and we formalize this result in Appendix C.2.

Remark 4.2. The folklore that DPO is equivalent to RLHF is often based on some assumption that is much stronger than Assumption 4.3: it requires that the learned policy has a point-wise accuracy guarantee β ln( π dpo ( y | x ) /π ref ( y | x )) = r ∗ ( x, y ) for all x, y . Such a point-wise guarantee is unrealistic in reality and does not hold in general in the supervised learning sense. The in-distribution style guarantee in Assumption 4.3 is the best one could hope for from a supervised learning algorithm.

4.2 Global Coverage is Sufficient for Offline Contrastive Algorithms

After showing that global coverage is necessary for DPO to guarantee any performance, we now show that it is sufficient for the performance guarantee.

Theorem 4.1. Let π ref be any reference policy such that Assumption 4.1 holds. For any policy π dpo such that the event in Assumption 4.3 holds, we have that

and we can complete the proof by plugging in the error guarantee from Assumption 4.3.

Note that as the proof suggests, the result holds with the more general reward learning guarantee as in Lemma B.2 - one only needs to be accurate in predicting the relative rewards between response pairs.

4.3 Online RL Method Under Partial Coverage

Finally, we contrast the previous negative results in Section 4.1 for offline contrastive-based algorithms to a positive result for online RL-based algorithms, under the partial coverage setting. We will show that in general global coverage is not necessary for RLHF, i.e., it can guarantee performance under partial coverage. In fact, one might still be able to show an impossibility result for RLHF under partial coverage, by reusing the same counterexample as in the previous section (c.r., Proposition 4.1). Concretely, as long as the learned reward ̂ r ( y 3 ) →∞ , π rlhf ( y 3 ) will be 1 and thus the reverse KL will be unbounded. However, this is a rather unrealistic scenario, as the construction requires a reward model (e.g., a neural network) to output an unbounded value. Thus this motivates the following assumption:

Assumption 4.4. For all learned reward model ̂ r from the reward model class, we have that ∥ ̂ r ∥ ∞ ≤ R ′ .

At this point, one might ask why a similar assumption is missing for the offline contrastive-based analysis, since in Remark 4.2 we argued that a point-wise learning guarantee is unrealistic but Assumption 4.4 is indeed also a point-wise boundedness assumption. The reason lies in the different construction of the model class ̂ r for those algorithms: for DPO and IPO, the reward model is constructed as ̂ r dpo = β log ( π π ref · Z ) , and there is no natural function class for π such that point-wise assumptions such as the one in Remark 4.2 or Assumption 4.4 holds. In contrast, post-processing such as clipping, offline normalization and on-the-fly normalization of rewards is standard in practice, which means the policy will always witness bounded rewards [8, 9, 18, 1] during online RL training (e.g., PPO). As we will show in the following, the difference in the reward function (which is tied to the offline vs. online nature of the algorithms) can explain the different coverage requirement of the algorithms. Note that we use the same in-distribution reward learning assumption for both types of methods.

To relate to Assumption 4.2, we first show that the reverse KL divergence of the RLHF policy is always bounded under Assumption 4.4.

Lemma 4.1. Suppose that Assumption 4.4 holds. Then for any RLHF policy π rlhf , we have that

Then we can show that the RLHF algorithm can guarantee performance under partial coverage:

Theorem 4.2. Suppose that Assumption 4.4 holds. Then for any reference policy π ref for which Assumption 4.2 holds with ε kl = 2 R ′ , and any RLHF policy π rlhf with r such that (c.r. Assumption

β ̂

Conditioned on Lemma 4.1, the proof of this theorem is similar to that of Theorem 4.1 so we defer it to Appendix D. Similar to Theorem 4.1, we note that Theorem 4.2 holds under a weaker reward learning guarantee as in Lemma B.2. We also remark that as long as ε kl is finite, C ε kl is finite, so the bound is never vacuous. Since C ε kl ≤ C glo for all ε kl , it indicates the regret bound of RLHF is never worse and can be much better than the regret bound of DPO. Combining Theorem 4.1 and Theorem 4.2, we complete the separation result between offline contrastive methods and online RL methods.

A natural question at this point could be: can we further relax the local KL-ball coverage condition in Assumption 4.2 to a single-policy coverage condition, i.e., just assuming max x,y π ∗ ( y | x ) /π ref ( y | x ) ≤ C ? Prior work [59] shows that with explicit pessimism, it is possible. However, using pessimism makes the algorithm from [59] not computationally tractable and hard to scale to LLM experiments. Our conjecture is that for the RLHF policy π rlhf , it is not possible to achieve meaningful regret under the single policy coverage condition, due to KL not being strong enough to induce pessimism (i.e., bounded KL between π and π ref can still imply exponentially large density ratio π/π ref ). Developing a lower bound for π rlhf under single policy coverage in this case can be an interesting future work.

Algorithm 1 Hybrid Preference Optimization (HyPO)

require Pretrained LLM π θ 0 , reference policy π ref , offline data D , learning rate α , KL coefficient λ .

1: for t = 1 , . . . , T do

2: Sample a minibatch of offline data D off := { x, y + , y -} ∼ D .

3: Compute DPO loss ℓ dpo := ∑ x,y + ,y -∈ D off log ( σ ( β log ( π θ t -1 ( y + | x ) π ref ( y + | x ) ) -β log ( π θ t -1 ( y -| x ) π ref ( y -| x ) ))) .

4: Sample (unlabeled) online data D on := { x, y } where x ∼ D , y ∼ π θ t -1 ( x ) .

5: Compute ℓ kl := ∑ x,y ∈ D on log( π θ t -1 ( y | x )) · sg ( log ( ( π θ t -1 ( y | x )) ( π ref ( y | x )) )) .

6: Update θ t = θ t -1 + α · ∇ θ t -1 ( ℓ dpo -λℓ kl ) . return π T .

5 Hybrid Preference Optimization: Regularizing Offline Learning with Unlabeled Online Samples

In this section, we will provide a practical algorithm that bridges the gap between the offline contrastive-based algorithms and the online RL-based algorithms. As we see in the previous sections, the difference between the two types of algorithms is their reward model parametrization, and whether to perform online rollouts. In the following, we will show that these two properties are in fact tightly intervened with each other.

Here we will focus on the DPO algorithm. One way to fix the issue of the unbounded reverse KL of DPO (which is caused by the unbounded reward model class) is to consider the following ideal procedure: at the beginning of the algorithm, we first go through the policy class Π , and then we filter out all the policies such that KL ( π || π ref ) ≥ 2 R ′ β , where R ′ is the boundedness of the reward function class for RLHF. Now applying the same analysis of Theorem 4.2, we can show that this revised DPO algorithm can guarantee performance under the partial coverage assumption, because now the Lemma 4.1, a sufficient condition for Theorem 4.2, is explicitly enforced by the constraints. We defer the detailed statement and analysis to Appendix F.1.

However, such a filtering procedure is not possible in practice, but we can instead consider the following constrained optimization problem: we call the definition of DPO loss in Eq. (4), we want to solve

using the KKT conditions, we can show that the following Lagrangian form is equivalent to Eq. (6):

where λ is the Lagrange multiplier. However, in reality, since we do not know the exact value of R ′ , we can consider setting λ to be a hyperparameter. We present the pseudocode in Algorithm 1. Note that due to the reverse KL term, the Hybrid Preference Optimization (HyPO) algorithm optimizes Eq. (7) via both offline and online samples where the offline samples are used for constructing and optimizing ℓ dpo (here σ denotes the sigmoid function), and the online samples y ∼ π ( · | x ) are for KL (i.e., ℓ kl ). Note that regularizing with reverse KL via online samples is widely used in online RLHF (e.g., PPO [40], APA [62], REBEL [18]). Here sg refers to the stop gradient operation, which is a common practice in optimizing reverse KL in the LLM fine-tuning setting [30, 47]. Finally, previous iterative RLHF methods [53] can be interpreted as hybrid methods as well, but they require labeling online samples from an additional reward model while HyPO only requires unlabeled online samples.

Summarization. Our first experiment is on the TL;DR dataset [40]. Our experiment setup mostly follows [18]: we use a maximum context length of 512 and a maximum generation length of 53. We use Pythia 1.4B and Pythia 2.8B [5] as the pre-trained model. For the supervised fine-tuning (SFT) model, we train it over 1 epoch of the dataset with human reference responses as labels. We train the reward model on top of the SFT over 1 epoch of preference data. Both HyPO and DPO are trained over 1 epoch of preference data with Low-rank Adaptation (LoRA) [21]. We defer more experiment details in Appendix F.

We summarize the results in Table 1: HyPO outperforms DPO in terms of GPT4 win-rage and reverse KL. Particularly, the significant reduction in reverse KL implies the impact of including a reverse KL term explicitly into the DPO objective. While comparing with PPO (e.g., Table 1 in [18]), HyPO's performance is still lower in winrate, HyPO does preserve the key advantages of DPO over PPO: we avoid training additional reward model and a value network.

General Chat. In the general chat setting, the model is required to produce a response y given user instruction x . We again follow the experiment setup in [18], where we finetune the Meta-Llama-38B-Instruct [27] model on the ultrafeedback dataset [14]. Due to the computation constrain, we follow the setup in [18] where we only train the last 4 layers of the network for both HyPO and DPO.

For evaluation, we use the common metrics including AlpacaEval 2.0 [15], MT-bench [61] and Open LLM leaderboard tasks: MMLU [20], GSM8K [13], Arc [12], TruthfulQA [26] and HellaSwag [57]. We provide the results for AlpacaEval and MT-bench in Table 2, and the the results of the remaining tasks can be found in Table 3.

0

2500

5000

7500 10000125001500017500

Timestep

0

25

50

75

100

125

150

175

Mean Score

Mean Validation KL

HyPO

DPO

HyPO Mitigates Overfitting in Contrastive Methods. Since the offline contrastive based methods only work with a static offline dataset, the overfitting issue has been observed [43]. In our last experiment, we show that HyPO can effectively address the overfitting issue by leveraging the unlabeled online data. We follow the setup of the summarization task with Pythia-2.8B base model. We train DPO and HyPO for 5 epochs respectively, and evaluate on the first 300 data in the validation dataset. We plot the validation KL in Figure 1: we observe that HyPO is better at preventing the deviation from the reference policy caused by overfitting from training on excessive epochs, even though the methods theoretically both have KL regularization to the reference policy.

6 Function Approximation Coverage: Can Fine-tuned Policies Extrapolate?

Our final result is a theoretical explanation of the extrapolation behavior of preference fine-tuning algorithms under the global coverage assumption in the function approximation (FA) setting. The extrapolation behavior refers to the phenomenon of RLHF algorithms (e.g., DPO) can improve SFT models despite the fact that during training the policies assign decreasing likelihood to both the preferred and rejected responses (i.e., they must increase the likelihood of responses outside of the training data) [31].

A previous attempt [32] to explain this behavior is based on the assumption that the responses from the reference policy have the same distribution as the preferred responses from the dataset, i.e., y + ∼ µ d = y ∼ π ref . However, as mentioned in Section 3, more realistically, one should assume that

y ∼ µ d = y ∼ π ref since it is more natural to use the reference policy to generate pairs of responses to collect labels; or even more generally by considering supp ( D ) ⊂ supp ( π ref ) . The latter is common in practice, for example, the dataset is often precollected, or the reference policy might have a small mass on some responses, so with a high probability they are not sampled during the data collection process.

In the following, we illustrate this behavior using linear function approximation:

Definition 6.1 (Linear function approximation) . Consider the promptless setting with response space Y . For all y ∈ Y , the ground truth reward r ∗ ( y ) = w ⊤ ϕ ( y ) , where w ∈ R d is a universal linear weight vector and ϕ : Y → R d is a d-dimensional feature map. In addition, all policies are parametrized as softmax linear policies, i.e., π ( y ) ∝ exp( w ⊤ π ϕ ( y )) . The feature map ϕ is known to the learner, and w is unknown. Finally, let y ∗ = argmax y ∈Y r ∗ ( y ) be the optimal action.

In general, we should not expect the offline dataset to contain the optimal action y ∗ under all situations. We show that thanks to the linear function approximation and the dataset coverage, DPO has hope to extrapolate correctly, i.e., it can increase the model's likelihood of the optimal action while decreasing the likelihood of both the preferred and rejected actions from the offline data:

Proposition 6.1. Under linear function approximation (Definition 6.1), there exists dataset collected from distribution µ that does not cover y ∗ , i.e., µ ( y ∗ ) = 0 , but has global coverage in the linear function approximation setting [54]: let Σ µ = E y ∼ µ ϕ ( y ) ϕ ( y ) ⊤ , then for all π , E y ∼ π ∥ ϕ ( y ) ∥ 2 Σ -1 µ ≤ C π . Then DPO will return a policy π dpo such that π dpo ( y ∗ ) > 0 , but π dpo ( y ) ≤ π ref ( y ) for all y in the offline data support, i.e., µ ( y ) > 0 .

We defer the proof to Appendix D.3. The above result shows when the training dataset together with the function approximation allow the learned function to generalize (e.g., learn a function that can predict well on test examples beyond the training data - a property supervised learning can have), algorithms like DPO can extrapolate correctly, i.e., they can push up the likelihood of the optimal responses outside of the training data while pushing down the likelihood of all the responses in the training data. Although it is not possible to show that extrapolation is guaranteed to always happen: suppose in the offline dataset, there is an action y ′ whose feature ϕ ( y ′ ) is almost identical to ϕ ( y ∗ ) , then in the finite sample case, with small learning error the final policy might increase the probability of y ′ instead of extrapolate to y ∗ , but in this case the policy is still near optimal. That said, our construction in the proof is general enough to explain non-edge-cases under function approximation.

To validate our theory result, in Appendix E we perform a synthetic experiment on global coverage with linear function approximation.

7 Discussion

There are a few limitations of our work: 1) our theoretical analysis only considers the statistical perspective of each algorithm, but we believe our result is complementary to the other work that considers the optimization perspectives [42]. 2) we only conduct experiments on limited models and benchmarks. 3) The experiment result shows that HyPO's performance is still below the one of online RLHF: this might suggest that our theory does not fully explain the benefit of all the components of online RLHF. For example, one hypothesis is that the learn reward function may have better generalization ability. 4) It is not clear that the KL-ball coverage is necessary for online RL-based methods. However, as we discussed, since a bounded reverse KL might still induce exponential error amplification, we conjecture that at least the single policy coverage [58] is not sufficient for online RLHF-based methods that use reverse KL. We believe these limitations lead to several interesting further directions. Finally, our method may not explicitly address the potential hallucinations or toxic behavior of LLMs, which is a common shortcoming of general-purpose fine-tuning algorithms.

References

A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, A. Üstün, and S. Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740 , 2024.

P. Amortila, D. J. Foster, N. Jiang, A. Sekhari, and T. Xie. Harnessing density ratios for online reinforcement learning. In The Twelfth International Conference on Learning Representations , 2024.

M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics , pages 4447-4455. PMLR, 2024.

J. Bagnell, S. M. Kakade, J. Schneider, and A. Ng. Policy search by dynamic programming. Advances in neural information processing systems , 16, 2003.

S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning , pages 2397-2430. PMLR, 2023.

R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324-345, 1952.

S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217 , 2023.

J. D. Chang, K. Brantley, R. Ramamurthy, D. Misra, and W. Sun. Learning to generate better than your llm. arXiv preprint arXiv:2306.11816 , 2023.

J. D. Chang, W. Shan, O. Oertell, K. Brantley, D. Misra, J. D. Lee, and W. Sun. Dataset reset policy optimization for rlhf. arXiv preprint arXiv:2404.08495 , 2024.

J. Chen and N. Jiang. Offline reinforcement learning under value and density-ratio realizability: the power of gaps. In Uncertainty in Artificial Intelligence , pages 378-388. PMLR, 2022.

P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.

P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 , 2018.

K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021.

G. Cui, L. Yuan, N. Ding, G. Yao, W. Zhu, Y. Ni, G. Xie, Z. Liu, and M. Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377 , 2023.

Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475 , 2024.

J. Eisenstein, C. Nagpal, A. Agarwal, A. Beirami, A. D'Amour, D. Dvijotham, A. Fisch, K. Heller, S. Pfohl, D. Ramachandran, et al. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.09244 , 2023.

L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning , pages 10835-10866. PMLR, 2023.

Z. Gao, J. D. Chang, W. Zhan, O. Oertell, G. Swamy, K. Brantley, T. Joachims, J. A. Bagnell, J. D. Lee, and W. Sun. Rebel: Reinforcement learning via regressing relative rewards. arXiv preprint arXiv:2404.16767 , 2024.

S. Guo, B. Zhang, T. Liu, T. Liu, M. Khalman, F. Llinares, A. Rame, T. Mesnard, Y. Zhao, B. Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792 , 2024.

D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 , 2020.

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

S. Kakade and J. Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning , pages 267-274, 2002.

R. Kirk, I. Mediratta, C. Nalmpantis, J. Luketina, E. Hambro, E. Grefenstette, and R. Raileanu. Understanding the effects of rlhf on llm generalisation and diversity. arXiv preprint arXiv:2310.06452 , 2023.

W. Kool, H. van Hoof, and M. Welling. Buy 4 REINFORCE samples, get a baseline for free! 2019.

J. Langford and T. Zhang. The epoch-greedy algorithm for multi-armed bandits with side information. In Advances in neural information processing systems , pages 817-824, 2008.

S. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958 , 2021.

Meta. Introducing meta llama 3: The most capable openly available llm to date, 2024. URL https://ai. meta. com/blog/meta-llama-3 , 2024.

R. Munos and C. Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research , 9(5), 2008.

R. Munos, M. Valko, D. Calandriello, M. G. Azar, M. Rowland, Z. D. Guo, Y. Tang, M. Geist, T. Mesnard, A. Michi, et al. Nash learning from human feedback. arXiv preprint arXiv:2312.00886 , 2023.

L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730-27744, 2022.

A. Pal, D. Karkhanis, S. Dooley, M. Roberts, S. Naidu, and C. White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.13228 , 2024.

R. Rafailov, J. Hejna, R. Park, and C. Finn. From r to q ∗ : Your language model is secretly a q-function. arXiv preprint arXiv:2404.12358 , 2024.

R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36, 2023.

S. Ross and J. A. Bagnell. Agnostic system identification for model-based reinforcement learning. arXiv preprint arXiv:1203.1007 , 2012.

C. Rosset, C.-A. Cheng, A. Mitra, M. Santacroce, A. Awadallah, and T. Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03715 , 2024.

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

A. Sharma, S. Keh, E. Mitchell, C. Finn, K. Arora, and T. Kollar. A critical evaluation of ai feedback for aligning large language models. arXiv preprint arXiv:2402.12366 , 2024.

P. Singhal, T. Goyal, J. Xu, and G. Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716 , 2023.

Y. Song, Y. Zhou, A. Sekhari, J. A. Bagnell, A. Krishnamurthy, and W. Sun. Hybrid rl: Using both offline and online data can make rl efficient. arXiv preprint arXiv:2210.06718 , 2022.

N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems , 33:3008-3021, 2020.

G. Swamy, C. Dann, R. Kidambi, Z. S. Wu, and A. Agarwal. A minimaximalist approach to reinforcement learning from human feedback. arXiv preprint arXiv:2401.04056 , 2024.

F. Tajwar, A. Singh, A. Sharma, R. Rafailov, J. Schneider, T. Xie, S. Ermon, C. Finn, and A. Kumar. Preference fine-tuning of llms should leverage suboptimal, on-policy data. arXiv preprint arXiv:2404.14367 , 2024.

Y. Tang, D. Z. Guo, Z. Zheng, D. Calandriello, Y. Cao, E. Tarassov, R. Munos, B. Á. Pires, M. Valko, Y. Cheng, et al. Understanding the performance gap between online and offline alignment algorithms. arXiv preprint arXiv:2405.08448 , 2024.

G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023.

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

M. Uehara and W. Sun. Pessimistic model-based offline reinforcement learning under partial coverage. arXiv preprint arXiv:2107.06226 , 2021.

L. von Werra, Y. Belkada, L. Tunstall, E. Beeching, T. Thrush, N. Lambert, and S. Huang. Trl: Transformer reinforcement learning. https://github.com/huggingface/trl , 2020.

R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8:229-256, 1992.

T. Xie, C.-A. Cheng, N. Jiang, P. Mineiro, and A. Agarwal. Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems , 34:66836694, 2021.

T. Xie, D. J. Foster, Y. Bai, N. Jiang, and S. M. Kakade. The role of coverage in online reinforcement learning. In The Eleventh International Conference on Learning Representations , 2023.

T. Xie, N. Jiang, H. Wang, C. Xiong, and Y. Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems , 34:27395-27407, 2021.

W. Xiong, H. Dong, C. Ye, Z. Wang, H. Zhong, H. Ji, N. Jiang, and T. Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models , 2023.

W. Xiong, H. Dong, C. Ye, Z. Wang, H. Zhong, H. Ji, N. Jiang, and T. Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Conference on Machine Learning , 2024.

W. Xiong, H. Zhong, C. Shi, C. Shen, L. Wang, and T. Zhang. Nearly minimax optimal offline reinforcement learning with linear function approximation: Single-agent mdp and markov game. arXiv preprint arXiv:2205.15512 , 2022.

S. Xu, W. Fu, J. Gao, W. Ye, W. Liu, Z. Mei, G. Wang, C. Yu, and Y. Wu. Is dpo superior to ppo for llm alignment? a comprehensive study. arXiv preprint arXiv:2404.10719 , 2024.

L. Yuan, G. Cui, H. Wang, N. Ding, X. Wang, J. Deng, B. Shan, H. Chen, R. Xie, Y. Lin, et al. Advancing llm reasoning generalists with preference trees. arXiv preprint arXiv:2404.02078 , 2024.

R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830 , 2019.

W. Zhan, B. Huang, A. Huang, N. Jiang, and J. Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory , pages 2730-2775. PMLR, 2022.

W. Zhan, M. Uehara, N. Kallus, J. D. Lee, and W. Sun. Provable offline preference-based reinforcement learning. In The Twelfth International Conference on Learning Representations , 2023.

Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P. J. Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425 , 2023.

L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36, 2024.

B. Zhu, H. Sharma, F. V. Frujeri, S. Dong, C. Zhu, M. I. Jordan, and J. Jiao. Fine-tuning language models with advantage-induced policy alignment. arXiv preprint arXiv:2306.02231 , 2023.

B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai , volume 8, pages 1433-1438. Chicago, IL, USA, 2008.

A Additional Related Works

Extrapolation Behavior of PFT. Recent work [56, 31, 32] has observed an interesting effect of the DPO procedure: a simultaneous decrease in the likelihood of both preferred and rejected responses. This behavior is surprising at first glance because one would expect that DPO will increase the likelihood of preferred responses and decrease the likelihood of rejected responses. We provide a rigorous statistical explanation of this behavior and show that this behavior is natural when the offline preference data only contains sub-optimal responses but the function approximation allows DPO to extrapolate and generalize to the correct optimal responses. This highlights the role of function approximation in the success of offline contrastive based methods.

Coverage. We analyze online RLHF and offline contrastive-based methods via the concept of coverage. Coverage measures how well an offline (data) distribution covers the support of the policy of interest, which has been the key technical tool in offline RL [28, 49, 46, 58], offline-online RL [34, 51, 39, 2] and online RL [22, 4, 50]. The data coverage plays an important role in our analysis since both online RLHF and offline contrastive-based methods rely on an offline preference dataset for learning.

B Auxiliary Lemmas

Lemma B.1 (Objective decomposition) . Let J ( π ) be the objective function defined in (1), and for reward function ˆ r , we let

then we have

Proof. We have

where the inequality is due to Eq. (8). To complete the proof, note that

Lemma B.2 (Lemma C.2 from [9]) . Assume that r ∗ is bounded, let R be the reward function class, and Let

then we have with probability at least 1 -δ that

where κ measures the non-linearity of the link function, and c is a constant, N := |D| is the size of the offline dataset.

C Additional Results

C.1 Results for IPO

In this section we give detailed technical details for IPO, and the negative results for IPO under partial coverage. Recall that the empirical objective of IPO is is π ipo ∈ argmin π ̂ ℓ ipo ( π ) , where

The empirical objective is derived from the following population loss

where

and I ( y 1 , y 2 ) is a Bernoulli random variable with parameter p = p ∗ ( y 1 ≻ y 2 ) , where here p ∗ can be any underlying human preference (that is not necessarily parametrized by the Bradley Terry model). To show the negative result, we can make the following learning assumption:

Assumption C.1 (In distribution guarantee for IPO) . We assume that the returned policy π ipo satisfies that

i.e., the returned policy π ipo induces the smallest possible in-distribution error on its population loss.

With the setup, we can state and prove the formal version of the result:

Proposition C.1 (Formal version of of Proposition 4.2) . Denote π ref as any reference policy such that Assumption 4.1 breaks. Let Π ipo be the set of IPO returned policies such that Assumption C.1 holds. Then there exists policy π ∈ Π ipo such that J ( π ) = -∞ .

Proof. Without loss of generality, we consider a promptless setting, and assume that the response space is Y = { y 1 , y 2 , y 3 } . Again without loss of generality, we assume π ref only covers y 1 and y 2 , and thus Assumption 4.1 breaks. Specifically, let π ref ( y 1 ) = π ref ( y 2 ) = 1 / 2 . Then we have

which gives

and thus we have the relation that

Let π ipo ( y 2 ) = α ∈ (0 , 1] , then for any α such that π ipo ( y 3 ) = 1 -(1 + exp( p ∗ ( y 1 ≻ y 2 ) /β )) α > 0 , we will have that KL ( π ipo || π ref ) is unbounded, and thus we complete the proof.

C.2 DPO Has Vacuous Forward KL

In this section, we show that in the worst case, the forward KL of DPO is vacuously large. We first see how we can relate the forward KL divergence of the DPO policy with the reward learning guarantee. Consider any DPO policy π dpo and its corresponding reward model ̂ r dpo . By construction of the DPO algorithm, we have, for any x, y pair that is covered in the dataset, π dpo ( y | x ) =

π ref ( y | x ) exp( ̂ r dpo ( x,y ) /β ) Z ( x ) , where Z ( x ) = ∑ y π ref ( y | x ) exp( ̂ r dpo ( x, y ) /β ) . Then the forward KL divergence is

Although the first term can be easily related to the reward learning guarantee, the second term ( E x ∼ ρ [log( Z ( x ))] ) can unfortunately be vacuous without further assumptions. We formalize in the following result:

Proposition C.2. There exist π dpo such that Assumption 4.3 holds, but KL ( π ref || π dpo ) is arbitrarily large.

̸

Proof. First without loss of generality let us consider that r ∗ > 0 . Now suppose there exists ˜ y such that π ref (˜ y | x ) = 1 n 4 for all x , where n will be determined soon. Now suppose that for all x , ̂ r dpo ( x, ˜ y ) -r ∗ ( x, ˜ y ) = n and ̂ r dpo ( x, y ) = r ∗ ( x, y ) for all y = ˜ y . Now we can check that

which is diminishing if we take n to be big enough. We can also check that

and thus the first term will have little impact on the final bound. However, the second term can be lower bounded as follows:

Putting everything together, we have

and since we can take n arbitrarily big we complete the proof.

D Omitted Proofs

D.1 Proof of Proposition 4.1

Proposition D.1 (Restatement of Proposition 4.1) . Denote π ref as any reference policy such that Assumption 4.1 breaks. Let Π dpo be the set of DPO returned policies such that Assumption 4.3 holds. Then there exists policy π ∈ Π dpo such that J ( π ) = -∞ .

Proof. Again as in the proof sketch, without loss of generality, we consider a promptless setting, and assume that the response space is Y = { y 1 , y 2 , y 3 } . Again without loss of generality, we assume π ref only covers y 1 and y 2 , and thus Assumption 4.1 breaks. Now consider the optimal policy

where Z ∗ = ∑ y ∈Y π ref ( y | x ) exp( r ∗ ( y ) /β ) , note that by construction π ∗ ( y 3 ) = 0 .

Then consider the following policy π such that

Then we have

thus π satisfies Assumption 4.3. Rearranging we can see that π ( y 1 ) < π ∗ ( y 1 ) and π ( y 2 ) < π ∗ ( y 2 ) . Now since π ∗ = 0 , we have

and combine we get π ( y 3 ) > 0 , which implies KL ( π || π ref ) is unbounded, since π ref ( y 3 ) = 0 .

D.2 Proof of Theorem 4.2

In this section we prove Theorem 4.2:

Theorem D.1 (Restatement of Theorem 4.2) . Suppose that Assumption 4.4 holds. Then for any reference policy π ref such that Assumption 4.2 holds with ε kl = 2 R ′ β , for any RLHF policy π rlhf with ̂ r such that (c.r. Assumption 4.3),

or more generally, the event in Lemma B.2 holds for ̂ r , we have

To prove this we first prove the following lemma so we can leverage Assumption 4.2:

Lemma D.1 (Restatement of Lemma 4.1) . Suppose that Assumption 4.4 holds. Then for any RLHF policy π rlhf , we have that

Proof. since we have that π rlhf ( y | x ) = π ref ( y | x ) exp(ˆ r ( x,y ) /β ) Z ( x ) for all x ∈ supp ( ρ ) , y ∈ Y , we have

Plugging in the definition of Z ( x ) we get

due to Jensen's inequality. Thus we have

Now with Lemma 4.1, we can prove Theorem 4.2:

Proof. By Lemma B.1, we have

D.3 Proof of Proposition 6.1

Proposition D.2 (Restatement of Proposition 6.1) . Under linear function approximation (Definition 6.1), there exists dataset collected from distribution µ that does not cover y ∗ , i.e., µ ( y ∗ ) = 0 , but has global coverage in the linear function approximation setting [54]: let Σ µ = E y ∼ µ ϕ ( y ) ϕ ( y ) ⊤ , then for all π , E y ∼ π ∥ ϕ ( y ) ∥ 2 Σ -1 µ ≤ C π . Then DPO will return a policy π dpo such that π dpo ( y ∗ ) > 0 , but π dpo ( y ) ≤ π ref ( y ) for all y in the offline data support, i.e., µ ( y ) > 0 .

Proof. Consider a response space Y = { y 1 , y 2 , y 3 } , with ϕ ( y 1 ) = [1 , 0] , ϕ ( y 2 ) = [1 / 2 , 1 / 2] , ϕ ( y 3 ) = [0 , 1] Let w ref = [1 , 1] , then we have π ref ( y i ) = 1 / 3 , ∀ i ∈ { 1 , 2 , 3 } . Let the ground truth reward function r ∗ ( y ) = [10 , 1] ⊤ ϕ ( y ) , and suppose supp ( µ ) = { y 1 , y 2 } , i.e., the data only covers y 1 and y 2 , and µ ( y 3 ) = 0 . And as always, the preference is based on the ground truth reward function under the Bradley-Terry model.

We can first check that the data distribution indeed has global coverage: for all π we have E y ∼ π ∥ ϕ ( y ) ∥ 2 Σ -1 µ ≤ C π . If we parameterize ̂ r ( y ) = ̂ w ⊤ ϕ ( y ) (or in case of DPO, we can still check and see that ̂ r dpo ( y ) = ̂ w dpo ⊤ ϕ ( y ) because of the softmax linear parametrization of the policies), for either direct reward learning or DPO, we can have the learned reward function ̂ r ( y ) = [10 , 1] ⊤ ϕ ( y ) + c , where c is the constant reward shift (c.r. Eq. (5)). Then a simple calculation (by π ( y ) ∝ π ref ( y ) exp( ̂ r ( y ) /β ) ) shows that, as long as c is small enough, the policies will decrease the likelihood of y 1 and y 2 and increase the likelihood of y 3 .

E Synthetic experiment for extrapolation

E.1 Extrapolation with function approximation

We first describe our experiment setup. We consider linear function approximation setting where we have 100 responses ( |Y| = 100 ). We consider a 16-dimensional feature vector ϕ : Y → R 16 , and we generate ϕ ( y ) by simply sampling 99 random 16-dimensional vectors where the ℓ 1 norm of each vector is 1. We add one final ϕ ( y ) = [1 , 0 , 0 , . . . ] .

We construct the implicit human reward r ∗ ( y ) = w ∗⊤ ϕ ( y ) , where w ∗ = [5 , ... ] , and the rest of the entries are sampled from Unif(-2,2).

We parametrize the policies as softmax linear policies, i.e., we parametrize each policy π with w π ∈ R 16 such that π ( y ) = w π ⊤ ϕ ( y ) ∑ y ∈Y w π ⊤ ϕ ( y ) . One can check in this formulation the implicit reward in DPO ( ̂ r dpo ) is linear in ϕ .

We generate 10000 preference pairs, according to the BT model under r ∗ , for the first 50 responses. We checked that the first responses indeed span R 16 . Thus the offline data has global coverage in linear function approximation setting.

For on-policy RL methods, we first train a reward model. Then we simply perform gradient descent on the KL-regularized bandit loss (we assume π ref is uniform). For DPO, we simply perform SGD

0

200

400

600

800

1000

Iteration

-10

-8

-6

-4

-2

0

Log Probability

Online RLHF with Linear FA

Preferred Probability

Best Probability

0

1000

2000

3000

4000

5000

Iteration

-7

-6

-5

-4

-3

-2

Log Probability

DPO with Linear FA

0

1000

2000

3000

4000

5000

Iteration

0.00200

0.00225

0.00250

0.00275

0.00300

0.00325

0.00350

0.00375

0.00400

Probability

DPO without FA

Out of Distribution Probability

on the offline preference dataset. We track two qualities over the training: the mean log probability of a random subset of preferred responses, and the log probability of best response ϕ ( y ) = [1 , 0 , 0 , . . . ] . We plot the results in Figure 2. We observe that both methods have the extrapolation behavior - the probability of preferred responses decays but the probability of the optimal response goes up.

E.2 Extrapolation without function approximation

Now we describe the setting where function approximation fails, and this reduces to a Multi-arm bandit setting. We set |Y| = 500 , and the offline data only covers the first half of the responses. The r ∗ ( y ) is set by sampling from Unif(-10,10), and we generate 10000 offline samples by uniformly sample pairs of responses from the first half of the response space, and then label them with BT model under r ∗ . We train DPO with 5000 iterations, and plot the mean probability of the responses outside of the data support in Figure 3: we observe that the mean probability of the out-of-distribution responses are increasing, however, this could be an undesirable behavior because the reward of the out-of-distribution responses could be arbitrarily bad.

F Details of Section 5

F.1 Theoretical guarantee

In this section, we consider the constrained optimization version of HyPO (Eq. (6)). Note that the reward function class is identical to DPO, i.e., R hypo = { β log ( π ( y | x ) π ref ( y | x ) Z ( x ) ) | π ∈ Π } , where Z ( x ) is the partition function. Then for each output policy π hypo , we can denote its implicit reward function

̂ r hypo ( x, y ) := β π hypo ( y | x ) π ref ( y | x ) · Z ( x ) , and similarly to Theorem 4.2, we can obtain the following guarantee in the partial coverage condition:

Theorem F.1. For any reference policy π ref such that Assumption 4.2 holds with ε kl = 2 R ′ β , for any HyPO policy π hypo such that the event in Lemma B.2 holds, i.e.,

we have

Proof. The proof mostly follows the proof of Theorem 4.2. It remains to show the following two properties:

1) Note that Theorem 4.2 requires Assumption 4.4, which does not hold for ̂ r hypo (note that ̂ r hypo is only bounded under ρ , but not for all x ), but we only use it to prove the sufficient condition in Lemma 4.1, which is satisfied by the constraint of HyPO.

2) We need to check that the premise of Lemma B.1 holds, i.e.,

note that with the reparametrization between π hypo and ̂ r hypo , π hypo is always among the minimizer of the unconstrained policy set, so we can still invoke Lemma B.2. The rest of the proof now follows the proof of Theorem 4.2 so we omit the details.

Finally, we remark the connection to the negative result of DPO, i.e, Proposition 4.1: note that given KL ( π hypo || π ref ) ≤ ∞ , we have that for all x such that ρ ( x ) > 0 , we have for all y , β log ( π hypo ( y | x ) π ref ( y | x ) ) < ∞ , (again with the convention that 0 0 = 0 ), which breaks the construction of Proposition 4.1.

F.2 Experiment details

F.2.1 Summarization

In this section, we provide more details of our summarization experiment. We use the Pythia 1.4B and 2.8B model [5] with hugging face model cards: EleutherAI/pythia-1.4b-deduped and EleutherAI/pythia-2.8b-deduped. The TL;DR dataset is available at https://github.com/openai/ summarize-from-feedback . The human reference dataset contains 117k training, 6.45K validation and 6.55K testing data. The preference dataset contains 92.9K training and 83.8K validation data. The reward evaluation and KL computation is performed on the whole validation data of the reference dataset. The GPT winrate is computed on a subset of 600 samples from the validation data. The GPT API checkpoint we use is gpt-4-0613. We follow the standard prompt for the winrate evaluation (e.g., see Appendix D.3 of [18]). Below we provide the hyperparameter for HyPO and DPO. Note that to optmize the online KL, we use Reinforce with Leave One Out (RLOO) [24] with two generations per prompt ( k = 2 ) and optimize trajectory-level KL.

For our experiment, we run on a cluster of mixture of Nvidia A6000 and L40 GPUs with 48 GB VRAM. We use 4 GPUs in parallel for training, and for DPO the experiment time varies from 1 hour to 2 hours to finish, and for HyPO the time varies between 4 hours to 5 hours.

F.2.2 General Chat

For the base model of general chat experiments, we use Llama3-8B-Instruct [27] with hugging face model card: meta-llama/Meta-Llama-3-8B-Instruct. The dataset card of the Ultrafeedback dataset [14] is HuggingFaceH4/ultrafeedback_binarized. In addition to the KL penalty, in the general chat task we add an additional length penalty, and the online penalty of a generation y with context x becomes log ( π ( y | x ) π ref ( y | x ) ) + α | y | . We summarize the hyperparameter of each baseline below.

We run the general chat experiment on a node of 8 Nvidia A100 80GB GPUs. DPO takes 3 hours to train one epoch while HyPO takes 18 hours to train one epoch.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have theoretical and empirical results supporting the claims made in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We provide a discussion about the limitations of our work in the discussion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We clearly state all our assumptions in all theorem statements, and the proofs can be found in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide pseudocode, hyparameter table and code in this submission.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We use public data on hugging face, and we submit the code for this submission.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all the training details and hyperparameters.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We repeat all our experiments over 3 seeds.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provided run time and type of GPUs we used in our experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We respect the code of conduct.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the broader impact in the discussion section.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release data or models in this work

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cited all the dataset and models used in this work.

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release any new asset.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not have human subject.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No human subject.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the optimization and learning methods for ``online'' RLHF and contrastive offline methods (DPO, IPO). The authors aim to understand the separation between these two type of methods, where they are different in terms of whether new responses can be sampled or not. The authors state that the difference in the reward parameterization is key to the separation and propose global coverage and local coverage to capture such a difference. \n\nThe theoretical insights also motivate a novel approach called Hybrid Preference Optimization (HyPO) that combines offline data with online samples, where the online samples are used to control the KL divergence. Empirical results are provided to verify the effectiveness of the proposed methods.\n\nSTRENGTHS:\n- The authors study the theory of RLHF under the KL-regularized target, which is more close to the practice, as compared to many previous works using the non-regularized reward;\n- The notion of local coverage condition is novel in the literature and is very natural in the analysis of KL-regularized target. I appreciate the paper writing at the beginning of section 4, which is easy to follow and informative.  \n- Building open the notions of local convergence and global convergence, the authors show a clear separation between the offline algorithms like DPO and online RL-based algorithms. This aligns with the recent observations that the online algorithms outperform their offline counterparts with a large margin. \n- This paper takes a step further to study the differences between the DPO and RLHF. While the original DPO paper states that the learning of DPO is equivalent to the RLHF, the empirical results do not align with this. The discussion related to the parametrization of reward for assumption 4.4 explains such a difference in practice. \n- Overall, I feel that the story of this paper is complete. Lemma 4.1 and the discussion  around assumption 4.4 clearly show that the offline algorithms can search for the policy with a large KL (possibly due to the parameterization of reward). This not only aligns with the separation between global coverage condition and the local coverage condition, but also motivates the practical algorithmic design to explicitly control the KL divergence.\n\nWEAKNESSES:\nThis is not a weakness but some clarification on the terminology. In the literature of RL theory, particularly for the preference learning paper, the learning (online exploration) is mentioned with querying the human preference so as to learn the $r^*$. In the setup of this paper, the online data is only used to compute the KL loss without querying the human feedback. Therefore, it is more related to an intermediate setting that we use the offline preference dataset to construct a proxy reward function and we are allowed to query the model to get new responses but not the human feedback. See the RSO [1], some discussions in [2] for proposing multi-step RSO, and some discussions in [3]. \n\n[1] Statistical rejection sampling improves preference optimization.\n[2] Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint\n[3] Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data.\n\nThen, it would be interesting to see the empirical results in terms of the ONLINE setting where we can query the new response and the new human preference signals because a lot recent works show that even with the DPO, the online framework outperforms the original one with a large margin. Moreover, this is standard in the PPO literature like Chat-GPT, Claude, and LLaMA2.\n\n[4] Training language models to follow instructions with human feedback\n[5] Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback\n[6] Llama 2: Open Foundation and Fine-Tuned Chat Models\n\nQUESTIONS:\nsee weakness part\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considers the statistical separation between contrastive algorithms (DPO and IPO) and RLHF. It proves that DPO/IPO requires global convergence assumption which is in general a very strong assumption while on the other hand RLHF only requires local coverage. This separation stems from the explicit KL constraint in the objective of RLHF while the implicit reward in contrastive algorithms can results in unbounded KL. Given the observation, a hybrid (offline + online) algorithm is proposed: the preference optimization remains offline DPO however it draws online samples to enforce KL constraints on DPO policy, recovering the statistical guarantee of RLHF while attains the practical computational strengths (w/o value and reward models) of DPO.\n\nSTRENGTHS:\n- Well written and easy to follow; understanding DPO and RLHF is important towards better preference optimization algorithms.\n\n- Good technical qualities and offer viable insights into DPO.\n\n- The proposed hybrid algorithm recovers the statistical guarantee of RLHF while keeps the practical computational strengths of DPO (w/o using value and reward models).\n\nWEAKNESSES:\nI only have a couple of minor comments:\n\n- Second sentence of L562 should be appended to L231 to make the proof sketch immediately clear.\n\n- Theorem E.1 could be relocated to Section 5 to be more self-contained.\n\nQUESTIONS:\n- I might have overlooked, what is the intuition behind online KL evaluation? Is it possible to enforce such KL constraint purely in offline setting?\n\nLIMITATIONS:\nOverall, I am convinced this is a good submission with good technical quality. However, as I did not extensively follow the theoretical results on preference optimization, I am not able to comment on the novelty of this work, hence I give confidence score of 3. \n\nIt would also be beneficial if the authors could create a table summarizing previous works and highlighting their contributions in comparison to prior research.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on the paradigm of fine-tuning large language models (LLMs) using human preference data. It delves into two primary techniques: online reinforcement learning (RL) and offline contrastive methods. The authors challenge the previous notion of these techniques being equivalent by conducting a theoretical analysis through the lens of dataset coverage. They introduce the concepts of global and partial coverage conditions, proving that the former is necessary for offline methods using forward KL like DPO to converge to the optimal policy, while the latter is sufficient for online RL methods using reverse KL. The paper proposes a hybrid preference optimization (HyPO) algorithm, demonstrating its empirical superiority over DPO while maintaining some computational efficiency. The authors also provide a coverage-based explanation for why RL and offline contrastive methods might decrease the probability of preferred responses.\n\nSTRENGTHS:\n1. The paper provides a rigorous mathematical analysis of the different conditions under which offline and online methods have provable performance guarantees, contributing to the theoretical foundation of preference learning in RL.\n\n2. The authors introduce the HyPO algorithm and support their theoretical findings with empirical results, demonstrating the effectiveness of HyPO on the TL;DR dataset.\n\nWEAKNESSES:\n1. The experiment is not sufficient. Since it is done with Pythia 1.4B and the TL;DR dataset, it is unclear if the proposed method is still valid on larger models and other datasets. While theoretically interesting, it is unclear if using reverse KL instead of forward KL can lead to a significant performance gain in practice. \n\n2. The proposed HyPO method only uses online samples to calculate the KL loss rather than to collect new preference feedback. While simpler, it may fail to fully leverage the benefits of online methods.\n\nQUESTIONS:\nSee the Weaknesses section.\n\nLIMITATIONS:\nThe authors discussed the limitations well in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies learning from human preference feedback for aligning large language models (LLMs). Many existing works share the same observation that offline alignment methods such as DPO underperforms their online counterpart such as PPO. But this phenomenon has not been well understood. This paper studies this difference through the lens of coverage. It unveils that the KL constrain in offline alignment methods can be violated due to incomplete coverage and thus offline methods may find bad solutions. Inspired by this insight, the authors propose a new method, hybrid preference optimization (HyPO), which combines the offline preference update with the online KL regularization. Experiments on the summarization task demonstrate that HyPO is better than DPO. This paper also sheds light on a counterintuitive observation that DPO often decreases the likelihood of preferred responses through an illustrative example.\n\nSTRENGTHS:\nThis work addresses an important matter and provides insights on a widely discussed question - why do offline alignment methods often underperform online methods? The paper is well written and easy to follow. The theoretical claims are sensible and easy to understand. I believe the community will benefit a lot from this work.\n\nWEAKNESSES:\nThe computational cost of the proposed HyPO algorithm is not sufficiently discussed. Usually the most computational cost in online alignment methods comes from the sampling process of LLMs. In practice, the computational cost plays an important role in comparing algorithms. Thus it will provide useful guidance if the computational cost is well discussed in the paper.\n\nThe discussion in Section 6 is bit handwavy. While Example 6.1 provides good intuition, a more rigorous analysis is needed to justify the claim.\n\nOne minor suggestion on presentation: It will be good to provide the PPO numbers in Table 1 to help the readers compare HyPO to PPO more easily.\n\nQUESTIONS:\nIn the TL;DR experiment, I am curious to know what happens if the authors also train HyPO for 4 epochs. Would it perform better than PPO?\n\nLIMITATIONS:\nThis paper includes a fairly thorough discussion on the limitations at the end. I would like to add one point: the analysis in this work only focuses on the solution space. It explains why offline methods may find bad solutions, but doesn't explain why they often find the bad ones in practice. As far as I know, people often use a very low learning rate for DPO in practice. For example, the original paper used 1E-6. One possible reason for such low learning rates is that it compensates the ineffective KL regularization. Despite such efforts, DPO still finds bad solutions. It will be insightful if we get a better understanding of why bad solutions seem inevitable.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-19"}