Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer

Authors: zhihan  liu
miao  lu
shenao  zhang
boyi  liu
hongyi  guo
yingxiang  yang
jose  blanchet
zhaoran  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 22:16:41

Abstract: Aligning generative models with human preference via RLHF typically suffers from overoptimization, where an imperfectly learned reward model can misguide the generative model to output even undesired responses. We investigate this problem in a principled manner by identifying the source of the issue as the distributional shift and uncertainty of human preference in dataset. To mitigate overoptimization, we first propose a theoretical algorithm which optimizes the policy against an adversarially chosen reward model, one that simultaneously minimizes its MLE loss and a reward penalty term. The penalty pessimistically biases the uncertain rewards so as to prevent the policy from choosing actions with spursiouly high proxy rewards, resulting in provable sample efficiency of the algorithm under a partial coverage style condition. Moving from theory to practice, the proposed algorithm further enjoys an equivalent but surprisingly easy to implement form. With a clever usage of the equivalence between reward models and the corresponding optimal policy, the algorithm features a simple objective that combines (i) a preference optimization loss that directly aligns the policy with human preference, and (ii) a supervised learning loss which explicitly imitates the policy with a baseline distribution. In the context of aligning large language models (LLM), this objective fuses the direct preference optimization (DPO) loss with the supervised fune-tuning (SFT) loss to help mitigate the overoptimization towards undesired responses, for which we name the algorithm Regularized Preference Optimization (RPO).
Experiments of aligning LLMs demonstrate the improved performance of our method when compared with DPO baselines. 
Our work sheds light on the interplay between preference optimization and SFT in tuning LLMs with both theoretical guarantees and empirical evidence.

Main paper: 
 introduction: 
 A key step in building state-of-the-art LLMs is Reinforcement Learning from Human Feedback (RLHF) [12, 87] , which aligns pretrained LLMs with human preferences using human assessment data, making the model more helpful, truthful, and harmless [42, 10] . Typically, RLHF first learns a reward model from data (pair-wise comparisons of responses) to quantify the human preferences of LLM outputs. Then it fine-tunes the LLM to maximize the learned reward using RL techniques.In this pipeline, a crucial challenge is reward overoptimization or reward hacking [40, 62, 25] . Since the reward model is learned from finite data, it might not be perfectly aligned with the underlying human preference. Optimizing the LLM towards such an imperfectly learned and potentially overfitted proposed by [46] . Our algorithm (RPO) significantly alleviates this decrease. See more discussions in Section 6.reward model leads to performance degeneration and a substantial decrease in the probability of choosing the preferred responses in the data [26, 45] . Given the importance of RLHF and the outlined challenge, a crucial research question is: How to mitigate reward overoptimization in RLHF in a principled and efficient manner for better alignment?To answer the question, we model RLHF as an offline contextual bandit [42] and ascribe overoptimzation to distributional shifts and reward uncertainty. Intuitively, when fine-tuning an LLM, the response (action) distribution of the tuned LLM could deviate from that of the training data. For the out-of-distribution responses, which are dissimilar with (or not well covered by) the responses in the data, the high inherent uncertainty of underlying human preferences could make the learned reward model misleading for out-of-distribution responses. In this situation, reward overoptimization can occur because the LLM is fine-tuned towards maximizing a reward model with defective out-ofdistribution prediction, giving a potential consequence that the LLM responses are favored by the learned reward but less preferred by a human [86] . We illustrate these types of distributional shift and reward uncertainty issues inherent to overoptimization in Figure 1 .In this paper, we propose a new RLHF algorithm to mitigate reward overoptimization. From a high level, our theoretical algorithm seeks the best LLM for an adversarially chosen reward model that minimizes the sum of its maximum likelihood estimation loss and its own expected reward value. Intuitively, since the reward value is also minimized when minimizing the sum, it can automatically prevent the misleadingly high reward caused by the uncertainty inherent in having access to finite data. Furthermore, we show that the theoretical algorithm enjoys an easy implementation: it simply adopts a supervised fine-tuning (SFT) loss as a regularizer during training. By explicitly regularizing the LLM to imitate high-quality responses (e.g., preferred responses in dataset), the algorithm can effectively mitigate the issue of overoptimization. We establish theoretical guarantees and conduct experiments to demonstrate our findings, which we summarize next.We summarize our contributions in three areas as follows.A theoretical algorithm under general function approximation. Our first contribution is a new theoretical algorithm (Algorithm 1). It features an unconstrained maximin problem, outputting the optimal policy (LLM) against an adversarially chosen reward model that minimizes the summation of: (a) the MLE loss for estimating the underlying reward; and (b) a reward expected value term as a penalty that aims to prevent spuriously high reward estimation caused by data uncertainty and insufficient coverage. Algorithm 1 is compatible with general function approximations of the reward model, meaning that we do not impose any specific structural form to the hypothesis class of reward, demonstrating its generality, especially in language modeling.In this regime of reward class, we establish the finite-sample suboptimality gap of Algorithm 1 as O(C 2 coverage N R /N ) when competing with any LLM in terms of the underlying true human reward (Theorem 5.3). Here N is the number of human comparison data, N R is the complexity of the reward model class R, and C coverage characterizes the coverage of the preference dataset with respect to the response distribution of the LLM to compete (please see Assumption 5.2 for details). This indicates that, as long as the training data well cover the LLM π to compete, the algorithm is guaranteed to align an LLM to output responses as good as π in terms of human reward, without suffering from overoptimization caused by distributional shifts and inherent uncertainty in human preference.An easy-to-implement practical objective. Moving towards practice, we show that the objective of Algorithm 1 adopts a surprisingly simple and equivalent form for its use in practice. Specifically, with mild regularity conditions, we prove that the maximin objective (Algorithm1) is equivalent to the corresponding minimax objective, which is further reduced to a single minimization problem for the reward model since its inner problem adopts a closed form solution. Inspired from recent progress in RLHF that explores reward-model-free methods to align LLMs [46] , we further re-parameterize the reward model via its corresponding KL-regularized optimal policy. Then the minimization objective of the reward modeling naturally translates to a target for directly aligning the LLM, which we call Regularized Preference Optimization (RPO; Algorithm 2). The objective of RPO features a simple weighted combination of two losses: RPO objective = Preference optimization loss + Imitation (SFT) loss.Here the Preference optimization loss coincides with the DPO [46] objective, tending to optimize the LLM towards maximizing the underlying true reward. The Imitation (SFT) loss explicitly supervises the LLM to mimic the responses from a proper distribution well covered by the dataset. The choice of the distribution is guided and justified by our theory of Algorithm 1, but can also be flexibly adapted in practice, e.g., the preferred response in the dataset, or the responses of the initial model.We highlight that the Imitation (SFT) loss serves as an important term to mitigate overoptimization. Even though the original DPO objective has already involved a KL regularization between the tuned LLM and the initial LLM, is not enough to prevent overoptimization. As we elaborate in Section 4, the KL-regularization weight of the DPO objective could only control the scale of the gradient per training example, while the RPO objective can further modify the gradient direction. Calling back to the theoretical Algorithm 1, such a modification of gradient direction originates from the reward penalty in the adversarial objective for the reward model. This modification, as we expose in our theoretical analysis, helps to mitigate overoptimization. Thus, incoporating SFT loss in RLHF gives you a regularizer that provably mitigates overoptimization.Empirical evaluations. Following the training setup of two series of released chat models Zephyr-7b-beta (trained on the Ultrafeedback dataset [16] by DPO) and Zephyr-7b-gemma (trained on the Argilla-DPO-Mix-7K dataset [3] by DPO) [64] , we implement RPO for the beta series and gemma series respectively to show that: (i) RPO is a flexible plug-in module and can be applied to different reference models. (ii) RPO can alleviate the overoptimization issue. (iii) RPO consistently achieves better alignment performance than DPO in in-data distribution. (iv) RPO can also achieve consistently better performance in standard LLM benchmarks like MT-bench and AlpacaEval 2.0, which shows its potential of mitigating overoptimization for better alignment performance, justifying our theory.Related works. Due to space limitation, we refer the readers to Appendix A for a detailed discussion. 
preliminaries of rlhf: 
 In this section, we introduce the mathematical framework of studying RLHF for aligning LLMs. We adopt the framework of offline contextual bandits [42] , where we identify the context space X as the space of prompts and the action space A as the space of responses. An LLM, defined as a policy π(•|•) : X → ∆(A), takes a prompt x ∈ X as input and output a response a ∈ A from a ∼ π(•|x).Preference model. Given any reward function r : X × A → R belonging to certain reward class R that represents the "human's ratin" of LLM responses given prompts, we consider the Bradley-Terry model [9] of human preference. That is, given a prompt x ∈ X and two responses a 1 , a 0 ∈ A, the probability of a 1 being preferred to a 0 (denoted by y = 1, otherwise y = 0) is given byEQUATIONwhere σ(z) = 1/(1 + exp(-z)) is the sigmoid function. For simplicity of future discussion, we explicitly write out the dependence of the preference probability P r (•) on the reward model r ∈ R.In the section of theory, i.e., Section 5, we specify the assumptions on the reward model class R.Learning protocol. Typically, the RLHF pipeline starts from certain reference policy π ref obtained from pretraining. Then RLHF aligns the LLM based on certain human preference data. In this work, we consider offline RLHF setup, where the LLM is aligned using a fixed offline preference dataset D.It consists of N i.i.d. tuples in the form of D = {(x i , a 1 i , a 0 i , y i )} N i=1 .Here the prompt x i and the responses a 1 i , a 0 i are distributed according to: (x, a 1 , a 0 ) ∼ µ D (•), and conditioning on (x i , a 1 i , a 0 i ), y i is distributed according to (2.1) for an underlying true (but unknown) reward model r ⋆ ∈ R. The target of RLHF is to align an LLM, or equivalently, to learn a policy π, so as to maximize the expected true reward r ⋆ . Thus, we define the value function of any policy π asJ(π) = E x∼d0,a∼π(•|x) r ⋆ (x, a) .(2.2)Here we allow the prompt distribution d 0 (•) to be different from that of the offline dataset distribution µ D (•), but is assumed to be known. In the meanwhile, we consider the policies that share the same support as the reference policy π ref [71] , that is, we take a policy class Π asΠ = π : X → ∆(A) Supp(π(•|x)) ⊆ Supp(π ref (•|x)), ∀x ∈ X . (2.3)The performance gap of a learned policy π ∈ Π w.r.t. any other policy π ∈ Π is measured asGap π ( π) = J(π) -J( π), given policy π. (2.4)The goal is to propose a sample-efficient and also implementation-friendly algorithm to learn a policy π ∈ Π able to compete with any given policy π ∈ Π in terms of Gap π ( π) ≤ ε, with sample complexity polynomial in 1/ε and logarithmic in the complexity of R. 
a theory-motivated objective: 
 Our method seeks to find the best policy π against an adversarially chosen reward model r adv that minimizes a weighted sum of its expected value and the maximum likelihood estimation (MLE) loss.Intuitively, such a reward model can prevent the overoptimization issue by taking its own value into account when minimizing the MLE loss. Since the reward value is also minimized when minimizing the sum, this method prevents the misleadingly high reward caused by the uncertainty due to finite data. Formally, given two hyperparameters β, η > 0 and a "baseline policy" π base , we defineT adv β,η (π) = min r∈R ηE x∼d0,a 1 ∼π(•|x), a 0 ∼π base (•|x) r(x, a 1 )-r(x, a 0 )-β • KL π(•|x)∥π ref (•|x) +L D (r) ,where the loss function L D (•) is the average negative log-likelihood function of the BT model (2.1) (and here it becomes the cross-entropy loss) over the preference dataset D, defined asL D (r) = -E D y i log σ r(x i , a 1 i ) -r(x i , a 0 i ) +(1 -y i ) log σ r(x i , a 0 i ) -r(x i , a 1 i ) .(3.1)As we can see, T adv β,η (π) is the minimum value of a weighted sum of the MLE loss and the expected reward value of π, but with two important modifications that we explain in the following.Firstly, we subtract another expected reward of certain policy π base . This is because the BT model (2.1) essentially only uses the reward differences to define the preference probabilities. As a result, the data can only reveal information of the differences between the true reward r ⋆ of different responses [78] . Accordingly, we subtract such a baseline expected reward value to match this observation. The choice of the baseline policy is discussed in the theory part (Section 5) and experiments (Section 6).Secondly, we subtract a KL divergence between π and π ref from the expected reward, weighted by the coefficient β > 0. Such a term is for practical considerations that would be explained in Sections 4 and 5.2. We note that the KL regularized reward is commonly adopted in RLHF practice to ensure the learned policy is not far away from the reference policy [42, 71] .Finally, the overall algorithm design (Algorithm 1) is to output the policy that maximizes T adv β,η (π), i.e., π ∈ argmax π∈Π T adv β,η (π), which gives the following theoretical target:π ∈ argmax π∈Π min r∈R ηE x∼d0,a 1 ∼π(•|x), a 0 ∼π base (•|x) r(x, a 1 )-r(x, a 0 )-β • KL π(•|x)∥π ref (•|x) +L D (r) . (3.2)Given the form of (3.2), we name it the maximin objective in the sequel. Upon seeing (3.2), one might be arguing that such a theory-motivated objective seems hard to implement in practice. Nevertheless, in the coming Section 4, we demonstrate that the maximin objective (3.2) adopts an easy-to-implement equivalent form, allowing us to design a practical algorithm for aligning LLMs. a 0 ∼π base (•|x) r(x, a 1 ) -r(x, a 0 ) -β • KL π(•|x)∥π ref (•|x) + L D (r) . (4.1)Such a minimax formulation (4.1) is the starting point of our practical algorithm. The magic of (4.1) is that the inner maximization problem adopts a closed form solution, which further simplifies such an objective. To see this, note that given any reward model r ∈ R, the inner problem is equivalent tomax π∈Π E x∼d0,a∼π(•|x) r(x, a) -β • KL π(•|x)∥π ref (•|x) . (4.2)It has been well established that the policy that maximizes the KL-regularized expected reward (4.2) has a closed form solution. Due to its importance, we present it as the following lemma.Lemma 4.1 (Oracle optimal KL-regularized policy). Given any reward model r ∈ R, the optimal policy π r to the maximization problem (4.2) is given byπ r (•|x) = 1 Z r (x) • π ref (•|x) • exp β -1 r(x, •) , Z r (x) = a∈A exp β -1 r(x, a) dπ ref (a|x),and correspondingly the optimal value of (4.2) is given by(4.2) = E x∼d0 [β • log(Z r (x))].Specifically, by Lemma 4.1, we can solve the inner maximization problem in (4.1) and obtain that(4.1) = min r∈R ηE x∼d0,a 0 ∼π base (•|x) -r(x, a 0 ) + β • log (Z r (x)) + L D (r) .Furthermore, from Lemma 4.1, one immediately see that given any reward model r ∈ R, we can reparameterize it via its corresponding optimal KL-regularized policy π r [46] , that is, EQUATION(•|x) -β • log(π r (a 0 |x)) + L D β • log π r (•|•) π ref (•|•) ,then the corresponding optimal KL-regularized policy π r also solves the maximin objective (3.2).Regularized Preference Optimization. Target (4.4) gives a quite simple objective to use in practice! Since (4.4) depends on r ∈ R only through its corresponding optimal policy π r , one can formulate a minimization objective over a parameterized policy π θ , i.e., the LLM to be aligned, and directly optimize the parameters θ ∈ Θ. More formally, the new RLHF objective becomesmin θ∈Θ L RPO (θ) := ηβ • E x∼d0,a 0 ∼π base (•|x) -log(π θ (a 0 |x))Imitation (SFT) lossEQUATIONIn (4.5), the second term coincides with the objective of DPO algorithm [46] which optimizes the policy towards maximizing the underlying true reward, and the first term stands for a regularization term weighted by η • β which explicitly regularizes the policy to imitate the baseline policy. Therefore, we name the resulting algorithm as Regularized Preference Optimization (RPO). We summarize it abstractly in Algorithm 2. As for DPO, implementing RPO does not require to maintain a reward model r. Thus it is computationally more friendly compared to reward-based algorithms.How does RPO improve DPO? We illustrate the effect of the imitation loss by analyzing the gradient of the RPO target L RPO (θ) in (4.5). Notice that by (4.5) we have∇ θ L RPO (θ) = ηβ • E x∼d0,a 0 ∼π base (•|x) -∇ θ log(π θ (a 0 |x))increase the alignment with the baseline policy+ ∇ θ L DPO (θ) decrease the DPO Loss ,where the derivative of the DPO loss ∇ θ L DPO (θ) is given by the following,∇ θ L DPO (θ) = -E D β •σ r θ (x, a rej ) -r θ (x, a cho ) gradient weight • ∇ θ log π θ (a cho |x)-∇ θ log π θ (a rej |x) .For simplicity we denoter θ (x, a) = β • log(π θ (x, a))/ log(π ref (x, a)), a cho for the chosen response and a rej for the rejected response. Intuitively, RPO (4.5) modifies the gradient direction of DPO to ensure the alignment with the baseline policy π base , and the hyper-parameter η controls the power of alignment. In comparison, the hyper-parameter β in DPO only controls the gradient weight when increasing the likelihood of a cho and decreasing the likelihood a rej . In this perspective, the hyperparameter β only changes the scale of the gradient instead of the direction. By introducing η, we stabilize the training and reduce the side-effect of uncertain labels in data to prevent overoptimization. 
theoretical analysis: 
 In this section, we establish theoretical analysis for Algorithms 1 and 2. We take the space of prompts and responses as compact subsets X ⊆ R d X and A ⊆ R d A . We take the policy class Π as (2.3).Objective (Algorithm 1) Assumption 5.1 (True reward model). We assume that the true reward model r ⋆ ∈ R, and for any r ∈ R and (x, a) ∈ X × A, it holds that r(x, a) ∈ [0, R].Assumption 5.2 (Partial coverage coefficient [78] ). Given a policy π ∈ Π, the coverage coefficient of the offline dataset distribution µ D w.r.t. reward model class R, policy π, and the baseline policy π base , denoted by C µ D (R; π, π base ), is defined asmax        0, sup r∈R E x∼d0,a 1 ∼π(•|x),a 0 ∼π base (•|x) (r ⋆ (x, a 1 ) -r ⋆ (x, a 0 )) -(r(x, a 1 ) -r(x, a 0 )) E (x,a 1 ,a 0 )∼µ D (r ⋆ (x, a 1 ) -r ⋆ (x, a 0 )) -(r(x, a 1 ) -r(x, a 0 )) 2        .We assume that C µ D (R; π, π base ) < +∞ for the policy π to compete. We remark that the quantityC µ D (R; π, π base ) is upper bounded by the density ratio ∥d 0 ⊗ π ⊗ π base /µ D ∥ ∞ .Assumption 5.1 is standard in sample complexity analysis [85, 78, 74] . Assumption 5.2 characterizes how well the dataset D covers the policy π to compete. To achieve provable sample efficiency, we only require that D covers the target policy π, a weak partial coverage style assumption for theoretical analysis. To illustrate it, when calling back to Figure 1 , the data distribution therein well covers those nearly optimal responses under r ⋆ , but does not sufficiently cover the responses with low r ⋆ .Under such a partial coverage data condition, however, human preference of responses a ∈ A that are not well covered by the dataset D can be poorly estimated, misguiding the policy π to behave suboptimally if it is overoptimized (recall Figure 1 ). Fortunately, the following theorem shows that Algorithm 1 provably mitigates the overoptimization issue and achieves a finite-sample convergence of the suboptimality gap (2.4) competing with π. Proof is in Appendix D.Theorem 5.3 (Suboptimality of Algorithm 1). Taking the policy class Π as (2.3), supposing that Assumptions 5.1 and 5.2 hold, and assuming that the reward model class R has a finite ε-epsiloncovering number under ∥ • ∥ ∞ -norm N ε (R, ∥ • ∥ ∞ ) < +∞ with ε = (6 • (1 + e R ) • N ) -1 . Setting η = (1 + exp(R)) -2 • 24 log (N ε (R, ∥ • ∥ ∞ )/δ) /N , β = 1/ √ N in Algorithm 1.Then the output policy π of Algorithm 1 satisfies that with probability at least 1δ,Gap π ( π) ≤ √ 6 1 + exp(R) 2 C µ D (R; π, π base ) 2 + 1 ι + 4E x∼d0 KL π(•|x)∥π ref (•|x) 4 √ N , where ι = log (N ε (R, ∥ • ∥ ∞ )/δ) with ε = (6 • (1 + e R ) • N ) -1 .Here, N denotes the number of preference pairs in D, R denotes the upper bound of the reward models, and the partial coverage coefficient C µ D (R; π, π base ) is defined in Assumption 5.2. Remark 5.4 (Choice of the baseline policy). As is indicated by Assumption 5.2, the least requirement is that π base can be covered by the offline data distribution. E.g., we can take π base as the distribution of the preferred responses in the data. In this case, the SFT loss in RPO explicitly regularizes the LLM to imitate the preferred responses. We choose this type of baseline policy in our experiments.Now we formally show that the theoretical target (maximin objective (3.2)) and the target for practical algorithm design (minimax objective (4.1)) are equivalent under certain regularity conditions. This can naturally extend the sample complexity of Algorithm 1 (Section 5.1) to that of minimax-based algorithms in Section 4, providing the theoretical guarantee for our practical algorithm design (RPO).First, for notational simplicity, we denote the optimization target we investigate in Sections 3 and 4 asEQUATIONfor any (π, r) ∈ Π × R. Our result relies on the following assumptions on the reward model class R.Assumption 5.5 (Regularity of reward model class). We assume the following things on the reward model class R: (i) the space R is a compact topological space; (ii) the function ϕ in (5.1) is convex-like on R, that is, for any r 1 , r 2 ∈ R and α ∈ [0, 1], there existsEQUATIONWe note if R is convex, e.g., a linear model class [85, 71, 86] or more general the Lipschitz continuous model class R, we can directly obtain that the function ϕ(π, •) is convex over R (since the dependence on r ∈ R is linear terms plus a convex loss L D of r ∈ R), which implies the convex-like property (5.2). Under Assumption 5.5, it holds that (Lemma E.1)max π∈Π min r∈R ϕ(π, r) = min r∈R max π∈Π ϕ(π, r). (5.3)Furthermore, thanks to the KL-divergence regularization in ϕ which intuitively makes ϕ "strongly concave" over the policy π, (5.3) can gives us the following stronger result, proved in Appendix E.1. (5.4)Then the policy π r also satisfies the maximin objective (3.2) of Algorithm 1, that is,π r ∈ argmax π∈Π min r∈R ϕ(π, r).Theorem 5.6 shows that the optimal KL-regularized policy associated with the reward model solving the minimax objective (3.2) also solves the maximin objective (i.e., objective (4.1) of Algorithm 1). This further allows us to extend our theoretical guarantee of Algorithm 1 (Section 5.1) to that of minimax-based algorithms, justifying our practical algorithm design in Section 4.Corollary 5.7 (Suboptimality of minimax-based algorithm). Take the policy class Π in (2.3) and the reward model class satisfying Assumption 5.5. Given any given policy π to compete, if Assumption 5.2 holds for π, then under the same choice of η and β as in Theorem 5.3, the policy π r defined in (5.4)satisfies that Gap π (π r ) ≤ O(1/ √ N ) with probability at least 1 -δ.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We submit the codes in the supplementary. All the datasets, reference models, and benchmarks used in this paper are open accessed.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 In this section, we provide a detailed empirical analysis of RPO to highlight the following four key points: (1) RPO is a flexible plug-in module and can be applied to different reference models.(2) RPO can alleviate the overoptimization issue in the training phase by giving more trust to the chosen responses in the preference dataset. (3) As a justification of our theoretical analysis, RPO achieves better alignment performance than DPO in in-data distribution. (4) RPO can also achieve consistently better performance in LLM benchmarks like MT-bench [83] and AlpacaEval 2.0 [20] , which shows the potential of mitigating overoptimization for better generalization performance. The code for the experiments can be found in https://github.com/YSLIU627/ Regularized-Preference-Optimization/tree/master.Experiment setup. To show that RPO is a flexible plug-in module regardless of the reference model, we follow the training setup for two well-studied series of released chat models with around 7 billion parameters trained by DPO: Zephyr-7b-beta and Zephyr-7b-gemma [64] to implement RPO in beta and gemma series. Mirrored by their training configurations, we introduce how we select the reference model and the preference dataset for our training on these two series as follows. For the beta series, we use mistral-7b-sft-beta as the reference model π ref . RPO alleviates overoptimization. As mentioned in the introduction part, DPO is observed to have a significant and continuous decrease in log probability on chosen responses [26, 45] during training and we regard it as the consequence of overoptimization. Implied by our theory, overoptimization could arise when the model maximizes its own proxy reward formed on the responses less covered by the data. Due to the overoptimization, the model tends to disprefer the chosen responses as they are away from the maximizers of the proxy reward despite that some chosen responses are highly preferred by humans. Consistent with our theoretical conclusion, we empirically find that RPO can indeed alleviate overoptimization in DPO. During the training phase of both beta and gemma series, we observe that the log probability given by the RPO-trained model is notably higher than that given by the DPO-trained model for the chosen responses, which are shown in Fig. 1 and 2 .RPO improves the alignment ability in in-data distribution. For the in-data distribution evaluation, we select the 200 prompts (which are not used in the selection of η) in the test split of the training dataset to let the reference model, DPO, and RPO generate the response respectively. We choose GPT-4 to annotate the preference in the response pairs. Though we instruct GPT-4 to give an annotation among win, lose, and, tie (please see the full prompt in Appendix F.2), GPT-4 may still give undesired annotations. Hence, we filter all the undesired annotations and collect 150 examples for evaluation. We report the pairwise win rate among Ref., RPO, and DPO in Table 1 for both the beta and gemma series. To show a more illustrative comparison between DPO and RPO, we provide the barplot to report the number of pairwise examples annotated by GPT-4 in Fig. 3 and Fig. 4 . We observe that for both beta and gemma series, RPO has a better performance than DPO in terms of both RPO/DPO-SFT and RPO-DPO win rates. The performance improvement matches our theoretical results in Corollary 5.7, which shows the credit of the alleviation of overoptimization. RPO consistently improves the benchmark performance. We further evaluate the reference model, RPO-trained model, DPO-trained model, and the officially released DPO-trained model for both beta and gemma series in two standard LLM chat benchmarks: MT-Bench and AlpacaEval 2.0. MT-Bench is a multi-turn benchmark that contains 160 questions across eight different domains of knowledge. The score for MT-Bench is evaluated by GPT-4 on a scale from 1 to 10. AlpacaEval 2.0 is a single-turn benchmark including 805 questions on different topics, mostly focused on helpfulness.The metrics of AlpacaEval 2.0 are the win rate and Length-Control (LC) win rate compared with GPT-4 Preview (11/06), where the annotator is also GPT-4 Preview (11/06) and LC win rate is proposed to mitigate the length bias of GPT-4. The results are summarized in Table 2 , which shows that RPO consistently exceeds the performance of all the competitors (DPO, Reference model, and the officially released model trained by DPO) on MT-Bench and AlpacaEval 2.0. We also provide additional results on the pairwise win rate for these two benchmarks in Appendix F.3 to illustrate the performance improvement. Finally, we remark that RPO is a flexible plug-in module and can steadily improve the benchmark performance without changing the original training configuration or accessing extra preference data. This also sheds light on the potential of mitigating overoptimization for better alignment and generalization performance. RPO also improves the math, reasoning, and coding abilities. In addition to the MT-Bench and AlpacaEval 2.0 benchmarks, we introduce more benchmarks on the math, reasoning, and coding tasks for evaluations of the RPO algorithm. Specifically, we choose the Grade School Math 8K (GSM8K) [14] , AI2 Reasoning Challenge (ARC) [13] , and Mostly Basic Python Programming (MBPP) [4] to measure math, reasoning, and coding abilities of the model trained by RPO, respectively. To due space limitation, we refer the readers to Appendix G for the setups and results of these experiments.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Please see the dataset choice and the detailed training configurations in Section 6 and Appendix F.1.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusions: 
 This paper proposes a new algorithm that provably mitigates reward overoptimization in RLHF. We establish its finite-sample convergence under a partial coverage style data condition, and provide an equivalent practical implementation, RPO. As a flexible plug-in module, RPO exhibits consistent improvement over the DPO baseline and effectively mitigates overoptimization. Future work includes extending our idea of algorithm design to online (iterative) RLHF where preference data are collected and updated iteratively during LLM fine-tuning. We give more detailed discussions in Appendix B.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Due to the budget of time and expense, we do not report the error bar, which is also common in many RLHF literature [46, 34, 71, 64] . However, we report all the training configurations and the random seed to ensure reproducibility.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We provide the information in Appendix F.1. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In the following, we relate our work to recent lines of RLHF research on both theory and practice sides. We also review related works on reward hacking and overoptimization in RLHF.RLHF: algorithm design. The technique of RLHF [12, 87, 42, 6, 18, 55] has recently demonstrated its great importance in building the state-of-the-art LLMs, including ChatGPT [1] , Gemini [61] , Claude [2] . In the RLHF pipeline, the LLM is fine-tuned towards maximizing a learned reward model for better alignment [52, 57] with human preference using RL algorithms such as Proximal Policy Optimization (PPO; [51] ). Meanwhile, PPO-style algorithm is also known for its instability, sample-inefficiency, and especially, a high demand for proper hyperparameter tuning [22] . This thus casts prohibitive computational cost to make the most effectiveness of PPO-based RLHF methods to align LLMs, especially for the open-source community.Given that, further research on RLHF has explored various alternatives to PPO-based methods, with the most popular approach being the direct preference optimization method [82, 46] , which skips the reward learning phase and directly optimizes the LLM to align it with the human preference. Our practical implementation (RPO) also harnesses the wisdom of reward-LLM equivalence to avoid explicit reward learning followed by PPO training.Besides the original DPO algorithm [46] , ever since it popularizing the direct preference learning style method, variants of the direct preference learning approach are proposed, including but not limited to [34, 5, 71, 59, 28, 74, 44, 26, 50, 32, 80, 58, 68, 30] . Each of them aims to address further challenges of direct preference learning from varying perspectives. Specifically, the algorithm proposed by [44, 26] share similar algorithmic components as RPO proposed in this work. Both work consider SFT style regularization during preference optimization. However, theoretical understanding of how SFT loss can help alignment remains unknown. In contrast, we provide theoretical justifications to the SFT loss as an implicit adversarial regularizer that provably mitigates overoptimization in preference learning.RLHF: theoretical investigation. Initiated from the literature of dueling bandits and dueling RL [76, 7, 43] , recent success of RLHF in fine-tuning LLMs also motivates a long line of research to investigate the theoretical foundations of RLHF under different settings [11, 85, 78, 79, 67, 31, 71, 74, 19, 84] , aiming to propose provably sample-efficient algorithms to learn a human-rewardmaximizing policy from human preference signals. Our theoretical study of RLHF falls into the paradigm of offline learning from a pre-collected preference dataset, and is mostly related to the work of [85, 78, 31, 71, 74] . In this setup, the main challenge is to address the overoptimization issues due to human reward uncertainty and distributional shifts when only a fixed dataset is available. In the sequel, we compare our work with them in more detail.Existing theoretical work on provably sample-efficient offline RLHF typically suffers from two drawbacks: they are either restricted to the linear function approximations setting [85, 71] which is far from the practical situations, or are generally unable to be implemented in the LLM experiments. Typically, to encompass the pessimistic principle in the face of uncertainty, the existing literature proposes to return the optimal policy against either an estimated reward model plus a structure-aware reward uncertainty penalty [71] or the most pessimistic reward model inside a confidence region [85, 78] . Both of these two types of method involve intractable components for implementation and needs for additional algorithmic design to approximate the theoretical algorithm in practice. In contrast, our theory works in the context of general function approximations while being friendly to be implemented. Finally, we remark that, while our study focuses on the standard Bradley-Terry model of human preference with general reward function approximations, the work of [74] further considers a general human preference model. But it remains unknown how their algorithms can be efficiently implemented in practice. It serves as an interesting direction to extend our technique to RLHF with general reward model and device new practical algorithms.Finally, we mention that the algorithm design of RPO is also related to the "pessimism" principle in the standard offline RL literature. It proposes to maintain a pessimistic estimate of the policy values or constrain the policy not to take unseen actions in the data to handle the challenge of the insufficient coverage of the dataset, e.g., [29, 65, 69, 70, 47, 75, 72, 36, 54, 77, 39, 48, 53, 8, 38, 33, 24] . In contrast, we consider the offline RLHF problem and the techniques to obtain the objective of the RPO algorithm (see Section 4) along with its sample complexity analysis are new and different from these works.Reward hacking and overoptimization in RLHF for LLM. As is discussed in the introduction, the challenge of reward hacking or overoptimization may prevent the successful alignment of LLMs, degenerating the performance of an LLM because of maximizing an imperfect, overfitted, and misgeneralized proxy reward learned from the finite data [40, 62, 25, 10] . Efforts have been made to mitigate this fundamental issue through the perspective of theory, e.g., [85, 71, 86] , and practice, e.g., [15, 21, 41, 81, 49, 56] . Our approach starts from the theoretical insights of handling inherent uncertainty in learning human preference from finite data, while being surprisingly easy to implement.One limitation of the current work is that we focus on the setting of offline RLHF where only a fixed preference dataset is available. Recent RLHF research has shown great potential of using iterative methods for LLM alignment with multiple rounds of preference data collection and tuning [71, 58] .Future works include extending our idea of theoretical algorithm design and analysis to the iterative RLHF setup where further preference data can be collected. Also, since our practical algorithm RPO is a plug-in module that effectively mitigates overoptimization and improves alignment performance, it serves as an exciting direction to combine it with explorative preference data collecting mechanism in iterative RLHF to further boost the performance of LLM alignment.Discussions on Algorithm 1 and Theorem 5.3. We compare our theory with [71] and [78] .Remark C.1 (Comparison with [71] ). Another theoretical work on RLHF [71] explicitly models the KL-regularization between the target policy and the reference policy in the learning objective, referred to as the KL-regularized contextual bandit. This means that their metric becomes the KL-regularized expected reward. In contrast, here we put the KL-regularization as a component of our algorithm design, but we still keep the metric as the expected reward (2.2). Therefore our theory in Section 5.1 directly reveals how the learned policy performs in terms of the expected reward compared to any given target policy (which can be a stochastic policy). Remark C.2 (Comparison with [78] ). We remark that in the work of [78] , they also mentioned a maximin object similar to (3.2) for offline preference-based RL as a complementary to their theoretical algorithm. However, the sample complexity of the maximin-style algorithm they presented is unknown, while we provide finite sample convergence result for Algorithm 1 in Section 5. Furthermore, our objective (3.2) features another KL-regularization term, which is essential for the proposal of our new practical algorithm design for aligning LLM in Section 4.A sufficient condition to make this partial coverage condition (Assumption 5.2) hold is that the distribution of the offline dataset, which is µ D , can well cover the joint distribution of (a 1 , a 0 ) ∼ (π, π base ). Here to discuss focus on π base = π chosen as we adopted in the experiment part.First, we clarify that the offline dataset distribution µ D is not simply (a 1 , a 0 ) ∼ (π unchosen , π chosen ), since according to our definition (see Section 2) whether a 1 or a 0 is chosen is random and is determined by y ∈ 0, 1 obeying the BT model. Thus, (a 1 , a 0 ) ∼ µ D can be interpreted as a mixture of (π unchosen , π chosen ) and (π chosen , π unchosen ). This mixture probability would not be too small as long as the quality of (a 1 , a 0 ) does not vary too much, i.e., both of them are possible to be chosen, which is the case in practice. As a result, in the offline data distribution (a 1 , a 0 ) ∼ µ D , both a 1 and a 0 partly comes from the chosen distribution π chosen . Then in order for µ D to cover the joint distribution of (a 1 , a 0 ) ∼ (π, π base ), it suffices to argue that π chosen can cover the target policy π, which is then reduced back to the traditional coverage condition. Thus our assumption essentially requires that π chosen well covers and only needs to cover the target policy π. This coincides with the spirit of the minimal data assumption in offline RL theory, i.e., the so-called partial coverage condition.On the relationship between observed chosen probability and reward overoptimization. First, we note that the actions and their chosen probabilities can be interpreted as a proxy of analyzing the underlying (estimated) reward model r due to the representation π r (a|x) ∝ π ref (a|x) exp(β -1 r(x, a)).Analyzing the (log) probabilities of the actions can be utilized to detect the mitigation of overoptimization, because according to the representation, an overestimated reward of a poor action would result in a higher probability of choosing this action, and would also cause a decay in the probability of choosing other better actions (since the probabilities are normalized to 1).To further showcase the ability of RPO to address overoptimization (through the lense of probability), consider the following theoretical example with only one state and three actions [73] where we can track everything clearly. which is better than the DPO policy. Thus, RPO is able to prevent overoptimization towards poor actions that are less covered by the dataset (action c here), therefore resulting in a better policy.About the relationships and distinctions between PTX loss in [60] and the SFT loss of RPO.The original PTX loss is an imitation loss calculated on the pretraining data. In contrast, the SFT loss in the RPO objective is an imitation loss calculated on the RLHF dataset. In more specific, our experiments use this SFT loss to imitate the chosen responses in the RLHF dataset. Thus the relationship is that they are both imitation loss which aims to mimic certain data distribution. The distinction is that they are calculated on different data sources. The SFT loss in the RPO objective naturally comes from our theoretical algorithm and provably serves as an important regularization term to mitigate overoptimization in offline RLHF.About the computational complexity of the SFT loss gradient. According to the paragraph Practical implementation in Section 6, RPO adds an additional SFT loss (the log probability of the chosen labels in the preference dataset) on the original DPO loss, where we highlight that the SFT loss is actually an intermediate quantity in the calculation of the DPO loss. Hence, our proposed method does not incur any additional computation overhead compared with the vanilla DPO.D.1 Proof of Theorem 5.3Proof of Theorem 5.3. By definition, the suboptimality gap of π w.r.t. π is decomposed as following,Gap π ( π) = E x∼d0,a∼π(•|x) r ⋆ (x, a) -E x∼d0,a∼ π(•|x) r ⋆ (x, a) = E x∼d0,a 1 ∼π(•|x),a 0 ∼π ref (•|x) r ⋆ (x, a 1 ) -r ⋆ (x, a 0 ) -β • KL π(•|x)∥π ref (•|x) -η -1 • min r∈R η • E x∼d0,a 1 ∼ π(•|x), a 0 ∼π base (•|x) r(x, a 1 ) -r(x, a 0 ) -β • KL π(•|x)∥π ref (•|x) + L D (r) + η -1 • min r∈R η • E x∼d0,a 1 ∼ π(•|x), a 0 ∼π base (•|x) r(x, a 1 ) -r(x, a 0 ) -β • KL π(•|x)∥π ref (•|x) + L D (r) -E x∼d0,a 1 ∼ π(•|x),a 0 ∼π base (•|x) r ⋆ (x, a 1 ) -r ⋆ (x, a 0 ) -β • KL π(•|x)∥π ref (•|x) + β • E x∼d0 KL π(•|x)∥π ref (•|x) -KL π(•|x)∥π ref (•|x) := Term (A) + Term (B) + Term (C), (D.1)where in the above Term (A), Term (B), and Term (C) are abbreviations forEQUATIONEQUATIONTerm (C) = β • E x∼d0 KL π(•|x)∥π ref (•|x) -KL π(•|x)∥π ref (•|x) .In the following, we analyze Term (A) and Term (B) respectively.Notice that by the optimality of our choice of policy π in (3.2), we haveTerm (A) = E x∼d0,a 1 ∼π(•|x),a 0 ∼π base (•|x) r ⋆ (x, a 1 ) -r ⋆ (x, a 0 ) -β • KL π(•|x)∥π ref (•|x) (D.2) -η -1 • min r∈R η • E x∼d0,a 1 ∼ π(•|x), a 0 ∼π base (•|x) r(x, a 1 ) -r(x, a 0 ) -β • KL π(•|x)∥π ref (•|x) + L D (r) ≤ E x∼d0,a 1 ∼π(•|x),a 0 ∼π ref (•|x) r ⋆ (x, a 1 ) -r ⋆ (x, a 0 ) -β • KL π(•|x)∥π ref (•|x) -η -1 • min r∈R η • E x∼d0,a 1 ∼π(•|x), a 0 ∼π base (•|x) r(x, a 1 ) -r(x, a 0 ) -β • KL π(•|x)∥π ref (•|x) + L D (r) = max r∈R E x∼d0,a 1 ∼π(•|x),a 0 ∼π base (•|x) r ⋆ (x, a 1 ) -r ⋆ (x, a 0 ) -r(x, a 1 ) -r(x, a 0 ) -η -1 • L D (r) ,where in the inequality we apply the optimality of the choice of policy π in (3.2).For this term, we directly consider the following bound,Term (B) = η -1 • min r∈R η • E x∼d0,a 1 ∼ π(•|x), a 0 ∼π ref (•|x) r(x, a 1 ) -r(x, a 0 ) -β • KL π(•|x)∥π ref (•|x) + L D (r) -E x∼d0,a 1 ∼ π(•|x),a 0 ∼π base(•|x) r ⋆ (x, a 1 ) -r ⋆ (x, a 0 ) -β • KL π(•|x)∥π ref (•|x) ≤ E x∼d0,a 1 ∼ π(•|x),a 0 ∼π base (•|x) r ⋆ (x, a 1 ) -r ⋆ (x, a 0 ) -β • KL π(•|x)∥π ref (•|x) + η -1 • L D (r ⋆ ) -E x∼d0,a 1 ∼ π(•|x),a 0 ∼π base(•|x) r ⋆ (x, a 1 ) -r ⋆ (x, a 0 ) -β • KL π(•|x)∥π ref (•|x) = η -1 • L D (r ⋆ ), (D.3)where in the inequality we apply the fact that r ⋆ ∈ R by Assumption 5.1.Combining Term (A), Term (B), and Term (C). Now by (D.1), (D.2), and (D.3), we have thatGap π β ( π) = Term (A) + Term (B) + Term (C) (D.4) ≤ max r∈R E x∼d0,a 1 ∼π(•|x), a 0 ∼π base (•|x) r ⋆ (x, a 1 ) -r ⋆ (x, a 0 ) -r(x, a 1 ) -r(x, a 0 ) + η -1 • L D (r ⋆ ) -L D (r) + β • E x∼d0 KL π(•|x)∥π ref (•|x) -KL π(•|x)∥π ref (•|x) .In the following, we upper bound the right hand side of (D.4) via relating the MLE loss difference term to the reward difference term through a careful analysis of the preference model. On the one hand, we invoke Lemma D.1 to give an upper bound of the difference of the MLE loss as following, with probability at least 1δ over random samples and ε = (6• (1 + e R ) • N ) -1 , for any reward model r ∈ R, it holds that L D (r ⋆ ) -L D (r) ≤ -2 • E (x,a 1 ,a 0 )∼µ D (•,•,•) D 2 Hellinger P r ⋆ (•|x, a 1 , a 0 )∥P r (•|x, a 1 , a 0 ) + 3 N • log N ε (R, ∥ • ∥ ∞ ) δ ,where we recall that we use the subscript r in P r to emphasize the dependence of the probabilistic model on the reward model. Here N ε (R, ∥ • ∥ ∞ ) denotes the ε-covering number of the reward model class and R is the upper bound on the reward functionss (Assumption 5.1). Now to facilitate the calculation, we lower bound the Hellinger distance by total variation (TV) distance asD 2 Hellinger P r ⋆ (•|x, a 1 , a 0 )∥P r (•|x, a 1 , a 0 ) ≥ D 2 TV P r ⋆ (•|x, a 1 , a 0 )∥P r (•|x, a 1 , a 0 ), By the expression of the probability model P r , we can further write the TV distance above asD TV P r ⋆ (•|x, a 1 , a 0 )∥P r (•|x, a 1 , a 0 ) = 1 2 • σ r ⋆ (x, a 1 ) -r ⋆ (x, a 0 ) -σ r(x, a 1 ) -r(x, a 0 ) + 1 2 • σ r ⋆ (x, a 0 ) -r ⋆ (x, a 1 ) -σ r(x, a 0 ) -r(x, a 1 ) = σ r ⋆ (x, a 1 ) -r ⋆ (x, a 0 ) -σ r(x, a 1 ) -r(x, a 0 ) , (D.5)where in the second equality we use the fact that σ(-z) = 1σ(z). Now by Lemma D.2 and the condition that r(x, a) ∈ [0, R] for any (x, a, r) ∈ X × A × R (Assumption 5.1), we know thatσ r ⋆ (x, a 1 ) -r ⋆ (x, a 0 ) -σ r(x, a 1 ) -r(x, a 0 ) ≥ κ • r ⋆ (x, a 1 ) -r ⋆ (x, a 0 ) -r(x, a 1 ) -r(x, a 0 ) , where κ = 1/(1 + exp(R)) 2 .As a result, the difference of the MLE loss is upper bounded byL D (r ⋆ ) -L D (r) ≤ -2κ 2 • E (x,a 1 ,a 0 )∼µ D (•,•,•) r ⋆ (x, a 1 ) -r ⋆ (x, a 0 ) -r(x, a 1 ) -r(x, a 0 ) 2 + 3 N • log N ε (R, ∥ • ∥ ∞ ) δ . (D.6)On the other hand, the reward difference term in (D.4), which is evaluated on actions from π and π base , can be related to the reward difference evaluated on the data distribution µ D via Assumption 5.2, i.e.,E x∼d0,a 1 ∼π(•|x),a 0 ∼π base (•|x) r ⋆ (x, a 1 ) -r ⋆ (x, a 0 ) -r(x, a 1 ) -r(x, a 0 ) (D.7) ≤ C µ D (R; π, π base ) E (x,a 1 ,a 0 )∼µ D r ⋆ (x, a 1 ) -r ⋆ (x, a 0 ) -r(x, a 1 ) -r(x, a 0 ) 2 .Finally, combining (D.6), (D.7), and (D.4), denoting∆ r := E (x,a 1 ,a 0 )∼µ D r ⋆ (x, a 1 ) -r ⋆ (x, a 0 ) -r(x, a 1 ) -r(x, a 0 ) 2 ,we have thatGap π ( π) ≤ max r∈R C µ D (R; π, π base ) • ∆ r -2η -1 κ 2 • ∆ 2 r + 3 ηN • log N ε (R, ∥ • ∥ ∞ ) δ + β • E x∼d0 KL π(•|x)∥π ref (•|x) -KL π(•|x)∥π ref (•|x) ≤ C µ D (R; π, π base ) 2 η 8κ 2 + 3 ηN • log N ε (R, ∥ • ∥ ∞ ) δ + β • E x∼d0 KL π(•|x)∥π ref (•|x) , = 1 N N i=1 y i • log σ r(x i , a 1 i ) -r(x i , a 0 i ) σ r † (x i , a 1 i ) -r † (x i , a 0 i ) + 1 N N i=1 (1 -y i ) • log σ r(x i , a 0 i ) -r(x i , a 1 i ) σ r † (x i , a 0 i ) -r † (x i , a 1 i ).Use the inequality that log(x) ≤ x -1, we can further upper bound L D (r † ) -L D (r) byEQUATION.Now since ∥r † -r∥ ∞ ≤ ε and r † ∈ [0, R],invoking Lemma D.2, we can derive thatL D (r † ) -L D (r) ≤ 1 N N i=1 r(x i , a 1 i ) -r(x i , a 0 i ) -r † (x i , a 1 i ) -r † (x i , a 0 i ) (1 + e R ) -1 + 1 N N i=1 r(x i , a 0 i ) -r(x i , a 1 i ) -r † (x i , a 0 i ) -r † (x i , a 1 i ) (1 + e R ) -1 ≤ 4 • ∥r † -r∥ ∞ • (1 + e R ) ≤ 4ε • (1 + e R ).(D.10)On the other hand, we upper bound the hellinger distance between P r and P r † , for any (x, a 1 , a 0 ) ∈ X × A × A,D 2 Hellinger P r † (•|x, a 1 , a 0 )∥P r (•|x, a 1 , a 0 ) ≤ D TV P r † (•|x, a 1 , a 0 )∥P r (•|x, a 1 , a 0 ) = σ r † (x, a 1 ) -r † (x, a 0 ) -σ r(x, a 1 ) -r(x, a 0 ) ≤ r † (x, a 1 ) -r † (x, a 0 ) -r(x, a 1 ) -r(x, a 0 ) ≤ 2 • ∥r † -r∥ ∞ ≤ 2ε, (D.11)where the first inequality uses the fact that D 2 Hellinger ≤ D TV , the equality uses the same argument as (D.5), and the second inequality applies Lemma D.2. Finally, combining (D.9), (D.10), and (D.11), we conclude thatL D (r ⋆ ) -L D (r) ≤ -2 • E (x,a 1 ,a 0 )∼µ D (•,•,•) D 2 Hellinger P r ⋆ (•|x, a 1 , a 0 )∥P r (•|x, a 1 , a 0 ) + 2 N • log N ε (R, ∥ • ∥ ∞ ) δ + 6ε • (1 + e R ).By taking the approximation error ε = (6• (1 + e R ) • N ) -1, we conclude that for δ < e -1 , with probability at least 1δ, for any r ∈ R, it holds thatL D (r ⋆ ) -L D (r) ≤ -2 • E (x,a 1 ,a 0 )∼µ D (•,•,•) D 2 Hellinger P r ⋆ (•|x, a 1 , a 0 )∥P r (•|x, a 1 , a 0 ) + 3 N • log N ε (R, ∥ • ∥ ∞ ) δ .This completes the proof of Lemma D.1.z 1 , z 2 ∈ [-R, R], it holds that κ • |z 1 -z 2 | ≤ |σ(z 1 ) -σ(z 2 )| ≤ |z 1 -z 2 |,where the constant κ = 1/(1 + exp(R)) 2 .Proof of Lemma D.2. Since the sigmoid function σ(•) is differentiable, we know that for anyz 1 , z 2 ∈ [-R, R], there exists some ξ(z 1 , z 2 ) ∈ [-R, R] such that σ(z 1 ) -σ(z 2 ) = σ ′ ξ(z 1 , z 2 ) • (z 1 -z 2 ). Notice that σ ′ (z) = σ(z) • (1 -σ(z)), we can obtain that 1 ≥ σ ′ ξ(z 1 , z 2 ) = σ ξ(z 1 , z 2 ) • 1 -σ ξ(z 1 , z 2 ) = 1 1 + exp(ξ(z 1 , z 2 )) • 1 - 1 1 + exp(ξ(z 1 , z 2 )) ≥ 1 1 + exp(R) • 1 - 1 1 + exp(-R) = 1 (1 + exp(R)) 2 .This completes the proof of Lemma D.2. This finishes the proof of Theorem 5.6.Lemma E.1 (Equivalence of maximin and minimax objectives). For the policy class Π defined in Proof of Lemma E.1. The foundation of this result is a minimax theorem given by [23] (Lemma E.2).In our setting, the policy class Π is a nonempty set, and the reward model class R is a nonempty compact Hausdorff space. Furthermore, by our choice of the policy class Π in (2.3), Π is a convex set. Meanwhile, the function ϕ is a concave function of π ∈ Π since the dependence on π is linear terms plus a negative KL term (concave). Finally, by our assumption, the function ϕ is convex-like on the reward model class R and is also continuous on R. As a result, all the conditions of Lemma E.2 are satisfied and the minimax theorem holds in our problem setup, finishing the proof of Lemma E.1.Lemma E.2 (Minimax theorem [23] ). Let X be a nonempty set (not necessarily topologized) and Y be a nonempty compact topological space. Let f : X × Y → R be lower semicontinuous on Y. Suppose that f is concave-like on X and convex-like on Y, i.e., for any x 1 , x 2 ∈ X , α ∈ [0, 1], there exists x 3 ∈ X such thatf (x 3 , •) ≥ α • f (x 1 , •) + (1 -α) • f (x 2 , •) on Y,and for any y 1 , y 2 ∈ Y, β ∈ [0, 1], there exists y 3 ∈ Y such thatf (•, y 3 ) ≤ β • f (•, y 1 ) + (1 -β) • f (•, y 2 ) on Y.Then the following equation holds, Then given any r ∈ R, the maximimzer of ϕ(•, r) is unique on the support of d 0 .Proof of Lemma E.3. Given any r ∈ R, consider thatmax π∈Π ϕ(π, r) = η • max π∈Π E x∼d0,a 1 ∼π(•|x) r(x, a 1 ) -β • D KL π(•|x)∥π ref (•|x) = η • max π∈Π C r -β • E x∼d0 D KL π(•|x) π ref (•|x) • exp(β -1 • r(x, •)) a ′ ∈A dπ ref (a ′ |x) • exp(β -1 • r(x, a ′ )), whereC r = E x∼d0 β • log a∈A dπ ref (a|x) • exp β -1 • r(x, a)is a constant independent of π. Therefore, the maximizer of ϕ(•, r) on the support of d 0 must equal toπ r (•|x) = π ref (•|x) • exp(β -1 • r(x, •)) a ′ ∈A dπ ref (a ′ |x) • exp(β -1 • r(x, a ′ )), which completes the proof of Lemma E.3.We train the gemma series models with 8 NVIDIA A6000 GPUs and the beta series models with 8 NVIDIA A100 GPUs, where they are all GPT-like models with around 7 billion parameters. It takes around three hours to train a beta series model and five hours to train a gemma one. Our codebase is adapted from the Alignment Handbook [63] . By comparing the validation loss on the test split (not used for later evaluation), we select the hyperparameter η of both RPO (beta) and RPO (gemma) to be 0.005. We list the remaining training configurations in GPT-4 evaluation on the test split. We use the following prompts to guide GPT-4 to annotate the preferences among win, lose, and tie (we denote them by A, B, and C, respectively).Prompts: Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below. You should choose the assistant that follows the user's instructions and answers the user's question better. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses. Begin your evaluation by comparing the two responses and provide a short explanation. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible. After providing your explanation, output your final verdict by strictly following this format: Here, we replace {answer A} and {answer B} with the answers of two models. Since GPT annotation has shown to prefer the answer in the first position [66] , we randomly exchange the positions between two answers during the evaluation to ensure a fair comparison.[[A]] if assistant A is better, [[B]] ifBenchmark evaluation. We use the default configuration for the evaluations on MT-Bench2 and AlpacaEval 2.03 . By default, the annotator of MT-Bench is the latest version of GPT-4. The default annotator and the competitor model are both GPT-4 (Preview 11/06). We only need to manually import the proper chat template that formats the training dataset, which are shown as follows.Chat Template for Beta Series: 


 {instruction}
 
 Chat Template for Gemma Series: <bos> 
user {instruction}
 
assistantIn this section, we provide the additional results to show the performance gain for RPO (beta) in MT-Bench and RPO (gemma) in AlpacaEval 2.0. We report the pairwise win rates in Tables 4, 5 , and 6 to analyze their performance gaps, where all the annotation configurations are the same in To provide a more comprehensive analysis of the trained LLM, we introduce more benchmarks on the math, reasoning, and coding tasks for evaluations. Specifically, we choose the Grade School Math 8K (GSM8K), AI2 Reasoning Challenge (ARC), and Mostly Basic Python Programming (MBPP) to measure math, reasoning, and coding abilities, respectively. In this section, we focus on the gemma series for the experiments. We do not use chain-of-thought or few shots in all the benchmarks. We compare the greedy decoding result (pass @1) on the MBPP benchmark. [27] and we average the SFT loss regularizer in RPO by the number of tokens of the chosen response. We do not use chain-of-thought or few shots in all the benchmarks. We compare the greedy decoding result (pass @1) for MBPP.Here we use the OpenRLHF codebase [27] to implement a new variant of RPO, where the SFT loss regularizer is averaged by the number of tokens of the chosen labels, that is, (log π θ (a cho |x))/|a cho |. Such a variant balances the weight of the averaged SFT loss regularizer between the shorter chosen response and the longer one. We set the coefficient for the SFT loss regularizer as 0.2. We use 8 NVIDIA A100 GPUs for the training and evaluation. The remaining hyperparameters are in Table 7 . Table 8 demonstrates that our proposed method still outperforms or performs equally to the vanilla DPO on these benchmarks of math, reasoning, and coding, which verifies the effectiveness of our proposed method.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judgehttps://github.com/tatsu-lab/alpaca_eval/tree/main 


Decision: The paper provides an interesting method to mitigate overoptimization in LLMs. The reviewers all agree with the contribution and novelty of the results. I recommend acceptance.

Reviews: review: paper_summary: paper_summary: The paper "Provably Mitigating Overoptimization in RLHF" addresses the issue of overoptimization in aligning large language models (LLMs) with human preferences using reinforcement learning from human feedback (RLHF). 

The main contributions include:
1. Identification of Overoptimization Source: The paper identifies the source of reward overoptimization as a distributional shift and uncertainty in learning human preferences.

2. Theoretical Algorithm Proposal: It proposes a theoretical algorithm that minimizes the maximum likelihood estimation of the loss and a reward penalty term to mitigate overoptimization, ensuring provable sample efficiency.

3. Practical Implementation: The algorithm is reformed into an easy-to-implement objective combining preference optimization and supervised learning loss, named Regularized Preference Optimization (RPO), demonstrating improved performance in aligning LLMs compared to existing methods​
strength_weakness: strength_weakness: The partial coverage condition lacks discussions since now it's a pair over (\pi,\pi^{base}), which is different from the traditional coverage condition. For example, for the linear case, $C_{\mu_D}$ would approximately become
$$
\mathbb{E}_{x,a^1\sim\pi^*,a^2\sim\pi^{pref}}\sqrt{(\phi(x,a^1) - \phi(x,a^0))^{\top} \Sigma_D^{-1} (\phi(x,a^1) - \phi(x,a^0))},
$$
where $\pi^{pref}$ means the distribution of the chosen samples. We know that $\Sigma_D$ is composed of pairs of chosen and unpreferred samples, but the $(\phi(x,a^1) - \phi(x,a^0))$ is there pair of optimal policy and the policy represents the chosen samples. Hence, if we want to compete with a policy $\pi$ better than $\pi^{chosen}$, how can the direction of $(\pi,\pi^{chosen})$ be covered by $(\pi^{unprefered},\pi^{chosen})$?
questions: questions: 1. I just wonder about the extension to online RLHF. For online RL, based on the optimism principle, it seems that then the objective should be subtracted from the SFT loss, which obliviates the wish to avoid overoptimization. So how to balance the exploration and avoiding overoptimization for the online setting?

2. What is the additional computational complexity brought by the gradient of SFT loss? Besides, the author doesn't mention how to approximate the gradient of SFT loss since there are expectations.
limitations: limitations: See weaknesses.
paper_summary: paper_summary: The paper introduces the concept of RPO, which combines DPO loss with SFT loss. This approach aims to align the policy with human preferences while simultaneously imitating a baseline distribution, effectively mitigating overoptimization. Empirical results from experiments with LLMs demonstrate that RPO outperforms traditional DPO methods, showcasing the practical applicability of the proposed algorithm.
strength_weakness: strength_weakness: The theoretical guarantees provided by the algorithm rely on specific conditions, such as partial coverage. These conditions might not always hold in practical scenarios, potentially limiting the generalizability of the results​.

The SFT loss + DPO seems very intuitive.
questions: questions: It is noticed believed that math problems are not very suitable with vanilla DPO. Some practitioners found similar algorithms should help the performance. Have you checked the reasoning benchmarks and the proposed algorithm?
limitations: limitations: Yes.
paper_summary: paper_summary: This paper aims to address the issue of over-optimization in RLHF. The authors introduce a method named RPO which concurrently minimizes the maximum likelihood estimation of the loss alongside a reward penalty term. Not only do the authors demonstrate that the proposed method is sample-efficient, but they also outline a straightforward yet effective implementation strategy. Experimental results underscore the efficiency of the method.
strength_weakness: strength_weakness: 1.	Additional experiments across a broader range of scenarios are required to more comprehensively demonstrate the method's efficiency.

2.	The current experiments are limited to evaluations using GPT and log probability, which do not offer intuitive insights into the over-optimization problem. In other words, it remains unclear whether the observed improvements in performance truly indicate a mitigation of the over-optimization issue. A more detailed analysis, perhaps focusing on rewards, could provide the necessary clarity.
questions: questions: 1.	The SFT loss appears similar to the commonly used PTX loss in [1]. Could you please elucidate their relationships and distinctions?

2.	How is the true reward (human reward) depicted in Figure 1 on the left derived?

3.	How does the proposed method address scenarios involving multiple reward models?

[1] Stanford alpaca: An instruction-following llama model.
limitations: limitations: Yes

Comments: title: Reply to the Official Comment by Reviewer 9VCJ
Reply to the Official Comment by Reviewer 1DUS
Reply to the Official Comment by Reviewer JyGr
comment: Dear Reviewer 9VCJ,

Thank you for your review and support. We will incorporate your valuable suggestions into our paper as we revise it based on the feedback from all reviewers. Your comments greatly assist us in strengthening the overall quality of our work.

Best regards, Authors
Thank you for your response. I have revised the score.
Dear Reviewer 1DUS,

Thank you for your review and support. We will incorporate your valuable suggestions into our paper as we revise it based on the feedback from all reviewers. Your comments greatly assist us in strengthening the overall quality of our work.

Best regards, Authors
Dear Reviewer JyGr,

Thank you for your review and support. We will incorporate your valuable suggestions into our paper as we revise it based on the feedback from all reviewers. Your comments greatly assist us in strengthening the overall quality of our work.

Best regards, Authors
Thank you for your response. I will keep my score positive.
The authors have addressed my question, and I choose to maintain my score.