Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer
Authors: Zhihan Liu
Miao Lu
Shenao Zhang
Boyi Liu
Hongyi Guo
Yingxiang Yang
Jose Blanchet
Zhaoran Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Alignment; Reinforcement Learning from Human Feedback
Primary area: generative_models
TL;DR: We propose a new algorithm that provably mitigates overoptimization in RLHF, evidenced by LLM experiments.

Abstract: Aligning generative models with human preference via RLHF typically suffers from overoptimization, where an imperfectly learned reward model can misguide the generative model to output even undesired responses. We investigate this problem in a principled manner by identifying the source of the issue as the distributional shift and uncertainty of human preference in dataset. To mitigate overoptimization, we first propose a theoretical algorithm which optimizes the policy against an adversarially chosen reward model, one that simultaneously minimizes its MLE loss and a reward penalty term. The penalty pessimistically biases the uncertain rewards so as to prevent the policy from choosing actions with spursiouly high proxy rewards, resulting in provable sample efficiency of the algorithm under a partial coverage style condition. Moving from theory to practice, the proposed algorithm further enjoys an equivalent but surprisingly easy to implement form. With a clever usage of the equivalence between reward models and the corresponding optimal policy, the algorithm features a simple objective that combines (i) a preference optimization loss that directly aligns the policy with human preference, and (ii) a supervised learning loss which explicitly imitates the policy with a baseline distribution. In the context of aligning large language models (LLM), this objective fuses the direct preference optimization (DPO) loss with the supervised fune-tuning (SFT) loss to help mitigate the overoptimization towards undesired responses, for which we name the algorithm Regularized Preference Optimization (RPO).
Experiments of aligning LLMs demonstrate the improved performance of our method when compared with DPO baselines. 
Our work sheds light on the interplay between preference optimization and SFT in tuning LLMs with both theoretical guarantees and empirical evidence.

Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer
Zhihan Liul+
Miao Lu
Shenao Zhang'
Boyi Liu'
Hongyi Guo'
Yingxiang Yang² Jose Blanchet Zhaoran Wang'
NNrthwestern University Stanford University *ByteDance Inc.
{zh fzhihanluu202, shenaozhang2028, how.aag222202ooogoooooooo00 hongyiguu22025)0u..m wow.wwwsee....t.eem..mmmm edu
fmiaolu, jose. blanchet blanchet)@stanff @eststtanoooo ord. edu, zhaoranvang@gmail com
Aboyi.liu00 yingxiang aang)abytedance....
Abstract
Aligning generative models with human preference via RLHF typically suffers from overoptinization, where an imperfectly learned reward model can misguide the generative model to output undesired responses. We investigate this problem in principled manner by identifying the source of the misalignment as form of distributional shift and uncertainty in learning human preferences. To mitigate overoptimization, we first propose theoretical algorithm that chooses the best policy for an adversarially chosen reward model; one that simultaneously mini- mizes the maximum likelihood estimation of the loss and reward penalty term. The penalty term is introduced to prevent the policy from choosing actions with spurious high proxy rewards, resulting in provable sample efficiency of the algo- rithm under a partial coverage style condition. Moving from theory to practice, the proposed algorithm further enjoys an equivalent but surprisingly easy-to-implement reformulation. Using the equivalence between reward models and the correspond- ing optimal policy, the algorithm features simple objective that combines: (i) preference optimization loss that directly aligns the policy with human preference, and (ii) supervised learning loss that explicitly imitates the policy with (suitable) baseline distribution. In the context of aligning large language models (LLM), this objective fuses the direct preference optimization (DPO) loss with the supervised fine-tuning (SFT) loss to help mitigate the overoptimization towards undesired responses, for which we name the algorithm Regularized Preference Optimization (RPO). Experiments of aligning LLMs demonstrate the improved performance of RPO compared with DPO baselines. Our work sheds light on the interplay between preference optimization and SFT in tuning LLMs with both theoretical guarantees and empirical evidence.
Introduction
A key step in building state-of-the-art LLMs is Reinforcement Learning from Human Feedback (RLHF) [12, 87], which aligns pretrained L.MSs with human preferences using human assessment data, making the model more helpful, truthful, and harmless [42, 10]. Typically, RLHF first learns reward model from data (pair-wise wise comparisons of responses) to quantify the human preferences of LLM outputs. Then it fine-tunes the LLM to maximize the learned reward using RL techniques.
In this pipeline, crucial challenge is reward overoptimization or reward hacking [40, 62, 25]. Since the reward model is learned from finite data, might not be perfectly aligned with the underlying human preference. Optimizing the LLM towards such an imperfectly learned and potentially overfitted
"quual contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Figure I: Left: Reward overoptimization due the distributional shift and uncer- tainty in reward. Right: Overoptimization causes the probability of outputting pre- ferred responses in the preference data to decrease substantially using original DPO proposed by [46]. Our algorithm (RPO) significantly alleviates this decrease. See more discussions in Section 6.
Log, Protabality Chesn Rleas um Daree Traeeig
Probabi Ity Pal Reward and?
distribution /" human roward
Dio (bves
-300
RPO (tvo]
ust inated raward
150
-22
100
Traiiig
reward model leads performance degeneration and a substantial decrease in the probability of choosing the preferred responses in the data 45]. Given the importance of RLHF and the outlined challenge, crucial research question is: How to mitigate reward overoptimization in RLHF in principled and efficient manner for better alignment?
To answer the question, we model RLHF an offline contextual bandit [42] and ascribe overopti- mzation to distributional shifts and reward uncertainty. Intuitively, when fine-tuning an LLM, the response (action) distribution of the tuned LLM could deviate from that of the training data. For the out-of-distribution responses, which are dissimilar with (or not well covered by) the responses in the data, the high inherent uncertainty of underlying human preferences could make the learned reward model misleading for ontorf-distribution responses. In this situation, reward overoptimization can occur because the LLM is fine-tuned towards maximizing reward model with defective out-of- distribution prediction, giving a potential consequence that the LLM responses are favored by the learned reward but less preferred by human [86]. We illustrate these types of distributional shift and reward uncertainty issues inherent to overoptimization in Figure 1.
In this paper, we propose new RLHF algorithm to mitigate reward overoptimization, From a high level, our theoretical algorithm seeks the best LLM for an adversarially chosen reward model that minimizes the sum of its maximum likelihood estimation loss and its own expected reward value. Intuitively, since the reward value is also minimized when minimizing the sum, it can automatically prevent the misleadingly high reward caused by the uncertainty inherent in having access to finite data. Furthermore, we show that the theoretical algorithm enjoys an easy implementation: it simply adopts supervised fine-tuning (SFT) loss regularizer during training. By explicitly regularizing the L.M to imitate high-quality responses (e.g., preferred responses in dataset), the algorithm can effectively mitigate the issue of overoptimization. We establish theoretical guarantees and conduct experiments to demonstrate our findings, which we summarize next.
1.1 Our Contributions and Related Works
We summarize contributions in three areas follows.
theoretical algorithm under general function approximation. Our first contribution is new theoretical algorithm (Algorithm 1). It features an unconstrained maximin problem, outputting the optimal policy (LLM) against an adversarially chosen reward model that minimizes the summation of: (a) the MLE loss for estimating the underlying reward; and (b) a reward expected value term as a penalty that aims to prevent spuriously high reward estimation caused by data uncertainty and insufficient coverage. Algorithm is compatible with general function approximations of the reward model, meaning that we do not impose any specific structural form to the hypothesis class of reward, demonstrating its generality, especially in language modeling.
In this regime of reward class, we establish the fnite-sampi suboptimality gap of Algorithm 1 as OCooverrge v NN// /N) when competing with any LLM in terms of the underlying true human reward (Theorem 5.3). Here N is the number of human comparison data, NR is the complexity of the reward model class R. and Ccoverage characterizes the coverage of the preference dataset with respect to the response distribution of the LLM to compete (please see Assumption 5.2 for details). This indicates that, as long as the training data well cover the LLM т to compete, the algorithm is guaranteed to align LLM to output responses as good as in terms of human reward, without suffering from overoptimization caused by distributional shifts and inherent uncertainty in human preference.
An easy- tos--le-mmpleeen practical objective. Moving towards practice, we show that the objective of A Algorithm| I adopts surprisingly simple and equivalent form for its use in practice. Specifically, with mild regularity conditions, we prove that the maximin objective (Algorithml) 1) is equivalent to the

---

corresponding minimax objective, which is further reduced to single minimization problem for the reward model since its inner problem adopts a closed form solution. Inspired from recent progress in methods to align LLMss [46], we further re-parameterize the
RLHF that explores reward-model-fie reward model via its corresponding Kl.-regularized optimal policy. Then the minimization objective of the reward modeling naturally translates to a target for directly aligning the LLM,, which we call Regularized Preference Optimization (RPO; Algorithm 2). The objective of RPO features a simple
weighted combination of two losses:
RPO objective Preference optimization loss Imitation (SFT) loss.
Here the Preference optimization loss coincides with the DPO [46] objective, tending to optimize the LLM towards maximizing the underlying true reward. The Imitation (SFT) loss explicitly supervises the L.LM to mimic the responses from a proper distribution well covered by the dataset. The choice of the distribution is guided and justified by our theory of Algorithm 1, but can also be flexibly adapted in practice, e.g., the preferred response in the dataset, or the responses of the initial model.
We highlight that the Imitation (SFT) loss serves as an important term to mitigate overoptimization. Even though the original DPO objective has already involved a KL regularization between the tuned LLM and the initial LLM. is not enough to prevent overoptimization. As we elaborate in Section 4, the KL-regularization weight of the DPO objective could only control the scale of the gradient per training example, while the RPO objective can further modify the gradient direction. Calling back to the theoretical A Algorithm 1, such a modification of gradient direction originates from the reward penalty in the adversarial objective for the reward model. This modification, as we expose in our theoretical analysis, helps to mitigate overoptimization. Thus, incoporating SFT loss in RLHF gives you regularizer that provably mitigates overoptimization.
Empirical evaluations. Following the training setup of two series of released chat models Zephyr- 7b-beta (trained on the Ultrafeedback dataset [ [16] by DPO) and Zephyr-7b-gr bbggemmaaaaaaaaaae (trained on the Argilla- Dipill--DO-Mix-- -ix--K dataset [3] by DPO) [64], we implement RPO for the beta series and gemma series respectively to show that: (i) RPO is a flexible plug-in in module and can be applied to different reference models. (ii) RPO can alleviate the overoptimization issue. (iii) RPO consistently achieves better alignment performance than DPO in in-data distribution. (iv) RPO can also achieve consistently better performance in standard I.I.M benchmarks like MT-hench and AlpacaEval 2.0, which shows its potential of mitigating overoptimization for better alignment performance, justifffyin justifying our theory. Related works. Due space limitation, we refer the readers Appendix A for detailed discussion.
2 Preliminaries of RLHF
In this section, we introduce the mathematical framework of studying RLHF for aligning LLMs. We adopt the framework of offline contextual bandits [42], where we identify the context space X' as the space of prompts and the action space A as the space of responses. An LLM, defined as a policy (-1-)) : (AA), takes prompt x E X' as input and output response a E AA from a ~ (-(2x)
Preference model. Given any reward function r : X' x A R belonging to certain reward class R that represents the "human's 's ratin" of LLM responses given prompts, consider the Bradley-Terry model [9] of human preference. That is, given a prompt x E X and two responses a', a E A, the probability of at being preferred to a" (denoted by y 1, otherwise y = 0) is given by
exp(r(e,
P, = 1|x, a.,aaa.a., aa0)=== cxp(r(z,a')) + exp(r(x,a9) = (r((,, r(x, (2.1)
where (2) = 1/(1 exp(-2)) is the sigmoid function. For simplicity of future discussion, explicitly write out the dependence of the preference probability (-) on the reward model r E R. In the section of theory, i.e., Section 5, we specify the assumptions on the reward model class R.
Learning protocol. Typically, the RLHF pipeline starts from certain reference policy oh obtained from pretraining. Then RLHF aligns the LLM based on certain human preference data. In this work, we consider offline RLHF setup, where the LLM is aligned using fixed offline preference dataset D. It consists of N i.i.d. tuples in the form of D {(x., a., aq J Here the prompt x; and the responses at, af are distributed according to: (ar, aa)a ~ ('((')) and conditioning on (#z, aQ), y is distributed according to (2.1) for an underlying true (but unknown) reward model r" E R.
3

---

Algorithm Theoretical Algorithm: Maximin Objective
Input: Preference dataset D, parameters 3.77 > O, reference policy n= baseline policy baase 2: Output: Policy रे given by (3.2) with the cross-entropy loss function Cp defined in (3.1)..
Performance metric. The target of RLHF is align an LLM, or equivalently, to learn policy so as to maximize the expected true reward r". Thus, we define the value function of any policy æ as
( = Ba-doaava((le) [r* a)]
(2.2)
Here we allow the prompt distribution d-(-) to be different from that of the offline dataset distribution (.)()), but is assumed to be known. In the meanwhile, consider the policies that share the same support as the reference policy ne [71]. that is, we take policy class II as
11 n: : ((A)) Supp(T(-1x)) S Supp( ref (-)p)) Vx YoEx
(2.3)
The performance gap of a learned policy मे E II w.r.t. any other policy E II is measured as Gap" (f) ( [ given policy
(2.4)
The goal is to propose a sample-efficient and also implementation-friendyy algorithm to learn policy स E Il able to compete with any given policy E II in terms of Gap" (f) S with sample complexity polynomial 1/c and logarithmic in the complexity of R.
3 A -heory-motivated Objective
Our method seeks to find the best policy स against an adversarially chosen reward model Fady that minimizes a weighted sum of its expected value and the maximum likelihood estimation (MLE) loss. Intuitively, such reward model can prevent the overoptimization issue by taking its own value into account when minimizing the MLE loss. Since the reward value is also minimized when minimizing the sum, this method prevents the misleadingly high reward caused by the uncertainty due finite data. Formally, given two hyperparameters B.n > 0 and "baseline policy" π basse we define
T3 (T)
min rER
We ((x, a (oan
a') (T ((o))) ((a))
where the loss function (-(-) is the average negative log-likelihood function of the BT model (2.1) (and here it becomes the cross-entropy loss) over the preference dataset D, defined as
CD(r) = -Eo y: log (a (r(1,, a!) r(x;, + +(1 v:) log (a(r(,,, at) r(a;, -((n,, (3.1)
As see, 13. (1) is the minimum value of a weighted sum of the MLE loss and the expected reward value of π. but with two important modifications that we explain in the following.
Firstly, we subtract another expected reward of certain policy 7 base This because the BT model (2.1) essentially only uses the reward differences to define the preference probabilities. As result, the data can only reveal information of the differences between the true reward r* of different responses [78]. Accordingly, we subtract such baseline expected reward value to match this observation. The choice of the baseline policy is discussed the theory part (Section 5) and experiments (Section 6).
Secondly, we subtract a KI. divergence between л and re from the expected reward, weighted by the coefficient 3 > O. Such term is for practical considerations that would be explained in Sections and 5.2. We note that the KL regularized reward is commonly adopted in RLHF practice to ensure the learned policy is not far away from the reference policy [42, 71].
Finally, the overall algorithm design (Algorithm 1) is to output the policy that maximizes Tady (T), i..., f E argmax, 1.3 (r), which gives the following theoretical target:

---

Tile allaadoo (T(())) r(z, a²) ) a)) B [L(+(<>>)| (-1z)) + +CD(r)) }.
Ā E argmax min
WEl
PER
a (
(3.2)
Given the form of (3.2), we name it the maximin objective in the sequel. Upon seeing (3.2), one might be arguing that such theory-motivatee objective seems hard to implement in practice. Nevertheless, in the coming Section 4, we demonstrate that the maximin objective (3.2) adopts an easy-to-implement equivalent form, allowing us to design a practical algorithm for aligning LLMs.
An Equivalent and Implementatioon-rieendy Objective
In this section, we propose another minimax-style objective that is equivalent to the maximin objective (3.2). Based on the minimax objective, we propose a new LLM aligning algorithm called Regularized Preference Optimization (RPO). It draws inspirations from the reparametrization technique originated in Direct Preference Optimization (DPO) [46] and goes beyond to further address the overoptimization issue in offline RLHF by incorprating an SFT loss as explicit adversarial regularizer.
An equivalent minimax objective. If the reward model class R satisfies certain regularity conditions, which we discuss in detail in Section 5.2, the minimax theorem holds: solving the maximin objective (3.2) equivalent to solving a minimax target, given by
T/ do (o r(z,a') r(a,a)) - B ([((( (+x)) + Lo(r) d' (s (r)))
} (4.1)
min max PER wEll
Such minimax formulation (4.1) is the starting point our practical algorithm. The magic of (4.1)
is that the inner maximization problem adopts closed form solution, which further simplifies such an objective. To see this, note that given any reward model R. the inner problem is equivalent to
E Ex-dd [r(z,a) - 3 (L( (1x)) TElL
max
(4.2)
It has been well established that the policy that maximizes the KL-regularized expected reward (4.2) has a closed form solution. Due to its importance, we present it as the following lemma. Lemma 4.1 (Oracle optimal KL-regularized policy). Given any reward model R, the optimal policy n. to the maximization problem (4.2) is given by
/
ny (-1x)) = (x(x) roo (-1x)) exp (8 ((), 2,(x) = SaSA exp (3-1r(x, a)) (alx),
and correspondingly the optimal value of (4.2) is given by (4.2) En [3 log(Z((z)))
Specifically, by Lemma 4.1, we can solve the inner maximization problem in (4.1) and obtain that
(4.1) min YER BB a hane (-1z) - r(z, a²) + log (Z+(4)) + Lo(r))
Furthermore, from Lemma 4.1, one immediately that given any reward model R, we can reparameterize via its corresponding optimal KL.-regularized policy n, [46], that is,
(1 (-|z)
r(z,-) B loog
+ B log (Z,())
(4.3)
Taking (4.3) back into (4.1), we are able to further simplify it as
n. (-(-+-)
}
(4.1) = min yER
E andg harm (-lw) [-
(og( + Co (. (looo
(4.4)
To (-1-)
Thanks to the KL-regularization term in the original minimax objective (4.1) (or equivalently, the maximin objective (3.2)), we have the following theorem. It theoretically shows that the policy me associated with the reward model 7 solving (4.4) also solves the maximin target (3.2) of the theoretical algorithm (Algorithm 1) that enjoys finite-sample convergence guarantees. (Please see Section 5.2 for formal statement and proof of Theorem 4.2).
5

---

Algorithm 2 Practical Algorithm: Regularized Preference Optimization (RPO)
Input:
Preference dataset D, parameters 3.7 17 > O, reference policy n-e,, baseline policy base,
2: Output: Policy Tg obtained by optimizing objective (4.5).
Theorem 4.2 (Equivalence between maximin and minimax algorithm (informal)). Under certain
regularity assumptions on R and given 17, B >> 0, solving the minimax objective (4.1) via (4.4), i.e.
(a))z)) + LD (3
Tr (-1-)
F == argmin FER
7 (B non hom (-12)
x (---)
then the corresponding optimal KL-regularized policy also solves the maximin objective (3.2).
Regularized Preference Optimization. Target (4.4) gives a quite simple objective to use practice! Since (4.4) depends on R only through its corresponding optimal policy Tr, one can formulate a minimization objective over a parameterized policy no. i.e., the LLM be aligned, and directly optimize the parameters 0 E O. More formally, the new RLHF objective becomes
{ (RPO :- -18 E En -a babee ( logg(T((a"a))) + LD (a (B-og log ee
min
Ie
Imitation (SFT) loss
Preference opt. loss
(4.5)
In (4.5), the second term coincides with the objective DPO algorithm [46] which optimizes the policy towards maximizing the underlying true reward, and the first term stands for a regularization term weighted by 17-8 which explicilly regularizes the policy to imitate the baseline policy. Therefore, name the resulting algorithm Regularized Preference Optimization (RPO). We summarize abstractly in Algorithm 2. As for DPO, implementing RPO does not require to maintain a reward model Thus computationally more friendly compared
seward-based algorithms.
How does RPO improve DPO? We illustrate the effect of the imitation loss by analyzing the gradient of the RPO target (f) in (4.5). Notice that by (4.5) we have
VeCRPO(O) n7. Ex-do base (-1z)
log (ag( +
VeLDPO (0)
increase the alignment with the baseline policy
decrease DPO Loss
where the derivative of the DPO loss VoLDPo(0) given by the following.
VoLDPo - 3.o arej) - fa actoo) (V log Ne (accool Vo log To (arej 1x))
gradient weight
For simplicity we denote Pa (z, B 3 log(<o (2x, a)) (oo a)), acho for the chosen response and arej for the rejected response. Intuitively, RPO (4.5) modifies the gradient direction of DPO to ensure the alignment with the baseline policy x buse and the hyper-paramene alignment. In comparison, the hyper-parameter /3 in DPO only controls the gradient weight when increasing the likelihood of acho and decreasing the likelihood arej- In this perspective, the hyper- parameter only changes the scale of the gradient instead of the direction. By introducing 7), stabilize the training and reduce the side-effect uncertain labels in data to prevent overoptimization.
controls the power of
5 Theoretical Analysis
In this section, establish theoretical analysis for Algorithms I and 2. We take the space of prompts and responses as compact subsets X' C dxxx and A S Rdd, We take the policy class II as (2.3).
5.1 Establishing the Sample Complexity of Maximin Objective (Algorithm 1)
Assumption 5.1 (True reward model). We assume that the true reward model y E R, and for any E R and (z, a) X x A. it holds that r(z, a.) [0, R).

---

Assumption 5.2 (Partial coverage coefficient [78)). Given policy т E II, the coverage coefficient of the offline dataset distribution ur w.r.t reward model class R. policy n, and the baseline policy base denoted by Opp (R; n, base)), is defined as
Exndg. (a)))) (-1a) [(r* at) (x, (r(a,, a+) r(0,, a²))]
max
0, susu YER
(arraa) ((r - r (x, a)) - (r(z, a²) r(z, a(x,a))) ]²)
We assume that Cu (R; TT, ग hasee < +o for the policy π to compete. We remark that the quantity C,;(R; nT, π base) is upper bounded by the density ratio || @ 2 π base b/bss////l //ppllo
Assumption 5.1 is standard in sample complexity analysis [85, 78, 74). Assumption 5.2 characterizes how well the dataset D covers the policy æ to compete. To achieve provable sample efficiency, we only require that D covers the target policy æ, weak partial coverage style assumption for theoretical analysis. To illustrate it, when calling back to Figure I, the data distribution therein well covers those nearly optimal responses under r*, but does not sufficiently cover the responses with low r*.
Under such a partial coverage data condition, however, human preference of responses E AA that are not well covered by the dataset D can be poorly estimated, misguiding the policy म to behave suboptimally if it is overoptimized (recall Figure 1). Fortunately, the following theorem shows that Algorithm provably mitigates the overoptimization issue and achieves finite-sample convergence of the suboptimality gap (2.4) competing with æ. Proof is in Appendix D.
Theorem 5.3 (Suboptimality of Algorithm 1). Taking the policy class IT as (2.3), supposing that Assumptions 5.1 and 5.2 hold, and assuming that the reward model class R has a finite s-epsilon covering number under || -1 -orrm (R, || lloo) < too with & = (6 (1 + e²) -N) Setting
n n= (1 + exp(R)) / log (Ne(R, 12)) /N, B B=1/VN
in Algorithm 1. Then the output policy of Algorithm satisfies that with probability at least 1 8,
v6(1 exp(R)) ((CpD (R; ))bss))) )² + 1)c + 1. KL
(-12))
Gap" (R)
4VN
where Vlog (N: (R, || 11)) with E (6- (1+ ) -NY Here, N denotes the number of preference pairs in D, R denotes the upper bound of the reward models, and the partial coverage coefficient Cup (R; n, n brase) is defined in Assumption 5.2.
Remark 5.4 (Choice of the baseline policy). As is indicated by Assumption 5.2, the least requirement that π base can be covered by the offline data distribution. E.g., we can take π base as the distribution of the preferred responses in the data. In this case, the SFT loss in RPO explicitly regularizes the LLM to imitate the preferred responses. We choose this type of baseline policy in our experiments.
5.2 Equivalence between Maximin and Minimax Objectives
Now we formally show that the theoretical target (maximin objective (3.2)) and the target for practical algorithm design (minimax objective (4. I)) are equivalent under certain regularity conditions. This can naturally extend the sample complexity of Algorithm (Section 5.1) to that of minimax-based algorithms in Section 4, providing the theoretical guarantee for our practical algorithm design (RPO).
First, for notational simplicity, we denote the optimization target we investigate in Sections 3 and 4 as
((T,T) := =1 Be-do.a' (x(1))) ((2 at) - -r(x, 3- DKL (I+14))] + CD(r), (5.1) a base
for any rT II X R. Our result relies on the following assumptions on the reward model class R. Assumption 5.5 (Regularity of reward model class). We assume the following things on the reward model class R: (i) the space R is a compact topological space; (ū) the function & in (5.1) is convex-like on R, that is, for any rT, r2 E R and a E [0, 1}. there exists r3 E R such that
δ(T,TA) S a ((T,TT) + (1 a) )(T,TT)) VT E 11,
(5.2)
7

---

We note R is convex, e.g., a linear model class [85, 71, 86] or more general the Lipschitz continuous model class R, we can directly obtain that the function (( -) convex over R (since the dependence on E R is linear terms plus a convex loss LDD of r E RR), which implies the convex-like property (5.2). Under Assumption 5.5, it holds that (Lemma F.1)
min (TT, min max ((T,T). rER
(5.3)
rER
Furthermore, thanks to the KL-divergence regularization in á which intuitively makes "strongly concave" over the policy т, (5.3) can gives us the following stronger result, proved in Appendix E.1.
Theorem 5.6 (Formal statement of Theorem 4.2). For the policy class defined in (2.3) and the reward model class R satisfying Assumption 5.5, consider the following policy defined as
T& E argmax ((F,)), R), where F E argmin max (T,rr)
(5.4)
TEll
TER
Well
Then the policy n, also satisfies the maximin objective (3.2) of Algorithm 1, that is,
Tp E argmax min r),
#CTl
rER
Theorem 5.6 shows that the optimal KL KL-regularized policy associated with the reward model solving the minimax objective (3.2) also solves the maximin objective (i.e., objective (4.1) of Algorithm 1). This further allows us to extend our theoretical guarantee of Algorithm (Section 5.1) to that of linimax-based algorithms, justifying our practical algorithm design in Section 4.
Corollary 5.7 (Suboptimality of minimax-basee algorithm). Take the policy class II in (2.3) and the reward model class satisfying Assumption 5.5. Given any given policy to compete, if Asssuptiion 5.2 holds for n, then under the same choice of and 3 as in Theorem 5.3, the policy πи defined in (5.4) satisfies that Gap" (T;) I ((IvVN) with probability least 1 8.
Experiments
In this section, we provide a detailed empirical analysis of RPO to highlight the following four key points: (1) RPO is a flexible plug-in module and can be applied to different reference mod- els. (2) RPO can alleviate the overoptimization issue in the training phase by giving more trust to the chosen responses in the preference dataset. (3) As a justification of our theoretical analysis, RPO achieves better alignment performance than DPO in in-data distribution. (4) RPO can also
achieve consistently better performance in LLM benchmarks like MT-bench [83] and AlpacaE-
val 2.0 [20], which shows the potential of mitigating overoptimization for better generalization performance. The code for the experiments can be found in https:///githbbb com/ Com/YSSIU622/ Regularized- Preferereeteeeee wwwrennce-pttmzzatteeeee Optimizationn wrweeeaatteee
Experiment setup. To show that RPO is a flexible plug-in module regardless of the reference model, follow the training setup for well-studied series of released chat models with around billion parameters trained by DPO: Zephyr-7b-beta and rephyr-7b-gemee beta and gemma series. Mirrored by their training configurations, reference model and the preference dataset for our training on these two series as follows. For the beta series, we use ssstsalll-ssstt beta as the reference model πeel" mistral-7b-ste -fttt-fettbet is fine-tuned version of disttral-bb-ue 1 on the distilled version of the UltraChat dataset [17], which contains approximately 200k examples of multi-turn dialogues generated by GPT-3.5-TURBO, For the training preference dataset, we use Ultrafeedback Dataset [16]. which consists of approximately 60k prompts. For the gemma series, we use zephyr-70 7w---maa---------------------- www.geemamaeeeeeee v0. 1 as our reference model re zephyr-7b teo---emmamattee -ft--0.. 1 is a fine-tuned version of gemma-7b 7b on the Deita dataset [35], which involves around 10k distilled SFT data. For the training preference dataset, we use Argilla- DPO-MiX-7K Dataset [3], which is mixture of multiple distilled public preference datasets. For simplicity, we denote Ref. (beta) as the reference model, DPO (beta) as the model trained by DPO, RPO (beta) the model trained by RPO, all for the beta series. We use the same notations for the gemma series.
[64] to implement RPO in introduce how select the
Practical implementation. According to Algorithm 2 and as discussed in Remark 5.4, we imple- ment RPO by adding an SFT loss (log probability of chosen responses in the preference dataset) to the original DPO loss. By comparing the evaluation performance on the test split of the training
8

---

dataset, we select the hyperparameter the training DPO and RPO, We keep the remaining hyperparameters including 5. batch size, and learning rate to be the same for a fair comparison. Please see Appendix F.1 for detailed training configuration.
0.005 for both RPO (beta) and RPO (gemma). During
RPO alleviates overoptimization. As mentioned in the introduction part. DPO is observed have a significant and continuous decrease in log probability on chosen responses [26, 45] during training and we regard it as the consequence of overoptimization. Implied by our theory, overoptimization could arise when the model maximizes its own proxy reward formed on the responses less covered by the data. Due to the overoptimization, the model tends to disprefer the chosen responses they are away from the maximizers of the proxy reward despite that some chosen responses are highly preferred by humans. Consistent with our theoretical conclusion, we empirically find that RPO can indeed alleviate overoptimization in DPO, During the training phase of both beta and gemma series, we observe that the log probability given by the RPO-trained model notably higher than that given by the DPO-trained model for the chosen responses, which are shown in Fig. and 2.
RPO improves the alignment ability in in-data distribution. For the in-data distribution eval- uation, we select the 200 prompts (which are not used in the selection of 1j) in the test split of the training dataset to the reference model, DPO, and RPO generate the response respectively. We choose GPT-4 annotate the preference in the response pairs. Though we instruct GPT-4 to give annotation among win, lose, and, tie (please see the full prompt in Appendix F.2), GPT-4 may still give undesired annotations. Hence, filter all the undesired annotations and collect 150 examples for evaluation. We report the pairwise win rate among Ref., RPO, and DPO Table I for both the beta and gemma series. To show more illustrative comparison between DPO and RPO, provide the barplot to report the number of pairwise examples annotated by GPT-4 in Fig. 3 and Fig. 4. We observe that for both beta and gemma series, RPO has better performance than DPO in terms of both RPO/DPO-SFT and RPO-DPO win rates. The performance improvement matches our theoretical results in Corollary 5.7, which shows the credit of the alleviation of overoptimization.
Table 1: Pairwise win rate (left vs. right) among RPO-trained model, DPO-trained model, and the reference model. Annotated by GPT-4, evaluations of beta and gemma series made the 150 examples of the test split of the Ultrafeedback and the Argill-DDOO-Mix-KK dataset, respectively.
Win rate (%)
RPO (beta)
Ref. (beta)
DPO (beta)
Win rate (%)
RPO (gemma)
Ref. (gemma)
DPO (gemma)
RPO (beta)
50.0
79.0
56.0
RPO (gemma)
50.0
71.7
54.0
Ref. (beta)
21.0
50.0
22.7
Ref. (gemma)
28.3
32.7
DPO (beta)
44.0
77.3
50.0
DPO (gemma)
46.0
67.3
50.0
Probubiiit Chosen Resporess Duting Trarrix
Painwise Annotations on RPO vs. DPO
Pairwise Annotations on RPO DPO
70
Number of Armotaations
Number Annotasions
(0)
30
DPD (pmrr) RPC (genmo)
40
40
30
300
20
-40
10
3.00 0.3
0.50
120
1.30 133 130
Win
Lose
Tie
Win
Lose
Traniing Spoch
Figure 2: Log probability of the model for chosen responses during the training of RPO (gemma) and DPO (gemma).
Figure 3: Pairwise annotations (by GPT-4) on RPO (beta) vs. DPO (beta) on the test split of the Ultra- feedback dataset.
Figure 4: Pairwise annotations (by GPT-4) on RPO (gemma) vs. DPO (gemma) on the test split the Argilla-DPO-Mix-T datasct.
RPO consistently improves the benchmark performance. We further evaluate the reference model, EPO-trainee model, DPO-trained model, and the officially released DPO-trained model for both beta and gemma series in two standard L.LM chat benchmarks: MT-Bench and AlpacaEval 2.0. MT-Bench is multi-turn benchmark that contains 160 questions across eight different domains of knowledge. The score for MT-Beecc evaluated by GPT-4 on scale from to 10. AlpacaFvval 2.0 a single-turn benchmark including 805 questions on different topics, mostly focused helpfulness.

---

The metrics of AlpacaEval 2.0 are the win rate and Songth-Control (LC) win rate compared with GPT-4 Preview (11/06), where the annotator is also GPT-4 Preview (11/06) and LC win rate is proposed to mitigate the length bias of GPT-4. The results are summarized in Table 2, which shows that RPO consistently exceeds the performance of all the competitors (DPO, Reference model, and the officially released model trained by DPO) MT-BBecch and AlpacaEval 2.0. We also provide additional results on the pairwise win rate for these two benchmarks in Appendix F.3 to illustrate the performance improvement. Finally, we remark that RPO is flexible plug-in module and can steadily improve the benchmark performance without changing the original training configuration or accessing extra preference data. This also sheds light the potential of mitigating overoptimization for better alignment and generalization performance.
Table 2: Results on MTT-Bench scores and AlpacaEval 2.0. zephyr-beta- be and zephyr-gemma- are the officially released models. win rates and Control (LC) win rates in AlpacaEval 2.0 are evaluated by GPT-4 compared with GPT-4.
MT-Bench Score
AlpacaEval 2.0
MT--Bench Score
AlpacaEval 2.0
Model Name
(%c)
Model Name
(56)
(%)
RPO (bcts)
7,381
23.28
21.01
RPO (gemma)
7,916
15.51
13.85
Ref. (heta)
5,088
7.19
4.69
(gemma)
4.61
DPO (heta)
7.278
21.15
17.27
DPO (gemmna)
7,688
15.36
13.69
zaphyr- -btta77 7,200
13.20
10.99
zephyr- aephyr-geman -ee--geaa--- 7b
14.78
RPO also improves the math, reasoning, and coding abilities. In addition to the MT-Bench and AlpacaEval 2.0 benchmarks, we introduce more benchmarks on the math, reasoning, and coding tasks for evaluations of the RPO algorithm. Specifically, we choose the Grade School Math 8K (GSM8K) [14], A12 Reasoning Challenge (ARC) [13], and Mostly Basic Python Programming (MBPP) [4] to measure math, reasoning, and coding abilities of the model trained by RPO, respectively. To due space limitation, we refer the readers to Appendix G for the setups and results of these experiments.
Conclusions
This paper proposes new algorithm that provably mitigates reward overoptimization in RLHF. We establish its umite-sample convergence under a partial coverage style data condition, and provide an equivalent practical implementation, RPO. As flexible plug-in module, RPO exhibits consistent improvement over the DPO baseline and effectively mitigates overoptimization. Future work includes extending our idea of algorithm design online (iterative) RLHF where preference data collected and updated iteratively during LLM fine-tuning. We give more detailed discussions in Appendix B.
Acknowledgement
Zhaoran Wang acknowledges National Science Foundation (Awards 2048075, 2008827, 2015568. 1934931), Simons Institute (Theory of Reinforcement Learning). Amazon, J.P. Morgan, and Two Sigma for their supports. The authors thank Junyan Zhang on valuable discussions on the equivalence between the min-max and max-min optimization.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08874, 2023. 16
[2] Anthropic. Introducing claude. hto.tciinnnauueeeee./////wwwwwwe hwwp:///wwwwww.oooooccccccoo///e 2023. 16
[3] argill. argilla-dpo-mix-7 http:://wuggghggteeerrrr/stte//s
3, 8
[4] Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint arxiv:2108.077:: 2021. 10
10

---

[5] Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo. Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm understand learning from human preferences. arXiv preprint arXiv:2310 10. 12036, 2023. 16
[6] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforeement learning from human feedback, arXiv preprint arXiv:2204. 0.5622, 2022. 16
[7] Viktor Bengs, Róbert Busa-Fekete, Adil EI Mesaoudi-Paule and Eyke Hüllermeier. Preference- based online learning with dueling bandits: A survey, Journal of Machine Learning Research, 22(7):1-108, 2021. 16
[8] Jose Blanchet, Miao Lu, Tong Zhang, and Han Zhong. Double pessimism is provably efficient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage. Advances in Neural Information Processing Systems, 36, 2024. 16
[9] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):324-345 1952. 3
[10] Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint drXiv:2307.15217 2023. 1, 17
[11] Xiaoyu Chen, Han Zhong, Zhuoran Yang. Zhaoran Wang, and Liwei Wang. Human-in-the-lonne Provably efficient preference- based reinforcement learning with general function approximation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 3773-3793. PMLR, 17-23 Jul 2022. 16
[12] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30. 2017. 1, 16
[13] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1.X3:v88803005555 2018. 10
[14] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthis Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021. 10
[15] Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv:2310.022 02.0433,, 2023. 17
[16] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377 2023. 3, 8
[17] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high qualityality instructional conversations. arXiv preprint arXiv:2305.1420 14233, 2023. 8
[18) Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.0676; 2023. 16
[19] Yihan Du, Anna Winnicki, Gal Dalal, Shie Mannor, and R Srikant. Exploration-driven pol- icy optimization in rlhf: Theoretical insights on efficient data utilization. arXiv preprint arXiv:2402.10342, 2024. 16
11

---

[20] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-conirolled alpacaeval: A simple way debias automatic evaluators. arXiv preprint S.Kiv:2404.04477 2024. 8
[21] Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D' Amour, DJ Dvi- jotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.09224, 2023. 17
[22] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos. Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: case study on ppo and upoo arXiv preprint 2005.:2005.12229 2020. 16
[23] Ky Fan. Minimax theorems. Proceedings of the National Academy of Sciences, 39(1):42 47, 1953. 25
[24] Zuyue Fu, Zhengling Qi, Zhaoran Wang. Zhuoran Yang. Yanxun Xu, and Michael R Kosorok. Offline reinforcement learning with instrumental variables in confounded markov decision processes. arXiv preprint ar arXiv:2209.0860 2022. 16
[25] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835-10866. PMLR, 2023. 1, 17
[26] Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, 2024. 2, 9, 16
[27] Jian Hu, Xibin Wu, Weixun Wang. Xianyu, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to- use, scalable and high pege-performance rlhf framework. arXiv preprint arXiv:2405. 114.: 2024. 28
[28] Haozhe Ji, Cheng Lu, Yilin Niu, Pei Ke, Hongning Wang, Jun Zhu, Jie Tang, and Minlie Huang. Towards efficient and exact optimization of language model alignment. arXiv preprint arXiv:2402.0088 00856, 2024. 16
[29] Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rI? In International Conference on Machine Learning, pages 5084-5096. PMLR, 2021. 16
[30] Chenliang Li, Siliang Zeng, Zeyi Liao, Jiaxiang Li, Dongycop Kang, Alfredo Garcia, and Mingyi Hong. Joint demonstration and preference learning improves policy alignment with human feedback. arXiv preprint S.I::24066068744 2024. 16
[31] Zihao Li, Zhuoran Yang, and Mengdi Wang. Reinforcement learning with human feedback: Learning dynamic choices via pessimism. arXiv preprint arXiv:2305.1844 18438, 2023. 16
[32] Xize Liang, Chao Chen, Jie Wang, Yue Wu, Zhihang Fu, Zhihao Shi, Feng Wu, and Jieping Ye. Robust preference optimization with provable noise tolerance for llss. arXiv preprint arXiv:2404.04 /02, 2024. 16
[33] Luofeng Liao, Zuyue Fu, Zhuoran Yang, Yixin Wang, Mladen Kolar, and Zhaoran Wang. Instrumental variable value itcration for causal offline reinforcement learning. arXiv preprint arXiv:2102.09907 2021. 16
[34] Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Salch. Peter Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. arXiv preprint arX1v:2309.06657 2023. 16. 32
[35] Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He, What makes good data for alignment? comprehensive study of automatic data selection in instruction tuning. arXiv preprint arXiv:23 /2. 15685, 2023. 8
[36] Zhihan Liu, Miao Lu, Zhaoran Wang, Michael Jordan, and Zhuoran Yang. Welfare maximiza- tion in competitive equilibrium: Reinforcement learning for markov exchange economy. In International Conference on Machine Learning, pages 13870-13911. PMLR, 2022. 16
12

---

[37] Zhihan Liu, Miao Lu, Wei Xiong, Han Zhong, Hao Hu, Shenao Zhang, Sirui Zheng, Zhuoran Yang, and Zhaoran Wang. Maximize to explore: One objective function fusing estimation. planning, and exploration. Advances in Neural Information Processing Systems, 36, 2024. 22
[38] Zhihan Liu, Yufeng Zhang, Zuyue Fu, Zhuoran Yang, and Zhaoran Wang. Learning from demonstration: Provably efficient adversarial policy imitation with linear function approxi- mation. In International conference on machine learning, pages 14094-1041 14138. PMLR, 2022.
16
[39] Miao Lu, Yifei Min, Zhaoran Wang, and Zhuoran Yang. Pessimism in the face of confounders: Provably efficient offline reinforcement learning in partially observable markov decision pro- cesses. arXiv preprint arXiv:2205. 13589, 2022. 16
[40] Eric J Michaud, Adam Gleave, and Stuart Russell. Understanding learned reward functions.
arXiv preprint arXiv:2012.05882, 2020. 1, 17
[41] Ted Moskovitz, Aaditya Singh. DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca D Dragan, and Stephen McAleer. Confronting reward model overoptimization with constrained rlhf. arXiv preprint arXiv:2310.04373, 2023.
[42] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022. 1, 2, 3, 4, 16
[43] Aldo Pacchiano, Aadirupa Saha, and Jonathan Lee. Dueling rl: reinforcement learning with trajectory preferences. arXiv preprint arXiv:2/ 1.04850 2021. 16
[44] Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive arXiv preprint arXiv:240..13228, 2024. 16
[45] Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From to g": Your language model secretly a q-function. arXiv preprint arXiv:2404. 12358, 2024. 2,
[46] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2023. 2, 3, 5, 6, 16, 32
[47] Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems, 34:1 1702-117166 2021. 16
[48] Paria Rashidinejad, Hanlin Zhu, Kunhe Yang, Stuart Russell, and Jiantao Jiao. Optimal conservative offline rl with general function approximation via augmented lagrangian. arXiv preprint 2rXiv:2211.007/6 2022. 16
[49] Mathieu Rita, Florian Strub, Rahma Chaabouni, Paul Michel, Emmanuel Dupoux, and Olivier Pietquin. Countering reward over-optimization in IIm with demonstration- guided reinforcement learning, arXiv preprint
2024. 17
[50] Corby Rosset, Ching An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models self-improve with general preferences. arXiv preprint arXiv:2404.03775. 2024. 16
[51] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal 2017. 16
policy optimization algorithms. arXiv preprint
[52] Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong. Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. Large language model alignment: A survey. arXiv preprint arXiv:2309. 15025, 2023. 16
[53] Laixi Shi and Yuejie Chi. Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity. Journal of Machine Learning Research, 25(200):1-91, 2024. 16
13

---

[54] Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Pessimistic q-learning for offline reinforcement learning: Towards optimal sample complexity. In International conference on machine learning, pages 19967-20025. PMLR, 2022. 16
[55] Hao Sun. Reinforcement learning in the era of lmss: What is essential? what is needed? an rl perspective on rlhf, prompting, and beyond. arXiv preprint urXiv:2310.06147, 2023. 16
[56] Hao Sun, Alex James Chan, Nabeel Seedat, Alihan Hüyük, and Mihaela van der Schaar. When is off-policy evaluation (reward modeling) useful in contextual bandits? data-centric perspective. Journal of Data-centric Machine Learning Research, 2024. 17
[57] Hao Sun and Mihacla van der Schaar. Inverse-lignmment Inverse reinforcement learning from demonstrations for Ilm alignment. arXiv preprint MXKi:2405./56244 2024. 16
[58] Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of Ilms should leverage suboptimal, on-policy data. arXiv preprint ar arXiv:2404. 14367, 2024. 16, 17
[59] Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. arXiv preprins 24Xiv:2402.057499 2024. 16
[60] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-followin Ilama model, 2023. 18
[61] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1180) 2023, 16
[62] Jeremy Tien, Jerry Zhi- Yang He, Zackory Erickson, Anca D Dragan, and Daniel S Brown. Causal confusion and reward misidentification in preference-baie based reward learning. arXiv preprint arXiv:2204. 06601, 2022. 17
[63] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alexander M. Rush, and Thomas Wolf. The alignment handbook. https://github. com com/huggingfaceo ace/ al -gw.enttaannooooo 2023. 26
[64] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiw:2310.16944, 2023. 3, 8, 32
[65] Masatoshi Uchara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial coverage. In International Conference on Learning Representations. 16
[66] Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao. Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305. 17926. 2023. 26
[67] Yuanhao Wang, Qinghua Liu, and Chi Jin. Is rlhf more difficult than standard rI? theoretical perspective. Advances in Neural Information Processing Systems, 36, 2023. 16
[68] Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.0067. 2024.
16
[69] Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman- consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34:6683-666666 2021. 16
[70] Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems, 34:27395-2743 2021. 16
14

---

[71] Wei Xiong, Hanze Dong, Chenlu Ye. Han Zhong, Nan Jiang, and Tong Zhang. Gibbs sam- pling from human feedback: A provable kl-constrained framework for rlhf. arXiv preprint arXiv:23/2. 1456, 2023. 4, 8, 16, 17,
[72] Wei Xiong, Han Zhong, Chengshuai Shi, Cong Shen, Liwei Wang, and Tong Zhang. Nearly minimax optimal offline reinforcement learning with linear function approximation: Single- agent mdp and markov game. arXiv preprint arXiv:2205.15512 2022. 16
[73] Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for Ilm alignment? a comprehensive study. arXiv preprint arXiv:2404. 10719, 2024. 18
[74] Chenlu Ye, Wei Xiong, Yuheng Zhang, Nan Jiang, and Tong Zhang. A theoretical analysis of nash learning from human feedback under general kl-regularized preference. arXiv preprint arXiv:2402.07314 2024. 7, 16
[75] Ming Yin and Yu-Xiang Wang. Towards instance-optimal offline reinforcement learning with pessimism. Advances in neural information processing systems, 34:4065-4078, 2021. 16
[76] Yisong Yue, Josef Broder, Robert Kleinberg, and Thorsten Joachims. The k-armed dueling bandits problem. Journal of Computer and System Sciences, 1535)115 2012. 16
[77] Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory, pages 2730-2775. PMLR, 2022. 16
[78] Wenhao Zhan, Masatoshi Uehara, Nathan Kallus, Jason D Lee, and Wen Sun. Provable offline preference- reinforcement learning. In The Twelfth International Conference on Learning Representations, 2023. 4, 7, 16, 17
[79] Wenhao Zhan, Masatoshi Uchara, Wen Sun, and Jason D Lee. How to query human feedback efficiently in rI? arXiv preprint arXiv:2305. 18505, 2023. 16
[80] Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse effective unlearning. arXiv preprint arXiv:2404.05868, 2024. 16
[81] Xiaoying Zhang, Francoia Ton, Wei Shen, Hongning Wang, and Yang Liu. Overcom- ing reward overoptimization via adversarial policy optimization with lightweight uncertainty estimation. arXiv preprint arXiv:2403.05171 2024. 17
[82] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic- hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 2023.
[83] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024. 8
[84] Han Zhong, Guhao Feng, Wei Xiong, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprini arXiv:2404. 10222224....2,... 2024. 16
[85] Banghua Zhu, Jiantao Jiao, and Michael I Jordan. Principled reinforcement learning with human feedback from pairwise or w-wise comparisons. arXiv preprint arXiv:2301. 2023. 7, 8, 16, 17
[86] Banghua Zhu, Michael Jordan, and Jiantao Jiao. Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf. arXiv preprint arXiv:2401.16335, 2024. 2, 8, 17
[87] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford. Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. 2019. 1, 16
arXiv preprint arXiv:
15

---

A Related Works
In the following, we relate our work to recent lines of RLHF research on both theory and practice sides. We also review related works on reward hacking and overoptimization in RLHF
algorithm design. The technique of RLHF [12, 87, 42, 6, 18, 55] has recently demonstrated its great importance in building the state-of-the-art LLMs, including ChatGPT [1], Gemini [61], Claude [2]. In the RLHF pipeline, the LLM is fine-tuned towards maximizing a learned reward model for better alignment [52, 57] with human preference using RL algorithms such as Proximal Policy Optimization (PPO; [51]). Meanwhile, PPO-style algorithm is also known for its instability, and especially, a high demand for proper hyperparameter tuning [22]. This thus casts prohibitive computational cost to make the most effectiveness of PPO-based RLHF methods to align L.LMs, especially for the open-source community.
RLHF:
Given that, further research on RL.F has explored various alternatives to PPO-based methods, with the most popular approach being the direct preference optimization method [82, 46), which skips the reward learning phase and directly optimizes the LLM to align it with the human preference. Our practical implementation (RPO) also harnesses the wisdom of reward-LLM equivalence to avoid explicit reward learning followed by PPO training.
Besides the original DPO algorithm [46], ever since it popularizing the direct preference learning style method, variants of the direct preference learning approach are proposed. including but not limited to [34, 5, 71, 59, 28, 74, 44, 26, 50, 32, 80, 58, 68, 30]. Each of them aims address further challenges of direct preference learning from varying perspectives. Specifically, the algorithm proposed by [44, 26] share similar algorithmic components as RPO proposed in this work. Both work consider SET style regularization during preference optimization. However, theoretical understanding of how SFT loss can help alignment remains unknown. In contrast, we provide theoretical justifications to the SFT loss as an implicit adversarial regularizer that provably mitigates overoptimization in preference learning.
RLHF: theoretical investigation. Initiated from the literature of dueling bandits and dueling RL [76, 7, 43], recent success of RLHF in fine-tuning LLMs also motivates a long line of research
to investigate the theoretical foundations of RLHF under different settings [11, 85, 78, 79, 67, 31, 71, 74, 19, 84], aiming to propose provably sample-efficient algorithms to learn a human-reward- maximizing policy from human preference signals. Our theoretical study of RLHF falls into the paradigm of offline learning from pre-collected preference dataset, and is mostly related to the work of [85, 78, 31. 71, 74]. this setup, the main challenge is to address the overoptimization issues due to human reward uncertainty and distributional shifts when only fixed dataset is available. In the sequel, we compare our work with them in more detail.
Existing theoretical work on provably sample-efficient offline RLHF typically suffers from two drawbacks: they are either restricted the linear function approximations setting [85, 71] which is far from the practical situations, or are generally unable to be implemented in the LLM experiments. Typically, to encompass the pessimistic principle in the face of uncertainty, the existing literature proposes to return the optimal policy against either an estimated reward model plus structure- aware reward uncertainty penalty [71] or the most pessimistic reward model inside a confidence region [85, 78). Both of these two types of method involve intractable components for implementation and needs for additional algorithmic design to approximate the theoretical algorithm practice. In contrast, our theory works in the context of general function approximations while being friendly to be implemented. Finally, we remark that, while our study focuses on the standard BradIcy-Teee Terry model of human preference with general reward function approximations, the work of [74] further considers a general human preference model. But it remains unknown how their algorithms can be efficiently implemented in practice. It serves as an interesting direction to extend our technique to RLHF with general reward model and device new practical algorithms.
Finally, we mention that the algorithm design of RPO is also related to the "pessimism" principle in the standard offline RL literature. It proposes to maintain a pessimistic estimate of the policy values or constrain the policy not to take unseen actions in the data to handle the challenge of the insufficient coverage of the dataset, e.g., [29, 65, 69, 70, 47, 75, 72, 36, 54, 77, 39, 48, 53, 8, 38, 33, 24]. In contrast, consider the offline RLHF problem and the techniques to obtain the objective of the
16

---

RPO algorithm (see Section 4) along with its sample complexity analysis are new and different from these works.
Reward hacking and overoptimization in RLHF for LLM. As is discussed in the introduction, the challenge of reward hacking or overoptimization may prevent the successful alignment of LLMs, imperfect, overfitted, and misgeneralized proxy reward learned from the finite data [40, 62, 25, 10]. Efforts have been made to mitigate this fundamental issue through the perspective of theory, e.g.. [85, 71, 86], and practice, e.g., [15, 21, 41, 81, 49, 56]. Our approach starts from the theoretical insights of handling inherent uncertainty in learning human preference from finite data, while being surprisingly easy to implement.
degenerating the performance of an LLM because of maximizing
B Limitations and Future Works
One limitation of the current work is that we focus on the setting of offline RLHF where only fixed preference dataset is available. Recent RLHF research has shown great potential of using iterative methods for LLM alignment with multiple rounds of preference data collection and tuning [71, 58).
Future works include extending our idea of theoretical algorithm design and analysis to the iterative RLHF setup where further preference data can be collected. Also, since our practical algorithm RPO a plug-in module that effectively mitigates overoptimization and improves alignment performance, it serves as an exciting direction to combine it with explorative preference data collecting mechanism in iterative RLHF to further boost the performance of LLM alignment.
C Further Discussions
Discussions on Algorithm 1 and Theorem 5.3. We compare our theory with [71] and (78).
Remark C.1 (Comparison with [71])- Another theoretical work RLHF [71] explicitly models the KL-regularization between the target policy and the reference policy in the learning objective, referved as the KL-regularized contextual bandit. This means that their metric becomes the KL-regularized expected reward. In contrast, here we put the KL-regularzzatinn as a component of our algorithm design, but we still keep the metric as the expected reward (2.2). Therefore our theory in Section 5. directly reveals how the learned policy performs in terms of the expected reward compared to any given target policy (which can be a stochastic policy).
Remark C.2 (Comparison with (78)). We remark that in the work of [78], they also mentioned maximin object similar to (3.2) for offline preference- based RL as a complementary to their theoretical algorithm. However, the sample complexity of the maximin-styye algorithm they presented is unknown. while we provide finite sample convergence result for Algorithm 1 in Section S. Furthermore, our objective (3.2) features another Kl-regularization ferm, which is essential for the proposal of our new practical algorithm design for aligning L.M in Section 4.
Discussions on the partial coverage assumption (Assumption 5.2). A sufficient condition to make this partial coverage condition (Assumption 5.2) hold is that the distribution of the offline dataset, which is prp can well cover the joint distribution of (a',a") .Q%) (Iz, . bhass). Here to discuss focus on n base chosen as we adopted in the experiment part.
First, we clarify that the offline dataset distribution pDD not simply (a aa.) ~ (Ix unchosen chosen)) since according to our definition (see Section 2) whether at or a" is chosen is random and is determined by E 0, 1 obeying the BT model. Thus, (a aa)) ~ μuD can be interpreted as a mixture of umchhsee . choson) ) and ( chosenn x zmmchossn) ). This mixture probability would not be too small as long as the quality of (a a²) does not vary too much, i.c., both of them are possible to be chosen, which is the case in practice. As result, in the offline data distribution (a* ,a)) (LD, both at and a partly comes from the chosen distribution n chosen
Then in order for ND to cover the joint distribution of (a²,a") ~ (TT, π base), it suffices to argue that T chosen can cover the target policy ग, which then reduced back to the traditional coverage condition. Thus our assumption essentially requires that x. thosen well covers and only needs to cover the target policy ग. This coincides with the spirit of the minimal data assumption in offline RI. theory, i.e., the so-called partial coverage condition.
17

---

On the relationship between observed chosen probability and reward overoptimization. First, we note that the actions and their chosen probabilities can be interpreted as a proxy of analyzing the un- derlying (estimated) reward model F due to the representation (=(a)z) x zee (alar) exp(8-1f(x,, a)). Analyzing the (log) probabilities of the actions can be utilized to detect the mitigation of over- optimization, because according to the representation, an overestimated reward of poor action would result in a higher probability of choosing this action, and would also cause a decay in the probability of choosing other better actions (since the probabilities are normalized to 1).
To further showcase the ability of RPO address overoptimization (through the lense of probability). consider the following theoretical example with only one state and three actions [73] where we can track everything clearly. It has three actions a, b, with R (a) 1, R
(b) (b) 0.5, R (c) 0. The reference policy ze (a) re (b) 0.4, x (c) 0. and the dataset consists of data point D (a, 6, 1) (meaning action is preferred in the data). Then an ideally solved DPO objective would be TDPO as long as π (b) (b) O, and the value of π DDPO can be arbitrarily chosen in [0, 1]. Thus possible solution to DPO would be (DP = 0.5, π DPP (b) O, and by the normalizing condition л DPO (c) = 0.5, which undesirable since the action e has reward R (c) 0. In contrast, solving the RPO objective would additionally require the maximization of TRPO (a) due to the SFT regularization term, and thus the solution is shifted towards TRPO (a) 1, TRPO (b) (c)
which is better than the DPO policy. Thus, RPO is able to prevent overoptimization towards poor actions that are less covered by the dataset (action e here), therefore resulting better policy.
About the relationships and distinctions between PTX loss in [60] and the SFT loss of RPO. The original PTX loss an imitation loss calculated on the pretraining data. contrast, the SFT loss in the RPO objective is an imitation loss calculated on the RLHF dataset. In more specific, our experiments use this SFT loss to imitate the chosen responses the RLHF dataset. Thus the relationship is that they are both imitation loss which aims to mimic certain data distribution. The distinction is that they are calculated on different data sources. The SFT loss the RPO objective naturally comes from theoretical algorithm and provably serves important regularization term to mitigate overoptimization in offline RLHE
About the computational complexity of the SFT loss gradient. According to the paragraph Practical implementation in Section 6, RPO adds an additional SFT loss (the log probability of the chosen labels in the preference dataset) on the original DPO loss, where we highlight that the SFT loss actually an intermediate quantity in the calculation of the DPO loss. Hence, our proposed method does not incur any additional computation overhead compared with the vanilla DPO.
18

---

D Proofs for Sample Complexity Analysis
D.1
Proof of Theorem 5.3
Proof of Theorem 5.3. By definition, the suboptimality gap of ir w.r.t. x is decomposed as following,
Gap" " (7)
Bggdd (x;
(a,, a)]
ax (r -x" - (x()))
Eggdd mot( (t)) (1)
1)
min rER
E 1)
(1(r,at) -r(o) - 3 (T((7(1>)] vet (11:1) + Lo(r)
a mas bave (-10)
17 aadd ((z,a")-r(z,, 3 ([T((1))] + co(r)
+ 17
min
rER
nox 1o0e (-1o)
Rxwddo at (00 (r' (x, (xr 33 ([7(((1)]==**
B Ex KL (T (+1=)][ (-1z)) KL (+((1))))
Term (A) + Term (B) + Term (C);
(D.1)
where in the above Term (A), Term (B), and Term (C) are abbreviations for
Term (A)
(r' (2, ((x,aaaa)r"aa) ax, a -3- KL (T
Ez al
(r(x, rr a(x, 8 (L( ro (-1x)) + co(r) }.
11
min
17 E
wa (W((12)) a' barn (1(z)
and
Term (B)
1)
min rER
7) Exndo (-)laa) a have (-12)
((z,aa) r(x, - (1 + Co(r))
(r (Ix - (xx 3 <=(((100)
and
Term (C) 3 Exrwdo KL (=((((1)))] KL( (+[x))
In the following, analyze Term (A) and Term (B) respectively.
19

---

Upper bound Term (A). Notice that by the optimality of our choice of policy मे in (3.2), we have
Term (A)
Bo wa ( (x"r,a)) ([)(
-
(D.2)
Bando ((a (1 + Co(r))
11
min rER ?
a (Puxphernn)aa)
< Bxndo
(1 - (
1) E. Bando ( r(z r(x, 3 + + Col(t)
1)
rER
a bare (.))) (.(x)
(r" -r'( (x(r))))) -(r(x,a)) r(x,,))) - -n7² ca/r))
max rER
E Ex al
where in the inequality we apply the optimality the choice of policy (3.2).
Upper bound Term (B). For this term, we directly consider the following bound,
Term (B)
= 1)
min rER
1) Beddo a' (-(le)) -(z,,)))r (L(((())]] (+x)) + Co(r) Nor (11x)
(1r),)))))(x - 3- (1 < Bxndooal al (r) (2, - (x, - - B (.. (1) + + LTD(r*)
E Eg
E Enwdo with aa boot ax) - ax, - 3 (1((1))))===+<+11)))))))
= - Cp(r"),
(D.3)
where in the inequality we apply the fact that r* E R by Assumption 5.1.
Combining Term (A), Term (B), and Term (C). Now by (D.1), (D.2), and (D.3), we have that
Gap](a) ( = Term (A) + Term (B) + Term (C)
(D.4)
(1* (r=(xxaa) - -x"( - - (r(z, a²) -r(,,a))) + +
(Co(r")-Co(r))
< max PER.
atndinatnatta m (-(a))
3. Ea (T. (-lx)) (z) (+1=))
In the following, we upper bound the right hand side of (D.4) via relating the MLE loss difference term to the reward difference term through a careful analysis of the preference model. On the one hand, we invoke Lemma D.1 give an upper bound of the difference of the MLE loss as following, with probability at least 1 - 8 over random samples and (6 (1 (1+ eR) N)-1, for any reward model r E R, it holds that
C - (r)
5 -2 Eixaa B aa)
D2 Ditelliager (Pp,- (-|2, a¹ (a (-|xx, a", ,a,a))))
20

---

3 N
+
log
where we recall that we use the subscript r in P. to emphasize the dependence of the probabilistic model on the reward model. Here No. (R, - Iloo) denotes the e-covering number of the reward model class and R is the upper bound on the reward functionss (Assumption 5.1). Now to facilitate the calculation, we lower bound the Hellinger distance by total variation (TV) distance
Ditliinge (Pr- (-|z,, a', a a')[[P((-lx a', ,a²) 2 Div (-|2, a², aa))|IP(((aa.... a' aa)).
By the expression of the probability model IPr, we can further write the TV distance above as Drv p. (Pr (
2 of (r, -(rat)) r(z,,a))))
dt)) - a(r(x, a r(z,a')))
- - -
= 0
r(z, a()x
(D.5)
where in the second equality we use the fact that (- condition that r(z, a) E [0, R| for any a)) E X' x A x R (Assumption 5.1), we know that
a(2). Now by Lemma D.2 and the
| (ort((, a+) - (x,a") - ( (²) - rxxx
2x ( (r(z,a') r(x, a0){
where K = 1/(1 + exp(R))2 As result, the difference of the MLE loss is upper bounded by LD((*) (r(
< 22k² E(z,a' a mmp(r)))))) (1 (r*,)))))) - (r(z, r(z, a |²]
3 N
Ne(RR || 1100)
log
(D.6)
8
On the other hand, the reward difference term in (D.4), which is evaluated on actions from т and π hase be related to the reward difference evaluated on the data distribution μED via Assumption 5.2, i.e.
E Eg ~x(l (-1x)) (r (x, a²) r* (x, (r(a, r(), -
(D.7)
S Cwo (R; R., π base
/ B pappp (1 - a((x,a)) r(z, r(x,
Finally, combining (D.6), (D.7), and (D.4), denoting
(1-(0,0)) r(()))))) - (z) 2
Ar V
we have that
Gap th (7) < max
Cno (R; n. x base)
A - 22- 22
3
loog
.N,RR,
-100))
FR
n/V
8
+ 3 Exwdo
(L(T((12)||1
(-|x)) KL (Z
(Cup (R; n., क base 2
3
No 11
+
log
8k.²
nN
8
+ B Egndo
[T((T(-)2))] (-1=))
21

---

where in the second inequality we use that fact that az - bz² < /²/(4b) for any 2 E R and that KL-divergence non -egaaivaeee. Consequently, with the choice of
11 = 2=266 log (W&(R. || 16x)//) B VN
K (1 exp R))2
N
we conclude that with probability at least 1 and = (6 (1 + elt) NN
Gap"(n
(Ri n ) + +1) 4BBadd KL (=([++))] ref (-1x)) v6(1+ exp(1))) 2 (CCwo(R;
AVN
where denote 1 Vlog (N: (Ne (l..)//))))/))) with (6 (1++ee))-NN- of Theorem 5.3.
This finishes the proof
D.2 Technical Lemmas
Lemma D.1 (Uniform concentration). Consider the MLE loss (3.1) and define the approximation
error as E = (6- (1 (I+e").NN) where R is the upper bound on the reward functions (Assumption 5.2). Suppose that the reward model class R has finite e-covering number No (R, (lle) < oc. Then for any < 1/e it holds with probability at least ô that
CD(r") Lr(r)
< 2 Biz,a* a²) (ompp())))) Ditellinge (Prr- (-/22,,, u², al,r,aaaa'')) aa)))
3
log (N, ol0)
+ N
5
Proof of Lemma D.1. For notational simplicity, we use C. (R, ||x))) to denote an e-cover of the reward model class R under the || -norm. It holds that (R, || || - Cc (R, I| ll First we invoke Proposition 5.3 of [37] to obtain a uniform concentration over the finite set of E-cover CR, llxc). Specifically, with probability least 1 8, for any C. (R, lloo).
Co(r*) - (r(r)
< - Ei ,a (()=w((()))) Di Ditlllingrr (Pr, (-122, a² aa)) (-Iz,, (l1lalaace, aaaraa"aa))
2
log (N, || (1.10)))
+ N
(D.8)
5
Now for any reward model r E R, we take art CR, || (loo) satisfying ||r jrr-rilla S E. We have
Co(r') )- - LD(rr
Co(r") Lo(rt) Lx(rt) - Lo(r)
-2 (( Ditellnger (P (- at (())
Ne (I,, -llee))) + -o(rt)- -(o()
log
22 Brr,,' (20) [ Diielligerr (-12, ad)) aa)]
N. |1 11.10))) Co(rt) - Lr(rr
log
+ 4 E (a,a" a (m Diiillinger (+,t (f|or, al (I) (-1,,a aaon,at,a00 a²)]) (D.9)
where in the fir inequality we (D.8) for r' and in the second inequality utilize the triangular inequality for Hellinger distance. Therefore, it remains to upper bound the approximation error induced by rt. On the one hand, by the definition of Lp in (3.1), we have that
CD(rt) -p(()
22

---

N
o ( at) ( (()i,))))) a?))
yi log
o ofr (xi, at) - rt (2 a.))
N
( a②) r (Ci, (x),
(1 n.) log
N
(x a²))
==1
Use the inequality that log(z) S x - 1, we can further upper bound LD(r!) (D(r) by
(D(rt)= (r(r)
N
(r(xz, -(), (r at) - (s1,
N
a(ri(1,,,)) (2,,
a(((z;,))) - 7( at:) - o (x,, (Z;;,])) r (z,, a;))
v:)
a ( a.! (z),
1=1
Now since ||rt - 1/ S E and rt E [0, RR, invoking Lemma D.2, we can derive that
(r(a,, at) r(wi, a()) (1+ (1+
(Ji at) (u rl (2 (x)),())
CD(rt) - -())) -
N
(r(x;, - r(o;, a ))
(rl (21, ag) (2,, at))
N
(1 (1+e"))-
1
S 4
||r r so (1 4e (1 + e
(D.10)
On the other hand, we upper bound the hellinger distance between P, and P,r, for any (a,a",a") X' A.
D Ditellnggr (P,1(11, |
Drv(P. ) -
((("(xx, r a)))
o(r(x, a²) -((,,,)
- -rl - (r(z,a²) r(z, (((=,a)))r((aa))))
S ||r r11o0 2E,
(D.11)
where the first inequality uses the fact that Ditellnge DTV, the equality uses the same argument as (D.5), and the second inequality applies Lemma D.2. Finally, combining (D.9), (D.10), and (D.11), we conclude that
Colr*) Cv(r) -2 B a .0) (~p Ditelllngerr (-rr- a², (lo
+ N log (Ne(R,
2
|1 (1~0) -||1=>)))) + 6E (1 (R).
8
By taking the approximation probability at least 1 - ō, for any E R, holds that
(6 ) NN)1, conclude that for ze_1, with
(D(r*) (r(r)
< - Biz,a' ad) (mppr))))))))
Ditelllngerr (P,2,111,, (allz,aa,a)))0 ,a"))|P, aa))
3
Ne(R,
ll00)
log
N
8
This completes the proof of Lemma D.I.
Lemma D.2 (Sigmoid function). For any real numbers 21, E [-R, RJ. it holds that
k |ž| #2| S (a(21) (z2)) |21 =2l.
where the constant 1/(1 exp(())
23

---

Proof of Lemma D.2. Since the sigmoid function a(-) is differentiable, we know that for any 21, RR, R), there exists some E(21. 22) E [-R, R] such that
o(21) - (22)) o' (5(21, 22)) (21 - 22).
Notice that d'(z) = 0(2) - (1-o(z)), we can obtain that
1 2 o' 05((21, (2)) σ (5(21, 22)) (1 (1-0(((21) 0(((21)
11-1- (2))
1 1 exp(E(21, 22))
eexp-( (1 (1-7+ exeepppp
exp(R))
(1 +
This completes the proof of Lemma D.2.
E Proofs for Equivalence between Maximin and Minimax Objectives
E.1 Proof of Theorem 5.6
Proof of Theorem 5.6. Consider denoting an auxiliary policy as
argmax min ((TT, r).
(E.1)
#Ell
rER
By the definition of F and मं the duality gap of (f, 7), defined as
Dual(f, ft) = max (fn, f) min (f,
#ell
reR
is zero. This is because the following deduction,
Dual(f,
max (TT, - min max 6( (TT,))) TEl
rER fell
max min ((π min ((F,
REll rRR
PR
0,
(F.2)
where in the first equality we apply Lemma E.I that the minimax objective and the maximin objective equivalent, and the last equality applies the definition of f and î respectively. Note that rewrite the duality gap as following
Dual (f, .))
max ((T + ((..)) well
(d(T)))
min )(
(E.3)
FER
Combining (F.2) and (E.3), we can conclude that
max *ell
((,,
(
A argmax (f,, R).
(E.4)
sell
Now comparing what me and satisfy in (5.4) and (E.4) respectively, invoking Lemma E.3 that maximizer of d(;r)) given any R. is unique on the support of do, we can conclude that
πサ(-|z) = (-|rr), Vr E Supp(do).
(E.5)
Therefore, by (E. 1) and (E.5), and the fact that )(π, r) depends on ा only through its value on the support of dg. we can conclude that
E argmax min ((T, r). TEnl PER
This finishes the proof of Theorem 5.6.
24

---

E.2
Auxiliary Lemmas
Lemma E.1 (Equivalence of maximin and minimax objectives). For the policy class II defined in (2.3) and the reward model class R satisfying Assumption 5.5, it holds that the maximin objective is equivalent to the minimax objective, i.e.,
max min @(T,T) = min max ((T, Telf FER.
FRR Tell
Proof of Lemma E.I. The foundation of this result a minimax theorem given by [23] (Lemma E.2). In setting, the policy class IT is nonempty set, and the reward model class R. is nonempty compact Hausdorff space. Furthermore, by our choice of the policy class II in (2.3), IT is convex set. Meanwhile, the function is a concave function of II since the dependence on is linear terms plus a negative KL term (concave). Finally, by our assumption, the function á is convex-like on the reward model class R and is also continuous on R. As result, all the conditions of Lemma B.2 satisfied and the minimax theorem holds in our problem setup, finishing the proof of Lemma E.1. O
Lemma E.2 (Minimax theorem [23]). Let X be a nonempty set (not necessarily topologized) and
y be nonempty compact topological space. Let f X x y + R be lower semicontinuous on y. Suppose that is concave-like X and conver-like on V, i.e., for any 21, x2 X, a E [0, 1], there exists D3 E X such that
f(zs.) )) 2 f(x1, (1- fx2,, V,
and for any y1, 1/2 V. [0. 1), there exists ys E y such that
[(-38)) S 3 ((-3n) + (1 - 3) (1,31/2 on V
Then the following equation holds,
max min (xz y) min max f (I, 4) zEX yEy
yEy xE
Lemma F.3 (Unique maximizer of @). Consider the function & defined as
((T,1) r) 17 watl)) + Lr(r).
((22, at) - -r(x - DKD (1(11))))
Then given any R, the maximimzer of ((.T)) is unique the support of do-
Proof of Lemma E.3. Given any R. consider that
max (TTTT) Tlll
E Bl (((,,a)) B. ((((
=1 max TEI
-1)( exp(8-1 r(x. +) Sa de x) exp(8 r(z, a²))
}
DKL (((1)))
= 7) max Tell
C, - B Ezndo
where
C, Exands B log (1 da da exp (g- r(x, r(x.,a))) m))]
a constant independent of π. Therefore, the maximizer o(- r) on the support de must equal to (+(x) expp3-1.rrxz Tz (-1x)) Jaea dr (a/lx) exp(p- 3- r(x, a'))
which completes the proof of Lemma E.3.
25

---

F
Additional Details on Experiments
Training Details
We train the gemma series models with 8 NVIDIA A6000 GPUs and the beta series models with 8 NVIDIA A100 GPUs, where they are all GPT-like models with around billion parameters. It takes around three hours to train a beta series model and five hours to train gemma one. Our codebase is adapted from the Aligmment Handbook [63]. By comparing the validation loss on the test split (not used for later evaluation), we select the hyperparameter 1) of both RPO (beta) and RPO (gemma) to be 0.005. We list the remaining training configurations in Table 3, which are recommended by the Alignment Handbook.
Configuration
Beta Series
Gemma Series
learning rate learning scheduler type
5.0e-7
5.0e-7
cosine
cosine
warmup ratio batch size gradient accumulation batch size per device training epoch B optimizer seed precision
1.0
1.0
128
128
16
8
2
0.01
0.05
adamw torch
adamw torch
42
42
bfloat16
bfloat16
Table 3: Training configurations for beta series and gemma series models in this paper.
F.2
Evaluation Details
GPT-4 evaluation on the test split. We use the following prompts to guide GPT-4 to annotate the preferences among win, lose, and tie (we denote them by A, B, and C, respectively).
Prompts: Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the question displayed below. You should choose the assistant that follows the user's ' instructions and answers the user's question better. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level detail of their responses. Begin your evaluation by comparing the two responses and provide short explanation. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length the responses to influence your evaluation. Do not favor certain names of the assistants. Be objective as possible. After providing your explanation, output your final verdict by strictly following this format: [[A]] if assistant A is better, [[B]] if assistant B is better, and [[C]) for a tie. [Instruction] instruction [Thhe Start of Assistant A's Answer] (answer A) [The End of Assistant A's Answer] [The Start of Assistant B's Answer] [answer B) [The End of Assistant B's Answer]
Here, we replace (answer A} and { (answer B) with the answers of two models. Since GPT annotation has shown prefer the answer in the first position [66], we randomly exchange the positions between two answers during the evaluation to ensure a fair comparison.
Benchmark evaluation. We use the default configuration for the evaluations on MT- MT-Bench' and AlpacaEval 2.0 By default, the annotator of MT-Bench is the latest version of GPT-4. The default annotator and the competitor model are both GPT-4 (Preview 11/06). We only need to manually import the proper chat template that formats the training dataset, which are shown as follows.
wttp:uu..commmmmmaemeeaeee http://:hhhhhtoocccccccc///////t Phttps://githen www./tttottneneeeeee http://ww./oaaaaeeeaaaaaaaeaeeaa
26

---

Chat Template for Beta Series: <lsystemb<<<><<uusrr>> (instruction] <//>> <lassistanti>
Chat Template for Gemma Series: <bos> uttatttbbbuesrr (instruction] <lim _nddl>
him_startbaassistat
F.3 Additional Results Experiments
In this section, we provide the additional results to show the performance gain for RPO (beta) in MT-Bench and RPO (gemma) in AlpacaEval 2.0. We report the pairwise win rates in Tables 4. 5, and 6 to analyze their performance gaps, where all the annotation configurations are the same in Table 2. Results show that RPO still exceeds DPO in the metric of the pairwise win rates on the benchmarks for both beta series and gemma series.
win rate (%)
RPO (beta)
Ref. (beta)
DPO (beta)
RPO (beta)
50.00
83.75
57.81
Ref. (beta)
16.25
50.00
21.25
DPO (beta)
78.75
42.19
50.00
Table 4: Pairwise win rates (left right) for beta series models on MT-Benchmark.
win rate (%)
RPO (beta)
Ref. (beta)
DPO (beta)
(bO(beta)
50.00
80.13
52.02
Ref.
19.87
50.00
20.61
DPO (beta)
47.98
79.39
50.00
Table Pairwise win rates (left vs. right) for gemma series models on AlpacaEval 2.0.
win rate (%)
RPO (beta)
Ref. (beta)
DPO (beta)
RPO (beta)
50.00
64.93
51.33
Ref. (beta)
35.07
50.00
36.44
DPO (beta)
48.67
64.56
50.00
Table 6: Pairwise Length-Control (LC) win rates (left vs. right) for gemma series models on AlpacaEval 2.0.
G Experiments on Math, Reasoning, and Coding Tasks
G.1 Experimental Details
To provide a more comprehensive analysis of the trained LLM, we introduce more benchmarks on the math, reasoning, and coding tasks for evaluations. Specifically, we choose the Grade School Math (GSM8K), AI2 Reasoning Challenge (ARC), and Mostly Basic Python Programming (MBPP) to measure math, reasoning, and coding abilities, respectively. In this section, we focus on the gemma series for the experiments. We not use chain-of-thought or few shots all the benchmarks. We compare the greedy decoding result (pass @1) on the MBPP benchmark.
27

---

GSM8K (%)
ARC
MBPP (Pass @1)
Model Name
Easy (%)
Challenge (%)
Normal (%)
Plus (%)
RPO
49.9
79.1
49.8
54.2
46.3
DPO
45.3
75.7
50.0
54.2
43,9
Ref.
45.4
75.0
45.8
50.3
44.2
zephyr-gemma- 7b
47.3
77.6
48.6
54.5
44.7
Table 8: Results on GSM8K, ARC, and MBPP. Here, zephyr-gemma- is the officially released models trained by DPO and Ref. denotes the reference model zephyr-7b gem-a-mmmmttttt used for our training. RPO and DPO are trained with the OpenRI.HF codebase [27] and we average the SFT loss regularizer in RPO by the number of tokens of the chosen response. We not use chain-of-thouge or few shots in all the benchmarks. We compare the greedy decoding result (pass @1) for MBPP.
Here we use the OpenRIHF codebase [27] to implement new variant of RPO, where the SFT loss regularizer is averaged by the number of tokens of the chosen labels, that is, (log H ( [z)) / Such variant balances the weight of the averaged SFT loss regularizer between the shorter chosen response and the longer one. We set the coefficient for the SFT loss regularizer as 0.2. We use NVIDIA A100 GPUs for the training and evaluation. The remaining hyperparameters are in Table 7.
Configuration
Gemma Series
learning rate learning scheduler type batch size
5.0e-7
cosine with minimum learning rate
128
gradient accumulation batch size per device
8
2
training epoch |
2
0.5
optimizer seed precision
adamw torch
42
bfloat16
Table 7: Training configurations for DPO and RPO for the experiments in Appendix G.
G.2
Experimental Results
Table 8 demonstrates that our proposed method still outperforms or performs equally to the vanilla DPO these benchmarks of math, reasoning, and coding, which verifies the effectiveness of our proposed method.
28

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We support all the claims made in the abstract and the introduction sections by our theory and experiment sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
29

---

Justification:
Please see the discussion of limitation of the work in Appendix B.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sellispecifiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
We provide the complete and accurate proof in Appendix D.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all the detailed training and evaluation configurations in Appendix F.1 and F.2 for reproducibility. We also submit the codes the supplementary for the purpose of reproducibility.
Guidelines:
30

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We submit the codes in the supplementary. All the datasets, reference models, and benchmarks used in this paper are open accessed.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
public/guides (Cooeeeummsssioonaaaaa
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / inips.cc/publle htt//uuiessssee (CodeSubmiii wwt.sSuubiisioonee. aasionoooicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
31

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see the dataset choice and the detailed training configurations in Section 6 and Appendix F.I.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to the budget of time and expense, we do not report the error bar, which is also common in many RLHF literature [46, 34, 71, 64]. However, we report all the training configurations and the random seed to ensure reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the information in Appendix F.I.
32

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : //nuurrpss.... caesscc//// http/uublc//EEtlee t cewpiiicssiiiiiiea ines?
Answer: [Yes]
Justification:
We have reviewed and followed the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no socictal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
33

---

Answer: [NA]
Justification: The scope of our work is not to publish a new released model but to analyze the overoptimization pheromone in RLHF both theoretically and empirically.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We respect all the licenses and terms of codes, models, and datasets used in this paper. We also properly cite their creators in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that re-packaged, both the original license and the license of derived asset (if has changed) should be provided. this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
34

---

14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "yCpwdZ7jrL", "reviewer": "Reviewer_JyGr", "summary": "The paper \"Provably Mitigating Overoptimization in RLHF\" addresses the issue of overoptimization in aligning large language models (LLMs) with human preferences using reinforcement learning from human feedback (RLHF). \n\nThe main contributions include:\n1. Identification of Overoptimization Source: The paper identifies the source of reward overoptimization as a distributional shift and uncertainty in learning human preferences.\n\n2. Theoretical Algorithm Proposal: It proposes a theoretical algorithm that minimizes the maximum likelihood estimation of the loss and a reward penalty term to mitigate overoptimization, ensuring provable sample efficiency.\n\n3. Practical Implementation: The algorithm is reformed into an easy-to-implement objective combining preference optimization and supervised learning loss, named Regularized Preference Optimization (RPO), demonstrating improved performance in aligning LLMs compared to existing methods​", "questions": "1. I just wonder about the extension to online RLHF. For online RL, based on the optimism principle, it seems that then the objective should be subtracted from the SFT loss, which obliviates the wish to avoid overoptimization. So how to balance the exploration and avoiding overoptimization for the online setting?\n\n2. What is the additional computational complexity brought by the gradient of SFT loss? Besides, the author doesn't mention how to approximate the gradient of SFT loss since there are expectations.", "limitations": "See weaknesses.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper not only provides rigorous analysis but also has solid experiments to solve the overoptimization problem for DPO.", "weaknesses": "The partial coverage condition lacks discussions since now it's a pair over (\\pi,\\pi^{base}), which is different from the traditional coverage condition. For example, for the linear case, $C_{\\mu_D}$ would approximately become\n$$\n\\mathbb{E}_{x,a^1\\sim\\pi^*,a^2\\sim\\pi^{pref}}\\sqrt{(\\phi(x,a^1) - \\phi(x,a^0))^{\\top} \\Sigma_D^{-1} (\\phi(x,a^1) - \\phi(x,a^0))},\n$$\nwhere $\\pi^{pref}$ means the distribution of the chosen samples. We know that $\\Sigma_D$ is composed of pairs of chosen and unpreferred samples, but the $(\\phi(x,a^1) - \\phi(x,a^0))$ is there pair of optimal policy and the policy represents the chosen samples. Hence, if we want to compete with a policy $\\pi$ better than $\\pi^{chosen}$, how can the direction of $(\\pi,\\pi^{chosen})$ be covered by $(\\pi^{unprefered},\\pi^{chosen})$?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The partial coverage condition lacks discussions since now it's a pair over $(\\pi,\\pi^{base})$, which is different from the traditional coverage condition. Hence, if we want to compete with a policy $\\pi$ better than $\\pi^{\\mathrm{chosen}}$, how can the direction of $(\\pi, \\pi^{\\mathrm{chosen}})$ be covered by $(\\pi^{\\mathrm{unchosen}},\\pi^{\\mathrm{chosen}})$?**\n\n**A1:** Thanks for raising this question! and we appreciate your suggestion to include more detailed discussions and explanations on this partial coverage coefficient. Here, we briefly explain this condition and address your concern on its rationality. \n\nEssentially, a sufficient condition to make this partial coverage condition (Assumption 5.2) hold is that the distribution of the offline dataset, which is $\\mu_\\mathcal{D}$, can well cover the joint distribution of $(a^1, a^0)\\sim (\\pi, \\pi^{\\mathrm{base}})$. We can focus on $\\pi^{\\mathrm{base}} = \\pi^{\\mathrm{chosen}}$ as we adopted in the paper. \n\nFirst, we would like to clarify that the offline dataset distribution $\\mu_\\mathcal{D}$ is not simply $(a^1, a^0)\\sim(\\pi^{\\mathrm{unchosen}}, \\pi^{\\mathrm{chosen}})$ as understood by the reviewer, since according to our definition (see Section 2) whether $a^1$ or $a^0$ is chosen is random and is determined by $y \\in\\{0, 1\\}$ obeying the BT model. Thus, $(a^1, a^0)\\sim \\mu_\\mathcal{D}$ can be interpreted as a mixture of $(\\pi^{\\mathrm{unchosen}}, \\pi^{\\mathrm{chosen}})$ and $(\\pi^{\\mathrm{chosen}}, \\pi^{\\mathrm{unchosen}})$. This mixture probability would not be too small as long as the quality of $(a^1, a^0)$ does not vary too much, i.e., both of them are possible to be chosen, which is the case in practice. As a result, in the offline data distribution $(a^1,a^0)\\sim \\mu_{\\mathcal{D}}$, both $a^1$ and $a^0$ partly comes from the chosen distribution $\\pi^{\\mathrm{chosen}}$.\n\nThen in order for $\\mu_\\mathcal{D}$ to cover the joint distribution of $(a^1, a^0)\\sim (\\pi, \\pi^{\\mathrm{base}})$, it suffices to argue that $\\pi^{\\mathrm{chosen}}$ can cover the target policy $\\pi$, which is then reduced back to the traditional coverage condition. Thus our assumption essentially requires that $\\pi^{\\mathrm{chosen}}$ well covers and only needs to cover the target policy $\\pi$. This coincides with the spirit of the minimal data assumption in offline RL theory, i.e., the so-called partial coverage condition.\n\nWe will make this clearer in the revision of our paper.\n\n**Q2: For online RL, based on the optimism principle, it seems that then the objective should be subtracted from the SFT loss, which obliviates the wish to avoid overoptimization. So how to balance the exploration and avoiding overoptimization for the online setting?**\n\n**A2:** Thank you for pointing this out! Actually online RLHF is a different theoretical setup than offline RLHF. The goal of *regret minimization* in online learning does not face the problem of overoptimization, because the data are not precollected but are collected and updated interactively. This in turn requires exploration and needs the algorithm to be optimistic. \n\nWhen our technique is applied to online RLHF, it does induce a similar SFT loss subtracted from the preference optimization loss, but the baseline policy $\\pi^{\\mathrm{base}}$ in the SFT loss needs to be chosen carefully and is not necessarily $\\pi^{\\mathrm{chosen}}$ as we considered in the offline setup. Possible candidates for the baseline policy could be the LLM at the previous iteration (serve as the reference policy for the current iteration). In this way, the data distribution of the actions (responses generated by the currently learned LLM) can be gradually shifted towards that of the optimal actions.\n\nThus, theoretically, we only need to subtract a properly designed SFT loss for online RLHF in terms of regret minimization. For practical situations where one might still need to handle the overoptimization issue after online data collection, we conjecture that the optimal way is to use optimism during the online data collection stage (subtract a properly designed SFT loss) and perform pessimism after all the data have been collected (add SFT loss as RPO). Still, online RLHF is beyond the scope of this paper. We leave this interesting question of addressing overoptimization in online RLHF to future work.\n\n**Q3: About the computational complexity and the implementations of the SFT loss gradient**\n\n**A3:** According to the paragraph **Practical implementation** in Section 6, RPO adds an additional SFT loss (the log probability of the chosen labels in the preference dataset) on the original DPO loss, where the SFT loss is a intermediate quantity in the calculation of DPO loss. Hence, our proposed method will not incur any additional computation overhead compared with the vanilla DPO. As for the justification of the approximation of the SFT loss, we use the linear property of the expectation to show that the population form of the RPO loss $\\mathcal{L}_ {\\text{RPO}}$ can be rewritten as\n$$\n\\mathcal{L}_ {\\text{RPO}}(\\theta) = \\mathbb{E}_ {(x,a_{\\text{cho}},a_{\\text{rej}})\\sim \\mu_ {\\mathcal{D}}}\\Bigl[-\\log \\pi_\\theta(a_ {\\text{cho}}\\mid x) +\\sigma \\bigl(\\hat r_ \\theta(x,a_ {\\text{cho}}) -  \\hat r_ \\theta(x,a_ {\\text{rej}})\\bigr)\\Bigr],\n$$\nwhere we denote by $\\hat r_\\theta(x,a)=\\beta\\cdot\\log(\\pi_\\theta(a\\mid x))/\\log(\\pi_{\\text{ref}}(a\\mid x))$ and denote $\\mu_{\\mathcal{D}}$ as the population distribution of the preference dataset $\\mathcal{D}$. It suggests that we only need to sample a mini-batch $\\mathcal{D}_ {\\text{mini}}$ from $\\mu_{\\mathcal{D}}$ (or equivalently sample from $\\mathcal{D}$) and calculate the gradient w.r.t. $\\theta$ on \n$$\n\\mathbb{E}_ {(x,a_{\\text{cho}},a_{\\text{rej}})\\sim {\\mathcal{D}_ {\\text{mini}}}}\\Bigl[-\\log \\pi_\\theta(a_ {\\text{cho}}\\mid x) +\\sigma \\bigl(\\hat r_ \\theta(x,a_ {\\text{cho}}) -  \\hat r_ \\theta(x,a_ {\\text{rej}})\\bigr)\\Bigr],\n$$\nwhich approximates the gradient $\\nabla_\\theta\\mathcal{L}_ {\\text{RPO}}(\\theta)$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j3zDlmLlja", "reviewer": "Reviewer_9VCJ", "summary": "The paper introduces the concept of RPO, which combines DPO loss with SFT loss. This approach aims to align the policy with human preferences while simultaneously imitating a baseline distribution, effectively mitigating overoptimization. Empirical results from experiments with LLMs demonstrate that RPO outperforms traditional DPO methods, showcasing the practical applicability of the proposed algorithm.", "questions": "It is noticed believed that math problems are not very suitable with vanilla DPO. Some practitioners found similar algorithms should help the performance. Have you checked the reasoning benchmarks and the proposed algorithm?", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper provides a robust theoretical framework for addressing overoptimization in RLHF. By identifying the source of misalignment as distributional shift and uncertainty, it offers a principled approach to the problem\n\nThe algorithm includes a reward penalty term to prevent the policy from exploiting spurious high proxy rewards, resulting in provable sample efficiency under partial coverage conditions\n\nThe paper provides empirical evidence demonstrating that RPO improves performance compared to DPO baselines in aligning LLMs. This practical validation strengthens the theoretical claims made in the study", "weaknesses": "The theoretical guarantees provided by the algorithm rely on specific conditions, such as partial coverage. These conditions might not always hold in practical scenarios, potentially limiting the generalizability of the results​.\n\nThe SFT loss + DPO seems very intuitive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The theoretical guarantees provided by the algorithm rely on specific conditions such as *partial coverage*, which might not always hold in practical scenarios, potentially limiting the generalizability of the results.**\n\n**A1:** Thanks for raising the question. We would like to comment that all the assumptions we imposed to obtain the theoretical guarantees are quite standard in the RL theory literature. Actually, our theory features a *minimal assumption* on the data distribution (the partial coverage assumption) thanks to our algorithm design. Similar kinds of data assumptions also appear in recent theoretical works on RLHF, e.g., [1, 2, 3].\n\nTo explain more, the partial coverage assumption (Assumption 5.2) only requires the dataset to cover the policy $\\pi$ to compete. As is shown by [4], the partial-coverage-style data assumption is the minimal assumption such that provably sample-efficient offline RL is possible. That being said, this assumption is actually a weak condition in terms of the data distribution, especially in comparison with the stronger notion of uniform coverage [5, 6, 7] where the offline dataset needs to cover all possible policies. Moreover, our theory works in the regime of general function approximation (instead of linear regimes [1, 3]), which also exhibits its generality.\n\nBut still, when going from theory to practice, the implementation of our algorithm RPO itself does not require the knowledge of these assumptions. It can be directly applied to handle overoptimization in RLHF for real-world problems. This is demonstrated by the effectiveness of RPO in LLM fine-tuning shown in the paper.\n\n**Q2: The proposed algorithm (SFT loss + DPO loss) seems very intuitive.**\n\n**A2:** Yes! the resulting algorithm does look very intuitive. This is actually an *advantage* of our algorithm design. Typically, to *provably* address overoptimization in offline RLHF with general function approximations, the theoretical algorithm would rely on solving complicated non-convex constrained optimizations over certain confidence regions [1, 2], which is prohibitive to scale to practice like LLMs without modifications or adaptations. \n\nIn contrast, our proposed theoretical object (Algorithm 1) naturally induces the simple but equivalent form of RPO (Algorithm 2) after delicate mathematical deductions (see Section 4). That is, it suffices to add an SFT loss to the preference optimization loss to implement the theoretical algorithm in an equivalent manner. Therefore, the simple form of RPO as well as the theoretical guarantees it enjoys is one of our main contributions. Also, our experimental results demonstrate its effectiveness despite its simple form. \n\nFinally, we notice that adding an SFT-style loss as a regularizer in RLHF object is becoming more and more popular, which has been adopted by the fine-tuning of Llama 3.1 [8] (see Section 4.1.4). Given that, our work also serves as a theoretical foundation for such an effective practice in large-scale RLHF.\n\n**Q3: About the reasoning benchmarks of the proposed algorithm**\n\n**A3:** Please refer to the **General Response**.\n\n**References:**\n\n[1] Zhu, B., Jordan, M., & Jiao, J. (2023, July). Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning (pp. 43037-43067). PMLR.\n\n[2] Zhan, Wenhao, et al. \"Provable Offline Preference-Based Reinforcement Learning.\" The Twelfth International Conference on Learning Representations.\n\n[3] Xiong, W., Dong, H., Ye, C., Wang, Z., Zhong, H., Ji, H., ... & Zhang, T. (2024). Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Conference on Machine Learning.\n\n\n[4] Jin, Ying, Zhuoran Yang, and Zhaoran Wang. \"Is pessimism provably efficient for offline rl?.\" International Conference on Machine Learning. PMLR, 2021.\n\n[5] Munos, Rémi. \"Error bounds for approximate policy iteration.\" ICML. Vol. 3. 2003.\n\n[6] Chen, Jinglin, and Nan Jiang. \"Information-theoretic considerations in batch reinforcement learning.\" International Conference on Machine Learning. PMLR, 2019.\n\n[7] Xie, Tengyang, and Nan Jiang. \"Q* approximation schemes for batch reinforcement learning: A theoretical comparison.\" Conference on Uncertainty in Artificial Intelligence. PMLR, 2020.\n\n[8] Dubey, Abhimanyu, et al. \"The Llama 3 Herd of Models.\" arXiv preprint arXiv:2407.21783 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mrn6yUFYHr", "reviewer": "Reviewer_1DUS", "summary": "This paper aims to address the issue of over-optimization in RLHF. The authors introduce a method named RPO which concurrently minimizes the maximum likelihood estimation of the loss alongside a reward penalty term. Not only do the authors demonstrate that the proposed method is sample-efficient, but they also outline a straightforward yet effective implementation strategy. Experimental results underscore the efficiency of the method.", "questions": "1.\tThe SFT loss appears similar to the commonly used PTX loss in [1]. Could you please elucidate their relationships and distinctions?\n\n2.\tHow is the true reward (human reward) depicted in Figure 1 on the left derived?\n\n3.\tHow does the proposed method address scenarios involving multiple reward models?\n\n[1] Stanford alpaca: An instruction-following llama model.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe research introduces a novel RLHF method specifically designed to tackle the issue of over-optimization.\n\n2.\tDespite its simplicity in implementation, the method proves to be highly effective.\n\n3.\tThe authors support their proposed method with thorough theoretical analysis, convincingly demonstrating that it benefits from finite-sample convergence guarantees.", "weaknesses": "1.\tAdditional experiments across a broader range of scenarios are required to more comprehensively demonstrate the method's efficiency.\n\n2.\tThe current experiments are limited to evaluations using GPT and log probability, which do not offer intuitive insights into the over-optimization problem. In other words, it remains unclear whether the observed improvements in performance truly indicate a mitigation of the over-optimization issue. A more detailed analysis, perhaps focusing on rewards, could provide the necessary clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Additional experiments are required to more comprehensively demonstrate the method's efficiency**\n\n**A1:** Please refer to the **General Response**.\n\n**Q2: The experiments are limited to evaluations using GPT and log probability. It remains unclear whether the observed improvements in performance truly indicate a mitigation of the over-optimization issue.**\n\n**A2:** Thanks for the question! First, we would like to point out that the actions and their chosen probabilities **can be** interpreted as a proxy of analyzing the underlying (estimated) reward model $\\widehat{r}$ [1] due to the representation $\\pi_{\\widehat{r}}(a|x)\\propto\\pi^{\\mathrm{ref}}(a|x)\\exp(\\beta^{-1}\\widehat{r}(x,a))$. Analyzing the (log) probabilities of the actions can be used to detect the mitigation of over-optimization, because according to the representation, an overestimated reward of a poor action would result in a higher probablity of choosing this action, and would also cause a decay in the probability of choosing other better actions (since the probabilities are normalized to $1$).\n\nTo further showcase the ability of RPO to address overoptimization (through the lense of probability), consider the following theoretical example with only actions [2] where we can track everything clearly. It has three actions $\\{a, b, c\\}$ with $R^\\star(a) = 1, R^\\star(b)=0.5,R^\\star(c)=0$. The reference policy $\\pi^{\\mathrm{ref}}(a)=\\pi^{\\mathrm{ref}}(b)=0.4, \\pi^{\\mathrm{ref}}(c)=0.1$, and the dataset consists of one data point $\\mathcal{D} = (a,b,1)$ (meaning action $a$ is preferred in the data). Then an ideally solved DPO objective would be $\\pi_{\\mathrm{DPO}}$ as long as $\\pi^{\\mathrm{DPO}}(b)=0$, and the value of $\\pi^{\\mathrm{DPO}}(a)$ can be arbitrarily chosen in $[0,1]$. Thus a possible solution to DPO would be $\\pi^{\\mathrm{DPO}}(a)=0.5,\\pi^{\\mathrm{DPO}}(b)=0$, and by the normalizing condition $\\pi^{\\mathrm{DPO}}(c)=0.5$, which is undesirable since the action $c$ has reward $R^{\\star}(c)=0$. In contrast, solving the RPO objective would additionally require the maximization of $\\pi_{\\mathrm{RPO}}(a)$ due to the SFT regularization term, and thus the solution is shifted towards $\\pi_{\\mathrm{RPO}}(a)=1, \\pi_{\\mathrm{RPO}}(b)=\\pi_{\\mathrm{RPO}}(c)=0$, which is better than the DPO policy. Thus, RPO is able to prevent overoptimization towards poor actions that are less covered by the dataset (action $c$ here), therefore resulting in a better policy.\n\n**Q3: Could you please elucidate the relationships and distinctions between PTX loss in [2] and your SFT loss?**\n\n**A3:** Thanks for the question! The original PTX loss is an imitation loss calculated on the pretraining data. In contrast, the SFT loss in the RPO objective is an imitation loss calculated on the RLHF dataset. In more specific, our experiments use this SFT loss to imitate the chosen responses in the RLHF dataset. Thus the relationship is that they are both imitation loss which aims to mimic certain data distribution. The distinction is that they are calculated on different data sources. Moreover, the SFT loss in the RPO objective naturally comes from our theoretical algorithm and provably serves as an important regularization term to mitigate overoptimization in offline RLHF. We would make this comparison clearer in the revision.\n\n**Q4: How is the true reward (human reward) depicted in Figure 1 on the left derived?**\n\n**A4:** Thanks for pointing this out! Figure 1 (left) is an illustrative figure to showcase the mechanism behind overoptimization as a consequence of distributional shifts. The rewards therein does not correspond to actual human rewards and is plotted for illustrative purposes. Meanwhile, as we demonstrated in the answer of **Q2**, RPO can effectively address overoptimization depicted in Figure 1 (left) where data coverage is not sufficient. We will make this clearer in the revision.\n\n**Q5: How does the proposed method address scenarios with multiple reward models?**\n\n**A5:** Thanks for raising this interesting question! How our methods can address the overoptimization issue in this scenario depends on the specific learning target in the face of multiple objects, e.g., [4, 5, 6] and more references therein.\n\nFor instance, when the goal is to find the optimal policy that maximizes a linear scalarized reward model [4, 6], the idea of RPO suggests to use the linearization of the multiple reward models to be learned as the regularizer in the object (3.2), which roughly give the object \n$$\\max_{\\pi}\\min_{r^1\\in\\mathcal{R}^1,\\cdots,r^m\\in\\mathcal{R}^m}\\eta\\mathbb{E}_{a^1\\sim\\pi,a^0\\sim\\pi^{\\mathrm{base}}}[\\mathbf{w}^\\top\\mathbf{r}(x,a^1)-\\mathbf{w}^\\top\\mathbf{r}(x,a^0)]-\\beta\\mathrm{KL}(\\pi\\|\\|\\pi^{\\mathrm{ref}}) + \\mathcal{L}\\_{\\mathcal{D}^1,\\cdots,\\mathcal{D}^m}(\\mathbf{r}).\n$$ \nInspired by our theory, such an algorithm can find the optimal policy as long as each of the responses in the data can cover the target policy in terms of the linearized reward (see Assumption 5.2, where the test function $r$ is replaced by the scalarization of the multiple rewards), thus overcoming the issue of overoptimization in offline RLHF. We leave the study of RPO for such multiple reward models as our future work. \n\n**References:**\n\n[1] Rafailov, Rafael, et al. \"Direct preference optimization: Your language model is secretly a reward model.\" NeurIPS 36 (2024).\n\n[2] Xu, Shusheng, et al. \"Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study.\" 41th ICML.\n\n[3] Stanford alpaca: An instruction-following llama model.\n\n[4] Zhou, Zhanhui, et al. \"Beyond one-preference-for-all: Multi-objective direct preference optimization.\" ArXiv:2310.03708 (2023).\n\n[5] Chakraborty, Souradip, et al. \"MaxMin-RLHF: Towards equitable alignment of large language models with diverse human preferences.\" ArXiv:2402.08925 (2024).\n\n[6] Yang, Rui, et al. \"Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment.\" 41th ICML.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**General Response:**

We thank all the reviewers for their time and effort reviewing our paper and we appreciate all your support of our work! We have responded to each of you detailedly. 

Here we provide a general response to **Q3** of **Reviewer 9VCJ** and **Q1** of **Reviewer 1DUS** about more experimental evaluations of the proposed algorithm RPO. To this end, we use extra benchmarks on the math, reasoning, and coding tasks to showcase the effectiveness of our method. Please refer to the PDF document attached to this response for detailed results. Thank you!

Meta-review: The paper provides an interesting method to mitigate overoptimization in LLMs. The reviewers all agree with the contribution and novelty of the results. I recommend acceptance.
Decision: Accept (poster)