Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms
Authors: Rafael Rafailov
Yaswanth Chittepu
Ryan Park
Harshit Sikchi
Joey Hejna
W. Bradley Knox
Chelsea Finn
Scott Niekum
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning From Human Feedback
Direct Preference Optimization
Reward Hacking
Primary area: natural_language_processing
TL;DR: We define and explore the reward over-optimization phenomenon in direct alignment algorithms, such as DPO

Abstract: Reinforcement Learning from Human Feedback (RLHF)has been crucial to the recent success of Large Language Models (LLMs), however it is often a complex and brittle process. In the classical RLHF framework, a reward model is first trained to represent human preferences, which is in turn used by an online reinforcement learning (RL) algorithm to optimized the LLM. A prominent issue with such methods is reward over-optimization or reward hacking, where the performance as measured by the learned proxy reward model increases, but the true model quality plateaus or even deteriorates. Direct Alignment Algorithms (DDAs), such as Direct Preference Optimization (DPO) have emerged as alternatives to the classical RLHF pipeline. However, despite not training a separate proxy reward model or using RL, they still commonly deteriorate from over-optimization. While the so-called reward hacking phenomenon is not well-defined for DAAs, we still uncover similar trends: at higher KL-budgets, DAA algorithms exhibit similar degradation patters to their classic RLHF counterparts. In particular, we find that DAA methods deteriorate not only across a wide range of KL-budgets, but also often before even a single epoch of the dataset is completed. Through extensive empirical experimentation this work formulates the reward over-optimization or hacking problem for DAAs and explores its consequences across objectives, training regimes, and model scales.

Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms
Rafael Rafailov" Stanford University
Yaswanth Chittepu"
Ryan Park' Stanford University
UMass Amherst
rafailov8cs. stanford.edd
ychittepuQumass.cm edu
ryparkOstanford.cm
Harshit Sikchi'
Joey Hejna Stanford University
W. Bradley Knox UT Austin bradknoxccs utexas.ecoom edu
UT Austin hsikchi@utexas.
jhe stanfccsstaaorr...o edu
Chelsea Finn
Scott Niekum
Stanford University
UMass Amherst
cbf bafinndca.co stanford edu
aniekum@cs. umass.
Abstract
Reinforcement Learning from Human Feedback (RLHF) has been crucial to the recent success of Large Language Models (LLMs), however, it is often complex and brittle process. In the classical RLHF framework, reward model is first trained represent human preferences, which is in turn used by online reinforcement learning (RL) algorithm to optimize the LLM. A prominent issue with such meth- ods is reward over-optimization reward hacking, where performance measured by the learned proxy reward model increases, but true quality platcaus or even dete- riorates. Direct Alignment Algorithms (DAAs) like Direct Preference Optimization have emerged as alternatives to the classical RLHF pipeline by circumventing the reward modeling phase. However, although DAAS do not use a separate proxy reward model, they still commonly deteriorate from over-optimization. While the so-called reward hacking phenomenon is not well-defined for DAAs, we still uncover similar trends: at higher KL budgets, DAA algorithms exhibit similar degradation patterns to their classic RLHF counterparts. In particular, we find that DAA methods deteriorate not only across wide range of KI. budgets but also often before even a single epoch of the dataset is completed. Through extensive empirical experimentation, this work formulates and formalizes the reward over-optimization or hacking problem for DAAs and explores its consequences across objectives, training regimes, and model scales.
Introduction
Recent advancements Large Language Models (LLMs) have broadened their capabilities signifi- cantly, enabling applications in code generation, mathematical reasoning, tool use, and interactive communication. These improvements have popularized LLMs across various domains. Reinforce- ment Learning from Human Feedback (RLHF) has been instrumental in these advances and is now integral to sophisticated LLM training regimes [00 (55]. Before alignment, LLMs, trained on text corpses to predict subsequent tokens
45 8 are often unwieldy and hard to use. Today, leading LLMs
incorporate variants of the RLHF framework 11
69 36 to align them with human intent, which
'Equal Contribution, Dice Rolling
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

generally involves a multi-stage process. Specifically, users evaluate model responses to assorted prompts order to train reward model that encapsulates human preferences [TO, 55]
72 S 62
Then, the refined L.M maximizes the expected learned reward function using reinforcement learn- ing (RL. algorithm 50 [6]. Despite its efficacy, this procedure is complex and computationally intensive, particularly in its latter stages.
Goodhart's is Law [2 that "when a measure becomes target, it ceases to be a good measure", has often been cited as a core shortcoming of RLHF, Standard RLHF methods optimize a learned, but imperfect reward function which ends up amplifying the reward model's shortcomings. Empirically. this phenomenon was first extensively characterized by Gao et al. [2I], who coined the term "reward over-optimization" and has been seen consistently in recent findings [62] [6 14). While reward over-optimization has been studied in the context of the aforementioned RLHF procedure, recent contemporary methods for aligning L.LMs circumvent the reward learning procedure, necessitating a new characterization of the over-optimization phenomena.
This new broad class of algorithms, which we refer as Direct Alignment Algorithms (DAAs), bypass the traditional RLHF pipeline by re-parameterizing the reward model directly through the optimal policy derived during the reinforcement learning phase. DAA methods, like Direct Preference Optimization [46]. have gained popularity 14 [8 as they often reduce computational demands. Yet, despite not fitting a reward function, DAAS still exhibit over-optimization trends similar to those of traditional RLHF methods using learned reward function. In some sense, this is puzzling: DAAs can be viewed as simply learning a reward function with supervised learning from which the optimal policy is deterministically mapped, however more seems to be at play than simple supervised learning.
In this work, we investigate the over-fitting phenomena present in DAA algorithms through extensive experimentation. First, we unify a number of different recent methods [46] (68, 4 under the DAA framework. Then, across different model scales and hyper-parametee we show that DAAS exhibit type of reward over-optimization consistent with that previously observed in RLHF (21). Specifically, we find that different KL-divergence budgets DAAs exhibit degradation patterns similar those found in RLHF. Interestingly. we also find that performance within single epoch is not always as consistent as expected for DAAs. Finally, we explain why this happens by appealing to the constroontt nature of the optimization problem used in DAAs.
2 Preliminaries
In this section, we first outline the core components of the standard RLHF pipeline 72 55 Then, we examine prior literature to characterize the reward over-optimization exhibited by standard RLHF methods. Finally, we provide unifying view of direct alignment algorithms (DAAs) which will guide our analysis of their training dynamics in the next section.
2.1 Reinforcement Learning From Human Feedback
The standard RLHF pipeline consists of three distinct stages with the goal of aligning the LLM with human preferences.
Supervised Fine Tuning (SFT): First, dataset of prompts and high-qualit, answers y are used to train an LLM for instruction following via maximum likelihood estimation over next-tokens. We refer to the resultant model as Tser(y]z)) and consider the entire prompt and answer strings to be single variables.
Second, the SFT model (FF(()|=) is used to learn reward function over human preferences. Specifically, the SFT model queried to produce pairs of answers U1:3/) ~ TSFT x)). for every prompt æ in dataset. Then, users select their preferred answers, resulting in ranking You > yt x where you and yı are the preferred and dispreferred answers respectively. Typically, user rankings are assumed to be distributed according to the Bradley-Terry (BT) model 7
Reward Modeling:
exp (r(#, v1))
ply1 U2 x))
=r(( vn) r(x. U2))
(1)
exp (r(z. y1)) + exp (r(z,
where the preference distribution p results from an unobserved latent reward r(x, y), and is the
logistic function. Given this model and dataset of rankings, denoted D (z((), (1) the vi 1
(8) (6) N

---

can train a parameterized model rai (x, y) to predict the unobserved reward using maximum likelihood estimation. This yields the following loss function.
Lew (Ta) (a, w:)) [log (T (io) (z m))]
Reinforcement Learning (RL): The final stage of the standard RLHF pipeline uses the learned reward model r (2, y) to update the LLM Th with an on-policy RL algorithm like PPO [50,, optimizing the model provide responses more preferred by human raters. The most common objective
max RewDD. (2, v)] xL [T [ Th
a:) || [<r[(]l=)]]
which enforces a Baillback-Leible (KL) divergence penalty with a reference distribution (yr(y)a) (usually taken to be T (v(z)) to prevent the LLM Ta from straying too far from its initialization. Thus, the hyper-parameter / directly trades off exploiting the reward function and deviating from [e)()
2.2 Reward Exploitation in RLHF
Unfortunately, repeating the above procedure without careful tuning of the RL phase can lead to disastrous performance. This is because in the context of RLHF the LLM policy is optimizing the surrogate reward estimate To y) and not the true reward function as often the case in other domains. Thus, prior works have observed that while the LLM's expected reward according to
eq. 3 increases the actual quality of the model's outputs can decrease [54 43 34
This particular
instantiation of the reward exploitation or hacking problem 3 is often referred to as reward "over-
optimization" RLHF literature and has been studied empirically both controlled experiments
21 and user studies [14] There are two prevailing explanations for why this phenomenon occurs.
OOD Robustness: In the classical RLHF pipeline, the RL objective (eq. (3) optimized using on-policy samples from тæ. This means that the reward function is continuously queried using unseen model samples which are potentially oul-of-distribution. Beyond the support of the reward modeling distribution, T may assign high rewards to sub- par responses, leading the policy to believe it is doing well when it may not be. While the KL-regularization term is designed prevent the model from drift- ing too far out of distribution, this term alone has proven inadequate to prevent reward hacking [21].
2. Reward Mis-specification. Learned reward functions may exhibit spurious correlations that cause them to prefer unintended behaviors. While this issue not at the forefront of LLM research, is known to be pervasive in RL [3] 34). Most efforts to address these problems exist at the intersection and use measures of epistemic uncertainty to 16
of robustness and offline RL literature
[13
67
penalize the predicted reward.
2.3 Direct Alignment Algorithms
Due to its complex multi-step nature, recent works have sought alternatives to the classic RLHF pipeline. A new class of algorithms, which we broadly classify as Direct Alignment Algorithms (DAAs), directly update the LLM's policy Ty using user feedback instead of fitting a reward function to and then employing an RL algorithm. Perhaps the most known example is Direct Preference Opti- mization (DPO). DPO, as well as other DAAS, are derived using the closed form solution to the RLHF objective in eq. 771. x (y(zz) & Tret (z)la)) ((r.w))) where r(x, y) is the ground-trune truth reward. By isolating r(z, y) in this relationship and substituting it into the reward optimization objective in eq. (2), we arrive at a general objective that allows us to train the LLM directly using feedback data:
LDA (mTH Tref) (i (g 33 log
To (yhe [x)
no (y1
x))
log
Tref (yw
x))
Tref (M xr)
where g is convex loss function. Using g(x) Bradley-Terry model and the original DPO objective. Other methods choose different loss functions: IPO 4 uses the quadratic objective g(z) = (x 1)² and SLiC-HF [68] 38 uses the hinge loss
- log (x) coincides with the standard
g(z) max(0, 1 - æ). Additional objectives were also considered in (59 but due to limited computational resources, focus the three objectives outlined above.
Crucially, the DAA approach allows us to recover the optimal policy using a straightforward classifi- cation loss without the need for learning reward function, on-policy sampling, or RL, which can be
3

---

B13 (endof-eaach cnedlpdrral
P0 scolin curve to Jend stopect checlporti
SUC Scalnng lan corve fiys land olippecs checkis rtal
2:00
639
GFTY alrad
0075
8
01-
03
04
Squarc 1221 Shargenas
Squar nott shorgass
0.5
(OPT4 ente
100
root
Square shergence
Figure 1: Results on over-optimization in Direct Alignment Algorithms for DPO, IPO and SLiC. Results show model win-rates over the dataset summary on an evaluation set of prompts as judged by GPT-4. The top row shows the final performance after epoch of training, while the second row also includes 4 intermediate checkpoints as well. The fitted dotted curves utilize scaling laws from [2T) applied to direct alignment, with GPT4 winrates taking the place of the gold reward model score.
notoriously difficult to tune and computationally expensive. Because of this, DAAs have emerged as popular alternative. However, just like classical RLHF methods, DAAs exhibit strong over-fitting and even reward-hacking like behaviors. For example, Park et al. [44] show that LLMs trained with DPO generate responses with increasing length throughout the course of training, but do not improve in ground-truth win-rate after a certain point. Since DAAs do not explicitly learn a reward function, is unclear how treward-overoptinzzzie light on this phenomenon in DAAs.
fits into the picture. In this work, aim shed some
Empirical Analysis of Overoptimization in DAAS
First, we examine the over-optimization problem in DAAs and compare it to those observed in tradi- tional RLHF methods. All our experiments carried out using the Reddit TL:DR summarization dataset (55) and the Pythia family of Large Language Models [6]. Additional plots illustrating similar over-optimization trends for Direct Alignment Algorithms on the Gemma2-26 model [61] and the Anthropic
Harmlessness dataset S are provided in Appendix F
3.1 Evaluating Modee-Overoopimmzzaion
In our first set of experiments, we evaluate the reward model over-optimization phenomenon. We evaluate three training objectives DPO, IPO, and SLiC using seven parameters, representing different KI. budgets at three model sizes 1B, 2.8B, and 6.9B. Our main results are shown in Fig. which presents results for different configurations after epoch of training (row 1) and including uniform intermediate checkpoints (row 2). We include additional results the training dynamics in Fig. [ which shows win rates and KL bounds for intra-epoch training. We present our findings below.
Model OvverOOtimizzttion We see clear over-optimization for all objectives as performance exhibits a hump-shaped pattern, where an additional increase in the KL budget leads to decreasing model performance. Moreover in Fig. 2 we observe similar intra-epoch training dynamics patterns as configurations with wider KI. budgets achieve their best performance after training on only 25% of the data, after which performance starts decreasing conjunction with increasing KI. divergence metrics.

---

Werrat ruciation ove epeen for SPO 648
IN rrate endittion ston epach Sy U.K.
Winrasa eveluticn ovr epech for IP 1.40
SPTE-RISE
pro (3-00075)
SAC BUC---BB ELT
F0 1013-a 31351
DPo 300
SAR GERS)
P0 10 2734
14.0 200.11
P0 13
GrOlepod tectin
Tacion
End reztion
OPO G-0001
SUK-3-2-00 23---
-8" UnO &. GUD
3:5-13 2851
DRO
(swcion)
50
DRD
12-222
12
0.5
Figure 2: Results intra-epoch optimization dynamics. The top row shows win-rates against the fraction of an epoch so far, while the bottom row shows the corresponding KL values. Under a lower KL constraint, most experiments reach their best performance degrade with additional training, while the model deviates from the reference under increasing KL. All models 6.9B and vary across DPO, SLiC, and IPO loss formulations.
the first 25% of the epoch and
Effect of Training Objective: In the IPO work 4 the authors present theoretical arguments that due to the monotone sigmoid objective in the DPO formulation. the KL constraint is not effectively enforced and propose the quadratic fixed-margin loss as an alternative. Across all objectives, there are clear dependencies between the , parameter and the corresponding KL achieved at the end of training. While DPO and SLiC exhibit similar performance, IPO indeed seems to less prone to over-optimization and in general, achieve lower KLs under the same constraint. Our observations with IPO also align with prior works in preference-bante RL and imitation learning where imposing a fixed margin led to more stable and performant methods [48] 51
Effect of Model Size: The results also show strong parameter count scaling effect. The Pythia 1B model achieves low performance under the same set of constraints it reaches much higher KL values, while almost immediately exhibiting signs of ver-optimization. This behavior holds under all three objectives. At larger scales, the 6.9B Pythia model tends to exhibit more win-rate KL trade-offs and be less prone over-optimization, with both models significantly outperforming the IB model. In the case of the IPO objective, the 6.9B also exhibits significantly better control over the KL objective and shows little no optimiitinnotion behavior.
3.2 Scaling Law Fits
Given we have established framework for evaluating over-optimization in DAAs and empirically validated it (section 3.T), we now develop scaling laws for this phenomenon. Previous work in classical RLHF has established such scaling laws for reward model scores function of the KL.
divergence between initial and optimized policies (1 The relevant functional of the reward R(d) is
R(d) d(a B log
(5)
where o, B are constants dependent on the size of the reward model dataset and parameter count, and d = DDxL (Ter). As DAAs do not train a proxy reward model, we treat GPT4 winrates over dataset completions as proxy for gold reward. Somewhat surprisingly, we find that this scaling law accurately relates d and winrates for DAAs. Compared to quadratic fit between DRr. (r]] (et) and winrates, this scaling law halves the RMSE. worth noting, however, that a quadratic between d and winrates yields similar error compared to Equation
5

---

RL-iieurs curve for unith eegulloeeeeeee lariedd and stardarrs DPQ 12 801
length
Lorgt whee DPQ x 0.61
6.35
2,85
T..mecc meserd and samplee desddle-eggg
6:98
0.25
x
8.15
x
32
0:05
de Sivrgarce
Figure 3: Left: KL budget versus win-rates (over dataset human answer) with and without length- regularization [4]. While including a length correction in the optimization objective changes the KL--win-rate Pareto Frontier, it does not alleviate reward over-optimization and might even exacerbate it. Right: Scaling behavior for length extrapolation smaller capacity models (either by size or KI.
budget) extrapolate more strongly on simpler features such as length.
3.3 Length Correlations
Prior work 44] has shown that the DPO algorithm is prone to length exploitation as it amplifies verbosity biases in preference datasets. Here we show that length is not the only dimension on which exploitation can occur. Our experimental results are shown in Fig. 3 On the left, we show results for the 2.8B Pythia model with standard training plus the length-eegllarization approach from [4 Both approaches suffer from vver-optimization, but the dynamics differ depending on the KL budget. Moreover, even though the regularized model achieves higher win rates on length-correct correct basis, it under-performs the model trained with the standard objective in the lower KL constraint region.
Recent work 27 has also shown that DAAS prioritize features of the data based their complexity and prevalence (with length a clear example of human datasets). 44 further showed that models trained with the DPO algorithm extrapolate significantly based on length. We extend this analysis in Fig, 3 (right). We consider a linear regression of the form
log To (z(((0))((1) Tees (y(t))
5/ (1 + e0
where ()) are held-out prompts and (+) are samples from the corresponding model between the DPO implicit reward and length. We fit a different regression for each model size and checkpoint and plot the corresponding R² values. We observe two main effects; first, there a clear scaling law behavior. Weaker models extrapolate across the simple length feature to much higher degree than stronger ones. This is especially clear when comparing the behavior of the Pythia IB versus the 2.8B and 6.9B models. Second, we significant effects based on the KL budget- under smaller budget all model sizes exhibit higher extrapolation behavior. Based on these results we formulate the hypothesis that under limited capacity, either from model capability or limited KL budgets, the model will extrapolate more strongly based on simpler features, which can lead to OOD issues.
3.4 Reward Metries Correlations
Prior works have measured reward model quality in ranking settings by classification accuracy. We evaluate the relationship between the DAA implicit reward model accuracy and policy performance in Figure 4 The DPO and SLiC algorithms exhibit little to no correlation between reward model accuracy and downstream model performance. The IPO model shows a weak positive relationship. but upon further examination, this is entirely due to model size scaling stronger models both fit the data better and produce better generations as well, however within each particular model size, there is no discernible relationship between the DAA implicit reward accuracy and the actual policy performance. Similar observations hold when comparing the empirical DAA loss with model performance, which is contrary to observations in supervised pre-training and instruction tuning [30].

---

DPO whrates 22003 of checoorts
FO rretes ECOS chockpoers
SUC winrates ECOOS chectponts
04
Depiiit Paait---100 163
GFF airaaa
OPFE 4T000
Be EL4)
2
03-
08
1.555 6.00
8,05
6.00
145
025
0.5
0.5
030
044
6.70
Exst accuraccee
chectpoitts
Mede 2013
Mede 2013
C374 1 Trats
CPTT GF
COPTL
02
8.2
lett
Fuol
Figure 4: Top: We plot the DAA implicit reward accuracy in preference classification versus win
rates. Bottom: DAA optimization loss versus checkpoint win rates. Model training statistics, do not exhibit a strong relationship with downstream performance.
3.5 Decreasing Likelihoods and Model Performance
A number of recent works have observed that the implicit DAA rewards of both preferred and dis-preferred responses decrease during training, which may be counter-intuitive. In [47] the authors make counter-point that in offine training of DAAS Wee is usually pre-trained with SFT on the preferred response and thus
Te (Bar (x)) log
log Te FB kz) Rd EE (x
Epp (Murl))
= DKi. [T)r(()):))
ma (v [x]]
(7)
Tret (ye [x:]]
Tret (Var [x:]
where pD (y)))) is the dataset distribution of preferred answers. That is the expected implicit reward represents a forward KI. divergence between the reference policy and the optimization policy, thus it expected to be negative and decrease with training the optimization model moves away from the reference. In this section, we study whether this empirical phenomenon presents a challenge for (negative)
DAA learning. Similar Fig. we plot the win rates against the snnure-roo--trasffrmme expected implicit reward of the preferred response (evaluated on a held-out evaluation dataset), which as stated above approximates the
forward KL IKK1, [Teeryy) (x:) || my (y z). Results are included in Fig. [5] which follow closely the pattern in Fig. [ with performance initially increasing before starts dipping down after a certain threshold. This indicates that under the standard DAA training pipeline decreasing likelihoods are not necessarily issue for performance, and are even necessary for improvement, but exhibit non-linear over-optimization dynamics.
4 Reward Exploitation in Direct Alignment Algorithms
While the phenomena observed in the previous section echo those observed in classical RLHF, their underlying causes may be distinct. Reward over-optimization in classical RLHF is largely attributed to querying a proxy reward function that is potentially OOD, while DAAS do not train a separate reward model. Instead, DAAS generally understood fitting "implicit" reward model to preference
data with the parameterization ro( v) 3 log (artsla))
me x) using the objective in eq. ( Therefore, the
OOD behavior of the policy is inextricably linked to the OOD behavior of the implicit reward model. We demonstrate below that the reward modeling objective used is heavily coner-constrained allowing for potentially large number of solutions that can place weight on OOD responses. This is especially problematic for DAAs which deterministically map the optimal policy from the "implicit" reward.
Rank Deficiency with Finite Preferences. In DAAs, the language modeling problem treated as contextual bandit. However, the space of possible prompts X' and answers y y are both
7

---

DPO scaling low carre ind of deaeh chedipo rta)
PO scolrp curve tos Jend Stopect checlporti
SUC Scaling lan curve fry lond- oriepecy checkss rial
2.00
230
144
230
03
0.4
04
Squars Iornar chepponc
of lrear SL deaiggttc
Squert limmere shapprs
ES Moncraie
(PPT oPtceeette
foraad chorgence
Square rost fowward duargence
Forwars chaeggence
Figure 5: Over-optimization results for / Forward KL vs. winrates. The top row shows the final performance after epoch of training, while the second row also includes 4 intermediate checkpoints. The fitted dotted curves are scaling laws from 21 applied to DAAS, with GPT4 winrates taking the place of the gold reward model score.
exponentially large in sequence length. However, as highlighted by Tang et al. 559. DA often assume full support of the reference distribution when mapping from the implicit reward to the optimal policy ı by eq. (10). However, in practice such coverage is impossible. Instead, preference datasets cover a minuscule portion of the prompt-response space. Unfortunately, as DAA objectives are not strictly convex, their loss functions (eq. (4) can have multiple global optimas, which may be undesirable. We demonstrate this below, using the regression interpretation from Hejna et al. (23
First, we re-write the DAA objective from eq. 44) using vectors in the prompt-response space X x V. Each preference query in the comparison dataset can be written as difference between indicator
(+) }}. This "query" vector simply selects the comparison from the prompt response space, with the entree corresponding to (x, y²) being +1 and the entree corresponding to (x, 3/) being -1. Similarly, we can consider the learned policy to be vector log log Teet X x V, to which the distributional constraint also applics in practice. Our generalized DAA loss function can then be re- re-written as
vectors, specifically g: = (((x,,) v) = (x) _ )} - [( y) = (xx(1,, yi
1 if (a. = (z(0) yhe
LDAa(Fo, D) = 212 (3q7 (log (t) m) log ==)()))))) where 2d2,, vl =
1 if (a. = (z(1), (0))
0 otherwise
with finite data. Choosing g to be the negative log sigmoid above recovers DPO with finite preferences, but also logistic regression with a data matrix Q of shape [D] by X x J constructed by stacking the aforementioned query vectors g. As XX x J'l >> (D), this matrix is likely to have non-trivial null space, making the problem not strictly convex. Thus, there are many possible policies π that can achieve the same optima, some of which will place high weight nut-of-distributie responses due to the distributional constraint of policy 23 [0]. To demonstrate this, we formalize the construction below.
Proposition 1 (Adapted from Hejna et al. (23)) Let S be the set of win-or-lose prompt-response vectors (I, v) in D. Provided:
1. The intersection of the null space of Q, (Q),, and the span of S. span(S), is non-trivial.
2. For every x there exists response GOod E y that is not in the data, (x, Noon) + S.
Then, there are infinite number of minima to eq. which place weight on dus---i-ttt responses y.
8

---

OOD Trijsstoriss:OO--bctaa0
OOD Trajsctorics-/PO--ctte.0..
OOD TralcctoricsSSUkkctttoo
Pre Probability
In Distribution Trujectories-DPo DPO.bela--O-eee 0.1
In Distribution Trajectoies--P--btta-.0
In Distribution Trajectories-UUC-betae
mote 13, 1.D.300
roleed St, B.-4
Probability
Figure 7: (Top row) Probability of OOD trajectories. DAA algorithms end up placing a substantial probability mass of some of the OOD trajectories during training. (Bottom row) Probability of in-distribution (preference-pair) trajectories decreases during training.
Proof. Let ने be the minima of the DAA loss fúnction. Choose vector such that N().. E Span(S), and u has at least one negative component. Modifying the log policy vector as log 9 + will not affect the DAA loss, as in the null space of Q. but the log- log-probability of the policy will decrease for least one prompt-response pair in S by construction. However, e e ++v may not integrate to one. To fix this, we can construct second vector v E N()) using the yood at each such that elug uut+ integrates one. For more details, we refer the reader to Hejna et al. (23 Appendix A.3.
The second constraint of proposition D is often trivially satisfied by the dimension of the response space as we are unlikely to see every response to a prompt. The first constraint harder, but can be satisfied by conflicting preferences. A trivial example which satisfies these constraints is simple MDP in which there is only single state (or prompt œ), but three possible actions (or responses), y1, V2, and ya- If construct the preference dataset DD {(yn > y2), (y2 > y))}, omitting for brevity, then we satisfy the above conditions: the null space of Q is non trivial in span of y1 and 1/2 and there is out-of-distee out-of-distributtion action V3- In this setting, the DPO loss is minimized by both # (v/[z) = (0.5, 0.5, 0) and f(y[=) = (0.0, 0.0, 1.0). In fact, it is minimized by infinitely many policies which place equal weight on yı and y2- To demonstrate this effect in higher dimensions across a number of different DAA methods, we conduct experiments in a Toy MDP which bears resemblance the language modeling setting.
Understanding OOD behavior for DAA algorithms with a Toy MDP: To illustrate that DAA algorithms, in general and not an artifact of training LLM's, end up placing probability mass OOD sequences during training we design a simple Tree MDP (shown in Figure ( to mimic the token-level MDP in LLMs. We and follow the standard procedure of running SFT on preferred responses before updating an RNN policy using DAA. Figure 7 shows that even this simple setup, popular DAAS (DPO/IPO/SLiC) end up extrapolating incorrectly out of distribution revealing a fundamental shortcoming. Unlike in standard RLHF, the non-strict convexity of the reward function in DAAs ends up directly affecting the policy. Detailed experimental details can be found in Appendix E
dataset containing single preference between two trajectories
5 Related Work
Broadly, over-optimization has been widely studied phenomenon across different settings
Over-fitting can be characterized as over-optimization in the supervised learning setting 39
32
which can harm generalization [19 12 24 or lead to susceptibility to adversarial attacks [56]
Reward hacking reinforcement learning (RL) [54), where an agent maximizes its reward through

---

behavior that deviates from the intended goal, can be viewed as different type of over-optimization, commonly observed in prior work 43 22
We study over-optimization in the context of aligning L.MMs with human feedback, for which the most common approach is RLHF as outlined in section 2.1] Similar RLHF techniques were originally pioneered for control 31 2 10). Standard RLHF methods suffer from both potential over-fitting of the reward function and reward exploitation by the RL algorithm. Several works have considered how to reduce over-fitting increase the robustness of learned reward functions using ensembles [33 67 16 or data smoothing 770. Other approaches,
01
92 86
consider how reward exploitation can be reduced by using different optimization techniques in the RL stage. Much of this work motivated by Gao et al. [21, which first characterized and provided scaling laws for over-optimization RLHF.
like Moskovitz et al. 40
Figure 6: An illustration of the Tree MDP. At each state, we can choose one of actions (4g, d1 a2). which deterministically maps the next state. Fur- thermore, all the leaf nodes in this tree MDP, transi- tion to the terminal absorbing state Seo, irrespective of the chosen action
Unlike Gao et al. 21 we consider the over- optimization problem in DAAs, which differs significantly from the standard RLHF pipeline. Different DA have been derived theoretically 47 46
68
4 64], and applied problems beyond language modeling like image generation these scenarios, over-optimization problems have persisted. Park et
63
and control 2 In show that DAAs commonly over-fit to length and the expense of performance, which has been linked to inherent bias in training data [53] 29 Other works have tried to allow DAAs to use more types of data like demonstrations [49 ratings [17] to get better performance. Recently, incorporating online data has proven critical to improving performance [66 (6 [7]. Concurrent to our work, Tang al. (58) study the differences between offline DAAS and standard RLHF methods. Unlike us, they focus on comparisons with online sampling whereas we focus on the purely offline setting.
Conclusion
In this work, we present analysis of the over-optimization problem in Direct Alignment Algorithms. Through extensive experimentation on different algorithms (DPO, IPO, SLIC) and at different model scales (1B, 2.8B, 6.9B), we observe consistent over-optimization trends different KL-divergence budgets. While our analysis is first step, it is not complete picture of understanding the over- optimization phenomena. More work can be done characterizing this effect at larger model scales, which we were unable to do due to computational limitations. Nevertheless, we believe our work sheds light on important problems in Direct Alignment Algorithms that can spur future research.
Acknowledgments
This work has taken place in part in the Safe, Correct, and Aligned Learning and Robotics Lab (SCALAR) at The University of Massachusetts Amherst. SCALAR research is supported in part by the NSF (I1--2323384), AFOSR (FA9550-20- (045550-001100077), and the Center for Al Safety (CAIS). This work has taken place in part in the Rewarding Lab at UT Austin. The Rewarding Lab is supported by NSF (IIS (US-240,,50), ONR (N00014-22-1- 1-244)), EA Ventures, Bosch, UT Austin' 's Good Systems grand challenge, and Open Philanthropy.
References
[1] A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, A. Üstün, and S. Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in lms. arXiv preprint 202.44240..14740, 2024.
10

---

[2] R. Akrour, M. Schoenauer, and M. Sebag. Preference-basee based policy learning. In Joint European Conference on Machine Learning and Knowledge Discovery Databases, 2011.
[3] D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete problems in ai safety. arXiv preprint
2016.
[4] M. G. Azar, M. Rowland, B. Piot, D. Guo, D. Calandriello, M. Valko, and R. Munos. A general theoretical paradigm to understand learning from human preferences, 2023.
[5] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, 'T. Conerly, S. El-Showk, N. Elhage, Z. Hatfield Dodds, D. Hernandez, 'T. Hume, S. Johnston,
Kravec, L., Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, B. Mann, and J. Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022.
[6] S. Biderman, H. Schoelkopf, Q. Anthony, H. Bradley, K. O'Brinn, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
[7] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 -45, 1952. doi:
[8] T. Brown, B. Mann, N. Ryder, M. Subbiah, D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020..
[9] S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freire, T. Wang. S. Marks, C.-R. Segerie, M. Carroll, A. Peng, P. Christoffersen, M. Damani, S. Slocum, U. Anwar, A. Siththaranjan, M. Nadeau, E. J. Michaud, J. Pfau, D. Krasheninnikov, X. Chen, L. Langosco, P. Hase, E. BIyık, A. Dragan, D. Krueger, D. Sadigh, and D. Hadfield-Menell Open problems and fundamental limitations of reinforcement learning from human feedback, 2023.
[10] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. V. Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Sys- tems, volume 30. Curran Associates, Inc., 2017, URL https:/ httppps:/prrocceeinnggs www.eeeedddinnnnnnnnieeeeee newrips....... cc/
[11] J. Clark and D. Amodei. Faulty reward functions in the wild, 2016. URL https://///paaa functions
caa//researrc/aallyy-eeeeae
[12] K. Cobbe, O. Klimov, C. Hesse, T. Kim, and J. Schulman. Quantifying generalization in reinforcement learning. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1282-1289. PMLR, 09-15 Jun 2019. /ttp/ttts:///proceeeee html
mlr. wptpss/v//ccbbssllpee
[13] T. Coste, U. Anwar. R. Kirk, and D. Krueger. Reward model ensembles help mitigate overopti- mization, 2023.
[14] Y. Dubois, X. Li, R. Taori, T. Zhang, I. Gulrajani, J. Ba, C. Guestrin, P. Liang, and T. B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2024.
[15] J. Ebrahimi, D. Lowd, and D. Dou. On adversarial examples for character-level neural machine translation. arXiv preprint arXiv: 09030, 2018
[16] Eisenstein, C. Nagpal, A. Agarwal, A. Beirami, A. D' Amour, D. Dvijotham, A. Fisch, K. Heller, S. Pfohl, D. Ramachandran, P. Shaw, and J. Berant. Helping or herding? reward model ensembles mitigate but do not climinate reward hacking, 2023.
[17] K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint 2rX0v:2402.01336, 2024.
11

---

[18] T. Everitt, V. Krakovna, L. Orseau, M. Hutter, and S. Legg. Reinforcement learning with corrupted reward channel. arXiv preprint arXiv:1705.08417 2017.
[19] J. Farebrother, M. C. Machado, and M. Bowling. Generalization and regularization in dqn. arXiv preprint arXiv: /810.00/3,, 2018.
[20] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration, 2019.
[21] L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. Interna- fional Conference machine Learning, 2023.
[22] D. Madeield-Menell, S. Milli, P. Abbeel, S. J. Russell, and A. Dragan. Inverse reward design. In
I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL
newrips...ppp.... cc/ http///arr...lleeppp//////////// 9
pdf
[23] J. Hejna, R. Rafailov, H. Sikehi, C. Finn, S. Niekum, W. B. Knox, and D. Sadigh. Contrastive preference learning: Learning from human feedback without reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. URL https://
openreview wtt.:eoormmmiidi111//00
[24] D. Hernandez, J. Kaplan, T. Henighan, and S. McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.01299. 2021.
[25] K. Hoskin. The 'awful idea of accountability': inseribing people into the measurement of objects. Accountability; Power, ethos and the technologies of managing, 265, 1996.
[26] A. Hosseini, X. Yuan, N. Malkin, A. Courville, A. Sordoni, and R. Agarwal. V-star: Training verifiers for self--tughht reasoners. arXiv preprint 47X1v:2402.06457, 2024.
[27] S. Im and Y. Li. Understanding the learning dynamics of alignment with human feedback, 2024.
[28] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot,
D. de las Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. R. Lavaud,
L. Saulnier, M.-A. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. L. Scao,
T. Gervet, T. Lavril, T. Wang. T. Lacroix, and W. E. Sayed. Mixtral of experts, 2024.
[29] S. Kabir, D. N. Udo-Imeh, B. Kou, and T. Zhang. Who answers it better? an in- analysis of chatgpt and stack overflow answers software engineering questions, 2023.
[30] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models, 2020.
[31] W. B. Knox and P. Stone. Tamer: Training an agent manually via evaluative reinforcement. In
2008 7th IEEE international conference on development and learning, pages 292-297. IEEE, 2008.
[32] V. Krakovna and R. Kumar. Classifying specification problems variants of goodhart's law, 8 2019. URL https:://trrakovaa..rrrtt wwwpwowwwwwwwwwooowowwwwooooe... cttp:/ss//ccc////////////// classifying-spen specificationn pww.llees-------rraattt-----eeee -ww.oaarto-oooooooooooooo------
[33] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-1199, 2020.
[34] N. Lambert and R. Calandra. The alignment ceiling: Objective mismatch in reinforcement learning from human feedback, 2023.
[35] S. Levine, A. Kumar, G. Tucker, and Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020.
12

---

[36] P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan,
Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning. C. Ré, D. Acosta-Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. Wang, K. Santhanam, I.. Orr, I.. Zheng, M. Yuksekgonul, M. Suzgun, N. Kim, N. Guha, N. Chatterji, O. Khattab, P. Henderson, Q. Huang, R. Chi, S. M. Xie, S. Santurkar, S. Ganguli, T. Hashimoto, T. Icard, T. Zhang, V. Chaudhary, W. Wang, X. Li, Y. Mai, Y. Zhang, and Y. Koreeda. Holistic evaluation of language models, 2023.
[37] Y.-C. Lin, Z.-W. Hong. Y.HH. Liao, M.-L. Shih, M.-Y. Liu, and M. Sun. Tactics of adversarial attack on deep reinforcement learning agents. arXiv preprint VwXIvv1703.066 2017.
[38] T. Liu, Y. Zhao, R. Joshi, M. Khalman, M. Salch, P. J. Liu, and J. Liu. Statistical rejection sampling improves preference optimization, 2024.
[39] D. Manheim and S. Garrabrant. Categorizing variants of goodhart's 's law, 2019.
[40] T. Moskovitz, A. K. Singh, D. Strouse, T. Sandholm, R. Salakhutdinov, A. Dragan, and S. M. McAleer. Confronting reward model overoptimization with constrained RLHF. In The Twelfth International Conference on Learning Representations, 2024. URL ttttts://oppnree net/forum? ii= id=ggk
[41] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, F. Christiano, J. Leike, and R. Lowe. Training language models follow instructions with human feedback. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27730-27744, Curran Associates, Inc., 2022. URL URL/attps: neuriss cc/ wwtpccpppppppeeeeeeeeeeeee Conaperr-Coofeeenceee pdf
[42] A. Pal, D. Karkhanis, Dooley, M. Roberts, Naidu, and C. White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.13225 2024.
[43] A. Pan, K. Bhatia, and J. Steinhardt. The effects of reward misspecification: Mapping mitigating misaligned models. International Conference on Learning Representations, 2022.
Mapping and
[44] R. Park, R. Rafailov, S. Ermon, and C. Finn. Disentangling length from quality in direct preference optimization, 2024.
[45] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners, 2019, OpenAI.
[46] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly reward model. In Thirtty-svennt Conference on Neural Information Processing Systems, 2023. URL httttt://///rtte http//axxvvoog//////// org/abs/////eoooooooeoogooeeeeet htsp//bb/3305.... 18290
[47] R. Rafailov, J. Hejna, R. Park, and C. Finn. From r to q": Your language model is secretly q-function, 2024.
[48] N. D. Ratiiff, J. A. Bagnell, and M. A. Zinkevich. Maximum margin planning. In Proceedings of the 23rd international conference on Machine learning, pages 729-736, 2006.
[49] M. Rita, F, Strub, R. Chaabouni, P. Michel, E. Dupoux, and O. Pictquin. Countering reward over-optimization in 11m with demonstrationnguuide reinforcement learning. arXiv preprint arXiv:2404.19400 2024.
[50] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, 2017.
[51] H. Sikchi, A. Saran, W. Goo, and S. Niekum. A ranking game for imitation learning. arXiv preprint 220..2200..0488... 2022.
[52] H. Sikchi, Q. Zheng, A. Zhang, and Niekum. Dual rl: Unification and new methods for reinforcement and imitation learning, arXiv preprint arXiv:2302.08566 2023.
13

---

[53] P. Singhal, T. Goyal, J. Xu, and G. Durrett. A long way to go: Investigating length correlations in rlhf, 2023.
[54] J. Skalse, N. H. R. Howe, D. Krasheninnikov, and D. Krueger. Defining and characterizing reward hacking, 2022.
[55] N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. Christiano. Learning summarize from human feedback, 2022.
[56] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:/312.6199 2013.
[57] F. Tajwar, A. Singh, A. Sharma, R. Rafailov, J. Schneider, T. Xie, S. Ermon, C. Finn, and A. Kumar. Preference fine-tuning of Ilms should leverage suboptimal, on-policy data. arXiv preprint arXiv:2404. 14367, 2024.
[58] Y. Tang, D. Z. Guo, Z. Zheng, D. Calandriello, Y. Cao, E. Tarassov, R. Munos, B. Á. Pires, M. Valko, Y. Cheng, et al. Understanding the performance gap between online and offline alignment algorithms. arXiv preprint arXiv:2405.0844 2024.
[59] Y. Tang, Z. D. Guo, Z. Zheng, D. Calandriello, R. Munos, M. Rowland, P. H. Richemond, M. Valko, B. Ávila Pires, and B. Piot. Generalized preference optimization: A unified approach to offline alignment, 2024.
[60] Taylor. Quantilizers: safer alternative to maximizers for limited optimization. In Workshops at the Thirtieth AAAI Conference Artificial Intelligence, 2016.
[61] G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, J. Ferret, P. Liu, P. Tafti, A. Friesen, M. Casbon, S. Ramos, R. Kumar, C. Lan, S. Jerome, A. Tsitsulin, N. Vieillard, P. Stanczyk, S. Girgin, N. Momchev, M. Hoff-
man, S. Thakoor, J.-B. Grill, B. Neyshabur, O. Bachem, A. Walton, A. Severyn, A. Parrish, A. Ahmad, A. Hutchison, A. Abdagic, A. Carl, A. Shen, A. Brock, A. Coenen, A. Laforge, A. Paterson, B. Bastian, B. Piot, B. Wu, B. Royal, C. Chen, C. Kumar, C. Perry, C. Welty, C. A. Proquette-Choo. D. Sinopalnikov, D. Weinberger, D. Vijaykumar, D. Rogozińska, D. Herbison, E. Bandy, E. Wang. E. Noland, E. Moreira, E. Senter, Eltyshev, F. Visin, G. Rasskin, G. Wei, H. Batra, H. Dhand, I. Nardini, Mein, J. Zhou, J. Svensson, J. Stanway, J. Chan, J. P. Zhou, J. Carrasqueira, Iljazi, Becker, J. Fernandez, J. van Amersfoort, J. Gordon, J. L.radon,,LL.ieeeee Lipschultz. J. Newlan, J. yeong Ji, K. Mohamed, K. Badola, K. Black, K. Millican, K. McDonell, K. Nguyen, K. Sodhia, K. Greene, L. L. Sjoesund, L. Usui, L. Sifrre, L. Heuermann, L. Lago, L. McNealus, L. B. Soares, L. Kilpatrick, L. Dixon, L. Martins, M. Reid, M. Singh, M. Iverson, M. Görner, M. Velloso, M. Wirth, M. Davidow, M. Miller, M. Rahtz, M. Watson, M. Risdal, M. Kazemi, M. Moynihan, M. Zhang, M. Kahng, M. Park, M. Rahman, M. Khatwani, N. Dao, N. Bardoliwalla, N. Devanathan, N. Du- mai. N. Chauhan, O. Wahltinez, P. Botarda, P. Barnes, P. Barham, P. Michel, P. Jin, P. Georgiev, P. Culliton, P. Kuppala, R. COmanescu, R. Merhej, R. Jana, R. A. Rokni, R. Agarwal, R. Mullins, S. Saadat, S. M. Carthy, S. Cogan, S. Perrin, S. M. R. Arnold, S. Krause, S. Dai, S. Garg, S. Sheth, S. Ronstrom, S. Chan, T. Jordan, T. Yu, T. Eccles, T. Hennigan, T. Kocisky, T. Doshi, Jain, V. Yadav, V. Meshram, V. Dharmadhikari, W. Barkley, W. Wei, W. Ye, W. Han, W. Kwon, X. Xu, Z. Shen, Z. Gong, Z. Wei, V. Cotruta, P. Kirk, A. Rao, M. Giang, L. Peran, T. Warkentin, E. Collins, J. Barral, Z. Ghahramani, R. Hadsell, D. Sculley, J. Banks, A. Dragan, S. Petrov, O. Vinyals, J. Dean, D. Hassabis, K. Kavukcuoglu, C. Farabet, E. Buchatskaya, S. Borgeaud, N. Fiedel, A. Joulin, K. Kenealy, R. Dadashi, and A. Andreev. Gemma 2: Improving open language models at a practical size, 2024. URL
G. Cameron, G. Martins, H. Hashemi, H. Klimezak-Pllucisste
org/ abs////s20 00118
[62] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro. F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[63] B. Wallace, M. Dang, R. Rafailov, L.. Zhou, A. Lou, S. Purushwalkam, S. Ermon, C. Xiong, S. Joty, and N. Naik. Diffusion model alignment using direct preference optimization, 2023.
14

---

[64] J. Watson, S. Huang, and N. Heess. Coherent soft imitation learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL ltttps://openrevieen htte/forumm?ddxcCDBBeee
[65] R. J. Williams. Simple statistical gradient-following algorithms for connectionist rein- forcement learning. Mach. Learn., 8(3 4):229-22222 256, may 1992. ISSN 0885-6125. doi: 10.1007/BF00099666.
org/ 10. 1
[66] R. Yuanzhe Pang. W. Yuan, K. Cho, H. He, S. Sukhbaatar, and J. Weston. Iterative reasoning preference optimization. arXiv e-prints, pages arXiv-2404, 2024.
[67] Y. Zhai, H. Zhang, Y. Lei, Y. Yu, K. Xu, D. Feng, B. Ding, and H. Wang. reinforcement learning from human feedback with diverse reward lora ensembles, 2023.
[68] Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P. J. Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 2002:2305./0425, 2023.
[69] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. E. P. Xing, H. Zhang, J. E. Gonzalez, and Stoica. Judging llm-as-a-judge with mt- mt-bench and chatbot arena. Conference Neural Information Processing Systems Track on Datasets and Benchmarks., 2023.
[70] B. Zhu, M. Jordan, and J. Jiao. Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf. arXiv preprint arXiv:2401.16335, 2024.
[71] B. D. Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.
[72] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and
G. Irving. Fine-tuning language models from human preferences, 2020.
15

---

A Limitations and Societal Impacts
Our discussion highlights number of issues with direct alignment algorithms used widely as means to align to human values. This work has mostly focused on pointing out those issues along with theoretical underpinning of the issue but does not provide way to resolve these issues. We still assume an underlying model of human preferences, which is an ongoing research area no model is perfect explaining the ways humans give preferences. Our work aims to drive the push towards better alignment algorithms that do not overoptimize and generate models that are safe to be deployed our society. We believe only through understanding and demonstrating the shortcomings current methods we can develop better alignment methods.
B Experiment Details
We largely follow the DPO setup unless otherwise mentioned and build on their code without changing any hyperparame-
http:///w/oeemiicc//lleeeeeeeeee ters unless otherwise mentioned.
For all DAA experiments, we used the curated OpenAl TL:DR dataset with 92K preferred-dispefeereed summary completions [55]. Each prompt is Reddit post belonging to one several topic forums, with titleppost metadata included. 256 prompts sampled from the held-out set are used for all evaluations (e.g. loss, accuracy, KL, winrates, length), with temperature 1.0 and max length 512.
Model sizes include IB, 2.8B, and 6.9B and were initialized from the base Pythia pre-trained weights. All models underwent supervised fine-tuning on TL;DR prior direct alignment. Across all SFT and DAA runs, we used batch size of 128 (8 gradient accumulation steps), and RMSProp with learning rate of 0.5 x 10-0 (linear warmup for 150 steps) for epoch. 1B models were trained on NVIDIA A40 GPUs, 2.8B models were trained on 4 NVIDIA A40 GPUs, and 6.9B models were trained on NVIDIA 100 GPUs. All evaluations were computed with "gpt-4-urbo-222-000000"" as judge, with random positional flips avoid known bias.
C Appendix A: Complete Intra-Epoch Training Dynamics
This appendix contains similar intra-epoch KL divergence and winrate evolution results as in Fig. across all model sizes.
16

---

Wiraat exolutie spech
evalation SUI 2.00
Werrat evolution 2.00
ow Su
40
CRQ (-003351
ong (HOLS)
OPF
See ELIS
P0 Doi
SUC
MAC god
82
00
SUC 200
6P0 scuch Tacks
Dro Gro-spadhtaa macaa
diel Foon OVDY maech (MS3
extlution over eeoch M.K
R1. reckkiot over epech
BFO12-E.8
2U5 1--0-00
68/11
00-00051
SUC D-o
Pr0 go6.21
6411
42201
033
8-420
0.29
sic p-43)
oroup 2001
0.5
2013 2:11
632
2019-100
(8018-300)
#013
057
(M0 hper
avolucion
expiutioon
SUI
over epach
C25
60
00010-3:00
070-0-330070 80700
P0 (-61315)
PO pw631
10110-3.
4001 Sratton
Figure 8: KL divergence and GPT4 winrate evolution for 2.8B and 1B models across DPO, SLiC, and IPO losses. Similar to the 6.9B models, performance tends to degrade after the first quarter epoch, particularly under a low KL budget, while KL. increases almost monotonically.
17

---

D Overoptimization from the lens of Implicit Bootstrapping
Reward over-optimization well understood in the classical RLHF setting, with a consensus that is driven by two main components using proxy reward function that is trained on limited data and continuous querying with new, potentially OOD samples during PPO training. At first glance, none of these conditions hold in DAAs as we do not train a separate proxy reward model generate new data during training. Therefore, understanding reward over-optimization in DAAs requires new theory. We will base our analysis on [47 using the token-level MDP and corresponding (soft) Q-learning formulation. Consider the class of dense per-token reward functions ro Vgs), where
Vgi denotes the first tokens of y, with sequence level-reward re(x, 1) Σ 2 ra(a, Mei). This is strictly more general class than the sparse reward function which returns a single score at the end of the sequence since
set all intermediate rewards as O. Within the framework of [47] given DAA-trained policy иa. there exists a dense per-token reward rg, that minimizes the reward modeling objective in Eq. and satisfy the below.
The (soft) Bellman Equation holds:
if y: is not EOS (yss (I, ver)) = rx, Vgs) + log Thet The Ve:)) + V ((z, Vgi)), if is EOS
(8)
Ves) + log Tha ( (ail(ar vc:)),
where V is the corresponding soft-value function:
(* log
Σ
(9)
NEIV
then the DAA policy satisfies:
Ta(vil(z, = exp( 1 Q- ( (z, (z,)) - ("((z,
(10)
in this interpretation, the LLM logits lo|i] = Q" (vi (z, (e:)))))// represent Q-values. With direct substitution, we then have
Q" (z, vee)) r(a, + 3 log T (x log
Σ ('(uecvss))1
(1I)
M.VV
OOD bootstrapping
That is in this framework DAAS may suffer from the classical OOD bootstrapping issue in offline RL [20] 35 33 (2). In this case, even though the objective trained fully offline we still effectively query the model the values of unseen tokens. This interpretation also provides further insight into the effect of the coefficient and the training dynamics. For small values of beta the estimate
Blog Σ
Q (v.(a,v)))18
max Q" (y, (I, (c:))))
(12)
yEVV
that is smaller parameter values yield more optimistic estimate, which results in a higher level of OOD bootstrapping. This interpretation would also explain the somewhat counter- cunter-intuitivv results of section 3.4 While the implicit reward function can adequately fit and model the data, the resulting LLM might behave sub-optimally, due OOD bootstrapping in the corresponding Q- Q-value estimate.
18

---

E Understanding Behavior of DAAS on OOD sequences
We have established that common DAA objectives allow for placing a high likelihood on OOD data. In practice, while one might expect the likelihood of preferred responses to increase during training, it has been observed that algorithms like DPO decrease the likelihood of both the preferred and dis-preferred responses [2]. In fact, this is expected from max-entropy RL perspective [47. Since the total probability mass must sum to one, the probability of OOD responses must increase during the course of training. A small amount of extrapolation may be necessary to reach the optimal policy, however, too much is potentially detrimental to performance. Because they are not adequately constrained to the reference distribution, current DAA objectives allow this to happen.
To understand how DAAS allocate probability mass out of distribution, we use a toy Markov Decision Process (MDP), that mimics the LLM setting. The MDP is modeled as a tree, originating from a single start state, featuring deterministic transitions. The Toy MDP is illustrated in fig. 6
E.1 Designing a toy LLM MDP
The MDP is modeled tree, originating from single start state. This configuration mirrors the token-level MDP in Direct Preference Optimization (DPO) 47], or the scenario where both preferred and dispreferred responses are conditioned on the same prompt in the broader Large Language Model alignment context. Each leaf node in the MDP transitions deterministically to a terminal absorbing state, regardless of the action taken. The deterministic transitions resemble the LLM setting, where the current state is represented by the sequence of encountered tokens (81, 82, 8:), and the action corresponds predicting the next word 8 from the vocabulary, given the context. In this simplified MDP, the deterministic transition is akin to a concatenation function, advancing the state to the next step (81: 82, 82; 81+1). Employing toy MDP enables us to systematically evaluate the trajectory probabilities for all feasible paths within the MDP, shedding light on the allocation of probability mass by Direct Alignment Algorithms (DAAs) towards out-of-distribution (OOD) trajectories.
The Experimental Setup. We adhere to the standard direct alignment protocol 46 ing two key stages:
encompass-
1. Supervised Fine-tuning (SFT) / Behavioral Cloning (BC): This phase involves fine- tnning the policy based on a limited number of trajectories. Specifically, we utilize three demonstrations for SFT: (s1, au, 82, ao, 85, ao, Soc ), ($1, 01 $3, 01 Sy, ao, Soo), and ($1, 02 St, 02 $13, a2 5.
2. Alignment with Preferences: In this stage, preferences extracted from trajectories are employed to align the policy. Notably, we have only one preference available: (s1, 01, $3, a1 Sg, ag, 5 > ($1, an $2, an $5, ag sxc). This deliberate constraint exag- gerates a scenario with limited data, enabling dutt----iistiittion (OOD) trajectories under such conditions. Insights garnered from this exaggerated low-data scenario hold relevance for Large Language Model (LLM) settings where preference datasets used for alignment are notably smaller compared to the scale of LLM models deployed.
to gauge the probability mass allocated to
We utilize a Recurrent Neural Network (RNN) policy to navigate through the MDP, facilitating a closer resemblance to
language modeling scenarios.
Subsequently, we explore three distinet direct alignment loss functions: Direct Preference Optimiza- tion (DPO) 46), Identity Preference Optimization (IPO) 4. and Sequence Likelihood Calibration (SLiC) 68 J Additionally, we investigate how the selection of the KL penalty coefficient 3 influences the distribution of probability mass on OOD trajectories. This exploration encompasses three values of 3: (0.01, 0.1, 0.5).
In general, the plots illustrate that Direct Alignment Algorithms (DAAs) tend to allocate a significant proportion of the probability mass to out- dfstiiiiibttii (OOD) trajectories during the alignment process. While Figure 9 may suggest that Direct Preference Optimization (DPO) can retain a substantial amount of probability mass on the selected trajectory the preference dataset, it's noteworthy that the plots for DPO exhibit considerable noise. To provide further insight, Figure T8 displays the plots resulting from three additional repetitions of the DPO experiment. Similar noisy trends were also observed in the experiments for IPO and SLiC. This elucidates the unconstrained
19

---

nature of the DPO problem: multiple solutions exist for the DPO loss, each distributing varying amounts of probability mass to OOD trajectories. In the experiments with IPO and SLiC, it's also observed that similar to DPO, the probability mass allocated to in-distribution trajectories can diminish substantially over the course of training. Notably, the probability mass, in our experiments, becomes concentrated on a select few out-of-distribution trajectories. Moreover, consistent trends are discernible across various values of 3, The results our experiments with the Toy-MDP can be found in the following figures 12
5
13
10
16
14
20

---

600 Bajectry yrotetiiiee ave DRD trening beta-d.1, Torel
60800
wh4hhald
130
50
80
20
10
10
3980
130
155
25
25
16
Distribution Trajectory Proboshiltiiss training:
1p
cheson (O, 02.12 05 0I]
rajected-(1, 3) [3, 86, 61]
Robubiliss
64.
0.2
25
3
103
125
-30
175
200
Training Staps
Figure 9: Trajectory probabilities throughout DPO training, 8 = 0.1. The top plot shows how the probability mass of different OOD trajectories, changes throughout training, The bottom plot shows how the probability mass of the trajectories in our preference dataset (size 1) changes over training. The trajectories are listed in the legends for the plots, as a sequence of state, action pairs.
21

---

DOD pubbesiitie
betwdh
1300
20
20
20
120
26
6008
4000
16
030
000
660
20
20
120
10
10
28
412
1011
030
0194
Seepe
In Distributiion Trajectory Probe over training: beta-0.1. 1pre
chooens(2) 06, 12 0L 05 di)
rejected (i), 13. 11.11. 8,
64
0.2
03
25
123
-30
200
Theining
Figure 10: Trajectory probabilities throughout IPO training, 8 0.1. The top plot shows how the probability mass of different OOD trajectories, changes throughout training. The bottom plot shows how the probability mass of the trajectories in preference dataset (size I) changes over training. The trajectories are listed in the legends for the plots, as a sequence of state, action pairs.
22

---

probabiities
seled.1, 1pet
20
20
20
12
30
1500
CO000
00001
508
20
25
20
138
10
90
3000
Taaning Seepe
Distribution Trajectory Probabilities over training. beta-0.1. 1pr
chooens(2, 0t, 12 0L di)
rejected (0), 13. 13.11,8 85,
0.0
0.-
0.4
0.2
03
25
123
-30
200
Theining
Figure 11: Trajectory probabilities throughout SLiC training, 0.1. The top plot shows how the probability mass of different OOD trajectories, changes throughout training. The bottom plot shows how the probability mass of the trajectories in preference dataset (size I) changes over training. The trajectories are listed in the legends for the plots, as a sequence of state, action pairs.
23

---

GOD Thgector protaciiliee new DPS training: sete-..0. Jpre!
380
30
-- 9432
18
26
18
600M
105
140
600
35
136
25
2
15
36
Distribution Trajectory Probsbilities training:
cheson (O, 0i, 05 0I]
0.8
rajected-(1,, (1,, 3) [3, 01]
Subabilive
0.4
25
103
125
-30
175
200
Training Stagps
Figure 12: Trajectory probabilities throughout DPO training, 3 = 0.01. The top plot shows how the probability mass of different OOD trajectories, changes throughout training, The bottom plot shows how the probability mass of the trajectories in our preference dataset (size 1) changes over training. The trajectories are listed in the legends for the plots, as a sequence of state, action pairs.
24

---

noo betadoo) 1pm
20R1
20
20
20
10
30
--MILEE 5/IL
698
20
20
130
10
9
1.0
Taining Seep
Distribution Trajectory Probabilities over IPO training. beta-0.01. 1pm
chooens(2) 06, 12 0L 05,0 di)
rejected (i), 13. 11.
0.0
0.6
0.4
0.7
03
25
123
-30
200
Theining
Figure 13: Trajectory probabilities throughout IPO training, 3 = 0.01. The top plot shows how the probability mass of different OOD trajectories, changes throughout training. The bottom plot shows how the probability mass of the trajectories in preference dataset (size I) changes over training. The trajectories are listed in the legends for the plots, as a sequence of state, action pairs.
25

---

con protahtries SIE
2300
3700
20
20
20
12
6-3211
2300
3300
00/8
1804
1800
20
20
20
10
10
2000
6808
600m
1.0
329
Taaning Seipe
Distribution Trajectory Probabilities over SUC training. beta-0 1pr
0.7
chooens (2, 06, 12 01 05,05 di)
rejected (0), (3. 13.11.85 8,,88
0.6-
0.5-
0.4-
0.3
0.2
61.
033
25
-30
200
Theining
Figure 14: Trajectory prohabilities throughout SLiC training, 3 = 0.01. The top plot shows how the probability mass of different OOD trajectories, changes throughout training. The bottom plot shows how the probability mass of the trajectories in preference dataset (size I) changes over training. The trajectories are listed in the legends for the plots, as a sequence of state, action pairs.
26

---

G03 Bajettry yrotabiitie owe DR trening betadd.s Torel
200
20h
508
6000
135
4000
10
2008
121
6000
600
1.4
380
156
305
300
20
Trajectory Probeshiltiis training:
1pe
cheson (O), 02.12 O1 0I]
0.7
(ajcced-(13, 3) (3,11, 85,00
0.0
0.4
0.3
61
25
3
103
125
-30
175
200
Training Staps
Figure 15: Trajectory probabilities throughout DPO training, 3 = 0.5. The top plot shows how the probability mass of different OOD trajectories, changes throughout training, The bottom plot shows how the probability mass of the trajectories in our preference dataset (size 1) changes over training. The trajectories are listed in the legends for the plots, as a sequence of state, action pairs.
27

---

DOD prbbesiitiee PPO betwd
380
20
20
20
6000
9491 1.1.8.13
38.0
3303
3303
2001
20
20
20
20
20
35
Seepe
In Distributiion Trajectory Probe over training: beta-0.5, 1pre
chooens(2) 06, 12 0L 05,05 di)
rejected (0), 13. 11, 8,,
0.0
0.6
0.4
0.7
03
25
123
-30
200
Theining
Figure 16: Trajectory probabilities throughout IPO training, 8 0.5. The top plot shows how the probability mass of different OOD trajectories, changes throughout training. The bottom plot shows how the probability mass of the trajectories in preference dataset (size I) changes over training. The trajectories are listed in the legends for the plots, as a sequence of state, action pairs.
28

---

probabitties
beled. 1pea
2001
3080
90
80
00
WERSSEN
3007
3000
28
89
30
80
09
60092
Seepe
Distribution Trajectory Probabilities over training. beta-0.5. 1pc
chooens(2, (2, 0t, 12 0L di)
rejected (0), 13. 11,11. 85,
0.1
0.3
02-
6.1
03
25
-30
200
Theining
Figure 17: Trajectory probabilities throughout SLiC training, 0.5. The top plot shows how the probability mass of different OOD trajectories, changes throughout training. The bottom plot shows how the probability mass of the trajectories in preference dataset (size I) changes over training. The trajectories are listed in the legends for the plots, as a sequence of state, action pairs.
29

---

Paderesc Eajedtry hockealiiiiee Bee coer traking betk.... Predenros DoRiuaion Tipedrr Frinadd Ple makking blt
slmenttf de 15.4
320
Proleuss Dutnktis hiaaayy hobubilio se basiny beiaf11
400
Figure 18: Trajectory probabilities throughout DPO training, over three different runs, with 3 = 0.1
divergence winrate Gamma2-26-Ht
divergence Gemma2-2bun
5he goaicitt wallt
276 8--8 e-20
4.5
Axb
SUE Babdi ent
FLR BeNCITI webts
SUK BOAL SHOEE
Sample Divergence
Sample Diueegence
Figure 19: KL divergence versus GPT-4 win rate for the Gemma2-2b model on the Anthropic-Hi dataset. The left plot shows DPO results, and the right plot shows SLiC results.
F
Overoptimization Trends in the Gemma2-2b Model and Anthropic-HH Dataset
We present KL divergence versus GPT-4 win rate plots in Figure 19 illustrate overoptimization trends in Direct Alignment Algorithms for the Gemma2-2b model 61 and the Anthropic-HH dataset [.. Results are shown for the DPO and SLiC variants, which sufficiently demonstrate that the overoptimization trends observed with the Pythia models are not specific to single model dataset. The figure illustrates the trade-off between KL divergence and GPT-4 win rate across different values of beta in the alignment objective.
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper faithfully adheres to the claims and motivation in the abstract and provides proof and detailed empirical studies in support.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: A discussion of our limitations be found separate section at the beginning of the appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
We provide proofs and empirical evidence to support all our theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch provide intuition.
31

---

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide detailed guidance on reproducibility by specifying all datasets, code, and hyperparameters used in this work.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have only used open-source models with apen-source datasets for all aspects of the work. Please refer to section for details on reproducing the results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc.
for more details.
32

---

While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https / cttpps.ccppulic///e.. guwdes/ Codeeubmis isi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We list detailed information about the training and test details in Section ??. Our experiments open-source datasets and models.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Training Large Language models is time-consuming and compute-intentn Our experiments do not run multiple seeds and financial budget. Instead, the focus this work is extensive evaluation across multiple configurations which spent all our compute resources into. Our evaluation protocol is similar to prior influential works in RLHF 46
configuration due limited computing
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
33

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (lype of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide information on compute resources in the experimental details section B in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We abide by the code of ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss socictal impacts in Section A of the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinfornation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
34

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: We use public models that fine-tuned for alignment on open-source datasets. Our models do not contribute any additional risk over the base models as we are explicitly training for alignment.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The pretrained models in this work come from the Pythia family all of which are classified under Apache License https://hugginggaee
2.8b/tree/main The TL:DR comparison dataset used in this work uses a modified MIT
License
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
35

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We use open-source pretrained models and provide details to reproduce our fine-tuning experiments.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not crowdsourcing research with human subjects in this work
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not crowdsourcing research with human subjects in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
36


Reviews:
[{"review_id": "1RBOSFUBwC", "reviewer": "Reviewer_Q9V7", "summary": "Reinforcement Learning from Human Feedback (RLHF) is a popular paradigm for aligning Large Language Models (LLMs) to human preferences. Direct Alignment Algorithms (DAA) are an alternative to traditional RLHF methods, which reduce the need to learn a reward and policy model separately. It has been shown that traditional RLHF methods suffer from reward over-optimization or reward hacking. However, because DAA do not learn a separate reward model, it is important to study whether these algorithms suffer from the same issues. This paper studies three popular DAA algorithms and how their performance deteriorates from over-optimization at scale. The paper has several compelling results that show the trade-off between different DAA objectives, KL divergence, and win rate when optimizing across different scales of models.", "questions": "- Could you provide best-fit lines for each model size in Figure 4 so that I can see trends? (This is the point raised on lines 196-200.)\n- Is it a fair conclusion that IPO performs the best among the DAA studied in this paper?\n- Are IPO and SCLI also prone to length exploitation? If so, could you provide results similar to those for DPO?\n- How did you perform model selection?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Empirical experiments provide insight into DAA algorithm's behavior at scale.\n- The empirical verification of the decreasing likelihood and model performance provides insight into an important issue in the literature: the decrease in the responses of both chosen and rejected samples.\n- The paper is generally well-written, and I was able to follow most empirical conclusions.", "weaknesses": "- The paper only performs experiments on a single model class and task, so it is not clear if these results generalize. For example, the paper observes that most models perform best after training on only 25% of the data, but it is not obvious if this is an artifact of the specific models used or the data. Does the same observation hold true on a different model or a different task?\n- The performance of DPO seems under-tuned compared to the results of DPO in the literature. (1, 2).\n- The paper does not provide any details regarding how model selection was performed and additional design choices.\n\n(1) REBEL: Reinforcement Learning via Regressing Relative Rewards by Zhaolin Gao\n\n(2) BPO: Supercharging Online Preference Learning by Adhering to the Proximity of Behavior LLM", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the informative review!\n\n**The paper only performs experiments on a single model class and task, so it is not clear if these results generalize…**\n\nWe are working on replicating our main experiment with the Gemma-2 2B model on the Anthropic Helpful and Harmless dataset. Our initial results suggest similar over-optimization dynamics to the Pythia TL;DR experiments presented in the paper. We will include the additional experiment in our revised submission.\n\n\n\n**The performance of DPO seems under-tuned compared to the results of DPO in the literature. (1, 2).**\n\nThese discrepancies are likely due to some parameter choices. For example REBEL (1) uses generation temperature of 0.1 and max-token length of 53, while we use sampling temperature 1.0 and max-token length of 512 for all of our experiments.  This can affect performance on TL;DR as shown in the original DPO work [1]. There is also a slight difference in the wording of the evaluation prompt as we also use “concise” in our evaluation prompt, which may lower win-rates for longer summaries. We will include all of these details in an updated appendix with our camera-ready version. \nWe could not find enough details on these parameters in the second reference. \n\n\n\n**The paper does not provide any details regarding how model selection was performed and additional design choices.**\n\nCould the reviewer expand on this point? We report performance of the final trained checkpoint after 1 epoch of DAA training, as well as intermediate checkpoints at regular intervals. We do not carry out any more involved checkpoint selection.\n\n\n\n**Could you provide best-fit lines for each model size in Figure 4 so that I can see trends? (This is the point raised on lines 196-200.)**\n\nWe will include those modifications in our updated camera-ready submission.\n\n\n\n**Is it a fair conclusion that IPO performs the best among the DAA studied in this paper?**\n\nFor all the algorithms we studied (DPO, IPO, SLiC) the best checkpoints reach similar performance. However, IPO indeed seems to be more robust to the over-optimization phenomenon.\n\n\n\n**Are IPO and SCLI also prone to length exploitation? If so, could you provide results similar to those for DPO?**\n\nWe have carried out additional evaluations on this issue. All the studied algorithms show significant length increases in the response, beyond the dataset coverage. However the relationship between length and implicit rewards seems to be more nuanced than the results shown in Figure 3 for DPO. We will include these additional experiments in our updated camera-ready submission. \n\n[1] Direct Preference Optimization: Your Language Model is Secretly a Reward Model\nRafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jD66uyQT9G", "reviewer": "Reviewer_r4HU", "summary": "This paper studies the scaling laws of DAAs for RLHF. The authors conducted extensive empirical studies and the discoveries are reported and discussed.", "questions": "Could the authors also provide error bars in Figure 2, second row? The results look very unstable. \n\nFor the scaling law fit, the author mentioned using win-rate to be the proxy of the golden reward, could the author provide detailed statistics of the results? I'm curious --- to what extent could different judges agree? If the win rate is only accurate in (for example,) 80% of the settings, does not that mean we are only able to draw a very rough conclusion using another proxy of this objective? \n\nOn the Tree-MDP, why should all states go to the same absorbing state? What is the consideration of using a single absorbing state rather than many (i.e., = the number of leaves).", "limitations": "please see weakness / question.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and easy to follow.\nThe authors conducted extensive empirical studies.\nThe current results are useful to the community to understand different algorithms.\nThe design of the diagnostic toy MDP is interesting and smart.", "weaknesses": "I like large-scale empirical study yet I disagree with calling those discoveries \"laws\". To draw a clear scaling **law** (from a scientific perspective), the current results are not supportive enough. Specifically, in Figure 1, the results do not seem to be a good fit, and more experimental results might be helpful to draw the conclusion. Also, why is the form of \"scaling law\" chosen as Eqn(5)? In Gal et al, BoN and PPO have different forms --- it would be highly possible that different DAAs also have different equation forms. \n\n\nThe authors should further highlight the takeaway messages of their empirical discoveries. For instance, with the discovered scaling law, is there a way to perform early stopping in training to achieve better performance? --- This could be a great contribution to the community as otherwise researchers may be prone to make unfair comparisons in evaluating algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the useful feedback!\n\n**Specifically, in Figure 1, the results do not seem to be a good fit, and more experimental results might be helpful to draw the conclusion.**\n\nWe used win rates as computed by GPT-4 for evaluation, which is now well-established in the “LLM-as-a-judge” framework to be a decent proxy for model quality as evaluated by humans [1]. The reviewer is indeed right that this approach could inject some noise in the evaluations.  \nWe do agree that further evaluations, such as higher number of evaluation samples (we used 256 held-out prompts), more training runs with different KL-parameter exploration and more intermediate checkpoints would reduce noise and draw a stronger statistical dependency. However, these require significant additional resources, both computational for model training as well as credits for LLM evaluations.\n\n**Also, why is the form of \"scaling law\" chosen as Eqn(5)?**\n\nWe evaluated a number of statistical formulations of the scaling law and found the ones presented in Gao et. al. [5] to provide as good statistical fit as any other formulation across the board. It is however possible that our search was not exhaustive and a function of similar complexity can better fit the data.\n\n**The authors should further highlight the takeaway messages of their empirical discoveries. For instance, with the discovered scaling law, is there a way to perform early stopping in training to achieve better performance? --- This could be a great contribution to the community as otherwise researchers may be prone to make unfair comparisons in evaluating algorithms.**\n\nThe goal of our work is to highlight the empirical phenomenon of reward over-optimization in DAAs, which had not been studied before as far as we are aware. We hope our work can incentivize a broader research direction of robustness for DAAs, similar to the line of research that the  Gao et. al. [5] publication spurred. \nThere are a number of promising directions to pursue on that front, such as model merging for example [2] or reward smoothing for DAAs [6]. We hope our work will incentivize researchers to pursue such ideas in follow-up publications. We ourselves are investigating mitigating issues, which we believe warrant independent investigations. \n\n**Could the authors also provide error bars in Figure 2, second row? The results look very unstable.**\n\nWe will include the modified graph in our updated camera-ready submission. \n\n**For the scaling law fit, the author mentioned using win-rate to be the proxy of the golden reward, could the author provide detailed statistics of the results? I'm curious --- to what extent could different judges agree? If the win rate is only accurate in (for example,) 80% of the settings, does not that mean we are only able to draw a very rough conclusion using another proxy of this objective?**\n\nIn the original DPO work [3], the authors carried out a human evaluation study on the TL;DR summarization task and found GPT-4 - human agreement to be 67-70%, which was higher than intra-human annotator agreement of 65%. A number of prior works have also studied this setting extensively [1], [4] and have established the use of “LLM-as-a-judge”. \n\n**On the Tree-MDP, why should all states go to the same absorbing state? What is the consideration of using a single absorbing state rather than many (i.e., = the number of leaves).**\n\nThe absorbing state in the toy-MDP is used as an analogue to the end-of-sequence token in order to faithfully reflect the true LLM finetuning setting. Theoretically, the absorbing state does not affect optimization as all actions from the pre-absorbing state leads to the absorbing state with 1 probability and its effect is canceled across the preference pairs and reference policy.\n\n[1] AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback, Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, Tatsunori B. Hashimoto\n\n[2] WARP: On the Benefits of Weight Averaged Rewarded Policies, Alexandre Ramé, Johan Ferret, Nino Vieillard, Robert Dadashi, Léonard Hussenot, Pierre-Louis Cedoz, Pier Giuseppe Sessa, Sertan Girgin, Arthur Douillard, Olivier Bachem\n\n[3] Direct Preference Optimization: Your Language Model is Secretly a Reward Model, Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn\n\n[4] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, Ion Stoica\n\n[5] Scaling Laws for Reward Model Overoptimization, Leo Gao, John Schulman, Jacob Hilton\n\n[6] Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF, Banghua Zhu, Michael I. Jordan, Jiantao Jiao", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uIMlvfeOoM", "reviewer": "Reviewer_bFR8", "summary": "This paper studies the reward over-optimization issue for offline alignment algorithms (e.g., DPO series) with massive experiment trials and discussions on why this phenomenon happens.", "questions": "See the comments in the weakness section.", "limitations": "The limitations have been discussed in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This is the first paper that studies the over-optimization issue for DPO-like algorithms systematically. The observations and findings can be beneficial to the community. The proposed experiments are organized rigorously with sufficient quantitative results. \n\n2. The paper presents concrete examples to reveal why over-optimizations could happen for DPO-like algorithms, which brings insights to the community.", "weaknesses": "There are no major weaknesses. Here are some of the minor comments for further improvement.\n\n1. Why do you choose DPO, IPO, and SLiC-HF? Why not some other variants, say ORPO? Although I do agree that the current selections can be sufficiently representative, some discussions on why they are prioritized can be appreciated. \n\n2. The presentation of Section 4 can be possibly improved. With its current presentation, it is difficult for a reader to understand the major argument of this section. There seems to be a collection of diverse arguments with different experiments. It may be better to highlight the overall goal of this section and the reason why these experiments are conducted in the very beginning before moving into more detailed experiments. \n\n3. Regarding the \"Rank Efficiency\" issue in Section 4, is it possible to have some more rigorous formulations in addition to statements only, say a theorem? \n\n4. Regarding citations and references, the OOD issue of DPO is also discussed in [1], which may be a good one to refer to. Also, the citation format is very strange. For example, many cited papers in the reference do not have any publication venues.\n\n[1] Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study, S. Xu et al., ICML 2024. https://arxiv.org/abs/2404.10719", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Why do you choose DPO, IPO, and SLiC-HF? Why not some other variants, say ORPO? Although I do agree that the current selections can be sufficiently representative, some discussions on why they are prioritized can be appreciated.**\n\nWe chose these versions of the DAAs as they were the most extensively studied in prior literature and used in practice at the time of our writing. We agree with the reviewer that it would be useful to expand the algorithm selection, which we could not do due to limited resources. \nThe ORPO algorithm specifically does not use a reference SFT model, but directly aligns the base model with the feedback data. This makes it harder to fit in the same performance-divergence framework. \n\n**The presentation of Section 4 can be possibly improved. With its current presentation, it is difficult for a reader to understand the major argument of this section. There seems to be a collection of diverse arguments with different experiments. It may be better to highlight the overall goal of this section and the reason why these experiments are conducted in the very beginning before moving into more detailed experiments.**\n\nWe believe the reviewer might be referring to Section 3. The goal of that section is to study the potential causes and dynamics of over-optimization, i.e. training objective (type of DAAA), capacity (model size), spurious correlations (length exploitation experiments) etc.. We will add language explaining this at the beginning and how each experiment fits within that framework.\n\n**Regarding the \"Rank Efficiency\" issue in Section 4, is it possible to have some more rigorous formulations in addition to statements only, say a theorem?**\n\n\nYes, we will present a more rigorous construction in final draft, which we will outline here. Consider the “query” vectors from section 4, which select the “win” and “loss” prompt-response pairs with a value of +1 or -1 from the prompt-response space, $X \\times Y$. As shown in the Appendix of Hejna et al. [1]  there are two sufficient conditions for the rank deficiency issue:\n\n1.  if the null space of the matrix formed by these query vectors $Q \\in \\mathbb{R}^{N \\times |X \\times Y|}$ is non-trivial on the support of the preference dataset (i.e. there exists a vector $v \\in N(Q)$ such that $v(x,y) \\ne 0$\n\n2. If for every prompt $x$, there is a response $y$ not included in the preference dataset.\n\nThe first condition is easily satisfied when the preference dataset contains any disagreeing preferences. The second is easily satisfied when the size of the preference dataset is smaller than the prompt response space, which is almost always the in practice as the prompt-response space is exponential in length. A trivial example of this is if the prompt space is empty and the response space has three tokens (a, b, c). If the preference dataset is (a < b, b > a) then the DAA loss is minimized by any policy that places equal probability on a and b, even if the probability of c is highest.\n\n\n**Regarding citations and references, the OOD issue of DPO is also discussed in [1], which may be a good one to refer to. Also, the citation format is very strange. For example, many cited papers in the reference do not have any publication venues.**\n\nThank you for bringing this to our attention, we will add the citation and rectify the formatting in our updated version!\n\n[1] Contrastive Preference Learning: Learning from Human Feedback without RL, Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W. Bradley Knox, Dorsa Sadigh", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QKQcDsxPsF", "reviewer": "Reviewer_TztD", "summary": "This paper analyse overoptimisation in the context of direct alignment algorithms (DAA). They show that even when no explicit reward model is being optimised against, a similar phenomena as Gao et al. is shown, where as KL budget increases \"gold\" reward increases and then decreases. This phenomena is shown across model sizes and different DAAs (DPO, IPO, SLiC). They fit scaling curves to this phenomena and find a similar style of fit to Gao et al.. This phenomena is then analysed from various angles, and the authors show that at low KL budgets best performance is reached early in training; length-correct adjusts the KL-winrate pareto frontier but doesn't mitigate overoptimisation; model training statistics aren't useful for predicting downstream performance. They also present some theoretical analysis and a toy MDP which demonstrates why reward exploitation can occur in DAAs, showing that lack of support in the sequence space combined with DAA's non-convex optimisation target can lead to DAAs placing high probability on OOD (and hence low-reward) sequences.\n\nL. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. Interna- tional Conference on machine Learning, 2023.", "questions": "My questions have been described in the weaknesses section.", "limitations": "The authors discuss limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper's analysis is interesting and novel - while overoptimisation has been demonstrated in online RL preference learning algorithms, this demonstration of the effect in DAAs hasn't been presented before to my knowledge.\n\nThe quality of the analysis of the paper is high - they discuss their results neutrally and clearly, and the results are somewhat general. The additional analysis answers many obvious questions that come to mind, which is beneficial; the authors have clearly investigated this phenomena thoroughly.\n\nThe paper is well-presented and easy to understand.\n\nThe significance of the work is reasonably high - DAAs are a very popular class of preference learning algorithm, and this work has demonstrated a previously unknown limitation of these algorithms, and provided much additional analysis and understanding into how they work and where they can break, which will be useful for the community going forward in improving upon these algorithms and understanding their limitations.", "weaknesses": "## Large points\n\nI think the main weaknesses with the paper are three-fold:\n* All analysis is only performed on the TL;DR summarisation dataset, which is somewhat different from Gao et al. and the setting these preference learning algorithms are generaly used in (dialogue and instruction-following). While reproducing the whole analysis on another dataset is too much to ask, testing out some of the core hypotheses, or reproducing some of figure 1, on a different dataset (e.g. alpaca farm, or anthropic HH) would increase the robustness and generality of the results.\n* Using GPT-4 winrate as the \"gold\" reward or reference output also makes the results less general. The preference distribution that produced the dataset the DAAs were trained on is not the same one being used to evaluate here. I would expect the results to hold up in the setting where these two distributions are the same, but it would be beneficial to have some empirical validation of that. This would likely mean mimicing the setting of Gao et al, and applying DPO on a preference distribution generated from an accessible gold reward function (for example, the data of Coste et al. is available and may be suitable, or alpaca farm gpt4 preference data could also work). The same reward function can then be used to produce the y-axis of the plots, which might even produce cleaner trends. This would more accurately mirror the real-world setting where human preferences would be used for both training and evaluation.\n* While the dicussion in section 4 is useful, it would be beneficial to have a better explanation of why this phenomena happens in DAAs, as it's still not immediately intuitive for me.\n\n## Small points\n\n* it would be useful for the axes in figure 1 to be unified across all the subplots, so it is easier to compare between algorithms. Similarly for the y-axis in figure 2.\n* some of the text in the figures (especially figure 7) is quite small.\n\n## Summary\n\nI think the paper is worthy of acceptance as is, and I'm reccomending accept. If one of the points mentioned in the Large Points section above was addressed thoroughly I would consider raising my score to a strong accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the kind words!\n\n**All analysis is only performed on the TL;DR summarisation dataset, which is somewhat different from Gao et al. and the setting these preference learning algorithms are generaly used in (dialogue and instruction-following)...**\n\nWe are working on replicating our main experiment with the Gemma-2 2B model on the Anthropic Helpful and Harmless dataset. Our initial results suggest similar over-optimization dynamics to the Pythia TL;DR experiments presented in the paper. We will include the additional experiment in our revised submission.\n\n**Using GPT-4 winrate as the \"gold\" reward or reference output also makes the results less general. The preference distribution that produced the dataset the DAAs were trained on is not the same one being used to evaluate here…**\n\nWe do agree that the use of GPT-4 win rates as an evaluation metric makes the results more noisy, but we believe this is a more realistic scenario. \nThe TL;DR dataset is based on human rater feedback. In original DPO work [3] the authors carried out a human evaluation study on the TL;DR summarization task and found GPT-4 - human agreement to be 67-70%, which was higher than intra-human annotator agreement of 65%. \n\n**While the dicussion in section 4 is useful, it would be beneficial to have a better explanation of why this phenomena happens in DAAs, as it's still not immediately intuitive for me.**\n\nWe will expand section 4 to try and make it more clear given additional space for the final paper.\nFundamentally, DAAs can be viewed as performing a type of generalized linear model, modulated by a convex regularization function $g$, where the number of datapoints $N$ is much much smaller than the number of features (prompt-response space). Normally, one would apply some type of regularization (like L2 in ridge regression) to make the problem strictly convex, but DAAs don’t do that. Instead, we are left with a vastly under-constrained problem. Because of this “rank-deficiency”, there are a large number of solutions that place probability on out-of-distribution sequences. Thus, DAA methods can easily start to converge to one of these solutions during training. \nA simple construction is as follows: consider a setting where the only prompt space is empty (ie no prompt) and the response is one of three tokens: (a, b, c). If my preference dataset does not include c and contains at least one conflicting preference (a > b and b < a), then the minima of a DAA will just ensure that the log-ratio of a and b equal some finite value. Since only the ratio of a and b is enforced at the optima, we can place any amount of probability mass on response c. \nWe have also detailed sufficient conditions for it to occur in response to Reviewer bFR8. \nWe would appreciate it if the reviewer could let us know what we could do to help make this section more understandable. \n\n\n\n[1] Direct Preference Optimization: Your Language Model is Secretly a Reward Model, Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for the useful comments!

1. We are working to expand the experiments in our work using the Gemma 2 2B model on the Anthropic Helpful and Homeless Dataset. We have attached our preliminary results here, which show the same general effects as our Pythia TL;DR experiments. We will include the final set of experiments in our camera-ready submissions. 

2. While we do not use the formulation of gold reward models, we use win rates as evaluated by GPT-4, which we believe is a more realistic evaluation. Prior work [1] has shown that GPT-4 achieves higher average agreement with human annotators than they do between themselves on the TL;DR summarization task. 
We believe this makes our setting and results a good proxy evaluation for real human preference, which a gold reward model was designed to approximate in the original Gao et. al. work [2]. 

3. We have attached the additionally requested analysis and figures on individual model fits and length correlations, which will also be included in our final camera-ready submission. 

4. We will expand on our theoretical formulation of Section 4  as outlined in individual reviewer responses. 

[1] Direct Preference Optimization: Your Language Model is Secretly a Reward Model, Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn

[2] Scaling Laws for Reward Model Overoptimization, Leo Gao, John Schulman, Jacob Hilton

Meta-review: This paper studies Direct Alignment Algorithms (DAAs), revealing that similar to traditional RLHF methods, DAAs suffer from degradation due to over-optimization, especially at higher KL-budgets. The authors also examine the consequences of reward over-optimization in DAAs across various objectives, training regimes, and model scales. The results seem to be interesting for a broad audience of NeurIPS. One issue to be careful about, as pointed out by "Reviewer TztD": there is a potential distribution shift when evaluation using GPT-4. I strongly recommend that the authors add a discussion on this point in the paper.
Decision: Accept (poster)