Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms

Authors: rafael  rafailov
yaswanth  chittepu
ryan  park
harshit  sikchi
joey  hejna
w bradley knox
chelsea  finn
scott  niekum
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 22:14:56

Abstract: Reinforcement Learning from Human Feedback (RLHF)has been crucial to the recent success of Large Language Models (LLMs), however it is often a complex and brittle process. In the classical RLHF framework, a reward model is first trained to represent human preferences, which is in turn used by an online reinforcement learning (RL) algorithm to optimized the LLM. A prominent issue with such methods is reward over-optimization or reward hacking, where the performance as measured by the learned proxy reward model increases, but the true model quality plateaus or even deteriorates. Direct Alignment Algorithms (DDAs), such as Direct Preference Optimization (DPO) have emerged as alternatives to the classical RLHF pipeline. However, despite not training a separate proxy reward model or using RL, they still commonly deteriorate from over-optimization. While the so-called reward hacking phenomenon is not well-defined for DAAs, we still uncover similar trends: at higher KL-budgets, DAA algorithms exhibit similar degradation patters to their classic RLHF counterparts. In particular, we find that DAA methods deteriorate not only across a wide range of KL-budgets, but also often before even a single epoch of the dataset is completed. Through extensive empirical experimentation this work formulates the reward over-optimization or hacking problem for DAAs and explores its consequences across objectives, training regimes, and model scales.

Main paper: 
 introduction: 
 Recent advancements in Large Language Models (LLMs) have broadened their capabilities significantly, enabling applications in code generation, mathematical reasoning, tool use, and interactive communication. These improvements have popularized LLMs across various domains. Reinforcement Learning from Human Feedback (RLHF) has been instrumental in these advances and is now integral to sophisticated LLM training regimes [10, 55] . Before alignment, LLMs, trained on vast text corpses to predict subsequent tokens [45, 8] are often unwieldy and hard to use. Today, leading LLMs incorporate variants of the RLHF framework [14, 69, 36] to align them with human intent, which generally involves a multi-stage process. Specifically, users evaluate model responses to assorted prompts in order to train a reward model that encapsulates human preferences [10, 55, 72, 5, 62] . Then, the refined LLM maximizes the expected learned reward function using a reinforcement learning (RL) algorithm [50, 1, 65] . Despite its efficacy, this procedure is complex and computationally intensive, particularly in its latter stages.Goodhart's Law [25, 11] , that "when a measure becomes a target, it ceases to be a good measure", has often been cited as a core shortcoming of RLHF. Standard RLHF methods optimize a learned, but imperfect reward function which ends up amplifying the reward model's shortcomings. Empirically, this phenomenon was first extensively characterized by Gao et al. [21] , who coined the term "reward over-optimization", and has been seen consistently in recent findings [62, 16, 14] . While reward over-optimization has been studied in the context of the aforementioned RLHF procedure, recent contemporary methods for aligning LLMs circumvent the reward learning procedure, necessitating a new characterization of the over-optimization phenomena.This new broad class of algorithms, which we refer to as Direct Alignment Algorithms (DAAs), bypass the traditional RLHF pipeline by re-parameterizing the reward model directly through the optimal policy derived during the reinforcement learning phase. DAA methods, like Direct Preference Optimization [46] , have gained popularity [14, 28] as they often reduce computational demands. Yet, despite not fitting a reward function, DAAs still exhibit over-optimization trends similar to those of traditional RLHF methods using a learned reward function. In some sense, this is puzzling: DAAs can be viewed as simply learning a reward function with supervised learning from which the optimal policy is deterministically mapped, however more seems to be at play than simple supervised learning.In this work, we investigate the over-fitting phenomena present in DAA algorithms through extensive experimentation. First, we unify a number of different recent methods [46, 68, 4] under the DAA framework. Then, across different model scales and hyper-parameters, we show that DAAs exhibit a type of reward over-optimization consistent with that previously observed in RLHF [21] . Specifically, we find that at different KL-divergence budgets DAAs exhibit degradation patterns similar to those found in RLHF. Interestingly, we also find that performance within a single epoch is not always as consistent as expected for DAAs. Finally, we explain why this happens by appealing to the under-constrained nature of the optimization problem used in DAAs.In the classical RLHF pipeline, the RL objective (eq. ( 3)) is optimized using on-policy samples from π θ . This means that the reward function is continuously queried using unseen model samples which are potentially out-of-distribution. Beyond the support of the reward modeling distribution, r ϕ may assign high rewards to sub-par responses, leading the policy to believe it is doing well when it may not be. While the KL-regularization term is designed to prevent the model from drifting too far out of distribution, this term alone has proven inadequate to prevent reward hacking [21] .2. Reward Mis-specification. Learned reward functions may exhibit spurious correlations that cause them to prefer unintended behaviors. While this issue is not at the forefront of LLM research, it is known to be pervasive in RL [43, 34] . Most efforts to address these problems exist at the intersection of robustness and offline RL literature [13, 67, 16] and use measures of epistemic uncertainty to penalize the predicted reward. 
preliminaries: 
 In this section, we first outline the core components of the standard RLHF pipeline [72, 55, 5, 41] ). Then, we examine prior literature to characterize the reward over-optimization exhibited by standard RLHF methods. Finally, we provide a unifying view of direct alignment algorithms (DAAs) which will guide our analysis of their training dynamics in the next section.The standard RLHF pipeline consists of three distinct stages with the goal of aligning the LLM with human preferences.Supervised Fine Tuning (SFT): First, a dataset of prompts x and high-quality answers y are used to train an LLM for instruction following via maximum likelihood estimation over next-tokens. We refer to the resultant model as π SFT (y|x) and consider the entire prompt and answer strings to be single variables.Reward Modeling: Second, the SFT model π SFT (y|x) is used to learn a reward function over human preferences. Specifically, the SFT model is queried to produce pairs of answers (y 1 , y 2 ) ∼ π SFT (y|x), for every prompt x in a dataset. Then, users select their preferred answers, resulting in ranking y w ≻ y l | x where y w and y l are the preferred and dispreferred answers respectively. Typically, user rankings are assumed to be distributed according to the Bradley-Terry (BT) model [7] p(y 1 ≻ y 2 | x) = exp (r(x, y 1 )) exp (r(x, y 1 )) + exp (r(x, y 2 )) = σ(r(x, y 1 ) -r(x, y 2 ))where the preference distribution p results from an unobserved latent reward r(x, y), and σ is the logistic function. Given this model and a dataset of rankings, denoted D = x (i) , yw , y(i) l N i=1 , wecan train a parameterized model r ϕ (x, y) to predict the unobserved reward using maximum likelihood estimation. This yields the following loss function,EQUATIONReinforcement Learning (RL): The final stage of the standard RLHF pipeline uses the learned reward model r ϕ (x, y) to update the LLM π θ with an on-policy RL algorithm like PPO [50] , optimizing the model to provide responses more preferred by human raters. The most common objective isEQUATIONwhich enforces a Kullback-Leibler (KL) divergence penalty with a reference distribution π ref (y|x) (usually taken to be π SFT (y|x)) to prevent the LLM π θ from straying too far from its initialization. Thus, the hyper-parameter β directly trades off exploiting the reward function and deviating from π ref (y|x).Unfortunately, repeating the above procedure without careful tuning of the RL phase can lead to disastrous performance. This is because in the context of RLHF the LLM policy is optimizing the surrogate reward estimate r ϕ (x, y) and not the true reward function as is often the case in other domains. Thus, prior works have observed that while the LLM's expected reward according to eq. ( 3) increases the actual quality of the model's outputs can decrease [54, 43, 9, 34] . This particular instantiation of the reward exploitation or hacking problem [3] is often referred to as reward "overoptimization" in RLHF literature and has been studied empirically in both controlled experiments [21] and user studies [14] . There are two prevailing explanations for why this phenomenon occurs.Due to its complex multi-step nature, recent works have sought alternatives to the classic RLHF pipeline. A new class of algorithms, which we broadly classify as Direct Alignment Algorithms (DAAs), directly update the LLM's policy π θ using user feedback instead of fitting a reward function to it and then employing an RL algorithm. Perhaps the most known example is Direct Preference Optimization (DPO). DPO, as well as other DAAs, are derived using the closed form solution to the RLHF objective in eq. ( 3) [71] , π * (y|x) ∝ π ref (y|x)e r(x,y)/β , where r(x, y) is the ground-truth reward. By isolating r(x, y) in this relationship and substituting it into the reward optimization objective in eq. ( 2), we arrive at a general objective that allows us to train the LLM directly using feedback data:EQUATIONwhere g is a convex loss function. Using g(x) = -log σ(x) coincides with the standard Bradley-Terry model and the original DPO objective. Other methods choose different loss functions: IPO [4] uses the quadratic objective g(x) = (x -1) 2 and SLiC-HF [68, 38] uses the hinge loss g(x) = max(0, 1 -x). Additional objectives were also considered in [59] , but due to limited computational resources, we focus on the three objectives outlined above.Crucially, the DAA approach allows us to recover the optimal policy using a straightforward classification loss without the need for learning a reward function, on-policy sampling, or RL, which can be notoriously difficult to tune and computationally expensive. Because of this, DAAs have emerged as a popular alternative. However, just like classical RLHF methods, DAAs exhibit strong over-fitting and even reward-hacking like behaviors. For example, Park et al. [44] show that LLMs trained with DPO generate responses with increasing length throughout the course of training, but do not improve in ground-truth win-rate after a certain point. Since DAAs do not explicitly learn a reward function, it is unclear how "reward-overoptimization" fits into the picture. In this work, we aim to shed some light on this phenomenon in DAAs.In this stage, preferences extracted from trajectories are employed to align the policy. Notably, we have only one preference available: (s 1 , a 1 , s 3 , a 1 , s 9 , a 0 , s ∞ ) ≻ (s 1 , a 0 , s 2 , a 0 , s 5 , a 0 , s ∞ ). This deliberate constraint exaggerates a scenario with limited data, enabling us to gauge the probability mass allocated to out-of-distribution (OOD) trajectories under such conditions. Insights garnered from this exaggerated low-data scenario hold relevance for Large Language Model (LLM) settings where preference datasets used for alignment are notably smaller compared to the scale of LLM models deployed.We utilize a Recurrent Neural Network (RNN) policy to navigate through the MDP, facilitating a closer resemblance to real-world language modeling scenarios.Subsequently, we explore three distinct direct alignment loss functions: Direct Preference Optimization (DPO) [46] , Identity Preference Optimization (IPO) [4] , and Sequence Likelihood Calibration (SLiC) [68] . Additionally, we investigate how the selection of the KL penalty coefficient β influences the distribution of probability mass on OOD trajectories. This exploration encompasses three values of β: (0.01, 0.1, 0.5).In general, the plots illustrate that Direct Alignment Algorithms (DAAs) tend to allocate a significant proportion of the probability mass to out-of-distribution (OOD) trajectories during the alignment process. While Figure 9 may suggest that Direct Preference Optimization (DPO) can retain a substantial amount of probability mass on the selected trajectory in the preference dataset, it's noteworthy that the plots for DPO exhibit considerable noise. To provide further insight, Figure 18 displays the plots resulting from three additional repetitions of the DPO experiment. Similar noisy trends were also observed in the experiments for IPO and SLiC. This elucidates the unconstrained nature of the DPO problem: multiple solutions exist for the DPO loss, each distributing varying amounts of probability mass to OOD trajectories. In the experiments with IPO and SLiC, it's also observed that similar to DPO, the probability mass allocated to in-distribution trajectories can diminish substantially over the course of training. Notably, the probability mass, in our experiments, becomes concentrated on a select few out-of-distribution trajectories. Moreover, consistent trends are discernible across various values of β. The results of our experiments with the Toy-MDP can be found in the following figures 12, 9, 15, 13, 10, 16, 14, 11, 17 . The trajectories are listed in the legends for the plots, as a sequence of state, action pairs. . The trajectories are listed in the legends for the plots, as a sequence of state, action pairs. .  
empirical analysis of overoptimization in daas: 
 First, we examine the over-optimization problem in DAAs and compare it to those observed in traditional RLHF methods. All our experiments are carried out using the Reddit TL;DR summarization dataset [55] and the Pythia family of Large Language Models [6] . Additional plots illustrating similar over-optimization trends for Direct Alignment Algorithms on the Gemma2-2b model [61] and the Anthropic Helpfulness-Harmlessness dataset [5] are provided in Appendix FIn our first set of experiments, we evaluate the reward model over-optimization phenomenon. We evaluate three training objectives DPO, IPO, and SLiC using seven β parameters, representing different KL budgets at three model sizes -1B, 2.8B, and 6.9B. Our main results are shown in Fig. 1 which presents results for different configurations after 1 epoch of training (row 1) and including 4 uniform intermediate checkpoints (row 2). We include additional results on the training dynamics in Fig. 2 , which shows win rates and KL bounds for intra-epoch training. We present our findings below.Model Over-Optimization: We see clear over-optimization for all objectives as performance exhibits a hump-shaped pattern, where an additional increase in the KL budget leads to decreasing model performance. Moreover in Fig. 2 we observe similar intra-epoch training dynamics patterns as configurations with wider KL budgets achieve their best performance after training on only 25% of the data, after which performance starts decreasing in conjunction with increasing KL divergence metrics. Effect of Training Objective: In the IPO work [4] the authors present theoretical arguments that due to the monotone sigmoid objective in the DPO formulation, the KL constraint is not effectively enforced and propose the quadratic fixed-margin loss as an alternative. Across all objectives, there are clear dependencies between the β parameter and the corresponding KL achieved at the end of training. While DPO and SLiC exhibit similar performance, IPO indeed seems to be less prone to over-optimization and in general, achieve lower KLs under the same constraint. Our observations with IPO also align with prior works in preference-based RL and imitation learning where imposing a fixed margin led to more stable and performant methods [48, 51] .The results also show a strong parameter count scaling effect. The Pythia 1B model achieves low performance under the same set of constraints it reaches much higher KL values, while almost immediately exhibiting signs of over-optimization. This behavior holds under all three objectives. At larger scales, the 6.9B Pythia model tends to exhibit more win-rate -KL trade-offs and be less prone to over-optimization, with both models significantly outperforming the 1B model. In the case of the IPO objective, the 6.9B also exhibits significantly better control over the KL objective and shows little to no over-optimization behavior.Given we have established a framework for evaluating over-optimization in DAAs and empirically validated it (section 3.1), we now develop scaling laws for this phenomenon. Previous work in classical RLHF has established such scaling laws for reward model scores as a function of the KL divergence between initial and optimized policies [21] . The relevant functional of the reward R(d) isEQUATIONwhere α, β are constants dependent on the size of the reward model dataset and parameter count, and d = D KL (π||π ref ). As DAAs do not train a proxy reward model, we treat GPT4 winrates over dataset completions as a proxy for gold reward. Somewhat surprisingly, we find that this scaling law accurately relates d and winrates for DAAs. Compared to a quadratic fit between D KL (π||π ref ) and winrates, this scaling law halves the RMSE. It is worth noting, however, that a quadratic fit between d and winrates yields a similar error compared to Equation 5.Figure 3 : Left: KL budget versus win-rates (over dataset human answer) with and without lengthregularization [44] . While including a length correction in the optimization objective changes the KL-win-rate Pareto Frontier, it does not alleviate reward over-optimization and might even exacerbate it. Right: Scaling behavior for length extrapolation -smaller capacity models (either by size or KL budget) extrapolate more strongly on simpler features such as length.Prior work [44] has shown that the DPO algorithm is prone to length exploitation as it amplifies verbosity biases in preference datasets. Here we show that length is not the only dimension on which exploitation can occur. Our experimental results are shown in Fig. 3 . On the left, we show results for the 2.8B Pythia model with standard training plus the length-regularization approach from [44] .Both approaches suffer from over-optimization, but the dynamics differ depending on the KL budget. Moreover, even though the regularized model achieves higher win rates on a length-correct basis, it under-performs the model trained with the standard objective in the lower KL constraint region.Recent work [27] has also shown that DAAs prioritize features of the data based on their complexity and prevalence (with length a clear example of human datasets). [44] further showed that models trained with the DPO algorithm extrapolate significantly based on length. We extend this analysis in Fig, 3 (right). We consider a linear regression of the formEQUATIONwhere x (i) are held-out prompts and y (i) are samples from the corresponding model between the DPO implicit reward and length. We fit a different regression for each model size and checkpoint and plot the corresponding R 2 values. We observe two main effects; first, there is a clear scaling law behavior. Weaker models extrapolate across the simple length feature to a much higher degree than stronger ones. This is especially clear when comparing the behavior of the Pythia 1B versus the 2.8B and 6.9B models. Second, we see significant effects based on the KL budget -under a smaller budget all model sizes exhibit higher extrapolation behavior. Based on these results we formulate the hypothesis that under limited capacity, either from model capability or limited KL budgets, the model will extrapolate more strongly based on simpler features, which can lead to OOD issues.Prior works have measured reward model quality in ranking settings by classification accuracy. We evaluate the relationship between the DAA implicit reward model accuracy and policy performance in Figure 4 . The DPO and SLiC algorithms exhibit little to no correlation between reward model accuracy and downstream model performance. The IPO model shows a weak positive relationship, but upon further examination, this is entirely due to model size scaling -stronger models both fit the data better and produce better generations as well, however within each particular model size, there is no discernible relationship between the DAA implicit reward accuracy and the actual policy performance. Similar observations hold when comparing the empirical DAA loss with model performance, which is contrary to observations in supervised pre-training and instruction tuning [30] . A number of recent works have observed that the implicit DAA rewards of both preferred and dis-preferred responses decrease during training, which may be counter-intuitive. In [47] the authors make a counter-point that in offline training of DAAs π ref is usually pre-trained with SFT on the preferred response and thusEQUATIONwhere p D (y w |x) is the dataset distribution of preferred answers. That is the expected implicit reward represents a forward KL divergence between the reference policy and the optimization policy, thus it is expected to be negative and decrease with training as the optimization model moves away from the reference. In this section, we study whether this empirical phenomenon presents a challenge for DAA learning. Similar to Fig. 1 we plot the win rates against the square-root-transformed (negative) expected implicit reward of the preferred response (evaluated on a held-out evaluation dataset), which as stated above approximates the (square-root-transformed) forward KLD KL π ref (y|x) || π θ (y | x).Results are included in Fig. 5 , which follow closely the pattern in Fig. 1 with performance initially increasing before it starts dipping down after a certain threshold. This indicates that under the standard DAA training pipeline decreasing likelihoods are not necessarily an issue for performance, and are even necessary for improvement, but exhibit non-linear over-optimization dynamics. 
reward exploitation in direct alignment algorithms: 
 While the phenomena observed in the previous section echo those observed in classical RLHF, their underlying causes may be distinct. Reward over-optimization in classical RLHF is largely attributed to querying a proxy reward function that is potentially OOD, while DAAs do not train a separate reward model. Instead, DAAs are generally understood as fitting an "implicit" reward model to preference data with the parameterization r θ (x, y) = β log π θ (y|x) πref(y|x) using the objective in eq. ( 2). Therefore, the OOD behavior of the policy is inextricably linked to the OOD behavior of the implicit reward model. We demonstrate below that the reward modeling objective used is heavily under-constrained, allowing for a potentially large number of solutions that can place weight on OOD responses. This is especially problematic for DAAs which deterministically map the optimal policy from the "implicit" reward.Rank Deficiency with Finite Preferences. In DAAs, the language modeling problem is treated as a contextual bandit. However, the space of possible prompts x ∈ X and answers y ∈ Y are both exponentially large in sequence length. However, as highlighted by Tang et al. [59] , DAAs often assume full support of the reference distribution when mapping from the implicit reward to the optimal policy π by eq. ( 10). However, in practice such coverage is impossible. Instead, preference datasets cover a minuscule portion of the prompt-response space. Unfortunately, as DAA objectives are not strictly convex, their loss functions (eq. ( 4)) can have multiple global optimas, which may be undesirable. We demonstrate this below, using the regression interpretation from Hejna et al. [23] .First, we re-write the DAA objective from eq. ( 4) using vectors in the prompt-response space X × Y. Each preference query in the comparison dataset can be written as difference between indicator vectors, specifically q i = 1{(x, y) = (x (i) , y(i) w )} -1{(x, y) = (x (i) , y (i)l )}. This "query" vector simply selects the comparison from the prompt response space, with the entree corresponding to (x, y w ) being +1 and the entree corresponding to (x, y l ) being -1. Similarly, we can consider the learned policy to be a vector log π -log π ref ∈ X × Y, to which the distributional constraint also applies in practice. Our generalized DAA loss function can then be re-written asL DAA (π θ , D) = |D| i=1 g βq ⊤ i (log π(y|x) -log π ref (y|x)) , where q i [x, y] =      1 if (x, y) = (x (i) , y (i) w ) -1 if (x, y) = (x (i) , y (i) l ) 0 otherwisewith finite data. Choosing g to be the negative log sigmoid above recovers DPO with finite preferences, but also logistic regression with a data matrix Q of shape |D| by |X × Y| constructed by stacking the aforementioned query vectors q. As |X × Y| >> |D|, this matrix is likely to have a non-trivial null space, making the problem not strictly convex. Thus, there are many possible policies π that can achieve the same optima, some of which will place a high weight on out-of-distribution responses due to the distributional constraint of policy [23, 70] . To demonstrate this, we formalize the construction below.Proposition 1 (Adapted from Hejna et al. [23] ) Let S be the set of win-or-lose prompt-response vectors (x, y) in D. Provided:1. The intersection of the null space of Q, N (Q), and the span of S, span(S), is non-trivial.2. For every x there exists a response y OOD ∈ Y that is not in the data, (x, y OOD ) ̸ = S.Then, there are infinite number of minima to eq. (4) which place weight on out-of-distribution responses y. . Proof. Let π be the minima of the DAA loss function. Choose a vector u such that u ∈ N (Q), u ∈ span(S), and u has at least one negative component. Modifying the log policy vector as log π + u will not affect the DAA loss, as u is in the null space of Q, but the log-probability of the policy will decrease for least one prompt-response pair in S by construction. However, e log π+u may not integrate to one. To fix this, we can construct a second vector v ∈ N (Q) using the y OOD at each x such that e log π+u+v integrates to one. For more details, we refer the reader to Hejna et al. [23] Appendix A.3.The second constraint of proposition 1 is often trivially satisfied by the dimension of the response space as we are unlikely to see every response to a prompt. The first constraint is harder, but can be satisfied by conflicting preferences. A trivial example which satisfies these constraints is a simple MDP in which there is only a single state (or prompt x), but three possible actions (or responses), y 1 , y 2 , and y 3 . If we construct the preference dataset D = {(y 1 ≻ y 2 ), (y 2 ≻ y 1 )}, omitting x for brevity, then we satisfy the above conditions: the null space of Q is non trivial in span of y 1 and y 2 and there is an out-of-distribution action y 3 . In this setting, the DPO loss is minimized by both π(y|x) = (0.5, 0.5, 0) and π(y|x) = (0.0, 0.0, 1.0). In fact, it is minimized by infinitely many policies which place equal weight on y 1 and y 2 . To demonstrate this effect in higher dimensions across a number of different DAA methods, we conduct experiments in a Toy MDP which bears resemblance to the language modeling setting.Understanding OOD behavior for DAA algorithms with a Toy MDP: To illustrate that DAA algorithms, in general and not an artifact of training LLM's, end up placing probability mass on OOD sequences during training we design a simple Tree MDP (shown in Figure 6 ) to mimic the token-level MDP in LLMs. We use a dataset containing a single preference between two trajectories and follow the standard procedure of running SFT on preferred responses before updating an RNN policy using a DAA. Figure 7 shows that even in this simple setup, popular DAAs (DPO/IPO/SLiC) end up extrapolating incorrectly out of distribution revealing a fundamental shortcoming. Unlike in standard RLHF, the non-strict convexity of the reward function in DAAs ends up directly affecting the policy. Detailed experimental details can be found in Appendix E.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We provide detailed guidance on reproducibility by specifying all datasets, code, and hyperparameters used in this work. 
related work: 
 Broadly, over-optimization has been a widely studied phenomenon across different settings [60, 18] . Over-fitting can be characterized as over-optimization in the supervised learning setting [39, 32] , which can harm generalization [19, 12, 24] or lead to susceptibility to adversarial attacks [56, 37, 15] . Reward hacking in reinforcement learning (RL) [54] , where an agent maximizes its reward through behavior that deviates from the intended goal, can be viewed as a different type of over-optimization, commonly observed in prior work [43, 3, 22] . Figure 6 : An illustration of the Tree MDP. At each state, we can choose one of 3 actions (a 0 , a 1 , a 2 ), which deterministically maps to the next state. Furthermore, all the leaf nodes in this tree MDP, transition to the terminal absorbing state s ∞ , irrespective of the chosen actionWe study over-optimization in the context of aligning LLMs with human feedback, for which the most common approach is RLHF as outlined in section 2.1. Similar RLHF techniques were originally pioneered for control [31, 2, 10] . Standard RLHF methods suffer from both potential over-fitting of the reward function and reward exploitation by the RL algorithm. Several works have considered how to reduce over-fitting or increase the robustness of learned reward functions using ensembles [13, 67, 16] or data smoothing [70] . Other approaches, like Moskovitz et al. [40] consider how reward exploitation can be reduced by using different optimization techniques in the RL stage. Much of this work is motivated by Gao et al. [21] , which first characterized and provided scaling laws for over-optimization in RLHF.Unlike Gao et al. [21] , we consider the overoptimization problem in DAAs, which differs significantly from the standard RLHF pipeline. Different DAAs have been derived theoretically [47, 46, 68, 4, 64] , and applied to problems beyond language modeling like image generation [63] and control [23] . In all of these scenarios, over-optimization problems have persisted. Park et al. [44] show that DAAs commonly over-fit to length and the expense of performance, which has been linked to inherent bias in training data [53, 29] . Other works have tried to allow DAAs to use more types of data like demonstrations [49] or ratings [17] to get better performance. Recently, incorporating online data has proven critical to improving performance [66, 26, 57] . Concurrent to our work, Tang et al. [58] study the differences between offline DAAs and standard RLHF methods. Unlike us, they focus on comparisons with online sampling whereas we focus on the purely offline setting. 
conclusion: 
 In this work, we present an analysis of the over-optimization problem in Direct Alignment Algorithms. Through extensive experimentation on different algorithms (DPO, IPO, SLIC) and at different model scales (1B, 2.8B, 6.9B), we observe consistent over-optimization trends at different KL-divergence budgets. While our analysis is a first step, it is not a complete picture of understanding the overoptimization phenomena. More work can be done characterizing this effect at larger model scales, which we were unable to do due to computational limitations. Nevertheless, we believe our work sheds light on important problems in Direct Alignment Algorithms that can spur future research. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: We use public models that are fine-tuned for alignment on open-source datasets. Our models do not contribute any additional risk over the base models as we are explicitly training for alignment. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The pretrained models in this work come from the Pythia family all of which are classified under Apache License https://huggingface.co/EleutherAI/pythia-2.8b/tree/main. The TL;DR comparison dataset used in this work uses a modified MIT License https://github.com/openai/summarize-from-feedback/blob/master/LICENSE. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: We use open-source pretrained models and provide details to reproduce our fine-tuning experiments. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Our discussion highlights a number of issues with direct alignment algorithms used widely as means to align to human values. This work has mostly focused on pointing out those issues along with a theoretical underpinning of the issue but does not provide a way to resolve these issues. We still assume an underlying model of human preferences, which is an ongoing research area as no model is perfect in explaining the ways humans give preferences. Our work aims to drive the push towards better alignment algorithms that do not overoptimize and generate models that are safe to be deployed in our society. We believe only through understanding and demonstrating the shortcomings of current methods we can develop better alignment methods.We largely follow the DPO setup unless otherwise mentioned and build on their code (https://github.com/eric-mitchell/direct-preference-optimization) without changing any hyperparameters unless otherwise mentioned.For all DAA experiments, we used the curated OpenAI TL;DR dataset with 92K preferred-dispreferred summary completions [55] . Each prompt is a Reddit post belonging to one of several topic forums, with title/post metadata included. 256 prompts sampled from the held-out set are used for all evaluations (e.g. loss, accuracy, KL, winrates, length), with temperature 1.0 and max length 512.Model sizes include 1B, 2.8B, and 6.9B and were initialized from the base Pythia pre-trained weights. All models underwent supervised fine-tuning on TL;DR prior to direct alignment. Across all SFT and DAA runs, we used a batch size of 128 (8 gradient accumulation steps), and RMSProp with a learning rate of 0.5 × 10 -6 (linear warmup for 150 steps) for 1 epoch. 1B models were trained on 2 NVIDIA A40 GPUs, 2.8B models were trained on 4 NVIDIA A40 GPUs, and 6.9B models were trained on 4 NVIDIA A100 GPUs. All evaluations were computed with "gpt-4-turbo-2024-04-09" as judge, with random positional flips to avoid known bias.This appendix contains similar intra-epoch KL divergence and winrate evolution results as in Fig. 2 , across all model sizes. Reward over-optimization is well understood in the classical RLHF setting, with a consensus that is driven by two main components -using a proxy reward function that is trained on limited data and continuous querying with new, potentially OOD samples during PPO training. At first glance, none of these conditions hold in DAAs as we do not train a separate proxy reward model or generate new data during training. Therefore, understanding reward over-optimization in DAAs requires a new theory. We will base our analysis on [47] using the token-level MDP and corresponding (soft) Q-learning formulation. Consider the class of dense per-token reward functions r θ (x, y ≤i ), where y ≤i denotes the first i tokens of y, with sequence level-reward r θ (x, y) = |y| i=1 r θ (x, y ≤i ). This is a strictly more general class than the sparse reward function which returns a single score at the end of the sequence since we can set all intermediate rewards as 0. Within the framework of [47] given a DAA-trained policy π θ , there exists a dense per-token reward r θ , that minimizes the reward modeling objective in Eq. 2 and satisfy the below.The (soft) Bellman Equation holds:Q * (y i , (x, y <i )) = r(x, y ≤i ) + β log π ref (y i |(x, y <i )) + V * ((x, y ≤i )), if y i is not EOS r(x, y ≤i ) + β log π ref (y i |(x, y <i )), if y i is EOS (8)where V * is the corresponding soft-value function:EQUATIONthen the DAA policy π θ satisfies:EQUATIONin this interpretation, the LLM logits l θ [i] = Q * (y i , (x, y <i ))/β represent Q-values. With a direct substitution, we then haveQ * (y i , (x, y <i )) = r(x, y ≤i ) + β log π ref (y i |(x, y <i )) + β log yi∈|V |e Q * (y,(x,y<i))/β OOD bootstrapping (11) That is in this framework DAAs may suffer from the classical OOD bootstrapping issue in offline RL [20, 35, 33, 52] . In this case, even though the objective is trained fully offline we still effectively query the model on the values of unseen tokens. This interpretation also provides further insight into the effect of the β coefficient and the training dynamics. For small values of beta the estimateEQUATIONthat is smaller parameter values yield a more optimistic estimate, which results in a higher level of OOD bootstrapping. This interpretation would also explain the somewhat counter-intuitive results of section 3.4. While the implicit reward function can adequately fit and model the data, the resulting LLM might behave sub-optimally, due to OOD bootstrapping in the corresponding Q-value estimate.We have established that common DAA objectives allow for placing a high likelihood on OOD data. In practice, while one might expect the likelihood of preferred responses to increase during training, it has been observed that algorithms like DPO decrease the likelihood of both the preferred and dis-preferred responses [42] . In fact, this is expected from a max-entropy RL perspective [47] .Since the total probability mass must sum to one, the probability of OOD responses must increase during the course of training. A small amount of extrapolation may be necessary to reach the optimal policy, however, too much is potentially detrimental to performance. Because they are not adequately constrained to the reference distribution, current DAA objectives allow this to happen.To understand how DAAs allocate probability mass out of distribution, we use a toy Markov Decision Process (MDP), that mimics the LLM setting. The MDP is modeled as a tree, originating from a single start state, featuring deterministic transitions. The Toy MDP is illustrated in fig. 6 .The MDP is modeled as a tree, originating from a single start state. This configuration mirrors the token-level MDP in Direct Preference Optimization (DPO) [47] , or the scenario where both preferred and dispreferred responses are conditioned on the same prompt in the broader Large Language Model alignment context. Each leaf node in the MDP transitions deterministically to a terminal absorbing state, regardless of the action taken. The deterministic transitions resemble the LLM setting, where the current state is represented by the sequence of encountered tokens (s 1 , s 2 , ..., s i ), and the action corresponds to predicting the next word s i+1 from the vocabulary, given the context. In this simplified MDP, the deterministic transition is akin to a concatenation function, advancing the state to the next step (s 1 , s 2 , ..., s i , s i+1 ). Employing a toy MDP enables us to systematically evaluate the trajectory probabilities for all feasible paths within the MDP, shedding light on the allocation of probability mass by Direct Alignment Algorithms (DAAs) towards out-of-distribution (OOD) trajectories.The Experimental Setup. We adhere to the standard direct alignment protocol [46] [41], encompassing two key stages:1. Supervised Fine-tuning (SFT) / Behavioral Cloning (BC): This phase involves finetuning the policy based on a limited number of trajectories. Specifically, we utilize three demonstrations for SFT: (s 1 , a 0 , s 2 , a 0 , s 5 , a 0 , s ∞ ), (s 1 , a 1 , s 3 , a 1 , s 9 , a 0 , s ∞ ), and (s 1 , a 2 , s 4 , a 2 , s 13 , a 2 , s ∞ ).We present KL divergence versus GPT-4 win rate plots in Figure 19 to illustrate overoptimization trends in Direct Alignment Algorithms for the Gemma2-2b model [61] and the Anthropic-HH dataset [5] . Results are shown for the DPO and SLiC variants, which sufficiently demonstrate that the overoptimization trends observed with the Pythia models are not specific to a single model or dataset.The figure illustrates the trade-off between KL divergence and GPT-4 win rate across different values of beta in the alignment objective.• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: This paper studies Direct Alignment Algorithms (DAAs), revealing that similar to traditional RLHF methods, DAAs suffer from degradation due to over-optimization, especially at higher KL-budgets. The authors also examine the consequences of reward over-optimization in DAAs across various objectives, training regimes, and model scales. The results seem to be interesting for a broad audience of NeurIPS. One issue to be careful about, as pointed out by "Reviewer TztD": there is a potential distribution shift when evaluation using GPT-4. I strongly recommend that the authors add a discussion on this point in the paper.

Reviews: review: paper_summary: paper_summary: Reinforcement Learning from Human Feedback (RLHF) is a popular paradigm for aligning Large Language Models (LLMs) to human preferences. Direct Alignment Algorithms (DAA) are an alternative to traditional RLHF methods, which reduce the need to learn a reward and policy model separately. It has been shown that traditional RLHF methods suffer from reward over-optimization or reward hacking. However, because DAA do not learn a separate reward model, it is important to study whether these algorithms suffer from the same issues. This paper studies three popular DAA algorithms and how their performance deteriorates from over-optimization at scale. The paper has several compelling results that show the trade-off between different DAA objectives, KL divergence, and win rate when optimizing across different scales of models.
strength_weakness: strength_weakness: - The paper only performs experiments on a single model class and task, so it is not clear if these results generalize. For example, the paper observes that most models perform best after training on only 25% of the data, but it is not obvious if this is an artifact of the specific models used or the data. Does the same observation hold true on a different model or a different task?
- The performance of DPO seems under-tuned compared to the results of DPO in the literature. (1, 2).
- The paper does not provide any details regarding how model selection was performed and additional design choices.

(1) REBEL: Reinforcement Learning via Regressing Relative Rewards by Zhaolin Gao

(2) BPO: Supercharging Online Preference Learning by Adhering to the Proximity of Behavior LLM
questions: questions: - Could you provide best-fit lines for each model size in Figure 4 so that I can see trends? (This is the point raised on lines 196-200.)
- Is it a fair conclusion that IPO performs the best among the DAA studied in this paper?
- Are IPO and SCLI also prone to length exploitation? If so, could you provide results similar to those for DPO?
- How did you perform model selection?
limitations: limitations: Yes
paper_summary: paper_summary: This paper studies the scaling laws of DAAs for RLHF. The authors conducted extensive empirical studies and the discoveries are reported and discussed.
strength_weakness: strength_weakness: I like large-scale empirical study yet I disagree with calling those discoveries "laws". To draw a clear scaling **law** (from a scientific perspective), the current results are not supportive enough. Specifically, in Figure 1, the results do not seem to be a good fit, and more experimental results might be helpful to draw the conclusion. Also, why is the form of "scaling law" chosen as Eqn(5)? In Gal et al, BoN and PPO have different forms --- it would be highly possible that different DAAs also have different equation forms. 


The authors should further highlight the takeaway messages of their empirical discoveries. For instance, with the discovered scaling law, is there a way to perform early stopping in training to achieve better performance? --- This could be a great contribution to the community as otherwise researchers may be prone to make unfair comparisons in evaluating algorithms.
questions: questions: Could the authors also provide error bars in Figure 2, second row? The results look very unstable. 

For the scaling law fit, the author mentioned using win-rate to be the proxy of the golden reward, could the author provide detailed statistics of the results? I'm curious --- to what extent could different judges agree? If the win rate is only accurate in (for example,) 80% of the settings, does not that mean we are only able to draw a very rough conclusion using another proxy of this objective? 

On the Tree-MDP, why should all states go to the same absorbing state? What is the consideration of using a single absorbing state rather than many (i.e., = the number of leaves).
limitations: limitations: please see weakness / question.
paper_summary: paper_summary: This paper studies the reward over-optimization issue for offline alignment algorithms (e.g., DPO series) with massive experiment trials and discussions on why this phenomenon happens.
strength_weakness: strength_weakness: There are no major weaknesses. Here are some of the minor comments for further improvement.

1. Why do you choose DPO, IPO, and SLiC-HF? Why not some other variants, say ORPO? Although I do agree that the current selections can be sufficiently representative, some discussions on why they are prioritized can be appreciated. 

2. The presentation of Section 4 can be possibly improved. With its current presentation, it is difficult for a reader to understand the major argument of this section. There seems to be a collection of diverse arguments with different experiments. It may be better to highlight the overall goal of this section and the reason why these experiments are conducted in the very beginning before moving into more detailed experiments. 

3. Regarding the "Rank Efficiency" issue in Section 4, is it possible to have some more rigorous formulations in addition to statements only, say a theorem? 

4. Regarding citations and references, the OOD issue of DPO is also discussed in [1], which may be a good one to refer to. Also, the citation format is very strange. For example, many cited papers in the reference do not have any publication venues.

[1] Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study, S. Xu et al., ICML 2024. https://arxiv.org/abs/2404.10719
questions: questions: See the comments in the weakness section.
limitations: limitations: The limitations have been discussed in the paper.
paper_summary: paper_summary: This paper analyse overoptimisation in the context of direct alignment algorithms (DAA). They show that even when no explicit reward model is being optimised against, a similar phenomena as Gao et al. is shown, where as KL budget increases "gold" reward increases and then decreases. This phenomena is shown across model sizes and different DAAs (DPO, IPO, SLiC). They fit scaling curves to this phenomena and find a similar style of fit to Gao et al.. This phenomena is then analysed from various angles, and the authors show that at low KL budgets best performance is reached early in training; length-correct adjusts the KL-winrate pareto frontier but doesn't mitigate overoptimisation; model training statistics aren't useful for predicting downstream performance. They also present some theoretical analysis and a toy MDP which demonstrates why reward exploitation can occur in DAAs, showing that lack of support in the sequence space combined with DAA's non-convex optimisation target can lead to DAAs placing high probability on OOD (and hence low-reward) sequences.

L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. Interna- tional Conference on machine Learning, 2023.
strength_weakness: strength_weakness: ## Large points

I think the main weaknesses with the paper are three-fold:
* All analysis is only performed on the TL;DR summarisation dataset, which is somewhat different from Gao et al. and the setting these preference learning algorithms are generaly used in (dialogue and instruction-following). While reproducing the whole analysis on another dataset is too much to ask, testing out some of the core hypotheses, or reproducing some of figure 1, on a different dataset (e.g. alpaca farm, or anthropic HH) would increase the robustness and generality of the results.
* Using GPT-4 winrate as the "gold" reward or reference output also makes the results less general. The preference distribution that produced the dataset the DAAs were trained on is not the same one being used to evaluate here. I would expect the results to hold up in the setting where these two distributions are the same, but it would be beneficial to have some empirical validation of that. This would likely mean mimicing the setting of Gao et al, and applying DPO on a preference distribution generated from an accessible gold reward function (for example, the data of Coste et al. is available and may be suitable, or alpaca farm gpt4 preference data could also work). The same reward function can then be used to produce the y-axis of the plots, which might even produce cleaner trends. This would more accurately mirror the real-world setting where human preferences would be used for both training and evaluation.
* While the dicussion in section 4 is useful, it would be beneficial to have a better explanation of why this phenomena happens in DAAs, as it's still not immediately intuitive for me.

## Small points

* it would be useful for the axes in figure 1 to be unified across all the subplots, so it is easier to compare between algorithms. Similarly for the y-axis in figure 2.
* some of the text in the figures (especially figure 7) is quite small.

## Summary

I think the paper is worthy of acceptance as is, and I'm reccomending accept. If one of the points mentioned in the Large Points section above was addressed thoroughly I would consider raising my score to a strong accept.
questions: questions: My questions have been described in the weaknesses section.
limitations: limitations: The authors discuss limitations.

Comments: title: Thanks
comment: Thank you for the clarification, I was not sure if you did any involved model checkpoint selection. All of my concerns have been addressed.
The responses all look good to me.
Dear reviewer,

We would like to follow up to see if the response addresses your concerns or if you have any further questions. We would really appreciate the opportunity to discuss this further if our response has not already addressed your concerns. Thank you again!
Dear reviewer,

We would like to follow up to see if the response addresses your concerns or if you have any further questions. We would really appreciate the opportunity to discuss this further if our response has not already addressed your concerns. Thank you again!
Dear reviewer,

We would like to follow up to see if the response addresses your concerns or if you have any further questions. We would really appreciate the opportunity to discuss this further if our response has not already addressed your concerns. Thank you again!
Regarding the evaluation using GPT4, we do agree that in theory this represents a distribution shift in terms of preferences. However, we would like to highlight the below table from the original DPO work:

|                           | DPO | SFT | PPO-1 |
|---------------------------|-----|-----|-------|
| **N respondents**          | 272 | 122 | 199   |
| **GPT-4 (S) win %**        | 47  | 27  | 13    |
| **GPT-4 (C) win %**        | 54  | 32  | 12    |
| **Human win %**            | 58  | 43  | 17    |
|---------------------------|-----|-----|-------|
| **GPT-4 (S)-H agree**      | 70  | 77  | 86    |
| **GPT-4 (C)-H agree**      | 67  | 79  | 85    |
| **H-H agree**              | 65  | -   | 87    |

It shows that specifically on the TL;DR summarization task, GPT4 agrees with the majority opinion at the same degree (and higher for DPO models) as humans. We would argue that GPT4 judgments are as good of a proxy for the majority human opinion as individual human raters (which is the way the training data was generated).

We do agree that the drivers behind the exact KL-quality dynamics are still somewhat unclear. However, we believe this is a challenging theoretical and empirical problem, which may warrant it's own research as some recent works have done [1].

[1] Understanding the Learning Dynamics of Alignment with Human Feedback
Shawn Im, Yixuan Li
Thank you for your response. I'm glad to here you're working on replicating the results in a different setting, and the preliminary results are very promising in this regard.

### on gpt-4 winrate

The scenario we care about is when we use humans to produce preferences, which we then optimise against using some DAA algorithm, and then we see that we have overoptimised the human preference function. Given it's expensive to do this analysis with real humans, we can produce an analogous setting where we replace humans with GPT4 as the rater. However, I think to make this setting more analagous we would like to have GPT4 both as the provider of preferences and as the as the judge used during evaluation, whereas in your experiments humans provide the preferences but GPT4 is the judge, resulting in a mismatch and a potentially less analogous setting. This is the issue with the experimental settings I was pointing to in that comment.

I acknowledge reproducing the experimental results with GPT-4 as the provider of preferences as well as the rater would be extremely expensive and infeasible in the remaining time, but I think it would be worth discussing this disanalogy between your setting and the real-world scenario that motivates this work in the paper.

### on better intuition

Thanks for the providing that explanation. I understand that intuition, but to me it doesn't seem that the explanation provided predicts the shape of the relationship between KL and gold reward. As described, that intuition would imply (to me) that as soon as you start optimising with the DAA method, KL would increase and gold reward would go down. Why do you think gold reward goes up and then down as you optimise (or as you choose different KL penalty coefficients)?

### summary

Overall, I am mainting my score of a 7, but I am keen to see the paper accepted, especially given the preliminary results reproduce the effect in a different setting.