Disentangling Length from Quality in Direct Preference Optimization (2024)

Abstract: Reinforcement Learning from Human Feedback (RLHF) has been a crucial component in the recent success of Large Language Models. However, RLHF is know to exploit biases in human preferences, such as verbosity. A well-formatted and eloquent answer is often more highly rated by users, even when it is less helpful and objective. A number of approaches have been developed to control those biases in the classical RLHF literature, but the problem remains relatively under-explored for Direct Alignment Algorithms such as Direct Preference Optimization (DPO). Unlike classical RLHF, DPO does not train a separate reward model or use reinforcement learning directly, so previous approaches developed to control verbosity cannot be directly applied to this setting. Our work makes several contributions. For the first time, we study the length problem in the DPO setting, showing significant exploitation in DPO and linking it to out-of-distribution bootstrapping. We then develop a principled but simple regularization strategy that prevents length exploitation while still maintaining improvements in model quality. We demonstrate these effects across datasets on summarization and dialogue, where we achieve up to 20% improvement in win rates when controlling for length, despite the GPT-4 judge’s well-known verbosity bias.

heading: 1 Introduction
text: Recently, Large Language Models (LLMs) have seen significant improvements in capabilities, such as code-generation, mathematical reasoning, and tool use. Importantly, they can now fluently interact with users and follow their instructions, leading to their widespread adoption. Fine-tuning with Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017; Stiennon et al.,
*Denotes equal contribution
2022) has been a significant component in those advances and is now a standard part of advanced LLM training pipelines (Ouyang et al., 2022; Bai et al., 2022a; Touvron et al., 2023; Jiang et al., 2024; Anil et al., 2023). Currently, all the leading LLMs deploy some sort of RLHF pipeline (Dubois et al., 2024; Zheng et al., 2023; Liang et al., 2023). The classical approach consists of three-stages. The first stage begins with a general model pre-trained with next-token prediction on a large corpus of text (Radford et al., 2019; Brown et al., 2020), which is then further-tuned for instruction-following purposes (Wei et al., 2022). In the second stage, the model is prompted with general requests, and generates multiple possible answers, which are then ranked by the user. These ratings are used to train a reward model, which represents human preferences (Christiano et al., 2017; Stiennon et al., 2022; Ziegler et al., 2020; Bai et al., 2022a; Touvron et al., 2023). In the final stage, the instruction-tuned LLM is further trained to maximize expected rewards from the previously trained reward model (a proxy for user preferences) using general purpose rein-
4998
forcement learning algorithms (Schulman et al., 2017; Mnih et al., 2016). While successful, this pipeline is technically complex and computationally expensive, mainly due to the final stage of RL optimization.
The quality of the learned reward model is crucial for the RLHF process (Touvron et al., 2023). However, prior works have demonstrated that reward models can be exploited (Casper et al., 2023; Gao et al., 2023) due to a Goodhart’s law effect (Clark and Amodei, 2016; Manheim and Garrabrant, 2019; Skalse et al., 2022; Lambert and Calandra, 2023). Under this phenomenon, the model can achieve high rewards during the RL training while generating undesirable behaviours (Gao et al., 2023; Dubois et al., 2024). A particular case of the reward exploitation phenomenon is the well-known verbosity issue - models fine-tuned with RLHF generate significantly longer answers, without necessarily improving the actual quality (Singhal et al., 2023; Kabir et al., 2023). This has been linked to an explicit bias in the preference data towards longer responses (Singhal et al., 2023). However, the statistical increase in verbosity of RLHF-trained models significantly outmatches
the the difference of distribution lengths between the preferred and rejected answers. This effect is even observed in in strong propriety models, such as GPT-4 (John Schulman et al., 2022), which is now frequently used to evaluate the performance of other LLMs (Dubois et al., 2024; Zheng et al., 2023; Zeng et al., 2023). However, even as an evaluator, GPT-4 exhibits strong preferences for length. Prior work (Wang et al., 2023) has noted that when evaluating 13B parameter models in head-to-head comparisons with the Davinci-003 model, win rates and the average number of unique tokens in the model’s response have correlation of 0.96.
Recently, Direct Preference Optimization (Rafailov et al., 2023) has emerged as an alternative to the standard RLHF pipeline. The key observation of DPO is that the reward model can directly be re-parameterized through the optimal LLM policy obtained in the reinforcement learning stage. This allows us to directly train the language model through the reward learning pipeline, eliminating the need for the reinforcement learning stage. This algorithm has become widely used, since it can train completely offline, yielding better simplicity of tuning, speed, and resource efficiency,
while still maintaining performance (Dubois et al., 2024; Jiang et al., 2024). For these reasons, it has also been widely adopted by the open-source community. At the time of this writing, 9 out of the top 10 models on the HuggingFace Open LLM Leaderboard use DPO as part of their training pipeline.
While the question of length exploitation has been extensively studied in the classical RLHF pipeline, it has not been explored in the DPO setting. RLHF-style reward models are explicit, making them susceptible to issues like reward overoptimization (Gao et al., 2023). It is unclear whether these issues transfer to DPO, where the lack of an explicit reward model means the problem of reward overoptimization is harder to define. To complicate this issue, others have argued that apparent gains in open-source model performance across automated benchmarks are driven by evaluator’s verbosity bias (Liu, 2024). These statistics are demonstrated in Figure 1, as open-source models can match the overall performance of proprietary ones, but lag significantly on length-corrected basis. We make several contributions in our work: First, we show the length exploitation is quite prevalent in DPO. We demonstrate empirically (for the first time) that in this settings OOD extrapolation issues emerge similarly to classical RLHF. Next, we derive a simple but efficient regularization approach, showing it can effectively control verbosity and minimally impact performance even under a length-biased judge which also explains other empirical phenomena in DPO training, such as early convergence.

heading: 2 Preliminaries
text: In this section, we will outline the core components of the standard RLHF pipeline Ziegler et al.; Stiennon et al.; Bai et al.; Ouyang et al.) and the Direct Preference Optimization algorithm (Rafailov et al., 2023), which is central to our analysis and regularization derivations.

heading: 2.1 Reinforcement Learning From Human Feedback
text: The standard RLHF pipeline consists of three stages: 1) we first pre-train a general LLM for instruction-following purposes with supervised fine-tuning (SFT); 2) next, we gather human feedback and train a parameterized reward model; 3) we further optimize the LLM in a reinforcement
learning loop using the trained reward model. SFT: During this stage, we use a dataset of prompts x and high-quality answers y to train an LLM with next-token prediction to obtain a model πSFT(y|x). In our notation, we treat the entire prompt and answer strings as a single variable. Reward modeling Phase: In the second phase the instruction-tuned model is given prompts x, and produces pairs of answers (y1,y2) ∼ πSFT(y|x). Users then rank the answers. We denote these preferences as yw ≻ yl | x, where yw and yl are the preferred and dispreferred answer. The rankings are usually assumed to be generated by the Bradley-Terry (BT) (Bradley and Terry, 1952), in which the preference distribution p is assumed to be driven by an unobserved latent reward r(x,y) and the following parametrization:
p(y1 ≻ y2 | x) = exp (r(x,y1))
exp (r(x,y1)) + exp (r(x,y2)) .
(1) Then, given a dataset of user rankings D ={ x(i),y (i) w ,y (i) l }N i=1
, we can train a parameterized reward model rϕ(x,y) using maximum likelihood:
LR(rϕ,D) = −E(x,yw,yl)∼D [ log σ(rϕ(x,yw)− rϕ(x,yl)) ]
(2)
where σ is the logistic function. Reinforcement Learning Phase: During the final phase, we use the learned reward function in an RL loop where the LLM is treated as a policy. The most common optimization objective is the following:
max πθ
Ex∼D,y∼πθ(y|x) [ rϕ(x,y) ] −
βDKL [ πθ(y | x) || πref(y|x) ] (3)
where πref(y|x) is a reference distribution (usually taken to be πref(y|x)) and β is a hyper-parameter. This objective trades off maximizing the reward rϕ(x,y) and the regularizing divergence term, which prevents the policy from drifting far from πref(y|x). This objective is then optimized using a general purpose RL algorithm, such as PPO (Schulman et al., 2017).

heading: 2.2 Direct Preference Optimization
text: Direct Preference Optimization (Rafailov et al., 2023) starts with the same objective as Eq. 3. However, DPO assumes we have access to the ground
truth reward r(x,y) and derives an analytical transformation between the optimal reward and optimal policy. This can be substituted back into the reward optimization objective in Eq. 2, which allows us to train the optimal model directly on the feedback data using the following objective:
LDPO (πθ;πref) =
−E(x,yw,yl)∼D [ log σ ( β log πθ (yw | x) πref (yw | x) −
β log πθ (yl | x) πref (yl | x)
)] (4)
Here, the parameter β is the same as in Eq. 3, and similarly controls the trade-off between expected reward and divergence from the model initialization. The DPO objective is attractive since it allows us to recover the optimal model using a standard classification loss, without the need for on-policy sampling or significant amount of hyper-parameter tuning. Eq. 4 resembles the reward modeling objective in Eq. 2 under the parameterization
rθ(x,y) = β log πθ (y | x) πref (y | x)
(5)
We will refer to this as the DPO "implicit reward". Theorem 1 in (Rafailov et al., 2023) shows that this is indeed a valid parameterization of a reward model without loss of generality. If we substitute this form of rθ(x,y) into the RL objective 3, we can obtain the optimal solution in a closed form, which happens to be πθ. We will return to the interpretation of DPO as an implicit reward function later on in our analysis of out-of-distribution bootstrapping.

heading: 3 Building in Explicit Regularization in DPO
text: Prior works have explicitly considered lengthregularization in the classical RLHF pipeline (Singhal et al., 2023), however these methods do not transfer directly to direct alignment algorithms, such as DPO (Rosset et al., 2024). We derive a length-regularized version of the algorithm by adding a regularized term to Eq. 3. The below considerations hold for a general regularizer, but we focus on a length term α|y|, where α is a hyperparameter and |y| denotes the token-length of the answer y. We then formulate the regularized RL
problems in the following objective:
max πθ
Ex∼D,y∼πθ(y|x) [ r(x,y) ] − α|y|−
βDKL [ πθ(y | x) || πref(y|x) ] (6)
where we assume that r(x,y) is still the same latent reward driving human preferences. We can follow the same derivations in (Rafailov et al., 2023) for the reward function r(x,y) ] − α|y| and obtain the optimal solution to Eq. 6 as
π∗(y|x) = 1 Z(x) πrefe 1 β (r(x,y)−α|y|) (7)
where Z(x) = ∑
y πrefe 1 β (r(x,y)−α|y|). With some
simple algebra, we can then obtain the equivalent regularized reward re-formulation:
r(x,y) = β log π∗(y|x) πref(y|x)
+ β logZ(x) + α|y| (8)
We can then plug in Eq. 8 into the reward modeling stage in Eq. 2, which yields the following regularized DPO objective:
LR−DPO (πθ;πref) =
−E(x,yw,yl)∼D [ log σ ( β log πθ (yw | y) πref (yw | x) −
β log πθ (yl | x) πref (yl | x) + (α|yw| − α|yl|) )] (9)
This is similar to the standard DPO objective, except for an additional regularization term (α|yw| − α|yl|) in the logit of the binary classification loss. Recent work (Zhou et al., 2023), aims to develop a multi-objective optimization approach and arrives at a similar objective involving a mixture of reward models, while we focus on optimization robustness of DPO. Another concurrent work (Chen et al., 2024) also consider the length exploitation problem in the classical RLHF pipeline, suggesting a similar regularization in the reward modeling stage in Eq. 2. This regularizer helps disentangle answer quality from length, demonstrating meaningful improvement in length-controlled performance. Our derivations can be seen as the DPO implicit reward counterpart to this classical RLHF approach, explicitly linking the regularized reward modeling problem to an equivalent regularized RL setup.
Similar to the original DPO formulation, the regularized objective still aims to increase the likelihood along the preferred answer, while decreasing
the likelihood along the dis-preferred answer, modulated by a weighting term. This term is equivalent to the original DPO formulation with the addition of the regularization margin α|yw|−α|yl|. We can interpret this as an additional per-example learning rate, which up-weighs the gradient on feedback pairs, in which the selected answer is shorter and down-weights the gradient on pairs in which the selected answer is longer, proportional to the difference in length.

heading: 4 Experiments
text: In this section we will empirically investigate the verbosity exploitation issues in DPO, the effectiveness of our regularization strategy and the potential causes of these effects. We beging with a description of our evaluation tasks and models.

heading: 4.1 Datasets and Models
text: We utilize three different setups in our experimental setting based on summarization, dialogue and general instruction-following. Summarization We use the standard Reddit TL;DR (TL;DR) summarization dataset from (Stiennon et al., 2022), which consists of a Reddit post and several short summaries, judged for quality and informativeness by human evaluators. Dialogue: For our dialogue experiment we use the Anthropic Helpful and Harmless (HH) datasets (Bai et al., 2022b), which consists of general conversations with a language model assistants, which are also ranked by human annotators.
Datasets statistics are included in Table 1 where exhibit a small length bias in the preferred response. Following (Rafailov et al., 2023), we use the Pythia 2.8B (Biderman et al., 2023) for both the dialogue and summarization tasks, carrying out full-parameter fine-tuning using the DPO original codebase2 with default hyperparameters, except when noted otherwise. See C for more details.
2https://github.com/eric-mitchell/direct-preferenceoptimization

heading: 4.2 Length Exploitation in DPO and Effectiveness of Regularization
text: We first consider the Anthropic Helpful and Harmless and Reddit TL;DR datasets. For both tasks, we train models with three parameter values β ∈ [0.5, 0.1, 0.05] and then sample 256 answers using prompts from the evaluation dataset. The length histograms are shown in Fig. 2. The first two columns show the answer length distribution for the set of preferred, rejected and DPO-generated answers, with each row corresponding to a different β parameter value. We see that the DPO generated answers are, on average, significantly longer than both the preferred and rejected answers. Models trained with smaller values of β generate longer responses on average, which is expected since β controls the deviation from the initial policy. Not only does the DPO model generate longer answers, it also generates answers that are significantly outof-distribution in terms of length from the offline preference dataset.
The third and fourth column in Fig. 2 show results for the SFT, DPO the length-regularized DPO model introduced in Section 3. We use α = 0.01 and α = 0.05 for the Anthropic Helpful and Harmless and Reddit TL;DR datasets respectively. While the length-regularized models still show mild increase in average length, they match the SFT model much more closely. Moreover, they do not generate answers with significantly out-of-distribution lengths. This indicates that the proposed algorithm can efficiently regularize the verbosity of the trained model.

heading: 4.3 Length Versus Quality Trade-Offs
text: In this section, we evaluate the length versus quality model trade-offs. For the Anthropic Helpful and Harmless and Reddit TL;DR datasets, we use the answers sampled from DPO policies and compare them head-to-head against the dataset preferred answer, with GPT-4 as an evaluator. Our main results are shown in Fig. 3, which plots
model win rates against average answer length, with 90% confidence intervals. We again evaluate all models with β ∈ [0.05, 0.1.0.5]. For HH, we use α ∈ [0, 0.005, 0.01]; for TL;DR, we use α ∈ [0, 0.2, 0.5] (α = 0 is standard DPO). Similar to before, we see that the length-regularized training can efficiently control verbosity, significantly decreasing the average length of the answers as compared to standard DPO. Moreover, on the HH task, regularization also leads to mild improvement in win rates, but a slight decrease on TL;DR (although both of these are not statistically significant). These results are quite promising, as GPT-4 is known to have a significant length bias in its preferences (Wang et al., 2023; Singhal et al., 2023). On both HH and TL;DR, the length-regularized experiments with β = 0.05 and β = 0.01 match the average lengths of the corresponding β = 0.5 runs, but achieve statistically significant higher corresponding win rates, with close to 20% improvement on HH and close to 15% improvement on TL;DR.

heading: 4.4 Is Length a Proxy for KL-Divergence?
text: In the constrained RL problem in Eq. 3 and the corresponding DPO objective in Eq. 4, the β parameter controls the degree of policy divergence from the initial reference model. In Fig. 2 and Fig. 3, we see that average length of the model generated answers is inversely proportional to the β parameter. In this section, we investigate the relationship between the length-regularized DPO objective in Eq. 9 and the KL divergence from the initial policy. In Fig. 4, we plot the trained policy’s KL divergence from πref for the different values of β and α parameters. We see only a weak correlation between KL divergence and length. For both
HH and TL;DR, length-regularized models trained with β = 0.05 and β = 0.01 match the average length of train runs with β = 0.5 (Fig. 3). At the same time, these runs have statistically significant higher KL divergences and win rates as shown in Fig. 3. We hypothesize that this indicates the existence of different factors driving human preference, with length only partially accounting for the policy KL divergence.

heading: 4.5 DPO and Early Convergence
text: In (Rafailov et al., 2023), the authors show early convergence of the DPO algorithm on the HH dataset. DPO achieves its best performance within a few hundred gradient steps, and does not improve with further training. Similar observations have also been made within the open-source community. We claim that this effect is likely due to length exploitation and the biased GPT-4 evaluator. In Fig. 5, we consider the training progression on the HH dataset with β = 0.1. We compare the regular DPO run (α = 0) with the length-regularized one (α = 0.1). We train for two epochs and evaluate intermediate checkpoints on the same set of prompts for average answer length, win rates, and KL divergence. Within the first 10% of the epoch, the standard DPO run produces answers almost twice as long as the SFT model. Standard DPO achieves its highest win rate here, with only KL divergence and average length increasing steadily with further training. In contrast, the length-regularized run sees little to no intermediate increase in length, but steady improvement in win rates throughout training and slow increases in divergence from the reference policy. Our final regularized checkpoint outperforms the non-regularized model along all fronts (KL, winrate, and length) at the end of 2
epochs. We hypothesize that the regular DPO training quickly increases length, exploiting the evaluator’s bias, but fails to capture more complex preference features. On the other hand, the lengthregularized training run is able to disentangle the verbosity component and fit other, more difficult quality features over a longer training period.

heading: 4.6 What Drives Length Exploitation?
text: In classical RLHF, excessive model verbosity (John Schulman et al., 2022) has been well understood as a reward exploitation problem (Gao et al., 2023; Casper et al., 2023; Lambert and Calandra, 2023), driven by a bias in the feedback datasets for longer answers. In particular, in the classical RLHF pipeline from Section 2.1, the reward model is continuously queried on new data generated by the model, creating an out-of-distribution (OOD) robustness issue. These results do not directly transfer to the DPO algorithm, as it does not train a separate reward model and only uses the offline feedback dataset for training. Surprisingly, we find that
the exploding length issue in DPO training is similarly driven by out-of-distribution exploitation. We consider the DPO algorithm as an implicit reward training method, as outlined in Section 2.2. We investigate the behavior of the implicit reward rθ as defined in Eq. 5. Since the DPO policy πθ is the optimal solution to the constrained RL problem in Eq. 3 corresponding to rθ, any exploitation behaviour from the policy must be driven by the reward function. In Fig. 6, we evaluate rθ trained with β = 0.1 and different α parameters on the offline feedback dataset (within its training distribution) and on answers generated by the corresponding DPO policy (out of distribution). Surprisingly, within distribution, the corresponding implicit reward models exhibit weak to no length correlation (and even negative length correlation with strong α regularization). However, they all show significant length bias out-of-distribution, with length explaining 30- 46% of the reward variance (as measured by the R2 of a linear regression of the implicit DPO reward on answer length).

heading: 5 Related Work
text: In this section we outline relevant work on reward exploitation in RLHF and the verbosity bias. Reward Exploitation in RLHF: RLHF reward exploitation, also known as reward over-optimization, is a well-known issue (Skalse et al., 2022; Pan et al., 2022; Casper et al., 2023; Lambert and Calandra, 2023) in which during the reinforcement learning stage, the expected reward keeps improving, but the quality of the model begins to degrade after some point. These effects were confirmed analytically in controlled experiments (Gao et al., 2023), as well as empirically in user studies (Dubois et al., 2024). Increased model verbosity has been explicitly linked to this phenomenon (John Schulman et al., 2022). A number approaches have been proposed to mitigate this issue, such as penalizing epistemic uncertainty (Coste et al., 2023; Zhai et al., 2023; Ahmed et al., 2024) or using mixture reward models (Moskovitz et al., 2023), but they do not explicitly target the length issue. Mitigating Length Biases in RLHF: A number of works have sought to explicitly address length biases in RLHF policies. (Ramamurthy et al., 2023) suggest setting a simple discount factor, which improves naturalness of the generated language, (Singhal et al., 2023) carry out an extensive study of length correlations in classical RLHF and suggest a number of heuristic-based mitigating approaches. The closes to our approach are the works of (Shen et al., 2023) and the concurrent work of (Chen et al.,
2024), which propose to disentangle length-biases from quality during the reward modeling stage. Our work can be seen as a DPO equivalent counter-part to these approaches.
As far as we are aware, this is the first work to study the length exploitation problem for Direct Alignment Algorithms such as DPO.

heading: 6 Conclusion
text: In this work, we study the problem of length exploitation in the Direct Preference Optimization (DPO) algorithm, extending its analysis from classical RLHF to DPO for the first time. On two standard human feedback datasets, we empirically show that DPO exhibits significant length hacking across a range of hyperparameters. We then specifically link this phenomenon to out-of-distribution bootstrapping. We derive an analytical lengthregularized version of the DPO algorithm and show empirically that we can maintain model performance, as evaluated by GPT-4 without significant increases in verbosity, boosting length-corrected win rates by up to 15-20%. Given the strong length bias in public feedback datasets and the prominence of DPO in the open source community, we hypothesize that many open-source models suffer from similar length-exploitation issues, driving the observations of Fig. 1. Our results are encouraging, suggesting that open-source models could match proprietary ones on automated evaluations on a length corrected basis as well.

heading: 7 Limitations
text: Our work addresses the particular issue of length exploitation in Direct Preference Optimization. Our regularization objective requires explicit penalty function (such as length) and may not be suitable to avoid general exploitation issues along axes separate from verbosity. Furthermore, we only study the DPO objective, which might behave differently from other direct alignment algorithms, which use different objective functions. We also e4valuate our approach on one model size and two smaller-scale public datasets of human feedback. It is unclear what the scaling laws of such exploitation behaviours might be and to what degree they are dependent on model size, capability and data quality beyond length biases. We believe these questions are a promising direction for future work.
Ethics Statement
This work focuses on alleviating and empirical extrapolation issues during DPO training, specifically an increasing verbosity bias. Experiments are ran on publicly-available data and pre-trained models, and do not release any new models for public use. As a result, there should be no ethical concerns.

heading: Acknowledgements
text: Chelsea Finn is a CIFAR Fellow in the Learning in Machines and Brains program. This work was also supported by ONR grant N00014-22-1-2621 and the Volkswagen Group.

heading: A Phi-2 UltraFeedback Experiments
text: To further validate the effectiveness of the proposed length regularization schena, we run a small set of experiments with the 2.7B Microsoft model Phi-2 (Javaheripi et al., 2023) on the Ultrafeedback binarized dataset3 (Cui et al., 2023). This dataset consists of 64K prompts, whose completions are generated by LLMs and then ranked with GPT. The chosen response is considered to be the highestscoring completion, and the rejected prompt is chosen at random from the other 3 responses. We use the same experimental setup as for the TL;DR/HH Pythia 2.8B experiments (see Appendix C), and run 1 epoch of supervised fine-tuning on Ultrafeedback prior to alignment with DPO. We compare α = 0.0 and α = 0.05, using β = 0.1 for both models. For evaluation, we use the multi-turn MT Bench harness (Zheng et al., 2023). The results (Table 2) indicate that the length regularization strategy decreases length, while actually increasing downstream performance, though both gains are small.

heading: B Additional Out-of-Distribution Experiments
text: In Fig. 7 and Fig. 8, we provide the rest of the reward-length correlation experiments across both TL;DR and HH (see Fig. 6). The results across different α and β parameter values indicate a similar pattern of out-of-distribution extrapolation along the length axis.

heading: C Experimental Details
text: We follow the original DPO codebase (https://github.com/eric-mitchell/directpreference-optimization) with default hyperparameters unless otherwise noted (1 epoch of training, batch size of 128 with 16 gradient
3https://huggingface.co/datasets/ HuggingFaceH4/ultrafeedback_binarized
accumulation steps, RMSProp with a learning rate of 0.5× 10−6 with linear warm-up for 150 steps).
For the experiments in the main body of the paper, we used the OpenAI TL;DR dataset (92K preferred/dispreferred response pairs) (Stiennon et al., 2022). Each prompt is a Reddit post belonging to one of several topic forums, with title/post metadata included. 256 prompts sampled from the held-out set are used for all evaluations (e.g. loss, accuracy, KL, winrates, length), with temperature 1.0 and max length 512. All models in the main experiments were initialized from Pythia 2.8B base pre-trained weights, and underwent supervised fine-tuning on TL;DR for 1 epoch prior to DPO. All experiments were carried out on 4 NVIDIA A40 GPUs for a total of about 2000 GPU hours. All evaluations were computed with gpt-4-1106-preview as judge, with random positional flips to avoid known bias. We use the same GPT-4 evaluation prompts as in (Rafailov et al., 2023), they are listed below for completeness.

heading: HH GPT-4 winrate prompt.
text: For the following query to a chatbot, which response is more helpful?
Query: <the user query>
Response A: <either the test method or baseline>
Response B: <the other response>
FIRST provide a one-sentence comparison of the two responses and explain which you feel is more helpful. SECOND, on a new line, state only "A" or "B" to indicate which response is more helpful. Your response should use the format: Comparison: <one-sentence comparison and explanation>
More helpful: <"A" or "B">
TL;DR GPT-4 winrate prompt. Which of the following summaries does a better job of summarizing the most important points in the given forum post?
Post: <post>
Summary A: <Summary A>
Summary B: <Summary B>
FIRST provide a one-sentence comparison of the two summaries, explaining which you prefer and why. SECOND, on a new line, state only "A" or "B" to indicate your choice. Your response should use the format: Comparison: <one-sentence comparison and explanation> Preferred: <"A" or "B">

heading: D Sample Responses Across Models
text: In this section, we provide examples of model responses generated by length-regularized DPO (RDPO), standard DPO (DPO), and the initial policy (SFT), in response to prompts from the HH/TL;DR evaluation sets. These examples are selected to highlight cases where regularized DPO wins against the dataset response (according to the GPT4 evaluator), but both the DPO and SFT-produced answers do not. All samples were generated with the standard procedure in Appendix C from the same set of 256 prompts used for the rest of evaluations. See Tables 3 through 6 for samples.

heading: Future Work
text: We believe these questions are a promising direction for future work