SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The operator  $f$  simply normalizes the reward function with the logarithm of the partition function of  $\pi_r$ . Since the added normalization term is only a function of the prefix  $x$ ,  $f(r; \pi_{\mathrm{ref}}, \beta)(x, y)$  is a reward function in the equivalence class of  $r(x, y)$ . Finally, replacing  $r$  with the RHS of Eq. [5] (which holds for any reward function), we have  $f(r; \pi_{\mathrm{ref}}, \beta)(x, y) = \beta \log \frac{\pi_r(y|x)}{\pi_{\mathrm{ref}}(y|x)}$ . That is, the projection  $f$  produces a member of the equivalence class of  $r$  with the desired form, and we do not lose any generality in our reward model from the proposed reparameterization.
We can alternatively view Theorem  $\square$  as specifying exactly which reward function within each equivalence class the DPO reparameterization selects, that is, the reward function satisfying:
$$
\sum_ {y} \underbrace {\pi_ {\operatorname {r e f}} (y \mid x) \exp \left(\frac {1}{\beta} r (x , y)\right)} _ {= \pi (y \mid x), \text {u s i n g T h m .} ] \text {r e p a r a m .}} = 1, \tag {9}
$$
i.e.,  $\pi(y \mid x)$  is a valid distribution (probabilities are positive and sum to 1). However, following Eq. 4 we can see that Eq. 9 is the partition function of the optimal policy induced by the reward function  $r(x, y)$ . The key insight of the DPO algorithm is that we can impose certain constraints on the under-constrained Plackett-Luce (and Bradley-Terry in particular) family of preference models, such that we preserve the class of representable reward models, but explicitly make the optimal policy in Eq. 4 analytically tractable for all prompts  $x$ .
5.2 Instability of Actor-Critic Algorithms
We can also use our framework to diagnose instabilities with standard actor-critic algorithms used for the RLHF, such as PPO. We follow the RLHF pipeline and focus on the RL fine-tuning step outlined in Section 3. We can draw connections to the control as inference framework [20] for the constrained RL problem outlined in 3. We assume a parameterized model  $\pi_{\theta}(y \mid x)$  and minimize  $\mathbb{D}_{\mathrm{KL}}[\pi_{\theta}(y|x) \mid \pi^{*}(y \mid x)]$  where  $\pi^{*}$  is the optimal policy from Eq. 7 induced by the reward function  $r_{\phi}(y, x)$ . With some algebra this leads to the optimization objective:
$$
\max  _ {\pi_ {\theta}} \mathbb {E} _ {\pi_ {\theta} (y | x)} \left[ \underbrace {r _ {\phi} (x , y) - \beta \log \sum_ {y} \pi_ {\mathrm {r e f}} \exp \left(\frac {1}{\beta} r _ {\phi} (x , y)\right)} _ {f \left(r _ {\phi}, \pi_ {\mathrm {r e f}}, \beta\right)} - \underbrace {\beta \log \frac {\pi_ {\theta} (y \mid x)}{\pi_ {\mathrm {r e f}} (y \mid x)}} _ {\text {K L}} \right] \tag {10}
$$
This is the same objective optimized in prior works [48, 38, 1, 26] using the DPO-equivalent reward for the reward class of  $r_{\phi}$ . In this setting, we can interpret the normalization term in  $f(r_{\phi}, \pi_{\mathrm{ref}}, \beta)$  as the soft value function of the reference policy  $\pi_{\mathrm{ref}}$ . While this term does not affect the optimal solution, without it, the policy gradient of the objective could have high variance, making learning unstable. We can accommodate for the normalization term using a learned value function, but that can also be difficult to optimize. Alternatively, prior works have normalized rewards using a human completion baseline, essentially a single sample Monte-Carlo estimate of the normalizing term. In contrast the DPO reparameterization yields a reward function that does not require any baselines.
Figure 2: Left. The frontier of expected reward vs KL to the reference policy. DPO provides the highest expected reward for all KL values, demonstrating the quality of the optimization. Right. TL;DR summarization win rates vs. human-written summaries, using GPT-4 as evaluator. DPO exceeds PPO's best-case performance on summarization, while being more robust to changes in the sampling temperature.
In this section, we empirically evaluate DPO's ability to train policies directly from preferences. First, in a well-controlled text-generation setting, we ask: how efficiently does DPO trade off maximizing reward and minimizing KL-divergence with the reference policy, compared to common preference learning algorithms such as PPO? Next, we evaluate DPO's performance on larger models and more difficult RLHF tasks, including summarization and dialogue. We find that with almost no tuning of hyperparameters, DPO tends to perform as well or better than strong baselines like RLHF with PPO as well as returning the best of  $N$  sampled trajectories under a learned reward function. Before presenting these results, we describe the experimental set-up; additional details are in Appendix C.
Tasks. Our experiments explore three different open-ended text generation tasks. For all experiments, algorithms learn a policy from a dataset of preferences  $\mathcal{D} = \{x^{(i)},y_w^{(i)},y_l^{(i)}\}_{i = 1}^N$ . In controlled sentiment generation,  $x$  is a prefix of a movie review from the IMDb dataset [22], and the policy must generate  $y$  with positive sentiment. In order to perform a controlled evaluation, for this experiment we generate preference pairs over generations using a pre-trained sentiment classifier, where  $p(\text{positive} \mid x,y_w) > p(\text{positive} \mid x,y_l)$ . For SFT, we fine-tune GPT-2-large until convergence on reviews from the train split of the IMDB dataset (further details in App C.1). In summarization,  $x$  is a forum post from Reddit; the policy must generate a summary  $y$  of the main points in the post. Following prior work, we use the Reddit TL;DR summarization dataset [41] along with human preferences gathered by Stiannon et al. We use an SFT model fine-tuned on human-written forum post summaries with the TRLX [42] framework for RLHF. The human preference dataset was gathered by Stiannon et al. on samples from a different, but similarly-trained, SFT model. Finally, in single-turn dialogue,  $x$  is a human query, which may be anything from a question about astrophysics to a request for relationship advice. A policy must produce an engaging and helpful response  $y$  to a user's query; we use the Anthropic Helpful and Harmless dialogue dataset [1], containing 170k dialogues between a human and an automated assistant. Each transcript ends with a pair of responses generated by a large (although unknown) language model along with a preference label denoting the human-preferred response. In this setting, no pre-trained SFT model is available; we therefore fine-tune an off-the-shelf language model on only the preferred completions to form the SFT model.
Evaluation. Our experiments use two different approaches to evaluation. In order to analyze the effectiveness of each algorithm in optimizing the constrained reward maximization objective, in the controlled sentiment generation setting we evaluate each algorithm by its frontier of achieved reward and KL-divergence from the reference policy; this frontier is computable because we have access to the ground-truth reward function (a sentiment classifier). However, in the real world, the ground truth reward function is not known; therefore, we evaluate algorithms with their win rate against a baseline policy, using GPT-4 as a proxy for human evaluation of summary quality and response helpfulness in the summarization and single-turn dialogue settings, respectively. For summarization, we use reference summaries in the test set as the baseline; for dialogue, we use the preferred response in the
$^{2}$ https://huggingface.co/CarperAI/openai_summarize_tldr_sft
Figure 3: Left. Win rates computed by GPT-4 for Anthropic-HH one-step dialogue; DPO is the only method that improves over chosen summaries in the Anthropic-HH test set. Right. Win rates for different sampling temperatures over the course of training. DPO's improvement over the dataset labels is fairly stable over the course of training for different sampling temperatures.
test dataset as the baseline. While existing studies suggest LMs can be better automated evaluators than existing metrics [10], we conduct a human study to justify our usage of GPT-4 for evaluation in Sec. 6.4. We find GPT-4 judgments correlate strongly with humans, with human agreement with GPT-4 typically similar or higher than inter-human annotator agreement.
Methods. In addition to DPO, we evaluate several existing approaches to training language models to adhere to human preferences. Most simply, we explore zero-shot prompting with GPT-J [43] in the summarization task and 2-shot prompting with Pythia-2.8B [3] in the dialogue task. In addition, we evaluate the SFT model as well as Preferred-FT, which is a model fine-tuned with supervised learning on the chosen completion  $y_{w}$  from either the SFT model (in controlled sentiment and summarization) or a generic LM (in single-turn dialogue). Another pseudo-supervised method is Unlikelihood, which simply optimizes the policy to maximize the probability assigned to  $y_{w}$  and minimize the probability assigned to  $y_{l}$ ; we use an optional coefficient  $\alpha \in [0,1]$  on the 'unlikelihood' term. We also consider PPO [37] using a reward function learned from the preference data and PPO-GT, which is an oracle that learns from the ground truth reward function available in the controlled sentiment setting. In our sentiment experiments, we use two implementations of PPO-GT, one of-the-shelf version [42] as well as a modified version that normalizes rewards and further tunes hyperparameters to improve performance (we also use these modifications when running 'normal' PPO with learned rewards). Finally, we consider the Best of  $N$  baseline, sampling  $N$  responses from the SFT model (or Preferred-FT in dialogue) and returning the highest-scoring response according to a reward function learned from the preference dataset. This high-performing method decouples the quality of the reward model from the PPO optimization, but is computationally impractical even for moderate  $N$  as it requires sampling  $N$  completions for every query at test time.
6.1 How well can DPO optimize the RLHF objective?
The KL-constrained reward maximization objective used in typical RLHF algorithms balances exploitation of reward while restricting the policy from deviating far from the reference policy. Therefore, when comparing algorithms, we must take into account both reward achieved as well as the KL discrepancy; achieving slightly higher reward but with much higher KL is not necessarily desirable. Figure  $\boxed{2}$  shows the reward-KL frontier for various algorithms in the sentiment setting. We execute multiple training runs for each algorithm, using a different hyperparameter for policy conservativeness in each run (target KL  $\in$  {3,6,9,12} for PPO,  $\beta \in \{0.05, 0.1, 1, 5\}$ ,  $\alpha \in \{0.05, 0.1, 0.5, 1\}$  for unlikelihood, random seeds for preferred-FT). This sweep includes 22 runs in total. After each 100 training steps until convergence, we evaluate each policy on a set of test prompts, computing the average reward under the true reward function as well as the average sequence-level KL with the reference policy KL ( $\pi || \pi_{\mathrm{ref}}$ ). We find that DPO produces by far the most efficient frontier, achieving the highest reward while still achieving low KL. This result is particularly notable for multiple reasons. First, DPO and PPO optimize the same objective, but DPO is notably more efficient;
3That is, the sum of the per-timestep KL-divergences.
DPO's reward/KL tradeoff strictly dominates PPO. Second, DPO achieves a better frontier than PPO, even when PPO can access ground truth rewards (PPO-GT).
6.2 Can DPO scale to real preference datasets?
Next, we evaluate fine-tuning performance of DPO on summarization and single-turn dialogue. For summarization, automatic evaluation metrics such as ROUGE can be poorly correlated with human preferences [38], and prior work has found that fine-tuning LMs using PPO on human preferences to provide more effective summaries. We evaluate different methods by sampling completions on the test split of TL;DR summarization dataset, and computing the average win rate against reference completions in the test set. The completions for all methods are sampled at temperatures varying from 0.0 to 1.0, and the win rates are shown in Figure 2 (right). DPO, PPO and Preferred-FT all fine-tune the same GPT-J SFT model. We find that DPO has a win rate of approximately  $61\%$  at a temperature of 0.0, exceeding the performance of PPO at  $57\%$  at its optimal sampling temperature of 0.0. DPO also achieves a higher maximum win rate compared to the best of  $N$  baseline. We note that we did not meaningfully tune DPO's  $\beta$  hyperparameter, so these results may underestimate DPO's potential. Moreover, we find DPO to be much more robust to the sampling temperature than PPO, the performance of which can degrade to that of the base GPT-J model at high temperatures. Preferred-FT does not improve significantly over the SFT model. We also compare DPO and PPO head-to-head in human evaluations in Section 6.4, where DPO samples at temperature 0.25 were preferred  $58\%$  times over PPO samples at temperature 0.
On single-turn dialogue, we evaluate the different methods on the subset of the test split of the Anthropic HH dataset [1] with one step of human-assistant interaction. GPT-4 evaluations use the preferred completions on the test as the reference to compute the win rate for different methods. As there is no standard SFT model for this task, we start with a pre-trained Pythia-2.8B, use Preferred-FT to train a reference model on the chosen completions such that completions are within distribution of the model, and then train using DPO. We also compare against the best of 128 Preferred-FT completions (we found the Best of  $N$  baseline plateaus at 128 completions for this task; see Appendix Figure 4) and a 2-shot prompted version of the Pythia-2.8B base model, finding DPO performs as well or better for the best-performing temperatures for each method. We also evaluate an RLHF model trained with PPO on the Anthropic HH dataset from a well-known source, but are unable to find a prompt or sampling temperature that gives performance better than the base Pythia-2.8B model. Based on our results from TL;DR and the fact that both methods optimize the same reward function, we consider Best of 128 a rough proxy for PPO-level performance. Overall, DPO is the only computationally efficient method that improves over the preferred completions in the Anthropic HH dataset, and provides similar or better performance to the computationally demanding Best of 128 baseline. Finally, Figure 3 shows that DPO converges to its best performance relatively quickly.
6.3 Generalization to a new input distribution
To further compare the performance of PPO and DPO under distribution shifts, we evaluate the PPO and DPO policies from our Reddit TL;DR summarization experiment on a different distribution, news articles in the test split of the CNN/DailyMail dataset [24], using the best sampling temperatures from TL;DR (0 and 0.25). The results are presented in Table I. We computed the GPT-4 win rate against the ground-truth summaries in the datasets, using the same GPT
Alg.	Win rate vs. ground truth
Temp 0	Temp 0.25
DPO	0.36	0.31
PPO	0.26	0.23
Table 1: GPT-4 win rates vs. ground truth summaries for out-of-distribution CNN/DailyMail input articles.
4 (C) prompt we used for Reddit TL;DR, but replacing the words "forum post" with "news article". For this new distribution, DPO continues to outperform the PPO policy by a significant margin. This experiment provides initial evidence that DPO policies can generalize similarly well to PPO policies, even though DPO does not use the additional unlabeled Reddit TL;DR prompts that PPO uses.
4https://huggingface.co/CarperAI/openai_summarize_tldr_sft
https://huggingface.co/reciprocate/ppo_hh_PYthia-6B
https://github.com/CarperAI/trlx/tree/main/examples/hh
6.4 Validating GPT-4 judgments with human judgments
We conduct a human study to verify the reliability of GPT-4's judgments, using the results of the TL;DR summarization experiment and two different GPT-4 prompts. The GPT-4 (S) (simple) prompt simply asks for which summary better-summarizes the important information in the post. The GPT-4 (C) (concise) prompt also asks for which summary is more concise; we evaluate this prompt because we find that GPT-4 prefers longer, more repetitive summaries than humans do with the GPT-4 (S) prompt. See Appendix C.2 for the complete prompts. We perform three comparisons, using the highest (DPO, temp. 0.25), the lowest (PPO, temp. 1.0), and a
middle-performing (SFT, temp. 0.25) method with the aim of covering a diversity of sample qualities; all three methods are compared against greedily-sampled PPO (its best-performing temperature). We find that with both prompts, GPT-4 tends to agree with humans about as often as humans agree with each other, suggesting that GPT-4 is a reasonable proxy for human evaluations (due to limited human raters, we only collect multiple human judgments for the DPO and PPO-1 comparisons). Overall, the GPT-4 (C) prompt generally provides win rates more representative of humans; we therefore use this prompt for the main results in Section 6.2. For additional details about the human study, including the web interface presented to raters and the list of human volunteers, see Appendix D.3
DPO	SFT	PPO-1
N respondents	272	122	199
GPT-4 (S) win %	47	27	13
GPT-4 (C) win %	54	32	12
Human win %	58	43	17
GPT-4 (S)-H agree	70	77	86
GPT-4 (C)-H agree	67	79	85
H-H agree	65	-	87
Table 2: Comparing human and GPT-4 win rates and per-judgment agreement on TL;DR summarization samples. Humans agree with GPT-4 about as much as they agree with each other. Each experiment compares a summary from the stated method with a summary from PPO with temperature 0.
Learning from preferences is a powerful, scalable framework for training capable, aligned language models. We have introduced DPO, a simple training paradigm for training language models from preferences without reinforcement learning. Rather than coercing the preference learning problem into a standard RL setting in order to use off-the-shelf RL algorithms, DPO identifies a mapping between language model policies and reward functions that enables training a language model to satisfy human preferences directly, with a simple cross-entropy loss, without reinforcement learning or loss of generality. With virtually no tuning of hyperparameters, DPO performs similarly or better than existing RLHF algorithms, including those based on PPO; DPO thus meaningfully reduces the barrier to training more language models from human preferences.
Limitations & Future Work. Our results raise several important questions for future work. How does the DPO policy generalize out of distribution, compared with learning from an explicit reward function? Our initial results suggest that DPO policies can generalize similarly to PPO-based models, but more comprehensive study is needed. For example, can training with self-labeling from the DPO policy similarly make effective use of unlabeled prompts? On another front, how does reward over-optimization manifest in the direct preference optimization setting, and is the slight decrease in performance in Figure 3-right an instance of it? Additionally, while we evaluate models up to 6B parameters, exploration of scaling DPO to state-of-the-art models orders of magnitude larger is an exciting direction for future work. Regarding evaluations, we find that the win rates computed by GPT-4 are impacted by the prompt; future work may study the best way to elicit high-quality judgments from automated systems. Finally, many possible applications of DPO exist beyond training language models from human preferences, including training generative models in other modalities.
EM gratefully acknowledges funding from a Knight-Hennessy Graduate Fellowship. CF and CM are CIFAR Fellows. This work was supported in part by the Stanford Accelerator for Learning (SAL) and Stanford Institute for Human-Centered Artificial Intelligence (HAI) Generative AI for the Future of Learning seed grant program. The Stanford Center for Research on Foundation Models (CRFM) provided part of the compute resources used for the experiments in this work. This work was supported in part by ONR grant N00014-20-1-2675.