SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rafael Rafailov*†
Archit Sharma*†
Eric Mitchell*†
Stefano Ermon†‡
Christopher D. Manning†
Chelsea Finn†
†Stanford University ‡CZ Biohub
{rafailov,architsh,eric.mitchell}@cs.stanford.edu
While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model. In this paper, we leverage a mapping between reward functions and optimal policies to show that this constrained reward maximization problem can be optimized exactly with a single stage of policy training, essentially solving a classification problem on the human preference data. The resulting algorithm, which we call Direct Preference Optimization (DPO), is stable, performant, and computationally lightweight, eliminating the need for fitting a reward model, sampling from the LM during fine-tuning, or performing significant hyperparameter tuning. Our experiments show that DPO can fine-tune LMs to align with human preferences as well as better than existing methods. Notably, fine-tuning with DPO exceeds RLHF's ability to control sentiment of generations and improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train.
Large unsupervised language models (LMs) trained on very large datasets acquire surprising capabilities [11,7,40,8]. However, these models are trained on data generated by humans with a wide variety of goals, priorities, and skillsets. Some of these goals and skillsets may not be desirable to imitate; for example, while we may want our AI coding assistant to understand common programming mistakes in order to correct them, nevertheless, when generating code, we would like to bias our model toward the (potentially rare) high-quality coding ability present in its training data. Similarly, we might want our language model to be aware of a common misconception believed by  $50\%$  of people, but we certainly do not want the model to claim this misconception to be true in  $50\%$  of queries about it! In other words, selecting the model's desired responses and behavior from its very wide knowledge and abilities is crucial to building AI systems that are safe, performant, and controllable [26]. While existing methods typically steer LMs to match human preferences using reinforcement learning (RL),
*Equal contribution; more junior authors listed earlier.
Figure 1: DPO optimizes for human preferences while avoiding reinforcement learning. Existing methods for fine-tuning language models with human feedback first fit a reward model to a dataset of prompts and human preferences over pairs of responses, and then use RL to find a policy that maximizes the learned reward. In contrast, DPO directly optimizes for the policy best satisfying the preferences with a simple classification objective, without an explicit reward function or RL.
we will show that the RL-based objective used by existing methods can be optimized exactly with a simple binary cross-entropy objective, greatly simplifying the preference learning pipeline.
At a high level, existing methods instill the desired behaviors into a language model using curated sets of human preferences representing the types of behaviors that humans find safe and helpful. This preference learning stage occurs after an initial stage of large-scale unsupervised pre-training on a large text dataset. While the most straightforward approach to preference learning is supervised fine-tuning on human demonstrations of high quality responses, the most successful class of methods is reinforcement learning from human (or AI) feedback (RLHF/RLAIF; [12, 2]). RLHF methods fit a reward model to a dataset of human preferences and then use RL to optimize a language model policy to produce responses assigned high reward without drifting excessively far from the original model. While RLHF produces models with impressive conversational and coding abilities, the RLHF pipeline is considerably more complex than supervised learning, involving training multiple LMs and sampling from the LM policy in the loop of training, incurring significant computational costs.
In this paper, we show how to directly optimize a language model to adhere to human preferences, without explicit reward modeling or reinforcement learning. We propose Direct Preference Optimization (DPO), an algorithm that implicitly optimizes the same objective as existing RLHF algorithms (reward maximization with a KL-divergence constraint) but is simple to implement and straightforward to train. Intuitively, the DPO update increases the relative log probability of preferred to dispreferred responses, but it incorporates a dynamic, per-example importance weight that prevents the model degeneration that we find occurs with a naive probability ratio objective. Like existing algorithms, DPO relies on a theoretical preference model (such as the Bradley-Terry model; [5]) that measures how well a given reward function aligns with empirical preference data. However, while existing methods use the preference model to define a preference loss to train a reward model and then train a policy that optimizes the learned reward model, DPO uses a change of variables to define the preference loss as a function of the policy directly. Given a dataset of human preferences over model responses, DPO can therefore optimize a policy using a simple binary cross entropy objective, without explicitly learning a reward function or sampling from the policy during training.
Our main contribution is Direct Preference Optimization (DPO), a simple RL-free algorithm for training language models from preferences. Our experiments show that DPO is at least as effective as existing methods, including PPO-based RLHF, for learning from preferences in tasks such as sentiment modulation, summarization, and dialogue, using language models with up to 6B parameters.
Self-supervised language models of increasing scale learn to complete some tasks zero-shot [31] or with few-shot prompts [6, 25, 11]. However, their performance on downstream tasks and alignment with user intent can be significantly improved by fine-tuning on datasets of instructions and human-written completions [23, 36, 13, 39]. This 'instruction-tuning' procedure enables LLMs to generalize to instructions outside of the instruction-tuning set and generally increase their usability [13]. Despite the success of instruction tuning, relative human judgments of response quality are often easier to collect than expert demonstrations, and thus subsequent works have fine-tuned LLMs with datasets of human preferences, improving proficiency in translation [18], summarization [38, 48], story-telling [48], and instruction-following [26, 32]. These methods first optimize a neural network reward function for compatibility with the dataset of preferences under a preference model such as the
Bradley-Terry model [5], then fine-tune a language model to maximize the given reward using reinforcement learning algorithms, commonly REINFORCE [44], proximal policy optimization (PPO; [37]), or variants [32]. A closely-related line of work leverages LLMs fine-tuned for instruction following with human feedback to generate additional synthetic preference data for targeted attributes such as safety or harmlessness [2], using only weak supervision from humans in the form of a text rubric for the LLM's annotations. These methods represent a convergence of two bodies of work: one body of work on training language models with reinforcement learning for a variety of objectives [33, 27, 45] and another body of work on general methods for learning from human preferences [12, 19]. Despite the appeal of using relative human preferences, fine-tuning large language models with reinforcement learning remains a major practical challenge; this work provides a theoretically-justified approach to optimizing relative preferences without RL.
Outside of the context of language, learning policies from preferences has been studied in both bandit and reinforcement learning settings, and several approaches have been proposed. Contextual bandit learning using preferences or rankings of actions, rather than rewards, is known as a contextual dueling bandit (CDB; [47, 14]). In the absence of absolute rewards, theoretical analysis of CDBs substitutes the notion of an optimal policy with a von Neumann winner, a policy whose expected win rate against any other policy is at least  $50\%$  [14]. However, in the CDB setting, preference labels are given online, while in learning from human preferences, we typically learn from a fixed batch of offline preference-annotated action pairs [46]. Similarly, preference-based RL (PbRL) learns from binary preferences generated by an unknown 'scoring' function rather than rewards [9, 35]. Various algorithms for PbRL exist, including methods that can reuse off-policy preference data, but generally involve first explicitly estimating the latent scoring function (i.e. the reward model) and subsequently optimizing it [16, 9, 12, 34, 19]. We instead present a single stage policy learning approach that directly optimizes a policy to satisfy preferences.
We review the RLHF pipeline in Ziegler et al. (and later [38, 11, 26]). It usually includes three phases: 1) supervised fine-tuning (SFT); 2) preference sampling and reward learning and 3) RL optimization.
SFT: RLHF typically begins by fine-tuning a pre-trained LM with supervised learning on high-quality data for the downstream task(s) of interest (dialogue, summarization, etc.), to obtain a model  $\pi^{\mathrm{SFT}}$ .
Reward Modelling Phase: In the second phase the SFT model is prompted with prompts  $x$  to produce pairs of answers  $(y_{1}, y_{2}) \sim \pi^{\mathrm{SFT}}(y \mid x)$ . These are then presented to human labelers who express preferences for one answer, denoted as  $y_{w} \succ y_{l} \mid x$  where  $y_{w}$  and  $y_{l}$  denote the preferred and dispreferred completion amongst  $(y_{1}, y_{2})$  respectively. The preferences are assumed to be generated by some latent reward model  $r^{*}(y, x)$ , which we do not have access to. There are a number of approaches used to model preferences, the Bradley-Terry (BT) [5] model being a popular choice (although more general Plackett-Luce ranking models [30, 21] are also compatible with the framework if we have access to several ranked answers). The BT model stipulates that the human preference distribution  $p^{*}$  can be written as:
$$
p ^ {*} \left(y _ {1} \succ y _ {2} \mid x\right) = \frac {\exp \left(r ^ {*} \left(x , y _ {1}\right)\right)}{\exp \left(r ^ {*} \left(x , y _ {1}\right)\right) + \exp \left(r ^ {*} \left(x , y _ {2}\right)\right)}. \tag {1}
$$
Assuming access to a static dataset of comparisons  $\mathcal{D} = \{x^{(i)},y_w^{(i)},y_l^{(i)}\}_{i = 1}^N$  sampled from  $p^*$ , we can parametrize a reward model  $r_{\phi}(x,y)$  and estimate the parameters via maximum likelihood. Framing the problem as a binary classification we have the negative log-likelihood loss:
$$
\mathcal {L} _ {R} \left(r _ {\phi}, \mathcal {D}\right) = - \mathbb {E} _ {\left(x, y _ {w}, y _ {l}\right) \sim \mathcal {D}} \left[ \log \sigma \left(r _ {\phi} \left(x, y _ {w}\right) - r _ {\phi} \left(x, y _ {l}\right)\right) \right] \tag {2}
$$
where  $\sigma$  is the logistic function. In the context of LMs, the network  $r_{\phi}(x,y)$  is often initialized from the SFT model  $\pi^{\mathrm{SFT}}(y\mid x)$  with the addition of a linear layer on top of the final transformer layer that produces a single scalar prediction for the reward value [43]. To ensure a reward function with lower variance, prior works normalize the rewards, such that  $\mathbb{E}_{x,y\sim \mathcal{D}}[r_{\phi}(x,y)] = 0$  for all  $x$ .
RL Fine-Tuning Phase: During the RL phase, we use the learned reward function to provide feedback to the language model. In particular, we formulate the following optimization problem
$$
\max  _ {\pi_ {\theta}} \mathbb {E} _ {x \sim \mathcal {D}, y \sim \pi_ {\theta} (y \mid x)} \left[ r _ {\phi} (x, y) \right] - \beta \mathbb {D} _ {\mathrm {K L}} \left[ \pi_ {\theta} (y \mid x) \right| | \pi_ {\text {r e f}} (y \mid x) ] \tag {3}
$$
where  $\beta$  is a parameter controlling the deviation from the base reference policy  $\pi_{\mathrm{ref}}$ , namely the initial SFT model  $\pi^{\mathrm{SFT}}$ . In practice, the language model policy  $\pi_{\theta}$  is also initialized to  $\pi^{\mathrm{SFT}}$ . The added constraint is important, as it prevents the model from deviating too far from the distribution on which the reward model is accurate, as well as maintaining the generation diversity and preventing mode-collapse to single high-reward answers. Due to the discrete nature of language generation, this objective is not differentiable and is typically optimized with reinforcement learning. The standard approach [48, 38, 11, 26] has been to construct the reward function  $r(x,y) = r_{\phi}(x,y) - \beta (\log \pi_{\theta}(y|x) - \log \pi_{\mathrm{ref}}(y|x))$ , and maximize using PPO [37].
4 Direct Preference Optimization
Motivated by the challenges of applying reinforcement learning algorithms on large-scale problems such as fine-tuning language models, our goal is to derive a simple approach for policy optimization using preferences directly. Unlike prior RLHF methods, which learn a reward and then optimize it via RL, our approach bypasses the reward modeling step and directly optimizes a language model using preference data. As we will describe next in detail, our key insight is to leverage an analytical mapping from reward functions to optimal policies, which enables us to transform a loss function over reward functions into a loss function over policies. This change-of-variables approach allows us to skip the explicit reward modeling step, while still optimizing under existing models of human preferences, such as the Bradley-Terry model. In essence, the policy network represents both the language model and the reward.
Deriving the DPO objective. We start with the same RL objective as prior work, Eq. 3 under a general reward function  $r$ . Following prior work [29, 28, 17, 15], it is straightforward to show that the optimal solution to the KL-constrained reward maximization objective in Eq. 3 takes the form:
$$
\pi_ {r} (y \mid x) = \frac {1}{Z (x)} \pi_ {\text {r e f}} (y \mid x) \exp \left(\frac {1}{\beta} r (x, y)\right), \tag {4}
$$
where  $Z(x) = \sum_{y} \pi_{\mathrm{ref}}(y \mid x) \exp \left(\frac{1}{\beta} r(x, y)\right)$  is the partition function. See Appendix A.1 for a complete derivation. Even if we use the MLE estimate  $r_{\phi}$  of the ground-truth reward function  $r^{*}$ , it is still expensive to estimate the partition function  $Z(x)$  [17, 15], which makes this representation hard to utilize in practice. However, we can rearrange Eq.4 to express the reward function in terms of its corresponding optimal policy  $\pi_r$ , the reference policy  $\pi_{\mathrm{ref}}$ , and the unknown partition function  $Z(\cdot)$ . Specifically, we first take the logarithm of both sides of Eq.4 and then with some algebra we obtain:
$$
r (x, y) = \beta \log \frac {\pi_ {r} (y \mid x)}{\pi_ {\text {r e f}} (y \mid x)} + \beta \log Z (x). \tag {5}
$$
We can apply this reparameterization to the ground-truth reward  $r^*$  and corresponding optimal model  $\pi^*$ . Fortunately, the Bradley-Terry model depends only on the difference of rewards between two completions, i.e.,  $p^*(y_1 \succ y_2 \mid x) = \sigma(r^*(x, y_1) - r^*(x, y_2))$ . Substituting the reparameterization in Eq. 5 for  $r^*(x, y)$  into the preference model Eq. 1, the partition function cancels, and we can express the human preference probability in terms of only the optimal policy  $\pi^*$  and reference policy  $\pi_{\mathrm{ref}}$ . Thus, the optimal RLHF policy  $\pi^*$  under the Bradley-Terry model satisfies the preference model:
$$
p ^ {*} \left(y _ {1} \succ y _ {2} \mid x\right) = \frac {1}{1 + \exp \left(\beta \log \frac {\pi^ {*} \left(y _ {2} \mid x\right)}{\pi_ {\text {r e f}} \left(y _ {2} \mid x\right)} - \beta \log \frac {\pi^ {*} \left(y _ {1} \mid x\right)}{\pi_ {\text {r e f}} \left(y _ {1} \mid x\right)}\right)} \tag {6}
$$
The derivation is in Appendix A.2 While Eq. 6 uses the Bradley-Terry model, we can similarly derive expressions under the more general Plackett-Luce models [30, 21], shown in Appendix A.3
Now that we have the probability of human preference data in terms of the optimal policy rather than the reward model, we can formulate a maximum likelihood objective for a parametrized policy  $\pi_{\theta}$ . Analogous to the reward modeling approach (i.e. Eq. 2), our policy objective becomes:
$$
\mathcal {L} _ {\mathrm {D P O}} \left(\pi_ {\theta}; \pi_ {\text {r e f}}\right) = - \mathbb {E} _ {(x, y _ {w}, y _ {l}) \sim \mathcal {D}} \left[ \log \sigma \left(\beta \log \frac {\pi_ {\theta} \left(y _ {w} \mid x\right)}{\pi_ {\text {r e f}} \left(y _ {w} \mid x\right)} - \beta \log \frac {\pi_ {\theta} \left(y _ {l} \mid x\right)}{\pi_ {\text {r e f}} \left(y _ {l} \mid x\right)}\right) \right]. \tag {7}
$$
This way, we simultaneously bypass the explicit reward modeling step while also avoiding the need to perform reinforcement learning optimization. Moreover, since our procedure is equivalent to fitting
a reparametrized Bradley-Terry model, it enjoys certain theoretical properties, such as consistencies under suitable assumption of the preference data distribution [4]. In Section 5 we further discuss theoretical properties of DPO in relation to other works.
What does the DPO update do? For a mechanistic understanding of DPO, it is useful to analyze the gradient of the loss function  $\mathcal{L}_{\mathrm{DPO}}$ . The gradient with respect to the parameters  $\theta$  can be written as:
$$
\begin{array}{l} \nabla_ {\theta} \mathcal {L} _ {\mathrm {D P O}} \left(\pi_ {\theta}; \pi_ {\text {r e f}}\right) = \\ - \beta \mathbb {E} _ {(x, y _ {w}, y _ {l}) \sim \mathcal {D}} \left[ \underbrace {\sigma (\hat {r} _ {\theta} (x , y _ {l}) - \hat {r} _ {\theta} (x , y _ {w}))} _ {\text {h i g h e r w e i g h t w h e n r e w a r d e s t i m a t e i s w r o n g}} \left[ \underbrace {\nabla_ {\theta} \log \pi (y _ {w} \mid x)} _ {\text {i n c r e a s e l i k e l i h o o d o f} y _ {w}} - \underbrace {\nabla_ {\theta} \log \pi (y _ {l} \mid x)} _ {\text {d e c r e a s e l i k e l i h o o d o f} y _ {l}} \right] \right], \\ \end{array}
$$
where  $\hat{r}_{\theta}(x,y) = \beta \log \frac{\pi_{\theta}(y|x)}{\pi_{\mathrm{ref}}(y|x)}$  is the reward implicitly defined by the language model  $\pi_{\theta}$  and reference model  $\pi_{\mathrm{ref}}$  (more in Section 5). Intuitively, the gradient of the loss function  $\mathcal{L}_{\mathrm{DPO}}$  increases the likelihood of the preferred completions  $y_w$  and decreases the likelihood of dispreferred completions  $y_l$ . Importantly, the examples are weighed by how much higher the implicit reward model  $\hat{r}_{\theta}$  rates the dispreferred completions, scaled by  $\beta$ , i.e., how incorrectly the implicit reward model orders the completions, accounting for the strength of the KL constraint. Our experiments suggest the importance of this weighting, as a naive version of this method without the weighting coefficient can cause the language model to degenerate (Appendix Table 3).
DPO outline. The general DPO pipeline is as follows: 1) Sample completions  $y_{1}, y_{2} \sim \pi_{\mathrm{ref}}(\cdot \mid x)$  for every prompt  $x$ , label with human preferences to construct the offline dataset of preferences  $\mathcal{D} = \{x^{(i)}, y_w^{(i)}, y_l^{(i)}\}_{i=1}^N$  and 2) optimize the language model  $\pi_{\theta}$  to minimize  $\mathcal{L}_{\mathrm{DPO}}$  for the given  $\pi_{\mathrm{ref}}$  and  $\mathcal{D}$  and desired  $\beta$ . In practice, one would like to reuse preference datasets publicly available, rather than generating samples and gathering human preferences. Since the preference datasets are sampled using  $\pi^{\mathrm{SFT}}$ , we initialize  $\pi_{\mathrm{ref}} = \pi^{\mathrm{SFT}}$  whenever available. However, when  $\pi^{\mathrm{SFT}}$  is not available, we initialize  $\pi_{\mathrm{ref}}$  by maximizing likelihood of preferred completions  $(x, y_w)$ , that is,  $\pi_{\mathrm{ref}} = \arg \max_{\pi} \mathbb{E}_{x, y_w} \sim \mathcal{D}[\log \pi(y_w \mid x)]$ . This procedure helps mitigate the distribution shift between the true reference distribution which is unavailable, and  $\pi_{\mathrm{ref}}$  used by DPO. Further details related to the implementation and hyperparameters can be found in Appendix B.
5 Theoretical Analysis of DPO
In this section, we give further interpretation of the DPO method, provide theoretical backing, and relate advantages of DPO to issues with actor critic algorithms used for RLHF (such as PPO [37]).
5.1 Your Language Model Is Secretly a Reward Model
DPO is able to bypass both explicit reward estimation and RL to learn the policy using a single maximum likelihood objective. However, the optimization objective Eq. 5 is equivalent to a Bradley-Terry model with a reward function  $r^{*}(x,y) = \beta \log \frac{\pi_{\theta}^{*}(y|x)}{\pi_{\mathrm{ref}}(y|x)}$  and we optimize our parametric model  $\pi_{\theta}$ , equivalently to the reward model optimization in Eq. 2 under the this change of variables. In this section we will build the theory behind this reparameterization, show that it does not constrain the class of learned reward models, and allows for the exact recovery of the optimal policy. We begin with by defining an equivalence relation between reward functions.
Definition 1. We say that two reward functions  $r(x,y)$  and  $r'(x,y)$  are equivalent iff  $r(x,y) - r'(x,y) = f(x)$  for some function  $f$ .
It is easy to see that this is indeed an equivalence relation, which partitions the set of reward functions into classes. We can state the following two lemmas:
Lemma 1. Under the Plackett-Luce, and in particular the Bradley-Terry, preference framework, two reward functions from the same class induce the same preference distribution.
Lemma 2. Two reward functions from the same equivalence class induce the same optimal policy under the constrained RL problem.
The proofs are straightforward and we defer them to Appendix A.5. The first lemma is a well-known under-specification issue with the Plackett-Luce family of models [30]. Due to this under-specification,
we usually have to impose additional identifiability constraints to achieve any guarantees on the MLE estimates from Eq. 2 [4]. The second lemma states that all reward functions from the same class yield the same optimal policy, hence for our final objective, we are only interested in recovering an arbitrary reward function from the optimal class. We prove the following Theorem in Appendix A.6:
Theorem 1. Under mild assumptions, all reward classes consistent with the Plackett-Luce (and Bradley-Terry in particular) models can be represented with the reparameterization  $r(x,y) = \beta \log \frac{\pi(y|x)}{\pi_{ref}(y|x)}$  for some model  $\pi(y|x)$  and a given reference model  $\pi_{ref}(y|x)$ .
Proof Sketch. Consider any reward function  $r(x, y)$ , which induces a corresponding optimal model  $\pi_r(y \mid x)$ , specified by Eq. 4. We will show that a reward function from the equivalence class of  $r$  can be represented using the reparameterization given above. We define the projection  $f$  as
$$
f (r; \pi_ {\mathrm {r e f}}, \beta) (x, y) = r (x, y) - \beta \log \sum_ {y} \pi_ {\mathrm {r e f}} (y \mid x) \exp \left(\frac {1}{\beta} r (x, y)\right) \tag {8}
$$