Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Authors: Rafael Rafailov
Archit Sharma
Eric Mitchell
Christopher D Manning
Stefano Ermon
Chelsea Finn
Venue: NeurIPS 2023 oral (2023)
Keywords: reinforcement learning from human feedback
language models
RLHF
preferences
TL;DR: Fine-tuning with RLHF is complicated; we show that it doesn't need to be.

Abstract: While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model. In this paper, we leverage a mapping between reward functions and optimal policies to show that this constrained reward maximization problem can be optimized exactly with a single stage of policy training, essentially solving a classification problem on the human preference data. The resulting algorithm, which we call Direct Preference Optimization (DPO), is stable, performant, and computationally lightweight, eliminating the need for fitting a reward model, sampling from the LM during fine-tuning, or performing significant hyperparameter tuning. Our experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods. Notably, fine-tuning with DPO exceeds RLHF's ability to control sentiment of generations and improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train.

Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafael Rafailoy
Archit Sharma
Eric Mitchell
Stefano Ermon
Christopher Manning
Chelsea Finn'
Stanford University $CZ Biohub (rafailov, architsh, eric. mi mitcchell@@ss wtt.icssssaaoo..r.oo.ooo edu
Abstract
While large-scale unsupervised language models (LMs) learn broad world knowl- edge and some reasoning skills, achieving precise control their behavior difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality model generations and fine-tune the unsupervised LM to align with these prefer- ences, often with reinforcement learning from human feedback (RLHF). However, RLHF is complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model. In this paper, we leverage a mapping between reward functions and optimal policies to show that this constrained reward maxi- mization problem can be optimized exactly with a single stage of policy training, essentially solving classification problem the human preference data. The resulting algorithm, which we call Direct Preference Optimization (DPO), is stable, performant, and computationally lightweight, eliminating the need for fitting reward model, sampling from the LM during fine-tuning, or performing significant hyperparameter tuning. Our experiments show that DPO can fine-tune LMs align with human preferences as well as or better than existing methods. Notably, fine-tuning with DPO exceeds RLHF's ability control sentiment of generations and improves response quality in summarization and single-turn dialogue while being substantially simpler implement and train.
Introduction
Large unsupervised language models (LMs) trained very large datasets acquire surprising capabili- ties 711 40 &
However, these models are trained on data generated by humans with wide variety of goals. priorities, and skillsets. Some of these goals and skillsets may be desirable imitate; for example, while we may want our AI coding assistant to understand common programming mistakes order correct them, nevertheless, when generating code, we would like to bias our model toward the (potentially rare) high-quality coding ability present in its training data. Similarly, our language model to aware of common misconception believed by 50% of people, but we certainly do not want the model to claim this misconception to be true in 50% of queries about it! other words, selecting the model's desired responses and behavior from its very wide knowledge and abilities is crucial to building AI systems that are safe, performant, and controllable [26). While existing methods typically steer LMs to match human preferences using reinforcement learning (RL),
might want
"qqual contribution; junior authors listed earlier.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Reinforcement Learning from Human Feedback (RLHF)
Direct Preference Optimization (DPO)
"eweeee about
label rewards
Use
reward model
LM policy
final LM
preference data
maximum
sample completions
preference data
maximum Likelihood
Likelihood
reinforcement learning
Figure I: DPO optimizes for human preferences while avoiding reinforcement learning, Existing methods for fine-tuning language models with human feedback first fit reward model dataset of prompts and human preferences over pairs responses, and then use RL find policy that maximizes the leamed reward. In contrast, DPO directly optimizes for the policy best satisfying the preferences with a simple classification objective, without an explicit reward function or RL.
we will show that the RL-based objective used by existing methods can be optimized exactly with a simple binary cross-entropy objective, greatly simplifying the preference learning pipeline.
At a high level, existing methods instill the desired behaviors into a language model using curated sets of human preferences representing the types of behaviors that humans find safe and helpful. This preference learning stage occurs after an initial stage of large-scale unsupervised pre-training on large text dataset. While the most straightforward approach to preference learning is supervised fine-tuning on human demonstrations of high quality responses, the most successful class of methods reinforcement learning from human (or AI) feedback (RLHF/RLAIF; 112 21. RLHF methods fit a reward model to a dataset of human preferences and then use RL to optimize a language model policy produce responses assigned high reward without drifting excessively far from the original model. While RLHF produces models with impressive conversational and coding abilities, the RLHF pipeline considerably more complex than supervised learning. involving training multiple LMs and sampling from the LM policy in the loop of training, incurring significant computational costs.
In this paper, we show how to directly optimize a language model adhere to human preferences, without explicit reward modeling or reinforcement learning. We propose Direct Preference Optimiza- tion (DPO), an algorithm that implicitly optimizes the same objective existing RLHF algorithms (reward maximization with a KI Kl--ivergence constraint) but is simple to implement and straight- forward to train. Intuitively, the DPO update increases the relative log probability of preferred to dispreferred responses, but it incorporates a dynamic, per-example importance weight that prevents the model degeneration that we find occurs with naive probability ratio objective. Like existing algorithms, DPO relies on a theoretical preference model (such as the Bradley- -eerryyyTeryy model; (5) that measures how well a given reward function aligns with empirical preference data. However, while existing methods use the preference model to define a preference loss to train a reward model and then train a policy that optimizes the learned reward model, DPO uses change of variables to define the preference loss as a function of the policy directly. Given a dataset of human preferences over model responses, DPO can therefore optimize policy using a simple binary cross entropy objective, without explicitly learning a reward function or sampling from the policy during training.
Our main contribution is Direct Preference Optimization (DPO), a simple RL-free algorithm for training language models from preferences. Our experiments show that DPO is at least as effective as existing methods, including PPO-based RLHF, for learning from preferences in tasks such as sentiment modulation, summarization, and dialogue, using language models with up to 6B parameters.
2 Related Work
Self-supervised language models of increasing scale learn to complete some tasks zero-shot ti or with few-shot prompts 6 |25
[. However, their performance on downstream tasks and alignment
with user intent can be significantly improved by fine-tuning on datasets of instructions and human-
written completions 2 Bo
procedure enables LLMs to generalize [3]. Despite the success of instruction tuning, relative human judgments of response quality are often easier to collect than expert demonstrations, and thus subsequent works have fine-tuned LLMs with datasets of
13 39 This instruction-tuniin
to instructions outside of the instruction-tunien
set and generally increase their usability
human preferences, improving proficiency in translation [18], summarization [38 48). story-telling
48 and dsstuctioo-followinn function for compatibility with the dataset of preferences under preference model such as the
20 52 These methods first optimize neural network reward
2

---

Bradley- Terry model 15. then fine-tune a language model to maximize the given reward using reinforcement learning algorithms, commonly REINFORCE ++ proximal policy optimization (PPO; (37)) ) or variants 32 A closely-related line of work leverages L.MSs fine-tuned for instruction following with human feedback to generate additional synthetic preference data for targeted attributes such as safety or harmlessness (2), using only weak supervision from humans in the form of a text rubric for the LLM's annotations. These methods represent a convergence of two bodies of work: one body of work on training language models with reinforcement learning for variety and another body of work on general methods for learning from human Despite the appeal of using relative human preferences, fine-tuning large language models with reinforcement learning remains a major practical challenge; this work provides theoretically- justtfffeed approach to optimizing relative preferences without RL.
of objectives 33 27
45
preferences
12 19
Outside of the context language, learning policies from preferences has been studied in both bandit and reinforcement learning settings, and several approaches have been proposed. Contextual bandit learning using preferences or rankings of actions, rather than rewards, known as a contextual dueling bandit (CDB; 47 14). In the absence of absolute rewards, theoretical analysis of CDBs substitutes the notion an optimal policy with a von Neumann winner, a policy whose expected win rate against any other policy is at least 50% [14] However, in the CDB setting, preference labels are given online, while in learning from human preferences, we typically learn from a fixed batch of offline preference-ad annorttte action pairs [6). Similarly, preference- based RL (PbRL) learns from binary preferences generated by an unknown 'scoring' function rather than rewards M B5I. Various algorithms for PbRL exist, including methods that can reuse off-policy preference data, but generally involve first explicitly estimating the latent scoring function (i.e. the reward model) and subsequently
optimizing it [6. 9 12
34 19 We instead present single stage policy learning approach that directly optimizes a policy to satisfy preferences.
3 Preliminaries
We review the RLHF pipeline in Ziegler al. (and later 38 1 26 It usually includes three phases:
1) supervised fine-tuning (SFT); 2) preference sampling and reward learning and 3) RL optimization.
SFT: RLHF typically begins by fine-tuning pre-trained LM with supervised learning on high-quality data for the downstream task(s) of interest (dialogue, summarization, etc.), to obtain a model SPT
Reward Modelling Phase: In the second phase the SFT model is prompted with prompts x to produce pairs of answers (y1, 122 ~ x TSPT (y x).. These are then presented to human labelers who express preferences for one answer, denoted as yu > 3t | x where Ya, and yı denotes the preferred and dispreferred completion amongst (y) 1/2) respectively. The preferences are assumed to be generated by some latent reward model r" "y, .).. which we do not have access to. There are number of approaches used model preferences, the Bradley-Terry (BT) 5 model being a popular choice (although more general Plackett-Luce ranking models 30 21 are also compatible with the framework if we have access to several ranked answers). The BT model stipulates that the human preference distribution p" can be written as:
exp (r (z, 91)) cxp (r" (x, + exp (r* (I. U2))
p² (v1 > 9/2 xx)
Assuming access to static dataset of comparisons D {.((9, yo vi Jim1 sampled from p", can parametrize reward model ro(2, v) and estimate the parameters via maximum likelihood. Framing the problem as a binary classification we have the negative log-likelihood loss:
N
CR(TS, D) - -E -B....... [log (ro(z, yw) gx)))
(2)
where a is the logistic function. In the context of LMs, the network (2 y) is often initialized from the SFT model ,. ,.SET(y æ) with the addition of a linear layer on top of the final transformer layer that produces single scalar prediction for the reward value 4 To ensure a reward function with lower variance, prior works normalize the rewards, such that Ey [r v)] = 0 for all x.
During the RL phase, we use the learned reward function to provide feedback to the language model. In particular, we formulate the following optimization problem
RL Fine-Tunne Tuning Phase:
max (.y.D.yll (To (x. v) Ml.. To (yy (x) Teer(y) xx) 89
3

---

where 3 is parameter controlling the deviation from the base reference policy Tiet. namely the ini- tial SFT model T In practice, the language model policy To also initialized T The added constraint is important, as it prevents the model from deviating too far from the distri- bution on which the reward model is accurate, as well as maintaining the generation diversity and preventing mode-collapse to single high-reward answers. Due to the discrete nature of lan- guage generation, this objective is not differentiable and is typically optimized with reinforce- ment learning. The standard approach [48 38 II 26 has been to construct the reward function r(z, 2) = (T y) - B(ogg ( x) log Tref r)), and maximize using PPO 37
Direct Preference Optimization
Motivated by the challenges of applying reinforcement learning algorithms large-scale problems such as fine-tuning language models, our goal is to derive a simple approach for policy optimization using preferences directly. Unlike prior RLHF methods, which learn a reward and then optimize it via RL, our approach bypasses the reward modeling step and directly optimizes a language model using preference data. As we will describe next detail, our key insight is leverage analytical mapping from reward functions to optimal policies, which enables us to transform a loss function over reward functions into loss function over policies. This change-of-variable us to skip the explicit reward modeling step, while still optimizing under existing models of human preferences, such the Bradley-Tenie -erry model. In essence, the policy network represents both the language model and the reward.
approach allows
Deriving the DPO objective. We start with the same RL objective prior work, Eq. 3 under general reward function r. Following prior work (29] 28 17 [5], it is straightforward to show that the optimal solution to the KL-constrained reward maximization objective in Eq. 3 Itakes the form:
1 Z(x) Trer(y x:) exp (fr(e..))
(y x)
where Z(x) E, Ter x) exp (ar(2. is the partition function. See Appendix 1 for complete derivation. Even we use the MLE estimate To of the ground-truth reward function r it is still expensive estimate the partition function ((x) [17 [3), which makes this representation hard to utilize in practice. However, we can rearrange Eq. to express the reward function in terms of its corresponding optimal policy k,, the reference policy Tref, and the unknown partition function Z--).
Specifically, we first take the logarithm of both sides of Eq. 4 and then with some algebra we obtain:
r(z, y) = Blg (y x) x))
+ Blog Z(x).
(5)
We apply this reparameterization to the ground-truth reward rM and corresponding optimal model T² Fortunately, the Bradley-Tene model depends only on the difference of rewards between two completions, i.e., p" (y) y/ x)) =xr((z,)) 91)
((r,y2)), Substituting the reparameterization in Eq. 5) for rxx y) into the preference model Eq. [ the partition function cancels, and we can express the human preference probability in terms of only the optimal policy л" and reference policy Tef. Thus, the optimal RLHF policy IT" under the Bradley-Ters Terry model satisfies the preference model:
p" (yn 12 v21xx
1 exexp 3 log
(valm) Res ())
3 log
(21 Tar( 20)
similarly The derivation is in Appendix A.2 While Eq. uses the Bradley-Tern -erry model,
derive expressions under the more general Plackett- Luce models (30 21 shown in Appendix A.3
Now that we have the probability of human preference data in terms of the optimal policy rather than the reward model, we can formulate a maximum likelihood objective for parametrized policy ma- Analogous to the reward modeling approach (i.e. Eq. 2). our policy objective becomes:
(DPO Te))) = - -B.r..... log a log To(2oo x)
To (yr
x
B log
(7)
(Ne (x)
Teet (1
This way, we simultaneously bypass the explicit reward modeling step while also avoiding the need to perform reinforcement learning optimization. Moreover, since our procedure is equivalent to fitting

---

reparametrized Bradley-Tecd Terry model, it enjoys certain theoretical properties, such as consistencies under suitable assumption of the preference data distribution In Section [ we further discuss theoretical properties of DPO in relation to other works.
What does the DPO update do? For mechanistie understanding of DPO, it is useful to analyze the gradient of the loss function DPO. The gradient with respect to the parameters can be written as:
V VDLDOO (oo Ted))
]].
(fo( to (z, Ylw)
Vo log πT (1h (z)
Vo log (3 1
higher weight when reward estimate is wrong
increase likelihooo yi
docrease likelihood yu
where f0 y) 8 log Tyr(y is the reward implicitly defined by the language model Te and refer- ence model Ther (more in Section Sp. Intuitively, the gradient of the loss function LDoO increases the likelihood of the preferred completions Yor and decreases the likelihood of dispreferred completions yt- Importantly, the examples are weighed by how much higher the implicit reward model n'g rates the dispreferred completions, scaled by 3, i.c how incorrectly the implicit reward model orders the completions, accounting for the strength of the KL constraint. Our experiments suggest the importance of this weighting, as a naïve version of this method without the weighting coefficient can cause the language model to degenerate (Appendix Table 3
ma (vlx)
DPO outline. The general DPO pipeline is as follows: 1) Sample completions y1. V/ ~ Teel( xx) for every prompt x, label with human preferences to construct the offline dataset of preferences D = (x(6), yie y1 (1) (6)
Ji=1 and 2) optimize the language model wa minimize CDPO for the given Tet and DD and desired B. In practice, one would like to reuse preference datasets publicly available, rather than generating samples and gathering human preferences. Since the preference datasets are sampled using π SFT we initialize Tre SST whenever available. However, when x is not available, we initialize Tet by maximizing likelihood of preferred completions (x,, Vve), that is, Feer arg max, Ex.ya ~D (U x)]. This procedure helps mitigate the distribution shift between the true reference distribution which unavailable, and Tee used by DPO. Further details related to the implementation and hyperparameters can be found in Appendix B
5 Theoretical Analysis of DPO
In this section, we give further interpretation of the DPO method, provide theoretical backing, and relate advantages of DPO to issues with actor critic algorithms used for RLHF (such as PPO (37))
5.1 Your Language Model Is Secretly a Reward Model
DPO is able to bypass both explicit reward estimation and RL to learn the policy using a single maximum likelihood objective. However, the optimization objective Eq. 5 is equivalent to Bradley- Terry model with a reward function r" r"(x, 3) = B log tar(pla)
=āly)a) and we optimize our parametric model Te, equivalently to the reward model optimization in Eq. uuder the this change of variables. this section will build the theory behind this reparameterization, show that it does not constrain the class of learned reward models, and allows for the exact recovery of the optimal policy. We begin with by defining equivalence relation between reward functions.
Definition 1. We say that two reward functions r(æ, y) and r'(x,)) are equivalent iff r(z,)) 8) r' v) f(x) for some function f.
It is easy to see that this is indeed an equivalence relation, which partitions the set of reward functions into classes. We can state the following two lemmas:
Lemma 1. Under the Plackett-Lucc and in particular the Bradley-Terry preference framework, two reward functions from the same class induce the same preference distribution.
Lemma 2. Two reward functions from the same equivalence class induce the same optimal policy under the constrained RI. problem.
The proofs are straightforward and we defer them to Appendix A.5
The first lemma a well-known
under-specification issue with the Plakett-Luce family of models 30
Due to this under-specification.
5

---

we usually have to impose additional identifiability constraints to achieve any guarantees on the MLE estimates from Eq. 2 4 The second lemma states that all reward functions from the same class yield the same optimal policy, hence for our final objective, we are only interested in recovering an arbitrary reward function from the optimal class. We prove the following Theorem in Appendix A.6
Theorem 1. Under mild assumptions, all reward classes consistent with the Plackett-Luce (and Bradley-Ters (erry in particular) models can be represented with the reparameterization s(ylz) for some model (t x)) and a given reference model TeyKy xz). r(z,y) 3 log mx)
Proof Sketch. Consider any reward function r(z, y). which induces corresponding optimal model (r z). specified by Eq. f We will show that a reward function from the equivalence class of can be represented using the reparameterizatio given above. We define the projection f as
[(1 Trets (), r(x, v) Bogg Σ Trery (x) exp (3)(0.v)
(8)
The operator f simply normalizes the reward function with the logarithm the partition function of π... Since the added normalization term is only function of the prefix æ, f(r Teet (z, y) is reward function in the equivalence class of r(x, y). Finally, replacing r with the RHS of Eq. (which
holds for any reward function), we have (r; Tref, (2,, y) 3 log Fwily
m-(vlz) That is, the projection
produces member of the equivalence class of with the desired form, and we do not lose any generality in our reward model from the proposed reparameterization.
We can alternatively view Theorem I as specifying exactly which reward function within each equivalence class the DPO reparameterization selects, that is, the reward function satisfying:
Σ Teerl x)) exp (f)(e.))
(y)la). using Thmn D reearaam.
ie., Tt/ x)) is a valid distribution (probabilities are positive and sum to 1). However, following Eq. 4 we can that Eq. is the partition function of the optimal policy induced by the reward function r(r, y). The key insight of the DPO algorithm is that we can impose certain constraints on the under-constrained Plackett-Luce (and Bradley-Terry in particular) family of preference models, such that we preserve the class of representable reward models, but explicitly make the optimal policy in Eq. f analytically tractable for all prompts x.
5.2 Instability of Actor-Critic Algorithms
We can also use our framework to diagnose instabilities with standard actor-critic algorithms used for the RLHF, such PPO. We follow the RLHF pipeline and focus on the RL fine-tuning step outlined in Section 3 We can draw connections to the control as inference framework (20) for the constrained RL problem outlined in g We assume a parameterized model To xx) and minimize
DeL (oolTTo(ll (x))) || (y x)] where x² is the optimal policy from Eq. 7 Iinduced by the reward function
(o(y, .). With some algebra this leads to the optimization objective:
(Fraeesw)) log
Te (3 x))
max E (vl=)
To (T) - 3 log Σ
Tref exp
(10)
Teet (y
31
This is the same objective optimized in prior works 48 38 for the reward class of To- In this setting, we can interpret the normalization term in f(ro, Wret, .) as the soft value function of the reference policy Tea- While this term does not affect the optimal solution, without it, the policy gradient of the objective could have high variance, making learning unstable. We can accommodate for the normalization term using a learned value function, but that can also be difficult to optimize. Alternatively, prior works have normalized rewards using human completion baseline, essentially single sample Monte-Carke estimate the normalizing term. In contrast the DPO reparameterization yields reward function that does not require any basclines.
26 using the DPO-equivalent reward

---

IMDb Sentiment Generation
TL:DR Summarization Win Rate Reference
1.0
o0
DTO
GP%
SF1
Bist
0.--
0.8
05
Reward
rate
0.
0.6-
0.-
DPO (Ours)
GT impl.) PPO-GT (TRL)
0.4
PPO impl))
Un ike kellhood
Proferred-FT
01-
255
50
10.5 125
15.0
17.
0.20
0.5
0.50
0.75
1.00
KL(ma m
Sampling temperature
Figure 2: Left. The frontier of expected reward vs KL to the reference policy. DPO provides the highest expected reward for all KL values, demonstrating the quality of the optimization. Right. TL:DR summarization win rates vs. human-written summaries, using GPT-4 as evaluator. DPO exceeds PPO's best-case performance on summarization, while being more robust to changes in the sampling temperature.
Experiments
In this section, empirically evaluate DPO's ability train policies directly from preferences. First, in a well-controleed text- -ex-eennratiion setting, we ask: how efficiently does DPO trade off maximizing reward and minimizing Kivdiveree with the reference policy, compared to common preference learning algorithms such as PPO? Next, we evaluate DPO's performance on larger models and more difficult RLHF tasks, including summarization and dialogue. We find that with almost tuning of syperparameters, DPO tends to perform as well or better than strong baselines like RLHF with PPO well as returning the best of N sampled trajectories under a learned reward function. Before presenting these results, we describe the experimental set-up; additional details are in Appendix C
Tasks. Our experiments explore three different open-ended text generation tasks. For all experiments, (i)
algorithms learn a policy from a dataset of preferences D = {2((0, , (
"/ (1)
1-1" In controlled sentiment generation, is a prefix of a movie review from the IMDb dataset [22] and the policy must generate y with positive sentiment. In order to perform a controlled evaluation, for this experiment we generate preference pairs over generations using a pre-trained sentiment classifier, where p(positive x, ym.) > p(positive x, y1). J. For SFT, we fine-tune GPT-2-large until convergence on reviews from the train split of the IMDB dataset (further details in App C.1 In summarization, & is a forum post from Reddit; the policy must generate a summary of the main points in the post. Following prior work, use the Reddit TL;DR summarization dataset [41] along with human preferences gathered by SStiennon et al. We use an SFT model fine-tuned on human-written forum post summaries with the TRLX 42 framework for RLHE. The human preference dataset was gathered by Stiennon et al.) on samples from a different, but similarly-trained SFT model. Finally, in single-turn dialogue, x a human query, which may be anything from a question about astrophysies to request for relationship advice. A policy must produce an engaging and helpful response y to user's query; we use the Anthropic Helpful and Harmless dialogue dataset M. containing 17kk dialogues between human and an automated assistant. Each transcript ends with pair of responses generated by a large (although unknown) language model along with a preference label denoting the human-preferee response. In this setting, no pre-trained SFT model is available; we therefore fine-tune an off-the-shelf language model on only the preferred completions to form the SFT model.
Evaluation. Our experiments use two different approaches to evaluation. In order to analyze the effectiveness of each algorithm in optimizing the constrained reward maxinization objective, in the controlled sentiment generation setting we evaluate each algorithm by its frontier of achieved reward and KL-divergence from the reference policy; this frontier is computable because we have acccess to the ground-trum -ruth reward function (a sentiment classifier). However, in the real world, the ground truth reward function is not known; therefore, we evaluate algorithms with their win rate against baseline policy, using GPTT4 as proxy for human evaluation of summary quality and response helpfulness in the summarization and single-turn dialogue settings, respectively. For summarization, we use reference summaries in the test set as the baseline; for dialogue, we use the preferred response in the

---

Anthropic-HH Dialogue Win Rate vs Chosen
Dialogue Win Rate Evolution
0.20
0.6
0.3
0.60
Win rate
Win rate
0.55
0.50
0.43
Preferred-FT Pythia-2.88
0:35
0.1
Best of 128
(temp
0.25
0.50
6,75
1.09
0 300
coc
900 1200 1500 1800 2120 2400 2700 1000 sabo
Sampling temperature
Fine-tuning step
Figure 3: Left. Win rates computed by GPT-4 for Anthropic-HH one-step dialogue; DPO the only method that improves over chosen summaries in the Anthropic-HH test set. Right. Win rates for different sampling temperatures over the course of training. DPO's improvement over the dataset labels is fairly stable over the course of training for different sampling temperatures.
test dataset as the baseline. While existing studies suggest LMs can be better automated evaluators than existing metrics (IO). we conduct a human study to justify our usage of GPT-4 for evaluation in Sec. 6..4 We find GPT-4 judgments correlate strongly with humans, with human agreement with GPT-4 typically similar or higher than inter-human annotator agreement.
Methods. In addition to DPO, we evaluate several existing approaches to training language models to adhere human preferences. Most simply, we explore zero-shot prompting with GPT-J [3] in the summarization task and 2-shot prompting with Pythia-2.8B K in the dialogue task. In addition, we evaluate the SFT model as well as Preferred-FT, which is a model fine-tuned with supervised learning on the chosen completion Vo from either the SFT model (in controlled sentiment and summarization) or a generic LM (in single-turn dialogue). Another sgeudo-supervisee method Unlikelihood, which simply optimizes the policy maximize the probability assigned to Yw and minimize the probability assigned to yr; we use an optional coefficient a E [0, 1] on the 'unlikelihood" term. We also consider PPO F using reward function learned from the preference data and PPO-GT, which is an oracle that learns from the ground truth reward function available in the controlled sentiment setting. In sentiment experiments, one of-the-shelf version [42] as well as a modified version that normalizes rewards and further tunes hyperparameters improve performance (we also these modifications when running 'normal' PPO with learned rewards). Finally, we consider the Best of N baseline, sampling N responses from the SFT model (or Preferred-FT in dialogue) and returning the highest-scoring response according to a reward function learned from the preference dataset. This high-performing method decouples the quality of the reward model from the PPO optimization, but is computationally impractical even for moderate N as it requires sampling N completions for every query at test time.
implementations of PPO-GT,
6.1 How well can DPO optimize the RLHF objective?
The KL-constrained reward maximization objective used in typical RLHF algorithms balances exploitation of reward while restricting the policy from deviating far from the reference policy. Therefore, when comparing algorithms, we must take into account both reward achieved as well the KL discrepancy; achieving slightly higher reward but with much higher KL is not necessarily desirable. Figure shows the reward-KL. frontier for various algorithms the sentiment setting. We execute multiple training runs for each algorithm, using a different hyperparameter for policy conservativeness in each run (target KL {3.6, 9, 12) for PPO. unlikelihood, random seeds for preferred-FT). This sweep includes 22 runs in total. After each 100 training steps until convergence, we evaluate each policy on a set of test prompts, computing the average reward under the true reward function as well as the average sequence-leve KI with the reference policy KL (iT Teef). We find that DPO produces by far the most efficient frontier, achieving the highest reward while still achieving low KL. This result is particularly notable for multiple reasons. First, DPO and PPO optimize the same objective, but DPO is notably more efficient;
{0.05, 0.1, 1, 5). E {0.05, 0.1, 0.5, 1) for
That is. the sum of the per-timestep KL-divergences.
8

---

DPO's reward/KL tradeoff strictly dominates PPO. Second, DPO achieves a better frontier than PPO, even when PPO can access ground truth rewards (PPO-GT).
6.2 Can DPO scale to real preference datasets?
Next, we evaluate fine-tuning performance of DPO summarization and single-turn dialogue. For summarization, automatic evaluation metrics such as ROUGE can be poorly correlated with human preferences [&S and prior work has found that fine-tuning LMs using PPO on human preferences to provide more effective summaries. We evaluate different methods by sampling completions on the test split of TL;DR summarization dataset, and computing the average win rate against reference completions in the test set. The completions for all methods are sampled at temperatures varying from 0.0 to 1.0, and the win rates are shown in Figure - (right). DPO, PPO and Preferred-FT all fine-tune the same GPT-J SFT mode I We find that DPO has a win rate of approximately 61% % at a temperature of 0.0, exceeding the performance of PPO at 57% at its optimal sampling temperature of 0.0. DPO also achieves a higher maximum win rate compared to the best of N baseline, We note that we did not meaningfully tune DPO's | hyperparameter, so these results may underestimate DPO's potential. Moreover, we find DPO to be much more robust to the sampling temperature than PPO, the performance of which can degrade that of the base GPT-J model high temperatures. Preferre--FT does not improve significantly over the SFT model. We also compare DPO and PPO head-to-head in human evaluations in Section 6.4, where DPO samples at temperature 0.25 were preferred 58% times over PPO samples at temperature O.
On single-turn dialogue, we evaluate the different methods on the subset of the test split of the Anthropic HH dataset [I] with one step of human-assistant interaction. GPT-4 evaluations use the preferred completions on the test as the reference to compute the win rate for different methods. As there is no standard SFT model for this task, we start with pre-trained Pythia-2.8B. use Preferred-FT to train reference model on the chosen completions such that completions are within distribution of the model, and then train using DPO. We also compare against the best of 128 Preferred FT completions (we found the Best of N baseline plateaus at 128 completions for this task; see Appendix Figure E and 2-shot prompted version of the Pythia-2.89 base model, finding DPO performs as well or better for the best pest-performing temperatures for each method. We also evaluate an RL model trained with PPO on the Anthropic HH dataset f from well-known source but are unable to find a prompt or sampling temperature that gives performance better than the base Pythia-2.8B model. Based on results from TL:DR and the fact that both methods optimize the same reward function, we consider Best of 128 rough proxy for PPO-Ievel performance. Overall, DPO is the only computationally efficient method that improves over the preferred completions in the Anthropic HH dataset, and provides similar or better performance to the computationally demanding Best of 128 bascline. Finally, Figure 3 shows that DPO converges to its best performance relatively quickly.
6.3 Generalization to a new input distribution
Win rate vs. ground truth
To further compare the performance of PPO and DPO un- der distribution shifts, we evaluate the PPO and DPO poli-
Alg.
Temp 0
Temp 0.25
cies from our Reddit TL:DR summarization experiment on a different distribution, news articles in the test split of the CNN/Daily dataset [24]. using the best sampling temper- atures from TL.DRR (0 and 0.25). The results are presented in Table T We computed the GPT-4 win rate against the ground-truth summaries in the datasets, using the same GPT- 4 (C) prompt we used for Reddit TL:DR, but replacing the words "forum post" with "news article". For this new distribution, DPO continues to outperform the PPO policy by significant margin. This experiment provides initial evidence that DPO policies can generalize similarly well to PPO policies, even though DPO does not use the additional unlabeled Reddit TL;DR prompts that PPO uses.
DPO
0.36
0.31
PPO
0.26
0.23
Table 1: GPT-4 win rates vs. ground truth summaries for out-of-distribution CNN/DailyMail input articles.
http:p//hhnnguuueecccc/////e//ss tidr_aadrrr sft

---

6.4 Validating GPT-4 judgments with human judgments
We conduct a human study to verify the reliability of GPT-4's judgments, using the results of the TL;DR summarization experiment and two different GPT-4 prompts. The GPT4 (S) (sim- ple) prompt simply asks for which summary better-summarizes the important information in the post. The GPT-4 (C) (concise) prompt also asks for which summary is more concise; we eval- uate this prompt because we find that GPT-4 prefers longer, more repetitive summaries than hu- mans do with the GPT-4 (S) prompt. See Appendix C.2 for the complete prompts. We perform three comparisons, using the highest (DPO, temp. 0.25), the lowest (PPO, temp. 1.0), and a middle-performing (SFT, temp. 0.25) method with the aim of covering a diversity of sample qualities; all three methods are compared against greedily- sampled PPO (its best- persfreefo temperature). We find that with both prompts, GPT-4 tends to agree with humans about as often as humans agree with each other, suggesting that GPT-4 is reason- able proxy for human evaluations (due to limited human raters, we only collect multiple human judg- ments for the DPO and PPO-1 comparisons). Over- all, the GPT-4 (C) prompt generally provides win rates more representative of humans; we therefore use this prompt for the main results in Section 66.2 For additional details about the human study. in- cluding the web interface presented to raters and the list of human volunteers, see Appendix D.3
DPO
SFT
PPO-1
N respondents
272
122
199
GPT-4 (S) win %
47
27
13
GPT-4 (C) win %
54
32
12
Human win %
58
43
17
GPT-4 (S)-H agree GPT-4 (C)-H agree
70
77
86
67
79
85
H-H agree
65
Table 2:
Comparing human and GPT-4 win rates and per-judgment agreement on TL:DR summariza- tion samples. Humans agree with GPT-44 about as much as they agree with each other. Each experi- ment compares a summary from the stated method with summary from PPO with temperature O.
7 Discussion
Learning from preferences is powerful, scalable framework for training capable, aligned language models. We have introduced DPO, a simple training paradigm for training language models from preferences without reinforcement learning. Rather than cocrcing the preference learning problem into a standard RL setting in order to use off-the-shelf RL algorithms, DPO identifies a mapping between language model policies and reward functions that enables training language model to satisfy human preferences directly, with a simple cross-entropy loss, without reinforcement learning or loss of generality. With virtually no tuning of hyperparameters, DPO performs similarly or better than existing RLHF algorithms, including those based on PPO; DPO thus meaningfully reduces the barrier to training more language models from human preferences.
Limitations & Future Work. Our results raise several important questions for future work. How does the DPO policy generalize out of distribution, compared with learning from an explicit reward function? Our initial results suggest that DPO policies can generalize similarly to PPO-based models, but more comprehensive study is needed. For example, can training with self-labeling from the DPO policy similarly make effective use of unlabeled prompts? On another front, how does reward over-optimization manifest in the direct preference optimization setting, and is the slight decrease in performance in Figure right an instance of it? Additionally, while we evaluate models up to 6B parameters, exploration of scaling DPO state-of- of-te--o--he-rrt models orders of magnitude larger is an exciting direction for future work. Regarding evaluations, we find that the win rates computed by GPT-4 are impacted by the prompt; future work may study the best way to elicit high-quality judgments from automated systems. Finally, many possible applications of DPO exist beyond training language models from human preferences, including training generative models in other modalities.
Acknowledgements
EM gratefully acknowledges funding from a are CIFAR Fellows. This work was supported in part by the Stanford Accelerator for Learning (SAL) and Stanford Institute for Cluttn-eentered Artificial Intelligence (HAI) Generative Al flror the Future of Leurning seed grant program. The Stanford Center for Research on Foundation Models (CRFM) provided part of the compute resources used for the experiments in this work. This work was supported in part by ONR grant N00014-20-1-2655.
Graduate Fellowship. CF and CM
10

---

References
[1] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph. S. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage. Z. Hatfield Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, B. Mann, and J. Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022.
[2]
Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirho- seini, C. McKinnon, C. Chen, C. Olsson, C. Olah, D. Hernandez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, K. Lukosuite, L. Lovitt, M. Sellitto, N. Elhage, N. Schiefer, N. Mercado, N. DasSarma, R. Lasenby, R. Lar- son, S. Ringer, S. Johnston, S. Kravec, S. E. Showk, S. Fort, T. Lanham, T. Telleen-Lawton, Mann, D. Amodei, N. Joseph, S. McCandlish, T. Brown, and J. Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022.
Conerly, T. Henighan. T. Hume, S. R. Bowman, Z. Hatfield-Dodds
[3] S. Biderman, H. Schoelkopf. Q. Anthony, Bradley, K. O'Brien, Hallahan, M. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
[4] H. Bong and A. Rinaldo. Generalized results for the existence and consistency of the MLE in the Bradley-Terry-Luee model. International Conference on Machine Learning, 2022. arXiv:21 10.1 1487.
[5] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952. doi: htpps//ddo..or/11033////3333
[6] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herhert- Voss,, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Lan- guage models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processsing Systems, volume 33, pages 1877- 1901. Curran Associates, Inc., 2020. URL httpss/t////proooddnngs neurips. wwwpuripssccppppppe...eeeee cc/paper f
pdf
[7] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, Askell, et al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1900 2020.
[8] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4, 2023. arXiv preprint arXiv:2303.12711
[9] R. Busa-Fekete, B. Szörényi, P. Weng. W. Cheng, and E. Hüllermeier. Preference-biie based reinforcement Iearning: cvolutionary direct policy search using a preference-basee racing algorithm. Machine Learning, 97(3):377-311, July 2014. doi: 10.1007/s109944-14444444 URL holl:httss////oi....
[10] Y. Chen, R. Wang, H. Jiang, S. Shi, and R.-I.. Xu. Exploring the use of large language models for reference-free text quality evaluation: A preliminary empirical study. ArXiv, abs/2304.00723, 2023.
[11] A. Chowdhery, S. Narang. J. Devlin, M. Bosma, G. Mishra, A. Roberts. P. Barham, H. W. Chung, C. Sutton, Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
[12] P. F. Christiano, J. Leike. T. Brown, M. Martic. S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. V. Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Syy- tems, volume 30. Curran Associates, Inc., 2017. URL
cc/
paper_

---

[13] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, Brahma, A. Webson, S. S. Gu, Z. Dai, M. Suzgun, X. Chen. A. Chowdhery, A. Castro-Ros, M. Pellat, K. Robinson, D. Valter, S. Narang, G. Mishra, A. Yu, V. Zhao, Y. Huang, A. Dai, H. Yu, S. Petrov, E. H. Chi, J. Dean, Devlin, A. Roberts, D. Zhou, Q. V. Le, and J. Wei. Scaling instruction- instruction-fieetueed language models, 2022.
[14] M. Dudík, K. Hofmann, R. E. Schapire, A. Slivkins, and M. Zoghi. Contextual dueling bandits. In P. Grünwald, E. Hazan, and S. Kale, editors, Proceedings of The 28th Conference on Learning Theory, volume 40 of Proceedings of Machine Learning Research, pages 563 587, Paris, France, 03 06 Jul 2015. PMLR.
mlr. press/v40/Dudikze hhtmiiik1.hmmml
[15] D. Go. T. Korbak, G. Kruszewski, J. Rozen, N. Ryu, and M. Dymetman. Aligning language models with preferences through f-divergence minimization. In Proceedings of the 40th International Conference on Machine Learning. ICML'22 23. JMLR.org. 2023.
[16] A. Jain, B. Wojeik, T. Joachims, and A. Saxena. Learning trajectory preferences for manip- ulators via iterative improvement. In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. URL http:////////rrceeiiings newrips......... wttp:ewurr..ccppppppepp.pp.epeee fitess/alepppp//////
20117111100065544777777422222262 pdf
[17] T. Korbak, H. Elsahar, G. Kruszewski, and M. Dymetman. On reinforcement learning and distribution matching for fine-tuning language models with no catastrophic forgetting. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 16200-16220. Curran Associates, Inc., 2022. URL htt/https: wwwpaannnnnneeuusee newrips. cc. wwpppprrrcppppppppppeeeeee httppppprrraappppeeepeepe pttpppppr..llaaepppp//a/////11a. pttp:////pppe////leee/e//// Paper- Comperr-Coneeee ceefffrenne.eo pdf
[18] J. Kreutzer, J. Uyheng, and S. Riezler. Reliability and learnability of human bandit feedback for sequence-1o-sequence reinforcement learning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1777-1788, Melbourne, Australia, July 2018. Association for Computational Linguisties. doi: 10.18653/v1/ 1165. URL https:t::: http:///caaanhloooooo org/ 1165
[19] A. Kupcsik, D. Hsu, and W. S. Lee. Learning Dynamic Sabollto-Humman Object Handover from Human Feedback, pages 161-176. Springer International Publishing, 01 2018 ISBN 978-3-319-51531-1, doi: 10.11077978-3-319953
10.
[20] S. Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review, 2018.
[21] R. D. Luce. Individual choice behavior: A theoretical analysis. Courier Corporation, 2012.
[22] A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng. and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Techmologies, pages 142-150 Portland, Oregon, USA, June 2011. Association for Computational Linguisties. URLpttp:: www.b.lweb..... .ogg anthology/P11-1000 1015
[23] S. Mishra, D. Khashabi, C. Baral, and H. Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 3470-3487, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/1/20222.aal wwtp////00222 acl acng. 244
244. URL https:://aclattthllllet
[24] R. Nallapati, B. Zhou, C. dos Santos, Ç. Gulçehre, and B. Xiang. Abstractive text summarization using sequence-to-sseueene RNNs and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pages 280-290, Berlin, Germany, Aug. 2016 6. Association for Computational Linguistics. doi: 10. 1685//11KK16-1022 URL https://
scllattolloy.org//KK K128
12

---

[25] D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, Kashinkunti, J. Bernauer, Catanzaro, Phanishayee, and M. Zaharia. Efficient large-scale language model training on gpu clusters using megatron-Im. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC: '21, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 47834444.44 doi: 100...145555 0145/3458817.338 3476209
URL
[26]
L. Ouyang. J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, F. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27730-27744 Curran Associates, Inc., 2022. URL httptttpss//pporeeeiiis www.:ss..rrrrpp.ocoooo. wttparppp.ccppppprrrer.... files/ .ddf
[27] R. Paulus, C. Xiong, and R. Socher. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations, 2018. URL htpp:::////ooenrreeee net/forum id Ad-HkAAiQgA-
(28) X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage-weighteed regression: Simple and scalable off- aff-policy reinforcement learning. arXiv preprint arXiv: 2019.
[29] J. Peters and S. Schaal. Reinforcement learning by reward- weighte regression for operational space control. In Proceedings of the 24th international conference Machine learning, pages 745-750, 2007.
[30] R. L. Plackett. The analysis of permutations. Journal of the Royal Statistical Society. Series C (Applied Statistics), 24(2):193-22,, 1975. doi: https://ob.oog/100337//336
[31] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners, 2019, Ms., OpenAL.
[32] R. Ramamurthy, P. Ammanabrolu, K. Brantley, J. Hessel, R. Sifa, C. Bauckhage, H. Hajishirzi, and Y. Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In The Eleventh International Conference on Learning Representutions, 2023. URL http::://///o //penreveee 2
[33] M. Ranzato, S. Chopra, M. Auli, and W. Zaremba. Sequence level training with recurrent neural networks. CoRR, abs/151 0bs/15511.065 1.06332, 2015.
[34] D. Sadigh, A. D. Dragan, S. Sastry, and S. A. Seshia. Active preference-based learning of reward functions. In Robotics: Science and Systems (RSS), 2017.
[35] A. Saha, A. Pacchiano, and J. Lee. Dueling rl: Reinforcement learning with trajectory preferences. In F. Ruiz, J. Dy, and J.-W. van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statisties, volume 206 of Proceed- ings of Machine Learning Research, pages 6263-6289. PMLR, 25-27 Apr 2023. URL
mlr press/v206/ sahaaaa...a. html
[36] V. Sanh, A. Webson, C. Raffel, S. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, A. Raja, M. Dey, M. S. Bari, C. Xu, U. Thakker, S. S. Sharma, E. Szezechla, T. Kim, G. Chh- ablani, N. Nayak, D. Datta, J. Chang, M. T.-. Jiang, H. Wang, M. Manica, S. Shen, Z. X. Yong, H. Pandey, R. Bawden, T. Wang, T. Neeraj. J. Rozen, A. Sharma, A. Santilli, T. Fevry, J. A. Fries, R. Teehan, T. L. Scao, S. Biderman, L. Gao, T. Wolf, and A. M. Rush. Multi- task prompted training enables zero-shot task generalization. In International Conference on hett/forum?.d--VrbboDo
Learning Representations, 2022. URL httpttt//ppeerrriiii
[37] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, 2017.
[38] N. Stiennon, I.. Ouyang, J. Wa, D. M. Ziegler, R. Lowe, C. Voss, A. Radford, Amodei, and P. Christiano. Learning summarize from human feedback, 2022.
13

---

[39] R. Thoppilan, D. D. Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos,
L. Baker, Du, Y. Li, H. Lee, H. S. Zheng, Ghafouri, M. Menegali, Y. Huang, M. Krikun, D. Lepikhin, J. Qin, D. Chen, Y. Xu, 7. Chen, A. Roberts, M. Bosma, V. Zhao, Y. Zhou,
C.-C. Chang, Krivokon, W. Rusch, M. Pickett, P. Srinivasan, L.. Man, Meier-Hellstern,
M. R. Morris, T. Doshi, R. D. Santos, T. Duke, J. Soraker, B. Zevenbergen, V. Prabhakaran, M. Diaz, Hutchinson, K. Olson, A. Molina, E. Hoffman-John, J. Lee, L. Aroyo, R. Rajakumar, A. Butryna, M. Lamm, V. Kuzmina, J. Fenton, A. Cohen, R. Bernstein, R. Kurzweil, B. Aguera- Arcas, C. Cui, M. Croak, Chi, and Q Le. Lamda: Language models for dialog applications, 2022.
[40] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B, Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 230,.1311,.1 2023.
[41] M. Völske, M. Potthast, S. Syed, and B. Stein. TL:DR: Mining Reddit learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 59-63, Copenhagen, Denmark, Sept. 2017. Association for Computational Linguistics. doi: 10. 00.18653/11/W17-44500 URL
[42] I.. von Werra, J. Tow, reciprocated, S. Matiana, A. Havrilla, cat state, I.. Castricato, Alan, D. V. Phung. A. Thakur, A. Bukhtiyarov, aaronrmm, F. Milo, Daniel, D. King. D. Shin, E. Kim, J. Wei, M. Romero, N. Pochinkov, O. Sanseviero, R. Adithyan, S. Siu, T. Simonini, V. Blagojevic, X. Song. Z. Witten, alexandremuzio, and crumb. CarperAl/trx:: v0.6.0: LLaMa (Alpaca), Benchmark Util, T5 ILQL, Tests, Mar. 2023. URL https:://oo...oo 10. 1111100002222/ddd. 7790115
[43] B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.
jax, May 2021.
[44] R. J. Williams. Simple statistical gradient-following algorithms for connectionist rein- forcement learning. Mach. Learn., 8(3-4)222--26, may 1992. ISSN 0885-6125. doi:
10. 10.000//F00099669
httptttts///oooorrtt 10. 1007 1007/BF00992699
[45] Y. Wu and B. Hu. Learning to extract coherent summary via deep reinforcement learning. In Proceedings of the T'hirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI' 18/AAA 18/EAAI' 18. AAAI Press,
2018. ISBN 978-1-57735-800-8
[46] X. Yan, C. Luo, C. I.. A. Clarke, N. Craswell, E. M. Voorhees, and P. Castells. Human preferences as dueling bandits. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22, page 567-577, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 978 4503873333 doi: 1145/3477495... 3531991
10.1145/3477495.. 1991. http:tt:t///o...ooo
[47] Y. Yue, J. Broder, R. Kleinberg, and T. Joachims. The k-armed dueling bandits problem. Journal of Computer and System Sciences, 78(5):1538-1553 2012. ISSN 0022-0000. doi: https: 12.000 URL htllhttss:/wwwe scw sciencedirect. com/science JCSS Special Issue: Cloud Computing 2011.
article/s11/8022000100002
[48] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodci, P. Christiano, and
G. Irving. Fine-tuning language models from human preferences, 2020.
14


Reviews:
[{"review_id": "oyzUJXYWbM", "reviewer": "Reviewer_FksS", "summary": "RLHF is generally done by training a reward model on a dataset of preferences and then using on-policy RL to finetune a language model with the reward model. This work proposes to bypass explicit learning of a reward model and finetune the language model directly on the preference dataset. This is conceptually similar to offline RL but the authors derive a direct preference objective such that the training is supervised not RL-based.\n\nUsing a standard preference model, a supervised objective (DPO) is derived, theoretically justified, and compared to PPO. The authors test their method on three tasks: sentiment completion (IMDB), summarization (as in Stiennon et al), and single-turn dialogue with Anthropic's HH dataset. They find that DPO achieves higher reward while staying closer to the reference model (KL) in the IMDB task. Using GPT-4 as a proxy for human judgement on summarization, DPO can match PPO's performance while being less sensitive to sampling temperature. They validate GPT-4 with human annotators and find that DPO is even slightly more preferred over PPO. On dialogue, they find that DPO outperforms supervised and re-ranking (again using GPT-4 eval) but do not compare against PPO.", "questions": "What are the compute differences between DPO and PPO? It feels like DPO should be much more efficient and this information would only strengthen your work.\n\nWhy do you not compare against ILQL for sentiment? As an offline RL method, it seems the closest RL comparison to DPO\n\nFor reproducibility, what exact API are you using for the GPT-4 evaluation (e.g. GPT-0314)\n\nCan you move Figure 2 and 3 to the next pages so it is closer to the text that references it?", "limitations": "The authors did quite well to address limitations in the paper, I believe they covered most of my concerns there.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The overall idea is relatively simple but the justification and theorems provide an excellent basis. The paper is timely and gives an excellent alternative to RLHF, that should be notably more efficient. The paper is clearly written and has the potential to be very impactful if used at scale in modern methods. GPT-4 evaluations and the follow-up human evaluations are strong and demonstrate both improved performance and robustness which can be notably bad in RLHF. The authors include the minimal code for DPO in the appendix which is commendable and many GPT-4 evaluation details for excellent reproducibility. The presentation is also very clear and breaking down the DPO loss into its components with clear language is an excellent addition. ", "weaknesses": "The main weakness is the unexplored main difference between DPO and PPO: generalization. The authors note it in limitations but since PPO is trained on-policy and DPO is trained offline, looking at generalization outside the dataset would be helpful. The work, as is, feels sufficient without those experiments but the difference does play into an inconsistency with some theorems. The other major issue are some incongruenties in the evaluation that the authors should respond to.\n\nThe proofs and theorems contain a hidden assumption that is not explicitly stated. In the formulations for PPO, the optimization uses $y \\sim \\pi_\\theta$ whereas in the formulations for DPO $y \\sim D$ i.e. DPO's algorithm is offline and the model learned from its implicit reward has guarantees over the dataset. In contrast, PPO's algorithm is learned over a new set of datapoints sampled from the model during training. This means that Equation 3 is maximized over a different set of data points than Equation 7. So even though Theoerem 1 is correct, Lemma 2 does not fully encompass this situation. With offline DPO vs on-policy PPO, equivalent reward models could induce different optimal policies since they are being optimized over a different set of points. The section seems to imply that PPO and DPO can induce the same optimal policies because of the ambiguity with $y$. Clarifying this would be helpful. This also means that section 5.2 and the substitution used to make Equation 10 is not exactly accurate as the optimal policy for PPO is not the same as the optimal policy for DPO. I still feel the point of section 5.2 is reasonable but the authors should explicitly note the issue with the substitution. \n\nThere are issues with each of the different evaluations, even though the overall picture does suggest that the method is sound. I will increase my score if the authors simply clarify some questions about the evaluations (not even necessary to do changes / re-run experiments)\n\nFor controlled sentiment (IMDB), the authors do not give many details, seem to deviate from previous work on the task without explanation. The GRUE benchmark (RL4LMs) specifically uses GPT-2 base and finds that models should not be supervised-finetuned before RL training. In contrast, the authors use GPT-2 large (not even GPT-2 medium) and do supervised finetuning before PPO. No details of the baseline are given except for the graph legend which implies the PPO baseline comes from the `trl` library, although the authors only claim to use the `trlx` library. It is also unclear whether the evaluation is over the train set or the test set. Since this task is the only one where the authors show the tradeoff between reward achieved and KL, these details seem important.\n\nFor summarization, the issue is relatively minor. GPT-4 may be an unfair evaluator (see Large Language Models are not Fair Evaluators) and the work would benefit from evaluating samples in both positions (i.e. option A then B and option B then A)\n\nFor Anthropic's HH, the authors claim there is no SFT model available but there are many available pretrained models for this benchmark that the authors do not compare against. The most prominent example being a GPT-J 6B model `reciprocate/ppo_hh_gpt-j` hosted on huggingface which also has wandb runs and repro code available on the `trlx` library. The authors have chosen Pythia 2.8B when there exists a similar Pythia 1.6B PPO-trained on the HH dataset and it is unclear why the authors do not compare to this model or another PPO-trained model.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for appreciating our work and providing a great summary for it!\n\n### DPO vs PPO generalization.\n\nWe performed a follow-up experiment to assess the performance of PPO and DPO under distribution shifts. We evaluate the summarization policies trained on Reddit TL;DR data on the CNN/DailyMail dataset, and evaluate the win rate of each policy's sample vs the ground truth summary. We use the sampling temperatures that performed best for TL;DR. For space reasons, please see the general comment for full experimental details.\n\n**Overall, DPO shows superior performance to PPO for OOD inputs as well:**\n\n**DPO-0 winrate vs ground truth: 0.359 (+/- 0.030)**\n\n**DPO-0.25 winrate vs ground truth: 0.309 (+/- 0.029)**\n\n**PPO-0 winrate vs ground truth: 0.258 (+/- 0.027)**\n\n**PPO-0.25 winrate vs ground truth: 0.230 (0.026)**\n\n### PPO $y \\sim \\pi_\\theta$ vs DPO $y \\sim D$\n\nWe note that Eq. 3 and Eq. 7 represent different stages of RLHF. Eq. 3 shows the objective of the policy optimization stage, while Eq. 7 shows the reward learning stage (using the DPO parameterization). Thus the interpretation of y is different in these equations. The DPO reward parameterization allows us to extract the optimal solution to the problem in Eq. 3 exactly, in closed form; this parameterization simply means that we don't have to perform any additional RL optimization (such as online learning with PPO), which conventional RLHF does.\n\n### DPO vs PPO optimal policy for finite data\n\nIn the limit of infinite data and perfect optimization, the policy found by PPO will be equivalent to the DPO policy, which is the optimal policy (wrt the learned DPO reward) in Eq. 4, which holds for all prompts & answers (this assumes the PPO reward model agrees with the implicit DPO reward model; Theorem 1 and our empirical findings show that the two reward model classes have equivalent expressiveness, and our empirical findings suggest that the two parameterizations do achieve very similar classification accuracy on held out preference pairs). However, in the case of finite data or imperfect optimization, PPO might not recover the theoretically optimal policy that DPO does (Eq. 4). Our new generalization experiment in the summarization setting further suggests that the DPO policy generalizes at least as well, if not better than, the PPO policy. Finally, Lemma 2 only states equivalence of the optimal policy, but suboptimal policies found from finite data or imperfect optimization may indeed differ for two reward functions in the same equivalence class.\n\n### Substitution in Equation 10\n\nThanks for pointing this out; this is a typo. The optimal policy is the one induced by Eq. 4, not Eq. 7. We apologize for the confusion and will correct the mistake in the final version of the paper. The result (closed-form optimal policy) of Eq 4 holds on all prompts and answers, given a reward function. We agree that the paper would benefit from this discussion, and the points raised about the difference in the PPO / DPO problem settings and the optimal policies being different. Specifically, we will revise Section 5.2 to note the discrepancy between optimal policies.\n\n### IMDb details\n\nThanks for raising the concern about the missing details. We provide the details of the controlled sentiment generation task in the common response, along with the details for the PPO baseline. We follow the TRL library which first does an SFT step before running PPO. This is necessary since the experiment uses shorter prompts and the model might deviate from the movie review task. The evaluation is done on the test set. We will add all the details to the paper in the next revision.\n\n### GPT-4 evaluation sample ordering\n\nOur present evaluation protocol randomly flips options A and B for every evaluation prompt for this reason. We make a brief note of this in Appendix C.1, but will certainly clarify in the main text.\n\n### Anthropic HH baselines\n\nWe have added a comparison of DPO to the oracle best-of-N method using a reward model learned on all the preference data; see Fig. 3 in the attached rebuttal figure pdf. We observe that DPO performs comparably to the best-of-N method. Note, this method is computationally infeasible as it requires generating N samples at test-time (for example, 128) and choosing the best one, but, this baseline is often used as an oracle proxy for PPO [1][2]. We use best-of-N because it is easy to implement, as strong as PPO, and public RLHF checkpoints that we found for Anthropic use undocumented training splits and/or skip the SFT phase (e.g., the reciprocate/ppo_hh_gpt-j checkpoint you mentioned) which is non-standard and makes fair comparison with DPO difficult.\n\n[1] Scaling Laws for Reward Model Overoptimization. Gao, Schulman, Hilton.\n\n[2] AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback, Dubois, et. al. \n\n### PPO vs DPO compute\nDepending on design choices DPO can require as little as half as much accelerator memory than PPO, since it does not maintain separate value function and reward models. In terms of run times, we found that DPO training requires about 5-8 times less compute time than the standard RLHF pipeline, depending on the model and dataset.  \n\n### ILQL baseline\n\nIt is possible to use many different RL algorithms to align a model with the learned reward function and in this work we focused on PPO due to its prevalence. Concurrent work (Appendix D3 in [1]) evaluated several offline RL algorithms on the Anthropic HH datasets using a learned reward model and found that they yielded little to no improvement. \n[1] Fine-Tuning Language Models with Advantage-Induced Policy Alignment, Zhu et. al. \n\n### GPT-4 API details\n\nWe used the `gpt-4-0314` for evaluation. We will add the details to the experiments section of the paper.\n\n### Figure 2/3 positioning\nThanks for the suggestion! We will reformat the paper so that the figures are closer to the text referencing it."}, {"review_id": "NnHugasEGZ", "reviewer": "Reviewer_GfFq", "summary": "The authors propose DPO, a method to fine-tune a language model to human preferences that does not use reinforcement learning or learn a reward model, but rather, directly optimizes the language model with preference data.\n\nThrough a change of variables, the authors express the reward function in terms of the optimal policy $\\pi*$ and $\\pi^{ref}$. The authors write the human preference distribution in terms of $\\pi*$ and $\\pi^{ref}$, and thus a maximum likelihood objective can be written in terms of $\\pi_\\theta$ and $\\pi^{ref}$. This formulation allows for a simple DPO gradient update.\n\nThe method is equivalent to fitting a reparameterized Bradley-Terry model. With mild assumptions, DPO does not constrain the class of learned reward models.\n\nExperimental results are run on three different open-ended generation tasks. Baselines include PPO, zero-shot and few-shot prompting, a SFT model, and Preferred-Ft. DPO’s reward/KL tradeoff dominates that of PPO. Furthermore, win-rates of DPO compared to other models (or test-set labels) beat baselines.\n\n", "questions": "Please provide experimental details for the IMDB Sentiment Generation task.\n\nPlease clarify what dataset, (specifically, whether the “helpful-online” or “helpful-rejection-sampled” data) was used for the single-turn dialogue experiment?\n\nWhy does win-rate in the single-turn dialogue task increase with higher sampling temperature in Figure 3?", "limitations": "Assumes we use Plackett-Luce models to model preferences.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "The paper is a significant contribution in a high impact area -- namely, fine-tuning language models to human preferences. DPO opens a new paradigm of learning human preferences without RL, and is a viable and performant alternative to the RLHF pipeline, reducing computational demands, and requires little tuning of hyperparameters\n\nDPO is grounded theoretically, equivalent to fitting a reparameterized Bradley-Terry model. The DPO update is simple and interpretable. Theorem 1 shows we do not necessarily constrain ourselves in terms of the reward model after reparameterization.\n\nExperimental results are good. DPO learns a better Reward / KL tradeoff compared to baselines in the sentiment generation task. DPO also has excellent win-rates compared to a number of baselines as evaluated with GPT-4 on the summarization and single-turn dialogue tasks. Human evaluators show as high correlations with each other as compared with the GPT-4 evaluation.", "weaknesses": "Single-turn dialogue experimental results did not compare DPO with a RLHF baseline tuned from the same base model, and it appears that helpful-rejection-sampled data from the Anthropic HH dataset was used, a relatively weaker baseline compared to the helpful-online dataset.\n\nThere are very few details on the experimental setup for the IMDB Sentiment Generation task.\n\nImprovements to the clarity of the paper can be made, including more detailed descriptions of experimental setup in the Appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for assessing our paper to be a significant contribution, and for your feedback!\n\n### Single-turn dialogue experimental results did not compare DPO with a RLHF baseline…\n\nPlease, see Fig. 3 in the rebuttal pdf for an additional baseline to our dialogue experiments, the oracle best-of-N method that uses a reward model learned on all the preference data to pick the best of N samples from the SFT model. We observe that DPO performs comparably to the best-of-N method. Note that best-of-N is computationally infeasible as it requires generating N samples at test-time (for example, 128) and choosing the best one, but this baseline is often used as an oracle proxy for PPO [1] and achieves comparable performance (Table 2 in [2]). We use best-of-N because it is easy to implement, as strong as PPO, and public RLHF checkpoints that we found for Anthropic use undocumented training splits and/or skip the SFT phase, which is non-standard and makes fair comparison with DPO difficult.\n\n[1] Scaling Laws for Reward Model Overoptimization. Gao, Schulman, Hilton.\n\n[2] AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback, Dubois, et. al. \n\n\n### There are very few details on the experimental setup for the IMDB Sentiment Generation task…\n\nThanks for the suggestion! Here are additional details about the IMDB task, which we will add to Section 6.1 of the next revision of the paper:\n\nFollowing the trl library, the prompts are prefixes from the IMDB dataset of length 2-8 tokens. We use the pre-trained sentiment classifier \"siebert/sentiment-roberta-large-english\" as a ground-truth reward model and GPT2-large as a base model. We use these larger models as we found the default ones to generate low-quality text and rewards to be somewhat inaccurate.  We first use supervised fine-tuning on a subset of the IMDB data for 1 epoch. We then use this model to sample 4 completions for 25000 prefixes and create 6 preference pairs for each prefix using the ground-truth reward model. The RLHF reward model is initialized from the GPT2-large model and trained for 3 epochs on the preference datasets, and we take the checkpoint with the highest validation set accuracy. The “TRL” run uses the same hyper-parameters as the trl library implementation. Our implementation uses larger batch samples of 1024 per PPO step.\n\n### Please clarify what dataset, (specifically, whether the “helpful-online” or “helpful-rejection-sampled” data) was used for the single-turn dialogue experiment?\n\nWe used the entire Anthropic/hh-rlhf available on HuggingFace. We will add the details to the paper.\n\n### Why does win-rate in the single-turn dialogue task increase with higher sampling temperature in Figure 3?\nFor low-temperature (0.25) evaluation in the dialogue task, we observe a significant number of responses that get stuck in repetition loops, a common failure mode of smaller LMs being sampled at low temperature. For the two higher-temperature (0.7, 1.0) evaluations, the win rates are within one standard error of each other. We will note this observation of repetition loops at low temperature in Section 6.2 of the revised paper."}, {"review_id": "4ge0CQgWoO", "reviewer": "Reviewer_cpnK", "summary": "This paper presents a new policy optimization algorithm with human preferences, called Direct Preference Optimization (DPO), which eliminates the need for explicitly fitting a reward model. The authors formulate the process of RLHF into a single stage policy learning by leveraging a mapping between reward functions and optimal policies. The main contribution of this paper is that they theoretically derive a new objective function that is equivalent to existing RLHF methods but can be simply learned by a single objective, and experimentally shows that it has better performance.", "questions": "- In Figure 2 of the experiment results, why does DPO perform better than PPO-GT even though PPO-GT uses a true reward function?\n- The values of the target KL used in PPO-GT are large (target KL $\\in$ {3, 6, 9, 12}). Is there a reason why the hyperparameter search range is set this way? In the previous paper (ex. [1]), hyperparameter search was done in a much wider range (target KL $\\in$ {0.02, 0.05, 0.1, inf}), and there were good results at low values, so I wonder if the results of the PPO-GT used in the paper were suboptimally learned.\n- In the results according to the sampling temperature, why does DPO have robust results to the sampling temperature?\n- How does the fluency (or naturalness) of the generated sentences from the learned policy according to training iteration change in the DPO experiment? In the case of RLHF, the fluency of the generated sentence shows a result that is sensitive to the coefficient of the KL divergence term (i.e. Fluency deteriorates as learning progresses in RLHF algorithm). I wonder if DPO does not have a similar problem. \n- Also, it was mentioned in the paper that DPO learning is stable regardless of hyperparameter beta, but I wonder if learning according to various beta values is stable without degeneration issues (i.e. degrading the fluency). \n- Unlikelihood learning (i.e. decreasing the likelihood) objectives are easy to degrade fluency (or naturalness), but how is it possible to stably learn while including unlikelihood learning objectives in DPO? It seems that the main difference between the equation for the gradient of the DPO and unlikelihood learning is that the weight term is multiplied. What exactly does the weight do to make learning stable?\n- What does the star mean in the right plot of Figure 2?\n\n[1] Rajkumar Ramamurthy and Prithviraj Ammanabrolu et al, Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization, ICLR 2023", "limitations": "- Since the reward model is not explicitly learned and used, generalization to unseen prompts may not work well.\n- A comparison of RLHF and DPO in various experimental settings is lacking.\n- As mentioned in Questions and Weaknesses, it seems that there are still some parts that are not clearly explained or verified.\n- Details of the experimental setup are omitted.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- This paper presents a new algorithm that can simply learn the policy from a given human preference dataset without explicit reward learning. \n- This paper not only experimentally shows better results than RLHF, but also provides theoretical connections with RLHF.", "weaknesses": "- Since the reward model is not explicitly learned and the policy is learned only with the given human preference dataset, there is a possibility that the generalization performance for unseen prompts is relatively low compared to the existing RLHF.\n- Experiments are insufficient to show that DPO is more stable than conventional RLHF. In order to show that the DPO can be learned more stably, it seems that the learning curve should be provided together with the results shown in the experiment.\n- A detailed description of the experiment is lacking. (ex. amount of human preference dataset used in the experiment)\n- Depending on the amount of human preference dataset, DPO may have better or worse performance than RLHF, but there is no analysis or explanation about this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your helpful feedback and questions!\n\n### Generalization of DPO vs existing RLHF\n\nFirst, we note that all of the evaluations in the paper already compute test win rates on unseen test prompts.\n\nWe performed a follow-up experiment to assess the performance of PPO and DPO under distribution shifts. We evaluate the summarization policies trained on Reddit TL;DR data on the CNN/DailyMail dataset, and evaluate the win rate of each policy's sample vs the ground truth summary. We use the sampling temperatures that performed best for TL;DR. For space reasons, please see the general comment for full experimental details.\n\n**Overall, DPO shows superior performance to PPO for OOD inputs as well:**\n\n**DPO-0 winrate vs ground truth: 0.359 (+/- 0.030)**\n\n**DPO-0.25 winrate vs ground truth: 0.309 (+/- 0.029)**\n\n**PPO-0 winrate vs ground truth: 0.258 (+/- 0.027)**\n\n**PPO-0.25 winrate vs ground truth: 0.230 (0.026)**\n\n### DPO stability\n\nThanks for the suggestion! To more clearly illustrate DPO's stability across training runs, we trained 4 DPO models using EleutherAI/Pythia-1b on the Anthropic-HH dataset. Please find the training loss curves in Fig. 2 in the rebuttal figure pdf and the classification accuracy of the implicit DPO reward function in Fig. 4. We find that these learning curves are very consistent across training runs. Further, we evaluate the final GPT-4 win rate of each policy against the chosen response for unseen prompts in the test set. The win rates for the final policy from each training run were 0.426, 0.410, 0.414, 0.438 (standard dev: 0.0110). In comparison, sampling from the first policy four times, changing only the random seed used for sampling, we observe a similar distribution of scores, with slightly higher standard deviation: 0.426, 0.412, 0.406, 0.441 (standard dev: 0.0135). These results suggest that the variance in performance of DPO policies across different training seeds is less significant than the variance due to simply using different random seeds for sampling at test time.\n\nAdditionally, we also experiment with how the \\beta hyperparameter controls the KL-divergence, we find that the \\beta hyperparameter very reliably controls the KL-divergence of the final policy, as shown in Fig. 1 in the rebuttal figure pdf. In contrast, PPO implementations typically require an adaptive beta (based on a target KL divergence) because the same fixed beta can lead to very different KLs for different training runs.\n\n### Size of human preference dataset\nThanks for bringing up the missing details. We use publicly available preference datasets for summarization (HuggingFace dataset CarperAI/openai_summarize_comparisons) with 92.5k comparisons and Anthropic HH dialogue (HuggingFace dataset Anthropic/hh-rlhf) with 161k comparisons. We will include these and other details about the preference datasets in the camera ready version. In our experiments, DPO and PPO use the same preference datasets (and thus, the same amount of preference data). While our new experiment evaluating the OOD performance of PPO and DPO gives some reason for optimism about DPO's data efficiency, directly comparing the performance of DPO and PPO as the amount of preference data is varied is an important question for analysis, which we defer to future work. Thank you for this suggestion!\n\n### Why does DPO perform better than PPO-GT even though PPO-GT uses a true reward function?\nIn the controlled sentiment generation problem, we find that the learned PPO reward can achieve accuracy well over 90%. Hence we believe the discrepancy between DPO and PPO is not due to issues with reward modeling, but optimization of the reward. PPO only approximately optimizes the KL-constrained reward problem, while DPO samples from the closed-form optimal policy (without approximation). We hypothesize that PPO is less efficient than DPO, even with the ground truth reward, due to this noisy optimization.\n\n### KL target values scaling\nFigure 2 reports sequence-level KL, which adjusting for sequence length results in per-token KLs of about 0.05, 0.1, 0.15 and 0.2, comparable to those used in Ramamurthy et. al.\n\n### Fluency during training\nDPO has a very stable relationship between the KL divergence of the final policy and the hyperparameter $\\beta$, as shown in the attached Fig. 1. We do not observe any degeneration as training progresses, however, in the limit of $\\beta$ going to zero, the KL constraint vanishes, and DPO would purely maximize the (implicit) learned reward. In this case, optimizing a reward function trained on a small preference dataset would likely show some deterioration.\n\n### Unlikelihood stability\nIndeed, unlikelihood objectives can degrade fluency. As we note in Section 4, the weight scaling the unlikelihood updates in DPO in would be < 1 for all comparisons, but for comparisons where the preferred completion has a higher reward than the dispreferred completion, the weight can be substantially lower, effectively stopping learning on those examples. Thus, the adaptive weights only change the model whenever the preference pair is ordered incorrectly under the LM, which we hypothesize leads to the stable training regime. Further, as beta is larger, this weight goes to zero more rapidly as the example is learned, effectively stopping optimization more quickly. Nonetheless, we cannot rule out the possibility that carefully tuned unlikelihood methods can also be successfully used for fine-tuning LM on comparison pairs.\n\n### Figure 2 star\nStars represent the win rates as computed in human evaluation, and we include 3 stars corresponding to the win rates for DPO @ 0.25, SFT @ 0.25 and PPO @ 0.0, all compared against PPO @ 0.0. We will clarify this in the figure caption."}, {"review_id": "gnEf05SVuG", "reviewer": "Reviewer_eRhN", "summary": "This paper tackles a very important problem in LLM training, i.e., how to simplify the complex process of reinforcement learning from human feedback? RLHF is different to use, it is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the LLM using RL to maximize this estimated reward. The authors propose DPO, a training paradigm for training language models from preferences without reinforcement learning. DPO identifies a mapping between language model policies and reward functions that enables training a language model to satisfy human preferences directly, with a cross-entropy loss, without reinforcement learning. The experimental results show that DPO can fine-tune LLMs as well or better than existing algorithms.", "questions": "None", "limitations": "Yes", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This article addresses a crucial issue in LLM training, i.e., how to perform alignment in a simpler way without using RL. The proposed DPO is remarkably simple yet effective, and it exhibits good theoretical properties. I believe DPO is highly significant for researchers in the field and will serve as a powerful tool for their work.", "weaknesses": "This is a very solid piece of work. The proposed method is simple yet effective. I don't have any particular concerns or issues with it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed summary of the paper, your feedback, and believing that our work is highly significant for researchers!"}]

Author rebuttal:
We appreciate the detailed feedback provided by the reviewers. We address a few common points in this response. All other questions are addressed in reviewer specific responses.

## Re: Generalization of PPO and DPO

First, we note that all of the evaluations in the paper compute test win rates on unseen test prompts.

To further assess the performance of PPO and DPO under distribution shifts, we evaluated the PPO and DPO policies from our Reddit TL;DR summarization experiment on a different distribution, news articles in the test split of the CNN/DailyMail dataset, using the best sampling temperatures from TL;DR (0 and 0.25). We computed the GPT-4 win rate against the ground-truth summaries in the datasets, using the same GPT-4 (C) prompt we used for Reddit TL;DR, but replacing the words "forum post" with "news article". We found that for this new distribution, DPO continues to outperform the PPO policy by a significant margin:

**DPO-0 winrate vs ground truth: 0.359 (+/- 0.030)**

**DPO-0.25 winrate vs ground truth: 0.309 (+/- 0.029)**

**PPO-0 winrate vs ground truth: 0.258 (+/- 0.027)**

**PPO-0.25 winrate vs ground truth: 0.230 (0.026)**

While it is not a comprehensive evaluation of generalization of different RLHF algorithms, **this experiment suggests that DPO policies can generalize similarly well to PPO policies, even without training on the additional unlabeled Reddit TL;DR prompts that PPO uses**. We will include this experiment in our next revision, and defer a more extensive investigation to future work.

## Re: Baselines on Anthropic-HH dialogue

Some reviewers asked about additional baselines for the dialogue experiment.

We have added a comparison of DPO to the best-of-N method (i.e., generate N samples, return the one with highest reward under the reward model) using a reward model learned on all the preference data; see Fig. 3 in the attached rebuttal figure pdf. **We observe that DPO performs comparably to the strong best-of-N baseline.** Note, this method is computationally infeasible as it requires generating N samples at test-time (for example, 128) and choosing the best one, but, this baseline is often used as an oracle proxy for PPO [1][2]. We use best-of-N because it is easy to implement, as strong as PPO, and public RLHF checkpoints that we found for Anthropic use undocumented training splits and/or skip the SFT phase (e.g., the reciprocate/ppo_hh_gpt-j checkpoint you mentioned) which is non-standard and makes fair comparison with DPO difficult.

[1] Scaling Laws for Reward Model Overoptimization. Gao, Schulman, Hilton.

[2] AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback, Dubois, et. al. 


## Re: Details about IMDb sentiment task

We provide additional details for the IMDb sentiment task, and will revise Section 6.1 to include these details in the next revision of the paper. Following the Transformers Reinforcement Learning (TRL) library, the prompts are prefixes from the IMDB dataset of length 2-8 tokens. We use the pre-trained sentiment classifier "siebert/sentiment-roberta-large-english" as a ground-truth reward model and GPT2-large as a base model. We use these larger models as we found the default ones to generate low-quality text and rewards to be somewhat inaccurate.  We first use supervised fine-tuning on a subset of the IMDB data for 1 epoch. We then use this model to sample 4 completions for 25000 prefixes and create 6 preference pairs for each prefix using the ground-truth reward model. The RLHF reward model is initialized from the GPT2-large model and trained for 3 epochs on the preference datasets, and we take the checkpoint with the highest validation set accuracy. The “TRL” run uses the same hyper-parameters as the TRL library implementation. Our implementation uses larger batch samples of 1024 per PPO step.


Meta-review: Reviewers unanimously agree that the paper addresses an important problem and timely gives an excellent alternative to RLHF to LLMs. Theoretical analysis and thorough experimental results (plus the additional experiments in the rebuttal period) make the paper even stronger. The paper potentially has large impact in the LLM community. Therefore I recommend acceptance as an oral.
Decision: Accept (oral)