Direct Preference Optimization with an Offset (2024)

Abstract: Direct preference optimization (DPO) is a successful fine-tuning strategy for aligning large language models with human preferences without the need to train a reward model or employ reinforcement learning. DPO, as originally formulated, relies on binary preference data and fine-tunes a language model to increase the likelihood of a preferred response over a dispreferred response. However, not all preference pairs are equal. Sometimes, the preferred response is only slightly better than the dispreferred one. In other cases, the preference is much stronger. For instance, if a response contains harmful or toxic content, the annotator will have a strong preference for that response. In this paper, we propose a generalization of DPO, termed DPO with an offset (ODPO), that does not treat every preference pair equally during fine-tuning. Intuitively, ODPO requires the difference between the likelihood of the preferred and dispreferred response to be greater than an offset value. The offset is determined based on the extent to which one response is preferred over another. Our experiments on various tasks suggest that ODPO significantly outperforms DPO in aligning language models, especially when the number of preference pairs is limited. https://github.com/rycolab/odpo

text: Findings of the Association for Computational Linguistics ACL 2024, pages 9954–9972 August 11-16, 2024 ©2024 Association for Computational Linguistics
https://github.com/rycolab/odpo

heading: 1 Introduction
text: Reinforcement learning from human feedback (RLHF) is a key building block in training the most modern large language models. The algorithm aligns the language model’s responses to human preferences (Ouyang et al., 2022; Touvron et al., 2023; Anil et al., 2023; OpenAI et al., 2023). A typical implementation of RLHF is as follows. First, humans compare paired responses from a language model, i.e., they determine which of the responses is better, e.g., more helpful or less toxic. Second, a reward model is trained to give higher rewards to the responses preferred by humans. Lastly, a reinforcement learning algorithm is used to update
the model’s parameters to maximize the expected reward, while not diverging too much from the model’s initial parameters.
The aforementioned implementation of RLHF, however, is tedious for two reasons. First, the reward model is usually a model as large as the language model itself, which is expensive to train and store. Second, reinforcement learning algorithms are known to be very sensitive to the choice of hyperparameters (Zheng et al., 2023), and, thus, hard to tune. Therefore, a fruitful line of work attempts to design alternative simpler implementations of RLHF (Welleck et al., 2020; Lu et al., 2022; Zhao et al., 2023).
A highly successful and straightforward alternative to RLHF is direct preference optimization (DPO; Rafailov et al., 2023). Unlike the conventional implementation of RLHF, DPO does not train a reward model and sidesteps the use of any reinforcement learning algorithm. Instead, it finetunes the language model’s weights to maximize the likelihood of the preference data directly using the Bradley–Terry model (Bradley and Terry, 1952). Intuitively, each DPO gradient update to the model’s parameters increases the likelihood of the preferred response and decreases the likelihood of the dispreferred response.
However, DPO only takes the ordering between the model’s responses into account and not the extent to which one response is preferred over another. In many settings, the difference in quality between the two responses is known—either by asking humans (Stiennon et al., 2020; Touvron et al., 2023)1 or through point-wise scores, e.g., toxicity scores, sentiment scores, code-based execution scores, etc. In that context, a natural question is the following: how can we make use of this information to better align language models with human preferences?
To answer this question, we propose a new method, which we term DPO with an offset
1For example, Touvron et al. (2023) ask humans to specify the extent to which they prefer one output over another, i.e., by choosing from: significantly better, better, slightly better, negligibly better / unsure.
9954
!
" "
!
" "
(ODPO). ODPO is a generalization of DPO that incorporates the difference between responses when modeling preference data. The intuition behind ODPO is simple; it requires the language model to increase the likelihood of the preferred responses compared to the dispreferred responses by an offset that is determined based on the difference between their associated reward values. Therefore, the larger the reward of the preferred response in comparison to the dispreferred response, the higher the likelihood needs to be over the likelihood of the dispreferred response. We further show that when the offset is set to zero, ODPO is equivalent to DPO. This process is illustrated in Fig. 1.
In our experiments, we fine-tune language models of various sizes with DPO and ODPO and compare the two methods. In cases where a ground-truth reward function is given, we measure two competing metrics: (i) the average reward associated with the generations from the fine-tuned model, and (ii) the KL divergence between the language model before and after the fine-tuning. Based on the results of sentiment and toxicity control tasks, we observe that ODPO more often appears on the Pareto frontier of reward and KL compared to DPO. We then apply ODPO to a summarization task, where we use the scores given by humans on a Likert scale to define an offset between the two summaries. We observe that on average and across different sampling temperatures, ODPO results in a higher win rate over human-written summaries compared to DPO.

heading: 2 Preliminaries
text: Given a prompt x ∈ Σ∗, a language model πθ is a distribution over the responses y ∈ Σ∗, where Σ is the alphabet.2 As is standard, we parameterize the conditional distribution over responses given a
2An alphabet is a finite, non-empty set.
prompt as an autoregressive language model πθ(y | x) parameterized by θ as,
πθ(y | x) =
πθ(EOS | y,x) T∏
t=1
πθ(yt | y<t,x), (1)
where EOS /∈ Σ is a distinguished end-of-sequence token. To deploy a language model to a downstream task, it is usually necessary to additionally fine-tune it on high-quality data. Given a dataset of prompts and desired responses (x,y) ∼ DSFT, the standard fine-tuning objective (to be maximized) is simply the log-likelihood of DSFT
J ML(θ) = E (x,y)∼DSFT
[ log πθ(y | x) ] . (2)
We use πSFT to refer to the language model finetuned with this objective on DSFT.
Such a fine-tuning approach can be effective for solving downstream tasks with language models. However, there is a disparity between the maximum-likelihood objective and the goal of the downstream task, which is to leverage the language model to generate high-quality responses, as judged by humans. This misalignment arises partly because the maximum-likelihood objective, Eq. (2), fails to distinguish between major and minor errors. For instance, when fine-tuning a language model to perform summarization, a significant error might involve hallucination, while a minor one might involve a failure to employ the appropriate synonyms (Stiennon et al., 2020). Training with the maximum-likelihood objective encourages the model to assign nonzero probability mass to all responses in DSFT, even those of lower quality.
To address this issue, one can optimize the expected reward of language model generations instead of using the maximum-likelihood objective.
Consider a reward function r(x,y), estimated from quality assessments performed by humans. The reward function assigns real values to prompt– response pairs; we will discuss in §3 how such a reward model can be learned from human feedback. Our objective is to make sure that the responses generated by the language model have high quality. Therefore, we can directly optimize the expected reward of the responses generated by the language model,
J RL(θ) = E x∼D,y∼πθ(·|x)
[ r(x,y) ] , (3)
where D = {x(n)}Nn=1 is a multiset of prompts. To prevent reward hacking (Amodei et al., 2016) and to make sure that we do not diverge too much from the supervised fine-tuned model πSFT, a regularization term is often added to the objective (Stiennon et al., 2020),
J RL(θ) = E x∼D,y∼πθ(·|x)
[ r(x,y) ]
− β DKL [ πθ(y | x) ∥πSFT(y | x) ] . (4)
The above objective is optimized using proximal policy optimization (PPO; Schulman et al., 2017) or another actor–critic algorithm (Mnih et al., 2016; Glaese et al., 2022).

heading: 3 Reward Modeling
text: Pointwise Rewards. A key component in Eq. (4) is the task-dependent reward function that assigns pointwise real-valued rewards to each output. In many tasks, learning such reward functions is straightforward. For example, in open-ended text generation, a desired attribute could be the presence of indicators of positive sentiment in the generated text, while an undesired attribute could be toxicity. In such cases, the reward model might take the form of a classifier that assesses responses based on their sentiment or toxicity. Similarly, in code generation tasks, the quality of the code can be automatically evaluated, providing another straightforward example.
Pairwise Preferences. Learning a point-wise reward function for tasks like summarization or dialogue generation is more complex. Judging the absolute quality of a summary can depend on several factors, e.g., coherence, faithfulness, and conciseness, which makes it hard to collect human feedback datasets for reward model training. An effec-
tive strategy in such cases is to collect human preferences instead of point-wise judgments (Ziegler et al., 2020; Wu et al., 2021; Ouyang et al., 2022). In this setup, humans are shown two (or more) responses to a prompt x and are asked to select the response they prefer. Therefore, a datapoint in a human feedback dataset is a triple (x,yw,yl) ∼ DHF, where yw is preferred over yl. Given such a dataset, one needs to learn the point-wise reward function. It is common to assume that the preference data can be modeled by a Bradley–Terry model (Bradley and Terry, 1952),
pBT(yw ≻ yl | x)
= exp
( r(x,yw) )
exp ( r(x,yw) ) + exp ( r(x,yl) ) (5a)
= σ ( r(x,yw)− r(x,yl) ) , (5b)
where σ(x) = 11+exp(−x) is the sigmoid function. Assuming that DHF ∼ pBT, we can train a reward model rϕ, parameterized by ϕ, as a binary classifier that maximizes the following log-likelihood
E [ log σ ( rϕ(x,yw)− rϕ(x,yl) )] , (6)
where the expectation is over (x,yw,yl) ∼ DHF. Intuitively, the reward function should assign higher reward values to the responses that are preferred by humans. The estimated reward function is then plugged into Eq. (4) to enable policy estimation.
It is often the case that directly optimizing Eq. (4) is complex and resource-intensive. Next, we will elaborate on an alternative method to using RLHF that has demonstrated comparable or even superior effectiveness to PPO.

heading: 4 Direct Preference Optimization
text: Rafailov et al. (2023) introduce a method to avoid reward model training and, thus, to directly optimize the language model. Their method, termed direct preference optimization (DPO) works as follows. The critical observation is that the optimal solution that maximizes the Eq. (4) is,
π⋆θ(y | x) = (8) 1
Z(x) πSFT(y | x) exp ( 1 β r(x,y) ) ,
where
Z(x) = ∑
y∈Σ∗ πSFT(y | x) exp
( 1 β r(x,y) ) , (9)
LDPO(θ) = − E (x,yw,yl)∼DHF
[ log σ ( β log
πθ(yw | x) πSFT(yw | x) − β log πθ(yl | x) πSFT(yl | x)
)] (7a)
= − E (x,yw,yl)∼DHF
[ log σ ( r̂θ(x,yw)− r̂θ(x,yl) )] (7b)
LODPO(θ) = − E (x,yw,yl)∼DHF
[ log σ ( r̂θ(x,yw)− r̂θ(x,yl)−∆r )] (7c)
Note that Z(x) is, in general, difficult to compute as it involves a sum over a countably infinite set. Nevertheless, we can write the reward as a function of π⋆θ by rearranging the terms,
r(x,y) = β log π⋆θ(y | x) πSFT(y | x) + β logZ(x). (10)
Under the assumption that the preference data is well-modeled by the Bradley–Terry model (Eq. (5a)), we substitute the reward in Eq. (6) with Eq. (10), and formulate the loss function in terms of the language model parameters. Therefore, we directly optimize the language model by maximizing the likelihood of the preference data in Eq. (7b). In this equation, r̂θ(x,y) = β log πθ(y|x) πSFT(y|x) is called the estimated reward. Note that the intractable normalization factor Z(x) cancels out.
Simply put, Eq. (7b) requires the estimated reward for the preferred response to be larger than the estimated reward for the dispreferred response. A strong assumption behind the DPO’s loss is that the preference data is well-modeled by a Bradley– Terry model. However, the Bradley–Terry model only tells us the probability that one response is preferred over another, and not the extent to which this preference will hold. To enhance DPO, we will introduce our modification to DPO, which generalizes DPO and introduces an offset between the responses in its loss function.

heading: 5 DPO with an Offset
text: The intuition behind ODPO is simple: we want the estimated reward for the preferred response to be larger than the estimated reward for the dispreferred response, by an offset that is a function of the actual reward differences assigned to the two responses. To begin our exposition of ODPO, we first discuss the connection between the Bradley–Terry model and Gumbel random variables, which we then use to construct the loss function for ODPO.
Theorem 1. Let yw and yl be two responses to a prompt x, and let r̂θ(x,yw) and r̂θ(x,yl)
be their associated estimated rewards. Finally, let Rw ∼ Gumbel(r̂θ(x,yw), 1) and Rl ∼ Gumbel(r̂θ(x,yl), 1) be Gumbel random variables. Then, we have,
P(Rw −Rl > 0) = pBT(yw ≻ yl | x) = σ(∆r̂θ), (11)
where pBT(yw ≻ yl | x) is a Bradley–Terry model Eq. (5a) parameterized by r̂θ(x,yw) and r̂θ(x,yl), and ∆r̂θ def = r̂θ(x,yw) − r̂θ(x,yl) is the difference between the estimated rewards.
Proof. The proof is simple and follows directly from the Gumbel-max trick (Maddison and Tarlow, 2017). See App. A for details. ■
Theorem 2. Let yw and yl be two responses for a prompt x. Given a desired offset ∆r between the responses’ reward values,
P ( Rw −Rl > ∆r ) = σ(∆r̂θ −∆r). (12)
where ∆r̂θ def = r̂θ(x,yw)− r̂θ(x,yl).
Proof. The difference between two independent Gumbel variables Rw ∼ Gumbel(r̂θ(x,yw), 1) and Rl ∼ Gumbel(r̂θ(x,yl), 1) follows the logistic distribution. Therefore, the probability of the difference between the noisy estimated rewards being greater than ∆r is,
P ( Rw −Rl > ∆r ) = 1−F ( ∆r ) , (13)
where F(·) is the CDF for logistic distribution. The mean of this distribution is ∆r̂θ , and the variance is 1. Substituting the CDF in Eq. (13) with its definition,
P ( Rw −Rl > ∆r ) = 1−F(∆r) (14a)
= 1 2 − 1 2 tanh (1 2 (∆r −∆r̂θ) ) (14b) = 1
2 − 1 2
( 2σ(∆r −∆r̂θ)− 1 ) (14c)
= 1− σ(∆r −∆r̂θ) (14d) = σ(∆r̂θ −∆r). (14e)
■
Similar to DPO, we then maximize the likelihood of the preference data, as shown in Eq. (7c). Intuitively, minimizing the above loss function enforces the estimated reward for the preferred response yw to be higher than yl by the offset ∆r. Comparing Eq. (7b) and Eq. (7c), when ∆r = 0, ODPO will be equivalent to DPO.
Designing ∆r. We choose to model the offset as a monotonically increasing function f(·) of the difference between the scores associated with the responses:
∆r = α f ( score(x,yw)− score(x,yl) ) . (15)
where α is a hyperparameter that controls the extent to which an offset should be enforced. When α = 0, ODPO becomes equivalent to DPO. In tasks where a ground-truth reward model r(·) is given we set score(·) = r(·). In other tasks, one can use the quality scores given by humans to responses as score (see §6.3), or ask humans directly for the difference between the quality of the responses (Touvron et al., 2023). We ablate f in §7.1 and α in §7.2.
Connection to Softmax Margin. We now show how ODPO is connected to softmax margin (Gimpel and Smith, 2010). The idea behind the softmax margin is to augment the softmax with a cost function, such that high-cost responses get penalized more heavily. For two responses per prompt, we maximize
E (x,yw,yl)
[ log σ ( ∆r̂θ − cost(yw,yl) )] . (16)
Importantly, the cost function is by definition a nonnegative function. Therefore, ODPO loss Eq. (7c) is only equivalent to softmax-margin loss if we restrict ∆r ≥ 0.

heading: 6 Experiments
text: In this section, we empirically compare the performance of ODPO with DPO across different tasks. We refer the reader to App. C for the computational budget used for conducting the experiments.

heading: 6.1 Sentiment Control
text: In this experiment, we steer generations of GPT2-Large (Radford et al., 2019) model towards positive sentiment. Following the experimental setup in (Rafailov et al., 2023), we fine-tune GPT2-Large on the train split of IMDB dataset (Maas et al., 2011) until convergence. This gives us the SFT checkpoint πSFT.
Reward Model. We use a state-of-the-art binary sentiment classifier3 with sentiments {POS, NEG} as the reward model. Concretely if the sentiment of the response is negative, we set r(x,y) to 1 − p(NEG | ·), and if the sentiment is positive r(x,y) = 1 + p(POS | ·), where p is given by the classifier. For notational ease, we show r(x,y) with r(y).
Bootstraping the Preference Data. DPO, as opposed to RLHF, only works on top of preference data. Therefore, in tasks that a ground-truth reward model is given, the preference data needs to be bootstraped. We follow Rafailov et al. (2023) to bootstrap the preference dataset (DHF) from point-wise rewards. Given the reward function, which in this experiment is given by the sentiment classifier, and a dataset of prompts that are prefixes of movie reviews, we proceed as follows. First,
3Specifically, we use https://huggingface.co/ lvwerra/distilbert-imdb.
for each prompt x ∈ D, M responses are sampled from the language model πSFT(· | x)4 to form a multiset Yx = {yi}Mm=1,5 where ym is the mth response generated for prompt x. Next, for each i ̸= j ∈ {1, . . . ,M} a tuple (x,yw,yl) is added to DHF if r(yi) ̸= r(yj), where
(yw,yl) = { (yi,yj) if r(yi) > r(yj) (yj ,yi) if r(yi) < r(yj) . (17)
While DPO only uses the order of responses’ rewards, with ODPO we further use the reward model to determine the offset between the two responses and set ∆r = log ( r(yw) − r(yl) ) . Following Eq. (15), this is equivalent to choosing f(·) as the log function and setting α = 1.
We vary the KL regularization term β in {0.1, 0.2, . . . , 1}∪{1, 2, 3, 4, 5}6 and sample from the fine-tuned language model πθ two times with two different random seeds. This gives us 28 different samples for each method. We use these samples to approximate the average probability of the generations having positive sentiment as well as the KL divergence between the fine-tuned model πθ and the initial model πSFT. Ideally, we want not to diverge too much from the SFT policy, while generating movie reviews with a positive sentiment.
To capture the tradeoff between the KL divergence and achieved reward, we evaluate the two
4One can also compare the responses generated by the model with human written responses (Stiennon et al., 2020).
5In this experiment, we generate two responses per prompt. 6Within the range of 0 to 1, increased resolution is necessary due to considerable variance in results. However, within the range of 1 to 5, the points are closely clustered, therefore, we increase the step size when covering this range.
methods based on the Pareto frontier of achieved reward and KL divergence. Concretely, we report the percentage of points on the Pareto set that belong to each method. We compare the performance of ODPO and DPO in 3 different settings by varying the number of data points in the preference dataset, i.e., |DHF|. As depicted in Fig. 2, in all experimental setups, ODPO is more effective in generating more samples with positive sentiment, while not diverging too far from πSFT.7 The difference between the two methods is more pronounced with smaller datasets, i.e., when |DHF| = 5000.8

heading: 6.2 Toxicity Control
text: In this task, our goal is to reduce the toxicity of the generations. We use GPT-neo-2.7b (Black et al., 2021)9 as the SFT checkpoint, πSFT. We adversarially sample the prompts from REALTOXICITYPROMPTS (Gehman et al., 2020), where we sample 10000 prompts that have toxicity scores of more than 0.3. We generate two preference pairs from πSFT for each prompt and compute their rewards using a toxicity classifier.10 Similar to the previous experiment, we choose the offset as the log scaled differences between the rewards of preferred and dispreferred responses.
7Further experiments comparing SLiC-HF and ODPO show the same trends App. D.
8When using more data, e.g., |DHF| = 10000, we observe the positive sentiment probability gets closer to the maximum value 1., while this comes at the cost of diverging too much from πSFT; we see an order of magnitude larger DKL with |DHF| = 10000 compared to |DHF| = 7500.
9Specifically, we use https://huggingface.co/ EleutherAI/gpt-neo-2.7B.
10We use https://huggingface.co/facebook/ roberta-hate-speech-dynabench-r4-target.
We vary the KL regularization term β ∈ {0.05, 0.1, 0.2, 0.3, 0.4, 0.5}11 and sample from the fine-tuned model πθ. We evaluate the samples based on their average toxicity probabilities and the KL divergence between πθ and πSFT.
We observe that ODPO significantly outperforms DPO when using a dataset of size 8000 or 9000, where all the points on the Pareto front are from ODPO in Fig. 3. Concretely, in those two setups, DPO fails to meaningfully reduce the toxicity of the generations, as the toxicity scores are all close to the toxicity scores of generations from πSFT (shown with a dashed horizontal line). However, ODPO manages to significantly reduce the toxicity of the generations, cutting it almost by half. As we increase the size of the dataset, the performance of DPO gets closer to ODPO.

heading: 6.3 Summarization
text: Next, we test our method on a summarization task, where we directly use the scores given by humans to the pair of summaries for designing the offset. We use REDDIT TL;DR dataset collected by Stiennon et al. (2020). We take the subset of this dataset that is assessed by humans for quality using a 7- point Likert scale.12 For each of the two summaries of a given Reddit post, we add a datapoint to DHF if the overall quality score of one is greater than the other. Moreover, we set the offset between the two summaries as the log difference between their
11For β > 0.5, we observe that no toxicity reduction compared to the SFT model; therefore, for this experiment we only test β ≤ 0.5.
12More specifically, we take the posts under relationships and relationship_advice subreddits. For more details regarding the datasets used refer to App. B.
associated scores by humans. The supervised fine-tuned model (πSFT) is a GPTJ-6B (Wang and Komatsuzaki, 2021) model that is fine-tuned on human-written forum post summaries.13 We further fine-tune this model with DPO and ODPO on DHF.14 We use GPT-4 to compare the generations from the fine-tuned models against the reference summary written by humans, as GPT-4 judgments have been shown to strongly correlate with human judgments (Rafailov et al., 2023). See App. E for the prompt used for the evaluation.
We use 100 test prompts and sample from the fine-tuned models with different temperatures. The
13Specifically, we use https://huggingface.co/ CarperAI/openai_summarize_tldr_sft
14Following Rafailov et al. (2023), we set β = 0.5 for both methods.
win rates of different methods against human written summaries are reported in Fig. 5. We observe that both DPO and ODPO improve upon the SFT model. At lower temperatures, i.e., 0, 0.25, 0.5, both methods are on average preferred over humanwritten summaries. These results are consistent with the results reported in (Rafailov et al., 2023).
Importantly, across all sampling temperatures, the average win rate of ODPO is higher than DPO. Specifically, in temperatures 0 and 0.5 ODPO significantly outperforms DPO. We further perform a head-to-head comparison between DPO and ODPO and observe that the win rate of ODPO over DPO is [51, 50, 62, 48, 57] at temperatures [0, 0.25, 0.5, 0.75, 1.] respectively. For qualitative results refer to App. F.

heading: 7 Ablation Studies
text: In this section, we investigate and ablate certain design decisions that were made in the experiments. Specifically, we investigate the effect of the offset hyperparameter α, and the scaling function f(·) on the performance of ODPO.

heading: 7.1 Ablating the Scaling Function
text: In our experiments, we scaled the reward difference between the preferred and dispreferred responses with a log function. To better understand the effect of this scaling function, we compare it to two other alternatives, (i) using the reward difference without scaling, i.e., setting f(·) to the identity function, and (ii) using the difference between log of reward values as the offset.
We repeat the movie review generation experiment, where the goal is to generate movie reviews with a positive sentiment. For computational efficiency, we choose |DHF| = 5000. We vary β between {0.1, 0.2, . . . , 0.9} ∪ {1, 2, 3, 4, 5} and finetune πθ with the aforementioned offsets.
The results are depicted in Fig. 4. Overall, ODPO outperforms DPO with all three choices of the offset. However, there exist some notable differences between the three offsets. Scaling the offset with the log function makes the model not diverge too far from πSFT, while achieving high rewards: While the models trained with log scaled offset (left plot) achieve high rewards (around 0.8) with KL of 0.4, models without log scaling reach to 0.8 rewards only when the KL between πθ and πSFT is around 1 (right plot). Scaling each reward value separately lies somewhere in between (middle plot).

heading: 7.2 Ablating the Offset Hyperparameter
text: In the previous experiment, we set α = 1. To understand the effect of enforcing an offset in the loss function we experiment with different values of α. Repeating the movie review experiment, we vary α ∈ {0.0, 0.1, 0.2, 0.3, 0.5, 0.8, 1.0} and set the offset to ∆r = α log ( r(yw) − r(yl) ) . Note that when α = 0 ODPO loss is exactly equal to DPO loss. After fine-tuning πθ on a dataset of size 7500 with the ODPO loss Eq. (7c) (with β = 0.5), we sample from the models two times with two different random seeds and report the results in Fig. 6.
We observe that higher values of α can lead to higher reward values at the expense of diverging from the SFT model. Lower values of α on average lead to lower reward values. On the Pareto front of the Fig. 6, points with the highest rewards are associated with models trained with α = 1, while points with lower DKL are fine-tuned with smaller α values.

heading: 8 Related Work
text: In this section, we review alternative methods to RLHF for aligning language models to human preferences.
Sequence-Level Contrastive Methods. The intuition that is shared by these methods is simple: they encourage the language model πθ to assign more probability mass to the preferred response compared to the dispreferred response(s). Zhao et al. (SLIC; 2023) employ a rank calibration loss that requires log πθ(yw | x) to be greater than log πθ(yl | x), by an offset δ. Importantly, in their
formulation δ is a hyperparameter and does not depend on the the responses yw,yl. Similarly, Yuan et al. (RRHF; 2023) uses a ranking loss without the offset. While intuitive, the objective that contrastive losses are optimizing for, and its connection to the RLHF objective (maximizing the expected reward) is unclear.
DPO and Variants. DPO proposes an alternative method to optimize the regularized expected rewards without using RL algorithms. Importantly, DPO shares the objective with conventional methods for RLHF (Stiennon et al., 2020), but the optimization is done without training a separate reward model nor using RL algorithms. Since the introduction of DPO, several follow-up studies attempted to improve DPO along different dimensions. To prevent DPO from overfitting to the preference dataset, Azar et al. (2023) introduce Identity Preference Optimization (IPO). IPO replaces the unbounded function of preference probabilities in DPO loss formulation with the bounded identity function. Ethayarajh et al. (2023) propose a method called Kahneman-Tversky Optimisation (KTO) that dispenses the need for paired preference data altogether. KTO’s loss function relies on unpaired examples that are labeled either as “good” or “bad”. Zhou et al. (2024) suggest another variant of DPO specifically for multi-objective alignment.
Our Approach. ODPO attempts to solve another shortcoming of DPO, which is to treat every preference pair equally and not take into account the extent to which the two responses differ from each other. ODPO’s loss requires the estimated reward for the preferred response to be larger than the dispreferred response by an offset that depends on the difference between the quality of the responses.

heading: 9 Conclusion
text: We propose ODPO, a generalization of DPO for aligning language models with human preferences. Just as with DPO, ODPO does not rely on a pretrained reward model and does not require an RL algorithm. However, in contrast to DPO, ODPO does not treat every preference pair equally and incorporates the extent to which one response should be preferred over another in its loss function. Experiments on a variety of tasks suggest that ODPO is more effective than DPO in aligning language models to human preferences.
Limitations
Human Preference Data. Not all datasets with human feedback contain judgments regarding the extent to which one response is preferred over another. In our experiments, we focused on tasks and datasets where we had either access to such information or point-wise reward functions. The results presented in the paper provide strong motivation to ask humans to indicate their degree of preference when collecting human feedback data.
Offset values. Deciding how to scale offset values can depend on the task. In this study, we experimented with offsets based on Likert scores and classifier probabilities. We defer extending ODPO to different tasks for future work.
Ethical Considerations
We foresee two main ethical concerns regarding the use of direct preference optimization and, in general, any RLHF method for aligning language models with human preferences. First, as with DPO and other RLHF methods, malicious actors can use ODPO to steer the responses to generate harmful or toxic content. However, we must note that we foresee no particular reason for DPO to be more suitable for malicious use cases compared to DPO or other RLHF methods. Second, reward functions might inherit unwanted biases from the datasets that they were trained on. Therefore, fine-tuning to increase the expected reward can then inject such biases into the language models’ generations. We view ODPO as a fine-tuning strategy that can work with any given reward function or preference dataset. Therefore, as we develop more accurate and less biased reward functions, or find more inclusive ways of collecting human feedback, less unwanted bias will propagate to the fine-tuning process.

heading: Acknowledgements
text: We thank Alizée Pace for her feedback on an early version of this paper. Afra Amini is supported by ETH AI Center doctoral fellowship.

heading: A Proof of Thm. 1
text: Theorem 1. Let yw and yl be two responses to a prompt x, and let r̂θ(x,yw) and r̂θ(x,yl) be their associated estimated rewards. Finally, let Rw ∼ Gumbel(r̂θ(x,yw), 1) and Rl ∼ Gumbel(r̂θ(x,yl), 1) be Gumbel random variables. Then, we have,
P(Rw −Rl > 0) = pBT(yw ≻ yl | x) = σ(∆r̂θ), (11)
where pBT(yw ≻ yl | x) is a Bradley–Terry model Eq. (5a) parameterized by r̂θ(x,yw) and r̂θ(x,yl), and ∆r̂θ def = r̂θ(x,yw)− r̂θ(x,yl) is the difference between the estimated rewards.
Proof. We define a random variable I as,
I def = argmax l,w {Rl, Rw} (18)
To prove the theorem, it suffices to prove,
P(I = w) = exp
( r̂θ(x,yw) )
exp ( r̂θ(x,yw) ) + exp ( r̂θ(x,yl) ) (19)
For ease of notation, we define gr̂w def = Gumbel(r̂θ(x,yw), 1), r̂w def = r̂θ(x,yw), and r̂l def = r̂θ(x,yl). Then, consider the following manipulation
P(I = w) = E m∼gr̂w
[ P(Rl < m) ] (20a)
=
∫ +∞
−∞ gr̂w(m) exp
( − exp(r̂l −m) ) dm (20b)
=
∫ +∞
−∞ exp
( r̂w −m− exp(r̂w −m) ) exp ( − exp(r̂l −m) ) dm (20c)
=
∫ +∞
−∞ exp(r̂w) exp(−m) exp
( − exp(−m) ( exp(r̂w) + exp(r̂l︸ ︷︷ ︸
:=Z
) )) dm (20d)
= exp(r̂w)
∫ +∞
−∞ exp(−m) exp
( −Z exp(−m) ) dm (20e)
= exp(r̂w)
exp(r̂w) + exp(r̂l) . (20f)
■

heading: B Datasets Statistics
text: We used the following datasets for either fine-tuning or prompting language models, which is consistent with the intended use case of the datasets. All the datasets are in English. We refer to the corresponding papers for data collection and postprocessing procedures, as well as the demographics of human annotators.
Sentiment Control. We train the SFT model on the train set of IMDB dataset (Maas et al., 2011), which consists of 25000 movie reviews. For fine-tuning with DPO and ODPO, we sample 10000 prompts from the train set and use the language model to generate continuations. For evaluation, we sample 256 prompts from the test set of IMDB dataset.
Toxicity Control. We sample 10000 prompts for fine-tuning and 256 for evaluation from REALTOXICITYPROMPTS (Gehman et al., 2020). The dataset is released under Apache-2.0 license.
Summarization. We take 20000 posts under relationships and relationship_advice subreddits in REDDIT TL;DR dataset (Stiennon et al., 2020) and 100 posts for evaluation. The dataset is released under a modified MIT license.

heading: C Computational Budget
text: For sentiment control experiments, a single fine-tuning and evaluation run takes approximately 20 minutes on 2 rtx_4090 GPUs. For toxicity control experiments, a single fine-tuning and evaluation run takes approximately 2 hours on 2 a100_40gb GPUs. For the summarization task, a single fine-tuning and evaluation run takes approximately 15 hours on 2 a100_80gb GPUs. Notably, the reported runtimes are the same for DPO and ODPO, and there is no extra computation cost for ODPO compared to DPO.

heading: D Comparison to SLiC-HF
text: We compare DPO and ODPO to SLiC-HF (Zhao et al., 2023) on the sentiment control task. Following Zhao et al. (2023), we set the margin hyperparameter to 1 for SLiC and β to 0.5 for DPO and ODPO. We observe that none of the SLiC runs end up on the Pareto frontier of KL divergence and reward. We report the mean values and 0.9 confidence intervals for DKL(πθ || πSFT) and reward in Tab. 1 for 3 dataset sizes.

heading: E GPT-4 Evaluation Prompt for the Summarization Task
text: Following Rafailov et al. (2023) we use the prompt below to evaluate the win rates of generated summaries against human-written summaries. Which of the following summaries does a better job of summarizing the most important
points in the given forum post, without including unimportant or irrelevant details? A good summary is both concise and precise.
Post: <post> Summary A: <summary_a> Summary B: <summary_b> FIRST provide a one-sentence comparison of the two summaries, explaining which you
prefer and why. SECOND, on a new line, state only "A" or "B" to indicate your choice. Your response should use the format:
Comparison: <one-sentence comparison and explanation> Preferred: <"A" or "B">
F Qualitative Results

heading: Future Work
text: We defer extending ODPO to different tasks for future work. Ethical Considerations
We foresee two main ethical concerns regarding the use of direct preference optimization and, in general, any RLHF method for aligning language models with human preferences. First, as with DPO and other RLHF methods, malicious actors can use ODPO to steer the responses to generate harmful or toxic content. However, we must note that we foresee no particular reason for DPO to be more suitable for malicious use cases compared to DPO or other RLHF methods. Second, reward functions might inherit unwanted biases from the datasets that they were trained on. Therefore, fine-tuning to increase the expected reward can then inject such biases into the language models’ generations. We view ODPO as a fine-tuning strategy that can work with any given reward function or preference dataset. Therefore, as we develop more accurate and less biased reward functions, or find more inclusive ways of collecting human feedback, less unwanted bias will propagate to the fine-tuning process