Please explain the following like I am a layperson: HTML conversions sometimes display errors due to content that did not convert correctly from the source. This paper uses the following packages that are not yet supported by the HTML conversion tool. Feedback on these issues are not necessary; they are known and are being worked on.

failed: inconsolata
Authors: achieve the best HTML results from your LaTeX submissions by following these best practices.

License: arXiv.org perpetual non-exclusive license
arXiv:2402.13228v1 [cs.CL] 20 Feb 2024
Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive
Arka Pal
Correspondence to: <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
Deep Karkhanis
Samuel Dooley
Manley Roberts
Siddartha Naidu
Colin White
Abacus.AI
Abstract
Direct Preference Optimisation (DPO) is effective at significantly improving the performance of large language models (LLMs) on downstream tasks such as reasoning, summarisation, and alignment. Using pairs of preferred and dispreferred data, DPO models the relative probability of picking one response over another. In this work, first we show theoretically that the standard DPO loss can lead to a reduction of the model’s likelihood of the preferred examples, as long as the relative probability between the preferred and dispreferred classes increases. We then show empirically that this phenomenon occurs when fine-tuning LLMs on common datasets, especially datasets in which the edit distance between pairs of completions is low. Using these insights, we design DPO-Positive (DPOP), a new loss function and training procedure which avoids this failure mode. Surprisingly, we also find that DPOP significantly outperforms DPO across a wide variety of datasets and downstream tasks, including datasets with high edit distances between completions. By fine-tuning with DPOP, we create and release Smaug-34B and Smaug-72B, which achieve state-of-the-art open-source performance. Notably, Smaug-72B is nearly 2% better than any other open-source model on the HuggingFace Open LLM Leaderboard and becomes the first open-source LLM to surpass an average accuracy of 80%.

1Introduction
Aligning large language models (LLMs) with human preferences is important for their fluency and applicability to many tasks, with the natural language processing literature using many techniques to incorporate human feedback [Christiano et al., 2017, Stiennon et al., 2020, Ouyang et al., 2022]. Typically in LLM alignment, we first collect large amounts of preference data, consisting of a context and two potential completions; one of these is labelled as the preferred completion, and the other as the dispreferred. We use this data to learn a general policy for generating completions in a given context. Direct Preference Optimisation (DPO) [Rafailov et al., 2023] is a popular method for learning from human preferences, and it has shown to be effective at improving the performance of pretrained LLMs on downstream tasks such as reasoning, summarisation, and alignment [Wang et al., 2023, Tunstall et al., 2023]. The theoretical motivation for DPO is based on a preference-ranking model with an implicit reward function that models the relative probability of picking the preferred completion over the dispreferred.

Refer to caption
Figure 1:DPOP avoids a failure mode of DPO. When preference pairs differ on only a few tokens, DPO receives no loss incentive at all for the early tokens, and a loss incentive that in some cases can lead to degradation of the log-probs of later tokens (Section 3). We introduce DPOP, which adds a new term to the loss which leads every token to be incentivised toward the preferred completion (Section 4).
In this work, first we show theoretically that the standard DPO loss can lead to a reduction of the model’s likelihood of the preferred completions (as long as the relative probability between the preferred and dispreferred classes increases), and we empirically show that this phenomenon occurs when fine-tuning current LLMs on common datasets. Our theoretical explanation for the phenomenon suggests that the problem occurs most frequently in preference datasets with small edit distances between each pair of completions, such as in math-based preference datasets.

Using these insights, we design a new loss function: DPO-Positive (DPOP), which adds a new term to the loss function that penalises reducing the probability of the positive completions. We also create new preference datasets based on ARC [Clark et al., 2018], HellaSwag [Zellers et al., 2019], and MetaMath [Yu et al., 2023] and use them along with DPOP to create new models.

We introduce the Smaug class of models which use DPOP and achieve state-of-the-art open-source performance. We fine-tune 72B, 34B, and 7B models on our new datasets and show that DPOP far outperforms DPO. We evaluate our resulting models on multiple benchmarks including the HuggingFace Open LLM Leaderboard [Beeching et al., 2023, Gao et al., 2021], which aggregates six popular benchmarks such as MMLU [Hendrycks et al., 2021] and GSM8K [Cobbe et al., 2021], and MT-Bench [Zheng et al., 2023], a challenging benchmark that uses a strong LLM to score candidate model responses across eight different categories of performance. On the HuggingFace Open LLM Leaderboard, Smaug-72B achieves an average accuracy of 80.48%, becoming the first open-source LLM to surpass an average accuracy of 80% and improving by nearly 2% over the second-best open-source model, and our Smaug-34B model is the best in its class of models of similar parameter count. We release our code and pretrained models at https://github.com/abacusai/smaug.

Our contributions.

• We theoretically and empirically show a surprising failure mode of DPO: running DPO on preference datasets with small edit distances between completions can result in a catastrophic decrease in accuracy.
• We introduce DPO-Positive (DPOP) which we theoretically and empirically show ameliorates the performance degradation. In particular, DPOP often outperforms DPO, even on preference datasets with high edit distances between completions.
• We create new preference-based versions of ARC, HellaSwag, and MetaMath.
• Using DPOP and our new datasets, we create and release the Smaug class of models, with Smaug-72B becoming the first open-source model to achieve an average accuracy of 80% on the HuggingFace Open LLM Leaderboard. We open-source our trained models, datasets, and code.
2Background and Related Work
Large language models (LLMs) have shown impressive zero-shot and few-shot performance [Radford et al., 2019, Brown et al., 2020, Bubeck et al., 2023]. Recently, researchers have fine-tuned pretrained LLMs on downstream tasks by using human-written completions [Chung et al., 2022, Mishra et al., 2021] or by using datasets labelled with human-preferred completions relative to other completions [Ouyang et al., 2022, Bai et al., 2022, Ziegler et al., 2020]. These techniques have been used to improve performance on a variety of downstream tasks such as translation [Kreutzer et al., 2018] and summarisation [Stiennon et al., 2020], as well as to create general-purpose models such as Zephyr [Tunstall et al., 2023]. Two of the most popular techniques for learning from human preference data are reinforcement learning from human feedback (RLHF) [Ouyang et al., 2022, Bai et al., 2022, Ziegler et al., 2020] and direct preference optimisation (DPO) [Rafailov et al., 2023]. We summarise these approaches below.

RLHF
Consider a dataset of pairwise-preference ranked data 
𝒟
=
{
�
(
�
)
,
�
�
(
�
)
,
�
�
(
�
)
}
�
=
1
�
 where 
�
(
�
)
 are prompts and 
�
�
(
�
)
 and 
�
�
(
�
)
 are respectively the preferred and dispreferred completions conditioned on that prompt. We have an initial LLM 
�
ref
 that parameterises a distribution 
�
ref
⁢
(
�
|
�
)
. Often, we initialise 
�
ref
 as an LLM that has undergone supervised fine-tuning (SFT) to improve performance on downstream task(s).

RLHF begins by modelling the probability of preferring 
�
�
 to 
�
�
 using the Bradley-Terry model [Bradley and Terry, 1952] which posits the following probabilistic form:

�
⁢
(
�
�
≻
�
�
|
�
)
=
�
⁢
(
�
⁢
(
�
,
�
�
)
−
�
⁢
(
�
,
�
�
)
)
where 
�
 is the logistic function and 
�
⁢
(
�
,
�
)
 corresponds to some latent reward function that is assumed to exist for the completion 
�
 given the prompt 
�
. Given 
𝒟
, we can learn a parameterised estimate of 
�
 by minimising the negative log-likelihood of the dataset:

ℒ
�
(
�
�
,
𝒟
)
=
−
𝔼
(
�
,
�
�
,
�
�
)
∼
𝒟
[
log
(
�
(
�
�
(
�
,
�
�
)
−
�
�
(
�
,
�
�
)
)
]
.
For RLHF, we use reinforcement learning to optimise based on this learned reward function 
�
�
 (with a regularising KL-constraint to prevent model collapse), and obtain a new LLM distribution 
�
�
.

DPO
Rafailov et al. [2023] showed that it is possible to optimise the same KL-constrained reward function as in RLHF without having to learn an explicit reward function. Instead, the problem is cast as a maximum likelihood optimisation of the distribution 
�
�
 directly, with the objective:

ℒ
DPO
⁢
(
�
�
;
�
ref
)
=
−
𝔼
(
�
,
�
�
,
�
�
)
∼
�
⁢
[
log
⁡
�
⁢
(
�
⁢
log
⁡
�
�
⁢
(
�
�
|
�
)
�
ref
⁢
(
�
�
|
�
)
−
�
⁢
log
⁡
�
�
⁢
(
�
�
|
�
)
�
ref
⁢
(
�
�
|
�
)
)
]
(1)
where 
�
 is a regularisation term corresponding to the strength of KL-regularisation in RLHF. In this case, the implicit reward parameterisation is 
�
⁢
(
�
,
�
)
=
�
⁢
log
⁡
�
�
⁢
(
�
|
�
)
�
ref
⁢
(
�
|
�
)
, and Rafailov et al. [2023] further showed that all reward classes under the Plackett-Luce model [Plackett, 1975, Luce, 2005] (such as Bradley-Terry) are representable under this parameterisation. For an abbreviation, we define 
�
ratio
⁢
(
�
|
�
)
=
�
�
⁢
(
�
|
�
)
�
ref
⁢
(
�
|
�
)
.

Since the release of DPO, various alternatives have been proposed. We discuss the most relevant to our work below and in Appendix A.

IPO
Azar et al. [2023] aim to understand the theoretical underpinnings of RLHF and DPO. They identify that DPO may be prone to overfitting in situations where the preference probability of the preferred over the dispreferred examples is close to 1. They propose an alternative form of pairwise preference loss—‘Identity-PO (IPO)’. IPO tries to prevent overfitting to the preference dataset by penalising exceeding the preference margin beyond this regularised value. Conversely, we identify that DPO can lead to underfitting as well—even complete performance degradation.

3Failure Mode of DPO
In this section, we take a step back and examine the DPO loss in Equation 1, specifically with an eye towards how it can reduce the probability of the preferred completion. The loss is a function only of the difference in the log-ratios, which means that we can achieve a low loss value even if 
�
ratio
⁢
(
�
�
|
�
)
 is lowered below 1, as long as 
�
ratio
⁢
(
�
�
|
�
)
 is also lowered sufficiently. This implies that the log-likelihood of the preferred completions is reduced below the original log-likelihood from the reference model!

Why is this an issue? The original use-case of RLHF did not explicitly denote the preferred completions as being also ideal completions (rather than just the preferred completion out of the two choices 
�
�
 and 
�
�
), and hence the DPO objective is a good modelling choice. However, since then, a large body of work has focused on distilling the knowledge of powerful models into smaller or weaker models, while also showing that doing so with RLHF/DPO outperforms SFT [Taori et al., 2023, Tunstall et al., 2023, Xu et al., 2023, Chiang et al., 2023]. In this paradigm, it is often the case that in each pair of completions, the better of the two is indeed also an ideal completion. Furthermore, a new technique is to transform a standard labelled dataset into a pairwise preference dataset [Ivison et al., 2023, Tunstall et al., 2023], which also has the property that for each pair of completions, one is an ideal completion.

Edit Distance 1
While the above illustrates a hypothetical situation, now we provide a specific case in which DPO may cause a decrease in the probability of the better completion. Consider the case of trying to improve a model’s math or reasoning abilities by comparing a completion of “2+2=4” to “2+2=5.” This process creates a pair of preferred and dispreferred completions which have an edit (Hamming) distance of 1, i.e., all tokens in the completion are the same except for one. In the following, we will explore how the location of the differing token impacts the computation of the DPO loss. For sake of argument, we will examine what happens when the differing token is the first token, though the argument also follows if it appears elsewhere.

For preliminaries, consider two completions with an edit distance of 1 which differ at token 
�
 with 
1
≤
�
≤
�
, i.e., consider 
�
�
=
(
�
1
,
…
,
�
�
)
 and 
�
�
=
(
�
1
,
…
,
�
�
−
1
,
�
�
′
,
�
�
+
1
,
…
,
�
�
)
. Denote 
�
<
�
=
(
�
1
,
…
,
�
�
−
1
)
 and 
�
≥
�
=
(
�
�
,
…
,
�
�
)
. Assume that the vocabulary length of the LLM is 
�
. Let 
�
�
{
�
}
 represent the probability of the 
�
-th token in the model’s vocabulary given the input 
�
. While the LLM model parameters 
�
 are numerous, we restrict our attention to the logits, 
�
�
 with 
�
∈
[
�
]
.

The gradient of Equation 1 with respect to 
�
 is proportional to the following:

∇
�
ℒ
�
⁢
�
⁢
�
⁢
(
�
�
;
�
ref
)
∝
−
[
∇
�
log
⁡
�
�
⁢
(
�
�
|
�
)
−
∇
�
log
⁡
�
�
⁢
(
�
�
|
�
)
]
.
We note first that for 
�
>
1
, all tokens from 1 to 
�
−
1
 have no effect on the gradient, simply because for all 
1
≤
�
<
�
, 
�
�
⁢
(
�
�
|
�
�
<
�
,
�
)
=
�
�
⁢
(
�
�
|
�
�
<
�
,
�
)
, causing these tokens’ contribution to the gradient to cancel out. Therefore, without loss of generality, assume 
�
=
1
, i.e., 
�
�
 and 
�
�
 differ only at the first token. Without loss of generality, we also assume that 
�
�
 takes vocabulary position 1. Then we have the following for each 
�
>
1
 (derivation in Section B.1):

∇
�
�
log
⁡
�
�
⁢
(
�
�
|
�
�
<
�
,
�
)
−
∇
�
�
log
⁡
�
�
⁢
(
�
�
|
�
�
<
�
,
�
)
=
�
�
{
�
�
<
�
,
�
}
−
�
�
{
�
�
<
�
,
�
}
.
(2)
As we typically run DPO after SFT, the model is likely to be reasonably well optimised, so we should have 
�
�
{
�
�
<
�
,
�
}
≤
�
�
{
�
�
<
�
,
�
}
 for 
�
≠
1
 and 
�
1
{
�
�
<
�
,
�
}
≥
�
1
{
�
�
<
�
,
�
}
. Therefore, while this analysis only extends to gradients with respect to the logits, we see that the gradient vector is decreasing in the correct logit dimension and increasing in the wrong logit dimensions. Surprisingly, this suggests that under DPO, all tokens that follow a mismatched token should have reduced probability of emitting the correct token when compared to 
�
ref
. We will later give empirical evidence for this in Section 5 and Figure 3.

4DPOP
Now, we introduce DPO-Positive (DPOP), which is a solution to fix the failure mode described in the previous section. While there is no incentive for DPO to maintain the high log-likelihood of the preferred completions, DPOP adds the penalty term 
max
⁢
(
0
,
log
⁡
�
ref
⁢
(
�
�
|
�
)
log
⁡
�
�
⁢
(
�
�
|
�
)
)
 to the loss. It is 0 when 
�
ratio
⁢
(
�
�
|
�
)
≥
1
 and increases as the ratio goes below 1. Thus, the DPOP loss function is:

ℒ
DPOP
(
�
�
;
�
ref
)
=
−
𝔼
(
�
,
�
�
,
�
�
)
∼
�
[
log
�
(
�
log
�
�
⁢
(
�
�
|
�
)
�
ref
⁢
(
�
�
|
�
)
−
�
log
�
�
⁢
(
�
�
|
�
)
�
ref
⁢
(
�
�
|
�
)
)
(3)
−
�
max
(
0
,
log
�
ref
⁢
(
�
�
|
�
)
�
�
⁢
(
�
�
|
�
)
)
]
where 
�
>
0
 is a hyperparameter that can be tuned. This form of loss retains the property that we are fitting parameters on the preference data under the Bradley-Terry model. The implicit reward parameterisation is

�
⋅
�
�
⁢
(
�
|
�
)
�
ref
⁢
(
�
|
�
)
⁢
 for 
⁢
�
=
�
�
,
�
⁢
[
log
⁡
�
�
⁢
(
�
�
|
�
)
�
ref
⁢
(
�
|
�
)
−
�
⁢
max
⁢
(
0
,
log
⁡
�
ref
⁢
(
�
|
�
)
�
�
⁢
(
�
|
�
)
)
]
⁢
 for 
⁢
�
=
�
�
.
By applying this optimisation pressure, the model can no longer minimise the loss by significantly reducing the log-likelihood of the dispreferred examples more than it reduces the log-likelihood of the preferred examples; it must also ensure that the log-likelihood of the preferred examples remains high relative to the log-likelihood under the reference model.

Now, we show that Equation 3 mitigates our examples of failure modes from the previous section. Recall from Section 3 that we focused on two completions, 
�
�
 and 
�
�
, which differ by one token at location 
�
=
1
. We showed in Equation 2 that for standard DPO, the gradient of the 
�
-th token in the completions with respect to the 
�
-th logit is 
�
�
{
�
�
<
�
,
�
}
−
�
�
{
�
�
<
�
,
�
}
. However, for DPOP, if 
�
ratio
<
1
, the gradients become

∇
�
�
[
log
�
�
(
�
�
|
�
�
<
�
,
�
)
−
log
�
�
(
�
�
|
�
�
<
�
,
�
)
−
�
⋅
log
�
�
(
�
�
|
�
�
<
�
,
�
)
]
)
]
=
{
�
⁢
(
1
−
�
�
{
�
�
<
�
,
�
}
)
+
�
�
{
�
�
<
�
,
�
}
−
�
�
{
�
�
<
�
,
�
}
⁢
�
=
�
−
(
�
+
1
)
⁢
�
�
{
�
�
<
�
,
�
}
+
�
�
{
�
�
<
�
,
�
}
⁢
�
≠
�
where 
�
 is the vocabulary index of token 
�
�
. Since 
�
�
{
�
�
<
�
}
≤
1
, for the case 
�
=
�
, the gradient is guaranteed to be positive for a large enough choice of 
�
. Similarly, for the case 
�
≠
�
, the gradient is guaranteed to be negative for a large enough 
�
 (as long as 
�
�
{
�
�
<
�
}
>
0
). This therefore fixes the issue from Section 3. The derivation of the above is given in Section B.1.

Connection to Contrastive Loss
While the main motivation for DPOP is to avoid the failure mode described in Section 3, we also note its connection to contrastive loss. Contrastive learning is a popular technique in areas such as computer vision for datasets of similar and dissimilar pairs [Oord et al., 2018, Chen et al., 2020, He et al., 2020], and the loss function often uses a margin factor. Equation 3 can be viewed as similar to contrastive loss with margin 
�
=
log
⁡
1
�
ref
⁢
(
�
�
|
�
)
. We give further details in Appendix C.

5DPOP Datasets & Experiments
In this section, we empirically validate that the failure mode does arise in practice and that DPOP is able to mitigate the failure. We also show that even when the edit distance is large and DPO does not show degradation in performance, DPOP can still outperform on downstream task evaluation.

5.1Dataset Creation
For our empirical analysis, we focus on the downstream tasks of GSM8K, ARC, and HellaSwag, and we introduce and release associated paired preference-ranked datasets.

GSM8K [Cobbe et al., 2021], a dataset of diverse grade school math word problems, has been adopted as a measure of the math and reasoning skills of LLMs [Chowdhery et al., 2023, Touvron et al., 2023b, a, Beeching et al., 2023, Gao et al., 2021]. We create a paired preference-ranked version of MetaMath [Yu et al., 2023], an extended version of the GSM8K training data [An et al., 2023, Yu et al., 2023]. The correct completions in the MetaMath dataset consist of a series of steps which lead to the final answer. To create a dispreferred version, we randomly corrupt one of the results of an intermediate calculation. This dataset has a low (normalised) edit distance of 6.5%.

ARC [Clark et al., 2018] is a dataset that tests the level of understanding of science at grade-school level. We focus specifically on ARC-Challenge, the more difficult of the two subsections of ARC, which has been widely adopted as a measure of LLM reasoning and world understanding [Chowdhery et al., 2023, Touvron et al., 2023b, a, Beeching et al., 2023, Cobbe et al., 2021]. The ARC-Challenge dataset consists of four choices of responses to each question, one of which is correct. To create a paired preference-ranked dataset, for each correct response in the training split, we create three pairs using each incorrect response. Due to the differences in the responses, this dataset has a high normalised edit distance of 90%.

HellaSwag [Zellers et al., 2019] is a dataset containing commonsense inference questions known to be hard for LLMs. Similar to ARC, each question has one correct completion and three incorrect completions, and