Title: Preference Learning Algorithms Do Not Learn Preference Rankings
Venue: NeurIPS 2024
Accepted: yes
Keywords: preference learning

large language models

DPO

RLHF

ranking accuracy
Abstract: Preference learning algorithms (e.g., RLHF and DPO) are frequently used to steer LLMs to produce generations that are more preferred by humans, but our understanding of their inner workings is still limited. In this work, we study the conventional wisdom that preference learning trains models to assign higher likelihoods to more preferred outputs than less preferred outputs, measured via *ranking accuracy*.
Surprisingly, we find that most state-of-the-art preference-tuned models achieve a ranking accuracy of less than 60% on common preference datasets. We furthermore derive the *idealized ranking accuracy* that a preference-tuned LLM would achieve if it optimized the DPO or RLHF objective perfectly. We demonstrate that existing models exhibit a significant *alignment gap* -- *i.e.*, a gap between the observed and idealized ranking accuracies. 
We attribute this discrepancy to the DPO objective, which is empirically and theoretically ill-suited to correct even mild ranking errors in the reference model, and derive a simple and efficient formula for quantifying the difficulty of learning a given preference datapoint.
Finally, we demonstrate that ranking accuracy strongly correlates with the empirically popular win rate metric when the model is close to the reference model used in the objective, shedding further light on the differences between on-policy (e.g., RLHF) and off-policy (e.g., DPO) preference learning algorithms.

Preference Learning Algorithms Do Not Learn Preference Rankings

Abstract

Preference learning algorithms (e.g., RLHF and DPO) are frequently used to steer LLMs to produce generations that are more preferred by humans, but our understanding of their inner workings is still limited. In this work, we study the conventional wisdom that preference learning trains models to assign higher likelihoods to more preferred outputs than less preferred outputs, measured via ranking accuracy . Surprisingly, we find that most state-of-the-art preference-tuned models achieve a ranking accuracy of less than 60% on common preference datasets. We also derive the idealized ranking accuracy that a preference-tuned LLM would achieve if it optimized the DPO or RLHF objective perfectly. We demonstrate that existing models exhibit a significant alignment gap -i.e. , a gap between the observed and idealized ranking accuracies. We attribute this discrepancy to the DPO objective, which is empirically and theoretically ill-suited to fix even mild ranking errors in the reference model, and derive a simple and efficient formula for quantifying the difficulty of learning a given preference datapoint. Finally, we show that ranking accuracy strongly correlates with the empirically popular win rate metric when the model is close to the reference model, shedding further light on the differences between on-policy (e.g., RLHF) and off-policy (e.g., DPO) preference learning algorithms.

1 Introduction

Recent work on aligning LLMs has focused predominantly on tuning models to adhere to human preferences - commonly through reinforcement learning (RLHF; Stiennon et al. [47]) or directly via offline supervision (DPO; Rafailov et al. [41]). Preference learning algorithms [20, 55, 58] were originally designed to use a dataset of pairwise preferences over candidates to train a model with high ranking accuracy - that is, the model can precisely rank preferred outputs over dispreferred ones. In the case of language models, the ranking is determined by the likelihood assigned to each candidate.

Many LLM alignment techniques are designed to yield models with a high preference ranking accuracy, including SLiC [68, 67], RAFT [8], PRO [46], and RRHF [64]. Most prominently, Rafailov et al. [41] claimed that their popular direct preference optimization (DPO) algorithm 'increases

the relative log probability of preferred to dispreferred response." It is standard to evaluate these various objectives by measuring how often the resulting model's generations are preferred over another model's (i.e., a win rate ) [71]. However, the relationship between the loss, ranking accuracy, and win rate is unclear, leaving open the question of what these alignment techniques are actually accomplishing during training.

In this work, we demonstrate that RLHF and DPO struggle to increase ranking accuracy in practice and explore both the theoretical and empirical reasons why. Our findings highlight an intricate relationship between offline optimization and online behavior, and motivate the need for more fine-grained analyses of preference training dynamics. Our contributions are as follows:

Existing models do not achieve high ranking accuracies. We demonstrate that a wide variety of open-access preference-tuned LLMs (e.g., LLAMA 2 7B CHAT, GEMMA 7B IT, and ZEPHYR 7B DPO) achieve a ranking accuracy below 60% across a range of validation splits from commonly used preference datasets, such as UltraFeedback [7], Anthropic helpfulness and harmlessness (HH-RLHF, [14]), and Stanford Human Preferences (SHP, [11]) (Figure 1). Although we do not advocate for ranking accuracy as a measure of model quality, we analyze LLMs' ranking accuracies nonetheless because (1) ranking accuracy has motivated the design of many preference learning algorithms, and (2) DPO directly optimizes for ranking accuracy (Theorem 3.1).

Existing models exhibit a significant alignment gap between the ranking accuracy they achieve and the accuracy achievable under idealized conditions. We derive a simple formula (Theorem 3.1) for the idealized ranking accuracy (i.e., the ranking accuracy achieved from training on ground-truth preference data and perfectly optimizing the DPO or RLHF objective). We observe that models suffer from a significant alignment gap in that they achieve ranking accuracy far below the idealized ranking accuracy (Table 1, Figure 1).

Preference learning rarely corrects incorrect rankings. We prove theoretically that even mild ranking errors in the reference model can make it virtually impossible for DPO and its variants to correct the ranking (Theorem 4.1), and demonstrate that in practice, the rankings are rarely flipped (Fig. 2) and the reference model likelihoods generally determine the ranking accuracy (Fig. 3). Our results permit straightforward and efficient identification of hard-to-learn preference datapoints without any tuning.

Ranking accuracy and win rate are closely correlated when the model is close to the reference model. We observe that the ranking accuracy and win rate trend together when the model is close to the reference model during the early phase of alignment, but become anti-correlated once the model has moved too far away, adding to the ongoing discussion on the differences between on-policy and off-policy behaviors of preference-tuned LLMs.

Crucially, our work highlights fundamental flaws in RLHF and DPO that prevent the preference-tuned model from achieving a high ranking accuracy even on the training dataset .

2 Preliminaries

2.1 Learning from Human Preferences

Preference Data Human preference data typically takes the form of pairwise preferences. Each prompt x is paired with two possible continuations y 1 and y 2 . One or more human raters then annotate which continuation is preferred. When there are multiple raters, we use α ( x, y 1 , y 2 ) to denote the proportion of raters who prefer y 1 over y 2 . 1

Definition 2.1 (Aggregated Preference Datapoint) . Consider a prompt x with two possible continuations y 1 and y 2 and the proportion of raters α ( x, y 1 , y 2 ) who preferred y 1 over y 2 . Then, the aggregated preference datapoint for each prompt x is denoted ( x, y w , y l ) where y w is the completion preferred by the majority of voters.

We note that at the time of writing, the vast majority of datasets either use a single rater [14] or only release aggregated preference data [11, 26], so we often do not have access to α ( x, y 1 , y 2 ) .

A standard assumption is that the ground-truth human preferences obey the Bradley-Terry model (Assumption A.1).

Supervised Fine-Tuning (SFT) In the first step of the preference learning pipeline, the model is typically trained using the standard cross-entropy objective on some choice of offline instructiontuning dataset(s). In some implementations [52], a variety of third-party datasets are selected, whereas in other implementations [47, 41, 43] the model is instead trained on the preferred continuations ( x, y w ) from the same preference learning dataset that is used in downstream preference learning. The resulting model is often used as a reference model , denoted as π Ref or π SFT, and it typically serves as the initialization when learning from human preferences.

Reinforcement Learning from Human Feedback (RLHF) Learning from human feedback originally required using reinforcement learning [47]. In this setting, the possible continuations for each prompt are sampled from a reference model (i.e., ( y w , y l ) ∼ π Ref ( · | x ) ) and then annotated and aggregated to create a preference dataset D . Then, one frames the problem as binary classification between the two continuations and trains a reward model r ϕ ( x, y ) to minimize L R ( r ϕ , D ) = -E ( x,y w ,y l ) ∼D [log σ ( r ϕ ( x, y w ) -r ϕ ( x, y l ))] . Finally, one trains the model π θ to maximize the reward without straying too far from the reference model π Ref . Because sampling generations from the model is non-differentiable, it is common to use PPO to maximize the reward r ( x, y ) = r ϕ ( x, y ) -β (log π θ ( y | x ) -log π Ref ( y | x )) , where β > 0 is a regularization coefficient designed to prevent the model from straying too far from its initialization.

Preference Learning with DPO Rafailov et al. [41] demonstrated that one can avoid using PPO by reparametrizing the objective to operate over policies instead of over rewards. Then, one can minimize the differentiable DPO objective.

Definition 2.2 (DPO Objective [41]) . Let σ be the sigmoid function and β > 0 be a hyperparameter. Then, the DPO objective for an aggregated preference dataset D and a reference model π Ref is defined as

We denote the DPO loss on the aggregated datapoint ( x, y w , y l ) as L DPO ( x, y w , y l ; π θ , π Ref ) .

2.2 Evaluation Metrics

Evaluating the alignment of a preference-tuned LLM is both under-specified and multi-dimensional. Many knowledge-based and logic-based benchmarks (e.g. MMLU, GLUE, BIG-Bench, HELM) already exist, but these benchmarks largely fail to capture nuanced aspects of human preference, such as helpfulness or harmlessness [14]. As such, one standard evaluation is to ask human or machine raters how often the model produces a favorable completion compared to a baseline ( i.e. , win rate). Human win rate is the gold standard but is costly to compute and can be biased based on size and nature of the worker pool [19, 25]. Rating completions using another LLM (e.g., MT-bench) can be cheaper but similarly suffers from various biases [39, 69, 62], and several studies have revealed failures in many LLM judges to identify violations of instruction-following [66, 29]. Nevertheless, since win rate evaluations are so prevalent, we compare ranking accuracy against win rate in Sec. 5 and describe when the former off-policy metric is correlated with the popular on-policy metric.

Besides the win rate, preference learning algorithms are also benchmarked by the frontier of the rewards versus the divergence from the initialization [41], which serves as a heuristic of how well the model can incorporate preference data without unlearning prior information. However, it is unclear how well rewards can describe the success of alignment.

As aforementioned, the current paper investigates the ranking accuracy , which is defined as follows:

Definition 2.3 (Ranking Accuracy) . The ranking accuracy R of a model π θ on an aggregated preference datapoint ( x, y w , y l ) is defined as

Analogously, the ranking accuracy of policy π θ on a dataset D = { ( x, y w , y l ) } is R ( D ; π θ ) = E ( x,y w ,y l ) ∼D R ( x, y w , y l ; π θ ) . In the rare case where a dataset has more than two outputs y per prompt x , we use the generalized ranking accuracy definition stated in App. A.6. We do not advocate for ranking accuracy as a metric of model quality, but as a lens into the inner workings of common preference learning algorithms.

Remark 2.4 (Lengths of Completions) . We note that y w and y l can have different lengths; for example, Singhal et al. [45] showed that y w is usually longer. Length can deflate π θ ( y w | x ) and reduce the ranking accuracy. One can normalize the likelihoods by the length of the response, but the length-normalized ranking accuracy may not be meaningful in practice, because it is currently unclear how to sample from the length-normalized likelihood. For completeness, we report the ranking accuracies of both the unnormalized and normalized policies, denoted R and ˜ R , respectively.

Remark 2.5 (Difference between Ranking Accuracy and Reward Accuracy) . For RLHF models and DPO models, the ranking accuracy is not equivalent to the reward accuracy ( i.e. , the metrics evaluated in RewardBench [29]). In the RLHF case, we are evaluating the ranking accuracy of the final policy rather than the reward model. In the DPO case, reward accuracy measures whether β log π θ ( y w | x ) π Ref ( y w | x ) >

β log π θ ( y l | x ) π Ref ( y l | x ) instead of whether π θ ( y w | x ) > π θ ( y l | x ) . Since we ultimately sample from π θ rather than π θ ( y | x ) π Ref ( y | x ) , we find the ranking accuracy to be of greater practical importance.

Moreover, we demonstrate that under very stringent conditions, minimizing the DPO objective results in a model with high ranking accuracy. We characterize the phenomenon on individual datapoints, as is the case throughout the paper, but note that Markov's inequality can be straightforwardly applied to extend the results to a low loss on the entire dataset.

Proposition 2.6 (Sanity Check) . Recall the definition of y w , y l in Definition 2.1. If π Ref ( y w | x ) ≥ π Ref ( y l | x ) and L DPO ( x, y w , y l ; π θ , π Ref ) ≤ 0 . 6 , then R ( x, y w , y l ) = 1 .

This result, proved in App. A.1, requires the condition that the reference model already has the correct ranking, so it is unlikely to hold across all datapoints in practice and somewhat moot. The remainder of the paper focuses on more realistic settings where the reference model is imperfect.

3 The Alignment Gap

Prop. 2.6 showed that training a low DPO loss with a perfect reference model yields a model with perfect ranking accuracy. However, Fig. 1a shows that real-world reference models exhibit low ranking accuracies, which prompts us to study more realistic, imperfect reference models.

3.1 Existing Reference Models Rarely Have Correct Rankings

Fig. 1a indicates that reference models rarely achieve high ranking accuracy on common preference datasets (except Synthetic Instruct GPT-J Pairwise), even though many were likely trained on the preferred completions (see Sef. 2.1). Many of the models do not have documented training data so we do not know which preference datasets, if any, are in-distribution. We also fine-tune several pretrained LLMs on the preferred completions (see App. B.1) and observe that ranking accuracy does not increase significantly. 2 Based on our findings, we turn to the case of imperfect reference models.

3.2 Idealized Ranking Accuracy

We showed above that empirically, reference models exhibit poor accuracy when ranking the plausible completions. However, the RLHF reward and DPO objective were formulated to ensure that the

0%

20%

40%

60%

80%

UltraFeedback

HH-RLHF

SHP

Synthetic Instruct

GPT-J Pairwise

StackExchange

Preferences

Alpaca Farm Validation

Likelihood Calculation Method

Length-Normalized

Non-Length-Normalized

0%

20%

40%

60%

80%

UltraFeedback

HH-RLHF

SHP

Synthetic Instruct

GPT-J Pairwise

StackExchange

Preferences

Alpaca Farm Validation

Likelihood Calculation Method

Length-Normalized

Non-Length-Normalized

Figure 1: Both reference and preference-tuned models exhibit low ranking accuracy on most preference datasets. Each point represents the length-normalized or non-length-normalized ranking accuracy of individual (1a) reference models (pre-trained or fine-tuned), or (1b) preference-tuned models (trained with DPO or RLHF). The random chance accuracy for each dataset is indicated with a black 'X'. We sub-sample 1K examples from each dataset and use the test split when available. We describe datasets in B.2 and list all numbers in Tables 2, 3, and 4. For UltraFeedback, ranking accuracy is measured with exact match across all 4 outputs (see App. A.6).

model learns the preference dataset but does not move too far from the reference model π Ref, so there may be a limit on the possible accuracy of the preference-tuned model. Here, we formalize this intuition by studying what the optimal policies would be when perfectly optimizing DPO or RLHF with access to perfect data (i.e., true proportions of human preferences). 3

Theorem 3.1 (Simulating Perfect RLHF 4 ) . Fix a reference model π Ref and an aggregated preference datapoint ( x, y w , y l ) ∼ D . Assume the dataset includes the ground-truth human preferences: that is, α ( x, y w , y l ) = P ( y w ≻ y l ) , and that these preferences obey the Bradley-Terry model (Assumption A.1). Let π ∗ be the model resulting from perfectly optimizing the DPO or RLHF objective on ( x, y w , y l ) as described in Section 2.1. Then, π ∗ satisfies

where α ( x, y w , y l ) is the proportion of raters who preferred y w over y l and β is a hyperparameter in the DPO and RLHF objectives.

Remark 3.2. We prove this result in App. A.2. Note that deterministic preferences, i.e., α ( x, y w , y l ) = 1 , should not be confused with settings that only report a single individual's preference, i.e., ˆ α ( x, y w , y l ) = 1 . In the former, the optimal probability ratio is infinity. The latter requires a better estimate of P ( y w ≻ y l ) , e.g., more samples.

This result allows us to simulate the policy resulting from perfect optimization of either the RLHF or the DPO learning objective. As such, given a reference model π Ref and preference dataset D , we can easily measure the idealized ranking accuracy of a model. We prove this result in App. A.3.

Corollary 3.3 (Idealized Ranking Accuracy) . Given a reference model π Ref, the DPO or RLHF hyperparameter β , a dataset of aggregated preferences D = { ( x, y w , y l ) } and their corresponding rater proportions α ( x, y w , y l ) , the ranking accuracy of the optimum of the RLHF or DPO objective π ∗ is given by

where ✶ [ · ] is the indicator function. When computed on length-normalized likelihoods from ˜ π Ref, we denote the idealized ranking accuracy as ˜ R ∗ .

3.3 Measuring the Alignment Gap

Given access to π Ref , β , and the α ( x, y w , y l ) values for each triple ( x, y w , y l ) in a given preference dataset, we can compute the idealized ranking accuracy from Eq. 3. 5 The results are shown in Table 1 and further details are given in App. B.4.

We identify several surprising findings. Firstly, even under ideal conditions ( i.e. perfectly optimizing the objective on ground-truth preference data), the idealized ranking accuracy is still sometimes below 100% . This distance varies with the choice of β , which indicates that the limits of DPO/RLHF depend largely upon how strong the reliance on π Ref is. Furthermore, we find that many state-of-the-art models do not achieve a ranking accuracy anywhere close to the idealized ranking accuracy, exhibiting alignment gaps ranging from 19 to 59 percentage points (measured to the median idealized R or ˜ R ).

4 Understanding Ranking Accuracy with DPO

We now turn to the training objectives to account for the alignment gap. We focus our analysis on the DPO objective (Definition 2.2), because its failure to achieve high ranking accuracy is especially surprising (Table 1). In particular, DPO directly maximizes the reward margin between preferreddispreferred pairs over an offline dataset so we would expect it to perform well on in-distribution held-out data. We also note that DPO is a popular choice in the community for aligning LLMs, because it is less costly than performing RLHF.

In this section, we study real-world characteristics of DPO. First, we show empirically that DPO rarely flips the ranking of the two continuations. This result combined with the observation that reference models exhibit poor ranking accuracy (Sec. 3.1) provides an explanation for the observed poor ranking accuracies in Table 1. We then formally characterize how hard it is for DPO to correct the ranking of each datapoint.Our result highlights how the reference model conditions the optimization: as the reference model log-ratio (Definition 2.2) grows larger, one has to reduce the DPO loss to a dramatically small value to flip the incorrect ranking (Fig. 3).

4.1 DPO Rarely Flips Preference Rankings

To study how ranking accuracy changes over the course of DPO training, we train three sizes of models (GPT-2 [40], Pythia 2.8B [4], and Llama 2-7B [52]) across three seeds each on the Anthropic HH-RLHF [3] preference dataset and study the ranking accuracy on different partitions of the training dataset. We present the results from training one seed of Pythia 2.8B in Fig. 2, and defer training

0

2000

4000

6000

8000

10000 12000

Step

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

DPO Loss

Pythia 2.8B

Label

correct->correct

correct->incorrect

incorrect->correct

incorrect->incorrect

(a) DPO loss.

0

2000

4000

6000

8000

10000 12000

Step

1

0

1

2

3

4

5

6

Reward Margin

Pythia 2.8B

Label

correct->correct

correct->incorrect

incorrect->correct

incorrect->incorrect

(b) DPO reward margin.

0

2000

4000

6000

8000

10000 12000

Step

0%

20%

40%

60%

80%

100%

% of Dataset

Pythia 2.8B

Label

correct->correct

correct->incorrect

incorrect->correct

incorrect->incorrect

(c) Proportion of the dataset corre- sponding to each category of data.

details to App. C.1 and results on the other two models to App. C.2. In Fig. 2, we partition a random subsample of 1K examples from the training dataset into four groups based on whether the reference model π Ref had the correct ranking and whether the current model π θ has the correct ranking.

Surprisingly, Fig. 2 demonstrates that DPO rarely flips the ranking of ( y w , y l ) over the course of training despite consistently reducing the loss L DPO. Aside from the group of points for which the model unlearns the correct preference ranking, we observe that the loss decreases and the reward margin increases consistently while training. However, at the point of lowest validation loss (marked by the vertical dashed line in Fig. 2c), less than 10% of the originally incorrectly ranked points have been flipped to have the correct ranking . Past this point, the model begins to overfit ( i.e. , the validation loss begins to increase). DPO does not substantially improve the ranking accuracy until far past the point of overfitting. Although Theorem 3.1 predicts that the theoretically optimal DPO model exhibits high ranking accuracy, in practice the empirical endpoint of training occurs far from the theoretical optimum. This indicates that the DPO objective is ill-formulated to induce a high ranking accuracy in practice.

4.2 Analysis: How Easy Is It To Flip A Ranking?

In the result below, we show that the DPO loss can decrease substantially without any improvement on the ranking accuracy of the model. Specifically, the DPO loss that the model needs to reach in order to have the correct ranking on an example ( x, y w , y l ) depends on the quality of the reference model, quantified by the reference model log-ratio. This dependence is highly ill-conditioned, whereby using a reference model with moderately incorrect likelihoods assigned to each continuation can effectively prevent DPO from learning the correct ranking.

Theorem 4.1. Consider an aggregated preference datapoint ( x, y w , y l ) such that the reference model log-ratio is some constant c , i.e. log π Ref ( y l | x ) π Ref ( y w | x ) = c. Then, R ( x, y w , y l ) = 1 if and only if L DPO ( x, y w , y l ) ≤ -log σ ( βc ) , where σ is the sigmoid function.

Remark 4.2. It is straightforward to extend our analysis to popular variants of DPO. For illustration, we prove an analogous result for identity preference optimization (IPO, Azar et al. [2]) in App. A.5.

We prove this result in App. A.4. Our theoretical result allows us to formally identify the points that will be hard to flip in their rankings. Fig. 3 visualizes the reference model log-ratio for several settings and highlights that datapoints with even mild ranking errors in the reference model will require the loss to be reduced to a very low value in order to flip the ranking. App. E contains examples of hard-

500

0

500

log

Ref

(

y

|

x

)

Ref

(

yw

|

x

)

0.0

0.5

1.0

1.5

2.0

DPO

(

x

,

yw

,

y

;

*

,

Ref

)

GPT2

Ranking Acc.

Correct

Incorrect

500

0

500

log

Ref

(

y

|

x

)

Ref

(

yw

|

x

)

0

1

2

3

DPO

(

x

,

yw

,

y

;

*

,

Ref

)

Pythia 2.8B

Ranking Acc.

Correct

Incorrect

500

250

0

250

500

log

Ref

(

y

|

x

)

Ref

(

yw

|

x

)

0.0

0.5

1.0

1.5

2.0

DPO

(

x

,

yw

,

y

;

*

,

Ref

)

Llama 2 7B

Ranking Acc.

Correct

Incorrect

to-learn, easy-to-learn, and easy-to-flip datapoints. We observe that the hard-to-learn datapoints are substantially longer than the easy ones, and that the easy datapoints generally contain less ambiguous preference annotations. More generally, our result motivates the use of stronger π Ref models and iterative or on-policy variants of DPO [50, 65, 24, 53].

5 Ranking Accuracy and Win Rate

Our results on ranking accuracy illuminate how well DPO and RLHF can align to preference data, but we have not yet related these insights to how the generative behavior of the model changes during alignment. In particular, ranking accuracy is a convenient but off-policy metric and is thus not as widely adopted as the on-policy metric of win rate (see Sec. 2.2). Indeed, one could maximize the ranking accuracy by learning a strong classifier on the preference data, but that model may not generate high-quality text. Here, we explore the gap between on-policy (i.e., generative) and off-policy (i.e., classification) behaviors of LLMs through the lens of ranking accuracy and win rate. Since the DPO objective directly optimizes for ranking accuracy (Proposition 2.6), the relationship between these two metrics is a direct reflection of how off-policy training affects on-policy behavior.

We study the relationship between win rate and ranking accuracy in two settings: (1) during DPO training, and (2) in a DPO variant modulating the influence of π Ref. We measure the win rate on 500 responses to prompts from the training dataset using the Alpaca Eval GPT-4 [30] auto-annotator.

Setting 1: DPO Training. We measure the win rate and the ranking accuracy of a Pythia 2.8B model [4] during DPO training with the same configuration as in Section 4. See Fig. 9 for the results.

Setting 2: Attenuating the reference model. Theorem 4.1 showed that π Ref exerts a negative influence on the ranking accuracy in most cases, so we design a new objective that scales the reference model log-ratio in L DPO to further characterize how win rate and ranking accuracy relate.

Note that L γ DPO = L DPO (Definition 2.2) when γ = 1 , and a larger value of γ increases the role of the reference model. Also, γ directly scales c in Theorem 4.1, thereby controlling how easy it is to fit the data and increase the ranking accuracy. We train a range of Pythia-2.8B models using the L γ DPO objective for γ ∈ { 0 , 0 . 25 , 0 . 5 , 0 . 75 , 1 . 0 , 1 . 25 , 1 . 5 , 1 . 75 , 2 . 0 } and measure the ranking accuracies and win rates of the best model for each γ value. 6

Takeaway: Ranking accuracy correlates with win rate when the model is close to the reference model. In both settings, we observe that the win rate and ranking accuracy are highly correlated with one another in the early phase of training but become anti-correlated (i.e., ranking accuracy increases but win rate declines) as the model π θ moves away from the reference π Ref (Fig. 4). Unlike traditional overfitting, the test loss is continuing to decline at this point (Fig. 9b). Experiments in Fig. 10 with the attenuated objective in Equation (4) further show that ranking accuracy and win rate trend together when the influence of the reference model is stronger (i.e., γ is larger).

We speculate that when the model is far from the reference model, overly optimizing the reward margin can harm the generative capabilities of the model, which are primarily acquired during pre-training. In other words, the off-policy behavior of the model can no longer predictably describe the on-policy generations when the reference model used in the offline objective is far from the current model. Our findings confirm the fundamental tradeoff between fitting the preference data and maintaining generative capabilities acquired during pre-training [22] and align with prior observations that adding on-policy preference data can make offline learning more effective [50, 65, 24, 53].

1.25

1.50

1.75

2.00

2.25

t

Ref

2

2

0.56

0.57

0.58

0.59

0.60

0.61

0.63

0.64

0.65

0.66

Ref

2.5

3.0

3.5

4.0

4.5

Ref

2

2

0.6

0.7

0.8

0.9

0.60

0.65

0.70

0.75

0.80

0.85

Ref

(b) Ranking accuracy and win rate of various γ -scaled models (trained with L γ DPO for 5 epochs), versus the distance travelled by the model weights θ γ from the initialization.

6 Related Work

Analyses of Preference Learning Algorithms Many works have investigated the role of the preference dataset [56, 61], the reliability of the evaluations [69, 29], and the confounding factor of response length [45, 10, 57, 37]. Theoretical works have unified the many preference learning algorithms into clear taxonomies that permit analysis and, sometimes, yield new variants [2, 61, 48, 50, 63, 34]. Several works study idiosyncrasies of preference learning, such as why DPO decreases the likelihood of both rejected and chosen outputs from the dataset [42, 13, 35] and why RLHF exhibits vanishing gradients [44]. In contrast, our work approaches understanding DPO and RLHF through the lens of ranking accuracy, and our findings emphasize the role of the reference model regularization in preference learning. Relatedly, SliC-HF [67], CPO [59], and pairwise cringe loss [60] optimize log probability margins log π ( y w | x ) -log π ( y l | x ) , effectively removing the regularization toward the reference model. Liu et al. [32] recommend using the reference model at inference time to exert more granular control over the regularization. Meng et al. [34] remove the π Ref terms altogether and use a target reward margin to prevent over-optimization. Additionally, Chennakesavalu et al. [6] design a DPO-like objective that includes an additional hyperparameter controlling the strength of the π Ref terms, similar to our L γ DPO objective (Eq. 4). Tang et al. [50] also analyze the role of regularizing toward a reference model, though our work focuses the effect of this regularization on ranking accuracy.

On-policy and Off-policy Preference Learning Preference-tuning LLMs originally required using an on-policy algorithm [47], but many recent works have derived off-policy methods that can use a static preference dataset for supervision [41, 12, 18, 37]. Off-policy methods are preferred for their efficiency and ease of implementation, but several works have suggested that on-policy methods

are superior [48, 61, 49, 31]. Several iterative training methods aim to bridge this gap, where the reference model and the dataset are refreshed during the alignment procedure to contain annotated preferences on generations from the model at that point in training [65, 24, 53]. These intuitions align strongly with our observation that win rate and ranking accuracy, and thus, on-policy and off-policy behavior, are strongly correlated when the model is close to the reference model.

7 Discussion

Our work highlights the significant but nuanced relationship between preference learning and ranking accuracy. We have demonstrated both theoretically and empirically that RLHF and DPO struggle to teach the model to correctly rank preferred and dispreferred outputs, even in the training dataset. Although the learning objective promotes high ranking accuracy in theory (Proposition 2.6), we observed a prominent alignment gap resulting from the poor conditioning of reference models. We then drew connections between the off-policy nature of ranking accuracy and the on-policy evaluations of win rate, identifying specific scenarios in which on-policy behavior can or cannot be reliably predicted by off-policy behavior. App. 8 details the limitations of our work.

Connections to Safety Our work shows that it is difficult to steer pre-trained LLMs to adhere to even the preference data used for training. When LLMs are used to judge responses from other models [30, 69] or to improve their own abilities [33, 65], poor ranking accuracies can induce strong negative feedback loops that are costly to mitigate.

We also observe that win rate does not monotonically increase during training (Fig. 9a), despite the decrease in both train and test loss (Fig. 9b) and the modest gain in ranking accuracy (Fig. 9a). As such, it is clear that we still do not understand the behaviors of preference learning. For example, others have observed that DPO can cause the likelihoods of both chosen and rejected outputs to decrease [42, 13, 35, 36], which implies that the policy must be moving probability mass to possibly undesirable sequences outside the data distribution. Moreover, our investigation of the non-monotonic relationship between ranking accuracy and win rate emphasizes the need for concrete evaluations that can more reliably and transparently measure the success of preference learning.

Future Work Our theoretical results only describe the behavior of the model on the preference data used during training, but they can serve as a starting point for understanding generalization to different distributions of data, especially the one prescribed by the model itself [48]. Furthermore, we hope to analyze the optimization dynamics of preference learning, given the intriguing relationship observed between ranking accuracy and win rate. For instance, identifying when the win rate begins to diverge from the ranking accuracy can motivate adding fresh on-policy training data. Our initial investigation into ranking accuracy also suggests that it is worthwhile to explore how alignment techniques interact with other calibration metrics.

8 Limitations

Although we reproduce our main results (Sec. 4) on three types of models across three seeds each, these models are trained on a single dataset due to the computational constraints. Our theoretical results lead us to believe that our findings would generalize to other datasets, but empirical verification is still valuable. As mentioned previously, our work is optimization-agnostic and only describes model behavior on the training dataset, making it difficult to draw general claims about other distributions. In particular, we cannot rigorously describe the generative capabilities, though Sec. 5 initiates an investigation into when off-policy behavior can describe on-policy behaviors.

References

Alex Havrilla. synthetic-instruct-gptj-pairwise (revision cc92d8d), 2023. URL https://huggingface.co/datasets/Dahoas/ synthetic-instruct-gptj-pairwise .

Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences, 2023.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.

Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning , pages 2397-2430. PMLR, 2023.

Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324-345, 1952. ISSN 00063444. URL http://www.jstor.org/stable/2334029 .

Shriram Chennakesavalu, Frank Hu, Sebastian Ibarraran, and Grant M. Rotskoff. Energy rank alignment: Using preference optimization to search chemical space at scale, 2024.

Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023.

Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research , 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=m7p5O7zblY .

Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.

Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators, 2024.

Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V -usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 59886008. PMLR, 17-23 Jul 2022.

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization, 2024.

Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, and Wenqiang Lei. Towards analyzing and understanding the limitations of dpo: A theoretical perspective, 2024.

Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned, 2022.

Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April 2023.

Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymetman. Aligning language models with preferences through f-divergence minimization. arXiv preprint arXiv:2302.08215 , 2023.

Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, A. Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Daniel Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hanna Hajishirzi. Olmo: Accelerating the science of language models. arXiv preprint , 2024. URL https://api.semanticscholar.org/ CorpusID:267365485 .

Jiwoo Hong, Noah Lee, and James Thorne. Orpo: Monolithic preference optimization without reference model, 2024.

Tom Hosking, Phil Blunsom, and Max Bartolo. Human feedback is not gold standard, 2024.

Eyke Hüllermeier, Johannes Fürnkranz, Weiwei Cheng, and Klaus Brinker. Label ranking by learning pairwise preferences. Artificial Intelligence , 172(16):1897-1916, 2008. ISSN 0004-3702. doi: https://doi.org/10.1016/j.artint.2008.08.002. URL https://www. sciencedirect.com/science/article/pii/S000437020800101X .

Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. Camels in a changing climate: Enhancing lm adaptation with tulu 2, 2023.

Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández-Lobato, Richard E Turner, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In International Conference on Machine Learning , pages 1645-1654. PMLR, 2017.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.

Dahyun Kim, Yungi Kim, Wonho Song, Hyeonwoo Kim, Yunsu Kim, Sanghoon Kim, and Chanjun Park. sdpo: Don't use your data all at once, 2024.

Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, Bertie Vidgen, and Scott A. Hale. The prism alignment project: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models, 2024.

Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Minh Nguyen, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Alexandrovich Glushkov, Arnav Varma Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Julian Mattick. Openassistant conversations - democratizing large language model alignment. In Thirty-seventh Conference on Neural

Information Processing Systems Datasets and Benchmarks Track , 2023. URL https:// openreview.net/forum?id=VSJotgbPHF .

Tomasz Korbak, Ethan Perez, and Christopher L Buckley. Rl with kl penalties is better viewed as bayesian inference. arXiv preprint arXiv:2205.11275 , 2022.

Nathan Lambert, Lewis Tunstall, Nazneen Rajani, and Tristan Thrush. Huggingface h4 stack exchange preference dataset, 2023. URL https://huggingface.co/datasets/ HuggingFaceH4/stack-exchange-preferences .

Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024.

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models, 2023.

Ziniu Li, Tian Xu, and Yang Yu. Policy optimization in rlhf: The impact of out-of-preference data, 2024.

Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel. Decoding-time realignment of language models, 2024.

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/forum?id=S37hOerQLB .

Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734 , 2024.

Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive, 2024.

Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization, 2024.

Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization, 2024.

Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning , pages 745-750, 2007.

Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity to the order of options in multiple-choice questions. arXiv preprint arXiv:2308.11483 , 2023.

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/forum?id=HPuSIXJaa9 .

Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q ∗ : Your language model is secretly a q-function, 2024.

Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=8aHzds2uUyB .

Noam Razin, Hattie Zhou, Omid Saremi, Vimal Thilak, Arwen Bradley, Preetum Nakkiran, Joshua M. Susskind, and Etai Littwin. Vanishing gradients in reinforcement finetuning of language models. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=IcVNBR7qZi .

Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf, 2023.

Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. Proceedings of the AAAI Conference on Artificial Intelligence , 38(17):18990-18998, Mar. 2024. doi: 10.1609/aaai.v38i17.29865. URL https://ojs.aaai.org/index.php/AAAI/article/view/29865 .

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 3008-3021. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/ 1f89885d556929e98d3ef9b86448f951-Paper.pdf .

Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of llms should leverage suboptimal, on-policy data, 2024.

Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, and Will Dabney. Understanding the performance gap between online and offline alignment algorithms, 2024.

Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749 , 2024.

Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikuła, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross McIlroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open models based on gemini research and technology, 2024.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.

Hoang Tran, Chris Glaze, and Braden Hancock. Iterative dpo alignment. Technical report, Snorkel AI, 2023.

Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alexander M. Rush, and Thomas Wolf. The alignment handbook. https://github. com/huggingface/alignment-handbook , 2023.

Shankar Vembu and Thomas Gärtner. Label ranking algorithms: A survey. In Preference learning , pages 45-64. Springer, 2010.

Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, Songyang Gao, Nuo Xu, Yuhao Zhou, Xiaoran Fan, Zhiheng Xi, Jun Zhao, Xiao Wang, Tao Ji, Hang Yan, Lixing Shen, Zhan Chen, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. Secrets of rlhf in large language models part ii: Reward modeling, 2024.

Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. How far can camels go? exploring the state of instruction tuning on open resources, 2023.

Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research , 18 (136):1-46, 2017. URL http://jmlr.org/papers/v18/16-634.html .

Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation, 2024.

Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Iterative preference optimization with the pairwise cringe loss, 2024.

Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for llm alignment? a comprehensive study, 2024.

Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Yang Wang. Perils of self-feedback: Self-bias amplifies in large language models, 2024.

Joy Qiping Yang, Salman Salamatian, Ziteng Sun, Ananda Theertha Suresh, and Ahmad Beirami. Asymptotics of language model alignment, 2024.

Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 10935-10950. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ 23e6f78bdec844a9f7b6c957de2aae91-Paper-Conference.pdf .

Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models, 2024.

Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum? id=tr0KidwPLc .

Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. Slic-hf: Sequence likelihood calibration with human feedback, 2023.

Yao Zhao, Mikhail Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. Calibrating sequence likelihood improves conditional language generation. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview. net/forum?id=0qSOodKmJaN .

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2023. URL https://openreview.net/forum?id=uccHPGDlao .

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36, 2024.

Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. Secrets of rlhf in large language models part i: Ppo. arXiv preprint arXiv:2307.04964 , 2023.

A Proofs and Additional Results

Assumption A.1 (Bradley-Terry [5]) . Given a prompt x and two possible continuations y 1 and y 2 , the ground truth human preference distribution satisfies

for some ground truth reward model r ∗ .

A.1 Proof of Proposition 2.6

Proposition A.2 (Reproduced from Proposition 2.6) . Recall the definition of y w , y l in Definition 2.1. If π Ref ( y w | x ) ≥ π Ref ( y l | x ) and L DPO ( x, y w , y l ; π θ , π Ref ) ≤ 0 . 6 , then R ( x, y w , y l ) = 1 .

Proof. For notational convenience, write the log probability ratios as

Then we can express the probability that the model places on each response as

Suppose for ( x, y w , y l ) , L DPO ( x, y w , y l ; π θ , π Ref ) ≤ 0 . 6 . Expanding the DPO loss, we have

Rearranging the inequality and exponentiate on both sides, we have

Note that σ (0) = 0 . 5 < exp( -0 . 6) ≤ σ ( β ( a x -b x )) . Since the logistic function is monotonic, this implies that 0 ≤ β ( a x -b x ) and a x ≥ b x .

Writing out this last inequality,

where the last inequality is by the assumption on π Ref . We conclude that R ( x, y w , y l ) = 1 by definition.

A.2 Proof of Theorem 3.1

Theorem A.3 (Simulating Perfect RLHF) . Fix a reference model π Ref and an aggregated preference datapoint ( x, y w , y l ) ∼ D . Assume the dataset includes the ground-truth human preferences: that is, α ( x, y w , y l ) = P ( y w ≻ y l ) , and that these preferences obey the Bradley-Terry model (Assumption A.1). Let π ∗ be the (adequately expressive) model that perfectly minimizes the DPO objective on ( x, y w , y l ) , or perfectly maximizes the PPO objective on the optimal reward function as described in Section 2.1. Then, the optimal policy π ∗ satisfies

where β is a hyperparameter in the DPO and RLHF objectives.

Proof. We first prove the statement for DPO. Following the notation from [41], fix a reward function r , let π r ( y | x ) be the optimal model under the KL-constrained RL objective given the reward function r . We can express r ( x, y ) in terms of π r ( y | x ) and π ref ( y | x ) :

where Z ( x ) is the partition function for prompt x . Then, under the Bradley-Terry model, the probability of preferring y w over y l under the model π r is

Given the ground-truth human preferences, DPO's maximum likelihood objective minimizes the binary classification loss on ( x, y w , y l ) :

Let π ∗ denote an optimal policy from the loss above, the optimal preference probabilities satisfy

and the optimal policy in turn satisfies

Rearranging, we have

taking a log on both sides and divide by β ,

and finally exponentiating both sides,

the result follows by taking an inverse.

Now we show the result for RLHF, starting from the optimal solution of the KL-constrained reward maximization problem under the reward r ϕ , as derived in [41]:

The condition is straightforward to derive

where the last equality is due to the Bradley-Terry model.

A.3 Proof of Corollary 3.3

Corollary A.4 (Reproduces Corollary 3.3) . Given a reference model π Ref, the DPO or RLHF hyperparameter β , a dataset of aggregated preferences D = { ( x, y w , y l ) } and their corresponding

rater proportions α ( x, y w , y l ) , the ranking accuracy of the optimum of the RLHF or DPO objective π ∗ is given by

where ✶ [ · ] is the indicator function. When computed on length-normalized likelihoods from ˜ π Ref, we denote the idealized ranking accuracy as ˜ R ∗ .

Proof. We can see from the definition of ranking accuracy (Definition 2.3) that the accuracy on a datapoint ( x, y w , y l ) is 1 when π ( y w | x ) > π ( y l | x ) . In other words, π ( y w | x ) /π ( y l | x ) > 1 ensures that the ranking accuracy R ( x, y w , y l ) = 1 . Theorem 3.1 shows a formula for the ratio of the optimal policies, and plugging this in to the condition for ranking accuracy immediately yields the given formula for R ∗ .

A.4 Proof of Theorem 4.1

Theorem A.5. Consider an aggregated preference datapoint ( x, y w , y l ) such that the reference model log-ratio is some constant c , i.e.

Then, R ( x, y w , y l ) = 1 if and only if L DPO ( x, y w , y l ) ≤ -log σ ( βc ) , where σ is the sigmoid function.

Proof. Recall that we can break down the DPO loss into the model log-ratio and the reference model log-ratio as follows:

Observe that if L DPO ( x, y w , y l ; π θ , π Ref ) ≤ -log σ ( βc ) , then

by the monotonicity of the log and sigmoid functions. This implies that π θ ( y w | x ) π θ ( y l | x ) ≥ 1 and R ( x, y w , y l ) = 1 .

Now we show the other direction. Suppose R ( x, y w , y l ) = 1 , then π θ ( y w | x ) ≥ π θ ( y l | x ) and

Using this relationship, we have

and the other direction follows by taking the negative of both sides.

A.5 Extending Theorem 4.1 to IPO

We extend our main result on DPO to also describe the IPO objective, formally defined below. Definition A.6 (Identity Preference Optimization, a.k.a. IPO [2]) . The Identity Preference Optimization (IPO) loss is defined as L IPO ( x, y w , y l ) = ( log π θ ( y w | x ) π Ref ( y w | x ) -log π θ ( y l | x ) π Ref ( y l | x ) -1 2 τ ) 2 .

Proposition A.7. Under the IPO loss, if

then zero IPO loss implies that R ( x, y w , y l ) = 0 . On the other hand, if

then L IPO ( x, y w , y l ) ≤ ( c + 1 2 τ ) 2 guarantees that R ( x, y w , y l ) = 1 .

Different from the DPO loss, the IPO loss is a regression loss where the optimal model log-ratio has a constant margin over the reference model log-ratio. This can be seen by the following decomposition of the IPO loss:

Clearly, the optimization target of the model log-ratio is the sum of the reference model log-ratio and the margin 1 2 τ .

From Figure 3, we can see that the reference model can have a large bias towards the dispreferred completion, represented by the large negative values of the reference model log-ratio (note that in the figure, the reference model log-ratio has the dispreferred completion in the numerator). Therefore, for the optimal model under the IPO loss to have perfect ranking accuracy, the margin 1 2 τ needs to be large enough to overcome this bias for all datapoints. Alternatively, a per-example margin dependent on the reference model log-ratio of the example can be used.

A.6 Generalization of the Ranking Accuracy for Preference Datasets with n > 2 Outputs

Some datasets ( e.g. UltraFeedback [7]) contain examples with more than two responses per prompt x . In these cases, we extend our definition of ranking accuracy (Def. 2.3) to exact-match of the rankings over all choices . Suppose each aggregated datapoint consists of a prompt x and n responses ( y 1 , · · · , y n ) . Since the question of how to best aggregate rankings over multiple raters (when n > 2 ) is an open research question, we assume that there already exists some aggregated social ranking ρ ( x, y 1 , · · · , y n ) = ( ρ i ( x, y 1 , · · · , y n )) n i =1 ∈ Π[ n ] where ρ i ( x, y 1 , · · · , y n ) < ρ j ( x, y 1 , · · · , y n ) implies that y i is preferred over y j . Nowlet the ranking assigned by a policy π θ be ν ( x, y 1 , · · · , y n ) = ( ν i ( x, y 1 , · · · , y n )) n i =1 ∈ Π[ n ] , where π θ ( y i | x ) > π θ ( y j | x ) if and only if ν i ( x, y 1 , · · · , y n ) < ν j ( x, y 1 , · · · , y n ) . Then the generalized ranking accuracy is defined as follows:

R n> 2 ( x, y 1 , · · · , y n ; π θ ) = ✶ [ ρ ( x, y 1 , · · · , y n ) = ν ( x, y 1 , · · · , y n )] (11) where ✶ [ · ] is the indicator function. Analogously, the ranking accuracy over a dataset D = { ( x, y 1 , · · · , y n ) } is E ( x,y 1 , ··· ,y n ) ∼D R n> 2 ( x, y 1 , · · · , y n ; π θ ) (12)

B Experimental Details for Computing Ranking Accuracy of Open-Access LLMs

B.1 Implementation of Ranking Accuracy

We evaluate ranking accuracy for a wide range of LLMs by evaluating the likelihoods that each model π θ assigns to y w given x and y l given x , as described in Def. 2.3. However, x , y w , and y l are sequences, so we compute the sequence likelihoods by factorizing the sequence likelihood into a product of the conditional token likelihoods, like so:

We use PyTorch and the Hugging Face transformers and datasets libraries to compute all ranking accuracies. For each dataset that we evaluate ranking accuracy on, we take a random sample of 1000 examples.

Handling Ties In some datasets, such as the cross-human-annotated validation split of the Alpaca Farm dataset [9] (described in Sec. B.2), ties exist in the human annotations. When this is the case, we make a mild adjustment in the calculation of ranking accuracy to accommodate ties.

Definition B.1 (Ranking Accuracy with Ties) .

✶ where [ · ] is the indicator function. In other words, if the human annotations indicate tied preferences between y 1 and y 2 , then π θ achieves the correct ranking if and only if it assigns y 1 and y 2 approximately the same likelihood (within some tolerance level ϵ ). For all other cases, the R Ties is equivalent to R .

Throughout this paper, we use a tolerance ϵ of 0.01. In Fig. 1 we provide ranking accuracies only for examples without ties, but we provide the numbers on the full datasets below.

Length Normalization To compute the length-normalized ranking accuracy ( ˜ R ), we replace π θ ( y | x ) in Defs. 2.3 and A.8 with ˜ π θ ( y | x ) , where

B.2 Datasets

HH-RLHF [3] (helpful set) consists of two model responses for each query (often the history of a multi-turn conversation between human and chatbot), based on generations from three different classes of models (context-distilled 52B model, same model with rejection sampling using a reward model, RLHF-finetuned models). Queries and preferences annotations are obtained from crowdworkers.

Synthetic Instruct GPT-J Pairwise [1] is a synthetic dataset of queries and pairwise model generations spanning different subjects.

StackExchange Preferences [28] consists of questions and answers from Stack Exchange, where within a pair of answers, the preference between two answers is determined by a function of the number of upvotes and whether the answer was selected.

UltraFeedback [7] consists of model-generated responses from four different language models out of a larger set of models (meaning a different set of models is considered for different samples). Queries are obtained by a mixture of existing QA datasets, and the preference is annotated by GPT-4.

Stanford Human Preferences [11] is a dataset created from Reddit posts across different subject matters, where within a pair a response is considered preferred to another if was created later and has more upvotes.

Alpaca Farm Validation [9] is sourced from the Alpaca Eval dataset, but with new splits repurposed for training preference-tuned models. Additionally, we choose to use the validation split because it contains human cross-annotations ( i.e. multiple human ratings per triple of ( x, y 1 , y 2 ) ). This particular split can be found at https://huggingface.co/datasets/tatsu-lab/alpaca_eval/ blob/main/alpaca_farm_human_crossannotations.json . The original dataset, Alpaca Eval [30], is a mixture of several test sets including Open Assistant [26], a dataset of human-constructed chatbot conversation turns, HH-RLHF [3], and the Vicuna [70] and Koala test sets [15], where the former consists of user-shared queries from ShareGPT, and the latter consists of human queries from online interactions.

B.3 Full Results

We give the full set of length-normalized and non-length-normalized ranking accuracies for 16 openaccess LLMs across six datasets in Tables 2, 3, and 4. For the UltraFeedback [7] and StackExchange

Preferences [28] datasets, we use the generalized definition of ranking accuracy for n > 2 outputs instead (Def. A.8). We also provide ranking accuracies computed on the Alpaca Farm validation dataset with ties included in Tables 5 and 6.

B.4 Computation of the Idealized Ranking Accuracy

When computing the idealized ranking accuracy in Theorem 3.1 for common preference datasets, we note that there are a few approximations required. First, most datasets do not report the proportion

α ( x, y w , y l ) of raters who preferred y w over y l , and using α ( x, y w , y l ) = 1 results in errors. As a result, we measure the alignment gap on the Alpaca Farm validation split [9], which contains individual votes for each triple. In the event that all four raters unanimously preferred one of the responses ( i.e. α ( x, y l , y w ) = 0 ), we add a small constant ϵ = 0 . 001 to α ( x, y l , y w ) to prevent divison by zero. Second, the formula depends on the choice of β , which we do not know for many closed proprietary models. We circumvent this issue by computing the quantity for a range of β values ( β ∈ { 0 . 01 , 0 . 1 , 1 , 5 , 10 } ) and reporting the minimum, median, and maximum.

C Dynamics of DPO Training

C.1 Training Details

For our results in Section 4, we trained three different scales of models (GPT2 [40], Pythia 2.8B [4], and Llama 2 7B [52]) across three seeds each on the HH-RLHF dataset [3]. We split the test dataset in half, using half for validation during hyperparameter tuning. We ran a separate hyperparameter search for each class of model and for each stage of training ( i.e. SFT versus DPO). The hyperparameter ranges we searched were:

GPT2

-SFT: learning rate ∈ { 5e-7, 1e-6, 5e-6, 1e-5 } , batch size ∈ { 64, 128, 256, 512 }

-DPO: learning rate ∈ { 5e-7, 1e-6, 5e-6, 1e-5 } , batch size ∈ { 32, 64, 128 } , β ∈ { 0.01, 0.1, 1.0, 10.0 }

Pythia 2.8B

-SFT: learning rate ∈ { 1e-7, 1e-6, 1e-5 } , batch size ∈ { 16, 32, 64 }

-DPO: learning rate ∈ { 5e-7, 1e-6, 5e-6, 1e-5 } , batch size ∈ { 32, 64 } , β ∈ { 0.01, 0.1, 1.0, 10.0 }

Llama 2 7B

-SFT: learning rate ∈ { 1e-7, 1e-6, 1e-5 } , batch size ∈ { 32, 64 }

-DPO: learning rate ∈ { 1e-6, 1e-7 } , batch size ∈ { 32, 64 } , β ∈ { 0 . 1 }

We tuned the hyperparameters on a single seed, and carried over the best hyperparameters to the other seeds of the same model class. We trained the GPT2 and Pythia2.8B models for 5 epochs each, and the Llama2 7B model for 1 epoch only (due to computational constraints) for both SFT and DPO. However, most seeds of the GPT2 and Pythia 2.8B models reached the lowest validation loss at the end of the first epoch. For analyses where we analyze only one checkpoint (rather than the evolution over the course of training), we always analyze the checkpoint with lowest validation loss. We use the AdamW optimizer (with β 1 = 0 . 9 , β 2 = 0 . 999 , ϵ = 1e-8) for SFT and the RMSProp optimizer (with α = 0 . 99 , weight decay = 0 , momentum = 0 , ϵ = 1e-8) for DPO.

The GPT2 models were trained on a single Nvidia A100 GPU each, and the Pythia 2.8B and Llama 2 7B models were trained on two Nvidia A100 GPUs each. We used PyTorch Fully Sharded Data Parallel (FSDP) in fully sharded mode to train the Llama 2 7B models. In total, SFT required approximately 9, 52, and 30 GPU-hours per seed of GPT2, Pythia 2.8B, and Llama 2 7B, respectively. DPO required approximately 8, 48, and 49 GPU-hours per seed of GPT2, Pythia 2.8B, and Llama 2 7B, respectively. (Longer training times were required for Pythia 2.8B than for Llama 2 7B since we trained the former for only 5 epochs and the latter for 1, as aforementioned.)

C.2 Results

We provide the DPO loss, reward margin, and dataset trends during training across all 9 models (three seeds each of GPT2 [40], Pythia 2.8B [4], and Llama 2 8B [52]) in Figs. 5, 6, and 7.

0

5000

10000

Step

0.2

0.4

0.6

0.8

1.0

1.2

DPO Loss

0

5000

10000

Step

0

5000

10000

Step

Category of Data

correct->correct

correct->incorrect

incorrect->correct

incorrect->incorrect

(a) Three different seeds of GPT-2 [40].

0

5000

10000

Step

0.00

0.25

0.50

0.75

1.00

1.25

1.50

DPO Loss

0

5000

10000

Step

0

5000

10000

Step

Category of Data

correct->correct

correct->incorrect

incorrect->correct

incorrect->incorrect

(b) Three different seeds of Pythia 2.8B [4].

500

1000

1500

2000

2500

Step

0.0

0.5

1.0

1.5

2.0

DPO Loss

500

1000

1500

2000

2500

Step

500

1000

1500

2000

2500

Step

Category of Data

correct->correct

correct->incorrect

incorrect->correct

incorrect->incorrect

(c) Three different seeds of Llama 2-7B [52].

D Ranking Accuracy and Win Rate

We include additional plots from our exploration of the relationship of ranking accuracy and win rate in Figs. 9 and 10.

D.1 Results on the Test Set

We also provide results computed on the test set in Figs. 11 and 12.

E Qualitative Analysis

Theorem 4.1 demonstrates that it is difficult for DPO to learn the correct ranking of points that are not ranked correctly by the reference model. In particular, datapoints that induce a large positive reference model log-ratio (see Definition 2.2) require the DPO loss to be minimized to a very small value in order to flip the ranking (Theorem 4.1 and Figure 3).

Here, we document a few of the datapoints that induce a large positive reference model log-ratio and are hard to learn (Table 7), as well as datapoints that induce a very negative reference model log-ratio and are easy to learn (Table 8). These datapoints are measured using the Pythia-2.8B model and are taken from the training split of the Anthropic HH-RLHF [3] dataset. We note that datapoints

0

5000

10000

Step

1

0

1

2

Reward Margin

0

5000

10000

Step

0

5000

10000

Step

Category of Data

correct->correct

correct->incorrect

incorrect->correct

incorrect->incorrect

(a) Three different seeds of GPT-2 [40].

0

2500

5000

7500 10000 12500

Step

0

2

4

6

Reward Margin

0

2500

5000

7500 10000 12500

Step

0

2500

5000

7500 10000 12500

Step

Category of Data

correct->correct

correct->incorrect

incorrect->correct

incorrect->incorrect

(b) Three different seeds of Pythia 2.8B [4].

500

1000

1500

2000

2500

Step

2

1

0

1

2

3

Reward Margin

500

1000

1500

2000

2500

Step

500

1000

1500

2000

2500

Step

Category of Data

correct->correct

correct->incorrect

incorrect->correct

incorrect->incorrect

(c) Three different seeds of Llama 2-7B [52].

with a very negative reference model log-ratio are already ranked correctly at the start of DPO. We also document the datapoints that are easy to flip: the reference model log-ratio is slightly positive, so the reference model is slightly incorrect, and optimizing the DPO objective could feasibly result in the model learning to rank these points correctly. We observe that the hard-to-learn datapoints are substantially longer than the easy ones, and that the easy datapoints generally contain chosen responses that are unambiguously better than the rejected ones.

0

5000

10000

Step

0.0

0.1

0.2

0.3

0.4

0.5

% of Dataset

0

5000

10000

Step

0

5000

10000

Step

Category of Data

correct->correct

correct->incorrect

incorrect->correct

incorrect->incorrect

(a) Three different seeds of GPT-2 [40].

0

5000

10000

Step

0.0

0.1

0.2

0.3

0.4

0.5

% of Dataset

0

5000

10000

Step

0

5000

10000

Step

Category of Data

correct->correct

correct->incorrect

incorrect->correct

incorrect->incorrect

(b) Three different seeds of Pythia 2.8B [4].

500

1000

1500

2000

2500

Step

0.0

0.1

0.2

0.3

0.4

0.5

0.6

% of Dataset

500

1000

1500

2000

2500

Step

500

1000

1500

2000

2500

Step

Category of Data

correct->correct

correct->incorrect

incorrect->correct

incorrect->incorrect

(c) Three different seeds of Llama 2-7B [52].

500

0

500

log

Ref

(

y

|

x

)

Ref

(

yw

|

x

)

0.0

0.5

1.0

1.5

2.0

DPO

(

x

,

yw

,

y

;

*

,

Ref

)

GPT2

Ranking Acc.

Correct

Incorrect

500

0

500

log

Ref

(

y

|

x

)

Ref

(

yw

|

x

)

0

1

2

3

DPO

(

x

,

yw

,

y

;

*

,

Ref

)

Pythia 2.8B

Ranking Acc.

Correct

Incorrect

500

250

0

250

500

log

Ref

(

y

|

x

)

Ref

(

yw

|

x

)

0

1

2

DPO

(

x

,

yw

,

y

;

*

,

Ref

)

Llama 2 7B

Ranking Acc.

Correct

Incorrect

(a) Results for the second seed.

500

0

500

log

Ref

(

y

|

x

)

Ref

(

yw

|

x

)

0.0

0.5

1.0

1.5

2.0

DPO

(

x

,

yw

,

y

;

*

,

Ref

)

GPT2

Ranking Acc.

Correct

Incorrect

500

0

500

log

Ref

(

y

|

x

)

Ref

(

yw

|

x

)

0

1

2

3

DPO

(

x

,

yw

,

y

;

*

,

Ref

)

Pythia 2.8B

Ranking Acc.

Correct

Incorrect

500

250

0

250

500

log

Ref

(

y

|

x

)

Ref

(

yw

|

x

)

0

1

2

3

DPO

(

x

,

yw

,

y

;

*

,

Ref

)

Llama 2 7B

Ranking Acc.

Correct

Incorrect

(b) Results for the second seed.

500

1000

1500

2000

2500

Training Steps

0.56

0.57

0.58

0.59

0.60

0.61

0.63

0.64

0.65

0.66

Ref

(a) Ranking accuracy and win rate,

computed on

prompts from the training dataset.

500

1000

1500

2000

2500

Training Steps

0.3

0.4

0.5

0.6

DPO Loss

Train

Test

(b) Training and test DPO loss during train-

ing, up to the point of lowest val. loss.

0.0

0.5

1.0

1.5

2.0

0.6

0.7

0.8

0.9

0.60

0.65

0.70

0.75

0.80

0.85

Ref

(a) Ranking accuracy and win rate versus

γ

.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

Ref

2

2

(b) γ versus distance travelled by model param- eters during DPO training.

Ref

500

1000

1500

2000

2500

Training Steps

0.540

0.545

0.550

0.555

0.560

0.565

0.66

0.68

0.70

0.72

(a) Ranking accuracy and win rate versus training steps.

1.25

1.50

1.75

2.00

2.25

t

Ref

2

2

0.540

0.545

0.550

0.555

0.560

0.565

0.66

0.68

0.70

0.72

Ref

(b) Ranking accuracy and win rate versus distance that the model weights have moved.

0.0

0.5

1.0

1.5

2.0

0.54

0.56

0.58

0.60

0.6

0.7

0.8

Ref

(a) Ranking accuracy and win rate versus training steps.

2.5

3.0

3.5

4.0

4.5

Ref

2

2

0.54

0.56

0.58

0.60

0.6

0.7

0.8

Ref

(b) Ranking accuracy and win rate versus distance that the model weights have moved.

Table 7: Preference datapoints that are hard to learn. We print several datapoints from the training split of Anthropic HH-RLHF [3] that are hard for the Pythia-2.8B model to learn. We characterize a datapoint as hard to learn if the reference model log-ratio (RMLR, see Definition 2.2) is large. This characterization follows Theorem 4.1 and Figure 3. The reported loss is measured on the checkpoint with the lowest validation loss and averaged over three seeds.

Table 8: Preference datapoints that are easy to learn. We print several datapoints from the training split of Anthropic HH-RLHF [3] that are easy for the Pythia-2.8B model to learn. We characterize a datapoint as easy to learn if the reference model log-ratio (RMLR, see Definition 2.2) is small, or very negative. This characterization follows Theorem 4.1 and Figure 3. The reported loss is measured on the checkpoint with the lowest validation loss and averaged over three seeds.

Table 9: Preference datapoints that are easy to flip. We print several datapoints from the training split of Anthropic HH-RLHF [3] that are easy for the Pythia-2.8B model to flip. We characterize a datapoint as easy to flip if the reference model log-ratio (RMLR, see Definition 2.2) is a small positive value. This characterization follows Theorem 4.1 and Figure 3. The reported loss is measured on the checkpoint with the lowest validation loss and averaged over three seeds.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Each claim in the abstract and introduction maps directly to a proposition, theorem, corollary, and/or figure containing empirical data. The formal assumptions required for the results are summarized in natural language in the abstract and introduction. The results contain additional commentary about how the analysis can be extended to other related algorithms and how measurements can be taken for new datasets.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: Assumptions required for the theory are stated formally for each result. The discussion section lists limitations of the experimental and conceptual scopes of the paper. Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Each theorem is self-contained in stating the required assumptions needed. Proofs are provided in detail in Appendix A. We omit proof sketches due to space constraints and because the formal results are mostly straightforward algebraic manipulations.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We detail datasets, models, and precise algorithms to take measurements in the appendix. Our training runs can be reproduced with minimal modification to the HuggingFace transformer reinforcement learning (TRL) library 7 . We additionally provide anonymized code.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide an anonymized version of our code. Our results can be reproduced with minimal modification to the HuggingFace transformer reinforcement learning (TRL) library, and the datasets used are all open-access and easily downloadable from the HuggingFace Hub.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Appendices B and C detail all of the information about the data splits, hyperparameter grids, optimizer, and so on.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report measurements on numerous models and datasets, and our training runs are conducted with multiple seeds.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Appendix C.1 contains a detailing of the computational resources needed for our results. We also ran several initial experiments with IPO (Appendix A.5) but did not use the results in the main paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Confirmed.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our discussion in Section 7 discusses the impact of our findings, especially as it relates to measuring and verifying the success of preference learning and alignment of LLMs.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Not needed.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite all open-access models and datasets used in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We did not release any new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work did not involve crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the effectiveness of preference learning algorithms, include the RLHF (PPO-based) and DPO in training LLMs. In particular, the authors study the non-regularized ranking accuracy, which is largely ignored in the previous RLHF literature. The authors approach this investigation through both theoretical analysis and empirical experiments and show that whether the ranking is correct or not  highly depends on the initial reference policy.\n\nSTRENGTHS:\nThe research problem is interesting. While the RLHF and DPO mainly care about the KL-regularized target, it is debatable that the likelihood of the response under the resulting policy is what we are really interested in. Therefore, the notion of ranking policy in terms of this non-regularized target deserves more attention. \n\nWith the goal in mind, the DPO loss target in definition 2.2 clearly shows that the log-ratio on the reference model serves as an adaptive margin for the data point. Therefore, the discussions around the reference model is natural and informative. \n\nI particularly appreciate the section 4 on the empirical studies of DPO, which can largely help us to understand the dynamic of this widely adopted algorithm. \n\nOverall, the authors identify an interesting question and the present a complete story with solid evidence in both theory and experiments.\n\nWEAKNESSES:\nMy major concern is that this paper only studies the goal of helpfulness in chat style. \n\nFirst, while it is fair to say that DPO is not good at optimizing the ranking policy in terms of the non-regularized target, it is not immediately clear that the non-regularized ranking policy is a better metric. For instance, a concurrent work (so this should not hurt the novelty of this paper) [1] proposes simpo with many shared insights. In particular, they also notice the mismatch between the ranking policy and the target of DPO. However, in their experiments, as the win rates evaluated on the standard benchmark significantly increase, we also observe a significant drop in some academic benchmarks, particularly those on the mathematical problems solving. \n\nBased on my personal experience, if we adopt a smaller KL coefficient (e.g., 0.01, so less regularization), the final model performance is worse than the model with a moderate one (say 0.1). This is particularly true for the online iterative learning, which we are more interested in as the authors also notice that these variants outperform the offline one with a large margin. In most of the cases, the ability of maximizing the reward/ranking accuracy, while maintaining a moderate KL divergence is more suitable to evaluate the algorithm performance in my viewpoint. Therefore, the ranking accuracy alone is also not a good learning target, which is also evidenced by figure 4 of this work.\n\nBased on the above two points, I feel that the paper can still be largely improved if a more comprehensive evaluations can be done, like the trade-off between the performance on instruction-following benchmarks (alpaca-eval, mt-bench, gpt4 win rate) and other  academic benchmarks. But overall, I support the acceptance.\n\n[1] SimPO: Simple Preference Optimization with a Reference-Free Reward\n\nQUESTIONS:\nsee weakness\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the effectiveness of preference learning algorithms, RLHF and DPO, in training LLMs to rank human-preferred outputs higher. It challenges the assumption that these algorithms can successfully teach models to rank outputs according to human preferences. Empirical findings show that most state-of-the-art models achieve low ranking accuracy, often below 60% on standard datasets. The paper identifies a significant alignment gap due to the limitations of the DPO objective and provides a theoretical analysis of the difficulty in correcting ranking errors. It concludes by highlighting the need for a reevaluation of these algorithms and calls for more nuanced analyses of preference training dynamics.\n\nSTRENGTHS:\n1. The paper provides a thorough empirical analysis, revealing the limited effectiveness of RLHF and DPO in improving ranking accuracy, which is a significant contribution to the field.\n\n2. It offers theoretical insights into why these algorithms fail to correct ranking errors, providing a deeper understanding of the underlying issues in preference learning for LLMs.\n\n3. The findings have clear practical implications for the development of alignment techniques in LLMs, motivating the need for more effective methods to train models that better reflect human preferences.\n\nWEAKNESSES:\n1. While the paper suggests a link between preference learning and win rate, it would be strengthened by a more explicit exploration of how these metrics interrelate. Specifically, the authors could benefit from a deeper analysis of the implications of using ranking accuracy versus win rate as measures of model performance, including potential discrepancies and how they might be reconciled.\n\n2. The paper identifies a significant determinism of ranking accuracy by reference model likelihoods, which is a crucial observation. However, it would be advantageous for the authors to expand on this point by discussing potential techniques or algorithmic adjustments that could alleviate the heavy reliance on the reference model. This would bolster the paper's contribution by addressing a key challenge in the field of preference learning for LLMs.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nYes, the authors fully address the limitations and potential social implications of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the preference learning algorithms like DPO in Large Language Models (LLMs). It focuses on the ranking accuracy and shows that existing algorithms fail to achieve high ranking accuracy. Then, the paper further proposes an idealized ranking accuracy and finds that there is an alignment gap.\n\nSTRENGTHS:\n- A metric called ranking accuracy and a variant of this metric are proposed. \n\n- Extensive empirical observations of DPO are provided, which may be interesting for algorithm analysis and understanding.\n\nWEAKNESSES:\n- The motivation of studying the rank accuracy is weak and not justified well. \n\n- The analysis and results are somewhat superficial.\n\nQUESTIONS:\nThe contributions, compared with similar works, are not clear for reviewers to give an acceptance recommendation.\n\n[1] Xu, Shusheng, et al. \"Is DPO superior to PPO for LLM alignment? A comprehensive study.\" arXiv preprint arXiv:2404.10719 (2024).\n\n[2] Tajwar, Fahim, et al. \"Preference fine-tuning of LLMs should leverage suboptimal, on-policy data.\" arXiv preprint arXiv:2404.14367 (2024).\n\n[3] Tang, Yunhao, et al. \"Understanding the performance gap between online and offline alignment algorithms.\" arXiv preprint arXiv:2405.08448 (2024).\n\nThe analysis of PPO in terms of ranking accuracy is missing, which somehow makes the conclusion regarding on-policy and off-policy weak.\n\nLIMITATIONS:\n- The motivation for studying ranking accuracy is weak. The ranking accuracy, especially when defined on the offline dataset, usually does not correlate with the quality of generation behaviors. Let us split the response set into two parts, R_good and R_bad. In practice, we care that sum\\_{y} P( y in R_good) > sum\\_{y'} P(y' in R_bad). The proposed ranking accuracy, instead cares about P(y in R_good) > P(y' in R_bad), where y and y' are two events from R_good and R_bad, respectively.  Thus, a good generative model can provide many good responses with high probability and, at the same time, it may have a low ranking accuracy. This says that ranking accuracy measures the worst-case behavior of generative models. Thus, we do not need to bother ourselves to care about the ranking accuracy, especially for a small dataset where only a few pairs of preference are given. \n- The analysis in terms of ranking accuracy is somehow superficial. In fact, it is the algorithmic bias that DPO introduces the KL-regularized optimization, and as such, it cannot recover the preference-generating distribution. Briefly, the BTL assumption does not invovle the KL term. See the recent work in [1] for more discussion (since this work was posted after NeurIPS submission deadline, it does not affect the review decision). This can also be reflected in equation (2): assume that pi_REF is a uniform policy so its effect can be eliminated and the parameter beta = 1, we see that it recovers the preference-generating policy in terms of alpha. The BTL model assumes that alpha + 1 - alpha = 1, however it may not be true that pi*(y\\_w|x) + pi*(y\\_l|x) = 1.  The procedure I mentioned is also reflected in equation (4), where the authors introduce a new loss. It in fact corresponds to add a entropy regularization to eliminate the bias of pi\\_ref.\n\nBased on the considerations mentioned above, the reviewer does not clearly perceive the technical depth and the contributions of this paper.\n\n[1] Xiao, Jiancong, et al. \"On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization.\" *arXiv preprint arXiv:2405.16455* (2024).", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper empirically highlights a few potential flaws in RLHF and DPO that prevent preference-tuned models from achieving high ranking accuracy. It presents a collection of empirical and theoretical findings:\n+ Existing preference-tuned models achieve low ranking accuracies.\n+ The idealized policy and the DPO/RLHF policy has a significant gap in their ranking accuracy.\n+ Preference learning algorithms such as DPO rarely flip incorrect rankings throughout the training process.\n+ Ranking accuracy and win rate are positively correlated when models are close to the reference model.\n\nSTRENGTHS:\nThe paper provides a thorough empirical and theoretical examination of the performance of preference learning algorithms, particularly RLHF and DPO, in aligning LLMs with human preferences. The paper highlights the significant alignment gap between the observed and idealized ranking accuracies.\nThe empirical study is extensive, covering a variety of state-of-the-art preference-tuned models across multiple datasets, providing robust evidence supporting its claims on the gap between the idealized and learnt policy under the DPO/RLHF objective.\n\nWEAKNESSES:\n1. While it is already argued in the paper that ranking accuracy is practically more meaningful than the reward accuracy, it is still questionable whether totally discarding $\\pi_{ref}$ is reasonable. Specifically, the paper should have included experiment results on whether there is also a gap in reward accuracy between the learnt and ideal policy. After all, the DPO and RLHF algorithm explicitely maintain the reference model $\\pi_{ref}$, and the implicit reward should be indicative on human preference. \n\n2. Due to the small size of preference labels for any given response pair (1 for most dataset, and 4 for the Alpaca Farm as stated in Sec C.4), it is more reasonable to consider the soft ranking accuracy instead of the hard ranking accuracy. The authors could have reported both or discuss on whether hard ranking accuracy is more meaningful.\n\nQUESTIONS:\n1. One concern is that the ranking accuracy considered here is the hard accuracy (accuracy of the Bayesian optimal classifier). I wonder if the \"soft\" ranking accuracy will exhibit more noticable improvement after preference fine-tuning. The soft ranking accuracy is $\\frac{\\pi(y_w)}{\\pi(y_w) + \\pi(y_l)}$ instead of hard thresholding the predicted accuracy.\n\n2. It is not clear to me what is the main failing part in DPO/RLHF training for high ranking accuracy. In Theorem 3.1, it seems that the perfect RLHF policy should encode the information from $\\pi_{ref}$ and the dataset perfectly at the same time. Does the DPO/RLHF model fail to capture $\\pi_{ref}$ or the preference dataset?\nHave the authors tried to plot the quantity $\\frac{\\pi_{DPO}(y_w)\\pi_{ref}(y_l)}{\\pi_{DPO}(y_w)\\pi_{ref}(y_l)}$ and compare it to $(\\frac{\\alpha}{1-\\alpha})^{\\beta}$ (perhaps a scatter plot)? This can illustrate more clearly what DPO/RLHF failed in reaching the perfect policy.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}