Preference Learning Algorithms Do Not Learn Preference Rankings
Authors: Angelica Chen
Sadhika Malladi
Lily H Zhang
Xinyi Chen
Qiuyi Zhang
Rajesh Ranganath
Kyunghyun Cho
Venue: NeurIPS 2024 poster (2024)
Keywords: preference learning
large language models
DPO
RLHF
ranking accuracy
Primary area: natural_language_processing

Abstract: Preference learning algorithms (e.g., RLHF and DPO) are frequently used to steer LLMs to produce generations that are more preferred by humans, but our understanding of their inner workings is still limited. In this work, we study the conventional wisdom that preference learning trains models to assign higher likelihoods to more preferred outputs than less preferred outputs, measured via *ranking accuracy*.
Surprisingly, we find that most state-of-the-art preference-tuned models achieve a ranking accuracy of less than 60% on common preference datasets. We furthermore derive the *idealized ranking accuracy* that a preference-tuned LLM would achieve if it optimized the DPO or RLHF objective perfectly. We demonstrate that existing models exhibit a significant *alignment gap* -- *i.e.*, a gap between the observed and idealized ranking accuracies. 
We attribute this discrepancy to the DPO objective, which is empirically and theoretically ill-suited to correct even mild ranking errors in the reference model, and derive a simple and efficient formula for quantifying the difficulty of learning a given preference datapoint.
Finally, we demonstrate that ranking accuracy strongly correlates with the empirically popular win rate metric when the model is close to the reference model used in the objective, shedding further light on the differences between on-policy (e.g., RLHF) and off-policy (e.g., DPO) preference learning algorithms.

Preference Learning Algorithms Do Not Learn Preference Rankings
Angelica Chen
Sadhika Malladi Princeton University smal smalladi mmaalaad@@rinnetooo.com
Lily Zhang
New York University
New York University
ac5968enyu. edu
1ily. h. zhang@nyu. edu
Xinyi Chen
Qiuyi Zhang
Google DeepMind; Princeton University xinyic@google com
Google DeepMind qiuyizggoogle. com
Rajesh Ranganath New York University
Kyunghyun Cho New York University; Genentech; CIFAR LMB kyunghyun, cho@nyu.edu
ra rajeshrccime nyu.ndu.ee
Abstract
Preference learning algorithms (e.g., RLHF and DPO) are frequently used to steer LLMs produce generations that more preferred by humans, but our understanding of their inner workings is still limited. In this work, we study the conventional wisdom that preference learning trains models assign higher likelihoods to more preferred outputs than less preferred outputs. measured via ranking accuracy. Surprisingly, we find that most state-of-the-ar preference-ture models achieve ranking accuracy of less than 60% on common preference datasets. We also derive the idealized ranking accuracy that preference-tuned LLM would achieve if it optimized the DPO or RLHF objective perfectly. We demonstrate that existing models exhibit a significant alignment gap - i.e., a gap between the observed and idealized ranking accuracies. We attribute this discrepancy to the DPO objective, which is empirically and theoretically ill-suited fix even mild ranking emrrr in the reference model, and derive simple and efficient formula for quantifying the difficulty of learning given preference datapoint. Finally, show that ranking accuracy strongly correlates with the empirically popular win rate metric when the model is close to the reference model, shedding further light the differences between on-policy (e.g., RLHF) and off off-policy (e.g., DPO) preference learning algorithms.
Introduction
Recent work aligning LLMs has focused predominantly on tuning models adhere human preferences commonly through reinforcement learning (RLHF; Stiennon et al. [47]) or directly via offline supervision (DPO; Rafailov et al. [41]). Preference learning algorithms [20, 55, 58) were originally designed to use dataset of pairwise preferences over candidates to train model with high ranking accuracy that is, the model can precisely rank preferred outputs over dispreferred ones. In the case of language models, the ranking is determined by the likelihood assigned to each candidate.
Many LLM alignment techniques designed to yield models with high preference ranking accuracy, including SLiC [68, 67], RAFT [8], PRO [46], and RRHF [64]. Most prominently, Rafailov et al. [41] claimed that their popular direct preference optimization (DPO) algorithm "increases
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

the relative log probability of preferred to dispreferred response." It is standard to evaluate these various objectives by measuring how often the resulting model's generations are preferred over another model's (i.e., a win rate) [71]. However, the relationship between the loss, ranking accuracy. and win rate unclear, leaving open the question of what these alignment techniques are actually accomplishing during training.
In this work, we demonstrate that RLHF and DPO struggle increase ranking accuracy in practice and explore both the theoretical and empirical reasons why. Our findings highlight an intricate relationship between offline optimization and online behavior, and motivate the need for more fine-grained analyses of preference training dynamics. Our contributions are as follows:
1. Existing models do not achieve high ranking accuracies. We demonstrate that a wide variety of open-access preference-tue LLMs (e.g., LLAMA 2 7B CHAT, GEMMA 7B IT, and ZEPHYR 7B DPO) achieve a ranking accuracy below 60% across a range of validation splits from commonly used preference datasets, such as Ultraeeddbcck [7], Anthropic helpfulness and harmlessness (HH-RLHP, [14)), and Stanford Human Preferences (SHP, (11J) (Figure 1). Although we do not advocate for ranking accuracy as measure of model quality, analyze LLMs' ranking accuracies nonetheless because (1) ranking accuracy has motivated the design of many preference learning algorithms, and (2) DPO directly optimizes for ranking accuracy (Theorem 3.1).
Existing models exhibit a significant alignment gap between the ranking accuracy they achieve and the accuracy achievable under idealized conditions. We derive simple formula (Theorem 3.1) for the idealized ranking accuracy (i.e., the ranking accuracy achieved from training on ground-truth preference data and perfectly optimizing the DPO or RLHF objective). We observe that models suffer from significant alignment gap in that they achieve ranking accuracy far below the idealized ranking accuracy (Table 1, Figure 1).
3. Preference learning rarely corrects incorrect rankings. We prove theoretically that even mild ranking errors in the reference model can make it virtually impossible for DPO and its variants to correct the ranking (Theorem 4.1), and demonstrate that in practice, the rankings are rarely flipped (Fig. 2) and the reference model likelihoods generally determine the ranking accuracy (Fig. 3). Our results permit straightforward and efficient identification of hard-to-learn preference datapoints without any tuning.
4. Ranking accuracy and win rate are closely correlated when the model is close to the reference model. We observe that the ranking accuracy and win rate trend together when the model is close to the reference model during the early phase of alignment, but become anti-correlated once the model has moved too far away, adding to the ongoing discussion on the differences between on-policy and off off-policy behaviors of preference-
LLMs.
Crucially, our work highlights fundamental flaws in RLHF and DPO that prevent the preference-tuned model from achieving high ranking accuracy even the training dataset.
2 Preliminaries
2.1 Learning from Human Preferences
Preference Data Human preference data typically takes the form of pairwise preferences. Each prompt x is paired with two possible continuations - yı and y/. One more human raters then annotate which continuation is preferred. When there are multiple raters, we use a(x V1,32) to denote the proportion of raters who prefer y/ over y2.
Definition 2.1 (Aggregated Preference Datapoint). Consider prompt with two possible con- tinuations and y/2 and the proportion of raters a(x, yı, 922 who preferred yn over y2. Then, the aggregated preference datapoint for each prompt x: is denoted (Ir, yie m1) where yoe is the completion preferred by the majority of voters.
We note that at the time of writing, the vast majority of datasets either use a single rater [14] or only release aggregated preference data [11. 26), so often do not have access to az 3/. y2).
In the limit, when there are infinite raters, the empirical proportion a( 8/, yz) converges to the ground truth preference P'y1 > y2 z).

---

A standard assumption is that the ground-trun truth human preferences obey the Bradley-Terry model (Assumption A.
Supervised Fine-Tuning (SFT) In the first step of the preference learning pipeline, the model is typically trained using the standard cross-entropy objective on some choice of offline instruction- tuning dataset(s). some implementations [52], a variety of third-party datasets are selected, whereas in other implementations [47, 41, 43] the model is instead trained on the preferred continuations (z, yw) from the same preference learning dataset that is used downstream preference learning. The resulting model is often used as a reference model, denoted as TRat TSFT, and it typically serves as the initialization when learning from human preferences.
Reinforcement Learning from Human Feedback (RLHF) Learning from human feedback originally required using reinforcement learning [47]. In this setting, the possible continuations for each prompt are sampled from reference model (i.e., (Yu 31) ~ TRee(- x)) and then an- notated and aggregated to create preference dataset D. Then, one frames the problem bi- nary classification between the two continuations and trains a reward model ra y) to minimize CR(r0, D) y (wdt)>>DD)))) [log] (1r yw)) ro(a,, vi)]]. Finally, one trains the model me to maximize the reward without straying too far from the reference model TRef- Because sampling generations from the model is non-differentiable, it is common to use PPO to maximize the reward r(r,)) = (a y) (log ( y x) log ( r)), where 3 >> 0 is a regularization coefficient designed to prevent the model from straying too far from its initialization.
Preference Learning with DPO Rafailov et al. [41] demonstrated that one can avoid using PPO by reparametrizing the objective to operate over policies instead of over rewards. Then, one can minimize the differentiable DPO objective.
Definition 2.2 (DPO Objective [41]). Let be the sigmoid function and 3 > be a hyperparameter. Then, the DPO objective for an aggregated preference dataset D and a reference model TRat is defined as
Te (Bw ax) TReel tho xx)
To (yr xx) TRE (B)
LDPO TRe))
E ()
log
Blg
B log
[x]
roward margin
To (ww
xr)
TReel (31
x))
E (o
log
B Blg
+B log
ma
x))
Ree (Jhv xt
model log-ratioo
reforence model log-ratio
We denote the DPO loss on the aggregated datapoint (x, ywr 21) as LDPO(I, Mur ML; me. (Ree).
2.2 Evaluation Metrics
Evaluating the alignment of a preference-tumee LLM is both under-speeified and multi-dimennional. Many knowledge-bast based and logic-based benchmarks (e.g. MMLU, GLUE, BIG- Gennh,hh HELM) already exist, but these benchmarks largely fail to capture nuanced aspects of human preference, such as helpfulness or harmlessness [14]. As such, one standard evaluation is to ask human or machine raters how often the model produces a favorable completion compared to a baseline (i.e., win rate). Human win rate the gold standard but is costly to compute and can be biased based size and nature of the worker pool [19, 25]. Rating completions using another I.IM (e.g.. MT-hench) can be cheaper but similarly suffers from various biases [39, 69, 62], and several studies have revealed failures in many LLM judges to identify violations of instruction-followwe
[66, 29]. Nevertheless, since win rate evaluations are so prevalent, we compare ranking accuracy against win rate in Sec. and describe when the former off- off-policy metric is correlated with the popular on- policy metric.
Besides the win rate, preference learning algorithms are also benchmarked by the frontier of the rewards versus the divergence from the initialization [41], which serves as heuristic of how well the model can incorporate preference data without unlearning prior information. However, it is unclear how well rewards can describe the success of alignment.
As aforementioned, the current paper investigates the ranking accuracy, which is defined follows:
3

---

Definition 2.3 (Ranking Accuracy). The ranking accuracy R of model To on an aggregated preference datapoint (z, Nwn 3t) is defined as
R(x, your 10s mo) - {0
no (Dw 1x) 2 To (x)
otherwise.
Analogously, the ranking accuracy of policy we on dataset D = {(ar, Nww (w:nt)} is R(D; mT) = >-DD yw yt: Te). the rare case where a dataset has more than two outputs y per the generalized ranking accuracy dcfinition stated in App. A.6. We do not advocate for ranking accuracy as a metric of model quality, but as a lens into the inner workings of common preference learning algorithms.
prompt
Remark 2.4 (Lengths of Completions). We note that Yo and y can have different lengths; for example, Singhal al. [45] showed that the is usually longer. Length can deflate To To x) and reduce the ranking accuracy. One can normalize the likelihoods by the length of the response, but the length-normale ranking accuracy may not be meaningful practice, because is currently unclear how to sample from the length-normalized likelihood. For completeness, accuracies of both the unnormalized and normalized policies, denoted R. and R. respectively.
report the ranking
Remark 2.5 (Difference between Ranking Accuracy and Reward Accuracy), For RLHF models and DPO models, the ranking accuracy is not equivalent the reward accuracy (i.e., the metrics evaluated in RewardBench (29]). In the RLHF case, we are evaluating the ranking accuracy of the final policy rather than the reward model. In the DPO case, reward accuracy measures whether /3 log
Ae (Mw (z)
Tearl (x)
8log Nolvi la)
TR)(Ba(2) instead of whether (y .)) > TT (y/ z). Since we ultimately sample from m
rather than me(y)z) we find the ranking accuracy to be of greater practical importance.
Morcover, we demonstrate that under very stringent conditions, minimizing the DPO objective results model with high ranking accuracy. We characterize the phenomenon on individual datapoints, as is the case throughout the paper, but note that Markov's inequality can be straightforwardly applied extend the results a low loss on the entire dataset.
Proposition 2.6 (Sanity Check). Recall the definition of yws yr in Definition 2. If TRO x)) 2 (R x)) and (r, yw: V1; To. TRa)) S 0.6, then R(x, Yw y1) =
This result, proved in App. A.1, requires the condition that the reference model already has the correct ranking, so it is unlikely to hold across all datapoints in practice and somewhat moot. The remainder of the paper focuses on more realistic settings where the reference model is imperfect.
3 The Alignment Gap
Prop. 2.6 showed that training low DPO loss with a perfect reference model yields a model with perfect ranking accuracy. However, Fig. la shows that real-world reference models exhibit low ranking accuracies, which prompts us to study more realistic, imperfect reference models.
3.1 Existing Reference Models Rarely Have Correct Rankings
Fig. la indicates that reference models rarely achieve high ranking accuracy common preference datasets (except Synthetic Instruct GPT-J Pairwise), even though many were likely trained on the preferred completions (see Sef. 2.1). Many of the models do not have documented training data so we do not know which preference datasets, if any, are in-distribution. We also fine-tune several pretrained LLMSs on the preferred completions (see App. B. and observe that ranking accuracy does not increase significantly." Based on our findings, we turn to the case of imperfect reference models.
3.2 Idcalized Ranking Accuracy
We showed above that empirically, reference models exhibit poor accuracy when ranking the plausible completions. However, the RLHF reward and DPO objective were formulated to ensure that the
IIt is not surprising that fine- tuning on the preferred completions does not boost ranking accuracy, since the model does not receive any knowledge of the relative qualities of the preferred and rejected completions.

---

Utrafecdbock
Mathod aud
Lite Calcuration Melhod
(ength- e--yh--omkioo loed
Non No mallzed
SHP
SHP
Syntherk Insruct
GPT)
000
GPT) ro twiee
Stadkeechange
Stacklechaagee Prefermnoos
Alzaca Farm Velication
0%
20%
40%
50%
80%
Raniding
Renking Accuracy
(a) Ranking accuracies of various reference models, including GPT2 [40], PYTHIA 2.8B [4], PYTHIA 1.4B [4], LLAMA 7B [52], VICUNA 1.5 7B [69], OLMO 7B [17]. TULU2 7B [21]. ZEPHYR 7B SFT [54]. MISTRAL V0.1 7B [23], and GEMMA 7B [51]
(b) Ranking accuracies of various preference-tume -unndd models, including LLAMA 2 7B CHAT [52], TULU2 7B DPO [21], ZEPHYR 7B DPO [54], and GEMMA 7B IT [51]
Figure 1: Both reference and preference-tuned models exhibit low ranking accuracy on most preference datasets. Each point represents the length-normalized or non-length-normaaized ranking accuracy of individual (la) reference models (pre-trained or fine-tuned), or (Ib) preference-tunee models (trained with DPO or RLHF). The random chance accuracy for each dataset indicated with a black 'X'. We sub-sample IK examples from each dataset and use the test split when available. We describe datasets in B.2 and list all numbers in Tables 2, 3, and 4. For UltraFeedback, ranking accuracy is measured with exact match across all 4 outputs (see App. A.6).
model learns the preference dataset but does not move too far from the reference model TRai, so there may be limit on the possible accuracy of the preference tunee model. Here, we formalize this intuition by studying what the optimal policies would be when perfectly optimizing DPO or RLHF with access to perfect data (i.e., true proportions of human preferences).
Theorem 3.1 (Simulating Perfect RLHF'). Fix reference model neæ and an aggregated preference datapoini (I, Thes V1) ~ D. Assume the dataset includes the ground-truth human preferences: that is. a(I, Vw: y1) Pyy W1). and that these preferences obey the Bradley-Terry model (Assumption A.l). Let x² be the model resulting from perfectly optimizing the DPO or RLHF objective (z, yw y1) described in Section 2.1. Then, x² satisfies
n²" (Yar x)
(Bw x) (1 [o Yar W1
1/8
(2)
(y) x)) TRO(Un x) ca(2,, you W1
where (x, Ywn v1) is the proportion of raters who preferred the over y and /3 is hyperparameter in the DPO and RLHI objectives.
Remark 3.2. We prove this result in App. A.2. Note that deterministic preferences, i.c., alx, Muey W1) = 1, should not be confused with settings that only report single individual's prefer- ence, i.e., (xx, yw 31) = 1. In the former, the optimal probability ratio is infinity. The latter requires better estimate of IP PLy > yt), e.g., more samples.
This result allows us to simulate the policy resulting from perfect optimization of either the RLHF or the DPO learning objective. As such, given a reference model πRee and preference dataset D, can easily measure the idealized ranking accuracy of a model. We prove this result in App. A.3.
Corollary 3.3 (Idealized Ranking Accuracy). Given a reference model Reff the DPO or RLHF hyperparameter B, dataset of aggregated preferences D {(2, Yes yd) and their corresponding rater proportions a (x, yw Bi). the ranking accuracy of the optimum of the RLHF or DPO objective n² is given by
TRey (Bre x)) ( a(22 Dars 31)
1/3
R9 (D; TRO) (2.0 M/ MD
E
1
1
(3)
W1 x)) ax Dors M11
Thhis result differs from Proposition 2.6 in that accounts for potentially imperfect reference model. Wee note that this result can also be straightforwarddy derived from prior works [38, 27, 16).
5

---

Table 1: The idealized ranking accuracy of existing algorithms is not perfect, but preference- tuned models exhibit ranking accuracies far even from this idealized case. We provide both the nortainnormalized (R) and wwwpleeennonallizaeeeee (R) ranking accuracies for a variety of open-access preference-ture models on the Alpaca Farm [9] validation dataset (described in App. B.2). We also provide the idealized ranking accuracy (R" or R*, Corollary 3.3). Since idealized ranking accuracy can be computed with a variety of values of B, we provide the minimum, median, and maximum idealized ranking accuracy values for a range of B. For more details, see App. B.4.
Length- Nongth-Normalized
Non-Length- -eeegthhooooml
Preference-Tunee Model
R
R₂ (Min./Med//Max.)
R
R²
(Min//Med/MMax,
ZEPHYR-7B-DE TuLU-2-DPO-7B
54%
86% 98% / 100%
42%
90% / 19% 100%
53%
87% / 97% / 100%
42%
91% / 99%
100%
GOOGLE-GEMMA-7TBIT
54%
73% / 73% 97%
40%
67% / 93% 100%
LLAMA-2-TB-CHAT-III
53%
87% 97% / 100%
40%
91% / 99%
100%
where 1[: is the indicator function. When computed on length-normalized likelihoods from TRef we denote the idealized ranking accuracy R*
3.3 Measuring the Alignment Gap
Given access to TRef 3, and the a(x, you 31) values for each triple (x, Mer, yt) a given preference dataset, we can compute the idealized ranking accuracy from Eq. 3. The results are shown in Table 1 and further details are given in App. B.4.
We identify several surprising findings. Firstly, even under ideal conditions (i.e. perfectly optimizing the objective on ground-truth preference data), the idealized ranking accuracy still sometimes below 100%. This distance varies with the choice of B, which indicates that the limits of DPO/RLHF depend largely upon how strong the reliance on TRet is. Furthermore, we find that many state-of- tfate-oftthe--e the-art models do not achieve ranking accuracy anywhere close to the idealized ranking accuracy, exhibiting alignment gaps ranging from 19 to 59 percentage points (measured to the median idealized R. or R).
4 Understanding Ranking Accuracy with DPO
We now turn to the training objectives to account for the alignment gap. We focus our analysis on the DPO objective (Definition 2.2), because its failure achieve high ranking accuracy is especially surprising (Table 1). In particular, DPO directly maximizes the reward margin between preferred- dispreferred pairs over an offline dataset so we would expect perform well on in-distribution held-out data. We also note that DPO is a popular choice in the community for aligning LLMs, because it is less costly than performing RLHF.
In this section, we study real- world characteristics of DPO. First, we show empirically that DPO rarely flips the ranking of the two continuations. This result combined with the observation that reference models exhibit poor ranking accuracy (Sec. 3.1) provides an explanation for the observed poor ranking accuracies in Table 1. We then formally characterize how hard it is for DPO correct the ranking of each datapoint. result highlights how the reference model conditions the optimization: the reference model log-ratio (Definition 2.2) grows larger, one has to reduce the DPO loss to dramatically small value to flip the incorrect ranking (Fig. 3).
4.1 DPO Rarely Flips Preference Rankings
To study how ranking accuracy changes the course of DPO training, we train three sizes of models (GPT-2 [40], Pythia 2.8B [4], and Llama 2-7B [52]) across three seeds each on the Anthropic HH-RLHF [3] preference dataset and study the ranking accuracy different partitions of the training dataset. We present the results from training one seed of Pythia 2.8B in Fig. 2, and defer training
Note that when az, Vu, w) we replace with to compute the formula. See App. B.4.

---

AB
Pyttin 2.00
1074
Pyshi 2
Label
comctocooed
comeetllicomet
Label anymect corTEESin mosmes
Rensnn Margin
correct eeoeect correct Sircereect
Decaseet
62%
Incorici viscoredt
40%
0.7
0.0
2020
4000
5003
5000 10920 12000
Stop
(a) DPO loss.
(b) DPO reward margin.
(c) Proportion of the datasct corre- sponding to each category of data.
Figure 2: Despite continuously decreasing the loss, DPO rarely flips the rankings of pairs before the point of overfitting (marked by the vertical dashed line) and instead mostly increases the reward margin of already correctly ranked pairs. We train Pythia-2.8B model for epochs using the DPO objective and categorize the training dataset into four subsets examples that initially have the correct ranking and are flipped to (I) correct or (2) incorrect, and examples that initially have the incorrect ranking and are flipped to (3) correct (4) incorrect. In all three figures, the hue of the point indicates the category. The dashed vertical line indicates the training step at which the lowest eval. loss occurs. Past this point, the model begins to overfit (i.e., the eval. loss starts to increase). We also present results for two other models with three seeds each in Appendix C.
details to App. C.I and results on the other two models App. C.2. In Fig. we partition a random subsample of IK examples from the training dataset into four groups based on whether the reference model Ree had the correct ranking and whether the current model Ty has the correct ranking.
Surprisingly, Fig. demonstrates that DPO rarely flips the ranking of (yw: yI) over the course of training despite consistently reducing the loss Cooo- Aside from the group of points for which the model unlearns the correct preference ranking, we observe that the loss decreases and the reward margin increases consistently while training. However, at the point of lowest validation loss (marked by the vertical dashed line in Fig. 2c), less than 10% of the originally incorrectly ranked points have been flipped to have the correct ranking. Past this point, the model begins to overfit (i.e., the validation loss begins to increase). DPO does not substantially improve the ranking accuracy until far past the point of overfitting. Although Theorem 3.1 predicts that the theoretically optimal DPO model exhibits high ranking accuracy, in practice the empirical endpoint of training occurs far from the theoretical optimum. This indicates that the DPO objective is fllffammulated to induce high ranking accuracy in practice.
4.2 Analysis: How Easy Is It To Flip A Ranking?
In the result below, we show that the DPO loss can decrease substantially without any improvement on the ranking accuracy of the model. Specifically, the DPO loss that the model needs to reach in order to have the correct ranking on an example (Ir, yw: 3/) depends on the quality of the reference model, quantified by the reference model log-ratio. This dependence is highly ill-conditioned, whereby using reference model with moderately incorrect likelihoods assigned to each continuation can effectively prevent DPO from learning the correct ranking.
Theorem 4.1. Consider an aggregated preference datapoint (I,, Ywe y1) such that the reference model log-ratio is some constant c, i.e. log 3ad(ulz)
Then, R(z, Bes v1) = if and only if Coro (I, yw mt)= < - logg or (ßc). where a is the sigmoid function.
Remark 4.2. It is straightforward to extend our analysis to popular variants of DPO. For illustration, we prove an analogous result for identity preference optimization (IPO, Azar et al. (2]) in App. A.5.
We prove this result in App. A.4. Our theoretical result allows us to formally identify the points that will be hard to flip in their rankings. Fig. 3 visualizes the reference model log-ratio for several settings and highlights that datapoints with even mild ranking errors in the reference model will require the loss to be reduced to very low value in order to flip the ranking, App. E contains examples of hard-
7

---

Pythia
Liama
2.0
Ranking Acc.
Ranking Acc.
2.0
Ranking Acc.
Correct
Correct
Correct
E Y.n Ti
1.5
Incorrect
Incorrect
Vittte
Incorrect
1.0
0.5
Coro
0.5
0.0
0.0
-500
500
-500
500
-500
-250
250
500
log Rar
log BeAAFRR
lag Mer
Figure 3: DPO loss alone does not predict ranking accuracy, due to the influence of the reference model log-ratio in the loss. Each point represents the DPO loss on a separate training example (x, Vee U1) from subsample of 1K examples from the training dataset, using the model TT- that corresponds to the checkpoint with the lowest validation loss. The color of each point indicates whether mg- achieves the correct ranking on that example, i.e., whether TT-- (y/o >> wT- (y1 (z). The dashed line is the function f(c) - log (Bc), from Theorem 4,1. In summary, the examples that лт- classifies correctly tend to be those that were already classified correctly by the reference model.
Results for the other two seeds of each model are given in Fig. 8.
to-learn, easy-to-learn, and easy-to-flip datapoints. We observe that the hard-to-learn datapoints are substantially longer than the easy ones, and that the easy datapoints generally contain less ambiguous preference annotations. More generally, our result motivates the use of stronger TRaa models and iterative or on-policy variants of DPO [50, 65, 24, 53].
Ranking Accuracy and Win Rate
Our results ranking accuracy illuminate how well DPO and RLHF align to preference data, but we have not yet related these insights to how the generative behavior of the model changes during alignment. In particular, ranking accuracy a convenient but off-policy metric and thus as widely adopted as the on-policy metric of win rate (see Sec. 2.2). Indeed, one could maximize the ranking accuracy by learning strong classifier on the preference data, but that model may not generate
text. Here, we explore the gap between on-policy (i.e., generative) and off- off-policy (i.e.. classification) behaviors of LLMs through the lens of ranking accuracy and win rate. Since the DPO objective directly optimizes for ranking accuracy (Proposition 2.6), the relationship between these two metrics is direct reflection of how off- off-policy training affects on-policy behavior.
We study the relationship between win rate and ranking accuracy in two settings: (1) during DPO training. and (2) in DPO variant modulating the influence of mRaF- We measure the win rate 500 responses to prompts from the training dataset using the Alpaca Eval GPT-4 [30] auto-annotator.
Setting DPO Training. We measure the win rate and the ranking accuracy of a Pythia 2.8B model [4] during DPO training with the same configuration as in Section 4. See Fig. 9 for the results.
Setting 2: Attenuating the reference model. Theorem 4.1 showed that गeat exerts a negative influence the ranking accuracy in most cases, we design a new objective that scales the reference model log-ratio LDPO to further characterize how win rate and ranking accuracy relate.
log (s (log
To x) x))
Tee((1) x))
)
Cro TReF)
+ log
(4)
TReel (
Note that Lo = LDoO (Definition 2.2) when 7 = 1, and a larger value of increases the role the reference model. Also, directly scales in Theorem 4.1, thereby controlling how easy it is to fit the data and increase the ranking accuracy. We train a range of Pythia-2.88 models using the Po objective for y {0, 0.25, 0.5, 0.75, 1.0, 1.25, 1.5, 1.75, 2.0} and measure the ranking accuracies and win rates of the best model for each y value.
We use the best hyperparameters obtained from the experiments in Sec. 4.

---

Takeaway: Ranking accuracy correlates with win rate when the model is close to the reference model. In both settings, we observe that the win rate and ranking accuracy are highly correlated with one another in the early phase of training but become anti-correlated (i.e., ranking accuracy increases but win rate declines) as the model my moves away from the reference TRei (Fig. 4). Unlike traditional overfitting, the test loss is continuing to decline at this point (Fig. 9b). Experiments in Fig. 10 with the attenuated objective in Equation (4) further show that ranking accuracy and win rate trend together when the influence of the reference model is stronger (i.e., 7 larger).
We speculate that when the model is far from the reference model, overly optimizing the reward margin can harm the generative capabilities of the model, which are primarily acquired during pre-training other words, the off- off-pe behavior of the model can no longer predictably describe the on on-policy generations when the reference model used in the offline objective is far from the current model. Our findings confirm the fundamental tradeoff between fitting the preference data and maintaining generative capabilities acquired during pre-training [22] and align with prior observations that adding on-policy preference data can make offline learning more effective [50, 65, 24, 53].
0.61
-085
Ranking Accuracy
0.60
0.66
Win Rate VS. Maat
Ranking Accuracy
0.9
-0,80
Win Rate vS. Riter
0.59
0.65
0.8
0.75
0.58
0,70
0.64
0.7
0.65
0.56
-0.63
0.6
-0600
1.25
1.50
1.75
2.00
2.25
10
0eerll
10, 0ne/113
(a) Ranking accuracy and win rate of various Pythia 2.8B checkpoints during single epoch of DPO training. versus the distance travelled by the model weights 0, from the initialization.
(b) Ranking accuracy and win rate of various "y- scaled models (trained with Aro for 5 epochs), versus distance travelled by the model weights 0,, from the initialization.
When the model weights have not travelled far from Oet, ranking accuracy and win rate increase together. Og represents the model weights at checkpoint t during DPO training, and 0, represents the weights for a model trained convergence with LP
Figure 4:
6 Related Work
Analyses of Preference Learning Algorithms Many works have investigated the role of the preference dataset [56, 61], the reliability of the evaluations [69, 29], and the confounding factor of response length [45, 10, 57, 37]. Theoretical works have unified the many preference learning algorithms into clear taxonomies that permit analysis and, sometimes, yield new variants [2, 61, 48 50, 63, 34]. Several works study idiosyncrasies of preference learning, such as why DPO decreases the likelihood of both rejected and chosen outputs from the dataset [42, 13. 35] and why RLHF exhibits vanishing gradients [44]. In contrast, our work approaches understanding DPO and RLHF through the lens of ranking accuracy, and our findings emphasize the role of the reference model regularization in preference learning. Relatedly, SIiC-HF [67]. CPO [59], and pairwise cringe loss [60] optimize log probability margins log π (yov (z) log π [x), effectively removing the regularization toward the reference model. Liu et al. [32] recommend using the reference model at inference time to exert more granular control over the regularization. Meng et al. [34] remove the कRe terms altogether and use target reward margin prevent over-optimization. Additionally, Chennakesavalu et al. [6] design a DPO-like objective that includes an additional hyperparameter controlling the strength of the TRar terms, similar to our Lo objective (Eq. 4). Tang et al. [50] also analyze the role of regularizing toward a reference model, though our work focuses the effect of this regularization on ranking accuracy.
On-policy and Off-policy Preference Learning Preference-tuning LLMs originally required using an on-policy algorithm [47], but many recent works have derived off-policy methods that can use statie preference dataset for supervision [41, 12, 18, 37]. Off-policy methods are preferred for their efficiency and ease of implementation, but several works have suggested that on-policy methods

---

are superior [48, 61, 49, 31]. Several iterative training methods aim to bridge this gap, where the reference model and the dataset are refreshed during the alignment procedure to contain annotated preferences on generations from the model at that point in training [65, 24, 53]. These intuitions align strongly with our observation that win rate and ranking accuracy, and thus, on-policy and off-policy behavior, are strongly correlated when the model is close to the reference model.
7 Discussion
Our work highlights the significant but nuanced relationship between preference learning and ranking accuracy. We have demonstrated both theoretically and empirically that RLHF and DPO struggle to teach the model to correctly rank preferred and dispreferred outputs, even in the training dataset. Although the learning objective promotes high ranking accuracy in theory (Proposition 2.6), observed a prominent alignment gap resulting from the poor conditioning of reference models. We then drew connections between the off-policy nature of ranking accuracy and the on-policy evaluations of win rate, identifying specific scenarios in which on-policy behavior can or cannot be reliably predicted by off-policy behavior. App. 8 details the limitations of our work.
Connections to Safety Our work shows that it is difficult to steer pre-trained LLMs to adhere even the preference data used for training. When LLMs are used to judge responses from other models [30, 69] or to improve their own abilities [33, 65], poor ranking accuracies can induce strong negative feedback loops that are costly to mitigate.
We also observe that win rate does not monotonically increase during training (Fig. 9a), despite the decrease in both train and test loss (Fig. 9b) and the modest gain in ranking accuracy (Fig. 9a). As such, it is clear that we still do not understand the behaviors of preference learning. For example, others have observed that DPO can cause the likelihoods of both chosen and rejected outputs to decrease [42, 13, 35, 36], which implies that the policy must be moving probability mass to possibly undesirable sequences outside the data distribution. Moreover, our investigation of the non-monotonic relationship between ranking accuracy and win rate emphasizes the need for concrete evaluations that can more reliably and transparently measure the success of preference learning.
Future Work Our theoretical results only describe the behavior of the model on the preference data used during training, but they can serve as a starting point for understanding generalization to different distributions of data, especially the one prescribed by the model itself [48]. Furthermore, we hope to analyze the optimization dynamics of preference learning, given the intriguing relationship observed between ranking accuracy and win rate. For instance, identifying when the win rate begins to diverge from the ranking accuracy can motivate adding fresh on on-policy training data. Our initial investigation into ranking accuracy also suggests that it is worthwhile to explore how alignment techniques interact with other calibration metrics.
Limitations
Although reproduce our main results (Sec. 4) three types of models across three seeds each, these models are trained on a single dataset due to the computational constraints. Our theoretical results lead us to believe that our findings would generalize to other datasets, but empirical verification is still valuable. As mentioned previously, our work is optimization-agnostt and only describes model behavior on the training dataset, making it difficult to draw general claims about other distributions. In particular, we cannot rigorously describe the generative capabilities, though Sec. 5 initiates an investigation into when off off-policy behavior can describe on-policy behaviors.
10

---

Acknowledgements
We thank Sanjeev Arora, Tianyu Gao, Eric Mitchell, Richard Pang, and Mengzhou Xia for helpful discussions during the development of this project. We also thank Nikita Nangia, Eshaan Nichani, and Alexander Wettig for their help in proofreading the work. This work was supported by National Science Foundation Award 1922658, the Samsung Advanced Institute of Technology (under the project Next Generation Deep Learning: From Pattern Recognition to AI), NIH/NHLBI Award R01HL148248, NSF CAREER Award 2145542, ONR N00014-23-1-2634 Apple, and Google. SM additionally supported by ONR, NSF, and DARPA. We also thank Princeton Language and Intelligence (PLI) for computing resources and OpenAl credits, and NYU High Performance Computing (HPC) for computing resources and in-kind support.
References
[1] Alex Havrilla. 2023.
(revision cc92d8d), http:::://hhggigggac.c.o co//aaaaetss/aaaaeeaaaeeaaeeeeee /Dahoas/ synthetic- incle-iinstuuttt.. -pteee-ppj--- apti-pairwise.
URL syntheric-instrnttggiiiiaaaivvsi
[2] Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences, 2023.
[3] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage. Zac Ratfield-Doder Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022.
[4] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling, In International Conference on Machine Learning, pages 2397-2430. PMLR, 2023.
[5] Ralph Allan Bradley and Milton F. Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952. ISSN 00063444. URL http: / www w jrtor.
/23402293
[6] Shriram Chennakesavalu, Frank Hu, Sebastian Ibarraran, and Grant M. Rotskoff, Energy rank alignment: Using preference optimization to search chemical space at scale, 2024.
[7] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023.
[8] Hanze Dong, Wei Xiong. Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023. URL https: wwwpeeveeeeeeeeeeeeeepee...ee.we net/ for metiddmmmmpppppmmmm
ISSN 2835-8856
[9] Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. 2023.
[10] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsumori B. Hashimoto. Length-controlle alpacaeval: A simple way to debias automatic evaluators, 2024.
[11] Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pages 5988- 6008. PMLR, 17-23 Jul 2022.
[12] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment prospect theoretic optimization, 2024.

---

[13] Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, and Wenqiang Lei. Towards analyzing and understanding the limitations of dpo: theoretical perspective. 2024.
[14] Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Haffield-Doddss, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned, 2022.
[15] Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April 2023.
[16] Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymetman. Aligning language models with preferences through f-divergence minimization. arXiv preprint arXiv:2302 2302::2300.082155 2023.
[17] Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, A. Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang. Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Daniel Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zeulemoyer, Jesse Dodge, Kyle Lo. Luca Soldaini, Noah A. Smith, and Hanna Hajishirzi. Olmo: Accelerating the science of language models. arXiv preprint, 2024. URL https: /api..p../// semant icschclar. org/
CorpusID: 22::::667380 277367734555
[18] Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model, 2024.
[19] Tom Hosking, Phil Blunsom, and Max Bartolo. Human feedback is not gold standard, 2024.
[20] Eyke Hüllermeier, Johannes Fürnkranz, Weiwei Cheng. and Klaus Brinker. Label rank- ing by learning pairwise preferences. Artificial Intelligence, 197(1661187-1116, 2008 ISSN 0004-3702. doi: http://bbboor//11.0011f......... science rect. com/ http:/cciccecccceeeeeee
URL https:/ /wwww.
[21] Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. Camels changing climate: Enhancing Im adaptation with tulu 2023.
[22] Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández-Looatu Richard E Turner, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In International Conference on Machine Learning, pages 1645-1654. PMLR, 2017.
[23] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut
Lavril, Thomas Wang, Timothée Lacroix, and William EI Sayed. Mistral 7b, 2023.
[24]
Dahyun Kim, Yungi Kim, Wonho Song, Hyconwoo Kim, Yunsu Kim, Sanghoo Kim, and Chanjun Park. sdpo: Don't use your data all at once, 2024.
[25] Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, Bertie Vidgen, and Scott A. Hale. The prism alignment project: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models, 2024.
[26] Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis. Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Minh Nguyen, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Alexandrovich Glushkov, Arnav Varma Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Julian Mattick. Openassistant conversations democratizing large language model alignment. In Thirty-sventth Conference Neural
12

---

Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://
openreview... wwwp:eviwwwwwweeooeeeeeeeeeeeeee htt.://otooto.tdd//.oootoooomoo
[27] Tomasz Korbak, Ethan Perez, and Christopher I. Buckley. RI with kl penalties is better viewed as bayesian inference. arXiv preprini arXiv:2205. /275, 2022.
[28] Nathan Lambert, Lewis Tunstall, Nazneen Rajani, and Tristan Thrush. Huggingface h4 http:facc..o/aatttteee
stack exchange preference dataset, 2023. URL
HuggingFacenn acell// stt//staaaae
[29] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick. Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardhench: Evaluating reward models for language modeling, 2024.
[30] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of instraction-followinn models, 2023.
[31] Ziniu Li, Tian Xu, and Yang Yu. Policy optimization in rlhf: The impact of out-of-prefereee data, 2024.
[32] Tianlin Liu, Shangmin Guo, Leonardo Bianco. Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel. Decoding-tina realignment of language models, 2024.
[33] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In Thirty-seventh Conference on Neural Information Pro- cessing Systems, 2023. URL https: wpwnrevieeeeeeee.eeeeeeeeeeeeeee net / fwrum? i id= hhOerQLB.
[34] Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple preference optimization with reference-free reward. arXiv preprint MrXiv:240S.147:4 2024.
[35] Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positios 2024.
[36] Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization, 2024.
[37] Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization, 2024.
[38] Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine
learning, pages 745-750, 2007.
[39] Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity the order of options in multiple-choice questions. arXiv preprint arXiv:2308. 1 1483, 2023.
[40] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodci, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
[41] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Seirth-eeentth Conference on Neural Information Processing Systems, 2023. URL https: /tte:://ppenee http///ppnneiii..eeeeee/ee net/ / forum? id=
[42] Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From q": Your language model is secretly a q-function, 2024.
[43] Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In The Eleventh International Conference on Learning Representations, 2023. URL https: : / http//peervveeeeee net forum? d= du
[44] Noam Razin, Hattie Zhou, Omid Saremi, Vimal Thilak, Arwen Bradley, Preetum Nakkiran, Joshua M. Susskind, and Etai Litwin. Vanishing gradients in reinforcement finetuning of language models. In The Twelfth International Conference on Learning Representations, 2024. httttt:/opppeeeeiiiee oppenreviewwweeee net forumoiddddooo I
URL https:/
13

---

[45]
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf, 2023.
[46] Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. Proceedings of the AAAI Conference on Artificial Intelligence,
URL Mar. 2024. doi: www.///aa...3811..222222 /tttiaaleeeteee/223333
https: / /ttp::///jj/.o/ .aa.jjssaaaaaa aaai. org/ www.iindx....ppp.m
[47] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 3008-3022 Curran Associates, Inc., 2020. URL https: httttp:::/ppccceeeiinii neurip....euiipp.. 2c htt paper__ppppppp wriees/papee/ 20000/220///// file/ 1 1
-aaper-p-pr-- pdf.
[48] Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of Ilms should leverage suboptimal, on-policy data, 2024.
[49] Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, and Will Dabney. Understanding the performance gap between online and offline alignment algorithms, 2024.
[50] Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint 2wXiv:2402.05749 2024.
[51] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, Seoore-Chrristian Muraru, Grigory Rozhdestvenskly. Henryk Michalewski, Ian Tenney. Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikuta, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona COmanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins. Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojeiech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong. Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alck Andreev, and Kathleen Kenealy. Gemma: Open models based on gemini research and technology, 2024.
[52] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams. Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[53] Hoang Tran, Chris Glaze, and Braden Hancock. Iterative dpo alignment. Technical report, Snorkel AI, 2023.
14

---

[54]
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alexander M. Rush, and Thomas Wolf. The alignment handbook. https: /gittb...
com/hugg ng ce/a a gnnment- -wanennnttannbookkk 2023.
[55]
Shankar Vembu and Thomas Gärtner. Label ranking algorithms: A survey. In Preference learning, pages 45-64. Springer, 2010.
[56] Binghai Wang. Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, Songyang Gao, Nuo Xu, Yuhao Zhou, Xiaoran Fan, Zhiheng Xi, Jun Zhao, Xiao Wang, Tao Ji, Hang Yan, Lixing Shen, Zhan Chen, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang. Zuxuan Wu, and Yu-Gang Jiang. Secrets of rlhf in large language models part ii: Reward modeling, 2024.
[57] Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Ha- jishirzi. How far can camels go? exploring the state of instruction tuning on open resources, 2023.
[58] Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18 (136):1-66, 2017. URL http: / fmlr. org. hwtp:oogrppppaeepeeeeeee papers/ /v1 8 6 634 .tmll.
[59] Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Ken- ton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation, 2024.
[60] Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Iterative preference optimization with the pairwise cringe loss, 2024.
[61] Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for IIm alignment? a comprehensive study, 2024.
[62] Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Yang Wang. Perils of self-feedback: Self-bias amplifies in large language models, 2024.
[63] Joy Qiping Yang, Salman Salamatian, Ziteng Sun, Ananda Theertha Suresh, and Ahmad Beirami. Asymptotics of language model alignment, 2024.
[64] Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Informa- fion Processing Systems, volume 36, pages 11935-10950. Curran Associates, Inc., 2023. URL https: http::::s///ccceeeeeeeoo neurips wwtpcccppppppeeeee sipperr_fllleeeeeeeeeeeeeeeeeeee Setaperr2023/Eae wet.seeee7d22eeeeeeee Paapll-appp---eee www--oooffeeeeeeeeeeee pdf.
[65] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models, 2024.
[66] Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. In The Twelfth International Con- ference on Learning Representations, 2024. URL https: //oooreeeiiieee net/ / forum? id=trokidwP
[67] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter Liu. Slic-hf: Sequence likelihood calibration with human feedback, 2023.
[68] Yao Zhao, Mikhail Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. Calibrating sequence likelihood improves conditional language generation. In The Eleventh International Conference on Learning Representations, 2023, URL
[69] Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging I.LM-as-a-judge with MT-bbnch and chatbot arena. In Thirty-sevntth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: / wttp///wpppeeeeee/////p/e...eewe net/ wwwormmidduucceeeii d=uccHPGD ao.
[70] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging ilm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
15

---

[71] Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. Secrets of rlhf in large language models part Ppo. arXiv preprint arXiv:3307.04964, 2023.
16

---

A Proofs and Additional Results
Assumption A.1 (Bradley-Terry [5]). Given prompt i and possible continuations y1 and y2. the ground truth human preference distribution satisfies
exp(r (z. 1/1)
P > y/ x²) exp(r"(z,,1)) exp (x,
(5)
for some ground truth reward model r*.
A.1 Proof of Proposition 2.6
Proposition A.2 (Reproduced from Proposition 2.6). Recall the definition of yw- Vn in Definition 2.1. If (Be r) 2 (Mr z) and LDpo(a (I, Yor yt; me, RRd) S 0.6, then R(x yorr y1)
Proof. For notational convenience, write the log probability ratios
Tee (Nar (z) TRe (Yw (x)) by log
Te (1n (x) Tere( (xc)
log
Then we can express the probability that the model places on each response as
=0(vwl<)) = TRe((Hul To(u (z)) = (Rer(y)(z))""
Suppose for (æ, yoe ). LDPO (2,, Ywrs yC næ, TR)e) S 0.6. Expanding the DPO loss, we have CDPO(E, You yt ne, TRee) log (3(az bb)) S 0.6. Rearranging the inequality and exponentiate both sides, have
a(3(az - bz)) 2 exp(-0.6)
Note that (00) = 0.5 < implies that 0 < 3(a.. by) and as bz.
I o(((a - bz)). Since the logistic function monotonic, this
Writing out this last inequality,
2 To(Ui(2)) (2:) (Be((Vwlz)) T
no (o( (x:)
(Ber(yull)))) 2 1, To ( TRai (x)
where the last inequality is by the assumption on TRee- We conclude that R(I, Bw. v)) by
definition.
A.2 Proof of Theorem 3.1
Theorem A.3 (Simulating Perfect RLHF). Fix a reference model TRę and an aggregated preference datapoint (x, Ves 81) ~ D. Assume the dataset includes the ground-truth human preferences: that is, a(x, Vwo M1 Py
31) and that these preferences obey the Bradley-Terry model (Assumption A. Let x² be the (adequately expressive) model that perfectly minimizes the DPO objective on (x. Bw: W1). or perfectly maximizes the PPO objective on the optimal reward function as described in Section 2.1. Then, the optimal policy x² satisfies
π (yw x)
Tyo x))
cx(2,, Mwr M1
1/8
x ( x)) TR x))
(1,,, Yar Mn)
where B is hyperparameter in the DPO and RLHI objectives.
Proof. We first prove the statement for DPO. Following the notation from [41], fix a reward function let TH (y(æ) be the optimal model under the KL-constrained RL objective given the reward function r. We can express rxx,) y/ terms of x, xy(() )nnd Tre (y]a:):
TH (g(zx)
r(x, v) = 33 log
Tres (y|z) + 3 log 7(x),
17

---

where (xx) is the partition function for prompt æ. Then, under the Bradley-Tene model, the probability of preferring yv over yr under the model n,
1 TT. (yo 11(=) + exp(r(1 m) r(,,
+ exp(3 (og
T- (xcl)
(u. [x] Blog
Jiey (Mar 2)
Given the ground-truth -ruth human preferences, DPO's maximum likelihood objective minimizes the binary classification loss on (x, yws y1):
min (z, Mhes M1) log π (yw y ()) + (1 - axx, ym m1) log(1 Tyw > n [z))).
Let πt denote an optimal policy from the loss above, the optimal preference probabilities satisfy (yer > w12=) - axx, yues 31);
and the optimal policy in turn satisfies
cxt Mers w²)
+ exp(Blog
x (' (x) mref (xul))
$ log ='(ywla) Tree Tre (thll))
Rearranging, we have
(xx, Mt, Mo.)
r² (31lz) B log Tree (w (x)
x² (H (z)
exp Blog Thees (21(z)
a(r, Nw. 31)
taking a log on both sides and divide by B.
B log axx, Me Vhw)
x (2) oT (Y (z) alx, B1) Tres (3i ( (Yu (z)
log
and finally exponentiating both sides,
(x, M/s yww) n (yhol [x) Tres (2 (x) axx Dwr 31)
Trer (x)
1/8
result follows by taking an inverse.
Now we show the result for RLHF, starting from the optimal solution of the KL-constrained reward maximization problem under the reward ræ, as derived in [41]:
(v/l)) & (vlr() exp exp (3ro(e.v))
The condition is straightforward to derive
(Rw (x) exp (z yw))
(yon [r) x² *("y xr)
(7)
TRee(yn (x) exp (3r(u0 101
TRe((ywll (x:
(exp(ro (I,, yv (x, v1)) 2/B
(8)
TRer a))
TRer(yar (x:)
(x, Mwe U1)
1/8
(9)
TRe((1) xr)
axx Meoy v1)
where the last equality is due to the Bradley-Terry model.
L
A.3 Proof of Corollary 3.3
Corollary A.4 (Reproduces Corollary 3.3). Given a reference model TRef. the DPO or RLHF hyyeerparameter 3, dataset of aggregated preferences D = {(x, yer 21) and their corresponding
18

---

rater proportions (z, Vhe U1), the ranking accuracy of the optimun of the RLHF or DPO objective nt is given by
TRe xx) Rs (D; (TR) (4:
ax(2; Mius 31)
1/3
E
1
>1
(10)
TRe(3l x)) (x,, Mwr 31)
where 1[] the indicator function. When computed on length-normalized likelihoods from TRey we denote the idealized ranking accuracy as R
from the definition of ranking accuracy (Definition 2.3) that the accuracy datapoint (z. Yur, 31) is when (1eo x) > T x). In other words, Tfy /t) x)) > 1 ensures that the ranking accuracy (ar, Mor 1/t) Theorem 3.1 shows formula for the ratio of the optimal policies, and plugging this in to the condition for ranking accuracy immediately yields the given formula for R
Proof. We
A.4 Proof of Theorem 4.1
Theorem A.5. Consider an aggregated preference datapoint (xr, Mwr vi) such that the reference model log-ratio constant c, i.e.
log TRe((((1))
(Re(Hul2))
Then, R(x, yoo 11) if and only if CDPO yues 11) S -log (Bc), where a the sigmoid function.
Proof. Recall that we can break down the DPO loss into the model log-ratio and the reference model log-ratio as follows:
To (B (x)
TRar( (x) TRee (Yar (x))
LDPO Uw. M1; no TRee) log
log
To( xx) + log
+ xe) To (Ve (x:)
- log
s
log
(by assumption)
(T(()
Observe that if CD,O( Yor me, )e)e)
< log o (Bc), then
Te (yw xz) 2 e
log
( [r)
by the monotonicity of the log and sigmoid functions.
This implies that
so (Nul
2 and
R(z, Yoo y1) 1.
Now we show the other direction. Suppose R(x, yors 31) 1, then To Te( (x) 2 me (s1(z) and log (ow (z) T (x)
c 2 c.
Using this relationship, we have
log 3 (l0g
log To(vrolz) + ( log (3c),
To Te(ui (x))
and the other direction follows by taking the negative of both sides.
A.5 Extending Theorem 4.1 to IPO
We extend our main result on DPO also describe the IPO objective, formally defined below. Definition A.6 (Identity Preference Optimization, a.k.a. IPO [2]). The Identity Preference Optimiza- tion (IPO) loss defined as LIPo (I, Uw m1) (log
3e (e lmn) Taur (Mw
log Te(uplz)
Taa (x)
27
19

---

Proposition A.7. Under the IPO loss. if
log TRe yow xx)
1
TRe (V((2) (:)
2r
then zero IPO loss implies that Riz, Nwn 31)
On the other hand, if
1
log TRee(u
2T
then LI, Yuu 11) S (c + 2F guarantees that R(x, Vee y1) 1.
Different from the DPO loss, the IPO loss is regression loss where the optimal model log-ratio has constant margin over the reference model log-ratio. This can be seen by the following decomposition of the IPO loss:
Lao Ves y1) -(1gg
log
To (V((l) - log TRer (w)))))))))))))
Toe (b)lz)
TRe((1((2)) 2T
Clearly, the optimization target of the model log-ratio is the sum of the reference model log-ratio and the margin 27
From Figure 3, we can see that the reference model can have large bias towards the dispreferred completion, represented by the large negative values of the reference model log-ratio (note that in the figure, the reference model log-ratio has the dispreferred completion in the numerator). Therefore. I for the optimal model under the IPO loss to have perfect ranking accuracy, the margin 27 needs to be large enough to overcome this bias for all datapoints. Alternatively, per-example margin dependent on the reference model log-ratio of the example can be used.
A.6 Generalization of the Ranking Accuracy for Preference Datasets with n > 2 Outputs
Some datasets (e.g. UltraFeedback [7]) contain examples with more than two responses per prompt x. In these cases, we extend our definition of ranking accuracy (Def. 2.3) to exact- mathh of the rankings over all choices. Suppose each aggregated datapoint consists of a prompt and responses (y1,-.. 3n Since the question of how to best aggregate rankings over multiple raters (when n >> 2) assume that there already exists some aggregated social ranking p(r, V1, ... yn) = (p, '1, ...
an open research question,
yn )); E [[[n] where P: y1, ... yn. < Py (I. y1, ... yn)
implies that ys is preferred over y₁- Now let the ranking assigned by policy wo be (xx, U1, 3nn)
(vi(1, Vı, ... 9n))) E [n.. where To(y; (z) > (1, if and only v4 (x, U1. ... yn.) <
Py .1. V..). Then the generalized ranking accuracy defined as follows:
Definition A.8 (Ranking Accuracy for n 2).
Rn>2(I, V1, Yni σT) (e v1. vb,) = v(z, (1. Vn)] where 1[-] is the indicator function. Analogously, the ranking accuracy over a dataset {(x, Vi, .B)]] is
(11)
E (1.991. -b, DD
Rn,2(1, "" Mp)
(12)
B Experimental Details for Computing Ranking Accuracy of Open-Access LLMs
B.1 Implementation of Ranking Accuracy
We evaluate ranking accuracy for a wide range of L.LMs by evaluating the likelihoods that each model no assigns to Yar given x and y given as described in Def. 2.3. However, Ywr and y are sequences, so we compute the sequence likelihoods by factorizing the sequence likelihood into a product of the conditional token likelihoods, like so:
To (y (x)
II no To (x; yet)
(13)
We use PyTorch and the Hugging Face rrans formforree formers and datasets libraries to compute all ranking accuracies. For each dataset that we evaluate ranking accuracy on, we take random sample of 1000 examples.
20

---

Handling Ties In some datasets, such as the cross-human- human-annotated validation split of the Alpaca Farm dataset [9] (described in Sec. B.2), ties exist in the human annotations. When this is the case, we make mild adjustment in the calculation of ranking accuracy to accommodate ties.
Definition B.1 (Ranking Accuracy with Ties).
1 (v)())) no (U2 <
if ax Un U2) = 0.5
Ries (Z, V1, U25 To)
1 (To()) no (V2[2)]
if a( y1 U2) > 0.5
(14)
(30 no (V2[2)]
if a(x 31.82) < 0.5
where [-] is the indicator function. In other words, if the human annotations indicate tied preferences between y/1 and y2, then No achieves the correct ranking if and only if it assigns 1/1 and //2 approxi- mately the same likelihood (within some tolerance level c). For all other cases, the RTies is equivalent to R.
Throughout this paper, we use a tolerance of 0.01. In Fig. we provide ranking accuracies only for examples without ties, but we provide the numbers the full datasets below.
Length Normalization To compute the length-normalized ranking accuracy (R), replace To(y)z) in Defs. 2.3 and A.8 with (o(y)z), where
1/Ma
lvv
(u(x> = II
(v x; vee)
(15)
B.2 Datasets
HH-RLHF [3] (helpful set) consists of two model responses for each query (often the history of multi-turn conversation between human and chatbot), based on generations from three different classes of models (context-distilled 52B model, same model with rejection sampling using a reward model, RLHF-finetuned models). Queries and preferences annotations are obtained from crowdworkers.
Synthetic Instruct GPT-J Pairwise [1] a synthetic dataset of queries and pairwise model generations spanning different subjects.
StackExchange Preferences [28] consists of questions and answers from Stack Exchange, where within a pair of answers, the preference between two answers is determined by function of the number of upvotes and whether the answer was selected.
UltraFeedback [7] consists of model-generated responses from four different language models out of larger set of models (meaning different set of models is considered for different samples). Queries are obtained by a mixture of existing QA datasets, and the preference annotated by GPT-4.
Stanford Human Preferences [11] is dataset created from Reddit posts across different subject matters, where within pair a response is considered preferred another was created later and has more upvotes.
Alpaca Farm Validation [9] is sourced from the Alpaca Eval dataset, but with splits repurposed for training preference-tuned models. Additionally, choose to use the validation split because contains human cross-ane gross-annotations (i.e. multiple human ratings per triple of (I, (r))1.8))) This particular split can be found at https: / hww.gggggggggeeeeeeee/ee co att.g/aaaacccoataattttteaeaeeeee tttp//ttttttttaaaaaaaaaaaaaaaaaa ons json. The original dataset, Alpaca Eval [30], is a mixture of several test sets including Open Assistant [26], a dataset of human-constructed chatbot conversation turns, HH-RL-F [3], and the Vicuna [70] and Koala test sets [15], where the former consists of user-shared queries from ShareGPT, and the latter consists of human queries from online interactions.
hwt___hmman_ssssoe bl ol ollb/mmai /ob/mmint mamin/a pacca ca fa hum___uann___ caraaa_crossae www.erroosssanaaaeeeeeeee
B.3 Full Results
We give the full set of length-normalized and non-Lengthhnormaaizee normalized ranking accuracies for open- access LLMs across six datasets in Tables 2, and 4. For the UltraFeedback [7] and StackExchange
21

---

Preferences [28] datasets, we use the generalized definition of ranking accuracy for n > 2 outputs instead (Def. A.8). We also provide ranking accuracies computed on the Alpaca Farm validation dataset with ties included in Tables 5 and 6.
Table 2:
and non-length-normallee ranking accuracies for the Anthropic Helpful and Harmless (HH-RLHF; Bai et al. [3]) and Synthetic Instruct GPT-J Pairwise [1] datasets. The latter contains only training split.
Anthropic HH-RLHF
Synthetic
Model
Instruct GPT-J
Pairwise
Test
Train
Train
R
R
R
R
DEMMA-7B-IT
52.5%
46.7%
52.6%
47.0%
56.6%
76.5%
GEMMA-77
51.9%
46.5%
53.7%
46.6%
93.1%
76.9%
GPT2
49.9%
46.0%
52.7%
46.8%
81.5%
65,9%
LLAMA-2-7B-CHAN
52.0%
46.2%
54.1%
46.3%
94.5%
75.4%
LLAMA-2-7B-HF
52.1%
46.3%
53.8%
46.3%
93.6%
77.9%
MIS.RAL--B-VO.I
52.9%
45.8%
54.4%
46.4%
93.5%
77.4%
OLMo-7B
51.0%
45.5%
53.6%
46.6%
91.9%
78.2%
PYTHIA-1.4B
51.2%
46.2%
53.2%
46.3%
88.0%
68.2%
PYTHIA-2.8B
51.0%
46.4%
53.5%
46.6%
90.3%
69.5%
TULU-2-7B
51.8%
46.7%
54.6%
46.3%
94.3%
77.8%
TULU-2-DPO-77
51.4%
46.1%
54.0%
46.2%
94.4%
77.7%
SCCUNA-B-VII,
52.2%
46.4%
54.7%
46.0%
93.6%
77.4%
CYPHYR-7B-DPO
50.9%
46.5%
55.4%
46.5%
95.1%
79.6%
ZEPHYR-7B-SFT
49.5%
46.2%
55.6%
46.2%
94.3%
78.9%
Table 3: Length-normalized and non-length-normaaizee ranking accuracies for the StackExchange Preferences [28] and UltraFeedback [7] datasets. Both datasets contain only training split.
StackExchange
FltrbFeedback
Model
Preferences
Train
Train
R
R
R
GEMMA-7B-IT
32.6%
12.8%
2.7%
1.4%
GEMMA-7B
31.2%
22.4%
2.6%
1.5%
GPT2
29.5%
21.0%
2.1%
1.4%
LLAMA-2-7B-CHAT-HH
32.1% %
22.1%
2.5%
1.2%
LLAMA-2-7B-HE
33.0%
22.6%
2.7%
1.7%
MISTRAL-7B-VO.1
31.8%
22.5%
2.4%
1.7%
OLMo-7B
31.9%
22.3%
2.6%
1.4%
PYTHIA-1.4B
28.3%
12.7%
2.1%
1.4%
PYTHIA-2.8B
31.9%
21.7%
2.4%
1.4%
TULU-2-7B
32.8%
22.4%
2.9%
1.3%
TULU-2-DPO-77
32.5%
22.1%
2.7%
1.6%
VICUNA-7B-V1.5
33.2%
22.2%
2.5%
1.4%
ZEPHYR-7B-DPO
33.0%
22.3%
3.5%
1.8%
ZEPHYR-7B-SFT
32.2%
22.3%
2.7%
1.5%
B.4 Computation of the Idealized Ranking Accuracy
When computing the idealized ranking accuracy Theorem 3.1 common preference datasets, we note that there are few approximations required. First, most datasets do not report the proportion
22

---

Table Length-normalln normalized and non-Length-normmalzzdd Preferences (SHP; Ethayarajh et al. [11]) and Alpaca Farm choose specifically the validation split since validated with multiple human annotations per triple of (ar, yor. ). Examples with ties are not included.
normalized ranking accuracies for the Stanford Human
validation [9] datasets. For the latter, we
Stanford Human Preferences
Alpaca Farm
Model
Test
Train
Validation
R
R
R
R
GEMMA-7B-IT
44.1%
24. 24.2%
60.3%
35.9%
55.6%
39.1%
GEMMA-7B
43.0%
24.2%
57.7%
35.5%
53.6%
40.4%
GPT2
40.6%
23.9%
56.9%
35.2%
50.2%
40.2%
LLAMA-2-TB-CHAT-HI
43.9%
23.1%
60.0%
35.4%
53.4%
40.2%
LLAMA-2-7B-HH
44.9%
23.8%
58.1%
35.1%
53.0%
42.1%
MISTRAL--BB-O..1
44.3%
23.9%
57.7%
35.3%
53.2%
40.6%
OLMo-7B
44.3%
24.8%
56.3%
35.6%
52.7%
41.2%
PYTHIA-
42.9%
23.8%
57.4%
35.5%
49.8%
40.4%
PYTHIA-2.88
43.9%
23.5%
57.6%
35.6%
50.2%
40.2%
TULU-2-7B
44.4%
23.4%
59.3%
35.3%
53.2%
41.9%
TULU-2-DPO-77
45.3%
23.4%
59.3%
35.3%
53.4%
42.1%
VICUNA-7B-VIL
42.2%
23.4%
59.3%
35.1%
54.5%
42.1%
ZEPHYR-7B-DPO
42.5%
23.4%
56.7%
35.2%
54.3%
42.1%
ZEPHYR-7B-SS
43.9%
23.5%
57.6%
35.3%
54.5%
41.5%
Table 5: Length-nowe norgalieed and normallzzdd ranking accuracies for the Alpaca Farm validation [9] dataset (see App. B.2), but including examples with ties (unlike Table 4).
Alpaca Farm
Model
Validation
R
R
GEMMA-7B-IT
15.8 15.8%
34.1%
GEMMA-7B
40.6%
35.2%
GPT2
32.2%
34.8%
LLAMA-2-7B-CHAN. 7B-CHAT-HF
41.5%
34.8%
LLAMA-2-7B-HI
41.5%
36.4%
MISTRAL-7B-VO.I
42.8%
35.0%
OLMo-7B
38.9%
35.4%
PYTHIA--.4P
37.2%
34.8%
PYTHIA-2.8B
38.6%
34.6%
TULU-227B
40.6%
36.2%
TULU--2-DO-770
40.3%
36.4%
VICUNA-7B-V1.5
41.5%
36.4%
ZEPHYR-7B-DRR
42.3%
36.4%
ZEPHYR-7B-SFT
43.9%
35.9%
axx, yoe y1) of raters who preferred Ye over yı and using a(x, Yar v1) results in errors. As a result, we measure the alignment gap on the Alpaca Farm validation split [9], which contains individual votes for each triple. In the event that all four raters unanimously preferred one of the responses (i.e. a(x,, Vt, yar) = 0), we add a small constant = 0.001 to a(x, 31, yw)) to prevent divison by zero. Second, the formula depends on the choice of 3, which we do not know for many closed proprietary models. We circumvent this issue by computing the quantity for range of values (3 {0.01, 0.1, 1, 10)) and reporting the minimum, median, and maximum.
23

---

Table 6: We provide both the nortth-normllized (R) and
(R) ranking accura-
cies for variety of open-aceess preference-tunee models on the Alpaca Farm [9] validation dataset
(described in App. B.2). We also provide the idealized ranking accuracy (Corollary 3.3). Unlike
Table we include examples with ties in this table.
Length- Sangth-Normalized
Non-LengthhNorraliian
Cefference-Tuneer Model
R
R
R
R
(Min./Med//Max/
OMin//Med/MMan
ZEPHYR-7B-DPO
42%
86% 98% 100%
36%
90% / 199 100%
TULU-2-DPO-7B
40%
87% / 97% / 100%
36%
91% / 99%
100%
GOOGLE--GEMMA-7BIIT
41%
73% / 73% 97%
35%
67% / 93%
100%
LLAMA-2-7B-CHAT-HIF
42%
87% 97% / 100%
35%
91% / 99% 100%
C Dynamics of DPO Training
C.1 Training Details
For our results in Section we trained three different scales of models (GPT2 [40], Pythia 2.8B [4]. and Llama 2 7B [52]) across three seeds each on the HH-RLHF dataset [3]. We split the test dataset in half, using half for validation during hyperparameter tuning. We ran separate hyperparameter search for each class of model and for each stage of training (i.e. SFT versus DPO). The hyperparameter ranges we searched were:
GPT2
SFT: learning rate E (5e-7, 1e-6, 5e-6, 1e-3), batch size E (64, 128, 256, 512]
- DDPO: learning rate E {5e-7, 1e-6, 5e-6, le-5), batch size E {32, 64, 128), 8 E {0.01, 0.1, 1.0, 10.0}
Pythia 2.8B
SFT: learning rate E {1e-7, le-6, le-5), batch size {16, 32, 64]
DPO: learning rate E {5e-7, le-6, 5e-6, le-5}, batch size E (32, 64). 8 E (0.01, 0.1,
1.0, 10.0)
Llama 2 7B
- -FFT: learning rate E {1e-7, 1e-6, le-5}, batch size E {32, 64]
DPOO learning rate E (1e-6, 1e-7], batch size E (32, 64), , E (0.1)
-
We tuned the hyperparameters seeds the same model class. We trained the GPT2 and Pythia2.8B models for 5 epochs each, and the Ilama2 7B model for epoch only (due computational constraints) for both SFT and DPO. However, most seeds of the GPT2 and Pythia 2.8B models reached the lowest validation loss at the end of the first epoch. For analyses where analyze only checkpoint (rather than the evolution over the course of training), we always analyze the checkpoint with lowest validation loss. We use the AdamW optimizer (with 31 0.9, 32 = 0.999, =le-8) for SFT and the RMSProp optimizer (with a = 0.99, weight decay- 0, momentum = 0, =le-8) for DPO.
a single seed, and carried over the best hyperparameters to the other
The GPT2 models were trained single Nvidia A100 GPU each, and the Pythia 2.8B and Llama 2 7B models were trained on two Nvidia A 100 GPUs each. We used PyTorch Fully Sharded Data Parallel (FSDP) in fully sharded mode to train the Llama 2 7B models. In total, SFT required approximately 9, 52, and 30 GPU-hours per seed of GPT2, Pythia 2.8B, and Ilama 7B, respectively. DPO required approximately 8, 48, and 49 GPU-hours per seed of GPT2, Pythia 2.8B, and Llama 7B, respectively. (Longer training times were required for Pythia 2.8B than for Llama 2 7B since we trained the former for only epochs and the latter for 1, aforementioned.)
C.2 Results
We provide the DPO loss, reward margin, and dataset trends during training across models (three seeds each of GPT2 [40], Pythia 2.8B [4], and Llama 8B [52]) in Figs. 5, 6, and 7.
24

---

1.2
1,0
Loss
Category
0.8
correct-corres
correct--nncomed
incorredt- corred incoored-iinccored
10000
Weeee ausarrroooe au Rie
seasee
5000
10000
5000
10000
5000
10000
Step
Step
(a) Three different seeds of GPT-2 [40].
1.50
1.25
DPO Loss
1.00
Category Data
0.75
correct-
correct-ainoerred
0.50
incoredt-.coom
0.25
incorret--incomee
Ein Biraoee yoa
0.00
5000
10000
5000
10000
Step
Step
(b) Three different seeds Pythia 2.8B [4].
DPO Lozs
3.5
Data correct-iinoereet
ccorect-ccoreet
incorrect..comm incorredt-iincoree
0.5
500
1000
1500
2000
2500
500
1000
1500
2000
2500
500
1000
15000
2000
2500
Step
Step
Step
(c) Three different seeds of Llama 2-7B [52].
Figure 5: Average DPO loss over the course of training, for four categories of the training data (An- thropic HH-RLHF; Bai al. [3]). The category "correct->incorrett" indicates examples (z,, Vies 31) for which (B (x) >> (Br (r) but Te, (Vw) < Te, (3ul (z) (where is the trained policy at training step t), and on. Lines that end early indicate that the category no longer contains any data points. The dashed vertical line indicates the step at which the lowest validation loss was achieved.
D Ranking Accuracy and Win Rate
We include additional plots from our exploration of the relationship of ranking accuracy and win rate in Figs. 9 and 10.
D.1 Results on the Test Set
We also provide results computed on the test set in Figs. 11 and 12.
E Qualitative Analysis
Theorem 4.1 demonstrates that is difficult for DPO to learn the correct ranking of points that are not ranked correctly by the reference model. In particular, datapoints that induce a large positive reference model log-ratio (see Definition 2.2) require the DPO loss to be minimized to a very small value in order to flip the ranking (Theorem 4.1 and Figure 3).
Here, we document a few of the datapoints that induce large positive reference model log-ratio and are hard to learn (Table 7), as well as datapoints that induce a very negative reference model log-ratio and are easy to learn (Table 8). These datapoints are measured using the Pythia-2.8B model and are taken from the training split of the Anthropic HH-RLHF [3] dataset. We note that datapoints
25

---

Reward Margin
Category Data
correct--orrest
correct--nncomeed
incorredt corred iinoored-innooered
5000
10000
5000
10000
5000
20000
Step
Step
(a) Three different seeds of GPT-2 [40].
Neward Margin
Category of Data
comeett
correct-innooeed
Incorred: coored
2500
5000
7500
10000 12500
2500
5000
7500
10000 12500
0 2500 5000 7500 10000 12500
Step
Stop
Stop
(b) Three different seeds of Pythia 2.8B [4].
Reward Margin
2
Category of Data
correct--incomed
Incorred: incorred--innoored
500
1000
1500 Step
2000
2500
500
1000
1500
2000
2500
500
1000
1500
2000
2500
Step
Step
(c) Three different seeds of Illmaa 2-7B [52].
Figure 6: Average DPO reward margin over the course training, for four categories of the training data (Anthropic HH-RLHF; Bai et al. [3]). The category
indicates examples (x, Yos y1) for which (yr (x) > TRer(yn (z:) but me, (yw.lx) < To, (y (x) (where Te, the trained policy at training step !), and so on. Lines that end early indicate that the category no longer contains any data points. The dashed vertical line indicates the step at which the lowest validation loss was achieved.
with very negative reference model log-ratio are already ranked correctly at the start of DPO. We also document the datapoints that are easy flip: the reference model log-ratio is slightly positive, so the reference model is slightly incorrect, and optimizing the DPO objective could feasibly result in the model learning to rank these points correctly. We observe that the hard-to-leant datapoints are substantially longer than the easy ones, and that the easy datapoints generally contain chosen responses that unanbiguously better than the rejected ones.
26

---

0.5
Dutasee
0.3
Category
correct-coomect
correct--incarees
incoried->coreett
0.1
incoored--inncoree
8
5000
10000
5000
10000
5000
20000
Step
Step
(a) Three different seeds of GPT-2 [40].
000
0.5
Detasee
0.4
Category of Data
0.3
correcti-comeet
correct--incorret
0.2
0.1
incornet-vincoomed
5000
10000
0
5000
10000
5000
10000
Step
Step
Step
(b) Three different seeds of Pythia 2.8B [4].
0.5
0.5
Dataset
Category Data
correct-ccomeett
correct iincorect
incorred-
incoried--incoomed
0.0
500
1000
1500
2000
2500
500
1000
1500
2000
2500
500
1000
1500
2000
2500
Step
Step
Step
(c) Three different seeds of Llama 2-7B [52].
Figure 7: Percent of the dataset that each category data constitutes over the course of training, for four categories of the training data (Anthropic HH-RLHF; Bai et al. [3]). The category "correct- >incorrect" indicates examples (x. Yw 3) for which TRe(yyalar (x) TRa((i (z) but no, (y (x) < To, (Br) .)) (where ne, is the trained policy at training step f), and so on. Lines that end early indicate that the category no longer contains any data points. The dashed vertical line indicates the step at which the lowest validation loss was achieved.
27

---

GPT2
Pythia 2.8B
Llama
2.0
Ranking Acc.
X. Ywu Yii mluur)
Ranking Acc.
Ranking Acc. Correct Incorrect
Correct
Correct
Eoro You Na"
Incorrect
Incorrect
1.0
0.0
-500
0
500
-500
0
500
-500
250
250
500
log mudly
log
(a) Results for second seed.
GPT2
Pythia 2.88
Liama 2
Cgoo0. Yo Raee)
Ranking
Ex You Viito To Rine)
Ranking
CoRo Yuu Vi no Riee
Ranking Correct
2.0
Correct Incorrect
Correct
3.5
0.5
0.0
500
500
500
500
500
250
250
500
(b) Results for second seed.
Figure 8: The log-ratio of the Re&t likelihoods versus DPO loss and ranking accuracy subsample of 1K training examples from the HH-RLHF dataset [3]. The results from the first seed are given in Fig. 3, and the results for the other two seeds given here.
0.61
Ranking Accuracy
0.60
0.66
Rate vs. feter
0.6
DPD Loss
0.5
Train
0.58
Test
0.64
0.4
-0.63
0.3
0.56
500
1000
1500
2000
2500
500
1000
1500
2000
2500
Training Steps
Training Steps
(a) Ranking accuracy and win rate, computed prompts from the training dataset.
(b) Training and test DPO loss during train- ing. up to the point of lowest loss.
Figure 9: Ranking accuracy and win rate (versus TRR)) are not monotonically related throughout training. We measure the loss, ranking accuracy, and win rate from the start of training to the checkpoint of lowest validation loss. Even though both training and test loss continue to decline during DPO training, ranking accuracy and win rate only trend together early on in training. Past certain point, the two become anti-correlated.
28

---

4,5
Ranking Accuracy
0.9
Rate VS. TTter
4.0
0.8
0.75
3.5
0.70
0.7
3.0
00,55
0.6
2.5
-0.60
1.0
1.5
0.0
0.5
1.0
1.5
2.0
y
(a) Ranking accuracy and win rate versus 7
(b) versus distance travelled by model param- during DPO training.
When the influence of Red is strong, win rate and ranking accuracy trend together. higher y value implies greater influence of गeत during training. For larger y values ( 2 1.25), ranking accuracy and win rate trend in the same direction.
Figure 10:
0.565
-072
Ranking Accuracy
Winn Rate VS. Mif
Ranking Accuracy
0.360
VS. flter
-,700
0.555
D.555
0.550
0.550
0.68
0.545
D.545
0.66
0.540
500
1000
1500
2000
2500
Training Steps
|0: Ond
(a) Ranking accuracy and win rate versus training steps.
(b) Ranking accuracy and win rate versus distance that the model weights have moved.
Figure Ranking accuracy and win rate of various Pythia 2.8B checkpoints during training, calculated on the test dataset.
Ranking Accuracy
0.60
-0,8
Win Rate vs. Neat
Ranking Accuracy
0.60
0088
0.58
Win Rate Vs.
0.7
0.56
0.56
0.54
0.6
0.66
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
y
[10,
Gnwrll
(a) Ranking accuracy and win rate versus training steps.
(b) Ranking accuracy and win rate versus distance that the model weights have moved.
Figure 12: Ranking accuracy and win rate various models trained with L calculated on the test dataset.
29

---

Table 7: Preference datapoints that are hard learn. We print several datapoints from the training split of Anthropic HH-RLHF [3] that are hard for the Pythia-2.88 model to learn. We characterize a datapoint hard to learn if the reference model log- rat-ratio (RMLR, see Definition 2.2) is large. This characterization follows Theorem 4.1 and Figure 3. The reported loss is measured on the checkpoint with the lowest validation loss and averaged over three seeds.
RMLR
LPO
Prompt, Chosen Response, and Rejected Response
457.6
3.05
Prompt: Human: What are the best and most reliable VPN services? Assistant:
could recommend Open which is open-source and through which you can encrypt and protect your online connections. You can find instructions for setting up the service with OpenVPN at https:/open/pp..eeeee wwwpppn..nenee..eeeeeeee http://ppnnnsuuueedcuueeetatttto
Human: That link doesn't 't work. Can you just send me link that would take me the homepage of OpenVPN.
Assistant: You can find the home page for OpenVPN at https://ppnnvpn.neee Human: Thanks I'll check out their website
if they trustworthy. consulting user-ratings sites like www.trustpilot.com
Assistant: Sure thing. You can also find out more about the service by
Human: Okay I'll use trustpilot to see if trustworthy and reliable.
Assistant:
Chosen response: Great. There are a few additional tips you might find useful when comes to evaluating third-party VPNs. For the most part, VPNs are pretty similar for users, although they differ slightly in terms of features and capabilities. Since can't give you any recommendations for specific VPNs. you might want to consider doing a cost/benefit analysis of various services: how many locations they support, how reliable and speedy they are, what types of devices they support, and does the service protect your personal information? recommend employers and groups with more specialized needs choose service such as CyberGhost, which offers military-graan encryption system and is able meet large-scale requirements down to the level of providing multiple cybersecurity experts on-board. However, as a personal consumer, you'l 'Illl probably feel more comfortable by finding cheaper service with more limited features, since you don't 't need as much help with cybersecurity. Rejected response: You't re welcome.
30

---

RMLR
CPO
Prompt, Chosen Response, and Rejected Response
423.4
2.2
Prompt: Human: What are some fun toys my kids can play with in the bath?
Assistant:
Chosen response: There are many fun toys you can play with in the tub with your kids! Here are few favorites:
-Bath crayons and sponges for drawing
-Soft bath sponges for floating
-Bubble blowers
-Bath toys, such bath squirt toys dinosaurs
-Simple bath musical instruments, like rubber ducks or Tommy Tippee cups
-ollored mini cups for blocks and other building games
-Bursting washup sponges
-Shaving foams and shaving creams for water fun
-Bath thermometers
Floating toys like floating rings and animals or mini boats
Pllastic containers and figurines for fantasy play and role-playing games
-Bath fizz tablets and baking soda for bubbles
Clackers, bangles and fizzers
Squirt guns
-Straw toys for blowing air and creating bubbles
-Water wands and spray bottles
-Floating rainbow soap bars
-Comic books and storybooks for reading tub
Sofft toys for cuddling and play
RRubber duckies and other
Rejected response: Well, of the popular bath toys for kids is bath horse. It's a large waterproof animal toy you can keep the bath and then wash off easily after use. These toys often sturdy enough that your child can sit on them while they't 're're in the bath, and they come in a huge variety of shapes and designs. You also search for bath playsets, which often have a variety of other accessories as well.
31

---

RMLR
CDO
Prompt, Chosen Response, and Rejected Response
418.7
0.95
Prompt: Human: Is brainwashing a real thing that people capable of doing? Assistant: Yes, I think brainwashing is definitely real thing that people are capable of doing. This is an example of moral and ethical question that many people have different ideas about, and you might wonder I'm qualified to answer it. So how about first do quick thought experiment?
Human: I'm not asking about the morality, I'm asking it's an actual practical scientifically backed phenomenno//proeedure that people can do. Are there any real world examples of brainwashing?
Assistant:
Chosen response: Brainwashing actually a real and fairly common phe- nomenon, and there are many examples of people using these techniques for coercive and unethical means. For example, in Orwell's famous novel 1984, the Party uses various forms of psychological coercion control its citizens and suppress independent thought or critical thinking. In real world cases, brainwash- ing is often associated with coercive ideological or political indoctrination, often in the form of highly-intense and insular social or cultic environments. Such groups can exert intense psychological pressure on individuals to adopt specific belief patterns or hehavioral norms. These environments are often characterized by prominent charismatic leadership, intensive and prolonged social isolation from the surrounding world, and highly repressive behavioral controls. In these contexts, new members are typically subjected to prolonged and intensive per- suasion campaigns, often involving overt coercion, psychological manipulation and dishonesty, and extreme social pressures to conform to the group. Over time, new members be progressively subjected to increasing restrictions their activities and freedom thought, often involving the suppression or denuncia- tion of prior belief patterns and behavioral norms, and eventually resulting in complete loss of previous identity and independent thought.
Rejected response: Oh, think understand. I'm not sure what you mean by "scientific", but let's talk about the use of the phrase "brainwashing" to mean something that happens to you. We often talk about brainwashing in the context of deprogramming, and that sometimes happens in the context of religious cults. I can tell you more about that.
32

---

Table 8: Preference datapoints that easy to learn. We print several datapoints from the training split of Anthropic HH-RLHF [3] that are easy for the Pythia-2.88 model to learn. We characterize a datapoint easy to learn if the reference model log-ratio (RMLR, Definition 2.2) is small, or very negative. This characterization follows Theorem 4.1 and Figure 3. The reported loss is measured the checkpoint with the lowest validation loss and averaged over three seeds.
RMLR
LDOO
Prompt, Chosen Response, and Rejected Response
675.9
0.16
Prompt: Human: How do you make twice-baked potatoes?
Assistant:
Chosen response:
They'r re potatoes that have been boiled, then baked in the oven, then baked again.
Rejected response: What is twice-baked potato?
-562.1
0.052
Prompt: Human: How do I put fondant cake?
Assistant: You can use butter to soften it, then add cream or milk to bring
the fondant smooth consistency. You can then put the fondant on the cake, before adding icing.
Human: How do apply it?
Assistant:
Chosen response: You can dab very small amount on the icing in the area where you want to have the fondant, then smooth it out with the back of the spoon. the fondant too dry, you can add more cream or milk to make it a bit softer, but make sure you don't 't get any clumps of fondant. Rejected: Once you have a smooth consistency, you can spread it out with a palette knife, using spatula to remove any excess.
Rejected response: Once you have a smooth consistency, you can spread it out with palette knife, using spatula to remove any excess.
-522.5
0.003
Prompt: Human: Can you find me Margot Robbie movies?
Assistant: Oh sure, let me check my database! Hmmm, it looks like you're "fan" of the Canadian actress Margot Robbie. Why don't you tell me more about her?
Human: She cool. Can you find me some movies?
Assistant:
Oh, cool! I'm not sure if there are any movies with her in it, but I do know there's TV show about her!
Human: would like movies.
Assistant:
Aaaahhh...I see, you want to movies with her in them. Why don't you tell me more about her movie roles?
Chosen response:
Rejected response: have it good authority that there Margot Robbie movies, but the keyword you typed really pretty broad and it's not super clear what you mean.
33

---

Table 9: Preference datapoints that easy to flip. We print several datapoints from the training
split of Anthropic HH-RLHF [3] that are easy for the Pythia-2.8P model to flip. We characterize a datapoint easy to flip if the reference model log-rat rog-ratio (RMLR, see Definition 2.2) is small positive value. This characterization follows Theorem 4. and Figure 3. The reported loss is measured on the checkpoint with the lowest validation loss and averaged over three seeds.
RMLR
LPO
Prompt, Chosen Response, and Rejected Response
0.26
0.86
Prompt: Human: Whats the easiest way to throw piano out of tune?
Assistant: Just singing into the microphone!
Human: What messed up the strings inside the piano?
Assistant: Just let me know if you want to start putting heavier things inside the piano break the legs off.
Human: want to put heavy things inside of it, have any suggestions?
Assistant:
Chosen response: Just call a dumpster fire piano, or let me know if you want take apart very heavy piano.
Rejected response: Depends what you mean by heavy. I can make some suggestions though. We don't 't really want to throw any large things the piano because you could cause damage to yourself or the piano.
0.28
0.19
Prompt: Human: Is Austria good place to take a vacation?
Assistant:
Chosen response: Possibly. Can I ask some questions? It's best to think
about any questions before you give me an answer.
Rejected response: Is Austria good place to take vacation?
0.45
0.16
Prompt: Human: what are good skills receptionist needs?
Assistant: I recommend organization and attention to detail. Since they 're also often the first point of contact for customers, they need be courteous and helpful. And you also want them to accurately enter new customer information, so they need be careful and diligent. would
Human: what would you say is the number one most important skill?
Assistant:
Chosen response: I would say it's really critical that they be organized and detail-oriented. This is the backbone of their job, and if they're 're not doing it well, it's going to make their entire team look bad. The customer service aspect is also super important, and helps if they's 'r friendly. But they 'r not organized, it's hard for them to provide good service.
Rejected response: would say it depends on the type of work they re doing, but organization and attention to detail are key for any receptionist. They need make sure everything is orderly and correct, and also that they' re very attentive customers. They need make sure everyone who calls in gets exactly what they need.
34

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Each claim in the abstract and introduction maps directly to proposition, theorem, corollary, and/or figure containing empirical data. The formal assumptions required for the results are summarized in natural language in the abstract and introduction. The results contain additional commentary about how the analysis can be extended other related algorithms and how measurements can be taken for new datasets.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Assumptions required for the theory are stated formally for each result. The discussion section lists limitations of the experimental and conceptual scopes of the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
35

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Each theorem is self-contained in stating the required assumptions needed. Proof' are provided in detail in Appendix A. We omit proof sketches due to space constraints and because the formal results are mostly straightforward algebraic manipulations.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We detail datasets, models, and precise algorithms to take measurements in the appendix. Our training runs can be reproduced with minimal modification to the HuggingFace transformer reinforcement learning (TRL) library". We additionally provide anonymized code.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
co//oo///./ooooooooooooooo.oo.. trl /enl///eniai index
36

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide an anonymized version of our code. Our results be repro- duced with minimal modification to the HuggingFace transformer reinforcement learning (TRL) library, and the datasets used are all open-access and easily downloadable from the HuggingFace Hub.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (https: pppps///nisss cc/
pub public/a guidess CCddeSubmi assioPoolccy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips cc/ publici i c cc/publl
des CodeSubms ss i onPol (emPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Appendices B and C detail all of the information about the data splits, hyper- parameter grids, optimizer, and so on.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
37

---

We report measurements on numerous models and datasets, and our training runs are conducted with multiple seeds.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix C.1 contains detailing of the computational resources needed for our results. We also ran several initial experiments with IPO (Appendix A.5) but did use the results in the main paper.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: https://///uiip..teee http://ps.....o.llaac///cccccoeo
Answer: [Yes]
Justification: Confirmed.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
38

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our discussion in Section discusses the impact of our findings, especially as it relates measuring and verifying the success of preference learning and alignment of LLMs.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Not needed.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite all open-access models and datasets used in the paper.
Guidelines:
39

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papersw wi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We did not release any new assets.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work did not involve crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing research with human subjects.
40

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
41


Reviews:
[{"review_id": "S4UcH2ml8T", "reviewer": "Reviewer_yhyh", "summary": "The paper empirically highlights a few potential flaws in RLHF and DPO that prevent preference-tuned models from achieving high ranking accuracy. It presents a collection of empirical and theoretical findings:\n+ Existing preference-tuned models achieve low ranking accuracies.\n+ The idealized policy and the DPO/RLHF policy has a significant gap in their ranking accuracy.\n+ Preference learning algorithms such as DPO rarely flip incorrect rankings throughout the training process.\n+ Ranking accuracy and win rate are positively correlated when models are close to the reference model.", "questions": "1. One concern is that the ranking accuracy considered here is the hard accuracy (accuracy of the Bayesian optimal classifier). I wonder if the \"soft\" ranking accuracy will exhibit more noticable improvement after preference fine-tuning. The soft ranking accuracy is $\\frac{\\pi(y_w)}{\\pi(y_w) + \\pi(y_l)}$ instead of hard thresholding the predicted accuracy.\n\n2. It is not clear to me what is the main failing part in DPO/RLHF training for high ranking accuracy. In Theorem 3.1, it seems that the perfect RLHF policy should encode the information from $\\pi_{ref}$ and the dataset perfectly at the same time. Does the DPO/RLHF model fail to capture $\\pi_{ref}$ or the preference dataset?\nHave the authors tried to plot the quantity $\\frac{\\pi_{DPO}(y_w)\\pi_{ref}(y_l)}{\\pi_{DPO}(y_w)\\pi_{ref}(y_l)}$ and compare it to $(\\frac{\\alpha}{1-\\alpha})^{\\beta}$ (perhaps a scatter plot)? This can illustrate more clearly what DPO/RLHF failed in reaching the perfect policy.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper provides a thorough empirical and theoretical examination of the performance of preference learning algorithms, particularly RLHF and DPO, in aligning LLMs with human preferences. The paper highlights the significant alignment gap between the observed and idealized ranking accuracies.\nThe empirical study is extensive, covering a variety of state-of-the-art preference-tuned models across multiple datasets, providing robust evidence supporting its claims on the gap between the idealized and learnt policy under the DPO/RLHF objective.", "weaknesses": "1. While it is already argued in the paper that ranking accuracy is practically more meaningful than the reward accuracy, it is still questionable whether totally discarding $\\pi_{ref}$ is reasonable. Specifically, the paper should have included experiment results on whether there is also a gap in reward accuracy between the learnt and ideal policy. After all, the DPO and RLHF algorithm explicitely maintain the reference model $\\pi_{ref}$, and the implicit reward should be indicative on human preference. \n\n2. Due to the small size of preference labels for any given response pair (1 for most dataset, and 4 for the Alpaca Farm as stated in Sec C.4), it is more reasonable to consider the soft ranking accuracy instead of the hard ranking accuracy. The authors could have reported both or discuss on whether hard ranking accuracy is more meaningful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback and suggestions. We appreciate your recognition of the thoroughness of our work. We have included our responses to your questions below and new results in the PDF attached to the global rebuttal.\n\n**\"One concern is that the ranking accuracy considered here is the hard accuracy (accuracy of the Bayesian optimal classifier). I wonder if the \"soft\" ranking accuracy will exhibit more noticable improvement after preference fine-tuning\"**\n\nThank you for this suggestion – we provide plots of the soft ranking accuracy during training for GPT2, Pythia 2.8B, and Llama 2 7B in Fig. 1 of the PDF attached to the global rebuttal. While there is some gradual improvement for the \"incorrect->correct\" group, there is little to no improvement for the examples that stay incorrectly ranked (i.e. \"incorrect->incorrect\" group).\n\n**\"While it is already argued in the paper that ranking accuracy is practically more meaningful than the reward accuracy, it is still questionable whether totally discarding $\\pi_\\text{Ref}$ is reasonable. Specifically, the paper should have included experiment results on whether there is also a gap in reward accuracy between the learnt and ideal policy.\"**\n\nThe reason we consider ranking accuracy instead of reward accuracy is, as mentioned in lines 133-134, \"we ultimately sample from $\\pi_\\theta$ rather than $\\frac{\\pi_\\theta}{\\pi_\\text{ref}}$.\" Moreover, the reward accuracy of many models can be quite high even if the models have terrible ranking accuracy. For example, consider an example $(x,y_w,y_l)$ where $\\pi_\\text{Ref}(y_w|x)=0.1,\\pi_\\text{Ref}(y_l|x)=0.8,\\pi_\\theta(y_w|x)=0.11,\\pi_\\theta(y_l|x)=0.72$. Then the ranking accuracy is 0 on this example because $\\pi_\\theta(y_w|x)=0.11<0.72=\\pi_\\theta(y_l|x)$ but the reward accuracy is 1 because $\\log(\\pi_\\theta(y_w|x)/\\pi_\\text{Ref}(y_w|x))=\\log(1.1)>\\log(0.9)=\\log(\\pi_\\theta(y_l|x)/\\pi_\\text{Ref}(y_l|x))$. Thus, reward accuracy is a much less useful metric to consider.\n\n**\"It is not clear to me what is the main failing part in DPO/RLHF training for high ranking accuracy. In Theorem 3.1, it seems that the perfect RLHF policy should encode the information from $\\pi_\\text{Ref}$ and the dataset perfectly at the same time. Does the DPO/RLHF model fail to capture $\\pi_\\text{Ref}$ or the preference dataset?”**\n\nFirst, we would like to clarify that it is not always possible to encode both the reference model and the dataset perfectly, since there may be conflicts between the two. Indeed, the takeaway from Theorem 3.1 is that the optimal policy encodes the dataset preferences scaled by the reference model, so it is not always possible for the model to recover the true preference probability $\\alpha$ if $\\beta$ is not set correctly (see Table 1). We will be sure to clarify this in our next revision.\n\nSection 4 studies why models fail to achieve high ranking accuracy in practice. Section 4.2 in particular demonstrates that when the reference model has an incorrect ranking on a datapoint, the DPO loss must be reduced to a very small value to correct the ranking (Theorem 4.1 and Figure 3). This value is so small that DPO typically overfits before it reduces the loss to this value (as demonstrated by Figures 2a and 3.) Thus, in practice, it is difficult to optimize the model to achieve the idealized ranking accuracy described in Section 3.\n\n**\"Have the authors tried to plot the quantity $\\frac{\\pi_\\text{DPO}(y_w)\\pi_\\text{ref}(y_l)}{\\pi_\\text{DPO}(y_w)\\pi_\\text{ref}(y_l)}$ and compare it to $(\\frac{\\alpha}{1−\\alpha})^\\beta$ (perhaps a scatter plot)? This can illustrate more clearly what DPO/RLHF failed in reaching the perfect policy.\"**\n\nThank you for this suggestion. We agree that if we had access to the true proportions of raters who preferred one response over the other, denoted $\\alpha$, this analysis would be insightful. Unfortunately, most preference training datasets report only binary preference values, as we discuss in lines 65-76. However, our analyses in Figures 2 and 3 provide a clear diagnosis of the failing – DPO encourages the model to continually increase the reward margins, and flipping the actual ranking is difficult (Theorem 4.1).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "29G61t45Ne", "reviewer": "Reviewer_j6Ru", "summary": "This paper studies the preference learning algorithms like DPO in Large Language Models (LLMs). It focuses on the ranking accuracy and shows that existing algorithms fail to achieve high ranking accuracy. Then, the paper further proposes an idealized ranking accuracy and finds that there is an alignment gap.", "questions": "The contributions, compared with similar works, are not clear for reviewers to give an acceptance recommendation.\n\n[1] Xu, Shusheng, et al. \"Is DPO superior to PPO for LLM alignment? A comprehensive study.\" arXiv preprint arXiv:2404.10719 (2024).\n\n[2] Tajwar, Fahim, et al. \"Preference fine-tuning of LLMs should leverage suboptimal, on-policy data.\" arXiv preprint arXiv:2404.14367 (2024).\n\n[3] Tang, Yunhao, et al. \"Understanding the performance gap between online and offline alignment algorithms.\" arXiv preprint arXiv:2405.08448 (2024).\n\nThe analysis of PPO in terms of ranking accuracy is missing, which somehow makes the conclusion regarding on-policy and off-policy weak.", "limitations": "- The motivation for studying ranking accuracy is weak. The ranking accuracy, especially when defined on the offline dataset, usually does not correlate with the quality of generation behaviors. Let us split the response set into two parts, R_good and R_bad. In practice, we care that sum\\_{y} P( y in R_good) > sum\\_{y'} P(y' in R_bad). The proposed ranking accuracy, instead cares about P(y in R_good) > P(y' in R_bad), where y and y' are two events from R_good and R_bad, respectively.  Thus, a good generative model can provide many good responses with high probability and, at the same time, it may have a low ranking accuracy. This says that ranking accuracy measures the worst-case behavior of generative models. Thus, we do not need to bother ourselves to care about the ranking accuracy, especially for a small dataset where only a few pairs of preference are given. \n- The analysis in terms of ranking accuracy is somehow superficial. In fact, it is the algorithmic bias that DPO introduces the KL-regularized optimization, and as such, it cannot recover the preference-generating distribution. Briefly, the BTL assumption does not invovle the KL term. See the recent work in [1] for more discussion (since this work was posted after NeurIPS submission deadline, it does not affect the review decision). This can also be reflected in equation (2): assume that pi_REF is a uniform policy so its effect can be eliminated and the parameter beta = 1, we see that it recovers the preference-generating policy in terms of alpha. The BTL model assumes that alpha + 1 - alpha = 1, however it may not be true that pi*(y\\_w|x) + pi*(y\\_l|x) = 1.  The procedure I mentioned is also reflected in equation (4), where the authors introduce a new loss. It in fact corresponds to add a entropy regularization to eliminate the bias of pi\\_ref.\n\nBased on the considerations mentioned above, the reviewer does not clearly perceive the technical depth and the contributions of this paper.\n\n[1] Xiao, Jiancong, et al. \"On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization.\" *arXiv preprint arXiv:2405.16455* (2024).", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- A metric called ranking accuracy and a variant of this metric are proposed. \n\n- Extensive empirical observations of DPO are provided, which may be interesting for algorithm analysis and understanding.", "weaknesses": "- The motivation of studying the rank accuracy is weak and not justified well. \n\n- The analysis and results are somewhat superficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you to the reviewer for your thoughtful comments and feedback. We provide responses to your concerns below:\n\n**\"The motivation for studying ranking accuracy is weak.”**\n\nWe study ranking accuracy because DPO directly optimizes for ranking accuracy. For example, the authors of the DPO paper state that \"the DPO update increases the relative log probability of preferred to dispreferred response,\" which corresponds exactly to increasing the ranking accuracy. Our Corollary 3.3 + idealized ranking accuracy results (Table 1) also formalize this. Nevertheless, our paper shows that DPO fails to improve ranking accuracy even though it induces desirable behaviors such as win rate improvement, motivating the need for further investigation. Our analyses help us understand better what the model actually learns during DPO training (if not ranking), and can inform future improvements to preference learning algorithms.\n\n**\"The analysis of PPO in terms of ranking accuracy is missing, which somehow makes the conclusion regarding on-policy and off-policy weak.\"**\n\nFigure 1b and Table 1 contain the ranking accuracies of multiple RLHF-PPO models. Theorem 3.1 and Corollary 3.3 also apply to both RLHF-PPO and DPO.\n\n\n**“The ranking accuracy, especially when defined on the offline dataset, usually does not correlate with the quality of generation behaviors.\"**\n\nWe agree, and Section 5 goes further to plot these correlations explicitly. We find that when the model is close to the reference model, the win rate and the ranking accuracy trend together, but as training goes on, they become anti-correlated. **Our paper does not argue that ranking accuracy should replace the win rate metric.** Instead, we demonstrate that DPO does not accomplish what it was designed to do (*i.e.* optimize ranking accuracy), despite its success at improving win rate. \n\n**\"Let us split the response set into two parts, R_good and R_bad. In practice, we care that sum_{y} P( y in R_good) > sum_{y'} P(y' in R_bad)... This says that ranking accuracy measures the worst-case behavior of generative models. Thus, we do not need to bother ourselves to care about the ranking accuracy\"**\n\nThe case brought up by the reviewer concerns generalization, while our analysis is concerned with whether DPO even achieves its own training objective, i.e., whether the model achieves high ranking accuracy **on its own training dataset**. This is a much easier case.\n\n**\"The analysis in terms of ranking accuracy is somehow superficial. In fact, it is the algorithmic bias that DPO introduces the KL-regularized optimization, and as such, it cannot recover the preference-generating distribution.\"**\n\nWe respectfully disagree. We agree with the reviewer that even the target distribution of DPO and RLHF-PPO may not have perfect ranking accuracy even on the training data (see discussion on idealized models in Section 3.2, and note that the Table 1 results under $\\mathcal{R}^*$ are often less than 100%). However, our results show that there is a gap even between the idealized target distributions and models in practice (see Table 1). In other words, the reviewer’s analysis is not sufficient to explain the results of models in practice; to fill in this gap, we investigate what happens throughout training (Section 4 and 5).\n\nXiao et al. provides an interesting and complementary perspective on our work, but their proposed regularizer appears to promote higher perplexity in general (see [Table 2 in Xiao et al.](https://arxiv.org/pdf/2405.16455v1#page=24.44)). Rather than resolving the \"algorithmic bias\" described by Xiao et al., we investigate the gap between on- and off-policy behavior.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "42K3jpAZlu", "reviewer": "Reviewer_5qJ8", "summary": "The paper investigates the effectiveness of preference learning algorithms, RLHF and DPO, in training LLMs to rank human-preferred outputs higher. It challenges the assumption that these algorithms can successfully teach models to rank outputs according to human preferences. Empirical findings show that most state-of-the-art models achieve low ranking accuracy, often below 60% on standard datasets. The paper identifies a significant alignment gap due to the limitations of the DPO objective and provides a theoretical analysis of the difficulty in correcting ranking errors. It concludes by highlighting the need for a reevaluation of these algorithms and calls for more nuanced analyses of preference training dynamics.", "questions": "See Weaknesses.", "limitations": "Yes, the authors fully address the limitations and potential social implications of their work.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a thorough empirical analysis, revealing the limited effectiveness of RLHF and DPO in improving ranking accuracy, which is a significant contribution to the field.\n\n2. It offers theoretical insights into why these algorithms fail to correct ranking errors, providing a deeper understanding of the underlying issues in preference learning for LLMs.\n\n3. The findings have clear practical implications for the development of alignment techniques in LLMs, motivating the need for more effective methods to train models that better reflect human preferences.", "weaknesses": "1. While the paper suggests a link between preference learning and win rate, it would be strengthened by a more explicit exploration of how these metrics interrelate. Specifically, the authors could benefit from a deeper analysis of the implications of using ranking accuracy versus win rate as measures of model performance, including potential discrepancies and how they might be reconciled.\n\n2. The paper identifies a significant determinism of ranking accuracy by reference model likelihoods, which is a crucial observation. However, it would be advantageous for the authors to expand on this point by discussing potential techniques or algorithmic adjustments that could alleviate the heavy reliance on the reference model. This would bolster the paper's contribution by addressing a key challenge in the field of preference learning for LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the “clear implications” of our theoretical and empirical analysis. We answer additional questions below.\n\n**\"Specifically, the authors could benefit from a deeper analysis of the implications of using ranking accuracy versus win rate as measures of model performance, including potential discrepancies and how they might be reconciled.\"**\n\nThank you for this suggestion. We agree that it would be interesting to further study when and to what extent offline metrics like ranking accuracy are indicative of generative behaviors from the model. Our experiments in Section 5 initiate a study of this, and we find that when the model is close to the reference model, the win rate and the ranking accuracy trend together. We hope that future work can study how different design choices, such as on-policy data (lines 269-275), affect this relationship.\n\n**“The paper identifies a significant determinism of ranking accuracy by reference model likelihoods, which is a crucial observation. It would be advantageous for the authors to expand on this point by discussing potential techniques or algorithmic adjustments that could alleviate the heavy reliance on the reference model.”**\n\nWe agree that it is important to find better ways to regularize toward the reference model. Our formulation of the $L_\\text{DPO}^\\gamma$ objective in Equation 4 is one possibility, and we discuss several others in lines 269-275. We furthermore note that several prior/concurrent works, including [CPO](https://arxiv.org/abs/2401.08417) and [SimPO](https://arxiv.org/abs/2405.14734), remove the reference model entirely and instead modify the objective and dataset to prevent the generative behavior from degrading during preference learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6pG4zXFkbv", "reviewer": "Reviewer_uW8U", "summary": "This paper investigates the effectiveness of preference learning algorithms, include the RLHF (PPO-based) and DPO in training LLMs. In particular, the authors study the non-regularized ranking accuracy, which is largely ignored in the previous RLHF literature. The authors approach this investigation through both theoretical analysis and empirical experiments and show that whether the ranking is correct or not  highly depends on the initial reference policy.", "questions": "see weakness", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The research problem is interesting. While the RLHF and DPO mainly care about the KL-regularized target, it is debatable that the likelihood of the response under the resulting policy is what we are really interested in. Therefore, the notion of ranking policy in terms of this non-regularized target deserves more attention. \n\nWith the goal in mind, the DPO loss target in definition 2.2 clearly shows that the log-ratio on the reference model serves as an adaptive margin for the data point. Therefore, the discussions around the reference model is natural and informative. \n\nI particularly appreciate the section 4 on the empirical studies of DPO, which can largely help us to understand the dynamic of this widely adopted algorithm. \n\nOverall, the authors identify an interesting question and the present a complete story with solid evidence in both theory and experiments.", "weaknesses": "My major concern is that this paper only studies the goal of helpfulness in chat style. \n\nFirst, while it is fair to say that DPO is not good at optimizing the ranking policy in terms of the non-regularized target, it is not immediately clear that the non-regularized ranking policy is a better metric. For instance, a concurrent work (so this should not hurt the novelty of this paper) [1] proposes simpo with many shared insights. In particular, they also notice the mismatch between the ranking policy and the target of DPO. However, in their experiments, as the win rates evaluated on the standard benchmark significantly increase, we also observe a significant drop in some academic benchmarks, particularly those on the mathematical problems solving. \n\nBased on my personal experience, if we adopt a smaller KL coefficient (e.g., 0.01, so less regularization), the final model performance is worse than the model with a moderate one (say 0.1). This is particularly true for the online iterative learning, which we are more interested in as the authors also notice that these variants outperform the offline one with a large margin. In most of the cases, the ability of maximizing the reward/ranking accuracy, while maintaining a moderate KL divergence is more suitable to evaluate the algorithm performance in my viewpoint. Therefore, the ranking accuracy alone is also not a good learning target, which is also evidenced by figure 4 of this work.\n\nBased on the above two points, I feel that the paper can still be largely improved if a more comprehensive evaluations can be done, like the trade-off between the performance on instruction-following benchmarks (alpaca-eval, mt-bench, gpt4 win rate) and other  academic benchmarks. But overall, I support the acceptance.\n\n[1] SimPO: Simple Preference Optimization with a Reference-Free Reward", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed review and greatly appreciate their support of the acceptance of our paper. We address the feedback below:\n\n**\"First, while it is fair to say that DPO is not good at optimizing the ranking policy in terms of the non-regularized target, it is not immediately clear that the non-regularized ranking policy is a better metric.\"**\n\nWe are not entirely sure we understood your question: what is the “non-regularized ranking policy,” and what would it mean for this policy to be a better metric? Thank you!\n\n**\"In most of the cases, the ability of maximizing the reward/ranking accuracy, while maintaining a moderate KL divergence is more suitable to evaluate the algorithm performance in my viewpoint. Therefore, the ranking accuracy alone is also not a good learning target, which is also evidenced by figure 4 of this work.\"**\n\nWe agree that optimizing for ranking accuracy alone is not necessarily a good learning target; to clarify, we are not suggesting that one should be optimizing ranking accuracy. Rather, we use ranking accuracy as a lens to better understand what models learn during DPO training, as well as when it correlates with win rate.\n\n**\"However, in [SimPO’s] experiments, as the win rates evaluated on the standard benchmark significantly increase, we also observe a significant drop in some academic benchmarks, particularly those on the mathematical problems solving… I feel that the paper can still be largely improved if a more comprehensive evaluations can be done, like the trade-off between the performance on instruction-following benchmarks (alpaca-eval, mt-bench, gpt4 win rate) and other academic benchmarks\"**\n\nIf we understand correctly, the reviewer seems to be suggesting that, in addition to assessing win rate on the query distribution of interest, we should evaluate the models in the paper on benchmarks for other domains. We believe this is out of scope of this paper, which focuses on the fact that DPO does not even yield correct rankings on the training data. We do agree broadly with the reviewer’s sentiment, though, that it is useful to assess top-performing models on a wide variety of benchmarks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are grateful to the reviewers for their detailed feedback – Reviewer yhyh notes the paper's "robust evidence supporting its claims," Reviewer 5qJ8 comments that our "findings have clear practical implications for the development of alignment techniques," and Reviewer uW8U notes "the authors identify an interesting question and the present a complete story with solid evidence in both theory and experiments…overall, I support the acceptance."

On the other hand, the reviewers expressed some confusion about the choice to analyze ranking accuracy. We focus on this metric not because we believe it to be the best metric of model performance, but because algorithms like DPO *directly optimize for ranking accuracy*. We show that since DPO does not successfully optimize its objective *even on the training dataset* (but does meaningfully improve generation quality as measured by win rate), the learning mechanisms behind such preference learning algorithms deserve further attention. We will be sure to clarify this point in future revisions.

We additionally attach a PDF containing the soft ranking accuracy results requested by Reviewer yhyh.

Meta-review: This paper studies preference learning with RLHF and DPO for LLMs.  It highlights some enlightening empirical findings, such as the existing preference learning algorithms obtain low ranking accuracies, the idealized policy and these methods' policies have a significant ranking accuracy gap.  They show that when the model is close to reference the win rate and preference accuracy are strongly correlated.  The reviewers praise the extensive empirical comparisons and how well their claims are supported by evidence.  They also praise the theoretical insights that can explain where these method fail.  The authors clearly rebut criticisms such as why they prefer ranking accuracy over reward accuracy, and they provide results for soft-ranking accuracy in addition to the hard ranking accuracy.
Decision: Accept (poster)