Preference Learning Algorithms Do Not Learn Preference Rankings

Authors: angelica  chen
sadhika  malladi
lily h zhang
xinyi  chen
qiuyi  zhang
rajesh  ranganath
kyunghyun  cho
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 20:53:44

Abstract: Preference learning algorithms (e.g., RLHF and DPO) are frequently used to steer LLMs to produce generations that are more preferred by humans, but our understanding of their inner workings is still limited. In this work, we study the conventional wisdom that preference learning trains models to assign higher likelihoods to more preferred outputs than less preferred outputs, measured via *ranking accuracy*.
Surprisingly, we find that most state-of-the-art preference-tuned models achieve a ranking accuracy of less than 60% on common preference datasets. We furthermore derive the *idealized ranking accuracy* that a preference-tuned LLM would achieve if it optimized the DPO or RLHF objective perfectly. We demonstrate that existing models exhibit a significant *alignment gap* -- *i.e.*, a gap between the observed and idealized ranking accuracies. 
We attribute this discrepancy to the DPO objective, which is empirically and theoretically ill-suited to correct even mild ranking errors in the reference model, and derive a simple and efficient formula for quantifying the difficulty of learning a given preference datapoint.
Finally, we demonstrate that ranking accuracy strongly correlates with the empirically popular win rate metric when the model is close to the reference model used in the objective, shedding further light on the differences between on-policy (e.g., RLHF) and off-policy (e.g., DPO) preference learning algorithms.

Main paper: 
 introduction: 
 Recent work on aligning LLMs has focused predominantly on tuning models to adhere to human preferences -commonly through reinforcement learning (RLHF; Stiennon et al. [47] ) or directly via offline supervision (DPO; Rafailov et al. [41] ). Preference learning algorithms [20, 55, 58] were originally designed to use a dataset of pairwise preferences over candidates to train a model with high ranking accuracy -that is, the model can precisely rank preferred outputs over dispreferred ones. In the case of language models, the ranking is determined by the likelihood assigned to each candidate.Many LLM alignment techniques are designed to yield models with a high preference ranking accuracy, including SLiC [68, 67] , RAFT [8] , PRO [46] , and RRHF [64] . Most prominently, Rafailov et al. [41] claimed that their popular direct preference optimization (DPO) algorithm "increases the relative log probability of preferred to dispreferred response." It is standard to evaluate these various objectives by measuring how often the resulting model's generations are preferred over another model's (i.e., a win rate) [71] . However, the relationship between the loss, ranking accuracy, and win rate is unclear, leaving open the question of what these alignment techniques are actually accomplishing during training.In this work, we demonstrate that RLHF and DPO struggle to increase ranking accuracy in practice and explore both the theoretical and empirical reasons why. Our findings highlight an intricate relationship between offline optimization and online behavior, and motivate the need for more fine-grained analyses of preference training dynamics. Our contributions are as follows:1. Existing models do not achieve high ranking accuracies. We demonstrate that a wide variety of open-access preference-tuned LLMs (e.g., LLAMA 2 7B CHAT, GEMMA 7B IT, and ZEPHYR 7B DPO) achieve a ranking accuracy below 60% across a range of validation splits from commonly used preference datasets, such as UltraFeedback [7] , Anthropic helpfulness and harmlessness (HH-RLHF, [14] ), and Stanford Human Preferences (SHP, [11] ) (Figure 1 ). Although we do not advocate for ranking accuracy as a measure of model quality, we analyze LLMs' ranking accuracies nonetheless because (1) ranking accuracy has motivated the design of many preference learning algorithms, and (2) DPO directly optimizes for ranking accuracy (Theorem 3.1).Existing models exhibit a significant alignment gap between the ranking accuracy they achieve and the accuracy achievable under idealized conditions. We derive a simple formula (Theorem 3.1) for the idealized ranking accuracy (i.e., the ranking accuracy achieved from training on ground-truth preference data and perfectly optimizing the DPO or RLHF objective). We observe that models suffer from a significant alignment gap in that they achieve ranking accuracy far below the idealized ranking accuracy (Table 1 , Figure 1 ).Preference learning rarely corrects incorrect rankings. We prove theoretically that even mild ranking errors in the reference model can make it virtually impossible for DPO and its variants to correct the ranking (Theorem 4.1), and demonstrate that in practice, the rankings are rarely flipped (Fig. 2 ) and the reference model likelihoods generally determine the ranking accuracy (Fig. 3 ). Our results permit straightforward and efficient identification of hard-to-learn preference datapoints without any tuning. 4 . Ranking accuracy and win rate are closely correlated when the model is close to the reference model. We observe that the ranking accuracy and win rate trend together when the model is close to the reference model during the early phase of alignment, but become anti-correlated once the model has moved too far away, adding to the ongoing discussion on the differences between on-policy and off-policy behaviors of preference-tuned LLMs.Crucially, our work highlights fundamental flaws in RLHF and DPO that prevent the preference-tuned model from achieving a high ranking accuracy even on the training dataset. 
learning from human preferences: 
 Preference Data Human preference data typically takes the form of pairwise preferences. Each prompt x is paired with two possible continuations -y 1 and y 2 . One or more human raters then annotate which continuation is preferred. When there are multiple raters, we use α(x, y 1 , y 2 ) to denote the proportion of raters who prefer y 1 over y 2 .1 Definition 2.1 (Aggregated Preference Datapoint). Consider a prompt x with two possible continuations y 1 and y 2 and the proportion of raters α(x, y 1 , y 2 ) who preferred y 1 over y 2 . Then, the aggregated preference datapoint for each prompt x is denoted (x, y w , y l ) where y w is the completion preferred by the majority of voters.We note that at the time of writing, the vast majority of datasets either use a single rater [14] or only release aggregated preference data [11, 26] , so we often do not have access to α(x, y 1 , y 2 ).A standard assumption is that the ground-truth human preferences obey the Bradley-Terry model (Assumption A.1).Supervised Fine-Tuning (SFT) In the first step of the preference learning pipeline, the model is typically trained using the standard cross-entropy objective on some choice of offline instructiontuning dataset(s). In some implementations [52] , a variety of third-party datasets are selected, whereas in other implementations [47, 41, 43] the model is instead trained on the preferred continuations (x, y w ) from the same preference learning dataset that is used in downstream preference learning. The resulting model is often used as a reference model, denoted as π Ref or π SFT , and it typically serves as the initialization when learning from human preferences.Reinforcement Learning from Human Feedback (RLHF) Learning from human feedback originally required using reinforcement learning [47] . In this setting, the possible continuations for each prompt are sampled from a reference model (i.e., (y w , y l ) ∼ π Ref (• | x)) and then annotated and aggregated to create a preference dataset D. Then, one frames the problem as binary classification between the two continuations and trains a reward model r ϕ (x, y) to minimize L R (r ϕ , D) = -E (x,yw,y l )∼D [log σ(r ϕ (x, y w ) -r ϕ (x, y l ))]. Finally, one trains the model π θ to maximize the reward without straying too far from the reference model π Ref . Because sampling generations from the model is non-differentiable, it is common to use PPO to maximize the reward r(x, y) = r ϕ (x, y) -β(log π θ (y | x) -log π Ref (y | x)), where β > 0 is a regularization coefficient designed to prevent the model from straying too far from its initialization.Preference Learning with DPO Rafailov et al. [41] demonstrated that one can avoid using PPO by reparametrizing the objective to operate over policies instead of over rewards. Then, one can minimize the differentiable DPO objective.Definition 2.2 (DPO Objective [41] ). Let σ be the sigmoid function and β > 0 be a hyperparameter. Then, the DPO objective for an aggregated preference dataset D and a reference model π Ref is defined as We denote the DPO loss on the aggregated datapoint (x, y w , y l ) as L DPO (x, y w , y l ; π θ , π Ref ).L DPO (π θ , π Ref ) = - E (x,Evaluating the alignment of a preference-tuned LLM is both under-specified and multi-dimensional. Many knowledge-based and logic-based benchmarks (e.g. MMLU, GLUE, BIG-Bench, HELM) already exist, but these benchmarks largely fail to capture nuanced aspects of human preference, such as helpfulness or harmlessness [14] . As such, one standard evaluation is to ask human or machine raters how often the model produces a favorable completion compared to a baseline (i.e., win rate). Human win rate is the gold standard but is costly to compute and can be biased based on size and nature of the worker pool [19, 25] . Rating completions using another LLM (e.g., MT-bench) can be cheaper but similarly suffers from various biases [39, 69, 62] , and several studies have revealed failures in many LLM judges to identify violations of instruction-following [66, 29] . Nevertheless, since win rate evaluations are so prevalent, we compare ranking accuracy against win rate in Sec. 5 and describe when the former off-policy metric is correlated with the popular on-policy metric.Besides the win rate, preference learning algorithms are also benchmarked by the frontier of the rewards versus the divergence from the initialization [41] , which serves as a heuristic of how well the model can incorporate preference data without unlearning prior information. However, it is unclear how well rewards can describe the success of alignment.As aforementioned, the current paper investigates the ranking accuracy, which is defined as follows:Definition 2.3 (Ranking Accuracy). The ranking accuracy R of a model π θ on an aggregated preference datapoint (x, y w , y l ) is defined asEQUATIONAnalogously, the ranking accuracy of policy π θ on a dataset D = {(x, y w , y l )} is R(D; π θ ) = E(x,y w ,y l )∼D R(x, y w , y l ; π θ ). In the rare case where a dataset has more than two outputs y per prompt x, we use the generalized ranking accuracy definition stated in App. A.6. We do not advocate for ranking accuracy as a metric of model quality, but as a lens into the inner workings of common preference learning algorithms. Remark 2.4 (Lengths of Completions). We note that y w and y l can have different lengths; for example, Singhal et al. [45] showed that y w is usually longer. Length can deflate π θ (y w | x) and reduce the ranking accuracy. One can normalize the likelihoods by the length of the response, but the length-normalized ranking accuracy may not be meaningful in practice, because it is currently unclear how to sample from the length-normalized likelihood. For completeness, we report the ranking accuracies of both the unnormalized and normalized policies, denoted R and R, respectively. Remark 2.5 (Difference between Ranking Accuracy and Reward Accuracy). For RLHF models and DPO models, the ranking accuracy is not equivalent to the reward accuracy (i.e., the metrics evaluated in RewardBench [29] ). In the RLHF case, we are evaluating the ranking accuracy of the final policy rather than the reward model. In the DPO case, reward accuracy measures whetherβ log π θ (yw|x) πRef(yw|x) > β log π θ (y l |x) πRef(y l |x) instead of whether π θ (y w | x) > π θ (y l | x).Since we ultimately sample from π θ rather than π θ (y|x) πRef(y|x) , we find the ranking accuracy to be of greater practical importance.Moreover, we demonstrate that under very stringent conditions, minimizing the DPO objective results in a model with high ranking accuracy. We characterize the phenomenon on individual datapoints, as is the case throughout the paper, but note that Markov's inequality can be straightforwardly applied to extend the results to a low loss on the entire dataset. This result, proved in App. A.1, requires the condition that the reference model already has the correct ranking, so it is unlikely to hold across all datapoints in practice and somewhat moot. The remainder of the paper focuses on more realistic settings where the reference model is imperfect.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Assumptions required for the theory are stated formally for each result. The discussion section lists limitations of the experimental and conceptual scopes of the paper. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
the alignment gap: 
 Prop. 2.6 showed that training a low DPO loss with a perfect reference model yields a model with perfect ranking accuracy. However, Fig. 1a shows that real-world reference models exhibit low ranking accuracies, which prompts us to study more realistic, imperfect reference models.Fig. 1a indicates that reference models rarely achieve high ranking accuracy on common preference datasets (except Synthetic Instruct GPT-J Pairwise), even though many were likely trained on the preferred completions (see Sef. 2.1). Many of the models do not have documented training data so we do not know which preference datasets, if any, are in-distribution. We also fine-tune several pretrained LLMs on the preferred completions (see App. B.1) and observe that ranking accuracy does not increase significantly. 2 Based on our findings, we turn to the case of imperfect reference models.We showed above that empirically, reference models exhibit poor accuracy when ranking the plausible completions. However, the RLHF reward and DPO objective were formulated to ensure that the Length-Normalized Non-Length-Normalized (b) Ranking accuracies of various preference-tuned models, including LLAMA 2 7B CHAT [52] , TULU2 7B DPO [21] , ZEPHYR 7B DPO [54] , and GEMMA 7B IT [51] Figure 1 : Both reference and preference-tuned models exhibit low ranking accuracy on most preference datasets. model learns the preference dataset but does not move too far from the reference model π Ref , so there may be a limit on the possible accuracy of the preference-tuned model. Here, we formalize this intuition by studying what the optimal policies would be when perfectly optimizing DPO or RLHF with access to perfect data (i.e., true proportions of human preferences). 3Theorem 3.1 (Simulating Perfect RLHF 4 ). Fix a reference model π Ref and an aggregated preference datapoint (x, y w , y l ) ∼ D. Assume the dataset includes the ground-truth human preferences: that is, α(x, y w , y l ) = P(y w ≻ y l ), and that these preferences obey the Bradley-Terry model (Assumption A.1). Let π * be the model resulting from perfectly optimizing the DPO or RLHF objective on (x, y w , y l ) as described in Section 2.1. Then, π * satisfiesπ * (y w | x) π * (y l | x) = π Ref (y w | x) π Ref (y l | x) α(x, y w , y l ) 1 -α(x, y w , y l ) 1/β (2)where α(x, y w , y l ) is the proportion of raters who preferred y w over y l and β is a hyperparameter in the DPO and RLHF objectives. Remark 3.2. We prove this result in App. A.2. Note that deterministic preferences, i.e., α(x, y w , y l ) = 1, should not be confused with settings that only report a single individual's preference, i.e., α(x, y w , y l ) = 1. In the former, the optimal probability ratio is infinity. The latter requires a better estimate of P(y w ≻ y l ), e.g., more samples. This result allows us to simulate the policy resulting from perfect optimization of either the RLHF or the DPO learning objective. As such, given a reference model π Ref and preference dataset D, we can easily measure the idealized ranking accuracy of a model. We prove this result in App. A.3. Corollary 3.3 (Idealized Ranking Accuracy). Given a reference model π Ref , the DPO or RLHF hyperparameter β, a dataset of aggregated preferences D = {(x, y w , y l )} and their corresponding rater proportions α(x, y w , y l ), the ranking accuracy of the optimum of the RLHF or DPO objective π * is given by denote the idealized ranking accuracy as R * .EQUATIONGiven access to π Ref , β, and the α(x, y w , y l ) values for each triple (x, y w , y l ) in a given preference dataset, we can compute the idealized ranking accuracy from Eq. 3. 5 The results are shown in Table 1 and further details are given in App. B.4.We identify several surprising findings. Firstly, even under ideal conditions (i.e. perfectly optimizing the objective on ground-truth preference data), the idealized ranking accuracy is still sometimes below 100%. This distance varies with the choice of β, which indicates that the limits of DPO/RLHF depend largely upon how strong the reliance on π Ref is. Furthermore, we find that many state-of-the-art models do not achieve a ranking accuracy anywhere close to the idealized ranking accuracy, exhibiting alignment gaps ranging from 19 to 59 percentage points (measured to the median idealized R or R).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: Each theorem is self-contained in stating the required assumptions needed. Proofs are provided in detail in Appendix A. We omit proof sketches due to space constraints and because the formal results are mostly straightforward algebraic manipulations. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
understanding ranking accuracy with dpo: 
 We now turn to the training objectives to account for the alignment gap. We focus our analysis on the DPO objective (Definition 2.2), because its failure to achieve high ranking accuracy is especially surprising (Table 1 ). In particular, DPO directly maximizes the reward margin between preferreddispreferred pairs over an offline dataset so we would expect it to perform well on in-distribution held-out data. We also note that DPO is a popular choice in the community for aligning LLMs, because it is less costly than performing RLHF.In this section, we study real-world characteristics of DPO. First, we show empirically that DPO rarely flips the ranking of the two continuations. This result combined with the observation that reference models exhibit poor ranking accuracy (Sec. 3.1) provides an explanation for the observed poor ranking accuracies in Table 1 . We then formally characterize how hard it is for DPO to correct the ranking of each datapoint.Our result highlights how the reference model conditions the optimization: as the reference model log-ratio (Definition 2.2) grows larger, one has to reduce the DPO loss to a dramatically small value to flip the incorrect ranking (Fig. 3 ).To study how ranking accuracy changes over the course of DPO training, we train three sizes of models (GPT-2 [40] , Pythia 2.8B [4] , and Llama 2-7B [52] ) across three seeds each on the Anthropic HH-RLHF [3] preference dataset and study the ranking accuracy on different partitions of the training dataset. We present the results from training one seed of Pythia 2.8B in Fig. Surprisingly, Fig. 2 demonstrates that DPO rarely flips the ranking of (y w , y l ) over the course of training despite consistently reducing the loss L DPO . Aside from the group of points for which the model unlearns the correct preference ranking, we observe that the loss decreases and the reward margin increases consistently while training. However, at the point of lowest validation loss (marked by the vertical dashed line in Fig. 2c ), less than 10% of the originally incorrectly ranked points have been flipped to have the correct ranking. Past this point, the model begins to overfit (i.e., the validation loss begins to increase). DPO does not substantially improve the ranking accuracy until far past the point of overfitting. Although Theorem 3.1 predicts that the theoretically optimal DPO model exhibits high ranking accuracy, in practice the empirical endpoint of training occurs far from the theoretical optimum. This indicates that the DPO objective is ill-formulated to induce a high ranking accuracy in practice.In the result below, we show that the DPO loss can decrease substantially without any improvement on the ranking accuracy of the model. Specifically, the DPO loss that the model needs to reach in order to have the correct ranking on an example (x, y w , y l ) depends on the quality of the reference model, quantified by the reference model log-ratio. This dependence is highly ill-conditioned, whereby using a reference model with moderately incorrect likelihoods assigned to each continuation can effectively prevent DPO from learning the correct ranking. Theorem 4.1. Consider an aggregated preference datapoint (x, y w , y l ) such that the reference model log-ratio is some constant c, i.e. log πRef(y l |x) πRef(yw|x) = c. Then, R(x, y w , y l ) = 1 if and only if L DPO (x, y w , y l ) ≤ -log σ(βc), where σ is the sigmoid function. Remark 4.2. It is straightforward to extend our analysis to popular variants of DPO. For illustration, we prove an analogous result for identity preference optimization (IPO, Azar et al. [2] ) in App. A.5.We prove this result in App. A.4. Our theoretical result allows us to formally identify the points that will be hard to flip in their rankings. Fig. 3 visualizes the reference model log-ratio for several settings and highlights that datapoints with even mild ranking errors in the reference model will require the loss to be reduced to a very low value in order to flip the ranking. App. E contains examples of hard- to-learn, easy-to-learn, and easy-to-flip datapoints. We observe that the hard-to-learn datapoints are substantially longer than the easy ones, and that the easy datapoints generally contain less ambiguous preference annotations. More generally, our result motivates the use of stronger π Ref models and iterative or on-policy variants of DPO [50, 65, 24, 53] .Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We detail datasets, models, and precise algorithms to take measurements in the appendix. Our training runs can be reproduced with minimal modification to the HuggingFace transformer reinforcement learning (TRL) library 7 . We additionally provide anonymized code. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
ranking accuracy and win rate: 
 Our results on ranking accuracy illuminate how well DPO and RLHF can align to preference data, but we have not yet related these insights to how the generative behavior of the model changes during alignment. In particular, ranking accuracy is a convenient but off-policy metric and is thus not as widely adopted as the on-policy metric of win rate (see Sec. 2.2). Indeed, one could maximize the ranking accuracy by learning a strong classifier on the preference data, but that model may not generate high-quality text. Here, we explore the gap between on-policy (i.e., generative) and off-policy (i.e., classification) behaviors of LLMs through the lens of ranking accuracy and win rate. Since the DPO objective directly optimizes for ranking accuracy (Proposition 2.6), the relationship between these two metrics is a direct reflection of how off-policy training affects on-policy behavior.We study the relationship between win rate and ranking accuracy in two settings: (1) during DPO training, and (2) in a DPO variant modulating the influence of π Ref . We measure the win rate on 500 responses to prompts from the training dataset using the Alpaca Eval GPT-4 [30] auto-annotator.Setting 1: DPO Training. We measure the win rate and the ranking accuracy of a Pythia 2.8B model [4] during DPO training with the same configuration as in Section 4. See Fig. 9 for the results.Setting 2: Attenuating the reference model. Theorem 4.1 showed that π Ref exerts a negative influence on the ranking accuracy in most cases, so we design a new objective that scales the reference model log-ratio in L DPO to further characterize how win rate and ranking accuracy relate.EQUATIONNote that L γ DPO = L DPO (Definition 2.2) when γ = 1, and a larger value of γ increases the role of the reference model. Also, γ directly scales c in Theorem 4.1, thereby controlling how easy it is to fit the data and increase the ranking accuracy. We train a range of Pythia-2.8B models using the L γ DP O objective for γ ∈ {0, 0.25, 0.5, 0.75, 1.0, 1.25, 1.5, 1.75, 2.0} and measure the ranking accuracies and win rates of the best model for each γ value. 6Takeaway: Ranking accuracy correlates with win rate when the model is close to the reference model. In both settings, we observe that the win rate and ranking accuracy are highly correlated with one another in the early phase of training but become anti-correlated (i.e., ranking accuracy increases but win rate declines) as the model π θ moves away from the reference π Ref (Fig. 4 ). Unlike traditional overfitting, the test loss is continuing to decline at this point (Fig. 9b ). Experiments in Fig. 10 with the attenuated objective in Equation ( 4) further show that ranking accuracy and win rate trend together when the influence of the reference model is stronger (i.e., γ is larger).We speculate that when the model is far from the reference model, overly optimizing the reward margin can harm the generative capabilities of the model, which are primarily acquired during pre-training. In other words, the off-policy behavior of the model can no longer predictably describe the on-policy generations when the reference model used in the offline objective is far from the current model. Our findings confirm the fundamental tradeoff between fitting the preference data and maintaining generative capabilities acquired during pre-training [22] and align with prior observations that adding on-policy preference data can make offline learning more effective [50, 65, 24, 53] .1. 25 Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: We provide an anonymized version of our code. Our results can be reproduced with minimal modification to the HuggingFace transformer reinforcement learning (TRL) library, and the datasets used are all open-access and easily downloadable from the HuggingFace Hub.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
related work: 
 Analyses of Preference Learning Algorithms Many works have investigated the role of the preference dataset [56, 61] , the reliability of the evaluations [69, 29] , and the confounding factor of response length [45, 10, 57, 37] . Theoretical works have unified the many preference learning algorithms into clear taxonomies that permit analysis and, sometimes, yield new variants [2, 61, 48, 50, 63, 34] . Several works study idiosyncrasies of preference learning, such as why DPO decreases the likelihood of both rejected and chosen outputs from the dataset [42, 13, 35] and why RLHF exhibits vanishing gradients [44] . In contrast, our work approaches understanding DPO and RLHF through the lens of ranking accuracy, and our findings emphasize the role of the reference model regularization in preference learning. Relatedly, SliC-HF [67] , CPO [59] , and pairwise cringe loss [60] optimize log probability margins log π(y w |x) -log π(y l |x), effectively removing the regularization toward the reference model. Liu et al. [32] recommend using the reference model at inference time to exert more granular control over the regularization. Meng et al. [34] remove the π Ref terms altogether and use a target reward margin to prevent over-optimization. Additionally, Chennakesavalu et al.[6] design a DPO-like objective that includes an additional hyperparameter controlling the strength of the π Ref terms, similar to our L γ DP O objective (Eq. 4). Tang et al. [50] also analyze the role of regularizing toward a reference model, though our work focuses the effect of this regularization on ranking accuracy.On-policy and Off-policy Preference Learning Preference-tuning LLMs originally required using an on-policy algorithm [47] , but many recent works have derived off-policy methods that can use a static preference dataset for supervision [41, 12, 18, 37] . Off-policy methods are preferred for their efficiency and ease of implementation, but several works have suggested that on-policy methods are superior [48, 61, 49, 31] . Several iterative training methods aim to bridge this gap, where the reference model and the dataset are refreshed during the alignment procedure to contain annotated preferences on generations from the model at that point in training [65, 24, 53] . These intuitions align strongly with our observation that win rate and ranking accuracy, and thus, on-policy and off-policy behavior, are strongly correlated when the model is close to the reference model.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Appendices B and C detail all of the information about the data splits, hyperparameter grids, optimizer, and so on.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Our discussion in Section 7 discusses the impact of our findings, especially as it relates to measuring and verifying the success of preference learning and alignment of LLMs. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
discussion: 
 Our work highlights the significant but nuanced relationship between preference learning and ranking accuracy. We have demonstrated both theoretically and empirically that RLHF and DPO struggle to teach the model to correctly rank preferred and dispreferred outputs, even in the training dataset. Although the learning objective promotes high ranking accuracy in theory (Proposition 2.6), we observed a prominent alignment gap resulting from the poor conditioning of reference models. We then drew connections between the off-policy nature of ranking accuracy and the on-policy evaluations of win rate, identifying specific scenarios in which on-policy behavior can or cannot be reliably predicted by off-policy behavior. App. 8 details the limitations of our work.Our work shows that it is difficult to steer pre-trained LLMs to adhere to even the preference data used for training. When LLMs are used to judge responses from other models [30, 69] or to improve their own abilities [33, 65] , poor ranking accuracies can induce strong negative feedback loops that are costly to mitigate.We also observe that win rate does not monotonically increase during training (Fig. 9a ), despite the decrease in both train and test loss (Fig. 9b ) and the modest gain in ranking accuracy (Fig. 9a ). As such, it is clear that we still do not understand the behaviors of preference learning. For example, others have observed that DPO can cause the likelihoods of both chosen and rejected outputs to decrease [42, 13, 35, 36] , which implies that the policy must be moving probability mass to possibly undesirable sequences outside the data distribution. Moreover, our investigation of the non-monotonic relationship between ranking accuracy and win rate emphasizes the need for concrete evaluations that can more reliably and transparently measure the success of preference learning.Future Work Our theoretical results only describe the behavior of the model on the preference data used during training, but they can serve as a starting point for understanding generalization to different distributions of data, especially the one prescribed by the model itself [48] . Furthermore, we hope to analyze the optimization dynamics of preference learning, given the intriguing relationship observed between ranking accuracy and win rate. For instance, identifying when the win rate begins to diverge from the ranking accuracy can motivate adding fresh on-policy training data. Our initial investigation into ranking accuracy also suggests that it is worthwhile to explore how alignment techniques interact with other calibration metrics. 
limitations: 
 Although we reproduce our main results (Sec. 4) on three types of models across three seeds each, these models are trained on a single dataset due to the computational constraints. Our theoretical results lead us to believe that our findings would generalize to other datasets, but empirical verification is still valuable. As mentioned previously, our work is optimization-agnostic and only describes model behavior on the training dataset, making it difficult to draw general claims about other distributions. In particular, we cannot rigorously describe the generative capabilities, though Sec. 5 initiates an investigation into when off-policy behavior can describe on-policy behaviors.Assumption A.1 (Bradley-Terry [5] ). Given a prompt x and two possible continuations y 1 and y 2 , the ground truth human preference distribution satisfiesEQUATIONfor some ground truth reward model r * .A. ; π θ , π Ref ) ≤ 0.6, then R(x, y w , y l ) = 1.Proof. For notational convenience, write the log probability ratios asa x = log π θ (y w |x) π Ref (y w |x) , b x = log π θ (y l |x) π Ref (y l |x) .Then we can express the probability that the model places on each response asπ θ (y w |x) = π Ref (y w |x)e ax , π θ (y l |x) = π Ref (y l |x)e bx .Suppose for (x, y w , y l ), L DPO (x, y w , y l ; π θ , π Ref ) ≤ 0.6. Expanding the DPO loss, we haveL DPO (x, y w , y l ; π θ , π Ref ) = -log σ(β(a x -b x )) ≤ 0.6.Rearranging the inequality and exponentiate on both sides, we haveσ(β(a x -b x )) ≥ exp(-0.6).Note that σ(0) = 0.5 < exp(-0.6) ≤ σ(β(a x -b x )). Since the logistic function is monotonic, this implies that 0 ≤ β(a x -b x ) and a x ≥ b x .Writing out this last inequality,π θ (y w |x) π Ref (y w |x) ≥ π θ (y l |x) π Ref (y l |x) ⇒ π θ (y w |x) π θ (y l |x) ≥ π Ref (y w |x) π Ref (y l |x) ≥ 1,where the last inequality is by the assumption on π Ref . We conclude that R(x, y w , y l ) = 1 by definition.A.2 Proof of Theorem 3.1Theorem A.3 (Simulating Perfect RLHF). Fix a reference model π Ref and an aggregated preference datapoint (x, y w , y l ) ∼ D. Assume the dataset includes the ground-truth human preferences: that is, α(x, y w , y l ) = P(y w ≻ y l ), and that these preferences obey the Bradley-Terry model (Assumption A.1). Let π * be the (adequately expressive) model that perfectly minimizes the DPO objective on (x, y w , y l ), or perfectly maximizes the PPO objective on the optimal reward function as described in Section 2.1. Then, the optimal policy π * satisfiesπ * (y w | x) π * (y l | x) = π Ref (y w | x) π Ref (y l | x) α(x, y w , y l ) 1 -α(x, y w , y l ) 1/β (6)where β is a hyperparameter in the DPO and RLHF objectives.Proof. We first prove the statement for DPO. Following the notation from [41] , fix a reward function r, let π r (y|x) be the optimal model under the KL-constrained RL objective given the reward function r. We can express r(x, y) in terms of π r (y|x)and π ref (y|x):r(x, y) = β log π r (y|x) π ref (y|x) + β log Z(x),where Z(x) is the partition function for prompt x. Then, under the Bradley-Terry model, the probability of preferring y w over y l under the model π r isπ r (y w ≻ y l |x) = 1 1 + exp(r(x, y l ) -r(x, y w )) = 1 1 + exp(β log πr(y l |x) π ref (y l |x) -β log πr(yw|x) π ref (yw|x) ).Given the ground-truth human preferences, DPO's maximum likelihood objective minimizes the binary classification loss on (x, y w , y l ):min π α(x, y w , y l ) log π(y w ≻ y l |x) + (1 -α(x, y w , y l )) log(1 -π(y w ≻ y l |x)).Let π * denote an optimal policy from the loss above, the optimal preference probabilities satisfyπ * (y w ≻ y l |x) = α(x, y w , y l ),and the optimal policy in turn satisfies1 1 + exp(β log π * (y l |x) π ref (y l |x) -β log π * (yw|x) π ref (yw|x) ) = α(x, y w , y l ).Rearranging, we haveα(x, y l , y w ) α(x, y w , y l ) = exp β log π * (y l |x) π ref (y l |x) -β log π * (y w |x) π ref (y w |x) ,taking a log on both sides and divide by β, Now we show the result for RLHF, starting from the optimal solution of the KL-constrained reward maximization problem under the reward r ϕ , as derived in [41] :1 β log α(x, y l , y w ) α(x, y w , y l ) = log π * (y l |x)π ref (y w |x) π ref (y l |x)π * (y w |x) ,π * (y|x) ∝ π Ref (y|x) exp 1 β r ϕ (x, y) .The condition is straightforward to deriveEQUATIONEQUATIONwhere the last equality is due to the Bradley-Terry model.Corollary A.4 (Reproduces Corollary 3.3). Given a reference model π Ref , the DPO or RLHF hyperparameter β, a dataset of aggregated preferences D = {(x, y w , y l )} and their corresponding rater proportions α(x, y w , y l ), the ranking accuracy of the optimum of the RLHF or DPO objective π * is given byR * (D; π Ref ) = E (x,yw,y l )∼D 1 π Ref (y w | x) π Ref (y l | x) α(x, y w , y l ) 1 -α(x, y w , y l ) 1/β > 1 (10)where 1[•] is the indicator function. When computed on length-normalized likelihoods from π Ref , we denote the idealized ranking accuracy as R * .Proof. We can see from the definition of ranking accuracy (Definition 2.3) that the accuracy on a datapoint (x, y w , y l ) is 1 when π(y w | x) > π(y l | x). In other words, π(y w | x)/π(y l | x) > 1 ensures that the ranking accuracy R(x, y w , y l ) = 1. Theorem 3.1 shows a formula for the ratio of the optimal policies, and plugging this in to the condition for ranking accuracy immediately yields the given formula for R * .A.4 Proof of Theorem 4.1Theorem A.5. Consider an aggregated preference datapoint (x, y w , y l ) such that the reference model log-ratio is some constant c, i.e.log π Ref (y l |x) π Ref (y w |x) = c.Then, R(x, y w , y l ) = 1 if and only if L DPO (x, y w , y l ) ≤ -log σ(βc), where σ is the sigmoid function.Proof. Recall that we can break down the DPO loss into the model log-ratio and the reference model log-ratio as follows:L DPO (x, y w , y l ; π θ , π Ref ) = -log σ β log π θ (y w |x) π θ (y l |x) + log π Ref (y l |x) π Ref (y w |x) = -log σ β log π θ (y w |x) π θ (y l |x) + c (by assumption) Observe that if L DPO (x, y w , y l ; π θ , π Ref ) ≤ -log σ(βc), then log π θ (y w |x) π θ (y l |x) + c ≥ c,by the monotonicity of the log and sigmoid functions. This implies that π θ (yw|x) π θ (y l |x) ≥ 1 and R(x, y w , y l ) = 1. Now we show the other direction. Suppose R(x, y w , y l ) = 1, then π θ (y w |x) ≥ π θ (y l |x) andlog π θ (y w |x) π θ (y l |x) + c ≥ c.Using this relationship, we havelog σ β log π θ (y w |x) π θ (y l |x) + c ≥ log σ(βc),and the other direction follows by taking the negative of both sides.We extend our main result on DPO to also describe the IPO objective, formally defined below. Definition A.6 (Identity Preference Optimization, a.k.a. IPO [2] ). The Identity Preference Optimization (IPO) loss is defined asL IPO (x, y w , y l ) = log π θ (yw|x) πRef(yw|x) -log π θ (y l |x) πRef(y l |x) -1 2τ 2 .Proposition A.7. Under the IPO loss, iflog π Ref (y w |x) π Ref (y l |x) < - 1 2τ ,then zero IPO loss implies that R(x, y w , y l ) = 0. On the other hand, iflog π Ref (y w |x) π Ref (y l |x) = c ≥ - 1 2τ , then L IPO (x, y w , y l ) ≤ c + 1 2τ 2 guarantees that R(x, y w , y l ) = 1.Different from the DPO loss, the IPO loss is a regression loss where the optimal model log-ratio has a constant margin over the reference model log-ratio. This can be seen by the following decomposition of the IPO loss:L IPO (x, y w , y l ) = log π θ (y w |x) π θ (y l |x) -log π Ref (y w |x) π Ref (y l |x) - 1 2τ 2 .Clearly, the optimization target of the model log-ratio is the sum of the reference model log-ratio and the margin 1 2τ . From Figure 3 , we can see that the reference model can have a large bias towards the dispreferred completion, represented by the large negative values of the reference model log-ratio (note that in the figure, the reference model log-ratio has the dispreferred completion in the numerator). Therefore, for the optimal model under the IPO loss to have perfect ranking accuracy, the margin 1 2τ needs to be large enough to overcome this bias for all datapoints. Alternatively, a per-example margin dependent on the reference model log-ratio of the example can be used.Some datasets (e.g. UltraFeedback [7] ) contain examples with more than two responses per prompt x. In these cases, we extend our definition of ranking accuracy (Def. 2.3) to exact-match of the rankings over all choices. Suppose each aggregated datapoint consists of a prompt x and n responses (y 1 , • • • , y n ). Since the question of how to best aggregate rankings over multiple raters (when n > 2) is an open research question, we assume that there already exists some aggregated social ranking ρ(x,y 1 , • • • , y n ) = (ρ i (x, y 1 , • • • , y n )) n i=1 ∈ Π[n] where ρ i (x, y 1 , • • • , y n ) < ρ j (x, y 1 , • • • , y n )implies that y i is preferred over y j . Now let the ranking assigned by a policy π θ be ν(x,y 1 , • • • , y n ) = (ν i (x, y 1 , • • • , y n )) n i=1 ∈ Π[n], where π θ (y i |x) > π θ (y j |x) if and only ifν i (x, y 1 , • • • , y n ) < ν j (x, y 1 , • • • , y n ).Then the generalized ranking accuracy is defined as follows: Definition A.8 (Ranking Accuracy for n > 2).EQUATIONwhere 1[•] is the indicator function. Analogously, the ranking accuracy over a dataset D =EQUATIONB Experimental Details for Computing Ranking Accuracy of Open-Access LLMsWe evaluate ranking accuracy for a wide range of LLMs by evaluating the likelihoods that each model π θ assigns to y w given x and y l given x, as described in Def. 2.3. However, x, y w , and y l are sequences, so we compute the sequence likelihoods by factorizing the sequence likelihood into a product of the conditional token likelihoods, like so:EQUATIONWe use PyTorch and the Hugging Face transformers and datasets libraries to compute all ranking accuracies. For each dataset that we evaluate ranking accuracy on, we take a random sample of 1000 examples.Handling Ties In some datasets, such as the cross-human-annotated validation split of the Alpaca Farm dataset [9] (described in Sec. B.2), ties exist in the human annotations. When this is the case, we make a mild adjustment in the calculation of ranking accuracy to accommodate ties. Definition B.1 (Ranking Accuracy with Ties).EQUATIONwhere [•] is the indicator function. In other words, if the human annotations indicate tied preferences between y 1 and y 2 , then π θ achieves the correct ranking if and only if it assigns y 1 and y 2 approximately the same likelihood (within some tolerance level ϵ). For all other cases, the R Ties is equivalent to R.Throughout this paper, we use a tolerance ϵ of 0.01. In Fig. 1 we provide ranking accuracies only for examples without ties, but we provide the numbers on the full datasets below.Length Normalization To compute the length-normalized ranking accuracy ( R), we replace π θ (y|x) in Defs. 2.3 and A.8 with π θ (y|x), whereEQUATION• HH-RLHF [3] (helpful set) consists of two model responses for each query (often the history of a multi-turn conversation between human and chatbot), based on generations from three different classes of models (context-distilled 52B model, same model with rejection sampling using a reward model, RLHF-finetuned models). Queries and preferences annotations are obtained from crowdworkers. • Synthetic Instruct GPT-J Pairwise [1] is a synthetic dataset of queries and pairwise model generations spanning different subjects. • StackExchange Preferences [28] consists of questions and answers from Stack Exchange, where within a pair of answers, the preference between two answers is determined by a function of the number of upvotes and whether the answer was selected. • UltraFeedback [7] consists of model-generated responses from four different language models out of a larger set of models (meaning a different set of models is considered for different samples). Queries are obtained by a mixture of existing QA datasets, and the preference is annotated by GPT-4. • Stanford Human Preferences [11] is a dataset created from Reddit posts across different subject matters, where within a pair a response is considered preferred to another if was created later and has more upvotes. • Alpaca Farm Validation [9] is sourced from the Alpaca Eval dataset, but with new splits repurposed for training preference-tuned models. Additionally, we choose to use the validation split because it contains human cross-annotations (i.e. multiple human ratings per triple of (x, y 1 , y 2 )).This particular split can be found at https://huggingface.co/datasets/tatsu-lab/alpaca_eval/ blob/main/alpaca_farm_human_crossannotations.json. The original dataset, Alpaca Eval [30] , is a mixture of several test sets including Open Assistant [26] , a dataset of human-constructed chatbot conversation turns, HH-RLHF [3] , and the Vicuna [70] and Koala test sets [15] , where the former consists of user-shared queries from ShareGPT, and the latter consists of human queries from online interactions.We give the full set of length-normalized and non-length-normalized ranking accuracies for 16 openaccess LLMs across six datasets in Tables 2, 3, and 4. For the UltraFeedback [7] and StackExchange Preferences [28] datasets, we use the generalized definition of ranking accuracy for n > 2 outputs instead (Def. A.8). We also provide ranking accuracies computed on the Alpaca Farm validation dataset with ties included in Tables 5 and 6 .Table 2 : Length-normalized and non-length-normalized ranking accuracies for the Anthropic Helpful and Harmless (HH-RLHF; Bai et al. [3] ) and Synthetic Instruct GPT-J Pairwise [1] datasets. The latter contains only a training split.Anthropic HH-RLHF Synthetic Instruct GPT-J Pairwise α(x, y w , y l ) of raters who preferred y w over y l , and using α(x, y w , y l ) = 1 results in errors. As a result, we measure the alignment gap on the Alpaca Farm validation split [9] , which contains individual votes for each triple. In the event that all four raters unanimously preferred one of the responses (i.e. α(x, y l , y w ) = 0), we add a small constant ϵ = 0.001 to α(x, y l , y w ) to prevent divison by zero. Second, the formula depends on the choice of β, which we do not know for many closed proprietary models. We circumvent this issue by computing the quantity for a range of β values (β ∈ {0.01, 0.1, 1, 5, 10}) and reporting the minimum, median, and maximum. -SFT: learning rate ∈ {1e-7, 1e-6, 1e-5}, batch size ∈ {32, 64} -DPO: learning rate ∈ {1e-6, 1e-7}, batch size ∈ {32, 64}, β ∈ {0.1}Test Train Train R R R R R R GEMMA-7B-ITWe tuned the hyperparameters on a single seed, and carried over the best hyperparameters to the other seeds of the same model class. We trained the GPT2 and Pythia2.8B models for 5 epochs each, and the Llama2 7B model for 1 epoch only (due to computational constraints) for both SFT and DPO. However, most seeds of the GPT2 and Pythia 2.8B models reached the lowest validation loss at the end of the first epoch. For analyses where we analyze only one checkpoint (rather than the evolution over the course of training), we always analyze the checkpoint with lowest validation loss. We use the AdamW optimizer (with β 1 = 0.9, β 2 = 0.999, ϵ =1e-8) for SFT and the RMSProp optimizer (with α = 0.99, weight decay = 0, momentum = 0, ϵ =1e-8) for DPO.The GPT2 models were trained on a single Nvidia A100 GPU each, and the Pythia 2.8B and Llama 2 7B models were trained on two Nvidia A100 GPUs each. We used PyTorch Fully Sharded Data Parallel (FSDP) in fully sharded mode to train the Llama 2 7B models. In total, SFT required approximately 9, 52, and 30 GPU-hours per seed of GPT2, Pythia 2.8B, and Llama 2 7B, respectively. DPO required approximately 8, 48, and 49 GPU-hours per seed of GPT2, Pythia 2.8B, and Llama 2 7B, respectively. (Longer training times were required for Pythia 2.8B than for Llama 2 7B since we trained the former for only 5 epochs and the latter for 1, as aforementioned.)We provide the DPO loss, reward margin, and dataset trends during training across all 9 models (three seeds each of GPT2 [40] , Pythia 2.8B [4] , and Llama 2 8B [52] ) in Figs. 5, 6 , and 7. We include additional plots from our exploration of the relationship of ranking accuracy and win rate in Figs. 9 and 10.We also provide results computed on the test set in Figs. 11 and 12 .Theorem 4.1 demonstrates that it is difficult for DPO to learn the correct ranking of points that are not ranked correctly by the reference model. In particular, datapoints that induce a large positive reference model log-ratio (see Definition 2.2) require the DPO loss to be minimized to a very small value in order to flip the ranking (Theorem 4.1 and Figure 3 ).Here, we document a few of the datapoints that induce a large positive reference model log-ratio and are hard to learn (Table 7 ), as well as datapoints that induce a very negative reference model log-ratio and are easy to learn (Table 8 ). These datapoints are measured using the Pythia-2.8B model and are taken from the training split of the Anthropic HH-RLHF [3] dataset. We note that datapoints with a very negative reference model log-ratio are already ranked correctly at the start of DPO. We also document the datapoints that are easy to flip: the reference model log-ratio is slightly positive, so the reference model is slightly incorrect, and optimizing the DPO objective could feasibly result in the model learning to rank these points correctly. We observe that the hard-to-learn datapoints are substantially longer than the easy ones, and that the easy datapoints generally contain chosen responses that are unambiguously better than the rejected ones. Table 7 : Preference datapoints that are hard to learn. We print several datapoints from the training split of Anthropic HH-RLHF [3] that are hard for the Pythia-2.8B model to learn. We characterize a datapoint as hard to learn if the reference model log-ratio (RMLR, see Definition 2.2) is large. This characterization follows Theorem 4.1 and Figure 3 . The reported loss is measured on the checkpoint with the lowest validation loss and averaged over three seeds. Chosen response: There are many fun toys you can play with in the tub with your kids! Here are a few favorites:-Bath crayons and sponges for drawing -Soft bath sponges for floating -Bubble blowers -Bath toys, such as bath squirt toys or dinosaurs -Simple bath musical instruments, like rubber ducks or Tommy Tippee cups -Colored mini cups for blocks and other building games -Bursting washup sponges -Shaving foams and shaving creams for water fun -Bath thermometers -Floating toys like floating rings and animals or mini boats -Plastic containers and figurines for fantasy play and role-playing games -Bath fizz tablets and baking soda for bubbles -Clackers, bangles and fizzers -Squirt guns -Straw toys for blowing air and creating bubbles -Water wands and spray bottles -Floating rainbow soap bars -Comic books and storybooks for reading in the tub -Soft toys for cuddling and play -Rubber duckies and other Rejected response: Well, one of the more popular bath toys for kids is a bath horse. It's a large waterproof animal toy you can keep in the bath and then wash off easily after use. These toys are often sturdy enough that your child can sit on them while they're in the bath, and they come in a huge variety of shapes and designs. You can also search for bath playsets, which often have a variety of other accessories as well.RMLR L DPO Prompt, Chosen Response, and Rejected Response 418.7 0.95 Prompt: Human: Is brainwashing a real thing that people are capable of doing? Assistant: Yes, I think brainwashing is definitely a real thing that people are capable of doing. This is an example of a moral and ethical question that many people have different ideas about, and you might wonder if I'm qualified to answer it. So how about we first do a quick thought experiment? Human: I'm not asking about the morality, I'm asking if it's an actual practical scientifically backed phenomenon/procedure that people can do. Are there any real world examples of brainwashing? Assistant:Chosen response: Brainwashing is actually a real and fairly common phenomenon, and there are many examples of people using these techniques for coercive and unethical means. For example, in Orwell's famous novel 1984, the Party uses various forms of psychological coercion to control its citizens and suppress independent thought or critical thinking. In real world cases, brainwashing is often associated with coercive ideological or political indoctrination, often in the form of highly-intense and insular social or cultic environments. Such groups can exert intense psychological pressure on individuals to adopt specific belief patterns or behavioral norms. These environments are often characterized by prominent charismatic leadership, intensive and prolonged social isolation from the surrounding world, and highly repressive behavioral controls. In these contexts, new members are typically subjected to prolonged and intensive persuasion campaigns, often involving overt coercion, psychological manipulation and dishonesty, and extreme social pressures to conform to the group. Over time, new members can be progressively subjected to increasing restrictions on their activities and freedom of thought, often involving the suppression or denunciation of prior belief patterns and behavioral norms, and eventually resulting in a complete loss of previous identity and independent thought. Rejected response: Oh, I think I understand. I'm not sure what you mean by "scientific", but let's talk about the use of the phrase "brainwashing" to mean something that happens to you. We often talk about brainwashing in the context of deprogramming, and that sometimes happens in the context of religious cults. I can tell you more about that. Chosen response: I would say it's really critical that they be organized and detail-oriented. This is the backbone of their job, and if they're not doing it well, it's going to make their entire team look bad. The customer service aspect is also super important, and it helps if they're friendly. But if they're not organized, it's hard for them to provide good service.Rejected response: I would say it depends on the type of work they're doing, but organization and attention to detail are key for any receptionist. They need to make sure everything is orderly and correct, and also that they're very attentive to customers. They need to make sure everyone who calls in gets exactly what they need.1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: Each claim in the abstract and introduction maps directly to a proposition, theorem, corollary, and/or figure containing empirical data. The formal assumptions required for the results are summarized in natural language in the abstract and introduction. The results contain additional commentary about how the analysis can be extended to other related algorithms and how measurements can be taken for new datasets. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Not needed. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification:We cite all open-access models and datasets used in the paper. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: We did not release any new assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. 38th Conference on Neural Information Processing Systems (NeurIPS 2024).In the limit, when there are infinite raters, the empirical proportion α(x, y1, y2) converges to the ground truth preference P[y1 ≻ y2 | x].It is not surprising that fine-tuning on the preferred completions does not boost ranking accuracy, since the model does not receive any knowledge of the relative qualities of the preferred and rejected completions.This result differs from Proposition 2.6 in that it accounts for a potentially imperfect reference model.We note that this result can also be straightforwardly derived from prior works[38,27,16].Note that when α(x, yw, y l ) = 1, we replace it with 1 -ϵ to compute the formula. See App. B.4.We use the best hyperparameters obtained from the experiments in Sec. 4.https://huggingface.co/docs/trl/en/index 
 
 Appendix:  


Decision: This paper studies preference learning with RLHF and DPO for LLMs.  It highlights some enlightening empirical findings, such as the existing preference learning algorithms obtain low ranking accuracies, the idealized policy and these methods' policies have a significant ranking accuracy gap.  They show that when the model is close to reference the win rate and preference accuracy are strongly correlated.  The reviewers praise the extensive empirical comparisons and how well their claims are supported by evidence.  They also praise the theoretical insights that can explain where these method fail.  The authors clearly rebut criticisms such as why they prefer ranking accuracy over reward accuracy, and they provide results for soft-ranking accuracy in addition to the hard ranking accuracy.

Reviews: review: paper_summary: paper_summary: The paper empirically highlights a few potential flaws in RLHF and DPO that prevent preference-tuned models from achieving high ranking accuracy. It presents a collection of empirical and theoretical findings:
+ Existing preference-tuned models achieve low ranking accuracies.
+ The idealized policy and the DPO/RLHF policy has a significant gap in their ranking accuracy.
+ Preference learning algorithms such as DPO rarely flip incorrect rankings throughout the training process.
+ Ranking accuracy and win rate are positively correlated when models are close to the reference model.
strength_weakness: strength_weakness: 1. While it is already argued in the paper that ranking accuracy is practically more meaningful than the reward accuracy, it is still questionable whether totally discarding $\pi_{ref}$ is reasonable. Specifically, the paper should have included experiment results on whether there is also a gap in reward accuracy between the learnt and ideal policy. After all, the DPO and RLHF algorithm explicitely maintain the reference model $\pi_{ref}$, and the implicit reward should be indicative on human preference. 

2. Due to the small size of preference labels for any given response pair (1 for most dataset, and 4 for the Alpaca Farm as stated in Sec C.4), it is more reasonable to consider the soft ranking accuracy instead of the hard ranking accuracy. The authors could have reported both or discuss on whether hard ranking accuracy is more meaningful.
questions: questions: 1. One concern is that the ranking accuracy considered here is the hard accuracy (accuracy of the Bayesian optimal classifier). I wonder if the "soft" ranking accuracy will exhibit more noticable improvement after preference fine-tuning. The soft ranking accuracy is $\frac{\pi(y_w)}{\pi(y_w) + \pi(y_l)}$ instead of hard thresholding the predicted accuracy.

2. It is not clear to me what is the main failing part in DPO/RLHF training for high ranking accuracy. In Theorem 3.1, it seems that the perfect RLHF policy should encode the information from $\pi_{ref}$ and the dataset perfectly at the same time. Does the DPO/RLHF model fail to capture $\pi_{ref}$ or the preference dataset?
Have the authors tried to plot the quantity $\frac{\pi_{DPO}(y_w)\pi_{ref}(y_l)}{\pi_{DPO}(y_w)\pi_{ref}(y_l)}$ and compare it to $(\frac{\alpha}{1-\alpha})^{\beta}$ (perhaps a scatter plot)? This can illustrate more clearly what DPO/RLHF failed in reaching the perfect policy.
limitations: limitations: N/A
paper_summary: paper_summary: This paper studies the preference learning algorithms like DPO in Large Language Models (LLMs). It focuses on the ranking accuracy and shows that existing algorithms fail to achieve high ranking accuracy. Then, the paper further proposes an idealized ranking accuracy and finds that there is an alignment gap.
strength_weakness: strength_weakness: - The motivation of studying the rank accuracy is weak and not justified well. 

- The analysis and results are somewhat superficial.
questions: questions: The contributions, compared with similar works, are not clear for reviewers to give an acceptance recommendation.

[1] Xu, Shusheng, et al. "Is DPO superior to PPO for LLM alignment? A comprehensive study." arXiv preprint arXiv:2404.10719 (2024).

[2] Tajwar, Fahim, et al. "Preference fine-tuning of LLMs should leverage suboptimal, on-policy data." arXiv preprint arXiv:2404.14367 (2024).

[3] Tang, Yunhao, et al. "Understanding the performance gap between online and offline alignment algorithms." arXiv preprint arXiv:2405.08448 (2024).

The analysis of PPO in terms of ranking accuracy is missing, which somehow makes the conclusion regarding on-policy and off-policy weak.
limitations: limitations: - The motivation for studying ranking accuracy is weak. The ranking accuracy, especially when defined on the offline dataset, usually does not correlate with the quality of generation behaviors. Let us split the response set into two parts, R_good and R_bad. In practice, we care that sum\_{y} P( y in R_good) > sum\_{y'} P(y' in R_bad). The proposed ranking accuracy, instead cares about P(y in R_good) > P(y' in R_bad), where y and y' are two events from R_good and R_bad, respectively.  Thus, a good generative model can provide many good responses with high probability and, at the same time, it may have a low ranking accuracy. This says that ranking accuracy measures the worst-case behavior of generative models. Thus, we do not need to bother ourselves to care about the ranking accuracy, especially for a small dataset where only a few pairs of preference are given. 
- The analysis in terms of ranking accuracy is somehow superficial. In fact, it is the algorithmic bias that DPO introduces the KL-regularized optimization, and as such, it cannot recover the preference-generating distribution. Briefly, the BTL assumption does not invovle the KL term. See the recent work in [1] for more discussion (since this work was posted after NeurIPS submission deadline, it does not affect the review decision). This can also be reflected in equation (2): assume that pi_REF is a uniform policy so its effect can be eliminated and the parameter beta = 1, we see that it recovers the preference-generating policy in terms of alpha. The BTL model assumes that alpha + 1 - alpha = 1, however it may not be true that pi*(y\_w|x) + pi*(y\_l|x) = 1.  The procedure I mentioned is also reflected in equation (4), where the authors introduce a new loss. It in fact corresponds to add a entropy regularization to eliminate the bias of pi\_ref.

Based on the considerations mentioned above, the reviewer does not clearly perceive the technical depth and the contributions of this paper.

[1] Xiao, Jiancong, et al. "On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization." *arXiv preprint arXiv:2405.16455* (2024).
paper_summary: paper_summary: The paper investigates the effectiveness of preference learning algorithms, RLHF and DPO, in training LLMs to rank human-preferred outputs higher. It challenges the assumption that these algorithms can successfully teach models to rank outputs according to human preferences. Empirical findings show that most state-of-the-art models achieve low ranking accuracy, often below 60% on standard datasets. The paper identifies a significant alignment gap due to the limitations of the DPO objective and provides a theoretical analysis of the difficulty in correcting ranking errors. It concludes by highlighting the need for a reevaluation of these algorithms and calls for more nuanced analyses of preference training dynamics.
strength_weakness: strength_weakness: 1. While the paper suggests a link between preference learning and win rate, it would be strengthened by a more explicit exploration of how these metrics interrelate. Specifically, the authors could benefit from a deeper analysis of the implications of using ranking accuracy versus win rate as measures of model performance, including potential discrepancies and how they might be reconciled.

2. The paper identifies a significant determinism of ranking accuracy by reference model likelihoods, which is a crucial observation. However, it would be advantageous for the authors to expand on this point by discussing potential techniques or algorithmic adjustments that could alleviate the heavy reliance on the reference model. This would bolster the paper's contribution by addressing a key challenge in the field of preference learning for LLMs.
questions: questions: See Weaknesses.
limitations: limitations: Yes, the authors fully address the limitations and potential social implications of their work.
paper_summary: paper_summary: This paper investigates the effectiveness of preference learning algorithms, include the RLHF (PPO-based) and DPO in training LLMs. In particular, the authors study the non-regularized ranking accuracy, which is largely ignored in the previous RLHF literature. The authors approach this investigation through both theoretical analysis and empirical experiments and show that whether the ranking is correct or not  highly depends on the initial reference policy.
strength_weakness: strength_weakness: My major concern is that this paper only studies the goal of helpfulness in chat style. 

First, while it is fair to say that DPO is not good at optimizing the ranking policy in terms of the non-regularized target, it is not immediately clear that the non-regularized ranking policy is a better metric. For instance, a concurrent work (so this should not hurt the novelty of this paper) [1] proposes simpo with many shared insights. In particular, they also notice the mismatch between the ranking policy and the target of DPO. However, in their experiments, as the win rates evaluated on the standard benchmark significantly increase, we also observe a significant drop in some academic benchmarks, particularly those on the mathematical problems solving. 

Based on my personal experience, if we adopt a smaller KL coefficient (e.g., 0.01, so less regularization), the final model performance is worse than the model with a moderate one (say 0.1). This is particularly true for the online iterative learning, which we are more interested in as the authors also notice that these variants outperform the offline one with a large margin. In most of the cases, the ability of maximizing the reward/ranking accuracy, while maintaining a moderate KL divergence is more suitable to evaluate the algorithm performance in my viewpoint. Therefore, the ranking accuracy alone is also not a good learning target, which is also evidenced by figure 4 of this work.

Based on the above two points, I feel that the paper can still be largely improved if a more comprehensive evaluations can be done, like the trade-off between the performance on instruction-following benchmarks (alpaca-eval, mt-bench, gpt4 win rate) and other  academic benchmarks. But overall, I support the acceptance.

[1] SimPO: Simple Preference Optimization with a Reference-Free Reward
questions: questions: see weakness
limitations: limitations: yes

Comments: title: Thanks for the clarification!
Thanks for the responses
comment: I appreciate the authors' detailed responses and have reviewed both their individual and global responses. I maintain my current stance and recommend acceptance.
Thanks for the responses, and I am keeping my overall score at 6.
**“My main point is that I do not think dropping the reference is not a good idea.”**

Thank you for the clarification! We agree with you on this point, and our paper does not promote dropping $\pi_\text{Ref}$ altogether without other interventions to prevent generation degradation. (SimPO, for example, uses a target reward margin, on-policy training data, and length normalization *in addition* to removing $\pi_\text{Ref}$.) Our experiments with $L_\text{DPO}^\gamma$ in Section 5 also explore the concept of modifying the strength of the $\pi_\text{Ref}$ terms rather than completely removing them. We show in Figure 10b that some values of $\gamma$ offer an improved tradeoff of ranking accuracy vs. win rate when compared to DPO (DPO results are in Figure 10a, and $L_\text{DPO}^\gamma$ results are in Figure 10b).
The ``non-regularized ranking policy'' refers to equation (1), which does not consist of the regularization of the reference model in the original DPO reward. 

My main point is that I do not think dropping the reference is a good idea. This is because even with the strong KL regularization, the DPO algorithm (and its variants) already suffers from considerable alignment tax. Therefore, chasing for a high ranking policy is too aggressive in my viewpoint, because it typically leads to performance regression in the out-of-distribution tasks. These have been widely observed in almost all the practices with DPO algorithm and has been mentioned in many original technical report of PPO-based framework (e..g instruct-gpt), as well as the more recent kto and simpo [1].


[1] The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism