Preference learning made easy: Everything should be understood through win rate
Authors: Lily H Zhang
Rajesh Ranganath
Venue: ICML 2025 poster (2025)
Keywords: preference learning
alignment
RLHF
Primary area: social_aspects->alignment
TL;DR: The paper motivates and develops a win rate-centric framework to understand preference learning, with takeaways for current practice and future research.

Abstract: Preference learning, or the task of aligning generative models to preference comparison data, has yet to reach the conceptual maturity of classification, density estimation, etc. To close this gap, this work presents a framework to understand preference learning starting from the sampling distribution of pairwise preference data. First, we prove that the only evaluation of a generative model that respects both preferences and prevalences in the data distribution is a form of win rate, justifying win rate as the focal point to understand preference learning. We then analyze preference learning methods as win rate optimization (WRO) or non-WRO. We present novel instances of WRO beyond existing examples (RLHF, NLHF) and identify two key theoretical benefits of all such methods. We prove that common non-WRO methods like DPO and SFT on preferred samples lack these properties and suggest ways to mitigate such theoretical limitations. We also show that WRO underperforms in practice due optimization difficulties and that optimization success predicts performance better than choices which affect the objective's solution. Our analysis highlights best practices for existing methods and provides recommendations for future research, guided by the principle that one should either align non-WRO methods more closely with WRO or improve the optimization of WRO objectives.

Preference learning made easy:
Everything should be understood through win rate
Lily H. Zhang'
Rajesh Ranganath
12
Abstract
ment learning from human feedback (RLHF) (Christiano et al., 2017) on state-of-the-at and high-profile language models such as GPT-4 (OpenAI, 2024). The goal of learning from preference data to finetune powerful base language models to output generations more in line with human pref- erences (Stiennon al.. 2020; Ouyang et al., 2022), moti- vated by the fact that pretraining internet-scale data has enabled large language models to exhibit fluent generations of text (Minaee et al., 2024) but not necessarily responses aligned with what humans prefer to see.
Preference learning, or the task of aligning genera- tive models to preference comparison data, has yet to reach the conceptual maturity of classification, density estimation, etc. To close this gap, this work presents framework understand prefer- ence learning starting from the sampling distribu- tion of pairwise preference data. First, we prove that the only evaluation of a generative model that respects both preferences and prevalences the data distribution is a form of win rate, justifying win rate as the focal point to understand prefer- ence learning. We then analyze preference learn- ing methods win rate optimization (WRO) non-WRO. We present novel instances of WRO beyond existing examples (RLHF, NI.FF) and identify two key theoretical benefits of all such methods. We prove that common non- OORO meth- ods like DPO and SFT on preferred samples lack these properties and suggest ways to mitigate such theoretical limitations. We also show that WRO underperforms in practice due optimization dif- ficulties and that optimization success predicts performance better than choices which affect the objective's ' solution. Our analysis highlights best practices for existing methods and provides rec- ommendations for future research, guided by the principle that one should either align non-WRO methods more closely with WRO or improve the optimization of WRO objectives.
In recent years, the landscape algorithms and evaluations for preference learning has grown significantly (Kaufmann et al., 2023; Jiang et al., 2024), resulting a complex and often fragmented field that can pose challenges for practi- tioners and researchers deciding where to focus their efforts. Whereas well-established machine learning tasks such as classification are grounded in principles such as maximum likelihood, preference learning lacks such a unifying con- ceptual foundation. To advance the field, we thus first ask: what underlying principles underpin preference learning?
We address this question by developing framework start- ing from the sampling distribution implied by pairwise pref- erence data. We first tackle the question of evaluating a model under the preference learning paradigm. We show that the only evaluation of generative model rooted in the preference data sampling distribution itself form of win rate we call h-win rate; any other notions of good either do not evaluate the model respect the preference data, are based on assumptions outside of either (Sec- tion 3). From this insight, we introduce a win rate- cen-rcet framework for understanding the landscape of preference learning methods. Given h-win rate is the only relevant evaluation without additional assumptions, we relate com- mon preference learning algorithms directly optimizing for win rate, dividing the preference learning space into win rate optimization (WRO) and nRR objectives. In Section 4, we generalize the space of WRO beyond exist- ing methods in the literature and identify two theoretical benefits of such methods: (1) win rate-correspondeece, i.e., optimizing the objective corresponds to optimizing for win rate: and (2) win rate-consistency. i.c., the solution can achieve the maximum win rate possible over a competitor
1. Introduction
Learning from preference data, often referred as human feedback, has emerged as a key step in training large lan- guage models, particularly given the success of reinforce-
'Center for Data Science, New York University, New York, USA *Courant Institute, New York University, New York, USA. Correspondence to: Lily H. Zhang <lily.h.zhang& nyu.edu>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Preference learning made casy: Everything should be understood through win rate
as regularization strength goes zero. In Section 5, we show that direct preference optimization (DPO) and other direct alignment algorithms (Rafailov et al., 2024a) fail win rate- coreresrrse,eeece, and that supervised finetuning (SFT) on preferred samples additionally fails win rate-consistency. In Section 6, we show that despite their theoretical benefits, WRO methods underperform relative to expectations due to difficulties in optimization, which has a much more signif- icant effect performance than other design choices that distinguish WRO methods from each other. We conclude with takeaways for current practice and future research in preference learning (Section7) 7).
(WRO) methods beyond the specific instances that currently exist in the literature. Moreover, our analysis points the theoretical merit of these approaches, while our experiments demonstrate the implementation bottlenecks of them.
Analyzing RLHE, DPO, and other preference learn- ing methods. Our work is related to work that seeks to better understand RLHF, DPO, and other existing meth- ods in preference learning (e.g., best-of-n). Examples in- clude benchmarking generalization and diversity (Kirk et al., 2024), comparing on- vs off-policy approaches (Tajwar et al., 2024), investigating length bias (Singhal et al., 2024), and disentangling design choices empirically (Ivison et al., 2024). For RLHF, existing works consider the complexity of proximal policy optimization (Ahmadian et al., 2024), vanishing gradients (Razin et al., 2024b), reward model overoptimization (Zhu et al., 2024), or limitations of the Bradley-Terry assumption to relate preferences to rewards (Wang et al., 2024b; Azar et al., 2023; Munos et al., 2023; Siththaranjan et al., 2024). For DPO, there exists not only large space of alternative direct alignment algorithms (e.g.. (Zhao et al., 2023; Azar al., 2023; Xu al., 2024a; Huang et al., 2024; Pal et al., 2024; Xu et al., 2024b)) but also works which analyze its limited ability to flip rankings (Chen et al., 2024), the decrease in chosen and rejected log probabilities (Razin et al., 2024a), and its limitations under heteroge- neous preferences (Shirali et al., 2025). Gui et al. (2024) show that the target distribution of best-of-n for different choices of n yields win rate close to that of RLHF's for different choices of divergence constraint; Theorem 5.1 of this work generalizes their win rate result for oest-of-2 from a deterministic to an arbitrary preference environment. Azar et al. (2023) present a family of y V-Preference Optimization objectives, show that RLHF falls within this family, and propose Identity Preference Op- timization (IPO); this work presents larger family of WRO objectives beyond V-Preference Optimization and catego- rizes V-Preference Optimization and IPO separately into WRO and non-WRO objectives, highlighting theoretical benefits of the former and limitations of the latter.
Our contributions can be summarized as follows:
1. We prove that the only evaluation of a generative model grounded in the preference data distribution is win rate. This result justifies using win rate as the focal point to understand the landscape of preference learning.
2. We present a win rate-centric framework to understand preference learning. Under this framework, we:
(a) introduce win rafe-correspoondence and consis- tency and show that WRO methods satisfy both while non-WRO do not;
(b) highlight the central role of optimization success for the performance;
(c) motivate best practices for existing methods (e.g., win rate checkpointing with direct alignment al- gorithms, multiple seeds with WRO algorithms, inducing diversity for SFT, combining different approaches); and
KL divergence curve
(d) provide recommendations for future research- namely, to focus on better surrogates for or im- proved optimization of WRO.
2. Related Work
Our work is most closely related to previous work in win rate evaluation and optimization as well as analysis of RLHF and preference learning objectives.
3. What evaluations make sense for preference learning?
Win rate evaluation and optimization. Win rate is already a central evaluation in preference learning (Li et al., 2023b: Zheng et al., 2024); however, our work goes further and proves that it is the only evaluation grounded in the sam- pling distribution itself, thus motivating its use as the central object to understand the rest of the preference learning land- scape. Several works have proposed methods that perform some form of win rate optimization (Munos et al., 2023; Swamy et al., 2024; Rosset et al., 2024), including as a minimax optimization with a dynamic (rather than fixed) opponent model. Our work provides a more general frame- work to characterize the landscape of win rate optimization
In this section, we analyze how to evaluate a generative model under the preference learning landscape. Various evaluations have been used in the preference learning litera- ture, from win rate (Li et al., 2023b) and reward (Ahmadian et al., 2024) implicit reward accuracy (Rafailov et al., 2024a) and ranking accuracy (Chen et al., 2024). Here, we critically assess what evaluations are meaningful under the pairwise preference comparisons data itself.
We first describe the underlying sampling distribution of the pariwise comparison data used in preference learning (Sec-
2

---

Preference learning made casy: Everything should be understood through win rate
tion 3. Then we define and motivate two important condi- tions for evaluation function, weefrrence-onsiitense and prevalence-consistennyy and prove that the only evaluations that satisfy both conditions is form of win rate we call h-win rate (Section 3.2). This result justifies win rate as the singular focus in preference learning.
scalar: Øpix, x)) (p(y x)) R.! Intuitively, o should
respect properties of the preference sampling distribution. We formalize criterion for & below.
Definition 3.2. Any evaluation function ø is grounded in given preference distribution if
1. (ø is preference-consistemee given a strictly increas-
3.1. The Sampling Distribution
ing function h, then for any singleton query environ-
The goal of preference learning is to learn a generative model that performs well in a given context. However, whereas typical maximum likelihood training employs sam- ples from the distribution of interest, the setup of preference learning does not: only samples from generative model competitors (sometimes the same model) and their relative preference within pair are available.
ment p(x) = 1 |x x'), generative model p(y x) 1 [y = y'), anchor distribution p(yo x) = Iyo = Yól:
<(p(y x). = h- p(e x yo. y) and
2. (Ø is prevalence-consissen))) for a, b 2 and a a b 1:
(a) for generator p(y x) ap1 (y x) + bp (y |x):
The sampling distribution for preferences consists of input x, candidate outputs Yu and y1. and a label e {0, 1 denoting which of Yo or y1 is preferred. Let e = 1 denote that yı is preferred, and = 0 denote that yo is preferred. Then, the overall sampling distribution can be defined as follows:
<(ply| x), = aplp) (y x) + (bb (y) xx)
(b) for query distribution p(x) api (x) + 6p2(x), letting & = (p:(x). p(e = x.
o(p(y x)) = ap(p(y Ex) + be(p(s xx), E2);
Definition 3.1. A sampling distribution for (pairwise) preference learning is a distribution over input x E X, candidate outputs Yo. E V, and preference label é E {0, 1) defined by:
(c) for anchor distribution p(vo xx = apı (Yo x) + (y2 x)
@ptr. (o6oo1)((Pt) (y) ()) =a0p1so (x) (ply xx)E)
1. Query distribution: p(x)
+ boppaloo (p) xx), 8).
2. Generation competitor 0: p(yol Ix)
wreference-consistene ensures that the evaluation can be reduced to an increasing function of the preference classi- fier the base case where query, model, and anchor dis- tributions are all a single deterministic value. The trans- formation function h dictates that the evaluation need not exactly equal the preference probability. just that the eval- uation should be higher when the preference probability is higher. Prevalence-consistenee
3. Generation competitor 1: p(y1 |x)
4. Preference classifier: p(( x. Yo- Y1)-
1, 2, and 3 are user-specified distributions; 1 denotes the inputs of interest, and and 3 are the candidate competitors one chooses evaluate. 4 is only distribution that cannot be directly specified; rather, it is defined by the environment in which the user chooses to collect the preferences.
ensures that the contribution of p(E = 1 x, yo for a given (x, Yo- Y1) gets weighted appropriately by its prevalence in the query environment, anchor distribution, and generative model, respectively.
Generation competitor 0 and 1 can be the same distribution and often are in existing open-source preference datasets (Lambert et al., 2023; Ethayarajh et al., 2022).
What are the consequences if these properties not met? If preference-consistee assigned high value under & even if it only outputs dis- preferred response according to the ground truth preference. For prevalence-consisteneyy consistency, if 2a is not satisfied, then o weights preferences according to a distribution that is not how the model generates; if 2b is not satisfied, then & does not reflect the chosen queries; and if 2c is not satisfied, then & does not compare to the chosen anchor.
is not satisfied, a model can be
3.2. Win Rate is the only evaluation that can matter
The goal of preference learning is to learn some generative model p"(5 (y x) that performs well under the preference envi- ronment for given query distribution (we refer to this as the query-preference environment). Learning such generative model requires a definition of what is good.
The only evaluation that satisfies Definition 3.2 is form of win rate:
Consider an evaluation function & which maps a gener- ative model p(y x). query-preference environment E = ((() x, No. ))), and anchor distribution p(yo |x) to
We will optionally write Optr (x) when the anchor is clear from context.
3

---

Preference learning made easy: Everything should be understood through win rate
Proposition 3.3. For p with discrete support, & is grounded. as defined in Definition 3.2, if and only if
4. Preference learning through the Lens of Win Rate Optimization
Optolx) (p) (x).
In this section, we characterize the space of objectives which directly optimize for win rate, discussing both Win Rate Optimization (WRO) objectives (Section 4.1) and games (Section 4.2). Then, we identify the theoretical benefits shared by all methods in the WRO family (Section 4.3).
Pp) (x) Eps. x)) p(e= 1xx,oo)))
(I)
for choice of strictly increasing function h.
4.1. Win Rate Optimization Objectives
See Appendix B for proof. Equation (1) is the win rate of the generator ply x) against an anchor distribution p(yu x) un- der the sneery-preeerence environment &, for some choice of order-preserving transformation function h. When h is the identity, we have vanilla win rate, which we will refer to as win rate or Win Ratepís, v) ( x)]- Often it is estimated via samples from the preference classifier, i.c., Ep where t ~ p(& x, Y) (Li et al., 2023b). When h any other strictly increasing function, we have an h-variant of win rate, which we will refer to as h- Win Rate.
The fact that Equation (1) is the only grounded evaluation immediately provides an objective (0) to optimize:
max (+) = max h- Win Rate,yrr wylpoyy [x]]
max pp(xpoo 1x)p6 10) p(l No: (2)
WRO can be optimized by using score function/policy gradi- ents (Mohamed et al., 2020a; Weng, 2018). The solution to any objective in this family the generator that maximizes the h-w rate over a given anchor.
Different h-variants can stretch preference probabilities in some regions while contracting them in others, effectively prioritizing certain preference probability differences over others in the evaluation of generator vs. anchor. Note that placing non-identity functions f. g in any other position, i.e., Eptx)] Pyyy x)9 Ep x) [h- p(E =1 Yo, y)), breaks prevalence-consissennyy of query and generator respectively.
Ko-constra RLHF is KL-regularized WRO with h=logit and the Bradley-Terry assumption to estimate the preference classifier via a reward function (Azar et al., 2023). However, many other WRO instances exist, e.g., for any combination of h, anchor distribution, and estimate of p(E = Xo Yi) or Epix, 1x) [h. p(l- = x, No-N, V1)] yields a different WRO objective.' 3 Different choices of la and anchor yield different optimal solutions in general. When the BT assumption holds, however, all objectives in Equation (2) maximize all h-variants of win rate for all anchors.
What about other evaluations? Proposition 3.3 proves that all other evaluations either break preference- or prevalance- consistency. For instance, reward and ranking accuracy, typically computed on offline data, do not respect model prevalences given the pairs in the computation may be un- likely overall under the model; empirically, they have also been shown to correlate little with win rate (Rafailov et al., 2024a). Average reward Fx(, (x, y) is equivalent to translated logit-Wing Win Rate under the Bradlcy- Terry assump- tion p(2 = 1 No- y) = ofr(x. y) - r(x. yo)l (Azar et al., 2023), but the underlying preference environment need not satisfy this assumption."
Proposition 4.1. (informal) Under the Bradley- Terry as- sumption, all objectives in Equation (2) with strictly in- creasing h share the same optimal solution. This solution maximizes win rate overall all possible anchor distributions.
See Appendix C for proof.
Given the insight that the only evaluation grounded in pref- erence data alone is h-win rate with respect to some anchor distribution, we next analyze common preference learning algorithms based on how they relate to optimizing for win rate. In Section 4, we characterize the space of Win Rate Optimization (WRO) objectives, highlighting existing and new methods that fall under this family, and discuss their theoretical benefits. In Section 5, we analyze instances of NRO methods and their limitations.
4.2. Win Rate Optimization Games
The choice of anchor need not be fixed up front. Instead, the anchor distribution could also optimize for its own win
'For instance, Eptro 1x) hh p(č=1 y. V,) can estimated by learning the preference distribution p(f = No> N1) from data and estimating the expectation with samples from the an- chor or training a model to estimate this expectation directly, removing the need to additionally sample from p(y, x) during the policy optimization step. The RLHF setting of WRO par- ticular (namely, choice of = logit and BT assumption) also allows the expectation over the anchor p(yo (x) to be dropped, i.e., maxpos, (x) Ppp(y [x] Fy 1x) p(E 1 xx Yo: max [x) Epir, xx [x. y)], thus requiring only that reward model be estimated.
Note that the Bradley Terry assumption suggests not only tran- sitivity but a functional relationship in the preference probabilities, e.g., if A is preferred over B with probability 6 and B is preferred over C with probability .7, then the Bradley-Terry assumption states that A must preferred over C with probability Rs 0.778.
4

---

Preference learning made easy: Everything should be understood through win rate
rate, resulting in a family of WRO games:
the model and the target distribution KL (y x) x Py x) exp( 3 Bp(rals) [h -p(e Yo. y)]); see Azar et al. (2023) or Appendix D.I for derivation. As such, the reverse- KL regularized WRO objective is form of black- box variational inference (Ranganath et al., 2014).
max Bpix) po Ep, (x p(l- - 1 y',, yY),
0
max Bxix) (x (r p(e y,
7
Nash Learning from Human Feedback (Munos et al., 2023) is one instance of a WRO game with h=identity (and ad- ditional KL. regularization); the choice of & leads to an antisymmetric, constant-sume game. However, WRO games need not be anti-symmetric or constant sum; for instance, a WRO game with h = log is neither. Different choices of h dictate different prioritization of preference probabilities to target, and it is not obvious that /==identity, the specific setting considered in Munos et al. (2023), should be the only WRO game to consider. After all, aligning to preferences is not inherently& a constant sum endeavor.
Regularized WRO (n,,(0) does not satisfy win rate- correspondence nor consistency in general but modified versions of both: 1. (regularized win rate-correspondeece)) Improving the objective implies either improvement in h-win rate or divergence to the reference, i.e., the regu- larization term; and 2. (regularized win rate-consistency): #* = arg max; 2, maximizes h-win rate for all distri- butions with divergence within (3) = D(po- Pret).
5. Preference algorithms that are not WRO
5.1. DPO fails win rate-correspondence
4.3. Benelits of Win Rate Optimization
Direct Preference Optimization (DPO) (Rafailov et al., 2024c) shares the same target distribution as RLHF but employs a different objective to optimize for that target. Denoting (o x. No: as Pr,O and q(x, as some distribution of offline data, DPO
Letting dh (0) denote the li-win rate, (Po ( efits of WRO objectives (0) are:
x)), two ben-
1. (Win rate-correspondence). Improving the objective improves h-win rate:
min LDPO (0) min 9
-E Eg ( (olle//o.))0ll/p....... Pogrt.)) (3)
( (00) = 6(0) > o(0")
where pe, is parametrized to include language model inside of it. Namely, pe,o(E = xo: is
2. (Win rate-consistency): The optimum of the objective maximizes h-win rate:
po (y xx) Prr x) 3 3log
po (Yn xx) Pref(Yo [x]
For t = arg max 0(0) 6(0") max ((0).
o 3 log
(4)
Win rate-correspondence makes gradient- based optimiza- tion practical, while win rate-consistency ensures that the loss itself does not exhibit fundamental limitations with respect its optimal solution.* Both properties follow triv- ially for WRO from the fact that O o. These properties need not hold for non OR- (see Section 5).
DPO fails win rate-correspondeece, as better (i.c., lower) DPO loss does not necessarily mean better (i.e., higher) win rate. This correspondence fails for two reasons:
1. Using offline data breaks gevaalencecoonistenng For instance, a model change that improves the ranking of responses YA VB in the offline data but moves more mass overall generate dispreferred responses not covered in the offline data will improve DPO loss but hurt win rate. Even sampling from the initial model is not sufficient (see Appendix F for an example), as throughout training.
Regularization. As p({ = 1 x. Yo: is typically esti- mated from data, to prevent overfitting to it one can reg- ularize with with a divergence penalty D(po. (pe) with reg- ularization parameter |::
- max h- Win Ratepty, 1) (p xx] - 3Ep(s) ((po.
of the generator does not hold
Options for D include sequence-level reverse KL, chi-sq, or sum of token-level divergences (Azar et al., 2023; Huang et al., 2024; Rafailov et al., 2024b).
2. Even if we satisfy prevalence-consistencyy by switch- ing to a fully online setting', win rate correspon- dence still does not hold; see Figure for a visual- ization. This is because the likelihood of preference
Optimizing WRO with reverse KL regularization is equiv- alent to minimizing the reverse KL divergence between
"This would consist of sampling from the appropriate query, generator, and anchor distributions throughout training. Guo et al. (2024); Calandriello et al. (2024) do this but use Faap--gadiest (y xx) instead of Tp (x) as loss; as evaluations both are the same.
For readers interested in the precedence for these properties, win citeccorrespeneecce and consistency are closely linked to Question 2 in (Steinwart, 2007) and H-consistency (Awasthi et al., 2021) respectively.
5

---

Preference learning made easy: Everything should be understood through win rate
distribution in Equation (3) is not an h-win rate: (x.N. pe. Pr,o) # Ppp(xx80.1 (a.pppppp pe]. Since prevalence-consissenccy holds, preference consistency of this online objective must not hold, which can be seen by noting that the argument inside Ep 3x. is not a fixed function of pe.
5.2. SFT fails win rafe-correspondeece and consistency
Supervised finetuning (SFT) is often viewed as an initial step for necessary precursor to other preference learning algorithms (Wang et al., 2024a; Razin et al., 2024b), but it is also preference learning algorithm itself based on filtering and maximum likelihood estimation.
Supervised finetuning on preferred samples (sometimes de- noted y,.)) seeks to maximize the likelihood of sample y when L = 1 and Yo when é O. When p(y1 x) p(Yo x). then p(y,, |x) = p(y1 x, l =1 ) p(yo xx l = 0) and the SFT loss can be written follows:
0.8
0.7-
0.6
CEFT = min E () log po xx)
0.5
0.45
0.50
0.55
0.60
0.65
0.70
online
Like DPO and direct alignment algorithms, this fails win rate-correspondence, as the above expression is not an h- win rate. What about win rate consistency? The target distribution can be written as
Figure 1: Plot of win rate vs. DPO loss for different (y |x) for a given choice of anchor and preference classifier (see Appendix F.2 for details). Any pair of points that would form line segment with positive slope form setting of 0, 0' such that win rate correspondence does not hold. Grey lines denote the win rate and DPO loss at initialization. See Appendix F.2 for additional discussion and examples.
Py x) x ply [x) tx)p(e = Y.. [x))))
(5)
which tilts the original distribution towards sequences with higher average preference probabilities over the anchor. However, there are limits to the amount of improvement Equation (5) can achieve (see Appendix E for visualization). In fact, we can characterize the exact win rate expected from the SFT target distribution over the original model:
Note that even though DPO is derived to match the target distribution of a regularized- WRO objective, DPO does not satisfy regularized win rate-correspondence either. Namely, is possible to improve (i.e., decrease) the DPO loss without improving win rate or divergence to the reference model (see Appendix F.3 for example).
Theorem 5.1 (Win rate improvement of SFT on preferred). Let p(yo x) be the initial generative model, and Psrtly xx) be the larget distribution of SFT on preferred samples (ply, x, E = 1), p(yo x) = ply, xx). Then,
The insight that DPO fails win rate- rate- correspondence offers explanation for empirical results such as the loss vs. win rate misalignment observed in Chen et al. (2024); Rafailov et al. (2024a), as well as for why it can helpful to perform very early stopping (e.g., one epoch, far from any evidence of overfitting) rather than model selection with DPO validation loss (Rafailov et al., 2024c), or choose checkpoints using win rate directly (Yuan et al., 2024).
Win Rate (x) (PSFTT( |x)] = 0.5 p)xx Varpo, [x) [Epor, (x) Ip(E 1 |x,,,.>)))] (6)
The theorem states that variance in the average preference probability under the initial model dictates the extent of win rate improvement. Intuitively, one should not expect any possible improvement in win rate if the existing model only outputs sequences which are equally preferred to each other; on the other hand, there is more room for improvement in win rate the more differentially preferred some of the model's sequences are to others. As long as the model has support over more than two sequences, however, the win rate cannot be maximized. See Appendix G for proof.
The above also applies to direct alignment algorithms (DAA) in general (Azar et al., 2023; Tang et al., 2024; Huang et al., 2024; Ji et al., 2024): the use of offline data means that prevalence-consiseeme does not hold, and an inner objective f(x ((x y1) (x,,,,)),)) 3i other than the prefer- ence classifier (including divergence to the true classifier, distance to some function of the true classifier) means that preference-consissennyy does not hold.
We can generalize Theorem 5.1 other filtering strategies:
Takeaways. Even though DPO and other DAA often share the same optimal solution as a regularized-We -RO objective (i.e., regularized-wee win rate consistency holds), these alterna- tive objectives fail win nate-correspondencce complicating their use training and model selection.
Theorem 5.2 (Win rate improvement of filter + SFT). Let p(f y,,y,, E) be a filter that selects data (f = 1) for supervised finetuning, fnetunigg and plyo (x) = ply, [x)). Then, de- noting Epptss 1x) [p( = xx,oo 11: E)) as Aughilter(x.y))
6

---

Preference learning made easy: Everything should be understood through win rate
and Episo (x) "p(l- No)o)11) as y)gPre/(xx,1))
Win Ratepoo (x) (pSErTy x)] 0.5
50
Win Rati
method
method
+ Epxx) Corpl, (x) ( AvgPrefr..))))) (7)
RUHF
RLHF
DPO
DPO
Epor (vgg
SFT
50
logil win rate (oracle
See Appendix G for proof. Doing SFT on preferred sets the filter as p(f No. e) 1[e 1]. Win rate increases as the denominator of the second term, measuring how often a response is selected, decreases and/or the numer- ator, measuring how closely the average filter and preference co-vary increases, In other words, both the threshold for filtering as well as the diversity (in p of the initial model matter for improvement possible in filter SFT algorithm. This could explain the success of works such as Li et al. (2023a), who use backtranslation to generate larger set of responses than the model's original generations and filter a subset for SFT, and Brandfonbrener et al. (2021), who filter based on quantile.
B
0.1 0.01 0.001
B
(a) Expected win rates
(b) Actual win rates
Figure 2: Expected versus observed win rates of RLHF, DPO, and SFT over the original model for OASST. RLHF notably underperforms relative to expectations. Moreover, substituting the learned reward model for the oracle pref- erence classifier (logit win rate (oracle)) does not improve performance, suggesting other factors are more important.
for OASST and 81.36 for HH) and 2. an imperfect judge model (accuracy is 80.47 for OASST and 85.16 for HH), As expected, the BT assumption is helpful for preference classifier estimation in OASST but not in HH, as the OASST directly abides by this assumption (outputs are globally ranked) whereas HH does not explicitly. See Appendix L for additional training and evaluation details.
Takeaways. SFT on preferred samples fails win rate- correspondence and consistency. The improvement possible is function of the initial model's diversity in preference. Other filtering strategies with stricter criteria can yield fur-
ther win rate improvements.
Implementing WRO-KL. Variants of WRO-KL can be im- plemented via different & in max; Epoty (xx
6. Investigating the empirical impact of different design choices
8 Blog x) + B log Pre((y |x)], e.g., (xx Yo. We approximate expec- tations over generator distributions with single-sample Monte-Carlo estimate. Concretely, for a given query we sample one response each from the current and original model to compute 1h, which is either a function of 1. the preference probability under the judge model which takes in the pair as input, or 2. the BT preference probability under the reward model which takes each sequence as input separately. The one exception is WRO-KL--lgii-BT (i.e., RLHF), where we drop Epcr, (x) (r( ( and only optimize r(x,y) (see Appendix M for comparison between keeping and dropping this term). For optimization, we use the PPO algorithm from the TRL library (von Werra et al., 2020).
Eprr ix)pp(e
We next compare WRO and non- WRO experimentally to complement the above theoretical analysis with an empiri- cal one. First, we compare SFT, DPO, and RLHF on equal footing based on how much they are able to improve win rate over the starting model. Moreover, we also compare dif- ferent variants of KL-regularized WRO, varying / (identity, log, and logit), 8 (1, 0.1, 0.01, 0.001), and the estimation of the preference classifier (perfect and estimated with and without the Bradley Terry assumption).
6.1. Experimental setup
We employ Pythia-2.8b (Biderman et al., 2023) as our base model and the OpenAssistant (OASST) (Kopf et al., 2023) and Anthropic Helpfulness and Harmlessness (HH) (Bai et al., 2022) as datasets. We train the base model on the data outputs and use these finetuned models our initial models. We train an oracle judge model per dataset to estimate p( 1 x, F1) and relabel the preference annotations using this judge model. We use the same oracle to evaluate win rate after training. See Appendix K for further details on the judge model. We additionally train reward model on the oracle-labeled judge annotations (accuracy is 82.8
6.2. Results: Comparing RLHF, DPO, and SFT
Figure 2 compares methods. While SFT performance aligns with expectations, as does DPO performance with more reg- ularization, RLHF substantially underperforms relative to expected given the aforementioned analysis. as does DPO with less regularization. The non-monotonic nature of both point to the influence of factors beyond target distribution for win rate improvements. To test if RL.FF underperfor- mance is due to error from using an estimated reward model instead of the true preference classifier, we additionally run a WRO-logit variant with oracle preference classifier, but this performs even worse, suggesting another factor at play.
Recall that Covpor, (x) [AVgFiiitrrr, y,), AvgProf(x.y) v,) <
Vargón (x) [AvgFilter(x Y1) Varpor, (x) AvggPref(x,
7

---

Preference learning made easy: Everything should be understood through win rate
Table I: Win rate of WRO-KL variants over the reference model. Row corresponding to RLHF is shaded grey. No choice of pe. h or / systematically outperforms all others.
win rate is the only evaluation that can matter based on preference data alone, and thus all of preference learning should be understood in relation to it (e.g., how well given method optimizes for win rate, the role of additional assumptions). This work takes the first steps to do that (see Appendix A for summary). High level takeaways from the analysis are that WRO is the theoretical ideal (as existing non- non-WRO fail win coteccorrepooddenc or consistency), but optimization difficulties can lead to its underperformance in practice. Thus, optimization is a first-order factor in the practical success of WRO, while achieving / improving win cate-corresooneenc and consistency is important for advancing non- WRO-
Dataset
Fr
/ 0:001
0.01
/-0.1
A=
HII
non BT
log
(8.36(L.21)
16.38.11.34)
(0.94)
67.00 (0.33)
logit
69.15
41.58
48.74
69.15 (0.55)
identity
69.94 (0.99)
69.94 (0.99)
65.92 (8.95)
43.31 (0.54)
BT
log
35.41 (1.98)
(1.59)
41.76 49)
49.49 (0.00)
logii
63.93
$4.60
identity
65.71 (0.95)
64.59 (0:95)
69.91 (0.99)
51.72 (0.59)
oracle
Ing
34.98 (1.24)
52.74 (0.26)
47 57.06 (1 (09)
43.29 (0.66
55.13 (1.07)
48.55
identity
65.34(0..33)
(1.0)
68 39 (8:94)
45.14 (0.44)
OASST non-BT
log
(3.88)
59.02 (3.96)
61 98 (1
58.63 (1 98)
lagit
(3.55)
(3.20)
(1.20)))
48.69 (1.33)
identiiy
56.65 (3.26)
54.20 (2.28)
64 90 22)
53.64
BT
log
(3.40)
62.80 (3.55)
60.94 (3:46)
54.64 (1.26)
logit
(2.05(33.65)
63.95 (3.38)
(1 16)
58.45 (1.82
idietiity
(3.50)
$1.98 (2.96)
54 (2. 16)
50.42 (1.58)
What does this mean for current practice? The impor- tance and difficulty of optimization for WRO suggests that strategies to improve optimization success could help, e.g., multiple seeds. The lack of win rate-correspondence in DPO/DAA clarifies the importance of checkpointing with metrics other than loss. And SFT's dependence on variance (in preference) suggests that increasing preference diversity of generations to annotate could yield gains.
onacle
lng
59,72.(3.79)
$6.00 (3.69)
60.99
logit
49.55 (3.06)
44 (2.61)
53.66 (2.8 87)
(1.55
identity
(2.47)
66.69 (1.29)
66.21 (1.22)
52.31 (1.25)
-02
-0.2
-0.3
04
Itann
Thale
-0.6
0.
-1.0
35
40
45
55 Win rate
60
65
What's the value of many methods? Interestingly, the- ory and experiments yield opposing trends: based on win rate-corresoondence and consistency, RLHF > DPO > SFT; but based on ease of optimization, SFT > DPO > RLHE." The fact that no existing method is optimal with respect to both the objective being optimized and ease of optimization suggests why it might be useful to combine methods (to take advantage of different strengths) and develop new ones (to strike better overall balance).
Wih rate
Figure 3: Train loss vs. test win rate across all settings
tested in Table 1.
6.3. Results: Comparing WRO-KL variants
Table compares different WRO-KL objectives with dif- ferent choices of h, 3, and the estimation of the classifier. Notably, no WRO-KL method (i.e., choice of h) outper- forms the others systematically across settings. Moreover, the settings that correspond to better target distributions (us- ing a perfect preference classifier, low 8) do not necessarily yield better win rates empirically. These results suggest that there is a more important consideration than the target distribution implied by the objective, namely the success of optimization. Indeed, we see that training loss correlates more with test win rate across WRO-KL than any of the target distribution design choices pe, h, or 3: the p-value of Spearman rank correlation test is 8.27e-5 for train loss vs. win rate, compared to 0.968 and 0.133 for Pr and B, and 0.54/0.87/0.88 for all pairs of i choices under Kolmogorov- Smirnoy test. Figure 3 plots train loss vs. test win rate for all WRO-KL variants. Interestingly, even though one would expect the losses for different WRO variants to dif- fer in scale under different choices in h and B, there still exists surprisingly noticeable global trend of better loss corresponding to better win rate.
What's next for preference learning? Our analysis sug- gests that the most important improvements in preference learning will likely fall under the umbrella of moving closer to directly optimizing for win rate, either in the objective itself (i.e., coming up with better surrogates for win rate optimization) or the practical optimization of it (c.g., be- yond curent policy gradient approaches). How might we make progress? For one, the connection between RLHF/ / WRO-KL objectives and variational inference suggests the possibility of drawing upon the rich field of probabilistic inference to improve preference learning, from variance re- duction techniques (Mohamed et al., 2020b) to alternative optimization objectives and algorithms altogether (Naesseth et al., 2020). Moreover, the strategies and theoretical tools used to understand and develop surrogate objectives for other machine learning tasks (Bridle, 1990; Steinwart, 2007; Awasthi et al., 2022) could provide a useful starting point for developing better objectives for preference learning.
Several concrete directions appear in Section 4 and Sec- tion 5 including non-constant sum WRO games to reflect
7. Discussion
Amidst the complex landscape of aligning language models to human preferences, this work offers a simplifying insight:
7PF >> DPO given failures at low B and DPO training issues mentioned in Pal et al. (2024); Chen et al. (2024)
8

---

Preference learning made easy: Everything should be understood through win rate
the fact that aligning to huamn preferences need not be constant-sum endeavor; fully online versions of direct align- ment algorithms or contrastive post-training algorithms to help satisfy prevalence consistency (and fewer degrees of freedom for win rate correspondence failures); and alter- native piter-ppus-SFT style algorithms that take diversity and filter strength into account for solutions with better win rates. There are probably many more promising ideas to explore- long win rate remains the guide.
Sutawika, L.., and van der Wal, O. Pythia: A suite for analyzing large language models across training and scaling. ArXiv, abs/2304.01373, 2023. URL https: /app::///pp..../ http:///ppp...mma.a.. org/CorpusTD: 25712212:22211288 wwt.:esisaattstttoeeeeeeee
Skowron, A.,
Brandfonbrener, D., Whitney, W. E, Ranganath, R., and Bruna, J. Quantile filtered imitation learning. arXiv preprint arXiv:21 2021.
Bridle, Probabilistic interpretation of feedforward classi- fication network outputs, with relationships to statistical pattern recognition. Neurocomputing. 1990.
Impact Statement
This paper presents work whose goal is to advance methods for aligning generative models (especially language models) to human preferences. There are many potential societal consequences of this work; perhaps the most critical is the question of whose preferences are being represented and how.
Calandriello, D., Guo, D., Munos, R., Rowland, M., Tang, Y., Pires, B. A., Richemond, P. H., Lan, C. L.., Valko, M., Liu, T., Joshi, R., Zheng, Z., and Piot, B. Hu- man alignment of large language models through online preference optimisation, 2024. URL https: //aps:://ar //arxiv.. org/ abs / 2403/4433330 240..08660 08635.
Acknowledgements
Chen, A., Malladi, S., Zhang, L. H., Chen, X., Zhang, Q.. Ranganath, R., and Cho, K. Preference learning algo- rithms do not learn preference rankings. NeurIPS, 2024.
This work was partly supported by the NIH/NHLBI Award R01HL148248, NSF Award 1922658 NRT-HDR: FUTURE Foundations, Translation, and Responsibility for Data Sci- ence, NSF CAREER Award 2145542, ONR N00014-23-1- 2634, NSF Award 2404476, Apple, Google, and Meta.
Christiano, P. F., Leike, J., Brown, T. B., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. ArXiv,
2017.
References
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, Raft: Reward ranked finetuming for generative foundation model alignment. TMLR, 2023.
Ahmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J.. Üstün, A., and Hooker, S. Back to basics: Revisit- ing reinforce style optimization for learning from human feedback in Ilms. arXiv, 2024.
Ethayarajh, K., Choi, Y., and Swayamdipta, S. Understand- ing dataset difficulty with v-usable information. ICML, 2022.
Awasthi, P., Frank, N., Mao, A., Mohri, M., and Zhong, Y. Calibration and consistency of adversarial surrogate losses. Advances in Neural Information Processing Sys- tems, 34:9804-9811 2021.
Gui, L., Gârbacea, C., and Veitch, V. Bonbon alignment for large language models and the sweetness of best-of-n sampling. NeurIPS, 2024.
Awasthi, P., Mao, A., Mohri, M., and Zhong, Y. Multi-class h --coniistency bounds. NeurIPS, 2022.
Gulcehre, C., Paine, T. L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., Macherey, W., Doucet, A., Firat, O., and de Freitas, N. Reinforced self self-training (rest) for language modeling. arXiv, 2023.
Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., and Munos, R. A general theoretical paradigm understand learning from human preferences. arXiv, 2023.
Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F., Rame, A., Mesnard, T., Zhao, Y., Piot, B., Ferret, J., and Blondel, M. Direct language model alignment from online ai feedback, 2024. httstttss: //trtttsss///iv..../ //rxivv org/ abs 224/224022222 04792.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv, 2022.
Han, J. Jiang, M., Song, Y., Leskovec, J., Ermon, S., and Xu, M. f-po: Generalizing preference optimiza- tion with f-divergence minimization. arXiv preprint arXiv:2410.2 2024.
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E.,

---

Preference learning made easy: Everything should be understood through win rate
Huang, A., Zhan, W., Xie, T., Lee, J. D., Sun, W., Krishna- murthy, A., and Foster, D. J. Correcting the mythos of Kl-regularization: Direct alignment without overoptimiza- tion via chi-squared preference optimization, 2024. URL https:/ / arxiv. wttp/iiooo.//oooooo abs/ 2407 13399.
Mohamed, S., Rosca, M., Figurnov, M., and Mnih, A. Monte carlo gradient estimation in machine learning. JMLR, 2020a.
Mohamed, S., Rosca, M., Figurnov, M., and Mnih, A. Monte carlo gradient estimation in machine learning. JMLR, 2020b.
Ivison, H., Wang, Y., Liu, J., Wu, Z.., Pyatkin, V., Lambert, N., Smith, N. A., Choi, Y., and Hajishirzi, H. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. NewrIPS, 2024.
Munos, R., Valko, M., Calandriello, D., Azar, M. G., Row- land, M., Guo, Z. D., Tang, Y., Geist, M.. Mesnard, T., Michi, A., Selvi, M., Girgin, S., Momchev, N., Bachem, O., Mankowitz, D. J., Precup, D., and Piot, B. Nash learning from human feedback. arXiv, 2023.
Ji, H., Lu, C.,, Niu, Y., Ke, P., Wang, H., Zhu, J., Tang, J., and Huang, M. Towards efficient and exact opti- mization language model alignment. arXiv preprint arXiv:2402.00856. 2024.
Naesseth, C. A., Lindsten, F., and Blei, D. Markovian score climbing: Variational inference with kl(p)(q). NeurIPS, 2020.
Jiang R., Chen, K., Bai, X., He, Z., Li, J., Yang, M., Zhao, T., Nie, L.,, and Zhang, M. A survey on human preference learning for large language models, 2024. URL https: //arxiv. org/ abs / 2406. 11191.
OpenAL. Gpt-4 technical report. arXiv, 2024.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L. E., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. J. Training language mod- els to follow instructions with human feedback. ArXiv, 0bs/2203.02155. 2022.
Kaufmann, T., Weng, P., Bengs, V., and Hüllermeier, E. A survey of reinforcement learning from human feedback.
ArXiv, abs/aa/b/.... 14925, 2023.
Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Ham- bro, E.. Grefenstette, E., and Raileanu, R. Understanding the effects of rihf on Ilm generalisation and diversity. arXiv, 2024.
Pal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., and White, C. Smaug: Fixing failure modes of preference optimisation with dpo-positive, 2024. URL https:// arxiv. arxiv.org/abe 222.2122222222...
Kopf, A., Kilcher, Y., von Rutte, D., Anagnostidis, S., Tam, Z. R., Stevens, K., Barhoum, A., Duc, N. M., Stanley, O., Nagyfi, R., Shahul, E., Suri, S., Glushkov, D., Dantu- luri, A., Maguire, A., Schuhmann, C., Nguyen, H., and Mattick, A. Openassistant conversations democratizing large language model alignment. ArXiv, abs/2304.07327. 2023.
Rafailov, R., Chittepu, Y., Park, R., Sikchi, H., Hejna, J.. Knox, B., Finn, C., and Nickum, S. Scaling laws for reward model overoptimization in direct alignment al- gorithms, 2024a. URL https: //tps////aavvv/eeeeeeeeeeeeievee abs/ 2406. 02900.
Lambert, N., Tunstall, L., Rajani, N., and Thrush, T. Huggingface h4 stack exchange preference dataset. 2023. URL. https:/ https://huggingteee /twuggingfaccee... / co/catasstts/uuggggaaaaa stack-exchaage-preeeereecc.
Rafailov, R., Hejna, J., Park, R., and Finn, C. From r g Your language model is secretly u-ffucction. COLM, 2024b.
Li, X., Yu, P., Zhou, C., Schick, T., Levy, O., Zettlemoyer, L., Weston, J., and Lewis, M. Self-alignment with instruc- tion backtranslation. arXiv preprint arXiv:2308.06254 2023a.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. NeurIPS, 36, 2024c.
Li, X.. Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacaeval: An automatic evaluator of instructionnfollooinng models. Gitllub repository, 2023b.
Ranganath, R., Gerrish, S., and Blei, D. Black box varia- tional inference. In Artificial intelligence and statistics, pp. 814-822. PMLR, 2014.
Razin, N., Malladi, S., Bhaskar, A., Chen, D., Arora, S., and Hanin, B. Unintentional unalignment: Likelihood displacement in direct preference optimization, 2024a. URL hwt.httpp///// /atpss////xvv..////// wtt.v/.rggbbb//eeeeeeeo 08847.
Minaee, S., Mikolov, T., Nikzad, N., Chenaghlu, M. A., Socher, R., Amatriain, X., and Gao, J. Large language models: A survey. ArXiv, 0tdp/240.00696 2024.
10

---

Preference learning made easy: Everything should be understood through win rate
Razin, N., Zhou, H., Saremi, O., Thilak, V., Bradley, A., Nakkiran, P., Susskind, J., and Littwin, E. Vanishing gradients in reinforcement finetuning of language models. ICLR, 2024b.
Weng, L. Policy gradient algorithms. lilianweng siilawwengggilhub.ic 2018. URL httpps://sslssiieeeeee posts/2018 04-08 08 1-a--oolicyy grwdyyytggad///....a
gi teu...ubbaa... io/
Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Durme, B. V., Murray, K., and Kim, Y. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation. JCML, 2024a.
Rosset, C.,, Cheng, C.-A., Mitra, A., Santacroce, M., Awadal- lah, A., and Xie, T. Direct nash optimization: Teach- ing language models to self-improve with general prefer- ences, 2024.
Xu, J., Lee, A., Sukhbaatar, S., and Weston, Some things are more cringe than others: Iterative preference opti- mization with the pairwise cringe loss, 2024b. URL https: /ttts//arxiii www.xxiw....gass./oeeeaaa
Shirali, A., Nasr-Esfahany, A., Alomar, A., Mirtaheri, P. Abebe, R., and Procaccia, A. Direct alignment with heterogeneous preferences. ArXiv, abs/2502.16320,
2025. URL httpp///ppp...mmaiiccccoooo semanticschooarr org/CorpusID: 222 276574796.
Yuan, W., Pang. R. Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, J. Self-rewarding language models. arXiv, 2024.
Singhal, P. Goyal, T., Xu, J., and Durrett, G. A long way to go: Investigating length correlations in rlhf, 2024, URL
Zhao, Y., Joshi, R., Liu, T., Khalman, M., Salch, M.. and Liu, J. Slic-hf: Sequence likelihood calibration with human feedback. arXiv, 2023.
https: /arxiv. org/ abs/ 2210 03716.
Siththaranjan, A., Laidlaw, C., and Hadfield-Menell D. Distributional preference learning: Understanding and accounting for hidden context in rlhf. ICLR, 2024.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D.. Xing, E., et al. Judging im-as-a-judge with mt-bench and chathot arena. NeurIPS, 36, 2024.
Steinwart, How to compare different loss functions and their risks. Constructive Approximation, 26(2):225-287, 2007. doi: 10.1007/500365-006-66666
Zhu, B., Jordan, M. I., and Jiao, J. Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf. arXiv, 2024.
Stiennon, N., Ouyang, L.,, Wu, J., Ziegler, D. M., Lowe, R. J., Voss, C., Radford, A., Amodei, D., and Christiano, P. Learning summarize from human feedback. ArXiv, abs/2009.013 01355, 2020.
Swamy, G., Dann, C., Kidambi, R., Wu, Z. S., and Agarwal, A. A minimaximalist approach to reinforcement learning from human feedback. ICML, 2024.
Tajwar, F., Singh, A., Sharma, A., Rafailov, R., Schneider, J., Xie, T., Ermon, S., Finn, C., and Kumar, A. Preference fine-tuning of llms should leverage suboptimal, on-policy data. ICML, 2024.
Tang, Y., Guo, Z. D., Zheng, Z., Calandriello, D., Munos, R., Rowland, M., Richemond, P. H., Valko, M., Avila Pires, B., and Piot, B. Generalized preference optimization: A unified approach to offline alignment. ICML, 2024.
von Werra, L., Belkada, Y., Tunstall, L., Beeching, E.,
Thrush, T., Lambert, N., and Huang. S. Trl: Transformer reinforcement learning. GitHub repository, 2020.
Wang, Z., Hou, L., Lu, T., Wu, Y.. Li, Y., Yu, H., and Ji, H. Enabling language models to implicitly learn self- improvement. ICLR, 2024a.
Wang. Z., Nagpal, C., Berant, J.. Eisenstein, J., D' Amour, A., Koyejo, S., and Veitch, V. Transforming and combin- ing rewards for aligning large language models. arXiv, 2024b.

---

Preference learning made easy: Everything should be understood through win rate
A. Summary of theoretical contributions
Below in Table 2, we present a table summarizing the win rate-centric framework introduced in this work, as well as the original source for each insight within the overall framework:
WRO membership
Generalization
Win rate-correepoodeece
Win rate-consistency
RLHF
v (Azar et al., 2023)
h-win rate (Azar et al., 2023)
V
(Section 4.3)
v (Section 4.3)*
NLHF
v (Munos et al., 2023)
h-win rate games (Section 4.2)
(Section 4.3)
v (Section 4.3)²
DPO
(Section 5.1)
direct alignment algorithms (*)
(Section 5.1)
/ (Rafailov al., 2024c) X(Section 5.2)
SFT
X(Settion 5.2)
best-of-n, RAFT, ReST (*)
((ection 5.2)
Table 2: Summary of the win-rate centric framework introduced this work. (*) means that the given insight can be attributed to multiple works. Works which generalize DPO and propose alternative direct alignment algorithms: Azar et al. (2023); Huang al. (2024); Tang al. (2024); Han al. (2024). Works that generalize SFT the preferred sample to more samples include Dong et al. (2023); Gulcehre et al. (2023). 'RLHF and NLHF typically include regularization term and satisfy regularized win cate-correspondecce and consistency. Without regularization, win ratc-correspondenc and consistency are satisfied. *DOO only satisfies regularized win rate-consistency (3 must be greater than zero for meaningful objective) and was constructed specifically with this goal in mind.
B. Proposition 3.3 proof
First we show a lemma that breaks down prevalance consistency into sum over atoms.
Lemma B.1. Let 3 be functional of distribution p with support over countable sample space X, where if Op1 + (1 a2p the functional satisfies 3(p) = aB((1)) + (p2)) Then,
3(v) Σ ((1)) x= =x]).
XEX
Proof. We prove this by induction the size of the support of X. The base case for = 2 is true by assumption, now assume the statement is true for support size of n, then we show it is true for n + 1.
Let be a support set of size n + 1. Let A be a set containing an element a in X, where a = =p(x E A). Then, p = x A) + (1 - (x x A²) and
B(p) p(x E A))(pp(
+ p(x A))8(p(X|xE A9))
p(x a) x) p(x ())
p(( ())3(1(=a))) = (1-p(x A))
= p(( 3] = a)) - (1 - -P(x E A)) Σ X =xD)), A of size n xEd=
= p(x = 311x = a)) + Σ
(9) x1))
XEE
p(x = a)3(((x = al) + Σ
p(x= x==(xx((=XxxI))
YEAee
Σ xxx)8(1|==X]
XEX
This completes the induction step.
Proposition 3.3. For discrete p. an evaluation o is grounded, defined in Definition 3.2, if and only ffr every generator ply, [x), anchor plyo x) in quefe-preference environment (p(x), ())
Oply, (x) (ply xx) E) Ep(r) x) E (x)
(x) [h p(e 1 xo,
(8)
Proof. For the forward direction, we can check both conditions on Expx Epty, 1x) RE 1x) p(e= = x,
12

---

Preference learning made easy: Everything should be understood through win rate
Preference consistency holds because if all distributions are point masses, the expectation becomes evaluation those points
Prevalence consistency holds by the general statement Ex-an + (f(z)) = qBq, [f(z)] + (1- a) (z)).
For the reverse, we need to show that if an evaluation á meets preference and prevalence consistency, it can be written as Oplro [x) (p(y1 x), = Epox)pppr, s) 1x) p(e = 1 Yu. for some h.
By preference consistency for all triples x. x. No obtained for generator ply, xx), anchor p(yo x), and query query-preference environment p(( Y.)), the following holds for some h:
@ (1 = víl. (1 x(), = h p(e yo:
Then, by prevalence consistency with X as the support of p(x), Vi as the support of p(yol x), and Yi as the support of ply, x).
@pos (x) (p(y, (x),8x
Σ
8pbo (ply, (x), (L[x xxxx=x],pee)il xe),)e))p(x)
NEX
Σ Σ
11 x1 (x), (1[x =x]. Ix)p(x)
VEX MEYS
Σ
PI1y 91 (1 -1 = =1x x')((s xx)(('))
xEX Exy' yer
E
Σ
Σ
p(e 1|x. v1)) x))(>' xx)((x), since preference consistency holds for all atoms
X'EX YEER'S vEEN
= Ex(x Epty| v) Pprr w/th- p(e = 1 No. y)].
To generalize the results to continuous density, one would need to additionally assume some notion of convergence for Φ as well to use weak convergence from a discrete measure.
C. Proposition 4.1 Proof
We provide both the informal statement the main paper as well its formal version.
Proposition 4.1. (informal) (informal) Under the Bradley- Terry assumption, all objectives in Equation (2) with strictly increasing h share the same optimal solution. This solution maximizes win rate overall all possible anchor distributions.
Proposition 4.1. (formal) Denote by Pi, the set of distributions ply xx) that optimize the WRO-h objective in Equation (2). Assume a hypothesis class induced by 8 E such that all optima are realizable. Then, for a given anchor distribution plyo x), Pj = Pj for any strictly increasing under the Bradley-Terry assumption with finite rewards.
Proof. We first introduce the P Pit the set of all distributions p(y x) which for each x place all their probability mass over only the highest-reward sequences or some subset of them. Then, we show that Pieward = Pj for any strictly increasing h and any anchor distribution p(yo x). In other words, any maximum-reward distribution Preward (y x) Prwward maximizes any WRO objective and vice versa:
E Ep(x) Bpinunty ty xx) Ep I 1x) p(e = 1 yx: yo.
(9)
Epp E.p (y) E pprr. x)) [h hoor (r(x,y) - r(x.x)))
(10)
max (x E Epoly, 1x) Epo x / - -
(11)
max pp()) po (x) EpG. w)hho(( x,,o,
(12)
follows that P Pi. for any strictly increasing h.
13

---

Preference learning made easy: Everything should be understood through win rate
D. Target distribution derivations
D.I. WRO-KL
Here, wc show that the WRO-KL objective is equivalent to minimizing the reverse KL divergence of the model and following target distribution:
PiRO-KL (y x) x Prerly x) expl R
TeelgBBuoe Epo p(e= 1|x Jo.
(13)
Derivation:
max pp(x) [Epots Ix) Bptro uy p(e- = N.)o (po (x) Prerly x))]
(14)
Epiro x)lh- pill-pp(e- = X. No. v1)] -loog po (y x)
min
pp)( poo (x)
(15)
0
Pret (y x)
log Po [x)
1
-min Bp(x B polly, (x)
Bris 1x) p(l
81)
(16)
0
Pr x)
B
ps xx)
=min Bppx) E Bolty, (x)
log
(17)
Prerly| x) exp(( EE x) [h p(t = 1
paly|x xx)
min Bp(x) B Spoly, (x)
log
Z(x) Pre x) exp( Epoo 1x) . p(l = (.)) log Z(x)
(18)
mmin Ep(x (po x) PW xx)]];
(19)
0
PiRO.I (y (x) Praly xx) exp (3 Episow p(e = 1/x.
(20)
D.2. RLHF
Here, we show that, under the BT assumption, the RLHF objective equivalent to minimizing the reverse KL divergence with the following target distribution:
Py x) x Prerly| xx) exp( Eptraix) (logit p(c = yo.
(21)
max px))) Spolyis)r(xx,)) (KL( x) Prerly xx)] 0
(22)
max (p)) [ppoty Ix) Eptro (x) (logit p(E = 1 1|x, y)] - (y (x) ll Prorly xx)]
(23)
min Ep(x) Epo(slx) log (po (x)
Prerly x)) 3 Enso syllogi p(e 1 Nx.
(24)
min Ep(s) (po (yl x) ll (y [x)]].
(25)
cxp P (x) x p(y| x) exp (3 B p(l = 1|x, No.
(26)
E. Comparing target distributions
In Figure 4, we present visualization of the target distributions of different preference learning objectives. Panel (a) shows an example preference environment defined by the true preference probabilities between responses (left) and initial starting model (right). Panels (b), (c), and (d) visualize the resulting target distribution of different objectives. In each, the figure on the left shows how the average preference probabilities Eply. xx 'p(E = 1 x, No. v,] (red dots) are translated into the tilt applied to the starting model, i.c. g(y, x) in p' (y x) x p(y x)g(y, x) (green dots). The figure on the right shows the optimal distribution under the objective. Notably, among the WRO-KL family, the choice of h can make a substantial difference in the target distribution of the objective (panels b vs. c). Moreover, SET is limited in how much mass can put on the preferred sample A, as g(y, x) is only the average preference probabilities themselves (panel d).
14

---

Preference learning made easy: Everything should be understood through win rate
1.0
1.0
1.00
0.8
0.778
0.8
0.7
0.8
0.737
0.75
0.6
0.6
0.6
A
0.6
A
0.4
0.4
0.4
0.3
0.50
B
0.4
0.2
0.25
C
B
0.235
0.2
0.2
0.0
0.0
0.00
C
0.0
0.028
A
C
A
B
C
B
B vS
A Avs
Average preference
DWRO tilt, normalized
(a) Initial generator and preference probabilities
(b) WRO-KL, æ= identity, 0.1
1.00
+A
1.0
0.981
1.00
1.0
0.8
0.8
0.75
0.75
0.6
0.529
A
A
0.50
B
0.50
B
0.4
0.4
0.333
0.25
C
0.25
C
C
0.2
0.2
0.138
0.00
BC
0.019
0,0
0.00
0.0
A
B
B
Average preferences
Average preference
RLHF tilt, normalized
SFT tilt
(c) RLHF// DPO, h= logit, 3 0.1
SFT on preferred samples
Figure 4: Different preference learning objectives have different target distributions. Consider the initial setting in (a). WRO-KL with ı== identity, 3 = 0.1 yields the target distribution depicted in (b). RLHF (h= logit, 3 0.1) yields sharper distribution given that average logit probabilities can differ more from each other than average probabilities in (c). SFT yields the least sharp distribution as it can only apply weights between and (d).
F. DPO win rafe-correspondenee counterexamples
F11 Example where data is sampled from original model
Consider query environment p(x)
[x = x') and sample space of three responses, Yor No. Under the preference environment of interest, ya is preferred over Ys and both are very preferred over y., i.e., Ys >>>> Ye- Let starting anchor distribution (yer(y x) be supported over y. and yn only, where Prer(y, x') > Prerly, x'). The dataset is generated from the initial model itself.
Consider pe (y x) that ranks ya and y, correctly but places most of its mass on ye- Then, (CDPo x)) will be high, higher than the LDPO (Perly x)), yet (y x) will generally output response that is dispreferred over one generated from P x). i..., Ya vs. Ya or Ys- Note that this example even considers online data (i.c., from the initial model) and no estimation error from finite samples; however, even so, the DPO objective does not respect prevalences from the generator itself, which can also be seen as a failing condition Item 2a.
F.2 Example where data is sampled in fully online manner
We now consider the DPO objective for each under a fully online setting, i.e., one response sampled from the anchor, another sampled from the generator, and the ability to obtain the oracle preference probability for every response pair. We show below that even then, DPO does not obey win noteccorrespondence.
Consider query environment pxx) [1] x') and a sample space of three responses, Ya. No. where y, >>> yo x
15

---

Preference learning made easy: Everything should be understood through win rate
0.8
0.75
0.70
win rale
0.65
win
0.60
0.6
0.55
0.50
0.5
0.45
0.50
0.55
0.60
0.65
0.70
0.4
0.5
0.6
0.7
online DPO loss
online DPO loss
(a) p( = X.. = .9. p(t = 1 Xx,.o. = .6,
(b) pl = X6, = 99, p(e = No: = .9,
pl( = 1 X.Yo 93.
ple = 1 Xx. = .9.
Py,y, x) Poly
(y
1/3.
PY x) Poly, [x) Po x) 1/3.
Figure 5: Plot of win rate vs. DPO loss for different (y x) under given setting of Bradley-Terry preference classifier p(é = 11 x, Yo. Y1) and imitial and anchor distribution (y x). These plots are analogous to Figure 1 but with different experimental settings. Each point on the plot is a different instantiation of ( x), sampled from a Dirichlet with a = 1 uniformly over all three responses. 3 = 1. Grey dotted lines are the values the win rate (horizontal) and DPO loss (vertical) at initialization. Any two pairs of points that would form a line segment with positive slope is a pair of generator settings 0, 0' such that win rate-correspondence does not hold.
Ye under the preference environment. Let starting anchor distribution Prer(y] x) rank the responses as y, > Ye >>>> Ya i.e. the rank ordering of y, and ye is correct, but ya is very unlikely relative them even though it is most preferred. In particular, assume Bradley-Terry preference classifier that greatly prefers ya to yı, i.c., p(l = 1| Yo. mildly prefers y, to ye- i.e., p(f = 1 No: starting model assign probabilities .1, .5, and .4 respectively to yar -ontaalyy it's ranking for pairs with ya is very wrong but its ranking for y, vs. y. is correct.
.9, and only Vo) .6. By the BT assumption, p(e = X. ya) = .93. Let the original
This is the setting used to generate the plot in Figure the main paper. We generate points on the plot by sampling p(((y) x) uniformly from the probability simplex. We only plot points whose loss improves upon the initial model (as those the practical model instances consider during training), and sample until the plot contains 5000 points.
We now provide a concrete example of (y x) and g x) where - (Pe) > - yet k(ps) < d(g)) words, pa has better DPO loss but worse win rate than qa. Namely, we construct pa such that it continues to increase the correct probability gap between Y, and y. without adding probability mass to ya, while we construct go to improve the probability y. but at the expense of worse probability gap between y, and Ye- As a concrete example, set po(y, [x) = .1, Po(y, x) 66, (y Ix) .3, and (y, x) .8. (y x) ,001, (o a x) = .199. The online DPO loss for pe and 4o is 51 and .78 respectively. The win rate the original model, however, is .54 and .67 respectively. The logit-win rate (i.e., h=logit in Equation (1)) is .66 and 1.7 respectively.
The intuition for large mismatch in this example comes from the differential impact to the DPO loss between improving the ranking of the initially incorrect y. vs. Ys vs. the initially correct Y& vs. duedue to the role of the initial model log probability margins on the loss shelf-comppred to the differential impact to the win rate between increasing the probability over ya versus yb.
However, this example is just one instance of the mismatch between DPO and win rate; there exist mismatched pairs even when the initial model is uniform over responses, well as when the BT preference classifier maintains the same probability gap between neighboring examples in the rank order; see Figure 5 for visualization.
The more fundamental issue is the fact that Equation (4) inside the DPO loss is meant to approximate the preference classifier, but this approximation is only correct when the model is exactly the target distribution for the objective. Then, even in the online setting where the expectations consider the correct distributions, the estimate of the preference classifier is incorrect, i.e., po x,y', y) # p(l x, y.y' y), meaning the DPO objective is not the same as win rate under the
16

---

Preference learning made easy: Everything should be understood through win rate preference environment of interest. Namely,
-prro(poy) xx) pp) x)poly xx)p((( lxy''") + p(e
0/x.y',y
+ pox xhooly w)(p(( x,y', h-Win Ratep(y, (x) xylpoy |x]].
E.3. Example that also fails regularized win rate-correspondence
Below is counterexample prove that DPO does not obey regularized win rate-correspondence, which is the property that an improvement in the objective implies win rate improvement decrease in the divergence to the reference model.
Consider the setting described in the previous section: query distribution is deterministic with single prompt universe of responses to the query are (y.: - Vol. preference classifier follows Bradley-Tenee Terry assumption with p(f = y, V,) ) .9 and p({ = 1 x. Nb. N.) = -6, and initial model Py(y x) has probabilities .1, .5, and .4 over Ya. and y, respectively, and .6, .07, and .33 over Ya Vo: and Ye- vs. qo(y x) be .56, .43, and .01 over Na. and y.. Then, C(pe) .59- < C(ge) .64, i... Py has better DPO loss. However, pe has both worse win rate and larger divergence from the initial model, with win rate and logit-win rate of .69 and 1.2 (vs. .70 and 1.3 for qo) and reverse-Ki KL divergence of .87 (vs. .86 for 4)).
Let Po (y x)
The intuition for this counterexample is that pe has slightly more mass on the best response than qe does, but has qo has much more mass on the second best y, than Pa, which places the remaining mass on y.- other words, Py and qo have different strengths. this specific setting, P9 gets better DPO loss whereas qe gets better win rate over the original model as the small difference in ya mass matters less than larger mass difference between y, and y.. Ultimately, go slightly closer the original model, which had most of its mass on Yo.
The broader takeaway is that although DPO has the same target distribution regularized WRO objective, the loss is not the same as a linear combination of win rate and explicit regularization term, such that it is possible to improve the DPO objective in a way that does not improve win rate or the regularization.
G. SFT proofs
We provide results for SFT the preferred sample in pair, well SFT with arbitrary filtering based pair and its preference.
We first provide a more general setting of the theorems in the main paper (Lemma G.1 and Lemma G.2); then we specialize to the setting in the main paper, where p(yo x) ply, x), i.e., both samples come from the same distribution.
Lemma G.1. Let plyol (x) be the initial generative model, and psprly (x) be the target distribution of supervisedfinettunnng on preferred samples Fo.N1 ~ P(o.31 (x) = p x))((, (x). Then,
Win Rate pb. 1x) (Psrr [x)]
(27)
Win Rate pba xx) ppr xx]] / p(x)
Variance pór, (x) pSr x))((e = 1 y,
dr.
(28)
Jol: (x) p(e 1/x, D'rro x)dy)dy
17

---

Preference learning made easy: Everything should be understood through win rate
Proof.
Win Ratep(y x) [PSEr(y (y [x]] = / p(x) x)PSET(Y x)p(( = 1 No.
(29)
ply, x) p(8 1|x, yo: N.lly' x)dy'
= p(x)p(yo x) Joly' x) f ple = 1 X, y. Dis dydy"dy" p(e = 1 x. No N2)dyddy,d ply x) p({
(30)
/ p(x)
|x. Yo. y Pay' xjdy
p(yo x) ppyi x) f p(( 1|x, Y8 11) dyd p(e 1 x, Yo: V1 Joy
dx
(31)
/ ply x) p(y' x) J'([ = 1x y0 Fi 1) dyd ple = 1 x. 32) Joyyddy dx
ply: x) p(e 1x, v, x)dy'
p(x)
(32)
/ Pply () = 1 x. No
xx) p(e= 1 y )p(y6 x)dy dy, dx
p(x)
(33)
p x) f p(c v0. (y dyd
/ p(x)
TU pLyo x)p(e - x, y1 / [x)dy dx
(34)
T S p(c
y1 ( x)dy6ds1
(S (oorooX)p(( x)p((( x. you y1 Joy p((; (x)dy,) 27 SpyL' f(( = 1|x, Vo (F dyd
v(x)
dx
(35)
+ f p(x)
Variancepts, (x) [S U(Fo x)p(( -= No. )dyo))] |p(y)x)) p(E = y0x P dy
dx
(36)
/(N)
plyo ix)p(e = 1 11 V1 dy),)) ))(() |x)dy,dx dx Variancepís, (x) [ p(yo x)p(( x Yo. N1 ddy,)
(37)
+ p(x)
dx
(38)
JPl'' x) J ple 1/x, 11 dyy
-Wi Ratepo s)b(y,, xx] + ff p(x)
Variance ply, 1x) [ p(yo x)p(e=11x )dyo)
dx.
(39)
pPls x) p(e p( 11x Fo d)d
Lemma G.2. Let plyo (x) be the initial generative model, and Pyterly (x)x be the target distribution of fietanngg on samples where the filter f = is potentially randomized data point filter p(f 1x.Y1.Yo. E) that selects data for supervised finetuning finetunnng
based on the any of the context observed x, generations y ViyYo and preference E.
Pyteer(yxx) = ply, Ix.f f = 1) ply
= 1 p(f 1(x)
plr xx) Epts 1x.)) IP( 1 1 0] E Epo,ya x)) jp(f = 1 ,x, []]
Then,
Win Rate, (x) (pptter (r |x)]
Win Rate (x) [ply, [x)] + f / p(x)
Covnin (Eplevo 1x) Ix) 1 1o.
1x) lp(e Enoy,
16x, dx.
E
Epo Ip( = 11x, Q)
18

---

Preference learning made easy: Everything should be understood through win rate
Proof.
Win pate (x) [Paner (y) x)] p(x)p(yo (y x)p((= 1 No: No:
= p(x)p(F0 x) p(y1 EpEE Ixs, Ix,, 1 C)
Epiy, Fo (1 (p(s = y1: p(e Jdyyddyy
f p(x)
ply x) Epcero ko,
O)]
plyo 1x)
p(e 1 No V1 dd
dx
(1x) 1 0
= -f p(x)
p(y1 Bppee Ixy, Ip(s N,y1, 2) plx, (x) 1p(e 1 x, No: V1l
dy
dx
E [P
No: 2)
f p(x)
E Eptr (x) Eptex xy, x [p( x, y1 No: 2) Pay. x) ip(e 1 1/. v1)]]
dy
dx
Pply Jo. ( [
No ))
Cov 1x) (E Ixy, 1xy))li [P( = 1 O) Epiry (x) pp(( = 1|x, No. v111)
p(x)
dx
Epty
N1. No:
+ -D(x))
Epleson (x) = 1 X 11. No:
(p(e- (p) 1, No. dx
p(x)
Epis, No. [P[ = x, ())
= - p(x))
[oovpy, 1) (EE (n)p(( No: 3n vi)] Epdr (x) molppe 1|x, V1) dx + Win Rate (x)
p(x)
(x)pp(y xx)
2 (x) = x1
For SFT on the preferred sample, p(f Yo. C) 1le 1]. Other options which decrease the denominator in the second term while maintaining covariance (e.g., quantile filtering based on p(E x have the potential to enable further win rate improvements.
Theorem 5.1. (Win rate improvement of SFT on preferred sample) Let plyo |x) be the initial generative model, and finetuning on preferred samples (ply, x E 1), plyo x) = ply, (x))
PsFrly x) be the target distribution of finetnnng Then,
/piro // piro (x)p(( = (
Win Rate plr, (x) (PSET [x)] = 0.5 2xx Varptr, (xx
(40)
which is less than 1.0 as long as there exist at least three responses in supp(ply) xx)) for any x.
Proof. We first use the result of Lemma G.1 and plug in the condition p(yo xx = p(y, xx):
Win Ratepóro (x) ()SPSF(( xx]
(41)
0.5 + f p(x)
Variancepts, (N) (xp0)_xxppee111/pp/
dx
(42)
J Jr(y x) p(e =
dydy'dy'
0.5 2 / Varianceppt lx) [ ((xolx)p(e==1) x. 1
dx.
(43)
I "pxo x))((E 1|xx No. yyy can only take values between 0 and 1. Win rate is maximized for random variable taking
values between and one when the variance is .25, but Proposition H.1 proves that this is unachievable long as P(No x))((== = No. Yi
(0, 1) for any y1- Proposition H.2 shows that f p(yo x)p(( = Fo E (0, 1) as long as there exist three or more elements in supp(p(y |x)). Thus, as long as there exist at least three responses in supp(p(y] x)) for any x, win rate ove the initial model is less than 1.
Note that under deterministic and transitive preferences that are evenly spaced apart, Theorem 5.1 matches the win rate result in Theorem 2 reported in (Gui et al., 2024) for best-of-n when n = 2. In other words, Theorem 5.1 generalizes the best-of-n result to general preference probabilities.
19

---

Preference learning made easy: Everything should be understood through win rate
Theorem 5.2. (Win rate improvement of filter + SFT) Let X.N: {) be filter that selects data (S 1) for supervised finetuning, and p(yo x) p(y, xx)). Then, denoting Pyles lx) [p(S No:N1:) as AvgFilter(x,y)) and Epixo (x) [p(e= 1 Yo: N1)] as Av))
Win Ratepty, (x) (Prr(y x)] = 0.5 + pxxx) Covpiy, x)) AvgFilter((xxx (.gefiier(x.)))...vvee((.....
(44)
Boi, 1x) AvgFitte(x.)
Proof. We use the result of Lemma G.2 with the condition p(yo (x) =p(y, lx).
H. Additional SFT Propositions
Below, Proposition H.1 and Proposition H.2 support Theorem 5.1, the result that the win rate improvement for SFT on preferred samples is limited.
Proposition H.1. Let X be a random variable over [0, 11. Then, Var(X) < 25 if < <1)>0.
Proof. First, recall that for convex function f. f(x) f(11 - x)) ++ 1) < -1-x) f(0) - x f(1). Then, letting
f(x) = (x - 11²2, IEX, the variance of a random variable is upper bounded as follows:
Var X)
E[(X - n]²]
< E[([ - x)- f(0) + X f())
-x)
E[(1 -))x² + n)
(1 p)y² + p(1 p
n1² 1º l 2/² 11
112 = pb - p.)
For distribution over [0, 1]. this quantity is maximized when = .3. resulting in variance of .25. This variance is achieved when X places mass on only 0 and symmetrically: Var(X) .5(0- - 5) this distribution that changes μ will have a lower upper bound and thus not achieve variance .25, and any other change that does not change y but moves probability mass away from the endpoints will only decrease variance.
-5)2 + .(1 - -5)² = .25. Any change to
Since the win-rate is Win Ratepor 1x) (psrrry xx]] = 0.5 + px)( Var (x) proposition establishes that Ep(ro [x) (p(L = No, y1)] has to be zero or one in order for win rate to be I. For a fixed y1. this expectation being one implies
[E xx) Ip(e xx Jo.Y (.)ll
[]] the former
Epir, (x) [p(E- = =1 x = (VY. E A}, for an such that EpGo [y E A]]
More plainly, y, must always be preferred over every generation with probability one. By symmetry, the expectation being zero implies that y must always be dispreferred with probability one over every generation.
Being always dispreferred or preferred is impossible due to self-comparison, e.g., for fixed Y1, the expectation over p(yo x) will include y, in the support when samples generated from the same model. Omitting self-comparisons, as long the model has support over more than two generations, Epts, xx) [p(e No) must be less than one for at least one y, E suppíp(y, [x))
Proposition H.2. Let ply xx) have support three more elements. Then there exists B in the support ply x) that is neither always preferred or dispreferred over every element in the support of ply (x).
Proof. Let a: be an element that is deterministically preferred over every element in the support of p(y [x), then by symmetry of (e- 1 Yo. Y1) no other element can be deterministically preferred over every other element. Preferences for at p(E- = 1 y = No (3) 1 and p( y, a.Yo 7) =
Then let 7 be an element that deterministically dispreferred over every element in the support of p(y x). Preferences for 7: p(E = y, = 7. = a) = 0 and p(E = x, = 7. = 3) = 0. The preference probabilities for are fixed by symmetry from the above assumptions
20

---

Preference learning made easy: Everything should be understood through win rate
Since p(e 1 x, N.N0 No = () 1. then p(e x, =,,1 7)
Since p(e X, 11 7:3o 3) 0. p(p
- 1 = 3. = 7) =
Thus, the 3 generation beta is not deterministically preferred or dispreferred.
Since being always determinstically preferred dispreferred is generally not possible, SFT's improvement is limited.
I. Expected Win Rate Improvement Expressions
Below, we present the expected win rate improvement over the original model for WRO-KL objectives. Letting p(yo x)
p(y x) = p(y6 x) and p(y, x) = p(y' x), we have:
Win Rate (x) (yWPRO-KL (y x)]
Epor. (x) Eptro xip(l 1 exp ( Bpss h
pll 1 Nx.
= =/ p(x)
(45)
dx.
Epti (x) exp ( Eplp x)lt p(e 86.))
Consequently, assuming the BT assumption holds the preference environment, the expected win rate improvement for the target of RLHF//PO as follows:
Win Ratep(ro (PRLHEDPPO x)]
- -/ p(x)
Eprr (x)
Epivo x)p(l 1 x X,, V1) exp
Bprr pir (x) [logit p(6 - x, Vó. 16..%11)
(46)
dx.
Epo: xx) exp
} Epor (x) (logit p(e = 11x.
For completion, we write the expected win rate improvement for SFT in the same form, connecting the expressions in this section the result of Theorem 5.1:
Win Ratep xx) (PSFFF(() xx]
= -f p(x)
Epty, x) [Eptro iNsple = N1) 1 1x
(47)
dx.
Epirt (x) Exp ix)lp(e = 1, F0.
J. Comparing methods
In Table 3, we compare various preference learning methods along the axes of target distribution, objective, and the method of estimating the preference classifier. Ideally, we want method's target solution able put more probability mass over the more preferred sequences; its objective directly seeks to approximate that target over some other goal; and its estimate of the preference classifier is as accurate as possible.
Table 3: Comparison of preference learning algorithms along three dimensions of design choices: target distribution, objective, and estimation of the preference classifier. For readability, we substitute the preference classifier distribution p(E = 1 xx y) with Par-
Target (unnormalized)
Objective (per-query)
Preference Classifier
WRO-KL
p(y x) exp( 1
a T 1x [h Paul)
(Lo x) p0(yy x))
Per
RLHF
ply x) exp( @ E
Eptr, flogit Peal)
(x)
KL (y x) || p1 [x))
f(x, y). BT
DPO
p(y exp( Epiro (logit Par))
KL || poçar)
Equation (4), BT
SFT
p(y x))Epr [Parl
(p (y x) Po(y x))
Bypass
21

---

Preference learning made easy: Everything should be understood through win rate
K. Judge model
To train the judge model, we finetune Pythia-2.8b base model using the same pairwise training set used to train all SFT models by simply modifying the prompt take in x, No into account. The prompt template is as follows:
Anin Human: + <Instruction>
Anin Candidate Response A: + <response a> +
Anin Candidate Response B: + <response b> +
In(n The better answer is Candidate Response
The model is trained with a language modeling loss on the output, which is either 'A' or 'B'. For each pair of responses 11 the training set includes two rows, one where Yo Response A, and one where y1 is Response
The model is trained using RMSProp with a learning rate of 5e-7 and batch size of 64, The model is trained with maximum sequence length of 512 and maximum input length of 511. On the evaluation dataset, the model achieves per-row classification accuracy of 68.8. (Training the same judge model with sequence length of 1024 achieves the same accuracy, so we choose to stick with 512 for efficiency.)
To obtain the preference probability of a pair of outputs, we run forward pass through the judge model twice, once with each order of output pairs, and average the results.
To simulate more opinionated preferences in this preference environment, we sharpen the judge preference prob- abilities with temperature scaling (T' 0.2) the logit-transfommed probabilities: p(f Yo: (logit Pjudgo (C x, Yo.Y))
L. Experiment details
Dataset processing. For the Open Assistant dataset (Kopf et al., 2023), we keep only the first turn in each conversation and English-only examples, following Yuan al. (2024). The dataset only has a train and validation split, so we split the original train set into train and validation set and leave the validation set for testing / evaluation. The dataset includes multiple candidate responses for each input, all ranked, and to match the pairwise preference learning setup, we create dataset of all possible pairs for each input. For instance, for given input with three candidate responses (A, B, C), our paired dataset includes all three pairs (AB, BC, AC). For SHP, we keep only pairs where the score ratio is >> 2, following Rafailov et al. (2024c).
Training the initial model. For each dataset, we finetune the base Pythia-2.8b models on all outputs, preferred and dispreferred. The resulting finetuned models serve as our initial models for preference learning. To train these models, we utilize batch size of 64 and learning rate of 5e-7 chosen based hyperparameter sweep between [le-8, 5e-8, 1e-7, 5e-7, 1e-6] on OASST. Following Rafailov et al. (2024c), we use the RMSProp optimizer with a learning rate warm up of 150 steps and constant learning rate schedule otherwise. We evaluate every 100 steps and choose the best checkpoint based on validation loss.
SFT and DPO experiments. SFT and DPO experiments follow the same training configuration as the initial model.
RL experiments. We use the implementation of reward model training and PPO from the TRL. library (von Werra et al., 2020). For reward model training, we use a batch size of 64, learning rate of 5e-7 for Pythia2.8b. and checkpoint every 100 steps, matching the SFT and DPO experiments. For PPO, we use learning rate=le-6 (obtained through hyperparameter sweep of [1c-7, 5c-7, lc-6] on OASST), batch size=128, and PPOConfig defaults for all other hyperparameters. We checkpoint every five steps and choose checkpoint with the best policy loss (namely, ignoring the estimation of the value head).
Win rate evaluations. We sample a set of 100 input prompts from the test set a given dataset (same 100 prompts for all models) and perform win rate evaluation using the oracle judge for the dataset.
22

---

Preference learning made easy: Everything should be understood through win rate
M. Exploring optimization benefits of RLHF
Below, we compare the win rate results when optimizing the RLHF objective versus a non-optimized WRO-KL-BT-logit objective that keeps the Ep(x, (x) term. Namely, the RLHF objective we optimize
LRLII (0) max Ep(x) E po ( y)]
(K x) Pre(yy [x)]-
(48)
The non-optimized WRO-KL-BT-lopy objective optimize is -CRLHH (0) = mnxx Ep(x) Ep (x)()))))(((x))))))))))))))))))) Eptx, (x) (Pa Pe [x)]]
(49)
Win rate results be found in Table 4. All experimental details match that of the main paper.
Table Comparison of win rates between RLHF (optimized) and WRO-KI-BBT-loir without additional optimization (non-optimized), RLHF's dropping of the constant in the objective decreases variance in the gradient estimates and generally improves results with small enough B. However, this change does not yield systematic benefits for every setting, suggesting that there is still room for improvement.
0.001
3= 0.01
B = 0.1
3=1 1
Dataset
HH
non-optimized
51.79 (1.12)
54.59 (0.90)
68.68 (0.80)
56.88 (0.46)
optimized
70.46 (0.87)
63.93 (1.24)
69.44 (0.90)
54.60 (0.67)
OASST
non-optimized
60.74 (3.65)
61.79 (3.64)
73.11 (1.16)
60.41 (2.05)
optimized
62.05 (3.65)
63.95 (3.38)
66.72 (1.16)
58.45 (1.82)
23


Reviews:
[{"review_id": "JJvMXBzh1O", "reviewer": "Reviewer_nt27", "summary": "The paper argues that win rate should be the primary evaluation metric in preference learning, as it is the only measure that respects both preferences and prevalences in pairwise comparison data. The authors introduce a win rate-centric framework and classify preference learning methods into WRO and non-WRO approaches.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review. Responding to your concerns below:\n\n1. [Overly idealized assumptions (in-distribution assumption, unreliable preference models)  diminish practical significance of conclusion]\n  - Our framework is not focused on idealized assumptions per se but rather what can be learned from the preference data distribution alone, specifically to clarify the role of assumptions vs. information inherent in preference data itself. In fact, our analysis is meant to make it easier to consider challenges such as those posed by the reviewer: Namely, our work clarifies what is optimal when nothing else is considered besides the information contained in preference data alone. This understanding of what is in preference data makes it possible to disentangle anything extra in a given method as an additional assumption or strategy. We are not arguing that anything extra is bad if the resulting objective is not pure WRO; instead, win rate optimization should be seen as the starting point, with any addition or modification understood modularly based on the underlying assumption being encoded.\n  - To drive that point home, we've added the following lines to the discussion: \n    - …this work offers a simplifying insight: win rate is the only evaluation that can matter based on preference data alone, and thus all of preference learning should be understood in relation to it—both how well a given method optimizes for win rate as well as the role of additional assumptions that move a method beyond pure WRO. \n\n2. [Limited practical guidance for real world preference learning]\n  - We respectfully disagree; some immediately implementable practical guidance that comes out of our analysis includes 1. given how important and finicky optimization success is for WRO objectives (including RLHF), it could be helpful to kick off multiple seeds to find one that optimizes best; 2. given that DPO loss fails win rate correspondence on multiple fronts, one should consider performing model selection with a metric other than validation loss; 3. greater generation diversity as well as alternative filtering strategies can improve the win rate limits of supervised fine tuning on preferred samples; 4. studying optimization strategies (e.g.,  contemporaneous paper [1]) would be a high leverage direction of inquiry, given the importance of optimization success in practical performance of preference learning algorithms.\n\nBased on our response, we hope you will consider raising your score. Thank you!\n\n[1] https://arxiv.org/pdf/2503.14286", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes,", "relation_to_broader_scientific_literature": "The authors’ analysis is well-reasoned; however, I believe their theoretical framework relies on overly idealized assumptions, which diminishes the practical significance of their conclusions. I agree that win rate is the most aligned metric with preference, but the analysis overlooks key challenges:\n\n1. In-Distribution Assumption: The authors assume that training and evaluation data are drawn from the same distribution. In practice, training data cannot fully cover real-world applications, necessitating regularization to improve OOD generalization.\n\n2. Unreliable Preference Models: The preference model may be biased or unreliable, which is why additional optimization objectives—such as length regularization—are incorporated into preference optimization. While these objectives break WRO, they often lead to better empirical results.\n\nGiven these limitations, I disagree with the claim that \"everything should be understood through win rate.\", and WRO methods, as acknowledged by the authors, not necessarily outperform non-WRO ones. While the there are some theoretical insights, the conclusions provide limited practical guidance for real-world preference learning and this area.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tfrU5JJgPk", "reviewer": "Reviewer_1f3H", "summary": "This paper introduces two consistency measures to study preference models. The paper proves that the only evaluation criteria that respects both is the win-rate. This finding is generalized into h-WinRate -- win rate under a monotonically non-decreasing transformation $h$. This measure is used as an optimization objective, called win rate optimization (WRO), which is later used to study preference learning algorithms. The paper shows that while RLHF respects this objective (under KL-regularization), DPO and SFT do not. Empirical analysis of DPO exhibits that improving DPO loss does not necessarily improve the win-rate. Further experiments with varying $h$, $\\beta$ and preference model suggests there is no one winning setting across different benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review. We answer your questions (paraphrased for brevity) below:\n\n1. [Theory in the paper is similar to IPO. Proposition 4.1 can be proven using equation 7 in IPO with $\\tau$ going to zero. More empirical analysis can distinguish the paper.] \n- We agree that one can use equation 7 in the IPO paper to gain intuition for proposition 4.1, but equation 7 in the IPO paper is the target distribution specifically for KL-regularized WRO objectives and does not include the additional analysis required to take the limit of $\\tau$ to zero (e.g., proving existence of a limit, connection between limit of solution and solution of limit`)  nor the specific implication of the Bradley Terry assumption. In contrast, Proposition 4.1 in this work  is meant to emphasize a consequence of the Bradley Terry assumption on the dependence of WRO on choice of anchor distribution: Namely, it proves that under the Bradley-Terry assumption and finite rewards, the resulting solution optimizes win rate over all possible anchors regardless of the choice of anchor in the objective. In other words, the Bradley-Terry assumption effectively gives us the ability to optimize over all anchors, whereas without that assumption we choose the anchor. We have updated the text to emphasize this point.\n2. [Expected versus observed win rates; the authors suggest there might be other factors at play, such as optimization etc., but these are not supported by any empirical evidence.]\n- Section 6.3 / Figure 3 provides empirical evidence of optimization being an important factor for the misalignment between expected versus observed win rate patterns. Namely, the train loss achieved by a given run is significantly predictive of the test win rate, even as other factors which dictate the target distribution of the objective are not (Table 1). This result is especially striking given that the losses for different objectives do not necessarily share the same scale. \n3. [It is interesting to me that using oracle $\\hat{p}_\\ell$  doesn't exhibit the best performance]\n- We agree. This result suggests that there is a more important first order factor, which Figure 3  (a scatterplot of train loss versus test win rate for selected models across different WRO runs), and the corresponding significance test suggests is optimization success.\n4. [The paper uses only Pythia-2.8b model to train preference models as well as to judge results. I think this limits the analysis. Using the same base model for both preference learning and judging can introduce biases. It would help if other similar sized models, such as Gemma2-2b, Qwen2-1.5b, are used as well.]\n- Due to resource constraints, we are unfortunately unable to run the same experiments on additional models at this time (requires 7 offline finetuning jobs and 36 online finetuning jobs for just a single dataset with no hyperparameter sweep), but we take the reviewer's point that it would be helpful to add more models to the final paper and are working to do so. And just to clarify, we are using separately trained models for the judge and policy model even if they are based on the same base model.\n5. [I think proposition 3.3 and definition 3.2 can be practically useful to check whether a preference learning method respects the win-rate objective. The alternative, comparing the objective directly to win-rate objective, can be more challenging.]\n- Thank you for highlighting the practical usefulness of proposition 3.3 and definition 3.2. We agree!\n6. [Section 4.2 notation] \n- Thanks for the feedback, fixed!\n\nWe hope we have addressed your concerns, and we hope you will consider raising your score. Thank you!", "claims_and_evidence": "The paper claims that win-rate is the unique measure that should be used to evaluate preference models. This is also supported by other work in literature, including IPO or BonBon. The paper hypothesizes two consistency measures that uniquely suggest win-rate is the only evaluation that respects these measures. These claims are supported by theoretical analysis of RLHF, DPO, and SFT.\n\nThe paper empirically shows that DPO-loss is not necessarily correlated with win-rate, which is suggested by the theory. But the remaining experimental results are mixed.\n\n- One main criticism is that the theory in the paper is similar to IPO, especially the optimization and analysis of preference learning. Similar to the paper, IPO uses a non-decreasing function with win-rate. The proposition 4.1 can be proven using Eq (7) in IPO with $\\tau$ going to zero. I think having more empirical analysis can distinguish the paper better than related work.\n\n- Expected win-rates suggest a global ordering amongst RLHF, DPO, and SFT, with a monotonically non-decreasing relationship between win-rate and $\\beta$. But, empirical results suggest otherwise; DPO performs the best and RLHF is slightly better than SFT. While the authors suggest there might be other factors at play, such as optimization etc. these are not supported by any empirical evidence. \n\n- It is interesting to me that using oracle $\\hat{p}_l$ doesn't exhibit the best performance. Given that the true objective is to improve the win-rate under the oracle preferences, this is counter-intuitive.\n\n- The paper uses only Pythia-2.8b model to train preference models as well as to judge results. I think this limits the analysis. Using the same base model for both preference learning and judging can introduce biases. It would help if other similar sized models, such as Gemma2-2b, Qwen2-1.5b, are used as well.", "methods_and_evaluation_criteria": "The paper studies win-rate through newly-proposed measures and introduces a new objective for preference learning. It compares different methods under win-rate; which is the main metric of study. There are two datasets that are commonly studied in the literature.", "theoretical_claims": "I checked the correctness of proofs in the main text.", "experimental_designs_or_analyses": "As I explained above, experimental results are limited. I think using other similar sized LLMs, analyzing the results in more detail would help the paper.", "relation_to_broader_scientific_literature": "The paper is broadly relevant to preference learning.", "essential_references_not_discussed": "Related work sufficiently covers the literature.", "other_strengths_and_weaknesses": "I think proposition 3.3 and definition 3.2 can be practically useful to check whether a preference learning method respects the win-rate objective. The alternative, comparing the objective directly to win-rate objective, can be more challenging.", "other_comments_or_suggestions": "One minor comment is about the notation. Functional composition in section 4.2 is different from the rest. Please use a consistent notation.", "questions_for_authors": "Please see above respective sections for related questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hMUjRhCsdn", "reviewer": "Reviewer_7gys", "summary": "The paper examines what constitutes a \"grounded\" evaluation of a policy or language model's alignment with (human) preferences. Assuming the evaluation function is both preference-consistent and prevalence-consistent, meaning it is linear in the distribution of contexts, the distribution of alternatives, and the policy under evaluation, the paper establishes that the only grounded evaluation metric is the $h$-win rate.\n\nBuilding on this, the paper argues that maximizing the $h$-win rate (for some $h$) is a natural objective for aligning generative models. This approach offers two key benefits: (1) win rate correspondence, meaning that optimizing the objective directly improves the $h$-win rate, and (2) win rate consistency, ensuring that the optimal policy obtained from the optimization also maximizes the $h$-win rate. RLHF is one such win rate optimization (WRO) algorithm. However, methods like DPO and SFT do not optimize win rates. In particular, DPO lacks win rate correspondence.\n\nThe paper then presents an empirical study to test whether theoretical insights translate to practice and whether WRO methods outperform others. Interestingly, the empirical results contradict theoretical expectations, highlighting the importance of optimization and other influencing factors.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review! Questions (paraphrased) and responses below:\n1. [Undesirable that optimal solution depends on anchor, not true for reward maximization]\n- This dependence is a limitation of the information contained in the distribution of pairwise comparisons itself. As mentioned in Prop 4.1, we agree that the issue can be bypassed with additional assumptions (e.g., under BT and finite rewards, optimizing any ℎ-win rate over any anchor optimizes for all ℎ-win rates over all anchors). We have added more discussion.\n2. [Better motivations for properties, prevalence-consistency for generator rules out non-RL]\n- We want preference-consistency or else a model can be deemed good even if it generates dispreferred responses. We want prevalence-consistency wrt generator or else we are not evaluating the model’s generation behavior. Enforcing prevalence-consistency means objectives include an expectation over the model, but this could be approximated with off-policy samples and importance weighting for instance. \n3. [I don’t think you need BT to prove Prop 4.1]\n- Good point. Prop 4.1 is meant to show what BT gives us (see 1), but we have generalized the proof.\n4. [Two anchor/competitor distributions is confusing]\n- Could you clarify what you mean? If you mean $p_{anchor}$ vs. $p_{ref}$, we distinguish them to highlight that the anchor we are optimizing against can be different from the reference we are regularizing to (but please let us know if we are misunderstanding!)\n5. [Highlight that DPO is win rate-consistent]\n- DPO’s optimal solution matches that of a regularized WRO objective; thus we can say that it satisfies regularized win rate consistency. We mention this in appendix A and have added a line in the main paper, but it is also the motivation for DPO, i.e., an objective whose solution matches that of RLHF. \n6. [Is win-rate correspondence a big deal? Can you clarify the intuition for DPO?]\n- Failing win rate correspondence makes training and model selection difficult (i.e., why reduce loss if it doesn’t help with the goal, how to select checkpoint when best loss is not best win rate).\n- Intuition: DPO performs MLE of the preference classifier, where this classifier is a function of the policy model. If loss goes down, this implicit preference classifier is getting closer to the true preference classifier (in expectation over the offline distribution). However, this does not mean that the corresponding policy model is getting better at win rate under the true preference classifier. First, prevalence mismatch between offline data and model means the preference classifier could get worse for responses that are relevant under the model even as it gets closer to ground truth for the offline data. Second, DPO breaks preference consistency as the implicit preference classifier is a function of the policy model itself and thus changes throughout training as the policy model changes.\n7. [Elaborate on three cases for $p_{\\ell}$]\n- Oracle is a model used to label the preference data and evaluate the win rate of the trained models. BT = True is a reward model trained on the oracle-labeled preference data. BT = False is a preference classifier trained on the oracle judge-labeled preference data without BT (lines 377-384).\n8. [SFT > DPO?]\n- Summarizing footer 4, we conclude SFT > DPO with respect to ease of optimization but based on works which show optimization difficulties of DPO, e.g., failure to increase the probability of the chosen response or improve the rankings.\n8. [I suspect some win rate correspondence violations of DPO are due to KL, Fig 1 is extreme]\n- To formalize, we add a new property called regularized win rate correspondence, i.e., improvement in loss implies improvement in win rate or divergence to reference. This is very inclusive, as even a change that results in a large decrease in win rate will still meet regularized win rate correspondence if divergence to reference decreases. Any regularized WRO objective satisfies this. DPO (off- or online) does not: loss can improve even though win rate and divergence get worse (as DPO does not directly optimize for either).\n- Note that Fig 1 is less extreme than DPO as setting is online and infinite data. Fig 5 is even less extreme with a uniform starting model and equally spaced rewards, and Fig 6 further decreases $\\beta$ in the least extreme setting in Fig 5. These figures show that DPO still does not satisfy win rate correspondence even as we continue to make the setting more and more favorable (and at some point, smaller $\\beta$ doesn't help). Even so, given that the Fig 1 is quite specific, we've moved it to the appendix and have replaced it with a discussion on regularized win rate correspondence.\n10. [Additional related work] \n- Thank you, they were a pleasure to read, and we have added them to related work.\n\nThank you again for your questions. If we have answered them to your satisfaction, we hope you will consider raising your score.", "claims_and_evidence": "The paper's main claims are theoretical. Given the proposed properties of the evaluation function, namely, preference-consistency and prevalencepconsistency, are desirable, the paper then provides a sound theoretical foundation. Additionally, the authors are transparent about the empirical evaluations, which I appreciate.\n\nThere is a subtlety in interpreting the results that I wish the authors had discussed further. Specifically, the dependence of the optimal solution on the anchor distribution is, by design, undesirable. This issue affects all $h$-win rate methods except those using BT with $h = \\text{logit}$ (i.e., RLHF). It is counterintuitive that what makes a generative model optimal depends on the anchor distribution used for comparison. Compare this to reward maximization where the optimal policy does not depend on how the preference dataset is sampled. For further discussion, see [2] in the additional related work.", "methods_and_evaluation_criteria": "Again, assuming that the evaluation function should be preference-consistent and prevalence-consistent, WRO makes sense. However, I wish there were better motivations for these properties. In particular, as I understand, prevalence-consistency for the generator basically rules out non-RL methods, including all direct methods (that's my read from Eq. 2 where I don't have any other way than RL to maximize w.r.t. $\\theta$).", "theoretical_claims": "As far as I checked, the theory is sound. I think you don't need BT to prove Prop. 4.1. I suspect any increasing symmetric $sigma$ works here. Another suggestion is that having two anchor/competitor distributions was confusing. Since you will assume these are the same, maybe do not introduce them differently. \n\nThe analysis of DPO was interesting. First, I think DPO is win rate-consistent, which could be better highlighted ([1] and [2] have similar observations, I believe). Second, this analysis made me think if win rate-correspondence is a big deal. Doing a little exploration, I came across this intuition: calculating the gradient of the DPO's objective, we can see that DPO works to make the win rate induced by a policy get closer to the true win rate under true reward. When these two completely match, DPO's policy internally represents the true reward, which means it is the optimal policy. However, this does not mean that we are getting closer to the optimal reward in every step. This is only my intuition but it would help if the authors clarify/elaborate further on their intuition in Sec 5.1. I think this will be very helpful as the readers generally don't expect and MLE estimator such as DPO to have an undesirable property.", "experimental_designs_or_analyses": "I appreciate the transparency of the results. I think the experiments were insightful and interesting. A few minor questions: Please further elaborate on how three cases for $\\hat{p}_l$ are obtained in Table 1. How did you conclude from experiments that SFT > DPO? My read of Fig. 2 is that DPO is superior to all methods. \n\nAnother room for improvement is Fig. 1. I suspect some of the win rate correspondence violations of DPO are due to its KL-regularization. I think Fig. 6 partly confirms this, though this cannot completely explain why DPO is not win rate correspondent. Compared to Fig. 6, I think Fig. 1 is a very extreme case. I'd choose a smaller $\\beta$. Also, it helps to have the same plot for logit-WRO-KL so we can see how many of these violations are due to regularizations.", "relation_to_broader_scientific_literature": "The paper mostly reminds me of $\\psi$-PO (Azar et al.) but still has new insights. It is also very related to GPO (Tang et al.). All of these works try to understand preference optimization in its most generality. I like how this work further categorizes these methods in terms of WRO and non-WRO.", "essential_references_not_discussed": "The paper has done a good job of covering relevant literature. When I heard the win rate, I immediately thought of two very recent works from social choice theory and alignment [1, 2]. These two also discuss how RLHF and DPO are related to win rate, or in other words, Borda count. Their setting is slightly different from the current work as they consider the possibility of different reward functions for individuals, but I thought they can still be interesting and relevant for the authors.\n[1] Distributional preference learning: Understanding and accounting for hidden context in RLH\n[2] Direct Alignment with Heterogeneous Preferences", "other_strengths_and_weaknesses": "The paper is well-written. Just as a suggestion, maybe use consistent capitalization in the title and sections.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please refer to above points.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JcRoB5Wdh9", "reviewer": "Reviewer_F6Z7", "summary": "The paper first introduce the concepts of preference consistency and prevalence consistency, and then proved that the only form of loss function that satisfies both preference and prevalence consistency is a type of win rate, proposing the h-win rate. The paper then argues the benefits of h-win rate, and analyzed DPO and SFT through this lens, showing that they do not fit the consistencies proposed. Empirical results shows a correspondance between win rate and the loss proposed.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We answer your questions (paraphrased for brevity) below:\n\n1. [datasets and model of Figure 2?]\n-  The reviewer is correct that the model is Pythia 2.8B. The dataset is Open Assistant.\n2. [Table 1 shows different h win rate methods, but did not show a clear winner, and thus it's hard to draw a useful conclusion there]\n- We'd like to emphasize that the lack of a clear winner is the primary conclusion of Table 1. This result suggests that  design choices for WRO which affect the optimal solution do not seem to have a first-order systematic effect on performance in practice;  optimization success does though, as a show later in Figure 3. \n3. [Figure 3 shows the correspondence between train loss and win rate, but I'm curious about the sudden drop of loss on the left subfigure.] \n- Great question. Each point on either plot in figure 3 is a different final model from a training run of a given WRO variant. The sudden drop of loss denotes a subset of models which achieve final loss past a certain threshold (around -0.35). These models achieve much better win rates than models which are not able to reach such a loss value, suggesting that simply being able to optimize past a certain threshold is strongly indicative of test win rate performance.\n4. [The experiments, while well intended, look a bit small in scale and not very strong. It would be helpful to conduct higher scale experiments, and show better correspondence between win rates and train loss, which is the main benefit of the methodology part.]\n- We'd like to emphasize that the current experiments consist of 72 distinct RL fine-tuning runs on a several billion parameter model. Unfortunately, limited resources prevent us from significantly scaling up past the current experiments during the response period, but if the reviewer has specific suggestions, we'd be happy to incorporate them into the final version of the paper. \n- It is also worth noting that the level of correspondence we are already seeing between train loss and test win rate is extremely striking, especially given the fact that different WRO variants do not necessarily even share the same scale for loss (due to choice of $\\beta$ and $h$). In other words, we would not expect, a priori, such significant rank correlations, much less a better correspondence. Instead, the fact that the correlation exists with train loss, even when it doesn't with other design choices we hypothesized a priority to be relevant, suggests that optimization success is the first order consideration for WRO.\n5. [Is it possible to extend the proposed pairwise metrics to listwise?]\n- Yes! In the case of preference for a set of $k$ responses, the binary preference classifier would become a k-class preference classifier, and h-win rate would become a multi-way win rate over $k-1$ anchor distributions, which could optionally be the same distribution. Additional assumptions also generalize, such as Plackett-Luce for Bradley-Terry. We’ve added this discussion to the appendix.\n\nThank you for your review. Please let us know if you have any additional questions or concerns; otherwise, we would greatly appreciate it if you would consider raising your score.", "claims_and_evidence": "yes. proofs look correct and empirical results indicate relavance.", "methods_and_evaluation_criteria": "The paper mostly proposes two criteria to look at when analyzing alignment algorithms. Well in experiments the resulting algorithm from the theory does not seem to outperform existing methods, it does help elucidate why DPO often do not achieve SOTA results in practice.", "theoretical_claims": "mostly checked prop 3.3 and skimmed the others. Looks correct.", "experimental_designs_or_analyses": "The experiments, while well indended, look a bit small in scale and not very strong. In particular, there are three sets of experiments. Figure 2 compares expected and actual win rates of different methods, but it's not clear the datasets and the model (I assume it's pythia 2.8B and Anthropic HH, but unclear); Table 1 shows different h win rate methods, but did not show a clear winner, and thus it's hard to draw a useful conclusion there; Figure 3 shows the correspondance between train loss and win rate, but I'm curious about the sudden drop of loss on the left subfigure. \nIn general, while the idea is interesting and useful, the experiments are not well supported, it would be helpful to conduct higher scale experiments, and show better correspondance between win rates and train loss, which is the main benefit of the methodology part.", "relation_to_broader_scientific_literature": "The paper helps elucidate some of the alignment research problems in practice.", "essential_references_not_discussed": "no.", "other_strengths_and_weaknesses": "no", "other_comments_or_suggestions": "no", "questions_for_authors": "Is it possible to extend the proposed pairwise metrics to listwise? Practitioners often have a list of N_i responses to the i-th prompt, and preference scores assigned to each of the responses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a well-motivated and theoretically grounded framework for understanding preference learning through the lens of win rate optimization (WRO). By formalizing the concepts of preference and prevalence consistency, the authors demonstrate that win rate is the only evaluation metric that faithfully reflects the underlying preference data distribution. The work introduces a useful taxonomy distinguishing WRO and non-WRO methods, and provides both theoretical and empirical analyses to support its claims. While one reviewer expressed concerns about the idealized assumptions and practical implications, the authors’ rebuttal offered thoughtful clarifications and highlighted concrete guidance for real-world applications. Overall, this paper offers valuable conceptual clarity to the preference learning literature and advances the foundation for future research in alignment. I recommend acceptance.
Decision: Accept (poster)