Win Rate is All that Can Matter from Preference Data Alone

Authors: Lily H Zhang, Rajesh Ranganath
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: alignment, preference learning, RLHF, win rate, language model

Abstract:
The surging interest in learning from preference data has resulted in an elaborate landscape of methods and evaluations. 
This work offers a framework to simplify this landscape, starting from the underlying sampling distribution for preference data.
First, we show that the only evaluation of a generative model that is grounded in the preference data sampling distribution is win rate. 
Given that win rate is all that can matter from preference data alone, we relate common preference learning algorithms to direct win rate optimization (DWRO). We outline the theoretical benefits of RLHF as a variant of DWRO; explain why checkpointing is difficult with DPO as a non-DWRO objective; and characterize the limits of SFT on preferred samples with regard to the extent of win rate improvement possible.
Furthermore, we provide closed-form expressions for the expected win rate improvement of the above objectives, formalizing the role of a model's starting point in the win rate improvement possible. Finally, we conduct an empirical analysis of existing methods and alternative DWRO objectives which suggests that optimization improvements are likely key to advancing preference learning.

Decision: reject
Reviewer scores: 3, 1, 6, 5

Reviews and discussion:
<thread>
creation date:
1730091359920

thread_type:
official_review

<thread_object>
creator: reviewer_41g3

<content>
summary:
The authors present a framework for analyzing different preference learning algorithms. They begin by defining the sampling distribution for pairwise preference learning and introduce a grounded evaluation function, demonstrating that the only grounded function, under a monotonic condition of function h, is the win rate. Building on this result, they propose an objective to optimize the win rate (DWRO) and explore various objective variants. Furthermore, they classify existing popular methods within the DWRO-KL variant. Despite not directly optimizing the DWRO objective, the authors show that SFT can still improve win rates under certain, intuitive conditions. Finally, they provide extensive results based on a single base model.

strengths:
- The paper introduces a general framework that encompasses cases like RLHF or DPO.
- This intuitive framework offers insights into current BT model-based preference learning methods.
- The authors present ablation studies on various functions h and different parameters beta.
- This framework can motivate further studies on finding improved optimization objectives.

weaknesses:
- The framework does not appear to account for unpaired/ranked preference methods.
- The judge model used seems underperforming, achieving only 68.8 on the evaluation set.
- Diverse models could have been employed for more comprehensive results.
- The framework does not address biases in preference learning, such as length bias.
- The paper focuses solely on the DWRO-KL variant without exploring other possible variants.

questions:
- The judge model Pythia 2.8B model seems weak.
- Why do you fine-tune on dispreferred outputs? Which step is that in the framework?
- Stronger judge models should be evaluated for comparison.
- Could you clarify what the anchor distribution is? It is used interchangeably as p or q, which adds to the confusion. Is it solely based on y_0 as defined in Definition 2?
- In the base case of Definition 2, where l=1, there is no y_1 provided. Could you explain this?
- Why is preference limited to the case where l=1? Why doesn’t it include the case when l=0 with x_0 being preferred?
- What is Property 3 in the proof of Proposition 1?
- Since you're using sequence-level KL divergence for DWRO-KL and token-level for PPO, the results may not be directly comparable. Could you comment on this?
- There are some grammatical issues throughout the paper that make comprehension more difficult.
- How many samples are used for each set in your experiments?
- The Appendix mentions Mistral 7B, but no results for this model are included.
- Given that beta is an important parameter, further studies or experiments focusing on its impact would be valuable.
- While DPO is discussed in the analysis, it is not shown in any empirical results. Why is that?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730265880558

thread_type:
official_review

<thread_object>
creator: reviewer_hqd1

<content>
summary:
The authors propose an overarching conceptual framework—a “North Star”—to guide work on learning from human preferences, centered around optimizing for win-rate. This framework consists of a family of Direct Win Rate Optimization (DWRO) objectives for optimizing generative models to achieve higher win-rates. DWRO objectives have of three central components: (1) the strictly monotonic increasing function h, which takes in a preference probability as an input (2) the preference classifier and (3) the anchor distribution over which win-rate is computed. 

The authors then present the KL-regularized DWRO (DWRO-KL) objective, which helps mitigates issues related to reward hacking, and which RLHF falls under. Notably the authors assert that optimizing for the DWRO-KL objective is equivalent to approximating a target distribution, which draws an important parallel to variational inference. The authors characterize the target distribution as being the original distribution that has been tilted to have more probability mass placed over sequences that are more likely to be preferred over the anchor distribution. In particular, the notion of sharpness as controlled by the parameter beta and the function h, is used to characterize the target distribution. Under this framing, the authors then introduce the following insights:
1. RLHF is a DWRO-KL objective 
2. DPO is a DWRO-KL objective with the same target distribution as RLHF, but used a different DWRO-KL objective than RLHF
3. SFT is not DWRO objective but still optimizes for win-rate via targeting a different distribution 

The authors present some empirical results illustrating the effects of varying the DWRO-KL objective on two popular preference learning datasets (i.e., HH and OASST). The authors also provide closed-form solutions for the expected win-rate improvement for DWRO-KL objectives and the SFT objective.

strengths:
The authors present a well-structured, sound, and clear framework for understanding different approaches to learning from preferences. By viewing these methods as either optimizing for win-rate or approximating a specific target distribution, they provide a useful lens for interpreting past and future work in this domain. I appreciate this framework as an original and valuable tool for conceptualizing advancements in preference-based learning.

The authors also establish a concrete parallel between learning from preferences (specifically through optimizing for the DWRO-KL objective) and black-box variational inference. This connection could be especially meaningful for researchers in black-box variational inference, potentially inspiring new research directions. To my knowledge, this connection is novel and holds promise for further impact.

Lastly, the authors offer both closed-form solutions for the expected gain in win-rate for DWRO-KL objectives and empirical results for various DWRO-KL variants. While I am uncertain about the practical utility of these results for other researchers or practitioners, they do provide insights into various design decisions for learning from preferences revolving around the sharpness of the target distribution.

weaknesses:
The paper feels lacking in actionable insights. While the authors offer a useful framework for understanding different approaches to learning from preferences, leading to a range of alternative objectives beyond the standard RLHF objective, RLHF still appears to be the top-performing method. This raises questions about the practical value of the additional flexibility provided by the proposed framework. 

Additionally, although the finding that SFT improves win-rate is intriguing, I would appreciate more clarity on why this insight is valuable for the community.

Furthermore, Figure 1 could be clearer. This figure would be significantly improved with guidance on how to interpret it and by highlighting any specific conclusions drawn from it. I was surprised that Figure 1 was not discussed in the results section; perhaps added explanation there would be helpful.

questions:
What practical insights can we draw from the provided closed-form solutions for win-rate improvement for the DWRO-KL objectives?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730350204687

thread_type:
official_review

<thread_object>
creator: reviewer_qhau

<content>
summary:
This paper studies preference learning algorithms for language models. The authors propose a family of objectives called DWRO to directly optimize the win rate. The authors discuss the relationship between DWRO, reinforcement learning from human feedback (RLHF), and supervised fine-tuning (SFT), examining how SFT can improve win rates. They also conduct experiments to evaluate different DWRO variants.

strengths:
a. This paper studies a general preference learning problem, with the goal of improving win rates. The widely-used Bradley-Terry model is a special case within this general preference framework.

b. The authors conduct experiments to evaluate various DWRO variants with different choices of $\psi$.

weaknesses:
a. The major contribution, DWRO, is not new, as it has already been proposed (see Equation 6, the preference optimization objective, in [1]). The function $\Psi$ serves the same role as $h$. Additionally, the relationship between the objective and RLHF has also been discussed in [1].

b. Section 3.3, which discusses SFT, is unclear. In line 313, the authors state that “SFT is analogous to estimating the distribution $p(\mathbf{y}_1 \mid \mathbf{x}, \ell=1)$” but do not provide justification. The quantity $p(\mathbf{y}_1 \mid \mathbf{x}, \ell=1)$ is confusing, as it’s unclear how another response, $y_0$, is sampled. Is it sampled from a fixed reference distribution? More details are needed on how SFT is performed on preferred samples. Furthermore, the justification for why Eq. 16 is the target distribution for SFT is lacking. The authors show equivalence between Eq. 15 and Eq. 16 but do not explain the derivation of Eq. 16.

c. Experimental details are also insufficient. There is no explanation of how the reference model is trained—does it use the same preference dataset as the models with DWRO variants? In addition, details on constructing the preference pairs for training are missing. The purpose of the experiments is also unclear. If the goal is to compare different DWRO objectives, it would be more meaningful to have them compete directly with each other, rather than against a fixed reference model. Furthermore, the judge model, Pythia-2.8b, is lightweight, whereas higher-standard models like GPT-4 are typically used as judges for evaluating win rates.

d. The overall presentation requires improvement. Section 2.2 all talks about the evaluation function $\phi$, but it does not appear in later sections. Additionally, the notation for the anchor distribution is inconsistent, with $q$ used in some expressions and $p$ in others, which makes the paper difficult to follow.

[1] Azar, Mohammad Gheshlaghi, et al. "A general theoretical paradigm to understand learning from human preferences." International Conference on Artificial Intelligence and Statistics. PMLR, 2024.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730726029385

thread_type:
official_review

<thread_object>
creator: reviewer_wxdn

<content>
summary:
This paper presents a framework for preference learning centered around the concept of "win rate" as the fundamental metric derived from preference data. Recognizing that preference learning has become increasingly complex, the authors propose Direct Win Rate Optimization (DWRO) as a more straightforward family of objectives for preference learning. They demonstrate that Reinforcement Learning from Human Feedback (RLHF) is a KL-regularized DWRO objective, unlike Supervised Fine-Tuning (SFT), although SFT still improves win rate. Through theoretical analysis, the paper explores how various preference learning objectives impact the sharpness of target distributions and provides closed-form solutions for expected win rate improvements. The authors support their framework with empirical results, showing that alternative KL-regularized DWRO objectives can outperform RLHF, and conclude by offering guidance for future research based on their win-rate-centric perspective.

strengths:
1. Theoretical Contributions: It offers formal proofs showing that only the win rate is grounded in preference data, providing justification for the proposed DWRO framework.

2. Empirical Validation: The experimental results support the theoretical claims and illustrate the efficacy of DWRO objectives compared to RLHF.

weaknesses:
1. The writing is poor. Especially for the contribution is poor, first, the authors should use mathematical formulas to accurately show their novelty instead of confusing words; second, the contribution should be split into fewer points, like 3 to 4. Now 7 points are too many.

2. The novelty of this work is little. Although the authors claim that they are the first to consider the win rate, the win rate is basically the same as the general preference, which is already been studied by a series work of Nash learning, such as Munos 484 et al. (2023), [1], [2], [3]. In this work, the authors fix the policy of one response as the based one and optimize the others, this is exactly the same as each iteration of Nash learning optimization of Munos 484 et al. (2023). Besides, although the authors consider different functions h, all the types they list in Table 3 have been considered in the previous works. 

3. Based on the theory of Munos 484 et al. (2023) and [3], fixing one response and optimizing the other may not be efficient and achieve the best performance since one iteration will not lead to the Nash equilibrium. Hence, the authors are suggested to compare the performance of their method to Munos 484 et al. (2023), and [3].



[1] Ye, Chenlu, et al. "A theoretical analysis of nash learning from human feedback under general kl-regularized preference." arXiv preprint arXiv:2402.07314 (2024).

[2] Rosset, Corby, et al. "Direct nash optimization: Teaching language models to self-improve with general preferences." arXiv preprint arXiv:2404.03715 (2024).

[3] Calandriello D, Guo D, Munos R, et al. Human alignment of large language models through online preference optimisation[J]. arXiv preprint arXiv:2403.08635, 2024.

questions:
1. Line 124: If h is a function, what is its' mapping: from what space to what space?

2. Line 354: the notation for the variance is confusing and I cannot find a clarification. To which variance the authors are taking variance?

3. Some typos: Line 228 should be "between the model and the target distribution";  Line 354 and 357, ".5" is somehow unprofessional and easily leads to confusion. It is better to change it to 0.5 or 1/2.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732510845438

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
Thank you all for your reviews. We are especially pleased that 41g328 acknowledges that the paper presents a “general” and “intuitive” framework and hqd1 appreciates our proposed framework as “an original and valuable tool for conceptualizing advancements in preference-based learning.” We want to emphasize that the **proposed win-rate centric framework is the main contribution of our work; this paper is not a methods paper.** To summarize and clarify our primary contributions: 
1. We are the first to point out that given simple desiderata, win rate is the only evaluation that is mathematically grounded in the preference data sampling distribution alone. Any other evaluation must be a function of external choices or assumptions. 
- We do not claim to be the first to discuss win rate as an evaluation or an objective, but we are the first to present its unique position for preference learning. This result provides a focal point to understand this otherwise seemingly elaborate field.
2. We present a win-rate centric analysis of common preference learning algorithms, with insights that are broadly relevant to variants of these algorithms as well. Our analysis proceeds by introducing direct win rate optimization (DWRO) as the theoretical ideal for preference learning and describing how common algorithms relate to or deviate from it, as well as what this implies about their benefits and limitations. 
- As correctly pointed out by WXDN and QHAU, as well as mentioned in our related work, other works that have proposed directly optimizing win rate. We do not claim DWRO as a methodological contribution but rather as an analytical tool. We are the first to our knowledge to motivate and use DWRO as the starting point to understand the preference learning landscape.

Our analysis offers a range of results and takeaways:
1. **There are theoretical benefits to being as close to DWRO as possible.** These include (1) optimizing train loss corresponds to optimizing for the test evaluation we care about (up to noise and overfitting); and (2) its solution has no theoretical limits as regularization strength goes to zero.
- RLHF is a variant of DWRO under assumptions, meaning it enjoys the above benefits. However, there are other variants that could work too.
- DPO is not a DWRO; this provides insight into why DPO loss is bad for checkpointing, i.e., (1) does not hold.
- SFT is not a DWRO either; while it can improve win rate, there are limits.
2. **We can characterize the expected win rate improvement of common preference learning algorithms like RLHF, DPO, and SFT.** This result highlights that **improvement depends on the initial model.** In the case of SFT in particular, win rate improvement is exactly a function of the variance of average preference probabilities, providing a formal definition of the role of diversity in self-improvement.
3. Preference learning objectives (e.g., KL-regularized DWRO) can be mapped to those in probabilistic inference (e.g., variational inference). This presents an opportunity to use insights from the latter to improve the former.

In response to reviewer comments, we made the following additions / modifications:
1. Previously, the introduction provided a step-by-step summary of our analysis, which we viewed as novel in itself. We have now modified the introduction to focus on only the primary contributions and takeaways of the paper. 
2. We have moved the related work section to the beginning of the paper, to emphasize the differences with prior work before the analysis begins.
3. In each section of the analysis, we have added a “Takeaways” paragraph to make it easier to recognize the contributions of the analysis.
4. We have expanded the experimental section. We:
- Increased the number of DWRO-KL settings tested (more values of beta, oracle and non-oracle preference classifier)
- Added comparisons to DPO and SFT 

With these additional experimental results, we have added a new empirically-driven contribution to the work: **optimization success plays a primary role in the success of preference learning algorithms.** DWRO objectives underperform empirically relative to theoretical expectations, and across DWRO variants (including RLHF), the train loss achieved by a given run is more predictive of test win rate than any setting of regularization, regularization strength, preference classifier estimate, or $h$ transformation. This suggests that although we may have a theoretical ideal for learning from preference data, much work remains to fully capitalize on these objectives in the form of optimization improvements. Moreover, the fact that RLHF > DPO > SFT in their relation to DWRO but SFT > DPO > RLHF in ease of optimization gives a useful perspective on the benefit of mixing and matching methods (e.g., SFT then RLHF, DPO + SFT).

We look forward to engaging in discussion and would deeply appreciate it if the reviewers would reconsider their reviews / raise their scores with the above in mind.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a general framework for preference learning with the win rate. The authors instantiate several existing algorithms in their framework, analyze them, and support their findings empirically. The scores of this paper are 6, 5, 3, and 1; and did not change during the discussion. The reviewers had several major concerns:

* **Poor writing:** Notation is sometimes inconsistent and not all technical novelty is stated mathematically.

* **Novelty:** Many insights in this work already appeared in papers since 2023.

* **Experiments:** Not all experimental details are stated. The judge (Pythia-2.8b) is weak.

I agree with the authors that there is novelty in their work over prior works. However, it needs to be clearly stated and well evaluated. This is not the case now and therefore the paper cannot be accepted.