Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts

## Abstract

Abstract We introduce Simultaneous Weighted Preference Optimization ( Swepo ) , a novel extension of Direct Preference Optimization (DPO) designed to accommodate multiple dynamically chosen positive and negative responses for each query. Swepo employs a weighted group contrastive loss, assigning weights to responses based on their deviation from the mean reward score. This approach effectively prioritizes responses that are significantly better or worse than the average, enhancing optimization. Our theoretical analysis demonstrates that simultaneously considering multiple preferences reduces alignment bias, resulting in more robust alignment. Additionally, we provide insights into the training dynamics of our loss function and a related function, InfoNCA. Empirical validation on the Ultra-Feedback dataset establishes Swepo as state-of-the-art, with superior performance in downstream evaluations using the AlpacaEval dataset.

## 1 Introduction

Alignment of language models with human preferences is a critical aspect of developing AI systems that are not only capable of generating coherent text but also of producing outputs aligned with human values and expectations (Liu et al., 2023b,; Ouyang et al.,, 2022; Christiano et al.,, 2017). Traditional methods like Reinforcement Learning from Human Feedback (RLHF) through explicit reward modeling (Christiano et al.,, 2017; Schulman et al.,, 2017) require large amounts of human-annotated training data (Wang et al.,, 2023), which can be resource-intensive.

In this context, Direct Preference Optimization (DPO) has become a cornerstone method for aligning language models with human preferences, as it directly optimizes model parameters based on preference data without the need for explicit reward modeling (Rafailov et al.,, 2024; Xu et al.,, 2024). DPO traditionally contrasts pairwise responses – using one positive and negative response per query.

However, the landscape of alignment datasets has evolved to include multiple positive and negative responses per query (Cui et al.,, 2023; Liu et al.,, 2024), reflecting the diverse ways in which responses can be good, or bad.
Furthermore, the complexity of language means that for any given query, there are numerous acceptable responses, and similarly, multiple unacceptable ones.
The ability to consider multiple preferences simultaneously allows for a more robust alignment process – jointly incorporating different response information into the model parameters (Yang et al.,, 2024). Such multi-preference optimization accounts for the spectrum of responses and leverages awareness of this response distribution to align models more effectively (Zhou et al.,, 2023; Wang et al.,, 2024).

Figure: Figure 1: The workflow shows how responses from a multi-preference dataset are used for alignment. Each response (4 responses in the illustration) receives a rating, which is compared against the mean. We then calculate weights based on the absolute difference between each rating and the mean. Accepted responses (above mean) and rejected responses (below mean) are grouped separately, with their respective weights determining their influence in the optimization process. The final step employs a Bradley-Terry style loss function to simultaneously maximize the probability of the accepted group while minimizing the probability of the rejected group.
Refer to caption: /html/2412.04628/assets/x1.png

In essence, for any given query, there exists a distribution of acceptable and suboptimal responses. Sampling only a small subset from these distributions may introduce alignment biases. Examples of such biases include length bias (Holtzman et al.,, 2019; Wolf et al.,, 2023), format bias (Zhang et al.,, 2024; Long et al.,, 2024), safety bias (Cui et al.,, 2024; Arditi et al.,, 2024), cultural bias (Tao et al.,, 2024), and annotator bias (Zhou et al.,, 2024). By sampling a larger number of preferences from both the accepted and rejected groups, we can achieve a closer approximation to the true distribution, thereby mitigating these biases.

Weighting responses based on their quality rating further refines this process. The most important responses for alignment are those that are significantly better or worse than average. Focussing the model alignment on such responses, is akin to importance sampling and curriculum learning (Bengio et al.,, 2009; Graves et al.,, 2017), where more informative samples are given greater influence during the optimization process. Responses of median quality contribute to learning but have less influence on alignment – enabling us to focus on the outliers to steer the optimization process.

To address multiple preferences, one might naively aggregate DPO losses to handle k-positive and k-negative responses – but this would require $k^{2}$ pairwise comparisons. A more nuanced approach would be to apply contrastive learning techniques like InfoNCE to the multi-preference alignment problem (Oord et al.,, 2018; Chen et al.,, 2020, 2024). But this loss function, in the presence of subpar responses, may increase the probability of a negative response (Gao et al.,, 2021). Moreover, standard InfoNCE treats all negative samples equally, failing to differentiate between responses of varying quality.

To address these challenges, we introduce a novel method, which we term Simultaneous Weighted Preference Optimization (Swepo), that extends DPO to handle multiple preferences per query using a weighted group contrastive loss. Our approach incorporates rating awareness by assigning weights to responses based on their deviation from the mean reward score. This ensures that strongly positive and strongly negative responses have a higher impact on the model’s optimization – effectively prioritize the most informative examples during training.

### 1.1 Our Contributions

Our work makes the following key contributions:

- 1.
Algorithm for Rating-aware Multi-preference Optimization: We extend Direct Preference Optimization to account for multiple positive and negative responses per query, incorporating weighting based on response ratings. This allows the model to jointly consider a spectrum of responses and prioritize them according to their quality during optimization.
- 2.
Theoretical Analysis of Bias Reduction: We provide theoretical insights demonstrating that joint contrastive losses are less prone to alignment bias compared to traditional DPO, which relies on samples from the distributions of accepted and rejected responses. Our analysis shows that by considering multiple preferences simultaneously, the model achieves a more robust alignment with human values.
- 3.
State-of-the-Art Results: We validate our method by training on the Ultra-Feedback dataset and demonstrate state-of-the-art results on downstream evaluation using the AlpacaEval dataset. Our approach significantly improves the alignment of language models with human preferences.

## 2 Related Work

Proximal Policy Optimization (PPO):
Proximal Policy Optimization (PPO) (Schulman et al.,, 2017) has emerged as a robust reinforcement learning algorithm for policy optimization by clipping policy updates to prevent instability. Building upon earlier policy gradient methods (Sutton et al.,, 1999), PPO has been successfully applied to alignment tasks in Reinforcement Learning from Human Feedback (RLHF), allowing language models to produce outputs aligned with human preferences (Ziegler et al.,, 2019; Ouyang et al.,, 2022). Its simplicity and efficiency make it a standard approach for fine-tuning large-scale models. Prior to PPO, Trust Region Policy Optimization (TRPO) (Schulman,, 2015) introduced constraints to improve learning stability, influencing the development of PPO. Early applications of policy gradient methods in natural language processing (Ranzato et al.,, 2015) demonstrated the potential of reinforcement learning for language model training.

Direct Preference Optimization (DPO):
Direct Preference Optimization (DPO) simplifies the alignment of language models by optimizing a contrastive loss directly over paired preference data, bypassing the intermediate step of reward modeling (Rafailov et al.,, 2024). Unlike RLHF, DPO does not require explicit reward functions, making it computationally efficient and suitable for limited preference datasets. Recent extensions of DPO, such as Identity Preference Optimization (IPO) (Azar et al.,, 2024), self-play preference optimization (Wu et al.,, 2024), preference ranking optimization (Song et al.,, 2024), rejection sampling optimization (Liu et al., 2023a,), and generalized preference optimization (Tang et al.,, 2024) are amongst the other recent works improve on the DPO method.

Multi-Preference Optimization:
Traditional preference optimization methods, like DPO, consider pairwise comparisons. However, datasets such as UltraFeedback (Cui et al.,, 2023) highlight the necessity of multi-preference optimization. Multi-preference methods, such as InfoNCA (Chen et al.,, 2024), leverage all available positive and negative responses simultaneously, reducing alignment bias and better approximating the true preference distribution. These methods mitigate limitations inherent to pairwise approaches by incorporating the diversity of acceptable and suboptimal responses. Earlier works in search have also used multiple user preferences to optimize models in various applications such as search (Joachims,, 2002).

Reward Modeling in Preferences:
Reward modeling is essential for translating qualitative human feedback into quantitative metrics that guide AI behavior optimization. Traditional methods, such as Reinforcement Learning from Human Feedback (RLHF), utilize reward models trained on human annotations to inform policy updates (Christiano et al.,, 2017; Stiennon et al.,, 2020). Early approaches like inverse reinforcement learning (Ng et al.,, 2000) and apprenticeship learning (Abbeel and Ng,, 2004) demonstrated the feasibility of inferring reward functions from observed behaviors. Recent advancements have diversified reward modeling techniques. For instance, the Adversarial Preference Optimization (APO) framework employs adversarial training to adapt reward models to the evolving generation distribution of language models (Cheng et al.,, 2023).

Contrastive Predictive Coding and InfoNCA:
Contrastive learning, particularly methods like InfoNCE (Oord et al.,, 2018), maximizes mutual information between positive samples while discriminating against negatives. InfoNCA adapts these principles for preference optimization, aligning responses with scalar rewards through noise-contrastive estimation (Chen et al.,, 2024). Despite its strengths, InfoNCA can overemphasize less informative negative samples, which motivates methods like Swepo that dynamically weigh responses based on deviation from the mean reward.

UltraFeedback Dataset:
The UltraFeedback dataset (Cui et al.,, 2023) is a significant advancement in preference-based training resources. It comprises GPT-4 annotated feedback for over 64,000 instructions, including scalar reward evaluations. UltraFeedback has been pivotal in developing models like UltraLM-13B-PPO and UltraRM, which achieve state-of-the-art performance across benchmarks such as AlpacaEval. This dataset’s granularity enables advanced preference optimization methods like Swepo to leverage diverse response quality levels effectively.

## 3 Notations and Preliminaries

In this section, we establish the notations and preliminaries necessary for our proposed weighted multi-preference optimization method.

Let $\mathcal{X}$ denote the set of all possible queries, with $x\in\mathcal{X}$ representing a specific query. For each query $x$, let $\mathcal{Y}_{x}$ be the set of all potential responses. Our dataset $\mathcal{D}$ consists of $N$ queries, where each query $x$ is associated with $n$ responses $\{y_{i}\}_{i=1}^{n}$ and corresponding reward scores $\{S_{i}\}_{i=1}^{n}$.

The mean reward score for query $x$ is calculated as:

$$ $S_{\text{mean}}=\frac{1}{n}\sum_{i=1}^{n}S_{i}.$ (1) $$

The deviation of each response’s reward score from the mean is:

$$ $\Delta S_{i}=S_{i}-S_{\text{mean}}.$ (2) $$

We partition the responses into positive and negative sets:

$$ $\displaystyle Y^{+}$ $\displaystyle=\{y_{i}\mid\Delta S_{i}>0\},$ (3) $\displaystyle Y^{-}$ $\displaystyle=\{y_{j}\mid\Delta S_{j}\leq 0\}.$ (4) $$

Weights are assigned based on the deviation, using an exponential function or a power function. For positive responses ($y_{i}\in Y^{+}$):

$$ $w_{i}^{+}=\exp\left(\alpha\Delta S_{i}\right)\quad\text{or}\quad w_{i}^{+}=\left(\Delta S_{i}\right)^{p},$ (5) $$

and for negative responses ($y_{j}\in Y^{-}$):

$$ $w_{j}^{-}=\exp\left(\alpha\left(-\Delta S_{j}\right)\right)\quad\text{or}\quad w_{j}^{-}=\left(-\Delta S_{j}\right)^{p},$ (6) $$

where $\alpha>0$ is a scaling hyperparameter and $p\in\{0,1,2\}$.

The language model parameterized by $\theta$ provides the conditional probability $P_{\theta}(y\mid x)$ of generating response $y$ given query $x$. The logit or score function is:

$$ $s_{\theta}(y\mid x)=\log\left(\frac{P_{\theta}(y\mid x)}{P_{\text{ref}}(y\mid x)}\right)=\log P_{\theta}(y\mid x)-\log P_{\text{ref}}(y\mid x).$ (7) $$

Incorporating the weights into the probabilities, we have:

$$ $w_{i}\times\exp\left(s_{\theta}(y_{i}\mid x)\right)=\exp\left(\alpha\Delta S_{i}+s_{\theta}(y_{i}\mid x)\right),$ (8) $$

which leads to the modified score:

$$ $s^{\prime}_{\theta}(y_{i}\mid x)=s_{\theta}(y_{i}\mid x)+\alpha\Delta S_{i}.$ (9) $$

The weighted contrastive loss function is defined as:

$$ $L_{\text{weighted}}(\theta)=-\log\frac{\sum\limits_{y\in Y^{+}}\exp\left(s^{\prime}_{\theta}(y\mid x)\right)}{\sum\limits_{y\in Y}\exp\left(s^{\prime}_{\theta}(y\mid x)\right)},$ (10) $$

where $Y=Y^{+}\cup Y^{-}$.

## 4 Algorithm and Methodology

We present the *Simultaneous Weighted Preference Optimization* (Swepo) algorithm, which aligns the language model with human preferences by incorporating multiple responses per query and weighting them based on their deviation from the mean reward score.

### 4.1 Algorithm

Figure: Algorithm 1 Simultaneous Weighted Preference Optimization (Swepo)

### 4.2 Weight Computation and Modified Scores

In the notations, we defined weights using an exponential function of the deviation $\Delta S_{i}$. Specifically, the weight for each response is:

$$ $w_{i}=\exp\left(\alpha\Delta S_{i}\right),$ (11) $$

for positive responses, and

$$ $w_{j}=\exp\left(\alpha\left(-\Delta S_{j}\right)\right),$ (12) $$

for negative responses.

By incorporating these weights into the loss function, we observe that:

$$ $w_{i}\times\exp\left(s_{\theta}(y_{i}\mid x)\right)=\exp\left(\alpha\Delta S_{i}+s_{\theta}(y_{i}\mid x)\right).$ (13) $$

This demonstrates that weighting the probabilities is equivalent to adjusting the logits by adding the scaled deviation. Thus, the modified score for each response becomes:

$$ $s^{\prime}_{\theta}(y_{i}\mid x)=s_{\theta}(y_{i}\mid x)+\alpha\Delta S_{i}.$ (14) $$

### 4.3 Generalization with Power p

In the algorithm, we generalize the weighting scheme by defining the weights as the $p$-th power of the deviation:

$$ $\displaystyle w_{i}^{+}$ $\displaystyle=\left(\Delta S_{i}\right)^{p},\quad\text{for }y_{i}\in Y^{+},$ (15) $\displaystyle w_{j}^{-}$ $\displaystyle=\left(-\Delta S_{j}\right)^{p},\quad\text{for }y_{j}\in Y^{-},$ (16) $$

where $p\in\{0,1,2\}$. This allows flexibility in modifying the impact of the deviation on the weights. When $p=0$,
all weights are equal to $1$, reducing the method to unweighted contrastive loss.

## 5 Theoretical Analysis

In this section, we provide theoretical insights into why incorporating multiple preferences per query, as in our proposed Swepo method, leads to better alignment with human values compared to methods that rely on pairwise preferences, such as Direct Preference Optimization (DPO). We also differentiate our weighted group contrastive loss from the InfoNCA loss (Chen et al.,, 2024) and discuss the implications for model optimization.

### 5.1 Bias Reduction through Multiple Preferences

A key motivation for using multiple preferences per query is to reduce alignment bias, which arises when sampling a limited subset of responses from the distribution of acceptable and suboptimal responses. We formalize this intuition by analyzing how the expected bias with respect to an attribute decreases as the number of samples increases.

To analyze biases, we introduce an attribute function $a(y):\mathcal{Y}_{x}\rightarrow\mathbb{R}$, which maps responses to real numbers (e.g., response length, politeness).

The expected attribute value over the model’s distribution is defined as:

$$ $\mu_{\theta}=\mathbb{E}_{x\sim\mathcal{X}}\left[\mathbb{E}_{y\sim P_{\theta}(\cdot\mid x)}\left[a(y)\right]\right],$ (17) $$

where $P_{\theta}(\cdot\mid x)$ is the model’s conditional distribution over responses given query $x$.

The true expected attribute value over acceptable responses is:

$$ $\mu_{\mathcal{A}}=\mathbb{E}_{x\sim\mathcal{X}}\left[\mathbb{E}_{y\sim\mathcal{A}_{x}}\left[a(y)\right]\right],$ (18) $$

where $\mathcal{A}_{x}$ is the distribution of acceptable responses for query $x$.

The bias with respect to attribute $a$ is then defined as:

$$ $B^{(k)}=\left|\mu_{\theta}^{(k)}-\mu_{\mathcal{A}}\right|,$ (19) $$

where $\mu_{\theta}^{(k)}$ is the expected attribute value under the model after training with $k$ positive and $k$ negative samples per query.

#### 5.1.1 Assumptions

We make the following assumptions:

- •
Finite Variance: The attribute $a(y)$ has finite variance over the acceptable response distribution $\mathcal{A}_{x}$ for each query $x$, i.e., $\operatorname{Var}_{y\sim\mathcal{A}_{x}}[a(y)]=\sigma_{\mathcal{A}_{x}}^{2}<\infty$.
- •
Independent Sampling: Responses are independently sampled from their respective distributions.
- •
Model Capacity: The model can represent the true distribution given sufficient data.
- •
Uniform Bounded Variance: There exists a constant $\sigma_{\max}^{2}$ such that $\sigma_{\mathcal{A}_{x}}^{2}\leq\sigma_{\max}^{2}$ for all $x\in\mathcal{X}$.

###### Theorem 1 .

Under the stated assumptions, the expected bias $\mathbb{E}[B^{(k)}]$ decreases with the number of samples $k$ as

$$ $\mathbb{E}[B^{(k)}]\leq\frac{C}{\sqrt{k}},$ (20) $$

where $C=\sigma_{\max}$ is a constant depending on the maximum variance of $a(y)$ over acceptable responses.

Proof Sketch.
The proof relies on the Central Limit Theorem and the fact that the standard error of the mean decreases with the square root of the number of samples. For each query $x$, the sample mean of the attribute over the $k$ positive responses converges to the true mean $\mu_{\mathcal{A}_{x}}$ at a rate proportional to $1/\sqrt{k}$. Averaging over all queries, we obtain the bound on $\mathbb{E}[B^{(k)}]$.
$\square$

###### Corollary 1 .

As $k\to\infty$, the expected bias $\mathbb{E}[B^{(k)}]$ approaches zero:

$$ $\lim_{k\to\infty}\mathbb{E}[B^{(k)}]=0.$ (21) $$

Implications: This theorem formalizes the intuition that using multiple positive and negative responses per query reduces the bias in the model’s alignment with human preferences. By considering more samples from the distributions of acceptable and suboptimal responses, the model’s expected attribute values converge to the true values, leading to better alignment.

###### Theorem 1 .

###### Corollary 1 .

### 5.2 Comparison with InfoNCA Loss

While one might consider applying contrastive learning techniques like InfoNCA to the multi-preference alignment problem, we argue that our weighted group contrastive loss is better suited for this task.

#### 5.2.1 Definitions of Loss Functions

##### InfoNCA Loss Function

The InfoNCA loss function, proposed by Chen et al., (2024), is defined as:

$$ $L_{\text{InfoNCA}}=-\sum_{i=1}^{K}p_{i}^{\text{target}}\log p_{i}^{\text{model}},$ (22) $$

where:

$$ $p_{i}^{\text{target}}=\frac{e^{r(x,y_{i})/\alpha}}{\sum_{j=1}^{K}e^{r(x,y_{j})/\alpha}}\qquad p_{i}^{\text{model}}=\frac{e^{s_{\theta}(y_{i}\mid x)}}{\sum_{j=1}^{K}e^{s_{\theta}(y_{j}\mid x)}}.$ (23) $$

In this context:

- •
$x$ is the query or instruction provided to the model.
- •
$\{y_{i}\}_{i=1}^{K}$ represents a set of $K$ responses generated for query $x$.
- •
$r(x,y_{i})$ is the reward associated with response $y_{i}$.
- •
$s_{\theta}(y_{i}\mid x)=\log\left(\frac{P_{\theta}(y_{i}\mid x)}{P_{\text{ref}}(y_{i}\mid x)}\right)$ is the score for response $y_{i}$.
- •
$\alpha$ is a temperature parameter controlling the influence of the reward.

##### Weighted Contrastive Loss Function

Our proposed weighted contrastive loss function is expressed as:

$$ $L_{\text{weighted}}=-\log\left(\frac{\sum\limits_{i\in Y^{+}}w_{i}e^{s_{\theta}(y_{i}\mid x)}}{\sum\limits_{j=1}^{K}w_{j}e^{s_{\theta}(y_{j}\mid x)}}\right),$ (24) $$

where:

- •
$Y^{+}=\{y_{i}\mid\Delta S_{i}>0\}$ is the set of positive responses, with $\Delta S_{i}=S_{i}-S_{\text{mean}}$, and $S_{\text{mean}}=\frac{1}{K}\sum_{j=1}^{K}S_{j}$.
- •
$w_{i}=e^{\alpha\Delta S_{i}}$ for positive responses, and $w_{i}=e^{\alpha(-\Delta S_{i})}$ for negative responses.
- •
$s_{\theta}(y_{i}\mid x)=\log\left(\frac{P_{\theta}(y_{i}\mid x)}{P_{\text{ref}}(y_{i}\mid x)}\right)$ is the score for response $y_{i}$.
- •
$\alpha$ is a scaling hyperparameter controlling the influence of the deviation $\Delta S_{i}$.

#### 5.2.2 Gradient Analysis

To understand how each loss function influences the model during training, we derive the gradients with respect to the model logits $s_{\theta}(y_{i}\mid x)$ for both methods.

###### Lemma 1 .

The gradient of the InfoNCA loss with respect to the model logits $s_{\theta}(y_{i}\mid x)$ is:

$$ $\frac{\partial L_{\text{InfoNCA}}}{\partial s_{\theta}(y_{i}\mid x)}=p_{i}^{\text{model}}-p_{i}^{\text{target}}.$ (25) $$

Proof Sketch.
By computing the derivative of the loss with respect to the logits and recognizing that the target probabilities $p_{i}^{\text{target}}$ are constants with respect to the model parameters, we obtain the gradient as the difference between the model’s predicted probabilities and the target probabilities.
$\square$

###### Lemma 2 .

The gradient of the weighted contrastive loss with respect to the model logits $s_{\theta}(y_{i}\mid x)$ is:

$$ $\frac{\partial L_{\text{weighted}}}{\partial s_{\theta}(y_{i}\mid x)}=p_{i}^{\text{weighted}}-p_{i}^{\text{pos}},$ (26) $$

where:

$$ $p_{i}^{\text{weighted}}=\frac{w_{i}e^{s_{\theta}(y_{i}\mid x)}}{\sum_{j=1}^{K}w_{j}e^{s_{\theta}(y_{j}\mid x)}},\qquad p_{i}^{\text{pos}}=\frac{w_{i}e^{s_{\theta}(y_{i}\mid x)}}{\sum_{k\in Y^{+}}w_{k}e^{s_{\theta}(y_{k}\mid x)}}\cdot\mathbb{I}_{y_{i}\in Y^{+}},$ (27) $$

with $\mathbb{I}_{y_{i}\in Y^{+}}$ being the indicator function, equal to $1$ if $y_{i}\in Y^{+}$ and 0 otherwise.

Proof Sketch.
By differentiating the weighted contrastive loss with respect to the logits and applying the chain rule, we express the gradient in terms of the weighted probabilities of the responses.
$\square$

###### Lemma 1 .

###### Lemma 2 .

#### 5.2.3 Optimization Behavior and Stationary Points

Understanding the optimization behavior under different loss functions sheds light on how models learn from the training data.

###### Theorem 2 .

For the InfoNCA loss, the stationary points occur when:

$$ $p_{i}^{\text{model}}=p_{i}^{\text{target}},\quad\forall i\in\{1,\dots,K\}.$ (28) $$

Proof Sketch.
Setting the gradient of the InfoNCA loss to zero leads to the condition that the model’s predicted probabilities must match the target probabilities for all responses.
$\square$

###### Remark 1 .

Rather than equating the soft-max of the score $s_{\theta}(y\mid x)$ to $p_{i}^{\text{target}}$, optimality may require directly setting the soft-max of the $\log P_{\theta}(y\mid x)$ to match the soft-max of the target scores.

Implications: This stationary point may not lead to the desired alignment, in light of Remark [1](#Thmrem1).

###### Theorem 2 .

###### Remark 1 .

#### 5.2.4 Stationary Points of the Weighted Contrastive Loss

We now analyze the stationary points of our weighted contrastive loss function.

###### Lemma 3 .

For the weighted contrastive loss function, the stationary point occurs when the probabilities of the negative samples approach zero:

$$ $P_{\theta}(y_{i}\mid x)\to 0\quad\text{for all }y_{i}\in Y^{-}.$ $$

Proof Sketch.
At a stationary point, the gradient of the loss with respect to the scores $s_{\theta}(y_{i}\mid x)$ must be zero. From Lemma [2](#Thmlemma2), the gradient for negative samples is:

$$ $\frac{\partial L_{\text{weighted}}}{\partial s_{\theta}(y_{i}\mid x)}=p_{i}^{\text{weighted}},\quad\text{for }y_{i}\in Y^{-}.$ $$

Since $p_{i}^{\text{weighted}}$ is always positive unless $P_{\theta}(y_{i}\mid x)\to 0$, setting the gradient to zero requires that $P_{\theta}(y_{i}\mid x)\to 0$ for all negative samples. For positive samples, the gradient becomes zero when the model’s weighted probabilities match the positive probabilities, which occurs when the negative probabilities vanish.
$\square$

###### Remark 2 .

Intuitively, as the probabilities of the negative samples approach zero, the weighted contributions of negative samples to both the numerator and denominator in $L_{\text{weighted}}$ become negligible. The loss function simplifies to:

$$ $L_{\text{weighted}}=-\log\left(\frac{\sum\limits_{i\in Y^{+}}w_{i}e^{s_{\theta}(y_{i}\mid x)}}{\sum\limits_{j=1}^{K}w_{j}e^{s_{\theta}(y_{j}\mid x)}}\right)\approx-\log\left(\frac{\sum\limits_{i\in Y^{+}}w_{i}e^{s_{\theta}(y_{i}\mid x)}}{\sum\limits_{j\in Y^{+}}w_{j}e^{s_{\theta}(y_{j}\mid x)}}\right)=-\log 1=0.$ $$

This indicates that the loss vanishes when the model assigns negligible probabilities to negative samples, effectively focusing entirely on positive responses.

###### Lemma 3 .

###### Remark 2 .

#### 5.2.5 Characterization of Optimization Dynamics

We further analyze the optimization dynamics of our weighted contrastive loss for positive examples under some simplifying assumptions.

###### Lemma 4 .

Consider a simplified scenario with $N^{+}$ positive examples, each with weight $w^{+}$, and $N^{-}$ negative examples, each with weight $w^{-}$. All positive examples have the same score $s^{(t)}$ at iteration $t$, and all negative examples have the same score. The update rule for the score $s^{(t)}$ of the positive examples at iteration $t+1$ is given by:

$$ $s^{(t+1)}=s^{(t)}+\eta\left(\frac{N^{-}w^{-}}{N^{+}(N^{+}w^{+}+N^{-}w^{-})}\right),$ (29) $$

where $\eta$ is the learning rate.

Proof Sketch.
By computing the gradient of the weighted contrastive loss with respect to the scores and applying the update rule, we derive the expression for the score at the next iteration.
$\square$

This theoretical analyses shows that the scores of the positive examples increase over iterations based on their weights, and thus leading to better alignment with human preferences.

Conclusion: These theoretical insights provide strong justification for our proposed Swepo method over traditional pairwise preference optimization methods. By leveraging multiple, weighted preferences, we achieve a more nuanced and effective alignment with human preferences.

###### Lemma 4 .

## 6 Experimental Setup

##### Model and Training Settings:

For our experiments, we utilized the Ultrafeedback Dataset Cui et al., (2023), an instruction-following dataset annotated by GPT-4, containing approximately 64,000 instructions. Each instruction includes four responses generated by different language models, with GPT-4 assigning scalar rewards on a scale of 0 to 10 for each response. Previous research has shown a strong correlation between these GPT-4 ratings and human annotations, establishing their reliability as a cost-effective alternative to human feedback.

Based on these scalar rewards, we categorized the responses into two sets: chosen responses and rejected responses. This categorization was determined using the mean of the scalar rewards. Responses scoring above the mean were classified as chosen, while the remaining responses were categorized as rejected.

For our approach, we bypassed the supervised fine-tuning (SFT) stage and directly employed the Mistral-7B-Instruct-v0.1 model for preference alignment. We applied our proposed preference optimization method, Swepo, which consists of two configurations:

- 1.
Absolute-weighting
- 2.
Squared-weighting

**Table 1: Comparison of preference optimization methods on AlpacaEval benchmarks. LC-WR represents length-controlled win rate, and WR represents raw win rate. Best results are highlighted in blue, second-best in green, and third-best in yellow.**
| Evaluator LM | Method | AlpacaEval2 |  |
| --- | --- | --- | --- |
| WR | LC-WR |  |  |
| mistral-7b-instruct-v0.1 | best vs worst | 9.29 | 12.61 |
| mistral-7b-instruct-v0.1 | best vs random | 9.37 | 12.59 |
| mistral-7b-instruct-v0.1 | DPO x (k-1) | 8.41 | 13.36 |
| mistral-7b-sft-beta | info-nca-reward (Paper) | 10.44 | 16.82 |
| mistral-7b-sft-beta | info-nca-preference (Paper) | 10.62 | 14.63 |
| mistral-7b-instruct-v0.1 | info-nca-reward (our implementation) | 10.51 | 11.88 |
| mistral-7b-instruct-v0.1 | Swepo (1 vs k) (p = 0) | 10.73 | 15.19 |
| mistral-7b-instruct-v0.1 | Swepo (k1 vs k2) (p = 0) | 11.14 | 16.06 |
| mistral-7b-instruct-v0.1 | Swepo (k1 vs k2) (p = 1) | 11.7 | 16.3 |
| mistral-7b-instruct-v0.1 | Swepo (k1 vs k2) (p = 2) | 11.94 | 16.64 |

These configurations were designed to harness scalar rewards effectively and improve model alignment. Our findings suggest that these setups significantly enhance performance, placing our models among the top contenders on the Alpaca leaderboard.

##### Evaluation Benchmarks:

We evaluate our models using three widely recognized open-ended instruction-following benchmarks: MT-Bench Zheng et al., (2023), AlpacaEval 2, and Arena-Hard v0.1 Zheng et al., (2023). These benchmarks test the models’ conversational versatility across a broad range of queries and are broadly utilized in the research community. AlpacaEval2 Dubois et al., (2024) includes 805 questions derived from five datasets, while MT-Bench spans eight categories with a total of 80 questions. Arena-Hard, a recently updated version of MT-Bench, focuses on 500 well-defined technical problem-solving queries. Scores are reported based on the evaluation protocols of each benchmark. For AlpacaEval2, both the raw win rate (WR) and the length-controlled win rate (LC) are reported, with LC specifically designed to mitigate biases related to model verbosity. For Arena-Hard, the win rate is reported relative to a baseline model. For MT-Bench, the average score is calculated using evaluations by GPT-4 and GPT-4-Preview as judges.
For decoding details, we generate responses using both greedy decoding and multinomial sampling with temperatures of 0.2, 0.5, and 0.8. To address potential biases introduced by multinomial sampling at varying temperatures, we generate responses three times for each setting and average their performance across the datasets.

We provide details regarding baselines in Appendix [D](#A4)

## 7 Experimental Results

##### Swepo outperform preference-based methods given reward dataset:

The results of our ablation study demonstrate that our proposed methods consistently outperform all baseline preference models. Specifically, our methods exhibit a significant performance improvement of 24% over LC-WR and 42% over WR in the AlpacaEval2 benchmark when compared to the best-performing preference baseline, DPOx(k-1). This substantial improvement underscores the effectiveness of our approach, Swepo, which leverages its ability to fully exploit all the information available in the reward dataset. The enhanced utilization of reward signals enables Swepo to achieve superior performance, highlighting the importance of comprehensive information integration in preference-based models.

##### Swepo vs. InfoNCA on Preference Data with Scalar Rewards

We implemented the InfoNCA Chen et al., (2024) baseline under our custom settings, adhering to the hyperparameters specified in their original method. The primary difference lies in the finetuning approach: while InfoNCA utilizes QLoRA, we opted for full finetuning of their model.

Using this approach and from table [2](#S7.T2), we observed that Swepo outperformed InfoNCA on downstream datasets such as AlpacaEval and AlpacaEval2. This result underscores the significance of Swepo and its clear advantage over InfoNCA, particularly in scenarios where preference data with scalar rewards is available.

**Table 2: Performance Evaluation of Swepo and InfoNCA on AlpacaEval Datasets**
| Evaluator LM | Method | AlpacaEval2 | AlpacaEval1 |  |
| --- | --- | --- | --- | --- |
| WR | LC-WR | WR |  |  |
| mistral-7b-instruct-v0.1 | info-nca-reward (our implementation) | 10.51 | 11.88 | 80.26 |
| mistral-7b-instruct-v0.1 | Swepo (k1 vs k2) (p = 1) | 11.7 | 16.3 | 88.93 |
| mistral-7b-instruct-v0.1 | Swepo (k1 vs k2) (p = 2) | 11.94 | 16.64 | 87.9 |

##### Importance of Suboptimal Responses:

Suboptimal responses are also important. Previous practices always ensure selecting the responses with highest scaler reward when constructing preference data. The assumption behind this strategy is that the dataset’s best-performing response determines the upper limit of alignment performance.However, our experiments contradict this assumption. Results in table [1](#S6.T1) indicate that extra suboptimal responses can also be advantageous for policy training. Specifically, we observe consistent performance improvements when increasing the number of data responses from K = 2 to K = 4 for Swepo.

##### Contrasting Many Positive Responses with Many Negative Responses vs. One Positive Response with Many Negative Responses:

In this ablation study, we analyze the difference between contrasting multiple positive responses with multiple negative responses and contrasting a single positive response against all negative responses. Considering only one positive response in a one-vs-all-negative framework may lead to an incomplete representation, as other responses with high positive scores could also contribute meaningful signals. Treating these high-scoring responses as negatives may distort the learning process and hinder model performance.
The same has been demonstrated in table [3](#S7.T3) and we are getting significant improvement in performance over one-vs-all-negative framework.

By incorporating multiple positive responses, we better account for nuanced variations and alignments in the data, leading to more robust and generalizable model behavior. This approach ensures that all positively relevant responses are effectively utilized rather than being misclassified as negatives.

**Table 3: Effect of Single Positive vs. Multiple Positive Responses in Contrastive Learning**
| Evaluator LM | Method | AlpacaEval2 | AlpacaEval1 |  |
| --- | --- | --- | --- | --- |
| WR | LC-WR | WR |  |  |
| mistral-7b-instruct-v0.1 | Swepo (1 vs k) (p = 0) | 10.73 | 15.18 | 89.12 |
| mistral-7b-instruct-v0.1 | Swepo (k1 vs k2) (p = 0) | 11.14 | 16.06 | 88.81 |

##### Effect of Logit Weighting on Group Contrastive Loss:

In this ablation study, we explore the impact of weighting logits using two approaches: (1) the absolute deviation of scalar rewards, and (2) the squared deviation of scalar rewards from the mean reward value (termed Swepo). These weighting schemes are compared against unweighted logits in the context of group contrastive loss.

Our results in table [4](#S7.T4) demonstrate a significant performance improvement with the introduction of logit weighting. In the AlpacaEval2 benchmark, the Swepo-based weighting scheme outperforms unweighted group contrastive loss, achieving an improvement of 3.6% in LC-WR and 7.2% in WR. This highlights the effectiveness of incorporating deviation-based weighting mechanisms to enhance the model’s capability in aligning rewards with contrastive loss objectives.

**Table 4: Performance Comparison of Logit Weighting Methods in Group Contrastive Loss**
| Evaluator LM | Method | AlpacaEval2 | AlpacaEval1 |  |
| --- | --- | --- | --- | --- |
| WR | LC-WR | WR |  |  |
| mistral-7b-instruct-v0.1 | Swepo (k1 vs k2) (p = 0) | 11.14 | 16.06 | 88.81 |
| mistral-7b-instruct-v0.1 | Swepo (k1 vs k2) (p = 1) | 11.7 | 16.3 | 88.93 |
| mistral-7b-instruct-v0.1 | Swepo (k1 vs k2) (p = 2) | 11.94 | 16.64 | 87.9 |

## References

- Abbeel and Ng, (2004)
Abbeel, P. and Ng, A. Y. (2004).
Apprenticeship learning via inverse reinforcement learning.
In Proceedings of the twenty-first international conference on Machine learning, page 1.
- Arditi et al., (2024)
Arditi, A., Obeso, O., Syed, A., Paleka, D., Panickssery, N., Gurnee, W., and Nanda, N. (2024).
Refusal in language models is mediated by a single direction.
arXiv preprint arXiv:2406.11717.
- Azar et al., (2024)
Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. (2024).
A general theoretical paradigm to understand learning from human preferences.
In International Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR.
- Bengio et al., (2009)
Bengio, Y., Louradour, J., Collobert, R., and Weston, J. (2009).
Curriculum learning.
In Proceedings of the 26th annual international conference on machine learning, pages 41–48.
- Chen et al., (2024)
Chen, H., He, G., Yuan, L., Cui, G., Su, H., and Zhu, J. (2024).
Noise contrastive alignment of language models with explicit rewards.
arXiv preprint arXiv:2402.05369.
- Chen et al., (2020)
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020).
A simple framework for contrastive learning of visual representations.
In International conference on machine learning, pages 1597–1607. PMLR.
- Cheng et al., (2023)
Cheng, P., Yang, Y., Li, J., Dai, Y., and Du, N. (2023).
Adversarial preference optimization.
arXiv preprint arXiv:2311.08045.
- Christiano et al., (2017)
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. (2017).
Deep reinforcement learning from human preferences.
Advances in neural information processing systems, 30.
- Cui et al., (2023)
Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G., Liu, Z., and Sun, M. (2023).
Ultrafeedback: Boosting language models with high-quality feedback.
arXiv preprint arXiv:2310.01377.
- Cui et al., (2024)
Cui, J., Chiang, W.-L., Stoica, I., and Hsieh, C.-J. (2024).
Or-bench: An over-refusal benchmark for large language models.
arXiv preprint arXiv:2405.20947.
- Dubois et al., (2024)
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. (2024).
Length-controlled alpacaeval: A simple way to debias automatic evaluators.
arXiv preprint arXiv:2404.04475.
- Gao et al., (2021)
Gao, T., Yao, X., and Chen, D. (2021).
Simcse: Simple contrastive learning of sentence embeddings.
arXiv preprint arXiv:2104.08821.
- Graves et al., (2017)
Graves, A., Bellemare, M. G., Menick, J., Munos, R., and Kavukcuoglu, K. (2017).
Automated curriculum learning for neural networks.
In international conference on machine learning, pages 1311–1320. Pmlr.
- Holtzman et al., (2019)
Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. (2019).
The curious case of neural text degeneration.
arXiv preprint arXiv:1904.09751.
- Joachims, (2002)
Joachims, T. (2002).
Optimizing search engines using clickthrough data.
In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 133–142.
- (16)
Liu, T., Zhao, Y., Joshi, R., Khalman, M., Saleh, M., Liu, P. J., and Liu, J. (2023a).
Statistical rejection sampling improves preference optimization.
arXiv preprint arXiv:2309.06657.
- Liu et al., (2024)
Liu, Y., Cao, J., Liu, C., Ding, K., and Jin, L. (2024).
Datasets for large language models: A comprehensive survey.
arXiv preprint arXiv:2402.18041.
- (18)
Liu, Y., Yao, Y., Ton, J.-F., Zhang, X., Guo, R., Cheng, H., Klochkov, Y., Taufiq, M. F., and Li, H. (2023b).
Trustworthy llms: a survey and guideline for evaluating large language models’ alignment.
arXiv preprint arXiv:2308.05374.
- Long et al., (2024)
Long, D. X., Ngoc, H. N., Sim, T., Dao, H., Joty, S., Kawaguchi, K., Chen, N. F., and Kan, M.-Y. (2024).
Llms are biased towards output formats! systematically evaluating and mitigating output format bias of llms.
arXiv preprint arXiv:2408.08656.
- Ng et al., (2000)
Ng, A. Y., Russell, S., et al. (2000).
Algorithms for inverse reinforcement learning.
In Icml, volume 1, page 2.
- Oord et al., (2018)
Oord, A. v. d., Li, Y., and Vinyals, O. (2018).
Representation learning with contrastive predictive coding.
arXiv preprint arXiv:1807.03748.
- Ouyang et al., (2022)
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022).
Training language models to follow instructions with human feedback.
Advances in neural information processing systems, 35:27730–27744.
- Rafailov et al., (2024)
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. (2024).
Direct preference optimization: Your language model is secretly a reward model.
Advances in Neural Information Processing Systems, 36.
- Ranzato et al., (2015)
Ranzato, M., Chopra, S., Auli, M., and Zaremba, W. (2015).
Sequence level training with recurrent neural networks.
arXiv preprint arXiv:1511.06732.
- Schulman, (2015)
Schulman, J. (2015).
Trust region policy optimization.
arXiv preprint arXiv:1502.05477.
- Schulman et al., (2017)
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017).
Proximal policy optimization algorithms.
arXiv preprint arXiv:1707.06347.
- Song et al., (2024)
Song, F., Yu, B., Li, M., Yu, H., Huang, F., Li, Y., and Wang, H. (2024).
Preference ranking optimization for human alignment.
In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18990–18998.
- Stiennon et al., (2020)
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. (2020).
Learning to summarize with human feedback.
Advances in Neural Information Processing Systems, 33:3008–3021.
- Sutton et al., (1999)
Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. (1999).
Policy gradient methods for reinforcement learning with function approximation.
Advances in neural information processing systems, 12.
- Tang et al., (2024)
Tang, Y., Guo, Z. D., Zheng, Z., Calandriello, D., Munos, R., Rowland, M., Richemond, P. H., Valko, M., Pires, B. Á., and Piot, B. (2024).
Generalized preference optimization: A unified approach to offline alignment.
arXiv preprint arXiv:2402.05749.
- Tao et al., (2024)
Tao, Y., Viberg, O., Baker, R. S., and Kizilcec, R. F. (2024).
Cultural bias and cultural alignment of large language models.
PNAS nexus, 3(9):pgae346.
- Tunstall et al., (2023)
Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Rasul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., et al. (2023).
Zephyr: Direct distillation of lm alignment.
arXiv preprint arXiv:2310.16944.
- Wang et al., (2024)
Wang, C., Zhao, Z., Zhu, C., Sankararaman, K. A., Valko, M., Cao, X., Chen, Z., Khabsa, M., Chen, Y., Ma, H., et al. (2024).
Preference optimization with multi-sample comparisons.
arXiv preprint arXiv:2410.12138.
- Wang et al., (2023)
Wang, Y., Liu, Q., and Jin, C. (2023).
Is rlhf more difficult than standard rl? a theoretical perspective.
Advances in Neural Information Processing Systems, 36:76006–76032.
- Wolf et al., (2023)
Wolf, Y., Wies, N., Avnery, O., Levine, Y., and Shashua, A. (2023).
Fundamental limitations of alignment in large language models.
arXiv preprint arXiv:2304.11082.
- Wu et al., (2024)
Wu, Y., Sun, Z., Yuan, H., Ji, K., Yang, Y., and Gu, Q. (2024).
Self-play preference optimization for language model alignment.
arXiv preprint arXiv:2405.00675.
- Xu et al., (2024)
Xu, S., Fu, W., Gao, J., Ye, W., Liu, W., Mei, Z., Wang, G., Yu, C., and Wu, Y. (2024).
Is dpo superior to ppo for llm alignment? a comprehensive study.
arXiv preprint arXiv:2404.10719.
- Yang et al., (2024)
Yang, J. C., Korecki, M., Dailisan, D., Hausladen, C. I., and Helbing, D. (2024).
Llm voting: Human choices and ai collective decision making.
arXiv preprint arXiv:2402.01766.
- Zhang et al., (2024)
Zhang, X., Xiong, W., Chen, L., Zhou, T., Huang, H., and Zhang, T. (2024).
From lists to emojis: How format bias affects model alignment.
arXiv preprint arXiv:2409.11704.
- Zheng et al., (2023)
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. (2023).
Judging llm-as-a-judge with mt-bench and chatbot arena.
Advances in Neural Information Processing Systems, 36:46595–46623.
- Zhou et al., (2024)
Zhou, Z., Liu, J., Shao, J., Yue, X., Yang, C., Ouyang, W., and Qiao, Y. (2024).
Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization.
In Findings of the Association for Computational Linguistics ACL 2024, pages 10586–10613.
- Zhou et al., (2023)
Zhou, Z., Liu, J., Yang, C., Shao, J., Liu, Y., Yue, X., Ouyang, W., and Qiao, Y. (2023).
Beyond one-preference-for-all: Multi-objective direct preference optimization.
arXiv preprint arXiv:2310.03708.
- Ziegler et al., (2019)
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. (2019).
Fine-tuning language models from human preferences.
arXiv preprint arXiv:1909.08593.

## Appendix A Bias Analysis:

In the first part of this section section, we analyze how the number of positive and negative examples per query, $k$, affects the bias with respect to an attribute $a(y)$. We provide a formal theorem establishing the relationship between bias and $k$, followed by a corollary discussing the behavior as $k\to\infty$.

The reason for this analysis is to show that multi-preference sampling of accepted and rejected answers from a distribution is better than using a single sample as DPO does. The more accepted and rejected samples you have, the lower the bias, provably.

### A.1 Assumptions

We make the following assumptions:

- 1.
Attribute Function: Let $a(y):\mathcal{Y}_{x}\rightarrow\mathbb{R}$ be an attribute function mapping responses to real numbers (e.g., response length).
- 2.
Finite Variance: The attribute $a(y)$ has finite variance over the acceptable response distribution $\mathcal{A}_{x}$ for each query $x$, i.e., $\operatorname{Var}_{y\sim\mathcal{A}_{x}}[a(y)]=\sigma_{\mathcal{A}_{x}}^{2}<\infty$.
- 3.
Independent Sampling: Responses are independently sampled from their respective distributions.
- 4.
Model Capacity: The model can represent the true distribution given sufficient data.
- 5.
Uniform Bounded Variance: There exists a constant $\sigma_{\max}^{2}$ such that $\sigma_{\mathcal{A}_{x}}^{2}\leq\sigma_{\max}^{2}$ for all $x\in\mathcal{X}$.

### A.2 Bias Definition

The bias with respect to attribute $a$ is defined as:

$$ $B^{(k)}=\left|\mu_{\theta}^{(k)}-\mu_{\mathcal{A}}\right|,$ (30) $$

where:

- •
$\mu_{\theta}^{(k)}$ is the expected attribute value under the model after training with $k$ positive and $k$ negative samples per query.
- •
$\mu_{\mathcal{A}}=\mathbb{E}_{x\sim\mathcal{X}}\left[\mu_{\mathcal{A}_{x}}\right]$, with $\mu_{\mathcal{A}_{x}}=\mathbb{E}_{y\sim\mathcal{A}_{x}}\left[a(y)\right]$.

### A.3 Main Bias Result

###### Theorem 3 .

Under the stated assumptions, the expected bias $\mathbb{E}[B^{(k)}]$ decreases with the number of samples $k$ as:

$$ $\mathbb{E}[B^{(k)}]\leq\frac{C}{\sqrt{k}},$ (31) $$

where $C=\sigma_{\max}$ is a constant depending on the maximum variance of $a(y)$ over the acceptable responses.

###### Proof.

For each query $x$, consider the sample mean of the attribute over the $k$ positive responses:

$$ $\overline{a}_{x}^{(k)}=\frac{1}{k}\sum_{i=1}^{k}a(y_{x,i}^{+}),\quad y_{x,i}^{+}\sim\mathcal{A}_{x}.$ (32) $$

Since the $y_{x,i}^{+}$ are independent and identically distributed samples from $\mathcal{A}_{x}$, the expected value and variance of $\overline{a}_{x}^{(k)}$ are:

$$ $\displaystyle\mathbb{E}\left[\overline{a}_{x}^{(k)}\right]$ $\displaystyle=\mu_{\mathcal{A}_{x}},$ (33) $\displaystyle\operatorname{Var}\left(\overline{a}_{x}^{(k)}\right)$ $\displaystyle=\frac{\sigma_{\mathcal{A}_{x}}^{2}}{k}\leq\frac{\sigma_{\max}^{2}}{k}.$ (34) $$

Using the fact that for any random variable $Z$ with finite variance, the expected absolute deviation from its mean satisfies:

$$ $\mathbb{E}\left[\left|Z-\mathbb{E}[Z]\right|\right]\leq\sqrt{\operatorname{Var}[Z]},$ (35) $$

we have:

$$ $\mathbb{E}\left[\left|\overline{a}_{x}^{(k)}-\mu_{\mathcal{A}_{x}}\right|\right]\leq\sqrt{\frac{\sigma_{\mathcal{A}_{x}}^{2}}{k}}\leq\frac{\sigma_{\max}}{\sqrt{k}}.$ (36) $$

Averaging over all queries $x\in\mathcal{X}$:

$$ $\mathbb{E}_{x}\left[\left|\overline{a}_{x}^{(k)}-\mu_{\mathcal{A}_{x}}\right|\right]\leq\frac{\sigma_{\max}}{\sqrt{k}}.$ (37) $$

Since $\mu_{\theta}^{(k)}=\mathbb{E}_{x}\left[\overline{a}_{x}^{(k)}\right]$ and $\mu_{\mathcal{A}}=\mathbb{E}_{x}\left[\mu_{\mathcal{A}_{x}}\right]$, the expected bias is:

$$ $\displaystyle\mathbb{E}[B^{(k)}]$ $\displaystyle=\left|\mu_{\theta}^{(k)}-\mu_{\mathcal{A}}\right|=\left|\mathbb{E}_{x}\left[\overline{a}_{x}^{(k)}-\mu_{\mathcal{A}_{x}}\right]\right|$ (38) $\displaystyle\leq\mathbb{E}_{x}\left[\left|\overline{a}_{x}^{(k)}-\mu_{\mathcal{A}_{x}}\right|\right]\leq\frac{\sigma_{\max}}{\sqrt{k}}.$ (39) $$

Thus, the expected bias decreases with $k$ as $\frac{1}{\sqrt{k}}$.

∎

###### Corollary 2 .

As $k\to\infty$, the expected bias $\mathbb{E}[B^{(k)}]$ approaches zero:

$$ $\lim_{k\to\infty}\mathbb{E}[B^{(k)}]=0.$ (40) $$

###### Theorem 3 .

###### Proof.

###### Corollary 2 .

##### Implications

This theorem establishes a quantitative relationship between the bias $B^{(k)}$ and the number of samples $k$. It shows that incorporating multiple positive and negative responses per query reduces the bias with respect to attribute $a(y)$ at a rate proportional to $1/\sqrt{k}$. As $k$ increases, the model’s expected attribute value converges to the true expected attribute value over acceptable responses, leading to better alignment with human preferences.

## Appendix B Differentiating the Group Contrastive Loss from InfoNCE Loss

In this subsection, we compare our proposed weighted contrastive loss function with the InfoNCA loss function. We present both loss functions, derive their gradients rigorously, and characterize their stationary points. Based on this characterization, we discuss the properties of the convergence points in terms of what the models learn and their alignment with human preferences.

### B.1 Definitions of Loss Functions

##### InfoNCA Loss Function

The InfoNCA loss function is defined as:

$$ $L_{\text{InfoNCA}}=-\sum_{i=1}^{K}p_{i}^{\text{target}}\log p_{i}^{\text{model}},$ $$

where $p_{i}^{\text{target}}$ represents the target probability for the $i$-th response, calculated as

$$ $p_{i}^{\text{target}}=\frac{e^{r(x,y_{i})/\alpha}}{\sum_{j=1}^{K}e^{r(x,y_{j})/\alpha}},$ $$

and $p_{i}^{\text{model}}$ denotes the model’s predicted probability for the $i$-th response, given by

$$ $p_{i}^{\text{model}}=\frac{e^{s_{\theta}(y_{i}\mid x)}}{\sum_{j=1}^{K}e^{s_{\theta}(y_{j}\mid x)}}.$ $$

In this context, $x$ is the instruction or prompt provided to the model, and $\{y_{i}\}_{i=1}^{K}$ represents a set of $K$ responses generated for the instruction $x$. The term $r(x,y_{i})$ is the reward associated with the response $y_{i}$, while $s_{\theta}(y_{i}\mid x)=\log\left(P_{\theta}(y_{i}\mid x)/P_{\text{ref}}(y_{i}\mid x)\right)$ is the score for response $y_{i}$. The parameter $\alpha$ serves as a temperature parameter that controls the influence of the reward, and $K$ is the total number of responses considered for the instruction $x$.

##### Weighted Contrastive Loss Function

Our proposed weighted contrastive loss function is expressed as:

$$ $L_{\text{weighted}}=-\log\left(\frac{\sum\limits_{i\in Y^{+}}w_{i}e^{s_{\theta}(y_{i}\mid x)}}{\sum\limits_{j=1}^{K}w_{j}e^{s_{\theta}(y_{j}\mid x)}}\right),$ $$

where $Y^{+}$ is the set of positive responses with rewards above the mean, defined as $Y^{+}=\{y_{i}\mid S_{i}>S_{\text{mean}}\}$. Each response $y_{i}$ is assigned a weight $w_{i}=e^{\alpha\delta_{i}}$, where $\delta_{i}$ is the deviation of the reward score $S_{i}$ from the mean reward score $S_{\text{mean}}$. Specifically, $\delta_{i}=S_{i}-S_{\text{mean}}$ for responses in $Y^{+}$ and $\delta_{i}=S_{\text{mean}}-S_{i}$ for responses not in $Y^{+}$. The mean reward score $S_{\text{mean}}$ is calculated as

$$ $S_{\text{mean}}=\frac{1}{K}\sum_{j=1}^{K}S_{j},$ $$

where $K$ is the total number of responses for the query $x$. The term $s_{\theta}(y_{i}\mid x)$ denotes the model’s logit for response $y_{i}$, and $\alpha$ is a scaling hyperparameter that controls the influence of the deviation $\delta_{i}$.

### B.2 Gradient Analysis

To understand how each loss function influences the model during training, we derive the gradients with respect to the model logits $s_{\theta}(y_{i}\mid x)$ for both methods.

##### Gradient of InfoNCA Loss

###### Lemma 5 .

The gradient of the InfoNCA loss with respect to the model logits $s_{\theta}(y_{i}\mid x)$ is:

$$ $\frac{\partial L_{\text{InfoNCA}}}{\partial s_{\theta}(y_{i}\mid x)}=p_{i}^{\text{model}}-p_{i}^{\text{target}}.$ (41) $$

###### Proof.

The InfoNCA loss is:

$$ $L_{\text{InfoNCA}}=-\sum_{k=1}^{K}p_{k}^{\text{target}}\log p_{k}^{\text{model}}.$ (42) $$

Our goal is to compute $\frac{\partial L_{\text{InfoNCA}}}{\partial s_{\theta}(y_{i}\mid x)}$.

Since $p_{k}^{\text{target}}$ does not depend on $s_{\theta}(y_{i}\mid x)$ (the rewards are constants with respect to the model parameters), the derivative only affects the terms involving $p_{k}^{\text{model}}$.

First, express $\log p_{k}^{\text{model}}$ explicitly:

$$ $\log p_{k}^{\text{model}}=s_{\theta}(y_{k}\mid x)-\log\left(\sum_{j=1}^{K}e^{s_{\theta}(y_{j}\mid x)}\right).$ (43) $$

Now, compute the derivative of $\log p_{k}^{\text{model}}$ with respect to $s_{\theta}(y_{i}\mid x)$:

$$ $\frac{\partial\log p_{k}^{\text{model}}}{\partial s_{\theta}(y_{i}\mid x)}=\frac{\partial s_{\theta}(y_{k}\mid x)}{\partial s_{\theta}(y_{i}\mid x)}-\frac{\partial}{\partial s_{\theta}(y_{i}\mid x)}\log\left(\sum_{j=1}^{K}e^{s_{\theta}(y_{j}\mid x)}\right).$ (44) $$

Compute each term separately.

First term:

$$ $\frac{\partial s_{\theta}(y_{k}\mid x)}{\partial s_{\theta}(y_{i}\mid x)}=\delta_{ik},$ (45) $$

where $\delta_{ik}$ is the Kronecker delta, equal to $1$ if $i=k$ and 0 otherwise.

Second term:

Let $Z=\sum_{j=1}^{K}e^{s_{\theta}(y_{j}\mid x)}$. Then,

$$ $\frac{\partial}{\partial s_{\theta}(y_{i}\mid x)}\log Z=\frac{1}{Z}\frac{\partial Z}{\partial s_{\theta}(y_{i}\mid x)}.$ (46) $$

Compute $\frac{\partial Z}{\partial s_{\theta}(y_{i}\mid x)}$:

$$ $\frac{\partial Z}{\partial s_{\theta}(y_{i}\mid x)}=e^{s_{\theta}(y_{i}\mid x)}.$ (47) $$

Therefore,

$$ $\frac{\partial}{\partial s_{\theta}(y_{i}\mid x)}\log Z=\frac{e^{s_{\theta}(y_{i}\mid x)}}{Z}=p_{i}^{\text{model}}.$ (48) $$

Putting it all together:

$$ $\displaystyle\frac{\partial\log p_{k}^{\text{model}}}{\partial s_{\theta}(y_{i}\mid x)}$ $\displaystyle=\delta_{ik}-p_{i}^{\text{model}}.$ (49) $$

Now, compute the gradient of the loss:

$$ $\displaystyle\frac{\partial L_{\text{InfoNCA}}}{\partial s_{\theta}(y_{i}\mid x)}$ $\displaystyle=-\sum_{k=1}^{K}p_{k}^{\text{target}}\frac{\partial\log p_{k}^{\text{model}}}{\partial s_{\theta}(y_{i}\mid x)}$ (50) $\displaystyle=-\sum_{k=1}^{K}p_{k}^{\text{target}}\left(\delta_{ik}-p_{i}^{\text{model}}\right)$ (51) $\displaystyle=-\left(p_{i}^{\text{target}}-p_{i}^{\text{model}}\sum_{k=1}^{K}p_{k}^{\text{target}}\right).$ (52) $$

Since $\sum_{k=1}^{K}p_{k}^{\text{target}}=1$, we have:

$$ $\sum_{k=1}^{K}p_{k}^{\text{target}}=1\implies\sum_{k=1}^{K}p_{k}^{\text{target}}=1.$ (53) $$

Therefore,

$$ $\frac{\partial L_{\text{InfoNCA}}}{\partial s_{\theta}(y_{i}\mid x)}=-\left(p_{i}^{\text{target}}-p_{i}^{\text{model}}\cdot 1\right)=p_{i}^{\text{model}}-p_{i}^{\text{target}}.$ (54) $$

∎

###### Lemma 5 .

###### Proof.

##### Gradient of Weighted Contrastive Loss

###### Lemma 6 .

The gradient of the weighted contrastive loss with respect to the model logits $s_{\theta}(y_{i}\mid x)$ is:

$$ $\frac{\partial L_{\text{weighted}}}{\partial s_{\theta}(y_{i}\mid x)}=p_{i}^{\text{weighted}}-p_{i}^{\text{pos}}$ (55) $$

where:

$$ $p_{i}^{\text{weighted}}=\frac{w_{i}e^{s_{\theta}(y_{i}\mid x)}}{\sum_{j=1}^{K}w_{j}e^{s_{\theta}(y_{j}\mid x)}},\qquad\qquad p_{i}^{\text{pos}}=\frac{w_{i}e^{s_{\theta}(y_{i}\mid x)}}{\sum_{k\in Y^{+}}w_{k}e^{s_{\theta}(y_{k}\mid x)}}\cdot\mathbb{I}_{y_{i}\in Y^{+}},$ (56) $$

and $\mathbb{I}_{y_{i}\in Y^{+}}$ is the indicator function, equal to $1$ if $y_{i}\in Y^{+}$ and 0 otherwise.

###### Proof.

Let us denote:

$$ $A=\sum_{k\in Y^{+}}w_{k}e^{s_{\theta}(y_{k}\mid x)},\quad Z=\sum_{j=1}^{K}w_{j}e^{s_{\theta}(y_{j}\mid x)}.$ (57) $$

The weighted contrastive loss is:

$$ $L_{\text{weighted}}=-\log\left(\frac{A}{Z}\right)=-\log A+\log Z.$ (58) $$

Compute the derivative with respect to $s_{\theta}(y_{i}\mid x)$:

$$ $\frac{\partial L_{\text{weighted}}}{\partial s_{\theta}(y_{i}\mid x)}=-\frac{1}{A}\frac{\partial A}{\partial s_{\theta}(y_{i}\mid x)}+\frac{1}{Z}\frac{\partial Z}{\partial s_{\theta}(y_{i}\mid x)}.$ (59) $$

Compute $\frac{\partial A}{\partial s_{\theta}(y_{i}\mid x)}$:

$$ $\frac{\partial A}{\partial s_{\theta}(y_{i}\mid x)}=w_{i}e^{s_{\theta}(y_{i}\mid x)}\cdot\mathbb{I}_{y_{i}\in Y^{+}}.$ (60) $$

Compute $\frac{\partial Z}{\partial s_{\theta}(y_{i}\mid x)}$:

$$ $\frac{\partial Z}{\partial s_{\theta}(y_{i}\mid x)}=w_{i}e^{s_{\theta}(y_{i}\mid x)}.$ (61) $$

Substitute back into the gradient:

$$ $\displaystyle\frac{\partial L_{\text{weighted}}}{\partial s_{\theta}(y_{i}\mid x)}$ $\displaystyle=-\frac{1}{A}w_{i}e^{s_{\theta}(y_{i}\mid x)}\cdot\mathbb{I}_{y_{i}\in Y^{+}}+\frac{1}{Z}w_{i}e^{s_{\theta}(y_{i}\mid x)}$ (62) $\displaystyle=w_{i}e^{s_{\theta}(y_{i}\mid x)}\left(\frac{1}{Z}-\frac{\mathbb{I}_{y_{i}\in Y^{+}}}{A}\right).$ (63) $$

Recognize that:

$$ $p_{i}^{\text{weighted}}=\frac{w_{i}e^{s_{\theta}(y_{i}\mid x)}}{Z},\quad p_{i}^{\text{pos}}=\frac{w_{i}e^{s_{\theta}(y_{i}\mid x)}}{A}\cdot\mathbb{I}_{y_{i}\in Y^{+}}.$ (64) $$

Therefore:

$$ $\frac{\partial L_{\text{weighted}}}{\partial s_{\theta}(y_{i}\mid x)}=p_{i}^{\text{weighted}}-p_{i}^{\text{pos}}.$ (65) $$

Since $p_{i}^{\text{pos}}=0$ when $y_{i}\notin Y^{+}$, we have:

$$ $\frac{\partial L_{\text{weighted}}}{\partial s_{\theta}(y_{i}\mid x)}=\begin{cases}p_{i}^{\text{weighted}}-p_{i}^{\text{pos}},&\text{if }y_{i}\in Y^{+},\\ p_{i}^{\text{weighted}}-0=p_{i}^{\text{weighted}},&\text{if }y_{i}\in Y^{-}.\end{cases}$ (66) $$

However, this suggests that the gradient is always positive for negative examples. In other words, given $w_{i}$ and $Z$ are positive, $e^{s_{\theta}(y_{i}\mid x)}$ keeps increasing. But note that $s_{\theta}(y_{i}\mid x)=-\log\left(P_{\theta}(y_{i}\mid x)\right)$. Hence $\frac{1}{P_{\theta}(y_{i}\mid x)}$ keeps increasing implying that $P_{\theta}(y_{i}\mid x)$ keeps decreasing. i.e. at the stationary point, $P_{\theta}(y_{i}\mid x)\to 0$ for all negative examples, $y_{i}\in Y^{-}$.

Now let us examine the positive examples. The gradient simplifies to $w_{i}e^{s_{\theta}(y_{i}\mid x)}\left(\frac{1}{Z}-\frac{1}{A}\right)$. Since $Z\geq A$, $\frac{1}{Z}\leq\frac{1}{A}$. Hence the gradient term with respect to $s_{\theta}(y_{i}\mid x)$ is negative.
Notice that $e^{s_{\theta}(y_{i}\mid x)}=\frac{1}{P_{\theta}(y_{i}\mid x)}$. A negative gradient implies that $\frac{1}{P_{\theta}(y_{i}\mid x)}$ decreases, implying that $P_{\theta}(y_{i}\mid x)$ increases for all positive examples $y_{i}\in Y^{+}$.

∎

We now provide the gradients directly in terms of $P_{\theta}(y_{j}\mid x)$ instead of the scores $s_{\theta}(y_{j}\mid x)$, for easy interpretibility in terms of the probabilities.

###### Lemma 7 .

Let the weighted contrastive loss be defined as:

$$ $L_{\text{weighted}}=-\log\left(\frac{V}{U}\right)=-\log V+\log U,$ $$

where

$$ $U=\sum_{j=1}^{K}u_{j}P_{\theta}(y_{j}\mid x),\quad V=\sum_{i\in Y^{+}}u_{i}P_{\theta}(y_{i}\mid x),$ $$

and

$$ $u_{i}=\frac{w_{i}}{P_{\text{ref}}(y_{i}\mid x)},$ $$

with $w_{i}=e^{\alpha\delta_{i}}$, $P_{\theta}(y_{i}\mid x)$ being the model probability for response $y_{i}$, and $P_{\text{ref}}(y_{i}\mid x)$ being the reference model probability.

Then, the gradient of the weighted contrastive loss with respect to $P_{\theta}(y_{i}\mid x)$ is given by:

- •
For positive examples ($y_{i}\in Y^{+}$):
$\frac{\partial L_{\text{weighted}}}{\partial P_{\theta}(y_{i}\mid x)}=u_{i}\left(\frac{1}{U}-\frac{1}{V}\right),$
(67)
- •
For negative examples ($y_{i}\notin Y^{+}$):
$\frac{\partial L_{\text{weighted}}}{\partial P_{\theta}(y_{i}\mid x)}=\frac{u_{i}}{U}.$
(68)

###### Proof.

Using the score function $s_{\theta}(y_{i}\mid x)=\log\left(\frac{P_{\theta}(y_{i}\mid x)}{P_{\text{ref}}(y_{i}\mid x)}\right)$, we have $e^{s_{\theta}(y_{i}\mid x)}=\dfrac{P_{\theta}(y_{i}\mid x)}{P_{\text{ref}}(y_{i}\mid x)}$.

The weighted contrastive loss becomes:

$$ $L_{\text{weighted}}=-\log\left(\frac{\displaystyle\sum_{i\in Y^{+}}w_{i}e^{s_{\theta}(y_{i}\mid x)}}{\displaystyle\sum_{j=1}^{K}w_{j}e^{s_{\theta}(y_{j}\mid x)}}\right)=-\log\left(\frac{\displaystyle\sum_{i\in Y^{+}}w_{i}\dfrac{P_{\theta}(y_{i}\mid x)}{P_{\text{ref}}(y_{i}\mid x)}}{\displaystyle\sum_{j=1}^{K}w_{j}\dfrac{P_{\theta}(y_{j}\mid x)}{P_{\text{ref}}(y_{j}\mid x)}}\right)=-\log\left(\frac{V}{U}\right),$ $$

where $u_{i}=\dfrac{w_{i}}{P_{\text{ref}}(y_{i}\mid x)}$, $V=\sum_{i\in Y^{+}}u_{i}P_{\theta}(y_{i}\mid x)$, and $U=\sum_{j=1}^{K}u_{j}P_{\theta}(y_{j}\mid x)$.

We compute the gradient of $L_{\text{weighted}}$ with respect to $P_{\theta}(y_{i}\mid x)$:

$$ $\frac{\partial L_{\text{weighted}}}{\partial P_{\theta}(y_{i}\mid x)}=-\frac{1}{V}\cdot\frac{\partial V}{\partial P_{\theta}(y_{i}\mid x)}+\frac{1}{U}\cdot\frac{\partial U}{\partial P_{\theta}(y_{i}\mid x)}.$ $$

Case 1: For $y_{i}\in Y^{+}$:

$$ $\frac{\partial V}{\partial P_{\theta}(y_{i}\mid x)}=u_{i},\quad\frac{\partial U}{\partial P_{\theta}(y_{i}\mid x)}=u_{i}.$ $$

Thus,

$$ $\frac{\partial L_{\text{weighted}}}{\partial P_{\theta}(y_{i}\mid x)}=-\frac{u_{i}}{V}+\frac{u_{i}}{U}=u_{i}\left(\frac{1}{U}-\frac{1}{V}\right).$ $$

Case 2: For $y_{i}\notin Y^{+}$:

$$ $\frac{\partial V}{\partial P_{\theta}(y_{i}\mid x)}=0,\quad\frac{\partial U}{\partial P_{\theta}(y_{i}\mid x)}=u_{i}.$ $$

Thus,

$$ $\frac{\partial L_{\text{weighted}}}{\partial P_{\theta}(y_{i}\mid x)}=0+\frac{u_{i}}{U}=\frac{u_{i}}{U}.$ $$

∎

###### Corollary 3 .

The sign of the gradient indicates the optimization direction:

- •
For positive examples ($y_{i}\in Y^{+}$), since $V\leq U$, we have $\dfrac{1}{U}-\dfrac{1}{V}\leq 0$. Therefore, the gradient $\dfrac{\partial L_{\text{weighted}}}{\partial P_{\theta}(y_{i}\mid x)}\leq 0$, and minimizing $L_{\text{weighted}}$ involves increasing $P_{\theta}(y_{i}\mid x)$.
- •
For negative examples ($y_{i}\notin Y^{+}$), the gradient $\dfrac{\partial L_{\text{weighted}}}{\partial P_{\theta}(y_{i}\mid x)}>0$, and minimizing $L_{\text{weighted}}$ involves decreasing $P_{\theta}(y_{i}\mid x)$.

###### Proof.

As established in the lemma:

For positive examples ($y_{i}\in Y^{+}$):
Since $V=\sum_{i\in Y^{+}}u_{i}P_{\theta}(y_{i}\mid x)$ and $U=V+\sum_{j\notin Y^{+}}u_{j}P_{\theta}(y_{j}\mid x)$, it follows that $V\leq U$ and thus $\dfrac{1}{U}-\dfrac{1}{V}\leq 0$.

Therefore, the gradient:

$$ $\frac{\partial L_{\text{weighted}}}{\partial P_{\theta}(y_{i}\mid x)}=u_{i}\left(\frac{1}{U}-\frac{1}{V}\right)\leq 0.$ $$

A negative gradient indicates that increasing $P_{\theta}(y_{i}\mid x)$ will decrease $L_{\text{weighted}}$. Hence, to minimize the loss, we should increase $P_{\theta}(y_{i}\mid x)$ for positive examples.

For negative examples ($y_{i}\notin Y^{+}$):
The gradient is:

$$ $\frac{\partial L_{\text{weighted}}}{\partial P_{\theta}(y_{i}\mid x)}=\frac{u_{i}}{U}>0,$ $$

since $u_{i}>0$ and $U>0$. A positive gradient indicates that decreasing $P_{\theta}(y_{i}\mid x)$ will decrease $L_{\text{weighted}}$. Therefore, to minimize the loss, we should decrease $P_{\theta}(y_{i}\mid x)$ for negative examples.
∎

###### Lemma 6 .

###### Proof.

###### Lemma 7 .

###### Proof.

###### Corollary 3 .

###### Proof.

## Appendix C Characterization of Stationary Points

We now characterize the stationary points of both loss functions.

### C.1 Stationary Points of the InfoNCA Loss Function

###### Theorem 4 .

For the InfoNCA loss, the stationary points occur when:

$$ $p_{i}^{\text{model}}=p_{i}^{\text{target}},\quad\forall i\in\{1,\dots,K\}.$ (69) $$

###### Proof.

Stationary points are defined by the condition:

$$ $\frac{\partial L_{\text{InfoNCA}}}{\partial s_{\theta}(y_{i}\mid x)}=0,\quad\forall i.$ (70) $$

From the gradient:

$$ $\frac{\partial L_{\text{InfoNCA}}}{\partial s_{\theta}(y_{i}\mid x)}=p_{i}^{\text{model}}-p_{i}^{\text{target}},$ (71) $$

setting the gradient to zero yields:

$$ $p_{i}^{\text{model}}=p_{i}^{\text{target}},\quad\forall i.$ (72) $$

∎

###### Remark 3 .

This stationary point is suboptimal because $p_{i}^{\text{model}}$ expands to:

$$ $p_{i}^{\text{model}}=\frac{e^{\log P_{\theta}(y_{i}|x)-\log P_{\text{ref}}(y_{i}|x)}}{\sum_{j=1}^{K}e^{\log P_{\theta}(y_{j}|x)-\log P_{\text{ref}}(y_{j}|x)}}$ $$

Rather than equating the soft-max of the difference between $\log P_{\theta}(y_{i}|x)$ and $\log P_{\text{ref}}(y_{i}|x)$ to $p_{i}^{\text{target}}$, optimality may require directly setting $\log P_{\theta}(y|x)$ to match the softmax of the target scores.

###### Theorem 4 .

###### Proof.

###### Remark 3 .

### C.2 Stationary Points of the Weighted Contrastive Loss under Simplifying Assumptions

###### Lemma 8 .

Consider the weighted contrastive loss function in a simplified scenario with the following conditions: There are $N^{+}$ positive examples, each with weight $w^{+}$, and $N^{-}$ negative examples, each with weight $w^{-}$. All positive examples have the same score $s^{(t)}$ at iteration $t$, and all negative examples have the same score $s^{(t)}$ at iteration $t$. Then, the update rule for the score $s^{(t)}$ of the positive examples at iteration $t+1$ is given by

$$ $s^{(t+1)}=s^{(t)}+\eta\left(\frac{N^{-}w^{-}}{N^{+}(N^{+}w^{+}+N^{-}w^{-})}\right),$ (73) $$

where $\eta$ is the learning rate.

###### Proof.

Let $Y^{+}$ denote the set of positive examples and $Y^{-}$ the set of negative examples, with $N^{+}$ and $N^{-}$ examples respectively for a total of $K=N^{+}+N^{-}$ examples. With weights $w^{+}$ and $w^{-}$ assigned to positive and negative examples respectively, and logits $s^{(t)}$ for both classes at timestep $t$, the weighted contrastive loss function is defined as:

$$ $L_{\text{weighted}}(\theta)=-\log\left(\frac{\sum\limits_{i\in Y^{+}}w_{i}e^{s_{i}}}{\sum\limits_{j=1}^{K}w_{j}e^{s_{j}}}\right),$ (74) $$

where $w_{i}=w^{+}$ and $s_{i}=s^{(t)}$ for $i\in Y^{+}$, and $w_{j}=w^{-}$ and $s_{j}=s^{(t)}$ for $j\in Y^{-}$.

Compute the numerator $A$ and the denominator $Z$ of the loss function:

$$ $\displaystyle A$ $\displaystyle=\sum_{i\in Y^{+}}w_{i}e^{s_{i}}=N^{+}w^{+}e^{s^{(t)}},$ (75) $\displaystyle Z$ $\displaystyle=\sum_{j=1}^{K}w_{j}e^{s_{j}}=N^{+}w^{+}e^{s^{(t)}}+N^{-}w^{-}e^{s^{(t)}}=e^{s^{(t)}}(N^{+}w^{+}+N^{-}w^{-}).$ (76) $$

For positive examples $i\in Y^{+}$, the weighted probability $p_{i}^{\text{weighted}}$ and the positive probability $p_{i}^{\text{pos}}$ are:

$$ $\displaystyle p_{i}^{\text{weighted}}$ $\displaystyle=\frac{w^{+}e^{s^{(t)}}}{Z}=\frac{w^{+}}{N^{+}w^{+}+N^{-}w^{-}},$ (77) $\displaystyle p_{i}^{\text{pos}}$ $\displaystyle=\frac{w^{+}e^{s^{(t)}}}{A}=\frac{w^{+}}{N^{+}w^{+}}=\frac{1}{N^{+}}.$ (78) $$

The gradient of the loss with respect to $s^{(t)}$ for positive examples is:

$$ $\frac{\partial L_{\text{weighted}}}{\partial s^{(t)}}=p_{i}^{\text{weighted}}-p_{i}^{\text{pos}}=\frac{w^{+}}{N^{+}w^{+}+N^{-}w^{-}}-\frac{1}{N^{+}}.$ (79) $$

To simplify this expression, we find a common denominator $D=N^{+}(N^{+}w^{+}+N^{-}w^{-})$:

$$ $\displaystyle\frac{\partial L_{\text{weighted}}}{\partial s^{(t)}}$ $\displaystyle=\frac{w^{+}N^{+}-(N^{+}w^{+}+N^{-}w^{-})}{D}$ (80) $\displaystyle=\frac{w^{+}N^{+}-N^{+}w^{+}-N^{-}w^{-}}{N^{+}(N^{+}w^{+}+N^{-}w^{-})}$ (81) $\displaystyle=\frac{-N^{-}w^{-}}{N^{+}(N^{+}w^{+}+N^{-}w^{-})}.$ (82) $$

The update rule for $s^{(t)}$ is then:

$$ $s^{(t+1)}=s^{(t)}-\eta\frac{\partial L_{\text{weighted}}}{\partial s^{(t)}}=s^{(t)}+\eta\left(\frac{N^{-}w^{-}}{N^{+}(N^{+}w^{+}+N^{-}w^{-})}\right).$ (83) $$

This completes the proof.
∎

###### Corollary 4 .

Assuming the initial scores are zero ($s^{(0)}=0$), the score $s^{(t)}$ of the positive examples at iteration $t$ is given by

$$ $s^{(t)}=t\eta\left(\frac{N^{-}w^{-}}{N^{+}(N^{+}w^{+}+N^{-}w^{-})}\right).$ (84) $$

###### Proof.

From the update rule established in the lemma,

$$ $s^{(t+1)}=s^{(t)}+c,$ (85) $$

where

$$ $c=\eta\left(\frac{N^{-}w^{-}}{N^{+}(N^{+}w^{+}+N^{-}w^{-})}\right).$ (86) $$

Since $s^{(0)}=0$, we have

$$ $\displaystyle s^{(1)}$ $\displaystyle=s^{(0)}+c=c,$ (87) $\displaystyle s^{(2)}$ $\displaystyle=s^{(1)}+c=2c,$ (88) $\displaystyle\ \ \vdots$ (89) $\displaystyle s^{(t)}$ $\displaystyle=tc.$ (90) $$

Substituting $c$ back into the expression, we obtain

$$ $s^{(t)}=t\eta\left(\frac{N^{-}w^{-}}{N^{+}(N^{+}w^{+}+N^{-}w^{-})}\right).$ (91) $$

∎

###### Corollary 5 .

In the special case where there is one positive example ($N^{+}=1$) and one negative example ($N^{-}=1$), and the weights are $w^{+}=w^{-}=1$ (as in Direct Preference Optimization), the score $s^{(t)}$ at iteration $t$ is:

$$ $s^{(t)}=\frac{\eta t}{2}.$ (92) $$

###### Proof.

Substituting $N^{+}=N^{-}=1$ and $w^{+}=w^{-}=1$ into the expression for $s^{(t)}$:

$$ $\displaystyle s^{(t)}$ $\displaystyle=t\eta\left(\frac{1\times 1}{1\times(1\times 1+1\times 1)}\right)$ (93) $\displaystyle=t\eta\left(\frac{1}{1\times(1+1)}\right)$ (94) $\displaystyle=t\eta\left(\frac{1}{2}\right)$ (95) $\displaystyle=\frac{\eta t}{2}.$ (96) $$

∎

###### Lemma 9 .

Consider the general case where positive examples may have different weights $w_{i}^{+}$, and each positive example $i$ has its own score $s_{i}^{(t)}$ at iteration $t$. Assuming initial scores $s_{i}^{(0)}=0$ for all positive examples, the score $s_{i}^{(t)}$ of positive example $i$ at iteration $t$, up to a linear approximation, is given by

$$ $s_{i}^{(t)}=t\eta w_{i}^{+}\left(\frac{B_{0}}{A_{0}Z_{0}}\right),$ (97) $$

where $A_{0}=\sum_{k\in Y^{+}}w_{k}^{+}$, $B_{0}=\sum_{j\in Y^{-}}w_{j}^{-}$, $Z_{0}=A_{0}+B_{0}$, and $\eta$ is the learning rate.

###### Proof.

At iteration $t=0$, the initial scores are $s_{i}^{(0)}=0$ for all $i\in Y^{+}$. The sums are:

$$ $\displaystyle A_{0}$ $\displaystyle=\sum_{k\in Y^{+}}w_{k}^{+}e^{s_{k}^{(0)}}=\sum_{k\in Y^{+}}w_{k}^{+}=W^{+},$ (98) $\displaystyle B_{0}$ $\displaystyle=\sum_{j\in Y^{-}}w_{j}^{-}e^{s_{j}^{(0)}}=\sum_{j\in Y^{-}}w_{j}^{-}=W^{-}.$ (99) $$

The total sum is $Z_{0}=A_{0}+B_{0}=W^{+}+W^{-}$.

The gradient for each positive example $i$ at $t=0$ is:

$$ $\frac{\partial L_{\text{weighted}}}{\partial s_{i}^{(0)}}=-w_{i}^{+}e^{s_{i}^{(0)}}\left(\frac{B_{0}}{A_{0}Z_{0}}\right)=-w_{i}^{+}\left(\frac{B_{0}}{A_{0}Z_{0}}\right).$ (100) $$

The update rule is:

$$ $s_{i}^{(1)}=s_{i}^{(0)}-\eta\frac{\partial L_{\text{weighted}}}{\partial s_{i}^{(0)}}=\eta w_{i}^{+}\left(\frac{B_{0}}{A_{0}Z_{0}}\right).$ (101) $$

Assuming that the term $\frac{B_{0}}{A_{0}Z_{0}}$ remains approximately constant over iterations (which holds when $\eta$ is small and changes in $s_{i}^{(t)}$ are small), the score at iteration $t$ is:

$$ $s_{i}^{(t)}=t\eta w_{i}^{+}\left(\frac{B_{0}}{A_{0}Z_{0}}\right).$ (102) $$

∎

###### Remark 4 .

The approximation assumes that $A_{t}$, $B_{t}$, and $Z_{t}$ remain close to their initial values $A_{0}$, $B_{0}$, and $Z_{0}$ over the iterations considered, and the score values remain small. This is reasonable for small learning rates $\eta$ and a limited number of iterations $t$.

###### Lemma 8 .

###### Proof.

###### Corollary 4 .

###### Proof.

###### Corollary 5 .

###### Proof.

###### Lemma 9 .

###### Proof.

###### Remark 4 .

### C.3 Stationary Points of the Weighted Contrastive Loss

We now analyze the stationary points of our weighted contrastive loss function.

###### Lemma 10 .

For the weighted contrastive loss function, the stationary point occurs when the probabilities of the negative samples approach zero, i.e.,

$$ $P_{\theta}(y_{i}\mid x)\to 0\quad\text{for all }y_{i}\in Y^{-}.$ (103) $$

###### Proof.

From Lemma [2](#Thmlemma2), the gradient of the weighted contrastive loss with respect to the model logits $s_{\theta}(y_{i}\mid x)$ is:

$$ $\frac{\partial L_{\text{weighted}}}{\partial s_{\theta}(y_{i}\mid x)}=\begin{cases}p_{i}^{\text{weighted}}-p_{i}^{\text{pos}},&\text{if }y_{i}\in Y^{+},\\ p_{i}^{\text{weighted}},&\text{if }y_{i}\in Y^{-}.\end{cases}$ (104) $$

At a stationary point, the gradient must be zero for all $y_{i}$. Consider the negative samples $y_{i}\in Y^{-}$. Setting the gradient to zero yields:

$$ $\frac{\partial L_{\text{weighted}}}{\partial s_{\theta}(y_{i}\mid x)}=p_{i}^{\text{weighted}}=0.$ (105) $$

Since $p_{i}^{\text{weighted}}$ is the normalized weighted probability of $y_{i}$, given by:

$$ $p_{i}^{\text{weighted}}=\frac{w_{i}e^{s_{\theta}(y_{i}\mid x)}}{\sum_{j=1}^{K}w_{j}e^{s_{\theta}(y_{j}\mid x)}},$ (106) $$

and $w_{i}>0$, the only way for $p_{i}^{\text{weighted}}$ to be zero is if $e^{s_{\theta}(y_{i}\mid x)}=0$, which implies:

$$ $s_{\theta}(y_{i}\mid x)\to-\infty\quad\implies\quad P_{\theta}(y_{i}\mid x)\to 0\quad\text{for }y_{i}\in Y^{-}.$ (107) $$

Similarly, for positive samples $y_{i}\in Y^{+}$, the gradient is:

$$ $\frac{\partial L_{\text{weighted}}}{\partial s_{\theta}(y_{i}\mid x)}=p_{i}^{\text{weighted}}-p_{i}^{\text{pos}}=0.$ (108) $$

This implies:

$$ $p_{i}^{\text{weighted}}=p_{i}^{\text{pos}}.$ (109) $$

Since the probabilities of the negative samples approach zero, the denominator in $p_{i}^{\text{weighted}}$ becomes:

$$ $\sum_{j=1}^{K}w_{j}e^{s_{\theta}(y_{j}\mid x)}\approx\sum_{k\in Y^{+}}w_{k}e^{s_{\theta}(y_{k}\mid x)}.$ (110) $$

Therefore, $p_{i}^{\text{weighted}}\approx p_{i}^{\text{pos}}$, satisfying the condition for the gradient to be zero for positive samples.

Thus, at the stationary point, the probabilities of the negative samples approach zero.
∎

###### Remark 5 .

When the probabilities of the negative samples approach zero, the scores $s_{\theta}(y_{i}\mid x)$ for $y_{i}\in Y^{-}$ tend to $-\infty$. Since:

$$ $e^{s_{\theta}(y_{i}\mid x)}=\frac{P_{\theta}(y_{i}\mid x)}{P_{\text{ref}}(y_{i}\mid x)}\to 0,$ (111) $$

the weighted contributions of the negative samples to the numerator and denominator of $L_{\text{weighted}}$ become negligible.

Consequently, the numerator and denominator of $L_{\text{weighted}}$ become equal:

$$ $\sum_{i\in Y^{+}}w_{i}e^{s_{\theta}(y_{i}\mid x)}\approx\sum_{j=1}^{K}w_{j}e^{s_{\theta}(y_{j}\mid x)}.$ (112) $$

Therefore:

$$ $L_{\text{weighted}}=-\log\left(\frac{\sum\limits_{i\in Y^{+}}w_{i}e^{s_{\theta}(y_{i}\mid x)}}{\sum\limits_{j=1}^{K}w_{j}e^{s_{\theta}(y_{j}\mid x)}}\right)\approx-\log 1=0.$ (113) $$

This implies that the loss vanishes when the probabilities of the negative samples approach zero, indicating that the model has successfully minimized the loss by focusing entirely on the positive responses.

###### Lemma 10 .

###### Proof.

###### Remark 5 .

## Appendix D Baselines Used for Comparison

When dealing with reward datasets where each instruction has more than two
$K>2$ responses, one common approach is to convert the data into pairwise preferences and then apply preference optimization techniques such as Direct Preference Optimization (DPO). Several strategies can be adopted for this purpose, each offering distinct trade-offs in terms of dataset richness and computational overhead.One straightforward method, as implemented by Zephyr Tunstall et al., (2023), involves selecting the response with the highest reward and pairing it with a randomly chosen response from the remaining responses for each instruction. Another variant involves pairing the highest-rewarded response with the lowest-rewarded response for each instruction, ensuring a clear distinction between preferences.Additionally, alternative baselines can be explored to enhance performance by incorporating more suboptimal responses during training. By applying DPO to combinations of responses, we can significantly expand the preference dataset and potentially achieve improved optimization. Two notable baselines in this context are:

DPOx$\binom{K}{2}$:
In this approach, all possible pairwise combinations of $\binom{K}{2}$ are generated, and DPO is applied to the entire combinatorial dataset. This method ensures the model is exposed to a comprehensive range of preference relationships, including those involving suboptimal responses.

DPOx$(K-1)$:
Here, the response with the highest reward is paired individually with each of the remaining $(K-1)$ responses. This strategy emphasizes the contrast between the top response and all others, potentially reinforcing the model’s understanding of optimal preferences.

Other baselines, such as InfoNCA and NCA, suggest that naively applying DPO to combinations of responses may lead to suboptimal performance. They leveraged Noise Contrastive Estimation (NCE) to bridge the gap in handling reward datasets explicitly annotated with scalar evaluations. According to their findings, the theoretical guarantees they provide ensure convergence, which is not guaranteed when applying DPO in this manner.

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: - Could the authors elaborate on potential failure modes for MPO? For instance, how would the mean-based partitioning perform if the reward distribution for a prompt is strongly bimodal?
- Regarding the On-Policy Data: In your on-policy experiments, you discard the median-reward response. What was the rationale for this decision?
- You mentioned skipped sets. How common are these in your experiments?
soundness: 3
strengths: - The paper addresses an important challenge in the alignment process, and the presented method is a clean and intuitive generalization of DPO, moving from pairwise to set-wise comparisons.
- The method is shown to achieve state-of-the-art results across a variety of models, benchmarks, and training paradigms (off-policy, on-policy, iterative), demonstrating its robustness and effectiveness, especially on AlpacaEval2.
- The paper provides both theoretical motivation (Theorems 1 & 2) and strong empirical validation for its core claims, particularly the benefits of using more responses and the robustness of 2-bin partitioning over full-ranking.
- Strong selection of benchmarks like AlpacaEval 2.0, Arena-Hard, and MT-Bench indicates generalizability of the approach.
summary: The paper tackles the important challenge of aligning LLMs in the post-training and introduces  Multi-Preference Optimization (MPO), a generalization of Direct Preference Optimization (DPO) that extends beyond pairwise comparisons. The approach optimizes over the entire sets of selected and rejected responses, potentially capturing valuable supervisory signal. The method is supported by a theoretical analysis proving that using n responses leads to a faster convergence rate (O(1/√n)) and that MPO's 2-bin partitioning is more robust to reward model noise than full-ranking methods like Plackett-Luce. The paper also introduces Weighted MPO (W-MPO), which uses reward score deviations to create an implicit curriculum by up-weighting informative outlier responses. The approach shows strong empirical results with an improvement of up to 17.5% win rate on AlpacaEval2 in the on-policy iterative setting, and state-of-the-art results in off-policy settings and mostly equivalent or better results on Arena-Hard and MT-Bench.
weaknesses: - The on-policy results depend on a single reward model. It is possible that MPO is particularly good at optimizing for the specific reward distribution of the Skywork RM, and its gains might be less pronounced with other RMs.
-  The theoretical result on noise robustness (Theorem 2) relies on a specific "spacing-scaled" noise model. It is unclear how realistic the assumption is.
- Relying on RM leaves actual alignment to be questionable; the study would benefit from human evaluation and more qualitative case studies.

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 2
questions: see weakness
soundness: 3
strengths: 1. This paper provides theoretical evidence on why MPO works better than other DPO-style methods.
2. The experiments are conducted and MPO are compared with several strong baselines.
summary: Traditional DPO methods only allow a pair of preference data to be trained. In this paper, they propose MultiPreferenceOptimization(MPO), a generalization of DPO that optimizes over entire sets of selected and rejected responses. Within the paper, authtors provide some theoretical evident on why MPO works better than other DPO-style methods. Experiments are conducted on several open-ended benchmarks and the results show that MPO achieves better performance than other methods.
weaknesses: 1. What is the difference between the "Off-policySetting" and "On-policySetting"? It seems that they only differ in the initial model (off-policy uses a weaker sft model, while the on-policy uses a stronger open-sourced instruct model). If so, why they get this name? Based on my understanding, Off-policy and On-policy should be different in how they are trained (sample from base model or from current policy model).
2. In the experiments, it seems some of strong baseline models are missing -- SimDPO [1]、BMC[2] etc. The authors should enrich their baseline comparsion.
3. Except for performance improvement, authors should also conduct some analysis on the strength of their approaches (e.g., case study). What is the additional advantages of MPO? (the authors provide results on data efficiency)

---

confidence: 5
contribution: 2
initial_rating: 0
presentation: 1
questions: N/A
soundness: 2
strengths: N/A
summary: The main text of this paper is 10 pages, which exceeds the page limit for ICLR submissions. According to ICLR policy, this paper should be desk-rejected.
weaknesses: N/A

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: See Weaknesses
soundness: 3
strengths: MPO retains the simplicity and elegance of DPO while naturally extending it to handle multiple responses per prompt, without the extra cost of full ranking or reward calibration. 
It achieves consistent state-of-the-art performance across model sizes and training regimes, scaling effectively with more responses per query and remaining competitive even under limited data or compute budgets.
summary: This paper introduces a set-level, contrastive preference optimization framework that generalizes DPO with groupwise. 
Empirically, MPO and W-MPO deliver state-of-the-art results on AlpacaEval2 (WR, LC-WR), Arena-Hard, and MT-Bench across both off-policy and on-policy training regimes, with performance in some cases approaching that of GPT-4o on AlpacaEval2.
weaknesses: Like DPO, MPO’s objective relies on the log-ratio between the policy and the reference model. It would be useful to analyze how sensitive performance is to the choice or vintage of the reference model (e.g., Llama vs. Qwen families).

W-MPO weights samples by their absolute deviation from the mean. Why use absolute deviation, and why the mean specifically? Including ablations with alternative robust statistics (e.g., median, trimmed mean, quantiles) could clarify stability under skewed or noisy reward distributions.

In on-policy settings, top-k and bottom-k responses depend on a particular reward model. How robust is MPO/W-MPO to miscalibration or domain shift in that scorer? Cross-reward or human-labeled validation would strengthen the reliability of the findings.

The main paper appears to be 10 pages long, while the conference strictly limits submissions to 9 pages. Please ensure the paper adheres to the page limit requirements.


Decision: Reject