Soft Preference Optimization: Aligning Language Models to Expert Distributions

## Abstract

Abstract We propose Soft Preference Optimization (SPO), a method for aligning generative models, such as Large Language Models (LLMs), with human preferences, without the need for a reward model. SPO optimizes model outputs directly over a preference dataset through a natural loss function that integrates preference loss with a regularization term across the model’s entire output distribution rather than limiting it to the preference dataset. Although SPO does not require the assumption of an existing underlying reward model, we demonstrate that, under the Bradley-Terry (BT) model assumption, it converges to a softmax of scaled rewards, with the distribution’s “softness" adjustable via the softmax exponent, an algorithm parameter. We showcase SPO’s methodology, its theoretical foundation, and its comparative advantages in simplicity and alignment precision.

## 1 Introduction

The alignment problem focuses on adjusting a generative model (e.g., Large Language Models (LLMs)) to align its outputs with human preferences and ethical standards or to tailor the model for specific tasks; and is especially important after supervised fine-tuning on datasets with mixed-quality samples.
A widely embraced approach involves refining these models based on expert (i.e., human) preferences, typically expert-provided comparisons of pairs of model-generated outputs (Christiano et al., ). Given a *preference dataset* $\mathcal{D}$ and a pre-trained model $\pi_{\textrm{ref}}$, preference alignment seeks to train a new model, $\pi_{\theta}$, whose outputs are better aligned with the preference in $\mathcal{D}$ (Radford et al., ; Ramachandran et al., ). A notable advancement in this field has been the application of Reinforcement Learning from Human Feedback (RLHF), which involves training a reward-model based of actions preferred by humans and then optimizing $\pi_{\theta}$ to maximize these learned rewards while ensuring closeness to the initial model behaviors (Ouyang et al., ). Despite the effectiveness of RLHF in addressing the alignment problem, RLHF involves a relatively complex pipeline, susceptible to propagation of reward-model’s biases over to the policy optimization.

Recently, several studies have introduced methods for the direct optimization of preferences, including Direct Preference Optimization (DPO) among others (Rafailov et al., ; Amini et al., ; Chowdhury et al., ; Xu et al., ; Yin et al., ; Xu et al., ; Tunstall et al., ). These approaches eliminate the need for a separate reward model training phase, instead adjusting the model directly using preference data, and often outperform RLHF-based approaches. These reward-model-free methods enjoy advantages over RLHF-based approaches, such as simplified pipelines, reduced computational complexity, and avoidance of the bias transfer from the reward model to policy optimization. Indeed, the rationale for incorporating an additional component, the reward model, into a supervised learning context with a supervised dataset, is debatable.

In this work, we propose a simple and effective reward-model-free alignment method, termed *Soft Preference Optimization* (SPO). SPO seeks to align the model’s *preference estimates* (detailed in Section 3) with expert preferences $\mathcal{D}$, through minimizing a loss function of the form

$$ $\textrm{AlignmentLoss}(\pi_{\theta},\pi_{\textrm{ref}},\mathcal{D})=\textrm{ PreferenceLoss}(\pi_{\theta},\mathcal{D})+\textrm{Regularizer}(\pi_{\theta}, \pi_{\textrm{ref}}),$ (1) $$

where the Regularizer may be chosen as the KL divergence. We discuss natural choices for the model’s preference estimates and the preference loss function in Sections 3 and 4.

Unlike RLHF and DPO, the development of SPO does not rely on assumptions regarding the existence of underlying rewards, such as the Bradley-Terry (BT) model (Bradley & Terry, ). Nevertheless, we demonstrate that if the BT model is applicable and given an asymptotically large preference dataset, SPO is theoretically guaranteed to converge to a softmax of the rewards, which inspires the designation “*Soft* Preference Optimization”. Unlike DPO, which tends toward a deterministic model even with an extremely large dataset if the regularization coefficient is nearly zero (Azar et al., ), SPO allows for the adjustment of the softmax’s exponent through an input parameter, thereby offering flexibility in modulating the “softness" of the output distribution.

SPO has two main distinctions from its successor reward-model-free alignment methods. The first distinction involves the choice of a preference loss that aligns model’s preference estimates with expert’s preferences, resulting in a favorable fixed point as discussed in the previous paragraph.
The other distinction of SPO with DPO and similar algorithms lies in the application of regularization. DPO restricts regularization to the preference dataset, which is counter-intuitive since the dataset already provides specific data points for the model to fit; thus, additional regularization within this limited scope is unnecessary. More critically, since the preference dataset represents a tiny subset of the potential outputs of the model, focusing regularization solely within this subset can lead to undesirable, extensive shift in the model’s distribution outside of the dataset, resulting in a non-coherent behaviours.
Acknowledging this limitation, SPO applies regularization across the entire output distribution of the model, not just within the confines of the preference dataset.

## 2 Background

Consider a finite context (or query) space $\mathcal{X}$ and a finite action (or response) space $\mathcal{Y}$. For a given query $x\in\mathcal{X}$, a behavior policy (such as a pre-trained model) is employed to generate responses $y_{1},y_{2}\in\mathcal{Y}$. These responses are subsequently evaluated by expert raters (e.g., humans) to determine which of $y_{1}$ or $y_{2}$ constitutes a more appropriate response to the query $x$. We adopt the notation $y_{1}\succ y_{2}$ to denote that $y_{1}$ is preferred over $y_{2}$ in a specific context. The true expert preferences are typically represented by a probability, $p^{*}(y_{1}\succ y_{2}|x)$, reflecting the inherent randomness due to the variable nature of the experts, who may be a group of humans with slightly differing preferences. A preference dataset, $\mathcal{D}$, is compiled by collecting expert preferences for multiple $(x;y_{1},y_{2})$ tuples. In detail, $\mathcal{D}$ comprises tuples $(x;y_{w},y_{l})$, where $y_{w}\succ y_{l}$ indicates the preferred (winner) and less preferred (loser) responses based on expert evaluations.

RLHF comprises two main phases: reward modeling and reinforcement learning (RL) fine-tuning. The initial phase, reward modeling, operates under the assumption that there exist latent rewards $r(y|x)$ that form the basis of expert preferences. This phase aims to develop a model capable of closely approximating these underlying rewards. A widely accepted method for defining these latent rewards is through the Bradley-Terry (BT) model (Bradley & Terry, ), alongside the Plackett-Luce (PL) ranking models (Luce, ; Plackett, ). The BT model posits that the distribution of expert preferences, $p^{*}$, is characterized by

$$ $p^{\mathrm{BT}}(y_{1}\succ y_{2}|x)\overset{\text{\tiny def}}{=}\sigma\big{(}r (y_{1}|x)-r(y_{2}|x)\big{)}=\frac{\exp\big{(}r(y_{1}|x)\big{)}}{\exp\big{(}r(y _{1}|x)\big{)}+\exp\big{(}r(y_{2}|x)\big{)}},$ (2) $$

where $\sigma(\cdot)$ represents the sigmoid function.
Subsequently, the reward model $r_{\phi}(y|x)$ is trained to minimize the negative log-likelihood loss, $-\mathbb{E}_{(x;y_{w},y_{l})\sim\mathcal{D}}\big{[}\sigma\big{(}r(y_{w}|x)-r(y
_{l}|x)\big{)}\big{]}$. The PL model generalizes the BT model for data involving rankings, modeling the expert distribution as

$$ $p^{\mathrm{PL}}(y_{1}\succ\cdots\succ y_{n}\,|\,x)\,\overset{\text{\tiny def}} {=}\,\prod_{k=1}^{n-1}\frac{\exp\big{(}r(y_{k}|x)\big{)}}{\sum_{i=k}^{n}\exp \big{(}r(y_{i}|x)\big{)}},$ (3) $$

for all $(x;y_{1},\ldots,y_{n})\in\mathcal{X}\times\mathcal{Y}^{n}$.

The RL fine-tuning phase aims to train a model, $\pi_{\theta}$, to maximize a loss function of the form

$$ $\mathcal{L}_{\mathrm{RLHF}}\big{(}\pi_{\theta},\pi_{\textrm{ref}},r_{\phi}\big {)}=-\mathbb{E}_{x\sim\mathcal{D},y\sim\pi_{\theta}(\cdot|x)}\big{[}r_{\phi}(y |x)\big{]}+\beta\mathcal{D}_{\textrm{KL}}\big{(}\pi_{\theta}\parallel\pi_{ \textrm{ref}}\big{)},$ (4) $$

where $\beta$ is a non-negative constant, $r_{\phi}$ is the trained reward function, and $\pi_{\mathrm{ref}}$ is a reference policy often acquired through supervised fine-tuning on high-quality data and is typically identical to the behavior policy. The $D_{KL}$ term in the loss function acts as a regularizer, ensuring the model does not significantly deviate from the distribution where the reward model is most accurate. RL fine-tuning employs reinforcement learning algorithms, like PPO (Schulman et al., ), to optimize the above loss function (Ouyang et al., ), introducing significant complexity into the RLHF pipeline. Additionally, the RLHF framework allows for the propagation of any generalization errors from the reward model to the RL fine-tuned model. The DPO framework (Rafailov et al., ) addresses these challenges by simplifying the problem into a single-phase supervised learning approach, thus avoiding the pitfalls associated with separate reward modeling and RL fine-tuning phases.

DPO circumvents the need for a reward model by directly optimizing the following loss function:

$$ $\mathcal{L}_{\mathrm{DPO}}\big{(}\pi_{\theta},\pi_{\textrm{ref}},\mathcal{D} \big{)}=-\mathbb{E}\left[\log\,\sigma\left(\beta\log\frac{\pi_{\theta}(y_{w}|x )}{\pi_{\textrm{ref}}(y_{w}|x)}-\beta\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{ \textrm{ref}}(y_{l}|x)}\right)\right].$ (5) $$

It was demonstrated in (Rafailov et al., ) that $\mathcal{L}_{\mathrm{DPO}}$ has the same minimizer as $\mathcal{L}_{\mathrm{RLHF}}$, under the conditions of the BT model, an asymptotically large dataset, and a sufficiently large model capacity (i.e., a *tabular model* that encodes the probability of $\pi_{\theta}(y|x)$ for all $x\in\mathcal{X}$ and $y\in\mathcal{Y}$ into a vector). The DPO framework was further extended in (Azar et al., ), aiming to directly maximize the *win-rate* of $\pi_{\theta}$ against $\pi_{\textrm{ref}}$.

## 3 SPO - Basic

Following (1), we consider a loss function of the form:

$$ $\mathcal{L}_{\mathrm{SPO}}(\pi_{\theta},\pi_{\textrm{ref}},\mathcal{D})\,=\, \mathcal{L}_{\mathrm{pref}}(\pi_{\theta},\mathcal{D})\,+\,\mathrm{Reg}(\pi_{ \theta},\pi_{\textrm{ref}}),$ (6) $$

where $\mathcal{L}_{\mathrm{pref}}$ and $\mathrm{Reg}$ stand for *preference loss* and *regularizer*, respectively. We proceed to further detail these components.

The regularization term, $\mathrm{Reg}(\pi_{\theta},\pi_{\textrm{ref}})$, aims to ensure that $\pi_{\theta}$ avoids producing outputs that are highly improbable under $\pi_{\textrm{ref}}$. A common and effective choice
is the KL divergence, $\mathcal{D}_{\textrm{KL}}(\pi_{\theta}\parallel\pi_{\textrm{ref}})$, although other regularization options are viable (Zhao et al., ). Importantly, $\mathrm{Reg}(\pi_{\theta},\pi_{\textrm{ref}})$ does not incorporate the preference dataset $\mathcal{D}$ as an input. This is because within $\mathcal{D}$, the model aims to fit to the target preferences, making additional regularization within $\mathcal{D}$ unnecessary. In fact, the regularization term primarily aims to regularize $\pi_{\theta}$ outside $\mathcal{D}$. This approach diverges from the DPO and several other existing loss functions (detailed in Section 8), which only consider the divergence of $\pi_{\theta}$ from $\pi_{\textrm{ref}}$ within the preference dataset.

We now turn our attention to the preference loss. Given a query $x$, let $\pi_{\theta}(y|x)$ denote the probability that model $\pi_{\theta}$ generates output $y$. When presented with a query $x$ and two responses, $y_{1}$ and $y_{2}$, we define the probability that $\pi_{\theta}$ prefers $y_{1}$ over $y_{2}$ as

$$ $\begin{split}\mathcal{P}_{\pi_{\theta}}(y_{1}\succ y_{2}\mid x)&\overset{\text {\tiny def}}{=}\mathrm{P}\big{(}\textrm{output of }\pi_{\theta}(\cdot|x) \textrm{ is }y_{1}\,\,\big{|}\,\,\textrm{output of }\pi_{\theta}(\cdot|x) \textrm{ is in }\{y_{1},y_{2}\}\big{)}\\ &=\frac{\pi_{\theta}(y_{1}|x)}{\pi_{\theta}(y_{1}|x)+\pi_{\theta}(y_{2}|x)}, \end{split}$ (7) $$

where the last equality follows from the definition of conditional probability.
We can then employ log-likelihood loss to measure the alignment of preference-probabilities’ with the preference-dataset labels,

$$ $-\mathbb{E}_{(x;y_{w},y_{l})\sim\mathcal{D}}\big{[}\log\,\mathcal{P}_{\pi_{ \theta}}(y_{w}\succ y_{l}\mid x)\big{]}.$ (8) $$

We consider a preference loss $\mathcal{L}_{\mathrm{pref}}^{\alpha}(\pi_{\theta},\mathcal{D})$ that extends the above cross entropy loss by employing arbitrary exponents for $\pi_{\theta}$.
Specifically, we let for any $\alpha>0$,

$$ $\mathcal{L}_{\mathrm{pref}}^{\alpha}(\pi_{\theta},\mathcal{D})\overset{\text{ \tiny def}}{=}-\frac{1}{\alpha}\mathbb{E}_{(x;y_{w},y_{l})\sim\mathcal{D}} \left[\log\,\frac{\pi_{\theta}(y_{w}\mid x)^{\alpha}}{\pi_{\theta}(y_{w}\mid x )^{\alpha}+\pi_{\theta}(y_{l}\mid x)^{\alpha}}\right],$ (9) $$

and for $\alpha=0$,

$$ $\mathcal{L}_{\mathrm{pref}}^{0}(\pi_{\theta},\mathcal{D})\overset{\text{\tiny def }}{=}-\frac{1}{2}\mathbb{E}_{(x;y_{w},y_{l})\sim\mathcal{D}}\left[\log\,\frac{ \pi_{\theta}(y_{w}\mid x)}{\pi_{\theta}(y_{l}\mid x)}\right].$ (10) $$

The $\mathcal{L}_{\mathrm{pref}}^{\alpha}$ takes the specific form (10) because the gradient of (9) approaches the gradient of (10) when $\alpha\to 0$, as can be easily verified from the following closed form expression for any $\alpha\geq 0$,

$$ $-\nabla_{\theta}\mathcal{L}_{\mathrm{pref}}^{\alpha}(\pi_{\theta},\mathcal{D}) =\mathbb{E}_{(x;y_{w},y_{l})\sim\mathcal{D}}\left[\frac{\pi_{\theta}(y_{l}|x)^ {\alpha}}{\pi_{\theta}(y_{w}|x)^{\alpha}+\pi_{\theta}(y_{l}|x)^{\alpha}}\big{( }\nabla_{\theta}\log\pi_{\theta}(y_{w}|x)-\nabla_{\theta}\log\pi_{\theta}(y_{l }|x)\big{)}\right].$ $$

Here, $\pi_{\theta}(y_{l}|x)^{\alpha}/\big{(}\pi_{\theta}(y_{w}|x)^{\alpha}+\pi_{
\theta}(y_{l}|x)^{\alpha}\big{)}$ serves as a measure of the model’s error in preferring $y_{w}$ over $y_{l}$. Consequently, the magnitude of this preference error proportionally scales the adjustment $\nabla_{\theta}\log\,\pi_{\theta}(y_{w}|x)-\nabla_{\theta}\log\,\pi_{\theta}(y
_{l}|x)$, leading to larger updates when the error is large.

The loss function $\mathcal{L}_{\mathrm{pref}}^{\alpha}(\pi_{\theta},\mathcal{D})$ contains the cross-entropy loss in (8) as a special case when $\alpha=1$.
The $\alpha$ parameter allows for tailoring the model to exhibit different entropies; models minimized under $\mathcal{L}_{\mathrm{pref}}^{\alpha}$ will display higher entropy for larger $\alpha$ values, gradually moving towards a deterministic model akin to DPO as $\alpha$ approaches zero; as established in the next theorem.

Although the SPO framework does not rely on existence of underlying reward functions, and in particular the BT assumption, it is insightful to study the preference loss $\mathcal{L}_{\mathrm{pref}}^{\alpha}$ under the conditions where the BT model assumption is valid. Intuitively, for a *BT expert model*, defined as $\pi(y|x)=\exp(r(y|x))/Z(x)$ with $Z(x)$ being the partition function, the preference probability in (7) would be identical to the BT preference formula (2).
In the next theorem, we further study the landscape of $\mathcal{L}_{\mathrm{pref}}^{\alpha}$ under the BT model assumption.
To eliminate local minima and saddle points that arise from nonlinear model spaces such as neural networks, in the theorems we consider a *tabular model* that encodes the probability of $\pi_{\theta}(y|x)$ for all $x\in\mathcal{X}$ and $y\in\mathcal{Y}$ into a large vector.

###### Theorem 1 .

Suppose that the BT model holds with rewards $r(\cdot|x)$, and fix any probability distribution $\mathcal{D}$ over $\mathcal{X}\times\mathcal{Y}\times\mathcal{Y}$ that has full support(^1^11Full support in this context means that the probability distribution assigns a non-zero sampling probability to all $(x;y_{w},y_{l})\in\mathcal{X}\times\mathcal{Y}\times\mathcal{Y}$.) and is consistent with the BT assumption.(^2^22Consistency with the BT holds if the relative probability of outcomes is determined by a logistic function of the reward differences. More specifically, $\mathcal{D}(x;y_{1},y_{2})/\mathcal{D}(x;y_{2},y_{1})=p^{\mathrm{BT}}(y_{1} \succ y_{2}|x)/p^{\mathrm{BT}}(y_{2}\succ y_{1}|x)=\exp\big{(}r(y_{1}\mid x)-r (y_{2}\mid x)\big{)}$, for all $(x;y_{1},y_{2})\in\mathcal{X}\times\mathcal{Y}\times\mathcal{Y}$, where $p^{\mathrm{BT}}$ is defined in (2) and $r(\cdot|\cdot)$ is the reward function in the BT model.) Then, for any $\alpha\geq 0$, in the tabular model, $\mathcal{L}_{\mathrm{pref}}^{\alpha}$ has a unique minimizer $\operatorname{Softmax}(r(\cdot|x)/\alpha)$ (reducing to $\operatorname{argmax}r(\cdot|x)$ for $\alpha=0$).
Furthermore, this minimizer is globally absorbing, and the landscape of $\mathcal{L}_{\mathrm{pref}}^{\alpha}$ contains no other first-order stationary point (i.e., no other local minima, local maxima, or saddle points).

The proof is provided in Appendix A.
According to Theorem 1, minimizer of $\mathcal{L}_{\mathrm{pref}}^{\alpha}$ is the softmax of BT rewards divided by $\alpha$, where $\alpha$ controls the entropy of the final model. Specifically, in the the asymptotically large dataset regime, when $\alpha=1$, the preference loss reaches its minimum at the hypothetical *BT expert model* that generates the preference dataset’s labels, defined as $\operatorname{Softmax}(r(\cdot|x))$.

###### Theorem 1 .

## 4 The General SPO Algorithm

We further expand the preference loss of SPO by considering a weighting over different samples, where the weights can depend on $\pi_{\theta}$. This weighting only affects (improves) the optimization process without changing the fixed point, as we show in this section.

We call a function $\mu:\mathcal{Y}\times\mathcal{Y}\times\mathcal{X}\to\mathbb{R}^{+}$ *symmetric positive* if $\mu(y_{1},y_{2}\mid x)=\mu(y_{2},y_{1}\mid x)>0$, for all $x\in\mathcal{X}$ and all $y_{1},y_{2}\in\mathcal{Y}$. Given a symmetric positive function $\mu$ and an $\alpha\geq 0$, we define *weighted* preference loss as

$$ $\mathcal{L}_{\mathrm{pref}}^{\alpha,\mu}(\pi_{\theta},\mathcal{D})\overset{ \text{\tiny def}}{=}-\frac{1}{\alpha}\mathbb{E}_{(x;y_{w},y_{l})\sim\mathcal{D }}\left[\mu(y_{w},y_{l}\mid x)\,\log\,\frac{\pi_{\theta}(y_{w}\mid x)^{\alpha} }{\pi_{\theta}(y_{w}\mid x)^{\alpha}+\pi_{\theta}(y_{l}\mid x)^{\alpha}}\right]$ (11) $$

if $\alpha>0$, and for $\alpha=0$ we let

$$ $\mathcal{L}_{\mathrm{pref}}^{0,\mu}(\pi_{\theta},\mathcal{D})\overset{\text{ \tiny def}}{=}-\frac{1}{2}\mathbb{E}_{(x;y_{w},y_{l})\sim\mathcal{D}}\left[\mu (y_{w},y_{l}\mid x)\,\log\,\frac{\pi_{\theta}(y_{w}\mid x)}{\pi_{\theta}(y_{l} \mid x)}\right].$ (12) $$

The weight-function $\mu$ controls the impact of individual samples within the loss calculation.
The utility of $\mu$ emerges from the observation that not all sample pairs in the preference dataset hold equivalent significance. For instance, diminishing the weights of dataset samples $(x;y_{w},y_{l})$ where both responses $y_{w}$ and $y_{l}$ are of low quality (e.g., low probability) can be particularly advantageous. This can be achieved for example by setting

$$ $\mu(y_{1},y_{2}\mid x)=2\,\sigma\Big{(}\big{(}\pi_{\theta}(y_{1}|x)+\pi_{ \theta}(y_{2}|x)\big{)}^{\gamma}-\hat{\mathbb{E}}_{(y_{1}^{\prime},y_{2}^{ \prime}|x^{\prime})\sim\mathcal{D}}\left[\big{(}\pi_{\theta}(y_{1}^{\prime}|x^ {\prime})+\pi_{\theta}(y_{2}^{\prime}|x^{\prime})\big{)}^{\gamma}\right]\Big{)},$ (13) $$

where $\sigma$ is the sigmoid function, $\gamma\geq 0$ is a hyperparameter, and $\hat{\mathbb{E}}$ can be obtained by averaging over the current batch. The $\mu$ function boils down to uniform weights for $\gamma=0$.

While $\mu$ may depend on $\pi_{\theta}$, it is important to note that gradient propagation through $\mu$ is not permitted. Specifically,
the gradient $\nabla_{\theta}\mathcal{L}_{\mathrm{pref}}^{\alpha,\mu}(\pi_{\theta},\mathcal{
D})$ is given by

$$ $-\mathbb{E}_{(x;y_{w},y_{l})\sim\mathcal{D}}\left[\mu(y_{w},y_{l}|x)\,\frac{ \pi_{\theta}(y_{l}|x)^{\alpha}}{\pi_{\theta}(y_{w}|x)^{\alpha}+\pi_{\theta}(y_ {l}|x)^{\alpha}}\,\big{(}\nabla_{\theta}\log\,\pi_{\theta}(y_{w}|x)-\nabla_{ \theta}\log\,\pi_{\theta}(y_{l}|x)\big{)}\right].$ (14) $$

Interestingly, the weight function, $\mu$, mainly influences the optimization process, not the ultimate fixed point, in the tabular setting and under asymptotically large preference dataset, as we show in the next theorem. The proof is given in Appendix A.

###### Theorem 2 .

Suppose that the conditions of Theorem 1 hold. Then for any $\alpha\geq 0$ and any symmetric positive function $\mu$,
the softmax of the BT rewards divided by $\alpha$, $\operatorname{Softmax}(r(\cdot|x)/\alpha)$ (reducing to $\operatorname{argmax}r(\cdot|x)$ for $\alpha=0$), is the unique globally absorbing fixed point of the differential equation $\dot{\boldsymbol{\pi}}=\prod\big{(}-\nabla_{\theta}\mathcal{L}_{\mathrm{pref}}
^{\alpha,\mu}(\pi_{\theta},\mathcal{D})\big{)}$, where $\prod(\cdot)$ stands for projection onto the probability simplex, and the gradient is given in (14).

We now proceed to discuss the computation of $\mathcal{D}_{\textrm{KL}}$ regularizer in (6). In order to estimate $\mathcal{D}_{\textrm{KL}}(\pi_{\theta}\lVert\pi_{\textrm{ref}})=\mathbb{E}_{x}
\mathbb{E}_{y\sim\pi_{\theta}(\cdot|x)}\big{[}\log\big{(}\pi_{\theta}(y|x)/\pi
_{\textrm{ref}}(y|x)\big{)}\big{]}$, we generate online samples from the current model $\pi_{\theta}$. This is however costly for sequential models, where sequence generation necessitates sequential calls to the model. To mitigate this problem, we generate a batch of samples from $\pi_{\theta}$ intermittently, for example once every $T$ steps, and keep using samples from this batch for approximating $\mathcal{D}_{\textrm{KL}}$, until the next batch of samples is generated.

Given a batch of samples $(x,y)$ with $y\sim\pi_{\theta}(\cdot|x)$,
in order to we obtain a reduced variance approximation of $\mathcal{D}_{\textrm{KL}}$, we employ the following token-wise $\mathcal{D}_{\textrm{KL}}$ formula:

$$ $\begin{split}\widehat{\mathcal{D}_{\textrm{KL}}}(\pi_{\theta}\parallel\pi_{ \textrm{ref}})\overset{\text{\tiny def}}{=}&\,\,\mathbb{E}_{(x;y)\in\textrm{ batch}}\Bigg{[}\sum_{\tau=1}^{\lvert y\rvert}\,\mathcal{D}_{\textrm{KL}}\Big{( }\pi_{\theta}(Y_{\tau}\mid x,y_{:\tau})\parallel\pi_{\textrm{ref}}(Y_{\tau} \mid x,y_{1:\tau})\Big{)}\Bigg{]}\\ =&\,\,\mathbb{E}_{(x;y)\in\textrm{batch}}\Bigg{[}\sum_{\tau=1}^{\lvert y\rvert }\,\sum_{s\in\mathcal{S}}\pi_{\theta}(Y_{\tau}=s\mid x,y_{1:\tau})\log\frac{ \pi_{\theta}(Y_{\tau}=s\mid x,y_{1:\tau})}{\pi_{\textrm{ref}}(Y_{\tau}=s\mid x ,y_{1:\tau})}\Bigg{]},\end{split}$ (15) $$

where $\mathcal{S}$ is the set of all possible tokens. Note that $\pi(Y_{\tau}=s\mid x,y_{1:\tau})$ is readily available from the softmax of the logits, in the network’s output. Therefore, the sum in (15) can be computed with negligible computational overhead (excluding the initial forward path). Note that $\widehat{\mathcal{D}_{\textrm{KL}}}$ is a biased estimate of the sequence-$\mathcal{D}_{\textrm{KL}}$.
However, we empirically found that the benefit of reduced variance brought by the token-wise approximation out-weights the potential negative impact of the resulting bias.
Moreover, similar to sequence-$\mathcal{D}_{\textrm{KL}}$, the token-wise $\mathcal{D}_{\textrm{KL}}$ is a proximity measure for $\pi_{\theta}$ and $\pi_{\textrm{ref}}$, and is therefore a conceptually sound choice of regularizer.
Algorithm 1 summarized the SPO algorithm.

Figure: Algorithm 1 SPO

###### Theorem 2 .

## 5 SPO for Other Data-Types: Best-of- n 𝑛 n italic_n Preference and Ranked Preference

In this section, we generalize the SPO algorithm for other types of preference data: best-of-$n$ preference data and ranked-data. We extend the definition of a symmetric function to $n$-responses by calling a function $\mu:\mathcal{Y}^{n}\times\mathcal{X}\to\mathbb{R}^{+}$ *symmetric positive* if $\mu(y_{\tau(1)},\ldots,y_{\tau(n)}\mid x)=\mu(y_{1},\ldots,y_{n}\mid x)>0$, for all $x\in\mathcal{X}$ all $y_{1},\ldots,y_{n}\in\mathcal{Y}$, and all permutations $\tau$ of $(1,\ldots,n)$.

Best-of-$n$
preference data: Given an $n\geq 2$, a sample $(x;y_{1},\ldots,y_{n};i^{*})$ of a best-of-$n$ preference dataset consists of a query $x$ along with $n$ responses $y_{1},\ldots,y_{n}$, one of which (i.e., $y_{i^{*}}$) is labeled by the expert as the best response. Given a symmetric positive function $\mu$ and an $\alpha>0$, we propose the following preference loss for a best-of-$n$ preference dataset $\mathcal{D}$:

$$ $\mathcal{L}_{\mathrm{pref}\textrm{-}n}^{\alpha,\mu}(\pi_{\theta},\mathcal{D}) \overset{\text{\tiny def}}{=}-\frac{1}{\alpha}\mathbb{E}_{(x;y_{1},\ldots,y_{n };i^{*})\sim\mathcal{D}}\left[\mu(y_{1},\ldots,y_{n}\mid x)\,\log\,\frac{\pi_{ \theta}(y_{i^{*}}\mid x)^{\alpha}}{\sum_{i=1}^{n}\pi_{\theta}(y_{i}\mid x)^{ \alpha}}\right].$ (16) $$

As before, we stop the gradient from propagating through $\mu$, even though $\mu$ may depend on $\pi_{\theta}$.
Similar to the case of pairwise preferences, we show in the following theorem that the loss in (16)
is minimized at the softmax of rewards, if we assume existence of an underlying reward function.
In particular, given a reward function $r(\cdot|x):\mathcal{X}\to\mathcal{Y}$ and a distribution $\mathcal{D}$ over $\mathcal{X}\times\mathcal{Y}^{n}\times\{1,\ldots,n\}$, we say that $\mathcal{D}$ is *consistent with $n$-ary BT model* if for any $(x;y_{1},\ldots,y_{n})\in\mathcal{X}\times\mathcal{Y}^{n}$ and any $i,j\in\{1,\ldots,n\}$, $\mathcal{D}(x;y_{1},\ldots,y_{n};i)/\mathcal{D}(x;y_{1},\ldots,y_{n};j)=\exp
\big{(}r(y_{i}\mid x)-r(y_{j}\mid x)\big{)}$.
Note that this definition boils down to the definition of consistency with BT model for $n=2$ in Section 3.
Proof of the following theorem is given in Appendix B.

###### Theorem 3 .

Consider a reward function $r(\cdot\mid x)$ and a probability distribution $\mathcal{D}$ with full support over $\mathcal{X}\times\mathcal{Y}^{n}\times\{1,\ldots,n\}$ that is consistent with the $n$-ary BT model.
Then, for any $\alpha>0$ and any symmetric positive function $\mu$, in the tabular model, $\operatorname{Softmax}(r(\cdot|x)/\alpha)$
is the unique globally absorbing fixed point of the differential equation $\dot{\boldsymbol{\pi}}=\prod\big{(}-\nabla_{\theta}\mathcal{L}_{\mathrm{pref}
\textrm{-}n}^{\alpha,\mu}(\pi_{\theta},\mathcal{D})\big{)}$, where $\prod(\cdot)$ stands for projection onto the probability simplex.

Ranked Preference Data:
A ranked preference dataset consists of samples of the form $(x;y_{1},\ldots,y_{n};\tau)$, where $x$ is a query, $y_{1},\ldots,y_{n}$ are $n$ responses, and $\tau$ is a permutation representing the relative preference $y_{\tau(1)}\succ\cdots\succ y_{\tau(n)}$ of the expert over these responses. Given an $\alpha>0$ and a sequence of symmetric positive function $\mu_{k}:\mathcal{X}\times\mathcal{Y}^{k}\to\mathbb{R}$ for $k=2,\ldots,n$, we propose the following preference loss for a ranked preference dataset $\mathcal{D}$:

$$ $\mathcal{L}_{\mathrm{rank}}^{\alpha,[\mu]}(\pi_{\theta},\mathcal{D})\overset{ \text{\tiny def}}{=}-\frac{1}{\alpha}\mathbb{E}_{(x;y_{1},\ldots,y_{n};\tau) \sim\mathcal{D}}\left[\sum_{k=1}^{n-1}\mu_{k}(y_{\tau(k)},\ldots,y_{\tau(n)}|x )\,\log\,\frac{\pi_{\theta}(y_{\tau(k)}\,|\,x)^{\alpha}}{\sum_{j=k}^{n}\pi_{ \theta}(y_{\tau(j)}\,|\,x)^{\alpha}}\right].$ (17) $$

We can control the importance weight of responses in different ranks through appropriate adjustment of weight functions $\mu^{1},\ldots,\mu^{n-1}$.
For example, by setting $\mu_{k}=0$ for $k=2,\ldots,n-1$, $\mathcal{L}_{\mathrm{rank}}^{\alpha,[\mu]}$ boils down to $\mathcal{L}_{\mathrm{pref}\textrm{-}n}^{\alpha,\mu^{1}}$.
Here again, the gradient is not allowed to propagate through $\mu^{1},\ldots,\mu^{n-1}$, even though these functions may depend on $\pi_{\theta}$.
The following theorem shows that, assuming existence of underlying rewards under the PL model (3), the softmax of these rewards is the unique minimizer of
$\mathcal{L}_{\mathrm{rank}}^{\alpha,\mu}$. The proof relies on Theorem 3, and is given in Appendix C.

###### Theorem 4 .

Suppose that the PL model holds with rewards $r(\cdot|x)$, and a probability distribution $\mathcal{D}$ with full support over $\mathcal{X}\times\mathcal{Y}^{n}\times\{\mathrm{Identity\,permutation}\}$ that is consistent with the PL model.(^3^33Consistency with the PL model holds if $\mathcal{D}(x;y_{1},\ldots,y_{n};\tau)/\mathcal{D}(x;y_{1},\ldots,y_{n};\tau^{ \prime})=p^{\mathrm{PL}}(y_{\tau(1)}\succ\cdots\succ y_{\tau(n)}|x)/p^{\mathrm {PL}}(y_{\tau^{\prime}(1)}\succ\cdots\succ y_{\tau^{\prime}(n)}|x)$, for all $(x;y_{1},\ldots,y_{n})\in\mathcal{X}\times\mathcal{Y}^{n}$ and all permutations $\tau$ and $\tau^{\prime}$, where $p^{\mathrm{PL}}$ is defined in (3).)
Then, for any $\alpha>0$ and any sequence $[\mu]=\mu^{1},\ldots,\mu^{n-1}$ of symmetric positive functions, in the tabular model, $\operatorname{Softmax}(r(\cdot|x)/\alpha)$
is the unique globally absorbing fixed point of the differential equation $\dot{\boldsymbol{\pi}}=\prod\big{(}-\nabla_{\theta}\mathcal{L}_{\mathrm{rank}}
^{\alpha,[\mu]}(\pi_{\theta},\mathcal{D})\big{)}$, where $\prod(\cdot)$ stands for projection onto the probability simplex.

###### Theorem 3 .

###### Theorem 4 .

## 6 Comparative Analysis: SPO Versus DPO

This section contrasts the SPO method with the DPO algorithm conceptually. A detailed empirical comparison follows in Section 7.

A key difference between SPO and DPO lies in how regularization ($\mathcal{D}_{\textrm{KL}}$) is applied. In RLHF and SPO, $\mathcal{D}_{\textrm{KL}}$ is used to prevent $\pi_{\theta}$ from straying too far from $\pi_{\textrm{ref}}$ in unexplored regions, reducing the risk of distribution shifts. In contrast, the DPO loss function (5) applies regularization only to preference dataset samples, which is suboptimal because 1) it fails to prevent distribution shifts in unexplored regions, and 2) regularizing within the dataset can hinder alignment with the preferences.

Moreover, SPO has an advantage over DPO and RLHF in avoiding determinism. In cases where the preference dataset is comparable to pre-training data size, regularization ($\mathcal{D}_{\textrm{KL}}$) becomes unnecessary, and RLHF and DPO loss functions tend to produce deterministic models; that is they tend to return a single high-quality response per query. This reduced diversity makes DPO prone to mode collapse (Azar et al., ). SPO, however, controls entropy via the $\alpha$ parameter in (9), even without the use of regularizer (see Theorem 1), preserving response diversity. This makes SPO more adaptable for continual learning and future alignments.

It is noteworthy that unlike RLHF and DPO, the SPO framework does not assume the existence of an underlying reward model or rely on assumptions like the BT model. Instead, SPO’s preference loss directly aligns $\pi_{\theta}$ with the preferences in the dataset, making it potentially more adaptable to broader alignment contexts. Additionally, SPO is not limited to using $\mathcal{D}_{\textrm{KL}}$ for regularization, unlike DPO and IPO, which depend on $\mathcal{D}_{\textrm{KL}}$ for derivations of their loss functions.

We also note that the DPO loss cannot be separated into components like (6), where the preference loss is independent of $\pi_{\textrm{ref}}$ and paired with a regularizer like $\mathcal{D}_{\textrm{KL}}$. As a short proof, consider a case where $\pi_{\theta}(y_{w}|x)=\pi_{\theta}(y_{l}|x)$ for a sample $(x;y_{w},y_{l})\in\mathcal{D}$. Here, the alignment loss in (6) remains symmetrical with respect to $\pi_{\textrm{ref}}(y_{w}|x)$ and $\pi_{\textrm{ref}}(y_{l}|x)$; swapping their values wouldn’t affect either the preference loss or $\mathcal{D}_{\textrm{KL}}$. This symmetry doesn’t hold in the DPO framework, as seen in the DPO loss in (5). Therefore, the DPO loss cannot be represented in the separable form of (6).

## 7 Experiments

This section presents empirical evaluations of SPO.

### 7.1 Alignment to Pairwise Preference Data

Experiment setting:
To evaluate the performance of SPO, we trained a Llama2-7B model (Touvron et al., ) on a pairwise preference dataset for question-answering available in AlpacaFarm (Dubois et al., ), and computed the win-rates against the Llama2-7B SFT model on AlpacaEval 2 (Li et al., ), using GPT4-Turbo API. More specifically, we used the following pipeline. We downloaded the pretrained Llama2-7B model and performed supervised fine-tuning on the AlpacaFarm SFT dataset available at Dubois et al. (), to obtain the SFT model.
Initializing the weights on the SFT model, we then performed alignment to the preference dataset from AlpacaFarm that contains pairs of question-answering samples and preference labels provided by GPT-4. We compared the performance of SPO with several reward-model-free alignment algorithms, namely DPO (Rafailov et al., ), IPO (Azar et al., ), KTO (Ethayarajh et al., ), CPO (Xu et al., ), R-DPO (Park et al., ), and SimPO (Meng et al., ).
For SPO, the experiment includes both the basic and weighted versions of SPO, with the weight function $\mu$ given in (13). We trained each algorithm for a few epochs, used GPT4-Turbo to compute its win-rate against the SFT model at the end of each epoch, and reported the maximum win-rate for each algorithm. Additional details about setting of this experiment is provided in Appendix D.1.

Results:
Table 1 presents the win-rates and length-controlled (LC) win-rates (Dubois et al., ) of different algorithms against the SFT model. The SPO algorithm outperforms the other tested algorithms in both conventional win-rate and LC win-rate. Furthermore, the weighted version of SPO performs better than the basic version, demonstrating the advantage of incorporating the weight function.

**Table 1: Alignment of the Llama2-7B model on AlpacaFarm dataset.**
| Alignment method | Win-rate(%) | LC Win-rate(%) |
| --- | --- | --- |
| SFT | 50.00 | 50.00 |
| R-DPO | 52.50 | 41.49 |
| CPO | 54.10 | 39.38 |
| SimPO | 58.48 | 49.73 |
| KTO | 58.50 | 51.94 |
| IPO | 58.59 | 49.60 |
| DPO | 59.16 | 51.26 |
| SPO-basic (unweighted) | 60.83 | 53.17 |
| SPO | 61.63 | 56.25 |

### 7.2 Alignment to Ranking and Best-of- n 𝑛 n italic_n Preference Data

Experiment setting:  Dataset: Alignment research has largely focused on pairwise preferences, and publicly available datasets for other preference types are rather scarce. To address this, we generated an $n$-ary ranked preference dataset ($n=4$) using GPT4o API as the labeler. Building on TinyStories (Eldan & Li, ) –a synthetic collection of short stories aimed at children aged 3 to 4– we created a preference dataset to align the stories to an older audience. The dataset contains around 5,000 samples, each with four stories generated by the TinyStories pre-trained 110M model, ranked by GPT4o-2024-08-06 based on coherence and engagement for older students. The best-of-$n$ dataset was then derived by removing the rank labels for the 2nd, 3rd, and 4th responses. Further details are in Appendix E.

Training: Using the implementation from (Karpathy, ) and the supervised fine-tuned model from (Karpathy, ), we aligned a 110M parameter model with ranking and best-of-$n$ versions of SPO. We compared this with three baselines: the ranking version of DPO (AppendixA3 of (Rafailov et al., )), S-DPO (Chen et al., ), and Best-response-SFT. Best-response-SFT is derived by all performing supervised fine-tuning on all top-rank responses. See Appendix D.2 for further details.

Results:
Tables 3 and 3 present peak win rates against the Best-response-SFT model for aligning the 110M TinyStories SFT model to ranking and best-of-$n$ datasets using different algorithms. The SPO algorithm outperforms all baselines for both types of alignment.

**Table 2: Alignment of TinyStories SFT model to the ranking dataset.**
| Alignment method | Win-rate (%) |
| --- | --- |
| Best-response-SFT | 50.0 |
| DPO (ranking version) | 67.0 |
| S-DPO (ranking) | 55.1 |
| SPO (ranking) | 68.5 |

### 7.3 Ablation Study

Global regularization vs in-dataset regularization:
To evaluate the significance of global regularization, we trained a Llama2-7B SFT model the SPO-basic algorithm while replacing the global $\mathcal{D}_{\textrm{KL}}$ regularizer of (15) with in-dataset regularizers.
In particular, we tested three different in-dataset regularizers, including the popular $-\log\pi(y_{w}|x)$ regularizer that is used in several alignment algorithm like CPO, SLiC-HF (Zhao et al., ), and RRHF (Yuan et al., ).
Other experiment settings are similar to the setting discussed in Section 7.1.
Table 4 provides the list of these in-dataset regularizers, as well as the peak win-rates of SPO-basic using these regularizers against the Llama2-7B SFT model. As it can be seen from the table, the global $\mathcal{D}_{\textrm{KL}}$ regularizer of (15) achieved better performance compared to in-dataset regularizers.

Weight function $\mu$:
As observed in the experiment of Section 7.1 on the alignment to AlpacaFarm preferences dataset, the weighted version of SPO achieved win-rate $61.63\%$ compared to the win-rate of $60.83\%$ for SPO-basic.
In the TinyStories experiment of Section 7.1, use of weight function $\mu$ improved win-rate from $xx\%$ to $xx\%$. In the experiment of Section 7.2 on the alignment to best-of-$n$ preferences dataset, the weighted version of SPO improved the win-rate to $70.5\%$, up from $67.8\%$ for SPO-basic. In the ranking experiment, use of non-uniform $\mu$ did not improve the win-rate.
In all of these experiments, the $\mu$ function in the weighted SPO algorithm is of the form (13) with parameter $\gamma=0.01$. We performed no sweeping on the parameter $\gamma=0.01$.

**Table 4: Ablation of SPO regularizer (model: Llama2-7B, dataset: AlpacaFarm)**
| Regularizer type | Regularizer formula | ​​​$\begin{array}[]{c}\textrm{{Win-rate}}\\<br>\textrm{\small{vs. SFT}}\end{array}$​​​ |
| --- | --- | --- |
| in-dataset (log probability) | $-\mathbb{E}_{(x;y_{w},y_{l})\in\mathcal{D}}[\log\pi_{\theta}(y_{w}|x)]$ | 54.1 |
| in-dataset (tokenwise) | $\mathbb{E}_{(x;y)\in D}\left[\sum_{\tau=0}^{\lvert y\rvert-1}\mathbb{E}_{Y\sim<br>\pi_{\theta}(\cdot|x,y_{1:\tau})}\log\frac{\pi_{\theta}(Y|x,y_{1:\tau})}{\pi_{<br>\textrm{ref}}(Y|x,y_{1:\tau})}\right]$ | 59.7 |
| in-dataset (importance sampling) | $\mathbb{E}_{(x;y)\in D}\left[\frac{\pi_{\theta}(y|x)}{\pi_{\textrm{ref}}(y|x)}<br>\log\frac{\pi_{\theta}(y|x)}{\pi_{\textrm{ref}}(y|x)}\right]$ | 59.9 |
| global | Eq. (15) | 60.8 |

## 8 Related Works

RLHF aims to align AI systems with human preferences, relying on human judgments rather than manual rewards or demonstrations. This method has been successfully applied in fine-tuning large language models (LLMs) (Achiam et al., ; Touvron et al., ; Ouyang et al., ), but faces challenges including data quality issues, reward misgeneralization, and policy optimization complexities. Research to enhance RLHF includes methods such as rejection sampling for response generation (Dong et al., ; Touvron et al., ), where the highest-reward response from a fixed number is selected for fine-tuning. Zhang et al. () simplified instruction alignment with language models into a goal-oriented reinforcement learning task, utilizing a two-phase approach of high-temperature online sampling and supervised learning with relabeled data during offline training. A two-loop learning algorithm, Grow and Improve, has also been proposed for iterative model alignment and training on a fixed dataset (Gulcehre et al., ). The Grow loop leverages the existing model to create and sample a dataset while the Improve loop iteratively trains the model on a fixed dataset.

Given the challenges of RLHF, reward-model-free alignment methods emerged fairly recently and have gained a lot of popularity.
Reward-model-free approach to alignment was popularized specifically after introduction of DPO in (Rafailov et al., ), which is breifly outlined in Section 2.
Recently, several works have been proposed methods to improve DPO. Azar et al. () considered an
objective called $\Psi$PO for learning from human preferences that is expressed in terms of
pairwise preferences, with no need for assumption of the BT model. The authors focused on a specific instance, IPO, of $\Psi$PO by setting $\Psi$ as the identity, aiming to mitigate the overfitting and tendency-towards-deterministic-policies issues observed in DPO. Munos et al. () formulated the alignment problem as finding the Nash equilibrium (NE) of a maximin game with two policies $\pi$ and $\pi^{\prime}$ as two players where each policy receives pay off of probability of winning over the other policy. The authors showed that the NE point can be approximated by running a mirror-descent-like algorithm. Rosset et al. () and Swamy et al. () proposed other approaches to approximate the NE point based on no-regret algorithms (Freund & Schapire, ).
Chowdhury et al. () proposed a loss function which is an unbiased estimate of the original DPO loss, and aims to alleviate sensitivity to flipped labels due to labeling noise. Amini et al. () added an offset term within the sigmoid function in the DPO loss. In this manner, the model puts more weight on the winning response.
To control the length of the output, R-DPO (Park et al., ) modified the DPO loss by adding a regularization term that penalizes lengthy responses. To achieve the same goal, SimPO (Meng et al., ) replaced the log-likelihood ratio between the current policy and the baseline model with the average log probability of the sequence under the current policy.
In (Rafailov et al., ), a token-level formulation of DPO has been proposed which enables a likelihood search over a DPO model by classical search-based algorithms, such as MCTS. Inspired by cringe loss previously proposed for binary feedback, Xu et al. () adapted cringe loss for the pairwise preference context.
Recently, in (Ethayarajh et al., ), KTO loss has been proposed for alignment from non-paired preference datasets.

In practice, the performance of an alignment technique highly depends on the quality of the human preference dataset. Noisy preference pairs could potentially limit the language models from capturing human intention.
In (Liu et al., ), DPO was used in conjunction with an improved preference dataset via a rejection sampling technique, arguing that DPO suffers from a mismatch between the sampling distribution and the policy corresponding to true expert preferences. (Tunstall et al., ) formed a dataset of conservative pairs by collecting AI feedback through an ensemble of chat model completions, followed by GPT-4 scoring. Then, they employed DPO for alignment to this improved dataset.
The work in (Yin et al., ) leveraged semantic correlations of prompts in the dataset to form more conservative response pairs. For a given prompt $(x;y_{w},y_{l})$, a prompt $x^{\prime}$ with a similar semantic to a tuple $(x^{\prime};y^{\prime}_{w},y^{\prime}_{l})$ is used to form more conservative pairs. In particular, they propose a weighted version of the DPO loss where for a given labeled data $(x;y_{w},y_{l})$, $y_{w}$ is approved while $y_{l}$ and any $y^{\prime}_{l}$ (from a similar prompt $x^{\prime}$) are penalized.

Zhao et al. () proposed a separable alignment technique, called SLiC, where, similar to SPO, the alignment loss is the sum of two terms: a calibration loss that contrasts a winner and loser responses encouraging the model $\pi_{\theta}$ to assign more probability to the winner, and a regularizer term. SLiC was further developed in (Zhao et al., ) to be used in alignment to preference data, where they proposed the SLiC-HF algorithm. SLiC-HF involves a rectified contrastive loss as its calibration loss and a log-likelihood term as the regularization. Other than a different choices for preference loss and regularization, SLiC-HF diverges from the SPO framework in that the regularization in SLiC-HF is limited to the preference or pre-training datasets, not using online samples from $\pi_{\theta}$ as in the $\mathcal{D}_{\textrm{KL}}$ regularize.

Concurrent to our work, CPO (Xu et al., ) motivated the preference loss in (9) as a heuristic approximation of DPO to reduce its complexity by removing $\pi_{\textrm{ref}}$ from the DPO loss. This results in a contrastive loss similar to SPO-basic (9). In contrast, we derived (9) and its extension in (11) as what should be truly minimized, regardless of complexity considerations. Other major differences include CPO’s use of in-dataset regularization versus SPO’s global regularization, the incorporation of a weighting mechanism in SPO, the generalization of SPO to other data types, and theoretical guarantees proposed in this work. As we demonstrated in experiments (see Table 4), the choice of regularizer significantly impacts performance.

## 9 Limitations

This paper introduced SPO, a class of algorithms designed for alignment to the expert’s distribution, with controlled softness. The paper also presents theoretical results demonstrating favorable landscape and convergence properties of SPO.
Below, we discuss some limitations of this work.

Limitations of the SPO Algorithm:
The main limitation of the SPO framework is the computational complexity of the regularizer, which requires online sampling from $\pi_{\theta}$. This limitation was discussed in Section 4, when intermittent batch generation of samples was proposed to mitigate the computational overhead .

Limitations of the Current Study:
An interesting question not addressed in this work is how the performance of different methods varies with dataset size and the level of noise in preference labels.
Another promising research direction is generalizing these alignment methods to other forms of implicit preference information hidden in data (e.g., in chat logs), instead of relying on labeled preference datasets.

## Appendix A Proof of Theorems 1 and 2

In this appendix, we present the proof of Theorems 1 and 2. The high-level proof idea is to show that moving along the projected(^4^44Projection on the probability simplex.) negative gradient of the preference loss (i.e., the ODE direction) results in an absolute reduction of the Euclidean distance of $\pi_{\theta}$ from $\operatorname{Softmax}\big{(}r(\cdot|x)/\alpha\big{)}$.

Without loss of generality, we prove the theorem for a single fixed $x\in\mathcal{X}$, and remove $x$ from the notations, for the sake of notation simplicity.

Given the rewards $r(\cdot)$ in the Bradley-Terry model, let

$$ $\pi^{*}(\cdot)\overset{\text{\tiny def}}{=}\operatorname{Softmax}\big{(}r( \cdot)\big{)}.$ (18) $$

For any $\alpha\in[0,1)$, let

$$ $\pi^{*}_{\alpha}(\cdot)\overset{\text{\tiny def}}{=}\operatorname{Softmax}\big {(}r(\cdot)/\alpha\big{)},$ (19) $$

and let $\boldsymbol{\pi}^{\alpha}$ be its vector representation. Therefore, for any $\alpha\in[0,1]$, and for any $y$,

$$ $\pi^{*}(y)=z_{\alpha}\,\times\,\big{(}\pi^{*}_{\alpha}(y)\big{)}^{\alpha}, \qquad\textrm{where }z_{\alpha}\overset{\text{\tiny def}}{=}\frac{\Big{(}\sum_ {y^{\prime}}e^{r(y^{\prime})/\alpha}\Big{)}^{\alpha}}{\sum_{y^{\prime}}e^{r(y^ {\prime})}}.$ (20) $$

Moreover, it follows from the consistency of distribution $\mathcal{D}$ with the Bradley-Terry model that for any pair $(y_{1},y_{2})$,

$$ $\frac{\mathcal{D}(y_{1},y_{2})}{\mathcal{D}(y_{1},y_{2})+\mathcal{D}(y_{2},y_{ 1})}=\mathcal{P}_{\mathcal{D}}\big{(}y_{1}\succ y_{2}\big{)}=\frac{\exp{r(y_{1 })}}{\exp{r(y_{1})}+\exp{r(y_{2})}}=\frac{\pi^{*}(y_{1})}{\pi^{*}(y_{1})+\pi^{ *}(y_{2})}.$ (21) $$

For any $y_{1},y_{2}\in\mathcal{Y}$ let

$$ $\tilde{\mu}(y_{1},y_{2})\overset{\text{\tiny def}}{=}\mu(y_{1},y_{2})\big{(} \mathcal{D}(y_{1},y_{2})+\mathcal{D}(y_{2},y_{1})\big{)}.$ (22) $$

Note that the symmetry of $\mu$ implies symmetry of $\tilde{\mu}$ with respect to its first and second arguments. Then,

$$ $\mu(y_{1},y_{2})\,\mathcal{D}(y_{1},y_{2})=\tilde{\mu}(y_{1},y_{2})\,\frac{ \mathcal{D}(y_{1},y_{2})}{\mathcal{D}(y_{1},y_{2})+\mathcal{D}(y_{2},y_{1})}= \tilde{\mu}(y_{1},y_{2})\,\frac{\pi^{*}(y_{1})}{\pi^{*}(y_{1})+\pi^{*}(y_{2})},$ (23) $$

where the last equality follows from (21).

Consider a $\boldsymbol{\pi}_{\theta}$ in the relative interior of the probability simplex and let $\mathbf{v}$ be the negative gradient of the preference loss

$$ $\mathbf{v}\overset{\text{\tiny def}}{=}-\nabla_{\pi_{\theta}}\,\mathcal{L}_{ \mathrm{pref}}^{\alpha,\mu}(\pi_{\theta},\mathcal{D}),$ (24) $$

where $\mathcal{L}_{\mathrm{pref}}^{\alpha,\mu}$ is defined in (14). For any $y\in\mathcal{Y}$, let $v(y)$ be the entry of $\mathbf{v}$ that corresponds to $y$.
Then,

$$ $\begin{split}v(y)=&\sum_{y^{\prime}\in\mathcal{Y}}\mathcal{D}(y,y^{\prime})\mu (y,y^{\prime})\,\frac{\pi_{\theta}(y^{\prime})^{\alpha}}{\pi_{\theta}(y)^{ \alpha}+\pi_{\theta}(y^{\prime})^{\alpha}}\,\left(\frac{d}{d\,\pi_{\theta}(y)} \log\,\pi_{\theta}(y)-\frac{d}{d\,\pi_{\theta}(y)}\log\,\pi_{\theta}(y^{\prime })\right)\\ &+\sum_{y^{\prime}\in\mathcal{Y}}\mathcal{D}(y^{\prime},y)\mu(y^{\prime},y)\, \frac{\pi_{\theta}(y)^{\alpha}}{\pi_{\theta}(y)^{\alpha}+\pi_{\theta}(y^{ \prime})^{\alpha}}\,\left(\frac{d}{d\,\pi_{\theta}(y)}\log\,\pi_{\theta}(y^{ \prime})-\frac{d}{d\,\pi_{\theta}(y)}\log\,\pi_{\theta}(y)\right)\\ =&\sum_{y^{\prime}\in\mathcal{Y}}\tilde{\mu}(y,y^{\prime})\,\frac{\pi^{*}(y)}{ \pi^{*}(y)+\pi^{*}(y^{\prime})}\,\,\frac{\pi_{\theta}(y^{\prime})^{\alpha}}{ \pi_{\theta}(y)^{\alpha}+\pi_{\theta}(y^{\prime})^{\alpha}}\,\,\frac{1}{\pi_{ \theta}(y)}\\ &-\sum_{y^{\prime}\in\mathcal{Y}}\tilde{\mu}(y,y^{\prime})\,\frac{\pi^{*}(y^{ \prime})}{\pi^{*}(y)+\pi^{*}(y^{\prime})}\,\,\frac{\pi_{\theta}(y)^{\alpha}}{ \pi_{\theta}(y)^{\alpha}+\pi_{\theta}(y^{\prime})^{\alpha}}\,\,\frac{1}{\pi_{ \theta}(y)}\\ =&\sum_{y^{\prime}\in\mathcal{Y}}\,\frac{\tilde{\mu}(y,y^{\prime})\,\big{(}\pi ^{*}(y)\pi_{\theta}(y^{\prime})^{\alpha}-\pi^{*}(y^{\prime})\pi_{\theta}(y)^{ \alpha}\big{)}}{\pi_{\theta}(y)\,\big{(}\pi^{*}(y)+\pi^{*}(y^{\prime})\big{)} \,\big{(}\pi_{\theta}(y)^{\alpha}+\pi_{\theta}(y^{\prime})^{\alpha}\big{)}}, \end{split}$ (25) $$

where the first equality follows from (14) and by considering all the terms that include $y$ either as winner (the first sum) or loser (the second sum); the second equality is due to (23) and the fact that $\tilde{\mu}$ is symmetric.
To simplify the notation, for any $y$ and $y^{\prime}$, let

$$ $h(y,y^{\prime})\overset{\text{\tiny def}}{=}\frac{\tilde{\mu}(y,y^{\prime})}{ \pi_{\theta}(y)\,\pi_{\theta}(y^{\prime})\,\big{(}\pi^{*}(y)+\pi^{*}(y^{\prime })\big{)}\,\big{(}\pi_{\theta}(y)^{\alpha}+\pi_{\theta}(y^{\prime})^{\alpha} \big{)}}.$ (26) $$

Then, (25) simplifies to

$$ $v(y)=\sum_{y^{\prime}\in\mathcal{Y}}h(y,y^{\prime})\,\pi_{\theta}(y^{\prime}) \,\big{(}\pi_{\theta}(y^{\prime})^{\alpha}\pi^{*}(y)-\pi_{\theta}(y)^{\alpha} \pi^{*}(y^{\prime})\big{)}.$ (27) $$

Consequently,

$$ $\begin{split}\mathbf{v}^{T}&(\boldsymbol{\pi}_{\theta}-\boldsymbol{\pi}^{*}_{ \alpha})=\sum_{y\in\mathcal{Y}}\,v(y)\,\big{(}\pi_{\theta}(y)-\pi^{*}_{\alpha} (y)\big{)}\\ =&\sum_{y,y^{\prime}\in\mathcal{Y}}\,h(y,y^{\prime})\,\big{(}\pi_{\theta}(y^{ \prime})^{\alpha}\pi^{*}(y)-\pi_{\theta}(y)^{\alpha}\pi^{*}(y^{\prime})\big{)} \,\,\pi_{\theta}(y^{\prime})\,\big{(}\pi_{\theta}(y)-\pi^{*}_{\alpha}(y)\big{) }\\ =&\,\,\frac{1}{2}\sum_{y,y^{\prime}\in\mathcal{Y}}\,h(y,y^{\prime})\,\big{(} \pi_{\theta}(y^{\prime})^{\alpha}\pi^{*}(y)-\pi_{\theta}(y)^{\alpha}\pi^{*}(y^ {\prime})\big{)}\,\,\pi_{\theta}(y^{\prime})\,\big{(}\pi_{\theta}(y)-\pi^{*}_{ \alpha}(y)\big{)}\\ &+\frac{1}{2}\sum_{y^{\prime},y\in\mathcal{Y}}\,h(y^{\prime},y)\,\big{(}\pi_{ \theta}(y)^{\alpha}\pi^{*}(y^{\prime})-\pi_{\theta}(y^{\prime})^{\alpha}\pi^{* }(y)\big{)}\,\,\pi_{\theta}(y)\,\big{(}\pi_{\theta}(y^{\prime})-\pi^{*}_{ \alpha}(y^{\prime})\big{)}\\ =&\,\,\frac{1}{2}\sum_{y,y^{\prime}\in\mathcal{Y}}\,h(y,y^{\prime})\,\big{(} \pi_{\theta}(y^{\prime})^{\alpha}\pi^{*}(y)-\pi_{\theta}(y)^{\alpha}\pi^{*}(y^ {\prime})\big{)}\,\,\,\big{(}\pi_{\theta}(y^{\prime})\pi_{\theta}(y)-\pi_{ \theta}(y^{\prime})\pi^{*}_{\alpha}(y)\big{)}\\ &+\frac{1}{2}\sum_{y,y^{\prime}\in\mathcal{Y}}\,h(y,y^{\prime})\,\big{(}\pi_{ \theta}(y^{\prime})^{\alpha}\pi^{*}(y)-\pi_{\theta}(y)^{\alpha}\pi^{*}(y^{ \prime})\big{)}\,\,\,\big{(}\pi_{\theta}(y)\pi^{*}_{\alpha}(y^{\prime})-\pi_{ \theta}(y)\pi_{\theta}(y^{\prime})\big{)}\\ =&\,\,\frac{1}{2}\sum_{y,y^{\prime}\in\mathcal{Y}}\,h(y,y^{\prime})\,\big{(} \pi_{\theta}(y^{\prime})^{\alpha}\pi^{*}(y)-\pi_{\theta}(y)^{\alpha}\pi^{*}(y^ {\prime})\big{)}\,\,\,\big{(}\pi_{\theta}(y)\pi^{*}_{\alpha}(y^{\prime})-\pi_{ \theta}(y^{\prime})\pi^{*}_{\alpha}(y)\big{)}\\ =&\,-\frac{z_{\alpha}}{2}\sum_{y,y^{\prime}\in\mathcal{Y}}\,h(y,y^{\prime})\, \Big{(}\big{(}\pi_{\theta}(y^{\prime})\pi^{*}_{\alpha}(y)\big{)}^{\alpha}-\big {(}\pi_{\theta}(y)\pi^{*}_{\alpha}(y^{\prime})\big{)}^{\alpha}\Big{)}\,\,\, \big{(}\pi_{\theta}(y^{\prime})\pi^{*}_{\alpha}(y)-\pi_{\theta}(y)\pi^{*}_{ \alpha}(y^{\prime})\big{)}\\ =&\,-\frac{z_{\alpha}}{2}\sum_{y,y^{\prime}\in\mathcal{Y}}\,h(y,y^{\prime}) \big{(}\pi_{\theta}(y)\pi_{\theta}(y^{\prime})\big{)}^{1+\alpha}\,\left(\left( \frac{\pi^{*}_{\alpha}(y)}{\pi_{\theta}(y)}\right)^{\alpha}-\left(\frac{\pi^{* }_{\alpha}(y^{\prime})}{\pi_{\theta}(y^{\prime})}\right)^{\alpha^{\vphantom{A} }}\right)\,\,\,\left(\frac{\pi^{*}_{\alpha}(y)}{\pi_{\theta}(y)}-\frac{\pi^{*} _{\alpha}(y^{\prime})}{\pi_{\theta}(y)}\right),\end{split}$ (28) $$

where the second equality follows from (27), the fourth equality is due to the symmetry of $h(y,y^{\prime})$ with respect to $y$ and $y^{\prime}$, i.e., $h(y,y^{\prime})=h(y^{\prime},y)$, and the sixth equality is from (20).
It is easy to see that all terms in the sum in the last line are non-negative, and the sum contains at least one non-zero term if $\boldsymbol{\pi}_{\theta}\neq\boldsymbol{\pi}^{*}_{\alpha}$. Therefore, $\mathbf{v}^{T}(\boldsymbol{\pi}_{\theta}-\boldsymbol{\pi}^{*}_{\alpha})<0$ if $\boldsymbol{\pi}_{\theta}\neq\boldsymbol{\pi}^{*}_{\alpha}$. Consequently, $\lVert\boldsymbol{\pi}_{\theta}-\boldsymbol{\pi}^{*}_{\alpha}\rVert$ is strictly decreasing when moving along $\mathbf{v}$. Since both $\boldsymbol{\pi}_{\theta}$ and $\boldsymbol{\pi}^{*}_{\alpha}$ lie on the probability simplex, we have $\prod(\mathbf{v})^{T}(\boldsymbol{\pi}_{\theta}-\boldsymbol{\pi}^{*}_{\alpha})
\leq\mathbf{v}^{T}(\boldsymbol{\pi}_{\theta}-\boldsymbol{\pi}^{*}_{\alpha})<0$. It follows that for any $\boldsymbol{\pi}_{\theta}$ in the relative interior of the probability simplex, projection of $\mathbf{v}$ on the probability simplex is a strictly decent direction for $\lVert\boldsymbol{\pi}_{\theta}-\boldsymbol{\pi}^{*}_{\alpha}\rVert$.

As a result, $\pi^{*}_{\alpha}$ is the globally absorbing unique fixed point of the ODE. Furthermore, when $\mu$ is not a function of $\pi_{\theta}$, then $\pi^{*}_{\alpha}$ is the unique first order stationary point of the preference loss $\mathcal{L}_{\mathrm{pref}}^{\alpha,\mu}$. In other words, $\mathcal{L}_{\mathrm{pref}}^{\alpha,\mu}$ contains no other local mininum, local maximum, or saddle-point in the probability simplex.

## Appendix B Proof of Theorem 3

This appendix presents the proof of Theorem 3. The high-level idea, akin to Appendix A, is to show that moving along the ODE direction results in an absolute reduction of the Euclidean distance of $\pi_{\theta}$ from $\operatorname{Softmax}\big{(}r(\cdot|x)/\alpha\big{)}$. The details are however substantially different from Appendix A.

We begin with the following lemma.

###### Lemma 1 .

For any $\eta>0$ and any pair of $n$-dimensional vectors $\boldsymbol{a}$ and $\boldsymbol{b}$ with positive entries, we have

$$ $\sum_{i=1}^{n}\left(\frac{a_{i}}{b_{i}}\right)^{\eta}\left(\frac{b_{i}}{\sum_{ j=1}^{n}b_{j}}-\frac{a_{i}}{\sum_{j=1}^{n}a_{j}}\right)\leq 0,$ (29) $$

and the equality holds only if $\boldsymbol{a}=c\boldsymbol{b}$ for some scalar $c$.

###### Proof of Lemma 1 .

Fix an arbitrary vector $\boldsymbol{a}$ with positive entries, and consider the following function

$$ $f(\mathbf{x})\overset{\text{\tiny def}}{=}\sum_{i=1}^{n}\left(\frac{a_{i}}{x_{ i}}\right)^{\eta}\left(\frac{x_{i}}{\sum_{j=1}^{n}x_{j}}-\frac{a_{i}}{\sum_{j= 1}^{n}a_{j}}\right),\qquad\textrm{for}\quad\mathbf{x}\in\mathbb{R}_{+}^{n},$ (30) $$

defined on the positive quadrant. We will show that $f(\mathbf{x})\leq 0$, for all $x\in\mathbb{R}^{n}_{+}$. Note that if $f(\mathbf{x})>0$ for some $x$, then
$f(c\mathbf{x})=f(\mathbf{x})/c^{\eta}>0$, for all $c>0$. Therefore, without loss of generality, we confine the domain to a compact set, say to the probability simplex $\mathcal{S}\overset{\text{\tiny def}}{=}\{\mathbf{x}\in\mathbb{R}_{+}^{*}:\,\,
\sum_{i=1}^{n}x_{i}=1\}$,
and show that $f(\mathbf{x})\leq 0$ for all $\mathbf{x}\in\mathcal{S}$. In the same vein, without loss of generality we also assume that

$$ $\sum_{i=1}^{n}a_{i}=1.$ (31) $$

Note that $f(\mathbf{x})=-\infty$ on the boundary of the probability simplex, that is if $x_{i}=0$ for some $i$. Therefore, the maximizer $\mathbf{x}^{*}$ of $f$ over $\mathcal{S}$, lies in the relative interior of $\mathcal{S}$. Consequently, the gradient of the Lagrangian of $f$ at $\mathbf{x}^{*}$ is zero.
The Lagrangian $L$ of $f$ is as follows:

$$ $L(\mathbf{x},\lambda)\overset{\text{\tiny def}}{=}f(\mathbf{x})+\lambda\left( \sum_{i=1}^{n}x_{i}-1\right),\qquad\textrm{for}\,\,\mathbf{x}\in\mathcal{S},\, \lambda\in\mathbb{R}.$ (32) $$

Then,

$$ $\begin{split}\frac{d}{d\,x_{k}}L(\mathbf{x},\lambda)&=\frac{d}{d\,x_{k}}f( \mathbf{x})+\lambda\\ &=\frac{d}{d\,x_{k}}\sum_{i=1}^{n}\left(\frac{a_{i}}{x_{i}}\right)^{\eta}\left (\frac{x_{i}}{\sum_{j=1}^{n}x_{j}}-\frac{a_{i}}{\sum_{j=1}^{n}a_{j}}\right)+ \lambda\\ &=\frac{d}{d\,x_{k}}\sum_{i=1}^{n}\left(\frac{a_{i}^{\eta}x_{i}^{1-\eta}}{\sum _{j=1}^{n}x_{j}}-a_{i}^{1+\eta}x_{i}^{-\eta}\right)+\lambda\\ &=\frac{(1-\eta)a_{k}^{\eta}x_{k}^{-\eta}}{\sum_{j=1}^{n}x_{j}}-\frac{\sum_{i= 1}^{n}a_{i}^{\eta}x_{i}^{1-\eta}}{\left(\sum_{j=1}^{n}x_{j}\right)^{2}}+\eta a _{k}^{1+\eta}x_{k}^{-\eta-1}+\lambda\\ &=(1-\eta)\left(\frac{a_{k}}{x_{k}}\right)^{\eta}+\eta\left(\frac{a_{k}}{x_{k} }\right)^{1+\eta}+\left[\lambda-\sum_{i=1}^{n}a_{i}^{\eta}x_{i}^{1-\eta}\right ]\end{split}$ (33) $$

where the third equality is due to (31), and the last equality is because $\sum_{j}x_{j}=1$. Consider a scalar function $h:\mathbb{R}_{+}\to\mathbb{R}_{+}$ as follows

$$ $h(y)\overset{\text{\tiny def}}{=}(1-\eta)\,y^{\eta}\,+\,\eta\,y^{1+\eta}\qquad \textrm{for}\quad y\geq 0.$ (34) $$

Then, (33) simplifies to

$$ $\frac{d}{d\,x_{k}}L(\mathbf{x},\lambda)=h\left(\frac{a_{k}}{x_{k}}\right)+C( \lambda,\mathbf{x},\boldsymbol{a}),$ (35) $$

where $C(\lambda,\mathbf{x},\boldsymbol{a})=\lambda-\sum_{i=1}^{n}a_{i}^{\eta}x_{i}^{
1-\eta}$ is independent of $k$.
Therefore, letting $\nabla_{\mathbf{x}}L(\mathbf{x},\lambda)=0$ at $\mathbf{x}=\mathbf{x}^{*}$, it follows that for any $1\leq i<j\leq n$,

$$ $h\left(\frac{a_{i}}{x_{i}^{*}}\right)=h\bigg{(}\frac{a_{j}}{x_{j}^{*}}\bigg{)}.$ (36) $$

We now consider two cases for $\eta$.

Case 1 ($\eta\leq 1$). In this case, $h$ defined in (34) is an strictly increasing function. Therefore, (36) implies that $a_{i}/x_{i}^{*}=a_{j}/x_{j}^{*}$, for all $i,j\leq n$. Equivalently, $\mathbf{x}^{*}=c\boldsymbol{a}$ for some scalar $c>0$. In this case, from (30), $f(\mathbf{x}^{*})=0$. The lemma then follows from the fact that $\mathbf{x}^{*}$ is the maximizer of $f$.

Case 2 ($\eta>1$). In this case, $h$ is no longer increasing. In this case, $h$ is unimodal. Specifically, $h$ is strictly decreasing over $\big{[}0,(\eta-1)/(\eta+1)\big{]}$ and is strictly increasing over $\big{[}(\eta-1)/(\eta+1),\infty\big{]}$. This unimodality implies that the pre-image of any $y\in\mathbb{R}_{+}$ (i.e., $h^{-1}(y)$) is a set of at most two points. Consequently, (36) implies that we can partition the indices $1,\ldots,n$ into two groups $S_{1}$ and $S_{2}$ such that within each group, we have $a_{i}/x_{i}^{*}=a_{j}/x_{j}^{*}$. In other words, $a_{i}/x_{i}^{*}=a_{j}/x_{j}^{*}$ for all $(i,j)\in S_{1}\times S_{1}$ and all $(i,j)\in S_{2}\times S_{2}$. Equivalently, the maximum point, $\mathbf{x}^{*}$, belongs to the set

$$ $X^{*}\overset{\text{\tiny def}}{=}\big{\{}\mathbf{x}\in\mathbb{R}_{+}^{n}:\,\, x_{i}=c_{1}a_{i}\textrm{ for }i\leq k,\textrm{ and }x_{i}=c_{2}a_{i}\textrm{ for }i>k,\textrm{ for some }c_{1},c_{2}>0\textrm{ and }k<n\big{\}},$ (37) $$

where we have assumed without loss of generality that $S_{1}=\{1,\ldots,k\}$ and $S_{2}=\{k+1,\ldots,n\}$ for some $k\leq n$.
We will show that $f(\mathbf{x})\leq 0$ for all $\mathbf{x}\in X^{*}$.

Fix some $\mathbf{x}\in X^{*}$, and corresponding constants $c_{1}$, $c_{2}$, and $k$, as per (37). Let $A=\sum_{i=1}^{k}a_{i}$ and $B=\sum_{i=k+1}^{n}a_{i}$. Then,

$$ $\begin{split}f(\mathbf{x})&=\sum_{i=1}^{n}\left(\frac{a_{i}}{x_{i}}\right)^{ \eta}\left(\frac{x_{i}}{\sum_{j=1}^{n}x_{j}}-\frac{a_{i}}{\sum_{j=1}^{n}a_{j}} \right)\\ &=\sum_{i=1}^{n}\left(\frac{a_{i}}{x_{i}}\right)^{\eta}\left(\frac{x_{i}}{c_{1 }A+c_{2}B}-\frac{a_{i}}{A+B}\right)\\ &=\sum_{i=1}^{k}c_{1}^{-\eta}\left(\frac{c_{1}a_{i}}{c_{1}A+c_{2}B}-\frac{a_{i }}{A+B}\right)+\sum_{i=k+1}^{n}c_{2}^{-\eta}\left(\frac{c_{2}a_{i}}{c_{1}A+c_{ 2}B}-\frac{a_{i}}{A+B}\right)\\ &=\left(\frac{c_{1}^{1-\eta}A}{c_{1}A+c_{2}B}-\frac{c_{1}^{-\eta}A}{A+B}\right )+\left(\frac{c_{2}^{1-\eta}B}{c_{1}A+c_{2}B}-\frac{c_{2}^{-\eta}B}{A+B}\right )\\ &=\frac{c_{1}^{1-\eta}A+c_{2}^{1-\eta}B}{c_{1}A+c_{2}B}-\frac{c_{1}^{-\eta}A+c _{2}^{-\eta}B}{A+B}\\ &=\frac{(c_{1}^{1-\eta}A+c_{2}^{1-\eta}B)(A+B)-(c_{1}^{-\eta}A+c_{2}^{-\eta}B) (c_{1}A+c_{2}B)}{(c_{1}A+c_{2}B)(A+B)}\\ &=\frac{(c_{1}-c_{2})(c_{1}^{-\eta}-c_{2}^{-\eta})AB}{(c_{1}A+c_{2}B)(A+B)}\\ &\leq 0,\end{split}$ $$

and the inequality in the last line holds with equality iff either $A$ or $B$ are zero (note that $c_{1},c_{2},\eta>0$), which is the case only if $\mathbf{x}=c_{1}\boldsymbol{a}$ or $\mathbf{x}=c_{2}\boldsymbol{a}$. The lemma then follows from the fact that $\mathbf{x}^{*}$ is the maximizer of $f$.

This completes the proof of Lemma 1.
∎

We proceed with the proof of the theorem.
Given the rewards $r(\cdot|\cdot)$ in the $n$-ary BT model (see Section 5), let

$$ $\pi^{*}(\cdot|\cdot)\overset{\text{\tiny def}}{=}\operatorname{Softmax}\big{(} r(\cdot|\cdot)\big{)}.$ (38) $$

For any $(x;y_{1},\ldots,y_{n})\in\mathcal{X}\times\mathcal{Y}^{n}$, let

$$ $\bar{\mathcal{D}}(x;y_{1},\ldots,y_{n})\overset{\text{\tiny def}}{=}\frac{\sum _{i=1}^{n}\mathcal{D}(x;y_{1},\ldots,y_{n};i)}{\sum_{i=1}^{n}\pi^{*}(y_{i}|x)}.$ (39) $$

It then follows from the consistency of $\mathcal{D}$ with the $n$-ary BT model that for any $(x;y_{1},\ldots,y_{n})\in\mathcal{X}\times\mathcal{Y}^{n}$ and $i=1,\ldots,n$

$$ $\mathcal{D}(x;y_{1},\ldots,y_{n};i)=\bar{\mathcal{D}}(x;y_{1},\ldots,y_{n})\, \pi^{*}(y_{i}|x).$ (40) $$

We further define

$$ $\tilde{\mathcal{D}}(x;[y])\overset{\text{\tiny def}}{=}\bar{\mathcal{D}}(x;y_{ 1},\ldots,y_{n})\,\mu(x;y_{1},\ldots,y_{n}).$ (41) $$

For brevity of notation, we denote $y_{1},\ldots,y_{n}$ by $[y]$ and denote $1,\ldots,n$ by $[n]$.
The loss function $\mathcal{L}_{\mathrm{pref}\textrm{-}n}^{\alpha,\mu}(\pi,\mathcal{D})$ defined in (16) can then be simplified to

$$ $\begin{split}\mathcal{L}_{\mathrm{pref}\textrm{-}n}^{\alpha,\mu}(\pi,\mathcal{ D})&=-\frac{1}{\alpha}\mathbb{E}_{(x;y_{1},\ldots,y_{n};i^{*})\sim\mathcal{D}} \left[\mu(y_{1},\ldots,y_{n}\mid x)\,\log\,\frac{\pi(y_{i^{*}}\mid x)^{\alpha} }{\sum_{i=1}^{n}\pi(y_{i}\mid x)^{\alpha}}\right]\\ &=-\frac{1}{\alpha}\sum_{(x;[y];i^{*})\in\mathcal{X}\times\mathcal{Y}^{n} \times[n]}\mathcal{D}(x;[y];i^{*})\,\mu([y]|x)\,\log\,\frac{\pi(y_{i^{*}}\mid x )^{\alpha}}{\sum_{i=1}^{n}\pi(y_{i}\mid x)^{\alpha}}\\ &=-\frac{1}{\alpha}\sum_{(x;[y];i^{*})\in\mathcal{X}\times\mathcal{Y}^{n} \times[n]}\tilde{D}(x;[y])\,\pi^{*}(y_{i^{*}}|x)\log\,\frac{\pi(y_{i^{*}}\mid x )^{\alpha}}{\sum_{i=1}^{n}\pi(y_{i}\mid x)^{\alpha}}\\ &=-\frac{1}{\alpha}\sum_{(x;[y])\in\mathcal{X}\times\mathcal{Y}^{n}}\tilde{D}( x;[y])\,\sum_{i=1}^{n}\pi^{*}(y_{i}|x)\log\,\frac{\pi(y_{i}\mid x)^{\alpha}}{ \sum_{j=1}^{n}\pi(y_{j}\mid x)^{\alpha}},\end{split}$ $$

where the third equality is due to (40) and (41).

In the rest of the proof, without loss of generality, we consider a single fixed $x\in\mathcal{X}$, and remove $x$ from the notations for the sake of notation brevity.
Let

$$ $\pi^{*}_{\alpha}(\cdot)\overset{\text{\tiny def}}{=}\operatorname{Softmax}\big {(}r(\cdot)/\alpha\big{)}.$ (42) $$

It follows that for any $y\in\mathcal{Y}$,

$$ $\pi^{*}_{\alpha}(y)=\frac{\pi^{*}(y)^{1/\alpha}}{\sum_{\tilde{y}\in\mathcal{Y} }\pi^{*}(\tilde{y})^{1/\alpha}}.$ (43) $$

Let $\boldsymbol{\pi}$ and $\boldsymbol{\pi}^{*}_{\alpha}$ be the vector representation of $\pi(y)$ and $\pi^{*}_{\alpha}(y)$ for all $y\in\mathcal{Y}$.
Then, for $\mathbf{v}\overset{\text{\tiny def}}{=}-\nabla_{\pi}\mathcal{L}_{\mathrm{pref}
\textrm{-}n}^{\alpha,\mu}(\pi,\mathcal{D})$ we have

$$ $\begin{split}\,\,\,\,\,\,\,&\!\!\!\!\!\!\!\left(\boldsymbol{\pi}-{\boldsymbol{ \pi}^{*}}^{1/\alpha}\right)^{T}\,\mathbf{v}=-\left(\boldsymbol{\pi}-{ \boldsymbol{\pi}^{*}_{\alpha}}\right)^{T}\,\nabla_{\pi}\mathcal{L}_{\mathrm{ pref}\textrm{-}n}^{\alpha,\mu}(\pi,\mathcal{D})\\ &=\frac{1}{\alpha}\sum_{[y]\in\mathcal{Y}^{n}}\tilde{D}([y])\,\left( \boldsymbol{\pi}-{\boldsymbol{\pi}^{*}_{\alpha}}\right)^{T}\,\nabla_{\pi}\sum_ {i=1}^{n}\pi^{*}(y_{i})\,\log\,\frac{\pi(y_{i}\mid x)^{\alpha}}{\sum_{j=1}^{n} \pi(y_{j}\mid x)^{\alpha}}\\ &=\frac{1}{\alpha}\sum_{[y]\in\mathcal{Y}^{n}}\tilde{D}([y])\,\sum_{\tilde{y} \in\mathcal{Y}}\left(\pi(\tilde{y})-{\pi^{*}_{\alpha}}(\tilde{y})\right)\, \frac{d}{d\,\tilde{y}}\sum_{i=1}^{n}\pi^{*}(y_{i})\,\log\,\frac{\pi(y_{i}\mid x )^{\alpha}}{\sum_{j=1}^{n}\pi(y_{j}\mid x)^{\alpha}}\\ &=\frac{1}{\alpha}\sum_{[y]\in\mathcal{Y}^{n}}\tilde{D}([y])\,\sum_{k=1}^{n} \left(\pi(y_{k})-{\pi^{*}_{\alpha}}(y_{k})\right)\,\frac{d}{d\,y_{k}}\sum_{i=1 }^{n}\pi^{*}(y_{i})\,\log\,\frac{\pi(y_{i}\mid x)^{\alpha}}{\sum_{j=1}^{n}\pi( y_{j}\mid x)^{\alpha}}.\end{split}$ (44) $$

For any $[y]=(y_{1},\ldots,\mathbf{y}_{n})\in\mathcal{Y}^{n}$, let

$$ $A\big{(}[y]\big{)}\overset{\text{\tiny def}}{=}\frac{1}{\alpha}\sum_{k=1}^{n} \left(\pi(y_{k})-{\pi^{*}_{\alpha}}(y_{k})\right)\,\frac{d}{d\,y_{k}}\sum_{i=1 }^{n}\pi^{*}(y_{i})\,\log\,\frac{\pi(y_{i}\mid x)^{\alpha}}{\sum_{j=1}^{n}\pi( y_{j}\mid x)^{\alpha}}.$ (45) $$

It then follows from (44) that:

$$ $\mathbf{v}^{T}\left(\boldsymbol{\pi}-{\boldsymbol{\pi}^{*}}^{1/\alpha}\right)= \sum_{[y]\in\mathcal{Y}^{n}}\tilde{D}([y])\,A\big{(}[y]\big{)}.$ (46) $$

We proceed to compute $A\big{(}[y]\big{)}$.
For $k=1,\ldots,n$,

$$ $\begin{split}\frac{d}{d\,y_{k}}\sum_{i=1}^{n}\frac{\pi(y_{i}\mid x)^{\alpha}}{ \sum_{j=1}^{n}\pi(y_{j}\mid x)^{\alpha}}&=\frac{d}{d\,y_{k}}\sum_{i=1}^{n}\pi^ {*}(y_{i})\left(\log\,\pi(y_{i})^{\alpha}\,-\,\log\,\sum_{j=1}^{n}\pi(y_{j})^{ \alpha}\right)\\ &=\alpha\frac{\pi^{*}(y_{k})}{\pi(y_{k})}\,-\,\left(\sum_{i=1}^{n}\pi^{*}(y_{i })\right)\frac{d}{d\,y_{k}}\log\,\sum_{j=1}^{n}\pi(y_{j})^{\alpha}\\ &=\alpha\frac{\pi^{*}(y_{k})}{\pi(y_{k})}\,-\,\alpha\left(\sum_{i=1}^{n}\pi^{* }(y_{i})\right)\frac{\pi(y_{k})^{\alpha-1}}{\sum_{j=1}^{n}\pi(y_{j})^{\alpha}} \\ \end{split}$ (47) $$

Plugging this into the definition of $A\big{(}[y]\big{)}$ in (45), we obtain

$$ $\begin{split}A\big{(}[y]\big{)}&=\sum_{k=1}^{n}\left(\pi(y_{k})-{\pi^{*}_{ \alpha}}(y_{k})\right)\,\left(\frac{\pi^{*}(y_{k})}{\pi(y_{k})}\,-\,\left(\sum _{i=1}^{n}\pi^{*}(y_{i})\right)\frac{\pi(y_{k})^{\alpha-1}}{\sum_{j=1}^{n}\pi( y_{j})^{\alpha}}\right)\\ &=\sum_{k=1}^{n}\pi(y_{k})\,\left(\frac{\pi^{*}(y_{k})}{\pi(y_{k})}\,-\,\left( \sum_{i=1}^{n}\pi^{*}(y_{i})\right)\frac{\pi(y_{k})^{\alpha-1}}{\sum_{j=1}^{n} \pi(y_{j})^{\alpha}}\right)\\ &\quad+\sum_{k=1}^{n}{\pi^{*}_{\alpha}}(y_{k})\left(\left(\sum_{i=1}^{n}\pi^{* }(y_{i})\right)\frac{\pi(y_{k})^{\alpha-1}}{\sum_{j=1}^{n}\pi(y_{j})^{\alpha}} \,-\,\frac{\pi^{*}(y_{k})}{\pi(y_{k})}\right)\\ &=\sum_{k=1}^{n}\pi^{*}(y_{k})\,-\,\left(\sum_{i=1}^{n}\pi^{*}(y_{i})\right) \frac{\sum_{k=1}^{n}\pi(y_{k})^{\alpha}}{\sum_{j=1}^{n}\pi(y_{j})^{\alpha}}\\ &\quad+\sum_{k=1}^{n}{\pi^{*}_{\alpha}}(y_{k})\left(\left(\sum_{i=1}^{n}\pi^{* }(y_{i})\right)\frac{\pi(y_{k})^{\alpha-1}}{\sum_{j=1}^{n}\pi(y_{j})^{\alpha}} \,-\,\frac{\pi^{*}(y_{k})}{\pi(y_{k})}\right)\\ &=\sum_{k=1}^{n}{\pi^{*}_{\alpha}}(y_{k})\left(\left(\sum_{i=1}^{n}\pi^{*}(y_{ i})\right)\frac{\pi(y_{k})^{\alpha-1}}{\sum_{j=1}^{n}\pi(y_{j})^{\alpha}}\,-\, \frac{\pi^{*}(y_{k})}{\pi(y_{k})}\right)\\ &=\frac{\sum_{i=1}^{n}\pi^{*}(y_{i})}{\sum_{i=1}^{n}\pi^{*}(y_{i})^{1/\alpha}} \,\,\sum_{k=1}^{n}\left(\frac{\pi^{*}(y_{k})}{\pi(y_{k})^{\alpha}}\right)^{1/ \alpha}\left(\frac{\pi(y_{k})^{\alpha}}{\sum_{j=1}^{n}\pi(y_{j})^{\alpha}}\,- \,\frac{\pi^{*}(y_{k})}{\sum_{i=1}^{n}\pi^{*}(y_{i})}\right)\\ &\leq 0\qquad(\textrm{\tt``$=$'' only if }\boldsymbol{\pi}^{*}=c\boldsymbol{ \pi}\textrm{\tt\,\,\,for some scalar }c>0),\end{split}$ (48) $$

where the last equality is due to (43), and the inequality in the last line follows from Lemma 1 by letting $a_{k}=\pi^{*}(y_{k})$, $b_{k}=\pi(y_{k})^{\alpha}$, and $\eta=1/\alpha$.
Plugging this into (46), it follows that

$$ $-\left(\nabla_{\pi}\mathcal{L}_{\mathrm{pref}\textrm{-}n}^{\alpha,\mu}(\pi, \mathcal{D})\right)^{T}\left(\boldsymbol{\pi}-{\boldsymbol{\pi}^{*}_{\alpha}} \right)=\mathbf{v}^{T}(\boldsymbol{\pi}-\boldsymbol{\pi}^{*}_{\alpha})<0$ (49) $$

if $\boldsymbol{\pi}\neq\boldsymbol{\pi}^{*}_{\alpha}$. Consequently, $\lVert\boldsymbol{\pi}-\boldsymbol{\pi}^{*}_{\alpha}\rVert$ is strictly decreasing when moving along $\mathbf{v}$.
Since both $\boldsymbol{\pi}$ and $\boldsymbol{\pi}^{*}_{\alpha}$ lie on the probability simplex, we have $\prod(\mathbf{v})^{T}(\boldsymbol{\pi}-\boldsymbol{\pi}^{*}_{\alpha})\leq
\mathbf{v}^{T}(\boldsymbol{\pi}-\boldsymbol{\pi}^{*}_{\alpha})<0$. It follows that for any $\boldsymbol{\pi}$ in the relative interior of the probability simplex, projection of $\mathbf{v}$ on the probability simplex is a strictly decent direction for $\lVert\boldsymbol{\pi}-\boldsymbol{\pi}^{*}_{\alpha}\rVert$.
As a result, $\boldsymbol{\pi}^{*}_{\alpha}$ is the globally absorbing unique fixed point of the ODE.
This completes the proof of Theorem 3.

###### Lemma 1 .

###### Proof of Lemma 1 .

## Appendix C Proof of Theorem 4

Here we present the proof of Theorem 4. The high level idea is to show that $\mathcal{L}_{\mathrm{rank}}^{\alpha,[\mu]}(\pi,\mathcal{D})$ can be equivalently written as the sum of $\mathcal{L}_{\mathrm{pref}\textrm{-}n}^{\alpha,\mu_{k}}(\pi,\mathcal{D}_{k})$ for appropriately defined $\mathcal{D}_{k}$, for $k=1,\ldots,n-1$; where each $\mathcal{D}_{k}$ is consistent with the $(n-k+1)$-ary BT model (defined in Section 5).
We then use Theorem 3, and in particular (49) in the proof of Theorem 3, to conclude that the softmax distribution is a globally absorbing fixed point of $-\nabla\mathcal{L}_{\mathrm{pref}\textrm{-}n}^{\alpha,\mu_{k}}(\pi,\mathcal{D}
_{k})$ for $k=1,\ldots,n-1$, and is therefore a globally absorbing fixed point of their sum, $-\nabla\mathcal{L}_{\mathrm{rank}}^{\alpha,[\mu]}(\pi,\mathcal{D})$.

As in the previous appendices, without loss of generality we prove the theorem for a single fixed $x\in\mathcal{X}$, and remove $x$ from the equations for notation brevity.
To further simplify the notation, without loss of generality, we also remove the permutation $\tau$ from the equations, and represent the ranking by mere order of the indices, that is we assume that $y_{1}\succ y_{2}\succ\cdots\succ y_{n}$. With these new conventions, the ranking loss (17) simplifies to

$$ $\mathcal{L}_{\mathrm{rank}}^{\alpha,[\mu]}(\pi,\mathcal{D})\overset{\text{ \tiny def}}{=}-\frac{1}{\alpha}\mathbb{E}_{(y_{1},\ldots,y_{n})\sim\mathcal{D} }\left[\sum_{k=1}^{n-1}\mu_{k}(y_{k},\ldots,y_{n})\,\log\,\frac{\pi(y_{k})^{ \alpha}}{\sum_{j=k}^{n}\pi(y_{j})^{\alpha}}\right].$ (50) $$

For $k=1,\ldots,n-1$, we define an $(n-k+1)$-ary preference distribution $\mathcal{D}_{k}$ as follows. For any $(y_{1},\ldots,y_{n-k+1})\in\mathcal{Y}^{n}$ and $i=1,\ldots,n-k+1$,

$$ $\mathcal{D}_{k}(y_{1},\ldots,y_{n-k+1};i)=\frac{1}{(n-k)!}\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\sum_{\begin{subarray}{c}(z_{1},\ldots,z_{k-1})\in\mathcal {Y}^{k-1}\\ \textrm{Permutation }\tau:(1,\ldots,n-k)\to(1,\ldots,\cancel{i},\ldots,n-k+1) \end{subarray}}\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\mathcal{D}(z_{1}, \ldots,z_{k-1},y_{i},y_{\tau(1)},\ldots,y_{\tau(n-k)}).$ (51) $$

From (50), we have

$$ $\begin{split}\,\,\,\,\,\,\,\,&\!\!\!\!\!\!\!\!\mathcal{L}_{\mathrm{rank}}^{ \alpha,[\mu]}(\pi,\mathcal{D})=-\frac{1}{\alpha}\mathbb{E}_{(y_{1},\ldots,y_{n })\sim\mathcal{D}}\left[\sum_{k=1}^{n-1}\mu_{k}(y_{k},\ldots,y_{n})\,\log\, \frac{\pi(y_{k})^{\alpha}}{\sum_{j=k}^{n}\pi(y_{j})^{\alpha}}\right]\\ &=-\frac{1}{\alpha}\sum_{k=1}^{n-1}\mathbb{E}_{(y_{1},\ldots,y_{n})\sim \mathcal{D}}\left[\mu_{k}(y_{k},\ldots,y_{n})\,\log\,\frac{\pi(y_{k})^{\alpha} }{\sum_{j=k}^{n}\pi(y_{j})^{\alpha}}\right]\\ &=-\frac{1}{\alpha}\sum_{k=1}^{n-1}\sum_{(y_{1},\ldots,y_{n})\in\mathcal{Y}^{n }}\mathcal{D}(y_{1},\ldots,y_{n})\left[\mu_{k}(y_{k},\ldots,y_{n})\,\log\, \frac{\pi(y_{k})^{\alpha}}{\sum_{j=k}^{n}\pi(y_{j})^{\alpha}}\right]\\ &=-\frac{1}{\alpha}\sum_{k=1}^{n-1}\sum_{y_{k},\ldots,y_{n}}\sum_{(y_{1}, \ldots,y_{k-1})\in\mathcal{Y}^{k-1}}\!\!\!\!\mathcal{D}(y_{1},\ldots,y_{n}) \left[\mu_{k}(y_{k},\ldots,y_{n})\,\log\,\frac{\pi(y_{k})^{\alpha}}{\sum_{j=k} ^{n}\pi(y_{j})^{\alpha}}\right]\\ &=-\frac{1}{\alpha}\sum_{k=1}^{n-1}\sum_{y_{k},\ldots,y_{n}}\frac{\mathcal{D}_ {k}(y_{1},\ldots,y_{n-k+1};1)}{\big{(}(n-k)!\big{)}^{2}}\left[\mu_{k}(y_{k}, \ldots,y_{n})\,\log\,\frac{\pi(y_{k})^{\alpha}}{\sum_{j=k}^{n}\pi(y_{j})^{ \alpha}}\right]\\ &=-\frac{1}{\alpha}\sum_{k=1}^{n-1}\frac{1}{(n-k+1)!\,(n-k)!}\mathbb{E}_{(y_{k },\ldots,y_{n};i)\sim\mathcal{D}_{k}}\left[\mu_{k}(y_{k},\ldots,y_{n})\,\log\, \frac{\pi(y_{k})^{\alpha}}{\sum_{j=k}^{n}\pi(y_{j})^{\alpha}}\right]\\ &=\sum_{k=1}^{n-1}\frac{\mathcal{L}_{\mathrm{pref}\textrm{-}n}^{\alpha,\mu_{k} }(\pi,\mathcal{D}_{k})}{(n-k+1)!\,(n-k)!}.\end{split}$ $$

Let $\boldsymbol{\pi}$ and $\boldsymbol{\pi}^{*}_{\alpha}$ be the vector representations of $\pi$ and the softmax distribution $\pi^{*}_{\alpha}$ (defined in (43)), and $\mathbf{v}\overset{\text{\tiny def}}{=}-\nabla_{\pi}\mathcal{L}_{\mathrm{rank}
}^{\alpha,[\mu]}(\pi,\mathcal{D})$. Then,

$$ $\begin{split}\left(\boldsymbol{\pi}-{\boldsymbol{\pi}^{*}_{\alpha}}\right)^{T} \mathbf{v}&=-\left(\boldsymbol{\pi}-{\boldsymbol{\pi}^{*}_{\alpha}}\right)^{T} \nabla\sum_{k=1}^{n-1}\frac{\mathcal{L}_{\mathrm{pref}\textrm{-}n}^{\alpha,\mu _{k}}(\pi,\mathcal{D}_{k})}{(n-k+1)!\,(n-k)!}\\ &=\sum_{k=1}^{n-1}\frac{-\left(\boldsymbol{\pi}-{\boldsymbol{\pi}^{*}_{\alpha} }\right)^{T}\nabla\mathcal{L}_{\mathrm{pref}\textrm{-}n}^{\alpha,\mu_{k}}(\pi, \mathcal{D}_{k})}{(n-k+1)!\,(n-k)!}\\ &\leq 0,\end{split}$ $$

where the last inequality follows from (49), and it holds with equality only if $\boldsymbol{\pi}\neq\boldsymbol{\pi}^{*}_{\alpha}$.
Since both $\boldsymbol{\pi}$ and $\boldsymbol{\pi}^{*}_{\alpha}$ lie on the probability simplex, we have $\prod(\mathbf{v})^{T}(\boldsymbol{\pi}-\boldsymbol{\pi}^{*}_{\alpha})\leq
\mathbf{v}^{T}(\boldsymbol{\pi}-\boldsymbol{\pi}^{*}_{\alpha})<0$. Then, following a similar argumnet as in the last paragraph of Appendix B, we conclude that $\boldsymbol{\pi}^{*}_{\alpha}$ is the globally absorbing unique fixed point of the ODE.
This completes the proof of Theorem 4.

## Appendix D Experiment Details

### D.1 Details for the AlpacaFarm experiment of Section 7.1

We performed the alignment procedure on 4 NVIDIA H100 (94 GiB) GPUs. For each method, we trained the model for four epochs and reported the maximum win-rate against the SFT model.
The training batch size per GPU device was set to one and the gradient accumulation step was 16. The range of hyper-parameters considered for each method is given as follows: R-DPO: $\beta\in\{0.01,0.1\}$, $\alpha\in\{0.001,0.01,0.1\}$, CPO: $\beta\in\{0.001,0.01,0.1\}$, $\alpha\in\{0.0001,0.001,0.01\}$, SimPO: $\beta\in\{2,2.5\}$, $\gamma\in\{1,1.5\}$ (the suggested range in SimPO), KTO: $\beta\in\{0.01,0.1\}$, $\lambda_{D}=\lambda_{U}=1$, IPO: $\tau\in\{0.001,0.01,0.1\}$, DPO: $\beta\in\{0.001,0.01,0.1\}$, and SPO: $\alpha\in\{0.001,0.01,0.1\}$.

### D.2 Details for the experiment of Section 7.2

We trained the models on NVIDIA A100 (40 GiB) GPUs.
We used a batch size of 32 samples (each containing four responses) for all algorithms. The reference model in all algorithms was identical to the SFT model. All alignment loss functions were optimized using AdamW with 5,000 warm-up iterations.

For SPO, we trained both basic (i.e., $\gamma=0$) and weighted version. For weighted SPO, we set $\gamma=0.01$ without sweeping, and in the ranking experiment we used decayed weight functions $\eta^{k}\mu_{k}$ for $\eta\in{1,0.5}$, see (17). For other SPO parameters, we swept over $\beta\in\{0.01,0.1\}$, and $\alpha\in\{0.001\}$. For $\mathcal{D}_{\textrm{KL}}$ computation, we used intermittent batch generation of samples, generating a batch of 32 samples from $\pi_{\theta}$ once every 8 iterations (i.e., $T=8$).
For other algorithms, we swept over the following sets of hyperparameters: DPO: $\beta\in\{0.0001,0.001,0.01\}$, S-DPO for ranking: $\beta\in\{0.0001,0.001,0.01\}$, and S-DPO for best-of-$n$: $\beta\in\{0.0001,0.001,0.01\}$. For For training the S-DPO algorithm on the best-of-$n$ dataset, we consider each top-rank response as the positive response and the corresponding lower-rank responses as the corresponding negative set of responses. For training S-DPO on ranking dataset, each of the 1st, 2nd, and 3rd rank responses serve as positive responses with the corresponding negative set containing the corresponding lower rank responses.

We computed the win rates of all methods against the best-of-$n$ SFT model using GPT4o-2024-08-06 once every 1000 iterations, and reported the peak win-rate for each method. Each win-rate was averaged over 1,000 story-pair instances, resulting in an estimation error with standard deviation smaller than $0.015$.

## Appendix E Details of Generation of Ranking Dataset for the TinyStories Experiment

We created a preference dataset to align stories with older age groups. Specifically, for each pair of stories generated by the reference model, we asked GPT4o–2024-08-06 to evaluate them based on clarity and coherence, writing quality, and whether they are interesting and engaging for high school students. The API was asked to evaluate each story independently based on these criteria, and identify its strengths and weaknesses compared to other stories; and then suggest a ranking of stories from best to worst. The prompt used for generating the dataset is provided at the end of this subsection.

We generated a set of 100,000 stories independently from the 110M-parameter pre-trained model (Karpathy, ), and grouped them into a set of 25,000 samples each containing four stories. To enhance the quality of the ranking dataset, for each sample we used the prompt to rank the stories twice, reversing the order of the stories in the second evaluation. We retained samples only if both evaluations showed a consistent ranking. After this filtration, 5,000 samples remained for use in the ranking dataset.

Prompt for Generation of the Ranking Dataset for TinyStories Experiment:
You are tasked with deciding which of the four short stories below, written by high school students, is better suited for publication in the high school newspaper.
**Story 1:**  {}
**Story 2:**  {}
**Story 3:**  {}
**Story 4:**  {}
**Your Task:**
1. **Evaluate Each Story Individually:**
Identify the **strengths** and **weaknesses** of each story, focusing on:
- **Engagement:** Is the story interesting and likely to captivate high school students?
- **Clarity and Coherence:** Is the story well-organized and easy to follow?
- **Writing Quality:** Assess the grammar, vocabulary, and overall language use.
2. **Make a Final Decision:**
- Based on your evaluations, decide which story is better suited for publication. Rank the stories from best to worst.
**Response Format:**
***
**Evaluation:**
**Story 1:**
- **Strengths compared to other stories:**
- [List strengths]
- **Weaknesses compared to other stories:**
- [List weaknesses]
---
**Story 2:**
- **Strengths compared to other stories:**
- [List strengths]
- **Weaknesses compared to other stories:**
- [List weaknesses]
---
**Story 3:**
- **Strengths compared to other stories:**
- [List strengths]
- **Weaknesses compared to other stories:**
- [List weaknesses]
---
**Story 4:**
- **Strengths compared to other stories:**
- [List strengths]
- **Weaknesses compared to other stories:**
- [List weaknesses]
---
**Conclusion:**
- **Overall Ranking of the Stories:**
1. **1st Place:** Story [1, 2, 3, or 4]
2. **2nd Place:** Story [1, 2, 3, or 4]
3. **3rd Place:** Story [1, 2, 3, or 4]
4. **4th Place:** Story [1, 2, 3, or 4]
***
**Guidelines:**
- In each evaluation, compare the story to the others, noting unique strengths and weaknesses.
- Evaluation of each story shold not be influenced from prior evaluations that you provided earlier.
- Do not let the presentation order affect your judgment; treat all stories equally.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: I put my questions in the weakness
soundness: 2
strengths: * The paper studies the important question of what regularization data for the alignment methods
* The proposed objective is interesting and has connection to the reward model
summary: This paper proposes an offline alignment objective composed of the preference loss and regularization loss. In the preference loss, it uses a temperature parameter to control the "softness" of the preference; in the regularization loss, it uses KL under different prompts (than preference prompts). Theoretically, it shows the connection to the reward models. Empirically, it shows improved win rates against SFT among baselines
weaknesses: I think the main weakness is that the proposed method is a combination of many components, and it's difficult to assess the utility of each one
* First, the preference loss rewritten as $log \sigma(log \pi_\theta(y_1) - log \pi_\theta(y_2))$ is similar to the DPO loss, but without calibration using $\pi_0(.)$. The DPO has the calibration terms because of the KL regularization using in-distribution dataset. The paper can't convince me that getting rid of the calibration terms brings any advantage in itself (either theoretically or empirically). I think the calibration term could be important when for example $y_1$ is much more/less likely than $y_2$ under $\pi_0$. Maybe the author could consider an ablation experiment comparing the two objectives (you can also use $\alpha$ in both cases too). 
* Second question is when might out-of-domain regularization helps. Again, I am not convinced if, and when, such regularization helps. 

Besides, 
* I can't confidently assess the method's utility just looking at win rates in the experiments. A method could do very well in win rate by having a huge KL from the SFT --- maybe plotting KL vs win rate gives more convincing arguments. 
* The paper says RLHF uses out-of-domain prompts --- is that true? 

I will raise my points if my concerns get addressed.

---

confidence: 3
contribution: 3
initial_rating: 3
presentation: 3
questions: 1. Regarding the preference representation: While your method defines $p(y_w > y_l | x)$ without depending on BT/PL model assumptions, isn't it effectively equivalent in expressivity? Please correct me if I am wrong. It seems that $\pi_\theta(y_w | x)^\alpha$ could be seen as an alternative parametrization of $\exp(\frac{1}{\beta}r_\phi(y_w, x))$, where $\alpha$ and $\beta$ serve similar roles, and $r_\phi$ and $\pi_\theta$ map to equivalent functions. Could you clarify if/how your formulation provides additional expressivity?
2. How does the choice of $\alpha$ interact with different types of preference data (pairwise, best-of-n, ranked)? Is there a principled way to select optimal $\alpha$ values for different scenarios or data distributions?
soundness: 1
strengths: 1. The paper presents a well-structured framework for preference loss without relying on Bradley-Terry/Plackett-Luce model assumptions. The development from basic to general algorithm is logical and easy to follow.
2. The framework shows impressive versatility by handling multiple data types (pairwise, best-of-n, and ranked preference data), demonstrating a comprehensive approach to preference optimization.
3. The introduction of global regularization rather than in-dataset regularization is a novel and well-justified approach that shows better empirical results.
4. The theoretical foundations are solid, with clear proofs and convergence guarantees for each variant of the algorithm.
summary: This paper introduces SPO (Soft Preference Optimization), a method for aligning language models with human preferences without requiring a reward model. SPO operates by optimizing model outputs directly using a loss function that combines preference loss with regularization across the full output distribution. While SPO doesn't require reward model assumptions, the authors prove it converges to a softmax of scaled rewards under the Bradley-Terry model, with adjustable softmax temperature. The method differs from predecessors like DPO in two ways: its preference loss formulation, and its application of regularization across the entire output distribution rather than just the preference dataset.
weaknesses: 1. The experimental evaluation has several limitations:
* The experimental evaluation would benefit from broader model and dataset coverage. While the current results on Llama-2-7B and AlpacaFarm are present, only one setting might be not enough. Here are some suggestions for reference:  
The baseline performance is concerning, with several methods showing negative results on length-controlled win rates, suggesting potential issues with the experimental setup or hyperparameter tuning. You may want to consider including models from different families such as Mistral-7B to show architecture-agnostic performance. To mention but not significant problem, llama-2-7B is currently not among the most capable open-source LLMs according to leaderboards like AlpacaEval. Showing results for llama-3-8B might make the results more significant. Additionally, evaluating on diverse datasets like UltraFeedback (~3 times larger than the alpacaFarm dataset) would verify generalizability across different instruction-tuning distributions.  These additions would provide stronger evidence for the method's broad applicability.

* Specific concerns are raised about the baseline performance that warrant further investigation. The baseline methods has at most 51.94% on the LC Win-rate against SFT, which seems unusually poor given their reported performance in original papers. Since you are fine-tuning starting from the SFT model, that means all the preference optimization baseline methods nearly not improve the quality at all, even worse the performance for R-DPO, CPO, SimPO, IPO.    
                                           Is there any specific reason? While there could be various reasons, one possibility is suboptimal hyperparameter choices. I recommend using similar hyperparameter settings as successful implementations in previous work (e.g., Meng et al. 2024, SimPO paper reported strong performance with their specific configuration), to ensure a fair comparison of these baseline methods.

2. Could you clarify the evaluation dataset sizes for each experiment? This information is particularly relevant when examining the global regularizer results, where some improvements show modest gains (e.g., from 59.9% to 60.8%). Given these subtle differences, I recommend including statistical significance testing to strengthen the results' interpretation.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 4
questions: 1. 
> Moreover, SPO has an advantage over DPO and RLHF in avoiding determinism. In cases where the
preference dataset is comparable to pre-training data size, regularization (DKL) becomes unnecessary,
and RLHF and DPO loss functions tend to produce deterministic models; that is they tend to return a
single high-quality response per query.

Doesn't that depend on the value of the $\beta$ parameter? See, for instance, Theorem 1 in https://arxiv.org/abs/2206.00761. You can see that as $\beta$ becomes large, the optimal distribution reverts to the original model.
soundness: 4
strengths: * S1: The paper proposes a simple and elegant model to fit preference data, including pairwise, best-of-n, and ranked data. The model is so simple in fact that it is surprising it had not been proposed before. (*1)
* S2: It also includes an extensive theoretical analysis of the optimal solution to the proposed loss when the preference data is sampled from Bradley-Terry or Plackett-Luce models (*2)
* S3: The presentation is very clear
* S4: The empirical evaluation is sufficiently convincing of the competitiveness of the method

(*1) In the related work, the authors do acknowledge a similar proposal in CPO by Xu et al.
(*2) I have not carefully checked the calculations of the theorems proofs.
summary: This paper introduces Soft Preference Optimization (SPO), a technique to fine-tune a LM to fit a dataset containing the preferred response to a given query, given some alternative, or given a ranking of possible responses. The main contribution of this paper is to model the preference probability $P_{\pi_\theta}(y_1 \succ y_2)$ as $\frac{\pi_\theta(y_1)}{\pi_\theta(y_1) + \pi_\theta(y_2)}$ and then fit the preference data using a log-likelihood loss plus a regularization term, which here is taken to be the commonly used KL divergence to the original model.

The model is further extended to include an (inverse) "temperature" parameter $\alpha$ to regulate the entropy of the resulting distribution, and a weighting function $\mu$ that is intended to down-weight preference data given on poor-quality samples (as judged by the model itself).

Also, a generalized loss to accommodate best-of-n preference data and ranked preference data is provided. 

The model was evaluated on win-rates against a SFT-based competitor, comparing positively to a number of competitive baselines on pairwise data and also to DPO on best-of-n and ranked data. For the latter task, a dataset was generated which is provided in the anonymous link.

Finally, a (biased, low-variance) estimator of the KL is proposed using token-level subsequences of the samples, which I think is another contribution of this paper, although I am not 100% sure of that.
weaknesses: None that I can think of

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Does the performance of the SPO come at the expense of other capabilities?

2. Diversity and accuracy are inherently trade off, but why is there no discussion of this in the experimental section of the article?
soundness: 2
strengths: 1. This paper introduces the concept of Soft Preference Optimization (SPO) and elaborates on its theoretical foundations. The authors not only outline the goals and motivations behind SPO but also explore its distinctions from existing approaches, such as Direct Preference Optimization (DPO). The mathematical proofs presented in the paper are rigorous, particularly those of the main theorems detailed in the appendix. These proofs not only substantiate the effectiveness of the SPO method but also establish that SPO can converge to an optimal softmax distribution under specific conditions. Such rigorous mathematical derivations lay a solid foundation for the scientific validity and credibility of the SPO approach.

2. In the experimental section, SPO outperforms some current baseline alignment methods, demonstrating its effectiveness in tasks such as instruction following. Moreover, SPO significantly outperforms other baselines in its ability to focus on the best data.
summary: This paper presents Soft Preference Optimization (SPO), a generative model approach that aligns with human preferences without a reward model. SPO directly optimizes the model output by combining a natural loss function with a preference loss and a regularization term, covering the entire output distribution. It was theoretically proved that under the assumption of the Bradley-Terry model, SPO could converge to the Softmax distribution, and the output softness was adjusted by the algorithm parameters. Experiments on AlpacaFarm benchmark show that SPO outperforms SimPO, DPO and other alignment algorithms. The authors claim the advantage of SPO in avoiding determinism compared to DPO and RLHF.
weaknesses: 1.The paper presents Soft Preference Optimization (SPO) as a novel method for aligning generative models with human preferences without relying on a reward model.  While the theoretical foundation is well-elaborated, the experimental section lacks a comprehensive evaluation of SPO across a broader range of pairwise alignment datasets and benchmarks.  A more extensive set of experiments is necessary to substantiate the claim that SPO can effectively focus on optimal data alignment without sacrificing performance in other areas.  It is crucial to include more experimental results and analyses to demonstrate that SPO enhances model alignment while maintaining or improving performance in related tasks. 


2. The authors highlight the differences between SPO and existing approaches like Direct Preference Optimization (DPO) and Reinforcement Learning from Human Feedback (RLHF).  While it is noted that SPO introduces a superior regularization term to prevent overfitting on preference data, the paper does not sufficiently address whether this approach might come at the expense of other model capabilities.  DPO, being a reward-model-free algorithm, is already designed to balance alignment with human preferences and model performance.  Therefore, it is essential for the authors to conduct additional experiments that validate whether SPO's emphasis on diversity and alignment comes at the cost of the model's ability to perform well on a broader range of tasks.  A thorough analysis comparing SPO against DPO and other baselines in terms of overall model performance, as well as alignment, would significantly strengthen the paper's contributions and claims.

3. The experimental section of the paper, while demonstrating the effectiveness of Soft Preference Optimization (SPO), does not include an ablation study on the hyper-parameters used within the SPO framework. By including such analysis, the paper would offer a more complete picture of the method's robustness and the sensitivity to hyper-parameter tuning, which is valuable for both theoretical understanding and practical application.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Could the authors provide a more detailed empirical comparison between PPO and SPO? Since SPO requires on-policy data sampling to compute the KL-regularization, introducing additional complexity that is typically avoided by off-policy alignment methods. Thereby, the key difference in performance between PPO and SPO might stem from the first term in Eq.(1). Would optimizing on off-policy preference data offer a significant advantage over on-policy sampled data, as in PPO, and could this be further explored? For example, training both methods on the same preference dataset, then evaluating them on a held-out test set (in-distribution) and a separate dataset from a different domain (out-of-distribution) respectively.

2. Could the authors provide a clearer interpretation of the solution obtained by SPO when regularization is applied under the BT assumption and show how the regularizer affects the convergence properties of the algorithm, as this represents the actual objective being optimized. 

3. Could the authors conduct a more thorough analysis of the weighting function by breaking down its individual components and performing an ablation study, as it would help to justify the necessity of its complex form.
soundness: 3
strengths: The approach of integrating preference loss with explicit regularization makes sense. The experimental results across different preference settings appear promising.
summary: The authors propose Soft Preference Optimization (SPO), which optimizes two key components: (1) a preference loss function aimed at maximizing the likelihood of favoring the chosen response over the rejected one, and (2) a KL-divergence regularization term applied to the model's samples. The authors extend SPO to handle other types of preferences, such as best-of-n and ranked preferences. They conduct experiments using various types of preference data, including pairwise, best-of-n, and ranked preferences. The ablation study further validates the choice of the regularization term and the additional weight function used in the objective.
weaknesses: 1. While the authors claim that reward-model-free alignment methods offer advantages, these methods, which solely optimize a preference dataset, may overfit or overestimate out-of-distribution data. This can result in degraded performance during inference. Therefore, to convincingly demonstrate the effectiveness of the proposed method, it is crucial to compare it against RLHF approaches like PPO. This can be achieved by replacing the preference loss with an on-policy loss term for a fair comparison.

2. The derivation of Theorem 1, as well as the subsequent theorems, does not account for the regularization term, which is typically present in the analytic solutions of PPO or DPO. This omission compromises the theoretical results, given that the proposed method essentially optimizes a combination of two loss terms.

3. The design of $\mu$ lacks intuition despite its complication, e.g., subtracting the average in a batch. Also the effect of $\gamma$ would be clearer with an ablation study of sweeping this hyper-parameter.

4. Missing data on the top of page 9: "improved win-rate from xx% to xx%".

5. Missing reference to related alignment methods [1,2,3] that match the model's output distribution.

[1] Sessa, Pier Giuseppe, et al. "Bond: Aligning llms with best-of-n distillation." Arxiv (2024).

[2] Gui, Lin, Cristina Gârbacea, and Victor Veitch. "BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling." Arxiv (2024).

[3] Ji, Haozhe, et al. "Towards efficient and exact optimization of language model alignment." ICML (2024).

Meta-review: This paper proposes SPO (Soft Preference Optimization), a method for aligning language models with human preferences without requiring a reward model. After rebuttal, it received mixed scores of 55568. On one hand, reviewers commented that the proposed objective is interesting and insightful. The reviewer who gave a score of 8 also commented that none of the pointed out weaknesses seriously undermine the soundness or the novelty of this work. On the other hand, several reviewers also commented that the experiment evaluation is not sufficient enough to show the effectiveness of the proposed method. The experimental evaluation would benefit from broader model and dataset coverage. During rebuttal, the authors have made clarifications and added new experiments. Overall, the flaws slightly outweigh the merits, and the AC would have to make the hard decision to reject the paper by the end.

Decision: Reject