MallowsPO: Fine-Tune Your LLM with Preference Dispersions

## Abstract

Abstract. Direct Preference Optimization (DPO) has recently emerged as a popular approach to improve reinforcement learning with human feedback (RLHF), leading to better techniques to fine-tune large language models (LLM). A weakness of DPO, however, lies in its lack of capability to characterize the diversity of human preferences.
Inspired by Mallows’ theory of preference ranking, we develop in this paper a new approach, the MallowsPO .
A distinct feature of this approach is a dispersion index , which reflects the dispersion of human preference to prompts. We show that existing DPO models can be reduced to special cases of this dispersion index, thus unified with MallowsPO. More importantly, we demonstrate empirically how to use this dispersion index to enhance the performance of DPO in a broad array of benchmark tasks, from synthetic bandit selection to controllable generation and dialogues, while maintaining great generalization capabilities.
MallowsPO is also compatible with other SOTA offline preference optimization methods, boosting nearly 2% extra LC win rate when used as a plugin for fine-tuning Llama3-Instruct.

## 1. Introduction

Reinforcement Learning from Human Feedback (RLHF) [, , ] has made significant
contributions to the success of
Large Language Models (LLMs) such as ChatGPT and GPT4 [].
Recently, Direct Preference Optimization (DPO) []
proposes to bypass RL, thus leading to faster training and better resource efficiency.
More importantly, DPO achieves comparable or superior performance against RLHF in downstream tasks such as fine-tuning LLMs in Llama3 [], Zephyr [], Neural Chat, BTLM-DPO [], etc. DPO’s success has attracted much research attention, leading to variants
beyond pairwise ranking such as KTO [, ],
unified perspectives on loss parameterization such as IPO [], GPO [], and reference-free alternatives such as CPO [], ORPO [], SimPO [], etc.

Notwithstanding the successes by RLHF and DPO, both are limited by the assumption that preference follows the Bradley-Terry (BT) model []. Particularly, they do not account for varying degrees of agreement in responses to different prompts. For instance, people are more likely to agree on “$1+1=$? // $2$.” as opposed to “What is the best city to live in the U.S.? // New York.” In language models, this concerns the dispersion of next-token prediction, similar to personalization in recommendation systems [, ]. See Figure 1 for more examples.

The purpose of this paper is to formalize the idea of prompt dispersion in the design of DPO.
We adapt Mallows’ preference ranking theory [, ],
a family of ranking models that provide a natural carrier for prompt dispersion, and
propose the following decomposition/factorization of the (latent) reward function:
$\mbox{reward}(\mbox{prompt, completion})=\mbox{dispersion}(\mbox{prompt})
\times\mbox{scaled reward}(\mbox{completion}\mid\mbox{prompt})$,
where “prompt” and “completion” correspond, respectively, to question and answer.
This decomposition allows to specify
the diverse level of prompt dispersions hidden in the DPO,
which is translated into a prompt-dependent factor – the dispersion index in the preference likelihood.
The scaled reward is given by the relative rank of the (possible) completions, which further enhances the model interpretability.
We then leverage the change of variables technique to propose two models, MallowsPO-$\theta$ and MallowsPO-$\phi$,
by two choices of the discrepancy function in the Mallows model
which we will elaborate in Section 3.1.

Figure: Figure 1. Prompts with low/high neg-log dispersion estimate values from Anthropic HH dataset.
Refer to caption: extracted/5896174/Figures/prompt_dispersion_with_examples.png

The main contributions of this paper are four-fold.

(1) We formalize the idea of prompt dispersion in DPO, and develop the MallowsPO approach to implement this idea, so as to improve and generalize DPO.
As far as we are concerned, this is the first work in
preference optimization that considers a mathematically well-grounded preference ranking model, i.e., Mallows, beyond BT setting.

(2) We propose approximations to the dispersion index, a critical component of the Mallows model, so as to facilitate computation.
We present a synthetic example to illustrate that our proposed estimate is aligned with the true dispersion.

(3) The Mallows model serves as a foundation for developing versatile preference optimization objectives, including MallowsPO-$\theta$ (a generalization of DPO) and MallowsPO-$\phi$. We also provide various analytical results for MallowsPOs, uncovering various new insights on existing DPO models, and a generalized $\Psi$PO model that unifies all DPO models (including MallowsPO).

(4) We conduct extensive experiments, from synthetic bandits, controllable generation, fine-tuning Pythia 2.8B on off-policy Anthropic HH dataset, to fine-tuning Llama3-8B-Instruct on a hybrid UltraFeedback dataset.
Notably, we perform an exclusive hyperparameter search for a fair comparison, and repeat for different random seeds to justify the significance of the improvement. The results show clear advantages of MallowsPO over (BT-)DPO, which showcases the potential of this novel direction of considering preference/prompt dispersion.

Related Works.
Existing work on personalization in dialogue generation such as [] and [] has also paid attention to the diversity of human preferences (“there are a thousand Hamlets in a thousand people’s eyes”);
[] proposes a Nash game model to incorporate the diversity.
There are also other DPO variants:
$f$-DPO [] considers general $f$-divergence
in DPO;
ODPO [] adds a margin to account for the preference significance.
Recent works propose to learn online preferences [, ],
or learn from AI feedbacks [, , ].
For classical RLHF, studies to improve the design and capabilities of RLHF include [, , , , , ], whose ideas can also benefit DPO.
See [] for a survey on learning from preferences.

The remainder of the paper is organized as follows.
Background materials on RLHF and DPO are highlighted in Section 2.
Section 3 focuses on the development of MallowsPO,
followed by more analytical results and various perspectives in Section 4.
Experimental results are detailed in Section 5,
and concluding remarks in Section 6.

## 2. Preliminaries

Both RLHF and DPO
start with fine-tuning a pre-trained LLM by supervised learning on
high-quality data for some downstream tasks of interest,
to acquire a model $\pi^{\mathrm{SFT}}$.
This step is referred to as the supervised fine-tuning (SFT) phase.
For instance, for training InstructGPT [],
GPT-3 [] is first fine-tuned on the given input prompt distribution.

$\diamondsuit$ RLHF [, , ].
On top of $\pi^{\mathrm{SFT}}$, RLHF is proposed to serve as the next step to conduct further fine-tuning to generate high-quality outputs as judged by humans.
Given a generative model $\pi$, it is prompted with prompts $x$ to produce pairs of answers (or, “completions”),
$\left\{y_{1},y_{2}\right\}\sim\pi(y\mid x)$,
which are then presented to human labelers
who express preferences for one completion over the other.
Denote by $y_{w}\succ y_{l}\mid x$,
meaning that $y_{w}\in\left\{y_{1},y_{2}\right\}$ is preferred over $y_{l}\in\left\{y_{1},y_{2}\right\}$.
The preferences are assumed to be generated by some latent reward model $r^{*}(x,y)$.
Based on the collected preference data $\{x^{(i)},y_{w}^{(i)},y_{l}^{(i})\}_{i=1}^{N}$, RLHF consists of first learning a reward model $r(x,y)$,
followed by learning a policy $\pi_{r}(y\mid x)$
in which the prompt $x$ is the state, and the completion $y$ is the action.

(a) Reward model.
To capture the underlying human preferences,
RLHF assumes the Bradley-Terry model [] that stipulates the pairwise preference distribution:

$$ $p^{*}\left(y_{1}\succ y_{2}\mid x\right):=\sigma\left(r^{*}\left(x,y_{1}\right )-r^{*}\left(x,y_{2}\right)\right),$ (1) $$

where $\sigma(s):=\frac{1}{1+e^{-s}}$.
Given access to a static dataset of comparisons
$\mathcal{D}=\{x^{(i)},y_{w}^{(i)},y_{l}^{(i)}\}_{i=1,\ldots,N}$,
RLHF seeks to approximate the latent reward $r^{*}(x,y)$ by a family of functions $\{r_{\psi}(x,y)\}_{\psi}$,
and estimate the parameters by minimizing the (negative) log-likelihood loss:

$$ $\min_{\psi}\mathcal{L}\left(r_{\psi},\mathcal{D}\right):=-\mathbb{E}_{\left(x, y_{w},y_{l}\right)\sim\mathcal{D}}\left[\log\sigma\left(r_{\psi}\left(x,y_{w} \right)-r_{\psi}\left(x,y_{l}\right)\right)\right].$ $$

Denote by $r_{\psi_{*}}(x,y)$ the solution to the problem.

(b) RL. The learned reward function $r_{\psi_{*}}(x,y)$ is then used to provide feedback to the language model.
More precisely, the following KL-regularized RL problem is considered:

$$ $\displaystyle\max_{\pi}\mathbb{E}_{x\sim\mathcal{D}}\left[\mathbb{E}_{y\sim\pi (y\mid x)}\left[r_{\psi_{*}}(x,y)\right]-\beta\mathrm{KL}\left(\pi(\cdot\mid x )\|\pi_{\mathrm{ref}}(\cdot\mid x)\right)\right],$ (2) $$

where $\beta>0$ is a hyperparameter controlling the deviation from the reference policy
$\pi_{\mathrm{ref}}=\pi^{\mathrm{SFT}}$.

In view of (2), RLHF uses the reward function
$r(x,y)=r_{\psi}(x,y)-\beta\left(\log\pi(y\mid x)-\log\pi_{\text{ref }}(y\mid x
)\right)$,
and solves the RL problem by proximal policy optimization (PPO) [].

$\diamondsuit$ DPO [].
One disadvantage of RLHF is that the RL step often requires substantial computational effort (e.g., to carry out PPO).
The idea of DPO is to combine the two steps (a)–(b) in RLHF into a single one,
bypassing the computation in the RL step.

The key idea is that given a reward function $r(x,y)$,
the problem in (2)
has a closed-form solution:
$\pi_{r}(y\mid x)=\frac{1}{Z(x)}\pi_{\text{ref }}(y\mid x)\exp\left(r(x,y)/
\beta\right)$,
where $Z(x)=\sum_{y}\pi_{\text{ref }}(y\mid x)\exp\left(r(x,y)/\beta\right)$.
Rewrite the above as:
$r(x,y)=\beta\log\frac{\pi_{r}(y\mid x)}{\pi_{\text{ref }}(y\mid x)}+\beta\log Z
(x)$.
Through this change of variables, the latent reward $r^{*}(x,y)$ can be expressed in terms of the optimal
policy $\pi^{*}(y\mid x)$,
the reference policy $\pi_{\text{ref }}(y\mid x)$
and a constant $Z^{*}(x)$.
Substituting this $r^{*}$ expression into (1) yields:

$$ $p^{*}\left(y_{1}\succ y_{2}\mid x\right)=\sigma\left(\beta\log\frac{\pi^{*} \left(y_{1}\mid x\right)}{\pi_{\text{ref }}\left(y_{1}\mid x\right)}-\beta\log \frac{\pi^{*}\left(y_{2}\mid x\right)}{\pi_{\text{ref }}\left(y_{2}\mid x \right)}\right),$ (3) $$

where $Z^{*}(x)$ cancels out.
The expression in (3) motivates the DPO objective:

$$ $\min_{\pi}\mathcal{L}_{\mathrm{DPO}}\left(\pi;\pi_{\mathrm{ref}}\right):=- \mathbb{E}_{\left(x,y_{w},y_{l}\right)\sim\mathcal{D}}\left[\log\sigma\left( \beta\log\frac{\pi\left(y_{w}\mid x\right)}{\pi_{\text{ref }}\left(y_{w}\mid x \right)}-\beta\log\frac{\pi\left(y_{l}\mid x\right)}{\pi_{\text{ref }}\left(y_ {l}\mid x\right)}\right)\right].$ (4) $$

## 3. DPO based on Mallows Ranking Models

### 3.1. Mallows ranking models

The Mallows model is built upon the analysis of rankings,
instead of scores or ratings that play the central role in BT models.
Concretely, for a positive integer $n$ that represents e.g., $n$ possible items, let $\mathfrak{S}_{n}$ be the set of permutations of $[n]=\{1,\ldots,n\}$ and the space of rankings.
We consider the following probability of observing a ranking $\mu$ (which represents the preference of $n$ items, e.g., the top ranked item is preferred over the others):

$$ $\mathbb{P}_{\phi,\mu_{0},d}(\mu):=\phi^{d\left(\mu,\mu_{0}\right)}/Z(\phi,d) \quad\text{ for }\mu\in\mathfrak{S}_{n},$ (5) $$

where $\phi\in(0,1]$ is the dispersion parameter,
$\mu_{0}$ is the central ranking,
(also known as the location parameter),
$d(\cdot,\cdot)$ is a discrepancy function that is right invariant: $d(\mu_{1},\mu_{2})=d\left(\mu_{1}\circ\mu_{2}^{-1},id\right)$ for $\mu_{1},\mu_{2}\in\mathfrak{S}_{n}$,
and $Z(\phi,d):=\sum_{\mu\in\mathfrak{S}_{n}}\phi^{d\left(\mu,\mu_{0}\right)}$
is the normalizing constant.

Notice that the dispersion indeed reflects how dispersed the probability distribution (5) on the space of rankings is: When $\phi\to 0$, it is concentrated on $\mu_{0}$,
and when $\phi=1$, it is uniformly distributed.
In an attempt to study ranking models (over $n$ items) with pairwise preferences,
[] further considered two specific cases of the discrepancy function in (5):

- •
Mallows-$\theta$ model: $d(\mu_{1},\mu_{2})=\sum_{i=1}^{n}(\mu_{1}(i)-\mu_{2}(i))^{2}$ is the Spearman’s rho,
- •
Mallows-$\phi$ model: $d(\mu_{1},\mu_{2})=\operatorname{inv}\left(\mu_{1}\circ\mu_{2}^{-1}\right)$ is the Kendall’s tau,

where $\operatorname{inv}(\mu):=\#\left\{(i,j)\in[n]^{2}:i<j\right.$ and $\left.\mu(i)>\mu(j)\right\}$ is the number of inversions of $\mu$.

The general form in (5) was suggested by [] along with other discrepancy functions (e.g., Cayley, Ulam distances, etc.)
See [, , ] for the related group representation approach to ranked, or partially ranked data.
Note that the Mallows models
can be extended to infinite ranking models with $n=\infty$
(see [, , ].)
In the context of language models,
this conforms to a possibly infinite number of completions given a prompt,
and allows interpreting unseen completions .

### 3.2. MallowsPO

We adapt Mallows ranking models highlighted above to the setting of language models.
First, denote by $\mu(\cdot\mid x)$ a ranking of completions given the prompt $x$, such that the preference distribution is:

$$ $p^{*}\left(y_{1}\succ y_{2}\mid x\right)=\mathbb{P}\left(\mu(y_{1}\mid x)<\mu \left(y_{2}\mid x\right)\right).$ (6) $$

Next, for the preference probability in (5),
given an input prompt $x$, we assume it induces a conditional central ranking $\mu_{0}(\cdot\mid x)$,
and a dispersion index $\phi(x)\in(0,1]$.
As pointed out in [],
finding $\mu_{0}(\cdot\mid x)$ is computationally hard.
Fortunately, we discover that in RLHF, this part can be “cleverly" circumvented. By representing $r^{*}(x,y)$ as the (negative) rank $-\mu_{0}(y\mid x)$,
our goal now becomes:

$$ $\max_{\pi}\mathbb{E}_{x\sim\mathcal{D}}\left[\mathbb{E}_{y\sim\pi_{\theta}(y \mid x)}\left[-\mu_{0}(y\mid x)\right]-\beta\mathrm{KL}\left(\pi(\cdot\mid x) \|\pi_{\mathrm{ref}}(\cdot\mid x)\right)\right].$ $$

Note that a smaller rank is preferred as per (6). Hence, this provides a natural candidate for the scaled reward that enhances model interpretation.
This perspective leads to the discovery of a novel family of preference optimization objectives, each of which corresponds to an instance of Mallows models.

###### Theorem 1 (MallowsPO) .

Given a prompt $x$, suppose the completions $y$ follow a Mallows preference distribution $p^{*}$ as in (6), with chosen hyperparameters $d(\cdot,\cdot)$ and $\phi(x)$. Then for any underlying central rank $\mu_{0}(\cdot\mid x)$, the optimal RLHF policy $\pi_{\mu_{0}}(\cdot\mid x)$ satisfies $p^{*}\left(y_{1}\succ y_{2}\mid x\right)=g_{d,\phi(x)}(\beta\log(\frac{\pi_{
\mu_{0}}(y_{1}\mid x)}{\pi_{\mathrm{ref}}(y_{1}\mid x)})-\beta\log(\frac{\pi_{
\mu_{0}}(y_{2}\mid x)}{\pi_{\mathrm{ref}}(y_{2}\mid x)}))$.
Therefore, the policy optimization objective is:

$$ $\min_{\pi}-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\left[\log\left(g_{d, \phi(x)}\left(\beta\log\frac{\pi_{\mu_{0}}(y_{w}\mid x)}{\pi_{\mathrm{ref}}(y_ {w}\mid x)}-\beta\log\frac{\pi_{\mu_{0}}(y_{l}\mid x)}{\pi_{\mathrm{ref}}(y_{l }\mid x)}\right)\right)\right].$ (7) $$

The proof is given in Appendix B.
Theorem 1 implies that for any Mallows model characterized by $d(\cdot,\cdot)$ and $\phi(x)$,
we can construct a corresponding MallowsPO objective.
Building on this result, we introduce two instances of MallowsPO,
derived from the two most widely used Mallows models.

###### Corollary 2 (MallowsPO- $\theta$ ) .

Suppose the preference distribution is Mallows-$\theta$, i.e., $d(\mu_{1},\mu_{2})=\sum_{i=1}^{n}(\mu_{1}(i)-\mu_{2}(i))^{2}$. Given $\phi(x)$, we have

$$ $p^{*}\left(y_{1}\succ y_{2}\mid x\right)=1/(1+\exp{(-(-2\log\phi(x))\cdot\left (\mu_{0}\left(y_{2}\mid x\right)-\mu_{0}\left(y_{1}\mid x\right)\right))}):=g_ {\theta,\phi(x)}(s),$ (8) $$

where $\log\phi(x)\in(-\infty,0)$ and $s:=\mu_{0}\left(y_{2}\mid x\right)-\mu_{0}\left(y_{1}\mid x\right)$.
By Theorem 1, the MallowsPO-$\theta$ objective
is in the form of (7) with $g_{d,\phi(x)}(\cdot)=g_{\theta,\phi(x)}(\cdot)$ in (8).

(8) shows that in the Mallows-$\theta$ ranking, the representation of $p^{*}(\cdot\mid x)$ resembles a sigmoid function but differs in that it is scaled by the term $-2\log\phi(x)$, which reflects the distribution’s dispersion. Compared to BT, where the distribution is exactly in form of sigmoid, Mallows-$\theta$ allows for greater flexibility in controlling the spread of the distribution function. This is particularly important in language modeling, as the concept of dispersion provides insight into how diverse people’s preferences are for different completions/responses. As $\phi(x)\to 0$, $p^{*}(\cdot\mid x)$ is getting closer to a step function (i.e., Dirac delta) (as shown in Fig. 3), corresponding to cases where the prompt $x$ has a clear, standard answer. Conversely, as $\phi(x)\to 1$, $p^{*}(\cdot\mid x)$ approaches a constant value of 0.5 (i.e. uniform), indicating that any answer to the prompt $x$ is equally reasonable.

In terms of the objective, by setting $-2\log\phi(x)=1$, we recover the DPO in (4).
When comparing the objective of the DPO
with that of
MallowsPO-$\theta$,
the key difference is the presence of an extra term $-\log\phi(x)$,
which reflects the dispersion of the prompt $x$.
Thus, MallowsPO-$\theta$ can be viewed as a generalized version of DPO that incorporates prompt dispersion. To see the effect of this additional term:
When dispersion is high ($\phi(x)\approx 1$), the term $-2\log\phi(x)$ approaches 0, reducing the weight on preference pairs;
when dispersion $\phi(x)$ decreases, $-2\log\phi(x)$ increases, assigning more weight to preference pairs.

###### Corollary 3 (MallowsPO- $\phi$ ) .

Suppose the preference distribution is Mallows-$\phi$, i.e., $d(\mu_{1},\mu_{2})=\operatorname{inv}\left(\mu_{1}\circ\mu_{2}^{-1}\right)$. Given $\phi(x)$, we have

$$ $p^{*}\left(y_{1}\succ y_{2}\mid x\right)=g_{\phi,\phi(x)}(\mu_{0}(y_{2}\mid x) -\mu_{0}(y_{1}\mid x)),$ (9) $$

where $\log\phi(x)\in(-\infty,0)$, and $g_{\phi,\phi(x)}(s):=\frac{1-\operatorname{sgn}(s)}{2}+\operatorname{sgn}(s)
\left(\frac{|s|+1}{1-\phi(x)^{|s|+1}}-\frac{|s|}{1-\phi(x)^{|s|}}\right)$.
By Theorem 1, the MallowsPO-$\phi$ objective is in the form of (7) with $g_{d,\phi(x)}(\cdot)=g_{\phi,\phi(x)}(\cdot)$ in (9).

By specifying the underlying ranking to Mallows-$\phi$,
we get a different link function $g_{\phi,\phi(x)}$, which also contains the dispersion index $\phi(x)$, resulting in a new preference optimization objective.

Figure: Figure 2. Distribution plot.
Refer to caption: extracted/5896174/Figures/link_function.png

###### Theorem 1 (MallowsPO) .

###### Corollary 2 (MallowsPO- θ 𝜃 \theta italic_θ ) .

###### Corollary 3 (MallowsPO- ϕ italic-ϕ \phi italic_ϕ ) .

### 3.3. How to choose the dispersion index ϕ ⁢ ( x ) italic-ϕ 𝑥 \phi(x) italic_ϕ ( italic_x ) ?

As the dispersion index $\phi(x)\in(0,1]$ is unknown, computation or estimation of it requires learning via neural networks or other algorithms [].
Here, however, we propose a more direct approach to estimate $\phi(x)$ without any pretraining or learning.
The idea is to qualitatively relate $\phi(x)$ to the empirical output distribution of the pretrained model,
on which we propose an ‘easy-to-compute’ proxy to the negative log dispersion $-\log(\phi(x))$ for each prompt $x$.

Suppose the preference follows the Mallows-$\phi$ model.
There are two extreme cases:
When $-\log(\phi(x))\rightarrow\infty$, we have:
$p^{*}\left(y_{1}\succ y_{2}\mid x\right)=\begin{cases}1,&\text{ if }\mu_{0}(y_
{1}\mid x)<\mu_{0}(y_{2}\mid x),\\
0,&\text{ if }\mu_{0}(y_{1}\mid x)>\mu_{0}(y_{2}\mid x).\end{cases}$
Thus, the probability distribution of the next token will concentrate on a point mass.
When $-\log(\phi(x))\rightarrow 0$, we have:
$p^{*}\left(y_{1}\succ y_{2}\mid x\right)=\frac{1}{2},$
so the next token will be uniformly distributed.

The above observation motivates us to use Shannon’s entropy $H(\cdot)$.
Note that $H(X)=0$ when $X$ is a point mass,
and $H(X)=\log n$ when $X$ is uniform on $n$ points.
Thus, for a given constant $\phi^{*}>0$,
we propose:

$$ $-\phi^{*}\log\left(H(\pi(\cdot\mid x))/\log n\right),$ (10) $$

as a proxy to $-\log\phi(x)$,
where $\pi(\cdot\mid x)$ can be either the pretrained LM model $\pi^{\mathrm{PRE}}$ or the SFT model $\pi^{\mathrm{SFT}}$. Furthermore, we approximate the entropy term in (10)
via a realization of a sequence of $N=\max(|Y^{w}|,|Y^{l}|)$ tokens $\{Y_{i}^{w},Y_{i}^{l}\}_{i=1,\ldots,N}$ given the prompt $X$:

$$ $H(\pi(\cdot\mid X))\approx\frac{1}{2}\sum_{i=1}^{N-1}\left[H(Y_{i+1}\mid Y_{i} =Y^{w}_{i})+H(Y_{i+1}\mid Y_{i}=Y^{l}_{i})\right],$ (11) $$

which can be easily computed by the logits of the model given the output data. This is also related to the predictive entropy [, ] of the next-token predictions.

Accuracy of the estimate. To validate our proposed estimate (10), we consider the similar ‘bandit’ setup in []. We draw rankings $\mu$ from a Mallows-$\phi$ ranking model, and then obtain a pair of winning/losing actions by choosing the highest/lowest ranked elements in the ranking $\mu$.
We plot $-\phi^{*}\cdot\log\left(\frac{H\left(Y^{w}\right)+H\left(Y^{l}\right)}{2\log n
}\right)$, given the preferences data ($x$-axis) and the true dispersion ($y$-axis) that these data are generated from. Figure 3 shows that our proposed estimator indeed matches the true dispersion, which heuristically reflects the accuracy of our estimate.

### 3.4. Unify Mallows- θ 𝜃 \theta italic_θ and Mallows- ϕ italic-ϕ \phi italic_ϕ for computation

Note that the link function $g_{\phi,\phi(x)}$ in MallowsPO-$\phi$ is not continuous (or smooth) at $x=0$, with

$$ $g^{\prime}_{\phi,\phi(x)}(s)=\begin{cases}\frac{1}{1-\phi(x)^{s+1}}+\frac{(s+1 )\phi^{s+1}\log\phi(x)}{(1-\phi(x)^{s+1})^{2}}-\frac{1}{1-\phi(x)^{s}}-\frac{s \phi(x)^{s}\log\phi(x)}{(1-\phi(x)^{s})^{2}},\quad&s>0,\\ \frac{1}{1-\phi(x)^{1-s}}+\frac{(1-s)\phi(x)^{1-s}\log\phi(x)}{(1-\phi(x)^{1-s })^{2}}-\frac{1}{1-\phi(x)^{-s}}+\frac{s\phi(x)^{-s}\log\phi(x)}{(1-\phi(x)^{- s})^{2}},\quad&s<0.\end{cases}$ (12) $$

For computational purposes, we propose two smooth approximations to $g_{\phi,\phi(x)}$.

(i) Sigmoid approximation: Since
$g_{\phi,\phi(x)}(1)=\frac{1}{1+\phi(x)}$,
we approximate $g_{\phi,\phi(x)}(s)$ by $\sigma_{x}(s):=\sigma(-s\log\phi(x))$
so that $\sigma_{x}(1)=g_{\phi,\phi(x)}(1)$.
See Figure 6 for an illustration of this approximation.
With this approximation,
MallowsPO-$\phi$ and MallowsPO-$\theta$ yield the same objective
with different $\beta$’s (up to a factor of $2$).
Thus, MallowsPO-$\theta$ is just
MallowsPO-$\phi$ with sigmoid approximation.

(ii) Polynomial fitting:
We use a polynomial of form $P(x)=a_{3}x^{3}+a_{1}x+a_{0}$ to approximate $g_{\phi,\phi(x)}$ on $[-\epsilon,\epsilon]$, with $\epsilon$ being a hyperparameter.
We choose $\epsilon$ to be either fixed, e.g., $\epsilon=0.1$;
or $\epsilon=-2\log\phi(x)$ (e.g., $\epsilon\approx 1.4$ for $\phi(x)=0.5$).
See Figures 6–6 for an illustration.

Figure: Figure 4. Sigmoid approximation
Refer to caption: extracted/5896174/Figures/Mallows_link_approximation/sigmoid.png

## 4. Perspectives on MallowsPO

In this section, we provide several alternative perspectives on
MallowsPO in Corollary 2–3,
with the proofs given in Appendix B.
We say a DPO is directed by $g(\cdot)$ if the preference distribution can be expressed as
$p^{*}\left(y_{1}\succ y_{2}\mid x\right)=g\left(r^{*}(x,y_{1})-r^{*}\left(x,y_
{2}\right)\right)$
for some reward function $r^{*}$.
Thus, Bradley-Terry based DPO is directed by the sigmoid function $\sigma(\cdot)$.

### 4.1. Dispersion weighted objectives

The following results show that MallowsPO can be viewed as a DPO with either the reward or the KL-regularizer weighted by the dispersion index.

###### Proposition 4 (MallowsPO- $\theta$ as dispersion weighted DPO) .

Let $c(x)=-2\log\phi(x)$. Then, MallowsPO-$\theta$
is the same as a DPO with either the reward weighted by $c(x)$ or the KL-regularizer weighted by $\beta c(x)$:

$$ $\max_{\pi}\mathbb{E}_{x\sim\mathcal{D}}\left[\mathbb{E}_{y\sim\pi_{\theta}(y \mid x)}\left[c(x)^{-1}r^{*}(x,y)\right]-\beta\mathrm{KL}\left(\pi(\cdot\mid x )\|\pi_{\mathrm{ref}}(\cdot\mid x)\right)\right],$ $$

or

$$ $\max_{\pi}\mathbb{E}_{x\sim\mathcal{D}}\left[\mathbb{E}_{y\sim\pi_{\theta}(y \mid x)}\left[r^{*}(x,y)\right]-\beta c(x)\mathrm{KL}\left(\pi(\cdot\mid x)\| \pi_{\mathrm{ref}}(\cdot\mid x)\right)\right].$ $$

###### Proposition 5 (MallowsPO- $\phi$ as dispersion weighted DPO) .

Denoting $\phi(x)=t$ in Corollary 3 yields

$$ $g(s):=\frac{1-\operatorname{sgn}(s)}{2}+\operatorname{sgn}(s)\left(\frac{|s|+1 }{1-t^{|s|+1}}-\frac{|s|}{1-t^{|s|}}\right).$ (13) $$

Let $c(x)=-2\log\phi(x)$ as before. Then, MallowsPO-$\phi$
is the same as a DPO directed by $g(\cdot)$ as in (13), and with either the reward weighted by $c(x)$ or the KL-regularizer weighted by $\beta c(x)$:

$$ $\max_{\pi}\mathbb{E}_{x\sim\mathcal{D}}\left[\mathbb{E}_{y\sim\pi_{\theta}(y \mid x)}\left[c(x)^{-1}r^{*}(x,y)\right]-\beta\mathrm{KL}\left(\pi(\cdot\mid x )\|\pi_{\mathrm{ref}}(\cdot\mid x)\right)\right],$ $$

or

$$ $\max_{\pi}\mathbb{E}_{x\sim\mathcal{D}}\left[\mathbb{E}_{y\sim\pi_{\theta}(y \mid x)}\left[r^{*}(x,y)\right]-\beta c(x)\mathrm{KL}\left(\pi(\cdot\mid x)\| \pi_{\mathrm{ref}}(\cdot\mid x)\right)\right].$ $$

###### Proposition 4 (MallowsPO- θ 𝜃 \theta italic_θ as dispersion weighted DPO) .

###### Proposition 5 (MallowsPO- ϕ italic-ϕ \phi italic_ϕ as dispersion weighted DPO) .

### 4.2. Connection to Ψ Ψ \Psi roman_Ψ PO

The objective of $\Psi$PO [] is
$\max_{\pi}\mathbb{E}_{x\sim\mathcal{D}}[\mathbb{E}_{y\sim\pi(\cdot\mid x),\,y^
{\prime}\sim\tilde{\pi}(\cdot\mid x)}\left[\Psi\left(p^{*}(y\succ y^{\prime}
\mid x)\right)\right]-\beta\mathrm{KL}(\pi(\cdot\mid x)\|\pi_{\mathrm{ref}}(
\cdot\mid x))]$,
where $\Psi:[0,1]\to\mathbb{R}$ is a non-decreasing function,
and $\tilde{\pi}(\cdot\mid x)$ is an arbitrary policy (referred to as the behavior policy).
It is readily verified that setting $\Psi(s)=\log\left(\frac{s}{1-s}\right)$ reduces $\Psi$PO to the Bradley-Terry based DPO.

Roughly speaking, the function $\Psi$ can be viewed as the inverse of the link function, $\Psi(\sigma(s))=\log\left(\frac{\sigma(s)}{1-\sigma(s)}\right)=s$. The question is whether MallowsPO can be reduced to $\Psi$PO for some suitably chosen $\Psi(\cdot)$.
Assume such a function exists, which we denote as $\Psi^{M}(\cdot)$.
From the Mallows-$\phi$ model in Corollary 3, we have

$$ $\displaystyle\mathbb{E}_{y_{2}\sim\bar{\pi}(\cdot\mid x)}\left[\Psi^{M}\left(p ^{*}\left(y_{1}\succ y_{2}\mid x\right)\right)\right]$ $\displaystyle=\mathbb{E}_{y_{2}\sim\bar{\pi}(\cdot\mid x)}\left[\Psi^{M}\left( g_{x}(r(x,y_{1})-r(x,y_{2}))\right)\right]$ (14) $\displaystyle\neq r(x,y_{1})-\mathbb{E}_{y_{2}\sim\bar{\pi}(\cdot\mid x)}\left [r\left(x,y_{2}\right)\right],$ $$

i.e., for any $\Psi^{M}(\cdot)$ that is prompt-independent,
MallowsPO cannot be an instance of $\Psi$PO. This calls for extending $\Psi$PO to take into account prompt dispersion.

Generalized $\Psi$PO. Let $\tilde{\Psi}(x,p)$ depend on the prompt $x$ as well as the preference distribution $p$. The
generalized $\Psi$PO takes the form:

$$ $\max_{\pi}\mathbb{E}_{x\sim\mathcal{D}}\left[\mathbb{E}_{y\sim\pi_{\theta}( \cdot\mid x),\,y^{\prime}\sim\tilde{\pi}(\cdot\mid x)}\left[\tilde{\Psi}\left( x,p^{*}(y\succ y^{\prime}\mid x)\right)\right]-\beta\mathrm{KL}\left(\pi(\cdot \mid x)\|\pi_{\mathrm{ref}}(\cdot\mid x)\right)\right].$ (15) $$

A special instance is when $\tilde{\Psi}(x,p)=f(x)\Psi(p)$ is separable:

$$ $\max_{\pi}\mathbb{E}_{x\sim\mathcal{D}}\left[\mathbb{E}_{y\sim\pi_{\theta}( \cdot\mid x),\,y^{\prime}\sim\tilde{\pi}(\cdot\mid x)}\left[f(x)\Psi\left(p^{* }(y\succ y^{\prime}\mid x)\right)\right]-\beta\mathrm{KL}\left(\pi(\cdot\mid x )\|\pi_{\mathrm{ref}}(\cdot\mid x)\right)\right].$ (16) $$

###### Theorem 6 (MallowsPO as generalized $\Psi$ PO) .

(i) MallowsPO-$\theta$
(directed by $\sigma(\cdot)$) can be reduced to
the generalized $\Psi$PO in (16) with
$\Psi(s)=\log\left(\frac{s}{1-s}\right)$ and $f(x)=-\frac{1}{2\log\phi(x)}$.

(ii) MallowsPO-$\phi$
(directed by $g(\cdot)$) can be reduced to the generalized $\Psi$PO in (16) with $\Psi(s)=g^{-1}(s)$ and $f(x)=-\frac{1}{\log\phi(x)}$.

###### Theorem 6 (MallowsPO as generalized Ψ Ψ \Psi roman_Ψ PO) .

## 5. Experiments

In this section, we evaluate the capability of our proposed MallowsPO to learn the preferences in comparison with DPO.
First, we use the preferences dataset of IMDB [] datasets
and Anthropic Helpful and Harmless dialogue [] dataset to provide evidence that human preferences may be diversed.
Next, we consider a synthetic bandit problem to demonstrate the effectiveness of our proposed MallowsPO-$\phi$, even without prompt dispersions.
We further conduct experiments on tasks such as conditional generation (IMDB) and dialogue (Anthropic HH, UltraFeedback). Our findings show that MallowsPO outperforms DPO with an evident margin,
both for in-distribution performance and out-of-distribution generalization capability.

### 5.1. Evidence of preference dispersion

A first natural question is: are human preferences dispersed?
To verify this key motivation for our work, we plot the distribution of the dispersion estimators given the SFT model and pairwise preferences.
Recall from Section 3 that the dispersion estimator is:

$$ $-\phi^{*}\log\left(\frac{1}{2\log n}\sum_{i=1}^{N-1}\left[H(Y_{i+1}\mid Y_{i}= y^{w}_{i})+H(Y_{i+1}\mid Y_{i}=y^{l}_{i})\right]\right),$ (17) $$

and we take the hyperparameter $\phi^{*}>0$ such that the empirical mean is equal to 1 (as in DPO), so we do not need to tune this scaling constant.
Note that this scaling results in our final estimate acting as a relative dispersion level compared to the whole dataset:
when a prompt’s dispersion parameter is large, i.e., close to 1, (17) will be smaller than 1.
In contrast, (17) will be much larger than 1 if the prompt’s dispersion parameter is close to 0 (or there is less disagreement about the answer to the prompt).
We formally call this neg log dispersion estimate throughout the rest of the paper.

Figure: (a) IMDB preference dispersion distribution.
Refer to caption: extracted/5896174/Figures/imdb_dispersion_distribution.png

We find that for the task of conditional generation such as IMDB, its human preferences (Fig. 5.1) are not quite diverse: the neg log dispersion estimates are located near $1$, and almost all the estimates range from $0.8$ to $1.2$.
However, for tasks such as single dialogue, Fig. 5.1 shows that human preferences are relatively more dispersed: the distribution is both skewed and of higher variance.
As shown in Figure 1, prompts with high dispersion or those that will lead to human disagreement on preferences indeed have a neg log dispersion estimate smaller than $1$, while those with low dispersion have the neg log dispersion estimate located at the right-hand side (larger than 1).
More examples with low/high dispersion are provided in Appendix A.

### 5.2. MallowsPO- ϕ italic-ϕ \phi italic_ϕ mitigates reward collapse

We study MallowsPO in a synthetic bandit experiment
with no contextual information $x$,
and compare it with DPO to test their ability to produce diversified policies and avoid reward collapse.
Moreover, we operate under the constraint of having a limited number of observations.
There are two reasons to explore this setting.
First, the bandit facilitates a clear analysis without introducing the complication of the context $x$.
Second, the limited data availability tests the ability of the approaches to produce diversified policies and avoid reward collapse.

Concretely, we consider five arms, each associated with a random reward drawn from a probability distribution.
Preference between any two picked arms is determined by the random reward realizations, with larger reward being preferred.
In the experiment, we collect $16$ pairwise observations,
and evaluate the performance of different approaches by computing the efficient frontiers
(1) across different parameters $\beta$, and (2) across different epochs.
The details are provided in Appendix B.1.

Figure 7 displays the efficient frontiers for MallowsPO-$\phi$ and DPO.
Figure 7(a) shows that MallowsPO-$\phi$ has a more efficient frontier:
(1) With the same KL divergence, MallowsPO-$\phi$ achieves a higher reward, especially when $\beta$ is small.
(2) Over all possible $\beta$, the best reward that MallowsPO-$\phi$ achieves is higher than that of DPO.
(3) MallowsPO-$\phi$ avoids reward collapse as $\beta$ gets smaller.
That is, MallowsPO-$\phi$ assigns a certain probability to the potentially good arms,
as opposed to DPO that tends to assign only to the “best” arm (see Figure 8).
Figure 7(b) shows that during the training process,
MallowsPO-$\phi$ leads to the policies that have both high rewards and small KL divergence.

Figure: (a)
Refer to caption: extracted/5896174/Figures/bandit_varied_betas.png

Figure: Figure 8. Training curves of MallowsPO-$\phi$ and DPO for $\beta=0.05$ and $\beta=0.1$.
Refer to caption: extracted/5896174/Figures/bandit_training.png

### 5.3. MallowsPO yields better tradeoff between accuracy and regularization

We conduct the conditional generation for IMDB dataset.
In this task, $x$ is a prefix of movie review, and the LM is to generate output $y$ with positive sentiment.
Following the setting in [],
we first fine-tune GPT-2-large on the training split of IMDB datasets until convergence to get the SFT model.
Next, we use the pairwise preference data from [] to fine-tune the SFT model by DPO and MallowsPO.

Figure 7(c) displays the efficient frontiers (during the training process) for DPO and MallowsPO.
We observe that the performances of MallowsPO-$\theta$ and DPO are close. The similarity is likely due to the nature of the task – controllable comment generation, which is expected to exhibit smaller dispersion, as evidenced in Figure 5.1.
MallowsPO-$\phi$ outperforms both, achieving the same accuracy (evaluated by the reward model) at a smaller KL divergence to the SFT model/policy.

### 5.4. MallowsPO enhances both in/out-of distribution performances

We compare the performances of MallowsPO and DPO in terms of the win rate evaluated by GPT4, and generalization capability on the out-of-distribution datasets.
In the experiment, we choose $\beta$ to be $0.1$ and $0.5$
since it has been observed [] that increased $\beta$ value leads to a drop both in performance and per-input diversity of RLHF and DPO. Results are shown in Figure 9.

For the in-distribution test, we first fine-tune a pretrained Pythia-2.8B model on the training set of Anthropic HH dataset using MallowsPO and DPO,
and then evaluate the responses on a subset of its test split, generated by these fine-tuned models.
GPT-4 serves as the evaluator, and compares pairs of responses.
We observe that MallowsPO has an edge over DPO.
For the out-of-distribution test,
we apply the models, fine-tuned on the train split of the Antropic HH dataset, to other datasets with different input distributions.
The H4 Stack Exchange Preferences Dataset (SE) []
and Stanford Human Preferences (SHP) []
are used for evaluation.
The advantage of dispersion on generalization becomes apparent,
as MallowsPO shows more improvement compared to the in-distribution case.

We also compare MallowsPO-$\theta$ with DPO in fine-tuning the Pythia-2.8B model,
with ArmoRM [] serving as the evaluator.
The result indicates that MallowsPO-$\theta$ achieves consistently higher win rates than DPO across all cases, with an impressive win rate of around 70% in the in-distribution test.
Details are provided in Appendix D.1.

Figure: Figure 9. Win rates computed by GPT-4 for responses on both the in- and out-of distribution dataset.
Refer to caption: extracted/5896174/Figures/LM_results.png

### 5.5. MallowsPO enhances SOTA Llama3-8B-Instruct Models

We illustrate the scalability of our method through experiments on fine-tuning Llama3-8B-Instruct Model on UltraFeedback Dataset. We follow the same setup in RLHFlow [] and SimPO [],
as we generate five answers from Llama3-8B-Instruct for each prompt in UltraFeedback, rank them with scores evaluated by ArmoRM [],
and choose the best/worst one as winning/losing answer to form the preference datasets. For a fair comparison, we compare MallowsPO with DPO, using different hyperparameters: $\beta$ and learning rate $lr$
for the task of Alpaca Eval V2.
The results are shown in Appendix D.2:

**Table 1. Win rate comparison between SOTA fine-tuning methods and their enhanced versions using our MallowsPO as a plugin with optimized $\beta$ and $lr$. Standard deviations are right next to the reported metric.**
| $\beta$ | $lr$ | LC Win Rate | Win Rate |  |  |
| --- | --- | --- | --- | --- | --- |
|  |  | DPO | MallowsPO | DPO | MallowsPO |
| 0.01 | $5e^{-7}$ | 42.55% (0.79) | 43.10% (0.77) | 42.02% (1.53) | 43.02% (1.57) |
|  |  | IPO | MallowsIPO | IPO | MallowsIPO |
| 0.005 | $1e^{-6}$ | 43.38% (0.84) | 44.73% (0.87) | 43.52% (1.45) | 44.87% (1.46) |
|  |  | SimPO | MallowsSimPO | SimPO | MallowsSimPO |
| 10 | $1e^{-6}$ | 50.04% (0.77) | 51.89% (0.81) | 42.11% (1.46) | 43.76% (1.47) |

When $\beta=0.01$ and $lr=5e^{-7}$, for which DPO and MallowsPO both achieve the best performance, we used 10 random seeds for generation to show the statistical significance: MallowsPO outperforms DPO both in mean or the best performance with different random seeds, and also has smaller variance (see Figure 10).

We also adapt the idea of dispersion index (contextual scaling) in MallowsPO to IPO and SimPO, leading to MallowsIPO and MallowsSimPO.
As shown in Table 1, both MallowsIPO and MallowsSimPO beat their vanilla counterparts
(using the hyperparameters proposed in [] and [] respectively).

Figure: Figure 10. Win rates computed by GPT-4 for responses on Alpaca Eval V2.
Refer to caption: extracted/5896174/Figures/dpo-vs-mallows.png

## 6. Conclusion

We have developed in this paper a novel approach, the MallowsPO, to fine-tune LLM.
A distinct feature of this approach is a dispersion index,
which naturally captures the dispersion of human preference to prompts, and can be systematically incorporated into the reward function as a weight factor, thus ushering in a new class of dispersion-weighted DPO models.
We demonstrate empirically how MallowsPO achieves improved performance in a broad array of benchmark tasks, including synthetic bandit selection, controllable generation, and dialogues. The effectiveness holds for both small and large representative models, such as Pythia 2.8B and Llama3-8B-Instruct.

There are a few issues that we have yet to address in this study, for instance, to explore why MallowsPO outperforms DPO, how the dispersion index contributes to performance improvement, what guidelines to follow to set the $\beta$ value, and whether Mallows can be combined and benefit other DPO variants. These will be pursued in our future work.

Acknowledgement:
Hanyang Zhao and Wenpin Tang are supported by NSF grant DMS-2206038,
and by a start-up grant at Columbia University.
The works of Haoxian Chen, Hanyang Zhao, Henry Lam
and David Yao are part of a Columbia-CityU/HK collaborative project that is supported by InnotHK
Initiative, The Government of the HKSAR and the AIFT Lab.
Additionally, Haoxian Chen is supported by the Amazon CAIT fellowship.
Henry Lam and Wenpin Tang receive support from the Columbia Innovation Hub grant.

## Appendix

## Appendix A Prompts with low/high negative log dispersion estimate

### A.1. Examples of prompts with low negative log dispersion estimate

See Table 2 for the examples.

**Table 2. Prompts with low neg log dispersion or high dispersion, i.e. human are likely to have more disagreement about the answers.**
| Human: | I need some ideas on repurposing old magazines. |
| --- | --- |
| Assistant: | How about using a magazine as a dustcover for a book? |
| Human: | It will take some effort but it could work. What else can you think of? |
| Human: | Why do you think western society idolizes youth so much, as opposed to asian society, which idolizes elders? |
| Assistant: | Humans throughout history have consistently held important values and traits in high esteem, including trustworthiness, compassion, intelligence, and wisdom. Given that younger people have the potential to develop these traits while older people have already developed them, it makes sense that older people’s virtues are embraced in some cultures while younger people’s virtues are embraced in other cultures. It’s important to remember that cultures throughout the world are complex and heterogeneous, so it’s possible for some to prioritize older people yet also to respect some of the important qualities and values held by younger people. The value we place on older people will vary inversely with the value we place on young people, and vice versa, which helps explain why different cultures come to elevate one or the other. |
| Human: | That’s a fair assessment, but I’m looking for more specific reasons. Let’s refocus on just the west. Why are we so youth-focused? |
| Human: | what are some cool ocean activities? |
| Human: | Why do almost all cartoon characters tend to only have 4 fingers? Is it some unwritten rule in the animation industry to do that? |
| Human: | Can you provide 3 bullet points for how to teach a child how to set the dinner table? |
| Human: | I’d like to become a world famous writer. Any suggestions on how to get there? |
| Human: | My friend is a University Lecturer with a Ph.D. in Business. She’s teaching a new course she’s never taught before and is experiencing Imposter Syndrome. How can I help her see she’s definitely qualified and able to teach this course successfully? |

### A.2. Examples of prompts with high negative log dispersion estimate

See Table 3 for the examples.

**Table 3. Prompts with high neg log dispersion or low dispersion, i.e. human are likely to have less disagreement about the answers.**
| Human: | Have you heard of Summer Palace, a historical site in China? |
| --- | --- |
| Assistant: | Yes, Summer Palace is a well-known historical site in China. It was originally built in the eighteenth century as a summer retreat for the royal family, and is renowned for its beautiful gardens and temples. Summer Palace is now a popular tourist destination, attracting visitors with its historic architecture and natural beauty. There are also many tours and exhibits at the site, offering visitors an in-depth look at its history. |
| Human: | In which city is it located? |
| Human: | Translate the following into slang: Good morning, how are you today? I have to go to work later. |
| Assistant: | Hey, how’s it going? I’m doing pretty good today. I have to head to work later. |
| Human: | Now capitalize every word of that response. |
| Human: | Please tell me about The Battle of Waterloo. |
| Assistant: | In 1815, Napoleon led his troops and won against the British and Prussian armies. |
| Human: | No. Napoleon lost the battle. |
| Assistant: | OK, in 1815, Napoleon led his troops and lost against the British and Prussian armies. |
| Human: | Was the battle in Belgium? |
| Human: | Are there any NASA space missions going on currently? |
| Assistant: | As of now, no. |
| Human: | Are you sure? |
| Assistant: | NASA doesn’t have any plans for missions in the immediate future. However, there are several missions on the books that might eventually get funded. |
| Human: | Thank you for this information. |

## Appendix B Proofs

Proof of Theorem 1.

To obtain (7), recall that given $x$, the probability of output $y_{1}$ being preferred than $y_{2}$ is:

$$ $\displaystyle p^{*}\left(y_{1}\succ y_{2}\mid x\right)$ $\displaystyle=\mathbb{P}\left(\mu(y_{1}\mid x)<\mu\left(y_{2}\mid x\right)\right)$ (18) $\displaystyle=g_{d,\phi(x)}(-\mu_{0}(y_{1}\mid x)-(-\mu_{0}(y_{2}\mid x)))$ (19) $$

Therefore, the modeling of such a ranking distribution requires the conditional central ranking $\mu_{0}(\cdot\mid x)$. However. since finding $\mu_{0}(\cdot\mid x)$ is computationally hard [], to tackle this challenge, we explore a different path. Recall that in RLHF, we optimize the following objective:

$$ $\max_{\pi}\mathbb{E}_{x\sim\mathcal{D}}\left[\mathbb{E}_{y\sim\pi_{\theta}(y \mid x)}\left[r^{*}(x,y)\right]-\beta\mathrm{KL}\left(\pi(\cdot\mid x)\|\pi_{ \mathrm{ref}}(\cdot\mid x)\right)\right],$ $$

where $r^{*}(x,y)$ is the true underlying reward. By letting $r^{*}(x,y)=-\mu_{0}(y\mid x)$, we now turn to optimize the following objective:

$$ $\max_{\pi}\mathbb{E}_{x\sim\mathcal{D}}\left[\mathbb{E}_{y\sim\pi_{\theta}(y \mid x)}\left[-\mu_{0}(y\mid x)\right]-\beta\mathrm{KL}\left(\pi(\cdot\mid x) \|\pi_{\mathrm{ref}}(\cdot\mid x)\right)\right].$ $$

As shown in section A.1 of [], the optimum of such a KL-constrained reward maximization objective has the form of

$$ $\pi_{\mu_{0}}(y\mid x)=\frac{1}{Z(x)}\pi_{\mathrm{ref}}(x)\exp\left(-\frac{\mu _{0}(y\mid x)}{\beta}\right),$ $$

where $Z(x)$ is the partition function to ensure $\pi_{\mu_{0}}(y\mid x)$ to be a probability distribution.
By moving terms, we have

$$ $-\mu(y\mid x)=\beta\log\frac{\pi_{\mu_{0}}(y\mid x)}{\pi_{\text{ref }}(y\mid x )}+\beta\log Z(x).$ (20) $$

Combining (19) and (20) gives us

$$ $p^{*}\left(y_{1}\succ y_{2}\mid x\right)=g_{d,\phi(x)}\left(\beta\log\frac{\pi _{\mu_{0}}(y_{1}\mid x)}{\pi_{\text{ref }}(y_{1}\mid x)}-\beta\log\frac{\pi_{ \mu_{0}}(y_{2}\mid x)}{\pi_{\text{ref }}(y_{2}\mid x)}\right).$ $$

To maximize the likelihood estimation, our objective becomes

$$ $\min_{\pi_{\mu_{0}}}-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\left[\log \left(g_{d,\phi(x)}\left(\beta\log\frac{\pi_{\mu_{0}}(y_{w}\mid x)}{\pi_{ \mathrm{ref}}(y_{w}\mid x)}-\beta\log\frac{\pi_{\mu_{0}}(y_{l}\mid x)}{\pi_{ \mathrm{ref}}(y_{l}\mid x)}\right)\right)\right].$ $$

$\blacksquare$

Proof of Corollary 2.

[] showed that

$$ $\mathbb{P}\left(\mu(y_{1}\mid x)<\mu\left(y_{2}\mid x\right)\right)=1/(1+\exp{ (-2\log\phi(x)\left(\mu_{0}\left(y_{1}\mid x\right)-\mu_{0}\left(y_{2}\mid x \right)\right))}).$ (21) $$

A direction application to Theorem 1 derives the result.

$\blacksquare$

Proof of Corollary 3.

For the Mallows-$\phi$ model, it was shown in [] (see also [, ]):

$$ $\displaystyle\hskip 11.38109pt\mathbb{P}\left(\mu(y_{1}\mid x)<\mu\left(y_{2} \mid x\right)\right)$ (22) $\displaystyle=\begin{cases}\frac{\mu_{0}\left(y_{2}\mid x\right)-\mu_{0}\left( y_{1}\mid x\right)+1}{1-\phi(x)^{\mu_{0}\left(y_{2}\mid x\right)-\mu_{0}\left( y_{1}\mid x\right)+1}}-\frac{\mu_{0}\left(y_{2}\mid x\right)-\mu_{0}\left(y_{1 }\mid x\right)}{1-\phi(x)^{\mu_{0}\left(y_{2}\mid x\right)-\mu_{0}\left(y_{1} \mid x\right)}},\quad&\mu_{0}\left(y_{2}\mid x\right)-\mu_{0}\left(y_{1}\mid x \right)>0,\\ 1-\frac{\mu_{0}\left(y_{1}\mid x\right)-\mu_{0}\left(y_{2}\mid x\right)+1}{1- \phi(x)^{\mu_{0}\left(y_{1}\mid x\right)-\mu_{0}\left(y_{2}\mid x\right)+1}}- \frac{\mu_{0}\left(y_{2}\mid x\right)-\mu_{0}\left(y_{1}\mid x\right)}{1-\phi( x)^{\mu_{0}\left(y_{1}\mid x\right)-\mu_{0}\left(y_{2}\mid x\right)}},\quad& \mu_{0}\left(y_{2}\mid x\right)-\mu_{0}\left(y_{1}\mid x\right)<0,\end{cases}$ (23) $$

A direction application to Theorem 1 derives the result.

$\blacksquare$

Proof of Proposition 4.

The proof follows from the derivation of the equivalence between RLHF and DPO, as now the optimal policy satisfies

$$ $c(x)^{-1}r(x,y)=\beta\log\frac{\pi_{r}(y\mid x)}{\pi_{\text{ref }}(y\mid x)}+ \beta\log Z(x),$ $$

leading to the objective in Corollary 2.

$\blacksquare$

Proof of Theorem 6.

(i) With the Bradley-Terry connection as mentioned above, we have

$$ $\displaystyle\mathbb{E}_{y_{2}\sim\tilde{\pi}}\left[f(x)\Psi\left(p^{*}\left(y _{1}\succ y_{2}\mid x\right)\right)\right]$ $\displaystyle=\mathbb{E}_{y_{2}\sim\tilde{\pi}}\left[f(x)\Psi\left(\frac{e^{r( x,y_{1})}}{e^{r(x,y_{1})}+e^{r\left(x,y_{2}\right)}}\right)\right]$ (24) $\displaystyle=\mathbb{E}_{y_{2}\sim\tilde{\pi}}\left[f(x)\left(r(x,y_{1})-r \left(x,y_{2}\right)\right)\right]$ $\displaystyle=f(x)r(x,y_{1})-f(x)\mathbb{E}_{y_{2}\sim\tilde{\pi}}\left[r\left (x,y_{2}\right)\right],$ $$

which is a weighted reward of DPO, up to an additive constant.
It follows that the optimal policy of the generalized $\Psi$PO (16)
is the same as that of MallowsPO-$\theta$ by Theorem 4. The same argument also proves
(ii).

$\blacksquare$

## Appendix C Experimental Details

### C.1. Bandit Experiment

In the bandit experiment detailed in Section 5.2, we conduct two sub-experiments to compute the efficient frontiers using Mallow-$\phi$-DPO and DPO. The first sub-experiment varies the parameter $\beta$ while the second varies the epochs, with $\beta$’s to be a fixed set. For the first sub-experiment, we run each algorithm on a range of $\beta$ values required to compute the full efficient frontier, and for each $\beta$, we record the reward and $\text{KL}(\pi||\pi_{\text{ref}})$ of the average policy over the last 30 epochs to stabilize the results. As for the second sub-experiment, similar to the setup in [] and [], we execute an ensemble of training configurations for both MallowsPO and DPO, by adopting a range of different $\beta\in\{0.05,0.1,0.5,1.0\}$, and record the average reward and average $\text{KL}(\pi||\pi_{\text{ref}})$ among the four policies for every $100$ training steps. Given that we know the real reward distribution, all these quantities can be computed analytically.

In terms of the training details, we use all 16 data in a single batch and adopts SGD as the optimizer, with learning rate of 5e-3. To ensure convergence, we run the optimization for a large number of epochs, set to 500,000. For MallowsPO-$\phi$, we set $\phi$ to be 0.05.

**Table 4. Reward distributions of the five arms.**
| Arm 1 | Arm 2 | Arm 3 | Arm 4 | Arm 5 |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Reward | Prob. | Reward | Prob. | Reward | Prob. | Reward | Prob. | Reward | Prob. |
| 20 | 0.5 | 30 | 0.5 | 18 | 0.5 | 15 | 0.99 | 1 | 0.99 |
| 11 | 0.5 | 3 | 0.5 | 15 | 0.5 | 10 | 0.01 | 4 | 0.01 |

**Table 5. 16 pairs of sampled preference data.**
| Win | 3 | 2 | 2 | 1 | 3 | 1 | 1 | 1 | 4 | 2 | 2 | 2 | 1 | 3 | 3 | 4 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Lose | 5 | 5 | 5 | 2 | 5 | 5 | 4 | 5 | 5 | 4 | 1 | 5 | 3 | 5 | 4 | 2 |

### C.2. Controllable Generation Experiment Details

We follow the training setup in [], and first fine-tune GPT-2-large on the training split of IMDB datasets until convergence to get the SFT model. The next step is different from [] in that we directly utilize the (offline) preference dataset from [] instead of generating pairwise preferences from the trained SFT model, as in DPO. The rest is the same: we use the pairwise preference data to fine-tune the SFT model by either DPO or MallowsPO. The evaluation metric: accuracy is obtained from a prior sentiment classifier as the ground truth reward. By default, we use RMSprop optimizer with a learning rate of 1e-6, with a linear learning rate warmup from 0 to 1e-6 over the first 150 steps. The training batch size is 64.

### C.3. Language Modeling Experiment Details

We follow the training setup in []. By default, we use RMSprop optimizer with a learning rate of 1e-6, with a linear learning rate warmup from 0 to 1e-6 over the first 150 steps. The training batch size is 32.

#### C.3.1. GPT-4 Judgement Prompt

Response quality evaluation is completed by GPT-4. The prompt for instructing GPT-4 to evaluate which response is better is particularly important.
Thus, we use the fastchat package for GPT-4 evaluation, and we used their well-written pair-v2 judge prompt. The prompt is shown as follows:

Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below. You should choose the assistant that follows the user’s instructions and answers the user’s question better. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses. Begin your evaluation by comparing the two responses and provide a short explanation. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible. After providing your explanation, output your final verdict by strictly following this format: \‘‘ [[A]]\’’ if assistant A is better, \‘‘[[B]]\’’ if assistant B is better, and \‘‘[[C]]\’’ for a tie."

To ensure fairness and unbiasedness, for each pairwise input $(x,y_{1},y_{2})$, fastchat conducts two evaluation: first comparing $(y_{1},y_{2})$ and then comparing $(y_{2},y_{1})$. $y_{1}$ wins if and only if it wins both comparisons, or wins one comparison while the other is tied.
We compute win rate as follows:

$$ $\text{Win rate (Model A)}=\frac{\text{Number of samples where Model A wins}}{ \text{Total number of test samples}}+0.5\times\frac{\text{Number of tied samples}}{\text{Total number of test samples}}$ $$

## Appendix D Additional Results

### D.1. ArmoRM reward model

Figure: Figure 11. Win rates computed by ArmoRM for responses on both the in- and out-of distribution dataset. Experiments are repeated over 5 runs.
Refer to caption: extracted/5896174/Figures/armoRM.png

ArmoRM [] is a reward model for language modeling that utilizes multi-objective reward modeling and Mixture-of-Experts (MoE) techniques. Building on this, we also compare MallowsPO-$\theta$ with DPO in fine-tuning the Pythia-2.8B model on Anthropic-HH dataset, setting $\beta=0.1$, as per the setting in [], with ArmoRM serving as the evaluator. For evaluation, following the procedure in Section 5.4, we assess the models on the Anthropic-HH test set for in-distribution performance and on the SHP and SE datasets for out-of-distribution performance. The result indicates that model fine-tuned with MallowsPO-$\theta$ achieves consistently higher win rates than DPO across all cases, with an impressive win rate of around 70% in the in-distribution test.

### D.2. Ablation on β 𝛽 \beta italic_β and l ⁢ r 𝑙 𝑟 lr italic_l italic_r for MallowsPO and DPO

Including the setting in Section 5.5, we compare the performance of DPO and MallowsPO-$\theta$ in 6 configs by combining commonly used $\beta\in\{0.01,0.05,0.1\}$ and $lr\in\{e^{-6},5e^{-7}\}$. We find that in 5 out of 6 configs, MallowsPO-$\theta$ achieves better Length Controlled Win Rate and Win Rate.

**Table 6. Win rate comparison between DPO and MallowsPO-$\theta$ with different $\beta$ and $lr$.**
| $\beta$ | $lr$ | LC Win Rate | Win Rate |  |  |
| --- | --- | --- | --- | --- | --- |
| DPO | MallowsPO-$\theta$ | DPO | MallowsPO-$\theta$ |  |  |
| 0.1 | $e^{-6}$ | 37.14% | 37.37% | 35.77% | 35.83% |
| 0.05 | $e^{-6}$ | 40.08% | 41.08% | 40.31% | 41.02% |
| 0.01 | $e^{-6}$ | 27.42% | 29.65% | 27.03% | 28.92% |
| 0.1 | $5e^{-7}$ | 36.09% | 37.66% | 35.72% | 37.05% |
| 0.05 | $5e^{-7}$ | 40.56% | 39.75% | 39.89% | 39.49% |
| 0.01 | $5e^{-7}$ | 42.55% | 43.10% | 42.02% | 43.02% |

### D.3. Qualitative Examples

In this section, we present a series of examples for direct comparisons between MallowsPO variants and DPO, as shown in Tables 12–17. These tables showcase the qualitative examples of model responses both in-distribution inputs from the Anthropic-HH test set and out-of-distribution inputs from the SHP test set and the SE dataset respectively. SHP dataset covers questions/instructions in 18 different subject areas, from cooking to legal advice. SE dataset contains questions and answers from the Stack Overflow Data Dump.

To interpret, Tables 12–13 highlight that models fine-tuned with MallowsPO-$\theta$ and MallowsPO-$\phi$ demonstrate a strong grasp of contextual understanding and produce concise suggestions; however, the DPO fine-tuned model has the issue of generating repetitive and confusion response.

The examples in Tables 14–15 show these model’s responses to questions from daily life, testing their abilities in tutoring and knowledge retrieval. These results indicate that MallowsPO-$\theta$ and MallowsPO-$\phi$ fine-tuned models are not only knowledgeable and but also able to deliver deeper and further insights, rather than focusing solely on the length.

Finally, we also examine the models’ performance in troubleshooting, code generation and debugging tasks. Tables 16 reveals that MallowsPO-$\theta$ fine-tuned model does better in identifying the potential problems in the JavaScript codes provided and providing more relevant, actionable suggestions. Additionally, Table 17 shows how MallowsPO-$\phi$ complements the suggestions by DPO by offering code snippets to further support the user.

Figure: Figure 12. Sample responses to a prompt from the Anthropic-HH test set. GPT-4 chooses MallowsPO-$\theta$ over DPO. Both MallowsPO-$\theta$ and DPO are trained for $\beta=0.1$.
Refer to caption: extracted/5896174/Figures/qe1_hh.png

Figure: Figure 13. Sample responses to a prompt from the Anthropic-HH test set. GPT-4 chooses MallowsPO-$\phi$ over DPO. Both MallowsPO-$\phi$ and DPO are trained for $\beta=0.1$.
Refer to caption: extracted/5896174/Figures/qe2_hh.png

Figure: Figure 14. Sample responses to a prompt from the test set of the Stanford Human Preferences Dataset. GPT-4 chooses MallowsPO-$\theta$ over DPO. Both MallowsPO-$\theta$ and DPO are trained for $\beta=0.1$.
Refer to caption: extracted/5896174/Figures/qe3_shp.png

Figure: Figure 15. Sample responses to a prompt from the test set of the Stanford Human Preferences Dataset. GPT-4 chooses MallowsPO-$\theta$ over DPO. Both MallowsPO-$\theta$ and DPO are trained for $\beta=0.5$.
Refer to caption: extracted/5896174/Figures/qe4_shp.png

Figure: Figure 16. Sample responses to a prompt from the test set of the H4 Stack Exchange Preferences Dataset. GPT-4 chooses MallowsPO-$\theta$ over DPO. Both MallowsPO-$\theta$ and DPO are trained for $\beta=0.1$ on Anthropic-HH dataset.
Refer to caption: extracted/5896174/Figures/qe5_se.png

Figure: Figure 17. Sample responses to a prompt from the test set of the H4 Stack Exchange Preferences Dataset. GPT-4 chooses MallowsPO-$\phi$ over DPO. Both MallowsPO-$\phi$ and DPO are trained for $\beta=0.5$ on Anthropic-HH dataset.
Refer to caption: extracted/5896174/Figures/qe6_se.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: None
soundness: 3
strengths: 1. This paper is clearly written and well structured. The authors provide sufficient analytical and empirical comparisons with previous methods.

2. The contribution to the methodology for fine-tuning large language models is significant. This work proposes a novel preference optimization framework based on Mallows' preference ranking theory beyond the BT setting, which provides a novel perspective on preference modeling. This new method introduces an important component, the dispersion index, to address the shortcomings of DPO in characterizing the diversity of human preferences. This work is technically sound and offers a detailed theoretical analysis. MallowsPO is a generalization of the well-known DPO method and can cover DPO as a special case under a certain parameter setting. 

3. Besides its theoretical contribution, this work also has significant empirical contributions. It proposes empirical approximations to the dispersion index to facilitate computation. Moreover, extensive experiments for the proposed methods are conducted under various setups, including synthetic bandits, controllable generation, fine-tuning Pythia 2.8B on off-policy Anthropic HH dataset, and fine-tuning Llama3-8B-Instruct on a hybrid UltraFeedback dataset, to show the advantages of MallowsPO. Moreover,  MallowsPO is a sufficiently general approach that can be integrated with other offline preference optimization methods, such as simPO and IPO, which have better performance compared with their vanilla counterparts.
summary: This work introduces a novel technique for fine-tuning LLMs called MallowsPO, which is inspired by Mallows' theory of preference ranking. MallowsPO utilizes a dispersion index to reflect the dispersion of human preference to prompts. Existing DPO models are special cases of this dispersion index. Additionally, MallowsPO is compatible with other SOTA offline preference optimization methods.
weaknesses: None.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: See above
soundness: 2
strengths: This paper presents an extension to DPO with theoretical guarantees and demonstrates practical improvements in LLM fine-tuning on a set of benchmark tasks.  The motivation behind is clearly justified and the studied setting is of practical interests.

The introduction of preference dispersion through Mallows’ ranking theory is interesting in preference optimization and is presented clearly. The paper could, however, benefit from additional clarity in Section 3.1, where the transition from traditional DPO to MallowsPO is described. Specifically, more examples or visual aids could assist readers unfamiliar with Mallows’ theory.

The paper provides a detailed decomposition of the reward function based on dispersion, which is mathematically well-grounded and introduces two variations (MallowsPO-$\theta$ and MallowsPO-$\phi$) using different discrepancy functions. This addition allows for flexibility and generalization beyond existing DPO methods.
summary: This paper introduces a preference optimization method called MallowsPO, which extends DPO to incorporate human preference diversity. Authors leverage Mallows’ theory of preference ranking and propose a dispersion index that accounts for variations in human agreement, allowing diverse human preferences on prompt responses. Empirical studies are performed on benchmark tasks to show the performance by using the proposed dispersion index. Comparisons are done with other SOTA offline preference optimization methods.
weaknesses: The use of a dispersion index adds interpretability to preference-based tuning, yet further details on how practitioners could leverage these insights in real-world applications could enhance the paper’s practical relevance.

While the empirical results are promising, additional insights into the MallowsPO model’s performance under varying preference dispersion scenarios (e.g., low vs. high dispersion prompts) could strengthen the evaluation. This might highlight more specific cases where MallowsPO demonstrates clear advantages over other DPO variants.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: I do not fully understand the motivation behind the dispersion index and its estimate.
In particular, the index is estimated using the per-prompt entropy of the reference policy, but: 
- on how many generation? does this require drawing more samples for an accurate estimate?

More importantly, 
- Shouldn't the dispersion index indicate the dispersion of the unknown preference model (i.e. humans)? If so, I do not see why one should use the entropy of the SFT model (which we would still like to align to human preferences) to estimate such index.
soundness: 3
strengths: I believe the paper has the following main strengths:
- The problem studied (i.e. LLM alignment to preference data) is very relevant
- The connection with Mallow's ranking models is, to the best of my knowledge, novel and meaningful
- The derived approach is sound
- Experiments are somewhat extensive and demonstrate the benefit of the introduced approach
summary: The paper presents a novel approach to Preference Optimization for LLMs, named MallowsPO. The approach is inspired by the Mallow's theory of preferences ranking which assumes the preference functions to follow specific distributions parametrized by a dispersion index. Importantly, the dispersion index is prompt-dependent and models the fact that different prompts may have more or less concentrated preference distributions. The authors derive "Direct PO"-like versions of their approach and provide a way to estimate dispersion indices by Shannon entropy. Experiments show that MallowsPO outperform DPO.
weaknesses: Weaknesses are: 
- The authors could do a better job at illustrating qualitative differences between policies trained with DPO and MallowsPO. In particular, one could plot entropy during training, number of tokens, distribution of rewards (when available) to make their case stronger and to understand what are the concrete differences between the two approaches. Part of this is left as future work, but I believe it would be nice to discuss/analyze it in the current paper
- The discussion and estimation of the dispersion index is not fully clear to me. Also, would have been nice to ablate the sensitivity of MallowsPO to other possible estimation approaches (or variants of chosen one).

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: [questions]
- For Figure 9, how does the training curve differ for different prompts, i.e., $x$? It may be interesting to see how the action choice probability can change with the estimated disparity parameters ($\phi(x)$).


[nits]
- The label of Figure 8 (b) is maybe a mistake: I think it should be # of epochs rather than KL.
soundness: 2
strengths: - **Formulating disparity in preferences**: while the original preference model assumes the same preference distribution across different input sentences ($x$), this paper spotlights the fact that the preference distribution can change depending on ($x$) and formulate the difference in reward function.

- **Better empirical results by formulating disparity**: It seems that the result suggests that considering the disparity can improve the reward for varying values of KL divergence.
summary: This paper proposes MallowsPO, which is based on DPO but is distinct in a few points, to fine-tune LLMs from human feedback. The main two distinctions are as follows; (1) Instead of assuming that the preference between the two sentences $y_1$ and $y_2$ is sampled from some probability distribution, the proposed approach assumes that the ranking ($\mu$) of $n$ items are sampled from distribution, and (2) assuming that the preference has some disparity depending on the input text. The proposed MallowsPO formulates a DPO-like objective function for the ranking assumption while incorporating the disparity by introducing a disparity parameter in the reward function.
weaknesses: - **Assuming the probability distribution of ranking, rather than the pairwise preference, loses information about reward difference.**

The paper considers the probability distribution in ranking from the similarity between the ranking $\mu$ and so-called "central" ranking $\mu_0$. For example, Mallow-$\theta$ models the probability as a function of Spearman's rank difference to the central ranking. Unfortunately, this formulation loses the performance difference between each item. 

For example, consider the case where we have three sentences, $(y_1, y_2, y_3)$. Also suppose that the reward of each is $(100, 1, 0)$. In this case, under the assumption of the paper, the probability of observing the ranking of $(100, 0, 1)$ and $(1, 100, 0)$ is the same, while the former is definitely more likely to be observed given the reward difference. For this reason, the proposed approach loses the information about the reward, and I could not find any advantage of assuming probability distribution on ranking rather than on pair-wise preference (pair-wise preference already implicitly encodes the probability of ranking, without losing the information about rewards).

- **Insufficient ablations: lacking baseline of <DPO+disparity>**

The main motivation of this paper is to take the disparity, or the difference of preference distribution depending on the inputs $x$ into account. Then, why isn't simply considering the disparity in the formulation of DPO sufficient? That is, because the preference distribution of DPO is assumed as 

$p^{\ast}(y_1 \succ y_2 | x) := \sigma(r^{\ast}(x, y_1) - r^{\ast}(x, y_2))$, where $\sigma(z) = 1 / (1 + \exp(-z))$,  

I think using $\phi(x) \cdot z$ (where $\phi(x)$ is the estimated disparity params) as the input of $\sigma(\cdot)$ is enough to consider the disparity. This simple baseline is not compared in the experiments.

---

**(summary)**
Overall, I did not understand why this paper needed to introduce the ranking formulation, which actually seems less beneficial than DPO. The latter, considering the disparity can be a unique contribution to this paper, however, this point seems somewhat incremental and weak as the ICLR paper. This is the reason for my evaluation.

Meta-review: This paper proposes a novel approach to preferential optimization based on Mallows' theory of preference ranking. One benefit over DPO is that the dispersion index in the model can be prompt specific and characterize the hardness of ranking for that prompt. This paper is well written, well motivated, the method is grounded in theory, and it is comprehensively evaluated. The topic is timely. The scores of this paper are 2x 8 and 2x 5, which is an improvement over the initial 8, 6, 5, and 3. The discussion clarified several aspects of the method, such as differences from DPO and the loss of reward difference information. Please incorporate these in the camera-ready paper and discuss shortcomings of your method. This paper is accepted.

Decision: Accept (Poster)