MallowsPO: Fine-Tune Your LLM with Preference Dispersions
Authors: Haoxian Chen
Hanyang Zhao
Henry Lam
David Yao
Wenpin Tang
Venue: ICLR 2025 Poster (2025)
Keywords: Language Models fine-tuning
learning from human feedback
Mallows ranking model
human preference dispersions
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Direct Preference Optimization (DPO) has recently emerged as a popular approach to improve reinforcement learning from human feedback (RLHF), leading to better techniques to fine-tune large language models (LLM). A weakness of DPO, however, lies in its lack of capability to characterize the diversity of human preferences. Inspired by Mallows' theory of preference ranking, we develop in this paper a new approach, the *MallowsPO*. A distinct feature of this approach is a  *dispersion index*, which reflects the dispersion of human preference to prompts. We show that existing DPO models can be reduced to special cases of this dispersion index, thus unified with MallowsPO. More importantly, we demonstrate empirically how to use this dispersion index to enhance the performance of DPO in a broad array of benchmark tasks, from synthetic bandit selection to controllable generation and dialogues, while maintaining great generalization capabilities. MallowsPO is also compatible with other SOTA offline preference optimization methods, boosting nearly 2\% extra LC win rate when used as a plugin for fine-tuning Llama3-Instruct.

Published as a conference paper at ICLR 2025
MALLOWSPO: FUNE-TUNE YOUR LLM WITH PREFER- ENCE DISPERSIONS
Haoxian Chen'r, Hanyang Zhao' Henry Lam', David D. Yao!, Wenpin Tang' CColmbia University, Department of IEOR
ABSTRACT
Direct Preference Optimization (DPO) has recently emerged popular approach to improve reinforcement learning from human feedback (RLHF), leading to better techniques to fine-tune large language models (LLM). A weakness of DPO, how- ever. lies in its lack of capability to characterize the diversity of human preferences. Inspired by Mallows' theory of preference ranking. develop in this paper a new approach, the MallowsPO. A distinct feature of this approach is a dispersion index, which reflects the dispersion of human preference prompts. We show that existing DPO models can be reduced to special cases of this dispersion index, thus unified with MallowsPO. More importantly. use this dispersion index to enhance the performance of DPO in a broad array of benchmark tasks, from synthetic bandit selection controllable generation and dialogues, while maintaining great generalization capabilities. MallowsPO is also compatible with other SOTA offline preference optimization methods, boosting nearly 2% extra LC win rate when used as plugin for fine-tuning Llama3- Instruct..
demonstrate empirically how
INTRODUCTION
Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022: Stiennon et al., 2020; Ziegler et al., 2019) has made significant contributions to the success of Large Language Models (LLMs) such as Chat GPT T and GPT4 (Achiam et al., 2023). Recently, Direct Preference Optimization (DPO) (Rafailov et al.. 2023) is proposed as a way bypass RL, thus leading to faster training and better resource efficiency. More importantly, DPO achieves comparable or superior performance against RLHF in downstream tasks such as fine-tuning L.LMs in Llama3 (Dubey et al., 2024), Zephyr (Tunstall et al., 2023), BTLM-DPO (Ivison et al., 2023), etc. DPO's s success has attracted much research attention, leading to variants beyond pairwise ranking such as KTO (Ethayarajh et al., 2023), unified perspectives
loss parameterization such IPO (Azar al., 2024), GPO (Tang et al., 2024), and reference-Fie free alternatives such as CPO (Xu et al., 2024), ORPO (Hong et al., 2024), SimPO (Meng et al., 2024), etc. See (Winata et al., 2024) for a comprehensive survey.
Notwithstanding the successes by RLHF and DPO, both are limited by the assumption that preference follows the Bradley-Terry (BT) model (Bradley & Terry, 1952). In particular, this does not account for varying degrees of agreement in response to different prompts. For instance, people are more likely to agree on "1 =? // 2." as opposed "What is the best city live in the U.S.? // New York." " In language models, this concerns the dispersion of next-token prediction, which is reminiscent to personalization in recommendation systems (Chan al., 2022; Fu al., 2022). See Figure for more illustrative examples of prompts in Anthropic HH (Bai et al., 2022a) dataset.
The purpose of this paper is to formalize the idea of prompt dispersion in the design of DPO. We adapt Mallows' preference ranking theory (Diaconis, 1988; Mallows, 1957), a family of ranking models that provide natural carrier for prompt dispersion, and propose the following decomposttion/iccooiiaatoo of the (latent) reward function:
reward(prompl completion)
(promp))oo((ropp)) X scaled reward(compleet prompt),
where "prompt" and "completion" correspond, respectively, to question and answer. This decom- position allows us specify the diverse level of prompt dispersions hidden in the DPO, which is
"Equal Contribution, a-93; Email: (ho3 136, hz2684, kh1211 14, yao, wt:31 9) @columbia.cdu

---

Published as a conference paper at ICLR 2025
translated into prompt-dependent factor the dispersion index in the preference likelihood. The scaled reward is given by the relative rank of the (possible) completions, which further enhances model interpretability. We then leverage the change of variables technique propose two models, MallowsPO-8 and MallowsPO-oi, by two choices of the discrepancy function in the Mallows model which we will elaborate in Section 3.1.
Figure 1: Prompts with low/high neg-log dispersion estimate values from Anthropic HH dataset.
Distribution Plot of the Dispersion Statistics
8000
How do you got over fear?
What
Normal
Frequency
6000
Can you tell me what you did westenday?
Was the Burtle Waterloo Belgiun?
Above 95% Quantile
Below 5% Quantile
4000
most thing any human
located?
2000
8.6
0.7
Value
The main contributions of this paper are three-fold.
(1) We formalize the idea of prompt dispersion in DPO, and develop the MallowsPO approach to implement this idea, so as to improve and generalize DPO. To the best of our knowledge, this is the first work in preference optimization that considers a mathematically well groundroodded preference ranking model (Mallows) beyond the BT model.
(2) We propose approximations to the dispersion index, a critical component of the Mallows model, so to facilitate computation and validate the approximation through a synthetic example. This allows Mallows model to serve as a foundation for developing versatile preference optimization objectives, including MallowsPO-0 (a generalization of DPO) and MallowsPO-ó. We also provide various analytical results for MallowsPOs, uncovering various new insights on existing DPO models, and generalized PO model that unifies all DPO models (including MallowsPO).
(4) We conduct extensive experiments, from synthetic bandits, controllable generation. fine-tuning Pythia 2.8B off- off-policy Anthropic HH dataset, to fine-tuning (a-ma--8--Instuue UltraFeedback prompts based dataset. Notably, we perform an exclusive hyperparameter search for a fair comparison, and repeat for different random seeds justify the significance of the improvement. The results show clear advantages of MallowsPO over (BT-)DPO, highlighting the potential of considering preference/prompt dispersion.
on-policy
Related Works. Existing work on personalization in dialogue generation such as Fu et al. (2022) and Li al. (2016) has also paid attention to the diversity of human preferences ("there are a thousand Hamlets in a thousand people's ' eyes"): Munos et al. (2023) proposes a Nash game model to incorporate the diversity. There are also other DPO variants: f-POO (Wang et al., 2023) considers general f-divergence in DPO; ODPO (Amini et al., 2024) adds a margin to account for the preference significance. Recent works propose to learn online preferences (Calandriello et al., 2024; Tajwar et al., 2024), or learn from Al feedbacks (Bai et al., 2022b; Chen et al., 2024b; Lee et al., 2023). For classical RLHF, studies to improve the design and capabilities of RLHF include Dubois et al. (2024); Kirk al. (2023); Wang et al. (2024a); Zhai et al. (2023); Zhao et al. (2023); Zheng et al. (2023), whose ideas can also benefit DPO. See Winata et al. (2024) for a survey on learning from preferences.
The remainder of the paper is organized as follows. Background materials on RLHF and DPO are highlighted Section 2. Section 3 focuses on the development of MallowsPO, followed by more analytical results and various perspectives in Section 4. Experimental results are detailed in Section 5, and concluding remarks in Section 6.
PRELIMINARIES
Both RLHF and DPO start with fine-tuning pre- pre-trained LLM by supervised learning on hugl-quality data for some downstream tasks of interest, acquire model SFT This step is referred to as the supervised fine-tumin fine-tuning (SFT) phase. For instance, for training InstructGPT (Ouyang et al., 2022), GPT-3 (Brown al., 2020) is first fine-tuned on the given input prompt distribution.
2

---

Published as a conference paper at ICLR 2025
RLHF (Ouyang et al., 2022; Stiennon et al., 2020; Ziegler et al., 2019). On top of T RLHF is proposed to serve as the next step to conduct further fine-tuning to generate high-quality outputs as judged by humans. Given a generative model it is prompted with prompts of answers (or, "completions"), ), {yn, V2) ~ ( at,, which are then presented to human labelers who express preferences for completion over the other. Denote by 1/e > Vt Ir meaning that You E {v1. y is preferred over E {y1, 923. The preferences are assumed to be generated by some latent reward model r* (ar, y). Based on the collected preference data (a:(6), Mh Vi
produce pairs
(6) 16 Ji= [)) RLHF
consists of first learning a reward model r(x, y). followed by learning a policy (T x)) in which the prompt is the state, and the completion y the action.
(a) Reward model. To capture the underlying human preferences, RLHF assumes the Bradley-Terry model (Bradley & Terry, 1952) that stipulates the pairwise preference distribution:
p" (y1 > x) := a (r" (x, v1) - r" (x, 12))
where (8) Ite- Given access static dataset of comparisons D {x ynu Vi ===.....N RLHF seeks to approximate the latent reward r" (a. y) by a family of functions (ru (z. he and estimate the parameters by minimizing the (negative) log-likelihodd loss: ming L (ri. D) ==
(x >>D [log a (Ti (I, yw) - Tw (z, m))]. Denote by rw. (2, y) the solution to the problem.
(b) RL. The learned reward function rv, (z. y) is then used to provide feedback to the language model. More precisely, the following KL-regularized RL problem is considered:
(2) where 3 >> 0 is hyperparameter controlling the deviation from the reference policy neaf n SFT, In view of (2), RLHF uses the reward function r(x; y) = (x, y) - 3 (log Tty x)) - log Thet (y) x)), and solves the RL problem by proximal policy optimization (PPO) (Schulman et al., 2017).
max EnwD (yn-(vll)) Irv. (a,, v)] - 3KL (T(- xz) Trer [z))]
& DPO (Rafailov et al., 2023). One disadvantage of RLHF is that the RL step often requires substantial computational effort (e.g.. carry out PPO). The idea of DPO is combine the two steps (a)-(b) in RLHF into a single one, bypassing the computation in the RL step.
The key idea is that given reward function r(x; 3), the problem in (2) has closed-form solution:
Tr 2) = Ziz) ziz Tett (y z) exp (r(z, B)) where Z(x) = Ey Tet (y | x)) exp (r(z, B).
Rewrite the above as: r r(x, 3) Blog ==-(vlz) suer [y]=] + , log Z(z). Through this change of variables, the latent reward r" (x; v) can be expressed terms of the optimal policy r"yy] x)), the reference policy Tet (y æ)) and constant 2 (ar). Substituting this r* expression into (1) yields:
H" (11 x) p" (31 > x) = o (3 (3log [og
π (y/ x))
B log
(3)
Tref (y1 x)
Tee (u2 xx)
where (z) cancels out. The expression in (3) motivates the DPO objective:
min LDPO (TT Tret) E.... log (8log (Yur x²) Teet (N/w x:) Tee (y) x)) )
(31 xx)
B log
(4)
3
DPO BASED ON MALLOWS RANKING MODELS
3.1 MALLOWS RANKING MODELS
The Mallows model is built upon the analysis of rankings, instead of scores ratings that play the central role in BT models. Concretely, for a positive integer n that represents e.g., n possible items, let 6, be the set permutations of [n] (1..... ......,77 n and the space of rankings. We consider that the probability of observing a ranking y (which represents the preference of n. items, e.g., the top ranked item preferred over the others) follows a exponential family characterized as:
IP (p) = (d(() /Z(().) d)
for j E Gn;
(5)
where & E (0, 1] is the dispersion parameter, #0 is the central ranking, d(..-) is a discrepancy function that is right invariant: d(H1; P2) = d (E 12 -1 id) for P1, E Sn. and Z d) Exes, (dlu.o) is the normalizing constant. Intuitively, the Mallows model is probabilistic ranking
3

---

Published as a conference paper at ICLR 2025
model in which the probability of rank decreases exponentially with its distance from the central rank. The greater the difference between a rank μ and the central rank, the lower its probability.
Notice that the dispersion indeed reflects how dispersed the probability distribution defined by (5) on the space of rankings is: When Φ + 0, it is point pass concentrated on Po. and when & = 1. is uniformly distributed. In attempt study ranking models (over items) with pairwise preferences, Mallows (1957) further considered two specific cases of the discrepancy function in (5):
M model: d(wn, = ( (i) 22))))² is the Spearman's tho,
Mallows-o model: d((1);))) inv (14 2 is the Kendall's tau,
where inv(p) := # {i..]) [n]3 i < and pli) > n(}} the number of inversions of j..
Example. Consider the central ranking of three items being No (1, with the observation 1 = (3, 2, 1). If the underlying model is Mallows-8 or Mallows-o, we have
(1, (o) = (3 1)² + (2 - 2)² + (1 - 2)² = ds( = inv (3, 2) =
See more examples and visual plots in Appendix C. Notably, different discrepancy functions are usually not consistent, thus they define different Mallows Ranking Models, see an example in C.4.
As remark, the general form (5) was suggested by Diaconis (1988) along with other discrepancy functions (e.g., Cayley, Ulam distances, etc.) See Critchlow (1985); Diaconis (1988; 1989) for the related group representation approach to ranked, or partially ranked data. Note that the Mallows models can be extended to infinite ranking models with n oo (see Meila & Bao (2010); Pitman & Tang (2019); Tang (2)19).) In the context of language models, this conforms to a possibly infinite number of completions given prompt, and allows interpreting unseen completions.
3.2 MALLOWSPO
We adapt Mallows ranking models highlighted above to the setting of language models, First, denote by (( x) a ranking of completions given the prompt x, such that the preference distribution is:
p" (y1 > y/2 x) P (u(y1 2)) (y2 x)).
(6)
Next, for the preference probability in (5), given an input prompt x, we assume it induces a conditional central ranking po(- | x)), and a dispersion index o(x) E (0, 1]. As pointed out in Tang (2019), finding Mo(-1 x) is computationally hard. Fortunately, we discover that in RLHE, this part can be "cleverly" circumvented. By representing r" (x.y) y) as the (negative) rank Ly (z),, our goal now becomes:
max Ez-D [Bgnraly (o [)] 3KL (T ()) []rer x))]
(7)
Note that smaller rank is preferred as per (6). Hence, this provides natural candidate for the scaled reward that enhances model interpretation. This perspective leads to the discovery of a novel family of preference optimization objectives, each of which corresponds to an instance of Mallows models.
MallowsPO-0. For Mallows-f model, we show by Proposition that, if we have the paired- comparison probability of ranking item y/1 before item y/
P (M(11 z) < (32 x)) = (2 log ((z) - (o (x)) = (s)
(8)
where log (xx) E (-00, 0) and s := pá (v2 x) pr (y1 1)), then under Spearman's is distance d and given d(z), we can recover (5) under the assumptions of consistent pairwise probabilities.
Proposition 1 (Probability of rank μ in Mallows-O) Suppose that IP (u(Y | I p (y2 x)) isfies (8) with given & and central ranking po, then we have P(μ) x ((-()))))----- drawn from Mallows-0 (with Spearman's tho distance d(p1, (2) 20=1(m )²)
i.e. μ is
The pairwise probability in (8), which yields Mallows-6 model, leads to our following optimization objective.
Theorem (MallowsPO-t) Given a prompt x, its associated dispersion 6(x) and the underlying central rank (-(-)2)), suppose for any yr y2 the preference probability satisfies p" (yn > y/2 x) as
4

---

Published as a conference paper at ICLR 2025
(8), and thus the preference distribution of y is Mallows-O. Then, the optimal RLHF policy T 1x)
for (7) satisfies p" (y1 > y2 1 (-2 log cb(x) (3 log s(ywlz)
B log
stys (x) Ray (vilz) Therefore,
Har (yw)
the MallowsPO-8 policy oprimization objective is:
(yor xx) Thy (Yer x))
(31 xx)
min (() log o
-2log (b(a)
log
3log
(9)
Tef (y) x3)
neg-log dispersioo of
The proof of Proposition 1 and Theorem 2 are given in Appendix B.
The expression in (8) shows that in the Mallows-O ranking, the representation of p-(- x),, with link function ge, (e))())))) resembles sigmoid function but differs in that it is scaled by the term lol 6(x), which reflects the distribution's dispersion. Compared to BT, where the distribution is exactly in the form of sigmoid, Mallows-O allows for greater flexibility in controlling the spread the distribution function. This is particularly important in language modeling, as the concept of dispersion provides insight into how diverse people's preferences are for different completionssresponsses As ((x)) 0, p- x) is getting closer
step function (i.e., Dirac delta) (as shown in Fig. 2), corresponding to cases where the prompt x has clear, standard answer. Conversely, as Φ(x) + 1. (- z) approaches constant value of 0.5 (i.e. uniform), indicating that any answer the prompt is equally reasonable.
In (8), by setting log 6(x) 1, we recover the DPO in (4). Comparing the objective of the DPO with that of MallowsPO-6, 8, the key difference is the presence of an extra term - log 6(x), which reflects the dispersion of the prompt Thus, MallowsPO-I can be viewed as a generalized version of DPO that incorporates prompt dispersion. To see the effect of this additional term: When dispersion high (ó(x) = 1), the term -2 log σxx) approaches O, reducing the weight on preference pairs; when dispersion ((x) decreases, -2 log o(x) increases, assigning more weight to preference pairs.
MallowsPO-eh The consistent pairwise probability can be derived for the Mallows-d model. As shown in Mallows (1957) (see also (Busa-Fekete et al., 2014; Mao & Wu, 2022)), the probability that a random permutation from the Mallows-ó model with dispersion & agrees with μu on {y1, 3/2 is
PP (v(11 a:)
< (y2 x)) = Is. (No(U2) xx)
po (v1 x)),
(10)
where
0,
(o)o(z)())
1- (-)
(11)
(()- T-
Similarly, using the (consistent) pairwise probability (10) derived from the Mallows-o model, we formulate our policy optimization objective.
Theorem (MallowsPO-d) Given prompt I, suppose the completions y follow a Mallows preference distribution p" as in (10), i.e. j is drawn from Mallows- with chosen ((x) and cen- tral ranking pa, then for any y1 and y2, the optimal RLHF policy satisfies T x)) satisfies
p" (y1 y/ z)) (0.d(a) (Blog alymlw) 3 3log <(y]=)) optimization objective is:
Therefore, the MallowsPO-g policy
and (bwlz)
and [w][z)
min EEG.
log log (o.)(x)
Blog
B log
(yen x) B log Teee
(in x)
(12)
Tie (Ywr
x))
(y1
x
By specifying the underlying ranking to Mallows-o, we get a different link function Jo.o(x)) which also contains the dispersion index ((x), resulting in a new preference optimization objective. We include more discussions about unified perspectives of Mallows-O and Mallows-ó in Appendix C.5.
3.3 How TO CHOOSE THE DISPERSION INDEX ((x)?
As the dispersion index ((x) E (0, 1] is unknown, computation or estimation of it requires learning via neural networks or other algorithms (Meila & Bao, 2010). Here, however, we propose a more direct approach to estimate Ø(z) without any pretraining or learning. The idea is to qualitatively relate ((x) to the empirical output distribution the pretrained model, on which we propose an -ass--o--coppute proxy to the negative log dispersion log()(x)) for each prompt ..
5

---

Published as a conference paper at ICLR 2025
y-x
Probablityy
2-
Saakee 000C
0.2
kink (2-2116 (10)
0
Naa -i
-20
-15
-10
15
20
-9" logg 65 1105
Figure 2: Distribution plot.
Figure 3: Our proposed estimate matches the true (neg-log) dispersion under Mallows model.
Suppose the preference follows the Mallows-o model. There are two extreme cases: When
= 1, if (o 2) ( x). Thus, the prob-
log((())) + oo, have: p" (yn y2 x))
0. if Ho 2)) > Ho(y2 x²).
ability distribution of the next token will concentrate on point mass. When loo(((x)) + have: p" (y) > 1/2 x) 1.2 so the next token will be uniformly distributed.
The above observation motivates us use Shannon's 's entropy H-). Note that H(X) when X is point mass, and (X) = log n when X is uniform on points. Thus, we propose:
6" log (= x))/ logn), for given constant o² >
(13)
as a proxy to - log (x)) where x(- x) can be either the pretrained LM model n PRE or the SFT model SFS Furthermore, we approximate the entropy term in (13) via realization of a sequence of = max([V"), [V]] tokens (Y,", Y given the prompt X:
H( X))
Σ [M044 IY, = Y,") + H(Y Y = Y()],
(14)
which can be easily computed by the logits of the model given the output data. In this case, = KN where k is the token size. This is also related to the predictive entropy Srrránnde--Lobtto al., 2014; MacKay, 1992) of the next-token predictions.
Accuracy of the estimate. To validate our proposed estimate (13), consider the similar 'bandit' setup in Tang (2019). We draw rankings μ from Mallows-ó ranking model, and then obtain a pair of winning/losing actions by choosing the highest/lowest ranked elements in the ranking μ. We plot H((")+H((*)
given the preferences data (æ-axis) and the true dispersion (y-axis) that these data are generated from. Figure shows that our proposed estimator indeed matches the true dispersion, which heuristically reflects the accuracy of our estimate.
-lg
21og
4 PERSPECTIVES ON MALLOWSPO
In this section, we provide several alternative perspectives MallowsPO in Theorem 2-3, with the proofs given in Appendix We say DPO directed by g(-) the preference distribution can be expressed p* (1/1 > V/2 x²) (r"(2, y1) (æ, V2)) for some reward function r*. Thus, Modley--erry based DPO is directed by the sigmoid function (-)).
4.1 DISPERSION WEIGHTED OBJECTIVES
The following results show that MallowsPO can be viewed DPO with either the reward the KL-regularizer weighted by the dispersion index.
Proposition 4 (MallowsPO-8 as dispersion weighted DPO) Let c(x) -2 log 6(x). Then,
MallowsPO-9 is the as a DPO with either the reward weighted by c(x) or the Kl-regularizer

---

Published as a conference paper at ICLR 2025
weighted by Bc(x): max, EZ~D (oylo) (0)) v) BKL (I(- [x)e[r [x]]] or max, EnwD (gy-l(l)ls) (x, v)] 3c(x)KL (T(- [rer x))].
Proposition 5 (MallowsPO-g) as dispersion weighted DPO) Denoting ((x) = in Equ. 11 yields
g(s) == 1 ssgg((s) + sgn(s)
[s| + - M+++ 1 1-
[s]
(15)
2
Let c(z) -2 log ((x) as before. Then, MallowsPO-0 is the same DPO directed by g(-) as in (15), and with either the reward weighted by c(x) or the KL-regularizer weighted by Be(x): max; BEAD [Bywnalul)) [c(x)) (x, -BKL (TT. ()) x))], or max, EAND [Bnwaolale) [r (I. v) Bc(x)KL ( [) |+)]]]
4.2 CONNECTION TO PP
The objective of 4PO (Azar et al., 2024) is max, D ( y" ('>a(-lx)) | (p" > y' x))]
(KL(T(- x) ][ r))], where [0, 1] + IR is a non-decreasing function, and न(: x) is
an arbitrary policy (referred to the behavior policy). It is readily verified that setting (s)
log reduces PPO to the Bradley-Tery based DPO. Roughly speaking, the function Y can be
viewed as the inverse of the link function, ((((s)) = log (I-a(s) )
a(s)
The question is whether
MallowsPO can be reduced to 4PO for some suitably chosen t (-). Assume such function exists, which we denote as y (-). From the Mallows-o model in Theorem 3, we have
(-2))) [ (p" (yn > y2 x)]] = (-)-)(1))) [w. M (r< vn) - r(x, 82)))] # r(x, y1) (11=) [r (xz 32)1
(16)
ie., for any M (-) that is prompt-independeer MallowsPO cannot be an instance of VPO. This calls for extending TPPO to take into account prompt dispersion.
Generalized PO. Let T (z. p) depend on the prompt x as well as the preference distribution p. The generalized PP takes the form:
max EYDD ( (ll)), yf (-11))
V (x, s/ x))) [K.. (x) [ ( (17)
A special instance is when (z. f (x) is separable:
max E Byy ( (p" (y > =)) - 5KL | []]o x))] (18)
Theorem (MallowsPO as generalized v/OO (i) MallowsPO-9 (directed by a()) can be reduced
to the generalized PO in (18) with (i)) = log - and f(z) = 2log
(ii) MallowsPO-ó (directed by g(-)) can be reduced to the generalized PP in (18) with (s)
(s) and fxx) Tzz)
5 EXPERIMENTS
In this section, evaluate the capability of our proposed MallowsPO to learn the preferences in comparison with DPO, First, we use the preferences dataset of IMDB (Maas et al., 201 datasets and Anthropic Helpful and Harmless dialogue (Bai et al., 2022a) dataset to provide evidence that human preferences may be diversed. Next, we consider a synthetic bandit problem to demonstrate the effectiveness of our proposed MallowsPO-o, even without prompt dispersions. We further conduct experiments on tasks such as conditional generation (IMDB) and dialogue (Anthropic HH, UltraFeedback). Our findings show that MallowsPO outperforms DPO with an evident margin, both for in in-distribution performance and out-of- generalization capability.
Our code is available at https://gittie com/hacxit an- che.che///ee /Mal owaPQ.

---

Published as a conference paper at ICLR 2025
5.1 EVIDENCE OF PREFERENCE DISPERSION
A first natural question is: are human preferences dispersed? To verify this key motivation for our work, we plot the distribution of the dispersion estimators given the SFT model and pairwise preferences. Recall from Section that the dispersion estimator is:
0 log 2N log
[Y | Y
= v(") + H(Y+11 Y, = -/))),
(19)
and we take the hyperparameter o" > 0 such that the empirical mean is equal to 1 (as in DPO), so we do not need to tune this scaling constant. Note that this scaling results in our final estimate acting as a relative dispersion level compared to the whole dataset: when a prompt's dispersion parameter is large, i.e., close to 1, (19) will be smaller than I. In contrast, (19) will be much larger than if the prompt's 's dispersion parameter is close to 0 (or there is less disagreement about the answer to the prompt). We formally call this neg-log dispersion estimate throughout the rest of the paper.
1500
3500
DPO
DPD
1200
Mallows
2800
Nallows
Counts
900
Counts
2100
600
1400
300
700
8.6
1.2 Scaled negative log dispersion
0.8
1.0
1.1
1.3
1.4
8.6
0.7
0.9
1.2
1.4
Scaled negative log dispersion
(a) IMDB preference dispersion distribution.
(b) Anthropie-HH preference dispersion distribation.
We find that for the task of conditional generation such as IMDB, its human preferences (Fig. 4a) are not quite diverse: the neg-log dispersion estimates are located near 1, and almost all the estimates range from 0.8 to 1.2. However, for tasks such as single dialogue, Fig. 4b shows that human preferences are relatively more dispersed: the distribution is both skewed and of higher variance. As shown in Figure 1, prompts with high dispersion or those that will lead to human disagreement on preferences indeed have neg-log dispersion estimate smaller than 1, while those with low dispersion have the neg-log dispersion estimate located at the right-hand side (larger than 1). More examples with low/high dispersion are provided in Appendix A.
5.2 MALLOWSPO-6 MITIGATES REWARD COLLAPSE
We study MallowsPO in a synthetic bandit experiment with no contextual information x. and compare it with DPO to test their ability to produce diversified policies and avoid reward collapse. Moreover, we operate under the constraint of having limited number of observations. There are two reasons to explore this setting. First, the bandit facilitates clear analysis without introducing the complication of the context æ. Second, the limited data availability tests the ability of the approaches to produce diversified policies and avoid reward collapse.
Concretely, we consider five arms, each associated with a random reward drawn from a probability distribution. Preference between any two picked arms is determined by the random reward realizations, with larger reward being preferred. In the experiment, we collect 16 pairwise observations, and evaluate the performance of different approaches by computing the efficient frontiers (1) across different parameters 3, and (2) across different epochs. The details are provided in Appendix B.
Figure 5 displays the efficient frontiers for MallowsPO-ó and DPO. Figure 5a shows that MallowsPO- & has a more efficient frontier: (1) With the same KI. divergence, MallowsPO-o achieves higher reward, especially when 3 is small. (2) Over all possible 3, the best reward that MallowsPO-g achieves is higher than that of DPO. (3) MallowsPO-6 avoids reward collapse as 3 gets smaller. That is, MallowsPO-G assigns a certain probability to the potentially good arms, as opposed to DPO that tends to assign only to the "best" arm (see Figure 6). Figure 5b shows that during the training process, MallowsPO-d leads to the policies that have both high rewards and small KL divergence.
5.3 MALLOWSPO YIELDS BETTER TRADEOFF BETWEEN ACCURACY AND REGULARIZATION
In the task of conditional generation for IMDB, x is a prefix of movie review, and LM is to generate output y with positive sentiment, Following the setting in Rafailov et al. (2023), we first fine-tune
8

---

Published as a conference paper at ICLR 2025
Trontlers
16.20
15.75
15.3
13.30
15.0
0.96
1..25
Rewwnn
15:20
0.94
0.02
DPO
155-
DPO
Mellwart-9
MallowsPO-
13.0
MallowsPOOg
0.80
MallwwP0
0.0
0.2
04
KLinl[mee)
0.9
1.
11
0.00
0.02
0.04
kLinl|ne))
0.00
0.00
0.30
0.2
024
30
40
60
(a)
(b)
(c)
Figure 5: (a). Reward vs KL for the policy with different 8's. (b). Reward vs KL every 100 epochs, averaging over the four policies with $ E {0.05, 0.1, 0.5, 1.0). (c) Accuracy vs KL achieved by MallowsPO and DPO,
MallowsPO-0
DPO
8-0.05
8-0.1
A-0.55
8-0.1
1.00
1.00
0.3
0.3
0.75
0.75
0.2
0.50
0.50
0.1
0.1
0.00
0.00
200000
400000
200000
400000
200000
400000
200000
400000
Step
Step
Step
Step
Figure 6: Training curves of MallowsPO-o and DPO for 0.05 and B 0.1.
GPT-2-large the training split of IMDB datasets until convergence to get the SFT model, and use the pairwise preference data from Wang et al. (2023) to further fine-tune it by DPO and MallowsPO.
Figure 5e displays the efficient frontiers (during the training process) for DPO and MallowsPO. We observe that the performances of MallowsPO-I and DPO are close. The similarity is likely due to the nature the task controllable comment generation, which is expected exhibit smaller dispersion, as evidenced in Figure 4a. MallowsPO-ó outperforms both, achieving the same accuracy (evaluated by the reward model) at smaller KL divergence the SFT model/policy.
5.4 MALLOWSPO ENIIANCES BOTH IN/OUT-OF DISTRIBUTION PERFORMANCES
We compare the performances of MallowsPO and DPO in terms of the win rate evaluated by GPT4, and generalization capability on the -iitt--distiiuution datasets. the experiment, we choose / to be 0.1 and 0.5 since it has been observed (Kirk et al., 2023) that increased | value leads to a drop both in performance and per-input diversity of RLHF and DPO. Results are shown in Figure 7.
For the in-distribution test, we first fine-tune a pretrained Pythia-2.8B model on the training set of Anthropic HH dataset using MallowsPO and DPO, and then evaluate their responses on subset of its test split. GPT-4 serves the evaluator, and compares pairs of responses. We observe that MallowsPO has an edge over DPO. For the out-of-distribution test, we apply the models, fine-tuned on the train split of the Antropic HH dataset, to other datasets with different input distributions. The H4 Stack Exchange Preferences Dataset (SE) (Lambert et al., 2023) and Stanford Human Preferences (SHP) (Ethayarajh al., 2022) are used for evaluation. The advantage of dispersion generalization becomes apparent, as MallowsPO shows more improvement compared to the in-distribution case.
We also compare MallowsPO-6 with DPO in fine-tuning the Pythia-2.88 model, with ArmoRM (Wang et al., 2024b) serving as the evaluator. The result indicates that MallowsPO-9 achieves consistently higher win rates than DPO across all cases, with an impressive win rate of around 70% in the in-distribution test. Details are provided in Appendix F.I.
5.5 MALLOWSPO ENHANCES SOTA LLAMA3- -EL-----IINNTEE INSTRUCT MODELS
We illustrate the scalability of our method through experiments on fine-tuning Llama3-8B --8--Instruct Model Ulrraeeddback Dataset. We follow the same setup in RLHFlow (Dong al., 2024) and

---

Published as a conference paper at ICLR 2025
In distribution
Out distribution
Dataset
Anthropic
H4 Stack Exchange
Stanford Human Preferences
B
0.1
0.5
0.1
0.5
0.1
0.5
MallowsPO-9 DPO
57.57%
50.67%
54.36%
55.03%
53.33%
56.00%
MallowsPO-g
53.33%
54.33%
55.78%
51.07%
54.33%
56.67%
Figure 7: Win rates computed by GPT-4 for responses on both the in- and out-of distribution dataset.
SimPO (Meng et al., 2024), generate five answers from Mlama--8B-Instiee for each prompt in UltraFeedback, rank them with scores evaluated by ArmoRM (Wang et al., 2024b), and choose the best/worst one winning/losing answer to form the preference datasets. For fair comparison, compare MallowsPO with DPO, using different hyperparameters: and learning rate ir for the task of Alpaca Eval V2. The results shown in Appendix F:2:
Ir
LCC. Win Rate
Win Rate
DPO
MallowsPO
DPO
MallowsPO
0.01
-7
42.55% (0.79)
43.10% (0.77)
42.02% (1.53)
43.02% (1.57)
IPO
MallowsIPO
IPO
MallowsIPO
0.005
Le
43.38% (0.84)
44.73% % (0.87)
43.52% (1.45)
44.87% (1.46)
SimPO
MallowsSimPO 51.89% (0.81) 42.11% (1.46)
SimPO
MallowsSimPO 43.76% (1.47)
10
le 6
50.04% (0.77)
Table 1: Win rate comparison between SOTA iin-ttunig methods and their enhanced versions using our MallowsPO as a plugin with optimized / and Ir. Standard deviations are right next to the reported metric.
When 8 0.01 and Ir 5e"7, -, for which DPO and MallowsPO both achieve the best performance, we used 10 random seeds show the statistical signif- icance: MallowsPO outperforms DPO both mean or the best performance across random seeds, and also has smaller variance (see Figure 8).
Apers Eat siae 301
We also adapt the idea of dispersion index in Mal- lowsPO to IPO and SimPO, leading to MallowsIPO and MallowsSimPO., As shown Table both Mal- lowsIPO and MallowsSimPO beat their vanilla coun- terparts (using hyperparameters proposed in Azar et al. (2024) and Meng et al. (2024)).
Figure 8: Win rates computed by GPT-4 for sponses on Alpaca Eval V2.
6 CONCLUSION
We have developed in this paper a novel approach, the MallowsPO, to fine-tune LLM. A distinct feature of this approach is a dispersion index, which naturally captures the dispersion of human preference to prompts, and can be systematically incorporated into the reward function as a weight factor, thus ushering in a new class of dispersion-weiihteed DPO models. We demonstrate empirically how MallowsPO achieves improved performance in broad array of benchmark tasks, including synthetic bandit selection, controllable generation, and dialogues. The effectiveness holds for both small and large representative models, such Pythia 2.8B and Llama3-8B -i-mma----Insstuct...
Furthermore, both curriculum learning and personalization alignment through contextual scaling present promising future directions and can be expanded based
work (see Appendix G).
There are a few issues that we have yet to address this study, for instance, to explore why MallowsPO outperforms DPO, how the dispersion index contributes performance improvement, what guidelines to follow to set the 3 value, and how dispersion index can more accurately approximated. These will be pursued in future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
Hanyang Zhao, Haoxian Chen and Wenpin Tang are supported by NSF grant DMS-220602 Wenpin Tang acknowledges financial support by the Tang Family Assistant Professorship. The works of Haoxian Chen, Hanyang Zhao, Henry Lam and David Yao are part of a Columbia-City U/HK collaborative project that supported by InmotHK Initiative, The Government of the HKSAR and the AIFT Lab. Additionally, Haoxian Chen is supported by the Amazon CAIT fellowship. Henry Lam and Wenpin Tang receive support from the Columbia Innovation Hub grant.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, and Shyamal Anadkat. GPT-4 technical report. 2023. arXiv:2303.08774.
Afra Amini, Tim Vieira, and Ryan Cotterell. Direct preference optimization with an offset. 2024. arXiv:2402.10571.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm understand learning from human preferences. In AISTATS, pp. 4447-4455, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, and Tom Henighan. Training a helpful and harmless assistant with reinforcement learning from human feedback. 2022a. arXiv:2204.0586..
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, and Cameron McKinnon. Constitutional AI: Harmlessness from AI feedback. 2022b. arXiv:2212.08073..
Yoshua Bengio. Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, and Amanda Askell. Language models are few-shot learners. In Neurips, volume 33, pp. 1877-1901, 2020.
Róbert Busa-Fekete, Eyke Hüllermeier, and Balázs Szörényi. Preference-based rank elicitation using statistical models: The case of Mallows. In ICML, pp. 1071-1079, 2014.
Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, and Tianqi Liu. Human alignment of large language models through online preference optimisation. 2024. arXiv:2403.0863.
David M Chan, Yiming Ni, David A Ross, Sudheendra Vijayanarasimhan, Austin Myers, and John Canny. Distribution aware metrics for conditional natural language generation. 2022. arXiv:2209.07518.
Ruizhe Chen, Xiaotian Zhang, Meng Luo, Wenhao Chai, and Zuozhu Liu. Pad: Personalized alignment at decoding-time, arXiv preprint 2400v22410.04070, 2024a.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. 2024b. arXiv:2401.01335.
Douglas Critchlow. Metric methods for analyzing partially ranked data, volume 34. Lecture notes in Statistics, Springer, 1985.
Persi Diaconis. Group representations in probability and statistics, volume 11. Lecture Notes- Monograph Series, 1988.
11

---

Published as a conference paper at ICLR 2025
Persi Diaconis. A generalization of spectral analysis with application to ranked data. Ann. Stat., pp. 949-979, 1989.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang. Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. 2024. (rXii:2405.07863.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, and Angela Fan. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. In Neurips, volume 36, 2024.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. ICML, pp. 9988-6008, 2022.
Kawin Ethayarajh, Winnie Xu, Dan Jurafsky, and Douwe Kiela. Technical report. Contextual
Human- AI, 2023.
aware loss functions (halos).
Tingchen Fu, Xueliang Zhao, Chongyang Tao, Ji-Rong Wen, and Rui Yan. There are a thousand hamlets in a thousand people's ss eyes: Enhancing knowledge-grounded dialogue with personal memory. 2022. arXiv:2204.02624.
Alex Graves, Mare G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In international conference machine learning, pp. 1311-1320. Pmlr, 2017.
José Miguel Hernáneez-Lobato, Matthew W Hoffman, and Zoubin Ghahramani. Predictive entropy search for efficient global optimization black-box functions. NIPS, volume 27, pp. 918-926, 2014.
Jiwoo Hong, Noah Lee, and James Thomn. Orpo: Monolithic preference optimization without reference model. 2024. 1.X.v.2403.07691.
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, and Iz Beltagy. Camels in a changing climate: Enhancing LM adaptation with Tulu 2. 2023. 1xXi:22311.1071
Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of RLHF on LLM generalisation and diversity. 2023. arXiv:2310.06452.
Nathan Lambert, Lewis Tunstall, Nazneen Rajani, and Tristan Thrush. Huggingface h4 stack exchange preference dataset, 2023. URL https:: htatts:://huginngfeee. co/ datasets/ HlugginFFacein stack wecckk-exchnggegee preferences.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with AI feedback. 2023. arXiv:2309.00267.
Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. Deep reinforce- ment learning for dialogue generation. 2016. arXiv: 011451
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In ACL, pp. 142-150, 2011.
David JC MacKay. Information-bass objective functions for active data selection. Neural computa- tion, 4(4):590-604, 1992.
Colin I. Mallows. Non-null ranking models. I. Biometrika, 14(1/2):114-111 1957.
12

---

Published as a conference paper at ICLR 2025
Cheng Mao and Yihong Wu. Learning mixtures of permutations: groups of pairwise comparisons and combinatorial method of moments. Ann. Statist., 50(4)) 2022.
Marina Meila and Le Bao. An exponential model for infinite rankings. J. Mach. Learn. Res., 11: 3481-3518, 2010.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference- free reward. 2024, arXiv:2405.144 14734.
Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, and Andrea Michi. Nash learning from human feedback. 2023. arXiv:2312.00886.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, and Alex Ray. Training language models to follow instructions with human feedback. In Neurips, volume 35, pp. 27730-27777 277444, 2022.
Jim Pitman and Wenpin Tang. Regenerative random permutations of integers. Ann. Probab., 47(3): 1376-1416, 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Neurips, volume 36, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. 2017. arXiv:1707.066: 006347.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In Neurips, volume 33, pp. 3008-3021, 2020.
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xic, Ste- fano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of LLMs should leverage suboptimal, on-policy data. 2024. Net:2244..13677
Wenpin Tang. Mallows ranking models: maximum likelihood estimate and regeneration. In ICML, pp. 2125-61344 2019.
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. 2024. arXiv:2402.05749
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, and Nathan Habib. Zephyr: Direct distillation LM alignment. 2023, atXiv:2310.169M
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, and Chenyu Shi. Secrets of rlhf in large language models part II: Reward modeling. 2024a. arXiv:2401.06m 000000000
Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse KL: General- izing direct preference optimization with diverse divergence constraints. 2023. MiXi::3309.16444
Haoxiang Wang. Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. 2024b. arXiv:2406.I 122.222466....5....
Daphna Weinshall, Gad Cohen, and Dan Amir. Curriculum learning by transfer learning: Theory and experiments with deep networks. In International conference on machine learning, pp. 5238-522-22 5246. PMLR, 2018
Genta Indra Winata, Hanyang Zhao, Anirban Das, Wenpin Tang, David D Yao, Shi-Xiong Zhang, and Sambit Sahu. Preference tuning with human feedback on language, speech, and vision tasks: A survey. 2024. #xXiv:2409.11566
13

---

Published as a conference paper at ICLR 2025
Haoran Xu, Amr Sharaf, Yummo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries LLM performance in machine translation. 2024. arXiv:2401.0841
Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. Investigating the catastrophic forgetting in multimodal large language models. 2023. arXiv:2309.100
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. 2023. arXiv:2305.1443 10425.
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, and Yuhao Zhou. Secrets of RLHF in large language models part I: PPO. 2023. arXiv:2307.0499 239.4444....4.
Daniel M Ziegler. Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford. Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. 2019.
19Xiv:190,00899..
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A PROMPTS WITH LOW/HIGH NEGATIVE LOG DISPERSION ESTIMATE
A.I EXAMPLES OF PROMPTS WITH LOW NEGATIVE LOG DISPERSION ESTIMATE
See Table 2 for the examples.
Human:
need some ideas repurposing old magazines.
Assistant:
How about using magazine dusteover for book?
Human:
will take some effort but it could work. What else can you think of?
Human:
Why do you think western society idolizes youth much, opposed asian society, which idolizes elders?
Assistant:
Humans throughout history have consistently held important values and traits in high esteem, including trustworthiness, compassion, intelligence. and wisdom. Given that younger people have the potential to develop these traits while older people have already developed them, makes sense that older people's virtues are embraced in some cultures while younger people's virtues are embraced in other cultures. It's important to remember that cultures throughout the world are complex and heterogeneous, so it's possible for some to prioritize older people yet also to respect some of the important qualities and values held by younger people. The value we place on older people will vary inversely with value we place on young people, and vice versa, which helps explain why different cultures come to elevate one or the other.
Human:
That's fair assessment. but I'm looking for more specific reasons. Let's refocus just the west. Why are we so routh-focusces
Human:
what
cool ocean activities?
Human:
Why do almost all cartoon characters tend to only have 4 fingers? Is it some unwritten rule in the animation industry to do that?
Human:
Can you provide 3 bullet points for how teach child how to set the dinner table?
Human:
I'd like to become a world famous writer. Any suggestions on how to get there?
Human:
My friend is a University Lecturer with a Ph.D. in Business. She's 's teaching a new course she's 's never taught before and is experiencing Imposter Syndrome. How can help her see she's definitely qualified and able to teach this course successfully?
Table Prompts with low neg-log dispersion or high dispersion, i.e. human are likely to have more disagreement about the answers.
A.2 EXAMPLES OF PROMPTS WITH HIGH NEGATIVE LOG DISPERSION ESTIMATE
See Table 3 for the examples.
15

---

Published as a conference paper at ICLR 2025
Human:
Have you heard of Summer Palace, historical site China? Yes, Summer Palace is well-known historical site in China. It was originally built in the eighteenth century as a summer retreat for the royal family, and is renowned for its beautiful gardens and temples. Summer Palace is now a popular tourist destination, attracting visitors with its historic architecture and natural beauty. There are also many tours and exhibits the site, offering visitors in-depth look at its history.
Assistant:
Human:
In which city is it located?
Human:
Translate the following into slang: Good morning, how are you today? I have to go to work later.
Assistant:
Hey, how's it going? I'm doing pretty good today. have to head to work later. Now capitalize every word of that response.
Human:
Human:
Please tell me about The Battle of Waterloo.
Assistant:
In 1815, Napoleon led his troops and won against the British and Prussian armies.
Human:
No. Napoleon lost the battle.
Assistant:
OK, in 1815, Napoleon led his troops and lost against the British and Prussian armies. Was the battle in Belgium?
Human:
Human:
Are there any NASA space missions going on currently? now, no.
Assistant:
Human:
Are you sure?
Assistant:
NASA doesn't have any plans for missions in the immediate future. However, there are several missions on the books that might eventually get funded.
Human:
Thank you for this information.
Table 3: Prompts with high neg-log dispersion low dispersion, i.e. human likely have less disagreement about the answers.
B PROOFS
Proof of Proposition 1.
Mallows (1957) showed that the paired-comparison probability
IP (u(y) xx) (y2 x)) 1 expp-22ogg ((r) (No (y1 x)) no (y2 x)))
(20)
yields the Mallows-8 model. Here we sketch a proof for case of reference. Write y == -2 log 6(x) for simplicity. Note that
exp (-7/40 (11))
(y/ is preferred to y2) exp (-YPo (g1)) + exp (-7/4g (82))
(21)
By Equation 21, the probability of observing (full) ranking p
P())) x II ((((i) is preferred to ((j)) & exp
Σ (n 1) (n-)))--au (-mo(t
(22)
Σ 1=1
i Ho (a- (o))
exp
Moreover, we have:
> i pol (1)) = Σ (molu 1(0) -1 3 +C C=
----((0 - - +C, (23)
where C = En polu- (i)) (n + 12 1)/3 is a constant, and the second equality due to the right invariance of the Spearman's rho. Combining Equation 21 and Equation 23 yields:
P(u) x - (m(o) mool))² x ((x)) Ei (m(00~-00000000 mo()))
(24)
16

---

Published as a conference paper at ICLR 2025
This is exactly the Mallows- model.
Proof of Theorem
To obtain (9), recall that given æ, the probability of output y1 being preferred than y/2 is:
p (y1 3/2 x² P (μl3) 2) μl (y2 2²)
(25)
= (_ x) - (0 x)))
(26)
Therefore, the modeling of such a ranking distribution requires the conditional central ranking Mo(- z). However. since finding Mo(- z) is computationally hard (Tang, 2019), to tackle this challenge, we explore a different path. Recall that RLHF, we optimize the following objective:
max ExwD [By=so(ula)) [r" (xx, v)] 3KL (TT(- [=]] [))]
where (2( y) is the true underlying reward. By letting r (xx, u) -moly x), we now turn to optimize the following objective:
BEzD BBgnaol m) [-Mo(y [x)] 3KT. (m( [))ee [=)]]
As shown in section A.1 of Rafailov et al. (2023), the optimum of such KL-constrained reward maximization objective has the form of
1 ((x)
Po x²)
no (y x)
Neet (I) exp
where (ar) the partition function ensure noo (v w) to be probability distribution. By moving terms, we have
Mo / x)) log Too (y x)
+ Blog Z(x).
(27)
Tel y x)
Combining (26) and (27) gives
an (y1 x)) Jeet (V1 x)
To (V2 xr) Neet (3/2 x))
p" (11 > 3/2 x) = (d.
Blog
B log
To maximize the likelihood estimation, our objective becomes
log (d.
Ta (Ye x.) Tref (Nw
THo (eu x)) Teef (y) x))
min - (e.
(Blog
B log
Tug
x))
To derive the MallowsPO-t loss objective, direct application of above derives the result, which is:
(thw ())
(yu x))
SMallows-@TTe Teet) -E log
log ((x)
3lo
33 log
Tyef (yw x)) Tooe (y1 x))
dispersion of
(28)
in which the expectation is with respect (I, Mw M1) D.
Proof of Theorem 3.
For the Mallows-O model, was shown in Mallows (1957) (see also Busa-Fekete al. (2014); Mao & Wu (2022)):
IP (n(v) |x) < (22 x))
(29)
No (V2 xx) 4o (v1 2x) 0,
(zz)"oo0z -)-Po(U1 I- I-0(z))0(0z) (o) f #o)=
n4 (y2 (x) 140 (y1 (x) 0, (30)
no uon
I- = P I- PotvgleT)
A similar discuss as in the proof of Theorem 2 derives the desired result in (12).
17

---

Published as a conference paper at ICLR 2025
Proof of Proposition 4.
The proof follows from the derivation of the equivalence between RLHF and DPO, as now the optimal policy satisfies
(y x)) Tea (y x)
dx)) v) =3 log
+ 3 log ((x).
Proof of Theorem
(i) With the Bradley-Terry connection as mentioned above, we have
(a.
Bygna [f(r) (p" (y1 > Ix)] = Eyyr~ ʃ(x) (e(x,0)) + e
(31)
Ey (x) (r(z, M1) - (2)))))
f(x)r(z,,1) f |r (zz y2)]
which is weighted reward of DPO, up to an additive constant. It follows that the optimal policy of the generalized vPO (18) is the same as that of MallowsPO-O by Theorem 4, The same argument also proves (ii).
18

---

Published as a conference paper at ICLR 2025
C MORE EXAMPLES OF MALLOWS RANKING MODEL
Here provide more detailed concrete examples and corresponding computations for Mallows Ranking Model. We also illustrate the implicit effects of the total number of items v..
Recall that, as defined in Section for positive integer n that represents e.g-, n possible items, let C be the set of permutations of [rx] = ( {1,...,,u n and the space of rankings. The probability of observing ranking μ (which represents the preference of items, c.g., the top ranked item is preferred over the others) under the Mallows Ranking Model is:
a == dd(umuo)) /Z(ó, d) for Gn.
(32)
where á (0, is the dispersion parameter, pn is the central ranking, d(.,-) is discrepancy function that is right invariant: (12 (Mi 12 id) for ıı; M2 G.. for which:
Mallows-O model: d(2 Σi= (i) 2 the Spearman's rho,
Mallows-o model: d(p1, (2) = inv (m2 |مク ). in which inv(p) = # {(t.)) E [n]² < j, and ((1) n()} is the number of inversions of μ..
and Z(o, Epec, (d)p.)o) is the normalizing constant.
C.I MALLOWS- MODEL.
To both qualitatively illustrate Mallows-O Ranking Model and quantitatively compute the correspond- ing probability, here we consider three items example, thus the total number of possible rankings 6, and we compute the discrepancy function values, normalization constant when dispersion index is 0.5, and the resulting probability in Table 4:
Ranking
Discrepancy
Normalization Constant
Probability
(1,2,3)
0
0.651
(1,3,2)
2
0.163
(2,1,3)
2
0.163
1.535
(2,3,1)
0.010
(3,1,2)
6
0.010
(3,2,1) 8
0.0025
Table 4: Mallows-O Model probabilities when dispersion index Ø 0.5.
We also plot the probability density function (in æ-axis we arrange the rankings by their distance to
the central ranking, the lefter the short distance to central ranking) in Figure 9a. We also plot the effects of dispersion on the resulting probabilities as in Figure 9b. We can see as dispersion parameter goes up (close to 1), the probability density function becomes flattened.
Probobiitty tordored
Probebility Density Function Mallows-9
Awd27
0w04
0.5
0.4
(32.3)
0,A20
04.29
0.2.1
Rantings foreece dstance central rankng
(a) Mallows-@ Probability Density function.
(b) PDF with respect dispersion ø
19

---

Published as a conference paper at ICLR 2025
C.2 MALLOWS-O MODEL
Here we consider the same example in the previous Mallows-O except the discrepancy function is d(p, (2) inv (mn t) The values be seen in Table 5.
Ranking
Discrepancy
Normalization Constant
Probability
(1,2,3)
0
0.381
(1,3,2)
0.191
(2,1,3)
1
0.19
2.625
(2,3,1)
2
0.095
(3,1,2)
2
0.095
(3,2,1)
3
0.048
Table 5: Mallows-ó Model probabilities when dispersion index 0.5
Again we plot the probability density function (in æ-axis we arrange the rankings by their distance to the central ranking, the lefter the short distance to central ranking) in Figure 10a. The effects of dispersion on the resulting probabilities as in Figure 10b.
Probebillty by El Rankings lordered by distance to centrel renking)
Probebility Density Function for Mellows-j
0.40
0.35
0.15
0.19
0.55
112.31
21.
(B2.1)
(1:23)
(1.2))
(2120
[8A1)
(112)
(92.1)
Rantings femercs by dstance to central aniing
(a) Mallows-o + Probability Density function.
(b) PDF with respect to dispersion ş
C.3 THE EFFECTS OF n >> 1 IITES
Here also illustrate the effect of additional items for the relative probability of observing two rankings which differ only in the rankings of 3 items. If there are no additional items, then observing relative order of items of values 1100.1,2) with the ground truth (100,2,1) under Mallows-O is (i.e., observing (1,3,2) for central ranking (1,2,3)):
p(100, 1,2) 2- -03-22 2. 0z,
which will be the same observing (2,100,1) since:
p(2, 100, 1) = Z (2 (-- 1) = 1Z.65 0Z.
However, if there are additional 97 items of values (100 items in total of values from to 100), things will be quite different. If uut) holds for all other i's from to 99, we have:
p(100, 1, Z (99- +110=++11990000
Z Φ
while
100, Z (99- 19) 911-9) 7 2x98*
Thus, p(2, 100, < p(100, 1,2) because o E (0, 1).
20

---

Published as a conference paper at ICLR 2025
C.4 DIFFERENT MALLOWS MODELS ARE INCONSISTENT.
Consider two discrepancy functions used to measure the difference between two rankings (which are exactly Mallows-O and Mallows-o models in this paper):
A. Squared Difference (SD):
SD n²)
Σ (n(1) x()))
where ((1) and (i) denote the positions of item i in the true ranking and the predicted ranking μ', respectively. B. Number of Inversions (NT):
N((H, = (((,)) (m(1) - ~()) (1' - (1))) < 0
which counts the number of pairs (i.j) where the relative order of items i and j is different in μ and μ.. Let the ground truth ranking f of four items {A, B, C, D3 be:
(A 1, B C: 3, 4:
consider two predicted rankings μl and p2:
1. Ranking 11 and μ12
(A: 2, B B C: :4, 3) 12= (A:1, B:4, C: D:
A. Squared Difference (SD):
For p1:
SD(M, u1) = (1 (1-2) 22 (3 4) (4 3)²
---1)² +-1+ +12
-1+1=1+1=
For N2:
SD(M, = - 11- +)² (2 4) + (3 - 2)² + +0 (4-3)-
(-2)- 1
=0+4-1+1-6.
Conclusion: SD(H, 11) 4 < SD(H, u2) 6, μr is better than μl2 under the squared difference.
B. Number of Inversions (NI):
The number of inversions is calculated by counting the number of pairs (i,j) where the order is reversed between and jz'
Pairs to Consider:
BA, (A, C), (A,D), (B,C), (B, (C,
For μ1, Inversion Count:
(A, B):
Inverted (j:: A B, p1 A> B) inversion.
D): Inverted (p:: C. < D. C> D) = inversion.
Other pairs maintain the order.
Total Inversions in M1 2 inversions.
For (12, Inversion Count:
-BB, C): Inverted (j:: < C.
M2: B> +)=1
= inversion.
-B,,), Inverted (μ: B < D, N2: B D) = 1 inversion.
-Other pairs maintain the order.
Total Inversions in N2 2 inversions.
21

---

Published as a conference paper at ICLR 2025
Conclusion: NI(H, p1) N1(,, 2, so Yr and μg equally good under the number of inversions measure.
This example demonstrates inconsistency between the squared difference and the number of inversions discrepancy functions:
Under the squared difference measure, 14 is better than A22- Under the number of inversions measure, μ4r and P2 are equally good.
C.5
UNIFY MALLOWS-P AND MALLOWS- FOR COMPUTATION
Note that the link function (p.0(x) in MallowsPO-O continuous (or smooth)
with
1- 10+ +1 lng d(a))
adb(n)" log d(())
8 > 0,
(8)
(1 o(
I- ((x)"
(1 ((x))
(33)
T-0(z)) (1 =))(2)
log ((ar)
s0(2) log o(x)
< 0.
((x)
I-o(0)
(1-p(x) 2)2
For computational purposes, we propose smooth approximations 96.0(x)
(i) Sigmoid approximation: Since (1) ++)(a)> we approximate (o.d(a) by T2 (s)
a-=slop (xx) so that (1) (s.d(=) (1). See Figure 11 for an illustration of this approximation. With this approximation, MallowsPO-o and MallowsPO-I yield the same objective with different S's
(up to factor of 2). Thus, MallowsPO-9 is just MallowsPO-g with sigmoid approximation.
(ii) Polynomial fitting: We use a polynomial of form P(x) agæx + d1 + du approximate (()) on [-c, c). with e being hyperparamntte. We choose e to be either fixed, e.g., 0.1: or 6 log (xx) (e.g., R: 1.4 for ((x) = 0.5). See Figures 12-13 for an illustration.
or sigmold
original
0.8
0.63
orginal Poyyomia ft
0.8
PMlyoomia IRt
0.00
0.1
0.5
0.50
0.4
0.45
02
0.00
0.3
-1.
-05
0.0
0.5
1.0
Figure 11: Sigmoid approxima- Figure 12: Poly-fitting on tc tion
Figure
13:
Poly-fitting on
422 log Ø
22

---

Published as a conference paper at ICLR 2025
D FURTHER DISCUSSION ON MALLOWSPO AND DPO (RAFAILOV ET AL., 2023)
MallowsPO-O. When the responses y1, 2/2 for given prompt follow the preference distribution in (8), Theorem 2 establishes the optimization objective as follows:
MallowssO- 9 (T; Tyet)
-llgg (st xt) Trtt (Yw x)) Nret (3) xx)
(Yer [x)
E (a log
2 log ((x))
Blog
neg log disspersion of
(34)
where log (xx) (0, oc). To compare with the objective of DPO, where the underlying ranking model is Bradley-Terry:
" (Bw a)) Niet (yw xx)
π (yt x)) Jea (y 2z)
)
LDP (I;; Tret) == - -B (a..emm)> log(or
Blog
Blog
the objective of MallowsPO-A in (9) has an extra term -2 ((x),, which reflects the dispersion of the prompt x. The similarity of MallowsPO-O and DPO is resulted from the similarity of the foundational ranking models between Bradley-Terry and Mallows-O.
Alternatively, MallowsPO-D can be viewd DPO with an added element of prompt dispersion or disparity. Rather than interpreting MallowsPO-6 solely Id.o(=) (0,0(x))
reinterpret that such an objective still maintains the link function of sigmoid (-),, but the logit difference term (1 la) 8log
log
For (2e]a]] is sacled by the dispersion -2 log (zx). This scaling introduces
<tvl[a) (x)
Ter (Bw [
nuanced modification to the original DPO framework, accounting for the influence prompt-specific dispersion.
MallowsPO-d. Similarly, when the responses y1, y/2 for given prompt æ follow the preference distribution in (10), Theorem yields the following optimization objective:
MMallowsPO-g (x; Tref) =
π (the x:)
π (yn x.)
)
(35)
log (o)
log
3 log
Tiet (yo xr)
Jeet (y1 x:)
where (o.6(a) (8) 1-ggn(e)
sgn(s)
[e|+1 1-0(2)1
lel
1-
In comparison with the DPO in (4), MallowsPO-g) replaces the sigmoid function a with a (different)
link function (o.0(x) that also contains the dispersion index (zx).
An overview of the comparison of these methods is provided in Table 6.
Model Name
Ranking model class
Dispersion
Link function
DPO (Rafailov et al., 2023)
Bradley- Terry (Bradley & Terry, 1952)
X
sigmoid a(.)
MallowsPO-0
Mallows-O (Mallows, 1957)
sigmoid a(-)
MallowsPO-6
Mallows-o (Mallows, 1957)
(o.o(n) in (10)
Table 6: Key Features of DPO and MallowsPO
23

---

Published as a conference paper at ICLR 2025
E EXPERIMENTAL DETAILS
E. BANDIT EXPERIMENT
In the bandit experiment detailed in Section 5.2, we conduct two sub-experiments to compute the efficient frontiers using Mallow-p-DPO and DPO. The first sub-experiment varies the parameter / while the second varies the epochs, with 3's to be a fixed set. For the first sub-experiment, we run each algorithm we record the reward and (L(TT Teet) of the average policy over the last 30 epochs to stabilize the results. As for the second sub-experiment, similar to the setup in Rafailov al. (2023) and Wang et al. (2023), we execute an ensemble of training configurations for both MallowsPO and DPO, by adopting range of different / E (0.05, 0.1, 0.5, 1.0), and record the average reward and average KL(T] (ett) among the four policies for every 100 training steps. Given that we know the real reward distribution, all these quantities can be computed analytically.
range of /3 values required to compute the full efficient frontier, and for each B,
In terms of the training details, we use all 16 data in a single batch and adopts SGD as the optimizer, with learning rate of 5e-3. To ensure convergence, we run the optimization for a large number of epochs, set to 500, 000. For MallowsPO-gl we set & to be 0. 05.
Table 7: Reward distributions of the five arms.
Arm
Arm 2
Arm 3
Arm 4
Arm 5
Reward
Prob.
Reward
Prob.
Reward
Prob.
Reward
Prob.
Reward
Prob.
20
0.5
30
0.5
18
0.5
15
0.99
0.99
0.5
3
0.5
15
0.5
0.01
0.01
Table 8: 16 pairs of sampled preference data.
Win
3
2
2
1
4
2
2
3
4
Lose
5
2
5
5
5
2
E.2 CONTROLLABLE GENERATION EXPERIMENT DETAILS
We follow the training setup in Rafailov et al. (2023), and first fine-tune GPT-2-1arge on the training split of IMDB datasets until convergence get the SFT model. The next step is different from Rafailov et al. (2023) in that we directly utilize the (offline) preference dataset from Wang al. (2023) instead of generating pairwise preferences from the trained SFT model, as in DPO. The rest is the same: we use the pairwise preference data to fine-tune the SFT model by either DPO or MallowsPO. The evaluation metric: accuracy is obtained from a prior sentiment classifier as the ground truth reward. By default, we use RMSprop optimizer with learning rate of 1e 6, with linear learning rate warmup from 1e-6 over the first 150 steps. The training batch size is 6 64.
E.3 LANGUAGE MODELING EXPERIMENT DETAILS
We follow the training setup in Rafailov et al. (2023). By default, we use RMSprop optimizer with learning rate of 1e- 6, with a linear learning rate warmup from 0 to 1e- 6 over the first 150 steps. The training batch size 32.
E.3.1 GPT-4 JUDGEMENT PROMPT
Response quality evaluation is completed by GPT-4. The prompt for instructing GPT-4 to evaluate which response is better is particularly important. Thus, use the fastchat package for GPT4 evaluation, and we used their well-written pair-v2 judge prompt. The prompt is shown as follows:
Please act responses provided by two AI assistants to the user question displayed below. You should choose the assistant that follows
impartial judge and evaluate the quality of the
24

---

Published as a conference paper at ICLR 2025
the user's instructions and answers the user's question better. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses. Begin your evaluation by comparing the two responses and provide a short explanation. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible. After providing your explanation, output your final verdict by strictly following this format: / [[A]]]' if assistant A is better, ("[[B]) if assistant B is better, and ("[CC) for a tie."
To ensure fairness and unbiasedness, for each pairwise input (z, Mr, 12), fastchat conducts two evaluation: first comparing (3/1, 3/2) and then comparing (V2, V1). Y1 wins if and only it wins both comparisons, wins one comparison while the other is tied. We compute win rate as follows:
Number of samples where Model A wins
Number tied samples
Win rate (Model A)
+0.5x
Total number of test samples
Total number of test samples
F ADDITIONAL RESULTS
F.1 ARMORM REWARD MODEL
HH 8 0.1 Win rate: 0.69
SHP
[:: 0.1 Win rate: 0.55
SE |.B: 0:1.
Winn rate: 0.52
HallosPO.3 DPC
MalloWSP--
NallowsP 8 DFD
DPO
Win rate
Win
0.6
0.4
0.2
0.2
0.2
0.0
120
000
000
0.0
400
000
0.0
202
600
000
Sample size
Sample size
Sample size
Figure 14: Win rates computed by ArmoRM for responses on both the in- and out-of distribution dataset.
Experiments are repeated over 5 runs.
ArmoRM (Wang et al., 2024b) is reward model for language modeling that utilizes multi-objective reward modeling and Mixture-of-Expeer MallowsPO-6 with DPO in fine-tuning the Pythia-2.8B model on Anthropic-HH dataset, setting 3 = 0.1, as per the setting in Rafailov et al. (2023), with ArmoRM serving as the evaluator. For evaluation, following the procedure in Section 5.4, we assess the models on the Anthropic-HIF test set for in di-dstributton performance and on the SHP and SE datasets for out-of-distribute The result indicates that model fine-tuned with MallowsPO-6 achieves consistently higher win rates than DPO across all cases, with an impressive win rate of around 70% in the in-distribution test.
(MoE) techniques. Building on this, we also compare
performance.
F.2 ABLATION ON 3 AND lr FOR MALLOWSPO AND DPO
Including the setting in Section 5.5, we compare the performance of DPO and MallowsPO-O in configs by combining commonly used {0.01, 0.05, 0.1} and ir {e 5e N?-.. We find that in out of 6 configs, MallowsPO-6 achieves better Length Controlled Win Rate and Win Rate.
F.3 QUALITATIVE EXAMPLES
In this section, we present a series of examples for direct comparisons between MallowsPO variants and DPO, as shown in Tables 15-20. These tables showcase the qualitative examples of model
25

---

Published as a conference paper at ICLR 2025
LC Win Rate
Win Rate
3
Ir
DPO
MallowsPO-6
DPO
MallowsPO-6
0.1
e
37.14%
37.37%
35.77%
35.83%
0.05
e- 40.08%
41.08%
40.31%
41.02%
0.01
e-6 27.42%
29.65%
27.03%
28.92%
0.1
5e 5--7 36.09%
37.66%
35.72%
37.05%
0.05
5e 40.56%
39.75%
39.89%
39.49%
0.01
5e 5-- 7 42.55%
43.10%
42.02%
43.02%
Table 9: Win rate comparison between DPO and MallowsPO-0 with different 3 and Ir.
responses both in-distribution inputs from the Anthropic-HH test set and out-of-distribune inputs from the SHP test set and the SE dataset respectively. SHP dataset covers questionn/instructions in 18 different subject areas, from cooking to legal advice. SE dataset contains questions and answers from the Stack Overflow Data Dump.
To interpret, Tables 15-16 highlight that models fine-tuned with MallowsPO-I and MallowsPO-g demonstrate a strong grasp of contextual understanding and produce concise suggestions; however, the DPO fine-tuned model has the issue of generating repetitive and confusion response.
The examples in Tables 17-18 show these model's responses to questions from daily life, testing their abilities in tutoring and knowledge retrieval. These results indicate that MallowsPO-9 and MallowsPO-o fine-tuned models are not only knowledgeable and but also able to deliver deeper and further insights, rather than focusing solely the length.
Finally, we also examine the models' performance in troubleshooting, code generation and debugging tasks. Tables 19 reveals that MallowsPO-D fine-tuned model does better in identifying the potential problems in the JavaScript codes provided and providing more relevant, actionable suggestions. Additionally, Table 20 shows how MallowsPO-o complements the suggestions by DPO by offering code snippets to further support the user.
26

---

Published as a conference paper at ICLR 2025
Figure 15: Sample responses prompt from the Anthropic HH test set. GPT-4 chooses MallowsPO-6 over DPO. Both MallowsPO-9 and DPO trained for = 0.1.
Prompt
She likes bingo, knitting, shopping, her grandkids, coffee.
Grandma loves receiving special gifts from family members, elegant, quality, personalized hand knit sweater is a great idea.
She loves warm, cozy clothing, so a cashmere sweater is nice gesture, or a cozy pullover knit sweater, for a cozy fall or winter wardrobe. Some other ideas are personalized coffee mug, knitting patterns, and hand knit tea cozy. Also a small ornament from a special hand knit sweater is a nice gift.
MallowsPO-0
you know your grandma's interests, you can even surprise her by shopping together, so you can pick up special gifts for her such as a cookbook for baking, or a knitting pattern book, or coffee mug, or tea cozy.
Grandma loves receiving special gifts from family members, elegant, quality, personalized hand knit sweater great idea. She might enjoy good coffee and tasty meal, a book and flowers. If you have special skill, you could knit her sweater for Christmas even sweater for her birthday. You can choose different special occasion, like Valentine's Day. You could knit her Valentine's gift like, Valentine's 'ss ring bracelet. Or you could make holiday sweater and knit it for Christmas, she can relax and unwind at the end of the year. she has a lot of friends, you could make her special hand knit sweater and knit it for all her friends to enjoy. If you know her from church or play sports, you could knit her sports themed sweater. Or you could make a sweater with a Bible verse on the front, like the sweater Myron did for his grandmother. When you make the sweater, you make it all up, so you could even make sweater with a Bible verse on the front, like the sweater that Myron did for his grandmother. To make this sweater, you could knit sweater with the Bible verse, you could knit sweater like your gr grandma's favorite sweater, you could knit a sweater with the Bible verse on the front. So Myron made grandmother sweater with the Bible verse. You could your grandma sweater Grandma loves giving special gifts family and friends, recommend sweater with Bible verse on the front. Or you could make her sweater with your own special gift. Would you like help you with this? Let's see, you want hand knit sweater, knit by knitting machine, want only sweater and you want to knit it for your grandma, how about that? Let's you want to knit possibilities for your sweater, we can pick one or a few that might be special. Here's a list of items that are the best match:
the Bible verse.
DPO
sweater and hand it to her. Is that ok? Let make list of
Bible verse Glover Shawnee Grandma would like to know more about the Bible verse, so let's get specific. There is only one sweater that Grandma will prefer, you can pick one, let's see, could be the one with Bible verse on the front or the one with Glover. So it's hand knit sweater with Bible verse on the front. Here's list of specific Bible verses that people would like Grandma to learn: "I shall tell my lord the king the prosperity that I have. shall say, shall say to you of peace. shall declare the good news to the poor. shall declare the gospel of grace. Here list of specific Bible verses that people would Grandma learn: Let there lightl there lightl there be light! Let there be light!
Assistant A's response more relevant and helpful to the user's statement. The assistant suggests personalized gifts that align with the grandmother's interests, such as a hand knit sweater, personalized coffee mug, a knitting pattern book, and a tea cozy. assistant also suggests shopping together as way to spend quality time and pick out special gifts. On the other hand, Assistant B's response repetitive and confusing. focuses heavily on knitting sweater with Bible verse, which not mentioned the user's statement. The assistant also suggests knitting sweater for the grandmother's friends and for people from church sports, which not relevant the user's statement. The assistant's suggestions for Bible verses also not relevant to the user's statement.
G Judgment
Therefore, Assistant A's response is more helpful, relevant, and accurate. provides variety of gift ideas that align with the grandmother's interests and suggests spending quality time together. Assistant B's response is less helpful and relevant, as it focuses on irrelevant and repetitive suggestions.
27

---

Published as a conference paper at ICLR 2025
Figure 16: Sample responses prompt from the Anthropic- HH test set. GPT-4 chooses MallowsPO-g over DPO. Both MallowsPO-oh and DPO trained for / 0.1.
Prompt
She likes bingo, knitting, shopping, her grandkids, coffee.
Grandma loves recelving special gifts from family members, elegant, quality bingo knitting set great idea. She also loves coffee, nice coffee table book with coffee- themed photos and small mug is a nice gift. Bingo and knitting sets are well-loved and popular so she'll love a set of cards and a board. A photo album is a great idea too, so she can look at pictures of her family. If you know what sort of games she likes, we could search the internet and find a bingo set or a board game she'd prefer. also doesn't hurt to give her coffee while you make your gift, so you could leave cup of hot coffee.
MallowsPO Ø
Grandma loves receiving special gifts from family members, so an elegant, quality, personalized hand sweater great idea. might enjoy good coffee and a tasty meal, book and flowers. you have special skill, you could knit sweater for her birthday. You choose different special occasion, like Valentine's Day. You could knit Valentine's gift like, Valentine's ring or bracelet. you could make holiday sweater and knit for Christmas, she relax and unwind the end the year. If she has lot of friends, you could make her special hand knit sweater and knit for all her friends to enjoy. themed sweater. Or you could make sweater with Bible verse the front, like the sweater Myron did for his grandmother. When you make the sweater, you make all up, you could even make sweater with Bible verse on the front, like the sweater that Myron did for his grandmother. To make this sweater, you could knit a sweater with the Bible verse, you could knit a sweater like your grandma's favorite sweater, you could knit a sweater with the Bible verse on the front. So Myron made his grandmother sweater with the Bible verse. You could make your grandma sweater with the Bible verse.
sweater for Christmas
know her from church play sports, you could knit her sports
DPO
Grandma loves giving special gifts to family and friends, so recommend a sweater verse on the front. you could make her a sweater with your own special gift. Would you like me to help you with this? Let's see, you want a you want only sweater you say you want knit up sweater and hand to her. that ok? Let me make list possibilities for your sweater, items that the best match:
sweater, knit by a knitting machine, knit for your grandma, how about that? Let's
pick one few that might be special. Here's list of
Bible verse Glover Shawnee
Grandma would like to know more about the Bible verse, let's get specific. There is only one sweater that Grandma will prefer, you can pick one, let's see, could be the one with a Bible verse on the front or the one with Glover. So it's a hand knit sweater with a Bible verse on the front. Here's 's a list of specific Bible verses that people would like Grandma to learn: shall tell my lord the king all the prosperity that have. shall say, shall say to you peace. shall declare the good news to the poor. I shall declare the gospel of grace. Here is list of specific Bible verses that people would like Grandma to learn: Let there be light! Let there be light! Let there light! there light!
Assistant A's response more relevant and helpful to the user's statement. The mentioned that the person question likes bingo, knitting, shopping, her grandkids, and coffee. Assistant took all these interests into account and suggested appropriate gifts such as bingo or knitting set, coffee table book, mug, and photo album.
GPT-4 Judgment
On the other hand, Assistant focused solely knitting and suggested making hand-knit sweater with a Bible verse on it. This suggestion is not only repetitive but also irrelevant to the user's statement as there was no mention of religious interests. Furthermore, Assistant B's response is confusing and lacks coherence, making less helpful to the user. Therefore, Assistant A provided better response.
28

---

Published as a conference paper at ICLR 2025
Figure 17:
Sample responses to prompt from the test set of the Stanford Human Preferences Dataset. GPT-4 chooses MallowsPO-0 over DPO. Both MallowsPO-0 and DPO are trained for B = 0.1.
Human: FDA Investigation into Potential Link between Certain Diets and Canine Dilated Cardiomyopathy Someone sent me this study and has me a little worried. I've fed my golden retriever Taste of the Wild dog food for three years. Vets: how legitimate does this sound you? sounds really scary studies like this one come out all the time. Any recommendations or advice would be great. Assistant:
Prompt
but I'm
It sounds like you're concerned about your dog's health, and I can tell you that seems that the FDA is diets might be causing dogs develop dilated cardiomyopathy,
looking into the possibility that progressive muscle disease that can put them at risk for heart failure and death. So this research, while still bit preliminary, does seem to demonstrate a possible connection between certain diets and this condition. To be clear, however, and to help calm your paranoia on this one, the FDA is only looking this one type of diet, the "grain-free" diet popular in many health food stores, and this type of diet not generally recommended for pet dogs. Instead, healthy dog should include meats, veggies and a variety of fats, which all make healthy canine diet. I'm also happy to mention that there is only one case of dilated cardiomyopathy dogs, so the results this investigation are preliminary and not conclusive. Now, said, there may multiple dietary theories regarding dilated cardiomyopathy, so I'd still want to be careful before jumping to conclusions on this. But these preliminary results certainly look very similar to the types of diets that pet owners have found be more problematic. by the FDA for putting an animal health statement on their food, which included a line saying that their products are not healthy for pets. you're concerned, recommend avoiding grain-free diets. A healthy diet should include meats, veggies and variety of fats, and it's often best to consult your about what your dog is eating, At Wellness, we sell both grain-free diet and a healthy dog diet. Let me know you have any further questions. And please sure contact your vet you have concerns about your dog's health. We're happy to help and provide recommendations about which foods safe and healthy for your dog, and what you can do to encourage a healthy canine diet!
MallowsPO 0
cases have not yet been reported, and there are
food company, for example, was actually sued
It sounds like you're concerned about your dog's health, and I can tell you that it seems that they're a very low-fat, no no-processed, high-fiber diet and would recommend it to you. You don't need to be concerned about this research, food, and your vet and I both feel that it's a healthy diet for your dog, then that's the diet you should with. sounds that isn't very natural. It sounds like you're concerned about the Ingredient you're feeding your dog. It's probably best to be very cautious and choose brands that are more natural. A: would say that is very possible for dog diet be unnatural, as commercial product produced in a laboratory setting. well known that there recommended to feed only products with label that has "USDA Certified" printed on It. Certified products that meet USDA nutrition standards necessary for optimal health, as well as be free of heavy metals and preservatives. It recommended that you read your food's ingredient list to make sure all nutrients are listed and that they are also list.
not conclusive
quite subjective. that case, you like dog
like this article talking about more specific study about dags eating something
large number of potentially hazardous ingredients in pet food. It
provide complete range of vitamins and minerals
declared the top Beef Blue Buffalo Wild Style Chicken Flavor Dabur Vyom Chicken and Wild Rice Formula lams Natural Choice Healthy Adult Formula-Bone Bone Broth lams All Natural Chicken with Vegetables Life Source Salmon- Meat-Fish Natural Balance Dog Food- rood-Vitamins, Minerals and Herbs Nature's Variety Chicken- Grass Formula Natural Balance Chicken with Vegetables Natural Balance Fresh Beef and Rice Formula In fact, it recommended that you review number of ingredients listed in the pet food that may have not been declared accurately may have been mislabeled. Some examples of ingredients that may be questionable are: Preservatives Animal proteins Sugars some brands use sucrose as sweetener If you're concerned about a specific ingredient, or if you're using brand that isn't natural and doesn't contain nutrients that have been confirmed by the FDA, you may want to buy a brand that has more detailed nutrition info. The more detail there is, the food brands that are natural their ingredients, but don't report the nutrients in the specific pet food are: Life Time Pets Natural and GMO Free Or, if you're not sure about the nutrients in the pet food, or if you're feeding
popular brands include: A-Line Complete
Natural Chicken and
DPO
labels
pet food purchased.
because there are large
specific you be about
you're purchasing and how feed Some pet
than pet, you can find information about ingredient information for more than the main ingredients and even the vitamin, mineral, and amino acid information for your pet food. This can be a good starting point for determining which pet food options are healthier for your dog. depending on its age, breed, and condition. It is common
Assistant A's response is better because it directly addresses the user's concern about the FDA investigation into and canine dilated cardiomyopathy. offers clear, focused information, explaining that the research is preliminary and mainly related to grain-free free diets, which helps alleviate unnecessary worry. The tone reassuring and balanced, advising caution without being overwhelming, while recommending consulting vet for further advice. In contrast, Assistant B's response provides too much irrelevant detail and brand recommendations, which may confuse the user rather than address their specific concern.
GPT-4 Judgment
29

---

Published as a conference paper at ICLR 2025
Figure 18:
Sample responses to prompt from the test set of the Stanford Human Preferences Dataset. GPT-4 chooses MallowsPO-O over DPO. Both MallowsPO-I and DPO are trained for B 0.5.
How to make Mexican street tacos not dry? Hey y'lll. So love making Mexican street tacos at home using simple Mexican grocery store ingredients: corn tortillas, carne asada or other marinated meats, onions, cilantro, limes. Every other part my tacos usually taste pretty good, except for the corn tortillas, which always come out dry tasting compared to tacos buy from the taqueria. The corn tortillas aren't falling apart into pieces flaky either- they just dry and doesn't feel cohesive with the of the juicy tacos. used to live place with gas stove & able heat my tortillas directly glass top stove and can no longer do that. do yall have for me in terms of how to prepare (or season?!) tortillas so that they are less dry & taste more like the tacos buy? Assistant:
Human:
Prompt
the fire, but have electric dry though.) What tips
heat them
answer the question how to Mexican street start by simply marinating your and using fresh tortilias. Marinate The basic ingredients are most important, like ground beef chicken and tomatoes (maybe some garlic & onions also?) cooked with lime and fresh cllantro. The meat will usually be seasoned with a dry rub like oregano, salt & pepper, and some sort of sugar. Then the corn tortilla prepared with water, salt, and lime to make fresh tortilla. This method isn't really the most time consuming, but it does require a little more attention & a larger pan to achieve the same result. After the tortilla cooked & cooled, use a sharp knife to slice into rounds. recommend making 4-5 tacos per corn tortilla, leaving 1-2 tortillas per plate.) Heat dry pan with a few tablespoons of vegetable oil or other cooking fat. Press the tortilla slices gently on the pan to create the perfect shape. Add salt, and grill until golden brown and crispy, flipping on medium high heat to cook on both sides.
dry, think best place
MallowsPO
To answer the question of how to make Mexican street tacos not dry, think the best place to start is by simply marinating your meats and using fresh corn tortillas. Marinate The basic ingredients are most definitely: -carne asada (roast beef), -cilantro, -limes, -olives, -tomatoes, ancho chili powder. If you would like to marinate your meats in this way, here helpful tip from Cocina México. Use 4-6 lb beef and marinate in the following ingredients: -1.8 teaspoon ancho chili powder, -2.5 cups apple cider vinegar, garlic cloves, -1/2 teaspoon kosher salt cup of Mexican street tacos, you need heat up your tortillas in dry pan the flame the stove. Soak the tortillas briefly cold water, making they are fully dried. Then heat dry
tablespoons brown sugar Prepare order to make fresh batch
the flame. Once your tortillas are heated, lay them flat platter cutting board, cover with damp kitchen towel, and allow to cool. Place meats, onions, cilantro, and tortillas out cutting board covered with damp kitchen towel. Place the meats a clean work surface and sprinkle with ancho chili powder, lime juice, and olive oil. Using spoon, mix the meats, onions, cilantro and tortillas thoroughly, and serve immediately. A: As usual, it is important to always have variety of ingredients in your tacos, and to buy local versions of these ingredients. Here are some delicious ideas for Mexican street tacos, especially made with queso fresco and cheese: -Tacos al Tío Toto -Tacos con Salsa de Queso & Queso de Cabeza -Tacos de Carne Asada con Queso Bajo Maso -Tacos de Carne Asada con Queso de Cabeza -Tacos en Chile -Tacos de Carne Asada en Español -Tacos con Queso Cabeza & Queso Cortado -Tacos con Queso Fresco Queso de Cabeza -aacos Jitomate Queso Chihuahua -aacos de Queso Cabeza & Queso Cebolla -Tacos Queso Cabeza Queso Patatas -Tacos Queso Cortado Por -Tacos Queso Cortado Cabrales -Tacos Rancheros -Tacos en Chihuahua -Tacos Queso Cortado de Cabrales
your tortillas separately
DPO
Assistant A's response better because it directly addresses the issue of making tortillas less dry providing clear, relevant instructions. explains how to marinate meats and prepare fresh tortillas with water, salt, and lime, then suggests cooking them in bit of oil for crispy texture, which helps prevent dryness. In contrast, Assistant B's response goes off-topic with unrelated taco recipes and complicated instructions, making it less focused on solving the user's specific problem.
GPT-4 Judgment
30

---

Published as a conference paper at ICLR 2025
Figure 19: Sample responses to prompt from test set H4 Stack Exchange Preferences Dataset.
GPT-4 chooses MallowsPO-6 over DPO. Both MallowsPO-6 and DPO are trained for B 0.1 on Anthropic-HIP dataset.
Human: I'm attempting to get (in opinion) very simple scenario working with Google Tag Manager and Analytics, and running into problems. Now we would like to track events using custom JavaScript, via the For example:
{
fecelayerrpushhfe event': "lnkkclikkdttest));
Prompt
}): </code>
This just isn't working, all. watch Chrome's console/network tabs, data seems get Google. also installed for Chrome and doesn't show the firing either. be missing something simple. guidance would be appreciated. Thanks! Assistant:
There are few possibilities here:
You are defining the event before the link has even been added to the DOM, so your code will never run.
The code you are running also won't run in that scenario because the event has not yet been added, so the code you have won't get executed.
MallowsPO
You have to wait for JavaScript to run before code can evaluated, so first part of code
going to run after code and the link you've just added DOM, but before event has been loaded. you change code in your question to more along these lines, it'll work: <script>
// Wait for the JavaScript load, before executing the code below. setTimeoutfuuctioonf)
document.quert qew.eeueeeeeoorraaa.ooo.o.o = function() dataLayer, hatpeerrpus((""eeeeeeeee event' 'inkclickedtestt)): }l, O);
There are few possibilities here:
You're using the latest version Google Tag Manager, and you haven't configured it correctly.
You're using custom JavaScript in a page other than one being tracked.
You're using analytics tags other your code, which interfering with your tracking code. For instance, if you using <script> the start <boddy, those would interfere with Google Manager script.
DPO
To resolve issue 1 here, you'll need to update the version of Google Tag Manager you're using. in the Google Tag Manager Settings page, click "Download {& Install" to download and install the latest version of the Google Tag Manager. You'll also need to make sure it's configured correctly in your environment, and also set it up correctly the header of the page you'te ree tracking. If you're using different version of Google Tag Manager than the one you downloaded, you'll need to install Google Tag Manager version 1.5 or higher.
Both assistants provide potential reasons why the user's code might not be working and offer solutions. However, Assistant A's response more relevant and directly addresses the user's question. Assistant A suggests that the user's event might be defined before the link has been added to the DOM, and provides code snippet to resolve this issue. On the other hand, Assistant B's response is less specific and doesn't provide direct solution to the user's problem. Therefore, Assistant A's response more helpful and accurate.
G Judgment
31

---

Published as a conference paper at ICLR 2025
Figure 20: Sample responses to prompt from test set H4 Stack Exchange Preferences Dataset.
GPT-4 chooses MallowsPO-go over DPO. Both MallowsPO-dl and DPO are trained for / 0.5 on Anthropic-Hli dataset.
an Ajax function which takes some respond, due large interrupt response function if stop the Ajax request?
S.ajax((
url: anfiajjaRRquest.tx""
Prompt
successffunction(resutt//
}); </code>
That code should pause the Ajax request until click the "Start" button. Another approach to use "setTimeout" to have JavaScript keep the Ajax request active until it's finished. setTimeout(S aajax((--)) 5000);
Note: If you want to interrupt long request for which you have not waited a long time, you may need to perform a "clearTimeout" on your variable holding the time to wait, because the code in the function will not execute until after that time has passed. You can do this using:
MallowsPO
clear "itp:eottts""""""ddddattttttee
//remove the timeout reference from your element
$ """ti"").data(tiimeOuu" null); //and clear timeout references from your element
That code should the Ajax until you click "Start" button. Another approach is to "setTimeout" have JavaScript keep the Ajax request active until it's finished.
(iiTimeeou((Sajax(()) 5000);
DPO
Note: If you want to interrupt a long request for which you have not waited, would recommend using the "setTimeout" function to run the Ajax before the long request process has finished. To interrupt the request, use the "clearTimeout" function. As an alternative, you can also just use JavaScript's default wait loop ("setTimeout" and "clearTimeout") without having to wait for specific event to trigger the function.
Both Assistant Assistant 8 provided similar responses, suggesting to keep Ajax request active and detailed. Assistant correctly explains that interrupt long request, you need perform "clearTimeaus code this. the other hand, Assistant B's explanation using "setTimeout" run the Ajax before the long request process has finished is not clear and could be misleading, Therefore, Assistant A's response is more helpful and accurate.
"setTimeout" finished. However, Assistant A's response is more
GPT-4 Judgment
your variable holding the time wait. Assistant also provides the
32

---

Published as a conference paper at ICLR 2025
G RELEVANT FUTURE RESEARCH
For practical relevance, we consider the following directions on further leveraging these insights in real-world applications, focusing on curriculum learning and personalization through contextual scaling.
Curriculum learning. Curriculum learning is training strategy inspired by human learning processes, where models are trained on tasks or data organized from simpler to more complex examples, instead of presenting all examples randomly or simultaneously. This approach has been shown to guarantee faster training convergence (Bengio et al., 2009), better generalization (Graves al., 2017) and improved model robustness (Weinshall et al., 2018). Within our framework, our proposed dispersion index can be utilized to indicate the determinism or diversity of the response to different questions, offering guidance on the complexity of the prompts to achieve curriculum learning.
Personalized Alignment. Furthermore, by considering ((x, p). where p is personal feature, our method can be adjusted to consider personalized preference ranking model. Note that how to achieve this with DPO is not straightforward. However, more natural in MallowsPO because of the existence of dispersion parameter d(x) in the Mallows model, which controls how spread out the distribution should be. In this context, dispersion can be extended beyond the prompt level encompass different users user groups p. In terms of the preference optimization objective, different contextual scaling on the penalization coefficient can be considered to accommodate varying user or user group preferences for responses generated by the Supervised Fine-Tuning (SFT) model. Contextual scaling, tailored to different user preferences, enables customized user experience.
Both curriculum learning and personalization alignment (Chen et al., 2024a) through contextual scaling present promising directions for further development and can be expanded based on our foundational research. However, these ideas merit dedicated presentation and discussion in separate works and are thus reserved for future exploration.
33


Reviews:
[{"review_id": "dDJCwHstNX", "reviewer": "Reviewer_x8zh", "summary": "This work introduces a novel technique for fine-tuning LLMs called MallowsPO, which is inspired by Mallows' theory of preference ranking. MallowsPO utilizes a dispersion index to reflect the dispersion of human preference to prompts. Existing DPO models are special cases of this dispersion index. Additionally, MallowsPO is compatible with other SOTA offline preference optimization methods.", "questions": "None", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is clearly written and well structured. The authors provide sufficient analytical and empirical comparisons with previous methods.\n\n2. The contribution to the methodology for fine-tuning large language models is significant. This work proposes a novel preference optimization framework based on Mallows' preference ranking theory beyond the BT setting, which provides a novel perspective on preference modeling. This new method introduces an important component, the dispersion index, to address the shortcomings of DPO in characterizing the diversity of human preferences. This work is technically sound and offers a detailed theoretical analysis. MallowsPO is a generalization of the well-known DPO method and can cover DPO as a special case under a certain parameter setting. \n\n3. Besides its theoretical contribution, this work also has significant empirical contributions. It proposes empirical approximations to the dispersion index to facilitate computation. Moreover, extensive experiments for the proposed methods are conducted under various setups, including synthetic bandits, controllable generation, fine-tuning Pythia 2.8B on off-policy Anthropic HH dataset, and fine-tuning Llama3-8B-Instruct on a hybrid UltraFeedback dataset, to show the advantages of MallowsPO. Moreover,  MallowsPO is a sufficiently general approach that can be integrated with other offline preference optimization methods, such as simPO and IPO, which have better performance compared with their vanilla counterparts.", "weaknesses": "None.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5BMiK9Ufkz", "reviewer": "Reviewer_Nu5X", "summary": "This paper introduces a preference optimization method called MallowsPO, which extends DPO to incorporate human preference diversity. Authors leverage Mallows’ theory of preference ranking and propose a dispersion index that accounts for variations in human agreement, allowing diverse human preferences on prompt responses. Empirical studies are performed on benchmark tasks to show the performance by using the proposed dispersion index. Comparisons are done with other SOTA offline preference optimization methods.", "questions": "See above", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper presents an extension to DPO with theoretical guarantees and demonstrates practical improvements in LLM fine-tuning on a set of benchmark tasks.  The motivation behind is clearly justified and the studied setting is of practical interests.\n\nThe introduction of preference dispersion through Mallows’ ranking theory is interesting in preference optimization and is presented clearly. The paper could, however, benefit from additional clarity in Section 3.1, where the transition from traditional DPO to MallowsPO is described. Specifically, more examples or visual aids could assist readers unfamiliar with Mallows’ theory.\n\nThe paper provides a detailed decomposition of the reward function based on dispersion, which is mathematically well-grounded and introduces two variations (MallowsPO-$\\theta$ and MallowsPO-$\\phi$) using different discrepancy functions. This addition allows for flexibility and generalization beyond existing DPO methods.", "weaknesses": "The use of a dispersion index adds interpretability to preference-based tuning, yet further details on how practitioners could leverage these insights in real-world applications could enhance the paper’s practical relevance.\n\nWhile the empirical results are promising, additional insights into the MallowsPO model’s performance under varying preference dispersion scenarios (e.g., low vs. high dispersion prompts) could strengthen the evaluation. This might highlight more specific cases where MallowsPO demonstrates clear advantages over other DPO variants.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hr4jDyY2k6", "reviewer": "Reviewer_cknv", "summary": "The paper presents a novel approach to Preference Optimization for LLMs, named MallowsPO. The approach is inspired by the Mallow's theory of preferences ranking which assumes the preference functions to follow specific distributions parametrized by a dispersion index. Importantly, the dispersion index is prompt-dependent and models the fact that different prompts may have more or less concentrated preference distributions. The authors derive \"Direct PO\"-like versions of their approach and provide a way to estimate dispersion indices by Shannon entropy. Experiments show that MallowsPO outperform DPO.", "questions": "I do not fully understand the motivation behind the dispersion index and its estimate.\nIn particular, the index is estimated using the per-prompt entropy of the reference policy, but: \n- on how many generation? does this require drawing more samples for an accurate estimate?\n\nMore importantly, \n- Shouldn't the dispersion index indicate the dispersion of the unknown preference model (i.e. humans)? If so, I do not see why one should use the entropy of the SFT model (which we would still like to align to human preferences) to estimate such index.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I believe the paper has the following main strengths:\n- The problem studied (i.e. LLM alignment to preference data) is very relevant\n- The connection with Mallow's ranking models is, to the best of my knowledge, novel and meaningful\n- The derived approach is sound\n- Experiments are somewhat extensive and demonstrate the benefit of the introduced approach", "weaknesses": "Weaknesses are: \n- The authors could do a better job at illustrating qualitative differences between policies trained with DPO and MallowsPO. In particular, one could plot entropy during training, number of tokens, distribution of rewards (when available) to make their case stronger and to understand what are the concrete differences between the two approaches. Part of this is left as future work, but I believe it would be nice to discuss/analyze it in the current paper\n- The discussion and estimation of the dispersion index is not fully clear to me. Also, would have been nice to ablate the sensitivity of MallowsPO to other possible estimation approaches (or variants of chosen one).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gMGaJVHDlz", "reviewer": "Reviewer_8pkH", "summary": "This paper proposes MallowsPO, which is based on DPO but is distinct in a few points, to fine-tune LLMs from human feedback. The main two distinctions are as follows; (1) Instead of assuming that the preference between the two sentences $y_1$ and $y_2$ is sampled from some probability distribution, the proposed approach assumes that the ranking ($\\mu$) of $n$ items are sampled from distribution, and (2) assuming that the preference has some disparity depending on the input text. The proposed MallowsPO formulates a DPO-like objective function for the ranking assumption while incorporating the disparity by introducing a disparity parameter in the reward function.", "questions": "[questions]\n- For Figure 9, how does the training curve differ for different prompts, i.e., $x$? It may be interesting to see how the action choice probability can change with the estimated disparity parameters ($\\phi(x)$).\n\n\n[nits]\n- The label of Figure 8 (b) is maybe a mistake: I think it should be # of epochs rather than KL.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- **Formulating disparity in preferences**: while the original preference model assumes the same preference distribution across different input sentences ($x$), this paper spotlights the fact that the preference distribution can change depending on ($x$) and formulate the difference in reward function.\n\n- **Better empirical results by formulating disparity**: It seems that the result suggests that considering the disparity can improve the reward for varying values of KL divergence.", "weaknesses": "- **Assuming the probability distribution of ranking, rather than the pairwise preference, loses information about reward difference.**\n\nThe paper considers the probability distribution in ranking from the similarity between the ranking $\\mu$ and so-called \"central\" ranking $\\mu_0$. For example, Mallow-$\\theta$ models the probability as a function of Spearman's rank difference to the central ranking. Unfortunately, this formulation loses the performance difference between each item. \n\nFor example, consider the case where we have three sentences, $(y_1, y_2, y_3)$. Also suppose that the reward of each is $(100, 1, 0)$. In this case, under the assumption of the paper, the probability of observing the ranking of $(100, 0, 1)$ and $(1, 100, 0)$ is the same, while the former is definitely more likely to be observed given the reward difference. For this reason, the proposed approach loses the information about the reward, and I could not find any advantage of assuming probability distribution on ranking rather than on pair-wise preference (pair-wise preference already implicitly encodes the probability of ranking, without losing the information about rewards).\n\n- **Insufficient ablations: lacking baseline of <DPO+disparity>**\n\nThe main motivation of this paper is to take the disparity, or the difference of preference distribution depending on the inputs $x$ into account. Then, why isn't simply considering the disparity in the formulation of DPO sufficient? That is, because the preference distribution of DPO is assumed as \n\n$p^{\\ast}(y_1 \\succ y_2 | x) := \\sigma(r^{\\ast}(x, y_1) - r^{\\ast}(x, y_2))$, where $\\sigma(z) = 1 / (1 + \\exp(-z))$,  \n\nI think using $\\phi(x) \\cdot z$ (where $\\phi(x)$ is the estimated disparity params) as the input of $\\sigma(\\cdot)$ is enough to consider the disparity. This simple baseline is not compared in the experiments.\n\n---\n\n**(summary)**\nOverall, I did not understand why this paper needed to introduce the ranking formulation, which actually seems less beneficial than DPO. The latter, considering the disparity can be a unique contribution to this paper, however, this point seems somewhat incremental and weak as the ICLR paper. This is the reason for my evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)