Beyond Pairwise: Empowering LLM Alignment With (Ranked) Choice Modeling
Authors: Yuxuan Tang
Yifan Feng
Venue: ICLR 2026 Poster (2026)
Keywords: Language Models Fine-tuning
Discrete Choice Model
Ranked Choice Model
Alignment
Preference Optimization
Learning From Human Feedback
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Alignment of large language models (LLMs) has predominantly relied on pairwise preference optimization, where annotators select the better of two responses to a prompt. While simple, this approach overlooks the opportunity to learn from richer forms of human feedback, such as multiway comparisons and top-$k$ rankings. We introduce \textit{Ranked Choice Preference Optimization} (RCPO), a unified framework that bridges preference optimization with (ranked) choice modeling via maximum likelihood estimation. RCPO supports both utility-based and rank-based models, subsumes several pairwise methods (such as DPO and SimPO) as special cases, and provides principled training objectives for richer feedback formats. We instantiate this framework with two representative models (Multinomial Logit and Mallows-RMJ). Experiments on Llama-3-8B-Instruct, Gemma-2-9B-it, and Mistral-7B-Instruct across in-distribution and out-of-distribution settings show that RCPO consistently outperforms competitive baselines. RCPO shows that directly leveraging ranked preference data, combined with the right choice models, yields more effective alignment. It offers an extensible foundation for incorporating (ranked) choice modeling into LLM training.

Published as a conference paper at ICLR 2026
BEYOND PAIRWISE: EMPOWERING LLM ALIGNMENT WITH (RANKED) CHOICE MODELING
Yuxuan Tang
Yifan Feng
Institute of Operations Research and Analytics National University of Singapore yuxuan. tanggu. nus edd
Department of Analytics and Operations NUS Business School y fan, fengênus edu. sg
ABSTRACT
Alignment of large language models (LLMs) has predominantly relied on pairwise preference optimization. where annotators select the better of two responses to a prompt. While simple, this approach overlooks the opportunity to learn from richer forms of human feedback, such as multiway comparisons and top-k rank- ings. We introduce Ranked Choice Preference Optimization (RCPO), a unified framework that bridges preference optimization with (ranked) choice modeling via maximum likelihood estimation. RCPO supports both utility-based based and rank- based models, subsumes several pairwise methods (such DPO and SimPO) special cases, and provides principled training objectives for richer feedback formats. We instantiate this framework with representative models (Multinomial Logit and Mallows-RMII, Experiments on --8--------- Gemma-2-9B-in and Mistral- In-Inrruct across in mn-iisributton and nut-of-distribution settings show that RCPO consistently outperforms competitive baselines. RCPO shows that directly leveraging ranked preference data, combined with the right choice models, yields more effective alignment. It offers an extensible foundation for incorporating (ranked) choice modeling into LLM training.
INTRODUCTION
Large language models (LLMs), a prominent form of generative AI, have rapidly transformed human- computer interaction, powering applications from open-ended dialogue (Thoppilan et al., 2022) and content creation (Brown et al., 2020) to code generation (Chen et al., 2021; Li et al., 2023) and healthcare decision support (Nori al., 2023). A key driver of this success is alignment: training models to produce outputs that are factual, helpful, safe, and aligned with social norms.
Reinforcement learning from human feedback (RLHF) (Ziegler et al., 2019; Stiennon al., 2020; Ouyang et al., 2022) has emerged as the dominant paradigm for aligning Al systems, exemplified by ChatGPT and GPT-4 (Achiam et al., 2023). More recently, Direct Preference Optimization (DPO) Rafailov et al. (2023) has achieved comparable results through a simpler and more efficient objective, and has been used fine-tune LLMs such as Llama-3 (Dubey et al., 2024) and Zephyr (Tunstall et al., 2023). The success of RLHF and DPO has spurred surge of research, resulting in numerous extensions (Zhao al., 2024; Winata al., 2025).
Despite their differences, most alignment methods rely pairwise preference data, where for each prompt x, a preferred response yo and a dispreferred response Y) are selected by human annotators or Al judges. In practice, however, preference feedback is often richer than simple pairs. Annotators may provide partial rankings, top-k selections, or single-best judgments from larger candidate set. Current approaches typically reduce this richer information to pairs-c.2g in training InstructGPT. Ouyang et al. (2022) collected rankings of K responses per prompt but converted them into all (K) pairs; in many academic alignment studies (Meng et al., 2024; Chen et al., 2025; Zhao et al., scored by reward model, but only the highest-
2024; Gupta et al., 2025), multiple responses and lowest-scoring are kept. These reductions, while convenient for pairwise-bard algorithms, risk distorting the original preference structure and discarding potentially valuable information.

---

Published as a conference paper at ICLR 2026
Pairwise Feedback
Single-Best Foedback
Top-k Feedback
"'eeecee pave dool
LM
maximum
macemmm
iBollhood
Figure I: Ranked Choice Preference Optimization (RCPO)
To address this gap, we propose Ranked Choice Preference Optimization (RCPO), general framework that generalizes pairwise preference alignment to ranked choice feedback (see Figure 1). Rather than restricting evaluators to comparing only two responses. RCPO presents a of candidates and asks them to choose either the single-best or the top-k responses for given prompt. Our approach grounded in the theory of (ranked) choice models, particularly discrete choice modeling, which have been extensively studied in psychology. marketing, economics, and operations research. To the best of our knowledge, RCPO is among the first attempts to systematically apply (ranked) choice modeling to LLM alignment.
The main contributions this paper are threefold:
We establish a systematic connection between LLM fine-tuning and choice modeling, showing that fine-tuning can be essentially reduced to maximum likelihood estimation (MLE) of choice models. Building this insight, develop RCPO principled extension of pairwise preference aligmment that directly incorporates ranked choice feedback. RCPO general and flexible framework: any choice model that satisfies certain regularity conditions can be integrated. By preserving the richness of the original annotations whether single-best or top-k preferences RCPO mitigates the information inefficiency in pairwise conversion and enables more effective alignment with human intent.
(1) Conceptual Framework:
(2) Concrete Examples: We showcase how two broad classes of choice models, i.e., utility- or rank-based, can be accommodated within the RCPO framework. We then instantiate RCPO with representative model from each class: the Multinomial Logit (MNL) model (McFadden, 1972) for utility-based choices and the Mallows-RMI model (Feng & Tang, 2022) for rank-based choices. For both models, we derive alignment objectives under single-best and top-k settings (see Table 1). We also use gradient analysis to shed light on the theoretical underpinnings of these preference optimization methods.
Table 1: Various Preference Optimization Objectives in RCPO
Choice Model
Preference
Objective
Pairwise (DPO)
log (x, Buy v11)
MNL
Single-Best
log o log En sesestow exp (Jols, Ni.)).)))
Top-k Choice
Σh. log o log 2nyest exp (Fo Vy. wi))
Pairwise Single-Best
log ((x) o (fo wi Mw))
Mallows-RMI
log 6(x) m. Soc.ss.tn (mmr) o (Jo vi.
Top-k Choice
log ((z) 2 1 (1$) (foo2, (1)) +
En ES (f.
Notes: fa, 91, (1)
log (o
To
log Tu(y2lz) (v2]))
We evaluate RCPO on state-of-the-art instruction-tuned LLMs (Llama-3-8B- Instruct, Gemma-2-9B-it. and -istrall)B-IInttruee)) using in-distribution test and widely adopted out-of- distrriituutinn benchmarks (AlpacaEval 2 and Arena-Hard), Results consistently show that RCPO improves model performance and demonstrates flexibility across base models, preference feedback, and evaluation settings. We highlight the Mallows-RM. RMJ-based preference optimization, which achieves strong results under both pairwise and ranked choice setups.
(3) Experiments:
2

---

Published as a conference paper at ICLR 2026
Collectively, these contributions position RCPO a principled and practical framework for leveraging ranked choice feedback to advance the alignment of large language models.
A CHOICE BASED ALIGNMENT FRAMEWORK
2.1 RLHF AND DPO
We start by recapping the key concepts in RLHF and DPO. Let x denote an input prompt and denote a candidate response. A language model is parameterized by a policy Te, where To x) represents the probability of generating response y given prompt
RLHF comprises three sequential phases. First, fine-tunes a pre-trained LLM through supervised learning supervised data and produces an SFT model, which
the reference policy Tref- Second, RLHF fits reward model r" (x, y). which can be a neural network itself, based on a separate pairwise preference dataset D {(x((), You (5) v1" ) Here 2:(1) represents the prompt provided to annotator i, and you and vi. are the preferred and dispreferred responses, respectively. Third, the LLM is further fine-tuned reinforcement learning maximize the regularized expected reward:
max Ez=D [ [r" (z, v) - 3 KL( (- xx) || Tref(t [x)]]
(1)
where 3 >> O is hyperparameter controlling the deviation from the reference policy Tref.
While RLHF has shown impressive results, both reward model fitting and reinforcement learning require substantial computational effort. In this light, DPO analytically solves (I), yielding a closed- form relationship between the optimal policy and the reward model given by:
r" (x. y) Blog
Tu- (ulz) ae (via)
Blog (zz).
where Z(x) = Ey (y x) exp(r" (z, 3)/3) the partition function. As a result, based on a Bradley-Teryy preference assumption on how the preserred/dispreferree responses are generated, DPO consolidates the last two steps of RLHF into direct optimization problem with the following loss function:
min DDP (To: Tret) = E (m. log (3 log
(o at)
3 log ( )}
mo (ar)
(3)
where (--) is the sigmoid function. In other words, DPO directly fine-tunes To to match human pairwise preferences in a single step, thereby greatly reducing computational overhead.
2.2 (Ranked) CHOICE MODELING
Discrete Choice Models. We also introduce the key concepts for (ranked) choice modeling, First, rich body of research across economics, marketing, and operations research has developed a variety of discrete choice models represent human preferences. Let A {y1, 1/..... Vn denote the universe of items. A discrete choice model specifies the probability of selecting an item from an assortment S C ' under given context x. denoted by PPyy [S ::).). This is abstraction of many business and economics use cases. For instance, in retail settings, items typically correspond to products, assortments represent the sets of available products the time of choice, and the context encompasses covariates such as prices, promotions, or product features. There are many ways define discrete choice model, which essentially reduce specifying the probability function P(y S 5z).
Ranked Choice Models. Discrete choice models can be extended to richer feedback in the form of ranked choices. We adopt the ranked choice framing of Feng & Tang (2022). Let j y1 > l/2 > ... . y/ with (y1,..., } C S denote a top-k list of items from A ranked choice model defines P(u S ;)), which specifies the probability of observing the partial ranking j² under context This notion of ranked choices generalizes several common feedback structures, such as (i) pairwise comparisons; (ii) discrete choices (or multiway comparisons); (iii) listwise feedback (or full rankings over the items in any given assortment); (iv) top-k rankings over a full item set, to name a few. For example, when k 1, ranked choice model reduces to discrete choice model.
Assumptions. this paper, we will put (ranked) choice models in the context of LLM alignment, and focus on those that satisfy the following assumptions:
3

---

Published as a conference paper at ICLR 2026
Assumption Al: Reward sufficiency. There exists real-valued reward function & y IR and a mapping such that, for every S and x,
F(u" 8 ;x) = , (. v) byes). for every n² = 1/2 > within S. ,,,
That is, the effect of x: and S on ranked choice probabilities enters only through the item rewards (r(x, yES. Note that in the special case of discrete choices, the condition above can be simplified to P(y S ;2) = g(v, S, ( r(z, y²) hrss), for every y E S.
Assumption A2: MLE estimability. Given observations D = {(z,, St, u Ji are identifiable up to the usual invariances (e.g., additive shift and positive scaling) and admit tractable log- likeliioloooo function
)) the rewards
Σn log g St, (r(2,,)) y) hess.)
Here the meaning of "tractable" will be clearer later. In short, the log- -ikkliiikoood function should admit simple gradients to be passed to the training of no.
As can be easily seen, many choice models satisfy the assumptions above. In Sections 2.4 and 2.5, we present two approaches: utility-based models and rank-based models.
2.3 RCPO: CONNECTION BETWEEN LLM ALIGNMENT AND RANKED CHOICE MODELING
Our framework starts with a conceptual insight into LLM alignment: if interpret prompt the context, a candidate response y as an item, and set of candidate responses as an assortment S, then every choice model offers a distinct way to incorporate annotators' preference feedback via MLE objective. For instance, consider the case where = {yw. yt J. and let g denote the Bradley-Terry choice rule. Then the probability that you is preferred over y is given by:
P(y (yw 31 x)) g(Me, (vw, 31 (r(zz yw), w)) (r(2,, Mw) rxx, 2)),
where denotes the sigmoid function. Substituting the reward function defined in (2) into the Bradley-Terry likelihood yields the DPO objective described in (3). This establishes that DPO is special case of our formulation, where preferences follow the Bradley-Terry pairwise comparison model.
The RCPO Framework. Motivated by these connections, we introduce general framework for preference optimization grounded in choice model theory. Specifically, we extend DPO from pairwise Brarley-Terry comparisons to arbitrary ranked choice models that satisfy the assumptions outlined above. this framework, once the functional forms of r(æ,8) y) are specified, and the choice rule is determined by ranked choice model, the corresponding preference optimization procedure is defined by the following maximum likelihood estimation (MLE) objective:
log (uti, Si, (r (Zi v) ves.)
max 37
where T're (2, (2,8) y) is reward function derived from the policy πe.
Beyond the reward function (2) used in DPO, the literature has proposed many alternative definitions. For example, Wang et al. (2023) introduces f-divergence generalizations, while Meng et al. (2024) and Gupta et al. (2025) propose length-normalized log-likelshoods. These alternative reward formulations can likewise be incorporated into the RCPO framework. Consequently, methods such as R-DPO (Park et al., 2024), SimPO (Meng et al., 2024), and AlphaPO (Gupta al., 2025) can also be viewed as special cases of RCPO. In particular, while these approaches adopt the Bradley-Terry choice rule, they differ the functional form of the reward r(x, y). Although our framework can accommodate a wide range of reward functions, for clarity, we restrict attention to the reward defined in (2) thereafter in the paper.
What truly demonstrates the versatility of RCPO is how it gives birth to new preference optimization methods. In this paper, we consider two prominent classes of choice models: -bieet models, which rely the numerical magnitudes of the items' utilities, and rank- based models, which depend on the rankings of the items. Despite using different representations of the underlying preference distribution, they are both compatible with the unified RCPO framework.
4

---

Published as a conference paper at ICLR 2026
2.4 UTILITY-BAN BASED CHOICE MODELS
The random utility model (RUM) perhaps the most widely studied class of discrete choice models. Originally proposed by Thurstone (2017), it has been extensively developed in the economics and operations management literature (Anderson et al., 1992; Train, 2009). RUM assumes that every item y/ E W comes with a utility, which takes the form Vy Vu Ey: where Vy, is the mean utility, and Ey. is an exogenous random utility shock term. In this paper, we also write Uy; and Vy, in the form of Uy, (z) and Vyy (a:) to emphasize that they can be
Given the mean utility vector v(x) = (V', (ar),.. ................................ Py, (x)) and a distribution f over the utility shocks & (Eg1....,, the probability of choosing alternative from an assortment S C N is
P(y 5 / 14 (x) V4j (a) > F - Eys My E E (u))f(()) de.
(4)
RUMs are categorized by the distribution of their stochastic terms. The multinomial logit (MNL) model, introduced by McFadden (1972), assumes i.i.d. Gumbel noise. Alternatives include the probit model (joint normal distribution (Daganzo, 2014)), the nested logit model (correlated extreme value distributions (McFadden, 1980)), and the exponomial model (negative exponential distributions (Alptekinoglu Semple, 2016)).
The RUM can be extended to a ranked choice model, where the probability of observing the top-k ranking μ.at = y1 > U2 > > y/k: from an assortment
II Kf (vc vv (4) Ev, Ever Vy, E S') (vn....,.e)) we3] f(e) de. (5)
P(p xz)
As (4) and (5) show, the choice probabilities depend only the mean utility vector vxxr), which corresponds to the reward vector in our framework. Thus, any RUM that admits MLE estimation of v(x) can be incorporated into LLM fine-tuning.
2.5 RANK-BAR CHOICE MODELS
While utility-based models operate on numerical utilities, rank-based models represent preferences as complete orderings over N, depending only on the relative positions of items'. See Jagabathula & Venkataraman (2022) for a survey of such models.
The Mallows-type model (Mallows, 1957; Fligner & Verducci, 1986) among the most widely used classes of ranking models. Let 6, be the set of permutations of W. A Mallows-type model assigns probability distribution over permutations μ 6,, based on their distance from a central ranking Po:
(p) == Enp' dillopie))
/A E Ga,
where .(.,)) is a distance function between permutations, & E (0,1) is a dispersion parameter. Intuitively, rankings closer to l40 are exponentially more likely. Different distance choices yield different variants, such as Kendall's Tau (Mallows, 1957), Spearman's 's rank and footrule (Diaconis & Graham, 1977). Hamming distance (Bookstein et al., 2002), Cayley distance (Irurozki et al., 2018), and Reverse Major Index (Feng& & Tang, 2022). To capture context dependence, we parameterize the central ranking and dispersion as functions of x. denoted (-o(-lz)) and Φ(x). For readability, we may suppress x in the notation when its dependence is unambiguous.
For ranking μ E Gn and an item y, denote μ (y) as the position of y in μ (smaller means higher preference). Given an assortment S C N and a top-k ranking jutt = y1 > y2 > ... > ye with (y1,...,hk) .uk...,k]] C S, the implied ranked choice probability of observing μ² from given by
P( [8;z)= :5:))=EEa
MEB. Po(e).po((e))a())) 1H,,",
(6)
where I/ S}
1 if μ ranks [yI,..., Wbk in the specified order, and each of them ranked above all remaining items in S. or more formally, IfH, "..
{f (a-16)))a--(
(13) Yy, (v........ vEE When k = this reduces standard discrete choice probability. If d((x) is known, the choice probability depends only on Ma(-|æ), which can be represented as a vector of normalized ranks. Consequently, any Mallows-type model that supports MLE estimation of po (-((z) can be embedded within our framework.
Although technically, an RUM can be transformed into distribution of rankings, not every RUM can necessarily be transformed into Mallows-type model. Therefore, we treat them separately in this paper, with different ways to parametrize the choice prohabilities.
5

---

Published as a conference paper at ICLR 2026
3 TWo EXAMPLES OF THE RCPO FRAMEWORK
In this section, we consider two representative examples: the Multinomial Logit (MNL) model as a utility-based choice model, and the Mallows-RMI model rank-based choice model. These models are selected for the simplicity of their choice probabilities and the tractability of MLE. For each model, we derive the corresponding objectives for both single-best and top-k feedback, demonstrating the framework's flexibility across diverse preference structures.
3.1 MULTINOMIAL LOGIT MODEL (MNL)
Discrete Choice. If we take the random shock e to be i.i.d. Gumbel, we recover the Multino- mial Logit (MNL) model (McFadden, 1972), arguably the most widely used RUM. The corre- sponding choice probabilities in (4) admit simple closed-form expressions, given by P(yi|S; x) = (x),())/ / E/S (²0,(=) As such, it naturally extends the Bradley-Tery model from pairwise compar- isons to multi-item choice settings. By representing the mean utility as the reward defined in (2), we have the following theorem.
Theorem (MNL-PO-Discrete) Suppose the underlying single- best choice preference distribution follows MNL, the corresponding policy optimization objective is given by:
me(aul=) min DD log log Σ 1. exp (3 log (ece(Mile) -33logg T6 (z)
Reef 13))
(7)
As in vanilla DPO, 3 controls the deviation from the reference policy. When empirically solving (7), the size of each prompt's s assortment is allowed to vary, enabling the sampling of additional responses, and thus finer-grained preferences, for certain prompts to further enhance alignment. Similar objectives appear in Ziegler al. (2019) and Chen al. (2024), where they are treated as softmax loss functions rather than being interpreted through the lens of discrete choice model.
Top-k Ranked Choice. shown in Feng Tang (2023), the MNL model also extends to simple ranked choice model. Specifically, the probability that a top-k ranked choice 1* V1 > 1/2 > yn chosen out an assortment 5 S W given by
P(u 15
"u, (x)
(8)
Ej + +vv cs me) " (x)
Hence we can write down the corresponding policy optimization objective as follows.
Theorem (MNI-PO-Top--) Suppose the underlying top-k choice preference distribution follows (8). the corresponding policy optimization objective is given by:
min -E MND
loga( (ogo(-log Σ 25 ( exp (oog (ur(())(z)
so(uylz)
xx(uulls) -glog
Jewf (3g ))
i=1
3.2 MALLOWS-RMI MODEL
Discrete Choice. A notable challenge in applying Mallows-type models to (ranked) choice model- ing that the ranked choice probabilities in (6) are usually difficult to obtain, since the sum taken over all permutations. In this regard, an exception is Feng & Tang (2022), who adopt Mallows-type model using the Reverse Major Index (RMJ) as the distance function. They show that, unlike other Mallows-type models, the Mallows-RN RMJ distribution admits a closed-form expression for the choice probabilities derived from (6):
P(y: S;x) 1+0(2)+ (1)
where d(y,, (): ->:= H (yr |x) > Ho 1 (wy })} equals the number of items in S that
ranked higher than y/ according the global ranking mol æ)). In other words, d(y:, 5) defines the relative ranking position of item y, within the assortment S, so that its choice probability decays exponentially with its rank position in S.
notable feature of this model is its exclusive dependence on ordinal information. For instance, when the assortment size is restricted two, the model reduces to the classic noisy comparison

---

Published as a conference paper at ICLR 2026
model, in which the superior item is chosen with fixed probability 1/(1+ (1[++)(x))) ((x)), independent of the absolute difference between the options. This is in contrast to the Multinomial Logit (MNL) model, where choice probabilities are directly tied to the cardinal utilities of items. This reliance relative rankings, rather than precise utility estimates, provides the Mallows-RMe model a form of robustness against model misspecification and noise in preference feedback. Such robustness may help explain the model's favorable empirical performance observed in our experiments (Section 4).
We next derive the corresponding policy optimization objectives. By normalizing the negative rank ==(U)=) + 3 log (z), we have the following.
as the reward defined in (2), i..., -Ho (ylz) = 3 log [
Theorem
) Suppose the undertying single-best choice preference distribution follows (9), the corresponding policy optimization objective is given by:
min S.g, log ((x) >
T(B log
melyul
Bogg
Jely x) ( < 0}}.
(10)
(uul)
m
We also derive the alignment objective for the pairwise comparison case, with details in Section C.
Top-k Ranked Choice. The top-k ranked choice probability under the Mallows-RMa model is also given by a simple expression:
Pk * 1
=((()-)()(0)) ((x)duu..)) ((S|A())
(11)
where d(x
E ( (os x) MD (Mi+1 x)) (JS)
i)
Eyy...( [6 (uk 12) Mo (y5 x and (6 ((x)) = IT= (1+ (1....+ ...... ((z)i-1). The
corresponding policy optimization objective is thus given by the result below.
Theorem (Mallows- Bisllows-RM-PPOL HMMJ-PO-TOp-K Suppose the underlying top-k choice preference distribu- tion follows (11), the corresponding policy optimization objective is given by:
k-1 ((SI- ())1-+)
min
E Sa log ((x)
1) Blog
relvil=) (uor(uiaa)
Blog
<0}
(
Σ
1/81og Blog
malyelm) [
log ma(a,(m) (agl < =o)]
(12)
To make Mallows-RMi objectives practical for LLM training, we also overcome two challenges along the way: estimating the unknown dispersion parameter Φ(x) and handling the step functions in the objectives that hinder optimization. For the first challenge, we follow a similar approach to Chen et al. (2025) and adapt the entropy proxy of - log Φ(x). For the second challenge. step functions by replacing them with sigmoid approximations, which preserve the preference-basee structure while yielding smoother, more informative gradients. More details in Section D. The resulting objectives are summarized in Table 1.
smooth the
3.3 GRADIENT ANALYSIS
To gain deeper mechanistic understanding of the RCPO framework, we analyze the gradient structure of its loss functions. We focus here on the gradient of the Mallows-RMJ-PO-TT objective representative example. Full derivations and gradient expressions for other settings are deferred to Section The gradient with respect to model parameters 0 is given by:
Mallooss RMJ-PO Top- (mo)
BE log (xx)
(5)- vf) (ff 2011, 24)
greater weight For Insi-dispersion peompts
greater weight for highes ranks
grenter weight when rewards similar
Vo To (x)
Ve log To x))
disconangg 1
enoourage
)
Σ
o (folz, - (fo vn)
Vo log To (x)
Vo log (ou xx)
My (k)
comparisoo difhoulty
discomunge M/
encourage yk

---

Published as a conference paper at ICLR 2026
where the expectation is taken over ranked choice triples (x, S. D. Intuitively, this gradient update increases the likelihood of higher-ranked responses while decreasing that of lower-ranked ones. The magnitude of the update is amplified when: (i) the prompt context exhibits low dispersion (i.e., more confident preferences); (ii) the response occurs higher-ranked position; and (iii) the reward estimates drive the model to sharpen distinctions near the top of the ranking and better align its outputs with fine-grained preference structures.
close, indicating a more informative comparison. As such, these properties
4 EXPERIMENTS
In this section, we assess our methods on several widely used evaluation protocols. Additional experimental details are provided in Appendix
4.1 EXPERIMENTAL. SETUP
We adopt Mamaa--8B-Instreeee Gemma-2-9B-it and Mistral-7B-Instenn as our fine-tuning bases, as they are widely used flagship instruction-tunee models that represent the state-of-the-arr. We generate multiple responses to each prompt in the UltraFeedback train dataset (Cui et al., 2023) and use the Skywork-Reward-V22 V2--ardd-2-Lla---33 3-B reward model to provide feedback, which achieves sate-of-the-ane performance on seven major reward model benchmarks at the time of writing this paper (Liu et al., 2025a). We refer readers to Appendix G regarding details of constructing the ranking-based preference dataset.
Evaluation. We evaluate our fine-tuned models out-t--distribuutinn and in-distribution tests.
For the out-of-distribution test, we assess our models on two widely used custruction-ooloviiii benchmarks: AlpacaEval 2.0 (Dubois et al., 2024) and Arena-Hard-vO.1 (Li et al., 2024). AlpacaEval 2.0 comprises 805 questions drawn from distinct datasets. Performance is measured by the win rate (WR) of model outputs against reference answers generated by GPT-4-Turbo. We also report the congh--onntooled win rate (LC), which adjusts WR to control for output length. wrena-Hard-vO. consists of 500 well-defined technical groblem-solving
prompts and evaluates models using WR against GPT-4-0314. Previous works (Meng et al., 2024) have demonstrated that Arena-Hard-vO.1 achieves stronger model separability than AlpacaEval 2.0. For both benchmarks, we use 1PPT-4.1-min as the judge, replacing the default GPT-4-Turbo due to its improvements (OpenAI, 2025). To enhance cross-judge robustness, we additionally employ GPT-5-mini as the judge on results are in Section G.4.
The
For the in-distribution test, we first let each fine-tuned model generate responses on the UltraFeedback test dataset, and then compute its win rate against the preferred responses in the dataset, using GPT- 4.1 4.1-min 1-mmni as the judge. This methodology aligns with standard evaluation protocols in related literature (e.g., Rafailov al., 2023; Liu et al., 2023; 2025b).
4.2 MAIN RESULTS
Generalization performance. All nine choice- methods four existing and five newly proposed RCPO variants- outperform other preference optimization baselines across most evaluation metrics. Notably, the best-performing RCPO method, Malllos-RMM-PO--EE Top-2, surpasses the strongest non-RCPO baseline, IPO, by 4.00 percentage points on AlpacaEval LC, 19.5 percentage points on AlpacaEval WR, 6.2 percentage points on Arena-Hard WR, and 9.47 percentage points on UltraFeedback WR. These gains demonstrate the strong potential of choice modeling as a principled framework for aligning LLMs with human preferences.
Impact of feedback structure. Fixing the reward function (2), we find that training top-2 feedback generally leads to better performance than top-1. This reflects the benefit of richer feedback structure. We will discuss the impact of longer feedback structures in the later ablation study.
Impact of choice models. The selection of the choice model can have a substantial impact on performance. Fixing the reward function as (2), Mallows-RMI performs strongly across feedback types. In the pairwise setting, it outperforms all baselines on AlpacaEval WR, Arena-Has Hard WR, and UltraFeedhack WR, even surpassing MNL models trained richer top-2 feedback. Its performance
8

---

Published as a conference paper at ICLR 2026
Table 2: Evaluation Results for Mama-3-8B-Instrucee
AlpacaEval
Arena-Hard
UltraFeedback
Method
LC (%)
WR (%)
WR (%)
WR (%)
Base Model
24.76 (0.42)
24,40 (1.44)
23.6 (21.9,25.4)
42.51 (1.04)
CPO (Xu al., 2024)
32.25 (0.31)
34.18 (1.59)
31.3 (29.5,33.0)
58.71 (1.05)
IPO (Azar et al., 2024)
37.95 (0.33)
33.51 (1.63)
31.0 (29.2.3.))
58,44 (1.05)
ORPO (Hong et al., 2024)
35.01 (0.39)
28.72 (1.52)
27.5 (25.5,29.3)
51.68 (1.06)
RRHE (Yuan et al., 2023)
31.04 (0.21)
25.36 (1.47)
27.1 (25.3,28.9)
44.28 (1.06)
SLiC-HF (Zhao et al.. 2023)
31.04 (0.21)
25.36 (1.47)
26.9 (25.2.28.9)
44,09 (1.06)
KTO (Ethayarajh et al., 2024)
32.55 (0.35)
(1.54)))))
25.8 (24.0,27.7)
53.33 (1.06)
DPO (Rafailov et al., 2023)
41.24 (0.35)
40.24 (1.66)
32.6 (30.6,347)))
62.36 (1.03)
R-DPO (Park et al., 2024)
39.88 (0.34)
38.01 (1.63)
32.3 (30.1.34.3)
60.68 (1.05)
SimPO (Meng al., 2024)
44.15 (0.25)
38,84 (1.58)
33.5 (3133,588)
50.17 (1.09)
DPO-AlIPairs
33.02 (0.24)
38,47 (1.65)
29.6 (27.3.11)))
51.95 (1.08)
Mallows-RMIL ------MM----P---- PO-Pairwise
39.33 (0.28)
48.71 (1.67)
36.5 (34.3.38.6)
66.28 (1.02)
MNL--OO-Discrete
41.33 (0.29)
48.08 (1.68)
(33.6,37)7))
64.64 (1.03)
SAL-ows-RMU-PO--Disee MNL-PO-TTp-2 Mallows-RMI-PO-Top22
39.19 (0.28)
51.17 (1.67)
36.3 (34.6.37))
67.56 (1.01)
40.12 (0.22)
47.69 (1.67)
35.8 (33.2,38.2)
64.01 (1.04)
41.95 (0.26)
53.01 (1.68)
37.2 (35.0.39.4)
68.91 (1.00)
Notes: Methods grouped by horizontal miles. The first block reports the base model. The second block reports existing prefercace optimization baselines. The third block reports prior preference optimization methods that can be viewed as special cases within the RCPO framework, DPO AllPairs denotes the DPO method trained puirwise data implied from the Top 2 ranking. The final black reports new preference optimization variants proposed in this paper under the RCPO framework. Standard errors (in parentheses) are reported for AlpacaEval and Ulraaseddackk and 95% Cls parcntheses) reported for Arena-Hard WR.
further improves with discrete top-2 choice data. We also observe that, when trained the same feedback structure, the two choice models require nearly identical training time. Further details can be found in the Appendix G.3.
Taken together, our results highlight the promising potential of leveraging broader feedback structures, when paired with appropriate choice models, achieve more effective alignment.
Robustness check on Gemma-2-9B-it and -I-nstut- As shown in Table 3, the Mallows- RMJ-PO-Top-2 consistently outperforms competitive baselines on the benchmarks, highlighting strong generalization across base models.
Table 3: Evaluation Results for Other Base Models.
Gemma-2-99B-a
Mistral-7B-Instree
AlpacaEyal 2
Arena-Hard
Alpacalisal 2
Arena-Hard
Method
LC
WR (%)
WR(%) (%)
(%)
WR (%)
WR (%)
Base Model
46.74 (0.15)
31.81 (1.58)
43.3 (41.3,4.77
14.53 (0.44)
11.94 (1.09)
10.8 (9.5.1220
SimPO (Meng et al., 2024)
(0.17)
47.23 (1.68)
57.4 57.4 (55,,,66)))
26.32 (0.38)
30.13 (1.54)
(17.2200))))
DPO (Rafailov al., 2023)
58.01 (0.18)
56.13 (1.66)
59.9 (57.6.62.2)
23.32 (0.39)
19.91 (1.34)
16.8 (15.5,18.3)
Mallows-B RMI-PO-Top-2
55.64 (0.11)
59.82 (1.65)
60.9 (58.9,62.6)
29.57 (0:22)
37.58 (1.68)
16.9 (15.4,18.3)
Notes: Standard errers (in parentheses) follow AlpacaEval 2 metrics, and 95% CIs (in parentheses) follow Arena-Hard WR.
4.3 ABLATION STUDY
The ablation studies are conducted using the Llama-3-8B-Instruue Due to space constraints, the main text focuses the impact of the ranked choice length k and the assortment size SS). The sample efficiency of training on ranked choice data, the effects the parameter, and different treatments on ranked responses deferred to the Appendix G.5.

---

Published as a conference paper at ICLR 2026
4.3.1 IMPACT OF RANKED CHOICE LENGTH & AND ASSORTMENT SIZE |S]
We study the impact of the ranked choice length k and assortment size |S] from two perspectives: generalization performance and time efficiency. The results are summarized in Table 4 below.
Impact of on generalization performance. A large k may necessarily improve performance. For example, when |S| 5, k = 2 is the best. One possible reason is that higher values of A make the alignment process more reliant on high-quality data. Specifically, when & is small, selecting the top-k items is relatively easy. but as k becomes larger, distinguishing the relative order among the remaining candidates may become more difficult. Hence a too large can introduce additional noise or place unnecessary burden annotators. This finding is conceptually consistent with the prior literature in a best-item selection context (Feng Tang, 2023). It also highlights the benefit of top-k feedback compared to the more extreme options, such as pairwise comparisons full ranking.
Impact [S] on generalization performance. Increasing [S] from two to three and five generally leads to better performance. Notably, even |S| can achieve significant improvement over pairwise ([S) = 2). One possible reason that incorporating more negative samples enables the language model learn better distinctions.
Impact of and [S] on time efficiency. The training time is insensitive to the value of k. It scales approximately linearly with the assortment size |S]. The reason is that when |S] is larger, fewer responses can be contained in one batch, and therefore more time to complete an epoch.
In general, we believe that moderately small (IS) k)) will stand in the sweet spot between keeping feedback simple and making efficient use of data.
Table 4: Evaluation Results of Varied k and S|].
Method
AlpacaEval (%)
AlpacaEval 2 WR (%)
Arena-Hard WR (%)
Training Time (Hours)
41.24 (0.35)
(30.6,
Mallows-RMI-PO
Mallows RMU-PO Pairwise
39.33 (0.28)
48.71 (1.67)
36.5 (34.3, 38.6)
1.5
Sstort3-Mlllon RMJ-PO POsDrece
40.28 (0.27)
$0.49 (1 (8)
36.3 (34,1, 38.8)
2.3
assori--Mlllows- RMI-PO-Topp
Mooot5-Malloos-RMM RMI-s--M--P--- PO- Discrete
39.19 (0.28)
51.17 (1.67)
36.3 (34.6, 37.9)
3.5
assort5- RMI-PO PO--OOT
41.95 (0.26)
53.01 (1.68)
37.2 (35.0, 39.4)
3.5
ssorts-Mallows-BB
41.05 (0.25)
(35.8,
assor5-Malllowe RMJ- PO- PO-Top--
39.92 (0.26)
51.08 (1.68)
37.4 (35.5, 39.2)
3.5
Notes: Standard (in parentheses) follow AlpacaEval 2 metrics, and 95% confidence intervals parentheses) follow Arena-
Hard WR.
5 CONCLUSION
In this work, we propose Ranked Choice Preference Optimization (RCPO), general framework that connects preference optimization with choice model estimation. By leveraging maximum likelihood, RCPO unifies pairwise, single-best, and top-k preference data within principled formulation. Exam- ples of both utility- and rank-based choice models, together with empirical evaluations, demonstrate that RCPO preserves richer feedback and improves alignment over pairwise-only methods. We hope this work provides a foundation for integrating more advanced choice models into LLM alignment and inspires future exploration of richer preference signals.
10

---

Published as a conference paper at ICLR 2026
Ethics statement This work adheres the ICLR Code of Ethics. All authors have read and explicitly acknowledged compliance with the Code of Ethics during the submission process. Our study does not involve human subjects, personal or sensitive data, or any procedures that would raise ethical concerns. Therefore, we believe that this work poses no ethical risks.
Reproducibility statement We have made significant efforts ensure reproducibility. Complete proofs of all theoretical results and detailed derivations of the gradient analysis presented in Section are provided in Section E and Section F, respectively. The full experimental details, including the computing environment, dataset construction process, the use of open-source models, hyperparameter settings, and evaluation protocols, are provided in the Section G. addition, we provide the codes and training scripts in the supplementary materials. Together, these materials allow independent researchers to reproduce our results.
ACKNOWLEDGMENTS
This work is supported by the NUS Startup Grant (WBS Number: A-0003856-00-00), the Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier (WBS Number: A-8003890- 00-00), and the Singapore National Supercomputing Centre (NSCC) Young Investigator Seed Grant (Project ID: 31010027).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Aydın Alptekinoglu and John H Semple. The exponomial choice model: A new alternative for assortment and price optimization. Operations Research, 64(1):79-93, 2016.
Simon Anderson, Andre De Palma, and Jacques-Francois Thisse. Discrete choice theory of product differentiation. MIT press, 1992.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandricllo. A general theoretical paradigm to understand learning from human preferences. In International Conference Artificial Intelligence and Statistics, pp- 1447-4455 PMLR, 2024.
discrete choice models in retail operations. Management Science, 68(6):405540023 2022. Gerardo Berbeglia, Agustín Garassino, and Gustavo Vulcano. A comparative empirical study of
Jose Blanchet, Guillermo Gallego, and Vineet Goyal. A markov chain approximation choice modeling. Operations Research, 64(4))886-905, 2016.
Abraham Bookstein, Vladimir A Kulyukin, and Timo Raita. Generalized hamming distance. Infor- mation Retrieval, 5(4):353-375, 2002.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Sebastian Bruch, Masrour Zoghi, Michael Bendersky, and Mare Najork. Revisiting approximate metric optimization in the age of deep neural networks. In Proceedings of the 42nd international ACM SIGIR conference on research and development information retrieval, pp. 1241-124
2019.
Shihao Cai, Chongming Gao, Yang Zhang, Wentao Shi, Jizhi Zhang, Keqin Bao, Qifan Wang, and Fuli Feng. K-order ranking preference optimization for large language models. arXiv preprint arXiv:2506.00444 00441, 2025.
Haoxian Chen, Hanyang Zhao, Henry Lam, David Yao, and Wenpin Tang. Mallowspo: Fine-tune your Il with preference dispersions. The Thirteenth International Conference on Learning Representations, 2025.
11

---

Published as a conference paper at ICLR 2026
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374, 2021.
Yuxin Chen, Junfei Tan, An Zhang, Zhengyi Yang, Leheng Sheng. Enzhi Zhang, Xiang Wang, and Tat-Seng Chua. On softmax direct preference optimization for recommendation. Advances in Neural Information Processing Systems, 37:27463-27489, 2024.
Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley H Holliday, Bob M Jacobs, Nathan Lambert, Milan Mossé, Eric Pacuit, Stuart Russell, Hailey Schoelkopf. et al. Social choice should guide ai alignment dealing with diverse human feedback. arXiv preprint arXiv:2404.10271. 2024.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultrafeedbcck: Boosting language models with scaled ai feedback. arXiv preprint arXiv:2310.01377, 2023.
Carlos Daganzo. Multinomial probit: the theory and its application to demand forecasting. Elsevier, 2014.
Jessica Dai and Eve Fleisig. Mapping social choice theory to rlhf. arXiv preprint arX1v:2404.13038, 2024.
Persi Diaconis and Ronald L Graham. Spearman's footrule as measure of disarray. Journal of the Royal Statistical Society Series B: Statistical Methodology, 39(2):62--268, 1977.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way debias automatic evaluators. arXiv preprint arXiv:2404.04477 044454 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024.
Vivek F Farias, Srikanth Jagabathula, and Devavrat Shah. A nonparametric approach to modeling choice with limited data. Management science, 59(2):305-322, 2013.
Yifan Feng and Yuxuan Tang. On a mallows-type model for (ranked) choices. Advances in Neural Information Processing Systems, 35:3052-3065, 2022.
Yifan Feng and Yuxuan Tang. A mallows-type model for preference learning from (ranked) choices. Available SSRN 4539900, 2023.
Michael A Fligner and Joseph S Verducci. Distance based ranking models. Journal of the Royal Statistical Society: Series (Methodological), 48(3):359 -699999399991 1986.
Guillermo Gallego, Richard Ratliff, and Sergey Shebalov. A general attraction model and sales-based linear program for network revenue management under customer choice. Operations Research, 63 (1):212-232, 2015.
Guillermo Gallego, Huseyin Topaloglu, et Revenue management and pricing analytics, volume 209. Springer, 2019.
Luise Ge, Daniel Halpern, Evi Micha, Ariel Procaccia, Itai Shapira, Yevgeniy Vorobeychik, and Junlin Wu. Axioms for ai alignment from human feedback. Advances in Neural Information Processing Systems, 18880099-04666 2024.
Aman Gupta, Shao Tang, Qingquan Song, Sirou Zhu, Jiwoo Hong, Ankan Saha, Viral Gupta, Noah Lee, Eunki Kim, Siyu Zhu, et al. Alphapo: Reward shape matters for Ilm alignment. arXiv preprint arXiv:2501. 003884. 2025.
12

---

Published as a conference paper at ICLR 2026
José Miguel Hemáández-Lobtto Matthew W Hoffman, and Zoubin Ghahramani. Predictive entropy search for efficient global optimization black-box functions. In NIPS, volume 27, pp. 918-926, 926, 2014.
Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. arXiv preprini arXiv:2403.07691, 2024.
Saffron Huang, Divya Siddarth, Liane Lovitt, Thomas I Liao, Esin Durmus, Alex Tamkin, and Deep Ganguli. Collective constitutional ai: Aligning a language model with public input. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pp. 1395-1417, 2024.
Ekhine Irurozki, Borja Calvo, and Jose A Lozano. Sampling and learning mallows and generalized mallows models under the cayley distance. Methodology and Computing in Applied Probability, 20(1):1-35, 2018.
Srikanth Jagabathula and Ashwin Venkataraman. Nonparametric Extimation of Choice Models, pp. 177-209. Springer International Publishing, Cham, 2022. ISBN 978-3-031--0192--5 doi: 10.1007/ 978-3-03 10--3-031--1126-5.3 URL https: : doi. wttp://../..///.../// 01010//099000--00------------
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint urXiv:2305.06 2023.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data high-quality benchmarks: Arena-hare and benchbuilder pipeline. arXiv preprint wrXiv:2406.11933 2024.
Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, et al. Skywork-reward-22. Scaling preference data curation via human-ai synergy. arXiv preprint arXiv:2507.01352, 2025a.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.0666 0066666777 2023.
Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Moham- mad Saleh, Simon Baumgartner, Jialu Liu, et al. Lipo: Listwise preference optimization through learning-to-rank. arXiv preprint arXiv:2402.01887, 2024.
Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Moham- mad Saleh, Simon Baumgartner, Jialu Liu, et al. Lipo: Listwise preference optimization through learning-to-rant took..n. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pupers), pp. 2404-2420, 2025b.
David JC MacKay. Information-based objective functions for active data selection. Neural compula- fion, 4(4):590, 1992.
Colin L Mallows. Non-null ranking models. i. Biometrika, 44(1/2):114 14-1 1957.
Daniel McFadden. Conditional logit analysis of qualitative choice behavior. 1972.
Daniel McFadden. Econometric models for probabilistic choice among products. Journal of Business, pp. S13-S29, 1980.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference- free reward. Advances in Neural Information Processing Systems, 37:124198-124235 2024.
Abhilash Mishra. Ai alignment and social choice: Fundamental limitations and policy implications. arXiv preprint arXiv:2310.1600 2023.
Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capabilities of gpt-4 medical challenge problems. arXiv preprint arXiv:2303. /3375, 2023.
13

---

Published as a conference paper at ICLR 2026
OpenAL. Introducing GPT-4.1 in the API. 2025. Accessed: 2025-07-30.
matrootuiing--ptt------------eee
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403. 19159, 2024.
Mahendra Prasad. Social choice and the value alignment problem. Artificial intelligence safety and security, pp. 291-314, 2018
Tao Qin, Tie Yan Liu, and Hang Li. A general approximation framework for direct optimization of information retrieval measures. Information retrieval, 13(4):375-397, 2010.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 0115332288-7444 2023.
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang. Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18990-18998, 2024.
Nisan Stienmon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008-3021, 2020.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint SEti:22001.02.399 2022.
Louis L. Thurstone. A law of comparative judgment. In Scaling, pp. 81-92. Routledge, 2017.
Kenneth E Train. Discrete choice methods with simulation. Cambridge university press, 2009.
Lewis Tunstall, Edward Beeching. Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro Von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprini arXiv:2310. 2tt.::330..69000 16944, 2023.
Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: Gen- cralizing direct preference optimization with diverse divergence constraints. arXiv preprint MiXiv:2309.16240. 2023.
Genta Indra Winata, Hanyang Zhao, Anirban Das, Wenpin Tang, David Yao, Shi-Xiong Zhang, and Sambit Sahu. Preference tuning with human feedback on language, speech, and vision tasks: A survey, Journal of Artificial Intelligence Research, 82:2595-2661, 2025.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation. arXiv preprint arXiv:2401.084// 2024.
Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback. Advances in Neural Information Processing Systems, 16:10955-10950 2023.
Mengxi Zhang, Wenhao Wu, Yu Lu, Yuxin Song, Kang Rong, Huanjin Yao, Jianbo Zhao, Fanglong Liu, Haocheng Feng, Jingdong Wang, et al. Automated multi-level preference for milms. Advances in Neural Information Processing Systems, 37:26171-26199 2024.
Hanyang Zhao, Genta Indra Winata, Anirban Das, Shi-Xit Shi-Xiong Zhang, David D Yao, Wenpin Tang, and Sambit Sahu, Rainbowpo: A unified framework for combining improvements in preference optimization. arXiv preprini 1rX0v:2410.042033, 2024.
14

---

Published as a conference paper at ICLR 2026
Yang Zhao, Yixin Wang, and Mingzhang Yin. Permutative preference alignment from listwise ranking of human judgments. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 310-334, 2025.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprini arXiv:2305. 2023.
Tan Zhi-Xuan, Micah Carroll, Matija Franklin, and Hal Ashton. Beyond preferences in ai alignment. Philosophical Studies, pp. 1-51, 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint orXiv:1909.0899; 2019.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
Large Language Models used to assist in editing and polishing the writing. The authors take full responsibility for all content of the paper, and acknowledge that LLMs are not eligible for authorship.
B RELATED WORK
Preference optimization. DPO's 's success has attracted significant research attention in the LLM aligmment community, yielding numerous variants with alternative objectives. These include formula- tions based on ranking objectives (Yuan et al., 2023; Song et al., 2024; Liu et al., 2024; Zhao et al., 2025; Cai et al., 2025), pairwise comparisons employing other preference models (Chen et al., 2025), multi-level preference data (Zhang et al., 2024), other implicit reward formulations (Wang et al., 2023; Meng et al., 2024; Gupta et al., 2025), and approaches leveraging binary feedback on individual prompt-response pairs (Ethayarajh et al., 2024). Distinct from prior studies, we investigate novel forms of preference feedback that enable alignment methods to directly leverage richer choice- signals.
Discrete choice modeling. Research marketing, economics, and operations research has studied various specifications of discrete choice models, including the multinomial logit model (McFadden, 1972). the general attraction model (Gallego et al., 2015), the Markov chain choice model (Blanchet et al., 2016), rank-based choice models (Farias et al., 2013), among others. These models play a critical role in informing key operational decisions such as inventory management, assortment planning. pricing, and matching optimization. For comprehensive overviews, refer readers to Train (2009), Gallego et al. (2019), and Berbeglia et al. (2022). To the best our knowledge, we are the first apply discrete choice model theory in LLM alignment, enabling principled use of richer preference feedback beyond pairwise comparisons.
Social choice and AI aligmment. Social choice theory is field of study that deals with the aggregation of individual preferences to form a collective decision. Current approaches to LLM alignment involves the experimental studies (Huang al., 2024), conceptual frameworks (Prasad, 2018; Mishra, 2023; Dai Fleisig, 2024; Conitzer et al., 2024; Zhi-Xuan et al., 2024), and axiomatic frameworks (Ge al., 2024). contrast to the standard social choice setting (Dai & Fleisig, 2024), where voters provide full rankings over all alternatives, we focus on aggregating ranked choice preferences.
C MALLOWS-RNE RMJ IN PAIRWISE SETUP
When this model is restricted to the pairwise comparison setting, the preference probability simplifies to
Pairwise Setup.
P (yw > Vt ;x) ((x) (mul))s0""
(13)
1+((2))
This formulation aligns with the widely studied noisy pairwise comparison models in the computer science literature, where only two items are compared at time, and the preferred one is selected with a fixed probability that does not depend on the specific pair, The pairwise probability in (13) leads to our following optimization objective.
Theorem 5 (Mallows-RMal RA-.-M-------iii-ii Suppose the underlying pairwise preference distribution follows (13), the corresponding policy optimization objective is given by:
min - E E (e wDD log 6(x) ()) 11,33 log =x(gma (wal(
3 log (tar( < K}]]
(14)
16

---

Published as a conference paper at ICLR 2026
Sigmoid Derivative vs Reward Differencc
Indicator Sigmold Smoothing
0.25
c'lc) dteil -otel.
Gradent Weight oriz)
0.8
0.4
-6
-2
Reward Difference
(a) Indicator with sigmoid approximation.
(b) Derivative of sigmoid function.
D PRACTICAL SCHEME OF MALLOWS-RMN METHODS
To make Mallows- Hallows-RMJJ-an RMJ-based objectives practical for LLM training, we address two key challenges: estimating the dispersion parameter ((x) and stabilizing optimization via smooth approximations.
Token-level Entropy-based Proxy of - log (<z) for Any Mallows-Type Models. Chen et al. (2025) studies using Mallows-O and Mallows-o model to conduct alignment on pairwise prefer- ence data. They propose a direct approach to estimate the dispersion parameter ((x) without any pretraining learning.
The idea is to qualitatively relate ((x) to the empirical output distribution of the LLM. Intuitively, when - log(d(zz) is large, preferences are highly concentrated and the next-token distribution collapses to a point mass, whereas when - log((á (z(()) approaches zero, the distribution becomes uniform. On the other hand, Shannon's entropy H (X) = 0 when X is point mass, and H (X) log n when X is uniform on points. Motivated by this observation, they propose:
- log (H( x))/ Io
(15)
as proxy to log ((x), where T(. Furthermore, they approximate the entropy term in (15) via a realization of a sequence of N max([Ywl. (Yl)) tokens {Yú, Ji given the prompt X:
æ) can be either the pretrained LM model the SFT model.
N-1
H(T(- (x) ~ Σ 1=1
[( y x + H(Y+1 Y =Y))
(16)
which can be easily computed by the logits of the model given the output data. In this case, = VN where is the token size. This is also related to the predictive entropy (Hemaáneez-Lobato al., 2014; MacKay, 1992) of the next-token predictions.
Finally, the authors validate that this entropy- based estimator closely matches the true dispersion in synthetic experiment setup.
In this paper, we adopt Chen al. (2025) to approximate dispersion using Shannon entropy. While their method assumes pairwise comparisons, we extend it to multiple responses. Specifically, for prompt X with response set = (Y....,,Y, Yis)), , we approximate - loo(((X)) as
Σ [s| Σ H(Y -r'=,,))
log
[S] logn
where H-II-) denotes the conditional Shannon entropy, which can be directly computed from the model's logits. Here, y. is the jeh token of response i, N max( m.. MYss)),, and = V with vocabulary size V.
Sigmoid-Smoothen Objectives. The objectives in (10), (12), and (14) cannot be directly optimized with mondienttbased methods, since they involve step functions that are either discontinuous or flat almost everywhere, yielding zero gradients for most values of 0. To make these objectives amenable
17

---

Published as a conference paper at ICLR 2026
to efficient optimization with preference data, replace the indicator functions with sigmoid approximation. This smoothing technique widely adopted in the machine learning literature (see, e.g., (Qin et al., 2010; Bruch et al., 2019)). Specifically, we approximate Ifx < 0} using of o(-z). An illustration is provided in Figure 2a.
This design brings two significant benefits:
Computational benefits: The original indicator-based objectives are discontinuous at zero, making medient-bbsee training unstable. The sigmoid function smooths these objectives, facilitating efficient optimization.
(ii) Soft and robust penalties: the indicator, yielding a "soft" loss that reflects the magnitude of preference violations rather than just their direction. For example, in the loss function (10), under the indicator function, ( matters-i.e. the loss
The sigmoid function offers continuous, differentiable alternative
only the relative ordering between 3 log merlly [r]
(a(uwlm) and / log Ter(gi]z))
zero as long as the preferred response scores higher. In contrast, the sigmoid penalty decreases smoothly as the score gap widens, encouraging the model to only rank preferred responses above dispreferred ones, but to do so confidently. For instance, even if the inequality holds, a small margin will still incur non-tri non-trivial loss, while large margin will yield smaller loss. This is conceptually similar to incorporating margin term in the loss function, as seen in prior works (Zhao al., 2023; Meng et al., 2024; Azar et al., 2024; Hong et al., 2024).
Overall, this smoothing approach allows us to retain the structure of preference- prefeeence-based training while enabling more stable and informative gradient signals during optimization. The final objectives summarized in Table
E
PROOFS
Proof of Theorem The Multinomial Logit (MNL) model (McFadden, 1972) is one of the most widely used utility-based discrete choice models. It assumes that each alternative y/i E S. is associated with latent utility Uy (xx) = Vy, (ar) + <,, where Yy, (1) a deterministic component and e follows an independent Gumbel distribution. Under this assumption, the choice probability of selecting alternative you from assortment S takes the closed form
P( |8 Enes explys, (x)
exp(P (x))
By identifying the deterministic utility My (x)) the reward function defined in (2), the normalization constant (xx) cancels out, and we are left with:
(Vin
PIo (Vw
SS [2)))
Enes
look
Teat (3r
Maximizing the likelihood yields the following objective:
B log me(Mil (.rr((((2))
min - -E -E
B log
Te me(ywll) (x) merr(y-lar (w)
log
3/ E.S
which is equivalent to the objective in (7). 0
Proof of Theorem Following Feng & Tang (2023), the probability that an individual selects a top-k choice 1² V1 1/2 y% out of an assortment S C NVi.
P( 1 18;z)
II
byg
+I,, ES c'vy (m)
By identifying the deterministic utility (I( the reward function defined in (2), we have that the optimal RLHF policy satisfics
Ka Ka [a) Tyer
Pro (1*
|1 15;z)
Tol 2
Sal Ta x
Σ
E E. (M)(())...M))
18

---

Published as a conference paper at ICLR 2026
To maximize the likelihood, objective becomes:
log ma(x:1a) Ter (1) +
ma [tr) a log Feee 3e (z)
min (E.S. log Viee (acla)
mafzyla)
log
Σ
76
j=
8% ME)
which is equivalent to the objective in Theorem 2. M
Proof of Theorem 3. We consider optimizing the following objective:
max EgnD
E Ey-ro( [- (y x)] 8- KL x)) Fooe( [))]
(17)
As shown section A.1 of Rafailov al. (2023), the optimum of such a KL-constrained reward maximization objective has the form of
Toly x) 2(x) Tref x)) exp (-15" (ulx))
9
where Z(x) = En Teo x) exp( - Mo ' (y xx) is the partition function. By moving terms, have
-o (y σ) = B log
ma(v]z) Wea (M))
+ f log Z(x).
(18)
Combining (9) and (18), have that the optimal RLHF policy no(: x)) for (17) satisfies
Zo (z)
To (o(y)l)
Pr (yw S :z) d(x) (aac
I0 lox
Red (M low (0ox>-Blox
Weat (3t ar)
-P lox 2(x))
1+ 18-1
Maximizing the likelihood leads to the following objective:
log d(a) d(a) Eweestvw) " plox Mo (2t)
Teat (Bw 2n)
Box
Meat (M6(
min E Sow
1+0(#)+ 80
min (,
Σ BE
T(-3 log aa
B log
Tol ar) West (3t mc)
log ((x) - C(x)
Weat (Bh (x)
where C(x) = log(1 + (xx) + ............. ((xx)55-- is constant with respect to the policy and thus does
not affect the optimal solution. This formulation is equivalent to the objective in (10). D
Proof of Theorem 4. Following (11) and a similar discussion as in the proof of Theorem 3, we have the optimal RLHF policy na(- 2)) for (17) satisfies
PTu (n (5;z)
(((sI=k,0(2)) *[([),(())
(xx) .8)
where the exponent term d(p= 5 is
k-1
dr (1*,
Σ
T {
log (x( -gg
No No(w1_1l1) (15) i)+
Tea (n)
Toat m)
Σ (f-3 -308
=e(ux(z) (
no (x, -Blog
Maximizing the likelihood leads to the following objective:
min D(DDS,u")D Togg P(I (xx)) do (u (u.,s)s)) ((())
E(,,5,N\~D [oog k.)()) + da (n* (s) log o(x))
min m1
((S)+(E)
where log S0( S1 kz))())))) ((S]. is constant with respect to the policy and thus does not affect the optimal
solution. This formulation is equivalent to the objective in (12).
Proof of Theorem 5. Theorem 5 is special of Theorem 3 with S (Ywe 31). 0
19

---

Published as a conference paper at ICLR 2026
F GRADIENT ANALYSIS
Let f0 V1, V2) == B log No (a)
(1)(())))) B log (val)) for shorthand notation. We next provide an
analysis of various preference optimization models to shed light their training procedures.
MNL
In this section, derive the gradients of DPO, MNL-PO--Discrett, and MNI--PO-Top-K, and then compare their update mechanisms.
F.1.1 DPO
Recap the DPO gradient below:
VoLDPO(TO) -B WD)
(Fo(z, yts yw))
Vo log TolUme
e log no (x)
higher weight when reward wring
incrense likelihood
decrcase likelihood
F.1.2 MNL-PO-DISCRETE
The gradient of LMII.-PO
Discrete with respect to parameters takes the following formulation:
VoLMNL- Do-eee--e-eeceteeem
-B olog
Σ
Σ
Va Pglogga (3s (x)
exp( Mi 9:))
By
higher weight when reward deviates preference
highor wiight where aaberooannt Leggrr
where the expectation is with respect to (x, 5,yw) D D. As MNL-PO-Discrete is special case of MNL-PO-Topk with ranking length & = 1, we defer its derivation to the next section.
F.1.3 MNL-PO-TOP-K
The gradient of -CMIL-POOOOT Top-k with respect to parameters 0 takes the following formulation:
= -B
log
Σ
exp(Ja(z,V,,.0))
By v
ranked preference
higher weight
preference
2/1 / --.-------0 ve15
x V log (vil=)
E
log rolygle)
(19)
Σ
exp(fola, v.1)
(Bn....)
My (B
higher soight when rounst bryer
where the expectation is with respect to (a, S, (ea))) D.
The gradient of the MNL-PO-Top-k loss increases the likelihood of the chosen responses while decreasing the likelihood of all unchosen responses. Specifically, each preference relation (y1 > > Vi S .2) is weighted by the degree to which the implicit reward deviates from the ob- served preference. Furthermore, MNL-PO-Topp Too-k differentiates among the gradients of unchosen
responses: the gradient for unchosen response yA is scaled by Σny ((t.... exp(Ja(R.wy 153]
(3ty expl& log (u)
This factor captures the relative reward of ززو compared with the
Eygrest (od ex log T other unchosen responses.
20

---

Published as a conference paper at ICLR 2026
Derivation. Recall that fa V1. y2) loss takes the following form:
B log mo(vila)
B Blog
Nalue m)
The MNL-PO-
mee((V1(z)
(vala
k
Σ
logo (-log log
Σ
The gradient of fo, y/, V2) can be formulated as:
V (2 yr, y2) = B(V, mag - Vo log mo(ul(())))
(20)
Using properties of the sigmoid function that -=)=)=(()) o(x)) a (xx)((->) and thus (log (x)) )و=) (x))((-1) = o-=x), have:
PO
=-
E
D log
-log
Σ
My (B1...)))
log
Σ
exp(fe(z.
Va log
Σ
exp(fo(a, v3.
i=
(logg(<)) =0 ((-2))
E
log Σ
exp(fo(z,35.0))
Enf Eov exp(Jola.a,..36)
3R
log
Σ
exp(fo(z,U5:30)
Voloe Eny ES E exppfo (1. -oloam 2611
x
(Eq. (20))
My ( (m..)
BE
log
Σ
exp(fo(z,8), By,
Vo log (2)())))
Σ
Ve Vologno(yg (e)
E exp(foe..
The last equation is because:
Eaycal(yi.... oxp..ll....... Eosen exp(/o 1,
Σ
DEE SV(UL.....)))
E ()
F.1.4 COMPARISON OF GRADIENT UPDATES
DPO updates the model by increasing the likelihood of the preferred response yo and decreasing that the dispreferred response yt, The update weight is larger when the model's reward estimate disagrees with the preference. This ensures that learning focuses on correcting mistakes, especially in cases where the model is misaligned with the observed preference.
MNL-PO-Discreds compares the preferred response Yo against all other alternatives y, E S (Bw)), The update weight grows when the reward deviates from the preference, and it is further adjusted according the relative reward magnitudes across the choice set. As result, the gradient not only enforces the winner against individual competitors but also reflects the overall reward distribution of the choice set.
MIN-PPOTTop-k extends this to ranked preferences 3/ ... > yk S. The gradient sequentially enforces each ranking position by comparing Vi. i 1,..., k against the remaining alternatives.
21

---

Published as a conference paper at ICLR 2026
Similar to MNL-PO-Diss DiDscreeee the update is stronger when the reward diverges from the observed preference and when the competitor's s reward is larger. This design enables the model to learn the entire preference ranking rather than only the top choice, sharpening distinctions across multiple ranking positions.
F.2 MALLOWS-RND. RMM
In this section, we derive the gradients of Nallows-RMI-PO-Paivve Mallows-RMI-PO-Disceee and
and then compare their update mechanisms.
F.2.1 MALLOWS-RMJJPOL PAIRWISE
The Mallows-RMMIPP-Parrviie loss takes the following form:
LMlloows- RM---MM-PPaaa Pairwise (Ta) = - -E (ll.....)))) [log ((x) (ffo(z, yt, Bw))].
The gradient of Malloows RMJ-PP PPPPP-Piiiie with respect to parameters 0 takes the following formu- lation:
V Mollooo Pair-oo-PPii (TO)
E ((z) (o((z, VoJola, (Kw)
8TR
log log ((x) of dfo(z, U1, yw of fo U1 Bw)))
Vo log No (o
log Te (w)c)r))
lurger dispersed
lurger weight reward estimates closer
decrcase liselihood of
likelihood
where the expectation is with respect to (x,, Vve M1) ~ D. See Figure 2b for an illustration of o'()).
F.2.2
MALLOW-RRI-PP-DDSCCENN
The Mallows-RMMIPP-Dissree loss takes the following form:
-Mallows-RMJ PMMalows-RM-----Dierree PO----oteeeeeeeeeeee (Te) S. log (xx) Emesroww affolz Wh, Bw)).
The gradient of RMallowa-RMJ--O- with respect to parameters & takes the following formu- lation:
V LMallows RMJ PO-D-Po-DiaccetettT)
-B log δ(x)
Σ
((oo(z) Vofolz, B.
k
JB
log ((x)
Σ
dfolz, We, vw)) jo, Ba, y. vn)))
larger weight for less dispersed
larger weight when roward ostimates closer
Ve log (v())
Vo log (Va
decroase likelihood of ys
inerease likelihond of teu
where the expectation with respect to (x. y.) D.
F.2.3 MALLOWS-RMI-PO-TOO
The Mallows-RMIIPP--Tpp- loss takes the following form:
MMallows-RM--PO--1
- -E,,Suu
log (()) Σ (IS) - 1) v(f00))
Σ
o(fo(,
22

---

Published as a conference paper at ICLR 2026
The gradient of -MMllows-RMI--P-I--- with respect parameters 9 takes the following formula- tion:
- log ()))
(IS) - i) (folo, 1641, Vefo(a, v441,
Σ
o' d'felz vn)) Vofa (o) Bk))
M
3E
log ((x)
(1S-
(fo(z, y++1, v.) (f 1++1, 0.))
larger weight for less dispersed
larger weight for top position
weight whan reward extimates claser
V log (V++1 x))
V log To (v)
decrease Likelihood of
increase likeihhood of
Σ M
Vo log (os (x) Vo log (Bk x))
larger weight when reward estimates ane closer
decrense likelihood
incresse likelihood of yhu
where the expectation with respect to S, ~ D.
F.2.4 COMPARISON OF GRADIENT UPDATES
Mallows-RMJ-PO-Paiivee focuses pairwise comparisons between preferred response you and a dispreferred response yı- The gradient increases the likelihood of yu while decreasing that of yr with stronger updates when preferences less dispersed and when the two responses have similar rewards. This concentrates learning on the harder, more informative comparisons.
generalizes this idea to of responses. The update pushes up the likelihood of the preferred response against all alternatives in \ }. Again, the updates are stronger when preferences concentrated and when the competing responses close in reward, ensuring sharper separation between the winner and its competitors.
Hallows--MMJ-PO-Dissse
Mallows- Mallows-RMJ-PG RMJ-PO-Top-k extends ranked choice feedback. Here the gradient simultancously enforces the full ranking: higher-ranked responses are promoted while lower-ranked ones are sup- pressed. The update amplified for top positions, for less dispersed preferences, and for cases where neighboring rewards are close. This design emphasizes both the most critical ranking positions and harder comparisons, yielding sharper alignment to the preference order.
G EXPERIMENTAL DETAILS
Our method is implemented by modifying the DPO Trainer and DPO Config in the TRL library.
Computation Environment All training 2.6.0 on server with 7 NVIDIA H100 GPUs each with 80 GB of memory, equipped with Ubuntu 22.04.2 LTS. The response-generation step in evaluations is performed NVIDIA Quadro RTX 6000 GPU with 24 GB of memory.
performed in Python 3.10.16 with PyTorch
Dataset We provide descriptions of the UltraFeedback train dataset in Table 6. We construct ranking-based preference set for ranked choice training in the following way. We generate five distinct responses for each prompt in the UltraFeedback dataset (Cui et al., 2023) using a sampling temperature 0.8. We then score and rank these five responses with the Skywork-Rene Reward- -2-Llama- 3.1-8B reward model. In the pairwise setup. adopt the literature's 's tradition and use the top- and bottom- ranked responses as the assortment. For ranked choice training, we truncate each full ranking generate the required data.
23

---

Published as a conference paper at ICLR 2026
The UltraFeedback test dataset contains 1961 prompts, and we use the chosen responses the baseline. This dataset serves as an an-distribution test.
Benchmarks
We provide descriptions of the evaluation benchmarks in Table 7.
Open Source Models
The Hugging Face IDs of the base models and reward model used in our experiments are listed in Table 5.
Table 5: Base Models and Reward Models Used in Experiments.
Model
Hugging Face ID
Llama-3 Hamaa---BB 3-i------B----- I-S-ttrrtuctttt
meta meta-llama//Meta- Llama-3-88 3-8-innsttu----
Gemma- 1ommma-2-9Bii 2-9B-it
google/gemma- 2-9b-it
Mistral-7B-Inssreen
mistalai/MMssaal---nnmmaaaaa
Skywork-
Skywork-Reward-Vo5 Neoward-2-LLlam--33
Skywork/Skywork Beweenk---wwedWWWWee Rew.rd--V-LLlama----
Table 6: Overview of the UltraFeedback Train Dataset (Cui et al., 2023).
Item
Description
Data Scale
~64K prompts
Prompt Sources
UltraChat, ShareGPT, Exvol-Instruct TruthfulQA, FalseQA, FLAN
Open-ended dialogue
intructioo-followinn (writing, coding, summarization, translation)
Prompt Types
Factual QA, adversarial/false QA Standard NLP tasks (classification, reasoning, reading comprehcnsion)
Response Source
Generated by base model
Ranking Method
Scored by Skywork-V2 (er- -3. 1B8B)
Table Evaluation details for AlpacaEval (Dubois et al., 2024) and Arena-Hard (Li al., 2024).
Exs.
Baseline Model
Judge Model
Scoring Type
Metric
AlpacaEvaa 2
80S
GPI-4-Turbo
GPT-4.1-minh Pairwise Comparison LC & Win Rate
Arena-Hard
500
GPT-4-0314
SPT-4.1-mine
Pairwise Comparison
Rate
Notes: The baseline model refers the model compared against.
corresponds Preview
GP1-4. 4.1 1-minii corresponds to GPT-4. 1.t-...-mmni---- 2osinii-225-0---2 04--14.
G.1 TRAINING HYPER-PARAMETER TUNING
By default of TRL, we use adamw optimizer.
Haama-3-SB-Instruce We adopt global batch size of 112, a maximum sequence length 4096, and cosine learning rate schedule for one epoch across all training settings. We retrieve the latest models of the baseline methods; their Huggingface IDs are listed in Table 8. The hyperparameters we used for training in Table 9.
We adopt a global batch size of 112, a maximum sequence length of 2048, and a cosine learning rate schedule for one epoch. We retrieve the latest models of the baseline methods; their Huggingface IDs are listed in Table 10. The hyperparameters we used for training are in Table 11.
Gemma-2-9B-it
24

---

Published as a conference paper at ICLR 2026
Table List of Baseline Models Used in Experiments on 1aama---B--intruuee
Baseline Method
Huggingface ID
SimPO (Meng et al., 2024)
DPO (Rafailov al., 2023)
RDPO (Park al., 2024)
pinnetooniiyyLLaaaaa3-nnttttttt -DP0-0..
CPO (Xu ct al., 2024)
-PB-v000
IPO Azar ct al. (2024)
ORPO (Hong et al., 2024)
RRHF (Yuan et al., 2023)
SLiC-HF (Zhao et al., 2023)
KTO (Ethayarajh et al., 2024)
Table 9: Hyperparameters for training on Llama-3- 3-8B--Intrrucc
Method
Learning Rate
Warmup
Mallows-RMe R-I--MMM-PPPPPEAA
PO-Pai--eee
0.03
3 10 7
0.2
MNL-PO- MNL-PO-Diserete
0.01
5 X 10
0.1
Mallows-RMi RMI-PO-Discrete
0.03
3 10
0.2
MNL-PO-Top-
0.01
5 x 10-7
0.1
Mallows-RMIIPP-Top-22
0.01
3 10
0.1
Table 10: List of baseline models used in experiments Gemma-2-9B-ii.
Baseline Method
Huggingface II)
SimPO (Meng et al., 2024)
princeton-allygenmma---999 2-9---m---------------------- prinetoo-algeeemmss299ii.i.
DPO (Rafailov et al., 2023)
Table 11: Hyperparameters for training on Gemma-2-9B-ii.
Method
Learning Rate
Warmup
Mallows-RNe -EI---M--PPPOOOO 20-T222
0.01
5 x 10
0.2
Table 12: List of baseline models used in experiments on 7istral-7B-Insrruet..
Baseline Method
Huggingface ID
SimPO (Meng et al., 2024) DPO (Rafailov al., 2023)
- -an-saraattt---tttttt------ii In--ntrruutt--mPPOO
princcton-nlp/Miitrral -B--iiittraa-----ntttttt 7B-Instruct-DPO
-B-rrl-----nstuett We adopt global batch size of 112, maximum sequence length 2048, and a cosine learning rate schedule for one epoch. We retrieve the latest models of the baseline methods; their Huggingface IDs are listed in Table 12. The hyperparameters we used for training are in Table 13.
Table 13: Hyperparameters for training Histral-BB-Instruet
Method
9
Learning Rate
Warmup
Mallows-RMI-PO-Topp2
0.05
5 10-7
0.2
G.2 DECODING HYPERPARAMETERS
For AlpacaEval 2.0, adopt the default template for evaluators provided by AlpacaEval. For Llama-3-8B -nnstruttt settings, we adopt the following fixed generation config: max new tokens 4096. temperature 0.7 and top-p 0.1. For Gemma-2-9B-it settings, adopt the following fixed generation
25

---

Published as a conference paper at ICLR 2026
config: max new tokens 4096, temperature 0.5 and top-p 1.0. For Arena-Hard-we the default greedy decoding for all settings and methods. For Ultrafeedback, we use the same configurations as those in AlpacaEval 2.0.
G.3 TRAINING TIME OF MNL AND MALLOWS-RM. RMM
Table 14: Training Time (Hours) of MNL and Mallows-RMJJ
Feedback
MNL
Mallows-RMJ
Pairwise Comparison
1.5
1,5
Diserete Choice
3.5
3,5
Top Choice
3.5
3.5
G.4 ROBUSTNESS CHECK
To enhance cross-judge robustness, additionally employ GPT-5-mmni-2025--08-00 the judge Arena-Hard. We report the win rate and 95% confidence interval. The results are shown in Table 15.
Table 15: Evaluation Results for Llama-3-8B. 1a-ma-3-8BIInstruce Instructt and Gemma-2-9B-it (Judged by GPT-5-mini).
Base
Method
WR (%)
95% CI
Base Model
20.9
[19.3, 22.7]
CPO (Xu et al., 2024)
23.7
[22.1, 25.4]
IPO (Azar et al., 2024)
23.2
[21.6, 24.7]
ORPO (Hong al., 2024)
22.1
[20.5, 23.8]
RRHF (Yuan et al., 2023)
22.3
[20.7, 23.88
SLiC-HF (Zhao et al., 2023)
23.1
[21.5, 24.7]
KTO (Ethayarajh et al., 2024)
21.1
[19.4,22.5)
DPO (Rafailov et al., 2023)
25.9
[24.7. 27.3]
R-DPO (Park et al., 2024)
24.9
[22.8, 26.4]
SimPO (Meng al., 2024)
27.2
[25.4, 28.71
Mallows-RMI-PO-Paavviis
27.4
125.8, 28.91
MNL-PO-Diserete
25.1
[23.1, 27.01
Mallows-RMI-PO-DDisoe
25.4
[23.8, 27.31
MNL-PPOTTp--2
24.9
[23.6, 26.5]
Mallows-RMI-PO-Topp2
25.8
124.1,27.5)
Base Model
38.2
[35.9, 40.3]
SimPO (Meng et al., 2024)
47.0
[45.3, 49.0]
Gemma-2-99
DPO (Rafailov et al., 2023)
49.4
14.36,5.66
Mallows-RMI-PO-TT
51.0
[48.7,
G.5 MORE ABLATION STUDIES
Sample Efficiency of Ranked Choice Training Method. We have evaluated the generalization performance as the amount of training data increases. For reference, when using all training data, the LC and WR for DPO are 41 4.224 and 40.24, respectively. As shown in Figure 3, Mallows-RMJ- Top-2 can achieve comparable performance to full-data DPO when trained much smaller data. Notably, strong performance can already be observed when using roughly 40% of the full dataset. This shows that ranked choice method can achieve high sample efficiency.
Impact on generalization performance. Figure shows the sensitivity.
Impact of different treatments for ranked responses. Given set of ranked responses, can (i) use only the top- bop-bottom pair (Meng et al., 2024; Chen et al., 2025; Zhao et al., 2024; Gupta et al., 2025), (ii) decompose the ranking into all possible pairwise comparisons (Ouyang et al., 2022: Zhang et al., 2024), or (iii) directly train on ranked choice data (ours). We analyze how these different treatments affect both generalization performance and training efficiency.
26

---

Published as a conference paper at ICLR 2026
Sample Efficiency
50
Win Rate
45
40
100
200
300
400
500
Training Steps
Figure 3: AlpacaEval 2 metrics evolution with training steps for Mallows-RMI-PO-Topp2
Figure 4: Effect of tuning / for Mallows-RMI-PO-TER
Llama-3-8B-Instruct
50
$5.0
48
52.5
46
50.0
LC Win Rate
44
42
45.0
42.5
40
40.0
38
36
35.0
0.003
0.01
0.03
0.05
As shown in Table 16, our methods achieve the best generalization performance with only modest increase in training time. In contrast, decomposing the ranking into pairs greatly inflates the data size, resulting in much longer training time while yielding the worst performance.
Table 16: Evaluation Results of Different Treatments for Ranked Responses.
Method
AlpacaEval LC (%)
AlpacaEval WR (%)
Arena-Hard WR (%)
Training Time (Hours)
DPO
41.24 (0.35)
40.24 (1.66)
32.6 (30.6, 34.7)
1.5
DPO-AlIPairs
(27.3.
MNL- PO-LPO- Mallows-RMU-PO-Tppp
40.12 (0,22)
47.69 (1.67)
35.8 (33.2, 38.2)
3.5
(0.26)
(1.68)
(35.0. 39.4)
Noles: DPO and DPO AllPaarss currespond trealment (i) and (i8), respectively, Stundard emvors parentheses) follow AlpacalE val metrics, and 95% confidence intervals follow Arena-Hard WR.
H QUALITATIVE EXAMPLES
In this section, we present series of examples for direct comparisons between our RCPO and benchmarks, as shown in Table 17-21. These tables showcase the qualitative examples of model responses.
27

---

Published as a conference paper at ICLR 2026
Table 17: Qualitative Example (SAT Word)
Question: Find the answer that best describes the underlined SAT word. Select the correct option and explain the meaning of the underiined word.
Despite the cacophony, the student tried to study.
A. Loud sounds
B. Difficult subject Late hour
Low lighting
Malllon-RMJ--O-Teue
DPO
The currect answer is A. Lond sounds.
The correct
Lond sounds.
The word "cacophony"" refers to harsh, discordant sound mixture of sounds, often unpleasant to the car. In this context, "despire the cacophony" noise, the student tried to study.
The ward "cacophony" refers harsh, discordant sound mixture of loud, unpleasont noises, In this context, "despite the that despite the laud and unplensant sounds g- noise from construction, traffic, etc.) surrounding the studenr, they still tried stndy.
that despite the load and unpleasant
cacophony"
The construction site outside my window created a cacophony hammering, drilling, and shouting, hut managed to focus on my work.
Example:
If stadent is trying study busy city noisy café, the cacophony of sounds from cars bonking. people talking, and music playing might he distracting, hut the student trying to focus despite these unpleasant sounds.
Exumple:
The other options don't fit the context:
The word "cacophony" doesn't iasstt relate to challenging academic subject.
B. Difficult subject:
Late hour: While it's possible study late hour, the word "cacophony" docsn't 't deseribe the time of day.
D. Low lighting: This option is unrelated to the concept of loud, unpleasant sounds.
Compared with DPO, the response generated by Mallows-RMI PM To-Top-22 explains why other options are incorrect.
Table 18: Qualitative Example (Excel Test)
Question:
Please write the Excel function name ussociuted with each deseription.
Renans the number of days between two dates
Renamns the starting position of text string within another text string
Retamn the namber in the middle of the set of given numbers
Mallows-RML Pop----T--
DPO
Here are the Excel function names ussociated with euch description: Here are the Excel function names associated with each description:
1. Returns the number of days between two dates:
1. Returns the number of days between two dates: DATEDIF or DAYS
DATEDIP DAYS
However. if you want a single l'unction that returns the exact
CATECIF is more flexible function that can caleu- the difference between two dates various units (days, months, years, etc.). while DAYS is a simpler function that returns exact number days tween two dates.
number of days between two dates, you use custom
formula: =22 -AL+1 (assuming Al and A2 are the start and cod dates).
2. Returns the starting position of text string within an- other string: 2IND SEARCH
2. Returns the starting position of a text string within an- other text string: IND SBARCU
PIND is caso-sensitive, while SEARCU is not.
SIND case sensitive, while SEARCH is not.
3. Returns the number in the middle of the set of given numbers: AVERAGE MEDIAN
3. Returns the number in the middle of the set of given numbers: MED DAN
you to return the exact middle value (in case an even number of values), you can use
This function returns the middle value in set of numbers, assuming the is sorted in ascending order. If the set has an even number of values, MEDIAN returns the average of middle values.
-AVERAGE (MID (Al:A10 C, coUNT (A1 :a10) (assuming A1:A10 the range of numbers), where XIDIF is custom arry formala.
you -TNDFIX (A1 (A1:A1D; NATOR (2, (A1 :A10 <MIN (810) (ML :A10;,
Alternatvely.
can
use
:A10), **1, 001 (assuming A1:A10 the range of numbers). For a simpler approach, you can use "AVERAGE (21 :a10) and then round to the nearest intoger noodod.
For u more rohust appmach, use
-MEDTAN (A1 A1101
Notes: the test, Mallows RMI-ROIIPPOOOO PO.Top-2 answered three questions correctly, whereas incorectll calculated median the last question using the average function.
28

---

Published as a conference paper at ICLR 2026
Table 19: Qualitative Example (Open Question)
Question: Which libraries are the best for developing deep learning scripts in Python?
Mallows- RMI-oo----IOOOppp.....
DPO
For developing deep learning scripts in Pythoo. you'll Il want to focus libraries that provide efficient implementations of neural net- works, automatic diffcrentiation. and optimization algorithms, Here some of the most popular and widely-used libraries: I. TensorFlow (TF): An open-source library developed by Google. Provides Python API, supports both CPU and GPU acoeleration, and has a large community of developers.
For developing doep learning scripts in Python, you IIll want to focus on libraries that provide efficient implementaions of neural net- works and related algorithms. Here are some of the most popular and widely used libraries:
1. TeusorFlow (TF); An open-soures library developed by Google. Popular for deep learning, especially for largo-seale projects, Pro- vides wide range of tools and APIs for building and training acural networks.
Pros: Wide range of pre-built estimators, flexible architecture, good documentation.
2. PyTorch: Another open-source library. developed by Facebook's 's Al Research Lab (FATR). Known for its dynamic computation graph, automatic differentiation, and ease of use. Particularly popular among researchers and developers who prefer a Pythonic API.
Cons: Steeper leamnngg curve, can be slow for small models.
2. Py PoTch:: Known for its dynamic computation graph and auto- matic differentiation. Popular among researchers who prefer more Pythonic API.
3. Keras: A high-lovel neural networks API writtea in Python, capa- ble of running on top of TensorFlow, Py Torch, or Theano. Provides an easy- to-use interface for building and experimenting with deep learning models, suitable for beginners and experts.
Pros: Easier to learn, dynamic computation graph. fast prototyping.
Cons: Less mature than TensorFlow, fewer pre- built estimators.
3. Keras. High-level neural networks API written in Python. Can
nun on top of TensorFloww Py Terchh or Theano.
4. SciPy: While not exclusively deep leamingg librury, SciPy's in ze and scipy is signsl 1 modules provide use- tooks for optimization and signal processing, which are essential for many deep learning algorithmss
Pres: Easy to use, rapid prototyping, supports multiple backends.
soi
Cons: Limited control over level details, not suitable for com-
plex models.
4. Theane: Library from the University of Montreal, facused on automatic differentiation and symbolic math.
specific tasks, you may also want to consider following libraries:
Pros: Stron support for symbolic math, research friendl. high- performance.
5. OpenCV (Compater Vision): tasks as image video processing, feature extraction, and object detection.
Cons:
Steep learning curve, user friendly.
NLTK (Natural Language Tholkit) and spaCy: For natural lan- processing (NL P) tasks, such as text processing, tokenization. language madeling.
S. Microsoft Cognitive Toelkit (CNTK): A commercial-grade DL
library from Microsoft, known high- performance computing and
largo-scale support.
7. PyCaffe (optional): A Python wrapper for the Caffe deep learn- ing framework, useful for deploying pre- pre-trained models.
Pros: High-perfone perf--eefom,eee, supports large-scall models, good docu-
mentation.
When choosing library, consider the following factors:
Cons: Commercial license required for some features, less widely used.
Fase use: you'le re new deep learning. Keras or Py PyTorch
might be better choice due to their more Pythonic APIs.
Hugging Face Transformers: Focused NL.P transfomer..
Performance: TensorFlow and PyTorch are generally more per-
based models, provides pre-trained models and simple interfaces.
formant for large-scale doep learning tasks.
Pros: Fasy to use. large community, pre- praitrdined models.
Specific requirements: you need work with computer vision or NLP. OpenCV and NLTK/spaCY might be more suitable.
Cons: Limited to NLP and hansforer--hseed models.
When choosing library, consider:
Community support: All of these libraries have large commu- nities and extensive documentation, but TensorFlow and PyTorch have more extensive ecosy stem.
Your project's specific requirements (large datasets/comple mod-
els: TensorFlow, PyTorch; simple/apid prototyping: Keras, Hug.
ging Facc).
Here's rough guideline:
Your team's expenise (familiarity makes adoption easier).
heginners: Kens or PyTorch
Computational resources (if limited, Keras or Hugging Face can
For largo-scale deep leammng: TensorFlow PyTorch.
run on CPU/GPU).
specific tasks (computer vision, OpenCV, NITTK/ppaC. or library-specife options (e.g. OpenCV for vision).
Community support (active communities provide better docshutoriaks).
Remember, you always use multiple lihvaries together achieve your goals. Happy deep leamming
Uhimately, it's good idea to experiment with differeot libraries to determine which hest fits your needs preferences. can also use libraries like selkit-learn, which provides unified interface for various MI. algorithms, including some DL. techniques.
Notes: The response generated by Mallows RMJ PO 20-Top-2 gives more details of pros and coms when listing the options compured to that generated hy DPO.
29

---

Published as a conference paper at ICLR 2026
Table 20: Qualitative Example (Factual Explanatory Question)
Question: Do you know why turkeyss became the official food of thanksgiving?
Hallows-RMJ-PO-Top..
DPO
The origins of turkey as the centerpicce of the modemn American Thanksgiving meal are not entirely clear, but there are several theo- rics. Here's s brief history:
A common misconception! Turkeys didn't actually become the oflicial food of Thanksgiving because some historical decree. Instead, wild turkeys were a natural part of the traditional diet and celebrations of early American settlers, particularly Pilgrims and Wampanceg Native Americans. Here's the backstory: In 1621. Pilgrims, who had established Plymouth Colony. hold harvest feast to celebrate their first successful crop yield, They invited Wampanoug tribe, new land and taught them agricultura tschniques. The feast, now known the First Thanksgiving, lasted three days and featured variety of foods, including.
The Wampanoag and the Pilgrims:
Thanksgiving. which took place in 1621, was harvest feast celebrated by the Pilgrims, early European settlers Plymouth Colony, and the Wampanoug Native American tribe. The Wampanosg had helped the Pilgrims adapt new land and laught them
farm. The featured foods native to the region, such as venison, corn, squash, and beans. There no direct evidence turkey was served this specific feast.
likely
helped them adapt the
Wild turkeys (Meleagris gaf- (oporo) were abundant in North America, and Native Americans had hunted
Wild Turkeys in
America:
1. Venison (deer meat)
centuries. The Pilgrims would been faamiliar with wild turkeys, which were a common food source in the region. Prtb-Century European Traditions: Europe, especially Eng- land, roasted capon (a castrated male chicken) or goose was a tra- ditional centerpiece special occasions like harvest festivals and celebrations. When Europcan settlers ariivd in North America, culinary traditions
2. Wild fowl (likely including ducks, goese, and passenger pi- geans)
3. corn (maize)
4. Squash
they brought
Beans
18th and 19th Centuries: The Rise of Turkey as a Symbol of Thanksgiving ing:
6. Seufood (like cod and eel)
time. turkey became popular choice spccial occasious in America, including Thanksgiving. Several factors contributed rise:
Turkeys present, but not the centerpiece. Wild turkeys abundant in the region, and they would have been a natural additioo to the harvest meal, along with other game Over time, as European customs merged with American traditions, roasted turkey became staple of annual harvest celebattons, cluding Thanksgiving. which was not officially declared a national holiday until 1863 by President Ahraham innoln. By then, turkey had already become papular chaice due to its waillaiilit, afford- ahility, and symbolic association with abundance and gratitude.
Availability: Domesticated turkeys were introduced North America in the early 1600s and became widely available in the century.
2. Marketing and Promotion: In the mid-1800s turkey farm- ers and butchers began promoting turkey as centerpiece for Thanksgiving, cupitalizing en its ussociation with the boliday.
So, while turkeys weren't specifically designated as the "official food" of Thanksgiving, their presence the First Thanksgiving and their subsequent popularity in American cuisine cemented their place classic Thanksgiving dish.
3. Sarah Josepha Hale's Influence: Sarah Josepha Hale, prominent American writer and editor, campaigned for na- tional Thanksgiving holiday through magazine, Godey's Lady's 's Book: In 1827, she published recipe for roasted turkey with stuffing, which helped popolarize the dish.
4. Abraham Lincoln's Pruclamation: In 1863, during the Civil War, President Abraham Lincola declared Thanksgiv- ing national holiday, and turkey becamee integral part of the celebration. His wile. Mary Todd Lincoln, was known to serve rousted turkey the White House.
By the early 20th century, turkey had become the traditional main course for Thanksgiving in the United States, largely due to com- hination these factors, While it's unlikely that turkey primury protein at the first Thanksgiving. it has since become an integral part the boliday's culinary tradition.
Notes: These outputs illustrative rather than fact-checked. Mallows-MMI-PO-Togy responds by providing historical narrative aligned with the question's is framing, whereas DPO places greater emphasis on questioning and qualitying the premise of the question.
30

---

Published as a conference paper at ICLR 2026
Table 21: Qualitative Example (Simplification)
Question: Simplify "Most of basic functions of current virtudlization sofware are isherited from early
viruolization
software is now represented with the tern hypervisor, Hondware virtualizaiinn is used to share the resources the cloud providers effectively the customers by generating illusion dedicated compuring, storage networhing on computing infrastructure: The concept of virtuaiiaation is physically implemented using the hypervisor modules, and the operation and processing of hypervisors are In other words, hypervisor modules are installed top OSS, which hypervisor packages."
materialized
interface hardwure and
Mallows-RMJ-PO.
DPO
Here's a simplified version:
Here's 's simplified version:
"Virualizaaionn software, now called hypervisors, inherited many functions early virmal machine operating systems, Hypervisors use hardware virtualization to provide customers with dedicated computing. storage, networking experience cloud, while sharing resources efficiently.
Virtualization technology, now known hypervisor, originated from early
cloud providers efficiently share resources with customers by creating simulated dedicated computing. storage. networking experience. Hyper- visor modules, built on Open-Source Software (OS8), hetween hardware hypervisor packages, physically implementing virtualization concepts.
fundamentals,
The concept of virmalization is implemented through hypervisor modules, which are (OSS). OSS acts an interface between hardware and hypervisor packages, enabling the operation and processing hypervisons."
top Operating System Software
removed some recunnant phrases, rephrased sentences clarity, and condensed the text to preserve the main ideas. Let you'd like any
simplification!
Notes: These oulpats are illustrative rather than cact-checked, Mallows RMJ PO-Top-2 exhibits higher fidelity the sourve text, whereas DPO introduces semantic distortion during simplification.
31


Reviews:
[{"review_id": "xOlqiJ0cMY", "reviewer": "Reviewer_fgCa", "summary": "Pairwise comparison forms the foundation of model alignment, and despite the emergence of alternative approaches such as KTO and joint preference optimization, pairwise methods remain dominant in the community. In this work, the authors propose a framework that extends beyond traditional pairwise comparisons by incorporating rank-order information. They introduce two complementary frameworks under this paradigm (called RCPO) and evaluate them both against each other and against established baselines such as DPO, demonstrating consistent improvements from the proposed approach.", "questions": "- It is important to consider the practical challenges of collecting ranked-choice data, both from an implementation standpoint and a psychological one. Given the current data collection pipelines used by major industry LLMs, adopting a ranking-based approach would represent a significant paradigm shift. Moreover, the usefulness of rank information—specifically, the choice of k, |S|, is highly task-dependent. This raises questions about whether the additional computational and monetary costs of obtaining such data are justified, especially when the observed performance gains appear relatively modest.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "I appreciate an alternate modeling for the preference alignment, the associated literature, and the corresponding MLE estimator formulations in the LLM alignment setup, with a given optimal reward structure. I also appreciate the gradient analysis, and to my understanding, the MNL style approach seems close to one-vs-many, where the MALLOWS-RMJ MODEL seems close to one-vs-one, but weighted by the size of the universe.", "weaknesses": "### Overall Comments\nThis work would benefit from more experiments and broader comparisons with related methods. Given the current limitations, I don’t think it’s yet ready for this venue.\n\n---\n\n### Related Work on Going Beyond Pairwise Ranking\nThe paper should include quantitative comparisons with several recent methods that also move beyond pairwise preference modeling:\n\n- **LiPO-λ (Listwise Preference Optimization through Learning-to-Rank):** Uses LambdaLoss with permutation-aware weighting and improves over DPO on Reddit TL;DR and Anthropic-HH.  \n- **KPO (K-order Ranking Preference Optimization):** Extends Plackett–Luce models with adaptive \\(K\\) selection and shows better sample efficiency on recommendation benchmarks.  \n- **PPA (Permutative Preference Alignment):** Optimizes differentiable NDCG-based objectives and outperforms pairwise methods on AlpacaEval.  \n- **AMP/MDPO (Automated Multi-level Preference Optimization):** Handles multi-level preferences and achieves strong results on hallucination reduction benchmarks.  \n\nAdding comparisons with these works would make the evaluation more complete.\n\n---\n\n### More Models\nThe results are currently based on only two models. It would help to include additional models such as **Qwen2.5** or **Mistral** to test how well the method generalizes.\n\n---\n\n### More Datasets\nIt would also strengthen the paper to include more datasets, such as **TL;DR**, **Anthropic-HH**, **UltraFeedback**, and **WildBench** (which is more challenging than AlpacaEval 2.0).\n\n---\n\n### Limited and Inconsistent Improvements\nThe improvements are not consistent across models or datasets.  \nFor example:\n- On **LLaMA**, LC–SimPO performs much better.  \n- On **Gemma**, **DPO** performs better.  \n- On **Arena-Hard**, the improvements are small and may not be significant.\n\n---\n\n### Ablation Studies\n1. The paper mentions that \\(k=2\\) performs best, but it would be useful to show how performance changes as \\(|S|\\) and \\(k\\) vary together.  \n2. It would help to test what happens if the \\(|S|-i\\) weighting in the Mallows RMJ model is removed, since that still represents a valid one-vs-one setup.  \n3. The paper should also study **sample efficiency**—how performance changes as the amount of training data increases (e.g., WR and LC at different data budgets).\n4. Sensitivity analysis on $\\beta$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1tDpaeZIYR", "reviewer": "Reviewer_XFCx", "summary": "This paper introduces Ranked Choice Preference Optimization (RCPO), a unified framework designed to improve Large Language Model (LLM) alignment by leveraging richer human feedback, such as top-k rankings, instead of relying solely on pairwise comparisons.\nEmpirical evaluations on Llama-3-8B and Gemma-2-9B show that RCPO variants consistently outperform competitive baselines on major benchmarks, with the rank-based Mallows-RMJ model achieving particularly strong results using top-2 feedback.", "questions": "Some of the equations in the appendix move out of bounds. You should fix that.\n\nDid you run any Top-k experiments with k > 2? If not, why not? Just that it might get worse doesn’t sound like a good reason. It might get better as well.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents the generalized framework: Ranked Choice Preference Optimization (RCPO) which subsumes several existing pairwise methods, including Direct Preference Optimization (DPO), SimPO, and R-DPO, viewing them as special cases.\n\nIt is not limited to pairwise comparisons, but can directly leverage richer human feedback formats, such as multi wise comparisons, single-best selections from a set, and top-k rankings.\n\nEmpirical studies on Llama-3-8B-Instruct and Gemma-2-9B-it show that RCPO variants consistently outperform competitive baselines (including DPO, SimPO, and IPO) on widely adopted benchmarks like AlpacaEval 2 and Arena-Hard. The best-performing variant, Mallows-RMJ-PO-Top2, surpassed the strongest non-RCPO baseline (IPO) by significant margins.\n\nThe improvements are shown on Llama-3 and Gemma-2 base models.", "weaknesses": "The paper only shows experimental results with Top-2 feedback. The authors say that they stop at Top-2 since the performance does not always increase with more feedback. It would be very interesting to see when this happens. Only seeing Top-2 results, when the obvious next step would be to try a higher number Top-k, gives me that the authors are trying to hide something.\n\nThe experiments are conducted only on relatively smaller models, specifically Llama-3-8B-Instruct and Gemma-2-9B-it. The paper does not demonstrate whether the gains observed with RCPO scale to larger models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q0ef8xbg0o", "reviewer": "Reviewer_Zc22", "summary": "The paper introduces RCPO, a framework that generalizes LLM alignment beyond pairwise preferences. It connects preference optimization to choice modeling via MLE, unifies DPO and S-DPO (as MNL-PO-Discrete) under this framework, and derived loss functions for MNL and Mallows-RMJ models choice models towards single-best and top-k rankings.\n\nThe authors demonstrate its effectiveness by fine-tuning Llama-3-8B-Instruct and Gemma-2-9B-it on preference data derived from UltraFeedback and Skywork-Reward-V2-Llama-3.1-8B reward mode. The results consistently outperformed baselines like DPO/KPO and S-DPO.\n\nThe paper also covers practical tricks to implement the Mallows-RMJ objectives, like sigmoid-smoothing approximation for non-differentiable step functions, and use entropy to estimate the dispersion parameter.", "questions": "Could the authors provide further reasoning/evidence on the suitability of Skywork-Reward-V2 for labeling the complex reasoning tasks in AlpacaEval 2 and Arena-Hard? Asking because the qualitative example in Table 15 shows the aligned models replicating a mathematical error (x**2 − x − 8 = (x − 2)(x + 4)), and curious if that could be related to RM quality (i.e. did the preference labeler catch the error?)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a principled and unified view connecting preference optimization, choice modeling, and MLE, bridging the literature on choice models with LLM alignment. The derivation of the new methods is very principled, and the presentation is clear.\n\nThe paper proposes two new methods (MNL and Mallows-RMJ based) for leveraging single-best and top-k rankings in preference optimization. The proposed methods are practical, providing practitioners with useful tools for exploring different sampling and ranking strategies in LLM post-training.\n\nThe experimental design and outcomes are overall convincing.", "weaknesses": "While the authors provide reasonable baselines for the pairwise and single-best scenarios, the evaluation of the top-k methods could be more robust by including something like a vanilla DPO method trained on all pairs implied by the top-2 ranking, especially given the impact of negative sampling on DPO; it is unclear how much of improvement from the “-Top2” methods come from a sheer negative sampling change.\n\nAdditionally, while the paper positions RCPO as a generalized framework and provides good insights on how choice modeling and MLE connect, the core methods derived feels a bit fragmented as their derivation requires several very specific assumptions. The paper feels more like a collection of methods derived under a similar principle. E.g., Citing  “For the baseline method, I used RCPO with MNL choice model for top-k choice” sounds a bit awkward.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)