Multiplayer Nash Preference Optimization
Authors: Fang Wu
Xu Huang
Weihao Xuan
Zhiwei Zhang
Yijia Xiao
Guancheng Wan
Xiaomin Li
Bing Hu
Peng Xia
Jure Leskovec
Yejin Choi
Venue: ICLR 2026 Oral (2026)
Keywords: Preference Optimization
RLHF
LLM Post-training
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Reinforcement learning from human feedback (RLHF) has emerged as the standard paradigm for aligning large language models with human preferences. However, reward-based methods grounded in the Bradley–Terry assumption struggle to capture the nontransitivity and heterogeneity of real-world preferences. To address this, recent studies have reframed alignment as a two-player Nash game, giving rise to Nash learning from human feedback (NLHF). While this perspective has inspired algorithms such as INPO, ONPO, and EGPO that offer strong theoretical and empirical guarantees, they remain fundamentally restricted to two-player interactions, introducing a single-opponent bias that fails to capture the full complexity of realistic preference structures. 
This work introduces Multiplayer Nash Preference Optimization (MNPO), a novel framework that generalizes NLHF to the multiplayer regime. It formulates alignment as an $n$-player game, where each policy competes against a population of opponents while being regularized toward a reference model. 
We demonstrate that MNPO inherits the equilibrium guarantees of two-player methods while enabling richer competitive dynamics and improved coverage of diverse preference structures. Comprehensive empirical evaluation shows that MNPO consistently outperforms existing NLHF baselines on instruction-following benchmarks, achieving superior alignment quality under heterogeneous annotator conditions and mixed-policy evaluation scenarios. Together, these results establish MNPO as a principled and scalable framework for aligning LLMs with complex, non-transitive human preferences. Code is available at~\url{https://github.com/smiles724/MNPO}.

Published as a conference paper at ICLR 2026
MULTIPLAYER NASH PREFERENCE OPTIMIZATION
Fang Wu ; Xu Huang Weihao Xuan V.A, Zhiwei Zhang* Yijia Xiao
Guancheng Wan Xiaomin Li Bing Hu", Peng Xia", Jure Leskovec", Yejin Choi Stanford University, * Georgia Institute of Technology. VThe University of Tokyo
ARIKEN AIP, Pennsylvania State University, Universsty of California, Los Angeles, Harvard University, 'Independent Researcher, *UNC-Chapel Hill annforrsstaoood.... edu, xu, huang@gatech. edu, yejiněcs. www.sssataooroooooo.om edu
ABSTRACT
Reinforcement learning from human feedback (RLHF) has emerged as the standard paradigm for aligning large language models with human preferences. However, reward-based methods grounded in the Bradlcy-Terry assumption struggle capture the non-transitive and heterogeneous nature of real-world preferences. To address this, recent studies have reframed alignment two-player Nash game, giving rise to Nash learning from human feedback (NLHF). While this perspective has inspired algorithms such INPO, ONPO, and EGPO that offer strong theoretical and empirical guarantees, they remain fundamentally restricted to two-player interactions, introducing a single-opponent bias that fails capture the full complexity of realistic preference structures. This work introduces Multiplayer Nash Preference Optimization (MNPO), novel framework that generalizes NLHF to the multiplayer regime. It formulates alignment as n-player game, where each policy competes against a population of opponents while being regularized toward reference model. We demonstrate that MNPO inherits the equilibrium guarantees two-player methods while enabling richer competitive dynamics and improved coverage of diverse preference structures. Comprchensive empirical evaluation shows that MNPO consistently outperforms existing NLHF baselines on instruction- following benchmarks, achieving superior alignment quality under heterogeneous annotator conditions and mixed-policy evaluation scenarios. Together, these results establish MNPO principled and scalable framework for aligning LLMs with complex, non-transitive human preferences. Code is available at https:// github. com smi les 724 MNPP
INTRODUCTION
Large language models (LLMs) have achieved remarkable progress in instruction following and open-ended reasoning through reinforcement learning from human feedback (RLHF) (Christiano)
ct al. 2017). Traditional RLHF pipelines built upon the Bradley-Terry (Bradley & Terry 1952)
model have enabled widely deployed systems (e.g., InstructGPT (Ouyang et al.] 2022), Claude (Bai
et al. 2022). and Gemini (Team et al. 2023)), but they assume transitive preferences and scalar reward functions. Recent empirical studies reveal that human preferences often exhibit non-transitive
patterns and heterogeneous structures that violate these assumptions (Ethayarajh et al. 2024 Wu
ct al. 2024).
This has motivated game-theoretic formulations of alignment that treat preference optimization as finding Nash equilibria in games defined by general preference oracles (Munos et al. 2023). In the Nash learning from human feedback (NLHF) paradigm, well-aligned policy constitutes optimal strategy that competing policies cannot exploit, thereby achieving strategic optimality rather than mediocrity. Subsequent work has explored this paradigm through no-regret learning (Zhang et al.) 2025b), optimistic mirror descent (Zhang et al. 2025a). and extragradient updates (Zhou et al. 2025). thereby advancing both theoretical guarantees and empirical stability relative traditional R....
"Equal contribution. Corresponning authors.

---

Published as a conference paper at ICLR 2026
A. Two-Player Baseline
Rfof (Reference Policy)
Preference Dataset
New IT [Updsted Policy)
Responses
Reward Model (RM)
New Preference Dstasel
Skas
Current (Poliyy Model)
Bampling
Multi--laayer MNPO
Reward Models
Nash Update (MNPO)
Homogeneous RM
Helerogeneous RM
T A
RM Helptunness)
Indecaction
T
A
BB
TC
B
Coure
(Homogenaaus)
RM
T
RMy (Bafety)
TC
RRMe (Truiffuhees))
WC
Figure 1: Overview of the two-player baseline and our multiplayer MNPO training paradigm.
Despite these advances, existing NLHF formulations remain constrained two-player settings, where a single policy competes against one opponent. However, real-world preference alignment rarely resembles a two-agent interaction. Instead, it often involves mixture of annotators, het- erogencous evaluation criteria, multiple reward models, even a sequence of historical model checkpoints-crrattee inherently multi-source, and sometimes conflicting, preference signals (Fre- und & Schapire 1999). Reducing this complex landscape single opponent introduces bottleneck: the policy is optimized against only one distribution at a time, resulting in oscillatory behavior, narrow exploration, and a brittle approximation of the broader preference population.
These limitations highlight the need for a framework that explicitly models alignment as competition against entire population rather than single synthetic opponent. In this work, we show that extending to the multiplayer setting (Freund & Schapire 1999) provides principled mean-field approximation that reduces gradient variance, stabilizes optimization, and more precisely captures diverse preference structures. Crucially, when all policies share the same preference oracle-as naturally occurs when competing against historical versions of single policy trajectory- the resulting symmetric game admits strong theoretical guarantees via multiplicative weights update.
We address these challenges by proposing Multiplayer Nash Preference Optimization (MNPO). principled framework that generalizes two-player preference optimization to n-player games. In MNPO, each policy simultaneously competes against population of other policies while being regularized toward reference model, producing a competitive equilibrium that balances performance against the population with adherence to a trusted baseline. Our contributions are threefold:
Theoretical Framework: We establish that MNPO with homogeneous preference oracles admits natural equilibrium characterizations, including well-defined Nash policies and duality gaps that measure alignment quality. We prove that MNPO inherits the desirable convergence properties of existing two-player formulations while enabling richer equilibrium dynamics.
Algorithmic Innovation: We introduce TD-MNPO, where opponent sets evolve adaptively using weighted combinations of historical policies, naturally yielding provable convergence guarantees. We further propose heterogeneous extension (HT-MNPO) that, although it lacks formal guarantees, demonstrates strong empirical performance across diverse preference sources.
Empirical Validation: Through comprehensive experiments on eestruction-folloiinn and reason- ing benchmarks, we demonstrate that MNPO consistently outperforms existing NLHF baselines, particularly excelling in scenarios involving diverse preferences and complex evaluation criteria.
Our analysis reveals that MNPO provides unifying perspective RLHF, subsuming many existing methods as special cases while offering improved robustness multi-agent alignment scenarios. By bridging recent advances in NLHF with the practical demands of aligning LLMs with diverse, potentially non-trane transitive human preferences, MNPO establishes a scalable foundation for next- generation alignment techniques.
2

---

Published as a conference paper at ICLR 2026
2 RLHF PRELIMINARIES
Notation. x' E denotes a prompt and X' the prompt space, z is assumed to be sampled from a fixed but unknown distribution dg. An LLM is characterized by a policy π X + A()) that takes prompt æ as input and outputs a distribution over the response space y. The response y E y is then sampled from the distribution IT-- 2).
Bradley-Term Model Assumption. The prevalent RLHF framework (Christiano et al. 2017 Ouyang al. 2022) assumes the Bradley-Terry model. It assumes that there exists a reward function
exp(r" r' (z.g')
r" such that for any æ X and y² y² E Y. we have (y' > y² x) exp(r"(z..) ))+xp(() (a,"")
(r" (Ir,y") -r" (r: y²) ) After learning reward function R(:,.). RLHF algorithms aim maxi- mize the following KI Kl-regulrrized objective with preference optimization RI. algorithms such as
PPO (Schulman et al. 2017):
(mT Eg [ay= y)] KL (x(- 2)] [z)]]-
Here Ter is the reference policy, which is usually supervised fine-tuned LLM,, and 7 > 0 is the regularization parameter. By maximizing the objective, the obtained policy simultaneously achieves high reward and stays close to Tref. which can mitigate reward hacking (Tien et al. 2022 Skalse ct al. 2022) to some extent.
General Preference Oracle. The Bradley-Temy model assumption may not always hold in practice.
Recent studies (Munos et al.] 2023] Calandriello et al. 2024 Ye et al., [024 Zhang et al. 2025b a) have instead directly considered the general preference distribution IP without imposing additional assumptions, framing the preference optimization problem as a two-player game. They assume the existence of a preference oracle F X' y y + [0, 1]. It can be queried to obtain binary preference signals as ~ Ber (P (y' y² I)), where = indicates that y' is preferred to y², and = = 0 indicates the opposite. The preference distribution is introduced as Au (Ir, y, v²) (V,y') [U < P (y y' [>]] + (y', I 2 ( > y' x)]. where ~ Uniform 1) a random variable. Given two policies कi and π2, LLMs are aligned using this general preference oracle, and the game objective is written as:
(11, x2) = Ezwdg Bm
worw [P (In > V/ x>] - KL (TIl + KL (
(2)
where the max-player тi aims to maximize the objective, and the min-player T2 aims to minimize the objective. The goal of both players is to maximize their winning rates against the opponent while not deviating too far from Tat. which shares a similar spirit with the objective of Eq.
Nash Policy and Duality Gap. Without loss of generality, we restrict our attention to the policy class II containing the policies with the same support set as कee. The Nash equilibrium the game in Eq. 2 is then defined as:
22 = argmax argmin. (TI, KiENT 72ElI
Due to the symmetry of the two players, their Nash policies are unique and coincide, meaning कं = nे nr (Ye et al., 2024). Interestingly, J (T²,, T) 2 0.5 always holds for VT E II, as [(F,))) 0.5 indicates that n" is the best response against itself. To quantify how well policy approximates the Nash policy x", we define the duality gap as:
DualGap(#) == max (Ii, x) min J (T, π2).
The duality gap is nonnegative and DualGap(m) if and only if ग n". Hence, our goal is to find a policy that minimizes the duality gap. Once we achieve DualGap(11 (T) S we say that π is an e-approximate Nash policy.
METHOD
3.1 RLHF AS MULTIPLAYER GAMES
To extend the two-player preference optimization objective to a multiplayer setting (n,)i_ we consider framework where each policy seeks to maximize its average preference probability against
3

---

Published as a conference paper at ICLR 2026
all other policies while regularizing toward reference policy. We first focus on the homogeneous multiplayer setting where all players share the same preference oracle. This symmetric structure underpins our theoretical guarantees.
Plackett-Luce Reward Learning Assumption. To generalize the maximum-likelihood reward learning objective for the Bradley-Terry model to one-vs-many comparisons, we adopt the Plackett- Luce framework. Specifically, we replace the pairwise logistic term log a (R (1, y') R (z. y y²)) with seftmax over multiple alternatives, extending the Bradley-Terry model to accommodate listwise comparisons. This Plackett-Luce model (Debreu 1960 Plackett 1975) maintains the interpretability of reward-based preferences while scaling to more complex decision- maaiing scenarios. Formally, given learned reward function R(x, v) and a dataset D containing tuples (xx, (u", ..... 3/² })
where {y' ; ..... remaining pool of entities {y²) jztt
......... y²) } a pool of to-be-ranked items, the probability that y is preferred over the under the Plackett-Luut model is:
P P(V' > {rr) Stt x) exp(R(z, E5 exp( ( (xp(R(x. (I.
exp(R ( (x, 3²)
(5)
The corresponding negative log- likelihood for single comparison is: log P (y' {s' 1y3e x
log (exp (R (Ix, Eiet exp (R (a,, (z))))))) - (x,y²) Consequently, the generalized reward
learning objective for learning R becomes:
(o vie R R(I.8') Riveluy
exp R (z,
Σ exp (R (R(ZN')) J
arg max RE
log
Per Per-oompaiion log likelihood
(6) Several key observations arise from this formulation. First, when k = 2 for a one- vs-one comparison, Eq. 6 reduces to the vanilla Bradley-Terry objective, given by log a (R (x. - R (z, y')), since T+ Second, this objective maximizes the gap between the reward of y and the log-sum- exp (LSE) of all alternatives, effectively applying soft maximum over competitors. This penalizes cases where yt fails to dominate the collective "strength" of the dispreferred items {x' Jaye
(a) = e"
Homogeneous Multiplayer Preference Oracle.
We consider the case where all players share the same universal preference oracle P : X x y (D))~1 [0, 1]. It directly compares y with group of responses (y' jizi and outputs binary preference signals z ~ Ber (PP y²* {v') J ar)
Then, each policy n, competes against the other n players, and the objective function becomes:
x) x)]
(Ri, (T)l)es) = Exwdo
T yyg {wr (y)n
KL (T.( ( ( x))
(7)
Here, each policy गy maximizes its expected preference against all other players {Ti} while remaining close to the reference policy wee via a KL penalty. The KL term, weighted by т. prevents over-optimization and ensures behavioral stability. All policies are updated concurrently: at each step, player improves its own objective J, leading the population toward competitive equilibrium that balances performance against opponents and adherence to Tref.
Eq. exhibits several important properties. (i) Symmetric treatment: All policies are treated equally and compete in a symmetric manner, ensuring that कं _ =
x at equilibrium. (ii) Decentralized optimization: Each policy's 'scyss update depends only on its own actions and the aggregate behavior of its opponents, thereby avoiding complex interdependencies. (ii)) Generalization of the two-player case; When n = 2, each policy's 's objective reduces to maximizing its pairwise preference probability subject its own KL penalty, as shown in Eq. 2
Nash Equilibrium and Duality Gap. In this n-player game with homogeneous preference oracles, the Nash equilibrium is policy =² where no player can improve their respective objectives by unilaterally deviating. Formally, for all m E II:
J (mz bz)) 2 J (m (m)))z)), Wi (I......)
(8)
4

---

Published as a conference paper at ICLR 2026
To quantify how far given policy π is from the Nash policy л², define the unified duality gap in the multiplayer setting. For player with opponents fixed as O₂ = {
DualGap(T) max J( 0=) J On). x'Ell
This measures the maximum gain player could achieve by unilaterally switching to an alternative strategy while all opponents remain fixed. A Nash equilibrium characterized by DualGap(T*) and (ualGap(T) & indicates that π is an e-approximate Nash policy.
Multiplayer Nash Preference Optimization. There are well-known algorithms that approximately solve the Nash equilibrium a constant-sum multiplayer game. this work, we follow (Freund & Schapire 1999) to establish an iterative framework that can asymptotically converge to the optimal policy average. Given learning rate y of online mirror descent update, we start with theoretical analysis that conceptually solves the homogeneous multiplayer game as follows]
P (y> 1=) 1s))
(e+1) (y (x)
5 (
exp
71
Σ S33
(10)
This iterative framework starts from base policy x0)) such Tret. In each iteration, the updated policy (((-1) is obtained from the reference policy x(1) following the multiplicative weight update. Particularly, a response y should have higher probability weight if has a higher average advantage over the current policy (*)), Notably, Eq. 10 provides convergence guarantees to Nash equilibrium this homogeneous circumstance, ensuring that the average policy म x(T)
E (1) converges to an e-approximate Nash equilibrium with = ((1/((()))))) regret bound (Hart & Mas-Colell, 2000).
Eq. 10 is equivalent to प: (++1) (y x) = 11i T (y x))
exp n-1
m
xx) 1)) ne (z).
where z, (x) is the normalization factor (a.k.a, the partition function). Then, for any fixed x and y. (++1) should satisfy:
each ideal update policy n.
Σ log
W
(y x)
" n- (y>
E x 12)
log 20
(1I)
(1)
Note that direct computation of π (1+ involves normalization factor, which is intractable for the exponentially large response space y: To avoid computing this normalization factor, consider the logarithmic ratio between response pair y and y', and define the function h, (T,, y.y') as:
Th 2+) log
( T
be (T, v') log
(12)
( (v'' xx)
From Eq. 11 we know that the following equality holds for any response pair y, v/ E Supp (Tret):
Isc ( (++1) w.
7
T. 1x)
(v' m.
(13)
Based on this observation, we define the loss function (T) and update the policy n (1+1) as:
argmin Bow.gr [ (/h (IT, Bws vi)
x (1+1)
11
Σ P ( =11) -P -P(' (U'>,()))
(14)
Ze(T))
It is clear to that a (1+1) is the minimizer of (T))) since L, (I((+1)) 0. Furthermore, in the following lemma, we show that af (+++) is the unique minimizer of T,, within the policy class II. Lemma 1. For each E [T], T in Eq. 14 is the unique minimizer of Le(T) within II.
For notational conciseness and avoiding confusion, we use superscript to denote time here, so that of cquivalent गe.
5

---

Published as a conference paper at ICLR 2026
The proof is deferred to Appendix F.I Moreover, we replace the tricky term P (v x0) with a hyperparameter 77 and propose the following loss to bypass it:
(
27
(ne (Be(TT,
[T) = yy
war was tw - Ao
(TT, ywu v1)
(15)
27/
Proposition 1. For any policy E 11 and any iteration t E [T], (T) is equivalent to Le(T), differing only by an additive constant that is independent of π.
See the proof in Appendix F.2 Here, the response pair (y, v') is directly sampled from the current policy x (1), which is crucial for the equivalence between (z) and (x)..
3.2 MULTIPLAYER NASH PREFERENCE OPTIMIZATION
Reward-Enhanced MNPO. While our framework is motivated by moving beyond the limitations of purely scalar reward- wwarrd-based approaches, this does not preclude the beneficial incorporation of reward information when available. The key distinction lies in how rewards are utilized: rather than relying solely on reward-based rankings with implicit transitivity assumptions (as in classical RLHF), MNPO can leverage reward information as auxiliary guidance while maintaining the flexibility to handle non-transitive preferences through its same-theoretic structure.
Reward-awan preference optimization (RPO) (Sun et al. 2025) demonstrates how quantitative reward signals can complement qualitative preference comparisons. It aligns learned implicit preference models with explicit reward models that provide graded assessments of response quality. This shift allows MNPO move beyond binary preference margins and instead minimize discrepancies between the learned reward function Iza (2, y) and the target reward model r" (x. y). Concretely, RPO defines the loss over preference pairs as:
Caro (To, (x,0", v²)) Ir", Tnd, 3 m) = [TT (2,1²) (.,'')=a - Tro (I,8") Ilr" v²) nr (z,2"), (16)
where (z,y'.2) v²)) represents preference pair and the distance metric D R + R measures alignment between the model's implicit reward differences and the scaled reference reward differences. The hyperparameters 7 IR and B R² control the reward scale and regularization, respectively. This formulation directly encourages the policy Te to internalize numan-aligne, reward values, bridging qualitative preference optimization with quantitative reward modeling.
Importantly, the loss function Eq. 15 can be interpreted as special case of RPO under a squared dis- tance metric D Specifically, L employs a learned reward model Tno (x, y) = EE,,, log x(ylx)
()]
and a reference reward model gap 6,- == 17 (r" (z. y¹) - r* (x,y²) = 1 This connection 2n'
highlights that integrating reward awareness into multiplayer preference games enhances stability, interpretability, and alignment fidelity.
Time-dependent Multiplayers A central challenge in multiplayer optimization lies in defining and updating the set of opponent players (() n-1 in Eq.
13 Inspired by recent iterative preference
optimization methods such DNO (Rosset al. 2024 SPIN (Chen et al. 2024). and INPO (Zhang) et al. 2025b), which typically rely past policy iterations (e.g., Tee and . (1-1) to construct opponents, we adopt dependenn opponent selection mechanism.
At any iteration 4, we construct the opponent set from a mixture of recent time-indexed historical policies { (n t+ 1), weighted by coefficients Xy with Ay E [0. 1] and optionally Σ, S The resulting time-dependent MNPO (TD-MNPO) loss is formulated as follows:
C (M) 8, [,,),, v) Ey..y' non tw ) log
x)
Σ A log ==0
n-2 Te- you x)) कe
76
(17)
xr)
Te (1n
x))
where 8" encodes the target reward gap. By blending multiple past policies, this formulation stabilizes training, mitigates overfitting to transient fluctuations, and preserves temporal consistency.

---

Published as a conference paper at ICLR 2026
Table I: Time-dependent MNPO recovers many existing offline or online preference optimization
algorithms. We denote the target reward gap as 6,t == 1j (r* (z, y1) - (z, y2)) D and IDwdd represent the squared distance and backward Bernoulli KL divergence, respectively,
Algorithm Num. Players
Opponents
Importance Weights
Dist.
Target Reward Gap
SimPO
Dyww
00
CPO
Dew
DPO
Weet
A
12bwd
00
Distill-DPO
Teee
X
D
DNO
11
Dowd
00
SPIN
11
A =
Dowd
00
SPPO
71
X 1
D
P (M
me
x)
IPO
71
DD9
if
INPO
=
A
if
1
2:
Connections to Existing RLHE. This unified MNPO formulation in Eq. 17 reveals that many preference optimization algorithms can be recovered as special cases by varying the number of players n. choice of opponents O,, distance metric I3, and target reward gap 8 For instance, DPO emerges by setting n = 2. Oz = Tet. and X 1. Table T summarizes these reductions, showing how dime-dependent MNPO unifies offline and online preference optimization under principled framework. We provide broader overview of RLHF objectives in Appendix
Compared to static reference-based approaches, the reward-aware multiplayer formulation offers: (i) Smoother policy evolution. Unlike methods that rely solely on the most recent past policy, MNPO gradually incorporates multiple past policies, preventing abrupt shifts and stabilizing policy updates. (ii) Greater robustness. By combining historical opponents with a weighted mixture, MNPO mitigates the risk of overfitting to transient fluctuations in recent iterations. (iii) Unified interpretation. TD- MNPO seamlessly extends existing approaches into unified formulation, allowing for flexible adaptation different training scenarios. (iv) Stable convergence. The weighting scheme ensures that recent policies exert greater influence while preserving the broader learning trajectory, thereby leading to more stable convergence. By dynamically leveraging historical policies as opponent players, TD- MNPO enhances preference optimization, making more adaptable and robust in evolving learning environments. The aseudo-algorithm is described in Appendix
3.3 MULTIPLAYER GAME WITH HETEROGENEOUS PREFERENCE ORACLES
Multiplayers with Heterogeneous Preference Oracles. In many real-world alignment scenarios, preference signals originate from multiple heterogeneous sources such as annotators with different evaluation criteria or distinct reward models trained for separate dimensions of quality (e.g.. helpful-
ness (Tan et al. 2025), safety (Dai et al.. (023),, conciseness (Dumitru et al. (025)). We therefore generalize MNPO to the heterogeneous setting by replacing the historical mixture over past policies with a mixture over opponent policies associated with distinct preference oracles.
Concretely, each player is paired with distinct reward model (2, v), inducing player-specific
preference oracle P; : X y 122p-1 + [0, 1]. The objective J TT, { becomes
Ezwdo Eytwse.ginn P. (ut (w') j#l
1x) KL (m²( x)]( x))] which preserves the
multiplayer structure while allowing each agent to learn under its own notion of preference.
Game Thene-tic Properties. When IP; + Pj, the resulting game is general-sum and lacks the symmetry needed for formal Nash equilibrium guarantees. In this setting, each player has its own objective J; with its own oracle
P.
breaking the constant-sum structure that underpins the convergence guarantees of multiplicative weights update. Consequently, the iterative framework in Eq. does not have formal convergence to the Nash equilibrium in the heterogencous case (Daskalakis et al. 2009 Hart & Mas-Colell 2000). To quantify the quality of policy profile, detine player-specific duality gap: DualGap, (m))
7

---

Published as a conference paper at ICLR 2026
J,, Oz), where O₂ { ii the fixed opponent policies. This measures player i's incentive to deviate from the current strategy. We consider the policy profile {x,))== to be near stationary point when max; DualGap, (T:) S e, indicating that player has a strong incentive to deviate unilaterally. Nevertheless, the algorithmic framework remains natural and principled: each policy optimizes with respect to the current opponent distribution using its own oracle and empirically yields effective solutions.
max Ja O+) Ox)
Heterogeneous MNPO. Let og denote the target reward gap induced by reward model ri. following RPO's 's reward-aware interpretation. The heterogeneous MNPO (HT-MMPOO) for player becomes:
(Be We (30
(yw
x) r)
Cer (TT [8, ()),,)) B. ww.. (f )
log
A log
(18)
x)
(yt
where the mixture is over opponent policies (Ty b and {Ay} are importance weights. Eq. 18 retains the equilibrium-seeking nature of MNPO, but each policy now internalizes a reward-gap signal specific to its own reward model. As result, MNPO can align with heterogeneous even conflicting evaluators, potentially yielding a population equilibrium that balances multiple dimensions of quality. As empirically demonstrated later, this heterogeneous formulation achieves strong performance in multi-rewart -model scenarios, suggesting that it can find effective stationary points even without formal equilibrium guarantees. The pseudo-algorithm is illustrated in Appendix B
4 EXPERIMENTAL SETUP
Models and Training Settings. We implement an online RLHF framework (Dong et al. 2024 with Gemma-2-9n-
(Team et al., 2024) the base model. Our MUPO training consists of T = 3 iterations, where each iteration generates responses from the current policy on fresh prompt and updates the policy using preference feedback. To eliminate the need for costly human annotations, we employ the reward model
(Wang et al., 2024a) to provide preference signals for TD-MNPO. To simulate heterogeneous preference oracles, we select Skywork -eerkk---wwareee rd- rd-vd-dl---. V2llaama- - 3.1--311111 1-8B (Liu et al.] (025) and Athene Mainee-MM-PP 8B (Frick et al.) 2024) as additional reward models for HT-MNPO. Hyperparameter optimization critical, as optimal configurations vary across base models and across iterations of the same model. Through empirical analysis, we find that maintaining 3 within the range [0.01, 10] consistently produces strong results. Furthermore, observe that gradually increasing throughout training effectively mitigates training degradation while enabling continued model improvement. Complete implementation details and hyperparameter specifications are provided in Appendix C
Evaluation Benchmarks. We evaluate primarily on three widely used open-ended instruction- following benchmarks: MT-Beech (Zheng et al. (003), AlpacaEval 2 (Li et al. (023), and Arena- Hard v0.1 (Li et al., 2024). As suggested by Dubois et al. (024),, we report the win rate (WR) for Arena-Hard and the length-controlled (LC) WR for AlpacaEval 2, judged by GPT-5-mini rather than the outdated GPT-4 Turbo (Preview-1106).
Since RLHF alignment known sometimes degrade reasoning, calibration, and factual accu- racy (Ouyang et al. 2022 Dong et al. 2024)), we further assess performance on broader set of 11 academic benchmarks, These benchmarks span multiple abilities, including explicit instruction following 7Zhou et al., 2023), general knowledge (Clark et al. 2018 Rein et al., 2024 Hendrycks jtt al. 2020), commonsense reasoning (Sakaguchi al. 2021 al.[
in et al. 2021 Zellers et al. 2019).
and math/coding problem-solving (Lewkowycz al. 2022 Chen et al. 2021).
In addition, we compare MUPO with a group of open-source LLMs including SmolLM3-33 3B (Bakouch et al.] 2025). Llama-3, 3.1--8-11----- (Dubey et al. 2024).
1imo-2-32B-Instrnne (OLMo et al. 2025). Tulu-2-DPO-T
Llama-3. Llama-3.1-Tulle 1-1--uul---77----0-
(Lambert et al.) 2025).
and
Qwen3-235B-it (Yang et al.] (025), and closed-source LLMs such as Gemini-2.5-P8 GPT-5, and -ouude-Sonnet-4..
8

---

Published as a conference paper at ICLR 2026
Table 2: Performance of various models instruction-follooinng and wreference-alignneen bench-
marks (AlpacaEval 2.0, Arena-Hard, and MT-Bench), evaluated using GPT-5-mini as the judge.
Model
Size
AlpacaEval 2.0
Arena-Hard MTT-Bench
SFT
9B
50.15
DPO
9B
54.35
45.63
6.87
SimPO
9B
55.16
45.04
SPPO
9B
55.97
43.89
6.86
INPO
9B
56.09
48.03
6.95
TD-MNPO
9B
57.27
52.26
7,03
HT-MNPO (ArmoRM-Llamaa)
9B
57.63
50.93
HT-MNPO (Skywork- R2)
9B
56.01
50.34
7.40
HT-MNPO (Atheme-MM RM-8B)
9B
59.64
51.17
SmolL M3-3L
M3-3B
3B
11.81
22.12
6.38
LLaMA-3.1-8B-4
8B
5.24
39.16
Olno 2-32B Instruct
32B
32.91
31.62
6.62
Tulu-2-DPO-709F
70B
8.82
27.88
5.91
Llama-3, 1-Tuul-3-70B-DPO
70B
52.28
71.34
7.45
LLaMA-3.3-70B-i
70B
29.44
59.21
7.73
8x22B-it
141B
@wen 235B--23---
235B
84.97
88.71
8.27
OpenAL/GPT:S
72.80
98.11
8.46
Anthoppi:Clande-Sonns
62.24
77.58
8.26
2oo
90.93
86.98
7.78
Table 3: Model performance instruction, knowledge, and commonsense benchmarks.
Model
Instruction IFEval
Knowledge
Commonsense
AVG
GPQA
MMLU
ARC
HellaSwag
TruthfulQA
Winogrande
SFT Model
72.27
28.28
75.35
91.29
80.30
70.75
73.72
70.28
DPO
72.96
29.29
75.77
91.26
80.37
71.24
73.88
70.68
SimPO
73.79
32.32
76.79
91.09
23.90
63.40
72.93
69.60
SPPO
75.47
26.26
75.37
91.17
80.10
71.48
73.48
70.19
INPO
73.20
27.78
74.79
91.07
80.22
71.24
73.48
70.25
TD-MNPO
73.94
33.33
75.63
91.15
80. 18
70.26
73.09
71
HT-MNPO MALmoRM)
75.05
29,30
75.66
91.25
80.29
70.58
73.48
70.33
HT-MNPO Skyyook--Rwarde
75.26
36.36
75.39
91.12
80.22
70.87
73.40
71.80
HT-MNPO (Athene RM-8B)
74.42
30.81
75.40
91.18
80.44
71.36
73.32
70.99
5
EMPIRICAL RESULTS
Instruction- Followwig and Preference Alignment. Table presents the performance of MNPO compared existing preference optimization methods on three widely used instruction-followe benchmarks: AlpacaEval 2.0 (Length-Controlled Controlled Win Rate, %), Arena-Hare (Win Rate, %), and MT- Bench (Score/10). All models were evaluated using GPT-5-min n as the judge. MNPO consistently outperforms all baseline methods across all three benchmarks. On AlpacaEval 2.0. MUPO achieves score of 57. 37.27. improving by 2.92 points over DPO (54.35), 2. points over SimPO (55.16), 1.30 points SPPO (55.97), and 1.18 points pronounced on Arena-Hard where MNPO scores 52.26, compared with the next-best method. INPO, 48.03, representing 4.23-point improvement. Notably, on this challenging benchmark, MNPO not only outperforms other preference optimization algorithms but also competes favorably with much larger open-source, fine-tuned models and even the latest closed- source models. surpasses prominent models such as Tulu-2-DPO (70B) and Mi Mil-IIIIIT (14 (141B). On MT-Beech, MNPO achieves 7.03, outperforming all baselines, with closest competitor being INPO 6.95. These results demonstrate that the multiplayer formulation in MNPO provides significant advantages for instruction-Fe -ooloooiiooooooowinnggo tasks. The consistent improvements across all benchmarks suggest that the framework' ability to accommodate diverse preferences and non-tree non-transitive relationships yields better alignment with human expectations in open-ended generation tasks.
INPO (56.09). The improvements are more
Knowledge and Reasoning Capabilities. Table evaluates model performance on academic bench- marks covering instruction following, knowledge, and commonsense reasoning. The results show that MNPO maintains strong performance across diverse cognitive tasks while achieving preference alignment. MNPO achieves the highest average score of 71. .08 across all benchmarks, outperforming

---

Published as a conference paper at ICLR 2026
Table 4: Model performance math and coding benchmarks.
Math
Code
Model
AVG
GSM8K
Minerva-Math
AIME-24
HumanEval
SFT Model
81.96
44.12
60.37
46.61
DPO
82.03
45,96
59.76
46.94
SimPO
82,56
43.38
57.32
45.82
SPPO
$2.11
47.43
59.76
47.33
INPO
82.94
46.32
59.15
47.10
TD MIPO
82,64
44.85
3.33
61.59
48.10
HT-MNPO (AmooRM-Llama3)
82.64
47.79
3.33
60.98
48.68
HT-MNPO Skyyork--Rewand-22
82.03
49.63
0
59.76
47.86
HI-MNPO (Athene-RM) (M-Bn)
82.11
47.06
59.15
47.08
the SFT baseline (70.28) and all other preference optimization methods. Notably, MNPO achieves the best performance GPQA (33.33), demonstrating strong graduate-level reasoning capabilities. The method also performs competitively on instruction following (IFEval: 73.94) and maintains solid performance on knowledge benchmarks such MMLU (75.63) and on commonsense reasoning tasks. Importantly, unlike some preference optimization methods that show degradation on certain academic benchmarks (e.g., SimPO's drop to 63.40 on TruthfulQA), MNPO maintains relatively stable performance across all domains. This suggests that the multiplayer framework helps preserve model's foundational capabilities while improving preference alignment.
Mathematical and Coding Performance. Table presents results on mathematical reasoning and coding benchmarks. MNPO achieves the highest average score of 48. 10 across math and coding tasks, outperforming all baseline methods, including SPPO (47.33) and INPO (47.10) On the challenging AIME-24 benchmark, MNPO is the only method to achieve non-zero performance (3.33), while all other methods, including the SFT baseline, score O. This demonstrates MNPO's superior capability in handling complex mathematical reasoning tasks. On HumanEval, MNPO achieves 61.59, representing the best coding performance among all methods. The results on GSM8K and Minerva-Math show that MNPO maintains competitive performance with existing methods while achieving superior results on the most challenging tasks. This pattern suggests that the multiplayer optimization framework particularly beneficial for complex reasoning tasks that require handling multiple solution strategies.
6 RELATED WORK
Reward-Mooel- RLHE Classical RLHF trains reward model human preference data and optimizes a policy with KL-regularized policy gradients (e.g., PPO) (Christiano et al. 2017 Bai al. 2022 Schulman al. 2017). While effective, PPO-style updates can suffer from instability and high memory cost. To address this, GRPO removes the critic to improve training stability and memory efficiency at scale (e.g., DeepSeek-R1) (Shao al.] [2244; Guo al., (025). ) and DAPO further improves sample efficiency through dynamic sampling and refined loss objectives (Yu et al. 2025). VAPO shows that value learning can strengthen RLHF under appropriate design choices (Yuan) al. 2025). Nonetheless, optimizing against imperfect reward proxies remains vulnerable to reward
hacking (Weng 2024 Wen et al. 2024).
RLHF with General Preferences. DPO bypasses reward modeling by directly optimizing a log- odds margin between preferred and dispreferred responses (Rafailov al. 2023). This idea has inspired a family of extensions, including IPO (Azar et al. (024). KTO (Ethayarajh et al. 2024).
SimPO (Meng al. (024)), and WPO (Zhou al. 2024), which refine the constraint, scaling, sampling strategy. Iterative and online forms introduce exploration and continual updates (Xiong
et
2023
Dong et al. 2024 Xie al. 2024), but most remain limited to static pairwise supervision.
CONCLUSION
We introduce Multiplayer Nash Preference Optimization, extending Nash learning from human feed- back multiplayer settings. Our framework admits approximates well-defined Nash equilibria and
10

---

Published as a conference paper at ICLR 2026
unifies existing preference optimization methods special cases. Empirically, MNPO outperforms baselines across instruction instruction-folloinng preference-alignment. and reasoning benchmarks. These results validate that multiplayer formulations better capture heterogeneous human preferences and provide more robust alignment for L.LMs.
ACKNOWLEDGMENT
This work was supported in part by RS-2024-00455882, National Al Research Lab Project.
REFERENCES
Gholamali Aminian, Amir R Asadi, Idan Shenfeld, and Youssef Mrouch. Theoretical analysis of kl-regularized rlhf with multiple reference models. arXiv preprint arXiv:2502.01203, 2025.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm understand learning from human preferences. International Conference on Artificial Intelligence and Statistics, pp. 4447 1447-4455. PMLR, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204. 05862, 2022.
Elie Bakouch, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Lewis Tunstall, Carlos Miguel
Patifio, Edward Beeching, Aymeric Roucher, Aksel Joonas Reedi, Quentin Gallouédec, Kashif Rasul, Nathan Habib, Clémentine Fourrier, Hynek Kydlicek, Guilherme Penedo, Hugo Larcher, Mathieu Morlon, Vaibhav Srivastav, Joshua Lochner, Xuan-Son Nguyen, Colin Raffel, Leandro von Werra, and Thomas Wolf. SmolLM3: smol, multilingual, long-context reasoner. https
//huggingfacee http:/afa....c.bboooo//e/e cw/blog.b/ooooo...cc.. smo 1m3 2025.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, et al. Human alignment of large language models through online preference optimisation. arXiv preprint arXiv:2403.08633 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01333. 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: I 1803.05457, 2018.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao. Wei Zhu, Yuan Ni. Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ulraaeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01371 2023.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310. 12773, 2023.
11

---

Published as a conference paper at ICLR 2026
Constantinos Daskalakis, Paul W Goldberg, and Christos H Papadimitriou. The complexity of computing a nash equilibrium. SIAM Journal on Computing, 39(1):195-255 2009.
2009.
Gerard Debreu. Individual choice behavior: A theoretical analysis, 1960.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way debias automatic evaluators. arXiv preprint 000:2244.044775. 2024.
Bazvan-Gabriel Dumitru, Darius Petelcaza, Vikas Yadav, and Liangming Pan. Conciserl: Conciseness- guided reinforcement learning for efficient reasoning models. arXiv preprint arXiv:2505.17250, 2025.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.0 2024.
Adam Fisch, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, and Jonathan Berant. Robust preference optimization through reward model distillation. arXiv preprint arXiv:2405. 220.:2295569966 /9316, 2024.
Yoav Freund and Robert E Schapire. Adaptive game playing using multiplicative weights. Games and Economic Behavior, 29(1-2):79-103, 1999.
Evan Frick, Peter Jin, Tianle Li, Karthik Ganesan, Jian Zhang, Jiantao Jiao, and Banghua Zhu. Athene-70b: Redefining the boundaries of post-training for open models, July 2024. URL
ai ailaalllaaao /blogsll///////
sellogs/athene athene
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprini arXiv:2501. 2025.
Sergiu Hart and Andreu Mas-Colell. simple adaptive procedure leading correlated equilibrium. Econometrica, 68(5): 127-1150, 2000.
Dan Hendrycks, Collin Burn,, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.088 000000000000300, 2020.
Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, 2024.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman,
Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing frontiers open language model post-training. 2025. URL
abs/2411. 15124
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski. Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances neural information processing systems, 35:3843-3857, 2022.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The arena-hard pipeline. Blog post. [Accesse 07-02-2025). 2024.
12

---

Published as a conference paper at ICLR 2026
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of instruction instructionfoolooinng models, 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07956. 2021.
Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, et al. Skywork-reward--2. Scaling preference data curation via human-ai synergy. arXiv preprint arXiv:2507.01352, 2025.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
Xufei Lv, Haoyuan Sun, Xuefeng Bai, Min Zhang, Houde Liu, and Kehai Chen. The hidden link between rlhf and contrastive learning. arXiv preprint arXiv:2506.22576 2025.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference- free reward. Advances in Neural Information Processing Systems, 77:124198-12441 2024.
Rémi Munos, Michal Valko, Danicle Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. arXiv preprint arXiv:23 00886. 18, 2023.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo. Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Allyson Ettinger, Michal Guerquin, David Heineman, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Jake Poznanski, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg. David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 olmo 2 furious, 2025. URL https:
//arxiv. wwwriiiorrgaaii aataaabss551. 2501.1000 00656
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Junshu Pan, Wei Shen, Shulin Huang, Qiji Zhou, and Yue Zhang. Pre-dpo: Improving data utilization in direct preference optimization using a guiding reference model. arXiv preprint arXiv:2504. 15843, 2025.
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. arXiv preprint 24XX::2403.1119, 2024.
Robin L Plackett. The analysis permutations. Journal of the Royal Statistical Society Series C: Applied Staristics, 24(2):193 19,-202, 1975.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-53745 2023.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: graduate-level google-prood q&a benchmark. In First Conference Language Modeling, 2024.
Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self- celf-improve with general preferences. arXiv preprint arXiv:2404 037 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106. 2021.
13

---

Published as a conference paper at ICLR 2026
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint urXiv:2402.03300 2024.
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35:9460-9471, 2022.
Shengyang Sun, Yian Zhang, Alexander Bukharin, David Mosallanezhad, Jiaqi Zeng, Soumye Singhal, Gerald Shen, Adithya Renduchintala, Tugrul Konuk, Yi Dong, et al. Reward-aware preference optimization: A unified mathematical framework for model alignment. arXiv preprint
arXiv:2502.00203, 2025.
Yingshui Tan, Yilei Jiang, Yanshi Li, Jiaheng Liu, Xingyuan Bu, Wenbo Su, Xiangyu Yue, Xiaoyong Zhu, and Bo Zheng. Equilibrate rlhf: Towards balancing helpfulness- safety trade-off large language models. arXiv preprint arXiv:2502. 1555, 2025.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Bean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.0011 2024.
ModelScope Team. EvalScope: Evaluation framework for large models, 2024. //qithub. wtt.c//ooommecccccccccc http:opppeealllpppp....
URL https:
Jeremy Tien, Jerry Zhi Yang He, Zackory Erickson, Anea D Dragan, and Daniel S Brown. Causal confusion and reward misidentification in preference-based reward learning. arXiv preprint arXiv:2204.06600 06601, 2022.
Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: Gen- eralizing direct preference optimization with diverse divergence constraints. arXiv preprint arXiv:2309. 16240, 2023.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts In EMNLP, 2024a.
Mingzhi Wang, Chengdong Ma, Qizhi Chen, Linjian Meng, Yang Han, Jiancong Xiao, Zhaowei Zhang, Jing Huo, Weijie Su, and Yaodong Yang. Magnetic preference optimization: Achieving last-iterate convergence for language model alignment. arXiv preprint arXiv:2410.161 14, 2024b.
Jiaxin Wen, Ruiqi Zhong, Akbir Khan, Ethan Perez, Jacob Steinhardt, Minlie Huang, Samuel R Bowman, He He, and Shi Feng. Language models learn to mislead humans via rlhf arXiv preprint arXiv:2409.1282, 2024.
Lilian Weng. Reward hacking in reinforcement learning. lilianweng. Nov 2024. URL github..tb............
28 hewardd-hakkee
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675, 2024.
Tengyang Xie, Dylan J Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit apprroixxiittion for sample-efficient rlhf. arXiv preprint arXiv:2405.2100
2024.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji. Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. arXiv preprini 122 1456, 2023.
14

---

Published as a conference paper at ICLR 2026
Haoran Xu, Amr Sharaf, Yummo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation. arXiv preprint arXiv:2401.0844 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09386 2025.
Chenlu Ye, Wei Xiong, Yuheng Zhang, Hanze Dong, Nan Jiang, and Tong Zhang. Online iterative reinforcement learning from human feedback with general preference model. Advances in Neural Information Processing Systems, 37:81773-81807, 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503 14476, 2025.
Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang, Rrhf: Rank responses to align language models with human feedback. Advances in Neural Information Processing Systems, 16110955-19900, 2023.
Yufeng Yuan, Qiying Yu, Xiaochen Zuo. Ruofci Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.051
2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: 190..0090 2019.
Yuheng Zhang, Dian Yu, Tao Ge, Linfeng Song, Zhichen Zeng, Haitao Mi, Nan Jiang, and Dong Yu. Improving I1m general preference alignment via optimistic online mirror descent. arXiv preprint 2685. 2025a.
Yuheng Zhang, Dian Yu, Baolin Peng, Linfeng Song. Ye Tian, Mingyue Huo, Nan Jiang, Haitao Mi, and Dong Yu. Iterative nash policy optimization: Aligning LLMs with general preferences via no-regret learning. In The Thirteenth International Conference on Learning Representations, 2025b. URLhtttss. /ttps:///ppeeeeeeeee renneeeiieeee eew net
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging ilm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems. 36:46595 46623, 2023.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instrwction-folloiingg evaluation for large language models. arXiv preprint (xXiv:3311.07911 2023.
Runlong Zhou, Maryam Fazel, and Simon S Du. Extragradient preference optimization (egpo): Beyond last-iterate convergence for nash learning from human feedback. arXiv preprint arXiv:2503.08942, 2025.
Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, and Chenguang Zhu, Wpo: Enhancing rlhf with weighted preference optimization. arXiv preprint 1187, 2024.
15

---

Published as a conference paper at ICLR 2026
A ADDITIONAL RELATED WORK ON TAME-THEORETIC RLHF
Another perspective frames preference optimization game between the model and its opponents, in which equilibrium gqullibrium-seeking methods yield stronger last-iterate guarantees. Self-play methods like SPIN (Chen al. 2024). SPPO (Wu al.) 2024), and INPO (Zhang et 2025b) use no-regret dynamics, while Pre-DPO (Pan et al. 2025) and MPO (Wang et al., 2024b) adapt mirror descent. More recent methods, such ONPO (Zhang et al. 2025a) and HGPO (houu al. 2025), incorporate optimism and extragradient techniques to ensure stable convergence under noisy preferences. These advances primarily focus on two-player games but highlight the importance equilibrium views in preference optimization. Extending RLHF to multiplayer interactions, pursued by MNPO, generalizes beyond pairwise dynamics and enables richer equilibrium structures for alignment.
B PSEUDO-ALGORITHN OF MNPO
Algorithm Time-dependent Multiplayer Nash Preference Optimization (TD-MNPO)
1: Require: Number of iterations T,, distance metric DD weight coefficients { }. reward scaling parameter 7), regularization parameter B, external policy {TT reference policy Tats policy class II, preference oracle P..
2: for iteration f. = 1, .,, do
Use current policy म generate response pairs {ur vi ) where y v5 Tt.
4: Query the preference oracle P to get the preference dataset Dr = {,, Vi i=1'
5: Calculate TT-1 as: 1++1 - argmin LtD MNPO
6: end for
7: Output TT+1
Algorithm 2 Heterogeneous Multiplayer Nash Preference Optimization (HT-MNPO)
1: Require: Number of players n, iterations T, distance metric D, importance weights {)g }
reward scaling parameter 7) regularization parameter 3, initial population of policies { (m) reference policy Teef policy class II, heterogeneous preference oracles {P:)
1i=1+
2: for iteration t ...,,... T do
3: for player j = 1,2,...,7 do
4: Use π. (t)
generate response pairs {VL,,) )(,)) where vl,a- vi, m, (. IK)- (.
5: Query the heterogeneous oracle P, obtain the dataset D {V v
6: Update player by n. (++1) arginin LiD MNPO
7:
end for
8: end for
9: Output: (T+1)
i=1
C EXPERIMENTAL DETAILS
Hardware and Implementation All experiments are conducted on 8 NVIDIA H100 GPUs with 96GB memory. For baseline implementations, DPO (Rafailov al. 2023) is trained using the official Hugging Face DPO Trainer, while SimPO (Meng et al. 2024 and SPPO (Wu et al. 2024 follow their official GitHub implementations. INPO (Zhang al. 2025bb is reproduced according to the settings described in the paper.
htttes//itttbb.coo//rrtntttttn httpss//giithbb.commmmbbba
16

---

Published as a conference paper at ICLR 2026
Table 5: Ablation the number of players (n) in TD-MNPO, where report AlpacaEval 2.0 (length- controlled win rate, %). Increasing n consistently improves alignment quality, with diminishing returns beyond n=3.
Players (n)
1
2
3
AlpacaEval 2.0
53.32
54,34 (+1.02)
57.27 (43.93)
57.42 (+4.10)
Table 6: Ablation on Different Base Models. AlpacaEval 2.0 (LC) results.
Model
Llama-3-8B-it
INPO
TD-MNPO
AlpacaEval 2.0
24.80
41.48 (+16.64)
42.94 (+18.14)
Hyperparameters For MNPO, we adopt hyperparameters consistent with SimPO and INPO, using a cosine learning-rate scheduler with a peak learning rate of 10-7, warmup ratio of 0.1, and global batch size of 128. The optimizer is AdamW (Loshchilov & Hutter 2017) without weight decay. We further perform grid search for history weights at timesteps and 2, selecting from {0, 0.1, 0.333, 0.5, 0.667, 0.9}. In addition, perform a grid search for 17 {0.1, 0.01, 0.0075, 0.005, 0.002) and set 0.0075.
Training Data For training data, we use which contains approximately 60K training samples and 2K test samples. We retain both prompts and responses in iteration 1, while only prompts are used in iterations and 3.
Armorm (Cui et al., 2023
Evaluation Framework For evaluation, we adopt the EvalScope framework Team 2024 (version 1.0.2) across all datasets in Tables and Table 2 MT-Benct evvaluttival are conducted following its official GitHub repository. The LLM judge configured with 5 mimmiiiiin? 7-2222 where the reasoning effort is set to "minimal," and all other parameters follow the default EvalScope settings.
[nd and also apply it for AlpacaEval 2.0 and Arena-Hard in
D ADDITIONAL RESULTS
E FORMULATIONS OF PREFERENCE OPTIMIZATION OBJECTIVES
Table provides a consolidated overview of various preference optimization objectives that have been proposed in the literature. Each method is presented in terms of its optimization objective given preference dataset D containing tuples (æ, ,,,,)))), where x denotes the input prompt, and yt and y denote the preferred (winning) and dispreferred (losing) responses, respectively. The table also specifies whether the method explicitly depends on reference policy Teet whether it leverages the current policy π. during training, and whether the algorithm falls into the offline or online reinforcement learning regime.
F MATHEMATICAL ANALYSIS
F.1 PROOF OF LEMMA
Proof. First, by construction (x(x(++))) = 0, hence (t+1) is minimizer. Suppose, for contradic-
tion, there exists क E II with मे # (++1) and ( 0. Then for every pair y.v' E Supp(TTer, we
must have
(T = 7
Ply > / xx) Ppy > m 1))
17

---

Published as a conference paper at ICLR 2026
Table Mean and standard deviation of model performance across three runs under different LLM judges. This illustrates both stability and relative improvements across RLHF methods.
Model
GPT-4-1106-preview
GPT-4.1
GPT-5-mini
SFT Model
47.83 + 0.92
41.95 + 0.13
49.95 + 0.18
DPO
48.32 1.26
43.87 0.45
53.98 + 0.34
INPO
49.30 + 0.95
:
56.16 = 0.06
MNPO
54.05 + 1.58
49.21 + 0.14
57.20 + 0.09
Table 8: Ablation of 2-player vs. 3-player HT-MNPO on AlpacaEval 2.0. 2-player scores are averaged over all pairwise judge configurations for each reward model (e.g.. Armo (Skywork) and Armo (Athene) for AmmoRM-Llama3) while 3-player results are copied from the full HT-MNPO
setting in Tab. [2]
Reward model
2-player HT-MNPO 3-player HT-MNPO
A
ArmoRM-Lllmaa'
55.43
57.63
+2.20
Skywork-Rewe Reward-V V2
54.25
56.01
+1.76
Athene-RM-8B
55.71
59.64
+3.93
Unrolling h2 and rearranging yields the pairwise ratio identity
cxp -1 Ei Ply > n. (1) (z)
I a (yl=))
i
x))
exp Σ P `j x)) 11 Tf (z)
(1)
Because Supp(#) Supp(Fret) and Σ, fyy|z) 1, these pairwise ratios determine 7(- uniquely- and that unique solution is exactly the normalized distribution implied by Eqs. I0 and ie., (f+1), Hence म (t+1), contradiction. Therefore, the minimizer is unique.
F.2 PROOF OF PROPOSITION D
Proof. Introduce an indicator I ~ Ber(P(y > of 2x) for a pair (/.3)) ~ T(1) x (t)). Consider
(T) = E Ey. [ (.0T),,.//) w.
Expanding and comparing (x) with (TT) shows the only difference lies the cross-term E hy( y, v.y. (P > x(1) x) Ply n (t) x)) versus E[he(T, y. s') v') One verifies these equal by writing he as a linear form in log R, log 7/ and using that y and y' are i.i.d. draws from (*):
The v. VP 10 x)
P(y n (1)
[x)] Ey (/k/ v., 1].
(See INPO Appendix A.5 for the algebraic steps; the symmetry argument applies verbatim.) Now expand (T() by conditioning on (y. y/) and the preference sampler Ae 1/):
2
= Ey. [P x 3/ |x) [ v. v. 2.) + + P(y > v/ x)) (1(( 2n
he( v' -1e ys y²) and completing the square shows L4(T) = L.(T) + const, where the constant depends only on the distribution of (y, v') and P(.), not Combining with the equality of cross-terms above gives T((TT) [x(=) + const, as claimed.
Using
F.3 THEORETICAL ANALYSIS OF EQ. 10
Here, provide theoretical analysis of the iterative update in Eq.
10
m ((+1)
(y) (1l x)) x II n (v
x)
exp n
"1
P
11
(0)
x))
(19)
1
18

---

Published as a conference paper at ICLR 2026
Table 9: Various preference optimization objectives given preference data D = (xr, y+ y where is input, and y* and y the winning and losing responses. f is a class of divergence functions. (æ y) is the uncertainty estimator. I is a convex decreasing loss function. P (y x Te x) is the win rate over the distribution estimated by the average win rate over all the sampled responses
yi:K Tep 2).
Method
Objective
RL Type
DPO Rafailov
2023
Rooan juu
logar
sefa"ik)
log
safu"ie)
Offline
f-DPO
Wang
2023
Binst (osstar)o logo (18
Forla
xula [2]
Offline
R-DPO
Park et al.
2024
Fizati 1-b
loge
(Alg Sutts
selvtH)
Offline
Disii-DDO [isch
2074
Ma loy sefx"in)
(1"(ni")) ((,,)))))
Offline
GSHF
Xiong
2025
Bpxatanii logar
BBlg (olly Blu wwis -I ( ((aa")))
roly (o)
ruly (o)
Offline
no (art [x)
Feal ) Apa Fret Bllg so [2] Nest by (x)
Kaushh Juu Aun where Bxxat=o
Ethayaraji
Taat [a)
Offiine
(rslv) Ioarll x)))
roly'le) 1e2 log tolys (o) luy Tal
IPO
2024
Offline
SIUC-HP Zhao et al.
2025
Bisatin D max 60.6 log (y* [x]
log (v" (2)
lng toly" [8)
x
Offinne
RRHF Yuan et al.
2023
Bjawt max पम7 log (o"
(a))
log no (1)) Alggooi" (x)
X
Oflline
Meng
Biratia">s logo L log to (9" a2)
log (or Jx) 7)
al.
Egoutior)sn log (,g (ar
Blog Iy" x)) Alog Wa (y
Eication (ooat logze (s" [x))
Aloog [1oe yus by (x)) po (2r (x)
ORPO
Hong et al. 2024
Pu (y
Py (or 1x))
X
Offline
where pulyi (a) exp
log motor (x)
[v]
ylog (s6 (xx)
log (al (x)
Brratation, -o
n, (.x)
my (st" [x)
DNO
Rosset et all..,200000 2024
-o (ri (crx)e)) - (x. o1 lox
(olox (86
=
Online
DNO- Pret
Rosset et al.
2024
[ Jaa, log
Flog
7olg n(wile) (r) infor 14]
Online
SPIN
Chen al
2024
E(covo Jov.
(Blog miluk) flog m.(y (3)
Online
SPPO
Wi et al.
2024
R (ixp,Puu-sst11---
log PAINEET
r (P
+)}
Onlinee
INPO
Zhang al.
2025b
(esi wi joiz
log rofvc)
log nofor) (on
log
mafal
log rdu
Online
Furlsi)
(x2))
(xr)
kon (o()
Jow (oC)
ONPO
Zhang et al. 2025a
Online
log (w")
log (wr
where ri = argminiB (n:wd J-.n
(vi)
MIO L.v
2025
loy
sudo"le) "iiis/i77
log
(1- Vurl
log
Offlnne
Derivation from Mirror Descent. Eq. 19 can viewed as an instance of online mirror descent (OMD) with the KL divergence the Bregman potential. At each step. player seeks maximize the expected win probability against the population the opponent mixture:
(t) subject to KL regularization toward
((+1) n
1
Σ(T, P(. j+1
5 (0) 1)) KL. π
argmax rEll
n
Taking first-order conditions yields exactly the multiplicative-weighh update in Eq. 19 Thus, the MNPO update inherits the regret guarantees of OMD: the average regret after T rounds scales as ( (1/VT), ensuring convergence equilibrium in the no-regret learning sense.
19

---

Published as a conference paper at ICLR 2026
Pairwise Ratio Dynamics. pairwise log-ratio
To avoid computing the intractable partition function, we analyze the
Jj (1) x)
(V 2)) (., = log (15 x)
Σ 3
log
n
(9) (1/ x))
Eq. 11 in the main text shows that at the fixed point o (++11, these ratios satisfy
he (1+2), v.o') --1 E E(p(~ (1) Tj x)) Ply > xj 22)
Hence, the log-ratio dynamies correspond consistent linearization of the preference margins across all opponents, ensuring that the update increases probability mass on responses with strictly higher average advantage.
Equilibrium Properties. By standard arguments in no-regret game dynamics (Freund & Schapire 1999), if all players update according to Eq. 19 the joint empirical distribution converges to an n-player Nash equilibrium. Moreover, because the update is multiplicative in form, probabilities remain strictly positive on the support of Tetts preventing premature collapse.
Interpretation.
Eq. 19 admits two complementary interpretations:
Population averaging: the geometrie mean term aggregates beliefs of all opponents, ensuring stability against heterogeneous policies.
Advantage weighting: the exponential term amplifies responses that consistently outperform others, with learning rate 17 controlling the exploration exploration-exploitttio exploitation trade-off.
Together, these properties explain why MNPO achieves robust convergence in multiplayer preference optimization while generalizing the two-player INPO update.
G LIMITATIONS AND FUTURE WORK
Like other algorithms in the RLHF paradigm, MNPO's performance is fundamentally linked the quality of its preference data. Three primary limitations warrant consideration for future work.
First, the fidelity of the preference oracle serves as performance ceiling. As the policy model improves and its generations become consistently high-quality, distinguishing between chosen and rejected responses becomes increasingly difficult for the preference oracle in practice. This diminishing discriminative capability can become a bottleneck for further improvement.
Second, the paradigm of simply increasing the probability of chosen responses and decreasing that of rejected ones may face diminishing returns the preference gap narrows. When rejected responses are themselves of high quality, the binary preference signal becomes less informative, potentially slowing down stalling the convergence of the policy model. Future research could explore more nuanced feedback mechanisms to address learning in this high-performance regime.
Third, theoretical analysis focuses the homogeneous multiplayer setting where all players share the same preference oracle. This restriction is necessary to ensure the game has a constant-sum structure and that multiplicative weight updates converge Nash equilibrium. The heterogeneous extension (HT-MNPO), while empirically effective, operates in general-sum game where formal convergence guarantees do not apply. Future work could explore alternative equilibrium concepts (e.g.. coarse correlated equilibrium) or game structures that provide theoretical grounding for heterogeneous preference optimization.
G. EXTERNAL OPPONENT PLAYERS
Formulation. An alternative to using past-time policies opponents is leveraging external LLMs to introduce broader range of competitive dynamics. Given set of n - LLM policies { 1-=1 these opponent models can come from different sources: they may belong distinct model families,
20

---

Published as a conference paper at ICLR 2026
have varying parameter scales within the same architecture, be trained on different data distributions, or specialize in different domains. The external opponent MNPO (EO-MNPO) loss in this case is defined
C EO-MNNO (TT 18,(2)), =D Σ> log < ( ( 1))
log TT x)) Ty (3' xx) no,-
(20)
Here, the constraint Σ, A, ensures that the contributions of different opponent policies are appro- priately weighted. This formulation introduces key advantage: it enables preference optimization across diverse knowledge sources rather than being restricted to a single training trajectory.
To better interpret this formulation, we can draw parallels to knowledge distillation. Specifically, consider a scenario where we have a collection of teacher models { (r.accher) and seek to refine an updated policy that remains close to both these teacher models and the original reference model Tie. The objective function can be expressed as:
Σ KI. (T]]|: )
J(T) Exzdda Eysa [R(x, v)] KL (m][
(21)
This formulation shows that MNPO with external opponent players can be viewed as extension of knowledge distillation (Aminian et al. (025). in which the learned policy integrates information from multiple expert models while balancing divergence from the reference model. Using the same derivation technique as DPO, we can recover Eq. 20 linking preference optimization to generalized knowledge alignment framework.
Proposition 2.
The optimal solution the maximum reward objective in Eq. 27 is equivalent the learned reward model in Eq. 20
straightforward to show that the solution to the maximum reward objective in Eq. 21 takes the form: x" (y x) 2(x) exp( R(x,
/0// II n tcacche (y 2/// where ((x) the
partition function and T = + E,
Analysis. EO-MNPO gains several benefits. First, unlike the self-comparison TD-MNPO, using a diverse pool of external LLMs exposes the policy to broader range of feedback, leading more robust optimization. Second, leveraging domain-speciftt expert models allows MNPO to fine-tune policies for specialized applications, improving generalization. Third, this framework aligns with broader trends in multi-agent RL, in which agents iteratively refine their strategies against multiple opponents, thereby enabling
sophisticated decision-making making. By considering dynamic set of external LLMs as opponent players, EO-MNPO extends beyond self-refcrential optimization, making more flexible and generalizable approach for preference optimization in LLMs.
21


Reviews:
[{"review_id": "k656U97vlh", "reviewer": "Reviewer_Thdj", "summary": "The paper presents Multiplayer Nash Preference Optimization (MNPO), a new framework for aligning Large Language Models (LLMs). \n\nThe authors identify a key limitation in existing alignment methods:\n* Classical Reinforcement Learning from Human Feedback (RLHF) built on the Bradley-Terry model assumes preferences are transitive, which often fails in practice.\n* Recent game-theoretic approaches, known as Nash Learning from Human Feedback (NLHF), overcome the transitivity assumption by reframing alignment as a two-player Nash game (e.g., INPO, EGPO).\n* The authors argue that even these NLHF methods are limited, as their two-player \"single-opponent bias\" fails to model the full complexity of real-world preferences, which are often heterogeneous (e.g., from diverse annotators or model checkpoints).\n*To address this, MNPO generalizes NLHF from a two-player game to an n-player game. \n\nThe paper's main contributions are:\n1. Theoretical Framework: It formally defines an n-player alignment game, introducing a multiplayer objective function (Eq. 8), an n-player Nash Equilibrium (Eq. 9), and a generalized Duality Gap (Eq. 10) to measure alignment quality.\n2. Algorithmic Innovation: It derives a practical and tractable loss function from the complex multiplayer game dynamics. The final proposed algorithm, Time-Dependent MNPO (TD-MNPO), defines the $n$ players as a weighted mixture of historical policy checkpoints, inspired by methods like INPO and SPIN.\n3. Conceptual Unification: The paper demonstrates that this TD-MNPO framework is a generalization that can recover many existing preference optimization algorithms (e.g., DPO, INPO, SPPO) as special cases by simply varying the number of players ($n$), the choice of opponents, and the distance metric (Table 1).\n4. Empirical Validation: Through comprehensive experiments, the authors show that their 9B MNPO model consistently outperforms strong baselines (DPO, SimPO, SPPO, INPO) on instruction-following benchmarks (AlpacaEval 2, Arena-Hard, MT-Bench) and also shows superior performance and capability preservation on academic benchmarks for math, code, and reasoning.", "questions": "1. Could you comment on the disconnect between the motivation (handling heterogeneous annotators) and the experimental setup (using a single RM)? How can you be sure the observed gains are due to the multiplayer formulation's strengths, rather than it just being a more stable version of an $n=2$ (INPO-like) algorithm? Would a more \"true\" test involve training $n$ distinct RMs (e.g., on different data slices) and having each player $\\pi_i$ optimize against a different oracle?\n\n2. Could you please verify the Arena-Hard scores in Table 2? The GPT-5 score of 41.42 seems exceptionally low, especially given your judge is \"GPT-5-mini\" and other models like Claude-Sonnet-4 scored 77.58. Is this a typo? If not, can you explain this result?\n\n3. The value of $n$ (number of players) and the weighting scheme $\\{\\lambda_j\\}$ seem critical.\n    * What value of $n$ was used for the main results in Tables 2-4?\n    * Could you provide an ablation study, even if brief, showing how performance on a key benchmark (e.g., Arena-Hard) changes as you vary $n$ from 2 (like INPO) to the value you used? This would be a smoking gun to prove that the multiplayer aspect is the key to the performance gain.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* **High Significance and Novelty:** The paper provides a clear, logical, and significant next step in the alignment literature (i.e., moving from 2-player to n-player games). This is a much more realistic model for real-world alignment problems involving diverse and non-transitive preferences.\n\n* **Theoretical Soundness:** The work is theoretically well-grounded. The derivation of a practical loss (Eq. 16) from the intractable multiplicative weight update (Eq. 11)  using log-ratio techniques is elegant and builds directly on the foundations of DPO and IPO.\n\n* **Excellent Conceptual Unification:** The framing in Table 1, which shows how TD-MNPO unifies a family of recent algorithms, is a major conceptual strength. It provides a \"recipe\" for understanding and creating new alignment methods. Appendix D (Table 5)  is also an outstanding resource for the community.\n\n\n* **Strong Empirical Results:** The experimental results are impressive. MNPO shows consistent and sometimes significant gains over SOTA baselines across all benchmark categories. The 4.23-point improvement over INPO on Arena-Hard and the fact that it is the only method to score on AIME-24  are particularly noteworthy.\n\n* **Comprehensive Evaluation:** The authors correctly evaluate their model not just on alignment (Table 2) but also on capability preservation (Tables 3, 4), demonstrating that MNPO avoids the performance degradation on reasoning tasks that can plague other alignment methods.\n\n\n* **Clarity:** The paper is exceptionally well-written and clear. The authors do an excellent job of motivating the work and explaining the complex theoretical concepts.", "weaknesses": "* **Disconnect Between Motivation and Experiment:** The primary motivation is to handle \"heterogeneous annotator conditions\". However, the experiment uses a single reward model (ArmoRM-Llama3-8B-v0.1) as the preference oracle. This setup does not actually test the core hypothesis. Instead of a true multiplayer game against diverse preference functions, the experiment is an n-player game where all players compete to align with the same static oracle. This is a significant gap between the problem statement and the empirical validation.\n\n* **Anomalous Result in Table 2:** The empirical results in Table 2 contain a suspicious data point. The GPT-5 model scores 41.42 on Arena-Hard, which is significantly lower than the 9B MNPO model (52.26), other baselines, and especially Claude-Sonnet-4 (77.58). This score seems anomalously low for GPT-5 and raises questions about the evaluation setup or a potential typo.\n\n* **Missing Ablation Study:** The core mechanism of TD-MNPO is the use of $n$ players (historical policies). The main paper lacks a crucial ablation study showing how performance is affected by the choice of $n$. The authors claim a win for the multiplayer framework, but it is not empirically proven that $n>2$ is better than $n=2$ (which would be an INPO-like baseline). The performance gain could be due to other implementation details rather than the multiplayer formulation itself.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7OWsGlvX2w", "reviewer": "Reviewer_iCKV", "summary": "This paper proposes a multiplayer Nash preference optimization (MNPO) framework that extends two-player NLHF to an $n$-player setting. The authors derive an iterative multiplicative-weights update that leads to a practical surrogate loss minimizing squared log-probability ratios between a policy and a set of opponent policies. Experiments with Gemma-2-9B-it show modest improvements over strong baselines on standard alignment benchmarks.", "questions": "1. Please comment on the weaknesses above. \n2. Have you more qualitatively looked at the NE in your multi-player formulation compared to the two-player formulation? The preference function induces a symmetric game and the NE is symmetric as you write on page 4. Under the reward-based preference function the game then reduces again to the two-player game where the equilibrium condition is that a player plays the best response to the mixture of the other players (which are symmetric). Similarly for non-transitive preferences, for example, $n$-player rock-paper-scissors (or the preference function induced by it), we do not need to model more than two players to represent the equilibrium condition (I think) as it is enough for a player to best respond to the population-average / mixture. I think this is a quite standard perspective for symmetric games and in mean-field games where the population-average acts as the second player and provides us with a fixed point condition that defines the same equilibrium as the $n$-player game. What I want to say with this is that often adding more players does not necessarily generate new symmetric equilibria or notable differnt dynamics unless players differ (heterogeneous preferences, asymmetric roles, etc.). I'd be curious whether you have thought about this in some basic examples to understand the solution concept of your multi-player Nash approach and whether there is anything we can learn from this.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper provides a clear and mathematically coherent derivation of multiplayer Nash preference optimization extending two-player Nash optimization. \n2. The reduction of several other optimization frameworks and approaches to the proposed time-dependent MNPO algorithm in Table 1 is nice.", "weaknesses": "My primary concern is the lack of motivation. What is the actual problem with existing two-player Nash optimization that we are addressing here? Why should preference optimization be modeled as an $n$-player game at all? Even the standard two-player Nash formulation only arises incidentally, because we wish to optimize for intransitive pairwise preferences / against a general preference model $p$, which naturally induces a constant-sum (or zero-sum) game and the NE is the most intuitive solution concept. Extending this to multiple players is not obviously motivated. In the paper, I only see this motivating paragraph:  \n> However, real-world preference alignment often involves diverse\nannotators, heterogeneous evaluation criteria, or mixtures of historical model checkpoints - contexts\nthat are better modeled as multiplayer games (Freund & Schapire, 1999).\n\nBeyond the \"mixtures of historical model checkpoints\" (in case you want to use several), it don't see why we want a multiplayer formulation? This would be more plausible *if* you were to model heterogeneous or conflicting preferences explictly (as in pluralistic alignment) where each player represents a distinct preference profile or reward model, e.g., each corresponding to a distinct demographic group. However, the MNPO formulation assumes a symmetric, player-invariant universal preference function, which eliminates this justification entirely. Without better motivation the resulting formulation seems to generalize two-player NLHF without offering a clear conceptual benefit. In fact, optimizing in multi-player games is harder as the dynamics are less stable. What do we gain in turn for the additional complexity of optimizing a $n$-player game and more hyperparameters versus a $2$-player game (which is already hard to optimize)? Empirically, the results also fail to justify the additional complexity. \n\nAlso, the \"universal preference oracle\" can itself be problematic to obtain if it is not reward-based. If this oracle is implemented via an LLM-as-a-judge, prompting it with $n$ candidate responses becomes increasingly unreliable as $n$ grows. Generally, you'd expect the quality and consistency of this feedback to be worse when querying for multiway preferences compared to pairwise comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sOTiO4cIrI", "reviewer": "Reviewer_RaDU", "summary": "The paper extends the field of reinforcement learning from human feedback and alignment of LLMs.\nTheir primary assumption is that reward models are limited in expressing the complexity and diversity of human preferences therefore more general preferences models should be considered. In particular, they model n-wise comparisons instead of pairwise comparisons when one completions out of a pool of n is chosen as the best response by the LLM for a given prompt.\nSince this is a non-transitive model, standard reward models can't be used and the authors formulate the problem as a game between n players and aim to approximate the Nash Equilibrium. However, they model generalises to settings when reward models are adequate choice for human preferences.\nThe proposed algorithm, MNPO, builds on online mirror descent that provides on average convergence guarantee and follow the previous line of work on Nash Learning from Human Feedback (Munos et al. 2023) and extends it to time-dependent opponent selection (TD-MNPO).\nTheir experimental results show consistently better performance than comparable algorithms and results on par with open-weight and closed-source models on standard benchmarks.", "questions": "I have the following questions to the authors that would be great to clarify beyond the points mentioned in the Weaknesses section\n1. In the definition of the DualGap for multiplayer games, Eq. (10), the first term takes both the expectation and the maximum over $\\pi_j$. Could the authors clarify this notation and definition?\n2. In the experiments, Gemma-2-9B-it was used as a base model which is an older version and has already been preference fine-tuned. Why did the authors choose this model and do the results hold when MNPO is initialised with an on SFT model? Due to this, the base model already achieves high scores on most benchmarks leaving small room for improvement and the differences between algorithms and models are marginal.\n3. How does the model compare against more recent open-source LLMs e.g. Tulu3 instead of Tulu2, Olmo2, or SmolLM3? Some models for comparison seems outdated and would be better to compare against more recent variants.\n4. Does the results carry over more general preference oracle? Current experiments use a reward model, ArmoRM-Llama3-8B-v0.1, which results in transitive preference signals while game-theoretic approaches are meant to optimise preference signals that are more diverse.\n5. Why did the authors choose to use GPT-5 as judge for the benchmarks? It is not standard for some of them, e.g., AlpacaEval 2.0 and its alignment with the human annotators is not evaluated, therefore, results with these models are less reliable. I would suggest to validate results with some of the standard annotators provided by the original codebase.\n6. How did the authors decide that 3 iterations for MNPO is sufficient? Results and comments on convergence and DualGap would be appreciated to judge whether the model really converged.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strength of the paper lies in its generality as highlighted in Table 1 furthermore in the time-dependent opponent selection that extends on the standard OMD approach.\nThe experimental results look promising and the comprehensive evaluation on a wide range of benchmarks is appreciated, however, I have some questions as discussed below.", "weaknesses": "I found the following points unclear and potential weaknesses of the paper:\n1. On top of page 2, the authors motivate the multi-player formulation via diverse annotators and heterogeneous evaluation criteria that are parts of the feedback signal that evaluates multiple completions for a single prompt, however, the MNPO framework focuses on multiple models generating the responses. In my opinion, this motivation is more suitable to argue that preference modelling should be improved and not that alignment algorithms need to compare more than 2 responses. It would be great if the authors could clarify the connection between the motivation and the framework.\n2. While the authors claim that the iterative framework they build upon has an asymptotic convergence guarantee to the optimal policy on average (Line 215-216), it is not discussed whether this guarantee holds in their case after introducing modifications to the online mirror descent in Eq. 15 and 16 especially as it moves from an offline dataset $D_t$ in Eq. (15) to an online optimisation problem in Eq. (16). Furthermore, how it carries over to the TD-MNPO algorithm. The last paragraph in Section 3 discussed the benefits of TD-MNPO without theoretical or empirical validation. Discussion on this would be much appreciated.\n3. TD-MNPO requires a mixture of all previous policies of opponents in its loss function defined in Eq. 18. This can be limiting when either n or T are large. I suggest the authors to comment on how this can be scaled to large n or T.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)