Multiplayer Nash Preference Optimization
Authors: name: Fang Wu; name: Xu Huang; name: Weihao Xuan; name: Zhiwei Zhang; name: Yijia Xiao; name: Guancheng Wan; name: Xiaomin Li; name: Bing Hu; name: Peng Xia; name: Jure Leskovec; name: Yejin Choi
Published: 2025-09-27 04:18:33

Abstract: Reinforcement learning from human feedback (RLHF) has emerged as the standard
paradigm for aligning large language models (LLMs) with human preferences.
However, reward-based methods built on the Bradley-Terry assumption struggle to
capture the non-transitive and heterogeneous nature of real-world preferences.
To address this, recent studies have reframed alignment as a two-player Nash
game, giving rise to Nash learning from human feedback (NLHF). While this
perspective has inspired algorithms such as INPO, ONPO, and EGPO with strong
theoretical and empirical guarantees, they remain fundamentally restricted to
two-player interactions, creating a single-opponent bias that fails to capture
the full complexity of realistic preference structures. In this work, we
introduce Multiplayer Nash Preference Optimization (MNPO), a novel framework
that generalizes NLHF to the multiplayer regime. It formulates alignment as an
n-player game, where each policy competes against a population of opponents
while being regularized toward a reference model. Our framework establishes
well-defined Nash equilibria in multiplayer settings and extends the concept of
duality gap to quantify approximation quality. We demonstrate that MNPO
inherits the equilibrium guarantees of two-player methods while enabling richer
competitive dynamics and improved coverage of diverse preference structures.
Through comprehensive empirical evaluation, we show that MNPO consistently
outperforms existing NLHF baselines on instruction-following benchmarks,
achieving superior alignment quality under heterogeneous annotator conditions
and mixed-policy evaluation scenarios. Together, these results establish MNPO
as a principled and scalable framework for aligning LLMs with complex,
non-transitive human preferences. Code is available at
https://github.com/smiles724/MNPO.