Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment
Authors: Mingzhi Wang
Chengdong Ma
Qizhi Chen
Linjian Meng
Yang Han
Jiancong Xiao
Zhaowei Zhang
Jing Huo
Weijie J Su
Yaodong Yang
Venue: ICLR 2025 Poster (2025)
Keywords: AI Alignment
Mirror Descent
Self-play
Nash Equilibrium
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We propose MPO, a novel game-theoretic approach for preference alignment through self-play，which achives last-iterate convergence to the Nash equilibrium of the original game.

Abstract: Self-play methods have demonstrated remarkable success in enhancing model capabilities across various domains. In the context of Reinforcement Learning from Human Feedback (RLHF), self-play not only boosts Large Language Model (LLM) performance but also overcomes the limitations of traditional Bradley-Terry (BT) model assumptions by finding the Nash equilibrium (NE) of a preference-based, two-player constant-sum game. However, existing methods either guarantee only average-iterate convergence, incurring high storage and inference costs, or converge to the NE of a regularized game, failing to accurately reflect true human preferences. In this paper, we introduce Magnetic Preference Optimization (MPO), a novel approach capable of achieving last-iterate convergence to the NE of the original game, effectively overcoming the limitations of existing methods. Building upon Magnetic Mirror Descent (MMD), MPO attains a linear convergence rate, making it particularly suitable for fine-tuning LLMs. To ensure our algorithm is both theoretically sound and practically viable, we present a simple yet effective implementation that adapts the theoretical insights to the RLHF setting. Empirical results demonstrate that MPO can significantly enhance the performance of LLMs, highlighting the potential of self-play methods in alignment.

Published as a conference paper at ICLR 2025
MAGNETIC PREFERENCE OPTIMIZATION: ACHIEVING LAST-ITERATE CONVERGENCE FOR LANGUAGE MODEL ALIGNMENT
Mingzhi Wangh.2, Chengdong Ma', Qizhi Chen', Linjian Meng", Yang Han Jiancong Xiao*, Zhaowei Zhang', Jing Huo Weijie J. Su Yaodong Yang!*
Institute for Artificial Intelligence, Peking University Beijing Academy of Artificial Intelligence National Key Laboratory for Novel Software Technology, Nanjing University "China Telecom, University of Pennsylvania
ABSTRACT
Self-play methods have demonstrated remarkable success in enhancing model capabilities across various domains. In the context of Reinforcement Learning from Human Feedback (RLHF), self-play not only boosts Large Language Model (L.LM) performance but also overcomes the limitations of traditional Bradley-Terry (BT) model assumptions by finding the Nash equilibrium (NE) of preference- based, two-player constant-sum game. However, existing methods either guarantee only average-iterate convergence, incurring high storage and inference costs, converge to the NE of a regularized game, failing to accurately reflect true human preferences. In this paper, we introduce Magnetic Preference Optimization (MPO), a novel approach capable of achieving last-iterate convergence to the NE of the original game, effectively overcoming the limitations of existing methods. Building upon Magnetic Mirror Descent (MMD), MPO attains a linear convergence rate, making particularly suitable for fine-tuning LLMs. To ensure our algorithm both theoretically sound and practically viable, we present a simple yet effective implementation that adapts the theoretical insights to the RLHF setting. Empirical results demonstrate that MPO can significantly enhance the performance of LLMs, highlighting the potential of self-play methods in alignment.
INTRODUCTION
Self-play has emerged as an effective method for improving model performance, particularly in domains that require strategic decision- -aaing and complex groblem-solving (Silver et al.) [2017] Vinyals et al. 2019 Perolat et al.] (022). By allowing models to iteratively refine their strategies through self-competition, self-play enables them to discover optimal policies. In the realm of
Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022 Peng et al.
2023
Achiam et al. 2023 Jin et al., 2025), self-play not only has proven effective in enabling
Large
Language Models (LLMs) to better align with human preferences (Chen et al. 2024 WWu et al. 2024
Zhang et al. 2024), but also offers unique advantages by addressing the limitations of traditional
preference modeling methods (Munos al. 2023 (Swamy et al. 2024).
Conventional RLHF methods typically rely on the Bradley-Terry (BT) (Bradley & Terry 1952)
assumption for preference modeling, which presumes transitivity in human preferences-iff response A is preferred over B, and B over C, then A should also be preferred over C. While this may hold for individuals in specific contexts, generalizing transitive preferences across broader populations often fails due to the presence of non-transitive preferences (Swamy et al.) 2024). This limitation undermines the ability of existing RLHF methods capture the complexity of human preferences. Self-play, however, offers a solution by finding the Nash equilibrium (NE) of a two-player constant- sum game based on human preferences (Munos et al. 2023 Swamy et 2024).
$ Correspondennc Yandong Yang <yandong. yang Loann@ppk.coo @pk..edu.cn>

---

Published as a conference paper at ICLR 2025
Diverse populations
Optimize preference 2-player games
Non-transitive preferences
Align
Age
Education
B
Gender
Profession
LLM
LLM N3
Game solving
Misalign
RLHF with BT model assumption
Historical models
PPO
He Me :8:
Sealar roward
LLM
Single model
Response ALBIC
B
C
Only transitive preferences
Existing mechods
MPO
Reward model
Regularised NE Original
Reference policy
Figure 2: An illustration of MPO and its background. Non-transitive preferences are prevalent across diverse populations, necessitating more generalized preference model that frames the alignment problem as a two-player constant-sum game. Existing methods either converge to the NE of a regularized game require maintaining multiple models. In contrast, MPO achieves last-iterate convergence to the original NE, aligning with diverse human preferences using only single model.
Despite its promise, self- in the context of LLM alignment presents unique challenges. Most existing methods, such as Self-Play Preference Optimization (SPO) (Swamy et al. 2024), rely on Mirror Descent (MD) (Beck & Teboulle 2003) based Deep RL methods like PPO (Schulman et al., 2017) and SAC (Haarnoja et al. 2018) to learn the NE of the preference-based game. However, from theoretical perspective, MD only guarantees average-iterate convergence to the NE, while the last-iterate policy tends to oscillate around the NE (Mertikopoulos et al. 201 8b a. Perolat et al.) 2021). This limitation implies that a single LLM cannot fully align with human preferences without maintaining multiple models for joint inference, leading to increased storage and computational costs. As shown in Figure [ where the duality gap measures the distance between the current policy and the NE, classic Deep RI methods exhibit poor last-iterate convergence, even in simple Kuhn Poker game. This underscores the importance of achieving last-iterate convergence in RLHF tasks.
On the other hand, Nash Learning from Human Feedback (NLHF) (Munos et al.) 2023) also leverages MD but achieves last-iterate convergence by employing geometric mixture of the current policy and a reference poliy, commonly referred to first-order approximation of the reference policy (Munos) jt al. 2023). However, this approximation lacks rigorous the- oretical guarantees and ultimately only converges to the NE of the KL regularized game, failing to capture true human pref- erences. summary, existing methods fail to obtain a single LLM policy that aligns with human preferences in the orig- inal game. The reliance on multiple LLMss proxies leads to inefficiency and high cost (Swamy et al.) 2024 Wu et al.) 2024| Rosset et al. 2024), while various approximation meth-
PPO
0.8
SAC
Duality Cup
DQN
MMD
147
Figure 1: Kuhn Poker Experiments.
ods result in misalignment (Munos et al. 2023 Calandriello et al. 2024 Zhang et al. 2024). These limitations collectively represent the core challenges in preference alignment LLMs.
In this paper, we introduce the Magnetic Preference Optimization (MPO) framework, which guaran- tees last-iterate convergence to the NE of the original game. This method offers a lightweight and efficient solution for aligning diverse human preferences by utilizing only the final trained model, without the need for storing multiple policies. Specifically, we adapt the insight of Magnetic Mirror Descent (MMD) (Sokota et al. 2022) to the RLHFF context derive MPO and further established
2

---

Published as a conference paper at ICLR 2025
theoretical guarantees for convergence to the original NE. The key insight lies in the periodically updated magnetic policy, which effectively guides the policy towards the NE. Our results show that MPO achieves last-iterate convergence at significantly faster rate than standard Mirror Descent (MD), with empirical evaluations demonstrating substantial improvements in LLM performance, further emphasizing the potential of self-play methods for preference alignment.
PRELIMINARIES
We consider a Large Language Model (LLM) denoted by π II and parametrized by 0 e O. The model receives a prompt x = [a1....,,Tul Fnl, and generates corresponding output sequence y = [y),.... Mmm The output y is sampled from the conditional probability distribution .(- x). In LLMs, Hj and Yi represent individual tokens from predetermined vocabulary V. The model generates tokens autoregressively. producing each token sequentially based on the input and all previously generated tokens. This autoregressive property allows to decompose the conditional probability as
T (y x) = I Ttv xxy,,,,
where Yer = 1 for > 1, and y=1 is an empty sequence.
2.1 TOKEN-LEVEL MDP FORMULATION FOR LLMS
We frame the RLHFF problem as a Markov decision process (MDP) (Puterman, 2014), defined by the tuple M AS, P. R, p, T). In this formulation, S represents the state space, where each state 8; = yet] includes the prompt x and all response tokens produced up to that point. The action space A consists of possible tokens, where each action a, Y/ represents token from the vocabulary V. The policy π. S - (A4) maps states to distributions over actions. The transition kernel p S A - A(S) describes the dynamics of the environment. In the context of LLMs, Yt, the environment will transition 8++1 [x, Y [++1] with probability 1. The token-wise reward function R S x A I is defined as R == R(st, ar) = R( [x, y=c, 3r). The accumulative reward for the generated text is DT [ < Mt). The initial state distribution is determined by the distribution of input prompts, while denotes the maximal interaction steps, characterizing the length limit for outputs.
this transition is deterministic: given
yx, and
2.2 TWO-PLAYER CONSTANT-SUM GAMES AND MIRROR DESCENT
We consider a constant-sum game where the sum of payoff's for any outcome remains constant. Let I {1, 2) represent set of two players, and II; R denote the compact, convex strategy space for player i. The joint strategy space is defined as x II, The strategy of player i is denoted by T, E II, while the strategies of all other players, denoted by ズ_ lie in II, x (e II, where -i refers to all players except player i. Each player has a continuous payoff function f. - R. In two-player constant-sum normal-form game, both players simultaneously choose their strategies, T E Пh and T2 E IT,, respectively. The payoff for player is then given by fa(T;; T-i), where the sum of the payoffss satisties LieT fi(T,, T-)) = c, with E 12 being constant.
To determine the optimal strategies for both players, it is essential to find the Nash equilibrium (NE) of the game. An NE strategy profile (xf, nT such that neither player can improve their payoff by unilaterally deviating from it:
f n & f 2 & f(Tt,(1))
V(T T2) e II.
In a two-player constant sum game, the NE strategies for both players are unique and identical, i.c.,
of n. * (Zhang et al. 2024 Ye et al. the game (i.e., f is convex-concave
2024: Swamy et al. 2024). Given the monotonicity of
it is well known that finding the NE is equivalent to solving the
associated Variational Inequality (VI) problem (Mertikopoulos & Zhou 2019 Sokota et al. 2022).
Formally, let F be the monotone operator defined as F(T)) (V f( [[TI 72), - V T2))
The NE # is solution the VI problem VI(II, which requires finding 7* e Il such that:
<F ) n 20,
VT e IT.

---

Published as a conference paper at ICLR 2025
To measure the distance from given strategy profile π to the NE, we define the duality gap (Wei ct al. 2020 Abe et al. (2024) as
<(T)
= max ætell
V. T
n.>
Mirror Descent (MD) [Bekk & Teboulle 2003 Beck 2017) is a first-order optimization algorithm capable solving such games. The update rule for MD applied to player i given by:
1
++1 arg min (F(T*) π) By m²)
" where 0 is the learning rate, and (,(TTx)) (TT) ( (v)r'), is the Bregman divergence associated with a strongly convex function t). In a two-player constant-sum game, if both players follow the MD update rule, the algorithm achieves average- average-iterate iterate convergence to the Nash equilibrium (NE). Formally, average-iteria convergence is defined as follows:
Definition 2.1 Evverag--ttertte Convergence). Given non-empty set of equilibria IT = II, sequence (k=) is said to exhibit average-iteran convergence t converges to some n* e II as
K - cc, where , K Σ Th
2.3 RLHF WITH BRADLEY- PRADLEY-TERRY MODEL.
In the standard RLHF pipeline, where the true reward function is unknown, reward model ra parameterized by Φ is trained using a dataset D = (x, Ym, where yw: represents the preferred response over yt. The distribution of the preference dataset is assumed follow the Bradley-Teryy (BT) model (Bradley & Terry 1952 Christiano et al. 2017)
exp )n
Fy v1(x) exp(ra(x, yw)) + exp(r a(ro(x,yw) To vx))
(1)
where a 1/(1 + (xp(-x)) is the sigmoid function. Based the dataset D, the reward model is trained by minimizing the negative log-likelihood [
((r²) yxx.y....) (x,, yw) rx(xx, yx)]
Given the trained reward model ræ. online RL algorithms, typically PPO (Schulman et al. 2017 are leveraged to optimized the following objective
max BaxxExDDyy-tte [r y)] - a [xL[(T() [ (er( xx]]
(2)
where a controls the strength of KL penalty. The KL-regularized objective (Li et al. 2024 is
widely adopted to prevent from deviating too much from the reference policy (Ziegler et al. 1909
Liu et 2020 Ouyang et
2022, Zheng et al.) (023).
2.4 RLHF WITH GENERAL PREFERENCE
Although the BT model has been widely adopted in RLHF for modeling human preferences, it has many limitations, including independence comparisons, linearity of preferences, transitivity and so on (Shah et al. [0166] Lanctot et al., 2023). Recent works (Munos et al. 2023 (wamm et al. (024) propose modeling the RLHF problem symmetric two-player constant-sum game. This approach introduces a preference model P y2 x). which defines the preference between two policies as
P(T 2 = -.r (P > y2 x)]
Typically, leveraging the capability of L.MMs as next-token predictors for preference modeling (Dong ct al. 2024. Munos et al. 2023 Jiang et al. 2023). The preference model can be trained via a cross
entropy loss
C(P) =
P(y1 > y2 x)],
where is the dataset of annotated preference pairs. Unlike the BT model, the preference model does not assume a global intrinsic quality score for each response, thereby enabling the modeling of intransitive preferences.
Given the preference model, the NE of this game is defined as
arg max min P(T1 > 72). W1
(3)
Intuitively, this NE represents a policy that minimizes the worst-case scenario dissatisfaction and satisfies variety of desirable consistency properties in social choice theory (Swamy al.) 2024).
4

---

Published as a conference paper at ICLR 2025
3 MAGNETIC PREFERENCE OPTIMIZATION
In this section, we introduce the Magnetic Preference Optimization (MPO) algorithm based on Magnetic Mirror Descent (MMD) (Sokota al. 2022), which enjoys theoretical guarantee for last-iterate convergence to the Nash equilibrium (NE) of the game.
3.1 MAGNETIC MIRROR DESCENT
While the game defined in B can be solved when both players employ Mirror Descent (MD) (Swamy et al. 2024), only the average-sequence { 1ka1 converges to the NE, where Æ* 1
K Σx= n².
The actual sequence {x*) kaa, as is shown in Figure 3 does not converge and cycles around the
NE (Mertikopoulos al. 2018b Perolat al. 2021
In the context of LLMs, this limitation poses significant practical challenges. Average-iterate convergence necessitates the storage of historical policies, leading to prohibitively high storage and inference costs. This limitation raises key question: can we devise an algorithm that achieves last-iterate convergence, thereby circumventing the need for storing and averaging over historical policies?
MNDD. an1d MMD a-23
One solution to this problem is Magnetic Mirror Descent (MMD) (Sokota et al. (022). To better understand MMD, we first define last-iterate convergence.
Definition 3.1 (Last-Iterate Convergence). Consider nonempty set of equilibria II c II, we say that sequence (T*) Jas1 exhibits last- iterate convergence if nh converges to n* II as k - sp.
Figure MD and MMD,
Compared to MD, MMD introduces an additional magnetic term. Formally, the MMD update rule can be expressed as
1 7)
arg min xell
( (Fiin) (B meet) + Bw: n²)),
where Tet is the magnet, which means * is attracted to either min (T)) or Tiets is the regularization temperature, is the learning rate. In contrast to MD, MMD solves the regularized game
min max ag([T) + [(11, T2) - ag(m2),
(5)
Kiell sgell,
where f and g are both convex functions and g can be taken either v or Bs Rret) for some Tref- Let n. be the solution of (5). This problem corresponds to the following VI problem VI(II, F Vg)
(F(#;) Vg(=;)) N, O, VI
The key advantage of MMD lies in its ability to achieve linear last-iterate convergence to the NE of the regularized game. This property formally stated the following theorem Sokota al. (022):
Theorem 3.2 (Theorem 3.4, (Sokota et al.) (2022). Consider the MMD update rule in (4. Assume k+1 E int dom &p and IT is bounded, I-strongly convex relative to over II with differentiable over int dom tb. Then the sequence haat generated by MMD exhibits linear last-iterate convergence to the solution n. if & it. Specifically,
is monotone and I-smooth with respect to 1. 9 is
Bølm;; (o(T#; x1
1 17ce
where a > is the regularization temperature and v > is the learning rate.
Theorem 3.2 demonstrates that when both players follow the MMD update rule in (4) their policies converge to the NE क of the regularized game with a last-iterate convergence rate of ((11(1 )) This property is particularly valuable in the context of LLMs, as it eliminates the need for storing and averaging over historical policies, and achieves much faster convergence rate than vanilla MD, which converges at ((1/Vk) (Beck 2017). This substantially reducing computational and storage requirements.
5

---

Published as a conference paper at ICLR 2025
3.2 CONVERGENCE TO THE NASH EQUILIBRIUM OF THE ORIGINAL GAME
While Theorem 3.2 establishes the last-iterate convergence property of MMD, it only converges to the NE of the regularized game, not the original one. As illustrated in Figure increasing the regularization strength accelerates MMD convergence, but simultaneously causes the learned NE to deviate further from the NE of the original game. This deviation potentially leads to equilibrium that fails to accurately reflect the true human preferences. Consequently, we face a crucial challenge: how can we achieve last-iterate convergence to the NE of the original game defined in
Formally, we define the n-th regularized game as
(2) min max P( > π2) + (1 --) a=DKL(TTll + mell mllll,
"
(6)
where x.. -- is the NE of the (n 1)-th regularized game. Intuitively, as the number of iterations increases, expect the sequence of regularized NEs, [T...) In31. to converge to the original NE π*.
This intuition formalized the following theorem.
Lemma 3.3. Let ( juat be the sequence of NEs of the regularized games generated by iteratively solving (6.. where n. is an arbitrary inilial reference policy in the interior of II. For any n. 2 1, if
T II $ II", we have
min DKL (T (m < min (T **
**
Otherwise, if ,, _11,, then n. **,+ = म. " II*
Theorem 3.4. If Lemma 3.3 holds, the sequence { Jnal converges to the NE n* II of the original game defined in (3) as n cC.
Theorem 3.4 suggests a two-stage convergence process for MMD to reach the NE of the original game. First, as established in Theorem 3.2. MMD achieves linear last-iterate convergence
the NE of each regularized game. Then, by iteratively updating the magnet policy to the most recent regularized NE, we guide the sequence of regularized NEs (m*""" Jnal towards the NE n* of the original game (Meng et al.
2023 Perolat et al. [202T). Importantly, any algorithm that guarantees last-iterate convergence
(e.g. Nash- MD (Munos et al., (023)) ) can be employed to solve the regularized games. Additionally,
Theorem F.4 extends the analysis to scenarios where only an approximate solution to π. T. is obtained.
3.3
PRACTICAL IMPLEMENTATION
We now describe how to adapt these theoretical insights into a practical, computationally efficient algorithm for RLHF with general preference models.
In the context of RLHF, the MMD update rule in 4 can be expressed as:
k+1 arg max Ex-p.l1- [A (x, - aell
DKL
(7)
where 4 denotes the advantage function. To compute A assign the preference P(y > Y2 x)
between two responses y, and y2 as the token-level reward RG- The advantage function then calculated via RHINFORCE (Williams) 1992), where the baseline can be computed using ReMax (Li
let al. 2023) or Leave-One-Oun (Kool et al., 2019) Ahmadian et al. 2024). The KI. divergence in between two LLM policies गi and T2 estimated as:
( (x2)
Σ (mu(m [x. x [x,y=())).
This estimation, also known as the sequential KL divergence (Zeng et al. 2024), has shown effective in controlling the KL divergence between two LLM policies. The optimiattion objective in (7) can be written in parameter space θ as:
max MMD (0) max Ex-D By Boy= (x. aDKI mor) 1 DK.. (TollaaT))) (8)
To optimize this objective, we draw inspiration from Mirror Descent Policy Optimization (MDPO) (Tomar et al. 2020), which provides an RI. implementation of MD, making it suitable for

---

Published as a conference paper at ICLR 2025
algorithm. Similar to MDPO, take multiple gradient steps at each iteration i: to ensure trust region constraints and introduce an annealed stepsize, 7f = 1 k/Tg. where Tę is the maximum number of iterations. These considerations culminate in our MPO algorithm, detailed in Algorithm
An important aspect of MMD that it is equivalent to solving the regularized objective in (5) using standard MD with specific stepsize (Sokota al. 2022). formalized in the following theorem.
Theorem 3.5 (Proposition D.7, (Sokota al. 2022) to the following rule:
The update rule of MMD in is equivalent
π h+1
arg min (F(**) + aV Bp moot), T) + == Tell
where the stepsize is defined as īj I4na*
7
This theorem establishes connection between MMD and MD, showing that MMD can be seen special case of MD with an adjusted gradient and stepsize. This insight allows us to derive a theoreti- cally equivalent algorithm, which refer to MPO-RT (i.e., Reward Transformation (Perolat et al., 2021; 2022 Meng et al. (202)).. ) In contrast to MPO, MPO-RT enforces a hard constraint on KL divergence by directly modilying the reward function. This approach aligns with the idea of standard RLHF methods (Orabona) 2019 Ouuang et al. 2022 Zheng et al. 2023). Detailed discussion and additional results for MPO-RT are provided in Appendix A.2
Although the theoretical framework requires simultaneous updates for both players to converge to the NE, this presents practical challenges RLHF due the added memory costs. By leveraging the symmetry of the game, we can simplify the approach by requiring only single player to play against its own iterates (Swamy et al. 2024 Ye al. 2024).
Another challenge arises from Theorem 3.4 which implies that convergence to the NE of the original game requires periodically updating the reference policy with the NE the previous regularized game. In practice, however, it can be difficult to determine when the current policy has reached the NE. Therefore, we update the reference policy every T iteration, when a predefined number of iterations Z'g is reached (Abe et al. 22024), i.e., गh π TT',, In this case, the convergence rate to the NE of the regularized game depends on the value of Tk. Formally, we present the following lemma.
Lemma 3.6. Consider the sequence ( generated by the update rule in 7 where the reference policy N.. is updated every Th iterations x, TTTE. Assume o. e int dom s and that P is monotone and L-smooth with respect to . Then, the sequence [x1) satisfies:
DKL (T (m#Im=++)))) & On (1
Ts 11+ (1+na) )
(
where > 0 is the regularization temperature, and 1 > O is the learning rate.
Lemma 3.6 immplies that when the update interval Tę is sufficiently large, m,+1 closely approximates
n.. Based on this result, we derive the following theorem.
Theorem 3.7. If Lemma 3.6 holds, the sequence )r]1 converges to n* as T - sc, where x the Nash equilibrium of the original game defined in (3)
In practice, detailed in Algorithm I we simultaneously replace both the opponent and the reference policy with the current policy T every Tę iterations. This implementation reduces storage requirements by maintaining only one additional model, while capturing the key insight of Theorem b.4by periodically updating the reference policy.
4 EXPERIMENTS
In this section, we conduct comprehensive experiments to validate the effectiveness of our proposed MPO algorithm. We start by focusing on safety as the primary alignment metric. Since safety is a single, well-defined dimension, it provides clear and straightforward way to evaluate the algorithm's 's ability to align with human preferences. We then extend the evaluation to the model's general capabilities, offering
complex, diltiiidmeesionnll analysis. This broader evaluation explores MPO's s scalability and examines how specific abilities improve or decline during self-play, providing deeper insights into the strengths and limitations of self-play methods in alignment.
7

---

Published as a conference paper at ICLR 2025
4.1 SAFETY ALIGNMENT EVALUATION
Experiment Setup. We use the open-source LLM Ge (Team et al., 2024) as our base model. Following the methodology of (Dai et al. 2023), we first perform supervised fine-tuning on the Alpaca TTaori et al. 2023) dataset, which we refer to as the Gemma-2B-STe train a preference model that avoids overfitting to specific tasks and is suitable for general-puroose use, train our preference model
model. To
mixture of widely-used open-source preference datasets] resulting in a preference model we term Gemma-2B-PMM The RewardBench (Lambert et al. 2024) scores for Gemma-2B- P- are presented in the Appendix B Next, we fine-tune the SFT model using prompts sourced from the PKU-SafeRLEE the Anthropic Helpful and Harmless dialogue (HH) (Bai et al.) 2022) dataset. These prompts are equally divided and used over three rounds of self-play. More experimental details are available in Appendix B
(Ji et al., 2024 dataset and the HH-Harmless section of
Evaluation Metrics. We employ two evaluation metrics to validate the effectiveness of our methods: (I) Cost Model Based Evaluation. We evaluate the performance our method using the publicly available cost models released by (Dai et al. 2023). (2) GPT-4o Based Evaluation. We use the same prompts as (Dai et 2023) ask GPT-4o to compare the quality of responses generated by two models under identical imputs. We use evaluation questions from the official codebase (Dai et al. 2023), which includes eight safety-related categories. This approach allows us analyze the models' performance across various safety-related dimensions.
Models/Datasets PKU-SafeRLHF |
HH- HHH-Hamleess
SFT
6.37
11.33
MPO Iter.1
-3.76
4.93
MPO Iter.2
-8.96
-0.87
MPO Iter.3
-11.38
-3.86
MPO wo. SP
-4.18
6.41
Table 1: Cost model evaluation results.
Result Analysis. We present our cost model evaluation results in Table where lower cost means safer outputs. From the table, across three self-play iterations on both datasets. The GPT-4o evaluation results, shown in Table 2 reveal that method substantially improves the model's win rate compared to the SFT model. This pattern is further illustrated in Figure 4 where our approach consistently boosts the win rate across eight safety-related categories. These comprehensive results underscore the effectiveness of our method in aligning LLMs with human preferences. Additionally, we also investigate the case where celf-play omitted, and the results show significantly poorer performance compared the self-play setting. Since both settings are fine-tuned using the same amount of prompts, this reveals an important insight: while RLHF with BT models runs the risk of overfitting to the reward model, RLHF with general preference models faces the risk of overfitting to the opponent. In this context, self-play is not simply a strategy to enhance performance but a necessity to prevent degradation. Without self-play, the model's performance is severely compromised, underscoring the critical role of self-play in maintaining robust performance.
observe that method significantly enhances model safety
Ablation Study. To further evaluate the effectiveness of each in- dividual component of MPO, we conduct an ablation study shown baseline method to scenario
Baseline
in Figure 5 First, we compare where the reference policy is fixed throughout the entire training process. Theoretically, this approach is expected converge only to the Nash equilibrium (NE) of the regularized game. The results in the figure confirm this, with the baseline method significantly outperforming the fixed policy case, indicating the importance of periodically update the reference policy. We also explored replac- ing the KL divergence loss with clipping to enforce trust region constraints. The results show it perform worse than KI. loss.
Fix Ref.
Clip Adv.
-6
-8
-10
1ax.1
Figure 5: Ablation study.
wttp:://wnnnncccccocaeeeee//eeep mixturci aaaeeeaaannnndnnn_neeaaaaee safe_pku
8

---

Published as a conference paper at ICLR 2025
GPT-4o-Evaluation
100
50
MPO
MPO
Settings
Win
Lose
Tie
MPO Iter.
51.8%
21.7%
26.5%
MPO Iter.2
69.9%
10.8 %
19.3%
MPO Iter.3
79.5%
9.6 %
10.9%
MPO wo.SP
30.1% %
15.7%
54.2%
Spciil Blas
Poregraphii
Crinar
Privney
Emotiona Harm
Plysbaal Herm
lnndl
Immural
Table 2: MPO demonstrates a steady improve- ment in win rates across three iterations. In contrast, MPO without self-play underper- forms, even compared to the first iteration of self-play.
Figure 4: Performance across each safety- related category for three self-play iterations of MPO.
4.2 GENERAL CAPABILITY ALIGNMENT AND ANALYSIS
base model. Following the recipe (Dong et al. 2024), we first perform supervised fine-tuning on the open-source SFT- SFT-OppeHermes- -tannard dataset. The obtained SFT model serves good foundation for our experiments. Next, we train a preference model based on mixture of open- preference datase We then fine-tune the SFT model using 30K prompts selected from collection of Ultra Feerbaee (Cui et al. 2023), HelpSteer (Wang et al. 2023), OpenOrea (Lian et al., 2023). Ultrlltteract (Yuan et al.) 2024), Capyhara (Daniele Suphavadeeprasit 2023) datasets for four rounds of self-play. More experimental details are available in Appendix and we also conduct
Experiment Sctup. We the open-souce LLM Llama-3-8B et al. 2024)
general capability alignment experiments of Gemma-2B-SS detailed in Appendix D.
Evaluation Metrics. To evaluate the general capabilities of the self- play fine-tuned models, we employ MixEval (Ni al.. 2024) and Open LLM Leaderboard v2 (Fourrir et al.) 2024) for benchmark evaluation. MixEval generates scores by combining real-world user queries with traditional benchmark queries, creating a more compre- hensive and realistic assessment. It exhibits strong correlation 0.93 with human preferences, as demonstrated by its alignment with the Chatbot Arena Elo (Chiang et al. 2024), a widely recognized gold standard for user-facing evaluations. Additionally, MixEval- Hard, a more challenging subset of this benchmark, shows a higher correlation of 0.96 with Chatbot Arena Elo. Open LLM Leaderboard v2 a popular benchmark released by Huggingface for evaluating the performance of LLMs. By replacing the original evaluation tasks with much more difficult ones, Open LLM Leaderboard v2 are more challenging and meaningful for evaluating LLMs. These benchmark evaluation results ensure that our evaluation closely aligns with human judgment.
46
40.
75.5
Figure 6: MixEval Hard score.
Model
IFEva
BBH
Math Hard
GPQA
MUSR
MMLU PRO
Average
SFT
41.63
48.54
4.87
28.95
42.32
32.64
33.16
MPO Iter.
41.61
50.72
5.02
30.12
42.25
32.79
33.75
MPO Iter.2
42.36
50.30
4.61
30.29
41.93
32.81
33.72
MPO Iter.3
42.75
51.22
5.51
30.12
40.61
32.81
33.84
MPO Iter.4
42.97
51.38
5.06
30.54
40.87
32.85
33.95
Table Evaluation results on Open LLM Leaderboard v2.
Result Analysis. We present the overall scores from the MixFval-Hard evaluation in Figure 6
with detailed results shown in Figures and baseline comparisons included in Appendix D. From Figure 6 we observe consistent improvements in model capabilities across iterations of self-

---

Published as a conference paper at ICLR 2025
play. Notably, Figure shows significant gains in categories like MBPP (programming) and PIQA (physical commonsense reasoning). reflecting enhanced logical reasoning and problem- skills. CommonsenseQQA and BBH also demonstrate steady progress, indicating better performance in general knowledge and high-level reasoning tasks. However, categories like OpenBookQA and SIQA (social interaction reasoning) show limited improvement after Iteration 2, while BoolQ experiences a slight decline. Overall, self-play significantly enhances the model's logical reasoning and commonsense understanding, but specific knowledge tasks may require additional fine-tuning, For Open LLM Leaderboard v2, the evaluation results in Table 3 show consistent improvement in average scores. Iteration 4 outperforms earlier iterations several key benchmarks, including IFEval, BBH, and MMLU PRO, highlighting the model's enhanced ability to handle reasoning and general knowledge tasks. However, progress remains slower in the Math Hard, where scores are consistently lower. This suggests that while MPO iterations are becoming more robust overall, further optimization needed for specialized tasks like complex mathematical reasoning.
5 RELATED WORK
Bradley-Teryy Model Based RLHE RLHF has seen significant success in aligning LLMs with
human preferences (Ouyang al. 2022 Peng al., 2023 Achiam et al.) 2023). Classical RLHF
methods typically scalarize human preferences into rewards (Zheng et al. [023] Wang et al. 2024
Dong et al.
2024 Xiao al., 2024) and optimize a KL-regularized objective using PPO (Schulman
et al.
2017 To streamline the RLHF process, Direct Preference Optimization (DPO) (Rafailov et al.
2024)
directly learns a policy from preference datasets. Extensions like Online DPÒ (Dong et al.
2024
Tang al. 22024] Chen et al. 2024) enhance DPO's performance by adopting an iterative learning approach. However, all these methods are fundamentally based on the Bradley-Terry (BT) model (Bradley & Terry 1952). which assumes transitivity in human preferences
limitation that
has been noted in the literature (Cattelan, (2022] Swamy et al.) [2024] Munos et al. 2023). In contrast, explore a more general preference model and adopt a theorett approach to better capture the complexity of human preferences.
RLHF Two-Player Constant-Sum Game To address the limitations of the BT model, recent research has proposed reformulating the RLHF problem as a two-player constant-sum game (Munos) al. 2023 Swamy et al. 2024, Chen et al., 2024). In this setup, the goal is to find the Nash equilibrium (NE), which represents the optimal policy distribution that accounts for diverse and often conflicting human preferences. Following this framework, Self-play Preference Optimization (SPO) (Swamy et al. 2024) learns the NE of the original game through MD. This approach only achieves average-iterate convergence while the last-iterate policy cycles around the NE. contrast, our method achieves last-iterate convergence. Nash Learning from Human Feedback (NLHF) (Munos) al., (023) learns the NE of the KI.-regularized game through Mirror Descent (MD) (Beck & Teboulle 2003 Beck 2017). However, NLHF achieves only sublinear last-iterate convergence, relying on a geometric mixture reference policy. In contrast, our method achieves linear last-iterate convergence and ensures convergence to the NE of the original game. Another line of work seeks to directly learn the NE over a preference dataset, these methods either only guarantees average-iterate convergence (Wu al.] [2024] Rosset al. 2024) only achieves last-iterate convergence of the NE of the regularized game (Calandriello et al. 2024 Zhang et al. 2024).
CONCLUSION
This paper introduced Magnetic Preference Optimization (MPO), a novel framework for aligning Large Language Models (LLMs) with human preferences through self-play. MPO achieves last- iterate convergence to the Nash equilibrium (NE) of the original preference game, offering significant improvements over traditional approaches that rely on average-iterate convergence or regularized games. Our experiments demonstrate that MPO significantly improves model performance. Overall, MPO represents a robust and efficient approach to LLM alignment, highlighting the potential of self-play methods in aligning diverse human preferences.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Kenshi Abe, Kaito Ariu, Mitsuki Sakamoto, and Atsushi Iwasaki. Adaptively perturbed mirror descent for learning in games, 2024. httptttsss///axxv.oooo
2305. 16610)
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprini www.x:2333.087722 2023.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXi:: 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204. 05862, 2022.
Heinz H. Bauschke, Jonathan M. Borwein, and Patrick L. Combettes. Bregman monotone opti- mization algorithms. SIAM Journal on Control and Optimization, 42(2):596-66 636, 2003. doi: 10.1 137//000663119992077222
Amir Beck. First-order methods in oplimization. SIAM, 2017.
Amir Beck and Marc Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31(3):167-175, 2003.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-444 1952.
Yang Cai, Argyris Oikonomou, and Weiqiang Zheng. Finite-time last-iterate convergence for learning in multi mullti-layer games. Advances Neural Information Processing Systems, 35:33904-33919 2022.
Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, et al. Human alignment of large language models through online preference optimisation. arXiv preprint arXiv:2403.08633 2024.
Manuela Cattelan. Models for paired comparison data: A review with emphasis on dependent data. Statistical Science, 27(3), August 2012. ISSN 0883-4237. doi: 10. 10.1214/12-st5396 URL http://áx. wttp://.........1111221121-1-010 121.101-11111111111
Volkan Cevher, Georgios Piliouras, Ryann Sim, and Stratis Skoulakis. Min-max optimization made simple: Approximating the proximal point method via contraction maps. In Symposium on Simplicity Algorithms (SOSA), pp. 192-206. SIAM, 2023.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.0133 2024.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating 11ms by human preference. arXiv preprint arXiv:2403.04132. 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao. Wei Zhu, Yuan Ni. Guotong Xie, Zhiyuan Liu, and Maosong Sun. Utrafeedback: Boosting language models with high-quality feedback. arXiv preprint ar arXiv:2310.01377 2023.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310. 12773, 2023.
11

---

Published as a conference paper at ICLR 2025
Luigi Daniele and Suphavadeeprasit. Amplify-instruct Synthetically generated diverse multi- turn conversations for efficient Ilm training. arXiv preprint arXiv:(coming soon), 2023. URL
https://huggiagface co/ datasets/ Sittasets/LDur//// LDUnr/Capybar.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang. Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07866 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2477.2 1783, 2024.
Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open IIm leaderboard v2. https: hattppa:/hhgge /uggg http:/acc.....pppcccpeeeeeeeeeee 110 11m- 1m-leemerboarrrd open 11m leaderboard 2024.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning. pp. 1861-1870. PMLR, 2018.
Jian Hu, Xibin Wu, Weixun Wang, Dehao Zhang, Yu Cao, et al. Openrlhf: An easy-to-use, scalable and -eigh-peffrmmacee rihf framework. arXiv preprint arXiv:2405. 11.4005.1113,,... 2024.
Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun,
Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of I1m via human-prefee -uman-preference dataset. Advances in Neural Information Processing Systems, 36, 2024.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LIm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint wwXiv:2330.02256// 2023.
Ruochen Jin, Bojian Hou, Jiancong Xiao, Weijie Su, and Li Shen. Fine-tuning attention modules only: Enhancing weight disentanglement in task arithmetic, 2025. URL
0bbs/2407.07089
Wouter Kool, Herke van Hoof, and Max Welling. Buy REINFORCE samples, get baseline for free!, 2019.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024. URL htthttps: //arxiv
org/abs/2403. 13787
Mare Lanctot, Kate Larson, Yoram Bachrach, Luke Marris, Zun Li, Avishkar Bhoopchand, Thomas Anthony, Brian Tanner, and Anna Koop. Evaluating agents using social choice theory. arXiv preprint arXiv:2312.03121.1 2023.
Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: simple, effective, and efficient reinforeement learning method for aligning large language models. In Forty-first International Conference on Machine Learning. 2023.
Ziniu Li, Congliang Chen, Tian Xu, Zeyu Qin, Jiancong Xiao, Ruoyu Sun, and Zhi Zhi-Quan Luo. Entropic distribution matching in supervised fine-tuning of 1lms: Less overfitting and better diversity. arXiv preprint arXiv:2408. 16673, 2024.
W Lian, B Goodson, E Pentland, et al. Openorca: An open dataset of gpt augmented flan reasoning traces, 2023.
Fei Liu et al. Learning to summarize from human feedback. In Proceedings of the 58th Annual Meeting of the Asxociation for Computational Linguistics, 2020.
Linjian Meng, Zhenxing Ge, Wenbin Li, Bo An, and Yang Gao. Efficient last-iterate convergence algorithms in solving games. arXiv preprint arXiv:2308. /256, 2023.
12

---

Published as a conference paper at ICLR 2025
Panayotis Mertikopoulos and Zhengyuan Zhou. Learning in games with continuous action sets and unknown payoff functions. Mathematical Programming. 173:465 5076655577220 2019.
Panayotis Mertikopoulos, Bruno Lecouat, Houssam Zenati, Chuan-Sheng Foo, Vijay Chandrasekhar, and Georgios Piliouras. Optimistic mirror descent in saddle-pon problems: Going the extra (gradient) mile. arXiv preprint
2018a.
Panayotis Mertikopoulos, Christos Papadimitriou, and Georgios Piliouras. Cycles in adversarial regularized learning. In Proceedings of the twenty-ninth annual ACM-SIAM symposium on discrete algorithms, pp. 2703-2717. SIAM, 2018b.
Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, al. Nash 2023.
learning from human feedback. arXiv preprint
Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, and Yang You. Mixeval: Deriving wisdom of the crowd from II benchmark mixtures. arXiv preprint 240.064000066555, 2024.
Francesco Orabona. A modern introduction to online learning, arXiv preprint arXiv:1912.13213, 2019.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, arXiv preprint arXiv:2304.03277, 2023.
Julien Perolat, Remi Munos, Jean- Cean-Baptiste Lespiau, Shayegan Omidshafici, Mark Rowland, Pedro Ortega, Neil Burch, Thomas Anthony, David Balduzzi, Bart De Vylder, et From poincaré recurrence to convergence in imperfect information games: Finding equilibrium via regularization. In International Conference on Machine Learning, pp. 8525-8535. PMLR, 2021.
Julien Perolat, Bart De Vylder, Daniel Hennes, Fugene Tarassov, Florian Strub, Vincent de Boer, Paul Muller, Jerome T Connor, Neil Burch, Thomas Anthony, et al. Mastering the game of stratego with model-free multiagent reinforcement learning. Science, 378(6623):990 :9900996, 2022.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
J. B. Rosen. Existence and uniqueness of equilibrium points for concave n-person games. Econo- metrica, 3203)54, 1965. ISSN 00129682. 14680262. URL http:://wwwwjstrrroo
stable/1911749
Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.037155 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17Xiv:1707.06347, 2017.
Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Nino Vieillard, Alexandre Ramé, Bobak Shariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, et al. Bond: Aligning 1lms with best-of-n distillation. arXiv preprint arXiv:2407.1462. 2024.
Nihar Shah, Sivaraman Balakrishnan, Aditya Guntuboyina, and Martin Wainwright. Stochastically transitive models for pairwise comparisons: Statistical and computational issues. In International Conference Machine Learning, pp. 11-20. PMLR, 2016.
13

---

Published as a conference paper at ICLR 2025
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou. Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 55017676)355-3555 2017.
Samuel Sokota, Ryan D' D'Orazio, Zico Kolter, Nicolas Loizou, Marc Lanctot, Ioannis Mitliagkas, Noam Brown, and Christian Kroer. A unified approach to reinforcement learning, quantal response equilibria, and two-player zero-sum games. arXiv preprint arXiv:2206. 2200::2206008825,, 2022.
Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. A minimaxi- malist approach reinforcement learning from human feedback. arXiv preprini arXiv:2401.00 044056 2024.
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning Ilms should leverage suboptimal, on-policy data. arXiv preprint vrXi:2244.14337. 2024.
Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, et al. Understanding the perfor- mance gap between online and offline alignment algorithms. arXiv preprint M.Niv:2405008444 2024.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An cnstructionfoolloiing Ilama model. st st anford alpaca, 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08290 2024.
Manan Tomar, Lior Shani, Yonathan Efroni, and Mohammad Ghavamzadeh. Mirror descent policy optimization. arXiv preprint arXiv:2005.09814 2020.
Paul Tseng. On accelerated proximal gradient methods for convex-concave optimization. SIAM Journal on Optimization, 2(3), 2008
Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg. Wojeiech M Czarnekki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, et al. Alphastar: Mastering the real-time strategy game starcraft ii. DeepMind blog, 2:20, 2019.
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rlhf in large language models part i:: Reward modeling. arXiv preprint arXiv:2401.06006 2024.
Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, et al. Helpsteer: Multi-attribute helpfulness dataset for steerlm. arXiv preprint 1rXiv:2311.09528 2023.
Chen-Ye Yu Wei, Chung-Wei Wei Lee, Mengxiao Zhang, and Haipeng Luo. Linear last-iterate convergence in constrained saddle-point optimization. arXiv preprint vrXiv:2006.09517, 2020.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229-256, 1992.
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:240S.00675. 2024.
Jiancong Xiao, Ziniu Li, Xingyu Xie, Emily Getzen, Cong Fang. Qi Long, and Weijie J Su. On the algorithmic bias of aligning large language models with rlhf: Preference collapse and matching regularization. arXiv preprint arXiv:2405. 16455, 2024.
Chenlu Ye, Wei Xiong, Yuheng Zhang, Nan Jiang, and Tong Zhang. A theoretical analysis of nash learning from human feedback under general kl-regularized preference. arXiv preprint 2:X1::2400.077344 2024.
14

---

Published as a conference paper at ICLR 2025
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. Advancing Ilm reasoning generalists with preference trees. arXiv preprint XXiv:2444.02078,, 2024.
Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token-level direct preference optimization. arXiv preprini arXiv:2404. 1999, 2024.
Yuheng Zhang, Dian Yu, Baolin Peng, Linfeng Song, Ye Tian, Mingyue Huo, Nan Jiang, Haitao Mi, and Dong Yu. Iterative nash policy optimization: Aligning Ilms with general preferences via no-regret learning, arXiv preprint rrXiv:2477.0061, 2024.
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. Secrets rlhf large language models part Ppo. arXiv preprint TrXiv:2307.04964 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffreey Wu, Tom B Brown, Alec Radford. Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arxiv 2019. arXiv preprint arXiv: (900:v:909.000000 090908593, 1909.
15

---

Published as a conference paper at ICLR 2025
A PSEUDOCODE AND IMPLEMENTATION DETAILS
In this section, present the pseudocode and implementation details for our proposed MPO and MPO-RT algorithms. Both algorithms take multiple SGD steps to ensure trust region constraints. The key difference lies in how they handle KL regularization: MPO employs KL loss prevent significant deviation from the reference policy, while MPO-RT modifies the reward directly. The baseline can be computed using ReMax (Li al.
)023 Leave-One- Ont (Ahmadian al., 2024). or simply set to 1/2 as suggested in (Munos et al. 2023) In our experiments, we adopt ReMax for both algorithms due its ability to provide strong baseline with only one additional sample.
A.1 MPO
The pseudocode of MPO provided in Algorithm
Algorithm MPO
Input: Initial policy Ty. preference model Pa. dataset D of prompts, regularization temperature a, learning rate 7), update interval Tk, max iterations K
Initialize: Reference policy Thr Tp. l: <- 0, -
for k ...... K do
2: Sample batch D, {x from D
3: for i 1, ,,.....33....... NN do Sample responses y1 for / ,, T
4:
To(-1x;) V2 ~ no. (+1x;)
5:
6:
Compute preference R. Plyl y2 x:)
Estimate advantage A, 21 2i-1 ff
7:
8:
end for end for
9:
10:
Update policy Tge by perform m SGD steps on g
11:
of = Ok
(0)
12:
for j = L...., m do
13:
A (j+1) (3)
V TTDD l0
14:
end for
15:
gk+11
of
(m)
16:
if mod Th
0 then
Th- Tgh
18:
end if
19:
-
20:
end for
Output: Final policy Max
A.2 MPO-RT
For MPO-RT, we directly incorporate the KI. regularization into the reward function, resulting in hard constraint on the policy updates. Specifically, the reward RG is transformed as follows:
R, Plyi y2 x) (log x) - llog Trer(y1 x)))
and the optimization objective becomes:
max MD (0)
max Ex By [Age (x, N1.
DKL. (To Tgx)
17
The key difference between MPO and MPO-RT lies how they handle the KL regularization. MPO soft constraint KL divergence via KL loss, while MPO-RT applies hard constraint by directly modifying the reward function, which aligns well with the standard RLHF methods (Orabona 2019
Ouyang al. 2022 Zheng et al. 2023). The choice between these algorithms depends
16

---

Published as a conference paper at ICLR 2025
the specific task requirements. The pseudocode of MPO-RT is provided in Algorithm 2 provide additional results for MPO-RT D.2
We also
Algorithm 2 MPO-RT
Input: Initial policy тe. preference model Ps. dataset D prompts, regularization temperature a, learning rate 7, update interval Tk. max iterations K Initialize: Reference policy Ter + To. k < 0, T -
1: for k = 1, ................................ K do
2: Sample batch D,, {x from D
3: for 1, ................................ .,.............................. N do
4: Sample responses y1 ~ To( (00(1xx)) ~ Tor (11x;)
5: for t l..... do
6:
Compute preference RG Ply1 > V2 x,) - allog (e x.. log गer (y1 x,))
Estimate advantage A, E / Rs end for end for
7:
8:
9:
10: Update policy The by perform m SGD steps on
g (0)
8¹
12:
for = 1,..., 111 do
0 (+1) - ((i) VDDD (0,0))))
13:
14:
end for
15:
gh+11
16:
if' mod
Th = 0 then
Th0-
Tat.
18:
end if
7
T4
20:
end for
Output: Final policy TgK
B DETAILS FOR SAFETY ALIGNMENT EXPERIMENTS
In this section, we provide details of our safety alignment experiments. These experiments are conducted on an 8xA800-40GB GPU server.
B. PREFERENCE MODEL TRAINING DETAILS
We train our preference model based the official codebase of (Dong 2024). The preference
model is initialized with Gemma-2B-II (Team et al. 2024). To train preference model that avoids overfitting to specific tasks and is suitable for ceneral-purpoo on open-source preference 5 which is a mixture of widely-used preference dataset. The hyper-parameter used during training are listed in Table 4
use, we train our preference model
We also evaluate the trained preference model Gemma- 2B-MM on RewardBench 2024). The RewardBench scores are presented in Table
Lambert et al.
Model
Chat
Chat Hard
Safety
Reasoning
Gemma-2B-PM
95.0
42.3
81.4
81.2
Table 5: RewardBench scores for Gemma-2B-PM.
http::/wwnnnnnntccccoeeeeeeeeeea mixturc2_and _aade_aae_nddddadt
17

---

Published as a conference paper at ICLR 2025
Hyper-parameters
Gemma-2B-PM
Llama-3-8B-PM
num _en_ee
1
1
Warmup_steps
40
40
sequence_len
3072
3072
gradient_checkpointing
false
true
gradient gradient_accumuune accumulation steps
8
micro bat__a_tth_tieee size
1
Ir s_sheduuler
cosine
cosine
learning _ate
le-5
5e-6
weight_ decay
0.0
0.0
Lok__rad_norm _nrm sample_packing pad sequence len
1.0
1.0
true
true
true
true
flash attention optimizer
true
true
tottttthcchchcdddddt fused
adam_tored tas__tttcc__feee_eetttee_ fused
Table 4: Hyper-parameters of preference model training.
B.2 SUPERVISED FINE-TUNING DETAILS.
We Gemma-2B Team al. 2024 pretrained model and perform supervised fine-tuning (SFT) based on the official codebase of Safe-RLHF" (Dai et al. (202)). For dataset, we use open-source
Alpaca TTori et 2023) dataset. The hyper-parameters used during SFT training process are presented in Table
Hyper-parameters epochs
Gemma-2B-SFT
Llama-3-8B-SFT
length
512
2048
per per_device_traid ttt_trn_atthhssiieee
16
8
mnndient_checkpointiigg
true
true
seaffent_aacumulatiinn
micro_batch size
Ir tt_scheduler_tye
cosine
cosine
learning rate
2e-5
2e-5
Ir warmup_ ratio weight_ _deca max_ max_ggradnoovl _orm flash attention zero zero_stage
0.03
0.03
0.0
0.0
1.0
1.0
true
truc
2
2
offload
none
none
Table 6: Hyper-parameters of supervised fine-tuning.
B.3 SELF-PLAY TRAINING DETAILS.
We implement our proposed algorithms based on the official Safe-RLHF codebase (Dai al. 2023
and fine-tune the SFT model with prompts sourced from the PKU-SafeRIHPP dataset (Ji et al.
2024
and the HH-Harmless subset of the Anthropic Helpful and Harmless dialogue dataset (HH) (Bai et 2022). These prompts are evenly split across three rounds of self-play. The hyper-parameters used for training on both datasets are detailed in Table 7
B.4 EVALUATION DETAILS
For the cost model based evaluation, employ the publicly available beaver-7b-vl 7b--c-l-vvloostttt mode released by (Dai et al. 2023). Note that this model is not included in the training process. For GPT-40
1.-coott
18

---

Published as a conference paper at ICLR 2025
Hyper-parameters
PKU-SafeRLHF
HH-Harmless
Prompt-Collectionvv..
sp_e ip_epochs max_ mea__ennghh
3
3
4
512
1024
1024
0.2
0.2
0.3
m
4
4
4
0.95
0.95
1.0
per por_device_traee batch size
4
4
4
peadient_checkpointiee
true
true 2
true
serffent_accumulatinn
2
2
miero _atcc__athh___ size
I
Ir r__shhdduler__ype
cosine
cosine
cosine
learning_rate rate
5e-6
5e-6
5e-7
Ir warmup ratio weight decay max faa__grad_non norm flash attention zero_stage offload
0.03
0,03
0,03
0.01
0.01
0,01
1.0
1.0
1.0
true
true
true
2
3
2
none
optimizer
none
Table 7: Hlyer-paraameter of self-play.
based evaluation, we utilize the official evaluation problem and prompts in the official codebase by (Dai et al. 2023). We also provide specific evaluation prompts used for GPT-4o.
You are a helpful, harmless, and precise assistant for checking the quality of the answer. [Question] {
(The Start of Assistant I's Answer] (answerl)
[The End of Assistant I's Answer]
[The Start of Assistant 2's Answer] [
[The End of Assistant 2's Answer]
[System]
We would like to request your feedback the performance of two AI assistants in response to the user question displayed above.
Please rate the harmlessness, helpfulness, and level of detail of their responses. Please prioritize the harmlessness. Please focus on whether there discrimination and prejudice in the reply, whether it abides by the law, whether avoids pornography and insults, whether avoids porn content, and whether it protects privacy. Each assistant receives an overall score on scale of to 10, where higher score indicates better overall performance. Please first output single line containing only two values indicating the scores for Assistant 1 and 2, respectively. The two scores are separated by space. In the subsequent line, please provide comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment.
19

---

Published as a conference paper at ICLR 2025
C DETAILS FOR GENERAL CAPABILITY ALIGNMENT
In this section, we provide details of our general capability alignment experiments. These experiments conducted on a 8xA800-80GB GPU server.
C.I PREFERENCE MODEL. TRAINING DETAILS
We initialize our preference model with Llama-3-88 3-8B--Insrrut We use the same codebase train Gemma-2B-PMN To train good preference model, we use an open-source preference datase which is mixture of widely-used preference dataset. The myper-parametery used during training listed in Table 4
We also evaluate the trained preference model Llama-3-8B-PP RewardBench. The RewardBench scores are presented in Table The performance of this preference model significantly better than that of Gemma-2B-PM.
Model
Chat
Chat Hard
Safety
Reasoning
Llama-3-8B-PP
97.6
66.7
90.4
93.3
Table RewardBench scores for Llama-3-8B-PM.
C.2 SUPERVISED FINE-TUNING DETAILS.
We use Llama-3-8h Team et al. 2024) as pretrained model and perform supervised fine-tuning (SFT) based the official codebase of Rate-RLHF"T Dai al. 2023). For dataset, open-source
SFT-OpenHerrmes-2.55 2.5-tandda dataset. The hyper-parameters used during SFT training process
presented Table
C.3 SELF-PLAY TRAINING DETAILS.
We fine-tune the SFT model with 30K prompts selected from open-source prompt-collection-vo. dataset. These prompts are evenly split across four rounds of self-play. The for training the dataset detailed in Table 7
used
D ADDITIONAL RESULTS
In this section, we provide additional results for our proposed algorithms.
D. ADDITION RESULTS FOR MPO
To demonstrate the effectiveness of MPO, perform comparative analysis with two baseline methods for fine-tuning LLMs: PPO and Iterative DPO (Dong et al. 2024).
To ensure fair comparison, employ the same SFT model and datasets outlined in Appendix C Specifically, the datasets used for training the preference model are also utilized to train the reward model, referred Llama-3-888 3-BB-RM. The RewardBench Table 9 This reward model is then used for PPO and Iterative DPO training. For PPO, we conduct experiments based on the official codebase of Safe RLHF (Dai et al., 2023). We follow the official default hyper-pann parameters for PPO, with few adjustments to align the settings with those of MPO. Specifically, we set the actor learning rate 5e-7, the max length 1024, the batch size 64, the coefficient to and the critic learning rate to 9e-6. For Iterative DPO, we use the implementation
for Llama-3-8B-E
is provided in
"htps:/huuggingfacecc///// http://hhhgggga..coocccce/a/aala "httss//huggingfacccoo////
12 "htpp://hngginggaacc.coo///assss wttpss//uuggingfac.ccm
20

---

Published as a conference paper at ICLR 2025
provided in OpenRLHF (Hu et al. 2024). The default hyper-parametter used, with the max length adjusted to 1024 to match the MPO setup.
PPO
leraiiv DPO oC-4)
42
DPO (Bes-oo6))
Midiival Hard
40
158
-66
Figure 7: Baseline Comparison of MPO, PPO, and Iterative DPO on MixEval-Hard Benchmark
The evaluation results are presented in the Figure 7 From the results, we observe that MPO significantly outperforms PPO, Iterative DPO (Best-of-8), and Iterative DPO (Best-of-4). Additionally, Iterative DPO generally performs better than PPO, which can be attributed to the complementary effects of on-policy sampling and the negative gradient, detailed in (Tajwar al., 2024). We also find that the performance of Iterative DPO largely depends on the Best-of-N sampling strategy, where larger N leads improved results but comes the cost of increased computational overhead (Sessa et al. 2024). These evaluation results demonstrate that MPO delivers strong performance compared PPO and Iterative DPO.
Model
Chat
Chat Hard
Safety
Reasoning
Llama-3-8B-RM
98.9
66.1
88.5
91.7
Table 9: RewardBench scores for Llama-3-8B-RMM.
We also provide detailed MixEval evaluation results for MPO in Figure 9 Across the four rounds of self-play, several benchmarks saw improvements while others experienced declines. Notably, PIQA, BoolIQ, and OpenBookQA demonstrated consistent gains across iterations, indicating enhanced reasoning and factual recall abilities. Commonsense(Q and ARC also exhibited stable performance. In contrast, WinoGrande showed fluctuating performance with drop in Iter.4, and GPQA demon- strated volatility with sharp decrease in Iter.3 followed by partial recovery. This suggests that while general reasoning abilities improved, there are still challenges in areas requiring specific contextual understanding nuanced judgment. This aligns with the results of MixEval Hard.
Additionally, we conduct experiments with Gemma-22 -a-aaa--BFFF on general capability using the same setup, as detailed in Appendix C.3 The MixEval-Hard evaluation results are presented in Figure [T] From this figure, we can observe that the model's capability improves consistently over the four rounds self-play, aligning with the experimental results of Llama-3. This trend demonstrates the effectiveness of self-play in enhancing the model's general capabilities across multiple evaluation categories. We also provide detailed the detailed evaluation results in Figure 10
D.2 ADDITIONAL RESULTS FOR MPO-RT
Following the same experimental setup for safety alignment, we train the Gemma-2B-S0 model over three rounds self-play. The hyperparameters are aligned with those used in MPO, with the exception of a, which is set 0.02 due the application of KL regularization directly the reward. The evaluation results of the cost model are presented Table I0 From the table, we can observe that while the model's 's performance improves significantly after three rounds of self-play, the magnitude of improvement is notably smaller than that of MPO, with only a slight advantage over the non-self-play baseline.
To further understand the performance MPO-RT, we also conduct experiments with Gemma-22 SFT general capability using the same setup, as detailed in Appendix C.3 The evaluation results
21

---

Published as a conference paper at ICLR 2025
Llana-3-8B-SP MPO Iter.
MPO Iter
Iter.3
MPO Iter4
Score (%)
DROP
MUP
AGR
MATH
GaMe
MMLU
Commommett
TricseGA
Bloid
PRC
SKGA
Heilsfoo
Openlbok
ARC
orgA
Figure 8: Detailed MixEval- Hard evaluation results across tasks for four self-play iterations of MPO.
Llama-3-8B-SFF MPO Iter.
MPO Iter.2
Iter.3
Iter.4
Score (%)
Tmmia4
INGW
OEMBE
MATII
BRI
AGleadl
MUE
MO1
Basli
HellsSon
ARC
CommonatioAAA
SYCA
Opelbel
UPEA
hnpp
Figure 9: MixEval general ability evalution results.
22

---

Published as a conference paper at ICLR 2025
Gemma-2B-SF7
MPO Itenl
50
MPO Iter2
MPO Iter3
MPO Tten.4
Score (%)
DROP
MOPP
Aribval
MATH
BHI
MMME
Commossamt
TrivictA
Bosk?
SKO4
Hollafoog
Opelhest
ARC
OP04
Figure 10: MixEval Hard general ability evalution results for MPO with Gemma-2B-SFT
on MixEval Hard shown in Figure From these results, observe that the performance gains from each round of self-play in MPO-RT are significantly lower than those in MPO, especially in the final two rounds. This result is consistent with the results from the safety alignment evaluation. Additionally, we provide detailed MixEval Hard results for each category, presented in Figure [2 These results indicate that, although theoretically equivalent, directly employ KI. regularization on rewards performs much worse than using a KL loss.
28
Models/Datasets
PKU-SafeRLHF
MPO
SFT
6.37
MixEoal Hard (%)
26
MPO-RT
MPO Iter. I
-3.76
24
MPO Iter.2
-8.96
MPO Iter.3
-11.38
20
MPO-RT Iter:1
4.36
18
MPO-RT Iter:2
0.83
MPO-RT Iter.3
-4.81
16
SET
ltc.11
Her.
her.3
Iter
MPO wo. SP
-4.18
Figure 11: MixEval Hard overall scores for MPO and MPO-RT.
Table 10: Cost model evaluation results.
E PROOFS
E ADDITIONAL LEMMAS
Lemma F.1 (Proposition 1, Munos al. 2023 ). for the game J(m1, #2) defined in (2) and कं
There exists a unique Nash equilibrium (n n
Proof. Since we have that P(r > π) = - P(r > x'). The minimax game can be repressed as symmetric two-player game with payoffs of policy n and क2 are defined
R(T; P( 7/) aD (Tllii//TTeee (T|())et),
and
R()'s)) P( > क) - a aDL (<' ((at)
We first prove the existence the Nash equilibrium. Since the payoff the game is concave with respect and n', thus the game processes Nash equilibrium (Rosen) 1965).
23

---

Published as a conference paper at ICLR 2025
Gemma-2B-SF7
MPO-RT Iter.
MPO-RT Iter2
50
MPO-RT Iter.3
MPO-RT Tter4
Score (%)
20
DROP
MOPP
Aribva
MATI
0310
BBI
MIMI
Commosumet
TrivactA
Bosk?
SKOt
Hollsfoog
Op
ARC
OPO4
Figure 12: MixEval Hard general ability evalution results for MPO-RT.
For uniqueness, need to show that the corresponding variational incquality is strictly mono- tone (Rosen 1965). Let [ x' and u()) [V, ((;(()=))) R(m'' For every Nash
equilibrium of the game satisfy
( オ) <0 0, II.
The variational inequality is strictly monotone if and only for every मं and T2, we have
(u(*)) - v(T2)) (m T2) 0,
with equality only holds at #2 (ooenn 1965 We can show this inequality holds by expanding the terms on LHS. For every context denote u (x) as the partial derivative for x. We have:
(x)) p(x)[P(y > x' x) a log(m/maa |x)
P(y > x|x)
log( [og("'/met x) 1].
Combining this with Eq E. and then exploiting the non-negativity of KL-divergence implies:
(u(#1) - v(52)) (5) m2) = P(m > و|) + P( > P(m P(Tx m2) -P +1 P ग2) + P( T2) + P(T +
> π
- -T + ( + D + D (~)
= -a((DK(( + (KI(m2/mm) m) + (KI( + (Ti( & 0.
with equality only at # 52- This completes the proof.
Lemma E.2 (Lemma D.3, (Sokota et al. 2022)). Under the assumptions of Theorem 3.2 for all π E II.
we have
(T n k+1
Bx(=:=*))
k+1 =)) (x(x
+ (nF((tt) voVg(T*+))) *22).
Proof. Note that
n k+1 E arg min (,, T) + nag(T) (TTTT)
xell
have
24

---

Published as a conference paper at ICLR 2025
0 & " x -
na(x(=) g(T + (T;T)) - (T
tt ;
); - - ((( g(***+1) + 1/(x)) ((( - [Vrb(="),, -m
) - na (g((+t+) π nbtz (Vu((*tt)) Ve(TT)) π x xbtz2
n ++1 <Vp(T - - Ve((T), t xht2)
=
= (a
> > B (,; - By (TT; (T; B B (
The first inequality relies the fact that n *+1 achieves the minimum value of the objective function. The second inequality results from the first-order optimality condition. Finally, the last equality derived from either the non-Euclidean prox theorem or the three-point property, as detailed in
Bauschke et al. (2003). Beck (0117) Tseng (0088)
Lemma E.3 (Lemma D.4, (Sokota et al. 2022)) ). Under the assumptions of Theorem 3.2 let पं be
the solution to VI(II, F + a g)) then, for Vn E 11 o int domb, the following equality holds
+ -. T) <
Proof. We have
(T) + na
- = + -
-
+ + +v
= (NF(TT - -F(F;),, -) + t |.( 17
+ + +(FF(=))+ +
< n)(V)(()) ,. -T)
< -na(Vo(( Ve((T,,, nr-)
( x + (T)))))
The first inequality follows from the monotonicity of the function F and the definition of the solution π.. The second inequality holds because is 1-strongly convex relative to 1 and the final equality is derived from the same references in Lemma E.2
I
Lemma E.4. Under the assumptions of Theorem 3.2 let म. be the Nash equilibrium of the regularized game and x² be the Nash equilibrium of the original game. Then the following inequality holds:
Σ Vn.gi T
t. m, O.
Proof. Since π² is the Nash equilibrium of the regularized game, from the first-order optimality condition for TH. we have for all π E IT:
V.. T2 V.. - =²
n, & O.
iE
Taking & as * I and rearranging the inequality, we obtain
V #m), m
Kc Se
Si a mt)
IE
ieI
1
Σ
म. at)
Ne
where the second inequality holds because the game is monotonous. Since n* is the Nash equilibrium of the original game, the first-order optimality condition implies that for all π II,
1 Σ VV. at) + 50, VT e II.
Q
25

---

Published as a conference paper at ICLR 2025
Then,
v
α2 i
This concludes the proof.
E.2
PROOF THEOREM 3.2
Proof.
Bo(
++1) ) Bo B+ ; + (m]((=*) naVglT + ), - x
-B((ffi²)) - (x(
k+1 ;-=² + (1)(((T*)) - 1/ ( ), xhtl> + (1)[(( x+1) + naV *+ ), )
-r - nhtl aa Bu (T+ ; + ( x & &Bo(kfim") - B ( x²) (n)((*t) - nF
Bp((kia*) B ++1 1L || x² k+l
प.
b+l
1a ; B (TT
& Bo( - By BT( +1 x²) n k+1 L²
h+1
na (B (1 + Bu x x(t))
2
- Bu ++ ; no(BB(n ; + (
Ba(=fia")) -no (m= ; **1).
The first inequality follows from Lemma E.2 the second inequality from Lemma E.3 the third inequality by the Cauchy-Schwarz inequality and the smoothness of F. The fourth inequality is derived from an elementary inequality, and the last inequality follows from the strong convexity of and By. Finally, we obtain
(T xht))
& + na
(x(
By iteration, obtain the result in Theorem 3.2
E.3 PROOF LEMMA 3.3
Proof. To prove this lemma, we first show that n. m,++1 * x.." for I > 1, we have
DKL (nt DKL (I (l)
Consider the KL. divergence between consecutive iterates. By definition:
DKI(TT (
Σ
JEJ
For any Nash equilibrium x IT can write:
26

---

Published as a conference paper at ICLR 2025
V ViDKL (
Fri
#,++1
mi?
Tig
rij
s,ntl)
4,8+1
Frij
IEI jEJ
Trij
Texp (In (7
#
+
rij
IeT
Prij
> eexp & +
rij
+,n+1
iEI
e (D (1 || +
DKL (T (+)))) I,
where the first equality follows from the gradient of KL divergence, the third inequality follows from Jensen's 's inequality since in(-) is concave, and the last equality is by the definition of KL divergence.
Since o. n# # o.. can rearrange get:
DKL.( (==" DRI. (m,") [1 +
1 Σ
v. V.D. 4.3
(T
#,n+1 Im"""""""""""""""""""""""""""""" miittt))
7
TE
(V.DLL (T "
Tr2 mnnt1 -x,
where the the second inequality uses the fact that (x + 1) & x for -.
Now, we have that
DKL( [ m.
DKL ( () &
VKL ( 4.n |
s.a+1), 8,n+1 Rri
Since * is the Nash equilibrium of the original game, म, is the Nash equilibrium of the regularized game, and n. ++n+1 + = according Lemma E.4 we have:
DEL (| DTI( I "
V.DKL
* 1) "
*,+t1
x² < 0.
xEI
Finally, let arg +ill+ DKL. (T+|| (.). If .. IIIII* then:
min DKI( (|=== DEL. (* || 1 *
DKL( म.
min DKI(T (171)
**EII*
This shows that the sequence of regularized Nash equilibria strictly approaches the of Nash equilib- rium of the original game, completing the proof.
E.4 PROOF OF THEOREM 3.4
Proof. We will prove that If Lemma 3.3 holds, then the sequence { Jn31 converges to x* II* as n 00 where π is the Nash equilibrium of the original game.
Let us begin by noting that since min (RL((T** (..")) is bounded below by and according to Lemma 3.3 this sequence is strictly decreasing unless ..."" e II", we can conclude that the sequence converges some value > We will prove by contradiction that b 0.
Suppose 5 > 0. Let us define:
e min DKL (x (m
Do (T, III b & min D (T)) & cj. **ell+
Since min,*enT DKL ( () decreases monotonically according to Lemma 3.3 we have π ,... Dg,e for all > 1.
First, we prove that Sr.e is a compact set. Note that min (TI.(T')) [|-) continuous function on II. and [b is closed set. Therefore, Sa,co being the preimage of [b, c] under continuous
27

---

Published as a conference paper at ICLR 2025
function, is closed. Furthermore, since II is bounded by assumption, Ro is also bounded. Thus, Rb.c compact as is both closed and bounded.
Let F(T.): II II be the function that maps πy ... As proved in Lemma F.7 of Abe al. (2024). Fis a continuous function our case. When F is continuous, min DxL π F(m,)) #eIT* DKL ( (x,) is also continuous.
Since So,c is compact and this difference is continuous, there exists a maximum value:
max min DRI. (xI.(T=1[(FF)) [F(TT))
min ( [T,))- **
T.elb **
From Lemma 3.3 we know that O. Therefore:
min DKL( 1x (,,, *
min *#ell*
DKL(T (==,,"
Σ
min (
min **ell*
*ello
& + Nv.
This implies that there exists N large enough such that e No 0, contradicting assumption that b > 0. Therefore, we must have b = 0 and this proves that the sequence { Jaal " converges x 11 completing our proof.
E.5
PROOF OF THEOREM 3.5
Proof. According to the definition of * can directly derive the following equivalence
π k+1 arg min ((F + Tret)) + (x x²)
mell
<>n
k+1 arg (nn(n)((77)) ma V ((Tre)) Ve(T*), T) + (1 + na)((T)
nell
F((*)) na VulTtet Veb(T*)) TT) ((T))
kt arg min
well
nja
FF (T(T na ( na Vol(m*+) Ve((a"), , T) + (Tx)
60 arg min well
1 qa
k+1 arg (in(7(( ((( + (V - a (V((Te))) - VV((T*), ④) ((*)
well
k+1 arg (7i (F(**) + aV By (T : VV((T)), + (TT)
well
4
k+1
arg
arg min Ff + V BBs ;rot), + Bx(T;T²
sell
This completes the proof.
E.6 PROOF OF LEMMA 3.6
Proof. Following the proof of Theorem
3.2, we have
( k+1) & DKI((T [ - njcr DKL. (Tr [
for any le (TTR, 7Ty + 1, TTk + 2,
(k 1)TT - 1).
1 x-
|| & (T |
nja
Taking & = (T 1)T - 1, we have
DKL (x +l & DKL(
Tk.
+ na
28

---

Published as a conference paper at ICLR 2025
Since T. T complete proof.
E.7 PROOF OF THEOREM 3.7
Proof. From Lemma 3.6 we have when T3 00, ( 7 +1 0. Since DeL, (T. |. +11 is
never negative, we have DKr. (n,² -+; 0. Then, for any E ( 7Te + 7 + 2, 1)Tk - 1), we obtain our results follows the proof of Theorem 3.4
F DUALITY GAP AND CONVERGENCE RATE
In this section, provide addition theorem for duality gap and show that Theorem 3.2 can be leveraged to guarantee linear convergence of the gap.
Theorem F.1 (Proposition D.8, (Sokota et al., 2022)). Assume that is twice continuously differen- tiable over int dom v. II is bounded, and the assumptions in Theorem 3.2 hold. Let G F For all k 2 1, the duality gap is bounded as
<x k1)) (up((((*)))
x) =0(( 1 na
)²)
nell
h U ( is eventually within closed ball centered at " So there exists K" and a closed ball B such th JaEK U {=*] S int dom th. Since B is compact and V is continuous over B, we have that V2g(z) is bounded on B. Therefore, there exists LB such that Vg(=') - Vg(TT)" & Lg(T - x'|l for any n, a e B. We have that for any N, n B. [G() G( & L||T - r'll for L = + aT.B.
Proof. From Theorem 3.2
have that {nth Jaet
For any π E II, we have
(G(#*), 1 x+1 T) (G(+)) n h+1 σ) (G(x*t1) G(T*), m n n) (G(r"), ), n x) (G( x k+1 x²) ?++ <G( - G(T*), ktl R)
I|ll| h+ x+1 *1 + fe|l - x + - π
(IG(T") )11+ + LD) || x*
& C By(
< C 1 1jo
v (T)))t
where such that max
T'|l D and C |G([*) + ŻD.
The first inequality is by the generalized Stuchy-Schwarz inequality and the Lipschitz property of G. The second inequality is by boundness of II. The third inequality is by the fact that B x/) > 2| 11
Lemma F22 Let #..9 )nsl be the sequence of NEs of the regularized games, and n. be the approximation of n. solved via the update rule of (4). Under the assumptions of Theorem 3.2 for any n > 1, if n. n.n 11 11 we have the following inequality:
B : m ") Bp(x=:=²) - (,(x;"";;,)
Proof. By the definition of Bregman divergence, we have:
By(n*tm Bo( + B ( m
[vo(T=)) VV(T)), -+))
29

---

Published as a conference paper at ICLR 2025
Since n. is the Nash equilibrium of the n-th regularized game, by the first-order optimality condition, we have:
( a. (a m.).
" E 0, VT II.
VEZ
Taking x², obtain:
1
Σ
Vv, Ba
sn イ,
4.1
+n x?)
n.
KEI
i27
1
-,,)
where the second inequality holds because the game is monotonous. Since n is the Nash equilibrium of the original game, the first-order optimality condition implies that for all π II,
1
Σ Vi, iEI
=0)=0, E0 e IT.
Then, taking mi.. we have:
V TEI
n. (T a.n TT)) s.
-m,)=
- x
IEI
<
(
0
ie
Thus, we have:
B - (xx :T)) + Ba( =
Σ
#. = -
60
IEI
This completes the proof.
Lemma E33 Under the assumptions of Lemmal F22
the duality gap
gap for क.. is bounded as:
S <(m="")) + O I1.
Proof. By the definition of duality gap, we have
c(x,)) max x.ell
fo (TT
n. (m
4.n
*n
mi) + max
fi(m TI 2. m
x.ell
welll
IEI
max
<
Fi ri
2. -,>>>
w,ell
ieI
max xiell
nz
fa(m
4.m
* V fi TIn), Ti)
iez
< <((T!".)
+m)
=(T +
L|| - TT
SET
where the second inequlity follows from the Lipschitz continuity of the gradient with constant This completes the proof.
30

---

Published as a conference paper at ICLR 2025
Theorem E4. Under the assumptions of Lemma F22 v1 < n & N, we have
suppose is Lm-smooth. Then, for any N > 1,
<(T,"'1
0
Proof. From Lemma
F3
wc have
( = c(TT,:1)) + Lo 2nn
n +1+1
where Lo is game-dependant constant.
Following the bounding technique for the gap function with tangent residuals (Cai et al. 2022) and the first-order optimality condition for n.t."" we have
pen (T ()
L1 c - T.7
where plan (m, denotes the tangent residual (Cai et al. depends on the original game. From Lemma of (Cai et al. 2022). for VT e II, we obtain
2022) of x and I1 is a constant that
c( #. & Lyrtan tan (mr""""(m"""))
where La a game-dependent constant. According to Theorem
3.2 let vb 1 12, we have
<(
& L1 m. 2. T. LO
n.
+++11 ||.
n. s.n - Lo m
4
Ne we have
where is arbitrary constant. According to Lemma
F22
Ba(=": ") < By (,] ( M." ; x
(T x. (T * πT)) By(T : m #.))
B (TT ; (TT " TF) By( ((("'==]])) प + (x (1 :m
(vi(TT - VilTm ),
),, n n.. +
where the last line comes from the three-point property of Bregman divergence. For (Vr((**.)) V Vib(m=-1) ), n x.+1>, we have
(VD(T**)) Vu((T," ),, ), x" n Vú((+"))
V V
|||
V +1
1
m.
+1
2
2Noc
n R.n
+
1 - ... 112
2N
inequality, the second inequality follows from ab & pa? // + b2/2p, Vp O. and the third inequality follows the smoothness of (-).. Thus, we have
where the the first inequality follows from the
0 < (T+ ;,,5) - (T, ;
m/)) - B +1 - (1
(Vb(T**")) V
7+1 ), xx+11)
Bo(==ii7)) - (, - -B( m
; x++) B (
1 - + + 1
2N
31

---

Published as a conference paper at ICLR 2025
Further, we obtain " (,)) & (TT ;m,) - (T ;
: - B +- (
|Tp""""
| -.. 1
2
2N
| TT 1
< B " x
B&s BT ; x +
2Nc
Then, summing up from n 1 to N, we have
N
N
| n 11
Σ
B " m & Ba ( - BO (,; +1
2
Thus, we obtain
Σ (T & Cs.
where C3 is game-dependen, constant, and this inequality comes from the derivation of (Cevher al. )023). Since is an arbitrary constant, we then have
Bg 7 8. m
(T t x x-1 + B f. ) प.
(V+(T= - rb [,, n-11 ), = 211- n
$ Bo प.. 1 + BB n.,
s.n ;; ) + |, 4
8,
8
T.y 2 n
& BTTT, म. 1 + B (Tf 2- + | "
+.n-11
TT,
+ (p m.) + | 2, || | x x-1
< Bel -m--1)=-)))))-0 2B ( i m 1) + *
1
-Tp
+ |, n. =-== ,
o. --
& BB( 1 -1 + ,. n
113 1
2C π x
/.-- -
1
2Ca
& (T | 1
| m
n 112
|m
N4
N2
C2 2C²
& BT - -r
N
N
where C₁ = max | - -11 Then, we have
C²
202
(T " mi) By (T
; - +
No N
T-2) 2) + 203
+ 403
Ba(T " ,,,)) & (T 2,;,,,,7777
NT
N
Therefore, we have
N
C2N2
20
NB (T 6 m
Σ By B.
,,
N44
K2
C²
NB m.m. : M
C3
2C
N2
1
+ |, -. | ||
v 0
vN
Therefore,
||
( & L2 n
., + Lo
$ 0
N4
N
This completes the proof.
32


Reviews:
[{"review_id": "laOhGl1Yrw", "reviewer": "Reviewer_8BmD", "summary": "In this paper the authors study the problem of Reinforcement Learning from Human Feedback. As it was shown in bibliography this problem can be seen as a two-player constant sum game with Nash equilibrium as a solution. In this paper the authors study the last-iterate convergence to a Nash equilibrium in this context giving new dynamics for this. They show last-iterate convergence and give experiments that show evidence that there is improvement of the problem with their algorithm.", "questions": "In 145 line it is stated that the constant sum game has unique and symmetric Nash equilibrium in this context (which is not true for any constant-sum game in general). Do similar results hold without having the uniqueness assumption? In what exact point in the proofs the uniqueness of the Nash equilibrium is needed?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "I think that it is nice to see how different areas of research are combined and results of the one area can be used to the other. In this direction the authors study/prove the last-iterate convergence to a Nash equilibrium of a game in their context, giving new techniques and ideas. Furthermore, this paper shows another example of the importance of last-iterate convergence to Nash equilibria in constant sum games.", "weaknesses": "I think the presentation of paper has space of improvement, especially in the technical/mathematical part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QYhEwTJEKf", "reviewer": "Reviewer_wd8o", "summary": "The paper presents the Magnetic Mirror Descent Self-Play Preference Optimization (MDSPO) framework, a novel approach in the field of Reinforcement Learning from Human Feedback (RLHF). It addresses the limitations of the Bradley-Terry model by reframing preference learning as a two-player constant-sum game and aims to converge to the Nash equilibrium (NE) of this game. MDSPO introduces a magnetic term to achieve linear convergence to the NE of the regularized game and provides theoretical guarantees for last-iterate convergence to the NE of the original game. The empirical results demonstrate significant improvements compared to the SFT method under the MixEval-Hard benchmark.", "questions": "Please see the Weakness part.\n\nTypo: It should be $y_{<t}$ instead of $ y_{<k}$ in line 116.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. By using the proposed magnetic mirror descent (MMD) method, this work achieves the last iteration convergence guarantee for the NE of the game under the general preference setting, which is better than previous works.\n2. The proposed MDSPO method achieves a great performance compared to the SFT method.", "weaknesses": "1. The motivation for introducing the magnetic term in MMD is currently unclear. It would be helpful to discuss the technique challenge to achieve last-iteration convergence when using mirror descent. After that, this work can show the reason why the MMD method can solve the above problem and achieve great performance, which would deepen the understanding of MMD.\n2. For the experiment part, this work uses the SFT method as the baseline. It would be better to add additional experiments such as the RLHF with PPO to do a more comprehensive comparsion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hRl8WvLjz9", "reviewer": "Reviewer_92Yr", "summary": "This paper studies Reinforcement Learning from Human Feedback methods by formulating the problem as finding the Nash Equilibrium of one regularized game. By introducing the Magnetic Mirror Descent approach, they achieve linear last-iterate convergence instead of average-iterate convergence in previous works. Consequently, they provide an algorithm with a theoretical guarantee for convergence, and provide two slightly different ways for implementation (MDSPO and MDSPO-RT) with theoretical insight.", "questions": "The questions are provided in the weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The extension from SPO to MDSPO algorithm is well-motivated, and the motivation presents well.\n2. The experiment is solid, and the performance is good.\n3. The idea of periodically updating the magnet policy is interesting, and the theoretical convergence result is provided.\n4. In addition to MDSPO, the authors also provide a slightly different algorithm MDSPO-RT, and provide empirical comparisons.", "weaknesses": "1. The empirical part:\n\n(1). The connection between theoretical contribution and the empirical implementation is not clear. More explanations about why MMD induces the eq (6) will enhance the paper. For example, the first term on the right side of eq (6) seems unrelated to the policy $pi$ and seems could be ignored, making this equation not similar to the MMD update rule.\n\n(2). In Line 331-333, the paper argues that the choice between MDSPO and MDSPO-RT depends on the requirements of tasks. However, it lacks further guidance on how to make this choice. Additionally, the empirical results only include a task where MDSPO-RT performs worse than MDSPO. Could the authors provide more insight into selecting between the two algorithms?\n\n(3). Since the paper is motivated by the problem of average-iterate convergence, the empirical observation about this phenomenon and how the MDSPO alleviates this problem would significantly strengthen the work. Additionally, empirical comparisons between SPO and MDSPO seem essential for a comprehensive evaluation.\n\n3. The theoretical part: \n\n(1). First, Theorem 3.2 almost follows the same derivation as Theorem 3.4 in [1]. It would be clearer to state explicitly that Theorem 3.2 is a modification of [1] and include the citation prior to presenting it.\n\n(2). The essential theoretical contribution is Lemma 3.3 and Theorem 3.4. However, the proof of these theorems could be polished. For example,  In line 976, what's the term $\\langle D_{KL}(\\pi_r^{\\*,n+1}|| \\pi_r^{\\*,n}), \\pi_r^{\\*,n+1}, \\pi^\\*\\rangle $ mean? I assume it's $\\langle D_{KL}(\\pi_r^{\\*,n+1}|| \\pi_r^{\\*,n}), \\pi_r^{\\*,n+1}-\\pi^\\*\\rangle.$ Also, in Line 992, it is unclear why the first-order optimality condition can be applied here since the first term $\\nabla D_{KL}(\\pi_r^{\\*,n+1}|| \\pi_r^{\\*,n})$ doesn't depend on $\\pi^\\*$. Moreover, in Line 998 and 999, the 3rd and 4th terms in the inequality should involve $\\pi_\\*$ instead of $\\pi^\\*$.\n\n(3). In footnote 4, the paper [2] suggests that one could apply optimistic mirror descent (OMD) to achieve a last-iterate guarantee. Why did the authors choose the MMD rather than the OMD? Are there some technical difficulties in directly applying OMD?\n\n\n\n\n\n[1]. Samuel Sokota, Ryan D’Orazio, J Zico Kolter, Nicolas Loizou, Marc Lanctot, Ioannis Mitliagkas, Noam Brown, and Christian Kroer. A unified approach to reinforcement learning, quantal response equilibria, and two-player zero-sum games.  \n\n[2]. Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. A minimaximalist approach to reinforcement learning from human feedback.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OzXLWyA1lJ", "reviewer": "Reviewer_D3mq", "summary": "This paper proposes mirror descent self-play preference optimization (MDSPO), which has a last-iteration convergence to the Nash equilibrium (NE) of the two-player constant-sum normal form game induced by a reinforcement learning from human feedback (RLHF) problem with non-transitive preference. In the regularized magnetic mirror descent setting, the policy converges to the regularized NE at a linear rate, which is faster than the result in previous work. By solving a series of regularized games sequentially, one can have a last-iterate convergence to the NE of the original game. The authors also give practical implementation for the theoretical algorithm and it shows an improvement in several benchmarks.", "questions": "1. The approach described in Section 3.2 seems hard to implement, since we cannot get the exact NE policy $\\pi_\\text{reg}^{*,n-1}$. What we have from MMD is an approximation of this NE. What will the convergence result be if the approximation error is taken into consideration?\n2. In the paper \"Nash Learning from Human Feedback\" by Munos et. al., the algorithm Nash-MD achieves last-iterate convergence (cf. their Theorem 1) to a regularized NE. Can their algorithm be adapted to the approach in your section Section 3.2 and achieve a last-iterate convergence to the original NE?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This work studies the Nash learning setting, which has broader significance than the Bradley-Terry preference setting as the preference could be arbitrarily non-transitive.\n2. The guarantee of last-iteration convergence to NE has practical benefits for large language models (LLMs) as the inference time and stability (low variance) is important.\n3. This paper presents the first theoretical result for last-iterate convergence to the NE of the original game, as previous result was for the regularized game.", "weaknesses": "1. The convergence rate to the original NE is not studied. The authors should either give an analysis of the convergence rate or explain why such an analysis may be challenging.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)