Improving LLM General Preference Alignment via Optimistic Online Mirror Descent
Authors: Yuheng Zhang
Dian Yu
Tao Ge
Linfeng Song
Zhichen Zeng
Haitao Mi
Nan Jiang
Dong Yu
Venue: NeurIPS 2025 spotlight ()
Keywords: LLM Alignment
RLHF
Primary area: deep_learning

Abstract: Reinforcement learning from human feedback (RLHF) has demonstrated remarkable effectiveness in aligning large language models (LLMs) with human preferences. Many existing alignment approaches rely on the Bradley-Terry (BT) model assumption, which assumes the existence of a ground-truth reward for each prompt-response pair. However, this assumption can be overly restrictive when modeling complex human preferences. In this paper, we drop the BT model assumption and study LLM alignment under general preferences, formulated as a two-player game. Drawing on theoretical insights from learning in games, we integrate optimistic online mirror descent into our alignment framework to approximate the Nash policy. Theoretically, we demonstrate that our approach achieves an $\mathcal{O}(T^{-1})$ bound on the duality gap, improving upon the previous $\mathcal{O}(T^{-1/2})$ result. Meanwhile, it enjoys a linear convergence rate in the last iterate, a property not achieved by previous methods. More importantly, we implement our method and show through experiments that it outperforms state-of-the-art RLHF algorithms across multiple representative benchmarks.

Improving LLM General Preference Alignment via Optimistic Online Mirror Descent
Yuheng Zhang
Dian Yu Tencent Al Lab
Tao Ge Tencent AI Lab
Linfeng Song Tencent Al Lab
Zhichen Zeng UICC
UITC
Haitao Mi
Nan Jiang UICC
Dong Yu Tencent AI Lab
Tencent Lab
Abstract
Reinforcement learning from human feedback (RLHF) has demonstrated remark- able effectiveness aligning large language models (LLMs) with human prefer- ences. Many existing aligument approaches rely on the Bradley-Terry (BT) model assumption, which assumes the existence of ground-truth reward for each prompt- response pair. However, this assumption can be overly restrictive when modeling complex human preferences. In this paper, we drop the BT model assumption and study LLM alignment under general preferences, formulated as two-player game. Drawing on theoretical insights from learning games, we integrate optimistic online mirror descent into our alignment framework to approximate the Nash pol- icy. Theoretically, we demonstrate that our approach achieves an O bound on the duality gap, improving upon the previous (T22) result. Mcanwhile, it enjoys a linear convergence rate in the last iterate, property not achieved by previous methods. More importantly, we implement our method and show through experiments that it outperforms tfte-off-the-an RLHF algorithms across multiple representative benchmarks.
Introduction and Related Works
Reinforcement learning from human feedback (RLHF) has played pivotal role in aligning large language models (LLMs) with human preferences. The goal of RLHF is to fine-tune LLMs to generate responses that preferred by humans. It has been successfully deployed in state-of-the-art models, including Instruct-GPT Ouyyng et al. [0022] and Claude [Bai et al. 2022b]. The first RLHF framework for LLMs was developed by Ouyang
20022] where after the pre-training stage, the LLM is fine-tuned to maximize the reward signal from a reward model using the proximal policy optimization (PPO) algorithm (Schulman al. 2017 This pipeline requires training both the reward model and the policy model. In addition, policy gradient approaches such as PPO often exhibit high variance and instability during training [Peng al. 2023 leading increased computational costs.
To develop more stable and computationally lightweight alignment approach, Rafailov et al. 2024bb propose the Direct Preference Optimization (DPO) algorithm, which directly trains the LLM a preference dataset and bypasses the need for a reward model. DPO uses an offline preference dataset, and since its development, line of research has explored different exploration strategies and proposed online direct preference alignment algorithms (Xiong et al.) (2024) Xie et al.] 2024 Dong al. 2024 Yuan al. 2024]. All these methods assume that human preferences can be modeled using the Bradley-Temi (BT) model, where a reward function R. exists such that, for any prompt
'Email: yuhengz281llinois edu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

and response pair (y' ./)) the preference between y' and y² satisfies:
P(y' > x² x) = (x, - R (z. v²),,
where (2) 1+exp(-2) is the sigmoid function.
1
However, the existence of reward function and the BT model are strong assumptions that can be overly restrictive when modeling complex human preferences. For example, the preference signals in the BT model are always transitive: if A is preferred to B and B is preferred to C. then A must always be preferred to C. This transitive property contradicts evidence from human decision -aainn [May] 1954 T 1969), especially when preferences are at the population level and aggregated from different human groups MMay] 1954 Ye et al. 2024). Furthermore, the limitations of the BT model have also been observed in RLHF practice. Jiang al. 2023) show that preference model with 0.4B parameters achieves performance comparable to 1iiiaa----3--bassee reward models. Ye et al. [2024) train BT reward model and a preference model separately using the same base model and preference dataset, and their results demonstrate that the preference model consistently outperforms the reward model on Heward-Bence [Lambert al. 2024] under different base models. These findings motivate us to drop the BT model assumption and instead consider general preferences.
In this work, we study the problem of aligning LLMs with general preferences and formulate it as a two-player zero-sum game. Our objective is to approximate the Nash policy of the game, which ensures a win of at least 50% against any other policy. As established in the game theory literature [Bai et al. 2020 Liu et al. 2021], self-play algorithms have proven to be highly effective in approximating Nash policies. Building on this, aim to propose a novel online RLHF algorithm that further leverages the self-play streture to enhance general preference alignment for LLMs. Our contributions are summarized as follows.
Contributions. We propose novel online general preference alignment algorithm, Optimistic Nash Policy Optimization (ONPO). Inspired by recent advancements in game theory, our algorithm integrates optimistic online mirror descent [Rakhlin and Sridharan] 2013 SSrgkanis et al. 2015 into the self-play framework. By utilizing a reward predictor in a two-step update strategy, ONPO more effectively leverages the self-play mechanism and achieves a faster convergence rate of OTT on the duality gap, improving upon the previous ((2-1/22 result. Moreover, ONPO enjoys linear convergence rate in the last iterate, a property not achieved by previous methods such as INPO Zhang et al. 2024
ONPO can be efficiently implemented by directly minimizing loss objective preference dataset, making it computationally lightweight in practice. We evaluate ONPO on several representative benchmarks, comparing it with state-of-the-art general preference alignment algorithms. Experimen- tal results demonstrate that ONPO consistently outperforms or achieves performance comparable to the baselines across different base models and benchmarks. Notably, on the AlpacaEval 2.0 benchmark [Li et al.] [2023a]. ONPO achieves 21.2% and 9.9% relative improvement over the strongest baseline when using Mistral-Instruct and Llama-3-8B as the base models, respectively.
2 Preliminary
Problem Setup. We study the contextual formulation which is extensively used in previous RLHF literature [Ratailov et al. 2024b. Xiong et al., 2024]. The prompt x E X' is sampled from an unknown prompt distribution dj. y is the response space and an LLM is characterized by a policy π: X + A())) which outputs the response probability given the context. For any policy n, we use E,, to denote the expectations under π.
General Preferences. In this work, we drop the BT model assumption [Bradley and Terry 1952 and focus on directly aligning LLMs with general preferences. To this end, we define a general preference oracle as follows:
Definition 1 (General Preference Oracle). There exists preference oracle IP Xxxy + y' 10,1] which can be queried to obtain the binary preference signal:
Ber > >y² xx),
where 1 indicates n² is preferred to y², and 0 indicates the opposite.

---

Unlike the BT model assumption, which assumes the existence of a reward function R. for each x and y, the general preference oracle always compares y' another y². This setup aligns with practical scenarios, where it is often easier for users to compare two responses than to assign an absolute score to single response. Since the preference signal always involves two responses, potentially come from two different policies, we formulate the LLM alignment problem as two-player zero-sum game, The objective of this game is the expected win rate between the two players:
( T2) = Ra Pyl">xin
[P(y' > y² [)]]
Here ni is the policy of the max-player, aiming maximize the objective, while is the policy of the min-player, aiming to minimize it.
Nash Policies and Duality Gap. which is defined as:
Our learning goal to find the Nash equilibrium of the game,
T1: 22 :- argmax argmin (TTI T22.
Due to the symmetric nature of the game, the Nash policies for both players are identical, i.e., Ti = _े = n², and the game value is J(x,, , )²) = 0.5. Since Nash policies are the best responses each other, for any policy т, we have [ π) 2 0.5, indicating that the Nash policy will not lose to any other policy. To quantify how well policy æ approximates x we define the duality gap as:
DualGap(#) == max ( x) min [ x2).
122
The duality gap is non-negative and (ualGap(T) = if and only i = TT" Hence, our goal is to find a policy that minimizes the duality gap. Once we achieve DualGap(rt (T) we say that is an c-approximate Nash policy.
3
Algorithm
In this section, we begin by briefly reviewing the self-play algorithm with online mirror descent (OMD) updates, which is used in previous general preference aligmment algorithm (Zhang et al. 2024] Next, we present our proposed algorithm, which leverages the faster convergence properties of optimistic OMD, inspired by advancements in game theory [Rakhlin and Sridharan, 2013 Syrgkanis et al. [015]. Through theoretical analysis, show that our approach achieves improved bound on the duality gap and linear convergence rate in the last iterate. Finally, we describe the implementation of our algorithm. Following Azar al. 2024 Zhang al. (2024) we omit the context æ throughout the rest of the paper since each context is independent.
3.1 Self-play Algorithm with OMD Update
Self-play algorithms are widely used in approximating the Nash policy [ai et al. 2020 Liu et al.
2021]. The key idea is let the policy play against itself, enabling iterative iorovemprovement. The algorithm is performed in an online manner, with each iteration using online mirror descent (OMD) to update the policy. Specifically, iteration we find the policy that maximizes the following objective:
Te+1 argmnax (T. re) 1 (L 17
(I)
x
where rv(y) Ppy mx) Byg.., Ply y' is the expected win rate of response y against the current policy ny, and 1j > 0 is the learning rate. This objective ensures that T441 not only aims to maximize the win rate over ne but also remains close to I., as measured by the KL divergence term. The stability introduced by the KL regularization is critical for achieving a sublinear regret bound. Without this regularization, one can construct examples where the algorithm suffers from linear regret, which is undesirable [Lattimore and Szepesvári 2020].
We can show that the uniform mixture of æ.. achieves an (TT-//2) duality gap, as stated in the following theorem. The proof is deferred to Appendix C.I
3

---

Theorem 1. Let D - max, (L(T]|TT and क unif(T1, , Self-play algorithm in Eq.
with 11 / / satisfiex:
DualGap(R) 4VD
In RLHF practice, we typically use a small number of iterations (e.g., T 3), so the uniform mixture policy can be directly deployed. Unlike UnlikeZZang et al. (2024), which adopts KL-regularized game formulation, we directly use the win rate between two policies as the game objective. This formulation has two advantages: 1. The Nash policy in our formulation guarantees at least a 50% win rate against any other policy, which aligns directly with the goal of general preference alignment. In contrast, the Nash policy in the KL-regularized game only ensures this when the KL terms are negligible. 2. In the Kl-rreuulrrized setting, the analysis of the OMD algorithm relies assumption that the log-density ratio log π (v)/ log (ye is uniformly bounded for all т, and the regret bound depends linearly the coverage coefficient. Our analysis avoids this assumption and its associated dependence by directly optimizing the win rate, resulting in improved results.
coverage
3.2 Optimistic Nash Policy Optimization
While self-play with OMD update already achieves an O(VT) regret bound, which is near-optimal in many online learning scenarios, there is still room for improvement by better leveraging the self play structure. Recent advancements in learning in games [Rakhlin and Sridharan, 2013 Syrgkanis et al.) 2015) demonstrate that faster convergence rate of (TT-1) can be achieved when both players adopt optimistic OMD update. In this subsection, we introduce how to integrate optimistic OMD into the self-play algorithm, resulting in algorithm called Optimistic Nash Policy Optimization (ONPO).
The key idea of optimistic OMD is to incorporate reward or loss predictor at each iteration. Recall that in OMD update, we use the expected win rate over the current policy Te as the reward vector Tt compute 1- While in optimistic OMD, the learner utilizes a reward predictor my and adopts two-step update strategy:
argmax (T, m²) (L((T|)(
7)
1
21+1 argiaax (T,
KLLLLT
Here ny aims to maximize the reward predictor m, and the auxiliary policy क(41 is updated after observing the actual reward The word "optimistic" comes from that the learner believes that the predictor m, provides a good approximation of the true reward rz.
Next, we describe how to apply optimistic OMD in our self-play algorithm. In both OMD and optimistic OMD, the KL. regularization term is consistently used to ensure that the next policy remains close to the previous policies. This regularization provides stability, making it reasonable to assume that the change from N. to TT+1 small. Based on this observation, we directly use the reward information from the previous iteration as the predictor, i.e., let m, Tt-1 Ey--ss- [P(y > s/)].
In the following theorem, demonstrate that ONPO achieves O(/TT) duality gap, improving over the previous O(I/VT) result.
Theorem 2. Let D max, (T(T)]] and म (n,, mT). ONPO algorithm with n) min vDJ satisfies:
DualGap(F) 4VD
Here, Ti = mT is the initialization policy. Theoretically, ग| can be set as a uniform policy, in which case D is bounded by log [/]. In RLHF practice, π is typically supervised fine-tuned policy.
The proof is provided Appendix C.2 The key to achieving the O1/T)) rate lies the regret bounded by variation in utilities (RVU) property of optimistic OMD. Specifically, the stability terms ||r, - are canceled out by the negative term - || - 1. which arises from the self- play mechanism where represents the win rate over ne. Additionally, the stability inherent in

---

optimistic OMD ensures that the learned policy remains close to the initial policy, This aligns with the motivation behind incorporating KL regularization into the game objective in prior works (Munos et al. 2023 Zhang et al., 2024). Since our update rule already implicitly enforces this stability, explicit regularization the game objective is unnecessary.
Although the uniform mixture policy is implementable, more common choice in RLHF practice directly deploy the last policy. In the following theorem, we show that ONPO also achieves linear convergence rate in the last iterate.
Theorem Assume that the Nash policy '' unique, with rj S 5. we have KL(TT [|))) - O(C
where C' > 1 is a constant.
The proof follows directly from the analysis of Theorem 3 in Wei et al. also demonstrate that self-play with OMD achieves last-iterate convergence. H However, their result relies on the strong convexity induced by the KL regularization terms in their game objective and does not apply to our formulation. This highlights another key advantage of using optimistic OMD: it not only improves the duality gap bound but also ensures last-iterate convergence without requiring explicit regularization.
2020 Zhang et al. [024
3.3 Implementation of ONPO
In this subsection, we describe the implementation of ONPO with query access to the preference oracle IP. The primary challenge in implementing ONPO lies in computing re(y), which involves taking an expectation over the entire policy Tr. Fortunately, this challenge can be addressed by avoiding the direct estimation (3) and instead relying on binary preference feedback between responses.
To achieves this, our goal to design loss function that does not involve Pyy > me) for policy optimization. We focus on obtaining the loss objective for n here and the derivation for o. is similar. The key observation is that, m. has a closed-form solution which satisfies Vy. s/ V
(e())
T'()) = (P > TT-1) - Ply > mt- (v')
log (3))
log
Therefore, similar to the techniques used in Azar et al. [024], Zhang et al. 2024]. solving ne is equivalent to finding the minimizer of the following loss function:
Eyw (T..).))))) (P(y > --- - P(y' > -
where ( v.y' 3/) = log [(y log Foa Since the inside win rate term is with respect to TT-1 and we also have an expectation over - outside, the loss function can be further written as
m(y)
[.(y)
Egg.'' y.yy'' wole 1. 1ww. -n>Ap(()a))
where Ap is the preference distribution [Calandriello et al. 2024]:
(v.v') with probability P(y 3²)
\p(
(v/.v) with probability 1 Pyy 3/)
To calculate the loss function, we only need the access to sample from the current policy, which is standard and easy to implement in practice. Putting everything together, the implementation of ONPO is summarized in Algorithm[]
In the beginning, we initialize тं and with the supervised fine-tuned policy TSFT. At each iteration I, we sample responses from the current policy n, and use the preference feedback from the oracle to construct the dataset Dr. Then we can directly minimize the corresponding loss functions on D; to find T(+ and T++1 respectively. We use the last iteration policy कт as the output policy, which is consistent with online RLHF practice (Dong et al. 2024 Wu et al. 2024 Zhang et al. 2024].
4 Discussion
In this section, we discuss the differences between ONPO and other general preference alignment methods.
5

---

Algorithm Implementation of ONPO
1: Input: Number of iterations T, learning rate 7/. preference oracle P, supervised fine-tuned policy TSFT-
2: Initialize π| <- STFT, TSF.-
3: for iteration f = 1, ..2...., do
4: Sample response pairs from the current policy Tei
{vi
9/2 Ji=1 Mt.
(4)
(6) in 5: Construct preference dataset D. = {yle yi with feedback from the oracle P.
6: Calculate 4/41 as:
(9 Mhus 31) ")
21
M++1 argmin E yw D D.
7: Calculate T+41 as:
(aLt2(T, yw u1) ")
T++1 = argmin Eg MnnD
8: end for Output TT.
9:
IPO. Azar et al. [2024] is the first to address general preference alignment in LLMs. The optimiza- tion objective of IPO is:
max E Byn [P( y')] - - (
where μ is fixed policy. From game-theoretic perspective, the goal of IPO is find the best response to However, this approach only ensures that the learned policy outperforms μe, which leaves the possibility that another policy could outperform the learned policy. In contrast, our approach focuses on learning the Nash policy in a two-player game. This provides stronger theoretical guarantees, as the Nash policy will not lose any other policy.
Nash-MD. Munos et al. [2023] is the first to formulate the alignment problem as two-player zero-sum game. Their game objective includes KL regularization terms, which ensure that the player's 's policy remains close to the reference policy Tet The KL terms are weighted by parameter T. They proposed an iterative algorithm, Nash-MD, to learn the Nash policy of the game. At each iteration t, the policy is updated as:
T++1 argmax P(T m!)
KT (T m!),
?)
where म/ is geometric mixture policy of the current policy and the reference policy mel
The Te(y)) are te7 Ey (v)')) 1 neT (
<V(
Nash-MD requires sampling from the mixture policy n/K However, the response space ) is often exponentially large, making the exact computation of n. intractable. To address this, Munos al. [2023) propose sampling from approximate policy. The theoretical guarantees of this approxima- tion remain unclear. contrast, our approach only requires sampling from the current policy n. which is straightforward to implement in practice.
Online IPO. Calandriello et 2024 propose the online IPO population loss:
( log (Buw))Ter(1) (Te)) (Hm)) 2- }}
E y.s/ SS][]]
Mu (1re))))
where SG is the stop-gradient operator, which prevents gradients from propagating through the data-generation process. Unlike the offline IPO approach, which always samples from a fixed policy μ, online IPO leverages responses generated by the current policy π.

---

Since the policy is updated throughout training, policy gradient methods are used to minimize the objective. However, as discussed earlier. policy gradient methods in RLHF have limitations, including being resource-intensive and unstable to train. In contrast, ONPO avoids these challenges by directly minimizing loss function over a preference dataset, offering more stable and efficient implementation.
DNO. The theoretical version of DNO (Algorithm in Rosset et 2024] relies on computing
(y()) = By' (2 > y')l, which requires taking an expectation over the current policy тy This
computation is challenging to implement in practice, Rosset et al. 2024 propose practical
version, DNO-Pret (Algorithm 2), where T is updated as follows:
(11 11 ((
argmax (e D. log σ
Ty. (V (T
When constructing the dataset D,, only response pairs with large margins are selected. This selection is motivated by the fact that, to approximate DNO, the ideal condition is (1r rr ( )) R 1. However, this cannot identical to the DPO objective [Rafailov et al. 2024b Therefore, DNO-Pret can be viewed as an iterative version of DPO.
fully achieved since ry( E [0, Notably, the objective of DNO-Pret
SPPO.
Wu et al. 2024 propose a self-play algorithm SPPO, The policy update in SPPO is:
TT)) (v(
(Py > me) - )
T++1 argmin Bywwr log
where P is heuristic approximation of (P n²)- However, obtaining accurate estimation of Pyy > TT) is challenging in practice. For example, Hoeffding's inequality suggests that more than 100 queries are needed to ensure P(y > πT) Ply > n.) s 0.1. This requirement results in high annotation and computation costs, 100 oracle queries are needed for single response y. In contrast, ONPO bypasses the need to estimate Ply > πt) and instead relies on binary preference signals between two responses.
Zhang et al. 2024] propose a self-play algorithm, INPO, which employs OMD to iteratively update the policy, described in Section 3.1 Leveraging the faster convergence properties of optimistic OMD, ONPO achieves an improved duality gap bound of O(T-)), )), compared to the O(T-1/3) bound of INPO.
INPO.
Two concurrent works also study general preference alignment, Liu al. (2024] propose two-level algorithmic framework that differs significantly from ours. In each iteration, their meta-algorithm defines new KL-regularized game and applies a general preference alignment algorithm, such as INPO, to learn its Nash policy. Their theoretical analysis shows that the resulting policy converges asymptotically to the true Nash policy. Wu et al. 22055) focus the multi-turn setting and adopt a deep reinforcement learning approach based on the actor-critic framework. In contrast, work targets computationally lightweight algorithm and therefore focuses on the single-turn setting.
Concurrent Works.
Experiments
5.1 Main Results
Experiment Setup. We implement ONPO following the online RLHF workflow described in
Dong et al. (2024). Two base models used as the initial policy πj: Llama-3-SFIE] based on Llama-3-8B [Dbeey et al. 2024], and (nsstralInstruue-vo..]]] an instruct fine-tuned version of the Mistral-7B-vol For the general preference oracle,
use pairwise preference mode which
demonstrates better performance compared to the BT reward model [Zhang et al. 2024 Training
details for the preference model are available in Dong et al. (2044.
http:::/huuggugggaae...///ttLLtt www.:/preeerreeeeeeeeeeeeee mode-- LLLMM3--LLMM3---0

---

Table 1: Results on three benchmarks. "ONPO+Mistall-Ir refers tuning the Mistral-Insttuee model with ONPO, while "ONPO+LLama-3-SSE" refers to tuning the Llama-3-SFT model with ONPO. Results where the baseline outperforms ONPO underlined.
Model
Size
AlpacaEval 2.0
Arena-Hare Hard
MT-Bench
Iterative DPO + Mistral-It
7B
32.0
22.2
7.35
SPPO Mistral-It
7B
33.1
24.5
7.51
INPO + Mistral-It
7B
35.3
25.3
7.46
ONPO Mistral-It
7B
42.8
29.7
7.68
Iterative DPO Llama-3-SFT
8B
28.3
31.9
8.34
SPPO + Llama-3-SFT
8B
38.5
32.9
8.23
INPO + Llama-3-SFT
8B
44.2
37.0
8.28
ONPO + Llama-3-SFT
8B
48.6
36.4
8.40
Llama-3-8B-it
8B
24.8
21.2
7.97
Tulu-2-DPO-P1
70B
21.2
15.0
7.89
Llama-3-70BB
70B
34.4
41.1
8.95
141B
30.9
36.4
8.66
HPP-33.-tturbo-6633
22.7
24.8
8.39
GPT-4-0613
30.2
37.9
9.18
Claude-3-Opus
40.5
60.4
9.00
GPT-4 Turbo (04/09)
55.0
82.6
At each iteration, the current policy generates K = 8 responses using a different set of prompt 5 To
select yo (winner) and yt (loser), we follow the tournament approach in Zhang al. (2024), where the eight responses are compared pairwise to identify the winning and losing responses.
Since online iterative alignment methods have been shown to outperform offline counterparts, focus on comparing ONPO with other online methods for a fair evaluation. These include iterative DPO (Dong al. 2024). SPPO [Wu et al. 2024 and INPO [Zhang et al. 2024]. where the latter two are general preference aligmment approaches.
We evaluate the models three representative benchmarks: AlpacaEval 2.0 Li al. 2023a Arena- Hard [Li al. 2024) and MT-Bench (Zheng al.) [024]. AlpacaEval 2.0 has 805 instructions from five datasets, including self-instruct test set Wang et al. 2022]. Open Assistant test set, Anthropic's helpful test set [aii et al. 2022b), Vicuna test set [Zheng et al. 2024] and Koala test set [Geng et al.) 2023). Arena-Hard includes 500 challenging user queries from Chatbot Arena. Both AlpacaEval 2.0 and Arena-Hard compare model-genenee generreede answers against reference answers from a baseline model, using GPT-4 Preview- 106 as the judge model. We report the win rate for Arena-Hard and the length-controlled (LC) win rate [Dubois et al.) 2024) for AlpacaEval 2.0. MT-Bench consists of 80 multi-turn questions, where responses rated by GPT-4 on 1-10 scale, with the average rating reported.
Results. The model performance is summarized in Table Our results show that ONPO consis- tently outperforms or achieves comparable performance to the baselines across both base models. Among the three benchmarks, the length-controlled (LC) win rate in AlpacaEval 2.0 exhibits the highest 0.98 Spearman correlation with Chatbot Arena rankings [Dubois et al.) 2024). In this bench- mark, ONPO outperforms the strongest baseline by a clear margin achieving 9.9% improvement on Llama-3-SFI and 21 .%% improvement on Mistral-It. These results align with our theoretical findings, demonstrating that ONPO benefits from an improved bound on the duality gap. We also compare ONPO with other LLMs that have significantly larger parameters, such as Llama-3-70B-in Mixtral-8x22B-it and GPT-4-Turbo. Remarkably, our ONPO even outperforms models with least nine times more parameters.
8

---

Table 2: Model performance academic benchmarks (AVG: average).
Model
GPQA
Hellaswag
MML.U-ro
Winogrande
TruthfulQA
GSM8K
AVG
Mistral-It
30.1
83.5
30.4
74.2
59.7
49.5
54.6
Iterative DPO
29.6
83.3
28.0
75.1
64.0
45.7
54.3
SPPO
28.7
83.5
28.1
73.9
66.4
49.9
55.1
INPO
28.8
82.9
28.9
74.9
64.7
46.3
54.4
ONPO
30.4
83.7
29.9
75.1
65.5
47.8
55.4
+ AlpacaEval 2.0
00.0
37.5
Win
32.5
27.5
60
90
100
120
140
160
180
200
Value
Figure 1: Performance of ONPO with different values of on Arena-Hard and AlpacaEval 2.0. ONPO consistently outperforms the best baseline, which achieves win rate of 25.3 on Arena-Hard and 35.3 on AlpacaEval, respectively.
5.2 More Results Academic Tasks
In this subsection, we evaluate the model's reasoning and calibration abilities across six academic benchmarks: GPQA [Rein et al., 2023) for graduate-level science question answering, MMLU- Pro [Wang al. 2024] for multitask language understanding. Hellaswag [Zellers et al. 2019 for commonsense inference, Winogrande (Sakaguchi et al. 2021] for difficult commonsense reason- ing, TruthfulQA [Lin al. 2021] to assess the model's 's tendency to reproduce falsehoods, and GSM8K [Cobbe al. 2021 for mathematical reasoning.
It is important to note that these benchmarks primarily evaluate model's intrinsic knowledge and capabilities, which are developed during the pre- praitngin stage rather than the alignment stage. However, as observed in prior work [Ouyang et al. 2022 OpenAl, 2023]. alignment can sometimes have a negative impact on these abilities a phenomenon known as the "alignment tax". Therefore, purpose in presenting these results is to verify that our alignment method preserves the model's abilities rather than demonstrating performance improvements.
We show the results using baselines as well as the base model itself. The results Table[]shov that ONPO achieves slightly higher average performance than both the base model and the baselines, demonstrating that ONPO does not over-align the model and effectively preserves its intrinsic knowledge and abilities.
as the base model and compare ONPO with three
5.3 Hyperparameter Sensitivity Analysis
In this subsection, we analyze the sensitivity of ONPO to the hyperparameter 77, which serves as the learning rate in the update rule. We conduct experiments using Mistral--Isttrut--0..3 as the base model and vary 7) from 200/3 to 200. The results, presented in Figure indicate that ONPO consistently achieves strong performance across different values of 1) and outperforms the baselines, demonstrating its robustness to hyperparamter variations.

---

Conclusion and Future Work
We propose Optimistic Nash Policy Optimization (ONPO), a novel approach for aligning LLMs with general preferences through self-play. By integrating optimistic online mirror descent, ONPO achieves improved theoretical guarantees for approximating the Nash policy of the game. More importantly, our experimental results demonstrate that ONPO consistently outperforms or matches state-of-the-art general preference alignment methods across multiple benchmarks. limitation of this work is that our experiments focuses on the single-turn setting. In the future, we plan to explore the implementation of ONPO under the multi-turn setting.
potential
Acknowledgements
Nan Jiang acknowledges funding support from NSF (NS-2112471, NSF CAREER IIS-2141781, Google Scholar Award, and Sloan Fellowship.
References
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandricllo. A general theoretical paradigm understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447-4455. PMLR, 2024.
Yu Bai, Chi Jin, and Tiancheng Yu. Ncar-optimal reinforcement learning with self-play. Advances in neural information processing systems, 33:2159-21177 2020.
Yu Bai, Chi Jin, Song Mei, and Tiancheng Yu. Near-optimal learning of extensive-form games with imperfect information. In International Conference on Machine Learning, pages 1337-1382. PMLR, 2022a.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2.Xiv:2204.08862. 2022b.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-344 1952.
Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, et al. Human alignment of large language models through online preference optimisation. arXiv preprint arXiv:2403.08633 2024.
Xi Chen and Binghui Peng. Hedging in games: Faster convergence of external and swap regrets. Advances in Neural Information Processing Systems, 33: 18:18990-1899, 2020.
Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. On the weaknesses of reinforcement learning for neural machine translation. arXiv preprint (000:0/9900077700.000 2019.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 2rXiv:2110.14166 2021.
Constantinos Daskalakis, Alan Deckelbaum, and Anthony Kim. Near-optimal no-regret algorithms for zero-sum games. In Proceedings of the twenty-second annual ACM-SIAM symposium on Discrete Algorithms, pages 235-254. SIAM, 2011.
Constantinos Daskalakis, Maxwell Fishelson, and Noah Golowich. Near-optimal no-regret learning in general games. Advances in Neural Information Processing Systems, 34:27604-2766 2021.
10

---

Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang. Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint wrXiv:2405.07863, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama herd of models.
arXiv preprint arXiv:2407.21788 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length--ontrolled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04477. 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024.
Yoav Freund and Robert E Schapire. Adaptive game playing using multiplicative weights. Games and Economic Behavior, 29(1-2):79-1033 1999.
Xinyang Geng, Arnv Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and
Dawn Song. Koala: A dialogue model for academic research. Blog post, April 2023. URL
https: www.eeeearrrbeeeeeeee..eeeee
Dongfu Jiang. Xiang Ren, and Bill Yuchen Lin. Lim-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXXiv:336..02550 2023.
Chi Jin, Qinghua Liu, Yuanhao Wang, and Tiancheng Yu. V-learning-a simple, efficient, decentralized algorithm for multiagent rl. arXiv preprint 1tti:22100115555 2021.
Tomasz Korbak, Ethan Perez, and Christopher L Buckley. RI with kl penalties is better viewed as bayesian inference, arXiv preprint arXiv:2205.1 11755, 2022.
Tadashi Kozuno, Pierre Ménard, Rémi Munos, and Michal Valko. Model-free learning for two-player zero-sum partially observable markov games with perfect recall. arXiv preprint arXiv:2106.06279 2021.
Christian Kroer, Kevin Waugh, Fatma Kilmç-Karzan and Tuomas Sandholm. Faster algorithms for Eitensive-form game solving via improved smoothing functions. Mathematical Programming. 179 (1):385-411 2020.
Nathan Lambert, Valentina Pyatkin. Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.I 13787, 2024.
Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
Chung-Wei Lee, Christian Kroer, and Haipeng Luo. Last-iterate convergence extensive-form games. Advances in Neural Information Processing Systems, 34:14293-14305 2021.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, 2024.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: automatic evaluator of instruction-follooinng models, 2023a.
Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. In Forty-first International Conference Machine Learning, 2023b.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint 1rXiv:2109.07990 2021.
Qinghua Liu, Tiancheng Yu, Yu Bai, and Chi Jin. A sharp analysis of model-based reinforcement learning with self-play. International Conference on Machine Learning, pages 7001-7010. PMLR, 2021.
11

---

Yixin Liu, Argyris Oikonomou, Weiqiang Zheng, Yang Cai, and Arman Cohan. Comal: A convergent meea-algorithm for aligning Ilms with general preferences. arXiv preprint arXiv:2410.25223, 2024.
Tung Mai, Milena Mihail, loannis Panageas, Will Ratcliff, Vijay Vazirani, and Peter Yunker. Cycles in zero-sum differential games and biological diversity. In Proceedings of the 2018 ACM Conference on Economics and Computation, pages 339-350, 2018.
Weichao Mao and Tamer Başar. Provably efficient reinforcement learning in decentralized general- sum markov games. Dynamic Games and Applications, 13(1):165-186, 2023.
Kenneth May. Intransitivityy utility, and the aggregation of preference patterns. Econometrica: Journal of the Econometric Society, pages 1-13, 1954.
Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. arXiv preprint arXiv:23 00.22322.008880 2023.
OpenAL. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2(5), 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Baolin Peng, Linfeng Song, Ye Tian, Lifeng Jin, Haitao Mi, and Dong Yu. Stabilizing rlhf through advantage model and selective rehearsal. arXiv preprint arXiv:2309. 10202, 2023.
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn, From to g": Your language model is secretly a q- q-function. arXiv preprint arXiv:2404. 12358, 2024a.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024b.
Sasha Rakhlin and Karthik Sridharan. Optimization, learning, and games with predictable sequences. Advances in Neural Information Processing Systems, 26, 2013.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12000 2023.
Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.0377 2024.
Abhishek Roy, Yifang Chen, Krishnakumar Balasubramanian, and Prasant Mohapatra. Online and bandit algorithms for nonstationary stochastic saddle-point optimization. arXiv preprint 1912::1912.01669 2019.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106. 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17Xiv:1707.06347, 2017.
Lior Shani, Aviv Rosenberg, Asaf Cassel, Oran Lang, Daniele Calandriello, Avital Zipori, Hila Noga, Orgad Keller, Bilal Piot, Idan Szpektor, al. Multi-turn reinforcement learning from preference human feedback. arXiv preprint arXiv:2405. 14655, 2024.
Vasilis Syrgkanis, Alekh Agarwal, Haipeng Luo, and Robert E Schapire. Fast convergence of regularized learning games. Advances in Neural Information Processing Systems, 28, 2015.
12

---

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.09288, 2023.
Amos Tversky. Intransitivity of preferences. Psychological review, 76(1):31, 1969.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct Aligning language models with self- seff-generate instructions. arXiv preprint wxxi:2212..0500 2022.
Yubo Wang, Xueguang Ma, Ge Zhang. Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574, 2024.
Chen-Yo Yu Wei, Yi-Te Hong, and Chi-Jen Lu. Online reinforcement learning stochastic games. Advances in Neural Information Processing Systems, 30, 2017.
Chen-Ye Yu Wei, Chung- Wei Lee, Mengxiao Zhang, and Haipeng Luo. Linear last-iterate convergence in constrained saddle point optimization. arXiv preprint arXiv:2006. 2020.
Yongtao Wu, Luca Viano, Yihang Chen, Zhenyu Zhu, Kimon Antonakopoulos, Quanquan Gu, and Volkan Cevher. Multi-step alignment as markov games: An optimistic online gradient descent approach with convergence guarantees. arXiv preprint arXiv:2502. 12678, 2025.
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00m0 00675, 2024.
Tengyang Xie, Dylan Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit q apaapooximation for sample-efficient rlhf. arXiv preprint 24XXi:2405.21000 2024.
Wei Xiong. Hanze Dong, Chenlu Ye, Ziqi Wang. Han Zhong. Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Conference on Machine Learning. 2024.
Chenlu Ye, Wei Xiong, Yuheng Zhang, Nan Jiang, and Tong Zhang. A theoretical analysis of nash learning from human feedback under general kl-regularized preference. arXiv preprint arXiv:2402.07314, 2024.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arX1v:2401.10020, 2024.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv: 2019.
Yuheng Zhang, Dian Yu, Baolin Peng, Linfeng Song. Ye Tian, Mingyue Huo, Nan Jiang, Haitao Mi, and Dong Yu. Iterative nash policy optimization: Aligning Ilms with general preferences via 2024.
no-regret learning, arXiv preprint arX1v:2407.0061
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging ilm-as-a-judge with mt-bench and chatbot arena. Advonces Neural Information Processing Systems, 36, 2024.
Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret minimization in games with incomplete information. Advances in neural information processing systems, 20, 2007.
13

---

A Related Literature
Reward-Based RLHE.
Since the first RLHF framework proposed by Christiano et al. [2017).
RLHF has achieved tremendous success in aligning large language models (LLMs), powering models
such Instruct-GPT [Ouyang et al.) 2022]_ Llama 2 [Tovrron et al. 2023], and Claude (Bai et al.]
2022b]. The RLHF pipeline typically involves training a reward model followed by applying policy gradient methods such as PPO (Schulman et al. 2017] to optimize KL-regularized objective [Korbak et al. 2022 Li et al. 2023b]. Nevertheless, the use of PPO in RLHF introduces challenges, including instability during training [Choshen al., 2019] and high computational costs [Yuan al., 20233 To address these limitations, |afailov et al. [2024b] proposed the DPO algorithm, which directly optimizes preferences by minimizing a loss objective on offline datasets. Additionally, other direct preference learning algorithmms have been developed, including offline methods [Ethayarajh et al.
2024] and online (iterative) methods [Xie al. 2024 Xiong al.) 2024 Yuan et al.] [2024]. However, all these algorithms are reward- based and rely on the Bradley-Tere (BT) model assumption. In this paper, we remove the BT model assumption and consider general preference alignment.
RLHF with General Preferences. Azar et al. [0244] is the first to consider the general preference without BT model assumption. They propose the offline IPO algorithm to learn the optimal policy
when the comparator policy is fixed. MMnos et al. (2023) formulate the alignment problem as a two-player zero-sum game and propose the iterative Nash-MD algorithm to find the Nash policy of the game. Subsequently, there has been a line of work (Ye et al. 2024 Calandriello et al.] 2024 Rosset et al.) 2024 Wu et al., 2024] developing online algorithms for learning the Nash policy. The closest work related to ours is Zhang et al. [2024), which also employs a no-regret learning algorithm for self-play. However, our algorithm incorporates an optimistic predictor into the policy update, achieving improved theoretical guarantees and better empirical performance. A detailed comparison between our algorithm and other general preference alignment algorithms is provided in Section 4
Learning in Games. Online learning and self-play algorithms are widely used in approximating the equilibrium of games, including normal- form games [Freund and Schapire 1999
Daskalakis
ct al. 2011 Mai et al. 2018 Roy et al., 2019 Chen and Peng [0200] Wei et al.
2020
Daskalakis
ct al. 2021]. extensive- form games [Zinkevich et al. [007] et al. [2020] Kozuno et al.) 2021
Lee et al. [002]] Bai et al.
2022a] and Markov games [eei et al.) 2017] 7 Jin et al. 2021 Liu et al.
202]] and Basar 2023 Our work is inspired by the faster convergence properties of optimistic
online mirror descent in equilibrium learning (Rakhlin and Sridharan 2013 Syrgkanis et al. 2013).
B Extension to the Multi-Ture Setting
In this section, we describe how ONPO can be extended to the multi-turn setting, which is formulated as a contextual Markov decision process (CMDP) (Shani et al. 2024]. The interaction between the LLM and the environment unfolds as follows: the LLM starts at a fixed initial state $1 E S and takes an action y/1 x(- s1). The environment then transitions to the next state 82 P(- $1, y1) according to the transition dynamics P, and the LLM subsequently takes action y/2 ~ T[- $2). This process repeats for H steps, ultimately reaching the final state s11+1- 1- At the end of the interaction, the preference oracle compares two final states and provides a preference signal: z ~ Ber (P (+) > .)1+1)). This CMDP formulation effectively captures various LLM applications, including chatbot interactions and token-level MDPs [Rafallov et al. 2024a).
In the multi-turn setting, the challenge is that preferences are only provided for the final states, and there is no direct feedback for intermediate states. To address this, we use Q-value functions, which capture the long-term expected outcomes, in the optimization objective. For each state 8s, the update rule for 7.41(- sn) is:
argmax (TT, QTA (sh; -)) 1 KL | ( 8n)),
where Tue (Sh: yh) = E. [P(s TT) 8h yhu] and P(s > Te) represents E.,, [I((s sH+1)] Here (IT, Q (8h. )) measures the probability of æ outperforming n. at state sn. The update rule for 7(-1 is similar, except that the KL divergence is computed between π and m/
The primary challenge in implementing ONPO in the multi-turn setting lies the efficient estimation of (e Shani et al. 2024 propose to use actor-critic framework that employs policy-gradien
14

---

methods such as PPO [Schulman et al. 2017] for policy optimization. However, methods are known exhibit high variance and sensitivity to implementation details, leading to increased computational costs. In this paper, we focus on implementing ONPO in the single-turn setting and leave the implementation under the multi-turn setting for future work.
C Proofs for Section
C.1 Proof for Theorem
Proof. According the regret analysis of OMD we have
[Lattimore and Szepesvári 2020]. for any policy x,
KL(T (T))
(T,, re)
(Te re) 5
+ 17
I 2VTD.
The rest proof follows from Theorem 3 in Zhang et [2024]
C.2 Proof for Theorem 2
Proof. Let (T)) = Ey (ly log क (yy) the KI. divergence between क. and 22 can also be written as the Bregman divergence term:
(L(x)]lT2) = D (m. m22 (b())) (Tx2) (Ve(T2) x2)
Since v strongly convex with respect to L, norm, we can apply regret analysis from Rakhlin and Sridharan 2013 Syrgkanis et al. (2015 and obtain that for any N
KL(x'||<1)
Σ (T' Te, r() 5
Ir. r 11
[T Te
"
41
We observe that for any t 2 2 and any
(ve(y) - (r)1 Σ
(/)()))(T(t))) m--1(1)) S || Te-1l11-
Once we have bn 2 17, the terms
n/(r: r1-1l13- and - - || T- cancel out and get
T
Σ
( - Te, 2VVD.
-
t=1
Next, we decompose the duality gap as:
DualGap(#) = max J (TI, 7)
min [ π2)
2
2
Tem
Term
We show how to bound Term and Term bounded similarly due to the symmetric nature the game. Let n' argmax,, ( we have
J J( Re)
( me)
T
-
Σ
{n' nc. T()
2/D
T
The proof is finished by also having ming, (R, < 2/D.
p
15

---

D Additional Experiment Details
For the implementation of ONPO, we follow the hyperparameters in Dong al.) 2024]. including the cosine learning rate scheduler with a peak learning rate of 5 x 10- 0.03 warm-up ratio, and a global batch size of 128. We use grid search for 1/1 over [0.1, 0.05, 0.02, 0.01, 0.005] and set 1/n 0.01. Llama-3-Sh 3-FT is trained for 5 iteration [ where in each iteration Ti is trained for epochs and के for epoch. While Mistral-Instruct, having already undergone instruction fine-tuning, thereby trained for 3 iterations, with ni trained for I epoch and for 2 epochs in each iteration. To ensure fair comparison, all baselines are trained using the same number of iterations and the same prompt set ONPO. All experiments are conducted on 8xA100 GPUs with 40GB memory each.
'teration
Iteration 2
Iteration
Iteration 4
Iteration
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
We present the theoretical results in Section and experimental results in Section
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

Justification:
We state the assumptions in Section 3 and provide the proofs in Appendix C
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section 5 and Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification: We provide details about our data that can be found online. We have uploaded our codes.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nipss.cc/pulli///uuu guwde//ddee/ee/ wwt.:Suumisssispoooooooooooooooe
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Section 5 Jand Appendix D
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Conducting LLM experiments with statistically significant justifications is challenging due to the high computational costs and the substantial carbon emissions generated.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
19

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Conducted.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets are used properly.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
23


Reviews:
[{"review_id": "zzP48UO8Ww", "reviewer": "Reviewer_9enM", "summary": "The paper proposes a method for preference alignment of LLMs by formulating the problem as a two-player game via game theory. This integrates online mirror descent into an alignment framework to approximate the Nash policy. The goal is to leverage the faster convergence properties of online mirror descent in the preference learning use case.", "questions": "- Even without full implementation, demonstrating ONPO on multi-turn with toy examples would help validate its broader application and scope. \n- Runtime and efficiency comparisons. Training time, GPU setup, convergence speed, against more relevant baselines listed above. This will help validate the claim of \"computational lightness\". \n- Add discussion/experiments considering the possibility of a noisy oracle, to try and model how this will behave in real world.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- Paper aims to integrate classical mirror descent with preference learning. This is an interesting and novel approach. \n- Optimistic Nash Policy Optimization improving the convergence rate on the duality gap with additional guarantees is interesting and non-trivial. The theoretical contribution does integrate well with the paper, and shows understanding of learning with general preferences beyond the Bradley-Terry framework. \n- Compares against DPO\n\nWeaknesses: \n- Only compares against DPO and 2 minor methods. Other methods in this area (SimPO, ORPO, R-DPO, RL-based methods etc) are not explored. \n- No discussion of experimentation setup. Hardware, training time, codebase, etc. Makes it hard to assess practical trade-offs between methods. \n- Method assumes access to a general preference oracle (possibly learned), but does not consider the effect of noisy/imperfect oracle feedback. Since human preferences can be inconsistent, this is a critical.", "was_revised": true, "final_justification": "The rebuttal clarified certain points, yet detracted from others. The authors concede experiments are on \"toy examples\" which is concerning. The main practical deployment possibilities of this method are not apparent, especially in this particularly dense area of research where many topics have been published in recent months. I would like to state my overall score as **3.5** ideally. Although the authors do not demonstrate a practical application where their work excels at, the concept of incorporating traditional mirror descent is interesting. However this work will significantly benefit from an additional round of reviews as well, to clearly solidify their theoretical contributions/practical superiority in a specific use-case.", "rebuttal": "We thank the reviewer for their valuable comments and respond to the weaknesses and questions below.\n\n**Only compares against DPO and 2 minor methods. Other methods in this area (SimPO, ORPO, R-DPO, RL-based methods etc) are not explored.**\n\nWe would like to clarify that the two baselines we compare against, SPPO and INPO, are state-of-the-art approaches for online preference alignment.\n\nThe methods mentioned by the reviewer are designed for the **offline** alignment setting, where the algorithm optimizes a learning objective over a fixed, pre-collected dataset. In contrast, our work focuses on the **online** setting, where no response pairs are pre-collected and the model must iteratively generate responses and improve itself based on collected preferences.\nAs stated on line 263, prior work has shown that online or iterative alignment methods can outperform their offline counterparts [Guo et al., 2024; Dong et al., 2024]. For this reason, we compare ONPO with other online methods to ensure a fair and meaningful evaluation.\n\nAmong the offline methods, SimPO has demonstrated strong performance. To further strengthen our comparison, we implemented an iterative variant of SimPO, where the current model generates responses and optimizes the SimPO objective at each iteration. As shown below, ONPO significantly outperforms it across all three evaluation benchmarks:\n| Model               | AlpacaEval 2.0 | Arena-Hard | MT-Bench |\n|---------------------|----------------|------------|----------|\n| SimPO + LLaMA-3-SFT | 35.0           | 31.7       | 8.19     |\n| ONPO + LLaMA-3-SFT  | 48.6           | 36.4       | 8.40     |\n\n\n**No discussion of experimentation setup. Hardware, training time, codebase, etc. Makes it hard to assess practical trade-offs between methods.**\n\nDetails of our experimental setup are provided in Section 5.1 and Appendix D. As noted on line 564, all experiments were conducted on 8×A100 GPUs with 40GB memory each.\n\nWe apologize for not reporting the actual training time in the main text. All methods follow the same online training setup and differ only in their learning objectives. In each iteration, approximately 3–4 hours are spent generating the preference dataset. Each baseline is then trained for two epochs, which takes around 3 hours. In the Mistral experiments, ONPO uses the same total number of training epochs per iteration, one for $\\pi'\\_{t+1}$ and one for $\\pi_{t+1}$, resulting in equivalent computational cost. In the LLaMA experiments, we use two epochs for $\\pi'\\_{t+1}$ and one for $\\pi_{t+1}$, introducing only one additional epoch (~1.5 hours). We also tested training the baselines with three epochs per iteration and did not observe performance improvements.\n\nOur codebase is adapted from Dong et al. [2024], and all baselines are implemented following their official code.\n\n**Method assumes access to a general preference oracle (possibly learned), but does not consider the effect of noisy/imperfect oracle feedback. Since human preferences can be inconsistent, this is a critical.**\n\nWe want to first emphasize that our preference oracle is a general formulation of the underlying human preferences we aim to align with. It is not “learned” as the reviewer suggested, since the oracle is simply a mathematical abstraction of the process that human generates preference labels. Specifically, given a prompt $x$ and two responses $y_1$ and $y_2$, the oracle gives the probability $\\mathbb{P}(y_1 \\succ y_2 \\mid x)$ that humans would prefer $y_1$ over $y_2$. \n\nIt is unclear to us what the reviewer means by “inconsistent”. The reviewer might be thinking of properties of commonly used models such as the Bradley-Terry (BT) model, which assumes that $\\mathbb{P}(y_1 \\succ y_2 \\mid x)$ follows a sigmoid function of some underlying reward/utility function $R$. In this case, a real human feedback may indeed be “inconsistent” with the BT model in that the feedback may not be consistent with any reward function.  In contrast, we only assume access to a binary preference signal sampled from $\\mathrm{Ber}(\\mathbb{P}(y_1 \\succ y_2 \\mid x))$.  As long as we are comfortable with modeling the human feedback as a probability distribution, this is really the **minimal** assumption. This binary signal plays a similar role to labels in supervised learning, it is essential for model training, and all LLM alignment methods rely on such preference data for supervision. \n\nWe acknowledge that modeling and mitigating the effect of noise in preference signals is an important direction for future work. In our case, stochastic noise will simply change the function $\\mathbb{P}(y_1 \\succ y_2 \\mid x)$, and our algorithm will still align with the preference function modified by noise. How to align with the “original” preference is analogous to handling noisy or corrupted labels in supervised learning. However, we consider this direction to be orthogonal to our primary focus, which is to develop a more effective alignment method with provably faster convergence. We will include a discussion of this point in the revised version.\n\n**Even without full implementation, demonstrating ONPO on multi-turn with toy examples would help validate its broader application and scope.**\n\nWe appreciate the suggestion. However, the core challenge of the LLM alignment problem lies in enabling large models to learn complex human preferences, which are difficult to meaningfully represent in toy examples. As discussed in Appendix B, extending our method to the multi-turn setting would require incorporating deep reinforcement learning and policy gradient techniques, which are computationally expensive and often unstable. Given our focus on developing a computationally lightweight method, we chose to concentrate on the single-turn setting in this work and leave the implementation and evaluation of the multi-turn extension to future work.\n\nShangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792, 2024.\n\nHanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863, 2024."}, {"review_id": "zf7rTkU8PU", "reviewer": "Reviewer_9tRL", "summary": "This paper addresses the problem of aligning large language models (LLMs) with human preferences without relying on the restrictive Bradley-Terry (BT) model assumption. The authors formulate LLM alignment as a two-player zero-sum game and propose Optimistic Nash Policy Optimization (ONPO), which integrates optimistic online mirror descent into a self-play framework. The authors obtain an improved theoretical convergence guarantees of $\\mathcal{O}\\left(\\frac{1}{T}\\right)$ duality gap bound versus previous $\\mathcal{O}\\left(\\frac{1}{\\sqrt{T}}\\right)$ results. The authors also prove a linear convergence in the last iterate which is a novel result for this setup. Finally the authors provide strong empirical performance across multiple benchmarks including AlpacaEval 2.0, Arena-Hard, and MT-Bench when the other models have same/comparable number of parameters.", "questions": "How much more computational resources would be needed for a multi turn setup?", "limitations": "The assumption of an oracle and single turn setup seem to be the only limitations I can think of. However, for a proof of concept paper these seem to be reasonable assumptions.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1.The paper identifies a key limitation of existing RLHF methods that rely on BT assumptions and provides evidence from prior literature that challenges its validity in modeling human preferences.\n2. Achieves state of the art convergence results for the duality gap as well as linear convergence rate for the last iterate. The latter is something that is novel.\n3. Dropping the Bradley-Terry model assumption is well-justified. The authors provide compelling evidence that human preferences often violate transitivity assumptions, especially at population levels, making the general preference setting more realistic.\n\nWeakness: \n1. While performance improvements against other models have been established, they have been done so for models of size 8B/7B. the question as to whether the same performance improvement would be had if the model sizes are scaled up to larger sizes remains open.\n2. Their is an assumption of a preference oracle here. Is that a realistic assumption? It appears not to be the case but I am open to the authors explaining why it might be.\n3. The self play here is done for a single turn. While I am not very familiar with the practical aspects of RLHF this seems significant.", "was_revised": true, "final_justification": "I have gone over the replies of the reviewers. They have answered the questions to my satisfaction, and I maintain my positive score.", "rebuttal": "We thank the reviewer for their valuable comments and respond to the weaknesses and questions below.\n\n**While performance improvements against other models have been established, they have been done so for models of size 8B/7B. the question as to whether the same performance improvement would be had if the model sizes are scaled up to larger sizes remains open.**\n\nDue to limited computational resources, our experiments are conducted on 7B/8B base models. This limitation is common in the LLM alignment literature, where experiments on larger-scale models are rarely presented. We acknowledge that it remains an open question whether the same performance gains hold at larger model scales, and we plan to explore this direction in future work as resources allow.\n\n**Their is an assumption of a preference oracle here. Is that a realistic assumption? It appears not to be the case but I am open to the authors explaining why it might be.**\n\nWe would like to clarify that the preference oracle is not an assumption but a general formulation of the underlying human preferences we aim to align with. Specifically, given a prompt $x$ and two responses $y_1$ and $y_2$, the oracle gives the probability $\\mathbb{P}(y_1 \\succ y_2 \\mid x)$ that humans would prefer $y_1$ over $y_2$. This offers a  mathematical abstraction of the process that humans generate preference labels.\n\nCommonly used models such as the Bradley–Terry (BT) model is a restrictive special case of the general preference formulation, where $\\mathbb{P}(y_1 \\succ y_2 \\mid x)$ is assumed to follow a specific parametric form, e.g., the sigmoid of the reward difference $R(x, y_1) - R(x, y_2)$. In contrast, we only assume access to a binary preference signal sampled from $\\mathrm{Ber}(\\mathbb{P}(y_1 \\succ y_2 \\mid x))$.  As long as we are comfortable with modeling the human feedback as a probability distribution, this is the **minimal** assumption. This binary signal plays a similar role to labels in supervised learning, it is essential for model training, and all LLM alignment methods rely on such preference data for supervision. \n\n**The self play here is done for a single turn. While I am not very familiar with the practical aspects of RLHF this seems significant.**\n\nAs discussed in Appendix B, our method can be extended to the multi-turn setting. However, implementing the multi-turn setting would necessitate using deep reinforcement learning techniques and policy gradient methods, which are known to be computationally expensive and unstable during training. Our primary goal in this work is to develop a computationally lightweight method and focus on the single-turn setting, leaving the implementation of the multi-turn extension as future work.\n\n**How much more computational resources would be needed for a multi turn setup?**\n\nWe do not have precise estimates for the additional computational resources required for a multi-turn setup. However, based on our experience from related projects involving policy gradient methods for post-training large language models, achieving satisfactory performance typically requires 10 or more epochs. In contrast, our current method uses only 2–3 epochs per iteration."}, {"review_id": "i86lprTF1Y", "reviewer": "Reviewer_eFXw", "summary": "The paper introduces Optimistic Nash Policy Optimization (ONPO), a novel online algorithm designed for aligning large language models with general human preferences beyond the restrictive Bradley-Terry assumption. By framing the preference alignment task as a two-player zero-sum game, ONPO leverages optimistic online mirror descent (OMD) to approximate the Nash equilibrium more efficiently. Theoretically, ONPO improves the convergence rate on the duality gap from the previously established and ensures linear convergence in the final iterate. Empirically, ONPO demonstrates superior performance compared to existing methods like iterative DPO, SPPO, and INPO across standard benchmarks including AlpacaEval 2.0, Arena-Hard, and MT-Bench, effectively maintaining model capabilities on reasoning and calibration tasks.", "questions": "See weaknesses. I am willing to raise my score if my concerns are addressed.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n* **Theoretical Advances**: The paper proposes Optimistic Nash Policy Optimization (ONPO), an innovative integration of optimistic online mirror descent (OMD) into LLM alignment. This method notably improves the convergence rate on the duality gap from $O(T^{-1/2})$ to $O(T^{-1})$, a significant theoretical advance.\n\n* **Strong Empirical Results**: Empirical evaluations demonstrate consistent improvements over state-of-the-art alignment methods (Iterative DPO, SPPO, INPO) across prominent benchmarks (AlpacaEval 2.0, Arena-Hard, MT-Bench). Notably, ONPO achieves substantial improvements on AlpacaEval 2.0, outperforming baseline methods even when compared to significantly larger models.\n\n### Weaknesses\n\n* **Lack of Statistical Significance Reporting**: The paper presents experimental results without reporting standard deviations or confidence intervals, casting doubt on the statistical significance of the observed improvements. Without this information, it is challenging to determine whether differences between methods are statistically meaningful or simply the result of random variance.\n\n* **Missing Empirical Comparison with Closely Related Methods**: Although the discussion section thoroughly situates ONPO among related algorithms like Nash-MD, DNO, and Online IPO, these methods are not included in the empirical comparison. This omission weakens the paper’s claim of superiority, especially since these algorithms are based on similar game-theoretic or general preference formulations.", "was_revised": true, "final_justification": "I thank the authors for the additional experiments and for implementing both Nash-MD and Online IPO during the rebuttal. While these results are helpful, my main concerns remain. The lack of comprehensive statistical significance reporting across all benchmarks is a notable limitation, especially for RL-based methods where variance can be high. Additionally, although the new baselines were included, the comparisons are still somewhat limited in scope, and the absence of broader variance analysis makes it difficult to fully assess the reliability of the reported improvements. I therefore keep my original rating.", "rebuttal": "We thank the reviewer for their valuable comments and respond to the weaknesses and questions below.\n\n**Lack of Statistical Significance Reporting**\n\nWe acknowledge the importance of statistical significance as raised by the reviewer. However, in contrast to traditional machine learning, LLM experiments are extremely resource-intensive, making statistical significance reporting relatively rare in the LLM community. This is not only the case in academic work, including closely related papers such as Iterative DPO [Dong et al., 2024], SPPO [Wu et al., 2024], and INPO [Zhang et al., 2024], but also in large-scale technical reports such as the Qwen3 Technical Report [Yang et al., 2025].\nHere we report standard deviations on the AlpacaEval 2.0 benchmark, one of the most widely used and representative alignment benchmarks:\n\n| Method                  | AlpacaEval 2.0 (LLaMA-3-SFT) | AlpacaEval 2.0 (Mistral-IT) |\n|-------------------------|------------------------------|------------------------------|\n| Iterative DPO           | 28.3 ± 1.3                   | 32.0 ± 1.3                   |\n| SPPO                    | 38.5 ± 1.4                   | 33.1 ± 1.2                   |\n| INPO                    | 44.2 ± 1.4                   | 35.3 ± 1.4                   |\n| ONPO                    | **48.6 ± 1.5**               | **42.8 ± 1.4**               |\n\n\nAs shown, the standard deviations are relatively small and ONPO consistently outperforms all baselines by a statistically meaningful margin.\n\n**Missing Empirical Comparison with Closely Related Methods**\n\nWe address the reviewer’s concern by discussing each method individually:\n\n**Nash-MD**. The Nash-MD algorithm cannot be directly implemented in practice for two main reasons: (1) computing the geometric mixture policy is computationally intractable, and (2) estimating the expected win rate between two policies is also challenging in practice. As a result, practical implementations of Nash-MD require heuristic approximations. At the time of our project, we did not find the official implementation, and prior related works did not include comparisons with Nash-MD.\n\nDuring the rebuttal period, we made our best effort to implement a variant of Nash-MD using the TRL trainer. Since our setting provides only binary preference signals rather than scalar win probabilities, we approximated the win rate by assigning a value of 1 to the preferred response and 0 to the other. The results on LLaMA3-8B-SFT are shown below.\n| Model               | AlpacaEval 2.0 | Arena-Hard | MT-Bench |\n|---------------------|----------------|------------|----------|\n| Nash-MD + LLaMA-3-SFT | 34.0           | 26.8       | 8.16     |\n| ONPO + LLaMA-3-SFT  | 48.6           | 36.4       | 8.40     |\n\nAs shown in the table, ONPO consistently outperforms Nash-MD across all three evaluation benchmarks.\n\n**DNO**. As stated in lines 224–232, the original DNO algorithm is not directly implementable in practice. Rosset et al. [2024] proposed a practical variant, DNO-Prct, which replaces the intractable update with the DPO objective. As a result, DNO-Prct can be interpreted as an iterative version of DPO and our experiments already include a comparison with Iterative DPO.\n\n**Online IPO**. There are two possible ways to implement online IPO. One is to apply the same loss transformation technique used in IPO and INPO; in this case, online IPO reduces to INPO without the KL stability term, which has been shown to underperform compared to INPO, see Table 3 in Zhang et al. [2024]. The other option is to use policy gradient methods to directly optimize the objective. However, as discussed in lines 215–223, policy gradient methods often suffer from high variance and can lead to unstable learning. Furthermore, we could not find any official policy gradient implementation of online IPO. We believe this is also why prior works have not included it in empirical comparisons.\n\nTaken together, we would like to emphasize that we have already compared ONPO with state-of-the-art preference alignment methods, including SPPO and INPO, demonstrating the effectiveness of our approach."}, {"review_id": "wAV8E7ZBnQ", "reviewer": "Reviewer_iMtJ", "summary": "This paper formulate LLM alignment as a two-player zero-sum game and elimate the assumption of Bradley-Terry model. The authors propose Optimistic Nash Policy Optimization (ONPO), which integrates optimistic online mirror descent into self-play to learn Nash policies. ONPO achieves improved theoretical guarantees with O(T^-1) duality gap convergence and demonstrates superior performance over state-of-the-art methods across multiple benchmarks.", "questions": "- Is it possible to implement the algorithm at the minibatch level? This could potentially lead to larger T, even if it introduces increased noise.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength：\n- The proposed method theoretically achieves O(1/T) convergence rate with solid theoretical guarantees.\n- The paper provides detailed comparison with similar methods, clearly articulating the differences and justifying the algorithm design.\n- Main experiments demonstrate the algorithm's effectiveness with notable improvements over baselines.\n\nWeaknesses：\n- The proposed method requires maintaining and training two models simultaneously, introducing additional computational overhead.\n- While the algorithm claims faster convergence, given the small T=3~5 in practice, it's unclear whether the theoretical analysis provides meaningful guidance for real applications (since these are just bounds).\n- The paper may need to supplement comparisons under equal computational budgets, as the algorithm's T=3~ 5 might correspond to baseline methods' T=6~8, suggesting that despite theoretically faster convergence, it may not be superior under limited computational resources.\n- \"Results where the baseline outperforms ONPO are underlined.\" This method of presenting results in Table 1 can easily lead to misunderstandings.", "was_revised": true, "final_justification": "I appreciate the authors’ response. Their discussion of related work has addressed my concerns. I will maintain my positive score.", "rebuttal": "We thank the reviewer for their valuable comments and respond to the weaknesses and questions below.\n\n**While the algorithm claims faster convergence, given the small T=3~5 in practice, it's unclear whether the theoretical analysis provides meaningful guidance for real applications (since these are just bounds).**\n\nWe would like to clarify that our theoretical bounds for both OMD and optimistic OMD share the same constant, and the improvement remains notable even for small values of $T$. For instance, at $T = 5$, the convergence rate improves from $1/\\sqrt{5}$ to $1/5$, yielding a 2× gain. While this improvement becomes more pronounced as $T$ increases, it is common in the RLHF and LLM alignment literature [Xie et al., 2024; Wu et al., 2024; Zhang et al., 2024] to develop theoretical analyses for relatively large $T$, even though practical experiments often use small values such as 3 or 5.\n\n\n**The paper may need to supplement comparisons under equal computational budgets, as the algorithm's T=3~ 5 might correspond to baseline methods' T=6~8, suggesting that despite theoretically faster convergence, it may not be superior under limited computational resources.**\n\nWe would like to clarify our online training setup, which follows from prior LLM alignment literature. In each iteration $t$, we construct the preference dataset $D_t$ by sampling a new set of prompts and generating responses using the current policy. Crucially, the prompt set changes across iterations. To ensure a fair comparison, we run all methods for the same number of iterations, ensuring that they are trained on the same set of prompts.\n\nRegarding computational cost, all baselines are trained for two epochs per iteration. In the Mistral experiments, ONPO uses the same total number of training epochs, one for $\\pi'\\_{t+1}$ and one for $\\pi_{t+1}$, resulting in the same computational cost. In the LLaMA experiments, we use two epochs for $\\pi'\\_{t+1}$ and one for $\\pi_{t+1}$, adding just one extra epoch per iteration. However, the majority of computation time per iteration (approximately 3–4 hours) is spent on generating the preference dataset, and the additional epoch only adds about 1.5 hours, which is relatively minor. Moreover, we experimented with training the baselines for three epochs per iteration but did not observe performance improvement. \n\n**The proposed method requires maintaining and training two models simultaneously, introducing additional computational overhead.**\n\nWe would like to clarify that $\\pi'\\_{t+1}$ and $\\pi_{t+1}$ are trained sequentially rather than simultaneously. After constructing the preference dataset $D_t$, we first minimize $g_t$ to obtain $\\pi'\\_{t+1}$, and then minimize $g_{t+1}$ to obtain $\\pi_{t+1}$. The log ratio of $\\pi'\\_{t+1}$ in the objective $g_{t+1}$ is fixed and precomputed. As a result, there is no need to maintain both models in memory at the same time, and the training does not introduce additional memory overhead from simultaneously handling two models.\n\n\n**Results where the baseline outperforms ONPO are underlined.\" This method of presenting results in Table 1 can easily lead to misunderstandings.**\n\nThank you for pointing this out. We agree that the current presentation could be misleading and will revise it in the next version.\n\n**Is it possible to implement the algorithm at the minibatch level? This could potentially lead to larger T, even if it introduces increased noise.**\n\nWe assume the reviewer is referring to using only a minibatch of prompts in each iteration to train the model. While this approach could reduce the computational cost per iteration and allow for a larger $T$, a small set of prompts may not provide sufficient training signal to accurately obtain $\\pi_{t+1}$, which serves as the target policy in mirror descent. Our current online training setup follows prior RLHF literature, which typically uses a large batch of prompts per iteration. That said, exploring a minibatch-level implementation is an interesting direction for future work."}]

Author rebuttal:

Meta-review: This work proposes a new RLHF method based on the optimism principle that outperforms several common benchmarks such as DPO. Theoretical analysis and experimental evaluation was sufficient to convince the reviewers of the merits of the proposed approach, to which I agree.
Decision: Accept (spotlight)