Title: Improving LLM General Preference Alignment via Optimistic Online Mirror Descent
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM Alignment

RLHF
Abstract: Reinforcement learning from human feedback (RLHF) has demonstrated remarkable effectiveness in aligning large language models (LLMs) with human preferences. Many existing alignment approaches rely on the Bradley-Terry (BT) model assumption, which assumes the existence of a ground-truth reward for each prompt-response pair. However, this assumption can be overly restrictive when modeling complex human preferences. In this paper, we drop the BT model assumption and study LLM alignment under general preferences, formulated as a two-player game. Drawing on theoretical insights from learning in games, we integrate optimistic online mirror descent into our alignment framework to approximate the Nash policy. Theoretically, we demonstrate that our approach achieves an $\mathcal{O}(T^{-1})$ bound on the duality gap, improving upon the previous $\mathcal{O}(T^{-1/2})$ result. Meanwhile, it enjoys a linear convergence rate in the last iterate, a property not achieved by previous methods. More importantly, we implement our method and show through experiments that it outperforms state-of-the-art RLHF algorithms across multiple representative benchmarks.


Peer reviews:
comments: SUMMARY:
This paper formulate LLM alignment as a two-player zero-sum game and elimate the assumption of Bradley-Terry model. The authors propose Optimistic Nash Policy Optimization (ONPO), which integrates optimistic online mirror descent into self-play to learn Nash policies. ONPO achieves improved theoretical guarantees with O(T^-1) duality gap convergence and demonstrates superior performance over state-of-the-art methods across multiple benchmarks.

STRENGTHS AND WEAKNESSES:
Strength：
- The proposed method theoretically achieves O(1/T) convergence rate with solid theoretical guarantees.
- The paper provides detailed comparison with similar methods, clearly articulating the differences and justifying the algorithm design.
- Main experiments demonstrate the algorithm's effectiveness with notable improvements over baselines.

Weaknesses：
- The proposed method requires maintaining and training two models simultaneously, introducing additional computational overhead.
- While the algorithm claims faster convergence, given the small T=3~5 in practice, it's unclear whether the theoretical analysis provides meaningful guidance for real applications (since these are just bounds).
- The paper may need to supplement comparisons under equal computational budgets, as the algorithm's T=3~ 5 might correspond to baseline methods' T=6~8, suggesting that despite theoretically faster convergence, it may not be superior under limited computational resources.
- "Results where the baseline outperforms ONPO are underlined." This method of presenting results in Table 1 can easily lead to misunderstandings.

QUESTIONS:
- Is it possible to implement the algorithm at the minibatch level? This could potentially lead to larger T, even if it introduces increased noise.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Optimistic Nash Policy Optimization (ONPO), a novel online algorithm designed for aligning large language models with general human preferences beyond the restrictive Bradley-Terry assumption. By framing the preference alignment task as a two-player zero-sum game, ONPO leverages optimistic online mirror descent (OMD) to approximate the Nash equilibrium more efficiently. Theoretically, ONPO improves the convergence rate on the duality gap from the previously established and ensures linear convergence in the final iterate. Empirically, ONPO demonstrates superior performance compared to existing methods like iterative DPO, SPPO, and INPO across standard benchmarks including AlpacaEval 2.0, Arena-Hard, and MT-Bench, effectively maintaining model capabilities on reasoning and calibration tasks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n* **Theoretical Advances**: The paper proposes Optimistic Nash Policy Optimization (ONPO), an innovative integration of optimistic online mirror descent (OMD) into LLM alignment. This method notably improves the convergence rate on the duality gap from $O(T^{-1/2})$ to $O(T^{-1})$, a significant theoretical advance.\n\n* **Strong Empirical Results**: Empirical evaluations demonstrate consistent improvements over state-of-the-art alignment methods (Iterative DPO, SPPO, INPO) across prominent benchmarks (AlpacaEval 2.0, Arena-Hard, MT-Bench). Notably, ONPO achieves substantial improvements on AlpacaEval 2.0, outperforming baseline methods even when compared to significantly larger models.\n\n### Weaknesses\n\n* **Lack of Statistical Significance Reporting**: The paper presents experimental results without reporting standard deviations or confidence intervals, casting doubt on the statistical significance of the observed improvements. Without this information, it is challenging to determine whether differences between methods are statistically meaningful or simply the result of random variance.\n\n* **Missing Empirical Comparison with Closely Related Methods**: Although the discussion section thoroughly situates ONPO among related algorithms like Nash-MD, DNO, and Online IPO, these methods are not included in the empirical comparison. This omission weakens the paper’s claim of superiority, especially since these algorithms are based on similar game-theoretic or general preference formulations.\n\nQUESTIONS:\nSee weaknesses. I am willing to raise my score if my concerns are addressed.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of aligning large language models (LLMs) with human preferences without relying on the restrictive Bradley-Terry (BT) model assumption. The authors formulate LLM alignment as a two-player zero-sum game and propose Optimistic Nash Policy Optimization (ONPO), which integrates optimistic online mirror descent into a self-play framework. The authors obtain an improved theoretical convergence guarantees of $\\mathcal{O}\\left(\\frac{1}{T}\\right)$ duality gap bound versus previous $\\mathcal{O}\\left(\\frac{1}{\\sqrt{T}}\\right)$ results. The authors also prove a linear convergence in the last iterate which is a novel result for this setup. Finally the authors provide strong empirical performance across multiple benchmarks including AlpacaEval 2.0, Arena-Hard, and MT-Bench when the other models have same/comparable number of parameters.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.The paper identifies a key limitation of existing RLHF methods that rely on BT assumptions and provides evidence from prior literature that challenges its validity in modeling human preferences.\n2. Achieves state of the art convergence results for the duality gap as well as linear convergence rate for the last iterate. The latter is something that is novel.\n3. Dropping the Bradley-Terry model assumption is well-justified. The authors provide compelling evidence that human preferences often violate transitivity assumptions, especially at population levels, making the general preference setting more realistic.\n\nWeakness: \n1. While performance improvements against other models have been established, they have been done so for models of size 8B/7B. the question as to whether the same performance improvement would be had if the model sizes are scaled up to larger sizes remains open.\n2. Their is an assumption of a preference oracle here. Is that a realistic assumption? It appears not to be the case but I am open to the authors explaining why it might be.\n3. The self play here is done for a single turn. While I am not very familiar with the practical aspects of RLHF this seems significant.\n\nQUESTIONS:\nHow much more computational resources would be needed for a multi turn setup?\n\nLIMITATIONS:\nThe assumption of an oracle and single turn setup seem to be the only limitations I can think of. However, for a proof of concept paper these seem to be reasonable assumptions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for preference alignment of LLMs by formulating the problem as a two-player game via game theory. This integrates online mirror descent into an alignment framework to approximate the Nash policy. The goal is to leverage the faster convergence properties of online mirror descent in the preference learning use case.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n- Paper aims to integrate classical mirror descent with preference learning. This is an interesting and novel approach. \n- Optimistic Nash Policy Optimization improving the convergence rate on the duality gap with additional guarantees is interesting and non-trivial. The theoretical contribution does integrate well with the paper, and shows understanding of learning with general preferences beyond the Bradley-Terry framework. \n- Compares against DPO\n\nWeaknesses: \n- Only compares against DPO and 2 minor methods. Other methods in this area (SimPO, ORPO, R-DPO, RL-based methods etc) are not explored. \n- No discussion of experimentation setup. Hardware, training time, codebase, etc. Makes it hard to assess practical trade-offs between methods. \n- Method assumes access to a general preference oracle (possibly learned), but does not consider the effect of noisy/imperfect oracle feedback. Since human preferences can be inconsistent, this is a critical.\n\nQUESTIONS:\n- Even without full implementation, demonstrating ONPO on multi-turn with toy examples would help validate its broader application and scope. \n- Runtime and efficiency comparisons. Training time, GPU setup, convergence speed, against more relevant baselines listed above. This will help validate the claim of \"computational lightness\". \n- Add discussion/experiments considering the possibility of a noisy oracle, to try and model how this will behave in real world.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}