Title: Last-Iterate Convergence of Smooth Regret Matching$^+$ Variants in Learning Nash Equilibria
Venue: NeurIPS 2025
Accepted: yes
Keywords: Regret Matching$^+$

Last-Iterate Convergence

Nash Equilibrium
Abstract: Regret Matching$^+$ (RM$^+$) variants are widely used to build superhuman Poker AIs, yet few studies investigate their last-iterate convergence in learning a Nash equilibrium (NE). Although their last-iterate convergence is established for games satisfying the Minty Variational Inequality (MVI), no studies have demonstrated that these algorithms achieve such convergence in the broader class of games satisfying the weak MVI. A key challenge in proving last-iterate convergence for RM$^+$ variants in games satisfying the weak MVI is that even if the game's loss gradient satisfies the weak MVI, RM$^+$ variants operate on a transformed loss feedback which does not satisfy the weak MVI. To provide last-iterate convergence for RM$^+$ variants, we introduce a concise yet novel proof paradigm that involves: (i) transforming an RM$^+$ variant into an Online Mirror Descent (OMD) instance that updates within the original strategy space of the game to recover the weak MVI, and (ii) showing last-iterate convergence by proving the distance between accumulated regrets converges to zero via the recovered weak MVI of the feedback. Inspired by our proof paradigm, we propose Smooth Optimistic Gradient Based RM$^+$ (SOGRM$^+$) and show that it achieves last-iterate and finite-time best-iterate convergence in learning an NE of games satisfying the weak MVI, the weakest condition among all known RM$^+$ variants. Experiments show that SOGRM$^+$ significantly outperforms other algorithms. Our code is available at https://github.com/menglinjian/NeurIPS-2025-SOGRM.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper study the convergence rates of Smooth PRM+ and ExRM+ under the assumption that the games satisfy the Minty condition, providing new results pertaining to *iterate* convergence, going beyond previously obtained results in Cai et al. 2025 pertaining to *duality gap* convergence. It also introduces a new algorithm (SOGRM+) that also converges under the weak Minty condition.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:** convergence of an algorithm in a new setting (weak MVI)  \n\n**Weaknesses:** \n- weak MVI vs. MVI: is the weak MVI condition an interesting setting? Can the authors provide some actual applications that satisfy weak MVI but not MVI, beyond the simple 2 player non-zero sum game from section G.2?\n- I find their version of OMD quite confusing; it is quite heavy notationally, and the authors fail to convince me that this framework is necessary (instead of the usual OMD formulation).\n- The paper is not well written, and I am not sure what can be done in that regards. Several sentences are hard to read and to understand. To me this is a crucial issue. Top conferences should only accept well-written papers. Some examples: \n\nlines 63 - 65: “the feedback is the loss gradient of the original games, satisfying the weak MVI, implying the weak MVI is recovered. By using the recovered weak MVI, […]”\n\nLines 109: “it ensures that the weak MVI can be used after the weak MVI is recovered,”\n\nl161 : “No-regret algorithms are the algorithms, which ensures the regret R = [XXX] to grow sublinearly”\n\nl272: “However, they do not achieve such convergences in games satisfying the weak MVI, covering games satisfying the MVI”\n\n* \n* To my understanding, none of the “technology” used in this paper is new. The main methodological tool (RM+ to OMD reduction, in the same space) is the same as Liu et al 2021. The convergence based on tangent residuals follows from the analysis in [0]. The authors mention obtaining the best-iterate convergence rates as one of their contributions. But as they themselves acknowledge in lines 88-100, this follows from the analysis in Cai et al. 25, verbatim.\n\n  [0] Yang Cai, Argyris Oikonomou, andWeiqiang Zheng. Finite-time last-iterate convergence for learning in multi-player games. In Proceedings of the 35th International Conference on Neural Information Processing Systems, volume 35, pages 33904–33919, 2022.\n\n* l252: this is the proof of what statement? \n* l754: broken references\n\nQUESTIONS:\n* Can the idea of the authors be extended beyond smoothness?\n* Cai et al. 2025 also obtain rates using a restarting algorithms, but no last-iterate convergence rates for ExRM+ and SPRM+. You also don’t give rates. Do you think that this is an inherent limitations of your analysis (and the one in Cai et al.) ?\n* I would recommend the authors to devote less space to equations in the main body and explain more the intuition behind their algorithms.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-06-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies several variants of smooth regret matching+ variants for learning NE in a class of games that satisfy MVIs. In the case of SExRM+ and SPRM+, two known algorithms from the literature, the authors provide last and best-iterate convergence results to the set of NE in games with the MVI property. Furthermore, a variant of RM+ called Smooth Optimistic Gradient-Based (SOGRM+) is proposed, and is shown to converge in the last and best-iterate sense to the set of NE in weak MVI games, a superset of MVI games. The key technique used to prove these results is showing that these RM+ variants can be transformed into instances of a general online mirror descent paradigm, which allows the authors to use tools from OMD literature to derive the convergence results.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper shows strong last iterate convergence results for a broader class of smooth games than was previously known, alongside the introduction of SOGRM+. The method of transforming the RM+ variants into an instance of OMD is useful and could be of independent interest. Overall, I find this paper introduces some useful ideas and does a good job substantiating them with both theory and experiments.\n\nSome weaknesses: the overall presentation of the paper is quite cluttered. The flow of the paper could be improved if some of the details from Section 5 are moved to the appendix and SOGRM+ is instead given more priority, given that the proof technique seems to be the same and the class of games studied is wider.\n\n_Edit: I erroneously mentioned that the main algorithm was left to the appendix, but actually the SOGRM+ update rule is in the main text. Apologies for that. However, my point that the paper is cluttered remains true, and a pass for formatting would greatly improve the readability of the paper._\n\nOne potential weakness in the results is that in Cai et al 2025, they show last/best iterate convergence in iterates (i.e. pointwise convergence), whereas the results in this paper pertain to convergence to the set of NE (in the case of MVI games). This seems to indicate that the results of Cai et al 2025 are stronger in the case of last-iterate convergence, since they also show this via the duality gap rather than the tangent residual. I feel a more comprehensive discussion and comparison of the results of this paper in relation to Cai et al 2025 would be helpful to clarify these points, and to emphasize that the best-iterate results of this paper are indeed new.\n\nOverall, while the paper has some issues with clarity of exposition, I believe the proof idea and usage of tangent residuals is interesting and thus would consider this paper a borderline accept.\n\nQUESTIONS:\n- In [1], they used the tangent residual not only as a proxy for the distance to NE, but also as a method to create a potential function for the dynamics, thus showing last iterate convergence. Would it be possible to extend similar ideas to the set of games with the MVI/weak MVI property (i.e., deriving a potential function as a function of the tangent residual)? More broadly, what properties of the tangent residual can be studied to give more results compared to the standard duality gap?\n\n- In the experiments section of Appendix I, it is stated that all existing works on RM+ variants focus on simultaneous updates. However, in Cai et al 2025 it is shown that alternating RM+ may cycle instead of converging to the NE. I would be curious to know if your proof technique can be used to study alternating variants SExRM+/SPRM+/SOGRM+ in games with MVI/weak MVI and potentially show last/best iterate convergence.\n\n[1] Cai, Yang, Argyris Oikonomou, and Weiqiang Zheng. \"Finite-time last-iterate convergence for learning in multi-player games.\" Advances in Neural Information Processing Systems 35 (2022): 33904-33919.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

comments: SUMMARY:
This paper investigates Regret Matching like algorithms in the context of Minty Variational Inequalities and weak Minty variational inequality.
They start considering a new analysis in the MVI setting of algorithm that already appeared in the literature like Smooth Extra-gradient RM+ (SExRM+) and Smooth Predictive RM+ (SPRM+). The new analysis crucially relies on a new reduction from RM+ to OMD whioch allows to show last iterate convergence to Nash exploiting the variational characterization of it.
Of particular importance in the reduction is that the feedback used by the algorithm satisfies the MVI condition.

Using this new reduction, the authors introduce. a new algorithm Smooth Optimistic Gradient Based Regret Matching+ (SOGRM+) which is shown to achieve last iterate convergence in the weak MVI setting for large enough $\rho$.Moreover a finite time rate is proven for the best iterate.

STRENGTHS AND WEAKNESSES:
***Strengths***
The paper is well written and easy to follow. It is nice that you could include proof sketches in the main text.

***Weaknesses***

Nothing major, but the minimum $\rho$ depends on the inverse of the maximum number of actions $D$ and on $L_u$. The authors should comment on the fact that the range tolerated by other algorithms is bigger up to $\rho > -1/2$ ( see for example https://openreview.net/pdf?id=Y7slJZPGCy ).

QUESTIONS:
In equation 6, how do you make sure that h is differentiable in $x^{t_2}_i$ and $x^{t_0}_i$ ?

Can you please relate more with the existing literature in the weak MVI setting. It would be useful to include a table with other existing results in particular to compare the range of $\rho$ ?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the last-iterate convergence properties of Regret-Matching+ (RM+) based algorithms in games. Recent works have established that in two-player zero-sum matrix games (1) the non-convergence of several popular variants, including RM+, alternating RM+, and PRM+; (2) last-iterate convergence of several smooth variants of RM+, including ExRM+ and SPRM+, where the decision set is truncated to ensure Lipschitzness of the regret operator. This paper's main contribution is to extend the convergence results of ExRM+ and SPRM+ to games satisfying the MVI condition, a class of games that encompasses two-player zero-sum games as a special case. This paper presents a new analysis framework based on (1) an equivalence between the RM+ type updates on the truncated positive orthant and the OMD type update on the simplex; (2) a proximity measure to Nash equilibria called tangent residual that upper bounds the duality gap. Using the new analysis framework, this work also generalizes the Optimistic Gradient (OG) algorithm to an RM+ based algorithm called SOGRM+, which has best-iterate convergence rates and asymptotic convergence to the set of Nash equilibria even in games with a weak MVI condition.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n1. This work studies the convergence properties of regret matching (RM)-type algorithms. This is an important question since regret matching (RM)-type algorithms (integrated with the CFR framework) are popular in practical game-solving. Although the average-iterate convergence rates of RM-type algorithms have been extensively studied, their last-iterate convergence properties have only been studied recently, and results are limited to the two-player zero-sum games setting. This work extends the convergence properties of the recently proposed smoothed RM+ variants, including ExRM+ and SPRM+, from two-player zero-sum games to general concave games that satisfy the MVI condition. \n2. This paper proposes an interesting proof paradigm that is based on (1) an equivalence between the RM+ type updates on the truncated positive orthant and the OMD type update on the simplex; (2) a proximity measure to Nash equilibria called tangent residual that upper bounds the duality gap. The main difficulty for analyzing RM-type algorithms is that the regret operator, unlike the gradient operator, may not be monotone or even satisfy MVI. The new proof paradigm overcomes this difficulty by translating the RM updates back to OMD updates on the simplex; thus, nice properties of the gradient operator can be used. Moreover, this proof paradigm is quite flexible, as the paper also extends another OMD-type algorithm, the Optimistic Gradient (OG) algorithm, to an RM-type algorithm called SOGRM+. The original analysis of OG can be reused to prove that SOGRM+ converges under weak MVI. \n\nWeakness:\n1. This paper's presentation could be improved. One particular example is the proof paradigm, which is the main technical contribution of the paper. It is claimed that the smooth RM+ variant can be transformed into an OMD instance, and an equivalence of the two updates is presented. However, in the main body,  there is no further explanation on why the equivalence holds or the intuition behind the equivalence. The OMD in equation (5) is also non-standard as it has both $f$ and $h$, making it unclear why (5) is equivalent to (4). Since this proof paradigm lays the foundation for subsequent analysis, a more detailed discussion would be helpful.\n\nQUESTIONS:\n1. Can you provide some intuition behind the equivalence between (4) and (5)? \n2. It is mentioned as future work that \"Future directions involve integrating our proof paradigm with other OMD algorithms, such as Reflected Gradient [Hsieh et al., 2019] and composite Fast Extra-Gradient [Cai et al.,2024], to develop novel RM+ variants that achieve stronger last-iterate convergence.\" Why does the current proof paradigm not directly extend to the Reflected Gradient or the composite Fast Extra-Gradient algorithm? Is it possible to use the equivalence between (4) and (5) to give RM variants of these two algorithms? If not, what might be the obstacles?\n\n\nMinor comment:\n1. \"As shown in Meng et, al. [2025], the tangent residual is as an upper bound for the duality gap, $\\dots$\". In this sentence, \"as\" is redundant. Besides, the fact that tangent residual upper bounds the duality gap is proved in Lemma 2 in [Cai et. al., 2022], where they refer to the duality gap as the total gap function.\n\nLIMITATIONS:\nYes (partially). The authors provide the assumptions of their theoretical results, but do not give an explicit discussion on the limitations of their approach.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a systematic method for proving convergence (both last-iterate and best-iterate rate) of smooth Regret Matching+ variants to the set of Nash equilibria in smooth concave games satisfying the weak Minty Variational Inequality (w-MVI). They leverage this method to prove convergence of two known algorithms under MVI, and of a newly-proposed algorithm under w-MVI. Experiments show that the newly-proposed algorithm outperforms existing ones.\n\nMore specifically:\n- The w-MVI -- the weakest condition among those assumed to prove convergence of RM+ variants -- loosely means that the game's loss gradient \"points sufficiently away\" from the equilibrium. The key challenge in leveraging the (weak) MVI to prove convergence is that RM+ variants update in (a subset of) the cone of original strategies, taking as input a *transformed* loss gradient which does not fulfil the (weak) MVI. The question then becomes how to recover the (weak) MVI.\n\n- To do so the authors recast a smooth RM+ instance as a generalised Online Mirror Descent instance updating in the strategy space itself and operating on the original loss gradient of the game, which does fulfil the (weak) MVI (Eqs. 4-5).\n\n- This reformulation allows to upper-bound a certain notion of distance between a strategy generated by a smooth RM+ variant and the set of Nash equilibria by the distance between consecutive iterates generated by the algorithm (Eq. 6).\n\n- The (weak)-MVI is then leveraged to show that said distance between consecutive iterates generated by the algorithm converges to zero for sufficiently small learning rates (Theorems 5.2, 5.3, 6.2).\n\n- This allows to prove last-iterate convergence and best-iterate convergence rate for existing RM+ variants under MVI (Theorem 5.1), and for the newly-proposed algorithm under w-MVI (Theorem 6.1).\n\nAn important characteristic of the newly-proposed algorithm is that it performs a single prox-map call per iteration (in the auxiliary step), while the main update step relies on a simple sum between the auxiliary iterate and the difference between transformed losses (Eq. 12).\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- investigating the convergence to Nash equilibria under the weak MVI is a challenging and worthy research project to pursue. The proposed algorithm is the first RM+ variant to achieve both last-iterate and finite-time best-iterate convergence to the set of NE under the weak Minty variational inequality.\n- experiments show that the proposed algorithm significantly outperforms existing ones across a variety of settings; displays a relatively low sensitivity to hyper-parameters; and allows for a relatively big learning rate.\n- the proposed proof paradigm seems flexible and powerful; it can be employed to show both last-iterate convergence and best-iterate convergence rate across a variety of RM+ settings. It stems from a novel combination of existing techniques, and it lends itself to fruitful applications from the community.\n- the authors exhibit a fine understanding of the problem, and accurately cite the related existing literature upon which their work is based.\n- the claims are technically sound and well-supported by detailed proofs.\n- the paper is well structured and organised. It provides in a nutshell a zoom-in from the setting of online convex optimization to that of smooth RM+ variants that can bring on board also a reader unfamiliar with the literature; it presents the proof paradigm in a concise way; and it applies it effectively.\n\n\n### Weaknesses\n- The main update step of the proposed algorithm (second line of Eq. 12) would benefit from further motivation / intuition / discussion (cf question below)\n- the authors do not address possible weaknesses or limitations of their setting\n- there are throughout a few minor typos and syntactical issues\n\nQUESTIONS:\n### Major\n- You do a lot of work to recast a RM+ instance as an OMD instance, where the (weak) MVI holds. Why not work with OMD in the first place then?\n- I find it interesting that the equivalence between a RM+ variant and an OMD variant can be established only for the generalized OMD of Joulani et al. and Liu et al. (Eq. 1), with a time-dependent Bregman term and an additional time-dependent regularization term. Why is this the case? More specifically:\n\t- what can you say about RM+ variants under other regularizers than the quadratic one? (line 194)\n\t- could the time-dependent $f_i$ and $h_i$ terms in eq. 5 (generalized OMD) be effectively captured by a fixed, non-quadratic regularizer in a standard OMD setting, or is their time-dependence crucial in establishing the equivalence with RM+?\n\t- In which sense are exactly the updates given by Eqs. 4 and 5 *equivalent*? I have not gone through the details of Appendix A, but it seems reasonable that $(4) \\implies (5)$. However, I do not see how it can be $(5) \\implies (4)$, since $(5)$ depends on the iterates generated by $(4)$ (as one cannot recover univocally $\\theta_i^t$ from $x_i^t$). Can you elaborate on this?\n- Can you provide further motivation / intuition / discussion around the design of the main update step of your algorithm (second line of eq 12)? In particular, it might be useful to add in the body a brief pointer to appendix I and J, where you conjecture that the relatively larger $\\eta$ allowed by your algorithm is related to the single prox-map call per step.\n- Can you see a generalization of your framework to smooth games played on generic convex compact sets (as opposed to the particular case of the simplex)?\n- Can you provide a counter-example of a game not fulfilling the w-MVI, in which your algorithm does not converge?\n\n\n### Minor\n- line 119: inconsistency on the dimension of $\\mathcal{X}_i$\n- l. 124, add comment on the fact that the upper bound $P$ on losses comes wlog\n- l. 128, consider making explicit mention to the Stampacchia variational inequality\n- l. 138, consider giving a more classical reference for monotone games, as [1]\n- l. 141: monotone games always admit a unique NE [1, 2]\n- l. 146: \"we provide an example...\" seems to fit better in the following paragraph\n- l. 679: the game admits also bottom-right -- $((0,1), (0,1))$  --  as pure, non-strict NE. \n- The argument \"in games satisfying the (weak) MVI the feedback of RM+ does not satisfy the (weak) MVI\" appears throughout the paper a few times (e.g. lines 6-9, 53-55, ...). The phrasing is not immediately clear at a first read; consider rephrasing it, for example along the lines of \"even if the game's loss satisfies the (weak) MVI, RM+ operates on a *transformed* loss which does not satisfy it\", or analogue.\n- a few minor typos and syntactical issues (e.g. *\"is the players\"* line 121,  *\"at sometimes\"* lines  801, 843, *\"such that that ensures\"* line 305, etc. )\n\n---\n[1] Rosen JB (1965) Existence and Uniqueness of Equilibrium Points for Concave N-Person Games. _Econometrica_ 33(3):520–534.\n\n[2] Mertikopoulos P, Zhou Z (2019) Learning in games with continuous action sets and unknown payoff functions. _Math. Program._ 173(1–2):465–507.\n\nLIMITATIONS:\n- The authors acknowledge the fact that the notion of convergence employed assures to reach the set of Nash equilibria - where the iterates may wander rather than settling at a specific point (Caption of Table 1)\n- The authors are encouraged to highlight further limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}