Title: Last Iterate Convergence in Monotone Mean Field Games
Venue: NeurIPS 2025
Accepted: yes
Keywords: mean-field game

learning in games
Abstract: In the Lasry--Lions framework, Mean-Field Games (MFGs) model interactions among an infinite number of agents.
However, existing algorithms either require strict monotonicity or only guarantee the convergence of averaged iterates, as in Fictitious Play in continuous time.
We address this gap with the following theoretical result.
First, we prove that the last-iterated policy of a proximal-point (PP) update with KL regularization converges to an MFG equilibrium under non-strict monotonicity.
Second, we see that each PP update is equivalent to finding the equilibria of a KL-regularized MFG.
We then prove that this equilibrium can be found using mirror descent with an exponential last-iterate convergence rate.
Building on these insights, we propose the Approximate Proximal-Point ($\mathtt{APP}$) algorithm, which approximately implements the PP update via a small number of Mirror Descent steps.
Numerical experiments on standard benchmarks confirm that the $\mathtt{APP}$ algorithm reliably converges to the unregularized mean-field equilibrium without time-averaging.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider an algorithm for learning equilibria in mean field games with last iterate convergence, to obtain low exploitability during final stages of learning, as opposed to playing suboptimal during learning in methods such as fictitious play. The algorithm is supported by rigorous theoretical analysis of convergence, as well as empirically by performing a successful numerical experiment.\n\nSTRENGTHS AND WEAKNESSES:\nThe claims of last-iterate convergence are supported by theoretical results, as well as experimentally in one example problem. The benchmark Beach Bar problem makes sense for the field of learning MF equilibria. The experiments are limited and could be extended to show the limitations of the proposed methodology beyond the Beach Bar example, beyond regularized MFGs or even beyond the theoretical assumptions made in the work. In particular, there is no MF interaction in the dynamics, which is a common theoretical assumption but might be circumvented in practice / experiments.\n\nThe contributions would seem relevant, to consider last iterate convergence in more general monotone settings, as opposed to existing works that require averaging after playing as in fictitious play or considering special cases such as graphon MFGs. However,  without application to a data-driven setting with N-agent systems and without access to the model, it seems although LIC is achieved, model knowledge and exact calculation of Q-functions is necessary. In that case, I am not sure of the advantage of LIC over any other convergence in existing literature, which could be compared more clearly.\n\nThe clarity could be improved by stating concisely the differences to existing theoretical results and algorithms, whenever a new result is introduced, e.g., Thm. 4.3 and Thm 4.4. Some things remained unclear for me, see the questions section below.\n\nMinor things:\n- I think [1] is also relevant to add to Table 1, as it has discrete-time convergence results for MFGs (under concavity assumptions)\n- Typo line 671: \"for Here\"\n\n[1] Geist, Matthieu, et al. \"Concave Utility Reinforcement Learning: The Mean-field Game Viewpoint.\" Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems. 2022.\n\nQUESTIONS:\n- Where does Figure 1 come from? Is it from the experiment further below?\n- I think the explanation on line 203 is unclear, how does Lemma E.7 provide the upper bound of exploitability? Can this be made more formal?\n- The convergence result in Thm. 4.4 given in $D_{\\mu^*}$, how does this imply exponential convergence in a standard metric for the policy, or in exploitability?\n- Why is (3.1) called intractable? If it is just a regularized MFG, can you not solve it using any regularized MFG algorithm?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the challenge of achieving last-iterate convergence (LIC) in Mean-Field Games (MFGs), especially under the more realistic assumption of non-strict monotonicity. The last-iterated policy of a proximal-point (PP) update, which incorporates Kullback–Leibler (KL) regularization, is shown to converge to an MFG equilibrium even under non-strict monotonicity. Each PP update step is shown to be equivalent to finding the equilibria of a KL-regularized MFG, and this regularized equilibrium can be found efficiently using Regularized Mirror Descent (RMD) with an exponential last-iterate convergence rate. Building on these theoretical insights, the paper proposes the Approximate Proximal-Point (APP) algorithm. APP approximates the PP update by executing a small number of RMD steps. Numerical experiments on standard benchmarks, such as the Beach Bar Process, show that APP reliably converges to the un-regularized mean-field equilibrium without time-averaging.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper directly tackles the crucial problem of LIC in MFGs, which is relevant for deep learning applications where time-averaged policies can be less meaningful due to nonlinearity in parameter space.\n2. Establishing convergence under non-strict monotonicity enhances the applicability in practice.\n3. Establishing exponential convergence rate for RMD is a useful result, improving previous results that achieved only polynomial or time-averaged convergence.\n4. The Proximal-Point (PP) method, with un-linearized objective, is expected to be less sensitive to approximation error and more robust in non-strictly monotone settings compared to standard Mirror Descent (MD).\n5. Proposition C.1 is interesting.\n\nWeaknesses:\n1. The model is assumed to be known, and naturally no sample complexity analyses like the most recent references. A more realistic setting would have been the case where the parameters are learned from the data. \n2. A couple of very strong assumptions in relation to the latest literature on MFGs (see Questions)\n3. It is not clear how the results are placed in the literature when considered in conjunction with the rather `outdated' assumptions used.\n\nQUESTIONS:\n1. How do the \"optimization-based methods for MFGs\" (e.g., Guo et al. 2024) that achieve local convergence without monotonicity compare in practical utility to the \"global\" convergence results achieved here with monotonicity?\n2. The assumption of ergodicity is too strong. Zaman et al. 2023, for example, establish results under weaker class of communicating MDPs. How would the results presented in the paper generalize to such MDPs?\n3. The assumption of mean-field independent MDP is also strong for MFGs (Zaman et.al 2023, Zeng et.al 2024, etc.) consider the case where $P$ depends on $\\mu$. How easy is it to relax this assumption in the paper?\n4. Can the authors elaborate why the initial policy~$\\pi_0$ should have full support in Prop. C.1?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a monotone mean field game without strict monotonicity. It proposes a proximal point-like algorithm that can solve the regularized version of the MFG exponentially fast. Numerical experiments also validate the proposed algorithm empirically.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is nicely presented with sufficient explanation and insights into how the algorithm is developed. \n\nStrength: the proximal point like algorithm is very well explained with figures to aid the understanding of it. The intuition from the continuous time version of the algorithm also help motivates and bridge the connection to previous work on continuous time mean field algorithms. \n\nWeakness: My main concern is that the exponential convergence rate is not new to regularized games (see Cen et al 2023, Zhang et al 2023, Dong et al 2025). To my knowledge, these papers also use a proximal point-like algorithm. From my brief skim of the proof, I cannot tell the technical novelty of the analysis and I am not sure if it could be straightforwardly extended from the existing analysis.\n\nI am not fully convinced by the contribution of this work because the proposed proximal-point–like algorithm does not appear significantly different from the extragradient-type algorithm used in Cen et al. (2023). Although the authors argue that the main challenge in extending the analysis of Cen et al. (2023) lies in the backward induction of the Q-function (Remark 4.6), this issue is already well-known in monotone mean-field games and has been addressed in Zhang et al. 2023 (Section F) and Dong et al. 2025 (Lemma B.1). \n\nOne key difference between the two work lies on that Dong et al uses the naive mirror descent algorithm, while this one uses the proximal point like algorithm. Combining bandit feedback is difficult with the proximal point algorithm, so I would have expected that the improvement on the convergence speed stem from here. Similar reasoning can be applied to Zhang et al. One improvement of the results in Theorem 4.3 is that it improves the terms in Equation E.1 that are due to the exploration with bandit feedback (Lemma I.3).\nBut we only care about the full information feedback, the exponential speedup brought by the proximal point like algorithm, combined with the speed up from not having to explore,  is not too surprising an this combination is known to be able to do that in regular non-mean-field monotone game (Cen et al).\n\nQUESTIONS:\nCould you explain the technical novelty of the analysis? It seems like the improvement over Zhang et al is on the non-strict monotonicity assumption. But Dong et al's results are also under non-strict monotonicity. \n\nCould the poly-> exp improvement on the convergence rate be attributed to different feedback types as Dong et al focused on bandit feedback?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel algorithmic framework for achieving last-iterate convergence in mean-field games under the assumption of non-strict monotonicity. Current methods often require stricter assumptions like contractivity or strict monotonicity, or only guarantee the convergence of time-averaged policies. The authors' approach is twofold. First, they introduce a Proximal Point method with KL-divergence regularization. They provide a proof that the last iterate of this method asymptotically converges to a mean-field equilibrium, a result established without requiring strict monotonicity. Second, they observe that each update of this PP method is equivalent to finding the equilibrium of a KL-regularized MFG. They then show that this regularized subproblem can be solved by the Regularized Mirror Descent  algorithm with an exponential last-iterate convergence rate, which is a significant improvement over previous polynomial-rate or time-averaged results. Based on these theoretical insights, the paper proposes the Approximate Proximal-Point algorithm, which uses a finite number of RMD steps to approximate the PP update, and demonstrates its empirical convergence on a standard MFG benchmark.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1.  **Addresses the Problem Under Weaker Assumptions:** The paper makes a contribution by tackling last-iterate convergence under non-strict monotonicity. This is a more general and practical assumption than the strict monotonicity or contractivity required by much of the prior work, and it correctly includes important cases, such as games with symmetric transitions, where strict monotonicity fails.\n\n2.  **Theoretical Improvements:** The paper delivers two important theoretical results. The first is the asymptotic last-iterate convergence of the PP method (Theorem 3.1) in this setting. The second, and perhaps more impactful, is the exponential convergence rate established for RMD in solving the regularized MFG subproblems (Theorem 4.3), which improves upon previous results that showed polynomial or time-averaged convergence.\n\n3.  **Well-Motivated and Elegant Algorithm Design:** The proposed APP algorithm is a well-motivated and practical method. The insight to frame the problem as approximating an intractable PP update by using RMD to solve a sequence of regularized MFGs is elegant.\n\n**Weaknesses:**\n\n1.  **Lack of Convergence Guarantee for the Proposed APP Algorithm:** While the paper provides convergence guarantees for the idealized PP method (asymptotic) and the RMD subproblem solver (exponential), it does not provide a formal convergence guarantee or rate for the final, practical APP algorithm (Algorithm 1). APP approximates the PP update with a finite number of RMD steps, introducing an error that is not theoretically analyzed. The authors state that deriving a convergence rate for APP is a conjecture and a future task.\n\n2.  **Reliance on a Model-Based Setting:** The entire analysis is conducted in a model-based setting, where the transition kernels and reward functions are assumed to be known. This assumption limits the direct applicability of the theoretical results to many reinforcement learning scenarios where the agent must learn from data without a model of the environment.\n\nQUESTIONS:\n1.  The APP algorithm's performance seems dependent on $\\tau$, the number of inner RMD iterations. The experiments show a difference in performance between $\\tau=10$ and $\\tau=1000$. What is the primary theoretical obstacle to deriving a convergence guarantee for the combined APP algorithm? How does the approximation error from using a finite $\\tau$ affect the stability and convergence of the outer PP loop?\n2.  Could the authors elaborate on the technical challenges specific to MFGs that complicate the convergence analysis? The paper correctly points out that the mean-field dependency, where the policy iterate $\\pi^k$ influences its own update through the mean-field $\\mu^k = m[\\pi^k]$, is a key difficulty not present in standard N-player games. Could you provide more intuition on how the use of the Łojasiewicz inequality (for the PP proof) and the detailed value function analysis (for the RMD proof) were tailored to overcome this specific challenge?\n3. Could the authors discuss how the RMD differentiates from that in zero-sum game (see e.g. [1] [2] ). Moreover, if the feedbacks are potentially delayed or asynchronous, could the authors give intuition on how to adapt the algorithms non-trivially in these scenarios (see e.g. [3])? A discussion clarifying the relationship to these works would further strengthen the paper's positioning.\n3.  How should one select the regularization parameter $\\lambda$ and the number of inner iterations $\\tau$ in practice? The parameter $\\lambda$ controls the gap between the regularized and unregularized equilibria, while $\\tau$ controls the accuracy of the PP update. Is there a theoretical relationship between these parameters that would guarantee convergence for the overall APP algorithm?\n\n\n\n**Numerical Scores:**\n\n* **Quality & Technical Soundness:** 4\n* **Clarity:** 3\n* **Originality & Significance:** 3\n* **Overall Score:** 4\n* **Confidence:** 4\n\n\n\n\n\n[1] Cen, S., Wei, Y., & Chi, Y. (2021). Fast policy extragradient methods for competitive games with entropy regularization. Advances in Neural Information Processing Systems, 34, 27952-27964.\n\n[2] Park, C., Zhang, K., & Ozdaglar, A. (2023). Multi-player zero-sum markov games with networked separable interactions. Advances in Neural Information Processing Systems, 36, 37354-37369.\n\n[3] Ao, R., Cen, S., & Chi, Y. (2023, January). Asynchronous Gradient Play in Zero-Sum Multi-agent Games. In International Conference on Learning Representations (ICLR).\n\nLIMITATIONS:\nThe authors provide a good discussion of the paper's limitations in Section 6, correctly identifying the model-based setting, the focus on iteration complexity over per-iteration computational cost, and the lack of analysis with function approximation. The most significant limitation, which is mentioned as a future goal but not framed as a limitation, is the absence of a convergence proof for the main APP algorithm. Moreover, it would be more concrete to discuss the relationship between the RMD in the mean-field game with that in Zero-sum Games and its adaptivity to the asynchronous multi-player scenarios, as mentioned before. A more direct discussion of the challenges involved in bridging this theoretical gap would strengthen the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to understand convergence guarantees for monotone finite-horizon MFGs. Existing results have some shortcomings, such as suboptimal rates, assumptions of strict monotonicity, not providing last-iterate convergence, and requiring regularization. This work proves a linear convergence bound for monotone MFGs for regularized MD, improving upon past work. This result is suggested to imply convergence to non-regularized NE by showing that it asmpyotically approximates an implicit proximal point update rule. Experimental evaluations are provided.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nI think the analysis of the work is interesting and potentially a good contribution to the understanding of MFGs and MF reinforcement learning. It is a clean result that is also meaningful for many practical problems.\n\nThe paper is well-written, and the theoretical results are easy to follow. The commentary on the theoretical developments compared to existing work on MFGs was helpful.\n\n## Weaknesses:\n\nI list some concerns below, I am happy to discuss further following the responses of the authors.\n\nFrom the theory side, one major concern is the convergence rate that has been proved. The exponential convergence of Theorem 4.3 matches intuitively the ODE rate, though in discrete time the fraction is $\\left(1 - \\frac{\\lambda\\eta}{2}\\right)$, where $\\eta < \\eta^{\\*}$. Learning rate $\\eta^\\*$ defined in D.5 determines the exponential convergence rate, and $\\eta^\\*$ can be very small. By tracing the results and constants, it seems $\\eta^\\*$ will scale with $e^{-H\\lambda^{-1}}$ and $(-\\log \\sigma_{min})^{-1}$, which might be extremely small. In this regard, while the convergence result is indeed linear, it is difficult to compare it with the sublinear rate of Zhang et al. 2023. Furthermore, since the convergence of APP is not known, one would typically set $\\lambda$ to be small, which exponentially worsens the linear rate.\n\nRelated to this, at some point in the main text, mentioning the dependence of $\\eta^*$ on the problem parameters might be helpful, at least asymptotically. It is currently somewhat difficult to understand how the rate of linear convergence changes depending on the problem parameters such as $|\\mathcal{S}|, |\\mathcal{A}|, H, \\sigma_{min}$.\n\nRegarding the presentation of the results, while I like the exposition, it was not clear if the PP method had implications beyond being a theoretically illuminating tool. As far as I understand, in the current state of the work the convergence of APP is not known, although MD seems to asymptotically approximate PP.\n\nFinally, I am not fully convinced regarding the strict monotonicity assumption. Looking at the work due to Zhang et al. 2023, it seems that the strict monotonicity assumption is not required for the convergence analysis, but rather for the uniqueness of the regularised equilibrium. Does Theorem 4.3 not also implicitly assume that there exists a unique equilibrium given by $\\bar{\\omega}^{\\*}$?\n\nMinor points:\n\n- potential formatting issue at Figure 1\n- Difficult to read axis labels in Figure 2-a\n\nQUESTIONS:\n- Could assumption 2.3 also be generalized to the case where the reward function $r$ depends on the mean-field flow over states and actions? That is, when $\\mu_h^\\pi(s,a)$ is the population flow on states and actions induced by policy $\\pi$. In some MFGs, the population's actions distribution is also significant.\n\n- Theorem 4.3 and correspondingly the update rule (4.4) tackle a similar setting as in Zhang et al., 2024. However, while their analysis only yields sublinear convergence,\n\n- The continuous time analysis of Perolat et al., 2021 is more straightforward than the analysis of the work, although it does not yield the more important discrete time convergence. I am curious to what extent the discretization of the ODE (corresponding to the (O)MD update) proposed by Perolate et al. can be used to still obtain a discrete time result. Clearly, in this work, the learning rate $\\eta$ is quite small, so the results might already be in the regime where an ODE is discretized. This is already somewhat suggested by Theorem 4.4, however, could the analysis directly incorporate the ODE up to a discretization error? To what extent does the analysis differ from standard ODE discretization methods that are used in optimisation literature?\n\n- The paper comes quite close to deriving a rate for the unregularized problem via APP and the RMD convergence. What is the main bottleneck that prevents merging the results to obtain a novel convergence result?\n\n- What are the major bottlenecks when generalizing the results to graphon MFGs?\n\nLIMITATIONS:\nDiscussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}