Title: Learning in Stackelberg Mean Field Games: A Non-Asymptotic Analysis
Venue: NeurIPS 2025
Accepted: yes
Keywords: Mean field game

Stackelberg game

reinforcement learning
Abstract: We study policy optimization in Stackelberg mean field games (MFGs), a hierarchical framework for modeling the strategic interaction between a single leader and an infinitely large population of homogeneous followers. The objective can be formulated as a structured bi-level optimization problem, in which the leader needs to learn a policy maximizing its reward, anticipating the response of the followers. Existing methods for solving these (and related) problems often rely on restrictive independence assumptions between the leader’s and followers’ objectives, use samples inefficiently due to nested-loop algorithm structure, and lack finite-time convergence guarantees. To address these limitations, we propose AC-SMFG, a single-loop actor-critic algorithm that operates on continuously generated Markovian samples. The algorithm alternates between (semi-)gradient updates for the leader, a representative follower, and the mean field, and is simple to implement in practice. We establish the finite-time and finite-sample convergence of the algorithm to a stationary point of the Stackelberg objective. To our knowledge, this is the first Stackelberg MFG algorithm with non-asymptotic convergence guarantees. Our key assumption is a "gradient alignment" condition, which requires that the full policy gradient of the leader can be approximated by a partial component of it, relaxing the existing leader-follower independence assumption.  Simulation results in a range of well-established economics environments demonstrate that AC-SMFG outperforms existing multi-agent and MFG learning baselines in policy quality and convergence speed.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nStackelberg mean field games (SFMGs) provide a model for studying settings in which a single \"leader\" takes an action and then \"follower\" agents react in a way modeled by a mean field capturing a collective response. This paper introduces AC-SMFG which is a single-loop actor critical algorithm that converges to a stationary point of a Stackelberg objective under a \"gradient alignment\" assumption that essentially implies the leader is able to improve their objective value by acting in response to the best response of the followers. This is a looser assumption than prior works which more so decouple the leaders and followers. Even with these looser assumptions, the authors establish the first non-asymptotic convergence guarantees for SMFGs. Additionally the authors provide numerical experiments demonstrating improved performance compared to baselines in mean field games such as market entrance and shop positioning.\n\nSTRENGTHS AND WEAKNESSES:\nThe authors provide compelling motivating examples and an introduction to Stackelberg mean field games. Algorithm 1 AC-SMFG is well-explained and presented. Their theoretical analysis is thorough, building all the necessary steps to demonstrate their theoretical contributions of obtaining non-asymptotic convergence guarantees for this setting. Additionally the authors are careful to compare their algorithm to rates that could be achieves by considering the policy optimization problem inherent to a SFMG as a bi-level optimization problem, showing that their convergence rate improves over the baseline comparable single-loop algorithm. This involves a key technical contribution regarding the MFG satisfying the PL condition with respect to the follower's softmax policy under regularization and discuss the potential individual interest of this technical contribution. This highlights the technical difficulty inherent to proving theoretical results of their algorithm. Additionally the authors provide relatively extensive numerical results in multiple MFG settings with detailed results and provided code. Their numerical results demonstrate faster leader convergence and smoother convergence trajectories compared to baselines. For each setting they compare to both the existing mean field approaches and a MARL approach, discussing and explaining the reasons for the relatively differences of how AC-SMFG compares in each setting. \n\nAs the authors mention in Section 1.1, many previous agent-based modeling approaches to settings such as SMFGs simulate agents under complex interactions which causes them to suffer from computational inefficiencies and lack of provable guarantees. But the tradeoff this paper makes is the homogeneity of the mean field response, which although inherent to such modeling approaches is a drawback when considering real world significance. Additionally, results that impose a certain learning algorithm for both parties do come with some questions as to applicability especially for some of the mentioned settings such as optimal liquidation.\n\nQUESTIONS:\nCould the authors provide an example of SMFG for which they expect the gradient alignment condition would not hold? To understand its limitations. \n\nCould the authors clarify whether followers would be incentivized to actually use this algorithm to update their policies or do there exist settings where this would not be in their best interest? \n\nThe authors make clear their advantages to previous agent-based modeling approaches, but could they discuss what they believe their drawbacks are compared to those approaches? What would be the line in practice between using AC-SMFG and such previous models?\n\nLIMITATIONS:\nLimitations are mostly addressed, lingering ones are related to my questions above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a learning algorithm for Stackelberg mean field games where a large number of followers follow a single leader. The proposed method is an actor-critic model that utilizes two trajectories to estimate the occupancy measure and the mean field distribution. Under the key gradient alignment that allows the improving gradients updates for the leader, the proposed method has a finite-sample convergence rate that improves upon the existing results for general bilevel optimization under convexity assumptions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n- Despite its technically dense content, the paper is written excellently. The existing methods, their weaknesses, and the proposed solution are explained well. \n- Under some assumptions (see the relevant weakness and question), the theoretical result improves upon the existing SMFG methods that only provide asymptotic convergence guarantees. Additionally, the proposed method improves upon the bilivel optimization methods that are applicable to the SMFG setting with additional assumptions. \n-Theoretical findings are supported by various numerical experiments.\n\nWeaknesses: \n-The problem setting has an unexplained addition compared to the existing formulations: The followers optimize an entropy regulated objective. To the best of my understanding, this term does not exist in the compared works (for example [1] and [2]). Furthermore, the followers are called ‘rational’, given that entropy-regularized objectives are very common in representing boundedly rational agents. I think that ‘rational’ does not accurately describe the followers considered in this paper. \n- The assumption on the gradient alignment may be required to derive the theoretical result. However, the authors should discuss in more detail when this assumption does not hold. I believe that when the leader and followers have conflicting objectives, for example a zero-sum setting, this assumption will break. \n\n\n[1] Cui, Kai, et al. \"Learning discrete-time major-minor mean field games.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 9. 2024.\n\n[2] Guo, Xin, Anran Hu, and Jiacheng Zhang. \"Optimization frameworks and sensitivity analysis of Stackelberg mean-field games.\" arXiv preprint arXiv:2210.04110 (2022).\n\nQUESTIONS:\n-Please explain the necessity of the entropy term. Is this term common in literature? Especially for the papers compared for the convergence rates. If the compared papers, does not have this term, I suggest the authors to make this an assumption or greatly emphasize that their setting is different than the others’.\n\n-When does the gradient alignment assumption hold? Is it possible to make a conclusion on the validity of the assumption based on the objective functions?\n\nLIMITATIONS:\nYes, but the discussion on limitations could be extended.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the strategy optimization problem in Stackelberg mean-field games (SMFG). The authors propose the AC-SMFG algorithm, a single-loop actor-critic algorithm that operates based on continuously generated Markov samples. Under a novel \"gradient alignment\" assumption, the paper not only proves that the algorithm converges to a stationary point of the Stackelberg objective at a rate of $\\tilde{O}(k^{-1/2})$ within finite time and finite samples but also validates its effectiveness through comprehensive numerical experiments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n（1）This work establishes the first non-asymptotic convergence guarantee for SMFG algorithms.\n（2）The AC-SMFG algorithm exhibits a significantly faster non-asymptotic convergence rate than previous SMFG algorithms, achieved under more challenging problem settings where the lower-level problem of the bi-level optimization is non-strongly convex.\n\nWeaknesses:\nThe authors should expand the discussion on the \"gradient alignment\" (Assumption 6). For instance, although $\\nabla_w \\Phi(\\phi_w)$ and $\\nabla_w J_l(\\phi_w, \\mu)_{\\mid\\mu=\\mu^* (\\phi_w)}$ differ by a derivative operation with respect to $\\mu^*(\\phi_w)$, it is necessary to analyze the impact of this difference in specific problem instances and whether the gradient alignment assumption still holds. As mentioned in the paper, the problem setting is more challenging than previous works. Thus, a paragraph discussing the techniques used in the convergence analysis should be added after Theorem 1 to highlight the improvements over prior research. The current description in the Introduction—i.e., developing an analytical argument to handle bi-level optimization with lower-level Polyak-Lojasiewicz (PL) condition—is too vague.\n\nQUESTIONS:\nCould the authors provide a paragraph introducing the techniques used in the convergence analysis of the AC-SMFG algorithm?\n\nLIMITATIONS:\nSee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of computing optimal leader policies in Stackelberg mean field games—an important framework for modeling strategic interactions in settings with a committed policy designer and a large population of followers, with applications in both public and private policy (e.g., economic platforms, government interventions).\n\nA central challenge in this setting is that each leader policy induces a new equilibrium in the follower mean field game, typically requiring a nested-loop optimization: the follower policy and mean field must converge before the leader can update. To address this, the paper adopts a single-loop algorithm that jointly updates the leader, follower, and mean field using carefully tuned step sizes, under appropriate assumptions on agent objectives. While the single-loop structure itself is not novel, the key technical contribution lies in relaxing the commonly assumed strong convexity of the follower’s objective. The proposed method is empirically evaluated against nested-loop baselines and MARL approaches across several benchmark mean field game environments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper tackles a timely and important problem in policy design, contributing an algorithmic improvement that is both technically sound and relevant.\n- The contribution is clear and the authors also provide example of settings that can be considered thanks to this technical generalization\n- The writing is clear, and the experimental evaluation is thoughtful, with careful analysis of the learned policies.\n\nWeaknesses:\n- The technical novelty of the proposed algorithm appears incremental in light of prior work by Hong et al. on single-loop methods.\n- While the experiments are well-executed, they are conducted in relatively simple environments and are restricted to tabular softmax policies, which limits the generalizability of the results.\n- The formulation assumes a well-posed leader problem, including the existence and uniqueness of the induced mean field equilibrium—guaranteed only through sufficiently large regularization weights. Although common in the literature, this assumption may limit the method’s practical applicability.\n\nQUESTIONS:\n- In the main text, you refer to the single-loop algorithms by Hong et al. as the foundation that your work generalizes. However, Appendix E presents examples that seem more directly related to the framework of Cui et al., whose assumptions appear more restrictive than those in Hong et al. I assume that extending the approach of Hong et al. to Stackelberg mean field games is non-trivial, and that your contribution is better viewed as a generalization of Cui et al.’s work—rather than Hong et al.’s. Could you kindly clarify this point?\n- I may have missed this in the paper, but could you elaborate on the rationale for using a tabular softmax policy parameterization? Is this choice required for the theoretical analysis, or was it made primarily for experimental convenience?\n- In the equilibrium pricing game, it appears that ADAGE outperforms your proposed method. Could you comment on why this might be the case, and whether all baselines—including ADAGE—are converging to equilibrium in your experiments?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}