Learning in Stackelberg Mean Field Games: A Non-Asymptotic Analysis
Authors: Sihan Zeng
Benjamin Patrick Evans
Sujay Bhatt
Leo Ardon
Sumitra Ganesh
Alec Koppel
Venue: NeurIPS 2025 poster ()
Keywords: Mean field game
Stackelberg game
reinforcement learning
Primary area: theory

Abstract: We study policy optimization in Stackelberg mean field games (MFGs), a hierarchical framework for modeling the strategic interaction between a single leader and an infinitely large population of homogeneous followers. The objective can be formulated as a structured bi-level optimization problem, in which the leader needs to learn a policy maximizing its reward, anticipating the response of the followers. Existing methods for solving these (and related) problems often rely on restrictive independence assumptions between the leader’s and followers’ objectives, use samples inefficiently due to nested-loop algorithm structure, and lack finite-time convergence guarantees. To address these limitations, we propose AC-SMFG, a single-loop actor-critic algorithm that operates on continuously generated Markovian samples. The algorithm alternates between (semi-)gradient updates for the leader, a representative follower, and the mean field, and is simple to implement in practice. We establish the finite-time and finite-sample convergence of the algorithm to a stationary point of the Stackelberg objective. To our knowledge, this is the first Stackelberg MFG algorithm with non-asymptotic convergence guarantees. Our key assumption is a "gradient alignment" condition, which requires that the full policy gradient of the leader can be approximated by a partial component of it, relaxing the existing leader-follower independence assumption.  Simulation results in a range of well-established economics environments demonstrate that AC-SMFG outperforms existing multi-agent and MFG learning baselines in policy quality and convergence speed.

Learning in Stackelberg Mean Field Games: A Non-Asymptotic Analysis
Sihan Zeng', Benjamin Patrick Evans", Sujay Bhatt', Leo Ardon',
Sumitra Ganesh' Alec Koppel'
J.P.Mon AI Research, United States
2 J.P.Morgan AI Research, United Kingdom
(sihan. ben jamin evans, beajay.bhat,, leo ardon, sumitra.ganesi. depc.oopelie (empppeei@pparaaan
jpmorgan com
Abstract
We study policy optimization in Stackelberg mean field games (MFGs), a hierarchical framework for modeling the strategic interaction between single leader and an infinitely large population of homogeneous followers. The objective be formulated as structured bi-level optimization problem, in which the leader needs to learn a policy maximizing its reward, anticipating the response of the followers. Existing methods for solving these (and related) problems often rely on restrictive independence assumptions between the leader's 's and followers' objectives. use samples inefficiently due nested-loop algorithm structure, and lack finite-time convergence guarantees. To address these limitations, we propose AC-SMFG, a single-loop actor-critic algorithm that operates on continuously generated Markovian samples. The algorithm alternates between (semi-)gradiee) updates for the leader, a representative follower, and the mean field, and is simple to implement in practice. We establish the finite-time and sample convergence of the algorithm objective. To our knowledge, this is the first Stackelberg MFG algorithm with non-asymptotic convergence guarantees. Our key assumption alignment" condition, which requires that the full policy gradient of the leader can approximated by a partial component it, relaxing the cxisting independence assumption. Simulation results in range of well-established economics environments demonstrate that AC-SMFG outperforms existing multi-agent and MFG learning baselines in policy quality and convergence speed.
stationary point of the Stackelberg
"gradient
Introduction
Mean field games (MFGs) provide a framework for studying the strategic interaction among an infinite number of rational agents, with wide range of applications in resource allocation [Li et al.
2020 telecommunication [Narasimha et al.] 2019), and power system optimization Alasseur et al. 2020 An important extension of MFGs is Stackelberg mean field games (SMFGs), incorporating a hierarchical structure where a single leader agent influences population of follower agents and enjoys first-mover advantage. A prominent example of SMFG optimal liquidation [Almgren and Chriss 1997]. in which a large institutional investor (leader) strategically sells assets, e.g., central bank treasuries, in market while accounting for the reactions of smaller traders (followers) who adjust their behavior, e.g., bond purchasing, in response [Chen et al. 2024]. Other examples arise in public policy domains such as taxation, mortgage regulation, and epidemic control [Zheng al.) 20200 Mi et al. 2024] Aurell et al. 2022]. where the leader, typically government, designs interventions to promote social welfare while anticipating and influencing the collective response of the population.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Despite the increasing number of empirical studies applying the SMFG framework to real-world problems, there remains a lack of practical and theoretically grounded algorithms for learning in such hierarchical environments, The central challenge lies in the inherently coupled dynamics among the leader, representative follower, and the mean field representing the collective behavior of the follower population. This interdependence gives rise to a complex bi-level structure: the leader's s objective depends on the equilibrium response of the follower population, which in turn is shaped by the leader's policy. Existing approaches [Dayaııkı and Laurière, 2024 Cui et al. (0244 to SMFGs (and the closely related major-minor MFGs) need to make strong assumptions to decouple the leader and followers, lack non-asymptotic guarantees for finding solution, and empirically exhibit slow convergence or tend towards sub-optimal limit points, possibly due to cyclic behaviors. In contrast, our work relaxes the assumptions and establishes a fast finite- sample rate of convergence for simple actor-critic algorithm to the stationary point of the SMFG objective.
1.1 Main Contributions
Convergence Rates
10a
We introduce AC-SMFG, a single-loop actor-critic al- gorithm for SMFGs which operates under continuously generated Markovian samples. Our first key contribution is to characterize the finite-time and finite-sample complexity of AC-SMFG, We achieve this through multi-time-scall analysis, where the updates of the leader's policy, the follower's policy, and the mean field are coordinated on distinct time scales via properly chosen step sizes. We show that AC-SMFG converges to stationary point of the Stackelberg objective with rate O(k-)/2), under new assumption termed "gradient alignment". Intuitively, the assumption allows the leader to improve its objective by acting to the best response from the followers. As the algorithm draws exactly two samples in each iteration, this translates to sample complexity of the same order. This is the first time that non-asymptotic convergence guarantees for any SMFG algorithm, and also the first time that a single-loop SMFG algorithm is analyzed.
10-1
Residual
20
10-3
Proposed: oit/v
S--ecel Converity [Basier probiem):
002/k*%)
Existing SHFO Method: (rate unimown)
Iterations (k)
Figure 1: Theoretical convergence rates. The proposed algorithm the first to have non-asymptotic convergence guaran- tees (black). Existing algorithms in sim- ilar settings are only known to converge asymptotically (red). Convergence rate of single-loop algorithm for bi-level opti-
established
mization [Hong et al.) [003) under a non-
convex upper-level objective and lower-
Notably, our convergence rate surpasses that of a com- parable single-loop algorithm for bi-level optimization under lower-level strong convexity [Hong et al.) [0033]
level strong convexity (blue).
(Ō(k-1/3) vs optimization, where the upper-level objective non-convex and the lower-level problem is solve standard MFG that does not exhibit any convexity. The reason that we achieve a better rate in a more challenging setting is two-fold. First, as a key technical innovation, we develop an analytical argument to handle bi-level optimization with lower-level Dolyak-Lojasiewice (PL) condition, which the MFG objective satisfies with respect to the follower's softmax policy parameter under regularization [Mei] et al. 2020). This technique allows us to obtain the same convergence rate in SMFGs as in bi-level optimization with lower-level strong convexity, and may be of independent interest and applicable to general bi-le optimization methods. Second, we make the rate improvement by incorporating the latest advances in multi-time-sce stochastic approximation [Han et al. 2024). leveraging the smoothness condition of the reward and transition.
The policy optimization problem in a SMFG can be viewed as bi-level
We support the merit of proposed method through numerical simulations. We apply AC-SMFG to a range of environments inspired by real-world economic scenarios involving a prominent leader and a population of followers. Previous agent-based modeling (ABM) approaches these problems simu- late large number of agents with complex interactions, leading to computational inefficiency and lack of theoretical guarantees [Evans et al., 2025). In contrast, we show that the SMFG formulation offers a more tractable alternative by summarizing the collective follower behavior through the mean field, thereby enabling faster computation of (possibly local) optimum for the leader's policy. Compared to the existing SMFG algorithms, AC-SMFG exhibits a significantly faster convergence rate.

---

1.2 Related Literature on Stackelberg Mean Field Games
(i) Continuous-time setting: Djete 2023] is among the first works to study SMFGs and to provide characterization of the problem structure. Building on this, Dayamıkh and Laurière [0024] in- troduces a penalty-based approach that reformulates a SMFG as a single-level mean field control problem, offering a conceptual simplification. However, this approach does not come with a provably convergent learning algorithm. leaving open the question of how to efficiently and reliably solve SMFGs. (ii) Discrete-time setting: [Guo al. 2022) consider
setup, and propose a minimax optimization framework (known model) for finding the policy of the leader considering worst-case objective for the followers. Cui al. [2024] consider finding an equilibrium in (related, but distinct setup) of major-minor MFGs and proposes a nested-loop algorithm based on fictitious play, which approximately alternates best-response updates between the leader and the follower. Such a nested-loop structure often poses practical challenges, as it requires the inner loop (best response computation) to converge before each outer-loop update. This results in increased computational burden and reduced flexibility, particularly in large-scale or online settings where full convergence at each iteration may be infeasible or undesirable. Moreover, Cui et al.. [204] establishes only asymptotic convergence, offering limited insight into the algorithm's 's performance in practice. Compared to the related literature diseussed above, our paper is the first work to propose a single-loop algorithm for learning equilibria in SMFGs, supported by non-asymptotic analysis.
Stackelberg Mean Field Game: Formulation
We formulate a discrete-time infinite-horizon discounted-reward SMFG between a leader and an infinite population of homogeneous rational followers. The leader's state and action spaces are St and B, while a representative follower's 's state and action spaces are Sy and A. The state transition of the leader depends not only on its own action, but also the aggregate behavior of the follower population. Such an aggregate behavior is denoted by μ E As,. where As, denotes the probability simplex over Sy. We use P: S x B x As, - As, to denote the transition kernel for the leader's state (s) st. b) denotes the probability that the next state is s when the leader takes action b under mean field µ in state st. Similarly, for the follower, we use (5 [8y, a, 6) to denote the probability that the next follower's state is s'y when the follower takes action and the leader takes action b under mean field μ in state sy. We define a product state space S S x Sy. For notational simplicity, we do not distinguish the leader's and follower's state in the rest of the paper and assume that S is the common state space observed and shared by both the leader and the representative follower.
In this way, we can compactly represent SMFG by the tuple AS B, P.ry. Tt, 7), where P x A x B As + As is the transition kernel and y E (0, 1) is the discount factor. The reward function of a representative follower is ry S x A x B x As - [0, 1]. with the understanding here that S is restricted to Sy. Similarly, the reward of the leader rı : S x B x As + [0, 1] does not depend on the state of the representative follower sy, but only on the mean field. The state and action spaces of all players is assumed to be finite to enable the analysis, but continuous space approximations are studied experimentally in Section 5
Followers' Interaction. The followers play mean field game in response to a policy Ø S - Ag of the leader. As in a standard MFG, let the representative follower take actions according to a randomized policy æ S AA- Given a leader-follower policy pair the sequentially generated states form Markov chain, for which the transition matrix is pT. e RIS)x(S] The matrix is entry-wise expressed as pr. |)) = Ea PM 8, s, ()x(( | (b | s). We denote by مو. the stationary distribution of the Markov chain, which is the singular vector of pr associated with the (only) largest singular value "I" (under an ergodicity assumption). Let the discounted visitatiion/occuaneyy measure d5 d..0... A Eso (d d. H) under the initial state distribution P. where
(Ø, and mean field jr,
de. S. 4 A (1- EE 7 10ss =s) 100 s),
where the expectation is taken over
ag ~ x[. sx), by ~ 06- (x), sk41 ~ PM sk, ax, bb).
3

---

Under (0, H.). the follower expects to collect the following (regularized) cumulative reward
J m) - dd [ 7tth (Ty(Sk, aus by, p)) 7 log क (ax sx)) so ~ p]
1-7 E smaz o.e, am)( ( bro))) (r
(1)
where the entropy function F(TT, 8) Σ. (aa 8) logz(a 8), and regularization weight 7 2 0. Define the follower's policy as π A x As + A such that
(6. n4) - argmax (,), Aμc
Let μ² Ag + As denote the mapping from the leader's policy to the mean field induced by the leader. It is known from the literature on standard MFGs that μ (0) satisfies for ø
" (Ø) nz "0.a" (+.""()) 0. (0)
(3)
With some abuse of notation, we write
(6) x (0, (0,44((0) (6)),
4))
Fixing the leader's policy to & reduces the environment to standard MFG among the followers. Using the terminology from the MFG literature, we refer to (x (0), x*))))(()))) (Ø)) as the mean field equilibrium (MFE) for such MFG. We will later impose an assumption which guarantees that x (Ø) and μ (Ø) are unique and well-defined.
Leader's Interaction and Game Objective. field μr, the leader's cumulative reward is
Given a leader's policy & and the follower's mean
Ji pa) Ex D kmo (s bu. 14) 80 p] - Bndd
snud. 6. (-) b, p)]- (5)
If the mean field were fixed, the aim of the leader would be to find policy & that maximizes Jı(ő.') pi). However, the stable mean field changes with & as the followers try best respond the leader and each other. To solve SMFG is to find an optimal policy for the leader, given that the followers best respond. We define (0)) = Ji(Ø, μr (Ø)) for all & and express the objective as follows
argina DEAS (()) = argmax&e4$ ((0,u(())
3 Algorithm
The algorithm developed in this work based on the principle of independent learning. To motivate our approach, consider a scenario where oracle knowledge is available on n² (Φ) for any ø. In this case, we can optimize iteratively with gradient descent - we maintain da (where is the iteration index) and update dh in the direction of the gradient ( which can be evaluated using μ ((x)).
In practice, oracle access to 4² (0x) is unavailable, and we must solve the lower-level MFG to approximate it. To this end, we introduce the iterates T& and üůş as estimates of I (Øxx and μ²' (Øx). and refine them via Jgradint descent. This naturally leads to nested-loop structure, where the follower policy and mean field are updated to convergence in the inner loop to support the outer-loop gradient computation. Our approach eliminates this nested structure: we perform alternating updates of the leader, follower, and mean field in a single loop, using appropriately chosen step sizes to implicitly approximate a nested loop. The updates are formally stated in Algorithm [
We use tabular softmax policy parameterization and maintain 0 E R| R|S| that encode the policies according to
cxp(w(s, b))
exp(f(s,a))
de (b18)
aa(a = Enees exp(O(s,
Exe
The tabular softmax parameterization is considered for the purpose of mathematical analysis optimizing over the space of softmax parameters allows us to exploit the PI. structure that each player's objective observes with respect to its parameter [Mei et al. 2020). In practice, the proposed algorithm can be applied with function approximations such as neural networks, discussed in detail

---

in Remark 1999].
We can express the policy gradients of w and & in the closed form below [Sutton et al.
V VuJi( (b) = ET (1r((s,b,,) + (V ()) V. log @w 8s)
(Td 6, - b1 ( VV (s') V Vo log mo(as))
The policy updates (7). (8) of Algorithm arr exactly based on the policy gradient expressions above, substituting in the latest leader's policy parameter wk, follower's policy parameter 0x. and mean field iterate ūgg. Here the value functions are defined as
(s0 =s) y (s, b, so ( log s)) su=s)
-0- V for policy gradient evaluation. Since these value functions are not directly available, we estimate them by Ví. and VI.A. which are updated via temporal difference (TD) learning (TO). Note that to ensure stability we leverage projections in (9) and (TO:: IIas denotes the projection to the probability simplex over the state space, and II
Algorithm/ requires access to v. .Px and
denotes the element projection to the interval [0, By], where By = 1/(1 - 7) + r log AA.
Our method exemplifies independent learning in the sense that wk., 8g. fu are each updated to optimize its own objective without any knowledge of the other two, though the samples used for the updates are from an environment determined by all variables. We note that the algorithm is single-loop and uses two trajectories of continuously generated samples" making significantly more practical than the existing methods with complex nested loops [Dayaıklı and Lauriere 2024
Cui et al. 2024
Algorithm 1 Single loop Actor-Critic Algorithm for Stackelberg Mean Field Games (AC-SMFG)
1: Initialize: leader's policy parameter wn, follower's policy parameter 9g value function estimates
Vi,, Vy,0, mean field estimate io As, initial state s0, 8g p. step sizes Ca &N S Or.. Sx.
2: for iteration k = 0, 1, 2,... do
3: Sample path I for tracking the discounted occupancy measure: Follower and leader take actions a* Te, (- 8t), by ~ Owr( sk), receive rewards
r (Sk, agg, by. (.)), and observe the next state according to the transition probability
8h+1 ~
P: (. Sk, ag, (g,bs); with probability y
with probability 1 1-7
4: Sample path 2 for tracking the stationary distribution (mean field): Follower and leader take actions äg ~ Te. 5x), by own (- 5R), and observe the next state 5k+1 PP (- 5g, ag. bx)
5: Leader's policy update:
Wk+1 We + Ce Va log dun (bx su) (Fi(Sk, bu, jux) + V(,0(5411) V(L((1))
(7)
6: Follower's policc (actor) update:
Oc+1 +a Volog Tay (ax (u)
(s))s()(((s( (r5(5k, an, bb, ) +T sn) + V
7: Mean field update:
Au+1 ITs (axx Ex ixx)) 8: Value function (critic) update:
(9)
V1,++1 = II (Vi,e Bres, (Fe(#n, be, (ax) + (.. ( Vi. (sa))
(10)
- (Vi,tt (Vess ax br. (xx) + EETT,, 34) + 7V -Vyx(ss)))
9: end for
'The second trajectory of samples is used solely to estimate the mean field and may be eliminated in applications where there exists generative model/oracle for the mean field.
5

---

4 Convergence Analysis
In this section, we establish the convergence of AC-SMFG. We first introduce the main technical assumptions, which can be segregated into the standard ones (Assumptions [ required for finite-sample analysis of standard MFGs, and the additional regularity assumption, Assumption 6 (termed gradient alignment)), required to deal with the more challenging setting of SMFGs. To illustrate that the gradient alignment assumption weaker than those in the existing literature, we provide a non-trivial example that satisfies our assumption but not the stronger ones in prior works.
Assumption (Lipschitz and Smooth Transition and Reward) There exist bounded constants Lp, by such that for all π, Φb P1, (42, 8,
pp pT || & Lpllm1 p22 I paidu - V PaPao S &Tlli - Hell: (ri - (s b. p2)l S L, | u2ll, Iry(s, a, b, - (s a, p2)1 S L pell. | (s, un) - Vari(s, b, p2)ll - L 1r H2ll: v a, b,, - Vurs(s, b, p2) S 4, pell.
The Lipschitz continuity is common assumption made in the literature MFGs (Cui and Koeppl 20216 Zeng et al. 2025| Cui et al. 2024). We additionally assume that they are smooth.
Assumption (Lipschitz Best Response) The best response operators j¹ and x¹ are Lipschitz and μ has Lipschitz gradients, i.e. there exists a constant L E [1, 00) such for all o, &' P.,
l| (0) 1² (6') || S L|l ø'Il, || (e. 14) H" (o', x')ll L(ll l'll + || IlI))-
|V (Ø) Von" (ø') Lll &'11,
(1I)
The condition (11) in fact can be shown to follow from the Lipschitz continuity and smoothness of the transition and reward function in Assumption T
We impose Assumption ddrrectly for simplicity.
Assumption (Exploration) There exists a constant Pmin > 0 such that the initial state distribution satisfies p(s) 2 Puii, Vs E S. In addition, the follower's policy iterates are uniformly bounded away from zero, Le. there exists a constant Pmii such that no, (a s) 2 Puin for all 0, S.a A.
The first part of Assumption 3 implies that the discounted occupancy measure d= d is bounded away from zero for any π, o. fr and is a standard assumption in RL [Agarwal et al. 2021 Mei et al. 2020], while the second part can be readily satisfied by capping the norm of O2 away from infinity. The assumption guarantee sufficient exploration of every state and action pair.
Assumption (Contraction) There exists a constant E (0, 1) such that for any Φ H
I (0.01):0.11 n² (0- || < 61|l1 p2ll-
(12)
Assumption states that in a standard MFG determined by any fixed Φ an "optimality operator contraction condition" holds, which implies the uniqueness of the MFE. This is a key
assumption made in the existing literature on MFGs [Xie et al. 2021 Zaman et al. 2023 Yardim
et al. 2023 and can be guaranteed when the regularization weight large enough.
Assumption (Uniform Geometric Ergodicity) For any π, &, p. the Markov chain { generated by p according to 8141 ~ p c, (- st) is irreducible and aperiodic. In addition, we have sup, drv (P((: = Su = s), v. I CoC Vt 1t20 0,
for some constants Co 2 1 and C E (0, 1), where drv denotes the total variation distance.
Assumption Dis is again standard in analyzing stochastic approximation and RL algorithms under Markovian samples (Zou et al. 2019 Wu et al. 2020 Wang et al. 2024
Assumption (Gradient Alignment) There exist positive bounded constants nh, 72 such that Vo m 4(ow), Vwni(ow 4) (o=)) T VV. Weni 1)) (a=n((>)) 2 ||V 11) (o=a) 112.
2 (
(13)
(14)

---

Note that V. Jo (4) (dw))((w)) is partial component of the full gradient V.. The condition states that the two gradients always make an acute angle. Without loss of generality, we assume ?,, 1/2 2 1, as if the conditions hold with 7),, '/2 < 1 they also hold with 7), 7/2 = 1. Conceptually, the condition allows the leader to improve its objective by acting to the best response from the followers. Prior works on SMFGs and the closely related major-minor MFGs need to make assumptions of a sim- ilar nature, but more restrictive. For example, Cui et al. [024] by their Assumptions 4.c) and 4.d) as- sumes that the leader's s reward is independent of the mean field and that the follower's reward and tran- sition are independent of the leader's action. Under such conditions, the leader and follower are effec- tively decoupled, reducing the hierarchical structure of the problem contrast, with the gradient alignment assumption we permit our leader to depend on j4, and follower re- wards and transitions to depend on b. In Appendix [ present non- coittiivial MFG which satisfies the gradient alignment assumption but not the leader-follower independence condition in Cui et al. [2024]
near- near-independent setting. In
4.1 Algorithm Complexity
Each variable in Algorithm | converges in the sense that an associated residual (measure of sub- optimality gap) decays to zero. The residual of the leader is the squared policy gradient norm. The mean field convergence is measured by its deviation from the (unique) equilibrium of a standard MFG induced by leader's latest policy iterate. The convergence the follower is assessed by the objective function gap relative to the best response against Tr: fag- Finally, the value function estimates are evaluated by their &2 distance to the true value functions under the latest policy and mean field iterates. The proof of Theorem relies analyzing the convergence of all variables jointly, through a coupled Lyapunov function.
# ( 1I3 EX = JJ (dw:: Pa), Own Px)) Jy( Pwn -w)) The ,wn RK 111
(15)
#x |iA 4 (o EY, IIVi, vi Re 1u1,, E/, IIVV,,
V
Theorem Consider the iterates of Algorithm 7 under the step sizes
CC 1)1/2
CE
C3
Sk
ag (k 1)
Ek (k: + 111/21/2
Bk
(k: + 1)1/2
with the properly selected constants ct, Cos ce2 CB. Under Assumptions
we have for all 0.
min E t<k
(K (k+1)
As the residuals are all non-negative, Theeremmiimplicn that the best iterate of Algorithm [ converges to stationary point of the leader's objective with rate O(k-1/2), where Ô hides constants and logarithmic factors of k. As exactly two samples are drawn in each iteration, this translates to a sample complexity of the same order. To our knowledge, this is the first result providing finite- mean field games. In contrast
time and sample complexity for learning in Stackelberg/majo-nniin to the most relevant prior work (Cui et al.) 2024), which establishes the asymptotic convergence of a nested-loop fictitious play method, our paper analyzes a single-loop algorithm and provides non-asymptotic guarantees. This rate improves upon that of a two-time-scale algorithm for bi-level optimization with a non-convex upper-level objective and lower-level strong convexity (Hong et al.. 2023]. which achieves ((1/22/5). Our improvement is obtained in a more challenging setting the lower-level problem is not strongly convex function but a MFG involving two coupled variables: the follower's policy and the mean field. The MFG does not exhibit any convex structure, and only non-uniform PL condition holds with respect to the follower's ''s policy under regularization. A key technical innovation made in our work is to handle such non-convexity, which we sketch in Remark 2 and which we believe is of independent interest and applicable to the analysis of general bi-level optimization methods with lover-level PL condition.
Note that our analysis relies on sufficiently large regularization weight 7 to ensure that Assumption 4 is satisfied. As a result, our method does not find the equilibrium of the original (unregularized) problem. In fact, solving unregularized MFGs even the standard, non oon-hierarc setting remains an open problem. Our work inherits this fundamental limitation.
Remark known in the standard RI. literature that the cumulative reward observes a "gradient domination" " condition with respect to the policy parameter. Using the notation of our paper, it means
7

---

that any stationary point (where V J. = 0) is globally optimal under a fixed μ. We are not able to extend the argument here and have to settle for convergence to stationary point, because & is a composite function involving pº and hence does not observe the gradient domination condition.
Two techniques allow us to achieve the convergence rate of ((k-)12), surpassing that in Hong et al. [2023] under lower-level strong convexity.
Remark 2
To analyze a bi-level optimization algorithm, we need to bound the residual of the follower's 's policy and show its iteration-wis reduction. If the lower-level objective J were strongly convex, we would select the residual to be EK = =I|TT, n (owy, D12 and show that Ek+1 - E is approximately negutive. As the learning larget shifts from iteration k to + 1, bounding Ek+1 &g requires
controlling the drift 1|=" (0unt1: Pk+1)) - π (owe fxx) which can be easily bounded by
O1 - &we 12 ll - mxx1?)
(16)
under the Lipschitz continuity of æ² However, without strong convexity, we cannot consider the residual Ex (as x² may not be unique in general) and instead employ eg defined in (15).
a residual in the value function space. We would like to bound the learning target shift J (0 ( Owesis Px+1) - -J ( ( Own Suurite ftx), preferably still by (T) to avoid rate deterioration. However, the Lipschitz continuity of Jj only yields a loose bound on the order of Ollowwe Own. + |l
II), where the norms are not squared. Detailed in Proposition 3 (proof in Section C.3). we develop a careful error decomposition scheme to break down the residual difference e + Eye and lightly bound the decomposed terms based on the PL. condition, thereby avoiding the suboptimal bound implied by naive application of the Lipschitz property of Jy-
The technique highlighted above allows us to achieve the same convergence rate in SMFGs as in bi-level optimization with lower-level strong convexity. We further improve upon the rate in Hong et al.) (2023y, leveraging a recent advance in the multi-time-scale stochastic approximation literature. Specifically, Han al. [2024] shows that a faster convergence rate can be achieved under lower-level strong convexity, if the lower-level learning target (equivalent to operators μ², a in our context) has Lipschitz gradients. Our work adapts and extends the argument to the case where the lower-level
objective is nonconvex.
5 Experiments
Proposed
OneByOne
Culkopgel
ADAGE
JANY
sanzin
Iterbion
Reation
Reratio
(a) Market Entrance (sell skew)
(b) Shop Positioning (c= 0.05)
(c) Equilibrium Pricing (i 5)
Figure 2: Convergence across environments, demonstrating bootstrapped mean and 5% confidence interval across 30 runs for the leader reward (left) and follower rewards (right).
We conduct comprehensive evaluation of the proposed methodology across a diverse set of canonical MFGs. Specifically, we extend three environments from MFGLib [Guo et al. [023a]] each exhibiting varying degrees of complexity. These environments, ordered by increasing difficulty, include Left-
Right, Beach Bar, and Equilibrium pricing [Guo et al. [0236]. with descriptions in Appendix FP] We also include in the appendix additional plots visualizing the evolution of the mean field as learning proceeds. All source code is available in the supplementary material.
Comparisons. The proposed approach is compared to the existing state-of-the-arr methods for the discrete time setting: 1) a nested-loop MFG algorithm which alternates between training the leader and representative follower to convergence (OneByOne): 2) weighted OneByOne update as presented in Algorithm 1 of [Cui et al. 2024) (CuiKoeppl); 3) MARL implementation, where the leader and followers are updated simultaneously with PPO, as presented in [Evans et al. 2025 (ADAGE). In the MARL setting, there are two policies, one for the leader, and a shared policy Vadori et al. 2020) for the followers, to keep the representation as fair as possible when compared to mean fields, and to prevent the need to learn N independent follower policies.
8

---

Poportoon
Popertio
Meurrietts
Heurrialts
Mearrieii
Leader Configuration
Crowd Aversion
Leader Inventory Target
(a) Market entrance
(b) Shop positioning
(c) Equilibrium Price
Figure 3: Equilibria Analysis
Market Entrance. The first environment is market entrance scenario inspired by the Left- Left-Right 2021a], akin to minority games in physics and EI Farol problems in complex
game [Cui and systems, as detailed Appendix F. Figure 3a illustrates the impact of various leader configurations. achieved by altering the leader's ' objective, demonstrating how the leader learns to execute optimal actions while followers adapt effectively, steering the mean field towards the target state (see also Figure(6) (6. The top row of Figure [a highlighs the leader's influence on the resulting mean fields, while the bottom row depicts the market skew. The leader shifts the market dynamics to align with the desired behavior. In each scenario, we observe convergence towards suitable MFE, with the MF adjusting appropriately to the task at hand. These findings underscore how follower behavior successfully adapts in response to the leader's actions, consistently achieving appropriate equilibria.
Shop Positioning. The shop positioning environment (Appendix is modeled after the beach bar MFG, where followers strive to position themselves near a desirable location while avoiding crowded areas, and a leader is tasked with establishing a new location to compete with k. We investigate how varying the crowd aversion parameter among followers influences both the resulting mean field and the leader's s chosen location. When crowd aversion is absent (c = 0), the leader strategically places the new location at the same spot x, capitalizing on the cxisting customer base, aligning with known models of spatial competition where competitors cluster nearby (Figure [bbleft), As followers' focus shifts entirely to crowd aversion (c - oc, Figure rightt,,, the leader places the shop uniformly across the state space, preventing any single state from becoming overly crowded. For intermediate crowd aversions (0 < oc), we observe spectrum of behaviors, balancing between the two limiting equilibria. In each scenario, the MFE is appropriately adjusted. These findings demonstrate how the leader's actions adapt automatically in response to the followers' behavior.
Finally, we examine an equilibrium pricing environment (Appendix F.3) as outlined in [uuo et al. 2023a b Cousin et al. 2017 In this environment, followers are homogeneous firms producing an identical product, with prices determined by the endogenous supply-deman equilibrium. These firms must effectively manage their inventory, production, and replenishment processes, and the leaders role is to incentivize firms to maintain a specific target inventory level. We explore various target scenarios, ranging from promoting lean operations with minimal inventories (suitable for predictable periods with stable demand) to encouraging robust firms with substantial inventory reserves (to withstand crises demand surges). As illustrated in Figure 3c the leader can effectively shift the mean field of the followers to achieve the desired inventory Tevels. This demonstrates the leader's ability to implement appropriate penalties or bonuses to guide followers towards either building a buffer (Figure right) or maintaining leaner operations (Figure 3c left).
Equilibrium Price.
Leacer Rewards
Talbowr Rewarck
Invetaryy
1.56
Loader Rewards
Falonee Rewards y % 1.
0.75
0.5
Inventory
0
0.00
-0.2
030
-035
Iteration
Ier atration
Iteration
teration
(a) Leader rewards
(b) Follower rewards
(c) Inventory costs (d) Mean Inventory
Figure 4: Function approximation with continuous states/actions in Equilibrium Pricing (with i = 0)
Key simulation findings. The convergence of the leader and follower rewards is compared in Figure 2 Overall, the proposed approach affords significantly faster leader convergence, and smoother convergence trajectories, characterized by reduced variability across runs and diminished intra-run

---

volatility. Notably, the more the followers' behavior is influenced by the leader's s actions, the greater the enhancement observed with the proposed approach. Breaking down the results per environment, we see significantly faster convergence for the proposed approach in the market entrance game than all the comparisons (Figure 2ap. while also resulting in an improved final equilibria (here, governed by the leaders reward which attempts to shape the equilibrium). In the shop positioning game (Figure(2b), (2b), we see significantly faster convergence for the leader, albeit with slightly slower follower convergence, yet still showing substantial improvements over the MARL scenario for both leaders and followers. The slower follower convergence here is because the leader has a smaller impact on the followers behaviour, due to their being the fixed desirable location, only a small part of the environment is changing (the additional location). In the equilibrium pricing game (Figure 2c), the proposed approach yields significantly better leader and follower rewards than existing SMFG algorithms. This environment is particularly challenging and exceeds the representational capacity of purely tabular methods due the continuous state and action spaces. ADAGE leverages function approximation, parameterizing the leader's and followers' policies by a neural neural work. We discuss in Remark [3] how function approximation integrates with the proposed SMFG approach we can parameterize the policies of the leader and representative follower by a neural network in a standard way, whereas for the mean field we make a distributional assumption and learn the parameters of the distribution. Specifically in the equilibrium pricing experiment, we assume that the mean field follows a Gaussian distribution and updates the mean and standard deviation of the mean field in AC-SMFG. The assumption may not be exactly accurate, which is a possible cause of the gap in leader's rewards between ADAGE and AC-SMFG, Nevertheless, the approximation suffices to yield competitive performance, and the proposed algorithm closely tracks the performance of ADAGE and outperforms other SMFG baselines.
(Continuous State and Action Spaces & Function approximation) In Algorithm ! we see that there are three key features that can be substituted with function approximation techniques rather than the exact update schemes presented: 1) the updates of the policies, 2) the updates of the value functions, 3) the updates of the mean field. Here, we show how each of these can be achieved. For the leader and follower, we can convert the tabular policy and value function into continuous representations with neural network function approximations (with an actor network and critic network)- - this is relatively straightforward and similar to how continuous state and action spaces are handled standard RL. What poses more challenge is the mean field update: in the tabular case we track the mean field by probability vector of dimension (S), but this does not scale to the large/continuous state space setting. For simplicity, in the experiments of this paper, we consider a Gaussian mean field (V, σ) and update the distribution parameters towards the sampled state with
Remark 3
Uk+1 = D% + (5(88-1))
Ok+1 = o + Ex((8 - -k++1* 2 ox).
However, more complex schemes could be utilized. For demonstration, we take the Equilibrium Price environment and assume that the leader takes a continuous action C2 E [-1, 1] to set the cost term, and the follower's 's inventory (and thus the state space) is positive, ordinal, and unbounded. An overview of the resulting learning dynamics is displayed in Figure { which shows rapid learning and matches or even surpasses the results shown in Figure of the appendix. This highlights the compatibility and effectiveness of our approach when integrated with function approximation in complex environments, underscoring the practical versatility and applicability of our method across diverse settings.
6 Conclusion & Discussions
We proposed a single-loop actor-critic algorithm for learning equilibria in SMFGs, and provided the first known non-asymptotic analysis. Numerical simulations show the algorithm's 's superior perfor- mance in various realistic problems, highlighting its potential for practical applications in hicrarchical decision-making environments. Our findings contribute to the growing body of research on SMFGs, offering a scalable and theoretically grounded solution for learning equilibria in complex SMFGs.
Finally, we acknowledge certain limitations of the SMFG framework in modeling practical problems, particularly when compared to agent- based modeling (ABM) approaches that simulate a finite number of followers directly. One limitation stems from the assumption of homogeneity among followers, which is embedded in the mean field formulation. Another key consideration is that MFGs serve as asymptotic approximations of N-agent Markov games, with the approximation error diminishing as N increases. In scenarios involving a small number of follower agents, the approximation error may become non-negligible, and alternative modeling paradigms such as ABM may be more appropriate.
10

---

Disclaimer
This paper was prepared for informational purposes by the Artificial Intelligence Research group of JPMorgan Chase & Co. and its affiliates ("JP Morgan") and is not product of the Research Department of JP Morgan. JP Morgan makes no representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. This document is not intended as investment research or investment advice, or a reconmendation, offer solicitation for the purchase sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute solicitation under any jurisdiction to any person, if such solicitation under such jurisdiction or to such person would be unlawful.
References
Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98))1-76, 2021.
Clémence Alasseur, Imen Ben Taher, and Anis Matoussi. An extended mean field game for storage in smart grids. Journal of Optimization Theory and Applications, 184:644-6777 670, 2020.
Robert Almgren and Neil A Chriss. Optimal liquidation. Available at SSRN 53501, 1997.
Alexander Aurell, Rene Carmona, Gokce Dayanikli, and Mathieu Lauriere. Optimal incentives to mitigate epidemies: a stackelberg mean field game approach. SIAM Journal on Control and Optimization, 60(2):5294-S322. 2022.
Yufan Chen, Lan Wu, Renyuan Xu, and Ruixun Zhang. Periodic trading activities financial markets: Mean-field liquidation game with major-minor players. arXiv preprint arXiv:2408. 09500, 2024.
Areski Cousin, Stéphane Crépey, Olivier Guéant, David Hobson, Monique Jeanblane, dean-Micher Lasry, Jean-Paul Laurent, Pierre-Louis Lions, Peter Tankov, Olivier Guéant, et al. Mean field games and applications. Paris-Princeed lectures on mathematical finance,, pages 205-266, 2011.
Kai Cui and Heinz Koeppl. Approximately solving mean field games via entropy regulayeee deep reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages 1917. PMLR, 2021a.
Kai Cui and Heinz Koeppl. Learning graphon mean field games and approximate nash equilibria. arXiv preprint arXiv:21 12.0/220,, 2021
Kai Cui, Gökçe Dayamkhı, Mathieu Laurière, Matthicu Geist, Olivier Pietquin, and Heinz Koeppl. Learning discrete-time major-minor mean field games. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 9616 9616-9625, 2024.
Gökçe Dayanıkhı and Mathicu Laurière. A machine learning method for stackelberg mean field games. Mathematics of Operations Research, 2024.
Mao Fabrice Djete. Stackelberg mean field games: convergence and existence results to the problem of principal with multiple agents in competition. arXiv preprint arXiv:2309.0064 2023.
Benjamin Patrick Evans, Sihan Zeng, Sumitra Ganesh, and Leo Ardon. Adage: A generic two-layer framework for adaptive agent based modelling. AAMAS, 2025.
X. Guo, A. Hu, M. Santamaria, M. Tajrobehkar, and J. Zhang. Mfglib: A library for mean field games. arXiv preprint (tt::23340086300 2023a.
Xin Guo, Anran Hu, and Jiacheng Zhang. Optimization frameworks and sensitivity analysis of stackelberg mean-fiche games. arXiv preprint 2211v:2210.0411111 2022.
Xin Guo, Anran Hu, Renyuan Xu, and Junzi Zhang. general framework for learning mean-field games. Mathematics of Operations Research, 48(2):656 686,, 2023b.

---

Yuze Han, Xiang Li, and Zhihua Zhang. Finite-time decoupled convergence in nonlinear two-time- scale stochastic approximation. arXiv preprint arXiv:2401.0389, 2024.
Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic. SIAM Journal on Optimization, 33(1):147-180, 2023.
Lixin Li, Qianqian Cheng, Xiao Tang, Tong Bai, Wei Chen, Zhiguo Ding, and Zhu Han. Resource allocation for noma-mec systems in ultra-dense networks: A learning aided mean-field game approach. IEEE Transactions on Wireless Communications, 20(3):1487-150 2020.
Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In International conference on machine learning, pages 0820-6829, PMLR, 2020.
Qirui Mi, Zhiyu Zhao, Siyu Xia, Yan Song, Jun Wang, and Haifeng Zhang, Learning macroeconomic policies based on microfoundations: A stackelberg mean field game approach. arXiv preprint arXiv:2403.12093 2024.
Dheeraj Narasimha, Srinivas Shakkottai, and Lei Ying. A mean field game analysis of distributed mac in ultra-dense multichannel wireless networks. In Proceedings of the Twentieth ACM International Symposium on Mobile Ad Hoc Networking and Computing, pages 1-10, 2019.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Nelson Vadori, Sumitra Ganesh, Prashant Reddy, and Manuela Veloso. Calibration shared equilibria in general sum partially observable markov games. Advances in Neural Information Processing Systems, 1114418-141288 2020.
Yudan Wang, Yue Wang, Yi Zhou, and Shaofeng Zou. Non-asymptotic analysis for single-loop (natural) actor-critic with compatible function approximation. In International Conference on Machine Learning, pages 51771-51824. PMLR, 2024.
Yue Frank Wu, Weitong Zhang, Pan Xu, and Quanquan Gu. A finite-time analysis of two time-scale actor-critic methods. Advances in Neural Information Processing Systems, 33: 33:17617-17628, 2020.
Qiaomin Xie, Zhuoran Yang, Zhaoran Wang, and Andreea Minca. Learning while playing in mean- field games: Convergence and optimality. In International Conference on Machine Learning, pages 1 11436-11447 PMLR, 2021.
Batuhan Yardim, Semih Cayci, Matthieu Geist, and Niao He. Policy mirror ascent for efficient and independent learning in mean field games. In International Conference on Machine Learning, pages -9754439754. PMLR, 2023.
Muhammad Aneeq uz Zaman, Alec Koppel, Sujay Bhatt, and Tamer Basar. Oracle-free reinforcement learning in mean-field games along a single sample path. In International Conference on Artificial Intelligence and Statistics, pages 10178-100 772-00 PMLR, 2023.
Sihan Zeng, Malik Aqeel Anwar, Thinh T Doan, Arijit Raychowdhury, and Justin Romberg. A decentralized policy gradient approach to multi-task reinforcement learning. In Uncertainty in Artificial Intelligence, pages 1002-1012. PMLR. 2021.
Sihan Zeng, Thinh Doan, and Justin Romberg. Regularized gradient descent ascent for two-player zero-sum markov games. Advances in Neural Information Processing Systems, 35:34546-34558, 2022.
Sihan Zeng, Sujay Bhatt, Alec Koppel, and Sumitra Ganesh. Learning in herding mean field games: Single-loop algorithm with finite-time convergence analysis. In International Conference on Artificial Intelligence and Statistics, pages 343-351. PMLR, 2025.
Stephan Zheng, Alexander Trott, Sunil Srinivasa, Nikhil Naik, Melvin Gruesbeck, David C Parkes, and Richard Socher. The ai economist: Improving equality and productivity with ai-driven tax policies. arXiv preprint ar arX1v:2004.13332, 2020.
12

---

Shaofeng Zou, Tengyu Xu, and Yingbin Liang. Finite-sample analysis for sarsa with linear function
approximation. Advances in neural information processing systems, 32, 2019.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly state the main contributions in the abstract and introduction, which to provide the first algorithm for solving Stackelberg mean field games with non-asymptotic convergence analysis, and to make the analysis under a relaxed assumption on the interaction between the leader and followers.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The main limitation lies in the gradient alignment assumption, and we clearly discuss how it is restrictive but still relaxes the prior ones, which are even stronger.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
14

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We clearly state all assumptions in Section 4 We provide the complete proofs all theoretical results the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide all source code for reproducing the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
15

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: We plan to make the code publicly available. In the review phase the code is submitted as part of the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
p CodeeubmiiSiie Sodeeuudsseeeoeuummiimmmmmm ooo.ooommmmmooooooooooooooooooic for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.. cc/ httpcc//pbbb pttpppullcc/aa guiiguiisssss /Codeoodeuulll Submi commisssonn ceiinoooicyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the source code and the most important experiment details the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: The figures show the boostrapped mean result over 30 repetitions, with a 5% confidence interval displayed as the shaded region.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [NA]
Justification: Numerical simulations conducted are small-scale and require minimal compu- tational resources.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httt:::t/ttstrr:t: http::/rwwppppppp.oc/cccccccccaa
Answer: [Yes]
Justification:
The authors confirm that the research conforms with the Code Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
The authors do not anticipate any social impact of the work at the moment.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No such threat is present our knowledge.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: To implement the bascline algorithms, used code from published works such as Evans et al. 2025]. These works are properly credited in our paper.
Guidelines:
The answer NA means that the paper does
existing assets. The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paapersw paperswithoooes com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have properly documented and added comments for the source code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing and/or research with human subjects involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No such risk present.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We did use LLM to generate any significant content of the paper.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: : //nuurripss. cc hwc./oonfeeeeee//////eeooea 2025/ LLM) for what should should not be described.
20

---

Appendix
Contents
1
Introduction
Main Contributions
12
Related Literature on Stackelberg Mean Field Games
Stackelberg Mean Field Game: Formulation
Algorithm
Convergence Analysis
Algorithm Complexity
5
Experiments
Conclusion & Discussions
10
Notations and Frequently Used Inequalities
22
Proof of Main Results
24
B. Proof of Theorem
26
Proof of Propositions
28
Proof of Proposition
28
Proof of Proposition]
29
Proof of Proposition
4 Proof of Proposition
34
Proof of Lemmas
36
D.I Proof of Lemma
36
D.2 Proof of Lemmal
38
Proof of Lemma
39
D.4 Proof of Lemmal
39
D.5 Proof of Lemma
41
D.6 Proof of Lemma
43
7 Proof of Lemma
44
Details on Example Satisfying Assumption 6
47
Appendix: Environment Definitions
48
21

---

Market Entrance
48
f:2
Shop position
49
Equilibrium price
Additional Experimental Results
49
Experimental Details
50
Notations and Frequently Used Inequalities
We introduce some shorthand notations frequently used in the of the paper. The operators below abstract the (semi-)gradice of the leader, follower, mean field, and value functions.
D(w, ue v,, V. log (b 8) ( () +V(())) vi(s)).
F(0, s,b,,, = Volog s) (ry(s,,, ( TT - Vj(()),
H(,, 8) = Cs -
(17)
Gilw,,, 6,8 = e, [r((, b, + +V((())) - vi(s)):
G, = e Ty(ss, a, b, p) T + -)
Given the notations
we can re-write the variable updates in Algorithm
wk+1 = wi + Duwk , Piy Vi,ks Sk, bR sk),
(18)
Ox+1 02 + Fx piu V 8k aNe bbs s%(),
(19)
px+1 1l.s pix + H (px: (5)555))))))
(20)
Vi,4+1 IT Vi. + Gu Pkk VLR. 8k, ak by, skb)
(21)
V Jx+1
BB
Vi + B (Ox, Prx, Vi Suy ag by. sts)
(22)
We also define the expected versions of the (semi- geradettraieett operators, where the expectation is taken over the stochastic samples from the stationary distribution.
D(w, n. Vi) E endd own
D(w n. s, s'),
(23)
F(0,w,,, Vy) & E and of Awn (n
[F V, b,a,,,'))
(24)
H E Eg
H (1,
(25)
Gi(w, Vi)
(G μ. V1, 8, a, b, b)))
(26)
G, 2 swaff Soid Swnl a=____
G, 14 8,
(27)
We re-iterate that due to the state separation discussed in Section 5 and G are independent of the single follower's policy æ, though appears the right hand side of (23) and (26).
We define the filtration F& = {ss, ap, bo. s ... Sks ake bb, 8/ ................................ ..........
Unless otherwise noted, we use || to denote the & norm of a vector and the operator norm of matrix.
We may use V.. ((w) and ( μ² (ow.) interchangeably in the rest the paper, which the true gradient of the leader. The partial gradient denoted by V.Ji((ou HI) (o=)"(
We also introduce few frequently used lemmas. The first on the Lipschitz continuity and smoothness of value functions.
22

---

Lemma 1 We have for any 0 0 w. w', M H.
(T ow 14) - J (TO ow: : H¹) & Lv - 0'll + wlow - -url + || - Il),,
(28)
1V, V0 - V V. || S Lv(110 - 0'1 + 11 - ou + 111u - 1 1I),
(29)
|| Vo Vi Vo Vorjur Aunt S Lvvll0 111, VV Vooidunt V - o1 Lvvll
w/1
(30)
I Vy V V V # || I L pl1l,
(31)
J) Ji n¹) & Lv (1W w'll + 1100 - , 1'I1),
(32)
11, 1,0- H1 || S Ly (1| - w/ll + |1 - 4 I)),
v
(33)
V V. 1, v,owrill S LVv ||W - w'll, 11 V,0 V Vi
S LVv n'1l,
(34)
15 { [1-gp3 (1-2)² + t and Lvv max { (I-7]a (1-7)3
2x V1s13.
27 //
where Lv max Lybp.
[1-3)²
The second lemma guarantees the Lipschitz continuity of the stationary distribution (of the current and next states jointly) with respect to the policies and mean field.
Lemma 2 For any 01, 02 w1, w2, p1, U2, we have
Σ
do a owr Hi pro owr Pn (. 8) dp a dwn H2 (s) pF own 22 (.) (-1 8))
1
[Plll1 - uell + 110: 0211 + ||w| - w2ll)
The next lemma establishes the boundedness of the
operators introduced in 17
Lemma 3 Recall that By is the entry-wise upper bound the magnitude of the value function. We define the constants
BD 2(1+ + BF = 2(1 + By + log A + 2,
(35)
=, =1
+ (1+
|A| + 1.
We have for all 0, w. n. V1, S,,,,
|D(w, M4 V,, s s')ll & Bp, VV b,)l, s Br, [H s)I S Bи, [[G[[,, μ. Vi, b, s')ll Ba, ||, p. Vj, s, s')ll s BG.
Lemma 4 further shows that the (semi-)gradien operators introduced 17 are Lipschitz.
Lemma 4 We define the constants
LD = Lp = max
{ 1 + log |A| + (1++ +
(4 + 8 8log| (AI)T
+ 1 1- 2L, + 1- 1- 4
Br
BrLr Br
(1 (1-7)3
Lp
)
(4+ 8 log (A1)T
Bg
BaLp BG
LH = max
1
LG= = max
+
Lr +
1 1- 1 1- 2
(1 (1- 7)3
1-
(36)
We have for all 02, Ti, T2, 21,,:11,,,0 W2 M1, U2, V1, V/2
[[D(w, M1, V1) - D(w2, u2, 1/211 _ Lp (l| w2l + 11:111-100 -ell + 1|V - Vell).
[[F (1,,w, Vi) F(22, w2, U2, 1/2 LF (||0: 02ll + ||ا] - -w2ll || ا2ll + 11V1 V2ll),
[[ - 11 (TT, W2, p2) - LH ([|= T2ll + [||] - + || -2ll),
[IG,(0,,w1,, Vi) - G) w2, /2, 1/2] I LG (I10, 02ll + ||41 will
+ ||| =2ll + 1| - Voll),
[[G] V1) - Gi(we, 22, 12) I Le (Ilw wel
+ || Ho/|
+ [|V VV2l).
23

---

Algorithm 2 Actor-Critic Algorithm for Hierarchical Mean Field Games (Simplified for Analysis)
1: Initialize: leader's policy parameter wo. follower's policy parameter On. value function estimate V.oo.ooo Vo, f,o. mean field estimate ño As, initial state so.
2: for iteration I-
3: Sample 1 for tracking the discounted occupancy measure:
Get samples 8k df akk Own (- 8k), by ~ twn (k(8)): 8/ -Dx (.1 8k, ak by) and receive reward (5k, ak by, (t.). TI (I(5 by, (ta).
4: Sample 2 for tracking the stationary distribution (mean field):
Get sample 5% n owy the Leader's 's policy update:
5:
Wk+1 = wk + Ca w log ows (bx (t) (m by. (aa) + (V)((sh) - V(r(ss))
(37)
6. Follower's policy (actor) update:
Volog 14+1=0,+oo Volog To, (ax (sx))(r(( ax, bk, +T 8k) *x)+7V Vy (8k))
(38)
7: Mean field update:
Bu+1 1lls (Ax + Ex(ezz pux)
(39)
8:
Value function (critic) update:
V1,++1 11 (Vix + BResr (ri(sk, bu. px) (V,((s)) VlL(SH)))
(40)
V1&+1=ITB BResss (s) ak byy (b:) + r F( 8k) + 7 (Vk)((k)) V)l(sk))
9: end for
B Proof of Main Results
We analyze slightly simplified version of Algorithm which we present Algorithm 2 The only difference between them is that @gorithm@@replaat continuously generated Markovian samples with i.i.d. samples from the stationary distribution. Note that stochastic approximation and RL algorithms have been extensively studied under Markovian samples (Zou et al. 2019 Wu et al., 2020), and it is well-known that Markovian sampling only incurs an additional logarithm factor in the convergence rates. This simplification allows focus the presentation the innovations develop for MFGs, without diverting attention to well-understood technical details on Markovian samples.
We state the theorem on the convergence of Algorithm below, with the exact conditions that the step sizes need satisfy.
Theorem Consider the iterates of Algorithu
under the step sizes
Cç ÇA (k + 1)2/2
Ca
C5
C9
ai
(k 1)1/2'
Ek
(k + 1)1/2'
Bk
(k + 1)1/2*
24

---

with the constants Cç. Ca. cg, Cg selected such that
1- 8
G fk S or S 3x 1,
&x S min
1-5 8)
81.2
1-7
2
1 1-7
0g S min(
}:
3: -
4Lvv (Bp
BD + B1))²
BB + B
1/6
1-6
1-8 8
1-8
11 (1110) 6))
1-8
Bи
LLH
S min-
&x
16-L2 24m 1 L 8/6LLy LD 16I
'6 BDL 3Lv
- (1-7)7² pm Pimin min
(1 7) 2 Pminn Piin
(1 7)
p Pminntiin Li
< min
og
16|5|, 11n
192 12
4
Ck
1-7 (1- (1-5)(1 7)
1-7
1 1-7
S min 96L2 38412 LE Li
8x
32t1
2/662
Ek
(1 - (1
6) Tp
Pimm Pn
min
(1 T Pinle
(1
(1-7)7 (1-7)72pm Prinn
Br + Bu) }
< min
a
LYSLH
Lv(2B) + +3Lv)
Ok
1-7
1-7
2Ba
1-7
}
<
min
8k
(384L; LybLyLD' VOL + Lvv (Bp + BD + BB) 4V6LV
Lvv
(41)
Then, under Assumptions
and we have for all k: > 0,
16
min
BiE
cc(k: + 1)1/2 (JilØ" , ()')) I[ws 1¹ (owe)) E/ + t + $ +
V
t<k
128c. log(k (K: +1) <c + 1)1/2
We note that the step sizes satisfying (4T) always exist, and the conditions are met by making the step sizes sufficiently small. To find the appropriate step sizes, we first make Cæ small enough that 3 satisfies all of its upper bounds, then og small enough with respect to Be. followed by ca with respect to & and Sk. and eventually eç. Note that cg, Ca: Cg, Cp all chosen as polynomial functions of the structural parameters, and do not depend on k.
The proof of Theorem I relies on intermediate convergence bound on the iteration- wise convergence of the leader's 's policy parameter, mean field, follower's 's policy parameter, and value function estimates. We state the intermediate results in the propositions below and defer their proofs to Section C
Proposition 1 (Leader Convergence) Under Assumptions and and step sizes satisfying we have for all k 2 O, [B]Ji 1² (dww)) Je(0wa (dww+))))
GN E[E2] + 2m CEEE + 2m Li +
Ly LBB
2m
2
Proposition 2 (Mean Field Convergence) Under Assumptions 4 and 6 and step sizes satisfying 41). we have for all le 2 0.
1 5 < (1
Ex) + 11
8I221I2 B|s
Elet k+
E[= +
(1 6)
(1 8)
321,213,62 B(eY,,) + 16
(1 (1- -)
Proposition 3 (Follower Convergence) Under Assumptions 7 and Band step sizes satisfying 41 we have for all k 2 0,
2) 13
El < B[E
ak Blllll ( OwL 13)
B[E
16
(1 (1- (1-7)72p²s P 2 Olk
16
815) 17
E(ery] + (1 (1-7)72p" 2 EleYA]
(1 (1- ? Puinn O
+ 2le + 3B (B + BH BH)
25

---

Proposition 4 (Value Function Convergence) Under Assumptions 7 4 and 6 and step sizes satisfy- ing (TT. we have for all l 2
(1 (L
6L
+ 6 Blejl
E|sy f.t1ll S 511
IEL f. + (1 7))3k
, (TO.. (T Own, prx) ]1x)]l3]
(1 (1-
6L ElEE] + 16L L. 22
241 k
+ (1 - 3)
B[
E B|EY,1 + 12
(1 - x)
(1- 33xx
(1 - 3k
az
6 L3 2 ElcE]
|| S (1-
+ (1 - 33
Eill JJ ( , Pwr (bx)) )|2]
4
(1 (1- 3) (1-
6L 2 2
161 S3 22
BiEl] + 6L
+
El=A1
+
ElEYAal + 12B8
(1 33
(1 - 3x
(1 71-7)33
B.1 Proof of Theorem 2
Collecting the results from Propositions 14 we have
E
(Ji(oun: 1² (d Je Je(ó 1 1 (dun1)))) + + + + + + + +
Sk ElEg] + 2m Li (( + 21 (4 +
ByBB
2m11
2
1-
5
11
TF(E7] + 8L 1222 22
32L2L EleY,) + 16B3
+ (1
4
Ex
[k
+ - 8)m
(1 - -)EE
B1
10
(1 - (1-8)5x
2/ 2
16/8|121411222226 BlE
+ E[<a]]
Okk
Ell V J (TO,, Pwr, Ax)1 + (1 (1-7)3 7)2 Pinin Pmin akk
Bleg1 + (1 - 222 PiniteP Okk
16
+ (1 7 Pmin Piin a.k
BlEY,1 + oL V f.al + 3B + B Ba)aε
+ (1 (1 - 3K
6L3,a7
EB|l V VoJs (T Owr A 6
3E (1 (1-7)3k
(1 -1-18 3x Blsa1
612 năCk B[Eg] + 161 L7150
24LE E|EY,| + 12B3 82
+ (1 - 3)
B[Et]
(1 - 33
(1 71
+ (1 (1- 18
+ 6 L 2
)Ele,,l+ + (1 33
EIll ( DwsK A )11
(1 - 3) E|sT]
4
61 122R E[EN]
16I, 13353 c2
+ (1 3)
+
ElEt||
+
(1- ElEY,A] + 12B3,8
(1 - (1-
Ck + (1 - 8)5kk
81.211222
2 2/
12
+ (1 k1
E[EA]
2m
(1 -1-18 7)T? p Priin Pinn Ol
Ti
1 1 8 4
321213 1633
+ El +
Ex + 2m 13 +
Elet
(1 7) Pinia min OKk
(1 7)kk
T2
12Tifaz
11 (1
1212 12 c2
+ E[
+
0k
+
B[ll Vol Per , 1 +
E(c
16
(1 73
Pmip
(1 [1)
23
+ E| + (- - 7 Bx + 2m + 321.2
815112-12
2413-13aa
B[EYA]
+
4
(1 - (1-
(1 - ? Priin P C.k:
(1 71
Ta
+ [.l= +
(-1
1 1
3x + 2 ax + (1 - 8)
BlEY,Nl
4
Ts
26

---

B L + $16 + +3BrBBn + Bu)o + 3
(42)
We bound each term of (42 individually. First, we can ensure 7i <
Ct by choosing the step sizes
8xy
such that every positive term is no larger than ft.
The required step size conditions are
8th
Ck &
1-8
(1 - 7)- Phinn ag S
Ck
SK 1-7
S
64L
16 m2
3
96
mm
1-6 making T2 For T2, we choose the step sizes such that each positive term is no larger than non-positive. The step size conditions are
12
Sk &k 24
-
aj S (1 8) Pmine Go (1-8)(1 (1
S
192 19
3. 384L
To bound T3, note that under entropy regularization, J, satisfies a non-uniform PL condition with respect to the follower's policy [Mei et al. 2020). Specifically, for any 0, db
IV Φ, p)l 2(1 P)
J, (0, (.. 0.) J (T 2.1))
(43)
[S]
The inequality in this specific form is adapted from Zeng et al.
2022
[Lemma 4).
Under
the step size
conditions (1->)² T P
1-7 3812
&
3r. and
minf
(1->X(1- S o Pini 352|S|L2 ii
implies
384 384
Li
11
12L 12
T3
E JJJ (Te.. Ouk 1+) (1 8)TPmnnn
E
B[=
32
T)
Pinin
11L3/5x
17
akk
B(eg]
5
8)T
(1 3)
For T4, each positive term is no larger than
1-y under the step size conditions
16
Sk
1-7
Sk
T/ (1 8)
Ct 1/17"" 7 piin 2 L?
Okk 1-7
321/1 1.7
&
167,2
Or
Bx 8V6LVLD
This ensures T4 < 0.
Similarly, have Tş under the conditions
1-7 2V6L3
3
161
3R
Finally, choose the step sizes such that each term in Tg is no larger than o
Ck < BDy
Ty &
3k S
minnf
4B BB + By)
agk
2
ag
a 2 2/6Ba
Plugging the bounds on Ti-Te into
42 we have
E (Jilow 1² (dww)) Ji n² (Own + Ehtt + + + + + cY,ut1]
< S* El + Elet + =ī + E + eV,ul + 4ak.
(44)
8m
Re-arranging the terms and plugging in the step size, we have
C
El < ((-w 4¹ (Dwe)) + at + E + sY + vVA.l
8(k: + 1)1/2
27

---

4Co
BI- 1² (owe+1)) + cktt + + + t + + +
(I: +
Recall that of is the optimal solution to the Stackelherg objective in ( The inequality above implies
8(1 + 1)1/2 B
C
Blei] + Blet+ + + + + e1,ttal
8(1 111+1)1222 1)1/2
t=
k-1 4ca
(0))) J(ówu p² (dw))) + # z + + + $ +
(45)
(t + 1)
The following relationships are standard results: for any k 2 0, we have
k-1
(k: +++11 1111/2
k-
Σ
S 2log(1 + 1).
(46)
11+1)
2
t+1
2=0
Combining
(45) and (46)
8(1+1)1/2 B|E
min Bi t<k
1=0
8(144))47
16
JilØ" Je(Ø",H'(0")) (0")))
p¹ (dwo)) Howo
E E + + f7,
< + 1)1/2
128c, log(A: + 1) cx + 1)1/2
C Proof of Propositions
C.1 Proof of Proposition
1
By the Ly-smoothness of the function Ji
Jilówz. /² (dwx)) Je(owers 1² (Owz+1))
Lv
- S-(Vw-i( n² (owe wh+1 wx) 2
||w+11 - wxll
Lv D(wks Pay Vi suy bu s1Il
- WI 4 (owe): D(wks Dis Vi,eo Sky bu. sh)) +
2
4² (owr) ), D(wes Pucs Vi, Sk, bR. sk)
D(wks Peks Vi,k))
-Cu(V n² (dwe), D(wk, ia, V(,w)) + Lv
ID(Wer Pur Vi, Sky bu, sk)11.
(47)
2
Taking the expectation and plugging in the bound on D from Lemma 3 we have
E[Ji(o 1² (0w.) Ji( 4¹ (Dwez)))
Lv B3S2
< -C 1² (owe). ), D(wn, pax V(,a))]
+
2
S V 1¹ (owz M19] LvBBSk
m11
2
G.E((VVW (Dwns (dhwa,) D (wn: n² (Dww (( D(wn, Du. Vr,))),, (48)
28

---

where the first inequality follows from
E((V Jilwow μ¹ (dwn); (ww Puo Vise sk, bR, 54) D(wks Pay, V1,x))] = [[(V /¹ (dwn); ([D( Dix, VLL, 8k, bu, sh)) D(wn. Pu. V.,)]] FR-1)) 1)] = 0.
and the second inequality follows from Assumption and the relationship
V (own p)) (ow) = D(w, p (dw), Vionn (ow)).
To bound the last term on the right hand side of 448 we apply Young's inequality
Sa.VV Wi 1²" (;w)): D (w' 1² (Dwe), ), Vi (ome) D(wk Pek. Vl.k))
Ce V 1¹ ()12
7iSk
[ (wns 1² (thw)) veenii if (dund) D(wss Pis Via) ||²
<
2m
2
Sk: J (bm))) + 1n 1. Vi V (dw)) 1/2 + m11 DibSallix - 1 (1
2m
Sk Ex. + 2m Li. vi Pk 1 + +m mLL3(L2 + 1)Ckll - 1² (owx)))
< 2m11
Sk Ex + 2m, L 27/ LRE
S 2m11
(49)
where the second and third inequalities follow from the Lipschitz continuity established in Lemma Substituting (49) into (48)
4
B[Ji(wun: 1tt (Dwe)) Ji( 1¹ (Pwe+1))
Sk E| +
Ly B7Sc + Ca Ji 1¹ (Dwe) ) D (w>:1" (dwr)) V D(wkk Pas V(,e
71
Sh Elef] 2m T.5 Cx 2m 1. + Ly B3B5
2m
C.2 Proof of Proposition
The analysis employs the following lemma, which bounds intermediate cross term. We provide the proof of the lemma in Section D.5
Lemma Under the assumptions and step sizes of Proposition 2 we have for all k: 2
Eljūx 1² (dww) + Ex H (T (dwe, pkx) Wk: (ux), μ³ (owr-1) 11" (x
8) -1
Elllix 1 (Dwe) + ExHl ( (owa 1121 48L2L,, [|
8
(1 - 8)
81.21352
Bie + 32T. BlEYAl + B3
(1 (1- 6)Ex
(1 (1-8)Er
We define the shorthand notation A A H(uu, 3x) H HTT Wk, (ax). Note that A/A = 0 and || < 4B
By the definition of e
liin+1 1= (1um))))1
I (PLx + ExH (py,, 5k)) 1¹ (owaLi)l
S Ilísk + EL (iK, 5k) 1¹ (bhwtt))l )1
= jixx 11" (tbwa) + Ax + Ex (or (ix) Wk,
29

---

+ EN (H Wis (ax) H (TT" (Ower p wa. (w)) (u"(wwx.)) - 1² (own)) 1
s 1lix 10 (0(,w)) + Ex H (x (dwes (k), Wks p=:)l 3 Ahall + ([wees)) 1 (Aww))
+ 35 | (Tp Wks (2) H (owe, Pa)) Whi px))
+ 2Ex (iu- 1¹ (pwr) Ex (TT (Own Px): whi pa), Ahu)
+ 24px u¹ (dme) Ex H (IT (dwn, Px) ), whi pa); 1¹" (Pwe+1) - 1¹ (dwe))
+ 2 (Du: - 1¹ (fwx) Ex (TT" (dwe Px) Whi Pa), H (T" (Owr Pk); wks wk) H ( px)).
(50)
where the first inequality is due to the fact that projection to a convex set is a non-expansive operator. As H (0), 0, μ² (0)) = O for any o. we have for the first term of (50)
Ai - 1 (dwn) + Eu (x=² (dwe: : Pa). Wei px))l 112
1lik 1¹ ( + Ex H (IT' (ower Pak) Wk: pak) Exx Ex (D ) We, Pt (owz ))12
lliee - (mmm) 12 + 2 H (TT (ower (K)), Wk, itx)) H(TT" (dhwn) whs p¹ (dww))) 1)
+ +Eulux - 1 (((ww)) 1²² (Dux Aa Pa - fuy - v²x (o )on (pue) + 1 (dwz))
S (1 - Exlliix - (+(w))) )11 + 21 22 iiv 10" ((hw))) )11 + 2T 2211
(oww - x (mhw))) )1?
+ Eellix 1¹ (ow, )1l| " (6Le (x) wan - 1x" (don wurk (dws))
< (1 (1 5) + 4L 1223 52 1| 1e" (Dme))l
$ (1 1 -
En 5x
(51)
2
where the second equation plugs in the definition of H in (5). and the first inequality is due to the Lipschitz continuity of H, the second inequality follows from Assumption and the Lipschitz continuity of π from Assumption 2 and the final inequality is due the step size condition
E
1-8 81.21
For the fourth term of (50g.
KiA 1¹ (6.r) + H. (dwe (ak); wkk (ak), Ahk)l
1 2f 1² (Pwe) + Hx (Owr (u))), Wh, pk), BlAhe FR-1lll
= 0.
(52)
For the fifth term of 50), we have from Lemma 5
2 2 - 1¹ (shw.) + Ex ( (dhr (46)) wis Pux)s 1t (Ownts) - 1 ((www)
(1 5) Blliu= 1 (Dmc) + E.H (TT (Oer (k), we, 11
4812222,..55 Bleg]
8
(1 8)
32 EleY, + +BBLC
+ (1 8)
B1a81 + (1 -))x
8)Ex
1
Ei +
481.212,..355.
BlEy]
+ (1- 8)E (1-5)EK
8T.21122 B[E
3212T.35
BiEY + BBLC
8
(1 8)
-0)
1 - 8
8L27ECR
32 EleY,Nl BZLLE
Eu
+
Elex]
(53)
4
8)-
(1 (1-8)E>
where the second inequality plugs in 51 and the last inequality is due to the step size condition
C < 1-5
& SVBLLyLD'
Similarly, for the final term of (50).
2Ex (Pxx 1²¹ (owe) + Ev Hx (owe Pa) Wes Pk), (Bwe: (ax), Wk. Pax) - H (ToL wky Pl)))
8 ExllpR - (dme) ) + H. (0 Pa)) wr. wh. px)ll
< 8
30

---

8
+ 5 [x (pwn px)) wk. p&)) ; wk: 12
8
8
E +
1 1- -iellll1 ( (Ower Pic) wks Pax) H (TT Wks Px)
(54)
8
Taking expectation and substituting
into (50)
1 8
Elek+1l S (1
& Bleg]l + 352 AB 3E||K" (dwnti) 1=" (owe 11
2
+
(To, Wk, (x) {x (سwم M&:) wk: 1sx)
8
8T,2
321. S 22
+
+
BlEx] 1:
Ve BLGLE
4
(1 - -)
(1 - 8)Ek
1 - 8
8
Ex Ex
17 ग
(Dwes iss) Wky jix)
H (T Wks pxx)
8
1-
8
< (1
Ex) B
+ 12
3. Wk+1 wxll
8
8L 122
3252L2552
+ (1 E)
Elsx] + (1-8)64 8)6
E| + BB
11 B[| (
+ - 6
(To wk; (xx) - H( (ohwr (is); wks fax)
1 - 8
8 Ex) Bl(g + 11L,7/5 B(l|TO, n (thn, [xx) 12 +3BB13
< (1
1-6
8L 1/2 Sk
E[s + 32122
+ (1 5)
ElEVAAl + BT
(1- (1-8)Ek 8)
1 8
11 Ei| - T²' (owe, 11
$1
8 &x
Bleg]]
+
1-6 8
2
32L L3
Ela
E[sy 16
(55)
(1 6)
(1 - 8)
where the second inequality follows from the step size condition &a S T-3 and the third inequality is result of the Lipschitz continuity of μt and H. and the last inequality applies the step size condition
ÇL
< Bg B
EL
Note that the following inequality holds for any x, T, as a result of the entropy regularization (adapted from Lemma of Zeng et al. [2022)
|| n"' (b,11))1
(J((="(0,)))))))) - JT(T,
(56)
TPann
Combining (55) and 56).
1 1-6
(1 11L3/54
Bieu
1 S (1
8 EN B[ee
+
E
( (Swn: Pu)), (wn..Pa) J -
K+1
8)
3
+ (1 6)
ElE
El=Y,l + 16B7,
(1 8)
1 -
11
8T.21ZC2 ()-
32T.2T.322
(1
& Bletl]
+
Blei]
ElEî]+
ElEY, 16B
4
(1 Pmin
(1-8)Ex
C,3 Proof of Proposition
The proof of the proposition relies on the intermediate result below which bounds an important cross term
31

---

Lemma 6 Under the assumptions and step sizes of Proposition 3
we have for all k. 2
0
VwJ Own (M)) 5 Swy, (As:)
(Rve
wk+1 wk
}1
V y Pwn. (b)) V J Ows ((ak) (dww SPa)
Mu+1
fuk
(1 - 7) Pinin are B[l|=* (dle (ix) - To 112] + (1 7)2 P Pmiin P min OAK
c2 2|5 2 L
1S
(chwett) To 10,
4|
2| 2
1, Blek] +
81
+ (1 (1-r)r²pmm Okk
E
B[c(,N)-
(1 (1-7) (1-7)r2 PriuuPin ak
(1 2-2 Priuu Ar
We provide the proof of Lemma 6 in Section D.6
As Jy has Lv-Lipschitz gradients.
J (To,, ( f (ToL+I :wets: Px+11
LV
$ - -(VoJ Qur_ 1x+1). 0s+1 0x) +
||0k+1 - 0x2l
2
-on (V (To. OWn Pu+ 1). F(Ox. Wh, Pa. Vy sh. ax. bes s'y)) + Lvo || F( War Pe. Vy S2, ag bu. s%,)1
2
-oa (VaJ S PR+ ) F( Whs Pu. VIN. 8ks oke bbs 51/) - F F(xxs wki Wiy Vs,z,)
Punns P2-1): F Wks Pk. Vy.+) F(0k, wks Mis V V. Payy ( (oan
an. Ownnn Pk- 1). V a (T owL. pw)) + Lv 2
F(BR, Peas V Sn, ar bn st)ll
2
(57)
where the final equation follows from Vg. ow: (1) = F(0, w. Mi Vileeeitt for all 0.w.
For the first term of (57)
- (V J (To Sween 1u+1), F(Ok, Wk. uu V Sk ax by, sh2) F(Oks Wks Pw VyKiil
-o F oJ TO Pun. ))); [F[ (0x, wk, Px, V Sk, ak, ak, bb, 8k) F(kk, Wky Pax, Vya.) Fk-
+ a. VoJ (To,, On Px.) - Vod Owets Pk+1), F0 Wks Pkx V 8k. ax by s/2) - F(Ox, Wki Pks Vzx)
=a,B((Ved ( Own (N)) ( Swe Px+1), (
F(RR, Wky pua Vg Sks au bb, si)) F F was Das vr,,
2Bra Lv Blll@ - Swall + 11 - -ullll
< 2BPLvar (BDC + BHEx)
2Br + Bu) Co.
(58)
where the last inequality follows from GK S Ex.
For the second term of (57).
a V (TTE Puelss 18+1) F(Ou, wk, Pas Vy,z) F(k, Pis Vita Pee
ak
VoJ (TTEL, Swess 1 + 20x F wh: Ar, Vrxk) - F wky Dik, V² vo Our (er
8
ak
VoJ (T Our 1= + ak VoJr(TTO
y Dwee 1xx)1 + 222
4
4
VoJJ Qor 12 + 12
(110ar4 - Owr 112 |1 mall3) + 21
4
2
VaJ Om )2 L3
(B7 + B1+52) + +2R
aKk ( Aay. 22 + 2L. f.k + 12
(59)
where the third inequality is a result of the Lipschitz continuity of the value function, and the last inequality follows from the step size condition Or I BB+B3,
2 and S+ < &
For the third term of (57).
-a (VoJ Pwess 1.41), (od own pk))
32

---

-oa
VoJ (TO OwL Px)] + au (VoJ) Pwr Oue. A&) V VoJ (TT (or (T Owz Pk))
ak
V VaJs TAx Pur )1 + ak
VoJ (Te,: Awx V Ve.J((Te Owzys 1x+1)1?
2
2
0k
V VoJ (TO Pwr )= as [1 1 || iis 1nal1²)
V V Te. We )= ( + B7152)
| To, Pwr )I 2
(60)
where the last inequality again uses the step size condition 0g S Bb B3
and St S Ex.
Substituting
588
60
into 57 and taking the expectation,
B (TB Pur 1x++)
[ Pwets D&++1]
2Br BD
BuI )ORE
OK E V oJ (To.
(TO, Awn (n)]1 (wm
21 BleY, T2
ag Blll V Our p + 22
Ok B[| VoJ (TO ( Pur Pa A + 212 Eey 3L3E2 +2Br(BB + BH)aakk. (61)
< 4
Under entropy regularization, J satisfies a non-uniform PL condition with respect to the follower's policy MMei al. 2020). Specifically. for any 0, o. μs,
2(1 7) Pinin
[VeJs(To, o, w)l2
(J((T"(d, -
(62)
[S]
The inequality in this specific form is adapted from Zeng et al. 2022 [Lemma 4).
Combining
61 with 62 leads to
, (TTR, Swe 11++1) - ( (1 Swest 1w++)
s OKk [ VoJ ( Pwni 11 + 2L + 33 2 + 222 BD + Bu)ouεk
16
3(1 T Ph atk (J ( (.:)) Own (x) J (T pun. (i)))
S
$ 0kk E[l V VoJ OwLr A + aq f.a. + 3 + 2Br + Bu)auε
16
3(1
7) p D
min
| (owz :olaas.pp Te 112
(63)
8|S|
where the last inequality follows from (56)
Within this subsection, we denote Is (owe, (.s.). Due to the Ly-smoothness of the function Jf. we have
(Te Owr (ux) J (Tee. Swats, PK+1)
Lv
< - -V (T Pwr. Plx)) xx+1 - Ixh) +
| - 2x.12
2
Lv Tk+ll
(V JJ (T Oun Pu) - v J (IT, ownn wux) I (p)) Ib+l xxk)
1xx1
2
(V 1 Ownn (x)) la=a (dlw .Ax) Fk+1 xxr)
Lv | 12
<V J (T0z Qwr Px) V J (IT, Pwn. (itt) la (b)) Fk+1 xx)) +
[ ( - (axe Own (xx) Ik+1 - Ik)
Lv
$ - (TTL Owr Pxx) - VzJ (TT, Own fix) Ii (Swn (P)) xx+1 xxx) +
|| |
xxx1
2
Lv
+ (J,(T Plx) owr. fixe) Jy (dwza Px+ 1), QwezT (xx+
||
xx
2
33

---

where the second inequality follows from the fact that for an L-smooth function f : I + R we have f(x) (x)) 2 (V z) f1y
=11?. The second equations above uses the relationship V=J ( Pa): Owny (4)) = ( oww, (ix) (p=, Ax)) which holds as the partial gradient with respect to is zero at the optimizer (under entropy regularization T >> 0, the policy ( vi) is in the interior of the probability simplex for any @, fr), i.e.
DJ y (Qwe, , Pkk) Pwn (K))
²π @ (owr (ts))
Taking the expectation and plugging in the result from Lemma 6
(TT, PwL, (ba)) J (TO. Swar +
7)r² 2 P'in Okk
El|l xt (1ll: (bw): - To, 112) (1-7)=² Pruin Pimii Okk
(dwe.be)) T0, 11
2
16 Elegl + (1 (1- PhrpPrinp min Okk
+ (1 (1- Price Pmin ak
El=g1 71- ax
B(EYA]
+ (J (dwer Pa)) Ownc Pa)) - 5 (Ówx 1R+ Dw ( + Lv
- 2x 11]
(64)
Now, we combine
63 and 64
Recall the definition of ez in (5)
Elek+1 - =x]
B[Jy Own Pm)) ( Swn+s 1: (x+11]
+ [ ) - J ( Dwnts , (*+1))
+ (TT (dwets Sweys - y (dwrr Px)) Pwn px)]
(1 2 PaiiiPPni Pinin Onk
2) L3 L3
(lll=((wwen Ux) Top [1 (1 7)² P Pmink Ork
-[Il==(@wweiiii (Dwe.de) Tep 11
4|81
2/
1618|12L7,13, Eleg] (1 (1- 2 Pmiiu Pasuummin OK
818 Bley,u
+ (1 (1-7) (1-7)5² 2 PriiuP Puin 0
Bie (1 71- PriiuPp ak
+ [v - 12 || - 1
as BBI ر (T wwee uxx)l [1 + ax +3125 - (BB + BB)BB) Ex
16
3(1 - 7) ph Pinn Od || (owrr pxx) - Tea 1|
S
ak E [ | J (TB PuL. A
$
16
2/
16| 16
+ (1 -22 m min Oi
B| (1 7) On
Els +
(1 7)7 p Pminee Pmin Oa
+ +3L + +2Br Bn + Bu)ONEE + Lv (B + B2
2/5 Li 2
18 El
E|l V J. (TOL. QwL 1p)
E|
16
(1-7)72pi) m O
(1 (1- P axk
8|S L
ElEY, + anE y. I + +B
+ BH)ONER,
(1 - 2) P P2 Oz
fo < (1-))"" Pin and the last inequality oy L.bt
where the second inequality uses the step size condition is due to the step size condition
St < &
Bi and Et < ay Lv Lv +3Lv))
Br(Bo + Bg)
Bp
C.4 Proof of Proposition 4
We introduce technical lemma below, which bounds an important cross term in the proof of the proposition.
34

---

Lemma? 7 Under the assumptions and step sizes of Proposition 4 we have for all k. 2 0
BRVV,A
VV
The -u As
+ +BaG, (0k, Wk. Ph., Vy,,),, V Top Au Ps no Mwats Batty)
3)
EE 1 f,k
V non A Ph B (0%, Wks Mis Vr,+))19)
2
6 I Sk
61221225 BlEg]
+
B[l
J
(To Owr (x)) )=)
[E( +
(1 - 3kk
(1 - 3)
(1 - 7)
16L LEL Blsk|
Blay,al + 24L2
EleY,a + Lvv (Br + Bn + Bu) a
(1-
(1-20) 7)
(1 7)
The proof of the lemma can be found in Section D.7
By the definition of eY,, and the update rule Vy.K.
V f.
1IV V To Bunt1 PK+1 1|
B[TTBV (ViAA + ( PA. VIA. Sk, ak. bk, sk2) Vi 2 Auutsi 1|
< |VEK + (0kG Pus VyN Sk, ax bn, 8k) - Vg" mo Aup Buta 11
= -IVEA Vioa An + Br.G (0x, Wh. Pa VyN) + 3x Gy Pu Vy SR, an bn, (6) G was Pe Vys))
An V Vi Aup41 Px41
|| Vi Voo An + BrGG Wkr Pas V1xk,
+ 2 Pk. VyNe Sk, ax, 0b, 81) - G, (O Wk: Pux Vrxx)1
2 V7 s0
Awwy PE V M1 P2+1 ||
+ Bx fV -V, voo S- Pe + B, wks PaN, Vr Go Pake Vy sps abs bu, sh) GO(kk, Pine Vy,z))
+ (V1,2
Vj you Own AK + (OR Pu. Vr.)), Vj yo oww frk
V
Touts a Awzzi RCt1) )
(65)
where the first inequality follows from the fact that the projection to a convex set is non-expansive. To bound the first term of (65).
V.. V To Smu Ar + BgG (0k, Wki Pis V V1,b 1²
-I1V,,- Vi buz Re 12 B G (0k, Wk: Pux. 1112 + 2,8 (V), V Vu GG wkr Piko Vy,h)
= -IIV,K- - -V.th owr Ru 122 + B G (0k, Wk: Pux. V/,x) - G, wk. pix. Vj
Aw An )1
+ 2,xx (VI,- ViT Aur Px G (0k, Wk: Pr. Vy,) G Why Prk. V Tup Aww Pk ))
-IV1,* Vq thwn MK 122 G (Ok, Wk: Pux, Vy,x) Gg Wk: fake Vj
Top Aw An !1
+ +28k (Vi.*
Vi
Tu Ax
(7 (7T
1) (Vi Vi Top 19 A
|1 f.k Vi yr yoo Ak 122 Lo (V V v0 Aux
Aus As 112 + 2(7 1)BBIl1 1.x-
Vi bbwny Ax |12
< (1 (1 7)kx))
(66)
where the last inequality follows from the step size condition BR S
1-2
Zzd
For the third term of (65)
2 Fo Owns de Vj Nou Seess - 12t2 S 2L2 (1
- To. 112 + [|owai - -duzll + 110x411 - iixll?)
g
35

---

I 6L2 BB + BIbS + B
- 61 (Br + BD + B
(67)
Note that the fourth term of (65) vanishes in expectation. Collecting the bounds from (66). (67 and
Lemma 7 we have
Ele f.++l S (1
(1 (1- )B 7. 2 iss Vy Sk, aks bk. s/)) Gy Wh. Pr. v1
+ (Bp + BD - B4) a
(1 1 31)
Elll V, V A Ar + 1, Wk. Pr.. 1,
2
6L
Bl + 6 BlEg1
E|| y (TO2. Pwr. 1
(1 7).3k
(1 7)
(1 - 3)
16L DOL
24L3 BlEY,N) + Lvv + BD + BH)
R(e + (1 73
El +
(1 33x
(1 71-7),8
(1 7)-7)
6T4-5
S (1-
+ (1 (1
f.be
V
+8B8 + (6L2 + Lvv) BD )
2
6L 2
6L BlaE]
6L7-1EC
TE[1 / J (T0 Owni [1x)1
+ (1 3)
B[e8]
(1 3)
(1-
16L 16L2LE,L3EE B[e
2302 BEEYA
(1 - 3)
(1 B)
Bill JJ (TBL Pwr. (bs) 1 6 L3 E[E
S (1
4
(1 -1-7) 3
(1-7)3c
16L 16L2 E
2z
EE + - 3) BlE + 12B3,03
(1 - 33x
(1 7)
where the second inequality follows from ||Vy, ViTe - Au B (Ok, Wk: juc v1,+ S e/P established in (66), and the last inequality follows from the step size conditions
Bx C < 1-y and
2066
D& PN
2Bc
VoLe (B
The bound on B|EY,]] can be established using an identical argument, under the step size condition
as 4/6LyLD
D Proof of Lemmas
D.1 Proof of Lemma 1
We focus on proving the inequalities under the follower's reward and note that the same argument can be used for the leader's cumulative reward and value function.
We show the Lipschitz continuity conditions first. Fixing the mean field and leader's policy, the follower's problem reduces to a standard infinite-horizon discounted-reward MDP, in which it is well-known that the value function is Lipschitz and smooth with respect to the follower's policy parameter. Specifically, adapting Zeng et al. [202 [2 B.5), we have
V Te. ow The Our #
< (1 7)2 (110 0'll + 10w - -w l).
2
(68)
This means that to show 31 it suffices for us to show the Lipschitz continuity of V.radwni with respect to u. The value function can be expressed as
Σ c 9. (s')rra((a s')o(b| as,
V
7
bab
36

---

1
7 (g.00. p.14 r, ,
e (I - 1700
(69)
where e, RI denotes vector with entry and zero otherwise, and r3 r0 R denotes the marginalized reward such that
rf (..00s)) = Σ br, b, s)o(b) s).
We have from 669
x. N... Vi..ba111 11
7PT.rol -T T )
(1 -pa.0.1) -T [ AA - -1
(1 - --pprre -p -T 11 - pa. T-T r"g 17,00 + 11 pppniiaa-- (T( -rg )3
(1g
1dd Aineera - (I ?.
) yT (I- (--pprao (1 - -1----ppsss ) -Trjoon
+ [1 - 7 (m. - " u.n. 19.001
< ||[[ prmaa. T|| ypm. ST - (1 - p T [II - ypt...l -T
+ 11( - - pa. o. y , 1m - , -17.0/1/
1-7 ? pro.u paoalll ||
1
1-7 .VI1 +
1
rg 1gg0 r r5y0
1
TPTV V
Tor
|1 - -1'll |1 + 1-7 11--111
(1 7)2
ipTvis 7)
Lr ) |1ue -''ll
(70)
where the second last inequality follows from Assumption 1
Combining (68) and (70) implies 29 which obviously leads to 28) J p)) is simply
(V7.00 .p)..
To show the smoothness conditions, we note that again adapting Zeng al. (202] [emmaa B.5], we have
|| V ro. ou.t
V VoV t Mun S (1 71-
8 10 - 0'11. 11
VV Vo V. V.V S (1 7)3 ||w wlll.
This implies 30
To show
31 we differentiate Vj .0 with respect to μ. It can be seen from 69 that
V V 7 -I- (1- WH T (V (T.iH)))))) T (I 11---ppp00 (.l -T
ry R. (I pp -T Vurgon
Therefore, we have
|V VT V.VTTAN
<> - M ( (V.prrd (I - piiiii - T (1-00--11-ppp0 (V, (V...) (1 (1-ypriow)) p
lا|| (I - - T V V - (I -I- (
V V
1lys (1 - - -1--71a16n -T
|| Lp
I- Visi
37

---

1
(1(V T - -(V (V..pro 11 1 vlsl
+y
-11
1 1-y
n -T
+ 7 7-7
d1
(1 ypaness (1 -1-pppoo)))
11 1
+ 11-7
1
d.
1 1-7 |V w uuggou V V
1
+7 (V pT) [ - (V..P..)H))) || Lr
+7bp V n T.0 -V Fprg.0n111 ||
2/Lr |
- -pppoaa) -T - -1-- (1-aprr)))
11 I1
T
(1 (1- + t, ),1,)|17, 11V prion pT.
1-7
yLp
+ 71)
+ LD || u - V Vurgionul
2/IPV15)
(1 - (I-ap=aa)) -T |||| (I - - -I- (p T |||| (I- (r- )
1-7
|||
7Lp
+
+ +2Ln) Lp |l اlll (1- + +rLp) Lrll 111 k'll
7)3
2y [S]
711 μ -a - V.pan. + 2 ///S/
(1 (1- + tLbpLr I| w'll
(1 -733
27 2 S
2%
SI
+2yLpLr |1 n'll, (1-
(1 11-7) 7)3
where we have plugged in the smoothness bounds on P and ry from Assumption
D.2 Proof of Lemma 2
Note that the discounted occupancy measure can be expressed
p d = (1 - (1 - pr P.
(71)
This implies
Σ>< (dp To own M1
do 6) M1 dug prdd (s)
tug proez (8) 22 tug
Ma (. ())
(1 - ->) i #1 (I - --
11 Hi Dunz im 1 11 tnn 7ppro,
7 srred Uun 2z) pll
$ (1 - p|l|||||| pat, Oug Mi
|||(T 2z 2rmg prat
pmo, 7 &ur Hol -
+ (1 - -1]||||||| proz, Owg 12 [|]1 -
11) dws paod - (1 - 7 :prio tuyy 12)-
1
S - 7) ( 6||| - p2ll + |To - Teall || + 1lwl owall) 1-7
+ (11-))1(11-- P Puz -=2--111 III((1 7 Footo sug
02)_ - (1 poo oun P1)1II[[1 pr . oun (i)
$ [Plll11 - #2ll + || - Teall + 1l@wn @uzlll 11) + 1 1-7 || paod
Oun 12 proz- oug 122 ||
< 1 [Pelin- - u2ll + ||TO - To,ll 11 + 11bws - (bezll)
[Pllk pall + 101 82ll + ||w - wall).
where the second inequality follows from Assumption I and the fact that the state transition matrix is 1-Lipschitz in the policies, and the final inequality is due the 1-Lipschitz continuity of the softmax function.
38

---

D.3
Proof of Lemma
Note that Ve log (a can be expressed entry-wise as
log (a 8) = s7 (1 d'-aao
To (a' 8)),
(72)
20
which implies
s)](2 S Velog 8)]( S 1 1 2. Vo log To(a
(73)
By the definition of F.
[[F(0, n, V,,, S Vo log Ta {)]l ry(3, a, b, EE(T,, 8) + +>Vs(s') - Vg(())
- 2( + log A + BBv + Bv)
2(1 27 log Al| 2,
where the second inequality applies 77
By a similar argument, we can show
Vi..,, 2 Bv +
For H, we have
|| (H(,,)) 8)ll = |le, Mlll s ||-l|| + |al|| <
For the G operators,
|G,(0,,,,,, b, b,8 8')ll s (+)
+VV - Vy(s)
$ 1 (1
(1+1 7 log A + +yBr Bv)
= -1+ + 1og | +1
Similarly,
|[r(w, n, V,,s (1+ +1.
D.4 Proof of Lemma 4
By the definition of F.
[[ wI, M1, V1) F(02, u2, V22
||T
Top Augins .amma b- (1)) (ll,0.8) [F(01, M1, V1, 8, a b,,,
soud
E
[F(02, b,,
sp
T0s Awa Jannt, (oo
Tap Pun (S) s)o (b) s)pi (s' s, s,a.
9.0. h,o"
Thg Dwg M2 (s) m)m (b (s' s, a) F(B2, 12, V,,
+ E
[F(O uI, a,,,, F(02, V2,s,
snd,
f0 owi is
FF(1,, V1, K,,8,,,,,,) - F(02, V2. a,,
snd Top Augins Jarxo, (c1a),
+ BF
(af our #1 (s) (a s) (b s)pp (s' s,,, dp dun P2 (s) s) (s p)z |s,a))
s.a.b.s'
soud no Swg (1a)) (1))) (f (-14,a, [F(O,, V,,s,,, - F(O,, u2, V2, v,
39

---

+ BF
Σ Gg our Mi (s) (s's 5) dx owz 12 Ouz
(s) Own Pa (()1))) (74)
To bound the first term of (74).
[[ M1, s,a,,
",,,,,,,) b,5') F(O2, u2, s,,
|/V To, (a b)
- vi(a))
Volog Ta (a s)(ry +)
V2(())II
| To (a 8) V log To (al (,) +>Vii(ss') (is)
+ Va log ma (a ا|] [ ( b, -1
8) = 8) +>Vi(s') (o)(6'))~V((s))-000 (5((s) V2(s))
I 110 0211(1 + 1og |A| + (1 + 7) Bv)
(4 + 8 llg |A)T
+ 2 (Lrll #2l +
|101 82ll + (1 7) - -Val)
7)3
(4 + log [A)T
1 1 1 log |A| + By +
|10] 82ll + +-2 H2ll + 2(1 + + 1/21,
(1 7)3
(75)
where the second inequality is due to Lipschitz continuity of the entropy function (see Zeng et al.) Lemma 6)) and the boundedness and 1-L-ischhtzz continuity of Vo log no, which is obvious
2022 from
and 73
By Lemma
2 we have for the second term of 74
Bf
Σ
(df
(o Pun #1 (s) pr tmg M2 (s') 1 8) dp
Toz duz 12 p ouz dun M2 (s' $))
5,8'
Br
owg TTp
M1 pTo ou, M2 (. 1 s) dp
dun 12 (s) pTo2 Bun p2 8))
< BF
Lp H2ll + 100 - B21l + ||w, - wall)
(76)
Substituting 75 and 76 into
have
[[F(0,, w1: H1, 1/1) F(02, W2, 22, V22
1 log |A| + +) + (4 + 8 log (A()T)
110, 021 + +22,1lm1 #ell + 2(1 + + V/2
(1 - 7)3
BF
+ 1-7 [Lllm1 - H2ll + |10 021l + ||w| - wall)
1 + +log |A| + (1+ (4 8log (A))T
+ BF 110 - -02l
(1 1 7)3
+ (2Lm + BpLp)
BF
1-> || - Pell
+
|| - - + 4 Vell-
1 1-
This
obviously
leads
to the claimed result by setting
LF
1 |A| + (1 + By (4+8log (1-7)²
LADr
Br 1- 2L, + BeLe Br
max
The Lipschitz continuity of D can be shown using almost identical steps, which we will skip.
For H, we can adopt similar argument
[[( 41) H( W2, p2)
E T1. [H(41, 5) H (12, 3)]l| + Bu
Σ (dp
our pM (8) - dp T Awy M2 (s))
40

---

$ &1lm- - -2ell +
Σ op o (s) 22, ue
0s) (-1s) [s)
auz
(s) oun u2 8))
SImm -22ll + 1- 7
Lrllun - /22 + 110 0211 + 1|w1 - -wal)
Ln - 02 + ||w| w2ll + (alllllua - u22l);
with LH = -max bp 1-y, 1}.
For Gj. we again split ||G (
W1, /1: Vi) - (02, μ2, V2) into two parts and invoke Lemma 2
([G W1, M1, V1) - G (02g w2, H2: V/2]l
S |IB sndd 1o
(1a,,))) (Gy(0,, 11, V22 a, Gy M2, V2, a, b, 8]]l
Awi 11 aarto, s).
+ BG
(dp p oun
uwy 11 (s) pmn, p own pn (s' 8) - d0 twn 12 (s) proze Swn 12 (5' {8)
8,8"
rr(s, (, (5 a, b, p2) (Te 8) TE(Te,, 3) + 7V'( - VV2(5') - Vi((s + v3(s))
Bc
[Pllmn #2ll + 110, -21l + |!w1 - walll)
+ 1-
(4 + 8log [A)T
+ 1 - 110 - 0211 + (Ir + Bobp }|m #2ll + 1-7 ||اا - w2ll - 2 - --
Bc
)101
BG
713
1 1-8
Similarly,
|[Gi(w,, (4,, Vi) - Cilws, 12, 1/21l
|E s-dd t0 Awn Ps anwty Brww (118),,'~mm (1);,,b) (Gi(H,, V1, a, b, - Gi(H2, V2, s, b, s s'lll
Tos:
+ Ba
(dp ( pwn M4 (s) pr tun M4 (s') 8) dp twn 12 (s) pao own u2 (s') s))
< [1s b, M1) - r1(sss 1z2) + (s') - ~s2(('') - (8( + V2($))
Ba
+ 1 y LP||H11 - M2ll + 110 - -221 + |w- - w2ll
Ba 10 02l + (Lr + BaLp m| uall + I I- ||wا w2 |l + 2 V/2l.
Ba
1 1-7
D.5 Proof of Lemma 5
Within the proof of Lemma
we use the shorthand notation
yk fiu 1 (Owx)
) + Ex.id (iT" (iT(( (dwn (u),, wk. pxx).
As the operator 1* is Lipschitz, we know from the mean- value theorem that there exists w zwkk + (1 - 2)WK+1 for some scalar 2 E [0, 1] such that 1r" (Dwer (1) 11" (bbr))))== = Vun" (DwErs) ) [wu++1 wk) = SN VWW (Ow ) D(wks Puk: Vi ski bbs sit)
SA VEN (Owg..) ) D(wn: Pex: ViR)
+ Ca Vwll (DwEen) T (D(wn, fux. Vi.ee 8h, be, sk) - D(wk, dix, ViLa).
This implies
Blia 12"""" (dwe) + (Ow (ux), wky (ux)s 1tt (Dwrer) 1² (owe)))
41

---

-uBluu V (OWEE .' D(wk Pk. V1,KD1]
+ (yk VWH (Ówe_.) ) (D Pr. Vi Ske bbs sk)) D(wk Paa V(xk)) )1
KN VHH (owE, . D(wks Pkk Vo,K)]]
+ (yM (owEr)) - Vul (oux)) (D(ws, pus, Vi sn bu. s6r) D(wks Pes Vur)))
+ Lyye VWH (di))) ) BID(WWK uus Vi oks bks 5%6) - D(wes De: Vex) ) Fk-
[Ku Vul (owiri) )' "(uus in. V(,a))]
+ (yre (Ww (dwen)) (ull(ae)) " (D(wn, Pas Vi 8ks bx, D(w Pa: vix)) M
(77)
To bound the first term of (77). note that by Assumption and the fact that the softmax operator is 1 1-Lipschitz
Ca (yke VuHa" "owE, .) D(wes Prs Vi,)))
< LG 11832111 ((W Pk: V1,+)
(1 5)
11v1112
[DWw, Pus 12
16
(1 (1-8)5,
(1 5)
81.252
[D(Wk, 1tt ( Vowe H (Owe) )11
|lue|1
+
16
(1 -8)Ex
SL
[i(we> isx, Vi,+) D(wks n² (Dew), Voue (ow)))
(1 &)
E)
81.22
1lval1
WWdiiddwa 11) ( 11?
16
(1 111-8% (1-
16L
11βa - 11" (dwrr) )112
16L2L,,6
(1-8)Ek Vi. Vi i (dwn)) 112
(1 6)
(1 ER
8L271322
16L2LL352
11
WWJi(own 10 (1
ll - 1 (dww))l
16
(1 - -))
(1 (1-6)E+
32I. 12,52
|| wu
As
Vi
youn un (own) 1 (1- (1-
V1 Vi Vi
(1 -x)
(1 6)
8L21352
48
32L L3 EY
5
16
11val12
(1 - -)E
&
(1-
1 +
(1 (1-8)E4
(78)
where the fifth inequality follows from Assumption 6 To bound the second term of (77).
(yk, (V=Hi" (Dwe))) Valt" (Dew)) (wHl")ww)) < 2Bn V w/1h" (1 - Vwll (Www))l)
((ww,, Puks Vi.es sk, bb, .%) - D(we, Mr, V(x))
< <2Br\4BB|llvlll @ue.: @well ll1
< <2BDLCHB]1l,, (iess dwelll II]
< 2233
< 13L BB
(79)
Substituting (78
and 79 into
11" (bbre
Ex H
(TT (Pww (bx) wk: Px)): (Pwe - 1¹" (bwww))))
(1 -5) E 111lve111 ||
(1 - E) Els7| + 48I,1 T.7 ElE + 321.213,622 E|
<
16
(1 - 8)
(1 (1-58 (1-8)6L
42

---

+ B U/ 12 + L
(1 11- B1llv1
48
32L2 B + BZ
Bl + (1 -8)EE
8
(1 1)-6)Ex
1--)Ex
where the last inequality follows from the step size conditions ÇA - 1 and
La I 1-5
16B
D.6
Proof of Lemma 6
Within the proof of Lemma we denote Ih (ower Pis)) By the law of total expectation,
-V(V y (TTe Owr pax) an (T, Owe (ax) T===" (O sin) 2k+1
Sk Dwsk Puk. Vi.k Sk, bus s'e) (in, Bx)
Fa-1
(T22 Owr Px)) - V J ( Dwe fit) (Bug Ra), E V
Sa D( luso V(se) Ex wh: px)
}
[V S Pwk. (x)
2 Swee Pax) (own
7)T² Prnin P
min akk
Blll [ ownr (x) V VzJ 2 Own fbx) In (dun, (oun Ae) 11?1
4|S|LT
S
|S| 1
+ (1 pmin ak
D(l (alkD(wwspps Pr. V) + (1 7)7 PriinPPinn ax
Eill (TTA Wk. Px) 112
(1 - Prin Okk
(B - 11
4|51
+ 2|S1 C
E [[[ N¹ (Dwn): Viun 2 (dwa)) )139
3)² 2 PmiiPPmnn O
2/5 12 2 C
E E|| (Wk Pr. Vi,x) Dlwk: 1s (dwx). V N (dhws) )
+
(1 1r2pminn ak
2|5] L
+
E1 [ (owe fux), wks pux) - H. (IT (dwn) wes
wikk 1² (owz)))
(1 T) Prinn OR
+ 2/5] C
El (Te, Wky (x) - H(T (dwrr (iw) wks (kk))(x))))))))))))))) 12
(Tx:
prinu Pmin Ok
(1 - 7)7 pinin Priin ak
(owe, pax)) T&x 1
4|81
+ 2|5 C
Bl an (Pwe H4) luma [13]
7)T² Pmin Pmi ak
4181L3-1333
415113,L358 Blll Vi, Vo H (own) 2 [12] E fl 1 (dwz))l1 + (1 (1- P axk
+ (1 7) Pmin aR
2
+
L3 Blliiu - 1² (brre) + LL21iuu - 11" (dwe) [13]
(1 2 D2 O
2|5) L3
+ (1
(dweni)) - 11 -
iin 0g
(1 - 7)7 2 P an
( fax) - 123
1|.8
+ 2|S| 2281L3 3
E V Jilwe HC (0wz)))11] + 4181L3
Elli 1= (dww)))
pmin O
(1 7) Pimn O*k
8181L3-L332
E ou your u (dwn) ) [1
BlIl v Akk [12]
+
12 OF 2 p
(1 -1-783 (1-7)72 p ak
L3
23 Elll 77 t (doe. To 11
+
El Ellix 1 (hwe )13]+ (1 - 7) Pi 12 Ccs
(1 - 2 P P2 min Og
43

---

(1-7)r² >)² PhinnPiii ane B1[l (Owe - Tv, -.,
(1--) 2/ -Bll= Tpp 11
Blll=((wwe
4|5|
Piii Okk
2/
16|S|1,914,133
8|S B(EYA),
+
Eleg
Eleg] +
(1 - 7)2 P min Okk
(1 (1-r)rr(mntue?? C.
(1 - 7)² Piiut min
(80)
where the second inequality follows from the fact that H (Ø), Wk: 1¹ (Ø)) = 0, and the last inequality simplifies and combines terms under the step size condition
Çt < LL
$4
D.7 Proof of Lemma
Within the proof of this lemma, employ the shorthand notation 2i. [0k, Wks [k, ((xx))
Vj A
and
Uk Vy Vi Aug Pe B0KG wk, Pk, Vy,+)-
The value function is smooth, i.c. has Lipschitz gradients. As result, have from the mean-value theorem that there exists xx+1 = 20 + (1 2) 1 for some scalar & E [0, 1] such that
((xx)= - f(Ix+1)
v TT
Ik+1)
(VoV, The : 1u24." (itl) T
(0x - Okt1) + (v,
Tohts OubeZ (iti) (wx wk+1)
Top -0-4 )
(px pu+11)
Tof A pb+1
an
Vol
F(Os, Wey V5,6))
mog_
(F(Ok, was Pas Vy Sks ax bu. (k) P(On, who Pen V2,,)
an
V
Cx V,
D(wks Puks Vi.t.)
+ Ca
VoV,
#0h AuRt1 )
(Dlwn, Pas Vi, Bu, bu, 8%) - D(wks Pase Vini))
Ouit P7 1)
+ Ex
V
V
h+1
HTT Wes Pak)
+ Ek
VoV,
bu (R.)) T Tug.
(i 5&6) ) H (TO wks (kte):
(81)
where we denote 02
z0, + - =0)=2))))(,,.. = zwn + (1 - 2)Wx+ Pi zίt + (1 -2 2)
Plugging
(8 into the cross term of interest, we have
(VgR V Tum: B,.G, We. Pics VyK)), V Aur Vi 1. 1.eee Butty
((N,(([x)) {(xx+1))
=au(yer (VoV, THE uuwg ab+1
F(ORTWKiPM Prk. Vyxk))
+ (un.
(VoV, toats Au Mi-))
[(Or..n. pex. Vr.)), ou, on, be, (.) F(Ox, was pix, VY.H)):
V VuV, 2+ Buf pi.i)
+ Cx (ye.
Dlwes Prcs Viee))
VuVV Toj -un m
(D(Wk, fik, Vi 8k. bu, s6)) D(wk, fux, Vire))
+ Ca (yi
V. Mog 2- (R+1))
+ Ex (Mle
H (TTO wks Pw))
44

---

+ Ex (yke V "i s Pa Hin) ( H 5k) H (To wk. (ix)
(82)
We bound each term of (2) individually. First, by Young's inequality
On (yk VoV, PuEss (itt)) T
FOOK,Ww, Pu, VFL))
$ Lv 1l F F(8k, Wk. Pr, v1,+
(1 B)
aL
11ux|12
12
(1 (1-7)
(1 B)
11yx|12
6T2
[1F(0 Pks Vj Non Own Be I|
12
(1 3)
aL
|| wk. jux, VyKx) F(Os, wky Mis V Bun Sl )1
+ (1 3k
(1 3k
6L
aL
11
+ (1 (1-7)) 7)
VoJs( Own px)ll + (1-7)3k E fik
(83)
12
For the second term of (82) we take the expectation
a E (un.
(VoV, "o Bu )
T
(((kk, Wkr Puk: Vy Sk, oks by, 8/6) F(Kk, wkr Puk Vysk))
=anB((ve, (VoV, moRts Au Mi+1 VoVaou Rix) (F(OR, pes Vy 82, au bb, s/)) F F(OKs was Dis VrA))1
+ +anBlyyk (VoV Suy (F(O:: Wh: Pr: Vr Sky ak bys skt) - F(Ox. Wki Pk. V..)))
Top Sug Ah+ss
VoV Nurn An) T (F(Ok) was wRr Vy 8k, ax bx, bx, - F(Ox, wk, Mas, Vy.e))
= =oyk
VoV,
< 2B yu VoV, T
P- Al+1 V VoV Top ruu
22B Lvv (ecel/vxe1((111100
(1
To.ll + 1 - Smi || + |l 1 - Mxll)
2BrLvva |l [|T02]1 Toall 10wxx Smel || + || - imll)
2ByLvva,EBlla (x SxBD + BN
2BrLvV (BF + BD + B11
S Lvv + Bp Bh 2 + Lvv (Br BB + Bh)a
(84)
where the fifth inequality follows from the step size condition Ca S & S osk, and the second equation follows from
(VoV Ton Awn (y)
F(Or. Wh. Pa. VyA+ Sk, ak be, si - F(Ox. Wh. Pa. VEa)))
Elke (VoVz Bee) Re) F(o,, Wk, Du. Vr Sk, au, bn, 52) - F(On, Wky Pux V/H) Fx 1)-1]]
0.
The third term of 82 can be bounded similar to the first term,
oug ouE kh+1) )
(u (V
D(wK, fins Vi,e))
< LV DI mrx Vi,+))l
5 (1 B)
32
1y 1?2 + (1 7)33k
[1 pix 11
12
7)
18kk ||2
+
62
[ID(w 1²" (owe)) V. oun wf (own)) )
12
(1 - ~3
45

---

|ID(wa, uun V(,) - D(wks 1 (owe) ); Viou u (Owe) 912
+ (1 7)
(1 7)
6
+ (1 7)x /wwlwower (4) (omn)) 11?
12
12 13
1 - 1 (0wz))) =1²+ 12 13252
- B)x 11V, - (AwN) ( 112
+ (1 7)
(1 B)
6L3-TESE
12L3-L9252 llie - 1et (Owr)))
1x
+ (1 - 3)
V Wi 11 (own)))1
12
(1 -7)
2412 1.
[1 de uu (owe) ) 11
11 - V.eua .Bk
+ (1 +3
(1 7).3x
< 12
(1 - 3)
|
1| +
612
SK
&
(1-7)3k
2412 17 V
(1 7) E1,
(85)
where the fifth inequality follows from Assumption
G
For the fourth term of (82) we again take the expectation and use the technique in (84
Buk, VV 24t Au (im) (D(WKs Puk, VI 8k, bk) D(wks PRo V(R)))
a+ PuRti 1ā VoV ) (" (D(WRs Pu.. Ve ski by, 24²) D(wks Piks Vuo))l
KykE (VoV,
+ (WV
Vo Vor (oo T
(D(wss Pes Vi Sk bu sh)) - D(wn Pas Vixe)
FoRti PoRLi Mi+l
VoV bur Ad) (Dlwns Pk: Vi ski bus sb) - D(wks Piks Vixs))
= =CuB(Wyk VoV,
< <2Bp< yy V #os+ Augti Ahr1 VoV, Nou mow A ||]
< 2A ||
[| mo.ll + 11 Suill ll + 11bb41 Pall)]
2BDLvv DeDLLvCABllla B1 |1
[| - To.ll + 11o &well ll + 11AAt11 Rull)]
< 22BDLvV Bllla% |I|
Bg + BB + BH
< BBlvv (Br + Bp + B ) [1bb || + (pLvv(Br + BD + By)
(86)
For the fifth term of 82 recall the definition x in (2) and
To 1 Ees" ) (ToA.
T (TT,, Wk. (ik)))
Ex (yre (v
Lv Ex 11 ||||
Wk: fix)ll
(1 - 7)
312,52
1/
(T Wk. 112)
12
- 7)
(1 - -7)
6L7752
11
+ (1-10 (1
|| (TT (dwer fuxe, (k, Rx)) - H (TT (own): 4² (bhw )112
12
6L752
+ (1 kL
|| (TOg. Wky Pax)) H( (Ower (a), P1:11
11 3k
6L7-22
11
+ (1 k)
L' (liAs 1rt 1 (bwm))1 +
12llite +iff+I21lji 10 (own)))
12
6L9 22
+ (1 B)
||
x5" (owe, 11x)
(1 8B
12T.TRR (1
11
cx
ck
(87)
12
(1-
3k
7)
where the third incquality follows from the fact that H (TT ((), Wk, μ¹ (d)) 0.
46

---

Finally, for the last term of 82 we have in expectation
[yVE, V
Pi 1
H(Lk; (k) - (To was (r))))
[(x VoV, not
V Vor T
(H(ie 8k) H (Tow Wes (es)s()))
+EEE (voV Au R4)
(H( 5k) H (ToL
=C VoV, mo Mal1
Vo Vo (in) T (H (H(p): 5k) H (TO was (ire
s $2B F.BHC:E(lUnH Ellu Vo,, "og 1 11 Aats VoVj A11l II]
L2B (1 Toe + 1 -welll + +1lut+1 - Rull))
$28n Ex (|tOUtt - To. |1 + Ilaue-1 - Swell + |1Ax+11 - (ica))-aell)
S LVB Ellleanll (anBF= + CB + +b
$ BuLvv(BA + BD + Bn Bl 11 + BB LeLLvBBrr + BD + B1)
(88)
Substituting 88 88
into 82 we have
EKVY f.b
The
Pug An
+ , Wk. Pug Vy.)), V7th Vin V -024 Batty)
(1 - 3) B1llyx111
6T2
aT EleYau
[|[ V VoJo To Dwr (c,,,)))1++++
12
(1 33
(1 7)8k
+ BBLLLV (BF + +B + BB )o B |1lmm + BpLvv + BD + Bu j
(1 (1 Billex|111 623
36L, L3 El= + 2412 L BlEY,]
B[Ey0]
12
(1- (1-
(1 7)
(1- 3)
+ BDLLV (Br + BD BBr )o yk |/
+ Bp (Br + BD Bu )
7) B[llvn 111
61,2 T2 c2
S
12T. 12 [[=s]]
[b(s5]]
+
12
(1-7))3
(1 (1-7)8n
+ BHLVV DRFLVBBBF+ + BD + BH Bx) B M/ || + BBLLV (By + BD + BH)ay
3) E1lle11 121+ (1 3)
a
L,, E|= + (1 - 7) B[=
6L2
E|| 5 (T Dwrr Ax)1
4
(1-
(1 (1- 3k
12T. 122 12
E[E 41 + (1 (1-7)3 Bleyna + 24T.2 1.3.0 EECY,A
62
+
+
(1 8)
(1 7)
(1 (1
+ LBv + Bp + BB) 2 a 1l [1 + LBv + BB + (aa)
11 3) E[llyx1 1|31 + (1 3)
aL 2
612² L2
6LEn
E[ J (To,. Oun Ar 11 + (1 B)
ElER] + (1- 7) Eleg]
2
6L 2
23 L3 ElEY + Lvv (Br + BD + Bz)
+ (1 71
BlEY + (1 7)
where the last inequality is due to the step size condition min(1, ( ).
Eo Ex
S
LLg 31y hp
and as S
1-7
E Details on Example Satisfying Assumption 6
Example 1 Consider a hierarchical mean field game in which the leader is stateless (S: = 0), and the follower's 's state space Sy, the follower's 's action space A, and the leader's 's action space all have equal cardinality n. Let the actions and states be indexed as = L, n A, 1,... n
47

---

B,sy 1, E S. Suppose the transition kernel and reward function ri satisfy the conditions [1) 1(b 1) u(b),
Pij a.. 1(s) 1),
where 1(-) denotes the indicator function (1 if the condition holds, 0 otherwise). We drop st as argument from Tı, since the leader's 's state is vacuous. The follower's s reward y can be any function of 8z, a. b. This MFG satisfies Assumption 6 with 1Ι1 2, 122 1/2. Clearly, the independence ussumption in Cui et al. (2024] is violated as the leader's reward depends on the mean field and the follower's 's reward may depend the leader's 's action.
Below we justify that Example 1 inndedd satisfies Assumption but not the leader-follower indepen- dence assumption in Cui et al. 2024
The mean field induced by any leader's policy is the same and satisfies the following μ² (+) = 1(s = 1).
As the leader is stateless, we have
(
(b)rr (b)) = [(b= 1)0
B
BEB
Σ BEB
(b))
1(b = 1)+ 1(b 1)) 20(1).
As result,
V+D())= = 21)
Similarly, we have
J( = Σ o(b) (166)
1(b 1) 1)+1(b)) = =0(1)= + Σ @()))(b),
bEB
which leads
VuJi(OwaH la=a" (1)
Since ((L)) = 2V 2V 1)) (a=)) the inequalities in Assumption 6 obviously hold with 1/ = and 7/2 = 1/2.
F
Appendix: Environment Definitions
F.1 Market Entrance
Followers Followers decide whether or not to buy (left) or sell (right) A = (buy, sell} a good at each timestep, leading to the corresponding state $*+1 ag. S = A. The payoff for followers is based on the proportion of other agents choosing the same action:
r (ak: bb, Sk, m4s) - (ax) + bonus Laz=by:
where bonus is awarded for choosing the same action as the leader, discussed below. The transition kernel is independent of the mean field and a deterministic function of the action of the representative agent:
P(sk+1 = s' sk = s, ag = = =
Leader We introduce a leader into this market, where the leader is trying to steer the mean field towards some desirable market state goal, for example, buying (goal = buy). selling (goal sell), or balanced market (goal ER (buy, sell)), by taking actions to bonus these states by B {buy, sell]. This can be seen as, for example, a market maker skewing prices to make certain trades more desirable, e.g., for managing their inventory. The reward for the leader is:
ri (sk, bb, pa) a pu (goal) + v(bx- goal).
where 0 < v & is a small reward shaping term, providing more immediate feedback to the leader than the delayed feedback from the mean field.
48

---

state
state
state
state
state 2
Figure 5: Beach Bar Environment. The blue stars indicates the fixed bar position, and the golden star shows the leader can add a new bar anywhere within the existing state.
F.2
Shop position
The state space S covers all the potential locations in a I-dimensional ring, where there is (fixed) desirable location in one of these states, as visualized in Fig. [5]
Followers The followers are rewarded hased on their proximity to the nearest desirable location (either by well the crowdedness of the state they currently in:
r (8k; (k: bk, 14a) min (dintsss(ss,, sk. *), dist( Sk: b&)) log(()((s:))
where 0 is coefficient specifying the follower's preference to avoid the crowd.
Leader The leader is rewarded based on the business of their chosen location: ri(sk: bys m4x) = μu (bb) e.g., more customers visiting store can lead to higher profits, making the positioning more desirable.
F.3 Equilibrium price
In this environment, the followers are homogeneous firms producing identical good, where the price determined by the (endogenous) supply-demand equilibrium around some baseline demand Each timestep. the representative firm chooses production quantity qs and replenishment quantity hy based on their inventory level captured by the state 8g. The state/inventory transitions according 5% min(qs, $&) hk.
to: 9k+1
The followers are rewarded based on the resulting (endogenous) price and their chosen quantities as:
T (s; a, b, u) (ps- - co)as- - ci - c2h4 - (e2 + ca) ca)max 8k. 0)
C482
(89)
where C2 are cost terms penalizing different inventory management and production capacities, and the price is determined based the supply-demand equilibrium pı
d 1/a
where
Eggl
Elgl = - u(inv) x[(g. h), inv] inv
inv h
the expected inventory of the mean field.
We introduce leader into this environment, where the leader attempts to encourage firms to maintain some target inventory level by taking actions to set the inventory holding cost c2. For example, large target inventory level would encourage firms to build buffers to withstand future supply shocks, inventory of zero would encourage optimal production under known and predictable demands. The leader is rewarded by
TF = (E[g) - i)² + + (Elg) - i)]
(90)
where the first term is based on the inventory gap, and the second term is again a reward shaping term for more immediate feedback.
G Additional Experimental Results
provide the evolution of the mean fields and resulting metries across the environments Figs. to for market entrance, shop positioning, and equilibrium price respectively.
49

---

Nearterd Time
190
190
740
160
1.00
buy skew
buy skew
-0.25
0.0
-0.50
0.25
-0.5
0.75
-1.00
-1.0%
0
500
1000
1500
2000
Iterations
Iterations
Iterations
(a) Goal=buy
(b) Goal=balance
(c) Goal=sell
Figure 6: Market Entrance: Evolution of follower mean fields μ (top row) and resulting skew (bottom row) under different leader goals.
on Heantels Over
Evelution Heantald Dver
2.0
1.5
distance
1.0
dislance
1.0
distance
1.5
MW
1.0
0.5
0.0
0.0
500 Iterations
1000
1500
2000
1000 Iterations
Iterations
(a) c 0
(b) c = 2.5
(c) C 00
Figure 7: Beach bar: Evolution of follower mean field μ (top row) and distance between the chosen bar position 6 (leaders action) and the fixed bar position (bottom row) under varying crowd aversions
H Experimental Details
We run each approach for 20, 000 iterations (plots show every 10th iteration). Each run repeated 30 times, with seeds set based on the run index for reproducibility.
Proposed. The proposed
with Co 0.5, ar 0.25, 3o 0.02, Eo 0.25.
PPO. For the PPO implementation, we base the implementation off CleanRL (in Torch), using a batch size of 256, hidden layer shape of (64, 64), learning rate of 3e 4. TanH activation functions for the hidden layers, and clipping epsilon 0.2. ADAM is used as the optimiser (as implemented in torch).
Nested. To ensure fair comparisons, the nested comparisons are run with the same hyperparamaters as the proposed. However, follower (resp. leader) specific parameters are decayed at rate based only on the number of follower (resp. leader) updates.
All approaches are run on a CPU, with Python3, and on an Amazon EC2 with Rói.large.
50

---

200
1900
150
1807
1.00
Inventory
Inventory
Inventory
4.0
2.5
eventery
2.5
2.5
2.0
2.0
3.8
0.
6.0
1,000
2020
0.
(a) Lean (Minimise inventory)
(b) Medium target inventory
(c) Robust (Maximize inventory)
Figure Equilibrium pricing: Evolution of follower mean fields μ (top row) and resulting inventories (bottom row) under different leader goals.
51


Reviews:
[{"review_id": "Jfo5QMt161", "reviewer": "Reviewer_qsnQ", "summary": "This paper tackles the problem of computing optimal leader policies in Stackelberg mean field games—an important framework for modeling strategic interactions in settings with a committed policy designer and a large population of followers, with applications in both public and private policy (e.g., economic platforms, government interventions).\n\nA central challenge in this setting is that each leader policy induces a new equilibrium in the follower mean field game, typically requiring a nested-loop optimization: the follower policy and mean field must converge before the leader can update. To address this, the paper adopts a single-loop algorithm that jointly updates the leader, follower, and mean field using carefully tuned step sizes, under appropriate assumptions on agent objectives. While the single-loop structure itself is not novel, the key technical contribution lies in relaxing the commonly assumed strong convexity of the follower’s objective. The proposed method is empirically evaluated against nested-loop baselines and MARL approaches across several benchmark mean field game environments.", "questions": "- In the main text, you refer to the single-loop algorithms by Hong et al. as the foundation that your work generalizes. However, Appendix E presents examples that seem more directly related to the framework of Cui et al., whose assumptions appear more restrictive than those in Hong et al. I assume that extending the approach of Hong et al. to Stackelberg mean field games is non-trivial, and that your contribution is better viewed as a generalization of Cui et al.’s work—rather than Hong et al.’s. Could you kindly clarify this point?\n- I may have missed this in the paper, but could you elaborate on the rationale for using a tabular softmax policy parameterization? Is this choice required for the theoretical analysis, or was it made primarily for experimental convenience?\n- In the equilibrium pricing game, it appears that ADAGE outperforms your proposed method. Could you comment on why this might be the case, and whether all baselines—including ADAGE—are converging to equilibrium in your experiments?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper tackles a timely and important problem in policy design, contributing an algorithmic improvement that is both technically sound and relevant.\n- The contribution is clear and the authors also provide example of settings that can be considered thanks to this technical generalization\n- The writing is clear, and the experimental evaluation is thoughtful, with careful analysis of the learned policies.\n\nWeaknesses:\n- The technical novelty of the proposed algorithm appears incremental in light of prior work by Hong et al. on single-loop methods.\n- While the experiments are well-executed, they are conducted in relatively simple environments and are restricted to tabular softmax policies, which limits the generalizability of the results.\n- The formulation assumes a well-posed leader problem, including the existence and uniqueness of the induced mean field equilibrium—guaranteed only through sufficiently large regularization weights. Although common in the literature, this assumption may limit the method’s practical applicability.", "was_revised": true, "final_justification": "The authors’ rebuttal has clarified the issues I had raised. I am therefore leaning toward recommending acceptance.", "rebuttal": "We thank the reviewer for the time, effort, and valuable feedback on our paper. Please find our response below, which we have incorporated into the revised paper.\n\n> The technical novelty of the proposed algorithm appears incremental in light of prior work by Hong et al. on single-loop methods.\n\n**Response:** Our analysis differs from that of Hong et al. [2023] in three key aspects, presented below in order of technical novelty and significance. We have revised the paper to include a detailed discussion of these distinctions in Section 4.1.\n\n1) The lower-level objective is strongly convex in Hong et al. [2023], but is non-convex and only satisfies the PL (Polyak-Lojasiewicz) condition in our case. Analyzing single-loop bi-level algorithms *without* strong convexity is a significant challenge, and handling this challenge is a main technical innovation of our paper, as we now explain.\n\n    To analyze a bi-level optimization algorithm, we need to bound the residual of the lower-level decision variable (in our context, the follower's policy) and show its iteration-wise reduction. If the lower-level objective $J_ f$ were strongly convex, we would select the residual to be $\\tilde\\epsilon_ k^\\pi = ||\\pi_ {\\theta_ k}-\\pi^\\star(\\phi_ {\\omega_ k},\\hat{\\mu}_ k)||^2$ and show that $\\tilde\\epsilon_ {k+1}^\\pi-\\tilde\\epsilon_ k^\\pi$ is approximately negative. As the learning target shifts from iteration $k$ to $k+1$, bounding $\\tilde\\epsilon_ {k+1}^\\pi-\\tilde\\epsilon_ k^\\pi$ requires controlling the drift $||\\pi^\\star(\\phi_ {\\omega_ {k+1}},\\hat{\\mu}_ {k+1})-\\pi^\\star(\\phi_ {\\omega_ k},\\hat{\\mu}_ k)||^2$, which can be easily bounded by $\\mathcal{O}(||\\phi_ {\\omega_ {k+1}}-\\phi_ {\\omega_ {k}}||^2+||\\hat{\\mu}_ {k+1}-\\hat{\\mu}_ {k}||^2)$ under the Lipschitz continuity of $\\pi^\\star$. \n    \n    However, without strong convexity, we cannot consider the residual $\\tilde\\epsilon_ k^\\pi$ (as $\\pi^\\star$ may not be unique in general) and instead employ $\\epsilon_ k^\\pi$ defined in Eq.(15), a residual in the value function space. We would like to bound the learning target shift $J_ f(\\pi^\\star(\\phi_ {\\omega_ {k+1}},\\hat\\mu_ {k+1}),\\phi_ {\\omega_ {k+1}},\\hat\\mu_ {k+1})-J_ f(\\pi^\\star(\\phi_ {\\omega_ k},\\hat\\mu_ k),\\phi_ {\\omega_ k},\\hat\\mu_ k)$, preferably still by $\\mathcal{O}(||\\phi_ {\\omega_ {k+1}}-\\phi_ {\\omega_ {k}}||^2+||\\hat{\\mu}_ {k+1}-\\hat{\\mu}_ {k}||^2)$ to avoid rate deterioration. However, the Lipschitz continuity of $J_ f$ only yields a loose bound on the order of $\\mathcal{O}(||\\phi_ {\\omega_ {k+1}}-\\phi_ {\\omega_ {k}}||+||\\hat{\\mu}_ {k+1}-\\hat{\\mu}_ {k}||)$, where the norms are not squared. Detailed in Proposition 3, we develop a careful error decomposition scheme to break down the residual difference $\\epsilon_ {k+1}^\\pi-\\epsilon_ k^\\pi$, and tightly bound the decomposed terms based on the PL condition, thereby avoiding the suboptimal bound implied by naive application of the Lipschitz property of $J_ f$. \n\n2) The technique highlighted above allows us to achieve the same convergence rate in SMFGs as in bi-level optimization with lower-level strong convexity. We further improve upon the rate in Hong et al. [2023], leveraging a recent advance in the multi-time-scale stochastic approximation literature. Specifically, Han et al. [2024] shows that a faster convergence rate can be achieved under lower-level strong convexity, if the lower-level learning target (equivalent to operators $\\mu^\\star,\\pi^\\star$ in our context) has Lipschitz gradients. Our work adapts and extends the argument to the case where the lower-level objective is nonconvex.\n\n3) Third, the analysis in Hong et al. deals with a bi-level optimization problems with two decision variables. We describe our problem as a bi-level problem for the ease of understanding, but we in fact have to deal with three decision variables on three levels (leader's policy, follower's policy, mean field). Although the argument used to analyze two levels may carry over in principle, actually implementing the analysis in the presence of a third level entails significantly greater complexity and effort.\n\n> While the experiments are well-executed, they are conducted in relatively simple environments and are restricted to tabular softmax policies, which limits the generalizability of the results.\n\n**Response:** We agree that some experiments associated with MFGLib are simplistic; however, worth mentioning is the fact that this collection of previously constructed environments must be extended in a particular manner to incorporate leader-follower dynamics, which is itself a contribution of this work. \n\nIn addition, we note that we conducted the Equilibrium Price experiment both in the tabular setting and under function approximation. Figure 2c) shows the results in the tabular setting, while Figure 4 shows the results when neural networks are used to represent the leader's policy. Please kindly refer to Remark 2 for a discussion on how to make our algorithm compatible with large state/action space and general function approximations, including neural networks. \n\n> In the main text, you refer to the single-loop algorithms by Hong et al. as the foundation that your work generalizes. However, Appendix E presents examples that seem more directly related to the framework of Cui et al., whose assumptions appear more restrictive than those in Hong et al. I assume that extending the approach of Hong et al. to Stackelberg mean field games is non-trivial, and that your contribution is better viewed as a generalization of Cui et al.’s work—rather than Hong et al.’s. Could you kindly clarify this point?\n\n**Response:** Our paper closely relates to Hong et al. in that we also study a bi-level optimization (loosely speaking) and analyze the convergence of a single-loop algorithm. However, from a technical perspective, we would not consider our work an extension of Hong et al. [2023], as the innovations discussed above to handle lower-level non-convexity and to achieve a faster convergence rate make the key technical analysis substantially distinct. The assumptions made in Hong et al. [2023] and our work may not exactly match, as they study a general optimization framework, and we specifically consider SMFGs and impose problem-specific assumptions.\n\nAs the reviewer correctly pointed out, our assumptions align more closely with those in Cui et al. [2024], which also studies MFGs with a hierarchical structure.\n\n> I may have missed this in the paper, but could you elaborate on the rationale for using a tabular softmax policy parameterization? Is this choice required for the theoretical analysis, or was it made primarily for experimental convenience?\n\n**Response:** The tabular softmax parameterization is only for the purpose of theoretical analysis. The algorithm itself can be implemented with any function approximation, which we discuss in detail in Remark 2. In fact, we have conducted the Equilibrium Price experiment both in the tabular setting and with function approximation.\n\nWe consider the tabular softmax parameterization in our analysis, so that we can adapt the innovations by Mei et al. [2020] on establishing the non-uniform Polyak-Lojasiewicz condition for the follower's objective. This condition allows us to bound the follower's optimality gap by its gradient norm.\n\n\n> In the equilibrium pricing game, it appears that ADAGE outperforms your proposed method. Could you comment on why this might be the case, and whether all baselines—including ADAGE—are converging to equilibrium in your experiments?\n\n**Response:** Yes, they reach equilibrium in the sense that the leader’s and every follower’s behavior converge to a stationary point and the magnitude of their updates tends to zero over time. \n\nADAGE indeed gets a slightly higher leader’s reward in the Equilibrium Pricing environment (Fig 2c). This environment is particularly challenging and exceeds the representational capacity of purely tabular methods, as the action space of the leader is continuous [-1,1], as we explain in Remark 2. ADAGE leverages function approximation, by parameterizing the leader’s and followers’ policies by a neural network, whereas the proposed algorithm shown in Figure 2c) does not use function approximation but rather uses a tabular policy with the leader's action space discretized into {-1, -0.5, 0, 0.5, 1}. The loss of precision due to the discretization may be a main cause of the performance gap.\n\nWe discuss in Remark 2 how function approximation can be integrated with the proposed SMFG approach, and conduct an additional experiment in the Equilibrium Pricing environment where we parameterize the leader's policy by a neural network. The results are shown in Fig 4. From Fig 4a, we see a much faster convergence rate for the proposed algorithm, showing the benefits of integrating function approximation in complex environments.\n\nWe thank the reviewer for the question and have added the discussion above to the revised version of our paper.\n\nReferences\n\nYardim, B., Goldman, A. and He, N., 2024. When is Mean-Field Reinforcement Learning Tractable and Relevant?. arXiv preprint arXiv:2402.05757.\n\nZhang, F., Tan, V., Wang, Z. and Yang, Z., 2023. Learning regularized monotone graphon mean-field games. Advances in Neural Information Processing Systems, 36, pp.67297-67308."}, {"review_id": "hCAzpgUKG1", "reviewer": "Reviewer_XdJs", "summary": "This paper investigates the strategy optimization problem in Stackelberg mean-field games (SMFG). The authors propose the AC-SMFG algorithm, a single-loop actor-critic algorithm that operates based on continuously generated Markov samples. Under a novel \"gradient alignment\" assumption, the paper not only proves that the algorithm converges to a stationary point of the Stackelberg objective at a rate of $\\tilde{O}(k^{-1/2})$ within finite time and finite samples but also validates its effectiveness through comprehensive numerical experiments.", "questions": "Could the authors provide a paragraph introducing the techniques used in the convergence analysis of the AC-SMFG algorithm?", "limitations": "See weakness.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n（1）This work establishes the first non-asymptotic convergence guarantee for SMFG algorithms.\n（2）The AC-SMFG algorithm exhibits a significantly faster non-asymptotic convergence rate than previous SMFG algorithms, achieved under more challenging problem settings where the lower-level problem of the bi-level optimization is non-strongly convex.\n\nWeaknesses:\nThe authors should expand the discussion on the \"gradient alignment\" (Assumption 6). For instance, although $\\nabla_w \\Phi(\\phi_w)$ and $\\nabla_w J_l(\\phi_w, \\mu)_{\\mid\\mu=\\mu^* (\\phi_w)}$ differ by a derivative operation with respect to $\\mu^*(\\phi_w)$, it is necessary to analyze the impact of this difference in specific problem instances and whether the gradient alignment assumption still holds. As mentioned in the paper, the problem setting is more challenging than previous works. Thus, a paragraph discussing the techniques used in the convergence analysis should be added after Theorem 1 to highlight the improvements over prior research. The current description in the Introduction—i.e., developing an analytical argument to handle bi-level optimization with lower-level Polyak-Lojasiewicz (PL) condition—is too vague.", "was_revised": true, "final_justification": "The authors' responses in their rebuttal have adequately addressed my concerns, leading me to decide to maintain my original score.", "rebuttal": "We thank the reviewer for the time, effort, and valuable feedback on our paper. Please find our response below, which we have incorporated into the revised paper.\n\n>The authors should expand the discussion on the \"gradient alignment\" (Assumption 6) ... it is necessary to analyze the impact of this difference in specific problem instances and whether the gradient alignment assumption still holds.\n\n**Response:** We appreciate the reviewer's comment on more discussion on Assumption 6. The leader's objective $J_ l(\\phi,\\mu^\\star(\\phi))$ has two components. The first is the partial gradient $\\nabla_ \\phi J_ l(\\phi,\\mu)\\mid_ {\\mu=\\mu^\\star(\\phi)}$ directly due to the leader's policy, and the second is $\\frac{\\partial J_ l}{\\partial \\mu^\\star(\\phi)}\\cdot\\frac{\\partial \\mu^\\star(\\phi)}{\\partial \\phi}$, which is the indirect component due to the effect of leader's policy on the mean field. The indirect partial gradient may be arbitrary in general SMFG instances, causing the gradient alignment assumption to break. For the assumption to hold, we need the SMFG to have at least one of the two following characteristics: 1) the leader's reward is such that $\\nabla_ \\mu J_ l(\\phi,\\mu)$ has a much smaller magnitude than $\\nabla_ \\phi J_ l(\\phi,\\mu)$, which allows $\\nabla_ \\phi\\Phi(\\phi)$ and the direct partial gradient $\\nabla_ \\phi J_ l(\\phi,\\mu)|_ {\\mu=\\mu^*(\\phi)}$ to always form an acute angle, and 2) the indirect gradient is always roughly \"aligned\" with the direct partial gradient. We provide a non-trivial example in Appendix E that falls into the second case and satisfies the gradient alignment assumption, but violates a comparable leader-follower independence assumption introduced in the recent related work Cui et al. [2024]. \n\nThough Assumption 6 is a relaxation of existing assumptions, we agree that it may not always hold in practical problems. This provides an important motivation for us to carry out extensive numerical simulations in environments that potentially violate the assumption. The environments in Section 5 cannot be verified to satisfy the gradient alignment condition, but the proposed algorithm still outperforms existing methods, demonstrating empirical effectiveness even in situations where the current analysis does not strictly apply.\n\n\n> * As mentioned in the paper, the problem setting is more challenging than previous works. Thus, a paragraph discussing the techniques used in the convergence analysis should be added after Theorem 1 to highlight the improvements over prior research. The current description in the Introduction—i.e., developing an analytical argument to handle bi-level optimization with lower-level Polyak-Lojasiewicz (PL) condition—is too vague.\n>* Could the authors provide a paragraph introducing the techniques used in the convergence analysis of the AC-SMFG algorithm?\n\n**Response:** We thank the reviewer for suggesting an in-depth discussion of the main technical innovation enabling the fast convergence rate and treatment of lower-level PL condition. Please find the discussion below. We will add a paragraph on the same in Section 4.1 after Theorem 1.\n\n1) In bi-level optimization and two-time-scale stochastic approximation literature, the lower-level objective is typically assumed to be strongly convex (for example, see Hong et al. [2023]). However, in our case, it is non-convex and only satisfies the PL condition. Analyzing single-loop bi-level algorithms without strong convexity is a significant challenge, and handling this challenge is a first main technical innovation of our paper, as we now explain.\n\n    To analyze a bi-level optimization algorithm, we need to bound the residual of the lower-level decision variable (in our context, the follower's policy) and show its iteration-wise reduction. If the lower-level objective $J_ f$ were strongly convex, we would select the residual to be $\\tilde\\epsilon_ k^\\pi = ||\\pi_ {\\theta_ k}-\\pi^\\star(\\phi_ {\\omega_ k},\\hat{\\mu}_ k)||^2$ and show that $\\tilde\\epsilon_ {k+1}^\\pi-\\tilde\\epsilon_ k^\\pi$ is approximately negative. As the learning target shifts from iteration $k$ to $k+1$, bounding $\\tilde\\epsilon_ {k+1}^\\pi-\\tilde\\epsilon_ k^\\pi$ requires controlling the drift $||\\pi^\\star(\\phi_ {\\omega_ {k+1}},\\hat{\\mu}_ {k+1})-\\pi^\\star(\\phi_ {\\omega_ k},\\hat{\\mu}_ k)||^2$, which can be easily bounded by $\\mathcal{O}(||\\phi_ {\\omega_ {k+1}}-\\phi_ {\\omega_ {k}}||^2+||\\hat{\\mu}_ {k+1}-\\hat{\\mu}_ {k}||^2)$ under the Lipschitz continuity of $\\pi^\\star$. However, without strong convexity, we cannot consider the residual $\\tilde\\epsilon_ k^\\pi$ (as $\\pi^\\star$ may not be unique in general) and instead employ $\\epsilon_ k^\\pi$ defined in Eq.(15), a residual in the value function space. We would like to bound the learning target shift $J_ f(\\pi^\\star(\\phi_ {\\omega_ {k+1}},\\hat\\mu_ {k+1}),\\phi_ {\\omega_ {k+1}},\\hat\\mu_ {k+1})-J_ f(\\pi^\\star(\\phi_ {\\omega_ k},\\hat\\mu_ k),\\phi_ {\\omega_ k},\\hat\\mu_ k)$, preferably still by $\\mathcal{O}(||\\phi_ {\\omega_ {k+1}}-\\phi_ {\\omega_ {k}}||^2+||\\hat{\\mu}_ {k+1}-\\hat{\\mu}_ {k}||^2)$ to avoid rate deterioration. However, the Lipschitz continuity of $J_ f$ only yields a loose bound on the order of $\\mathcal{O}(||\\phi_ {\\omega_ {k+1}}-\\phi_ {\\omega_ {k}}||+||\\hat{\\mu}_ {k+1}-\\hat{\\mu}_ {k}||)$, where the norms are not squared. Detailed in Proposition 3, we develop a careful error decomposition scheme to break down the residual difference $\\epsilon_ {k+1}^\\pi-\\epsilon_ k^\\pi$, and tightly bound the decomposed terms based on the PL condition, thereby avoiding the suboptimal bound implied by naive application of the Lipschitz property of $J_ f$. \n\n2) The technique highlighted above allows us to achieve the same convergence rate in SMFGs as in bi-level optimization with lower-level strong convexity. We further improve upon the rate in Hong et al. [2023], leveraging a recent advance in the multi-time-scale stochastic approximation literature. Specifically, Han et al. [2024] shows that a faster convergence rate can be achieved under lower-level strong convexity, if the lower-level learning target (equivalent to operators $\\mu^\\star,\\pi^\\star$ in our context) has Lipschitz gradients. Our work adapts and extends the argument to the case where the lower-level objective is nonconvex."}, {"review_id": "ouVLVVUwIZ", "reviewer": "Reviewer_pcnH", "summary": "This paper proposes a learning algorithm for Stackelberg mean field games where a large number of followers follow a single leader. The proposed method is an actor-critic model that utilizes two trajectories to estimate the occupancy measure and the mean field distribution. Under the key gradient alignment that allows the improving gradients updates for the leader, the proposed method has a finite-sample convergence rate that improves upon the existing results for general bilevel optimization under convexity assumptions.", "questions": "-Please explain the necessity of the entropy term. Is this term common in literature? Especially for the papers compared for the convergence rates. If the compared papers, does not have this term, I suggest the authors to make this an assumption or greatly emphasize that their setting is different than the others’.\n\n-When does the gradient alignment assumption hold? Is it possible to make a conclusion on the validity of the assumption based on the objective functions?", "limitations": "Yes, but the discussion on limitations could be extended.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- Despite its technically dense content, the paper is written excellently. The existing methods, their weaknesses, and the proposed solution are explained well. \n- Under some assumptions (see the relevant weakness and question), the theoretical result improves upon the existing SMFG methods that only provide asymptotic convergence guarantees. Additionally, the proposed method improves upon the bilivel optimization methods that are applicable to the SMFG setting with additional assumptions. \n-Theoretical findings are supported by various numerical experiments.\n\nWeaknesses: \n-The problem setting has an unexplained addition compared to the existing formulations: The followers optimize an entropy regulated objective. To the best of my understanding, this term does not exist in the compared works (for example [1] and [2]). Furthermore, the followers are called ‘rational’, given that entropy-regularized objectives are very common in representing boundedly rational agents. I think that ‘rational’ does not accurately describe the followers considered in this paper. \n- The assumption on the gradient alignment may be required to derive the theoretical result. However, the authors should discuss in more detail when this assumption does not hold. I believe that when the leader and followers have conflicting objectives, for example a zero-sum setting, this assumption will break. \n\n\n[1] Cui, Kai, et al. \"Learning discrete-time major-minor mean field games.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 9. 2024.\n\n[2] Guo, Xin, Anran Hu, and Jiacheng Zhang. \"Optimization frameworks and sensitivity analysis of Stackelberg mean-field games.\" arXiv preprint arXiv:2210.04110 (2022).", "was_revised": true, "final_justification": "The authors have addressed my concerns in their rebuttal. I will keep my original score of 5 (Accept).", "rebuttal": "We thank the reviewer for the time, effort, and valuable feedback on our paper. Please find our response below, which we have incorporated into the revised paper.\n\n1)  >* The problem setting has an unexplained addition compared to the existing formulations: The followers optimize an entropy regulated objective. To the best of my understanding, this term does not exist in the compared works (for example [1] and [2]). \n>* Please explain the necessity of the entropy term. Is this term common in literature? Especially for the papers compared for the convergence rates. If the compared papers, does not have this term, I suggest the authors to make this an assumption or greatly emphasize that their setting is different than the others’.\n\n**Response:** Our work adds regularization to ensure that Assumption 4 (optimality-consistency operator contraction condition) holds. The regularization (or the related Assumption 4) is not introduced in our paper to study Stackelberg MFGs, but rather is a common assumption inherited from the (standard, non-hierarchical) MFG literature. Below we explain how and why we adopt this assumption, and clarify its relationship to prior work.\n\nIn general not all MFGs (even without the leader) admit polynomial time solutions, a fact proved in Yardim et al. [2024]. This implies that only certain special subclasses of MFGs are tractable. Currently, three subclasses are known to admit efficient solutions. The first, and most widely studied, consists of MFGs that satisfy the contraction condition (guaranteed by a sufficient regularization). Papers making the contraction assumption includes our work, as well as Mao et al. [2022], Zaman et al. [2023], and others. The second solvable subclass includes MFGs satisfying the Lasry-Lions monotonicity condition, which can also be guaranteed if a sufficient regularization is added. Existing works making the Lasry-Lions monotonicity assumption include the paper [2] (Cui et al. [2024]) that the reviewer mentioned. The third subclass is the herding class introduced in Zeng et al. [2024]. We provide a comparison on the subclasses of solvable MFGs and the corresponding papers in the table below.\n| Existing Work     | Assumption | Achievable Through Entropy Regularization |\n|----------|-----------|-----------|\n| Guo et. al [2019]| Contraction | Yes |\n| Xie et. al [2021]| Contraction | Yes |\n| Mao et. al [2022]| Contraction | Yes |\n| Zaman et. al [2022]| Contraction | Yes |\n| Yardim et. al [2023]| Contraction | Yes |\n| Zhang et al. [2023]| Lasry-Lions Monotonicity | Yes |\n| Cui et al. [2024]| Lasry-Lions Monotonicity | Yes |\n| Zeng et al. [2025]| Herding Condition | Unknown |\n\nCurrent knowledge suggests that the three subclasses overlap but do not subsume one another. We consider the contraction assumption due to its common use in the literature, but believe a modified version of our analysis can handle the other two subclasses as well. \nThe paper [1] suggested by the reviewer (Guo et al. [2022]) does not make any of the aforementioned assumptions, as it only investigates the SMFG problem structure and performs general sensitivity analysis, and does not propose/analyze any specific algorithm.\n\n2) > Furthermore, the followers are called ‘rational’, given that entropy-regularized objectives are very common in representing boundedly rational agents. I think that ‘rational’ does not accurately describe the followers considered in this paper. \n\nWe agree with the comment. While we deal with the low temperature (regularization weight) case, as the temperature increases, this can be seen as modeling bounded rational behavior in the spirit of Quantal Response Equilibrium. However, since the level of regularization is relatively low (compared to what would typically be used in bounded rationality work), we feel it is still valid to consider the followers rational, as is common in existing MFG literature. Bounded rational behavior in MFGs (even without the leader or hierarchical structure) represents a compelling direction for future work, which we have been investigating in parallel to this paper.\n\n\n 3) >* The assumption on the gradient alignment may be required to derive the theoretical result. However, the authors should discuss in more detail when this assumption does not hold. I believe that when the leader and followers have conflicting objectives, for example a zero-sum setting, this assumption will break.\n>* When does the gradient alignment assumption hold? Is it possible to make a conclusion on the validity of the assumption based on the objective functions?\n\n**Response:**  The reviewer is correct that the gradient alignment assumption breaks when there is competition between the leader and followers. For the assumption to hold, we need the SMFG to have at least one of the two following characteristics: \n\n1) The leader's reward is such that $\\nabla_ \\mu J_ l(\\phi,\\mu)$ has a much smaller magnitude than $\\nabla_ \\phi J_ l(\\phi, \\mu)$, which allows $\\nabla_ \\phi \\Phi(\\phi)$ and $\\nabla_ \\phi J_ l (\\phi,\\mu)| _{\\mu=\\mu^ *(\\phi)}$ to always form an acute angle. \n\n2) The indirect gradient is always roughly \"aligned\" with the direct partial gradient. We provide a non-trivial example (Example 1 in Appendix E) that falls into the second case and satisfies the gradient alignment assumption. Note that the leader's reward function in Example 1 has two components $r_ l(b, \\mu)=\\textbf{1}(b=1)+\\mu(b)$. The first component is associated with the direct portion of the gradient $\\nabla_ \\phi\\Phi(\\phi)$ and the second component with the indirect portion. If we flip the second component and consider $r_ l(b,\\mu)=\\textbf{1}(b=1)-2\\mu(b)$, the example will become one violating the gradient alignment assumption, because of the conflict between the direct and indirect partial gradients.\n\nThe assumption imposed in prior work is that the leader's reward is independent of the mean field, and that the follower's reward and transition are independent of the leader's action. Our works relaxes the assumption of independence between the leader and the followers, by allowing a form of coupling. This extends the class of problems that can be theoretically analyzed. While it is difficult to verify in practice whether the given problem instance satisfies this assumption, we show in simulations that the proposed algorithm performs well empirically, even when the assumption is explicitly violated or when we cannot verify if it is satisfied. \n\n\nReferences\n \nGuo, X., Hu, A., Xu, R. and Zhang, J., 2019. Learning mean-field games. Advances in neural information processing systems, 32.\n\nMao, W., Qiu, H., Wang, C., Franke, H., Kalbarczyk, Z., Iyer, R. and Basar, T., 2022. A mean-field game approach to cloud resource management with function approximation. Advances in Neural Information Processing Systems, 35, pp.36243-36258.\n\nYardim, B., Goldman, A. and He, N., 2024. When is Mean-Field Reinforcement Learning Tractable and Relevant?. arXiv preprint arXiv:2402.05757.\n\nZhang, F., Tan, V., Wang, Z. and Yang, Z., 2023. Learning regularized monotone graphon mean-field games. Advances in Neural Information Processing Systems, 36, pp.67297-67308."}, {"review_id": "dvbpjRnsVF", "reviewer": "Reviewer_CXoA", "summary": "Stackelberg mean field games (SFMGs) provide a model for studying settings in which a single \"leader\" takes an action and then \"follower\" agents react in a way modeled by a mean field capturing a collective response. This paper introduces AC-SMFG which is a single-loop actor critical algorithm that converges to a stationary point of a Stackelberg objective under a \"gradient alignment\" assumption that essentially implies the leader is able to improve their objective value by acting in response to the best response of the followers. This is a looser assumption than prior works which more so decouple the leaders and followers. Even with these looser assumptions, the authors establish the first non-asymptotic convergence guarantees for SMFGs. Additionally the authors provide numerical experiments demonstrating improved performance compared to baselines in mean field games such as market entrance and shop positioning.", "questions": "Could the authors provide an example of SMFG for which they expect the gradient alignment condition would not hold? To understand its limitations. \n\nCould the authors clarify whether followers would be incentivized to actually use this algorithm to update their policies or do there exist settings where this would not be in their best interest? \n\nThe authors make clear their advantages to previous agent-based modeling approaches, but could they discuss what they believe their drawbacks are compared to those approaches? What would be the line in practice between using AC-SMFG and such previous models?", "limitations": "Limitations are mostly addressed, lingering ones are related to my questions above.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The authors provide compelling motivating examples and an introduction to Stackelberg mean field games. Algorithm 1 AC-SMFG is well-explained and presented. Their theoretical analysis is thorough, building all the necessary steps to demonstrate their theoretical contributions of obtaining non-asymptotic convergence guarantees for this setting. Additionally the authors are careful to compare their algorithm to rates that could be achieves by considering the policy optimization problem inherent to a SFMG as a bi-level optimization problem, showing that their convergence rate improves over the baseline comparable single-loop algorithm. This involves a key technical contribution regarding the MFG satisfying the PL condition with respect to the follower's softmax policy under regularization and discuss the potential individual interest of this technical contribution. This highlights the technical difficulty inherent to proving theoretical results of their algorithm. Additionally the authors provide relatively extensive numerical results in multiple MFG settings with detailed results and provided code. Their numerical results demonstrate faster leader convergence and smoother convergence trajectories compared to baselines. For each setting they compare to both the existing mean field approaches and a MARL approach, discussing and explaining the reasons for the relatively differences of how AC-SMFG compares in each setting. \n\nAs the authors mention in Section 1.1, many previous agent-based modeling approaches to settings such as SMFGs simulate agents under complex interactions which causes them to suffer from computational inefficiencies and lack of provable guarantees. But the tradeoff this paper makes is the homogeneity of the mean field response, which although inherent to such modeling approaches is a drawback when considering real world significance. Additionally, results that impose a certain learning algorithm for both parties do come with some questions as to applicability especially for some of the mentioned settings such as optimal liquidation.", "was_revised": true, "final_justification": "The authors clearly addressed my original questions, which increased my original confidence in my already positive review. Additionally, the authors appear to have sufficiently addressed the concerns of other reviewers and in the process raised the overall sentiment towards the paper. From my own perspective, the potential limited generality and applicability of the model setting is still a drawback, though within this assumed setting the work provides a clear contribution. Thus I will maintain my original score with increased confidence.", "rebuttal": "We thank the reviewer for the time, effort, and valuable feedback on our paper. Please find our response below, which we have incorporated into the revised paper.\n\n> ...the tradeoff this paper makes is the homogeneity of the mean field response, which although inherent to such modeling approaches is a drawback when considering real world significance. \n\n**Response:** The reviewer is correct that it is a requirement of the setting of MFGs that the agents have homogeneous reward functions and exert symmetric effect on the environment. While this indeed implies that the standard (Stackelberg) MFG framework may not be directly applicable to real-world problems with heterogeneous agents, there exist many applications where the homogeneity assumption holds true (such as the ones in the economics space discussed in introduction and experiment sections) in which the MFG framework has yielded insights and a good approximation. \n\nWe further point out that for problems involving heterogeneous agents, the graphon MFG framework offers a natural extension, where each agent's dynamics and reward depend on local interactions defined by a structured graph. While graphon MFG adds another layer of complexity on top of standard MFG and is beyond the scope of this paper, we believe that many components of our algorithm and analysis can carry over to the graphon setting as well.\n\n\n> Additionally, results that impose a certain learning algorithm for both parties do come with some questions as to applicability especially for some of the mentioned settings such as optimal liquidation.\n\n**Response:** We appreciate this insightful comment and have revised the paper to provide more clarification. Please find our response below.\n\nIt is true that the specific algorithm proposed and analyzed in our paper is one in which the leader and follower both run actor-critic-type updates. However, we clarify that in general our framework does not require the two parties to adopt the same learning algorithm. The follower may use any algorithm (e.g. actor-critic algorithm considered in our paper, Q learning, or others) to optimize its own objectives given the leader's policy. The only requirement is that the chosen algorithm leads the follower to an (approximate) best response.\n\nTo reflect this flexibility, we have deliberately structured our analysis in a modular way -- we individually establish the leader's policy convergence, mean field convergence, and follower's convergence in Propositions 1, 2, and 3, respectively. These components are then connected through a Lyapunov function that ensures joint convergence. If we switch the follower's algorithm, Propositions 1 and 2 remain unaffected, and we simply need to redo the convergence analysis of the follower. This modularity allows our framework to accommodate alternative follower dynamics and highlights the generality of our approach.\n\n\n> Could the authors provide an example of SMFG for which they expect the gradient alignment condition would not hold? To understand its limitations.\n\n**Response:** Generally speaking, the gradient alignment assumption breaks when there is competition between the leader and followers. For the assumption to hold, we need the SMFG to have at least one of the two following characteristics: 1) the leader's reward is such that $\\nabla_ \\mu J_ l(\\phi,\\mu)$ has a much smaller magnitude than $\\nabla_ \\phi J_ l(\\phi,\\mu)$, which allows $\\nabla_ \\phi\\Phi(\\phi)$ and the direct partial gradient $\\nabla_ \\phi J_ l(\\phi,\\mu)|_ {\\mu=\\mu^*(\\phi)}$ to always form an acute angle, and 2) the indirect gradient is always roughly \"aligned\" with the direct partial gradient. We provide a non-trivial example (Example 1 in Appendix E) that falls into the second case and satisfies the gradient alignment assumption. Note that the leader's reward function in Example 1 has two components $r_ l(b,\\mu)=\\textbf{1}(b=1)+\\mu(b)$. The first component is associated with the direct portion of the gradient $\\nabla_ \\phi\\Phi(\\phi)$ and the second component with the indirect portion. If we flip the second component and consider $r_ l(b,\\mu)=\\textbf{1}(b=1)-2\\mu(b)$, the example will become one violating the gradient alignment assumption, because of the conflict between the direct and indirect partial gradients.\n\nWe note that in prior work, the assumption is that the leader's reward is independent of the mean field, and that the follower's reward and transition are independent of the leader's action. Therefore, the notion of gradient alignment we consider here is more general in that it permits these quantities to be correlated. Though Assumption 6 is a relaxation, we agree that it may not hold in many practical problems, and even checking its validity can be challenging. This provides an important motivation for us to carry out extensive numerical simulations in environments that potentially violate the assumption. The environments in Section 5 cannot be verified to satisfy the gradient alignment condition, but the proposed algorithm still outperforms existing methods, demonstrating empirical effectiveness even in situations where the current analysis does not strictly apply.\n    \n\n> Could the authors clarify whether followers would be incentivized to actually use this algorithm to update their policies or do there exist settings where this would not be in their best interest?\n\n**Response:** The follower's part of the proposed algorithm is that it uses an actor-critic algorithm to maximize its objective, in response to the shifting leader's policy and mean field, and this is indeed in the best interest of the follower. However, we note that a Stackelberg MFG, by definition, is a problem in which the goal is to maximize the leader’s objective. The most natural application of this framework is one where a central authority (e.g. government) uses an SMFG algorithm to determine a taxation rule, or where a large institutional investor uses an SMFG algorithm to determine a liquidation strategy, while accounting for the responses of the followers. The follower's objective is not what the SMFG formulation ultimately aims to optimize. \n\n\n> The authors make clear their advantages to previous agent-based modeling approaches, but could they discuss what they believe their drawbacks are compared to those approaches? What would be the line in practice between using AC-SMFG and such previous models?\n\n**Response:** We thank the reviewer for suggesting a discussion of the potential drawbacks, which we will incorporate into the revised version of the paper. We are aware of two types of practical scenarios where an agent-based modeling (ABM) approach may be more naturally suited.\n\nFirst, as the reviewer noted in an earlier comment, ABM may offer greater flexibility in modeling problems involving heterogeneous agents. The standard SMFG framework assumes symmetry among followers by design. While graphon-based extensions can capture heterogeneity, they come with the cost of additional complexity and the need to specify the interactions between followers by a graph.\n\nSecond, MFGs are approximations of $N$-player symmetric Markov games, with the approximation error diminishing as $N$ increases. If the problem of interest involves only a small number of followers (e.g. $N<10$), the approximation error may be non-negligible, making an ABM approach more appropriate in such cases.\n\nBoth ABM and SMFG approaches have their distinct advantages and challenges. We believe they should be seen as one complimenting the other in providing solutions and insights into the analysis of the challenging framework of multi-agent decision making."}]

Author rebuttal:

Meta-review: This paper proposes a single-loop actor–critic method (AC-SMFG) for Stackelberg mean-field games and proves $\tilde{O}(k^{-1/2})$ non-asymptotic convergence under a gradient-alignment condition, avoiding nested bilevel loops. Empirically, it shows faster, smoother convergence on standard SMFG benchmarks, including a neural function-approximation variant.

During discussion, reviewers mainly flagged clarity (notably the gradient-alignment assumption and proof sketch), scope/novelty relative to prior single-loop bilevel work, and whether results extend beyond tabular policies. The authors responded by adding intuition and examples for the assumption, a concise technique sketch after the main theorem, and clarifying that tabular choices are for analysis while providing additional neural results. Remaining concerns (e.g., regularization regime, homogeneous followers) are acknowledged and have limited impact on the core contribution.

Overall, the theoretical advance is solid and timely, the algorithm is practical, and the added clarifications strengthen the presentation. As a result, I recommend accept.
Decision: Accept (poster)