SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Convex-Concave 0-Sum Markov Stackelberg Games
Denizalp Goktas
Brown University, Computer Science denizalp_goktas@brown.edu
Arjun Prakash
Brown University, Computer Science arjun_prakash@brown.edu
Amy Greenwald
Brown University, Computer Science
amy_greenwald@brown.edu
Zero-sum Markov Stackelberg games can be used to model myriad problems, in domains ranging from economics to human robot interaction. We develop a policy gradient method which we prove solves these games in continuous state, continuous action settings, using noisy gradient estimates computed from observed trajectories of play. When the games are convex-concave, we prove that our algorithm converges to Stackelberg equilibrium in polynomial time. We also prove that reach-avoid problems are naturally modeled as convex-concave zero-sum Markov Stackelberg games, and show experimentally that Stackelberg equilibrium policies are more effective than their Nash counterparts in these problems.<sup>1</sup>
Markov games [28, 65, 70] are a generalization of Markov decision processes (MDPs) comprising multiple players simultaneously making decisions over time, collecting rewards along the way depending on their collective actions. They have been used by practitioners to model many real-world multiagent planning and learning environments, such as autonomous driving [31, 59], cloud computing [77], and telecommunications [3]. Moreover, theoreticians are beginning to formally analyze policy gradient methods, proving polynomial-time convergence to optimal policies in MDPs [2, 16], and to Nash equilibrium policies [53] in zero-sum Markov games [24], the canonical solution concept. While Markov games are a fruitful way to model some problems (e.g., robotic soccer [46]), others, such as reach-avoid [48], may be more productively modeled as sequential-move games, where some players commit to moves that are observed by others, before they make their own moves. To this end, we study two-player zero-sum Markov Stackelberg [74] (i.e., sequential-move) games. While polynomial-time value-iteration (i.e., planning) algorithms are known for these games assuming discrete states [36], we develop a policy gradient method that converges to Stackelberg equilibrium in polynomial time in continuous state, continuous action games, using noisy gradients based only on observed trajectories of play. Furthermore, we demonstrate experimentally that Stackelberg equilibrium policies are more effective than their Nash counterparts in reach-avoid problems.
A (discounted discrete-time) zero-sum Markov Stackelberg game [36] is played over an infinite horizon  $t = 0,1,\ldots$  between two players, a leader and a follower. The game starts at time  $t = 0$ , at some initial state  $S^{(0)}\sim \mu \in \Delta (\mathcal{S})$  drawn randomly from a set of states  $S$ . At each time step  $t = 1,2,\dots$ , the players encounter a state  $s^{(t)}\in S$ , where the leader takes its action  $\pmb{a}^{(t)}$  first, from its action space  $\mathcal{A}(\pmb{s}^{(t)})$ , after which the follower, having observed the leader's action, makes it own move  $\pmb{b}^{(t)}$ , chosen from a feasible subset  $\mathcal{C}(s^{(t)},a^{(t)})$  determined by the leader's action  $\pmb{a}^{(t)}$
<sup>1</sup>A full and current version of the paper can be found at: https://arxiv.org/abs/2401.12437
37th Conference on Neural Information Processing Systems (NeurIPS 2023). New Orleans.
of its action space  $\mathcal{B}(\pmb{s}^{(t)})$ . After both players have taken their actions, they receive respective rewards,  $-r(\pmb{s}^{(t)},\pmb{a}^{(t)},\pmb{b}^{(t)})$  and  $r(\pmb{s}^{(t)},\pmb{a}^{(t)},\pmb{b}^{(t)})$ . The game then moves to time step  $t + 1$  and transitions either to a new state  $S^{(t + 1)}\sim p(\cdot \mid \pmb{s}^{(t)},\pmb{a}^{(t)},\pmb{b}^{(t)})$  with probability  $\gamma$ , called the discount factor, or the game ends with the remaining probability. Each player's goal is to play a (potentially history-dependent) policy that maximizes its respective expected (cumulative discounted) payoffs,  $-\mathbb{E}\left[\sum_{t = 0}^{\infty}\gamma^{t}r(S^{(t)},A^{(t)},B^{(t)})\right]$  and  $\mathbb{E}\left[\sum_{t = 0}^{\infty}\gamma^{t}r(S^{(t)},A^{(t)},B^{(t)})\right]$ .
In zero-sum Markov Stackelberg games, when the reward function  $(\pmb{a},\pmb{b})\mapsto r(\pmb{s},\pmb{a},\pmb{b})$  is continuous and bounded, for all  $s\in S$ , and the correspondence  $\pmb{a}\Vdash \mathcal{C}(\pmb{s},\pmb{a})$  is continuous, as well as non-empty- and compact-valued, a recursive (or Markov perfect) [49] Stackelberg equilibrium is guaranteed to exist [36], meaning a stationary policy profile (i.e., a pair of mappings from states to the actions of the leader and the follower, respectively) specifying the actions taken at each state s.t. the leader's policy maximizes its expected payoff assuming the follower best responds, while the follower indeed best responds to the leader's policy. In other words, the aforementioned assumptions guarantee the existence of a policy profile  $\pi^{*}\doteq (\pi_{a}^{*},\pi_{b}^{*})$ , with  $\pi_{a}^{*}:S\to \mathcal{A}$  and  $\pi_b^*:\mathcal{S}\rightarrow \mathcal{B}$ , that solves the following coupled min-max optimization problem:
$$
\min  _ {\boldsymbol {\pi} _ {a}: \mathcal {S} \rightarrow \mathcal {A}} \max  _ {\substack {\boldsymbol {\pi} _ {b}: \mathcal {S} \rightarrow \mathcal {B}:\\\forall s \in \mathcal {S}, \boldsymbol {\pi} _ {b} (s) \in \mathcal {C} (s, \boldsymbol {\pi} _ {a} (s))}} \mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r (S ^ {(t)}, \boldsymbol {\pi} _ {a} (S ^ {(t)}), \boldsymbol {\pi} _ {b} (S ^ {(t)})) \right] , \tag{1}
$$
where the expectation is with respect to  $S^{(0)} \sim \mu$  and  $S^{(t + 1)} \sim p(\cdot \mid s^{(t)},\pi_a(S^{(t)}),\pi_b(S^{(t)}))$ . The problem is "coupled" because the players' actions sets constrain one another; in particular, the set of actions available to the follower at each state is determined by the leader's choice.
In spite of multiple compelling applications, including autonomous driving [29, 43], reach-avoid problems in human-robot interaction [10], robust optimization in stochastic environments [15], and resource allocation over time [36], very little is known about computing recursive Stackelberg equilibria in zero-sum Markov Stackelberg games. A version of value iteration is known to converge in polynomial time when the state space is discrete [36], but this (planning) method becomes intractable in large or continuous state spaces. Furthermore, nothing is known, to our knowledge, about learning Stackelberg equilibria from observed trajectories of play. We develop an efficient policy gradient method for convex-concave zero-sum Markov Stackelberg games, and we show that reach-avoid problems naturally lie in this class of games.
Contributions. Equation (1) reveals that the problem of computing Stackelberg equilibria in zero-sum Markov Stackelberg games is an instance of a coupled min-max optimization problem. Goktas and Greenwald [33] studied coupled min-max optimization problems assuming an exact first-order oracle, meaning one that returns a function's exact value and gradient at any point in its domain. As access to an exact oracle is an unrealistic assumption in Markov games, we develop a first-order method for solving these problems, assuming access to a stochastic first-order oracle, which returns noisy estimates of a function's value and gradient at any point in its domain. We show that our method converges in polynomial-time (Theorem 3.1) in a large class of coupled min-max optimization problems, namely those which are convex-concave.
We then proceed to study zero-sum Markov Stackelberg games, providing sufficient conditions on the action correspondence  $\mathcal{C}: S \times \mathcal{A} \to \mathcal{B}$ , the rewards  $r: S \times \mathcal{A} \times \mathcal{B} \to \mathbb{R}$ , and the transition probabilities  $p: S \times S \times \mathcal{A} \times \mathcal{B} \to \mathbb{R}_+$  to guarantee that the game is convex-concave. Furthermore, we develop a policy gradient algorithm that provably converges to Stackelberg equilibrium in polynomial time when such games are convex-concave (Theorem 4.1), the first reinforcement learning algorithm of this kind. Our method specializes to continuous state, continuous action zero-sum Markov games; as such, we provide a provably-convergent policy gradient method for these problems as well. Finally, we prove that our framework naturally models reach-avoid problems, and run experiments which show that the Stackelberg equilibrium policies learned by our method exhibit better safety and liveness properties than their Nash counterparts.
To simplify notation, we drop the dependency of action spaces  $\mathcal{A}$  and  $\mathcal{B}$  on states going forward, but our theory applies in this more general setting.
${}^{3}$  Unlike  ${\mathbf{a}}^{\left( t\right) }$  and  ${\mathbf{b}}^{\left( t\right) }$  ,which are deterministic actions because they depend on a realized history of states and actions encountered,  ${A}^{\left( t\right) }$  and  ${B}^{\left( t\right) }$  are random variables,because they might depend on a random history.
Notation. All notation for variable types, e.g., vectors, should be clear from context; if any confusion arises, see Appendix A. Unless otherwise noted, we assume  $\|\cdot\|$  is the Euclidean norm,  $\|\cdot\|_2$ . We let  $\Delta_n = \{x \in \mathbb{R}_+^n \mid \sum_{i=1}^n x_i = 1\}$  denote the unit simplex in  $\mathbb{R}^n$ , and  $\Delta(A)$ , the set of probability distributions on the set  $A$ . We also define the support of any distribution  $f \in \Delta(\mathcal{X})$  as  $\operatorname{supp}(f) \doteq \{x \in \mathcal{X} : f(x) > 0\}$ . We denote the orthogonal projection operator onto a set  $C$  by  $\Pi_C$ , i.e.,  $\Pi_C(\boldsymbol{x}) = \arg \min_{\boldsymbol{y} \in C} \| \boldsymbol{x} - \boldsymbol{y} \|^2$ . We denote by  $\mathbb{1}_{\mathcal{C}}(\boldsymbol{x})$  the indicator function of a set  $\mathcal{C}$ , with value 1 if  $\boldsymbol{x} \in \mathcal{C}$  and 0 otherwise. Given two vectors  $x, y \in \mathbb{R}^n$ , we write  $x \geq y$  or  $x > y$  to mean component-wise  $\geq$  or  $>$ , respectively. For any set  $\mathcal{C}$ , we denote the diameter by  $\mathrm{diam}(\mathcal{C}) \doteq \max_{\boldsymbol{c}, \boldsymbol{c}' \in \mathcal{C}} \| \boldsymbol{c} - \boldsymbol{c}' \|$ . Given a tuple consisting of a sequence of iterates and weights  $(\{\boldsymbol{z}^{(t)}\}_t, \{\eta^{(t)}\}_t)$ , the weighted average of the iterates is given by  $\overline{z}_\eta = \frac{\sum_t \eta^{(t)} z^{(t)}}{\sum_t \eta^{(t)}}$ .
Mathematical Concepts. Given  $\mathcal{X} \subset \mathbb{R}^n$ , the function  $f: \mathcal{X} \to \mathcal{Y}$  is said to be  $\ell_f$ -Lipschitz-continuous w.r.t. norm  $\|\cdot\|$  iff  $\forall x_1, x_2 \in \mathcal{X}$ ,  $\|f(x_1) - f(x_2)\| \leq \ell_f\|x_1 - x_2\|$ . If  $\mathcal{Y} = \mathbb{R}$ , then  $f$  is convex (resp. concave) iff for all  $\lambda \in (0,1)$  and  $\pmb{x}, \pmb{x}' \in \mathcal{X}$ ,  $f(\lambda \pmb{x} + (1 - \lambda)\pmb{x}') \leq (\text{resp.} \geq) \lambda f(\pmb{x}) + (1 - \lambda)f(\pmb{x}')$ . For any  $\mathcal{Y}$ , if the relation holds with equality, then  $f$  is called affine. A two-sided function  $h: \mathcal{X} \times \mathcal{Y} \to \mathcal{Z}$  is biaffine if  $\pmb{x} \mapsto f(\pmb{x}, \pmb{y})$  is affine for all  $\pmb{y} \in \mathcal{Y}$ , and  $\pmb{y} \mapsto h(\pmb{x}, \pmb{y})$  is affine for all  $\pmb{x} \in \mathcal{X}$ .  $f$  is a strongly convex if  $f(x_1) \geq f(x_2) + \langle \nabla_x f(x_2), x_1 - x_2 \rangle + \mu/2 \|x_1 - x_1\|^2$ . For convenience, we say that an  $l$ -dimensional vector-valued function  $g: \mathcal{X} \to \mathcal{Y} \subset \mathbb{R}^l$  is log-convex, convex, log-concave, or concave, respectively, if  $g_k$  is log-convex, convex, log-concave, or concave, for all  $k \in [l]$ . A correspondence  $\mathcal{Z}: \mathcal{X} \to \mathcal{Y}$  is concave if for all  $\lambda \in (0,1)$  and  $x, x' \in \mathcal{X}$ ,  $\mathcal{Z}(\lambda x + (1 - \lambda)x') \subseteq \lambda \mathcal{Z}(x) + (1 - \lambda) \mathcal{Z}(x)$ , assuming Minkowski summation [22, 57].
We require notions of stochastic convexity related to stochastic dominance of probability distributions [7]. Given non-empty and convex parameter and outcome spaces  $\mathcal{W}$  and  $\mathcal{O}$  respectively, a conditional probability distribution  $\pmb{w} \mapsto p(\cdot \mid \pmb{w}) \in \Delta(\mathcal{O})$  is said to be stochastically convex (resp. stochastically concave) in  $\pmb{w} \in \mathcal{W}$  if for all continuous, bounded, and convex (resp. concave) functions  $v: \mathcal{O} \to \mathbb{R}$ ,  $\lambda \in (0,1)$ , and  $\pmb{w}'$ ,  $\pmb{w}^{\dagger} \in \mathcal{W}$  s.t.  $\overline{\pmb{w}} = \lambda \pmb{w}' + (1 - \lambda) \pmb{w}^{\dagger}$ , it holds that  $\mathbb{E}_{O \sim p(\cdot \mid \overline{\pmb{w}})}[v(O)] \leq (\text{resp} \geq) \lambda \mathbb{E}_{O \sim p(\cdot \mid \pmb{w}')}[v(O)] + (1 - \lambda) \mathbb{E}_{O \sim p(\cdot \mid \pmb{w}^{\dagger})}[v(O)]$ .
3 Coupled Min-Max Optimization Problems
A min-max Stackelberg game, denoted  $(\mathcal{X},\mathcal{Y},f,\pmb {g})$ , is a two-player, zero-sum game, where one player, called the leader, first commits to an action  $\pmb {x}\in \mathcal{X}$  from its action space  $\mathcal{X}\subset \mathbb{R}^n$ , after which the second player, called the follower, takes an action  $\pmb {y}\in \mathcal{Z}(\pmb {x})\subset \mathcal{V}$  from a subset of his action space  $\mathcal{V}\subseteq \mathbb{R}^{m}$  determined by the action correspondence  $\mathcal{Z}:\mathbb{R}^n\Rightarrow \mathcal{V}$ . As is standard in the optimization literature, we assume throughout that the follower's action correspondence can be equivalently represented via a coupling constraint function  $\pmb {g}:\mathbb{R}^n\times \mathbb{R}^m\to \mathbb{R}^d$  s.t.  $\mathcal{Z}(\pmb {x})\doteq \{\pmb {y}\in \mathcal{V}\mid \pmb {g}(\pmb {x},\pmb {y})\geq \pmb {0}\}$ . An action profile  $(\pmb {x},\pmb {y})\in \mathcal{X}\times \mathcal{V}$  comprises actions for both players. Once both players have taken their actions, the leader (resp. follower) receives a loss (resp. payoff)  $f(\pmb {x},\pmb {y})$ , defined by an objective function  $f:\mathbb{R}^n\times \mathbb{R}^m\to \mathbb{R}$ . We define the marginal function  $f^{*}(\pmb {x})\doteq \max_{\pmb {y}\in \mathcal{Z}(\pmb {x})}f(\pmb {x},\pmb {y})$ , which, given an action for the leader, outputs its ensuing payoff, assuming the follower best responds. The constraints in a min-max Stackelberg game are said to be uncoupled if  $\mathcal{Z}(\pmb {x}) = \mathcal{V}$ , for all  $\pmb {x}\in \mathcal{X}$ . A min-max Stackelberg game is said to be continuous iff 1. the objective function  $f$  is continuous; 2. the action spaces  $\mathcal{X}$  and  $\mathcal{V}$  are non-empty and compact; and 3. the action correspondence  $\mathcal{Z}$  is continuous, non-empty- and compact-valued. $^4$
Stackelberg Equilibrium. The canonical solution concept for min-max Stackelberg games is the  $(\varepsilon, \delta)$ -Stackelberg equilibrium  $((\varepsilon, \delta)$ -SE, or SE if  $\varepsilon = \delta = 0$ , an action profile  $(\pmb{x}^{*}, \pmb{y}^{*}) \in \mathcal{X} \times \mathcal{Y}$  s.t.  $\| \Pi_{\mathbb{R}^d}[g(\pmb{x}^*, \pmb{y}^*)] \| \leq \delta$  and  $\min_{\pmb{x} \in \mathcal{X}} f^*(\pmb{x}) + \varepsilon \geq f(\pmb{x}^*, \pmb{y}^*) \geq \max_{\pmb{y} \in \mathcal{Z}(\pmb{x}^*)} f(\pmb{x}^*, \pmb{y}) - \delta$ , for  $\varepsilon, \delta \geq 0$ . As a straightforward corollary of Theorem 3.2 of Goktas and Greenwald [33], a SE is guaranteed to exist in continuous Stackelberg games. Moreover, the set of SE can be characterized as solutions to the following coupled min-max optimization problem:  $\min_{\pmb{x} \in \mathcal{X}} \max_{\pmb{y} \in \mathcal{Z}(\pmb{x})} f(\pmb{x}, \pmb{y})$ .
<sup>4</sup>See Theorem 5.9 and Example 5.10 of Rockafellar and Wets [63] for conditions on  $\pmb{g}$  that guarantee the continuity of  $\mathcal{Z}$  or Section 3 of Goktas and Greenwald [33].
For  $\delta > 0$ , this definition of an  $(\varepsilon, \delta)$ -SE is more general than the one introduced by Goktas and Greenwald [33], as it allows for the coupling constraints to be satisfied only approximately, which is necessary in this paper, as the coupling constraints can only be accessed via a stochastic oracle.
An alternative but weaker solution concept previously considered for min-max Stackelberg games [71] is the  $\varepsilon$ -generalized Nash equilibrium ( $\varepsilon$ -GNE, or GNE if  $\varepsilon = 0$ ), i.e.,  $(\pmb{x}^{*},\pmb{y}^{*})\in \mathcal{X}\times \mathcal{Z}(\pmb{x}^{*})$  s.t.  $\min_{\pmb{x}\in \mathcal{X}}f(\pmb{x},\pmb{y}^{*}) + \varepsilon \geq f(\pmb{x}^{*},\pmb{y}^{*})\geq \max_{\pmb{y}\in \mathcal{Z}(\pmb{x}^{*})}f(\pmb{x}^{*},\pmb{y}) - \varepsilon$ , for some  $\varepsilon \geq 0$ . In general, the set of GNE and SE need not intersect; as such, GNE are not necessarily solutions of  $\min_{\pmb{x}\in \mathcal{X}}\max_{\pmb{y}\in \mathcal{Z}(\pmb{x})}f(\pmb{x},\pmb{y})$  (see, Appendix A of Goktas and Greenwald [33]). Furthermore, there is no GNE whose value is less than the SE value of a game. When a min-max Stackelberg game's constraints are uncoupled, a(n  $\varepsilon$ )-GNE is called a(n  $\varepsilon$ )-saddle point, or a(n  $\varepsilon$ )-Nash equilibrium, and is also an SE. Finally, a saddle point is guaranteed to exist [67, 73] in continuous min-max Stackelberg games with uncoupled constraints, a convex-concave objective  $f$ , and convex action spaces  $\mathcal{X}$  and  $\mathcal{Y}$ , in which case such games have traditionally been referred to as convex-concave min-max (simultaneous-move) games or saddle-point problems [13].
Convex-Concave Games. A min-max Stackelberg game is said to be convex-concave if, in addition to being continuous, 1.  $f^{*}$  is convex; 2.  $y \mapsto f(x, y)$  is concave, for all  $x \in \mathcal{X}$ ; 3.  $\mathcal{X}$  and  $\mathcal{Y}$  are convex; and 4.  $\mathcal{Z}$  is convex-valued. This definition generalizes that of convex-concave min-max (simultaneous-move) game, because in such games, the marginal function  $f^{*}$  is necessarily convex when  $f$  is convex, by Danskin's theorem [23]. Assuming access to an exact first-order oracle, an  $(\varepsilon, \delta)$ -SE of a convex-concave min-max Stackelberg game can be computed in polynomial time when  $f$  and  $g$  are Lipschitz-continuous [33], while the computation is NP-hard in continuous min-max Stackelberg games, even when  $\mathcal{X}$  and  $\mathcal{Y}$  are convex,  $f$  is convex-concave, and  $g$  is affine [47].
All the conditions that define a convex-concave Stackelberg game depend on the game primitives, namely  $(\mathcal{X},\mathcal{Y},f,\pmb {g})$ , and are well-understood (see, for instance Section 5 of Rockafellar and Wets [63]), with the exception of the condition that the marginal function  $f^{*}$  be convex. While it is difficult to obtain necessary and sufficient conditions on the game primitives that ensure the convexity of  $f^{*}$ , one possibility is to require  $f$  to be convex in  $(x,y)$  and  $\mathcal{Z}$  to be concave. The following sufficient conditions, which also guarantee concavity, were introduced by Goktas and Greenwald [33].
Assumption 1 (Convex-Concave Assumptions). 1. The objective function  $f(\pmb{x}, \pmb{y})$  is convex in  $(\pmb{x}, \pmb{y})$ , and concave in  $\pmb{y}$ , for all  $\pmb{x} \in \mathcal{X}$ ; 2. the action correspondence  $\mathcal{Z}$  is concave; 3. the action spaces  $\mathcal{X}$  and  $\mathcal{Y}$  are convex.
As these assumptions are only sufficient, they are not necessarily satisfied in all applications of convex-concave min-max Stackelberg game. Hence, the convexity of the marginal function must sometimes be established by other means. We thus provide the following alternative set of sufficient conditions, which we use to show that the reach-avoid problem we study in Section 5 is convex-concave.
Assumption 2 (Alternative Convex-Concave Assumptions). 1. (Convex-concave objective) The objective  $f(\pmb{x}, \pmb{y})$  is convex in  $\pmb{x}$ , for all  $\pmb{y} \in \mathcal{V}$ , and concave in  $\pmb{y}$ , for all  $\pmb{x} \in \mathcal{X}$ ; 2. (log-convex-concave coupling) the coupling constraint  $g(\pmb{x}, \pmb{y})$  is log-convex in  $\pmb{x}$  for all  $\pmb{y} \in \mathcal{Y}$ , and concave in  $\pmb{y}$  for all  $\pmb{x} \in \mathcal{X}$ ; and 3. the action spaces  $\mathcal{X}$  and  $\mathcal{Y}$  are convex.
Computation. We now turn our attention to the computation of  $(\varepsilon, \delta)$ -SE in convex-concave min-max Stackelberg games, assuming access to an unbiased first-order stochastic oracle  $(\widehat{F}, \widehat{G}, \mathcal{F}, \mathcal{G})$  comprising random functions  $\widehat{F}: \mathbb{R}^n \times \mathbb{R}^m \times \Theta \to \mathbb{R}$  and  $\widehat{G}: \mathbb{R}^n \times \mathbb{R}^m \times \Phi \to \mathbb{R}^d$  and sampling distributions  $\mathcal{F} \in \Delta(\Theta)$  and  $\mathcal{G} \in \Delta(\Phi)$  s.t.  $\mathbb{E}_{\theta \sim \mathcal{F}}[\widehat{F}(\boldsymbol{x}, \boldsymbol{y}; \boldsymbol{\theta})] = f(\boldsymbol{x}, \boldsymbol{y}), \mathbb{E}_{\phi \sim \mathcal{G}}[\widehat{G}(\boldsymbol{x}, \boldsymbol{y}; \phi)] = g(\boldsymbol{x}, \boldsymbol{y}), \mathbb{E}_{\boldsymbol{\theta}}[\nabla_{(\boldsymbol{x}, \boldsymbol{y})} \widehat{F}(\boldsymbol{x}, \boldsymbol{y}; \boldsymbol{\theta})] = \nabla f(\boldsymbol{x}, \boldsymbol{y})$ , and  $\mathbb{E}_{\phi}[\nabla_{(\boldsymbol{x}, \boldsymbol{y})} \widehat{G}(\boldsymbol{x}, \boldsymbol{y}; \phi)] = \nabla g(\boldsymbol{x}, \boldsymbol{y})$ . The following assumptions are required for the convergence of our methods.
Assumption 3. 1. (Lipschitz game)  $f$  and  $g$  are Lipschitz-continuous; 2. (concave representation) the coupling constraint function  $y \mapsto g(x, y)$  is concave for all  $x \in \mathcal{X}$ ; 3. (Slater's condition)  $\forall x \in \mathcal{X}, \exists \widehat{y} \in \mathcal{Y}$  s.t.  $g(x, \widehat{y}) > 0$ ; and 4. (stochastic oracle) there exists an unbiased first-order stochastic oracle  $(\widehat{F}, \widehat{G}, \mathcal{F}, \mathcal{G})$  with bounded variance s.t.  $\forall (x, y) \in \mathcal{X} \times \mathcal{Y}$ ,  $\mathbb{E}[\| \widehat{G}(x, y; \phi) \|^2] \leq \sigma_g$ ,  $\mathbb{E}[\| \nabla_{(x, y)} \widehat{F}(x, y; \theta) \|^2] \leq \sigma_{\nabla f}$ , and  $\mathbb{E}[\| \nabla_{(x, y)} \widehat{G}(x, y; \phi) \|^2] \leq \sigma_{\nabla g}$ , for  $\sigma_g, \sigma_{\nabla f}, \sigma_{\nabla g} \geq 0$ . In the sequel, we rely on the following notation and definitions. For any action  $x \in \mathcal{X}$  of the leader, we can re-express the marginal function in terms of the Lagrangian  $\ell(y, \lambda; x) = f(x, y) + \langle \lambda, g(x, y) \rangle$  (see, for instance, Section 5 of Boyd et al. [17]) as follows:  $f^*(x) = \max_{y \in Y} \min_{\lambda \in \mathbb{R}_+^d} \ell(y, \lambda; x)$ . Further, we define the follower's best-response correspondence
<sup>6</sup>A GNE is guaranteed to exist in continuous min-max Stackelberg games when the objective function  $f$  is convex-concave, the action spaces  $\mathcal{A}$  and  $\mathcal{B}$  are convex, and the action correspondence  $\mathcal{Z}$  is convex-valued [4].
See Section 2 of Nikodem [57] and Chapter 36 of Czerwik [22] for conditions on  $g$  which guarantee that  $\mathcal{Z}$  is concave and/or continuous and/or convex-valued.