SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$\mathcal{Y}^{*}(\pmb {x})\doteq \arg \max_{\pmb {y}\in \mathcal{Y}}\min_{\pmb {\lambda}\in \mathbb{R}^{d}_{+}}\ell (\pmb {y},\pmb {\lambda};\pmb {x})$  , and the KKT multiplier correspondence  $\Lambda^{*}(\pmb {x})\doteq$  arg  $\min_{\pmb {\lambda}\in \mathbb{R}_+^d}\max_{\pmb {y}\in \mathcal{Y}}\ell (\pmb {y},\pmb {\lambda};\pmb {x})$  . With these definitions in hand, under Assumption 3, we can build an unbiased first-order stochastic oracle  $\widehat{\mathcal{L}} (\pmb {y},\pmb {\lambda};\pmb {x},\pmb {\theta},\pmb {\phi})\doteq \widehat{F} (\pmb {x},\pmb {y};\pmb {\theta}) + <  \pmb {\lambda},\widehat{G} (\pmb {x},\pmb {y};\pmb {\phi})>$  for the Lagrangian  $\ell$  s.t.  $\mathbb{E}_{(\pmb {\theta},\phi)}[\widehat{\mathcal{L}} (\pmb {y},\pmb {\lambda};\pmb {x},\pmb {\theta},\pmb {\phi})]$  , where the expectation is taken over  $(\theta ,\phi)\sim \mathcal{F}\times \mathcal{G}$
Algorithms. Assuming access to an exact first-order oracle  $(f, g)$ , a natural approach to computing SE in convex-concave min-max Stackelberg games with uncoupled constraints games (i.e., saddle-point problems) is to simultaneously run projected gradient descent and projected gradient ascent on the objective function  $f$  w.r.t.  $\pmb{x} \in \mathcal{X}$  and  $\pmb{y} \in \mathcal{Y}$ , i.e., for  $t = 0,1,2,\ldots$ ,  $(\pmb{x}^{(t+1)}, \pmb{y}^{(t+1)}) \gets \Pi_{\mathcal{X} \times \mathcal{Y}}[(\pmb{x}^{(t)}, \pmb{y}^{(t)}) + (-\nabla_{\pmb{x}} f, \nabla_{\pmb{y}} f)(\pmb{x}^{(t)}, \pmb{y}^{(t)})]$ , a method known under the names of Arrow-Hurwicz-Uzawa, primal-dual, and (simultaneous) gradient descent ascent (GDA) [5, 6]. Intuitively, any fixed point of GDA in such games, i.e.,  $(\pmb{x}^*, \pmb{y}^*) \in \mathcal{X} \times \mathcal{Y}$  s.t.  $\| (\pmb{x}^*, \pmb{y}^*) - \Pi_{\mathcal{X} \times \mathcal{Y}}[(\pmb{x}^*, \pmb{y}^*) + (-\nabla_{\pmb{x}} f, \nabla_{\pmb{y}} f)(\pmb{x}^*, \pmb{y}^*)] \| = 0$ , satisfies the necessary and sufficient optimality condition for an action profile to be a SE. More generally, in convex-concave min-max Stackelberg games (without coupled constraints), this approach fails, as the necessary and sufficient optimality condition for an action profile  $(\pmb{x}^*, \pmb{y}^*) \in \mathcal{X} \times \mathcal{Y}$  to be a SE is  $\| (\pmb{x}^*, \pmb{y}^*) - \Pi_{\mathcal{X} \times \mathcal{Z}(\pmb{x}^*)}[(\pmb{x}^*, \pmb{y}^*) + (-\nabla_{\pmb{x}} f^*(\pmb{x}^*), \nabla_{\pmb{y}} f(\pmb{x}^*, \pmb{y}^*))] \| = 0$ , where, for any leader action  $\widehat{\pmb{x}} \in \mathcal{X}, \nabla_{\pmb{x}} f^*(\widehat{\pmb{x}}) \doteq \ell(\pmb{y}^*(\widehat{\pmb{x}}), \pmb{\lambda}^*(\widehat{\pmb{x}}); \widehat{\pmb{x}})$ , for some  $(\pmb{y}^*, \pmb{\lambda}^*)(\widehat{\pmb{x}}) \in \mathcal{Y}^*(\widehat{\pmb{x}}) \times \Lambda^*(\widehat{\pmb{x}})$ , by the subdifferential envelope theorem [33]. The observation that any subgradient of  $\nabla_{\pmb{x}} f^*$  depends on the optimal KKT multipliers motivates a first-order method based on the gradient of the Lagrangian.
A min-max Stackelberg game can be seen as a three-player game  $\min_{\boldsymbol{x} \in \mathcal{X}} \max_{\boldsymbol{y} \in \mathcal{Z}(\boldsymbol{x})} f(\boldsymbol{x}, \boldsymbol{y}) = \min_{\boldsymbol{x} \in \mathcal{X}} \max_{\boldsymbol{y} \in \mathcal{Y}} \min_{\boldsymbol{\lambda} \in \mathbb{R}_+^d} \ell(\boldsymbol{y}, \boldsymbol{\lambda}; \boldsymbol{x})$ , where the  $\boldsymbol{x}$ -player moves first, and the  $\boldsymbol{y}$ - and  $\boldsymbol{\lambda}$ -players move second, simultaneously, because strong duality holds under Assumption 3 (Slater's condition [68]) for the inner min-max optimization problem, i.e.,  $\max_{\boldsymbol{y} \in \mathcal{Y}} \min_{\boldsymbol{\lambda} \in \mathbb{R}_+^d} \ell(\boldsymbol{y}, \boldsymbol{\lambda}; \boldsymbol{x}) = \min_{\boldsymbol{\lambda} \in \mathbb{R}_+^d} \max_{\boldsymbol{y} \in \mathcal{Y}} \ell(\boldsymbol{y}, \boldsymbol{\lambda}; \boldsymbol{x})$ . The problem of computing an SE can thus be reduced to the min-max optimization  $\min_{(\boldsymbol{x}, \boldsymbol{\lambda}) \in \mathcal{X} \times \mathbb{R}_+^d} \max_{\boldsymbol{y} \in \mathcal{Y}} \ell(\boldsymbol{y}, \boldsymbol{\lambda}; \boldsymbol{x})$ , which we might hope to solve by running GDA on  $\ell(\boldsymbol{y}, \boldsymbol{\lambda}; \boldsymbol{x})$  w.r.t.  $(\boldsymbol{x}, \boldsymbol{\lambda})$  and  $\boldsymbol{y}$  over  $\mathcal{X} \times \mathbb{R}_+^d$  and  $\mathcal{Y}$ , respectively. Although  $\boldsymbol{y} \mapsto \ell(\boldsymbol{y}, \boldsymbol{\lambda}; \boldsymbol{x})$  is concave,  $(\boldsymbol{x}, \boldsymbol{\lambda}) \mapsto \ell(\boldsymbol{y}, \boldsymbol{\lambda}; \boldsymbol{x})$  is not convex, and its stationary points (i.e., points  $(\boldsymbol{y}^*, \boldsymbol{\lambda}^*; \boldsymbol{x}^*)$  s.t.  $\| (\boldsymbol{y}^*, \boldsymbol{\lambda}^*; \boldsymbol{x}^*) - \Pi_{\mathcal{Y} \times \mathbb{R}_+^d \times \mathcal{X}}[(\boldsymbol{y}^*, \boldsymbol{\lambda}^*; \boldsymbol{x}^*) + (\nabla_{\boldsymbol{y}}\ell, -\nabla_{\boldsymbol{\lambda}}\ell, -\nabla_{\boldsymbol{x}}\ell)(\boldsymbol{y}^*, \boldsymbol{\lambda}^*; \boldsymbol{x}^*)]\| = 0$ ) do not necessarily coincide with SE even in simple convex-concave min-max Stackelberg games [35].
Algorithm 1 Saddle-Point-Oracle SGD/Nested SGDA
Inputs:  $\mathcal{X},\mathcal{Y},\widehat{F},\widehat{G},\mathcal{F},\mathcal{G},\boldsymbol{x}^{(0)},T_{\boldsymbol{x}},\{\eta_{\boldsymbol{x}}^{(t)}\}_{t},\delta$
(+ for nested SGDA:)  $\Lambda, \pmb{y}^{\prime(0)}, \pmb{\lambda}^{\prime(0)}, T_{\pmb{y}}, \{\eta_{\pmb{y}}^{(t)}\}_{t}$
Outputs:  $(\pmb{x}^{(t)},\pmb{y}^{(t)},\pmb{\lambda}^{(t)})_{t = 0}^{T_{\pmb{x}}}$
1: for  $t = 0, \dots, T_{x}$  do
2: if Saddle-Point-Oracle SGD then
3: Find  $(\pmb{y}^{(t)},\pmb{\lambda}^{(t)})\in \mathcal{Y}\times \mathbb{R}_+^d$  s.t.
4:  $\max_{\pmb{y}\in \mathcal{Y}}\ell (\pmb{y}^{(t)},\pmb {\lambda};\pmb{x}^{(t)}) - \min_{\pmb {\lambda}\in \mathbb{R}_+^d}\ell (\pmb {y},\pmb{\lambda}^{(t)};\pmb{x}^{(t)})\leq \delta ,$
5: if Nested SGDA then
6: for  $s = 0,\dots ,T_{\mathbf{y}}$  do
7: Sample  $\pmb {\theta}\sim \mathcal{F},\phi \sim \mathcal{G}$
8:  $\pmb{y}^{\prime (s + 1)}\gets \Pi_{\mathcal{V}}\Big[\pmb{y}^{\prime (s)} + \eta_{\pmb{y}}^{(s)}\nabla_{\pmb{y}}\widehat{\mathcal{L}} (\pmb{y}^{\prime (s)},\pmb{\lambda}^{\prime (s)};\pmb{x}^{(t)},\pmb {\theta},\phi)\Big]$
9:  $\pmb{\lambda}^{\prime(s+1)} \gets \Pi_{\Lambda}\left[\pmb{\lambda}^{\prime(s)} - \eta_{\pmb{y}}^{(s)} \nabla_{\pmb{\lambda}} \widehat{\mathcal{L}}(\pmb{y}^{\prime(s)}, \pmb{\lambda}^{\prime(s)}; \pmb{x}^{(t)}, \pmb{\theta}, \phi)\right]$
10: Average iterates  $(\pmb{y}^{(t)},\pmb{\lambda}^{(t)})\gets (\overline{\pmb{y}}_{\eta_y}',\overline{\lambda}_{\eta_y}')$
11: Sample  $\pmb{\theta} \sim \mathcal{F}, \phi \sim \mathcal{G}$
12:  $\pmb{x}^{(t + 1)}\gets \Pi_{\mathcal{X}}\Big[\pmb{x}^{(t)} - \eta_{\pmb{x}}^{(t)}\nabla_{\pmb{x}}\widehat{\mathcal{L}} (\pmb{y}^{(t)},\pmb{\lambda}^{(t)};\pmb{x}^{(t)},\pmb {\theta},\phi)\Big]$
13: return  $(\overline{\pmb{x}}_{\pmb{\eta}_{\pmb{x}}},\pmb{y}^{(T_{\pmb{x}})},\pmb{\lambda}^{(T_{\pmb{x}})})$
As GDA fails in these games, Goktas and Greenwald [33] developed nested GDA, a nested first-order method for computing an  $(\varepsilon, \delta)$ -SE, which solves the inner maximization problem by running GDA on  $\ell$  w.r.t.  $\pmb{y}$  and  $\lambda$  over constraint sets  $\mathcal{V}$  and  $\mathbb{R}_+^d$  until convergence to a  $\delta$ -saddle point  $(\widehat{\pmb{y}}, \widehat{\pmb{\lambda}})$ . Then, exploiting the convexity of the marginal function  $f^*$ , the algorithm runs a descent step on  $f^*$  w.r.t.  $\pmb{x}$ , in which, for any leader action  $\pmb{x} \in \mathcal{X}$ , a subgradient  $\nabla_{\pmb{x}} f^*$  is approximated by  $\widehat{\nabla_{\pmb{x}} f^*}(\pmb{x}) = \ell(\widehat{\pmb{y}}, \widehat{\pmb{\lambda}}; \pmb{x})$ . In this paper, we replace the exact first-order oracle used by nested GDA with a stochastic one, the gradient descent step with a step of stochastic gradient descent (SGD), and GDA with stochastic GDA (SGDA), using in both cases the stochastic
Lagrangian oracle  $\widehat{\mathcal{L}}$ . We call our method nested stochastic gradient descent ascent (nested SGDA).
We begin by presenting saddle-point-oracle stochastic gradient descent algorithm (saddle-point-oracle SGD, Algorithm 1), whose analysis we build on to develop our primary contribution, nested SGDA. Following Goktas and Greenwald's [33] max-oracle gradient descent algorithm, saddle-point-oracle SGD runs SGD on  $f^{*}$ , assuming access to an oracle, which, for any leader action  $x \in \mathcal{X}$  returns a  $\delta$ -saddle point of  $(\pmb{y},\pmb{\lambda}) \mapsto \ell(\pmb{y},\pmb{\lambda};\pmb{x})$ . Our second algorithm, nested stochastic gradient descent ascent (nested SGDA, Algorithm 1), follows the same logic as saddle-point-oracle SGD, but implements the saddle-point oracle by running SGDA. The following theorem establishes conditions under which both of our algorithms converge to an  $(\varepsilon + \delta, \delta)$ -SE in a number of oracle calls that is polynomial in  $1 / \varepsilon$  and  $1 / \delta$ .<sup>8</sup>
Theorem 3.1. Let  $(\mathcal{X},\mathcal{Y},f,\pmb {g})$  be a convex-concave min-max Stackelberg game for which Assumption 3 holds. For any  $\varepsilon ,\delta \geq 0$  , if nested SGDA (resp. saddle-point-oracle SGD) is run with inputs that satisfy for all  $t\in \mathbb{N}_+$ $\eta_{\pmb{x}}^{(t)},\eta_{\pmb{x}}^{(t)}\in \Theta (1 / \sqrt{t + 1})$  , and outputs  $(\pmb {x}^*,\pmb {y}^*,\pmb {\lambda}^*)$  , then in expectation over all runs of the algorithm (i.e., sample paths of  $\pmb{\theta}$  and  $\phi$  ), the action profile  $(\pmb{x}^{*},\pmb{y}^{*})$  is an  $(\varepsilon +\delta ,\delta)$  -SE after  $\tilde{O} (1 / \varepsilon^2\delta^2)$  (resp.  $\tilde{O} (1 / \varepsilon^2)$ ) oracle calls. If, in addition,  $f^{*}$  is  $\mu$  -strongly-convex, then  $(\pmb {x}^{*},\pmb{y}^{*})$  is an  $(\varepsilon +\delta ,\delta)$  -SE after  $\tilde{O} (1 / \varepsilon \delta^2)$  (resp.  $\tilde{O} (1 / \varepsilon)$ ) oracle calls.
4 Policy Gradient in Convex-Concave Zero-Sum Markov Stackelberg Games
In this section, we reduce the computation of Stackelberg equilibrium in zero-sum Markov Stackelberg games to a coupled min-max optimization problem, which enables us to derive a policy gradient method for these games based on our nested SGDA algorithm.
We consider zero-sum Markov Stackelberg games  $\mathcal{M} \doteq (l, n, m, d, \mathcal{S}, \mathcal{A}, \mathcal{B}, \mu, r, \pmb{g}, p, \gamma)$  with state space  $\mathcal{S} \subset \mathbb{R}^l$  and action spaces  $\mathcal{A} \subset \mathbb{R}^n$  and  $\mathcal{B} \subset \mathbb{R}^m$  for the leader and follower, respectively, where the follower's actions are constrained by the leader's via vector-valued state-dependent coupling constraints  $\pmb{g}: \mathcal{S} \times \mathbb{R}^n \times \mathbb{R}^m \to \mathbb{R}^d$  s.t. that define a correspondence  $\mathcal{C}(s, \pmb{a}) \doteq \{\pmb{b} \in \mathcal{B} | g(s, \pmb{a}, \pmb{b}) \geq 0\}$ . We define the set of states with non-trivially coupled constraints  $\mathcal{N} \doteq \{s \in S | \exists (\pmb{a}, \pmb{b}) \in \mathcal{A} \times \mathcal{B}, g(\pmb{s}, \pmb{a}, \pmb{b}) < 0\}$ . A Markov policy for the leader (resp. follower)—hereafter policy for short—is one that is history independent, and thus a mapping from states to actions  $\pi_a: S \to \mathcal{A}$  (resp.  $\pi_b: S \to \mathcal{B}$ ). A policy profile  $\pi \doteq (\pi_a, \pi_b) \in \mathcal{A}^S \times \mathcal{B}^S$  is a tuple comprising policies for the leader and follower, respectively. The follower's feasible policy correspondence is given by  $\mathcal{Z}(\pi_a) = \{\pi_b: S \to \mathcal{B} | \forall s \in \mathcal{N}, g(s, \pi(s)) \geq 0\}$ .
A continuous action zero-sum Markov Stackelberg game is a game where 1. for all states  $s \in S$ , the reward function  $(a, b) \mapsto r(s, a, b)$  is continuous and bounded, i.e.,  $\| r(s, \cdot, \cdot) \|_{\infty} \leq r_{\max} < \infty$ , for some  $r_{\max} \in \mathbb{R}_+$ ; 2. the action spaces  $\mathcal{A}$  and  $\mathcal{B}$  are non-empty and compact; and 3. for all states  $s \in S$ , the correspondence  $a \Vdash C(s, a)$  is continuous, non-empty-, and compact-valued. A continuous state zero-sum Markov Stackelberg game is a game where 1.  $S$  is non-empty and compact and 2. the reward function  $r$  is continuous and bounded, i.e.,  $\| r \|_{\infty} < \infty$ .
A history  $\pmb{h} \in (\mathcal{S} \times \mathcal{A} \times \mathcal{B})^{\tau}$  of length  $\tau \in \mathbb{N}$  is a sequence of state-action tuples  $\pmb{h} = (s^{(t)}, a^{(t)}, b^{(t)})_{t=0}^{\tau-1}$ . Given a policy profile  $\pmb{\pi}$  and a history of play  $\pmb{h}$  of length  $\tau$ , we define the discounted history distribution as  $\nu^{\pi, \tau}(h) = \mu(s^{(0)}) \prod_{t=0}^{\tau-1} \gamma^t p(s^{(t+1)} \mid s^{(t)}, a^{(t)}, b^{(t)}) \mathbb{1}_{\pi(s^{(t)})}(a^{(t)}, b^{(t)})$ . Define the set of all realizable trajectories  $\mathcal{H}^{\pi, \tau}$  of length  $\tau$  under policy  $\pmb{\pi}$  as  $\mathcal{H}^{\pi, \tau} \doteq \mathrm{supp}(\nu^{\pi, \tau})$ , i.e., the set of all histories that occur with non-zero probability. For convenience, we denote by  $\nu^{\pi} \doteq \nu^{\pi, \infty}$ , and by  $H = (S^{(t)}, A^{(t)}, B^{(t)})_t$  any randomly sampled history from  $\nu^{\pi}$ . Finally, we define the discounted state-visitation distribution under any initial state distribution  $\mu$  as  $\delta_\mu^\pi(s) = \sum_{t=0}^\infty \sum_{\pmb{h} \in \mathcal{H}^{\pi, t}: S^{(t)} = s} \mu(s^{(0)}) \prod_{k=1}^t \gamma^k p(s^{(k)} \mid s^{(k-1)}, a^{(k-1)}, b^{(k-1)})$ .
Given a policy profile  $\pi$ , the (state-)value function  $v^{\pi}: S \to \mathbb{R}$  and the action-value function  $q^{\pi}: S \times \mathcal{A} \times \mathcal{B} \to \mathbb{R}$  are defined in terms of  $\nu^{\pi}$  as  $v^{\pi}(s) \doteq \mathbb{E}_{H \sim \nu^{\pi}}\left[\sum_{t=0}^{\infty} r\left(S^{(t)}, A^{(t)}, B^{(t)}\right) \mid S^{(0)} = s\right]$  and  $q^{\pi}(s, \boldsymbol{a}, \boldsymbol{b}) \doteq \mathbb{E}_{H \sim \nu^{\pi}}\left[\sum_{t=0}^{\infty} r\left(S^{(t)}, A^{(t)}, B^{(t)}\right) \mid S^{(0)} = s, A^{(0)} = \boldsymbol{a}, B^{(0)} = \boldsymbol{b}\right]$ , respectively. The cumulative payoff function  $u: \mathcal{A}^S \times \mathcal{B}^S \to \mathbb{R}$  is then defined in terms of the value function as  $u(\pi_{\boldsymbol{a}}, \pi_{\boldsymbol{b}}) \doteq \mathbb{E}_{S \sim \mu}\left[v^{\pi}(S)\right]$ , i.e., the total expected loss (resp. gain) of the leader (resp.
<sup>8</sup>We include detailed theorem statements and proofs in the full version of the paper.
${}^{9}\Lambda$  should be chosen as a superset of all optimal KKT multipliers,i.e.,  ${\cup }_{\mathbf{x} \in  \mathcal{X}}{\Lambda }^{ * }\left( \mathbf{x}\right)  \subseteq  \Lambda$  (see Appendix C).
follower). Additionally, the marginal action-value function  $q^{*\pi}(s, a) \doteq \max_{b \in \mathcal{C}(s, a)} q^{\pi}(s, a, b)$  is the payoff when play initiates at state  $s$  with the leader taking action  $a$ , after which the follower best responds (at state  $s$  only), with both players playing according to  $\pi$  thereafter. Finally, for any leader policy  $\pi_a \in \mathcal{A}^S$ , we define the marginal (state-value) function  $u^*(\pi_a) \doteq \max_{\pi_b \in \mathcal{Z}(\pi_a)} u(\pi_a, \pi_b)$ .
Recursive Stackelberg Equilibrium. A policy profile  $\pi^{*} \doteq (\pi_{a}^{*}, \pi_{b}^{*}) \in \mathcal{A}^{\mathcal{S}} \times \mathcal{B}^{\mathcal{S}}$  is called an  $(\varepsilon, \delta)$ -recursive (or Markov perfect) Stackelberg equilibrium iff  $\forall s \in \mathcal{S}$ ,  $\|\Pi_{\mathbb{R}^d}[g(s, \pi^*(s))]\| \leq \delta$  and  $\max_{\pi_b \in \mathcal{Z}(\pi_a)} v^{(\pi_a^*, \pi_b)}(s) - \delta \leq v^{\pi^*}(s) \leq \min_{\pi_a \in \mathcal{A}^s} \max_{\pi_b \in \mathcal{Z}(\pi_a)} v^{(\overline{\pi}_a, \pi_b)}(s) + \varepsilon$ . A recursive SE is guaranteed to exist in continuous state, continuous action zero-sum Markov Stackelberg games [36]. A policy profile  $\pi^{*} \doteq (\pi_{a}^{*}, \pi_{b}^{*}) \in \mathcal{A}^{\mathcal{S}} \times \mathcal{Z}(\pi_{a}^{*})$  is called an  $(\varepsilon, \delta)$ -Markov perfect GNE iff  $\forall s \in \mathcal{S}$ ,  $\max_{\pi_b \in \mathcal{Z}(\pi_a)} v^{(\pi_a^*, \pi_b)}(s) - \delta \leq v^{\pi^*}(s) \leq \min_{\pi_a \in \mathcal{A}^s} v^{(\pi_a, \pi_b^*)}(s) + \varepsilon$ .
Convex-Concave Markov Stackelberg Games. As we have shown (Theorem 3.1), Stackelberg equilibria can be computed in polynomial time in convex-concave min-max Stackelberg games, assuming access to an unbiased first order-stochastic oracle. We now define an analogous class of Markov Stackelberg games, namely zero-sum Markov Stackelberg games in which the min-max Stackelberg game played at each state is convex-concave. A convex-concave zero-sum Markov Stackelberg game is a continuous state, continuous action zero-sum Markov game where, for all policy profiles  $\pi \in \mathcal{A}^S \times \mathcal{B}^S$ , 1. the marginal action-value function  $(s, a) \mapsto q^{*\pi}(s, a)$  is convex, 2. the action-value function  $(s, b) \mapsto q^{\pi}(s, a, b)$  is concave, for all  $a \in \mathcal{A}$ , 3. the state and action spaces  $\mathcal{S}, \mathcal{A}$  and  $\mathcal{B}$  are convex, and 4. the action correspondence  $\mathcal{C}$  is convex-valued. We note that any continuous state, continuous action convex-concave zero-sum Markov game, i.e.,  $1. \mathcal{N} = \emptyset$ , 2.  $(s, a) \mapsto r(s, a, b)$  is convex, for all  $b \in \mathcal{B}$ , 3.  $(s, b) \mapsto r(s, a, b)$  is concave, for all  $a \in \mathcal{A}$ , 4.  $(s, a) \mapsto p(\cdot | s, a, b)$  is stochastically convex, for all  $b \in \mathcal{B}$ ; and 5.  $(s, b) \mapsto p(\cdot | s, a, b)$  is stochastically concave, for all  $a \in \mathcal{A}$ , is a convex-concave zero-sum Markov Stackelberg game for which the set of Markov perfect generalized Nash equilibria is a subset of the recursive SE.
As our plan is to use our nested SGDA algorithm to compute recursive Stackelberg equilibria, we begin by showing that zero-sum Markov Stackelberg games are an instance of min-max Stackelberg games. Assume parametric policy classes for the leader and follower, respectively, namely  $\mathcal{P}_{\mathcal{X}} \doteq \{\pi_{\mathbf{x}} : \mathcal{S} \to \mathcal{A} \mid \mathbf{x} \in \mathcal{X}\} \subseteq \mathcal{A}^{\mathcal{S}}$  and  $\mathcal{P}_{\mathcal{Y}} \doteq \{\pi_{\mathbf{y}} : \mathcal{S} \to \mathcal{B} \mid \mathbf{y} \in \mathcal{Y}\} \subseteq \mathcal{B}^{\mathcal{S}}$ , for parameter spaces  $\mathcal{X} \subset \mathbb{R}^{d}$  and  $\mathcal{Y} \subset \mathbb{R}^{d}$ . Using these parameterizations, we redefine  $v^{(\mathbf{x},\mathbf{y})} \doteq v^{(\pi_{\mathbf{x}},\pi_{\mathbf{y}})}$ ,  $q^{(\mathbf{x},\mathbf{y})} \doteq q^{(\pi_{\mathbf{x}},\pi_{\mathbf{y}})}$ ,  $u(\mathbf{x},\mathbf{y}) \doteq u(\pi_{\mathbf{x}},\pi_{\mathbf{y}})$ , etc., and thus restate the problem of computing a recursive SE as finding  $(\mathbf{x},\mathbf{y}) \in \mathcal{X} \times \mathcal{Y}$  that solves  $\min_{\mathbf{x} \in \mathcal{X}} \max_{\mathbf{y} \in \mathcal{Y} \in \mathcal{Z}(\mathbf{x})} v^{(\mathbf{x},\mathbf{y})}(\mathbf{s})$ , for all states  $\mathbf{s} \in S$ . As this optimization problem is infinite dimensional for continuous state games, we optimize the objective and satisfy the constraints, both in expectation over the initial state distribution  $\mu$ , thereby reducing the problem to the min-max Stackelberg game  $\min_{\mathbf{x} \in \mathcal{X}} \max_{\mathbf{y} \in \mathcal{Z}(\mathbf{x})} u(\mathbf{x},\mathbf{y})$ .
In Appendix D, assuming 1. biaffine parametric policy classes, i.e.,  $(s, x) \mapsto \pi_x(s)$  and  $(s, y) \mapsto \pi_y(s)$  are biaffine, and 2. non-empty, compact, and convex parameter spaces  $\mathcal{X}$  and  $\mathcal{Y}$ , we show that the min-max Stackelberg game associated with any convex-concave zero-sum Markov Stackelberg game is also convex-concave (Lemma 4). We also provide sufficient conditions on the primitives  $\mathcal{M}$  of any zero-sum Markov Stackelberg game to ensure that it is convex-concave (Lemma 5 and 6). At a high level, our results allow us to conclude that a zero-sum Markov Stackelberg game is convex-concave if the 1. reward (resp. transition probability) function is concave (resp. stochastically concave) in the state and the follower's action; 2. the reward (resp. transition probability) function is convex (resp. stochastically convex) in the state and the leader's follower's actions; and 3. the follower's action correspondence is concave.
Computation. We now turn our attention to the computation of recursive SE in convex-concave zero-sum Markov Stackelberg games. Mirroring the steps by which policy gradient has been show to converge in other settings [24], we first define an unbiased first-order stochastic oracle for zero-sum Markov-Stackelberg games, given access to an unbiased first-order stochastic oracle for the reward and probability transition functions. We then establish convergence of nested SGDA in this setting by invoking Theorem 3.1 under the following assumptions.
Assumption 4 (Convergence Assumptions). 1. The parameter spaces  $\mathcal{X}$  and  $\mathcal{Y}$  are non-empty, compact, and convex; 2. the policy parameterizations are biaffine, i.e.,  $(s,x)\mapsto \pi_x(s)$  and  $(s,y)\mapsto \pi_y(s)$  are biaffine; 3. the set of non-trivially constrained sets is finite  $\mathcal{N}$ , i.e.  $\| \mathcal{N}\| < \infty$ ;
4. (Slater's condition) for all  $s \in \mathcal{N}$  and  $\pmb{a} \in \mathcal{A}$ , there exists  $\widehat{\pmb{b}} \in \mathcal{B}$  s.t.  $g(\pmb{s},\pmb{a},\widehat{\pmb{b}}) > 0$ ; and 5. the reward  $r$ , probability transition  $p$ , and coupling constraint  $g$  functions are Lipschitz-continuous.