SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Stochastic nested GDA relies on an unbiased first-order stochastic oracle  $(\widehat{F},\widehat{G},\mathcal{F},\mathcal{G})$ , which we can use to obtain unbiased first-order stochastic estimators of  $u$  and  $g$ . Since the constraints are deterministic, we simply set  $\widehat{G} (x,y;s)\doteq (g(s,\pi_x(s),\pi_y(s)))_{s\in S}$  and  $\mathcal{G}(s)\doteq \rho (s)$ , for any distribution  $\rho \in \Delta (\mathcal{S})$  over the state space to obtain an unbiased first-order stochastic oracle for the constraints  $g$ . While for simplicity we define  $\widehat{G}$  as such,  $\widehat{G}$  is tractable to compute (i.e., finite-dimensional) only when  $\mathcal{N}$  is finite. When  $\mathcal{N}$  is infinite, our theoretical results generalize by setting  $\widehat{G} (x,y;s)\doteq (\min_{s\in \mathcal{N}}g_c(s,\pi_x(s),\pi_y(s)))_{c\in [d]}$ ; however, in practice, this estimator might be intractable, in which case one might choose to abandon our theoretical guarantees in favor of the biased estimator  $\widehat{G} (x,y;s)\doteq g(s,\pi_x(s),\pi_y(s))$ . In all cases, the definition of  $\nabla_{(x,y)}\widehat{G}$  follows directly, since  $\widehat{G}$  is deterministic. Now, for any history  $h$  of length  $\tau$ , define the cumulative payoff estimator  $\widehat{R} (\pi ;h)\doteq \sum_{t = 0}^{\tau -1}\mu (s^{(0)})\prod_{k = 0}^{t - 1}\gamma^{k}p(s^{(k + 1)}\mid s^{(k)},\pi (s^{(k)}))r(s^{(k)},\pi (s^{(k)}))$ . We then construct an estimator for  $u$  using first-order gradient estimator [69], i.e., we set  $\widehat{F} (x,y;h)\doteq \widehat{R} (\pi_x,\pi_y;h)$  and  $\nabla_{(x,y)}\widehat{F} (x,y;h)\doteq \nabla_{(x,y)}\widehat{R} (\pi_x,\pi_y;h)$ . Regarding the variances of this oracle model, as  $\widehat{G}$  and  $\nabla_{(x,y)}\widehat{G}$  are deterministic, they have bounded variance. Moreover, if the policy and the reward and transition probability functions are Lipschitz-continuous, then  $\widehat{R}$  and  $\nabla_{(x,y)}\widehat{R}$  are also Lipschitz-continuous if their domains are compact (i.e., if  $\mathcal{S},\mathcal{A}$ , and  $\mathcal{B}$  are compact). Hence  $\widehat{F}$  and  $\nabla F$  likewise must be Lipschitz-continuous, which implies that their variances must be bounded, e.g., there exists  $\sigma_{\nabla f}\in \mathbb{R}$  s.t.  $\mathbb{E}_h[\| \nabla \widehat{F} (x,y;h)\| ^2 ]\leq \| \nabla \widehat{F} (x,y;h)\|_\infty^2 = \sigma_{\nabla f}$  where the middle expression is well-defined since  $\nabla \widehat{F}$  is Lipschitz-continuous over its compact domain.
With all of this machinery in place, we can now extend nested SGDA to compute recursive Stackelberg equilibria in zero-sum Markov Stackelberg games (Algorithm 2; Appendix C). In the usual case, when the policy parameterization does not represent the space of all policies  $\mathcal{A}^S \times \mathcal{B}^S$ , this result should be understood as convergence to the recursive Stackelberg equilibria of a game in which the players' action spaces are restricted to the parameterized policies.
Theorem 4.1. Let  $\mathcal{M}$  be a convex-concave zero-sum Markov Stackelberg game. Under Assumption 4, for any  $\varepsilon, \delta \geq 0$ , if nested policy gradient descent ascent (Algorithm 2, Appendix C) is run with inputs that satisfy for all  $t \in \mathbb{N}_+$ ,  $\eta_{\pmb{x}}^{(t)}, \eta_{\pmb{x}}^{(t)} \in \Theta(1/\sqrt{t+1})$ , and outputs  $(\pmb{x}^*, \pmb{y}^*, \pmb{\lambda}^*)$ , then in expectation over all runs of the algorithm (i.e., sample paths of  $\pmb{\theta}$  and  $\phi$ ), the policy profile  $(\pi_{\pmb{x}^*}, \pi_{\pmb{y}^*})$  is an  $(\varepsilon + \delta, \delta)$ -recursive SE after  $\tilde{O}(1/\varepsilon^2\delta^2)$  oracle calls.
5 Application: Reach-Avoid Problems
In this section, we endeavor to apply our algorithms to a real-world application, namely reach-avoid problems. In a reach-avoid problem (e.g., [29, 32]), an agent seeks to reach one of a set of targets—achieve liveness—while avoiding obstacles along the way—ensuring safety. Reach-avoid problems have myriad applications, including network consensus problems [42], motion planning [21, 41], pursuit-evasion games [30, 44], autonomous driving [43], and path planning [80], to name a few.
The obstacles in a reach-avoid problem are not necessary stationary; they may move, either randomly or deliberately, around the environment. When the obstacles' movement is random, the problem can be modeled as an MDP. But when their movement is deliberate, so that they are more like a rational opponent than a stochastic process, the problem is naturally modeled as a zero-sum game, where the agent—the protagonist—aims to reach its target, while an antagonist—representing the obstacles—seeks to prevent the protagonist from doing so. Past work has modeled these games as simultaneous-move (e.g., [29], [32]), imposing what should be a hard constraint—that the agent cannot collide with any of the obstacles—as a soft constraint in the form of large negative rewards.
Using the framework of zero-sum Markov Stackelberg games, we model this hard constraint properly, with the leader as the antagonist, whose movements impose constraints on the moves of the follower, the protagonist. We then use nested policy GDA to compute Stackelberg equilibria and simultaneous SGDA to compute GNE, and show experimentally that the protagonist learns stronger policies in the sequential (i.e., Stackelberg) game than in the simultaneous.
A (discrete-time discounted infinite-horizon continuous state and action) reach-avoid game  $(l, S, \mathcal{T}, \mathcal{V}, \mathcal{A}, \mathcal{B}, \mu, r, h)$  comprises two players, the antagonist (or  $a$ -player) and the protagonist (or  $b$ -player), each of whom occupies a state  $s_a, s_b \in S$  in a state space  $S \subset \mathbb{R}^l$ , for some  $l \in \mathbb{N}$ . The protagonist's goal is to find a path through the safe set  $\mathcal{V} \subset S \times S$  that reaches a state in the target set  $\mathcal{T} \subset \mathcal{V}$ , while steering clear of the avoid set  $\overline{\mathcal{V}} = S \times S \setminus \mathcal{V}$ . This safe and avoid set formulation is intended to represent capture constraints, which have been the focus of the reach-avoid literature [80].
Initially, the players occupy some state  $s^{(0)} \sim \mu \in \Delta(\mathcal{V})$  drawn from an initial joint distribution  $\mu$  over all states, excluding the target and avoid sets. At each subsequent time-step  $t \in \mathbb{N}_+$ , the antagonist (resp. protagonist) chooses  $a^{(t)} \in \mathcal{A}$  (resp.  $b^{(t)} \in \mathcal{B}$ ) from a set of possible directions  $\mathcal{A} \subseteq \mathbb{R}^l$  (resp.  $\mathcal{B} \subseteq \mathbb{R}^l$ ) in which to move. After both the antagonist and the protagonist move, they receive respective rewards  $-r(s^{(t)}, a^{(t)}, b^{(t)})$  and  $r(s^{(t)}, a^{(t)}, b^{(t)})$ . Then, either the game ends, with probability  $1 - \gamma$ , for some discount rate  $\gamma \in (0,1)$ , or the players move to a new state  $s^{(t + 1)} \doteq h(s^{(t)}, a, b) = (h_a(s_a^{(t)}, a), h_b(s_b^{(t)}, b))$ , as determined by their respective displacement functions  $h_a : S \times \mathcal{A} \to S$  and  $h_b : S \times \mathcal{B} \to S$ . We can express this deterministic transition as the following probability transition function  $p(s' | s, a, b) \doteq \mathbb{1}_{s'}(h(s, a, b))$ .
We define the feasible action correspondence  $\mathcal{C}(s, a) \doteq \{b \in \mathcal{B} \mid \alpha(s, a, b) \geq 0\}$  via a vector-valued safety constraint function  $\alpha: S^2 \times S \times S \to \mathbb{R}^d$ , which outputs a subset of the protagonist's actions in the safe set, i.e., for all  $(s, a) \in S^2 \times \mathcal{A}$ ,  $\mathcal{C}(s, a) \subseteq \{b \in \mathcal{B} \mid h(s, a, b) \in \mathcal{V}\}$ . Note that we do not require this inclusion to hold with equality; in this way, the protagonist can choose to restrict itself to actions far from the boundaries of the avoid set, thereby increasing its safety, albeit perhaps at the cost of liveness. Overloading notation, we define the feasible policy correspondence  $\mathcal{C}(\pi_a) \doteq \{\pi_b: S \to \mathcal{B} \mid \pi_b(s) \in \mathcal{C}(s, \pi_a(s))\}$ , for all  $s \in S$ .
We consider two forms of reward functions. The first, called the reach probability reward,  $r(\boldsymbol{s},\boldsymbol{a},\boldsymbol{b}) = \mathbb{1}_{\mathcal{T}}(\boldsymbol{s}_b)$ , is an indicator function that awards the protagonist with a payoff of 1 if it enters the target set, and 0 otherwise. Under this reward function, the cumulative payoff function (i.e., the expected value of these rewards in the long term) represents the probability that the protagonist reaches the target, hence its name. The second reward function is the reach distance reward function,  $r(\boldsymbol{s},\boldsymbol{a},\boldsymbol{b}) = -\min_{\boldsymbol{s}' \in \mathcal{T}} \| \boldsymbol{s}_b - \boldsymbol{s}' \|^2$ , which penalizes the protagonist based on how far away it is from the target set. With all these definitions in hand, we can now cast the reach-avoid game as a zero-sum Markov Stackelberg game ( $2l, l, l, d, S, A, B, \mu, r, \alpha, p, \gamma$ ).
The next assumption ensures that 1. under the reach probability reward function, a reach-avoid game is a convex-non-concave zero-sum Markov Stackelberg game (i.e., the marginal function  $\pmb{x} \mapsto u^{*}(\pmb{x})$  is convex, and the cumulative payoff function  $\pmb{y} \mapsto u(\pmb{x}, \pmb{y})$  is non-concave, for all  $\pmb{x} \in \mathcal{X}$ ); and 2. under the reach distance reward function, a reach-avoid game is a convex-concave zero-sum Markov Stackelberg game. Furthermore, a Markov perfect GNE is guaranteed to exist under this assumption, assuming the reach distance reward but not under the reach probability distance. $^{10}$
To state this assumption, for convenience, we model the leader's policy  $\pi_{a}(s) \doteq xs_{a}$  as parameterized by  $x \in \mathcal{X} \subset \mathbb{R}^{l \times l}$ , and the follower's policy  $\pi_{b}(s) \doteq ys_{b}$  as parameterized by  $y \in \mathcal{Y} \subset \mathbb{R}^{l \times l}$ . Note also that we assume decentralized, play, meaning the players learn only from their own state and rewards, and maintain their policies independently of one another.
Assumption 5 (Convex-Concave Reach-Avoid Game). 1. The state space  $\mathcal{S}$  and the target set  $\mathcal{T}$  are non-empty and convex; 2. the action spaces  $\mathcal{A}, \mathcal{B}$  are non-empty, compact and convex; 3. the displacement functions  $h_{a}, h_{b}$  are affine; 4.  $a \mapsto \alpha(s, a, b)$  is log-convex for all  $b \in \mathcal{B}$ , and  $b \mapsto \alpha(s, a, b)$  for all  $(s, a) \in S \times \mathcal{A}$ ; 5. the players' parameter spaces  $\mathcal{X}$  and  $\mathcal{Y}$  are non-empty, compact, and convex; and 6. the players policies are biaffine, i.e.,  $\pi_x(s) \doteq xs_a$  and  $\pi_y(s) \doteq xs_b$ .
Part 1 is a standard assumption commonly imposed on reach-avoid games (see, for instance Fisac et al. [29]). Part 3 is satisfied by natural displacement functions of the type  $h(\boldsymbol{s}, \boldsymbol{a}, \boldsymbol{b}) = \boldsymbol{s} + \beta(\boldsymbol{a}, \boldsymbol{b})$ , for some  $\beta \in \mathbb{R}$ , which is a natural characterization of all displacement functions with constant velocity  $\beta$ , when  $\mathcal{A} = \mathcal{B} \subseteq \{z \in S \mid \| z \| = 1\}$ . Part 4 is satisfied by various action correspondences, such as  $\alpha(\boldsymbol{s}, \boldsymbol{a}, \boldsymbol{b}) \doteq \exp \{\min_{s' \in \overline{\mathcal{V}}} \| (\boldsymbol{h}_a(\boldsymbol{s}_a, \boldsymbol{a}), s_b) - s' \| \} - 1 - \| h_b(\boldsymbol{s}_b, \boldsymbol{b}) - s_b \|$ , which shrinks
10The existence of Markov perfect GNE, and hence GNE, is guaranteed by a straightforward generalization of Shapley's [65] result on the existence of Markov perfect Nash equilibria in zero-sum Markov games.
the space of actions exponentially as the protagonist approaches the antagonist, and can thus be interpreted as describing a safety-conscious protagonist. The following theorem states the convex-concavity properties of reach-avoid games, and shows polynomial-time computability of recursive SE under Assumption 5. Note that for the reach probability reward function, it is not possible to obtain a polynomial-time convergence result, result since the rewards are not even continuous.
Theorem 5.1. Under the reach distance (resp. reach probability) reward function, any reach-avoid game for which Assumption 5 hold is convex-concave (resp. convex-non-concave). Moreover, if  $\alpha$  is Lipschitz-continuous, then nested SGDA is guaranteed to converge in such games to recursive SE policies in polynomial time.
Experiments. We ran a series of experiments on reach-avoid problems, $^{11}$  which were designed to assess the efficacy of policies learned in a Stackelberg game formulation as compared to those learned in a simultaneous-move game formulation, assuming complex, i.e., neural, policy parameterizations.
We consider a variant of the two-player differential game introduced in Isaacs [38], played by two Dubins cars. A Dubins car is a simplified model of a vehicle with a constant forward speed  $\nu$  and a constrained turning radius  $\omega$ . We model both the protagonist and antagonist as Dubins cars [38] moving around a 2-dimensional state space. The target set is a select subset of the state space, while the avoid set, which defines the safe set, is a ball around the antagonist.
We experiment with only the reach distance, not the reach probability, reward function. In all safe states, the reward is actually a penalty, measuring the protagonist's distance to the target set, while a bonus  $\beta$  is awarded upon reaching a target, at which point the game ends. This reward function suffices for our Stackelberg game setup, which enforces the hard constraint that the protagonist cannot move into the avoid set. In our simultaneous-move game setup, we achieve a similar effect by enhancing the aforementioned reward function with a large penalty  $(- \beta)$  whenever the protagonist touches the avoid set. As in the case of reaching the target, touching the avoid set ends the game.
We note that this reach-avoid game is not actually a continuous game, as there is a discontinuity in the reward function when the target is reached. Additionally, it is possible for the antagonist to be "cornered," meaning left with an empty set of feasible actions (in which case the game ends). For these reasons, recursive SE are not guaranteed to exist in our setup.
Our experiments were run on a 7x7 square grid, with the target set  $\mathcal{T}$  a closed ball of radius 1 centered along the lower edge, and the avoid set  $\overline{\nu}$  a closed ball of radius 0.3 around the antagonist. We set the bonus (resp. penalty) for reaching the target (resp. avoid set)  $\beta = 200$ ,  $\omega = 30^{\circ}$ , and  $\nu = 0.25$ .
Using this experimental setup, we train two agents by playing two games, the Stackelberg and simultaneous-move variants of the reach-avoid game, using nested policy GDA and SGDA, respectively. We evaluate the protagonists' policies to assess their safety and liveness characteristics.
To assess liveness, we ran our agents against an opponent that plays actions sampled uniformly at random. To assess safety, we ran our agents against an
Match-up	Outcome	Mean win length	Loss/draw length
GNE vs. random	47 W, 18 L, 35 D	23.23 ± 7.53	33.71 ± 19.31
SE vs. random	95 W, 2 L, 3 D	18.16 ± 3.69	33.0 ± 20.8
GNE vs. chaser	0 W, 100 L, 0 D	N/A	8.53 ± 1.90
SE vs. chaser	63 W, 36 L, 1 D	21.63 ± 5.04	11.06 ± 7.71
Table 1: Game results summary for GNE and SE agents.
opponent who chases them, always taking actions that minimize their distance. Table 1 reports the number of wins (W), losses (L), and draws (D), and average game lengths, of 100 games against each opponent. An agent, playing the role of the protagonist, wins when it reaches the target set. A GNE agent loses if it enters  $\overline{\nu}$ , while a Stackelberg agent loses if it finds itself cornered. The game is a draw if neither player wins or loses within 50 time steps.
We find that the SE agent outperforms the GNE agent by a large margin. The SE agent wins almost all of its games against random, and roughly  $\frac{2}{3}$  of its games against the chaser, while the GNE agent wins only half of its games against random, and none of its games against the chaser. Moreover, even when the SE agent loses or draws, it tends to stay alive longer than the GNE agent. Not only does our Stackelberg approach outperform GNE, it is tractable as well. Our methods thus seem to offer a promising path to further progress solving the myriad of robotic applications of reach-avoid.
11Our code is found at: https://github.com/arjun-prakash/stackelberg-reach-avoid.
Denizalp Goktas was supported by a JP Morgan AI fellowship. Arjun Prakash was partially supported by ONR N00014-22-1-2592 and the Quad Fellowship.
[1] Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. Optimality and approximation with policy gradient methods in markov decision processes. In Conference on Learning Theory, pages 64-66. PMLR, 2020. 26
[2] Alekh Agarwal, Sham M. Kakade, Jason D. Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. 22(1), jan 2021. ISSN 1532-4435. 1, 26
[3] Eitan Altman. Flow control using the theory of zero sum markov games. IEEE transactions on automatic control, 39(4):814-818, 1994. 1
[4] Kenneth Arrow and Gerard Debreu. Existence of an equilibrium for a competitive economy. Econometrica: Journal of the Econometric Society, pages 265-290, 1954. 4
[5] Kenneth J. Arrow and Leonid Hurwicz. On the stability of the competitive equilibrium, i. Econometrica, 26(4):522-552, 1958. ISSN 00129682, 14680262. URL http://www.jstor.org/stable/1907515.5
[6] Kenneth Joseph Arrow, Leonid Hurwicz, and Hirofumi Uzawa. Studies in linear and non-linear programming. 1958. 5
[7] Alp E. Atakan. Stochastic convexity in dynamic programming. Economic Theory, 22(2): 447-455, 2003. ISSN 09382259, 14320479. URL http://www.jstor.org/stable/25055693.3, 26
[8] Yu Bai, Chi Jin, Huan Wang, and Caiming Xiong. Sample-efficient learning of stackelberg equilibria in general-sum games. Advances in Neural Information Processing Systems, 34: 25799-25811, 2021. 16
[9] Stefan Banach. Sur les operations dans les ensembles abstraits et leur application aux equations integrales. Fund. math, 3(1):133-181, 1922. 25
[10] Somil Bansal, Mo Chen, Sylvia Herbert, and Claire J Tomlin. Hamilton-jacobi reachability: A brief overview and recent advances. In 2017 IEEE 56th Annual Conference on Decision and Control (CDC), pages 2242-2253. IEEE, 2017. 2
[11] Richard Bellman. On the theory of dynamic programming. Proceedings of the National Academy of Sciences of the United States of America, 38(8):716, 1952. 24, 25
[12] Alain Bensoussan, Shaokuan Chen, and Suresh P Sethi. The maximum principle for global solutions of stochastic stackelberg differential games. SIAM Journal on Control and Optimization, 53(4):1956-1981, 2015. 16
[13] Dimitri Bertsekas. Convex optimization theory, volume 1. Athena Scientific, 2009. 4
[14] Dimitri Bertsekas. Dynamic programming and optimal control: Volume I, volume 4. Athena scientific, 2012. 24
[15] Dimitris Bertsimas, David B Brown, and Constantine Caramanis. Theory and applications of robust optimization. SIAM review, 53(3):464-501, 2011. 2
[16] Jalaj Bhandari and Daniel Russo. Global optimality guarantees for policy gradient methods. arXiv preprint arXiv:1906.01786, 2019. 1
[17] Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004. 4
[18] Steven J Bradtke and Andrew G Barto. Linear least-squares algorithms for temporal difference learning. Machine learning, 22:33-57, 1996. 24
[19] Yanling Chang, Alan L Erera, and Chelsea C White. A leader–follower partially observed, multiobjective markov game. Annals of Operations Research, 235(1):103–128, 2015. 16
[79] Kaiqing Zhang, Alec Koppel, Hao Zhu, and Tamer Basar. Global convergence of policy gradient methods to (almost) locally optimal policies. SIAM Journal on Control and Optimization, 58 (6):3586-3612, 2020. 26
[80] Zhengyuan Zhou, Jerry Ding, Haomiao Huang, Ryo Takei, and Claire Tomlin. Efficient path planning algorithms in reach-avoid problems. Automatica, 89:28–36, 2018. 8, 9