[59] Praveen Palanisamy. Multi-agent connected autonomous driving using deep reinforcement learning. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1-7. IEEE, 2020. 1
[60] Matteo Papini, Damiano Binaghi, Giuseppe Canonaco, Matteo Pirotta, and Marcello Restelli. Stochastic variance-reduced policy gradient. In International conference on machine learning, pages 4026-4035. PMLR, 2018. 26
[61] Giorgia Ramponi and Marcello Restelli. Learning in markov games: can we exploit a general-sum opponent? In The 38th Conference on Uncertainty in Artificial Intelligence, 2022. 16
[62] Sean C Rismiller, Jonathan Cagan, and Christopher McComb. Stochastic stackelberg games for agent-driven robust design. In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, volume 84003, page V11AT11A039. American Society of Mechanical Engineers, 2020. 16
[63] R Tyrrell Rockafellar and Roger J-B Wets. Variational analysis, volume 317. Springer Science and Business Media, 2009. 3, 4
[64] Sailik Sengupta and Subbarao Kambhampati. Multi-agent reinforcement learning in bayesian stackelberg markov games for adaptive moving target defense. arXiv preprint arXiv:2007.10457, 2020. 16
[65] Lloyd S Shapley. Stochastic games. Proceedings of the national academy of sciences, 39(10): 1095-1100, 1953. 1, 9
[66] Zebang Shen, Alejandro Ribeiro, Hamed Hassani, Hui Qian, and Chao Mi. Hessian aided policy gradient. In International conference on machine learning, pages 5729-5738. PMLR, 2019. 26
[67] Maurice Sion. On general minimax theorems. Pacific Journal of mathematics, 8(1):171-176, 1958. 4
[68] Morton Slater. Lagrange multipliers revisited. Cowles Foundation Discussion Papers 80, Cowles Foundation for Research in Economics, Yale University, 1959. URL https:// EconPapers.repec.org/RePEc:cwl:cwldpp:80.5
[69] Hyung Ju Suh, Max Simchowitz, Kaiqing Zhang, and Russ Tedrake. Do differentiable simulators give better policy gradients? In International Conference on Machine Learning, pages 20668-20696. PMLR, 2022. 8
[70] Masayuki Takahashi. Equilibrium points of stochastic non-cooperative  $n$ -person games. Journal of Science of the Hiroshima University, Series AI (Mathematics), 28(1):95-99, 1964. 1
[71] Ioannis Tsaknakis, Mingyi Hong, and Shuzhong Zhang. Minimax problems with coupled linear constraints: Computational complexity, duality and solution methods. arXiv preprint arXiv:2110.11210, 2021. 4
[72] Deepanshu Vasal. Stochastic stackelberg games, 2020. 16
[73] John von Neumann. Zur theorie der gesellschaftsspiele. Mathematische annalen, 100(1): 295-320, 1928. 4
[74] Heinrich von Stackelberg. Marktform und gleichgewicht. J. springer, 1934. 1
[75] Yevgeniy Vorobeychik and Satinder Singh. Computing stackelberg equilibria in discounted stochastic games. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 26, pages 1478-1484, 2012. 16
[76] Quoc-Liem Vu, Zane Alumbaugh, Ryan Ching, Quanches Ding, Arnav Mahajan, Benjamin Chasnov, Sam Burden, and Lillian J Ratliff. Stackelberg policy gradient: Evaluating the performance of leaders and followers. In ICLR 2022 Workshop on Gamification and Multiagent Solutions, 2022. 16
[77] Yuandou Wang, Hang Liu, Wanbo Zheng, Yunni Xia, Yawen Li, Peng Chen, Kunyin Guo, and Hong Xie. Multi-objective workflow scheduling with deep-q-network-based multi-agent reinforcement learning. IEEE access, 7:39974-39982, 2019. 1
[78] Yu Yuan, Zhibin Liang, and Xia Han. Robust reinsurance contract with asymmetric information in a stochastic stackelberg differential game. Scandinavian Actuarial Journal, pages 1-28, 2021. 16