[LYBJ21] Qinghua Liu, Tiancheng Yu, Yu Bai, and Chi Jin. A sharp analysis of model-based reinforcement learning with self-play. In International Conference on Machine Learning, pages 7001–7010. PMLR, 2021.
[LZBZ21] Tianyi Lin, Zhengyuan Zhou, Wenjia Ba, and Jiawei Zhang. Doubly optimal no-regret online learning in strongly monotone games with bandit feedback. Available at SSRN 3978421, 2021.
[MOP20] Aryan Mokhtari, Asuman E Ozdaglar, and Sarath Pattathil. Convergence rate of  $o(1 / k)$  for optimistic gradient and extragradient methods in smooth convex-concave saddle point problems. SIAM Journal on Optimization, 30(4):3230-3251, 2020.
[MPP18] Panayotis Mertikopoulos, Christos Papadimitriou, and Georgios Piliouras. Cycles in adversarial regularized learning. In Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 2703-2717. SIAM, 2018.
[MPS20] Vidya Muthukumar, Soham Phade, and Anant Sahai. On the impossibility of convergence of mixed strategies with no regret learning. arXiv preprint arXiv:2012.02125, 2020.
[Neu15] Gergely Neu. Explore no more: Improved high-probability regret bounds for non-stochastic bandits. Advances in Neural Information Processing Systems, 28, 2015.
[PAI69] MA Pollatschek and B Avi-Itzhak. Algorithms for stochastic games with geometrical interpretation. Management Science, 1969.
[PDVH+22] Julien Perolat, Bart De Vylder, Daniel Hennes, Eugene Tarassov, Florian Strub, Vincent de Boer, Paul Muller, Jerome T Connor, Neil Burch, Thomas Anthony, et al. Mastering the game of Stratego with model-free multiagent reinforcement learning. Science, 378(6623):990-996, 2022.
[Put14] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[Sha53] Lloyd S Shapley. Stochastic games. Proceedings of the national academy of sciences, 1953.
[SL99] Csaba Szepesvári and Michael L Littman. A unified analysis of value-function-based reinforcement-learning algorithms. Neural computation, 1999.
[SLY23] Zhuoqing Song, Jason D. Lee, and Zhuoran Yang. Can we find nash equilibria at a linear rate in markov games? In International Conference on Learning Representations, 2023.
[SPO22] Muhammed O Sayin, Francesca Parise, and Asuman Ozdaglar. Fictitious play in zero-sum stochastic games. SIAM Journal on Control and Optimization, 60(4):2095-2114, 2022.
[SSBD14] Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
$\left[\mathrm{SSS}^{+}17\right]$  David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 2017.
[SZL+21] Muhammed Sayin, Kaiqing Zhang, David Leslie, Tamer Basar, and Asuman Ozdaglar. Decentralized q-learning in zero-sum markov games. Advances in Neural Information Processing Systems, 34:18320-18334, 2021.
[TK22] Tatiana Tatarenko and Maryam Kamgarpour. On the rate of convergence of payoff-based algorithms to nash equilibrium in strongly monotone games. arXiv preprint arXiv:2202.11147, 2022.
[Tse95] Paul Tseng. On linear convergence of iterative methods for the variational inequality problem. Journal of Computational and Applied Mathematics, 60(1-2):237-252, 1995.