[51] James Hannan. Approximation to Bayes risk in repeated play. Contributions to the Theory of Games, 3:97-139, 1957.
[52] Drew Fudenberg and David K Levine. Consistency and cautious fictitious play. Journal of Economic Dynamics and Control, 19(5-7):1065-1089, 1995.
[53] Josef Hofbauer and Ed Hopkins. Learning in perturbed asymmetric games. Games and Economic Behavior, 52(1):133-152, 2005.
[54] Jeff S Shamma and Gurdal Arslan. Unified convergence proofs of continuous-time fictitious play. IEEE Transactions on Automatic Control, 49(7):1137-1141, 2004.
[55] Nicolo Cesa-Bianchi and Gabor Lugosi. Prediction, Learning, and Games. Cambridge University Press, 2006.
[56] Stefanos Leonardos, Will Overman, Ioannis Panageas, and Georgios Piliouras. Global convergence of multi-agent policy gradient in Markov potential games. In International Conference on Learning Representations, 2022.
[57] Sarath Pattathil, Kaiqing Zhang, and Asuman Ozdaglar. Symmetric (optimistic) natural policy gradient for multi-agent learning with parameter convergence. In International Conference on Artificial Intelligence and Statistics, pages 5641-5685. PMLR, 2023.
[58] Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Last-iterate convergence of decentralized optimistic gradient descent/ascent in infinite-horizon competitive Markov games. In Conference on Learning Theory, pages 4259–4299. PMLR, 2021.
[59] Ziyi Chen, Shaocong Ma, and Yi Zhou. Sample efficient stochastic policy extragradient algorithm for zero-sum Markov game. In International Conference on Learning Representations, 2021.
[60] Muhammed O Sayin, Kaiqing Zhang, and Asuman Ozdaglar. Fictitious play in Markov games with single controller. In Proceedings of the 23rd ACM Conference on Economics and Computation, pages 919-936, 2022.
[61] Lucas Baudin and Rida Laraki. Fictitious play and best-response dynamics in identical interest and zero-sum stochastic games. In International Conference on Machine Learning, pages 1664-1690. PMLR, 2022.
[62] Chinmay Maheshwari, Manxi Wu, Druv Pai, and Shankar Sastry. Independent and decentralized learning in Markov potential games. Preprint arXiv:2205.14590, 2022.
[63] Eyal Even-Dar and Yishay Mansour. Learning rates for  $Q$ -learning. Journal of Machine Learning Research, 5(Dec):1-25, 2003.
[64] Guannan Qu and Adam Wierman. Finite-time analysis of asynchronous stochastic approximation and  $Q$ -learning. In Conference on Learning Theory, pages 3185-3205. PMLR, 2020.
[65] Zaiwei Chen, Siva T Maguluri, Sanjay Shakkottai, and Karthikeyan Shanmugam. A Lyapunov theory for finite-sample guarantees of Markovian stochastic approximation. Operations Research, 2023.
[66] Guanghui Lan. Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes. Mathematical programming, pages 1-48, 2022.
[67] Yuling Yan, Gen Li, Yuxin Chen, and Jianqing Fan. The efficacy of pessimism in asynchronous  $Q$ -learning. IEEE Transactions on Information Theory, 2023.
[68] Srihari Govindan, Philip J Reny, Arthur J Robson, et al. A short proof of Harsanyi's purification theorem. Games and Economic Behavior, 45(2):369-374, 2003.
[69] Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3(1):9-44, 1988.