[51] Jakub Grudzien Kuba, Christian A Schroeder De Witt, and Jakob Foerster. Mirror learning: A unifying framework of policy optimisation. In International Conference on Machine Learning, 2022. (Cited on pages 1, 20, and 23.)
[52] Harold W. Kuhn and Albert W. Tucker. Nonlinear programming. In Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability, 1951. (Cited on page 26.)
[53] Solomon Kullback and Richard A. Leibler. On Information and Sufficiency. The Annals of Mathematical Statistics, 1951. (Cited on page 1.)
[54] Guanghui Lan. Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes. Mathematical programming, 2022. (Cited on pages 1, 8, 21, 22, 25, and 39.)
[55] Guanghui Lan. Policy optimization over general state and action spaces. arXiv preprint arXiv:2211.16715, 2022. (Cited on pages 6, 8, 21, and 24.)
[56] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020. (Cited on page 23.)
[57] Yan Li and Guanghui Lan. Policy mirror descent inherently explores action space. arXiv preprint arXiv:2303.04386, 2023. (Cited on pages 6, 10, and 30.)
[58] Yan Li, Tuo Zhao, and Guanghui Lan. Homotopic policy mirror descent: Policy convergence, implicit regularization, and improved sample complexity. arXiv preprint arXiv:2201.09457, 2022. (Cited on pages 22 and 25.)
[59] Zhize Li, Hongyan Bao, Xiangliang Zhang, and Peter Richtarik. PAGE: A simple and optimal probabilistic gradient estimator for nonconvex optimization. In International Conference on Machine Learning, 2021. (Cited on page 23.)
[60] Boyi Liu, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural trust region/proximal policy optimization attains globally optimal policy. Advances in Neural Information Processing Systems, 2019. (Cited on pages 6, 8, 9, 22, and 24.)
[61] Yanli Liu, Kaiqing Zhang, Tamer Basar, and Wotao Yin. An improved analysis of (variance-reduced) policy gradient and natural policy gradient methods. Advances in Neural Information Processing Systems, 2020. (Cited on pages 8, 22, 23, 24, and 38.)
[62] Stanislaw Lojasiewicz. Une propriété topologique des sous-ensembles analytiques réels. Les équations aux dérivées partielles, 1963. (Cited on page 22.)
[63] Chris Lu, Jakub Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, and Jakob Foerster. Discovered policy optimisation. Advances in Neural Information Processing Systems, 2022. (Cited on page 10.)
[64] Saeed Masiha, Saber Salehkaleybar, Niao He, Negar Kiyavash, and Patrick Thiran. Stochastic second-order methods improve best-known sample complexity of SGD for gradient-dominated functions. In Advances in Neural Information Processing Systems, 2022. (Cited on page 23.)
[65] Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In International Conference on Machine Learning, 2020. (Cited on page 22.)
[66] Jincheng Mei, Yue Gao, Bo Dai, Csaba Szepesvari, and Dale Schuurmans. Leveraging non-uniformity in first-order non-convex optimization. In International Conference on Machine Learning, 2021. (Cited on page 23.)
[67] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. (Cited on page 9.)