[16] John Fearnley. Exponential lower bounds for policy iteration. In Automata, Languages and Programming, pages 551-562, 2010.
[17] Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized Markov decision processes. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2160-2169. PMLR, 2019.
[18] Romain Hollanders, Jean-Charles Delvenne, and Raphael M. Jungers. The complexity of policy iteration is exponential for discounted markov decision processes. In 51st IEEE Conference on Decision and Control (CDC), pages 5997–6002, 2012.
[19] Sham Kakade and John Langford. Approximately Optimal Approximate Reinforcement Learning. In Proceedings of the 19th International Conference on Machine Learning, volume 2, pages 267-274, 2002.
[20] Sham M Kakade. A natural policy gradient. In Advances in Neural Information Processing Systems, volume 14, 2001.
[21] Michael Kearns and Satinder Singh. Finite-sample convergence rates for q-learning and indirect algorithms. In Advances in Neural Information Processing Systems, volume 11, 1998.
[22] Sajad Khodadadian, Prakirt Raj Jhunjhunwala, Sushil Mahavir Varma, and Siva Theja Maguluri. On the linear convergence of natural policy gradient algorithm. In 60th IEEE Conference on Decision and Control (CDC), pages 3794-3799, 2021.
[23] Vijay Konda and John Tsitsiklis. Actor-critic algorithms. In Advances in Neural Information Processing Systems, volume 12, 1999.
[24] Guanghui Lan. Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classes. Mathematical Programming, 2021.
[25] Alessandro Lazaric, Mohammad Ghavamzadeh, and Rémi Munos. Analysis of classification-based policy iteration algorithms. Journal of Machine Learning Research, 17(19):1-30, 2016.
[26] Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Breaking the sample size barrier in model-based reinforcement learning with a generative model. In Advances in Neural Information Processing Systems, volume 33, pages 12861-12872, 2020.
[27] Yan Li, Guanghui Lan, and Tuo Zhao. Homotopic policy mirror descent: Policy convergence, implicit regularization, and improved sample complexity. Mathematical Programming, 2023.
[28] Yanli Liu, Kaiqing Zhang, Tamer Basar, and Wotao Yin. An improved analysis of (variance-reduced) policy gradient and natural policy gradient methods. In Advances in Neural Information Processing Systems, volume 33, pages 7624-7636, 2020.
[29] Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley Series in Probability and Statistics. Wiley, 1994.
[30] R. Tyrrell Rockafellar. Convex Analysis. Princeton Landmarks in Mathematics and Physics. Princeton University Press, 1970.
[31] Bruno Scherrer. Improved and generalized upper bounds on the complexity of policy iteration. In Advances in Neural Information Processing Systems, volume 26, 2013.
[32] Bruno Scherrer. Approximate policy iteration schemes: A comparison. In Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 1314-1322. PMLR, 2014.
[33] Lior Shani, Yonathan Efroni, and Shie Mannor. Adaptive trust region policy optimization: Global convergence and faster rates for regularized mdps. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):5668-5675, 2020.
[34] Aaron Sidford, Mengdi Wang, Xian Wu, Lin Yang, and Yinyu Ye. Near-optimal time and sample complexities for solving markov decision processes with a generative model. In Advances in Neural Information Processing Systems, volume 31, 2018.