[33] Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized Markov decision processes. In International Conference on Machine Learning, 2019. (Cited on pages 19, 20, 22, 23, and 24.)
[34] Udaya Ghai, Elad Hazan, and Yoram Singer. Exponentiated gradient meets gradient descent. In International Conference on Algorithmic Learning Theory, 2020. (Cited on pages 6, 10, and 30.)
[35] Ian Goodfellow, *Yoshua Bengio*, and Aaron Courville. *Deep learning*. MIT press, 2016. (Cited on page 4.)
[36] Yuzheng Hu, Ziwei Ji, and Matus Telgarsky. Actor-critic is implicitly biased towards high entropy optimal policies. In International Conference on Learning Representations, 2022. (Cited on pages 6, 8, 20, 22, and 24.)
[37] Feihu Huang, Shangqian Gao, and Heng Huang. Bregman gradient policy optimization. In International Conference on Learning Representations, 2022. (Cited on pages 22 and 24.)
[38] Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, 2018. (Cited on page 9.)
[39] Ziwei Ji, Matus Telgarsky, and Ruicheng Xian. Neural tangent kernels, transportation mappings, and universal approximation. In International Conference on Learning Representations, 2019. (Cited on pages 9, 38, 43, and 44.)
[40] Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, 2020. (Cited on pages 22 and 23.)
[41] Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning, 2002. (Cited on page 33.)
[42] Sham M. Kakade. A natural policy gradient. Advances in Neural Information Processing Systems, 2002. (Cited on pages 1, 5, and 21.)
[43] William Karush. Minima of functions of several variables with inequalities as side conditions. Master's thesis, Department of Mathematics, University of Chicago, Chicago, IL, USA, 1939. (Cited on page 26.)
[44] Michael J. Kearns and Daphne Koller. Efficient reinforcement learning in factored mdps. In International Joint Conference on Artificial Intelligence, 1999. (Cited on page 23.)
[45] Sajad Khodadadian, Zaiwei Chen, and Siva Theja Maguluri. Finite-sample analysis of off-policy natural actor-critic algorithm. In International Conference on Machine Learning, 2021. (Cited on pages 22 and 24.)
[46] Sajad Khodadadian, Prakirt Raj Jhunjhunwala, Sushil Mahavir Varma, and Siva Theja Maguluri. On the linear convergence of natural policy gradient algorithm. In IEEE Conference on Decision and Control, 2021. (Cited on pages 22 and 25.)
[47] Sajad Khodadadian, Prakirt Raj Jhunjhunwala, Sushil Mahavir Varma, and Siva Theja Maguluri. On linear and super-linear convergence of natural policy gradient algorithm. Systems and Control Letters, 2022. (Cited on pages 22 and 25.)
[48] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. (Cited on page 10.)
[49] Vijay Konda and John Tsitsiklis. Actor-critic algorithms. In Advances in Neural Information Processing Systems, 2000. (Cited on page 1.)
[50] Walid Krichene, Syrine Krichene, and Alexandre Bayen. Efficient bregman projections onto the simplex. In IEEE Conference on Decision and Control, 2015. (Cited on pages 5, 6, 27, 29, and 30.)