[61] Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Proceedings of the Conference on Learning Theory, pages 2137-2143, 2020.
[62] Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global optimality and rates of convergence. In Proceedings of the International Conference on Learning Representations, 2020.
[63] Qinbo Bai, Amrit Singh Bedi, and Vaneet Aggarwal. Achieving zero constraint violation for constrained reinforcement learning via conservative natural policy gradient primal-dual algorithm. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 6737-6744, 2023.
[64] Alexander Rakhlin and Karthik Sridharan. Online learning with predictable sequences. In Proceedings of the Conference on Learning Theory, pages 993-1019. PMLR, 2013.
[65] Lin Xiao. On the convergence rates of policy gradient methods. Journal of Machine Learning Research, 23(282):1-36, 2022.
[66] Laurent Condat. Fast projection onto the simplex and the  $l_{1}$  ball. Mathematical Programming, 158(1-2):575-585, 2016.
[67] C Daskalakis and Ioannis Panageas. Last-iterate convergence: Zero-sum games and constrained min-max optimization. In Proceedings of the Innovations in Theoretical Computer Science (ITCS) Conference, 2019.
[68] Qi Lei, Sai Ganesh Nagarajan, Ioannis Panageas, et al. Last iterate convergence in no-regret learning: constrained min-max optimization for convex-concave landscapes. In Proceedings of the International Conference on Artificial Intelligence and Statistics, pages 1441-1449, 2021.
[69] Dongsheng Ding, Kaiqing Zhang, Tamer Başar, and Mihailo R Jovanović. Convergence and optimality of policy gradient primal-dual method for constrained Markov decision processes. In Proceedings of the American Control Conference, pages 2851-2856, 2022.
[70] Ruida Zhou, Tao Liu, Dileep Kalathil, Panganamala Kumar, and Chao Tian. Anchoring regularized natural policy gradient for multi-objective reinforcement learning. In Proceedings of the Advances in Neural Information Processing Systems, 2022.
[71] Aria HasanzadeZonuzy, Dileep Kalathil, and Srinivas Shakkottai. Model-based reinforcement learning for infinite-horizon discounted constrained Markov decision processes. In Proceedings of the International Joint Conference on Artificial Intelligence, 2021.
[72] Leonid Denisovich Popov. A modification of the Arrow-Hurwicz method for search of saddle points. Mathematical Notes of the Academy of Sciences of the USSR, 28:845-848, 1980.
[73] Ming Yu, Zhuoran Yang, Mladen Kolar, and Zhaoran Wang. Convergent policy optimization for safe reinforcement learning. In Proceedings of the Advances in Neural Information Processing Systems, volume 32, 2019.
[74] Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection-based constrained policy optimization. In Proceedings of the International Conference on Learning Representations, 2020.
[75] Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. In Proceedings of the Advances in Neural Information Processing Systems, volume 33, pages 15338-15349, 2020.
[76] Long Yang, Yu Zhang, Jiaming Ji, Juntao Dai, and Weidong Zhang. CUP: A conservative update policy algorithm for safe reinforcement learning, 2022.