SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$(--)$  is damped, and OPG-PD  $(-)$  is almost free of oscillation as PID Lagrangian  $(\dots)$ . However, oscillation of NPG-PD  $(-)$  causes its last-iterate policy violating the constraint  $V_{g}^{\pi}(\rho) \geq 0$ . OPG-PD  $(-)$  reaches the maximum reward value in four methods and RPG-PD  $(-)$  converges to a slightly smaller value due to regularization, while both meet the constraint at the end. However, PID Lagrangian  $(\dots)$  is highly sub-optimal. Hence, our methods OPG-PD and RPG-PD can overcome oscillation and approach a nearly-optimal constrained policy in the last-iterate fashion.
Figure 1: Convergence performance of RPG-PD, OPG-PD, and primal-dual methods. Learning curves of our RPG-PD  $(- - )$  and OPG-PD  $(-)$ , and NPG-PD [23]  $(- - )$  and PID Lagrangian [29]  $(\dots)$  methods. The horizontal axes mean the policy iterations  $\{\pi_t\}_{t\geq 0}$  that are generated by each method and the vertical axes mean the value functions of the policy iterates  $\{\pi_t\}_{t\geq 0}$ : reward value  $V_{r}^{\pi_{t}}(\rho)$  (Left) and utility value  $V_{g}^{\pi_{t}}(\rho)$  (Right). In this experiment, we use the same stepsize  $\eta = 0.1$  for all methods, the regularization parameter  $\tau = 0.08$  for RPG-PD, and the uniform initial distribution  $\rho$ .
We showcase the linear convergence of OPG-PD (9) with three constant step sizes in Figure 2. Three policy optimality gaps decrease linearly in the logarithmic scale plot, which verifies the linear last-iterate convergence of OPG-PD's policy iterates in Theorem 6. Noticeably, there is no oscillation behavior in OPG-PD's policy iterates, which perhaps is best for learning constraints [29, 10]. We also see that a large stepsize  $\eta = 0.2$  improves the convergence, which is reflected by our rate.
Figure 2: Convergence performance of OPG-PD with stepsize  $\eta$ :  $(\eta = 0.05, \dots)$ ,  $(\eta = 0.1, \dots)$ ,  $(\eta = 0.2, \dots)$ . The horizontal axis represents the policy iterations  $\{\pi_t\}_{t \geq 0}$  that are generated by OPG-PD and the vertical axis means the policy optimality gap that measures the distance of the policy iterates  $\{\pi_t\}_{t \geq 0}$  to an optimal policy  $\pi^{\star} \colon \sum_{s} \| \pi_t(\cdot | s) - \pi^{\star}(\cdot | s) \|^2$ . In this experiment, we take the initial distribution  $\rho$  to be a uniform one.
Please see Appendix E for more details of this experiment, more baselines, and sensitivity analysis.
We have presented two single-time-scale policy-based primal-dual methods for finding an optimal policy of a constrained MDP, with global non-asymptotic and last-iterate policy convergence guarantees. Our first regularized method enjoys a nearly dimension-free sublinear rate, while our second optimistic method possesses a linear rate that is problem-dependent. Our work stimulates a number of compelling future directions: (i) Our problem setting circumvents the exploration difficulty, which leaves online exploration open; (ii) Our convergence rates are not as sharp as solving convex-concave minimax optimization problems, regarding the order or instance-related constant; (iii) Last-iterate convergence is under-examined in constrained MDPs with other constraints, and unexplored for other gradient methods.
D. Ding and A. Ribeiro were supported by THEORINET Simons-NSF MoDL, and DCIST CRA. K. Zhang acknowledges the support from Simons-Berkeley Research Fellowship and Northrop Grumman - Maryland Seed Grant Program.
[1] Frederick J Beutler and Keith W Ross. Optimal policies for controlled Markov chains with a constraint. Journal of Mathematical Analysis and Applications, 112(1):236-252, 1985.
[2] Keith W Ross. Randomized and past-dependent policies for Markov decision processes with multiple constraints. Operations Research, 37(3):474-477, 1989.
[3] Alexey Piumovskiy. Control of random sequences in problems with constraints. Theory of Probability & Its Applications, 38(4):751-762, 1994.
[4] Eugene A Feinberg and Adam Shwartz. Constrained discounted dynamic programming. Mathematics of Operations Research, 21(4):922-945, 1996.
[5] Eitan Altman. Constrained Markov decision processes, volume 7. CRC press, 1999.
[6] Vivek S Borkar. An actor-critic algorithm for constrained Markov decision processes. Systems & Control Letters, 54(3):207-213, 2005.
[7] Naoki Abe, Prem Melville, Cezar Pendus, Chandan K Reddy, David L Jensen, Vince P Thomas, James J Bennett, Gary F Anderson, Brent R Cooley, Melissa Kowalczyk, et al. Optimizing debt collections using constrained reinforcement learning. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 75–84, 2010.
[8] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of the International Conference on Machine Learning, pages 22-31, 2017.
[9] Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. In Proceedings of the International Conference on Learning Representations, 2019.
[10] Gabriel Dulac-Arnold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, 110(9):2419-2468, 2021.
[11] Amol Mandhane, Anton Zhernov, Maribeth Rauh, Chenjie Gu, Miaosen Wang, Flora Xue, Wendy Shang, Derek Pang, Rene Claus, Ching-Han Chiang, et al. Muzero with self-competition for rate control in VP9 video compression. arXiv preprint arXiv:2202.06626, 2022.
[12] Santiago Paternain, Miguel Calvo-Fullana, Luiz FO Chamon, and Alejandro Ribeiro. Safe policies for reinforcement learning via primal-dual methods. IEEE Transactions on Automatic Control, 2022.
[13] Xiangkun He, Haohan Yang, Zhongxu Hu, and Chen Lv. Robust lane change decision making for autonomous vehicles: An observation adversarial reinforcement learning approach. IEEE Transactions on Intelligent Vehicles, 2022.
[14] Ziqing Gu, Lingping Gao, Haitong Ma, Shengbo Eben Li, Sifa Zheng, Wei Jing, and Junbo Chen. Safe-state enhancement method for autonomous driving via direct hierarchical reinforcement learning. IEEE Transactions on Intelligent Transportation Systems, 2023.
[110] Alp Yurtsever, Quoc Tran Dinh, and Volkan Cevher. A universal primal-dual convex optimization framework. In Proceedings of the Advances in Neural Information Processing Systems, volume 28, 2015.
[111] Bingsheng He and Xiaoming Yuan. On non-ergodic convergence rate of Douglas-Rachford alternating direction method of multipliers. Numerische Mathematik, 130(3):567-577, 2015.
[112] Ya-Feng Liu, Xin Liu, and Shiqian Ma. On the nonergodic convergence rate of an inexact augmented Lagrangian framework for composite convex programming. Mathematics of Operations Research, 44(2):632-650, 2019.
[113] Yangyang Xu. Iteration complexity of inexact augmented Lagrangian methods for constrained convex programming. Mathematical Programming, 185:199-244, 2021.
[114] Quoc Tran Dinh. Non-ergodic alternating proximal augmented Lagrangian algorithms with optimal rates. Proceedings of the Advances in Neural Information Processing Systems, 31, 2018.
[115] Quoc Tran-Dinh and Yuzixuan Zhu. Augmented Lagrangian-based decomposition methods with non-ergodic optimal rates. arXiv preprint arXiv:1806.05280, 2018.
[116] Shoham Sabach and Marc Teboulle. Faster Lagrangian-based methods in convex optimization. SIAM Journal on Optimization, 32(1):204-227, 2022.
[117] Tao Zhang, Yong Xia, and Shiru Li. Lagrangian-based methods in convex optimization: prediction-correction frameworks with non-ergodic convergence rates. arXiv preprint arXiv:2304.02459, 2023.
[118] Andrew Cotter, Heinrich Jiang, and Karthik Sridharan. Two-player games for efficient nonconvex constrained optimization. In Proceedings of the Algorithmic Learning Theory, pages 300-332, 2019.
[119] Richard Bellman and Stuart Dreyfus. Functional approximations and dynamic programming. Mathematics of Computation, 13(68):247-251, 1959.
[120] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[121] Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Proceedings of the Conference on Learning Theory, pages 2730-2775, 2022.
[122] John Nash Jr. Non-cooperative games. Annals of Mathematics, 54(2):286-295, 1951.
[123] Vivek S Borkar. A convex analytic approach to Markov decision processes. Probability Theory and Related Fields, 78(4):583-602, 1988.
[124] Junyu Zhang, Amrit Singh Bedi, Mengdi Wang, and Alec Koppel. Cautious reinforcement learning via distributional risk in the dual domain. IEEE Journal on Selected Areas in Information Theory, 2(2):611-626, 2021.
[125] Qinbo Bai, Amrit Singh Bedi, Mridul Agarwal, Alec Koppel, and Vaneet Aggarwal. Achieving zero constraint violation for concave utility constrained reinforcement learning via primal-dual approach. 2022.
[126] Donghao Ying, Mengzi Amy Guo, Yuhao Ding, Javad Lavaei, and Zuo-Jun Shen. Policy-based primal-dual methods for convex constrained Markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 10963-10971, 2023.
[127] Maurice Sion. On general minimax theorems. Pacific Journal of Mathematics, 8(1):171-176, 1958.
[128] Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the International Conference on Machine Learning, pages 928-936, 2003.
[129] Yasushi Terazono and Ayumu Matani. Continuity of optimal solution functions and their conditions on objective functions. SIAM Journal on Optimization, 25(4):2050-2060, 2015.
[130] Simon Lacoste-Julien, Mark Schmidt, and Francis Bach. A simpler approach to obtaining an  $\mathrm{O}(1 / t)$  convergence rate for the projected stochastic subgradient method. arXiv preprint arXiv:1212.2002, 2012.
[131] Yuhao Ding, Junzi Zhang, and Javad Lavaei. Beyond exact gradients: Convergence of stochastic soft-max policy gradient methods with entropy regularization. arXiv preprint arXiv:2110.10117, 2021.
[132] Haipeng Luo. Introduction to online optimization/learning: Lecture 1. https://haipeng-luo.net/courses/CSCI659/2022_fall/lectures/lecture1.pdf, 2022. Access on January 21, 2023.
Supplementary Materials for "Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPs"
A More Comparisons and Additional Related Works
In this section, we discuss more comparison details and other related works.
Iterate Type	Method	Single-time-scale	Complexity
Occupancy measure	Saddle flow dynamics [36]	Yes	asymptotic
ReLOAD [40]	Yes	asymptotic
OPD (18)	Yes	Ω (log2 1/ε)
Policy	Dual descent [34]	No	Ω (log2 1/ε)
Cutting-plane [35]	No	Ω (log3 1/ε)
Policy-based ReLOAD [40]	Yes	—
RPG-PD (6)	Yes	Ω (1/ε6 log2 1/ε)
OPG-PD (9)	Yes	Ω (log2 1/ε)
Table 1: Iteration complexities of our methods and representative algorithms for solving a constrained MDP problem: maximize $_{\pi} V_r^\pi(\rho)$  subject to  $V_g^\pi(\rho) \geq 0$  with reward/utility functions  $r \in [0,1]$ ,  $g \in [-1,1]$ . The iteration complexity is the number of gradient-based updates for an algorithm to output the last policy-iterate  $\pi_t$  that satisfies  $V_r^\star(\rho) - V_r^{\pi_t}(\rho) \leq \epsilon$  and  $-V_g^{\pi_t}(\rho) \leq \epsilon$ .
Last-iterate and value-average (or policy-mixture) performance in constrained MDPs. There has been a flurry of research activities in studying convergence behaviors of direct policy search or policy gradient-based algorithms for constrained MDPs in the infinite-horizon discounted setting. There are two main streams: (i) Lagrangian-based policy search and (ii) Approximate constrained policy search.
(i) Lagrangian-based policy search. In the Lagrangian-based framework, last-iterate performance has been established as asymptotic convergence for several policy-based primal-dual algorithms, e.g., naive policy gradient-based primal-dual method [21] and actor-critic variants of policy gradient primal-dual methods [6, 22, 16, 9]. These studies rely on modeling primal-dual updates in two separate time scales and/or two nested loops as continuous-time gradient flow dynamics, and their asymptotic convergence is often restricted to some stationary points. We notice that recent global asymptotic convergence results [36, 40] are in terms of occupancy measure iterates instead of instantaneous policy iterates, as highlighted in Table 1. In contrast, our first method: OPD strengthens the asymptotic last-iterate convergence to non-asymptotic and last-iterate convergence with a linear rate.
To provide efficiency and optimality performances, it is crucial to develop algorithms with global non-asymptotic convergence guarantees. Several recent policy-based primal-dual algorithms have been proved to converge with non-asymptotic convergence rates, e.g., policy gradient primal-dual method [69], natural policy gradient-based or policy mirror-descent style primal-dual methods [23, 27, 54], accelerated natural policy gradient-based primal-dual methods [24, 25, 28], actor-critic version of natural policy gradient-based primal-dual
method [26], and anchor-changing natural policy gradient-based primal-dual method [70]. These studies characterize non-asymptotic convergence to an optimal constrained policy regarding the average of value functions, except for [27] in which the convergence is for a mixture of past policies and [28] in which the convergence is for a policy induced by a history-weighted occupancy measure. Similar non-asymptotic convergence can also be found in an occupancy measure-based primal-dual method [42], where the convergence is in terms of the average of occupancy measures. Besides, sublinear non-asymptotic convergence can be found in generative model-based methods [71, 41], regarding a mixture of past policies. In contrast, our two policy-based methods: RPG-PD and OPG-PD strengthen the sublinear non-asymptotic convergence of average value functions or a mixture of past policies to sublinear and linear non-asymptotic convergence of policy iterates. In particular, we exploit the regularization technique [52] and the optimistic gradient method [72, 37] to augment typical policy-based primal-dual methods with novel identification of decreasing distances of policy iterates to an optimal constrained policy, which allows for stronger convergence.
Instead of working with policy primal and dual variables both, Lagrangian-based dual method formulates a constrained MDP as a convex dual problem that enables classical dual ascent method [12]. Despite guaranteed convergence in dual space from convex optimization, it is challenging to compute an optimal constrained policy in primal policy space even if we use an optimal dual variable [9, 47, 30]. Recently, regularization [52] has been used in dual ascent methods [34, 35] in which policy last-iterates of natural policy gradient-based subroutines can be nearly-optimal. These dual-based algorithms [34, 35] work with two nested loops and their non-asymptotic last-iterate convergence relies on tuning loop parameters optimally. In contrast, our two policy-based methods: RPG-PD and OPG-PD remove the double loop requirement as listed in Table 1, which permits outputting the last-iterate policy as a nearly-optimal constrained policy. In Table 1, we see that our RPG-PD method has a worse rate while OPG-PD achieves a linear rate which is similar as the dual-based methods [34, 35]. Importantly, our non-asymptotic convergence characterizes the stability of primal-dual iterates generated by the algorithms, which is theoretically stronger and more appealing in practice.
(ii) Approximate constrained policy search. Approximation of constrained MDPs with surrogate functions has been shown to be effective, e.g., constrained policy optimization [8], successive convex relaxation [73], projection-based constrained policy optimization [74], first-order constrained optimization [75], and conservative policy update [76]. These studies have shown impressive empirical performance by iteratively solving an approximated constrained optimization problem and such performance is characterized in the worst-case policy improvement, except for [73] in which local asymptotic convergence is established, which leaves non-asymptotic convergence of policy iterates open. A related approach is the primal method [77] that treats a constrained MDP as an unconstrained one and corrects policy iterates whenever constraint violation happens. Non-asymptotic convergence of primal method has been established in terms of mixture of past policies. Another approximation method is the interior-point policy optimization [78] that solves an unconstrained MDP by adding a logarithm barrier function into the objective function, while convergence of this method is unknown. In contrast, we have supported our methods with non-asymptotic last-iterate convergence. Since Lagrangian-based methods are typically used to solve an approximated constrained optimization problem [8, 74, 75, 76], our methodologies can be applied to these methods for better convergence, which we leave as future work.
For constrained MDPs in the finite-horizon episodic setting and the total or average-reward settings, there is a rich line of works that have developed learning algorithms with non-asymptotic convergence guarantees in terms of the average of value functions [79, 80, 81, 82, 83, 84, 85, 55, 86, 87, 88, 89, 56, 90, 91, 92, 93, 94, 95], except for [96] in which global asymptotic convergence of a policy gradient method has been established in the finite-horizon constrained MDP setting. Although being not directly comparable, their non-asymptotic and last-iterate convergence are not established yet, to the best of our knowledge.
Lagrangian-based policy gradient methods in constrained MDPs. Our methods are closely pertinent to Lagrangian-based policy gradient methods for solving constrained MDPs in the infinite-horizon discounted setting, e.g., policy gradient-based primal-dual methods [21, 6, 22, 16, 9, 69],
natural policy gradient-based or mirror-descent style primal-dual methods [23, 27, 28, 24, 54, 25, 26], dual descent methods [97, 34, 35, 12]. Regarding algorithm implementation, primal-dual methods [21, 6, 22, 16, 9, 23, 27, 28, 24, 54, 25, 26, 69] work with primal-dual iterates simultaneously in a single loop, which is similar to the classical gradient-based primal-dual methods in constrained optimization [31, 32, 33], while diminishing step sizes in different speeds is often required in many of them [21, 6, 16, 9]; dual descent methods [97, 34, 35, 12] intermittently operate the dual iterate only after a sufficient number of primal iterations, which often adds difficulty of tuning hyper-parameters of nested loops in practice. It is worth mentioning that, it is convenient to view such dual descent methods as primal-dual methods that update primal variable faster than iterating dual variable. With respect to convergence analysis, stochastic approximation has been widely used to establish asymptotic convergence of several primal-dual methods [6, 22, 16, 9] by analyzing the stability of limiting gradient flow dynamics, while recent methods [28, 24, 54, 25, 26, 69, 34, 35] exploit the connection between policy gradient and mirror-descent in convex optimization to prove non-asymptotic convergence. However, non-asymptotic convergence of primal-dual methods only characterizes the average of value functions [23, 54, 25, 26, 69] or a mixture of past policies [27, 28, 24] because of the dual update that results from regulating constraint violation. For the dual descent methods [34, 35], non-asymptotic convergence is characterized in terms of last-iterate policies that are computed by approximately solving unconstrained RL problems with fixed dual variables. Therefore, designing Lagrangian-based policy gradient methods that enjoy the single-loop simplicity and the non-asymptotic convergence of policy iterates is challenging, because of the oscillation and overshoot issues of updating primal-dual variables simultaneously [29, 40]. In this work, we have established non-asymptotic and last-iterate convergence of two single-loop Lagrangian-based policy gradient methods via the regularization and optimistic gradient techniques and our analysis builds on the mirror-descent analysis for policy gradient methods [49, 98, 65, 99] while focusing on the distance of policy iterates, which is stronger than the prior art. Compared with recent efforts [40, 36] as shown in Table 1, our algorithms are simpler and our theoretical guarantees are stronger.
Gradient-based methods with last-iterate convergence for learning in games. Since the Lagrangian-based approach for constrained MDPs can be viewed as solving a constrained saddle-point problem, another line of related work is gradient-based methods for solving saddle-point (or minimax optimization) problems with last-iterate convergence. Last-iterate convergence of gradient-based methods has been established in several scenarios, e.g., linear rates of extragradient methods for strongly convex problems [100, 101], asymptotic convergence of optimistic multiplicative weights updates for convex problems [67, 68], linear rates of optimistic gradient methods for convex problems [43], and lower bound-matching rates of extragradient and optimistic gradient methods for convex problems [39]. These studies focus on convex-concave saddle-point problems except for [102, 103, 104] in which non-asymptotic last-iterate convergence is achievable for saddle-point problems with special non-convexity structure. In contrast, our constrained saddle-point problem that result from constrained MDP is non-convex in policy primal variable, which prevents direct application of these last-iterate results. A slightly twisted exception is that our constrained saddle-point problem can be reformulated to be convex in occupancy measure instead of policy, which leads the second and third methods in Table 1. To solve our Lagrangian-based constrained saddle-point problem in policy space, our first policy-based method: RPG-PD relaxes the non-convexity by adding regularization into the objective function and we provide sublinear last-iterate policy convergence guarantee using the distance analysis for policy primal-dual iterates. To improve the convergence rate, we further develop another policy-based method: OPG-PD that extends the optimistic gradient method [37] for a class of non-convex constrained saddle-point problems. This extension departs from previous extensions for zero-sum Markov games [43, 44], because lacking of uniformly optimal policies prevents learning an optimal policy from per-state bilinear games. Instead, we provide a new distance analysis of policy primal-dual iterates of an optimistic gradient method for solving a new class of constrained non-convex saddle-point problems, with linear last-iterate policy convergence.
Non-asymptotic last-iterate (or non-ergodic) convergence in constrained optimization. Reduction of constrained optimization to saddle-point problems is a classical idea to solve constrained optimization problems by developing primal-dual algorithms, e.g., primal-dual interior-point methods [105], Uzawa and Arrow-Hurwicz algorithms [31, 106, 107], and Lagrange multiplier methods [108, 45]. Inspired by the Lagrange multiplier methods, many recent studies on constrained optimization have significantly advanced primal-dual algorithms with last-iterate convergence, e.g., accelerated augmented Lagrangian method [109], accelerated universal primal-