Title: Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPs
Venue: NeurIPS 2023
Accepted: yes
Keywords: Constrained Markov decision processes

policy gradient primal-dual methods

non-convex saddle-point problem

last-iterate convergence

entropy regularization

optimistic gradient
Abstract: We study the problem of computing an optimal policy of an infinite-horizon discounted constrained Markov decision process (constrained MDP). Despite the popularity of Lagrangian-based policy search methods used in practice, the oscillation of policy iterates in these methods has not been fully understood, bringing out issues such as violation of constraints and sensitivity to hyper-parameters. To fill this gap, we employ the Lagrangian method to cast a constrained MDP into a constrained saddle-point problem in which max/min players correspond to primal/dual variables, respectively, and develop two single-time-scale policy-based primal-dual algorithms with non-asymptotic convergence of their policy iterates to an optimal constrained policy. Specifically, we first propose a regularized policy gradient primal-dual (RPG-PD) method that updates the policy using an entropy-regularized policy gradient, and the dual variable via a quadratic-regularized gradient ascent, simultaneously. We prove that the policy primal-dual iterates of RPG-PD converge to a regularized saddle point with a sublinear rate, while the policy iterates converge sublinearly to an optimal constrained policy. We further instantiate RPG-PD in large state or action spaces by including function approximation in policy parametrization, and establish similar sublinear last-iterate policy convergence. Second, we propose an optimistic policy gradient primal-dual (OPG-PD) method that employs the optimistic gradient method to update primal/dual variables, simultaneously. We prove that the policy primal-dual iterates of OPG-PD converge to a saddle point that contains an optimal constrained policy, with a linear rate. To the best of our knowledge, this work appears to be the first non-asymptotic policy last-iterate convergence result for single-time-scale algorithms in constrained MDPs. We further validate the merits and the effectiveness of our methods in computational experiments.

Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPs

Abstract

We study the problem of computing an optimal policy of an infinite-horizon discounted constrained Markov decision process (constrained MDP). Despite the popularity of Lagrangian-based policy search methods used in practice, the oscillation of policy iterates in these methods has not been fully understood, bringing out issues such as violation of constraints and sensitivity to hyper-parameters. To fill this gap, we employ the Lagrangian method to cast a constrained MDP into a constrained saddle-point problem in which max/min players correspond to primal/dual variables, respectively, and develop two single-time-scale policy-based primal-dual algorithms with non-asymptotic convergence of their policy iterates to an optimal constrained policy. Specifically, we first propose a regularized policy gradient primal-dual (RPG-PD) method that updates the policy using an entropyregularized policy gradient, and the dual variable via a quadratic-regularized gradient ascent, simultaneously. We prove that the policy primal-dual iterates of RPG-PD converge to a regularized saddle point with a sublinear rate, while the policy iterates converge sublinearly to an optimal constrained policy. We further instantiate RPG-PD in large state or action spaces by including function approximation in policy parametrization, and establish similar sublinear last-iterate policy convergence. Second, we propose an optimistic policy gradient primal-dual (OPG-PD) method that employs the optimistic gradient method to update primal/dual variables, simultaneously. We prove that the policy primal-dual iterates of OPG-PD converge to a saddle point that contains an optimal constrained policy, with a linear rate. To the best of our knowledge, this work appears to be the first non-asymptotic policy last-iterate convergence result for single-time-scale algorithms in constrained MDPs. We further validate the merits and the effectiveness of our methods in computational experiments.

1 Introduction

Constrained Markov decision process (Constrained MDP) is the classical model for constrained dynamic systems in the early stochastic control literature (e.g., [1, 2, 3, 4, 5]) and the recent constrained reinforcement learning (RL) literature (e.g., [6, 7, 8, 9, 10, 11]). It is applicable to many constrained control problems by integrating other system specifications in constraints, and admits a natural extension of constrained optimization and Lagrangian in policy space. Lagrangian-based policy search methods, especially policy-based primal-dual methods that work simultaneously with

Chen-Yu Wei ∗

Alejandro Ribeiro

primal/dual variables, lie at the heart of recent successes of constrained MDPs, e.g., navigation [12], autonomous driving [13, 14], robotics [15], and finance [16]; see [17, 18, 19, 20] for more examples.

Despite the popularity of policy-based primal-dual algorithms, classical asymptotic convergence assumes that primal-dual updates are in two-time-scale 1 type [21, 6, 22, 16, 9] (and/or work in two nested loops 2 ), and considerable global non-asymptotic convergence guarantee is measured via an average of past objective/constraint functions [23, 24, 25, 26] or a mixture of past policies [27, 28]. These results are unfavorable in constrained dynamic systems, especially safety-critical ones, due to three reasons: (i) Average and mixture performance of non-asymptotic convergence conceals oscillating (or even overshooting) objective/constraint functions of immediate policy iterates [29, 30], and oscillation-incurred constraint violation impedes a policy iterate being optimal; (ii) Asymptotic convergence is not instructive, because arbitrarily slow convergence, and oscillation and overshoot in any finite time can happen; (iii) Two-time-scale algorithms including algorithms with nested loops are sensitive to hyper-parameters and are therefore typically difficult to tune [16, 9]. Thus, we ask the following question in constrained MDPs:

Can the policy iterates of a single-time-scale policy-based primal-dual algorithm converge to an optimal constrained policy with non-asymptotic rate ?

By 'single-time-scale', we refer to the classical methods [31, 32, 33] that iterate primal/dual variables concurrently (with the same constant stepsize). Only partial answers to this question are provided in recent studies [34, 35, 36] since they either do not work in the single-time-scale scheme or they do not have non-asymptotic convergence guarantees. In this work, we provide an affirmative answer in two methodologies. First, we initiate the design and analysis of single-time-scale policy-based primal-dual algorithms via regularization, while previous works [24, 28, 34] rely on two-time-scale schemes. Second, inspired by convex minimax optimization [37, 38, 39], we propose a new optimistic policy gradient for a single-time-scale policy-based primal-dual algorithm that solves a class of non-convex minimax problem. While preparing our work, we noticed a contemporaneous work [40], which has empirically validated the effectiveness of other optimistic methods in constrained MDPs, has further inspired the pursuit of our contributions, as outlined in detail below.

Contributions . To compute an optimal policy of an infinite-horizon discounted constrained MDP, we employ the Lagrangian method to cast it into a constrained saddle-point problem in which max/min players correspond to primal/dual variables, propose two single-time-scale policy-based primal-dual algorithms, and prove global non-asymptotic convergence of their policy iterates.

Nearly dimension-free sublinear last-iterate policy convergence. We propose a regularized policy gradient primal-dual (RPG-PD) method that updates the policy using an entropy-regularized policy gradient, and the dual using a quadratic-regularized gradient ascent, simultaneously. We prove that the policy primal-dual iterates of RPG-PD converge to a regularized saddle point with a sublinear rate, and the policy iterates converge to an optimal constrained policy sublinearly.

Sublinear last-iterate policy convergence with function approximation. We generalize RPG-PD for constrained MDPs with large state/action spaces by including function approximation in policy parametrization. We prove that the policy primal-dual iterates of an inexact RPG-PD converge to a regularized saddle point with a sublinear rate, but up to a function approximation error, and the policy iterates converge sublinearly to an optimal constrained policy when the error is small.

Problem-dependent linear last-iterate policy convergence. We propose an optimistic policy gradient primal-dual (OPG-PD) method that employs the optimistic gradient method to update the primal/dual, simultaneously. We prove that the policy primal-dual iterates of OPG-PD converge to a saddle point that contains an optimal constrained policy with a problem-dependent linear rate.

While last-iterate convergence is of importance in its own right, by adding proper conservatism in the constraint, both methods can ensure no constraint violation for the last policy iterate , which perhaps is best for safety-critical tasks [10, 20]. As far as we know, this work shows the first nonasymptotic and policy last-iterate convergence for single-time-scale algorithms in the constrained MDP literature. We further exhibit the merits and the effectiveness of our methods in experiments.

Technical comparisons with prior art . Although global asymptotic last-iterate convergence has been established for single-time-scale algorithms very recently [36, 40], and value-average or policymixture non-asymptotic convergence have been established for other algorithms [23, 24, 25, 27, 28, 41, 42, 26], these studies did not investigate global non-asymptotic and last-iterate convergence for single-time-scale algorithms. Our results not only strengthen these prior guarantees, but also set up a new framework for analyzing policy-based primal-dual algorithms via the distance of primal-dual iterates to a saddle point that contains an optimal constrained policy. Our RPG-PD and OPG-PD keep the simplicity of single-time-scale primal-dual methods and output a nearly-optimal policy in the last iterate, which is more convenient than the history-average policies [28, 24] or the policies from subroutines [34, 35]. Compared with the policy-based methods [40], our OPG-PD is a projected policy gradient method that enjoys policy last-iterate convergence with linear rate. Compared with the constrained saddle-point problems [38, 39], our minimax optimization that results from constrained MDP is non-convex . Hence, our OPG-PD extends the last-iterate convergence guarantee from convex minimax optimization to a class of non-convex ones, while preserving a linear rate. Compared with the analysis in the two-player zero-sum Markov game [43, 44], there is no reduction from constrained MDPs to per-state bilinear games. Please see more details in Appendix A.

2 Preliminaries

We consider an infinite-horizon discounted constrained Markov decision process [3, 5, 8] -CMDP ( S, A, P, r, u, b, γ, ρ ) - where S and A are state/action spaces, P is a transition kernel that specifies the transition probability P ( s ′ | s, a ) from state s to next state s ′ under action a ∈ A , r , u : S × A → [0 , 1] are reward/utility functions, b is a constraint threshold, γ ∈ [0 , 1) is a discount factor, and ρ is an initial state distribution. A stationary stochastic policy π : S → ∆( A ) determines a probability distribution over the action space A based on current state, i.e., a t ∼ π ( · | s t ) at time t , where ∆( A ) is a probability simplex over A . Let Π be the set of all possible stochastic policies. A policy π ∈ Π , together with the initial state distribution ρ , induces a distribution over trajectories τ = { ( s t , a t , r t , u t ) } ∞ t =0 , where s 0 ∼ ρ , a t ∼ π ( · | s t ) , r t = r ( s t , a t ) , u t = u ( s t , a t ) and s t +1 ∼ P ( · | s t , a t ) for all t ≥ 0 .

Given a policy π , the value functions V π r , V π u : S → R associated with the reward function r or the utility function u are given by the expected sums of discounted rewards or utilities under policy π :

where the expectation E is over the randomness of the trajectory τ induced by π . Their expected values under ρ are V π r ( ρ ) := E s ∼ ρ [ V π r ( s ) ] and V π u ( ρ ) := E s ∼ ρ [ V π u ( s ) ] . It is useful to introduce the discounted state visitation distribution, d π s 0 ( s ) = (1 -γ ) ∑ ∞ t =0 γ t Pr ( s t = s | π, s 0 ) which adds up discounted probabilities of visiting s in the execution of π starting from s 0 . Denote d π ρ ( s ) := E s 0 ∼ ρ [ d π s 0 ( s ) ] and thus d π ρ ( s ) ≥ (1 -γ ) ρ ( s ) for any ρ and s . Furthermore, for the reward function r , we introduce the state-action value function Q π r : S × A → R when the agent begins with a stateaction pair ( s, a ) and follows a policy π , and the associated advantage function A π r : S × A → R ,

Similarly, we define Q π u : S × A → R and A π u : S × A → R for the utility function u .

In this work, we aim to find a policy solution π glyph[star] of a constrained policy optimization problem,

where the objective is the reward value function V π r ( ρ ) and the constraint requires that the utility value function V π u ( ρ ) is above a given threshold b . For notational simplicity we assume a single constraint, but our algorithms are readily generalizable to the problems with multiple constraints, as well as our subsequent last-iterate convergence theory. Since V π r ( ρ ) and V π u ( ρ ) ∈ [ 0 , 1 / (1 -γ ) ] , we assume b ∈ ( 0 , 1 / (1 -γ ) ] to avoid trivial cases. Let g : S × A → [ -1 , 1] be g := u -(1 -γ ) b . We, equivalently, translate the constraint V π u ( ρ ) ≥ b into V π g ( ρ ) ≥ 0 that is our focal constraint. The optimal constrained policy π glyph[star] depends on the initial state distribution ρ ; see Appendix B.1.

By the method of Lagrange multipliers [45], we dualize the constraint in (1) and present a standard Lagrangian L ( π, λ ) := V π r ( ρ ) + λV π g ( ρ ) , where π ∈ Π is the primal variable and λ ∈ [0 , ∞ ] is the dual variable or the Lagrangian multiplier. Introduction of the Lagrangian L ( π, λ ) interprets Problem (1) as a max-min problem: maximize π ∈ Π minimize λ ∈ [0 , ∞ ] L ( π, λ ) , and thus we can view the Lagrangian L ( π, λ ) as a value function with a composite function r + λg ,

However, it's defective to view Problem (2) as a standard MDP problem by fixing a dual variable λ , even the optimal one; also see [9, 46, 47, 30]. This is often referred to as the scalarization fallacy [46]; see Appendix B.2 for the detail. From the perspective of game theory, we instead view V π r + λg ( ρ ) : Π × [ 0 , ∞ ] → R as a payoff function for a two-player zero-sum game in which maxplayer is the policy π ∈ Π and min-player is the dual variable λ ∈ [ 0 , ∞ ] , and study its saddle points. To proceed, we assume feasibility for Problem (1) throughout our analysis.

Assumption 1 (Feasibility) . There exists a policy ¯ π ∈ Π and ξ > 0 such that V ¯ π g ( ρ ) ≥ ξ .

Feasibility mirrors the Slater condition in the duality analysis of constrained optimization [45]. It can be verified by solving an unconstrained MDP problem with respect to V π g ( ρ ) .

A saddle point ( π ′ , λ ′ ) satisfies V π r + λ ′ g ( ρ ) ≤ V π ′ r + λ ′ g ( ρ ) ≤ V π ′ r + λg ( ρ ) for all π ∈ Π , λ ∈ [0 , ∞ ] , or equivalently, π ′ is the max-min point, i.e., π ′ ∈ argmax π ∈ Π V π r + λ ′ g ( ρ ) and λ ′ is the min-max point, i.e., λ ′ ∈ argmin λ ∈ [0 , ∞ ] V π ′ r + λg ( ρ ) . To view Problem (2) as a saddle-point problem, we denote V π P ( ρ ) := inf λ ∈ [0 , ∞ ] V π r + λg ( ρ ) as the primal function which takes V π r ( ρ ) when V π g ( ρ ) ≥ 0 and -∞ otherwise, and V λ D ( ρ ) := max π ∈ Π V π r + λg ( ρ ) as the dual function. Let an optimal dual variable be λ glyph[star] ∈ argmin λ ∈ [0 , ∞ ] V λ D ( ρ ) . For Problem (1) under Assumption 1, strong duality holds in policy space [12, Theorem 3] and optimal dual variables are bounded [48, Lemma 3].

Lemma 1 (Strong duality/Saddle point existence and boundedness) . Let Assumption 1 hold. Then, (i) strong duality holds for Problem (1) , i.e., V π glyph[star] P ( ρ ) = V λ glyph[star] D ( ρ ) ; (ii) optimal dual variables are bounded, i.e., λ glyph[star] ∈ [ 0 , ( V π glyph[star] r -V ¯ π r ) /ξ ] .

Let the set of max-min points be Π glyph[star] := argmax π ∈ Π min λ ∈ [0 , ∞ ] V π r + λg ( ρ ) and the set of minmax points be Λ glyph[star] := argmin λ ∈ [0 , ∞ ] max π ∈ Π V π r + λg ( ρ ) . From Lemma 1 (ii), Λ glyph[star] is contained in a bounded interval Λ := [ 0 , 1 / ((1 -γ ) ξ ) ] . Lemma 1 (i) shows that any pair ( π glyph[star] , λ glyph[star] ) ∈ Π glyph[star] × Λ glyph[star] solves the following constrained saddle-point problem,

Any saddle points associated with the set Λ glyph[star] are captured by Problem (3) due to the invariance of saddle points, and searching for any pair ( π glyph[star] , λ glyph[star] ) ∈ Π glyph[star] × Λ glyph[star] is sufficient by the interchangeability of saddle points; see Lemmas 8-9 in Appendix B.3 for the properties of saddle points. Thus, we view the policy (primal) as max-player and the dual as min-player in a zero-sum game.

Three structural properties from constrained MDPs distinguish Problem (3) from recent last-iterate convergence for learning in zero-sum games (e.g., [37, 38, 43, 39, 44]): (i) Two players are asymmetric . One plays a stochastic policy that affects the transition dynamics and the other selects an action in a continuous interval that only changes the payoff; (ii) Problem (3) is a non-convex game, because of the non-concavity of the payoff V π r + λg ( ρ ) in policy π (e.g., [49, Lemma 1]); (iii) A saddle-point policy for Problem (3) cannot be uniformly max-min optimal, i.e., being optimal across all states , since an optimal policy often depends on the initial state distribution ρ in a constrained MDP; see Appendix B.1. Hence, known last-iterate results in zero-sum convex games or symmetric Markov games that admit uniformly optimal policies can't be applied and new techniques are required to address this non-standard saddle-point problem, which warrants our contributions in this work.

Warm-up: Indirect policy search in occupancy-measure space . Finding a saddle point of a nonconvex game is hard in general [50]. Nevertheless, Problem (1) can be rewritten as a linear program regarding the occupancy measure [5], which permits indirectly searching for a saddle point of a bilinear Lagrangian [51, 42, 36]. These asymptotic or average-iterate convergence results can be easily strengthened by applying last-iterate convergence results for bilinear games (e.g., [38, 39]) to be

non-asymptotic and last-iterate. By doing so, we state an optimistic primal-dual (OPD) method (18) in Appendix B.4. Compared with a contemporaneous work [40], OPD is free of projection to an occupancy measure set, and enjoys strengthened linear convergence.

OPD is an indirect policy search method that iterates using occupancy measure-based gradients, not policy-based gradients. It is crucial to develop direct policy search methods that are widely-used in RL, which is our focus. We propose two such methods in Section 3 and Section 4, respectively.

3 Policy Last-Iterate Convergence: Regularized Method

Towards achieving policy last-iterate convergence, a practical strategy is using regularization [52] to 'convexify' Problem (3). We present a regularized method - Regularized Policy Gradient PrimalDual (RPG-PD) - that converges to a saddle point that yields an optimal constrained policy.

3.1 Regularized policy gradient primal-dual method

We introduce a regularized Lagrangian L τ ( π, λ ) := V π r + λg ( ρ ) + τ ( H ( π ) + 1 2 λ 2 ) by adding a regularization term H ( π ) + 1 2 λ 2 onto the original Lagrangian V π r + λg ( ρ ) , where τ is a regularization parameter, and H ( π ) := E [ ∑ ∞ t =0 -γ t log π ( a t | s t ) ] is an entropy-like regularization term [52]. We now introduce a regularized constrained saddle-point problem,

Problem (4) is well-defined, since there exists a saddle point for L τ ( π, λ ) over Π × Λ and it is unique; see Appendix C.1 for proof. A saddle point ( π glyph[star] τ , λ glyph[star] τ ) , i.e., π glyph[star] τ = argmax π ∈ Π min λ ∈ Λ L τ ( π, λ ) and λ glyph[star] τ = argmin λ ∈ Λ max π ∈ Π L τ ( π, λ ) , satisfies a sandwich-like property,

that states that ( π glyph[star] τ , λ glyph[star] τ ) is a saddle point of the original Lagrangian V π r + λg ( ρ ) , up to two τ -terms. We thus propose a regularized policy gradient primal-dual (RPG-PD) method by maintaining a sequence for policy and dual variables each: { π t } t ≥ 0 for the policy-player, and { λ t } t ≥ 0 for the dual-player,

where the gradient direction Q π t r + λ t g + τψ t ( s, a ) is the state-action value function under a composite function r + λ t g + τψ t in which ψ t ( s, a ) := -log π t ( a | s ) , KL ( p, p ′ ) := ∑ a p a log p a p ′ a is the Kullback-Leibler (KL) divergence, ˆ ∆( A ) := { π ( · | s ) ∈ ∆( A ) | π ( a | s ) ≥ glyph[epsilon1] 0 | A | , a ∈ A } is a restricted probability simplex set with parameter glyph[epsilon1] 0 ∈ (0 , 1) , η is the stepsize, and ( π 0 ( · | s ) , λ 0 ) ∈ ˆ ∆( A ) × Λ is an initial point. Projecting the policy iterate to the simplex set ˆ ∆( A ) ensures the boundedness of the gradient. Primal update (6a) works as the classical mirror descent with KL divergence [53] with a projection onto the set ˆ ∆( A ) . Dual update (6b) performs typical projected gradient descent. Hence, RPG-PD is a single-time-scale method. RPG-PD simplifies the two-time-scale method [34] to be single-time-scale and generalize the single-time-scale methods [23, 54] with regularization.

3.2 Policy last-iterate convergence

In Theorem 2, we show that the primal-dual iterates of RPG-PD converge in the last iterate; see Appendix C.2 for proof. We characterize the convergence via a distance metric Φ t = KL t ( ρ ) + 1 2 ( λ glyph[star] τ -λ t ) 2 , where KL t ( ρ ) := (1 / (1 -γ )) ∑ s d π glyph[star] τ ρ ( s ) KL t ( s ) and KL t ( s ) := KL ( π glyph[star] τ ( · | s ) , π t ( · | s )) . Theorem 2 (Linear convergence of RPG-PD) . Let Assumption 1 hold. If we set the stepsize η ≤ 1 /C τ,ξ,glyph[epsilon1] 0 , then the primal-dual iterates of RPG-PD (6) satisfy

Theorem 2 states that the primal-dual iterates of RPG-PD converge to a neighborhood of ( π glyph[star] τ , λ glyph[star] τ ) in a linear rate. The size of neighborhood scales with η/τ + ητ (1+log 2 glyph[epsilon1] 0 ) and the convergence rate is ητ . Even if glyph[epsilon1] 0 is very small, the log glyph[epsilon1] 0 -term is almost a constant. If we take η = min( glyph[epsilon1]τ, 1 /C τ,ξ,glyph[epsilon1] 0 ) and glyph[epsilon1] 0 = glyph[epsilon1] , then after O (1 /glyph[epsilon1] ) iterations the RPG-PD's primal-dual iterate ( π t , λ t ) is glyph[epsilon1] -close to ( π glyph[star] τ , λ glyph[star] τ ) , i.e., Φ t = O ( glyph[epsilon1] ) for any t ≥ (1 / ( glyph[epsilon1]τ 2 )) log(1 /glyph[epsilon1] ) . For small τ , we can translate the policy convergence for the value functions in Corollary 3; see Appendix C.3 for proof.

Corollary 3 (Nearly-optimal constrained policy) . Let Assumption 1 hold. For small glyph[epsilon1] > 0 , if we take η = Θ( glyph[epsilon1] 4 ) , τ = Θ( glyph[epsilon1] 2 ) , and glyph[epsilon1] 0 = glyph[epsilon1] , then the policy iterates of RPG-PD (6) satisfy

where Ω( · ) only has some problem-dependent constant.

Corollary 3 states that the last policy iterate of RPG-PD is an glyph[epsilon1] -optimal policy for Problem (1) after Ω(1 /glyph[epsilon1] 6 ) iterations. Compared with the single-time-scale methods [23, 54], RPG-PD improves the convergence from average-value (or regret-type) to last policy iterate . Not just being theoretically stronger, the last-iterate convergence is more appealing since it captures the stability of trajectories of an algorithm [29, 40]. Compared with the two-time-scale methods [28, 24, 34, 35], RPG-PD is free of nested loops, and uniform ergodicity and exploratory initial state distribution. We notice that the dual methods [28, 24] yield history-average policies and the dual methods [34, 35] return policies from a subroutine. In contrast, RPG-PD outputs a nearly-optimal policy in the last iterate, the firstof-its-kind in the constrained MDP literature, albeit the rate is worse than the average ones [23, 54].

To get zero constraint violation, i.e., V π t g ( ρ ) ≥ 0 at some t , it is straightforward to employ a conservative constraint V π g ′ ( ρ ) ≥ 0 with g ′ := g -(1 -γ ) δ for some δ > 0 . When glyph[epsilon1] is small enough, there always exists some δ such that the policy iterates of RPG-PD (6) satisfy V π glyph[star] r ( ρ ) -V π t r ( ρ ) ≤ glyph[epsilon1] and V π t g ( ρ ) ≥ 0 for large t ; see Appendix C.4 for proof. Our zero constraint violation ensures the last policy iterate of RPG-PD to satisfy the constraint, which is not the zero average constraint violation in the episodic setting [55, 56]. Compared with the zero constraint violation of a policy induced by an average of past occupancy measures [42], RPG-PD's zero constraint violation directly settles the policy iterates down, which appears to be the first policy-based zero constraint violation.

Last but not least, the iteration complexity of RPG-PD is nearly-free of the MDP dimension, except for an log | A | -term, which inherits the dimension-free property of the NPG methods [49, 57, 23]. Hence, it is ready to view RPG-PD as a variant of NPG methods and generalize RPG-PD to constrained MDPs with large state spaces in the function approximation setting.

3.3 Linear function approximation case

To deal with large state spaces, we use a parametrized policy π θ with θ ∈ R d for RPG-PD (6) without restricting ∆( A ) , where d is much smaller than the size of state/action spaces. To introduce function approximation, we begin with a tabular softmax policy π θ ( a | s ) = exp( θ s,a ) ∑ a ′ exp( θ s,a ′ ) for all ( s, a ) ∈ S × A and θ ∈ R | S || A | . Connecting NPG to mirror descent [49, 58], we express RPG-PD (6) as a NPG method with the following update; see Appendix C.5 for proof,

where F ρ ( θ t ) † · ∇ θ L τ ( π θ t , λ t ) is a NPG direction, and F ρ ( θ ) is the Fisher information matrix for a policy π θ , i.e., F ρ ( θ ) := E s ∼ d π θ ρ E a ∼ π θ ( · | s ) [ ∇ θ log π θ ( a | s )( ∇ θ log π θ ( a | s )) glyph[latticetop] ] . A useful property of (7a) is that NPG can be related to a linear regression. For any policy π θ and a state-action value function Q π θ , the associated compatible function approximation error is E Q ( w,θ, ν ) := E ( s,a ) ∼ ν [ ( w glyph[latticetop] ∇ log π θ ( a | s ) -Q π θ ( s, a )) 2 ] , where ν ( s, a ) = d π θ ρ ( s ) π θ ( a | s ) is a state-action distribution. It is known that (7a) is equivalent to θ t +1 = θ t + ηw glyph[star] t , where w glyph[star] t ∈ argmin w ∈ R d E Q ( w,θ t , ν t ) in which Q π θ t ( s, a ) = Q π θ t r + λ t g + τψ t ( s, a ) and ν t ( s, a ) = d π θ t ρ ( s ) π θ t ( a | s ) (e.g., [59, Lemma 1]). In practice, only an approximate minimizer w glyph[star] t is available if a sample-based algorithm is used, e.g., w t ≈ argmin ‖ w ‖ ≤ W E Q ( w,θ t , ν t ) , where W > 0 .

A useful generalization of the softmax policy to large state spaces is the log-linear policy based on linear function approximation. Let φ s,a ∈ R d be a feature map with ‖ φ s,a ‖ ≤ 1 for each state-action pair ( s, a ) . A log-linear policy π θ : S → ∆( A ) is parametrized by a parameter θ ∈ R d ,

which takes the tabular softmax policy as a special case, i.e., φ s,a is an indicator function. We notice that ∇ θ log π θ ( a | s ) = φ s,a -E a ′ ∼ π θ ( · | s ) [ φ s,a ′ ] . Since the log-linear policy is invariant to any action-independent term, it is convenient to replace ∇ θ log π θ ( a | s ) by φ s,a and we introduce a simplified compatible function approximation error, E Q ( w,θ, ν ) := E ( s,a ) ∼ ν [ ( φ glyph[latticetop] s,a w -Q π θ ( s, a )) 2 ] . Thus, we can take w glyph[star] t ∈ argmin w ∈ R d E Q ( w,θ t , ν t ) in which Q π θ t ( s, a ) = Q π θ t r + λ t g + τψ t ( s, a ) and ν t ( s, a ) = d π θ t ρ ( s ) π θ t ( a | s ) to update θ t +1 = θ t + ηw glyph[star] t . Using the log-linear policy class, we replace the primal gradient direction of RPG-PD (6) by its linear function approximation φ glyph[latticetop] s,a w glyph[star] t ,

which, together with Dual update (6b), leads to a general version of RPG-PD. The set ˆ ∆( A ) ensures bounded true gradient direction Q π θ t r + λ t g + τψ t ( s, a ) . When there is no function approximation error, (8) reduces to Primal update (6a). In practice, we can only compute w glyph[star] t approximately via

which leads to an inexact RPG-PD: Primal update (8) in which w glyph[star] t is replaced by w t and Dual update (6b), where d t,ν := (1 -γ ) E ( s 0 ,a 0 ) ∼ ν ∑ ∞ t =0 γ t Pr ( s t = s, a t = a | π θ t , s 0 , a 0 ) is a state-action distribution starting from any distribution ν . Noticeably, d t,ν is more general than ν t . To control the function approximation error, we divide E Q ( w t , θ t , d t,ν ) into a statistical error E Q ( w t , θ t , d t,ν ) - E Q ( w glyph[star] t , θ t , d t,ν ) that is similar to the excess risk in supervised learning, and an approximation error E Q ( w glyph[star] t , θ t , d t,ν ) that captures how well a linear function ( w glyph[star] t ) glyph[latticetop] φ s,a approximates the true value function under d t,ν . If the on-policy distribution d t,ν in E Q ( w glyph[star] t , θ t , d t,ν ) is replaced by ν glyph[star] ( s, a ) = d π glyph[star] τ ρ ( s ) Unif A ( a ) , we define a transfer error E Q ( w glyph[star] t , θ t , ν glyph[star] ) . Let the covariance matrix of φ s,a in any state-action distribution ν be Σ ν := E ( s,a ) ∼ ν [ φ s,a φ glyph[latticetop] s,a ] , and the relative condition number between Σ ν and Σ ν glyph[star] be κ ν := max w ∈ R d w glyph[latticetop] Σ ν glyph[star] w w glyph[latticetop] Σ ν w .

We make an assumption on the statistical error, the transfer error, and the relative condition number. Assumption 2. (i) There exist glyph[epsilon1] stat , glyph[epsilon1] bias > 0 such that E [ E Q ( w t , θ t , d t,ν ) -E Q ( w glyph[star] t , θ t , d t,ν ) ] ≤ glyph[epsilon1] stat and E [ E Q ( w glyph[star] t , θ t , ν glyph[star] ) ] ≤ glyph[epsilon1] bias ; (ii) The relative condition number is finite, i.e., κ ν < ∞ .

We assess the convergence of inexact RPG-PD via the distance metric E [ Φ t ] := E [ KL t ( ρ ) ] + 1 2 E [ ( λ glyph[star] τ -λ t ) 2 ] , where the expectation E is over the randomness of computing w t via a samplebased algorithm. We state the convergence in Theorem 4 and delay its proof to Appendix C.6.

Theorem 4 (Linear convergence of inexact RPG-PD) . Let Assumptions 1-2 hold. If we take the stepsize η ≤ 1 /C W , then the primal-dual iterates of inexact RPG-PD satisfy

Theorem 4 states that the primal-dual iterates of inexact RPG-PD converge to a neighborhood of ( π glyph[star] τ , λ glyph[star] τ ) in a linear rate. The convergence rate is ητ and the size of neighborhood scales with a sum of an η/τ -term and an 1 /τ -term that amplifies the effect of function approximation ( glyph[epsilon1] stat , glyph[epsilon1] bias ) . We note that, Theorem 4 does not require the strong duality in the parametrized policy class, and the function approximation error includes the duality gap caused by the inexpensiveness of function class and the policy representation error caused by the restricted policy set ˆ ∆( A ) . When there is no function approximation error, Theorem 4 has a similar result as Theorem 2. It is important to control

( glyph[epsilon1] stat , glyph[epsilon1] bias ) to be small: (i) Application of stochastic gradient methods to the linear regression leads to glyph[epsilon1] stat = O (1 / √ K ) or O (1 /K ) , where K is the number of gradient steps, and thus, it is easy to control glyph[epsilon1] stat; (ii) When glyph[epsilon1] 0 is very small, the parametrized policy iterate can be contained in ˆ ∆( A ) , and thus glyph[epsilon1] bias becomes zero in some cases, e.g., tabular softmax case [49] or low-rank MDPs [60, 61] with d ≥ | A | ; it can be made very small if the function class is rich, e.g., wide neural networks [62]. When the errors are small, it is ready to establish Corollary 5; see Appendix C.7 for proof.

Corollary 5 (Nearly-optimal constrained policy) . Let Assumptions 1-2 hold and glyph[epsilon1] stat , glyph[epsilon1] bias = O ( glyph[epsilon1] 8 ) for small glyph[epsilon1] , glyph[epsilon1] 0 > 0 . If we take the stepsize η = Θ( glyph[epsilon1] 4 ) and τ = Θ( glyph[epsilon1] 2 ) , then the policy iterates of inexact RPG-PD satisfy

where Ω( · ) only has some problem-dependent constant.

Corollary 5 states that the iteration complexity in Corollary 3 holds in the function approximation case. When glyph[epsilon1] is small enough, we can design a conservative constraint such that the policy iterates of inexact RPG-PD satisfy V π glyph[star] r ( ρ ) -V π θ t r ( ρ ) ≤ glyph[epsilon1] and V π θ t g ( ρ ) ≥ 0 for large t ; see Appendix C.8 for proof. Compared with the zero average constraint violation [63], this appears to be the first policybased zero constraint violation result in the function approximation setting. Moreover, we extend inexact RPG-PD to be a sample-based algorithm and provide its sample complexity in Appendix C.9.

4 Policy Last-Iterate Convergence: Optimistic Method

Having established sublinear policy last-iterate convergence via regularization, we turn to the optimistic gradient method [37] for a faster rate. We propose an optimistic method - Optimistic Policy Gradient Primal-Dual (OPG-PD) - that converges an optimal constrained policy at a linear rate.

4.1 Optimistic policy gradient primal-dual method

We propose an optimistic policy gradient primal-dual (OPG-PD) method by maintaining two sequences for policy and dual variables each: { π t } t ≥ 1 and { ˆ π t } t ≥ 1 for the policy-player, and { λ t } t ≥ 1 and { ˆ λ t } t ≥ 1 for the dual-player,

where η is the stepsize and (ˆ π 0 , ˆ λ 0 ) = ( π 0 , λ 0 ) ∈ Π × Λ is the initial point. OPG-PD concurrently works with two primal iterates and two dual iterates, and each two are updated consecutively to stabilize the algorithm dynamics. The 'optimistic' in optimization, e.g., [64] views (ˆ π t +1 , ˆ λ t +1 ) -update as a real policy gradient step and ( π t , λ t ) -update as a prediction step that generates an intermediate iterate ( π t , λ t ) . Not policy gradient at (ˆ π t , ˆ λ t ) , the real step uses a policy gradient at ( π t , λ t ) from prediction, exhibiting the optimism towards the prediction. Specifically, Primal update (9a) works as the projected Q -ascent [58, 65], an application of the classical mirror descent with Euclidean distance [53], where the projection onto a probability simplex can be solved efficiently [66]. Dual update (9b) performs standard projected gradient descent. We note that OPG-PD is different from the one-step multiplicative weights update in the policy-based ReLOAD [40].

When there is no MDP transition dynamics, i.e., constrained bandit [40], last-iterate convergence of OPG-PD to a saddle point is known in the minimax optimization [67, 68, 38, 39], because Problem (3) reduces to a bilinear zero-sum game in this case. However, it is prohibitive to apply such

bilinear game results to the Lagrangian V π r + λg ( s ) in every state s , as has been done for zero-sum Markov games [43, 44]. The main reason for this is that there may not exist an optimal constrained policy that is uniformly optimal across all states; see Appendix B.2.

4.2 Policy last-iterate convergence

We define the distribution mismatch coefficient over ρ as κ ρ := sup π ∈ Π ∥ ∥ d π ρ /ρ ∥ ∥ ∞ , which is the maximum distribution mismatch of policy π relative to ρ , where d π ρ /ρ is divided per state. Hence, ∥ ∥ d π ρ /d π glyph[star] ρ ∥ ∥ ∞ ≤ κ ρ / (1 -γ ) for any policy π ∈ Π and κ ρ ≤ 1 /ρ min where ρ min := min s ρ ( s ) . The projection operator P X on a closed convex set X defines P X ( x ) := argmin x ′ ∈ X ‖ x ′ -x ‖ .

We state the policy last-iterate convergence of OPG-PD (9) in Theorem 6.

Theorem 6 (Linear convergence of OPG-PD) . Let Assumption 1 hold. Assume the optimal state visitation distribution be unique, i.e., d π ρ = d π glyph[star] ρ for any π ∈ Π glyph[star] , and ρ min > 0 . If we set the stepsize η ≤ min ( 1 / (4 √ ι ) , (1 -γ ) 3 / (4 | A | ) , (1 -γ ) 3 / (2 κ ρ ) ) , where ι > 0 is defined in Appendix D.1, then the primal-dual iterates of OPG-PD (9) satisfy

where C := min(7(1 -γ ) / 8 , 7 η 2 (1 -γ ) 2 ( C ρ,ξ ) 2 ρ min / (6 κ ρ,γ ) 2 ) in which C ρ,ξ and κ ρ,γ are given by C ρ,ξ := cρ min / (2 √ | S || A | ) / (1 + 1 / ((1 -γ ) ξ )) , κ ρ,γ := max( κ ρ / (1 -γ ) , 1) , and c > 0 is a problem-dependent constant from Lemma 26.

Theorem 6 states that the primal-dual iterates of OPG-PD converge to Π glyph[star] × Λ glyph[star] in a linear rate, or putting it differently, (9) is contracting to a set of optimal primal/dual variables. The rate is governed by a problem-dependent constant. Proof of Theorem 6 is provided in Appendix D. A key to our analysis is to bridge the per-state policy gradient update and the policy improvement for V π r + λg ( ρ ) that is non-convex in policy π , which departs from the convex last-iterate analysis [38, 39]. In addition, we address two technical difficulties. First, the lack of uniformly optimal policies prevents learning an optimal policy from per-state bilinear games in zero-sum Markov games [43, 44]. Instead, we characterize the proximity of primal-dual iterates to a saddle point supported by an optimal state visitation distribution d π glyph[star] ρ . Second, Problem (3) is an asymmetric game since one plays a stochastic policy over a finite set of discrete actions and controls the transition dynamics, but the other selects an action in a continuous interval. Thus, our dual-player analysis handles the long-term effect of the policy-player, which did not appear in the symmetric game [43, 44].

A direct corollary of Theorem 6 is stated below; see Appendix D.3 for the proof.

Corollary 7 (Nearly-optimal constrained policy) . Let Assumption 1 hold and the optimal state visitation distribution be unique, i.e., d π ρ = d π glyph[star] ρ for any π ∈ Π glyph[star] , and ρ min > 0 . If we use the stepsize η from Theorem 6, then the policy iterates of OPG-PD (9) satisfy

where Ω( · ) only has some problem-dependent constant.

Corollary 7 states that the last policy iterate of OPG-PD is an glyph[epsilon1] -optimal policy for Problem (1) after an almost constant number of iterations, which improves the sublinear rate in Corollary 3. OPGPD also improves the average-value convergence of the single-time-scale methods [23, 54] and the two-time-scale methods [28, 24, 34, 35], and matches the last-iterate convergence rate of the twotime-scale methods [34, 35]. We stress that our last-iterate convergence indicates the stability of whole primal-dual iterates, which is not the last policy iterate from a subroutine [34, 35]. Again, when glyph[epsilon1] is small, we can design a conservative constraint such that the policy iterates of OPG-PD satisfy V π glyph[star] r ( ρ ) -V ˆ π t r ( ρ ) ≤ glyph[epsilon1] and V ˆ π t g ( ρ ) ≥ 0 for large t ; see Appendix D.4 for the proof.

5 Computational Experiment

We validate the effectiveness of RPG-PD (6) and OPG-PD (9) by comparing them with typical primal-dual methods in Figure 1. A few observations are in order. The initial oscillation of RPG-PD

( - -) is damped, and OPG-PD ( -) is almost free of oscillation as PID Lagrangian ( ···· ). However, oscillation of NPG-PD ( -· -) causes its last-iterate policy violating the constraint V π g ( ρ ) ≥ 0 . OPGPD ( -) reaches the maximum reward value in four methods and RPG-PD ( - -) converges to a slightly smaller value due to regularization, while both meet the constraint at the end. However, PID Lagrangian ( ···· ) is highly sub-optimal. Hence, our methods OPG-PD and RPG-PD can overcome oscillation and approach a nearly-optimal constrained policy in the last-iterate fashion.

Reward value

Utility value

Iteration

Iteration

8.0

7.5

7.0

6.5

6.0

5.5

5.0

0

200

400

600

800

1000

Q

200

400

600

800

1000

We showcase the linear convergence of OPG-PD (9) with three constant stepsizes in Figure 2. Three policy optimality gaps decrease linearly in the logarithmic scale plot, which verifies the linear lastiterate convergence of OPG-PD's policy iterates in Theorem 6. Noticeably, there is no oscillation behavior in OPG-PD's policy iterates, which perhaps is best for learning constraints [29, 10]. We also see that a large stepsize η = 0 . 2 improves the convergence, which is reflected by our rate.

Policy optimality gap

Iteration

101

100

10-1

0

250

500

750

1000

1250

1500

1750

2000

Please see Appendix E for more details of this experiment, more baselines, and sensitivity analysis.

6 Concluding Remarks

We have presented two single-time-scale policy-based primal-dual methods for finding an optimal policy of a constrained MDP, with global non-asymptotic and last-iterate policy convergence guarantees. Our first regularized method enjoys a nearly dimension-free sublinear rate, while our second optimistic method possesses a linear rate that is problem-dependent. Our work stimulates a number of compelling future directions: (i) Our problem setting circumvents the exploration difficulty, which leaves online exploration open; (ii) Our convergence rates are not as sharp as solving convexconcave minimax optimization problems, regarding the order or instance-related constant; (iii) Lastiterate convergence is under-examined in constrained MDPs with other constraints, and unexplored for other gradient methods.

Acknowledgments and Disclosure of Funding

Ding and A. Ribeiro were supported by THEORINET Simons-NSF MoDL, and DCIST CRA. K. Zhang acknowledges the support from Simons-Berkeley Research Fellowship and Northrop Grumman - Maryland Seed Grant Program.

References

Frederick J Beutler and Keith W Ross. Optimal policies for controlled Markov chains with a constraint. Journal of Mathematical Analysis and Applications , 112(1):236-252, 1985.

Keith W Ross. Randomized and past-dependent policies for Markov decision processes with multiple constraints. Operations Research , 37(3):474-477, 1989.

Alexey Piunovskiy. Control of random sequences in problems with constraints. Theory of Probability & Its Applications , 38(4):751-762, 1994.

Eugene A Feinberg and Adam Shwartz. Constrained discounted dynamic programming. Mathematics of Operations Research , 21(4):922-945, 1996.

Eitan Altman. Constrained Markov decision processes , volume 7. CRC press, 1999.

Vivek S Borkar. An actor-critic algorithm for constrained Markov decision processes. Systems & Control Letters , 54(3):207-213, 2005.

Naoki Abe, Prem Melville, Cezar Pendus, Chandan K Reddy, David L Jensen, Vince P Thomas, James J Bennett, Gary F Anderson, Brent R Cooley, Melissa Kowalczyk, et al. Optimizing debt collections using constrained reinforcement learning. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 75-84, 2010.

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of the International Conference on Machine Learning , pages 22-31, 2017.

Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. In Proceedings of the International Conference on Learning Representations , 2019.

Gabriel Dulac-Arnold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning , 110(9):2419-2468, 2021.

Amol Mandhane, Anton Zhernov, Maribeth Rauh, Chenjie Gu, Miaosen Wang, Flora Xue, Wendy Shang, Derek Pang, Rene Claus, Ching-Han Chiang, et al. Muzero with selfcompetition for rate control in VP9 video compression. arXiv preprint arXiv:2202.06626 , 2022.

Santiago Paternain, Miguel Calvo-Fullana, Luiz FO Chamon, and Alejandro Ribeiro. Safe policies for reinforcement learning via primal-dual methods. IEEE Transactions on Automatic Control , 2022.

Xiangkun He, Haohan Yang, Zhongxu Hu, and Chen Lv. Robust lane change decision making for autonomous vehicles: An observation adversarial reinforcement learning approach. IEEE Transactions on Intelligent Vehicles , 2022.

Ziqing Gu, Lingping Gao, Haitong Ma, Shengbo Eben Li, Sifa Zheng, Wei Jing, and Junbo Chen. Safe-state enhancement method for autonomous driving via direct hierarchical reinforcement learning. IEEE Transactions on Intelligent Transportation Systems , 2023.

Dan A Calian, Daniel J Mankowitz, Tom Zahavy, Zhongwen Xu, Junhyuk Oh, Nir Levine, and Timothy Mann. Balancing constraints and rewards with meta-gradient D4PG. In Proceedings of the International Conference on Learning Representations , 2021.

Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Riskconstrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research , 18(1):6070-6120, 2017.

Javier Garcıa and Fernando Fern´ andez. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research , 16(1):1437-1480, 2015.

Frits De Nijs, Erwin Walraven, Mathijs De Weerdt, and Matthijs Spaan. Constrained multiagent Markov decision processes: A taxonomy of problems and algorithms. Journal of Artificial Intelligence Research , 70:955-1001, 2021.

Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems , 5:411444, 2022.

Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205.10330 , 2022.

F Vazquez Abad, Vikram Krishnamurthy, Katerine Martin, and Irina Baltcheva. Self learning control of constrained Markov chains - a gradient approach. In Proceedings of the 41st IEEE Conference on Decision and Control , volume 2, pages 1940-1945, 2002.

Shalabh Bhatnagar and K Lakshmanan. An online actor-critic algorithm with function approximation for constrained Markov decision processes. Journal of Optimization Theory and Applications , 153:688-708, 2012.

Dongsheng Ding, Kaiqing Zhang, Tamer Bas ¸ar, and Mihailo R Jovanovi´ c. Natural policy gradient primal-dual method for constrained Markov decision processes. Proceedings of the Advances in Neural Information Processing Systems , 33:8378-8390, 2020.

Tao Liu, Ruida Zhou, Dileep Kalathil, PR Kumar, and Chao Tian. Policy optimization for constrained MDPs with provable fast global convergence. arXiv preprint arXiv:2111.00552 , 2021.

Arushi Jain, Sharan Vaswani, Reza Babanezhad, Csaba Szepesvari, and Doina Precup. Towards painless policy optimization for constrained MDPs. In Proceedings of the Uncertainty in Artificial Intelligence , pages 895-905, 2022.

Sihan Zeng, Thinh T Doan, and Justin Romberg. Finite-time complexity of online primaldual natural actor-critic algorithm for constrained Markov decision processes. In Proceedings of the IEEE 61st Conference on Decision and Control , pages 4028-4033, 2022.

Yi Chen, Jing Dong, and Zhaoran Wang. A primal-dual approach to constrained Markov decision processes. arXiv preprint arXiv:2101.10895 , 2021.

Tianjiao Li, Ziwei Guan, Shaofeng Zou, Tengyu Xu, Yingbin Liang, and Guanghui Lan. Faster algorithm and sharper analysis for constrained Markov decision process. arXiv preprint arXiv:2110.10351 , 2021.

Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by PID Lagrangian methods. In Proceedings of the International Conference on Machine Learning , pages 9133-9143, 2020.

Miguel Calvo-Fullana, Santiago Paternain, Luiz FO Chamon, and Alejandro Ribeiro. State augmented constrained reinforcement learning: Overcoming the limitations of learning with rewards. IEEE Transactions on Automatic Control , 2023.

Kenneth J. Arrow, Leonid Hurwicz, and Hirofumi Uzawa. Studies in linear and non-linear programming . Stanford University Press, 1958.

Galina M Korpelevich. The extragradient method for finding saddle points and other problems. Matecon , 12:747-756, 1976.

Angelia Nedi´ c and Asuman Ozdaglar. Subgradient methods for saddle-point problems. Journal of Optimization Theory and Applications , 142:205-228, 2009.

Donghao Ying, Yuhao Ding, and Javad Lavaei. A dual approach to constrained Markov decision processes with entropy regularization. In Proceedings of the International Conference on Artificial Intelligence and Statistics , pages 1887-1909, 2022.

Egor Gladin, Maksim Lavrik-Karmazin, Karina Zainullina, Varvara Rudenko, Alexander Gasnikov, and Martin Takac. Algorithm for constrained Markov decision process with linear convergence. In Proceedings of the International Conference on Artificial Intelligence and Statistics , pages 11506-11533. PMLR, 2023.

Tianqi Zheng, Pengcheng You, and Enrique Mallada. Constrained reinforcement learning via dissipative saddle flow dynamics. arXiv preprint arXiv:2212.01505 , 2022.

Yu-Guan Hsieh, Franck Iutzeler, J´ erˆ ome Malick, and Panayotis Mertikopoulos. On the convergence of single-call stochastic extra-gradient methods. Proceedings of the Advances in Neural Information Processing Systems , 32, 2019.

Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Linear last-iterate convergence in constrained saddle-point optimization. In Proceedings of the International Conference on Learning Representations , 2020.

Yang Cai, Argyris Oikonomou, and Weiqiang Zheng. Tight last-iterate convergence of the extragradient and the optimistic gradient descent-ascent algorithm for constrained monotone variational inequalities. arXiv preprint arXiv:2204.09228 , 2022.

Ted Moskovitz, Brendan O'Donoghue, Vivek Veeriah, Sebastian Flennerhag, Satinder Singh, and Tom Zahavy. ReLOAD: Reinforcement learning with optimistic ascent-descent for lastiterate convergence in constrained MDPs. arXiv preprint arXiv:2302.01275 , 2023.

Sharan Vaswani, Lin Yang, and Csaba Szepesv´ ari. Near-optimal sample complexity bounds for constrained MDPs. In Proceedings of the Advances in Neural Information Processing Systems , volume 35, pages 3110-3122, 2022.

Qinbo Bai, Amrit Singh Bedi, Mridul Agarwal, Alec Koppel, and Vaneet Aggarwal. Achieving zero constraint violation for constrained reinforcement learning via primal-dual approach. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 36823689, 2022.

Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Last-iterate convergence of decentralized optimistic gradient descent/ascent in infinite-horizon competitive Markov games. In Proceedings of the Conference on learning theory , pages 4259-4299, 2021.

Zhuoqing Song, Jason D Lee, and Zhuoran Yang. Can we find Nash equilibria at a linear rate in Markov games? In Proceedings of the International Conference on Learning Representations , 2023.

Dimitri P Bertsekas. Constrained optimization and Lagrange multiplier methods . Academic Press, 2014.

Csaba Szepesv´ ari. Constrained MDPs and the reward hypothesis. http://readingsml. blogspot.com/2020/03/constrained-mdps-and-reward-hypothesis.html , 2020. Access on January 21, 2023.

Tom Zahavy, Brendan O'Donoghue, Guillaume Desjardins, and Satinder Singh. Reward is enough for convex MDPs. Proceedings of the Advances in Neural Information Processing Systems , 34:25746-25759, 2021.

Dongsheng Ding, Kaiqing Zhang, Jiali Duan, Tamer Bas ¸ar, and Mihailo R Jovanovi´ c. Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPs. arXiv preprint arXiv:2206.02346 , 2022.

Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. The Journal of Machine Learning Research , 22(1):4431-4506, 2021.

Constantinos Daskalakis, Stratis Skoulakis, and Manolis Zampetakis. The complexity of constrained min-max optimization. In Proceedings of the Annual ACM SIGACT Symposium on Theory of Computing , pages 1466-1478, 2021.

Yujia Jin and Aaron Sidford. Efficiently solving MDPs with stochastic mirror descent. In Proceedings of the International Conference on Machine Learning , pages 4890-4900, 2020.

Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei, and Yuejie Chi. Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research , 70(4):2563-2578, 2022.

S´ ebastien Bubeck. Convex optimization: Algorithms and complexity. Foundations and Trends R © in Machine Learning , 8(3-4):231-357, 2015.

Dongsheng Ding and Mihailo R Jovanovi´ c. Policy gradient primal-dual mirror descent for constrained MDPs with large state spaces. In Proceedings of the IEEE 61st Conference on Decision and Control , pages 4892-4897, 2022.

Tao Liu, Ruida Zhou, Dileep Kalathil, Panganamala Kumar, and Chao Tian. Learning policies with zero or bounded constraint violation for constrained MDPs. Proceedings of the Advances in Neural Information Processing Systems , 34:17183-17193, 2021.

Honghao Wei, Xin Liu, and Lei Ying. Triple-Q: A model-free algorithm for constrained reinforcement learning with sublinear regret and zero constraint violation. In Proceedings of the International Conference on Artificial Intelligence and Statistics , pages 3274-3307, 2022.

Shicong Cen, Yuting Wei, and Yuejie Chi. Fast policy extragradient methods for competitive games with entropy regularization. Proceedings of the Advances in Neural Information Processing Systems , 34:27952-27964, 2021.

Jalaj Bhandari and Daniel Russo. On the linear convergence of policy gradient methods for finite mdps. In Proceedings of the International Conference on Artificial Intelligence and Statistics , pages 2386-2394, 2021.

Rui Yuan, Simon Shaolei Du, Robert M Gower, Alessandro Lazaric, and Lin Xiao. Linear convergence of natural policy gradient methods with log-linear policies. In Proceedings of the International Conference on Learning Representations , 2022.

Lin Yang and Mengdi Wang. Sample-optimal parametric Q-learning using linearly additive features. In Proceedings of the International Conference on Machine Learning , pages 69957004, 2019.

Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Proceedings of the Conference on Learning Theory , pages 2137-2143, 2020.

Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global optimality and rates of convergence. In Proceedings of the International Conference on Learning Representations , 2020.

Qinbo Bai, Amrit Singh Bedi, and Vaneet Aggarwal. Achieving zero constraint violation for constrained reinforcement learning via conservative natural policy gradient primal-dual algorithm. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 6737-6744, 2023.

Alexander Rakhlin and Karthik Sridharan. Online learning with predictable sequences. In Proceedings of the Conference on Learning Theory , pages 993-1019. PMLR, 2013.

Lin Xiao. On the convergence rates of policy gradient methods. Journal of Machine Learning Research , 23(282):1-36, 2022.

Laurent Condat. Fast projection onto the simplex and the l 1 ball. Mathematical Programming , 158(1-2):575-585, 2016.

C Daskalakis and Ioannis Panageas. Last-iterate convergence: Zero-sum games and constrained min-max optimization. In Proceedings of the Innovations in Theoretical Computer Science (ITCS) Conference , 2019.

Qi Lei, Sai Ganesh Nagarajan, Ioannis Panageas, et al. Last iterate convergence in no-regret learning: constrained min-max optimization for convex-concave landscapes. In Proceedings of the International Conference on Artificial Intelligence and Statistics , pages 1441-1449, 2021.

Dongsheng Ding, Kaiqing Zhang, Tamer Bas ¸ar, and Mihailo R Jovanovi´ c. Convergence and optimality of policy gradient primal-dual method for constrained Markov decision processes. In Proceedings of the American Control Conference , pages 2851-2856, 2022.

Ruida Zhou, Tao Liu, Dileep Kalathil, Panganamala Kumar, and Chao Tian. Anchorchanging regularized natural policy gradient for multi-objective reinforcement learning. In Proceedings of the Advances in Neural Information Processing Systems , 2022.

Aria HasanzadeZonuzy, Dileep Kalathil, and Srinivas Shakkottai. Model-based reinforcement learning for infinite-horizon discounted constrained Markov decision processes. In Proceedings of the International Joint Conference on Artificial Intelligence , 2021.

Leonid Denisovich Popov. A modification of the Arrow-Hurwicz method for search of saddle points. Mathematical Notes of the Academy of Sciences of the USSR , 28:845-848, 1980.

Ming Yu, Zhuoran Yang, Mladen Kolar, and Zhaoran Wang. Convergent policy optimization for safe reinforcement learning. In Proceedings of the Advances in Neural Information Processing Systems , volume 32, 2019.

Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projectionbased constrained policy optimization. In Proceedings of the International Conference on Learning Representations , 2020.

Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. In Proceedings of the Advances in Neural Information Processing Systems , volume 33, pages 15338-15349, 2020.

Long Yang, Yu Zhang, Jiaming Ji, Juntao Dai, and Weidong Zhang. CUP: A conservative update policy algorithm for safe reinforcement learning, 2022.

Tengyu Xu, Yingbin Liang, and Guanghui Lan. CRPO: A new approach for safe reinforcement learning with convergence guarantee. In Proceedings of the International Conference on Machine Learning , pages 11480-11491, 2021.

Yongshuai Liu, Jiaxin Ding, and Xin Liu. IPO: Interior-point policy optimization under constraints. In Proceedings of the AAAI Aonference on Artificial Intelligence , volume 34, pages 4940-4947, 2020.

Yonathan Efroni, Shie Mannor, and Matteo Pirotta. Exploration-exploitation in constrained MDPs. arXiv preprint arXiv:2003.02189 , 2020.

Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, Zhaoran Wang, and Mihailo R Jovanovi´ c. Provably efficient safe exploration via primal-dual policy optimization. In Proceedings of the International Conference on Artificial Intelligence and Statistics , pages 3304-3312, 2021.

Liyuan Zheng and Lillian Ratliff. Constrained upper confidence reinforcement learning. In Proceedings of the Learning for Dynamics and Control , pages 620-629, 2020.

Shuang Qiu, Xiaohan Wei, Zhuoran Yang, Jieping Ye, and Zhaoran Wang. Upper confidence primal-dual reinforcement learning for CMDP with adversarial loss. In Proceedings of the Advances in Neural Information Processing Systems , volume 33, pages 15277-15287, 2020.

Kiant´ e Brantley, Miro Dudik, Thodoris Lykouris, Sobhan Miryoosefi, Max Simchowitz, Aleksandrs Slivkins, and Wen Sun. Constrained episodic reinforcement learning in concaveconvex and knapsack settings. In Proceedings of the Advances in Neural Information Processing Systems , volume 33, pages 16315-16326, 2020.

Tiancheng Yu, Yi Tian, Jingzhao Zhang, and Suvrit Sra. Provably efficient algorithms for multi-objective competitive RL. In Proceedings of the International Conference on Machine Learning , pages 12167-12176, 2021.

Thiago D Sim˜ ao, Nils Jansen, and Matthijs TJ Spaan. AlwaysSafe: Reinforcement learning without safety constraint violations during training. In Proceedings of the 20th International Conference on Autonomous Agents and Multi-Agent Systems , pages 1226-1235, 2021.

Krishna C Kalagarla, Rahul Jain, and Pierluigi Nuzzo. A sample-efficient algorithm for episodic finite-horizon MDP with constraints. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 8030-8037, 2021.

Sobhan Miryoosefi and Chi Jin. A simple reward-free approach to constrained reinforcement learning. In Proceedings of the International Conference on Machine Learning , pages 1566615698, 2022.

Honghao Wei, Xin Liu, and Lei Ying. A provably-efficient model-free algorithm for infinitehorizon average-reward constrained Markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 3868-3876, 2022.

Liyu Chen, Rahul Jain, and Haipeng Luo. Learning infinite-horizon average-reward Markov decision process with constraints. In Proceedings of the International Conference on Machine Learning , pages 3246-3270, 2022.

Archana Bura, Aria HasanzadeZonuzy, Dileep Kalathil, Srinivas Shakkottai, and JeanFrancois Chamberland. DOPE: Doubly optimistic and pessimistic exploration for safe reinforcement learning. In Proceedings of the Advances in Neural Information Processing Systems , volume 35, pages 1047-1059, 2022.

Arnob Ghosh, Xingyu Zhou, and Ness Shroff. Provably efficient model-free constrained RL with linear function approximation. In Proceedings of the Advances in Neural Information Processing Systems , 2022.

Rahul Singh, Abhishek Gupta, and Ness B Shroff. Learning in constrained Markov decision processes. IEEE Transactions on Control of Network Systems , 10(1):441-453, 2022.

Honghao Wei, Arnob Ghosh, Ness Shroff, Lei Ying, and Xingyu Zhou. Provably efficient model-free algorithms for non-stationary CMDPs. In Proceedings of the International Conference on Artificial Intelligence and Statistics , pages 6527-6570, 2023.

Jacopo Germano, Francesco Emanuele Stradi, Gianmarco Genalti, Matteo Castiglioni, Alberto Marchesi, and Nicola Gatti. A best-of-both-worlds algorithm for constrained MDPs with long-term constraints. arXiv preprint arXiv:2304.14326 , 2023.

Krishna C Kalagarla, Rahul Jain, and Pierluigi Nuzzo. Safe posterior sampling for constrained MDPs with bounded constraint violation. arXiv preprint arXiv:2301.11547 , 2023.

Soumyajit Guin and Shalabh Bhatnagar. A policy gradient approach for finite horizon constrained Markov decision processes. arXiv preprint arXiv:2210.04527 , 2022.

Santiago Paternain, Luiz Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro. Constrained reinforcement learning has zero duality gap. In Proceedings of the Advances in Neural Information Processing Systems , 2019.

Wenhao Zhan, Shicong Cen, Baihe Huang, Yuxin Chen, Jason D Lee, and Yuejie Chi. Policy mirror descent for regularized reinforcement learning: A generalized framework with linear convergence. SIAM Journal on Optimization , 33(2):1061-1091, 2023.

Guanghui Lan. Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes. Mathematical Programming , 198(1):1059-1106, 2023.

Paul Tseng. On linear convergence of iterative methods for the variational inequality problem. Journal of Computational and Applied Mathematics , 60(1-2):237-252, 1995.

Yu Malitsky. Projected reflected gradient methods for monotone variational inequalities. SIAM Journal on Optimization , 25(1):502-520, 2015.

Jacob Abernethy, Kevin A Lai, and Andre Wibisono. Last-iterate convergence rates for minmax optimization: Convergence of Hamiltonian gradient descent and consensus optimization. In Proceedings of the Algorithmic Learning Theory , pages 3-47, 2021.

Yang Cai, Argyris Oikonomou, and Weiqiang Zheng. Accelerated algorithms for monotone inclusions and constrained nonconvex-nonconcave min-max optimization. arXiv preprint arXiv:2206.05248 , 2022.

Quoc Tran-Dinh. Extragradient-type methods with O (1 /k ) -convergence rates for cohypomonotone inclusions. arXiv preprint arXiv:2302.04099 , 2023.

Stephen J Wright. Primal-dual interior-point methods . SIAM, 1997.

Michele Benzi, Gene H Golub, and J¨ org Liesen. Numerical solution of saddle point problems. Acta Numerica , 14:1-137, 2005.

Kenneth J Arrow and Leonid Hurwicz. Reduction of constrained maxima to saddle-point problems. In Traces and Emergence of Nonlinear Programming , pages 61-80. 2013.

Kazufumi Ito and Karl Kunisch. Lagrange multiplier approach to variational problems and applications . SIAM, 2008.

Bingsheng He and Xiaoming Yuan. On the acceleration of augmented Lagrangian method for linearly constrained optimization. Optimization Online , 3, 2010.

Alp Yurtsever, Quoc Tran Dinh, and Volkan Cevher. A universal primal-dual convex optimization framework. In Proceedings of the Advances in Neural Information Processing Systems , volume 28, 2015.

Bingsheng He and Xiaoming Yuan. On non-ergodic convergence rate of Douglas-Rachford alternating direction method of multipliers. Numerische Mathematik , 130(3):567-577, 2015.

Ya-Feng Liu, Xin Liu, and Shiqian Ma. On the nonergodic convergence rate of an inexact augmented Lagrangian framework for composite convex programming. Mathematics of Operations Research , 44(2):632-650, 2019.

Yangyang Xu. Iteration complexity of inexact augmented Lagrangian methods for constrained convex programming. Mathematical Programming , 185:199-244, 2021.

Quoc Tran Dinh. Non-ergodic alternating proximal augmented Lagrangian algorithms with optimal rates. Proceedings of the Advances in Neural Information Processing Systems , 31, 2018.

Quoc Tran-Dinh and Yuzixuan Zhu. Augmented Lagrangian-based decomposition methods with non-ergodic optimal rates. arXiv preprint arXiv:1806.05280 , 2018.

Shoham Sabach and Marc Teboulle. Faster Lagrangian-based methods in convex optimization. SIAM Journal on Optimization , 32(1):204-227, 2022.

Tao Zhang, Yong Xia, and Shiru Li. Lagrangian-based methods in convex optimization: prediction-correction frameworks with non-ergodic convergence rates. arXiv preprint arXiv:2304.02459 , 2023.

Andrew Cotter, Heinrich Jiang, and Karthik Sridharan. Two-player games for efficient nonconvex constrained optimization. In Proceedings of the Algorithmic Learning Theory , pages 300-332, 2019.

Richard Bellman and Stuart Dreyfus. Functional approximations and dynamic programming. Mathematics of Computation , 13(68):247-251, 1959.

Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014.

Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Proceedings of the Conference on Learning Theory , pages 2730-2775, 2022.

John Nash Jr. Non-cooperative games. Annals of Mathematics , 54(2):286-295, 1951.

Vivek S Borkar. A convex analytic approach to Markov decision processes. Probability Theory and Related Fields , 78(4):583-602, 1988.

Junyu Zhang, Amrit Singh Bedi, Mengdi Wang, and Alec Koppel. Cautious reinforcement learning via distributional risk in the dual domain. IEEE Journal on Selected Areas in Information Theory , 2(2):611-626, 2021.

Qinbo Bai, Amrit Singh Bedi, Mridul Agarwal, Alec Koppel, and Vaneet Aggarwal. Achieving zero constraint violation for concave utility constrained reinforcement learning via primal-dual approach. 2022.

Donghao Ying, Mengzi Amy Guo, Yuhao Ding, Javad Lavaei, and Zuo-Jun Shen. Policybased primal-dual methods for convex constrained Markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 10963-10971, 2023.

Maurice Sion. On general minimax theorems. Pacific Journal of Mathematics , 8(1):171-176, 1958.

Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the International Conference on Machine Learning , pages 928-936, 2003.

Yasushi Terazono and Ayumu Matani. Continuity of optimal solution functions and their conditions on objective functions. SIAM Journal on Optimization , 25(4):2050-2060, 2015.

Simon Lacoste-Julien, Mark Schmidt, and Francis Bach. A simpler approach to obtaining an O (1 /t ) convergence rate for the projected stochastic subgradient method. arXiv preprint arXiv:1212.2002 , 2012.

Yuhao Ding, Junzi Zhang, and Javad Lavaei. Beyond exact gradients: Convergence of stochastic soft-max policy gradient methods with entropy regularization. arXiv preprint arXiv:2110.10117 , 2021.

Haipeng Luo. Introduction to online optimization/learning: Lecture 1. https: //haipeng-luo.net/courses/CSCI659/2022_fall/lectures/lecture1.pdf , 2022. Access on January 21, 2023.

Supplementary Materials for 'Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPs'

A More Comparisons and Additional Related Works

In this section, we discuss more comparison details and other related works.

Table 1: Iteration complexities of our methods and representative algorithms for solving a constrained MDP problem: maximize π V π r ( ρ ) subject to V π g ( ρ ) ≥ 0 with reward/utility functions r ∈ [0 , 1] , g ∈ [ -1 , 1] . The iteration complexity is the number of gradient-based updates for an algorithm to output the last policy-iterate π t that satisfies V glyph[star] r ( ρ ) -V π t r ( ρ ) ≤ glyph[epsilon1] and -V π t g ( ρ ) ≤ glyph[epsilon1] .

Last-iterate and value-average (or policy-mixture) performance in constrained MDPs . There has been a flurry of research activities in studying convergence behaviors of direct policy search or policy gradient-based algorithms for constrained MDPs in the infinite-horizon discounted setting. There are two main streams: (i) Lagrangian-based policy search and (ii) Approximate constrained policy search.

(i) Lagrangian-based policy search . In the Lagrangian-based framework, last-iterate performance has been established as asymptotic convergence for several policy-based primal-dual algorithms, e.g., naive policy gradient-based primal-dual method [21] and actor-critic variants of policy gradient primal-dual methods [6, 22, 16, 9]. These studies rely on modeling primal-dual updates in two separate time scales and/or two nested loops as continuous-time gradient flow dynamics, and their asymptotic convergence is often restricted to some stationary points. We notice that recent global asymptotic convergence results [36, 40] are in terms of occupancy measure iterates instead of instantaneous policy iterates, as highlighted in Table 1. In contrast, our first method: OPD strengthens the asymptotic last-iterate convergence to non-asymptotic and last-iterate convergence with a linear rate.

To provide efficiency and optimality performances, it is crucial to develop algorithms with global non-asymptotic convergence guarantees. Several recent policy-based primal-dual algorithms have been proved to converge with non-asymptotic convergence rates, e.g., policy gradient primal-dual method [69], natural policy gradient-based or policy mirror-descent style primal-dual methods [23, 27, 54], accelerated natural policy gradient-based primaldual methods [24, 25, 28], actor-critic version of natural policy gradient-based primal-dual

method [26], and anchor-changing natural policy gradient-based primal-dual method [70]. These studies characterize non-asymptotic convergence to an optimal constrained policy regarding the average of value functions, except for [27] in which the convergence is for a mixture of past policies and [28] in which the convergence is for a policy induced by a history-weighted occupancy measure. Similar non-asymptotic convergence can also be found in an occupancy measure-based primal-dual method [42], where the convergence is in terms of the average of occupancy measures. Besides, sublinear non-asymptotic convergence can be found in generative model-based methods [71, 41], regarding a mixture of past policies. In contrast, our two policy-based methods: RPG-PD and OPG-PD strengthen the sublinear non-asymptotic convergence of average value functions or a mixture of past policies to sublinear and linear non-asymptotic convergence of policy iterates. In particular, we exploit the regularization technique [52] and the optimistic gradient method [72, 37] to augment typical policy-based primal-dual methods with novel identification of decreasing distances of policy iterates to an optimal constrained policy, which allows for stronger convergence.

Instead of working with policy primal and dual variables both, Lagrangian-based dual method formulates a constrained MDP as a convex dual problem that enables classical dual ascent method [12]. Despite guaranteed convergence in dual space from convex optimization, it is challenging to compute an optimal constrained policy in primal policy space even if we use an optimal dual variable [9, 47, 30]. Recently, regularization [52] has been used in dual ascent methods [34, 35] in which policy last-iterates of natural policy gradientbased subroutines can be nearly-optimal. These dual-based algorithms [34, 35] work with two nested loops and their non-asymptotic last-iterate convergence relies on tuning loop parameters optimally. In contrast, our two policy-based methods: RPG-PD and OPG-PD remove the double loop requirement as listed in Table 1, which permits outputting the lastiterate policy as a nearly-optimal constrained policy. In Table 1, we see that our RPG-PD method has a worse rate while OPG-PD achieves a linear rate which is similar as the dualbased methods [34, 35]. Importantly, our non-asymptotic convergence characterizes the stability of primal-dual iterates generated by the algorithms, which is theoretically stronger and more appealing in practice.

(ii) Approximate constrained policy search . Approximation of constrained MDPs with surrogate functions has been shown to be effective, e.g., constrained policy optimization [8], successive convex relaxation [73], projection-based constrained policy optimization [74], first-order constrained optimization [75], and conservative policy update [76]. These studies have shown impressive empirical performance by iteratively solving an approximated constrained optimization problem and such performance is characterized in the worst-case policy improvement, except for [73] in which local asymptotic convergence is established, which leaves non-asymptotic convergence of policy iterates open. A related approach is the primal method [77] that treats a constrained MDP as an unconstrained one and corrects policy iterates whenever constraint violation happens. Non-asymptotic convergence of primal method has been established in terms of mixture of past policies. Another approximation method is the interior-point policy optimization [78] that solves an unconstrained MDP by adding a logarithm barrier function into the objective function, while convergence of this method is unknown. In contrast, we have supported our methods with non-asymptotic last-iterate convergence. Since Lagrangian-based methods are typically used to solve an approximated constrained optimization problem [8, 74, 75, 76], our methodologies can be applied to these methods for better convergence, which we leave as future work.

For constrained MDPs in the finite-horizon episodic setting and the total or average-reward settings, there is a rich line of works that have developed learning algorithms with non-asymptotic convergence guarantees in terms of the average of value functions [79, 80, 81, 82, 83, 84, 85, 55, 86, 87, 88, 89, 56, 90, 91, 92, 93, 94, 95], except for [96] in which global asymptotic convergence of a policy gradient method has been established in the finite-horizon constrained MDP setting. Although being not directly comparable, their non-asymptotic and last-iterate convergence are not established yet, to the best of our knowledge.

Lagrangian-based policy gradient methods in constrained MDPs . Our methods are closely pertinent to Lagrangian-based policy gradient methods for solving constrained MDPs in the infinitehorizon discounted setting, e.g., policy gradient-based primal-dual methods [21, 6, 22, 16, 9, 69],

natural policy gradient-based or mirror-descent style primal-dual methods [23, 27, 28, 24, 54, 25, 26], dual descent methods [97, 34, 35, 12]. Regarding algorithm implementation, primal-dual methods [21, 6, 22, 16, 9, 23, 27, 28, 24, 54, 25, 26, 69] work with primal-dual iterates simultaneously in a single loop, which is similar to the classical gradient-based primal-dual methods in constrained optimization [31, 32, 33], while diminishing stepsizes in different speeds is often required in many of them [21, 6, 16, 9]; dual descent methods [97, 34, 35, 12] intermittently operate the dual iterate only after a sufficient number of primal iterations, which often adds difficulty of tuning hyper-parameters of nested loops in practice. It is worth mentioning that, it is convenient to view such dual descent methods as primal-dual methods that update primal variable faster than iterating dual variable. With respect to convergence analysis, stochastic approximation has been widely used to establish asymptotic convergence of several primal-dual methods [6, 22, 16, 9] by analyzing the stability of limiting gradient flow dynamics, while recent methods [28, 24, 54, 25, 26, 69, 34, 35] exploit the connection between policy gradient and mirror-descent in convex optimization to prove non-asymptotic convergence. However, non-asymptotic convergence of primal-dual methods only characterizes the average of value functions [23, 54, 25, 26, 69] or a mixture of past policies [27, 28, 24] because of the dual update that results from regulating constraint violation. For the dual descent methods [34, 35], non-asymptotic convergence is characterized in terms of last-iterate policies that are computed by approximately solving unconstrained RL problems with fixed dual variables. Therefore, designing Lagrangian-based policy gradient methods that enjoy the single-loop simplicity and the non-asymptotic convergence of policy iterates is challenging, because of the oscillation and overshoot issues of updating primal-dual variables simultaneously [29, 40]. In this work, we have established non-asymptotic and last-iterate convergence of two single-loop Lagrangian-based policy gradient methods via the regularization and optimistic gradient techniques and our analysis builds on the mirror-descent analysis for policy gradient methods [49, 98, 65, 99] while focusing on the distance of policy iterates, which is stronger than the prior art. Compared with recent efforts [40, 36] as shown in Table 1, our algorithms are simpler and our theoretical guarantees are stronger.

Gradient-based methods with last-iterate convergence for learning in games . Since the Lagrangian-based approach for constrained MDPs can be viewed as solving a constrained saddlepoint problem, another line of related work is gradient-based methods for solving saddle-point (or minimax optimization) problems with last-iterate convergence. Last-iterate convergence of gradientbased methods has been established in several scenarios, e.g., linear rates of extragradient methods for strongly convex problems [100, 101], asymptotic convergence of optimistic multiplicative weights updates for convex problems [67, 68], linear rates of optimistic gradient methods for convex problems [43], and lower bound-matching rates of extragradient and optimistic gradient methods for convex problems [39]. These studies focus on convex-concave saddle-point problems except for [102, 103, 104] in which non-asymptotic last-iterate convergence is achievable for saddle-point problems with special non-convexity structure. In contrast, our constrained saddle-point problem that result from constrained MDP is non-convex in policy primal variable, which prevents direct application of these last-iterate results. A slightly twisted exception is that our constrained saddle-point problem can be reformulated to be convex in occupancy measure instead of policy, which leads the second and third methods in Table 1. To solve our Lagrangian-based constrained saddle-point problem in policy space, our first policy-based method: RPG-PD relaxes the non-convexity by adding regularization into the objective function and we provide sublinear last-iterate policy convergence guarantee using the distance analysis for policy primal-dual iterates. To improve the convergence rate, we further develop another policy-based method: OPG-PD that extends the optimistic gradient method [37] for a class of non-convex constrained saddle-point problems. This extension departs from previous extensions for zero-sum Markov games [43, 44], because lacking of uniformly optimal policies prevents learning an optimal policy from per-state bilinear games. Instead, we provide a new distance analysis of policy primal-dual iterates of an optimistic gradient method for solving a new class of constrained non-convex saddle-point problems, with linear last-iterate policy convergence.

Non-asymptotic last-iterate (or non-ergodic) convergence in constrained optimization . Reduction of constrained optimization to saddle-point problems is a classical idea to solve constrained optimization problems by developing primal-dual algorithms, e.g., primal-dual interiorpoint methods [105], Uzawa and Arrow-Hurwicz algorithms [31, 106, 107], and Lagrange multiplier methods [108, 45]. Inspired by the Lagrange multiplier methods, many recent studies on constrained optimization have significantly advanced primal-dual algorithms with last-iterate convergence, e.g., accelerated augmented Lagrangian method [109], accelerated universal primal-

dual gradient method [110], Douglas-Rachford alternating direction method [111], inexact augmented Lagrangian method [112, 113], alternating proximal augmented Lagrangian algorithm [114], augmented Lagrangian-based decomposition method [115], faster Lagrangian method [116], and prediction-correction-based primal-dual method [117]. However, all these studies build on augmented Lagrangian methods to solve the classical convex optimization problems with linear constraints. In comparison, we have studied a class of non-convex constrained optimization problems that result from constrained MDPs using the standard Lagrange multiplier method. We notice that a Lagrangian-based two-player game has been used to study general non-convex constrained optimization problems with average performance analysis [118]. Our two policy-based primaldual methods with sublinear and linear last-iterate convergence appear to be the first global nonasymptotic and last-iterate convergence result in non-convex constrained optimization.

B Proofs in Section 2

In this section, we make some helpful observations and provide proofs of the claims in Section 2.

B.1 Lack of uniformly optimal stationary policies in constrained MDPs

In unconstrained MDPs, there always exists an optimal policy that is optimal simultaneously for all states (e.g., see [119] and [120, Chapter 6]). In contrast, this is not true anymore for constrained MDPs. To see this, we adopt a counter-example from [46] and investigate it in the constrained MDP setting.

Figure 3: An example of a constrained MDP that has the objective function V π r ( ρ ) and the constraint set { π ∈ Π | V π g ( ρ ) ≥ 0 } . The pair ( a, r, g ) associated with a directed arrow represents ( reward, utility ) received when an action a at a certain state is taken.

We introduce a constrained MDP with two states: Left ( L ) and Right ( R ), in Figure 3. In each state, there are two actions { L, R } . The MDP transition dynamics is deterministic. In state L , if the agent chooses action L , then the next state is L and the reward/utility (0 , 0) is received; otherwise, action R leads to next state R and reward/utility (1 , -1) . In state R , no matter which action the agent takes, the next state is R and the reward/utility (1 , 1) is received.

Since the state R is trivial, a stationary policy π can be represented by the probability of taking action L in state L denoted by p . With a slight abuse of notation, we use notion ρ to represent the probability of starting off from state L . Thus, we can compute the value functions as follows via the Bellman equations, i.e., V ( s ) = ∑ a π ( a | s )( r ( s, a ) + γ ∑ s ′ P ( s ′ | s, a ) V ( s )) for all s .

(i) It is easy to check a basic case: γ = 0 [46]. We can compute the value functions as follows,

Feasibility of the policy p requires that V p g ( ρ ) ≥ 0 , i.e.,

Hence, the maximum V p r ( ρ ) within the feasible region can be reached at the optimal policy,

Therefore, the optimal policy p glyph[star] depends on the initial state distribution ρ . Moreover, except that ρ = 1 or 1 2 , the optimal policy p glyph[star] is a stochastic policy and is unique.

(ii) A slightly more general case is given by γ = 1 4 . Thus, we can compute the value functions as follows,

Feasibility of the policy p requires that V p g ( ρ ) ≥ 0 , i.e.,

In particular, if we take ρ = 7 9 , then p ≥ 1 2 . In this case, the maximization of V p r ( ρ ) yields an optimal policy p glyph[star] = 1 2 , which is a uniform policy and is unique.

B.2 Scalarization fallacy in constrained MDPs

In constrained RL, scalarization is often used to reduce a constrained MDP problem to a standard unconstrained one, which might permit many unconstrained RL algorithms [8, 9]. Unfortunately, as pointed out in the literature (e.g., [46, Part 4] and [30]), such a reduction does not necessarily provide a solution to the original constrained MDP problem. It is easy to see this from the previous examples in Figure 3. In the basic case: γ = 0 , if we take ρ = 3 4 , then from (10) the optimal policy is given by p glyph[star] = 2 3 , which is a stochastic policy; we see a uniform optimal policy when γ = 1 4 . By shaping a composite function r + λg with some fixed λ ∈ [0 , ∞ ] , the scalarization method aims to solve the following unconstrained MDP problem,

However, by the optimality of dynamic programming [120, Chapter 6], an optimal policy is given in a deterministic form which has been widely used in theory and practice. Therefore, solving the above scalarized version of a constrained MDP problem does not necessarily provide an optimal solution for the original constrained MDP problem. We also notice that this phenomenon is reported in recent empirical studies [9, 30] and a more formal statement [47, Lemma 1]. Hence, it can be infeasible for dual descent methods [12, 30] to find an optimal constrained policy, because solving Problem (11) often yields a deterministic policy, which can be sub-optimal for a constrained MDP with a unique stochastic optimal policy, e.g., constrained MDP examples in Appendix B.1.

B.3 Properties of saddle points

First, we state the invariance property of saddle points [121] for our constrained saddle-point problem (3). By the invariance property of saddle points, we can restrict the problem domain without changing the saddle-point property when the original saddle points are contained in the restricted domain. Let the set of max-min points be Π glyph[star] := argmax π ∈ Π min λ ∈ [0 , ∞ ] V π r + λg ( ρ ) and the set of min-max points be Λ glyph[star] := argmin λ ∈ [0 , ∞ ] max π ∈ Π V π r + λg ( ρ ) .

Lemma 8 (Invariance of saddle points) . Let ( π glyph[star] , λ glyph[star] ) ∈ Π glyph[star] × Λ glyph[star] be a saddle point of V π r + λg ( ρ ) over Π × [0 , + ∞ ] . For any subset Λ ′ ⊂ [0 , + ∞ ] , if ( π glyph[star] , λ glyph[star] ) ∈ Π × Λ ′ , then ( π glyph[star] , λ glyph[star] ) is a saddle point of V π r + λg ( ρ ) over Π × Λ ′ .

Proof. From the saddle-point property of ( π glyph[star] , λ glyph[star] ) , we have

It is straightforward to see that

Since Λ ′ ⊂ [0 , + ∞ ] and λ glyph[star] ∈ Λ ′ , V π glyph[star] r + λ glyph[star] g ( ρ ) = min λ ∈ Λ ′ V π glyph[star] r + λg ( ρ ) . Hence,

Finally, combining (12) and (13) defines ( π glyph[star] , λ glyph[star] ) as a saddle point of of V π r + λg ( ρ ) over Π × Λ ′ .

We next show the interchangeability of saddle points in two-player zero-sum games [122] for our non-convex game.

Lemma 9 (Interchangeability of saddle points) . Let ( π glyph[star] , λ glyph[star] ) , (¯ π glyph[star] , ¯ λ glyph[star] ) ∈ Π glyph[star] × Λ glyph[star] be two saddle points of V π r + λg ( ρ ) over Π × [0 , + ∞ ] . Then, both ( π glyph[star] , ¯ λ glyph[star] ) and (¯ π glyph[star] , λ glyph[star] ) are saddle points of V π r + λg ( ρ ) over Π × [0 , + ∞ ] .

Proof. By the definition of saddle points ( π glyph[star] , λ glyph[star] ) and (¯ π glyph[star] , ¯ λ glyph[star] ) ,

Then,

Therefore,

for all π ∈ Π and λ ∈ [0 , ∞ ] , which shows that ( π glyph[star] , ¯ λ glyph[star] ) is a saddle point of V π r + λg ( ρ ) over Π × [0 , + ∞ ] .

Similarly, we can prove it for (¯ π glyph[star] , λ glyph[star] ) .

B.4 Constrained MDPs in occupancy-measure space

In the analytic approach [123, 5], the value functions in Problem (1) are bilinear in the occupancy measure V π glyph[diamondmath] ( ρ ) = 〈glyph[diamondmath] , q 〉 for glyph[diamondmath] = r or u , where q π : S × A → R is an (un-normalized) occupancy measure over the state-action space,

which adds up discounted probabilities of visiting ( s, a ) in the execution of π . Furthermore, we let the operator P glyph[latticetop] : R | S || A | → R | S | be ( P glyph[latticetop] q )( s ) := ∑ s ′ ,a ′ P ( s | s ′ , a ′ ) q ( s ′ , a ′ ) , and the operator E glyph[latticetop] : R | S || A | → R | S | be ( E glyph[latticetop] q )( s ) := ∑ a q ( s, a ) or simply q ( s ) . With a slight abuse of notation, we denote ρ = [ ρ ( s 1 ) , . . . , ρ ( s | S | ) ] glyph[latticetop] . A valid occupancy measure q π ∈ R | S || A | satisfies the the Bellman flow equations,

It is known that the Bellman flow constraint is necessary and sufficient for any q ∈ R | S || A | to be a valid occupancy measure (e.g., [124, Lemma 1]).

Let the concatenation of r ( s, a ) , u ( s, a ) for all ( s, a ) be r , u ∈ R | S || A | , respectively. In the occupancy measure space, the goal of a constrained MDP is to find a solution q glyph[star] of a linear program,

Denoting g : S × A → [ -1 , 1] as g := u -(1 -γ ) b , we simplify the constraint 〈 u, q 〉 ≥ b as 〈 g, q 〉 ≥ 0 . By the method of Lagrange multipliers, we dualize two constraints in (16) and introduce a standard Lagrangian,

where π ∈ Π is the primal variable, λ ∈ [0 , ∞ ] is the dual variable for the constraint 〈 g, q 〉 ≥ 0 , and µ is the dual variable for the equality constraint in Q . Since the strong duality holds for any feasible linear program, the boundedness of λ glyph[star] in Lemma 1 holds for L ( q, λ, µ ) . Thus, any saddle point ( q glyph[star] , λ glyph[star] , µ glyph[star] ) is also a max-min and min-max point, i.e., q glyph[star] is the occupancy measure associated with the optimal policy π glyph[star] , and ( λ glyph[star] , µ glyph[star] ) ∈ argmin λ ≥ 0 ,µ max q ≥ 0 L ( q, λ, µ ) . Boundedness of q glyph[star] is straightforward from (14),

which allows us further restrict q ∈ Q ⊂ Q . We next show boundedness of ( λ glyph[star] , µ glyph[star] ) in Lemma 10. Lemma 10 (Boundedness) . Let Assumption 1 hold. Then, λ glyph[star] ∈ Λ and µ glyph[star] ∈ M , where Λ is stated below (3) and M := { µ | | µ ( s ) | ≤ µ max , ∀ s ∈ S } where µ max := 1 -γ +1 /ξ (1 -γ ) 2 .

Proof. From the saddle-point property of ( q glyph[star] , λ glyph[star] , µ glyph[star] ) , we have

equivalently, for any q ∈ Q , ∂ q L ( q glyph[star] , λ glyph[star] , µ glyph[star] ) glyph[latticetop] ( q -q glyph[star] ) ≤ 0 , and for any µ and λ ≥ 0 , ∂ µ L ( q glyph[star] , λ glyph[star] , µ glyph[star] ) glyph[latticetop] ( µ -µ glyph[star] ) + ∂ λ L ( q glyph[star] , λ glyph[star] , µ glyph[star] )( λ -λ glyph[star] ) ≥ 0 . Hence, for any q ∈ Q ,

Arbitrary q ∈ Q demands the inequality r +( γP -E ) µ glyph[star] + λ glyph[star] g ≤ 0 . By the definition of occupancy measure, we know that for any s ∈ S there exists an action a ∈ A such that q glyph[star] ( s, a ) > 0 . Thus, the equality r +( γP -E ) µ glyph[star] + λ glyph[star] g = 0 must hold at such state-action pairs in which we represent the associated reward and transition by (¯ r, ¯ P ) . Hence,

Thus, (1 -γ ) ‖ µ glyph[star] ‖ ∞ ≤ 1 + λ glyph[star] . However, by Assumption 1, L (¯ q, λ glyph[star] , µ glyph[star] ) ≤ L ( q glyph[star] , λ glyph[star] , µ glyph[star] ) . Hence,

which, together with the feasibility of ¯ q and the optimality of q glyph[star] , imply that 0 ≤ λ glyph[star] ξ ≤ 〈 r, q glyph[star] -¯ q 〉 . Hence, 0 ≤ λ glyph[star] ≤ 1 (1 -γ ) ξ , which further yields a bound on ‖ µ glyph[star] ‖ ∞ .

We now obtain a constrained saddle-point problem in terms of the q -based Lagrangian,

where we take bounded polytopes Q and Λ × M such that they contain q glyph[star] and ( λ glyph[star] , µ glyph[star] ) , respectively. For notational brevity, we introduce z = ( q, λ, µ ) , Z := Q × Λ × M , and Z glyph[star] := Q glyph[star] × Λ glyph[star] × M glyph[star] which contains all saddle points z glyph[star] := ( q glyph[star] , λ glyph[star] , µ glyph[star] ) . Let the gradient of L ( q, λ, µ ) be

Due to the bilinearity of L ( q, λ, µ ) over a compact domain, L ( q, λ, µ ) has a gradient Lipschitz constant L f . Let P X be the projection operator onto a set X , i.e., P X ( x ) := argmin x ′ ∈ X ‖ x ′ -x ‖ . Since the q -based Lagrangian L ( q, λ, µ ) is bilinear and the domains are polytopes, Problem (17) satisfies the metric subregularity condition [38, Theorem 5].

Lemma 11 (Metric subregularity condition) . Let Assumption 1 hold. Then, the gradient function F ( z ) satisfies that for any z ∈ Z/Z glyph[star] with z glyph[star] = P Z glyph[star] ( z ) ,

where C > 0 is a problem-dependent constant.

Hence, application of the optimistic gradient method [38] to Problem (17) yields an Optimistic Primal-Dual (OPD) algorithm that begins with two initial tuples of primal/dual variables ( q 0 , λ 0 , µ 0 ) = (ˆ q 1 , ˆ λ 1 , ˆ µ 1 ) ∈ Z , and performs two gradient steps for each primal/dual variable at time t ≥ 1 ,

where η is the stepsize. Let the squared distance of a point z ∈ Z to the set Z glyph[star] be dist 2 ( z, Z glyph[star] ) := ‖ z -P Z glyph[star] ( z ) ‖ 2 . It is straightforward to employ [38, Theorem 8] to claim last-iterate convergence guarantee of OPD (18) below.

Theorem 12 (Linear convergence of OPD) . Let Assumption 1 hold. If the stepsize η in OPD (18) satisfies η < 1 8 L f , then the iterates { z t } t ≥ 0 converge to the set of saddle points Z glyph[star] linearly,

where C 1 = 64 dist 2 (ˆ z 1 , Z glyph[star] ) and C 2 = min( 16 η 2 C 2 81 , 1 2 ) for a problem-dependent constant C > 0 .

Theorem 12 shows that the primal-dual iterates of OPD converge to the saddle point set Z glyph[star] in linear rate. Compared with a contemporaneous work [40], OPD is free of projection to a occupancy measure set, and enjoys non-asymptotic and last-iterate linear convergence. If the underlying policy is recovered via π t ( a | s ) = q t ( s,a ) ∑ a ′ q t ( s,a ′ ) for all ( s, a ) , these policy iterates π t associated with the occupancy measure iterates q t also converge to an optimal constrained policy π glyph[star] .

Corollary 13 (Nearly-optimal constrained policy) . Let Assumption 1 hold. Assume ρ min := min s ρ ( s ) > 0 and re-define Q := { q ∈ R | S || A | | 0 ≤ q ( s, a ) ≤ 1 / (1 -γ ) , q ( s ) ≥ ρ min / (1 -γ ) , ∀ ( s, a ) ∈ S × A } . If we set the stepsize η in a similar way as in Theorem 12, then the recovered policy iterates { π t } t ≥ 0 of OPD (18) satisfy

where π t ( a | s ) = q t ( s,a ) ∑ a ′ q t ( s,a ′ ) for all ( s, a ) , and Ω( · ) only has some problem-dependent constant.

Proof. The key to our proof is to connect the occupancy measure iterates q t with associated policy iterates π t . It is straightforward that Theorem 12 continues to hold, with a further restricted the domain Q . If we set the stepsize η in a similar way as in Theorem 12, then for any t = Ω(log 1 glyph[epsilon1] ) ,

Let π glyph[star] t be a policy that is associated with the occupancy measure q π glyph[star] t = P Q glyph[star] ( q t ) and π t be a policy that is associated with the occupancy measure iterate q t , i.e., π t ( a | s ) = q t ( s,a ) ∑ a ′ q t ( s,a ′ ) for all ( s, a ) . We denote ( λ glyph[star] t , µ glyph[star] t ) := ( P Λ glyph[star] ( λ t ) , P M glyph[star] ( µ t )) . Thus,

where the first inequality is due to triangle inequality, we use the fact: (1 -γ ) q π glyph[star] t ( s ) = d π glyph[star] t ρ ( s ) , d π glyph[star] t ρ ( s ) ≤ 1 , and ‖ q t ( s, · ) ‖ ≤ √ | A | 1 -γ in the second inequality, the third inequality is due to that q π glyph[star] t ( s ) , q t ( s ) ≥ ρ min , and ρ min > 0 , we apply Cauchy-Schwarz inequality in the fourth inequality, and finally we combine two square root terms by relaxing the first one in the last inequality.

First, we have

where the equality is due to performance difference lemma in Lemma 28, we use Cauchy-Schwarz inequality in the first inequality, and the second inequality is due to ‖ x ‖ 1 ≤ √ d ‖ x ‖ 2 for x ∈ R d , which shows V π glyph[star] t r ( ρ ) -V π t r ( ρ ) ≤ O ( √ glyph[epsilon1] ) . By the optimality of π glyph[star] t , V π glyph[star] t r ( ρ ) = V π glyph[star] r ( ρ ) . Therefore, V π glyph[star] r ( ρ ) -V π t r ( ρ ) ≤ O ( √ glyph[epsilon1] ) .

Second, we have

Similar to bounding V π glyph[star] t r ( ρ ) -V π t r ( ρ ) , we can show that (ii) ≤ O ( √ glyph[epsilon1] ) . By the optimality of π glyph[star] t , we have V π glyph[star] t g ( ρ ) ≥ 0 . Therefore, -V π t g ( ρ ) ≤ O ( √ glyph[epsilon1] ) .

Finally, we replace the accuracy √ glyph[epsilon1] by glyph[epsilon1] and combine all big O notation to conclude the proof.

Corollary 13 states that after an almost constant number of iterations a policy induced by the last occupancy measure iterate of OPD is an glyph[epsilon1] -optimal constrained policy for Problem (1). We notice that recent last-iterate convergence result for convex-concave saddle-point problems [39] is also applicable to Problem (17), which provides the optimal rate without problem-dependent constants. It is worth mentioning that direct application of such last-iterate convergence results in convex minimax optimization to constrained MDPs with general utilities [125, 126] and convex MDPs [124, 47] in occupancy-measure space is also straightforward. We omit these exercises in this paper, and focus on the design and analysis of algorithms in policy space.

C Proofs in Section 3

In this section, we provide proofs of the claims in Section 3.

C.1 Existence and uniqueness of regularized saddle points

Lemma 14 (Existence and uniqueness) . There exists a unique primal-dual pair (¯ π, ¯ λ ) ∈ Π × Λ such that L τ (¯ π, λ ) ≥ L τ (¯ π, ¯ λ ) ≥ L τ ( π, ¯ λ ) for any π ∈ Π and λ ∈ Λ .

Proof. We first re-write the regularized Lagrangian L τ ( π, λ ) in terms of occupancy measure q ∈ Q ,

We next use notation L π ( q, λ ) to represent L τ ( π, λ ) . We first check that H ( q ) is a concave function,

for any q 1 , q 2 ∈ Q and α ∈ [ 0 , 1 ] , where the inequality is because of the log sum inequality ( ∑ i a i ) ln ∑ i a i ∑ i b i ≤ ∑ i a i ln a i b i for non-negative a i and b i , and the equality holds if and only if

Therefore, L τ ( q, λ ) is concave in q ∈ Q and strongly convex in λ ∈ Λ . We notice that Q is a polytope and Λ is a bounded interval. By Sion's minimax theorem [127], L τ ( q, λ ) has a saddle point (¯ q, ¯ λ ) ∈ Q× Λ . From the one-to-one correspondence between policy and occupancy measure, ¯ q induces a policy ¯ π and (¯ π, ¯ λ ) serves as a saddle point of L τ ( π, λ ) , which proves the existence of saddle points.

To show the uniqueness of (¯ π, ¯ λ ) (or (¯ q, ¯ λ ) ), it is sufficient to show that L τ ( q, λ ) is strictly concave in q and strictly convex in λ . The second argument is straightforward from the strong convexity of L τ ( q, λ ) in λ . We next show the first argument that L τ ( q, λ ) is strictly concave in q for any λ ∈ Λ . Assume that there are two (different) policies π 1 and π 2 and the associated two occupancy measures are q 1 and q 2 . From the concavity of H ( q ) , there is another occupancy measure αq 1 + (1 -α ) q 2

that satisfies

where the inequality ≥ has to be strict since π 1 and π 2 differentiate by at least one state-action pair (¯ s, ¯ a ) , glyph[negationslash]

Hence, a new policy associated with αq 1 + (1 -α ) q 2 can achieve a strictly higher value of the convex combination of two objective functions, unless q 1 = q 2 or α equals either 0 or 1 . Therefore, L τ ( q, λ ) is strictly concave in q ∈ Q for any λ ∈ Λ .

Let a saddle point of L τ ( π, λ ) be π glyph[star] τ = argmax π ∈ Π min λ ∈ Λ L τ ( π, λ ) and λ glyph[star] τ = argmin λ ∈ Λ max π ∈ Π L τ ( π, λ ) . Existence of saddle points in Lemma 14 ensures that,

in which the first inequality implies that for any π ∈ Π ,

and the second inequality implies that for any λ ∈ Λ ,

Combination of two implied inequalities above leads to (5), i.e., ( π glyph[star] τ , λ glyph[star] τ ) is a saddle point of the original Lagrangian V π r + λg ( ρ ) , up to two τ -terms.

C.2 Proof of Theorem 2

Proof. We begin with the standard decomposition of the primal-dual gap,

and we next deal with (i) and (ii), separately. We notice that

Wenote that Q π t r + λ t g + τψ t ( s, a ) is a sum of a soft Q value function associated with a composite function r + λ t g -τ log π t , and -τ log π t . Because of the boundedness of the soft Q value function and the glyph[epsilon1] 0 -restriction on the policy domain ˆ ∆( A ) , | Q π t r + λ t g + τψ t ( s, a ) | ≤ 1 1 -γ (1+ 1 (1 -γ ) ξ + τ log | A | ) -

τ log glyph[epsilon1] 0 | A | := C τ,ξ,glyph[epsilon1] 0 . For the term (i),

where the first two equalities are because of the entropy regularization H ( π ) , we apply the performance difference lemma in Lemma 28 and ψ t ( s, a ) = -log π t ( a | s ) to the third equality, the first inequality is due to an application of Lemma 27 with x glyph[star] = π glyph[star] τ ( · | s ) , x = π t ( · | s ) , g = -Q π t r + λ t g + τψ t ( s, a ) / (1 -γ ) , and η ≤ 1 /C τ,ξ,glyph[epsilon1] 0 .

Similarly, for the term (ii),

where the inequality is due to the standard descent lemma [128] and V π t g ( ρ )+ τλ t ≤ 1 1 -γ (1+ τ ξ ) := C ′ τ,ξ .

Using the definition Φ t := KL t ( ρ ) + 1 2 ( λ glyph[star] τ -λ t ) 2 , we combine the two inequalities above to show that,

where the second inequality is due to Lemma 14. If we expand the inequality above recursively, then,

which completes the proof.

C.3 Proof of Corollary 3

Proof. According to Theorem 2, if we take τ = Θ( glyph[epsilon1] ) , η = Θ( glyph[epsilon1] 2 ) , and glyph[epsilon1] 0 = glyph[epsilon1] , then Φ t +1 = O ( glyph[epsilon1] ) for any t = Ω( 1 glyph[epsilon1] 3 log 1 glyph[epsilon1] ) , where Ω( · ) hides some problem-dependent constant. We next consider a primal-dual iterate ( π t , λ t ) for some t = Ω( 1 glyph[epsilon1] 3 log 1 glyph[epsilon1] ) . It is straightforward to check that

First, we have

For the term (ii), because KL t ( ρ ) = O ( glyph[epsilon1] ) , we have

where we use Cauchy-Schwarz inequality in the first and third inequalities, and the second inequality is due to Pinsker's inequality, which shows V π glyph[star] τ r ( ρ ) -V π t r ( ρ ) ≤ O ( √ glyph[epsilon1] ) . For the term (i), if we take π = π glyph[star] in (5), then,

Meanwhile, if we take λ = 0 in (5), then λ glyph[star] τ V π glyph[star] τ g ( ρ ) ≤ 0 . We notice that the feasibility of π glyph[star] yields V π glyph[star] g ( ρ ) ≥ 0 . Hence,

We now substitute the upper bounds of (i) and (ii) above into (20) to obtain V π glyph[star] r ( ρ ) -V π t r ( ρ ) ≤ O ( √ glyph[epsilon1] ) , where we take τ = Θ( glyph[epsilon1] ) .

Second, we have

Similar to bounding V π glyph[star] τ r ( ρ ) -V π t r ( ρ ) , we can show that (iv) ≤ O ( √ glyph[epsilon1] ) . Let λ max := 1 (1 -γ ) ξ . For the term (iii), if we take λ = λ max in (5), then,

By the definition λ glyph[star] τ := argmin λ ∈ Λ { λV π glyph[star] τ g ( ρ ) + τ 2 λ 2 } , there are three values for λ glyph[star] τ to take: -V π glyph[star] τ g ( ρ ) τ , or 0 , or λ max as follows

When -V π glyph[star] τ g ( ρ ) τ ≤ 0 , λ glyph[star] τ = 0 which shows that λ max -λ glyph[star] τ = λ max > 0 ;

Meanwhile, for any saddle point ( π glyph[star] , λ glyph[star] ) ∈ Π glyph[star] × Λ glyph[star] , V π glyph[star] τ r ( ρ ) -V π glyph[star] r ( ρ ) ≤ λ glyph[star] ( V π glyph[star] g ( ρ ) -V π glyph[star] τ g ( ρ )) , which in conjunction with (22) and V π glyph[star] g ( ρ ) ≥ 0 yields,

It is easy to see that we can always take λ glyph[star] < λ max . In fact, except for λ glyph[star] = 0 , we know that λ glyph[star] V π glyph[star] g ( ρ ) ≤ 0 leads to V π glyph[star] g ( ρ ) = 0 . By the definition λ glyph[star] ∈ argmin λ ∈ Λ V π glyph[star] r + λg ( ρ ) , any λ glyph[star] < λ max is a min-max point. Therefore,

By combining three cases above, we conclude that (iii) ≤ O ( τ ) = O ( glyph[epsilon1] ) .

We now substitute the upper bounds of (iii) and (iv) above into (21) to obtain -V π t ( ρ ) ≤ O ( glyph[epsilon1] ) .

Finally, we replace the accuracy glyph[epsilon1] by glyph[epsilon1] and combine all big O notation to conclude the proof.

C.4 Zero constraint violation of RPG-PD (6)

Corollary 15 (Zero constraint violation) . Let Assumption 1 hold. For small glyph[epsilon1] , there exists δ > 0 such that if we instead use the conservative constraint V π g ′ ( ρ ) ≥ 0 for g ′ = g -(1 -γ ) δ , and take η = Θ( glyph[epsilon1] 4 ) , τ = Θ( glyph[epsilon1] 2 ) , and glyph[epsilon1] 0 = glyph[epsilon1] , then the policy iterates of RPG-PD (6) satisfy

where Ω( · ) only has some problem-dependent constant.

Proof. We apply the conservatism to the translated constraint V π g ( ρ ) ≥ 0 in Problem (1). Specifically, for any δ < min( ξ, 1) , we let g ′ := g -(1 -γ ) δ and define a conservative constraint,

It is straightforward to see that Assumption 1 ensures that V π t g ′ ( ρ ) ≥ 0 is strictly feasible for a new slack variable ξ ′ := ξ -δ . We now can apply RPG-PD (6) to a new regularized Lagrangian,

and Corollary 3 holds if we replace g in RPG-PD by g ′ . Thus,

where Ω( · ) hides some problem-dependent constant, and π glyph[star] δ is an optimal policy to the δ -perturbed constrained policy optimization problem,

We notice that the above Ω( · ) has ξ ′ -dependence and we denote it by Ξ( ξ ′ ) , where Ξ : R + → R + is a positive function. Thus, we select δ such that δ ≥ glyph[epsilon1] Ξ( ξ ′ ) , which is always possible for small enough glyph[epsilon1] , for instance, δ = ξ 2 and ξ ′ = ξ 2 . Hence, if we take δ = ξ 2 and such small glyph[epsilon1] , then,

which shows that V π t g ( ρ ) ≥ 0 for some large t .

The rest is to show that V π glyph[star] r ( ρ ) -V π t r ( ρ ) ≤ O ( glyph[epsilon1] ) . We notice that π glyph[star] is an optimal policy to Problem (23) when δ = 0 . Let q glyph[star] and q glyph[star] δ be associated occupancy measures of policies π glyph[star] and π glyph[star] δ . In the occupancy measure space, Problem (23) becomes a linear program and it has a solution q glyph[star] δ . Thus, we can view q glyph[star] δ as a δ -perturbed solution of a convex optimization problem in which all functions are continuously differentiable and the domain is convex and compact. It is known from [129, Theorem 3.1] that the optimal solution q glyph[star] δ is continuous in δ , which implies that for any glyph[epsilon1] > 0 , there exists δ ′ > 0 such that |〈 r, q glyph[star] 〉 - 〈 r, q glyph[star] δ 〉| ≤ O ( glyph[epsilon1] ) for any δ < δ ′ . Thus, | V π glyph[star] r ( ρ ) -V π glyph[star] δ r ( ρ ) | ≤ O ( glyph[epsilon1] ) for small enough glyph[epsilon1] . Therefore,

for some large t . Collecting all conditions on δ leads to our final choice of δ = min( ξ 2 , 1 , δ ′ ) . Finally, we combine all big O notation to complete the proof.

C.5 Reduction of RPG-PD (6) as a NPG variant (7)

We introduce some useful notation in the regularized MDP [52]. Let V π τ ( ρ ) := V π r + λg ( ρ ) + τ H ( π ) . We introduce the softQ value function Q π τ : S × A → R and V π τ : S → R via Bellman equations,

We also define A π τ ( s, a ) := Q π τ ( s, a ) -τ log π ( a | s ) -V π τ ( s ) . Hence,

where ψ ( s, a ) := -log π ( a | s ) for all ( s, a ) ∈ S × A .

Setting glyph[epsilon1] 0 = 0 , it is easy to show that RPG-PD (6) is a case of (7) in the tabular case by introducing the softmax policy that is widely used in policy optimization. A softmax policy π θ : S → ∆( A ) is parametrized by a parameter θ ∈ R | S || A | via a softmax function,

With a slight abuse of notation, we also use notation π θ as a vector in R | S || A | .

Lemma16. Set glyph[epsilon1] 0 = 0 . Under the softmax policy parametrization, RPG-PD (6) is equivalent to (7) .

Proof. Dual update (7b) is straightforward. We next show the equivalence for the primal update by applying the softmax function to both sides of Primal update (7a).

We first notice that F ρ ( θ t ) † · ∇ θ L τ ( π θ t , λ t ) = F ρ ( θ t ) † · ∇ θ V π θ τ ( ρ ) . Thus,

where c ( s ) is an action-independent constant, and the second equality is the property of natural policy gradient (e.g., Lemma 29). Hence, after normalization over actions and some re-arrangement, we have

which is an explicit form of the policy update in (6). Since the above derivation holds in both directions, the proof is complete.

C.6 Proof of Theorem 4

Proof. We utilize the decomposition of the primal-dual gap as in (19) and analyze the term (i) and the term (ii), separately.

For the term (i), we have

where the first two equalities are because of the entropy regularization H ( π ) , we apply performance difference lemma in Lemma 28 and ψ t ( s, a ) = -log π θ t ( a | s ) to the third equality, the inequality is due to an application of Lemma 27 with x glyph[star] = π glyph[star] τ ( · | s ) , x = π θ t ( · | s ) , g = -φ glyph[latticetop] s,a w t , and η ≤ 1 /C W , where | φ glyph[latticetop] s,a w t / (1 -γ ) | ≤ 2 W/ (1 -γ ) := C W . Moreover, the cross term has the following decomposition,

We now deal with the four terms (a), (b), (c), and (d), separately. For the term (a),

where we recall the definition of E Q ( w glyph[star] t , θ t , ν glyph[star] ) . Similarly, we can bound the term (d) by | (d) | ≤ √ | A |E Q ( w glyph[star] t , θ t , ν glyph[star] ) . For the term (b),

where we recall the definition of κ ν to obtain the last line. Similarly, we can bound the term (c) by | (c) | ≤ √ | A | ‖ w t -w glyph[star] t ‖ 2 Σ ν glyph[star] . Moreover, the optimality of w glyph[star] t ∈ argmin w ∈ R d E Q ( w,θ t , d t,ν ) yields,

which further implies that for any ‖ w ‖ ≤ W ,

Therefore,

With a similar reasoning, we can bound the term (c) by

By applying the upper bounds above to the cross term, and then taking expectation over the randomness in w t , we have

Similarly, for the term (ii),

where the inequality is due to the standard descent lemma [128] and V π θ t g ( ρ )+ τλ t ≤ 1 1 -γ (1+ τ ξ ) := C ′ τ,ξ . By taking expectation over the randomness in w t ,

Using the definition E [ Φ t ] := E [ KL t ( ρ ) ] + 1 2 E [ ( λ glyph[star] τ -λ t ) 2 ] , we combine the two inequalities above to show that,

where the second inequality is due to Lemma 14. If we expand the inequality above recursively, then,

which completes the proof.

C.7 Proof of Corollary 5

Proof. The proof is similar to the proof of Corollary 3, except that we take the expectation over the randomness of computing w t via a sample-based algorithm.

According to Theorem 4 and glyph[epsilon1] stat = O ( glyph[epsilon1] 4 ) , glyph[epsilon1] stat = O ( glyph[epsilon1] 4 ) , if we take τ = Θ( glyph[epsilon1] ) and η = Θ( glyph[epsilon1] 2 ) , then E [Φ t +1 ] = O ( glyph[epsilon1] ) for any t = Ω ( 1 glyph[epsilon1] 3 log 1 glyph[epsilon1] ) . We next consider a primal-dual iterate ( π θ t , λ t ) for some t = Ω ( 1 glyph[epsilon1] 3 log 1 glyph[epsilon1] ) . It is straightforward to check that

First, we have

For the term (ii), because of KL t ( ρ ) = O ( glyph[epsilon1] ) , we have

where we use Cauchy-Schwarz inequality in the first and third inequalities, and the second inequality is due to Pinsker's inequality and Jensen's inequality, which shows E [ V π glyph[star] τ r ( ρ ) -V π θ t r ( ρ ) ] ≤ O ( √ glyph[epsilon1] ) . For the term (i), if we take π = π glyph[star] in (5), then,

Meanwhile, if we take λ = 0 in (5), then λ glyph[star] τ V π glyph[star] τ g ( ρ ) ≤ 0 . We notice that the feasibility of π glyph[star] yields V π glyph[star] g ( ρ ) ≥ 0 . Hence,

We now substitute the upper bounds of (i) and (ii) into (24) to obtain E [ V π glyph[star] r ( ρ ) -V π θ t r ( ρ ) ] ≤ O ( √ glyph[epsilon1] ) , where we take τ = Θ( glyph[epsilon1] ) .

Second, we have

Similar to bounding E [ V π glyph[star] τ r ( ρ ) -V π θ t r ( ρ ) ] , we can show that E [ (iv) ] ≤ O ( √ glyph[epsilon1] ) . For the term (iii), we can show that (iii) ≤ O ( τ ) = O ( glyph[epsilon1] ) in a similar way as dealing with (iii) in (21).

We now substitute the upper bounds of (iii) and (iv) above into (25) to obtain E [ -V π θ t g ( ρ ) ] ≤ O ( √ glyph[epsilon1] ) .

Finally, we replace the accuracy √ glyph[epsilon1] by glyph[epsilon1] and combine all big O notation to conclude the proof.

C.8 Zero constraint violation of inexact RPG-PD

Corollary 17 (Zero constraint violation) . Let Assumptions 1-2 hold and glyph[epsilon1] stat , glyph[epsilon1] bias = O ( glyph[epsilon1] 8 ) for small glyph[epsilon1] , glyph[epsilon1] 0 > 0 . For small glyph[epsilon1] , there exists δ > 0 such that if we instead use the conservative constraint V π g ′ ( ρ ) ≥ 0 for g ′ = g -(1 -γ ) δ , and take the stepsize η = Θ( glyph[epsilon1] 4 ) and τ = Θ( glyph[epsilon1] 2 ) , then the policy iterates of inexact RPG-PD satisfy

where Ω( · ) only has some problem-dependent constant.

Proof. We apply the conservatism to the translated constraint V π g ( ρ ) ≥ 0 in Problem (1). Specifically, for any δ < min( ξ, 1) , we let g ′ := g -(1 -γ ) δ and define a conservative constraint,

It is straightforward to see that Assumption 1 ensures that V π t g ′ ( ρ ) ≥ 0 is strictly feasible for a new slack variable ξ ′ := ξ -δ . We now can apply inexact RPG-PD (8) to a new regularized Lagrangian,

and Corollary 5 holds if we replace g in inexact RPG-PD by g ′ and glyph[epsilon1] stat , glyph[epsilon1] bias = O ( glyph[epsilon1] 8 ) for small glyph[epsilon1] > 0 . Thus,

where Ω( · ) has some problem-dependent constant, and π glyph[star] δ is an optimal policy to the δ -perturbed constrained policy optimization problem,

We notice that the above Ω( · ) has ξ ′ -dependence and we denote it by Ξ( ξ ′ ) , where Ξ : R + → R + is a positive function. Thus, we select δ such that δ ≥ glyph[epsilon1] Ξ( ξ ′ ) , which is always possible for small enough glyph[epsilon1] , for instance, δ = ξ 2 and ξ ′ = ξ 2 . Hence, if we take δ = ξ 2 and such small glyph[epsilon1] , then

which shows that E [ V π θ t g ( ρ ) ] ≥ 0 for some large t .

The rest is to show that E [ V π glyph[star] r ( ρ ) -V π θ t r ( ρ ) ] ≤ O ( glyph[epsilon1] ) . We notice that π glyph[star] is an optimal policy to Problem (26) when δ = 0 . Let q glyph[star] and q glyph[star] δ be associated occupancy measures of policies π glyph[star] and π glyph[star] δ . In the occupancy measure space, Problem (26) becomes a linear program and it has a solution q glyph[star] δ . Thus, we can view q glyph[star] δ as a δ -perturbed solution of a convex optimization problem in which all functions are continuous differentiable and the domain is convex and compact. It is known from [129, Theorem 3.1] that the optimal solution q glyph[star] δ is continuous in δ , which implies that for any glyph[epsilon1] > 0 , there exists δ ′ such that |〈 r, q glyph[star] 〉 - 〈 r, q glyph[star] δ 〉| ≤ O ( glyph[epsilon1] ) for any δ < δ ′ . Thus, | V π glyph[star] r ( ρ ) -V π glyph[star] δ r ( ρ ) | ≤ O ( glyph[epsilon1] ) for small enough glyph[epsilon1] . Therefore,

for some large t . Collecting all conditions on δ leads to our final choice of δ = min( ξ 2 , 1 , δ ′ ) . Finally, we combine all big O notation to complete the proof.

C.9 Sample-based inexact RPG-PD algorithm

We generalize the inexact RPG-PD to be a sample-based algorithm that only takes sample-based estimates. We propose a sample-based RPG-PD with linear function approximation as follows,

where ˆ w t and ˆ V π t g ( ρ ) are the sample-based estimates of NPG directions and value functions. It is standard to assume that there exists a policy simulator that generates policy rollouts for any given policies [49]. At time t , we can estimate ˆ w t by solving the regression problem E Q ( w,θ t , d t,ν ) = E ( s,a ) ∼ d t,ν [ ( φ glyph[latticetop] s,a w -Q π θ t ( s, a )) 2 ] with Q π θ t ( s, a ) = Q π θ t r + λ t g + τψ t ( s, a ) via a projected stochastic gradient descent (SGD) method,

where k ≥ 0 counts the number of SGD iterations, and G k t is a sample-based estimate of the population gradient ∇ w E Q ( w,θ t , d t,ν ) ,

From the projected SGD result [130], we use a weighted average 2 K ( K +1) ∑ K -1 k =0 ( k +1) w k t as our ˆ w t . We note that Q π θ t r + λ t g + τψ t ( s, a ) is a sum of a softQ value function associated with a composite function r + λ t g -τ log π θ t , and -τ log π θ t . Thus, we can estimate the value function ˆ Q π θ t r + λ t g + τψ t ( s, a ) using policy rollouts in Algorithm 2, which provides an unbiased estimate and it has bounded variance [131],

where the expectation E is taken over the randomness of drawing ( s, a ) ∼ d t,ν . Another value function ˆ V π θ t g ( ρ ) can be estimated using policy rollouts in Algorithm 3, E [ ˆ V π θ t g ( ρ ) ] = V π θ t g ( ρ ) , which is also unbiased and has bounded variance [23]. Hence, simply replacing all population quantities in inexact RPG-PD by their sample-based estimates leads to a sample-based inexact RPGPD that is detailed in Algorithm 1.

We are now ready to establish the sample complexity of Algorithm 1 by exploiting the projected SGD result [130].

Corollary 18 (Sample complexity of inexact RPG-PD) . Let Assumptions 1-2 hold. Assume Σ ν = E ( s,a ) ∼ ν [ φ s,a φ glyph[latticetop] s,a ] ≥ κ 0 I for any state-action distribution ν and some κ 0 > 0 . If we take the

Algorithm 1 Sample-based inexact RPG-PD algorithm with log-linear policy parametrization

1: Input : Learning rate η , number of SGD iterations K , SGD learning rate α .

2: Initialize θ 0 = 0 , λ 0 = 0 ,

3: for t = 0 , . . . , T -1 do

4: Initialize w 0 t = 0 .

5: for k = 0 , 1 , . . . , K -1 do

6: Estimate ˆ Q π θ t r + λ t g + τψ t ( s, a ) for some ( s, a ) ∼ d t,ν , using Algorithm 2 with policy π θ t .

7: Perform projected SGD step with α k = 2 κ 0 ( k +2) ,

8: end for

9: Set ˆ w t = 2 K ( K +1) ∑ K -1 k =0 ( k +1) w k t .

10: Estimate ˆ V π θ t g ( ρ ) using Algorithm 3 with policy π θ t .

11: Perform inexact RPG-PD update,

12: end for

Algorithm 2 Unbiased estimate Q

1: Input : Initial state-action distribution ν , policy π , dual variable λ , regularization parameter τ , discount factor γ .

2: Sample ( s 0 , a 0 ) ∼ ν , execute the policy π with probability γ at each step h ; otherwise, accept ( s h , a h ) as the sample.

3: Start with ( s h , a h ) , execute the policy π with the termination probability 1 - √ γ . Once terminated, add all composite values γ ( k -h ) / 2 ( r + λg + τψ ) from step k = h +1 onwards and -τ log π ( a h | s h ) as ˆ Q π r + λg + τψ ( s h , a h ) .

4: Output : ( s h , a h ) and ˆ Q π r + λg + τψ ( s h , a h ) .

Algorithm 3 Unbiased estimate V

1: Input : Initial state distribution ρ , policy π , discount factor γ .

2: Sample s 0 ∼ ρ , execute the policy π with the termination probability 1 -γ . Once terminated, add all utilities up as ˆ V π g ( ρ ) .

3: Output : ˆ V π g ( ρ ) .

stepsize η ≤ 1 /C W , then the primal-dual iterates of sample-based inexact RPG-PD in Algorithm 1 satisfy

Proof. The proof is based on the proof of Theorem 4. Additionally, we have to take the randomness of sample-based estimates into account and bound the statistical error glyph[epsilon1] stat using the projected SGD result [130]. We first check all conditions of the projected SGD [130] are indeed satisfied by the SGD step in Algorithm 1: (i) The domain ‖ w ‖ ≤ W is convex and bounded; (ii) The gradient G k t is an unbiased estimate of the population gradient; (iii) The minimizer of E Q ( w,θ t , d t,ν ) is unique, since Σ d t,ν ≥ κ 0 I for some κ 0 > 0 ; (iv) The squared norm of the estimated gradient G k t is bounded or the gradient has bounded variance. To show (iv), it is sufficient to check that

where we use the boundedness of ∥ ∥ w k t ∥ ∥ ≤ W and ‖ φ s,a ‖ ≤ 1 . We notice that | r + λ t g | ≤ 2 (1 -γ ) ξ . By [131, Lemma 3.5], we know that

where the first two terms in the upper bound is due to the variance of softQ value function, and the last term is due to | log π θ t | ≤ | log glyph[epsilon1] 0 | A | | for π θ t ∈ ˆ ∆( A ) . Hence, the estimated gradient G k t has bounded second-order moment (or variance), which verifies (iv). From the projected SGD result [130], if the SGD stepsize α k = 2 k +2 , then

which leads to √ glyph[epsilon1] stat ≤ 4( W +2 / ( ξ (1 - √ γ ) 2 )+ τ (2 log | A | + | log glyph[epsilon1] 0 | ) / ((1 - √ γ ) 2 ξ )) √ κ 0 ( K +1) . Substituting the bound of √ glyph[epsilon1] stat into the proof of Theorem 4 yields our desired result.

Corollary 18 states a similar result as Theorem 4. The effect of using sample-based estimates to update inexact RPG-PD appears as the number K of SGD steps at each time t . Thus, we can interpret the iteration complexity in Corollary 5 in terms of the number of SGD steps by taking glyph[epsilon1] stat = O ( glyph[epsilon1] 8 ) , i.e., K = Ω( 1 glyph[epsilon1] 8 ) . Thus, whenever glyph[epsilon1] bias = O ( glyph[epsilon1] 8 ) for small glyph[epsilon1] > 0 , if we take ( η, τ ) in Corollary 5 and K = Ω( 1 glyph[epsilon1] 8 ) for Algorithm 1, then,

where Ω( · ) only has some problem-dependent constant. In other words, the total number of policy rollouts or sampled trajectories tK = Ω ( 1 glyph[epsilon1] 14 ) is required for Algorithm 1 to output an glyph[epsilon1] -optimal constrained policy. Furthermore, the zero constraint violation in Corollary 17 can be interpreted similarly. When glyph[epsilon1] is small enough, we can design a conservative constraint such that the policy iterates of Algorithm 1 satisfy V π glyph[star] r ( ρ ) -V π t r ( ρ ) ≤ glyph[epsilon1] and V π θ t g ( ρ ) ≥ 0 for some t , K that satisfy tK = Ω ( 1 glyph[epsilon1] 14 ) . This appears to be the first sample-based zero constraint violation result for constrained MDPs in the function approximation setting. We leave achieving the optimal sample complexity as future work.

D Proofs in Section 4

In this section, we provide proofs for the claims in Section 4.

D.1 Preliminary last-iterate analysis of OPG-PD (9)

To measure the proximity of the primal-dual iterates of OPG-PD (9) to the optimal pair ( π glyph[star] , λ glyph[star] ) , we introduce the following two distance metrics Θ t and ζ t at time t ≥ 1 ,

and a problem-dependent constant ι ,

Lemma 19. Let the optimal state visitation distribution be unique, i.e., d π ρ = d π glyph[star] ρ for any π ∈ Π glyph[star] . If we set the stepsize η ≤ 1 / (4 √ ι ) , then the primal-dual iterates of OPG-PD (9) satisfy

Proof. We begin with the standard decomposition of the primal-dual gap at time t ≥ 1 ,

and we next deal with (i) and (ii), separately.

For the first term (i),

where the first equality is due to performance difference lemma in Lemma 28, the second equality is because of some re-arrangement, the inequality is from an application of Lemma 24 to ˆ π t +1 ( · | s ) and π t ( · | s ) :

Similarly, we deal with the second term (ii) with some re-arrangement, and apply Lemma 24 to ˆ λ t +1 and λ t ,

On the other hand, from Lemma 25, we have

and

Hence,

where the first inequality is due to Cauchy-Schwarz inequality and (33), we subtract and add Q π t r + λ t -1 g and apply the inequality ( x + y ) 2 ≤ 2 x 2 +2 y 2 in the second inequality, the third inequality

is due to Lemma 22 and the boundedness of value functions and dual variables, the fourth inequality comes from the property of κ ρ , and the last inequality is due to the inequality ( x + y ) 2 ≤ 2 x 2 +2 y 2 and the inequality ‖ p -p ′ ‖ 1 ≤ √ | A | ‖ p -p ′ ‖ 2 for two probability distributions p and p ′ . Meanwhile, using a similar reasoning, we can derive that

where the first inequality is due to Cauchy-Schwarz inequality and (34), the second inequality is due to Lemma 22, the third inequality is an application of Cauchy-Schwarz inequality, and the last inequality is due to the inequality ( x + y ) 2 ≤ 2 x 2 + 2 y 2 and the inequality ‖ p -p ′ ‖ 1 ≤ √ | A | ‖ p -p ′ ‖ 2 for two probability distributions p and p ′ .

We set notation,

After applying the established inequalities above to (31) and (32), we combine them into (30) as,

We notice that V π glyph[star] r + λ t g ( ρ ) -V π t r + λ glyph[star] g ( ρ ) ≥ 0 and non-expansiveness of projection operators P Π glyph[star] , P Λ glyph[star] ,

for any π glyph[star] ∈ Π glyph[star] and λ glyph[star] ∈ Λ glyph[star] . If we take π glyph[star] = P Π glyph[star] (ˆ π t ( · | s )) and λ glyph[star] = P Λ glyph[star] ( ˆ λ t ) , then after some re-arrangement, we have

where we also use the assumption d π glyph[star] ρ = d π ρ for any π ∈ Π glyph[star] . By taking η 2 ι ≤ 1 16 and using notation Θ t and ζ t , we have Θ t +1 ≤ Θ t -7 16 ζ t .

To show the convergence, we next relate ζ t with Θ t +1 . An intermediate step is to show that ζ t has the following lower bound.

Lemma 20. Let κ ρ,γ := max( κ ρ 1 -γ , 1) . If we set the stepsize η ≤ min( (1 -γ ) 3 4 | A | , (1 -γ ) 3 2 κ ρ ) , then the primal-dual iterates of OPG-PD (9) satisfy glyph[negationslash]

Proof. From the optimality of ˆ π t +1 ( · | s ) in OPG-PD, we know that for any π ∈ Π ,

Similarly, from the optimality of π t +1 ( · | s ) in OPG-PD, we know that for any π ∈ Π ,

On the other hand, from the optimality of ˆ λ t +1 in OPG-PD, we know that for any λ ∈ Λ ,

and the optimality of λ t +1 in OPG-PD yields,

First, we take the expectation of (35a) over some state distribution d π ρ on both sides and add it to (36a),

where the second inequality is due to performance difference lemma in Lemma 28 and CauchySchwarz inequality, and the last inequality results from Lemma 22 and the inequality ‖ p -p ′ ‖ 1 ≤ √ | A | ‖ p -p ′ ‖ 2 for two probability distributions p and p ′ . By using the inequality ac + bd ≤

where the second inequality comes from the property of κ ρ and the last inequality is straightforward from (37). We take η > 0 such that max ( 2 η | A | (1 -γ ) 3 , ηκ ρ (1 -γ ) 3 ) ≤ 1 2 and denote κ ρ,γ := max ( κ ρ 1 -γ , 1 ) . If we take the square of both sides of the inequality above, then the second product argument has the following upper bound,

where we use the inequality ( x + y ) 2 ≤ 2 x 2 +2 y 2 and Jensen's inequality.

Recall the definition of κ ρ and κ ρ ≤ 1 /ρ min , where ρ min := min s ρ ( s ) .

Lemma 21. Assume ρ min > 0 . For any t ≥ 1 , the primal-dual iterates of OPG-PD (9) satisfy

where C ρ,ξ := cρ min / (2 √ | S || A | ) / (1 + 1 / ((1 -γ ) ξ )) in which c > 0 is described in Lemma 26.

Proof. Denote V glyph[star] := V π glyph[star] r + λ glyph[star] g ( ρ ) and D max := max π,π ′ ∈ Π ,λ,λ ′ ∈ Λ (max s ‖ π ( · | s ) -π ′ ( · | s ) ‖ + | λ -λ ′ | ) . We observe that if we can prove that there exist constants c 1 , c 2 > 0 such that

then,

which proves the lemma by taking C ρ,ξ := min( c 1 ,c 2 ) D max .

We next prove (38) using the bilinear game result in Lemma 26. By the linear program formulation of constrained MDP, we can express the value function in terms of the occupancy measure q π , e.g., V π r + λg ( ρ ) = 〈 q π , r + λg 〉 , where q π is the occupancy measure that lives in a polytope Q that is given by (15). Hence, the constrained saddle-point problem (3) reduces to,

We notice that the game value keeps the same as V glyph[star] that is achieved at ( q π glyph[star] , λ glyph[star] ) , where q π glyph[star] is the occupancy measure under the policy π glyph[star] . Let the set of occupancy measures associated with Π glyph[star] be Q glyph[star] . According to Lemma 26, we know that there exists a constant c > 0 such that

It is more straightforward to see (38a) from (39a) if we take λ = ˆ λ t +1 and c 1 = c . We next show (38b) using (39b) by taking π glyph[star] ( · | s ) to be the policy associated with P Q glyph[star] ( q π ) ,

where the first inequality is due to triangle inequality, we use the fact: (1 -γ ) q π glyph[star] ( s ) = d π glyph[star] ρ ( s ) , d π glyph[star] ρ ( s ) ≤ 1 , and ∥ ∥ q π glyph[star] ( s, · ) ∥ ∥ ≤ 1 in the second inequality, the third inequality is due to that q π ( s ) ≥ ρ min , and we apply Cauchy-Schwarz inequality in the last inequality. Hence, we can further lower bound (39b),

which yields (38b) using c 2 = cρ min 2 √ | S || A | .

Finally, we combine all selected constants and take D max = 1 + 1 (1 -γ ) ξ to conclude the proof.

Lemma 22. For any policies π and π ′ , we have

Proof. By the Bellman equations, for each pair ( s, a ) ,

Hence, for each pair ( s, a ) ,

which yields the first inequality.

To show the second inequality, we employ the performance difference lemma to show that,

where we replace d π ρ by d π glyph[star] ρ using the inequality ∥ ∥ d π ρ /d π glyph[star] ρ ∥ ∥ ∞ ≤ κ ρ / (1 -γ ) for any policy π ∈ Π .

D.2 Proof of Theorem 6

Proof. From the non-increasing relation (29) in Lemma 19, we have

Meanwhile, from Lemma 20, we obtain that

where the third inequality is due to Lemma 21, the inequalities: ( x + y ) 2 ≥ x 2 + y 2 for x , y ≥ 0 , and d π glyph[star] ρ ( s ) ≥ (1 -γ ) ρ min , and C ′ ρ,ξ := (1 -γ ) C 2 ρ,ξ ρ min . Hence, the relation (29) reduces into,

which implies that Θ t is decreasing to zero at an exponential rate.

D.3 Proof of Corollary 7

Proof. According to Theorem 6, if we take the same stepsize η , then for any t = Ω(log 1 glyph[epsilon1] ) ,

Let ˆ π glyph[star] t ( · | s ) := P Π glyph[star] (ˆ π t ( · | s )) and ˆ λ glyph[star] t := P Λ glyph[star] ( ˆ λ t ) . Because of the interchangeability of saddle points from Lemma 9, (ˆ π glyph[star] t , ˆ λ glyph[star] t ) is a saddle point in the set Π glyph[star] × Λ glyph[star] for any t ≥ 0 . First, we have

where we use Cauchy-Schwarz inequality in the first and third inequalities, and the second inequality is due to ‖ x ‖ 1 ≤ √ d ‖ x ‖ 2 for x ∈ R d , which shows V ˆ π glyph[star] t r ( ρ ) -V ˆ π t r ( ρ ) ≤ O ( √ glyph[epsilon1] ) . By the saddle-point property of (ˆ π glyph[star] t , ˆ λ glyph[star] t ) , ˆ π glyph[star] t is also an optimal constrained policy, i.e., V ˆ π glyph[star] t r ( ρ ) = V π glyph[star] r ( ρ ) . Therefore, V π glyph[star] r ( ρ ) -V ˆ π t r ( ρ ) ≤ O ( √ glyph[epsilon1] ) .

Second, we have

Similar to bounding V ˆ π glyph[star] t r ( ρ ) -V ˆ π t r ( ρ ) , we can show that (ii) ≤ O ( √ glyph[epsilon1] ) . By the saddle-point property of (ˆ π glyph[star] t , ˆ λ glyph[star] t ) , V ˆ π glyph[star] t g ( ρ ) ≥ 0 . Therefore, -V ˆ π t g ( ρ ) ≤ O ( √ glyph[epsilon1] ) .

Finally, we replace the accuracy √ glyph[epsilon1] by glyph[epsilon1] and combine all big O notation to conclude the proof.

D.4 Zero constraint violation of OPG-PD (9)

Corollary 23 (Zero constraint violation) . Let Assumption 1 hold and the optimal state visitation distribution be unique, i.e., d π ρ = d π glyph[star] ρ for any π ∈ Π glyph[star] , and ρ min > 0 . For small glyph[epsilon1] , there exists δ > 0 such that if we instead use the conservative constraint V π g ′ ( ρ ) ≥ 0 for g ′ = g -(1 -γ ) δ , and take the stepsize η from Theorem 6, then the policy iterates of OPG-PD satisfy

where Ω( · ) only omits some problem-dependent constant.

Proof. We apply the conservatism to the translated constraint V π g ( ρ ) ≥ 0 in Problem (1). Specifically, for any δ < min( ξ, 1) , we let g ′ := g -(1 -γ ) δ and define a conservative constraint,

It is straightforward to see that Assumption 1 ensures that V π t g ′ ( ρ ) ≥ 0 is strictly feasible for a new slack variable ξ ′ := ξ -δ . We now can apply OPG-PD (9) to a new Lagrangian,

and Corollary 7 holds if we replace g in OPG-PD by g ′ . Thus,

where Ω( · ) hides some problem-dependent constant, and π glyph[star] δ is an optimal policy to the δ -perturbed constrained policy optimization problem,

We notice that the above Ω( · ) has some problem-dependent constant Υ > 0 . Thus, we select δ such that δ ≥ glyph[epsilon1] Υ , which is always possible for small enough glyph[epsilon1] , for instance δ = ξ 2 and ξ ′ = ξ 2 . Hence, if we take δ = ξ 2 and such small glyph[epsilon1] , then,

which shows that V ˆ π t g ( ρ ) ≥ 0 for some large t .

The rest is to show that V π glyph[star] r ( ρ ) -V ˆ π t r ( ρ ) ≤ O ( glyph[epsilon1] ) . We notice that π glyph[star] is an optimal policy to Problem (40) when δ = 0 . Let q glyph[star] and q glyph[star] δ be associated occupancy measures of policies π glyph[star] and π glyph[star] δ . In the occupancy measure space, Problem (40) becomes a linear program and it has a solution q glyph[star] δ . Thus, we can view q glyph[star] δ as a δ -perturbed solution of a convex optimization problem in which all functions are continuous differentiable and the domain is convex and compact. It is known from [129, Theorem 3.1] that the optimal solution q glyph[star] δ is continuous in δ , which implies that for any

glyph[epsilon1] > 0 , there exists δ ′ such that |〈 r, q glyph[star] 〉-〈 r, q glyph[star] δ 〉| ≤ O ( glyph[epsilon1] ) for any δ < δ ′ . Thus, | V π glyph[star] r ( ρ ) -V π glyph[star] δ r ( ρ ) | ≤ O ( glyph[epsilon1] ) for small enough glyph[epsilon1] . Therefore,

for some large t . Collecting all conditions on δ leads to our final choice of δ = min( ξ 2 , 1 , δ ′ ) . Finally, we combine all big O notation to complete the proof.

E Other Computational Experiments

In this section, we report details of our experimental setup and additional experimental results that verify merits and effectiveness of our methods: RPG-PD (6) and OPG-PD (9). We implement RPG-PD in the form of NPG [49] and the restricted probability simplex ˆ ∆( A ) by restraining policy parameter to be bounded.

To properly assess the convergence performance, our experiment is a tabular constrained MDP with a randomly generated transition kernel, a discount factor γ = 0 . 9 , uniform rewards r ∈ [0 , 1] and utilities g ∈ [ -1 , 1] , and a uniform initial state distribution ρ . The constraint is V π g ( ρ ) ≥ 0 . To check feasibility, we employ the standard policy iteration procedure to solve a standard MDP problem with respect to V π g ( ρ ) . If feasible, we then solve a linear program in occupancy-measure space to find the optimal reward value V π glyph[star] r ( ρ ) at an optimal policy π glyph[star] induced by the optimal occupancy measure. Throughout all experiments, the random seed is fixed and V π glyph[star] r ( ρ ) takes the value 8 . 16 at an optimal policy π glyph[star] .

We compare our methods RPG-PD and OPG-PD with three typical learning algorithms in the constrained MDP literature: (i) primal-dual methods [23, 29]; (ii) dual methods [28, 24, 34]; and (iii) primal method [77]. We have reported our comparison for primal-dual methods in Section 5 and Figure 1. We now report our comparison experiments on other two baseline methods: (ii) dual methods [28, 24, 34] and (iii) primal method [77], in Section E.1. In addition, we showcase the last-iterate zero constraint violation performance of our methods in Section E.2, and conduct sensitivity analysis of our methods to regularization and stepsize in Section E.3 and Section E.4, together with a variant of OPG-PD and policy-based ReLOAD [40]. All the experiments were conducted on an Apple MacBook Pro 2017 laptop equipped with a 2.3 GHz Dual-Core Intel Core i5 in Jupyter Notebook.

E.1 Last-iterate convergence comparison with other baselines

We report our comparison for dual methods in Figure 4. We notice that dual methods [28, 24, 34] work in a double-loop fashion, where a (regularized) NPG subroutine is executed to perform the dual update. To make a fair comparison, the number of dual updates and the number of NPG steps are set to be 53 and 20 to ensure that dual methods take the same number of policy gradient updates: 1060 , as RPG-PD and OPG-PD, and we evaluate all policy iterates inside the NPG subroutines of dual methods.

In Figure 4, RPG-PD ( - -) and OPG-PD ( -) outperform PMD-PD [24] ( - -), AR-CPO [28] ( -· -), and Accelerated Dual [34] ( ···· ) in several aspects. First, we see that the known oscillation behavior in primal-dual methods also shows up in these dual methods, which could result from that we can't exactly evaluate the search direction of the dual update via a NPG subroutine. Thus, dual methods can be viewed an instantiation of two-time-scale methods in which the primal update performs faster than the dual update. Regarding this, RPG-PD and OPG-PD show outstanding performance in suppressing oscillation behavior. Second, since RPG-PD and OPG-PD are single-time-scale primaldual methods, it is easy to tune algorithmic parameters compared with double-loop dual methods. For instance, in Accelerated Dual, it is relatively difficult to find a set of algorithmic parameters that avoid the sub-optimal performance in Figure 4. Third, OPG-PD reaches the maximum reward value 8.16 and RPG-PD converges to a slightly smaller reward value because of regularization, while both methods enjoy the utility constraint satisfaction in the last-iterate fashion, which seems to be difficult for dual methods because of oscillating utility values. Hence, using the same number of policy gradient updates, we have verified that OPG-PD and RPG-PD also can outperform several dual methods by yielding an optimal constrained policy in the last-iterate fashion.

Reward value

Utility value

Iteration

Iteration

8.0

7.5

7.0

6.5

6.0

5.5

5.0

0

200

400

600

800

1000

3

2

0

0

200

400

600

800

1000

We report our comparison for a primal method in Figure 5. In Figure 5, RPG-PD ( - -) and OPG-PD ( -) outperform CRPO [77] ( -· -). We notice that although CRPO [77] works in a single-time-scale fashion as RPG-PD and OPG-PD, the policy has to be updated by alternatively using the gradient directions of reward/utility value functions. To ensure constraint satisfaction, CRPO often switches between the gradient directions of reward/utility value functions depending on the amount of constraint violation. As a result, we see that CRPO reaches a slightly lower reward value than OPGPD's, and the constraint satisfaction is relatively conservative and has mild oscillation behavior. In contrast, OPG-PD achieves the maximum reward value 8.16 and RPG-PD converges to a slightly smaller reward value because of regularization, while both methods enjoy the utility constraint satisfaction in the last-iterate fashion. Last but not least, we have supported RPG-PD and OPG-PD with a policy last-iterate convergence theory, while such theory is unknown for CRPO as far as we know.

Reward value

Utility value

Iteration

Iteration

8.0

7.5

7.0

6.5

6.0

5.5

5.0-

0

200

400

600

800

1000

1.0

0.5

0.0

0.5

1.0

0

200

400

600

800

1000

E.2 Last-iterate zero constraint violation comparison

In this experiment, we continue our previous tabular constrained MDP with a random transition, a discount factor γ = 0 . 9 , uniform rewards r ∈ [0 , 1] and utilities g ∈ [ -1 , 1] , and an uniform initial state distribution ρ . Instead of the nominal constraint V π g ( ρ ) ≥ 0 , we use a conservative constraint V π g ′ ( ρ ) ≥ 0 in RPG-PD and OPG-PD, where g ′ := g -(1 -γ ) δ is a conservative utility function and δ is the conservative parameter. To get zero constraint violation regarding the nominal constraint, we apply RPG-PD and OPG-PD to the conservative constraint V π g ( ρ ) ≥ δ where we take the conservative parameter δ = 0 . 1 . As above, we compare conservative RPG-PD and OPG-PD with three typical learning algorithms in the constrained MDP literature: (i) primaldual methods [23, 29]; (ii) dual methods [28, 24, 34]; and (iii) primal approach [77]. We report our comparison for primal-dual methods in Figure 6, for dual methods [28, 24, 34] in Figure 7, and for primal approach [77] in Figure 8. We observe that conservative RPG-PD and OPG-PD achieve similar performance regarding the oscillation suppression and the optimality of reward values as shown in Figure 1, Figure 4, and Figure 5. Interestingly, in Figure 6, Figure 7, and Figure 8, RPGPD and OPG-PD converge to a utility value that is strictly above zero, i.e., V π t g ( ρ ) > 0 for large t , which is not guaranteed in many of other methods. To sum up, we have confirmed that RPG-PD and OPG-PD can ensure zero constraint violation of instantaneous policy iterates in a finite number of training time.

Reward value

Utility value

Iteration

Iteration

8.0

7.5

7.0

6.5

6.0

5.5

5.0

0

200

400

600

800

1000

3

0

200

400

600

800

1000

E.3 Sensitivity of RPG-PD (6) to regularization and stepsize

In this experiment, we use our previous tabular constrained MDP with a random transition, a discount factor γ = 0 . 9 , uniform rewards r ∈ [0 , 1] and utilities g ∈ [ -1 , 1] , and an uniform initial state distribution ρ . The constraint is V π g ( ρ ) ≥ 0 . We recall that when we set the regularization parameter τ = 0 , RPG-PD becomes a policy-based primal-dual method [23] that often suffers the oscillation issue as shown in Figure 1. However, larger regularization usually bias regularized methods more to sub-optimal solutions. Hence, it is important to reveal how the last-iterate convergence of RPG-PD depends on the regularization parameter τ and the stepsize η .

We first repeat executing RPG-PD with a fixed stepsize η = 0 . 1 , but varying three different regularization parameters τ ∈ { 0 . 1 , 0 . 05 , 0 . 01 } . In Figure 9, RPG-PD damps initial oscillations successfully for τ = 0 . 1 and 0 . 05 , and oscillates for τ = 0 . 01 . When we decrease τ from 0 . 1 to 0 . 05 , the reward value RPG-PD converges to becomes higher, and the utility value's oscillation is slightly amplified initially, but damped eventually. When τ is further reduced to 0 . 01 , although the reward value reaches a value around the optimal reward value 8 . 16 , the utility value behaves oscillating.

Reward value

Utility value

Iteration

Iteration

8.0

7.5

7.0

6.5

6.0

5.5

5.0

0

200

400

600

800

1000

3

A

0

0

200

400

600

800

1000

Reward value

Utility value

Iteration

Iteration

8.0

7.5

7.0

6.5

6.0

5.5

5.0

0

200

400

600

800

1000

1.0

0.5

0.0

0.5

1.0

0

200

400

600

800

1000

A reason for this is that RPG-PD with a relatively small regularization parameter (compared to the stepsize) works as usual un-regularized single-time-scale primal-dual methods. Hence, increasing the regularization parameter, τ = 0 . 1 can accelerate the convergence and attenuate the oscillation more effectively, although it makes the reward value more sub-optimal, which is also clearly shown in Figure 10.

To demonstrate the convergence of RPG-PD's policy iterates, we measure the policy optimality gap via the squared norm distance of policy iterates to an optimal policy π glyph[star] that is obtained from an occupancy-measure-based linear program. From the optimality gap in Figure 10, we see that three policy optimality gaps decrease sublinearly to some constants in the logarithmic scale plot, which verifies the sublinear last-iterate convergence of RPG-PD's policy iterates in Theorem 2. Hence, we

conjecture that it is impossible to improve the order of RPG-PD's sublinear rate, without introducing new algorithmic design.

To reduce the oscillation behavior, we repeat this experiment with a smaller stepsize η = 0 . 01 as suggested by Corollary 3. We report our result in Figure 11 and Figure 12. The oscillation in the utility value previously happened for τ = 0 . 01 becomes less frequent, and the best reward value is achieved in this case. A noticeable loss is that the convergence has been slowed down, apparently in Figure 12. Hence, we have shown that balancing the stepsize and the regularization parameter (e.g., Corollary 3) is important for RPG-PD to achieve better oscillation attenuation and last-iterate convergence in practice.

Reward value

Utility value

Iteration

Iteration

8.0

7.5

7.0

6.5

6.0

5.5

5.0

0

250

500

750

10001250150017502000

1.0

0.5

0.0

0.5

1.0

0

250

500

750

10001250150017502000

Policy optimality gap

Iteration

101

100

0

250

500

750

1000

1250

1500

1750

2000

Reward value

Utility value

8.0

7.5

7.0

6.5

6.0

5.5

5.0

0

250

500

750

10001250150017502000

1.0

0.5

0.0

0.5

1.0

0

250

500

750

10001250150017502000

Iteration

Iteration

Policy optimality gap

Iteration

101

6×100

4×100

3×100

0

250

500

750

1000

1250

1500

1750

2000

E.4 Sensitivity of OPG-PD (9) to stepsize

In this experiment, we use our previous tabular constrained MDP with a random transition, a discount factor γ = 0 . 9 , uniform rewards r ∈ [0 , 1] and utilities g ∈ [ -1 , 1] , and an uniform initial state distribution ρ . The constraint is V π g ( ρ ) ≥ 0 . We repeat executing OPG-PD by varying three different stepsizes η ∈ { 0 . 05 , 0 . 1 , 0 . 2 } . To demonstrate the optimality of OPG-PD's policy iterates, we measure the policy optimality gap via the squared norm distance of policy iterates to an optimal policy π glyph[star] that is obtained from an occupancy-measure-based linear program. To demonstrate the optimality of OPG-PD's policy iterates, we measure the policy optimality gap via the squared norm distance of policy iterates to an optimal policy π glyph[star] that is obtained from an occupancy-measure-based linear program. From the policy optimality gap in Figure 2, we see that three policy optimality gaps decrease linearly in the logarithmic scale plot, which verifies the linear last-iterate convergence of

OPG-PD's policy iterates in Theorem 6. Furthermore, in Figure 13 we show the optimality of OPGPD's policy iterates by plotting the optimality gap | V π glyph[star] r ( ρ ) -V π t r ( ρ ) | and the constraint violation | V π t g ( ρ ) | , where V π glyph[star] r ( ρ ) = 8 . 16 . We see that the optimality gap and the constraint violation both decay asymptotically in linear rates in spite of some oscillations, and larger stepsize enjoys faster convergence. It is worth mentioning that, these descending oscillations actually show that value functions are approaching the optimal ones. Hence, we have verified the linear last-iterate convergence of OPG-PD's policy iterates in Theorem 6 using a range of constant stepsizes.

Optimality gap

Constraint violation

Iteration

Iteration

100

10-1

102

10-3

10-4

5

0

250

500

750

10001250150017502000

100

10-1

102

10-3

10-4

10-5

10-6

10-7

0

250

500

750

10001250150017502000

Last but not least, we extend this experiment for a variant of OPG-PD that is based on the multiplicative weights update (MWU). We call this variant as an optimistic multiplicative weights update primal-dual (OMWU-PD) method which maintains two sequences for policy and dual variables each: { π t } t ≥ 1 and { ˆ π t } t ≥ 1 for the policy-player, and { λ t } t ≥ 1 and { ˆ λ t } t ≥ 1 for the dual-player,

where η is the stepsize and (ˆ π 0 , ˆ λ 0 ) = ( π 0 , λ 0 ) ∈ Π × Λ is the initial point. OMWU-PD concurrently works with two primal iterates and two dual iterates, which is similar to OPG-PD. The only difference is that Primal update (41a) works as the NPG or MWU updates [23, 54], instead of the projected Q -ascent [58, 65], which is also different from the one-step optimistic MWU in policy-based ReLOAD [40].

To further investigate the applicability of optimistic gradient methods, we execute OMWU-PD in the same setting and report our result in Figure 14 and Figure 15. We see that the policy optimality gaps decay sublinearly in Figure 14, and the optimality gap and the constraint violation asymptotically decay in sulinear rates in Figure 15. As a comparison, we repeat the same experiment for policybased ReLOAD [40], which is different from our OMWU-PD in using one-step optimistic gradient updates. In Figure 16 and Figure 17, we observe sublinear decay of policy optimality gap, optimality gap and constraint violation, where are similar as shown in Figure 14 and Figure 15.

The empirical results from two MWU-based optimistic primal-dual algorithms are suggestive. First, MWU-based optimistic algorithms can also have policy last-iterate convergence to an optimal policy. Second, convergence rates of MWU-based optimistic algorithms look slower than OPG-PD's under the same constant stepsize, which we leave as an immediate future work to establish sublinear convergence rates for MWU-based optimistic primal-dual algorithms.

Policy optimality gap

Iteration

101

100

0

250

500

750

1000

1250

1500

1750

2000

Optimality gap

Constraint violation

Iteration

Iteration

100

10-1

10-2

10-3

0

250

500

750

10001250150017502000

100

10-1

10-2

AANAii!

10-3

10-4

10-5

10-6

0

250

500

750

100012501500

17502000

F Supporting Lemmas

In this section, we collect some lemmas that are helpful to our analysis.

Policy optimality gap

Iteration

101

100

0

250

500

750

1000

1250

1500

1750

2000

Optimality gap

Constraint violation

Iteration

Iteration

100

10-1

10-2

10-3

0

250

500

750

10001250150017502000

100

10-1

102

10-3

10-4

10-5

10-6

10-7

0

250

500

750

100012501500

17502000

F.1 Lemmas in optimization

For any convex differentiable function ψ : X → R , the Bregman divergence of x , x ′ ∈ X is given by D ψ ( x ′ , x ) := ψ ( x ′ ) -ψ ( x ) - 〈∇ ψ ( x ) , x ′ -x 〉 . When ψ is σ -strongly convex, D ψ ( x ′ , x ) ≥ σ 2 ‖ x ′ -x ‖ 2 for any x ′ , x ∈ X . Specifically, when ψ ( x ) = 1 2 ‖ x ‖ 2 , D ψ ( x ′ , x ) = 1 2 ‖ x ′ -x ‖ 2 .

Lemma 24. Let X be a convex set. If x = argmin 〈 ¯ x, g 〉 + D ψ (¯ x, x ) , then for any x ∈ X ,

Proof. See [38, Lemma 10].

Lemma 25. Assume that D ψ ( x, x ′ ) ≥ 1 2 ‖ x -x ′ ‖ 2 p for some ψ and p ≥ 1 . If

then where 1 p + 1 q = 1 .

Proof. See [38, Lemma 11].

Lemma 26. Let X ⊂ R m and Y ⊂ R n be polytopes and M ∈ R m × n be a matrix. Then, there exists a problem-dependent constant c > 0 such that

where V glyph[star] is the game value,

and ( X glyph[star] , Y glyph[star] ) is the set of minimax optimal strategies.

Lemma 27. Let X ⊆ ∆( A ) be a convex set and g be a bounded vector in R | A | . If x ′ = argmin ¯ x ∈ X 〈 ¯ x, g 〉 + 1 η KL (¯ x, x ) , then for any x glyph[star] ∈ X ,

where η satisfies ηg a ≥ -1 for a ∈ A .

Proof. See [132, Theorem 2].

F.2 Properties of policy gradient

Lemma 28 (Performance difference lemma) . For any two policies π and π ′ , and any state s 0 ,

Lemma 29 (Regularized PG and NPG under softmax parametrization) . Let an entropy-regularized value function be V π τ ( ρ ) := V π r ( ρ ) + τ H ( π ) , and define Q π τ : S × A → R and V π τ : S → R via Bellman equations,

Let a parametrized policy be π θ for some parameter θ ∈ R d . If the policy π θ is differentiable and ∑ a π θ ( a | s ) = 1 , then,

for all ( s, a ) ∈ S × A , where A π θ τ ( s, a ) := Q π θ τ ( s, a ) -τ log π θ ( a | s ) -V π θ τ ( s ) . Moreover, if the policy π θ is in form of the softmax function π θ ( a | s ) = exp( θ s,a ) ∑ a ′ exp( θ s,a ′ ) for all ( s, a ) ∈ S × A , then

where c ( s ) is an action-independent constant.

Proof. See [52, Lemma 6].

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of policy searching for constrained MDPs. The authors devise two Lagrangian-based named regularized policy gradient primal-dual (RPG-PD) method and optimistic policy gradient primal-dual (OPG-PD) method, respectively. Their methods are single-time-scale and thus insensitive to hyperparameter changes. They also show that the proposed methods have the property of last-iterate convergence and give the convergence rate in theoretical analysis.\n\nSTRENGTHS:\nThe paper is well written and easy to follow. The related works section is particularly commendable, providing a comprehensive overview of relevant works in this field. The authors are the first to give a non-asymptotic rates for last-iterate convergence of single-timescale primal-dual methods. The theoretical analysis is technically sound. From my view point this paper can be a nice addition to the literature on constrained policy optimization.\n\nWEAKNESSES:\n* The convergence of the proposed methods relies on the strong duality property of CMDPs, which may not be true for general parametrized policy classes (like NNs).\n* The numerical experiments are only conducted on tabular cases. And I suggest the authors compare their results to more baselines, for example, primal methods like CRPO in [102].\n* The paper seems a little bit too long to appear in a conference proceeding. I suggest the author remove some redundant explanations or examples to shorten the paper.\n\nQUESTIONS:\n* The paper only considered constrained MDP with a single constraint as a simplified case. Can the results be generalized to the case of multiple constraints?\n* Could the homotopic strategy (gradually shrink the regularization term, see [Li et al. 2022]) be applied to the RPG-PD method? Would that yield better theoretical guarantees?\n\n\n[Li et al. 2022] Homotopic Policy Mirror Descent: Policy Convergence, Implicit Regularization, and Improved Sample Complexity\nYan Li, Guanghui Lan, Tuo Zhao arXiv:2201.09457\n\nLIMITATIONS:\nSee comments above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied the policy gradient primal-dual approach for constraint MDP with last iterate convergence guarantee. A regularized policy gradient primal-dual method is first proposed where regularization on the policy and dual variables are introduced to add curvature to the minimax problem and with appropriately chosen regularization factor, sublinear last-iterate convergence is obtained. An optimistic poicy gradient primal-dual method is then proposed with linear convergence of the squared distance between the last iterate (policy, dual variable) pair and the saddle region under some additional assumptions. The key analysis technique is to bridge the policy update and update in occupancy measure, and CMDP is convex w.r.t. the occupancy measure.\n\nSTRENGTHS:\nThe last iterate convergence of algorithm for CMDP is meaningful. The proposed methods with counterparts in minimax optimization are intuitive and explained very well. Multiple varations of CMDP has been studied, i.e., linear function approximation, zero constraint violations. The I went over a majority part of the proofs and have some questions. I will increase my score after the my questions are addressed. The majority of proofs are sound to me.\n\nWEAKNESSES:\nI did  not find  perticular weaknesses of the current paper. The ideas are explained clearly and easy to follow. The limitations are also discussed.\n\nQUESTIONS:\nThe first inequality in Page 29 between Line 1045 - Line 1046: could you justify the usage of Lemma 27, since \\pi^*_\\tau may not be the optimal w.r.t. to dual variable \\lambda_t?\n\nIt seems that there is some order mismatch of the terms (i), (ii) in Eq. (20), where (i) is O(\\sqrt{epsilon}) and (ii) is O(epsilon). It is possible to further reduce the sample complexity by carefully selecting \\tau?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

comments: SUMMARY:
This work shows the first non-asymptotic and policy last-iterate convergence for single-time-scale algorithms in the CMDP literature. In particular, it provides nearly dimension-free sublinear last-iterate policy convergence, sublinear last-iterate policy convergence with function approximation, and problem-dependent linear last-iterate policy convergence.

STRENGTHS:
1. This work strengthened the prior works which only guarantee asymptotic last-iterate convergence or value-average or policy-mixture non-asymptotic convergence.
2. It sets up a new framework for analyzing policy-based primal-dual algorithms via the distance of primal-dual iterates to a saddle point.

WEAKNESSES:
1. It is better to give a more formal definition of the "single-time-scale" and "two-time-scale" at the beginning of the paper to help readers better understand the introduction.

QUESTIONS:
1. Compared with the references [95, 24], if there is any other technical difficulty apart from replacing the convex inequality conditions in constrained convex optimization with the specialty property of CMDP, such as the performance difference lemma?
2. How is the last-iterate converge rate established in this paper compared with the last-iterate converge rates using two-time-scale methods? Are the convergence rates in this paper better? If not, why not, and if it is possible to reduce the gap?
3. For problem (4), why do you directly show that the strong duality holds? Lemma 1 can not be directly applied to the problem (4).
4. In Theorem 6, how strong is the assumption that optimal state visitation distribution is unique given that there may exist multiple optimal policies for CMDP.

LIMITATIONS:
See questions
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTwo single-timescale algorithms (RPG-PD and OPG-PD) are proposed, and their finite-time convergence rates have been derived. The iteration complexity guarantees for RPG-PD are (nearly) dimension-free but are sublinear. Guarantees for OPG-PD are linear but depend on problem-dependent quantities.\n\nSTRENGTHS:\n1. Last-iterate convergence and constraint satisfaction are useful and generally more challenging. Unlike average and mixture performance measures, these do not hide possible oscillations in objective/constraint functions of immediate policy iterates, which can be undesirable for constrained dynamic systems.\n\n2. Empirical studies showing improved performance over existing algorithms. Most notably -  RPG-PD and OPG-PD suppress oscillation in utility values while achieving optimal reward values.\n\n3. Very thorough literature survey.\n\nWEAKNESSES:\nSee Questions.\n\n Minor typos\n1. At several places, the references are to the results in the appendix, e.g., Theorem 18 below the statement of Thm. 4.\n\nQUESTIONS:\n1. I could not find any explanations/insights on why the previous approaches lead to oscillations while the newly proposed approaches do not. In particular, why does OPG-PD more or less have no oscillations? Also, in Figure 8, the oscillations are more pronounced for smaller parameter choices. Why does that happen? \n\n2. It is unclear how the proposed ideas can be implemented in practice: they involve (a) computing exact expectations or Monte-carlo estimates from a priori unknown distributions and (b) optimizing certain objective functions over the set of all policy distributions, e.g., 6a, 9a, or 27a. These seem very challenging. Can the authors comment on it? \n\n3. While the authors have mentioned the online version as part of future work, do they expect this version to also have the benefits of damped oscillations?\n\n4. In practice, one often may require that the constraints not be violated with high probability. Would the proposed approach extend to cover that scenario as well?\n\n5. What is `optimistic' about the OPG-PD method? Some insights would be helpful for the reader.\n\nLIMITATIONS:\nLimitations of the proposed techniques have been adequately discussed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-26"}