Title: Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Constrained Reinforcement Learning

Last-Iterate Convergence

Global Convergence

Policy Gradients
Abstract: *Constrained Reinforcement Learning* (CRL) tackles sequential decision-making problems where agents are required to achieve goals by maximizing the expected return while meeting domain-specific constraints, which are often formulated on expected costs. In this setting, *policy-based* methods are widely used since they come with several advantages when dealing with continuous-control problems. These methods search in the policy space with an *action-based* or *parameter-based* exploration strategy, depending on whether they learn directly the parameters of a stochastic policy or those of a stochastic hyperpolicy. In this paper, we propose a general framework for addressing CRL problems via *gradient-based primal-dual* algorithms, relying on an alternate ascent/descent scheme with dual-variable regularization. We introduce an exploration-agnostic algorithm, called C-PG, which exhibits global last-iterate convergence guarantees under (weak) gradient domination assumptions, improving and generalizing existing results. Then, we design C-PGAE and C-PGPE, the action-based and the parameter-based versions of C-PG, respectively, and we illustrate how they naturally extend to constraints defined in terms of *risk measures* over the costs, as it is often requested in safety-critical scenarios. Finally, we numerically validate our algorithms on constrained control problems, and compare them with state-of-the-art baselines, demonstrating their effectiveness.

Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning

Abstract

Constrained Reinforcement Learning (CRL) tackles sequential decision-making problems where agents are required to achieve goals by maximizing the expected return while meeting domain-specific constraints, which are often formulated as expected costs. In this setting, policy-based methods are widely used since they come with several advantages when dealing with continuous-control problems. These methods search in the policy space with an action-based or parameter-based exploration strategy, depending on whether they learn directly the parameters of a stochastic policy or those of a stochastic hyperpolicy. In this paper, we propose a general framework for addressing CRL problems via gradient-based primal-dual algorithms, relying on an alternate ascent/descent scheme with dualvariable regularization. We introduce an exploration-agnostic algorithm, called C-PG , which exhibits global last-iterate convergence guarantees under (weak) gradient domination assumptions, improving and generalizing existing results. Then, we design C-PGAE and C-PGPE , the action-based and the parameter-based versions of C-PG , respectively, and we illustrate how they naturally extend to constraints defined in terms of risk measures over the costs, as it is often requested in safety-critical scenarios. Finally, we numerically validate our algorithms on constrained control problems, and compare them with state-of-the-art baselines, demonstrating their effectiveness.

1 Introduction

When applying Reinforcement Learning (RL, Sutton and Barto, 2018) to real-world scenarios, we are tasked with addressing large-scale continuous control problems where, in addition to reaching a goal, it is necessary to meet structural or utility-based constraints. For instance, an autonomousdriving car has its main objective of getting to the desired destination (i.e., goal) while avoiding collisions, ensuring the safety of people on the streets, adhering to traffic rules, and respecting the physical requirements of the engine to avoid damaging it (i.e., constraints) (Likmeta et al., 2020). To pursue such an objective, it is necessary to extend the RL problem formulation with the possibility to account for constraints. Constrained Reinforcement Learning (CRL, Uchibe and Doya, 2007) aims at solving this family of problems by employing RL techniques to tackle Constrained Markov Decision Processes (CMDPs, Altman, 1999), which provide an established and widely-used framework for modeling constrained control tasks. The conventional CRL framework primarily focuses on constraints related directly to expected costs (Stooke et al., 2020; Ding et al., 2020; Ying

et al., 2022; Ding et al., 2024). However, especially in safety-critical contexts, the expected cost may not represent a reliable index of safe behavior. In response to this issue, chance constraints were introduced to ensure that the probability of unsafe events is minimized. Nonetheless, employing chance constraints presents several challenges (Chow et al., 2017). To strike a balance between these two extremes, constraints are defined in terms of risk measures over the costs. Examples include the Conditional Value at Risk (CVaR, Rockafellar and Uryasev, 2000) and the Mean-Variance (MV, Markowitz and Todd, 2000; Li and Ng, 2000). These risk measures offer a generalization of the previous concepts, allowing for the consideration of uncertainties while preserving the focus on the cost. When incorporating constraints on risk measures, CRL is often referred to as Risk-CRL (Chow et al., 2017).

Among the RL methods applicable to CMDPs, Policy Gradients (PGs, Deisenroth et al., 2013) are particularly appealing. Indeed, PGs have demonstrably achieved impressive results in continuouscontrol problems due to several advantages that make them well-suited for real-world applications. These advantages include the ability to handle continuous state and action spaces (Peters and Schaal, 2006), resilience to sensor and actuator noise (Gravell et al., 2020), robustness in partially-observable environments (Azizzadenesheli et al., 2018), and the possibility of incorporating expert knowledge during policy design (Ghavamzadeh and Engel, 2006), which can simplify the learning process and improve the efficacy, safety, and interpretability of the learned policy (Likmeta et al., 2020). PGs can be categorized into two key families depending on the way exploration is carried out in the policy space (Montenegro et al., 2024). Following their taxonomy, we distinguish between the action-based and the parameter-based exploration paradigms. The former, employed by REINFORCE (Williams, 1992) and GPOMDP (Baxter and Bartlett, 2001), focuses on directly learning the parameters of a parametric stochastic policy . The latter, employed by PGPE (Sehnke et al., 2010), is tasked with learning the parameters of a parametric stochastic hyperpolicy from which the parameters of the actual policy (often deterministic) are sampled.

Policy-based CRL has gained significant popularity in solving constrained control problems (Achiam et al., 2017). Within this field, algorithms are primarily developed using primal-dual methods (Chow et al., 2017; Tessler et al., 2019; Ding et al., 2020, 2021; Bai et al., 2022), which can be formulated through Lagrangian optimization of the primal (i.e., policy parameters) and dual variable (i.e., Lagrange multipliers). Even though the distinction between the exploration paradigms is well known in the PG methods literature, the current state of the art in Policy-based CRL focuses only on the action-based exploration approach (Achiam et al., 2017; Stooke et al., 2020; Bai et al., 2023), while the parameter-based one remains unexplored. A critical challenge for policy-based Lagrangian optimization algorithms is ensuring convergence guarantees. Existing works have spent a notable effort in this direction (Ying et al., 2022; Gladin et al., 2023; Ding et al., 2024). Recently, (Ying et al., 2022; Gladin et al., 2023; Ding et al., 2024) manage to ensure global last-iterate convergence guarantees. However, these approaches are affected by some notable limitations: ( i ) the provided convergence rates depend on the problem dimension (e.g., the cardinality of the state and action spaces), limiting their applicability to tabular CMPDs and preventing scaling to realistic continuous control problems; ( ii ) they focus on softmax policies only, disregarding other more realistic policy models; ( iii ) (Ding et al., 2024) ensure convergence when a single constraint only is present.

Original Contribution. The goal of this work is to introduce a framework for solving constrained continuous control problems using policy-based primal-dual algorithms that operate in both the action-based and parameter-based policy gradient exploration scenarios, while providing global last-iterate convergence guarantees with general (hyper)policy parameterization. Specifically, the main contributions of this work can be summarized as follows:

In Section 2, we introduce a general constrained optimization problem, which is agnostic w.r.t. both the action-based or parameter-based paradigm.

In Section 3, we introduce C-PG , a general policy-based primal-dual algorithm optimizing the regularized Lagrangian function associated with the general constrained optimization problem shown in Section 2. We show that, under (weak) domination assumptions, it simultaneously achieves the following: ( i ) last-iterate convergence guarantees to a globally optimal feasible policy (i.e., satisfying all constraints); ( ii ) compatibility with CMDPs having continuous state and action spaces ; ( iii ) the ability to handle multiple constraints .

In Section 4, we introduce C-PGAE and C-PGPE , the action-based and parameter-based versions of C-PG , respectively. Both algorithms are designed to also handle constraints on risk measures, thus extending the applicability space of the father algorithm C-PG . This is achieved by employing

a parametric unified risk measure formulation, for which we show the mapping to several risk measures of the unified one and we present the specific form of all the estimators.

In Section 5, we numerically validate our proposals against state-of-the-art baselines in constrained control problems. Related works are discussed in Appendix B. The proofs of all the statements are reported in Appendix E.

2 Preliminaries

Notation. For a measurable set X , we denote as ∆ p X q the set of probability measures over X . For P P ∆ p X q , we denote with p its density function and we will interchangeably use x ' P or x ' p to express that random variable x is distributed according to P . For n, m P N with n ď m , we denote J n K : ' t 1 , 2 , . . . , n u and with J n, m K : ' t n, n ` 1 , . . . , m u . For a vector x P R d , we denote as x i the i -th component of x . For a P R , we define p a q ` : ' max t 0 , a u and we extend the notation to vectors as p x q ` ' pp x 1 q ` , . . . , p x d q ` q J . Given a set X Ď R d , we denote with Π X the Euclidean norm projection, i.e., Π X x P arg min y P X } y ´ x } 2 for any x P R d . For two vectors x , y P R d , we denote with x x , y y their inner product. A function f : R d Ñ R is L 1 -Lipschitz continuous if | f p x q ´ f p x 1 q| ď L 1 } x ´ x 1 } 2 and L 2 -smooth if it is differentiable and } ∇ x f p x q ´ ∇ x f p x 1 q} 2 ď L 2 } x ´ x 1 } 2 , for every x , x 1 P R d .

0 p q ' ´ p q p q i p q P r max s P J K

Constrained Markov Decision Processes. AConstrained Markov Decision Process (CMDP, Altman, 1999) with U constraints is represented by M C : ' ` S , A , p, r, t c i u i P J U K , t b i u i P J U K , µ 0 , γ ˘ , where S Ď R d S and A Ď R d A are the measurable state and action spaces, p : S ˆ A Ñ ∆ p S q is the transition model, where p p s 1 | s , a q is the probability density of getting to state s 1 P S given that action a P A is taken in state s P S , r : S ˆ A Ñ r´ 1 , 0 s is the reward function, where r p s , a q is the instantaneous reward obtained by playing action a in state s , c i : S ˆ A Ñr 0 , 1 s is the i -th cost function, where c i p s , a q is the i -th instantaneous cost obtained by playing action a in state s , b i P r 0 , J max s with J max : ' 1 ´ γ T 1 ´ γ is the threshold for the i -th cost for every i P J U K , µ 0 P ∆ p S q is the initial state distribution, and γ P r 0 , 1 s is the discount factor. A trajectory τ of length T P N Yt`8u is a sequence of T state-action pairs: τ ' p s τ, 0 , a τ, 0 , . . . , s τ,T ´ 1 , a τ,T ´ 1 q . The discounted return over a trajectory τ is R p τ q : ' ř T ´ 1 t ' 0 γ t r p s τ,t , a τ,t q , while the i -th discounted cumulative cost is C i p τ q : ' ř T ´ 1 t ' 0 γ t c i p s τ,t , a τ,t q . We define the additional cost function c 0 p s , a q : ' ´ r p s , a q and C τ : R τ . Note that R τ , C τ 0 , J for every i U and trajectory τ .

Action-based Policy Gradients. Action-based (AB) PG methods focus on learning the parameters θ P Θ Ď R d Θ of a parametric stochastic policy π θ : S Ñ ∆ p A q , where π θ p a | s q represents the probability density of selecting action a P A being in state s P S . At each step t of the interaction with the environment, the stochastic policy is employed to sample an action a t ' π θ t p¨| s t q . To assess the performance of π θ w.r.t. the i -th cost function, with i P J 0 , U K , we employ the AB performance index J A ,i : Θ Ñ R , which is defined as J A ,i p θ q : ' E τ ' p A p¨| θ q r C i p τ qs , where p A p τ, θ q : ' µ 0 p s τ, 0 q ś T ´ 1 t ' 0 π θ p a τ,t | s τ,t q p p s τ,t ` 1 | s τ,t , a τ,t q is the density of trajectory τ induced by policy π θ .

Parameter-based Policy Gradients. Parameter-based (PB) PG methods focus on learning the parameters ρ P R Ď R d R of a parametric stochastic hyperpolicy ν ρ P ∆ p Θ q . The hyperpolicy ν ρ is used to sample parameter configurations θ ' ν ρ to be plugged into an underlying parametric policy π θ , that will be then used for the interaction with the environment. Notice that π θ can also be deterministic. To assess the performance of ν ρ w.r.t. the i -th cost function, with i P J 0 , U K , we employ the PB performance index J P ,i : R Ñ R , which is defined as J P ,i p ρ q : ' E θ ' ν ρ r E τ ' p A p¨| θ q r C i p τ qss .

Constrained Optimization Problem. Having introduced the AB and PB performance indices, we formulate a constrained optimization problem (COP) agnostic w.r.t. the exploration paradigm:

where : P t A , P u and υ is a generic parameter vector belonging to the parameter space V . When : ' A, we are considering the AB exploration paradigm, then V ' Θ . On the other hand, when : ' P, we are in the PB exploration paradigm, then V ' R .

3 Last-Iterate Global Convergence of C-PG

In this section, we present C-PG , a general primal-dual algorithm that optimizes a regularized version of the Lagrangian function (Section 3.1) associated with the COP of Equation (1). After having introduced the necessary assumptions (Section 3.2), we show that C-PG exhibit dimension-free 1 last-iterate global convergence guarantees (Section 3.3). While C-PG is designed to be agnostic w.r.t. the exploration approach, we introduce two specific versions of C-PG in Section 4 for AB or PB, respectively. For notation convenience, in the rest of this section, we use J i in place of J : ,i .

3.1 Regularized Lagrangian Approach

To solve the COP of Equation (1) we resort to the method of Lagrange multipliers (Bertsekas, 2014) introducing the Lagrangian function L 0 p υ , λ q : ' J 0 p υ q ` ř U i ' 1 λ i p J i p υ q ´ b i q ' J 0 p υ q ` x λ , J p υ q ´ b y , where υ P V is the primal variable and λ P R U ě 0 are the Lagrangian multipliers or dual variable, J ' p J 1 , . . . , J U q J , and b ' p b 1 , . . . , b U q J . This allows to rephrase the COP in Equation (1) as a min-max optimization problem min υ P V max λ P R U ě 0 L 0 p υ , λ q and we denote with H 0 p υ q : ' max λ P R U ě 0 L 0 p υ , λ q the primal function and with H ˚ 0 : ' min υ P V H 0 p υ q . To obtain a last-iterate convergence guarantee, we make use of a regularization approach. Specifically, let ω ą 0 be a regularization parameter, we define the ω -regularized Lagrangian function as follows:

The ridge regularization makes L ω p υ , λ q a strongly concave function of λ at the price of a bias that is quantified in Lemmas E.1, E.2 and E.3. Thus, we address the ω -regularized min-max optimization problem min υ P V max λ P Λ L ω p υ , λ q , where Λ : ' t λ P R U ě 0 : } λ } 2 ď ω ´ 1 ? UJ max u , in replacement of the original (non-regularized) one. For this problem, we introduce the primal function H ω p υ q : ' max λ P Λ L ω p υ , λ q , that, thanks to the ridge regularization, admits the closedform expression H ω p υ q ' J 0 p υ q ` 1 2 ω ř U i ' 1 pp J i p υ q ´ b i q ` q 2 ' J 0 p υ q ` 1 2 ω }p J p υ q ´ b q ` } 2 2 , where the optimal values of the Lagrange multipliers is given by λ ˚ p υ q ' Π Λ ` 1 ω p J p υ q ´ b q ˘ ' 1 ω p J p υ q ´ b q ` that is guaranteed to have norm smaller than ω ´ 1 ? UJ max . Furthermore, we define H ˚ ω : ' min υ P V H ω p υ q . C-PG updates the parameters p υ k , λ k q with an alternate gradient descentascent scheme for every k P N :

where ζ υ ,k , ζ λ ,k ą 0 are the learning rates and p ∇ υ L ω p υ k , λ k q , p ∇ λ L ω p υ k , λ k q are estimators of the gradients ∇ υ L ω p υ k , λ k q , ∇ λ L ω p υ k , λ k q of the regularized Lagrangian function.

3.2 Assumptions

Before diving into the study of the convergence guarantees of C-PG , in this part, we list and motivate the assumptions necessary for our analysis.

Assumption 3.1 (Existence of Saddle Points) . There exist υ ˚ 0 P V and λ ˚ 0 P R U ě 0 such that L 0 p υ ˚ 0 , λ ˚ 0 q ' min υ P V max λ P R U ě 0 L 0 p υ , λ q .

Assumption 3.1 ensures that the value of the min-max problem is attained by a pair of primal-dual values υ ˚ 0 P V and λ ˚ 0 P R U ě 0 which, consequently, satisfy L 0 p υ ˚ 0 , λ q ď L 0 p υ ˚ 0 , λ ˚ 0 q ď L 0 p υ , λ ˚ 0 q for every υ P V and λ P R U ě 0 . Analogous assumptions have been considered by Yang et al. (2020) and Ying et al. (2022). Thus, p υ ˚ 0 , λ ˚ 0 q is a saddle point of the Lagrangian function L 0 and, consequently, strong duality holds. Alternatively, as commonly requested in CRL works, assuming

Slater's condition combined with the requirement that the policy space covers all Markovian policies ensures strong duality (e.g., Paternain et al., 2019; Ding et al., 2020, 2024). 2

Assumption 3.2 (Weak ψ -Gradient Domination) . Let ψ P r 1 , 2 s . There exist α 1 ą 0 and β 1 ě 0 such that, for every υ P V and λ P Λ , it holds that:

Assumption 3.2 is customary in the convergence analysis of policy gradient methods and it is usually enforced on the objective J 0 only (Yuan et al., 2022; Masiha et al., 2022; Fatkhullin et al., 2023). In particular, when β 1 ' 0 , we speak of (strong) ψ -gradient domination. In this form, for a generic exponent ψ P r 1 , 2 s , this assumption has been employed by Masiha et al. (2022). Particular cases are when ψ ' 1 , which corresponds to the standard (weak) gradient domination (GD), while for ψ ' 2 , we have the so-called Polyak-Łojasiewicz (PL) condition. Notice that Assumption 3.2 is enforced on the non-regularized Lagrangian function L 0 (i.e., ω ' 0 ). However, it is easy to realize that it holds for the regularized one L ω by simply computing the terms of Equation (3) replacing L 0 with L ω .

Remark 3.1 (When does Assumption 3.2 holds?) . As remarked by Ding et al. (2024), the Lagrangian function, for a fixed value of λ , can be regarded as the expected return of a new reward function ´ C 0 ´ x λ , C y , where C ' p C 1 , . . . , C U q J . As a consequence, a sufficient condition for Assumption 3.2 is when the selected class of policies guarantees the ψ -gradient domination regardless of the reward function. For instance, in tabular environments with natural policy parametrization, i.e., π θ p s q ' θ s for every s P S , the PL condition ( ψ ' 2 and β 1 ' 0 ) holds (Bhandari and Russo, 2024). Moreover, in tabular environments with softmax policy, i.e., π θ p a | s q9 exp p θ p s, a qq , GD ( ψ ' 1 and β 1 ' 0 ) holds (Mei et al., 2020). This enables a meaningful comparison of our results with resorting to softmax policies (e.g., Ding et al., 2020; Gladin et al., 2023; Ding et al., 2024). More in general, when ( i ) the Fisher information matrix induced by policy π θ is non-degenerate for every θ P Θ , i.e., F p θ q ' E π θ r ∇ θ log π θ p a | s q ∇ θ log π θ p a | s q J s ľ µ F I for some µ F ą 0 and ( ii ) a compatible function approximation bias bound holds, i.e., E π θ ˚ rp A π θ p s , a q ´ p 1 ´ γ q u J ∇ θ log π θ p a | s qq 2 s ď ϵ bias being u ' F p θ q : ∇ θ J 0 p θ q and the advantage function A π θ computed w.r.t. reward ´ C 0 ´x λ , C y , the weak GD ( ψ ' 1 ) holds with α 1 ' Gµ ´ 1 F and β 1 ' p 1 ´ γ q ´ 1 ? ϵ bias, where G is such that } ∇ θ log π θ p a | s q} 2 ď G (Masiha et al., 2022).

In principle, we could have enforced Assumption 3.2 on the primal function H ω p υ q only. However, this would come with two drawbacks: ( i ) the assumption would now depend explicitly on ω ; ( ii ) the considerations of Remark 3.1 would no longer hold. Nevertheless, in Lemma E.4, we prove that Assumption 3.2 induces an analogous property on the primal function H ω p υ q in the regularized case.

Assumption 3.3 (Regularity of the Regularized Lagrangian L ω ) . There exists L 1 , L 2 , L 3 ą 0 such that, for every υ , υ 1 P V , and for every λ , λ 1 P Λ , the following hold:

Notice that, similarly to Assumption 3.2, we realize that if Assumption 3.3 holds for the nonregularized Lagrangian L 0 , it also holds (with the same constants) for the regularized one L ω for every ω ą 0 . The regularity conditions of Assumption 3.3 are common in the literature (Yang et al., 2020) and mild when regarded from the policy optimization perspective. Equation (4) is satisfied whenever the constraint functions J i are Lipschitz continuous w.r.t. υ . Indeed, } ∇ λ L 0 p υ , λ q ´ ∇ λ L 0 p υ 1 , λ q} 2 ' } J p υ q ´ J p υ 1 q} 2 . Equation (5) is fulfilled when the objective function J 0 and the constraint functions J i are smooth w.r.t. υ and the Lagrange multipliers are bounded (guaranteed thanks to the projection Π Λ ), since } ∇ υ L 0 p υ , λ q ´ ∇ υ L 0 p υ 1 , λ q} 2 ď | ∇ υ J 0 p υ q´ ∇ υ J 0 p υ 1 q| ` ř U i ' 1 λ i | ∇ υ J i p υ q´ ∇ υ J i p υ q| . Finally, Equation (6) is fulfilled whenever functions J i admit bounded gradients, since } ∇ υ L 0 p υ , λ q ´ ∇ υ L 0 p υ , λ 1 q} 2 ď } ∇ υ J p υ qp λ ´ λ 1 q} 2 . It is worth noting that L 2 depends on the norm of the Lagrange multipliers and, consequently, due

to the projection operator, we have that L 2 ' O p ω ´ 1 q , whereas L 1 and L 3 are independent on ω . 3 Explicit conditions on the constitutive elements of the MDP and (hyper)policies to ensure Lipshitzness and smoothness of these quantities are reported in (Montenegro et al., 2024, Appendix E) for both the AB and PB cases. These regularity properties enforced on L ω are inherited by the primal function H ω which results to be ` L 2 ` L 2 1 ω ´ 1 ˘ -smooth (Lemma E.7). Concerning the regularity of L ω w.r.t. λ , we observe that it is a quadratic function and, therefore, it is ω -smooth and satisfies the PL condition, i.e., Assumption 3.2 with ψ ' 2 , β 1 ' 0 , and with α 1 ' ω (Lemma E.5).

Assumption 3.4 (Bounded Estimator Variance) . For every υ P V and λ P Λ , the estimators p ∇ υ L ω p υ , λ q and p ∇ λ L ω p υ , λ q are unbiased for ∇ υ L ω p υ , λ q ' ∇ υ J 0 p υ q ` ř U i ' 1 λ i ∇ υ J i p υ q and ∇ λ L ω p υ , λ q ' J p υ q ´ b ´ ω λ with bounded variance, i.e., there exist V υ , V λ ă `8 such that:

Note that V υ typically depends on the Lagrange multipliers and, for standard sample mean estimators, it is of order V υ ' O p ω ´ 2 q thanks to the projection operator. Contrary, V λ is usually not affected by ω since the term ω λ is not estimated and, thus, it does not impact on the variance of the sample mean estimator. In Section 4, explicit estimators are provided for both the AB and PB cases. The variance of such estimators can be easily controlled by leveraging on the properties of the score function as done in previous works (see Papini et al. 2022 and Montenegro et al. 2024, Appendix E).

3.3 Convergence Analysis

We are now ready to attack the convergence analysis of C-PG to the global optimum of the COP of Equation (1). To this end, we study the potential function defined as P k p χ q : ' a k ` χb k , where a k : ' E r H ω p υ k q ´ H ˚ ω s and b k : ' E r H ω p υ k q ´ L ω p υ k , λ k qs and χ P p 0 , 1 q will be specified later. Since a k , b k ě 0 , intuitively, if P k p χ q « 0 we have that both a k , b k « 0 and, consequently, convergence is achieved. Let us start relating P k p χ q , with the solution of the COP in Equation (1).

Theorem 3.1 (Objective Function Gap and Constraint Violation) . Let ϵ ą 0 . Under Assumption 3.1, if P k ď ϵ , it holds that:

Theorem 3.1 justifies the study of the potential P k as a technical tool to ensure convergence. Indeed, whenever P k ď ϵ both the ( i ) objective function gap and ( ii ) the constraint violation scale linearly with ϵ and with the regularization parameter ω of the regularized Lagrangian L ω multiplied by the norm of the Lagrange multipliers of the non-regularized problem } λ ˚ 0 } 2 , which are finite under Assumption 3.1. This expression also suggests a choice of ω ' O p ϵ q to enforce an overall ϵ error on both quantities. Note that, from Theorem 3.1, it is immediate to employ a conservative constraint ( b 1 i « b i ´ 4 ϵ ´ ω } λ ˚ 0 } 2 2 ) to achieve zero constraint violation with no modification of the algorithm.

We are now ready to state the convergence guarantees for the potential function.

Theorem 3.2 (Convergence of P K ) . Under Assumptions 3.2, 3.3, 3.4, for χ ă 1 { 5 , sufficiently small ϵ and ω , and a choice of constant learning rates ζ υ , ζ λ , we have P K p χ q ď ϵ ` β 1 { α 1 whenever: 4

K ' O p ω ´ 1 ϵ ´ 2 ψ ´ 1 q if ψ P r 1 , 2 q and the gradients are exact (i.e., V υ ' V λ ' 0 );

K ' O p ω ´ 1 log p ϵ ´ 1 qq if ψ ' 2 and the gradients are exact (i.e., V υ ' V λ ' 0 );

K ' O p ω ´ 3 ϵ ´ 4 ψ ` 1 q if ψ P r 1 , 2 s and the gradients are estimated (i.e., V υ ' O p ω ´ 2 q and V λ ' O p 1 q ).

Some comments are in order. First, Theorem 3.2 holds for a specific choice of the constant χ P p 0 , 1 { 5 q defining the potential function P K . Second, the presented rates hold for sufficiently small values of ϵ and ω . This is just for presentation purposes, as the sample complexity 5 can only improve if we increase the values of ϵ and ω . Third, in the proof, an explicit expression of the learning rates is provided. Concerning their orders, for the case of exact gradients, we choose ζ λ ' ω ´ 1 and ζ υ ' O p ω q , whereas for the estimated gradient case, we choose ζ λ ' O p ωϵ 2 { ψ q and ζ υ ' O p ω 3 ϵ 2 { ψ q .

'

p

q

p

q

p

q

p

q

p

q

Assuming ω to be a constant, we observe that both learning rates display the same dependence on ϵ and, consequently, they are in single-time scale . However, as we have seen in Theorem 3.1, in order to obtain guarantees on the original non-regularized problem, we have to set ω ' O p ϵ q , leading to a two-time scales algorithm. Fourth, we observe that, for both exact and estimated gradients, the sample complexity degrades as the constant ψ of the gradient domination moves from 2 to 1 , delivering the smallest sample complexity when the PL condition holds. Finally, we highlight that C-PG jointly: ( i ) converges to the global optimum of the COP problem of Equation (1); ( ii ) delivers a last-iterate guarantee; ( iii ) has no dependence on the cardinality of the state or action spaces, making it completely dimension-free . Table 1 and Figure 1 summarize the results of Theorem 3.2.

1

1

.

2

1

.

4

1

.

6

1

.

8

2

2

4

6

8

ψ

-

log

K

log

ϵ

Exact gradients

Estimated gradients

4 Action-based and Parameter-based Primal-Dual Algorithms

In this section, we introduce C-PGAE and C-PGPE , the action-based and the parameter-based versions of C-PG , respectively. Both the algorithms have been designed to tackle continuous risk-constrained optimization problems (RCOP), generalizing the COP of Equation (1) and thus extending the applicability space of C-PG . This is done by employing a parametric unified risk measure formulation, which leads to having a framework for solving risk-constrained problems via policy-based primaldual methods operating in both the action-based and parameter-based PGs exploration scenarios. Moreover, this framework allows to handle constraints enforced on several risk measures, which will be discussed below. We first introduce a parametric unified risk measure and formulate an exploration-agnostic RCOP (Section 4.1). Then, we present both C-PGAE and C-PGPE (Section 4.2).

4.1 Risk-Constrained Optimization Problem

We start presenting the notion of unified risk measure originally introduced by Bisi et al. (2022). For the AB case, to evaluate the performance of policy π θ w.r.t. the i -th cost, with i P J 0 , U K , given two functions f i : R 2 Ñ R and g i : R Ñ R , we define the AB-risk measure as:

Similarly, for the PB case, to assess the performance of hyperpolicy ν ρ w.r.t. the i -th cost, with i P J 0 , U , we define the PB-risk measure as:

Some observations are in order. First, by selecting the functions f i and g i , we generate different risk measures (Table 2). Details on the presented risk measures and on the mappings can be found in Appendix C. Second, the risk measure itself is defined as another minimization problem over the additional real variable η i . In principle, if we replace the constraints on cost expectation of the COP in Equation (1) with the risk measures presented above, we are in the presence of a bilevel optimization problem. However, it is immediate to realize that we can merge variables η i with the primal variables υ of the COP without changing the optimum. Finally, let us appreciate the semantic difference between enforcing risk-based constraints in the AB and PB cases. Indeed, while for AB the stochasticity inducing the risk is the one generated by the policy π θ and the environment, for PB we have the joint stochastic process of the hyperpolicy ν ρ , policy π θ (when stochastic), and the

t

p

q ě

u

environment. To the best of our knowledge, this paper is the first proposing to enforce risk-based constraints for parameter-based exploration.

Having introduced the notions of AB and PB unified risk measures, we can formulate a riskconstrained optimization problem (RCOP) agnostic w.r.t. the exploration paradigm and the risk measure: 6

where : P t A , P u , υ P V is the parameter and η ' p η 1 , . . . , η U q P R U are the auxiliary variables.

As for the COP of Equation (1), we define the regularized Lagrangian function for the RCOP (Equation 11) as follows:

with : P t A, P u . We assume that r L : ,ω is differentiable w.r.t. υ and subdifferentiable 7 w.r.t. η , while, by construction, it is already differentiable w.r.t. λ .

Remark 4.1 (Do the Convergence Guarantees of Section 3 Apply to the RCOP?) . When all the f i and g i are selected in order to consider the expected values of the corresponding C i (i.e., f i p C i p τ q , η i q ' C i p τ q and g i p η i q ' 0 ), then r L : ,ω coincides with L ω , thus, all the theoretical results presented in Section 3 hold. This is not true in general. For some risk measures r L : ,ω may not be smooth or even differentiable in υ or η , or it may violate the weak gradient domination assumption. Although out of the scope of the present paper, studying the preservation of the (weak) gradient domination for risk-based objectives is an appealing future research direction.

4.2 Algorithms: C-PGAE and C-PGPE

Algorithms. Both algorithms, whose pseudo-codes are deferred to Appendix A, aim at solving the RCOP of Equation (11), finding the best feasible (hyper)policy parameterization. The alternate ascent/descent primal-dual prototypical algorithm is the same as described in Section 3. In particular, at each iteration k P J K K , the algorithms collect N trajectories (employing the chose exploration paradigm) to update the primal and dual variables as:

where the values are considered to be projected into their spaces, and ζ υ ,k , ζ λ ,k , ζ η ,k ą 0 are the learning rates at the k -th iteration for υ , λ , and η , respectively. Notice that, since we use alternate ascent/descent , the update for λ k ` 1 uses the updated primal values υ k ` 1 and η k ` 1 . In practice, this requires collecting a first batch of N { 2 trajectories using the (hyper)policy υ k to perform the primal updates, computing υ k ` 1 and η k ` 1 . Then, it is necessary to collect a new batch of N { 2 trajectories using the updated (hyper)policy υ k ` 1 in order to perform the dual update, computing λ k ` 1 , which also leverages η k ` 1 .

Estimators. Here, we provide the risk-agnostic form of the estimators used to perform the primal and dual updates. In particular, the gradient of r L : ,ω with respect to η is closely related to the choices of f i and g i . Therefore, the discussion is deferred to Appendix D, where we also explicitly derive all the estimators for the risk measures presented in Table 2 and for both exploration paradigms. The estimator of the gradient of r L : ,ω w.r.t. λ has the same form for both C-PGAE and C-PGPE , that is:

where the trajectories are collected with the AB or the PB exploration paradigm , f p C p τ q , η q : ' p f 1 p C 1 p τ q , η 1 q , . . . , f U p C U p τ q , η U qq J and g p η q : ' p g 1 p η 1 q , . . . , g U p η U qq J .

C-PGAE (Algorithm 1) is the action-based version of C-PG . At each iteration k P J K K , the agent collects N trajectories by playing the policy π θ k , then it alternatively updates the policy parameter θ and the risks parameter η , or the Lagrange multipliers λ . The θ update relies on the estimator:

which reduces to the prototypical action-based algorithm REINFORCE (Williams, 1992) in the risk-neutral case. When allowed by the choice of f i , we switch to a GPOMDP-style estimator (Baxter and Bartlett, 2001) which suffers less variance. See Appendix D for details.

C-PGPE (Algorithm 2) is the parameter-based version of C-PG . At each iteration k P J K K , the algorithm samples N parameter configurations t θ i u i P J N K from the hyperpolicy ν ρ k , then collects a single trajectory τ i , obtained by playing the policy π θ i , for each sampled parameterization θ i . The sampled trajectories and parameters are then used to alternatively update the hyperpolicy parameter vector ρ and the risks parameter vector η , or the vector of Lagrange multipliers λ . In particular, the ρ update relies on the following estimator:

which reduces to the parameter-based algorithm PGPE (Sehnke et al., 2010) in the risk-neutral case.

5 Numerical Validation

In this section, we empirically validate some of the theoretical results shown throughout this work. Experimental details and additional results are provided in Appendix H. The code to run the experiments in this paper is available at https://github.com/MontenegroAlessandro/MagicRL .

Comparison in DGWW. We compare our C-PGAE against the sample-based versions of NPGPD (Ding et al., 2020, Appendix H) and RPG-PD (Ding et al., 2024, Appendix C.9) on a Discrete Grid World with Walls (DGWW, see Appendix H) with a horizon of T ' 100 , and with a single constraint on the average trajectory cost. The methods are learning the parameters of a tabular softmax policy and the learning phase considers constant step sizes. Figure 2a shows the average return and the average cost curves over the trajectories seen during the learning. As can be noticed, C-PGAE strikes the objective of the COP with fewer trajectories w.r.t. the competitors. Indeed, both NPG-PD and RPG-PD require additional O p| S | ` | S || A |q trajectories per iteration.

Comparison in LQR. We compare our C-PGAE and C-PGPE against the continuous sample-based version of NPG-PD2 (Ding et al., 2022, Algorithm 1), working with generic policy parameterizations. Additionally, we consider RPG-PD2, a ridge-regularized version of NPG-PD2. The environment is a bidimensional CostLQR (Appendix H) with a horizon T ' 50 and a single cost that the algorithms should keep below b ' 0 . 2 on average. C-PGAE , NPG-PD2, and RPG-PD2 learn the parameter of a linear gaussian policy , while C-PGPE the ones of a gaussian hyperpolicy over a linear deterministic policy . All the step sizes are chosen with Adam (Kingma and Ba, 2015) scheduler. Figure 2b reports the learning curves for the average return and the cost, confirming that our methods solve the COP with fewer trajectories. Indeed, being both NPG-PD2 and RPG-PD2 actor-critic methods, they suffer from the inner critic loop, which requires the collection of additional trajectories ( 500 here). We stress that the actor-critic methods were very sentitive to the hyperparameters selection, especially the length and the step size of the inner loop (see Appendix H).

0

5

,

000

10

,

000

15

,

000

20

,

000

25

,

000

30

,

000

´

100

´

80

´

60

´

40

´

20

Trajectories

Average Trajectory Return

C-PGAE

NPG-PD

RPG-PD

0

5

,

000

10

,

000

15

,

000

20

,

000

25

,

000

30

,

000

0

1

2

3

4

Trajectories

Average Trajectory Cost

C-PGAE

NPG-PD

RPG-PD

b

(a) DGWW.

0

1

¨

10

5

2

¨

10

5

3

¨

10

5

4

¨

10

5

5

¨

10

5

6

¨

10

5

´

15

´

10

´

5

Trajectories

Average Trajectory Return

C-PGAE

C-PGPE

NPG-PD2

RPG-PD2

0

1

¨

10

5

2

¨

10

5

3

¨

10

5

4

¨

10

5

5

¨

10

5

6

¨

10

5

0

0

.

2

0

.

4

0

.

6

Trajectories

Average Trajectory Cost

C-PGAE

C-PGPE

NPG-PD2

RPG-PD2

b

(b) CostLQR.

10

20

30

40

50

0

10

20

30

Cost Value

Frequency

Chance

MV

CVaR

Average Cost

(a)

C-PGPE

.

35

40

45

0

5

10

15

20

Cost Value

Frequency

Chance

MV

CVaR

Average Cost

(b)

C-PGAE

.

Risk

C-PGPE

C-PGAE

Cost

26

.

91

˘

0

.

09

25

.

80

˘

0

.

14

CVaR

α

23

.

07

˘

0

.

26

23

.

08

˘

0

.

24

MV

24

.

23

˘

0

.

23

0

.

75

˘

0

.

26

Chance

26

.

34

˘

0

.

16

23

.

07

˘

0

.

26

(c) Avg. Ret. (5 runs, mean

˘

std).

Risk constraints on Swimmer. In Figure 3, we show the empirical distributions of costs over 100 trajectories of the learned (hyper)policies via C-PGPE and C-PGAE . This experiment considers the cost-based version of the Swimmer-v4 MuJoCo (Todorov et al., 2012) environment, with a single constraint over the actions (see Appendix H), for which we set b ' 50 . The experimental results show that C-PGPE learns a hyperpolicy paying less cost when using risk measures compared to average cost, with the smallest costs attained by CVaR α . C-PGAE shows similar results, although the difference between CVaR α or the Chance constraints and average cost constraints are not very significant. Notice that, the minimum amount of cost is obtained using MV constraints even if the learned policy exhibits poor performances (Table 3c). In all the other cases, both C-PGPE and C-PGAE learns (hyper)policies exhibiting similar performance scores.

6 Conclusions

In this work, we proposed a general framework to address continuous CRL problems via primal-dual policy-based algorithms, leveraging on an alternate ascent-descent approach. Our explorationagnostic proposal C-PG exhibits dimension-free global last-iterate convergence guarantees, under the standard (weak) gradient domination assumption. Furthermore, we introduced C-PGAE and C-PGPE , the action and parameter-based versions of C-PG which enable embedding risk-based constraints, enlarging the capabilities of our framework in addressing constrained real-world problems. Future works should focus on matching the sample complexity lower bound prescribed by (Vaswani et al., 2022) and devising algorithms with the same rates of C-PG with a single time-scale.

References

Richard S. Sutton and Andrew G. Barto. Reinforcement learning: an introduction . MIT Press, 2018.

Amarildo Likmeta, Alberto Maria Metelli, Andrea Tirinzoni, Riccardo Giol, Marcello Restelli, and Danilo Romano. Combining reinforcement learning with rule-based controllers for transparent and general decision-making in autonomous driving. Robotics Auton. Syst. , 131:103568, 2020.

Eiji Uchibe and Kenji Doya. Constrained reinforcement learning from intrinsic and extrinsic rewards. In IEEE International Conference on Development and Learning , pages 163-168. IEEE, 2007.

Eitan Altman. Constrained Markov Decision Processes . CRC Press, 1999.

Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by PID lagrangian methods. In Proceedings of the International Conference on Machine Learning (ICML) , volume 119 of Proceedings of Machine Learning Research , pages 9133-9143. PMLR, 2020.

Dongsheng Ding, Kaiqing Zhang, Tamer Basar, and Mihailo Jovanovic. Natural policy gradient primal-dual method for constrained markov decision processes. Advances in Neural Information Processing Systems (NeurIPS) , 33:8378-8390, 2020.

Donghao Ying, Yuhao Ding, and Javad Lavaei. A dual approach to constrained markov decision processes with entropy regularization. In International Conference on Artificial Intelligence and Statistics (AISTATS) , volume 151 of Proceedings of Machine Learning Research , pages 1887-1909. PMLR, 2022.

Dongsheng Ding, Chen-Yu Wei, Kaiqing Zhang, and Alejandro Ribeiro. Last-iterate convergent policy gradient primal-dual methods for constrained mdps. Advances in Neural Information Processing Systems (NeurIPS) , 36, 2024.

Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research , 18(1): 6070-6120, 2017.

R Tyrrell Rockafellar and Stanislav Uryasev. Optimization of conditional value-at-risk. Journal of risk , 2:21-42, 2000.

Harry M Markowitz and G Peter Todd. Mean-variance analysis in portfolio choice and capital markets , volume 66. John Wiley & Sons, 2000.

Duan Li and Wan-Lung Ng. Optimal dynamic portfolio selection: Multiperiod mean-variance formulation. Mathematical Finance , 10(3):387-406, 2000.

Marc Peter Deisenroth, Gerhard Neumann, and Jan Peters. A survey on policy search for robotics. Foundations and Trends in Robotics , 2(1-2):1-142, 2013.

Jan Peters and Stefan Schaal. Policy gradient methods for robotics. In IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 2219-2225. IEEE, 2006.

Benjamin Gravell, Peyman Mohajerin Esfahani, and Tyler Summers. Learning optimal controllers for linear systems with multiplicative noise via policy gradient. IEEE Transactions on Automatic Control , 66(11):5283-5298, 2020.

Kamyar Azizzadenesheli, Yisong Yue, and Animashree Anandkumar. Policy gradient in partially observable environments: Approximation and convergence. arXiv preprint arXiv:1810.07900 , 2018.

Mohammad Ghavamzadeh and Yaakov Engel. Bayesian policy gradient algorithms. Advances in Neural Information Processing Systems (NeurIPS) , 19, 2006.

Alessandro Montenegro, Marco Mussi, Alberto Maria Metelli, and Matteo Papini. Learning optimal deterministic policies with stochastic policy gradients. In Proceedings of the International Conference on Machine Learning (ICML) . PMLR, 2024.

Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning , 8:229-256, 1992.

Jonathan Baxter and Peter L. Bartlett. Infinite-horizon policy-gradient estimation. Journal of Artificial Intelligence Research , 15:319-350, 2001.

Frank Sehnke, Christian Osendorfer, Thomas Rückstieß, Alex Graves, Jan Peters, and Jürgen Schmidhuber. Parameter-exploring policy gradients. Neural Networks , 23(4):551-559, 2010.

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of the International Conference on Machine Learning (ICML) , volume 70 of Proceedings of Machine Learning Research , pages 22-31. PMLR, 2017.

Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization. In International Conference on Learning Representations (ICLR) , 2019.

Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, Zhaoran Wang, and Mihailo R. Jovanovic. Provably efficient safe exploration via primal-dual policy optimization. In International Conference on Artificial Intelligence and Statistics (AISTATS) , volume 130 of Proceedings of Machine Learning Research , pages 3304-3312. PMLR, 2021.

Qinbo Bai, Amrit Singh Bedi, Mridul Agarwal, Alec Koppel, and Vaneet Aggarwal. Achieving zero constraint violation for constrained reinforcement learning via primal-dual approach. In AAAI Conference on Artificial Intelligence , pages 3682-3689. AAAI Press, 2022.

Qinbo Bai, Amrit Singh Bedi, and Vaneet Aggarwal. Achieving zero constraint violation for constrained reinforcement learning via conservative natural policy gradient primal-dual algorithm. In AAAI Conference on Artificial Intelligence , pages 6737-6744. AAAI Press, 2023.

Egor Gladin, Maksim Lavrik-Karmazin, Karina Zainullina, Varvara Rudenko, Alexander V. Gasnikov, and Martin Takác. Algorithm for constrained markov decision process with linear convergence. In International Conference on Artificial Intelligence and Statistics (AISTATS) , volume 206 of Proceedings of Machine Learning Research , pages 11506-11533. PMLR, 2023.

Tao Liu, Ruida Zhou, Dileep Kalathil, PR Kumar, and Chao Tian. Policy optimization for constrained mdps with provable fast global convergence. arXiv preprint arXiv:2111.00552 , 2021.

Dongsheng Ding, Kaiqing Zhang, Jiali Duan, Tamer Basar, and Mihailo R. Jovanovic. Convergence and sample complexity of natural policy gradient primal-dual methods for constrained mdps. CoRR , abs/2206.02346, 2022.

Dimitri P Bertsekas. Constrained optimization and Lagrange multiplier methods . Academic press, 2014.

Junchi Yang, Negar Kiyavash, and Niao He. Global convergence and variance reduction for a class of nonconvex-nonconcave minimax problems. In Advances in Neural Information Processing Systems (NeurIPS) , volume 33, pages 1153-1165, 2020.

Santiago Paternain, Luiz F. O. Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro. Constrained reinforcement learning has zero duality gap. In Advances in Neural Information Processing Systems (NeurIPS) , pages 7553-7563, 2019.

Rui Yuan, Robert M Gower, and Alessandro Lazaric. A general sample complexity analysis of vanilla policy gradient. In International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 3332-3380. PMLR, 2022.

Saeed Masiha, Saber Salehkaleybar, Niao He, Negar Kiyavash, and Patrick Thiran. Stochastic second-order methods improve best-known sample complexity of sgd for gradient-dominated functions. Advances in Neural Information Processing Systems (NeurIPS) , 35:10862-10875, 2022.

Ilyas Fatkhullin, Anas Barakat, Anastasia Kireeva, and Niao He. Stochastic policy gradient methods: Improved sample complexity for fisher-non-degenerate policies. In Proceedings of the International Conference on Machine Learning (ICML) , volume 202 of Proceedings of Machine Learning Research , pages 9827-9869. PMLR, 2023.

Jalaj Bhandari and Daniel Russo. Global optimality guarantees for policy gradient methods. Operations Research , 2024.

Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In Proceedings of the International Conference on Machine Learning (ICML) , pages 6820-6829. PMLR, 2020.

Matteo Papini, Matteo Pirotta, and Marcello Restelli. Smoothing policies and safe policy gradients. Machine Learning , 111(11):4081-4137, 2022.

Lorenzo Bisi, Davide Santambrogio, Federico Sandrelli, Andrea Tirinzoni, Brian D. Ziebart, and Marcello Restelli. Risk-averse policy optimization via risk-neutral policy optimization. Artif. Intell. , 311:103765, 2022.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR) , 2015.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 5026-5033. IEEE, 2012.

Sharan Vaswani, Lin Yang, and Csaba Szepesvári. Near-optimal sample complexity bounds for constrained mdps. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerík, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. CoRR , abs/1801.08757, 2018.

Yinlam Chow, Ofir Nachum, Edgar A. Duéñez-Guzmán, and Mohammad Ghavamzadeh. A lyapunovbased approach to safe reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS) , pages 8103-8112, 2018.

Ming Yu, Zhuoran Yang, Mladen Kolar, and Zhaoran Wang. Convergent policy optimization for safe reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS) , pages 3121-3133, 2019.

Yongshuai Liu, Jiaxin Ding, and Xin Liu. IPO: interior-point policy optimization under constraints. In AAAI Conference on Artificial Intelligence , pages 4940-4947. AAAI Press, 2020.

Tengyu Xu, Yingbin Liang, and Guanghui Lan. CRPO: A new approach for safe reinforcement learning with convergence guarantee. In Proceedings of the International Conference on Machine Learning (ICML) , volume 139 of Proceedings of Machine Learning Research , pages 11480-11491. PMLR, 2021.

Felisa J. Vázquez-Abad, Vikram Krishnamurthy, Katerine Martin, and Irina Baltcheva. Self learning control of constrained markov chains - a gradient approach. In IEEE Conference on Decision and Control (CDC) , pages 1940-1945. IEEE, 2002.

Shalabh Bhatnagar and K. Lakshmanan. An online actor-critic algorithm with function approximation for constrained markov decision processes. J. Optim. Theory Appl. , 153(3):688-708, 2012.

Tianqi Zheng, Pengcheng You, and Enrique Mallada. Constrained reinforcement learning via dissipative saddle flow dynamics. In Asilomar Conference on Signals, Systems, and Computers (ACSSC) , pages 1362-1366. IEEE, 2022.

Ted Moskovitz, Brendan O'Donoghue, Vivek Veeriah, Sebastian Flennerhag, Satinder Singh, and Tom Zahavy. Reload: Reinforcement learning with optimistic ascent-descent for last-iterate convergence in constrained mdps. In Proceedings of the International Conference on Machine Learning (ICML) , volume 202 of Proceedings of Machine Learning Research , pages 25303-25336. PMLR, 2023.

Yu-Guan Hsieh, Franck Iutzeler, Jérôme Malick, and Panayotis Mertikopoulos. On the convergence of single-call stochastic extra-gradient methods. In Advances in Neural Information Processing Systems (NeurIPS) , pages 6936-6946, 2019.

Yin-Lam Chow and Marco Pavone. Stochastic optimal control with dynamic, time-consistent risk constraints. In American Control Conference (ACC) , pages 390-395. IEEE, 2013.

Vivek S. Borkar and Rahul Jain. Risk-constrained markov decision processes. IEEE Trans. Autom. Control. , 59(9):2574-2579, 2014.

Qiyuan Zhang, Shu Leng, Xiaoteng Ma, Qihan Liu, Xueqian Wang, Bin Liang, Yu Liu, and Jun Yang. Cvar-constrained policy optimization for safe reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems , 2024.

Aviv Tamar, Yonatan Glassner, and Shie Mannor. Policy gradients beyond expectations: Conditional value-at-risk. CoRR , abs/1404.3862, 2014.

Lorenzo Bisi, Luca Sabbioni, Edoardo Vittori, Matteo Papini, and Marcello Restelli. Risk-averse trust region optimization for reward-volatility reduction. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI) , pages 4583-4589. ijcai.org, 2020.

Yudong Luo, Guiliang Liu, Pascal Poupart, and Yangchen Pan. An alternative to variance: Gini deviation for risk-averse policy gradient. In Advances in Neural Information Processing Systems (NeurIPS) , 2023.

Richard S. Sutton, David A. McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems (NeurIPS) , pages 1057-1063. The MIT Press, 1999.

Maher Nouiehed, Maziar Sanjabi, Tianjian Huang, Jason D Lee, and Meisam Razaviyayn. Solving a class of non-convex min-max games using iterative first order methods. Advances in Neural Information Processing Systems (NeurIPS) , 32, 2019.

Brian DO Anderson and John B Moore. Optimal control: linear quadratic methods . Courier Corporation, 2007.

Appendix

Table of Contents

A Pseudo-codes

Algorithm 2: C-PGPE .

B Related Works

Policy Optimization Approaches for Constrained Reinforcement Learning. Policy Optimization-based algorithms for Constrained Reinforcement Learning mostly follow primal -only or primal-dual approaches. Primal -only algorithms (Dalal et al., 2018; Chow et al., 2018; Yu et al., 2019; Liu et al., 2020; Xu et al., 2021) avoid considering dual variables by focusing on the design of the objective function and by designing the update rules for the policy at hand incorporating the constraint satisfaction part.

The main benefit of employing primal-only algorithms lies in the fact that there is no need to consider another variable to learn, and therefore, no need to tune its learning rate. However, few of the existing methods establish global convergence to an optimal feasible solution. For instance, Xu et al. (2021) propose CRPO, an algorithm employing an unconstrained policy maximization update taking into account the reward when all the constraints are satisfied, while leveraging on-policy minimization updates in the direction of violated constraint functions. Moreover, it exhibits average global convergence guarantees for the tabular setting. On the other hand, primal-dual algorithms (Chow et al., 2017; Achiam et al., 2017; Tessler et al., 2019; Stooke et al., 2020; Ding et al., 2020, 2021; Bai et al., 2022; Ying et al., 2022; Bai et al., 2023; Gladin et al., 2023; Ding et al., 2024) are the most commonly used and investigated. Indeed, the effectiveness of using the primal-dual approach is justified by Paternain et al. (2019), which states that this kind of approach has zero duality gap under Slater's condition when optimizing over the space of all the possible stochastic policies. Among the reported works, Stooke et al. (2020) propose PID Lagrangian, a method to update the dual variable, smoothing the oscillations around the threshold value of the costs during the learning. The practical strength of such a method is that can be paired with any of the existing policy optimization methods. The other cited works are treated in details in the next paragraph.

Lagrangian-based Policy Search Convergence Guarantees. A lot of research effort has been spent in studying the convergence guarantees for primal-dual policy optimization methods. In this field, the goal is to ensure last-iterate convergence guarantees showing rates that are dimension-free, i.e., not relying on the state and action spaces' dimensions, and working with multiple constraints. In the rest of this paragraph, we talk about single time-scale algorithms when the methods at hand prescribe the usage of the same step sizes for both the primal and dual variables' updates. VázquezAbad et al. (2002) and Bhatnagar and Lakshmanan (2012) propose primal-dual policy gradient-based methods built upon distinct time-scales and relying on nested loops. Such methods only show asymptotic convergence guarantees. Chow et al. (2017) propose two primal-dual methods ensuring asymptotic convergence guarantees. The peculiarity of those methods lies in the fact that their notion of CMDP encapsulates risk-based constraints, introducing an additional learning variable. Their algorithms have guarantees of asymptotic convergence to stationary points. The recent works by Zheng et al. (2022) and Moskovitz et al. (2023) also propose methods ensuring asymptotic global convergence guarantees. These methods exploit occupancy-measure iterates rather than policy iterates. Ding et al. (2020) propose NPG-PD, which relies on a natural policy gradient approach and, under Slater's assumption, ensures dimension-dependent average -iterate global convergence guarantees in the single-constrained setting with a single time-scale and with exact gradients. This work has been extended by Ding et al. (2022), which strikes dimension-free rates, but still guaranteeing just average -rate convergence with exact gradients. However, sample-based versions of NPG-PD showing, under additional assumptions, the same convergence rates are provided by the authors. Another work ensuring an average -iterate rate is the one by Liu et al. (2021). The latter exhibits a convergence rate of order ˜ O p ϵ ´ 1 q , considering to act in tabular CMDPs with softmax policies and having access to exact graidents and to a generative model. Liu et al. (2021) propose also a sample-based version of their algorithm, keeping the same setting previously described, which ensures a convergence rate on average of order ˜ O p ϵ ´ 3 q . Both Ying et al. (2022) and Gladin et al. (2023) propose algorithms involving regularization. The proposed methods rely on natural policy-based subroutines and show dimension-dependent last-iterate global convergence guarantees, relying on two time-scales. These methods work also with multiple constraints. Finally, Ding et al. (2024) propose RPG-PD and OPG-PD, exhibiting last-iterate global convergence guarantees under Slater's condition in a singleconstraint setting. The former is a regularized version of the algorithm proposed by Ding et al. (2020), showing last-iterate global convergence at a sublinear rate. The latter leverages on the optimistic gradient method (Hsieh et al., 2019) to unlock a faster linear convergence rate. These methods show single time-scale dimension-dependent rates and both leverage on exact gradients. However, for

RPG-PD there exists an inexact version showing, under additional assumptions on the statistical and transfer errors and the relative condition number (Ding et al., 2024, Assumption 2), the same guarantees of the exact one. It is worth noticing that all the mentioned works just consider the action-based exploration approach for policy optimization, while the parameter-based one remains unexplored. For the sake of clarity, Table 3 shows a detailed comparison among our approach and the other presented methods exhibiting last-iterate global convergence guarantees.

Furthermore, Vaswani et al. (2022) have recently proposed a dimension-dependent lower bound for the sample complexity of O ` ϵ ´ 2 ˘ , assuming to be under the Slater condition and considering single-constrained CMDPs with finite state and action spaces.

Risk-Constrained Reinforcement Learning. For safety-critical problems, it is often insufficient to consider constraints solely on expected costs or utilities. Therefore, constraints are sometimes applied to risk measures over cost or utility functions (Chow and Pavone, 2013; Borkar and Jain, 2014; Chow et al., 2017; Zhang et al., 2024). The employed risk measures are those most commonly applied in risk-averse RL (Tamar et al., 2014; Bisi et al., 2020, 2022; Luo et al., 2023). For instance, Tamar et al. (2014) consider the Conditional Value at Risk (CVaR α , Rockafellar and Uryasev, 2000), while Bisi et al. (2022) consider a unified formulation embracing several risk measures, including the CVaR α and the Mean-Variance (MV, Markowitz and Todd, 2000; Li and Ng, 2000). More details on risk measures are discussed in Appendix C. Among the works on Risk-Constrained RL, the one by Chow et al. (2017) considers both CVaR α constraints and chance constraints over cost functions. In particular, the authors propose a trajectory-based policy gradient algorithm and an actor-critic one, both primal-dual methods. For what concerns the CVaR α -constrained problem, the authors incorporate into the constrained optimization problem the real-valued variable associated with the CVaR α . This leads to three variables to learn, requiring algorithms with three time-scales. Finally, the recent work by Zhang et al. (2024) considers an extension of the CPO algorithm (Achiam et al., 2017) imposing constraints over the CVaR α of cost functions.

C On the Unified Risk Measure

In this section, we deepen the risk measures presented in our work, showing also how to select the functions f and g to obtain the desired risk measure. For what follows, we define Z as a finite-mean random variable that has F Z p¨q as the cumulative distribution function. Formally:

For what follows, let such a random variable represent a loss.

C.1 Expected Cost and Mean-Variance

One of the most natural risk measures is represented directly by the expected cost on the trajectories induced by a (hyper)policy. This 'risk neutral' risk measure was introduced in Section 2 for both the PG exploration paradigms as J AC ,i and J PC ,i , where i is the cost function index.

However, in some scenarios, it may be desirable to minimize jointly the expected cost and its variance. To this end, it is possible to consider the Mean-Variance (MV, Markowitz and Todd, 2000; Li and Ng, 2000) risk measure which models this kind of objective. Indeed, the MV is a combination of the mean cost and its variance over the trajectories. The MV with parameter κ over the random variable Z , which represents a loss, is defined as follows:

C.2 Chance Constraints, Value at Risk, and Conditional Value at Risk

Value at Risk (VaR α ) and Conditional Value at Risk (CVaR α ) are two popular and strongly connected risk measures. In the following, we report their standard definition as provided by Rockafellar and Uryasev (2000) and Chow et al. (2017).

VaR α measures risk as the maximum cost that might be incurred with respect to a given confidence level α P p 0 , 1 q . Such a quantity models the potential loss and the probability that the loss will occur. In its classical definition, the VaR α is the worst-case loss associated with a probability and a time horizon. This risk metric is particularly useful when there is a well defined failure state. Formally:

CVaR α measures risk as the expected cost given that such cost is greater than or equal to VaR α , and provides a number of theoretical and computational advantages. Such a quantity represents the expected loss if the worst-case threshold is passed (i.e., beyond the VaR α breakpoint). Formally:

Another risk measure, similar to the VaR α and the CVaR α , is the chance one. Considering chance constraints means to consider constraints to hold with high probability. Indeed, constraints are imposed on the probability that the loss random variable exceeds a certain threshold n :

C.3 Unified Risk Measure Mapping

Here we introduce an objective for writing an optimization problem for different risk measures (Bisi et al., 2022). For every i P J 0 , U K , we introduce the functions f i : R 2 Ñ R and g i : R Ñ R in order to express the unified risk-minimization objective function over a risk measure on C i as:

Thus, by fixing a policy parameter configuration θ , we obtain an AB unified risk measure over C i as follows:

In order to switch to the PB exploration paradigm, we need to consider also the expectation over the parameter configuration of the underlying policy. Indeed, by fixing a hyperparameter configuration ρ , we can define:

By selecting the functions f i and g i we can consider distinct risk measures to minimize, as we show in Table 4.

The focus of this section is to derive the mapping between the functions f and g and the desired risk measure as shown in Table 4. In particular, we consider just the AB exploration scenario, since then passing to the PB one is straightforward. In this section, we consider a generic cost function c : S ˆ A Ñ r 0 , 1 s with its cumulative cost index over a trajectory τ which is C p τ q ' ř T ´ 1 t ' 0 γ t c p s τ,t , a τ,t q . In what follows, with a little abuse of notation, we will write the risk measures with also a dependence on the policy parameters θ which induce trajectories τ .

Mean Cost. This is the simplest case, indeed, we can express the minimization problem of the mean cost as:

Thus, it suffices to select f p C p τ q , η q ' C p τ q and g p η q ' 0 to make the minimization problem fit with the form of the unified cost minimization formulation.

Conditional Value at Risk. For what concern the CVaR α , we can start from its formulation:

Thus, we need to select f p C p τ q , η q ' 1 1 ´ α p C p τ q ´ η q ` and g p η q ' η to complete the mapping to the unified cost minimization objective.

Mean Variance. For the MV κ p θ q objective, we need to make some preliminary observations. In particular, for a generic finite-mean random variable X , we have that:

Moreover, by Fenchel duality, we have the following:

Given that, we can start the derivation from the minimization of the Mean Variance:

"

"

**

Thus, to complete the mapping, we need to select

and g p η q ' κη 2 .

Chance. For characterizing the chance constraints, we start by expressing the probability of an event A as the expected value of the indicator function:

P p A q ' E r ✶ t A us . (32) Thus, having to minimize a chance measure on the cost with a parameter n , we can rewrite it as:

Thus, to complete the mapping to the unified cost minimization objective, it suffices to select f p C p τ q , η q ' ✶ t C p τ q ě n u and g p η q ' 0 . D Estimators

Here, we provide all the explicit estimators' forms for both C-PGAE and C-PGPE and for each risk measure. For the sake of clarity, we report the mapping between the f i and g i functions and the risk measures in Table 4.

t

p

q ě

u

The general Lagrangian function for the problem in Equation (11) is the following:

For what follows we are going to compute the following gradients:

∇ υ r L : ,ω p υ , λ , η q ;

∇ η r L : ,ω p υ , λ , η q .

∇ λ r L : ,ω p υ , λ , η q ;

Before entering into the details of the estimators, we show the general forms of the gradients of the Lagrangian.

General Gradient w.r.t. υ . Notice that the following holds:

thus, in what follows, for the gradient w.r.t. υ we focus on the single terms ∇ υ J : ,u p υ , η u q for every u P J 0 , U K .

General Gradient w.r.t. λ . The general gradient w.r.t. λ has the form:

General Gradient w.r.t. η . As done for the gradient w.r.t. υ , the following holds:

Thus, also in this case, we will focus on the single terms ∇ η J : ,u p υ , η u q for every u P J 0 , U K .

D.1 Parameter-based Algorithm: C-PGPE

In the following we consider a generic hyperpolicy ν ρ and a generic parameterization ρ . Before starting with the derivations, we report the definition of J P ,i p ρ , η i q :

for every i P J 0 , U K .

D.1.1 Gradients w.r.t. Parameters

The first step of the derivation can be done via the log-trick for the parameter-based exploration paradigm as stated by Sehnke et al. (2010):

To switch to the sample-based versions of all the gradients, we consider the behavior of C-PGPE described in Section 4. In particular, we have the following:

Thus, to obtain the estimators for all the risk measures, it suffices to map the selection of the f i functions in Equation (42).

D.1.2 Gradients w.r.t. Lagrangian Multipliers

Given the general gradient w.r.t. λ of the regularized Lagrangian r L P ,ω in the parameter-based scenario, the partial derivative w.r.t. λ i is the following:

This is defined for any i U .

In order to switch to the sample-based version of the partial derivative, we consider the behavior of C-PGPE as described in Section 4. In particular, we have the following:

Thus, to obtain the estimator for all the risk measures, it suffices to map the choices of f i and g i in Equation (44).

D.1.3 Gradients w.r.t. Risk Parameters

For what concern the gradients w.r.t. η , we have to enumerate the mappings shown in Table 4. Notice that, when the employed risk measure is the expected cost or the chance , the estimator of the gradient w.r.t. η is not needed. Indeed, in the unified risk measure formulation such mappings do not depend on η . As shown at the beginning of this section, we can just focus on the terms ∇ η J P ,i p ρ , η i q , and in particular on the partial derivative ∇ η i J P ,i p ρ , η i q , that exhibits the common form:

Mean Variance. In this case we have that f i p C i p τ q , η i q ' p 1 ´ 2 κ i η i q C i p τ q ` κ i C i p τ q 2 and that g i p η i q ' κ i η 2 i . Thus, from Equation (46), we get the following:

Considering the behavior of C-PGPE described in Section 4, we obtain the following sample-based version:

Conditional Value at Risk. In this case we have that f i p C i p τ q , η i q ' 1 1 ´ α i p C i p τ q ´ η i q ` and that g i p η i q ' η i . As also shown by Chow et al. (2017), due to the presence of the non-differentiable term p¨q ` , we need to resort to sub-differentiability theory. The following holds:

Thus, from Equation (46), we can write what follows:

with q P r 0 , 1 s .

In particular, for q ' 1 , we obtain the following partial derivative:

Finally, according to the C-PGPE behavior described in Section 4, we obtain the following samplebased version:

With a little abuse of notation, we will use p ∇ η i J P ,i p ρ , η i q ' p B η i J P ,i p ρ , η i q for the η update in the case in which the CVaR α risk measure is employed.

D.2 Action-based Algorithm: C-PGAE

In the following we consider a generic policy π θ and a generic parameterization θ . Before starting with the derivations, we report the definition of J A ,i p θ , η i q :

for every i P J 0 , U K .

D.2.1 Gradients w.r.t. Parameters

Also in this case, the first step of the derivation can be done via the log-trick, which provides an analogous result of the Policy Gradient Theorem (PGT, Sutton et al., 1999):

To switch to the sample-based versions of all the gradients, we generally resort to the Monte Carlo version of ∇ θ J A ,i p θ , η i q , obtaining a REINFORCE-like (Williams, 1992) estimator, that is:

Thus, to obtain the estimators for all the risk measures, it suffices to map the selection of the f i functions in Equation (60).

It is worth noticing that some of the choices for f i and g i allow to switch to a GPOMDP-like (Baxter and Bartlett, 2001) estimator, which suffers from less variance. This holds for the expected cost and mean variance risk measures.

Expected Cost GPOMDP-like Estimator. In this case f i p C i p τ q , η i q ' C i p τ q , thus we can obtain exactly the GPOMDP estimator:

Mean Variance GPOMDP-like Estimator. In this case f i p C i p τ q , η i q ' p 1 ´ 2 κ i η i q C i p τ q ` 2

κ i C i p τ q , thus we can obtain the GPOMDP estimator just for the C i p τ q part:

D.2.2 Gradients w.r.t. Lagrangian Multipliers

The result is the same as the one obtained for C-PGPE . The difference lies in the way in which trajectories are collected. The estimator for the partial derivative w.r.t. λ i is:

To obtain the estimator for all the risk measures, it suffices to map the choices of f i and g i as prescribed by Table 4.

D.2.3 Gradients w.r.t. Risk Parameters

As for the exploration-based case, also here we need to enumerate the mappings reported in Table 4. However, the expected cost and the chance risk measures do not depend on η , thus they are not treated. As shown at the beginning of the section, we can just focus on the ∇ η J A ,i p θ , η i q terms and, in particular, we consider the partial derivative ∇ η i J A ,i p θ , η i q . For it, we can recover the common form:

Mean Variance. In this case we have that f i p C i p τ q , η i q ' p 1 ´ 2 κ i η i q C i p τ q ` κ i C i p τ q 2 and that g i p η i q ' κ i η 2 i . Thus, from Equation (67), the following holds:

Considering the behavior of C-PGAE described in Section 4, we obtain the following sample-based version:

Conditional Value at Risk. In this case we have that f i p C i p τ q , η i q ' 1 1 ´ α i p C i p τ q ´ η i q ` and that g i p η i q ' η i . Here we face the same issues we have discussed in the corresponding C-PGPE part. With the same procedure, we obtain the following partial derivative:

Finally, according to the C-PGAE behavior described in Section 4, we obtain the following samplebased version:

Also in this case, we will use p ∇ η i J A ,i p ρ , η i q ' p B η i J A ,i p ρ , η i q for the η update in the case in which the CVaR α risk measure is employed.

E Proofs

Lemma E.1 (Regularization Bias on Saddle Points - 1) . Under Assumption 3.1, for every ω ě 0 , let p υ ˚ ω , λ ˚ ω q be a saddle point of L ω , it holds that:

Proof. From the fact that υ , λ is a saddle point of L , we have for every υ V and λ Λ :

p ˚ ω ˚ ω q ω P P

From the fact that p υ ˚ 0 , λ ˚ 0 q is a saddle point of L 0 , we have for every υ P V and λ P Λ :

By setting p υ , λ q Ð p υ ˚ ω , λ ˚ ω q in Equation (75) and p υ , λ q Ð p υ ˚ 0 , λ ˚ 0 q in Equation (74), we obtain: (76)

thus:

Lemma E.2 (Regularization Bias on Saddle Points - 2) . Under Assumption 3.1, for every ω ě 0 , it holds that:

Proof. The first inequality follows from the observation that L 0 p υ , λ q ě L ω p υ , λ q for every ω ě 0 . For the second inequality, let us denote as p υ ˚ ω , λ ˚ ω q the saddle point for L ω and let Λ ˚ ' t λ ˚ 0 , λ ˚ ω u . We have:

where we used Lemma E.1 to conclude that λ ˚ 2 λ ˚ 2

Lemma E.3 (Objective bound and Constraint violation) . Under Assumption 3.1, for every ω ě 0 , letting p υ ˚ ω , λ ˚ ω q be a saddle point of L ω , it holds that:

} 0 } 2 ě } ω } 2 .

Proof. Since p υ ˚ 0 , λ ˚ 0 q is a saddle point of L 0 , it holds that υ ˚ 0 is feasible and, consequently, L 0 p υ ˚ 0 , λ ˚ 0 q ' J 0 p υ ˚ 0 q . Moreover, let ω ą 0 : since p υ ˚ ω , λ ˚ ω q is a saddle point of L ω it holds that λ ˚ ω ' λ ˚ p υ ˚ ω q ' Π Λ ` 1 ω p J p υ ˚ ω q ´ b q ˘ ' 1 ω p J p υ ˚ ω q ´ b q ` , since 1 ω }p J p υ ˚ ω q ´ b q ` } 2 ď ω ´ 1 ? UJ max . Thus, we have:

By summing 1 ω }p J p υ ˚ ω q ´ b q ` } 2 2 to all members, we have:

Now taking the first and last member, we conclude:

Since 1 ω }p J p υ ˚ ω q ´ b q ` } 2 2 ě 0 and plugging the latter inequality into the third member of (89) we obtain:

Lemma E.4 (Weak ψ -Gradient Domination on H ω p υ q ) . Under Assumption 3.2, if ω ą 0 , for every υ P V and λ P Λ , it holds that:

Proof. If ω ą 0 , the dual variable exist finite since the maximization problem over λ is concave:

Thus, we have from Lemma E.7 that ∇ υ H ω p υ q ' ∇ υ L ω p υ , λ q | λ ' λ ˚ p υ q and by Assumption 3.2 we have the following:

Lemma E.5. Let ω ą 0 and υ P V . The following statements hold:

L ω p υ , ¨q is ω -smooth, i.e., for every λ , λ 1 P Λ it holds that:

L ω p υ , ¨q satisfies the PL condition, i.e., for every λ P Λ it holds that:

L ω p υ , ¨q satisfies the error bound (EB) condition, i.e., for every λ , λ 1 P Λ it holds that:

where λ ˚ p υ q ' arg max λ P Λ L ω p υ , λ q .

L ω p υ , ¨q satisfies the quadratic growth (QG) condition, i.e., for every λ , λ 1 P Λ it holds that:

Proof. For the first property, it is enough to observe that L ω is twice differentiable in λ and that its Hessian is ω I . For the second property, we observe that L ω is quadratic in λ and, consequently it satisfies the PL condition with parameter ω :

For the third and fourth properties, we refer to Lemma A.1 of Yang et al. (2020).

Lemma E.6. Let ω ą 0 . For every υ P V , it holds that:

Proof. Let us consider the following derivation:

having exploited the fact that, from the saddle point property, L ω p υ ˚ ω , λ ˚ ω q ď L ω p υ , λ ˚ ω q and, then, Lemma E.5.

Lemma E.7. Let ω ą 0 . The following statements hold:

H ω is L H -smooth, i.e., for every υ , υ 1 P V , it holds that:

where L H : ' L 2 ` L 2 1 ω .

For every υ , υ 1 P V we have ∇ υ H ω p υ q ' ∇ υ L ω p υ , λ q | λ ' λ ˚ p υ q , where λ ˚ p υ q ' arg max λ P Λ L ω p υ , λ q .

Proof. The first and second statements follow from Lemma A.5 of Nouiehed et al. (2019).

Theorem 3.1 (Objective Function Gap and Constraint Violation) . Let ϵ ą 0 . Under Assumption 3.1, if P k ď ϵ , it holds that:

Proof. Since P k ď ϵ , it follows that a k ď ϵ and, consequently, 0 ď E r H ω p υ k q ´ H ˚ ω s ď ϵ . We start by bounding the norm of the dual variables:

where we applied the triangular inequality and Lemma E.6. The projection Π Λ is such that λ ˚ p υ q ' Π Λ ` 1 ω p J p υ q ´ b q ˘ ' 1 ω p J p υ q ´ b q ` and, consequently, we have:

By the last inequality, together with Lemma E.3, having applied the expectation on both sides:

Recalling that:

For the objective function bound, let us consider the following derivation. By definition of H ω p υ q and λ ˚ p υ q we have:

Taking the expectation on both sides and upper bounding } λ ˚ ω } with } λ ˚ 0 } from Lemma E.1:

The result is obtained by applying Lemma E.3 as follows:

Theorem 3.2 (Convergence of P K ) . Under Assumptions 3.2, 3.3, 3.4, for χ ă 1 { 5 , sufficiently small ϵ and ω , and a choice of constant learning rates ζ υ , ζ λ , we have P K p χ q ď ϵ ` β 1 { α 1 whenever: 8

K ' O p ω ´ 1 ϵ ´ 2 ψ ´ 1 q if ψ P r 1 , 2 q and the gradients are exact (i.e., V υ ' V λ ' 0 );

K ' O p ω ´ 1 log p ϵ ´ 1 qq if ψ ' 2 and the gradients are exact (i.e., V υ ' V λ ' 0 );

K ' O p ω ´ 3 ϵ ´ 4 ψ ` 1 q if ψ P r 1 , 2 s and the gradients are estimated (i.e., V υ ' O p ω ´ 2 q and V λ ' O p 1 q ).

Proof. The proof is subdivided into several parts. We will omit the ω subscript for notational easiness. Let us focus on a specific iteration k P N .

Part I: bounding the a k term. Let us start with the a k term:

where the first line is due to the fact that the function H ω is L H -smooth (Lemma E.7), the last inequality is due to the update rule of υ . Now, we apply the expected value on both sides of the inequality and we use the fact that the gradient estimation is unbiased and has variance bounded by V υ :

where F k ´ 1 is the filtration associated with all events realized up to interaction k ´ 1 . We recall that:

and that V ar p ∇ υ L ω p υ k , λ k q ď V υ by Assumption 3.4. Thus, selecting ζ υ ,k ď 1 { L H , we have that:

Thus, the following holds:

Thus, we have obtained:

holding via the selection of ζ υ ,k ď 1 { L H . Notice that, from A ⃝ the following directly follows:

Part II: bounding the b k term. We are ready to analyze the b k term. Recall that for ridge regularization of the Lagrangian function presented in the main paper, we have that L ω is ω -smooth and fulfills the PL condition with constant ω , as shown in Lemma E.5. Since L is a quadratic function of λ and λ ˚ p υ k ` 1 q P Λ , we have that considering the non-projected λ k ` 1 can only increase the distance. Thus, we will ignore projection for the rest of the proof. We have:

Consider that:

that is possible under Assumption 3.3 (i.e., L ω is L 2 -smooth) and due to the update rules we are considering. Now, by applying the expectation on both sides, we obtain the following:

where the last line follows by selecting ζ λ ,k ď 1 { ω . Since L ω enjoys the PL condition w.r.t λ with constant ω , for every pair p υ , λ q we have:

By applying the PL condition:

where we enforce 1 ´ ζ λ ,k 2 ω ě 0 , i.e., ζ λ ,k ď 2 { ω . However, we do not have a proper recursive term, thus consider the following:

A bound on D ⃝ has already been derived, so let us bound the term C ⃝ :

again because of Assumption 3.3 and the update rule. Now, as usual, we consider the expectation conditioned to the filtration F k ´ 1 and the properties of the variance, to obtain:

having set ζ υ ,k ď 1 { L 2 . We are finally able to conclude the bound of the term B ⃝ :

Now we apply the bounds on C ⃝ and D ⃝ (the latter is from Eq. 136), obtaining:

that is the second fundamental term.

Part III: bounding the potential function P k p χ q . Before going on, we recall that so far we enforced: ζ υ ,k ď 1 { L H (since L H ě L 2 ) and ζ λ ,k ď 1 { ω , for every t P J K K . What we want to bound here is the potential function P k ` 1 p χ q ' a k ` 1 ` χb k ` 1 . Using the final results of Part I and Part II:

¸

Now we can re-arrange the terms by noticing that:

where the last inequality holds by Young's inequality. Then we can write what follows:

ˆ

ˆ

˙˙˙

Let us now proceed to bound } ∇ υ L ω p υ k , λ k q ´ ∇ υ H ω p υ k q} 2 2 . By Lemma E.7, we have that ∇ υ H ω p υ q ' ∇ υ L ω p υ , λ ˚ p υ qq for every λ ˚ p υ q P arg max λ P Λ L ω p υ , λ q , thus we can write:

since Assumption 3.3 holds.

For a fixed value of υ , by Lemma E.5 it follows that L ω p υ , ¨q satisfies the quadratic growth condition (since it satisfies the PL condition), for which the following holds:

and thus we have:

By applying the total expectation, it trivially follows:

Thus, we have:

Part IV: apply the ψ -gradient domination. Now we need to bound the term } ∇ H ω p υ k q} 2 2 . We consider Assumption 3.2 and we get: } ∇ υ H ω p υ k q} ψ 2 ě α 1 p H ω p υ k q ´ H ˚ q ´ β 1 . By defining r H ˚ : ' H ˚ ` β 1 { α 1 , we also have:

If we apply the total expectation on both sides of the inequality, we get:

which is achieved by a double application of Jensen's inequality, since z 2 { ψ is convex for ψ P r 1 , 2 s and z ě 0 , and the maximum is convex. Let us start from Equation (198):

We first enforce the negativity of ´ C . To this end: ¨

˛

Thus, it is enough to enforce 5 χ ´ 1 ď 0 ùñ χ ď 1 { 5 . We now plug in the gradient domination inequalities:

.

Now we introduce the symbol r a k : ' E H ω p υ k q ´ r H ˚ ' a k ´ β 1 { α 1 , to get:

For what follows, we call B the term that is multiplying b k :

Let refer to r a k ` χb k as r P t p χ q with χ P p 0 , 1 q . For the sake of clarity, we re-write our main inequality as:

r P t ` 1 p χ q ' r a k ` 1 ` χb k ` 1 ď r a k ` Bb k ` C p χb k q 2 ψ ´ 2 1 ´ 2 ψ C max t 0 , r a k ` χb k u 2 ψ ` V. (227) By choosing χ so that χb k ď 1 , i.e., χ ď 1 { max k Pr K s b k , we have:

Then, from Lemma F.1 having set a Ð r a k and b Ð χb k , we have:

To unfold the recursion, we need to ensure that B ` χC ď χ , which leads to a condition relating the two learning rates:

where we exploited ζ υ ,k ď 1 { L 2 and ζ λ ,k ď 2 { ω . Thus, we have:

Collecting all conditions on the learning rates, we have:

$

,

As a further simplification, let us observe that:

Denoting with r C ' : 2 1 ´ 1 ψ ζ υ ,k α 2 ψ 1 2 , we are going to study the recurrence:

Part V: Rates Computation

Part V(a): Exact gradients We consider the case r V ' 0 . Let us start with ψ ' 2 . From Lemma G.3, we have:

The inequality on K holds under the conditions:

,

where the first one derives from the hypothesis of Lemma G.3 and the other two from the conditions on the learning rates derived in the previous parts. We set:

`

Consider now ψ P r 1 , 2 q . We have from Lemma G.3:

Thus, the sample complexity becomes K ' O ω ´ 1 log 1 ϵ .

˘

holding under the same conditions as before. With the same choices of learning rates, we obtain the sample complexity K ' O ´ ω ´ 1 ϵ ´ 2 ψ ` 1 ¯ as sample complexity.

Part V(b): Estimated gradients We consider r V ą 0 . In this case, from Lemma G.5, we have:

We enforce both terms to be smaller or equal to ϵ { 2 . With the first one, we can evaluate the sample complexity:

Regarding the second one, we have:

By enforcing the relation between the two learning rates, we set ζ υ ,k ' O p ω 2 ζ λ ,k q . By enforcing the previous inequality, recalling that L 2 ď O p ω ´ 1 q and V υ ď O p ω ´ 2 q , we obtain ζ λ ' O p ωϵ 2 { ψ q , from which ζ υ ' O p ω 3 ϵ 2 { ψ q . Substituting these values into the sample complexity upper bound, we get (highlighting the terms possibly depending on ω ):

˜

¸

having bounded the sum at the denominator with the second addendum.

F Technical Lemmas

Lemma F.1. Let a P R , b ě 0 , and ψ P r 1 , 2 s . It holds that:

Proof. Let us consider the following derivation:

(268)

where the first inequality follows from p x ` y q 2 ψ ď 2 2 ψ ´ 1 p x 2 ψ ` y 2 ψ q for x, y ě 0 , from Holder's inequality; the second inequality from observing that 2 1 ´ 2 ψ p a ` b q 2 ψ ´ b 2 ψ ď p 2 1 ´ 2 ψ ´ 1 q b 2 ψ ď 0 for ´ b ă a ď 0 .

G Recurrences

In this section, we provide auxiliary results about convergence rate of a certain class of recurrences that will be employed for the convergence analysis of the proposed algorithms. Specifically, we study the recurrence:

for a ą 0 , b ě 0 , and ϕ P r 1 , 2 s . To this end, we consider the helper sequence:

The line of the proof follows that of Montenegro et al. (2024). Let us start showing that for sufficiently small a , the sequence ρ k upper bounds r k .

Proof. By induction on k . For k ' 0 , the statement holds since ρ 0 ' r 0 . Suppose the statement holds for every j ď k , we prove that it holds for k ` 1 :

where the first inequality holds by the inductive hypothesis and by observing that the function f p x q ' x ´ a max t 0 , x u ϕ is non-decreasing in x when a ď 1 ϕρ ϕ ´ 1 k . Indeed, if x ă 0 , then f p x q ' x , which is non-decreasing; if x ě 0 , we have f p x q ' x ´ ax ϕ , that can be proved to be non-decreasing

in the interval ' 0 , p aϕ q ´ 1 ϕ ´ 1 ı simply by studying the sign of the derivative. Thus, we enforce the following requirement to ensure that ρ k falls in the non-decreasing region:

So does r k by the inductive hypothesis.

Thus, from now on, we study the properties of the sequence ρ k . Let us note that, if ρ k is convergent, then it converges to the fixed-point ρ computed as follows:

having retained the positive solution of the equation only, since the negative one never attains the maximum max t 0 , ρ u . Let us now study the monotonicity properties of the sequence ρ k .

Lemma G.2. The following statements hold:

If r 0 ă ρ and a ď 1 ϕρ ϕ ´ 1 , then for every k ě 0 it holds that: ρ ě ρ k ` 1 ě ρ k .

Proof. The proof is analogous to that of (Montenegro et al., 2024, Lemma F.3).

From now on, we focus on the case in which r 0 ě ρ , since, as we shall see later, the opposite case is irrelevant for the convergence guarantees. We now consider two cases: b ' 0 and b ą 0 .

G.1 Analysis when b ' 0

From the policy optimization perspective, this case corresponds to the one in which the gradients are exact (no variance). Recall that here ρ ' 0 . We have the following convergence result.

Lemma G.3. If a ď 1 ϕr ϕ ´ 1 0 , r 0 ě 0 , and b ' 0 it holds that:

Proof. Since r 0 ě 0 ' ρ , from Lemma G.2, we know that ρ k ě 0 and, thus, max t 0 , ρ k u ' ρ k . For ϕ ' 1 , we have:

For ϕ 1 , 2

We proceed by induction. For k ' 0 , the statement hold since ρ 0 ' r 0 and r 0 ď p ϕa q ´ 1 ψ ´ 1 ď pp ϕ ´ 1 q a q ´ 1 ψ ´ 1 from the condition on the learning rate. Suppose the thesis holds for j ď k , we prove it for k ` 1 . ρ k ` 1 ď r 0 by monotonicity, and, from the inductive hypothesis:

We now prove that p˚q is non-positive:

having observed that:

G.2 Analysis for b ą 0

From the policy optimization perspective, this corresponds to the case in which the gradients are estimated, i.e., the variance is positive. In this case, we proceed considering the helper sequence:

We show that the sequence η k upper bounds ρ k when ρ 0 ' r 0 ě ρ .

Lemma G.4. If r 0 ą ρ and a ď 1 ϕr ϕ ´ 1 0 , then, for every k ě 0 , it holds that η k ě ρ k .

Proof. The proof is analogous to that of (Montenegro et al., 2024, Lemma F.4).

Thus, we can provide the convergence guarantee.

Lemma G.5. If a ď 1 ϕr ϕ ´ 1 0 , r 0 ě 0 , and b ą 0 it holds that:

Proof. By unrolling the recursion:

H Experimental Details and Additional Results

H.1 Experimental Details

H.1.1 Employed Policies and Hyperpolicies

Linear Gaussian Policy. A linear parametric gaussian policy π θ : S ˆ A Ñ ∆ p A q with variance σ 2 samples the actions as a t ' N p θ J s t , σ 2 I d S q , where s t is the observed state at time t and θ is the parameter vector.

Tabular Softmax Policy. A tabular softmax policy π θ : S ˆ A Ñ ∆ p A q with a temperature constant τ is such that:

where θ i,j is the parameter associated with the i -th state and the j -th action. Notice that the total number of parameters for this kind of policy is | S || A | .

Linear Deterministic Policy. A linear parametric deterministic policy µ θ : S ˆ A Ñ A samples the actions as a t ' θ J s t , where s t is the observed state at time t and θ is the parameter vector.

Gaussian Hyperpolicy. A parametric gaussian hyperpolicy ν ρ : R Ñ ∆ p Θ q with variance σ 2 samples the parameters θ for the underlying generic parametric policy π θ as θ t ' N p ρ , σ 2 I d R q , where ρ is the parameter vector for the hyperpolicy.

H.1.2 Environments

Discrete Grid World with Walls. Discrete Grid World with Walls (DGWW) is a simple discrete environment we employed to compare C-PGAE against the sample-based versions of NPG-PD (Ding et al., 2020, Appendix H) and RPG-PD (Ding et al., 2024, Appendix C.9). DGWW is a grid-like bidimensional environment in which an agent can assume only integer coordinate positions and in which an agent can play four actions stating whether to go up, right, left, or down. The goal is to reach the center of the grid performing the minimum amount of steps, begin the initial state uniformly sampled among the four vertices of the grid. The agent is rewarded negatively and proportionally to its distance from the center, where the reward is 0 . Around the goal state there is a 'U-shaped' obstacle with an opening on the top side. In particular, when the agent lands in a state in which the wall is present, it receives a cost of 1 , otherwise the cost signal is always equal to 0 . In our experiments, we employed a DGWW environment of such a kind, with | S | ' 49 , i.e., with each dimension with length equal to 7 .

Linear Quadratic Regulator with Costs. The Linear Quadratic Regulator (LQR, Anderson and Moore, 2007) is a continuous environment we employed in the regularization sensitivity study of C-PGAE and C-PGPE , and in the comparison among the same algorithms against the sample-based version of NPG-PD2 (Ding et al., 2022, Algorithm 1) and its ridge-regularized version RPG-PD2 (not provided by the authors, but designed by us). LQR is a dynamical system governed by the following state evolution:

where A P R d S ˆ d S and B P R d S ˆ d A .

In the standard version of the environment, the reward is computed at each step as:

where R R d S ˆ d S and Q R d A ˆ d A .

We modified this version of the LQR environment introducing costs. In particular, in our CostLQR , the state evolution is treated as in the original case, while the reward at step t is computed as:

where R P R d S ˆ d S . Moreover, we added a cost signal c which is computed as follows at every time step t :

t ' t t where Q P R d A ˆ d A .

In our experiments, we consider a CostLQR environment whose main characteristics are reported in Table 5.

Additionally, we considered a uniform initial state distribution in r´ 3 , 3 s and the following matrices:

r´

s

p´8

`8q

MuJoCo with Costs. For our experiments on risk minimization, we utilized environments from the MuJoCo control suite (Todorov et al., 2012), which offers a variety of continuous control environments. To tailor these environments to our specific requirements, we introduced a cost function that represents the energy associated with the control actions. In standard MuJoCo environments, a portion of the reward is typically calculated as the cost of the control action, which is proportional to the deviation of the chosen action from predefined action bounds. In our MuJoCo modification, at each time step we make the environment return a cost computed as:

where a min and a max are respectively the bounds for the minimum and maximum value for each component of the action vector. Then, the action min t max t a t , a min u , a max u is passed to the environment. In our experiment we consider Swimmer-v4 and Hopper-v4 MuJoCo environments, whose main features are summarized in Table 5.

H.2 Details for the comparison against the baselines in DGWW

In Section 5, we compare our proposal C-PGAE against the sample-based versions of NPG-PD (Ding et al., 2020, Appendix H) and RPG-PD (Ding et al., 2024, Appendix C9). The environment in which the methods are tested is the Discrete Grid World with Walls (DGWW, see Appendix H) with a horizon of T ' 100 . In this experiment, the methods aim at learning the parameters of a tabular softmax policy with 196 parameters, maximizing the trajectory reward while considering a single constraint on the average trajectory cost, for which we set a threshold b ' 0 . 2 . All the methods were run for K ' 3000 iterations with a batch size of N ' 10 trajectories per iteration, and with constant learning rates. In particular, for both C-PGAE and NPG-PD, we employed ζ θ ' 0 . 01 and ζ λ ' 0 . 1 , while for RPG-PD we selected ζ θ ' 0 . 01 and ζ λ ' 0 . 01 . For C-PGAE and RPG-PD we used a regularization constant ω ' 10 ´ 4 . All the details about the experimental setting are summarized in Table 6. We would like to stress that, as prescribed by the respective convergence theorems, we chose a two time-scales learning rate approach for C-PGAE and a single time-scale one for RPG-PD. Figure 2a shows the performance curves (i.e., the one associated with the objective function and the cost ones. As can be noticed, C-PGAE manages to strike the objective of the constrained optimization problem with less trajectories. Indeed, the sample-based NPG-PD requires to estimate the value and the action-value functions for all the states and state-action pairs, resulting in analyzing | S | ` | S || A | additional trajectories w.r.t. C-PGAE for every iteration of the algorithm. The sample-based RPG-PD also requires additional trajectories to be analyzed, which in practice, for a correct learning behavior, result to be the same in number to the extra ones analyzed by NPG-PD.

H.3 Details for the comparison against baselines in CostLQR

In Section 5, we compare our proposals C-PGAE and C-PGPE against the continuous sample-based version of NPG-PD (Ding et al., 2022, Algorithm 1) with works with generic policy parameterizations. In the following, we refer to this version of NPG-PD as NPG-PD2. Moreover, we added a ridgeregularized version of NPG-PD2, that we call RPG-PD2, to resemble the type of regularization we employed for our proposed methods. For all the regularized methods (i.e., C-PGAE , C-PGPE , and RPG-PD2) we selected as regularization constant ω ' 10 ´ 4 . The setting for this experiment considers a bidimensional LQR environment with a single cost over the provided actions (see Appendix H) and with a fixed horizon T ' 50 . Here, the methods aim at maximizing the average reward over trajectories, while keeping the average cost over trajectories under the threshold b ' 0 . 9 . In particular, C-PGAE learns the parameters of a linear gaussian policy with a variance σ 2 A ' 10 ´ 3 and employing a learning rate schedule governed by the Adam scheduler (Kingma and Ba, 2015) with ζ θ , 0 ' 0 . 001 and ζ λ , 0 ' 0 . 01 . C-PGPE learns the parameters of a gaussian hyperpolicy, with a variance σ 2 P ' 10 ´ 3 , which samples the parameters of a deterministic linear policy. It employs a learning rate schedule

governed by Adam too with ζ ρ , 0 ' 0 . 001 and ζ λ , 0 ' 0 . 01 . Both C-PGAE and C-PGPE were run for K ' 6000 iterations with a batch of N ' 100 trajectories per iteration. NPG-PD2 and RPGPD2 are both actor-critic methods which were run for K ' 1000 iterations with a batch size of N ' 600 trajectories per iteration. In particular, among the trajectories of the reported batch size, N 1 ' 500 were used for the inner critic-loop, while N 2 ' 100 for performance and cost estimations. The inner loop step size was selected constant, as prescribed by the original algorithm, and with a value α ' 10 ´ 5 . Furthermore, since such methods were designed for infinite-horizon discounted environments, we tested them on the same LQR as for C-PGAE and C-PGPE , but leaving T ' `8 and γ ' 0 . 98 (the effective horizon is p 1 ´ γ q ´ 1 ' 50 ). The step sizes for the primal and dual variables updates were governed by Adam with ζ θ , 0 ' 0 . 003 and ζ λ , 0 ' 0 . 01 . As for C-PGAE , both NPG-PD2 and RPG-PD2 aimed at learning the parameters of a linear gaussian policy, with variance σ 2 A ' 10 ´ 3 . All the details about this experiment are summarized in Table 7. Figure 2b reports the learning curves for the average return and the cost over trajectories. As can be seen, our methods manage to solve the constrained optimization problem at hand by leveraging on less trajectories. Indeed, NPG-PD2 and RPG-PD2 suffer the inner critic loop, which add additional trajectories to be analyzed per iteration (in this specific case N 1 ' 500 ). We would like to stress that the actor-critic methods were very sensible to the hyperparameters selection, especially the length and the step size of the inner loop.

H.4 Details for the risk-constrained experiment on Swimmer-v4

In Section 5, we presented an experiment comparing what happens when considering the two exploration approaches of C-PGAE and C-PGPE and on different risk-constrained optimization problems on the cost version of Swimmer-v4 (details in Appendix H). In particular, we considered such an environment with a horizon of T ' 100 . Both the algorithms were run for K ' 3000 iterations, with a batch size of N ' 100 trajectories per iteration, and with step sizes governed by the Adam scheduler. Moreover, they had a regularization constant ω ' 10 ´ 4 .

C-PGAE employed a linear gaussian policy with variance σ 2 A ' 1 . In Table 8, we list the characteristics for all the experiments with all the risk measures.

On the other hand, C-PGPE employed a gaussian hyperpolicy with variance σ 2 P ' 0 . 01 . Such an hyperpolicy sampled parameters for an underlying linear deterministic policy . In Table 9, we list the characteristics for all the experiments with all the risk measures.

As also highlighted in the main paper, C-PGPE delivers an hyperpolicy which samples (after having sampled the parameters for the underlying policy) actions whose costs are always under the fixed threshold. Moreover, by considering risk measures different from the average cost one, the empirical distribution of costs shows that these are way lighter w.r.t. the ones associated with the deployment of

'

'

'

an hyperpolicy learned considering average cost constraints. C-PGAE also shows results of this kind. However, the displacement between the final empirical cost distributions is not as marked as the one shown with C-PGPE . The exception is from the MV risk measure side, which seems to make C-PGAE learn a policy able to pay lighter costs, but resulting in poor-performing policy. All the other found (hyper)policies, instead, provide similar performance scores.

H.5 Risk-constrained experiment on Hopper-v4

Here, we present a similar experiment to the one shown on Swimmer-v4 . Also in this case, we compare what happens when considering the two exploration approaches of C-PGAE and C-PGPE and on different risk-constrained optimization problems, but on the cost version of Hopper-v4 (details in Appendix H). The experimental setting is quite the same of the one considered above (i.e., T ' 100 , K ' 3000 , N ' 100 , ω ' 10 ´ 4 ).

C-PGAE employed a linear gaussian policy with variance σ 2 A ' 1 . The characteristics for all the risk measures' experiments are presented in Table 10. For the learning rates schedules, we employed the Adam scheduler.

'

C-PGPE employed a gaussian hyperpolicy with variance σ 2 P ' 0 . 1 over an underlying linear deterministic policy . The characteristics for all the experiments with all the risk measures are summarized in Table 11. For the learning rates schedules, we employed the Adam scheduler.

'

Figure 5 shows the empirical distributions of costs over 100 trajectories of the learned (hyper)policies via C-PGPE and C-PGAE . C-PGPE shows a behavior that is quite the same as the one shown in the Swimmer-v4 environment. Indeed, by considering constraints on the CVaR α , the MV, or on the Chance, the learned hyperpolicy selects parameters for the underlying policy selecting actions that pay a way lighter cost w.r.t. the ones provided by an hyperpolicy learned by considering constraints on average costs. It is worth noticing that the lightest costs are observed when considering the CVaR α or the MV risk measures. In this case, C-PGAE shows a similar behavior to C-PGPE for what concern the distance in costs between the ones observed by learning considering constraints on the average cost and the ones observed by learning with other risk measures. The lightest costs here can be observed by learning with chance constraints, while, by considering constraints on the MV, the observed costs exceed the ones observed under average cost constraints. For what concern the performances of the learned (hyper)policies (see Table 4c, C-PGPE exhibit high-performing hyperpolicies under average cost or chance constraints, while the ones under CVaR α and MV shows similar (low) performance scores. On the other hand, for C-PGAE the learned policy under CVaR α exhibits the same goodperforming behavior as the one of the policy learned under average cost constraints.

H.6 Regularization Sensitivity Study

Here, we study the sensitivity of C-PGAE and C-PGPE w.r.t. the regularization term ω . We tried the algorithms on a bidimensional LQR environment which has been modified to output a cost signal based on the selected action. For the environment at hand, we considered a horizon T ' 50 . We run

20

40

60

80

100

0

10

20

30

Cost Value

Frequency

Chance

MV

CVaR

Average Cost

(a)

C-PGPE

60

80

100

0

10

20

30

Cost Value

Frequency

Chance

MV

CVaR

Average Cost

(b)

C-PGAE

Algo.

Cost

CVaR

α

MV

Chance

C-PGPE

200

.

18

˘

1

.

6

126

.

37

˘

1

.

71

122

.

28

˘

1

.

91

190

.

81

˘

1

.

83

C-PGAE

228

.

04

0

.

48

216

.

81

0

.

46

136

.

45

1

.

35

159

.

25

2

.

06

˘

˘

˘

(c) Avg. Ret. (5 runs, mean ˘ std).

P

both the algorithms for K ' 10 4 iterations, with a batch size N ' 100 trajectories per iteration, and with a varying regularization term such that ω P t 0 , 10 ´ 4 , 10 ´ 2 u . We considered a single constraint on the average trajectory cost, for which we set a threshold b ' 0 . 2 . For the step size schedules, we employed Adam (Kingma and Ba, 2015) with initial rates ζ ρ , 0 ' ζ θ , 0 ' 10 ´ 3 and ζ λ , 0 ' 10 ´ 2 . Moreover, in this specific experiment C-PGAE employed a linear gaussian policy with a variance σ 2 A ' 10 ´ 3 . On the other hand, C-PGPE employed a linear gaussian hyperpolicy with a variance σ 2 P ' 10 ´ 3 over a linear deterministic policy. The experimental environment is summarized in Table 12. Figures 7 and 6 show the Lagrangian curves, the performance ones (i.e., the one associated with the objective function), and the cost-related ones. From the shown curves it is possible to notice that, for both C-PGAE and C-PGPE , a higher regularization ( ω ' 10 ´ 2 ) corresponds to a higher bias w.r.t. the constraint satisfaction. This bias is compliant with what shown by Theorem 3.1, indeed, the higher the regularization, the higher the constraint threshold should be made stricter. Finally, we report in Figure 8 the evolution of the values of the Lagrangian multipliers λ during the learning. As expected from the theory, for both C-PGAE and C-PGPE a higher regularization leads to have smaller values of λ . Moreover, we empirically notice that C-PGAE reaches higher value of λ w.r.t. the ones seen by C-PGPE .

˘

C-PGPE : Lagrangian Curve

0

2

,

000

4

,

000

6

,

000

8

,

000

10

,

000

5

10

15

20

Iterations

Lagrangian Value

ω

'

0

ω

'

0.0001

ω

'

0.01

0

2

,

000

4

,

000

6

,

000

8

,

000

10

,

000

´

20

´

15

´

10

´

5

Iterations

Average Trajectory Return

C-PGPE : Performance Curve

ω

'

0

ω

'

0.0001

ω

'

0.01

0

2

,

000

4

,

000

6

,

000

8

,

000

10

,

000

0

.

1

0

.

2

0

.

3

Iterations

Average Trajectory Cost

C-PGPE

: Cost Curve

ω

'

0

ω

'

0.0001

ω

'

0.01

b

C-PGAE : Lagrangian Curve

0

2

,

000

4

,

000

6

,

000

8

,

000

10

,

000

5

10

15

Iterations

Lagrangian Value

ω

'

0

ω

'

0.0001

ω

'

0.01

0

2

,

000

4

,

000

6

,

000

8

,

000

10

,

000

´

15

´

10

´

5

Iterations

Average Trajectory Return

C-PGAE : Performance Curve

ω

'

0

ω

'

0.0001

ω

'

0.01

0

2

,

000

4

,

000

6

,

000

8

,

000

10

,

000

0

.

1

0

.

2

0

.

3

Iterations

Average Trajectory Cost

C-PGAE

: Cost Curve

ω

'

0

ω

'

0.0001

ω

'

0.01

b

C-PGPE : λ Curve

0

2

,

000

4

,

000

6

,

000

8

,

000

10

,

000

0

5

10

Iterations

λ

Value

ω

0

'

0

ω

0

'

0

.

0001

ω

0

'

0

.

01

(a)

C-PGPE

0

2

,

000

4

,

000

6

,

000

8

,

000

10

,

000

0

5

10

Iterations

λ

Value

C-PGAE

:

λ

Curve

ω

0

'

0

ω

0

'

0

.

0001

ω

0

'

0

.

01

(b)

C-PGAE

H.7 Computational Resources

All the experiments were run on a 2019 16-inches MacBook Pro. The machine was equipped as follows:

In particular, N ' 100 trajectories of the MuJoCo environments with T ' 100 scored « 2 iterations per second for C-PGAE , while « 3 iterations per second for C-PGPE . N ' 100 trajectories of the CostLQR environment with T ' 100 scored « 5 and « 8 iterations per second respectively for C-PGAE and C-PGPE . All the performances are to be considered with a parallelization over 10 CPU cores.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and the introduction reflect the original contribution of the paper. Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of the work emerges in the final section of the work and in the related works section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the statements are correlated with proofs. The definitions and the quantity used are clearly presented.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All the experimental settings are described, together with choices of all the parameters for the settings at hand.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The implemented code is provided in a zip file which contains detailed instructions on how to use it.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the experimental settings are described, together with choices of all the parameters for the settings at hand.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The plots of all the experiments are made over an average of different trials and we report, for each shown curve, the 95% confidence interval.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The computational resources employed for th experiments are discussed.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We checked the guidelines and we are compliant with them.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: As a theoretical research paper, the authors do not see any negative societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The only existing asset employed is the MuJoCo (Todorov et al., 2012) control suite, that is properly mentioned.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: In order to run the experiments, we implemented a detailed code-base. This asset is provided in an anonymized zip file and, conditionally on the paper acceptance, it will be made public. The zip file includes a file explaining how to employ the code-base.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper targets the constrained RL problem and provides a primal-dual method C-PG to solve it. The proposed method is extended to C-PGAE and C-PGPE to handle the constraint cases with risk measures.  The paper provides a theoretical analysis of the global last-iterate convergence guarantees toward C-PG and empirically tests the C-PGAE and C-PGPE.\n\nSTRENGTHS:\nThe paper is well-written. All the assumptions are standard in literatures or justified. The theoretical result is rigorous.\n\nWEAKNESSES:\nThe main weakness of the paper comes from the technical novelty. From the algorithm design, the primal-dual method is widely used in constrained optimization problems and constrained RL problems, and the regularization term is also not new in constrained optimization problems [1]. The proposed algorithm basically follows the existing methods without a significant novel design. The algorithm can be naturally adapted to C-PGAE and C-PGPE to handle the constraint cases with risk measures due to the well-established policy gradient for the case. From the theoretical analysis, assumptions 1,2, 3, and 4 are standard assumptions for constrained optimization problems. With these assumptions, the constrained RL problem is transferred to a pure constrained optimization problem. Although the assumptions are justified in RL, I believe there are lots of theoretical results for constrained optimization problems with these assumptions, and can be directly used in constrained RL. Therefore, the theoretical novelty may also be marginal.\n\nAlthough the paper claims that it addresses some theoretical limitations of previous works, these limitations are somehow avoided, rather than being addressed. For example, the paper does not require the softmax policy, but assumption 2 is verified under the softmax policy. The provided convergence rates do not depend on the problem dimension, but the problem dimension may be included in the constants of assumptions 2, 3, and 4.\n\nThe experiment scenarios are relatively simple, but it is acceptable if the theoretical result is sufficiently solid.\n\n\n\n\n\n[1] Khuzani, Masoud Badiei, and Na Li. \"Distributed regularized primal-dual method: Convergence analysis and trade-offs.\"\n\nQUESTIONS:\nAs mentioned in Weaknesses, as the dimension-free property of the proposed algorithm is claimed many times in the paper, have you proven that the problem dimension is independent of the constants of assumptions 2, 3, and 4? What is the reason and the theoretical intuition that it is dimension-free?\n\nHow does the convergence rate change after introducing the regularization term for the last-iteration convergence? As it introduces an extra error term of $w$ in Theorem 3.1, it may deteriorate the convergence rate.\n\nI am willing to increase my score if weaknesses and questions are addressed.\n\nLIMITATIONS:\nNo limitations of the paper are explicitly stated by the author.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies policy-based methods in constrained RL. The author first establishes the last-iterate convergence of the algorithm C-PG under a form a gradient domination assumptions. Then, the author further designs action-based and parameter-based versions of C-PG to handle constraints defined in terms of risk measures over the costs. Finally, the proposed algorithms are validated by numerical examples of control problems.\n\nSTRENGTHS:\n* Despite not being the first paper to establish the last-iterate convergence of primal-dual type algorithms, I believe the convergence results for C-PG is also a meaningful contribution to the literature. \n* More general constrained RL formulations are studied in the paper, where constrains are defined in terms of risk measures. The author adapts C-PG to another two sample-based variants for these formulations.\n* Overall, the paper is well-written and easy to follow. The author uses highlights for the two different algorithms/settings, making them easy to distinguish.\n\nWEAKNESSES:\n* The role of Section 4 in the paper is unclear to me. Good theoretical results are established for Section 3, yet, there is no general theoretical results for Section 4, besides some cases where results from Section 3 can be directly applied. This makes Section 4 looks like an ''add-on'' to Section 3. \n* For theoretical results, e.g., Theorems 3.1 and 3.2, it would be better for the author to briefly discuss the proof idea in the main paper.\n\nMinor comments: it would be better to make figure legends exactly align with the names of algorithms, e.g., change CPGAE to C-PGAE.\n\nQUESTIONS:\n* I hope the author could further clarify the contribution/role of Section 4 in the paper.\n* Could the author also discusses how the techniques used in the paper different from that in Ding et al., 2024?\n\nLIMITATIONS:\nThe paper has stated assumptions clearly in the paper. Yet, although the author mentions \"The limitations of the work emerges in the final section of the work\", I failed to find that section in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-01"}

comments: SUMMARY:
This paper proposes a general framework for addressing safe RL problems via gradient-based primal-dual algorithms. The authors show that the proposed algorithm exhibit global last-iterate convergence guarantees under gradient domination assumptions. Additionally, the authors validate their algorithms on several constrained control problems.

STRENGTHS:
1. This paper is well-written and easy to follow.
2. The paper is technically sound with most claims supported sufficiently.
3. The theoretical analysis seems novel.

WEAKNESSES:
Quality:

1. In Theorem 3.1, it requires w=O(\epsilon) to enforce an overall \epsilon error, but w should be a fixed number in C-PG algorithm. 
2. It is better to have some experimental results about cost constraints on MuJoCo and include more baselines, e.g., [Zhang et al., 2020, First Order Constrained Optimization in Policy Space].

Clarity:

1. The range of reward value is [-1, 0], which is a bit weird compared to normal settings.
2. The advantage of parameter-based hyperpolicy is not clearly mentioned in the paper.

Significance:

One relevant literature is missing. In [Liu et al. 2021, Policy Optimization for Constrained MDPs with Provable Fast Global Convergence], a fast convergence \tilde{O}(1/\epsilon) result is proved without multiple assumptions in this paper.

QUESTIONS:
Please see the details in "weakness".

LIMITATIONS:
There is no potential negative social impact of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-07

comments: SUMMARY:
This paper studies the problem of constrained MDP. To solve this problem, this paper adopts the policy gradient methods, and specifically they considered the action-based policy gradient method and parameter-based policy gradient method. 

The algorithm proposed in this paper is a type of primal-dual method. Under certain assumptions, this algorithm is shown to have last iterate convergence.

This paper also executes numerical experiments on various environment with their algorithm, and the experiments validate the results in this paper.

STRENGTHS:
This paper is a well-written paper. The description of problem setup, theorems, assumptions are clear. 

Even though this is mainly a theory paper, there are numerical experiments which validates the theoretical results.

This paper has results on the last iterate convergence, which is a property not possessed by most of stochastic optimization algorithms.

WEAKNESSES:
I don't see significant weaknesses in this paper.

QUESTIONS:
Do you have lower bounds showing that these rates are tight?

LIMITATIONS:
Yes. The authors addressed all the limitations listed in the guidelines.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10