Title: Sample-Efficient Constrained Reinforcement Learning with General Parameterization
Venue: NeurIPS 2024
Accepted: yes
Keywords: Constrained MDP

Sample Complexity

Constraint Violation

Global Optimality.
Abstract: We consider a constrained Markov Decision Problem (CMDP) where the goal of an agent is to maximize the expected discounted sum of rewards over an infinite horizon while ensuring that the expected discounted sum of costs exceeds a certain threshold. Building on the idea of momentum-based acceleration, we develop the Primal-Dual Accelerated Natural Policy Gradient (PD-ANPG) algorithm that ensures an $\epsilon$ global optimality gap and $\epsilon$ constraint violation with $\tilde{\mathcal{O}}((1-\gamma)^{-7}\epsilon^{-2})$ sample complexity for general parameterized policies where $\gamma$ denotes the discount factor. This improves the state-of-the-art sample complexity in general parameterized CMDPs by a factor of $\mathcal{O}((1-\gamma)^{-1}\epsilon^{-2})$ and achieves the theoretical lower bound in $\epsilon^{-1}$.

Sample-Efficient Constrained Reinforcement Learning with General Parameterization

Abstract

We consider a constrained Markov Decision Problem (CMDP) where the goal of an agent is to maximize the expected discounted sum of rewards over an infinite horizon while ensuring that the expected discounted sum of costs exceeds a certain threshold. Building on the idea of momentum-based acceleration, we develop the Primal-Dual Accelerated Natural Policy Gradient (PD-ANPG) algorithm that ensures an ϵ global optimality gap and ϵ constraint violation with ˜ O ((1 -γ ) -7 ϵ -2 ) sample complexity for general parameterized policies where γ denotes the discount factor. This improves the state-of-the-art sample complexity in general parameterized CMDPs by a factor of O ((1 -γ ) -1 ϵ -2 ) and achieves the theoretical lower bound in ϵ -1 .

1 Introduction

Reinforcement learning (RL) is a framework where an agent repeatedly interacts with an unknown Markovian environment to find a policy that maximizes the expected discounted sum of its observed rewards. Such problems, often modeled via Markov Decision Processes (MDPs), find applications in many areas, including transportation [1], communication networks [2], robotics [3], etc. In many applications, however, the agents must also obey certain constraints. For example, in a food delivery network, the orders must be delivered within a stipulated time window; the marketing decisions of a firm must satisfy its budget constraints, etc. Such constraints are incorporated into RL by introducing a cost function. In constrained MDPs (CMDPs), the agents not only maximize the expected sum of discounted rewards but also ensure that the expected sum of discounted costs does not cross a predefined boundary.

Finding an optimal policy for a CMDP is a challenging problem, especially when the environment, i.e., the state transition function, is unknown. The efficiency of a solution to a CMDP is measured by its sample complexity, which essentially states how many state-transition samples it takes to yield a policy that is ϵ close to the optimal one while also ensuring that the expected sum of discounted costs does not violate the imposed boundary by more than ϵ amount. Many articles in the literature solve the CMDPwith an unknown environment. Most of these works, however, focus on the tabular case where the number of states is finite. These solutions cannot be applied to many real-life scenarios where the state space is either large or infinite. To tackle this issue, the concept of policy parameterization must be invoked. Unfortunately, as exhibited in Table 1, only a few works are available on CMDPs with parameterized policies. While, for softmax parameterization, the state-of-the-art (SOTA) sample complexity is O ( ϵ -2 ) , the same for the general parameterization is ˜ O ( ϵ -4 ) which is far from the lower bound Ω( ϵ -2 ) . It should be noted that the number of parameters needed in softmax parameterization is O ( SA ) where S, A are the sizes of the state and action spaces of the underlying CMDP. On the other hand, general parameterization uses d ≪ SA number of parameters, which makes it appropriate

Vaneet Aggarwal

for dealing with large or infinite states. Given the importance of general parameterization for large state space CMDPs, the following question naturally arises: "Is it possible to solve CMDPs with general parameterization and achieve a sample complexity better than the SOTA ˜ O ( ϵ -4 ) bound?"

In this article, we provide an affirmative answer to the above question. We propose a Primal-Dualbased Accelerated Natural Policy Gradient (PD-ANPG) algorithm to solve γ -discounted CMDPs with general parameterization. We theoretically prove that PD-ANPG achieves ϵ optimality gap and ϵ constraint violation with ˜ O ((1 -γ ) -7 ϵ -2 ) sample complexity (Theorem 1) that improves the SOTA ˜ O ((1 -γ ) -8 ϵ -4 ) sample complexity result of [4]. It closes the gap between the theoretical upper and lower bounds of sample complexity in general parameterized CMDPs (in terms of ϵ -1 ), which was an open problem for quite some time (see the results of [5], [6], [4] in Table 1).

1.1 Challenges and Key Insights

Our algorithm builds upon the idea of primal dual-based NPG [7, 4]. However, unlike the previous works, we use accelerated stochastic gradient descent (ASGD) in the inner loop to compute the estimate of the NPG. The improvement in the sample complexity results from two key observations. Firstly, we establish a global-to-local convergence lemma (Lemma 3), which dictates how the global convergence of the Lagrange function is related to the first and second-order estimation error of the NPG. Here, via careful analysis, we show that the first-order term can be written as the expected bias of the NPG estimator (i.e., the difference between the true NPG and the expectation of its estimate). Secondly, we show (Lemma 5 and its subsequent discussion) that the bias of the NPG estimate can be interpreted as the convergence error of an ASGD program with non-stochastic (i.e., deterministic) gradients. These, combined with the ASGD convergence result provided by [8], lead to a convergence result of the Lagrange function (Corollary 1).

Finally, Theorem 1 segregates the objective and constraint violation rates from Lagrange convergence. Corollary 1 shows that Lagrange convergence error is bounded by an independent function of ζ (the dual learning rate). One might, hence, be tempted to make ζ arbitrarily small. However, our analysis shows that although small ζ leads to better objective convergence, it worsens the constraint violation rate. We demonstrate how to optimally choose ζ to reach the middle ground which eventually leads us to ˜ O ( ϵ -2 ) sample complexity.

1.2 Related Works

Unconstrained RL: Many algorithms solve MDPs with exact gradients [12, 13, 14, 15, 16]. Moreover, many works use generative models to show either first-order [17, 18, 19, 20, 21] or global convergence [22, 23, 24, 25, 7, 26, 27].

Constrained RL: The tabular setting is well investigated, and many model-based [28, 29, 30, 31] and model-free [6, 30, 32, 33] algorithms are available in the literature. In comparison, there are relatively fewer works on parameterized policies. Policy mirror descent-primal dual (PMD-PD) algorithm was proposed by [9] that achieves O ( ϵ -3 ) sample complexity for softmax policies. For the same parameterization, [10] achieved O ( ϵ -6 ) sample complexity via their proposed Online Primal-Dual Natural Actor-Critic Algorithm. The primal-dual Natural Policy Gradient algorithm suggested by [5] yields O ( ϵ -2 ) and O ( ϵ -6 ) sample complexities for softmax and general parametrization respectively. [6] also proposed a primal policy-based algorithm that works for both the softmax and general function approximation cases. The state of the art for the general parameterization is given by [4], where the ˜ O ( ϵ -4 ) sample complexity is obtained. This work improves upon this direction to obtain ˜ O ( ϵ -2 ) sample complexity. The comparisons are summarized in Table 1.

2 Formulation

Let us consider a constrained Markov Decision Process (CMDP) characterized by the tuple M = ( S , A , r, c, P, γ, ρ ) where S , A denote the state space and the action space respectively, r : S × A → [0 , 1] defines the reward function, c : S × A → [ -1 , 1] is the cost function, P : S × A → ∆( S ) indicates the state transition kernel (where ∆( S ) is the collection of all probability distributions over S ), γ is the discount factor, and ρ ∈ ∆( S ) is the initial state distribution. Note that the state space S , in our setting, can potentially be a compact set of infinite size. However, for simplicity, we assume it to be countable. The action space, A , is assumed to be of finite size. The range of the cost function is chosen to be [ -1 , 1] , rather than [0 , 1] , to ensure that the constraint in our central optimization problem (defined later in (2)) is non-trivial. A policy, π : S → ∆( A ) is defined as a distribution over the action space for a given state of the environment. For a given policy, π , and a state-action pair ( s, a ) , we define the Q -value associated with g ∈ { r, c } as follows.

where E π is the expectation computed over all π -induced trajectories { ( s t , a t ) } ∞ t =0 where s t +1 ∼ P ( s t , a t ) and a t ∼ π ( s t ) , ∀ t ∈ { 0 , 1 , · · · } . Similarly, the V -value associated with policy π , state s , and g ∈ { r, c } is defined below.

Below we define the advantage value for a policy π , a state-action pair ( s, a ) , and g ∈ { r, c } .

Define a function J π g,ρ , ∀ g ∈ { r, c } as follows.

where d π ρ ∈ ∆( S ) is the state occupancy measure given by,

Similarly, the state-action occupancy measure is defined as,

Our goal is to maximize the function J π r,ρ over all policies π while ensuring that J π c,ρ does not lie below a predefined threshold. Without loss of generality, we can formally express this problem as,

If the state space, S , is large or infinite (which is the case in many application scenarios), the policies can no longer be represented in the tabular format; rather, they are indexed by a parameter, θ ∈ Θ . In

this paper, we assume Θ = R d . Such indexing can be done via, for example, neural networks (NNs). Let J g,ρ ( θ ) ≜ J π θ g,ρ . This allows us to redefine the constrained optimization problem as follows.

We assume the existence of at least one interior point solution of the above optimization. This is also known as Slater condition which can be formally expressed as follows.

Assumption 1. There exists ¯ θ such that J c,ρ ( ¯ θ ) ≥ c slater for some c slater ∈ (0 , 1 1 -γ ] .

3 Algorithm

The dual problem associated with the constraint optimization (3) can be written as follows.

The function, J L ,ρ ( · , · ) is called the Lagrange function while λ is said to be the Lagrange multiplier. The above problem can be solved by iteratively applying the following update rule ∀ k ∈ { 0 , · · · , K -1 } , starting with ( θ 0 , λ 0 ) where θ 0 ∈ Θ is arbitrary and λ 0 = 0 .

where η, ζ are learning rates, P Λ denotes the projection function onto the set, Λ ≜ [0 , λ max ] , and † is the Moore-Penrose pseudoinverse operator. The choice of λ max will be specified later. Note that the update rule of θ is similar to that of the standard policy gradient method except here the learning rate, η is modulated by the inverse of the Fisher matrix, F ρ ( θ ) which is defined below.

where ⊗ indicates the outer product. Using a variation of the classical policy gradient theorem [34], one can obtain the gradient of the Lagrange function as follows.

In most application scenarios, the learner is unaware of the state transition function, P , and thereby, of the advantage function, A π θ L ,λ and the occupancy measure, ν π θ ρ . This makes the exact computation of F ρ ( θ ) and H ρ ( θ, λ ) an impossible task. Fortunately, there is a way to obtain an approximate value of the natural policy gradient ω ∗ θ,λ ≜ F ρ ( θ ) † ∇ θ J L ,ρ ( θ, λ ) that does not require the knowledge of P . Invoking (8), one can prove that ω ∗ θ,λ is a solution of a quadratic optimization. Formally, we have,

The above reformulation opens up the possibility to compute ω ∗ θ,λ via a gradient descent-type iterative procedure. Observe that the gradient of L ν π θ ρ ( · , θ, λ ) can be calculated as follows.

Algorithm 1 describes a procedure to obtain unbiased estimates of this gradient. This is inspired by Algorithm 3 of [12]. Additionally, observe from (6) that the update of the Lagrange variable, λ requires the computation of J c,ρ ( θ ) which is also difficult to accomplish without having an explicit knowledge about P . Algorithm 1 also provides an unbiased estimation of the above quantity.

Algorithm 1 first samples a horizon length, T , from the geometric distribution with success probability (1 -γ ) and executes the CMDP for T instances following the policy, π θ , starting from a state s 0 ∼ ρ .

Algorithm 1 Unbiased Sampling

1: Input: Parameters ( θ, ω, λ, γ ) , Initial Distribution ρ

2:

3:

4:

Geo(1

∼

T

for

j

0

,

-

· · ·

∈ {

Execute

j

a

ρ

,

a

do

0

,

γ

)

s

∼

, T

1

-

}

, observe

s

j

+1

∼

∼

π

θ

P

(

(

s

s

0

j

)

, a

j

)

and sample

6:

7:

8:

9:

Geo(1

∼

T

for for

j

0

,

-

· · ·

∈ {

Execute

j

, a

,

γ

)

(

s

0

, T

1

-

0

}

, observe

)

do

s

j

+1

a

(ˆ

s,

∼

ˆ

a

P

g

do r, c

∈ {

}

11:

,

Geo(1

γ

)

s

-

12:

13:

14:

∼

∼

∼

T

for for

j

0

,

· · ·

∈ {

Execute

a

j

0

, T

1

}

-

, observe

,

a

s

ˆ

do

s

+1

j

g

do r, c

∈ {

}

16:

(ˆ

s,

ˆ

a

)

Q

ˆ

θ

17:

18:

ˆ

A

π

g

π

g

θ

(ˆ

s,

ˆ

a

)

-

ˆ

V

0

π

P

π

g

)

(

s

θ

(

j

s

(

θ

j

s

, a

j

0

)

, a

(ˆ

)

j

s

0

)

)

, and sample

, and sample

a

j

a

+1

j

a

+1

j

+1

∼

∼

∼

▷

+1

)

▷

j

s

(

π

θ

π

θ

π

Q-function Estimation

+1

)

▷

V-function Estimation

+1

)

(

θ

j

s

(

s

j

Estimation of Relevant functions

▷ Gradient Estimate

The total cost observed in the resulting trajectory is assigned as the estimate ˆ J c,ρ ( θ ) . The state-action pair ( s T , a T ) can be assumed to be an arbitrary sample (ˆ s, ˆ a ) chosen from the occupancy measure ν π θ ρ . The algorithm then generates a π θ -induced trajectory of length T ∼ Geo(1 -γ ) , taking (ˆ s, ˆ a ) as the starting point. The total reward and cost observed in this trajectory are assigned as ˆ Q π θ r (ˆ s, ˆ a ) and ˆ Q π θ c (ˆ s, ˆ a ) respectively. Next, another π θ -induced trajectory of length T ∼ Geo(1 -γ ) is generated assuming the state, ˆ s as the initiation point. The total reward and cost of this trajectory are assigned as ˆ V π θ r (ˆ s ) and ˆ V π θ c (ˆ s ) respectively. For g ∈ { r, c } , an estimate of the advantage value is computed as ˆ A π θ g (ˆ s, ˆ a ) = ˆ Q π θ g (ˆ s, ˆ a ) -ˆ V π θ g (ˆ s ) . Finally, the estimates of F ρ ( θ ) and H ρ ( θ, λ ) are obtained via (12) and (13) respectively which produces an estimation of the desired gradient in (14). The following Lemma demonstrates that the estimates produced by Algorithm 1 are unbiased.

Lemma 1. Let ˆ J c,ρ ( θ ) , ˆ ∇ ω L ν π θ ρ ( ω, θ, λ ) be the estimates produced by Algorithm 1 for a predefined set of parameters ( ω, θ, λ ) . The following equations hold.

In the absence of knowledge about the transition model, P , one can utilize the estimates generated by Algorithm 1 as good proxies for their true values. In particular, one can obtain an approximate value of the natural policy gradient ω ∗ θ,λ by iteratively minimizing the function L ν π θ ρ ( · , θ, λ ) using the gradient estimate ˆ ∇ ω L ν π θ ρ ( ω, θ, λ ) . On the other hand, using the estimate ˆ J c,ρ ( θ ) , an approximate update equation of the Lagrange parameter can be formed. Algorithm 2 uses these two ideas to obtain a policy that is close to the optimal one.

Algorithm 2 Primal-Dual Accelerated Natural Policy Gradient (PD-ANPG)

1: Input: Parameters ( θ 0 , λ 0 ) , Distribution ρ , Run-time Parameters K,H , Learning Parameters η, ζ, α, β, ξ, δ

2: for k ∈ { 0 , · · · , K -1 } do

3: x 0 , v 0 ← 0

4: for h ∈ { 0 , · · · , H -1 } do

5:

▷ Inner Loop ▷ Accelerated Stochastic Gradient Descent

7: Tail Averaging:

8: Obtain ˆ J c,ρ ( θ k ) via Algorithm 1.

9: Parameter Updates:

10: Output: { θ k } K -1 k =0

Algorithm 2 has a nested loop structure. The outer loop runs K number of times. At a given instance, k , of the outer loop, the policy parameter θ k and the Lagrange parameter, λ k are updated via (20) and (21). The estimate, ˆ J c,ρ ( θ k ) is computed via Algorithm 1. On the other hand, ω k , the approximate value of the natural policy gradient ω ∗ θ k ,λ k is obtained by iteratively minimizing L ν π θ k ρ ( · , θ k , λ k ) in H number of inner loop steps via the Accelerated Stochastic Gradient Descent (ASGD) procedure as stated in [8]. ASGD comprises the iterative updates (15) -(18) with tunable learning parameters ( α, β, ξ, δ ) followed by a tail-averaging step (19). The gradient estimate utilized in (16) and (18) is obtained via Algorithm 1. It is worth mentioning that existing NPG algorithms such as that given in [7] typically apply the SGD, rather than the ASGD procedure, to obtain ω k . The difference between these subroutines is that while SGD uses only the current gradient estimate to update ω k , ASGD considers the contribution of all previous gradient estimates (momentum) using its convoluted iteration and tail-averaging steps.

4 Analysis

Our goal in this section is to characterize the rate of convergence of the objective function and the constraint violation if policy parameters are generated via Algorithm 2. We start by stating a few assumptions needed for the analysis.

Assumption 2. The log-likelihood function is G -Lipschitz and B -smooth where B,G > 0 . Formally, the following relations hold ∀ θ, θ 1 , θ 2 ∈ Θ , and ∀ ( s, a ) ∈ S × A .

Remark 1. Assumption 2 is commonly applied in proving convergence guarantees of policy gradienttype algorithms [35, 12, 7]. This assumption is obeyed by many widely used policy classes such as the class of neural networks with bounded weights.

Assumption 2 implies the boundedness of the gradient of the Lagrange function. This can be formally expressed as follows.

Lemma 2. If Assumption 2 holds, then the following inequality is true ∀ θ ∈ Θ and ∀ λ ∈ Λ .

Proof. Statement (a) can be proven using (8) along with Assumption 2 and the facts that | A π θ L ,λ ( s, a ) | is bounded by (1 + λ ) / (1 -γ ) and λ ≤ λ max , ∀ λ ∈ Λ .

The result established by Lemma 2 will be pivotal in our further analysis.

Assumption 3. The compatible function approximation error defined in (9) satisfies the inequality L ν π ∗ ρ ( ω ∗ θ,λ , θ, λ ) ≤ ϵ bias / 2 , ∀ θ ∈ Θ and ∀ λ ∈ Λ where π ∗ is a solution to the original constrained optimization problem (2) and ω ∗ θ,λ is defined in (9) . The term ϵ bias is a non-negative constant. The factor 2 is used for notational convenience.

Remark 2. The term ϵ bias quantifies the expressivity of the parameterized policy class. For example, if the parameterization is complete i.e., includes all possible policies (such as in direct or softmax parameterization), then ϵ bias = 0 [12]. A similar result can be proven for linear MDPs [36]. For incomplete policy classes, we have ϵ bias > 0 . However, if the class is sufficiently rich (such as neural networks with a large number of parameters), ϵ bias can be assumed to be negligibly small [37].

Assumption 4. There exists a positive constant µ F such that F ρ ( θ ) -µ F I d is positive semidefinite i.e., F ρ ( θ ) ⪰ µ F I d , ∀ θ ∈ Θ where I d is a d × d identity matrix and F ρ ( · ) is defined in (7) .

The property of the policy classes laid out in Assumption 4 is called Fisher Non-Degeneracy (FND) which essentially ensures that ∀ θ ∈ Θ , the Fisher matrix F ρ ( θ ) is away from the zero matrix by a certain amount. Observe that the Hessian of the function, l θ,λ ( · ) ≜ L ν π θ ρ ( · , θ, λ ) is F ρ ( θ ) . Therefore, Assumption 4 also indicates that l θ,λ is µ F -strongly convex. This assumption is commonly applied in analyzing policy-gradient algorithms [4, 38, 7]. Assumption 4 also ensures that the matrix F ρ ( θ ) is invertible, which, in turn, implies the uniqueness of the maximizer ω ∗ θ,λ = arg min ω ∈ R d l θ,λ ( ω ) . [39] describes a concrete set of policies that obeys Assumption 2 -4 .

4.1 Local-to-Global Convergence Lemma

Recall that our goal is to establish the global convergence rates. Lemma 3 (stated below) is the first step in that direction. Specifically, it demonstrates how the average optimality gap of the Lagrange function can be bounded by the first and second-order error of the gradient estimates.

Lemma 3. If the parameters { θ k , λ k } K -1 k =0 are updated via (20) and (21) and assumptions 2 -4 hold, then the following inequality holds for any K .

where ω ∗ k ≜ ω ∗ θ k ,λ k , ω ∗ θ k ,λ k is the natural policy gradient defined in (9) , and π ∗ is the solution to the constrained optimization (2) . Finally, ω k is the approximation of ω ∗ k given by (19) and KL ( ·∥· ) is the KL-divergence.

Note the presence of the term, ϵ bias in (22). It shows that due to the incompleteness of the parameterized policy class, the average optimality error cannot be made arbitrarily small. It is worth mentioning that many existing CMDP analyses (such as [4]) follow a path similar to that of Lemma 3. However, while the first order term in those works turns out to be E ∥ ω k -ω ∗ k ∥ , we improved it to E ∥ E [ ω k | θ k , λ k ] -ω ∗ k ∥ . Such seemingly insignificant improvement has important ramifications for

our analysis as explained later in the paper. The second order term in (22) can be expanded as,

where (a) utilises ω ∗ k = F ρ ( θ k ) † ∇ θ J L ,ρ ( θ k , λ k ) and Assumption 4. The second inequality applies Lemma 2 together with the fact that λ k ∈ Λ . Our next subsection provides a bound on E ∥ ω k -ω ∗ k ∥ 2 and the first order term E ∥ E [ ω k | θ k , λ k ] -ω ∗ k ∥ .

4.2 Local Convergence of the Natural Policy Gradient

To deliver the promised bounds, we first provide some characterization of the gradient estimate.

Lemma 4. Let ˆ ∇ ω L ν π θ ρ ( ω, θ, λ ) be the estimate produced by Algorithm 1. Under assumptions 2 and 4, the following semidefinite inequality holds for any θ ∈ Θ and λ ∈ Λ .

where ω ∗ θ,λ , F ρ ( θ ) are given by (9) and (7) respectively and σ 2 is defined below.

The term σ 2 defined in Lemma 4 can be described as the scaled variance of the gradient estimate, ˆ ∇ ω L ν π θ ρ ( ω, θ, λ ) . Note that if the estimates were non-stochastic (i.e., deterministic), we would have σ 2 = 0 since ∇ ω L ν π θ ρ ( ω ∗ θ,λ , θ, λ ) = 0 . The last equation can be proved using the definition of ω ∗ θ,λ given in (9), the gradient expression provided in (10), and observing that the Fisher matrix, F ρ ( θ ) is invertible due to Assumption 4. The above information is crucial in bounding the first-order error, as stated in the following lemma.

Lemma5. If assumptions 2 and 4 hold, then the following relations are satisfied ∀ k ∈ { 0 , · · · , K -1 } with learning rates α = 3 √ 5 G 2 µ F +3 √ 5 G 2 , β = µ F 9 G 2 , ξ = 1 3 √ 5 G 2 , and δ = 1 5 G 2 provided that the inner loop length of Algorithm 2 obeys H > ¯ C G 2 µ F log ( √ d G 2 µ F ) for some universal constant, ¯ C .

where C denotes a universal constant, ω k is given by (19) , and σ 2 is defined in (24) .

The first bound (25) is a consequence of Lemma 4 and the ASGD convergence result provided in [8] (Corollary 2). To gain intuition about the second result (26), note that by taking the conditional expectation E [ ·| θ k , λ k ] on both sides of the ASGD iterations (15) -(18), and applying the unbiasedness of the gradient estimate (Lemma 1) we obtain the following ∀ h ∈ { 0 , · · · , H -1 } .

where ¯ l h = E [ l h | θ k , λ k ] , l ∈ { v , x , y , z } . Moreover, taking conditional expectation on both sides of the tail averaging process (19), we arrive at the following.

Note that the steps (27) -(28) resemble the iterative updates of a deterministic ASGD. This allows us to obtain E ∥ ¯ ω k -ω ∗ k ∥ by substituting σ 2 = 0 in (25) and applying the Cauchy-Schwarz inequality.

4.3 Global Convergence of the Lagrange

Combining Lemma 3, (23) and using the expected gradient errors provided by Lemma 5, we bound the average Lagrange optimality gap as a function of tunable parameters H and K as stated in the following corollary.

Corollary 1. Consider the same setup and the choice of parameters described in Lemma 3 -5. The following inequality holds if assumptions 2 -4 are met.

Corollary 1 bounds the optimality error of the Lagrange function as O ( √ ϵ bias +exp( -C 0 H ) + η + 1 ηK ) where C 0 is some problem specific constant. Interestingly, the dual learning parameter, ζ does not appear in this bound. However, the next section shows that ζ plays a pivotal role in deciding the objective and constraint violation rates.

4.4 Decoupling the Objective and the Constraint Violation Rates

The goal of the following theorem is to choose optimal values of the tunable parameters and decouple the objective and constraint violation rates from the Lagrange convergence result given in (29).

Theorem 1. Consider the same setup and the choice of parameters given in Lemma 3 -5. Assume η = (1 -γ ) 2 (1 + λ max ) -1 / √ K , ζ = λ max (1 -γ ) / √ K , and λ max = 2 / [(1 -γ ) c slater ] . For sufficiently small ϵ > 0 , the following inequalities hold

whenever assumptions 1 -4 are met, H = O (log( ϵ -1 )) and K = O ((1 -γ ) -6 ϵ -2 ) . Therefore, the sample complexity to ensure (30) is O ((1 -γ ) -1 HK ) = ˜ O ((1 -γ ) -7 ϵ -2 ) . It is to be clarified that the (1 -γ ) -1 factor in the sample complexity calculation appears due to the fact that it requires O ((1 -γ ) -1 ) samples on an average to obtain a gradient estimate via Algorithm 1.

Theorem 1 dictates that, with appropriate choice of the parameters, the rate of convergence of the objective and that of constraint violation can be bounded as O ( √ ϵ bias + ϵ ) with ˜ O ((1 -γ ) -7 ϵ -2 ) samples. This beats the SOTA ˜ O ( ϵ -4 ) sample complexity of [4] and achieves the theoretical lower bound. Our derived sample complexity is also dependent on c slater . However, this is not explicitly mentioned in Theorem 1. Interested readers can find such details in the appendix.

Remark 3. Note the importance of the nested expectation in the first-order term E ∥ E [ ω k | θ k , λ k ] -ω ∗ k ∥ in Lemma 3. Lemma 5 bounds this term as O (exp( -C 0 H )) where C 0 is a problem dependent constant. Other terms in the Lagrange optimality bound (Lemma 3) can be expressed as O ( η + 1 ηK ) . Moreover, following the proof of Theorem 1, one sees that decoupling the objective optimality error and constraint violation bounds incurs additional O ( ζ + 1 ζK ) terms. Choosing η, ζ as prescribed in Theorem 1, makes both the objective and constraint violation errors as O ( √ ϵ bias +exp( -C 0 H ) + K -0 . 5 ) . This allows us to take H = ˜ O (1) and K = O ( ϵ -2 ) , leading to ˜ O ( ϵ -2 ) sample complexity. Had the first-order term been E ∥ ω k -ω ∗ k ∥ , it would have resulted in O ( √ ϵ bias + H -0 . 5 + K -0 . 5 ) objective and constraint violation errors which would have lead to O ( ϵ -4 ) sample complexity.

5 Conclusions and Limitations

This paper considers the problem of learning a CMDP where the goal is to maximize the objective value function while guaranteeing that the cost value exceeds a predefined threshold. We propose an acceleration-based primal-dual natural policy gradient algorithm that ensures ϵ optimality gap and ϵ constraint violation with ˜ O ( ϵ -2 ) sample complexity. This improves upon the previous stateof-the-art sample complexity of O ( ϵ -4 ) and achieves the theoretical lower bound. Future works include applying the idea of acceleration-based NPG to improve sample complexities in other related domains of constrained reinforcement learning, e.g., non-linear CMDP, average reward CMDP, etc.

References

Al-Abbasi, A. O., A. Ghosh, V. Aggarwal. Deeppool: Distributed model-free algorithm for ridesharing using deep reinforcement learning. IEEE Transactions on Intelligent Transportation Systems , 20(12):4714-4727, 2019.

Geng, N., T. Lan, et al. A multi-agent reinforcement learning perspective on distributed traffic engineering. In 2020 IEEE 28th International Conference on Network Protocols (ICNP) , pages 1-11. IEEE, 2020.

Gonzalez, G., M. Balakuntala, M. Agarwal, et al. Asap: A semi-autonomous precise system for telesurgery during communication delays. IEEE Transactions on Medical Robotics and Bionics , 5(1):66-78, 2023.

Bai, Q., A. S. Bedi, V. Aggarwal. Achieving zero constraint violation for constrained reinforcement learning via conservative natural policy gradient primal-dual algorithm. In Proceedings of the AAAI Conference on Artificial Intelligence , vol. 37, pages 6737-6744. 2023.

Ding, D., K. Zhang, T. Basar, M. Jovanovic. Natural policy gradient primal-dual method for constrained markov decision processes. Advances in Neural Information Processing Systems , 33:8378-8390, 2020.

Xu, T., Y. Liang, G. Lan. Crpo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning , pages 11480-11491. 2021.

Liu, Y., K. Zhang, T. Basar, W. Yin. An improved analysis of (variance-reduced) policy gradient and natural policy gradient methods. Advances in Neural Information Processing Systems , 33:7624-7636, 2020.

Jain, P., S. M. Kakade, R. Kidambi, P. Netrapalli, A. Sidford. Accelerating stochastic gradient descent for least squares regression. In Conference On Learning Theory , pages 545-604. 2018.

Liu, T., R. Zhou, D. Kalathil, P. Kumar, C. Tian. Policy optimization for constrained mdps with provable fast global convergence. arXiv preprint arXiv:2111.00552 , 2021.

Zeng, S., T. T. Doan, J. Romberg. Finite-time complexity of online primal-dual natural actorcritic algorithm for constrained markov decision processes. In 2022 IEEE 61st Conference on Decision and Control (CDC) , pages 4028-4033. IEEE, 2022.

Vaswani, S., L. Yang, C. Szepesvári. Near-optimal sample complexity bounds for constrained mdps. Advances in Neural Information Processing Systems , 35:3110-3122, 2022.

Agarwal, A., S. M. Kakade, J. D. Lee, G. Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. The Journal of Machine Learning Research , 22(1):4431-4506, 2021.

Bhandari, J., D. Russo. On the linear convergence of policy gradient methods for finite mdps. In International Conference on Artificial Intelligence and Statistics , pages 2386-2394. 2021.

Cen, S., C. Cheng, Y. Chen, Y. Wei, Y. Chi. Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research , 70(4):2563-2578, 2022.

Lan, G. Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes. Mathematical programming , 198(1):1059-1106, 2023.

Zhan, W., S. Cen, B. Huang, Y. Chen, J. D. Lee, Y. Chi. Policy mirror descent for regularized reinforcement learning: A generalized framework with linear convergence. SIAM Journal on Optimization , 33(2):1061-1091, 2023.

Xu, P., F. Gao, Q. Gu. An improved convergence analysis of stochastic variance-reduced policy gradient. In Uncertainty in Artificial Intelligence , pages 541-551. 2020.

Gargiani, M., A. Zanelli, A. Martinelli, T. Summers, J. Lygeros. Page-pg: A simple and loopless variance-reduced policy gradient method with probabilistic gradient estimation. In International Conference on Machine Learning , pages 7223-7240. 2022.

Huang, F., S. Gao, J. Pei, H. Huang. Momentum-based policy gradient methods. In International conference on machine learning , pages 4422-4433. 2020.

Salehkaleybar, S., M. Khorasani, N. Kiyavash, N. He, P. Thiran. Momentum-based policy gradient with second-order information. Transactions on Machine Learning Research , 2024.

Shen, Z., A. Ribeiro, H. Hassani, H. Qian, C. Mi. Hessian aided policy gradient. In International conference on machine learning , pages 5729-5738. 2019.

Chen, Z., S. Khodadadian, S. T. Maguluri. Finite-sample analysis of off-policy natural actorcritic with linear function approximation. IEEE Control Systems Letters , 6:2611-2616, 2022.

Chen, Z., S. T. Maguluri. Sample complexity of policy-based methods under off-policy sampling and linear function approximation. In International Conference on Artificial Intelligence and Statistics , pages 11195-11214. 2022.

Khodadadian, S., T. T. Doan, J. Romberg, S. T. Maguluri. Finite sample analysis of two-timescale natural actor-critic algorithm. IEEE Transactions on Automatic Control , 2022.

Fatkhullin, I., A. Barakat, A. Kireeva, N. He. Stochastic policy gradient methods: Improved sample complexity for fisher-non-degenerate policies. In International Conference on Machine Learning , pages 9827-9869. 2023.

Masiha, S., S. Salehkaleybar, N. He, N. Kiyavash, P. Thiran. Stochastic second-order methods improve best-known sample complexity of SGD for gradient-dominated functions. Advances in Neural Information Processing Systems , 35:10862-10875, 2022.

Mondal, W. U., V. Aggarwal. Improved sample complexity analysis of natural policy gradient algorithm with general parameterization for infinite horizon discounted reward Markov decision processes. In International Conference on Artificial Intelligence and Statistics , pages 3097-3105. 2024.

Efroni, Y., S. Mannor, M. Pirotta. Exploration-exploitation in constrained mdps. arXiv preprint arXiv:2003.02189 , 2020.

Liu, T., R. Zhou, D. Kalathil, P. Kumar, C. Tian. Learning policies with zero or bounded constraint violation for constrained mdps. Advances in Neural Information Processing Systems , 34:17183-17193, 2021.

Ding, D., X. Wei, Z. Yang, Z. Wang, M. Jovanovic. Provably efficient safe exploration via primal-dual policy optimization. In International Conference on Artificial Intelligence and Statistics , pages 3304-3312. 2021.

He, J., D. Zhou, Q. Gu. Nearly minimax optimal reinforcement learning for discounted mdps. Advances in Neural Information Processing Systems , 34:22288-22300, 2021.

Wei, H., X. Liu, L. Ying. Triple-q: A model-free algorithm for constrained reinforcement learning with sublinear regret and zero constraint violation. In International Conference on Artificial Intelligence and Statistics , pages 3274-3307. 2022.

Bai, Q., A. S. Bedi, M. Agarwal, A. Koppel, V. Aggarwal. Achieving zero constraint violation for constrained reinforcement learning via primal-dual approach. In Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, pages 3682-3689. 2022.

Sutton, R. S., D. McAllester, S. Singh, Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems , 12, 1999.

Zhang, J., C. Ni, C. Szepesvari, M. Wang. On the convergence and sample efficiency of variance-reduced policy gradient method. Advances in Neural Information Processing Systems , 34:2228-2240, 2021.

Jin, C., Z. Yang, Z. Wang, M. I. Jordan. Provably efficient reinforcement learning with linear function approximation. In J. Abernethy, S. Agarwal, eds., Proceedings of Thirty Third Conference on Learning Theory , vol. 125 of Proceedings of Machine Learning Research , pages 2137-2143. PMLR, 2020.

Wang, L., Q. Cai, Z. Yang, Z. Wang. Neural policy gradient methods: Global optimality and rates of convergence. In International Conference on Learning Representations . 2019.

Zhang, K., A. Koppel, H. Zhu, T. Basar. Global convergence of policy gradient methods to (almost) locally optimal policies. SIAM Journal on Control and Optimization , 58(6):3586-3612, 2020.

Mondal, W. U., V. Aggarwal, S. Ukkusuri. Mean-field control based approximation of multiagent reinforcement learning in presence of a non-decomposable shared global state. Transactions on Machine Learning Research , 2023.

Ding, D., K. Zhang, J. Duan, T. Ba¸ sar, M. R. Jovanovi´ c. Convergence and sample complexity of natural policy gradient primal-dual methods for constrained mdps. arXiv preprint arXiv:2206.02346 , 2022.

A Helper Lemma

Lemma 6. With slight abuse of notation, define J L ,ρ ( π, λ ) ≜ J π r,ρ + λJ π c,ρ . The following relation holds for any two policies π 1 , π 2 and λ ∈ Λ .

Proof. This can be proved using Lemma 2 of [12] and the definition of the Lagrange function.

B Proof of Lemma 1

Proof. Fix arbitrary θ and λ . Note that the following equation holds due to the definition of ˆ J c,ρ ( θ ) .

To prove the unbiasedness of the gradient, we first prove that the distribution of the sample pair (ˆ s, ˆ a ) produced by Algorithm 1 is indeed ν π θ ρ . Observe the following.

Next, we show that for a given pair ( s, a ) , the estimate ˆ Q π θ g ( s, a ) , g ∈ { r, c } is unbiased.

In a similar fashion, one can establish that E [ ˆ V π θ g ( s ) ∣ ∣ θ, s ] = V π θ g ( s ) , ∀ g ∈ { r, c } . Combining this with (34) leads to: E [ ˆ A π θ L ,λ ( s, a ) | θ, λ, s, a ] = A π θ L ,λ ( s, a ) . We arrive at,

Finally, we arrive at the following by utilizing the definitions (10) and (14).

This concludes the proof.

C Proof of Lemma 3

Proof. Invoking the definition of KL divergence, we arrive at the following series of inequalities.

where the step (a) holds by Assumption 2 and step (b) holds by Lemma 6. Step (c) uses the convexity of the function f ( x ) = x 2 . Finally, step (d) comes from the Assumption 3. Rearranging the terms and taking expectations on both sides, we have,

where (a) follows from Assumption 2. Summing from k = 0 to K -1 , using the non-negativity of KL divergence and dividing the resulting expression by K , we obtain,

This concludes the proof.

D Proof of Lemma 4

Proof. Fix a θ ∈ Θ and a λ ∈ Λ . Observe the following equation.

To prove the lemma, it is sufficient to demonstrate that E [ ζ θ,λ ( s, a )] ≤ σ 2 , ∀ ( s, a ) . Notice the chain of inequalities stated below.

Inequality ( a ) follows from the Cauchy-Schwarz inequality and the fact that ( a + b ) 2 ≤ 2( a 2 + b 2 ) for any two numbers a, b . The same argument is also applied in inequality ( b ) . Additionally, it uses Assumption 2 and the definition of ω ∗ θ,λ . Finally, ( c ) is a consequence of Assumption 4, Lemma 2, and the following two bounds.

To establish the first bound, note that | ˆ Q π θ g ( s, a ) | is assigned a value of at most ( j +1) with probability (1 -γ ) γ j , ∀ g ∈ { r, c } . Therefore,

The second bound in (42) can be proven similarly. This concludes the lemma.

E Proof of Lemma 5

We establish Lemma 5 applying Corollary 2 of [8]. Note the following statements.

S1 : The following quantities exist and are finite ∀ θ ∈ Θ

.

S2 : There exists σ 2 such that the following is obeyed ∀ θ ∈ Θ where ω ∗ θ,λ minimizes L ν π θ ρ ( · , θ, λ ) .

S3 : There exists µ F , G > 0 such that the following statements hold ∀ θ ∈ Θ .

Statement S1 follows from Assumption 2 whereas S2 is a consequence of Lemma 4. Statement S3 ( a ) is identical to Assumption 4, S3 ( b ) results from Assumption 2, and finally, S3 ( c ) follows from Assumption 2 and 4. We can, therefore, apply Corollary 2 of [8] with κ = ˜ κ = G 2 /µ F and deduce the following convergence result whenever H > ¯ C √ κ ˜ κ log( √ d √ κ ˜ κ ) and the learning rates are set as α = 3 √ 5 √ κ ˜ κ 1+3 √ 5 κ ˜ κ , β = 1 9 √ κ ˜ κ , ξ = 1 3 √ 5 µ F √ κ ˜ κ , and δ = 1 5 G 2 .

The term, C is a universal constant. Note that l k ( ω ∗ k ) ≥ 0 and l k ( 0 ) is bounded above as follows.

where ( a ) is a result of the fact that | A π θ L ,λ ( s, a ) | ≤ (1 + λ max ) / (1 -γ ) , ∀ ( s, a ) ∈ S × A , ∀ θ ∈ Θ , and ∀ λ ∈ Λ . Combining (50), (51), and the fact that l k ( · ) is µ F -strongly convex, we establish,

This proves the first statement. We get the following for noiseless ( σ 2 = 0 ) gradient updates.

The second statement can be established from (53) by applying Jensen's inequality on the function f ( x ) = x 2 .

F Proof of Theorem 1

Applying the inequalities H ≥ 1 , exp( -( µ F / 20 G 2 ) H ) ≤ 1 and substituting the values of σ 2 and η (as stated in Lemma 4 and Theorem 1 respectively), we can rewrite (29) as follows.

The terms f 0 and f 1 are defined below.

Using the definition of J L ,ρ ( · , · ) , one can write,

where (a) follows from the fact that λ k ≥ 0 and J π ∗ c,ρ ≥ 0 due to feasibility. Combining (54) and (56), we obtain the following.

F.1 Convergence Rate of the Objective Function

Note that (57) can be alternatively written as,

To obtain the convergence rate of the objective function, we need to bound the last term in the above expression. Observe the following chain of inequalities.

where (a) uses λ 0 = 0 and (b) follows from the contraction property of the projection operator, P Λ . Using the above inequality, one can write,

Using the unbiasedness of ˆ J c,ρ ( θ k ) (Lemma 1), we deduce the following.

where ( a ) is a consequence of the fact that ˆ J c,ρ ( θ k ) and λ k are conditionally independent given θ k . Note that, ˆ J 2 c,ρ ( θ k ) is assigned a value of at most ( j +1) 2 with probability (1 -γ ) γ j , j ∈ { 0 , 1 , · · · } . Therefore, the RHS of (60) can be bounded as follows.

Combining (58), (60), (61), and (62), we finally obtain,

where ( a ) uses the substitution ζ = λ max (1 -γ ) / K .

F.2 Rate of Constraint Violation

The following inequality is satisfied for any k ∈ { 0 , 1 , · · · , K -1 } .

where (a) is due to the contractive property of P Λ . Performing an average over k ∈ { 0 , · · · , K -1 } , and applying expectations on both sides, we get,

where ( a ) utilises λ 0 = 0 , ( b ) applies (62), and ( c ) is derived using ζ = λ max (1 -γ ) / √ K . Note that one can write the following using Lemma 1 and the observation that ˆ J c,ρ ( θ k ) and λ k are conditionally independent given θ k .

Combining (65) and (66), we get,

Finally, combining (58) and (67), we arrive at,

Since the functions { J g,ρ ( θ k ) } , g ∈ { r, c } , k ∈ { 0 , · · · , K -1 } are linear in occupancy measure, there exists a policy ¯ π such that the following holds ∀ g ∈ { r, c } .

This allows us to rewrite (68) as,

Applying Lemma 9 and verifying (via Lemma 8) that λ max ≥ 2 λ ∗ where λ ∗ is the non-negative minimizer of the dual function corresponding to the unparameterized constrained optimization (2), we can write the constraint violation rate as follows.

F.3 Final Result

Substituting λ max = 2 / [(1 -γ ) c slater ] ≥ 2 in (63), we get the rate of convergence of the objective as follows.

Similarly, we obtain the constraint violation rate as,

Let, H and K are chosen as follows for an arbitrary ϵ > 0 .

For the above choice of H and K , we have,

Note that the expected number of steps required in Algorithm 1 is O ((1 -γ ) -1 ) . Therefore, the sample complexity required to ensure (74) is O ((1 -γ ) -1 HK ) = ˜ O ((1 -γ ) -7 ϵ -2 ) . Finally, note that Lemma 5 requires H > ¯ C G 2 µ F log ( √ d G 2 µ F ) . This can be ensured if ϵ is sufficiently small.

G Strong Duality and Related Lemmas

Define the dual function associated with the unparameterized constrained optimization (2) as follows.

The following lemma formally describes the strong duality result.

Lemma 7. [40] [Lemma 3] If λ ∗ ≜ arg min λ ≥ 0 J λ D ,ρ and π ∗ is a solution of (2) , then the following holds whenever Assumption 1 is true.

It is to be mentioned that strong duality, in general, does not hold for the parameterized optimization (3). The following lemma established a bound on λ ∗ which becomes the foundation for choosing the value of λ max in Algorithm 2.

Lemma 8. [40][Lemma 3] Let λ ∗ be the optimal dual variable as defined in Lemma 7. The following inequalities hold where c slater is defined in Assumption 1.

The following lemma is the main tool in decoupling the objective and constraint violation rates.

Lemma 9. Let Slater's condition (Assumption 1) hold. If C ≥ 2 λ ∗ where λ ∗ is defined in Lemma 7 and ¯ π is a policy such that J π ∗ r,ρ -J ¯ π r,ρ + C [ -J ¯ π c,ρ ] ≤ ζ for some ζ > 0 then

Proof. Define the function v ( · ) as follows.

Let τ = J ¯ π c,ρ . Therefore, one can write the following chain of inequalities.

where (a) follows from the definition of v ( τ ) , (b) uses λ ∗ ≥ 0 , (c) follows from the definition of the dual function, and (d) is a result of the strong duality (Lemma 7). Utilizing (79), we get,

This leads to -τ ≤ ζ/ ( C -λ ∗ ) ≤ 2 ζ/C .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state the paper's contribution and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Some limitations of the paper have been discussed in the conclusion section. Moreover, some assumptions are also accompanied by elaborate discussion on their limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have clearly stated the required assumptions and an accompanying complete proof in the appendix for each theory result.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: Our paper is primarily of theoretical nature and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: Our paper is primarily of theoretical nature and does not include experiments. Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: Our paper is primarily of theoretical nature and does not include experiments. Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: Our paper is primarily of theoretical nature and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: Our paper is primarily of theoretical nature and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conforms, in every respect, to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work is primarily of theoretical nature and has no immediate societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No high risk data or model have been used.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: No existing asset has been used in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new asset is introduced in the paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No experiments with human subjects were conducted.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We conducted no experiments with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper studies constrained Markov decision processes. Using momentum acceleration, the paper develops a primal-dual natural policy gradient-based algorithm and shows that it achieves an \epsilon global optimality with O(\epsilon^-3) samples. The policy class considered in the paper is the generic Fisher non-degenerate policies.

STRENGTHS:
* The paper is the first in the literature that derives the O(\epsilon^-3) sample complexity for constrained MDP under more general policy parameterizations than the softmax.
* The presentation of theoretical results are clear and easy to follow. I appreciate that the author presents a complete flow of proof in the main body of the paper.

WEAKNESSES:
I mainly have the following concern, and if it can be addressed, then I am willing to adjust my rating. In literature [25], a sample complexity of O(\epsilon^-2) is derived for unconstrained MDP. To me, it seems it is not impossible to translate this result to the constrained setting by adding a primal-dual component, and if this can be done, I think the sample complexity should be O(\epsilon^-3) or so. I wonder could the author discuss the feasibility of this generalization (and perhaps the underlying challenges if it is not generalizable).

QUESTIONS:
* I hope the author could discuss the concern I raised in the weakness section.
* Since the paper actually studies Fisher non-degenerate policies, I think a more common (and precise) way is to directly use the name "Fisher non-degenerate policies". Using the name "general parameterization" is a little bit misleading.

LIMITATIONS:
The author has discussed the limitation of the paper briefly in the conclusion section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the constrained MDP setting. Given a target error of $\\epsilon > 0$, the paper provides a primal-dual algorithm on that will return an epsilon optimal policy making at most epsilon constraint violations.  The claim is that the algorithm solves such problem with O($\\epsilon^{-3}$) sample complexity.\n\nSTRENGTHS:\nThe idea of incorporating momentum acceleration to a primal-dual algorithm seems interesting.\n\nWEAKNESSES:\nThe Slater's constant $c_{slater}$ is missing from the big O-notations in your $H$ and $K$ terms.  The slater's constant is an unique quantity in CMDP and should be clearly stated in your sample complexity.  In the relaxed-feasibility problem, where the algorithm returns a policy that is allowed to have some small constraint violation (such as epsilon constraint violation), the Slater's constant can be smaller than the target error epsilon.  In this case, the Slater's constant can be set to be the $\\epsilon$.  Your paper is in this relaxed-feasibility setting.  \n\nBy writing the Slater's constant $c_{slater}$ explicitly in the big O-notations, we see that $H = \\mathcal{O}((1-\\gamma)^{-3} \\epsilon^{-1} c_{slater}^{-1})$ and $K = \\mathcal{O}((1-\\gamma)^{-4} \\epsilon^{-2} c_{slater}^{-2})$. If the Slater's constant = epsilon, then we see that $H \\propto \\epsilon^{-2}$ and $K \\propto \\epsilon^{-4}$ making the total sample complexity to be proportional to $\\epsilon^{-6}$. Then, the result is independent of the Slater's constant.\n\nQUESTIONS:\nYour result stated in Lemma 6 is based on a previous result provided by [6]. In [6], the result relies on an oracle that, when given an input, provides a noisy unbiased gradient of a function. However, your algorithm uses a sampling procedure to obtain a gradient estimate, which appears to rely on access to a generative model (Kakade (2003) On the sample complexity of reinforcement learning). Are you considering the setting of having access to a generative model, or are you assuming access to an oracle as in [6], for which your Lemma 6 is based? If you are considering the setting of having access to a generative model, the paper should address the statistical challenges associated with sampling, and the sample complexity should include the number of queries to a simulator, incorporating $T$ into the sample complexity. Therefore, the result from [6] in Lemma 6 does not immediately apply. If you are considering the setting of having access to an oracle, it is not clear why Algorithm 1 is necessary.\n\nLIMITATIONS:\nBy stating the Slater's constant in the sample complexity, the sample complexity is in the order of epsilon to the minus 6 instead of minus 3 so to be independent of the Slater's constant.  See comments in the Weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied the sample complexity of learning a discounted-reward MDP with a discounted cost constraint under a general policy parameterization. \nIt proposes a policy-gradient-type of algorithm that combines natural policy gradient and an accelerating stochastic gradient descent algorithm proposed in [6]. \n\nThen the paper proves that with $O(1/\\epsilon^3)$ samples, the policy learned by this algorithm violates the constraint by at most $\\epsilon$, and is optimal up to $\\epsilon_{bias} + \\epsilon$ error, where $\\epsilon_{bias}$ is a constant coming from policy parameterization. \n\nThe sample complexity of this policy improves upon the prior SOTA, which was $O(1/\\epsilon^4)$.\n\nSTRENGTHS:\nThe paper is well-written. The main contribution, the algorithm, the key idea of the proof and the proof structure are clearly presented. \n\nThe improvement of sample complexity from $O(1/\\epsilon^4)$ to $O(1/\\epsilon^3)$ is non-trivial, following from a clever observation given in lemma 3. \n\nThe analysis looks solid, though I do have questions about a few minor details, see the questions 1 and 2 below.\n\nWEAKNESSES:\n1\nThe “sample efficient” in the title seems to overclaim the contribution of the paper, since there is still an order gap between the sample complexity of the algorithm in this paper, $O(\\epsilon^{-3})$, and the lower bound, $O(\\epsilon^{-2})$. \n\n2\nIt is not entirely accurate to say that the algorithm achieves “$\\epsilon$ global optimality gap” and “$\\epsilon$ constraint violation”, as the actual bounds in Theorem 1 contain constant terms involving $\\epsilon_{bias}$. \nAlthough $\\epsilon_{bias}$ is common in the analysis of RL algorithms with general parameterizations, it is crucial to discuss whether they are tight and optimal, how they compare to the prior work, and whether they are truly negligible when the policy is parameterized by a neural net of a reasonable scale. \nAt the very least, the sense in which your policy is epsilon-optimal should be clarified in a prominent part of the paper.\n\nQUESTIONS:\n1 I believe the definition of L in (9) has a typo: the square should be inside the expectation, rather than outside the expectation, so that its gradient with respect to w is given by (10). \nDoes this typo affect the proof? \n\n2 The logic from (75) to (76) is not very clear. How did you get rid of the first term on LHS of (76), and where does $\\lambda^*$ comes from?\n\nLIMITATIONS:\nThere is no negative social impact. A discussion of limitation is included in the submission.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the sample complexity of the constrained Markov Decision Process (CMDP), and derives an algorithm that attains the $O(\\epsilon^{-3})$ that improves the SOTA sample complexity bound for CMDP by a factor of $O(\\epsilon^{-1})$.\n\nSTRENGTHS:\nCMDP with general parameterization is a very challenging problem, and thus there have not been many papers making progress in this challenging setting compared to the tabular CMDP setting. The authors modify the classic primal-dual natural policy gradient (NPG) framework by leveraging an accelerated stochastic gradient descent (ASGD) method to improve the SOTA sample complexity bound by a factor of $O(\\epsilon^{-1})$.\n\nWEAKNESSES:\nEven though the authors claim they have improved the SOTA sample complexity bound for CMDP with general parameterization, I still think it is hard to say the authors have \"beat\" the previous results. $\\epsilon$ is not the only factor that matters for the sample complexity bound for discounted infinite-horizon MDPs, since $\\gamma$ is also another significant factor. Unfortunately, it seems that the authors' bound has a dependency of $O(1/(1-\\gamma)^{-8})$, so that makes the sample complexity bound read $O(1/(1-\\gamma)^8\\epsilon^3)$. Compared to a prior work [1] which has a sample complexity bound of $O(1/(1-\\gamma)^6\\epsilon^4)$, it is clear the two bounds are not necessarily comparable due to one bound is superior in terms of the order of $\\epsilon$ and the other superior in terms of the order of $\\gamma$. \n\nAlso, [1]'s algorithm has zero constraint violation while the authors' algorithm permits $O(\\epsilon)$ constraint violation, which is a worse result.\n\nHence, unless the authors can clarify their result has improvement in $\\epsilon$ and $\\gamma$ and constraint violations (or at least provide stronger justifications that their algorithm requires a smaller number of samples to achieve near-optimality), it is hard to evaluate the value of the authors' result.\n\n[1] Bai, Qinbo, Amrit Singh Bedi, and Vaneet Aggarwal. \"Achieving zero constraint violation for constrained reinforcement learning via conservative natural policy gradient primal-dual algorithm.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 6. 2023.\n\nQUESTIONS:\nAs mentioned, the major weakness of the paper is that you have overlooked the sample complexity bound's dependency in $\\gamma$ and it seems you have a worse constraint violation result. Can you update Table 1 with sample complexity bounds having dependency in $\\gamma$ and with constraint violations? More importantly, can you justify why your result has seemingly worse dependency in $\\gamma$ and constraint violations?\n\nLIMITATIONS:\nThe major limitation of the paper is the lack of proper comparison of the sample complexity bounds with prior works.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}