Title: Learning General Parameterized Policies for Infinite Horizon Average Reward Constrained MDPs via Primal-Dual Policy Gradient Algorithm
Venue: NeurIPS 2024
Accepted: yes
Keywords: Average Reward MDP

Constraint Violation

Regret.
Abstract: This paper explores the realm of infinite horizon average reward Constrained Markov Decision Processes (CMDPs). To the best of our knowledge, this work is the first to delve into the regret and constraint violation analysis of average reward CMDPs with a general policy parametrization. To address this challenge, we propose a primal dual-based policy gradient algorithm that adeptly manages the constraints while ensuring a low regret guarantee toward achieving a global optimal policy. In particular, our proposed algorithm achieves $\tilde{\mathcal{O}}({T}^{4/5})$ objective regret and $\tilde{\mathcal{O}}({T}^{4/5})$ constraint violation bounds.

Learning General Parameterized Policies for Infinite Horizon Average Reward Constrained MDPs via Primal-Dual Policy Gradient Algorithm

Abstract

This paper explores the realm of infinite horizon average reward Constrained Markov Decision Processes (CMDPs). To the best of our knowledge, this work is the first to delve into the regret and constraint violation analysis of average reward CMDPs with a general policy parametrization. To address this challenge, we propose a primal dual-based policy gradient algorithm that adeptly manages the constraints while ensuring a low regret guarantee toward achieving a global optimal policy. In particular, our proposed algorithm achieves ˜ O ( T 4 / 5 ) objective regret and ˜ O ( T 4 / 5 ) constraint violation bounds.

1 Introduction

The framework of Reinforcement Learning (RL) is concerned with a class of problems where an agent learns to yield the maximum cumulative reward in an unknown environment via repeated interaction. RL finds applications in diverse areas, such as wireless communication, transportation, and epidemic control [1, 2, 3]. RL problems are mainly categorized into three setups: episodic, infinite horizon discounted reward, and infinite horizon average reward. Among them, the infinite horizon average reward setup is particularly significant for real-world applications. It aligns with most of the practical scenarios and captures their long-term goals. Some applications in real life require the learning procedure to respect the boundaries of certain constraints. In an epidemic control setup, for example, vaccination policies must take the supply shortage (budget constraint) into account. Such restrictive decision-making routines are described by constrained Markov Decision Processes (CMDP) [4, 5, 6]. Existing papers on CMDPs utilize either a tabular or a linear MDP structure. This work provides the first algorithm for an infinite horizon average reward CMDP with general parametrization and proves its sub-linear regret and constraint violation bounds.

There are two primary ways to solve a CMDP problem in the infinite horizon average reward setting. The first one, known as the model-based approach, involves constructing estimates of the transition probabilities of the underlying CMDP, which are subsequently utilized to derive policies [6, 7, 5]. The caveat of this approach is the large memory requirement to store the estimated parameters, which effectively curtails its applicability to CMDPs with large state spaces. The alternative strategy, known as the model-free approach, either directly estimates the policy function or maintains an estimate of the Q function, which is subsequently used for policy generation [8]. Model-free algorithms typically demand lower memory and computational resources than their model-based counterparts. Although the CMDP has been solved in a model-free manner in the tabular [8] and linear [9] setups, its exploration with the general parameterization is still open and is the goal of this paper.

General parameterization indexes the policies by finite-dimensional parameters (e.g., weights of neural networks) to accommodate large state spaces. The learning is manifested by updating these

Table 1: This table summarizes the different model-based and mode-free state-of-the-art algorithms available in the literature for average reward CMDPs. We note that our proposed algorithm is the first to analyze the regret and constraint violation for average reward CMDP with general parametrization. Here, the parameter d refers to the dimension of the feature map for linear MDPs.

parameters using policy gradient (PG)-type algorithms. Note that PG algorithms are primarily studied in discounted reward setups. For example, [10] characterizes the sample complexities of the PG and the Natural PG (NPG) algorithms with softmax and direct parameterization. Similar results for general parameterization are obtained by [11, 12]. The regret analysis of a PG algorithm with the general parameterization has been recently performed for an infinite horizon average reward MDP without constraints [13]. Similar regret and constraint violation analysis for the average reward CMDP is still missing in the literature. In this paper, we bridge this gap.

Challenges and Contribution: We propose a PG-based algorithm with general parameterized policies for the average reward CMDP and establish its sublinear regret and constraint violation bounds. In particular, assuming the underlying CMDP to be ergodic, we demonstrate that our PG algorithm achieves an average optimality rate of ˜ O ( T -1 5 ) and average constraint violation rate of ˜ O ( T -1 5 ) . Invoking this convergence result, we establish that our algorithm achieves regret and constraint violation bounds of ˜ O ( T 4 5 ) . Apart from providing the first sublinear regret guarantee for the average reward CMDP with general parameterization, our work also improves the state-of-the-art regret guarantee, ˜ O ( T 5 / 6 ) in the model-free tabular setup [8].

Despite the availability of sample complexity analysis of PG algorithms with constraints in the discounted reward setup [14, 4] and PG algorithms without constraint in average reward setup [13], obtaining sublinear regret and constraint violation bounds for their average reward counterpart is challenging.

·[14, 4] solely needs an estimate of the value function V while we additionally need the estimate of the gain function, J .

·[14, 4] assume access to a simulator to generate unbiased value estimates. In contrast, our algorithm uses a sample trajectory of length H to estimate the values and gains and does not assume the availability of a simulator.

·The first-order convergence analysis (Lemma 6) differs from that in [13]. Note that both of these papers use an ascent-like inequality. In [13], this bounds the term J ( θ k +1 ) -J ( θ k ) . The final result is obtained by calculating a sum over k which cancels the intermediate terms and leaves us with J ( θ K ) -J ( θ 1 ) . We would like to emphasize that the cancellation of the intermediate terms is crucial to establishing the result. However, a similar effort in our case only leads to a bound of J L ( θ k +1 , λ k ) -J L ( θ k , λ k ) . Note that directly performing a sum over this difference does not lead to the cancellation of intermediate terms. We had to take a different route and apply the bounds of the Lagrange multipliers and the estimate of the constraint function to achieve that goal.

·After solving the problems mentioned above, we prove ˜ O ( T -1 5 ) convergence rate of the Lagrange function. Unfortunately, the strong duality property, which is central to proving convergence results of CMDPs for tabular and softmax policies, does not hold under the general parameterization. As a result, the convergence result for the dual problem does not automatically translate to that for the primal problem, which is a main difference from [13]. We overcome this barrier by introducing a novel constraint violation analysis and a series of intermediate results (Lemma 16-18) that help disentangle the regret and constraint violation rates from the Lagrange convergence. It is important to mention that although the techniques applied are inspired by the [14], those techniques cannot be directly adopted for average reward MDPs. This is primarily because the estimate ˆ J c ( θ k ) is biased in the average case. To the best of our knowledge, constraint violation analysis with a biased estimate of the cost value is not available in the literature and is performed for the first time in our paper.

·Due to the presence of the Lagrange multiplier, the convergence analysis of a CMDP is much more convoluted than its unconstrained counterpart. The learning rate of the Lagrange update, β , turns out to be pivotal in determining the growth rate of regret and constraint violation. Low values of β push the regret down while simultaneously increasing the constraint violation. Finding the optimal value of β that judiciously balances these two competing goals is one of the cornerstones of our analysis.

Related work for unconstrained average reward RL: In the absence of constraints, both modelbased and model-free tabular setups have been widely studied for infinite horizon average reward MDPs. For example, the model-based algorithms proposed by [15, 16] achieve the optimal regret bound of ˜ O ( √ T ) . Similarly, the model-free algorithm proposed by [17] for tabular MDP results in ˜ O ( √ T ) regret. Regret analysis for average reward MDP with general parametrization has been recently studied in [13], where a regret bound of ˜ O ( T 3 / 4 ) is derived.

Related work for constrained RL: The constrained reinforcement learning problem has been extensively studied both for infinite horizon discounted reward and episodic MDPs. For example, discounted reward CMDPs have been recently studied in the tabular setup [18], with both softmax [14, 19], and general policy parameterization [14, 19, 4, 12]. Moreover, [20, 21, 22] investigated episodic CMDPs in the tabular setting.

Recently, the infinite horizon average reward CMDPs have been investigated in model-based setups [5, 6, 7], tabular model-free setting [8] and linear CMDP setting [9]. For model-based CMDP setup, [6] proposed a model-based online mirror descent algorithm in the ergodic setting which achieves ˜ O ( √ T ) for regret and violation at the same time. [7] proposed algorithms based on the posterior sampling and the optimism principle that achieve ˜ O ( √ T ) regret with zero constraint violations in the ergodic setting. However, the above model-based algorithms cannot be extended to large state space. In the tabular model-free setup, the algorithm proposed by [8] achieves a regret of ˜ O ( T 5 / 6 ) with zero constraint violations. Finally, in the linear CMDP setting, [9] achieves ˜ O ( √ T ) regret bound with zero constraint violation. Note that the linear CMDP setting assumes that the transition probability has a certain linear structure with a known feature map which is not realistic. Table 1 summarizes all relevant works. Unfortunately, none of these papers study the infinite horizon average reward CMDPs with general parametrization which is the main focus of our article.

Additionally, for the weakly communicating setting, [6] proposed a model-based algorithm achieving ˜ O ( T 2 / 3 ) for both regret and violation in tabular case. [9] further extends such result to linear MDP setting with ˜ O ( T 3 / 4 ) regret and violation. In general, it is difficult to propose a model-free algorithm with provable guarantees for Constrained MDPs (CMDPs) without considering the ergodic model. [6] pointed out several extra challenges in Weakly communicating MDP compared to the ergodic case. For example, there is no uniform bound for the span of the value function for all stationary policies. It is also unclear how to estimate a policy's bias function accurately without the estimated model, which is an important step for estimating the policy gradient.

2 Formulation

This paper analyzes an infinite-horizon average reward constrained Markov Decision Process (CMDP) denoted as M = ( S , A , r, c, P, ρ ) where S denotes the state space, A is the action space of size A , r : S × A → [0 , 1] is the reward function, c : S × A → [ -1 , 1] is the constraint cost function, P : S × A → ∆ |S| is the state transition function where ∆ |S| denotes a probability simplex with dimension |S| , and ρ ∈ ∆ |S| is the initial distribution of states. A policy π ∈ Π : S → ∆ A maps the current state to an action distribution. The average reward and cost of a policy, π , is,

where g = r, c for average reward and cost respectively. The expectation is calculated over the distribution of all sampled trajectories { ( s t , a t ) } ∞ t =0 where a t ∼ π ( s t ) , s t +1 ∼ P ( ·| s t , a t ) , ∀ t ∈ { 0 , 1 , · · · } . For notational convenience, we shall drop the dependence on ρ whenever there is no confusion. Our goal is to maximize the average reward function while ensuring that the average cost is above a given threshold. Without loss of generality, we can mathematically write this problem as,

However, the above problem becomes difficult to handle when the underlying state space, S is large. Therefore, we consider a class of parametrized policies, { π θ | θ ∈ Θ } whose elements are indexed by a d -dimensional parameter, θ ∈ R d where d ≪|S||A| . Thus, the original problem in Eq (2) can be reformulated as the following parameterized problem.

We denote J π θ g = J g ( θ ) , g ∈ { r, c } for notational convenience. Let, P π θ : S → ∆ |S| be a transition function induced by π θ and defined as, P π θ ( s, s ′ ) = ∑ a ∈A P ( s ′ | s, a ) π θ ( a | s ) , ∀ s, s ′ . If M is such that for every policy π , the function, P π is irreducible and aperiodic, then M is called ergodic.

Assumption 1. The CMDP M is ergodic.

Ergodicity is a common assumption in the literature [23, 24]. If M is ergodic, then ∀ θ , there exists a unique stationary distribution, d π θ ∈ ∆ |S| given as follows.

Ergodicity implies that d π θ is independent of the initial distribution, ρ , and obeys P π θ d π θ = d π θ . Hence, the average reward and cost functions can be expressed as,

where g π θ ( s ) ≜ ∑ a ∈A g ( s, a ) π θ ( a | s ) , g ∈ { r, c } . Note that the functions J g ( θ ) , g ∈ { r, c } are also independent of the initial distribution, ρ . Furthermore, ∀ θ , there exist a function Q π θ g : S × A → R such that the following Bellman equation is satisfied ∀ ( s, a ) ∈ S × A .

where g ∈ { r, c } and V π θ g : S → R is given as V π θ g ( s ) = ∑ a ∈A π θ ( a | s ) Q π θ g ( s, a ) , ∀ s ∈ S . Note that if Q π θ g satisfies (6) , then it is also satisfied by Q π θ g + c for any arbitrary, c . To uniquely define the value functions, we assume that ∑ s ∈S d π θ ( s ) V π θ g ( s ) = 0 . In this case, V π θ g ( s ) is given by,

where the expectation is computed over all π θ -induced trajectories. In a similar way, ∀ ( s, a ) , one can uniquely define Q π θ g ( s, a ) , g ∈ { r, c } as follows.

Moreover, the advantage function A π θ g : S × A → R is defined such that A π θ g ( s, a ) ≜ Q π θ g ( s, a ) -V π θ g ( s ) , ∀ ( s, a ) , ∀ g ∈ { r, c } . Assumption 1 also implies the existence of a finite mixing time. Specifically, for an ergodic MDP, M , the mixing time is defined as follows.

Definition 1. The mixing time, t θ mix , of the CMDP M for a parameterized policy, π θ , is defined as, t θ mix ≜ min { t ≥ 1 ∣ ∣ ∥ ( P π θ ) t ( s, · ) -d π θ ∥ ≤ 1 4 , ∀ s } . The overall mixing time is t mix ≜ sup θ ∈ Θ t θ mix . In this paper, t mix is finite due to ergodicity.

Mixing time characterizes how fast a CMDP converges to its stationary state distribution, d π θ , under a given policy, π θ . We also define the hitting time as follows.

Definition 2. The hitting time of an ergodic CMDP M with respect to a policy, π θ , is defined as t θ hit ≜ max s ∈S [ d π θ ( s )] -1 . The overall hitting time is defined as t hit ≜ sup θ ∈ Θ t θ hit . In this paper, t hit is finite due to ergodicity as well.

Define π ∗ as the optimal solution to the unparameterized problem (2). For a given CMDP M , and a time horizon T , the regret and constraint violation of any algorithm A is defined as follows.

where the algorithm, A , executes the actions, { a t } , t ∈ { 0 , 1 , · · · } based on the trajectory observed up to time, t , and the state, s t +1 is decided according to the state transition function, P . For simplicity, we shall denote the regret and constraint violation as Reg T and Vio T respectively. Our goal is to design an algorithm A that achieves low regret and constraint violation bounds.

3 Proposed Algorithm

We solve (3) via a primal-dual algorithm based on the following problem.

where J L ( θ, λ ) ≜ J r ( θ ) + λJ c ( θ ) . The function, J L ( · , · ) , is called the Lagrange function and λ the Lagrange multiplier. Our algorithm updates the pair ( θ, λ ) following the policy gradient iteration as shown below ∀ k ∈ { 1 , · · · , K } with an initial point ( θ 1 , λ 1 ) , λ 1 = 0 .

where α and β are learning parameters and δ is the Slater parameter introduced in the following assumption. Finally, for any set, Λ , P Λ [ · ] denotes projection onto Λ . The assumption stated below ensures that we have at least one feasible interior point solution to (2).

Assumption 2 (Slater condition) . There exists a δ ∈ (0 , 1) and ¯ θ ∈ Θ such that J c ( ¯ θ ) ≥ δ .

Note that in (11), the dual update is projected onto the set [0 , 2 δ ] because the optimal dual variable for the parameterized problem is bounded in Lemma 16. The gradient of J L ( · , λ ) can be computed by invoking a variant of the well-known policy gradient theorem [25].

Lemma 1. The gradient of J L ( · , λ ) is computed as,

Algorithm 1 Primal-Dual Parameterized Policy Gradient

1: Input: Episode length H , learning rates α, β , initial parameters θ 1 , λ 1 , initial state s 0 ∼ ρ ( · ) ,

2: K = T/H

3: for k ∈ { 1 , · · · , K } do

4: T k ← ϕ

5: for t ∈ { ( k -1) H, · · · , kH -1 } do

6: Execute a t ∼ π θ k ( ·| s t )

7: Observe r ( s t , a t ) , c ( s t , a t ) and s t +1

8: T k ←T k ∪ { ( s t , a t ) }

9: end for

10: for t ∈ { ( k -1) H, · · · , kH -1 } do

11: Obtain ˆ A π θ k L ,λ k ( s t , a t ) via Algorithm 2 and T k

12: end for

13: Compute ω k using (15)

14: Update the parameters:

15: end for

where ∀ ( s, a ) , A π θ L ,λ ( s, a ) ≜ A π θ r ( s, a ) + λA π θ c ( s, a ) , and { A π θ g } g ∈{ r,c } are the advantage functions corresponding to reward and cost. In typical RL scenarios, learners do not have access to the state transition function, P , and thereby to the functions d π θ and A π θ L ,λ . This makes computing the exact

gradient a difficult task. In Algorithm 1, we demonstrate how one can still obtain good estimates of the gradient using sampled trajectories.

Algorithm 1 runs K epochs, each of duration H = 16 t hit t mix T ξ (log T ) 2 where ξ ∈ (0 , 1) defines a constant whose value is specified later. Clearly, K = T/H . Note that the learner is assumed to know the horizon length, T . This can be relaxed utilizing the well-known doubling trick [26]. Additionally, it is assumed that the algorithm is aware of the mixing time and the hitting time. This assumption is common in the literature [13, 17]. The first step in obtaining a gradient estimate is estimating the advantage value for a given pair ( s, a ) . This can be accomplished via Algorithm 2. At the k th epoch, a π θ k -induced trajectory, T k = { ( s t , a t ) } kH -1 t =( k -1) H is obtained and subsequently passed to Algorithm 2 that searches for subtrajectories within it that start with a given state s , are of length N = 4 t mix (log T ) , and are at least N distance apart from each other. Assume that there are M such subtrajectories. Let the total reward and cost of the i th subtrajectory be { r i , c i } respectively and τ i be its starting time. The value function estimates for the k th epoch are

This leads to the following advantage estimator.

where ˆ A π θ k g ( s, a ) = ˆ Q π θ k g ( s, a ) -ˆ V π θ k g ( s ) , g ∈ { r, c } . Finally, the gradient estimator is,

where t k = ( k -1) H is the starting time of the k th epoch. The parameters are updated following (12). To update the Lagrange multiplier, we need an estimation of J c ( θ k ) , which is obtained as the average cost of the k th epoch. It should be noted that we remove the first N samples from the k th epoch because we require the state distribution emanating from the remaining samples to be close enough to the stationary distribution d π θ k , which is the key to make ˆ J c ( θ k ) close to J c ( θ k ) . The following lemma demonstrates that ˆ A π θ k L ,λ k ( s, a ) is a good estimator of A π θ k L ,λ k ( s, a ) .

Algorithm 2 Advantage Estimation

1: Input: Trajectory ( s t 1 , a t 1 , . . . , s t 2 , a t 2 ) , state s , action a , Lagrange multiplier λ , and parameter θ

2: Initialize: M ← 0 , τ ← t 1

3:

mix

N

= 4

t

log

Define:

4: while τ ≤ t 2 -N do

5:

τ

s

=

s

if then

6:

7:

8:

9:

10:

11:

M

g

M

τ

M

+1

else

τ

end if

12:

end while

13: if M > 0 then

14:

Compute

Q

g

15:

16:

17:

18:

∑

τ

τ

,

τ

+

t

+2

N

=

N

τ

+1

.

s, a

ˆ

(

,

else

V

g

(

s

ˆ

end if return

.

)

,

(

M

τ

-

1

2

.

T

g

τ

(

s

t

, a

ˆ

g

V

(

s

t

,

)

g

∀

∈ {

)

s, a

Q

) = 0

( ˆ

-

via (13),

) = 0

ˆ

V

r

(

s

,

r, c

∀

g

∀

g

∈ {

}

∈ {

}

r, c

)) +

λ

( ˆ

Q

c

(

s, a

)

-

r, c

ˆ

V

c

(

}

s

))

Q

r

(

ˆ

g

s, a

)

Lemma 2. The following inequality holds ∀ k , ∀ ( s, a ) and sufficiently large T .

Lemma 2 shows that the L 2 error of our proposed advantage estimator can be bounded above as ˜ O ( T -ξ ) . We later utilize the above result to prove the goodness of the gradient estimator. It is to be clarified that our Algorithm 2 is inspired by Algorithm 2 of [17]. However, while the authors of [17] choose H = ˜ O (1) , we adapt H = ˜ O ( T ξ ) . This subtle change is important in proving a sublinear regret for general parametrization.

4 Global Convergence Analysis

This section first shows that the sequence { θ k , λ k } K k =1 produced by Algorithm 1 is such that their associated Lagrange sequence { J L ( θ k , λ k ) } ∞ k =1 converges globally. By expanding the Lagrange function, we then exhibit convergence of each of its components { J g ( θ k , λ k ) } K k =1 , g ∈ { r, c } . This is later used for regret and constraint violation analysis. Before delving into the details, we would like to state a few necessary assumptions.

Assumption 3. The score function (stated below) is G -Lipschitz and B -smooth. Specifically, ∀ θ, θ 1 , θ 2 ∈ Θ , and ∀ ( s, a ) , the following inequalities hold.

Remark 1 . The Lipschitz and smoothness properties of the score function are commonly assumed for policy gradient analyses [27, 28, 29]. These assumptions hold for simple parameterization classes such as Gaussian policies.

Note that by combining Assumption 3 with Lemma 2 and using the gradient estimator as given in (15) , one can deduce the following result.

Lemma 3. The following inequality holds ∀ k provided that assumptions 1 and 3 are true.

Lemma 3 claims that the gradient estimation error can be bounded as ˜ O ( T -ξ ) . We will use this result later to prove the global convergence of our algorithm.

Assumption 4. Let the transferred compatible function approximation error be defined as follows.

where π ∗ is the optimal solution of unparameterized problem in (2) and

We assume that L d π ∗ ,π ∗ ( ω ∗ θ,λ , θ, λ ) ≤ ϵ bias , λ ∈ [0 , 2 δ ] and θ ∈ Θ where ϵ bias is a positive constant.

Remark 2 . The transferred compatible function approximation error quantifies the expressivity of the parameterized policy class. We can show that ϵ bias = 0 for softmax parameterization [10] and linear MDPs [30]. If the policy class is restricted, i.e., it does not contain all stochastic policies, ϵ bias turns out to be strictly positive. However, if the policy class is parameterized by a rich neural network, then ϵ bias can be assumed to be negligibly small [31]. Such assumptions are common [29, 10].

Remark 3 . Note that ω ∗ θ,λ defined in (19) can be written as,

where † is the Moore-Penrose pseudoinverse and F ρ ( θ ) is the Fisher information matrix defined as,

Assumption 5. There exists a constant µ F > 0 such that F ρ ( θ ) -µ F I d is positive semidefinite where I d is an identity matrix of dimension, d .

Assumption 5 is also called Fisher-non-degenerate policy assumption and is quite common in the literature [29, 32, 33] in the policy gradient analysis. [29][Assumption 2.1] provided a detailed discussion on the requirement of policy class to satisfy the assumption 5. Moreover, [34] describes a class of policies that obeys assumptions 3 -5 simultaneously. The Lagrange difference lemma stated below is important in establishing global convergence.

Lemma 4. With a slight abuse of notation, let J L ( π, λ ) = J π r + λJ π c . For any two policies π , π ′ , the following result holds ∀ λ > 0 .

We now present a general framework for the convergence analysis of Algorithm 1.

Lemma 5. If the policy parameters, { θ k , λ k } K k =1 are updated via (12) and assumptions 3, 4,and 5 hold, then we have the following inequality for any K ,

where ω ∗ k := ω ∗ θ k ,λ k , ω ∗ θ k ,λ k is defined in (19) , and π ∗ is the optimal solution to the problem (2) .

Lemma 5 proves that the optimality error of the Lagrange sequence can be bounded by the average first-order and second-order norms of the intermediate gradients. Note the presence of ϵ bias in the result. If the policy class is severely restricted, the optimality bound loses its importance. Consider the expectation of the second term in (20). Note that,

where ( a ) follows from Assumption 5. The expectation of the third term in (20) can be bounded as

In both (4) and (20), E ∥ ω k -∇ θ J L ( θ k , λ k ) ∥ 2 is bounded above by Lemma 3. To bound the term, E ∥∇ θ J L ( θ k , λ k ) ∥ 2 , the following lemma is applied.

Lemma 6. Let J g ( · ) be L -smooth, ∀ g ∈ { r, c } and α = 1 4 L (1+ 2 δ ) . Then the following holds.

Note the presence of β in (20). To ensure convergence, β must be a function of T . Invoking Lemma 3, we get the following relation under the same set of assumptions and the choice of parameters as in Lemma 6.

Applying Lemma 3 and (21) in (20), we arrive at,

Similarly, using (4) , we deduce the following.

Inequalities (22) and (23) lead to the following global convergence of the Lagrange function.

Lemma 7. Let { θ k } K k =1 be as described in Lemma 5. If assumptions 1 -5 hold, { J g ( · ) } g ∈{ r,c } are L -smooth functions, α = 1 4 L (1+ 2 δ ) , K = T H , and H = 16 t mix t hit T ξ (log 2 T ) 2 , then

Lemma 7 establishes that the average difference between J L ( π ∗ , λ k ) and J L ( θ k , λ k ) is ˜ O ( √ β + T -ξ/ 2 + T -(1 -ξ ) / 2 ) . Expanding the function, J L , and utilizing the update rule of the Lagrange multiplier, we achieve the global convergence for the objective and the constraint in Theorem 1 (stated below). In its proof, Lemma 18 (stated in the appendix) serves as an important tool in disentangling the convergence rates of regret and constraint violation. Interestingly, Lemma 18 is built upon the strong duality property of the unparameterized optimization (2) and has no apparent direct connection with the parameterized setup.

Theorem 1. Consider the same parameters as in Lemma 7 and set β = T -2 / 5 , ξ = 2 / 5 . We have,

where π ∗ is a solution to (2) . In the above bounds, we write only the dominating terms of T .

Theorem 1 establishes ˜ O ( T -1 / 5 ) convergence rates for both the objective and the constraint violation.

5 Regret and Violation Analysis

In this section, we use the convergence result of the previous section to bound the expected regret and constraint violation of Algorithm 1. Note that the regret and constraint violation decompose as,

where I k ≜ { ( k -1) H, · · · , kH -1 } . Observe that the expectation of the first terms in regret and violation can be bounded by Theorem 1. The expectation of the second term in regret and violation can be expanded as follows,

where g ∈ { r, c } . Equality ( a ) uses the Bellman equation and ( b ) follows from the definition of Q g . The first term in the last line of Eq. (24) can be upper bounded by Lemma 8 (stated below). On the other hand, the second term can be upper bounded as O ( t mix ) using Lemma 9.

Lemma 8. If assumptions 1 and 3 hold, then for K = T H where H = 16 t mix t hit T 2 5 (log 2 T ) 2 , the following inequalities hold ∀ k , ∀ ( s, a ) and sufficiently large T :

where g ∈ { r, c } , and { s k } K k =1 is an arbitrary sequence of states.

Lemma 8 states that the obtained policy parameters are such that the average consecutive difference in the sequence { J g ( θ k ) } K k =1 , g ∈ { r, c } decreases with time horizon, T . We would like to emphasize that Lemma 8 works for both reward and constraint functions. Hence, we can prove our regret guarantee and constraint violation as shown below.

Theorem 2. If assumptions 1 -5 hold, J g ( · ) 's are L -smooth, ∀ g ∈ { r, c } and T are sufficiently large, then our proposed Algorithm 1 achieves the following expected regret and constraint violation bounds with learning rates α = 1 4 L (1+ 2 δ ) and β = T -2 / 5 .

The detailed expressions of these bounds are provided in the Appendix. Here, we keep only those terms that emphasize the order of T . Note that our result outperforms the state-of-the-art model-free tabular result in average-reward CMDP [8]. However, our regret bound is worse than that achievable in average reward unconstrained MDP with general parameterization [13]. Interestingly, the gap between the convergence results of constrained and unconstrained setups is a common observation across the literature. For example, in the tabular model-free average reward MDP, the state-of-the-art regret bound for unconstrained setup, ˜ O ( T 1 / 2 ) [17], is better than that in the constrained setup, ˜ O ( T 5 / 6 ) [8].

6 Conclusions

This paper establishes the first sublinear regret and constraint violation bounds in the average reward CMDP setup with general parametrization (and do not assume the underlying constrained Markov Decision Process (CMDP) to be tabular or linear). We show that our proposed algorithm achieves ˜ O ( T 4 / 5 ) regret and constraint violation bounds where T is the time horizon. Note that the state of the art in unconstrained counterpart is ˜ O ( T 3 / 4 ) . Closing this gap by designing more efficient algorithms is an open question in the average reward CMDP literature with the general parametrization. Moreover, our current algorithm requires the knowledge of mixing time. Relaxing such assumptions is another important future direction in realistic settings. For further discussions on future work directions, the readers are referred to [35].

References

Liu, C., N. Geng, et al. Cmix: Deep multi-agent reinforcement learning with peak and average constraints. In Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part I 21 , pages 157-173. Springer, 2021.

Al-Abbasi, A. O., A. Ghosh, V. Aggarwal. Deeppool: Distributed model-free algorithm for ridesharing using deep reinforcement learning. IEEE Transactions on Intelligent Transportation Systems , 20(12):4714-4727, 2019.

Ling, L., W. U. Mondal, S. V. Ukkusuri. Cooperating graph neural networks with deep reinforcement learning for vaccine prioritization. arXiv preprint arXiv:2305.05163 , 2023.

Bai, Q., A. S. Bedi, V. Aggarwal. Achieving zero constraint violation for constrained reinforcement learning via conservative natural policy gradient primal-dual algorithm. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 6737-6744. 2023.

Agarwal, M., Q. Bai, V. Aggarwal. Concave utility reinforcement learning with zero-constraint violations. Transactions on Machine Learning Research , 2022.

Chen, L., R. Jain, H. Luo. Learning infinite-horizon average-reward markov decision process with constraints. In International Conference on Machine Learning , pages 3246-3270. PMLR, 2022.

Agarwal, M., Q. Bai, V. Aggarwal. Regret guarantees for model-based reinforcement learning with long-term average constraints. In Uncertainty in Artificial Intelligence , pages 22-31. PMLR, 2022.

Wei, H., X. Liu, L. Ying. A provably-efficient model-free algorithm for infinite-horizon averagereward constrained markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 3868-3876. 2022.

Ghosh, A., X. Zhou, N. Shroff. Achieving sub-linear regret in infinite horizon average reward constrained mdp with linear function approximation. In The Eleventh International Conference on Learning Representations . 2023.

Agarwal, A., S. M. Kakade, J. D. Lee, G. Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. The Journal of Machine Learning Research , 22(1):4431-4506, 2021.

Mondal, W. U., V. Aggarwal. Improved sample complexity analysis of natural policy gradient algorithm with general parameterization for infinite horizon discounted reward markov decision processes. In International Conference on Artificial Intelligence and Statistics (AISTATS) . 2024.

-. Sample-efficient constrained reinforcement learning with general parameterization. arXiv preprint arXiv:2405.10624 , 2024.

Bai, Q., W. U. Mondal, V. Aggarwal. Regret analysis of policy gradient algorithm for infinite horizon average reward markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence . 2024.

Ding, D., K. Zhang, T. Basar, M. Jovanovic. Natural policy gradient primal-dual method for constrained markov decision processes. Advances in Neural Information Processing Systems , 33:8378-8390, 2020.

Agrawal, S., R. Jia. Optimistic posterior sampling for reinforcement learning: worst-case regret bounds. Advances in Neural Information Processing Systems , 30, 2017.

Auer, P., T. Jaksch, R. Ortner. Near-optimal regret bounds for reinforcement learning. Advances in neural information processing systems , 21, 2008.

Wei, C.-Y., M. J. Jahromi, H. Luo, H. Sharma, R. Jain. Model-free reinforcement learning in infinite-horizon average-reward markov decision processes. In International conference on machine learning , pages 10170-10180. PMLR, 2020.

Bai, Q., A. S. Bedi, M. Agarwal, A. Koppel, V. Aggarwal. Achieving zero constraint violation for constrained reinforcement learning via primal-dual approach. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 3682-3689. 2022.

Xu, T., Y. Liang, G. Lan. Crpo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning , pages 11480-11491. PMLR, 2021.

Efroni, Y., S. Mannor, M. Pirotta. Exploration-exploitation in constrained mdps. arXiv preprint arXiv:2003.02189 , 2020.

Qiu, S., X. Wei, Z. Yang, J. Ye, Z. Wang. Upper confidence primal-dual reinforcement learning for cmdp with adversarial loss. Advances in Neural Information Processing Systems , 33:1527715287, 2020.

Germano, J., F. E. Stradi, G. Genalti, M. Castiglioni, A. Marchesi, N. Gatti. A best-ofboth-worlds algorithm for constrained mdps with long-term constraints. arXiv preprint arXiv:2304.14326 , 2023.

Pesquerel, F., O.-A. Maillard. Imed-rl: Regret optimal learning of ergodic markov decision processes. In NeurIPS 2022-Thirty-sixth Conference on Neural Information Processing Systems . 2022.

Gong, H., M. Wang. A duality approach for regret minimization in average-award ergodic markov decision processes. In Learning for Dynamics and Control , pages 862-883. PMLR, 2020.

Sutton, R. S., D. McAllester, S. Singh, Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems , 12, 1999.

Lattimore, T., C. Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Agarwal, A., S. M. Kakade, J. D. Lee, G. Mahajan. Optimality and approximation with policy gradient methods in markov decision processes. In Conference on Learning Theory , pages 64-66. PMLR, 2020.

Zhang, J., C. Ni, C. Szepesvari, M. Wang. On the convergence and sample efficiency of variance-reduced policy gradient method. Advances in Neural Information Processing Systems , 34:2228-2240, 2021.

Liu, Y., K. Zhang, T. Basar, W. Yin. An improved analysis of (variance-reduced) policy gradient and natural policy gradient methods. Advances in Neural Information Processing Systems , 33:7624-7636, 2020.

Jin, C., Z. Yang, Z. Wang, M. I. Jordan. Provably efficient reinforcement learning with linear function approximation. In J. Abernethy, S. Agarwal, eds., Proceedings of Thirty Third Conference on Learning Theory , vol. 125 of Proceedings of Machine Learning Research , pages 2137-2143. PMLR, 2020.

Wang, L., Q. Cai, Z. Yang, Z. Wang. Neural policy gradient methods: Global optimality and rates of convergence. In International Conference on Learning Representations . 2019.

Yuan, R., R. M. Gower, A. Lazaric. A general sample complexity analysis of vanilla policy gradient. In International Conference on Artificial Intelligence and Statistics , pages 3332-3380. PMLR, 2022.

Fatkhullin, I., A. Barakat, A. Kireeva, N. He. Stochastic policy gradient methods: Improved sample complexity for fisher-non-degenerate policies. In International Conference on Machine Learning , pages 9827-9869. PMLR, 2023.

Mondal, W. U., V. Aggarwal, S. V. Ukkusuri. Mean-field control based approximation of multi-agent reinforcement learning in presence of a non-decomposable shared global state. Transactions on Machine Learning Research , 2023.

Aggarwal, V., W. U. Mondal, Q. Bai. Constrained reinforcement learning with average reward objective: Model-based and model-free algorithms. Found. Trends Optim. , 6(4):193-298, 2024.

Dorfman, R., K. Y. Levy. Adapting to mixing time in stochastic optimization with markovian data. In International Conference on Machine Learning , pages 5429-5446. PMLR, 2022.

Ding, D., K. Zhang, J. Duan, T. Ba¸ sar, M. R. Jovanovi´ c. Convergence and sample complexity of natural policy gradient primal-dual methods for constrained mdps. arXiv preprint arXiv:2206.02346 , 2022.

Bai, Q., V. Aggarwal, A. Gattami. Provably sample-efficient model-free algorithm for mdps with peak constraints. Journal of Machine Learning Research , 24(60):1-25, 2023.

A Proofs for Lemmas in Section 3

A.1 Proof of Lemma 1

Since the first step of the proof works in the same way for functions J r and J c , we use the generic notations J g , V g , Q g where g = r, c and derive the following.

where the step (a) is a consequence of ∇ θ log π θ = ∇ π θ π θ and the Bellman equation. Multiplying both sides by d π θ ( s ) , taking a sum over s ∈ S , and rearranging the terms, we obtain the following.

where ( a ) uses the fact that d π θ is a stationary distribution. Note that,

We can, therefore, replace the function Q π θ g in the policy gradient with the advantage function A π θ g ( s, a ) = Q π θ g ( s, a ) -V π θ g ( s ) , ∀ ( s, a ) ∈ S × A . Thus,

The proof is completed using the definitions of J L ,λ and A L ,λ .

A.2 Proof of Lemma 2

Proof. The proof is similar to the proof of [17, Lemma 6]. Consider the k th epoch and assume that π θ k is denoted as π for notational convenience. Let, M be the number of disjoint sub-trajectories of

length N that start with the state s and are at least N distance apart (found by Algorithm 2). Let, g k,i be the sum of rewards or constraint ( g = r, c accordingly) observed in the i th sub-trajectory and τ i denote its starting time. The advantage function estimate is,

Note the following,

where ( a ) follows from the definition of J π g as given in (5) , ( b ) is an application of the definition of V π g given in (7) , and ( c ) follows from the Bellman equation. Define the following quantity.

Using Lemma 10, we get δ π ( s, T ) ≤ 1 T 3 which implies, | E π T ( s, a ) | ≤ 1 T 3 . Observe that,

where ∆ π T ( s, a ) ≜ E T ( s, a ) -∑ a ′ π ( a ′ | s )E π T ( s, a ′ ) . Using the bound on E π T ( s, a ) , we derive, | ∆ π T ( s, a ) | ≤ 2 T 3 , which implies,

Note that (35) cannot be directly used to bound the bias of ˆ A π g ( s, a ) . This is because the random variable M is correlated with the variables { g k,i } i M =1 . To decorrelate them, imagine a CMDP where the state distribution resets to the stationary distribution, d π after exactly N time steps since the completion of a sub-trajectory. In other words, if a sub-trajectory starts at τ i , and ends at τ i + N , then the system 'rests' for additional N steps before rejuvenating with the state distribution, d π at τ i +2 N . Clearly, the wait time between the reset after the ( i -1) th sub-trajectory and the start of the i th sub-trajectory is, w i = τ i -( τ i -1 +2 N ) , i > 1 . Let w 1 be the difference between the start time of the k th epoch and the start time of the first sub-trajectory. Note that,

( a ) w 1 only depends on the initial state, s ( k -1) H and the induced transition function, P π ,

( b ) w i , where i > 1 , depends on the stationary distribution, d π , and the induced transition function, P π ,

( c ) M only depends on { w 1 , w 2 , · · · } as other segments of the epoch have fixed length, 2 N .

Clearly, in this imaginary CMDP, the sequence, { w 1 , w 2 , · · · } , and hence, M is independent of { g k, 1 , g k, 2 , · · · } . Let, E ′ denote the expectation operation and Pr ′ denote the probability of events in this imaginary system. Define the following.

where ∆ π T ( s, a ) is given in (34) . Note that we have suppressed the dependence on T , s, a , and π while defining ∆ i to remove clutter. Using (34) , one can write E ′ [∆ i ( s, a ) |{ w i } ] = 0 . Moreover,

where ( a ) uses the bound | ∆ π T ( s, a ) | ≤ 2 T 3 derived in (35) , and the fact that { ∆ i } are zero mean independent random variables conditioned on { w i } . Note that | g k,i | ≤ N almost surely, | A π g ( s, a ) | ≤ O ( t mix ) via Lemma 9, and | ∆ π T ( s, a ) | ≤ 2 T 3 as shown in (35) . Combining, we get, E ′ [ | ∆ i | 2 ∣ ∣ { w i } ] ≤ O ( N 2 /π ( a | s )) (see the definition of ∆ i in (36)). Invoking this bound into (37) , we get the following result.

Note that, one can use Lemma 11 to bound the following violation probability.

where ( a ) is a consequence of the fact that 4 t hit T ξ (log 2 T ) -1 ≥ 4 d π ( s ) log 2 T for sufficiently large T . Finally, note that, if M < M 0 , where M 0 is defined as,

then there exists at least one w i that exceeds 4 N log 2 T/d π ( s ) which can happen with the following maximum probability according to Lemma 11.

The above probability bound can be used to obtain the following result,

Injecting (39) and (42) into (38) , we finally obtain the following.

Eq. (43) demonstrates that our desired inequality is obeyed in the imaginary system. We now need a mechanism to translate this result to our actual CMDP. Note that ( ˆ A π g ( s, a ) -A π g ( s, a )) 2 = f ( X ) where X = ( M,τ 1 , T 1 , · · · , τ M , T M ) , and T i = ( a τ i , s τ i +1 , a τ i +1 , · · · , s τ i + N , a τ i + N ) . We have,

The last inequality uses the non-negativity of f ( · ) . Observe that, for a fixed sequence, X , we have, Pr( X ) = Pr( τ 1 ) × Pr( T 1 | τ 1 ) × Pr( τ 2 | τ 1 , T 1 ) × Pr( T 2 | τ 2 ) ×···

̸

̸

̸

The difference between Pr( X ) and Pr ′ ( X ) arises because Pr( τ i +1 | τ i , T i ) = Pr ′ ( τ i +1 | τ i , T i ) , ∀ i ∈ { 1 , · · · , M -1 } . Note that the ratio of these two terms can be bounded as follows,

̸

̸

̸

̸

where ( a ) is a consequence of Lemma 10. We have,

where ( a ) uses the fact that M ≤ T . Combining (44) and (48) , we get,

where ( a ) follows from (43) . Using the definition of A L ,λ , we get,

This concludes the proof.

B Proofs for the Section of Global Convergence Analysis

B.1 Proof of Lemma 3

Proof. Recall from Eq. (15) that,

Define the following quantity,

where t k = ( k -1) H is the starting time of the k th epoch. Note that the true gradient is given by,

Using Assumption 3, Lemma 9, and λ ∈ [0 , 2 δ ] , one can exhibit that | A π θ k L ,λ ( s, a ) ∇ θ log π θ ( a | s ) | ≤ O ( t mix G δ ) , ∀ ( s, a ) ∈ S × A which implies |∇ θ J L ,λ ( θ k ) | ≤ O ( t mix G δ ) . Applying Lemma 14, one, therefore, arrives at

Finally, the difference, E ∥ ω k -¯ ω k ∥ 2 can be bounded as follows.

where ( a ) follows from Assumption 3 and Jensen's inequality whereas ( b ) follows from Lemma 2. Combining, (53) and (54) , we conclude the result.

B.2 Proof of Lemma 4

Proof. Using the Lemma 12, it is obvious to see that

We conclude the lemma using the definition of J L ,λ and A L ,λ .

B.3 Proof of Lemma 5

Proof. We start with the definition of KL divergence.

where the step (a) holds by Assumption 3 and step (b) holds by Lemma 4. Step (c) uses the convexity of the function f ( x ) = x 2 . Finally, step (d) comes from the Assumption 4. Rearranging items, we have

Summing from k = 1 to K , using the non-negativity of KL divergence and dividing the resulting expression by K , we get the desired result.

B.4 Proof of Lemma 6

Proof. By the L -smooth property of the objective function and constraint function, we know that J L ( · , λ ) is a L (1 + λ ) -smooth function. Thus,

where step (a) follows from the fact that θ k +1 = θ k + αω k and inequality (b) holds due to the Cauchy-Schwarz inequality. Now, adding J L ( θ k +1 , λ k +1 ) on both sides, we have

where (a) holds by the definition of J L ( θ, λ ) and step (b) is true because | J c ( θ ) | ≤ 1 , ∀ θ and | λ k +1 -λ k | ≤ β | ˆ J c ( θ k ) | ≤ β where the last inequality uses the fact that | ˆ J c ( θ k ) | ≤ 1 . Summing over k ∈ { 1 , · · · , K } , we have,

which leads to the following.

Rearranging the terms and using 0 ≤ λ k ≤ 2 δ due to the dual update, we arrive at the following.

Choosing α = 1 4 L (1+ 2 δ ) and dividing both sides by K , we conclude the result.

Recall that | J L ( θ, λ ) | ≤ 1 + λ ≤ 1 + 2 δ ≤ 3 δ , ∀ θ ∈ Θ , ∀ λ ≥ 0 . Thus,

This completes the proof.

B.5 Proof of Theorem 1

B.5.1 Rate of Convergence of the Objective

Recall the definition of J L ( θ, λ ) = J r ( θ ) + λJ c ( θ ) . Using Lemma 7, we arrive at the following.

Thus, we need to find a bound for the last term in the above equation.

where (a) uses λ 1 = 0 and inequality (b) holds because θ ∗ is a feasible solution to the constrained optimization problem. Rearranging items and taking the expectation, we have,

where (a) results from | ˆ J c,ρ ( θ ) | 2 ≤ 1 , ∀ θ ∈ Θ and (b) uses the fact that ˆ J c,ρ ( θ k ) and λ k are conditionally independent given θ k . Finally, (c) is a consequence of Lemma 13. Combining (67) with (65), we deduce,

The last inequality presents only the dominant terms of β and T .

B.5.2 Rate of Constraint Violation

Since { λ k } K k =1 are derived by applying the dual update in Algorithm 1, we have,

where ( a ) is due to the non-expansiveness of the projection P [0 , 2 δ ] and ( b ) holds because ˆ J c ( θ ) ∈ [0 , 1] , ∀ θ ∈ Θ according to its definition in Algorithm 1. Finally, ( c ) is a consequence of the fact that ˆ J c ( θ k )

and λ k are conditionally independent given θ k whereas ( d ) applies | λ k -2 δ | ≤ 2 δ and Lemma 13. Averaging (69) over k ∈ { 1 , . . . , K } , we get,

where (a) uses λ 1 = 0 . Note that λ k J π ∗ c ≥ 0 , ∀ k . Adding the above inequality to (65) at both sides, we, therefore, have,

Since the functions { J g ( θ k ) } , k ∈ { 0 , · · · , K -1 } , g ∈ { r, c } are linear in occupancy measure, there exists a policy ¯ π such that the following holds ∀ g ∈ { r, c } .

Injecting the above relation to (71), we have

The last inequality presents only the dominant terms of β and T .

B.5.3 Optimal Choice of β and ξ

If we choose β = T -η for some η ∈ (0 , 1) , then following (68) and (74), we can write,

Clearly, the optimal values of η and ξ can be obtained by solving the following optimization.

One can easily verify that ( ξ, η ) = (2 / 5 , 2 / 5) is the solution of the above optimization. Therefore, the convergence rate of the objective function can be written as follows.

The last expression only considers the dominant terms of T . Similarly, the constraint violation rate can be computed as,

where the last expression contains only the dominant terms of T . This concludes the theorem.

C Proofs for the Regret and Violation Analysis

C.1 Proof of Lemma 8

Proof. Using Taylor's expansion, we can write the following ∀ ( s, a ) ∈ S × A , ∀ k .

where ¯ θ k is some convex combination 1 of θ k and θ k +1 and ( a ) results from Assumption 3. This concludes the first statement. Applying (80) and Lemma 12, we obtain the following for g ∈ { r, c } .

Inequality ( a ) uses Lemma 9 and the update rule θ k +1 = θ k + αω k . Step ( b ) holds by the Cauchy inequality and Jensen inequality whereas ( c ) can be derived using (22) and substituting K = T/H . This establishes the second statement. Next, recall from (5) that for any policy π θ , g π θ ( s ) ≜ ∑ a π θ ( a | s ) g ( s, a ) . Note that, for any policy parameter θ , and any state s ∈ S , the following holds.

Define the following quantity.

Lemma 10 states that for sufficiently large T , we have δ π θ ( s, T ) ≤ 1 T 3 for any policy π θ and state s . Combining this result with the fact that the g π θ function is absolutely bounded in [0 , 1] , we obtain,

where ( a ) follows from (81) and substituting N = 4 t mix (log 2 T ) . For the first term, note that,

Inequality ( a ) holds since every row of P π θ k sums to 1 and ∥ ( P π θ k ) t -1 g π θ k +1 ∥ ∞ ≤ 1 . Moreover, invoking (80), and the parameter update rule θ k +1 = θ k + αω k , we get,

Plugging the above result into (85) and using a recursive argument, we get,

Finally, we have

where ( a ) follows from (22). Moreover, notice that,

where ( a ) follows from (80) and the update rule θ k +1 = θ k + αω k whereas ( b ) is a consequence of (22). Combining (84), (86), and (87), we establish the third statement.

C.2 Proof of Theorem 2

Proof. Recall the decomposition of the regret in section 5 and take the expectation.

Using the result in (78), Lemma 8 and Lemma 9, we get,

Similarly, for the constraint violation, we have

Using the result in (79), Lemma 8 and Lemma 9, we get,

This concludes the theorem.

D Some Auxiliary Lemmas for the Proofs

Lemma 9. [17, Lemma 14] For any ergodic MDP with mixing time t mix , the following holds ∀ ( s, a ) ∈ S × A , any policy π and ∀ g ∈ { r, c } .

Lemma 10. [17, Corollary 13.2] Let δ π ( · , T ) be defined as written below for an arbitrary policy π .

If t mix < T/ 4 , we have the following inequality ∀ s ∈ S : δ π ( s, T ) ≤ 1 T 3 .

Lemma 11. [17, Lemma 16] Let I = { t 1 +1 , t 1 +2 , · · · , t 2 } be a certain period of an epoch k of Algorithm 2 with length N . Then for any s , the probability that the algorithm never visits s in I is upper bounded by

Lemma 12. [17, Lemma 15] The difference of the values of the function J g , g ∈ { r, c } at policies π and π ′ , is

Lemma 13. [6, Lemma 7] The term ˆ J c ( θ ) for any θ ∈ Θ is a good estimator of J c ( θ ) , which means

Lemma 14. [36, Lemma A.6] Let θ ∈ Θ be a policy parameter. Fix a trajectory z = { ( s t , a t , r t , s t +1 ) } t ∈ N generated by following the policy π θ starting from some initial state s 0 ∼ ρ . Let, ∇ L ( θ ) be the gradient that we wish to estimate over z , and l ( θ, · ) is a function such that E z ∼ d π θ ,π θ l ( θ, z ) = ∇ L ( θ ) . Assume that ∥ l ( θ, z ) ∥ , ∥∇ L ( θ ) ∥ ≤ G L , ∀ θ ∈ Θ , ∀ z ∈ S × A × R ×S . Define l Q = 1 Q ∑ Q i =1 l ( θ, z i ) . If P = 2 t mix log T , then the following holds as long as Q ≤ T ,

[

Q

2

]

(

2

L

E

∥

l

-∇

L

(

θ

)

∥

≤ O

G

log (

PQ

)

P

Q

(96)

Lemma 15 (Strong duality) . [37, Lemma 3] For convenience, we rewrite the unparameterized problem (2) .

Define π ∗ as the optimal solution to the above problem. Define the associated dual function as

and denote λ ∗ = arg min λ ≥ 0 J λ D . We have the following strong duality property for the unparameterized problem whenever Assumption 2 holds.

Although the strong duality holds for the unparameterized problem, the same is not true for parameterized class { π θ | θ ∈ Θ } . To formalize this statement, define the dual function associated with the parameterized problem as follows.

̸

and denote λ ∗ Θ = arg min λ ≥ 0 J λ D, Θ . The lack of strong duality states that, in general, J λ ∗ Θ D, Θ = J r ( θ ∗ ) where θ ∗ is a solution of the parameterized constrained optimization (3). However, the parameter λ ∗ Θ , as we demonstrate below, must obey some restrictions.

Lemma16. Under Assumption 2, the optimal dual variable for the parameterized problem is bounded as

Proof. The proof follows the approach in [37, Lemma 3], but is revised to the general parameterization setup. Let Λ a ≜ { λ ≥ 0 | J λ D, Θ ≤ a } be a sublevel set of the dual function for a ∈ R . If Λ a is non-empty, then for any λ ∈ Λ a ,

where ¯ θ is a Slater point in Assumption 2. Thus, λ ≤ ( a -J r ( ¯ θ )) /δ . If we take a = J λ ∗ Θ D, Θ ≤ J λ ∗ D, Θ ≤ J λ ∗ D = J π ∗ r , then we have λ ∗ Θ ∈ Λ a , which proves the Lemma. The last inequality holds since J π r ∈ [0 , 1] for any policy, π .

)

Since the above inequality holds for arbitrary Θ , we also have, 0 ≤ λ ∗ ≤ 1 δ . Define v ( τ ) ≜ max π ∈ Π { J π r | J π c ≥ τ } . Using the strong duality property of the unparameterized problem (97), we establish the following property of the function, v ( · ) .

Lemma 17. Assume that the Assumption 2 holds, we have for any τ ∈ R ,

Proof. By the definition of v ( τ ) , we have v (0) = J π ∗ r . With a slight abuse of notation, denote J L ( π, λ ) = J π r + λJ π c . By the strong duality stated in Lemma 15, we have the following for any π ∈ Π .

Thus, for any π ∈ { π ∈ Π | J π c ≥ τ } ,

Maximizing the right-hand side of this inequality over { π ∈ Π | J π c ≥ τ } yields

This completes the proof of the lemma.

We note that a similar result was shown in [38, Lemma 15]. However, the setup of the stated paper is different from that of ours. Specifically, [38] considers a tabular setup with peak constraints. Note that Lemma 17 has no direct connection with the parameterized setup since its proof uses strong duality and the function, v ( · ) , is defined via a constrained optimization over the entire policy set, Π , rather than the parameterized policy set. Interestingly, however, the relationship between v ( τ ) and v (0) leads to the lemma stated below which turns out to be pivotal in establishing regret and constraint violation bounds in the parameterized setup.

Lemma 18. Let Assumption 2 hold. For any constant C ≥ 2 λ ∗ , if there exists a π ∈ Π and ζ > 0 such that J π ∗ r -J π r + C [ -J π c ] ≤ ζ , then

Proof. Let τ = J π c . Using the definition of v ( τ ) , one can write,

Combining Eq. (106) and (108), we obtain the following.

The condition in the Lemma leads to,

Finally, we have,

which completes the proof.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The contribution and challenges are clearly described at the end of the introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We add all the assumptions in the work, list the gap with lower bound in Table 1, and give future work direction.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the assumptions are clearly stated with a remark to discuss. All the proof are given in the appendix in details.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: the paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: the paper does not include experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: the paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: the paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: the paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: the research conducted in the paper satisfies the NeurIPS Code of Ethics

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: there is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: the paper poses no such risks

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: the paper does not use existing assets

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: the paper does not use release new assets

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores RL in Infinite-horizon Ergodic Average-reward constrained \nMDPs under general policy parametrization. In this setting, the authors propose \na primal-dual based policy gradient algorithm that simultaneously optimizes the \npolicy parameter and a constraint violation penalty term $\\lambda$. They also \nequip their method with techniques to address challenges due to the \naverage-reward MDP setup, constraints on the MDP and general policy \nparametrization. They prove global convergence of their resulting method in \nterms of the Lagrange function and from this derive bounds on the expected \nregret as well as constraint violation. Furthermore, when the policy class is expressive enough to approximately contain an optimal policy of the constrained MDP so that the un-expressivity parameter $\\varepsilon_{bias}$ is zero or negligibly small, the authors prove that the \nexpected average regret and constraint violation of their method decreases at a rate of $O(T^{-1/5})$.\n\nSTRENGTHS:\n1. The authors consider the task of reinforcement learning \n\tin average-reward constrained MDPs and propose novel techniques to address \n\tunderlying challenges.\n2. This is a complete piece of work that explores provably \n\tefficient RL in average reward constrained MDPs. Their claims are backed by \n\ttheoretical analysis, and they highlight the strength as well as weaknesses \n\tof their work.\n3. This submission is clearly written. The authors highlight \n\tthe required assumptions and adequately discuss the nature of most relevant \n\tparameters.\n4. This work significantly contributes to existing theory \n\ton reinforcement learning when it is more desirable to optimize the average \n\treturn, rather than the typical discounted return, and in addition to \n\treturn optimization, the policy is required to adequately adhere to \n\tadditional constraints on the MDP.\n\nWEAKNESSES:\nSee questions.\n\nQUESTIONS:\n1. In Lines 243-246, the authors claim that there are scenarios under \n\twhich the un-expressivity parameter $\\varepsilon_{bias}$ is zero or \n\tnegligible. I could reason about this claim when $\\pi^{\\*}$ is an optimal \n\tpolicy in the unconstrained MDP, but unsure that the same holds when \n\t$\\pi^{*}$ is an optimal policy in the constrained MDP (see Line 240). Can \n\tthe authors kindly provide exact pointers in the referenced \n\tpapers that verify this claim, or a proof in the appendix?\n\n2. In Lines 323 and 324, the authors claim that the primal-dual approach to policy optimization\n\tis known to give worse results than in the unconstrained version, even for \n\tthe discounted setting. Could this be related to the fact that the method is policy, rather than value based? In the offline setting, [1] takes a value-based primal-dual approach and their result implicitly highlights that the constrained setting may not be worse off than the unconstrained setting.\n\n3. Can the authors kindly elaborate on the bias of single \n\ttrajectory-based estimations in this setting? Reference [13] in the paper seems to \n\tachieve better performance even with this biased estimation strategy.\n\n4.  How strong is Assumption 5?\n\n[1] Hong, K., \\& Tewari, A. (2024). A Primal-Dual Algorithm for Offline \nConstrained Reinforcement Learning with Low-Rank MDPs. arXiv preprint \narXiv:2402.04493.\n\n**Comments**\n1. Line 265 makes reference to equation 21 twice rather than the \n\texpressions after lines 263 and 264.\n2. Expectation is sometimes expressed without the parenthesis. For \n\texample, see Equations 20, 22, as well as Lines 265, 266 e.t.c.\n\nLIMITATIONS:\nNone. This is a purely theory paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the infinite horizon average reward Constrained MDP setting and proposes an analysis of regret and constraint violation under a general policy parametrization. They devise a primal-dual policy gradient algorithm achieving global optimality while ensuring sublinear bounds on the regret and constraint violation.\n\nSTRENGTHS:\nThe paper is interesting and well-written and places itself well in the related literature by filling the gap about infinite horizon average reward constraint MDPs under general parametrized policies. \nThey employ similar techniques that can be found in the related unconstrained setting but also highlight and overcome the challenges encountered while facing the constrained and average reward setting. In particular, they take inspiration from already existing techniques developed for the discounted setting and adapt them to cope with the average reward MDP.\n\nWEAKNESSES:\nFrom the theoretical perspective, I do not recognize specific weaknesses. \n\nHowever, I would like to highlight that the authors assume the knowledge of the mixing and hitting time of the MDP. I recognize that this is a standard assumption in this type of work, however, I wonder if they are necessary to achieve these results or if the authors are aware of similar techniques that can provide sublinear guarantees without this knowledge. Recently, the work of [1] highlighted some techniques that allow achieving global convergence results without this knowledge. Would it be possible to apply these techniques to this setting as well?\n\nFrom the simulation perspective, I believe that some experimental results showing the validity of the presented approach may be beneficial for the presented work.\n\n\n\n[1] Patel B. et al., Towards Global Optimality for Practical Average Reward Reinforcement Learning without Mixing Time Oracles, ICML 2024.\n\nQUESTIONS:\nSee the weakness section.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper studies learning in constrained MDPs with the long-run average-reward objective. 
It gives the first sample complexity, regret bound and constraint violation bound in this setting with general parameterization, whereas all prior work is restricted to either tabular or linear parameterizations.

STRENGTHS:
The result and analysis look solid. The presentation is also clear.

WEAKNESSES:
1 
The assumption that all policies induce aperiodic irreducible Markov chain is a bit strong. To what extend can you weaken the assumptions such that the current analysis in the paper still go through? Is it enough if we only assume the Markov chain is weakly-communicating?

2 
Claiming that the regret is O(T^{4/5}) in table 1 is a bit misleading, because in the regret and constraint violation bounds in Theorem 2, there are actually terms linear in T whose coefficients depend on the transfer error \epsilon_{bias}. 
I understand that the linear-in-T error terms due to the transfer error are common in the analysis of RL with general parameterizations, and the transfer error is zero when the parameterization is tabular.  
However, it is still worth discussing whether the coefficients of the linear-in-T terms in Theorem 2 are tight and optimal, how they compare to the prior work, and whether they are truly negligible when the policy is parameterized by a neural net of a reasonable scale. 
At the very least, how the regret and constraint violation are actually calculated should be clarified in a prominent part of the paper.

QUESTIONS:
See my questions in the weaknesses section.

LIMITATIONS:
There is no negative social impact. There are some discussion of limitation in the conclusion section, but not sufficient.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose a primal dual policy gradient method for solving average reward constrained MDPs. The primal problem is minimizing the usual RL objective with a averaged reward, plus a penalty induced constraint violation term. The dual problem is to find appropriate Lagrangian multiplier that balances the two objectives to reach equilibrium. The proposed method has reached the claimed regret bound and constraint violation bounds with general policy parameterization.\n\nSTRENGTHS:\nThe paper is well written and components are clearly explained. The presentation is good and easy to understand. The investigated topic is interesting and important for real life applications, and average reward is harder to study in general w.r.t to its discounted counterparts due to the absence of contraction of the Bellman operator under the infinity norm. \nThe analysis is easy to follow and plainly laid out. The basic flow of thought is easy to follow and nicely connected. The analysis itself is solid  in terms of its assumptions made in the paper.\n\nWEAKNESSES:\nThere are a few points that made this paper limited in its technical contribution.\n1. The claimed the general parameterization seems to rely on an accurate policy model such as neural network. In the paper the authors assumes somehow we can obtain an accurate policy for free. In reality, this is far from the truth. In this sense, the claimed contribution is not much of importance.\n2. Also related to the first point, it seems that for policy evaluation (algorithm 2), the values functions are still in tabular form. Additionally, this is closely following the proposed policy evaluation method in [17] referred in the paper, which seems to the hardest part in average-reward MDPs. Existing literature shows that for policy optimization alone, it is not so different from discounted setup. See\n\nLi, Tianjiao, Feiyang Wu, and Guanghui Lan. \"Stochastic first-order methods for average-reward markov decision processes.\" arXiv preprint arXiv:2205.05800 (2022).\n\n3. There is no numerical experiments to validate the proposed algorithm. There is a growing important for RL researcher to bridge the gap between theories and practice.\n\nQUESTIONS:\nIt seems uniform ergodicity is crucial in the paper. I wonder one can relax the assumption for the MDP to be unichain.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a primal-dual policy gradient algorithm for infinite horizon average-reward Constrained Markov Decision Processes with general policy parametrization. It uniquely addresses minimizing regret and managing constraints simultaneously, providing the first analysis showing sub-linear bounds of O(T^{4/5}) for both regret and constraint violations. This approach extends the applicability of reinforcement learning to complex, large-scale decision-making problems under practical constraints.\n\nSTRENGTHS:\n- The paper is the first to apply a primal-dual policy gradient framework to this particular setting of CMDPs, addressing a gap in the literature.\n- The problem is well-motivated.\n- The paper offers theoretical sub-linear bounds for both regret and constraint violations.\n\nWEAKNESSES:\n- The assumption of ergodicity is quite strong and may limit the applicability of the proposed algorithm in practical scenarios where such conditions are not met. This contrasts with the Upper Confidence Reinforcement Learning (UCRL) framework, which operates under the more relaxed condition of weakly communicating MDPs.\n- The literature review table omits the MDP modeling assumptions. Specifically, the regret bound offered by the proposed algorithm is O(T^{4/5}), while tabular constrained RL algorithms under more general multichain MDP (communicating/weakly communicating) assumptions in [6] achieve O(T^{2/3}). Related facts were not mentioned in the paper.\n- The mixing time, t_{\\text{mix}}, tends to depend unfavorably on the state space S and action space A, reflecting in the overall diameter of the MDPs.\n\nQUESTIONS:\n- Can you explain in more detail the dependence of the mixing time on the state space  S  and action space  A  is critical, especially in large-scale problems. Clarification on how  t_{\\text{mix}}  scales with  S  and  A  and its impact on the algorithm’s performance would provide a better understanding of the method’s applicability to complex environments.\n- Error propagation for estimated  t_{\\text{mix}} : Information on the numerical performance of the algorithm, particularly how  t_{\\text{mix}}  influences runtime and convergence in practical scenarios, would substantiate the theoretical claims. Experimental results or simulations that highlight these dynamics could significantly enhance the manuscript.\n\nLIMITATIONS:\nThe authors touched upon future work/limitations on theoretical complexity", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}