Title: Achieving $\tilde{O}(1/\epsilon)$ Sample Complexity for Constrained Markov Decision Process
Venue: NeurIPS 2024
Accepted: yes
Keywords: constrained MDP

reinforcement learning

online linear programming
Abstract: We consider the reinforcement learning problem for the constrained Markov decision process (CMDP), which plays a central role in satisfying safety or resource constraints in sequential learning and decision-making. In this problem, we are given finite resources and a MDP with unknown transition probabilities. At each stage, we take an action, collecting a reward and consuming some resources, all assumed to be unknown and need to be learned over time. In this work, we take the first step towards deriving optimal problem-dependent guarantees for the CMDP problems. We derive a logarithmic regret bound, which translates into a $O(\frac{1}{\Delta\cdot\epsilon}\cdot\log^2(1/\epsilon))$ sample complexity bound, with $\Delta$ being a problem-dependent parameter, yet independent of $\epsilon$. Our sample complexity bound improves upon the state-of-art $O(1/\epsilon^2)$ sample complexity for CMDP problems established in the previous literature, in terms of the dependency on $\epsilon$. To achieve this advance, we develop a new framework for analyzing CMDP problems. To be specific, our algorithm operates in the primal space and we resolve the primal LP for the CMDP problem at each period in an online manner, with \textit{adaptive} remaining resource capacities. The key elements of our algorithm are: i) a characterization of the instance hardness via LP basis, ii) an eliminating procedure that identifies one optimal basis of the primal LP, and; iii) a resolving procedure that is adaptive to the remaining resources and sticks to the characterized optimal basis.

Achieving ˜ O (1 /ε ) Sample Complexity for Constrained Markov Decision Process

Abstract

We consider the reinforcement learning problem for the constrained Markov decision process (CMDP), which plays a central role in satisfying safety or resource constraints in sequential learning and decision-making. In this problem, we are given finite resources and a MDP with unknown transition probabilities. At each stage, we take an action, collecting a reward and consuming some resources, all assumed to be unknown and need to be learned over time. In this work, we take the first step towards deriving optimal problem-dependent guarantees for the CMDP problems. We derive a logarithmic regret bound, which translates into a O ( 1 ∆ · ε · log 2 (1 /ε )) sample complexity bound, with ∆ being a problem-dependent parameter, yet independent of ε . Our sample complexity bound improves upon the state-of-art O (1 /ε 2 ) sample complexity for CMDP problems established in the previous literature, in terms of the dependency on ε . To achieve this advance, we develop a new framework for analyzing CMDP problems. To be specific, our algorithm operates in the primal space and we resolve the primal LP for the CMDP problem at each period in an online manner, with adaptive remaining resource capacities. The key elements of our algorithm are: i) a characterization of the instance hardness via LP basis, ii) an eliminating procedure that identifies one optimal basis of the primal LP, and; iii) a resolving procedure that is adaptive to the remaining resources and sticks to the characterized optimal basis.

1 Introduction

Reinforcement learning (RL) is pivotal in the realm of dynamic decision-making under uncertainty, where the objective is to maximize total reward through ongoing interaction with and learning from an enigmatic environment. Markov Decision Processes (MDPs) are a prevalent framework for encapsulating environmental dynamics. MDPs have been instrumental in various domains, such as video gaming [52], robotics [41], recommender systems [57], inventory control [11], and beyond. Yet, they fall short in accommodating additional constraints that may influence the formulation of the optimal policy and the decision-maker's engagement with the uncertain environment. Often, in MDP applications, there are stringent constraints on utilities or costs, emanating from areas like safe

autonomous driving [22], robotics [56], revenue management [33] and financial management [62]. These constraints might also symbolize limitations on resources within resource allocation applications. Constrained MDPs (CMDPs), as introduced in [4], enhance MDPs to factor in constraints affecting long-term policy results. In CMDPs, the decision-maker aims to optimize cumulative rewards while adhering to these constraints. Our paper focuses on CMDPs, and we aim to develop efficient algorithmic solutions.

The significance of RL in CMDP contexts has garnered substantial attention in recent years. A variety of methods for tackling CMDPs have been developed, including the primal-dual technique [21, 49, 12, 69, 18, 45, 25, 26], which leverages the Lagrangian dual of CMDPs and implements an online learning strategy for the iterative update of dual variables. Other methods encompass constrained optimization [1, 61], the Lyapunov technique [14], among others. Previous research has established minimax bounds for CMDPs, representing the optimal regret that can be achieved for the most challenging problem within a specific class of problems. Nonetheless, these minimax regret bounds and worst-case scenarios can be overly conservative, leading to a disconnect between theoretical guarantees and practical performance for specific problem instances. A more tailored approach is warranted-one that ensures great performance on every single problem instance and furnishes problem-dependent guarantees. Our research takes the first step towards deriving optimal problem-dependent guarantees for CMDP problems.

1.1 Preliminaries

We consider a CMDP problem with a finite set of states S = { 1 , 2 , . . . , |S|} and a finite set of actions A = { 1 , 2 , . . . , |A|} . We denote by γ ∈ (0 , 1) a discount factor. We also denote by P : S ×A → D ( S ) the probability transition kernel of the CMDP, where D ( S ) denotes a probability measure over the state space S . Then, P ( s ′ | s, a ) denotes the probability of transiting from state s to state s ′ when the action a is executed. The initial distribution over the states of the CMDP is denoted by µ 1 .

There is a stochaastic reward function r : S × A → D [0 , 1] and K stochastic cost functions c k : S × A → D [0 , 1] for each k ∈ [ K ] . We also denote by ˆ r ( s, a ) = E [ r ( s, a )] for each ( s, a ) and ˆ c ( s, a ) = E [ c k ( s, a )] for each ( s, a ) . For any Markovian policy π , where the action of π depends only on the current state and the action of π is allowed to be randomized, we denote by V r ( π, µ 1 ) the infinite horizon discounted reward of the policy π , with the formulation of V r ( π, µ 1 ) given below:

where ( s t , a t ) is generated according to the policy π and the transition kernel P with the initial state distribution µ 1 . For each k ∈ [ K ] , the infinite horizon discounted cost of the policy π is denoted by V k ( π, µ 1 ) , and the following constraint needs to be satisfied by the policy π ,

To solve the CMDP problem, we aim to find an optimal Markovian policy, denoted by π ∗ , that maximizes the reward in (1) while satisfying the cost constraint (2) for each k ∈ [ K ] , with α k ∈ [ 0 , 1 1 -γ ] being a pre-specified value for each k ∈ [ K ] . Importantly, we assume that the reward function r , the cost functions { c k } K k =1 , and the transition kernel P , are all unknown to the decision maker. Our goal is to obtain a policy π that approximates the optimal policy π ∗ with as few samples as possible. We now describe the sampling procedure and present the performance measure of our policy. We assume the existence of a stylized generative model M , as studied in [39, 38, 44]. The model M satisfies the following condition.

Assumption 1.1 For each state and action pair ( s, a ) , we can query the model M to obtain an observation of r ( s, a ) , c k ( s, a ) for each k ∈ [ K ] , and the new state s ′ ∈ S , where the transition from s to s ′ follows the probability kernel P ( s ′ | s, a ) independently.

Note that in reinforcement learning for CMDP problems, querying the generative model M can be costly. Therefore, it is desirable to query the model M as less as possible, while guaranteeing the

near optimality of the approximate policy. To this end, we use sample complexity as the measure of the performance of our policy. For any ε , we aim to find an ε -accurate policy π such that

with as few samples as possible.

1.2 Our Main Results and Contributions

The main result of our research is the introduction of a novel algorithm that promises a O ( 1 ∆ · ε · log 2 (1 /ε )) sample complexity bound, where ∆ is a positive constant that characterizes the gap between the optimal policy and the sub-optimal ones. Note that the state-of-the-art sample complexity bound under the worst-case scenarios is O (1 /ε 2 ) , which has been established in a series of work [68, 37, 21]. Though the O (1 /ε ) or better iteration complexity has been achieved in [45, 50, 74, 26], it comes with a sample complexity bound no better than O (1 /ε 2 ) . Our algorithm enjoys a sample complexity bound that has a better dependency in terms of ε . To achieve this improved result, we develop several new elements listed below.

Contribution 1 : we develop new characterizations of the problem instance hardness for CMDP problems. Note that a key component for achieving instance-dependent bounds is to characterize the 'hardness' of the underlying problem instance. That is, we need to identify a positive gap to separate the optimal policy from the sub-optimal policies for a particular problem instance. The importance of identifying such a gap has been demonstrated in instance-optimal learning for multi-arm-bandit problems (e.g. [42]) and reinforcement learning problems (e.g. [6]), among others. For CMDP, identifying such a gap is non-trivial because the optimal policies for CMDP are randomized policies [4]. Then, the policies can be represented by distributions over the action set and the sub-optimal policies can be arbitrarily close to the optimal policy as long as the corresponding sub-optimal distributions converge to the optimal one. To tackle this problem, we show that the feasible region for the policies can be represented as a polytope and we only need to focus on the corner points of this polytope to find an optimal policy. Therefore, the hardness can simply be characterized as the distance between the optimal corner point and the sub-optimal corner point, as illustrated in detail in Section 2.1. This is the first characterization of problem instance hardness for CMDP problems.

Contribution 2 : we devise a new algorithmic framework to analyze CMDP problems, inspired by the online packing/linear programming (LP) literature [3, 40, 47, 46]. Specifically, we utilize a linear programming reformulation of the CMDP problem, where policies are delineated via occupancy measures [4]. The optimal policy emerges from the LP's solution; however, the indeterminate model parameters mean the LP cannot be solved directly but must be approached online as we obtain more samples from the generative model. Each generative model query leads to solving an empirical LP with accrued samples, and our final policy is derived from averaging these solutions-an approach akin to the methodology in online LP. A critical feature of our algorithm is the adaptiveness of the LP constraints' right-hand side to the input samples, a technique demonstrated to achieve logarithmic regret in online LP literature, which we now apply to CMDP problems.

Contribution 3 : we extend our contributions to the online LP literature. Note that after adopting the LP reformulation, the corner points of the feasible region for policies can be represented by the basis of the LP. Separating the optimal policy from the others is equivalent to identifying one optimal basis of the LP. We utilize an approach that lexicographically restricts one variable to zero and tests whether the LP value has changed. We show that this approach systematically pinpoints a particular optimal LP basis with a high probability. Then, we develop a resolving procedure that capitalizes on the structure of the identified optimal basis, which involves only the non-zero basic variables and the active constraints. This is a new approach of deriving problem-dependent bound for online LP.

Other related literature. Problem-dependent guarantees have been considered extensively in the RL literature, where a series of work [72, 59, 54, 23, 17, 66, 65, 71, 20] establishes the log( N ) regret bound or the O ( κ · ε -1 ) sample complexity, with κ being a problem-dependent constant. Our approach can also be directly applied to the RL problems. Moreover, our approach can handle long-term constraints and can deal with multi-objective (safe) RL problems.

our work presents a new algorithm for safe RL problems. We adopt an occupancy measure representation of the optimal and obtain an LP to work with, which is similar to the previous work. However, our algorithm resolves an LP and operates in the primal space, which is fundamentally different from the previous work that adopts a primal-dual update (e.g. [60], [19], [73], [8], [53]). There is also

work developing primal-based algorithms, for example ([51], [13], [15], [16], [70]). Our algorithm is completely different from the previous work and we obtain new results. The result is that we are able to obtain an instance-dependent ˜ O (1 /ϵ ) sample complexity the first time in the literature, which improves upon the O (1 /ϵ 2 ) worst-case sample complexity established in the previous work. Though the constrained optimization approach and the Lyapunov approach have also been developed for CMDP problems, they do not enjoy a theoretical guarantee. In comparison to the literature, we develop a new primal-based algorithm and achieve the first instance-dependent sample complexity for CMDP problems.

Our new primal-based algorithm is motivated from the literature of online linear programming [3, 40, 47] and bandits with knapsack problems (e.g. [46], [48]). In these problems, the optimal policy can be written as an LP and we need to develop an online policy to solve this LP sequentially. Note that a prevalent strategy is to resolve the LP adaptive to the remaining resources, which has been developed in a long line of research on various applications, for example [31], [32], [5], [36], [64], [35] and [10]. We make the innovation of resolving the LP while sticking to the identified optimal basis, which distinguishes our algorithm from the previous ones in online LP. Note that the online LP techniques has also been extended to handle non-stationarity, for example in [9], [34]. It is an interesting future topic to explore whether our algorithm can be extended to non-stationary environment.

2 LP Reformulation

The infinite horizon discounted setting described in Section 1.1 admits a linear programming reformulation. To be specific, due to the existence of the constraints, the optimal policy of a CMDP can be randomized policies (e.g. [4]), where it is optimal to take a stochastic action given the current state. Therefore, it is convenient to represent a policy through the occupation measure , which gives us the desired linear programming reformulations of the CMDP problems.

For the infinite horizon discounted problem, the occupancy measure is defined as q π ( s, a ) for any state s , action a , and policy π . Note that q π ( s, a ) represents the total expected discounted time spent on the state-action pair ( s, a ) , under policy π , multiplied by 1 -γ . Then, following [4], the optimal policy (and the optimal occupancy measure) can be obtained from the following linear programming.

where δ s,s ′ = ✶ s = s ′ , and µ 1 ( s ) denotes the probability for the first state to be realized as s ∈ S following the initial distribution µ 1 . Note that for an optimal solution { q ∗ ( s, a ) } ∀ s ∈S , ∀ a ∈A to (4), the corresponding optimal policy π ∗ will be

where π ∗ ( s ) denotes the probability for the policy π ∗ to take the action a ∈ A given the state s ∈ S . In fact, when ∑ a ′ ∈A q ∗ ( s, a ′ ) = 0 , we can take an arbitrary action. In what follows, we rely on the linear programming formulation (4) to derive our results.

2.1 Characterization of Instance Hardness

We now rewrite the LP (4) into the following standard formulation to proceed with our illustration.

Note that one crucial step for achieving problem-dependent bounds is to characterize the hardness of the underlying problem instance and define a gap that separates the optimal policies from the others. For multi-arm-bandit problem, the characterization of the hardness can be the gap between the optimal arm and the best sub-optimal arm (e.g. [42]). For reinforcement learning problem, the characterization of the hardness can also be the gap between the optimal policy and the best sub-optimal policy (e.g. [6]). As long as this separation gap is a positive constant, denoted by ∆ , separating the optimal policy from the others with a probability at least 1 -ϵ would require samples at most 1 ∆ · log(1 /ϵ ) , which finally implies the instance-optimal sample complexity bound.

For CMDP problem, characterizing the hardness of the problem instance can be hard. Based on LP (6), we know that a feasible policy corresponds to a feasible solution and the sub-optimal solution can be arbitrarily close to the optimal solution since the feasible set is 'continuous'. Therefore, there is no direct way to identify a positive gap between the optimal policies and the sub-optimal ones. However, from standard LP theory, we know that one corner point of the feasible region must be one optimal solution. Therefore, we can simply focus on the corner points when solving LP (6) and we define the gap as the distance between the optimal corner point and the sub-optimal corner point , as illustrated in Figure 1. As we will show later, the LP reformulation (6) and such a characterization of hardness via corner points will inspire our entire approach.

Since the problem hardness is characterized via corner points, it is essential to provide further characterization of the corner points. Note that in LP theory, the corner point is called basic solution and can be represented by LP basis , which involves the set of basic variables that are allowed to be non-zero, and the set of active constraints that are binding under the corresponding basic solution. Our next lemma follows from standard LP theory, where the proof is provided in the appendix for completeness.

Lemma 2.1 Denote by b the number of rows in the matrix B . Then, there exists a subset J ∗ ⊂ [ K ] with K ′ = |J ∗ | and an optimal solution q ∗ to LP (6) such that there are b + K ′ variables in q ∗ are non-zero. Moreover, we denote by I ∗ the index set of the non-zero element in q ∗ . Then, the optimal solution q ∗ can be uniquely determined as the solution to the linear system

with I ∗ c being the complementary set of the index set I ∗ .

Remark . Note that the minimax lower bound Ω(1 /ϵ 2 ) has been established in previous work [7, 63]. However, this does not contradict with our ˜ O (1 /ϵ ) sample complexity after we introduce the instance

hardness measure ∆ . To be specific, for a problem instance I , we can denote by S ( I, ϵ ) the number of samples needed to construct an ϵ -optimal policy. Then the worst-case lower bound implies that max I S ( I, ϵ ) = Θ(1 /ϵ 2 ) . However, if we do not consider the worst-case guarantee, i.e., if we do not maximize over the problem instance I , then we can characterize an instance-dependent constant ∆( I ) (independent of ϵ ) such that S ( I, ϵ ) = ∆( I ) /ϵ · polylog (1 /ϵ ) . When the problem instance is favorable such that the constant ∆( I ) is smaller than 1 /ϵ , our bound strictly improves upon the worst-case bound.

Overview of our approach : in the first part, we aim to identify the optimal basis I and J . In this way, we identify the optimal corner point to look at. The detailed procedure is described in Section 3. In the second part, we learn the optimal solution given the optimal basis we have identified, which finally gives us a near-optimal policy with the desired sample complexity bound. The detailed procedure is described in Section 4.

3 Construct Estimates and Identify Optimal Basis

We describe how to construct estimates for LP (6). To this end, for a round N 0 , we denote by F N 0 the filtration of all the information collected up to round N 0 . Then, we denote by ¯ C N 0 (resp. ¯ B N 0 ) an estimate of the matrix C (resp. B ), constructed using the information in the set F N 0 . We also denote by ¯ r N 0 and estimate of ˆ r constructed from the information in the set F N 0 . To be specific, we define

where r n ( s, a ) denotes the n -th observation of the reward, and c n k ( s, a ) denotes the n -th observation of the k -th cost, and s n ( s, a ) denotes the n -th observation of the state transition for the state-action pair ( s, a ) , for n ∈ [ N 0 ] . Then, similar to [21], we can use the following LP to obtain an estimate of V (6).

with λ N 0 being a parameter that we specify later. To bound the estimation gap between ¯ V N 0 and V , it is useful to bound the optimal dual solution to (6). To this end, we adopt the approach in [29, 55] that utilizes Slater's condition, which is imposed as an assumption below.

Assumption 3.1 There exists a policy ¯ π such that all the resource constraints are satisfied strictly. In other words, there exists an occupancy measure ¯ q such that B ¯ q = µ and C ¯ q < α . In fact, for each state s ∈ S , there exists a null action that consumes no resource.

The Slater point ¯ q can be set as the policy that takes the null action given each state. The estimation error will be related to the gap between the Slater point ¯ q and the optimal point q ∗ . We then define the lower gap as and the upper gap as

with both inequalities (10) and (11) hold with probability at least 1 -ε .

3.1 Bound the Estimation Gap

We denote by Rad( N 0 , ε ) = √ log(2 /ε ) 2 N 0 . Following the standard Hoeffding's inequality, we know that | ¯ r N 0 ( s, a ) -ˆ r ( s, a ) | , | ¯ c k,N 0 ( s, a ) -ˆ c k ( s, a ) | , and | ¯ P N 0 ( s ′ | s, a ) -P ( s ′ | s, a ) | are all upper bounded by Rad( N 0 , ε ) with probability at least 1 -ε . We can simply set

and

We have the following result, where the proof is relegated to appendix.

Lemma 3.2 As long as λ = Rad( N , ε ) , the following inequality

holds with probability at least 1 -( K |S||A| - |S| 2 |A| ) · ε , where Gap 1 ( N 0 , ε ) is defined in (12) and Gap 2 ( N 0 , ε ) is defined in (13) .

3.2 Characterize One Optimal Basis

We now describe how to identify one optimal basis of the LP (6) as required in Lemma 2.1, by sequentially discarding the sub-optimal actions and the redundant constraints. The formal algorithm to identify such non-zero elements and the constraints is given in Algorithm 1.

Algorithm 1 Algorithm for identifying one optimal basis

1: Input: the historical sample set F N 0 that contains N 0 samples for each ( s, a ) ∈ S × A .

2: Compute the value of ¯ V N 0 as in (9).

3: Initialize I to be the whole index set that contains every column index of matrix B in (6) and J = [ K ] .

4: for i ∈ I do

5: Let I ′ = I\{ i } .

7: If | ¯ V I ′ ,N 0 -¯ V N 0 | ≤ 2Gap 1 ( N 0 , ε ) + 2Gap 2 ( N 0 , ε ) , then we set I = I ′ .

6: Compute the value of ¯ V I ′ ,N 0 as in (15).

8: end for

9: for k = 1 , . . . , K do

10: Let J ′ = J\{ q } .

11: Compute the value of Dual J ′ , I ,N 0 as in (18).

12: If | ¯ V N 0 -Dual J ′ , I ,N 0 | ≤ 2Gap 1 ( N 0 , ε ) + 2Gap 2 ( N 0 , ε ) , then we set J = J ′ .

13: end for

14: Output : the set of indexes I and J .

We now explain the intuition why Algorithm 1 works. Denote by q ∗ an optimal solution to LP (6). We describe how to identify the non-zero elements in q ∗ and how to identify the constraints such that the values of the non-zero elements of q ∗ can be uniquely determined by the corresponding linear equation. For each i -th element of q ∗ , we compare the value of V (6) against V with an additional constraint that q i = 0 . If the two values are different, we identify a non-zero element. To this end, for an index set I , we define an LP, as well as its estimate, as follows.

where I c denotes the complementary set of I . Note that if V -V I > 0 , we know that I c contains a non-zero basic variable. The steps 4-8 in Algorithm 1 reflect this point. Starting from I denoting the whole index set, we sequentially delete one element i (denoting ( s, a ) in the infinite horizon discounted problem) from the set I . Once we detected that V -V I\{ i } > 0 , we know that i is a non-zero basic variable and we add i back into the set I . In this way, we can classify all the basic variables into the set I . However, since we do not know the exact value of V and V I , we use the estimates and compare the value of ¯ V N 0 and ¯ V I ,N 0 . For this comparison to be valid, the estimation error has to be smaller than the intrinsic gap between V and V I . We define a constant

and we need N 0 to be large enough such that the estimation gap is smaller than ∆ 1 / 2 .

To find the corresponding active constraints, we consider the dual program of V I , where I is determined in steps 4-8 in Algorithm 1, and similarly, we test which dual variable can be set to 0 without influencing the dual objective value. For a dual variable index subset J ⊂ [ K ] , we consider the dual program as follows.

where J c = [ K ] \J . The estimate of Dual J , I , can be obtained from the estimate of its dual, which is given below.

Similarly, we compare the value of Dual I , where Dual I = Dual J ′ , I with J ′ = [ K ] , and Dual J , I . However, we can only compare the value of their estimates Dual I ,N 0 and Dual J , I ,N 0 . To this end, we define a constant

For the comparison to be valid, we need N 0 to be large enough such that the estimation error is smaller than ∆ 2 / 2 . In this way, we identify the linearly independent binding constraints corresponding to q ∗ , as described in steps 9-13 of Algorithm 1.

4 Our Final Algorithm

We now describe our formal algorithm. From the output of Algorithm 1, we characterize one optimal solution. If the sample size n is used in Algorithm 1, we denote by I n and J n the output of Algorithm 1. To be specific, we have q ∗ I c n = 0 and the non-zero elements q ∗ I n can be given as the solution to

However, in practice, both the matrices C ( J n , I n ) and B (: , I n ) are unknown. We aim to use the samples to learn the matrices C ( J n , I n ) and B (: , I n ) such that the q ∗ I n can also be determined. Our formal algorithm is given in Algorithm 2. The steps 3-8 in Algorithm 2 is to use Algorithm 1 as a subroutine to identify the set I and J that satisfy the conditions in Theorem 2.1. We can show that as long as n ≥ N ′ 0 , where N ′ 0 is a threshold that depends on the problem parameters, Algorithm 1 correctly obtains the set I and J satisfying the conditions in Theorem 2.1. Therefore, we exponentially increase the value of n as input to Algorithm 1 to reach N ′ 0 .

Acrucial element in Algorithm 2 (step 10) is that we adaptively update the value of α n J n -1 and µ n ( s ′ ) as in (22) and (23). We then use the updated α n J n -1 and µ n ( s ′ ) to obtain the value of ˜ q n I n as in (21). Such an algorithmic design follows the resolving idea from online LP to achieve a problem-dependent bound. In step 11, we further project ˜ q n +1 to the set { q ≥ 0 : ∥ q ∥ 1 ≤ 2 } to obtain q n +1 . This prevents ˜ q n from behaving ill when n is not large and the estimates of C ( J n , I n ) and B (: , I n ) are not accurate enough. We can show that when n is large enough, ˜ q n +1 automatically belongs to the set { q ≥ 0 : ∥ q ∥ 1 ≤ 2 } .

5 Theoretical Analysis

In this section, we conduct our theoretical analysis. The analysis can be divided into two parts. In the first part, we show that Algorithm 1 can successfully help us identify one optimal basis to work

Algorithm 2 The Adaptive-resolving Algorithm

1: Input: the number of samples N for each ( s, a ) ∈ S × A .

2: Initialize F 1 = ∅ , α 1 = N · α and µ 1 = N · µ .

3: for n = 1 , . . . , N do

4: if n = 2 m for an integer m then

5: Obtain the output I n and J n from Algorithm 1 with the input F n .

6: else

7: set I n = I n -1 and J n = J n -1 .

8: end if

10: Construct a solution ˜ q n such that ˜ q n I c n = 0 and ˜ q ∗ I n is the solution to

9: Construct estimates ¯ C n ( J n , I n ) and ¯ B n (: , I n ) using the sample set F n .

11: Project ˜ q n to the set { q : ∥ q ∥ 1 ≤ 2 } to obtain q n .

12: For each ( s, a ) ∈ I n ⊂ S × A , we query the model M to obtain a sample of the reward r n ( s, a ) and the costs c n k ( s, a ) for each k ∈ J n ⊂ [ K ] , as well as the state transition s n ( s, a ) .

14: Denote by c n ( s, a ) = ( c n k ( s, a )) ∀ k ∈J n and do the update:

13: Update F n +1 = F n ∪ { r n ( s, a ) , c n k ( s, a ) , s n ( s, a ) , ∀ ( s, a ) ∈ S × A , ∀ k ∈ [ K ] } .

.

15: Do the update:

where ✶ { s ′ = s n ( s,a ) } is an indicator function of whether the state transition s n ( s, a ) equals s ′ 16: end for 17: We define ¯ q N such that ¯ q N I c N = 0 and ¯ q N I N = 1 N · ∑ N n =1 q n I N . We then define a policy ¯ π N

18: Output: policy ¯ π N .

with. In the second part, we show how to learn the optimal distribution over the optimal basis we have identified.

We now present the theorem showing that Algorithm 1 indeed helps us identify one optimal basis with a high probability. In practice, the value of ε will be set to 1 /N in the following theorem.

Theorem 5.1 For any ε > 0 , as long as N 0 satisfies the condition

the outputs I N 0 and J N 0 of Algorithm 1 satisfy the conditions described in Lemma 2.1 with probability at least 1 -( K |S||A| - |S| 2 |A| ) · ε . Moreover, the sets I n and J n will be common for any n ≥ N 0 satisfying (25) , which we denote by I ∗ and J ∗ .

An important problem parameter related to I ∗ and J ∗ can be described as follows. Define

We then denote by { σ 1 ( A ∗ ) , . . . , σ |S| + K ′ ( A ∗ ) } the eigenvalues of the matrix A ∗ . We define σ as

From the non-singularity of the matrix A ∗ , we know that σ > 0 . We then have the following bound.

Theorem 5.2 With a sample complexity bound of

where ∆ = min { ∆ 2 1 , ∆ 2 2 } , ξ = min ( s,a ) ∈I ∗ { q ∗ ( s, a ) } and q ∗ denotes the optimal soluton to LP (6) corresponding to the optimal basis I ∗ and J ∗ , we obtain a policy ¯ π N from Algorithm 2 (defined in (24) ) such that

Our Algorithm 2 can be directly applied to solving MDP problems without resource constraints and our bounds in Theorem E.2 and Theorem 5.2 still hold. Note that in the MDP problems, the parameter σ can be lower bounded by 1 -γ , which follows from the fact that the matrix A ∗ can simply be represented by the probability transition matrix. Then, we have the following sample complexity bound for our Algorithm 2.

Proposition 5.3 For the MDP problems without resource constraints, i.e., K = 0 , with a sample complexity bound of

we obtain a policy ¯ π N from Algorithm 2 (defined in (24) ) such that

In terms of the dependency of our sample complexity bound on other problem parameters such as |S| , |A| , and 1 -γ , we compare to the series of work [58, 67, 68, 2, 27], that subsequently achieves a sample complexity bound of O ( |S|·|A| (1 -γ ) 3 · ε 2 ) , where the dependency over |S| , |A| , and 1 -γ is optimal [24, 43]. Our sample complexity bound in (28) has a worse dependency in terms of |S| and 1 -γ . This is because we construct an empirical LP to estimate the value of LP (6) and resolve the linear equation as in (21), where the size of the LP (which is |S| ) and the eigenvalues of the matrix A ∗ (which is bounded by 1 -γ ) will play a part. However, our bound (28) enjoys a better dependency in terms of ε .

6 Conclusions

In this paper, we develop the first instance-dependent ˜ O (1 /ϵ ) sample complexity for constrained MDP problems. We characterize the instance hardness via corner points of the LP formulation and we develop a resolving algorithm to learn the optimal solution while sticking to the identified optimal basis. The work presented by this paper advances the field of Machine Learning and the algorithmic ideas developed in this paper have a broader impact to inspire new algorithms. Our results are developed for the tabular settings, which pose some limitations to the real-world applications of our methods. We leave the extensions to more involved settings for future work.

Acknoledgement

Jiashuo Jiang is generously supported by the early career scheme 26210223 and the general research fund 16204024 from Research Grants Council, Hong Kong.

References

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning , pages 22-31. PMLR, 2017.

Alekh Agarwal, Sham Kakade, and Lin F Yang. Model-based reinforcement learning with a generative model is minimax optimal. In Conference on Learning Theory , pages 67-83. PMLR, 2020.

Shipra Agrawal, Zizhuo Wang, and Yinyu Ye. A dynamic near-optimal algorithm for online linear programming. Operations Research , 62(4):876-890, 2014.

Eitan Altman. Constrained Markov decision processes . vol7. CRCPress, 1999.

Alessandro Arlotto and Itai Gurvich. Uniformly bounded regret in the multisecretary problem. Stochastic Systems , 9(3):231-260, 2019.

Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. Advances in neural information processing systems , 21, 2008.

Mohammad Gheshlaghi Azar, Rémi Munos, Mohammad Ghavamzadeh, and Hilbert Kappen. Reinforcement learning with a near optimal rate of convergence. 2011.

Qinbo Bai, Amrit Singh Bedi, and Vaneet Aggarwal. Achieving zero constraint violation for constrained reinforcement learning via conservative natural policy gradient primal-dual algorithm. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 6737-6744, 2023.

Santiago Balseiro, Haihao Lu, and Vahab Mirrokni. Dual mirror descent for online allocation problems. In International Conference on Machine Learning , pages 613-628. PMLR, 2020.

Pornpawee Bumpensanti and He Wang. A re-solving heuristic with uniformly bounded loss for network revenue management. Management Science , 66(7):2993-3009, 2020.

Boxiao Chen, Jiashuo Jiang, Jiawei Zhang, and Zhengyuan Zhou. Learning to order for inventory systems with lost sales and uncertain supplies. Management Science , 2024.

Yi Chen, Jing Dong, and Zhaoran Wang. A primal-dual approach to constrained markov decision processes. arXiv preprint arXiv:2101.10895 , 2021.

Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research , 18(167):1-51, 2018.

Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. A lyapunov-based approach to safe reinforcement learning. Advances in neural information processing systems , 31, 2018.

Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. Lyapunov-based safe policy optimization for continuous control. arXiv preprint arXiv:1901.10031 , 2019.

Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.08757 , 2018.

Christoph Dann, Teodor Vanislavov Marinov, Mehryar Mohri, and Julian Zimmert. Beyond value-function gaps: Improved instance-dependent regret bounds for episodic reinforcement learning. Advances in Neural Information Processing Systems , 34:1-12, 2021.

Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, Zhaoran Wang, and Mihailo Jovanovic. Provably efficient safe exploration via primal-dual policy optimization. In International Conference on Artificial Intelligence and Statistics , pages 3304-3312. PMLR, 2021.

Dongsheng Ding, Kaiqing Zhang, Jiali Duan, Tamer Ba¸ sar, and Mihailo R Jovanovi´ c. Convergence and sample complexity of natural policy gradient primal-dual methods for constrained mdps. arXiv preprint arXiv:2206.02346 , 2022.

Yaqi Duan and Martin J Wainwright. Taming" data-hungry" reinforcement learning? stability in continuous state-action spaces. arXiv preprint arXiv:2401.05233 , 2024.

Yonathan Efroni, Shie Mannor, and Matteo Pirotta. Exploration-exploitation in constrained mdps. arXiv preprint arXiv:2003.02189 , 2020.

Jaime F Fisac, Anayo K Akametalu, Melanie N Zeilinger, Shahab Kaynama, Jeremy Gillula, and Claire J Tomlin. A general safety framework for learning-based control in uncertain robotic systems. IEEE Transactions on Automatic Control , 64(7):2737-2752, 2018.

Dylan J Foster, Alexander Rakhlin, David Simchi-Levi, and Yunzong Xu. Instance-dependent complexity of contextual bandits and reinforcement learning: A disagreement-based perspective. arXiv preprint arXiv:2010.03104 , 2020.

Mohammad Gheshlaghi Azar, Rémi Munos, and Hilbert J Kappen. Minimax pac bounds on the sample complexity of reinforcement learning with a generative model. Machine learning , 91:325-349, 2013.

Arnob Ghosh, Xingyu Zhou, and Ness Shroff. Achieving sub-linear regret in infinite horizon average reward constrained mdp with linear function approximation. In The Eleventh International Conference on Learning Representations , 2022.

Egor Gladin, Maksim Lavrik-Karmazin, Karina Zainullina, Varvara Rudenko, Alexander Gasnikov, and Martin Takac. Algorithm for constrained markov decision process with linear convergence. In International Conference on Artificial Intelligence and Statistics , pages 1150611533. PMLR, 2023.

Jiafan He, Dongruo Zhou, and Quanquan Gu. Nearly minimax optimal reinforcement learning for discounted mdps. Advances in Neural Information Processing Systems , 34:22288-22300, 2021.

Nicholas J Higham. Accuracy and stability of numerical algorithms . SIAM, 2002.

Jean-Baptiste Hiriart-Urruty and Claude Lemaréchal. Convex analysis and minimization algorithms I: Fundamentals , volume 305. Springer science & business media, 1996.

Roger A Horn and Charles R Johnson. Matrix analysis . Cambridge university press, 2012.

Stefanus Jasin and Sunil Kumar. A re-solving heuristic with bounded revenue loss for network revenue management with customer choice. Mathematics of Operations Research , 37(2):313345, 2012.

Stefanus Jasin and Sunil Kumar. Analysis of deterministic lp-based booking limit and bid price controls for revenue management. Operations Research , 61(6):1312-1320, 2013.

Jiashuo Jiang. Constant approximation for network revenue management with markoviancorrelated customer arrivals. arXiv preprint arXiv:2305.05829 , 2023.

Jiashuo Jiang, Xiaocheng Li, and Jiawei Zhang. Online stochastic optimization with wasserstein based non-stationarity. arXiv preprint arXiv:2012.06961 , 2020.

Jiashuo Jiang, Will Ma, and Jiawei Zhang. Degeneracy is ok: Logarithmic regret for network revenue management with indiscrete distributions. arXiv preprint arXiv:2210.07996 , 2022.

Jiashuo Jiang and Jiawei Zhang. Online resource allocation with stochastic resource consumption. arXiv preprint arXiv:2012.07933 , 2020.

Yujia Jin and Aaron Sidford. Efficiently solving mdps with stochastic mirror descent. In International Conference on Machine Learning , pages 4890-4900. PMLR, 2020.

Sham Machandranath Kakade. On the sample complexity of reinforcement learning . University of London, University College London (United Kingdom), 2003.

Michael Kearns, Yishay Mansour, and Andrew Y Ng. A sparse sampling algorithm for nearoptimal planning in large markov decision processes. Machine learning , 49:193-208, 2002.

Thomas Kesselheim, Andreas Tönnis, Klaus Radke, and Berthold Vöcking. Primal beats dual on online packing lps in the random-order model. In Proceedings of the forty-sixth annual ACM symposium on Theory of computing , pages 303-312, 2014.

Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research , 32(11):1238-1274, 2013.

Tze Leung Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics , 6(1):4-22, 1985.

Tor Lattimore and Marcus Hutter. Pac bounds for discounted mdps. In Algorithmic Learning Theory: 23rd International Conference, ALT 2012, Lyon, France, October 29-31, 2012. Proceedings 23 , pages 320-334. Springer, 2012.

Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Breaking the sample size barrier in model-based reinforcement learning with a generative model. Operations Research , 72(1):203-221, 2024.

Tianjiao Li, Ziwei Guan, Shaofeng Zou, Tengyu Xu, Yingbin Liang, and Guanghui Lan. Faster algorithm and sharper analysis for constrained markov decision process. arXiv preprint arXiv:2110.10351 , 2021.

Xiaocheng Li, Chunlin Sun, and Yinyu Ye. The symmetry between arms and knapsacks: A primal-dual approach for bandits with knapsacks. In International Conference on Machine Learning , pages 6483-6492. PMLR, 2021.

Xiaocheng Li and Yinyu Ye. Online linear programming: Dual convergence, new algorithms, and regret bounds. Operations Research , 70(5):2948-2966, 2022.

Shang Liu, Jiashuo Jiang, and Xiaocheng Li. Non-stationary bandits with knapsacks. Advances in Neural Information Processing Systems , 35:16522-16532, 2022.

Tao Liu, Ruida Zhou, Dileep Kalathil, Panganamala Kumar, and Chao Tian. Learning policies with zero or bounded constraint violation for constrained mdps. Advances in Neural Information Processing Systems , 34:17183-17193, 2021.

Tao Liu, Ruida Zhou, Dileep Kalathil, PR Kumar, and Chao Tian. Policy optimization for constrained mdps with provable fast global convergence. arXiv preprint arXiv:2111.00552 , 2021.

Yongshuai Liu, Jiaxin Ding, and Xin Liu. Ipo: Interior-point policy optimization under constraints. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 4940-4947, 2020.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013.

Ted Moskovitz, Brendan O'Donoghue, Vivek Veeriah, Sebastian Flennerhag, Satinder Singh, and Tom Zahavy. Reload: Reinforcement learning with optimistic ascent-descent for last-iterate convergence in constrained mdps. In International Conference on Machine Learning , pages 25303-25336. PMLR, 2023.

Wenlong Mou, Zheng Wen, and Xi Chen. On the sample complexity of reinforcement learning with policy space generalization. arXiv preprint arXiv:2008.07353 , 2020.

Angelia Nedi´ c and Asuman Ozdaglar. Subgradient methods for saddle-point problems. Journal of optimization theory and applications , 142:205-228, 2009.

Masahiro Ono, Marco Pavone, Yoshiaki Kuwata, and J Balaram. Chance-constrained dynamic programming with application to risk-aware robotic space exploration. Autonomous Robots , 39:555-571, 2015.

Guy Shani, David Heckerman, Ronen I Brafman, and Craig Boutilier. An mdp-based recommender system. Journal of Machine Learning Research , 6(9), 2005.

Aaron Sidford, Mengdi Wang, Xian Wu, Lin Yang, and Yinyu Ye. Near-optimal time and sample complexities for solving markov decision processes with a generative model. Advances in Neural Information Processing Systems , 31, 2018.

Max Simchowitz and Kevin G Jamieson. Non-asymptotic gap-dependent regret bounds for tabular mdps. Advances in Neural Information Processing Systems , 32, 2019.

Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning , pages 9133-9143. PMLR, 2020.

Zhongchang Sun, Sihong He, Fei Miao, and Shaofeng Zou. Constrained reinforcement learning under model mismatch. arXiv preprint arXiv:2405.01327 , 2024.

Aviv Tamar, Dotan Di Castro, and Shie Mannor. Policy gradients with variance related risk criteria. In Proceedings of the twenty-ninth international conference on machine learning , pages 387-396, 2012.

Sharan Vaswani, Lin Yang, and Csaba Szepesvári. Near-optimal sample complexity bounds for constrained mdps. Advances in Neural Information Processing Systems , 35:3110-3122, 2022.

Alberto Vera and Siddhartha Banerjee. The bayesian prophet: A low-regret framework for online decision making. Management Science , 67(3):1368-1391, 2021.

Andrew J Wagenmaker and Dylan J Foster. Instance-optimality in interactive decision making: Toward a non-asymptotic theory. In The Thirty Sixth Annual Conference on Learning Theory , pages 1322-1472. PMLR, 2023.

Andrew J Wagenmaker, Max Simchowitz, and Kevin Jamieson. Beyond no regret: Instancedependent pac reinforcement learning. In Conference on Learning Theory , pages 358-418. PMLR, 2022.

Martin J Wainwright. Variance-reduced q -learning is minimax optimal. arXiv preprint arXiv:1906.04697 , 2019.

Mengdi Wang. Randomized linear programming solves the markov decision problem in nearly linear (sometimes sublinear) time. Mathematics of Operations Research , 45(2):517-546, 2020.

Honghao Wei, Xin Liu, and Lei Ying. A provably-efficient model-free algorithm for constrained markov decision processes. arXiv preprint arXiv:2106.01577 , 2021.

Tengyu Xu, Yingbin Liang, and Guanghui Lan. Crpo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning , pages 11480-11491. PMLR, 2021.

Yunbei Xu and Assaf Zeevi. Towards optimal problem dependent generalization error bounds in statistical learning theory. Mathematics of Operations Research , 2024.

Andrea Zanette and Emma Brunskill. Tighter problem-dependent regret bounds in reinforcement learning without domain knowledge using value function bounds. In International Conference on Machine Learning , pages 7304-7312. PMLR, 2019.

Sihan Zeng, Thinh T Doan, and Justin Romberg. Finite-time complexity of online primal-dual natural actor-critic algorithm for constrained markov decision processes. In 2022 IEEE 61st Conference on Decision and Control (CDC) , pages 4028-4033. IEEE, 2022.

Ruida Zhou, Tao Liu, Dileep Kalathil, PR Kumar, and Chao Tian. Anchor-changing regularized natural policy gradient for multi-objective reinforcement learning. Advances in Neural Information Processing Systems , 35:13584-13596, 2022.

A Numerical Experiments

We implement our Algorithm 2 to study the numerical performance. We consider a CMDP problem with the state space |S| = 10 and the action space |A| = 10 . We set the discount factor γ = 0 . 7 . We then randomly generate the probability transition kernel P . To be specific, for each state s ∈ S , action a ∈ A , and the future state s ′ ∈ S , we uniformly generate a randomly variable p s,a,s ′ . Then, the transition probability is defined as P ( s ′ | s, a ) = p s,a,s ′ ∑ s ′′ ∈S p s,a,s ′′ . For each state-action pair ( s, a ) ∈ S × A , the expected reward ˆ r ( s, a ) is uniformly generated from the interval [1 , 2] (with the reward for the first action set to be 0 ). The actual reward r ( s, a ) = ˆ r ( s, a ) + η , where η is uniformly distributed among [ -0 . 5 , 0 . 5] . There are K = 5 constraints and for each constraint k ∈ [ K ] and each state-action pair ( s, a ) ∈ S × A , we define the expected cost ˆ c k ( s, a ) to be uniformly generated from [1 , 2] . The actual cost c k ( s, a ) = ˆ c k ( s, a ) + η ′ , where η ′ is uniformly distributed among [ -0 . 5 , 0 . 5] .

For each total iterations N , We apply Algorithm 2 and obtain the output q 1 , . . . , q N . We compare ¯ q N with the optimal occupancy measure. Since our algorithm is a randomized algorithm, we study the performance of our algorithm in expectation. To be specific, given the problem instance and a fixed N , we implement our algorithm repeatedly for M = 500 rounds. Denote by ¯ q N m the output of our Algorithm 2 at round m , for m ∈ [ M ] . We define the error term as Err ( N ) = 1 M · ∑ M m =1 ∥ ¯ q N m -q ∗ ∥ 1 / ∥ q ∗ ∥ 1 . We study how the error term Err ( N ) scales with N . The results are displayed in Figure 2. As we can see, the error term Err ( N ) decreases quickly with respect to N . Moreover, since our Algorithm 2 requires only solving a set of linear equations in each iteration, the computation cost of our Algorithm 2 is also moderate.

0.2

0.18

0.16

0.14

0.12

0.1

0.08

0.06

0.04

0.02

0

0

2

3

4

5

Size of N

×104

B Proof of Lemma 2.1

Let q ∗ be a basic optimal solution to LP (6). We also denote by I the index set of q ∗ such that q ∗ I > 0 . We then denote by J ′ ∈ [ K ] the set of active constraints at q ∗ . Then, the following linear equations must be satisfied by q ∗ .

From the property of the basic optimal solution, we must have |J ′ | + b ≥ |I| .

If |J ′ | + b = |I| , then we know that we already find the desired index set I and J = J ′ .

Otherwise, if |J ′ | + b > |I| , then we know that the LP (6) is degenerate. However, now the linear system (29) is over-determined, i.e., there must be |J ′ | + b -|I| number of equations can be implied by the others. It only remains to show that those redundant equations are all in the set J ′ . This step can be done can showing that the matrix B (: , I ) has a full row rank. We prove this by showing contradiction.

For the infinite horizon discounted problem, suppose that B (: , I ) does not have full row rank, there must exists β ∈ R |S| such that

On the other hand, denote by π ∗ the optimal policy corresponding to q ∗ . We then define a matrix Γ ∈ R |I|×S such that the s -th column of Γ is a vector that takes 0 for all the ( s ′ , a ) ∈ I -th element if s ′ = s , and take a value of π ∗ ( a | s ) for all a such that ( s, a ) ∈ I . We know that

with I ∈ R |S|×|S| being the identify matrix and P π ∗ being the transition probability matrix under the policy π ∗ , with the element at the s -th row and s ′ -th column denoting the probability of transiting from state s to state s ′ under the policy π ∗ . It is well known that the matrix I -γ · P π ∗ is non-singular (see for example the Gersgorin's Theorem in [30]). Therefore, for any vector β ∈ R |S| , we know that

̸

which contradicts with (60). Our proof is thus completed.

C Proof of Lemma 3.2

We now condition on the event that | ¯ r N 0 ( s, a ) -ˆ r ( s, a ) | , | ¯ c k,N 0 ( s, a ) -ˆ c k ( s, a ) | , and | ¯ P N 0 ( s ′ | s, a ) -P ( s ′ | s, a ) | for each k ∈ [ K ] , ( s, a ) ∈ S × A and s ′ ∈ S are all bounded by Rad( N 0 , ε ) . From the union bound, we know that this event happens with probability at least 1 -( K |S||A| - |S| 2 |A| ) · ε .

Note that for LP (4), by summing up the constraint (4c) for all s ∈ S , we obtain that any feasible solution q for LP (4) would satisfy

We first upper bound the gap V -¯ V N 0 . Denote by q ∗ one optimal solution to V . Then, from the feasibility of q ∗ , we know that

where the first inequality follows from ∥ q ∗ ∥ 1 = 1 (31) and all elements of ¯ C N 0 -C are upper bounded by Rad( N 0 , ε ) . Also, we know that

where the first inequality follows from ∥ q ∗ ∥ 1 = 1 (31) and all elements of ¯ B N 0 -B are upper bounded by γ · Rad( N 0 , ε ) . Similarly, we have that

Therefore, as long as

̸

we know that q ∗ is a feasible solution to ¯ V N 0 . We have that

by noting ∥ q ∗ ∥ 1 = 1 (31) and all elements of ¯ r N 0 -ˆ r are upper bounded by Rad( N 0 , ε ) . Therefore, we can obtain the bound

We then lower bound the gap V -¯ V N 0 . We first define

for any constant λ ′ N 0 . Clearly, any optimal solution ¯ q ∗ to ¯ V N 0 will be a feasible solution to ¯ V N 0 ( λ ′ N 0 ) . Moreover, by summing up the constraints ¯ B N 0 q ≤ µ + λ N 0 for all s ∈ S , we know that

Then, it holds that

We then compare the value between ¯ V N 0 ( λ ′ N 0 ) and V . The dual of ¯ V N 0 ( λ ′ N 0 ) is given below.

Denote by y ∗ and z ∗ one optimal solution to the dual of LP (6), given below.

We now show that y ∗ and z ∗ is also a feasible solution to Dual ′ N 0 , with λ ′ N 0 = 1 min k ∈ [ K ] { α k } · ( 1 + |S| 1 -γ ) · Rad( N 0 , ε ) . We have the following claim regarding the upper bound on ∥ y ∗ ∥ ∞ and ∥ z ∗ ∥ ∞ .

Claim C.1 There exists an optimal solution y ∗ and z ∗ to the Dual (42) such that

Then, we define ¯ y ∗ = y ∗ , z ∗ = max { 0 , z ∗ } and z ∗ = max { 0 , -z ∗ } . We have

Thus, we know that ¯ y ∗ and z ∗ 1 , z ∗ 2 is also a feasible solution to Dual N 0 ( λ ′ N 0 ) , and we have

Combing (43) with (40) and also noting that ¯ V N 0 ( λ ′ N 0 ) = Dual ′ N 0 ( λ ′ N 0 ) , we have

which completes our proof.

Proof of Claim C.1. We first bound ∥ y ∗ ∥ ∞ . We utilize the approach in [29, 55]. We define a Lagrangian function, with only y as the Lagrangian dual variable.

where the feasible set for q is { q ≥ 0 : B q = µ } and the feasible set for y is { y ≥ 0 } . Following Lemma 3 in [55], it is without loss of generality to restrict the feasible set of y to the set { y ≥ 0 : ∥ y ∥ 1 ≤ ρ } , where the constant ρ is defined as

where ¯ q is the occupancy measure that satisfies Slater's condition as stated in Theorem 3.1. Note that we have

Therefore, we obtain the following bound on y ∗ :

We now proceed to bound z ∗ . Denote by q ∗ the optimal solution corresponding to the optimal dual solution ( y ∗ , z ∗ ) , with y ∗ bounded as in (46). We also denote by π ∗ the optimal policy corresponding to q ∗ . Then, from the complementary slackness condition, as long as q ∗ ( s, a ) > 0 for a ( s, a ) , we must have

We now multiply both sides of (47) by q ∗ ( s, a ) / ∑ a ′ ∈A q ∗ ( s, a ′ ) , and sum up over a , for each state s . Then we get

Here, C π ∗ ∈ R K ×|S| and the element at k -th row and s -th column is ∑ a ∈A c k ( s, a ) · q ∗ ( s,a ) ∑ a ′ ∈A q ∗ ( s,a ′ ) . B π ∗ = I -γ · P π ∗ ∈ R |S|×S , where P π ∗ ( s, s ′ ) = ∑ a ∈A π ∗ ( a | s ) · P ( s ′ | s, a ) denotes the transition probability matrix under the policy π ∗ . Also, ˆ r π ∗ ∈ R |S| with ˆ r π ∗ ( s ) = ∑ a ∈A ˆ r ( s, a ) · q ∗ ( s,a ) ∑ a ′ ∈A q ∗ ( s,a ′ ) . Then, we have

Also, from the bound on y ∗ in (46), we know that

□

From [37], we know that

Therefore, we have that which completes our proof.

D Proof of Theorem 5.1

We now condition on the event that | ¯ r N 0 ( s, a ) -ˆ r ( s, a ) | , | ¯ c k,N 0 ( s, a ) -ˆ c k ( s, a ) | , and | ¯ P N 0 ( s ′ | s, a ) -P ( s ′ | s, a ) | for each k ∈ [ K ] , ( s, a ) ∈ S × A and s ′ ∈ S are all bounded by Rad( N 0 , ε ) . We know that this event happens with probability at least 1 -( K |S||A| - |S| 2 |A| ) · ε .

Following the procedure in Algorithm 1, we identify an index set I , where we set q i = 0 for each i ∈ I c . Note that we cannot further delete one more i from the set I . Otherwise, when deleting this particular i in I by setting I ′ = I\{ i } , we will have V I ′ = V and as a result of Lemma 3.2, we have

Therefore, if it is feasible (objective value does not change) to delete one more i from the index set I , our algorithm will already do so.

We denote by q ∗ the optimal solution to LP (6), corresponding to the optimal basis I identified by Algorithm 1. We know that q ∗ I > 0 and q ∗ I c = 0 , which implies that

Note that in the formulation of (54), we simply discard the columns of the constraint matrix in the index set I . Thus, one optimal solution to (54) will just be q ∗ I . The dual of (54) is

From the complementary slackness condition, we know that for the optimal dual variable corresponding to q ∗ I , all the constraints in (55) must hold with equality. Therefore, we have the following result.

Claim D.1 It holds that

The proof of the claim is relegated to the end of this proof. We now show that Algorithm 1 identify the linearly independent binding constraints for q ∗ such that the conditions in Lemma 2.1 are satisfied.

Denote by C ( J , I ) a sub-matrix of C such that the rows in the index set J ⊂ [ K ] and the columns in the index set I of the matrix C remain. Now if the matrix

is singular, there must be

(i). one row in C ( J , I ) can be expressed as the linear combination of the other rows in A (we know that the matrix B (: , I ) has full row rank from Lemma 2.1);

(ii). or one column in A can be expressed as the linear combination of other columns.

We consider the situation (i). Denote this row as k ′ and we know that restricting y k ′ = 0 will not change the objective value of Dual I , which implies that k ′ will be added to the index set J . This is because y k ′ cannot be a basic variable, otherwise, the optimal basis of the Dual (56) will be linearly dependent. Therefore, we obtain a contradiction and we know that the matrix A must have full row rank. Thus, situation (i) will not happen.

We then consider the situation (ii). Note that we have

Comparing the formulation (58) to the formulation (55), we only remain the rows of the matrix C (: , I ) in the index set J , where it is feasible (objective value does not change) to set y J c = 0 . The dual of (58) is

If situation (ii) happens and one column of A can be expressed as the linear combination of the other columns in A , then we denote the index of this column by i ′ and we know that we can simply restrict q i ′ = 0 without changing the objective value of (59). This is because q i ′ cannot be a basic variable otherwise the corresponding optimal basis will be linearly dependent. However, the index i ′ ∈ I . This means that we cannot further delete i ′ from the index set I by restricting q i ′ = 0 without changing the objective value of LP (4). The above argument leads to a contradiction. Therefore, we know that situation (ii) cannot happen.

Given the arguments above, we know that the matrix A is a non-singular matrix and thus the conditions in Lemma 2.1 are satisfied with the variable index set I and the constraint index set J .

It only remains to show that we can tell whether the objective value of V Infi or Dual I has changed by restricting one variable to be 0 . Following the same steps as in Lemma 3.2, we can show that for index sets I ′ and J ′ , it holds that

and the upper gap as with the formulation of Gap 1 ( N 0 , ε ) and Gap 2 ( N 0 , ε ) given in (12) and (13). Therefore, as long as

where δ 1 is defined in (16) and δ 2 is defined in (19), we can tell whether V Infi is different from V Infi I ′ and whether Dual I ′ is different from Dual J ′ , I ′ . Our proof is thus completed.

Proof of Claim D.1. We first show that

Denote by q ∗ the optimal solution to V corresponding to the optimal basis I . It is clear to see that q ∗ I is a feasible solution to V I with the same objective value. Then, we have

On the other hand, we denote by ˆ q one optimal solution to V I , and we construct

It is clear to see that ˜ q is a feasible solution to V with the same objective value, which implies that

Therefore, (63) is proved from combining (64) and (65). The dual of V I is given by

We denote by y ∗ , z ∗ the optimal dual variable to V corresponding to q ∗ . It is easy to see that y ∗ , z ∗ is also feasible to Dual I and

where the first equality follows from the strong duality between V I and Dual I , the second equality follows from (63), and the third equality follows from the strong duality between V and its dual. Therefore, we know that y ∗ and z ∗ is also an optimal solution to Dual I . Moreover, note that from the complementary slackness condition, since q ∗ I > 0 , we must have

We know that y ∗ , z ∗ is a feasible solution to

which implies that

On the other hand, any feasible solution to Dual ′ I must be a feasible solution to Dual I , and we have

Therefore, we must have Dual ′ I = Dual I and our proof is completed. □

E Proof of Theorem 5.2

We first prove the following lemma.

Lemma E.1 For the optimal basis identified in Algorithm 1 and the corresponding optimal solution q ∗ , we denote by I ∗ and J ∗ the output sets as long as N 0 satisfies the condition (25) . We also denote by ( y ∗ , c ∗ ) the corresponding optimal dual solution. Then, it holds that

Therefore, it suffices to analyze how the 'remaining resources' ( α n J ∗ , µ n ) behave. We now define

The key is to show that the stochastic process ˜ α k ( n ) and ˜ µ s possess some concentration properties such that they will stay within a small neighborhood of their initial value α 1 k and µ 1 s for a sufficiently long time. We denote by τ the time that one of ˜ α k ( n ) for each k ∈ J ∗ and ˜ µ s for each s ∈ S escape this neighborhood. Then, both Regret r ( π, N ) and Regret k ( π, N ) for each k ∈ [ K ] can be upper bounded by E [ N -τ ] . From the update rule (22) and (23), we know that

and

Ideally, both ˜ α k ( n +1) and ˜ µ s ( n +1) will have the same expectation as ˜ α k ( n ) and ˜ µ s ( n ) such that they become a martingale. However, this is not true since we have estimation error over C ( J ∗ , I ∗ ) and B (: , I ∗ ) , and we only use their estimates to compute q n . Nevertheless, we can show that ˜ α k ( n ) for each k ∈ J ∗ and ˜ µ s for each s ∈ S behave as a sub-martingale. Then, from the concentration property of the sub-martingale, we upper bound E [ α N k ] for each k ∈ J ∗ and E [ µ N s ] for each s ∈ S . The term | E [ α N k ] | for each k ∈ [ K ] \J ∗ can be upper bounded as well. The results are presented in the following lemma.

Lemma E.2 Denote by ¯ π N the output policy of Algorithm 2 and denote by N the number of rounds. Then, it holds that

where the parameters α = min k ∈ [ K ] { α k } , ∆ = min { ∆ 2 1 , ∆ 2 2 } with ∆ 1 given in (16) and ∆ 2 given in (19) , σ given in (27) . Also, for any k ∈ [ K ] , we have

E.1 Proof of Lemma E.1

Note that the distribution of q n is independent of the distribution of r n . We know that

Denote by q ∗ and y ∗ , z ∗ the optimal primal-dual variable corresponding to the optimal basis I ∗ and J ∗ . From the complementary slackness condition and noting that q ∗ I ∗ > 0 , we know that

Also, we can define a matrix C n ( J ∗ , I ∗ ) such that the element of C n ( J ∗ , I ∗ ) at the k ∈ J ∗ row and ( s, a ) ∈ I ∗ column is c n k ( s, a ) . We can also define a matrix B n (: , I ∗ ) such that the element of B n (: , I ∗ ) at the s ′ ∈ S row and ( s, a ) ∈ I ∗ column is δ s ′ ,s -γ · { s ′ = s n ( s,a ) } . It is easy to see that

Then, it holds that

Note that we have and

Plugging (76) and (77) back into (75), we get that

Note that

that holds from the strong duality of V Infi (4). Then, we have that

Our proof is thus completed.

E.2 Proof of Lemma E.2

We now condition on the event that | ¯ r n ( s, a ) -ˆ r ( s, a ) | , | ¯ c k,n ( s, a ) -ˆ c k ( s, a ) | , and | ¯ P n ( s ′ | s, a ) -P ( s ′ | s, a ) | for each k ∈ [ K ] , ( s, a ) ∈ S × A and s ′ ∈ S are all bounded by Rad( n, ε ) , for any n ∈ [ N ] . We know that this event happens with probability at least 1 -N · ( K |S||A| - |S| 2 |A| ) · ε . From now on, we set ε = 1 N 2 to guarantee that the event happens with probability at least 1 -K |S||A| + |S| 2 |A| N .

We consider the stochastic process ˜ α k ( n ) and ˜ µ s ( n ) defined in (70). For a fixed ν > 0 which we specify later, we define a set

and

It is easy to see that initially, ˜ α (1) ∈ X and ˜ µ (1) ∈ Y . We show that ˜ α ( n ) and ˜ µ ( n ) behave well as long as they stay in the region X and Y for a sufficiently long time. To this end, we define a stopping time

Note that in Algorithm 2, to stop q n from behaving ill when n is small, we project it to a set that guarantees ∥ q n ∥ 1 ≤ 2 . We now show in the following claim that when n is large enough but smaller than the stopping time τ , there is no need to do projection.

Claim E.3 There exist two constants N ′ 0 and ν 0 . When max { N 0 , N ′ 0 } ≤ n ≤ τ , and ν ≤ ν 0 , it holds that ∥ ˜ q n I ∗ ∥ 1 ≤ 2 , where ˜ q n I ∗ denotes the solution to the linear equations (21) . Specifically, N 0 is given in (25) and N ′ 0 is given as follows

We set ν satisfy the condition ν ≤ ν 0 with ν 0 satisfies the condition in Claim E.3. We bound E [ N -τ ] in the following claim.

Claim E.4 Let the stopping time τ be defined in (81) . It holds that

where N 0 is given in (25) and N ′ 0 is given in (82) , as long as

Also, for any N ′ such that max { N 0 , N ′ 0 } ≤ N ′ ≤ N , it holds that

From the definition of the stopping time τ in (81), we know that for each k ∈ J ∗ , it holds

Thus, we have that

and thus

Following the same procedure, we can show that for each s ∈ S , it holds that

We finally consider the other constraints k ∈ J ∗ c . Note that following the definition of α n and µ n , we have that

Also, from the bindingness of q ∗ regarding the optimal basis I ∗ and J ∗ , we have

Also, ν 0 is given as follows

Therefore, it holds that

and

Finally, for any k ∈ J ∗ c , we have

From the feasibility of q ∗ , we know that

Therefore, for any k ∈ J ∗ c , it holds that

Moreover, the definition of σ in (27) implies that following upper bound on the norm of the dual variable y ∗ and z ∗ .

Therefore, we know that the regret over the reward and the regret over the constraint violation can all be bounded by using (87), (88), and (94). We present the bounds as follows.

Meanwhile, the constraint violations are bounded by (87), (88), and (94). Our proof is thus completed.

Proof of Claim E.3. Denote by q ∗ the optimal solution corresponding to the optimal basis I ∗ and J ∗ . Then, it holds that

We compare ˜ q n I ∗ with q ∗ I ∗ when n large enough. Note that when n ≥ N 0 , ˜ q n is the solution to the following linear equations

Moreover, we know that the absolute value of each element of ¯ C n ( J ∗ , I ∗ ) -C ( J ∗ , I ∗ ) , and ¯ B n (: , I ∗ ) -B (: , I ∗ ) is upper bounded by Rad( n, ε ) . We now bound the distance between the solutions to the linear equations (97) and (98). The perturbation of the matrix is denoted as

Clearly, it holds that

Therefore, as long as

When n ≤ τ , we know that and

following standard perturbation analysis of linear equations [28], we have that

where κ ( A ∗ ) = ∥ A ∗ ∥ 1 · ∥ ( A ∗ ) -1 ∥ 1 denotes the conditional number of A ∗ . The last inequality follows from ∥ [ α J ∗ ; µ ] ∥ 1 ≥ 1 -γ . Further, note that ∥ q ∗ I ∗ ∥ 1 = 1 . Therefore, in order to satisfy the condition ∥ ˜ q n I ∗ ∥ 1 ≤ 2 , we only need the right hand side of (103) to be upper bounded by 1. Clearly, as long as n satisfies the condition (102) and the following condition

we only need to select a ν such that

Combining (102) and (104), we know that n needs to satisfy the following conditions: n ≥ N 0 and

Also, ν is selected to satisfy the following condition

□

Our proof is thus completed.

Proof of Claim E.4. Now we fix a k ∈ J ∗ . We specify a ¯ N 0 = max { N 0 , N ′ 0 } . For any ¯ N 0 ≤ N ′ ≤ N , it holds that

We define ξ k ( n ) = ˜ α k ( n +1) -˜ α k ( n ) . Then, we have

where F n denotes the filtration of information up to step n . Note that due to the update in (71), we have

Then, it holds that

where the inequality follows from the fact that the value of ˜ α k ( n ) is deterministic given the filtration F n and ∥ q n ∥ 1 ≤ 2 for any n . Note that

forms a martingale difference sequence. Following Hoeffding's inequality, for any N ′′ ≤ N ′ and any b > 0 , it holds that

Therefore, we have that

holds for any b > 0 .

We now bound the probability that τ > N ′ for one particular N ′ such that ¯ N 0 ≤ N ′ ≤ N . Suppose that N ′ ≤ τ , then, from Claim E.3, for each n ≤ N ′ , we know that ∥ ˜ q n ∥ 1 ≤ 2 and therefore q n = ˜ q n as the solution to (21). We have

It holds that

Then, we know that

for a N large enough such that

Combining (111) and (109) with b = ν/ 2 , and apply a union bound over all k ∈ J ∗ and s ∈ S , we know that

Therefore, we know that

which completes our proof.

E.3 Final Proof of Theorem 5.2

Note that in the proof of Lemma E.2, we have shown the following bounds.

holds for the each k ∈ J ∗ . For each s ∈ S , it holds that

The caveat of directly transferring the bound of (114) and (115) into the sample complexity bounds of the policy ¯ π N is that, the vector ¯ q N does not directly characterize an occupancy measure. This point

□

can be seen by noting that there is a gap between B ¯ q N and µ , though bounded by O (log( N ) /N ) by setting ε = 1 /N 2 . However, we can show that the gap between ¯ q N and q ∗ is upper bounded by O (log( N ) /N ) , which implies a bound over the gap between the policy ¯ π N and the optimal policy π ∗ that corresponds to the occupancy measure q ∗ . This bound over the gap between the policy distributions can be then transferred into the bound over the gap between the state-value functions under the policy ¯ π N and π ∗ . The regret bounds can be obtained then.

We first bound the gap between ¯ q N and q ∗ . Note that as long as n ≥ N 0 , we have I n = I ∗ following Theorem 5.1. Then, by noting I N = I ∗ , we know that

Also, note that following the definition of α n and µ n , we have that

Also, from the bindingness of q ∗ regarding the optimal basis I ∗ and J ∗ , we have

Then, from (117) and (118), we know that

From Markov's inequality, for each i ∈ I ∗ and any a > 0 , we know that

We denote by

The policy ¯ π N is essentially random by noting that q N is a random variable, where the randomness comes from the randomness of the filtration F N . For each s ∈ S and a ∈ A , we denote by ¯ π ( a | s ) the (ex-ante) probability that the random policy ¯ π takes the action a given the state s . Then, for any 0 < g ≤ ξ/ 2 , we note that

for each i = ( s, a ) ∈ I ∗ . For any 0 < g ≤ ξ/ 2 , note that

where the inequality follows from the bound (120) and the union bound over i ∈ I ∗ . Therefore, for any 0 < g ≤ ξ/ 2 and any ( s, a ) , we know that

From the above inequality, for any ( s, a ) , we have that

We finally transfer the bound (125) into the sample complexity bounds of policy ¯ π N . We use the state-value functions V r ( π, s ) , defined for any initial state s and any policy π as follows

where ( s t , a t ) is generated according to the policy π and the transition kernel P with the initial state s . Note that the value of V r ( π, s ) for any s ∈ S can be obtained from solving Bellman's equation under policy π

We define a matrix B π ∈ R |S|×|S| such that the s -th row s ′ -th column element is

Then, the matrix B π represents the state transition probability matrix under the policy π . Denote by

and

We have that the state values V r ( π ) is the solution to the linear equation

To bound the regret, we bound the solution to the linear equation (129) with π being ¯ π N and π ∗ separately. The perturbation of the right hand of the equation (129) is

Clearly, we have that

The perturbation of the matrix is denoted as

Clearly, it holds that

We plug the formulation of N 0 in (25) and N ′ 0 in (82) into the bound (130) and (131). We obtain

where C 1 is a constant, α = min k ∈ [ K ] { α k } , and ∆ = min { ∆ 2 1 , ∆ 2 2 } with ∆ 1 given in (16) and ∆ 2 given in (19). We also obtain

Therefore, as long as

following standard perturbation analysis of linear equations [28], we have that

where κ ( B π ∗ ) = ∥ B π ∗ ∥ ∞ · ∥ ( B π ∗ ) -1 ∥ ∞ denotes the conditional number of B π ∗ , and C 2 is a constant. Note that we have the regret

It is clear to see that and

Following [37], we have the following bound.

Also, from the definition, we have that

Plugging the bound (132), (133), (137), and (138), into the inequality (136), we have that

where C 3 is a constant. In a same way, for any k ∈ [ K ] , we obtain that

To show the sample complexity bound, we denote by ε a constant such that

Therefore, we have

Note that in each of the N rounds, we obtain a sample for each ( s, a ) ∈ S × A . Therefore, the bound on N above should be multiplied by |S| · |A| to obtain the final sample complexity bound. Our proof is thus completed.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The contributions and scope is accurately reflected.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed in the conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions and proofs are provided in full.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Experimental details given in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This is a theoretical paper and all the data is simulated with details given in the appendix.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the experimental details are given in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The experiment has been repeated enough times to guarantee that the expected regret has been appropriately approximated.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The simulated experiments are easy to implement and no significant computing resource is required.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: It conforms.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Presented in the conclusion.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Not applied here.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: Not applied here.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Not applied here.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Not applied here.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: None is included.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the linear program formulation of constrained MDPs. The paper first characterizes the instance hardness of the underlying LP. Then, by proposing an algorithm that operates in the primal space and resolves the primal LP in an online manner, the paper derives an overall sample complexity of O(1/\\epsilon) up to logarithm terms.\n\nSTRENGTHS:\n* The paper provides a strong bound on the constrained MDP using LP-based approaches. The obtained sample complexity is the first in the literature that achieves \\tilde O(1/\\epsilon).\n* The derivation and presentation of theoretical results are clear and easy to follow. The proposed theoretical framework has the potential to be applied to more general online LP problems.\n\nWEAKNESSES:\n* The introduction and related work parts of the paper are not comprehensive enough (especially the related work). In the beginning of Section 1.2, the paper compares itself with [25, 29, 44, 14]. Yet, all of these works focus on policy-based methods, and I think it is not directly comparable with the proposed method in this paper. For example, the dependency of the sample complexity in this work is |S|^4, and this is usually lower for policy-based methods. Moreover, I think the paper might miss several related literature also studies occupancy-measure based approaches, for example,  \"Achieving Zero Constraint Violation for Concave Utility Constrained Reinforcement Learning via Primal-Dual Approach\" by Bai et al. \n* The paper does not provide numerical simulations. If there is some simulation, I think it helps the reader to have a better sense of how the convergence rate of the algorithm depends on the problem size.\n\nQUESTIONS:\n* Can the author provide a more comprehensive literature review and problem introduction? And I think it is also important to consider the dependency of the sample complexity on the problem-related parameters (i.e., |S| and |A|) in the comparison.\n* It would be better to briefly discuss the proof idea/implications of theoretical results around the theories. Currently, the ending of the paper seems too abrupt?\n* If it is possible, can the author provides some preliminary numerical experiments (not mandatory)?\n\nLIMITATIONS:\nI did not find the place where the paper explicitly discuss the limitation (yet, the author claims \"Yes\" in the checklist question 2). BTW, it seems the author also forgot to provide justifications for other questions in the checklist as well.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new algorithm that solves the CMDP problem in $O(1/\\epsilon)$ sample complexity, which improves the best-known $O(1/\\epsilon^2)$ sample complexity in the literature. To achieve this, this paper made three contributions: (1) New characterizations of the problem instance hardness for CMDP problems. (2) A new algorithm based on LP literature instead of the traditional RL literature. (3) Extended results adopted to online LP literature.\n\nSTRENGTHS:\nThe proposed method is new. Though it is mainly inpired by existing LP literature, to my understanding similar approaches have not been applied in RL or ML research. The LP formulation also leads to a new perspective in understanding the CMDP problem.\n\nWEAKNESSES:\n1. The presentation of this work is relatively poor. The author listed three contributions but I feel hard to understand how these contributions have contributed to this work. For example, in the Contribution 1, the author should tell (a) what is the problem instance hardness for CMDP problem, (b) what is the motivation of proposing new characterizations of the problem instance hardness for CMDP problems, (c) why the new characterization is helpful in developing the new algorithm or achieving better complexity.  \n2. There is no sufficient discussions on Lemma 2.1. How is it connected to the whole section Characterization of Instance Hardness?\n3. The reward, cost, and transitions are all estimated during training. I cannot agree that this algorithm work with unknown transition probabilities since it requires to estimate that.\n4. I doubt if the theoretical analysis is correct since it has been known that the sample complexity lower bound is $O(1/\\epsilon^2)$. See [R1] and [R2].\n\n[R1] Azar, Mohammad Gheshlaghi, et al. \"Reinforcement learning with a near optimal rate of convergence.\" (2011).\n[R2] Vaswani, Sharan, Lin Yang, and Csaba Szepesvári. \"Near-optimal sample complexity bounds for constrained MDPs.\" Advances in Neural Information Processing Systems 35 (2022): 3110-3122.\n\nQUESTIONS:\nI am mainly concerned about the $O(1/\\epsilon)$ complexity. Can the author clarify the difference between this work and classical RL literature?\n\nLIMITATIONS:\nThis is a theoretical work so there is no any negative impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-04"}

comments: SUMMARY:
This paper addresses the reinforcement learning problem for CMDPs.  The authors derived a problem-dependent sample complexity bound that is $\tilde O(1/\epsilon)$, improving upon the state-of-the-art.  They introduce a novel way to characterize the hardness of CMDP instances using the LP basis, enabling problem-dependent guarantees. The proposed algorithm involves an elimination procedure to identify an optimal basis and a resolving procedure that adapts to remaining resources, ensuring the policy remains near-optimal with fewer samples.

STRENGTHS:
The paper is well written, and the intuition/ideas behind the algorithm and theoretical proofs are clearly explained, making the paper a pleasant read.  I've learned something interesting and new.

WEAKNESSES:
While this point may seem minor since the problem setting assumes a tabular formulation with finite and fully observable state and action spaces, it is important to note that the methodology becomes challenging to apply when dealing with large or infinite state spaces where function approximation is required.

QUESTIONS:
N/A

LIMITATIONS:
Suggestions:
1) A typo in 1.1 Preliminaries line 53: "stochaastic reward" -> stochastic reward
2) Define N in line 120
3) In RL, the notation $q_\pi$ is usually reserved for action-value function according to policy $\pi$.  Thus, it may be a bit disorientating for people from the RL community to see $q_\pi$ as a notation for occupancy measure. Perhaps use a different notation?  I've seen paper using $\nu$ or $d$ for defining occupancy measures. 
4) On line 244, I believe "...satisfy the condition in Theorem 2.1" should be "...Lemma 2.1".
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-04

comments: SUMMARY:
The strength of this paper is that it provides strong sample complexity results for the constrained MDP, enhancing the existing analysis in the literature by developing a new algorithm. However, despite presenting a promising method, it lacks thorough comparisons with existing methods in the literature.

STRENGTHS:
The strength of this paper is that it provides strong sample complexity results for the constrained MDP, enhancing the existing analysis in the literature by developing a new algorithm.

WEAKNESSES:
Although the paper presents a promising method, it does not present more through comparisons with existing methods in the literature.

QUESTIONS:
1) Please define the notation [K] in page 2
2) I wonder if the constraint (2) is commontly used in the literature. 
Please add some discussions on the constraint (2) and if they are used in other papers.
3) The term "problem instance hardness" is frequently used in the introdcution part, but it is not familier with the most readers in my opinion. Therefore, it is necessary to clearly define what is the problem instance hardness.
4) Although the authros develop some promising algorithms, it seems that comparison with other approaches is still weak. Therefore, it would be better if some thorough discussions with existing works is added.

LIMITATIONS:
The authors properly addressed the limitations of the paper in the document.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies reinforcement learning problem under Constrained Markov Decision Processes (CMDPs). It formulates the problem using linear programming and designs a novel algorithm to solve it. Using the newly designed algorithm, the authors prove a sample complexity of $\\tilde{O}(1/\\epsilon)$, albeit at the expense of having some additional factors.\n\nSTRENGTHS:\n- The sample complexity analyzed in this paper breaks the barrier of $O(1/\\epsilon^2)$ which is known as the lower bound for the problem that the paper studies.\n\n- The algorithm proposed in this paper is novel. Under the linear programming framework, It designs an algorithm which only focuses on the LP basis (corner points of the feasible region). The algorithm run in $O(1/\\epsilon)$ iterations and in each iteration, the order of the samples collected is independent of $\\epsilon$. Unlike some traditional methods that operate in the dual space or use primal-dual techniques, this algorithm operates directly in the primal space.\n\nWEAKNESSES:\n- It has many additional dependencies such as an additional $|\\mathcal{S}|^3$, $\\sigma$, and etc., compared to other complexity bounds.\n\n- Since the algorithm focuses on the corner points, it defines the separation gap $\\delta_1$ between the optimal corner point and the sub-optimal corner points. They are defined to ensure the estimation errors are bounded to distinguish the optimal policy from sub-optimal policies. However, since the algorithm outputs stochastic policies, it is possible that the sub-optimal policies are very close to the optimal policy. Similarly, $\\delta_2$ is the minimum gap in the dual values when some constraints are excluded. If the constraints do not change the value of the dual problem much, $\\delta_2$ will also be small. Therefore, an additional $\\delta = \\min \\{\\delta_1^2, \\delta_2^2 \\}$ term might not be worth substituting for $\\epsilon$.\n\n- In addition, from the definitions of $\\xi$ and $\\sigma$, it is very likely they will be small terms. For example, for rarely visited states in $q^*$ and small eigenvalues for $A^*$.\n\n- Due to the above concerns, it would be better to conduct experiments showing that the samples needed under such a framework are indeed fewer than those needed for other algorithms achieving $O(1/\\epsilon^2)$. However, there are no numerical experiments done in this work. Thus, it is hard to tell whether the newly proposed algorithm is more efficient or not in reality.\n\nQUESTIONS:\nCould you theoretically provide some cases to illustrate when those additional dependencies do not make the bound worse? Besides, it will better if the cases provided are not edge cases.\n\nLIMITATIONS:\nThere are no potential negative societal impact concerns for this theoretical work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}