Title: Confident Natural Policy Gradient for Local Planning in  $q_\pi$-realizable Constrained MDPs
Venue: NeurIPS 2024
Accepted: yes
Keywords: reinforcement learning

constrained MDP

sample complexity

q-pi realizability

local planning
Abstract: The constrained Markov decision process (CMDP) framework emerges as an important reinforcement learning approach for imposing safety or other critical objectives while maximizing cumulative reward. However, the current understanding of how to learn efficiently in a CMDP environment with a potentially infinite number of states remains under investigation, particularly when function approximation is applied to the value functions. In this paper, we address the learning problem given linear function approximation with $q_{\pi}$-realizability, where the value functions of all policies are linearly representable with a known feature map, a setting known to be more general and challenging than other linear settings. Utilizing a local-access model, we propose a novel primal-dual algorithm that, after $\tilde{O}(\text{poly}(d) \epsilon^{-3})$ iterations, outputs with high probability a policy that strictly satisfies the constraints while nearly optimizing the value with respect to a reward function. Here, $d$ is the feature dimension and $\epsilon > 0$ is a given  error. The algorithm relies on a carefully crafted off-policy evaluation procedure to evaluate the policy using historical data, which informs policy updates through policy gradients and conserves samples. To our knowledge, this is the first result achieving polynomial sample complexity for CMDP in the $q_{\pi}$-realizable setting.

Confident Natural Policy Gradient for Local Planning in qπ -realizable Constrained MDPs

Abstract

The constrained Markov decision process (CMDP) framework emerges as an important reinforcement learning approach for imposing safety or other critical objectives while maximizing cumulative reward. However, the current understanding of how to learn efficiently in a CMDP environment with a potentially infinite number of states remains under investigation, particularly when function approximation is applied to the value functions. In this paper, we address the learning problem given linear function approximation with q π -realizability, where the value functions of all policies are linearly representable with a known feature map, a setting known to be more general and challenging than other linear settings. Utilizing a localaccess model, we propose a novel primal-dual algorithm that, after ˜ O (poly( d ) ϵ -3 ) 1 queries, outputs with high probability a policy that strictly satisfies the constraints while nearly optimizing the value with respect to a reward function. Here, d is the feature dimension and ϵ > 0 is a given error. The algorithm relies on a carefully crafted off-policy evaluation procedure to evaluate the policy using historical data, which informs policy updates through policy gradients and conserves samples. To our knowledge, this is the first result achieving polynomial sample complexity for CMDP in the q π -realizable setting.

1 Introduction

In the classical reinforcement learning (RL) framework, optimizing a single objective above all else can be challenging for safety-critical applications like autonomous driving, robotics, and Large Language Models (LLMs). For example, it may be difficult for an LLM agent to optimize a single reward that fulfills the objective of generating helpful responses while ensuring that the messages are harmless (Dai et al., 2024). In autonomous driving, designing a single reward often requires reliance on complex parameters and hard-coded knowledge, making the agent less efficient and adaptive (Kamran et al., 2022). Optimizing a single objective in motion planning involves combining heterogeneous quantities like path length and risks, which depend on conversion factors that are not necessarily straightforward to determine (Feyzabadi and Carpin, 2014).

The constrained Markov decision process (CMDP) framework (Altman, 2021) emerges as an important RL approach for imposing safety or other critical objectives while maximizing cumulative

reward (Wachi and Sui, 2020; Dai et al., 2024; Kamran et al., 2022; Wen et al., 2020; Girard and Reza Emami, 2015; Feyzabadi and Carpin, 2014).

In addition to the single reward function optimized under a standard Markov decision process (MDP), CMDP considers multiple reward functions, with one designated as the primary reward function. The goal of a CMDP is to find a policy that maximizes the primary reward function while satisfying constraints defined by the other reward functions. Although the results of this paper can be applied to multiple constraint functions, for simplicity of presentation, we consider the CMDP problem with only one constraint function.

Our current understanding of how to learn efficiently in a CMDP environment with a potentially infinite number of states remains limited, particularly when function approximation is applied to the value functions. Most works studying the sample efficiency of a learner have focused on the tabular or simple linear CMDP setting (see related works for more details). However, there has been little work in the more general settings such as the q π -realizability, which assumes the value function of all policies can be approximated by a linear combination of a feature map with unknown parameters. Unlike Linear MDPs (Yang and Wang, 2019; Jin et al., 2020), where the transition model is assumed to be linearly representable by a feature map, q π -realizability only imposes the assumption on the existence of a feature map to represent value functions of policies.

Nevertheless, the generality of q π -realizability comes with a price, as it becomes considerably more challenging to design effective learning algorithms, even for the unconstrained settings. For the general online setting, we are only aware of one sample-efficient MDP learning algorithm (Weisz et al., 2023), which, however, is computationally inefficient. To tackle this issue, a line of research (Kearns et al., 2002; Yin et al., 2022; Hao et al., 2022; Weisz et al., 2022) applies the local-access model , where the RL algorithm can restart the environment from any visited states - a setting that is also practically motivated, especially when a simulator is provided. The local-access model is more general than the generative model (Kakade, 2003; Sidford et al., 2018; Yang and Wang, 2019; Lattimore et al., 2020; Vaswani et al., 2022), which allows visitation to arbitrary states in an MDP. The local-access model provides the ability to unlock both the sample and computational efficiency of learning with q π -realizability for the unconstrained MDP settings. However, it remains unclear whether we can harness the power of local-access for CMDP learning.

In this paper, we present a systematic study of CMDP for large state spaces, given q π -realizable function approximation in the local-access model. We summarize our contributions as follows:

We design novel, computationally efficient primal-dual algorithms to learn CMDP nearoptimal policies with the local-access model and q π -realizable function classes. The algorithms can return policies with small constraint violations or even no constraint violations and can handle model misspecification.

We provide theoretical guarantees for the algorithms, showing that they can compute an ϵ -optimal policy with high probability, making no more than ˜ O (poly( d ) ϵ -3 ) queries to the local-access model. The returned policies can strictly satisfy the constraint.

Under the misspecification setting with a misspecification error ω , we show that our algorithms achieve an ˜ O ( ω ) + ϵ sub-optimality with high probability, maintaining the same sample efficiency of ˜ O (poly( d ) ϵ -3 ) .

2 Related works

Most provably efficient algorithms developed for CMDP are in the tabular and linear MDP settings. In the tabular setting, most notably are the works by (Efroni et al., 2020; Liu et al., 2021; Zheng and Ratliff, 2020; Vaswani et al., 2022; Kalagarla et al., 2021; Yu et al., 2021; Gattami et al., 2021; HasanzadeZonuzy et al., 2021; Chen et al., 2021; Kitamura et al., 2024). Work by Vaswani et al. (2022) have showed their algorithm uses no more than ˜ O ( SA (1 -γ ) 3 ϵ 2 ) samples to achieve relaxed feasibility and ˜ O ( SA (1 -γ ) 5 ζ 2 ϵ 2 ) samples to achieve strict feasibility. Here, the γ ∈ [0 , 1) is the discount factor and ζ ∈ (0 , 1 1 -γ ] is the Slater's constant, which characterizes the size of the feasible region and hence the hardness of the CMDP. In their work, they have also provided a lower bound of

Ω ( SA (1 -γ ) 5 ζ 2 ϵ 2 ) on the sample complexity under strict feasibility. However, all the aforementioned results all scale polynomially with the cardinality of the state space.

For problems with large or possibly infinite state spaces, works by (Jain et al., 2022; Ding et al., 2021; Miryoosefi and Jin, 2022; Ghosh et al., 2024; Liu et al., 2022) have used linear function approximations to address the curse of dimensionality. All these works, except Jain et al. (2022); Liu et al. (2022), make the linear MDP assumption, where the transition function is linearly representable.

Under the generative model, for the infinite horizon discounted case, the online algorithm proposed in Jain et al. (2022) achieves a regret of ˜ O ( √ d/ √ K ) with ˜ O ( √ d/ √ K ) constraint violation, where K is the number of iterations. Work by Liu et al. (2022) is able to achieve a faster O (ln( K ) /K ) convergence rate for both the reward suboptimality and constraint violation. For the online access setting under linear MDP assumption, Ding et al. (2021); Ghosh et al. (2024) achieve a regret of ˜ O (poly( d ) poly( H ) √ T ) with ˜ O (poly( d ) poly( H ) √ T )) violations, where T is the number of episodes and H is the horizon term.

Miryoosefi and Jin (2022) presented an algorithm that achieves a sample complexity of ˜ O ( d 3 H 6 ϵ 2 ) , where d is the dimension of the feature space and H is the horizon term in the finite horizon CMDP setting. In the more general setting under q π -realizability, the best-known upper bounds are in the unconstrained MDP setting.

In the unconstrained MDP setting with access to a local-access model, early work by Kearns et al. (2002) have developed a tree-search style algorithms under this model, albeit in the tabular setting. Under v ∗ -realizability, Weisz et al. (2021) presented a planner that returns an ϵ -optimal policy using O (( dH/ϵ ) |A| ) queries to the simulator. More works by (Yin et al., 2022; Hao et al., 2022; Weisz et al., 2022) have considered the local-access model with q π -realizability assumption. Recent work by Weisz et al. (2022) have shown their algorithm can return a near-optimal policy that achieves a sample complexity of ˜ O ( d (1 -γ ) 4 ϵ 2 ) .

3 Problem formulation

Constrained MDP

We consider an infinite-horizon discounted CMDP ( S , A , P, r, c, γ, b, s 0 ) consisting a possibly countably infinite state space S with a finite set of actions A , a reward function r : S × A → [0 , 1] , a constraint function c : S × A → [0 , 1] , a discount factor γ ∈ [0 , 1) , a constraint threshold b ≥ 0 , and a fixed initial state s 0 . Let M 1 ( X ) denote the space of probability distributions supported on the set X . Then, the transition probability P : S × A → M 1 ( S ) .

Define a set of stationary randomized policies Π rand, and a policy π ∈ Π rand maps states to probability distributions over the actions (i.e., π : S → M 1 ( A ) ). Given a π ∈ Π rand, the policy π interacts with the CMDP starting from any state s ∈ S through discrete steps indexed by t ∈ N 0 , where N 0 = { 0 , 1 , 2 , . . . } . This interaction generates a trajectory of { S t , A t } t ∈ N 0 , where S 0 = s, A t ∼ π ( ·| S t ) , and S t +1 ∼ P ( ·| S t , A t ) . The reward action-value function is defined as q r π ( s, a ) = E [ ∑ ∞ t =0 γ t r ( S t , A t ) | S 0 = s, A 0 = a ] . Similarly, the constraint action-value function is defined as q c π ( s, a ) = E [ ∑ ∞ t =0 γ t c ( S t , A t ) | S 0 = s, A 0 = a ] . The reward state-value function v r π ( s ) = ⟨ π ( ·| s ) , q r π ( s, · ) ⟩ , where ⟨· , ·⟩ denotes the inner product over actions. Likewise, the constraint state-value function v c π ( s ) = ⟨ π ( ·| s ) , q c π ( s, · ) ⟩ .

The objective of the CMDP is to find a policy π that maximizes the state-value function v r π starting from a given state s 0 , while ensuring that the constraint v c π ( s 0 ) ≥ b is satisfied:

We assume the existence of a feasible solution to eq. (1) and let π ∗ denote a solution to eq. (1). A quantity unique to CMDP is the Slater's constant, which is denoted as ζ = max π v c π ( s 0 ) -b . Slater's constant characterizes the size of the feasibility region, and hence the hardness of the problem.

Because the state space can be large or possibly infinite, we use linear function approximation to approximate the values of stationary randomized policies. Let ϕ : S × A → R d be a feature map. We assume that both q r π and q c π satisfy the following condition:

Assumption 1 ( q π -realizability) There exists B > 0 and a misspecification error ω ≥ 0 such that for every π ∈ Π rand, there exists a weight vector w π ∈ R d , ∥ w π ∥ 2 ≤ B , and ensures | q π ( s, a ) -⟨ w π , ϕ ( s, a ) ⟩| ≤ ω for all ( s, a ) ∈ S × A .

A mixture policy is defined as a policy randomly selected from a finite set of policies { π 0 , · · · , π K } and executed for all subsequent steps. For example, a mixture policy ¯ π K is constructed by sampling a policy π k with probability 1 K and following it. The value function of such mixture policy for state s ∈ S is given by v ¯ π K ( s ) = 1 K ∑ K -1 k =0 v π k ( s ) , where v π k ( s ) is the value function of the individual policy π k . Note that ¯ π K is a non-stationary policy, and the set of non-stationary policies includes the set of stationary randomized policies Π rand .

We assume access to a local access model, where the agent can query the simulator only for states that have been encountered during previous simulations. Our goal is to design an algorithm that outputs a near-optimal mixture policy ¯ π K , whose performance can be characterized in one of two ways.

For a given target error ϵ > 0 , the relaxed feasibility requires the returned policy ¯ π K whose suboptimality gap v r π ∗ ( s 0 ) -v r ¯ π K ( s 0 ) is bounded by ϵ , while allowing for a small constraint violation. Formally, we require ¯ π K such that

On the other hand, strict-feasibility requires the returned policy ¯ π K whose sub-optimality gap v r π ∗ ( s 0 ) -v r ¯ π K ( s 0 ) is bounded by ϵ while not allowing any constraint violation. Formally, we require ¯ π K such that

Notations

For any real number a ∈ R , we let ⌊ a ⌋ to denote the smallest integer i such that i ≤ a . For vector x ∈ R d , let ∥ x ∥ 1 = ∑ i | x i | , ∥ x ∥ 2 = √∑ i x 2 i , and ∥ x ∥ ∞ = max i | x i | . For a positive definite matrix A ∈ R d × d , the ∥ x ∥ 2 A = x ⊤ Ax . We let proj [ a 1 ,a 2 ] ( λ ) = arg min p ∈ [ a 1 ,a 2 ] | λ -p | , and trunc [ a 1 ,a 2 ] ( y ) = min { max { y, a 1 } , a 2 } . For any two positive numbers a, b , we write a = O ( b ) if there exists an absolute constant c > 0 such that a ≤ cb . We use the ˜ O to hide any polylogarithmic terms.

4 Confident-NPG-CMDP, a local-access algorithm for CMDP

In this section, we introduce a primal-dual algorithm, which we call Confident-NPG-CMDP (see algorithm 1).

4.1 A primal-dual approach

We approach solving the CMDP problem by framing it as an equivalent saddle-point problem:

where L : Π rand × R + → R is the Lagrange function. For a policy π ∈ Π rand and a Lagrange multiplier λ ∈ R + , we have

Let ( π ∗ , λ ∗ ) be a solution to this saddle-point problem. By an equivalence to a LP formulation and strong duality (Altman, 2021), π ∗ is the policy that achieves the optimal value in the CMDP as defined in eq. (1). An optimal Lagrange multiplier λ ∗ ∈ arg min λ ≥ 0 L ( π ∗ , λ ) , Therefore, solving eq. (1) is equivalent to finding a saddle-point of the Lagrange function.

A typical primal dual algorithm that finds the saddle-point will proceed in an iterative fashion alternating between a policy update using policy gradient and a dual variable update using mirror descent. The policy gradient is computed with respect to the primal value q p π k ,λ k = q r π k + λ k q c π k and the mirror descent is computed with respect to the constraint value v c π k ( s 0 ) = ⟨ π k ( ·| s 0 ) , q c π k ( s 0 , · ) ⟩ .

Algorithm 1 Confident-NPG-CMDP

1: Input: s 0 (initial state), ϵ (target accuracy), δ ∈ (0 , 1] (failure probability); γ (discount factor) 2: Initialize:

3: Define K,η 1 , η 2 , m according to Theorem 1 for relaxed-feasibility and Theorem 2 for strictfeasibility,

4: Set L ←⌊⌊ K ⌋ / ( ⌊ m ⌋ +1) ⌋ .

5: For each iteration k ∈ { 0 , . . . , ⌊ K ⌋} : π k ← Unif( A ) , ˜ Q p k ( · , · ) ← 0 , ˜ V c k ( · ) ← 0 , and λ k ← 0 .

6: For each phase l ∈ { 0 , . . . , L +1 } : C l ← () , D l ←{}

7: For a ∈ A : if ( s 0 , a ) ̸∈ ActionCov( C 0 ) , then append ( s 0 , a ) to C 0 and set ⊥ to D 0 [( s 0 , a )] ▷ see ActionCov defined in eq. (4)

8: while True do

9: Let ℓ be the smallest integer s.t. D ℓ [ z ′ ] = ⊥ for some z ′ ∈ C ℓ

10: Let z be the first state-action pair in C ℓ s.t. D ℓ [ z ] = ⊥

11: If ℓ = L +1 , then return ¯ π K

12:

k

ℓ

ℓ

×

(

⌊

m

⌋

+1)

13: ( result, discovered ) ← Gather-data( π k ℓ , C ℓ , α, z )

14: if discovered is True then

15: Append result to C 0 and set ⊥ to D 0 [ result

16: Goto line 8

25:

26:

▷

update policy

For s, a

∈ S × A

:

28: ▷ update dual variable

30:

For

z

∈ C

ℓ

z

̸∈ C

ℓ

+1

: append

z

to

C

ℓ

+1

and set

⊥

to

D

ℓ

+1

[

z

]

▷ main loop

ℓ

▷

iteration corresponding to phase

] ▷ result is a state-action pair

Given that we do not have access to an oracle for exact policy evaluations, we must collect data to estimate the primal and constraint values. If we have the least-squares estimates of q r π k and q c π k , denoted by Q r k and Q c k , respectively, then we can compute the least-squares estimate Q p k = Q r k + λ k Q c k to be the estimate of the primal value q p π k ,λ k . Additionally, we can compute V c k ( s 0 ) = ⟨ π k ( ·| s 0 ) , Q c k ( s 0 , · ) ⟩ to be the least-squares estimate of the constraint value v c π k ( s 0 ) . Then, for any given ( s, a ) ∈ S × A , our algorithm makes a policy update of the following form:

followed by a dual variable update of the following form:

where the η 1 and η 2 are the step-sizes.

4.2 Core set and least square estimates

To construct the least-squares estimates, let us assume for now that we are given a set of state-action pairs, which we call the core set C . By organizing the feature vector of each state-action pair in C row-wise into a matrix Φ C ∈ R |C|× d , we can write the covariance matrix as V ( C , α ) = Φ ⊤ C Φ C + αI . For each ( s, a ) ∈ C , suppose we have run Monte Carlo rollouts using the rollout policy π with the local access simulator to obtain an averaged Monte Carlo return denoted by ¯ q ( s, a ) . Then we gather all the state-action pairs into a vector ¯ q ∈ R |C| . For any state-action pair ( s, a ) ∈ S × A , the least-square estimate of action-value q π is defined to be

Since the algorithm can only rely on estimates for policy improvement and constraint evaluation, it is imperative that these estimates closely approximate their true action values. In the local access setting, an algorithm may not be able to visit all state-action pairs, so we cannot guarantee that the estimates will closely approximate the true action values for all state-action pairs. However, we can ensure the accuracy of the estimates for a subset of states.

Given C , let us define a set of state-action pairs whose features satisfies the condition ∥ ϕ ( s, a ) ∥ V ( C ,α ) -1 ≤ 1 , then we call this set the action-cover of C :

Following from the action-cover, we have the cover of C . For a state s to be in the cover of C , all its actions a ∈ A , the pair ( s, a ) is in the action-cover of C . In other words,

For any s ∈ Cov( C ) , we can ensure the least square estimate Q ( s, a ) defined by eq. (3) closely approximates its true action value q π ( s, a ) for all a ∈ A . However, such a core set C is not available before the algorithm is run. Therefore, we need an algorithm that will build a core set incrementally in the local-access setting while planning. To achieve this, we build our algorithm on CAPI-QPI-Plan (Weisz et al., 2022), using similar methodology for core set building and data gathering.

4.3 Core set building and data gathering to control the accuracy of the least-square estimates

Confident-NPG-CMDP does not collect data in every iteration but collects data in interval of m = O ( ln(1 + ρ 0 ) poly( ϵ -1 (1 -γ ) -1 ) ) , where ρ 0 ≥ 0 is an user defined constant. During each data collection phase, the algorithm performs on-policy evaluation. Between these phases, it conducts ( ⌊ m ⌋ +1) off-policy evaluations, reusing data from the most recent on-policy iteration.

By setting ρ 0 to a positive value, we impose an upper bound of 1+ ρ 0 on the per-trajectory importance sampling ratio used in off-policy evaluations, and m is adjusted accordingly to maintain this bound. The total number of data collection phases is L = ⌊⌊ K ⌋ / ( ⌊ m ⌋ +1) ⌋ , where K is the total number of iterations. When ρ 0 is set to zero, we have L = K , resulting in a purely on-policy version of the algorithm.

Confident-NPG-CMDP maintains a set of core sets {C l } L +1 l =0 , one for each data collection phases. Each core set C l is a list of state-action pairs. Due to the off-policy evaluations, Confident-NPGCMDP also maintains a set of data sets { D l } L l =0 . Initially, all core sets are empty, all policies are initialized to the uniform policy, and all data sets are empty.

The algorithm begins by adding the feature vectors corresponding to ( s 0 , a ) for all actions a ∈ A that are not in the action-cover of C 0 . These feature vectors are considered informative. For every ( s, a ) ∈ C 0 , the algorithm adds an entry to D 0 and sets its value to the placeholder ⊥ , indicating that there is no roll-out data yet. Then, in line 9 of algorithm 1, the algorithm finds the smallest integer l ∈ { 0 , . . . , L } such that the corresponding D l has an entry without roll-out data (i.e., it contains the placeholder ⊥ ). When such a phase is found, a running phase begins, denoted by ℓ in algorithm 1. We note that when ℓ = L +1 , the algorithm returns and no roll-outs are stored.

Since only one running phase ℓ can be active at a time, and ℓ can only take value l ∈ { 0 , . . . , L } , the algorithm updates the policies of the corresponding iterations in line 27, updates the dual variables of these iterations in line 29, and extends the core set for the next phase in line 30.

Suppose during a running phase with ℓ = l , while performing the roll-out in Gather-data subroutine (algorithm 3 in Appendix A), if any state-action pair ( s, a ) ∈ S × A is not in the action-cover of C ℓ , the current running phase stops and the newly discovered state-action pair is added to C 0 in line 15. The same state-action pair is then propagated to C 1 and so on by line 30.

Once a state-action pair is added to a core set by line 7, line 15, and line 30, it remains in that core set for the duration of the algorithm. This means that any C l , l ∈ { 0 , . . . , L +1 } can grow in size and be extended multiple times during the execution of the algorithm. When any new state-action pair is added to a core set, the least-square estimate should be recomputed with the newly added information. This implies that the policy needs to be updated and data re-collected. However, we can avoid restarting the entire data collection procedure by updating only the policy for states that are newly added to the extended core set. We elaborate on this approach further in the next paragraph.

When the algorithm enters the running phase ℓ = l , and the Gather-data subroutine returns, the LSE subroutine (algorithm 4) computes the least-squares estimate Q r k , Q c k using the most recently extended core set C ℓ for each corresponding iteration k = k ℓ , . . . , k ℓ +1 -1 . Subsequently, ˜ Q p k of line 23 of algorithm 1 is updated with the newly updated least-square estimates Q r k , Q c k . However, the policy π k +1 will only be updated for states that are newly covered by C ℓ (i.e., s ∈ Cov( C ℓ ) \ Cov( C ℓ +1 ) ). For any states that are already covered by C ℓ (i.e., s ∈ Cov( C ℓ +1 ) ), the policy remains unchanged from its previous update using the ˜ Q p at that time. By updating the policy in this manner, the accuracy guarantee of ˜ Q p k ( s, a ) with respect to q p π k ,λ k ( s, a ) is ensured not just for π k , but for an extended set of policies defined as follows:

Definition 1 For any policy π from the set of randomized policies Π rand and any subset X ⊆ S , the extended set of policies is defined as:

By maintaining a set of core sets, gathering data via the Gather-data subroutine (algorithm 3 in Appendix A), making policy updates by line 27, and dual variable updates by line 29, we have:

Lemma 1 Whenever LSE subroutine in line 21 of Confident-NPG-CMDP is executed during a running phase ℓ = l for l ∈ { 0 , . . . , L } , the least-square estimate ˜ Q p k ( s, a ) satisfies the following condition for all iterations k = k ℓ , . . . , k ℓ +1 -1 associated with this phase and for all s ∈ Cov( C ℓ ) and a ∈ A ,

where ϵ ′ = (1+ U )( ω + √ αB +( ω + ϵ ) √ ˜ d ) with ˜ d = ˜ O ( d ) and U is an upper bound on the optimal Lagrange multiplier. Similarly, for initial state s 0 , we have

The accuracy guarantee of eq. (5) and eq. (6) are maintained throughout the execution of the algorithm. By lemma 4.5 of Weisz et al. (2022) (restated in lemma 6 in Appendix A), for any past version C past l

of C l and the corresponding policy π past k associated with C past l , we have Π π k , Cov( C l ) ⊆ Π π past k , Cov( C past l ) . This means that if eq. (5) and eq. (6) hold true for any policy in Π π past k , Cov( C past l ) , they will also hold true for any future updated policy π k .

4.4 Differences between Confident-NPG-CMDP and CAPI-QPI-Plan

CAPI-QPI-Plan is designed for unconstrained MDPs and returns a deterministic policy, which may not be feasible in the constrained setting. In contrast, Confident-NPG-CMDP returns a soft mixture policy ¯ π K , ensuring that ¯ π K ( a | s ) > 0 for all ( s, a ) ∈ S × A .

In constrained MDPs, controlling the dual variable via mirror descent adds an ϵ -2 factor to the sample complexity. Directly applying CAPI-QPI-Plan would increase the complexity to ˜ O ( ϵ -4 ) due to the need to manage both the dual variable and estimation error. To address this, Confident-NPG-CMDP employs the natural policy gradient for policy improvement and leverages the softmax policy structure to perform off-policy estimation, thereby reducing the complexity to ˜ O ( ϵ -3 ) .

By employing a per-trajectory importance sampling ratio, we weigh the Monte Carlo returns generated from data collected in earlier on-policy phases, resulting in unbiased estimates of action values with respect to the target policy. However, this ratio can become large if there is a substantial difference between the on-policy and target policies. To mitigate this, the algorithm collects data at intervals of m , effectively determining when to gather new data as the policy significantly diverges from an earlier recent data-gathering iteration. By setting ρ 0 > 0 , we can bound the per-trajectory importance sampling ratio, thus controlling the interval m for resampling on-policy data to produce well-controlled estimators.

Key algorithmic differences between Confident-NPG-CMDP and CAPI-QPI-Plan:

Policy Improvement Step: Confident-NPG-CMDP utilizes a softmax over the estimated action-values, whereas CAPI-QPI-Plan employs a greedy approach.

Dual Variable Computation: Confident-NPG-CMDP requires computation of the dual variable inherent in primal-dual algorithms.

Data Sampling Strategy: Unlike CAPI-QPI-Plan, Confident-NPG-CMDP does not sample data at every iteration but collects data at specific intervals to control the importance sampling ratio.

In the next two sections, we will demonstrate how these changes ensure a feasible mixture policy for the CMDP and address the additional analytical challenges.

5 Confident-NPG-CMDP satisfies relaxed-feasibility

With the accuracy guarantee of the least-square estimates, we prove that at the termination of Confident-NPG-CMDP, the returned mixture policy ¯ π K satisfies relaxed-feasibility. We note that because of the execution of line 30 in algorithm 1, at termination, one can show using induction that C 0 = C 1 = · · · = C L +1 . Therefore, Cov( C 0 ) = Cov( C 1 ) = · · · = Cov( C L ) . Thus, it is sufficient to only consider C 0 at the termination of the algorithm. By line 7 of algorithm 1, we have ensured s 0 ∈ Cov( C 0 ) .

By employing the primal-dual approach discussed in section 4, we reduce the CMDP problem to an unconstrained problem with a single reward function of the form r λ = r + λc . Therefore, we can apply lemma 12 from the Confident-NPG algorithm in the single-reward setting (see Appendix A) to our Confident-NPG-CMDP algorithm, replacing π with π ∗ . Consequently, the value difference between π ∗ and ¯ π K can be bounded, which leads to:

Lemma 2 Let δ ∈ (0 , 1] be the failure probability, ϵ > 0 be the target accuracy, and s 0 be the initial state. Assuming for all s ∈ Cov( C 0 ) and all a ∈ A , | ˜ Q p ( s, a ) -q p ′ ( s, a ) | ≤ ϵ ′ and

Confident-NPG-CMDP returns a mixture policy ¯ π K that satisfies the following,

where ϵ ′ = (1 + U )( ω +( √ αB +( ω + ϵ ) √ ˜ d )) with ˜ d = ˜ O ( d ) , and U is an upper bound on the optimal Lagrange multiplier.

By setting the parameters to appropriate values, it follows from lemma 2 that we obtain the following result:

Theorem 1 With probability 1 -δ , the mixture policy ¯ π K returned by confident-NPG-CMDP ensures that

if we choose n = ˜ O ( ϵ -2 ζ -2 (1 -γ ) -4 d ) , α = O ( ϵ 2 ζ 2 (1 -γ ) 4 ) , K = ˜ O ( ϵ -2 ζ -2 (1 -γ ) -6 ) , η 1 = ˜ O ( (1 -γ ) 2 ζK -1 / 2 ) , η 2 = ζ -1 K -1 / 2 , H = ˜ O ( (1 -γ ) -1 ) , m = ˜ O ( ϵ -1 ζ -2 (1 -γ ) -2 ) , and L = ⌊ K/ ( ⌊ m ⌋ +1) ⌋ = ˜ O ( ϵ -1 (1 -γ ) -4 ) total number of data collection phases.

Furthermore, the algorithm utilizes at most ˜ O ( ϵ -3 ζ -3 d 2 (1 -γ ) -11 ) queries in the local-access setting.

Remark 1: In the presence of misspecification error ω > 0 , the reward suboptimality and constraint violation is ˜ O ( ω ) + ϵ with the same sample complexity.

Remark 2: Suppose the Slater's constant ζ is much smaller than the suboptimality bound of ˜ O ( ω ) + ϵ , and it is reasonable to set ζ = ϵ . Then, the sample complexity is ˜ O ( ϵ -6 (1 -γ ) -11 d 2 ) , which is independent of ζ .

Remark 3: Our algorithm requires the Slater's constant ζ , which can be estimated by running CAPI-QPI-Plan only on the constraint function c , treating it as an unconstrained optimization problem. This yields an approximation of max π v c π ( s 0 ) , allowing us to estimate ζ . Performing this estimation before executing Confident-NPG-CMDP adds only an additive term to the overall sample complexity.

6 Confident-NPG-CMDP satisfies strict-feasibility

To address the strict feasibility problem, where no constraint violations are permitted (i.e., v c ¯ π K ≥ b ), the algorithm must solve a more conservative CMDP. We define a surrogate CMDP with the tuple ( S , A , P, r, c, γ, b ′ , s 0 ) , where b ′ = b + ∆ for some ∆ ≥ 0 . Note that b ′ ≥ b , imposing stricter constraints than the original problem. The optimal policy of this surrogate CMDP ensures compliance with the original constraint and is defined as follows:

Notice that π ∗ △ is a more conservative policy than π ∗ , where π ∗ is the optimal policy of the original CMDPobjective eq. (1). By solving this surrogate CMDP using Confident-NPG-CMDP and applying the result of theorem 1, we obtain a ¯ π K that would satisfy

where ¯ ϵ = ˜ O ( ω ) + ϵ . Expanding out b ′ , we have v c ¯ π K ( s 0 ) ≥ b + △-¯ ϵ . If we can set △ such that △-¯ ϵ ≥ 0 , then v c ¯ π K ( s 0 ) ≥ b , which satisfies strict-feasibility. We show this formally in the next theorem, where △ = O ( ϵ (1 -γ ) ζ ) and is incorporated into the algorithmic parameters for ease of presentation.

Theorem 2 With probability 1 -δ , a target ϵ > 0 , the mixture policy ¯ π K returned by confident-NPG-CMDP ensures that v r π ∗ ( s 0 ) -v r ¯ π K ( s 0 ) ≤ ϵ and v c ¯ π K ( s 0 ) ≥ b , if assuming the misspecification error ω ≤ ϵζ 2 (1 -γ ) 3 (1 + √ ˜ d ) -1 , and if we choose α = O ( ϵ 2 ζ 3 (1 -γ ) 5 ) , K = ˜ O ( ϵ -2 ζ -4 (1 -γ ) -8 ) , n = ˜ O ( ϵ -2 ζ -4 (1 -γ ) -8 d ) , H = ˜ O ( (1 -γ ) -1 ) , m = ˜ O ( ϵ -1 ζ -2 (1 -γ ) -3 ) , and L = ⌊ K/ ( ⌊ m ⌋ +1) ⌋ = ˜ O (( ϵ -1 ζ -2 (1 -γ ) -5 )) total data collection phases.

Furthermore, the algorithm utilizes at most ˜ O ( ϵ -3 ζ -6 (1 -γ ) -14 d 2 ) queries in the local-access setting.

Remark 1: We note that by solving this conservative CMDP incurs a higher sample complexity, necessitating a separate treatment for this setting. Additionally, in the presence of a misspecification error ω > 0 , the strict-feasibility setting requires additional assumptions on ω , whereas the relaxed-feasibility setting does not. The sample complexity of the relaxed-feasibility setting can be independent of Slater's constant, whereas for strict feasibility, the returned policy must strictly adhere to constraints, and we cannot simply set Slater's constant ζ to ϵ and disregard its impact.

7 A discussion on memory cost and some implementation details

The overall memory requirement is ˜ dnH ( L +1) + ˜ d +( L +1)( m +1) ˜ dd . The term ˜ dnH ( L +1) comes from maintaining L + 1 copies of the core sets, and each core set contains no more than ˜ d state-action pairs. For each state-action pair in C l for l ∈ { 0 , . . . , L } , the algorithm stores n trajectories consisting of H tuples ( s, a, r, c ) .

In phase L +1 , the algorithm terminates, so no roll-outs are stored. The second term ˜ d accounts for the elements stored in C L +1 , which has no more than ˜ d elements.

Finally, the last term is the memory required to store the least-square weights of the estimator during core set extensions. Each core set C l can undergo up to ˜ d extensions. Recall that one state-action pair is added to C 0 at a time, and subsequently propagated to C 1 , C 2 and so on, ensuring that each core set contains no more than ˜ d elements. During every extension of C l , the newly added state-action pairs are marked, and up to m +1 least-square weights are stored to account for the corresponding iterations associated with C l . Since each weight vector has a dimension d , and there are L +1 core sets maintained this manner, the total memory required to store all least-square weights is bounded by ( L +1)( m +1) ˜ dd .

We store the least-squares weights because the algorithm must return a mixture policy, which requires access to all policies π 0 , . . . , π K -1 . Instead of storing each π k for k = 0 , . . . , K -1 across the entire state-action space, the algorithm tracks the state-action pairs newly added to each core set during extensions and saves their corresponding least-squares weights for each extension. With this stored information and the initialization of π 0 , a subroutine can reconstruct the policies π k ( ·| s ) for any s and iteration k as needed. Please refer to Appendix E for a brief discussion on how to mark the state-action pairs, store the least-square weights, and use this information to reconstruct the policies as required.

8 Conclusion

We have presented a primal-dual algorithm for planning in CMDP with large state spaces, given q π -realizable function approximation. The algorithm, with high probability, returns a policy that achieves both the relaxed and strict feasibility CMDP objectives, using no more than ˜ O ( ϵ -3 d 2 poly( ζ -1 (1 -γ ) -1 )) queries to the local-access simulator.

Our algorithm does not query the simulator and collect data in every iteration. Instead, the algorithm queries the simulator only at fixed intervals. Between these data collection intervals, our algorithm improves the policy using off-policy optimization. This approach makes it possible to achieve the desired sample complexity in both feasibility settings.

Acknowledgments and Disclosure of Funding

Tian Tian would like to thank Roshan Shariff and Kenny Young for their insightful comments and helpful feedback during the preparation of this manuscript. Csaba Szepesvári also gratefully acknowledges funding from the Canada CIFAR AI Chairs Program, Amii, and NSERC. Lin Yang is supported in part by NSF # 2221871 and an Amazon Faculty Award.

References

Altman, E. (2021). Constrained Markov decision processes . Routledge.

Chen, Y., Dong, J., and Wang, Z. (2021). A primal-dual approach to constrained markov decision processes. arXiv preprint arXiv:2101.10895 .

Dai, J., Pan, X., Sun, R., Ji, J., Xu, X., Liu, M., Wang, Y., and Yang, Y. (2024). Safe RLHF: Safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations .

Ding, D., Wei, X., Yang, Z., Wang, Z., and Jovanovic, M. (2021). Provably efficient safe exploration via primal-dual policy optimization. In International Conference on Artificial Intelligence and Statistics , pages 3304-3312. PMLR.

Efroni, Y., Mannor, S., and Pirotta, M. (2020). Exploration-exploitation in constrained mdps. CoRR , abs/2003.02189.

Feyzabadi, S. and Carpin, S. (2014). Risk-aware path planning using hirerachical constrained markov decision processes. In 2014 IEEE International Conference on Automation Science and Engineering (CASE) , pages 297-303.

Gattami, A., Bai, Q., and Aggarwal, V. (2021). Reinforcement learning for constrained markov decision processes. In International Conference on Artificial Intelligence and Statistics , pages 2656-2664. PMLR.

Ghosh, A., Zhou, X., and Shroff, N. (2024). Towards achieving sub-linear regret and hard constraint violation in model-free rl. In International Conference on Artificial Intelligence and Statistics , pages 1054-1062. PMLR.

Girard, J. and Reza Emami, M. (2015). Concurrent markov decision processes for robot team learning. Engineering Applications of Artificial Intelligence , 39:223-234.

Hao, B., Lazic, N., Yin, D., Abbasi-Yadkori, Y., and Szepesvari, C. (2022). Confident least square value iteration with local access to a simulator. In International Conference on Artificial Intelligence and Statistics , pages 2420-2435. PMLR.

HasanzadeZonuzy, A., Kalathil, D., and Shakkottai, S. (2021). Model-based reinforcement learning for infinite-horizon discounted constrained markov decision processes. IJCAI 2021 .

Jain, A., Vaswani, S., Babanezhad, R., Szepesvari, C., and Precup, D. (2022). Towards painless policy optimization for constrained mdps. In Uncertainty in Artificial Intelligence , pages 895-905. PMLR.

Jin, C., Yang, Z., Wang, Z., and Jordan, M. I. (2020). Provably efficient reinforcement learning with linear function approximation. In Conference on learning theory , pages 2137-2143. PMLR.

Kakade, S. M. (2003). On the sample complexity of reinforcement learning . University of London, University College London (United Kingdom).

Kalagarla, K. C., Jain, R., and Nuzzo, P. (2021). A sample-efficient algorithm for episodic finitehorizon mdp with constraints. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 8030-8037.

Kamran, D., Simão, T. D., Yang, Q., Ponnambalam, C. T., Fischer, J., Spaan, M. T., and Lauer, M. (2022). A modern perspective on safe automated driving for different traffic dynamics using constrained reinforcement learning. In 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC) , pages 4017-4023. IEEE.

Kearns, M., Mansour, Y., and Ng, A. Y. (2002). A sparse sampling algorithm for near-optimal planning in large markov decision processes. Machine learning , 49:193-208.

Kitamura, T., Kozuno, T., Kato, M., Ichihara, Y., Nishimori, S., Sannai, A., Sonoda, S., Kumagai, W., and Matsuo, Y. (2024). A policy gradient primal-dual algorithm for constrained mdps with uniform pac guarantees. arXiv preprint arXiv:2401.17780 .

Lattimore, T. and Szepesvári, C. (2020). Bandit algorithms . Cambridge University Press.

Lattimore, T., Szepesvari, C., and Weisz, G. (2020). Learning with good feature representations in bandits and in rl with a generative model. In International conference on machine learning , pages 5662-5670. PMLR.

Liu, T., Zhou, R., Kalathil, D., Kumar, P. R., and Tian, C. (2021). Learning policies with zero or bounded constraint violation for constrained mdps. CoRR , abs/2106.02684.

Liu, T., Zhou, R., Kalathil, D., Kumar, P. R., and Tian, C. (2022). Policy optimization for constrained mdps with provable fast global convergence.

Miryoosefi, S. and Jin, C. (2022). A simple reward-free approach to constrained reinforcement learning. In International Conference on Machine Learning , pages 15666-15698. PMLR.

Sidford, A., Wang, M., Wu, X., Yang, L., and Ye, Y. (2018). Near-optimal time and sample complexities for solving markov decision processes with a generative model. Advances in Neural Information Processing Systems , 31.

Vaswani, S., Yang, L., and Szepesvári, C. (2022). Near-optimal sample complexity bounds for constrained mdps. Advances in Neural Information Processing Systems , 35:3110-3122.

Wachi, A. and Sui, Y. (2020). Safe reinforcement learning in constrained markov decision processes. In International Conference on Machine Learning , pages 9797-9806. PMLR.

Weisz, G., Amortila, P., Janzer, B., Abbasi-Yadkori, Y., Jiang, N., and Szepesvári, C. (2021). On query-efficient planning in mdps under linear realizability of the optimal state-value function. In Conference on Learning Theory , pages 4355-4385. PMLR.

Weisz, G., György, A., and Szepesvári, C. (2022). Confident approximate policy iteration for efficient local planning in q pi realizable mdps. arXiv preprint arXiv , 2210.

Weisz, G., György, A., and Szepesvári, C. (2023). Online rl in linearly q π -realizable mdps is as easy as in linear mdps if you learn what to ignore.

Wen, L., Duan, J., Li, S. E., Xu, S., and Peng, H. (2020). Safe reinforcement learning for autonomous vehicles through parallel constrained policy optimization. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC) , pages 1-7.

Yang, L. and Wang, M. (2019). Sample-optimal parametric q-learning using linearly additive features. In International conference on machine learning , pages 6995-7004. PMLR.

Yin, D., Hao, B., Abbasi-Yadkori, Y., Lazi´ c, N., and Szepesvári, C. (2022). Efficient local planning with linear function approximation. In International Conference on Algorithmic Learning Theory , pages 1165-1192. PMLR.

Yu, T., Tian, Y., Zhang, J., and Sra, S. (2021). Provably efficient algorithms for multi-objective competitive rl. In International Conference on Machine Learning , pages 12167-12176. PMLR.

Zheng, L. and Ratliff, L. (2020). Constrained upper confidence reinforcement learning. In Bayen, A. M., Jadbabaie, A., Pappas, G., Parrilo, P. A., Recht, B., Tomlin, C., and Zeilinger, M., editors, Proceedings of the 2nd Conference on Learning for Dynamics and Control , volume 120 of Proceedings of Machine Learning Research , pages 620-629. PMLR.

A Confident-NPG in a single reward setting

The pseudo code of Confident-NPG with a single reward setting is the same as Confident-NPG-CMDP in algorithm 1, except that line 24 and line 29 will not appear in Confident-NPG. Additionally, the LSE subroutine returns just Q r , and the policy update will be with respect to ˜ Q r . For the complete pseudo code of Confident-NPG in the single reward setting, please see algorithm 2. In the following analysis, for convenience, we omit the superscript r .

Algorithm 2 Confident-NPG Input: s 0 (initial state), ϵ (target accuracy), δ ∈ (0 , 1] (failure probability), c ≥ 0 , γ , K = 2 ln( |A| ) (1 -γ ) 4 ϵ 2 , η 1 = (1 -γ ) √ 2 ln( |A| ) K , m = ln(1+ ρ 0 ) 2 ϵ (1 -γ ) ln ( 4 ϵ (1 -γ ) 2 ) , L = ⌊⌊ K ⌋ / ( ⌊ m ⌋ +1) ⌋ . Initialize: for each iteration k ∈ { 0 , . . . , ⌊ K ⌋} : π k ← Uniform ( A ) , ˜ Q r k ( s, a ) ← 0 , for all s, a ∈ S × A , and λ k ← 0 . For each phase l ∈ { 0 , . . . , L +1 } : C l ← () , D l ←{} 1: for a ∈ A do 2: if ( s 0 , a ) ̸∈ ActionCov( C 0 ) then 3: Append ( s 0 , a ) to C 0 ; D 0 [( s 0 , a )] ←⊥ 4: while True do ▷ main loop 5: Get the smallest integer ℓ s.t. D ℓ [ z ′ ] = ⊥ for some z ′ ∈ C ℓ 6: Get the first state-action pair z in C ℓ s.t. D ℓ [ z ] = ⊥ 7: if ℓ = L +1 then return ¯ π K 8: k ℓ ← ℓ × ( ⌊ m ⌋ +1) ▷ iteration corresponding to phase ℓ 9: ( result, discovered ) ← Gather-data( π k ℓ , C ℓ , α, z ) 10: if discovered is True then 11: ▷ result is a state-action pair 12: Append result to C 0 ; D 0 [ result ] ←⊥ 13: break 14: ▷ result is a set of n H -horizon trajectories ∼ π k ℓ starting at z 15: D ℓ [ z ] ← result 16: if ̸ ∃ z ′ ∈ C ℓ s.t. D ℓ [ z ′ ] = ⊥ then 17: k ℓ +1 ← k ℓ +( ⌊ m ⌋ +1) if k ℓ +( ⌊ m ⌋ +1) ≤ ⌊ K ⌋ otherwise ⌊ K ⌋ 18: ▷ update policy for every k ∈ [ k ℓ , k ℓ +1 -1] using C ℓ , D ℓ 19: for k = k ℓ , . . . , k ℓ +1 -1 do 20: Q r k , _ ← LSE ( C ℓ , D ℓ , π k , π k ℓ ) 21: ▷ update variables and improve policy 22: for all s ∈ Cov( C ℓ ) \ Cov( C ℓ +1 ) , and for all a ∈ A do 23: ˜ Q r k ( s, a ) ← Trunc [ 0 , 1 1 -γ ] Q r k ( s, a ) 24: for all s, a ∈ S × A do 25: π k +1 ( a | s ) ← { π k +1 ( a | s ) if s ∈ Cov( C ℓ +1 ) π k ( a | s ) exp( η 1 ˜ Q r k ( s,a )) ∑ a ′ ∈A π k ( a ′ | s ) exp( η 1 ˜ Q r k ( s,a ′ )) otherwise 26: for z ∈ C ℓ s.t. z ̸∈ C ℓ +1 do 27: Append z to C ℓ +1 ; D ℓ +1 [ z ] ←⊥

A.1 The Gather-data subroutine

Given a core set C , a behaviour policy µ , a starting state-action pair ( s, a ) ∈ S × A along with some algorithmic parameters, the Gather-data subroutine (algorithm 3) will either 1) return a newly

discovered state-action pair, or 2) return a set of n trajectories. Each trajectory is generated by running the behaviour policy µ with the simulator for H consecutive steps. For i = 1 , . . . , n , let τ i s,a denote the i th trajectory starting from s, a to be { S i 0 = s, A i 0 = a, R i 1 , C i 1 , · · · , S i H -1 , A i H -1 , R i H , C i H , S i H } . Then the i -th discounted cumulative rewards G ( τ i s,a ) = ∑ H -1 h =0 γ h R i h +1 . For a target policy π , then the empirical mean of the discounted sum of rewards is as follows,

where ρ ( τ i s,a ) = Π H -1 h =1 π ( A i h | S i h ) µ ( A i h | S i h ) is the per-trajectory importance sampling ratio.

For some given ¯ s and ¯ a , we establish the following relationship between the target policy π and the behavior policy µ :

where f (¯ s, ¯ a ) : S × A → R + and ρ 0 ≥ 0 is a given constant. By establishing the relationship stated in eq. (9), the importance sampling ratio ρ ( τ i s,a ) can be bounded by 1 + ρ 0 as this is proven in the following lemma:

Lemma 3 Suppose the trajectory τ = ( S 0 , A 0 , R 1 , S 1 , A 1 , · · · , S H -1 , A H -1 , R H ) is sampled from a behaviour policy µ , and µ is related to the target policy π via eq. (9) . The per-trajectory importance sampling ratio

Proof: Let l 0 = sup s,a | f ( s, a ) | , where f is defined in eq. (9). For any ( s, a ) ∈ { ( S h , A h ) } H -1 h =1 ,

We see that π ( a | s ) µ ( a | s ) ≤ exp(2 l 0 ) , and it follows that Π H -1 h =1 π ( A h | S h ) µ ( A h | S h ) ≤ exp(2 l 0 H ) . By assumption, l 0 ≤ ln(1+ ρ 0 ) 2 H , then exp(2 l 0 H ) ≤ exp ( 2 H ln(1+ ρ 0 ) 2 H ) ≤ 1 + ρ 0 . ■

Define the probability distribution P π,s,a over trajectory { S h , A h , R h +1 } h ≥ 0 as follows: the initial state S 0 is set deterministically to s , and the initial action A 0 is set deterministically to a . For each subsequent time step h ≥ 0 , the next state S h +1 is sampled according to the transition probability P ( ·| S h , A h ) , and the next action A h +1 is sampled from the policy π ( ·| S h +1 ) . It follows that E π,s,a denotes the expectation with respect to distribution P π,s,a .

Now, we show that for all ( s, a ) ∈ C , | ¯ q ( s, a ) -q π ( s, a ) | ≤ ϵ , where ϵ > 0 is a given target error. Additionally, the accuracy guarantee of | ¯ q ( s, a ) -q π ( s, a ) | ≤ ϵ continues to holds for the extended set of policies defined in definition 1. Formally, we state the main result of this section.

Lemma 4 For any s, a ∈ S × A , X ⊂ S , the Gather-data subroutine will either return with (( s ′ , a ′ ) , True ) for some s ′ ̸∈ X , or it will return with ( D [( s, a )] , False ) , where D [( s, a )] is a set of n independent trajectories generated by a behavior policy µ starting from ( s, a ) . When Gather-data returns False for ( s, a ) , we assume 1) the behavior policy µ and target policy π for all the states and actions encountered in the trajectories stored in D [( s, a )] satisfy eq. (9) and 2) ¯ q ( s, a ) is an unbiased estimate of E π ′ ,s,a [ ∑ H -1 h =0 γ h R h +1 ] for all π ′ ∈ Π π, X . Then, the importance-weighted return ¯ q ( s, a ) constructed from D [( s, a )] according to eq. (8) will, with probability 1 -δ ′ ,

Proof: The proof follows similar reasoning to Lemma 4.2 Weisz et al. (2022).

Recall D [( s, a )] stores n number of trajectories indexed by i , where each trajectory τ i s,a = ( S i 0 = s, A i 0 = a, R i 0 , . . . , S i H -1 ) ∼ µ . The per-trajectory importance sampling ratio ρ ( τ i s,a ) = Π H -1 h =1 π ( A i h | S i h ) µ ( A i h | S i h ) , and the discounted cumulative return is ∑ H -1 h =0 γ h R i h +1 . By the triangle inequality,

The goal is to bound each of the two terms in eq. (10) by ϵ 4 so that the sum of the two is ϵ 2 .

By assumption, the policies π and µ satisfies eq. (9) for all state-action pairs ( S i h , A i h ) extracted from the i -trajectory τ i s,a . Second, ¯ q ( s, a ) is assumed to be an unbiased estimate of E π,s,a [ ∑ H -1 h =0 γ h R h +1 ] . Note that for all i = 1 , . . . , n , the importance weighted cumulative return ρ ( τ i s,a ) ∑ H -1 h =0 γ h R i h +1 are independent random variables, and the value of each such random variable ∈ [ 0 , 1+ ρ 0 1 -γ ] . This is because 1) ∑ H -1 h =0 γ h R i h +1 ≤ 1 1 -γ since the rewards take values in the range of [0 , 1] , and 2) ρ ( τ i s,a ) ≤ 1 + ρ 0 by lemma 3. We apply Hoeffding's inequality,

Then, we have with probability 1 -δ ′ / 2 , where δ ′ = 2exp ( -2 nϵ 2 16 ( 1+ ρ 0 1 -γ ) 2 ) , the first term in eq. (10) | 1 n ∑ n i =1 ρ ( τ i s,a ) ∑ H -1 h =0 γ h R i h +1 -E π,s,a ∑ H -1 h =0 γ h R h +1 | ≤ ϵ 4 . For the second term in eq. (10),

By the choice of H = ln(4 / (1 -γ ) ϵ ) 1 -γ , we have γ H 1 -γ ≤ ϵ 4 . Putting everything together, we get | ¯ q ( s, a ) -q π ( s, a ) | ≤ ϵ 2 . To get the final result, we need to upper bound | q π ( s, a ) -q π ′ ( s, a ) | by ϵ 2 , so that | ¯ q ( s, a ) -q π ′ ( s, a ) | ≤ | ¯ q ( s, a ) -q π ( s, a ) | + | q π ( s, a ) -q π ′ ( s, a ) | ≤ ϵ .

Recall that π and π ′ differs in distributions over states that are not in X . For a trajectory ( S 0 = s, A 0 = a, S 1 , . . . ) , let T be the smallest positive integer such that S T ̸∈ X , then the distribution of the trajectory ( S 0 = s, A 0 = a, S 1 , . . . , S T ) are the same under P π,s,a and P π ′ ,s,a because π ( ·| s ) = π ′ ( ·| s ) for all s ∈ X . Then,

Recall P π,s,a (1 ≤ T < H ) = ∑ H -1 t =1 ∑ s ′ ∈X ,a ′ ∈A P π,s,a ( S t -1 = s ′ , A t -1 = a ′ ) P ( S T | s ′ , a ′ ) , and recall S 0 = s, A 0 = a , then by the law of total probability, we have

To get eq. (11), we use lemma 3 and that 1 ≤ t ≤ H -1 .

Altogether, we have

Now, we bound P µ,s,a (1 ≤ T < H ) . For each ( s, a ) ∈ X , in the i th rollouts, let I i ( s, a ) be an indicator function, where it takes the value 1 when the event that S T ̸∈ X occurs during 1 ≤ T < H . Then E µ,s,a [ I i ( s, a )] = P µ,s,a (1 ≤ T < H ) . By another Hoeffding's inequality,

Then, with probability 1 -δ ′ / 2 ,

When Gather-data subroutine returns, all indicators I i ( s, a ) = 0 for all ( s, a ) ∈ X and i ∈ [ n ] , then we have

Putting everything together, we have the result.

A.2 The LSE subroutine

Algorithm 4 LSE

Input:

C , D, π k , π ′ k

1: for s, a ∈ C do

2: for every τ i s,a ∈ D [( s, a )] for every i ∈ [ n ] do

3: extract { S i 0 , A i 0 , R i 1 , C i 1 , S i 1 , A i 1 · · · S i H , A i H } from τ i s,a

4: compute G i r ( s, a ) ← ∑ H -1 h =0 γ h R i h +1 ; G i c ( s, a ) ← ∑ H -1 h =0 γ h C i h +1

5: compute ρ i ( s, a ) ← Π H -1 h =1 π k ( A i h | S i h ) π ′ k ( A i h | S i h )

8: Q r ( s, a ) ←⟨ w r , ϕ ( s, a ) ⟩ , Q c ( s, a ) ←⟨ w c , ϕ ( s, a ) ⟩ for all s, a return Q r , Q c

Given a core set C , a set of trajectories, a behaviour policy µ , a target policy π , the LSE subroutine (algorithm 4) returns a least-square estimate Q of q π .

If the core set C is empty, we define Q ( · , · ) to be zero. Then, for a target accuracy ϵ > 0 and a uniform misspecification error ω defined in assumption 1, we have a bound on the accuracy of ¯ q with respect to q π as given by the next lemma.

■

Lemma 5 [Lemma 4.3 of Weisz et al. (2022)] Let π be a randomized policy. Let C = { ( s i , a i ) } i ∈ [ N ] be a set of state-action pairs of set size N ∈ N . Assume for all i ∈ [ N ] , | ¯ q ( s i , a i ) -q π ( s i , a i ) | ≤ ϵ . Then, for all s, a ∈ S × A ,

Proof: If assumption 1 holds, then there exists a w π ∈ R d , ∥ w π ∥ 2 ≤ B such that for any ( s, a ) ∈ S × A , | ϕ ( s, a ) ⊤ w π -q π ( s, a ) | ≤ ω , where ω is the misspecification error. Let ¯ w π = V ( C, α ) -1 ∑ i ∈ [ N ] ϕ ( s i , a i ) ϕ ( s i , a i ) ⊤ w π . For any ( s, a ) ∈ S × A , recall Q ( s, a ) = ϕ ( s, a ) ⊤ w , where w = V ( C, α ) -1 ∑ i ∈ [ N ] ϕ ( s i , a i )¯ q ( s i , a i ) . It follows that | Q ( s, a ) -q π ( s, a ) |

≤ | ϕ ( s, a ) ⊤ ( w -¯ w π ) | + | ϕ ( s, a ) ⊤ ( ¯ w π -w π ) | + | ϕ ( s, a ) ⊤ w π -q π ( s, a ) | . (14)

To bound the second term in eq. (14), we have

Let α be the smallest eigenvalue of V ( C , α ) , then by eigendecomposition, V ( C , α ) = Q Λ Q ⊤ ≥ Q ( αI ) Q ⊤ = αQQ ⊤ ≥ αI since QQ ⊤ is orthonormal. This implies that V ( C , α ) -1 ≤ 1 α I , which leads to eq. (15).

Finally, we bound the first term in eq. (14). For every i ∈ [ N ] , let ξ i = ϕ ( s i , a i ) ⊤ w π -¯ q ( s i , a i ) . Then,

It follows that for all s, a ∈ S × A ,

Putting everything together complete the proof.

■

A.3 The accuracy of least-square estimates

Given a core set C and a target policy π , for any s ∈ Cov( C ) , a ∈ A , the feature vector ϕ ( s, a ) satisfies ∥ ϕ ( s, a ) ∥ V ( C ,α ) -1 ≤ 1 . Then, by lemma 5, we have | Q ( s, a ) -q π ( s, a ) | = O ( ω + ϵ ) for any s ∈ Cov( C ) . In this section, we verify whether this accuracy is maintained throughout the execution of our algorithm.

We note that policy improvements can only occur during a running phase ℓ = l . When all ( s, a ) pairs in C ℓ have their placeholder value ⊥ replaced by trajectories, algorithm 2 executes line 17 to line 27. During each iteration from k ℓ to k ℓ +1 -1 , the LSE subroutine is executed. The accuracy of ¯ q k is used to bound the estimation error in lemma 5. Therefore, we will first verify that the accuracy guarantee of ¯ q k ( s, a ) used in lemma 4 is indeed satisfied by the main algorithm and maintained throughout its execution.

Once a state-action pair is added to a core set, it remains in that core set for the duration of the algorithm. This means that any core set C l for l ∈ { 0 , . . . , L +1 } can grow in size over time. When a core set C l is extended during a running phase ℓ = l -1 , the least-square estimate will need be updated based on the newly extended C l in running phase ℓ = l , which contains newly discovered features. However, the policy is update only for states that are newly covered by the extended core set C l using the newly improved estimates. Meanwhile, the policy for other states that have already been updated by a prior softmax update remain unchanged. Note that after line 27 of algorithm 2 is run, the next phase's core set C l +1 will be set to C l , which means that any state that was once newly covered by C l is no longer considered newly covered. Consequently, the policy for those states will remain unchanged throughout the rest of the algorithm's execution. By updating the policies accordingly, we arrive at the following lemma, which will be crucial in proving the accuracy guarantee of the least-squares estimators.

Lemma 6 For any l ∈ { 0 , . . . , L } , let C past l be any past version of C l and let π past k for k = k l , · · · , k l +1 -1 be the corresponding policies associated with C past l . If at any later point during the execution of the algorithm, π k is updated again, then it holds that

Additionally, for any states that have been covered by C past l , it will continue to be covered by C l throughout the execution of the algorithm. In other words, s ∈ Cov( C past l ) ⊆ Cov( C l ) .

Proof: The proof follows similar logic to Lemma 4.5 of Weisz et al. (2022). Recall for matrices A,B , A ≥ B means that A -B is positive semidefinite.

Because C l ⊇ C past l , there may be more rows added to Φ C l than Φ C past l . Recall V ( C l , α ) = Φ ⊤ C l Φ C l + αI , and likewise for V ( C past l ) except Φ C l is replaced by C past l . Note that both V ( C l , α ) , V ( C past l , α ) are dimension d × d . Let C ′ l contain a set of state-action pairs that are in C l \ C past l . Then, V ( C l , α ) -V ( C past l , α ) = ∑ ( s,a ) ∈C ′ l ϕ ( s, a ) ϕ ( s, a ) ⊤ . Since any rank-1 matrices is positive semidefinite and their sum is also positive semidefinite, it follows that V ( C l , α ) ≥ V ( C past l , α ) .

Since V ( C l , α ) and V ( C past , α ) are symmetric positive definite matrices, then it follows that V ( C l , α ) -1 ≤ V ( C past l , α ) -1 . From this, we see that for any x ∈ R d , ∥ x ∥ V ( C l ,α ) -1 ≤ ∥ x ∥ V ( C past l ,α ) -1 . Then, it follows that ActionCov( C past l ) ⊆ ActionCov( C l ) , and likewise Cov( C past l ) ⊆ Cov( C l ) . Therefore, for an s ∈ Cov( C past l ) , the same state s ∈ Cov( C l ) , and the second result follows. Finally, by the definition of the extended policy set definition 1, Π π k , Cov( C l ) ⊆ Π π past k , Cov( C past l ) . ■

Lemma 7 For any l ∈ { 0 , . . . , L } and any ( s, a ) ∈ C l , the importance-weighted ¯ q k ( s, a ) computed in the LSE subroutine during the running phase ℓ = l is an unbiased estimator of the expected discounted reward:

for iterations k = k l , · · · , k l +1 -1 associated with this phase.

Proof: For the algorithm to execute the LSE subroutine, every ( s, a ) ∈ C l must have its placeholder value ⊥ in D l [( s, a )] replaced with trajectories. Trajectories are stored in D l only when the Gather-data subroutine returns 'discovered is False' during the running phase ℓ = l . This ensures that every state within these trajectories has passed the uncertainty test, thereby ensuring that all such states are in the cover of C l and will remain in the cover of C l for the duration of the algorithm, as established in lemma 6. Additionally, once trajectories for a ( s, a ) ∈ C l are stored in D l , they remain unchanged throughout the algorithm's execution. We aim to show that ¯ q k ( s, a ) computed using D l [( s, a )] , is an unbiased estimate of the stated quantity for all iterations associated with the phase. This will be established through the following inductive arguments.

Base case: for a ( s, a ) ∈ C l , the trajectories are generated and stored in D l [( s, a )] for the first time during the running phase ℓ = l .

We let τ i s,a denote the i -th trajectory ( S i 0 = s, A i 0 = a, R i 1 , S i 1 , . . . , S i H -1 , A i H -1 , R i H ) generated by π k l interacting with the simulator, and there are n such trajectories stored in D l [( s, a )] . Then, for all k = k l , · · · , k l +1 -1 , the return

where G ( τ i s,a ) = ∑ H -1 h =0 γ h R h +1 .

The behavior policy π k l is updated in a previous loop through the algorithm when ℓ = l -1 . For iterations, starting with k = k l +1 , . . . , k l +1 -1 , the policy π k is updated in iteration k -1 . Thus, the most recent policy π k and the behaviour policy π k l are available for the computation of the importance sampling ratio: ρ k ( τ i s,a ) = Π H -1 h =1 π k ( A i h | S i h ) π k l ( A i h | S i h ) . We show that the importance weighted return ρ k ( τ i s,a ) G ( τ i s,a ) is an unbiased estimate of E π k ,s,a [ G ( τ i s,a )] :

where δ ( s, a ) is the dirac-delta function. Note, for the on-policy iteration k = k l , the importance sampling ratio ρ k ( τ i s,a ) = 1 , and the result is trivially satisfied.

Finally, since all the states in the trajectories are in Cov( C l ) , it follows that any policy π ′ k ∈ Π π k , Cov( C l ) produces the same ρ k ( τ i s,a ) . The return ρ k ( τ i s,a ) G ( τ i s,a ) is an unbiased estimate of E π ′ k ,s,a [ G ( τ i s,a ] for all π ′ k ∈ Π π k , Cov( C l ) . This is true for all i = 1 , . . . , n . Consequently, ¯ q k ( s, a ) is an unbiased estimate of E π ′ k ,s,a [ ∑ H -1 h =0 R h +1 ] for all for all π ′ k ∈ Π π k , Cov( C l ) .

The requirement that the importance weighted ¯ q k be unbiased for all policies π ′ k ∈ Π π k , Cov( C l ) is important. This ensures that if π k is to be updated in a future loop through the algorithm again, the estimates remain unbiased and unchanged.

Previously generated trajectories: for any ( s, a ) ∈ C l , the trajectories have already been generated and stored in D l [( s, a )] during a previous loop of the algorithm when ℓ = l .

Let D past l [( s, a )] denote a past snapshot of the data stored for a ( s, a ) ∈ C past l . Let π past k for k = k l , . . . k l +1 -1 denote the policies associated with C past l after line 27 has been run. Finally, let τ i, past s,a denote the i -th trajectory stored in D past l [( s, a )] .

Assume the importance weighted return ρ k ( τ i, past s,a ) G ( τ i, past s,a ) is an unbiased estimate of E ˜ π k ,s,a [ G ( τ i, past s,a )] for all ˜ π k ∈ Π π past k , Cov( C past l ) for k = k l , . . . , k l +1 -1 . When the algorithm executes a loop with ℓ = l again, by lemma 6, the most recent policy π k ∈ Π π k , Cov( C l ) ⊆ Π π past k , Cov( C past l ) , then ρ k ( τ i, past s,a ) G ( τ i, past s,a ) is also an unbiased estimate of E π ′ k ,s,a [ G ( τ i, past s,a )] for all π ′ k ∈ Π π k , Cov( C l ) .

Once D past l [( s, a )] is populated with trajectories, D past l [( s, a )] remain unchanged throughout the execution of the algorithm. Therefore, G ( τ i s,a ) = G ( τ i, past s,a ) . Since all the states in the trajectories are in Cov( C past l ) ⊆ Cov( C l ) by lemma 6, any policy ˜ π k ∈ Π π past k , Cov( C past l ) produces the same ρ k ( τ i s,a ) . Thus, we have ρ k ( τ i s,a ) G ( τ i s,a ) = ρ k ( τ i, past s,a ) G ( τ i, past s,a ) . It follows that ρ k ( τ i s,a ) G ( τ i s,a ) is an unbiased estimate of E π ′ k ,s,a [ G ( τ i s,a )] for all π ′ k ∈ Π π k , Cov( C l ) . This is true for all i = 1 , . . . , n . Consequently, ¯ q k ( s, a ) is an unbiased estimate of E π ′ k ,s,a [ ∑ H -1 h =0 R h +1 ] for all π ′ k ∈ Π π k , Cov( C l ) . ■

Lemma 8 Whenever the LSE-subroutine of Confident-NPG is executed during a running phase ℓ = l for l ∈ { 0 , . . . , L } , the behaviour policy π k ℓ ( ·| s ) and target policy π k ( ·| s ) for k = k ℓ , . . . , k ℓ +1 -1 satisfy eq. (9) for any s ∈ Cov( C ℓ ) .

Proof: Recall that the behavior policy π k l is updated during a previous loop of the algorithm when ℓ = l -1 . By the time the LSE subroutine is executed, π k ℓ will be the policy that generated the data. Therefore, for the on-policy iteration where k = k ℓ , eq. (9) is trivially satisfied.

For subsequent iterations, starting with k = k ℓ + 1 , . . . , k ℓ +1 -1 , for any s ∈ Cov( C ℓ ) , the policy π k ( ·| s ) will have either performed a softmax update for the first time or remain unchanged from a previous softmax update based on an earlier least-square estimate. Either way, for any s ∈ Cov( C ℓ ) , the target policy π k and behaviour policy π k ℓ relate to each other in the form of π k ( ·| s ) ∝ π k ℓ ( ·| s ) exp( η 1 ∑ k -1 t = k ℓ ˜ Q t ( s, a )) . Since ˜ Q t ( s, a ) ∈ [0 , 1 1 -γ ] for any t = k ℓ , . . . , k -1 , then it follows that

By choosing η 1 = (1 -γ ) √ 2 ln( |A| ) K , H = ln(4 /ϵ (1 -γ )) 1 -γ , m = ln(1+ ρ 0 ) 2 Hϵ (1 -γ ) 2 , and K = 2 ln( A ) (1 -γ ) 4 ϵ 2 , we have η 1 (( ⌊ m ⌋ +1) -1) 1 -γ ≤ η 1 m 1 -γ = ln(1+ ρ 0 ) 2 H . Then it follows that eq. (9) is satisfied. ■

Lemma 9 For any l ∈ { 0 , . . . , L } and any ( s, a ) ∈ C l , the importance-weighted ¯ q k ( s, a ) computed in the LSE subroutine during the running phase ℓ = l satisfies the following with probability 1 -δ ′ ,

for all iterations k = k l , . . . , k l +1 -1 associated with this phase.

Proof: We apply lemma 4 to each ( s, a ) ∈ C l . To ensure the applicability of the lemma, we verify its two conditions: 1) the policies satisfy eq. (9) and 2) the estimate are unbiased.

We note that when the LSE-subroutine is executed during a running phase with ℓ = l , the Gather-data subroutine has already completed, and the algorithm trajectories for each state-action pair ( s, a ) ∈ C l are stored in D l [( s, a )] . For any trajectory to be stored in D l , this means that every state within the trajectories has passed the uncertainty test, ensuring that all such states are in the cover of C l . By lemma 6, these states will continue to be covered by C l throughout the execution of the algorithm. The implication of this is that all the states in a trajectory of D l [( s, a )] satisfy eq. (9) by lemma 8.

Second, by lemma 7, the importance weighted return ¯ q k ( s, a ) is unbiased estimate of any E π ′ k ,s,a [ ∑ H -1 h =0 γ h R h +1 ] for all π ′ k ∈ Π π k , Cov( C l ) . Altogether, by lemma 4, we can ensure eq. (16) holds.

Consider a past loop through the algorithm with ℓ = l , let C past l be the core set and π past k for k = k l , . . . , k l +1 -1 be the policies associated with C past l after line 27 has been run. If eq. (16) holds for all ˜ π k ∈ Π π past k , Cov( C past l ) , then the accuracy of ¯ q k will continue to hold for any future update of π k because π k ∈ Π π k , Cov( C l ) ⊆ Π π past k , Cov( C past l ) by lemma 6. ■

Lemma 10 (Weisz et al. (2022)) At any time during the execution of the main algorithm, for all l ∈ { 0 , . . . , L } , the size of each C l is bounded:

where the α is the smallest eigenvalue of V ( C , α ) and N is the radius of the Euclidean ball containing all the feature vectors.

Lemma 11 Whenever LSE subroutine of Confident-NPG is executed during a running phase ℓ = l for l ∈ { 0 , . . . , L } , the least-square estimate ˜ Q k ( s, a ) satisfies the following condition for all iterations k = k ℓ , · · · , k ℓ +1 -1 associated with this phase and for all s ∈ Cov( C ℓ ) and a ∈ A ,

Proof: We prove the result by induction similar to Lemma F.1 of Weisz et al. (2022). We let C -l , π -k , ˜ Q -k to denote the value of variable C l , π k , ˜ Q k at the time when line 17 to line 27 were most recently executed with ℓ = l in a previous loop through the algorithm. If such time does not exist, we let their values be the initialization values. Only after the execution of line line 27 will C -l change and as well as C l +1 , and this is the only time that C l +1 can be changed. Therefore, at the start of a new loop, we see that C l +1 = C -l . This also holds at the initialization of the algorithm, we conclude that at the start of each loop, Cov( C l +1 ) = Cov( C -l ) .

At initialization, ˜ Q k = 0 for any k ∈ { 0 , . . . , K } and C l = () for all l ∈ { 0 , . . . , L } . By applying lemma 5 (Lemma 4.3 of Weisz et al. (2022)), for any ( s, a ) ∈ S × A , which satisfies eq. (17).

Next, let us consider the start of a loop after ℓ = l is set and assume that the inductive hypothesis holds for the previous time line 17 to line 27 were executed with the same value of ℓ = l . For any s ∈ Cov( C -l -1 ) , policy π k l ( ·| s ) would have already been set in a previous loop with value l -1 and remains unchanged in the current loop. By lemma 9, the condition of lemma 5 holds, then by lemma 5, we have for any s ∈ Cov( C -l -1 ) ,

where ∥ ϕ ( s, · ) ∥ V ( C -l -1 ,α ) -1 ≤ 1 because s ∈ Cov( C -l -1 ) and | C -l -1 | ≤ ˜ d by lemma 10. Recall by definition, ˜ Q k l = ˜ Q -k l , π k l = π -k l , C l = C -l -1 , and Cov( C l ) = Cov( C -l -1 ) . It follows that for any s ∈ Cov( C l ) , | ˜ Q k l ( s, · ) -q π ′ k l ( s, · ) | ≤ ϵ ′ for π k ′ l ∈ Π π k l , Cov( C l ) .

For any s that is already covered by C l (i.e., s ∈ Cov( C -l ) ), and for any off-policy iteration k = k l +1 , · · · , k l +1 -1 , ˜ Q k ( s, · ) = ˜ Q -k ( s, · ) . Additionally, the policy π k ( ·| s ) would already have been set in a previous running loop with the same value of l and remains unchanged in the current loop. For s ∈ Cov( C -l ) , by lemma 9, the condition of lemma 5 holds, and then by lemma 5,

where ∥ ϕ ( s, · ) ∥ V ( C -l ,α ) -1 ≤ 1 because s ∈ Cov( C -l ) and |C -l | ≤ √ ˜ d by lemma 10. By lemma 6, Π π k , Cov( C l ) ⊆ Π π -k , Cov( C -l ) . By definition, ˜ Q k ( s, · ) = ˜ Q -k ( s, · ) for s ∈ Cov( C l +1 ) = Cov( C -l ) , | ˜ Q k ( s, · ) -q π ′ k ( s, · ) | ≤ ϵ ′ for any π ′ k ∈ Π π k , Cov( C l +1 ) .

Finally, for any s that is newly covered by C l (i.e., s ̸∈ Cov( C l +1 ) ), and for all k = k l , . . . , k l +1 -1 , ˜ Q k ( s, · ) = Q k ( s, · ) . By lemma 9, the condition of lemma 5 holds, and then by lemma 5, we have

where ∥ ϕ ( s, · ) ∥ V ( C l ,α ) -1 ≤ 1 and |C l | ≤ ˜ d by lemma 10.

■

Lemma 12 For any δ ′ ∈ (0 , 1] , a target accuracy ϵ > 0 , misspecification error ω ≥ 0 , and initial state s 0 , with probability at least 1 -δ ′ , the value difference between any π ∈ Π rand and the mixture policy ¯ π K returned by Confident-NPG has the following bound:

Proof: For any l ∈ { 0 , . . . , L } and for all iterations k = k l , . . . , k l +1 -1 associated with l , define

Then, for any s ∈ Cov( C l ) ,

where d π ( s ) is the discounted state occupancy measure induced by following π starting from s .

To bound term II , we note that for any s ∈ Cov( C l ) , we have π + k ( ·| s ) = π k ( ·| s ) and both π k , π + k ( ·| s ) ∈ Π π k , Cov( C l ) . By lemma 11, we have for any s ∈ Cov( C l ) , a ∈ A , | ˜ Q k ( s, a ) -q π ′ k ( s, a ) | ≤ ϵ ′ for any π ′ k ∈ Π π k , Cov( C l ) . Then, for any s ∈ Cov( C l ) , a ∈ A ,

It follows that for any s ∈ Cov( C l ) ,

To bound term I , we note that for any s ̸∈ Cov( C l ) , π + k ( ·| s ) = π ( ·| s ) and π + k ∈ Π π k , Cov( C l ) , then

In summary, for any l , for any k = k l , . . . , k l +1 -1 associated with l , and for any s ∈ Cov( C l ) ,

Because of line 27 of algorithm 2, one can use induction to show that by the time Confident-NPG terminates, all the C l for l ∈ { 0 , . . . , L + 1 } will be equal. Therefore, the cover of C l for all l ∈ { 0 , . . . , L +1 } are also equal. Thus, it is sufficient to only consider C 0 at the end of the algorithm. Because of line 3 of algorithm 2, s 0 ∈ Cov( C 0 ) . Putting everything together, the value difference can be bounded as follows,

■

B Confident-NPG-CMDP

We include the proofs of lemmas that appear in prior works and supporting lemmas that are helpful proving the lemmas in the main section. The lemmas that appear in the main section will have the same numbering here.

B.1 The accuracy of least-square estimates

Once a state-action pair is added to a core set, it remains in that core set for the duration of the algorithm. This means that any C l for l ∈ { 0 , . . . , L +1 } can grow in size. When a core set C l is extended during a running phase ℓ = l , the least-square estimates will need be updated based on the newly extended C l which contains newly discovered features. However, the policy is update only for states that are newly covered by the extended core set C l using the newly improved estimates. Note that after line 30 of algorithm 1 is run, the next phase's core set C l +1 will be set to C l , which means that any state that was once newly covered by C l is no longer considered newly covered. Consequently, the policy for those states will remain unchanged throughout the rest of the algorithm's execution.

We introduce hypothetical ˜ Q r k and ˜ Q c k to reflect the value of ˜ Q p k , used in the update of π k +1 for k = k l , . . . , k l +1 -1 associated with running phase ℓ = l . At initialization, ˜ Q r k ( s, a ) = 0 , ˜ Q c k ( s, a ) = 0 for all k = 0 , . . . , K , s ∈ S and a ∈ A . The values are specified in the following cases when line 27 is run:

where Q r k ( s, a ) , Q c k ( s, a ) are the least-square estimates using the most recently extended C l at that time. The dual variable λ k is defined in line 29. Therefore, the ˜ Q p k ( s, a ) used in the update of policy at line 27 can be written as ˜ Q p k ( s, a ) = trunc [0 , 1 1 -γ ] ˜ Q r k ( s, a ) + λ k trunc [0 , 1 1 -γ ] ˜ Q c k ( s, a ) .

Lemma 1 Whenever LSE subroutine in line 21 of Confident-NPG-CMDP is executed during a running phase ℓ = l for l ∈ { 0 , . . . , L } , the least-square estimate ˜ Q p k ( s, a ) satisfies the following condition for all iterations k = k ℓ , . . . , k ℓ +1 -1 associated with this phase and for all s ∈ Cov( C ℓ ) and a ∈ A ,

where ϵ ′ = (1+ U )( ω + √ αB +( ω + ϵ ) √ ˜ d ) with ˜ d = ˜ O ( d ) and U is an upper bound on the optimal Lagrange multiplier. Similarly, for initial state s 0 , we have

Proof: By using the primal-dual approach, we have reduced the CMDP problem to an unconstrained problem with a single reward of the form r λ = r + λc .

Because of line 7 have executed before entering the loop and line 30 have been executed in the previous phase ℓ = l -1 , the initial state s 0 ∈ Cov( C ℓ ) . If s 0 is in Cov( C ℓ ) for the first time (i.e. s 0 ∈ Cov( C ℓ ) \ Cov( C ℓ +1 ) ), then the dual variable λ k makes a mirror descent update in line 29 using V c k ( s 0 ) at that time. After line 30 is executed, the core set for the next phase C ℓ +1 = C ℓ . This means that any states, including s 0 , that are covered by C ℓ are then covered by C ℓ +1 . By lemma 6, the initial state s 0 will continue to be covered by C ℓ +1 for the remainder of the algorithm's execution. This implies that the dual variable λ k referenced in this lemma remains fixed at the value set when s 0 is covered by C ℓ for the first time and does not change thereafter for the duration of the algorithm's execution.

Then the proof of this lemma follows similar logic to lemma 11 in the single reward setting. The result of lemma 11 uses lemma 5. For lemma 5 to hold, lemma 9 is used to verify the conditions sufficient for lemma 5 to hold. For lemma 9 to hold, one of the requirement is that the behaviour policy π k ℓ and the target policy π k must satisfy eq. (9). In the following paragraphs, we show that eq. (9) indeed hold with appropriate changes to the parameters of interest. Then it follows that lemma 9 holds and consequently lemma 5 holds. Once all the sufficient conditions hold, by following similar logic as in lemma 11, we have the proof.

Since the policies are updated with respect to ˜ Q p instead of ˜ Q of the single-reward setting, we need to make adjustment to η 1 , H, m, K to ensure π k ℓ and π k indeed satisfy eq. (9). First, note the value ˜ Q p k for k = 0 , . . . , K are in the range of 0 and 1+ U 1 -γ . The upper bound value is the result of the primary reward function taking values in the range of [0 , 1] and the dual variable taking values in the range of [0 , U ] . The value U is defined in lemma 13 for relaxed-feasibility and in lemma 15 for strict-feasibility, and it is an upper bound on the optimal dual variable (i.e., λ ∗ ≤ U ). By similar argument to lemma 8, we make the following changes to η 1 , H, m, K . We set the step size η 1 = 1 -γ 1+ U √ 2 ln( |A| ) K , the total number of iterations K = 6 2 ( √ 2 ln( |A| )+1) 2 (1+ U ) 2 (1 -γ ) 4 ϵ 2 , and H = ln((30 √ ˜ d (1+ U )) / ((1 -γ ) 2 ϵ )) 1 -γ . Then, it follows that m = (1+ U ) ln(1+ ρ 0 ) 2 Hϵ (1 -γ ) 2 .

Next, from lemma 7, we have each ¯ q r k ( s, a ) and ¯ q c k ( s, a ) is an unbiased estimate of E π ′ k ,s,a [ ∑ H -1 h =0 γ h R h +1 ] and E π ′ k ,s,a [ ∑ H -1 h =0 γ h C h +1 ] respectively for all π ′ k ∈ Π π k , Cov( C l ) . Let δ ′ = 2exp ( -2 n ( ϵ 4 ) 2 ( (1+ ρ 0 ) 1 -γ ) 2 ) . By lemma 9, with probability 1 -δ ′ , we have for any ( s, a ) ∈ C l ,

Then the conditions of lemma 5 hold, and by similar argument to lemma 11 using lemma 5, we have for each ( s, a ) ∈ Cov( C l ) ,

for all π ′ k ∈ Π π k , Cov( C ℓ ) . Then it follows that for a given λ k ,

for all π ′ k ∈ Π π k , Cov( C ℓ ) .

Finally, since s 0 ∈ Cov( C ℓ ) and ˜ V c k ( s 0 ) = ⟨ π ′ k ( ·| s 0 ) , ˜ Q c k ( s 0 , · ) ⟩ , therefore | ˜ V c k ( s 0 ) -v c π k ( s 0 ) | = |⟨ π ′ k ( ·| s 0 ) , ˜ Q c k ( s 0 , · ) -q c π ′ k ( s 0 , · ) ⟩| ≤ ω + √ αB +( ω + ϵ ) √ ˜ d for all π ′ k ∈ Π π k , Cov( C ℓ ) . ■

C Relaxed-feasibility

Lemma 13 [Lemma 4.1 of Jain et al. (2022)] Let λ ∗ be the optimal dual variable that satisfies min λ ≥ 0 max π v r π ( ρ ) + λ ( v c π ( ρ ) -b ) . If we choose

then λ ∗ ≤ U .

Proof: Let π ∗ c ( ρ ) = arg max v c π ( ρ ) , and recall that ζ = v c π ∗ c ( ρ ) -b > 0 , then

After rearranging terms, we have

By choosing U = 2 ζ (1 -γ ) , we have λ ∗ ≤ U .

Definition 2

Lemma 2 Let δ ∈ (0 , 1] be the failure probability, ϵ > 0 be the target accuracy, and s 0 be the initial state. Assuming for all s ∈ Cov( C 0 ) and all a ∈ A , | ˜ Q p k ( s, a ) -q p π ′ k ,λ k ( s, a ) | ≤ ϵ ′ and | ˜ V c k ( s 0 ) -v c π ′ k ( s 0 ) | ≤ ω + √ αB +( ω + ϵ ) √ ˜ d for all π ′ k ∈ Π π k , Cov( C 0 ) , then, with probability 1 -δ , Confident-NPG-CMDP returns a mixture policy ¯ π K that satisfies the following,

where ϵ ′ = (1 + U )( ω +( √ αB +( ω + ϵ ) √ ˜ d )) with ˜ d = ˜ O ( d ) .

■

Proof: For the following result, we consider a k ∈ { 0 , . . . , K } with its corresponding l ∈ { 0 , . . . , L } . At the time of termination, all C l are equal.

To obtain a bound on the suboptimality and the constraint violation, we apply lemma 12 with π = π ∗ of CMDP, ˜ Q p k = ˜ Q r k + λ k ˜ Q c k instead of ˜ Q k , and lemma 1 instead of lemma 11 of the single reward setting. Then, we have

By Proposition 28.6 of Lattimore and Szepesvári (2020), the primal regret R p ( π ∗ , K ) ≤ 1+ U 1 -γ √ 2 K ln( |A| ) with η 1 = 1 -γ 1+ U √ 2 ln( |A| ) K . Expanding eq. (18) in terms of v r , v c , we have

Furthermore, by lemma 1, we have | ˜ Q c k ( s, a ) -q c π ′ k ( s, a ) | ≤ ω + √ αB + ( ω + ϵ ) √ ˜ d for any s ∈ Cov( C l ) . Recall ˜ V c k ( s 0 ) = ⟨ π k ( ·| s 0 ) , ˜ Q c k ( s 0 , · ) ⟩ , then it follows that λ k ( v c π k ( s 0 ) -˜ V c k ( s 0 )) ≤ | λ k ( v c π k ( s 0 ) -˜ V c k ( s 0 )) | ≤ U ( ω + √ αB +( ω + ϵ ) √ ˜ d ) ≤ ϵ ′ .

The update to the dual variable is a mirror descent algorithm. By Proposition 28.6 of Lattimore and Szepesvári (2020), the dual regret R d (0 , K ) ≤ U √ K 1 -γ with η 2 = U (1 -γ ) √ K . Altogether,

For bounding the constraint violations, we first incorporate R d ( λ, K ) into eq. (19) and rearrange terms to obtain:

There are two constraint cases. Case one is no violation: b -v c ¯ π K ( s 0 ) ≤ 0 . Then, it also holds that b -△-v c ¯ π K ( s 0 ) ≤ 0 for any △ ≥ 0 , which is what we want to show. Case two is violation: b -v c ¯ π K ( s 0 ) > 0 , for which case, λ = U . Using notation [ x ] + = max { x, 0 } , we have

By Lemma B.2 of Jain et al. (2022), we have

Theorem 1 With probability 1 -δ , the mixture policy ¯ π K returned by confident-NPG-CMDP ensures that

if we choose n = 30 2 (1+ ρ 0 ) 2 (1+ U ) 2 ˜ d 2 ϵ 2 (1 -γ ) 4 ln ( 8 ˜ d ( L +1) δ ) , α = (1 -γ ) 2 ϵ 2 30 2 (1+ U ) 2 B 2 , K = 6 2 ( 2 ln( |A| )+1) 2 (1+ U ) 2 (1 -γ ) 4 ϵ 2 , η 1 = 1 -γ 1+ U √ 2 ln( |A| ) K , η 2 = U (1 -γ ) √ K , H = ln((30 √ ˜ d (1+ U )) / ((1 -γ ) 2 ϵ )) 1 -γ , m = (1+ U ) ln(1+ ρ 0 ) 2 ϵH (1 -γ ) 2 , and U = 2 ζ (1 -γ ) .

Furthermore, the algorithm utilizes at most ˜ O ( d 2 (1 + U ) 3 ϵ -3 (1 -γ ) -8 ) queries in the local-access setting.

Proof: From lemma 2, we have

Let C = 1 ζ (1 -γ ) for a ζ ∈ (0 , 1 1 -γ ] . By lemma 13, we chose U = 2 C and λ ∗ ≤ C . It follows that 1 U -λ ∗ ≤ 1 C = ζ (1 -γ ) ≤ 1 , and thus the right hand side of eq. (23) is upper bounded by the right hand side of eq. (22). Recall ϵ ′ = (1 + U ) ( ω + ( √ αB +( ω + ϵ ) √ ˜ d )) . Then, the goal is to set the parameters H,n,K , and α appropriately so that the A,B and C of the following expression, when added together, is less than ϵ :

First, we set n appropriately so that the failure probability is well controlled. The failure probability depends on the number of times Gather-data subroutine (algorithm 3) is executed. Gather-data is run for phase 0 , . . . , L . Each phase has at most ˜ d elements, and recall ˜ d is defined in lemma 10. Therefore, Gather-data would return success at most ˜ d times. Altogether, Gather-data can return success at most ˜ d ( L +1) times, each with probability of at least 1 -δ ′ = 1 -δ/ ( ˜ d ( L +1)) . By a union bound, Gather-data returns success in all occasions with probability 1 -δ .

By setting H = ln((30 √ ˜ d (1+ U )) / ((1 -γ ) 2 ϵ )) 1 -γ and n = 30 2 (1+ ρ 0 ) 2 (1+ U ) 2 ˜ d 2 ϵ 2 (1 -γ ) 4 ln ( 8 ˜ d ( L +1) δ ) , we have for any l ∈ { 0 , . . . , L } , k = k l , . . . , k l +1 -1 , the | ¯ q r k ( s, a ) -q r π ′ k ( s, a ) | ≤ 4 6 (1 -γ ) ϵ 5(1+ U ) √ ˜ d and | ¯ q c k ( s, a ) -q c π ′ k ( s, a ) | ≤ 4 6 (1 -γ ) ϵ 5(1+ U ) √ ˜ d hold for all π ′ k ∈ Π π k , Cov( C l ) with probability at least 1 -δ . Then, this is used in the accuracy guarantee of the least-square estimate (lemma 1) and finally in the suboptimality bound of lemma 2.

Then, we can set α of eq. (24) to be equal to ϵ 6 and solve for α = ϵ 2 (1 -γ ) 2 30 2 (1+ U ) 2 B 2 . Finally, by setting K = 6 2 ( √ 2 ln( |A| )+1) 2 (1+ U ) 2 (1 -γ ) 4 ϵ 2 , we have C of eq. (24) be less than ϵ 6 . Altogether, we have the reward suboptimality satisfying eq. (20) and constraint satisfying eq. (21).

For the query complexity, we note that our algorithm does not query the simulator in every iteration, but at fixed intervals, which we call phases. Each phase is m iterations in length. There are total of L = ⌊ K/ ( ⌊ m ⌋ +1) ⌋ ≤ K/m = ˜ O ( (1 + U )(1 -γ ) -3 ϵ -1 ) phases. In each phases, Gather-data subroutine (algorithm 3) can be run. Each time Gather-data returns success with trajectories, the subroutine would have made at most nH queries. Gather-data is run for each of the elements in C l , l ∈ { 0 , . . . , L } . By the time the algorithm terminates, all C l 's are the same. Since there are at most ˜ O ( d ) elements in each C l , the algorithm will make a total of nH ( L + 1) |C 0 | number of queries to the simulator. Since we have H = ˜ O ((1 -γ ) -1 ) , n = ˜ O ((1 + U ) 2 dϵ -2 (1 -γ ) -4 ) and L = ˜ O ((1 + U ) ϵ -1 (1 -γ ) -3 ) , the sample complexity is ˜ O ( d 2 (1 + U ) 3 (1 -γ ) -8 ϵ -3 ) . ■

D Strict-feasibility

Lemma 14 Let π ∗ △ be defined as in eq. (7) and π ∗ be an optimal policy of CMDP. Then, for a △ > 0 ,

where λ ∗ is an optimal dual variable that satisfies min λ ≥ 0 max π v r π ( s 0 ) + λ ( v c π ( s 0 ) -b ′ ) .

Proof:

After rearranging the terms, we get the result.

■

Lemma 15 Let λ ∗ be the optimal dual variable that satisfies min λ ≥ 0 max π V r π ( s 0 )+ λ ( V c π ( s 0 ) -b ′ ) . If we choose

then λ ∗ ≤ U requiring that △∈ (0 , ζ 2 ) .

Proof: Let π ∗ ( s 0 ) = arg max V c ( s 0 ) , and recall that ζ = V c ∗ ( s 0 ) -b > 0 , then c π π c

If we require △∈ (0 , ζ 2 ) , then we have

After rearranging terms in eq. (25), we have

By choosing U = 4 ζ (1 -γ ) , λ ∗ ≤ U .

■

Theorem 2 With probability 1 -δ , a target ϵ > 0 , the mixture policy ¯ π K returned by confidentNPG-CMDP ensures that v r π ∗ ( s 0 ) -v r ¯ π K ( s 0 ) ≤ ϵ and v c ¯ π K ( s 0 ) ≥ b , if assuming the misspecificiation error ω ≤ △ (1 -γ ) 70(1+ U )(1+ √ ˜ d ) , and if we choose △ = ϵ (1 -γ ) ζ 8 , α = △ 2 (1 -γ ) 2 70 2 (1+ U ) 2 B 2 , K = √

Furthermore, the algorithm utilizes at most ˜ O ( d 2 (1 + U ) 3 (1 -γ ) -11 ϵ -3 ζ -3 ) queries in the localaccess setting.

Proof: Let λ ∗ be the optimal dual variable that satisfies the Lagrangian primal-dual of the surrogate CMDP defined by eq. (7) (i.e., λ ∗ = arg min λ ≥ 0 max π v r π ( s 0 ) + λ ( v c π ( s 0 ) -b ′ ) ).

where ¯ ϵ = 5(1+ U )(1+ √ ˜ d ) ω 1 -γ + 5(1+ U ) √ αB 1 -γ + 5(1+ U ) ϵ √ ˜ d 1 -γ + ( √ 2 ln( |A| )+1)(1+ U ) (1 -γ ) 2 √ K . By lemma 14, v r π ∗ ( s 0 ) -v r π ∗ △ ( s 0 ) ≤ λ ∗ △ . We can further upper bound λ ∗ by U = 4 ζ (1 -γ ) using lemma 15 and requiring △∈ ( 0 , ζ 2 ) . Together with theorem 1, we have Confident-NPG-CMDP return ¯ π K s.t.

Now, we need to set △ such that 1) △ ∈ ( 0 , ζ 2 ) and 2) △-¯ ϵ ≥ 0 are satisfied. If we choose △ = ϵ (1 -γ ) ζ 8 , then the first condition is satisfied. This is because ϵ ∈ ( 0 , 1 1 -γ ] , and thus △≤ ζ 8 < ζ 2 .

.

Next, we check if our choice of △ = ϵ (1 -γ ) ζ 8 satisfies △-¯ ϵ ≥ 0 . For the condition △-¯ ϵ ≥ 0 to be true, we make an assumption on the misspecification error ω ≤ △ (1 -γ ) 70(1+ U )(1+ √ ˜ d ) , and pick n, α, K, η 1 , η 2 , H, m to be the values outlined in this theorem. Consequently, we have ¯ ϵ = 1 2 △ Then, we have ensured the condition △-¯ ϵ ≥ 0 is satisfied.

We note that because ζ ∈ ( 0 , 1 1 -γ ) , we have ¯ ϵ ≤ ϵ 16 ≤ ϵ . following from eq. (26), we have v r π ∗ ( s 0 ) -v r ¯ π K ( s 0 ) ≤ ϵ and b ′ -V c ¯ π K ( s 0 ) ≤ △ 2 . Then it follows that b + △ 2 ≤ V c ¯ π K ( s 0 ) . Strictfeasilbility is achieved.

For the query complexity, we note that our algorithm does not query the simulator in every iteration, but at fixed intervals, which we call phases. Each phase is m iterations in length. There are total of L = ⌊ K/ ( ⌊ m ⌋ +1) ⌋ ≤ K/m = ˜ O ( (1 + U )(1 -γ ) -3 △ -1 ) phases. In each phase, Gather-data subroutine (algorithm 3) can be run. Each time Gather-data subroutine returns with trajectories, the subroutine would have made at most nH queries. Gather-data is run for each of the element in C l , l ∈ { 0 , . . . , L } . By the time the algorithm terminates, all C l 's are the same. Since there are at most ˜ O ( d ) elements in each C l , the algorithm will make a total of nH ( L + 1) |C 0 | number of queries to the simulator. Since we have H = ˜ O ((1 -γ ) -1 ) , n = ˜ O ((1 + U ) 2 d (1 -γ ) -4 △ -2 ) , L = ˜ O ( (1 + U )(1 -γ ) -3 △ -1 ) , and △ = ϵζ (1 -γ ) 8 , the sample complexity is ˜ O ( d 2 (1 + U ) 3 (1 -γ ) -11 ϵ -3 ζ -3 ) . ■

E A discussion on memory cost and some implementation details

By recording the states added to each core set during extensions and their corresponding least-squares weights, we can reconstruct the policy as needed. This section explains how to track this information and how it facilitates policy reconstruction.

In phase l , the policies π k for iterations k = k l + 1 , . . . , k l +1 -1 depend on the core set C l . Since C l can be extended multiple times, these policies may change accordingly. However, we do not want to change the action distribution for states have already passed the uncertainty test in previous extensions (i.e. s ∈ Cov( C l +1 ) ). For such states, action distributions are based on the least-square estimation of the core set at that time they passed the uncertainty test for the first time (i.e., s ∈ Cov( C l ) \ Cov( C l +1 ) ). Therefore, it is essential to track newly added states in each extension and store their corresponding least-square weights to recompute their action distributions.

To achieve this, C 0 is extended only via line 7 and line 15 of algorithm 1, while other core sets C l , where l ∈ { 1 , . . . , L +1 } are extended solely via line 30 during the running phase ℓ = l -1 . We mark newly added elements in line 7, line 15, and line 30. After executing line 21, we store the least-square weights associated with these newly added state-action pairs.

By keeping track of the state-action pairs that are newly added in each extension and saving the corresponding least-square weights, we can construct the policy π k +1 associated with C l . Let C 0 l = ∅ , and C i l denote all state-action pairs added to C l in extension i for i = 1 up to at most ˜ d . Let w i k represent the least-square weight computed using C l = C 0 l ∪ C 1 l ∪ C 2 l ∪ · · · ∪ C i l for the k -th iteration. When C l is extended for the ( i +1) -th time, let C i +1 l be the set of newly added state-action pairs,

making the latest C l = C 0 l ∪C 1 l ∪C 2 l ∪· · ·∪C i +1 l . The least-squares weight w i +1 k is then computed using C l . When line 27 of algorithm 1 is executed, π k +1 remains unchanged for the rest of the algorithm's execution for any states already in Cov( C 0 l ∪ C 1 l ∪ · · · ∪ C i l ) , equivalent to Cov( C l +1 ) in line 27, because line 30 would have been executed in the i -th extension, making C l +1 = C 0 l ∪ C 1 l ∪ · · · ∪ C i l . For states in Cov( C 0 l ∪C 1 l ∪·· ·∪C i +1 l ) \ Cov( C 0 l ∪C 1 l ∪·· ·∪C i l ) (equivalent to Cov( C l ) \ Cov( C l +1 ) in line 27), π k +1 makes a softmax update using w i +1 k . For all other states not in Cov( C l ) , the policy remains as π k .

A subroutine can start with π 0 , use the stored data to compute and return π k ( ·| s ) for any s and k . By tracking newly added elements and the corresponding least-square weights, the algorithm can reconstruct policies π 0 , . . . , π K . This approach enables the algorithm to return the value of a mixture policy at termination.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Reviewers may find results from section 4 leading to the main results stated in section 5 and section 6.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions are listed in every lemma and theorems. In the supplementary section, all supporting lemmas will be proven or cited. All the lemmas are presented in sequence leading up to the two main theorems.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: This paper does not include experiment

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This paper does not include experiment

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: This paper does not include experiment

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: This paper does not include experiment

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: This paper does not include experiment

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: I have read the ethics guidelines.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: By understanding the sample complexity of CMDP, a framework used by many of the safe reinforcement learning research, we can design more efficient algorithms. This potentially has broader positive impact for real world applications.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not pose such risk as it contains no data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not use any data, model or code.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not contain any new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor resarch with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This is the first work which addresses and achieves polynomial sample complexity for the learning problem of CMDPs in the more general setting of linear function approximation with $q_{\pi}$ realizability.  The authors propose a primal-dual algorithm and utilize a local access model (can be viewed to in between online learning and a generative model ). The algorithm reliably produces a policy that strictly adheres to the constraints while closely optimizing the reward function's value.

STRENGTHS:
- The paper is mostly well written with good summarization of the main ideas and theorem statements.
- It is the first work achieving polynomial sample complexity for CMDPs in the more general $q_{\pi}$ realizability setting.

WEAKNESSES:
- Lack of experimental results.
-  The paper seems an extension of earlier work i.e., (Weisz et al., 2022). 
- Despite general good writing, the main algorithm description in section 4.2-4.3 is very dense and difficult to parse.

QUESTIONS:
- Since the paper claims that $q_{\pi}$ realizability is more general than the linear MDP setting, does it mean that linear MDP setting implies $q_{\pi}$ realizability ? Is this result known or trivial ? Could this point be included in the paper?

- The paper seems an extension of earlier work i.e., (Weisz et al., 2022). Can the authors elaborate on what were the main technical challenges overcome on extending this work to the CMDP setting ?

LIMITATIONS:
There are no potential negative societal impact of their work
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied constrained Markov decision processes (CMDP) and proposed a confident policy gradient algorithm for $q_\\pi$ realizable MDPs. Here, a $q_\\pi$ realizable MDP assumes that the Q function can be approximated by a linear function w.r.t. some feature of state-action pairs. By using primal-dual methods, the proposed algorithm can find an $\\epsilon$-optimal policy satisfying the constraint or with small constraint violation. The proposed approach can also be applied to the misspecification case.\n\nSTRENGTHS:\n1. The paper is technically sound.\n2. The studied problem is important and well-motivated.\n\nWEAKNESSES:\n1. The major weakness is the presentation of contribution. According to the paper's presentation, it feels like the proposed method is a straightforward extension of Weisz et al., 2022 applying to the Lagrangian objective function. It would be better to highlight the difficulties of handling CMDP or this Lagrangian function.\n\n2. The presentation of the two learning goals is also confusing. In my understanding, the proposed algorithm can already achieve the stringent constraint, i.e. zero violation, and the relaxed feasibility is satisfied naturally. It seems redundant to spend a whole section to describe the result of relaxed feasibility. Some papers split the two results (Liu et al., 2021) because they need different assumptions. Perhaps the authors can explain or organize the presentation more clear.\n\n3. Another presentation issue: There is no introduction or justification of the terminology *natural policy gradient* in this paper. \n\n[1]. Weisz et al., Confident Approximate Policy Iteration for Efficient Local Planning in $q_\\pi$-realizable MDPs, 2022.\n\n[2]. Liu et al., Learning policies with zero or bounded constraint violation for constrained mdps, 2021\n\nQUESTIONS:\nPlease see the weaknesses part.\n\nLIMITATIONS:\nNo further limitations need to be addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n**Problem setup**\n\nThe authors consider the task of global planning in large Constrained \nDiscounted MDPs. They assume local access to a simulator which can be queried at previously \nencountered state-action pairs to obtain a next state sample and immediate \nreward, and that the $Q$-value of any policy is linearly realizable by a given feature map. Under this more general assumption on the MDP, they aim to develop an efficient method which learns a policy that maximizes return with minimal constraint violations.\n\n**Approach**\n\nIn the above context, the authors propose a primal-dual approach to solve a constrained \nreturn optimization problem. Under the weaker assumption of $Q^{\\pi}$-realizability and local access to a simulator, their method combines existing techniques on $Q$-value estimation in large MDPs and off-policy evaluation to nicely approximate the gradients of the primal-dual objective while conserving samples. Precisely, they apply the $Q$-values estimation sub-routine of CAPI-QPI-PLAN in [1] to construct least squares estimates which are guaranteed to closely approximate the true value at a subset of important (namely \\emph{core}) state-action pairs. Furthermore, contrary to the fully on-policy evaluation routine of CAPI-QPI-PLAN, their method Confident-NPG-CMDP reuses the core set and trajectory data for value estimation and policy improvement across a number of steps in each learning phase.\n\n**Result**\n\nThe authors derive performance guarantees for their method in terms of the value error and constraint violation of the output mixture policy. Precisely, they prove that w.h.p their method learns a near-optimal policy with minimal constraint violation with at most $\\mathcal{O}(poly(d)\\varepsilon^{-3})$ queries to the simulator and iterations. \n\n\n\n[1] Weisz, G., György, A., Kozuno, T., & Szepesvári, C. (2022). Confident Approximate Policy Iteration for Efficient Local Planning in $ q^\\pi $-realizable MDPs. Advances in Neural Information Processing Systems, 35, 25547-25559.\n\n\n**Update**: I also updated my score for contribution.\n\nSTRENGTHS:\n1. This paper addresses the task of planning in large Constrained Discounted MDPs under the weakest possible assumptions in the RL literature. This is achieved with a unique combination of existing techniques. Their value estimation scheme makes use of techniques from CAPI-QPI-PLAN to create a core set in addition to off-policy evaluation via importance sampling for value estimation at core state-action pairs. The authors also adequately cite related works.\n\n2. The submission is well written and, to the best of my knowledge, technically sound.\n\n3. In terms of the target accuracy and feature dimension, the results are significant especially in restrictive the setting of approximate policy completeness and local access to the simulator that they consider.\n\nWEAKNESSES:\n1. The relevance and impact on the theoretical analysis of the off-policy evaluation scheme is not fully accessible. It would help if the authors could clarify (with pointers in the main text), how this routine helps conserve samples in theory. Precisely, does the sample complexity get worse if the authors simply apply CAPI-QPI-PLAN?\n\n2. In Line 268, it should be $\\frac{1}{K}$ not $\\frac{1}{k}$ in the definition of $\\overline{\\pi}_{K}$.\n\nQUESTIONS:\n1. The nature of the output policy in Line 268 is a bit ambiguous and I spent a considerable amount of time trying to understand the meaning of the notation $\\frac{1}{K}\\sum_{k=0}^{K-1}\\pi_{k}$ means.\n\n    On one hand, the authors refer to the said notation as a mixture policy, which to my knowledge means that at deployment, the agent flips a coin at the beginning of each episode to decide which of the $K$ policies to use. In this case, it is unclear how you define $v_{\\overline{\\pi}\\_{K}}$, and more importantly, such policies are not suitable for constrained MDPs, especially with safety constraints as they are only required to perform well on average.\n\n    On the other hand, the notation seems to mean that $\\overline{\\pi}\\_{K}$ is an element-wise average for all state-action pairs. In this case, the policy is stochastic and $v_{\\overline{\\pi}\\_{K}}$ well defined.\n\n2. The complexity guarantees has worse dependence on the effective horizon $(1-\\gamma)$ e.g. $(1-\\gamma)^{-11}$ in Theorem 1 and $(1-\\gamma)^{-14}$  in Theorem 2. Is this a consequence of the method or an artefact of the analysis? Can this be improved?\n\n3. In Remark 3, the authors mention that the Slater's constant, which is required by their algorithm, can be approximated by another algorithm. Is this approximation trivial? How will the approximation error influence the current results?\n\n4. In appendix A, the authors provide Confident-NPG, a version of \nConfident-NPG-CMDP for the single-reward (or unconstrained MDP) case. \nFurthermore, their performance guarantees for the latter algorithm appear to be \nbased on first relating the former to CAPI-QPI-PLAN. Therefore, to \nbetter understand the contributions of this paper in the constrained case, the following questions are based on Confident-NPG. Can the authors kindly clarify the following:\n\n    - Why is $c > 0$ interesting to study?\n\n    - What is a suitable choice of $c$ and how does tuning this parameter show up in the performance guarantees?\n\n    - Can the authors comment on the memory requirement of Confident-NPG?\n\nLIMITATIONS:\nThis is a purely theory paper so there are no direct \nsocietal impacts. However, I would recommend that the authors highlight the  \ndependence on the effective horizon in their discussion. More \nimportantly, I recommend that the authors elaborate on the memory and \ncomputational complexity as these contribute to the efficiency of their method.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}