Title: Deterministic Policies for Constrained Reinforcement Learning in Polynomial Time
Venue: NeurIPS 2024
Accepted: yes
Keywords: Constrained Reinforcement Learning

Approximation Algorithms

Computational Complexity
Abstract: We present a novel algorithm that efficiently computes near-optimal deterministic policies for constrained reinforcement learning (CRL) problems. Our approach combines three key ideas: (1) value-demand augmentation, (2) action-space approximate dynamic programming, and (3) time-space rounding. Our algorithm constitutes a fully polynomial-time approximation scheme (FPTAS) for any time-space recursive (TSR) cost criteria. A TSR criteria requires the cost of a policy to be computable recursively over both time and (state) space, which includes classical expectation, almost sure, and anytime constraints. Our work answers three open questions spanning two long-standing lines of research: polynomial-time approximability is possible for 1) anytime-constrained policies, 2) almost-sure-constrained policies, and 3) deterministic expectation-constrained policies.

Deterministic Policies for Constrained Reinforcement Learning in Polynomial Time

Abstract

We present a novel algorithm that efficiently computes near-optimal deterministic policies for constrained reinforcement learning (CRL) problems. Our approach combines three key ideas: (1) value-demand augmentation, (2) action-space approximate dynamic programming, and (3) time-space rounding. Our algorithm constitutes a fully polynomial-time approximation scheme (FPTAS) for any timespace recursive (TSR) cost criteria. A TSR criteria requires the cost of a policy to be computable recursively over both time and (state) space, which includes classical expectation, almost sure, and anytime constraints. Our work answers three open questions spanning two long-standing lines of research: polynomial-time approximability is possible for 1) anytime-constrained policies, 2) almost-sure-constrained policies, and 3) deterministic expectation-constrained policies.

1 Introduction

Constrained Reinforcement Learning (CRL) traditionally produces stochastic, expectationconstrained policies that can behave undesirably - imagine a self-driving car that randomly changes lanes or runs out of fuel. However, artificial decision-making systems must be predictable, trustworthy, and robust. One approach to ensuring these qualities is to focus on deterministic policies, which are inherently predictable, easily implemented [19], reliable for autonomous vehicles [30, 23], and effective for multi-agent coordination [38]. Similarly, almost sure and anytime constraints [36] provide inherent trustworthiness and robustness, essential for applications in medicine [15, 37, 32], disaster relief [18, 50, 45], and resource management [35, 34, 40, 7]. Despite the advantages of deterministic policies and stricter constraints, even the computation of approximate solutions has remained an open challenge since NP-hardness was proven nearly 25 years ago [19]. Our work addresses this challenge by studying the computational complexity of computing deterministic policies for general constraint criteria.

Consider a constrained Markov Decision Process (cMDP) denoted by M . Let C represent an arbitrary cost criterion and B be the available budget. We focus on the set of deterministic policies denoted by Π D . Our objective is to compute: max π ∈ Π D V π M s.t. C π M ≤ B , where V π M is the value and C π M is the cost of π in M . This objective generalizes the example of a self-driving car calculating the fastest fixed route without running out of fuel. Our main question is the following:

Can near-optimal deterministic policies for constrained reinforcement learning problems be computed in polynomial time?

Although optimal stochastic policies for expectation-constrained problems are efficiently computable [3], the situation drastically changes when we require deterministic policies and general constraints. Computing optimal deterministic policies is NP-hard for most popular constraints, including expectation [19], chance [51], almost sure, and anytime constraints [36]. This complexity remains even if we relax our goal to finding just one feasible policy, provided that we are dealing

with a single chance constraint [51], or at least two of the other mentioned constraints [36]. Beyond these computational challenges, traditional solution methods, such as backward induction [41, 3], fail to apply due to the cyclic dependencies among subproblems: the value of any decision may depend on the costs of both preceding and concurrent decisions, preventing a solution from being computed in a single backward pass.

Past work. Past approaches fail to simultaneously achieve computational efficiency, feasibility, and optimality. Optimal and feasible algorithms, albeit inefficient, utilize Mixed-Integer Linear Programs [17] and Dual-guided heuristic forward searches [29] for expectation-constraints, and cost-augmented MDPs for almost sure [11] and anytime constraints [36]. Conversely, optimal and efficient, though infeasible, algorithms are known for expectation [43], almost sure, and anytime constraints [36]. A fully polynomial-time approximation scheme (FPTAS) [49] is known for expectation constraints, but it requires strong assumptions such as a constant horizon [31]. Balancing computational efficiency, feasibility, and optimality remains the bottleneck to efficient approximation.

Our contributions. We present an FPTAS for computing deterministic policies under any timespace recursive (TSR) constraint criteria. A TSR criteria requires the cost of a policy to be computable recursively in both time and (state) space, which captures expectation, almost sure, and anytime constraints. Since non-TSR criteria, such as chance constraints [51], are provably inapproximable, TSR seems pivotal for efficient computation. Overall, our general framework answers three open complexity questions spanning two longstanding lines of work: we prove polynomial-time approximability for 1) anytime-constrained policies, 2) almost-sure-constrained policies, and 3) deterministic expectation-constrained policies, which have been open for nearly 25 years [19].

Our approach breaks down into three main ideas: (1) value-demand augmentation, (2) action-space approximate dynamic programming, and (3) time-space rounding. We augment the states with value demands and the actions with future value demands to break cyclic subproblem dependencies, enabling dynamic programming methods. Importantly, we use values because they can be rounded without compromising feasibility [36] and can capture constraints that are not predictable from cumulative costs. However, this results in an exponential action space that makes solving the Bellman operator as hard as the knapsack problem. By exploiting the space-recursive nature of the criterion, we can efficiently approximate the Bellman operator with dynamic programming. Finally, rounding value demands result in approximation errors over both time and space, but carefully controlling these errors ensures provable guarantees.

1.1 Related work

Approximate packing. Many stochastic packing problems, which generalize the knapsack problem, are captured by our problem. Dean et al. [16], Frieze and Clarke [21] derived optimal approximation ratio algorithms for stochastic packing and integer packing with multiple constraints, respectively. Yang et al. [52], Bhalgat et al. [6] designed efficient approximation algorithms for variations of the stochastic knapsack problem. Then, Halman et al. [27] derived an FPTAS for a general class of stochastic dynamic programs, which was then further improved in [26, 1]. These methods require a single-dimensional state space that captures the constraint. In contrast, our problems have an innate state space in addition to the constraint. Our work forms a similar general dynamic programming framework for the more complex MDP setting.

Constrained RL. It is known that stochastic expectation-constrained policies are polynomial-time computable via linear programming [3], and many planning and learning algorithms exist for them [39, 46, 8, 28]. Some learning algorithms can even avoid violation during the learning process under certain assumptions [48, 4]. Furthermore, Brantley et al. [10] developed no-regret algorithms for cMDPs and extended their algorithms to the setting with a constraint on the cost accumulated over all episodes, which is called a knapsack constraint [10, 13].

Safe RL. The safe RL community [22, 25] has mainly focused on no-violation learning for stochastic expectation-constrained policies [14, 9, 2, 12, 5] and solving chance constraints [47, 53], which capture the probability of entering unsafe states. Performing learning while avoiding dangerous states [53] is a special case of expectation constraints that has also been studied [42, 44] under

non-trivial assumptions. In addition, instantaneous constraints, which require the expected cost to be within budget at all times, have also been studied [33, 20, 24].

2 Cost criteria

In this section, we formalize our problem setting. We also define our conditions for cost criteria.

Constrained Markov decision processes. A(tabular, finite-horizon) Constrained Markov Decision Process (cMDP) is a tuple M = ( S , A , P, r, c, H ) , where (i) S is the finite set of states , (ii) A is the finite set of actions , (iii) P h ( s, a ) ∈ ∆( S ) is the transition distribution, (iv) r h ( s, a ) ∈ R is the reward , (v) c h ( s, a ) ∈ R is the cost , and (vi) H ∈ N is the finite time horizon . We let S := |S| , A := |A| , [ H ] := { 1 , . . . , H } , and M denote the set of all cMDPs. We also let r max def = max h,s,a | r h ( s, a ) | denote the maximum magnitude reward, r min def = min h,s,a r h ( s, a ) denote the true minimum reward, and p min def = min h,s,a,s ′ P h ( s ′ | s, a ) denote the minimum transition probability. Since S is a finite set, we often assume S = [ S ] WLOG. Lastly, for any predicate p , we use the Iverson bracket notation [ p ] to denote 1 if p is true and 0 otherwise, and we let χ p denote the characteristic function which evaluates to 0 if p is true and ∞ otherwise.

Interaction protocol. The agent interacts with M using a policy π = ( π h ) H h =1 . In the fullest generality, π h : H h → ∆( A ) is a mapping from the observed history at time h to a distribution of actions. In contrast, a deterministic policy takes the form π h : H h →A . We let Π denote the set of all possible policies and Π D denote the set of all deterministic policies. The agent starts at the initial state s 0 ∈ S with observed history τ 1 = ( s 0 ) . For any h ∈ [ H ] , the agent chooses an action a h ∼ π h ( τ h ) . Then, the agent receives immediate reward r h ( s h , a h ) and cost c h ( s h , a h ) . Lastly, M transitions to state s h +1 ∼ P h ( s h , a h ) and the agent updates the history to τ h +1 = ( τ h , a h , s h +1 ) . This process is repeated for H steps; the interaction ends once s H +1 is reached.

Objective. For any cost criterion C : M× Π → R and budget B ∈ R , the agent's goal is to compute a solution to the following optimization problem:

Here, P π M denotes the probability law over histories induced from the interaction of π with M , and E π M denotes the expectation defined by this law. We let V π M def = E π M [ ∑ H t =1 r t ( s t , a t ) ] denote the value of a policy π , and V ∗ M denote the optimal solution value to (CON).

Cost criteria. We consider a broad family of cost criteria that satisfy a strengthening of the standard policy evaluation equations [41]. This strengthening requires not only the cost of a policy to be computable recursively in the time horizon, but at each time the cost should also break down recursively in (state) space.

Definition 1 (TSR) . We call a cost criterion C time-recursive (TR) if for any cMDP M and policy π ∈ Π D , π 's cost decomposes recursively into C π M = C π 1 ( s 0 ) . Here, C π H +1 ( · ) = 0 and for any h ∈ [ H ] and τ h ∈ H h ,

where s = s h ( τ h ) , a = π h ( τ h ) , and f is a non-decreasing function 1 computable in O ( S ) time. For technical reasons, we also require that f ( x ) = ∞ whenever ∞∈ x .

We further say C is time-space-recursive (TSR) if the f term above is equal to g τ h ,a h (1) . Here, g τ h ,a h ( S +1) = 0 and for any t ≤ S ,

where α is a non-decreasing function, and both α, β are computable in O (1) time. We also assume that α ( · , ∞ ) = ∞ , and β satisfies α ( β (0 , · ) , x ) = x to match f 's condition.

Since the TR condition is a slight generalization of traditional policy evaluation, it is easy to see that we can solve for minimum-cost policies using backward induction.

Proposition 1 (TR Intuition) . If C is TR, then C satisfies the usual optimality equations. Furthermore, arg min π ∈ Π D C π M can be computed using backward induction in O ( HS 2 A ) time.

Although the TR condition is straightforward, the TSR condition is more strict. We will see the utility of the TSR condition in Section 4 when computing Bellman updates. For now, we point out that the TSR condition is not too restrictive: it is satisfied by many popular criteria studied in the literature.

Proposition 2 (TSR examples) . The following classical constraints can be modeled by a TSR cost constraint.

(Expectation Constraints) are captured by C π M def = E π M [ ∑ H h =1 c h ( s h , a h ) ] ≤ B . We see C is TSR by defining α ( x, y ) def = x + y and β ( x, y ) def = xy .

(Almost Sure Constraints) are captured by C π M def = max τ ∈H H +1 , P π M [ τ ] > 0 ∑ H h =1 c h ( s h , a h ) ≤ B . We

(Anytime Constraints) are captured by C π M def = max t ∈ [ H ] max τ ∈H H +1 , P π M [ τ ] > 0 ∑ t h =1 c h ( s h , a h ) ≤

Remark 1 (Extensions) . Our methods can also handle stochastic costs and infinite discounting. We defer the details to Appendix F. Moreover, we can handle multiple constraints using vector-valued criteria so long as the comparison operator is a total ordering of the vector space.

Remark 2 (Inapproximability) . Our methods cannot handle chance constraints or more than one of our example constraints. However, this is not a limitation of our framework as the problem becomes provably inapproximable under said constraints [51, 36].

3 Covering algorithm

In this section, we propose an algorithm to solve (CON). Our approach relies on converting the original problem into an equivalent covering problem that can be solved using an unconstrained MDP. This covering MDP is derived using the key idea of value augmentation.

Packing and covering. We can view (CON) as a packing program , which wishes to maximize V π M subject to C π M ≤ B . However, we could also tackle the problem by reversing the objective: attempt to minimize C π M subject to V π M ≥ V ∗ M . If (CON) is feasible, then any optimal solution π to this covering program satisfies V π M ≥ V ∗ M and C π M ≤ B . Thus, we can solve the original packing program by solving the covering program.

Proposition 3 (Packing-Covering Reduction) . Suppose that C ∗ M def = min π ∈ Π D C π M s.t. V π M ≥ V ∗ M Then, C ∗ ≤ B ⇐⇒ V ∗ > -∞ . Furthermore, if V ∗ > -∞ , then,

Thus, any solution to the covering program is a solution to the packing program.

We focus on the covering program for several reasons. To optimize the value recursively, we would need to predict the final cost resulting from intermediate decisions to ensure feasibility. Generally, such predictions would require strict assumptions on the cost criteria. By treating the value as the constraint instead, we only need to assume the cost can be optimized efficiently. Moreover, values are well understood in RL and are more amenable to approximation [36]. Thus, the covering program allows us to capture many criteria, ensure feasibility, and compute accurate value approximations.

Algorithm 1 Reduction to RL

Value augmentation. We can solve the covering program by solving a cost-minimizing MDP ¯ M . The key idea is to augment the state space with value demands, ( s, v ) . Then, the agent can recursively reason how to minimize its cost while meeting the current value demand. If the agent starts at ( s 0 , V ∗ M ) , then an optimal policy for ¯ M should be a solution to the covering program.

The key invariant we desire is that any feasible policy π for ¯ M should satisfy ¯ V π h ( s, v ) ≥ v . To ensure this invariance, we recall the policy evaluation equations [41]. If π h ( s ) = a , then,

For the value invariant to be satisfied, it suffices for the agent to choose an action a and commit to future value demands v s ′ satisfying,

We can view choosing future value demands as part of the agent's augmented actions. Then, at any augmented state ( s, v ) , the agent's augmented action space includes all ( a, v ) ∈ A × R S satisfying (DEM). When M transitions to s ′ ∼ P h ( s, a ) , the agent's new augmented state should consist of the environment's new state in addition to its chosen demand for that state, ( s ′ , v s ′ ) . Putting these pieces together yields the definition of the cover MDP, Definition 2.

Definition 2 (Cover MDP) . The cover MDP ¯ M def = ( ¯ S , ¯ A , ¯ P, ¯ c, H ) where,

The objective for ¯ M is to minimize the cost function ¯ C def = C ¯ M with modified base case ¯ C π H +1 ( s, v ) def = χ { v ≤ 0 } .

Covering algorithm. Importantly, the action space definition ensures the value constraint is satisfied. Meanwhile, the minimum cost objective ensures optimal cost. So long as our cost is TR, ¯ M can be solved using fast RL methods instead of the brute force computation required for general covering programs. These properties ensure our method, Algorithm 1, is correct.

Theorem 1 (Reduction) . If SOLVE is any finite-time MDP solver, then Algorithm 1 correctly solves (CON) in finite time for any TR cost criterion.

Remark 3 (Execution) . Given a value-augmented policy π output from Algorithm 1, the agent can execute π using Algorithm 2. To compute V ∗ M as the starting value, it suffices for the agent to compute,

This computation can be easily done given ¯ C ∗ 1 ( s 0 , · ) in O ( |V| ) time.

Algorithm 2 Augmented interaction

4 Fast Bellman updates

In this section, we present an algorithm to solve ¯ M from Definition 2 efficiently. Although the Bellman updates can be as hard to solve as the knapsack problem, we use ideas from knapsack approximation algorithms to create an efficient method. Our approach exploits (SR) through approximate dynamic programming on the action space.

Even if V were small, solving ¯ M would still be challenging due to the exponentially large action space. Even a single Bellman update requires the solution of a constrained optimization problem:

Above, we used the fact that ( s ′ , v ′ ) ∈ ¯ P h (( s, v ) , ( a, v )) iff s ′ ∈ P h ( s, a ) and v ′ = v s ′ to simplify f 's input. Observe that even when each v s ′ only takes on two possible values, { 0 , w s ′ } , the optimization above can capture the minimization version of the knapsack problem, implying that it is NP-hard to compute.

Recursive approach. Fortunately, we can use the connection to the Knapsack problem positively to efficiently approximate the Bellman update. For any fixed ( s, v ) ∈ ¯ S and a ∈ A , we focus on the inner constrained minimization over v :

We use (SR) to transform this minimization over v into a sequential decision-making problem that decides each v s ′ . As above, we can use the definition of ¯ P to simplify g ( s,v ) , ( a, v ) h ( t, v ′ ) into a function of t alone:

Since v only constrains the valid ( a, v ) pairs, we can discard v and use the simplified notation g s,a h, v ( t ) instead of g ( s,v ) , ( a, v ) h ( t ) . It is then clear that we can recursively optimize the value of v t by focusing on g s,a h, v ( t ) .

To recursively encode the value constraint, we can record the partial value u = r h ( s, a ) + ∑ t -1 s ′ =1 P h ( s ′ | s, a ) v s ′ that we have accumulated so far. Then, we can check if our choices for v satisfied the constraint with the inequality u ≥ v . The formal recursion is defined in Definition 3.

Definition 3. For any h ∈ [ H ] , s ∈ S , v ∈ V , and u ∈ R , we define, g s,a h,v ( S +1 , u ) = χ { u ≥ v } and for t ≤ S ,

Recursive rounding. This approach can still be slow due to the exponential number of partial values u induced. Similarly to the knapsack problem, the key is to round each input u to ensure fewer subproblems. Unlike the knapsack problem, however, we do not have an easily computable lower bound on the optimal value. Thus, we turn to a more aggressive recursive rounding. Since rounding may cause originally feasible values to violate the demand constraint, we also relax the demand constraint to u ≥ κ ( v ) for some lower bound function κ .

Algorithm 3 Approx Bellman Update

Algorithm 4 Approx Solve

Input: ( ¯ M, ¯ C )

Definition 4. Fix a rounding function ⌊·⌋ G and a lower bound function κ . For any h ∈ [ H ] , s ∈ S , v ∈ V , and u ∈ R , we define, ˆ g s,a h,v ( S +1 , u ) = χ { u ≥ v } and for t ≤ S ,

Fortunately, the approximate version behaves similarly to the original. The main difference is the constraint now ensures the rounded sums are at least the value lower bound. This is formalized in Lemma 1.

Lemma 1. For any t ∈ [ S +1] and u ∈ R , we have that,

To turn this recursion into a usable dynamic programming algorithm, we must also pre-compute the inputs to any sub-computation. Unlike in standard RL, this computation must be done with a forward recursion. The details for the approximate Bellman update are given in Definition 5.

Definition 5 (Approx Bellman) . For any h ∈ [ H ] , s ∈ S , and a ∈ A , we define ˆ U s,a h (1) def = { r h ( s, a ) } and for any t ∈ [ S ] ,

Then, an approximation to the Bellman update can be computed using Algorithm 3. 2

Proposition 4. Algorithm 4 runs in O ( HS 2 A |V| 2 ˆ U ) time, where ˆ U def = max h,s,a | ˆ U s,a h | . When ⌊·⌋ G and κ are the identity function, Algorithm 4 outputs an optimal solution to ¯ M .

Remark 4 (Speedups) . The runtime of our methods can be quadratically improved by rounding the differences instead of the sums. We defer the details to Appendix F.

5 Approximation algorithms

In this section, we present our approximation algorithms for solving (CON). We carefully round the value demands over both time and space to induce an approximate MDP. Solving this approximate MDP with Algorithm 4 yields our FPTAS.

Although we can avoid exponential-time Bellman updates, the running time of the approximate Bellman update will still be slow if |V| is large. To reduce the complexity, we instead use a smaller set of approximate values by rounding elements of |V| . By rounding down, we effectively relax the value-demand constraint. More aggressive rounding not only leads to smaller augmented state spaces but also to smaller cost policies. The trade-off is aggressive rounding leads to weaker guarantees on the computed policy's value. Thus, it is critical to carefully design the rounding and lower bound functions to balance this trade-off.

Value approximation. Given a rounding down function ⌊·⌋ G , we would ideally use the rounded set { ⌊ v ⌋ G | v ∈ V } to form our approximate state space. To avoid having to compute V explicitly, we instead use the rounded superset { ⌊ v ⌋ G | v ∈ [ v min , v max ] } , where v min and v max are bounds on the extremal values that we specify later. To ensure we can use Algorithm 4 to find solutions efficiently, we must also relax the augmented action space to only include vectors that lead to feasible subproblems for (ADP). From Lemma 1, we know this is exactly the set of ( a, ˆ v ) for which ˆ σ s,a h, ˆ v (1 , r h ( s, a )) ≥ κ ( v ) . Combining these ideas yields the new approximate MDP, defined in Definition 6.

Definition 6 (Approximate MDP) . Given a rounding function ⌊·⌋ G and lower bound function κ , the approximate MDP ˆ M def = ( ˆ S , ˆ A , ˆ P, ˆ c, H ) where,

The objective for ˆ M is to minimize the cost function ˆ C def = C ˆ M with modified base case ˆ C π H +1 ( s, ˆ v ) def = χ { ˆ v ≤ 0 } .

We can show that rounding down in Definition 6 achieves our goal of producing smaller cost policies. This ensures feasibility is even easier to achieve. We formalize this observation in Lemma 2.

Lemma 2 (Optimistic Costs) . For our later choices of ⌊·⌋ G and κ , the following holds: for any h ∈ [ H +1] and ( s, v ) ∈ ¯ S , we have ˆ C ∗ h ( s, ⌊ v ⌋ G ) ≤ ¯ C ∗ h ( s, v ) .

Thus, Algorithm 5 always outputs a policy with better than optimal cost when the instance is feasible, V ∗ M > -∞ . If the instance is infeasible, all policies have cost larger than B by definition and so Algorithm 5 correctly indicates the instance is infeasible. The remaining question is whether Algorithm 5 outputs policies having near-optimal value.

Time-Space errors. To assess the optimality gap of Algorithm 5 policies, we must first explore the error accumulated by our rounding approach. Rounding each value naturally accumulates approximation error over time. Rounding the partial values while running Algorithm 3 accumulates additional error over (state) space. Thus, solving ˆ M using Algorithm 4 accumulates error over both time and space, unlike other approximate methods in RL. As a result, our rounding and threshold functions will generally depend on both H and S .

Arithmetic rounding. Our first approach is to round each value down to its closest element in a δ -cover. This guarantees that v -δ ≤ ⌊ v ⌋ G ≤ v . Thus, ⌊ v ⌋ G is an underestimate that is not too far from the true value. By setting δ to be inversely proportional to SH , we control the errors over time and space. The lower bound must also be a function of S since it controls the error over space.

Algorithm 5 Approximation Scheme

Definition 7 (Additive Approx) . Fix ϵ > 0 . We define,

where δ def = ϵ H ( S +1)+1 , v min def = -Hr max , and v max def = Hr max .

Theorem 2 (Additive FPTAS) . For any ϵ > 0 , Algorithm 5 using Definition 7 given any cMDP M and TSR criteria C either correctly outputs the instance is infeasible, or produces a policy π satisfying ˆ V π ≥ V ∗ M -ϵ in O ( H 7 S 5 Ar 3 max /ϵ 3 ) time. Thus, it is an additive-FPTAS for the class of cMDPs with polynomial-bounded r max and TSR criteria.

Geometric rounding. Since the arithmetic approach can be slow when r max is large, we can instead round values down to their closest power of 1 / (1 -δ ) . This guarantees the number of approximate values needed is upper bounded by a function of log( r max ) , which is polynomial in the input size. We choose a geometric scheme satisfying v (1 -δ ) ≤ ⌊ v ⌋ G ≤ v so that the rounded value is an underestimate and a relative approximation to the true value. To ensure this property, we must now require that all rewards are non-negative.

Definition 8 (Relative Approx) . Fix ϵ > 0 . We define,

Theorem 3 (Relative FPTAS) . For ϵ > 0 , Algorithm 5 using Definition 8 given any cMDP M and TSR criteria C either correctly outputs the instance is infeasible, or produces a policy π satisfying ˆ V π ≥ V ∗ M (1 -ϵ ) in O ( H 7 S 5 A log ( r max /r min p min ) 3 /ϵ 3 ) time. Thus, it is a relative-FPTAS for the class of cMDPs with non-negative rewards and TSR criteria.

Remark 5 (Assumption Necessity) . We also note the mild reward assumptions we made to guarantee efficiency are unavoidable. Without reward bounds, (CON) captures the knapsack problem which does not admit additive approximations. Similarly, without non-negativity, relative approximations for maximization problems are generally not computable.

6 Conclusions

In this paper, we studied the computational complexity of computing deterministic policies for CRL problems. Our main contribution was the design of an FPTAS, Algorithm 5, that solves (CON) for any cMPD and TSR criteria under mild reward assumptions. In particular, our method is an additiveFPTAS if the cMDP's rewards are polynomially bounded, and is a relative-FPTAS if the cMDP's rewards are non-negative. We note these assumptions are necessary for efficient approximation, so our algorithm achieves the best approximation guarantees possible under worst-case analysis. Moreover, our algorithmic approach, which uses approximate dynamic programming over time and the state space, highlights the importance of the TSR condition in making (CON) tractable. Our work finally resolves the long-standing open questions of polynomial-time approximability for 1) anytimeconstrained policies, 2) almost-sure-constrained policies, and 3) deterministic expectation-constrained policies.

Future work. Several interesting questions remain unanswered. First, it remains unresolved whether an FPTAS exists asymptotically faster than ours. Second, whether our TSR condition is necessary for efficient computation or whether a more general condition could be derived is unclear. Lastly, it is open whether there exist algorithms that can feasibly handle multiple constraints from Proposition 2. Although computing feasible policies for multiple constraints is NP-hard, special cases may be approximable efficiently. Moreover, an average-case or smoothed-case analysis could circumvent this worst-case hardness.

Acknowledgments and Disclosure of Funding

This work was supported in part by NSF grant 2023239.

References

T. Alon and N. Halman. Automatic generation of fptases for stochastic monotone dynamic programs made easier. SIAM Journal on Discrete Mathematics , 35(4):2679-2722, 2021. doi: 10.1137/19M1308633. URL https://doi.org/10.1137/19M1308633 .

M. Alshiekh, R. Bloem, R. Ehlers, B. Könighofer, S. Niekum, and U. Topcu. Safe reinforcement learning via shielding. Proceedings of the AAAI Conference on Artificial Intelligence , 32 (1), Apr. 2018. doi: 10.1609/aaai.v32i1.11797. URL https://ojs.aaai.org/index.php/ AAAI/article/view/11797 .

E. Altman. Constrained Markov Decision Processes . Chapman and Hall/CRC, 1999. doi: 10.1201/9781315140223.

Q. Bai, A. Singh Bedi, and V. Aggarwal. Achieving zero constraint violation for constrained reinforcement learning via conservative natural policy gradient primal-dual algorithm. Proceedings of the AAAI Conference on Artificial Intelligence , 37(6):6737-6744, 6 2023. doi: 10.1609/aaai. v37i6.25826. URL https://ojs.aaai.org/index.php/AAAI/article/view/25826 .

F. Berkenkamp, M. Turchetta, A. Schoellig, and A. Krause. Safe model-based reinforcement learning with stability guarantees. In I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/ paper_files/paper/2017/file/766ebcd59621e305170616ba3d3dac32-Paper.pdf .

A. Bhalgat, A. Goel, and S. Khanna. Improved approximation results for stochastic knapsack problems. In Proceedings of the Twenty-Second Annual ACM-SIAM Symposium on Discrete Algorithms , SODA '11, page 1647-1665, USA, 2011. Society for Industrial and Applied Mathematics.

A. Bhatia, P. Varakantham, and A. Kumar. Resource constrained deep reinforcement learning. Proceedings of the International Conference on Automated Planning and Scheduling , 29(1): 610-620, 5 2021. doi: 10.1609/icaps.v29i1.3528. URL https://ojs.aaai.org/index. php/ICAPS/article/view/3528 .

V. Borkar. An actor-critic algorithm for constrained markov decision processes. Systems & Control Letters , 54(3):207-213, 2005. ISSN 0167-6911. doi: https://doi.org/10.1016/j. sysconle.2004.08.007. URL https://www.sciencedirect.com/science/article/pii/ S0167691104001276 .

D. M. Bossens and N. Bishop. Explicit explore, exploit, or escape (e4): Near-optimal safetyconstrained reinforcement learning in polynomial time. Mach. Learn. , 112(3):817-858, 6 2022. ISSN 0885-6125. doi: 10.1007/s10994-022-06201-z. URL https://doi.org/10.1007/ s10994-022-06201-z .

K. Brantley, M. Dudík, T. Lykouris, S. Miryoosefi, M. Simchowitz, A. Slivkins, and W. Sun. Constrained episodic reinforcement learning in concave-convex and knapsack settings. In NeurIPS , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ bc6d753857fe3dd4275dff707dedf329-Abstract.html .

A. Castellano, H. Min, E. Mallada, and J. A. Bazerque. Reinforcement learning with almost sure constraints. In R. Firoozi, N. Mehr, E. Yel, R. Antonova, J. Bohg, M. Schwager, and M. Kochenderfer, editors, Proceedings of The 4th Annual Learning for Dynamics and Control Conference , volume 168 of Proceedings of Machine Learning Research , pages 559-570. PMLR, 6 2022. URL https://proceedings.mlr.press/v168/castellano22a.html .

R. Cheng, G. Orosz, R. M. Murray, and J. W. Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. Proceedings of the AAAI Conference on Artificial Intelligence , 33(01):3387-3395, Jul. 2019. doi: 10.1609/aaai.v33i01. 33013387. URL https://ojs.aaai.org/index.php/AAAI/article/view/4213 .

W. C. Cheung. Regret minimization for reinforcement learning with vectorial feedback and complex objectives. In Advances in Neural Information Processing Systems , volume 32, 2019. URL https://proceedings.neurips.cc/paper/2019/file/ a02ffd91ece5e7efeb46db8f10a74059-Paper.pdf .

Y. Chow, O. Nachum, E. Duenez-Guzman, and M. Ghavamzadeh. A lyapunov-based approach to safe reinforcement learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. CesaBianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/ paper/2018/file/4fe5149039b52765bde64beb9f674940-Paper.pdf .

A. Coronato, M. Naeem, G. De Pietro, and G. Paragliola. Reinforcement learning for intelligent healthcare applications: A survey. Artificial Intelligence in Medicine , 109:101964, 2020. ISSN 0933-3657. doi: https://doi.org/10.1016/j.artmed.2020.101964. URL https://www. sciencedirect.com/science/article/pii/S093336572031229X .

B. C. Dean, M. X. Goemans, and J. Vondrák. Adaptivity and approximation for stochastic packing problems. In Proceedings of the Sixteenth Annual ACM-SIAM Symposium on Discrete Algorithms , SODA '05, page 395-404, USA, 2005. Society for Industrial and Applied Mathematics. ISBN 0898715857.

D. A. Dolgov and E. H. Durfee. Stationary deterministic policies for constrained mdps with multiple rewards, costs, and discount factors. In IJCAI , volume 19, pages 1326-1331, 2005.

C. Fan, C. Zhang, A. Yahja, and A. Mostafavi. Disaster city digital twin: A vision for integrating artificial and human intelligence for disaster management. International Journal of Information Management , 56:102049, 2021. ISSN 0268-4012. doi: https://doi.org/10.1016/ j.ijinfomgt.2019.102049. URL https://www.sciencedirect.com/science/article/ pii/S0268401219302956 .

E. A. Feinberg. Constrained discounted markov decision processes and hamiltonian cycles. Mathematics of Operations Research , 25(1):130-140, 2000. doi: 10.1287/moor.25.1.130.15210. URL https://doi.org/10.1287/moor.25.1.130.15210 .

J. F. Fisac, N. F. Lugovoy, V. Rubies-Royo, S. Ghosh, and C. J. Tomlin. Bridging hamiltonjacobi safety analysis and reinforcement learning. In 2019 International Conference on Robotics and Automation (ICRA) , page 8550-8556. IEEE Press, 2019. doi: 10.1109/ICRA.2019.8794107. URL https://doi.org/10.1109/ICRA.2019.8794107 .

A. Frieze and M. Clarke. Approximation algorithms for the m-dimensional 0-1 knapsack problem: Worst-case and probabilistic analyses. European Journal of Operational Research , 15(1):100-109, 1984. ISSN 0377-2217. doi: https://doi.org/10.1016/0377-2217(84)90053-5. URL https://www.sciencedirect.com/science/article/pii/0377221784900535 .

J. García, Fern, and o Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research , 16(42):1437-1480, 2015. URL http://jmlr.org/ papers/v16/garcia15a.html .

F. Geißer, G. Povéda, F. Trevizan, M. Bondouy, F. Teichteil-Königsbuch, and S. Thiébaux. Optimal and heuristic approaches for constrained flight planning under weather uncertainty. Proceedings of the International Conference on Automated Planning and Scheduling , 30(1): 384-393, Jun. 2020. doi: 10.1609/icaps.v30i1.6684. URL https://ojs.aaai.org/index. php/ICAPS/article/view/6684 .

S. Gros, M. Zanon, and A. Bemporad. Safe reinforcement learning via projection on a safe set: Howto achieve optimality? IFAC-PapersOnLine , 53(2):8076-8081, 2020. ISSN 2405-8963. doi: https://doi.org/10.1016/j.ifacol.2020.12.2276. URL https://www.sciencedirect.com/ science/article/pii/S2405896320329360 . 21st IFAC World Congress.

S. Gu, L. Yang, Y. Du, G. Chen, F. Walter, J. Wang, and A. Knoll. A review of safe reinforcement learning: Methods, theory and applications, 2024. URL https://arxiv.org/abs/2205. 10330 .

N. Halman and G. Nannicini. Toward breaking the curse of dimensionality: An fptas for stochastic dynamic programs with multidimensional actions and scalar states. SIAM Journal on Optimization , 29(2):1131-1163, 2019. doi: 10.1137/18M1208423. URL https://doi.org/ 10.1137/18M1208423 .

N. Halman, D. Klabjan, C.-L. Li, J. Orlin, and D. Simchi-Levi. Fully polynomial time approximation schemes for stochastic dynamic programs. SIAM Journal on Discrete Mathematics , 28(4):1725-1796, 2014. doi: 10.1137/130925153. URL https://doi.org/10.1137/ 130925153 .

A. HasanzadeZonuzy, A. Bura, D. Kalathil, and S. Shakkottai. Learning with safety constraints: Sample complexity of reinforcement learning for constrained mdps. Proceedings of the AAAI Conference on Artificial Intelligence , 35(9):7667-7674, 5 2021. doi: 10.1609/aaai.v35i9.16937. URL https://ojs.aaai.org/index.php/AAAI/article/view/16937 .

S. Hong and B. C. Williams. An anytime algorithm for constrained stochastic shortest path problems with deterministic policies. Artificial Intelligence , 316:103846, 2023. ISSN 00043702. doi: https://doi.org/10.1016/j.artint.2022.103846. URL https://www.sciencedirect. com/science/article/pii/S0004370222001862 .

S. Hong, S. U. Lee, X. Huang, M. Khonji, R. Alyassi, and B. C. Williams. An anytime algorithm for chance constrained stochastic shortest path problems and its application to aircraft routing. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 475-481, 2021. doi: 10.1109/ICRA48506.2021.9561229.

M. Khonji, A. Jasour, and B. Williams. Approximability of constant-horizon constrained pomdp. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19 , pages 5583-5590. International Joint Conferences on Artificial Intelligence Organization, 7 2019. doi: 10.24963/ijcai.2019/775. URL https://doi.org/10.24963/ijcai. 2019/775 .

P. Kolesar. A markovian model for hospital admission scheduling. Management Science , 16 (6):B384-B396, 1970. ISSN 00251909, 15265501. URL http://www.jstor.org/stable/ 2628725 .

J. Li, D. Fridovich-Keil, S. Sojoudi, and C. J. Tomlin. Augmented lagrangian method for instantaneously constrained reinforcement learning problems. In 2021 60th IEEE Conference on Decision and Control (CDC) , page 2982-2989. IEEE Press, 2021. doi: 10.1109/CDC45484. 2021.9683088. URL https://doi.org/10.1109/CDC45484.2021.9683088 .

R. Li, Z. Zhao, Q. Sun, C.-L. I, C. Yang, X. Chen, M. Zhao, and H. Zhang. Deep reinforcement learning for resource management in network slicing. IEEE Access , 6:74429-74441, 2018. doi: 10.1109/ACCESS.2018.2881964.

H. Mao, M. Alizadeh, I. Menache, and S. Kandula. Resource management with deep reinforcement learning. In Proceedings of the 15th ACM Workshop on Hot Topics in Networks , HotNets '16, page 50-56, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450346610. doi: 10.1145/3005745.3005750. URL https: //doi.org/10.1145/3005745.3005750 .

J. McMahan and X. Zhu. Anytime-constrained reinforcement learning. In S. Dasgupta, S. Mandt, and Y. Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics , volume 238 of Proceedings of Machine Learning Research , pages 4321-4329. PMLR, 02-04 May 2024. URL https://proceedings.mlr.press/v238/mcmahan24a. html .

G. Paragliola, A. Coronato, M. Naeem, and G. De Pietro. A reinforcement learning-based approach for the risk management of e-health environments: A case study. In 2018 14th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS) , pages 711-716, 2018. doi: 10.1109/SITIS.2018.00114.

P. Paruchuri, M. Tambe, F. Ordonez, and S. Kraus. Towards a formalization of teamwork with resource constraints. In Third International Joint Conference on Autonomous Agents and Multiagent Systems , United States, 2004. IEEE Computer Society. Place of conference:USA.

S. Paternain, L. Chamon, M. Calvo-Fullana, and A. Ribeiro. Constrained reinforcement learning has zero duality gap. In Advances in Neural Information Processing Systems , volume 32, 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/ file/c1aeb6517a1c7f33514f7ff69047e74e-Paper.pdf .

H. Peng and X. Shen. Multi-agent reinforcement learning based resource management in mecand uav-assisted vehicular networks. IEEE Journal on Selected Areas in Communications , 39 (1):131-141, 2021. doi: 10.1109/JSAC.2020.3036962.

M. L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming . John Wiley & Sons, Inc., USA, 1st edition, 1994. ISBN 0471619779.

M. Roderick, V. Nagarajan, and Z. Kolter. Provably safe pac-mdp exploration using analogies. In A. Banerjee and K. Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , volume 130 of Proceedings of Machine Learning Research , pages 1216-1224. PMLR, 4 2021. URL https://proceedings.mlr.press/ v130/roderick21a.html .

M. A. Taleghan and T. G. Dietterich. Efficient exploration for constrained mdps. In 2018 AAAI Spring Symposium Series , 2018.

G. Thomas, Y. Luo, and T. Ma. Safe reinforcement learning by imagining the near future. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 13859-13869. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/ 73b277c11266681122132d024f53a75b-Paper.pdf .

Y. L. Tsai, A. Phatak, P. K. Kitanidis, and C. B. Field. Deep Reinforcement Learning for Disaster Response: Navigating the Dynamic Emergency Vehicle and Rescue Team Dispatch during a Flood. In AGU Fall Meeting Abstracts , volume 2019, pages NH33B-14, Dec. 2019.

S. Vaswani, L. Yang, and C. Szepesvari. Near-optimal sample complexity bounds for constrained mdps. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 3110-3122. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/ 2022/file/14a5ebc9cd2e507cd811df78c15bf5d7-Paper-Conference.pdf .

Y. Wang, S. S. Zhan, R. Jiao, Z. Wang, W. Jin, Z. Yang, Z. Wang, C. Huang, and Q. Zhu. Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 36593-36604. PMLR, 7 2023. URL https://proceedings.mlr.press/v202/wang23as.html .

H. Wei, X. Liu, and L. Ying. Triple-q: A model-free algorithm for constrained reinforcement learning with sublinear regret and zero constraint violation. In G. Camps-Valls, F. J. R. Ruiz, and I. Valera, editors, Proceedings of The 25th International Conference on Artificial Intelligence and Statistics , volume 151 of Proceedings of Machine Learning Research , pages 3274-3307. PMLR, 3 2022. URL https://proceedings.mlr.press/v151/wei22a.html .

D. P. Williamson and D. B. Shmoys. The Design of Approximation Algorithms . Cambridge University Press, USA, 1st edition, 2011. ISBN 0521195276.

C. Wu, B. Ju, Y. Wu, X. Lin, N. Xiong, G. Xu, H. Li, and X. Liang. Uav autonomous target search based on deep reinforcement learning in complex disaster scene. IEEE Access , 7: 117227-117245, 2019. doi: 10.1109/ACCESS.2019.2933002.

H. Xu and S. Mannor. Probabilistic goal markov decision processes. In Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence - Volume Volume Three , IJCAI'11, page 2046-2052. AAAI Press, 2011. ISBN 9781577355151.

S. Yang, S. Khuller, S. Choudhary, S. Mitra, and K. Mahadik. Correlated Stochastic Knapsack with a Submodular Objective. In S. Chechik, G. Navarro, E. Rotenberg, and G. Herman, editors, 30th Annual European Symposium on Algorithms (ESA 2022) , volume 244 of Leibniz International Proceedings in Informatics (LIPIcs) , pages 91:1-91:14, Dagstuhl, Germany, 2022. Schloss Dagstuhl - Leibniz-Zentrum für Informatik. ISBN 978-3-95977-247-1. doi: 10. 4230/LIPIcs.ESA.2022.91. URL https://drops.dagstuhl.de/opus/volltexte/2022/ 17029 .

W. Zhao, T. He, R. Chen, T. Wei, and C. Liu. State-wise safe reinforcement learning: a survey. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence , IJCAI '23, 2023. ISBN 978-1-956792-03-4. doi: 10.24963/ijcai.2023/763. URL https://doi.org/10.24963/ijcai.2023/763 .

A Appendix / supplemental material

B Proofs for Section 2

B.1 Proof of Proposition 1

The proof follows from the standard proof of backward induction [41]. The main ideas for the proof can also be seen in the proof of Lemma 4 and Lemma 5.

B.2 Proof of Proposition 2

Proof.

(Expectation Constraints) We claim that C π M captures expectation constraints. This is immediate as an expectation constraint takes the form E π M [ ∑ H h =1 c h ( s h , a h ) ] ≤ B and by definition C π M = E π M [ ∑ H h =1 c h ( s h , a h ) ] . Moreover, the standard policy evaluation equations for deterministic policies immediately imply,

Thus, (TR) holds. It is also easy to see that ∑ s ′ P h ( s ′ | s, a ) C π h +1 ( τ h , a, s ′ ) can be computed recursively state-wise by,

and so (SR) holds. The infinity conditions and non-decreasing requirements are also easy to verify.

(Almost Sure Constraints) We claim that C π M captures almost sure constraints. This is because that for tabular MDPs, P π M [ ∑ H h =1 c h ( s h , a h ) ≤ B ] = 1 if and only if for all τ ∈ H H +1 with P π M [ τ ] > 0 it holds that ∑ H h =1 c h ( s h , a h ) ≤ B if and only if C π M = max τ ∈H H +1 : P π M [ τ ] > 0 ∑ H h =1 c h ( s h , a h ) ≤ B .

Let c ( τ ) = ∑ H h =1 c h ( s h , a h ) denote the cost of a full history τ ∈ H H +1 and let c h : t ( τ ) = ∑ t k = h c k ( s k , a k ) denote the partial cost of τ from time h to time t . Our choice of α and β imply that,

To show that C π M satisfies (TR), we prove for all h ∈ [ H +1] and all τ h ∈ H h that

Then, we see that C π 1 ( s 0 ) = max τ ∈H H +1 : P π M [ τ | s 0 ] > 0 c 1: H ( τ ) = max τ ∈H H +1 : P π M [ τ ] > 0 ∑ H h =1 c h ( s h , a h ) = C π M . Thus, C π M satisfies (TR). Furthermore, it is clear that max s ′ ∈ P h ( s,a ) C π h +1 ( τ h , a, s ′ ) can be computed state-recursively by,

and so C π M satisfies (SR). The infinity conditions and non-decreasing requirements are also easy to verify.

We proceed by induction on h .

(Base Case) For the base case, we consider h = H +1 . Observe that for any history τ , we have c H +1: H ( τ ) = 0 since it is an empty sum. Then, by definition of C π M , we see that C π H +1 ( τ H +1 ) = 0 = max τ 0 = max τ c H +1: H ( τ ) .

(Inductive Step) For the inductive step, we consider h ≤ H . Let s = s h ( τ h ) and a = π h ( τ h ) . For any τ ∈ H H +1 for which P π M [ τ | τ h ] > 0 , we can decompose its cost by c h : H ( τ ) = c h ( s, a ) + c h +1: H ( τ ) . Since a is fixed, we can remove c h ( s, a ) from the optimization to get,

Next, we observe by the Markov property that P π M [ τ | τ h ] = ∑ s ′ P π M [ τ | τ h , a, s ′ ] P h ( s ′ | s, a ) . Thus, P π M [ τ | τ h ] > 0 if and only if there exists some s ′ ∈ P h ( s, a ) satisfying P π M [ τ | τ h , a, s ′ ] > 0 . This implies that,

By applying the induction hypothesis, we see that,

The second line used the induction hypothesis and the third line used the definition of C π M .

(Anytime Constraints) We claim that C π M captures anytime constraints. This is because that for tabular MDPs, P π M [ ∀ t ∈ [ H ] , ∑ t h =1 c h ( s h , a h ) ≤ B ] = 1 if and only if for all t ∈ [ H ] and τ ∈ H H +1 with P π M [ τ ] > 0 it holds that ∑ t h =1 c h ( s h , a h ) ≤ B if and only if C π M = max t ∈ [ H ] max τ ∈H H +1 : P π M [ τ ] > 0 ∑ t h =1 c h ( s h , a h ) ≤ B .

Our choice of α and β imply that,

To show that C π M satisfies (TR), we show that for all h ∈ [ H +1] and all τ h ∈ H h that

Then, we see that C π 1 ( s 0 ) = max t ∈ [ H ] max τ ∈H H +1 : P π M [ τ | s 0 ] > 0 c 1: t ( τ ) = max t ∈ [ H ] max τ ∈H H +1 : P π M [ τ ] > 0

∑ t h =1 c h ( s h , a h ) = C π M . Thus, C π M satisfies (TR). Furthermore, it is clear that max(0 , max s ′ ∈ P h ( s,a ) C π h +1 ( τ h , a, s ′ )) can be computed state-recursively by,

and so C π M satisfies (SR). The infinity conditions and non-decreasing requirements are also easy to verify.

We proceed by induction on h .

(Base Case) For the base case, we consider h = H +1 . Observe that for any history τ and t , we have c H +1: t ( τ ) = 0 since it is an empty sum. Then, by definition of C π M , we see that C π H +1 ( τ H +1 ) = 0 = max t max τ 0 = max t max τ c H +1: t ( τ ) 3 .

(Inductive Step) For the inductive step, we consider h ≤ H . Let s = s h ( τ h ) and a = π h ( τ h ) . By separately considering the case where t = h and t ≥ h +1 in the max t ≥ h , we see that max t ≥ h max τ ∈H H +1 : π c h : t ( τ )

M

P

[

τ

|

τ

h

]

>

0

The second line used the fact that c h : h ( τ ) = c h ( s, a ) and the recursive definition of c h : t ( τ ) . The fourth line used the result proven for the almost sure case above. The sixth line used the induction hypothesis. The last line used the definition of C π M .

C Proofs for Section 3

C.1 Helpful Technical Lemmas

Here, we use a different, inductive definition for V then in the main text. However, the following lemma shows they are equivalent.

Definition 9 (Value Space) . For any s ∈ S , we define V H +1 ( s ) def = { 0 } , and for any h ∈ [ H ] ,

We define V def = ⋃ h,s V h ( s ) .

Lemma 3 (Value Intution) . For all s ∈ S and h ∈ [ H +1] ,

and |V h ( s ) | ≤ A ∑ H t = h S H -t . Thus, V can be computed in finite time using backward induction.

Lemma 4 (Cost) . For any h ∈ [ H +1] , τ h ∈ H h , and v ∈ V , if s = s h ( τ h ) , then,

Lemma 5 (Value) . Suppose that π ∈ Π D . For all h ∈ [ H +1] and ( s, v ) ∈ ¯ S , if ¯ C π h ( s, v ) < ∞ , then ¯ V π h ( s, v ) ≥ v .

Remark 6 (Technical Subtlety) . Technically, V π h ( τ h ) is only well defined if P π M [ τ h ] > 0 and all of our arguments technically should assume this is the case. However, it is standard in MDP theory to define the policy evaluation equations on non-reachable trajectories using the standard recursion to simplify proofs, as we have done here. Formally, this is equivalent to assuming the process starts initially at τ h instead of just conditioning on reaching τ h , or defining the values to correspond to policy evaluation equations directly. This is consistent with the usual definition when P π M [ τ h ] > 0 but gives it a defined value also when P π M [ τ h ] = 0 . In either case, this detail only means our recursive definition of V is a superset rather than exactly the set of all values as we defined in the main text. This does not effect the final results since unreachable trajectories do not effect π 's overall value in the MDP anyway, and only effects the interpretations of some intermediate variables.

C.2 Proof of Proposition 3

Proof. By definition of V ∗ M and C ∗ M ,

For the second claim, we observe that if V ∗ M > -∞ then by the above argument any optimal deterministic policy π for COVER satisfies C π M = C ∗ M ≤ B and V π M ≥ V ∗ M . Thus, COVER ⊆ PACK .

C.3 Proof of Lemma 3

Proof. We proceed by induction on h . Let s ∈ S be arbitrary.

Base Case. For the base case, we consider h = H + 1 . In this case, we know that for any π ∈ Π D and any τ ∈ H H +1 , V π H +1 ( τ H +1 ) = 0 ∈ { 0 } = V H +1 ( s ) by definition. Furthermore, |V H +1 ( s ) | = 1 = A 0 = A ∑ H t = H +1 S t .

Inductive Step. For the inductive step, we consider h ≤ H . In this case, we know that for any π ∈ Π D and any τ h ∈ H h , if s = s h ( τ h ) and a = π h ( τ h ) , then the policy evaluation equations imply,

We know by the induction hypothesis that V π h +1 ( τ h , a, s ′ ) ∈ V h +1 ( s ′ ) . Thus, by (13), V π h ( τ h ) ∈ V h ( s ) . Lastly, we see by (13) and the induction hypothesis that,

This completes the proof.

C.4 Proof of Lemma 4

Proof. We proceed by induction on h . Let τ h ∈ H h and v ∈ V be arbitrary and suppose that s = s h ( τ h ) . We let C ∗ h ( τ h , v ) denote the minimum for the RHS of (15).

Base Case. For the base case, we consider h = H + 1 . Observe that for any π ∈ Π D , V π H +1 ( τ H +1 ) = 0 by definition. Thus, there exists a π ∈ Π D satisfying V π H +1 ( τ H +1 ) ≥ v if and only if v ≤ 0 . We also know by definition that any such policy π satisfies C π H +1 ( τ H +1 ) = 0 and if no such policy exists C ∗ H +1 ( τ H +1 , v ) = ∞ by convention. Therefore, we see that C ∗ H +1 ( τ H +1 , v ) = χ { v ≤ 0 } . Then, by definition of the base case for ¯ C , it follows that,

Inductive Step. For the inductive step, we consider h ≤ H . If C ∗ h ( τ h , v ) = ∞ , then trivially ¯ C ∗ h ( s, v ) ≤ C ∗ h ( τ h , v ) . Instead, suppose that C ∗ h ( τ h , v ) < ∞ . Then, there must exist a feasible π ∈ Π D satisfying V π h ( τ h ) ≥ v . Let a ∗ = π h ( τ h ) . By the policy evaluation equations, we know that,

For each s ′ ∈ S , define v ∗ s ′ def = V π h +1 ( τ h , a ∗ , s ′ ) and observe that v ∗ s ′ ∈ V h +1 ( s ′ ) ⊆ V by Lemma 3. Thus, we see that ( a ∗ , v ∗ ) ∈ A × V S and r h ( s, a ) + ∑ s ′ P h ( s ′ | s, a ) v s ′ ≥ v , which implies ( a ∗ , v ∗ ) ∈ ¯ A h ( s, v ) .

Since π satisfies V π h +1 ( τ h , a ∗ , s ′ ) ≥ v ∗ s ′ , it is clear that C ∗ h +1 ( s ′ , v ∗ s ′ ) ≤ C π h +1 ( τ h , a ∗ , s ′ ) . Thus, the induction hypothesis implies that ¯ C ∗ h +1 ( s ′ , v ∗ s ′ ) ≤ C ∗ h +1 ( s ′ , v ∗ s ′ ) ≤ C π h +1 ( τ h , a ∗ , s ′ ) . The optimality equations for ¯ M then imply that,

The first inequality used the fact that ( a ∗ , v ∗ ) ∈ ¯ A h ( s, v ) . The second inequality relied on f being non-decreasing and the induction hypothesis. The final equality used (TR).

Since π was an arbitrary feasible policy for the optimization defining C ∗ h ( τ h , v ) , we see that ¯ C ∗ h ( s, v ) ≤ C ∗ h ( τ h , v ) . This completes the proof.

C.5 Proof of Lemma 5

Proof. We proceed by induction on h . Let ( s, v ) ∈ ¯ S be arbitrary.

Base Case. For the base case, we consider h = H +1 . By definition and assumption, ¯ C π H +1 ( s, v ) = χ { v ≤ 0 } < ∞ . Thus, it must be the case that v ≤ 0 and so by definition ¯ V π H +1 ( s, v ) = 0 ≥ v .

Inductive Step. For the inductive step, we consider h ≤ H . We decompose π h ( s, v ) = ( a, v ) where we know ( a, v ) ∈ ¯ A h ( s, v ) since π has finite cost 4 . Moreover, it must be the case that for any s ′ ∈ S with P h ( s ′ | s, a ) > 0 that ¯ C π h +1 ( s ′ , v s ′ ) < ∞ otherwise the property that f outputs ∞ when inputted an ∞ would imply a contradiction:

Thus, the induction hypothesis implies that ¯ V π h +1 ( s ′ , v s ′ ) ≥ v s ′ for any such s ′ ∈ S . By the policy evaluation equations, we see that,

¯ . This completes the proof.

The third line uses the definition of A h ( s, v )

C.6 Proof of Theorem 1

Proof. If ¯ C ∗ 1 ( s 0 , v ) > B for all v ∈ V , then C ∗ M > B since otherwise we would have ¯ C ∗ 1 ( s 0 , v ) ≤ C ∗ 1 ( s 0 , v ) = C ∗ M ≤ B by Lemma 4. Thus, if Algorithm 1 outputs 'infeasible' it is correct.

On the other hand, suppose that there exists some v ∈ V for which ¯ C ∗ 1 ( s 0 , v ) ≤ B . By standard MDP theory, we know that since π ∈ Π D is a solution to ¯ M , it must satisfy the optimality equations. In particular, ¯ C π 1 ( s 0 , v ) = ¯ C ∗ 1 ( s 0 , v ) ≤ B . Since C π M = ¯ C π 1 ( s 0 , v ) 5 , we see that there exists a π ∈ Π D for which C π M ≤ B and so V ∗ M > -∞ .

Since V ∗ M is the value of some deterministic policy, Lemma 3 implies that V ∗ M ∈ V . Thus, Lemma 5 implies that V π 1 ( s 0 , V ∗ M ) ≥ V ∗ M and C π 1 ( s 0 , V ∗ M ) ≤ C ∗ 1 ( s 0 , V ∗ M ) ≤ B . Consequently, running π with initial state ¯ s 0 = ( s 0 , V ∗ M ) is an optimal solution to (CON). In either case, Algorithm 1 is correct.

D Proofs for Section 4

Definition 10. We define the exact partial sum,

Observation 1. We observe that both σ and ˆ σ can be computed recursively. Specifically, σ s,a h, v ( S + 1 , u ) = u and σ s,a h, v ( t, u ) = σ s,a h, v ( t, u + P h ( t | s, a ) v t ) . Similarly, ˆ σ s,a h, v ( S + 1 , u ) = u and ˆ σ s,a h, v ( t, u ) = σ s,a h, v ( t, ⌊ u + P h ( t | s, a ) v t ⌋ G ) .

For completeness, and to assist with other arguments, we also prove the exact recursion we presented in Definition 3 is correct using Lemma 6.

Lemma 6. For any t ∈ [ S +1] and u ∈ R , we have that,

Moreover, ¯ C ∗ h ( s, v ) = min a ∈A c h ( s, a ) + g s,a h,v (1 , r h ( s, a )) .

D.1 Proof of Lemma 6

Proof. We proceed by induction on t .

Base Case. For the base case, we consider t = S + 1 . Since ∑ S s ′ = S +1 P h ( s ′ | s, a ) v s ′ = 0 is the empty sum, the condition u + ∑ S s ′ = S +1 P h ( s ′ | s, a ) v s ′ ≥ v is true iff u ≥ v . Also, for any v , g s,a h, v ( S +1) = 0 by definition. Thus, the minimum defining g s,a h, v ( S +1 , u ) is 0 when u ≥ v and is ∞ due to infeasibility otherwise. In symbols, g s,a h,v ( S +1 , u ) = χ { u ≥ v } as was to be shown.

Inductive Step. For the inductive step, we consider t ≤ S . We see that g s,a h,v ( t, u )

The second lined used (SR). The third line split the optimization into the first decision and the remaining decisions and decomposed the sum in the constraint. The fourth line used the fact that α is a non-decreasing function of both its arguments and the fact that the second optimization only concerns the second argument. The last line used the induction hypothesis.

The observation that min a ∈A c h ( s, a )+ g s,a h,v (1 , r h ( s, a )) = ¯ C ∗ h ( s, v ) then follows from the definition of ¯ A h ( s, v ) and (BU):

D.2 Proof of Lemma 1

Proof. We proceed by induction on t .

Base Case. For the base case, we consider t = S +1 . By definition, ˆ σ s,a h, ˆ v ( S +1 , u ) = u so the constraint is satisfied iff u ≥ v . Since for any ˆ v , ˆ g s,a h, ˆ v ( S + 1) = 0 by definition, the minimum defining ˆ g s,a h, ˆ v ( S + 1 , u ) is 0 when u ≥ v and is ∞ due to infeasibility otherwise. In symbols, ˆ g s,a h,v ( S +1 , u ) = χ { u ≥ v } as was to be shown.

Inductive Step. For the inductive step, we consider t ≤ S . We see that,

The second lined used (SR). The third line split the optimization into the first decision and the remaining decisions and used the recursive definition of ˆ σ in the constraint. The fourth line used the fact that α is a non-decreasing function of both its arguments and the fact that the second optimization only concerns the second argument. The last line used the induction hypothesis.

D.3 Proof of Proposition 4

Proof. The runtime guarantee is easily seen since Algorithm 4 consists of nested loops. The fact that it computes an optimal solution for ¯ M absent rounding or lower bounding follows immediately from Lemma 6.

E Proofs for Section 5

E.1 Helpful Technical Lemmas (Additive)

The following claims all assume Definition 7.

Observation 2. For any v ∈ R ,

Lemma 7. For any h ∈ [ H ] , s ∈ S , a ∈ A , v ∈ R S , u ∈ R , and t ∈ [ S +1] , we have,

Lemma 8 (Cost) . For any h ∈ [ H +1] and ( s, v ) ∈ ¯ S , ˆ C ∗ h ( s, ⌊ v ⌋ G ) ≤ ¯ C ∗ h ( s, v ) .

Lemma 9 (Approximation) . Suppose that π ∈ Π D . For all h ∈ [ H + 1] and ( s, ˆ v ) ∈ ˆ S , if ˆ C π h ( s, ˆ v ) < ∞ , then ˆ V π h ( s, ˆ v ) ≥ ˆ v -δ ( S +1)( H -h +1) .

E.2 Helpful Technical Lemmas (Relative)

The following claims all assume Definition 8.

Observation 3. For any v ∈ R ,

Lemma 10. For any h ∈ [ H ] , s ∈ S , a ∈ A , v ∈ R S ≥ 0 , u ∈ R ≥ 0 , and t ∈ [ S +1] , we have,

Lemma 11 (Cost) . Suppose all rewards are non-negative. For any h ∈ [ H + 1] and ( s, v ) ∈ ¯ S , ˆ C ∗ h ( s, ⌊ v ⌋ G ) ≤ ¯ C ∗ h ( s, v ) .

Lemma12 (Approximation) . Suppose all rewards are non-negative and π ∈ Π D . For all h ∈ [ H +1] and ( s, ˆ v ) ∈ ˆ S , if ˆ C π h ( s, ˆ v ) < ∞ , then ˆ V π h ( s, ˆ v ) ≥ ˆ v (1 -δ ) ( S +1)( H -h +1) .

E.3 Proof of Observation 2

Proof. Using properties of the floor function, we can infer that,

E.4 Proof of Lemma 7

Proof. We proceed by induction on t .

Base Case. For the base case, we consider t = S +1 . By definition, we have ˆ σ s,a h, v ( S +1 , u ) = u = σ s,a h, v ( S +1 , u ) .

Inductive Step. For the inductive step, we consider t ≤ S . We first see that,

The first inequality used the induction hypothesis and the second inequality used the fact that ⌊ x ⌋ G ≤ x .

We also see that,

The first inequality used the induction hypothesis and the second inequality used the fact that ⌊ x ⌋ G ≥ x -δ .

E.5 Proof of Lemma 8

Proof. We proceed by induction on h . Let ( s, v ) ∈ ¯ S be arbitrary.

Base Case. For the base case, we consider h = H +1 . Since ⌊ v ⌋ G ≤ v , we immediately see,

and,

Inductive Step. For the inductive step, we consider h ≤ H . If ¯ C ∗ h ( s, v ) = ∞ , then trivially ˆ C ∗ h ( s, ⌊ v ⌋ G ) ≤ ¯ C ∗ h ( s, v ) . Instead, suppose that ¯ C ∗ h ( s, v ) < ∞ . Let π be a solution to the optimality equations for ¯ M so that ¯ C π h ( s, v ) = ¯ C ∗ h ( s, v ) < ∞ . Since ¯ C ∗ h ( s, v ) < ∞ , we know that ( a ∗ , v ∗ ) = π h ( s, v ) ∈ ¯ A h ( s, v ) . By the definition of ¯ A h ( s, v ) , we know that,

For each s ′ ∈ S , define ˆ v ∗ s ′ def = ⌊ v ∗ s ′ ⌋ G and recall that v ∗ s ′ ∈ V . We first observe that,

Since v ∗ s ′ ∈ V , the induction hypothesis implies that ˆ C ∗ h +1 ( s ′ , ˆ v ∗ s ′ ) ≤ ¯ C ∗ h +1 ( s ′ , v ∗ s ′ ) = ¯ C π h +1 ( s ′ , v ∗ s ′ ) . The optimality equations for ˆ M then imply that,

The first inequality used the fact that ( a ∗ , v ∗ ) ∈ ˆ A h ( s, v ) . The second inequality relied on f being non-decreasing and the induction hypothesis. The penultimate equality used (TR). This completes the proof.

E.6 Proof of Lemma 9

Proof. We proceed by induction on h . Let ( s, ˆ v ) ∈ ˆ S be arbitrary.

Base Case. For the base case, we consider h = H +1 . By definition and assumption, ˆ C π H +1 ( s, ˆ v ) = χ { ˆ v ≤ 0 } < ∞ . Thus, it must be the case that ˆ v ≤ 0 and so by definition ˆ V π H +1 ( s, ˆ v ) = 0 ≥ ˆ v .

Inductive Step. For the inductive step, we consider h ≤ H . As in the proof of Lemma 5, we know that π h ( s, v ) = ( a, ˆ v ) ∈ ˆ A h ( s, ˆ v ) and for any s ′ ∈ S with P h ( s ′ | s, a ) > 0 that ˆ C π h +1 ( s ′ , v s ′ ) < ∞ . Thus, the induction hypothesis implies that ˆ V π h +1 ( s ′ , ˆ v s ′ ) ≥ ˆ v s ′ -δ ( S +1)( H -h ) for any such

s ′ ∈ S . By the policy evaluation equations, we see that,

The first inequality used the induction hypothesis. The second inequality used Lemma 7. The third inequality used the fact that by definition of ˆ A h ( s, ˆ v ) and κ , ˆ σ s,a h, ˆ v (1 , r h ( s, a )) ≥ κ (ˆ v ) = ˆ v -δ ( S +1) . This completes the proof.

E.7 Proof of Theorem 2

Proof.

Correctness. If ˆ C ∗ 1 ( s 0 , v ) > B for all ˆ v ∈ ˆ V , then C ∗ M > B since otherwise we would have ˆ C ∗ 1 ( s 0 , ⌊ v ⌋ G ) ≤ ¯ C ∗ 1 ( s 0 , v ) ≤ C ∗ M ≤ B by Lemma 8. Thus, if Algorithm 5 outputs 'infeasible' it is correct.

On the other hand, suppose that there exists some ˆ v ∈ ˆ V for which ˆ C ∗ 1 ( s 0 , ˆ v ) ≤ B . By standard MDP theory, we know that since π ∈ Π D is a solution to ˆ M , it must satisfy the optimality equations. In particular, ˆ C π 1 ( s 0 , ˆ v ) = ˆ C ∗ 1 ( s 0 , v ) ≤ B . As in the proof of Theorem 1, since C π M = ˆ C π 1 ( s 0 , ˆ v ) , we see that there exists a π ∈ Π D for which C π M ≤ B and so V ∗ M > -∞ .

Since V ∗ M is the value of some deterministic policy, Lemma 3 implies that V ∗ M ∈ V . Thus, Lemma 9 implies that ˆ V π 1 ( s 0 , ⌊ V ∗ M ⌋ G ) ≥ ⌊ V ∗ M ⌋ G -δ ( S +1) H ≥ V ∗ M -δ (1 + ( S +1) H ) = V ∗ M -ϵ and ˆ C π 1 ( s 0 , V ∗ M ) ≤ C ∗ 1 ( s 0 , V ∗ M ) ≤ B . Consequently, running π with initial state ¯ s 0 = ( s 0 , ⌊ V ∗ M ⌋ G ) is an optimal solution to (CON). In either case, Algorithm 5 is correct.

Complexity. For the complexity claim, we observe that the running time of Algorithm 5 is O ( HS 2 A | ˆ V| 2 | ˆ U| ) . To bound | ˆ V| , we observe that the number of integer multiples of δ required to capture the range [ -Hr max , Hr max ] is at most O ( Hr max δ ) = O ( H 2 Sr max /ϵ ) by definition of δ . Moreover, | ˆ U| = O ( | ˆ V| + S ) = O ( | ˆ V| ) for sufficiently large r max ϵ .

In particular, we see that the range of the rounded sums defining ˆ U is at widest [ -2 Hr max -δS, 2 Hr max ] since for any t +1 the rounded input is,

which is at most 2 Hr max , and,

which is at least -2 Hr max -δS . Overall, we see that O ( | ˆ V| 2 | ˆ U| ) = O ( | ˆ V| 3 ) = O ( H 6 S 3 r 3 max /ϵ 3 ) implying that the total run time is O ( H 7 S 5 Ar 3 max /ϵ 3 ) as claimed.

E.8 Proof of Observation 3

Proof. Using properties of the floor function, we can infer that,

and,

E.9 Proof of Lemma 10

Proof. We proceed by induction on t .

Base Case. For the base case, we consider t = S +1 . By definition, we have ˆ σ s,a h, v ( S +1 , u ) = u = σ s,a h, v ( S +1 , u ) .

Inductive Step. For the inductive step, we consider t ≤ S . We first see that,

The first inequality used the induction hypothesis and the second inequality used the fact that ⌊ x ⌋ G ≤ x .

We also see that,

The first inequality used the induction hypothesis and the second inequality used the fact that ⌊ x ⌋ G ≥ x -δ and the fact that all rewards and values are non-negative allowing us to add a (1 -δ ) -factor to the other value demands.

E.10 Proof of Lemma 11

Proof. We proceed by induction on h . Let ( s, v ) ∈ ¯ S be arbitrary.

Base Case. For the base case, we consider h = H +1 . Since ⌊ v ⌋ G ≤ v , we immediately see,

Inductive Step. For the inductive step, we consider h ≤ H . If ¯ C ∗ h ( s, v ) = ∞ , then trivially ˆ C ∗ h ( s, ⌊ v ⌋ G ) ≤ ¯ C ∗ h ( s, v ) . Instead, suppose that ¯ C ∗ h ( s, v ) < ∞ . Let π be a solution to the optimality

equations for ¯ M so that ¯ C π h ( s, v ) = ¯ C ∗ h ( s, v ) < ∞ . Since ¯ C ∗ h ( s, v ) < ∞ , we know that ( a ∗ , v ∗ ) = π h ( s, v ) ∈ ¯ A h ( s, v ) . By the definition of ¯ A h ( s, v ) , we know that,

For each s ′ ∈ S , define ˆ v ∗ s ′ def = ⌊ v ∗ s ′ ⌋ G and recall that v ∗ s ′ ∈ V . We first observe that,

The second inequality used the fact that all rewards are non-negative. Then by Lemma 10,

Thus, ( a ∗ , ˆ v ∗ ) ∈ ˆ A h ( s, ⌊ v ⌋ G ) .

Since v ∗ s ′ ∈ V , the induction hypothesis implies that ˆ C ∗ h +1 ( s ′ , ˆ v ∗ s ′ ) ≤ ¯ C ∗ h +1 ( s ′ , v ∗ s ′ ) = ¯ C π h +1 ( s ′ , v ∗ s ′ ) . The optimality equations for ˆ M then imply that,

The first inequality used the fact that ( a ∗ , v ∗ ) ∈ ˆ A h ( s, v ) . The second inequality relied on f being non-decreasing and the induction hypothesis. The penultimate equality used (TR).

This completes the proof.

E.11 Proof of Lemma 12

Proof. We proceed by induction on h . Let ( s, ˆ v ) ∈ ˆ S be arbitrary.

Base Case. For the base case, we consider h = H +1 . By definition and assumption, ˆ C π H +1 ( s, ˆ v ) = χ { ˆ v ≤ 0 } < ∞ . Thus, it must be the case that ˆ v ≤ 0 and so by definition ˆ V π H +1 ( s, ˆ v ) = 0 ≥ ˆ v .

Inductive Step. For the inductive step, we consider h ≤ H . As in the proof of Lemma 5, we know that π h ( s, v ) = ( a, ˆ v ) ∈ ˆ A h ( s, ˆ v ) and for any s ′ ∈ S with P h ( s ′ | s, a ) > 0 that ˆ C π h +1 ( s ′ , v s ′ ) < ∞ . Thus, the induction hypothesis implies that ˆ V π h +1 ( s ′ , ˆ v s ′ ) ≥ ˆ v s ′ (1 -δ ) ( S +1)( H -h ) for any such

s ′ ∈ S . By the policy evaluation equations, we see that,

The first inequality used the induction hypothesis. The second inequality used the fact that the rewards are non-negative. The third inequality used Lemma 10. The fourth inequality used the fact that by definition of ˆ A h ( s, ˆ v ) and κ , ˆ σ s,a h, ˆ v (1 , r h ( s, a )) ≥ κ (ˆ v ) = ˆ v (1 -δ ) S +1 .

E.12 Proof of Theorem 3

Proof.

Correctness. If ˆ C ∗ 1 ( s 0 , v ) > B for all ˆ v ∈ ˆ V , then C ∗ M > B since otherwise we would have ˆ C ∗ 1 ( s 0 , ⌊ v ⌋ G ) ≤ ¯ C ∗ 1 ( s 0 , v ) ≤ C ∗ M ≤ B by Lemma 11. Thus, if Algorithm 5 outputs 'infeasible' it is correct.

On the other hand, suppose that there exists some ˆ v ∈ ˆ V for which ˆ C ∗ 1 ( s 0 , ˆ v ) ≤ B . By standard MDP theory, we know that since π ∈ Π D is a solution to ˆ M , it must satisfy the optimality equations. In particular, ˆ C π 1 ( s 0 , ˆ v ) = ˆ C ∗ 1 ( s 0 , v ) ≤ B . As in the proof of Theorem 1, since C π M = ˆ C π 1 ( s 0 , ˆ v ) , we see that there exists a π ∈ Π D for which C π M ≤ B and so V ∗ M > -∞ .

Since V ∗ M is the value of some deterministic policy, Lemma 3 implies that V ∗ M ∈ V . Thus, Lemma 12 implies that ˆ V π 1 ( s 0 , ⌊ V ∗ M ⌋ G ) ≥ ⌊ V ∗ M ⌋ G (1 -δ ) ( S +1) H ≥ V ∗ M (1 -δ ) ( S +1) H +1 = V ∗ M (1 -ϵ ( S +1) H +1 ) ( S +1) H +1 ≥ V ∗ M (1 -ϵ ) and ˆ C π 1 ( s 0 , V ∗ M ) ≤ C ∗ 1 ( s 0 , V ∗ M ) ≤ B . Consequently, running π with initial state ¯ s 0 = ( s 0 , ⌊ V ∗ M ⌋ G ) is an optimal solution to (CON). In either case, Algorithm 5 is correct.

Complexity. For the complexity claim, we observe that the running time of Algorithm 5 is O ( HS 2 A | ˆ V| 2 | ˆ U| ) . To bound | ˆ V| , we observe that the number of vmin -scaled powers of 1 / (1 -δ ) required to capture the range [0 , Hr max ] is at most one plus the largest power needed, which is

by definition of δ and the fact that log( 1 1 -δ ) = -log(1 -δ ) ≥ -log( e -δ ) = δ . Moreover, | ˆ U| = O ( | ˆ V| ) .

We see that the range of the rounded sums is at widest [0 , 2 Hr max ] since for any t +1 rounding non-negative sums is at least 0 and,

which is at most 2 Hr max . Then, the same analysis from before shows that the number of scaled powers of 1 / (1 -δ ) needed to cover this interval is O ( | ˆ V| ) . Thus, we see that O ( | ˆ V| 2 | ˆ U| ) = O ( | ˆ V| 3 ) = O ( H 6 S 3 log( r max p min r min ) 3 /ϵ 3 ) implying that the total run time is O ( H 7 S 5 A log( r max p min r min ) 3 /ϵ 3 ) as claimed.

F Extensions

F.1 Stochastic Costs

Suppose each cost c h ( s, a ) is replaced with a cost distribution C h ( s, a ) . Here, we consider finitely supported cost distributions whose supports are at most m ∈ N . Then, instead of the agent occurring cost c h ( s, a ) upon taking action a in state s at time h , the agent occurs a random cost c h ∼ C h ( s, a ) . Generally, this necessitates histories be cost dependent, and so the policy evaluation equations become,

Cover MDP. This implicitly changes the definition of V since the histories considered in the definition must now include cost history. Since the cost distributions are finitely supported, V remains a finite set. The main difference for ¯ M is that the future value demands must depend on both the immediate cost and the next state. This slightly changes the action space:

Bellman Updates. In order to solve ¯ M using Algorithm 4, we must extend the definition of TSR to also be recursive in the immediate costs. The key difference of the TSRC condition is that g 's recursion is now two dimensional.

Definition 11 (TSRC) . We call a criterion C time-space-cost-recursive (TSRC) if C π M = C π 1 ( s 0 ) where C π H +1 ( · ) = 0 and for any h ∈ [ H ] and τ h ∈ H h letting s = s h ( τ h ) and a = π h ( τ h ) ,

In the above, c ′ ∈ C h ( s, a ) and s ′ ∈ P h ( s, a ) . We now require that f be computable in O ( mS ) time. We also require that the f term above is equal to g τ h ,a h (1 , 1) , where, g τ h ,a h ( m + 1 , 1) = 0 , g τ h ,a h ( k, S +1) = g τ h ,a h ( k +1 , 1) , and,

In the above, we assume c k is the k th supported cost of C h ( s, a ) . Again, both α, β can be computed in O (1) time, but now α ( β ( y, · ) , x ) = x whenever 0 ∈ y .

Our examples from before also carry over to the stochastic cost setting.

Proposition 5 (TSCR examples) . The following classical constraints can be modeled by a TSCR cost constraint.

(Expectation Constraints) We capture these constraints by defining α ( x, y ) def = x + y and β ( x, y, z ) def = xyz .

(Almost Sure Constraints) We capture these constraints by defining α ( x, y ) def = max( x, y ) and β ( x, y, z ) def = [ x > 0 ∧ y > 0] z .

(Anytime Constraints) We capture these constraints by defining α ( x, y ) def = max(0 , max( x, y )) and β ( x, y, z ) def = [ x > 0 ∧ y > 0] z .

We can then modify our approximate recursion from before.

Definition 12. We define, ˆ g s,a h,v ( m +1 , 1 , u ) def = χ { u ≥ v } , ˆ g s,a h,v ( k, S +1 , u ) def = ˆ g s,a h,v ( k +1 , 1 , u ) and for t ≤ S ,

Approximation. Lastly, our rounding now occur error over time, space, and cost. Thus, we simply need to slightly modify our rounding functions. The main change is we use δ def = ϵ H ( mS +1)+1 . We also further relax our lower bounds to κ ( v ) def = v -δ ( mS +1) and κ def = v (1 -δ ) mS +1 respectively. Our running times correspondingly will have m 3 terms now.

F.2 Infinite Discounting

Approximations. Since we focus on approximation algorithms, the infinite discounted case can be immediately handled by using the idea of effective horizon. We can treat the problem as a finite horizon problem where the finite horizon H defined so that ∑ ∞ h = H γ h -1 r max ≤ ϵ ′ . By choosing ϵ ′ and ϵ small enough, we can get traditional value approximations. The discounting also ensures the effective horizon H is polynomially sized implying efficient computation. We just need to assume that 0 -cost actions are always available so that the policy can guarantee feasibility after the effective horizon has passed.

Hardness. We also note that all of the standard hardness results concerning deterministic policy computation carries over to the infinite discounting case even if all quantities are stationary.

F.3 Faster Approximations

We can significantly improve the running time of our FPTAS. The main guarantee is given in Corollary 1. They key step is to modify Algorithm 3 to use the differences instead of the sums. It is easy to see that this is equivalent since,

Since rounding down the differences make them larger, it becomes harder to be below r h ( s, a ) . Consequently, we now interpret κ as an upper bound for r h ( s, a ) instead of a lower bound on v The approximate dynamic programming method based on differences can be seen in Definition 13.

Definition 13. Fix a rounding down function ⌊·⌋ G and upper bound function κ . For any h ∈ [ H ] , s ∈ S , v ∈ V , and u ∈ R , we define, ˆ g s,a h,v ( S +1 , u ) = χ { u ≤ κ ( r h ( s,a )) } and for t ≤ S ,

The recursion is nearly identical to the originally, and unsurprisingly, it retains the same theoretical guarantees but in the reverse order. The guarantees can be seen in Lemma 13, which is straightforward to prove following the approach in the proof of Lemma 1.

Lemma 13. For any t ∈ [ S +1] and u ∈ R , we have that,

The difference version so far does not help us get faster algorithms. The key is in how we use it. Since the base case of the recursion is r h ( s, a ) and not v , we can compute the approximate bellman update for all v 's simultaneously. This ends up saving us a factor of |V| that we had in the original Algorithm 4. The new algorithm is defined in Algorithm 6. The inputs to the recursion are define in Definition 14.

Algorithm 6 Approx Solve

Definition 14. For any h ∈ [ H ] , s ∈ S , and a ∈ A , we define ˆ U s,a h (1) def = V and for any t ∈ [ S ] ,

Proposition 6. The running time of Algorithm 6 is O ( HS 2 A |V| ˆ σ ) .

Corollary 1 (Running Time Improvements) . Using Algorithm 6, the running time of our additiveFPTAS becomes O ( H 5 S 4 Ar 2 max /ϵ 2 ) , and the running time of our relative-FPTAS becomes O ( H 5 S 4 A log( r max r min p min ) 2 /ϵ 2 )

Approximation Details. Although the running times our clear from removing the factor of | ˆ V | , we need to slightly alter our approximation schemes for this to work. First, we need to use κ ( r h ( s, a )) def = r h ( s, a ) + δ for the additive approximation. The proof from before goes through almost identically.

However, for the relative approximation, no choice of upper bound can ensure enough feasibility. Thus, we simply use κ ( r h ( s, a )) def = r h ( s, a ) and apply a different analysis. We also note that technically, differences can become negative. To deal with this the relative rounding function should simply send any negative number to 0 : ⌊-x ⌋ G def = 0 . The analysis is mostly the same, but the feasibility statement must be slightly modified.

Lemma 14. Suppose all rewards are non-negative. For any h ∈ [ H + 1] and ( s, v ) ∈ ¯ S , ˆ C ∗ h ( s, ⌊ v (1 -δ ) H -h +1 ⌋ G ) ≤ ¯ C ∗ h ( s, v ) .

The idea is that since no fixed upper bound can capture arbitrary input values, we simply input relative values. Then, the feasibility part of Lemma 11 goes through as before. The proofs mostly remain the same, but the rounding must again change. We must now start at the smaller v min that is the original v min scaled by a factor of (1 -δ ) H to ensure that ⌊ V ∗ M (1 -δ ) H ⌋ is in ˆ V . This makes ˆ V larger, but not by too much as we argued in previous analyses.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We believe the claims made are accurate.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We point out our assumed reward bounds and note they are unavoidable under worst-case analysis.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Our paper provides all proofs in the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: Our paper is purely theoretical and does not include any experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: Our paper is purely theoretical and does not include any experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: Our paper is purely theoretical and does not include any experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: Our paper is purely theoretical and does not include any experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: Our paper is purely theoretical and does not include any experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We beleive our paper follows the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss in the introduction and conclusions potential positive impacts of our work. We do not believe there are any direct negative impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper is purely theoretical and does not include any releasable materials.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: Our paper is purely theoretical and does not include any experiments.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper is purely theoretical and does not include any experiments.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper is purely theoretical and does not use any crowdsourcing nor human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper is purely theoretical and does not use any crowdsourcing nor human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an algorithm that efficiently computes near-optimal deterministic policies for constrained reinforcement learning (CRL) problems. The proposed algorithm comprises three ideas: 1) value-demand augmentation, 2) action-space approximate dynamic programming, and 3) time-space rounding. Under some assumptions, their algorithm constitutes a fully polynomial-time approximation scheme (FPTAS) for a few cost criteria. This class requires that the cost of a policy can be computed recursively over both time and (state) space, which includes classical expectation, almost sure, and anytime constraints.\n\nSTRENGTHS:\n**Novelty.** I think this paper brings new insights into the field of safe RL (based on constraints) or CMDPs. I think the authors provide unique views for addressing or solving constrained RL problems. \n\n**Cover MDP.** It is an interesting approach to formulate a CMDP that covers diverse formulations for constrained RL. There have similar attempts as I will mention in Weakness, the proposed one is novel as far as I know.\n\n**Algorithm.** I think the proposed algorithm is technically sound. It is a good property to solve diverse problems using a single algorithm in a reasonable computational time.\n\nWEAKNESSES:\n**Unclear difference from previous work.** I do not think this paper clearly conveys the differences or contributions compared to previous work. I think this paper is not written after fully surveying existing work. Within the several major claims the authors make, there are several existing works where similar attempts were made. For example, in safe RL literature, there have been several papers on state augmentation or generalized formulations:\n- Sootla, Aivar, et al. \"Sauté RL: Almost surely safe reinforcement learning using state augmentation.\" ICML, 2022.\n- Wachi, Akifumi, Xun Shen, and Yanan Sui. \"A Survey of Constraint Formulations in Safe Reinforcement Learning.\" IJCAI (2024).\n\nI know existing works have not made the same contributions as this paper, but it is significant to properly make the differences clear rather than exaggerate the contributions.\n\n**Theoretical results.** Though I think the theoretical results are good, I think it is better to compare the theoretical results with existing ones. I understand no existing algorithm can solve diverse safe RL problems, unlike the proposed ones. However, there are a bunch of theoretical results on CMDPs based on expectations (and recent papers derive the upper bounds for other problem formulations). It is necessary to compare the upper bound or strength of the assumptions between the SOTA algorithm and the proposed ones.\n\n**Experiments.** I highly respect theoretical ML papers, but I think this paper needs empirical evaluation because the main claim is efficient computation. The algorithm is complicated and it is questionable whether the proposed algorithm actually works well even in toy problems.\n\nIn summary, this paper presents novel and interesting insights, but this paper is not written based on existing research. Thus, the contribution is not clear and I feel the contributions of this paper are exaggerated. Also, the theoretical and empirical advantages are not clear, which should be clearly written or supported by additional work. Therefore, I do not think this paper is ready for publication and resubmit to another venue.\n\nQUESTIONS:\n[Q] Could you discuss the computational complexity of the proposed algorithm while referring to those in SOTA algorithms (for constrained RL problems based on expectation)?\n\n\n### Minor Comments\n- Line 117-118: (SR) $\\rightarrow$ (TSR)\n\nLIMITATIONS:\nThe authors have slightly addressed the limitations. There is no discussion regarding potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-30"}

comments: SUMMARY:
The paper proposes a novel algorithm for constraint MDPs whose constraints have TR properties. The algorithm converts the original problem to an unconstraint MDP. To address the challenge of high complexity of the proposed approach, the method discovers that for constraints with TSR properties, it can accelerate the Bellman update. Furthermore, the paper proposes the approximate version of the problem by rounding the values effectively.

STRENGTHS:
1. The analysis is comprehensive and the logic is strong.
2. The fast bellman update for TSR constraints and the approximate rounding are interesting but very useful in practical computation.
3. The overall writing is good, esp. for math notations and explanation.

WEAKNESSES:
1. The problem extends knapsack problems to constraint MDP. It is unclear if this problem is novel or is a common interest in the RL and control community.
2. It is hard to think of practical problems that fit the setting on TSR, could you provide examples that needs constraint MDP with TSR (TR) constraints while cannot be modeled as a single MDP problem with cleverly designed rewards?
3. The overall algorithm and the improvement are important but also seems like straight-forward extensions on the basic algorithms. The overall complexity is still very high at last.

QUESTIONS:
1. Could you provide examples that needs constraint MDP with TSR (TR) constraints while cannot be modeled as a single MDP problem with cleverly designed rewards?
2. Is there existing works  using the same problem definition?

LIMITATIONS:
1. The complexity is still high.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an algorithm that computes deterministic policy for constraint RL. The work is majorly theoretical. The proposed algorithm for the worst-case analysis achieves the best approximation guarantees. Their approach incorporates ideas from state augmentation with value function, approximate dynamic programming, and time-space rounding.\n\nSTRENGTHS:\n- Computes a near-optimal policy\n- Consider a diverse range of constraint functions\n- Algorithm belongs to FPTAS\n\nWEAKNESSES:\n- Motivation sounds weak or rather superficial. In the case of stochastic policies, why can one not apply the mean of the policy to get a deterministic behaviour?\n\n- All the Algorithms are difficult to parse/not self-contained. What does it mean if a definition takes an input of M, C?  Maybe I missed it somewhere, what is V^s? Would be good if you can define the symbols in the algorithm at least input.\n\n- Theoretical statement doesn't contain assumptions. It would be good if you could incorporate assumptions in the theoretical statements, especially in propositions and theorems.\n\n- Finite state-action space, perhaps rather difficult to then have meaningful applications among the ones that the authors mention in the first paragraph, e.g., autonomous driving. On a high level I am wondering, since you are also solving for a finite horizon case, why would one not apply an open loop MPC type approach for a constraint finite horizon application (that also results in deterministic policies and is polynomial time for finite H).\n\nQUESTIONS:\n- Why is Proposition 2 a proposition but not just an example?\n- remove line 144. It creates more confusion than helps to understand. E.g., use of not defined terminology of covering MDP now and value augmentation. \n- In line 148, what does \"this covering program\" mean? Does it correspond to the reversed objective?\n- why value-demand augmentation and not value-function augmentation?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provide a novel polynomial-time algorithm for finding near-optimal deterministic policies in constrained MDPs with a large family constraints beyond expectation. The algorithm first considers in the dual covering problem minimizing cost for constrained value. The new problem is then viewed as solving a new augmented MDP with \"value demands.\" This MDP is more prone to approximate algorithms via approximate dynamic programming and careful rounding. In the process, the class of constraints supported by the algorithm (TSR constraints) is a step towards the class of constraints that allow polynomial-time algorithms.\n\nSTRENGTHS:\n- The paper solves a significant problem in the area. Apart form the new tractability result, the TSR class for constraints includes most kinds of constraints of interest and only excludes the ones that are known to be impossible to solve. This is a step towards classification of constraints that admit FPTAS.\n- I am not an expert in the area, and I cannot comment on novelty of techniques used by the paper in confidence. However, I find them not only novel, but also potentially inspiring for RL theory.\n- The paper is well-written and easy to follow despite its technical complexity.\n\nWEAKNESSES:\nI think more discussions on the optimality/tightness of each step of algorithm design would benefit the paper. I suppose some of the operations/steps are specific to this problem and therefore investigation of the optimality the whole algorithm might require a lot effort possibly in another paper. However, I wonder if for some of the steps we can have insights on whether the operation is optimal or not. For example, if I am not wrong, switching to the covering problem was harmless, same for the reduction to the value augmented MDP. I am not sure how optimal handling the augmented MDP is. However, I guess due to the MDP formulation some existing results may apply. Perhaps the authors can provide some insights on each of the steps.\n\nMinor comments:\n-  Some small comments in the paper for quantities that will be discussed later would be welcome. Specifically, in the discussions on tractability of solving the cover MDP the lack of knowledge of the set $V$ is not brought up. Only later in the paper this issue is addressed.\n \n- The Objective of the cover MDP could be better clarified. The objective appears to be determined by the function $f$ as opposed to the conventional summation and expectations of value functions.\n\nQUESTIONS:\nPlease see the weaknesses. Also I wonder whether the proof techniques to focus on covering problem in cMDPs, and approximate dynamic programming are common in the literature. If so, more citations could be useful.\n\nLIMITATIONS:\nThe paper does a good job outlining the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-16"}