Title: Provably Safe Reinforcement Learning with Step-wise Violation Constraints
Venue: NeurIPS 2023
Accepted: yes
Keywords: safe reinforcement learning

step-wise violation

reinforcement learning theory
Abstract: We investigate a novel safe reinforcement learning problem with step-wise violation constraints. Our problem differs from existing works in that we focus on stricter step-wise violation constraints and do not assume the existence of safe actions, making our formulation more suitable for safety-critical applications that need to ensure safety in all decision steps but may not always possess safe actions, e.g., robot control and autonomous driving.
We propose an efficient algorithm SUCBVI, which guarantees $\widetilde{\mathcal{O}}(\sqrt{ST})$ or gap-dependent $\widetilde{\mathcal{O}}(S/\mathcal{C}_{\mathrm{gap}} + S^2AH^2)$ step-wise violation and $\widetilde{\mathcal{O}}(\sqrt{H^3SAT})$ regret. Lower bounds are provided to validate the optimality in both violation and  regret performance with respect to the number of states $S$ and the total number of steps $T$. 
Moreover, we further study an innovative safe reward-free exploration problem with step-wise violation constraints. For this problem, we design algorithm SRF-UCRL to find a near-optimal safe policy, which achieves nearly state-of-the-art  sample complexity $\widetilde{\mathcal{O}}((\frac{S^2AH^2}{\varepsilon}+\frac{H^4SA}{\varepsilon^2})(\log(\frac{1}{\delta})+S))$, and guarantees $\widetilde{\mathcal{O}}(\sqrt{ST})$ violation during exploration.  Experimental results demonstrate the  superiority of our algorithms in safety performance and corroborate our theoretical results.

Provably Safe Reinforcement Learning with Step-wise Violation Constraints

Abstract

Weinvestigate a novel safe reinforcement learning problem with step-wise violation constraints. Our problem differs from existing works in that we focus on stricter step-wise violation constraints and do not assume the existence of safe actions, making our formulation more suitable for safety-critical applications that need to ensure safety in all decision steps but may not always possess safe actions, e.g., robot control and autonomous driving. We propose an efficient algorithm SUCBVI, which guarantees an ˜ O ( √ ST ) or gap-dependent ˜ O ( S/ C gap + S 2 AH 2 ) step-wise violation and an ˜ O ( √ H 3 SAT ) regret. Lower bounds are provided to validate the optimality in both violation and regret performance with respect to the number of states S and the total number of steps T . Moreover, we further study an innovative safe reward-free exploration problem with step-wise violation constraints. For this problem, we design the algorithm SRF-UCRL to find a nearoptimal safe policy, which achieves a nearly state-of-the-art sample complexity ˜ O (( S 2 AH 2 ε + H 4 SA ε 2 )(log( 1 δ ) + S )) , and guarantees an ˜ O ( √ ST ) violation during exploration. Experimental results demonstrate the superiority of our algorithms in safety performance and corroborate our theoretical results.

1 Introduction

In recent years, reinforcement learning (RL) (Sutton & Barto, 2018) has become a powerful framework for decision-making and learning in unknown environments. Despite the ground-breaking success of RL in games (Lanctot et al., 2019), recommendation systems (Afsar et al., 2022) and complex tasks in simulation environments (Zhao et al., 2020), most existing RL algorithms focus on optimizing the cumulative reward and do not take into consideration the risk aspect, e.g., the agent runs into catastrophic situations during control. The lack of strong safety guarantees hinders the application of RL to broader safety-critical scenarios such as autonomous driving, robotics and healthcare. For example, for robotic control in complex environments, it is crucial to prevent the robot from getting into dangerous situations, e.g., hitting walls or falling into water pools, at all times.

To handle the safety requirement, a common approach is to formulate safety as a long-term expected violation constraint in each episode. This approach focuses on seeking a policy whose cumulative expected violation in each episode is below a certain threshold. However, for applications where an agent needs to avoid disastrous situations throughout the decision process, e.g., a robot needs to avoid hitting obstacles at each step, merely reducing the long-term expected violation is not sufficient to guarantee safety.

Motivated by this fact, we investigate safe reinforcement learning with a more fine-grained constraint, called step-wise violation constraint, which aggregates all nonnegative violations at each step (no offset between positive and negative violations permitted). We name this problem Safe-RL-SW. Our step-wise violation constraint differs from prior expected violation constraint (Wachi & Sui, 2020; Efroni et al., 2020b; Kalagarla et al., 2021) in two aspects: (i) Minimizing the step-wise violation enables the agent to learn an optimal policy that avoids unsafe regions deterministically, while reducing the expected violation only guarantees to find a policy with low expected violation, instead of a per-step zero-violation policy. (ii) Reducing the aggregated nonnegative violation allows us to have a risk control for each step, while a small cumulative expected violation can still result in a large cost at some individual step and cause danger, if other steps with smaller costs offset the huge cost.

Our problem faces two unique challenges. First, the step-wise violation requires us to guarantee a small violation at each step, which demands very different algorithm design and analysis from that for the expected violation (Wachi & Sui, 2020; Efroni et al., 2020b; Kalagarla et al., 2021). Second, in safety-critical scenarios, the agent needs to identify not only unsafe states but also potentially unsafe states, which are states that may appear to be safe but will ultimately lead to unsafe regions with a non-zero probability. For example, a self-driving car needs to learn to slow down or change directions early, foreseeing the potential danger in advance in order to ensure safe driving (Thomas et al., 2021). Existing safe RL works focus mainly on the expected violation (Wachi & Sui, 2020; Liu et al., 2021b; Wei et al., 2022), or requiring some other assumptions such as imposing the prior knowledge of a safe action for each state (Amani et al., 2021). Moreover, many previous works also require strong assumptions that exclude discrete tabular MDP (Amani et al., 2021; Wachi et al., 2021; Wang et al., 2023) which is considered in our paper. Hence, techniques in previous works cannot be applied to handle step-wise violations. More detailed comparisons are provided in Section 2.

To systematically handle these two challenges, we formulate safety as an unknown cost function for each state without assuming safe actions, and consider minimizing the step-wise violation instead of the expected violation. We propose a general algorithmic framework called S afe UCBVI (SUCBVI). Specifically, in each episode, we first estimate the transition kernel and cost function in an optimistic manner, tending to regard a state as safe at the beginning of learning. After that, we introduce novel dynamic programming to identify potentially unsafe states and determine safe actions, based on our estimated transition and costs. Finally, we employ the identified safe actions to conduct value iteration. This mechanism can adaptively update dangerous regions, and help the agent plan for the future, which keeps her away from all states that may lead to unsafe states. As our estimation becomes more accurate over time, the safety violation becomes smaller and eventually converges to zero. Note that without the assumption of safe actions, the agent knows nothing about the environment at the beginning. Thus, it is impossible for her to achieve an absolute zero violation. Nevertheless, we show that SUCBVI can achieve a sub-linear ˜ O ( √ ST ) cumulative violation or an ˜ O ( S/ C gap + S 2 AH 2 ) gap-dependent violation that is independent of T . This violation implies that as the RL game proceeds, the agent eventually learns how to avoid unsafe states. We also provide a matching lower bound to demonstrate the optimality of SUCBVI in both violation and regret.

Furthermore, we apply our step-wise safe RL framework to the reward-free exploration (Jin et al., 2020) setting. In this novel safe reward-free exploration, the agent needs to guarantee small step-wise violations during exploration, and also output a near-optimal safe policy. Our algorithm achieves ε cumulative step-wise violation during exploration, and also identifies a ε -optimal and safe policy. See the definition of ε -optimal and ε -safe policy in Section 6.1. Another interesting application of our framework is safe zero-sum Markov games, which we discuss in Appendix B.

The main contributions of our paper are as follows.

We formulate the safe RL with step-wise violation constraint problem (Safe-RL-SW), which models safety as a cost function for states and aims to minimize the cumulative step-wise violation. Our formulation is particularly useful for safety-critical applications where avoiding disastrous situations at each decision step is desirable, e.g., autonomous driving and robotics.

We provide a general algorithmic framework SUCBVI, which is equipped with an innovative dynamic programming to identify potentially unsafe states and distinguish safe actions. We establish an ˜ O ( √ H 3 SAT ) regret and an ˜ O ( √ ST ) or ˜ O ( S/ C gap + S 2 AH 2 ) gap-dependent violation guarantees, which exhibits the capability of SUCBVI in attaining high rewards while maintaining small violation.

We further establish an Ω( √ HST ) regret and an Ω( √ ST ) violation lower bounds for Safe-RL-SW. The lower bounds demonstrate the optimality of algorithm SUCBVI in both regret minimization and safety guarantee, with respect to factors S and T .

We consider step-wise safety constraints in the reward-free exploration setting, which is called the Safe-RFE-SW problem. In this setting, we design an efficient algorithm SRF-UCRL, which ensures ε step-wise violations during exploration and plans a ε -optimal and ε -safe policy for any reward functions with probability at least 1 -δ . We obtain an ˜ O (( S 2 AH 2 ε + H 4 SA ε 2 )(log( 1 δ ) + S )) sample complexity and an ˜ O ( √ ST ) violation guarantee for SRF-UCRL, which shows the efficiency of SRF-UCRL in sampling and danger avoidance even without reward signals. To the best of our knowledge, this work is the first to study the step-wise violation constraint in the RFE setting.

2 Related Work

Safe RL. Safety is an important topic in RL, which has been extensively studied. The constrained Markov decision process (CMDP)-based approaches handle safety via cost functions, and aim to minimize the expected episode-wise violation, e.g., (Yu et al., 2019; Wachi & Sui, 2020; Qiu et al., 2020; Efroni et al., 2020b; Turchetta et al., 2020; Ding et al., 2020; Singh et al., 2020; Kalagarla et al., 2021; Simão et al., 2021; Ding et al., 2021), or achieve zero episode-wise violation, e.g., (Liu et al., 2021b; Bura et al., 2021; Wei et al., 2022; Sootla et al., 2022). Apart from CMDP-based approaches, there are also other works that tackle safe RL by control-based approaches (Berkenkamp et al., 2017; Chow et al., 2018; Dalal et al., 2018; Wang et al., 2022), policy optimization (Uchibe & Doya, 2007; Achiam et al., 2017; Tessler et al., 2018; Liu et al., 2020; Stooke et al., 2020) and safety shields (Alshiekh et al., 2018).

In recent years, there are also some works studying step-wise violations with additional assumptions. Now we provide detailed comparisons between existing papers with instantaneous constraints. Turchetta et al. (2016); Wachi et al. (2018) propose a GP-based algorithm, which assumes that the transition is deterministic and known, while modeling the reward and cost functions using Gaussian Processes. By using this particular structure, they can infer the safety cost by estimating the parameters. Wachi et al. (2021) assumes the reward and cost functions have a generalized linear structure. Their algorithm explores in a safe space until a time t ∗ when the agent explores sufficiently. However, the upper bound of the exploring time t ∗ is not given in their paper. In fact, under the tabular MDP setting t ∗ can be infinite.

Amani et al. (2021) further considers the reward and cost functions to have a linear structure. It makes two assumptions: (a) There exists a safe action in each state, which prevents the agent from going to a potentially unsafe state. (b) The feature set is a star convex set, which helps them change actions continuously. However, this makes their works infeasible in tabular MDPs: The feature set in tabular MDPs consists of one-hot vectors and is not a star convex set. Hence, the work Amani et al. (2021) cannot solve our problem. Shi et al. (2023) considers safe RL in linear mixture MDPs. Their work also contains assumption (b), making it infeasible in tabular MDPs. Moreover, although they do not have assumption (a), it assumes the transition set ∆( s, a ) is known, which is not needed in our paper. Thus our paper is more challenging since we need to estimate ∆( s, a ) in our algorithm adaptively.

There are some other papers investigating the safety of RL problems. Alshiekh et al. (2018) represents the safe state by the reactive system, and uses shielding to calculate and restrict the agent within a safe trajectory completely. The main difference between their work and our work is that we need to dynamically update the estimated safe state, while they require to know the mechanism and state to calculate the shield. Dalal et al. (2018) considers restricting the safe action by projecting the action into the closest safe actions. They achieve this goal by solving a convex optimization problem on the continuous action set. However, in their paper, they do not consider the situation where a state can have no safe actions. To be more specific, they do not consider the situation when the convex optimization problem has no solutions. Le et al. (2019) considers the decision-making problem with a pre-collected dataset. Turchetta et al. (2020) and Sootla et al. (2022) both consider cumulative cost constraints rather than step-wise constraints. The former uses a teacher for intervention to keep the agent away from the unsafe region, while the latter encourages safe exploration by augmenting a safety state to measure safety during training.

Reward-free Exploration with Safety. Motivated by sparse reward signals in realistic applications, reward-free exploration (RFE) has been proposed in Jin et al. (2020), and further developed

in Kaufmann et al. (2021); Ménard et al. (2021). In RFE, the agent explores the environment without reward signals. After enough exploration, the reward function is given, and the agent needs to plan a near-optimal policy based on his knowledge collected in exploration. Safety is also important in RFE: We need to not only guarantee that our outputted policy is safe, but also ensure small violations during exploration.

Recently, Miryoosefi & Jin (2022); Huang et al. (2022) also study RFE with safety constraints. Compared to our paper, Miryoosefi & Jin (2022) only considers the safety constraint after the exploration phase. Huang et al. (2022) differs from our work in the following aspects: (i) They allow different reward functions during exploration and after exploration, and the agent can directly know the true costs during training. In our work, the cost function is the same during and after exploration, but the agent can only observe the noisy costs of a state when she arrives at that state. (ii) They require prior knowledge of safe baseline policies, while we do not need such an assumption. (iii) They consider the zero-expected violation during exploration, while we focus on keeping small step-wise violations.

3 The Safe MDP Model

Episodic MDP. In this paper, we consider the finite-horizon episodic Markov Decision Process (MDP), represented by a tuple ( S , A , H, P , r ) . Here S is the state space, A is the action space, and H is the length of each episode. P = { P h : S ×A ↦→ △ S } h ∈ [ H ] is the transition kernel, and P h ( s ′ | s, a ) gives the transition probability from ( s, a ) to s ′ at step h . r = { r h : S × A ↦→ [0 , 1] } h ∈ [ H ] is the reward function, and r h ( s, a ) gives the reward of taking action a in state s at step h . A policy π = { π h : S ↦→ A} h ∈ [ H ] consists of H mappings from the state space to action space. In each episode k , the agent first chooses a policy π k . At each step h ∈ [ H ] , the agent observes a state s k h , takes an action a k h , and then goes to a next state s k h +1 with probability P h ( s k h +1 | s k h , a k h ) . The algorithm executes T = HK steps. Moreover, the state value function V π h ( s, a ) and state-action value function Q π h ( s, a ) for a policy π can be defined as

Safety Constraint. To model unsafe regions in the environment, similar to Wachi & Sui (2020); Yu et al. (2022), we define a safety cost function c : S ↦→ [0 , 1] . Let τ ∈ [0 , 1] denote the safety threshold. A state is called safe if c ( s ) ≤ τ , and called unsafe if c ( s ) > τ . Similar to Efroni et al. (2020b); Amani et al. (2021), when the agent arrives in a state s , she will receive a cost signal z ( s ) = c ( s ) + ζ , where ζ is an independent, zero-mean and 1-sub-Gaussian noise. Denote ( x ) + = max { x, 0 } . The violation in state s is defined as ( c ( s ) -τ ) + , and the cumulative step-wise violation till episode K is

Eq. (1) represents the accumulated step-wise violation during training. When the agent arrives in state s k h at step h in episode k , she will suffer violation ( c ( s k h ) -τ ) + . This violation setting is significantly different from the previous CMDP setting (Qiu et al., 2020; Ding et al., 2020; Efroni et al., 2020b; Ding et al., 2021; Wachi & Sui, 2020; Liu et al., 2021a; Kalagarla et al., 2021). They study the episode-wise expected violation C ′ ( K ) = ∑ K k =1 ( E [ ∑ H h =1 c ( s k h , a k h ) ] -µ ) . There are also some papers (Efroni et al., 2020a; Simão et al., 2021) considering a stricter constraint named episodewise clipped expected violation : C ′′ ( K ) = ∑ K k =1 ( E [ ∑ H h =1 c ( s k h , a k h ) ] -µ ) + . Compared to the episode-wise violation (including expected violation and clipped expected violation), our step-wise violation has two main differences: (i) First, the episode-wise violation constraints allow the agent to get into unsafe states occasionally. Instead, the step-wise violation constraint forces the agent to stay in safe regions at all times. (ii) Second, in the episode-wise constraints, the average violation

E [ c ( s h , a h )] -µ/H at step h is allowed to be positive or negative, and they can cancel out in one episode to achieve E [ ∑ H h =1 c ( s k h , a k h )] ≤ µ. Instead, we consider a nonnegative function ( c ( s ) -τ ) + at each step in our step-wise violation, which imposes a stricter constraint.

Define U := { s ∈ S | c ( s ) > τ } as the set of all unsafe states . Let ι = { s 1 , a 1 , · · · , s H , a H } denote a trajectory. Since a feasible policy needs to satisfy the constraint at each step, we define the set of feasible policies as Π = { π | Pr { ∃ h ∈ [ H ] , s h ∈ U | ι ∼ π } = 0 } . The feasible policy set Π consists of all policies under which one never reaches any unsafe state in an episode.

Learning Objective. In this paper, we consider the regret minimization objective. Specifically, define π ∗ = argmax π ∈ Π V π 1 , V ∗ = V π ∗ and Q ∗ = Q π ∗ . The regret till K episodes is then defined as

where π k is the policy taken in episode k . Our objective is to minimize R ( K ) to achieve a good performance, and minimize the violation C ( K ) to guarantee the safety at the same time.

4 Safe RL with Step-wise Violation Constraints

4.1 Assumptions and Problem Features

Before introducing our algorithms, we first state the important assumptions and problem features for Safe-RL-SW.

Suppose π is a feasible policy. Then, if we arrive at s H -1 at step H -1 , π needs to select an action that guarantees s H / ∈ U . Define the transition set ∆ h ( s, a ) = { s ′ | P h ( s ′ | s, a ) > 0 } for any ∀ ( s, a, h ) ∈S×A× [ H ] , which represents the set of possible next states after taking action a in state s at step h . Then, at the former step H -1 , the state s is potentially unsafe if it satisfies that ∆ H -1 ( s, a ) ∩ U ̸ = ∅ for all a ∈ A . (i.e., no matter taking what action, there is a positive probability of transitioning to an unsafe next state). Therefore, we can recursively define the set of potentially unsafe states at step h as

̸

where U H = U . Intuitively, if we are in a state s h ∈ U h at step h , no action can be taken to completely avoid reaching potentially unsafe states s h +1 ∈ U h +1 at step h +1 . Thus, in order to completely prevent from getting into unsafe states U throughout all steps, one needs to avoid potentially unsafe states in U h at step h . From the above argument, we have that s 1 / ∈ U 1 is equivalent to the existence of feasible policies. The detailed proof is provided in Appendix D. Thus, we make the following necessary assumption.

Assumption 4.1 (Existence of feasible policies) . The initial state s 1 satisfies s 1 / ∈ U 1 .

For any s ∈ S and h ∈ [ H -1] , we define the set of safe actions for state s at step h as

and let A safe H ( s ) = A . A safe h ( s ) stands for the set of all actions at step h which will not lead to potentially unsafe states in U h +1 . Here, {U h } h ∈ [ H ] and { A safe h ( s ) } h ∈ [ H ] are defined by dynamic programming: If we know sets of all possible next state { ∆ h ( s, a ) } h ∈ [ H ] and unsafe state set U = U H , we can calculate all potentially unsafe state sets {U h } h ∈ [ H ] and safe action sets { A safe h ( s ) } h ∈ [ H ] , and choose feasible policies to completely avoid unsafe states.

4.2 Algorithm SUCBVI

Now we present our main algorithm S afe UCBVI (SUCBVI), which is based on previous classic RL algorithm UCBVI (Azar et al., 2017), and equipped with a novel dynamic programming to identify potentially unsafe states and safe actions. The pseudo-code is shown in Algorithm 1. First, we provide some intuitions about how SUCBVI works. At each episode, we first estimate all the unsafe states based on the historical data. Then, we perform a dynamic programming procedure introduced

Algorithm 1 SUCBVI

̸

in Section 4.1 and calculate the safe action set A safe h ( s ) for each state s . Then, we perform value iteration in the estimated safe action set. As the estimation becomes more accurate, SUCBVI will eventually avoid potentially unsafe states and achieve both sublinear regrets and violations.

Now we begin to introduce our algorithm. In the beginning, we initialize ∆ h ( s, a ) = ∅ for all ( h, s, a ) ∈ [ H ] ×S × A . It implies that the agent considers all actions to be safe at first, because no action will lead to unsafe states from the agent's perspective. In each episode, we first estimate the empirical cost ˆ c ( s ) based on historical cost feedback z ( s ) , and regard state s as safe if ¯ c ( s ) = ˆ c ( s ) -β > τ for some bonus term β , which aims to guarantee ¯ c ( s ) ≤ c ( s ) (Line 4). Then, we calculate the estimated unsafe state set U k H , which is a subset of the true unsafe state set U = U H with a high probability by optimism. With U k H and ∆ k h ( s, a ) , we can estimate potentially unsafe state sets U k h for all h ∈ [ H ] by Eq. (2) recursively.

Then, we perform value iteration to compute the optimistically estimated optimal policy. Specifically, for any hypothesized safe state s / ∈ U k h , we update the greedy policy on the estimated safe actions, i.e., π k h ( s ) = max a ∈ A k,safe h ( s ) Q k h ( s, a ) (Line 13). On the other hand, for any hypothesized unsafe state s h ∈ U h , since there is no action that can completely avoid unsafe states, we ignore safety costs and simply update the policy by π k h ( s ) = max a ∈A Q ( s, a ) (Line 15). After that, we calculate the estimated optimal policy π k for episode k , and the agent follows π k and collects a trajectory. Then, we update ∆ k +1 h ( s, a ) by incorporating the observed state s k h +1 into the set ∆ k h ( s k h , a k h ) . Under this updating rule, it holds that ∆ k h ( s, a ) ⊆ ∆ h ( s, a ) for all ( s, a ) ∈ S × A .

The performance of Algorithm 1 is summarized below in Theorem 4.2.

Moreover, if C gap ≜ min s ∈U ( c ( s ) -τ ) + > 0 , we have C ( K ) = ˜ O ( S/ C gap + S 2 AH 2 ) .

Theorem 4.2 shows that SUCBVI achieves both sublinear regret and violation. Moreover, when all the unsafe states have a large cost compared to the safety threshold τ , i.e., C gap = min s ∈U ( c ( s ) -τ ) + > 0 is a constant, we can distinguish the unsafe states easily and get a constant violation. In particular, when τ = 1 , Safe-RL-SW degenerates to the unconstrained MDP and Theorem 4.2 maintains the same regret ˜ O ( √ H 2 SAT ) as UCBVI-CH (Azar et al., 2017), while CMDP algorithms (Efroni et al., 2020b; Liu et al., 2021a) suffer a larger ˜ O ( √ H 3 S 3 AT ) regret.

We provide the analysis idea of Theorem 4.2 here. First, by the updating rule of ∆ k h ( s, a ) , we show that U k h and A k,safe h ( s ) have the following crucial properties: U k h ⊆ U h , A safe h ( s ) ⊆ A k,safe h ( s ) . Based on this property, we can prove that Q ∗ h ( s, a ) ≤ Q k h ( s, a ) and V ∗ h ( s ) ≤ V k h ( s ) for all ( s, a ) , and then apply the regret decomposition techniques to derive the regret bound.

Recall that π k is a feasible policy with respect to the estimated unsafe state set U k H and transition set { ∆ k h ( s, a ) } h ∈ [ H ] . If the agent takes policy π k and the transition follows ∆ k h ( s, a ) , i.e., s k h +1 ∈ ∆ k h ( s, a ) , the agent never arrive any estimated unsafe state s ∈ U k H in this episode. Hence the agent will suffer at most an ˜ O ( √ ST ) step-wise violation or an ˜ O ( S/ C gap + S 2 AH 2 ) gap-dependent bounded violation. Yet, the situation s h +1 ∈ ∆ k h ( s, a ) does not always hold for all h ∈ [ H ] . In the case when s k h +1 / ∈ ∆ k h ( s, a ) , we add the newly observed state s k h +1 into ∆ k +1 h ( s k h , a k h ) (Line 21). We can show that this case appears at most S 2 AH times, and thus incurs O ( S 2 AH 2 ) additional violation. Combining the above two cases, the total violation can be upper bounded.

5 Lower Bounds for Safe-RL-SW

In this section, we provide a matching lower bound for Safe-RL-SW in Section 4. The lower bound shows that if an algorithm always achieves a sublinear regret in Safe-RL-SW, it must incur an Ω( √ ST ) violation. This result matches our upper bound in Theorem 4.2, showing that SUCBVI achieves the optimal violation performance.

Theorem 5.1. If an algorithm has an expected regret E π [ R ( K )] ≤ HK 24 for all MDP instances, there exists an MDP instance in which the algorithm suffers expected violation E π [ C ( K )] = Ω( √ ST ) .

Now we validate the optimality in terms of regret. Note that if we do not consider safety constraints, the lower bound for classic RL (Osband & Van Roy, 2016) can be applied to our setting. Thus, we also have an Ω( √ T ) regret lower bound. To understand the essential hardness brought by safety constraints, we further investigate whether safety constraints will lead to an Ω( √ T ) regret, given that we can achieve an o ( √ T ) regret on some good instances without the safety constraints.

Theorem 5.2. For any α ∈ (0 , 1) , there exists a parameter n and n MDPs M 1 , . . . , M n satisfying that:

1. If we do not consider any constraint, there is an algorithm that achieves an ˜ O ( T (1 -α ) / 2 ) regret compared to the unconstrained optimal policy on all MDPs.

If we consider the safety constraint, any algorithm with a O ( T 1 -α ) expected violation will achieve an Ω( √ HST ) regret compared to the constrained optimal policy on one of MDPs.

Intuitively, Theorem 5.2 shows that if one achieves sublinear violation, she must suffer at least an Ω( √ T ) regret even if she can achieve an o ( √ T ) regret without considering constraints. This theorem demonstrates the hardness particularly brought by the step-wise constraint, and corroborates the optimality of our results. Combining with Theorem 5.1, the two lower bounds show an essential trade-off between the violation and performance.

6 Safe Reward-Free Exploration with Step-wise Violation Constraints

6.1 Formulation of Safe-RFE-SW

In this section, we consider Safe RL in the reward-free exploration (RFE) setting (Jin et al., 2020; Kaufmann et al., 2021; Ménard et al., 2021) called Safe-RFE-SW, to show the generality of our proposed framework. In the RFE setting, the agent does not have access to reward signals and only receives random safety cost feedback z ( s ) = c ( s ) + ζ . To impose safety requirements, Safe-RFE-SW requests the agent to keep small safety violations during exploration, and outputs a near-optimal safe policy after receiving the reward function.

Definition 6.1 ( ( ε, δ ) -optimal safe algorithm for Safe-RFE-SW) . An algorithm is ( ε, δ ) -optimal safe for Safe-RFE-SW if it outputs the triple ( ˆ P , ˆ ∆( s, a ) , ˆ U H ) such that for any reward function r , with probability at least 1 -δ ,

where ˆ π ∗ is the optimal feasible policy with respect to ( ˆ P , ˆ ∆( s, a ) , ˆ U H , r ) , Π is the set of feasible policies with respect to (∆( s, a ) , U H ) . and V ( s ; r ) is the value function under reward function r . We say that a policy is ε -optimal if it satisfies the left inequality in Eq. (4) and ε -safe if it satisfies the right inequality in Eq. (4). We measure the performance by the number of episodes used before the algorithm terminates, i.e., sample complexity . Moreover, the cumulative step-wise violation till episode K is defined as C ( K ) = ∑ K k =1 ∑ H h =1 ( c ( s k h ) -τ ) + . In Safe-RFE-SW, our goal is to design an ( ε, δ ) -optimal safe algorithm, and minimize both sample complexity and violation.

6.2 Algorithm SRF-UCRL

The Safe-RFE-SW problem requires us to consider extra safety constraints for both the exploration phase and final output policy, which needs new algorithm design and techniques compared to previous RFE algorithms. Also, the techniques for Safe-RL-SW in Section 4.2 are not sufficient for guaranteeing the safety of output policy, because SUCBVI only guarantees a step-wise violation during exploration.

We design an efficient algorithm S afe RF-UCRL (SRF-UCRL), which builds upon previous RFE algorithm RF-UCRL (Kaufmann et al., 2021). SRF-UCRL distinguishes potentially unsafe states and safe actions by backward iteration, and establishes a new uncertainty function to guarantee the safety of output policy. Algorithm 2 illustrates the procedure of SRF-UCRL. Specifically, in each episode k , we first execute a policy π k h computed from previous episodes, and then update the estimated next state set { ∆ k h ( s, a ) } h ∈ [ H ] and unsafe state set U k H by optimistic estimation. Then, we use Eq. (2) to calculate the unsafe state set U h for all steps h ∈ [ H ] . After that, we update the uncertainty function W k defined in Eq. (5) below and compute the policy π k +1 that maximizes the uncertainty to encourage more exploration in the next episode.

Now we provide the definition of the uncertainty function , which measures the estimation error between the empirical MDP and true MDP. For any safe state-action pair s / ∈ U h , a ∈ A k,safe h ( s ) , we define

√

where M ( N k h ( s, a ) , δ ) = 2 H 2 γ ( N k h ( s,a ) ,δ ) N k h ( s,a ) + SHγ ( N k h ( s,a ) ,δ ) N k h ( s,a ) , and γ ( n, δ ) is a logarithmic term that is formally defined in Theorem 6.2. For other state-action pairs ( s, a ) , we relax the restriction of b ∈ A k,safe h +1 ( s ′ ) to b ∈ A in the max function. Our algorithm stops when W K 1 ( s 1 , π K ( s 1 )) shrinks to within ε/ 2 . Compared to previous RFE works (Kaufmann et al. (2021); Ménard et al. (2021)), our uncertainty function has two distinctions. First, for any safe state-action pair ( s, a ) ∈ ( S \ U h , A k,safe h ( s )) , Eq. (5) considers only safe actions A k,safe h +1 ( s ) , which guarantees that the agent focuses on safe policies. Second, Eq. (5) incorporates another term ( SHγ ( N k h ( s, a ) , δ ) /N k h ( s, a )) to control the expected violation for feasible policies. Now we present our result for Safe-RFE-SW.

Algorithm 2 SRF-UCRL

̸

Theorem 6.2. Let γ ( n, δ ) = 2(log(2 SAH/δ ) + ( S -1) log( e (1 + n/ ( S -1))) , Algorithm 2 is a ( ε, δ ) -PAC algorithm with sample complexity at most 2

The step-wise violation of Algorithm 2 during exploration is C ( K ) = ˜ O ( S 2 AH 2 + √ ST ) .

Compared to previous work (Kaufmann et al., 2021) with an ˜ O (( H 4 SA/ε 2 )(log(1 /δ ) + S )) sample complexity, our result has an additional term ˜ O (( S 2 AH 2 /ε )(log(1 /δ ) + S )) . This term is due to the additional safety requirement for the final output policy, which was not considered in previous RFE algorithms (Kaufmann et al., 2021; Ménard et al., 2021). When ε and δ are sufficiently small, the leading term is ˜ O (( H 4 SA/ε 2 ) log(1 /δ )) , which implies that our algorithm satisfies the safety constraint without suffering additional regret. 3

6.3 Analysis for Algorithm SRF-UCRL

For the analysis of step-wise violation (Eq. (1)), similar to algorithm SUCBVI, algorithm SRF-UCRL estimates the next state set ∆ h ( s, a ) and potentially unsafe state set U h , which guarantees a ˜ O ( √ ST ) step-wise violation. Now we give a proof sketch for the ε -safe property of output policy (Eq. (4)).

First, if π is a feasible policy for ( ˆ P k , ∆ k ( s, a ) , {U k h } h ∈ [ H ] ) and s h +1 ∈ ∆ K h ( s h , a h ) for all h ∈ [ H ] , the agent who follows policy π will only visit the estimated safe states. Since each estimated safe state

Average Reward in Safe-RL-SW

Step-wise Violation in Safe-RL-SW

AverageRewardinSafe-RFE-SW

Expected Violation inSafe-RFE-SW

Average Reward

2.0

1.5

iolation

3000

2500

SUCBVI (ours)

UCBVI (Azar et al.)

Reward

1.75

1.50

Expected Violation

0.06

0.05

1.25

0.04

1.0

0.5

Unconstrained optimal reward

Safe optimal reward

Constrained optimal reward

Step-wise

2000

1500

Average

0.75

1.00

SRF-UCRL(ours)

0.03

200

SRF-UCRL(ours)

RF-UCRL(Kaufmann et al.)

0.5

1.0

0.0

UCBVI (Azar et al.)

SUCBVI (ours)

1000

500

0

0.50

0.25

0.00

Unconstrained optimal reward

Safe optimal reward

RF-UCRL(Kaufmann et al.)

0.01

0.01

0.00

0

250

The Number of Episodes

500

750 1000 1250 1500 1750 2000

0

The Number of Episodes

500

1000

1500

2000

0

The Number of Episodes

100

200

300

400

500

0

The Number of Episodes

100

200

300

400

500

only suffers a O (1 / √ N t ( s h , π h ( s h ))) violation, the violation led by this situation is bounded by W k 1 ( s, π 1 ( s 1 )) . Next, we bound the probability that s h +1 / ∈ ∆ K h ( s h , a h ) for some step h ∈ [ H ] . For any state-action pair ( s, a ) , if there is a probability P ( s ′ | s, a ) ≥ ˜ O (log( S/δ ) /N K h ( s, a )) that the agent transitions to next state s ′ from ( s, a ) at step h , the state s ′ is put into ∆ K h ( s, a ) with probability at least 1 -δ/S . Then, we can expect that all such states are put into our estimated next state set ∆ K h ( s, a ) . Thus, the probability that s h +1 / ∈ ∆ K h ( s h , a h ) is no larger than ˜ O ( S log( S/δ ) /N K h ( s, a )) by a union bound over all possible next states s ′ . Based on this argument, W k h ( s h , a h ) is an upper bound for the total probability that s h +1 / ∈ ∆ K h ( s h , a h ) for some step h . This will lead to additional W K 1 ( s 1 , π 1 ( s 1 )) expected violation. Hence the expected violation of output policy is upper bounded by 2 W K 1 ( s 1 , π K 1 ( s 1 )) ≤ ε . The complete proof is provided in Appendix A.

7 Experiments

In this section, we provide experiments for Safe-RL-SW and Safe-RFE-SW to validate our theoretical results. For Safe-RL-SW, we compare our algorithm SUCBVI with a classical RL algorithm UCBVI (Azar et al., 2017) and three state-of-the-art CMDP algorithms OptCMDP-bonus (Efroni et al., 2020b) , Optpess (Liu et al., 2021a) and Triple-Q (Wei et al., 2022). For Safe-RFE-SW, we report the average reward in each episode and cumulative step-wise violation. For Safe-RFE-SW, we compare our algorithm SRF-UCRL with a state-of-the-art RFE algorithm RF-UCRL (Kaufmann et al., 2021). We do not plot the regret because unconstrained MDP or CMDP algorithms do not guarantee step-wise violation. Applying them to the step-wise constrained setting can lead to negative or large regret and large violations, making the results meaningless. Detailed experiment setup is in Appendix E.

As shown in Figure 1, the rewards of SUCBVI and SRF-UCRL converge to the optimal rewards under safety constraints (denoted by "safe optimal reward"). In contrast, the rewards of UCBVI and UCRL converge to the optimal rewards without safety constraints (denoted by "unconstrained optimal reward"), and those of CMDP algorithms converge to a policy with low expected violation (denoted by "constrained optimal reward"). For Safe-RFE-SW, the expected violation of output policy of SRF-UCRL converges to zero while that of RF-UCRL does not. This corroborates the ability of SRF-UCRL in finding policies that simultaneously achieve safety and high rewards.

8 Conclusion

In this paper, we investigate a novel safe reinforcement learning problem with step-wise safety constraints. We first provide an algorithmic framework SUCBVI to achieve both an ˜ O ( √ H 3 SAT ) regret and an ˜ O ( √ ST ) step-wise or an ˜ O ( S/ C gap + S 2 AH 2 ) gap-dependent bounded violation that is independent of T . Then, we provide two lower bounds to validate the optimality of SUCBVI in both violation and regret in terms of S and T . Further, we extend our framework to the safe RFE with a step-wise violation and provide an algorithm SRF-UCRL that identifies a near-optimal safe policy given any reward function r and guarantees an ˜ O ( √ ST ) violation during exploration.

References

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning , pp. 22-31. PMLR, 2017.

MMehdi Afsar, Trafford Crump, and Behrouz Far. Reinforcement learning based recommender systems: A survey. ACM Computing Surveys , 55(7):1-38, 2022.

Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Sanae Amani, Christos Thrampoulidis, and Lin Yang. Safe reinforcement learning with linear function approximation. In International Conference on Machine Learning , pp. 243-253. PMLR, 2021.

Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning , pp. 263-272. PMLR, 2017.

Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. Advances in neural information processing systems , 30, 2017.

Archana Bura, Aria HasanzadeZonuzy, Dileep Kalathil, Srinivas Shakkottai, and Jean-Francois Chamberland. Safe exploration for constrained reinforcement learning with provable guarantees. arXiv preprint arXiv:2112.00885 , 2021.

Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. A lyapunovbased approach to safe reinforcement learning. Advances in neural information processing systems , 31, 2018.

Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.08757 , 2018.

Dongsheng Ding, Kaiqing Zhang, Tamer Basar, and Mihailo Jovanovic. Natural policy gradient primal-dual method for constrained markov decision processes. Advances in Neural Information Processing Systems , 33:8378-8390, 2020.

Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, Zhaoran Wang, and Mihailo Jovanovic. Provably efficient safe exploration via primal-dual policy optimization. In International Conference on Artificial Intelligence and Statistics , pp. 3304-3312. PMLR, 2021.

Yonathan Efroni, Shie Mannor, and Matteo Pirotta. Exploration-exploitation in constrained mdps. arXiv preprint arXiv:2003.02189 , 2020a.

Yonathan Efroni, Shie Mannor, and Matteo Pirotta. Exploration-exploitation in constrained mdps. arXiv preprint arXiv:2003.02189 , 2020b.

Ruiquan Huang, Jing Yang, and Yingbin Liang. Safe exploration incurs nearly no additional sample complexity for reward-free rl. arXiv preprint arXiv:2206.14057 , 2022.

Chi Jin, Akshay Krishnamurthy, Max Simchowitz, and Tiancheng Yu. Reward-free exploration for reinforcement learning. In International Conference on Machine Learning , pp. 4870-4879. PMLR, 2020.

Krishna C Kalagarla, Rahul Jain, and Pierluigi Nuzzo. A sample-efficient algorithm for episodic finitehorizon mdp with constraints. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pp. 8030-8037, 2021.

Emilie Kaufmann, Pierre Ménard, Omar Darwiche Domingues, Anders Jonsson, Edouard Leurent, and Michal Valko. Adaptive reward-free exploration. In Algorithmic Learning Theory , pp. 865-891. PMLR, 2021.

Marc Lanctot, Edward Lockhart, Jean-Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, et al. Openspiel: A framework for reinforcement learning in games. arXiv preprint arXiv:1908.09453 , 2019.

Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In International Conference on Machine Learning , pp. 3703-3712. PMLR, 2019.

Tao Liu, Ruida Zhou, Dileep Kalathil, Panganamala Kumar, and Chao Tian. Learning policies with zero or bounded constraint violation for constrained mdps. Advances in Neural Information Processing Systems , 34:17183-17193, 2021a.

Tao Liu, Ruida Zhou, Dileep Kalathil, Panganamala Kumar, and Chao Tian. Learning policies with zero or bounded constraint violation for constrained mdps. Advances in Neural Information Processing Systems , 34:17183-17193, 2021b.

Yongshuai Liu, Jiaxin Ding, and Xin Liu. Ipo: Interior-point policy optimization under constraints. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pp. 4940-4947, 2020.

Pierre Ménard, Omar Darwiche Domingues, Anders Jonsson, Emilie Kaufmann, Edouard Leurent, and Michal Valko. Fast active learning for pure exploration in reinforcement learning. In International Conference on Machine Learning , pp. 7599-7608. PMLR, 2021.

Sobhan Miryoosefi and Chi Jin. A simple reward-free approach to constrained reinforcement learning. In International Conference on Machine Learning , pp. 15666-15698. PMLR, 2022.

Ian Osband and Benjamin Van Roy. On lower bounds for regret in reinforcement learning. arXiv preprint arXiv:1608.02732 , 2016.

Shuang Qiu, Xiaohan Wei, Zhuoran Yang, Jieping Ye, and Zhaoran Wang. Upper confidence primaldual reinforcement learning for cmdp with adversarial loss. Advances in Neural Information Processing Systems , 33:15277-15287, 2020.

Ming Shi, Yingbin Liang, and Ness Shroff. A near-optimal algorithm for safe reinforcement learning under instantaneous hard constraints. arXiv preprint arXiv:2302.04375 , 2023.

Thiago D Simão, Nils Jansen, and Matthijs TJ Spaan. Alwayssafe: Reinforcement learning without safety constraint violations during training. 2021.

Rahul Singh, Abhishek Gupta, and Ness B Shroff. Learning in markov decision processes under constraints. arXiv preprint arXiv:2002.12435 , 2020.

Aivar Sootla, Alexander Cowen-Rivers, Jun Wang, and Haitham Bou Ammar. Enhancing safe exploration using safety state augmentation. Advances in Neural Information Processing Systems , 35:34464-34477, 2022.

Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning , pp. 9133-9143. PMLR, 2020.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv:1805.11074 , 2018.

Garrett Thomas, Yuping Luo, and Tengyu Ma. Safe reinforcement learning by imagining the near future. Advances in Neural Information Processing Systems , 34:13859-13869, 2021.

Matteo Turchetta, Felix Berkenkamp, and Andreas Krause. Safe exploration in finite markov decision processes with gaussian processes. Advances in neural information processing systems , 29, 2016.

Matteo Turchetta, Andrey Kolobov, Shital Shah, Andreas Krause, and Alekh Agarwal. Safe reinforcement learning via curriculum induction. Advances in Neural Information Processing Systems , 33:12151-12162, 2020.

Eiji Uchibe and Kenji Doya. Constrained reinforcement learning from intrinsic and extrinsic rewards. In 2007 IEEE 6th International Conference on Development and Learning , pp. 163-168. IEEE, 2007.

Akifumi Wachi and Yanan Sui. Safe reinforcement learning in constrained markov decision processes. In International Conference on Machine Learning , pp. 9797-9806. PMLR, 2020.

Akifumi Wachi, Yanan Sui, Yisong Yue, and Masahiro Ono. Safe exploration and optimization of constrained mdps using gaussian processes. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Akifumi Wachi, Yunyue Wei, and Yanan Sui. Safe policy optimization with local generalized linear function approximations. Advances in Neural Information Processing Systems , 34:20759-20771, 2021.

Yixuan Wang, Simon Sinong Zhan, Ruochen Jiao, Zhilu Wang, Wanxin Jin, Zhuoran Yang, Zhaoran Wang, Chao Huang, and Qi Zhu. Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments. arXiv preprint arXiv:2209.15090 , 2022.

Yixuan Wang, Simon Sinong Zhan, Ruochen Jiao, Zhilu Wang, Wanxin Jin, Zhuoran Yang, Zhaoran Wang, Chao Huang, and Qi Zhu. Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments. In International Conference on Machine Learning , pp. 36593-36604. PMLR, 2023.

Honghao Wei, Xin Liu, and Lei Ying. Triple-q: A model-free algorithm for constrained reinforcement learning with sublinear regret and zero constraint violation. In International Conference on Artificial Intelligence and Statistics , pp. 3274-3307. PMLR, 2022.

Dongjie Yu, Haitong Ma, Shengbo Li, and Jianyu Chen. Reachability constrained reinforcement learning. In International Conference on Machine Learning , pp. 25636-25655. PMLR, 2022.

Ming Yu, Zhuoran Yang, Mladen Kolar, and Zhaoran Wang. Convergent policy optimization for safe reinforcement learning. Advances in Neural Information Processing Systems , 32, 2019.

Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE Symposium Series on Computational Intelligence (SSCI) , pp. 737-744. IEEE, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies safe RL with step-wise violation constraints, different from the popular CMDP with\nan additive expectation cost constraint. The step-wise violation constraint is more suitable for safety-critical\nsystems. The authors propose an algorithm that provides violation and regret bound. They then further\ndevelops an algorithm to learn a near-optimal safe policy and show its effectiveness in the experiments.\n\nSTRENGTHS:\n1) This paper studies an important problem. The paper is well-written and easy to follow. \nThe RL with step-wise violation constraint, as a formulation, is novel and more general than the popular CMDP with an additive expectation constraint. \n2) The proposed approach looks correct and sound to me although I didn't check every proof. \n3) The authors provide theoretical analyses of violation and regret bound. \n4) The proposed safe RL algorithm achieves better performance than the existing baselines.\n\nWEAKNESSES:\n1. This paper may miss some reference for safe RL with step-wise violations such as\n\na) Wang, Y., Zhan, S. S., Jiao, R., Wang, Z., Jin, W., Yang, Z., ... & Zhu, Q. (2022). \nEnforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environments. \narXiv preprint arXiv:2209.15090.\n\nThe authors have to at least discuss this paper in this work as the timepoint-level \"hard chance constraint\" looks very similar to the step-wise violations proposed by the authors. \n\n2. In the MDP model, it is not clear to me whether the paper solves safe RL with continuous/discrete deterministic/stochastic\nsystems or environments. The transition set \\Delta_h(s, a) could be an infinite set when considering continuous state space. \n\n3. Essentially, my understanding of this paper is that they are dealing with discrete action and state space, and because of this,\nthe RL agent can infer which state and/or future state is unsafe to visit by building the A_safe set via dynamical programming. \nwhich is the core of the proposed algorithms.\n\nQUESTIONS:\nI am using a Windows laptop and I cannot open the full_paper file (format) in the supplementary folder, so I cannot see the limitations of the approach.\n\nLIMITATIONS:\nI would like to authors to clarify the limitations of their formulation and approach in the response.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-24"}

comments: SUMMARY:
This paper formulates and studies a strict step-wise violation constraint reinforcement learning problem, where a non-negative state-dependent cost is cumulated at each step. They show lower bounds for regret and safety violations. A model-based algorithm that matches the lower bound is provided, while another O(1) gap-dependent violation is provided too. They also study the reward-free problem and provide an algorithm that has a sample complexity with a leading term matching the SOTA of a no-constraint case.

STRENGTHS:
The step-wise violation constraint problem is meaningful. This paper is technically sound. The proofs are correct as far as I can tell. The empirical result looks good.

WEAKNESSES:
1. The idea of the unsafe set can only apply to state-dependent safety problems. I doubt this technique can apply to a (state, action)-dependent safety problem. Additionally, it is weird that the reward is (state, action)-dependent, but the cost is state-dependent only.
2. It is weird that the reward-free can give cost feedback.
3. Definition 6.1 is confusing. I guess from your proof that \pi in (4) means all feasible policies defined by \Delta and U?
4. line 545, '<' should be '>'?

QUESTIONS:
See weakness.

LIMITATIONS:
Part of the limitations are shown. The authors don't discuss the limitations of state-dependent cost as well as the weird reward-free setting.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new formulation for the safe RL problem Safe-RL-SW, whose violation constraints are step-wise, different from the existing work. A model-based general algorithmic framework SUCBVI is proposed and the theoretical guarantees on the upper bound and lower bound of its regret are provided. The authors also propose a similar reward-free safe RL problem with step-wise violation constraints (Safe-RFE-SW) along with an algorithm SRF-UCRL, the theoretical upper bound on whose regret is provided. It is claimed to be the first result on step-wise violation constraints in the RFE setting. Experiments are provided in support of the theoretical results.\n\nSTRENGTHS:\nThe authors propose new problem formulations for safe RL as well as algorithms under these formulations. Theoretical results are provided for the proposed algorithms. Overall, the new problem formulations seem original and the results seem significant.\n\nWEAKNESSES:\nHow the results in this work compare with the existing work is somewhat unclear, despite some discussion in the Related Work Section. In Section 2, the authors mention two closely-related work Amani et al. (2021) and Shi et al. (2023) that also consider step-wise violation and state that more detailed comparisons can be found in Appendix B, but I did not find any comparison with these two works in Appendix B. I strongly suggest that the authors compare with all related works and clearly state how this work differentiates from them in the revision. For example, do all CMDP works define violation with $C'(K)$ in Line 136? Can you compare with the CMDP works that do not use expected violation like $C'(K)$, if there is any? As for Amani et al. (2021) and Shi et al. (2023), please explain whether there is any difference in your formulations and how your results compare with theirs, if your setting is comparable with theirs.\n\nQUESTIONS:\nI hope the authors can provide clarification for a few questions:\n\n1. What is the role of $n$ in Theorem 5.2? Why considers $M_1, \\cdots, M_n$ in this theorem instead of just focusing on the one MDP that establishes the lower bound?\n\n2. Do all CMDP works define violation with $C'(K)$ in Line 136? Is there any CMDP work that do not use $C'(K)$ and have a violation definition that is more comparable to the step-wise violation in this work? \n\n3. In the plot of \"Step-wise Violation in Safe-RL-SW\" of Figure 1, the cumulative step-wise violation curve for SUCBVI seems to grow much more slowly than $\\sqrt{T}$. However, shouldn't we expect something $\\Omega(\\sqrt{ST})$, especially given your lower bounds in Section 5?\n\nLIMITATIONS:\nThe limitation is currently discussed in Appendix A. I encourage the authors to move part of it in the conclusion so that it is more conspicuous.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper considers online RL problem for an MDP with stage-wise constraints. The stage-wise constraints basically specify the set of unsafe states which must be avoided at all times. While there is a lot of recent work on CMDPs, this formulation, which is actually more relevant is less studied. The authors propose a variant of UCBVI algorithm, and show that it achieves O(\sqrt{SATH^3}) regret which is order optimal except in H. It also achieves sublinear constraint violation, which is independent of T when it is gap-dependent. They also present a reward-free exploration algorithms with sublinear regret and constraint violation.

STRENGTHS:
This seems one of the few results on Online RL for MDP with stage-wise constraints. Attempt at numerical evaluation is commendable.

WEAKNESSES:
Unfortunately, the results are weaker than what have been achieved recently for CMDP problems: some of the formulations only consider clipped violation functions, and still achieve bounded constraint violation. The results for reward-free learning are actually much weaker as it only achieves sublinear constraint violation. Furthermore, novelty in the algorithm is limited based as it is on UCBVI, a well-trodden path. Recent work on online learning for CMDPs has several interesting algorithmic ideas which the authors could potentially use to propose algorithms for sharper results.

QUESTIONS:
Could you use some of the recent work of Lei Ying and Dileep Kalathil to design algorithms that achieve even sharper results. I believe it is possible without too much extra work.

LIMITATIONS:
N/A.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-21

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies an episodic constrained reinforcement learning problem with step-wise constraints on states. The authors first extend the classical UCB-VI to step-wise constraints and prove the sub-linear optimality gap and step-wise constraint violation. A lower bound is also provided to show optimal dependence on the state space size and the number of episodes. Second, the authors present a reward-free algorithm that takes nearly optimal sample complexity and subliner step-wise constraint violation during exploration. Some numerical experiments are provided to show the effectiveness of the proposed algorithms.\n\nSTRENGTHS:\n**originality**\n\n- The problem formulation follows the existing tabular constrained MDP with instantaneous constraints. The safety constraint used in this work is an inequality of a function of state. The learning objective is to minimize regret and step-wise violation. The instantaneous constraint is also studied in other papers, e.g., Safe Policy Optimization with Local Generalized Linear Function Approximations and Provable Safe Reinforcement Learning with Binary Feedback. \n\n-  The only problem assumption is the existence of an initial state that is not in the unsafe set. This assumption seems to be the weakest in the literature. Since the feasible policy has to satisfy the constraint at every step, transition dynamics can't guarantee the feasibility of the next state no matter the actions taken. So, it might not be intuitive that the feasibility has nothing to do with transition dynamics. \n\n- In the first algorithm, the authors extend the existing UCBVI by incorporating the estimation of safe and unsafe sets, and prove sublinear regret and step-wise violation. It is useful if the authors could discuss how this work differs from previous applications of UCBVI to constrained MDPs, e.g., (Amani et al., 2021), which studies linear function approximation and offers zero violation. \n\n- In the second algorithm, the authors generalize the existing reward-free algorithm to step-wise constraints, and prove nearly-optimal sample complexity.  \n\n**quality**\n\n- Most statements are clarified by remarks or proofs.\n\n- It is useful if the authors can make a more technical comparison with existing works since this paper is more theoretical. \n\n- In experiments, it might be not very fair to compare other methods that study value-based constraints. \n\n- It is less discussed about the weaknesses of the proposed method. It is useful to discuss if the proposed methods handle function approximation and provide zero constraint violation.\n\n\n**clarity**\n\n- The main results are delivered clearly, and the paper is organized well. \n\n**significance**\n\n- It is important to develop RL algorithms that can learn step-wise constraints. The authors have extended two known unconstrained RL algorithms to tabular constrained MDPs with step-wise constraints. However, step-wise constraints have been tackled in several other works, which warrant detailed comparison to show the significance of this paper. \n\n- The provided theoretical analysis and guarantees are more expected. It is less discussed new challenges of extending the existing unconstrained analysis.\n\n- The provided experiments do not seem to reflect the performance of the proposed algorithms. There is no variance in learning curves. Since all algorithms directly converge, exploration does seem to be an issue for convergence, which does not explains the adaptivity of online RL algorithms.\n\nWEAKNESSES:\n- It is useful to make a more detailed comparison with existing works on constrained MDP with instantaneous constraints, especially problem assumptions. Discussing some related applications is also useful. \n\n- The technical contributions can be better explained by comparing the constrained and unconstrained ones. It is useful to highlight the main technical challenges.  \n\n- It is useful to discuss extending to large problems using function approximation and providing better violation guarantees.\n\n- For experiments, it is important to make a fair comparison with existing works on constrained MDPs with instantaneous constraints. It is also important to experiment algorithmms in a correct way by showing convergence and exploration randomness.\n\nQUESTIONS:\nSome questions are stated in Strengths and Questions.\n\nHere are some other questions.\n\n- How the confidence bonus in Algorithm 1 is constructed? \n\n- Is there a lower bound for Algorithm 2 that shows the hardness of step-wise constraints?  \n\n- To see how the proposed algorithms deal with step-wise constraints, can the authors plot the instantaneous constraint violation?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-26"}