SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Nuoya Xiong $^{1,\star}$ , Yihan Du $^{2,\dagger}$ , Longbo Huang $^{1,\ddagger,*}$
$^{2}$ University of Illinois at Urbana-Champaign
*nuoyaxiong@gmail.com
†yihandu@illinois.edu
$\ddagger$ longbohuang@tsinghua.edu.cn
We investigate a novel safe reinforcement learning problem with step-wise violation constraints. Our problem differs from existing works in that we focus on stricter step-wise violation constraints and do not assume the existence of safe actions, making our formulation more suitable for safety-critical applications that need to ensure safety in all decision steps but may not always possess safe actions, e.g., robot control and autonomous driving. We propose an efficient algorithm SUCBVI, which guarantees an  $\widetilde{\mathcal{O}} (\sqrt{ST})$  or gap-dependent  $\widetilde{\mathcal{O}} (S / \mathcal{C}_{\mathrm{gap}} + S^2 AH^2)$  step-wise violation and an  $\widetilde{\mathcal{O}} (\sqrt{H^3SAT})$  regret. Lower bounds are provided to validate the optimality in both violation and regret performance with respect to the number of states  $S$  and the total number of steps  $T$ . Moreover, we further study an innovative safe reward-free exploration problem with step-wise violation constraints. For this problem, we design the algorithm SRF-UCRL to find a near-optimal safe policy, which achieves a nearly state-of-the-art sample complexity  $\widetilde{\mathcal{O}} ((\frac{S^2AH^2}{\varepsilon} +\frac{H^4SA}{\varepsilon^2})(\log (\frac{1}{\delta}) + S))$ , and guarantees an  $\widetilde{\mathcal{O}} (\sqrt{ST})$  violation during exploration. Experimental results demonstrate the superiority of our algorithms in safety performance and corroborate our theoretical results.
In recent years, reinforcement learning (RL) (Sutton & Barto, 2018) has become a powerful framework for decision-making and learning in unknown environments. Despite the ground-breaking success of RL in games (Lanctot et al., 2019), recommendation systems (Afsar et al., 2022) and complex tasks in simulation environments (Zhao et al., 2020), most existing RL algorithms focus on optimizing the cumulative reward and do not take into consideration the risk aspect, e.g., the agent runs into catastrophic situations during control. The lack of strong safety guarantees hinders the application of RL to broader safety-critical scenarios such as autonomous driving, robotics and healthcare. For example, for robotic control in complex environments, it is crucial to prevent the robot from getting into dangerous situations, e.g., hitting walls or falling into water pools, at all times.
To handle the safety requirement, a common approach is to formulate safety as a long-term expected violation constraint in each episode. This approach focuses on seeking a policy whose cumulative expected violation in each episode is below a certain threshold. However, for applications where an agent needs to avoid disastrous situations throughout the decision process, e.g., a robot needs to avoid hitting obstacles at each step, merely reducing the long-term expected violation is not sufficient to guarantee safety.
Motivated by this fact, we investigate safe reinforcement learning with a more fine-grained constraint, called step-wise violation constraint, which aggregates all nonnegative violations at each step (no offset between positive and negative violations permitted). We name this problem Safe-RL-SW. Our step-wise violation constraint differs from prior expected violation constraint (Wachi & Sui, 2020; Efroni et al., 2020b; Kalagarla et al., 2021) in two aspects: (i) Minimizing the step-wise violation enables the agent to learn an optimal policy that avoids unsafe regions deterministically, while reducing the expected violation only guarantees to find a policy with low expected violation, instead of a per-step zero-violation policy. (ii) Reducing the aggregated nonnegative violation allows us to have a risk control for each step, while a small cumulative expected violation can still result in a large cost at some individual step and cause danger, if other steps with smaller costs offset the huge cost.
Our problem faces two unique challenges. First, the step-wise violation requires us to guarantee a small violation at each step, which demands very different algorithm design and analysis from that for the expected violation (Wachi & Sui, 2020; Efroni et al., 2020b; Kalagarla et al., 2021). Second, in safety-critical scenarios, the agent needs to identify not only unsafe states but also potentially unsafe states, which are states that may appear to be safe but will ultimately lead to unsafe regions with a non-zero probability. For example, a self-driving car needs to learn to slow down or change directions early, foreseeing the potential danger in advance in order to ensure safe driving (Thomas et al., 2021). Existing safe RL works focus mainly on the expected violation (Wachi & Sui, 2020; Liu et al., 2021b; Wei et al., 2022), or requiring some other assumptions such as imposing the prior knowledge of a safe action for each state (Amani et al., 2021). Moreover, many previous works also require strong assumptions that exclude discrete tabular MDP (Amani et al., 2021; Wachi et al., 2021; Wang et al., 2023) which is considered in our paper. Hence, techniques in previous works cannot be applied to handle step-wise violations. More detailed comparisons are provided in Section 2.
To systematically handle these two challenges, we formulate safety as an unknown cost function for each state without assuming safe actions, and consider minimizing the step-wise violation instead of the expected violation. We propose a general algorithmic framework called Safe UCBVI (SUCBVI). Specifically, in each episode, we first estimate the transition kernel and cost function in an optimistic manner, tending to regard a state as safe at the beginning of learning. After that, we introduce novel dynamic programming to identify potentially unsafe states and determine safe actions, based on our estimated transition and costs. Finally, we employ the identified safe actions to conduct value iteration. This mechanism can adaptively update dangerous regions, and help the agent plan for the future, which keeps her away from all states that may lead to unsafe states. As our estimation becomes more accurate over time, the safety violation becomes smaller and eventually converges to zero. Note that without the assumption of safe actions, the agent knows nothing about the environment at the beginning. Thus, it is impossible for her to achieve an absolute zero violation. Nevertheless, we show that SUCBVI can achieve a sub-linear  $\widetilde{\mathcal{O}} (\sqrt{ST})$  cumulative violation or an  $\widetilde{\mathcal{O}} (S / \mathcal{C}_{\mathrm{gap}} + S^2 AH^2)$  gap-dependent violation that is independent of  $T$ . This violation implies that as the RL game proceeds, the agent eventually learns how to avoid unsafe states. We also provide a matching lower bound to demonstrate the optimality of SUCBVI in both violation and regret.
Furthermore, we apply our step-wise safe RL framework to the reward-free exploration (Jin et al., 2020) setting. In this novel safe reward-free exploration, the agent needs to guarantee small step-wise violations during exploration, and also output a near-optimal safe policy. Our algorithm achieves  $\varepsilon$  cumulative step-wise violation during exploration, and also identifies a  $\varepsilon$ -optimal and safe policy. See the definition of  $\varepsilon$ -optimal and  $\varepsilon$ -safe policy in Section 6.1. Another interesting application of our framework is safe zero-sum Markov games, which we discuss in Appendix B.
The main contributions of our paper are as follows.
- We formulate the safe RL with step-wise violation constraint problem (Safe-RL-SW), which models safety as a cost function for states and aims to minimize the cumulative step-wise violation. Our formulation is particularly useful for safety-critical applications where avoiding disastrous situations at each decision step is desirable, e.g., autonomous driving and robotics.
- We provide a general algorithmic framework SUCBVI, which is equipped with an innovative dynamic programming to identify potentially unsafe states and distinguish safe actions. We establish an  $\tilde{\mathcal{O}} (\sqrt{H^3SAT})$  regret and an  $\tilde{\mathcal{O}} (\sqrt{ST})$  or  $\tilde{\mathcal{O}} (S / \mathcal{C}_{\mathrm{gap}} + S^2 AH^2)$  gap-dependent violation guarantees, which exhibits the capability of SUCBVI in attaining high rewards while maintaining small violation.
- We further establish an  $\Omega(\sqrt{HST})$  regret and an  $\Omega(\sqrt{ST})$  violation lower bounds for Safe-RL-SW. The lower bounds demonstrate the optimality of algorithm SUCBVI in both regret minimization and safety guarantee, with respect to factors  $S$  and  $T$ .
- We consider step-wise safety constraints in the reward-free exploration setting, which is called the Safe-RFE-SW problem. In this setting, we design an efficient algorithm SRF-UCRL, which ensures  $\varepsilon$  step-wise violations during exploration and plans a  $\varepsilon$ -optimal and  $\varepsilon$ -safe policy for any reward functions with probability at least  $1 - \delta$ . We obtain an  $\widetilde{\mathcal{O}}\left((\frac{S^2AH^2}{\varepsilon} + \frac{H^4SA}{\varepsilon^2})(\log(\frac{1}{\delta}) + S)\right)$  sample complexity and an  $\widetilde{\mathcal{O}}(\sqrt{ST})$  violation guarantee for SRF-UCRL, which shows the efficiency of SRF-UCRL in sampling and danger avoidance even without reward signals. To the best of our knowledge, this work is the first to study the step-wise violation constraint in the RFE setting.
Safe RL. Safety is an important topic in RL, which has been extensively studied. The constrained Markov decision process (CMDP)-based approaches handle safety via cost functions, and aim to minimize the expected episode-wise violation, e.g., (Yu et al., 2019; Wachi & Sui, 2020; Qiu et al., 2020; Efroni et al., 2020b; Turchetta et al., 2020; Ding et al., 2020; Singh et al., 2020; Kalagarla et al., 2021; Simão et al., 2021; Ding et al., 2021), or achieve zero episode-wise violation, e.g., (Liu et al., 2021b; Bura et al., 2021; Wei et al., 2022; Sootla et al., 2022). Apart from CMDP-based approaches, there are also other works that tackle safe RL by control-based approaches (Berkenkamp et al., 2017; Chow et al., 2018; Dalal et al., 2018; Wang et al., 2022), policy optimization (Uchibe & Doya, 2007; Achiam et al., 2017; Tessler et al., 2018; Liu et al., 2020; Stooke et al., 2020) and safety shields (Alshiekh et al., 2018).
In recent years, there are also some works studying step-wise violations with additional assumptions. Now we provide detailed comparisons between existing papers with instantaneous constraints. Turchetta et al. (2016); Wachi et al. (2018) propose a GP-based algorithm, which assumes that the transition is deterministic and known, while modeling the reward and cost functions using Gaussian Processes. By using this particular structure, they can infer the safety cost by estimating the parameters. Wachi et al. (2021) assumes the reward and cost functions have a generalized linear structure. Their algorithm explores in a safe space until a time  $t^*$  when the agent explores sufficiently. However, the upper bound of the exploring time  $t^*$  is not given in their paper. In fact, under the tabular MDP setting  $t^*$  can be infinite.
Amani et al. (2021) further considers the reward and cost functions to have a linear structure. It makes two assumptions: (a) There exists a safe action in each state, which prevents the agent from going to a potentially unsafe state. (b) The feature set is a star convex set, which helps them change actions continuously. However, this makes their works infeasible in tabular MDPs: The feature set in tabular MDPs consists of one-hot vectors and is not a star convex set. Hence, the work Amani et al. (2021) cannot solve our problem. Shi et al. (2023) considers safe RL in linear mixture MDPs. Their work also contains assumption (b), making it infeasible in tabular MDPs. Moreover, although they do not have assumption (a), it assumes the transition set  $\Delta(s,a)$  is known, which is not needed in our paper. Thus our paper is more challenging since we need to estimate  $\Delta(s,a)$  in our algorithm adaptively.
There are some other papers investigating the safety of RL problems. Alshiekh et al. (2018) represents the safe state by the reactive system, and uses shielding to calculate and restrict the agent within a safe trajectory completely. The main difference between their work and our work is that we need to dynamically update the estimated safe state, while they require to know the mechanism and state to calculate the shield. Dalal et al. (2018) considers restricting the safe action by projecting the action into the closest safe actions. They achieve this goal by solving a convex optimization problem on the continuous action set. However, in their paper, they do not consider the situation where a state can have no safe actions. To be more specific, they do not consider the situation when the convex optimization problem has no solutions. Le et al. (2019) considers the decision-making problem with a pre-collected dataset. Turchetta et al. (2020) and Sootla et al. (2022) both consider cumulative cost constraints rather than step-wise constraints. The former uses a teacher for intervention to keep the agent away from the unsafe region, while the latter encourages safe exploration by augmenting a safety state to measure safety during training.
Reward-free Exploration with Safety. Motivated by sparse reward signals in realistic applications, reward-free exploration (RFE) has been proposed in Jin et al. (2020), and further developed
in Kaufmann et al. (2021); Menard et al. (2021). In RFE, the agent explores the environment without reward signals. After enough exploration, the reward function is given, and the agent needs to plan a near-optimal policy based on his knowledge collected in exploration. Safety is also important in RFE: We need to not only guarantee that our outputted policy is safe, but also ensure small violations during exploration.
Recently, Miryoosefi & Jin (2022); Huang et al. (2022) also study RFE with safety constraints. Compared to our paper, Miryoosefi & Jin (2022) only considers the safety constraint after the exploration phase. Huang et al. (2022) differs from our work in the following aspects: (i) They allow different reward functions during exploration and after exploration, and the agent can directly know the true costs during training. In our work, the cost function is the same during and after exploration, but the agent can only observe the noisy costs of a state when she arrives at that state. (ii) They require prior knowledge of safe baseline policies, while we do not need such an assumption. (iii) They consider the zero-expected violation during exploration, while we focus on keeping small step-wise violations.
3 The Safe MDP Model
Episodic MDP. In this paper, we consider the finite-horizon episodic Markov Decision Process (MDP), represented by a tuple  $(S, \mathcal{A}, H, \mathbb{P}, r)$ . Here  $\mathcal{S}$  is the state space,  $\mathcal{A}$  is the action space, and  $H$  is the length of each episode.  $\mathbb{P} = \{\mathbb{P}_h : S \times \mathcal{A} \mapsto \triangle_S\}_{h \in [H]}$  is the transition kernel, and  $\mathbb{P}_h(s'|s, a)$  gives the transition probability from  $(s, a)$  to  $s'$  at step  $h$ .  $r = \{r_h : S \times \mathcal{A} \mapsto [0,1]\}_{h \in [H]}$  is the reward function, and  $r_h(s, a)$  gives the reward of taking action  $a$  in state  $s$  at step  $h$ . A policy  $\pi = \{\pi_h : S \mapsto \mathcal{A}\}_{h \in [H]}$  consists of  $H$  mappings from the state space to action space. In each episode  $k$ , the agent first chooses a policy  $\pi^k$ . At each step  $h \in [H]$ , the agent observes a state  $s_h^k$ , takes an action  $a_h^k$ , and then goes to a next state  $s_{h+1}^k$  with probability  $\mathbb{P}_h(s_{h+1}^k | s_h^k, a_h^k)$ . The algorithm executes  $T = HK$  steps. Moreover, the state value function  $V_h^\pi(s, a)$  and state-action value function  $Q_h^\pi(s, a)$  for a policy  $\pi$  can be defined as
$$
V _ {h} ^ {\pi} (s) := \mathbb {E} _ {\pi} \left[ \sum_ {h ^ {\prime} = h} ^ {H} r _ {h ^ {\prime}} \left(s _ {h ^ {\prime}}, \pi_ {h ^ {\prime}} \left(s _ {h ^ {\prime}}\right)\right) \Bigg | s _ {h} = s \right],
$$
$$
Q _ {h} ^ {\pi} (s, a) := \mathbb {E} _ {\pi} \left[ \sum_ {h ^ {\prime} = h} ^ {H} r _ {h ^ {\prime}} (s _ {h ^ {\prime}}, \pi_ {h ^ {\prime}} (s _ {h ^ {\prime}})) \Bigg | s _ {h} = s, a _ {h} = a \right].
$$
Safety Constraint. To model unsafe regions in the environment, similar to Wachi & Sui (2020); Yu et al. (2022), we define a safety cost function  $c: S \mapsto [0,1]$ . Let  $\tau \in [0,1]$  denote the safety threshold. A state is called safe if  $c(s) \leq \tau$ , and called unsafe if  $c(s) > \tau$ . Similar to Efroni et al. (2020b); Amani et al. (2021), when the agent arrives in a state  $s$ , she will receive a cost signal  $z(s) = c(s) + \zeta$  where  $\zeta$  is an independent, zero-mean and 1-sub-Gaussian noise. Denote  $(x)_+ = \max\{x, 0\}$ . The violation in state  $s$  is defined as  $(c(s) - \tau)_+$ , and the cumulative step-wise violation till episode  $K$  is
$$
C (K) = \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left(c \left(s _ {h} ^ {k}\right) - \tau\right) _ {+}. \tag {1}
$$
Eq. (1) represents the accumulated step-wise violation during training. When the agent arrives in state  $s_h^k$  at step  $h$  in episode  $k$ , she will suffer violation  $(c(s_h^k) - \tau)_+$ . This violation setting is significantly different from the previous CMDP setting (Qiu et al., 2020; Ding et al., 2020; Efroni et al., 2020b; Ding et al., 2021; Wachi & Sui, 2020; Liu et al., 2021a; Kalagarla et al., 2021). They study the episode-wise expected violation  $C'(K) = \sum_{k=1}^{K} \left( \mathbb{E} \left[ \sum_{h=1}^{H} c(s_h^k, a_h^k) \right] - \mu \right)$ . There are also some papers (Efroni et al., 2020a; Simão et al., 2021) considering a stricter constraint named episode-wise clipped expected violation:  $C''(K) = \sum_{k=1}^{K} \left( \mathbb{E} \left[ \sum_{h=1}^{H} c(s_h^k, a_h^k) \right] - \mu \right)_+$ . Compared to the episode-wise violation (including expected violation and clipped expected violation), our step-wise violation has two main differences: (i) First, the episode-wise violation constraints allow the agent to get into unsafe states occasionally. Instead, the step-wise violation constraint forces the agent to stay in safe regions at all times. (ii) Second, in the episode-wise constraints, the average violation
$\mathbb{E}[c(s_h,a_h)] - \mu /H$  at step  $h$  is allowed to be positive or negative, and they can cancel out in one episode to achieve  $\mathbb{E}[\sum_{h = 1}^{H}c(s_h^k,a_h^k)]\leq \mu$ . Instead, we consider a nonnegative function  $(c(s) - \tau)_+$  at each step in our step-wise violation, which imposes a stricter constraint.
Define  $\mathcal{U} := \{s \in \mathcal{S} \mid c(s) > \tau\}$  as the set of all unsafe states. Let  $\iota = \{s_1, a_1, \dots, s_H, a_H\}$  denote a trajectory. Since a feasible policy needs to satisfy the constraint at each step, we define the set of feasible policies as  $\Pi = \{\pi \mid \operatorname*{Pr}\{\exists h \in [H], s_h \in \mathcal{U} \mid \iota \sim \pi\} = 0\}$ . The feasible policy set  $\Pi$  consists of all policies under which one never reaches any unsafe state in an episode.
Learning Objective. In this paper, we consider the regret minimization objective. Specifically, define  $\pi^{*} = \operatorname{argmax}_{\pi \in \Pi} V_{1}^{\pi}$ ,  $V^{*} = V^{\pi^{*}}$  and  $Q^{*} = Q^{\pi^{*}}$ . The regret till  $K$  episodes is then defined as
$$
R (K) = \sum_ {t = 1} ^ {K} \left(V _ {1} ^ {*} \left(s _ {1}\right) - V _ {1} ^ {\pi^ {k}} \left(s _ {1}\right)\right),
$$
where  $\pi^k$  is the policy taken in episode  $k$ . Our objective is to minimize  $R(K)$  to achieve a good performance, and minimize the violation  $C(K)$  to guarantee the safety at the same time.
4 Safe RL with Step-wise Violation Constraints
4.1 Assumptions and Problem Features
Before introducing our algorithms, we first state the important assumptions and problem features for Safe-RL-SW.
Suppose  $\pi$  is a feasible policy. Then, if we arrive at  $s_{H - 1}$  at step  $H - 1$ ,  $\pi$  needs to select an action that guarantees  $s_H\notin \mathcal{U}$ . Define the transition set  $\Delta_h(s,a) = \{s'|\mathbb{P}_h(s'|s,a) > 0\}$  for any  $\forall (s,a,h)\in S\times \mathcal{A}\times [H]$ , which represents the set of possible next states after taking action  $a$  in state  $s$  at step  $h$ . Then, at the former step  $H - 1$ , the state  $s$  is potentially unsafe if it satisfies that  $\Delta_{H - 1}(s,a)\cap \mathcal{U}\neq \emptyset$  for all  $a\in \mathcal{A}$ . (i.e., no matter taking what action, there is a positive probability of transitioning to an unsafe next state). Therefore, we can recursively define the set of potentially unsafe states at step  $h$  as
$$
\mathcal {U} _ {h} = \mathcal {U} _ {h + 1} \cup \{s \mid \forall a \in \mathcal {A}, \Delta_ {h} (s, a) \cap \mathcal {U} _ {h + 1} \neq \emptyset \}, \tag {2}
$$
where  $\mathcal{U}_H = \mathcal{U}$ . Intuitively, if we are in a state  $s_h \in \mathcal{U}_h$  at step  $h$ , no action can be taken to completely avoid reaching potentially unsafe states  $s_{h+1} \in \mathcal{U}_{h+1}$  at step  $h+1$ . Thus, in order to completely prevent from getting into unsafe states  $\mathcal{U}$  throughout all steps, one needs to avoid potentially unsafe states in  $\mathcal{U}_h$  at step  $h$ . From the above argument, we have that  $s_1 \notin \mathcal{U}_1$  is equivalent to the existence of feasible policies. The detailed proof is provided in Appendix D. Thus, we make the following necessary assumption.
Assumption 4.1 (Existence of feasible policies). The initial state  $s_1$  satisfies  $s_1 \notin \mathcal{U}_1$ .
For any  $s \in S$  and  $h \in [H - 1]$ , we define the set of safe actions for state  $s$  at step  $h$  as
$$
A _ {h} ^ {\text {s a f e}} (s) = \left\{a \in \mathcal {A} \mid \Delta_ {h} (s, a) \cap \mathcal {U} _ {h + 1} = \emptyset \right\}, \tag {3}
$$
and let  $A_H^{safe}(s) = \mathcal{A}$ .  $A_h^{safe}(s)$  stands for the set of all actions at step  $h$  which will not lead to potentially unsafe states in  $\mathcal{U}_{h+1}$ . Here,  $\{\mathcal{U}_h\}_{h \in [H]}$  and  $\{A_h^{safe}(s)\}_{h \in [H]}$  are defined by dynamic programming: If we know sets of all possible next state  $\{\Delta_h(s, a)\}_{h \in [H]}$  and unsafe state set  $\mathcal{U} = \mathcal{U}_H$ , we can calculate all potentially unsafe state sets  $\{\mathcal{U}_h\}_{h \in [H]}$  and safe action sets  $\{A_h^{safe}(s)\}_{h \in [H]}$ , and choose feasible policies to completely avoid unsafe states.
4.2 Algorithm SUCBVI
Now we present our main algorithm Safe UCBVI (SUCBVI), which is based on previous classic RL algorithm UCBVI (Azar et al., 2017), and equipped with a novel dynamic programming to identify potentially unsafe states and safe actions. The pseudo-code is shown in Algorithm 1. First, we provide some intuitions about how SUCBVI works. At each episode, we first estimate all the unsafe states based on the historical data. Then, we perform a dynamic programming procedure introduced