SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Akifumi Wachi
LINE Corporation
akifumi.wachi@linecorp.com
Wataru Hashimoto
hashimoto@is.eei.eng.osaka-u.ac.jp
Xun Shen
shenxun@eei.eng.osaka-u.ac.jp
Kazumune Hashimoto
hashimoto@eei.eng.osaka-u.ac.jp
Safe exploration is essential for the practical use of reinforcement learning (RL) in many real-world scenarios. In this paper, we present a generalized safe exploration (GSE) problem as a unified formulation of common safe exploration problems. We then propose a solution of the GSE problem in the form of a meta-algorithm for safe exploration, MASE, which combines an unconstrained RL algorithm with an uncertainty quantifier to guarantee safety in the current episode while properly penalizing unsafe explorations before actual safety violation to discourage them in future episodes. The advantage of MASE is that we can optimize a policy while guaranteeing with a high probability that no safety constraint will be violated under proper assumptions. Specifically, we present two variants of MASE with different constructions of the uncertainty quantifier: one based on generalized linear models with theoretical guarantees of safety and near-optimality, and another that combines a Gaussian process to ensure safety with a deep RL algorithm to maximize the reward. Finally, we demonstrate that our proposed algorithm achieves better performance than state-of-the-art algorithms on grid-world and Safety Gym benchmarks without violating any safety constraints, even during training.
Safe reinforcement learning (RL) is a promising paradigm that enables policy optimizations for safety-critical decision-making problems (e.g., autonomous driving, healthcare, and robotics), where it is necessary to incorporate safety requirements to prevent RL policies from posing risks to humans or objects [14]. As a result, safe exploration has received significant attention in recent years as a crucial issue for ensuring the safety of RL during both the learning and execution phases [6].
Safe exploration in RL has typically been addressed by formulating a constrained RL problem in which the policy optimization is subject to safety constraints [9, 18]. While there have been many attempts under different types of constraint representations (e.g., expected cumulative cost [3], CVaR [29]), satisfying constraints almost surely or with high probability received less attention to date. Imagine safety-critical applications such as planetary exploration where even a single constraint violation may result in catastrophic failure. NASA's engineers hope Mars rovers to ensure safety at least with high probability [8]; thus, constraint satisfaction "on average" does not fit their purpose.
While several algorithms have addressed this problem with this stricter notion of safety, there are several formulations in terms of how the constraints are represented, including cumulative [32], state [39], and instantaneous constraints [41], which respectively correspond to Problems 1, 2, and 3
as we will discuss shortly in Section 2. Unfortunately, there has been limited discussion on the relationships between these approaches, making it challenging for researchers to acquire a systematic understanding of the field as a whole. If a generalized problem were to be formulated, then the research community could pool their efforts to develop suitable algorithms.
A closer examination of existing algorithms that span the entire theory-to-practice spectrum reveals several areas for improvement. Practical algorithms using deep RL (e.g., [32],[39],[40]) may provide satisfactory performance after convergence, but do not usually guarantee safety during training. In contrast, theoretical studies (e.g., [4], [41]) that guarantee safety with high probability during training often have limitations, such as relying on strong assumptions (e.g., known state transition) or experiencing decreased performance in complex environments. In summary, many algorithms have been proposed in various safe RL formulations, but the creation of a safe exploration algorithm that is both practically useful and supported by theoretical foundations remains an open problem.
Contributions. We first present a generalized safe exploration (GSE) problem and prove its generality compared with existing safe exploration problems. By taking advantage of the tractable form of the safety constraint in the GSE problem, we establish a meta-algorithm for safe exploration, MASE. This algorithm employs an uncertainty quantifier for a high-probability guarantee that the safety constraints are not violated and penalizes the agent before safety violation, under the assumption that the agent has access to an "emergency stop" authority. Our MASE is both practically useful and theoretically well-founded, which allows us to optimize a policy via an arbitrary RL algorithm under the high-probability safety guarantee, even during training. We then provide two specific variants of MASE with different uncertainty quantifiers. One is based on generalized linear models (GLMs), for which we theoretically provide high-probability guarantees of safety and near-optimality. The other is more practical, combining a Gaussian process (GP, [27]) to ensure safety with a deep RL algorithm to maximize the reward. Finally, we show that MASE performs better than state-of-the-art algorithms on the grid-world and Safety Gym [28] without violating any safety constraints, even during training.
**Definitions.** We consider an episodic safe RL problem in a constrained Markov decision process (CMDP, [3]),  $\mathcal{M} = \langle \mathcal{S}, \mathcal{A}, H, \mathcal{P}, r, g, s_1 \rangle$ , where  $\mathcal{S}$  is a state space,  $\mathcal{A}$  is an action space,  $H \in \mathbb{Z}_{>0}$  is a (fixed) length of each episode,  $\mathcal{P}: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to [0,1]$  is a state transition probability,  $r: \mathcal{S} \times \mathcal{A} \to [0,1]$  is a reward function,  $g: \mathcal{S} \times \mathcal{A} \to [0,1]$  is a safety (cost) function, and  $s_1 \in \mathcal{S}$  is an initial state. At each discrete time step, with a given (fully-observable) state  $s$ , the agent selects an action  $a$  with respect to its policy  $\pi: \mathcal{S} \to \mathcal{A}$ , receiving the new state  $s'$ , reward  $r$ , and safety cost  $g$ . Though we assume a deterministic policy, our core ideas can be extended to stochastic policy settings. Given a policy  $\pi$ , the value and action-value functions in a state  $s$  at time  $h$  are respectively defined as
$$
V _ {r, h} ^ {\pi} (s) := \mathbb {E} _ {\pi} \left[ \sum_ {h ^ {\prime} = h} ^ {H} \gamma_ {r} ^ {h ^ {\prime}} r (s _ {h ^ {\prime}}, a _ {h ^ {\prime}}) \Bigg | s _ {h} = s \right]
$$
and  $Q_{r,h}^{\pi}(s,a) \coloneqq \mathbb{E}_{\pi}\left[\sum_{h' = h}^{H}\gamma_{r}^{h'}r(s_{h'},a_{h'})\mid s_{h} = s,a_{h} = a\right]$ , where the expectation  $\mathbb{E}_{\pi}$  is taken over the random state-action sequence  $\{(s_{h'},a_{h'})\}_{h'=h}^{H}$  induced by the policy  $\pi$ . Additionally,  $\gamma_r \in (0,1]$  is a discount factor for the reward function. In the remainder of this paper, we define  $V_{\max} \coloneqq \frac{1 - \gamma_r^H}{1 - \gamma_r}$  and let  $\mathcal{T}_h: (\mathcal{S} \times \mathcal{A} \to \mathbb{R}) \to (\mathcal{S} \times \mathcal{A} \to \mathbb{R})$  denote the Bellman update operator  $\mathcal{T}_h(Q)(s,a) \coloneqq \mathbb{E}[r(s_h,a_h) + \gamma_r V_Q(s_{h+1}) \mid s_h = s,a_h = a]$ , where  $V_Q(s) \coloneqq \max_{a \in \mathcal{A}} Q(s,a)$ .
Three common safe RL problems. We tackle safe RL problems where constraints must be satisfied almost surely, even during training. While such problems have garnered attention in the research community, there are several types of formulations, and their relations are yet to be fully investigated.
One of the most popular formulations for safe RL problems involves maximizing  $V_{r}^{\pi} \coloneqq V_{r,1}^{\pi}(s_{1})$  under the constraint that the cumulative cost is less than a threshold, which is described as follows:
Problem 1 (Almost surely safe RL with cumulative constraint [32]).
$$
\max  _ {\pi} V _ {r} ^ {\pi} \quad \text {s u b j e c t t o} \quad \Pr \left[ \sum_ {h = 1} ^ {H} \gamma_ {g} ^ {h} g \left(s _ {h}, a _ {h}\right) \leq \xi_ {1} \mid \mathcal {P}, \pi \right] = 1,
$$
where  $\xi_1\in \mathbb{R}_{\geq 0}$  is a constant representing a threshold, and  $\gamma_{g}\in (0,1]$  is a discount factor for  $g$
Observe that, the expectation is not taken regarding the safety constraint in Problem 1. This problem was studied in [32], which is stricter than the conventional one where the expectation is taken with respect to the cumulative safety cost function (i.e.,  $\mathbb{E}_{\pi}[\sum_{h = 1}^{H}\gamma_{g}^{h}g(s_{h},a_{h})]\leq \xi_{1})$
Another popular formulation involves leveraging the state constraints so that safety corresponds to avoiding visits to a set of unsafe states. This type of formulation has been widely adopted by previous studies on safe-critical robotics tasks [38-40, 45], which is written as follows:
Problem 2 (Safe RL with state constraints).
$$
\max  _ {\pi} V _ {r} ^ {\pi} \quad \text {s u b j e c t t o} \quad \mathbb {E} \left[ \sum_ {h = 1} ^ {H} \gamma_ {g} ^ {h} \mathbb {I} \left(s _ {h} \in S _ {\text {u n s a f e}}\right) \Bigg | \mathcal {P}, \pi \right] \leq \xi_ {2},
$$
where  $\xi_2\in \mathbb{R}_{\geq 0}$  is a threshold,  $\mathbb{I}(\cdot)$  is the indicator function, and  $S_{\mathrm{unsafe}}\subset S$  is a set of unsafe states.
Finally, some existing studies formulate safe RL problems via an instantaneous constraint, attempting to ensure safety even during the learning stage while aiming for extremely safety-critical applications such as planetary exploration [42] or healthcare [36]. Such studies typically require the agent to satisfy the following instantaneous safety constraint at every time step.
Problem 3 (Safe RL with instantaneous constraints).
$$
\max  _ {\pi} V _ {r} ^ {\pi} \quad \text {s u b j e c t t o} \quad \Pr \left[ g \left(s _ {h}, a _ {h}\right) \leq \xi_ {3} \mid \mathcal {P}, \pi \right] = 1, \quad \forall h \in [ 1, H ],
$$
where  $\xi_3\in \mathbb{R}_{\geq 0}$  is a time-invariant safety threshold.
This paper also requires an agent to optimize a policy under a safety constraint, as in the three common safe RL problems. We seek to find the optimal policy  $\pi^{\star}:\mathcal{S}\to \mathcal{A}$  of the following problem, which will hereinafter be referred to as the "generalized" safe exploration (GSE) problem:
Problem 4 (GSE problem). Let  $b_{h} \in \mathbb{R}$  denote a time-varying threshold.
$$
\max  _ {\pi} V _ {r} ^ {\pi} \quad \text {s u b j e c t t o} \quad \Pr \left[ g (s _ {h}, a _ {h}) \leq b _ {h} \mid \mathcal {P}, \pi \right] = 1, \quad \forall h \in [ 1, H ].
$$
This constraint is instantaneous, which requires the agent to learn a policy without a single constraint violation not only after convergence but also during training. We assume that the threshold is myopically known; that is,  $b_{h}$  is known at time  $h$ , but unknown before that. Crucially, at every time step  $h$ , since  $s_{h}$  is a fully observable state and the agent's policy is deterministic, we will use a simplified inequality represented as  $g(s_{h},a_{h})\leq b_{h}$  in the rest of this paper. This constraint is akin to that in Problem 3, with the difference that the safety threshold is time-varying.
Importance of the GSE problem. Though our problem may not seem relevant to Problems 1 and 2, we will shortly present and prove a theorem on the relationship between the GSE problem and the three common safe RL problems.
Theorem 3.1. Problems 1, 2, and 3 can be transformed into the GSE problem (i.e., Problem 4).
See Appendix B for the proof. In other words, the feasible policy space in the GSE problem can be identical to those in the other three problems by properly defining the safety cost function  $g$  and threshold  $b_{h}$ . Crucially, Problem 1 is a special case of the GSE problem with  $b_{h} = \eta_{h}$  for all  $h$ , where  $\eta_{h+1} = \gamma_{g}^{-1} \cdot (\eta_{h} - g(s_{h}, a_{h}))$  with  $\eta_{0} = \xi_{1}$ . It is particularly beneficial to convert Problems 1 and 2, which have additive constraint structures, to the GSE problem, which has an instantaneous constraint. The accurate estimation of the cumulative safety value in Problems 1 and 2 is difficult because they depend on the trajectories induced by a policy. Dealing with the instantaneous constraint in the GSE problem is easier, both theoretically and empirically. Also, especially when the environment is time-varying (e.g., there are moving obstacles), the GSE problem is more useful than Problem 3.
Typical CMDP formulations with expected cumulative (safety) cost are out of the scope of the GSE problem. In such problems, the safety notion is milder; hence, although many advanced deep RL algorithms have been actively proposed that perform well in complex environments after convergence,
their performance in terms of safety during learning is usually low, as reported by Stooke et al. [33] or Wachi et al. [43]. Risk-constrained MDPs are also important safe RL problems that are not covered by the GSE problem; they have been widely studied by representing risk as a constraint on some conditional value-at-risk [11] or using chance constraints [24, 26].<sup>1</sup>
Difficulties and Assumptions. Theorem 3.1 insists that the GSE problem covers a wide range of safe RL formulations and is worth solving, but the problem is intractable without assumptions. We now discuss the difficulties in solving the GSE problem, and then list the assumptions in this paper.
The biggest difficulty with the GSE problem lies in the fact that there may be no viable safe action given the current state  $s_h$ , safety cost  $g$ , and threshold  $b_h$ . When  $b_h = 0.1$  and  $g(s_h, a) = 0.5, \forall a \in \mathcal{A}$ , the agent has no viable action for ensuring safety. The agent needs to guarantee safety, even during training, where little environmental information is available; hence, it is significant for the agent to avoid such situations where there is no action that guarantees safety. Another difficulty is related to the regularity of the safety cost function and the strictness of the safety constraint. In this paper, the safety cost function is unknown a priori.; thus, when the safety cost does not exhibit any regularity, the agent can neither infer the safety of decisions nor guarantee safety almost surely.
To address the first difficulty mentioned above, we use Assumptions 3.2 and 3.3.
Assumption 3.2 (Safety margin). There exists  $\zeta \in \mathbb{R}_{>0}$  such that  $\operatorname*{Pr}[g(s_h,a_h)\leq b_h - \zeta \mid \mathcal{P},\pi^{\star}] = 1,\forall h\in [1,H]$
Assumption 3.3 (Emergency stop action). Let  $\widehat{a}$  be an emergency stop action such that  $\mathcal{P}(s_1\mid s,\widehat{a}) = 1$  for all  $s\in S$ . The agent is allowed to execute the emergency stop action and reset the environment if and only if the probability of guaranteed safety is not sufficiently high.
Assumption 3.2 is mild; this is similar to the Slater condition, which is widely adopted in the CMDP literature [13, 25]. We consider Assumption 3.3 is also natural for safety-critical applications because it is usually better to guarantee safety, even with human interventions, if the agent requires help in emergency cases. In some applications (e.g., the agent is in a hazardous environment), however, emergency stop actions should often be avoided because of the expensive cost of human intervention. In such cases, the agent needs to learn a reset policy allowing them to return to the initial state as in Eysenbach et al. [15], rather than asking for human help, which we will leave to future work.
As for the second difficulty, we assume that the safety cost function belongs to a class where uncertainty can be estimated and guarantee the satisfaction of the safety constraint with high probability. We present an assumption regarding an important notion called an uncertainty quantifier:
Assumption 3.4 ( $\delta$ -uncertainty quantifier). Let  $\mu : \mathcal{S} \times \mathcal{A} \to \mathbb{R}$  denote the estimated mean function of safety. There exists a  $\delta$ -uncertainty quantifier  $\Gamma : \mathcal{S} \times \mathcal{A} \to \mathbb{R}$  such that  $|g(s, a) - \mu(s, a)| \leq \Gamma(s, a)$  for all  $(s, a) \in \mathcal{S} \times \mathcal{A}$ , with a probability of at least  $1 - \delta$ .
We propose MASE for the GSE problem, which combines an unconstrained RL algorithm with additional mechanisms for addressing the safety constraints. The pseudo-code is provided in Algorithm 1, and a conceptual illustration can be seen in Figure 1.
The most notable feature of MASE is that safety is guaranteed via the  $\delta$ -uncertainty quantifier and the emergency stop action (lines 3 - 9). The  $\delta$ -uncertainty quantifier is particularly useful because we can guarantee that the confidence bound contains the true safety cost function, that is,  $g(s, a) \in [\mu(s, a) \pm \Gamma(s, a)]$  for all  $s \in S$  and  $a \in A$ . This means that, if the agent chooses actions such that  $\mu(s_h, a_h) + \Gamma(s_h, a_h) \leq b_h$ , then  $g(s_h, a_h) \leq b_h$  holds with a probability of at least  $1 - \delta$ . Regarding the first difficulty mentioned in Section 3, it is crucial that there is at least one safe action. Thus, at every time step  $h$ , the agent computes a set of actions that are considered to satisfy the safety constraints with a probability at least  $1 - \delta$  given the state  $s_h$  and threshold  $b_h$ . This is represented as
$$
\mathcal {A} _ {h} ^ {+} := \left\{a \in \mathcal {A} \mid \min  \left\{1, \mu \left(s _ {h}, a\right) + \Gamma \left(s _ {h}, a\right) \right\} \leq b _ {h} \right\}.
$$
Whenever the agent identifies that at least one action will guarantee safety, the agent is required to choose an action within  $\mathcal{A}_h^+$  (line 3). The emergency stop action is executed if and only if there is no
The solution in the GSE problem is guaranteed to be a conservative approximation of that in safe RL problems with chance constraints. For more details, see Appendix C.
Algorithm 1 Meta-Algorithm for Safe Exploration (MASE)
1: for episode  $t = 1,2,\ldots ,T$  do  
2: for time  $h = 1,2,\ldots ,H$  do  
3: Take "safe" action  $a_{h} = \pi (s_{h})$  within  $A_h^+$  ▷ Execute only safe actions  
4: Receive reward  $r(s_{h},a_{h})$ , safety cost  $g(s_{h},a_{h})$ , and next state  $s_{h + 1}$   
5: Update safety threshold  $b_{h + 1}$   
6: if  $A_{h + 1}^{+} = \emptyset$  then  
7: Compute  $\widehat{r}(s_{h},a_{h}) = -\frac{c}{\min_{a\in A}\Gamma(s_{h + 1},a)}$  ▷ Penalty for the emergency stop action  
8: Append  $(s_{h},a_{h},\widehat{r}(s_{h},a_{h}),s_{h + 1})$  to  $D$   
9: break (i.e., take action  $\widehat{a}$ ) ▷ Execute the emergency stop action  
10: else  
11: Append  $(s_{h},a_{h},r(s_{h},a_{h}),s_{h + 1})$  to  $D$   
12: Optimize a policy  $\pi$  based on  $D$  via an (unconstrained) RL algorithm  
13: Update the uncertainty quantifier  $\Gamma$  and rewrite  $D$
viable action satisfying the safety constraint (i.e.,  $\mathcal{A}_{h + 1}^{+}\neq \emptyset$ ); that is, the agent is allowed to execute  $\widehat{a}$  and start a new episode from an initial safe state (lines  $6 - 9$ ). The safety cost is upper-bounded by 1 because  $g\in [0,1]$  by definition. Note that MASE proactively avoids unsafe actions by selecting the emergency stop action to take beforehand; this is in contrast to Sun et al. [37], whose method terminates the episode immediately after the agent has already violated a safety constraint.
When safety is guaranteed in the manner described above, the question remains as to how to obtain a policy that maximizes the expected cumulative reward. As such, we first convert the original CMDP  $\mathcal{M}$  to the following unconstrained MDP
$$
\widehat {\mathcal {M}} := \langle \mathcal {S}, \{\mathcal {A}, \widehat {a} \}, H, \mathcal {P}, \widehat {r}, s _ {1} \rangle .
$$
The changes from  $\mathcal{M}$  lie in the action space and the reward function, as well as in the absence of the safety cost function. First, the action space is augmented so that the agent can execute the emergency stop action,  $\widehat{a}$ . The second modification concerns the reward function. When executing the emergency stop action  $\widehat{a}$ , the agent is penalized as its sacrifice so that the same situation will not occur in future episodes; hence, we modify the reward function as follows:
$$
\widehat {r} \left(s _ {h}, a _ {h}\right) = \left\{ \begin{array}{l l} - c / \min  _ {a \in \mathcal {A}} \Gamma \left(s _ {h + 1}, a\right) & \text {i f} \mathcal {A} _ {h + 1} ^ {+} = \emptyset , \\ r \left(s _ {h}, a _ {h}\right) & \text {o t h e r w i s e ,} \end{array} \right. \tag {1}
$$
where  $c \in \mathbb{R}_{>0}$  is a positive scalar representing a penalty for performing the emergency stop. This penalty is assigned to the state-action pair  $(s_h, a_h)$  that placed the agent into the undesirable situation at time step  $h + 1$ , represented as  $A_{h + 1}^{+} = \emptyset$  (see Figure 1).
To show that MASE is a reasonable safe RL algorithm, we express the following intuitions. Consider the ideal situation in which the safety cost function is accurately estimated for any state-action pairs; that is,  $\Gamma(s,a) = 0$  for all  $(s,a)$ . In this case, all emergency stop actions are properly executed, and the safety constraint will be violated at the next time step if the agent executes other actions. It is reasonable for the agent to receive a penalty of  $\widehat{r}(s,a) = -\infty$  because this state-action pair surely causes a safety violation without the emergency stop action. Unfortunately, however, the safety
Figure 1: Conceptual illustration of MASE. At every time step  $h$ , the agent chooses action  $a_h$  within  $\mathcal{A}_h^+$ . If there is no safe action at state  $s_{h + 1}$  satisfying the constraint, the emergency stop action  $\widehat{a}$  is executed and the agent receives a large penalty for  $(s_h,a_h)$ .
cost is uncertain and the agent conservatively executes  $\widehat{a}$  although there are still actions satisfying the safety constraint, especially in the early phase of training; hence, we increase or reduce the penalty according to the magnitude of uncertainty in (1) to avoid an excessively large penalty.
We must carefully consider the fact that the quality of information regarding the modified reward function  $\widehat{r}$  is uneven in the replay buffer  $\mathcal{D}$ . Specifically, in the early phase of training, the  $\delta$ -
uncertainty quantifier is loose. Hence, the emergency stop action is likely to be executed even if viable actions remain; that is, the agent will receive unnecessary penalties. In contrast, the emergency stop actions in later phases are executed with confidence, as reflected by the tight  $\delta$ -uncertainty quantifier. Thus, as in line 15, we rewrite the replay buffer  $\mathcal{D}$  while updating  $\Gamma$  depending on the model in terms of the safety cost function (as for specific methods to update  $\Gamma$ , see Sections 5 and 6).
Connections to shielding methods. The notion of the emergency stop action is akin to shielding [2, 20] which has been actively studied in various problem settings including partially-observable environments [10] or multi-agent settings [23]. Thus, MASE can be regarded as a variant of shielding methods (especially, preemptive shielding in [2]) that is specialized for the GSE problem. On the other hand, MASE does not only block unsafe actions but also provides proper penalties for executing the emergency stop actions based on the uncertainty quantifier, which leads to rigorous theoretical guarantees presented shortly. Such theoretical advantages can be enjoyed in many safe RL problems because of the wide applicability of the GSE problem backed by Theorem 3.1.