SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[38] Thananjeyan, B., Balakrishna, A., Nair, S., Luo, M., Srinivasan, K., Hwang, M., Gonzalez, J. E., Ibarz, J., Finn, C., and Goldberg, K. (2021). Recovery RL: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, 6(3):4915-4922.
[39] Thomas, G., Luo, Y., and Ma, T. (2021). Safe reinforcement learning by imagining the near future. In Neural Information Processing Systems (NeurIPS).
[40] Turchetta, M., Kolobov, A., Shah, S., Krause, A., and Agarwal, A. (2020). Safe reinforcement learning via curriculum induction. In Neural Information Processing Systems (NeurIPS), volume 33, pages 12151-12162.
[41] Wachi, A. and Sui, Y. (2020). Safe reinforcement learning in constrained Markov decision processes. In International Conference on Machine Learning (ICML).
[42] Wachi, A., Sui, Y., Yue, Y., and Ono, M. (2018). Safe exploration and optimization of constrained MDPs using Gaussian processes. In AAAI Conference on Artificial Intelligence (AAAI).
[43] Wachi, A., Wei, Y., and Sui, Y. (2021). Safe policy optimization with local generalized linear function approximations. In Neural Information Processing Systems (NeurIPS).
[44] Wang, Y., Wang, R., Du, S. S., and Krishnamurthy, A. (2020). Optimism in reinforcement learning with generalized linear function approximation. In International Conference on Learning Representations (ICLR).
[45] Wang, Y., Zhan, S. S., Jiao, R., Wang, Z., Jin, W., Yang, Z., Wang, Z., Huang, C., and Zhu, Q. (2023). Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments. In International Conference on Machine Learning, pages 36593-36604. PMLR.
[46] Yang, L. and Wang, M. (2019). Sample-optimal parametric Q-learning using linearly additive features. In International Conference on Machine Learning (ICML).
A Limitations and Potential Negative Societal Impacts
We will first discuss limitations and potential negative societal impacts regarding our work.
One of the major limitations of this study is that emergency stop actions are allowed for agents. Emergency stop actions should often be avoided because of the expensive cost of human intervention in many applications (e.g., the agent is in a hazardous or remote environment). In future work, we will investigate an algorithm that requires the agent to learn a reset policy allowing them to return to the initial state as in [15], rather than asking for human intervention via emergency stop actions.
Another limitation is how to construct the uncertainty quantifier. In our experiment, because we used a computationally inexpensive deep GP algorithm [30] and the uncertainty quantifier is updated at the end of the episode (see Line 13 in Algorithm 1), the computational time of the GP part was much smaller than the RL part in our experiment settings. However, since GP is generally a computationally expensive algorithm, GP can be a computational bottleneck in some cases.
A.2 Potential Negative Societal Impacts
We believe that safety is an essential requirement for applying RL in many real problems. While we have not found any potential negative societal impact of our proposed method, we must remain aware that RL algorithms are vulnerable to misuse and ours is no exception.
We first present lemmas regarding the relationship between the GSE problem and Problems 1, 2, and 3. After that, we present the proof for the Theorem 3.1 in Appendix B.4 by combining them.
B.1 Relationship between the GSE problem and Problem 1
Lemma B.1. Problem 1 can be transformed into the GSE problem.
Proof. We first utilize a safety state augmentation technique presented in Sootla et al. [32] by defining a new variable  $\eta_h$  such that
$$
\eta_ {h} := \gamma_ {g} ^ {- h} \cdot \left(\xi_ {1} - \sum_ {h ^ {\prime} = 1} ^ {h - 1} \gamma_ {g} ^ {h ^ {\prime}} g \left(s _ {h ^ {\prime}}, a _ {h ^ {\prime}}\right)\right), \quad \forall h \in [ 1, H ]. \tag {4}
$$
This new variable  $\eta_h$  means the remaining safety budget associated with the discount factor  $\gamma_g$ , which is updated as follows:
$$
\eta_ {h + 1} = \gamma_ {g} ^ {- 1} \cdot \left(\eta_ {h} - g \left(s _ {h}, a _ {h}\right)\right) \quad \text {w i t h} \quad \eta_ {0} = \xi_ {1}. \tag {5}
$$
By (4), the necessary and sufficient condition for satisfying the constraint in Problem 1 is
$$
\eta_ {h} \geq 0, \quad \forall h \in [ 1, H ]. \tag {6}
$$
By (5), we have
$$
\eta_ {h + 1} \geq 0, \quad \forall h \in [ 1, H ] \iff \eta_ {h} - g \left(s _ {h}, a _ {h}\right) \geq 0, \quad \forall h \in [ 1, H ]. \tag {7}
$$
In summary, by introducing the new variable  $\eta_h$ , Problem 1 is rewritten to the following problem:
$$
\max  _ {\pi} V _ {r} ^ {\pi} \quad \text {s u b j e c t t o} \quad \Pr \left[ g \left(s _ {h}, a _ {h}\right) \leq \eta_ {h} \mid \mathcal {P}, \pi \right] = 1, \quad \forall h \in [ 1, H ]. \tag {8}
$$
The aforementioned problem (8) is a special case of the GSE problem with  $b_{h} \coloneqq \eta_{h}$ . Therefore, we obtain the desired lemma.
B.2 Relationship between the GSE problem and Problem 2
Lemma B.2. Problem 2 can be transformed into the GSE problem.
Proof. The following chain of inequalities holds:
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {h = 1} ^ {H} \gamma_ {g} ^ {h} \mathbb {I} \left(s _ {h} \in S _ {\text {u n s a f e}}\right) \mid \mathcal {P}, \pi \right] \leq \xi_ {2} \\ \Longleftrightarrow \mathbb {E} \left[ \sum_ {h ^ {\prime} = 1} ^ {h} \gamma_ {g} ^ {h ^ {\prime}} \mathbb {I} \left(s _ {h ^ {\prime}} \in S _ {\text {u n s a f e}}\right) \mid \mathcal {P}, \pi \right] \leq \xi_ {2}, \quad \forall h \in [ 1, H ] \\ \Longleftrightarrow \mathbb {E} \left[ \gamma_ {g} ^ {h} \mathbb {I} \left(s _ {h} \in S _ {\text {u n s a f e}}\right) \mid \mathcal {P}, \pi \right] \leq \xi_ {2} - \mathbb {E} \left[ \sum_ {h ^ {\prime} = 1} ^ {h - 1} \gamma_ {g} ^ {h ^ {\prime}} \mathbb {I} \left(s _ {h ^ {\prime}} \in S _ {\text {u n s a f e}}\right) \mid \mathcal {P}, \pi \right], \quad \forall h \in [ 2, H ] \\ \Longleftrightarrow \mathbb {E} \left[ \mathbb {I} \left(s _ {h} \in S _ {\text {u n s a f e}}\right) \mid \mathcal {P}, \pi \right] \leq \gamma_ {g} ^ {- h} \left\{\xi_ {2} - \mathbb {E} \left[ \sum_ {h ^ {\prime} = 1} ^ {h - 1} \gamma_ {g} ^ {h ^ {\prime}} \mathbb {I} \left(s _ {h ^ {\prime}} \in S _ {\text {u n s a f e}}\right) \mid \mathcal {P}, \pi \right] \right\}, \quad \forall h \in [ 2, H ] \\ \end{array}
$$
Because we assume Markov property, we simply define the safety cost function  $g: S \times \mathcal{A} \to \mathbb{R}$  as
$$
g \left(s _ {h}, a\right) := \mathbb {E} \left[ \mathbb {I} \left(s _ {h} \in S _ {\text {u n s a f e}}\right) \mid \mathcal {P}, \pi \right], \quad \forall a \in \mathcal {A}.
$$
We now set
$$
b _ {h} := \gamma_ {g} ^ {- h} \left\{\xi_ {2} - \mathbb {E} \left[ \sum_ {h ^ {\prime} = 1} ^ {h - 1} \gamma_ {g} ^ {h ^ {\prime}}   \mathbb {I} (s _ {h ^ {\prime}} \in S _ {\text {u n s a f e}}) \Big | \mathcal {P}, \pi \right] \right\},
$$
then the Problem 2 can be transformed into
$$
\Pr \left[ g (s, a) \leq b _ {h} \mid \mathcal {P}, \pi \right] = 1.
$$
Finally, we obtained the desired lemma.
B.3 Relationship between the GSE problem and Problem 3
Lemma B.3. Problem 3 can be transformed into the GSE problem.
Proof. Set  $b_{h} = \xi_{3}$  for all  $h$ . Then, the GSE problem is identical to Problem 3.
B.4 Summary
Proof. Combining Lemma B.1, B.2, and B.3, we obtain the desired Theorem 3.1.
C Connections to Safe RL Problems with Chance Constraints
As a strongly related formulation to Problem 2, policy optimization under joint chance-constraints has been studied especially in the field of control theory such as Ono et al. [24] and Pfrommer et al. [26], which is written as
Problem 5 (Safe RL with joint chance constraints). Let  $\xi_5 \in \mathbb{R}_{\geq 0}$  be a constant representing a safety threshold. Also, let  $S_{\mathrm{unsafe}} \subset S$  denote a set of unsafe states. Find the optimal policy  $\pi^{\star}$  such that
$$
\max  _ {\pi} V _ {r} ^ {\pi} \quad \text {s u b j e c t t o} \quad \Pr \left[ \bigvee_ {h = 1} ^ {H} s _ {h} \in S _ {\text {u n s a f e}} \mid \mathcal {P}, \pi \right] \leq \xi_ {5}.
$$
Lemma C.1. Problem 2 is a conservative approximation of Problem 5.
Proof. This lemma mostly follows from Theorem 1 in Ono et al. [24]. Regarding the constraint in Problem 5, we have the following chain of equations:
$$
\begin{array}{l} \Pr \left[ \bigvee_ {h = 1} ^ {H} s _ {h} \in S _ {\text {u n s a f e}} \mid \mathcal {P}, \pi \right] \leq \sum_ {h = 1} ^ {H} \Pr \left[ s _ {h} \in S _ {\text {u n s a f e}} \mid \mathcal {P}, \pi \right] \\ = \sum_ {h = 1} ^ {H} \mathbb {E} \left[ \mathbb {I} \left(s _ {h} \in S _ {\text {u n s a f e}}\right) \mid \mathcal {P}, \pi \right] (9) \\ = \mathbb {E} \left[ \sum_ {h = 1} ^ {H} \mathbb {I} \left(s _ {h} \in S _ {\text {u n s a f e}}\right) \mid \mathcal {P}, \pi \right]. (10) \\ \end{array}
$$
In the first step, we used Boole's inequality (i.e.,  $\operatorname{Pr}[A \cup B] \leq \operatorname{Pr}[A] + \operatorname{Pr}[B]$ ). The final term in (10) is the LHS of the constraint in Problem 2, which implies that Problem 2 is a conservative approximation of Problem 5. Therefore, the GSE problem is also a conservative approximation of Problem 5.
Corollary C.2. Suppose we solve the GSE problem by properly defining the safety function  $g(\cdot, \cdot)$  and the threshold  $b_h$ . Then, the obtained policy is a conservative solution of Problem 5.
Remark C.3. It is extremely challenging to directly solve the Problem 5 characterized by joint chance-constraints without approximation, as discussed in Ono et al. [24]. Practically, it would be a promising and realistic approach to solve Problem 5 by converting it into the GSE problem.
More detailed explanations for the aforementioned remark are as follows. It is extremely challenging to directly solve the Problem 5 characterized by joint chance-constraints. Most of the previous work does not directly deal with this type of constraint and uses some approximations or assumptions. For example, Pfrommer et al. [26] assume a known linear time-invariant dynamics. Also, Ono et al. [24] approximate the joint chance constraint as in the above procedure and obtain
$$
\Pr \left[ \bigvee_ {h = 1} ^ {H} s _ {h} \in S _ {\text {u n s a f e}} \Bigg | \mathcal {P}, \pi \right] \leq \mathbb {E} \left[ \sum_ {h = 1} ^ {H} \mathbb {I} (s _ {h} \in S _ {\text {u n s a f e}}) \Bigg | \mathcal {P}, \pi \right].
$$
This is a conservative approximation with an additive structure, which is easier to solve than the original joint chance constraint. Ono et al. [24] deals with the above constraints with additive safety structure. By additionally transforming the conservatively-approximated problem into the GSE problem, the problem would become easier to handle because the safety constraint is instantaneous.
D Proof of Theorems 4.1 and 4.2
Proof. Recall that, at every time step  $h$ , the MASE chooses actions satisfying
$$
\mu \left(s _ {h}, a _ {h}\right) + \Gamma \left(s _ {h}, a _ {h}\right) \leq b _ {h}. \tag {11}
$$
By definition of the  $\delta$ -uncertainty quantifier, the actions that are conservatively chosen based on (11) also satisfy the safety constraint, with a probability of at least  $1 - \delta$ ; that is,
$$
g \left(s _ {h}, a _ {h}\right) \leq b _ {h}. \tag {12}
$$
In addition, when there is no action satisfying (11), the emergency stop action is executed; that is, safety is guaranteed with a probability of 1. Hence, the desired theorem is now obtained.
Proof. Assumption 3.2 implies that there exists a policy that satisfies a more conservative safety constraint written as
$$
g \left(s _ {h}, a _ {h}\right) \leq b _ {h} - \zeta , \quad \forall h \in [ 1, H ]. \tag {13}
$$
By combining (13) and the assumption  $\Gamma(s, a) \leq \frac{1}{2}\zeta, \forall (s, a) \in S \times \mathcal{A}$ , we have
$$
\mu \left(s _ {h}, a _ {h}\right) + \Gamma \left(s _ {h}, a _ {h}\right) \leq g \left(s _ {h}, a _ {h}\right) + \zeta \leq b _ {h}, \quad \forall h \in [ 1, H ], \tag {14}
$$
Algorithm 2 GLM-MASE
1: for episode  $t = 1,2,\dots ,T$  do   
2: for time  $h = 1,\ldots ,H$  do   
3: Take action  $a_{h} = \pi (s_{h})$  within  $\mathcal{A}_h^+$    
4:Receive reward  $r(s_h,a_h)$  and next state  $s_{h + 1}$    
5:Receive safety cost  $g(s_{h},a_{h})$  and update safety threshold  $b_{h + 1}$    
6: if  $\mathcal{A}_h^+ = \emptyset$  then   
7: Compute  $\widehat{r} (s_h,a_h) = -\frac{c}{\min_{a\in\mathcal{A}}\Gamma(s_{h + 1},a)}$    
8:Append  $(s_h,a_h,\widehat{r} (s_h,a_h),s_{h + 1})$  to  $\mathcal{D}$    
9: break (i.e., emergency stop action  $\widehat{a}$ )   
10: else   
11: Append  $(s_h,a_h,r(s_h,a_h),s_{h + 1})$  to  $\mathcal{D}$    
12: Update  $\widehat{\theta}_{h,t}^{g}$  and  $\widehat{\theta}_{h,t}^{Q}$    
13: Compute the optimistic Q-estimate   
 $\widehat{Q}_{\widehat{r},h}^{(t)}(s,a) = \min \{V_{\max},f(\langle \phi_{s,a},\widehat{\theta}_{h,t}^{Q}\rangle) + C_{Q / g}\Gamma (s,a)\}$    
14: Optimize the policy by   
 $\pi_h^{(t)}(s) = \arg \max_{a\in \mathcal{A}}\widehat{Q}_{\widehat{r},h}^{(t)}(s,a).$    
15: Update  $\Gamma (s,a)\coloneqq C_g\cdot \| \phi_{s,a}\|_{\Lambda_{h,t}^{-1}}$  and then rewrite  $\mathcal{D}$
which guarantees that there exists a policy that conservatively satisfies the safety constraint via the  $\delta$ -uncertainty quantifier  $\Gamma(\cdot, \cdot)$  at every time step  $h$ , with a probability of at least  $1 - \delta$ .
When we set  $c$  to be a sufficiently large scalar such that  $c > \frac{\zeta}{2\gamma_r^H} V_{\mathrm{max}}$ , the penalty  $\widehat{r}$  satisfies
$$
\begin{array}{l} \widehat {r} \left(s _ {h}, a _ {h}\right) = \frac {- c}{\operatorname* {m i n} _ {a \in \mathcal {A}} \Gamma \left(s _ {h + 1} , a\right)} \\ <   \frac {- \frac {1}{2} \zeta \cdot \frac {1}{\gamma_ {r} ^ {H}} V _ {\mathrm {m a x}}}{\frac {1}{2} \zeta} \\ <   - \gamma_ {r} ^ {- H} V _ {\max}. \\ \end{array}
$$
This means that, when the constraint violation happens even a single time, the value by a policy obtained in  $\widehat{\mathcal{M}}$  becomes negative because  $\max_s V_r^\pi (s)\leq V_{\mathrm{max}}$ .
Under Assumption 3.2, after convergence, the optimal policy in  $\widehat{M}$  will not violate the safety constraint, and thus the emergency stop action  $\widehat{a}$  will not be executed. In this case, the modified (unconstrained) MDP  $\widehat{\mathcal{M}}$  is identical to the original CMDP  $\mathcal{M}$ . Therefore, we now obtain the desired theorem.
E Supplementary materials regarding GLM-MASE
E.1 Pseudo-code for GLM-MASE
We first present the pseudo-code for GLM-MASE in Algorithm 2.
E.2 Proofs of Lemmas 5.4 and 5.5
Proof. See Lemma 1 (and Lemma 7) in Wang et al. [44].
E.3 Preliminary Lemmas
Lemma E.1. Suppose the assumptions in Lemma 5.4 and 5.5 hold. Let  $C_1$  and  $C_2$  be positive, universal constants. Also, with a sufficiently large  $T$ , let  $t^*$  denote the smallest integer satisfying
$$
\lambda_ {\min } (\Sigma) t H - C _ {1} \sqrt {t H d} - C _ {2} \sqrt {t H \ln \delta^ {- 1}} \geq 2 \cdot C _ {g} \cdot \zeta^ {- 1} \tag {15}
$$
where  $\lambda_{\mathrm{min}}(\Sigma)$  is the minimum eigenvalue of the second moment matrix  $\Sigma$ . Then, we have
$$
\Gamma \left(s _ {h} ^ {(t)}, a _ {h} ^ {(t)}\right) \leq \frac {1}{2} \cdot \zeta . \tag {16}
$$
Proof. By Proposition 1 of Li et al. [22],
$$
\lambda_ {\min } \left(\Lambda_ {h, t}\right) \geq \lambda_ {\min } (\Sigma) t H - C _ {1} \sqrt {t H d} - C _ {2} \sqrt {t H \ln \delta^ {- 1}}.
$$
By combining the aforementioned inequality with (15), we have
$$
\lambda_ {\min } (\Lambda_ {h, t}) \geq 2 \cdot C _ {g} \cdot \zeta^ {- 1}.
$$
Using the definition of  $\lambda_{\max}(\Lambda_{h,t}^{-1}) = \frac{1}{\lambda_{\min}(\Lambda_{h,t})}$ , the following chain of equations hold for all  $t \in [t^{\star}, T]$  and  $h \in [1, H]$ :
$$
\begin{array}{l} \Gamma (s _ {h} ^ {(t)}, a _ {h} ^ {(t)}) = C _ {g} \cdot \| \phi_ {s, a} \| _ {\Lambda_ {h, t} ^ {- 1}} \\ \leq C _ {g} \cdot \lambda_ {\max} (\Lambda_ {h, t} ^ {- 1}) \\ = C _ {g} \cdot \lambda_ {\min } ^ {- 1} (\Lambda_ {h, t}) \\ \leq \frac {1}{2} \zeta . \\ \end{array}
$$
Therefore, we have the desired lemma.
E.4 Proof of Theorem 5.6
Proof. By definition, the GLM-MASE chooses actions satisfying
$$
f \left(\left\langle \widetilde {\phi} _ {h} ^ {(\tau)}, \widehat {\theta} _ {h, t} ^ {g} \right\rangle\right) + \Gamma \left(s _ {h} ^ {(t)}, a _ {h} ^ {(t)}\right) \leq b _ {h}. \tag {17}
$$
By Lemma 5.4, the actions that are conservatively chosen based on (17) also satisfy the actual safety constraint, with a probability at least  $1 - \delta$ ; that is,
$$
g \left(s _ {h} ^ {(t)}, a _ {h} ^ {(t)}\right) \leq b _ {h}. \tag {18}
$$
In addition, when there is no action satisfying (17), the emergency stop action is executed where no unsafe action will be executed. Hence, the desired theorem is now obtained.
E.5 Proof of Theorem 5.7
By Assumption 3.2, the optimal policy  $\pi^{\star}$  satisfies
$$
g \left(s _ {h}, \pi^ {\star} \left(s _ {h}\right)\right) \leq b _ {h} - \zeta , \quad \forall h \in [ 1, H ]. \tag {19}
$$
Thus, the set of state-action pairs that are potentially visited by  $\pi^{\star}$  are written as
$$
\{(s, a) \in \mathcal {S} \times \mathcal {A} \mid g (s, a) \leq b _ {h} - \zeta \},
$$
which satisfies the following chain of inequalities:
$$
\begin{array}{l} \{(s, a) \mid g (s, a) \leq b _ {h} - \zeta \} \subseteq \{(s, a) \mid \mu (s, a) - \Gamma (s, a) \leq b _ {h} - \zeta \} \\ \subseteq \left\{\left(s, a\right) \mid \mu (s, a) + \Gamma (s, a) \leq b _ {h} \right\}. \\ \end{array}
$$
The state and action spaces in the last line represent the set of state-action pairs that may be visited by the policy obtained by the MASE algorithm. We used Lemma 5.4 in the first line and  $\Gamma(s, a) < \frac{\zeta}{2}$  in the second line.
By Lemma 8 in Strehl and Littman [34], the total regret can be decomposed into two parts as follows:
$$
\sum_ {t = t ^ {\star}} ^ {T} \left[ V _ {r} ^ {\pi^ {*}} - V _ {r} ^ {\pi_ {t}} \right] = \mathcal {R} (T) + \sum_ {t = t ^ {\star}} V _ {\max } \delta , \tag {20}
$$
where the first term is the regret under the condition that the confidence bound based on  $\delta$ -uncertainty quantifier successfully contains the true safety function. Also, the second term is the regret under the opposite condition, which occurs with a probability  $\delta$ .
The first term in (20) is upper-bounded based on Wang et al. [44] as follows:
$$
\begin{array}{l} \mathcal {R} (T) \leq O \left(H \sqrt {(T - t ^ {\star}) \ln ((T - t ^ {\star}) H)} \right. \\ + H \bar {\kappa} \underline {{\kappa}} ^ {- 1} \sqrt {M + \bar {\kappa} + d ^ {2} \ln \left(\frac {\bar {\kappa} + \alpha_ {\max}}{(T - t ^ {\star}) H}\right) \cdot (T - t ^ {\star}) d \ln \left(1 + \frac {(T - t ^ {\star})}{d}\right)} \\ \leq \widetilde {O} (H \sqrt {d ^ {3} (T - t ^ {\star})}). \\ \end{array}
$$
As for the second term in (20), set  $\delta = \frac{1}{TH}$  and then we have
$$
\begin{array}{l} \sum_ {t = t ^ {*}} ^ {T} V _ {\max } \cdot \delta = \sum_ {t = t ^ {*}} ^ {T} V _ {\max } \cdot \frac {1}{T H} \\ = \sum_ {t = t ^ {\star}} ^ {T} \frac {1 - \gamma^ {H}}{1 - \gamma} \cdot \frac {1}{T H} \\ \leq \sum_ {t = t ^ {*}} ^ {T} H \cdot \frac {1}{T H} \\ \leq O (1). \\ \end{array}
$$
In summary, the regret can be upper bounded by  $\widetilde{O}(H\sqrt{d^3(T - t^\star)})$ . We now obtain the desired theorem.
F Proof of Theorem 6.1
Lemma F.1 ( $\delta$ -uncertainty quantifier). Assume  $\| g \|_k^2 \leq B$  and  $N_n \leq \omega$  for all  $n \geq 1$ . Set
$$
\Gamma (s, a) := \beta_ {n} \cdot \sigma_ {n} (s, a), \quad \forall (s, a) \in \mathcal {S} \times \mathcal {A} \tag {21}
$$
with  $\beta_n^{1/2} \coloneqq B + 4\omega \sqrt{\nu_n + 1 + \ln \delta^{-1}}$ , where  $\nu_n$  is the information capacity associated with kernel  $k$ . Then,  $\Gamma$  is a  $\delta$ -uncertainty quantifier.
Proof. Recall the assumption that  $\| g\| _k^2\leq B$  and  $N_{n}\leq \omega ,\forall n\geq 1$  . Also, set
$$
\beta_ {n} ^ {1 / 2} := B + 4 \omega \sqrt {\nu_ {n} + 1 + \ln \delta^ {- 1}}.
$$
By Theorem 2 in Chowdhury and Gopalan [12], we have
$$
\left| g (s, a) - \mu_ {n} (s, a) \right| \leq \beta_ {n} \cdot \sigma_ {n} (s, a), \quad \forall (s, a) \in \mathcal {S} \times \mathcal {A}
$$
for all  $n\geq 1$  , with probability at least  $1 - \delta$  . Now, define
$$
\Gamma (s, a) := \beta_ {n} \cdot \sigma_ {n} (s, a), \quad \forall (s, a) \in \mathcal {S} \times \mathcal {A} \tag {22}
$$
then we interpret that  $\Gamma : S \times \mathcal{A} \to \mathbb{R}$  is a  $\delta$ -uncertainty quantifier based on GP.
F.1 Proof of Theorem 6.1
Proof. Based on Lemma F.1, when there is at least one safe action (i.e.,  $\mathcal{A}_h^+ \neq \emptyset$ ), the satisfaction of the safety constraint is guaranteed based on the  $\delta$ -uncertainty quantifier with a probability at least  $1 - \delta$ . Also, if there is no safe action (i.e.,  $\mathcal{A}_h^+ = \emptyset$ ), the emergency stop action is executed. In both cases, MASE guarantees the satisfaction of the safety constraint with probability at least  $1 - \delta$ .
Table 1: Hyper-parameters for Safety Gym experiments.
NAME	VALUE
COMMON PARAMETERS	NETWORK ARCHITECTURE	[64, 64]
ACTIVATION FUNCTION	tanh
LEARNING RATE (CRITIC)	5 × 10-3
LEARNING RATE (POLICY)	3 × 10-4
LEARNING RATE (PENALTY)	3 × 10-2
DISCOUNT FACTOR (REWARD)	0.99
DISCOUNT FACTOR (SAFETY)	0.99
STEPS PER EPOCH	10,000
NUMBER OF GRADIENT STEPS	80
NUMBER OF EPOCHS	500
TARGET KL	0.01
TRPO & CPO	DAMPING COEFFICIENT	0.1
BACKTRACK COEFFICIENT	0.8
BACKTRACK ITERATIONS	10
LEARNING MARGIN	FALSE
MASE	PENALTY FOR EMERGENCY STOP ACTIONS	-1
DEEP GP NETWORK ARCHITECTURE	[16, 16]
NUMBER OF INDUCING POINTS	128
KERNEL FUNCTION	RADIAL BASIS FUNCTION
G Details of Safety-Gym Experiment
We present the details regarding our experiments using Safety-Gym. Our experimental setting is based on Sootla et al. [32], which is slightly different from the original Safety Gym in that the obstacles (i.e., unsafe region) are replaced deterministically so that the environment is solvable and there is a viable solution. In this experiment, we used a machine with Intel(R) Xeon(R) Silver 4210 CPU, 128GB RAM, and NVIDIA A100 GPU. For a fair comparison, we basically used the same hyper-parameter as in Sootla et al. [32], which is summarized in Table 1.
In our experiment, when the agent identified that there was no safe action based on the GP-based uncertainty quantifier, we simply terminated the current episode (i.e., resetting) immediately after the emergency stop action and started the new episode. The frequency of the emergency stop actions is shown in Table 2.
Table 2: Frequency of the emergency stop actions.
TASK	TOTAL	LAST 100 EPISODES
POINTGOAL1	154/500	24/100
CARGOAL1	397/500	46/100
The emergency stop action is a variant of so-called resetting actions that are common in episodic RL settings, which prevent the agent from exploring the state-action spaces since the uncertainty quantifier is sometimes quite conservative. We consider that this is the reason why the reward performance of our MASE is worse than other methods (e.g., TRPO-Lagrangian, CPO). However, because we require the agent to solve more difficult problems where safety is guaranteed at every time step and episode, we consider that this result is reasonable to some extent. Though it is better for an algorithm for such a severe safety constraint to have a comparable performance as CPO, we will leave it for future work.
We also conducted an experiment to compare the performance of MASE with the early-terminated MDP (ET-MDP, [37]) algorithm. The ET-MDP is an algorithm to execute emergency stop actions immediately after safety constraints are violated. Figure 4 shows the experimental results. The ET-MDP and MASE exhibit similar learning curves on the average episode reward and average episode safety. However, while ET-MDP violated the safety constraint in most episodes (i.e., almost
(f) Maximum episode safety.
Figure 4: Experimental results on Safety Gym (Top: PointGoal1, Bottom: CarGoal1) with an additional implementation of the Early-terminated MDP (ET-MDP) algorithm (Sun et al. [37]).
all episodes are terminated after an unsafe action is executed), MASE did not violate any safety constraint.
H Grid-world Experiment