SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Advantages of MASE. Though certain existing algorithms for Problem 3 (i.e., the closest problem to the GSE problem) theoretically guarantee safety during learning, several strong assumptions are needed, such as a known and deterministic state transition and regular safety function as in [41] and a known feature mapping function that is linear with respect to transition kernels, reward, and safety as in [5]. Such algorithms have little affinity with deep RL; thus, their actual performance in complex environments tends to be poor. In contrast, MASE is compatible with any advanced RL algorithms, which can also handle various constraint formulations while maintaining the safety guarantee.
Validity of MASE. We conclude this section by presenting the following two theorems to show that our MASE produces reasonable operations in solving the GSE problem.
Theorem 4.1. Under Assumption 3.4, MASE guarantees safety with a probability of at least  $1 - \delta$ .
Theorem 4.2. Assume that the safety cost function is estimated for any state-action pairs with an accuracy of better than  $\frac{\zeta}{2}$ ; that is,  $\Gamma(s,a) \leq \frac{\zeta}{2}$  for all  $(s,a)$ . Set  $c \in \mathbb{R}$  to be a sufficiently large scalar such that  $c > \frac{\zeta V_{\max}}{2\gamma H}$ . Then, the optimal policy in  $\widehat{\mathcal{M}}$  is identical to that in  $\mathcal{M}$ .
See Appendix D for the proofs. Unfortunately, obtaining a  $\delta$ -uncertainty qualifier that works in general cases is highly challenging. To develop a feasible model for the uncertainty quantification, we assume that the safety cost can be modeled via a GLM in Section 5 and via a GP in Section 6.
5 A Provable Algorithm under Generalized Linear MCDP Assumptions
In this section, we focus on CMDPs with generalized linear structures and analyze the theoretical properties of MASE. Specifically, we provide a provable algorithm to use a class of GLMs denoted as  $\mathcal{F}$  for modeling  $Q_{r,h}^{\star} \coloneqq Q_{r,h}^{\pi^{\star}}$  and  $g$ , and then provide theoretical results on safety and optimality.
5.1 Generalized Linear CMDPs
We extend the assumption in Wang et al. [44] from unconstrained MDPs to CMDPs settings. Our assumption is based on GLMs as with [44] that makes a strictly weaker assumption than their Linear MDP assumption [19, 46]. As preliminaries, we first list the necessary definitions and assumptions.
Definition 5.1 (GLMs). Let  $d \in \mathbb{Z}_{>0}$  be a feature dimension and let  $\mathbb{B}^d \coloneqq \{\mathbf{x} \in \mathbb{R}^d : \| \mathbf{x} \|_2 \leq 1\}$  be the  $l_2$  ball in  $\mathbb{R}^d$ . For a known feature mapping function  $\phi: S \times \mathcal{A} \to \mathbb{B}^d$  and a known link function  $f: [-1,1] \to [-1,1]$ , the class of generalized linear model is denoted as  $\mathcal{F} \coloneqq \{(s,a) \to f(\langle \phi_{s,a}, \theta \rangle): \theta \in \mathbb{B}^d\}$  where  $\phi_{s,a} \coloneqq \phi(s,a)$ .
Assumption 5.2 (Regular link function). The link function  $f(\cdot)$  is twice differentiable and is either monotonically increasing or decreasing. Furthermore, there exist absolute constants  $0 < \underline{\kappa} < \overline{\kappa} < \infty$  and  $M < \infty$  such that  $\underline{\kappa} < |f'(\mathbf{x})| < \overline{\kappa}$  and  $|f''(\mathbf{x})| < M$  for all  $|\mathbf{x}| \leq 1$ .
This assumption on the regular link function is standard in previous studies (e.g., [16], [21]). Linear and logistic models are the special cases of the GLM where the link functions are defined as  $f(\mathbf{x}) = \mathbf{x}$  and  $f(\mathbf{x}) = 1 / (1 + e^{-\mathbf{x}})$ . In both cases, the link functions satisfy Assumption 5.2.
We finally make the assumption of generalized linear MDPs (GL-CMDPs), which extends the notion of the optimistic closure for unconstrained MDP settings in Wang et al. [44].
Assumption 5.3 (GL-CMDP). For any  $1 \leq h < H$  and  $u \in \mathcal{F}_{\mathrm{up}}$ , we have  $\mathcal{T}_h(u) \in \mathcal{F}$  and  $g \in \mathcal{F}$ .
Recall that  $\mathcal{T}_h$  is the Bellman update operator. In Assumption 5.3, with a positive semi-definite matrix  $A \in \mathbb{R}^{d \times d} \succ 0$  and a fixed positive constant  $\alpha_{\max} \in \mathbb{R}_{>0}$ , we define
$$
\mathcal {F} _ {\mathrm {u p}} := \left\{(s, a) \to \min \{V _ {\max}, f (\langle \phi_ {s, a}, \theta \rangle) + \alpha \| \phi_ {s, a} \| _ {A} \}: \theta \in \mathbb {B} ^ {d}, 0 \leq \alpha \leq \alpha_ {\max}, \| A \| _ {\mathrm {o p}} \leq 1 \right\},
$$
where  $\| \mathbf{x}\| _A\coloneqq \sqrt{\mathbf{x}^\top A\mathbf{x}}$  is the matrix Mahalanobis seminorm, and  $\| A\|_{\mathrm{op}}$  is the matrix operator norm. For simplicity, we suppose the same link functions for the Q-function and the safety cost function, but it is acceptable to use different link functions. Note that Assumption 5.3 is a more general assumption than Amani et al. [5] that assumes linear transition kernel, reward, and safety cost functions or Wachi et al. [43] that assumes a known transition and GLMs in terms of reward and safety cost functions.
5.2 GLM-MASE Algorithm
We introduce an algorithm GLM-MASE under Assumptions 5.2 and 5.3. Hereinafter, we explicitly denote the episode for each variable. For example, we let  $s_h^{(t)}$  or  $a_h^{(t)}$  denote a state or action at the time step  $h$  of episode  $t$ . We also let  $\widetilde{\phi}_h^{(t)} \coloneqq \phi(s_h^{(t)}, a_h^{(t)})$  for more concise notations.
Uncertainty quantifiers. To actualize MASE in the generalized linear CMDP settings, we first need to consider how to obtain the  $\delta$ -uncertainty quantifier in terms of the safety cost function. Since we assume  $g \in \mathcal{F}$ , we can define the  $\delta$ -uncertainty quantifier based on the existing studies on GLMs, especially in the field of multi-armed bandit [22, 16]. Based on Assumptions 5.2 and 5.3, we now provide a lemma regarding the  $\delta$ -uncertainty quantifier on safety.
Lemma 5.4. Suppose Assumptions 5.2 and 5.3 hold. Set  $\delta = \frac{1}{TH}$ . With a universal constant  $C \in \mathbb{R}_{>0}$ , let  $C_g := C\overline{\kappa} \underline{\kappa}^{-1} \sqrt{1 + M + \overline{\kappa} + d^2 \ln \left(\frac{1 + \overline{\kappa} + \alpha_{\max}}{\delta}\right)}$ . Define
$$
\Gamma (s, a) := C _ {g} \cdot \| \phi_ {s, a} \| _ {\Lambda_ {h, t} ^ {- 1}} \quad w i t h \quad \Lambda_ {h, t} := \sum_ {\tau \leq t} \widetilde {\phi} _ {h} ^ {(\tau)} \widetilde {\phi} _ {h} ^ {(\tau)} + I,
$$
where  $I$  is the identity matrix. Let  $\widehat{\theta}_{h,t}^{g}\in \mathbb{R}^{d}$  be the ridge estimate, which is computed by  $\widehat{\theta}_{h,t}^{g}:= \arg \min_{\| \theta \| _2\leq 1}\sum_{\tau \leq t}\left(g(s_h^{(\tau)},a_h^{(\tau)}) - f(\langle \widetilde{\phi}_h^{(\tau)},\theta \rangle)\right)^2$ . Then, the following inequality holds
$$
| g (s _ {h} ^ {(t)}, a _ {h} ^ {(t)}) - f (\langle \widetilde {\phi} _ {h} ^ {(t)}, \widehat {\theta} _ {h, t} ^ {g} \rangle) | \leq \Gamma (s _ {h} ^ {(t)}, a _ {h} ^ {(t)})
$$
for all  $(s,a)\in S\times \mathcal{A}$ , with a probability at least  $1 - \delta$ .
For the purpose of qualifying uncertainty in GLMs, the weighted  $l_{2}$ -norm of  $\phi$  (i.e.,  $\| \phi_{s,a} \|_{\Lambda_{h,t}^{-1}}$ ) plays an important role. Because we assume that the Q-function and safety cost function share the same feature, we have a similar lemma on the uncertainty quantifier regarding the Q-function as follows:
Lemma 5.5. Suppose Assumptions 5.2 and 5.3 hold. Let  $\widehat{\theta}_{h,t}^{Q}\in \mathbb{R}^{d}$  denote the ridge estimate; that is,  $\widehat{\theta}_{h,t}^{Q}\coloneqq \arg \min_{\| \theta \| _2\leq 1}\sum_{\tau \leq t}\left(y_h^{(\tau)} - f(\langle \widetilde{\phi}_h^{(\tau)},\theta \rangle)\right)^2$  where  $y_{h}^{(\tau)}\coloneqq r(s_{h}^{(\tau)},a_{h}^{(\tau)})+$ $\max_{a'\in \mathcal{A}}\widehat{Q}_{r,h + 1}^{(\tau)}(s_{h + 1}^{(\tau)},a')$  for all  $\tau \leq t$  with
$$
\widehat {Q} _ {r, h} ^ {(t)} (s, a) := \min \left\{V _ {\max}, f (\langle \phi_ {s, a}, \hat {\theta} _ {h, t} ^ {Q} \rangle) + C _ {Q / g} \Gamma (s, a) \right\}
$$
that is initialized with  $\widehat{Q}_{r,h}^{(0)} = 0$  for all  $h\leq H$  and  $\widehat{Q}_{r,H + 1}^{(t)} = 0$  for all  $1\leq t\leq T$ . Then, with a universal constant  $C_{Q / g}\in \mathbb{R}_{>0}$ , the following inequalities hold
$$
| Q _ {r, h} ^ {\star} (s _ {h} ^ {(t)}, a _ {h} ^ {(t)}) - f (\langle \widetilde {\phi} _ {h} ^ {(t)}, \widehat {\theta} _ {h, t} ^ {Q} \rangle) | \leq C _ {Q / g} \cdot \Gamma (s _ {h} ^ {(t)}, a _ {h} ^ {(t)})
$$
for all  $(s,a)\in \mathcal{S}\times \mathcal{A}$ , with a probability at least  $1 - \delta$ .
Note that  $\Gamma(s_h^{(t)}, a_h^{(t)})$  is the  $\delta$ -uncertainty quantifier with respect to the safety cost function. One of the biggest advantages of the generalized linear CMDPs is that the magnitude of uncertainty for the Q-function is proportional to that for the safety cost function. Hence, by exploring the Q-function
based on the optimism in the face of the uncertainty principle [7, 35], the safety cost function is also explored simultaneously, which contributes to the efficient exploration of state-action spaces.
Integration into MASE. The GLM-MASE is an algorithm to integrate the  $\delta$ -uncertainty quantifiers inferred by the GLM into the MASE sequence. Detailed pseudo code is presented in Appendix E.
To deal with the safety constraint, GLM-MASE leverages the upper bound inferred by the GLM; that is, for all  $h$  and  $t$ , the agent takes only actions that satisfy
$$
f \left(\langle \widetilde {\phi} _ {h} ^ {(t)}, \widehat {\theta} _ {h, t} ^ {g} \rangle\right) + \Gamma \left(s _ {h} ^ {(t)}, a _ {h} ^ {(t)}\right) \leq b _ {h}.
$$
By Lemma 5.4, such state-action pairs satisfy the safety constraint, i.e.  $g(s_h^{(t)}, s_h^{(t)}) \leq b_h$ , for all  $h$  and  $t$ , with a probability at least  $1 - \delta$ . If there is no action satisfying the safety constraint (i.e.,  $\mathcal{A}_h^+ = \emptyset$ ), the emergency stop action  $\widehat{a}$  is taken, and then the agent receives a penalty defined in (1).
As for policy optimization, we follow the optimism in the face of the uncertainty principle. Specifically, the policy  $\pi$  is optimized so that the upper-confidence bound of the Q-function characterized by  $\widehat{r}$  is maximized; that is, for any state  $s\in S$ , the policy is computed as follows:
$$
\pi_{h}^{(t)}(s) = \operatorname *{arg  max}_{a\in \mathcal{A}}\widehat{Q}_{\hat{r},h}^{(t)}(s,a).
$$
Intuitively, this equation enables us to 1) solve the exploration and exploitation dilemma by incorporating the optimistic estimates of the Q-function and 2) make the agent avoid generating trajectories to violate the safety constraint via the modified reward function.
Theoretical results. We now provide two theorems regarding safety and near-optimality. For both theorems, see Appendix E for the proofs.
Theorem 5.6. Suppose the assumptions in Lemma 5.4 hold. Then, the GLM-MASE satisfies  $g(s_h^{(t)},a_h^{(t)})\leq b_h$  for all  $t\in [1,T]$  and  $h\in [1,H]$ , with a probability at least  $1 - \delta$ .
Theorem 5.7. Suppose the assumptions in Lemmas 5.4 and 5.5 hold. Let  $C_1$  and  $C_2$  be positive, universal constants. Also, with a sufficiently large  $T$ , let  $t^\star$  denote the smallest integer satisfying  $\lambda_{\min}(\Sigma)tH - C_1\sqrt{tHd} - C_2\sqrt{tH\ln\delta^{-1}} \geq 2C_g \cdot \zeta^{-1}$ , where  $\lambda_{\min}(\Sigma)$  is the minimum eigenvalue of the second moment matrix  $\Sigma$ . Then, the policy  $\pi^{(t)}$  obtained by GLM-MASE at episode  $t$  satisfies
$$
\sum_ {t = t ^ {\star}} ^ {T} \left[ V _ {r} ^ {\pi^ {\star}} - V _ {r} ^ {\pi^ {(t)}} \right] \leq \widetilde {O} \left(H \sqrt {d ^ {3} (T - t ^ {*})}\right)
$$
Theorem 5.6 shows that the GLM-MASE guarantees safety with high probability for every time step and episode, which is a variant of Theorem 4.1 under the generalized linear CMDP assumption and corresponding  $\delta$ -uncertainty quantifier. Theorem 5.7 demonstrates the agent's ability to act near-optimally after a sufficiently large number of episodes. The proof is based on the following idea. After  $t^{\star}$  episodes, the safety cost function and the Q-function are estimated with an accuracy better than  $\frac{\zeta}{2}$ . Then, based on Theorem 4.2, the optimal policy in  $\widehat{\mathcal{M}}$  is identical to that in  $\mathcal{M}$ ; thus, the agent achieves a near-optimal policy by leveraging the (well-estimated) optimistic Q-function.
6 A Practical Algorithm
Though we established an algorithm backed by theory under the generalized linear CMDP assumption in Section 5, it is often challenging to obtain proper feature mapping functions in complicated environments. Thus, in this section, we propose a more practical algorithm combining a GP-based estimator to guarantee safety with unconstrained deep RL algorithms to maximize the reward.
Guaranteeing safety via GPs. As shown in the previous sections, the  $\delta$ -uncertainty quantifier plays a critical role in MASE. To qualify the uncertainty in terms of the safety cost function  $g$ , we consider modeling it as a GP:  $g(z) \sim \mathcal{GP}(\mu(z), k(z, z'))$ , where  $z \coloneqq [s, a]$ ,  $\mu(z)$  is a mean function, and  $k(z, z')$  is a covariance function. The posterior distribution over  $g(\cdot, \cdot)$  is computed based on  $n \in \mathbb{Z}_{>0}$  observations at state-action pairs  $(z_1, z_2, \ldots, z_n)$  with safety measurements  $\mathbf{y}_n \coloneqq \{y_1, y_2, \ldots, y_n\}$ , where  $y_n \coloneqq g(z_n) + N_n$  and  $N_n \sim \mathcal{N}(0, \omega^2)$  is zero-mean Gaussian noise
with a standard deviation of  $\omega \in \mathbb{R}_{\geq 0}$ . We consider episodic RL problems, and so  $n \approx tH + h$  for episode  $t$  and time step  $h$ , although the equality does not hold because of the episode cutoffs. Using the past measurements, the posterior mean, variance, and covariance are computed analytically as  $\mu_n(z) = k_n^\top(z)(K_n + \omega^2 I)^{-1}y_n$ ,  $\sigma_n(z) = k_n(z,z)$ , and  $k_n(z,z') = k(z,z') - k_n^\top(z)(K_n + \omega^2 I)^{-1}k_n(z')$ , where  $k_n(z) = [k(z_1,z),\dots,k(z_n,z)]^\top$  and  $K_n$  is the positive definite kernel matrix. We now present a theorem on the safety guarantee.
Theorem 6.1. Assume  $\| g\| _k^2\leq B$  and  $N_{n}\leq \omega$  for all  $n\geq 1$ . Set  $\beta_n^{1 / 2}\coloneqq B + 4\omega \sqrt{\nu_n + 1 + \ln(1 / \delta)}$  and construct the  $\delta$ -uncertainty quantifier by
$$
\Gamma (s, a) := \beta_ {n} \cdot \sigma_ {n} (s, a), \quad \forall (s, a) \in \mathcal {S} \times \mathcal {A}, \tag {2}
$$
where  $\nu_{n}$  is the information capacity associated with kernel  $k$ . Then, MASE based on (2) satisfies the safety constraint  $g(s_h^{(t)},a_h^{(t)})\leq b_h$  for all  $t$  and  $h$  with a probability of at least  $1 - \delta$ .
See Appendix F for the proofs. Theorem 6.1 guarantees that the safety constraint is satisfied by combining the GP-based  $\delta$ -uncertainty quantifier in (2) and the emergency stop action.
Maximizing reward via deep RL. The remaining task is to optimize the policy via the modified reward function  $\hat{r}$  in (1), whereby the agent is penalized for emergency stop actions. This problem is decoupled from the safety constraint and can be solved as the following unconstrained RL problem:
$$
\pi := \underset {\pi} {\arg \max } V _ {\hat {r}} ^ {\pi}. \tag {3}
$$
There are many excellent algorithms for solving (3) such as trust region policy optimization (TRPO, [31]) and twin delayed deep deterministic policy gradient (TD3, [17]). One of the key benefits of our MASE is such compatibility with a broad range of unconstrained (deep) RL algorithms.
We conduct two experiments. The first is on Safety Gym [28], where an agent must maximize the expected cumulative reward under a safety constraint with additive structures as in Problems 1 and 2. The safety cost function  $g$  is binary (i.e., 1 for an unsafe state-action pair and 0 otherwise), and the safety threshold is set to  $\xi_1 = 20$ . The reason for choosing Safety Gym is that this benchmark is complex and elaborate, and has been used to evaluate a variety of excellent algorithms. The second is a grid world where a safety constraint is instantaneous as in Problem 3. Due to the page limit, we present the settings and results of the grid-world experiment in Appendix H.
To solve the Safety Gym tasks, we implement the practical algorithm presented in Section 6 as follows. First, we convert the problem into a GSE problem by defining  $b_{h} \coloneqq \gamma_{g}^{-1} \cdot (\xi_{1} - \sum_{h' = 0}^{h - 1} \gamma_{g}^{h'} g(s_{h'}, a_{h'})))$  and enforcing the safety constraint represented as  $g(s_{h}, a_{h}) \leq b_{h}$  for every time step  $h$  in each episode. Second, to infer the safety cost, we use deep GP to conduct training and inference, as in [30] when dealing with high-dimensional input spaces. Finally, as for the policy optimization in  $\widehat{\mathcal{M}}$ , we leverage the TRPO algorithm. We delegate other details to Appendix G.
Baselines and metrics. We use the following four algorithms as baselines. The first is TRPO, which is a safety-agnostic deep RL algorithm that purely optimizes a policy without safety consideration. The second and third are CPO [1] and TRPO-Lagrangian [28], which are well-known algorithms for solving CMDPs. The final algorithm is Sauté RL [32], which is a recent, state-of-the-art algorithm for solving safe RL problems where constraints must be satisfied almost surely. We employ the following three metrics to evaluate our MASE and the aforementioned four baselines: 1) the expected cumulative reward, 2) the expected cumulative safety, and 3) the maximum cumulative safety. We execute each algorithm with five random seeds and compute the means and confidence intervals.
Results. The experimental results are summarized in Figure 2. The figures show that TRPO, TRPO-Lagrangian, and CPO successfully learn the policies, but violate the safety constraints during training and even after convergence. Sauté RL is much safer than those three algorithms, but the safety constraint is not satisfied in some episodes, and the performance of the policy significantly deteriorates in terms of the cumulative reward during training. Our MASE obtains better policies in a smaller number of samples compared with Sauté RL, while also satisfying the safety constraints with respect to both the average and the worst-case. Note that, after convergence, the policy obtained by MASE performs worse than those obtained by the baseline algorithms in terms of reward, as shown in
(a) Average episode return.
(b) Average episode safety.
(c) Maximum episode safety.
(d) Average episode return.
(e) Average episode safety.
(f) Maximum episode safety.
Figure 2: Experimental results on Safety Gym (Top: PointGoal1, Bottom: CarGoal1). The proposed MASE satisfies the safety constraint in every episode and achieves better performance in terms of the reward than the state-of-the-art method called Sauté RL. Conventional methods (i.e., TRPO, TRPO-Lagrangian, and CPO) repeatedly violate the safety constraint, especially in the early phase of training. Shaded areas represent  $1\sigma$  confidence intervals across five different random seeds.
(a) Average episode return.
Figure 3: Experimental results on Safety Gym (CarGoal1) with the epoch of 1000. The box plots show the converged performance. Though MASE performs worse than other baselines in terms of reward, the acquired policy is still near-optimal. As for safety, while baselines violate the safety constraint in most of the episodes, MASE guarantees the satisfaction of the severe safety constraint.
Figure 3. The emergency stop action is a variant of resetting actions that are common in episodic RL settings, which prevent the agent from exploring the state-action spaces since the uncertainty quantifier is sometimes quite conservative. We consider that this is a reason why the converged reward performance of MASE is worse than other methods. However, because we require the agent to solve difficult problems where safety is guaranteed at every time step and episode, we consider that this result is reasonable, and further performance improvements are left to future work.
In this article, we first introduced the GSE problem and proved that it is more general than three common safe RL problems. We then proposed MASE to optimize a policy under safety constraints that allow the agent to execute an emergency stop action at the sacrifice of a penalty based on the  $\delta$ -uncertainty qualifier. As a specific instance of MASE, we first presented GLM-MASE to theoretically guarantee the near-optimality and safety of the acquired policy under generalized linear CMDP assumptions. Finally, we provided a practical MASE and empirically evaluated its performance in comparison with several baselines on the Safety Gym and grid-world.
We would like to thank the anonymous reviewers for their helpful comments. This work is partially supported by JST CREST JPMJCR201 and by JSPS KAKENHI Grant 21K14184.
[1] Achiam, J., Held, D., Tamar, A., and Abbeel, P. (2017). Constrained policy optimization. In International Conference on Machine Learning (ICML).
[2] Alshiekh, M., Bloem, R., Ehlers, R., Konighofer, B., Niekum, S., and Topcu, U. (2018). Safe reinforcement learning via shielding. In AAAI Conference on Artificial Intelligence (AAAI).
[3] Altman, E. (1999). Constrained Markov decision processes, volume 7. CRC Press.
[4] Amani, S., Alizadeh, M., and Thrampoulidis, C. (2019). Linear stochastic bandits under safety constraints. In Neural Information Processing Systems (NeurIPS).
[5] Amani, S., Thrampoulidis, C., and Yang, L. (2021). Safe reinforcement learning with linear function approximation. In International Conference on Machine Learning (ICML).
[6] Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. (2016). Concrete problems in AI safety. arXiv preprint arXiv:1606.06565.
[7] Auer, P. and Ortner, R. (2007). Logarithmic online regret bounds for undiscounted reinforcement learning. In Neural Information Processing Systems (NeurIPS).
[8] Bajracharya, M., Maimone, M. W., and Helmick, D. (2008). Autonomy for mars rovers: Past, present, and future. Computer, 41(12):44-50.
[9] Brunke, L., Greeff, M., Hall, A. W., Yuan, Z., Zhou, S., Panerati, J., and Schoellig, A. P. (2022). Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5:411-444.
[10] Carr, S., Jansen, N., Junges, S., and Topcu, U. (2023). Safe reinforcement learning via shielding under partial observability. In AAAI Conference on Artificial Intelligence.
[11] Chow, Y., Ghavamzadeh, M., Janson, L., and Pavone, M. (2017). Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research (JMLR), 18(1):6070-6120.
[12] Chowdhury, S. R. and Gopalan, A. (2017). On kernelized multi-armed bandits. In International Conference on Machine Learning (ICML).
[13] Ding, D., Zhang, K., Basar, T., and Jovanovic, M. (2020). Natural policy gradient primal-dual method for constrained Markov decision processes. In Neural Information Processing Systems (NeurIPS).
[14] Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., and Hester, T. (2021). Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning.
[15] Eysenbach, B., Gu, S., Ibarz, J., and Levine, S. (2018). Leave no trace: Learning to reset for safe and autonomous reinforcement learning. In International Conference on Learning Representations (ICLR).
[16] Filippi, S., Cappe, O., Garivier, A., and Szepesvári, C. (2010). Parametric bandits: The generalized linear case. In Neural Information Processing Systems (NeurIPS).
[17] Fujimoto, S., van Hoof, H., and Meger, D. (2018). Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning (ICML).
[18] Garcia, J. and Fernández, F. (2015). A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research (JMLR), 16(1):1437-1480.