ActSafe: Active Exploration with Safety Constraints for Reinforcement Learning

## Abstract

Abstract Reinforcement learning (RL) is ubiquitous in the development of modern AI systems. However, state-of-the-art RL agents require extensive, and potentially unsafe, interactions with their environments to learn effectively.
These limitations confine RL agents to simulated environments, hindering their ability to learn directly in real-world settings. In this work, we present ActSafe , a novel model-based RL algorithm for safe and efficient exploration. ActSafe learns a well-calibrated probabilistic model of the system and plans optimistically w.r.t. the epistemic uncertainty about the unknown dynamics, while enforcing pessimism w.r.t. the safety constraints. Under regularity assumptions on the constraints and dynamics, we show that ActSafe guarantees safety during learning while also obtaining a near-optimal policy in finite time. In addition, we propose a practical variant of ActSafe that builds on latest model-based RL advancements and enables safe exploration even in high-dimensional settings such as visual control. We empirically show that ActSafe obtains state-of-the-art performance in difficult exploration tasks on standard safe deep RL benchmarks while ensuring safety during learning.

## 1 Introduction

Reinforcement learning (RL) is a powerful paradigm for sequential decision-making under uncertainty, with many applications in games (Mnih et al., ; Silver et al., ), recommender systems (Maystre et al., ), nuclear fusion control (Degrave et al., ), data-center cooling (Lazic et al., ) and robotics (Lee et al., ; Brohan et al., ; Cheng et al., ). Despite the notable progress, its application without any use of simulators remains largely limited. This is primarily because, in many cases, RL methods require
massive amounts
of data for learning while also being inherently unsafe during exploration.

In many real-world settings, environments are complex and rarely align exactly with the assumptions made in simulators.
Learning directly in the real world allows RL systems to close the sim-to-real gap and continuously adapt to evolving environments and distribution shifts. However, to unlock these advantages, RL algorithms must be sample-efficient and ensure safety throughout the learning process to avoid costly failures or risks in high-stakes applications. For instance, agents learning driving policies in autonomous vehicles must prevent collisions with other cars or pedestrians, even when adapting to new driving environments. This challenge is known as *safe exploration*, where the agent’s exploration is restricted by safety-critical, often unknown, *constraints that must be satisfied throughout the learning process*.

Several works study safe exploration
and have demonstrated state-of-the-art performance in terms of both safety and sample efficiency for learning in the real world (Sui et al., ; Wischnewski et al., ; Berkenkamp et al., ; Cooper & Netoff, ; Sukhija et al., ; Widmer et al., ). These methods maintain a “safe set” of policies during learning, selecting policies from this set to safely explore and gradually expand it. Under common regularity assumptions about the constraints, these approaches guarantee safety throughout learning. However, explictily maintaining and expanding a safe set, limits these methods to low-dimensional policies, such as PID controllers. This makes them difficult to scale to more complex tasks such as those considered in deep RL.

The goal of this work is to address this gap. To this end, we propose a scalable model-based RL algorithm – ActSafe – for efficient and safe exploration.
Crucially, ActSafe learns an uncertainty-aware dynamics model, which it uses to implicitly define and expand the safe set of policies.
We theoretically show that ActSafe ensures safety throughout learning and converges to a near-optimal policy within a finite number of episodes. Moreover, ActSafe is practical and integrates seamlessly with state-of-the-art dynamics modeling techniques, for instance Dreamer (Hafner et al., ), delivering strong empirical performance.
Thus, ActSafe advances the frontier of safe RL methods, both in theory and practice. Our main contributions are summarized below.

##### Contributions

- •
We propose ActSafe, a novel model-based RL algorithm for safe exploration in continuous state-action spaces.
ActSafe maintains a *pessimistic* set of safe policies and *optimistically* selects policies within this set that yield trajectories with the largest model epistemic uncertainty.
- •
We show that when the dynamics lie in a reproducing kernel Hilbert space (RKHS), ActSafe guarantees safe exploration. In addition, we provide a sample-complexity bound for ActSafe, illustrating that ActSafe obtains $\epsilon$-optimal policies in a finite number of episodes. To the best of our knowledge, we are the first to show *safety and finite sample complexity* for safe exploration in model-based RL with continuous state-action spaces.
- •
In our experiments, we demonstrate that ActSafe, when combined with a Gaussian process dynamics model, achieves efficient and safe exploration. Additionally, we show that ActSafe scales to high-dimensional environments of the Safety-Gym and RWRL benchmarks, excelling in challenging exploration tasks with visual control while being safe during learning.

## 2 Related Works

##### Constrained Markov decision processes (CMDP) for safe RL

Safety in reinforcement learning can be modeled in various ways (García et al., ; Brunke et al., ). Constrained Markov decision processes (CMDPs) serve as a natural option for this purpose, as they can encode unsafe behaviors through constraints and enjoy many classical results from planning in MDPs (Altman, ).
Learning and planning in CMDPs have been extensively explored in the RL community, both theoretically and in practice. Notably, the works of Efroni et al. (); Vaswani et al. (); Ding et al. (); Müller et al. () derive sample complexity bounds for CMDPs in discrete state-action spaces, whereas Achiam et al. (); Tessler et al. (); Stooke et al. (); Xu et al. (); Liu et al. (); As et al. (); Sootla et al. (); Huang et al. () develop deep RL algorithms for CMDPs in continuous state-action spaces. However, all the aforementioned works relax the requirement of safe exploration and thus do not ensure the safety during learning. This is in contrast to this work, where we tackle the hard problem of safe exploration.

##### Provably safe exploration

Turchetta et al. (); Wachi et al. (); Wachi & Sui () focus on safe exploration in CMDPs with *discrete* state-action spaces and a constraint function that lies in an RKHS. Zheng & Ratliff () study sample complexity for safe exploration in discrete CMDPs. For continuous state-action spaces, Berkenkamp et al. () and extensions thereof (Baumann et al., ; Sukhija et al., ; Hübotter et al., ), leverage ideas from safe Bayesian optimization (Sui et al., ) to directly optimize over the policy parameters in a model-free manner. The proposed algorithms guarantee safe exploration and finite sample complexity for learning an $\epsilon$-optimal solution. When evaluated on real-world systems,
these methods exhibit remarkable sample efficiency while also being safe during learning (Cooper & Netoff, ; Kirschner et al., ; Widmer et al., ). However, these approaches are limited to simple low-dimensional policies, e.g., PID controllers, and are hard to scale to policies with more than few parameters.
In a similar spirit, Berkenkamp et al. () propose a model-based RL algorithm for safe learning, where safety is modeled in terms of Lyapunov stability. Even though the method enjoys similar theoretical guarantees as Berkenkamp et al. (), it assumes access to a generative simulator and thus cannot be applied to traditional online RL settings. In contrast, Koller et al. (); Curi et al. () propose more practical safe learning methods in combination with model-predictive control (MPC). While these methods guarantee safety during learning, they lack optimality guarantees and are computationally expensive to run in real-time.

A common aspect among most of the aforementioned methods is their use of an intrinsic objective, such as the model epistemic uncertainty, to guide and restrain exploration. Crucially, these methods maintain a safe set of policies which they gradually expand during learning by sampling policies that yield the highest intrinsic reward. In this work, we build on this key insight to propose a model-based RL algorithm for online learning that enjoys the same kind of guarantees while also being applicable in real-world settings such as deep RL.

##### Safe exploration with deep RL

A common approach to safe exploration is the use of safety filters (Dalal et al., ; Wabersich & Zeilinger, ; Curi et al., ), which modify the actions produced by an unsafe policy to meet safety constraints before they are executed on the real system. A key advantage of safety filters is that they can be easily added to any “off-the-shelf” unsafe RL algorithm. However, while safety is ensured, safety filters can lead to arbitrarily bad exploration and therefore lack guarantees for optimality. Another line of work, relies on guaranteeing feasibility of policy optimization algorithms. Notably, Chow et al. () use Lyapunov functions to guarantee feasibility of policy gradients iterates and derive their analysis on discrete state-action spaces. Usmanova et al. () propose Log-Barriers SGD (LBSGD), an optimization algorithm that ensures feasibility of all its iterates with barrier functions, showcasing its application in navigation tasks with image observations. More recently, As et al. (); Ni & Kamgarpour () use LBSGD for safe learning with *greedy* policy gradients, i.e., without considering intrinsic rewards to expand the safe set of policies. Crucially, this form of greedy policy search may result in sub-optimal policies, as described in Section 4 and empirically shown in Section 5.

## 3 Problem Setting

We consider a discrete-time, episodic, constrained Markov decision process (CMDP), where the goal is to find a policy that not only maximizes the reward but also keeps the accumulated costs below a specified threshold, i.e., satisfies a safety constraint. This type of formulation is common in real-world scenarios, such as robot navigation. In this setting, the reward could represent the negative distance to a target destination, while the costs could represent penalties, such as a cost of 1 incurred for each collision with an obstacle. The CMDP formulation allows us to separate these two objectives, thus ensuring constraint satisfaction and safety, for an optimal policy.
In this setup, we consider dynamical systems with additive noise and bounded running rewards $r$ and costs $c$

$$ $\displaystyle\begin{aligned} {\bm{s}}_{t+1}={\bm{f}}^{*}({\bm{s}}_{t},{\bm{a}} _{t})+{\bm{w}}_{t},\ &({\bm{s}}_{t},{\bm{a}}_{t})\in{\mathcal{S}}\times{ \mathcal{A}},\ {\bm{s}}(0)={\bm{s}}_{0}\\ &r({\bm{s}},{\bm{a}})\in[0,R_{\max}]\qquad\text{(Running reward)}\\ &c({\bm{s}},{\bm{a}})\in[0,C_{\max}]\qquad\text{(Running cost)}.\end{aligned}$ (1) $$

Here ${\bm{s}}_{t}\in{\mathcal{S}}\subset\mathbb{R}^{d_{\bm{s}}}$ is the state, ${\bm{a}}_{t}\in{\mathcal{A}}\subset\mathbb{R}^{d_{\bm{a}}}$ the control input, and ${\bm{w}}_{t}\in{\mathcal{W}}\subseteq\mathbb{R}^{d_{\bm{s}}}$ the process noise. The dynamics ${\bm{f}}^{*}$ are unknown and without loss of generality, the reward $r$ and cost $c$ are assumed to be known.

##### Task

In this work, we study the following constrained RL problem (Altman, )

$$ $\begin{split}\max_{{\bm{\pi}}\in\Pi}J_{r}({\bm{\pi}},{\bm{f}}^{*})&\coloneq \max_{{\bm{\pi}}\in\Pi}\mathbb{E}_{{\bm{s}}_{0},{\bm{\pi}}}\left[\sum^{T-1}_{t =0}r({\bm{s}}_{t},{\bm{a}}_{t})\right]\ \text{s.t. }J_{c}({\bm{\pi}},{\bm{f}}^ {*})\coloneq\mathbb{E}_{{\bm{s}}_{0},{\bm{\pi}}}\left[\sum^{T-1}_{t=0}c({\bm{s }}_{t},{\bm{a}}_{t})\right]\leq d;\\ &{\bm{s}}_{t+1}={\bm{f}}^{*}({\bm{s}}_{t},{\bm{\pi}}_{t}({\bm{s}}_{t}))+{\bm{w }}_{t}.\end{split}$ (2) $$

We study the episodic setting, with episodes $n=1,\ldots,N$. At the beginning of the episode $n$, we deploy a policy ${\bm{\pi}}_{n}=({\bm{\pi}}_{n,0},{\bm{\pi}}_{n,1},\dots,{\bm{\pi}}_{n,T-1})$, chosen from the policy space $\Pi$ for a horizon of $T$ on the system. Next, we obtain the trajectory ${\bm{\tau}}_{n}=({\bm{s}}_{n,0},\ldots,{\bm{s}}_{n,T})$, which we add to a dataset of transitions ${\mathcal{D}}_{n}=\{({\bm{z}}_{n,i}=({\bm{s}}_{n,i},{\bm{\pi}}_{n,i}({\bm{s}}_
{n,i})),{\bm{y}}_{n,i}={\bm{s}}_{n,i+1})_{0\leq i<T}\}$ and use the collected data to learn a model of ${\bm{f}}^{*}$.

## 4 ActSafe : Active Exploration with Safety Constraints

A key challenge in learning with safety constraints is ensuring that these constraints are not violated during exploration. In the following, we introduce an idealized version of ActSafe, which guarantees safe exploration for dynamical systems with Gaussian process dynamics (^1^11These guarantees can be extended to more general well-calibrated models as in Curi et al. ()).
Moreover, we also provide a bound on the sample complexity of ActSafe for learning an $\epsilon$-optimal policy.
To the best of our knowledge, this is the first model-based safe RL algorithm for continuous state-action spaces that provides guarantees for both safety and sample complexity. In Section 4.3, we discuss a practical variant scaling to more complex domains.

### 4.1 Assumptions

Theoretically studying safe exploration without any assumptions on the underlying dynamical system is an ill-posed problem. In the following, we make some assumptions on the underlying problem that are common in the model-based RL (Curi et al., ; Kakade et al., ) and safe RL (Berkenkamp et al., ; Baumann et al., ) literature.

###### Assumption 4.1 (Continuity of ${\bm{f}}^{*}$ and ${\bm{\pi}}$ ) .

The dynamics model ${\bm{f}}^{*}$ is $L_{{\bm{f}}}$–Lipschitz, the cost $c$ is $L_{c}$–Lipschitz, and all ${\bm{\pi}}\in\Pi$ are continuous.

###### Assumption 4.2 (Process noise distribution) .

The process noise is i.i.d. Gaussian with variance $\sigma^{2}$, i.e., ${\bm{w}}_{t}\stackrel{{\scriptstyle\mathclap{i.i.d}}}{{\sim}}{\mathcal{N}}({
\bm{0}},\sigma^{2}{\mathbb{I}})$.

We focus on the setting where ${\bm{w}}$ is homoscedastic for simplicity. However, our framework can also be applied to the more general heteroscedastic and sub-Gaussian case (Sukhija et al., ; Hübotter et al., ).

###### Assumption 4.3 (Initial safe seed) .

We have access to an initial nonempty safe set ${\mathcal{S}}_{0}$ of policies, i.e.,
$\forall{\bm{\pi}}\in{\mathcal{S}}_{0}:J_{c}({\bm{\pi}})\leq d$ and ${\mathcal{S}}_{0}\neq\emptyset$.

This assumption is crucial since without any prior knowledge about the system, ensuring safety is unrealistic. Therefore, ${\mathcal{S}}_{0}$ allows us to start the learning process by selecting policies from this set.
In practice, this safe set could be obtained from a simulator or offline demonstration data.

In the following, we assume that at each step $n$ we learn a mean estimate ${\bm{\mu}}_{n}$ of ${\bm{f}}^{*}$ and can quantify our uncertainty ${\bm{\sigma}}_{n}$ over the estimate. This allows us to learn an uncertainty-aware model of ${\bm{f}}^{*}$, which is crucial for exploration and safety.
More formally, we learn a well-calibrated statistical model of ${\bm{f}}^{*}$ as defined below.

###### Definition 4.4 (Well-calibrated statistical model of ${\bm{f}}^{*}$ , Rothfuss et al. ( 2023 ) ) .

Let ${\mathcal{Z}}\operatorname*{\smash{\overset{\mathrm{def}}{=}}}{\mathcal{S}}
\times{\mathcal{A}}$.
An all-time well-calibrated statistical model of the function ${\bm{f}}^{*}$ is a sequence $\{{\mathcal{Q}}_{n}(\delta)\}_{n\geq 0}$, where

$$ $\displaystyle{\mathcal{Q}}_{n}(\delta)\operatorname*{\smash{\overset{\mathrm{ def}}{=}}}\left\{{\bm{f}}:{\mathcal{Z}}\to{\mathbb{R}}^{d_{s}}\mid\forall{\bm{ z}}\in{\mathcal{Z}},\forall j\in\{1,\ldots,d_{s}\}:|\mu_{n,j}({\bm{z}})-f_{j}( {\bm{z}})|\leq\beta_{n}(\delta)\sigma_{n,j}({\bm{z}})\right\},$ $$

if, with probability at least $1-\delta$, we have ${\bm{f}}^{*}\in\bigcap_{n\geq 0}{\mathcal{Q}}_{n}(\delta)$.
Here, $f_{j}$, $\mu_{n,j}$ and $\sigma_{n,j}$ denote the $j$-th element in the vector-valued functions ${\bm{f}}$, ${\bm{\mu}}_{n}$ and ${\bm{\sigma}}_{n}$ respectively, and $\beta_{n}(\delta)\in{\mathbb{R}}_{\geq 0}$ is sequence of scalar functions that depends on the confidence level $\delta\in(0,1]$ and is monotonically increasing in $n$.

Next, we assume that ${\bm{f}}^{*}$ resides in a Reproducing Kernel Hilbert Space (RKHS) of vector-valued functions and show that this is sufficient for us to obtain a well-calibrated model.

###### Assumption 4.5 .

We assume that the functions $f^{*}_{j}$, $j=\{1,\dots,d_{s}\}$ lie in a RKHS with kernel $k$ and have a bounded norm $B$, that is ${\bm{f}}^{*}\in{\mathcal{H}}^{d_{s}}_{k,B}$, with ${\mathcal{H}}^{d_{s}}_{k,B}=\{{\bm{f}}\mid\norm{f_{j}}_{k}\leq B,j=\{1,\dots,d
_{s}\}$. Moreover, we assume that $k({\bm{s}},{\bm{s}})\leq\sigma_{\max}$ for all ${\bm{s}}\in{\mathcal{S}}$.

Assumption 4.5 allows us to model ${\bm{f}}^{*}$ with GPs for which the mean and epistemic uncertainty (${\bm{\mu}}_{n}({\bm{z}})=[\mu_{n,j}({\bm{z}})]_{j\leq d_{s}}$, and ${\bm{\sigma}}_{n}({\bm{z}})=[\sigma_{n,j}({\bm{z}})]_{j\leq d_{s}}$) have an analytical formula (c.f., Equation 9 in Appendix A).

###### Lemma 4.6 (Well calibrated confidence intervals for RKHS, Rothfuss et al. ( 2023 ) ) .

Let ${\bm{f}}^{*}\in{\mathcal{H}}_{k,B}^{d_{s}}$.
Suppose ${{\bm{\mu}}}_{n}$ and ${\bm{\sigma}}_{n}$ are the posterior mean and variance of a GP with kernel $k$ after episode $n$.
There exists $\beta_{n}(\delta)$, for which the sequence $({\bm{\mu}}_{n},{\bm{\sigma}}_{n},\beta_{n}(\delta))_{n\geq 0}$ represents a well-calibrated statistical model of ${\bm{f}}^{*}$.

In summary, Assumption 4.5 and Lemma 4.6 show that in the RKHS setting, a GP is a well-calibrated model.
For more general models like Bayesian neural networks (BNNs), methods such as Kuleshov et al. () can be used for calibration. Overall, our results can also be extended beyond the RKHS setting to other classes of well-calibrated models similar to Curi et al. ().

###### Assumption 4.1 (Continuity of 𝒇 ∗ superscript 𝒇 {\bm{f}}^{*} bold_italic_f start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT and 𝝅 𝝅 {\bm{\pi}} bold_italic_π ) .

###### Assumption 4.2 (Process noise distribution) .

###### Assumption 4.3 (Initial safe seed) .

###### Definition 4.4 (Well-calibrated statistical model of 𝒇 ∗ superscript 𝒇 {\bm{f}}^{*} bold_italic_f start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT , Rothfuss et al. ( 2023 ) ) .

###### Assumption 4.5 .

###### Lemma 4.6 (Well calibrated confidence intervals for RKHS, Rothfuss et al. ( 2023 ) ) .

### 4.2 ActSafe : Algorithmic Framework

A crucial element of safe exploration algorithms is the exploration–expansion dilemma (Hübotter et al., ). In the following, we explain this in further detail, we then present a sketch of ActSafe and finally the formal algorithm.

##### Safe set expansion

To ensure the safety of the agent during the initial phases of learning, ActSafe begins exploration by selecting policies from ${\mathcal{S}}_{0}$ (Assumption 4.3). This reduces uncertainty ${\bm{\sigma}}_{n}$ about ${\bm{f}}^{*}$ for policies in ${\mathcal{S}}_{0}$, allowing us to infer the safety of policies beyond ${\mathcal{S}}_{0}$ and *expand* the safe set (see Figure 1).
Safe set expansion is critical in safe RL because the optimal policy may lie outside the initial safe set, and expanding the safe set is necessary to reach it. Unlike traditional RL, where exploration focuses on maximizing reward, safe RL methods must also explore to expand the safe set. Methods like optimism and Thompson sampling, which focus on reward maximization, do not address this need for safe set expansion (Sui et al., ).

##### Algorithm Sketch

ActSafe operates in two stages;
(i) expansion by intrinsic exploration and
(ii) exploitation of extrinsic reward.
In the first stage, ActSafe uses the model epistemic uncertainty as an intrinsic reward $r^{\text{\text{explore}}}({\bm{s}},{\bm{a}})=\norm{{\bm{\sigma}}_{n-1}({\bm{s}
},{\bm{a}})}$ and selects policies within the safe set that yield trajectories with high uncertainties. This enables ActSafe to efficiently reduce its uncertainty within the safe set and expand it. ActSafe performs the intrinsic exploration phase for a fixed number of episodes $n^{*}$ till the safe set is sufficiently large and then transitions to the second stage. In the exploitation stage, ActSafe greedily maximizes the extrinsic reward $r$, effectively aiming to solve the problem in Equation 2.

Most model-based safe RL methods (As et al., ) focus only on the second stage and ignore safe set expansion. In contrast, the theoretically grounded approaches of Berkenkamp et al. (); Baumann et al. (); Sukhija et al. (); Hübotter et al. () explicitly account for the expansion, but are not scalable to high dimensional policies typically considered in RL.

Figure: Algorithm 1 ActSafe: Active Exploration with Safety Constraints (Expansion stage)

Next, we present our main algorithm.
To ensure safety during learning, we maintain a conservative (pessimistic) estimate of the safe set which is defined below.

###### Definition 4.7 .

Let ${\mathcal{M}}_{n}\operatorname*{\smash{\overset{\mathrm{def}}{=}}}{\mathcal{M}
}_{n-1}\cap{\mathcal{Q}}_{n},\forall n\geq 1$ denote the set of plausible models, and $P_{n}({\bm{\pi}})=\max_{{\bm{f}}\in{\mathcal{M}}_{n}}J_{c}({\bm{\pi}},{\bm{f}})$ our *pessimistic* estimate of the expected costs w.r.t. ${\mathcal{M}}_{n}$. Then, we define the safe set ${\mathcal{S}}_{n}$ as

$$ ${\mathcal{S}}_{n}\operatorname*{\smash{\overset{\mathrm{def}}{=}}}{\mathcal{S} }_{n-1}\cup\left\{{\bm{\pi}}\in\Pi\setminus{\mathcal{S}}_{n-1};\exists{\bm{\pi }}^{\prime}\in{\mathcal{S}}_{n-1}\ \text{s.t.}\ P_{n}({\bm{\pi}}^{\prime})+D({ \bm{\pi}},{\bm{\pi}}^{\prime})\leq d\right\},$ (3) $$

where

$$ $\displaystyle D({\bm{\pi}},{\bm{\pi}}^{\prime})=$ $\displaystyle\mathbb{E}_{{\bm{\tau}}^{{\bm{\pi}}^{\prime}}}\left[\sum^{T-1}_{t =0}\min\left\{L_{c}\norm{{\bm{\pi}}^{\prime}({\bm{s}}_{t})-{\bm{\pi}}({\bm{s}} _{t})},2C_{\max}\right\}+TC_{\max}\min\left\{\frac{L_{f}\norm{{\bm{\pi}}^{ \prime}({\bm{s}}_{t})-{\bm{\pi}}({\bm{s}}_{t})}}{\sigma},1\right\}\right]$ $$

###### Definition 4.7 .

##### Interpretation of Definition 4.7

We maintain a pessimistic estimate, $P_{n}$ of the constraint value function $J_{c}$ w.r.t. our model set ${\mathcal{M}}_{n}$. In Equation 3 we define the expansion operator for the safe set.
This operator adds new policies ${\bm{\pi}}$ that are not yet in the safe set, i.e., those in $\Pi\setminus{\mathcal{S}}_{n-1}$, to ${\mathcal{S}}_{n}$ if they are close to some policy ${\bm{\pi}}^{\prime}$ from within the safe set.
The distance $D({\bm{\pi}},{\bm{\pi}}^{\prime})$ measures how close the two policies are in terms of the underlying cost function, and it is similar to other distance metrics, such as the one in Foster et al. (, Theorem 2.1).

Figure: Figure 1: Schematic illustration of the expansion process. We expand the safe set at iteration $n-1$ by reducing our uncertainty around policies at the boundary of ${\mathcal{S}}_{n-1}$. The pale blue area depicts the reachable set ${\mathcal{R}}_{H}^{\varepsilon}({\mathcal{S}}_{0})$ after $H$ expansion iterations. The arrow on the leftmost illustration demonstrates that without explicit expansion, finding the optimal policy $\pi^{*}$ is intractable.
Refer to caption: x1.png

The expansion operator is common in the safe BO and RL literature (Wischnewski et al., ; Fiducioso et al., ; Berkenkamp et al., ; Baumann et al., ; Cooper & Netoff, ; Sukhija et al., ; Holzapfel et al., ; Fiedler et al., ), and while it is generally difficult to evaluate in continuous spaces, it gives a key insight for safe RL methods: *to effectively expand our knowledge of what is safe, we have to reduce our pessimism across policies in our safe set*.

Accordingly, during the expansion phase, we use the following objective for ActSafe, which, in the $n$-the episode, selects the policy ${\bm{\pi}}_{n}$ that yields the high uncertainty about the underlying dynamics

$$ $\displaystyle{\bm{\pi}}_{n},{\bm{f}}_{n}$ $\displaystyle=\operatorname*{arg\,max}_{{\bm{\pi}}\in{\mathcal{S}}_{n},{\bm{f} }\in{\mathcal{M}}_{n}}\underbrace{\mathbb{E}_{{\bm{\tau}}^{{\bm{\pi}},{\bm{f}} }}\left[\sum_{t=0}^{T-1}\norm{{\bm{\sigma}}_{n-1}({\hat{\bm{s}}}_{t},{\bm{\pi} }({\hat{\bm{s}}}_{t}))}\right]}_{\operatorname*{\smash{\overset{\mathrm{def}}{ =}}}J_{r_{n}}({\bm{\pi}},{\bm{f}})}.$ (4) $$

Furterhmore, akin to Curi et al. (); Sukhija et al. (), we introduce additional exploration, by also optimistically picking the dynamics ${\bm{f}}_{n}$ from our set of plausible models ${\mathcal{M}}_{n}$. Moreover, since the true dynamics, ${\bm{f}}^{*}$ are unknown, we have to plan w.r.t. some dynamics model in ${\mathcal{M}}_{n}$. A theoretically grounded and well-established strategy for model-based RL methods is to pick an optimistic model ${\bm{f}}_{n}$ from ${\mathcal{M}}_{n}$. As we show in Theorem 4.8 this results in first-of-its-kind sample complexity and safety guarantees.
The expansion phase of the algorithm is summarized in Algorithm 1.

###### Theorem 4.8 .

Let Assumptions 4.1, 4.2, 4.5, 4.5 and 4.3 hold. Then, we have with probability at least $1-\delta$ that $J_{c}({\bm{\pi}}_{n},{\bm{f}}^{*})\leq d$ $\forall n\geq 0$, i.e., *ActSafe is safe during all episodes*.

Moreover, consider any $\epsilon>0$ and define ${\mathcal{R}}^{\epsilon}_{H}({\mathcal{S}}_{0})$ as the reachable safe set after $H$ expansions

$$ $\displaystyle{\mathcal{R}}^{\epsilon}_{H}({\mathcal{S}}_{0})$ $\displaystyle\operatorname*{\smash{\overset{\mathrm{def}}{=}}}{\mathcal{R}}^{ \epsilon}_{H-1}({\mathcal{S}}_{0})\cup\left\{{\bm{\pi}}\in\Pi\setminus{ \mathcal{R}}^{\epsilon}_{H-1}({\mathcal{S}}_{0});\exists{\bm{\pi}}^{\prime}\in {\mathcal{R}}^{\epsilon}_{H-1}({\mathcal{S}}_{0})\ \mathrm{s.t.}\ J_{c}({\bm{ \pi}}^{\prime})+D({\bm{\pi}},{\bm{\pi}}^{\prime})\leq d-\epsilon\right\}$ $\displaystyle{\mathcal{R}}^{\epsilon}_{0}({\mathcal{S}}_{0})$ $\displaystyle\operatorname*{\smash{\overset{\mathrm{def}}{=}}}{\mathcal{S}}_{0}.$ $$

Let $n^{*}$ be the smallest integer such that

$$ $\frac{n^{*}}{\gamma_{n^{*}}(k)\beta^{4}_{n^{*}}(\delta)}\geq\frac{(H+1)T^{6}C^ {4}\frac{d_{s}\sigma^{2}_{0}}{\log(1+\sigma^{-2}\sigma^{2}_{0})}}{\epsilon^{2} }.,$ (5) $$

where $C=(1+\sqrt{d_{s}})\max\{C_{\max},R_{\max},\sigma_{0}\}$, $\gamma_{n}(k)$ the maximum information gain (Srinivas et al., ), and $\tilde{{\bm{\pi}}}_{n}$ the solution to $\underset{{\bm{\pi}}\in{\mathcal{S}}_{n}}{\arg\max}\min_{{\bm{f}}\in{\mathcal{
M}}_{n}}J_{r}({\bm{\pi}},{\bm{f}})$.
Then we have $\forall n\geq n^{*}$ with probability at least $1-\delta$

$$ $\max_{{\bm{\pi}}\in{\mathcal{R}}^{\epsilon}_{H}({\mathcal{S}}_{0})}J_{r}({\bm{ \pi}})-J_{r}(\tilde{{\bm{\pi}}}_{n})\leq\epsilon.$ $$

The theorem shows that ActSafe is safe during all episodes.
Furthermore, it shows that after finishing the expansion phase, ActSafe achieves an $\epsilon$-optimal solution within ${\mathcal{R}}^{\epsilon}_{H}({\mathcal{S}}_{0})$ for the underlying reward function $r$, where ${\mathcal{R}}^{\epsilon}_{H}({\mathcal{S}}_{0})$ is the largest safe set we can obtain after $H$ expansion steps if we know the dynamics to $\epsilon$ precision. To the best of our knowledge, we are the first to prove safety and give sample complexity bounds for safe model-based RL algorithms in the episodic setting with continuous state-action spaces.

Intuitively, by maximizing the epistemic uncertainty, we explore our dynamics uniformly among all policies in the safe set ${\mathcal{S}}_{n}$, making our model more confident, i.e., reducing ${\bm{\sigma}}_{n}$. As our uncertainty within ${\mathcal{S}}_{n}$ shrinks, we add more policies to our safe set (cf. Definition 4.7) and thus facilitate its expansion. The proof of Theorem 4.8 is given in Appendix A.

While the algorithm itself is difficult to implement for continuous state-action spaces, it gives key insights that guide our practical implementation in Section 4.3:
(i) maximization of intrinsic rewards for expansion,
(ii) pessimism w.r.t. plausible dynamics to define a safe set of policies ${\mathcal{S}}_{n}$, and
(iii) selecting ${\bm{\pi}}_{n}$ only from ${\mathcal{S}}_{n}$ to ensure safety.
Building on these insights, we introduce a practical version of ActSafe designed to excel in real-world scenarios, such as visual control tasks.

###### Theorem 4.8 .

### 4.3 Practical Implementation

##### Optimizing over safe policies

In Equation 4 we optimize over the policies within the safe set, where the safe set is defined according to Definition 4.7.
This is particularly challenging in continuous state-action spaces since it requires us to maintain the model set ${\mathcal{M}}_{n}$ and the safe set ${\mathcal{S}}_{n}$.
We modify the definition of the safe set which makes the optimization problem more tractable.

$$ $\widehat{{\mathcal{S}}}_{n}=\left\{{\bm{\pi}}\in\Pi;\ \text{ s.t.}\max_{{\bm{f }}^{\prime}\in{\mathcal{Q}}_{n}}J_{c}({\bm{\pi}},{\bm{f}}^{\prime})\leq d\right\}$ (6) $$

Note that $\widehat{{\mathcal{S}}}_{n}\subseteq{\mathcal{S}}_{n}$, making it a conservative estimate of ${\mathcal{S}}_{n}$, therefore selecting policies from $\widehat{{\mathcal{S}}}_{n}$ still preserves the safety guarantees. Furthermore, in $\widehat{{\mathcal{S}}}_{n}$, we are pessimistic w.r.t. the dynamics ${\bm{f}}\in{\mathcal{Q}}_{n}$ and thus we can simply use ${\bm{\mu}}_{n},{\bm{\sigma}}_{n}$ to induce pessimism, i.e., we do not have to maintain the model set ${\mathcal{M}}_{n}={\mathcal{M}}_{n-1}\cap{\mathcal{Q}}_{n}$ (c.f., Definition 4.4). A similar relaxation is made by other safe RL algorithms such as Berkenkamp et al. (); Baumann et al. (); Sukhija et al. ().

To practically solve Equation 4 we use $\widehat{{\mathcal{S}}}_{n}$ instead of ${\mathcal{S}}_{n}$, resulting in the following problem

$$ $\displaystyle\operatorname*{arg\,max}_{{\bm{\pi}}\in\Pi}\max_{{\bm{f}}\in{ \mathcal{Q}}_{n}}J_{n}({\bm{\pi}},{\bm{f}})\ \text{s.t.}\ \max_{{\bm{f}}^{ \prime}\in{\mathcal{Q}}_{n}}J_{c}({\bm{\pi}},{\bm{f}}^{\prime})\leq d.$ (7) $$

Equation 7 is a constrained optimization problem with the added complexity of optimizing over the dynamics in ${\mathcal{Q}}_{n}$. Moreover, it does not require us to maintain $\widehat{{\mathcal{S}}}_{n}$ since we implicitly account for it in the constraint in Equation 7, making it tractable for continuous state-action spaces.
In Equation 7, we have to solve $\max_{{\bm{f}}^{\prime}\in{\mathcal{Q}}_{n}}J_{c}({\bm{\pi}},{\bm{f}}^{\prime})$ to enforce pessimism for safety. To this end, we use the methods from Yu et al. () for our experiments. In practice, we solve Equation 7 by using a CMDP planner based on Log-Barrier SGD (LBSGD, Usmanova et al., ). Further technical details can be found in Appendix B.

Figure: Algorithm 2 ActSafe: Practical Version

##### From CMDPs to visual control

ActSafe can be seamlessly integrated with state-of-the-art model-based RL methods for learning in visual control tasks (Hafner et al., ; ). To tighten the gap between RL and real-world problems, we relax the typical full observability assumption and consider problems where the agent receives an observation ${\bm{o}}_{t}\sim p(\cdot|{\bm{s}}_{t})$ instead of ${\bm{s}}_{t}$ at each time step.
To handle partial observability, we choose to base our dynamics model on the Recurrent State Space Model (RSSM) introduced in Hafner et al. (). The RSSM can be thought of as a sequential variational auto-encoder that learns the (latent) dynamics ${\bm{f}}$. We leverage approximate Bayesian inference techniques, in particular probabilistic ensembles (Lakshminarayanan et al., ), to approximate the posterior $p({\bm{f}}|{\mathcal{D}}_{n})$ over RSSMs. In particular, we learn an ensemble of $M$ models and define ${\mathcal{Q}}_{n}$ as ${\mathcal{Q}}_{n}=\bigcup^{M-1}_{i=0}\{{\bm{f}}^{i}\}$. The model’s epistemic uncertainty (disagreement) is then used to enforce pessimism w.r.t. the safety constraints and for the intrinsic reward exploration (see Algorithm 2).

##### Online policy improvement

After the first intrinsic exploration phase, it is often necessary to perform additional learning updates during the second exploitation phase (Sekar et al., ). Therefore, after $n^{*}$ iterations of intrinsic exploration, we optimize the extrinsic reward by solving

$$ $\displaystyle\operatorname*{arg\,max}_{{\bm{\pi}}\in\Pi}\max_{{\bm{f}}\in{ \mathcal{Q}}_{n}}J_{r}({\bm{\pi}},{\bm{f}})\ \text{s.t.}\ \max_{{\bm{f}}^{ \prime}\in{\mathcal{Q}}_{n}}J_{c}({\bm{\pi}},{\bm{f}}^{\prime})\leq d.$ (8) $$

## 5 Experiments

In the following, we evaluate ActSafe on state-based and visual control tasks. For the state-based tasks, we use GPs to model the dynamics ${\bm{f}}^{*}$. For the visual control tasks, we use the RSSM model from Hafner et al. () as described in Section 4.3. We thus validate both the theoretical and practical aspects of ActSafe in this section.

### 5.1 Does ActSafe Explore Safely with GPs?

Figure: Figure 2: Evaluation of safety via pessimism and intrinsic exploration. The cumulative cost accumulates all the incurred costs during learning, the reported objective performance is normalized. ActSafe maintains safety during learning while attaining high zero-shot performance on the test task.
Refer to caption: x2.png

We evaluate ActSafe on the Pendulum and Cartpole environments. Additional details on the experimental setup, including the safety constraints, are provided in Appendix B.
For both environments, we run the algorithms for ten episodes and then use the learned model to plan w.r.t. known extrinsic rewards after the expansion phase. For extrinsic rewards, we consider the Swingup task.
We study the effects of pessimism with respect to the model uncertainty for safety. To this end, we consider as baselines a version of ActSafe without pessimism, which only uses the mean model ${\bm{\mu}}_{n}$ for planning and Opax (Sukhija et al., ), an unsafe active exploration algorithm.

We present our results in Figure 2, where we report the performance and the total accumulated costs during exploration of our method.
We conclude that ActSafe does not incur any costs during learning. In contrast, the variant of ActSafe without pessimism and Opax are unsafe during learning. This validates the necessity of using the model epistemic uncertainty to enforce pessimism during exploration. Note that ActSafe pays a price in terms of performance for pessimism, as it converges to a lower reward value than the other algorithms.

Figure: Figure 3: Safe exploration in the PendulumSwingup task. Each plot above visualizes trajectories considered during exploration across all past learning episodes. The red box in the plot depicts the safety boundary in the state space. ActSafe maintains safety throughout learning.
Refer to caption: x3.png

In Figure 3 we visualize the trajectories of ActSafe and Opax in the state space during exploration. We observe that both algorithms cover the state space well, however, ActSafe remains within the safety boundary during learning whereas Opax violates the constraints.

### 5.2 Does ActSafe Scale to Vision Control?

While with GPs we can work closer to theory, scaling them to high dimensions with large data regimes, in particular visual control tasks, is challenging. We demonstrate our practical implementation (Algorithm 2) on high-dimensional RL tasks. For all experiments hereon, we assume to have access to offline-collected data of 200K environment steps, collected with a random policy  (akin to Dalal et al., ). We highlight that ensuring safety with an NN model with randomly initialized weights is impractical without any additional prior knowledge.
In Appendix C, we present additional experiments that study safe exploration under distribution shifts of the dynamics, imitating sim-to-real transfer, without offline data.

##### Safety

We investigate ActSafe’s performance in terms of constraint satisfaction during learning and compare it with state-of-the-art baseline algorithms for safe vision control  (As et al., ; Huang et al., ) and with CPO (Achiam et al., ). We use the same experimental setup from Safety-Gym(Ray et al., ) and As et al. (), with the Point robot in all tasks.

Figure: Figure 4: Safety of ActSafe in Safety-Gym with vision control. The dotted horizontal line depicts the safety constraint. We report the median and standard error across 5 seeds. The vertical dashed line illustrates the transition of ActSafe from the intrinsic exploraiton/expansion phase to the extrinsic reward phase.
Refer to caption: x4.png

As shown in Figure 4, compared to the baselines, ActSafe, significantly reduces constraint violation on all tasks. Notably, while ActSafe slightly underperforms LAMBDA, it incurs much smaller costs. This result may be interpreted by the conservatism needed to maintain safety during learning. Furthermore, we observe that BSRP-Lag generally underperforms both algorithms in terms of safety and performance. We provide more details on our comparison in Appendix B. Additionally, we ablate our choice of LBSGD in Appendix C and highlight its benefits.

##### Exploration

In this experiment, we examine the influence of using an intrinsic reward in hard exploration tasks. To this end, we extend tasks from Safety-Gym and introduce three new tasks with sparse rewards, i.e., without any reward shaping to guide the agent to the goal. We provide more details about the rewards in Appendix B. In Figure 5 we compare ActSafe with a Greedy baseline that collects trajectories only based on the sparse extrinsic reward. As shown, ActSafe substantially outperforms Greedy in all tasks, while violating the constraint only once in the GotoGoal task.

Figure: Figure 5: Performance on hard safe exploration tasks. The vertical dashed line illustrates the transition from data collection with intrinsic to extrinsic reward.
Refer to caption: x5.png

In addition to Safety-Gym, we evaluate on CartpoleSwingupSparse from RWRL (Dulac-Arnold et al., ) with additional penalty for large actions (see Curi et al., , and Appendix B). We compare ActSafe with three baselines.
(i) Uniform, which samples actions uniformly at random during exploration,
(ii) Optimistic, which uses the model epistemic uncertainty estimates as exploration reward bonuses and
(iii) Greedy, which optimizes the extrinsic reward directly.
Figure 6 indicates that using uncertainty quantification for exploration is crucial, as only ActSafe and Optimistic find non-trivial policies. Despite that, ActSafe outperforms Optimistic. Furthermore, even though Uniform initially explores with an unsafe policy, it is insufficient to learn a good dynamics model, and thus underperforms ActSafe. This is mainly due to the undirected exploration strategy of Uniform, which does not leverage the model’s epistemic uncertainty.

Figure: Figure 6: Hard exploration performance in CartpoleSwingupSparse of RWRL benchmark. We report the mean and standard error.
Refer to caption: x6.png

##### Discussion

Our experiments underscore the following key findings. First, intuitively, in the GP setting, where our implementation is closer to theory, pessimism w.r.t. the model uncertainty plays a crucial role as we achieve strict safe exploration. Second, in our visual control experiments, using a small fraction of offline data ($<$5$\%$ of total data collected) as initialization for the model and policy is sufficient for drastically reducing constraint violation. Learning safely typically requires some form of prior knowledge about the problem, hence, using offline data keeps the experiment setup realistic (e.g., Kidambi et al. (); Matsushima et al. (); Feng et al. () and references therein) without imposing specific domain knowledge and thus sacrificing generality. Third,
in addition to exploring safely ActSafe, also solves tough exploration problems with the intrinsic rewards playing a crucial role.
These results underline the importance of intrinsic exploration in RL, especially in safety-critical tasks. Moreover, ActSafe transfers directly from the GP setting to the vision control setting and in both cases our results show that ActSafe outperforms the baselines in terms of both safety and performance. We provide additional experiments in Appendix C, where ablate our choice of the LBSGD planner, evaluate ActSafe on a setting with distribution shifts in the dynamics and on a realistic robotics task from the state-of-the-art humanoid benchmark from Sferrazza et al. ().

## 6 Conclusions

In this paper, we introduce ActSafe, a safe model-based RL algorithm that leverages epistemic uncertainty as an intrinsic reward to learn a dynamics model efficiently and safely. We theoretically study systems with continuous state-action spaces and non-linear dynamics that lie in the RKHS, and provide guarantees on safety and near-optimality. We derive a practical variant of ActSafe, and demonstrate safe exploration and competitive performance with a Gaussian process dynamics model. Furthermore, we identify the key concepts that enable safe exploration with ActSafe and demonstrate how one can heuristically apply them to solve high-dimensional safe RL problems. Our empirical results showcase the importance of intrinsic rewards in the context of safety, demonstrating that ActSafe outperforms the baselines in the majority of tasks. In conclusion, ActSafe represents a significant advancement in safe reinforcement learning methods, enhancing both theoretical insights and practical applications.

#### Acknowledgments

We thank Jonas Hübotter for the insightful discussion and feedback on this work.
This project has received funding from the Swiss National Science Foundation under NCCR Automation, grant agreement 51NF40 180545, the Microsoft Swiss Joint Research Center, grant of the Hasler foundation (grant no. 21039) and the SNSF Postdoc Mobility Fellowship 211086.

## Appendix A Proofs

In the following, we prove Theorem 4.8. First, we provide the analytical formula for the mean and epistemic uncertainty of a GP model.
We denote ${\bm{x}}\coloneq({\bm{s}},{\bm{a}})$, so that

$$ $\displaystyle\begin{split}\mu_{n,j}({\bm{x}})&={\bm{k}}_{n}^{\top}({\bm{x}})({ \bm{K}}_{n}+\sigma^{2}_{\epsilon}\bm{I})^{-1}{\bm{y}}_{n,j}\\ \sigma^{2}_{n,j}({\bm{x}})&=k({\bm{x}},{\bm{x}})-{\bm{k}}^{T}_{n}({\bm{x}})({ \bm{K}}_{n}+\sigma^{2}_{\epsilon}\bm{I})^{-1}{\bm{k}}_{n}({\bm{x}})\end{split}$ (9) $$

where ${\bm{y}}_{n,j}=[s^{\prime}_{i,j}]^{\top}_{i\leq n}$ is the vector of the $j$-th element of the observed next states ${\bm{s}}^{\prime}_{i}$, $\bm{k}_{n}({\bm{x}})=[k({\bm{x}},{\bm{x}}_{i})]^{\top}_{i\leq n}$, and ${\bm{K}}_{n}=[k({\bm{x}}_{i},{\bm{x}}_{l})]_{i,l\leq n}$ is the kernel matrix. By concatenating the element-wise posterior mean and standard deviation, we obtain ${\bm{\mu}}_{n}({\bm{x}})=[\mu_{n,j}({\bm{x}})]^{\top}_{j\leq d_{s}}$ and
${\bm{\sigma}}_{n}({\bm{x}})=[\sigma_{n,j}({\bm{x}})]^{\top}_{j\leq d_{s}}$.

###### Corollary A.1 .

Let assumption 4.5 hold, then we have for all ${\bm{\pi}}\in\Pi$, $n\geq 0$, with probability at least $1-\delta$

$$ $P_{n}({\bm{\pi}})\geq J_{c}({\bm{\pi}},{\bm{f}}^{*})$ $$

###### Proof.

Note that ${\bm{f}}^{*}\in{\mathcal{Q}}_{n}$ for all $n\geq 0$ with probability at least $1-\delta$ (Lemma 4.6). Therefore, ${\bm{f}}^{*}\in{\mathcal{M}}_{n}$. Furthermore,

$$ $\displaystyle P_{n}({\bm{\pi}})$ $\displaystyle=\max_{{\bm{f}}\in{\mathcal{M}}_{n}}J_{c}({\bm{\pi}},{\bm{f}})$ $\displaystyle\geq J_{c}({\bm{\pi}},{\bm{f}}^{*})$ $$

∎

###### Lemma A.2 (Difference in Policy performance, Sukhija et al. ( 2024 ) ) .

Consider any function $r:{\mathcal{S}}\times{\mathcal{A}}\to\mathbb{R}$.
Let $J_{r,k}({\bm{\pi}},{\bm{f}}^{*},{\bm{s}}_{k})=\mathbb{E}_{{\bm{\tau}}^{{\bm{
\pi}}}}\left[\sum_{t=k}^{T-1}r({\bm{s}}_{t},{\bm{\pi}}({\bm{s}}_{t}))\right]$ and $A_{r,k}({\bm{\pi}},{\bm{s}},{\bm{a}})=\mathbb{E}_{{\bm{\tau}}^{{\bm{\pi}}}}
\left[r({\bm{s}},{\bm{a}})+J_{r,k+1}({\bm{\pi}},{\bm{f}}^{*},{\bm{s}}^{\prime}
)-J_{r,k}({\bm{\pi}},{\bm{f}}^{*},{\bm{s}})\right]$ with ${\bm{s}}^{\prime}={\bm{f}}^{*}({\bm{s}},{\bm{a}})+{\bm{w}}$. For simplicity we refer to $J_{r,0}({\bm{\pi}},{\bm{f}}^{*},{\bm{s}}_{0})=J_{r}({\bm{\pi}},{\bm{f}}^{*},{
\bm{s}}_{0})$.
The following holds for all ${\bm{s}}_{0}\in{\mathcal{S}}$:

$$ $J_{r}({\bm{\pi}}^{\prime},{\bm{f}}^{*},{\bm{s}}_{0})-J_{r}({\bm{\pi}},{\bm{f}} ^{*},{\bm{s}}_{0})=\mathbb{E}_{{\bm{\tau}}^{{\bm{\pi}}^{\prime}}}\left[\sum^{T -1}_{t=0}A_{r,t}({\bm{\pi}},{\bm{s}}^{\prime}_{t},{\bm{\pi}}^{\prime}({\bm{s}} ^{\prime}_{t}))\right]$ $$

###### Proof.

See Lemma 5. Sukhija et al. ().
∎

###### Lemma A.3 (Comparing safety costs of policies) .

$$ $J_{c}({\bm{\pi}},{\bm{f}}^{*},{\bm{s}}_{0})-J_{c}({\bm{\pi}}^{\prime},{\bm{f}} ^{*},{\bm{s}}_{0})\leq D({\bm{\pi}},{\bm{\pi}}^{\prime})$ $$

###### Proof.

For notational convenience we will omit the dependance on ${\bm{f}}^{*}$.

$$ $\displaystyle J_{c}({\bm{\pi}},{\bm{s}}_{0})-J_{c}({\bm{\pi}}^{\prime},{\bm{s} }_{0})=\mathbb{E}_{{\bm{\tau}}^{{\bm{\pi}}^{\prime}}}\left[\sum^{T-1}_{t=0}-A_ {c,t}({\bm{\pi}},{\bm{s}}^{\prime}_{t},{\bm{\pi}}^{\prime}({\bm{s}}^{\prime}_{ t}))\right]$ $\displaystyle=\mathbb{E}_{{\bm{\tau}}^{{\bm{\pi}}^{\prime}}}\left[\sum^{T-1}_{ t=0}-(c({\bm{s}}_{t},{\bm{\pi}}^{\prime}({\bm{s}}_{t}))-c({\bm{s}}_{t},{\bm{ \pi}}({\bm{s}}_{t})))\right]$ $\displaystyle+\mathbb{E}_{{\bm{\tau}}^{{\bm{\pi}}^{\prime}}}\left[\sum^{T-1}_{ t=0}\mathbb{E}_{{\bm{s}}^{\prime}_{t+1}|{\bm{s}}_{t},{\bm{\pi}}({\bm{s}}_{t})} \left[J_{c,k+1}({\bm{\pi}},{\bm{s}}^{\prime}_{t+1})\right]-\mathbb{E}_{{\bm{s} }^{\prime}_{t+1}|{\bm{s}}_{t},{\bm{\pi}}^{\prime}({\bm{s}}_{t})}\left[J_{c,k+1 }({\bm{\pi}},{\bm{s}}^{\prime}_{t+1})\right]\right]$ $\displaystyle=\mathbb{E}_{{\bm{\tau}}^{{\bm{\pi}}^{\prime}}}\left[\sum^{T-1}_{ t=0}c({\bm{s}}_{t},{\bm{\pi}}({\bm{s}}_{t}))-c({\bm{s}}_{t},{\bm{\pi}}^{\prime }({\bm{s}}_{t}))\right]$ $\displaystyle+\mathbb{E}_{{\bm{\tau}}^{{\bm{\pi}}^{\prime}}}\left[\sum^{T-1}_{ t=0}\mathbb{E}_{{\bm{s}}^{\prime}_{t+1}|{\bm{s}}_{t},{\bm{\pi}}({\bm{s}}_{t})} \left[J_{c,k+1}({\bm{\pi}},{\bm{s}}^{\prime}_{t+1})\right]-\mathbb{E}_{{\bm{s} }^{\prime}_{t+1}|{\bm{s}}_{t},{\bm{\pi}}^{\prime}({\bm{s}}_{t})}\left[J_{c,k+1 }({\bm{\pi}},{\bm{s}}^{\prime}_{t+1})\right]\right]$ $\displaystyle\leq\mathbb{E}_{{\bm{\tau}}^{{\bm{\pi}}^{\prime}}}\left[\sum^{T-1 }_{t=0}\min\left\{L_{c}\norm{{\bm{\pi}}^{\prime}({\bm{s}}_{t})-{\bm{\pi}}({\bm {s}}_{t})},2C_{\max}\right\}\right]$ $\displaystyle+\mathbb{E}_{{\bm{\tau}}^{{\bm{\pi}}^{\prime}}}\left[\sum^{T-1}_{ t=0}\sqrt{\mathbb{E}_{{\bm{s}}^{\prime}_{t+1}|{\bm{s}}_{t},{\bm{\pi}}({\bm{s}} _{t})}\left[J^{2}_{c,k+1}({\bm{\pi}},{\bm{s}}^{\prime}_{t+1})\right]}\min\left \{\frac{\norm{{\bm{f}}^{*}({\bm{s}}_{t},{\bm{\pi}}({\bm{s}}_{t}))-{\bm{f}}^{*} ({\bm{s}}_{t},{\bm{\pi}}^{\prime}({\bm{s}}_{t}))}}{\sigma},1\right\}\right]$ (Kakade et al., 2020 , Lemma C.2.) $\displaystyle\leq\mathbb{E}_{{\bm{\tau}}^{{\bm{\pi}}^{\prime}}}\left[\sum^{T-1 }_{t=0}\min\left\{L_{c}\norm{{\bm{\pi}}^{\prime}({\bm{s}}_{t})-{\bm{\pi}}({\bm {s}}_{t})},2C_{\max}\right\}+TC_{\max}\min\left\{\frac{L_{\bm{f}}\norm{{\bm{ \pi}}^{\prime}({\bm{s}}_{t})-{\bm{\pi}}({\bm{s}}_{t})}}{\sigma},1\right\}\right]$ $\displaystyle=D({\bm{\pi}},{\bm{\pi}}^{\prime})$ $$

∎

###### Lemma A.4 .

Let assumption 4.1 – assumption 4.5 hold. Then we have $\forall n\geq 0$, ${\bm{\pi}}\in{\mathcal{S}}_{n}\setminus{\mathcal{S}}_{n-1}$ with probability at least $1-\delta$,
$J_{c}({\bm{\pi}})\leq d$.

###### Proof.

Consider any ${\bm{\pi}}\in{\mathcal{S}}_{n}\setminus{\mathcal{S}}_{n-1}$. By Definition 4.7, we have that there exists a ${\bm{\pi}}^{\prime}$ in ${\mathcal{S}}_{n-1}$ such that

$$ $P_{n}({\bm{\pi}}^{\prime})+D({\bm{\pi}},{\bm{\pi}}^{\prime})\leq d$ $$

Therefore,

$$ $\displaystyle d$ $\displaystyle\geq P_{n}({\bm{\pi}}^{\prime})+D({\bm{\pi}},{\bm{\pi}}^{\prime})$ $\displaystyle\geq J_{c}({\bm{\pi}}^{\prime},{\bm{f}}^{*})+D({\bm{\pi}},{\bm{ \pi}}^{\prime})$ ( Corollary A.1 ) $\displaystyle\geq J_{c}({\bm{\pi}},{\bm{f}}^{*}).$ ( Lemma A.3 ) $$

∎

###### Corollary A.5 (All policies in ${\mathcal{S}}_{n}$ are safe) .

Let assumption 4.1 – assumption 4.5 hold. Then we have $\forall n\geq 0$, ${\bm{\pi}}\in{\mathcal{S}}_{n}$ with probability at least $1-\delta$, $J_{c}({\bm{\pi}})\leq d$.

###### Proof.

We prove this by induction. For $n=0$, this holds due to assumption 4.3.
Consider any $n>0$. By induction, $\forall{\bm{\pi}}\in{\mathcal{S}}_{n}$ we have that $J_{c}({\bm{\pi}},{\bm{f}}^{*})\leq d$. Hence, we focus on ${\bm{\pi}}\in{\mathcal{S}}_{n+1}\setminus{\mathcal{S}}_{n}$. In Lemma A.4, we show $J_{c}({\bm{\pi}},{\bm{f}}^{*})\leq d$ for all ${\bm{\pi}}\in{\mathcal{S}}_{n+1}\setminus{\mathcal{S}}_{n}$. This completes the proof.
∎

###### Lemma A.6 .

Consider any positive and bounded function $c\in[0,C_{\max}]$. Let assumption 4.1 – 4.5 hold. Then we have $\forall n\geq 0$, $\forall{\bm{f}}\in{\mathcal{M}}_{n}$. with probability at least $1-\delta$

$$ $|J_{c}({\bm{\pi}},{\bm{f}})-J_{c}({\bm{\pi}},{\bm{f}}^{*})|\leq TC_{\max} \mathbb{E}_{{\bm{\tau}}^{{\bm{\pi}}}}\left[\sum^{T-1}_{t=0}\frac{(1+\sqrt{d_{s }})\beta_{n-1}(\delta)\norm{{\bm{\sigma}}_{n-1}({\bm{s}}_{t},{\bm{\pi}}({\bm{s }}_{t}))}}{\sigma}\right].$ $$

###### Proof.

From Sukhija et al. (, Corollary 2.) we have,

$$ $\displaystyle J_{c}({\bm{\pi}},{\bm{f}})-J_{c}({\bm{\pi}},{\bm{f}}^{*})$ $\displaystyle=\mathbb{E}_{{\bm{\tau}}^{{\bm{\pi}}}}\left[\sum^{T-1}_{t=0}J_{c, t+1}({\bm{\pi}},{\bm{f}},{\bm{s}}_{t+1})-J_{c,t+1}({\bm{\pi}},{\bm{f}},{\bm{s} }^{\prime}_{t+1})\right],$ (Expectation w.r.t ${\bm{\pi}}$ under true dynamics ${\bm{f}}^{*}$ ) $\displaystyle\text{with }{\bm{s}}_{t+1}={\bm{f}}^{*}({\bm{s}}_{t},{\bm{\pi}}({ \bm{s}}_{t}))+{\bm{w}}_{t},$ $\displaystyle\text{and }{\bm{s}}^{\prime}_{t+1}={\bm{f}}({\bm{s}}_{t},{\bm{\pi }}({\bm{s}}_{t}))+{\bm{w}}_{t}.$ $$

Furthermore, $J_{c,t+1}({\bm{\pi}},{\bm{f}},{\bm{s}})\in[0,TC_{\max}]$ for all ${\bm{\pi}},{\bm{f}},{\bm{s}}$, and $t$. Therefore, given ${\bm{s}}_{t}$,

$$ $\displaystyle\left|\mathbb{E}_{{\bm{w}}_{t}}\left[J_{c,t+1}({\bm{\pi}},{\bm{f} },{\bm{s}}^{\prime}_{t+1})-J_{c,t+1}({\bm{\pi}},{\bm{f}},{\bm{s}}_{t+1})\right ]\right|$ $\displaystyle\leq\max\left\{\sqrt{\mathbb{E}_{{\bm{w}}_{t}}[J^{2}_{c,t+1}({\bm {\pi}},{\bm{f}},{\bm{s}}^{\prime}_{t+1})]},\sqrt{\mathbb{E}_{{\bm{w}}_{t}}[J^{ 2}_{c,t+1}({\bm{\pi}},{\bm{f}},{\bm{s}}_{t+1})]}\right\}\min\left\{\frac{\norm {{\bm{f}}^{*}({\bm{s}}_{t},{\bm{\pi}}({\bm{s}}_{t}))-{\bm{f}}({\bm{s}}_{t},{ \bm{\pi}}({\bm{s}}_{t}))}}{\sigma},1\right\}$ (Kakade et al., 2020 , Lemma C.2.) $\displaystyle\leq TC_{\max}\min\left\{\frac{\norm{{\bm{f}}^{*}({\bm{s}}_{t},{ \bm{\pi}}({\bm{s}}_{t}))-{\bm{f}}({\bm{s}}_{t},{\bm{\pi}}({\bm{s}}_{t}))}}{ \sigma},1\right\}$ $\displaystyle\leq TC_{\max}\min\left\{\frac{(1+\sqrt{d_{s}})\beta_{n-1}(\delta )\norm{{\bm{\sigma}}_{n-1}({\bm{s}}_{t},{\bm{\pi}}({\bm{s}}_{t}))}}{\sigma},1\right\}$ (Sukhija et al., 2024 , Corollary 3) $$

∎

From hereon let $C=\frac{(1+\sqrt{d_{s}})\max\{R_{\max},C_{\max},\sigma_{0}\}}{\sigma}$.

###### Lemma A.7 .

Let assumption 4.1 – 4.5 hold. Then we have $\forall n,N\geq 0$ with probability at least $1-\delta$

$$ $\max_{{\bm{\pi}}\in{\mathcal{S}}_{n}}\mathbb{E}_{{\bm{\tau}}^{{\bm{\pi}}}} \left[\sum^{T-1}_{t=0}\norm{{\bm{\sigma}}_{N+n-1}({\bm{s}}_{t},{\bm{\pi}}({\bm {s}}_{t}))}\right]\leq T^{2}C\frac{\sqrt{d_{s}}\sigma_{0}}{\sqrt{\log(1+\sigma ^{-2}\sigma^{2}_{0}})}\sqrt{\frac{\beta^{2}_{n+N-1}(\delta)\gamma_{n+N-1}(k)}{ N}}.$ $$

###### Proof.

Consider any $N>0$,

$$ $\displaystyle\max_{{\bm{\pi}}\in{\mathcal{S}}_{n}}\mathbb{E}_{{\bm{\tau}}^{{ \bm{\pi}}}}\left[\sum^{T-1}_{t=0}\norm{{\bm{\sigma}}_{N+n-1}({\bm{s}}_{t},{\bm {\pi}}({\bm{s}}_{t}))}\right]$ $\displaystyle\leq\frac{1}{N}\sum^{N-1}_{i=0}\max_{{\bm{\pi}}\in{\mathcal{S}}_{ n}}\mathbb{E}_{{\bm{\tau}}^{{\bm{\pi}}}}\left[\sum^{T-1}_{t=0}\norm{{\bm{ \sigma}}_{n+i}({\bm{s}}_{t},{\bm{\pi}}({\bm{s}}_{t}))}\right]$ (Monotonocity of the variance) $\displaystyle\leq\frac{1}{N}\sum^{N-1}_{i=0}\max_{{\bm{\pi}}\in{\mathcal{S}}_{ n+i}}\mathbb{E}_{{\bm{\tau}}^{{\bm{\pi}}}}\left[\sum^{T-1}_{t=0}\norm{{\bm{ \sigma}}_{n+i}({\bm{s}}_{t},{\bm{\pi}}({\bm{s}}_{t}))}\right]$ (Monotonocity of the safe set) $\displaystyle=\frac{1}{N}\sum^{n+N-1}_{i=n}\mathbb{E}_{{\bm{\tau}}^{{\bm{\pi}} ^{*}_{i}}}\left[\sum^{T-1}_{t=0}\norm{{\bm{\sigma}}_{i}({\bm{s}}_{t},{\bm{\pi} }^{*}_{i}({\bm{s}}_{t}))}\right]$ (Definition of ${\bm{\pi}}^{*}_{i}$ ) $\displaystyle=\frac{1}{N}\sum^{n+N-1}_{i=n}\mathbb{E}_{{\bm{\tau}}^{{\bm{\pi}} _{i}}}\left[\sum^{T-1}_{t=0}\norm{{\bm{\sigma}}_{i}({\bm{s}}_{t},{\bm{\pi}}_{i }({\bm{s}}_{t}))}\right]+\frac{1}{N}(J({\bm{\pi}}^{*}_{i})-J({\bm{\pi}}_{i}))$ $$

Let $r_{i}=J_{i}({\bm{\pi}}^{*}_{i})-J({\bm{\pi}}_{i})$, where ${\bm{\pi}}_{i}$ is the policy proposed by Equation 4. We analyze this regret term.
Note that since, we optimistically pick dynamics from ${\mathcal{M}}_{n}$, we have $J_{i}({\bm{\pi}}^{*}_{i})\leq J({\bm{\pi}}_{i},{\bm{f}}_{i})$, where ${\bm{f}}_{i}$ are the optimistic dynamics. Therefore, $r_{i}\leq J({\bm{\pi}}_{i},{\bm{f}}_{i})-J({\bm{\pi}}_{i},{\bm{f}}^{*})$. Hence, we can invoke Lemma A.6 to get

$$ $r_{i}\leq TC\left[\sum^{T-1}_{t=0}\beta_{i}(\delta)\norm{{\bm{\sigma}}_{i}({ \bm{s}}_{t},{\bm{\pi}}({\bm{s}}_{t}))}\right].$ $$

Therefore,

$$ $\displaystyle\max_{{\bm{\pi}}\in{\mathcal{S}}_{n}}\mathbb{E}_{{\bm{\tau}}^{{ \bm{\pi}}}}\left[\sum^{T-1}_{t=0}\norm{{\bm{\sigma}}_{N+n-1}({\bm{s}}_{t},{\bm {\pi}}({\bm{s}}_{t}))}\right]$ $\displaystyle\leq\frac{TC\beta_{n+N-1}(\delta)}{N}\sum^{n+N-1}_{i=n}\mathbb{E} _{{\bm{\tau}}^{{\bm{\pi}}_{i}}}\left[\sum^{T-1}_{t=0}\norm{{\bm{\sigma}}_{i}({ \bm{s}}_{t},{\bm{\pi}}_{i}({\bm{s}}_{t}))}\right]$ $\displaystyle\leq\frac{TC\beta_{n+N-1}(\delta)}{N}\sum^{n+N-1}_{i=n}\mathbb{E} _{{\bm{\tau}}^{{\bm{\pi}}_{i}}}\left[\sum^{T-1}_{t=0}\norm{{\bm{\sigma}}_{i}({ \bm{s}}_{t},{\bm{\pi}}_{i}({\bm{s}}_{t}))}\right]$ $\displaystyle\leq\frac{TC\beta_{n+N-1}(\delta)}{N}\sqrt{NT}\sqrt{\sum^{n+N-1}_ {i=n}\mathbb{E}_{{\bm{\tau}}^{{\bm{\pi}}_{i}}}\left[\sum^{T-1}_{t=0}\norm{{\bm {\sigma}}_{i}({\bm{s}}_{t},{\bm{\pi}}_{i}({\bm{s}}_{t}))}^{2}\right]}$ (Cauchy-Schwartz) $\displaystyle\leq\frac{TC\beta_{n+N-1}(\delta)}{N}\sqrt{NT}\sqrt{\sum^{n+N-1}_ {i=0}\mathbb{E}_{{\bm{\tau}}^{{\bm{\pi}}_{i}}}\left[\sum^{T-1}_{t=0}\norm{{\bm {\sigma}}_{i}({\bm{s}}_{t},{\bm{\pi}}_{i}({\bm{s}}_{t}))}^{2}\right]}$ $\displaystyle\leq\frac{TC\frac{\sqrt{Td_{s}}\sigma_{0}}{\sqrt{\log(1+\sigma^{- 2}\sigma^{2}_{0}})}\beta_{n+N-1}(\delta)}{N}\sqrt{NT}\sqrt{\gamma_{n+N-1}(k)}$ (Curi et al., 2020 , Lemma 17) $\displaystyle=T^{2}C\frac{\sqrt{d_{s}}\sigma_{0}}{\sqrt{\log(1+\sigma^{-2} \sigma^{2}_{0}})}\sqrt{\frac{\beta^{2}_{n+N-1}(\delta)\gamma_{n+N-1}(k)}{N}}$ $$

∎

###### Lemma A.8 .

Let assumption 4.1 – 4.5 hold and define $N_{n}$ to be the smallest integer such that

$$ $T^{3}C^{2}\frac{\sqrt{d_{s}}\sigma_{0}}{\sqrt{\log(1+\sigma^{-2}\sigma^{2}_{0} })}\beta^{2}_{n+N_{n}-1}(\delta)\sqrt{\frac{\gamma_{n+N_{n}-1}(k)}{N_{n}}}\leq\epsilon.$ $$

Then, we have $\forall{\bm{\pi}}\in{\mathcal{S}}_{n}$, ${\bm{f}}\in{\mathcal{M}}_{n+N_{n}-1}$ with probability at least $1-\delta$

$$ $|J_{c}({\bm{\pi}},{\bm{f}})-J_{c}({\bm{\pi}},{\bm{f}}^{*})|\leq\epsilon,\; \text{and,}\;|J_{r}({\bm{\pi}},{\bm{f}})-J_{r}({\bm{\pi}},{\bm{f}}^{*})|\leq\epsilon.$ $$

###### Proof.

$$ $\displaystyle|J_{c}({\bm{\pi}},{\bm{f}})-J_{c}({\bm{\pi}},{\bm{f}}^{*})|$ $\displaystyle\leq TC_{\max}\mathbb{E}_{{\bm{\tau}}^{{\bm{\pi}}}}\left[\sum^{T- 1}_{t=0}\frac{(1+\sqrt{d_{s}})\beta_{n+N_{n}-1}(\delta)\norm{{\bm{\sigma}}_{n+ N_{n}-1}({\bm{s}}_{t},{\bm{\pi}}({\bm{s}}_{t}))}}{\sigma}\right]$ ( Lemma A.6 ) $\displaystyle\leq TC\beta_{n+N_{n}-1}\left[\sum^{T-1}_{t=0}\norm{{\bm{\sigma}} _{n+N_{n}-1}({\bm{s}}_{t},{\bm{\pi}}({\bm{s}}_{t}))}\right]$ $\displaystyle\leq TC\beta_{n+N_{n}-1}T^{2}C\frac{\sqrt{d_{s}}\sigma_{0}}{\sqrt {\log(1+\sigma^{-2}\sigma^{2}_{0}})}\beta_{n+N_{n}-1}(\delta)\sqrt{\frac{ \gamma_{n+N_{n}-1}(k)}{N_{n}}}$ ( Lemma A.7 ) $\displaystyle\leq\epsilon$ $$

We can apply the same inequalities for $J_{r}$.
∎

###### Corollary A.9 .

Let assumption 4.1 – 4.5 hold. Consider any $n\geq 0$ and define $N_{n}$ as in Lemma A.8. Then we have with probability at least $1-\delta$

$$ ${\mathcal{S}}_{n+N_{n}}\supseteq{\mathcal{R}}^{\varepsilon}({\mathcal{S}}_{n}).$ $$

###### Proof.

From Lemma A.8, we have $\forall{\bm{\pi}}\in{\mathcal{S}}_{n}$, ${\bm{f}}\in{\mathcal{M}}_{n+N_{n}-1}$,
$|J_{c}({\bm{\pi}},{\bm{f}})-J_{c}({\bm{\pi}},{\bm{f}}^{*})|\leq\epsilon$, therefore $P_{n+N_{n}-1}({\bm{\pi}})\leq J_{c}({\bm{\pi}},{\bm{f}}^{*})+\epsilon$.
For the sake of contradiction, assume there exists a policy ${\bm{\pi}}\in{\mathcal{R}}^{\varepsilon}({\mathcal{S}}_{n})\setminus{\mathcal{
S}}_{n+N_{n}}$. We study the case where ${\bm{\pi}}\in{\mathcal{R}}^{\varepsilon}({\mathcal{S}}_{n})\setminus{\mathcal{
S}}_{n}$ else we have a contradiction (${\mathcal{S}}_{n}\subseteq{\mathcal{S}}_{n+N_{n}}$).
Since ${\bm{\pi}}\in{\mathcal{R}}^{\varepsilon}({\mathcal{S}}_{n})\setminus{\mathcal{
S}}_{n}$, there exists a ${\bm{\pi}}^{\prime}\in{\mathcal{S}}_{n}$ such that
$J_{c}({\bm{\pi}}^{\prime})+D({\bm{\pi}},{\bm{\pi}}^{\prime})\leq d-\epsilon$ (see Theorem 4.8). Hence, we get

$$ $\displaystyle d$ $\displaystyle\geq J_{c}({\bm{\pi}}^{\prime})+\epsilon+D({\bm{\pi}},{\bm{\pi}}^ {\prime})$ $\displaystyle\geq P_{n+N_{n}-1}({\bm{\pi}}^{\prime})+D({\bm{\pi}},{\bm{\pi}}^{ \prime}).$ $$

Since, ${\bm{\pi}}^{\prime}\in{\mathcal{S}}_{n}\subseteq{\mathcal{S}}_{n+N_{n}-1}$, by the definition of the safe set (c.f.  Definition 4.7), this implies that ${\bm{\pi}}\in{\mathcal{S}}_{n+N_{n}}$, which is a contradiction.
∎

A key property of $N_{n}$ is that it increases monotonously with $n$. Moreover, for a given $n\geq 0$, $N_{n}$ is the smallest integer satisfying

$$ $N_{n}\geq\frac{\gamma_{n+N_{n}-1}(k)\beta^{4}_{n+N_{n}-1}(\delta)T^{6}C^{4} \frac{d_{s}\sigma^{2}_{0}}{\log(1+\sigma^{-2}\sigma^{2}_{0})}}{\epsilon^{2}}.$ $$

Both functions $n\mapsto\gamma_{n}$, and $n\mapsto\beta_{n}$ are monotonically increasing with $n$. Hence increasing $n$, increases the right-hand side of the inequality, and therefore $N_{n}$.

###### Lemma A.10 .

Let assumption 4.1 – 4.5 hold and consider $n^{*}\geq(H+1)N_{n^{*}}$. Then we have with probability at least $1-\delta$ for all $n\geq n^{*}$

$$ $S_{n}\supseteq{\mathcal{R}}^{\varepsilon}_{H}({\mathcal{S}}_{0}).$ $$

###### Proof.

To prove this, we show for any positive integer $k\leq H$, that
$S_{kN_{n^{*}}}\supseteq{\mathcal{R}}^{\varepsilon}_{k}({\mathcal{S}}_{0})$ by induction.

Moreover for any $k$, let $T_{k}=T_{k-1}+N_{T_{k-1}}$ and $T_{0}=0$. We inductively show that $T_{k}\leq kN_{n^{*}}$ for all $k\leq H$.

For the base case $k=1$, we have $T_{1}=N_{0}\leq N_{n^{*}}$ since $n^{*}\geq 0$.
Consider any $k\leq H$, then, we have $T_{k}=T_{k-1}+N_{T_{k-1}}$. By induction $T_{k-1}\leq(k-1)N_{n^{*}}$. Therefore, $T_{k}\leq(k-1)N_{n^{*}}+N_{(k-1)N_{n^{*}}}$. Furthermore, note that $(k-1)N_{n^{*}}\leq n^{*}$ for all $k\leq H$. Therefore, $T_{k}\leq(k-1)N_{n^{*}}+N_{n^{*}}=kN_{n^{*}}$.

Next, we have from Corollary A.9, ${\mathcal{S}}_{T_{k}}\supseteq{\mathcal{R}}^{\varepsilon}({\mathcal{S}}_{T_{k-
1}})\coloneqq{\mathcal{R}}^{\varepsilon}_{k}({\mathcal{S}}_{0})$.
Moreover, ${\mathcal{S}}_{T_{1}}\coloneqq{\mathcal{S}}_{N_{0}}\supseteq{\mathcal{R}}^{
\varepsilon}({\mathcal{S}}_{0})$. Similarly, ${\mathcal{S}}_{T_{2}}\coloneqq{\mathcal{S}}_{N_{0}+N_{N_{0}}}\supseteq{
\mathcal{R}}^{\varepsilon}({\mathcal{S}}_{1})\coloneqq{\mathcal{R}}^{
\varepsilon}_{2}({\mathcal{S}}_{0})$, etc. Therefore, we get ${\mathcal{S}}_{HN_{n^{*}}}\supseteq{\mathcal{S}}_{T_{H}}\supseteq{\mathcal{R}}
^{\varepsilon}_{H}({\mathcal{S}}_{0})$. As $n^{*}\geq HN_{n^{*}}$, this completes the proof.
∎

###### Lemma A.11 .

Let assumption 4.1 – 4.5 hold and consider the smallest integer $n^{*}$ such that

$$ $\frac{n^{*}}{\gamma_{n^{*}}(k)\beta^{4}_{n^{*}}(\delta)}\geq\frac{(H+1)T^{6}C^ {4}\frac{d_{s}\sigma^{2}_{0}}{\log(1+\sigma^{-2}\sigma^{2}_{0})}}{\epsilon^{2}}.$ (10) $$

Then we have for all $n\geq n^{*}$

$$ $S_{n}\supseteq{\mathcal{R}}^{\varepsilon}_{H}({\mathcal{S}}_{0}).$ $$

Moreover, we have
for all $n\geq n^{*}$, ${\bm{\pi}}\in{\mathcal{R}}^{\varepsilon}_{H}({\mathcal{S}}_{0})$ that
$|J_{c}({\bm{\pi}},{\bm{f}})-J_{c}({\bm{\pi}},{\bm{f}}^{*})|\leq\epsilon$ and $|J_{r}({\bm{\pi}},{\bm{f}})-J_{r}({\bm{\pi}},{\bm{f}}^{*})|\leq\epsilon$.

###### Proof.

Note that for any $n$, $N_{n}$ is defined as the smallest integer satisfying:

$$ $\displaystyle\frac{N_{n}}{\gamma_{n+N_{n}-1}(k)\beta^{4}_{n+N_{n}-1}(\delta)} \geq\frac{T^{6}C^{4}\frac{d_{s}\sigma^{2}_{0}}{\log(1+\sigma^{-2}\sigma^{2}_{0 })}}{\epsilon^{2}}.$ $$

From Lemma A.10, for $n^{*}=(H+1)N_{n^{*}}$, we have for all $n\geq n^{*}$

$$ $S_{n}\supseteq{\mathcal{R}}^{\varepsilon}_{H}({\mathcal{S}}_{0}).$ $$

We show that the solution to Equation 5 satisfies this condition. Moreover, let $n^{*}=(H+1)N_{n^{*}}$

$$ $\displaystyle\frac{N_{n^{*}}}{\gamma_{n^{*}+N_{n^{*}}-1}(k)\beta^{4}_{n^{*}+N_ {n^{*}}-1}(\delta)}$ $\displaystyle=\frac{\frac{n^{*}}{H+1}}{\gamma_{n^{*}+\frac{n^{*}}{H+1}-1}(k) \beta^{4}_{n^{*}+\frac{n^{*}}{H+1}-1}(\delta)}$ $\displaystyle\geq\frac{\frac{n^{*}}{H+1}}{\gamma_{n^{*}}(k)\beta^{4}_{n^{*}}( \delta)}$ $$

Picking $n^{*}$ as the smallest integer satisfying

$$ $\frac{n^{*}}{\gamma_{n^{*}}(k)\beta^{4}_{n^{*}}(\delta)}\geq\frac{(H+1)T^{6}C^ {4}\frac{d_{s}\sigma^{2}_{0}}{\log(1+\sigma^{-2}\sigma^{2}_{0})}}{\epsilon^{2}},$ $$

ensures that

$$ $\displaystyle\frac{N_{n^{*}}}{\gamma_{{n^{*}}+N_{n^{*}}-1}(k)\beta^{4}_{{n^{*} }+N_{n^{*}}-1}(\delta)}\geq\frac{T^{6}C^{4}\frac{d_{s}\sigma^{2}_{0}}{\log(1+ \sigma^{-2}\sigma^{2}_{0})}}{\epsilon^{2}}$ $$

Finally, from Lemma A.8 we have that $S_{HN_{n^{*}}}\supseteq{\mathcal{R}}^{\varepsilon}_{H}({\mathcal{S}}_{0})$.

Therefore, for all $n\geq n^{*}$,
${\bm{\pi}}\in{\mathcal{R}}^{\varepsilon}_{H}({\mathcal{S}}_{0})$, ${\bm{f}}\in{\mathcal{M}}_{n^{*}+N_{n^{*}}-1}$ with probability at least $1-\delta$

$$ $|J_{c}({\bm{\pi}},{\bm{f}})-J_{c}({\bm{\pi}},{\bm{f}}^{*})|\leq\epsilon,\; \text{and,}\;|J_{r}({\bm{\pi}},{\bm{f}})-J_{r}({\bm{\pi}},{\bm{f}}^{*})|\leq\epsilon.$ $$

∎

###### Proof of Theorem 4.8 .

We prove in Corollary A.5, that all policies in ${\mathcal{S}}_{n}$ are safe for all $n\geq 0$.
ActSafe is safe since it picks policies only from ${\mathcal{S}}_{n}$.

For optimality, we showed in Lemma A.10 for all $n\geq n^{*}$ that $S_{n}\supseteq{\mathcal{R}}^{\varepsilon}_{H}({\mathcal{S}}_{0})$.
Moreover, we have $\forall{\bm{\pi}}\in{\mathcal{R}}^{\varepsilon}_{H}({\mathcal{S}}_{0})$, ${\bm{f}}\in{\mathcal{M}}_{n^{*}+N_{n^{*}}-1}\supseteq{\mathcal{M}}_{n}$,
$|J_{r}({\bm{\pi}},{\bm{f}})-J_{r}({\bm{\pi}},{\bm{f}}^{*})|\leq\epsilon$.
Let ${\bm{\pi}}^{*}$ be the optimal policy and let $\tilde{{\bm{\pi}}}_{n}$ denote the solution to $\arg\max_{{\bm{\pi}}\in{\mathcal{S}}_{n}}\min_{{\bm{f}}\in{\mathcal{M}}_{n}}J_
{r}({\bm{\pi}},{\bm{f}})$.
For the sake of contradiction, assume that

$$ $J_{r}(\tilde{{\bm{\pi}}}_{n})<\max_{{\bm{\pi}}\in{\mathcal{R}}^{\epsilon}_{H}( {\mathcal{S}}_{0})}J_{r}({\bm{\pi}},{\bm{f}}^{*})-\epsilon.$ (11) $$

Furthermore, let $P^{r}_{n}({\bm{\pi}})=\min_{{\bm{f}}\in{\mathcal{M}}_{n}}J_{r}({\bm{\pi}},{\bm
{f}})$ for all ${\bm{\pi}}\in\Pi$.

$$ $\displaystyle P^{r}_{n}({\bm{\pi}}^{*})$ $\displaystyle\leq\max_{{\bm{\pi}}\in{\mathcal{S}}_{n}}P^{r}_{n}({\bm{\pi}})$ $\displaystyle=P^{r}_{n}(\tilde{{\bm{\pi}}}_{n})$ $\displaystyle\leq J_{r}(\tilde{{\bm{\pi}}}_{n})$ $\displaystyle<J_{r}({\bm{\pi}}^{*},{\bm{f}}^{*})-\epsilon$ (contradiction assumption) $\displaystyle\leq P^{r}_{n}({\bm{\pi}}^{*}).$ ( Lemma A.11 ) $$

This is a contradiction, which completes the proof.

∎

###### Corollary A.1 .

###### Proof.

###### Lemma A.2 (Difference in Policy performance, Sukhija et al. ( 2024 ) ) .

###### Proof.

###### Lemma A.3 (Comparing safety costs of policies) .

###### Proof.

###### Lemma A.4 .

###### Proof.

###### Corollary A.5 (All policies in 𝒮 n subscript 𝒮 𝑛 {\mathcal{S}}_{n} caligraphic_S start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT are safe) .

###### Proof.

###### Lemma A.6 .

###### Proof.

###### Lemma A.7 .

###### Proof.

###### Lemma A.8 .

###### Proof.

###### Corollary A.9 .

###### Proof.

###### Lemma A.10 .

###### Proof.

###### Lemma A.11 .

###### Proof.

###### Proof of Theorem 4.8 .

## Appendix B Experiment Details

### B.1 GP Experiments

For the GP experiments, we approximate Equation 7 with the following unconstrained optimization problem.

$$ $\displaystyle\operatorname*{arg\,max}_{{\bm{\pi}}\in\Pi}\max_{{\bm{f}}\in{ \mathcal{Q}}_{n}}J_{n}({\bm{\pi}},{\bm{f}})-\lambda\max\left\{\max_{{\bm{f}}^{ \prime}\in{\mathcal{Q}}_{n}}J_{c}({\bm{\pi}},{\bm{f}}^{\prime})-d,0\right\}.$ (12) $$

Here $\lambda$ is a (large) penalty term that is used to discourage constraint violation.
We use the iCEM (Pinneri et al., ) optimizer to solve the constrained optimization above.
Effectively, given a sequence of actions $\{{\bm{a}}_{t}\}^{H}_{t=0}$, we roll them out on our learned GP model using the TS1 approach from Chua et al. ().
Moreover, we maintain $P$ particles, and given the state $({\bm{s}}^{p}_{t},{\bm{a}}^{p}_{t})$ for the $p$-th particle, we determine the next state ${\bm{s}}^{p}_{t+1}$, by sampling from ${\mathcal{N}}({\bm{\mu}}_{n}({\bm{s}}^{p}_{t},{\bm{a}}^{p}_{t}),{\bm{\sigma}}_
{n}({\bm{s}}^{p}_{t},{\bm{a}}^{p}_{t}))$. Accordingly, for each action sequence $\{{\bm{a}}_{t}\}^{H}_{t=0}$, we obtain $P$ trajectories and we empirically solve $\max_{{\bm{f}}^{\prime}\in{\mathcal{Q}}_{n}}J_{c}({\bm{\pi}},{\bm{f}}^{\prime})$ by taking the max over the $P$ trajectories. This approach is also proposed by Kakade et al. () as a heuristic for optimizing over the dynamics.

##### Rewards and constraints

The reward function is designed to penalize deviations in both the angular position and the control input from the target behavior. For both the Pendulum and Cartpole, the state of the pole can be defined as follows. Let $\theta$ be the current angle, $\omega$ the angular velocity, and $u$ the control input. The target angle is denoted as $\theta_{\text{target}}$, and the angular error between the current angle and the target angle is $\Delta\theta$. The reward and cost functions for the Pendulum environment are given by

$$ $r_{\text{Pendulum}}=-\left(\Delta\theta^{2}+0.1\cdot\omega^{2}+0.02\cdot u^{2} \right),\quad c_{\text{Pendulum}}=\max\{|\omega|-6.0,0.0\},d=0.0.$ $$

For the Cartpole environment, the position and velocity of the slider are defined as $p$ and $v$ respectively.
The reward for the Cartpole environment is the given by

$$ $r_{\text{Cartpole}}=-\left(\Delta\theta^{2}+p^{2}+0.1\cdot\left(v^{2}+\omega^{ 2}\right)\right)-0.01\cdot u^{2},\quad c_{\text{Cartpole}}=\max\{|p|-0.5,0.0\} ,d=0.75.$ $$

### B.2 Vision Control Experiments

We provide an open-source implementation of our experiments in [https://anonymous.4open.science/r/actsafe-CF2C/](https://anonymous.4open.science/r/actsafe-CF2C/). We encourage readers to use it, as it contains additional important implementation details.

##### Approximating Equation 7

We solve the constraint optimization problem in Equation 7 using the LBSGD solver from Usmanova et al. ().
LBSGD is a first-order optimizer that
uses a logarithmic barrier function to enforce constraint satisfaction. Previous works from Ni & Kamgarpour (); As et al. () have successfully applied LBSGD for planning in model-based RL with CMDPs, showing notably fewer constraint violations than alternative solvers like the augmented Lagrangian method (As et al., ).

To approximate Equation 7, we maintain an RSSM ensemble of $P$ particles and given the state action pair $({\bm{s}}_{t},\pi_{n}({\bm{a}}_{t}|{\bm{s}}_{t}))$, we obtain $P$ estimates $\{{\bm{s}}^{p}_{t+1}\}_{p=1}^{P}$ for the next state. We estimate ${\bm{\sigma}}^{2}_{n}$ with the variance/disagreement between the ensemble members, i.e., $\text{Var}\left(\{{\bm{s}}^{p}_{t+1}\}_{p=1}^{P}\right)$. We obtain the next state ${\bm{s}}_{t+1}$ by uniform sampling from $\{{\bm{s}}^{p}_{t+1}\}_{p=1}^{P}$, i.e., TS1 from Chua et al. ().
Akin to Yu et al. (), we approximate $\max_{{\bm{f}}^{\prime}\in{\mathcal{Q}}_{n}}J_{c}({\bm{\pi}},{\bm{f}}^{\prime})$ by penalizing the cost function with ${\bm{\sigma}}_{n}$

$$ $J_{c-\lambda{\bm{\sigma}}}({\bm{\pi}}_{n})=\mathbb{E}_{{\bm{\pi}}_{n}}\left[ \sum^{H}_{t=0}\gamma^{t}(c({\bm{s}}_{t},{\bm{a}}_{t})+\lambda\norm{{\bm{\sigma }}_{n}({\bm{s}}_{t},{\bm{a}}_{t})})\right],$ $$

where $\lambda$ is a pessimism parameter. Yu et al. () show that for an appropriate choice of $\lambda$, $J_{c-\lambda{\bm{\sigma}}}({\bm{\pi}}_{n})$ is indeed a pessimistic estimate of $J_{c}({\bm{\pi}}_{n})$. However, in our experiments we treat $\lambda$ as a hyper-parameter.

##### Safety experiments

We focus on Safety-Gym to showcase our practical algorithm design maintains constraint satisfaction during learning. Our experiments rely on a newer fork of Safety-Gym which is available via our open-source code. We follow the experimental setup of Ray et al. (); As et al. () and an episode length of $T=1000$. We set the cost budget for each episode to $d=25$ for Safety-Gym (see Ray et al., ). After each training epoch we estimate $J_{r}({\bm{\pi}}_{n})$ and $J_{c}({\bm{\pi}}_{n})$ by sampling 50 episodes, denoting the estimates with $\hat{J}_{r}$ and $\hat{J}_{c}$. Unless specified otherwise, in all our experiments we use 5 random seeds and report the median and standard error across these seeds. Finally, we use a budget of 5M training steps for each training run. To make a fair comparison with As et al. (); Huang et al. (), we fix the ratio of environment steps and update steps of the model and policy. While Huang et al. () use the RSSM model from Hafner et al. (), our implementation uses the (older) one from Hafner et al. () and As et al. ().

##### Sparse Safety-Gym

Let $d_{t}^{\text{RG}}$ be the euclidean distance between the robot and the goal/button at time step $t$, $d_{t}^{\text{BG}}$ the distance between the box and the goal position and $d_{t}^{\text{RB}}$ the distance between the robot and the box positions. Furthermore, denote $\text{tol}(x,l,u)$ as the tolerance function from Tassa et al. (), where $l,u$ denotes lower and upper bounds respectively.

**Table 1: Comparison of the reward functions in the base environments of Safety-Gym and our sparse rewards environments.**
| Environment | Dense Reward | Sparse Reward |
| --- | --- | --- |
| GotoGoal | $d_{t-1}^{\text{RG}}-d_{t}^{\text{RG}}+\bm{1}_{d_{t}^{\text{BG}}\leq 0.3}$ | $\text{tol}(d_{t}^{\text{RG}},0,0.45)\cdot(d_{t-1}^{\text{RG}}-d_{t}^{\text{RG}<br>})+\bm{1}_{d_{t}^{\text{RG}}\leq 0.3}$ |
| PressButton | $d_{t-1}^{\text{RG}}-d_{t}^{\text{RG}}+\bm{1}_{d_{t}^{\text{BG}}\leq 0.3}$ | $\bm{1}_{d_{t}^{\text{RG}}\leq 0.1}$ |
| PushBox | $d_{t-1}^{\text{RB}}-d_{t}^{\text{RB}}+d_{t-1}^{\text{BG}}-d_{t}^{\text{BG}}+<br>\bm{1}_{d_{t}^{\text{BG}}\leq 0.3}$ | $\text{tol}(d_{t}^{\text{RB}},0,0.5)\cdot(d_{t-1}^{\text{RB}}-d_{t}^{\text{RB}}<br>)+d_{t-1}^{\text{BG}}-d_{t}^{\text{BG}}+\bm{1}_{d_{t}^{\text{BG}}\leq 0.3}$ |

##### Cartpole exploration

Figure: Figure 7: Cartpole environment as an example of a problem instance of safe exploration. Each scene summarizes a trajectory that was collected in increasing training iterations. The agent incurs a cost whenever the cart goes outside of the area between the two red vertical lines. The goal is to learn a policy that swings the pole to the top position, while ensuring the expected accumulated cost is bounded *during learning*. Learning in this setting is much more challenging, as agents can only try out control policies that known to be safe.
Refer to caption: extracted/5921425/figures/cartpole-combined.png

In this task, the agent receives a sparse reward when it swings up a pendulum to the top position and when the slider (a.k.a cart) is centered. The RWRL benchmark (Dulac-Arnold et al., ) adds a safety constraint that enforces the slider to remain in a certain distance from the center (see Figure 7). As in Dulac-Arnold et al. (), we use a cost budget of $d=100$ and an episode length of $T=1000$ steps. Adding the safety constraint adds a significant challenge, as any safe policy is much more limited in exploration. In addition to the safety constraint, we add a cost for taking actions, as done in Curi et al. (). Combining all these factors together, makes a challenging exploration task, as we show in our experiments. Further implementation details can be found in our open-source code.

## Appendix C Additional Experiments

##### Experiments with the Doggo Robot

In this experiment we compare ActSafe with the same baseline algorithms from Section 5 on Safety-Gym’s Doggo robot. We omit our results in the PushBox environment as all baselines failed to solve it.

Figure: Figure 8: Performance and safety in with the Doggo robot.
Refer to caption: x7.png

As shown in Figure 8, similarly to the results in Section 5, ActSafe maintains safety during learning, while moderately underperforming LAMBDA. Overall, ActSafe outperforms CPO both in terms of safety and performance and BSRP-Lag of Huang et al. () in terms of performance.

##### Ablating LBSGD

One assumptions of LBSGD that we cannot formally satisfy relates to unbiasedness of the evaluation of the objective, constraints and their gradients. In principle, satisfying this assumption will allow us to guarantee that all iterates of Equation 7 are feasible, i.e., satisfy the pessimistic constraint.
This is in contrast to primal-dual methods, such as the Augmented Lagrangian of As et al. () that lacks any guarantees on feasibility during optimization. While it is hard to formally satisfy LBSGD’s unbiasedness assumption, we empirically observe that LBSGD allows us to keep constraint satisfaction during learning. We present this result in Figure 9.

Figure: Figure 9: Augmented Lagrangian from As et al. () compared to LBSGD of Usmanova et al. (). LBSGD significantly reduces the number of unsafe episodes.
Refer to caption: x8.png

As shown, even after initializing both variants with offline data, ActSafe[Lagrangian] fails to satisfy the constraints throughout learning. As in the main results on safety in Figure 4, compared to Augmented Lagrangian, LBSGD maintains safety during learning at a slight price of performance.

##### Safe Adaptation

Here, instead of initially providing offline data, we study the effect of first training on a “safe” environment, like a simulator, and then continuing training on a similar environment, but with shifted dynamics. To this end, we extend GotoGoal from Safety-Gym to two additional tasks, in which we change the motor gear and floor damping coefficients. The agent is first allowed to explore the “sim” environment for 300K interaction steps before being deployed on the “real” environment. We analyze the impact of our LBSGD optimizer and of pessimism in handling constraint violation during deployment.
As shown in Figure 10, without LBSGD and pessimism, ActSafe does not always transfer safely to the deployment environment. Furthermore, intuitively, while pessimism is crucial for maintaining safety while adapting to distribution shifts, it may sometimes hinder performance of the main objective. This experiment demonstrates that, if one has no offline data, one can use ActSafe in combination with a simulator to achieve safe exploration in practice, with a clear tradeoff of the simulator’s fidelity and the degree of pessimism in ActSafe.

Figure: Figure 10: Adaptation to domain shifts. The red dashed vertical line represents the step after which we switch dynamics. Black dashed vertical line represents changing from active exploration to greedily maximizing the reward. We report the mean metrics across 5 seeds.
Refer to caption: x9.png

##### Humanoid Proof-of-Concept

We further demonstrate the scalability of ActSafe on the HumanoidBench benchmark (Sferrazza et al., ).

Figure: (a) Pole task of HumanoidBench. The robot has to cross to the other side of the maze while avoiding hitting the poles.
Refer to caption: extracted/5921425/figures/pole.png

We use a robust, low-level walking policy provided with the benchmark, and input visual observations from a third-person camera view. We compare ActSafe with Opax (Sukhija et al., ) on the Pole task, where a humanoid robot must navigate through a field of pole obstacles, as illustrated in Figure 11. In this task, the agent incurs a cost of 1 for each pole it hits and when it falls, while the reward is based on the robot’s forward velocity. As shown in Figure 11, ActSafe significantly reduces the number of constraint violations compared to Opax, while maintaining competitive performance on the objective.

## Peer Reviews

confidence: 3
contribution: 4
initial_rating: 8
presentation: 3
questions: - The paper mentions the use of offline-collected data comprising 200K environment steps from a random policy. Can the authors clarify how this offline dataset is incorporated into ACTSAFE? Also, are all baseline algorithms utilizing this offline dataset consistently? If not, it may introduce unfairness in the comparisons.
soundness: 3
strengths: - Theoretical Rigor: The paper offers a thorough theoretical framework for ACTSAFE, deriving safety guarantees and sample-complexity bounds for safe exploration in continuous state-action spaces—a notable contribution in the safe RL domain.
Scalability in Practical Settings: The authors extend ACTSAFE to visual control tasks, demonstrating scalability beyond low-dimensional models. This practical application is a significant step towards bridging the gap between theoretical safe RL algorithms and real-world, high-dimensional deep RL applications.

- Comprehensive Experimental Analysis: The experimental analysis is extensive, covering both theoretical settings (under the Gaussian process assumptions) and more complex visual motor control tasks in the SAFETY-GYM environment, as well as sparse reward exploration. This breadth of evaluation provides valuable insights into the algorithm's performance under varied conditions.
summary: The paper introduces ACTSAFE, a model-based reinforcement learning (RL) algorithm designed for safe exploration within continuous state-action spaces. ACTSAFE utilizes epistemic uncertainty in the model as an intrinsic reward mechanism to encourage safe set expansion during exploration. The authors provide theoretical analysis that includes safety and sample-complexity guarantees, suggesting that ACTSAFE achieves near-optimal policy convergence within finite episodes. Additionally, a practical implementation of ACTSAFE is proposed, enabling the method to scale to high-dimensional tasks, including visual control. Empirical evaluations demonstrate ACTSAFE’s performance across various benchmarks, supporting both safe exploration and strong task performance.
weaknesses: - Reliance on Assumptions: The theoretical guarantees rely on idealized assumptions (e.g., well-calibrated Gaussian processes and specific Lipschitz conditions), which may limit generalizability. However, the authors provide strong empirical evidence that ACTSAFE performs well even when these assumptions are not strictly met, somewhat mitigating this concern.

---

confidence: 2
contribution: 3
initial_rating: 8
presentation: 3
questions: Safe RL is not my area of expertise, and I therefore ask the metareviewer to discount my (unfortunately brief) review accordingly.
soundness: 3
strengths: The paper is very well written. The high level idea of the algorithm is presented in an understandable way, without sacrificing mathematical rigour. By combining existing ideas, like intrinsic motivation or safe Expansion operators, the authors seem to have created a conceptually simple but very powerful algorithm for safe RL. 

Based on a strong theoretical foundation a practical implementation is provided, that crucially maintains the safety constraints. The experiments seem well designed, highlighting the clear strengths of the algorithm (its safety guarantees), while also discussing weaknesses. The authors especially discuss scalability weaknesses that may result from the choice of Gaussian Processes to approximate the system dynamics.
summary: The authors present a model-based algorithm, called ActSafe, that has strong theoretical safety guarantees.
weaknesses: While the two-phase approach that ActSafe employs are the foundation for its safety guarantees, I would expect that this comes at a cost. A comparison of total environment steps in both loops required, wall clock or memory requirements would have been a nice addition.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the list of weakness above, 

- Can the authors explain the disconnect between the motivations in the paper and the experiments? 

- In lines 233-235, it is mentioned that prior works do not have dimensional policies, but there is no explicit clarification about how high dimensional control tasks can the proposed approach tackle.  What are the relative differences? Are there any fundamental assumptions that prevent these prior works from being applicable to higher dimensional tasks?

- It is unclear why there are no evaluations in a model-free setting, and what are the motivations for just considering model-based RL, after proposing a safe exploration algorithm that seems to be fairly generic and broadly applicable. It will be helpful to clarify these points. 

- In lines 225-226 it seems that in the general case the epistemic uncertainty will be uncalibrated / unreliable due to function approximation errors in the neural network model. Does the theory account for this?

- Can the authors clarify relations to the missing related works, including very related safe exploration works that actually show results in real-world settings?
soundness: 3
strengths: - The paper targets a problem that is relevant to a broad community including controls, reinforcement learning, and robotics. Safe exploration in the context of model-based control is interesting because a reliable approach in this space has the potential for being deployed in safety critical scenarios as well as in scenarios that require sample efficiency and real-world exploration is challenging. 

- The proposed algorithm based on intrinsic exploration for reducing uncertainty of policies at the boundary of the current safe set and expanding the safe set based on reachability beyond it, is novel to the best of my knowledge. It is also intuitively sound and doesn't make any fundamentally limiting assumptions of the underlying system. 

- The theoretical guarantees of the base algorithm are strong, and the regularity assumptions seem reasonable to me. 

- The experiments on simulated environments are good, and it is nice to see results in challenging control tasks like a simulated humanoid walking, that go beyond the predominant evaluations of prior safe RL works on simple simulations like SafetyGym.
summary: This paper tackles the problem of safe exploration in reinforcement learning by proposing a new model-based constrained RL algorithm. Under the constrained MDP formulation, with regularity assumptions on the underlying dynamical system, the paper demonstrates theoretical safety guarantees and sample complexity guarantees. A practical version of the algorithm with weaker guarantees is shown to be applicable for visual control tasks in different simulated benchmarks. The results show less safety violations compared to prior constrained RL approaches.
weaknesses: - A major weakness is that the paper's motivations are disconnected from the experiments. For example the intro states: 

"In many real-world settings, environments are complex and rarely align exactly with the assumptions
made in simulators. Learning directly in the real world allows RL systems to close the sim-to-real
gap and continuously adapt to evolving environments and distribution shifts. However, to unlock
these advantages, RL algorithms must be sample-efficient and ensure safety throughout the learning
process to avoid costly failures or risks in high-stakes applications." 

However the experiments are all in simulated environments and there are no real-world experiments either in a controls setting or in a robotics setting.

 
- It is unclear how the model-based version of ActSafe can be reliably be deployed in the real-world for safety-critical applications. Both theoretically and empirically in the simulated experiments, the constraint violations are lower than the baselines but nowhere close to 0. In addition learning a recurrent state-space model is a more complex choice than directly trying to learn a policy through model-free RL. It is unclear why there are no evaluations in a model-free setting, and what are the motivations for just considering model-based RL, after proposing a safe exploration algorithm that seems to be fairly generic and broadly applicable. It will be helpful to clarify these points. 

- In lines 233-235, it is mentioned that prior works do not have dimensional policies, but there is no explicit clarification about how high dimensional control tasks can the proposed approach tackle.  What are the relative differences? Are there any fundamental assumptions that prevent these prior works from being applicable to higher dimensional tasks?


- The related works seem to miss a lot of prior approaches that also have similar ideas of conservative exploration  and actually demonstrate results on real-world settings where safety in important. For example, see [A-C] below. [A] in particular demonstrates results on real robotic manipulation tasks where safety is critical. 

[A] Thananjeyan, Brijen, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E. Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. "Recovery rl: Safe reinforcement learning with learned recovery zones." IEEE Robotics and Automation Letters (RA-L)

[B] Bharadhwaj, Homanga, Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Animesh Garg. "Conservative safety critics for exploration." ICLR 2021

[C] Srinivasan, Krishnan, Benjamin Eysenbach, Sehoon Ha, Jie Tan, and Chelsea Finn. "Learning to be safe: Deep rl with a safety critic."

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. It would be great if the author could address the main weaknesses I have outlined above. If they are properly addressed, I would be happy to raise my score, as I may have misunderstood the paper.
2. Considering that there are no hard constraints on safety during learning in practical implementations, can ActSafe maintain safety during training after removing the initial offline data?
3. I am very familiar with BSRP-Lag, but the algorithm's performance in the PUSHBOX task doesn't quite match my experience, can you disclose more experimental details?
soundness: 3
strengths: 1. The problem is well-motivated.
2. This paper is mostly well-written and has solid theoretical foundations. The code provided is very helpful for the reproducibility and review of this paper.
3. The empirical results in sparse reward tasks are impressive.
summary: This paper introduces ActSafe, a model-based algorithm for addressing the challenge of safe and efficient exploration in continuous state-action spaces, and provides a detailed theoretical analysis of it. This algorithm integrates OPAX and LBSGD with typical model-based RL methods to achieve safe and efficient exploration performance in vision control tasks. They conduct sufficient experiments in Safety-Gym domain, demonstrating that their approach outperforms prior works regarding safe exploration.
weaknesses: 1. The proposed algorithm ActSafe, is a combination of  OPAX, LBSGD, and Dreamer(or other model-based RL methods). The technical contributions are incremental.   But considering the experiments are solid and sound, that should be fine.
2. Inconsistency. This algorithm addresses the problem of safe exploration by maintaining a 'safe set' of policies. However, after checking the code provided, there is no such concept of a 'safe set', they just optimize the actor with the typical safe actor-critic method. Since the theoretical results might be unconvincing because of the inconsistency between the theory and practical implementation, it would be nice if the authors could explain this.
3. Baselines.  In Figure 5, the author compares ActSafe with existing methods like OPTIMISTIC、UNIFORM and GREEDY. Are there any other more competitive baselines?  Comparison with methods like GoSafeOpt、OPAX might be necessary.

Meta-review: This paper proposes a model-based approach for safe RL, which has an expansion stage and an exploitation stage. The overall algorithm appears novel and the theoretical analysis is solid. The empirical study also involves challenging vision-based control tasks. So I recommend accept. That being said, I do hope to see a few changes in the camera ready version. First, I agree that hardware experiments are not necessary and it's fine to use real world problems as motivation. But the authors need acknowledge that learning a good model in most real-world problems is extremely challenging. Second, Figure 4 now has only 5 seeds. It needs to have at least 10 seeds and it's better to report mean and standard errors. SE characterizes the concentration of the empirical average and does not match with median. Third, the discussion of related works should be expanded, e.g., for "these work do not ground their methods on theory", it is necessary to make it absolutely clear what kind of theories they have and what kind of theories they do not have.

Decision: Accept (Poster)