ActSafe: Active Exploration with Safety Constraints for Reinforcement Learning
Authors: Yarden As
Bhavya Sukhija
Lenart Treven
Carmelo Sferrazza
Stelian Coros
Andreas Krause
Venue: ICLR 2025 Poster (2025)
Keywords: Safe Exploration
Constrained Markov Decision Processes
Safe Reinforcement Learning
Primary area: reinforcement learning
TL;DR: We propose a novel model-based algorithm for safe exploration in continuous state-action spaces and derive theoretical guarantees for it. We design a practical variant of our algorithm that scales well to vision control tasks.

Abstract: Reinforcement learning (RL) is ubiquitous in the development of modern AI systems. However, state-of-the-art RL agents require extensive, and potentially
unsafe, interactions with their environments to learn effectively. These limitations
confine RL agents to simulated environments, hindering their ability to learn
directly in real-world settings. In this work, we present ActSafe, a novel
model-based RL algorithm for safe and efficient exploration. ActSafe learns
a well-calibrated probabilistic model of the system and plans optimistically
w.r.t. the epistemic uncertainty about the unknown dynamics, while enforcing
pessimism w.r.t. the safety constraints. Under regularity assumptions on the
constraints and dynamics, we show that ActSafe guarantees safety during
learning while also obtaining a near-optimal policy in finite time. In addition, we
propose a practical variant of ActSafe that builds on latest model-based RL advancements and enables safe exploration even in high-dimensional settings such
as visual control. We empirically show that ActSafe obtains state-of-the-art
performance in difficult exploration tasks on standard safe deep RL benchmarks
while ensuring safety during learning.

Published as a conference paper at ICLR 2025
ACTSAFE: ACTIVE EXPLORATION WITH SAFETY CON- STRAINTS FOR REINFORCEMENT LEARNING
Yarden As, Bhavya Sukhija" ETH Zürich
Lenart Treven ETH Zürrch
Carmelo Sferrazza UC Berkeley
Stelian Coros
Andreas Krause ETH Zürich
ETH Zürich
ABSTRACT
Reinforcement learning (RL) is ubiquitous in the development of modern Al sys- tems. However, state-of-the-an art RL agents require extensive, and potentially unsafe, interactions with their environments to learn effectively. These limitations confine RL agents to simulated environments, hindering their ability to learn directly in real-world settings. In this work, we present ACTSAFE, a novel model-based RL algorithm for safe and efficient exploration. ACTSAFE learns well-calibrated probabilistic model of the system and plans optimistically w.r.t. the epistemic uncertainty about the unknown dynamics, while enforcing pessimism w.r.t. the safety constraints. Under regularity assumptions we show that ACTSAFF guarantees safety during learning while also obtaining near-optimal policy in finite time. In addition, propose a practical variant of ACTSAFE that builds on latest model-based RL advancements and enables safe ex- ploration even in high-dimensional settings such as visual control. We empirically show that ACTSAFE obtains state-of-the-al performance in difficult exploration tasks on standard safe deep RL benchmarks while ensuring safety during learning.
the constraints and dynamics,
INTRODUCTION
Reinforcement learning (RL) a powerful paradigm for sequential decision-making under uncertainty, with many applications in games (Mnih et al., 2015; Silver et al., 2016), recommender systems (Maystre et al., 2023), nuclear fusion control (Degrave et al., 2022), data-center cooling (Lazic et al., 2018) and robotics (Lee et al., 2020; Brohan et al., 2022; Cheng et al., 2024). Despite the notable progress, its application without any use of simulators remains largely limited. This is primarily because, in many cases, RL. methods require massive amounts of data for learning while also being inherently unsafe during exploration.
In many real-world settings, environments are complex and rarely align exactly with the assumptions made in simulators. Learning directly in the real world allows RI. systems to close the sim-to-real gap and continuously adapt to evolving environments and distribution shifts. However, to unlock these advantages, RL algorithms must be sample-efficient and ensure safety throughout the learning process avoid costly failures risks in high-stakes applications. For instance, agents learning driving policies autonomous vehicles must prevent collisions with other cars or pedestrians, even when adapting to new driving environments. This challenge is known safe exploration, where the agent's 's exploration is restricted by safety-critical, often unknown, constraints that must be satisfied throughout the learning process.
Several works study safe exploration and have demonstrated state-of-the-art performance in terms of both safety and sample efficiency for learning in the real world (Sui al., 2015; Wischnewski et al., 2019; Berkenkamp et al., 2021; Cooper & Netoff, 2022; Sukhija et al., 2023; Widmer et al., 2023). These methods maintain a "safe set" of policies during learning, selecting policies from this set to safely explore and gradually expand it. Under common regularity assumptions about the constraints, these approaches guarantee safety throughout learning. However, explicitly maintaining and expanding a safe set, limits these methods to oiw-dimesiional policies, such as PID controllers. This makes them difficult to scale to more complex tasks such as those considered in deep RL.
"Equal contribution. Correspondence to: cardas@etth..ch

---

Published as a conference paper at ICLR 2025
The goal of this work is to address this gap. To this end, we propose scalable model-based RL algorithm - ACTSAFE- for efficient and safe exploration. Crucially, ACTSAFE learns an uncertainty- aware dynamics model, which it uses to implicitly define and expand the safe set of policies. We theoretically show that ACTSAFF. ensures safety throughout learning and converges to a near-optimal policy within finite number of episodes. Moreover, ACTSAFF, is practical and integrates seamlessly with state-of-the-ar dynamics modeling techniques, for instance Dreamer (Hafner et al., 2023), delivering strong empirical performance. Thus, ACTSAFE advances the frontier of safe RL methods, both in theory and practice. Our main contributions are summarized below.
Contributions
We propose ACTSAFE, novel model-based RL algorithm for safe exploration in continuous state-action spaces. ACTSAFE maintains pessimistic set of safe policies and optimistically selects policies within this set that yield trajectories with the largest model epistemic uncertainty.
We show that when the dynamics lie in reproducing kernel Hilbert space (RKHS), ACTSAFE guarantees safe exploration. addition, we provide a sample- sample-complest complexity bound for ACTS illustrating that ACTSAFE obtains e-optimal policies in a finite number of episodes. To the best of our knowledge, we are the first to show safety and finite sample complexity for safe exploration in model-based RI. with continuous state-action spaces.
In our experiments, we demonstrate that ACTSAFE, when combined with a Gaussian process dynamies model, achieves efficient and safe exploration. Additionally, we show that ACTS AFE seales to high-dimensional environments of the SAFETY-GYM and RWRL benchmarks, excelling in challenging exploration tasks with visual control while also incurring significantly fewer constraint violations.
2 RELATED WORKS
Constrained Markov decision processes (CMDP) for safe RL Safety in reinforcement learning can be modeled in various ways (Garefa et al., 2015; Brunke et al., 2022). Constrained Markoy decision processes (CMDPs) serve natural option for this purpose, they encode unsafe behaviors through constraints and enjoy many classical results from planning in MDPs (Altman, 1999). Learning and planning in CMDPs have been extensively explored in the RL community, both theoretically and in practice. Notably, the works of Efroni et al. (2020); Vaswani et al. (2022); Ding et al. (2022); Müller et al. (2024) derive sample complexity bounds for CMDPs in discrete state-action spaces, whereas Achiam et al. (2017); Tessler et al. (2018); Stooke et al. (2020); Xu et al. (2021); Liu et al. (2022); As al. (2022); Sootla et al. (2022); Huang et al. (2024) develop deep RL. algorithms for CMDPs in continuous state-action spaces. However, all the aforementioned works relax the requirement of safe exploration and thus do not ensure the safety during learning. This is in contrast to this work, where we tackle the hard problem of safe exploration.
Provably safe exploration Turchetta et al. (2016); Wachi et al. (2018); Wachi & Sui (2020) focus on safe exploration in CMDPs with discrete state-action spaces and a constraint function that lies in an RKHS. Zheng& & Ratliff (2020) study sample complexity for safe exploration in discrete CMDPs. For continuous state-action spaces, Berkenkamp et al. (2021) and extensions thereof (Baumann et al., 2021; Sukhija et al., 2023; Hübotter al., 2024), leverage ideas from safe Bayesian optimization (Sui et al., 2015) to directly optimize over the policy parameters in a model-free manner. The proposed algorithms guarantee safe exploration and finite sample complexity for learning an e-optimal solution. When evaluated on real-world systems, these methods exhibit remarkable sample efficiency while also being safe during learning (Cooper & Netoff, 2022; Kirschner et al., 2019; Widmer et al., 2023). However, these approaches are limited to simple low-dimensional policies, e.g., PID controllers, and are hard to scale to policies with more than few parameters. In a similar spirit, Berkenkamp et al. (2017) propose model-basee RL algorithm for safe learning, where safety is modeled in terms of Lyapunov stability. Even though the method enjoys similar theoretical guarantees as Berkenkamp et al. (2021), it assumes access to a generative simulator and thus cannot be applied to traditional online RL settings. In contrast, Koller et al. (2018); Curi et al. (2022) propose more practical safe learning methods in combination with model-predictive control (MPC). While these methods guarantee safety during learning, they lack optimality guarantees and are computationally expensive to run in real-time.
A common aspect among most of the aforementioned methods is their use of an intrinsic objective. such the model epistemic uncertainty, guide and restrain exploration. Crucially, these methods
2

---

Published as a conference paper at ICLR 2025
maintain a safe set of policies which they gradually expand during learning by sampling policies that yield the highest intrinsic reward. In this work, we build on this key insight to propose a model-bbsee RL algorithm for online learning that enjoys the same kind of guarantees while also being applicable in real-world settings such as deep RI.
Safe exploration with deep RL A common approach to safe exploration is the of safety filters (Dalal et al., 2018; Wabersich & Zeilinger, 2021; Curi et al., 2022), which modify the actions produced by an unsafe policy meet safety constraints before they are executed on the real system. A key advantage of safety filters is that they can be easily added to any tofftthe-shelf" unsafe RL algorithm. However, while safety is ensured, safety filters can lead arbitrarily bad exploration and therefore lack guarantees for optimality. The works of Srinivasan et al. (2020); Thananjeyan et al. (2021) rely on learning safety crities that certificate state-actions as safe, either for policy optimization or during online data collection. These works provide strong empirical results, including demonstra- tions of safe policies on real hardware. In addition, following this approach, Bharadhwaj et al. (2020) upper bound the the probability of making infeasible policy updates. Another line of work, relies on guaranteeing feasibility of policy optimization algorithms. Notably, Chow et al. (2019) use Lyapunov functions to guarantee feasibility of policy gradients iterates and derive their analysis state-action spaces. Usmanova et al. (2024) propose Log-Barriers SGD (LBSGD), an optimization algorithm that ensures feasibility of all its iterates with barrier functions, showcasing its application in navigation tasks with image observations. More recently, As et al. (2024); Ni & Kamgarpour (2024) use LBSGD for safe learning with greedy policy gradients, i.c., without considering intrinsic rewards to expand the safe set of policies. Crucially, this form of greedy policy search may result in sub-optimal policies, as described in Section 4 and empirically shown Section 5.
discrete
PROBLEM SETTING
We consider discrete-time episodic, constrained Markov decision process (CMDP), where the goal is to find a policy that not only maximizes the reward but also keeps the accumulated costs below specified threshold, i.e., satisfies safety constraint. This type of formulation is common in real-world scenarios, such as robot navigation. this setting, the reward could represent the negative distance to a target destination, while the costs could represent penalties, such for each collision with an obstacle. The CMDP formulation allows us to separate these two objectives, thus ensuring constraint satisfaction and safety, for optimal policy. In this setup, we consider dynamical systems with additive noise and bounded running rewards and costs
cost of incurred
8tt1 = (s1, at) + wt. (St, az) E S X A, s(0) So r(s, E [0, Rmaax]
(Running reward)
(I)
c(s, a) E [0, Cmaa]]
(Running cost).
Here 8; E S C. Rd is the state, a, E A C Rd the control input, and w, E W C Rd the process noise. The dynamics f" are unknown and without loss of generality, the reward and cost c assumed to be known.
Task In this work, we study the following constrained RL problem (Altman, 1999)
T--1
max Jn f*) max E NO. well
ai) f (8, T., (8t) + w..
s.t. Je(TT, f") == Eso.
Σ c(s at)
S d: (2)
* Flll
8141
We study the episodic setting, with episodes n = I,..., N. At the beginning of the episode n, we deploy a policy π, (In,, nn.1.-.. ...,, -1), chosen from the policy space II for horizon of T on the system. Next, we obtain the trajectory Tn = (Sn.u...., S, which we add to a dataset of transitions D,, {(Zn,i (Sn,i, An,i (8n,()n,))) yn.i 8n.i4 105iec) and use the collected data to learn a model of f"
4 ACTSAFE: ACTIVE EXPLORATION WITH SAFETY CONSTRAINTS
key challenge in learning with safety constraints is ensuring that these constraints are not violated during exploration. In the following, we introduce an idealized version of ACTSAFE, which guar- antees safe exploration for dynamical systems with Gaussian process dynamics' 1 Moreover, we also
'hheee guarantees be extended to more general well-calibrated models as in Curi et al. (2020)
3

---

Published as a conference paper at ICLR 2025
provide bound on the sample complexity of ACTEAFE for learning e-optimal policy. To the best of our knowledge, this is the first model-based safe RL algorithm for continuous state-action spaces that provides guarantees for both safety and sample complexity. In Section 4.3, we discuss a practical vari- ant scaling to more complex domains. Our choice of a model-based approach is motivated by its supe- rior empirical sample efficiency (Chua et al., 2018; As et al., 2022) as well as our theoretical analysis.
4.1 ASSUMPTIONS
Theoretically studying safe exploration without any assumptions on the underlying dynamical system an ill-posed problem. In the following, we make some assumptions on the underlying problem that are common in the model-based RL (Curi et al., 2020; Kakade et al., 2020) and safe RL (Berkenkamp et al., 2021; Baumann et al., 2021) y literature.
Assumption 4.1 (Continuity of f and т). The dynamics model f is L, Lipsccht, the cost c is L. Lipsschhi,,,, and all क E II are continuous.
Assumption 4.2 (Process noise distribution). The process noise is i.i.d. Gaussian with variance o², i.e., we ~ (0,, (TT)
We focus the setting where w is homoscedastic for simplicity. However, our framework can also be applied to the more general heteroscedastic and sub-Gaussian case (Sukhija et al., 2024; Hübotter al., 2024).
Assumption 4.3 (Initial safe seed). We have access to an initial nonempty safe set Sn of policies, i.e., VT So : J< < d and So A 0.
This assumption is crucial since without any prior knowledge about the system, ensuring safety is unrealistic. Therefore, So allows us start the learning process by selecting policies from this set. In practice, this safe set could be obtained from a simulator or offline demonstration data.
assume that at each step we learn mean estimate 14, of f and can quantify our uncertainty on over the estimate. This allows us to learn an uncertainty-aware model of f", which is crucial for exploration and safety. More formally, learn a well-calibrated statistical model of f as defined below.
In the following,
Definition 4.4 (Well-calibrated statistical model of f*, Rothfuss et al. (2023)). Let Z - S A. An all-time well well-calibrated statistical model of the function f is a sequence (d) where
(o) def { Z Rdd Vz E Vj E (l,...,dg) do} } (M),)(2) - fz(2)] S 3, (8))
if, with probability at least 1 8, we have f E E (n) Here, fz. Mn.j and on.i denote the j-th element in the vector-valued functions f. l%n and a respectively, and (5) RRo is sequence of scalar functions that depends on the confidence level 8 E (0, 1] and is monotonically increasing in n.
Next, assume that f resides in Reproducing Kernel Hilbert Space (RKHS) of vector- valued functions and show that this is sufficient for us to obtain a well-calibrated model.
Assumption 4.5. We assume that the functions fj. j = {l..... , d lie in RKHS with kernel k and have a bounded norm B, that is f E HH,n. with H {f |1 < B.j = (1,.,d, d.)).
Moreover, we assume that k(z, x) S Tmax for all x E Z. Assumption 4.5 allows us to model f" with GPs for which the mean and epistemic uncertainty (2)(22== = (2)](())), and or, (2) = [on.; (z)l have an analytical formula (c.f., Equation (9) Appendix A).
Lemma 4.6 (Well calibrated confidence intervals for RKHS, Rothfuss et al. (2023)). Let f E HR Suppose bhd and o are the posterior mean and variance of a GP with kernel k after episode n. There exists 3, (6), for which the sequence (Mys T.,, (5)) represents a well-calibrated statistical model of f
In summary, Assumption 4.5 and Lemma 4.6 show that in the RKHS setting, a GP is a well-calibrated model. For more general models like Bayesian neural networks (BNNs), methods such as Kuleshov et al. (2018) can be used for calibration. Overall, our results can also be extended beyond the RKHS setting to other classes of well-calibrated models similar to Curi et al. (2020).
4.2 ACTSAFF: ALGORRITMMIC FRAMKW
A crucial element of safe exploration algorithms is the exploration-expansionn dilemma (Hübotter et al., 2024). In the following, we explain this in further detail, we then present a sketch of ACTSAFE and finally the formal algorithm.
4

---

Published as a conference paper at ICLR 2025
Algorithm 1 ACTSAFE: ACTIVE EXPLORATION WITH SAFETY CONSTRAINTS (Expansion stage)
Init: Aleatoric uncertainty o, Probability 8, Statistical model (Ho, ou. (u (u, for episode n = I,....," n do
n arg max max Byn. ETE I π ( Dn (OLLOUT(TH))
Prepare policy
Collect data
Update (Ma, Sn) f 115
Update statistical model and safe set
end for
Safe set expansion To ensure the safety of the agent during the initial phases of learning, ACT begins exploration by selecting policies from Sn (Assumption 4.3). This reduces uncertainty on about f for policies in So. allowing us to infer the safety of policies beyond So and expand the safe set (see Figure I). Safe set expansion is critical in safe RL because the optimal policy may lie outside the initial safe set, and expanding the safe set is necessary to reach Unlike traditional RL, where exploration focuses on maximizing reward, safe RL methods must also explore to expand the safe set. Methods like optimism and Thompson sampling, which focus on reward maximization, do not address this need for safe set expansion (Sui et al., 2015).
Algorithm Sketch ACTSAFE operates in two stages; (i) expansion by intrinsic exploration and (ii) exploitation of extrinsic reward. In the first stage, ACTSAFE uses the model epistemic uncertainty as an intrinsic reward pesplore (s, a) = 1 alll and selects policies within the safe set that yield trajectories with high uncertainties. This enables ACTSAFE efficiently reduce its uncertainty within the safe set and expand it. ACTS AFF performs the intrinsic exploration phase for a fixed number of episodes n' till the safe set is sufficiently large and then transitions to the second stage. In the exploitation stage, ACTSAFE greedily maximizes the extrinsic reward r, effectively aiming to solve the problem in Equation (2).
Most model-based safe RL methods (As et al., 2022) focus only on the second stage and ignore safe set expansion. In contrast, the theoretically grounded approaches of Berkenkamp al. (2021); Baumann et al. (2021); Sukhija et al. (2023); Hübotter et al. (2024) explicitly account for the expansion, but are not scalable to high dimensional policies typically considered in RL.
Next, we present our main algorithm. To ensure safety during learning, we maintain a conservative (pessimistic) estimate of the safe set which defined below.
Definition 4.7. Let Mn Ma- n Qn, Vn 2 1 denote the set of plausible models, and (T( = fax, Je(T,, f) our pessimistic estimate of the expected costs w.r.t. M,. Then, we define the safe set Sn as
def Sn = Sp-1 { E 11 Sn-1:3 E SS s.t. Pa(TT) + D(T, x²) S dd,
(3)
where
D(T, TT) =
T-1
L ((x((ss) - π ((t))
E ppm
min [Le || (st) TTs:))l (.s:)lllCmmall 20max + TC min
t=
Interpretation of Definition 4.7 function Je w.r.t. our model set M,, Equation (3) we define the expansion operator for the safe set. This operator adds new policies π that are not yet in the safe set, i.e., those in II Sp-1. to S, they are close to some policy T' from within the safe set. The distance D( æ²) measures how close the two policies are in terms of the underlying cost function, and it is similar to other distance metrics, such as the one in Foster et al. (2024, Theorem 2.1).
We maintain a pessimistic estimate, Pr, of the constraint value
The expansion operator is common in the safe BO and RL literature (Wischnewski et al., 2019; Fiducioso et al., 2019; Berkenkamp et al., 2021: Baumann et al., 2021; Cooper Netoff, 2022; Sukhija et al., 2023; Holzapfel et al., 2024; Fiedler al., 2024), and while it is generally difficult to evaluate in continuous spaces, it gives a key insight for safe RL methods: to effectively expand our knowledge of what safe, we have reduce our pessimism across policies in our safe set.
5

---

Published as a conference paper at ICLR 2025
II
(S,)
(So
(r
π
π
π
So
Figure I: Schematic illustration of the expansion process. We expand the safe set iteration n by reducing our uncertainty around policies at the boundary of S,-1. The pale blue area depicts the reachable R3 (Sa) after H expansion iterations. The arrow on the leftmost illustration demonstrates that without explicit expansion, finding the optimal policy x² is intractable.
Accordingly, during the expansion phase, the n-th episode, selects the policy nr that yields the high uncertainty about the underlying dynamics
the following objective for ACTSAFE, which, in
To fn arg max E & TES., FEM.
onn- T (8+))))
(F.f)
Furthermore, akin to Curi et al. (2020); Sukhija et al. (2024), we introduce additional exploration, by also optimistically picking the dynamics fn from set of plausible models MA. Moreover, since the true dynamics, f are unknown, we have to plan w.r.t. some dynamics model in M,. A theoretically grounded and well-established strategy for model-based RL methods is pick an optimistic model fn from Mn- As we show in Theorem 4.8 this results in first-of-its-king sample complexity and safety guarantees. The expansion phase of the algorithm is summarized Algorithm 1.
Theorem 4.8. Let Assumptions 4.1 to 4.3 and 4.5 hold. Then, we have with probability at least 1 - that Jn f*) S d vn 0, i.e., ACTSAFE safe during all episodes.
Moreover, consider any >> 0 and define i2 (So) as the reachable safe set after H expansions
Rí (Sp) del RH (So) U E 11 (S)); 3r'
Bnr' E RH--((So) s.t. Jo(T') + D(T, T') - d- c}
(Sp) def So-
Let n* be the smallest integer such that
(H + CYT
d. Tog(1 to 20
n² Yn (k)
(5)
where C (1 Vd.) max CC maxx R Jo), 7n (k) the maximum information gain (Srinivas et al., 2012), and म, the solution to arg max min JEM, (T f). Then we have Vn 2 n² with probability MES
at least - 8
max J,(TT) J+(+x) Se. (S.)
The theorem shows that ACTSAFE safe during all episodes. Furthermore, it shows that after finishing the expansion phase, ACTS AFE achieves an e-optimal solution within Ri, for the underlying reward function r, where R (So) is the largest safe set we can obtain after H expansion steps if we know the dynamics to precision. To the best of our knowledge, safety and give sample complexity bounds for safe model-based RI. algorithms in the episodic setting with continuous state-action spaces.
the first to prove
Intuitively, by maximizing the epistemic uncertainty, we explore our dynamics uniformly among all policies in the safe set S,, making our model more confident, i.e., reducing aa As our uncertainty

---

Published as a conference paper at ICLR 2025
Algorithm 2 ACTSAFE: Practical Version
Init: Model Set Qo
for episode n I,...: n1" do
Intrinsic exploration phase
Select 7T by solving Equation (7)
Prepare policy
Dn f (OI.LOUT(TT.)
Collect data
Update 2n Di:
Update dynamics
end for for episode n²" ..... do
Extrinsic exploration phase
Select Tr by solving Equation (8)
Dn (OOLLOU(TTT)
Update On t- 1mm
end for
within S, shrinks, we add more policies to our safe set (c.f. Definition 4.7) and thus facilitate its expansion. The proof of Theorem 4.8 is given in Appendix A.
While the algorithm itself is difficult to implement for continuous state-action spaces, it gives key insights that guide our practical implementation Section 4.3: (i) maximization of intrinsic rewards for expansion, (ii) pessimism w.r.t. plausible dynamics to define safe set of policies Sn, and (iil) selecting nr only from S, to ensure safety. Building on these insights, we introduce a practical version of ACTSAFE designed to excel in real-world scenarios, such visual control tasks.
4.3 PRACTICAL. IMPLEMENTATION
Optimizing over safe policies In Equation (4) we optimize over the policies within the safe set, where the safe set is defined according Definition 4.7. This is particularly challenging continuous state-action spaces since it requires us to maintain the model set M, and the safe set S.,. We modify the definition of the safe set which makes the optimization problem more tractable.
S, π E II; s.t. max J < f'EQ..
(6)
Note that S, S,, making it conservative estimate of Sn, therefore selecting policies from S, still preserves the safety guarantees. Furthermore, in S,,, f E 2n and thus we can simply use fhn on to induce pessimism, i.c., we do not have to maintain the model set M, M On (c.f. Definition 4.4). A similar relaxation made by other safe RL algorithms such as Berkenkamp et al. (2021); Baumann et al. (2021).
pessimistic w.r.t. the dynamics
To practically solve Equation (4) we use S, instead of S,,, resulting in the following problem
arg max max Jn (n, f) s.t. max Je(n, S d. f'e2.
#ell fEQ..
Equation (7) is a constrained optimization problem with the added complexity optimizing over the dynamics in 2n. Moreover, it does not require us to maintain S, since we implicitly account for it in the constraint in Equation (7), making tractable for continuous state-action spaces. In Equation (7), we have to solve max Je(TT, f') to enforce pessimism for safety. To this end, we use the methods from Yu et al. (2020) for our experiments. In practice, we solve Equation (7) by using CMDP planner based on Log-Barrier SGD (LBSGD, Usmanova et al., 2024). Further technical details can be found in Appendix B.
From CMDPs to visual control ACTSAFE can be seamlessly integrated with Mate-of-theear model-based RL methods for learning in visual control tasks (Hafner et al., 2019; 2023). To tighten the gap between RL and real-world problems, relax the typical full observability assumption and consider problems where the agent receives an observation o, p(-[s;) instead of s, at each time step. choose to base our dynamics model on the Recurrent State Space
To handle partial observability, Model (RSSM) introduced in Hafner et al. (2019). The RSSM can be thought of as sequential varia- tional auto-encoder that learns the (latent) dynamics f. We leverage approximate Bayesian inference techniques, in particular probabilistic ensembles (Lakshminaray et al., 2017). to approximate the posterior (f)),,, over RSSMs. In particular, we learn an ensemble of M models and define Qn as Q,, = -U - ( The model's epistemic uncertainty (disagreement) is then used to enforce pessimism w.r.t. the safety constraints and for the intrinsic reward exploration (see Algorithm 2).
7

---

Published as a conference paper at ICLR 2025
Trajectorics
Safe Boundary
Initial State
Episode
Episode 3
Episode 7
Episade 10
ACTS AFF
-5-
OPAX
0
Figure 2: Safe exploration in the PENDULUMS task. Each plot above visualizes trajectories considered during exploration across all past learning episodes. The red box the plot depicts the safety boundary in the state space. ACTSAFE maintains safety throughout learning.
Online policy improvement After the first intrinsic exploration phase, is often necessary per- form additional learning updates during the second exploitation phase (Sekar et al., 2020). Therefore, after iterations of intrinsic exploration, we optimize the extrinsic reward by solving
arg max max J, (T, f) s.t. max Je(n, f') S d. *Ell FE2,
(8)
f'E..
EXPERIMENTS
In the following, we evaluate ACTSAFE on state-based and visual control tasks. For the state-based tasks, use GPs to model the dynamics f For the visual control tasks, use the RSSM model from Hafner et al. (2019) as described in Section 4.3. We thus validate both the theoretical and practical aspects of ACTEAFE in this section.
5.1 DOES ACTSAFE EXPLORE SAFELY WITH GPs?
We evaluate ACTSAFE on the PENDULUM and CARTPOLE environments. Additional details on the experimental setup, including the safety constraints, are provided in Appendix B. For both environments, ten episodes and then use the learned model plan w.r.t. known extrinsic rewards after the expansion phase. For extrinsic rewards, we con- sider the SWINGUP task. We study the effects of pessimism with respect to the model uncer- tainty for safety. To this end, baselines a version of ACTSAFE without pes- simism, which only uses the mean model p., for planning and OPAX (Sukhija et al., 2024), unsafe active exploration algorithm.
ACTSAFE (Ours)
ACTSAFE (No Pessimism)
OPAX
Swingup Objective
Cumulative
Cost
run the algorithms for
Pendulum Carppole
Pendulum Cartpole
Figure 3: Evaluation safety via pessimism and intrinsic exploration. The cumulative cost accumu- lates all the incurred costs during learning, the re- ported objective performance is normalized. ACT- SAFF maintains safety during learning while at- taining high zero-shot performance the PENDU- LUMSWINGUS objective at test time.
consider
We present our results in Figure 3, where we report the performance and the total accumulated costs during exploration our method. We conclude that ACTSAFE does not incur any costs during learning. In contrast, the variant of ACTS without pessimism and OPAX are unsafe during learning. This validates the necessity of using the model epistemic uncertainty to enforce pessimism during exploration. Note that ACTS AFF pays price in terms of performance for pessimism, converges to a lower reward value than the other algorithms.
In Figure 2 we visualize the trajectories of ACTSAFE and OPAX in the state space during exploration. We observe that both algorithms cover the state space well, however, ACTS AFE remains within the safety boundary during learning whereas OPAX violates the constraints.
8

---

Published as a conference paper at ICLR 2025
SATE (Ours)
LAMBDA
CPO
BSRP-LAG
GoToGOAL
PRESSBUTTONS
PUSHBOX
10
0
75
75
40
50
20
25
25
0
0
2000
4000
2000
4000
0
2000
4000
Episodes
Episodes
Episodes
Figure 4: Safety of ACTSAFE in SAFETY-GYS with vision control. The dotted horizontal line depicts the safety constraint. We report the mean and standard error across 10 seeds. The vertical dashed line illustrates the transition of ACTSAFE from the intrinsic exploraiton/expansion phase to the extrinsic reward phase. Grey shaded area represents the warm-up phase.
5.2 DOES ACTSAFE SCALE TO VISION CONTROL?
While with GPs in particular visual control tasks, is challenging. We demonstrate our practical implementation (Algorithm 2) on high-dimensional RL tasks. We highlight that ensuring safety with NN model with randomly initialized weights is impractical without any additional prior knowledge. To this end, for all experiments hereon, we assume access to an initial data collection (warm-up) period of 200K environment steps, where the agent collects data and uses it to calibrate its world model. This experimental setup simple as it seamlessly integrates with both off and on-policy algorithms, such as CPO. Furthermore, it simulates a realistic setting, where the agent can collect some data initially in controiled/supervviee setting where safety not directly penalized. However, after the initial data collection period, the agent is required to be safe during learning. We use the same training procedure across all baselines and environments (akin to Dalal et al., 2018). Appendix C, we present additional experiments that study safe exploration under distribution shifts of the dynamics, effectively leveraging the simulator to calibrate the model and imitating sim-to-real transfer.
work closer to theory, scaling them to high dimensions with large data regimes,
Safety We investigate ACTSAFE'S performance in terms of constraint satisfaction during learning and compare it with state-of-the-art baseline algorithms for safe vision control (As et al., 2022: Huang al.. 2024) and with CPO (Achiam al., 2017). We use the same experimental setup from LAFETY-GYM, (Ray et al., 2019) and As et al. (2022), with the POINT robot in all tasks. As shown in Figure compared to the baselines, ACTSAFE, significantly reduces constraint violation on all tasks. Notably, while ACTSAFE slightly underperforms I. LA M DA, it incurs much smaller costs. This result may be interpreted by the conservatism needed to maintain safety during learning. Furthermore, observe that BSRP- LLAG generally underperforms both algorithms in terms of safety and performance. We provide more details on our comparison in Appendix B. Additionally, we ablate choice of LBSGD in Appendix and highlight its benefits.
Exploration In this experiment, we examine the influence of using an intrinsic reward in hard exploration tasks. To this end, we extend tasks from SAFETY-GYe and introduce three new tasks with sparse rewards, i.e., without any reward shaping to guide the agent to the goal. We provide more details about the rewards in Appendix B. In Figure we compare ACTSAFE with GREEDY baseline that collects trajectories only based on the sparse extrinsic reward. As shown, ACTSAFE substantially outperforms GREEDY in all tasks, while violating the constraint only once the GOTOGOAL task. In addition to SAFETY-GYM we evaluate on
from RWRL (Dulac-Arnold et al., 2019) with additional penalty for large actions (see Curi et al., 2020, and Appendix B). We compare ACTSAFE with three baselines. (i) UNIFORM, which samples actions uniformly at random during exploration, (ii) OPTIMISTIC, which uses the model epistemic uncertainty estimates exploration reward bonuses and (iii) GREEDY, which optimizes the extrinsic reward directly.

---

Published as a conference paper at ICLR 2025
ACTSAFE (Ours)
GRERDY
[SPAREAL[SPARSSE
PRESSBUTTONS SPARSE]
PUSHBOX|
10
5
0
2000
4000
2000
4000
5000
10000
40
20
20
20
20000
4000
2000
4000
5000
10000
Episodes
Episodes
Episodes
Figure 5: Performance on hard safe exploration tasks. The vertical dashed line illustrates the transition from data collection with intrinsic extrinsic reward.
Figure indicates that using uncertainty quantification for exploration is crucial, only ACTSAFE and OPTIMISTIC find non-trivial policies. Despite that, ACTSAFE outperforms OPTIMISTIC. Furthermore, even though UNIFORM initially explores with an unsafe policy, is insufficient to learn good dynamics model, and thus underperforms ACTSAFE. This is mainly due to the undirected exploration strategy of UNIFORM, which does not leverage the model's epistemic uncertainty.
Discussion Our experiments underscore the following key findings. First, intuitively, in the GP setting, where implementation is closer to theory, pessimism w.r.t. the model uncertainty plays a crucial role as achieve strict safe exploration. Second, in our visual control experiments, using small fraction of data (<5% of total data collected) as the warm-up period for calibrating the model and policy is sufficient for drastically reducing constraint violation. Learning safely typically requires some form of prior knowledge about the problem, hence, using the data from the warm-up period keeps the experiment setup realistic without imposing specific domain knowledge and thus sacrificing generality. Third, in addition to exploring safely ACTSAFE, also solves tough exploration problems with the intrinsic rewards playing erucial role. These results underline the importance of intrinsic exploration in RL, especially in safety-critical tasks. Moreover, ACTS AFE transfers directly from the GP setting to the vision control setting and in both
ACTSAFE (Ours)
UNIFORM OPTIMISTIC
GREODY
1000
500
500
2000 Episodes
2000 Episodes
Figure 6: Hard exploration performance in CART- POLESWINGUPSPAA
of RWRL benchmark. We report the mean and standard error.
our results show that ACTSAFE outperforms the baselines in terms of both safety and performance. We provide additional experiments in Appendix C, where ablate our choice of the LBSGD planner, evaluate ACTSAFE setting with distribution shifts in the dynamics and on a realistic robotics task from the state-of-the -fthc-rrt humanoid benchmark from Sferrazza et al. (2024).
6 CONCLUSIONS
In this paper, we introduce ACTSAFE, a safe model-based RL algorithm that leverages epistemic uncertainty as an intrinsic reward learn dynamics model efficiently and safely. We theoretically study systems with continuous state-action spaces and non- non-linear dynamics that lie in the RKHS, and provide guarantees on safety and near-optimality. We derive a practical variant of ACTS and demonstrate safe exploration and competitive performance with a Gaussian process dynamics model. Furthermore, we identify the key concepts that enable safe exploration with ACTSAFE and demonstrate how one can heuristically apply them to solve high-dimensional safe RL problems. Our empirical results showcase the importance of intrinsic rewards in the context of safety, demonstrating that ACTSAFE outperforms the baselines in the majority of tasks. In conclusion, ACTSAFE represents a significant advancement in safe reinforcement learning methods, enhancing both theoretical insights and practical applications.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank Jonas Hübotter for the insightful discussion and feedback on this work. This project has received funding from the Swiss National Science Foundation under NCCR Automation, grant agreement 5INF40 180545, the Microsoft Swiss Joint Research Center, grant of the Hasler foundation (grant no. 21039) and the SNSF Postdoc Mobility Fellowship 211086.
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In ICML, 2017.
E. Altman. Constrained Markov Decision Processes. Chapman and Hall, 1999.
Yarden As, Ilnura Usmanova, Sebastian Curi, and Andreas Krause. Constrained policy optimization via bayesian world models. ICLR, 2022.
Yarden As, Bhavya Sukhija, and Andreas Krause. Safe exploration using bayesian world models and log-barrier optimization. arXiv preprint arXiv:2405.05880 2024.
Dominik Baumann, Alonso Marco, Matteo Turchetta, and Sebastian Trimpe. Gosafe: Globally optimal safe robot learning. In ICRA, 2021.
Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. NeurIPS, 2017.
Felix Berkenkamp, Andreas Krause, and Angela Schoellig. Bayesian optimization with safety constraints: safe and automatic parameter tuning in robotics. Machine Learning, 2021.
Homanga Bharadhwaj, Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Animesh Garg. Conservative safety critics for exploration. arXiv preprint arXiv:2010.1 2020.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.068// 2022.
Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela Schoellig. Safe learning in robotics: From learning- based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 2022.
Xuxin Cheng, Yandong Ji, Junming Chen, Ruihan Yang, Ge Yang, and Xiaolong Wang. Expressive whole-body control for humanoid robots. arXiv preprint 4rXiv:2402.1679, 2024.
Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Duenez-Guzman,, and Mohammad Ghavamzadeh. Lyapunov-based safe policy optimization for continuous control. arXiv preprint arXiv:1901.10031, 2019.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. NeurIPS, 31, 2018.
Scott E. Cooper and Théoden I. Netoff. Multidimensional bayesian estimation for deep brain stimulation using the safcopt algorithm. medRxiv, 2022.
Sebastian Curi, Felix Berkenkamp, and Andreas Krause. Efficient model-based reinforcement learning through optimistic policy search and planning. NeurIPS, 2020.
Sebastian Curi, Armin Lederer, Sandra Hirche, and Andreas Krause. Safe reinforcement learning via confidence-based filters. In CDC, 2022.
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint
2018.
11

---

Published as a conference paper at ICLR 2025
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmalcki, Diego de las Casas, Craig Donner, Leslic Fritz, Cristian Galperti, Andrea Huber, James Keeling, Maria Tsimpoukelli, Jackie Kay, Antoine Merle, Jean-Mare Moret, Seb Noury, Federico Pesamosca, David Pfau, Olivier Sauter, Cristian Sommariva, Stefano Coda, Basil Duval, Ambrogio Fasoli, Pushmeet Kohli, Koray Kavukcuoglu, Demis Hassabis, and Martin Riedmiller. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 2022.
Dongsheng Ding, Kaiqing Zhang, Jiali Duan, Tamer Başar, and Mihailo R Jovanović. Convergence and sample complexity of natural policy gradient primal-dual methods for constrained mdps. arXiv preprint arXiv:2206.02344 2022.
Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. arXiv preprint arXiv: /290/, 2019.
Yonathan Efroni, Shie Mannor, and Matteo Pirotta. Exploration-explotttion in constrained mdps. arXiv preprint arXiv:2003.02/180 2020.
Marcello Fiducioso, Sebastian Curi, Benedikt Schumacher, Markus Gwerder, and Andreas Krause. Safe contextual Bayesian optimization for sustainable room temperature PID control tuning. In IJCAI, 2019.
Christian Fiedler, Johanna Menn, Lukas Kreisköther, and Sebastian Trimpe. On safety in safe bayesian optimization. arXiv preprint vrXiv:2403112945 2024.
Dylan J Foster, Adam Block, and Dipendra Misra. Is behavior cloning all you need? understanding horizon imitation learning. arXiv preprint arXiv:2407.15007 2024.
Javier Garefa, Fern, and o Fernández. comprehensive survey on safe reinforcement learning. JMLR, 2015.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In ICML, 2019.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301. 140444 2023.
Antonia Holzapfel, Paul Brunzema, and Sebastian Trimpe. Event-triggered safe Bayesian optimiza- tion on quadcopters. In L4DC, 2024.
Weidong Huang, Jiaming Ji, Chunhe Xia, Borong Zhang, and Yaodong Yang. Safedreamer: Safe reinforcement learning with world models. In ICLR, 2024.
Jonas Hübotter, Bhavya Sukhija, Lenart Treven, Yarden As, and Andreas Krause. Information-bused transductive active learning. arXiv preprint wwXiv:2402.15898, 2024.
Sham Kakade, Akshay Krishnamurthy, Kendall Lowrey, Motoya Ohnishi, and Wen Sun. Information theoretic regret bounds for online nonlinear control. NeurIPS, 2020.
Johannes Kirschner, Mojmir Mutny, Nicole Hiller, Rasmus Ischebeck, and Andreas Krause. Adaptive and safe Bayesian optimization in high dimensions via one- miedimmesiooal subspaces. In ICML, 2019.
Torsten Koller, Felix Berkenkamp, Matteo Turchetta, and Andreas Krause. Learning- based model predictive control for safe exploration. In CDC, 2018.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In ICML, 2018
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. NeurIPS, 2017.
Nevena Lazic, Craig Boutilier, Tyler Lu, Eehern Wong, Binz Roy, MK Ryu, and Greg Imwalle. Data center cooling using model-predietive control. NeurIPS, 2018.
12

---

Published as a conference paper at ICLR 2025
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science Robotics, 2020.
Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo Li, and Ding Zhao. Constrained variational policy optimization for safe reinforcement learning. In ICML, 2022.
Lucas Maystre, Daniel Russo, and Yu Zhao. Optimizing audio recommendations for the long-term: A reinforcement learning perspective. arXiv preprini arXiv:2302.0356// 2023.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 2015.
Adrian Müller, Pragnya Alatur, Volkan Cevher, Giorgia Ramponi, and Niao He. Truly no-regret learning in constrained mdps. arXiv preprint arXiv:2402. 15776, 2024.
Tingting Ni and Maryam Kamgarpour. A safe exploration approach to constrained markov decision processes. In ICML 2024 Workshop: Foundations of Reinforcement Learning and Control- Connections and Perspectives, 2024.
Cristina Pinneri, Shambhuraj Sawant, Sebastian Blaes, Jan Achterhold, Joerg Stueckler, Michal Rolinek, and Georg Martius. Sample-efficient cross-entropy method for real-time planning. In CoRL, 2021.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv: 191000100 2019.
Jonas Rothfuss, Bhavya Sukhija, Tobias Birchler, Parnian Kassraic, and Andreas Krause. Hallucinated adversarial control for conservative offlin policy evaluation. UAI, 2023.
Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In ICMI, 2020.
Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Humanoid- bench: Simulated humanoid benchmark for whole-body locomotion and manipulation. arXiv preprint arXiv:2403. 2024.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Gracpel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 2016.
Aivar Sootla, Alexander I Cowen-Riverr Taher Jafferjee. Ziyan Wang, David H Mguni, Jun Wang, and Haitham Ammar. Sauté rl: Almost surely safe reinforcement learning using state augmentation. In ICML, 2022.
Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias W. Seeger. Information-theeorti regret bounds for gaussian process optimization in the bandit setting. IEEE Transactions on Information Theory, 2012.
Krishnan Srinivasan, Benjamin Eysenbach, Schoon Ha, Jie Tan, and Chelsea Finn. Learning to be safe: Deep rl with safety critic. arXiv preprint 2rxiv:2010.14660) 2020.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety reinforcement learning by PID lagrangian methods. In ICMI,, 2020.
Yanan Sui, Alkis Gotovos, Joel Burdick, and Andreas Krause. Safe exploration for optimization with gaussian processes. In /CML, 2015.
Bhavya Sukhija, Matteo Turchetta, David Lindner, Andreas Krause, Sebastian Trimpe, and Dominik Baumann. Gosafcopt: Scalable safe exploration for global optimization of dynamical systems. Artificial Intelligence, 2023.
13

---

Published as a conference paper at ICLR 2025
Bhavya Sukhija, Lenart Treven, Cansu Sancaktar, Sebastian Blaes, Stelian Coros, and Andreas Krause. Optimistic active exploration of dynamical systems. NeurIPS, 2024.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq. Timothy Lillicrap, and Martin Riedmiller. Deepmind control suite, 2018.
Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv: /1044, 2018.
Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho
Hwang, Joseph E Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery rl: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, 2021.
Matteo Turchetta, Felix Berkenkamp, and Andreas Krause. Safe exploration in finite markov decision processes with gaussian processes. NeurIPS, 29, 2016.
Ilnura Usmanova, Yarden As, Maryam Kamgarpour, and Andreas Krause. Log barriers for safe black-box optimization with application to safe reinforcement learning. JMLR, 2024.
Sharan Vaswani, Lin Yang, and Csaba Szepesvari. Near-optimal sample complexity bounds for constrained mdps. NeurlPS, 2022.
Kim Peter Wabersich and Melanie N Zeilinger. A predictive safety filter for learning-based control of constrained nonlinear dynamical systems. Automatica, 2021.
Akifumi Wachi and Yanan Sui. Safe reinforcement learning in constrained markov decision processes. In ICML,, 2020.
Akifumi Wachi, Yanan Sui, Yisong Yue, and Masahiro Ono. Safe exploration and optimization of constrained mdps using gaussian processes. AAAI, 2018.
Daniel Widmer, Dongho Kang, Bhavya Sukhija, Jonas Hübotter, Andreas Krause, and Stelian Coros. Tuning legged locomotion controllers via safe bayesian optimization. In CoRL, 2023.
Alexander Wischnewski, Johannes Betz, and Boris Lohmann. model-free algorithm to safely approach the handling limit of an autonomous racecar. In IEEE International Conference on Connected Vehicles and Expo, 2019.
Tengyu Xu, Yingbin Liang, and Guanghui Lan. Crpo: A new approach for safe reinforcement learning with convergence guarantee. In ICML, 2021.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. NewrIPS, 2020.
Liyuan Zheng and Lillian Ratliff. Constrained upper confidence reinforcement learning. In L4DC, 2020.
14

---

Published as a conference paper at ICLR 2025
A PROOFS
In the following, we prove Theorem 4.8. First, we provide the analytical formula for the mean and epistemic uncertainty of a GP model. We denote as, that
Mn.j ( = ki, (a)) K + -2 ynj n (z) k(x,, m) ki (ar) (K,, + - (x)())
where Yn.j = 1 is the vector the j-th element of the observed next states 8/, k,, [k( m)llign and K, = [k(2;; ay) Massan is the kernel matrix. By concatenating the element- wise posterior mean and standard deviation, obtain (a( [mng [nna(ar)] and ( (n)(())]]5
Corollary A.1. Let assumption 4.5 hold, then we have for all π E n 2 0, with probability least 1
Pa( 2 Je(
Proof. Note that f E Qn for all n 2 0 with probability at least 1 8 (Lemma 4.6). Therefore, f E Mn. Furthermore,
Pn((T) = max Jo( f) JEM.
2 Jo( fr)
Lemma A.2 (Difference in Policy performance, Sukhija et al. (2024)). Consider any function
r S x A - R. Let Jr, ( f", 88) E, ΣT- r(s,, π (8:)) and ArJ(TT 8, a) E,- (r(s,a) + Jr,k+1 T f 8¹) ( f" 8)) with s' (s, refer to Jr.0 , f su) Jr T f-, The following holds for all so S:
For simplicity
80) = =Eya' J,(T', f", - J ;0)
Σ 1=
Ara s/c, T. (s())
Proof. See Lemma 5. Sukhija al. (2024).
Lemma A.3 (Comparing safety costs of policies).
JT, s0) - Je(TT' ff" 80) s D(T, T²)
15

---

Published as a conference paper at ICLR 2025
Proof. For notational convenience will omit the dependance on f
Jo(TT, so) Jo(T'' 80) Byx'
AT, s α (si))
E
-(((8,, (81)) c(sc, ((.))))
+ E
1e [TattI(TT s14111 - Balaaleen (TA
E
c(s (8))) ci T'.)))
E mm
(m) [eatI(T, (( (o( [JoktI(T, ((41]
< E
min {Le (s:) TT(I) m0max
s(t1s (T, min [ff xs:))
f (8c, n (8:))
11]
B (ee [ee [e
+ E
(Kakade et al., 2020, Lemma C.2.)
Ly (st) π (
< E Eys
Σ
min (Le (8:)) -T():)l ((5ma))22mmax) + 7C min
1=0
= x')
Lemma A.4. Let assumption
assumption 4.5 hold. Then we have Vn 2 0, Sn Sx-1 with
probability at least - 8, JT(T) < d.
Proof. Consider any π E Sn, \ Sn By Definition 4.7, we have that there exists a 'T' in Sn-1 such that
Pa(TT') + D(T, S d
Therefore,
d 2 Pa(TT) + D(TT( T')
2 Ja(T', , fr) + D(T,,T))
(Corollary A.1) (Lemma A.3)
2 f()
Corollary A.5 (All policies in S, are safe). Let assumption 4. assumpption 4.5 hold. Then we have Vn π E Sn with probability at least
Jo(T) < d.
0. this holds due to assumption 4.3. Consider any n. >> O. By induction, Vπ E Sn we have that J.( π. f d. Hence, we focus on π E Sn+1 Sn. In Lemma A.4, we show JT(T, f*) < d for all π E Sn+1 S,, This completes the proof.
Proof. We prove this by induction. For
Lemma A.6. Consider any positive and bounded function c E [0, Cmaxe]- Let assumption 4. 4.5
hold. Then we have Vn vf Mn. with probability least - ò
d. 3 (0) Io (8t, T ((+))))
Je(m f) Jo(TT, fz)l T. Eyg
Σ
16

---

Published as a conference paper at ICLR 2025
Proof. From Sukhija al. (2024, Corollary 2.) we have,
Je( Je(TI f") = E,
Σ
Jetti fT. Jet41((T f. s
1=
(Expectation w.r. a under true dynamics f*)
with 8141 - (8,, T x(8)) w
and 8/41 - (8, π (B+)) + Wt.
Furthermore, V 8) E 10, maxx for all π, f. s, and Therefore, given 8t,
|Bao [Jatt S, 6/21) Jasti T 81+1)]
| [s T(:))) f f(s T ((S)))]
$ max { Eve ( (TT, f, 8, Ew, (1 f. s1+1)] min
1
0 (Kakade al., 2020, Lemma C.2.)
(s (s,)) f (8r, (8:))]]
TCmax min
(1+ (1
}
$ TCmax min
(Sukhija et al., 2024, Corollary 3)
From hereon let C (I+vdd.) max Remase Cmase.
Lemma A.7. Let assumption 4.1 4.5 hold. Then have Vn, N 2 0 with probability at least 1 6
Vigo Vlog(1 + o t00 V
3 Ba 1 18) N
max E,x WSS,
Σ I IoNta- (8,,
< TTC
N
Proof. Consider any > O.
T-1
N-1
max Eny TES.
Σ
1 (8:))l
S
max E rES,
a (st, x (a:)))
N
t=
(Monotonocity of the variance)
N
max Epd TS
(s,,
N
(Monotonocity of the safe set)
R+
E
o (st, n, (s:))l
N
(Definition of πς))
R+--1
E,n, Σ | n. (s))i
+ n (J(m)) -(m;))
N
Let J(TTT) Jmr),, where π. is the policy proposed by Equation (4). We analyze this regret term. Note that since, we optimistically pick dynamics from Mn, we have JT(x)) S J [[, f.). where fi are the optimistic dynamics. Therefore, I J fi) - ( f*). Hence, we can invoke Lemma A.6 to get
r S TC
Σ
3.(6) (8 (
17

---

Published as a conference paper at ICLR 2025
Therefore,
n+N-1
TC
max E TES.
Σ (st,
Σ E
o (Sc, n (s:)))
TC
n+--1
E
( n. (8x)))
N
TC3=+N-100)
a+N-1
NT
Σ B
Σ (s,, (s:)))
N
(Cauchy-Secte Schwartz)
TC VNT
n+N-1
B
(0, (8:)))
N
i=
TC
VTd.o
Bn+ N1+N-1(8)
v /log(1 to 200)
VNT /
N
N
(Curi al., 2020, Lemma 17)
Vd,00
n+N (0)
T
Vlog(1 20
N
Lemma A.8. Let assumption 4.1 - 4.5 hold and define Nn to be the smallest integer such that
Vd.00
Jat N. Sc. c.
C²
32 N.
Vlog(1 1 200)
Nn
Then, we have VT E Sn f E MM+N,N1 with probability at least 8
f) - Je(T,, f()l S e, and, [Jr( f) - Jr(TT, f(1)s Sc.
Proof.
1 Vd,) 3,+ N,, -1 o n (8 (8:)))
[Jo(T,, f) - Jo(TT, f() < TC Enr
(Lemma A.6)
T-1 Σ
< TC N.
Vigao Vlog(1 + 220
In-N. (kk Bu-NN
< TCB.-N.-ITTC
No
(Lemma A.7)
S6
We can apply the same inequalities for Jr.
Corollary A.9. Let assumption 4.1 4.5 hold. Consider any n 2 0 and define N., as Lemma A.8.
Then we have with probability at least 1 -
SaNN, 2 (Sw).
18

---

Published as a conference paper at ICLR 2025
Proof. From Lemma A.8, have Vn Sn, f E M1+N,,1, f) Je( f*1
therefore (I+ E JT f*) ) + e. For the sake of contradiction, assume there exists a policy E (S,) SS+NN. We study the case where π (S,,)) \ S, else have contradiction (Sn S SR,N.,). Since π E RR (S..) (,5)),es
Sn, there exists a ' E Sn such that J=(T')+ D(T,, x²) s d-6
(see Theorem 4.8). Hence, get
d 2
2 P1+N.-1(T'') + ((T, x²).
Since, n/ S,, S SS+NN.-1 by the definition of the safe set (c.f. Definition 4.7), this implies that E Sn+N., which a contradiction.
key property of N, is that it increases monotonously with n. Moreover, for a given n 2 0, N,, is the smallest integer satisfying
N₂ Tn+N.- -I) (1
d.
log(L+o 70
2²
Both functions n. + Yn. and n + 8n are monotonically increasing with n. Hence increasing n, increases the right-hand side of the inequality, and therefore Nn.
Lemma A.10. Lel assumption 4.1- 4.5 hold and consider n² 2 (H + 1)N,,, Then we have with probability at least - 8 for all 2 n²
Sn 2 (S))
Proof. To prove this, show for any positive integer k S H, that SEN.. 2 (So) by induction. Moreover for any k, let T3 = Tx-1 + Nr.,, and To = O. We inductively show that T₃ S kN for all k H.
For the base case 1, we have Ti = No S Nn. since n 2 O. Consider any S H, then, we have Th = T By induction Tx-1 (K- 1).,.. Therefore, Tr S (k - 1)N,, + N(K-I)N.. Furthermore, note that (k-1)N,,= 5 n for all k H. Therefore, Th I (k (k-18 1)N,, +Nn kN
Next, we have from Corollary A.9, ST. 2 R (S = R5(S)) Moreover, ST, = SNo (So). Similarly, ST, == SNo+Nwo 2 (S)) = R5)S)), etc. Therefore, we get SHN.-
ST 2 Ri (Sp). As 2 HN,, this completes the proof.
Lemma A.11. Let assumption 4.1 4.5 hold and consider the smallest integer such that
(H + C
d,
n
log 2
(10)
Tn- (A) (8)
Then we have for all 2 n²
S, 2 (Sp) R2y (So) that Ja((T, f) Jo(T, <e and [Jr(T, f)
Moreover, we have for all 2 n²,
Proof. Note that for any n, N,, is defined as the smallest integer satisfying: d.
Nn
1og(1+o-7od
7n (k) (4 nt N. (8)
2
From Lemma A.10, for n² = (H + 1)N we have for all m 2 n
S. 2 R5,
We show that the solution to Equation (5) satisfies this condition. Moreover, let = (H + 1)
N..
Tn N
(k)34 a' N
(8)
N + #+1
8k)
(8)
n
n
T-1
Yn (k)
19

---

Published as a conference paper at ICLR 2025
Picking n² the smallest integer satisfying
d (H + C)T
n1
log 200
Yn- (k)) (6)
ensures that
T
d,00
N
log 2001
7n- +N. (k: +NN.. (6)
Finally, from Lemma A.8 we have that SHN,.- 2 (So) Therefore, for all n. (S Mn with probability at least 1-6 8
[e( - Je( f S e, and, [ f) , f*)
Proof of Theorem 4.8. We prove in Corollary A.5, that all policies in Sn are safe for all n 2 ACTSAFE is safe since it picks policies only from Sn.
showed in Lemma A.10 for all 2 n" that S, 2 (S)). Moreover, have (T, f*) S Let n be the optimal policy and let π. denote the solution to arg maxres, minjeM, Jr (TT, f). For the sake contradiction, assume that
For optimality, VT R (S f E MM 2 Mn. [ f) - (
J, ( <
max (T f*)
(11)
(So)
Furthermore, let (T) = minfeM.. J, J f) for all क E II.
PT((TT) - max (T(
*ESn
$ (ftn)
(,( -
(contradiction assumption)
$ (TT"")
(Lemma A.11)
This is contradiction, which completes the proof.
20

---

Published as a conference paper at ICLR 2025
B EXPERIMENT DETAILS
B.1 GP EXPERIMENTS
For the GP experiments, we approximate Equation (7) with the following unconstrained optimization problem.
arg max max Jn( - A max TEll SEC.
Je(TT, ((] Je( -d, (Ee.
max
(12)
Here À is a (large) penalty term that is used to discourage constraint violation. We use the iCEM (Pin- neri al., 2021) optimizer to solve the constrained optimization above. Effectively, given sequence of actions {ar {=o) we roll them out on our learned GP model using the TS1 approach from Chua al. (2018). Moreover, we maintain P particles, and given the state (sk, a.) ) for the p-th particle, we determine the next state $1+,, by sampling from N (H (s a.), on (s a²))) Accordingly, for each we obtain P trajectories and we empirically solve max JT( f*) by taking the max over the P trajectories. This approach also proposed by Kakade et al. (2020) as heuristic for optimizing over the dynamics.
action sequence {ar H
Rewards and constraints The reward function is designed to penalize deviations in both the angular position and the control input from the target behavior. For both the PENDULUM and CARTPOLE, the state of the pole can be defined as follows. Let 0 be the current angle, w the angular velocity, and the control input. The target angle denoted as Olngeet and the angular error between the current angle and the target angle is A0. The reward and cost functions for the PENDULUM environment are given by
PPeadulum (A02 +011 w² + 0.02 . u²), CPendulum max[|w] 6.0, 6.0,0.0,, d == 0.0.
For the CARTPOLE environment, the position and velocity of the slider are defined as p and respectively. The reward for the CARTPOLE environment is the given by
''Crtpole (A0² +p² + 0.1 (v² (2²+w)) 0.01-uu.
CCaripole = mmax (Irl 0.5, 0.0), d = 0.75.
B.2 VISION CONTROL EXPERIMENTS
We provide an open-source implementation of our experiments in https:///itthut com/ yardenas/ / acttsafe We encourage readers to use it, as it contains additional important im- plementation details. In all the experiments below, our policy consists of 750K parameters, a several orders of magnitudes compared to previous works on provable safe explorations.
Approximating Equation (7) We solve the constraint optimization problem Equation (7) using the LBSGD solver from Usmanova al. (2024). LBSGD is first-order optimizer that uses logarithmic barrier function to enforce constraint satisfaction. Previous works from Ni & Kamgarpour (2024); As et al. (2024) have successfully applied LBSGD for planning in model-based RL with CMDPs, showing notably fewer constraint violations than alternative solvers like the augmented Lagrangian method (As et al., 2022).
To approximate Equation (7), we maintain an RSSM ensemble of P particles and given the state action pair (st, n. (a, st)), we obtain P estimates 1 for the next state. We estimate o² with the varianceoddsagrceemnnt between the ensemble members, i.e., Var (f+++1)p=)) We obtain the next state st+1 by uniform sampling from 1 i.e., TSI from Chua et al. (2018). Akin to Yu et al. (2020), we approximate f'e, Jo(T,, f¹) by penalizing the cost function with or,
H y (c(s), + Aallor, ( ax)ll)
Je- (TT) =
1=0
where À is a pessimism parameter. Yu et al. (2020) show that for an appropriate choice of A. Ja- (TT) ) is indeed a pessimistic estimate of J.(T.). However, in our experiments we treat À as hyper-parameter.
Safety experiments We focus on SAFETY-GYM to showcase our practical algorithm design maintains constraint satisfaction during learning. Our experiments rely on newer fork of SAFETY- GyM which is available via our open-source code. We follow the experimental setup of Ray et al. (2019); As et al. (2022) and an episode length of T 1000. We set the cost budget for each episode
21

---

Published as a conference paper at ICLR 2025
to 25 for SAFETY-GYE (see Ray et al., 2019). After each training epoch estimate (T, and (T(T) by sampling 50 episodes, denoting the estimates with J, and Je- Unless specified otherwise, in all our experiments we use 5 random seeds and report the median and standard error across these seeds. Finally, we use budget of 5M training steps for each training run. To make fair comparison with As et al. (2022); Huang et al. (2024), we fix the ratio of environment steps and update steps of the model and policy. While Huang et al. (2024) use the RSSM model from Hafner al. (2023), our implementation uses the (older) one from Hafner et al. (2019) and As et al. (2022).
Sparse SAFETY-GYM Let dRG be the euclidean distance between the robot and the goal/button at time step i, de the distance between the box and the goal position and dBB the distance between the robot and the box positions. Furthermore, denote tol(2, 1, u) as the tolerance function from Tassa al. (2018), where l,u denotes lower and upper bounds respectively.
Environment
Dense Reward
Sparse Reward
GOTOGOAL
JRG dec + 5003
tol(d O, 0.445
(d de +
100
PRESSBUTTON
dRG dRG 50.3
10050.1
PUSHBOX
d di + PG
di
tol RB 0, 0,0.5)-(d, -d, +d
50.3
50.3
dBG g0000.3
Table 1: Comparison of the reward functions in the base environments of SAFETY-GYM and our sparse rewards environments.
Safe
Unsafc
Training Progress
Figure 7: CARTPOLE environment as an example problem instance of safe exploration. Each scene summarizes trajectory that was collected in increasing training iterations. The agent incurs cost whenever the cart goes outside of the area between the two red vertical lines. The goal is to learn policy that swings the pole the top position, while ensuring the expected accumulated cost is bounded during learning. Learning in this setting is much more challenging, as agents can only try control policies that known be safe.
Cartpole exploration In this task, the agent receives sparse reward when it swings up pendulum to the top position and when the slider (a.k.a cart) is centered. The RWRL benchmark (Dulac-Aran Armold et al., 2019) adds safety constraint that enforces the slider to remain in certain distance from the center (see Figure 7). As Dulac-Arnold et al. (2019), we use cost budget of = 100 and episode length of T 1000 steps. Adding the safety constraint adds significant challenge, as addition to the safety constraint, we add a
any safe policy is much more limited in exploration. cost for taking actions, as done in Curi al. (2020). Combining all these factors together, makes challenging exploration task, as we show in our experiments. Further implementation details can be found in our open-souree code.
22

---

Published as a conference paper at ICLR 2025
ACTSAFE: (Ours)
LAMBDA
CPD
BSRP-LAG
GOTOGOAL
PRESSBUTTONS
10
50
2000
1000
6000
8000
10000
0
2000
4000
6000
8000
10000
Episodes
Episodes
Figure 8: Performance and safety in with the DOGGO robot.
(Ours)
SAISAAELLGGRAGIAAN
GoTOGOAL
PRESSBUTTONS
PUSHBOX
15
20
10
Mm
10
100
030
50
40
50
25
2000
4000
2000
4000
2000
4000
Episades
Episodes
Episodes
Figure 9: Augmented Lagrangian from As et al. (2022) compared to LBSGD of Usmanova al. (2024). LBSGD significantly reduces the number of unsafe episodes.
C ADDITIONAL EXPERIMENTS
Experiments with the DOGGO Robot In this experiment we compare ACTSAFE with the same baseline algorithms from Section on SAFETY-GYM's DOGGO robot. We omit our results in the PUSHBOX environment as all baselines failed to solve it. As shown in Figure 8, similarly to the results in Section ACTSAFE maintains safety during learning, while moderately underperforming LAMBDA. Overall, ACTSAFE outperforms CPO both in terms of safety and performance and BSRP-LAG Huang et al. (2024) terms of performance.
Ablating LBSGD One assumptions of LBSGD that we cannot formally satisfy relates to unbiased- ness of the evaluation of the objective, constraints and their gradients. principle, satisfying this assumption will allow us to guarantee that all iterates of Equation (7) are feasible, i.c., satisfy the pessimistic constraint. This is in contrast to primal-dual methods, such as the Augmented Lagrangian of As et al. (2022) that lacks any guarantees feasibility during optimization. While is hard to formally satisfy LBSGD's unbiasedness assumption, we empirically observe that LBSGD allows us to keep constraint satisfaction during learning. We present this result in Figure 9. As shown, even after initializing both variants with initial data from the burn-in period, fails to satisfy the constraints throughout learning. As in the main results safety in Figure 4. compared to Augmented Lagrangian, LBSGD maintains safety during learning at a slight price of performance.
23

---

Published as a conference paper at ICLR 2025
Safe Adaptation Here, instead of the warm-up period of data collection, we study the effect of first training on a "safe" environment, like a simulator, and then continuing training on a similar environment, but with shifted dynamics. To this end, extend GOTOGOAL from SAFETY-GYM to two additional tasks, in which we change the motor gear and floor damping coefficients. The agent is first allowed explore the "sim" environment for 300K interaction steps before heing deployed on the "real" environment. We analyze the impact of our LBSGD optimizer and of pessimism in handling constraint violation during deployment. As shown in Figure 10, without LBSGD and pessimism, ACTSAFE does not always transfer safely to the deployment environment. Furthermore, intuitively, while pessimism crucial for maintaining safety while adapting to distribution shifts, may sometimes hinder performance of the main objective. This experiment demonstrates that, if one has no initial data, one can use ACTSAFF in combination with simulator to achieve safe exploration in practice, with clear tradeoff of the simulator's s fidelity and the degree of pessimism in ACTSAFF.
ACTSAFE (Ours)
ACTSAFE (No PESSIMISM)
ACTSSA (No PESSIMISM)
GoToGOAL[[DMPIN
[oTOGOAL[MOTOR]
20
20
10
100
50
50
0
0
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Episodes
Episodes
Figure 10: Adaptation to domain shifts. The red dashed vertical line represents the step after which switch dynamies. Black dashed vertical line represents changing from active exploration to greedily maximizing the reward. We report the mean metrics across seeds.
Humanoid Proof-of-Concept We further demonstrate the scalability of ACTSAFE on the HU- MANOIDBENCE benchmark (Sferrazza et al., 2024). We use robust, low-level walking policy provided with the benchmark, and input visual observations from third-person camera view. We compare ACTSAFE with OPAX (Sukhija et al., 2024) on the POLE task, where a humanoid robot must navigate through a field of pole obstacles, illustrated in Figure 11. In this task, the agent incurs a cost of for each pole it hits and when it falls, while the reward is based on the robot's 's forward
ACTSAFE (Ours)
OPAX
1000
750
500
200
0
500
1000
1500
2000
2500
Episades
(a) POLE task HUMANOIDBENCH. The robot has to cross to the other side of the maze while avoiding hitting the poles.
(b) Performance and safety on the POLE task of HU- MANOIDBENCH.
Figure 11: Overview of the Pole task and its performance metrics.
24

---

Published as a conference paper at ICLR 2025
velocity. As shown in Figure 11, ACTSAFE significantly reduces the number of constraint violations compared to OPAX, while maintaining competitive performance on the objective.
Comparison with OPAX CARTPOLE We compare ACTSAFE with OPAX (Sukhija et al., 2023) on the from Section 5.2. Both ACT AFE and OPAX rely on intrinsic rewards for exploration and model learning. however, ACTS AFE only con- siders policies from within the pessimistic safe set. We compare AFTSAFE with OPAX trained for 1M and 1.25M steps of pure exploration. ACTSAFE uses 1M exploration steps, in Sec- tion 5.2. As shown in Figure 12, OPAX fails sufficiently explore the dynamics within 1M steps. The reason being that ACTSAFE can explore much more confined state-action space, and therefore visits states with non-zero rewards quicker. This is in contrast to OPAX which is permitted to explore unsafe action-states as well, and therefore less likely to visit these states within the given training budget. We note that result in similar spirit has been observed by Widmer al. (2023, Figure 2). While 1M steps not enough for OPAX to fully learn the dynamics when no constraints are imposed on the policy, in Figure 12 we show that after having explored the dynamics for 1. 1.25M steps, OPAX is able to recover optimal policy. Unsurprisingly, in both experiments OPAX fails to satisfy the constraints, as optimizes only for the intrinsic reward.
ACTSAPE (Ours) OPAX 1.25M
OPAX @ IM
task
1000
1000
3000
4000
Episndes
Figure 12: Comparison of ACTSAFE and OPAX in task of RWRL.
the
25


Reviews:
[{"review_id": "2WLK9tRg47", "reviewer": "Reviewer_VK9v", "summary": "The paper introduces ACTSAFE, a model-based reinforcement learning (RL) algorithm designed for safe exploration within continuous state-action spaces. ACTSAFE utilizes epistemic uncertainty in the model as an intrinsic reward mechanism to encourage safe set expansion during exploration. The authors provide theoretical analysis that includes safety and sample-complexity guarantees, suggesting that ACTSAFE achieves near-optimal policy convergence within finite episodes. Additionally, a practical implementation of ACTSAFE is proposed, enabling the method to scale to high-dimensional tasks, including visual control. Empirical evaluations demonstrate ACTSAFE’s performance across various benchmarks, supporting both safe exploration and strong task performance.", "questions": "- The paper mentions the use of offline-collected data comprising 200K environment steps from a random policy. Can the authors clarify how this offline dataset is incorporated into ACTSAFE? Also, are all baseline algorithms utilizing this offline dataset consistently? If not, it may introduce unfairness in the comparisons.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Theoretical Rigor: The paper offers a thorough theoretical framework for ACTSAFE, deriving safety guarantees and sample-complexity bounds for safe exploration in continuous state-action spaces—a notable contribution in the safe RL domain.\nScalability in Practical Settings: The authors extend ACTSAFE to visual control tasks, demonstrating scalability beyond low-dimensional models. This practical application is a significant step towards bridging the gap between theoretical safe RL algorithms and real-world, high-dimensional deep RL applications.\n\n- Comprehensive Experimental Analysis: The experimental analysis is extensive, covering both theoretical settings (under the Gaussian process assumptions) and more complex visual motor control tasks in the SAFETY-GYM environment, as well as sparse reward exploration. This breadth of evaluation provides valuable insights into the algorithm's performance under varied conditions.", "weaknesses": "- Reliance on Assumptions: The theoretical guarantees rely on idealized assumptions (e.g., well-calibrated Gaussian processes and specific Lipschitz conditions), which may limit generalizability. However, the authors provide strong empirical evidence that ACTSAFE performs well even when these assumptions are not strictly met, somewhat mitigating this concern.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JYPdbNW5K0", "reviewer": "Reviewer_3f3n", "summary": "The authors present a model-based algorithm, called ActSafe, that has strong theoretical safety guarantees.", "questions": "Safe RL is not my area of expertise, and I therefore ask the metareviewer to discount my (unfortunately brief) review accordingly.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is very well written. The high level idea of the algorithm is presented in an understandable way, without sacrificing mathematical rigour. By combining existing ideas, like intrinsic motivation or safe Expansion operators, the authors seem to have created a conceptually simple but very powerful algorithm for safe RL. \n\nBased on a strong theoretical foundation a practical implementation is provided, that crucially maintains the safety constraints. The experiments seem well designed, highlighting the clear strengths of the algorithm (its safety guarantees), while also discussing weaknesses. The authors especially discuss scalability weaknesses that may result from the choice of Gaussian Processes to approximate the system dynamics.", "weaknesses": "While the two-phase approach that ActSafe employs are the foundation for its safety guarantees, I would expect that this comes at a cost. A comparison of total environment steps in both loops required, wall clock or memory requirements would have been a nice addition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B4lg3aTm4b", "reviewer": "Reviewer_JgEe", "summary": "This paper tackles the problem of safe exploration in reinforcement learning by proposing a new model-based constrained RL algorithm. Under the constrained MDP formulation, with regularity assumptions on the underlying dynamical system, the paper demonstrates theoretical safety guarantees and sample complexity guarantees. A practical version of the algorithm with weaker guarantees is shown to be applicable for visual control tasks in different simulated benchmarks. The results show less safety violations compared to prior constrained RL approaches.", "questions": "Please refer to the list of weakness above, \n\n- Can the authors explain the disconnect between the motivations in the paper and the experiments? \n\n- In lines 233-235, it is mentioned that prior works do not have dimensional policies, but there is no explicit clarification about how high dimensional control tasks can the proposed approach tackle.  What are the relative differences? Are there any fundamental assumptions that prevent these prior works from being applicable to higher dimensional tasks?\n\n- It is unclear why there are no evaluations in a model-free setting, and what are the motivations for just considering model-based RL, after proposing a safe exploration algorithm that seems to be fairly generic and broadly applicable. It will be helpful to clarify these points. \n\n- In lines 225-226 it seems that in the general case the epistemic uncertainty will be uncalibrated / unreliable due to function approximation errors in the neural network model. Does the theory account for this?\n\n- Can the authors clarify relations to the missing related works, including very related safe exploration works that actually show results in real-world settings?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper targets a problem that is relevant to a broad community including controls, reinforcement learning, and robotics. Safe exploration in the context of model-based control is interesting because a reliable approach in this space has the potential for being deployed in safety critical scenarios as well as in scenarios that require sample efficiency and real-world exploration is challenging. \n\n- The proposed algorithm based on intrinsic exploration for reducing uncertainty of policies at the boundary of the current safe set and expanding the safe set based on reachability beyond it, is novel to the best of my knowledge. It is also intuitively sound and doesn't make any fundamentally limiting assumptions of the underlying system. \n\n- The theoretical guarantees of the base algorithm are strong, and the regularity assumptions seem reasonable to me. \n\n- The experiments on simulated environments are good, and it is nice to see results in challenging control tasks like a simulated humanoid walking, that go beyond the predominant evaluations of prior safe RL works on simple simulations like SafetyGym.", "weaknesses": "- A major weakness is that the paper's motivations are disconnected from the experiments. For example the intro states: \n\n\"In many real-world settings, environments are complex and rarely align exactly with the assumptions\nmade in simulators. Learning directly in the real world allows RL systems to close the sim-to-real\ngap and continuously adapt to evolving environments and distribution shifts. However, to unlock\nthese advantages, RL algorithms must be sample-efficient and ensure safety throughout the learning\nprocess to avoid costly failures or risks in high-stakes applications.\" \n\nHowever the experiments are all in simulated environments and there are no real-world experiments either in a controls setting or in a robotics setting.\n\n \n- It is unclear how the model-based version of ActSafe can be reliably be deployed in the real-world for safety-critical applications. Both theoretically and empirically in the simulated experiments, the constraint violations are lower than the baselines but nowhere close to 0. In addition learning a recurrent state-space model is a more complex choice than directly trying to learn a policy through model-free RL. It is unclear why there are no evaluations in a model-free setting, and what are the motivations for just considering model-based RL, after proposing a safe exploration algorithm that seems to be fairly generic and broadly applicable. It will be helpful to clarify these points. \n\n- In lines 233-235, it is mentioned that prior works do not have dimensional policies, but there is no explicit clarification about how high dimensional control tasks can the proposed approach tackle.  What are the relative differences? Are there any fundamental assumptions that prevent these prior works from being applicable to higher dimensional tasks?\n\n\n- The related works seem to miss a lot of prior approaches that also have similar ideas of conservative exploration  and actually demonstrate results on real-world settings where safety in important. For example, see [A-C] below. [A] in particular demonstrates results on real robotic manipulation tasks where safety is critical. \n\n[A] Thananjeyan, Brijen, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E. Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. \"Recovery rl: Safe reinforcement learning with learned recovery zones.\" IEEE Robotics and Automation Letters (RA-L)\n\n[B] Bharadhwaj, Homanga, Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Animesh Garg. \"Conservative safety critics for exploration.\" ICLR 2021\n\n[C] Srinivasan, Krishnan, Benjamin Eysenbach, Sehoon Ha, Jie Tan, and Chelsea Finn. \"Learning to be safe: Deep rl with a safety critic.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R37k8Zw2Fs", "reviewer": "Reviewer_XLTg", "summary": "This paper introduces ActSafe, a model-based algorithm for addressing the challenge of safe and efficient exploration in continuous state-action spaces, and provides a detailed theoretical analysis of it. This algorithm integrates OPAX and LBSGD with typical model-based RL methods to achieve safe and efficient exploration performance in vision control tasks. They conduct sufficient experiments in Safety-Gym domain, demonstrating that their approach outperforms prior works regarding safe exploration.", "questions": "1. It would be great if the author could address the main weaknesses I have outlined above. If they are properly addressed, I would be happy to raise my score, as I may have misunderstood the paper.\n2. Considering that there are no hard constraints on safety during learning in practical implementations, can ActSafe maintain safety during training after removing the initial offline data?\n3. I am very familiar with BSRP-Lag, but the algorithm's performance in the PUSHBOX task doesn't quite match my experience, can you disclose more experimental details?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The problem is well-motivated.\n2. This paper is mostly well-written and has solid theoretical foundations. The code provided is very helpful for the reproducibility and review of this paper.\n3. The empirical results in sparse reward tasks are impressive.", "weaknesses": "1. The proposed algorithm ActSafe, is a combination of  OPAX, LBSGD, and Dreamer(or other model-based RL methods). The technical contributions are incremental.   But considering the experiments are solid and sound, that should be fine.\n2. Inconsistency. This algorithm addresses the problem of safe exploration by maintaining a 'safe set' of policies. However, after checking the code provided, there is no such concept of a 'safe set', they just optimize the actor with the typical safe actor-critic method. Since the theoretical results might be unconvincing because of the inconsistency between the theory and practical implementation, it would be nice if the authors could explain this.\n3. Baselines.  In Figure 5, the author compares ActSafe with existing methods like OPTIMISTIC、UNIFORM and GREEDY. Are there any other more competitive baselines?  Comparison with methods like GoSafeOpt、OPAX might be necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)