Safe Exploration via Policy Priors
Authors: Manuel Wendl
Yarden As
Manish Prajapat
Anton Pollak
Stelian Coros
Andreas Krause
Venue: ICLR 2026 Poster (2026)
Keywords: Deep Reinforcement Learning
Safe Exploration
Safe RL
Constrained Markov Decision Processes
Primary area: reinforcement learning
TL;DR: We propose a safe and scalable reinforcement learning algorithm that leverages policy priors with probabilistic dynamics models to guarantee safety and convergence to optimal performance.

Abstract: Safe exploration is a key requirement for reinforcement learning agents to learn and adapt online, beyond controlled (e.g. simulated) environments. In this work, we tackle this challenge by utilizing suboptimal yet conservative policies (e.g., obtained from offline data or simulators) as priors. Our approach, SOOPER, uses probabilistic dynamics models to optimistically explore, yet pessimistically fall back to the conservative policy prior if needed. We prove that SOOPER guarantees safety throughout learning, and establish convergence to an optimal policy by bounding its cumulative regret. Extensive experiments on key safe RL benchmarks and real-world hardware demonstrate that SOOPER is scalable, outperforms the state-of-the-art and validate our theoretical guarantees in practice.

Published as a conference paper at ICLR 2026
SAFE EXPLORATION VIA POLICY PRIORS
Manuel Wendl, Yarden As" ETH Zurich
Manish Prajapat ETH Zurich
Anton Pollak ETH Zurich
Stelian Coros ETH Zurich
Andreas Krause ETH Zurich
ABSTRACT
Safe exploration is key requirement for reinforcement learning (RL) agents to learn and adapt online, beyond controlled (e.g. simulated) environments. In this work, we tackle this challenge by utilizing suboptimal yet conservative policies (c.g., obtained from offline data or simulators) as priors. Our approach, SOOPER, uses probabilistic dynamics models to optimistically explore, yet pessimistically fall back to the conservative policy prior needed. We prove that SOOPER guarantees safety throughout learning, and establish convergence policy by bounding its cumulative regret. Extensive experiments on key safe RL benchmarks and real-world hardware demonstrate that SOOPER is scalable, outperforms the state-of-the-ar and validate our theoretical guarantees in practice.
optimal
INTRODUCTION
defining feature of intelligence the ability utilize online streams of information to learn and adapt over time (Silver & Sutton, 2025). Reinforcement learning (RL) provides framework for online learning without supervision, driving several notable real- horldwrldd applications (Silver et al., 2017; Ouyang et al., 2022). Safety is key unlocking RL in the physical world-Re RL agents simply cannot risk taking actions that lead to catastrophic outcomes (Dalrymple et al., 2024), even while learning, when they lack complete knowledge about their environments (Legg, 2023). Such agents must explore safely (Amodei et al., 2016), gradually discovering new safe behaviors to solve their tasks.
Safe exploration is a major challenge in practice. Methods with theoretical safety guarantees often struggle to scale complex, general-purpose tasks, whereas scalable approaches typically fail to en- sure safety during learning (cf. Section 2). A key reason lies in how they incorporate prior knowledge about the environment-al crucial component for anticipating danger without learning through harmful trial-and-error. Such prior knowledge can be instantiated via policies. Beyond primarily maintaining safety, as done in much of prior work, we argue that prior policies can also guide exploration toward promising regions of the environment, enabling agents to provably learn near near-optimal policies.
Motivated by this insight, we propose Safe Online Optimism for Pessimistic Expansion in RI- SOOPER model-based RL. algorithm for safe exploration constrained Markov decision pro- cesses (CMDP, Altman, 1999). SOOPER uses prior policies that can be derived from scarce offline data or simulation under distribution shifts. Such policies are invoked pessimistically during online rollouts to maintain safety. These rollouts collected optimistically so to maximize information about a world model of the environment. Using this model, SOOPER constructs a simulated RL environment, used for planning and exploration (Amodei et al., 2016), whose trajectories terminate once the prior policy invoked. The key idea is that early terminations incentivize the agent to avoid the prior policy when trajectories with higher returns can be obtained safely. This design allows SOOPER to leverage standard RL methods, bypassing the complexity of directly solving CMDPs. More fundamentally, this enables a novel bound on the agent's cumulative regret.
Our contribution.
First, we propose SOOPER, a scalable model-based RL algorithm for safe exploration in continu- ous state-action spaces that leverages offline- and dimulation-traime policies as safe priors.
"Equal contribution. Corresponding authors: (mwend1, yardas) @ethz.cc

---

Published as a conference paper at ICLR 2026
Next, under standard regularity assumptions, prove that SOOPER guarantees constraint satisfaction with high probability throughout learning. We additionally show that SOOPER gradually expands implicit set of safe policies until it converges to the feasible set defined by the true CMDP. This is accomplished by reformulating the problem as an unconstrained MDP, making the approach naturally compatible with standard deep RI. methods, while avoiding min-max formulations typical for CMDPs. Using this formulation, we further establish a novel upper bound on the cumulative regret, improving over prior works that provide optimality guarantees only at the end of training with arbitrarily poor performance during exploration (Wagener et al., 2021; As et al., 2025b).
Finally, we conduct extensive empirical evaluation of SOOPER, comparing it with other of-e-of-the-aa safe exploration algorithms when initialized from offline- and simulation-train trained policies in RWRL benchmark (Dulac-Arnold et al., 2021) and SaferyGym (Ray et al., 2019). In all these experiments, SOOPER substantially outperforms the baselines in terms of safety and performance. We further validate SOOPER for safe online learning on hardware, giving empirical evidence that our theory translates to the real world.
2 RELATED WORKS
Prior works safe exploration fall broadly into two categories: theoretically principled methods with guarantees on safety and optimality and those prioritizing practicality and scalability.
Provable safety. A canonical line of work leverages safe Bayesian optimization (Sui et al., 2015), which has seen real-world success (Berkenkamp et al., 2016; Kirschner et al., 2019; Sukhija et al.. 2023b), but typically ignores the Markovian structure of sequential decision problems and is limited to few parameters. Berkenkamp et al. (2017) extend to continuous MDPs and provide safety and optimality guarantees via Lyapunov stability, though they rely on manual resets of the system at each timestep and state-space discretization for safe set computation. In contrast, Prajapat et al. (2025b) leverage model predictive control (MPC) to implicitly compute these sets and ensure that the agent can always return to a safe state. Other MPC-based methods ensure safety either through "backup" poli- cies (Koller et al., 2018; Prajapat al., 2024; 2025a) or via low-fidelity models with reachability anal- ysis (Hewing et al., 2019) and safety filters (Wabersich & Zeilinger, 2021; Curi et al., 2022). However, these approaches lack optimality guarantees. A substantial body of work addresses safe exploration in discrete CMDPs (Moldovan & Abbeel, 2012; Turchetta et al., 2016; Wachi et al., 2018; Wachi & Sui, 2020; Ding et al., 2021; Simão et al., 2021; Prajapat et al., 2022: Bura al., 2022). Akin our work, Bura et al. (2022) proposes a model-based approach that uses optimism and pessimism to provide opti- mality and safety guarantees for small-scale tabular problems. Although the above methods guarantee safety throughout learning, scaling them to high- -igh-dimensional problems is computationally hard.
Scalability. The works of Dalal et al. 22: Eysenbach et al. (2018); Srinivasan et al. (2020); Thananjeyan et al. (2021); Bharadhwaj et al. (2021); Thomas et al. (2021); Sun et al. (2021); Liu et al. (2022); As et al. (2022); Sootla et al. (2022); Huang et al. (2024) use scalable tools from deep RL for safe learning, with different levels of theoretical guarantees and effectiveness in maintaining safety throughout learning in practice. Safe exploration has also been studied from an optimization perspective, with methods aiming to guarantee feasibility of all optimization iterates. These techniques span trust regions (Achiam et al., 2017; Milosevic et al., 2024). Lyapunov stability (Chow et al., 2018), interior-point (Liu et al., 2020; Usmanova et al., 2024; Ni & Kamgarpour, 2025), and primal-dual optimization (Usmanova & Levy, 2025). While these methods scale well, they often overlook the exploration aspect of learning, namely how to actively sample data that maximizes information about the environment. As a result, they often lack optimality except under assumptions on the initial state distribution.
Optimality. As et al. (2025b) propose ActSafe, which guarantees safety and optimality for simple regret, relying on reward-free exploration, and therefore might perform poorly during exploration. Wachi et al. (2023) introduce MASE, which enforces safety under high-probability constraints on safe states but relies on a restrictive "emergency stop" action that can effectively 'stop time' at any state. Lastly, Wagener et al. (2021) propose SAILR, a state-of-the-as algorithm that like SOOPER, reformu- lates safety through terminations whenever their "backup" policy is invoked. However, its guarantee on simple regret hinges on the probability of such resets vanishing over time, yet no formal conditions are provided to ensure this occurs. In this work, we relax several of these assumptions and establish novel bound on the cumulative regret, while also demonstrating improved empirical performance.
2

---

Published as a conference paper at ICLR 2026
3 PROBLEM SETTING
Constrained Markov decision processes. Safety in RL has been conceptualized in several ways across the literature (García et al., 2015; Brunke et al., 2022; Gu et al., 2024). In this work, we focus on constrained Markov decision processes (CMDP, Altman, 1999), due their flexibility and capacity to generalize several notable formulations of safety (cf. Wagener et al., 2021; Curi et al., 2022; Wachi al., 2023). A discounted where S C dRds and A C Rda are the continuous state and compact action spaces, with the transition probability p(*++1 St, 0e) of reaching 8t+1 E S by applying at E in 8 S. We consider the transition probabilities p to be governed by the unknown stochastic dynamics
CMDP defined by the tuple Me. == A, p, c, 7. Po).
1 = (8t, a+) + wt: (8t; 01) E ES A, 80 ~ Po:
with the initial state distribution Pa and additive noise E W C Rds, Throughout our theoretical analysis, we consider the setting of additive zero-mean Gaussian noise.
Assumption 1 (Gaussian noise (Kakade et al., 2020)). The additive noise wt is independent and identically distributed (i.id.) zero-mean Gaussian noise with variance o².
We note that this assumption can be relaxed reward and the cost are given by r
richer class of sub-Gaussiaa distributions. The : S x A + [0, Rmax] and e : S x A + [0, Cmax] with discount factor 7 (0, 1). We assume the following regularity conditions for the dynamics, reward, and cost.
Assumption (Lipschitz-conninutty (Berkenkamp et al., 2017)). The dynamics f, reward r, cost e and epistemic uncertainty on, Yn 2 0 are Lipschitz continuous with known constants L L,, Le, Lo-
Assumption 3 (Regularity (Berkenkamp al., 2017)). The unknown dynamics f lie in RKHS asso- ciated with a kernel and have a component-wise bounded norm [| S. B with a known constant B. Moreover, assume the kernel k to be bounded by kc[(s, a), (s, kmax, V(s, a) E x A.
Intuitively, this assumption means that the complexity of f is controlled by the kernel, which in turn enables generalization from limited data. Our analysis uses Assumption 3 to relate the statistical complexity of learning the dynamics the number of online episodes required for convergence. In Section 5, we model it using neural networks that can learn features to represent f accurately.
The goal in our setting is to find a stationary, stochastic policy x (aals)) for the true dynamics f that maximizes value function (s(s) == E, 1 7"r(st, at) so = s). while ensuring that the cost value function V = E ot= r //c(st, az) 50 s) remains below safety budget d E Rzo:
Th E arg max JT, f) = Eo (V. (8o)
t. Je(x, f) = Esoxpo
(s))) S d.
(2)
We hereafter consider in our theoretical analysis strict feasibility of r:; namely that an optimal policy of the constrained problem does not lie on the boundary of the feasible set but may be arbitrarily close, such that J f) < d. Equation (2) naturally decouples the objective from safety, so each can be solved independently. This explicit separation prevents agents from exploiting the reward function to achieve high performance at the expense of harmful behaviors, a problem commonly referred to as reward hacking in the AI safety literature (Amodei et al., 2016).
Safe online learning. Since knowledge of the dynamics is limited priori, the agent must learn them through online interaction. We consider learning in an episodic setting, where in each episode 1,... N, infinite-horizon trajectories are truncated after T, steps and the agent is reset new initial state (Puterman, 2014; Sutton & Barto, 2015). The agent deploys a policy Tn, collecting transitions D, {(st, ar, st+1))=1.... }. The optimality gap due incomplete knowledge over episodes is quantified by cumulative regret, namely, the difference in performance between nn and the optimal policy given full access to Me:
R(N) Σ (JA(E)) -
()) t. Je(nn, S d, Vn (1....,N) N).
The crux of safe exploration is that unlike performance, where suboptimality during learning is tolerable, safety must be maintained throughout all episodes.
3

---

Published as a conference paper at ICLR 2026
Probabilistic world models. Given full knowledge of the CMDP Mc. the agent solve Equa- tion (3) with no regret. In practice however, the agent has only limited knowledge about Mc. We capture this uncertainty by defining set of plausible models of the dynamics Fn for each iteration
Definition 1. We define the set of plausible models in episode n' as Frl (f 1f parl ow h described by a nominal model ps :S x A + Rdds and the uncertainty on' : :S x A - -Rus given data D 1ox. This implies the model of f is always well-calibrated (Curi et al., 2020), there exist vn' < : 3 E R200 such that, with probability of at least 1 - it holds V(s a) E S x A that a(),a) Fn : Fo Fn'-
Definition 1 formalizes the agent's epistemic uncertainty about Me. At each iteration the agent maintains confidence set that is statistically consistent with the observed data. This is done by learning model with nominal dynamics pn S x A + R and epistemic uncertainty On : S x + Rdds for every n I,..., N, using transitions from previously collected trajectories D... This type of statistical models can be learned in practice via various (approximate) Bayesian inference methods (e.g., Rasmussen & Wiliams, 2006; Lakshminarayana et al., 2017; Liu & Wang. 2016). Additionally, calibration can be achieved empirically with post-hoe recalibration techniques (Kuleshov et al., 2018).
Pessimistic policy priors. The set Fo has unique interpretation: it formally represents the agent's prior knowledge about the environment. For instance, in fully data-driven settings, given offline dataset Dg. model Fo can be learned from Do and then used to construct a conservative MDP that penalizes the agent when it goes outside of the data distribution (Levine et al., 2020; Yu et al., 2020). Alternatively, in simulator-driven workflows, one has access to a nominal model Fa_: S A - Rds, and bounded 'sim-to-real gap' u. x A - Rds, giving I E Fo = { |f pol w). Probabilistic models of this sort formalize the conservatism needed for safe prior policies.
Assumption 4 (Safe policy prior). We assume access a pessimistic policy prior î S A that satisfies the constraint in Equation (2) for all dynamics| J E Fo such that under the true dynamics we have (z((s) I Vo (s) for all states with probability at least 8-
Learning such "worst-case" policies builds on the principle of pessimism in the face of uncertainty and has been extensively studied in the robust RL literature (Kitamura al., 2024; Zhang al.; Shi et al.; Shi & Chi, 2024), offline safe RL (Jin et al., 2021; Liu et al., 2023; Zheng et al., 2024; Wachi et al., 2025), safe learning from demonstrations (Schlaginhaufen & Kamgarpour, 2023; Lindner et al., 2024) and when transferring from simulators (Queeney & Benosman, 2023; As et al., 2025a). In our experiments in Section 5, we leverage a few of these methods and demonstrate the effectiveness of combining them with safe online learning.
SOOPER: SAFE ONLINE OPTIMISM FOR PESSIMISTIC EXPANSION IN RL
In nutshell, SOOPER can be explained by two modes of operation: (i) safe online data collection by pessimistically invoking a safe prior policy early enough so as to ensure constraint satisfaction; (ii) optimistic planning in "simulation" via model- rollouts, driving exploration of the unknown dynamics through intrinsic rewards.
Quantifying pessimism.
We define the pessimistic cost value function V for the policy prior
DC E''oss at)
V th (s) max E. JEF.
8t+1 = f(8,, at) + wt-
80 =
By definition, Equation (4) considers the worst-case model f in the well-calibrated set F, (per Definition 1), which implies that V+((s) upper-bounds v**(s) for all s E S with probability 1 8. We later formally show that, the number of episodes n increases, the agent refines its model F, and tightens the pessimistic upper-confidence bound V on the expected accumulated cost.
tractable upper bound. Computing the pessimistic cost- V in Equation (4) is intractable due to the maximization over F,,. Instead, upper-hound V by augmenting the cost function with
4

---

Published as a conference paper at ICLR 2026
uncertainty penalty such that
(,,( a) = Ex Σ az) + Apessnnisml/ a+) so s, ag =
(5)
(s) S Vel =Ban 12 (s, (s,a)], Se+11 = (1,, a4) wt
Crucially, directly penalizing the cost function avoids finding the worst-case model within Fn. and lends itself gracefully to existing temporal difference techniques for value function learning. The derivation of Apessimism and the proof for this upper bound are deferred to Lemma 2 in Appendix A.
Online cost tracking. SOOPER operates an online fashion, where for each of the trajectory rollouts, it uses Q to predict when a, Tx (-1s is potentially unsafe. Specifically, this may occur the sum of the accumulated cost until I, i.e., Cet = Σ==o? 7 7-c(8r; (+) and the pessimistic cost value 7 Qo (t,,))) 0x) exceeds the safety budget d. We exploit this idea in design of Algorithm 1. In Theorem we show that this design guarantees safety throughout all episodes n = 1,..., N.
Algorithm Safe Rollout via Online Cost Tracking
Require: Pessimistic cost value Qa policy *,, safe policy prior n, and safety budget d Initialize accumulated cost C<o = 0 and experience buffer D,, = 0
2: for step t = -1..., Tn do
3: Execute ae ~ nn (ael St: Cet, Q²,n) observe 8t, at, 8c-1
Theorem
4: Append the tuple (st, des 8141) Dn,
5: Update accumulated cost: Get+1 = Cet + 7
6: end for
7:
return Dr
Theorem (Safety guarantee). Suppose Assumptions I to hold and F,, well-calibrated Vn. 1...., according to Definition I. actions are executed for all timesteps t according to
(T((-1))) (m)(())))()))s)) if ((1,,, Cet, QT < otherwise,
Tn Tn(ailse Cet, Po = (A(s))
where t is the discounted sum of the realized accumuluted cost Cet == Etz, c(s,, until and the pessimistic cost value Q such that 4 (88,, at, Get, Q2,x) == Cet + 7 Q'Q
10,(0n (81, with
factor Appessimism = =O (1+vds) 88 (6) where Ö = max (Cmary kmar]- Then, Algorithm satisfies the safety constraint with probability 1 - 8 on f for every episode n =,,1,... N.
We refer Appendix A for the formal proof. Monitoring costs online allows the agent execute n. until Ā is required. This enables the agent to visit state-actions that î would avoid, since those would be deemed unsafe by (so).
Simulated exploration. While Theorem guarantees safety during online rollouts, the policy may explore potentially unsafe behaviors by using simulated rollouts on the learned world model. This allows SOOPER to train n, an unconstrained "planning MDP" M, that differs from the real CMDP M, that it reaches a terminal state 84 after those state-actions that would trigger á during online deployment:
[1(5 st} if o $1, Cen, Q5 do St (p(s441 st. st) otherwise.
p(s St, ..)
(6)
The resulting termination signal encourages the agent to find policies that outperform क if possible. This done by assigning the terminal reward on M to the pessimistic value V of A upon termination:
V (st)
((a, St, Czt, Q+ 2 d, if St = St. otherwise,
F(s,, 0t) =
0
a()
whereby,
(8)) = min Es
Σ
r (r(s,, ae)
St+1 = Í(s,, az) ) + wi-
(8)
jeF,
5

---

Published as a conference paper at ICLR 2026
E
Σ rr(s: a+)
+ (x)) s V (80)
Figure 1: The agent's goal to reach the cross marker while avoiding obstacles (tires). The agent deploys a policy Ras however at time t, it switches to the prior policy म. to maintain the safety criterion of Algorithm The prior policy क ensures safety by following con- servative route, but sacrifices performance, resulting in a lower return V (s,). The tra- jectory of iteration n is recorded improve models of subsequent iterations. After N it- erations, as more data is gathered, the agent learns a more rewarding trajectory via model- generated rollouts M. See provided video.
V (8t)
V.e (so)
81
4 (8t:
2en) d
V.= (80)
Exploration
a
Expansion
a
IE
II
ca
Si
10
I
We denote the implicit set of safe policies in iteration n by aa based on the learned model Fn. Left: search is limited to aa Right: expansion proactively enlarges the safe set and reaches the optimum.
Figure 2:
in constrained tasks may not find an optimal policy because
This choice of the terminal reward keeps the reward structure consistent with the rewards encountered during online rollouts on f. unlike prior works that treat the terminal reward as hyperparame- ter (Thomas et al., 2021; Wagener et al., 2021). Figure I illustrates concrete example of how this design encourages the agent to learn to outperform the prior policy.
and expansion. Probabilistic models have proven particularly effective in the unconstrained setting. Optimistic planning with these models enables principled balance between exploration and exploitation, forming the basis of the celebrated UCRL algorithm (Auer & Ortner, 2006; Curi et al., 2020). However, when safety must hold at every iteration n, exploration-exploiaaion is restricted to the set of policies that are safe with high probability. In particular, this set consists only those policies that satisfy the constraint under the worst-case model in FR.- Crucially, due to pessimism, an optimal policy = may not lie in this set initially, as shown in Figure 2. Nonetheless, this set can be expanded by updating the model using the collected measurements from the environment, even in those regions where rewards are predicted to be low under the agent's model iteration Prior works typically follow strategy (Sui et al., 2015; Berkenkamp et al., 2017: Bura et al., 2022: As et al., 2025b), which begins by learning the dynamics with reward-free trajectory sampling expand the safe set, followed by exploration-exploitaiii
Exploration--expoiiteee
to find an optimal and safe policy. This approach has two limitations: (i) requires agents first learn an auxiliary policy unrelated the task, leading to wasted compute and exploration; (ii) it restricts the theoretical analysis to simple regret, allowing arbitrarily poor performance during exploration long the final policy is near-optimal.
A unified objective. In contrast, SOOPER addresses the apploaaion-exploottion-eeppppppp dilemma through intrinsic rewards (Sukhija et al., 2025) with a single tractable objective:
Th E arg max E,
"F(s,, a.) + (1 texplore + / Aexpmma))lo (s,,
8t+1 Mrn (St, (3,,,ar)) ar) Wes or ~ (-1st) 80 po(-),
where Asxplore and Aaxpand weighting factors for the respective components of the intrinsic reward bonus, encouraging exploration and expansion. In Lemma 12 in Appendix B, we derive these quantities in closed form and prove that Equation (9) indeed enables expansion. Importantly, since

---

Published as a conference paper at ICLR 2026
only augment the reward with exploration bonuses, approximate solution to Equation (9) can be computed efficiently.
Converging to the optimum. By iteratively updating the model F, and planning according to Equation (9) in each episode, SOOPER achieves sublinear cumulative regret and therefore converges for a sufficient number of episodes in Algorithm 2 according to Theorem 2.
Algorithm 2 Safe Online Optimism for Pessimistic Expansion in RL (SOOPER)
Require: Initial dynamics set Fo. pessimistic cost value Q2 policy prior म, safety budget d Initialize experience buffer D
2: for episode n l..... N do
3: Execute Algorithm to collect D,,
On the real environment Refine model
4: Update Fr, 4 1im
5: Obtain ny: Qu,, via Equations (5) and (9)
D Model-based exploration and planning
6: end for
7:
return T
Theorem (Sublinear cumulative regret). Suppose Assumptions hold and the model Fn is well calibrated according to Definition 1. Then, Algorithm guarantees with probability 1
R(N) = =E (J,("e, Jr (Tn, s s (og(N)) VN),
and Je( S) d. Vn Enei (1....,N)
that is, the cumulative regret grows sub-linearts in N, dependent on the maximal information gain I N (og(N), while satisfying the constraint throughout learning by Theorem 1.
Proof sketch. Previous works explicitly expand the safe set of policies via pure exploration. This restricts them to a simple regret analysis, since performance can be evaluated only after the reward- free expansion phase. SOOPER optimizes single objective, thus expansion occurs implicitly, during learning the task. This allows us to extend the analysis to cumulative regret, providing guarantees on the agent's 's performance throughout learning. We bound the cumulative regret in Equation (3) by decomposing the performance gap in each iteration into two terms (Lemma 4). (i) The first term corresponds to the regret on the planning MDP M (cf. Equations (6) and (7)). Specifically, we compare the performance difference between an optimal policy of M and the learned policy n. Since this setting is purely unconstrained, we bound this term following the regret analysis for unconstrained MDPs of Kakade et al. (2020). (ii) Next, analyze the performance gap due to invoking the prior policy & (Algorithm 1). In particular, we study the performance of executing ~. an optimal policy of the true CMDP Mc. however under the safety criterion of Algorithm 1. Intuitively, due to the pessimism of Q instead of performing optimally with -. Algorithm I invokes म, thus leading to suboptimal performance. The gap in performance between executing =ं freely and under Algorithm 1 is bounded by the model uncertainty at iteration (Lemmas 6 and 8). The key intuition is that since -. fact safe, as model uncertainty shrinks. Algorithm needs invoke a less, hence converging to optimal performance, Finally, we show that model uncertainty shrinks as n grows under Assumption due to Chowdhury Gopalan (2017) and that the sum of the above two terms is upper- by the objective in Equation (9) in each episode n. The complete proof is detailed in Appendix B.
Practical implementation. Our practical implementation follows closely Algorithm 2. We in- stantiate a "deep" version of Algorithm that scales nigh-dimensional continuous control tasks, following model-based actor-critic architecture similar to MBPO (Janner et al., 2019a). Line 4 is implemented via a probabilistic ensemble of neural-networks (Chua et al., 2018). our practical implementation we additionally learn the reward and cost functions (see Appendix C). The standard deviation across ensemble predictions is used to estimate the epistemic uncertainty on (Depeweg al., 2018). We found this approach to deliver reliable uncertainty estimates in practice (cf. Section 5 and Appendix D). In Line 5, we solve Equations (5) and (9) using -enerrtee rollouts and by per- forming a fixed number of actor-critic updates described by Janner al. (2019a). Adapting MBPO to our setting is straightforward, as it only requires wrapping the model predictions within the MDP defined by M. This suggests that SOOPER readily extensible to future advancements in deep RI.
7

---

Published as a conference paper at ICLR 2026
SOOPER RaceCar
SAILR
CRPO
Primal-Dual
PointGoal2
CaripoleSwingup
HumanoidWalk
WalkerWalk
1.65
J,
1.4
1.3
1.55
1.3
1.50
1.2
0.9
Jetma)
5
25
Safe
0
-50
100
Unsafe
200
-25
Figure 3: Performance improvement the baseline policy and largest constraint recorded through- out learning. Among all methods, SOOPER remains safe in all tasks while consistently outperforming or being on par with the other baselines when they satisfy the constraints.
5 EXPERIMENTS
We now present our empirical results, demonstrating SOOPER's safety and performance guarantees practice. In the Appendix D, we provide additional ablations. illustrating how each component of SOOPER plays key role in attaining good performance while maintaining safety throughout learning.
Setup. In all experiments, we use the model-based architecture described in Section 4. We compare SOOPER with the following baselines: (i) SAILR (Wagener et al., 2021), state-of-the-art RL algorithm for safe exploration. (ii) CRPO (Xu et al., 2020), CMDP solver without safety guarantees during learning. (iii) Similarly, Primal-Dual (Bertsekas, 2016) is constrained optimization algorithm. which is commonly used as CMDP solver. CRPO and Primal-Dual serve as natural baselines, as they represent the standard approaches practitioners would typically adopt when fine-tuning policies initialized from simulation or offline training. Unless specified otherwise, we use batch of 128 trajectories to obtain empirical estimates of the undiscounted cumulative rewards J.(T) and costs (T) and run each experiment for five random seeds. We slightly abuse notation, that for SOOPER, denotes the policy induced by Algorithm and for SAILR the policy induced by their "advantage-based intervention". All baselines initialized with the same task-specific prior policy.
Safe transfer under dynamics mismatch. We study policies that are trained under mismatch in the dynamics on five tasks from RWRL (Dulac-Arnold et al., 2021). SafetyGym (Ray et al., 2019) and RaceCar (Kabzan et al., 2020). We provide additional details in Appendices E to G about how mismatches in the dynamics are implemented for each task. We refer to Appendix D for additional details on how we train the pessimistic prior policies. We train prior policies under the shifted dynamics (μa. cf. Section 3) and continue learning on the true dynamics. Figure 3 presents the largest cumulative cost recorded the true dynamics throughout training, along with the relative performance improvement at the end of training compared to the policy prior before online learning. As shown, SOOPER satisfies the constraints throughout learning all of the tasks. For each task. among all algorithms that satisfy the constraints, SOOPER is consistently on par or outperforms the baselines. This demonstrates SOOPER's ability to transfer safely under distribution shifts.
Scaling to vision control. Next, demonstrate that SOOPER scales to control tasks with image-based observations. Specifically, we use the CartpoleSwinge task from Dulac- Amold et al. (2021) and train policies that operate on three temporally-stacked 64 grayscale images as depicted in Fig- ure 4. As in Figure 3, we introduce distribution shift in the dynamics and pretrain a vision policy under the perturbed dy- namics using DrQ (Yarats et al., 2021). We then train SOOPER by learning the dynamics model directly on the embeddings pro- duced by the pretrained DrQ vision encoder. Figure 5b shows that even when trained on these embeddings. SOOPER satisfies the safety constraints and achieves near-optimal performance.
Figure 4: Image observations for the CartpoleSwingup task. The goal is to swing the pendulum to the upright position while avoiding range outside the vertical red lines.
8

---

Published as a conference paper at ICLR 2026
SOOPER
SAILR
CRPO
Primal- Dual
Baseline
20
MW
750
(T(
20
(
(.(
500
200
10
10
250
200
200
500
500
Iteration
Iteration
Iteration
Iteration v2
(a) Safe online learning from ofline--trined policy.
(b) Vision control task.
Figure 5: Learning curves of the objective and constraint when learning from offline and vision policies. SOOPER satisfies the constraints while significantly improving the initial prior policy.
SOOPER
CRPO
Baseline
Goal
SOOPER
100
40
(x(Ta)
20
CRPO
Obstacles
100
Iteration n
Figure 6: Safe exploration on real hardware with SOOPER. We report the mean and standard error across five seeds of the objective and constraint measured on the real system. SOOPER learns to improve over the prior policy while satisfying the constraints throughout learning. Video of training.
Safe offline-to-online We evaluate SOOPER when initialized with a conservative policy using only offline data. To this end, collect fixed offline data of 2M transitions from the PointGoall task of SafetyGym. We train pessimistic prior policy offline by following MOPO (Yu et al., 2020), well-established model-based offline RL algorithm. We use Primal-Dual optimizer solve the constraint optimization problem with this offline data. We found this approach to yield policies that satisfy the constraint while still delivering nontrivial performance. We use these policies to initialize all the baselines for our experiments. Figure 5a presents the objective and constraint during online learning. Notably, SOOPER outperforms all the baselines while maintaining safety.
Safe online learning on real hardware. Finally, evaluate SOOPER when trained online directly on real-world robotic hardware: a highly dynamic, remote-controlled race car operating 60 Hz. The task reach designated goal position located behind three obstacles (tires) that must be avoided (illustrated in Figure 6). This setting is particularly challenging due to stochasticity introduced by grgufrreyenccy control and delays in both actuation and motion-capture measurements. We initialize training with a prior policy derived from a first- princciless simulator (Kabzan et al., 2020), and then fine-tune it on the physical system using trajectories collected in real time. We repeat the experiment with five random seeds and report cumulative rewards and costs after each iteration in Figure 6. As shown, after training, SOOPER is roughly twice as good as the prior policy in terms of rewards while satisfying the safety constraint throughout learning. In Appendix D, extend this experiment to the offline setting, and report results when using prior policy trained offline with data collected from the real system.

---

complex tasks Published as a conference paper at ICLR 2026
6 CONCLUSION
In this work, we address a critical challenge in systems that learn autonomously, namely, maintaining safety during learning. We present SOOPER, novel algorithm that provably improves performance over policies trained in simulation or offline, while satisfying safety constraints throughout learning. SOOPER is simple implement, achieves state-of- of-tthe-rt performance and consistently maintains safety when deployed in practice. In addition, we establish new theoretical result on the cumulative regret, enabling guarantee of good performance during learning and not only at the last iteration. These contributions advance our understanding of safe exploration and make important step towards deployable RL. Nonetheless, many important challenges remain. In terms of theory and methodology, extending our work to high-prou high-probability constraints over particular states and developing new methods for the non-episodic setting, where agents learn from a single trajectory without resets, two challenging yet crucial problems. On the practical side, considering with more powerful models is promising direction for future work.
ACKNOWLEDGMENTS
We would like to thank Armin Lederer, Bruce D. Lee and Dongho Kang for insightful discussions during the development of this project. We thank the anonymous reviewers for their valuable com- ments and suggestions. This project has received funding from a grant of the Hasler foundation (grant no. 21039) and the ETH AI Center.
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference on Machine Learning, 2017. (Cited on page 2)
E. Altman. Constrained Markov Decision Processes. Chapman and Hall, 1999. (Cited on pages I and 3)
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety, 2016. (Cited on pages and 3)
Yarden As, Ilnura Usmanova, Sebastian Curi, and Andreas Krause. Constrained policy optimization via bayesian world models. International Conference on Learning Representations, 2022. (Cited on page
Yarden As, Chengrui Qu, Benjamin Unger, Dongho Kang, Max van der Hart, Laixi Shi, Stelian Coros, Adam Wierman, and Andreas Krause. SPiDR: A simple approach for zero-shot safety in sim-to-real transfer. In International Conference Neural Information Processing Systems, 2025a. (Cited on pages 4 and 36)
Yarden As, Bhavya Sukhija, Lenart Treven, Carmelo Sferrazza, Stelian Coros, and Andreas Krause. Actsafe: Active exploration with safety constraints for reinforcement learning. In International Conference on Learning Representations, 2025b. (Cited pages 2, 6, and 20)
Peter Auer and Ronald Ortner: Logarithmic online regret bounds for undiscounted reinforcement learning. In B. Schölkopf, J. Platt, and T. Hoffman (eds.), International Conference on Neural Information Processing Systems, 2006. (Cited page 6)
Felix Berkenkamp, Andreas Krause, and Angela P. Schoellig. Bayesian optimization with safety constraints: safe and automatic parameter tuning in robotics. Machine Learning, 2016. (Cited on
page 2)
Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model- based reinforcement learning with stability guarantees. In International Conference on Neural Information Processing Systems, 2017. (Cited on pages 2, 3, and 6)
D. Bertsekas. Nonlinear Programming. Athena scientific optimization and computation series. Athena Scientific, 2016. (Cited on page 8)
Homanga Bharadhwaj. Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Animesh Garg. Conservative safety critics for exploration. In International Conference on Learning Representations, 2021. (Cited on page 2)
10

---

Published as a conference paper at ICLR 2026
Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela Schoellig. Safe learning in robotics: From learning-bast based control to safe reinforcement learning. Anmual Review of Control, Robotics, and Autonomous Systems, 2022. (Cited on page
Archana Bura, Aria Hasanzadeeonnuz, Dileep Kalathil, Srinivas Shakkottai, and Gean-Francois Chamberland. Dope: Doubly optimistic and pessimistic exploration for safe reinforcement learning. International Conference on Neural Information Processing Systems, 2022. (Cited on pages and 6)
Yinlam Chow, Ofir Nachum, Edgar based approach to safe reinforcement learning. International Conference on Neural Information Processing Systems, 2018. (Cited on page
and Mohammad Ghavamzadeh. A Iyapunov-
Sayak Ray Chowdhury and Aditya Gopalan. On kernelized multi-armes bandits. In International Conference on Machine Leurning, 2017. (Cited pages 7, 30, and 31)
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. International Conference on Neural Information Processing Systems, 2018. (Cited on page
Sebastian Curi, Felix Berkenkamp, and Andreas Krause. Efficient model-based reinforcement learning through optimistic policy search and planning. In International Conference on Neural Information Processing Systems, 2020. (Cited on pages and 6)
Sebastian Curi, Armin Lederer, Sandra Hirche, and Andreas Krause. Safe reinforcement learning via confidence-based filters. In Conference on Decision and Control. IEEE, 2022. (Cited on pages and 3)
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint 11X1v:1801.08757, 2018. (Cited
on page
David Dalrymple, Joar Skalse, Yoshua Bengio, Stuart Russell, Max Tegmark, Sanjit Seshia, Steve Omohundro, Christian Szegedy, Ben Goldhaber, Nora Ammann, et al. Towards guaranteed safe ai: framework for ensuring robust and reliable systems, arXiv preprint arXiv:2405.0m 06664, 2024.
(Cited on page 1)
Stefan Depeweg, Jose-Miguel Hernandez-Lobato, Finale Doshi-Velez, and Steffen Udluft. Decom- position of uncertainty in bayesian deep learning for efficient and risk-sensitive learning. In International conference machine learning, 2018. (Cited on page 7)
Dongsheng Ding, Xiaohan Wei, Zhuoran Yang. Zhaoran Wang, and Mihailo Jovanovic. Provably efficient safe exploration via primal-dual policy optimization. In International conference on artificial intelligence and statistics, 2021. (Cited on page 2)
Gabriel Dulac-Amold, Nir Levine, Daniel J. Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. An empirical investigation of the challenges of real-world reinforcement learning, 2021. (Cited on pages 2, 8, and 39)
Benjamin Eysenbach, Shixiang Gu. Julian Ibarz, and Sergey Levine. Leave no trace: Learning to reset for safe and autonomous reinforcement learning. International Conference on Learning Representations, 2018. (Cited on page 2)
Javier García, Femn and Fernández. A comprehensive survey safe reinforcement learning. Journal of Machine Learning Research, 2015. (Cited on page 3)
Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll. A review of safe reinforcement learning: Methods, theories and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. (Cited on page 3)
Lukas Hewing, Juraj Kabzan, and Melanie N Zeilinger. Cautious model predictive control using gaussian process regression. IEEE Transactions on Control Systems Technology, 2019. (Cited on page 2)
11

---

Published as a conference paper at ICLR 2026
Weidong Huang, Jiaming Ji, Chunhe Xia, Borong Zhang, and Yaodong Yang. Safedreamer: Safe rein- forcement learning with world models. In International Conference on Learning Representations. 2024. (Cited page 2)
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When trust your model: Model-based policy optimization. International Conference Neural Information Processing Systems, 2019a. (Cited on page 7)
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: model- based policy optimization. In International Conference Neural Information Processing Systems, 2019b. (Cited on page 36)
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline ri? In International conference on machine learning, 2021. (Cited on page 4)
Juraj Kabzan, Miguel Valls, Victor JF Reijgwart, Hubertus FC Hendrikx, Claas Ehmke, Manish Prajapat, Andreas Bühler, Nikhil Gosala, Mehak Gupta, Ramya Sivanesan, et al. Amz driverless: The full autonomous racing system. Journal of Field Robotics, 2020. (Cited on pages 8, 9, and 40)
Sham Kakade, Akshay Krishnamurthy, Kendall Lowrey, Motoya Ohnishi, and Wen Sun. Information theoretic regret bounds for online nonlinear control. In International Conference on Neural Information Processing Systems, 2020. (Cited on pages 3, 7, 18, 20, 23, and 25)
Johannes Kirschner, Mojmir Mutny, Nicole Hiller, Rasmus Ischebeck, and Andreas Krause. Adaptive and safe Bayesian optimization in high dimensions via one-dimensional subspaces. In International Conference on Machine Learning, 2019. (Cited on page 2)
Toshinori Kitamura, Tadashi Kozuno, Wataru Kumagai, Kenta Hoshino, Yohei Hosoe, Kazumi Kasaura, Masashi Hamaya, Paavo Parmas, and Yutaka Matsuo. Near-optimal policy identi- fication in robust constrained markov decision processes via epigraph form. arXiv preprint arXiv:2408. 16286, 2024. (Cited on page 4)
Torsten Koller, Felix Berkenkamp, Matteo Turchetta, and Andreas Krause. Learning-borl based model predictive control for safe exploration. IEEE Conference on Decision and Control, 2018. (Cited on page 2)
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International Conference on Machine Learning, 2018. (Cited on page 4)
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. International Conference on Neural Information Processing Systems, 2017. (Cited on page 4)
Shane Legg. System two safety. The Alignment Workshop, 2023. (Cited on page 1)
Eric Lehman, Leighton Thomas, and Meyer Albert. Mathematics for Computer Science. 2018. (Cited on page 30)
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learnnn Tutorial, review, and perspectives open problems. arXiv preprint arXiv:20055006643, 2020. (Cited on page 4)
David Lindner, Xin Chen, Sebastian Tschiatschek, Katja Hofmann, and Andreas Krause. Learning safety constraints from demonstrations with unknown rewards. In International Conference on Artificial Intelligence and Statistics, 2024. (Cited page 4)
Qiang Liu and Dilin Wang. Stein variational gradient descent: general purpose bayesian inference algorithm. International Conference on Neural Information Processing Systems, 2016. (Cited on page-) 4)
Yongshuai Liu, Jiaxin Ding, and Xin Liu. Ipo: Interior-point policy optimization under constraints. In AAAI, 2020. (Cited page 2)
12

---

Published as a conference paper at ICLR 2026
Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bi, and Ding Zhao. Constrained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning, 2022. (Cited on page
Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu, Tingnan Zhang, Jie Tan, et al. Datasets and benchmarks for offline safe reinforcement learning. arXiv preprini arXiv:2306.09330
2023. (Cited on page
Nikola Milosevic, Johannes Müller, and Nico Scherf. Embedding safety into rl: new take on trust region methods. arXiv preprint arXiv:2411.02990 2024. (Cited on page 2)
Teodor Mihai Moldovan and Pieter Abbeel. Safe exploration in markov decision processes. arXiv preprint arviv /205. 220.48100, 2012. (Cited on page 2)
Tingting Ni and Maryam Kamgarpour. safe exploration approach constrained markov decision
processes. In International Conference on Artificial Intelligence and Statistics, 2025. (Cited on
page 2)
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. (Cited on page 1)
Manish Prajapat, Matteo Turchetta, Melanie Zeilinger, and Andreas Krause. Near-optimal multi-agent learning for safe coverage control. International Conference on Neural Information Processing Systems, 2022. (Cited on page 2)
Manish Prajapat, Amon Lahr, Johannes Köhler, Andreas Krause, and Melanie N Zeilinger. Towards safe and tractable gaussian process-based mpe: Efficient sampling within a sequential quadratic programming framework. In IEEE Conference on Decision and Control. IEEE, 2024. (Cited on
page 2)
Manish Prajapat, Johannes Köhler, Amon Lahr, Andreas Krause, and Melanie N Zeilinger. Finite- sample-based reachability for safe control with gaussian process dynamics. arXiv preprint arXiv:2505.07594, 2025a. (Cited on page 2)
Manish Prajapat, Johannes Köhler, Matteo Turchetta, Andreas Krause, and Melanie N. Zeilinger. Safe guaranteed exploration for non-linear systems. IEEE Transactions on Automatic Control, 2025b. (Cited on page 2)
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014. (Cited on page 3)
James Queeney and Mouhacine Benosman. Risk-averse model uncertainty for distributionally robust safe reinforcement learning. In International Conference on Neural Information Processing Systems, 2023. (Cited on pages and 40)
Carl Edward Rasmussen and Christopher K. I. Wiliams. Gaussian Processes for Machine Learning. 2006. (Cited page 4)
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking Safe Exploration in Deep Reinforce- ment Learning. 2019. (Cited on pages 2, 8, and 39)
Andreas Schlaginhaufen and Maryam Kamgarpour. Identifiability and generalizabilityy in constrained inverse reinforcement learning. In International Conference on Machine Learning, 2023, (Cited on page4)
Laixi Shi and Yuejie Chi. Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity. Journal of Machine Learning Research, 2024. (Cited on page
Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Pessimistic leaannnng for offline reinforcement learning: Towards optimal sample complexity. In International conference on machine learning. (Cited 4)
13

---

Published as a conference paper at ICLR 2026
David Silver and Richard Sutton. Welcome to the era of experience. Google AI, 2025. (Cited on page !)
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of Go without human knowledge. Nature, 2017. (Cited on page 1)
Thiago Simão, Nils Jansen, and Matthijs TJ Spaan. Alwayssafe: Reinforcement learning without safety constraint violations during training. 2021. (Cited on page 2)
Aivar Sootla, Alexander Cowen-Rivers, Taher Jafferjee. Ziyan Wang, David H Mguni, Jun Wang. and Haitham Ammar. Saute rl: Almost surely safe reinforcement learning using state augmentation. In International Conference on Machine Learning. 2022. (Cited on page 2)
Krishnan Srinivasan, Benjamin Eysenbach, Sehoon Ha. Jie Tan, and Chelsea Finn. Learning to be safe: Deep rl with safety critic. arXiv preprint 20 14603, 2020. (Cited on page 2)
Yanan Sui, Alkis Gotovos, Joel Burdick, and Andreas Krause. Safe exploration for optimization with gaussian processes, In International Conference on Machine Learning, 2015. (Cited on pages and 6)
Bhavya Sukhija, Lenart Treven, Cansu Sancaktar, Sebastian Blaes, Stelian Coros, and Andreas Krause. Optimistic active exploration of dynamical systems. In International Conference on Neural Information Processing Systems, 2023a. (Cited pages 25 and 27)
Bhavya Sukhija, Matteo Turchetta, David Lindner, Andreas Krause, Sebastian Trimpe, and Dominik Baumann. Gosafeopt: Scalable safe exploration for global optimization of dynamical systems.
Artificial Intelligence, 2023b. (Cited on page
Bhavya Sukhija, Lenart Treven, Carmelo Sferrazza, Florian Dorfler, Pieter Abbeel, and Andreas Krause. Optimism via intrinsic rewards: Scalable and principled exploration for model-based reinforcement learning. 7th Robot Learning Workshop: Towards Robots with Human-Level Abilities, 2025. (Cited on pages 6, 16, 17, 22, 24, 26, and 31)
Hao Sun, Ziping Xu, Meng Fang, Zhenghao Peng. Jiadong Guo, Bo Dai, and Bolei Zhou. Safe exploration by solving early terminated mdp. arXiv preprint arXiv:2 2021. (Cited on page 2)
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. 2015. (Cited on page 3)
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint 18X1:0/801.00690 2018. (Cited on page 39)
Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E. Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery rl: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, 2021.
(Cited on page 2)
Garrett Thomas, Yuping Luo, and Tengyu Ma. Safe reinforcement learning by imagining the near future. In International Conference on Neural Information Processing Systems, 2021. (Cited on pages and 6)
Matteo Turchetta, Felix Berkenkamp, and Andreas Krause. Safe exploration in finite markov decision processes with gaussian processes. International Conference Neural Information Processing Systems, 2016. (Cited on page 2)
Ilnura Usmanova and Kfir Yehuda Levy. Safe primal-dual optimization with a single smooth constraint. arXiv preprint 2rXiv:2505.09349 2025. (Cited page 2)
Ilnura Usmanova, Yarden As, Maryam Kamgarpour, and Andreas Krause. Log barriers for safe black-bor box optimization with application to safe reinforcement learning. Journal of Machine Learning Research, 2024. (Cited pages and 36)
14

---

Published as a conference paper at ICLR 2026
Kim Peter Wabersich and Melanie N Zeilinger. A predictive safety filter for learning-basee control of constrained nonlinear dynamical systems. Automatica, 2021. (Cited on page 2)
Akifumi Wachi and Yanan Sui. Safe reinforcement learning in constrained markov decision processes.
In International Conference on Machine Learning. 2020. (Cited on page 2)
Akifumi Wachi, Yanan Sui, Yisong Yue, and Masahiro Ono. Safe exploration and optimization of constrained mdps using gaussian processes. AAAI, 2018. (Cited page 2)
Akifumi Wachi, Wataru Hashimoto, Xun Shen, and Kazumune Hashimoto. Safe exploration in reinforcement learning: generalized formulation and algorithms. International Conference on Neural Information Processing Systems, 2023. (Cited on pages 2 and 3)
Akifumi Wachi, Kohei Miyaguchi, Takumi Tanabe, Rei Sato, and Youhei Akimoto. A provable approach for end-to-end safe reinforcement learning, arXiv preprint arXiv:2505.21882, 2025. (Cited on page 4)
Nolan Wagener, Byron Boots, and Ching-An Cheng. Safe reinforcement learning using advantage- based intervention. In International Conference Machine Learning, 2021. (Cited pages 2, 3, 6,
and 8)
Tengyu Xu, Yingbin Liang, and Guanghui Lan. Crpo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning, 2020. (Cited on page 8)
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021. (Cited on page 8)
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. International Conference on Neural Information Processing Systems, 2020. (Cited on pages and 36)
Zhengfei Zhang, Kishan Panaganti, Laixi Shi, Yanan Sui, Adam Wierman, and Yisong Yue. Dis- tributionally robust constrained reinforcement learning under strong duality. In Reinforcement Learning Conference. (Cited on page 4)
Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang. Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu. Safe offline reinforcement learning with casibblittyguuiee diffusion model. In International Conference on Learning Representations, 2024. (Cited on page 4)
15

---

Published as a conference paper at ICLR 2026
A SAFETY GUARANTEE
First, we derive Lemma that the pessimistic safety-condition with online cost tracking (Theorem 1) guarantees safety given the pessimistic cost value function V of Equation (4).
Lemma 1. Suppose Assumptions to 4 hold and Fn is well-calibrated Vn = .,....N according to Definition Given the safe prior â, the rollout of the policy nn satisfies the safety constraint with probability 1 - on f for every episode I....,,,, N, if the actions are executed for all timesteps t according the pessimistic coneytcondiiinn
(T((1sa) (ss au Cet Fn) d
St,
(f()))
otherwise.
where
B (88,, 04, Cet Fr) = eet ( 01) + 7 max Eas (+(( + wi)
(10)
fETn
indicates the sum of the accumulated costs Cct Ex c(st, ar) until 1, the immediate cost at and the worst-case pessimistic cost value V defined in Equation (5).
Proof. We prove the safe rollout of n. for the unknown f by induction.
1. Base Case (t 0): By definition of #u. only execute du (-((s))) if ((Su, do, Ceo: Fr)) < d with cço 0 and otherwise the safe prior म. By definition of &, either ag is safe to execute with
c(50: un) + ? max Es fEF.
V. (f(s0, ao) + wn)
(11)
and otherwise executing ao #sp) is safe since (so) < d by Assumption 4.
2. Inductive Step: Given the agent is safe up to timestep t i...
Cer-1 + 7-- c(s--1, + r max E.
v (f( a- + wi- < <d.
(12)
JEF
we show constraint satisfaction for time t, satisfying
d> ces + c/c(s,, a;) + + ++ E. [v (f(ss, ar) + +w)]
(13)
Using 7,, we either execute if Equation (13) is satisfied and therefore is safe by definition. Otherwise, we invoke the prior à = (st). Using cet = Cet-1 + ( 0t-1) we obtain
Cet fe C ài) + ++1 max E.,,
V=* az) + wi)]
(14)
FEF
= of c(81-1, 4-11 + + eec(s1, âz) - tt1 JeF.
max En (f ae) + wi)
(15)
S cct=l 2
c(t- (--1) + ldt max En feF.
(( + w--i) - d.
(16)
Next, we show that adding the cost penalties in Q (Equation (5)) upper-bounde the true cost value, using Sukhija et al. (2025, Lemma A.5):
Lemma 2. Suppose Assumptions to 3 hold and Fn is well-calibrated by Definition 1. Given
141 max Ens V (=*(((() ar) + we) feF.
Ex
Σ
Te(sr, ar)
ratt1
s.t. 8711 = f(sst ar) + Wrr, O, = ;
$111 = f() + wes
Apersimism = 7 (1+ + vds) (8)
1 1-
σ
16

---

Published as a conference paper at ICLR 2026
then for all episodes
1, ,, N with probability of - 6:
y+1 max
Bu
V.r (f(ss,, ax) + wi) 7 Apessimism cacisiimmmmllnnnnnne (sn arll
FeFa
sc
?" ar) + Apeasimiim Aannimimmllonsss (sn a+)1)]
E
8,41 = p ar) + Wr. 0, a 8141 = pn (n( 0e) + we-
Proof. Let us denote the value following the nominal dynamics pn of Fr as
(s) = Ex 7o(s ax) | 80 =
(17)
8. t. 81+1 = Pn (8t, a4) + wi,
a= == ((e)):
and the upper bound on the cost value following the worst dynamics f E Fn by
V. max Vo((s) 1E$$
Σ 7'c(ss, ar) s0 =
JeFo
(18)
s.t. 56+1 = f(ss, a+) + Wer Following Sukhija et (2025, Lemma A.5), the difference between these values is bounded by
at = #st)).
V.((s) - V. (s) S Apessimism Σ Il atll
so =
(19)
8t-1 Mn ax) We,
ar म (st).
Bringing V=* to the other side and using the definition in Equation (17) results in:
(+*(s) S V. (s) + Apessimism E
so =
(20)
= Ex Σ 1==
o (c(se, at) + Apessimismml (st, at)l
(21)
8. 81+1 = Mn (St, at) + we, at 1==1(0 #st).
(22)
Further, we use this upper bound having c(s;, az) already observed and rewriting
++ max E,, EEFn
V (f(s), ar) + wi)] = >'(=((()) - 3 ((c(s,a))
(23)
Ex
7" (c(sT, ar) + Apessimisml|/ww (o (+)l))
7'((s,,
(24)
< E
7 (c(ST, ar) + Apensnissmllall (0, (.)., ar)ll) + Apeessimiim ( ar)ll.
(25)
7=
8. t. 8741 (1 a,) + Wr, #sr)), 8t-1 = (n at) + wes
(26)
where step (i) follows from using the upper bound in Equation (21).
Theorem (Safety guarantee). Suppose Assumptions I hold and F. well-calibrated Vn 1,..., N according to Definition 1. If actions are executed for all timesteps i according to
(T((()))) (m)(((5)))) if Ø(8t, Cet, Q <d
The (a Cet, 2²,n)
( otherwise.
where is the discounted sum of the realized accumulated cost Cet == 2 Tc(s,, until t t- 1 and the pessimistic cost value Q such that (0st, Cet, Q == Cet + Q (St, 0z) with
factor Apersimism = - 2 (1+vd))0w(6) where Ö max Maxxcrs kmar). Then, Algorithm I satisfies the safety constraint with probability 8 on for every episode n. 1,..., N.
17

---

Published as a conference paper at ICLR 2026
Proof. Theorem 1 is proven using Lemmas and 2. Consider the accumulate cost (s ae, Fri,
from Lemma 1
(3 aes Ceti Fn) =c<t + + ((((8, + +, s+1 max
Ew (f(e,,)) + wi]
(27)
JEFn
(
Sece + (c(s,, az) + Apessimism (n a+)l)
+ E
Σ T=1+1
Y
ar) + Apessimisoal Ion (sr, 0r ar)ll)
(28)
(i)
(ii Cet + 7' 2n (st, ax) (( a2 Cee, Q < d.
(29)
Step (i) follows from the upper bound on the cost penalties in Lemma 2, and Step (ii) and (iii) follow from the definitions of Q and ((2,, ae, Cee, Q from Equation (5) and lemma 1.
B REGRET ANALYSIS
Overview of proof. Theorem 2 states that the tractable objective in Equation (9) achieves sublinear cumulative regret while satisfying the safety constraint. We start our proof by decomposing the cumulative regret into per-episode regret terms. Using Lemma 3, we lower-bound the performance of the true system M, by the performance on the planning MDP M. This bound allows us to decompose the per-episode regret into components in Lemma 4 that rollout vs. optimal rollout. The first regret term captures the performance gap between the optimal policy n5 and the same policy executed via Algorithm on Me- The difference arises from invoking the prior policy मे due to uncertainty in the learned dynamics. We express this gap in Lemma 5 in terms of action differences between πT and the potentially used safe prior म. By Theorem these action differences can be related in Lemma 6 via the safety criterion & to the accumulated model uncertainty along potentially unsafe trajectories starting with n- (at(8t) at step f, followed by Åt Since it may be unsafe to execute them, we instead express their cumulative uncertainty in terms of the corresponding safe trajectories starting with A (s;) at step (Lemma 7). This is achieved using the difference of Gaussians (Kakade et al., 2020, Lemma C.2). Consequently, we establish in Lemma 8 an upper bound on the first regret term, which depends on the model uncertainty along the safe rollout of n² under Algorithm 1. (ii) Planning model vs. true dynamics. The second regret term quantifies the value gap between an optimal policy for the planning MDP M (with true dynamics f) and the policy n. obtained by optimizing the nominal model Phn in episode n.. This gap can be bounded in terms of the model uncertainty evaluated along the optimal policy x' when it is safely executed using Algorithm 1. By Lemma 12, the tractable objective in Equation (9) an optimistic upper bound on the best achievable performance across all models F,, Consequently, Lemma 13 provides a bound on the sum of the two per-episode regret components along the safely rolled-out behavioral policy n using Algorithm 1. Summing these bounds over all episodes yields Theorem 2, which states that the objective in Equation (9) attains sublinear cumulative regret.
be bounded separately: (i) Safe
To decompose R(N), we first establish a lower bound on the performance of R,, obtained by safely rolling out n. on Me with Algorithm using T.
Planning at the lower performance bound. Given the policy T. on M, we derive in Lemma 3 the performance bound of the safe policy म. on Me with dynamics ff using the pessimistic value function V (so) defined in Equation (8).
Lemma 3. Suppose Assumptions I to 3 hold and the model Fn, is well-calibrated according to Definition! 1. Then the performance J, f) of the safe policy Rn induced by Algorithm I on Me is lower-bounded by the performance JT(Tn: . f) of the policy nn M following the true dynamics f:
Jr( J).
Proof. We consider for all timesteps k two cases:
1. Never fall buck to prior (i.e. m. is always safe). Then, by definition of M,
Je(Tnn f) = Ex
Σ r 7'r(st, at) = Jr (r
(30)
18

---

Published as a conference paper at ICLR 2026
This equality follows by definition from M and M. being identical in case the agent never falls back to the safe prior î during the rollout and therefore म. 7n.
2. Agent falls back to safe prior (i.e. 8 action à ((s)) is executed instead of potentially unsafe at). For the performance J, (Rnz f) of Tn on Me we get by combining the policy values of nn and
I
J, (T, f) = E Exn..0
Σ
Entr(se, az) + (sx),
(31)
whereas for the performance of Jr Jz( f) of on on M, we get
Jz Jz( f)
E 50
y (f)))) + F az) +
(32)
Σ >/((s,, at) + (sn)
(33)
Step (i) follows from the modified transitions in Equation (6) to the absorbing state s as soon as the agent would fall back to the safe prior ft, and (ii) follows from the terminal reward given in Equation (7) for transitioning to s;. Since v is defined pessimistically in Equation (8), it holds
that V (s) S V and thereby follows
JF(Tas f) S BT...D
Σ ('r(st; ail + + (sx)) = (Fn, t=
(34)
which completes the proof.
Regret decomposition. Lemma 3 enables us to decompose R(N) into two separate terms. The first term models the regret executing मn on Mc. caused by falling back to the safe prior when safely rolling out with Algorithm 1. The second component determines the (constraint-Fie regret of Th on M.
Lemma 4. Suppose Assumptions 1 to 4 hold and the model F,, is well-calibrated according to Definition Let n. be the optimal policy on Me with J . f) (a. JT f) and applying Algorithm I n yields H.,. with J, f) for the current learned model Fr episode n. Further, let nt be the optimal policy on M that has J+(mt" = max Jr(T,, for and Qe in episode n. Given Je(Tn. f) denotes the performance of the behavioral policy on on M, the cumulative regret R(N)= Em (Jr
Jr f)) of nn on MM can be decomposed into:
N
N
R(N) S Σ
(J Jr
+
(J((T", -J Ji(Tmn)))
==1
n=1
At. .PP.R of FC with
A4 I.E.R. of v,,0
with the per-episode regret (P.E.R.) terms A and A related to the safe prior and the regret on M.
Proof. Starting with the cumulative regret definition in Equation (2) for Mc. upper bound R(N) using
R(N)
(J J - Σ
(JA(Ta) Je(Tnn f))
(35)
n=1
Σ (J J, +J Jr f) (T(TTn f)),
(36)
where step (i) follows applying Lemma and (ii) introducing zero term by adding and subtracting J, f). Since M. and M are identical for safe policies (e.g. #*,)) by not transitioning to s (see Equations (6) and (7)) and n² is the optimal policy on M with F, in episode
J, ( f) J= J=(m", ,)).
(37)
19

---

Published as a conference paper at ICLR 2026
Thus, obtain the following upper bound of R(N) by plugging Equation (37) into Equation (36):
N (U, Jr(T f) + Σ (Js(TT, .f) Jz( f)).
R(N) & n=1
(38)
A regret bound. Recall from Lemma 4 that the episode regret of executing πī with & for Fn in episode n on the planning MDP Me is A, = (+())) (,."",,) We rewrite A, in terms of the distance between the optimal policy æ² and the safely rolled-out policy T2,, in Lemma 5.
Lemma 5. Suppose Assumptions I to 4 hold and the model F,, is calibrated according to Definition 1. (T f) admits the bound
Then the per- episode regret A = (T
A S Exg,00 ((
Typ + RL, ?
[lla] - Taull
7 (()T)())a)))
1=
Ben Ben (ea
81+1 = [(8, ai) + wr:
af ~ (11())
with the Lipschitz constants Loy T.g for the reward and dynamics, and R max Romnn kmar).
Proof. We bound the episodic regret with the advantage formulation (As et al., 2025b, Lemma A.2)
A Er
Bag ~x; (= (-1ae) [Ar St, at)] 7t
(39)
1=0
with A St, at)
r(s,, ar) Elw VV, ($t+1) VØ and 8141 f(( + Wt.
( A = E
Σ 7't Eag Eagami((ee)) (rlon, a²)
(40)
+ + Beesulenn (v - V, (st)
(1)
BBg 80
re (2177())))) [(r(se, (,) r(se, [(an)]] a ((m
(41)
++ (Beenlleo tem
(8(+1)] Eeetl lent w (Vr Te. (((+)(ss)))))))
+ Egg Esg 10
(iii
< E nm
Σ
LB of (-lee)
[minfLrllao"" Tam Rmazt]
(42)
a 2 (198)
max seth (a we [R(++ B. Ble [[((s+1))}
1=0
Lylla - Tan!l Ill..
x
a->x'' (-1s.) a (1se))
where step (i) follows from the advantage definition and (ii) uses the value function to derive the dependence the reward difference in timestep We rewrite the advantage in terms of the expected action difference in step (iii) using the expectation difference under two Gaussians (Kakade et al.,
2020, lemma C.2) with R(s) (v_ fe (s)) Step (iii) uses the Lipschitz constants L,, Lg for the reward and the dynamics (Assumption 2). Further, we bound this by:
A <Eg
Σ t=0
x 'E a mnin - a'all Rmaz)
(43)
80
20

---

Published as a conference paper at ICLR 2026
L [|
dzmll
+ R
E
og
min
7 1=0
RLs Σ
E
L,
ai (1sx) BBB
|lla &2uull
(44)
80
1=0
~ No (-10))
Eu Is, max Rmmxx kmax) due to and in (ii) we drop the min operator due to non-negativity.
Step (i) uses V max ( Ber [R( (1(-s-1]],,
(R(s++))]} & R , with R
Lemma 5 expresses the regret in terms of the difference between actions taken by an optimal policy Tē and those taken by के.. in episode with model Fr,. In the following, we use Lemma to relate A to model uncertainties "collected" along trajectories induced by these policies. Bounding these uncertainties ensures that the learner improves its model of the dynamics after each episode.
Lemma Suppose Assumptions I to 4 hold and the model Fn well calibrated according to Definition 1. Given & is defined as in Theorem 1, A depends on the accumulated and discounted uncertainty Eg along the trajectories of action a and the safe â, and the diameter of the bounded action space DA Given Vn= = (1 + Apeszimism) Apessimism with Apersimism being defined as in Theorem 1,
A I BFg BRi [L
Rmoz Jog
Vn (e(e,))))
DDaain Bauna
1-7)
dc
1=0
00
E, f) = (8c
Σ
1 (80, + wr B/y: 0l (s,))]|| 8741 =
r=1+1
and de min((um
(c<t + 7 (s(s + t Bw., (s(+1))))) e R
Proof. We can upper bound the expected difference between the policies by the action bound DA times the probability P of falling back to the safe prior
D= (st)
=E a: (clee)
[lla - az,ll]]
(45)
S D A Pa atllee
(Ges + Co az) 2 d)
(46)
(i)
DA PAPar (-
Cet + 7 7'c(se,
(47)
t (ww. VV= (un an) + we)] + Apossimism En (ac, pn)) 2 dd)
with 1/4 (n( a²) + Wh. Ee(at, Mm) (s, ai)l + Er=(+1 Blyr~11oo ((l and (sp) = Ei 7'c((%, a4) following the model dynamics (4n- Step (i) follows from the switching
criterion in Theorem and step (ii) by the definition in Equation (5). Next, we relate the uncertainty
along trajectories under the model dynamics (4, to those under the true dynamics f by
( DD DAPF a (1 (<<e + 7' d'c(ss, a.) + + (Bor. IV. (pn(s), + wi)l cee
Dac
(st)
(48)
+ E. VV. (f(s + wi)l E., IV. (f ar) + we)l
Apessimism (Ee, + Ex f) - Ea f1) 2
(8)
< D PPPeregtHa)))
Cet Cer ax) e+ (Bwn (V((((,, + wi)l
(49)
+ (1 +1+ Aasssismmssssssiiiiieeeipp 2 2d)
(10)
< DAP. a ~z((11)) (cc(tn'csssaa)) + +e+1 (Bui IV= (s1+1)l + t,,Eo((r,)) 2
(50)
21

---

Published as a conference paper at ICLR 2026
We first expand the inequality in (i) with two terms. In step (ii), we bound the difference of the value and the uncertainty for the different dynamics using Sukhija et al. (2025, Lemma A.5). In (iii) we Apessimism /pessimism and sample according to the true dynamics 84+11 f(,,, az twp-
define v,, (1 (1+ppsssimism))
Consider the set of all policies that are safely reachable given क
I = { [c/eer + (c((s,,))) + Ew. C.u.I4 IV= (sc+1)l < d, Vt}.
(51)
Since To is strictly feasible have Exg (<<e + //((s az)] + 7 Eur [Ve' (8t+1) d; in addi- tion, (s) I Ve"e (s) for all initial states (ef. Assumption 4), therefore EEz [c<t + 7'((s1, aa)) +
ofth gt E, [V (s1+1)]
implying that :h E II We define the smallest (time-wise) safety gap due
to switching from π. to f as
8 min Pa (cex + 2'((0,, ae) g IV, (+++) Ryo-
(52)
We thus rewrite the probability of invoking the safe prior
Dac,, (st) = DxPaamm (1ss)) pn PnI 2 d (czi + c(cs + +
+ Ew, (V (+(+1))))
(53)
(aPaaaat((1)) VnIx(ar: f) 25.)
(54)
Using Markov's inequality, we can bound the probability of invoking the prior by
Vn, (a f)
Doc, (8t) S DA min E Earnzg(le))
5, .1
(55)
Finally, we insert Equation (55) into Lemma and obtain
A , 10 ( (4, RLJ 7
Un Ex Eat,
.1]] (56)
x DA min Barwnggllaeee
de
t=0
Remark 1. Indeed, the beginning of learning, _. may not lie within the set {T Er|cer 7'c($2, a:)l + 7 [V= (s++1) < d, vt S II as the pessimistic cost value V < V for all 8 E S with probability - 5 by definition (recall Equation (4)). Reducing pessimism such that V converges V effectively expands this set until converges 11 Moreover, let us consider the case where the prior policy does not satisfy the assumption that V (s) < V. V²*((s for all s E S, i.e. the pessimistic prior policy in fact accumulates more costs (in expectation) than an optimal
policy π² when executed on the true dynamics f. In this case, T² is not guaranteed to lie within II and SOOPER converges to the optimum within the set of all feasible policies I <d that are safely reachable given क.
Lemma 6 establishes upper bound the first regret term A,, based on the probability of action 0e ~ ( (-1sr) to be unsafe given the model uncertainty. More concretely, even though actions ag ~ (-(x)) in Equation (56) are determined by safe policy πē. they may be regarded unsafe due to limited knowledge during learning, quantified by epistemic model uncertainty. Since such trajectories would trigger á when executing Algorithm 1, we must relate the bound in Equation (56) to uncertainty induced along trajectories executed by the prior policy f. Therefore, we relate in Lemma 7 these accumulated uncertainties when executing ग. freely in f, to trajectories induced under Algorithm 1, starting with the safe prior policy at timestep This relation visualized in Figure 7.
Me (si,
S; (st,, ae, Cen ( 2
Figure 7: Relating the uncertainty of safe trajectory (green) to trajectory that executes n. freely at / (i.e. not under Algorithm 1, in red) and therefore may be (possibly wrongly) considered unsafe due to model uncertainties.
22

---

Published as a conference paper at ICLR 2026
Lemma 7. Suppose Assumptions 1 to 4 hold and the model Fr, is well-calibrated according to Definition 1, the accumulated uncertainty along a trajectory induced by executing Te freely at t, quantified by Eay=-z (.ss).,,, [a f)). can be upper-bounder by the uncertainty along safe trajectory starting with the safe prior according to Algorithm 1:
(ean) f)] (1
Do
7
Ens (T))
(57)
Proof. Using the definition of Σ, from Lemma
E((ar. f) = (ll
Bh ( (s))))lI. r=t+1
Σ
(58)
we can bound the difference between the accumulated uncertainties of the two trajectories
A - (B,, (<(0, Re, 1e (*tts,, f)]]
(59)
27
@
E
B (s, (
(60)
1==++
(1 (n (++))1l| }
x min
( Lg DA
(61)
by using the expectation difference under two Gaussians (Kakade et al., 2020, Lemma C.2) in (i) and bound the uncertainty difference of under the different actions for the first timestep in (ii) using the closed set of actions with diameter DA and the Lipschitz constant Lo of the uncertainty. Thus Ew,,
can further bound this term using EaioTI se) [Er]
> ( (8x), f)} and obtain
Baysnn (.)s,), (ac fI 5 1 LoDA
VE,, (T
(62)
27
Lig DA
1
Eui Σ X (on [1,((,,
(63)
@
1+ DA σ
8c Σrr
Eui
y (sT:
(64)
LADA
X
1
Bas, 17t (|an((;; ({+)))1})))
(65)
7
2
(
LoDA
8
Es, [o,
(66)
1 DA
Ew, IVT Wanlss (+
(67)
where we get step (i) by restating the definition of E, (+t))) from Lemma 6 and thereafter applying the Schway-Schwarz inequality in (ii). In step (iii), factor out the infinite sum of 7" and can interchange the square with the summation in (iv) due to the non-negativity of [|a,||-
Following from the relation between uncertainty accumulated along trajectories that execute क. freely at f and thereafter follow स. and those that pessimistically invoke Lemma we can bound the first regret term when episodes collected safely using Algorithm 1.
23

---

Published as a conference paper at ICLR 2026
Lemma 8. Suppose Assumptions to 4 hold and the model is calibrated according Definition 1. Given the Lipschitz constants La, with bounded actions D the first regret term A is bounded with probability - 6 in terms of the safely rolled out policy Te."
A Bag
Th Σ t=0
El BlV (9
8241 on (nt, whs ae (an((188)
where In (Ir RLs
D1 (D))) (1 (1+ )2 1-7 (1-7)
Apeninii
6c
Proof. Given the derived regret bound in Lemma 6, bound the sum by the largest sum- mation term during the safe rollout (i.e. executing T under Algorithm 1) using abbreviation
C²
Lr
RLi
Vn (a f)
A
< Egg
C²
7 D min
E Banwagelles
.111
(68)
60
Un Ee
11)
SEg
C
D A max
min Baeweg
.1
(69)
80
Uy Ey f)
())
<
C²
D min
max
Bag
.
(70)
80
As we invoke the prior for the maximal term obtain together with Lemma 7
1
1
(1++4DDA)
LoDA
A
DA
min
{ max
V 11
(71)
--7
Vn
Eus [V>= +
11]
X
7=1
E
C²
1
DA 1+ LoDA
Un
Eu (s),
(72)
1-7
V 1 &
St+1 (() + We: at a~ (.
(73)
Since we now have the uncertainty along the entire safe rollout of म following the true dynamics f. we can now express the entire rollout in terms of the model dynamics pas using (Sukhija al., 2025, Lemma A.5):
A <E p
C C
DA (1 1+ LoDA
LoDA)A))
Vn 7
Σ
E., (1 ac)l
(74)
8n
+ 1=0
NEw.VV/lloo at)1l Σ
Ew, / (b))l
1=0
LD C 1-
DA (1+
LODA)
Vn
<Eng
(75)
1- d0
(1 + Apessimism)
Σ
Bui aoolVv at)
1=
8111 f(st. 0.) +w2,
141 = pn af) + wi.
(76)
In step (i), we introduce a zero term by adding and subtracting the accumulated uncertainties along the model dynamics. Next, we bound in (ii) the difference of the uncertainties along the two different dynamics with (Sukhija et al., 2025, Lemma A.5). Thus, the proof is complete by plugging back in C" and Vn.
24

---

Published as a conference paper at ICLR 2026
A regret bound. Let now consider the second term A that determines the regret of the policy Ty on M, given Fu. Therefore, we first bound the value difference between the pessimistic value estimate V and the true value of executing स on f. Leveraging this bound, we can upper-bound the value difference of executing aa M. and M in Lemma 10. Finally, we bound the second regret term A in Lemma 11.
Lemma Suppose Assumptions I to hold and the model Fu. well-calibrated according to Definition 1. Given the pessimistic cost value Ve from Equation (4), we get with probability -
E (5, + An Ryy 2(1 + vd2)
7 IV, (su) (xx) S An
t=k
where R = max Rmass kmaz]. The trajectory can either be along the worst-case dynamics Bi41 = (n( At ($t)) + a(1 + (6))),-1(500 (8,, ((+)) + wa with a E [-1,1], initial state
8k Sk. or the true dynamics 8241 = [(e, 7t (Br) wh. yielding the same bound.
Proof.
MV, (sx))
= E
(V++1) - V22 (B(+1))
(77)
t=k
> Eu, ($(+1)) - ($1+1))
(78)
where 5;-1 is in each step the worst next state given J leading to the worst value (see Equation (4)). Thus with 8141 Pn (5e, (5:)) + a(1 + vd.) 18)0 ((:)) + we, where a E [-1, and
R(s) = (ss) holds
① M (8x) (sk))
20
Σe
yE
maxx Ean, [f((s++))) Eu, (
(79)
t=k
[1ss,, t(+)))
Mn (5t, (3x)) + +a1 + (ol1+vdaalll
x- min
@ Ry 2(1+ Vd)) 1-7
B Σ
(80)
We rewrite the value difference dependent on the worst-case dynamics μ4, + a(1 vdx1+Vd))), Vn (0) and the true dynamics in step (i) using Kakade et al. (2020, Lemma C.2), which can be expressed in terms of the model uncertainty in (ii) (Sukhija et al., 2023a, Corollary 3).
Lemma 10. Suppose Assumptions to 3 hold and the model F, calibrated according to Defini- tion 1. We consider the following definitions
Ję(Tn, f) = Es,.o
Σ 1=0
((((s
[(81,a))+ + (ls,,a,ces Qt,)) 2 d, + 8t:
ar Tn (-1se))
8141
st
otherwise,
00
Jp(TnsMMn) = Es
Σ 1=0
2' 7Tr(
Mr Wn(Be,50 az) + wi Φ53,, at cet, Cet
2 + 81, 2
ā, ~ Th (Be),
8y-1
otherwise,
Ry 2(1 + Vdla
Bn
A, 1-7
0
25

---

Published as a conference paper at ICLR 2026
Then we have for all O, with probability 1 8
J+(Tan Hn) Jz( f) ()155 S An
BT, [ ( [e]l]]
[Je(Ha) Jz( 1)1 S An
B B [ 1 Σ 1=
Proof. We prove the first inequality [J pn) Jj , and the second one holds for analogous argument. We extend the argument of Sukhija al. (2025, Lemma A.5) to account for the termination state $t-
fI S An Ei Es,,, I (5r, a+)
[T Pa) - JT(Tns f)l = BR San
7 (V.)
(81)
1=
where Vey is the value following under dynamics S, and define the states 88+1 and 81+-1 as
[(E, āe) + we
(3,, Cet, Q.,,) 2 d, 8t F 8t otherwise,
āe ~ Tn((118))
(82)
(1)
[un(5s āt) we (5r, Cer, Cer, 2o,n) 2 8r f 8t:
$141
~~ ~ (-1xe)
(83)
otherwise.
Let (L+VT,)8... R(s) = V (s) and with R(s)= I An. if n. does not invoke the safe prior म, we obtain the following bound following Sukhija et al. (2025, Lemma A.5)
[Je[Tn Pn) Jn fll
Σ E....
Vma (), [R(44)))
(84)
||$241 11541118+111 -84111 11]
x 7 min
v /max Bur (1((3r.) }}, Bw,, [[(+++)]]
(85)
1=0
[|(()r, - (31, 11
7 min
Ry (1+ vd
Ex,, [ (1nn(3,,))))
(86)
In case the agent invokes some timestep k, formulate the regret
- Jr( AI
Ex
/ma. BE., ( (R($(+))) Ew., [[((41))))
(87)
t=0
|| -
x 71 min
+ , (3x) V (3x))
|| stll 1
+
Eww, [R((+)]]7 min
1=
k-1
Vmax(Buw (((5+11) En,, (((++1))
(88)
t=0
||s1+1 | - 1 + +1) (Sw)..
x 7 min
(i0) k-1
7 Ex
Viax(Een, [R( (x En,, [[(s++))))
(89)
1=
26

---

Published as a conference paper at ICLR 2026
[f π (3.) un (5 x (5+))
|}]
x 7f min
0
Ry 2(1 + /
+ 1 7
Σ
Ex,5a [1'11on ( (8)))]]
t=k
(i)) Ry (1 + /d (8)
Ex 10 at)
(90)
0
Ry 2(1 + vd B
Σ Ex (5,,
1-7
(V) Ry 2(1 + vdz) 3n (6)
Ba (y'1oon(5e,
(91)
In step (i), we split the infinite sum along the trajectory. when the policy prior is invoked. We therefore follow the trajectory along the model dynamics in Equation (83). Further, we eliminate the difference term for the terminal state 84 in (ii). In step (iii), we use Lemma 9 to bound the value difference between the true value V.* along f and the pessimistic value V... Using Sukhija et al. (2023a, Corollary 3) bound the difference between the next states in (iv). Finally, in step (V), formulate the bound over nn allowing us to upper bound both terms with the uncertainty using the Ryy 2(1+v7))).-100 safely rolled out policy म. and abbreviate >, = 1-y
Lemma 11. Suppose Assumptions I to 3 hold and the model is well-calibrated according to defini- tion 1. Given the optimal policy x²,, then we obtain the following per-episode regret on M in episode n, A J=(x=" .f) Jr,, f) with An Ry 2ds+/ds)8mnil with probability (1 (1-6)
1--
8
A n = J+(N", ,5 Jz( f) J=(n" pn) + An D
E (711 at)ll] J; (T f),
where 81+11 = Mmn (8t, az))
Proof. We can show this by applying Lemma 10 an optimal policy x" on M.
Intrinsic rewards for exploration and expansion. Next, we show with Lemmas 12 and 13 that the policy n, Equation (9) achieves both optimism and expansion by upper-bounding the sum of both regret terms A, = A + A2
Given Assumptions I to 4 hold and the model Fn is well-calibrated according to Definition I, the policy ua as defined in Equation (9)
Lemma 12.
(7(((s);))) az) ax) + (1" Vesplore + V2 Aesponn Ion lon(se, adll)]
Tn = arg max ET
Σ
S1+1 = Ma an) + Wey as ~ x-118)), 8p ~ po()),
with Nexplore 3dn (Lemma 11) and dexpand 3m (Lemma 8), satisfies with probability 1- 8:
An =1
2 7m
Exg oo (/
a:) a:) (=("" .Ha)
+ Am
B en [r' (lon ax)1l (p(
1=0
<
E Er [( Nesptore + v Aespoon) (se, at)ll + Jz(Tn, - Ji(TTT)) f)-
1=0
27

---

Published as a conference paper at ICLR 2026
Proof. Recall that by construction Tn maximizes
J(x) E Σ r F(st,
Σ t=
7 3m + +333,m) ol (8t, at))l
(92)
Hence
J ( 2 J(t) VT.
(93)
Apply Equation (93) first to the policy ~2. (i.e. the policy induced by using m5 and then invoking the safe prior policy R). By the linearity of expectations and non-negativity of all terms,
(TT)) (T Ho.) + E Brg 80
3nm
1l ax)l + 33n
(s,,, a:)l
(94)
t=0
r=
Next apply Equation (93) to the optimal policy xO on Fr, yielding
2 31n Σ
J(H,) J=(m" Ma) + Ext,s 3nm
+3,, (st, +33,,
(l
(95)
t=
Adding Equation (94) and Equation (95) results in
8 (s +3nn
2m(m) 2J (z (n) + E
37
(96)
20
+ J=(mm (n)) RoTt, ,80
3mm
(7 at)l
3An Σ
7 Il (82, at))
Let us derive bound for Jz( (, Mn), that we can relate to the optimal policy xt for M.
0 Jz Mn) "J((T",f) Bn ,80
(97)
( <J,(F", f) + Bar ,50
(st, (st, atl
(98)
@ {J f) + B,
"l ai)l
(99)
(i)
( f) + An Aa .50
(100)
+ 17 Egg F. 88
Σ (s,,
Step (i) follows from Lemma 10 and (ii) is derived from the definition of M in Equations (6) and (7). Next, step (iii) uses the optimality T. and finally step (iv) uses the upper bound of the first regret term A in Lemma 8. Hence, we can lower bound parts of the term by Jr(T", .,n). Therefore, 2J( in Equation (96) is lower bounded by
(7(TT)) 22J=(T ,p)) + 27, Ex
Vx /
(101)
+ 3A B
Bp.
Vi
+ E
2h, Σ
1=0
28

---

Published as a conference paper at ICLR 2026
Since all terms non-negative, drop the terms 3(A, The BF: and obtain by dividing both sides by two:
2J (TT) 22J=(m'* pn) + 2m :50
Σ
x Io (se, ax)l
(102)
+ Rgg 200
2mh Σ
t=
((Ta) 2J+(T", (n) + AnBar
Σ xd (oln
(103)
+ Exg 200 80
Th
( (s))
t=
By definition of ((,,) and Equation (103), subtracting (T f) completes the proof.
In Lemma 12 we show that the objective upper bounds the performance and the per-episode regret on the model dynamics. Next, we show in Lemma 13 that we can also bound the regret on the true dynamics using An.
Lemma 13. Given Assumptions 1 to hold and the model Fn is well-calibrated according to Definition the per-episode regret in An A + A is upper-bounded with probability of at least 1 8 for all n > 0 by
An < (3 4An + 3 +nm))
Σ t=0
Eu,,
at)
for 8t [81,, az) wi following the true dynamics with ar a. (11sk, Cess Q
Proof.
Using the derivation in Lemma 12, we can upper bound the regret in terms of nn by
An
E E oo dexplore + va (expand) (0) aDll + Jr( Um) [)
(104)
t=
(i)
33(m + Am Bn n..
Σ (o abill + J Mn) Jz(
(105)
where = (s a + follows the true dynamics with at (-1o)) and we insert Lemma 12
(i). Step (ii) follows from 7 E (0, 1) and the uncertainty along the safely executed policy R. being larger than for nne which terminates in st. We next use Lemma 10 to bound Jp(Tn, (hm (T. f) and obtain
An <3(m + E...
(7 ai)l
+ An Fa
(s))
(106)
=3
-317 Ex...o
V1 ac)l
+ +37m
ERn..D
Vorllon اللهl
(107)
0
)
E
3dm
Ex
V 41ll
) Ei
-E
Vo aill
where step (i) introduces zero terms by adding and subtracting accumulated uncertainties. We further bound the per episode regret using non non-negan non-negativity and Lemma 10
A <33 + 4h,, + +nn2 33.,
V (s,,
(108)
29

---

Published as a conference paper at ICLR 2026
In Lemma 13, derived upper bound on the per episode regret An, based on the policy nn executed on the true dynamics $1+1 = f(t, az) + w. Using Chowdhury & Gopalan (2017), we obtain a sublinear cumulative regret for our safe exploration algorithm SOOPER.
Theorem 2
(Sublinear cumulative regret). Suppose Assumptions to 4 hold and the model Fn is well calibrated according to Definition Then, Algorithm guarantees with probability
R(N) - (J.(T2,)) J4 (Tn, f)) S (1 log(N) VN).
and Je(Tn, S d, Vn E [I....,N}.
that is, the cumulative regret grows sub-linearly in N, dependent on the maximal information gain I Nog(N), while satisfying the constraint throughout learning by Theorem 1.
Proof. Given the per episode regret bound in Lemma 13, we sum over episodes n. = -I...,,
N
R(N)- = Σ 1
(3 + 4Am + 3 + 3mm)
Σ
E
[V74lo,n (11, (109)
2=1
As by definition Chowdhury & Gopalan (2017), for the well-calibrated model Fns Brg BN for all ,,...,,, N,, we rewrite
R(N) <(4AN 3(AN IN IN))
(110)
1=
@ <(4ANN + 3(A7 + + UN))
N
BR... [V791an((), aI
Eeg
n=
1=0
( S 24AN + 3(A + RN + nN))
N
N
Es (V
80
n=1
t=
(112)
We use Cauchy-Schware in step (i) for the outer sum over episodes I...., N and in (ii) for the inner sum over timesteps t. Further, we can bound the regret term for episodes n = 1,..., N, using the truncated horizon T, Tog(v?)
log(n) by decomposing the term into
Ex, [V
(113)
80
T
Ex
(1
Ea, (
(114)
t=
N T
Ex,
(s).
+
N
Tu
2
Paa
(115)
n=1
t=1
T.
k22
kmar
Ex,
+
(116)
7
n=1
Tx-1
Ex, (s,, + (log(N) + 1).
1.2
k
(117)
log, (n) and hence y log. In (i), we use the fact that 1 is non-decreasing in n and therefore due to Lehman et al. (2018, Theorem 3.9.1) bound ΣN log(N) 1. Further, bound the first term
Step (i) follows from log(n) log(>) with log(n)
log(y)
a
30

---

Published as a conference paper at ICLR 2026
Σ Ei= BR
lo as in Sukhija et al. (2025, theorem 5.7):
R N Nlg((N)
Ry N (log((M) NV +
S(4AN + 3(A + ni + )
(118)
de
Smaz where R. log(1 fyma))
As T, is non-decreasing, we fix TN and since we obtain from the definition of 3x from Chowdhury & Gopalan (2017) that
with Smaz
AN x 1-7' lh x I-g.
BN
Px
RN S SO (FWWoogtN)
(119)
C THEORETICAL EXTENSION TO WORLD MODELS
For our theoretical analysis in Appendices A and B, we assume, for simplicity in presentation, that the reward and cost functions are known. All arguments can be extended the more general setting of unknown costs and rewards, which are learned as part of a "world model", in addition to the unknown dynamics f. In practice, we consider this general setting in our experiments (Section 5). Below we briefly illustrate how one can extend the analysis above to this more general case.
First, one needs to impose an additional RKHS assumption on the cost and reward functions, analogous to Assumption 3 for the unknown dynamics f. Accordingly, we jointly learn with the unknown dynamics f calibrated model for the reward and cost function with nominal predictions Min 14,, and epistemic uncertainty o of Hence, it holds V(s,a) S x A r(s.a) E {r' x A c(s,a) E {c' |c' - M S 3
Ir' - S 3, and V(s,a)
Given only access to the calibrated model predictions, modify the reward and cost structure the planning MDP M as follows:
if o ( St, Ce, O d,
F(s ar)
if s; = 81, otherwise,
(120)
(s), a.) + (n ai) ö(5, a,) = (st, ar) + 0 (s,, oz).
(121)
By utilizing the above cost and reward, we demonstrate safe exploration and achieve sublinear cumulative regret, in the presence of unknown cost and reward functions.
Safety guarantee for unknown costs. Under the modified cost č in Equation (121), the update of the pessimistic Q-value given by:
( = Ex Σ of (ā(ss, at) + Apessimisml or (Se, dz)ll) = s. au
(122)
For this modified update using č, it possible to show that Lemma 2 holds using uncertain predictions. One can derive it by bounding the difference between the value v (s) following the model dynamics with the true cost function, and the value V. (x) following the model dynamics with the mean predictions
(s) V2(
E+
"(((ss, (z) (5 az)) s0
(123)
$ E
7 |c(s), ar) -Kii acll s0 =
(124)
E
7 ''Bn0 (st, s00 =
(125)
8t41 (st, at) wes su
(126)
31

---

Published as a conference paper at ICLR 2026
Hence, by bringing the actually observable value (s) the right side, it follows that
(s) - (s) + E&
> 3n (st, ax) 80 -
(127)
E
= X (sr ar) + (st,
(128)
Ex
7'((s,, ax))
80 -
(129)
81+1
(1 az)
+ we,
0t = fsa))
(130)
Using the derived upper bound and plugging it into Equation (20) yields
(s) S Ex
7 (ō(s); ar) + Apessimism (s a))l) 80 =
(131)
8t41 = pnn (8t, 0+) + whs
at =
(132)
Hence, the proof of Theorem holds using Q (s, a) from Equation (122) in Equation (29). We can also upper-bound this in more compact form by assuming the same kernel k for the cost and the dynamics prediction, so that we have 3, az) + Apessimism| (st az)ll S (Apessimism - (,,) ( dz)ll. This follows from the non-negativity of and with off being defined as the concatenation of the prediction uncertainties for the dynamics on and the cost of.
Regret bound for unknown reward and cost function. Similar to the modified cost update in Equation (122), use f in Equation (120), and optimize for
Fn = arg max E
Σ () ar) + (72 desplore + )l
(133)
81+1 = (8e; ae) + We, ar ~ (-186).
(134)
For the derivation of the regret bound, the first regret term is affected by enlarging the pessimistic additive cost with Apessimism + 3n. This contributes an additive 3n, term, which does not change the 3a proportionality of 1/ x -_ For the second regret term, we obtain an additional term that quantifies the uncertainty in the reward prediction, similar the derivation in Equation (129). Hence, we obtain the additive term in Lemma 11 while following the mean dynamics and using the model's mean predictions v
3 =J=("", .f) Je(Tnn
(135)
<Ja(a', ,Mn + An
Σ 1=0
Ex (lln)nl) Jr (T(TTn
(136)
< > Eg [ (1) + (5, a:)) so = s]
(137)
t=0
A, Ex so [7'1|0, a:] J=(TT.:
(138)
By definition of the modified objective in Equation (133), we still upper bound in Lemma 12 the regret with the behavioral policy T.e so that the following proof steps still hold. Further, we can again assume that the dynamics and the reward are part of the same kernel, resulting in the bound 3,00 (8t, at) + >m.lo,, (St, al)ll S (An + (,)]laa/( (x,,,, at)]l. where of defined as the concatenation of the prediction uncertainties of the dynamics on, and the reward o% Thereby, also for the second regret term, the proportionality remains X, & 1-y Consequently, the regret bound VN) for unknown reward and cost functions.
remains unchanged Ry O (NNiog()) VN)
32

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENTS
We conduct further empirical analysis of SOOPER, focusing on the following key aspects: (i) demon- strating the role of prior policies in accelerating learning: (ii) leveraging terminations for safe policy improvement: (iii) ablating the role of Algorithm I; (iv) robustness of SOOPER to prior policies with varying pessimism levels; (v) the necessity of acting pessimistically for safety, and (vi) the influence of optimism on the agent's regret. The goal of these ablations is to provide additional empirical evidence to support the different design choices of SOOPER.
Safe transfer under task misspecification. We demonstrate the effectiveness of policy priors compared to "backup" policies that are commonly considered previous works. For this, we reduce the size of the goal in the PointGoall task from SafetyGym (see Figure 18) from 0.3 to 0.15. This makes learning harder, as additional exploration is required to find the goal. We compare the following methods: (i) SOOPER that a policy trained the original 0.3 goal task safe prior; (ii) SAILR that uses a hand-crafted "backup" policy that brakes the robot upon triggering, following SAILR's "advantage-based filtering". The policy Ra in this case initialized with the same policy prior that SOOPER uses; (iii) SOOPER that uses the policy prior only within Algorithm 1, but learns its policy Ty completely from scratch; (iv) finally, baseline that uses the hand-crafted braking backup policy, falling back to it as done in in SAILR, while training the policy Tn from scratch. Figure 8 demonstrates two results: (a) as expected, both baselines that learn from scratch fail to converge within the training budget of 200 episodes. (b) On the other hand, when finetuning the prior policy, using hand-crafted braking "backup" policy indeed satisfies the constraint, however at the cost of slower learning compared to SOOPER. This result illustrates the importance of using policy priors not only to maintain safety, but also to guide and accelerate learning.
SOOPER
Braking Filter
From Scratch
+ From Scrntch
Braking Filter
-- Baseline
20
W
40
(r(
A
0
30
-10
20
10
150
200
100
150
200
Iteration
Iteration n
Figure 8: Learning curves of the objective and constraint for PointGoall goal size transfer. SOOPER performs significantly better using the prior policy trained larger goal size, significantly helping in exploration when learning to reach a smaller goal.
Learning from ferminations. We study the effect of learning from terminations on the agent's ability to reduce its dependency the policy prior. To this end, we ablate the use of terminations in Equations (6) and (7) and compare SOOPER with a variant that does not terminate in M. In Figure 9, observe decrease the incurred costs for SOOPER with terminations, while SOOPER without terminations still encounters high costs for both CartpoleSwingup and RaceCar, the optimal unconstrained policy violates the safety budget, the agent trained without terminations fails to achieve the optimal safe performance. This failure occurs because, in the absence of termination signals, the agent receives no feedback about triggering the prior policy during online rollouts.
Robustness to policy priors. Different degrees of sim-to-real gap or lack of data coverage in the offline setting require increasing levels of conservatism. We demonstrate that SOOPER can effectively learn from policy priors with varying levels of conservatism and initial performance. For this, we train three prior policies on PointGioal2 for different sim-to-real gaps with sufficient pessimism levels and show that SOOPER converges to optimal performance while satisfying the constraints using all three policy priors Figure 10. These results highlight SOOPER's ability to safely explore in highly uncertain environments, given conservative policy priors
Robustness to noise. Our problem formulation in Equation (1) focuses on additive Gaussian noise. While this modeling assumption is key theoretical development of SOOPER-in in particular
33

---

Published as a conference paper at ICLR 2026
SOOPER
SOOPER without Terminations
Baseline
(m.)
Ja(an)
# #1 Qā.) 2
RaceCar
Budget
100
50
50
50
100
0
50
100
50
100
CaripoleSwingup
150
800
Budgel
700
600
100
200
300
100
200
300
100
200
300
Iteration n
Figure 9: Learning curves of the objective, constraint, and number of times the safe prior is triggered during online rollouts. The number of times SOOPER uses the prior shrinks over training.
Sale prior #1
-
Sale
Safe prior R3
30
10
Budgel
(.(
8
Ja(
20
6-
5
0
Iteration
x10
Iteration
x103
Figure 10:
Learning curves of the objective and the constraint, starting from different safe policy priors with various levels of conservatism, differing in initial performances.
for the proof of Lemma 7-we demonstrate that SOOPER remains safe under substantial disturbances that deviate from the additive Gaussian assumption. In particular, we consider noise of the form ā, Oe + Th, T) Unif(-n, n).. Notably, this form of noise enters non-linearly the dynamics f. To study this setting, we sweep over the interval of the disturbance distribution with n E {0, 0.05, 0, 0.15..1111.. 0.2, 0.25] for all tasks considered Figure We use the same prior policies of the experiment in Figure 3; these policies are trained in simulation using O. Note that for all tasks in our experiments results in Figure 11 where we report the performance at the end of training and the largest average accumulated cost measured during training for all tasks. As shown, SOOPER maintains safety without suffering performance drops, even under significant noise disturbances. Crucially, these results show that even though To is obtained under 7 = O, SCOPER can still satisfy the constraint, as long is An is conservative enough. These results in line with Theorem 1: safety be guaranteed for a very general class of noise structures as long as Tp satisfies the constraint.
[-1, 1] hence 7 0.25 is non-negligible noise. We present our
Pessimism for safe exploration. Given the previous ablation on conservatism of safe policy priors, we now study the scale of pessimism used during learning. As shown for CartpoleSwinguu in Figure 12, using increased Apessimisms the agent consistently satisfies the constraints without significant sacrifice in performance. Moreover, SOOPER demonstrates robustness increased Apassimiam and showcases higher performance for lower chosen pessimism values. Crucially, this behavior expected it reflects an inherent tradeoff between pessimism and exploration.
34

---

Published as a conference paper at ICLR 2026
PointGoal2
RaceCar
CaripoleSwingup
HumanoidWalk
WalkerWalk
950
JeleN)
200
800
675
17.5
700
625
15.0
muxxa Jelme)
25
100
100
100
20
80
0.0
0.2
0.0
0.0
0.2
0.2
0.0
0.2
Y
Figure 11: SOOPER achieves good performance and satisfies the constraint under non-Gaussian, non additive noise.
125
Apssimich
800
Budget
J. (Fa))
Jol
75
700
50
600
25
2
Iterations TL
x10*
Iterations T
x10*
Figure 12: Learning curves of the objective and the constraint for SOOPER with increasing Apesiimism
Pessimism improves robustness constraint satisfaction.
Optimistic planning for sublinear regret. We now study the influence of SOOPER's intrinsic reward on its performance, safety during learning and on the cumulative regret. Specifically, we study how varying Aexpand together with Aexplore improves SOOPER's performance in the CartpoleSwingup task. Intuitively, high values of Aexpand + Aesplore may lead to excessive exploration, while small values of A + Aexplore may not explore enough. As visible in Figure 13, empirically, the cumulative regret x R(N) exhibits single optimum, since for small optimism scales the agent under-explores and fails to expand the safe region, and for large optimism scale the agent over-explores, which reduces short- term greediness and lowers cumulative reward over the finite horizon. As N increases, short-term suboptimality due to explorrtoo decreases and the cumulative reward becomes less sensitive the optimism scale, consistent with the sublinear-regret guarantee for sub-linear growth in N.
Azspooe
Auspent
Aceptor: Apgind 10
Aueptor: Apsprs 50
Budget
800
(m))
200
Je m
700
50
175
Optimal
-/2
600
150
25
50
75
0
25
50
75
100
0
25
50
75
Iterations n
Optimism
Iterations n
Figure 13: Learning curves of the objective and the constraint, well cumulative regret, for different optimism scales. A good value for the optimism value can be identified, where both under- and over-exploration are avoided and regret is minimized.
Safe offline-to-online real hardware. We repeat our hardware experiment in Section 5 on the race car however now use SOOPER using an offline-trained policy. We collect 25K roal-world transitions, corresponding roughly ten minutes of real-time data. To collect this data, the
35

---

Published as a conference paper at ICLR 2026
simulation-traiins policy used in sim-to-real experiment baseline. This dataset is then used to train a pessimistic policy using MOPO (Yu et al., 2020) with a primal-dual solver for the constraint. We introduce pessimism by penalizing the reward and cost with model uncertainty (see Yu et al., 2020). We repeat our hardware experiment with five random seeds, reporting the mean and standard emror of the accumulated rewards and costs of each iteration n. in Figure 14. As shown, SOOPER leverages the filline-trained policy prior to learn near-optimal policy while satisfying the constraint throughout learning.
SOOPER
Baseline
200
20
150
M
m
15
100
10
30
20
40
60
80
100
20
40
60
80
100
Iteration
Iteration n
Figure 14: Performance and safety the real race car given prior policy that is trained via fixed offline data. SOOPER improves the performance while satisfying the constraint during learning.
Learning directly from costs. While we demonstrate that SOOPER can learn safety from an unconstrained MDP with termination, our method can be extended to learning directly from costs via model-based rollouts with any off-the-shelf CMDP solver. While this variant enjoys the safety guarantee of Theorem 1 using Algorithm 1, it is not guaranteed to learn an optimal policy. We instantiate this variant with CRPO and refer to this baseline SafeCRPO. The evaluations across various tasks show that SOOPER's Algorithm 1 can be paired with alternative CMDP solvers and ensures safe learning. However, SafeCRPO consistently learns slower and achieves lower performance than SOOPER. This demonstrates the advantage of SOOPER using terminations paired with Algorithm 1 learn safely and guarantee convergence.
Runtime, Even though SOOPER builds on MBPO (Janner et al., 2019b) and therefore enjoys relatively simple im- plementation, its use of Algorithm 1 and post-hoe relabeling of transitions with terminations deviate from the standard
SOOPER
CRPO
SAILR
Primal
Steps second (t)
"RL toolbox". Here, we study the effect of these components on the runtime of SOOPER compared to other model-based baselines. Specifically, we record training throughput, namely, the total number of environment steps divided by the overall wall-clock time for the runs of the experiments in Figure 3. In Figure 16 aggregate the number of steps per second across all tasks (using five seeds) and report the mean and standard deviation. As shown, even though SOOPER has additional algorithmic components, it does not suffer from significant runtime overhead.
60
40
Figure 16: The number of environ- ment steps per second of SOOPER compared to other baselines.
Additional learning curves. Figure 17 complements Fig- ure in Section and shows the learning curves on all sim-to-sim tasks. SOOPER consistently achieves near-optimal performance without violating the constraint during learning. In addition to the baselines presented in Section 5, we include another CMDP solver that is based on Log Barriers (Usmanova et al., 2024). This solver is theoretically guaranteed to maintain feasibility during learning by penalizing iterates that approach the boundary of the feasible set. For all of our sim-to-sim exper- iments, we train the safe policy priors following SPiDR (As et al., 2025a). SPiDR leverages domain randomization to introduce the conservatism required for constraint satisfaction when deploying RL policies under distribution shifts. In particular, to obtain pessimistic policies, SPiDR penalizes the cost according to the disagreement in next-state predictions of the randomized dynamics.
36

---

Published as a conference paper at ICLR 2026
SOOPER
SafeCRPO
-- Baseline
(T.)
Jc(
CartpoleSwinge Swingup
150
800
Budget
30
600
X10
20
PointGoall
Budgel
10
0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
X10
X10
40
PointGoal2
Budgel
X10
700
WalkerWalk
Budgel
600
50
0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
X 10
X10
150
HumanoidWalk
940
Budget
920
50
900
3
0
x10
X10
200
Budgel
RaceCar
2
0
3
x10
x10
Iteration n
Figure 15: Learning curves of the objective and the constraint for SOOPER and SafeCRPO across all environments. SOOPER consistently outperforms and shows asymptotic convergence.
37

---

Published as a conference paper at ICLR 2026
SOOPER
SAILR
CRPO
Primal-Dual
Log Barriers
Baseline
(T.)
JT(TT)
300
CartpoleSwingur
800
200
600
Budgel
400
300
20
Budget
PointGoall
15
10
50
100
150
200
0
50
100
150
200
150
PointGoal2
20
100
50
Budget
500
1000
1500
2000
2500
3000
0
500
1000
1500
2000
2500
3000
150
700
WalkerWalk
600
Budget
500
100
300
200
400
600
1000
1200
200
400
600
800
1000
1200
300
950
Humanoid
200
900
850
Rudget
800
50
100
150
0
50
100
150
15
200
RaceCar
150
10
100
Budget
Iteration n
Figure 17: Learning curves of the objective and the constraint for SOOPER and comparison baselines.
SOOPER maintains safety throughout training in all tasks while outperforming the baselines in terms of performance.
38

---

Published as a conference paper at ICLR 2026
E SAFETYGYM
PointGoal environments. In these tasks, the agent must navigate to target location while avoiding hazards which include free-moving vases and designated hazard zones. The environment is depicted in Figure 18. The initial positions of the agent, goal, vases, and hazards are randomized at the beginning of each episode. The reward function is defined the change in Euclidean distance the goal between successive steps
=
1[4 S c).
where d, = || Xgoai 2 is the Euclidean distance from the robot to the goal. The term 1[d, S an indicator that gives a reward bonus when the agent reaches the goal, i.e. when it is within 0.3 of the center of the goal. The goal position is resampled to another free position in the environment once reached. PointGoall and PointGoal2 differ in the amount of obstacles located in the environment. A cost of incurred when the agent collides with crosses a linear velocity threshold (after collision) or when the agent inside hazard zone h:
when of the vases
(x( a+) = 1 V collides(x, (x, x.)) + 1 E X 2 >] + 1 E H de S p), where 7 5e and p = 0.2. Please see the implementation of Ray et al. (2019) for more specific details.
Simulation gap. In our sim-to-sim experiments (in Figure 3), before training, uniformly sample gear parameters of the actuators that control the linear and angular velocity of the robot. The precise ranges given in Table 1. Note that the z-joint is hinge joint that allows the agent to rotate around the z-axis and the z-joint a slide joint that allows translation the zy-plane.
Table 1:
Domain randomization parameters and ranges used during training.
Parameter
Value
Gear (x)
[-0.2, 0.2]
Gear (z)
[-0.1, 0.1]
Figure 18: Visualization of random initial- ization the PointGoal2 environment. The red pointmass is the agent, the green transpar- ent cylinder is the goal, the cyan boxes are vases and the blue circles hazard zones.
F RWRL BENCHMARK
We use the RWRL benchmark suite (Dulac-Arood et al., 2021), which adds safety constraints and distribution shifts to the tasks from the DeepMind Control benchmark suite (Tassa et al., 2018).
Figure 19: RWRL tasks.
Constraints. We use the joint position limits constraint for Humanoid Walk, joint velocity limits for WalkerWalk, and slider position limits for CartpoleSwingup. These are the standard constraints proposed by Dulac-Amold et al. (2021).
39

---

Published as a conference paper at ICLR 2026
Simulation gap. In sim-to-sim experiments from Figure follow similar experimental setup as Queeney & Benosman (2023), introducing variability the physical properties of the system during training simulate distribution shifts. In Table 2 we provide the specific parameters perturb in each task.
Table 2: Additive domain randomization parameters and ranges used during training of the policy prior across tasks from RWLR.
Parameter
Value
CartpoleSwingup
Pole Length
[-0.1, 0.1]
Gear
[-1.0, 1.0]
Knee Gear
[-40., 40.0]
Humanoid Walk
Friction
[-0.05, 0.05]
Hip Gear (x)
[-20.0, 20.0]
Hip Gear (y)
[-20.0, 20.0]
Hip Gear (z)
[-60.0, 60.0)
WalkerWalk
Torso Length
[-0.1, 0.1]
Gear
[-5.0, 5.0]
G RACECAR ENVIRONMENT
In this environment the agent is asked navigate goal position while avoiding static obstacles. This environment is implemented both in our sim-to-sim experiments, as well as on the world race car experiments.
Reward and cost.
The reward at timestep is given by
(s ax) == d,-1 de + 1[d S e| hellaell2 d ll: a.
where d, is the Euclidean distance to the goal, and E IR.2 denotes the action applied at time t (consisting of steering and throttle). The term [[d, < <] is an indicator function that gives a reward bonus when the agent is within 0.3 of the goal. The penalties A and λ weight the control effort (magnitude of the action) and the change in action between consecutive timesteps, respectively. The cost function at time is defined as
(s(, a4) = Σ 111 pil < p.. E + 1[x & V1,
where x, E R2 the car's position, Pt and p; are the position and radius of the i-th obstacle and E is the kinetic energy of the car at time l, simulating plastic collision between the car and obstacles. This choice cost function allows to penalize more severely collisions which the car smashes with high velocity into obstacles, as opposed to softly touching them. The second term penalizes the agent for leaving the valid area which corresponds a bounded rectangular arena.
Simulation gap. In the previous sim-to-sim environments, model the sim-to-sim gap for the experiments in Figure by introducing an auxiliary dynamics parameter (e.g., pendulum length) that not observed during training. In contrast, in the RaceCar environment, the car dynamics in the training environments are governed by a semi- -inematiiaaii bicycle model that does not account for interactions between the tire and the ground. On the other hand, in evaluation, bicycle model of Kabzan et al. (2020). We refer the reader to Kabzan et al. (2020) for the detailed equations of motion.
the dynamical
40


Reviews:
[{"review_id": "7pFHqC9jIA", "reviewer": "Reviewer_vK36", "summary": "The paper proposes SOOPER (Safe Online Optimism for Pessimistic Expansion in RL), a model-based algorithm for safe exploration in continuous CMDPs. It leverages pessimistic policy priors, which are safe but suboptimal policies learned offline or in simulation, to ensure constraint satisfaction, while using probabilistic world models for optimistic exploration.\n\nThe authors prove that SOOPER satisfies safety constraints at all times (Theorem 1) and achieves sublinear cumulative regret (Theorem 2), unlike prior methods that only guarantee asymptotic safety or simple-regret bounds. The analysis couples online cost-tracking with a \"planning MDP with pessimistic termination reward,\" allowing standard RL methods to be used while maintaining guarantees. Empirical validation on SafetyGym, RWRL, and real robots shows consistent safety and performance gains.", "questions": "How sensitive is the safety guarantee to imperfect calibration of $\\sigma_n$\t​\nCould Assumption 4 be relaxed to hold in expectation over $\\rho_0$ \nHow would the analysis change if the policy prior were stochastic or state-dependent only approximately safe?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Strong analytical results by combining always-safe learning with sublinear cumulative regret.\n+ The pessimistic-termination MDP approach converts a constrained problem into a standard RL one.\n+ Unification of optimism, pessimism, and expansion through a single intrinsic-reward objective.\n+ Empirically validated on diverse continuous-control tasks and real hardware, not only simulators.", "weaknesses": "- The proof seems to implicitly assume Lipschitz continuity of the uncertainty estimate $\\sigma_n,$ which is not formally stated.  I think this assumption is required for several bounds.\n- Assumption 4 requires the prior policy to be at least as safe as the optimal one for all states, which seems very strong.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r22TRdOxdE", "reviewer": "Reviewer_zYbx", "summary": "This paper investigates the use of conservative policy prior to enhance safe exploration in reinforcement learning (RL). The proposed method, SOOPER, performs optimistic exploration when safety can be guaranteed with high probability, and switches to the conservative policy prior when exploration becomes potentially unsafe. The authors provide theoretical guarantees for both high-probability constraint satisfaction and optimality. Extensive experiments demonstrate that SOOPER outperforms state-of-the-art baselines across various benchmark tasks.", "questions": "1. The proposed method relies heavily on the set of plausible models. In general, as the amount of data increases, the model can be refined and improved. How does such model improvement affect the theoretical guarantees presented in this paper? I did not find a clear discussion on this point.  \n2. Could the authors clarify why CRPO fails in the safe online learning experiment on real hardware?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. While the combination of offline training and online exploration is not entirely new, the paper demonstrates novelty through its theoretical development, particularly in providing safety and optimality guarantees.  \n2. The empirical evaluations are comprehensive and effectively support the theoretical findings, illustrating the method’s applicability to practical scenarios.  \n3. The paper is well written and easy to follow, even in the theoretical sections, which are presented with clarity and coherence.", "weaknesses": "1. In the Introduction, the authors state that their theoretical results hold under regularity assumptions, and in the “Optimality” subsection of Related Works, they claim to relax some assumptions from prior studies. However, in Section 3 (Problem Setting), Assumption 1 regarding Gaussian noise appears rather restrictive. Moreover, the assumption that the transition dynamics follow \\( s_{t+1} = f(s_t, a_t) + \\omega_t \\) is quite strong; a more general formulation such as \\( s_{t+1} = f(s_t, a_t, \\omega_t) \\) might be preferable. As far as I understand, these assumptions are stronger than those used in [1].  \n2. The proposed simulated exploration strategy is model-based, which may limit its applicability to environments with large or complex state spaces. It would be valuable for the authors to discuss these limitations and possible extensions to model-free settings.\n\n[1] Akifumi Wachi, and et al, Safe exploration in reinforcement learning: a generalized formulation and algorithms, in NeurIPS 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wCwAnSuTLK", "reviewer": "Reviewer_XArr", "summary": "This paper introduces SOOPER, a model-based reinforcement learning algorithm designed for safe exploration for constrained Markov decision processes. The method leverages conservative policy priors to ensure safety while optimistically exploring to improve performance. Conservative policy priors are derived from offline data or simulations. Theoretical guarantees are established for both safety and sublinear cumulative regret. The experiments demonstrate strong empirical performance on several benchmarks and real-world robotic systems.", "questions": "1. We consider the assumption of treating state transitions in the form of equation (1) and Assumptions 1–3 to be very strong. Could you share the authors' views on the strength of this set of assumptions and any attempts to relax them in future research?\n\n1. To what extent do the assumptions made in the paper hold in the experimental setting? While the paper provides a sound theoretical framework, it is somewhat unclear whether all these assumptions are necessary in the experimental setting. It would strengthen the paper to discuss or empirically examine how SOOPER behaves when some of these assumptions do not hold. Since the implementation builds on MBPO to learn the dynamics model, the method should, in practice, already be capable of handling moderately stochastic transitions. Confirming this empirically would provide valuable evidence for the robustness of the proposed approach, by evaluating SOOPER under relaxed or partially violated assumptions. I do not intend to criticize the fact that some assumptions may not hold; rather, if the proposed method still functions effectively as a kind of meta-algorithm even when certain assumptions are violated, that would further enhance the value of this paper.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is easy to follow and the motivations behind the proposed algorithm are clearly presented. The paper provides a well-motivated discussion of the need for safe exploration in reinforcement learning, emphasizing the trade-off between conservatism and exploration.\n\n1. The relevant literature has been well-covered and I do not find missing references.\n\n1. The proposed algorithm is technically sound.\n\n1. Theoretical results are rigorous, providing both safety guarantees and a sublinear cumulative regret bound. As far as I know, the theoretical results in this paper is novel.\n\n1. The experimental section is comprehensive, including evaluations on RWRL, SafetyGym, and a real hardware setup. The proposed method consistently outperforms or matches strong baselines such as SAILR, CRPO, and Primal-Dual.\n\n1. The implementation based on MBPO demonstrates that the proposed algorithm can scale to high-dimensional continuous control tasks. MBPO is known as a highly scalable algorithm, which is a good technique for making SOOPER practical.", "weaknesses": "1. The theoretical analysis relies on several strong assumptions (e.g., Gaussian noise, Lipschitz continuity, bounded RKHS norms, and the existence of a pessimistic policy prior that satisfies safety for all plausible dynamics). While such assumptions are sometimes seen in other safe RL literature, they may limit the practical applicability of the theoretical results.\n\n1. Although the paper claims that SOOPER can be implemented on top of standard model-based methods, the overall system involves several non-trivial components (uncertainty estimation, pessimistic value computation, termination logic). A short discussion on the computational overhead or design trade-offs would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)