SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
Authors: Yarden As
Chengrui Qu
Benjamin Unger
Dongho Kang
Max van der Hart
Laixi Shi
Stelian Coros
Adam Wierman
Andreas Krause
Venue: NeurIPS 2025 poster ()
Keywords: Safe Reinforcement Learning
Deep Reinforcement Learning
Sim-to-Real
Constrained Markov Decision Processes
Primary area: reinforcement_learning
TL;DR: We develop a practical algorithm for safe sim-to-real transfer

Abstract: Deploying reinforcement learning (RL) safely in the real world is challenging, as policies trained in simulators must face the inevitable *sim-to-real gap*. Robust safe RL techniques are provably safe, however difficult to scale, while domain randomization is more practical yet prone to unsafe behaviors. We address this gap by proposing SPiDR, short for Sim-to-real via Pessimistic Domain Randomization—a scalable algorithm with provable guarantees for safe sim-to-real transfer. SPiDR uses domain randomization to incorporate the uncertainty about the sim-to-real gap into the safety constraints, making it versatile and highly compatible with existing training pipelines. Through extensive experiments on sim-to-sim benchmarks and two distinct real-world robotic platforms, we demonstrate that SPiDR effectively ensures safety despite the sim-to-real gap while maintaining strong performance.

SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
Yarden As* ETH Zurich
Chengrui Qu Caltech
Benjamin Unger ETH Zurich
Dongho Kang ETH Zurich
Max van der Hart ETH Zurich
Laixi Shi Johns Hopkins University
Stelian Coros
Adam Wierman
Andreas Krause ETH Zurich
ETH Zurich
Caltech
Abstract
Deploying reinforcement learning (RL) safely in the real world is challenging, as policies trained in simulators must face the inevitable 'sim-to-real gap'. Robust safe RL techniques provably safe however difficult to scale, while domain ran- domization is more practical yet prone to unsafe behaviors. We address this gap by proposing SPIDR, short for Sim-to-real via Pessimistic Domain Randomization-4: scalable algorithm with provable guarantees for safe sim-to-real transfer. SPIDR domain randomization incorporate the uncertainty about the sim-to-real gap into the safety constraints, making it versatile and highly compatible with existing training pipelines. Through extensive experiments on sim-to-sim benchmarks and two distinct real-world robotic platforms, we demonstrate that SPiDR effectively ensures safety despite the sim-to-real gap while maintaining strong performance.
Introduction
Reinforcement learning (RL) has made significant strides in recent years, demonstrating remarkable progress across range of domains. These include achieving superhuman capabilities in games (Mnih et al., 2015; Silver et al., 2016), fine-tuning large language models (Ouyang et al., 2022), advancing applications healthcare (Fox et al., 2020; Zhu al., 2020), robotics (Lee al., 2020; Degrave et al., 2022; Lin et al., 2025) and autonomous driving
Sesuumano-Townee et al., 2025; Cornelisse et al., 2025). Yet despite these achievements, ensuring safety and preventing harmful behaviors remains critical challenge and prerequisite for unlocking the full potential of RL as a ubiquitous element in everyday life (Amodei al., 2016; Gu et al., 2022).
The use of simulators has been a key component behind the success of many of the mentioned applications (Visentin et al., 2014; Makoviychuk et al., 2021; Degrave et al., 2022; Kazemkhani al., 2024). Training in simulation allows agents to learn from unsafe interactions, which in reality would lead to catastrophic outcomes. In addition, learning complex behaviors fully online can be prohibitively time-consuming. Modern simulators accelerate training, reducing hours of real-world experience to minutes on consumer-grand GPUs (Rudin et al., 2022). However, while being major driver in the development of the above examples, even oftte-of-theartt simulators often fall short in precisely mirroring the real-world. Indeed, "all models are wrong" (Box, 1976) the so-called sim-to-real gap can make simulation-trainet policies violate real-world constraints, which can be particularly dangerous in high-stakes settings where safety must be guaranteed on first contact.
Existing literature address this challenge often relies tools from robust optimization (Queeney and Benosman, 2024; Kitamura et al., 2024; Zhang et al., 2024). While being theoretically grounded, such methods typically require practitioners to significantly alter their existing training pipelines,
*Corresponding author: yardaaRethz.c ch
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

1.2
1-26
1-32
1-3.8
1=42
t=
L-500
5.3
Terminal
Safety statns
0.175
0.088
1E130
0,300
0,000
0.664
0.856
0.8
Uncertainty
Nw
0.5
w
0.2
1.0
1.2
2.0
3.0
4.0
4.6
5.0
5.3
Time t [scc]
Figure 1: Uncertainty over a quadruped robot's trajectory. The snapshots illustrate the robot's pose at key moments, with corresponding uncertainty levels highlighted. High-uncertainty transitions are incorporated into the cost function to discourage the policy from entering regions where the simulator is inaccurate and behavior is more likely to become unsafe during real-works deployment.
rendering them less prevalent in practice. In contrast, due to its simplicity, domain randomization has become the de facto tool for sim-to-real transfer (Tobin al., 2017; Peng et al., 2018; Lee et al., 2020; Degrave et al., 2022). Despite its success, in problems that require adherence to safety constraints, domain randomization lacks safety guarantees and often fails to satisfy the constraints in practice (cf. Queeney and Benosman, 2024, and Figure 3). Therefore, method that provably guarantees safe sim-to-real transfer, while being highly compatible with standard training practices, is still missing.
In this work, we address this gap by presenting a simple method that builds on domain randomization while ensuring safety under sim-to-real transfer. We theoretically show that unsafe transfer can be associated with large uncertainty about the sim-to-real gap, quantified as the disagreement among next-state predictions from dandiieradd dynamies models. This key idea is illustrated in Figure 1, where spikes in uncertainty (e.g. at 4.6 and 5.3) coincide with unstable or unsafe behaviors, such as stumbling or flipping. Motivated by this insight, we propose to penalize the cost with the uncertainty to achieve safe sim-to-real transfer, leading to the design of SPIDR. Notably, SPIDR seamlessly integrates with state-of-the-art RL algorithms (Schulman et al., 2017: Haarnoja et al., 2019), delivering strong empirical performance both in simulated and real-world safe RL tasks while ensuring constraint satisfaction, even under severe model mismatch.
Out contribution.
We address an important challenge to real- woal-world adoption of RL: zero-shot safe sim-to sim-to-real real transfer, where an agent must learn safe and effective policy using only simulated interactions. We propose SPIDR, practical algorithm with formal safety guarantees that integrates easily into popular sim-to-real pipelines.
We validate SPiDR on two real-world robotic platforms, where it achieves zero-shot constraint satisfaction, substantially outperforming other baselines in terms of safety and performance. These results provide empirical evidence that our theoretical guarantees translate to the real-world, suggesting that SPIDR can safely used in real-world deployment.
Finally, extensively evaluate SPIDR on well established simulated continuous control bench- marks, including the RWRL benchmark (Dulac-Arnol, et al., 2020), Safety Gym (Ray al., 2019) and RaceCar environments (Kabzan et al., 2020), where SPIDR consistently satisfies safety constraints while achieving strong task performance.
2 Related Works
Safe sim-to-real transfer can be naturally framed as a constrained Markov decision process (CMDP, Altman, 1999) under model uncertainty. common approach to extend CMDPs using tools from robust optimization (Iyengar, 2005; Ben-Tal et al., 2009), which has led growing body

---

of work spanning both theoretical and practical contributions. We refer to Appendix N for a more comprehensive discussion of each line of work.
Provably robust algorithms. Zhang al. (2024) build game-theoretic formulation develop a tractable primal-dual algorithm with provable non-asymptotic convergence guarantees to safe policy. Kitamura et al. (2024) take this further by proposing policy gradient algorithm with formal guarantees for safety and optimality, via an epigraph form of the robust CMDP problem. Robust CMDPs are inherently challenging since the worst-case scenarios w.r.t. the reward and cost can differ (see Kitamura et al., 2024); both works make notable theoretical progress on this front. While sharing the goal of provably safe transfer, our work adopts more scalable and modular approach by building on domain randomization, and integrating with existing CMDP solvers, avoiding the complexity of solving the minimax formulation, common in robust optimization.
Scalable robust algorithms. Russel et al. (2020) and Mankowitz et al. (2020) were among the first to study robust CMDPs in the context of deep RL, proposing practical methods that scale to continuous control tasks. Similarly, Queeney and Benosman (2024) introduce RAMU, algorithm that uses coherent risk measures (Shapiro, 2017) with temporal differences (TD) learning, achieving strong empirical performance on the RWRL benchmark, though lacking formal safety guarantees. Sun et al. (2024) extends CPO (Achiam et al., 2017) to problems with model uncertainty, providing safety and performance guarantees. Lastly, Bossens (2024) proposes to learn adversary policy and show that their algorithm is robust w.r.t L, -n-rm uncertainty sets. Our work differs in that it provides safety guarantees while remaining scalable and not tied particular RL algorithm.
Practical methods for safe sim-to- to-real. While the previous works develop methods for solving robust CMDPs-often motivated by the practical problem of safe sim-to-real transfer-other prior works address this problem directly. Kaushik et al. (2022) use online data from the real system and employ safe Bayesian optimization (Sui et al., 2015) to select safe and geghpeerfrrmig policies from collection of policies trained in simulation. Similarly, Hsu et al. (2023) propose multi-fidelity approach (Cutler et al., 2014), incorporating a fine-tuning step in a high-fidelity simulator prior to deployment. Both works emphasize practical applicability and provide strong empirical validation in real robotics settings. Compared to these works, in this work we focus on guaranteeing safe transfer without access to online data computationally expensive intermediate simulator.
3 Problem Setting
Constrained Markov decision process. We study discounted infinite-horizon CMDPs, defined
by tuple M = (S.AAp.. C, 7, p). Here, S and A are the state and action spaces, S x AA + ASS) denotes the transition probability of the system dynamics. the reward function is given by S:S x A + [0, [ma] and the cost is given by e: S x A + [0, Cmax]. The discount factor is y E [0. 1) and p ASS) is probability distribution from which initial states are drawn. We consider the class of stationary policies IT5, where each policy is stochastic mapping from states to actions π : S > (AA). For given dynamics p. the value function under policy at state s is defined as (s) & E.. (Eeo 7'rz s). Similarly, the cost value is given by V..a (s) - Ep, iE cc s). We define the expected value function of when the initial state is sampled from P as JT(T)) A BV (s)), and the expected cost value as (T) & EEp (VP:((s)) The goal to find policy ा that solves
max (T( s.t. Cp(T) < d, #EIf'
where d >> is a predefined budget. This formulation enables explicit decoupling of safety from the objective. For example, in robotics, the cost can represent collisions with obstacles, while the reward encourages reaching a goal.
In this work, we consider a setting where the agent has access to a simulator, capable of generating any environment Mg = (S, A. fç. r, c, 7, p) given & E C RR4, The dynamics of each environment are parameterized by E. typically representing the physical properties of the system dynamics. The agent can freely interact with any simulated environment P5 with & E. but has access the unknown real environment M = (S.A. p", C, The
Safe sim-to-real transfer.
'While only the class of distory--ependee policies is formally complete under domain randomization, we focus on 11" due to its simplicity. Our theoretical and empirical results can be directly extended to history- dependent policies. See Dolgov and Durfee (2005); Kwon et al. (2021); Chen et al. (2022) for further discussions.
3

---

objective is to learn a policy x E II's purely within the simulator such that, when deployed in the real environment M satisfies the safety constraint Cp- (=) S d. Crucially, the agent must guarantee constraint satisfaction "zero-shot", without any direct interaction with the real environment M
4
SPiDR for Safe Zero-Shot Sim-to-Real Transfer
4.1
Domain Randomization
Extending domain randomization to CMDPs. Domain randomization is particularly well-suited
for the problem setting described above, it leverages set { Lid μ of parameterized environments, sampled independently from some probability distribution j.. This distribution acts as prior for the real, yet unknown system parameters. A natural approach for tackling safe sim-to-real problems is by formulating CMDPs over distribution of randomized domains, i.e., solving
max max Pe (TT s.t. TET
Cp (x) S d.
Domain randomization can be seen as a sample average approximation of Equation (2), making it both straightforward to implement and scalable, collect data from each environment in parallel.
meassively-parallee simulators can be used to
Domain randomization is not always safe. While Equation (2) provides a compelling formulation from a practical standpoint, it does not guarantee safety in the real environment M Specifically, since simulators only approximate the real world with limited precision, as well as due to averaging over dynamics, the costs in Equation (2) may underestimate the true costs in M This limitation empirically validated in Sections 5.1 and 5.2 (Figures and 6, respectively) and theoretically illustrated through an example in Appendix A. In what follows, we formally characterize constraint underestimation and show how SPIDR is designed to mitigate it.
A pessimistic upper bound. We quantify the extent by which constraints on the real system may be underestimated by establishing the following bound. To this end, we measure the discrepancy between the simulated and real dynamics using the Wassserste distance, denoted as Dw (fe. ps, a). whose formal definition is provided Definition C.1. While our analysis is based on this metric, it naturally extends to other discrepancy measures. We assume this discrepancy is finite for all & E E, which is a reasonable assumption in practice. For instance, the simulators by Makoviychuk et al. (2021) and Zakka et al. (2025) have been successfully used for zero-shot sim-to-real transfer across several robotic platforms, suggesting that they maintain high fidelity with real-world systems. We now present our bound below.
Lemma 4.1. Let Let.p.re (s) denote the probability of reaching the state s at step under the policy т and the dynamics p. and let dp,x - (1 (1-99 (alss P' denote the normalized discounted occupancy measure of policy under the dynamics p. The woal-world cost Cpt (π) can be upper-bounded by
Cp (T)
Cpe +Bg-n [Bo
7 Dw (',,,'))s
(3)
11-7
Constraint simulaaiio
where Le is the Lipschitz constant of the state cost function Vg" (s))
Lemma 4.1 shows that the true safety constraint function Cp- (T)) is upper-bounded by the constraint evaluated during training, and the expected L1- Wasseesstein distance with respect to the state-action occupancy measure of π over the simulated dynamics Pg. Importantly, establishes that even when the constraint is satisfied in simulation, i.e., Be (TT) S d, the constraint on the real system may be larger, depending on π and the degree of mismatch between the simulated and real dynamics. Therefore, by bounding the r.h.s. of Equation (3) with d, we guarantee that Cp- (i) & d, hence safe transfer to the real system. We refer to Appendix for the formal proof and assumptions of Lemma 4. We next show how this key insight is used in our design of SPIDR.
4.2 Algorithm Design
Reduction to penalized CMDPs. Observing that by linearity of expectation, the r.h.s. of Equation (3) can be written
LLC Dw (De, (s, (s.
Egnn
B( c(s,a)
4

---

This simple insight motivates the use of
a(s,a) = + LLC max Dw (pc, p" (s)
(4)
y EE
penalty
as a surrogate cost function during training. Crucially, it is infeasible to estimate the model discrepancy Dw (fe,, p² (s,a) for any 5 E, therefore we use the worst-case approximation. Using ö(;; .) yields a penalized CMDP (S, AS,AA, Pe, C,7, This CMDP is still fully compatible with domain randomization, allowing us to solve
conservative
max max Pe (TT s.t. Ee sd. TETP=
(5)
where Opg (T) denotes the constraint with ((---) following dgc While a(::) can be used as a conservative approximation that in principle guarantees safe transfer, direct access to the penalty term is generally intractable. This is due to the fact that Le and Dw (w are unknown a priori and can only be estimated using access to ground-truth data from the real system. Therefore, we propose approximating the penalty term using only simulated data.
iid Approximating the penalty term. We propose using an ensemble of {pE: ] j dynamics and measuring their disagreement in predicting the next state, as a proxy for the uncertainty about the model discrepancy maXçez Dw pp). Specifically, when s; p.. s, a) are i.i.d. samples, we define the sum of component-wist wise empirical variances of the ensemble next-state predictions as the estimator
dim(S)
vss a) _ ||Var (s1....,,s)ll. =
Σ Var ($1.f1..., Sn.)) j=1
(6)
where (S) is the dimension of the state space and $t denotes the j-th component of the i-th sample. The empirical variance v(s, a) measures the sensitivity of the environment in predicting the next state w.r.t. 5. making it an effective proxy for our uncertainty about the model discrep- ancy maxeg= Dw(Pe, p²), especially when the real environment lies near the simulated ones. In Appendix C, we formally show that for a suitable constant \, and under bounded model mismatch, Au(s,a) a) upper-bounds ThE maxçe= Dw (fe, pd) (s, a), with the bound becoming tighter as n in- creases, Additional practical guidance on how to pick empirically is provided in Appendix E. We find this approach simple to implement, computationally efficient and effective in practice, as demon- strated in Section 5 (Figures 3 and 7). Moreover, in Figure 8 we demonstrate how to pick n in practice.
The algorithm. With the above approximation v(:;.) in hand, entire algorithm SPiDR, summarized in Algorithm 1. Standard domain randomization typically involves policy search methods such as policy gradients or TD learning, which require collecting trajectory data from the simulator. These trajectories are collected independently and in parallel for each dynamics {E+), To incorporate pessimism, we modify only the procedure by which these trajectories are obtained. This abstraction allows us to keep using domain randomization as it is while remaining versatile w.r.t. the choice of policy search algorithm. Specifically, following standard domain randomization, SPiDR samples a batch of dynamics {E: - These dynamics are used only to rollout the policy (Lines and 7). For each sampled dynamics, we further employ an ensemble {pc, 1 This ensemble is used to estimate the penalty term as v(.,.) .) (Lines 8 to 10). These trajectories in the "penalized" CMDP using ē are collected and used to solve the penalized CMDP in Equation (5). Importantly, Lines 8 to 10 represent the only modifications made relative to standard domain randomization.
ready introduce the
4.3 Safety Guarantee
Next, we demonstrate our theoretical guarantees for a solution to Equation (5). We first assume the feasibility of the problem, otherwise, a solution can be recovered only by improving the simulator or with safe online learning techniques (As al., 2025). We show that any solution to the penalized CMDP provably achieves safety in the real environment by the following theorem.
Theorem 4.2. Let p" be the dynamics of the real environment M Let ã be a solution to the penalized CMDP introduced in Equation (5), i.e., मे max Eę pe (IT) Eg- Cpg (π) s d.
Then ñn satisfies the safety constraint in the real environment, namely, Cp. (7) S d.
5

---

Algorithm SPiDR: Safe Sim-to-Real via Pessimistic Domain Randomization
1: Input: pessimism A, initial distribution j4, behavior policy π
2: Init: Sample {<r) {Sig 15=1 iid
3: for i 1,, ., in parallel do
Collect data from each E
4: Initialize trajectory 7(0) 4- 0
5: for ( 0, do
Rollout policy
6: de ~ ~- kt)
7: Simulate 8t+1 (t St, az). obtain rı, Ct
8:
Simulate (2) ~ PE.,( St, as) with each {Ps., 7==1 Compute the penalty term v(st, at) in Equation (6)
> Fast parallel execution
9:
10:
Penalize cost & <- c(se, a4) (u oz)
11:
Append (82, a22 C2) to x()
12: Set next state 8, + 81+-1
13:
end for
14:
end for
Solve Equation (5) using {-(1) Ji=1 obtain क with any CMDP solver
16: return π
The formal proof of this theorem, including its assumptions, is provided Appendix Solving the CMDP with any penalty on the cost that dominates maxgez Dw (Pe,""") yields more conservative policy. Any such policy will remain safe in the real environment. The drop in performance due to cost penalization is not captured by Theorem 4.2 as it also depends on cost and reward function. However, our world experiments Section 5 (Figure 3) demonstrate that SPiDR consistently matches or outperforms standard domain randomization. Adopting any penalty the cost reduces the set of feasible policies. Therefore, the agent may avoid high-reward regions that are in reality safe but uncertain in simulation. However, the limited drop observed in practice suggests that bound o(s)) provides a reasonably tight werrapprooimation of the worst-case model discrepancy.
5
Experiments
Next, we demonstrate SPIDR's performance in practice. Below we provide our sim-to-real ex- periments two robotic tasks, followed by comprehensive ablations several tasks from three well-established safe RL benchmarks. We refer the reader to Appendices D and for additional details on how À is picked in practice and for more ablations.
Setup. Unless otherwise specified, all experiments use SAC (Haarnoja et al., 2019; Nauman et al., 2024) in combination with either CRPO (Xu et al., 2021) optimization method (Bertsekas, 2016). We use these CMDP solvers since they deliver strong results while remaining casy implement. We run each experiment with five random seeds and report the mean and standard error. Empirical estimates of the objective and constraint on the test enviromments are denoted by ((H) and C(#), respectively. We compare SPiDR with the following baselines: (i) Nominal is simple baseline that collects trajectories only from the nominal training dynamics; (fi) Domain Randomization collects trajectories only from the perturbed dynamics of the training distribution; and (iii) RAMU (Queeney and Benosman, 2024), a state-of-the-an robust safe RI algorithm designed specifically for TD learning methods.
simple primal-dual constrained
5.1 Real-World Deployment
We demonstrate SPiDR's effectiveness on two real-world robotic tasks: a highly dynamic remote- controlled race car, and a Unitree Gol quadruped robot, illustrated in Figure 2. Each policy is trained only in simulation and then evaluated in the real world: five trials for the remote-controiled car and ten trials for Unitree Gol. These trials are used to obtain ((#)) and (i+) where applicable. In both tasks, compare SPIDR with domain randomization evaluated on the real system. Additionally,
to demonstrate how domain randomization may underestimate the constraint in real, we compare its performance with SPiDR, when evaluated during training simulation. Further details about the tasks and hardware are provided in Appendices G and H.
Experiment I: Does SPIDR transfer safely real systems? We present our results in Figure As shown, in both tasks, SPIDR satisfies the constraint, whereas domain randomization dramatically

---

Goal
Obstacles
t 0[sec]
t= s[scc]
t= 6[sec]
t = [sec]
t
[[sec]
= 1.5[sec]
t = [.5[sec]
t = 3.5[
Figure 2: Example trajectories SPIDR with RaceCar and Unitree Gol.
SPIDR (sim-to-sim)
Domain Randomization (sim-to-sim)
SPiDR (sim-to-real)
Domain Randomization (sim-to-real)
RaceCar
Unitree Gol
100
300
150
6.2x higher
26. Ix higher than SPiDR
75
than SPiDR
((
100
100
50
Budget
Budegj
Figure 3: Performance on the race car and Unitree Gol. SPiDR sim--o-sim) and SPIDR (sim-to- real) represent evaluation in simulation and on the real system respectively. SPIDR transfers safely, while domain randomization dramatically violates the safety constraints.
violates it, even when the constraint is satisfied in simulation. This result consistent with Lemma 4.1, due to constraint underestimation in simulation. Remarkably, due to the highly dynamic behavior of the remote-control car, the performance of the objective with domain randomization underperforms compared to SPiDR. This is because domain randomization often overshoots the target, whereas SPIDR's more conservative hence slower policies avoid overshooting. To evaluate the performance we measure the rewards on the real system are report the performance in Figure 4. As demonstrated in Figure 4, both domain randomization and SPIDR exhibit comparable decrease in performance relative to their simulated counterparts. These results demonstrate that SPIDR transfers effectively to the real-world system without being overly conservative, successfully solving the task. We additionally provide five trajectories for each policy and algorithm, resulting in total of 75 video demonstrations, provided the following link. These recordings indicate that using SPiDR does lead to a noticeable degradation in locomotion performance. In comparison, RAMU, trained with its default hyperparameters, succeeds following the commands in 12 + 4.8% of the trials, falling the rest, while SPIDR completes all trials successfully without falling. See Appendix G for more details. These results suggest that SPIDR works well across different real-world robotic tasks, satisfying the constraints while maintaining strong performance.
the Unitree Gol robot,
SPIDR to sim)
Domain Randomization (sim-to-sim)
SPIDR (sim-to-real)
Domain Randomization (sim-to-real)
20
10
Figure 4: Performance is main- tained on the Unitree Gol.
Experiment How versatile is SPIDR? We replace SAC with PPO (Schulman al., 2017) and a primal-dual optimizer as CMDP solver, repeating our experiment on the Unitree Gol robot. We present our results in Figure 5, demonstrating that SPIDR satisfies the constraints on the real system, while domain randomization with PPO satisfies the constraint only in simulation. We provide additional sim-to-sim experiments with PPO in Appendix Throughout this work, three different CMDP solvers, with two different policy search methods, demonstrating

---

SPIDR
RAMU
Domain Randomization
Nominal
RaceCar
CartpoleSwingup
Humanoid Walk
30
400
200
20
300
10
Budget
Budget
Budget
0
0
2
0
2
X107
x1u
X10
QuadrupedRun
WalkerWalk
PointGoal2
300
200
75
200
50
Budget
Budget
Budgel
x10
x10
x10
Simulation Steps
Figure 6: Costs over training iterations, evaluated on the simulated test environments. SPiDR consistently satisfies the constraints across all tasks.
competitive performance while remaining safe, including on real hardware. These results highlight the versatility and broad applicability of SPIDR across different CMDP solvers and RL algorithms.
5.2 Evaluation in the Sim-to-Sim Sandbox
SPIDR (sim-to-sim)
Next, we demonstrate that SPiDR consistently achieves strong task performance while adhering safety constraints in a series of sim-to-sim experiments. This set of experiments serves as a controlled testbed for systematically ablating SPIDR. We provide additional ablations on the robustness to choice of A empirical intuition about u(:.)) Appendix D. Full learning curves and further details on the experimental setup can be found in Appendices F and L.
Domain Randomization (sim-to-sim)
SPIDR (sim to real)
Domain Randomization (sim to real)
200
Experiment Can SPIDR satisfy the constraints under sim-to-sim gap? We demonstrate results in Figure 6, reporting the learning curves of constraint across tasks. Figure 6 demonstrates that SPIDR satisfies the constraints all six tasks. This is in contrast to Nominal and Domain Randomization, which fail to satisfy the constraints across all tasks, expected. In the RaceCar task, where the sim-to-sin gap primarily arises due to the training simulator failing to capture all the relevant phenomena present in the test environment, SPIDR is the only algorithm that satisfies the constraint.
Budgej
Figure 5: Safe transfer to real Uni- tree Gol with PPO.
Experiment 4: What is the tradeoff between safety and performance? We study the tradeoff between constraint satisfaction and objective performance, showing that SPIDR does not suffer significant performance decrease. We present our results Figure 7. As shown, SPIDR maintains balance of high performance while satisfying the constraints across all tasks. SPIDR and RAMU demonstrate comparable performance on the RWRL environments. On the other hand, on the RaceCar task, which involves more realistic and challenging model discrepancies, SPIDR significantly outperforms RAMU. Notably, in environments like Humanoid QuadrupedRum, WalkerWalk and RaceCar, SPiDR not only meets the safety constraints but also achieves competitive performance with respect to the optimal performance on these tasks. These results illustrate that performance not significantly compromised, even in the presence of large model mismatch.
Experiment 5: How does SPIDR scale? We analyze how the performance of SPiDR is affected by the ensemble size terms of training wall-clock time and performance. test SPiDR on Walker- Walk, QuadrupedRun and HumanoidWalk while varying 21, 4, 8, 16, 32, 64, 128). In Figure 8

---

SPIDR X RAMU
Domain Randomization
Nominal
* Optinum on AM
RaceCar
CartpoleSwingup
HumamoidWalk
1.0
Normalized Objective
0.5
Budget X2.0
X4.0
Budget
x1.2
x1.4
Budget
x2.0
x3.0
QuadrupedRun
WalkerWall Walk
PointGoal2
1.0
Normalized
Objective
0.5
x
Unsafe
0.0
Budget x2.0
x3.0
x0.5
Budget
x1.5
x2.0
Budget
x2.0
C(#)/a
C(*)/a
C(#)/a
Figure Normalized objective and constraint performance after training. The constraint normalized relative to the budget, while the objective is normalized against the optimal performance on M Shaded red area represents unsafe policies. SPiDR consistently satisfies the constraints while achieving competitive performance on different locomotion and navigation tasks.
WalkerWalk
Quadrupe:RRun
HumanoidWalk
((#)
Normalized Runtime
1000
750
150
Constraint satistied
x2.0
Good runtime
500
Budget
x1.5
50
0
x1.0
2'
2
27
2
27
2
Ensemble Size
Ensemble Size
Ensemble Size
Figure 8: Runtime overhead of SPIDR is marginal even for 8. For 2 8 safety is maintained across environments.
we report the objective, constraint and the relative training runtime compared to standard domain randomization. As shown, for 2 the constraint remains within budget in all environments, while increasing n has little effect on the objective. Furthermore, on all three environment the relative run- time for 32 only marginally longer compared to standard training with domain randomization. All experiments are run using MuJoCo XLA (Freeman et al., 2021), enabling us to train SAC for 5M and PPO for 200M environment steps in under an hour on single NVIDIA RTX 4090 GPU.
Does SPIDR scale to vision control tasks? We investigate whether SPIDR scales to partially observable vision-based settings, where the policy operates directly on rendered images. To evaluate this, implement an asymmetric teacher-student Lee et al. (2020) setup: the policy receives only image observations, while the cost is penalized using privileged state information available in simulation, following Equation (6). Specifically,
combine DrQ (Yarats al., 2021) with CRPO (Xu et al., 2021) ) as the penalizer and apply SPiDR on the CartpoleSwingup task. This setup follows our previous sim-to-sim experiment on the CartpoleSwingu task, only differing in the observation given to the agent. The policy trains on a sequence of stacked grayscale 64 x 64 pixels images, while the cost penalty uses the true system states (not shown the agent). We report our results on the evaluation environment in Figure 9. These results show that even in a partially observable setup, SPIDR satisfies the constraint by leveraging privileged information during training. Training the vision-basee based policy for 1M simulation steps with SPIDR takes roughly 33 minutes on an RTX 4090 GPU, compared to roughly 30 minutes without SPIDR. The main computational bottleneck lies in computing critic gradients, while simulation efficiency comes from Madrona- MIX (Rosenzweig et al., 2024), which renders rollouts from 128+ environments parallel. This experiment suggests that SPIDR scales effecttivlyy to vision-based control tasks.

---

SPIDR
Domain Randomization
1000
800
800
600
600
400
400
200
200
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Simulation Steps
x 100
Simulation Steps
X106
Figure 9: Learning curves of vision-based CartpoleSwinguy task. SPIDR satisfies the constraints under dynamics mismatch even when the policy observes grayscale images of the cartpole.
6 Conclusions
In this work address safe sim-to-real transfer, key challenge limiting the broader adoption of RL in real-world applications. We theoretically link constraint violations upon deployment to uncertainty about the sim-to-real gap and propose simple and provably safe algorithm that penalizes the cost function using an estimated gap under domain randomization. Our proposed method SPiDR is theoretically sound, easy implement and be readily combined with popular safe RI. algorithms. We empirically show that SPiDR consistently achieves strong performance while maintaining safety different tasks spanning three different simulated safe RL benchmarks. Moreover, SPIDR solves two real-world robotic tasks as it ensures safe transfer, proving its practical applicability to the full-scale problem it aims to address. Although the simplicity of SPIDR, we believe that future work can extend it to other Finally, since our method focuses on the "zero-shot" transfer setting, where access to real- world data is restricted, proposing hybrid approaches that combine simulation-traines policies with safe online exploration techniques an important direction for future work.
focus our experiments
robotic tasks, due to domains.
10

---

Acknowledgments and Disclosure of Funding
We would like to thank Taerim Yoon, Yonathan Efroni, Kishan Panaganti, Chenhao Li and James Queeney for insightful discussions during the development of this project. We thank the anonymous reviewers for their valuable comments and suggestions. This project has received funding from grant of the Hasler foundation (grant no. 21039) and the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No. 866480). Adam Wierman supported by NSF through CNS-2 2168 14, CPS-2136197, CNS-2106403, NGSDI2105648, IIS-2336236, and by the Resnick Sustainability Institute at Caltech.

---

References
Abbas Abdolmaleki, Jost Tobias Springenherg. Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint arXiv: /800:06966 2018. 8. (Cited on page 41)
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization, 2017. (Cited on pages 3 and 43)
Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement Learning: Theory and Algorithms. 2021. (Cited on page 28)
E. Altman. Constrained Markov Decision Processes. Chapman and Hall, 1999. (Cited on pages and 43)
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in safety, 2016. (Cited on page 1)
OpenAl: Marein Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray. et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 2020. (Cited on page 43)
Yarden As, Ilnura Usmanova, Sebastian Curi, and Andreas Krause. Constrained policy optimization via bayesian world models, 2022. (Cited on page 43)
Yarden As, Bhavya Sukhija, Lenart Treven, Carmelo Sferrazza, Stelian Coros, and Andreas Krause. Actsafe: Active exploration with safety constraints for reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025, (Cited on page 5)
Kishan Panaganti Badrinath and Dileep Kalathil. Robust reinforcement learning using least squares policy iteration with provable performance guarantees. In International Conference Machine Learning. PMLR, 2021. (Cited on page 43)
A. Ben- Ben-Tal, L.E. Ghaoui, and A. Nemirovski. Robust Optimization. Princeton Series in Applied Mathematics. Princeton University Press, 2009. (Cited page 2)
D. Bertsekas. Nonlinear Programming. Athena scientific optimization and computation series.
Athena Scientific, 2016. (Cited on page 6)
Homanga Bharadhwaj. Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and
Animesh Garg. Conservative safety critics for exploration. arXiv preprint arXhv:2010.14497 2020. (Cited on page 43)
David M Bossens. Robust lagrangian and adversarial policy gradient for robust constrained markoy decision processes. In 2024 IEEE Conference on Artificial Intelligence (CA/). IEEE, 2024. (Cited on page 3)
George EP Box. Science and statistics. Journal of the American Statistical Association, 1976. (Cited on page 1)
Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. Safe learning in robotics: From learning- based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 2022. (Cited on page 43)
Weiqin Chen, Sandipan Mishra, and Santiago Paternain. Domain adaptation for offline reinforcement learning with limited samples. arXiv preprint arXiv:2408. 12136, 2024. (Cited on page 43)
Xiaoyu Chen, Jiachen Hu, Chi Jin, Lihong Li, and Liwei Wang- Understanding domain randomization for sim-to-real transfer. In International Conference on Learning Representations, 2022. (Cited on pages 3 and 43)
Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. A lyapunov- based approach to safe reinforcement learning. Advances neural information processing systems, 2018. (Cited page 43)
12

---

Duphne Cornelisse, Aarav Pandya, Kevin Joseph, Joseph Suárez, and Eugene Vinitsky. Building reliable sim driving agents by scaling self play, arXiv preprint OIXv:2502.14706, 2025, (Cited on page !)
Sebastian Curi, Iija Bogunovic, and Andreas Krause. Combining pessimism with optimism for robust and efficient model-based deep reinforcement learning, 2021. (Cited on page 43)
Marco Desumano-Towwir David Hafner, Alex Hertzberg, Brody Huval, Aleksei Petrenko, Eugene Vinitsky, Erik Wijmans, Taylor Killian, Stuart Bowers, Ozan Sener, et al. Robust autonomy emerges from self-play. arXiv preprint arXiv:2502. 03049, 2025. (Cited on page 1)
Mark Cutler, Thomas J Walsh, and Jonathan P How. Reinforcement learning with multi-fidelity simulators. 2014 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2014. (Cited page 3)
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Hwalds, Roland Hafner, Abbas Abdolmaleki, Diego de las Casas, Craig Donner, Leslie Fritz, Cristian Galperti, Andrea Huber, James Keeling, Maria Tsimpoukelli, Jackie Kay, Antoine Merle, Jean-Marc Moret, Seb Noury, Federico Pesamosca, David Pfau, Olivier Sauter, Cristian Sommariva, Stefano Coda, Basil Duval, Ambrogio Fasoli, Pushmeet Kohli, Koray Kavukcuoglu, Demis Hassabis, and Martin Riedmiller. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 2022. (Cited on pages and 2)
Esther Derman and Shie Mannor. Distributional robustness and regularization in reinforcement learning. arXiv preprint arXiv:2003. 02894, 2020. (Cited on page 43)
Wenhao Ding, Laixi Shi, Yuejie Chi, and Ding Zhao. Seeing is not believing: Robust reinforce- ment learning against spurious correlation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023, (Cited on page 43)
Dmitri A Dolgov and Edmund H Durfee. Stationary deterministic policies for constrained mdps with multiple rewards, costs, and discount factors. In ICCAI, 2005. (Cited page 3)
R. M. Dudley. Real Analysis and Probability. Cambridge Studies in Advanced Mathematics. Cambridge University Press, 2 edition, 2002. (Cited on page 28)
Gabriel Dulac-Amold, Nir Levine, Daniel Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. An empirical investigation of the challenges of real-world reinforcement learning. 2020. (Cited on pages and 40)
Ian Fox, Joyce Lee, Rodica Pop-Busui, and Jenna Wiens. Deep reinforcement learning for closed- loop blood glucose control. Finale Deshi-Velez, Jim Fackler, Ken Jung, David Kale, Rajesh Ranganath, Byron Wallace, and Jenna Wiens, editors, Proceedings of the 5th Machine Learning for Healthcare Conference, Proceedings of Machine Learning Research. PMLR, 2020. (Cited on pages and 23)
C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax differentiable physies engine for large scale rigid body simulation, 2021. (Cited pages and 39)
Uri Gadot, Kaixin Wang, Navdeep Kumar, Kfrr Yehuda Levy, and Shie Mannor. Bring your own (non- robust) algorithm to solve robust MDPs by estimating the worst kernel. In Forty-first International Conference Machine Learning, 2024. (Cited page 43)
Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 2015. (Cited page 43)
Clark R. Givens and Rae Michael Shortt. A class of Wasserstein metrics for probability distributions. Michigan Mathematical Journal, 1984. (Cited page 29)
Vineet Goyal and Julien Grand-Clement, Robust markov decision processes: Beyond rectangularity. Mathematics of Operations Research, 2022. (Cited on page 43)
13

---

Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205. 10330, 2022. (Cited on pages and 43)
Shangding Gu, Bilgehan Sel, Yuhao Ding, Lu Wang, Qingwei Lin, Ming Jin, and Alois Knoll. Balance reward and safety optimization for safe reinforcement learning: perspective of gradient manipulation. In AAAI, 2024. (Cited page 43)
Tuomas Haarooja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critt algorithms and applications, 2019. (Cited on pages 2, and 41)
Tairan He, Weiye Zhao, and Changliu Liu. Autocost: Evolving intrinsic cost for zero-violation reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023. (Cited on page 38)
Chin Pang Ho, Marek Petrik, and Wolfram Wiesemann. Fast bellman updates for robust MDPs. In International Conference on Machine Learning. PMLR, 2018. (Cited on page 43)
Chin Pang Ho. Marek Petrik, and Wolfram Wiesemann. Partial policy iteration for 11-robust markov decision processes. Journal of Machine Learning Research, 2021. (Cited on page 43)
Kai-Chieh Hsu, Allen Z Ren, Duy P Nguyen, Anirudha Majumdar, and Jaime Fisac. Sim-to- lab-to-real: Safe reinforcement learning with shielding and generalization guarantees. Artificial Intelligence, 2023. (Cited on page 3)
Weidong Huang, Jiaming Ji, Chunhe Xia, Borong Zhang, and Yaodong Yang. Safedreamer: Safe reinforcement learning with world models. arXiv preprint 2rXiv:2307.07176, 2023. (Cited page 38)
Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 2005. (Cited on pages and 43)
Anders Jonsson, Emilie Kaufmann, Pierre Ménard, Omar Darwiche Domingues, Edouard Leurent, and Michal Valko. Planning in markov decision processes with gap-dependent sample complexity. arXiv preprint arXiv:2006. 05879, 05879, 2020. (Cited on page 30)
Juraj Kabzan, Miguel Valls, Victor JF Reijgwart, Hubertus FC Hendrikx, Claas Ehmke, Manish Prajapat, Andreas Bühler, Nikhil Gosala, Mehak Gupta, Ramya Sivanesan, et al. Amz driverless: The full autonomous racing system. Journal of Field Robotics, 2020. (Cited on pages and 41)
David L Kaufman and Andrew Schaefer. Robust modified policy iteration. INFORMS Journal on Computing, 2013. (Cited page 43)
Rituraj Kaushik, Karol Arndt, and Ville Kyrki. Safeapt: Safe simulation-to-real robot learning using diverse policies learned in simulation. IEEE Robotics and Automation Letters, 2022. (Cited on page 3)
Saman Kazemkhani, Aarav Pandya, Daphne Cornelisse, Brennan Shacklett, and Eugene Vinit- sky. Gpudrive: Data-driven, multi-agent driving simulation at million fps. arXiv preprint arXiv:2408.01584, 2024. (Cited page
Dohyeong Kim, Kyungjae Lee, and Songhwai Oh. Trust region- safe distributional reinforce- ment learning for multiple constraints. Advances in neural information processing systems, 2023, (Cited on page 43)
Yunho Kim, Hyunsik Oh, Jeonghyun Lee, Jinhyeok Choi, Gwanghyeon Moonkyu Jung. Donghoon Youm, and Jemin Hwangbo. Not only rewards but also constraints: Applications on legged robot locomotion. IEEE Transactions on Robotics, 2024. (Cited on page 43)
Toshinori Kitamura, Tadashi Kozuno, Wataru Kumagai, Kenta Hoshino, Yohei Hosoe, Kazumi Kasaura, Masashi Hamaya, Paavo Parmas, and Yutaka Matsuo. Near-optimal policy identi- fication in robust constrained markov decision processes via epigraph form. arXiv preprint arXiv:2408. 16286, 2024. (Cited pages and 3)
14

---

Jeongyeol Kwon, Yonathan Efroni, Constantine Caramanis, and Shie Mannor. RI for latent mdps: Regret guarantees and lower bound. Advances in Neural Information Processing Systems, 2021. (Cited on page
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science Robotics, 2020. (Cited on pages 1, 2, and 39)
Joonho Lee, Lukas Schroth, Victor Klemm, Marko Bjelonie, Alexander Reske, and Marco Hutter. Evaluation of constrained reinforcement learning algorithms for legged locomotion. arXiv preprint arXiv:2309. 15430, 2023. (Cited on page 43)
Toru Lin, Kartik Sachdev, Linxi Fan, Jitendra Malik, and Yuke Zhu. Sim-to-real reinforcement learning for vision-based dexterous manipulation on humanoids. arXiv preprint arXiv:2502.20334 2025. (Cited page 1)
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprini arXiv:2 10470, 2021. (Cited on pages and 4)
Daniel J Mankowitz, Dan A Calian, Rae Jeong. Cosmin Paduraru, Nicolas Heess, Sumanth Dathathri, Martin Riedmiller, and Timothy Mann. Robust constrained reinforcement learning for continuous control with model misspecification. arXiv preprint arXiv:2010. 10644, 2020. (Cited on page 3)
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, February 2015. (Cited on page 1)
Pierre Ménard, Omar Darwiche Domingues, Anders Jonsson, Emilie Kaufmann, Edouard Leurent, and Michal Valko. Fast active learning for pure exploration in reinforcement learning. In Pro- ceedings of the 38th International Conference on Machine Learning. PMLR, 2021. (Cited on page 30)
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, 2020. (Cited on page 43)
Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miloš, and Marek Cygan. Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control. arXiv preprint arXiv:2405. /6/58, 2024. (Cited page 6)
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. (Cited on page !)
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforeement learning with adversarial attacks. arXiv preprint arXiv:1712.03632, 2017.
(Cited on page 43)
Xue Bin Peng, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, May 2018. (Cited on pages and 43)
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial rein- forcement learning. In International Conference on Machine Learning. PMLR, 2017. (Cited on page 43)
15

---

James Queeney and Mouhacine Benosman. Risk-averse model uncertainty for distributionally robust safe reinforcement learning. In Proceedings of the 37th International Conference on Neural Information Processing Systems, 2024. (Cited on pages 1, 2, 3, 6, 36, 40, and 41)
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking Safe Exploration in Deep Reinforce- ment Learning. 2019. (Cited on pages and 39)
Luc Guy Rosenzweig, Brennan Shacklett, Warren Xia, and Kayvon Fatahalian. tigh-throughput batch rendering for embodied ai. 2024. (Cited on page 9)
Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Aleksandra Faust, David Hsu, and Gerhard Neumann, editors, Proceedings of the 5th Conference on Robot Learning, Proceedings of Machine Learning Research. PMLR, 2022. (Cited on page
Reazul Hasan Russel, Mouhacine Benosman, and Jeroen Van Baar. Robust constrained mdds: Soft- constrained robust policy optimization under model uncertainty. arXiv preprint arXiv:2010.04870, 2020. (Cited page 3)
Fereshteh Sadeghi and Sergey Levine. Cad2rl: Real single-image flight without single real image. arXiv preprint arXiv:1611.04200 2016. (Cited on page 43)
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization, 2015. (Cited on page 41)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. (Cited on pages 7, and 39)
Alexander Shapiro. Distributionally robust stochastic programming, SIAM Journal on Optimization, 2017. (Cited on page 3)
David Silver, Aja Huang, Christopher J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schritwiieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 2016. (Cited on page 1)
Elena Smirnova, Elvis Dohmatob, and Jérémie Mary. Distributionally robust reinforcement learning. arXiv preprint arXiv:
2019. (Cited on page 43)
Aivar Sootla, Alexander Cowen- Taher Jafferjee, Ziyan Wang, David H Mguni, Jun Wang, and Haitham Ammar. Sauté rl: Almost surely safe reinforcement learning using state augmentation. In International Conference on Machine Learning. PMLR, 2022. (Cited on page 39)
Krishnan Srinivasan, Benjamin Eysenbach, Sehoon Ha, Jie Tan, and Chelsea Finn. Learning to be safe: Deep rl with safety critic. arXiv preprint 2010. 14603, 2020. (Cited on page 43)
Yanan Sui, Alkis Gotovos, Joel Burdick, and Andreas Krause. Safe exploration for optimization with gaussian processes. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Lille, France, 2015.
PMLR. (Cited on page 3)
Zhongchang Sun, Sihong He, Fei Miao, and Shaofeng Zou. Constrained reinforcement learning under model mismatch. arXiv preprint arXiv:2405.01327) 2024. (Cited on page 3)
Shyam Sundhar Ramesh, Pier Giuseppe Sessa, Yifan Hu, Andreas Krause, and Ilija Bogunovic. Distributionally robust model-based reinforcement learning with large state spaces. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research. PMLR, 2024. (Cited on page 43)
Mohammad Sadegh Talebi and Odalric-Ambrym Maillard. Variance-aware regret bounds for undis- counted reinforcement learning in mdps. In Firdaus Janoos, Mehryar Mohri, and Karthik Sridharan, editors, Proceedings of Algorithmic Learning Theory, Proceedings of Machine Learning Research. PMLR, 2018. (Cited on page 30)
16

---

Aviv Tamar, Shie Mannor, and Huan Xu. Scaling up robust MDPs using function approximation. In International conference on machine learning. PMLR, 2014. (Cited on page 43)
Takumi Tanabe, Rei Sato, Kazuto Fukuchi, Jun Sakuma, and Youhei Akimoto. Max-min off-policy actor-critic method focusing worst-case robustness model misspecification. Advances in Neural Information Processing Systems, 2022. (Cited on page 43)
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applica- tions in continuous control, 2019. (Cited on page 43)
Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery rl: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, 2021. (Cited on page 43)
Garrett Thomas, Yuping Luo, and Tengyu Ma. Safe reinforcement learning by imagining the near future. Advances in Neural Information Processing Systems, 2021. (Cited page 43)
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world, 2017. (Cited on pages and 43)
Roberto Visentin, Chiara Dalla Man, Boris Kovatchev, and Claudio Cobelli. The university of virginia/padova type I diabetes simulator matches the glucose traces of clinical trial. Diabetes technology & therapeutics, 2014. (Cited on page 1)
Kaixin Wang, Uri Gadot, Navdeep Kumar, Kfir Levy, and Shie Mannor. Bring your (non-robust) algorithm to solve robust mdps by estimating the worst kernel. arXiv preprint WxXv:23306.0589. 2023. (Cited page 43)
Eric M Wolff, Ufuk Topcu, and Richard M Murray. Robust control of uncertain Markov decision processes with temporal logic specifications. In 2012 IEEE 51st IEEE Conference on Decision and Control (CDC), IEEE, 2012. (Cited on page 43)
Huan Xu and Shie Mannor. Distributionally robust Markov decision processes. Mathematics of Operations Research, 2012. (Cited on page 43)
Tengyu Xu, Yingbin Liang, and Guanghui Lan. Crpo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference Machine Learning. PMLR, 2021. (Cited on pages 6, 9, 41, and 43)
Long Yang. Jiaming Ji, Juntao Dai, Linrui Zhang, Binbin Zhou, Pengfei Li, Yaodong Yang, and Gang Pan. Constrained update projection approach to safe policy optimization. Advances in Neural Information Processing Systems, 2022. (Cited page 43)
Tsung-Yer Yang, Justinian Rosca, Karthik Narasimhan, and Peter J. Ramadge. Projection- -based constrained policy optimization. In International Conference on Learning Representations, 2020. (Cited on page 43)
Denis Yarats, Ilya Kostrikov, and Rob Fergus, Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International conference learning representations, 2021. (Cited on page 9)
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In NeurIPS, 2020. (Cited on page 43)
Kevin Zakka, Baruch Tabanpour, Qiayuan Liao, Mustafa Haiderbhai, Samuel Holt, Jing Yuan Luo, Arthur Allshire, Erik Frey, Koushil Sreenath, Lueder A. Kahrs, Carmelo Sferrazza, Yuval Tassa, and Pieter Abbeel. Mujoco playground, 2025. (Cited pages 4, 37, and 40)
Moritz Zanger, Karam Daaboul, and J. Marius Zöllner. Safe continuous control with constrained model-based policy optimization, 2021. (Cited page 43)
17

---

Runyu Zhang, Yang Hu, and Na Li. Soft robust mdps and risk-sensitive mdps: Equivalence, policy gradient, and sample complexity. arXiv preprint arXiv:2306. 16..2,666..1,,, 2023. (Cited page 43)
Zhengfei Zhang, Kishan Panaganti, Laixi Shi, Yanan Sui, Adam Wierman, and Yisong Yue. Dis- tributionally robust constrained reinforcement learning under strong duality, arXiv preprint 111V:2400.115788, 2024. (Cited on pages and 3)
Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforce- ment learning for robotics: survey. In 2020 IEEE symposium series computational intelligence (SSCI). IEEE, 2020. (Cited on page 43)
Taiyu Zhu, Kezhi Li, Lei Kuang, Pau Herrero, and Pantelis Georgiou. An insulin bolus advisor for type I diabetes using deep reinforcement learning. Sensors, 2020. (Cited on pages and 23)
18

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
In the abstract, we claim to have developed a novel algorithm with theoretical safety guarantees and a comprehensive empirical study of it. Our theory can be found in Section 4, with proofs in Appendices B and C. In addition, we claim to have conducted experiments on several simulated benchmarks and on two real-world robotic platforms. These results can be found in Section 5 and Appendices D and F to
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We explicitly detail the limitations of our work, including possible venues for how they can be addressed by future work in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess..
19

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Our main theoretical results are presented in Section 4. Formal assumptions required for these results are stated explicitly in Appendices B and C. The proofs for our main theoretical result (Lemma 4.1 and Theorem 4.2) are presented in Appendix B, Additionally, further justify practical implementation in Appendix C.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the open-source code used run our experiments as well as exhaustive details on our experimental setup in Appendices G, H and to L. The physical trivial reproduce, since they rely
remote control car experiments are unfortunately on the licensed software for the motion capture system. However, the training code for this environment open-sourced and the experimental setup itself is fairly simple. For the Unitree Go1 robot, aside from providing the code used for the experiments, we provide the policies we deployed on the real platform in https: / //anonymous. 4open. science/r/ Prfe-sim2reall1Be
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same
20

---

dataset, provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: A URL with anonymized open-source code is provided in https: / //anonymous 4open. wwt.44.ppeeenccee. www.sscceenesseeeeeeeeeeeeeeeeee ssaannceee///eeeeeeeeeeeeeeeeeee simsreall 1EAC. The link includes specific in- stallation instructions. Furthermore, for the Unitree Gol experiments, provide the policies used on hardware onnx format, essentially allowing anyone with this robot to evaluate the policies used in our experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.ce/ coollc/guides///e how.eeeeeeoooeeeeeeiiiiiiiiiioi Policyy)) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: htttlss.cc/pubbccuuuucce wwwpiiicgggie/eeeeeeeeeeeeeeeeee http://///giieee/eeeeubbbiiiiiii wetteubbmisiiiiiiooio sww.iiosisiipiinonoooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
21

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All training details for the experiments are provided in Appendices G, H and J I..
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We explicitly mention in Section 5 that we report the mean and standard eπror in all of our reported results. Our hardware experiments are evaluated across several seeds and trials for each experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
22

---

Answer: [Yes]
Justification:
We explicitly mention in Section the compute used in all of our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https httppp:////eurppee/ htt///eurripp.e.
Answer: [Yes]
Justification: Ethics".
The research conducted in this work fully follows the "NeurIPS Code of
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our algorithm SPIDR, aims make RL agents safe and trustworthy, potentially unlocking many domains in which RL can be deployed reliably (e.g. in healthcare, Fox et al., 2020; Zhu et al., 2020) and improve our lives. While malicious usage cannot be strictly excluded, we believe that if SPIDR is used bona fide, its overall societal impact will be net positive. We discuss this in furthre details in Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
23

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have explicitly referenced all works upon which this research is based; for example, we have credited the authors of MuloCo appropriately throughout the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi tecoiehoooe... com dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our open-source code includes a README file with specific instructions for installation and usage.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: Our work does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: We used LLMs only for language editing purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https httpp::///uurrss.coo http://p//.///oeeeeee/////////// for what should or should be described.
26

---

Appendix
A Failure modes of Domain Randomization
The following example illustrates how domain randomization can fail to ensure constraint satisfaction in practice, even if all the simulator environments are arbitrarily close to the real environment.
Example A.1. Suppose S 2 3, A 2 2, then for all 0 < e S 1/4, there exists set of simulated CMDPs (M, (S, A. Ti C, 7, p) an unknown real CMDP M = AS,,A,P,,,,,,,, satisfying
(w Pg. . (s.) a) S (I.
with fixed budget d > 0 for all CMDPs, such that the domain randomization policy aDn returned by Equation (2) with any μ. will be unsafe the real environment,
Cp. (TDR) > d.
Proof. Consider the CMDPs given in Figure 10, where there are three states, the initial state su. two absorbing states $1 and $2, two actions a₁ and ag. The state space is in E, and assume so 0. $1 = and 82 = 2. The reward and cost function are given by:
r(sa a) = 0, c(so; a) Va A: r(81, a) = 1, c(s), a) = 1, Va E A: r(82, a) = c(s2, a) Va A: In the i-th simulated environment, at the initial state so. the transition probability is given by:
Pa. (50 so, a) = 0, Va E A,
Pe. (81 8o; a) +el fa 01), AaEA,
Pe. (s2 So, a) = =l fa = a1)) AaEA.
In the real environment, at the initial state so, the transition probability is given by:
p (su sp, a) = 0, Va E A,
p* (s1 su, a) +E(1 + 1 fa a1)), Y.EA.
p² (82 So,
-(1 + 1 {a = a1)), AaEA.
For all these environments, the absorbing states $1 and $2, have p(s2 a) p(s) 81.0) = for Va E A. Then the maximum Wasserstein distance can be bounded by epsilon, i.c. Dw (p' (s, e, a) E S x A for every simulated environment. Take the budget to be d = 1 +)).
To achieve the highest cumulative reward, the optimal domain randomization policies returned by Equation (2) is supported only on ai for any training distribution p. but the cost incurred by these policies in the real environment is + 2s) 2 > d. violating the constraint in the real environment.
We see that policy trained via standard domain randomization in the example Example A.1 can still be unsafe in the real environment. The only safe policy in the real environment is π (Su) ag, which cannot be learned with pure domain randomization in this example. This unsafe transfer is not only due to the averaging cost constraint, but also due to inherent mismatch between any simulated and real dynamics. This mismatch may lead the learned policy to frequently visit regions with underestimated cost in simulation, but which incur high actual cost in the real world.
27

---

sim-to-real deployment
Simulated Dynamics
Real Dynamies
0.
0. 1/4 e(1 (a a
+ a
z a1 an) : al (a=
r= 0. c=
Figure 10: Pathological unsafe transfer. Light grey states denote the initial state sy- Constraints prone to be violated, under small mismatch between the simulated and the real system dynamics.
B Proofs
B.1 Preliminaries
We first present well-established lemma, the proof of which can be found in many works, e.g. Agarwal et al. (2021, Lemma 1.16).
Lemma B.1 (Telescoping lemma). Given a policy n. for different dynamics and q, let g (8, a) & ( (P.=(("), where f is used to overload the notation for the constraint
Begagl-ls.a) (8') (p''-(--ss.aa) fF. function Vo. Then, we have
7 "
- = 1- 11s
Starting from this point, assume the following Lipsehitz continuity condition holds throughout the remainder this section:
Assumption B.2 (Continuity). For any E II,, the state cost value V/r*(s) and state reward value (*) are T-c- and LyyLLipschizz continuous s E S w.r.t. the 1-norm respectively, over both M and Me for all & E.
Assumption B.3 (Finite Discrepancy). We assume that the worst-case model discrepancy between the real and all simulated environments for all (s,a) E S x A is finite, i.e.,
max Dw (fc, < ESx EE
B.2 Proof of Lemma 4.1
Proof. Given policy и,, for dynamics p* and pç. let o (e V2' (s') Ex'rp" (- Vp (s"). Let ||-1 denote the Lipschitz constant a function. Under the assumption that the state cost function is Lipschitz, by the Kantorovich-Rubinstenn representation (Dudley, 2002,
Section 11.8) for L Wasserstein distance, we have:
J S Lo
(()))))))))) E. [e) I1AllapsS1
sup
< Le Dw (fg, (s,
By Lemma B.I, we have
(T) Ee- Cp. (T) S E Ee (T) - Cpc((T)
& Bg BeNu
E Bloa))dng I0p c(,,
(Lemma B.1)
LLC
Benu
B(osa))dpe: (D ((ssa)))))
(Equation (7))
(1 7)
This ends the proof of Lemma 4.1.
28

---

B.3 Proof of Theorem 4.2
Proof. We first prove that (Cpp (T)] 2 Cp- (T) for any policy E IT'. By definition of the
penalized cost Ope (T) and using Lemma 4.1, we have
Es (T) -p- Cp-(TT)
= Be-pl (Cpp (T) - Cp-(TT)
7Lc
+ Ban
((saa)dde
max Dw (pe, (e)e:"")ss,, (s,
eeE
2 7 1lc
Egnu
B(o,a)ddp a DDw (=s,
(1 7)
C
+ (1 7)
Egaugs
B(saa)ds
max Dw (pe: (s.
(Lemma 4.1)
e'E
2
(8)
Where in the last inequality we used that Dw (pe (p): (s,) I maxere Dw (pe, (s, Next, since Eę~u (Cpg (+) S d, and with Equation (Lemma 4.1) directly gives us:
Cp- S BeNu (C) s d.
This ends the proof of Theorem 4.2.
C Designing v v(s a)
In this section we provide additional theoretical intuition for the variance estimator introduced in Equation (6). Next, we show that, under that under the assumptions stated below, v(:.-) upper- the true sim-to-real discrepancy, measured by the Liaasseesteinn distance, with high probability, using data collected simulation. Importantly, when v(::) :) is indeed an upper bound for the true model discrepancy, SPIDR is provably guaranteed to be safe, as described above in Appendix B.
C.1 Modeling Assumptions
Let d(.,.) be the Euclidean 2-norm on S, and let py A Eg [Pe] denote the domain domaain-randomizaaioo kernel.
Definition C.1 Wassesssnn distance (Givens and Shortt, 1984)), Given a o-algebra F, for any two probability measures P1, P2 E F). the Wasssssernni distance between them is defined as:
/ d(si, (d87 x d82),
Dw (Pi P2)
inf rET(F, sxs
where r(P,, P2) E x F x 7(A x 8) = (A), x A) = Pa(A), E F} is the set of all couplings of P1 and P2, and d(. is a x F-measurable metric defined on E. In our setting, we assume this metric is given by 2-norm
112.
Assumption C.2 (Mild sim-to-real gap). There exists a constant & > 0, such that for any (s, a) x A, KL(p", (p)) I e and maxçe= (L(p E.
This assumption states that the real dynamics are close to the comain-randomization kernel and that the tomaan-randomizztion support using the domain knowledge of the real system; the uniform KI. radius formalizes this intuition. Additionally, assuming that KT.(p", ,,,)", S ensures that the simulator provides a reasonably accurate approximation of the real system. While the Wasserstein distance Dw serves the primary metric for model mismatch in our analysis, we use the KL divergence as it yields tighter concentration bounds, which we use below.
not overly large. practice, practitioners design =
Assumption C.3 (Bounded state space). The state space compact with diameter d..
This is fairly mild assumption in many physical problems, common for example in robotics. We next assume the sample average distance the average next state in Algorithm is uniformly lower bounded.
29

---

Assumption C.4 (Non- -enedeegee variance). Let be n i..d. draws from Py(- a. and $,,a - - 8; their empirical average. Furthermore, let (s s,a) Σ 1[s si] denote the empirical measure associated with the sample set. We assume the expected distance between 8; and $5,a is uniformly lower bounded by constant C3. i.e.,
E B a.0) [d(s', )s,a)] 2 ca, ((s,a) x
the next-state distribution collapses to point (C3 00) the empirical variance trivially underestimates any model mismatch. Requiring minimal spread rules out this degenerate case. In addition, many real systems are not fully deterministic and exhibit process and sensor noise. Hence a uniform lower bound 03 is both mild and practically satisfiable.
C.2 Auxiliary Lemmas
The following lemmas used in our analysis below.
Lemma C,5 (Bernstein transportation (Talebi and Maillard, 2018)), L.t p. E Es, where Eg denotes the probability simplex of dimension S 1. For all >> 0, for all functions f defined on S with 0 S f(k) & b. for all E S. KL q) S a then
lpf - all S 2Var,((f) + be,
3
where we use the expectation operator defined as pf & Esnpf(s) and the variance operator defined [n -n
as Var, (f) 2 (f)
pps -pf}²
Lemma C.6 (Variance bound for change of measure (Ménard et al., 2021)), Letp, Es and f is function defined on S such that 0 S f()) < b for all 8 E S. KT 4) & B then
Varg(f) & (f) + 4b2 and Var,(f) - 2ar q (f) - 482
Lemma C.7 (Jonsson al. (2020, Proposition 1)), For all p E Em and for all a [0,
Vn EN nKT.(fn. & log ( + (m - 1)log (e (c(1+
m - 1) 2 m=1)))
n
a.
C.3 High-probability Wasserstein bound
We ready to state the theoretical guarantee for v(:--), showing that upper-bounds the worst-case Wasserssn distance between the simulated and real dynamics.
Theorem C.8. Under Assumptions C.2-C.4, given confidence level (0, for any (s, a) x A. there exist constants C1, C2, Ca, C4 > 0 that contain only S, A, a, de E and log factor of n, such that with probability least O; have:
C +
C4
max Dw (py, (Pg. (s, s Civ(s,a) + C2E CEE
The leading term Ciu(s, a) captures how the local spread of simulated transitions bounds the worst- case Wasserstein gap. The term reflects irreducible model mismatch, and the remaining terms standard finite-sample corrections. The key insight lies in the first term, which uses the variance in algorithm. We note that while the design of v(..)) is necessarily the tightest possible, is easy to implement, making it more widely applicable. Our proof for Theorem C.8 is stated below.
Proof. We first establish an upper bound on Dw (pe.p")( (s, a) using the 2-norm. Specifically, for each (s, a), consider the coupling between pe(: 8, a) and p²(- s, a), defined by:
(8),82)s.a
(*( 8, a) (#2 8, a).
By the definition of the L1 Wasserstein distance given in Equation (9), we have:
Dw (w(ps,p')ss, < E Espro (d(() s2)]
(10)
30

---

Recall that 3,,a -i= where s, are i..d. drawn from Pa(: 8, s, By the triangle inequality of the 2-norm, have:
d(s1, $2) - d(s1, Sc,a) + d(22, 3,,0)
Substituting this into Equation (10) yields:
Dw (pe, (p,, S E (1)a)) ss,a)] (1))) s,,a)l.
(11)
Since Pg and p" are unknown, approximate them using Pa via Lemma C.5. Under Assumption C.2 and Assumption C.3, we obtain:
2
Bas-pe(thaa 3s,a)
ss,a)l I Es Enrnn ss,a)l + 2ar,, (d(s), Ba,a))E d E,
E Pe2pp"(1sa) [(22, ss,a)l I E Begr [(82 s.aa) + /2Varp, (d(s2, 8 8,,a)) +
d
3
Substituting into Equation (1I) gives:
4 Dw ("), < 2E, - (lba))d(ss 33,a)l 2 (d(3'3aa,)) + t
(12)
+3
Recall that (s' a, -i 1 denote the empirical measure. By Lemma C.7, we have w.p. a, for any (s, a), the following inequality holds:
log ( + +S - 1ol (e (e(I+s")
5-1 n
KL(Pp. <
Define g(S, A, n, a)
log (o)
(e (1 ((+gmm) )) Applying Lemma C.5 and Lemma C.6
the terms appearing in Equation (12) gives:
g(8, A,
E Ex Id(s' 8s,a)] S ( (d(s,, Sxa)] Varr,, (((5),8,,))
n
+ 2d, a,)
(Lemma C.5)
3n
(Rgg(S,Ann,) 2d, A, n, a.
$ E. B. sz,all
n
3n (Var, S [p(
and
/Var,, (d(s',8a,a)) S 2 Vara, (d(s', Br,a))e + 2033 g(S, A, n. a)
(Lemma C.6)
n
<2 /arp,, (d(s', a),e))e + 2 203 g(S, g(S, A,
(Vz + + S /x vv)
31

---

Substituting these into Equation (12) yields that with probability at least 1 a, for any (s, a) and any & E E, the following inequality holds:
Dw(pe,P p' (s) & / (a (d(s ss,a)) + 2/ (S A,n,a)
4d.g(S, A,
3n
+ 4v 2dds A,S,A,n,))
+ 1 Varp, 3 3s, (d(s',3,,))))
2 E (1 {saa)) + 21 2d A,n, a))
Ad,g(S, a,,
n2
3n
+ 4v Varp, (d(s', 33,))) + 4v 2dp A,,,A,
+ 2 dg E
3
(Assumption C.4)
(d A,
4d,g(S, A,
E. EE A ( 1s) 2
C3
3n
pp (d(s ((,5', + /1 2d2e g(S, A,
4 (jd, + 2c²)
+
Varg,
63
n
(Young's 's inequality)
2
22 A,
4d.g(S, A, a)
u(s, a) +
3n
4 3 9(S, (n,,,n,a)
+ 41 2d.
+ +203) -E
(13)
n
where the last equality utilizes that B[ [([[]) + Varlf) ff and the fact that
dim(()
(( =
Var (81.j....,.n.) = [d(s', ss,a))
[d(s',
This ends the proof.
32

---

D Additional Sim-to-Sim Ablations
We conduct further empirical analysis of SPIDR, focusing on the following key aspects: (1) sensitivity to the penalty parameter A, (ii) further analysis of standard domain randomization, (iii) empirical validation Lemma 4.1 and, (iv) behavior of the uncertainty approximation v(s, a) Through these ablations, highlight SPiDR's robustness the choice of X, its efficiency in scaling larger domain ensembles, and its principled handling uncertainty in challenging state- aate-action regions.
How robust SPIDR to the choice of X? We study the performance of SPIDR across varying values of Ả and under different magnitudes of distribution shifts in the CartpoleBalance task. To this end, we vary the magnitude of the actuator's gear parameter, denoted as |E], with slight abuse of notation. We ablate À E {0.85, 0.95, 0.75, 0.6, 0.5, 0,25, 0.35, 0.1, O} and | across {350, 400, 500, 500). We present our results in Figure 11, where we compare the objective and constraint for different values. Notably, for À = 0.6, while some performance is sacrificed in the objective, depending on the magnitude of | the constraint is satisfied across all settings of |=].
350
400
450
500
550
400
800
300
Constraint satisfied
((
600
Good performance
Budget
400
0.2
0.4
0.6
0.8
0.2
0.4
0.G
0.8
Figure 11: SPIDR's 's performance under different values of motor gear parameter. One choice of X consistently maintains constraint satisfaction.
Can domain randomization still be safe if enlarge the set E? We study the dependence of standard domain randomization on the size of the set E, presenting an example where even significantly increasing this set, and therefore the "diversity" of training environments, fails enable safe transfer. To this end, we use the simulated RaceCar environment and vary the lower and upper limits of the car's throttle parameter. This parameter controls how fast can the car drive, directly relating to its ability to operate safely upon deployment. Specifically, we use the following values {00.4, 0.6), (0.3, 0.7), (0.2, 0.8), (0.1, 0.9)). where each tuple defines the minimum and maximum throttle values. The default range used in our experiments (including sim mon-to-real experiments) is (0.4. 0.6). As in the previous experiment, we slightly abuse notation and refer the diameter of E as [3) computed as the difference between the upper and lower bounds. We report our results in Figure 12, demonstrating that for this task, safety upon transfer is not maintained, even when using
30
180
25
160
f(
Optimum on MM
Domain Randomization
15
140
10
120
Budget
E
Figure 12: Constraint and objective of domain randomization when varying the size of the car's throttle parameter. Constraints are violated even when increasing the range of paramteres.
wide range of parameters with domain randomization. More generally, argue that for some
33

---

problems, simply enlarging E\t could generally improve safety, however when safety must be ensured, pessimism is necessary.
Can we quantify constraint underestimation? We investigate the degree of constraint underesti- mation from Lemma 4. To this end, we measure the difference in the constraint measured in the collect empirical estimates of C.(T)) and E Cig (a) and report C. Ce (#) for SPiDR and standard domain ran- domization in Figure 13. As shown, in all tasks, compared to SPIDR, standard domain randomization underestimates the cost, leading to unsafe behavior on the test tasks (cf. Figure 6).
training environment against the evaluation environment. Concretely,
SPIDR
Domain Randomization
RaceCar
PointGoal2
CartpoleSwingup
Humanoid
QuadrupedRun
WalkerWalk
10
20
25
100
Gap
50
0
x107
x10
x 10
x10
x x10
x10
Simulation Steps
Figure 13: Constraint performance gap between training and test (lower is better). Domain random- ization underestinates the constraint on the test tasks.
Does vs,, a) upper-bounds the discrepancy in practice? We continue our previous study and demonstrate that (s,,))-d,,-- [Au a)] - (T) Cpe (x)) Z O, namely, that estimate -(.-)) can be used to sufficiently penalize the cost. Figure 14 we report this error across all six simulated tasks. As shown, in all tasks, our approximation error positive, suggesting that Xu indeed
RaceCar
PointGoal2
CartpoleSwingup
HumanoidWalk
QuadrupedRun
WalkerWalk
20
Error
100
2
100
x100
x10
x10
x10
x10
x10
Simulation Steps
Figure 14: Our estimator u(s,a) a) upper bounds the model discrepancy.
upper-bounds the gap in constraint. Notably, in the HumanoidWalk environment, the error of Au a) the constraint gap is close zero. This result is line with the performance-safety tradeoff shown in Figure 7, were SPiDR is both safe and achieving strong performance compared to the best solution on M
How does vss a) vary across states? We analyze v(s, a) measured across different states and actions the CartpoleSwingup environment. The state space consists of the linear cart position and velocity, along with the angular position (0) and angular velocity (6) of the pole. The continuous action space lies in []. and due to the symmetry of the system,
restrict our analysis to a E [0, 1]. Figure 15 visualizes the uncertainty over the angular position and velocity dimensions for representative actions (0.0, 0.3, 0.7, 1.0). We note the following observations: (i) uncertainty generally increases with action magnitude, and, (ii) for a o O, uncertainty peaks when the pole is the upright position (0 = π), which corresponds to inherently unstable equilibrium. This observation aligns with intuition: near the unstable upright position, small perturbations can lead to significantly different outcomes, making these regions harder model. From a safety perspective, this analysis motivates penalizing high-uncertainty regions during policy learning, particularly when deploying real systems where model inaccuracies may have significant consequences.
34

---

0.0
0. = 0.3
= 0.7
= 1.0
3 10
Uncertainty
10
-5
1 10
-7 X -X 0
Figure 15: Uncertainty v(ss, a) across angular states for actions a E {0.0, 0.3, 0.7, 1.7,1.00 1.0). Uncertainty increases with action magnitude and is highest near the upright position T)) highlighting states where the simulator's 's predictions are less reliable.
E Picking À in Practice
We demonstrate ablations on À when deploying SPIDR in real, providing guidance how to pick effectively in practice. We describe below the procedure we use our sim-to-real experiments for selecting A.
Step 1: Estimate the magnitude of v(-;' +). Evaluate and record .)) in training, aggregated over state-action pairs and across time. This estimate is not required be precise, its purpose to determine the general order of magnitude. This can be done purely in simulation.
Step 2: Select an appropriate range for A. Choose a candidate range of À values such that 1/- R 1. This heuristic ensures that the penalty term has roughly the same magnitude cost function, assuming Cmax RJ 1.
Step 3: Iteratively refine A. On the real system, begin testing with the largest value in the selected range and iteratively decrease it until the constraint satisfaction the real system closely matches the desired budget.
In Figure 16 we report the performance of SPIDR across different values of Importantly, for
RaceCar
Unitree Gol
x 10:01
26.12
25
Budget
Budget
Constraint
in simulation
40
0.0
0.025
0.05
0.075
0.1
No Pessimism
No Pessimism
Safety performance under different À values in real-world robotic tasks. The blue horizontal line represents the value of the constraint measured in simulation under = 0. SPIDR satisfics the constraints both of the physical systems.
Figure 16:
large enough initial choice of X, which can be obtained in practice using prior domain knowledge (e.g. degree of of simulator fidelity), the very first deployment of SPIDR is safe. In Step 3, online data used improve performance, while maintaining safety. With this procedure, safety is maintained zero-shot, while performance can be improved with online rollouts, in line with our safety guarantee in Section 4.3.
F Learning Curves for Simulated Environments
In Figure we provide the full learning curves of the experiment trials used for Section 5.2, including the standard error intervals across five random seeds.
35

---

SPIDR
RAMU
Domain Randomization
Nominal
PointGoal2
QuadrupedRun
Humanoid Walk
25
800
600
500
10
109
10
PointGoal2
QuadrupedRun
Humanoid Walk
1000
1000
75
500
Budget
Budgg
Budeg
2
x10
x10
x10"
CartpoleSwingup
RaceCar
WalkerWalk
100
500
300
0
2
2
10
10
X10
CartpoleSwingup
RaceCar
WalkerWalk
30
1000
750
(
500
500
250
10
Budget
Budget
Budget
0
2
1
2
x10
x10
x10
Simulation Steps
Figure 17: Learning curves used in Section 5.2. SPiDR consistently satisfies the constraints while maintaining good performance on the objective. Domain Randomization and Nominal fail to satisfy the constraints.
Simulating the sim-to-sim gap. We simulate the sim-to-sim gap as follows. In PointGoal2 and the RWRL tasks, follow similar approach Queeney and Benosman (2024) and introduce in evaluation an additional dynamics parameter (e.g.. mass or motor gains) that not encountered during training. In the RaceCar environment, the agent is trained using simplified bicycle model, but evaluated on more realistic variant that incorporates tire forces and detailed motor dynamics. Further details the tasks and their sim-to-sim gap design can be found in Appendices J L.
36

---

G Locomotion Experiments with Unitree Gol
We train policies using the FlatTerrainGol Joystick environment from MuJoCo Playground (Zakka et al., 2025). In this environment, the agent is tasked with following randomly sampled velocity commands in the forward, lateral, and yaw directions. Excessive joint motion can cause the legs to self-collide, even when joint limits are not reached. These collisions lead to falls or serious hardware damage.
Constraints. To prevent these joint limit violations we define cost function to measure the number of joint violations. The cost function is defined by the indicator function of any joint being outside the soft limit. More formally, the cost is defined as
{0
if any of E J such that > 0.75 ghax or < <.77 gfin otherwise
c(s, a) =
where I the set of joint indices, Oj the joint angle and ginn 93 are the angle limits of joint angle j. The soft factor limit of 0.75 is applied meaning that if a joint enters the outermost 25% of its feasible range of motion, the cost is set to 1. We choose this constraint since joint- position measurements are accurate and reliable on the real system, without relying indirect filtering/estimation methods.
Training in simulation. Each policy is trained for roughly one hour NVIDIA RTX4090 GPU. For both SAC and PPO, we train policies with primal-dual solver, using different values of À penalize the uncertainty. Each À value is trained across five different random seeds.
Command distribution. Target commands are uniformly sampled from the ranges [+0.45, +0.2, 1.3), corresponding to the forward velocity, lateral velocity, and yaw rate, respec- tively. Each sampled command is applied for fixed number of 1000 control steps. After this period, the command is reversed, perturbed with additive noise, and then reapplied for the same duration, after which new command is sampled from the same distribution stated above.
Real-world evaluation. We evaluate the trained policies on the Unitree Gol quadruped robot. To ensure that policies are evaluated under identical conditions, the same sequences of commands are sampled for each episode. Every policy is tested across 10 independent trials. We provide onnx files for the policies used in this experiment, as well the code to port simulation-trained policies to onnx format in the following link.
Comparison with RAMU. In Figure 18 we demonstrate a comparison of SPiDR and RAMU, con- ditioned on the same sequence of commands. We note that RAMU uses only single "nominal" environment, together with robust estimation value function estimation as means for robust transfer. We believe that RAMU fails follow the commands mainly because using only the nominal environ- ment, without incorporating domain randomization, might not be sufficient for robust sim-to-real transfer. We note that conceptually RAMU and domain randomization therefore RAMU could in principle transfer better if used with domain randomization. However, combining the two goes beyond the proposed algorithmic solution and official implementation of RAMU, and therefore not considered in this experiment. Despite that, we believe that combining the two approach is an interesting direction for future work.
mutually exclusive,
H RaceCar Experiments
In this task we implement the simulated RaceCar environment on real remote-controlled car, illustrated in Figure 2. Please see Appendix L. for further details about the reward and cost functions.
Real-world implementation. We measure the position and orientation of the car using a motion- capture system. Velocities estimated using a first-order low pass filter. These measurements are sufficient to recover the full state of the system as the goal and obstacle positions are fixed. All trajectory measurements start from the roughly same initial position in the world frame. The goal position is at the origin. Further details, regarding the parameters used for domain randomization and training hyper-parameters, can be found in our open-source implementation.
Additional results. In Figure 19 we provide the objective and constraint measured on the real system in this experiment. As shown, for X = 45, SPIDR able satisfy the constraint, while finding a performant policy.
37

---

RAMU
SPIDR
Figure 18: Comparison RAMU and SPIDR on the Unitree Gol robot. SPIDR follows the given commadns without falling.
55
65
100
60
40
20
Budget
Figure 19: Safety and objective performance for À E (0, 55, 60, 65] on the real system. We report the mean and standard eтror across five seeds. SPiDR transfers safely the real system while solving the task, i.c., reaching to the goal position.
Comparison with constraint tightening. We additionally compare SPIDR with a simple baseline that tightens the constraint budget dl in simulation. To this end, we evaluate "constraint tightening" on the real-world RaceCar, ablating it when using budgets d {0, 7.5] when training simulation. We present our results in Figure 20. As shown, while reducing the budget d in simulation indeed reduces the degree of accumulated costs on the real system, might still not be sufficient maintain safety, opposed to SPIDR. While this approach is even simpler than SPIDR, there are two main
SPIDR
7.5
300
[(m)
50
200
100
Rudget
Figure 20: Comparison of SPiDR with constraint tightening on the real RaceCar. Constraint tightening fails to satisfy the constraint.
challenges with it. First, severe tightening may yield "zero budget" training regime that CMDP solvers like CRPO and primal-dual methods struggle to solve in practice (Huang et al., 2023; He al., 2023). This behavior observerd in Figure 20; even though the budget is set d 0, the primal-dual CMDP solver we use fails to converge to constraint satisfying solution. Second, as hinted, reducing the budget equivalent to finding uniform upper- Equation (4), that does not depend on state-actions. Therefore, in order to achieve the same level of penalty required satisfy the constraint, this for "uniform pessimism" can degrade performance by being over- per-peessiistic in those states where the uncertainty about the sim-to-real gap might in fact be low, effectively penalizing the "wrong" states. We visualize such state-action sate-action-deee -deponnd uncertainty in Figure 15.
the penalty term in
38

---

I Additional Experiments with PPO
To further study SPIDR's performance with CMDP solvers that utilize policy gradients, we use PPO (Schulman et al., 2017). PPO is common choice used in many robotics tasks (for instance, see Lee et al., 2020). We use SauteRL with counter of the online accumulated cost, and penalizes the reward once this counter exceeds the budget. We ablate À E (0, 0.001, 0.01, } and report the performance on the CartpoleBalance task from RWRL in Figure 21. As shown, the constraints are violated for X 0, corresponding to
CMDP solver (Sootla et al., 2022). SuateRL augments the state
0.01
0.001
À
800
300
j(
600
200
400
Budget
200
Training Steps
100
Training Steps
x107
Figure 21: PPO with SauteRL as CMDP solver. Domain randomization fails to satisfy the constraints, while for À 2 0.01 constraints are satisfied with minor performance drop.
standard domain randomization. In contrast, the constraints are satisfied as À increases. We note that, while SauteRL designed for constraint satisfaction with high probability, instead of bounding the expectation of the cumulative costs, and thus in principle it adds additional conservatism, this might still not guarantee safe transfer under modeling mismatches. Figure demonstrates that SPIDR mitigates this issue.
J Safety Gym
To compare policies in environments from OpenAl Safety Gym by Ray et al. (2019), we port the PointGoal2 environment from standard MuJoCo MJX (MuJoCo XLA). MJX is a JAX-compiled MuloCo backend that is tightly integrated with Brax (Freeman et al., 2021). This port enables us to massively parallelize training by collecting trajectories from thousands of environments parallel on single GPU, accelerating training by several orders of magnitude compared to the original implementation. This contribution is independent interest to the safe RL community, Extending our work to additional Safety Gym environments is left for future work.
PointGoal2 environment. In this environment, the agent must navigate target location while avoiding hazards which include free-mi rre-mnovig vases and designated hazard zones. The environment is depicted in Figure 22. The initial positions of the agent, goal, vases, and hazards are randomized the beginning of training. The environment reward defined as the change in Euclidean distance to the goal between successive steps
re(s;, a.) A d - - 1dd S c).
where d, |x, Xgoal| is the Euclidean distance from the robot to the goal. The term [dd S e) an indicator function that gives a reward bonus when the agent reaches the goal, i.e., when is within 0.3 of the center of the goal. The goal position is resampled to another free position in the environment once reached.
cost of is incurred when the agent collides with velocity threshold (after collision), or when the agent inside a hazard zone h:
when of the crosses linear
Cr (st, az) A 1 B E collides(x,, x,)] 1 e V x 2 2 + e d. S pl.
where 7 = - and p = 0.2. Please see the implementation of Ray et al. (2019) for more specific details and open-source implementation.
39

---

Sim-to-sim gap. During evaluation, we uniformly sample three key parameters of the environment' s dynamics: multiplicative joint damping and mass factors and additive values for the actuator gear ratio. The precise training and evaluation ranges are given in Table 1. Note that the z-joint is hinge joint that allows the agent to rotate around the z-axis and the z-joint is slide joint that allows translation in the zyyplane.
Table 1:
Domain randomization parameters and ranges used during training and evaluation. and x denote additive and multiplicative terms respectively. By fixing the damping and mass parameters in train- ing, but not in evaluation, knowledge of 5", but also modeling mismatch due to imperfect simulation.
simulate both lack of
Parameter
Train
Eval
Damping (x, y)
1.0 (fixed)
[0.6, 1.0]
Damping (z) X
1.0 (fixed)
[0.7, 1.0]
Figure 22: Visualization of random initial- ization of the PointGoal2 environment. The red pointmass is the agent, the green transpar- ent cylinder is the goal, the cyan boxes are vases and the blue circles are hazard zones.
Gear (x) +
[-0.2, 0.2]
[0, 0.1]
Gear (z)
[-0.1, 0.1]
[0, 0.05]
Mass X
1.0 (fixed)
[1.0, 1.05]
K RWRL Benchmark
We evaluate SPiDR on four robotic tasks using the RWRL benchmark (Dulac-Amold et al., 2020), which adds safety constraints and distribution shifts to DeepMind Control suite tasks. We build on MuloCo Playground (Zakka et al., 2025), an MJX-based reimplementation that enables faster, parallelized training, by incorporating RWRL's modifications. See open-source implementation for details.
Figure 23: RWRL tasks.
Constraints. We use the joint position limits constraint for HumanoidWalk and QuadrupedRun. joint velocity limits for WalkerWalk, and slider position limits for CartpoleSwingup. These are the standard constraints proposed by Dulac-Amold al. (2020).
Sim-to-sim gap. We follow a similar experimental setup as Queeney and Benosman (2024) and introduce additional variability in the evaluation task to simulate modeling mismatches. In Table we provide the specific parameters we perturb for the RWRL environments.
L RaceCar Environment
Reward and cost.
The reward at timestep is given by
re (e at) A d d, + [d4 < c] >ellael12 >ellar ax-
where d, = ||xe - Xpnall: is the Euclidean distance the goal, and or E IR denotes the action applied at time (consisting of steering and throttle). The term (d; is an indicator function that gives a reward bonus when the agent within = 0.3 of the goal. The penalties X and X weight the control effort (magnitude of the action) and the change in action between consecutive timesteps,
40

---

Table Domain randomization parameters and ranges used during training and evaluation across tasks from RWLR. + and x denote additive and multiplicative terms respectively.
Parameter
Train
Eval
CartpoleSwingup
Pole Length
[0.0, 0.0)
[-0.25, 0.25)
Gear
[0.0, 5.0]
[0.0, 5.0]
QuadrupedRun
Torso Density +
[0.0, 0.0]
[-300.0, 300.0]
Friction +
[0.0, 0.0]
[-0.95, 0.0]
Lift Gear x
[0.75, 1.5]
[0.75, 1.5]
Yaw Gear
[1.0, 1.0] (fixed)
[0.5, 2.0]
Extend Gear X
[0.75, 1.5]
[0.75, 1.5]
HumanoidWalk
Friction
[0.0, 0.0]
[-0.05, 0.05]
Hip Gear (x) +
[-20., 20.]
[-20., 20.]
Hip Gear (y) +
[-20., 20.]
[-20., 20.]
Hip Gear (z) [-60., 60.]
[-60., 60.]
Knee Gear +
[0, 0]
[-40., 40.1
WalkerWalk
Torso Length
[0.0, 0.0] (fixed)
[-0.75, 0.75)
Gear +
[0.0, 20.0]
[0.0, 20.0]
respectively. The cost function at time t defined as
Ct 1 || - pill pt + 1sx $ VJ.
where H, E R2 is the agent's 's position, p; and p; are the position and radius of the i-th obstacle and E the kinetic energy of the car at time f, simulating plastic collision between the car and obstacles. This choice of cost function allows to penalize more severely collisions in which the car smashes into obstacles, as opposed to softly touching them. The second term penalizes the agent for leaving the valid area V, which corresponds bounded rectangular arena.
Sim-to-sim gap. In the previous sim-to-sim environments, we model the sim-to-sim gap by intro- ducing an auxiliary dynamics parameter (e.g., pendulum length) that is not observed during training. In contrast, in the RaceCar environment, the car dynamics in the training environments are governed by -inimminematic bicycle model that does not account for interactions between the tire and the ground. On the other hand, in evaluation, we use the dynamical bicycle model of Kabzan et al. (2020). We refer the reader to Kabzan al. (2020) for the detailed equations of motion as well as to our open-source implementation for more details.
M Additional Details on our CMDP Solvers
CRPO. CRPO (Xu et al., 2021) is CMDP solver that uses primal subgradient switching method to solve the constraint optimization problem. By doing so, CRPO is not tied to any specific RL algorithm and can be used with different policy search methods. For instance, Xu et al. (2021) use this technique with TRPO (Schulman al., 2015) and Queeney and Benosman (2024) use with MPO Abdolmaleki et al. (2018). In our experiments, mainly use CRPO together with SAC (Haarnoja
41

---

et al., 2019). To apply CRPO with SAC, we learn cost critic Qc(s, trained similarly to the reward critic but without the standard entropy term of SAC. To update the policy, we estimate the constraint with C, and it exceeds the budget d, we switch from a neward-maxiiinn update to cost-minimizing one. The procedure is detailed in Algorithm 2.
Algorithm 2 CRPO with SAC
1: Input: Constraint budget d, replay buffer DD
2: Initialize: Policy my, reward critic Q, cost critic Qe
3: for each environment interaction step do
4: Collect action (Tp- |
5: Observe reward r, cost next state s'
6: Store transition (s, a, 7, c, s') in D
7: end for
8: for each gradient update step do
9: Sample batch {(*;, di, Ct2 + from D
Update reward critic Q, using SAC critic loss
11: Update cost critic Q. using TD error (no entropy bonus)
12:
Estimate constraint Ć < N E Qc(si, mo(s))
>> Draw action from
13:
if C then
Constraint violated: prioritize cost minimization
Update policy тe using gradient of Ò
14:
else
Constraint satisfied: optimize for reward
16:
Update policy using SAC policy loss
end if
18:
end for
19:
return Policy R8
Primal-dual solvers. The primal-dual approach solves CMDPs by augmenting the standard loss used in PPO SAC with penalty term derived from the constraint using Lagrangian duality. Specifically, we define the Lagrangian as
[(0 App) = (TT) APp (C(TT)) d), -)),
where ApD 2 is the Lagrange multiplier. The goal to find saddle point of the Lagrangian by performing gradient descent on the policy parameters 0 (primal variables) while performing gradient ascent on Äpp (dual variable) to enforce the constraint.
When integrating this method with SAC, we learn a cost critic together with the reward critic. When computing the policy loss, we evaluate the policy in reward critic to evaluate the constraint. The loss is computed via Ta = E,, (log Ta + (s, x() + Xn (s, 7 #)). The same estimate Qe (8, is used update the dual
way the cost critic as do in the
A [X + 7A (C(mr) d))+
where V/y is the dual learning rate.
For methods that rely policy gradient estimators like PPO, constraint via a learned cost value function Vs(s). and compute advantage estimates accordingly. The policy update then follows the same principle. where the objective is penalized by the constraint estimate weighted by App ensuring that constraint violations are actively discouraged during training.
value-based estimate of the
More specific implementation details can be found in our open-source implementation.
42

---

N Additional Related Work
Safe reinforcement learning. A prominent line of work addressing safety in RL utilizes constrained Markov decision processes. In CMDPs, agents optimize cumulative rewards while satisfying constraints on cumulative costs (Altman, 1999). To solve CMDPs, two main algorithm categories are used: primal-dual and primal methods. Primal-dual approaches leverage the Lagrangian form, aiming to find a saddle-point of it. This is typically done by iteratively updating policy parameters and dual multipliers to balance rewards and constraints (Chow et al., 2018; Achiam et al., 2017; Yang et al., 2020, 2022). On the other hand, primal methods bypass dual variables entirely, focusing on directly optimizing the primal problem by embedding constraints through gradient combination (Xu et al.. 2021; Gu et al., 2024). Other works use the CMDP formulation but propose alternative methods to satisfy the safety constraints. Srinivasan et al. (2020) learn a safety actor-critic together with a behavior policy, Thananjeyan et al. (2021) use recovery backup policy, Bharadhwaj et al. (2020) propose a conservative safety critic together with online rejection sampling of actions. Further discussions on alternative formulations of safety in RL are provided by Garcia and Fernández (2015); Gu et al. (2022); Brunke et al. (2022). This work differs from standard CMDPs in that it focuses on addressing additional model mismatch within the CMDP framework.
Model uncertainty in reinforcement learning. Various works address model uncertainty or model mismatch of the environment (e.g., reward function, dynamics, or task itself) during train- and test-time. This challenge is particularly relevant in sim-to-real transfer, which typically serves as a natural motivation for these works. Within this context, a line of research applies robust optimization to optimize worst-case performance, known as robust RL (Iyengar, 2005; Xu and Mannor, 2012;
Wolff et al., 2012; Kaufman and Schaefer, 2013; Tamar et al., 2014; Pinto et al., 2017; Pattanaik et al., 2017; Ho et al., 2018; Tessler et al., 2019; Smirnova et al., 2019; Derman and Mannor, 2020;
Ho et al., 2021; Badrinath and Kalathil, 2021; Curi al., 2021: Tanabe et al., 2022: Goyal and Grand-Clement 2022: Ding et al., 2023; Wang et al., 2023; Sundhar Ramesh et al., 2024). Robust RL algorithms often employ a minimax formulation, significantly increasing design and implementation complexity. Similar to our approach, Gadot et al. (2024) propose an algorithm that avoids explicitly solving the minimax problem, instead relying entirely on how trajectories are sampled. Their key result demonstrates that by sampling next states pessimistically with respect to reward value function, one can derive a robust policy under a rectangular KL-divergence uncertainty set. In addition, Thomas et al. (2021); As et al. (2022); Zanger et al. (2021) 1) use ensembles and pessimism to enforce safety constraints under model uncertainty as done in SPIDR. This work differs in that our models are not learned from data but are derived via domain randomization. In addition, we formally show that by only using domain randomization, one can achieve safe transfer to the real system. Finally, similar to this work, Yu et al. (2020) propose to penalize rewards based model uncertainty. However, their focus is in unconstrained offline RL problems. Model uncertainty is also addressed in other formulations such risk tiskssensttive RL (Zhang al., 2023; Kim et al., 2023), domain adaptation in RL (Chen et al., 2024), and curriculum RL (Narvekar et al., 2020). Although these do not focus directly on safety, work can potentially be adapted to these settings.
Domain randomization. Domain randomization enhances policy robustness by training across a variety of environmental scenarios and optimizing average performance. It is widely used in robotics (Sadeghi and Levine, 2016; Tobin et al., 2017; Peng et al., 201 8; Andrychowicz et al., 2020). This work utilizes domain randomization as a key component for robustness, integrating it into the constrained RL framework by adding a robust penalty term to the cost function. Given the limited the- oretical analysis on domain randomization (Chen et al., 2022), we theoretically study it when facing with additional safety requirements, highlighting its limitations and addressing them. Further, similar to our work, Lee et al. (2023) and Kim et al. (2024) combine CMDPs with domain randomization. However, these works apply CMDPs primarily to legged locomotion, using constraints to shape stylis- tie gait qualities, rather than ensuring safety. Additionally, their evaluations are largely qualitative and assess constraint satisfaction in simulation. Lastly, comprehensive survey on domain randomization, and other methods for sim-t--raal transfer is discussed in more details by Zhao et al. (2020).
43


Reviews:
[{"review_id": "lmkFLr1ER0", "reviewer": "Reviewer_3yuV", "summary": "This paper presents SPiDR, a method for safe zero-shot sim-to-real transfer in reinforcement learning. Motivated by the safety risks from the sim-to-real gap, SPiDR combines domain randomization with a pessimistic cost penalty based on uncertainty from an ensemble of simulators. This penalization ensures policies avoid risky regions during training. Experiments on both simulated benchmarks and real robots (a race car and a quadruped) show that SPiDR achieves safe and effective transfer, outperforming standard domain randomization and robust RL baselines.", "questions": "How sensitive is SPiDR to the choice of the pessimism coefficient (λ), and how practical is it to tune this parameter in real-world applications?\nDoes SPiDR still ensure safety if the real environment lies significantly outside the range of the domain randomization distribution?\nWhat are the limitations of using ensemble variance as a proxy for model uncertainty? Are there alternative metrics that might work better?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- SPiDR provides provable zero-shot safety during sim-to-real transfer without relying on real-world data\n- SPiDR is theoretically sound\n- It is simple to implement and integrates easily with existing RL methods \n- It shows strong performance in both simulated benchmarks and real-world robot tasks\n\nWeaknesses:\n- The pessimistic penalty can lead to overly conservative policies, potentially reducing task performance.\n- SPiDR requires manual tuning of the pessimism parameter", "was_revised": true, "final_justification": "This work demonstrates a sim-to-real transfer method with robust RL with theoretically guarantees.\nThe rebuttal has addressed my concerns over how to tune the lambda parameter and scalability. I would like to maintain my original recommendation of accept.", "rebuttal": "Thank you for your thoughtful review and accurate summary of our paper! We appreciate your acknowledgment that SPiDR “shows strong performance in both simulated benchmarks and real-world robot tasks”. Please find below our response to some of your questions.\n\n> The pessimistic penalty can lead to overly conservative policies, potentially reducing task performance.\n\nWe fully agree with this statement. When the sim-to-real gap is not precisely known, we believe safety should be maintained first, and only then performance is ought to be improved by gradually reducing conservatism with online data. SPiDR allows practitioners to achieve the first step in this procedure.\n\n> how practical is it to tune this parameter in real-world applications?\n\nWe acknowledge that the choice of $\\lambda$ is crucial for the success of SPiDR. In addition, we would like to point out, that since $\\lambda$ has a physical interpretation (see Equation 4), domain knowledge about the problem, such as knowledge about the fidelity of the simulator and the cost function of the CMDP, can greatly simplify this process.\nFor instance, in both of our real-world experiments, we initially select $\\lambda$ by matching the scale of the penalty (in simulation) to be roughly the same scale of the cost function. The simulator of the Go1 robot is much more accurate compared to the RaceCar, and therefore it required lower values for $\\lambda$. Thank you for raising this important question! We will clarify this point in the paper by including the discussion above to Appendix E.\n\n> Does SPiDR still ensure safety if the real environment lies significantly outside the range of the domain randomization distribution?\n\nThank you for raising this question. Yes, this is exactly how we conduct our sim-to-sim experiments. The parameters used to perturb the environment in evaluation are different than those used in domain randomization during training. For instance, in the Walker task, we randomize only the motor gear parameters, while in evaluation we additionally also randomize the torso length. This is mentioned in detail in Appendices J-L, however we will clarify this point further in the main text of our revised version.\n\nWe greatly appreciate your feedback and we are happy to address any further questions or comments."}, {"review_id": "zIIaerfg5a", "reviewer": "Reviewer_HvGa", "summary": "This paper introduces SPIDR, a novel algorithm designed to address the challenge of zero-shot safe sim-to-real transfer in reinforcement learning. The authors state that standard domain randomization is scalable but typically unsafe, while formal robust methods tend to be too complex for practical use and scale. SPIDR addresses this gap by augmenting standard domain randomization with a pessimistic penalty. The idea behind this pessimism term is that uncertain transitions, i.e., transitions that result in a variety of successor states under an ensemble of randomized simulators, should be penalized so as to avoid exploitation of uncertain dynamics. By adding this state-dependent uncertainty measure as a negative bonus to the safety cost function, SPIDR encourages the learned policy to avoid regions where a variety of simulators do not result in the same successor state, thereby acting more conservateively and improving safety upon real-world deployment. The method is evaluated empirically on a variety of simulated benchmarks and, notably, on two distinct real-world robotic platforms.", "questions": "My main questions relate to the weaknesses I have outline above: \n\n- Could you clarify your definition of \"zero-shot\" in the context of this paper? And does it include the tuning process of $\\lambda$? A more explicit discussion of this definition in the main text would be valuable.\n- How would you expect SPIDR to perform against a simple baseline as the one I outlined above (i.e., vanilla DR where the cost budget itself is treated as a tunable hyperparameter that is allowed to be tuned online)? A discussion of state-dependent vs. uniform pessimism could also be insightful in general. \n- How would you expect SPIDR to perform in highly stochastic environments and are there ways to circumvent potential issues? See above for my reasoning.\n\nI am willing to raise my score if the mentioned points are addressed clearly in a revision.", "limitations": "Yes. However, a few points, e.g., the need for real-world trials to tune $\\lambda$ or stochastic transitions (if I understood correctly) may be addressed more centrally.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: \n\n- Significance: The paper addresses a problem of high importance to the field, the reliable deployment of simulated agents to real-world applications. Ensuring safety in sim-to-real transfer is a bottleneck for deploying RL in real-world applications, and a practical, scalable solution may be impactful for both researchers and practitioners. \n\n- Clarity: This paper is written clearly, and the approach is simple conceptually, indicating that it may be easy to integrate into existing pipelines. It modifies standard domain randomization setups in a relatively minimal way, which is by altering the cost signal. Relevant background is explained well and the derivations are straightforward to follow. \n\n- Quality: The empirical evaluation of the proposed evidence is strong, in particular due to the inclusion of real-world experiments with robotic sim-to-real settings. The experiments support the paper's claim, albeit more baseline methods or ablation experiments (see below) could have been run. I believe, however, that this is acceptable given that they are real-world experiments.  is exceptionally strong. \n- The theoretical grounding looks sound in principle. The practical algorithm however deviates from the theoretical grounding in a number of ways (see below). I believe fairly similar related bounds (albeit not exactly the same) have been proposed in the literature I believe. \n\nWeaknesses: \n\n- Originality: The algorithmic novelty seems somewhat limited to me. While the specific application to the model-free sim-to-real setting may be novel, the concept of using ensemble variance as an uncertainty penalty to ensure safety is a rather well-established pattern for example in the literature on safe model-based RL. I believe this line of research is discussed rather sparingly in the current form of the paper, for example several works derive highly related bounds and algorithmic components to the proposed ones that precede the cited work by Sun et al.:\n- Safe continuous control with constrained model-based policy optimization (Zanger et al., 2021) \n- Constrained Policy Optimization via Bayesian World Models (As et al. 2022)\n- Safe Reinforcement Learning by Imagining the Near Future (Thomas et al. 2021)\nthe paper would be strengthened by discussing and differentiating itself from these works in a bit more detail.\n\n- Parameter tuning: An algorithmic weakness in my view lies in the practical need to tune the pessimism parameter $\\lambda$. Parameters of this kind are generally known to be hard to tune, which warrants the use of Lagrangian methods and constrained solvers with access to online data many works in the literature. It is commendable that the authors suggest a procedure for finding this parameter (Appendix E), but from what I understand, this procedure requires tests on the online MDP. In my view, this procedure also contradicts the definition of \"zero-shot\" transfer since it seems likely that one requires at least a \"few shots\" to find an appropriate pessimism level. This distinction should be made more explicit, or (even better) be addressed by directly the inference of $\\lambda$ as an algorithmic component (e.g., by phrasing the problem as a POMDP). I do, however, realize that the latter would be a rather tall ask for a rebuttal. \n\n- Experimental baseline: The above point also affects the experimental evaluation. The comparison with the standard domain randomization is important but feels slightly imbalanced as is due to the presence of the additional tunable hyperparameter. An extremely simple baseline that would be granted a similar tuning pipeline for a pessimism parameter $\\lambda$ could use vanilla domain randomization but simply changes the admissable cost limit as a hyperparameter. \n\n- Stochastic environments: It seems that the current practical implementation of the algorithm would be very susceptible to stochastic transitions, since it replaces the wasserstein distance $D_W$ with variances. For highly stochastic environments, this should result in overly conservative agents if I am not wrong.", "was_revised": true, "final_justification": "The authors have provided an extensive rebuttal, including additional empirical studies, that addressed many of my initial concerns. Specifically, the authors addressed and clarified\n- concerns about the online tuning of the pessimism parameters and the implication on the nature of the \"zero-short\" property \n- differentiation to related works\n- additional empirical ablations / experiments to an additional intuitive baseline\n\nI believe the online tuning of pessimism still requires future investigation, but this is to be expected and would not be a reasonable requirement for this submission. I do believe the additional of models that could seperate aleatoric and epistemic uncertainty to deal with stochastic systems more fundamentally would be a sensible, method-specific addition to SPIDR. Overall, I find that enough questions / weaknesses have been addressed to raise my score. I thank the authors for their work and the extensive rebuttal.", "rebuttal": "We would like to thank you for carefully reviewing our paper and for your clear and constructive feedback! You can find below our response to your questions.\n\n> Quality: The empirical evaluation of the proposed evidence is strong, in particular due to the inclusion of real-world experiments with robotic sim-to-real settings. is exceptionally strong. Significance: The paper addresses a problem of high importance to the field, the reliable deployment of simulated agents to real-world applications.\n\nThank you again for acknowledging our work and for the recognition of our contributions.\n\n> Could you clarify your definition of \"zero-shot\" in the context of this paper? And does it include the tuning process of $\\lambda$? A more explicit discussion of this definition in the main text would be valuable.\n\nThank you for pointing this out. To be concrete, our definition of “zero-shot” relates only to _safety_. More specifically, under standard regularity assumptions, one can guarantee safe transfer on the first trial. We do not state however guarantees about the _performance_ upon transfer, precisely because attaining optimal performance may require additional online trials on the real system. \n\nIn appendix E, we provide an informal procedure that serves as a recommendation to practitioners. In this procedure, safe transfer is guaranteed on the first trial, while performance can be improved after additional post-simulation trials. We agree that this manual process should be automated, e.g. by running online safe Bayesian Optimization techniques [1, 2, 3]. A similar approach has been recently proposed in [4], in the context of safe offline learning and is highly relevant, yet complementary to SPiDR. We view an online procedure for the selection of $\\lambda$ as a highly valuable direction for future work.\n\nIn a broader sense, the choice of hyperparameters in the context of safe learning is a critical, yet an extremely challenging problem, which we do not aim to tackle. For example, in SafeOpt [1, 2], a well-established algorithm with probabilistic guarantees for constraint satisfaction during learning, the choice of GP kernel and Lipschitz constants are left to the practitioner. In a similar vain, $\\lambda$ is in fact derived from the Lipschitz constant of the constraint, and can be viewed intuitively as a prior over the sim-to-real gap. To the best of our knowledge, an initial attempt to address safe hyperparameters selection was done in [5, FrontierSearch], yet significant work still remains to be done. Based on your suggestion and the discussion above, we will clarify further in the main text of the paper, that zero-shot safety relates only to ensuring safety on the first trial, without guaranteeing optimality on the real system.\n\n> How would you expect SPIDR to perform against a simple baseline as the one I outlined above? A discussion of state-dependent vs. uniform pessimism could also be insightful in general.\n\nThank you for raising this important question. \nThere are two main challenges with tuning the budget (“constraint tightening”). First, severe tightening may yield a “zero budget” training regime that CMDP solvers like CRPO and primal-dual methods struggle to solve in practice [6, 7]. Second, as hinted, reducing the budget is equivalent to finding a uniform upper-bound to the penalty term in Equation 4, that does not depend on state-actions (“uniform pessimism”). Therefore, in order to achieve the same level of penalty required to satisfy the constraint, uniform pessimism can hurt performance by being over-pessimistic in those states where the uncertainty about the sim-to-real gap might in fact be low, effectively penalizing the “wrong” states. We visualize such state-action-dependent uncertainty in Figure 13.\nThank you for the great suggestion to include this discussion in the paper, we will add it to Appendix D.\n\nFollowing your suggestion, we have also conducted an additional ablation on the real-world race car system, repeating our experiment with five trials per seed across five seeds, testing constraint tightening with values $d \\in \\{0, 7.5\\}$ (where the standard budget is $d = 15$). We present below the mean and standard error across the five seeds (on the real system)\n\n|  | Cumulative Cost (± SE)    | Cumulative Reward (± SE)  |\n|----------|---------------------------|----------------------------|\n| d = 0        | 58.7074 ± 10.5819         | 14.1819 ± 3.3733           |\n| d = 7.5      | 141.1890 ± 29.7231        | 29.1217 ± 5.9963           |\n| SPiDR     | 10.7956 ± 1.2856          | 62.6296 ± 6.5359           |\n\nAs shown, in both settings, SPiDR outperforms constraint tightening both in terms of safety and performance. We note that for $d =0$, the constraint was not satisfied even in simulation, precisely as described in [6].\n\nIn total, we compare SPiDR _on hardware_ with three other baselines: RAMU (Appendix G), Constraint Tightening (will be added in Appendix H) and Domain Randomization (main text). This additional experiment strengthens our confidence in SPiDR's effectiveness in satisfying constraints upon dynamics mismatch, supporting our main contribution: providing a simple, yet effective algorithm for safe sim-to-real transfer.\n\n> How would you expect SPIDR to perform in highly stochastic environments and are there ways to circumvent potential issues? See above for my reasoning.\n\nIndeed, our implementation uses only one sample per dynamics model, which could add conservatism in highly stochastic environments. In principle, one would need to separate the aleatoric noise from uncertainty that originates from model ambiguity. This can be done by drawing several samples from each model and marginalizing the noise [8]. We opt for using a single sample as simplicity is an important design choice of SPiDR. Empirically, we observe that even on very stochastic environments, such as the Go1 (noise added to observations, commands are sampled randomly) and PointGoal2 (target positions are resampled during rollouts), SPiDR works well, outperforming other baselines and transferring safely to a real Go1 robot while maintaining competitive performance.\n\n> several works derive highly related bounds and algorithmic components...the paper would be strengthened by discussing and differentiating itself from these works in a bit more detail.\n\nWe agree with this statement; our theoretical argument relies at its core on the simulation lemma, which is indeed a standard tool for analyzing many other related problems. We will add a paragraph in the related works section discussing pessimism in the context of model-based CMDP solvers, differentiating more precisely our contribution relating to the important works mentioned above. Thank you for bringing this to our attention and helping us improve our paper!\n\nWe thank you once again for your feedback and hope our responses adequately address your concerns. We'd be happy if you would consider revising our score, and we are happy to answer any other open questions. Thanks again for your feedback and taking the time to help us improve our work.\n\n---\n\n[1] Berkenkamp, Felix, Andreas Krause, and Angela P. Schoellig. “Bayesian optimization with safety constraints: safe and automatic parameter tuning in robotics.” Machine learning 112, no. 10 (2023): 3713-3747.\n\n[2] Sui, Yanan, Alkis Gotovos, Joel Burdick, and Andreas Krause. “Safe exploration for optimization with Gaussian processes.” In International conference on machine learning, pp. 997-1005. PMLR, 2015.\n\n[3] Kaushik, Rituraj, Karol Arndt, and Ville Kyrki. “Safeapt: Safe simulation-to-real robot learning using diverse policies learned in simulation.” IEEE Robotics and Automation Letters 7, no. 3 (2022): 6838-6845.\n\n[4] Wachi, Akifumi, Kohei Miyaguchi, Takumi Tanabe, Rei Sato, and Youhei Akimoto. “A Provable Approach for End-to-End Safe Reinforcement Learning.” arXiv preprint arXiv:2505.21852 (2025).\n\n[5] Rothfuss, Jonas, Christopher Koenig, Alisa Rupenyan, and Andreas Krause. “Meta-learning priors for safe Bayesian optimization.” In Conference on robot learning, pp. 237-265. PMLR, 2023.\n\n[6] He, Tairan, Weiye Zhao, and Changliu Liu. “Autocost: Evolving intrinsic cost for zero-violation reinforcement learning.” In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 12, pp. 14847-14855. 2023.\n\n[7] Huang, Weidong, Jiaming Ji, Chunhe Xia, Borong Zhang, and Yaodong Yang. “Safedreamer: Safe reinforcement learning with world models.” arXiv preprint arXiv:2307.07176 (2023).\n\n[8] Depeweg, Stefan, Jose-Miguel Hernandez-Lobato, Finale Doshi-Velez, and Steffen Udluft. \"Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning.\" In International conference on machine learning, pp. 1184-1193. PMLR, 2018."}, {"review_id": "mXEu8tcBKU", "reviewer": "Reviewer_5h5Y", "summary": "This paper addresses the critical challenge of zero-shot safe sim-to-real transfer in reinforcement learning, where a policy trained entirely in simulation must satisfy safety constraints upon its first deployment in the real world. The proposed SPIDR builds on domain randomization by incorporating a pessimistic penalty into the safety constraint. Specifically, SPIDR approximates this uncertainty by measuring the variance of next-state predictions from an ensemble of simulators and adds this variance as a penalty to the cost function during training. This encourages the agent to avoid state-action regions where the simulator is unreliable, leading to more robustly safe policies.", "questions": "1) The paper's framework assumes that the dynamics $p$ differ between sim and real, does the difference of cost function $c$ is also considered? If not, the \"cost-to-real\" gap can also exist due to sensor noise, calibration errors, or differing physics of cost-inducing events (e.g., contacts). How robust do you expect SPIDR to be if the cost function itself is also mismatched? \n2) The success of SPIDR hinges on the pessimism parameter λ, and the tuning guide in Appendix E suggests an iterative process on the real system. This seems to conflict with the paper's \"zero-shot\" premise. Could you please elaborate on this process? Specifically, for the RaceCar and Unitree Go1 experiments, how many real-world rollouts or trials were required to select the final λ values reported in the paper?\n3) SPIDR increases the cost, which can potentially render the CMDP infeasible if λ is too high or the sim-to-real gap is too large. This could lead to the agent learning a trivial \"freezing\" policy to avoid all costs. Take an example that might not be very accurate: what if the $\\lambda v(s_t, a_t) > d$? Did you encounter this failure mode in your experiments?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1) The paper is very well-written, structured, and easy to understand. The motivation is laid out clearly, and the progression from the theoretical concept to the final algorithm design is logical and intuitive .\n2) The proposed method is well-founded in theory. It provides a formal upper bound on the real-world cost (Lemma 4.1) and a clear safety guarantee (Theorem 4.2) showing that policies trained with SPIDR will satisfy constraints in the real world.\n3) The deployment in real-world scenarios validates its ability of zero-shot transfer of sim2real.\n\nWeaknesses:\n1) The primary weakness lies in the practical requirement to tune the pessimism hyperparameter λ. The paper provides a heuristic for selecting λ in Appendix E, but this process involves iterative testing on the physical system to find a suitable balance between safety and performance. This need for real-world tuning somewhat contradicts the \"zero-shot\" claim and could be costly or impractical in many real-world scenarios. \n2) The effectiveness of the empirical variance as an estimator depends on the ensemble size n. For systems with very high-dimensional state spaces (e.g., from vision), a much larger ensemble might be required to get a reliable uncertainty estimate, which could impact computational performance. The paper's experiments are limited to robotics tasks with proprioceptive states, and it does not analyze how the method would scale to such high-dimensional problems.", "was_revised": true, "final_justification": "The authors provided a good rebuttal that addressed most of my initial concerns.\n\n* Concern on \"Zero-Shot\": My primary concern about the \"zero-shot\" claim was resolved. The authors' clarification that it refers to safety on first deployment (not optimal performance) is a crucial distinction.  This, combined with their confirmation that the initial real-world deployments were indeed safe without prior tuning, validates their central claim.\n\n* Concern on Scalability: The authors conducted a new vision-based experiment that demonstrates the method's applicability, turning a potential weakness into a strength.\n\nSince authors has provided extensive experiment and addressed most of my concerns, and I will raise my score.", "rebuttal": "Thank you for thoroughly reviewing our paper and for helping us improve it.\nPlease find below our detailed responses to the concerns raised. \n\n> The proposed method is well-founded in theory. It provides a formal upper bound on the real-world cost (Lemma 4.1) and a clear safety guarantee (Theorem 4.2) showing that policies trained with SPIDR will satisfy constraints in the real world.\nThe deployment in real-world scenarios validates its ability of zero-shot transfer of sim2real.\n\nThank you acknowledging these aspects of our work!\n\n> The primary weakness lies in the practical requirement to tune the pessimism hyperparameter $\\lambda$. The paper provides a heuristic for selecting $\\lambda$ in Appendix E, but this process involves iterative testing on the physical system to find a suitable balance between safety and performance.\n\nWe recognize that selecting an appropriate value for $\\lambda$ is critical to the success of SPiDR. Notably, incorporating domain knowledge, such as insights into simulator fidelity or the CMDP's cost function makes the selection process more straightforward in practice. Additionally, tuning $\\lambda$ was usually not the main bottleneck in tuning parameters, but rather first finding policies that satisfy the constraints in simulation, e.g., tuning primal-dual parameters, or parameters that relate to SAC/PPO.\n\nIn addition, we highlight that the goal of the paper is satisfy the constraints on the real system upon deployment.\nMore formally, we give a safety guarantee and not a performance guarantee for SPiDR. In Appendix E we show how after the first safe deployment, one can indeed improve performance by gradually reducing $\\lambda$. This appendix should rather be viewed as an additional guidance for practitioners when using SPiDR, instead of a formal statement about SPiDR's safety guarantees. We apologize for this confusion and wil clarify this ambiguity in the main text of our revised version. We will additionally clarify in our revised version that the term “zero-shot” refers only to safety, and not to performance. Thank you for this comment and for helping us improve the presentation of our work!\n\n> For systems with very high-dimensional state spaces (e.g., from vision), a much larger ensemble might be required to get a reliable uncertainty estimate, which could impact computational performance.\n\nFigure 7 indicates a “diminishing returns” trend when increasing $n$ beyond $8$, while generally requiring feasible computation resources, even on the Humanoid task with observations of 244 dimensions. Since training happens in simulation, where privileged state information is available, one can still penalize the cost w.r.t. variance in states, instead of penalizing disagreement of images, even if the policy operates on these images without direct access to the states. Such asymmetric training, via a teacher-student architecture, is quite common in practice [1, 2].\nWe agree that the addition of vision control experiments would improve our work. Following your suggestion, we conducted an additional experiment, implementing the asymmetric architecture described above. We use DrQ [3] with CRPO [4] as a penalizer and run it on the CartpoleSwingup task. The policy operates on a sequence of 3 stacked greyscale images, while the cost is penalized with SPiDR by using the privileged system states that are not shown to the agent but are known in simulation. We provide our results below.\n|           | Cumulative Cost (± SE)    | Cumulative Reward (± SE)  |\n|-----------|---------------------------|----------------------------|\n| Domain Randomization | 267.156 ± 405.2127        | 872.506 ± 64.9             |\n| SPiDR     | 44.8047 ± 20.8520         | 674.608 ± 85.54            |\n\nAs shown, even in a partially observable setup, where the policy operates on a 64x64x3 image, penalizing the cost with privileged data (available in simulation), allows the agent to satisfy the constraint, even under a significant model mismatch.\nTraining a vision-based policy for 1M time steps with SPiDR, using an ensemble of $n = 8$ environments, takes roughly 33 minutes on an RTX 4090 GPU. Training the same setup without SPiDR takes about 30 minutes and the main bottleneck is in computing the critic's gradients. This speedup is achieved by using Madrona-MJX, allowing us to render rollouts from 128+ environments in parallel. Thank you for this great question! We believe the addition of this experiment gives SPiDR an interesting direction which we did not think about originally.\n\n> The paper's framework assumes that the dynamics differ between sim and real, does the difference of cost function is also considered?\n\nWe mainly consider perturbations in the dynamics (referred to as “domain randomization”), as it seems, at least empirically, that such perturbations play a critical role in successful sim-to-real transfer across many real-world tasks. Theoretically speaking, our framework can naturally be generalized for mismatches in the cost function. Assuming bounded mismatch between the simulated and the real cost functions, one can show that an additional term relating to this mismatch will appear in Equation 4. Intuitively, as this gap increases, the feasible set of policies shrinks, making it harder to solve the task. However, in practice, such modeling discrepancies are typically modest, allowing us to to compute safe policies in simulation.\nThank you for raising this question, this is an interesting aspect we did not consider, and we will add a discussion on that in the appendix.\n\n> Specifically, for the RaceCar and Unitree Go1 experiments, how many real-world rollouts or trials were required to select the final $\\lambda$ values reported in the paper?\n\nFor both tasks our initial choice of $\\lambda$ (i.e., the largest one) satisfied the safety constraints, as shown in Figure 14. This is in line with our theory and the main claim of the paper. Our initial choices of $\\lambda$ are based on our domain knowledge about the sim-to-real gap of the problems. For instance, the lower value of $\\lambda$ for the Go1 was indeed chosen since the simulator we have for this task is much better compared to the one of the RaceCar (mainly due to difficulties in modeling tire dynamics).\n\n> SPIDR increases the cost, which can potentially render the CMDP infeasible if $\\lambda$ is too high or the sim-to-real gap is too large. Did you encounter this failure mode in your experiments?\n\nWe appreciate this valuable comment. Theoretically speaking, it is possible to think of problems where the sim-to-real gap is large enough so that the feasible set of the penalized CMDP (Equation 5) is empty. More generally, we believe that many real-world scenarios are a bit more optimistic than the theoretical worst-case: simulators are becoming better in many tasks and therefore an optimal policy within the feasible set of the penalized CMDP might not deviate too much from an optimal policy of the real CMDP. Empirically, in our sim-to-real experiments, for increasing values of $\\lambda$, policies qualitatively seemed to be more “cautions”. In example, in the RaceCar task, increasing $\\lambda$ led to policies that drove slower, causing the car to sometimes fail to reach the goal within the specified time horizon. This result is captured quantitatively by the lower costs and rewards in Figure 14.\n\nWe believe our response above answers your questions and would appreciate it if you would consider increasing your score. Please let us know if you have any further questions, we would be very happy to answer.\n\n---\n\n[1] Chen, Fu, Rui Wan, Peidong Liu, Nanxing Zheng, and Bo Zhou. \"VMTS: Vision-Assisted Teacher-Student Reinforcement Learning for Multi-Terrain Locomotion in Bipedal Robots.\" arXiv preprint arXiv:2503.07049 (2025).\n\n[2] Lee, Joonho, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. \"Learning quadrupedal locomotion over challenging terrain.\" Science robotics 5, no. 47 (2020): eabc5986.\n\n[3] Yarats, Denis, Ilya Kostrikov, and Rob Fergus. \"Image augmentation is all you need: Regularizing deep reinforcement learning from pixels.\" In International conference on learning representations. 2021.\n\n[4] Xu, Tengyu, Yingbin Liang, and Guanghui Lan. \"Crpo: A new approach for safe reinforcement learning with convergence guarantee.\" In International Conference on Machine Learning, pp. 11480-11491. PMLR, 2021."}, {"review_id": "ZYeGuYZauq", "reviewer": "Reviewer_qaCC", "summary": "This paper studies the problem of ensuring safety in sim-to-real transfer. Existing robust safe reinforcement learning (RL) methods are difficult to be fitted in large-scale training pipelines, while domain randomization technique often cannot ensure safety in practice. This paper proposes an enhanced version of domain randomization called Sim-to-real via Pessimistic Domain Randomization (SPiDR), which approximates the upper bound of the constraint function under the true dynamics by a set of dynamics generated through domain randomization. It is proved that if the upper bounded constraint is satisfied, the constraint under the true dynamics is also satisfied. SPiDR is tested on two real-world robotic tasks and several simulated tasks from mainstream safe RL benchmarks. Results show that SPiDR remains safe when transferred from simulation to the real world without sacrificing too much performance.", "questions": "1. How are the next states for variance estimation computed (Line 8 in Algorithm 1)? How the dynamics model be called at any state-action pair?\n2. Theorem 4.2 does not consider the approximation of model discrepancy with the variance of next-state predictions. Does the safety guarantee still hold if considering this approximation?\n3. In Figure 5, it is wired that the costs of all baselines keep increasing in RaceCar. Is there any explanation for this result?\n4. In Experiment 4, how is the optimal performance computed?\n5. From Figure 7, it's hard to tell how the time complexity of SPiDR grows with the ensemble size. It's better to first provide a theoretical analysis on time complexity and then compare it with the experiment results. Also, analysis on memory complexity is needed.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Theoretical guarantee on the safety during sim-to-real transfer is provided.\n2. The proposed algorithm is validated in real-world robotic tasks, showing potential real-world applications.\n3. This paper is written clearly. The methods, theories, and experiments are well described and easily understandable.\n\nWeaknesses:\n\n1. The proposed algorithm requires the dynamics model to be directly callable for estimating the next-state prediction variance. That is, given an arbitrary state-action pair, the simulator can return a next state. This requirement cannot be satisfied by many existing simulators, which only supports rollout from a randomly chosen initial state.\n2. The safety guarantee provided by Theorem 4.2 does not consider the approximation of model discrepancy with the variance of next-state predictions introduced in Section 4.2.", "was_revised": true, "final_justification": "The authors responded to four of my five questions. Questions 2, 3, and 4 are addressed. The response to Question 1 indicates that the algorithm does require the simulator to support next-state computation at an arbitrary state, which I believe is a limitation.", "rebuttal": "We would like to thank you for carefully reviewing our paper and for your valuable feedback! Please find below our response to some of your questions.\n\n> The proposed algorithm requires the dynamics model to be directly callable for estimating the next-state prediction variance.  That is, given an arbitrary state-action pair, the simulator can return a next state. This requirement cannot be satisfied by many existing simulators, which only supports rollout from a randomly chosen initial state.\n\nThank you for this important question. To the best of our knowledge, state-of-the-art simulators support setting the system's state arbitrarily out of the box:\n1. In IsaacGym [1] via the `set_actor_root_state_tensor` API.\n2. In DeepMind Control Suite [2] (and MuJoCo) in example: \n```\n    with physics.reset_context():\n        # Set cart and pole initial state [cart_position, cart_velocity, pole_angle, pole_angular_velocity]\n        physics.named.data.qpos[\"slider\"] = init_state[0]  # Cart position\n        physics.named.data.qvel[\"slider\"] = init_state[1]  # Cart velocity\n        physics.named.data.qpos[\"hinge_1\"] = init_state[2]  # Pole angle\n        physics.named.data.qvel[\"hinge_1\"] = init_state[3]  # Pole angular velocity\n    env._task.after_step(physics)\n```\n3. In Brax [3] and Mujoco Playground [4] environments by just calling step, as environments are purely-functional (i.e. stateless), which allows one to directly call the step function with any state-action pair.\n\nWhile our implementation uses Brax and MujocoPlayground, using other simulators is possible. We are keen to provide an implementation of SPiDR in other simulators such as IsaacGym, if there is interest from the robotics and safe RL communities.\n\n> Theorem 4.2 does not consider the approximation of model discrepancy with the variance of next-state predictions. Does the safety guarantee still hold if considering this approximation?\n\nIn appendix C we formally analyze the conditions under which our empirical estimator provides a guarantee for Theorem 4.2. Please let us know if we should highlight this point further in our main text—we are happy to do so.\n\n> In Figure 5, it is wired that the costs of all baselines keep increasing in RaceCar. Is there any explanation for this result?\n\nThank you for raising this interesting question. We note that the Nominal baseline fails simply because it is trained only on the nominal environment, which has different dynamics from the training environment. Domain Randomization fails due to underestimation in the constraint, as shown in Lemma 4.1. Both of these baselines are expected to fail, which leaves us only with RAMU. We believe that the reason for RAMU's failure on this task, relates to how it computes the next-state prediction when updating its risk-averse Bellman update (see [1], page 7). In short, these next-states are sampled as a (random) convex combination between the current state and the next state that was originally predicted by the simulator. In the RaceCar environment, the training dynamics significantly deviate from the evaluation dynamics since they only use a \\emph{kinematic} model, not accounting for tire and motor dynamics. We hypothesize that by taking a convex combination of the current state and the next one, RAMU cannot capture possible states that lie outside the line connecting the current and next state, therefore possibly failing to penalize states that are not seen during training. SPiDR on the other hand uses domain randomization, therefore has possibly richer set of states to learn from.\n\n> In Experiment 4, how is the optimal performance computed?\n\nWe compute the optimal performance by running SAC with CRPO or a primal-dual method (see also Appendix M) on the test dynamics. To the best of our knowledge, these results match the established benchmark performance on these tasks [4]. We will clarify this point in our revised version. Thank you for this question.\n\nWe hope our response addresses all of your concerns, and would appreciate it if you would consider increasing our score. We are happy to answer any other open questions. Thanks again for your active engagement!\n\n---\n\n[1] Makoviychuk, Viktor, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller et al. \"Isaac gym: High performance gpu-based physics simulation for robot learning.\" arXiv preprint arXiv:2108.10470 (2021).\n\n[2] Tassa, Yuval, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden et al. \"Deepmind control suite.\" arXiv preprint arXiv:1801.00690 (2018).\n\n[3] Freeman, C. Daniel, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. \"Brax--a differentiable physics engine for large scale rigid body simulation.\" arXiv preprint arXiv:2106.13281 (2021).\n\n[4] Zakka, Kevin, Baruch Tabanpour, Qiayuan Liao, Mustafa Haiderbhai, Samuel Holt, Jing Yuan Luo, Arthur Allshire et al. \"Mujoco playground.\" arXiv preprint arXiv:2502.08844 (2025).\n\n[5] Queeney, James, and Mouhacine Benosman. \"Risk-averse model uncertainty for distributionally robust safe reinforcement learning.\" Advances in Neural Information Processing Systems 36 (2023): 1659-1680."}]

Author rebuttal:

Meta-review: This paper presents SPiDR, an effective method for safe zero-shot sim-to-real transfer in reinforcement learning. The approach is supported by theoretical guarantees and validated on real-world robotic platforms, demonstrating strong performance. The paper is well-written and easy to follow, and it received unanimous positive reviews. While reviewers raised concerns about parameter tuning (with respect to lambda), simulator querying issue, and scalability to high-dimensional problems, the authors effectively addressed the main issues during the rebuttal and discussion by providing clarifications along with new vision-based and hardware experiments. Overall, this is a good paper that merits acceptance.
Decision: Accept (poster)