Title: SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
Venue: NeurIPS 2025
Accepted: yes
Keywords: Safe Reinforcement Learning

Deep Reinforcement Learning

Sim-to-Real

Constrained Markov Decision Processes
Abstract: Deploying reinforcement learning (RL) safely in the real world is challenging, as policies trained in simulators must face the inevitable *sim-to-real gap*. Robust safe RL techniques are provably safe, however difficult to scale, while domain randomization is more practical yet prone to unsafe behaviors. We address this gap by proposing SPiDR, short for Sim-to-real via Pessimistic Domain Randomization—a scalable algorithm with provable guarantees for safe sim-to-real transfer. SPiDR uses domain randomization to incorporate the uncertainty about the sim-to-real gap into the safety constraints, making it versatile and highly compatible with existing training pipelines. Through extensive experiments on sim-to-sim benchmarks and two distinct real-world robotic platforms, we demonstrate that SPiDR effectively ensures safety despite the sim-to-real gap while maintaining strong performance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of ensuring safety in sim-to-real transfer. Existing robust safe reinforcement learning (RL) methods are difficult to be fitted in large-scale training pipelines, while domain randomization technique often cannot ensure safety in practice. This paper proposes an enhanced version of domain randomization called Sim-to-real via Pessimistic Domain Randomization (SPiDR), which approximates the upper bound of the constraint function under the true dynamics by a set of dynamics generated through domain randomization. It is proved that if the upper bounded constraint is satisfied, the constraint under the true dynamics is also satisfied. SPiDR is tested on two real-world robotic tasks and several simulated tasks from mainstream safe RL benchmarks. Results show that SPiDR remains safe when transferred from simulation to the real world without sacrificing too much performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Theoretical guarantee on the safety during sim-to-real transfer is provided.\n2. The proposed algorithm is validated in real-world robotic tasks, showing potential real-world applications.\n3. This paper is written clearly. The methods, theories, and experiments are well described and easily understandable.\n\nWeaknesses:\n\n1. The proposed algorithm requires the dynamics model to be directly callable for estimating the next-state prediction variance. That is, given an arbitrary state-action pair, the simulator can return a next state. This requirement cannot be satisfied by many existing simulators, which only supports rollout from a randomly chosen initial state.\n2. The safety guarantee provided by Theorem 4.2 does not consider the approximation of model discrepancy with the variance of next-state predictions introduced in Section 4.2.\n\nQUESTIONS:\n1. How are the next states for variance estimation computed (Line 8 in Algorithm 1)? How the dynamics model be called at any state-action pair?\n2. Theorem 4.2 does not consider the approximation of model discrepancy with the variance of next-state predictions. Does the safety guarantee still hold if considering this approximation?\n3. In Figure 5, it is wired that the costs of all baselines keep increasing in RaceCar. Is there any explanation for this result?\n4. In Experiment 4, how is the optimal performance computed?\n5. From Figure 7, it's hard to tell how the time complexity of SPiDR grows with the ensemble size. It's better to first provide a theoretical analysis on time complexity and then compare it with the experiment results. Also, analysis on memory complexity is needed.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the critical challenge of zero-shot safe sim-to-real transfer in reinforcement learning, where a policy trained entirely in simulation must satisfy safety constraints upon its first deployment in the real world. The proposed SPIDR builds on domain randomization by incorporating a pessimistic penalty into the safety constraint. Specifically, SPIDR approximates this uncertainty by measuring the variance of next-state predictions from an ensemble of simulators and adds this variance as a penalty to the cost function during training. This encourages the agent to avoid state-action regions where the simulator is unreliable, leading to more robustly safe policies.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) The paper is very well-written, structured, and easy to understand. The motivation is laid out clearly, and the progression from the theoretical concept to the final algorithm design is logical and intuitive .\n2) The proposed method is well-founded in theory. It provides a formal upper bound on the real-world cost (Lemma 4.1) and a clear safety guarantee (Theorem 4.2) showing that policies trained with SPIDR will satisfy constraints in the real world.\n3) The deployment in real-world scenarios validates its ability of zero-shot transfer of sim2real.\n\nWeaknesses:\n1) The primary weakness lies in the practical requirement to tune the pessimism hyperparameter λ. The paper provides a heuristic for selecting λ in Appendix E, but this process involves iterative testing on the physical system to find a suitable balance between safety and performance. This need for real-world tuning somewhat contradicts the \"zero-shot\" claim and could be costly or impractical in many real-world scenarios. \n2) The effectiveness of the empirical variance as an estimator depends on the ensemble size n. For systems with very high-dimensional state spaces (e.g., from vision), a much larger ensemble might be required to get a reliable uncertainty estimate, which could impact computational performance. The paper's experiments are limited to robotics tasks with proprioceptive states, and it does not analyze how the method would scale to such high-dimensional problems.\n\nQUESTIONS:\n1) The paper's framework assumes that the dynamics $p$ differ between sim and real, does the difference of cost function $c$ is also considered? If not, the \"cost-to-real\" gap can also exist due to sensor noise, calibration errors, or differing physics of cost-inducing events (e.g., contacts). How robust do you expect SPIDR to be if the cost function itself is also mismatched? \n2) The success of SPIDR hinges on the pessimism parameter λ, and the tuning guide in Appendix E suggests an iterative process on the real system. This seems to conflict with the paper's \"zero-shot\" premise. Could you please elaborate on this process? Specifically, for the RaceCar and Unitree Go1 experiments, how many real-world rollouts or trials were required to select the final λ values reported in the paper?\n3) SPIDR increases the cost, which can potentially render the CMDP infeasible if λ is too high or the sim-to-real gap is too large. This could lead to the agent learning a trivial \"freezing\" policy to avoid all costs. Take an example that might not be very accurate: what if the $\\lambda v(s_t, a_t) > d$? Did you encounter this failure mode in your experiments?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SPIDR, a novel algorithm designed to address the challenge of zero-shot safe sim-to-real transfer in reinforcement learning. The authors state that standard domain randomization is scalable but typically unsafe, while formal robust methods tend to be too complex for practical use and scale. SPIDR addresses this gap by augmenting standard domain randomization with a pessimistic penalty. The idea behind this pessimism term is that uncertain transitions, i.e., transitions that result in a variety of successor states under an ensemble of randomized simulators, should be penalized so as to avoid exploitation of uncertain dynamics. By adding this state-dependent uncertainty measure as a negative bonus to the safety cost function, SPIDR encourages the learned policy to avoid regions where a variety of simulators do not result in the same successor state, thereby acting more conservateively and improving safety upon real-world deployment. The method is evaluated empirically on a variety of simulated benchmarks and, notably, on two distinct real-world robotic platforms.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n- Significance: The paper addresses a problem of high importance to the field, the reliable deployment of simulated agents to real-world applications. Ensuring safety in sim-to-real transfer is a bottleneck for deploying RL in real-world applications, and a practical, scalable solution may be impactful for both researchers and practitioners. \n\n- Clarity: This paper is written clearly, and the approach is simple conceptually, indicating that it may be easy to integrate into existing pipelines. It modifies standard domain randomization setups in a relatively minimal way, which is by altering the cost signal. Relevant background is explained well and the derivations are straightforward to follow. \n\n- Quality: The empirical evaluation of the proposed evidence is strong, in particular due to the inclusion of real-world experiments with robotic sim-to-real settings. The experiments support the paper's claim, albeit more baseline methods or ablation experiments (see below) could have been run. I believe, however, that this is acceptable given that they are real-world experiments.  is exceptionally strong. \n- The theoretical grounding looks sound in principle. The practical algorithm however deviates from the theoretical grounding in a number of ways (see below). I believe fairly similar related bounds (albeit not exactly the same) have been proposed in the literature I believe. \n\nWeaknesses: \n\n- Originality: The algorithmic novelty seems somewhat limited to me. While the specific application to the model-free sim-to-real setting may be novel, the concept of using ensemble variance as an uncertainty penalty to ensure safety is a rather well-established pattern for example in the literature on safe model-based RL. I believe this line of research is discussed rather sparingly in the current form of the paper, for example several works derive highly related bounds and algorithmic components to the proposed ones that precede the cited work by Sun et al.:\n- Safe continuous control with constrained model-based policy optimization (Zanger et al., 2021) \n- Constrained Policy Optimization via Bayesian World Models (As et al. 2022)\n- Safe Reinforcement Learning by Imagining the Near Future (Thomas et al. 2021)\nthe paper would be strengthened by discussing and differentiating itself from these works in a bit more detail.\n\n- Parameter tuning: An algorithmic weakness in my view lies in the practical need to tune the pessimism parameter $\\lambda$. Parameters of this kind are generally known to be hard to tune, which warrants the use of Lagrangian methods and constrained solvers with access to online data many works in the literature. It is commendable that the authors suggest a procedure for finding this parameter (Appendix E), but from what I understand, this procedure requires tests on the online MDP. In my view, this procedure also contradicts the definition of \"zero-shot\" transfer since it seems likely that one requires at least a \"few shots\" to find an appropriate pessimism level. This distinction should be made more explicit, or (even better) be addressed by directly the inference of $\\lambda$ as an algorithmic component (e.g., by phrasing the problem as a POMDP). I do, however, realize that the latter would be a rather tall ask for a rebuttal. \n\n- Experimental baseline: The above point also affects the experimental evaluation. The comparison with the standard domain randomization is important but feels slightly imbalanced as is due to the presence of the additional tunable hyperparameter. An extremely simple baseline that would be granted a similar tuning pipeline for a pessimism parameter $\\lambda$ could use vanilla domain randomization but simply changes the admissable cost limit as a hyperparameter. \n\n- Stochastic environments: It seems that the current practical implementation of the algorithm would be very susceptible to stochastic transitions, since it replaces the wasserstein distance $D_W$ with variances. For highly stochastic environments, this should result in overly conservative agents if I am not wrong.\n\nQUESTIONS:\nMy main questions relate to the weaknesses I have outline above: \n\n- Could you clarify your definition of \"zero-shot\" in the context of this paper? And does it include the tuning process of $\\lambda$? A more explicit discussion of this definition in the main text would be valuable.\n- How would you expect SPIDR to perform against a simple baseline as the one I outlined above (i.e., vanilla DR where the cost budget itself is treated as a tunable hyperparameter that is allowed to be tuned online)? A discussion of state-dependent vs. uniform pessimism could also be insightful in general. \n- How would you expect SPIDR to perform in highly stochastic environments and are there ways to circumvent potential issues? See above for my reasoning.\n\nI am willing to raise my score if the mentioned points are addressed clearly in a revision.\n\nLIMITATIONS:\nYes. However, a few points, e.g., the need for real-world trials to tune $\\lambda$ or stochastic transitions (if I understood correctly) may be addressed more centrally.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper presents SPiDR, a method for safe zero-shot sim-to-real transfer in reinforcement learning. Motivated by the safety risks from the sim-to-real gap, SPiDR combines domain randomization with a pessimistic cost penalty based on uncertainty from an ensemble of simulators. This penalization ensures policies avoid risky regions during training. Experiments on both simulated benchmarks and real robots (a race car and a quadruped) show that SPiDR achieves safe and effective transfer, outperforming standard domain randomization and robust RL baselines.

STRENGTHS AND WEAKNESSES:
Strengths:
- SPiDR provides provable zero-shot safety during sim-to-real transfer without relying on real-world data
- SPiDR is theoretically sound
- It is simple to implement and integrates easily with existing RL methods 
- It shows strong performance in both simulated benchmarks and real-world robot tasks

Weaknesses:
- The pessimistic penalty can lead to overly conservative policies, potentially reducing task performance.
- SPiDR requires manual tuning of the pessimism parameter

QUESTIONS:
How sensitive is SPiDR to the choice of the pessimism coefficient (λ), and how practical is it to tune this parameter in real-world applications?
Does SPiDR still ensure safety if the real environment lies significantly outside the range of the domain randomization distribution?
What are the limitations of using ensemble variance as a proxy for model uncertainty? Are there alternative metrics that might work better?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03