Robust Adaptive Multi-Step Predictive Shielding
Authors: Tanmay Ambadkar
Darshan Chudiwal
Greg Anderson
Abhinav Verma
Venue: ICLR 2026 Poster (2026)
Keywords: Safe Reinforcement Learning
Control Barrier functions
Model Predictive shielding
Primary area: reinforcement learning
TL;DR: A robust multi-step control barrier function for a minimally invasive shielding without sacrificing performance

Abstract: Reinforcement learning for safety-critical tasks requires policies that are both high-performing and safe throughout the learning process. While model-predictive shielding is a promising approach, existing methods are often computationally intractable for the high-dimensional, nonlinear systems where deep RL excels, as they typically rely on a patchwork of local models. We introduce **RAMPS**, a scalable shielding framework that overcomes this limitation by leveraging a learned, linear representation of the environment's dynamics. This model can range from a linear regression in the original state space to a more complex operator learned in a high-dimensional feature space. The key is that this linear structure enables a robust, look-ahead safety technique based on a *multi-step Control Barrier Function (CBF)*. By moving beyond myopic one-step formulations, **RAMPS** accounts for model error and control delays to provide reliable, real-time interventions. The resulting framework is minimally invasive, computationally efficient, and built upon robust control-theoretic foundations. Our experiments demonstrate that **RAMPS** significantly reduces safety violations compared to existing safe RL methods while maintaining high task performance in complex control environments.

Published as a conference paper at ICLR 2026
ROBUST ADAPTIVE MULTI-STEP PREDICTIVE SHIELD- ING
Tammay Ambadkar"
Darshan Chudiwal
The Pennsylvania State University University Park, PA 16802, USA tsa5252epsu.edh
The Pennsylvania State University University Park, PA 16802, USA dsc5 dsc5636@psu.ede
Greg Anderson
Abhinav Verma
Reed College Portland, OR 97202, USA prega@reed..o
The Pennsylvania State University University Park, PA 16802, USA verma@psu.ce .dd
ABSTRACT
Reinforcement learning for safety-critical tasks requires policies that are both high-performing and safe throughout the learning process. While preec-pre shielding is a promising approach, existing methods are often computationally intractable for the high-dimensional, nonlinear systems where deep RL excels, as they typically rely on a patchwork local models. We introduce RAMPS, a scalable shielding framework that overcomes this limitation by leveraging learned, linear representation of the environment's 's dynamics. This model can range from more complex operator learned
linear regression in the original state space in a high-dimensional feature space. The key is that this linear structure enables a robust, look-ahead safety technique based on a nulti-step Control Barrier Function (CBF). By moving beyond myopic one-step formulations, RAMPS accounts for model error and control delays to provide reliable, real-time interventions. The resulting framework is minimally invasive, computationally efficient, and built upon robust -theoretccc foundations. Our experiments demonstrate that RAMPS significantly reduces safety violations compared to existing safe RL methods while maintaining high task performance in complex control environments.
INTRODUCTION
Deep reinforcement learning (RL) has achieved remarkable success in solving complex control problems, yet its deployment safety-critical applications like autonomous vehicles and robotics remains a grand challenge Gu al. (2022). A requirement in these domains is not only that the final policy safe, but that safety is maintained throughout the entire learning process. This problem of safe exploration has motivated range of solutions, among which model- preeicipre shielding has emerged as a promising paradigm Jovanovi' c et al. (2020): Brunke et al. (202T).
Existing shielding frameworks present difficult trade-off. On one hand, neural shields learn safety critics from data, offering flexibility but often requiring vast experience and failing to prevent violations during early training [haradhwajj et (202Tb); Dalal al. (0188). On the other hand, symbolic shields provide formal, mathematical guarantees from the first interaction by analyzing an environment model Berkenkamp et al. (2017); Anderson al. (2020); Wang & Zhu (2024). However, these methods have a critical limitation that has confined them to low-dimensional systems: they rely on explicitly partitioning the state space to construct a patchwork of local línear models. This approach suffers from the curse of dimensionality, rendering it computationally intractable for the complex, high- nigh-dimensiioal environments (> 10 dimensions) where modern deep RL excels.
This paper introduces RAMPS, framework that bridges this critical gap by making formal shielding scalable to high-dimensional, nonlinear systems through a novel theoretical advance in safety cer-
"Wehsite: htttp:://abaakkeeeeee

---

Published as a conference paper at ICLR 2026
tification. At the core of RAMPS is new robust multi-step Control Barrier Function formulation that fundamentally changes how safety is guaranteed in discrete-time stochastic systems with model uncertainty. RAMPS achieves both theoretical soundness and practical scalability through unified approach.
The key insight enabling RAMPS is the synergy between robust multi-step CBF theory and the use of linear dynamics models. By representing the system dynamics through single linear model, whether a linear regression in the original space or a learned operator high-dimensional feature space like the Deep Koopman OperatorShi & Meng (022). we can efficiently propagate safety constraints multiple steps into the future while formally accounting for model error. Our CBF formulation explicitly incorporates accumulated prediction error through novel tightening mechanism, provides model-relative safety guarantees even with imperfect models. At each timestep, RAMPS's shield solves comparatively-sman Quadratic Program to find the minimally invasive safe action, with adaptive horizon selection that maximizes foresight while avoiding excessive conservatism.
Our contributions are threefold:
We introduce RAMPS, a scalable shielding framework that provides probabilisitic safety guarantees, in high-dimensions nonlinear systems by unifying robust CBF theory with learned linear dynamics representations.
We develop novel robust multi-step CBF formulation for discrete-time stochastic systems featuring accumulated error tightening and adaptive horizon selection, providing principled solution high relative-degree safety constraints under model uncertainty.
We demonstrate that RAMPS significantly outperforms state-of-the-art safe RL methods, reducing safety violations by up to 90% and scaling to 348-dimensional environments, while maintaining competitive task performance across challenging high-dimensional control environments including quadrupedal locomotion.
2 RELATED WORK
Research in safe reinforcement learning (safe RL) can be categorized by what kind of safety guarantees are provided and when those guarantees apply. Safety usually defined in two ways: formulation, where each action may incur some penalty and the long-term cost must remain below a threshold, or (ii) a state-based formulation, where specific regions of the state space are marked unsafe and must never be entered. Our work adopts the state-based view.
cost- cost-based
Worst- Case Guarantees. One line of work provides deterministic safety guarantees under a worst- case environment model, ensuring forward invariance by construction (Anderson et al. 2020 Gillula & Tomlin 2012 Alshiekh et al., 2018 Zhu et al.) 2019: Fulton & Platzer 201 [Baci et al. (021). These approaches offer strong guarantees but require explicit model and are limited to low- dimensional settings due to the computational cost of state-space partitioning. In contrast, RAMPS does not require predefined model and remains tractable in high-dimensional systems.
Statistical Guarantees. Another family of methods offers probabilistic or statistical safety guaran- tees. These approaches build learn an approximate dynamics model and optimize policies that are likely to be safe with respect to that model (Achiam et al., 2017 Liu et al. [2020; Yang et al. 2020 Ma al. 2021 Zhang et al. 2020; Satija al.] 2020). While more scalable than worst-case methods, they typically allow safety violations during training. In contrast, RAMPS enforces hard constraints with respect to its learned model, reducing violations in practice.
Model-Predietive Shielding. A complementary paradigm is model-predictive shielding (MPS), where a shield monitors the agent's proposed action and intervenes only when it threatens safety (Wabersich & Zeilinger 2018 Bastani (021: (Anderon et al.] 2020 2023 Goodall & Belar) dinellli 2023) Banerjee et al.] 2024). Prior works differ in how they construct models and shields, but most struggle with scalability, particularly when moving beyond simple one-step predictions. Model-predicte shielding is closely related to model predictive control (MPC): both use a model to roll out multi-step trajectories and solve constrained optimization problems. However, their objectives differ fundamentally. MPC optimizes long-howd horizon performance and effectively replaces the policy with its control solution, whereas MPS acts purely safety filter: it retains the agent's 's action
2

---

Published as a conference paper at ICLR 2026
whenever it is safe, and otherwise solves a feasibility problem to return the closest safe alternative. This shifts the role of prediction from planning to minimal, targeted intervention, making shielding compatible with arbitrary RL policies while still enforcing hard safety guarantees.
Koopman Operators and Safety. Prior work has combined Koopman models with safety mech- anisms, typically through one-step CBF filters. This includes Gcopman-acceleratee backup-CBF controllers, and neural or deep approaches that learn Koopman embeddings together with one-
CBF-QP filters or command governors Folkestad al. (2020); Zinage & Bakolas (2022); Chen et al. (2024): Mitjans et al. (2024); Liang el al. (2025). Robust Koopman-MPC methods provide predictive control with error guarantees Mamakoukas et al. (2022); de Jong et al. (2024). However, these methods either assume known backup controller. rely on SMT-based CBF certification, or
remain limited to one-step filtering and moderate-dimenniinn
systems.
Cost-Based Safe RL. Cost-based methods enforce safety indirectly by shaping the reward with carefully designed cost signals and applying constrained optimization techniques (Achiam et al.. 2017 Sootla al.] 2022a; Gu et al.) [2024] [Sootla et al., [022b] Zhang et al. [2022] Yang et al. (022)) These approaches are flexible but inherently allow violations while the agent learns the cost structure. Compared to these methods, RAMPS enforces stricter state-based safety constraints, leading to fewer
violations.
Safe RL methods balance trade-off: formal and symbolic methods offer strong guarantees but do not scale, while statistical and cost-based methods scale but permit many violations early in training. Existing model- prreii-pre shielding methods are typically limited to systems with state dimensions in the tens, as they rely on computationally expensive state-space partitioning or explicit nonlinear model propagation. RAMPS bridges this gap by combining a learned, linear model with novel robust multi-step control barrier function, enabling scalable shielding with strong safety assurances in complex, high-ding nigh-dimensional environments, successfully operating on systems with over 300 state dimensions (where current formal techniques struggle above 10-dimensions), while maintaining real-time computational efficiency.
PRELIMINARIES
Safe Exploration. We model the environment Markov decision process (MDP) M (S,Ar P.7), where S is the state space, .A is the action space, r : x A - R is a re- ward function, P(x' xxa) is a probabilistic transition function, and Po is an initial distribu- tion over states. A policy π maps states to distributions over actions. The long-term return
of a policy is R(T)- = E Es, Ei (r(ss,))) The goal of RL is to find an optimal policy π = arg max, R(T).
Most deep RL algorithms generate a sequence of policies To, ,,,...,,. N with TN + '.. We refer to this sequence as a learning process. In safe exploration, the aim to ensure that every intermediate policy remains safe with high probability. Formally, given safety threshold 8 and unsafe set St, we require V < i I N, Prown, E Su) S 6, while the final policy NN maximizes reward among all safe policies. Following prior work (Anderson et al. [023 Wang & Zhu 2024), we do not require the initial policy No to be safe, since no prior model of the environment is assumed.
Safety Specification. We adopt the common state-based notion of safety in safe RL. The unsafe set Sp defined as a union of convex polyhedra over features of the state space (Anderson et al. 2023 Wang
& Zhu 2024). Equivalently, the safe set can be expressed as S \ Sy = - for matrices G2 and vectors h4. Unions of convex polyhedra are sufficient to approximate any compact safe set to arbitrary precision, and are widely used in model-predictive safety methods.
G S h4 }.
4 ROBUST ADAPTIVE MULTI-STEP PREDICTIVE SHIELDING
RAMPS, provides strong, real-time safety guarantees for reinforcement learning agents by integrating learned, linear dynamics model with robust, certificate-baa based safety shield. The framework is composed of three core components: (1) a learned linear dynamics model that provides a single, global representation of the environment's dynamics from data; (2) a Robust Control Barrier Function
3

---

Published as a conference paper at ICLR 2026
(CBF) that uses this model to certify safety and correct potentially unsafe actions online; and (3) standard deep RL agent that learns a seigh-eeffrmancee policy inside the protection of the shield.
The key requirement for the dynamics model is that it must be linear, this structure enables the effi- cient, multi-step evolution required by RAMPS. This allows for flexible range of modeling choices, from simple linear regression operating in the original state space to the Koopman Operator (Shi & Meng 2022) that learns a linear transition function in a high-dimensional feature space.
RAMPS operates in an iterative loop. The agent first collects dataset of environment interactions. This data is used to train the linear dynamics model and a worst-case error bound, which in turn parameterize the CBF shield. The RI. agent is then trained, with every action being verified and potentially corrected by the shield to ensure safety. The newly collected, safe data is added back to the dataset, allowing the dynamics model and error bound be periodically refined. This creates a eycle where more accurate model leads to a less conservative shield, allowing the agent to explore more freely and learn better policy. This is illustrated in Algorithm
4.1 SAFETY SHIELDING WITH MULTI-STEP ROBUST CONTROL BARRIER FUNCTIONS
We propose a safety shield designed to address a fundamental limitation of standard Control Barrier Functions (CBFs) when applied to discrete-time stochastic systems. Although one-step CBFs offer strong guarantees in continuous time, their discrete-ting time analogues may fail when a system's 's control inputs do not immediately affect the safety constraints; a challenge characterized by a relative degree greater than one. To resolve this issue, we construct multi-step robust CBF by drawing upon principles from the theories of High-Order CBFs (HOCBFs; Tan et al. (2022)) and multi-step predictive control (Chriat & Sun| 2023). Our shield enforces safety over variable prediction horizon H, which ensures that control authority is maintained despite such actuation delays. By adaptively selecting the largest feasible horizon each timestep. the shield maximizes its predictive capability to eliminate "trap" states, which are configurations that appear safe in the short term but lead to inevitable future violations. This is accomplished while remaining minimally invasive the actions proposed by the reinforcement learning agent's policy.
Control Barrier Functions (CBFs) Nagumo 1942 Prajna & Jadbabaie 2004 Wieland & Allgöwer
2007 Ames Ames et al. 2019) are a powerful tool for enforcing safety constraints in control systems by rendering specific region of the state space forward invariant. In the continuous-timg setting, for
system with dynamics å [(xx) + g(z)u and safe set defined as C {xr E IR h(x)) > 0}, a function h is a CBF if there exists class-K function a such that for all E C, the condition suPaeu (L)) + Lgh(z)u ju + a(h(r))] 2 0 holds. This Lie derivative condition ensures that for any state on the boundary of the safe set, there exists a control action that prevents the system from instantaneously exiting C.
In contrast, for a discrete-time system Æk+1 = F F( 44), the condition is fundamentally different. A function h is discrete CBF if for all 24 E C, there exists control 2n E U such that h(F ((((Ix, ux)) 2 Xh(x)). where X E [0, 1] is a decay rate. The key distinction lies in their temporal nature: the continuous condition is infinitesimal, guaranteeing safety based the instantaneous velocity of the system, while the discrete condition provides a guarantee over a finite time step, ensuring that the state step A: + remains safe given the state at step This often makes the discrete condition more conservative, as it must account for the system's 's evolution over the entire sampling period. Reinforcement learning typically deals with discrete-time systems.
Linear Dynamics. The core of our shielding framework relies on learned, linear dynamics model, as this structure is essential for performing the efficient, multi-step predictions needed for robust safety analysis. For systems with simple dynamics, this can be a direct linear model operating in the original state space. For more complex, non-linear environments, the state can be "lifted" via a learned, non-linear embedding into digher-dimensional feature space (Shi & Meng) 2022). The fundamental principle is that within this lifted space, the intricate dynamics can be accurately captured by simple linear transition, 2 Az + Bug- c. This transformation from non-linear to linear dynamics is what enables the shield to efficiently propagate safety constraints far into the future, making the approach scalable to wide range of complex systems.
Safe Set and Dynamics. Let the lifted state space be Ra+d (d 2 0) with discrete-time affine dynamics 2 Azg Bug + We, where e is learned constant offset representing the
4

---

Published as a conference paper at ICLR 2026
system's 's drift, and wi is an additive model enror satisfying ||w S E. The admissible control set C IR". We define a polyhedral safe set as the intersection of half-spaces, such that
M
c = c= I 50)
For each face i of the polyhedron, we define a corresponding safety function h2(2) as hz(2) = -mp z + b.), which means the safe set can be expressed as C = {= hç(=) 2 0, W...
One-Step Robust CBF Condition. To guarantee safety under model uncertainty, we formulate robust CBF condition similar to (Cosner et al. 2023). The safety requirement that the true next state, 2k-11 Az + Bug + + Wk, must remain in the safe set C. This implies that for each face i, the condition p; (Aza + Bu, + c wx) + b, S must hold for bounded disturbances wk-
To ensure this, design the constraint based on the worst-case disturbance, which has value of ellp<lla. 112 By incorporating this worst-case term, we arrive at the robust CBF condition: for any state E C, there must exist control input u E U such that
T Ps (Az + Bu + c) b < x
el|p-||ı, Vi,
where À E (0, 1] is decay parameter that governs the conservatism of the barrier condition. Values of Ả close to 1 require the safety function (z) to remain nearly constant across timesteps, leading to stricter constraints and stronger invariance. Smaller values of relax this requirement by permitting hz(z) to decay over time, which can improve feasibility but reduces the safety margin. The term -Ellpilaa provides an additional robust margin, ensuring safety under the worst-case model error.
Relative Degree. The relative degree of a safety constraint h(2) under dynamics 24+1 = f(x)) + g 24 ) is the smallest integer r such that the control input uy appears explicitly the r-step evolution of h22)), i.e. through 8h(2x
+ O.
Dos
Multi-Step Robust CBF Condition. The one-step condition in equation D insufficient for systems where the control input has delayed effect on a safety constraint (i.e., relative degree r 4.2 To eliminate the trap states that arise in such systems, our method ensures that the safety condition is met every intermediate timestep j over a chosen horizon H, for all j 2 r:s For each such step j. we define the nominal reachable state under control sequence (Mo,.... -H as
2) A's
Σ 10-1 Buk k=
Σ A*c, k=
where the final term represents the cumulative effect of the affine drift. The total accumulated error over this j-step horizon bounded by a tightening term, (p(), which sums the worst-case error at each step:
Ep(P,
Σ Ellp A+ k=0
This leads to set of robust CBF conditions, one for each valid step j and face i:
2,, u) + b. < X
(p.).
Each of these inequalities is linear with respect to the full control sequence u. We aggregate all such constraints into single system of linear inequalities, Gu S h, which guarantees that any feasible control sequence maintains the system within the safe set C throughout the entire horizon.
Minimally Invasive Action Selection. For a horizon H, the shield solves a Quadratic Program (QP) to find a safe control sequence that is minimally invasive to the RL agent's 's intended action, 0,, The primary objective is to find a control sequence u (ug.... ... uH-1) that minimizes the deviation of the first action, up. from the agent's proposal:
min
Iluo ax 22
s.t.
Gu - h, (representing all constraints from equation 2)
uk E U, k= 0...... H H-1.
5

---

Published as a conference paper at ICLR 2026
Following the receding horizon principle, only the first action of the solution, to. is applied to the system. The subsequent actions, M1:H-1, are optimized to ensure a feasible trajectory exists but are discarded, preserving flexibility at the next timestep.
Adaptive Horizon Selection and Safety Guarantee. At each timestep, we select the horizon H via bounded binary search within miii: Hmax] where miin is the maximum relative degree among active constraints. Candidate horizons are tested by solving the QP in equation (3 feasible horizons remain candidates while the search continues toward larger values, and infeasible ones shrink the range. The largest feasible horizon H* determines the minimally invasive action Un-
If no feasible horizon is found, a backup policy (backup (=) (A.4) is applied. Otherwise, the chosen action uo guarantees forward invariance: under disturbances [w l/ll/ oo S E, the closed-loop system satisfies 2yz C, Vk S H.
4.2 ANALYSIS OF THE SHIELDING FRAMEWORK
The efficacy of our framework stems from the powerful synergy between a learned linear dynamics model and the multi-step robust CBF shield. Each component is designed address a fundamental challenge in safe control, and their integration yields solution that is formally sound, robust to model error, and computationally tractable.
Synergy of Linear Model and Multi-Step Shielding. The foundational element of our approach the use of a linear dynamics model. This structure the key enabler for our multi-step shield; allows safety constraints, defined as simple polyhedra, to be accurately and efficiently propagated through time. Unlike methods that rely on repeated local linearizations or computationally expensive nonlinear propagation, our approach maintains tractability even over extended prediction horizons. This synergy is critical: the linear model makes multi-step prediction feasible, and the multi-step prediction is what gives the shield its foresight and power.
core design principle of our framework is that it does not assume a perfect dynamics model. Instead, it achieves robustness by formally accounting for model error. The shield's safety guarantee is not based on the model's nominal prediction alone, but on a worst-case analysis that considers the maximum possible deviation. The robust tightening term, Ey (pt). is derived from data-driven error bound e, effectively creating a tube of uncertainty around the predicted trajectory. By ensuring this entire tube remains within the safe set, the shield remains effective even when the learned linear model is an imperfect approximation of the true, complex dynamics. This allows the framework to work well even with simple models like linear regression, as it plans for their inherent inaccuracies.
Robustness to Model Error.
Resolving High Relative-Degree Traps in Pendulum. The multi-step CBF framework also addresses traps in systems where the safety constraint depends on state that the control input does not influence in a single step. Consider the pendulum environment with state (0,w), representing angle and angular velocity. Its dynamics can be written in affine form as
Illustrative Example:
AI
0
0
2k+11 = Azz + Bug + c(2x), with A
B
c(2x)
0 1
34t
gAt 2
sin (0x))
Suppose we impose safety constraint on the angle, b O. with normal p = [1 T The influence of the control input in one step is determined by p B, which evaluates to 0. Thus, one-step CBF cannot act directly on 0 to prevent it from exceeding the bound. This creates relative-degree trap: the shield has no immediate authority over the constrained variable.
In contrast, our multi-step formulation evaluates terms such T A For the pendulum,
T AB [1
3dta
+ 0. This non-zero term indicates that the control input does affect 0, mi?
but only after two steps. By enforcing constraints over horizon H 2 r (here, = 2), our framework ensures that the control authority is accounted for, thereby resolving the trap. The affine term c(2g) shifts the dynamics but does not alter the relative-degree analysis. This mirrors the role of High-Order
Control Barrier Functions in continuous-time systems (Tan et 2022 Chriat
Sun 2023).
Conditional Safety Guarantees The safety guarantee provided by our framework a probabilistic certificate, which is standard for systems with learned dynamics. The argument is twofold: we first

---

Published as a conference paper at ICLR 2026
establish deterministic guarantee of safety relative our learned model and its error bound, and then connect this guarantee to the true physical system with a probabilistic bound.
Guarantee Relative to the Learned Model. Let the true, unknown, discrete- time dynamics of the system be governed by the function F S A - S, such that the true next state is 5k+1 = F Fsk ux). Our framework learns a linear model, which we denote as F. that approximates these dynamics in a lifted space.
k++1
F(z u.) = Az + Bug +
The residual dynamics, or one-step prediction error, is the difference between the true evolution of the lifted state and the model's prediction, denoted by wa = 2k+1 λx+1- Our shield is constructed using the model E and worst-case bound on this error, | loo This leads to the following guarantee.
Theorem (Conditional Model-Relative Forward Invariance). Given the learned dynamics model F' and an error bound s, if at every timestep k the multi-step robust CBF problem defined by the constraints in equation is feasible, and the residual dynamics satisfy ||w S then the state of the system n₁ is guaranteed to remain within the safe set C for all k 2 20 (Blanchini 1999).
Proof. The proof is by construction and induction. At any state 2 E C, the feasibility of the QP equation jimplies the existence of a control sequence u that satisfies the robust multi-step CBF condition in equation [ This condition, by its formulation, ensures that all intermediate states Zk+1...... H remain within C for any possible realization of the error sequence where each ||w5 < By applying the first action uo of this sequence, the resulting state 2k+1 is guaranteed to be in C. The argument then applies recursively at timestep k + 1, as long as the condition stays feasible k +
L
While Theorem D establishes conditional recursive feasibility under the assumption that the QP remains feasible at every timestep, this requirement is standard but difficult to analytically guarantee in practical safe-control or safe-RL settings, especially when the dynamics model learned. Prior model-based shielding and safe-exploration methods similarly rely on stepwise feasibility assumptions in their theoretical guarantees, while noting that aninite-horizon feasibility cannot be fully certified in practice and is instead supported empirically (Wang & Zhu, 2024 Anderson al., 2023 Banerjee jet al. 2024; Wachi et al.] (023). Consistent with this common limitation, we find that the QP in our framework is feasible in over 98% of timesteps, indicating that the theoretical assumption is well-satisfied in practice.
Probabilistic Connection to the Physical System. The deterministic guarantee of Theorem D is conditioned on the validity of the error bound e. In practice, E is estimated empirically from a finite hold-out validation dataset, Deal, the maximum observed one-step prediction error. The connection between this empirical bound and the true, underlying error distribution is necessarily probabilistic, but the bound is maintained with high probability. Theorem formalizes this connection. The proof is given in Appendix A.2
Theorem 2 (High-Prooe Probability Model Accuracy). Lel EN be a set of i.i.d. sampled model errors from our learned model F. Assume that the probability of any two samples being equal is zero. Choose quantile q 1 and let be the [qN] 'th smallest value among El EI...., Then
F(l) mx) F(ss,
ox ((ss, a] + (2N))/3
4
4(21/3)
Corollary 1 (Probabilistic Forward Invariance over Finite Horizon). Let = q + (2N)773
N775 be the failure probability of the empirical error bound efrom Theorem 2 If the multi-step robust CBF problem (Eq. 3) is feasible at every timestep k E {0..... K - 1} over finite horizon of K steps, then the true system state zx remains within the safe set for all k E (0...., K} with probubility P
K5.
Proof Sketch. By Theorem 2 Pr wwwle 20 <0) for each timestep k. By union bound over K
timesteps, Pr(vi E {O..... ......K K - 1}: /wwales I <) 2 1 - K8. Conditioning Theorem [ forward invariance on this high-probabilltty event yields the result.
7

---

Published as a conference paper at ICLR 2026
5 EXPERIMENTAL EVALUATION
We conduct experiments to evaluate RAMPS on suite of challenging control tasks. Our evaluation designed to answer three primary research questions: Safety Analysis: Does RAMPS reduce safety violations more effectively than state-of-the-ar safe RL algorithms? 2. Safety-Performant Tradcoff:
Does the minimally invasive nature RAMPS allow the agent to learn a tigh-perfonning policy? 3. Role of Model Expressiveness Does improved representational power of the learned dynamics model enhance the shielding performance of RAMPS?
Environments. We evaluate method on five challenging environments. Pendulum is a clas- sic cww-dimensional control task. SafeHopper, SafeCheetah, SafeAnt and SafeHumanoid are high-dimensional locomotion tasks from the Sirett--Gymniiumm benchmark (Ji et al., (023) Safe- Humanoid is a challenging benchmarks due to their high-dimensional state (348) and action spaces (17) and the complex, unstable dynamics of legged locomotion, where sophisticated coordination is required to prevent falling.
Baselines. We compare RAMPS against two classes of baselines. First, we consider state-of-the Constrained Markov Decision Process (CMDP) algorithms that optimize for reward while treating safety as constraint: PPOSaute (Sootla et al. 2022a) P30 (Zhang al. (022). and CUP (Yang al. 2022). We use the implementations from the OmniSafe-Re RL. library (Ji et al. 2024). We compare against these methods because, unlike many symbolic approaches, they are capable of operating in the high-dimensional environments we consider. We discuss additional baselines in Appendix A.5.3
Second, we compare against methods architecturally similar to RAMPS, which also learn a dynamics model for shielding. We selected SPICE (Anderson et al. 2023), which learns simple linear model; we refer to this as SPICE + L. To provide a direct comparison of modeling techniques, we also implemented SPICE + K, a variant where replace the original linear model with our learned Koopman operator. We found that while SPICE L failed to scale to the high-dimensional SafeHopper and SafeCheetah environments, SPICE + K was able to produce a stable model. We attempted comparisons with other relevant MPS/MPC techniques DMPS (Banerjee et al.) (2024), VELM (Wang& & Zhu 2024), MASE (Wachi et al. 2023), and Conservative Safety Crities (Bharad- hwaj et al. 2021b), but these methods failed to achieve stable training on the high-dimensional locomotion tasks, accumulating over 1000 violations within the first 20-30k environment interactions. More details are in Appendix A.5.3
Implementation Details. To analyze the impact of the learned dynamics model, we evaluate two versions of our RAMPS framework: RAMPS + L, which uses simple linear model learned via regularized regression in the original state space, and RAMPS + K. which uses the Deep Koopman Operator. The underlying policy for RAMPS variants is trained with PPO and SAC. For all baselines, we add a penalty reward of -100 and terminate the episode upon a safety violation to provide a clear learning signal. For the CMDP baselines, the cost is 1 for a violation and O otherwise. We also ran CMDP baselines using only the sparse violation cost (cost = 1 for a violation, otherwise) without the 100 ppisode-termination penalty; under this protocol the CMDP baselines failed to learn safe or performant policy within our training budget. For all experiments involving AAMPS and SPICE, use maximum prediction horizon of H 5, which is justified A.3 We use Stellato et al. (2020) to solve the quadratic program.
5.1 RESULTS AND ANALYSIS
Safety Analysis. As detailed in Table all variants of RAMPS demonstrates a substantial reduction in cumulative safety violations compared various baselines across different environments. This effect is particularly pronounced in high-dimensional tasks like SafeHopper, SafeCheetah, SafeAnt and SafeHumanoid, where RAMPS variants typically exhibit significantly fewer violations than visually reinforce these findings; while other
other methods. The violation curves in Figure methods often show a continued accumulation of violations during training, the curves for RAMPS variants tend to flatten much earlier in the training phase, indicating the shield's success in mitigating unsafe actions. This suggests that our multi-step shielding approach provides robust safety assurances, especially where optimization- based other model-predictiv predictive methods may face challenges.
The effectiveness of RAMPS stems from its robust shielding framework, rather than solely relying on the underlying dynamics model's accuracy. A comparison between RAMPS + K and SPICE K, both
8

---

Published as a conference paper at ICLR 2026
(a) Hopper
(b) Cheetah
(c) Ant
(d) Humanoid
Cumulative Safety violations (top row in each subfigure) and episodic reward (bottom row) for all sigh-dimensional environments.
Figure 1:
Table I:
Cumulative safety violations during training. Failed indicates that training quit or the agent
never completed a safe episode. Linear Regression baseline; Koopman Dynamics model.
Algorithm
Pendulum
SafeHopper
SafeCheetah
SafeAnt
SafeHumanoid
SauteRL
91 + 22
703 + 78
183 + 25
1221 + 203
319 +: 106
CUP
184 225
673 63
122 + 22
1883 + 221
172 +
P3O
173 + 166
620 + 6
185 +
1481 + 446
183 + 45
SPICE +
495 + 128
Failed
Failed
Failed
Failed
SPICE + K
87 8
459 105
169 + 70
Failed
Failed
RAMPS + + PPO
69
193
7
162 + 42
137 = 134
RAMPS + K PPO
53
172 1
26
111 + 23
154 + 25
RAMPS + K + SAC 25 26
49 + 10
21
242 + 38
11 +
utilizing Koopman dynamics, reveals that RAMPS consistently achieves superior safety performance. This difference highlights RAMPS'S ability to operate effectively even with imperfect model, due to its explicit accounting for model error through robust multi-step predictions. While methods like SPICE typically require highly accurate model with minimal error ensure stable performance, RAMPS'S design allows to maintain safety guarantees across a broader range of model accuracies. Furthermore, the real-time operation of the RAMPS shield is highly efficient. As detailed in Table 2. the mean per-step computation time ranges from just 0.23 ms for Pendulum to 0.40 ms for the high-dimensional Ant environment, suggesting feasibility for real-time control loops.
The Safeet--Performance Tradeoff. A critical aspect of safe RL is balancing stringent safety with high task performance. The reward curves Figure I illustrate that RAMPS effectively navigates this tradcoff. Across environments, RAMPS achieves strong safety while obtaining competitive, and often superior, task rewards compared to the baselines. This suggests that the shield provides necessary interventions without being overly conservative, allowing the policy to exploit high-reward regions.
Policy-Agnostic Shielding, We evaluate RAMPS with both PPO (on-policy) and SAC (off- (off-policy) to highlight that the shield operates independently of the underlying RI. algorithm. SAC is generally more reliable, especially in high-dimensional settings such as HafeHnma while PPO performs competitively and even surpasses SAC SafeAnt. These results indicate that RAMPS is compatible with multiple learning paradigms and scales effectively to challenging continuous-conrol tasks.
The PPO instability observed Humanoid is not shield-specific failure but known limitation of on-policy methods under action modification. Prior work shows that even simple invalid-action masking, structurally analogous shielding because the executed action differs from the policy's proposal, can cause PPO's KL divergence to spike and training to collapse (Huang & Ontañón 2020 Hou et al. 2023). Similar sensitivity has been documented in safe-RI. algorithms such CPO and

---

Published as a conference paper at ICLR 2026
primal-duet CMDP methods, where constraint- distribution shift destabilizes updates without
additional safeguards (Achiam et al.) 2017] Paternain et al. 2019, Ding et al., 2020). In contrast, off- off-policy approaches exhibit greater robustness to distribution mismatch (Liu et al. 2022)
Empirical Feasibility of Multi-step constraints A crucial element of our framework's 's reliability is the practical feasibility of the multi-step QP. We analyzed the action selection distribution, and the results (detailed in Appendix [A.5.5] Table confirm our shield is highly robust. For the complex locomotion tasks, the backup policy was invoked in less than 2% of all timesteps, and for Pendulum and Humnnida it was never used at all. This demonstrates that our primary shield consistently finds a feasible, safe solution, validating the empirical stability of our approach and showing that the conditional guarantee of Theorem D is almost always active.
Role of Model Expressiveness. The choice of dynamics model within RAMPS can influence this balance between safety and reward, particularly in environments with complex dynamics. While both RAMPS + (simple linear model) and AAMPS K (Koopman model) offer significant safety improvements, the more expressive Koopman model generally supports better reward performance. This is observed in all environments, but particularly in SafeCheetah, where RAMPS + achieves extremely low violations but shows lower reward accumulation compared to RAMPS + K. As shown in Appendix AA.6 this is an outcome of the simpler linear model leading to a more conservative shield (due to larger estimated error bounds), resulting in interventions with larger deviations from the neural action. This hinder the agent's ability to learn a policy that maximizes the reward. In contrast, the more accurate Koopman model allows for less conservative, yet still provably safe, shield, thereby improving the overall safety- pereyy-peffrmance balance.
A Ablation Analysis. We performed an extensive ablation analysis, detailed in Appendix A.3 to validate the design principles of the RAMPS framework. These studies confirm that robust, multi-step shielding is a co-designed system requiring a careful balance of competing factors. Our most critical finding is that explicit robustness to model error is the essential component for safety; removing the error-aware tightening term proved catastrophic, leading to continuous safety violations regardless of other hyperparameter settings (Figure 2
Furthermore, the ablations justify our hyperparameter choices by exploring key trade-offs. The prediction horizon H must be long enough to resolve high relative-degree traps but short enough to avoid compounding model error (Figure The CBF decay rate X must be permissive enough to ensure the underlying QP remains feasible, as an overly conservative setting harms both safety and reward (Figure B) Finally, we show that a high-confidence error bound (99th percentile) is prerequisite for achieving both safety and high reward, as it creates more stable learning environment (Figure 5) We further evaluate RAMPS under multi--immnniioal safety constraints to demonstrate scalability A.7 In the SAFEHUMANOD benchmark, we simultaneously constrain the coordinate and 18 joint angular velocities (a 21 dimeenniiaa safety set). RAMPS accumulates only 256 violations, whereas CMDP-bbaed baselines exceed 3000 violations and fail learn a safe policy, reflected by their steadily increasing violation curves. Additonally, RAMPS is the only method that attains high task reward of 5,000, while CMDP baselines plateau near 500. Together, these results show that RAMPS maintains safety even under high-dimensional constraints without sacrificing performance. Collectively, these results validate our methodology and demonstrate that effective shielding arises from a calibrated synthesis of all framework components.
6 CONCLUSION
We present RAMPS, a scalable model-predictive shielding framework that enables safe policy learning for complex, high-dimensional systems. The core our approach is the synergy between a learned, linear dynamics model and a robust, multi-step safety shield. By leveraging a linear representation. which can range from simple regression to a more complex latent model like the koopman operator, RAMPS remains computationally tractable. Its multi-step. adaptive-horizon Control Barrier Function provides strong foresight to prevent safety violations, even when the learned model is an imperfect approximation of the true dynamics. Experiments on suite of challenging environments demonstrate the efficacy of RAMPS, showing it can dramatically reduce safety violations while maintaining high task performance. Its ability to learn a reliable safety model from few samples makes it particularly well-suited for deployment of reinforcement learning agents in safety-critical applications.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Picter Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17. pp. 22-31. org, 2017.
Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topeu. Safe reinforcement learning via shielding. Sheila A. Mcllraith and Kilian Q- Weinberger (eds.), Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligncce (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana. USA, February 2-7, 2018, pp. 2669-2678. AAAI Press, 2018. URL https: /www: aaai org/ocs index.
/
AAAI /AAIAAAAIII /ttpa//AAAAA/aaappeeea/a//////1/
Aaron D Ames, Samuel Coogan, Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, and Paulo Tabuada. Control barrier functions: Theory and applications. In 2019 18th European control
conference (ECC), pp. 3420-3331. Icee, 2019.
Greg Anderson, Abhinav Verma, Isil Dillig, and Swarat Chaudhuri. Neurosymbolic reinforcement learning with formally verified exploration. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6172-6183, Curran Associates, Inc., 2020, URL htttps::/prooceediiiii neurips cc/ pdÉ
Greg Anderson, Swarat Chaudhuri, and Isil Dillig. Guiding safe exploration with weakest precondi- tions. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023, URL poster/12258
Edoardo Bacci, Mireo Giacobbe, and David Parker. Verifying reinforcement learning up to infinity, In Zua Zhou (ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 2154-2160. International Joint Conferences on Artificial Intelligence Organization, 8 2021. doi: (0.24933jjaai.2221//99 Utthttps: /ttp:///0//o..011.044440400
ijcai. 2021/297
Main Track.
Arko Banerjee, Kia Rahmani, Joydeep Biswas, and Isil Dillig. Dynamic model predictive shielding for provably safe reinforcement learning. In The Thirty-ightth Annual Conference Neural Information Processing Systems, 2024. URL hittps://openreeee net Ftt.://uwww.eeeeoooo..iit
x2zY 4hz44h
Osbert Bastani. Safe reinforcement learning with nonlinear dynamics via model predictive shielding. In 202/ American Control Conference (ACC), pp. 3488 -1944, 2021, doi: 10.23919/ACC50511 2021 9483122.
Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. Advances in neural information processing systems, 30, 2017
Homanga Bharadhwaj, Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Animesh Garg. Conservative safety crities for exploration. In International Conference Learning Representations, 2021a. URL htt.hhttpss//peeeeeeeeeee net http://www.ee/eee..tto... 1id:idd=1i00 fd=///rrmmmmmmaa 6DUuK
Homanga Bharadhwaj. Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Animesh Garg. Conservative safety critics for exploration. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021b. URL URLLhttpss:/ppeereee wwwpiirrwveeieeeeteeeeeeee fow./urumteetttmmmmmmmm htt.rrummmiida00000000000
Franco Blanchini. Set invariance in control. Automatica, 35(11)::747-1766) 1999.
Lukas Brunke, Melissa Greeff, Adam W. Hall, Zhaocong Yuan, Sigi Zhou, Jacopo Panerati, and Angela P. Schoellig. Safe learning in robotics: From learning- based control safe reinforcement learning. ArXiv, abs/2108.06266 2021. URL http:///wwwwwwoop/////////// pddf
11

---

Published as a conference paper at ICLR 2026
Hao Chen, Xiangkun He, Shuo Cheng, and Chen Lv. Deep koopman operator- infeoor- safety command governor for autonomous vehicles. IEEE/ASME Transactions on Mechatronics, 29(5): 3568-3578, 2024.
Alaa Eddine Chriat and Chuangchuang Sun. On the optimality, stability, and feasibility of control barrier functions: An adaptive learning-basee approach. IEEE Robotics and Automation Letters, (11):7865-7872, 2023.
Ryan K Cosner, Preston Culbertson, Andrew Taylor, and Aaron Ames. under stochastic uncertainty with discrete-time control barrier functions. arXiv preprint arXiv:2302.07469 2023.
Robust safety
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerík, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. CoRR, abs/1801 bbss/100108857,, 2018. URLhttp
08757
Thomas de Jong, Valentina Breschi, Maarten Schoukens, and Mircea Lazar. Koopman data-driven predictive control with robust stability and recursive feasibility guarantees. In 2024 IEEE 63rd Conference on Decision and Control (CDC), pp. 140-145. IEEE, 2024.
Dongsheng Ding, Kaiqing Zhang, Tamer Basar, and Mihailo Jovanovic. Natural policy gradient primal-dual method for constrained markov decision processes. Advances in Neural Information Processing Systems, 03:8378-8390, 2020.
Carl Folkestad, Yuxiao Chen, Aaron D Ames, and Joel W Burdick. Data-driven safety -rittccl control: Synthesizing control barrier functions with koopman operators. IEEE Control Systems Letters, (6):2012-2017, 2020.
Nathan Fulton and André Platzer. Verifiably safe off-model reinforcement learning, In International Conference on Tools and Algorithms for the Construction and Analysis of Systems, pp. 413-430. Springer, 2019.
Jeremy H. Gillula and Claire J. Tomlin. Guaranteed safe online learning via reachability: tracking a ground target using quadrotor. In IEEE International Conference Robotics and Automation, ICRA 2012, 14-18 May, 2012, St. Paul, Minnesota, USA, pp. 2723-2730, 2012. doi: 10.1 109/ ICRA.2012.62251380 URL httss://///oo.oooo http://0011199/1111 2012. 6225136
Alexander W. Goodall and Francesco Belardinelli. Approximate model-based shielding for safe reinforcement learning. In Kobi Gal, Ann Nowé, Grzegorz Nalepa, Roy Fairstein, and Roxana Radulescu (eds.), ECAI 2023 26th European Conference on Artificial Intelligence, September 30 October 2023, Kraków, Poland Including 12th Conference on Prestigious Applications of Intelligent Systems (PAIS 2023), volume 372 of Frontiers in Artificial Intelligence and Applications, pp. 883 890. IOS Press, 2023. doi: 10.3233/FAIA220035. URL /tttps://doi.com
32.33 FAA/A203A7277
Shangding Gu, Longyu Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang. Yaodong Yang. and Alois Knoll. A review of safe reinforeement learning: Methods, theory and applications. ArXiv, abs/2205. 2022. URL Ut/ 248965265
www.steescchottoooe org/Corpusiii
Shangding Gu, Bilgehan Sel, Yuhao Ding, Lu Wang, Qingwei Lin, Ming Jin, and Alois Knoll. Balance reward and safety optimization for safe reinforcement learning: perspective of gradient manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 11099-211066 2024.
Yueqi Hou, Xiaolong Liang, Jiaqiang Zhang, Qisong Yang, Aiwu Yang, and Ning Wang. Exploring the use of invalid action masking reinforcement learning: comparative study of on-policy and off- algorithms in real-time strategy games. Applied Sciences, 13(14):8283, 2023. 2023.
Shengyi Huang and Santiago Ontañón. A closer look at invalid action masking in policy gradient algorithms. arXiv preprint arXiv:2006. 14171, 2020.
12

---

Published as a conference paper at ICLR 2026
Jiaming Ji, Borong Zhang, Jiayi Zhou, Xuehai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong, Josef Dai, and Yaodong Yang. Safety gymnasium: A unified safe reinforcement learning benchmark. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. httttttt:////// http:://///eeeeeeeeeeee net. forum? d= x XITGGR
Jiaming Ji, Jiayi Zhou, Borong Zhang, Juntao Dai, Xuehai Pan, Ruiyang Sun, Weidong Huang, Yiran Geng, Mickel Liu, and Yaodong Yang. Omnisafe: An infrastructure for accelerating safe reinforcement learning research. Journal of Machine Learning Research, 25(285):1 2024. URL http:://mlr wttp://...rgppppsss//2333
Mihailo R. Jovanovi'e, Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, and Zhaoran Wang. Provably efficient safe exploration via primal-dual policy optimization. In International Conference on Artificial Intelligence and Statisticx, 2020. URL htttss::: //api.apppp///// semanti org
Kaier Liang, Guang Yang, Mingyu Cai, and Cristian-loan Vasile. Safe navigation in dynamic environments using data-driven koopman operators and conformal prediction. arXiv preprint arXiv:2504.00352, 2025.
Zuxin Liu, Hongyi Zhou, Baiming Chen, Sicheng Zhong, Martial Hebert, and Ding Zhao. Constrained model-based reinforcement learning with robust cross-entropy method, 2020.
Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning, pp. 13644-13668. PMLR, 2022.
Bo Li, and Ding Zhao. Constrained
Yecheng Jason Ma, Andrew Shen, Osbert Bastani, and Dinesh Jayaraman. Conservative and adaptive penalty for model-based safe reinforcement learning, 2021. URL httpptt///xxii...ortttt /
2112.07701
Giorgos Mamakoukas, Stefano Di Cairano, and Abraham P Vinod. Robust model predictive control with data-driven koopman operators. In 2022 American Control Conference (ACC), pp. 3885-3892. IEEE, 2022.
Marc Mitjans. Liangting Wu, and Roberto Tron, Learning deep koopman operators with convex stability constraints. arXiv preprint arXiv:2404. 15978, 2024.
Mitio Nagumo. Über die lage der integralkurven gewöhnlicher differeentalgleechungen. Proceedings of the ahysico-mathemattcal society of Japan. 3rd Series, 24:551-559 1942.
Santiago Paternain, Miguel Calvo-Fullana, Luiz FO Chamon, and Alejandro Ribeiro. Safe poli- cies for reinforcement learning via primal-dual methods. arxiv e-prints, art. arXiv preprins arXiv:1911.09101 2019.
Stephen Prajna and Ali Jadbabaie. Safety verification of hybrid systems using barrier certificates. In International workshop on hybrid systems: Computation and control, pp. 477-492. Springer, 2004.
Harsh Satija, Philip Amortila, and Joelle Pineau. Constrained markov decision processes via backward value functions. In Proceedings of the 37th International Conference on Machine Learning, ICML'20, JMLR.org, 2020.
Haojie Shi and Max Q H. Meng. Deep koopman operator with control for nonlinear systems, 2022. URL BRRL/htts://axxii wtwpa/vvrrgga////eeeee 08004]
Aivar Sootla, Alexander Cowen-Rivers, Taher Jafferjee, Ziyan Wang, David H Mguni, Jun Wang, and Haitham Ammar. Sauté rl: Almost surely safe reinforcement learning using state augmentation. In International Conference on Machine Learning, pp. 20423-20443, PMLR, 2022a.
Aivar Sootla, Alexander Nowen-Rivers, Jun Wang. and Haitham Bou Ammar. Effects of safety state augmentation safe exploration. arXiv preprint arXiv:2206. 2266:22:000000.55 2022b.
B. Stellato, G. Banjac, P. Goulart, A. Bemporad, and S. Boyd. OSQP: operator splitting solver for quadratic programs. Mathematical Programming Computation, 12(4):637 672))))37-66622000 2020. doi: 1007/ 12232-020-000111 URL /ttp://////////ooo.oo0/000000000 http:///w..1000000/0050002000200
13

---

Published as a conference paper at ICLR 2026
Xiao Tan, Wenceslao Shaw Cortez, and Dimos V. Dimarogonas. High-order barrier functions: Robustness, safety, and cetoomannce-ritiicee control. IEEE Transactions on Automatic Control, 67 (6):3021-3028, 2022. doi: 10.1 10.1109/AAC.2201330896333
Kim P Wabersich and Melanie N Zeilinger. Linear model predictive safety certification for learning- based control. In 2018 IEEE Conference on Decision and Control (CDC), pp. 7130-7135. IEEE, 2018.
Akifumi Wachi, Wataru Hashimoto, Xun Shen, and Kazumune Hashimoto. Safe exploration in reinforcement learning: A generalized formulation and algorithms. Advances in Neural Information Processing Systems, 36:29252-22277 2023.
Yuning Wang and He Zhu. Safe Exploration in Reinforcement Learning by Reachability Analy- sis over Learned Models, pp. 232-255. 07 2024. ISBN 978-3-031-66632-3, doi: 10.1007/
978-3-031-65633-0.111
Peter Wieland and Frank Allgöwer. Constructive safety using control barrier functions. IFAC Proceedings Volumes, 40(12):462 077:4624467,.. 2007.
Long Yang, Jiaming Ji, Juntao Dai, Linrui Zhang, Binbin Zhou, Pengfei Li, Yaodong Yang, and Gang Pan. Constrained update projection approach to safe policy optimization. In S. Koyejo, Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 9111-9124. Curran Associates, Inc., /ttp::://///cceeee / httppss//prroceeeingg...t neuriss. cccpppscccppppppp httpeeeriiisse/aa////222.... Conference.
2022. URL
file/ 3eeeebbaa55566 2tre://0000cceeeeeooooooooooaaa0 Prapreaseeeeppppeer-o-eereeeeeee
Tsung-Yes Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter Ramadge. Projection-base
constrained policy optimization. arXiv preprint arXiv:2010.03//53 2020.
Linrui Zhang, Li Shen, Long Yang, Shixiang Chen, Xueqian Wang, Bo Yuan, and Dacheng Tao. Penalized proximal policy optimization for safe reinforcement learning. In Lud De Raedt (ed.), Proceedings of the Thirty-First International Joint Conference Artificial Intelligence, IJCAI-22, pp. 3744 3744-3750. International Joint Conferences on Artificial Intelligence Organization, 2022. doi: 2e244963/jcai.222255555 http:///pee////aa..o//0/0022222/ Track.
Main
Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in pol- icy space. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 15338-15349, Curran As- sociates, Inc., 2020. URL httpD///R/sttte///eeeeeeoioooo oag.aa448...t.cc0oaaaaaaaeeaaaaa
wwwp:rooceeedninno..neeeeeeeeee. nwwrips httpsscccpapppee wttppi/.....aapp///////EL/LL
.dff
He Zhu, Zikang Xiong. Stephen Magill, and Suresh Jagannathan. An inductive synthesis framework for verifiable reinforcement learning. In ACM Conference on Programming Language Design and Implementation (SIGPLAN), 2019.
Vrushabh Zinage and Efstathios Bakolas. Neural koopman control barrier functions for safety-critical control of unknown nonlinear systems. arXiv preprint arXiv:2209.0700 2022.
14


Reviews:
[{"review_id": "HkiMkUuMGD", "reviewer": "Reviewer_nR3a", "summary": "This paper proposes a safety shield for RL that uses learned linear (or Koopman-lifted) dynamics and multi-step robust CBFs. At each step, a QP (Eq 3) adjusts the policy's action to stay safe under bounded model error, using an adaptive prediction and a backup controller when the QP becomes infeasible. A high-provability bound on model accuracy is provided, together with the standard forward invariance argument. Experiments on several Safety-Gym tasks show large reductions in safety violations while maintaining comparable rewards.", "questions": "- Can you link model accuracy (Theorem 2) to actual policy safety or optimality? Even a rough probabilistic bound or empirical measure of policy violations would help.\n\n- Are there analytical conditions (e.g., invariant sets or terminal tubes) that guarantee recursive feasibility of the QP in Eq (3)? How often did the backup controller trigger in practice?\n\n- How do you make sure the model keeps improving if exploration stays inside the safe region? Would a mild risk budget or uncertainty-aware relaxation help?\n\n- Do model error and violation frequency really decrease over time?\n\n- In the Koopman setup, have you tried anything beyond zero-padding to encode safety constraints more tightly?\n\n- Since you call the shield \"policy-agnostic\", have you tested it with more than one RL algorithm (e.g., PPO, SAC, TD3)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- I like the idea of combining multi-step CBFs with learned linear or Koopman-lifted dynamics. The adaptive horizon and tightening scheme are a nice way to deal with higher relative-degree constraints and bounded model uncertainty. \n\n- The method is technically consistent. The QP formulation for CBFs is well-posed, and the robust tightening is well-motivated, and Theorem 2 adds a probabilistic model-accuracy bound, albeit without new theoretical depth.\n\n- The framework improves the practical scalability of safety-shielded RL. It could become a helpful module for safe data collection or model-based control, though its broader impact remains to be seen.", "weaknesses": "- Theorem 2 only bounds model accuracy; it doesn't actually tell us whether the learned policy is actually safe or near-optimal. It seems implicitly assumed that having an accurate model automatically leads to a safe, near-optimal policy, but the paper doesn't clearly establish the connection between model accuracy, constraint satisfaction, and policy performance. \n\n- The paper doesn't really say when the QP is guaranteed to be (recursively) feasible. When the QP becomes infeasible, a backup controller is used, but its activation rate and effect on performance are not studied. \n\n- It seems the agent rarely visits high-uncertainty regions as the shield cuts off risky actions near the safety boundary. That means the model might never improve near the boundary, creating a kind of \"self-reinforcing conservatism loop\".\n\n- While Figures 1 and 2 show empirical decreases in model error and safety violations, there's no theoretical or probabilistic guarantee that these quantities improve over time.\n\n- In the Koopman-lifted version, polyhedral safety constraints are zero-padded into the latent space. That's a conservative embedding, not a true preservation of the constant geometry. This may cause unnecessary conservatism.\n\n- Although the proposed framework is described as algorithm-agnostic, experiments employ only one RL backbone. It remains unclear how the shield behaves similarly with algorithms of different exploration styles (e.g., PPO vs SAC vs TD3).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aXqV77UdsY", "reviewer": "Reviewer_8JnD", "summary": "The authors propose RAMPS, a framework that combines multi-step Control Barrier Functions (CBFs) with linear dynamics models derived via Koopman operator learning to enable robust, certified safe exploration in reinforcement learning and control.\nRAMPS formulates safety as the probability of entering an unsafe set and uses a quadratic program (QP) to minimally modify a given policy action to satisfy the multi-step CBF constraints.\nA key theoretical result (Theorem 1) provides a probabilistic safety guarantee under bounded model error, and experiments on simulated control benchmarks demonstrate improved safety and performance compared to existing shielded RL methods.", "questions": "1. Can you provide a sufficient or verifiable condition under which the QP remains feasible at all time steps? If not, how frequently does infeasibility occur in practice?\n2.\tHow sensitive is RAMPS to modeling errors or distribution shift as the policy evolves? Is the error bound ε recalibrated online?\n3.\tHave you evaluated how the number of polyhedral facets affects safety and computational overhead? Could more flexible representations (e.g., neural implicit sets) be incorporated?\n4.\tCan you report the proportion of time steps when the backup policy is triggered and its effect on reward and safety violation rates?\n5.\tWhat are the main limitations of using a linear Koopman model compared to nonlinear learned models, and do you foresee ways to relax this assumption while keeping the QP tractable?\n6.\tWhat is the empirical runtime of the QP and the precomputation stage for large-scale environments (e.g., SafeAnt)? How does the method scale with latent dimension and horizon length?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tExtending formal safety certificates to nonlinear and high-dimensional systems is a central challenge in safe RL, and the paper directly addresses this gap.\n2.\tThe idea of leveraging a Koopman-based linear model for multi-step safety prediction is conceptually elegant and enables computationally tractable QP formulations.\n3. The paper provides a formal safety theorem under bounded model uncertainty, and the derivation is mathematically coherent within the assumed linear structure.\n4.\tThe experiments are extensive and show that RAMPS achieves a good trade-off between safety and task performance, outperforming strong baselines.\n5.\tThe method includes a fallback (backup) policy to handle infeasible QP cases, showing practical awareness of real-world constraints.", "weaknesses": "1. Theorem 1 assumes that the QP problem is feasible at each step but does not provide conditions ensuring that feasibility is preserved over time. Without a proof of recursive feasibility or a characterization of when the QP remains solvable, the safety guarantee is conditional rather than absolute.\n2. While Koopman operator learning is used to obtain linear dynamics, it remains unclear how accurately such models can approximate complex nonlinear environments. Theoretical guarantees rely on bounded model error ε, but the method for estimating or maintaining this bound under policy drift is empirical.\n3.  The unsafe set is modeled as a union of convex polytopes. Although theoretically sufficient for approximation, this assumption may not hold in realistic, highly nonconvex safety scenarios, and the computational cost scales poorly with the number of facets.\n4. The framework’s tractability and guarantees hinge on having a linear model. It is unclear whether similar safety certificates can be extended to fully nonlinear learned models without sacrificing computational efficiency.\n5. The paper could provide quantitative analysis of QP infeasibility frequency, the impact of the backup policy, and how Koopman model accuracy affects safety margins.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xpGuAu6VcJ", "reviewer": "Reviewer_mgYK", "summary": "In this paper, authors propose a shielding-based safe RL framework. Leveraging the linear modelling of the dynamic model and robust multi-step CBF, this framework can provide formal safety guarantee for the RL agent training.", "questions": "1, How is the actuation delay mentioned in line 166 modeled in the proposed dynamic system?\\\n2, How is the shielding mechanism involved in the RL training? Do we need to solve the QP for each action selection?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "The paper overall is well written with clear structure. The method part is clearly illustrated with intuitive explanation. Illustrative example is provided for easy interpretation.", "weaknesses": "1, RL part has been mentioned as main component of the paper in line 143 and 144 Section 4. However, there is no clear illustration how the RL problem is solved combined with the safety shielding throughout the main paper. As a RL track paper, the contribution is doubted.\\\n2, Since the dynamic system is already linearized with particular formulation shown in line 198, it is more natural for review to consider framing it as a MPC/LQR problem, where at each time step combining with proposed multi-step robust CBF can be formulated as an QP. Thus, it is essential to address the motivation of current combination. \\\n3, Combining Koopman operator with control barrier function has already been explored by dozen of works[3,4].\n4, Authors have addressed actuation delay in line 166 and introduction, however, the approach to mitigate this actuation delay has not been discussed in the rest of the methodology part and experiment section.\\\n5, More elaboration is needed on the RL framework side, including how the shielding mechanism apply in the original RL loop.\\\n6, The training details of linear dynamic model is also missing in the paper. It would be much better to have heuristic algorithm diagram to depict each component. \\\n7, For experiment section, more baselines are needed: (1), comparison with MPC based methods are encouraged to include; (2) comparison with CBF-based RLs are needed, particularly those optimization-based methods with formal guarantees[1,2]\n\n[1]:Choi, Jason, et al. \"Reinforcement learning for safety-critical control under model uncertainty, using control lyapunov functions and control barrier functions.\" arXiv preprint arXiv:2004.07584 (2020).\\\n[2]:Wang, Yixuan, et al. \"Joint differentiable optimization and verification for certified reinforcement learning.\" Proceedings of the ACM/IEEE 14th International Conference on Cyber-Physical Systems (with CPS-IoT Week 2023). 2023.\\\n[3]:Folkestad, Carl, et al. \"Data-driven safety-critical control: Synthesizing control barrier functions with Koopman operators.\" IEEE Control Systems Letters 5.6 (2020): 2012-2017.\\\n[4]:Zinage, Vrushabh, and Efstathios Bakolas. \"Neural koopman control barrier functions for safety-critical control of unknown nonlinear systems.\" arXiv preprint arXiv:2209.07685 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3T3z4GdDfV", "reviewer": "Reviewer_XTg3", "summary": "This paper proposes a scalable model-predictive shielding framework that enables safe policy learning for complex, high-dimensional systems.", "questions": "The Weaknesses include my questions.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "The paper is well-written and easy to follow.", "weaknesses": "1. The paper claims that current model-predictive shielding (MPS) frameworks have a limitation: they are \"computationally intractable for high-dimensional, nonlinear systems.\" However, the paper does not clearly define what constitutes \"high-dimensional\" or provide evidence for this limitation. When considering the dimensionality of system states, the proposed method does not appear to address this issue. Furthermore, the dynamics model in MPS can be simplified using order-reduction techniques to reduce dimensionality, which the paper does not acknowledge.\n\n2. The proposed dynamics model is a time-invariant linear model (with fixed system and control matrices), and the multi-step look-ahead derivations depend on this model. However, time-invariant linear models often exhibit significant mismatch with real-world systems, which are typically highly nonlinear. Consequently, the results will lack robustness, and the prediction accuracy will be unreliable in practical applications.\n\n3. MPS can be viewed as a fault-tolerant RL approach for safety-critical systems and can also be considered an advanced variant of Simplex [1]. Considering this, related safe RL frameworks include runtime assurance [2,3], neural Simplex [4], and the runtime learning machine [5]. Compared to the approach proposed in this paper, these frameworks do not require a multi-step look-ahead to guarantee safety; they can ensure safety even when model mismatches are present. Additionally, these frameworks have been validated on real, complex autonomous systems rather than toy simulators. However, the paper lacks comprehensive comparisons with these methods, making it difficult to conclude whether the proposed approach can outperform existing solutions.\n\n4. The experimental systems evaluated in the paper are not particularly high-dimensional, which undermines the motivation stated in the introduction regarding the need to handle high-dimensional, nonlinear systems.\n\n**References**\n\n[1] Sha, L. (2001). Using simplicity to control complexity. IEEE Software, 18(4), 20-28.\n\n[2] Brat, G., & Pai, G. Runtime assurance of aeronautical products: preliminary recommendations. NTRS - NASA Technical Reports Server, 2023.\n\n[3] Sifakis, J., & Harel, D. (2023). Trustworthy autonomous system development. ACM Transactions on Embedded Computing Systems, 22(3), 1-24.\n\n[4] Phan, D. T., Grosu, R., Jansen, N., Paoletti, N., Smolka, S. A., & Stoller, S. D. (2020). Neural Simplex architecture. In NASA Formal Methods: 12th International Symposium, Moffett Field, CA, USA, May 11–15, 2020, Proceedings 12 (pp. 97-114). Springer International Publishing.\n\n[5] Cai, Yihao, Yanbing Mao, Lui Sha, Hongpeng Cao, and Marco Caccamo. \"Runtime Learning Machine.\" ACM Transactions on Cyber-Physical Systems (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)