Dynamic Model Predictive Shielding for Provably Safe Reinforcement Learning
Authors: Arko Banerjee
Kia Rahmani
Joydeep Biswas
Isil Dillig
Venue: NeurIPS 2024 poster (2024)
Keywords: Safe Reinforcement Learning
Model Predictive Shielding
Planning
MCTS
Primary area: reinforcement_learning
TL;DR: The paper proposes a novel method for integrating a dynamic planner within a safe reinforcement learning framework, enabling progress towards the goal while recovering from unsafe situations.

Abstract: Among approaches for provably safe reinforcement learning, Model Predictive Shielding (MPS) has proven effective at complex tasks in continuous, high-dimensional state spaces, by leveraging a *backup policy* to ensure safety when the learned policy attempts to take risky actions. However, while MPS can ensure safety both during and after training, it often hinders task progress due to the conservative and task-oblivious nature of backup policies.
This paper introduces *Dynamic Model Predictive Shielding* (DMPS), which optimizes reinforcement learning objectives while maintaining provable safety. DMPS employs a local planner to dynamically select safe recovery actions that maximize both short-term progress as well as long-term rewards. Crucially,  the planner and the neural policy  play a synergistic role in DMPS. When planning recovery actions for ensuring safety,  the planner utilizes the neural policy to estimate long-term rewards, allowing it to *observe* beyond its short-term planning horizon. 
Conversely, the neural policy under training learns from the recovery plans proposed by the planner, converging to policies that are both *high-performing* and *safe* in practice.
This approach guarantees safety during and after training, with bounded recovery regret that decreases exponentially with planning horizon depth. Experimental results demonstrate that DMPS converges to policies that rarely require shield interventions after training and achieve higher rewards compared to several state-of-the-art baselines.

Dynamic Model Predictive Shielding for Provably Safe Reinforcement Learning
Arko Banerjee
Kia Rahmani
The University of Texas at Austin arko. banerjee weeuueeas.eeeeeeeeeeee..
The University of Texas at Austin kia@durable.ai
Joydeep Biswas The University of Texas at Austin joydeepb@utexas.cem
Isil Dillig
The University of Texas at Austin
isil@utexas.cm
Abstract
Among approaches for provably safe reinforcement learning, Model Predictive Shielding (MPS) has proven effective complex tasks in continuous, high- dimensional state spaces, by leveraging a backup policy ensure safety when the learned policy attempts to take unsafe actions. However, while MPS can ensure safety both during and after training, it often hinders task progress due to the conservative and task-oblivious nature of backup policies. This paper introduces Dynamic Model Predictive Shielding (DMPS), which optimizes reinforcement learn- ing objectives while maintaining provable safety. DMPS employs local planner to dynamically select safe recovery actions that maximize both short-term progress as well as long-term rewards. Crucially, the planner and the neural policy play synergistic role in DMPS. When planning recovery actions for ensuring safety, the planner utilizes the neural policy to estimate long-term rewards, allowing it observe beyond its short-term planning horizon. Conversely, the neural policy under training learns from the recovery plans proposed by the planner, converging policies that are both high-performing and safe practice. This approach guar- antees safety during and after training, with bounded recovery regret that decreases exponentially with planning horizon depth. Experimental results demonstrate that DMPS converges to policies that rarely require shield interventions after training and achieve higher rewards compared several state-of-the-arr baselines.
Introduction
Safe Reinforcement Learning (SRL) [1, 2] aims learn policies that adhere important safety requirements and is essential for applying reinforcement learning to safety-critical applications, such autonomous driving. Some SRI approaches give statistical guarantees and reduce safety violations by directly incorporating safety constraints into the learning objective [3, 4, 5, 6]. In contrast, Provably Safe RL (PSRL) methods aim learn policies that never violate safety and essential in high-stakes domains where even a single safety violation can be catastrophic [7].
A common approach PSRL is shielding [8], where actions proposed by the policy monitored for potential safety violations. If necessary, these actions are overridden by a safe action that is guaranteed to retain the system's 's safety. Model Predictive Shielding (MPS) is emerging PSRL method that has proven effective in high-dimensional, continuous state spaces with complex dynamics, surpassing previous shielding approaches in applicability and performance [9, 10]. At high level, MPS methods leverage the concept of recoverable states that lead to a safe equilibrium in time-steps by following
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

a so-called backup policy. MPS approaches dynamically check (via simulation) if proposed action results in a recoverable state and follow the backup policy if does not.
However, an important issue with existing MPS approaches is that the backup policies are not necessarily aligned with the primary task's objectives and often propose actions that, while safe, impede progress towards task completion-even when alternative safe actions are available that do not obstruct progress in the task. Intuitively, this occurs because the backup policy designed with the sole objective of driving the agent to the closest equilibrium point rather than making task-specific progress, For instance, in an autonomous navigation task, if the trained policy suggests an action that could result in a collision, MPS methods revert to backup policy that simply instructs halting, rather than trying find more optimal recovery approach, such as finding a route that steers around the obstacle. As a result, the recovery phase of MPS frequently inhibits the learning process and incurs high recovery regret, meaning that there a large discrepancy between the return from executed recovery actions and the maximum possible returns from the same states.
Motivated by this problem, we propose novel PSRL approach called Dynamic Model Predictive Shielding (DMPS), which aims to optimize RL objectives while ensuring provable safety under complex dynamics. The key idea behind DMPS employ a local planner [11, 12] dynamically identify& a safe recovery action that optimizes finite-horizon progress towards the goal. Although the computational overhead of planning grows exponentially with the depth of the horizon, in practice, a reasonably small horizon is sufficient for allowing the agent to recover from unsafe regions.
In DMPS, the planner and the neural policy under training play synergistic role. First, when using the planner to recover from potentially unsafe regions, our optimization objective not only uses the finite-horizon reward but also uses the Q-function learned by the neural policy. The integration of the Q-function into the optimization objective allows the planner to take into account long-term reward beyond the short planning horizon needed for recovering safety. As a result, the planner benefits from the neural policy in finding recovery actions that optimize for long-term reward. Conversely, the integration of the planner into the training loop allows the neural policy to learn from actions suggested by the planner: Because the planner dynamically figures out how to avoid unsafe regions while making task-specific progress, the policy under training also learns how avoid unsafe regions in smart way, rather than taking overly conservative actions. From theoretical perspective, DMPS can guarantee provable safety both during and after training. We also provide a theoretical guarantee that the regret from recovery trajectories identified by DMPS is bounded and that it decays at an exponential rate with respect to the depth of the planning horizon. These properties allow DMPS to learn policies that are both high-performing and safe.
We have implemented the DMPS algorithm in an open-source library and evaluated it on suite of 13 representative benchmarks. We experimentally compare our approach against Constrained Policy Optimization (CPO) [3], PPO-Lagrangian (PPO-Lag) [13, 14], Twin Delayed Deep Deterministic Policy Gradient (TD3) [15] and -fate-of-the-aa PSRL approaches, MPS [9] and REVEL [16]. Our results indicate that policies learned by DMPS outperform all basclines in terms of total episodic return and achieve safety with minimal shield invocations. Specifically, DMPS invokes the shield 76% less frequently compared to the next best baseline, MPS, and achieves 29% higher returns after convergence.
To summarize, the contributions of this paper are as follows: First, we introduce the DMPS algorithm, a novel integration of RL and planning, that aims to address the limitations of model predictive shielding. Second, we provide a theoretical analysis of our algorithm and prove that recovery regret decreases exponentially with planning horizon depth. Lastly, we present detailed empirical evaluation of our approach on a suite of PSRL benchmarks, demonstrating superior performance compared to several state-of of-the-art art baselines.
2 Related Work
PSRL. There is growing body of work addressing safety issues in RL [1, 2, 17, 18, 19, 20, 21]. Our approach falls into the category of provably safe RL (PSRL) techniques [7, 18], treating safety as hard constraint that must never be violated. This is in contrast to statistically safe RL techniques, which provide only statistical bounds on the system's 's safety by constraining the training objectives [3, 22, 23, 24, 25). These soft guarantees, however, are insufficient for domains like autonomous driving, where each failure can be catastrophic. Existing PSRL methods can be categorized based on whether

---

they guarantee safety throughout the training phase [26, 27, 16, 28] or only post-training upon deployment [29, 30, 31, 32, 33). Our approach falls into the former category and employs shielding mechanism that ensures safety both during training and deployment.
Safety Shielding, Many PSRL works use safety shields [8, 34, 35, 36, 37, 16, 38, 39j. These methods typically synthesize a domain-specitnn policy ahead of time and use it to detect and substitute unsafe actions at runtime. However, traditional shielding methods tend to be computationally intractable, leading to limited usability. For instance, [8] presents one of the early works in shielding, where safety constraints are specified in Linear Temporal Logic (LTL), and a verified reactive controller is synthesized to act as the safety shield at runtime. However, this approach is restricted to discrete state and action spaces, due to the complexity of shield synthesis. Another example is [39], which enhances PSRL performance by substituting shield-triggering actions with verified actions and reducing the frequency of shield invocations. In Model Predictive Shielding (MPS) [9]. a backup policy dynami- cally assesses the states from which safety can be reinstated and, if necessary, proposes substitute actions. Unlike pre-computed shields, MPS can handle nigh-dimensional, non-linear systems with continuous states and actions, without incurring an exponential runtime overhead [40]. MPS has also been successfully applied to stochastic dynamics systems [41] and multi-agent environments [10]. However, a significant limitation of current MPS methods is the separation of safety considerations from the RL objectives, which hinders learning when employing the recovery policy.
RL and Planning. Planning has traditionally been considered a viable complement to reinforcement learning, combining real-time operations in both the actual world and the learned environment model [42, 43, 44, 45, 46]. With recent developments in deep model-based RL [44, 47, 48, 49, 50] and the success of planning algorithms in discrete [51, 52] and continuous spaces [53, 54, 55, 56, 57], the prospect of combining these methods holds great promise for solving challenging real-world problems. Integrating planning within RL has also been applied to safety measures [58, 59, 21, 60, 61, 62, 63, 64). For instance, Safe Model-Basee -ased Policy Optimization [21] minimizes safety violations by detecting non-recoverable states through forward planning using an accurate dynamics model of the system. However, it only employs planning identify unsafe states, not to find optimal recovery paths from such situations. To the best of our knowledge, DMPS is the first method that leverages dynamic planning for optimal recovery, within the framework of model predictive shielding.
Classical Control. There is long line of classical control approaches to the problem of safe navigation [65, 66, 67, 68, 69, 70]. One common approach, for instance, is the use of control barrier functions (CBFs) [71, 72]. CBFs have been used across many domains in robotics to achieve safety with high confidence [73, 74, 75, 76]. While useful, these approaches tend to make assumptions about the environment (e.g. differentiability, closed-form access dynamics) that cannot easily be reconciled with the highly general RL framing of this work.
3 Preliminaries
MDP. We formulate our problem using standard Markov Decision Process (MDP) M (S, S,,S, A, T, 7), where S C IR" is a set of states, Sy C S is set of unsafe states, Sn C the initial states, A C Rm is a continuous action space, T S x A S is deterministic transition function, R S x A + IR is deterministic reward function, and y is the discount factor. We define 11 to be the set of all agent policies, where each policy π E 11 is a function from environment states to actions, i.e., π : S + A. For any set S C S, the set of reachable states from 5 in steps under policy n, is denoted by reach, ST, and defined recursively as follows: reach1 (T, = (T(s, (s)) E S} and reach;+1(T,, (T, = reach; (T,, reach,(T, The set of all reachable states under a policy ι is reach(T) = U1gr reach, (T).
PSRL The standard objective in RL is to find a policy æ that maximizes a performance measure, J(T), typically defined as the expected infinite-horizon discounted total return. In provably safe reinforcement learning (PSRL), the aim to identify a safe policy that maximizes the above measure. The set of safe policies is denoted by aage C II and consists of policies that never reach an unsafe state, i.e., π E Tsgf < reach((T) n Sy = 0. Therefore, the objective of PSRL to find a policy Tufe = arg max (TT)
3

---

of t- Mouctrpp (8)
planßec( Vange
(Simulate Base Policy)
(s))
IF
-isRec /hackup)
planßec (s, Q Vonee
a' aj
(Execute Next Action) s + T(s
T ()
4- Tradmpp3) (8)
ieRecls, Funkop)
a'
Figure 1: Overview of an execution cycle in MPS (0, 2) and DMPS
Model Predictive Shielding
Under the Model Predictive Shielding (MPS) framework [9], a safe policy, Tmps. is constructed by integrating two sub-policies: a learned policy, Tmps, and a backup policy, Thackup- Depending on the current state of the system, control of the agent's 's actions is delegated to one of these two policies. The learned policy is typically implemented as a neural network that is trained using standard deep RL techniques to optimize J(.). However, this policy may violate safety during training or deployment, i.e., Amps & Ia On the other hand, the backup policy, Toackup- is specifically designed for safety and is invoked to substitute potentially unsafe actions proposed by the learned policy.
Due to domain-specifhr constraints on system transitions, the backup policy effective only from a certain subset of states, called recoverable states. For instance, given the deceleration limits of an agent, a backup policy that instructs the agent to halt can only avoid collisions from states where there is sufficient distance between the agent and the obstacle. At high level, the recoverability of a given state s in MPS is determined by a function isRec:
II + B. This function performs an N-step forward simulation of buckup from and checks whether a safety equilibrium can be established.
Figure 1 gives an overview of the control delegation mechanism in Tmps. Given state E S, the agent first forecasts the next state á that would be reached by following the learned policy (double-bordered. yellow box). If isRec($, Tbuckup) is true, then Tmps simply returns the same action as mmps. as marked by O. Otherwise, if i isRec($, 8, Tuuckkp) is false, mpp delegates control to the backup policy Thackups as indicated by The selected action, a', is then executed in the environment single-borderee green box), resulting in a new state 8'. From this state, Tmps is executed again, and the process repeats.
The safety of Tmps relies on the fact that all recoverable states are safe and that the backup policy is closed under the set of recoverable states. Thus, we can inductively show that the agent remains in safe and recoverable states; thus Tmps E IIgeee
Example.
Consider an agent a 2D plane aiming to reach a goal while avoiding static obstacles. Figure 2 (a) presents an unsafe trajectory pro- posed by the learned policy. Figure 2 (b) presents the trajectory under MPS. As dis- cussed above, mpps delegates control to the backup policy in such potentially unsafe sit- uations, which corresponds here to applying maximum deceleration away from the obsta- cle. This causes the agent to come to a halt, which suboptimal. Figure 2 (c) depicts a DMPS trajectory. Figure 2 (d) depicts DMPS planning in an environment containing static obstacle and low-reward puddle region. The latter two are explained in section 5.
(a)
(b)
(c)
(d)
Figure 2: (a) Unsafe trajectory leading to a collision. (b) Safe but sub-optimal trajectory. (c) Optimal and safe trajectory. (d) An instance of the planning phase.
4.1 Recovery Regret
While MPS guarantees worst-case safety, shield interventions can hinder the training of Amps and compromise overall efficacy. To formalize this limitation, we introduce the concept of Recovery
Arrows @ and 4 used for planner-guided recovery, and explained more in section 5.

---

Regret, which measures the expected performance gap between Tbockup and the optimal policy. To this end. we first introduce a helper function kuckup + {0. 1). which, given state s, indicates whether control in is delegated to the backup policy, ie., hackup( (s) if -isRec (s, kmpx( (8), Thackup) ) and Tupckup (8) 0 otherwise. In any state s where control is delegated to the backup policy, the optimal value V* represents the maximum expected reward that can be achieved from 8, and the optimal action-value (s, (s)) represents the value of executing Thackup in s and thereafter following an optimal policy. Thus, we can define Recovery Regret (RR) as the expected discrepancy between these two values across states where Thackup is invoked, i.e.,
RR( Tbackup Tmps) = Euripp [Ibackup()] (s) (s) Q* Tbackup (s))
In the above formula, na is the discounted state visitation distribution associated with Tmps i.e. " (s) = 7) 7) Et=o Pr((s = Su E Su. Vi20- $1+1 TT mpps (s:))). This term assists in quantifying how frequently each state is visited when measuring the regret of a backup policy.
mmps():))
In existing MPS approaches, the misalignment between backup policies and the optimal policy can result in substantial recovery regrets. For example, Figure (c) illustrates the optimal sequence of actions in the previously discussed scenario, where the agent avoids a collision by maneuvering around the obstacle. However, in MPS, the training process lacks mechanisms to teach such optimal behavior to mpp and only teaches overly cautious and poorly rewarded actions depicted in Figure 2 (b).
5 Dynamic Model Predictive Shielding
In this section, we introduce the Dynamic Model Predictive Shielding (DMPS) framework that builds on top of MPS but aims to address its shortcomings. Similar to the control delegation mechanism in MPS, the policy Tdmps initially attempts to select its next action using learned policy mämpe- If the action proposed by Tdmps leads to a state that is not recoverable using Thackup- an alternative safe action is executed instead, as in standard MPS. However, rather than using the task-oblivious policy Tbackup the backup action is selected using a dynamic backup policy "buckup More formally,
Admps (s) Tackop (s) otherwise
if isseec fdmps(s) (8)), Tbackup)
dmpss (s)
(1)
The core innovation behind DMPS the dynamic backup policy Thackup which realized using a local planner [45, 77], denoted as a function planRec(). At a high level, planRec performs a finite- horizon lookahead search over predetermined number of steps (n E N) and identifies a sequence of backup actions optimizing the expected returns during recovery. Specifically, the function planRec takes an initial state so and state-action value function Q : S x A + IR, and returns sequence of task-optimal actions d0.n E An+1 defined as follows:
planRec(So. Q) = arg max a EAA+1
R(s;, a;)) +7" Q(xn Q(snian))
(2)
1=0
such that, Vogien
T(ss,, a.) and VosignisRec(ss, Toackup).
Crucially, the recovery plan is required to only lead to recoverable states within the finite planning horizon (i.e., isRec(s;, Thackup)). Note that the backup policy Thackup is used by the planner in deciding the recoverability of a state. Beyond satisfying the hard safety constraint, the planner is also required to optimize the objective function shown in Equation 2. Importantly, this objective accounts for both the immediate rewards within the planning horizon, R(s,, a.). well as the estimated long-term value from the terminal state sn beyond the planning horizon, as defined by Q. As result, the planner benefits from the long-term reward estimates learned by the neural policy Admps.
Figure (d) illustrates an agent planning a recovery path around an obstacle, The actions considered by planRec are represented by gray edges. Two optimal paths on this tree, depicted in green and red, yield similar rewards due to the symmetric nature of the reward function relative to the goal
position. The planner selects the green path on the left as its final choice due to a water puddle on the right side of the goal, which, if traversed, would result in lower returns. Since the puddle lies outside of the planning horizon, the decision to opt for the green path is informed by access to the
5

---

Algorithm 1: Reinforcement Learning with Dynamic Recovery Planning
1: Inputs: (M: Markov Decision Process). (E: Episode Count), (Fraccapi Task-oblivious backup Policy)
2: Output: (Rángs! Optimal Learned Policy)
3: Raups initteeuraPPlleey
Tritialize neural network act as the learned policy.
4: B := 0 Initialize an emptl replay butllr..
5: for [1. B] do
6: s == begiaRpisode(e) # Initialize a new episode and receive the first observed state.
7: while -torminated(e) da
8:
"seu :m Ramos # Choose a candidate for the next action using the learned policy.
9:
-iinRee(TT (uer)) Thuccup) then
10:
B = B (a, Creen, Ø, r) RRcord a large negative penalty for triggaring the backup policy. dg.n planRee(s, Stap.)) then duze = og # Plan recovery and choose the next action.
11:
12:
else ueest == (busup()) Use the tisk-oblivious backup policy if planning fails. Lr:: execute( (seu) # Execute the selected next action on the environment.
13:
14:
(e, Graai 8 .r) Update the buffer with the reeent transition record.
15:
16:
B) Update the leamned policy using buffered records.
17:
return Adaus
Q-function. By executing the first action on the green path and repeating dynamic recovery, the agent can demonstrate the desired behavior shown in Figure (c).
Given the plan returned by planRec, the dynamic backup policy Taackup returns the first action in the plan aj as its output. However, planRec could, in theory, fail to find a safe and optimal plan, even though one exists. In such cases, planRec returns a special symbol 1, and 7 backup reverts the task-oblivious backup policy Tbackup- Thus, we have:
{ a
If planRec(s, Q QRamp)) Oo:n
backup (s)
( Thackup (s)
If planRec QFamp))
An outline of Tdmps is shown in Figure where the control delegation mechanism is represented by the red dashed arrows (marked as O and @) instead of the blue solid arrow (marked as 2).
5.1 Planning Optimal Recovery
The specific choice of the planning algorithm to solve Equation 2 depends the application domain. However, DMPS requires two main properties to be satisfied by the planner: probabilistic completeness and asymptotic optimality. The former property states that the planner will eventually find solution if one exists, while the latter states that the found plan converges to the optimal solution as the allocated resources increase. There exist several state-of-ther ofh--e--rttttttt planners that satisfy both of these requirements, including sampling-base based planners such as RRT* [78] and MCTS [79], which have been shown to be particularly effective finding high-quality solutions in high-dimensional continuous search spaces [53, 54, 55]. These methods construct probabilistic roadmaps or search trees and deal with the exponential growth in the search space by incrementally exploring and expanding only the most promising nodes based on heuristics or random sampling. Given an implementation of planRec that satisfies the aforementioned requirements, significant outcome DMPS is that, as the depth of the planning horizon (n) increases, the expected return from Tackup approaches the globally optimal value. The following theorem states the optimality of recovery in Tmmps, in terms of exponentially diminishing recovery regret of Tuacku as n increases.
Theorem 5.1.* 2 (Simplified) Suppose the use of a probabilistically complete and asymptotically
optimal planner with planning horizon n and sampling limit m. Under mild assumptions of the MDP, the recovery regret of policy Taackup used in Tamps is almost surely bounded by order 7 m goes to infinity. In other words, with probability 1,
lim RR(Tho Tdmps) 0(
77-200
5.2 Training Algorithm
Extended theorem statement and proof are provided in Appendix A.I.

---

While our proposed recovery can be used during deployment irrespective of how the neural policy is trained, our method integrates the planner into the training loop, allowing the neural policy to learn to "imitate" the safe actions of the planner while making task-specific progress. Hence, the training loop converges to neural policy that is both high-performant and safe. This is very desirable because DMPS can avoid expensive shield interventions that require on-line planning during deployment.
Algorithm I presents generic deep RL algorithm for training a policy #dmps- Lines 3-4 of the algorithm perform initialization of the neural policy Admps as well as the replay buffer B, which stores set of tuples (s, a. s', ,r) that capture transitions and their corresponding reward r. Each training episode begins with the agent observing the initial state s (line 6) and terminates when the goal state reached or after a predetermined number of steps are taken (line 7). The agent first attempts to choose its next action Gnext using the learned policy Tdmps (Line 8). If the execution of Onext leads to a non-recoverable state according to Thackup (line 9), the algorithm first adds a record to the replay buffer where the current state and action are associated with a high negative reward r- (line 10), It then performs dynamic model predictive shielding to ensure safety (lines 11-12) as discussed earlier: If the planner yields valid policy, the the next action is chosen as the first action in the plan (line 1 11); otherwise, the backup policy Teackup is used to determing the next action. Then, Onext is executed at line 13 to obtain new state s' and its corresponding reward r. This new transition and its corresponding reward are again added to the replay buffer (line 14), which is then used to update the neural policy at line 16, after the termination of the current training episode.
Experiments
In this section, we present an empirical study of DMPS on 13 benchmarks and compare it against 4 baselines. The details of our implementation and experimental setup are presented in Appendix A.3.
Benchmarks. We evaluate our approach on five static benchmarks (ST), where the agent's envi- romment is static, and eight dynamic benchmarks, where the agent's 's environment includes moving objects. Static benchmarks (obstacle, obstacle2, mount-car, road, and road2d) are drawn from prior work [16, 27] and include classic control problems. The more challenging dynamic benchmarks (dynamic-
single-gate, double-gates, and double-gates+) require the agent to adapt its policy to accommodate complex obstacle movements. Specifically, dynamic-obs features moving obstacles along the agent's path. In single-gate, a rotating circular wall with small opening surrounds the goal position. double- Couble-gate is similar but includes two concentric circular walls around the goal, and double- coubleegate++ is the most challenging, featuring increased wall thickness to force more efficient navigation through the openings. For each dynamic benchmark, we consider two different agent dynamics: differential drive dynamics (DD), featuring an agent with two independently driven wheels, and double integrator dynamics (DI), where the agent's acceleration in any direction can be adjusted by the policy. A detailed description of benchmarks can be found in Appendix A.3.
Baselines. We compare DMPS (with a planning horizon of 5) with five baselines. Our first baseline is MPS, the standard model predictive shielding approach. The second baseline is REVEL [16], recent PSRL approach that learns verified neurosymbolic policies through iterative mirror descent. REVEL requires a white-box model of the environment's ss worst-case dynamics, which challenging to develop for dynamic benchmarks; hence, we apply REVEL only to static benchmarks. We also compare DMPS against three established RL. methods: CPO [3], PPO-Lag (PPO Lagrangian) [13, 14), and TD3 [15]. All aim to reduce safety violations during training by incorporating safety constraints into the objective. In TD3, a negative penalty is applied to unsafe steps. In CPO, a fixed number of violations are tolerated. Finally, in PPO-Lag, a Lagrangian multiplier is used to balance safety cost with reward.
6.1 Safety Results
Table presents our experimental results regarding safety. All results are averaged over 5 random seeds. For PSRL methods (DMPS, MPS, and REVEL), which guarantee worst-case safety, we report the average number of shield invocations per episode. Generally, less frequent shield invocation indicates higher performance of the approach. For SRL methods (TD3, PPO-Lag and CPO), we report the average number of safety violations per episode.
7

---

Table Safety Results
Shield Invocations Episode
# Safely Violalions Episode
Benchmark
DNPS
MPS
REVEL
CPO
P90-1ag
TD3
incan
ad
mnca
sd
macnn
sd
mcan
sd
moan
sd
mcan
sd
obstacle2
0.0
0.0
0.0
0.0
33.8
30.3
0.9
0.2
6.42
0.19
0.0
0.0
ST
car
0.0
0.0
0.0
0.0
4.0
4.0
2.0
2.1
22.2
28.9
0.0
0.0
0.0
0.0
0.0
0.8
0.74
0.0
0.0
0.0
0.0
road2d
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
dynanic-obat
iiggleggate
0.1
0.0
0.2
0.1
2.0
1.2
10.0
5.5
0.0
0.0
D1
double-gates
4.5
1.2
28.3
6.9
2.1
1.7
11.8
0.3
0.5
double-gates+
6.4
239.8
16.3
2.9
6.5
dynamic-obst
105.2
39.9
144.9
39.9
1.7
1.9
2.9
2.2
0.8
0.16
DD
single-gate
3.1
1.6
6.9
5.2
6.7
0.1
double-gater
5.5
1.9
52.5
17.6
3.9
1.7
7.9
9.2
3.0
5.5
double- double-gates+
18.4
13.0
106.2
18.5
32.1
2.9
6.9
6.1
0.2
0.4
254
522
150 13 113
15
103
151
125
360
123
double- gate (DD)
double-gast gate (DI)
double- gatee (DD)
double couble-gate+ gatetttee (DI)
Figure 3: Shield Invocations in double-gate and double-gate+
Due to the relative simplicity of the static benchmarks, PSRL baselines achieve safety with very infrequent shield invocations. Even the SRL baselines mostly able to avoid safety violations in these benchmarks. On the other hand, in dynamic benchmarks, PSRL approaches heavily rely shielding to achieve safety, and SRL approaches violate safety more frequently. Notably, the number DMPS shield invocations is significantly lower than in other baselines. Across all dynamic benchmarks, DMPS invokes the shield average of 24.7 times per episode, whereas MPS triggers the shield 124.1 times. The results also indicate that the standard deviation values for shield invocations in DMPS are consistently lower than those of MPS, indicating more stable and predictable performance.
Figure shows the number of shield triggers plotted against episodes for the double saatle-gate and double-gater dynamic benchmarks. The error regions are 1-sigma over random seeds. Under both agent dynamics, DMPS achieves significantly fewer shield invocations compared MPS. Moreover, the number of shield invocations in DMPS consistently decreases training progresses, whereas this trend not present in MPS. In fact, in many scenarios, the number of shield invocations in MPS increases with more training because the learned policy reduces its exploratory actions and adheres more strictly to a direct path to the goal. However, this frequently leads to being blocked by obstacles and results in repeated shield invocations.
6.2 Performance Results
Table 2 presents the per-episode return over the last 10 test episodes run. The reported mean and standard deviations are computed over random seeds. The results indicate that DMPS and MPS exhibit comparable performance across most static benchmarks, with the exception of the more challenging obstacle and obstacle2 benchmarks for which DMPS significantly outperforms MPS. In dynamic benchmarks, DMPS outperforms MPS in all benchmarks except for single-gas (DI), where both methods achieve equivalent results. Both DMPS and MPS consistently outperform REVEL. The SRL approaches (CPO, PPO-Lag, and TD3) perform reasonably well in most static benchmarks their performance significantly deteriorates dynamic benchmarks, failing to achieve positive returns in any instance. This is because the policies in CPO, PPO-Lag, and TD3 rapidly learn to avoid both the obstacles and the goal due to harsh penalties for safety violations, thus accruing the negative rewards for each timestep spent away from the goal.

---

Table Performance Results
Total Return
Final 10 Episade
Benchmark
DRPS
KPS
REVEL.
PPO-Lag
mecn
ad
mean
sd
mnzan sd
mocn sd
mean
sd
mean
5d
32.7
0.3
47.9
52.7
32.9
obstacle2
20.2
15.2
-1.8
3.2
9.3
21.1
33.0
0.2
34.1
0.1
-1.2
3.0
ST
nount-car
81.2
0.3
85.
11.4
34.1
9.6
35.3
-21.3
2.9
-30.0
35.5
0.0
22.7
9.7
16.4
22.7
22.7
0.05
22.7
0.0
road2d
24.0
0.2
24.0
0.2
11.2
16.5
24.0
0.0
24.0
0.1
24.0
0.1
dynamic-oba
24.9
-5.0
single-gate
11.6
0.0
11.6
0.0
19.6
17.0
-2.0
1.1
-2.1
0.2
DI
double-gates
12.7
1.0
11.3
1.1
6.77
5.4
3.9
11.1
-3.6
1.0
double-gates+
0.6
17.4
12.8
-4.1
dynamic-obst
7.4
3.7
6.3
2.1
4.5
0.5
3366
0.8
-5.3
0.5
DD
single-gate
2.5
-244
-3.1
double-gates
13.1
0.5
10.9
1.8
2.4
119
0.9
-3.1
0.5
double-gates+
13.0
0.6
8.3
2.3
2.5
0.3
20.7
22.8
-3.6
0.3
UnC
14.
DM's
110 12 113
11
103
151
12
12.3
106 110
300
12.
double-gate gate (DD)
double-gate (DI)
double-gater (DD)
double- eageee (DI)
Figure 4: Episodic Returns in double-gate and double-gate+
Figure 4 presents the total return plotted against the episode number for the double-gater dynamic benchmarks. The error regions are I-sigma over random seeds. The CPO and PPO-Lag baselines are omitted due to their significantly poor performance, which distorts the scale of the graphs. In all benchmarks, DMPS demonstrates superior performance compared to the other methods. While MPS performs adequately in three of the benchmarks, it exhibits poor performance in the double- -ogte+ (DI) benchmark.
and
6.3 Analysis
We analyze the performance of DMPS and MPS on the double-gate+ environment under double-integrator dynamics. Figure 5a shows traeectories from the first half of training when the agent's 's policy and Q function are still under- trained. When the agents approach the first gate and attempt to cross it, the shield is triggered. In the case of MPS, this shield simply pushes the agent back outside the gate (see blue tra- jectory), and the agent is unable to make any progress. In contrast, DMPS plans actions that maximize an n-step return target, allowing the agent to initially make progress. However, we can observe from the green trajectory that the agent's Q function and policy are under-trained after having made it through the obstacles, the agent is not sufficiently goal-aware to reliably make it the center. This makes the DMPS planner objective inaccurate with respect to long-term return, causing suboptimal actions to be selected by the shield. In the red trajectory, for instance, the agent spends a large portion the trajectory trying to cross the second gate, only to retreat and get stuck in that position until eventually getting through.
SAPS
(a) Early Stage
(b) Late Stage
Figure 5: Example trajectories in double-gatet
MPS backup policy hinders the agent's exploration the environment, impeding the learning of the neural policy. By the end, most runs of the MPS agent are unable to progress past the first gate. Figure 5b shows one of the few MPS runs that successfully make through the first gate towards the end of training: however, the MPS agent still fails to make it through
As training progresses,

---

the second gate. In contrast, the DMPS agent can learn from the actions suggested by the planner, improving its policy and Q function. This improvement makes the DMPS planner's objective a more accurate estimation of long-term return, further strengthening the planner. Consequently, the DMPS agent demonstrates significantly better behaviors, as shown by the red trajectory in Figure 5b.
7
Limitations
7.1
Determinism
First, our approach requires a perfect-information deterministic model, which could limit its usability in real-world deployment. Much prior work provably safe locomotion makes the same determinism assumptions that we do [10, 31, 80], with some such algorithms even having been deployed on real robots [3]. However, extending our DMPS approach to stochastic environments is a promising direction for future work. In particular, since prior MPS work has been extended to work in stochastic settings [40], we believe our DMPS approach can be similarly extended to the stochastic setting,
7.2 Computational Overhead
Another potential limitation of our approach is the computational overhead of the planner. We use MCTS in an anytime planning setting, where we devote a fixed number of node expansions to searching for plan. The clock time needed is linear in the allocated compute budget. However, the worst-case computational complexity to densely explore the search space, as in general plan- ning problems, would be ((cxp(H) since the planning search space grows exponentially. Our implementation used MCTS with 100 node expansions, plan horizon of 5, and a branch factor of 10, which amounts to exploring 1000 states total. On average, we found that when the shield is triggered, planning takes 0.4 seconds per timestep. The code we used is unoptimized, written in Python, and single-threaded. Since MCTS a CPU-intensive search process, switching to language C++ would likely yield significant speed improvements, and distributing computation over multiple cores would further slash the compute time by the number of assigned cores. We perform some additional experiments the double- gates+ environment, outlined in subsection A.4, to see how necessary compute budget scales with planner depth, confirming a rough exponential relationship.
7.3 Sufficiency of Planning Horizon
Finally, it can be asked whether small planning horizons (in our case, we used H = 5) are sufficient to solve tricky planning problems. Our planner objective designed ensure that the planner accounts for both short-term and long-term objectives, preventing overly myopic behavior even with short horizons. However, the length the horizon still affects how close the globally optimal solution the result is, with a tradeoff of computational cost. To see this empirically, we conducted an experiment on the double- double-gates+ gates+ environment, re-evaluating it under different planner depths and observing performance differences. Despite better initial performance, the low horizon agent converged to the same performance once the policy had fully stabilized. As part of our analysis, we also found that trivial planning (H- 1) does not work, reaffirming the necessity of planner. More analysis and experimental results can be found subsection A.5.
8 Conclusions
In this paper, we proposed Dynamic Model Predictive Shielding (DMPS), which is variant of Model Predictive Shielding (MPS) that performs dynamic recovery by leveraging a local planner. The proposed approach takes less conservative actions compared to MPS while still guaranteeing safety, and it learns neural policies that are both effective and safe in practice. Our evaluation on several challenging tasks shows that DMPS improves upon the state-of-the-a methods in Safe Reinforcement Learning.
Impact Statement. Over the past decade. reinforcement learning has experienced significant advance- ments and is increasingly finding applications in critical safety environments, such as autonomous driving. The stakes in such settings are high, with potential failures risking considerable property dam- age or loss of life. This study seeks to take a meaningful step for mitigating these risks by developing reinforcement learning agents that are rigorously aligned with real-word world safety requirements.
10

---

Acknowledgements. This work is partially supported by the National Science Foundation (CCF- 2319471 and CCF-1901376), Any opinions, findings, and conclusions expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors.
References
[1] Garcia, J., F. Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16()):1437-1480, 2015.
[2] Gu, S., L. Yang, Y. Du, et al. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205.
2022.
[3] Achiam, J., D. Held, A. Tamar, et al. Constrained policy optimization. International
conference machine learning, pages 22-31. PMLR, 2017.
[4] Qin, Z., Y. Chen, C. Fan. Density constrained reinforcement learning. In International Conference on Machine Learning, pages 8682-8692 PMLR, 2021.
[5] Miryoosefi, S., C. Jin. simple reward-free approach to constrained reinforcement learning. In International Conference Machine Learning, pages 15666-15698. PMLR, 2022.
[6] Wagener, N. C., B. Boots, C.-A. Cheng. Safe reinforcement learning using advantage-basee intervention. In International Conference on Machine Learning, pages 10630-10640. PMLR, 2021.
[7] Krasowski, H., J. Thumm, M. Müller, et al. Provably safe reinforcement learning: Conceptual analysis, survey, and benchmarking. Transactions on Machine Learning Research, 2023.
[8] Alshiekh, M., R. Bloem, R. Fhlers, et al. Safe reinforcement learning via shielding. In Proceedings of the AAAI conference on artificial intelligence, vol. 32. 2018.
[9] Bastani, O. Safe reinforcement learning with nonlinear dynamics via model predictive shielding. In 2021 American control conference (ACC), pages 3488-3494 IEFE, 2021.
[10] Zhang, W., O. Bastani, V. Kumar. Mamps: Safe multi-agent reinforcement learning via model predictive shielding. arXiv preprint arXiv: /9/0. /2639, 2019.
[11] Moerland, T. M., J. Broekens, C. M. Jonker. A framework for reinforcement learning and planning. arXiv preprint arXiv:2006. 55099, 127, 2020.
[12] Den Hengst, F., V. François-Lavet, M. Hoogendoomn, al. Planning for potential: efficient safe reinforcement learning. Machine Learning, 111(6):225-22274, 2022.
[13] Schulman, J., F. Wolski, P. Dhariwal, et al. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017.
[14] Altman, E. Constrained markov decision processes with total cost criteria: Lagrangian approach and dual linear program. Math. Methods Oper. Res., 48(3):387-417, 1998.
[15] Fujimoto, S., H. Hoof, D. Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[16] Anderson, G., A. Verma, I. Dillig, et al. Neurosymbolic reinforcement learning with formally verified exploration. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 6172-6183. 2020.
[17] Zhao, W. T. He, R. Chen, al. State-wise safe reinforcement learning: A survey. arXiv preprint arXiv:2302.03312. 2023.
[18] Xiong, N., Y. Du, L. Huang. Provably safe reinforcement learning with step-wise violation constraints. Advances in Neural Information Processing Systems, 36, 2024.
[19] Odriozola-Olalde, H., M. Zamalloa, N. Arana-Arexollleiba. Shielded reinforcement learning: review of reactive methods for safe learning. In 2023 IEEE/SICE International Symposium
on System Integration (SIII, pages 1-8. IEEE, 2023.
[20] Shperberg, S. S., B. Liu. A rule-based shield: Accumulating safety rules from catastrophic action effects. In Proceedings of The 1st Conference on Lifelong Learning Agents. 2022.
[21] Thomas, G., Y. Luo, T. Ma. Safe reinforcement learning by imagining the near future. Advances in Neural Information Processing Systems,
2021.

---

[22] Wen, M., U. Topcu. Constrained cross-entropy method for safe reinforcement learning. Ad- vances in Neural Information Processing Systems, 31, 2018.
[23] Alur, R., O. Bastani, K. Jothimurugan, et al. Policy synthesis and reinforcement learning for discounted ltl. arXiv preprint arXiv:2305. 17115, 2023.
[24] Liu, Z., H. Zhou, B. Chen, al. Constrained model-based reinforcement learning with robust cross-entropy method. arXiv preprint (rXiv:2010.07966 2020.
[25] Satija, H., P. Amortila, Pineau. Constrained markov decision processes via backward value functions. In International Conference Machine Learning, pages 8502-8511. PMLR, 2020.
[26] Bacci, E., M. Giacobbe, D. Parker. Verifying reinforcement learning up to infinity. In Pro- ceedings of the International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence Organization, 2021.
[27] Fulton, N., A. Platzer. Safe reinforcement learning via formal methods: Toward safe control through proof and learning. In Proceedings of the AAAI Conference on Artificial Intelligence,
vol. 32. 2018.
[28] Anderson, G., S. Chaudhuri, I. Dillig. Guiding safe exploration with weakest preconditions, In The Eleventh International Conference on Learning Representations. 2023.
[29] Bastani, O., Y. Pu, A. Solar-Lezama. Verifiable reinforcement learning via policy extraction.
Advances in neural information processing systems, 31, 2018.
[30] Schmidt, L. M., G. Kontes, A. Plinge, et Can you trust your autonomous car? interpretable and verifiably safe reinforcement learning. In 2021 IEEE Intelligent Vehicles Symposium (IV), pages 171-178 IEEE, 2021.
[31] Zhu, H., Z. Xiong, S. Magill, et al. An inductive synthesis framework for verifiable reinforce- ment learning. In Proceedings of the 40th ACM SIGPLAN conference on programming language design and implementation, pages 116-701.2019
[32] Verma, A., V. Murali, R. Singh, et al. Programmatically interpretable reinforcement learning. In International Conference on Machine Learning, pages 5045-5054. PMLR, 2018.
[33] Ivanov, R., J. Weimer, R. Alur, et al. Verisig: verifying safety properties of hybrid systems with neural network controllers. In Proceedings of the 22nd ACM International Conference on Hybrid Systems: Computation and Control, pages 169-178. 2019.
[34] Carr, S., N. Jansen, S. Junges, et al. Safe reinforcement learning via shielding under partial observability. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, pages 14748-14756. 2023.
[35] Könighofer, B., J. Rudolf, A. Palmisano, et al. Online shielding for reinforcement learning. Innovations in Systems and Software Engineering, 19(4):379-39 2023.
[36] Jansen, N., B. Könighofer, S. Junges, et al. Shielded mecisoon in mdps. arXiv preprint arXiv: /0fv://8000000 0800186077600.. 2018.
[37] Könighofer, B., F. Lorber, N. Jansen, et al. Shield synthesis for reinforcement learning. In Margaria, B. Steffen, eds., Leveraging Applications of Formal Methods, Verification and Validation: Verification Principles, pages 290-306. Springer International Publishing, Cham, 2020.
[38] Yang, W.-C., G. Marra, G. Rens, et al. Safe reinforcement learning via probabilistic logic shields. arXiv preprint arXiv:2303.03226, 2023.
[39] Thumm, J., G. Pelat, M. Althoff. Reducing safety interventions in provably safe reinforcement learning. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7515-7522. IEEE, 2023.
[40] Bastani, O., S. Li, A. Xu. Safe reinforcement learning via statistical model predictive shielding. In Robotics: Science and Systems, pages 1-13. 2021.
[41] Li, S., O. Bastani. Robust model predictive shielding for safe reinforcement learning with stochastic dynamics. In 2020 IEEE International Conference on Robotics and Automation (ICRA). pages 7166-7172. IEEE, 2020.
[42] Sutton, R. S. Integrated architectures for learning, planning, and reacting based on approximat- ing dynamic programming. In B. Porter, R. Mooney, eds., Machine Learning Proceedings 1990, pages 216-224 Morgan Kaufmann, San Francisco (CA), 1990.
12

---

[43] Lin, L.-J. Selroiipg reactive agents based on reinforcement learning, planning and teaching. Machine learning. 8:293-321, 1992.
[44] Moerland, T. M., Broekens, A. Plaat, et al. Model- reinforcement learning: A survey. Foundations and Trends@ in Machine Learning, 16(1):1-118, 2023.
[45] Schrittwiese,, J., T. Hubert, A. Mandhane, et al. Online and offline reinforcement learning by planning with learned model. Advances in Neural Information Processing Systems, 34:27580- 27591, 2021.
[46] Srouji. M., H. Thomas, Y.-H. H. Tsai, et al. Safer: Safe collision avoidance using focused and efficient trajectory search with reinforcement learning. In 2023 IEEE 19th International Conference on Automation Science and Engineering (CASE), pages 1 18. IEEE, 2023.
[47] Efroni, Y., N. Merlis, M. Ghavamzadeh, et al. Tight regret bounds for model-based reinforcement learning with greedy policies. Advances in Neural Information Processing Systems, 32, 2019.
[48] Hamrick, J. B., A. L. Friesen, F. Behbahani, et al. On the role of planning in model-based deep reinforcement learning, arXiv preprint arXiv:2011.0402. 2020.
[49] Zhao, M., Z. Liu, S. Luan, et al. A consciousness-inspired planning agent for model-based reinforcement learning. Advances in neural information processing systems, 14:1569-1581, 2021.
[50] Wang, T., J. Ba. Exploring model-based planning with policy networks. In International Conference on Learning Representations. 2020.
[51] Silver, D., A. Huang, C. Maddison, al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-489, 2016.
[52] Schrittwieser, J., I. Antonoglou, T. Hubert, et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839):604-666 2020.
[53] Hubert, T., J. Schrittwisser, I. Antonoglou, al. Learning and planning in complex action spaces. In International Conference on Machine Learning, pages 4476-4486. PMIR, 2021.
[54] Kujanpää, K., A. Babadi, Y. Zhao, et al. Continuous monte carlo graph search. arXiv preprint arXiv:22 10.01220.... 2022.
[55] Kim, B., K. Lee, S. Lim, et al. Monte carlo tree search in continuous spaces using voronoi optimistic optimization with regret bounds. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pages 9916-9924. 2020.
[56] Yee, T., V. Lisý, M. H. Bowling, et al. Monte carlo tree search in continuous action spaces with execution uncertainty. In IJCAI, pages 690-697. 2016.
[57] Rajamäki, J., P. Hämäläänen. Continuous control monte carlo tree search informed by multiple experts. IEEE transactions visualization and computer graphics, (00):2540-5555 2018.
[58] Leurent, E. Safe and Efficient Reinforcement Learning for Behavioural Planning in Autonomous Driving. Theses, Université de Lille, 2020.
[59] Huang, W., J. Ji, B. Zhang, et al. Safe dreamerv3: Safe reinforcement learning with world models. arXiv preprint arXiv:2307.07176, 2023.
[60] Rong. J., N. Luan. Safe reinforcement learning with policy-guided planning for autonomous driving. In 2020 IEEE International Conference on Mechatronics and Automation (ICMA), pages 320-326. 2020.
[61] Wang, H., J. Qin, Z. Kan. Shielded planning guided data-efficient and safe reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 2024.
[62] Leung, K., E. Schmerling, M. Zhang, et al. On infusing reachability-based safety assurance within planning frameworks for hobon-robot vehicle interactions. The International Journal of Robotics Research,
2020.
[63] Selim, M., A. Alanwar, S. Kousik, et al. Safe reinforcement learning using black-box reachability analysis. IEEE Robotics and Automation Letters, 7(4): 16665-066611672 2022.
[64] Rong, J., N. Luan. Safe reinforcement learning with policy-guided planning for autonomous driving. In 2020 IEFE International Conference on Mechatronics and Automation (ICMA), pages 320-326. IEEE, 2020.
13

---

[65] Breeden, J. D. Panagou. Predictive control barrier functions for online safety critical control. In 61st IEEE Conference on Decision and Control, CDC 2022, Cancun, Mexico, December 6-9, 2022, pages 924-931 IEFF, 2022.
[66] Wabersich, K. P., M. N. Zeilinger. Predictive control barrier functions: Enhanced safety mechanisms for learning- baaned----ddddee control. IEEE Trans. Autom. Control., 68(5):2638228) 22511888,66,,,, 2023.
[67] Wabersich, K. P., A. J. Taylor, J. Choi, et al. Data-driven safety filters: Hamilton-jacobi reachability, control barrier functions, and predictive methods for uncertain systems. IEEE Control Systems Magazine, 43(5):137-177 2023.
[68] Qin, Z., K. Zhang, Y. Chen, et al. Learning safe multi-agent control with decentralized neural barrier certificates. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. Reviewen,, 2021.
[69] Qin, Z., D. Sun, C. Fan. Sablas: Learning safe control for black-box dynamical systems. IEEE Robotics Autom. Lett., 7(2): (12):1928-1935 2022.
[70] Bui, M., M. Lu, R. Hojabr, et al. Real-time hamilton-jacobi reachability analysis of autonomous system with FPGA. In IEEE/RSI International Conference on Intelligent Robots and Systems, IROS 2021, Prague, Czech Republic, September 27 Oct. 1, 2021, I, pages 1666-1673. IEEE, 2021.
[71] Ames, A. J. W. Grizzle, P. Tabuada. Control barrier function based quadratic programs with application to adaptive cruise control. 53rd IEEE Conference on Decision and Control, CDC 2014, Los Angeles, CA, USA. December 15-17, 2014, pages
IEEE, 2014.
[72] Ames, A. D., X. Xu, J. W. Grizzle, et al. Control barrier function based quadratic programs for safety critical systems. IEEE Trans. Autom. Control., 62(8) 2017.
[73] Xu, X., P. Tabuada, J. W. Grizzle, al. Robustness of control barrier functions for safety critical control. CoRR, abs/1612.01154, 2016.
[74] Xu, X., T. Waters, D. Pickem, al. Realizing simultancous lane keeping and adaptive speed regulation on accessible mobile robot testbeds. In IEEE Conference on Control Technology and Applications, CCTA 2017, Mauna Lani Resort, HI, USA, August 27-30, 2017, pages 1769-1775 IEEE, 2017.
[75] Borrmann, U., L. Wang. A. D. Ames, et al. Control barrier certificates for safe swarm behavior. In M. Egerstedt, Y. Wardi, eds., 5th IFAC Conference Analysis and Design of Hybrid Systems, ADHS 2015, Atlanta, GA, USA, October 14-16, 2015, vol. 48 of IFAC-PapersOnline, pages 68 78-73. Elsevier, 2015.
[76] Wu, G., K. Sreenath. Safety-critical control of planar quadrotor. In 2016 American Control Conference, ACC 2016, Boston, MA, USA, July 6-8, 2016, pages 2252-2255 -258.. IEEE, 2016.
[77] Walsh, T., S. Goschin, M. Littman. Integrating sample-based planning and model-based reinforcement learning. In Proceedings of the aaai conference on artificial intelligence, vol. 24, pages 612-617. 2010.
[78] Karaman, S., E. Frazzoli. Sampling-based algorithms for optimal motion planning. The international journal of robotics research, 30(7):846 894, 2011.
[79] Coulom, R. Efficient selectivity and backup operators in monte-carlo tree search. In Interna- tional conference on computers and games, pages 72-83. Springer, 2006.
[80] Vinod, A. P., S. Safaoui, A. Chakrabarty, et al. Safe multi-agent motion planning via filtered reinforcement learning. In 2022 International Conference on Robotics and Automation, ICRA 2022, Philadelphia, PA, USA, May 23-27, 2022, pages 7270-7276. IEEE, 2022.
[81] Silver, D., T. Hubert, J. Schrittwieser, et al. general reinforcement learning algorithm that 2018. [82] Couêtoux, A., J.-B. Hoock, N. Sokolovska, et al. Continuous upper confidence trees. In Learning and Intelligent Optimization: 5th International Conference, LION 5, Rome, Italy, January 17-21, 2011. Selected Papers 5, pages 433-445. Springer. 2011.
masters chess, shogi, and go through self-play. Science, 362(6419):
[83] Auer, P. Using confidence bounds for exploitation-explorrtion trade-offs. Journal of Machine Learning Research, 39N0v2397--422, 2002.
[84] Brockman, G., Cheung, Pettersson, et al. Openai gym. arXiv preprint arXiv: 1eee::66660000 04440, 2016.
14

---

[85] Achiam, J., A. Ray, D. Amodei. Safety gym. http:///pppaiiiicorrrrrrrrrraaa/ gym. 2019. Accessed: Nov 27, 2023.
15

---

A Supplemental Materials
In this section, we provide supplemental material complementary to the main content of the paper. This includes a complete proof for Theorem 5.1 and additional details omitted from section 6 due to space constraints.
A.1 Proof of Theorem 5.1
Throughout this section, we denote Q, Q", and V to be the agent Q function, the optimal Q function, and the optimal V function respectively.
Our planner is parameterized by fixed plan depth n and sampling limit m. The planner is tasked with finding actions d0m optimize the objective JQ So: (o:n) -E 7²
Rs) ( an)- Let cm be the supremum of JQ( do:n) ( anm) over S, where ag.n planRec(s) and 4on is the optimal sequence of actions with respect to the .qq objective.
We assume that our planner is probabilistically complete and asymptotically optimal. In context, with probability With this grounding established, we state the extended version of Theorem 5.1.
this means that n Cm
Theorem A.1. With probability I, we have:
1. Under the assumption that Q e-close to Q", we have limm--400 RR(Fbackup Tdmpss) Oeer").
2. Under the assumptions that the maximum reward per timestep is bounded, that S x A is compact, and that Q is continuous function, have lit0m-000 RR bockups Tempss) ).
RRR(TTockkpp
3. Under the assumptions that Q and Q* are both Lipschitz continuous, that (T(ss,)) - all2 is bounded over all (s, a) A, and that the initial state space Sy and the action space A are both bounded, we have limm-+20 RR ZZR(Thacenn backogs Temmps) Onn)").
We first show the following general lemma.
Lemma A.2. Fix a state E S, and let a = Tbacku(s)
(s) ) be the first action returned by planRec. Let Sn be the set of all states reachable from 8 in steps, and let = su a' 1Q' Then, 0 < (*) (*(ssa) $ 2B + Cm-
-('s' a'))
Proof. The bound (s) (s, a) 0 follows trivially from the definitions of V and Q* We now prove the upper bound.
Denote do.n as the sequence of actions returned by the planner. Note that ag. Denote a the sequence of actions that are optimal with respect to true infinite-horizon return from s (not necessarily with respect to JQ)-
We allow Ro:n-1 and R be the reward sequences attained by rolling out ao:--1 and 0p: respectively. Similarly, we allow 8n and 8 to be the states reached at the end of executing a0:n-1, " and d respectively.
First, we have
(s)
Σx'R: Re + *0
This follows immediately from the definition of Q" and the fact that a = ag- Since (sn, an) - Q(nn, an) S B, have
ErtR, + +-" an) 2
Σ T + ((((( By"
1=0
16

---

[q(s, do:n) - By
We know that don optimizes JQ to within tolerance of em, can write Jo (s, 0o:n) 2
Jo (s, (o:n) em- We get
JQ(s, (s, 00:)
By 2 Jo(s, ap.) - -m BY
Σ
7 R
7" Q" anj) - cm - By
Invoking again IQ" an) - ((5 az))
S B, we get
R-1 Σ i=D
n-1
R
+ +>" - -m -By" 2
Σ
R
+ +~"Q"(" o,2) - -mm - -BB
i==
=V*(s) - 2B
Putting everything together gives us (s 2 V (s) - c 2By". Rearranging this gives us the desired claim.
With this, we can begin proving the main theorem.
Lemma A.3. (Part of Thm A.1) Under the assumption that Q is e-close to Q", we have limm +20 RR( TTakuppp To dmps) = (eer") with probability 1.
Proof. In the context of Lemma A.I., we see that B S 6. Thus, we conclude that for all states
S, we have V*(s) Q' (s, [s)p 20 + cm Since RR is simply an expectation of
(*())-Q" (", "uuckup (s)) over some state distribution over S we can conclude RR( backup: Tdmps) S
2cy" " Cm- We know from asymptotic optimality assumption that cm decays to 0 as m goes to infinity,
get limm RR( buckupp Tdmps) O(er)).
L
Lemma A.. (Part 2 of Thm A.l) Under the assumption that the maximum reward per timestep is bounded, that S x A is compact, and that Q is a continuous function, we have limm-+0o RR( TT backups Tdmpps O(7") with probability 1.
Proof. It suffices show that (s, a) Q(s, a)) is bounded over all (s,a) S A. We can then simply apply Lemma A.3 to get the O()²) bound.
It is known that continuous function with compact domain has maximum and minimum value. Thus, there exists some Co such that [Q(s, ()s < Cq for all (s, a) E S x A.
Pick constant I such that the absolute value of the reward at any timestep is less than R. We see R and greater than (- = - I-y letting Co I we get [Q" a)l Cq over all (s, ESXA.
that Q*(s, a) is less than Ee= Ryd
-R Consequently,
With this, (Q" a) (ss, a)l < (s, +Q)ss a)l Cq + CQo over all (s, S A. This shows the boundedness we needed to attain the asymptotic bound.
Lemma A.5. (Part 3 of Thm A.1) Under the assumptions that Q and Q* are both Lipschitz continuous, that [[T a) s|l2 is bounded over all (s. a)) E S x A, and that the initial state space So and the action space A are both bounded, we have limm-+oo R.R backup: T. dmmps) = O(nor) with probability 1.
Proof. Suppose that both Q and Q" are K-Lipschitz continuous. Let d be the diameter of A. Let dy be the supremum of [s, a) sl|a over (s. a) E S A. We fix some arbitrary s E Sg and a E A. Intuitively, this will act as a "central" point upon which we can estimate values for other states. Let D (Q(s,a) (s, a)]. Let do be the radius of So when centered
We first prove subclaim. Take an arbitrary s' S, and d be the &z distance between s and Then,
17

---

V-('") (" bockuu (6')) S 27 D 2K v d2 + (d + nd cm.
The idea is to invoke Lemma A.1. To do so, we need to attain bound on B as defined by the Lemma. Consider state s" that is steps away from s'. The Triangle Inequality lets us bound the distance between s' and s" to dr. A second application of the Triangle Inequality lets us bound the distance between and s" d + dr. We can further use the boundedness of the action space to note that the maximum distance between the stteeactiion pair (s. and (s", a") for arbitrary a" E is d" + (s, Q* (s", d')l S Kd" and (((saa) - Q(s", "a") S Kd". With this,
+ (d [[++ndy )2 Finally, use Lischpitz continuity to get
IQ"(d" - Q(s",
$ "") -
+ 1Q(s" d²) - Q(s a) + (Q) Q(s,a)
S 2Kdd"= D
Using this as bound on and invoking Lemma A.I. demonstrates the subclaim.
Now, consider some arbitrary trajectory 80, 40, 81, d2,..... 1,-1, 1, 8k, with 80 E Sq and some fixed non-negative integer k. Via repeated application of the Triangle Inequality, one can bound the distance between s and to at most do + kdy. Invoking the subclaim, we see that
[D+2K 2K /dg (do + kdy ndd)2) + Em
(*x))) - Q* Taakkup (sx)) S 27
We can repeatedly use the fact that k + 2 to clean up the bound here. Namely, write
D D+2K V L + (do + kdy + ndT)²
< D(k + n) + 2K / m2 + ((k: + n)ddo + kdy + ndr
mk)
D+ 2K V & (do d7)2
Setting C = DD + (do + d we can sum up the equations above by writing
(***)) Q (8k. Taackup (sx)) S 20 + n) + 6m-
Now, we can finally bound RR. Let aa be the discounted state visitation distribution. We define Tauss to be the state visitation distribution at timestep t. We get
Pr.
RR RR( backkup Tdmpp)
= Eggrg" "ttrpp" [backup (V*(s) (x, Backup (8)))
1 ES "0omppane Tamps (V) Q (s)]
1 (s) - Q' Thackup (Tm
2
(s) - Qs, क Thuckup (s)] (1
*
Ps
Tame
(s) ds
km
(1 7i / /IV*(S) Q²* Toackup)
1P Tanes (s)ds
K=U
20
Σ (1 --)))
Tangs (V) (s, Thackup (s))
simpy
k=U
18

---

Algorithm 2 Monte Carlo Tree Search (planRec)
1: Inputs: (#1: Start State), (Q: state-action function)
2: Hyper-parameters: (K: Branching, Factor), (F: ltemtion Count), (T.: Maximum Path Length)
3: Q = initilastnap(S x A RR) Initialize emply data structure for state-action value estimates.
4: : initHashhappsS x
N) Initialize emply data structure for state action visit counts. sig expand(#1 T, N) Attempe expanding the root with actions leading recoverable states. 8: E [1,
5: T = initTree(S1) # Initialize
with
6:
7: if -res then return 1 # Return 1 if the root state cannot be expanded.
9: # #Solect path of length less than L from the root using the Upper Confidence Bound (UCB) formula.
10:
(#1:a1, Ri)...., (Ar 1r 1. R2 1), 8, selectPachtcc(( I.)
11:
(s then
12:
expand(#r. Q, N) Expand leaf state at the cad the path with actions leading to recoverable states.
13: a. = selectAction(er. Q) # Choose the best oungoing action from o, with respect to Q.
14: for l
15:
Update the sinte action estimates with rewards collected on the selected path.
16:
Qis.. = N( [N(s,, a.) (e,, a.) (E)=l 72- R3) +-- -0-"" Q(s..a:))
17: for E do
18:
((s):a)) :m (s);; + Increment state
count for selected path.
19: (1): (#2: 02).... (8x, (x)) :m Pathhredad (Q) Q greedily select the oprimal path from root to leaf.
20:
return (0 (n)) Retum the actions on the groedily selected path.
Algorithm 3 Sampling-Basse -aaed Tree Expansion (expand)
1: Inputs: (s: Expanding Stute), (T:; State-Action Tree). (Q: Local Value Estimate), (Q: Long term Value Function), (N: Visit Count)
2: #i..... aK ~A A Sample K actions.
3: success == Falee
4: for [1,
5: if (+,, E Seec then
6:
(() == Q(+ as) # Initialize value estimate for (s.
7: [ss.a,)
Record first visit for (e, a.) pair.
8:
updateTree a.) Add (s, the sucness zm True Record at least one recoveralle action was found.
9:
10:
return 8u0cess
Σ
(1
7h 120 m) + Gml
k=U
30
20(1 7)7"
th +n)
Σ
(1 7)
km0
km0
=0 - 7)?""
koy
+n
+ Em
k=0
We can evaluate the titthmeiicgeommette series Σx= ly equal
(1-7)² The geometric sequence
Ei * evaluates - Thus, our becomes
7 + 71 20 - ->7" (1 (1- 1-
+ 6m = O(ngm) + 6m-
Due to asymptotic optimality, taking the limit of this as m goes to infinity gives a bound of O(n7")
A.2 Monte Carlo Tree Search
The planning function pl planRec used in DMPS is realized via Monte Carlo Tree Search (MCTS) [79]. which facilitates efficient online planning by balancing exploration and exploitation. MCTS has been shown to be highly effective for planning in large search spaces with either discrete [52, 81, 51] or continuous [54, 57, 82] states and actions. DMPS utilizes a continuous variant MCTS that employs sampling-based methods to manage the exponential increase in search space size as the planning horizon expands [53]. A high-level overview of this approach follows. Algorithms and present implementation of MCTS.
19

---

*
(a) gate
(b) double-ged double-gates
(c) double-ger eagttee-
Figure 6: Visualization of the dynamic environments. The agent is depicted as a red circle. The direction of rotation the walls is shown with red arrows. The goal position is shown with *.
Starting from given state So the root node, the function planRec maintains and iteratively expands a tree structure, with states as nodes and actions as edges. During each iteration, the algorithm selects path from the root to leaf node. Upon reaching leaf node, it extends the tree by sampling number (k) of actions from the leaf, thereby adding new nodes and edges to the tree. The path selection process is based on the Upper Confidence Bound (UCB) formula [83], which utilizes an internal representation of the state-action values at each tree node (denoted by Q Q::. x A R), to balance exploration of less frequently visited paths with exploitation of paths known to yield high returns. Once path is selected, the algorithm updates the value of Q for each node on the selected path by backpropagating the accumulated rewards from subsequent nodes. These updates are averaged by each state-action's visit count to achieve more precise estimates.
After a specified number of iterations (/), the algorithm uses Q to greedily select an optimal path from the root leaf node and returns the corresponding sequence of actions as its final result.
A.3
Additional Experimental Results
A.3.1 Benchmarks
A detailed description of benchmarks used in our evaluation is presented below:
This benchmark involves a robot moving on a 2D plane, aiming to reach a goal position without colliding with a stationary obstacle. This obstacle is positioned to the side, impacting the robot only during its exploration phase and not on the direct, shortest route to the goal.
obstacle:
obstacle2: This benchmark involves robot moving a 2D plane, aiming reach goal position without colliding with a stationary obstacle. This obstacle is positioned between the starting point and the goal, requiring the robot to learn how to maneuver around it.
mount-car: This benchmark requires moving underpowered vehicle up hill without falling off the other side.
road: This benchmark requires controlling an autonomous car to move one dimension to specified end position while obeying a given speed limit.
road2d: This benchmark requires controlling autonomous car to move in two dimensions to a specified end position while obeying a given speed limit.
dynamic-obs: This benchmark features multiple non-stationary obstacles that move deter- ministically in small circles on the path from the agent to the goal.
single gate: In this benchmark, the goal position is surrounded by a circular wall with small opening, allowing the agent to pass through. The position the opening continuously rotates around the goal position. This is visualized in Fig. 6a.
double-gate -gaee:::: This benchmark is similar to single-gast e-gate; however, the goal position is surrounded by two concentric rotating walls. This is visualized in Fig. 6b.
double -ablle-gttet::: This benchmark similar double -gates; however, the thickness of the rotating walls is increased. This poses greater challenge, the agent has shorter
20

---

time window to cross through the opening without colliding with the rotating wall. This is visualized in Fig. 6c.
For the dynamic benchmarks above (dynamic-obs single-gate, double-gate, and double-gate+y the action space of the double integrator agent consists of acceleration in the r and y directions. The action space for the differential drive agent consists of the torque value on the left wheel and the torque value on the right wheel.
Additionally, the observation space of benchmarks with a double integrator agent consists of the position and velocity vectors of the agent, while for a differential drive agent, the observation space includes position, velocity, and pose angle. The observation space for single-gat -gate, double -gatt, and double e--gte+++++ also contains an angle term for each wall, corresponding to the current rotation of each wall. The observation space in dynami obs additionally includes an angle term for each obstacle, indicating how far each obstacle has moved along its circular trajectory.
A.3.2 Implementation
We implemented DMPS by modifying the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm [15], following the description presented in Algorithm 1.
Each of the dynamic benchmarks were trained for 200,000 timesteps with maximum episode length of 500. The static environments were trained for the number of timesteps prescribed by the sources of the environments. Namely, mount-car car was trained for 200,000 timesteps with a maximum episode length of 999 [84], obstacle and obstacle2 were trained for 400,000 timesteps with a maximum episode length of 200 [16], and road and road2d were trained for 100,000 timesteps with a maximum episode length of 100 [16]. Each experiment was run on five independent seeds. We used prior implementations of REVEL [16], PPO-Lag [85], and CPO [85] to run our experiments. Trained models were test evaluated every 10K timesteps, independently from the training loop. Each test evaluation consisted of 10 runs. The final test evaluation was used to determine the average per-episode return and per-episode shield invocation rate for that random seed. These are the values displayed in the tables from section 6.
The reward functions used in [16] for the static environments are non-standard, and based upon a cost-based framework of reward. In light of this, we used the canonical reward functions for our evaluations. For mount-car, we use the original reward function defined in [84]. For obstacle, obstacle2, road, and road2d, we use the canonical goal environment shaped reward function.
The negative penalty for safety violations in TD3 was taken to be large enough so that the agent could not move through obstacles and still maintain positive reward. In most cases, the penalty was simply the negation of the positive reward incurred upon successfully completing the environment. The episode did not terminate upon the first unsafe action. For CPO, we reduced tolerance for safety violations by reducing the parameter for number of acceptable violations to
Our experiments were conducted @2.30GHz, 264GB of available memory, and eight NVIDIA GeForce RTX 2080 Ti GPUs. Each benchmark ran on one CPU thread and on one GPU,
server with 64 available Intel Xeon Gold 5218 CPUs
A.4 Analysis of Computational Overhead
As mentioned in the main text, our implementation uses MCTS in an anytime planning setting, where we devote a fixed number of node expansions to searching for a plan, and we choose the best plan found at the end of the procedure. This bounds the clock time needed for planning lincarly in the compute budget allocated. However, in the worst case, may need compute budget exponential in size with respect to the planning horizon H if one wishes to explore the planning state densely. This is common to all general planners.
Here, we try to experimentally evaluate how the required compute budget scales as a function of the planning horizon. We re-evaluate the double + environment (under double integrator dynamics). For each planning horizon H range [2, 9), we count the average number of node expansions that MCTS needs before it successfully explores 10 states at depth H. We use this as proxy for successful exploration of the horizon H search space. The results of this experiment can be found in Figure 7. As expected, exponential relationship emerges.
21

---

10a
Node Expansions (Log Scale)
102
10
100
3
5
8
Horizon
Figure Planner Computation Scaling
14
160
12
140
10
120
Shild
100
80
4-
40
122
H49
2.5
5.0
7.3
10.0
12.5
13.0
17.5
20.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
Timesteps [10K]
Timesteps (10K)
Figure 8: Multiple Horizons Experiments
A.5
Effect of Planning Horizon
We expand on the question of whether small planning horizons are sufficient to solve tricky planning problems. Our algorithm is designed to ensure that the planner accounts for both short-term and long-term objectives. As detailed in section 5, the objective of the planner consists of two terms: 1) the discounted sum of the first n rewards, and 2) the estimated long-term value of the penultimate step in the plan horizon, determined by the agent's 's Q function. The second part of the objective function is specifically included to ensure that the planner does not return myopic plans, and accounts for progress towards the long-horizon goal. Since the planner optimization objective includes this second term, even a small-horizon planner can output actions with proper awareness of long-horizon recovery events. The length of the horizon affects how close to the globally optimal solution the result is, with tradeoff of computational cost, as was established in Theorem 1.
To this empirically, reevaluated the double -ouble-gates++ environment (double integrator dynamics) with horizons of 5, and 9. The graphs of attained reward and shield triggers from this experiment shown in Figure 8.
Comparing the agent in both shield triggers and reward. Comparing H = 5 and H 9, the H agent reached high performance and low shield triggers faster than the = 5 agent. However, both the H 5 and Il agents converge to the same performance eventually.
and
5 agents, the
agent substantially outperforms the H
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claim in this paper is that MPS, state-of-the-ar framework for Provably Safe RL, has significant shortcoming due to its inability to recover from poten- tially unsafe situations while making task progress. We introduce solution for this problem and claim that adequately resolves the shortcoming of MPS. Throughout the paper, we extensively elaborate the nature of MPS's shortcoming. We present solution based on a local planner and provide a theoretical analysis explaining the asymptotic optimality of our solution. Our experiments validate performance compared to MPS. We also show that our approach is superior to several other well-known SRL and PSRL methods.
claims that our approach achieves superior
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We present discussion of the limitations of our approach in section 8. In summary, our approach requires an accurate world model, and the computational overhead of planning increases exponentially with the planning horizon.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
23

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: In this paper, we present make theoretical contribution, which is presented in Theorem 5.1. We have included a complete, correct and detailed proof for this theorem in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We formally define all objectives solved by our solution and present detailed algorithms implementing the main approach in Algorithms 1, 2, and 3. Our submission also includes the source code of our implementations and scripts for reproducing the results. We also plan to create open-source repository and make our implementation publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
24

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b)
If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which case authors are welcome to deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described supplemental material?
Answer: [Yes]
Justification: We provide the code necessary to run our own method (DMPS) and MPS, as well as implementations of the environments on which the algorithms were run. All other baselines are publically available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: wweedeeeeeeeeeiiiiieeeeeeeeeeeee
//nips. wwcpppcccc/eeeeecccccccccceeeeee
sesissssssspioioooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and basclines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Appendix A.3.2 we present the specifics of our implementation and experi- mental setup in detail. This includes the hyperparameters, the training and test set details, the number of runs, and the hardware used for experimentation.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
25

---

The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The statistical significance of all presented empirical results is justified. In particular, we include standard deviation values for all results presented in Table and Table 2. Additionally, we include confidence belts in all plots presented in Figure 3 and
Figure 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We present the details of the hardware and other resources used for experiments in Appendix A.3.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: htt///nnuiip.er/ cetpurpps.ccppulllllee http::/........lcctcac.cc.uaaaaa
Answer: [Yes]
26

---

We have reviewed the NeurIPS Code of Ethics carefully and affirm that our work adheres all listed requirements.
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: To the best of our knowledge, work cannot be misused in any way cause any form negative social impact. We have included a discussion of the potentially positive social impacts of our work in section 8.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
To the best of our knowledge, our work does not pose a risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided relcasing unsafe images.
27

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have only used publicly available open-source libraries in our implementa- tion and credited all third-party sources accordingly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release any new assets with this paper.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
28

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "bUuTH1ZONd", "reviewer": "Reviewer_BFvm", "summary": "The paper seeks to address provably safe RL problems where safety must be ensured even during training. It proposed DMPS, which enhances prior Model Predictive Shielding approach, to dynamically select safe actions when danger is imminent. DMPS employs local planner to plan for recovery actions and the planner objective consists of both short-term and long-term rewards. Feedback from the planner can then be used to incrementally train the neural policy to guide it towards safe policy set.", "questions": "1. In the first sentence of Section 6.2, you mentioned that the total return is averaged over the last 10 training episodes. Are you evaluating them using the same (single) random seed and only use the last 10 training episodes for evaluation?  \n\n2. Given that both the states and actions are continuous, how do you apply MCTS to the local planner?  \n\n3. TD3 only maximizes a single reward objective. In your experiments, I guess you performed some sort of reward shaping for it to balance between safety and reward. Can you elaborate how do you incorporate safety into its objective and is there any weighting used?  \n\n4. Similarly for CPO, how do you incorporate safety into it? Do you specify a safety violation constraint?  \n\n5. (Related to Qn 3 & 4) I am surprised that TD3 and CPO rapidly overfits to conservative policy in dynamic environment. What do you think is the reason and is the weighting between safety and reward dynamically tuned?", "limitations": "The paper discussed the known environment model as its limitation. I agree that this is a limitation which warrants further investigation. As studied in [5], it is very challenging for a model to accurately predict future trajectories with long horizon. Since DMPS relies on having an accurate environment model for safety adherence, this may limit its applicability to practical scenarios where environment model is not given and needs to be learned.  \n\nAnother related point is that it is unclear what value to set for the local planner horizon. Different tasks may require corrective action sequence of different lengths. Setting the horizon too short may revert DMPS performance back to MPS and setting the horizon too long may increase the computational overhead beyond acceptable threshold.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Quality\n* Overall, the approach described in the paper is sound and it combines many established components (e.g. backup policy, local planner, estimate future reward using model unrolling and Q-estimate) to facilitate safe RL.  \n* The paper provides theoretical bound on the recovery regret as the sampling limit in the local planner approaches inifinity.\n2. Clarity\n* The paper is written in a clear and lucid manner. The figures, algorithm and equations are structured in a way that is easily understandable to the readers.", "weaknesses": "1. Originality\n*  The main difference between DMPS and MPS is the use of local planner when backup policy is triggered. The technical approach used in DMPS is not particularly new as there are already some similar approaches of estimating a safety Q-value and perform planning based on the Q-value [1, 2].\n2. Significance\n* The only difference between DMPS and the prior MPS seems to be the local planner and (as discussed in point 1) this local planner is not particularly novel. Having said that, I do agree that the proposed DMPS does show improvement over MPS in some experiment scenarios.\n* While the paper mentions a small planning horizon is sufficient for the local planner to plan safe yet rewarding actions, I feel that this may not be true in most cases. To steer the agent back to safety (and yet rewarding), a long sequence of actions may be required. If the planning horizon is set too small, then DMPS falls back to backup policy and the performance would be the same as MPS. In this case, I guess the only solution is to increase in planning horizon and in turn increase the computational overhead of DMPS exponentially.\n* The local planner requires perfect information of the transition and the transition must be deterministic. This may restrict its applicability, especially given that there're prior work on model-based RL where transition can be stochastic and learned instead.  \n\nReferences  \n[1] Clavera, I., Fu, Y. and Abbeel, P., Model-Augmented Actor-Critic: Backpropagating through Paths. In International Conference on Learning Representations.  \n[2] Thomas, G., Luo, Y. and Ma, T., 2021. Safe reinforcement learning by imagining the near future. Advances in Neural Information Processing Systems, 34, pp.13859-13869.  \n[3] Nagabandi, A., Kahn, G., Fearing, R.S. and Levine, S., 2018, May. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE international conference on robotics and automation (ICRA) (pp. 7559-7566). IEEE.  \n[4] Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems. 2018.  \n[5] Janner, M., Fu, J., Zhang, M. and Levine, S., 2019. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their many insightful comments and suggestions. We respond to their questions and concerns below.\n\n**Concerns over originality and significance of DMPS**\n\nThe main novelty of our approach is the synergistic relationship between a local planner tasked with finding good recovery paths around unsafe regions and a neural policy that can learn from the planner's recovery strategies to navigate unsafe regions on its own. We thank the reviewer for highlighting the related papers (which we will cite in the revision), but note that both papers take quite different approaches from DMPS: \n\nThe first paper does not deal with safe navigation, focusing instead on performance optimization in general RL settings. In that work, the training of the policy is disconnected from the planner; they first train a policy using their MAAC algorithm and directly insert it into an MPC planner. However, in the PSRL setting, adding a planner after the fact limits the performance of the policy since it was not trained to avoid safety-violating scenarios while simultaneously optimizing the task objective. The synergistic relationship between planner and policy in our approach avoids this problem.\n\nThe second paper deals with safe navigation, but it does not use planning, instead taking a model-based RL approach. The key insights of this paper are dynamically adjusting the negative penalty for safety violations such that the penalty can \"carry over\" a discounted horizon, and using a model to synthetically generate more rollouts in training. \n\n**Reporting of average return in the experimental section**\n\nThe mean is computed with randomized seeds. We take five random seeds, and for each random seed, we compute the mean return over the last ten training episodes. We then report the average score over the five random seeds. The standard deviation reported is across the seeds, not the episodes. We will clarify this in the text.\n\n**Using MCTS in continuous settings**\n\nWe apply MCTS to the continuous domain following a strategy similar to previous works such as [2,3]. We deal with a continuous action space having a branch factor hyperparameter $B$, and then sampling $B$ continuous actions from the action space for each node expansion.\n\n**Reward shaping in TD3 to incorporate safety priorities** \n\nWe tested two variants: one where TD3 received a negative reward penalty and ended the episode upon the first unsafe action, and another where each unsafe action incurred a penalty. The former led to frequent crashes, so we chose the latter. We set the penalty magnitude equal to the positive reward for completing the environment. Hyperparameter tuning showed that reducing the penalty too much led the policy to ignore safety constraints, while beyond a certain point, the penalty magnitude had little effect on behavior.\n\n**CPO safety parameters**\n\nCPO has a hyperparameter specifying the maximum tolerable number of safety collisions. Given our goal is provably safe RL, we set this parameter to 1 following the methodology of REVEL, [1]. We also tested higher tolerance values to ensure CPO wasn't hindered by the safety constraint but saw no performance improvement, so we maintained our original setting. \n\n**Overfitting of TD3 and CPO**\n\nIn dynamic environments, reaching the goal requires safely avoiding obstacles. Hyperparameter tuning showed that low penalties for unsafe actions lead agents to tolerate collisions, while high penalties make them overly conservative. This indicates that avoiding obstacles while simultaneously achieving task goals is complex and can't be learned through reward signals alone.\n\nWe did not use dynamic tuning. In CPO, the safety constraint is separate from the reward, so dynamic tuning was not possible. In TD3, changing the reward function mid-training is theoretically unsound and would likely destabilize the learned Q function, as it minimizes loss using a replay buffer with rewards spanning 10^6 timesteps.\n\n[1] Anderson, G., Verma, A., Dillig, I., & Chaudhuri, S. (2020). Neurosymbolic Reinforcement Learning with Formally Verified Exploration. NeurIPS 2020\n\n[2] Hubert, T., Schrittwieser, J., Antonoglou, I., Barekatain, M., Schmitt, S., & Silver, D. (2021). Learning and Planning in Complex Action Spaces. ICML 2021\n\n[3] Rajamäki, J., & Hämäläinen, P. (2019). Continuous Control Monte Carlo Tree Search Informed by Multiple Experts. IEEE Trans. Vis. Comput. Graph., 25(8), 2540–2553.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xVqcTwIpOt", "reviewer": "Reviewer_xs2Y", "summary": "The approach called dynamic model-predictive shielding for safe reinforcement learning is proposed as an improvement over its static counterpart. The main idea is to optimize for expected return on action with respect to the reinforcement-learning task when choosing a shielding backup action, and to incorporate planning horizon prediction into learning for the policy to learn to avoid unsafe actions. This dynamic version is evaluated on several static and dynamic obstacle-avoidance benchmarks and compared to static model-predictive shielding and three more planning-based approaches.", "questions": "Questions\n1. In Figure 1, what are green and red lines, and a blue blob?\n2. How does the local planner scale with respect to the look-ahead?\n3. Does the local planner have to recompute the look-ahead prediction every time it is invoked or does it reuse previous results if the agent continues along the same trajectory?\n4. What is the overhead of the planner's computations?\n5. How does the planning limit affect safety and optimality guarantees?\n6. MCTS typically struggles to plan for overly constrained problems and complex planning tasks. How does the approach scale with respect to the planning task complexity?", "limitations": "The authors explicitly discuss limitations which are fairly common for problem domain.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The core idea of the approach is interesting and potentially valuable: to achieve synergy between safety and optimal performance in model-predictive shielding via incorporating planning into policy learning and taking expected performance into account during backup planning. Similar attempts have been done previously. In comparison, this work proposes a novel notion of \"recovery regret\" as a heuristic to guide mutual integration of planning and reinforcement learning. \n\nThe strength of the paper is in extensive evaluation and comparison to multiple approaches. The notion of recovery regret can also be of independent interest for model-predictive shielding research. Dynamic shielding outperforms other approaches in the evaluation in terms of the number of shielding invocations, which indicates synergy between planning and learning over time.", "weaknesses": "Potential weaknesses of the approach are in scalability of the planner and tightness of the probabilistic bounds on safety.\n\nMinor:\n- \"more optimal recovery approach\" --> an optimal/a better", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their many insightful comments and suggestions. We respond to their questions and concerns below.\n\n**Figure 2 clarification**\n\nWe assume that the reviewer’s question is referring to Figure 2. Figure 2 demonstrates the example described in the text, and in particular, part (d) of the figure is discussed in Section 5 (Lines 208-215). The red and green lines represent two locally optimal paths in the planning tree. The blue blob represents a water puddle that would result in lower returns if crossed. The planner selects the green planner here, informed by its access to the long-term $Q$ function.\n\n**Reusing lookahead predictions in the local planner**\n\nThe planner is only used when the neural policy suggests an action that cannot be certified as recoverable. Reusing prior results is beneficial only if the planner is invoked consecutively due to the base policy acting recklessly. In such cases, previous search results can be reused, similar to sub-tree reuse in the POMCP algorithm [2]. Our current implementation does not include this feature, but it can be easily added.\n\n**Planner overhead**\n\nBeyond the computational cost of running the planner (discussed above), there is minimal overhead in our implementation. To minimize the overhead, we designed the planner to leverage the environment's simulated transition function so that the state and action space between the planning problem and the RL formulation is identically the same. Thus, when DMPS needs to invoke the planner, it just initializes a planning environment with the current world state, and the planner directly queries the transition function internally during its search.\n\n**Effect of planning limit on safety and optimality guarantees**\n\nWhen using a planner with proven completeness (or probabilistic completeness) properties, DMPS can always guarantee a safe recovery plan, given enough planning time. However, due to finite computational limits, the planner might not find a solution in a fixed time. Thus, DMPS can rely on its fallback policy when the planner fails to provide a valid solution (branch 4 in Figure 1). As such, the incorporation of a planner in DMPS does not affect the provable safety guarantees. In practice, we find that DMPS never uses the fallback policy in our experimental settings.\n\nThe relationship between the optimality guarantee and the planning depth is outlined in Theorem 5.1, with more specific statements and proofs given in Appendix A.1. The planner’s regret decays exponentially as a function of the planning depth.\n\n**Scaling of approach with respect to planning task complexity**\n\nAs the reviewer noted, MCTS struggles with overly constrained problems, making the choice of planner crucial. We used MCTS in our evaluation due to its wide applicability, but some domains may have more effective planners. For instance, Informed RRT* excels at motion planning in constrained environments with dense clusters [1].\n\n[1] Gammell, J. D., Srinivasa, S. S., & Barfoot, T. D. (2014). Informed RRT*: Optimal sampling-based path planning focused via direct sampling of an admissible ellipsoidal heuristic. 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2014, Chicago, IL, USA, September 14-18, 2014, 2997–3004. doi:10.1109/IROS.2014.6942976\n\n[2] Silver, D., & Veness, J. (2010). Monte-Carlo Planning in Large POMDPs. In J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, & A. Culotta (Eds.), Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010. Proceedings of a meeting held 6-9 December 2010, Vancouver, British Columbia, Canada (pp. 2164–2172). Retrieved from https://proceedings.neurips.cc/paper/2010/hash/edfbe1afcf9246bb0d40eb4d8027d90f-Abstract.html", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0DJo7Vrlze", "reviewer": "Reviewer_NRWz", "summary": "The authors introduce Dynamic Model Predictive Shielding (DMPS) an extension of Model Predictive Sheilding (MPS) that adress some of its key limitations, such as overconservatism when deploying the backup policy which consequently hinders exploration of the neural 'task' agent and slows down conergence. The key innovation of DMPS is that it incoropoates a local planner for dynamic recovery that leverages both a pre-computed backup policy and the neural 'task' policies Q values for planning for short and long horizon returns while maintaining safety of the system. DMPS is a provably safe reinforcement learning (PSRL) method, meaning that it guarantees safety of the system (regardless of the underlying neural 'task' policy) by only exploring within the set of safe and recoverable states defined by the backup policy.  This realised by planning for a fixed n step trajectory and checking whether the system can be recovered by the backup policy given the action proposed by the agent. The authors demonstrate that DMPS outperforms MPS and other baselines in terms of performance and safety in various benchmarks. It also emphasizes the importance of aligning reinforcement learning agents with real-world safety requirements, while discussing some of the limitations of their approach.", "questions": "Most of my questions are technical:\n\nFor each of the enviroments you consider how are the backup policies constructed and how are the invariat safe sets determined?\n\nFor each of the environments what are the maxmimum number of steps needed to deccelerate the vehicle to zero or avoid obstacles and is your choice of n=5 sufficient?\n\nWhat would be suitable ways of modelling the environment from experience to obtain uncertainty estimates, for example would Gaussian Process modelling suffice?\n\nDo you assume any observation noise or just perfect access to the current state, if not how would you incorporate this into your framework?", "limitations": "See weaknesses.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper has several strengths: I find that the paper is very well written and easy to follow, with sufficient details in necessary places and abstractions in other places where the details may not immediately matter, as such, it is a very nice read. The theoretical analysis of the recovery regret is convincing and interesting. Furthermore, the overall framework is attractive from the point of view that it is provably safe, something I personally find is crucial for deploying RL in the real world, rather than safe at convergence or in expectation like a lot safe RL methods. I find that the dynamic planning module is an innovative solution to the intuitive issue faced by most shielding methods (Figure 2) and I feel that this work constitutes a step in the right direction for improving shielding methods and making them more practical. The experimental evaluation I feel is strong and thorough as in most cases DMPS clearly outperforms MPS and REVEL, although I think it is missing something (see weaknesses).", "weaknesses": "The key weakness of the PSRL framework is the reliance on a perfect (or sufficiently accurate) dynamics model of the environment, the safety performance of the backup policy and the computation of the safe invariant set. In contrast to the first shielding approaches for RL [1], which operate primarly on discrete state and actions spaces, DMPS does not need to compute the shield before learning can begining which significantly reduces the engineering overhead before training. This of course comes at a cost, in practice the shields in [1] are substatially more lightweight during \"inference\", (although in theory there could be exponential blow up) in part due to only operating on discrete or discretized state/action spaces but also because a lot of the necessary computation is done before hand. This is a key limitation of DMPS as it relys on planning at each timestep which might be costly and infeasible for on-board computation or edge devices. Fruthermore, it seems that there is still a significant amount of prior knowledge required for DMPS to work effectively, first we have to have a \"perfect\" dynamics model (for provable guarantees) secondly I presume we need to handcraft a backup policy and then compute its invariant safe set so as to plan for recoverability. The first limitation is mentioned in the paper but not really discussed in much detail, the second limitation is find is crucial and I don't think is really mentioned in the paper. In particular it is a non-trivial challenge to come up with a backup policy that has a maximal safe invariant set, perhaps for the environments the authors consider it is easy (just decelerate) but for more dynamics environments and in general this is not the case and I feel like more discussion about both these limitations (i.e. the limitations of the PSRL setting) is needed. \n\nWhile I find the experimental evaluation compelling I feel it is slightly misleading and it is missing something. In Table 2 CPO and TD3 score the same or higher in a few of the static benchmarks but there scores are not in bold, is there a reason for this that I am missing? I also feel like a comparison to PPO-Lag or DDPG-Lag would really help make the results that bit more convincing.\n\nAll that being said, in principle I advocate for acceptance of this paper.\n\n[1] Alshiekh, Mohammed, et al. \"Safe reinforcement learning via shielding.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their many insightful comments and suggestions. We respond to their questions and concerns below.\n\n**Construction of backup policies and determination of the invariant sets**\n\nIn static environments, the backup policy involves braking as hard as possible. In dynamic environments, the agent checks if it is in an obstacle's trajectory; if so, it accelerates away, otherwise, it brakes. The backup policy is straightforward, aiming to halt the agent quickly. It can be determined through basic understanding of environment dynamics, or by training a neural backup policy.\n\nThe Model Predictive Shielding (MPS) framework automatically synthesizes the invariant safe set from the backup policy. The MPS framework defines “stable states” where safety is guaranteed. In static environments, stable states are those where the agent has zero velocity; in dynamic environments, the agent must have zero velocity and be outside the trajectory of all obstacles. A state s is within the invariant safe set ($S_{rec}$) if a trajectory from s to a stable state can be found by forward-simulating the backup policy without violating safety conditions.\n\nThe invariant set's construction is automatic, requiring no manual engineering, and is generally sufficient for most safety problems. States not within $S_{rec}$​ that could allow safe navigation are usually undesirable, as they are too close to obstacles for braking to be safe.\n\n**Maximum number of steps needed to decelerate the vehicle and sufficiency of n=5**\n\nThe planning horizon doesn't need to cover all steps to decelerate the vehicle to zero. Each state in MCTS expansions is verified to be in $S_{rec}$ through forward simulation, ensuring every state in the search is safe. This simulation isn't part of the planning horizon.\n\nFor context, it takes a maximum of 10 timesteps to decelerate to a stop in both the differential drive and double-integrator dynamics.\n\n**Suitable ways to model environment noise**\n\nWe assume the environment is deterministic with no observation noise. However, in RL, Gaussian modeling can effectively handle observational uncertainty, as shown in [2]. More complex models, like uncertainty-aware deep learning, have also been explored [1].\n\n**Experiments**\n\nWe thank the reviewer for their comments on the experimental results. We have included PPO-lag as a baseline and corrected the bolded numbers in the tables. The tables are attached in the global rebuttal.\n\n[1] Chua, K., Calandra, R., McAllister, R., & Levine, S. (2018). Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models. In S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, & R. Garnett (Eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada (pp. 4759–4770). Retrieved from https://proceedings.neurips.cc/paper/2018/hash/3de568f8597b94bda53149c7d7f5958c-Abstract.html\n\n[2] Kaufmann, E., Bauersfeld, L., Loquercio, A., Müller, M., Koltun, V., & Scaramuzza, D. (2023). Champion-level drone racing using deep reinforcement learning. Nat., 620(7976), 982–987. doi:10.1038/S41586-023-06419-4", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "etJk49jzQH", "reviewer": "Reviewer_jAJK", "summary": "This paper proposes a new method for safety shielding. More precisely, the authors extend Model Predictive Shielding (MPS), where an agent reverts to a safe backup policy if, for the next predicted state, this policy would not be able to guarantee safety anymore. MPS is often overly conservative, particularly in cases where the backup policy is very different from the final policy (for example, it may only consider breaking, while the final policy may be able to steer around an object). To improve this, whenever an action is potentially unsafe, the agent first uses a short-horizon planner to see if there exists some safe action that may be better than the one of the backup policy (i.e., one for which the backup policy could still recover in the future, but for which our learned agent predicts a higher reward). The authors formalize this framework and show recovery regret for this framework diminishes exponentially with the horizon. Next, they show that an implementation of this framework outperforms prior methods, both in terms of performance and the number of required shield invocations.", "questions": "As mentioned in the 'weaknesses' section, I have one main question: how does the computational complexity of your method compare to those of the benchmarks, particularly MPS? I will change my rating if this question is not adequately answered.", "limitations": "Limitations are adequately addressed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The topic of the paper, safety shielding, is relevant and significant. Safe RL (and particularly, safety shielding) is a promising line of research but is often overly conservative in practice: the methods proposed in this paper take a step toward reducing this problem while still giving formal guarantees about safety. The topic is relevant for the NeurIPS community (particularly those interested in RL), both as a method that could immediately be used or to extend the method to more complex settings (i.e., with a stochastic/unknown model).\n\nThe paper is well-written and easy to read: the intuition behind the method is clear, and the analysis of the results is easy to follow. The framework is well formalized (using visualizations where helpful), and the given pseudo-code helps with reproducibility. The experiments are extensive and convincingly show the advantages of the proposed method.", "weaknesses": "Apart from some minor remarks that I add below, this paper has one main weakness: it does not clearly indicate the computational complexity of its method nor the scalability. The results do not show computation times, and (as far as I could tell) no mention is made of either the average planning time or some time limit for this planning phase. From some ball-parking, the additional time required for this method may be significant (solving up to millions of short-horizon planning problems), so a quantification of this computational cost should be provided.\n\nSome more minor remarks:\n* The paper only mentions how the framework is implemented (i.e., what RL & planning method it uses) in the appendix: it would be nice to (briefly) mention this in the results section as well;\n* In Table 2, the results of CPO and TD3 are not bold, even though some are equal to those of the best frameworks: this should be fixed;\n* One limitation of the proposed framework is that it assumes the environment is deterministic: it would be nice to mention this in the limitations section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their many insightful comments and suggestions. We respond to their questions and concerns below.\n\n**Questions over computational complexity**\n\nWe analyze the question of the planner’s computational cost in more depth in the global rebuttal. We restate a short summary of the global response here.\n\nThere is a tradeoff between the quality of the recovery plan, and the computational cost incurred in the planner searching for it. The look-ahead controls this tradeoff. In our implementation, we use MCTS in an anytime planning setting, where we devote a fixed number of node expansions to searching for a plan, and we choose the best plan found at the end of the procedure. The clock time needed would simply be linear in the allocated compute budget. However, the worst-case computational complexity to densely explore the search space, as in general planning problems, would be O(exp(H)) where H is the look-ahead length, since the planning search space grows exponentially. Since the remaining baselines do not do any kind of planning, they use constant time per timestep, though this comes at an optimality cost.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sFToyBBhSi", "reviewer": "Reviewer_d437", "summary": "Naive model predictive shielding may overly restrict exploration thereby preventing an RL agent from learning a policy with good performance. In order to prevent this, the authors propose a method to optimise a backup policy that is provably safe using an online planner. An approximate model such as double integrator or differential drive is used for planning. Improvements are demonstrated on five benchmarks that involve static or dynamic obstacle avoidance as compared to provably safe and approximately safe RL methods. A provable guarantee is provided for recovery regret.", "questions": "Have the authors explored the space of RL training algorithms and methods to test this approach?\n\nAre there any advantages of using DMPS if the performance policy is not using RL and uses imitation learning or no learning at all? Exploration is important only for RL.", "limitations": "The authors have discussed limitations and there is potential for the approach to scale even though the experiments in the paper are on simple examples.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Presentation is clear with backing proofs and demonstrable results\n\nProblem that is being solved is clearly delineated and addressed using sound techniques\n\nExperimental comparisons are performed rigorously with attention to detail", "weaknesses": "Literature review and comparisons are partial to the RL literature. There is a long-standing literature in control [A, B, C] to use an approximate model to plan using predictive control. A whole host of methods to learn a safety-filter/shielding on the fly has been explored with robust optimization-based offline and online control techniques. Most of these methods would implicitly solve the problem this paper is trying to address. However, it is interesting that the paper uses the Q function in the online optimization. This aspect is novel and unique to this paper.\n\nIt is unclear how much computation and time it takes to run MCTS online at each time in order to do dynamic shielding at runtime.\n\nDynamics model such as double integrator and differential drive are too simple. It would be interesting to see how well these would work with more complicated and/or higher-dimensional dynamics.\n\n[A] Breeden, Joseph, and Dimitra Panagou. \"Predictive control barrier functions for online safety critical control.\" 2022 IEEE 61st Conference on Decision and Control (CDC). IEEE, 2022.\n\n[B] Wabersich, Kim P., and Melanie N. Zeilinger. \"Predictive control barrier functions: Enhanced safety mechanisms for learning-based control.\" IEEE Transactions on Automatic Control 68.5 (2022): 2638-2651.\n\n[C] Wabersich, Kim P., et al. \"Data-driven safety filters: Hamilton-jacobi reachability, control barrier functions, and predictive methods for uncertain systems.\" IEEE Control Systems Magazine 43.5 (2023): 137-177.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their many insightful comments and suggestions. We respond to their questions and concerns below.\n\n**Adding traditional control and non-RL methods to literature review**\n\nWe appreciate the reviewer's paper recommendations and will include them, along with classical control papers, in our literature review.\n\nWe did not compare our approach to non-RL methods experimentally. One of RL's strengths is its generality—agents can satisfy complex specifications with just an abstract environment and a simple reward signal. While this generality is valuable, specialized methods often outperform RL in specific tasks. Since our paper is primarily RL-focused, we believe that the fairest comparison is against other RL approaches.\n\n**Concerns over simplicity of the environment dynamic models**\n\nSince the planner and RL algorithm treat environment dynamics as black boxes, more complex dynamics don't necessarily challenge the agent, as long as they are deterministic and allow sufficient range of motion for task completion. We agree that increasing the dimensionality of the dynamics would make the task harder for both the RL algorithm and the planner and believe that this would be an interesting avenue for future research.\n\n**Using other RL training algorithms**\n\nOur algorithm can generally work with any off-policy RL algorithm. We briefly explored using DDPG and SAC as the training algorithm and did not find reliable improvement.\n\n**Advantages of using DMPS if performance policy does not use RL (instead using IL or no learning)**\n\nRL-based approaches like DMPS can learn from interactions with the environment, allowing them to explore states and actions essential for model learning. Unlike imitation learning, which often suffers from distribution shift, RL methods are more resilient. Moreover, RL methods are more general than alternatives and need minimal input. Imitation learning requires expert trajectories and sometimes an expert controller, which can be costly. Classical learning-free methods rely on environment-specific assumptions. We believe that the generality of RL makes it a valuable paradigm on its own.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their insightful feedback. We summarize the responses to common questions.

**Computational cost of the planner**

There is a tradeoff between the quality of the recovery plan, and the computational cost incurred in the planner searching for it. The look-ahead controls this tradeoff. In general, DMPS can leverage any planner available for the domain, and the computational cost of the planner with respect to the look-ahead will depend on the exact choice of the planner, and its hyperparameters.

In our implementation, we use MCTS in an anytime planning setting, where we devote a fixed number of node expansions to searching for a plan, and we choose the best plan found at the end of the procedure. The clock time needed would simply be linear in the allocated compute budget. However, the worst-case computational complexity to densely explore the search space, as in general planning problems, would be O(exp(H)) where H is the look-ahead length, since the planning search space grows exponentially.

Our implementation used MCTS with 100 node expansions, a plan horizon of 5, and a branch factor of 10, which amounts to exploring 1000 states in total. On average, we found that when the shield is triggered, planning takes 0.4 seconds per timestep. The code we used is unoptimized, written in Python, and single-threaded. Since MCTS is a CPU-intensive search process, switching to a language C++ would likely yield significant speed improvements, and distributing computation over multiple cores would further slash the compute time by the number of assigned cores.

Figure 1 shows the number of simulator state expansions needed by a search-based planner as a function of horizon length in the double-gates+ (double integrator) environment. Each node expansion requires equal computation time, hence this graph illustrates how the computation cost of the planner scales as a function of horizon length. Since we use an anytime planner, the number of actual states expanded (and hence the computational cost) is capped by a hyperparameter that controls the planner computation budget. 

**Sufficiency of planning horizon H=5**

Our algorithm is designed to ensure that the planner accounts for both short-term and long-term objectives. As detailed in Section 5, the objective of the planner consists of two terms: 1) the discounted sum of the first $H$ rewards, and 2) the estimated long-term value of the penultimate step $s_H$ in the plan horizon, as determined by the agent’s $Q$ function. 
The second part of the objective function is specifically included to ensure that the planner does not return myopic plans, and accounts for progress towards the long-horizon goal. Since the planner optimization objective includes this second term, even a small-horizon planner can output actions with proper awareness of long-horizon recovery events. The length of the horizon affects how close to the globally optimal solution the result is, with a tradeoff of computational cost.

To see this empirically, we reevaluated the double-gates+ environment (double integrator dynamics) with horizons of 1, 5, and 9. The graphs of attained reward and shield triggers from this experiment are attached as Figure 2 in the pdf. Comparing the H=1 and H=5 agents, the H=5 agent substantially outperforms the H=1 agent in both shield triggers and reward. Comparing H=5 and H=9, the H=9 agent reached high performance and low shield triggers faster than the H=5 agent. However, both the H=5 and H=9 agents converge to the same performance eventually.

**Experimental results**

We thank reviewer NRWz for suggesting the inclusion of PPO-Lagrangian as an additional baseline. We have evaluated it on all environments and appended the values to the tables of results. The new tables are attached to this response. Similar to CPO, PPO-lag was unable to make headway on the dynamic environments. We also correct bolding errors in the tables. In Table 1, DMPS, MPS, and REVEL are provably safe RL methods while CPO, PPO-lag, and TD3 are not. The table shows shield triggers for the former and safety violations for the latter.

**Concerns about assumption of perfect information and determinism**

The majority of prior work on PSRL assumes perfect information [1,2,3,4,5]. While traditional model-based RL methods can learn environment models even under partial observability/imperfect information, they cannot provably guarantee safety and are thus not suitable for tasks where safety violations are completely unacceptable.

Much prior work on provably safe locomotion also assumes determinism [3,4,5], with some such algorithms even having been deployed on real robots [3]. However, we agree with reviewers that extending our DMPS approach to stochastic environments is a promising direction for future work. In particular, since prior MPS work has been extended to work in stochastic settings [1,2], we believe our DMPS approach can be similarly extended to the stochastic setting as well.

[1] Anderson, G., Verma, A., Dillig, I., & Chaudhuri, S. (2020). Neurosymbolic Reinforcement Learning with Formally Verified Exploration. NeurIPS 2020

[2] Bastani, O., & Li, S. (2021). Safe Reinforcement Learning via Statistical Model Predictive Shielding. In D. A. Shell, M. Toussaint, & M. A. Hsieh (Eds.), RSS 2021.

[3] Vinod, A. P., Safaoui, S., Chakrabarty, A., Quirynen, R., Yoshikawa, N., & Cairano, S. D. (2022). Safe multi-agent motion planning via filtered reinforcement learning, ICRA 2022

[4] Zhang, W., & Bastani, O. (2019). MAMPS: Safe Multi-Agent Reinforcement Learning via Model Predictive Shielding. CoRR, abs/1910.12639.

[5] Zhu, H., Xiong, Z., Magill, S., & Jagannathan, S. (2019). An inductive synthesis framework for verifiable reinforcement learning, PLDI 2019

Meta-review: The paper addresses provably safe RL problems where safety must be ensured even during training. In particular, it considers the optimal way to recover from potentially dangerous states with respect to the main objective.

The proposed approach is based on the Model Predictive Shielding, which follows a backup policy when the current policy proposes an unsafe action. The new method, called dynamic MPS (DMPS), integrates a local planner to select safe recovery actions that also aim to maximize the short and long-term rewards.

Although the reviewers consider that the method uses multiple known techniques, they also point out that the combination of such methods is novel. Moreover, they also praised the quality of the paper and the theoretical contribution related to the recovery regret. Furthermore, the rebuttal clarified some concerns regarding the computational overhead of the proposed method. Overall, the discussion raised some limitations of DMPS, which should be clarified in a new version of the paper.

I believe this paper makes a clear contribution to safe RL. Taking that into account and the unanimously positive recommendation of the reviewers, I recommend its acceptance.
Decision: Accept (poster)