Model-free Safe Control for Zero-Violation Reinforcement Learning
Authors: Weiye Zhao
Tairan He
Changliu Liu
Venue: CoRL2021 Poster (2021)
Keywords: Robots
Safe Reinforcement Learning
Safe Control

Abstract: While deep reinforcement learning (DRL) has impressive performance in a variety of continuous control tasks, one critical hurdle that limits the application of DRL to physical world is the lack of safety guarantees. It is challenging for DRL agents to persistently satisfy a hard state constraint (known as the safety specification) during training. On the other hand, safe control methods with safety guarantees have been extensively studied. However, to synthesize safe control, these methods require explicit analytical models of the dynamic system; but these models are usually not available in DRL. This paper presents a model-free safe control strategy to synthesize safeguards for DRL agents, which will ensure zero safety violation during training. In particular, we present an implicit safe set algorithm, which synthesizes the safety index (also called the barrier certificate) and the subsequent safe control law only by querying a black-box dynamic function (e.g., a digital twin simulator). The theoretical results indicate the implicit safe set algorithm guarantees forward invariance and finite-time convergence to the safe set. We validate the proposed method on the state-of-the-art safety benchmark Safety Gym. Results show that the proposed method achieves zero safety violation and gains $ 95\% \pm 9\%$ cumulative reward compared to state-of-the-art safe DRL methods. Moreover, it can easily scale to high-dimensional systems.

for Model-free Safe Control Reinforcement Learning
Weiye Zhao
Tairan He
Changliu Liu
Carnegie Mellon University
(weiyezha, tairanh, cliu6 (aiudee cma, edu
Abstract: While deep reinforcement learning (DRL) has impressive performance in a variety of continuous control tasks, one critical hurdle that limits the applica- tion of DRL to physical world is the lack of safety guarantees. It is challenging for DRL agents persistently satisfy hard state constraint (known as the safety specification) during training. On the other hand, safe control methods with safety guarantees have been extensively studied. However, synthesize safe control, these methods require explicit analytical models of the dynamic system; but these models usually not available in DRL. This paper presents model-free safe control strategy to synthesize safeguards for DRL agents, which will ensure zero safety violation during training. In particular, we present an implicit safe set algo- rithm, which synthesizes the safety index (also called the barrier certificate) and the subsequent safe control law only by querying black-box dynamic function (e.g., digital twin simulator). The theoretical results indicate the implicit safe set algorithm guarantees forward invariance to the safe set. We validate the pro- posed method on the sfte-of-thhe-a safety benchmark Safety Gym. Results show that the proposed method achieves zero safety violation and gains 95% 9% cu- mulative reward compared to state-of-the-ard safe DRL methods. Moreover, the proposed method can easily scale to high-dimensional systems.
Keywords: Mobile Robots, Safe Reinforcement Learning, Safe Control
1 Introduction
Safety in terms of persistently satisfying hard state constraint always an important consideration in robotics. For example, vehicles should not crash into pedestrians; robot arms should not hit walls. More safety concerns are raised during robot learning, where the robot needs figure out the optimal control policy by exploring the surrounding environment. It is not enough to merely penalize the robot for entering the unsafe states using posterior measures [1, 2]. The robot's action should indeed be constrained to not enter the unsafe states using prior measures.
Safe control methods that constrain the robot motion to persistently satisfy a hard safety constraint in predictable environments have been extensively studied. The most widely used ones are energy snnction-based methods. These methods [3, 4, 5, 6] first design an energy function such that the safe states are with low energy and then synthesize control laws that are constrained to make the system dissipate energy. Then the system will never leave the safe set forward invariance). These methods were first introduced for deterministic systems. They have been recently extended to stochastic and uncertain systems [7, 8] (uncertainties in either the state measurements or the future evolution of the ego robot or obstacles) and combined with robot learning for safety assurance [9]. However, the major limitation of these methods is that they require white-box analytical models of the system dynamics (c.g., Kinematic Bicycle Model [10]) and/or the associated uncertainties, which are challenging to obtain during robot learning.
This paper proposes a model-free safe control method that enables the application of energy function-based methods to ensure robot safety in environments where no white-box analytical model is available. The model-free safe control is able to safeguard any robot learning algorithm. The key insight we have is that the safe control laws can be synthesized by only querying black-box non- analytical dynamic functions. These black-box functions can be high-fidelity digital twin simulators
5th Conference on Robot I Learning (CoRI. 2021), London, UK.

---

or deep neural network models, which are usually available or can be constructed by data-driven learning. The key questions that we want to answer are: 1) how synthesize the energy function (in our case called the safety index) so that there always exists feasible control input to dissipate energy at any state, when the dynamics are black-box; and 2) how to efficiently sample the control space to find the best safe control (e.g., the control that dissipates energy and optimizes the reward) with black-box dynamics. We propose safety index design rule to address the first question, and sample-efficient algorithm to perform black-box constrained optimization to address the second question. By combining these approaches with the (model-based) safe set algorithm [5], we propose the (model-free) implicit safe set algorithm (ISSA). We then use ISSA to safeguard deep reinforce- ment learning (DRL) agents. Extension to other energy function- based methods and robot learning approaches is straight forward and will be explored in the future.
The key contributions of this paper are summarized below:
We propose two techniques to synthesize safe control laws without white- box models: safety index design rule for mobile robots in the 2D plane and a sample-efficient black-box optimization algorithm using adaptive momemtum boundary approximation (AdamBA).
We propose the implicit safe set algorithm using these two techniques, which guarantees to generate safe controls for all system states without knowing the explicit system dynam- ics. We show that ISSA safeguard DRL agents to ensure zero safety violation during training in Safety Gym. Our code is available on Github.'
2 Problem Formulation
Dynamics Let we E X C R be the robot state at time step L, where na is the dimension of the state space X; E ul C R" be the control input to the robot at time step I, where n. is the dimension of the control space U. The system dynamics are defined as:
T++1
f We),
(1)
where f: X' x U + X is function that maps the current robot state and control to the robot state in the next time step. For simplicity, this paper considers deterministic dynamics. The proposed method can be easily extended to stochastic case through robust safe control [11, 12], which will be left for future work. Moreover, it is assumed that we can only access an implicit black-box form of f. e.g., as an implicit digital twin simulator or a deep neural network model. Note that the word implicir refers to that we evaluate fxx u) without any explicit knowledge or analytical form of (x.
Safety Specification The safety specification requires that the system state should be constrained in a closed subset in the state space, called the safe set Xs.. The safe set can be represented by the zero-sublevel set of continuous and piecewise smooth function do R= + R, i.c., X's {o <o(z) S 0}. Xg and do are directly specified by users. The design of Φn is straightforward in most scenarios. For example, for collision avoidance, Φo be designed as the negative closest distance between the robot and environmental obstacles.
Reward and Nominal Control A robot learning controller generates the nominal control which is subject to modification by the safeguard. The learning controller aims to maximize rewards in an infinitehoriioon deterministic Markov decision process (MDP). An MDP is specified by a tuple U, r, f), where r X x 2 + IR is the reward function, 0 S < 1 is the discount factor, and f is the deterministic system dynamics defined in (1).
Problem Building on top of the nominal learning agent, the core problem of this paper is to syn- thesize a safeguard for the learning agent, which monitors and modifies the nominal control to ensure forward invariance in subset of the safe set X'. Forward invariance of set means that the robot state will never leave the set if starts from the set. The reason why we need to find a subset instead of directly enforcing forward invariance of X's is that X's may contain states that will inevitably go to the unsafe set no matter what the control input is. These states need to be penalized when
pattps://gee /ttt::://g/ttt www.eeeeeeeeeeeeeeeeeeeee contentttoooooo-ooe Latetroo--aab/ISSS Lttpaaaaaaaaaraaa..aaaaaaaa

---

we synthesize the energy function. For example, when vehicle is moving toward an obstacle with high speed, it would be too late to stop. Even if the vehicle is safe now (if X's only constrains the position), it will eventually collide with the obstacle (unsafe). Then we need to assign high energy values to these inevitably-unsafe states.
Related Work
Safe Reinforcement Learning Safe RL either considers soft safety constraints or hard safety constraints. Typical safe RL methods for soft safety constraints include risk-sensitive safe RL [13], Lagrangian methods [14] and constrained policy optimization (CPO) [15]. These methods are able to find policies that satisfy the safety constraint in expectation, but cannot ensure all visited states safe. The methods that more closely related to ours are safe RL methods with hard safety constraints. Richard el al. [7] proposes a general safe RL framework, which combines CBF-based controllers with RL algorithms to guarantee safety and guide the learning process by constraining the set of explorable policies. However, the theory guarantee of CBF-based controller strongly relies on known control affine dynamics system, which restricts its application to general nonlinear dy- namics systems. More related works of safe RL methods with hard safe constraints are summarized in Appendix A.
Safe Control Representative energy function-ann methods for safe control include potential field methods [3], control barrier functions (CBF) [4], safe set algorithms (SSA) [5], sliding mode algo- rithms [6], and a wide variety of bio-inspired algorithms [16]. The first step to synthesize safe controller is to compute a desired energy function offline such that 1) the low energy states are safe and 2) there always exists feasible control input to dissipate the energy. SSA has introduced a rule- based approach to synthesize the energy function as a continuous, piece-wise smooth scalar function on the system state space Φ T== -
RR. And the energy function (xx) is called safety index in (n) this approach. The general form of the safety index was proposed as Φ = % + ki d0 + + k where 1) the roots of + k + + s = 0 are all negative real (to ensure pero-overshooting of the original safety constraints); 2) the relative degree from (61) to is one (to avoid singularity); and 3) o8 defines the same set as du (to nonlinear shape the gradient of 6 at the boundary of the safe set). It is shown in [5] that if the control input is unbounded (l I²") ) then there always exist a control input that satisfies the constraint à when ø = O: and if the control input always satisfies
that constraint, then the set far ((x) S 0} n {æ (x)x)) < 0) is forward invariant. In practice, the actual control signal is computed through a quadratic projection of the nominal control to the control constraint
arg min l u²1 s.t. á -n()), uEM
(2)
where à < (m))) is general form of the constraint; 77 : + R is a non decreasing function that n7)) > 0. For example, in CBF, 7()) is designed to be Xo for some positive scalar X. In SSA, (0)) is designed to be a positive constant when ø 2 0 and -o0 when ø < 0. Note there two major differences between this paper and the existing results. First, this paper considers constrained control space, which then require careful selection of the parameters in ó to make sure the control constraint Us(x) == (u E 2 & < -n(()) is nonempty for states that ø 2 O. Secondly and most importantly, this paper considers general black-box dynamics, while the existing work considers analytical control-affine dynamics. For analytical control-affine dynamic models, (xxr) essentially half-space and the projection (2) can be efficiently computed by calling a quadratic programming solver. However, for black-box dynamics, this constraint is challenging to quantify.
4 Method
This section introduces the implicit safe set algorithm (ISSA), which is able to leverage energy cauction-based methods (SSA in particular) with black-box dynamics, and be used to safeguard any nominal policy. ISSA contains two parts: 1) a safety index synthesis rule to make sure Us (x) is nonempty for all æ, and 2) sample-efficient black-box optimization method to solve the projection of the nominal control to (x)).. With these two components, the overall pipeline of the implicit safe set algorithm is summarized as follows:
3

---

Offline: Design the safety index ((x) according to the safety index design rule.
Online: Project nominal control into (s(2)) during online robot maneuvers.
For simplicity, consider discrete-time control only since real-world robot control systems always implemented in discrete-time. Thus, we consider the discrete-time set of safe control ug (z) = {u E u 6(f(2, u)) I maxfo(x) 17. 0}} in the following discussion.
4.1 Synthesize Safety Index
In this paper, the safety constraint that we are specifically interested in collision avoidance for mobile robots in 2D planes. We treat the robot and the obstacles as point-mass circles with bounded collision radius. The safety specification is define as do max; 60₃, and Φo; = dmin d; where d, denotes the distance between the center points of the robot and the i-th obstacle (static non-static). It is assumed that the dynamics f(z u) is a bounded Lipschitz function. Denote a and w as the rela- tive acceleration and relative angular velocity of the robot in the obstacle frame, respectively, as shown in fig. 1. The safety index for collision avoidance in 2D will be syn- thesized without referring to the specific dynamic model, but under the following assumptions.
Robot
Obstacle
Figure I: Notations.
Assumption 1 (2D Collision Avoidance). 1) The state space is bounded, and the relative accel- eration and angular velocity are bounded and both can achieve zeros, i.e., w E [Wmin Wenar] for Wmin I 0 S Wmar and a E [Amin, Omar for Omin I 0 S amax: 2) For all possible values of and w, there always exists control u to realize such a and w; 3) The discrete- time system time step dt + 0. 4) At any given time, there at most be one obstacle becoming safety critical, such that Φ 7 2 0 (Sparse Obstacle Environment).
These assumptions are easy to met in practice. The bounds in the first assumption will be directly used to synthesize é. The second assumption enables us to turn the question on whether these exists feasible control in ug to the question on whether there exists and to decrease d. The third assumption ensures that the discrete time approximation error small. The last assumption enables safety index design rule applicable with multiple moving obstacles. Following the rules in [5], parameterize the safety index as max; Oi and Φ; share the same set of tunable parameters o, n. k. 77 E RT. Our goal is to choose these parameters such that U (x) is always nonempty. Under the above assumptions, the safety index design rule is constructed below.
+ d'nnn di kd,, where all Φ;
Safety Index Design Rule: By setting = O, the parameters k. n, and should be chosen such that
n(o d + kvvaa
Oqnin
(3)
Umar
where Umaz is the maximum relative velocity that the robot can achieve in the obstacle frame.
4.2 Sample-Efficient Black-Box Constrained Optimization
The nominal control up needs to be projected to 2 (x) by solving the following optimization:
min | - u/12 Elell
(4)
s.t. of (Iz, uz) S max 17, 0}
Since the objective of (4) is convex, its optimal solution will always lie on the boundary of Ug (I:) if w" & ug(x). (x). Therefore, it is desired to have an efficient algorithm to find the safe controls on the boundary of ug (z). To efficiently perform this black-box optimization, we propose sample- efficient boundary approximation algorithm called Adaptive Momentum Boundary Approximation Algorithm (AdamBA), which is summarized in Algorithm of Appendix B. We illustrate the main boundary approximation procedures of AdamBA in Figure 2, where AdamBA supposed to find the boundary points of US (x) (green area) with respect to the reference control n" & Ug (a:) (red star). The core idea of the AdamBA follows the adaptive line search [17], where three main procedures are included. I. AdamBA first initialize several unit gradient vectors (green vectors) to be the sampling

---

A u'
u
u'
u
Unit Grodiest Vector
2700 Conrol
2700
Salt
Custnol Space W
Control Space W
Costrol Space W
Contiol Space W
(a) Initialize gradient vectors
(b) Exponential outreach.
(c) Exponential decay.
(d) Boundary points found.
Figure 2: Illustration of the procedure of the AdamBA algorithm.
directions as shown in Figure 2a. II. AdamBA enters the exponential outreach phase by exponential increasing the gradient vector length until they reach US (a:) as shown Figure 2b. Note that we discard those gradient vectors that go out of control space (red vectors). III. Next, AdamBA enters exponential decay phase by iteratively applying binary search to find boundary points as shown in Figure 2c. Finally, set of boundary points will be returned after AdamBA converges shown in Figure 2d. Note that AdamBA and the line search methods are fundamentally similar to each other, while the purpose of AdamBA is to find the boundary of safe/unsafe action, while the line search methods are to find the minimum error can be upper bounded, where the bound depends the resolution of the sampling directions.
function. It can be shown that the boundary approximation
4.3 Implicit Safe Set Algorithm
Leveraging AdamBA and the safety index design rule, we construct the implicit safe set algorithm (ISSA). The proposed ISSA is summarized in Algorithm 2 of Appendix B. ISSA contains an offline stage and an online stage. In the offline stage, we synthesize the safety index according to the design rule (3). There are two major phases in the online stage for solving (4), In online-phase 1, we directly use AdamBA to find the safe controls on the boundary of Ug (x). and choose the control with minimum deviation from the reference control the final output. In the case that safe control activated. We uniformly
returned in online- -hanee due to sparse sampling, online-phase sample the control space and deploy AdamBA again these samples to find the safe control on the boundary of ug (z). It can be shown that the ISSA algorithm is guaranteed to find a feasible solution of (4) and that solution ensures forward invariance the set S = X's O {ar Φ(x)) 0).
Although the ISSA algorithm builds upon the safe set algorithm [5]. the proposed safety index synthesis and AdamBA algorithm can be applied to other energy fanction-based methods to generate safe controls with or without an explicit analytical dynamics model.
Theoretical Results
We first present two propositions for the feasibility of the safety index synthesis rule and of Al- gorithm 2. The proofs for Proposition 1 and Proposition are summarized in Appendix and Appendix D. respectively. Then, we present the main theorem stating that the implicit safe set algorithm ensures forward invariance. The proof of Theorem is summarized in Appendix E.
Proposition 1 (Nonempty set of safe control). If the dynamic system satisfies the assumptions in Assumption 1, then the safety index design rule in Section 4.1 ensures that the robot system in 2D plane has nonempty set of safe control at any state, i.e.. U (x) + 0, Vx.
Proposition 2 (Feasibility Algorithm 2). If the set of safe control is non-empty. Algorithm 2 can always find local optimal solution of (4) with finite number of iterations.
Theorem (Forward Invariance). If the control system satisfies the assumptions in Assumption and the safety index design follows the rule described in Section 4.1, the implicit safe set algorithm guarantees the forward invariance to the set S C Xs.
Experimental Results
In our experiments, aim answer the following questions:
Q1: How does ISSA compare with other state-of-the-arr methods for safe RL. Can ISSA achieve zero-violation of the safety constraint?
5

---

(a) Point robot: simple (b) Goal: navigating the (c) Push: pushing the yel- (d) Hazards: non-physical inside the green dangerous
(e) Pillarw: fixed dangerous obstacles
2D robot that can turn and robot inside the green goal
goal area.
Figure 3: The environmental settings for benchmark problems in Safety Gym.
Q2: How does the design of the safety index affect the set of safe control?
Q3: How do the hyper-parametenn of ISSA and the dimensionality of the system impact its performance?
To demonstrate the effectiveness of the proposed implicit safe set algorithms, we conduct evalu- ation experiments on the safe reinforcement learning benchmark environment Safety Gym [14]. Our experiments adopt the Point robot (U S RR2) as shown in Figure 3a and the Doggo robot (M S 122 shown in Figure 1. We design 8 experimental environments with different task types, constraint types, constraint numbers and constraint sizes. We name these environments as (Cons))-Coostrain Type) (Con)))Costtriin
Size]. Note that Constraint Si Size equals dmin the safety index design. Two tasks are considered:
Goal: The robot must navigate to goal as shown in Figure 3b.
Push: The robot must push a box to a goal as shown in Figure 3c.
And two different types of constraints are considered:
Hazard: Dangerous (but admissible) areas as shown in Figure 3d. Hazards are circles on the ground. The agent penalized for entering them.
Pillar: Fixed obstacles shown in Figure 3e. The agent is penalized for hitting them.
The methods in the comparison group include: uncon- strained RL algorithm PPO [18] and constrained safe RL algorithms PPO-Lagrangian, CPO [15] and PPO-SL (PPO-Safety Layer) [19]. We select PPO as our baseline method since it is state-of-the-ar and already has safety- constrained derivatives that can be tested off-the-shelf. We set the limit of cost to for both PPO-Lagrangins and CPO since we aim avoid any violation of the constraints. To make sure ISSA can complete tasks while guarantecing safety, we use a PPO agent as the nominal policy and ISSA as safety layer to solve (4), we call this structure as PPO-ISSA, and it illustrated in Figure Such safety layer structure has also been used in PPO-SL [19] which leverages offline dataset to learn a linear safety- signal model and then construct safety layer via analytical optimization. For all experiments, we use neural network policies with separate feedforward MLP policy and value networks of size (256, 256) with tanh activations. More details are provided in Appendix F.I.
Norebul policy
Sakty lager
Figure 4: The PPO-ISSA structure.
PFO
PRO I5SA
6.1 Evaluating PPO-ISSA and Comparison Analysis
To compare the reward and safety performance of PPO-ISSA to the baseline methods in different tasks, constraint types, and con- straint sizes, we design 4 test suites with 4 constraints which are summarized in Figure The comparison results reported in Fig- ure demonstrate that PPO-ISSA is able to achieve zero average episode cost and zero cost rate across experiments while slightly sacrificing the reward performance. The baseline soft safe RL meth- ods (PPO-Lagrangian and CPO) fail to achieve zero- violation safety when the cost limit is set to be O. PPO-Lagrangian and CPO
Figure
changes over time steps of PPO and PPO-ISSA starting from the same unsafe state over 20 trails.

---

400
WD Do
Thea Pruiioatts
TeadPhlt
Chi
Tcall caaaca
Cho
13
Dall
Tea Cheneeee
(a) Goal- HazaaH---.0DD 0.05
(b) Goal- Hazard4-0.15
(c) Goul-Pillard-O.E
(d) Push Push-Haaard440.11
Figure 6: Average episodic return, episodic cost and overall cost rate of constraints of PPO-ISSA and baseline methods 4-constraint environments over five seeds.
fail since both methods rely on trial-and-error enforce constraints while ISSA is able to guar- antee forward invariance by Theorem 1. We also observe that PPO-SL fails to lower the vio- lation during training. due to the fact that the linear approximation of cost function c(x++1) x c(2+) + g( w²" u [19] becomes inaccurate when the dynamics are highly nonlinear like the ones we used MuloCo [20]. More importantly, PPO-SL cannot guarantee that these always exist a feasible safe control to lower the cost, since they directly use the user defined cost function which cannot always ensure feasibility. More detailed metrics for comparison and experimental results on test suites with 1 constraint are summarized in Appendix F.1.4.
We further compare the cost evolution of PPO and PPO-ISSA agents when starting from the same unsafe state (i.c., cost >> 0). The com- parison results are shown in Figure 5, which shows that PPO-ISSA can converge to X's within 100 time steps across all experiments while cost evolution of PPO agents fluctuates wildly without preference to con- verge to safe set.
20
ce2.75
1id
Figure Average return of PPO-ISSA with different safety index design on Goal- Hazard4-0.15.
6.2 Feasibility of Safety Index Synthesis
To demonstrate how the set of safe control is impacted by different safety index definition, we randomly pick an unsafe state x² such that <(x*) > O. and visualize the corresponding set of safe control 4g under different safety index def- initions, which are shown in Figure 8. Red area means Ao > 0 (i.e. unsafe control) and blue area means Ao 0 (i.e. safe control). Figure 8a shows the set of safe control of distance safety index - + dmin - d, which is the default cost definition of Safety Gym. The heatmap is all red in Figure 8a, which means that the set of safe control under the default da is empty. Figure 8b shows the set of safe control of the synthesized safety index σ d d2 kd with different value of k. With the synthesized safety index, Figure 8b demonstrates that the size of the set of safe con- trol grows as k increases, which aligns with the safety index synthesis rule discussed in Section 4.1 as larger k is easier to satisfy (3). To demonstrate the reward performance of PPO-ISSA under different safety index designs, we select Goal- Hoall-azardd40.11 test suite. Figure demonstrates the average return of PPO-ISSA under different value of k, which shows that the reward performance of PPO-ISSA deteriorates as value increases (since larger & makes the control more conservative). Note that the set of safe control increases as the l value increases, thus the optimal & should be the smallest & that makes the set of safe control nonempty for all states. Our safety index synthesis rule in (3) provides the condition to pick the optimal k.
da =
7

---

KC-0.75
K-5.00
10
100
44
Distancn Safery Index
(b) Synthesized Salery Index
Figure 8: Heat maps of the difference of safety index A of u)) d(x). The x-axis uy represents the
control space of moving actuator, and the y-axis u2 represents the control space of turning actuator.
Number of vectors
Simulation Time Tam Overall ISSA Time Tau
Return J,
0.301
0.738
5
0.504
0.511
0.826
0.987
1,000
1,000
Table I: Normalized computation time and return under different number of vectors in ISSA. These results average on 100 ISSA rums over five random seeds on Goal- Soal-Hazard4-0.15.
6.3 Sensitivity Analysis and Scalability Analysis
To demonstrate the scalability and the performance of PPO-ISSA when ISSA chooses different parameters, we conduct additional tests using the test suite Goal-Haaard4-00.
Among all input pa- rameters of ISSA, the gradient vector number n is critical impact the quality of the solution of (4). Note in the limit when - oc, ISSA is able to traverse all boundary points of the set of safe control, hence able to find the global optima of (4). We pick three different values: 3, 5, 10; and report the average episode reward of PPO-ISSA, and the computation time of ISSA when solving (4), which includes the normalized average ISSA computation time and the normalized average simulation time for each run. The results are summarized in Table 1, which demonstrates that the reward per- formance of PPO-ISSA would improve as n gets bigger since we get better optima of Equation (4). In practice, find that the reward performance will stop improving when is big enough (n 10).
The computation time scales linearly with respect to while the majority (98%) of computation cost is used for environment simulation, which can be im- proved in the future by replacing the simulator with a more computationally efficient surrogate model. We also evaluate the scalability of ISSA in 12- dimensional control system, showing ISSA is able achieve cero-violation and best performance with acceptable computation cost in Figure 9. Besides, ISSA can be largely accelerated using parallel com- putation. The detailed discussion can be found in Appendix G.
Seatthee
Average reward
(b) Average cost
Figure Average episodic return and episodic cost PPO-ISSA and baseline methods on Goal- Hazard1-O.15 environment of doggo robot over five seeds.
7 Conclusion and Future Work
Safety guarantee is critical for robotic applications in real world, such that robots can persistently satisfy safety constraints. This paper presents a model-free safe control strategy to synthesize safe- guards for DRL agents, which will ensure zero safety violation during training. In particular, we present an implicit safe set algorithm as a safeguard, which synthesizes the safety index (also called the barrier certificate) and the subsequent safe control law only by querying black-box dynamics function (c.g., a digital twin simulator). The theoretical results indicate that the synthesized safety index guarantees nonempty set of safe control for all system states, and ISSA guarantees forward invariance to the safe set. We further validate the proposed safeguard with DRL, on state-of-the- the-art safety benchmark Safety Gym. Our proposed method achieves zero safety violation and 95% + 9% reward performance compared to state-of-the--e safe DRL methods.
There are two major directions for future work. Firstly, we will further generalize the safety index synthesis rule to cover wider range applications other than collision avoidance in 2D. Secondly, we will further speed up the implicit model evaluation step by replacing the physical engine based simulator with a learned surrogate model while taking the learned dynamics error into account.
8

---

Acknowledgments
This work is supported by Amazon Research Award.
References
[1] M. Papini, M. Pirotta, and M. Restelli. Smoothing policies and safe policy gradients. CoRR, abs/1905.03231, 2019.
[2] M. Pirotta, M. Restelli, A. Pecorino, and D. Calandriello. Safe policy iteration. In International Conference on Machine Learning, pages 307-315. PMLR, 2013.
[3] O. Khatib. Real-time obstacle avoidance for manipulators and mobile robots. In Autonomous robot vehicles, pages 396-404. Springer, 1986.
[4] A. D. Ames, J. W. Grizzle, and P. Tabuada. Control barrier function based quadratic programs with application to adaptive cruise control. In 53rd IEEE Conference on Decision and Control, pages 6271-6277 IEEE, 2014.
[5] C. Liu and M. Tomizuka. Control in safe set: Addressing safety in human-robot interactions. In ASME 2014 Dynamic Systems and Control Conference, American Society of Mechanical Engineers Digital Collection, 2014.
[6] L. Gracia, F. Garelli, and A. Sala. Reactive sliding- algorithm for collision avoidance in robotic systems. IEEE Transactions on Control Systems Technology, 21 2013.
[7] R. Cheng, G. Orosz, R. M. Murray, and J. W. Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3387-3395, 2019.
[8] J. Taylor and A. D. Ames. Adaptive safety with control barrier functions. In 2020 American Control Conference (ACC), pages 1399-1405. IEEE, 2020.
[9] F. Fisac, A. K. Akametalu, M. N. Zeilinger, S. Kaynama, J. Gillula, and C. J. Tomlin. A general safety framework for learning- control in uncertain robotic systems. IEEE Transactions on Automatic Control, 64(7):2737-2752 2018.
[10] J. Kong, M. Pfeiffer, G. Schildbach, and F. Borrelli. Kinematic and dynamic vehicle models for autonomous driving control design. In 2015 IEEE Intelligent Vehicles Symposium (IV), pages 1094-1099, IEEE, 2015.
[11] C. Liu and M. Tomizuka. Safe exploration: Addressing various uncertainty levels in human robot interactions. In 2015 American Control Conference (ACC), pages 465-470. IEEE, 2015.
[12] C. Noren, W. Zhao, and C. Liu. Safe adaptation with multiplicative uncertainties using robust safe set algorithm. In Modeling, Estimation and Control Conference, 2021.
[13] Garcia and F. Fernández. A comprehensive survey safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
[14] A. Ray, J. Achiam, and D. Amodei. Benchmarking safe exploration in deep reinforcement learning. CoRR, abs/1910.01 708, 2019.
[15] Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained policy optimization. In International Conference Machine Learning, pages 22-31. 31. PMLR, 2017.
[16] J. Zhang, M. Kang, X. Li, and G.-y. Liu. Bio-inspired genetic algorithms with formalized crossover operators for robotic applications. Frontiers in neurorobotics, 11:56, 2017.
[17] L. Armijo. Minimization of functions having lipschitz continuous first partial derivatives. Pacific Journal of mathematics, 16(1):1-3, 1966.
[18] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR, abs/ 1707. abs/1770.06347, 2017.

---

[19] G. Dalal, K. Dvijotham, M. Vecerik, T. Hester, C. Paduraru, and Y. Tassa. Safe exploration in continuous action spaces. CoRR, abs/1801.08757, 2018.
[20] E. Todorov, T. Erez, and Tassa. Mujoco: physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026- 5033. IEEE, 2012.
[21] J. Ferlez, M. Elnaggar, Y. Shoukry, and C. Fleming. Shieldnn: A provably safe nn filter for unsafe controllers. CoRR, abs/2006.0955 09564,, 2020.
[22] T.-H. Pham, G. De Magistris, and R. Tachibana. Optlayer-practical constrained optimization for deep reinforcement learning in the real world. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 6236-6243, IEEE, 2018.
[23] Berkenkamp, M. Turchetta, A. Schoellig, and Krause. Safe model-based reinforcement learning with stability guarantees. Advances in Neural Information Processing Systems, 30, 2017.
[24] J. Grover, C. Liu, and K. Sycara. Feasible region-based identification using duality (extended version). CoRR, abs/2011 1.09944 2020.
[25] S. Grover, C. Liu, and K. Sycara. Parameter identification for multirobot systems using optimization based controllers (extended version). CoRR, 0bs/22009118177 2020.
[26] Y. Chow, O. Nachum, A. Faust, E. Duenez-Guzman, and M. Ghavamzadeh. Lyapunov-based safe policy optimization for continuous control. ICML 2019 Workshop RLARealLife.
10


Reviews:
[{"review_id": "uDRO7tw4gH8", "reviewer": "Reviewer_VywJ", "summary": "In this paper the authors present a method for safe reinforcement learning where the dynamics of the system are unknown, i.e. black box. The crux of the problem is how to synthesize safe controls given that no explicit analytical form for the system can be used. To do so, the authors design a safety index, a Lyapunov-like function, which implicitly defines the set of safe actions. If the nominal action to be taken is outside the safe set, a new action is produced, which is the L2 projection of the nominal action onto the safe set. The problem is that for black-box optimization problems this cannot be done efficiently without querying. The authors recognize that the optimal solution will always be at the boundary of the safe set of controls, and so they design AdamBA to find near-optimal points using a 2 phase algorithm. The first phase generates units vectors which are extended exponentially until the control bounds are exceeded, or the safe set is found. If the safe set is found then binary search is used to find safe controls at the boundary. If no safe controls are found, an anchor control point is found by uniform sampling, and AdamBA is executed again. In the result section, the authors showcase the algorithm on a Point Robot system from Safety Gym and show how PPO-ISSA is able to learn competitive policies when compared to PPO, PPO-Lagrangian and CPO, without violating safety. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper addresses a crucial important problem in robotics, which is that of learning while remaining safe.\n* The paper uses well-grounded concepts from control theory through the use of the safety index.\n\nWeaknesses:\n* It is not clear how well AdamBA can scale to higher dimensional control systems. As the control dimension increases I would expect the sampling strategies to fail or take too long.\n* In the same spirit as the previous comment, the experimental section deals only with a single system with only a 2-dimensional control space.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I did enjoy reading the paper, but I am afraid that without some more solid analysis and evidence for the scalability of the method to systems with more control dimensions, I would not accept it in its current form.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "As previously stated my main concern is with the scalability of the method. Most robotic systems have more than 2 control dimensions. To seriously consider this paper for acceptance I would expect a more realistic robotic system than a point robot."}}, {"review_id": "kf97xf555PY", "reviewer": "Reviewer_yCZE", "summary": "This paper presents a method for safe reinforcement learning based on the notion energy-based methods to ensure invariance of certain properties, or sets. Importantly for the context of modern RL, the method is model free and comes with the capability to reason about black box or unknown dynamics and provide a kind of barrier certificate at training time. The paper is well written and the ideas are sound. I have a few intellectual questions below, but the large part of my comments are editorial. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strengths of this paper include: organization, clarity of writing, comparison to related works and/or context of approach. The approach and its presentation in the paper is/are technically sound.\n\nI do not have any major suggestions for improvement, other than that the significance of the contribution is unclear to me. I am not saying there is *not* a significant amount of originality or impact, but based on my knowledge of the topic, this paper feels somewhat incremental. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Safety in robotics applications, particularly vis a vis reinforcement learning and black box models, is an important and active area. This is a sound contribution. Assuming that some of my comments are addressed, I believe this paper would represent a nice contribution to the conference. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "This is not a huge issue, but on lines 70 and 71 the authors use the word \"implicit\", which I believe I understand from the context; however, \"implicit\" is used in the related context of model-predictive control to mean something different. If the authors can think of a synonym, this might be helpful, e.g. \"unknown\"???\n\nIn section 2, is it trivial for the user to specify phi_0 for the safety specification? In practice, I believe this can be quite challenging - I recognize this is likely out of scope for the contributions of the paper but simply note this is a potential hurdle to adoption. \n\nLine 77: \"subjected to be modified\" reads a bit clunky. I suggest: \"subject to modification by the safe guard\". \n\nLine 87: why would X_s contain states that will inevitably go to an unsafe state? Why not simply define an invariant set and just call that the \"safe set\"? It is a bit strange to call something a safe set when it will inevitably lead to an unsafe state - perhaps X_S should be called something else? Just a generic 0-sublevel set and remove the word \"safe\"? Sorry, this is just a bit strange to me, although I understand the technical rationale for needing a subset.\n\nLine 95: These methods *are* able to... (typo)\n\nMinor but general comment: the paper is well-written, including very strong prose. However, there are several typos, including things like subject-verb agreement, etc. Please clean this up in the final (again, not a huge deal but worth a bit of extra effort, grammerly, etc)\n\nLine 119: perhaps we will get to this later in the paper, but the authors suggest that safe control is guaranteed, regardless of the geometry of the safe set. Can the set be disjoint or unconnected?\n\nIn lines 120-143, the authors provide a sketch of safe control, focusing on safe set algorithms. Utlimately, most of the detail is in Appendix A, which is fine to some extent, but for \"self contained-ness\", the paper would be improved by helping the reader a bit more here. I am struggling to come up with a helpful suggestion here, but hopefully this helps: instead of vaguely discussing things like adding higher-order terms, the discourse could be made more precise by directly including the details from appendix A (i.e. equation 5 and the subsequent text), as well as the discussion in appendix A about what differs between this paper and the literature. I recognize that there are space constraints, but I believe this would improve readability\n\nGeneral question: is it correct to say that SSA is a generalization of control barrier functions? \n\nNote that in line 149-150, the authors essentially do what I had asked earlier about the term \"implicit\". Perhaps this statement could be used earlier, although this is again not a huge deal\n\nEquation 3 brings to bear the fact that (I think) obstacles are assumed to be static. I might have missed this assumption in the intro or elsewhere, but it is worth making this assumption clear\n\nIs the approximation summarized in line 158, and described in Appendix B, guaranteed to be a conservative approximation? I do not believe that either the main body or App B actually makes this assertion and/or proof. Appendix G provides a numerical example but no proof of conservatism. I could be wrong here - if that is the case, please make this clear either in section 4 or Appendix B, or both\n\nAdamBA seems to be \"expensive\", and is subject to the number of unit gradient vectros that are initialized, the dimensionality of the safe control set, etc. Can the authors comment on computational cost?\n\nThis is an extremely minor point (almost embarrasing to say this), but the gradient directions in figure 1(a) are slightly misaligned from those in (b)-(d), especially the 3 vectors that are pointed \"down\". It seems like an easy fix and is worth it, in my opinion, to clarify the AdamBA method\n\nThe authors state that the reward performance of PPO-ISSA is comparable to the other methods, but I am not sure this claim is justified. The method constitently has lower performance than the other methods. It might be better to couch this argument in terms of the \"cost\" of a safety violation and emphasize that there is a tradeoff here, and (depending on how performance and safety are traded off) ISSA does better\n\nPPO is a reasonable starting place to provide a baseline. I have no problem with this, but I suggest the authors justify this choice, somewhere in the neighborhood of line 236. For example, one reason this is a good baseline is because (a) PPO is relatively state-of-the-art but (b) is generic and (c) already has safety-constrained derivatives that can be tested off the shelf. There is so much going on with RL that it is impossible to have a perfect baseline; again, I agree with the choice but feel that the text could provide a stronger justification. \n\nThe figures are in strange locations, which I understand: there are space limitations and typesetting is difficult. However, figure 6, for example, belongs closer to the body text that discusses it. On line 275, figure 7 is mentioned but the figure immediately to the right is figure 6. Perhaps after all other revisions are complete, the authors could attempt to re-optimize in terms of figure placement. Again, not a huge deal and this is not a technical comment, but a readability comment. \n\nIn the conclusion, the authors report 98% pm 22% reward performance relative to SotA methods. The \"zero safety violation\" claim is easy enough to see from the paper, but I am not sure where this 98% number comes from. It does not make sense based on figure 5, and I am unclear about the method used to derive these numbers. Did I miss it? Is it in the appendix? I believe the number is derived from section G.2.4, but this section seems to only include one scenario. "}}, {"review_id": "NCsXV6SYq63", "reviewer": "Reviewer_2HkF", "summary": "This paper proposes the model-free Implicit Safe Set Algorithm (ISSA), a framework for zero-violation control that uses energy function-based methods to ensure robot safety (represented in this paper by collision avoidance) when only a black-box dynamics simulator is available, rather than an analytical model. ISSA is inspired by the model-based safe set algorithm of Liu and Tomizuka (2014), developed with the goal of extending their algorithm's capabilities to systems where only a black-box simulator is available. To formulate ISSA, a design rule is proposed to synthesize the energy function (termed the safety index in this paper) so that a feasible control is available at any state, and a sample-efficient algorithm for constrained black-box optimization is proposed for sampling the control space to find a control that dissipates energy and maximizes reward. ISSA is evaluated over several Safety Gym benchmark problems, serving as a safety layer used in conjunction with the PPO reinforcement learning algorithm. The resulting PPO-ISSA framework, in exchange for a modest reduction in rewards, avoids collision entirely while competing RL algorithms do not. \n\n\n\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n\n-The paper proposes a practical approach for achieving a well-motivated and highly desirable capability - avoiding collision entirely in the course of learning, in the absence of an analytical dynamics model for the system of interest. \n\n-The paper is well-organized and includes helpful illustrative figures to explain the architecture and working principles of the proposed algorithm, the settings in which it is evaluated, and the results of the experiments performed. \n\n-The authors establish proof of the beneficial properties of forward invariance and finite-time convergence for the proposed algorithm.\n\n-The authors have made their source code freely available as a supplement to their paper. \n\nWeaknesses: \n\n-The paper describes a framework for safe control that can be highly useful for learning, but is not a learning method at its core.\n\n-The safety index design rule adopted in the paper is specific to the problem of collision avoidance for a mobile robot. \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I believe the algorithm proposed and evaluated in this paper, intended for use as a safety layer in a reinforcement learning framework, constitutes a novel and highly useful capability when safety is a primary concern. Although the proposed framework is composed of a collection of arguably incremental advances, there seems to be great potential for impact and adoption by others in safe RL applications. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "\n-The safety index design rule used in the paper is specific to collision avoidance by a mobile robot. It would be helpful if the authors could clarify the intended pathway toward more general, systematic formulation of appropriate design rules for the various systems of interest for safe RL.  \n\n-Table 1 is helpful in addressing the dependence of ISSA's computational expense on the number of vectors, but the normalization of the required computation time also obfuscates ISSA's compatibility with real-time robotics applications. It would be helpful if the authors could comment further on the computational expense of using an ISSA safety layer, and the computing arrangements that would be needed to support its use for safe real-time control in the course of learning with real robotic systems.\n\n-Here are a few grammatical and typographical errors that I noticed while reading the paper: \n--Line 95: \"methods is able\" should be \"methods are able\"\n--Line 107: \"methods still exploits\" should be \"methods still exploit\"\n--Line 119: \"property of set\" should be \"properties of the set\"\n--To my knowledge, safeguard is typically spelled as one word, rather than \"safe guard\" as used throughout the paper\n"}}, {"review_id": "8ew8WkYaK0f", "reviewer": "Reviewer_nDua", "summary": "This paper proposed a method, Implicit Safe Set Algorithm (ISSA), which focuses on collision avoidance. ISSA contains two parts: a parameter design rule and a sample-based optimization method. Firstly, the original constraints are reformulated into the Safe Index Form [1] that includes hyperparameters. And the hyperparameters should satisfy the parameter design rule, which incorporates the maximum velocity and acceleration. Secondly, a sample-based line search method AdamBA is applied to find the closest control action w.r.t the nominal action from the RL agent. The proposed method is tested in the Safety Gym environment.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. This reformulation of the constraints considers the forward invariance and the finite-time convergence. This reformulation is beneficial for real robot applications.\n2. The proposed method can be applied to other RL algorithms.\n\nWeaknesses\n1. Applying the proposed action search method AdamBA in high dimensional action space is challenging. You will need more samples to find a feasible direction or even not able to find one. Besides, the feasible action could highly deviate from the action sampled from the policy. This deviation could potentially destroy the learning process of the value function. In addition, it’s not possible to ensure that the optimized actions are the same at each time .\n2. Improper baselines. The proposed method focuses on the safe exploration problem. But the author chose safe policy search methods as baselines. For comparison, safe exploration methods, e.g., Safety Layer[2] and SDDPG/SPPO[3], should be better choices.\n3. The paper requires a forward dynamics model for the sample-based optimization. The claim of “model-free” is wrong. \n4. The experiment only runs in the Safety Gym environment. It would be better to have some simulations with physical robots.\n5. Each experiment has only three runs which can be highly biased. My suggestion is to run 25 random seeds to justify the claim for the RL problem.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper builds upon the work [1] and proposes a rule to choose the hyperparameter. The consideration of the forward invariance and the finite-time convergence is essential for robotics. The Safety Index Rule is determined based on the known dynamics model of the 2d differential drive robots. However, the proposed method is not model-free. In addition, the author did not choose proper baselines in comparison. And three runs for RL experiments are not sufficient to prove the performance.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "1. There is no Figure 10a in the appendix. \n2. Need more explanation about the safety index rule. It is not clear how the rule is determined.\n3. “ It can be shown that the ISSA algorithm is guaranteed to find a feasible solution ...”. How?\n\n[1] Liu, C., & Tomizuka, M. (2014). Control in a safe set: Addressing safety in human-robot interactions. ASME 2014 Dynamic Systems and Control Conference (DSCC), 3(November). https://doi.org/10.1115/DSCC2014-6048\n\n[2] Dalal, G., Dvijotham, K., Vecerik, M., Hester, T., Paduraru, C., & Tassa, Y. (2018). Safe Exploration in Continuous Action Spaces. ArXiv Preprint ArXiv:1801.08757.\n\n[3] Chow, Y., Nachum, O., Faust, A., Duenez-Guzman, E., & Ghavamzadeh, M. (2019). Lyapunov-based Safe Policy Optimization for Continuous Control. Reinforcement Learning for Real Life (RL4RealLife) Workshop in the 36 Th International Conference on Machine Learning. "}}]

Author rebuttal:

Meta-review: ### Final Meta-Review

The authors have carefully responded to the questions and concerns raised in the reviews and meta-review; they have made improvements to the clarity and rigor of the exposition in response to issues brought up by reviewers; and they have included additional simulation results for a high-dimensional nonlinear system (quadrupedal robot) to provide empirical evidence of the method's scalability.

These changes have significantly strengthened the final manuscript and, together with the thoughtful clarifications offered during the discussion, have brought all reviewers to an agreement on a positive recommendation for the paper, which I am happy to endorse.

### Original Meta-Review

The paper proposes a safe reinforcement learning approach that uses a black-box dynamics function to ensure safety. The reviewers agree on the importance of the topic and overall find the paper to be clearly written. However, the reviewers bring up a number of concerns, primarily regarding the exposition and results.

There appear to be a number of unstated assumptions made throughout the paper that detract form the claimed generality of the approach. Reviewers yCZE and 2HkF point out that the safety index design rule (3) only applies to mobile robot collision avoidance and seems to implicitly require obstacles to be static, something that is not stated in the text. There is also reviewer concern that the exposition of the proposed approach is not sufficiently self-contained, with some critical information relegated to the Appendix.

Evaluation results are limited to a comparatively simple dynamical system; reviewers Vyw and nDua indicate likely scalability challenges with the control space dimension, which are not addressed in the paper. There are further reviewer concerns that the number of simulation runs is insufficient for statistical significance, and that the chosen baselines may not provide an appropriate comparison because they are not fully safe learning methods, which could lead to the incorrect perception that the proposed method is the first to enable zero-violation learning.

Overall, the authors must respond to the detailed issues raised by the reviewers around the paper's claims, technical exposition, and experimental evaluation.

In addition, I ask that the authors clarify the following issue that I encountered while preparing this meta-review: given that there are no stated assumptions on the system dynamics f(x,u), the Safety Index Design Rule (3) is implied to be valid for extremely general cases, and the results in Proposition 1 and Theorem 1 therefore seem quite strong as stated. However, in the Appendix it is revealed that both of these results rely on the dynamics corresponding to a differential drive robot. This seems very restrictive (especially for an assumption that only appears in the Appendix), and appears to violate a central claim of the paper: if constructing a safety index requires knowledge of the specific form of the dynamics f(x,u), this would mean that that this is not in fact a black-box-compatible method.
Decision: Accept (Poster)