Safe Nonlinear Control Using Robust Neural Lyapunov-Barrier Functions
Authors: Charles Dawson
Zengyi Qin
Sicun Gao
Chuchu Fan
Venue: CoRL2021 Poster (2021)
Keywords: Certified control
learning for control

Abstract: Safety and stability are common requirements for robotic control systems; however, designing safe, stable controllers remains difficult for nonlinear and uncertain models. We develop a model-based learning approach to synthesize robust feedback controllers with safety and stability guarantees. We take inspiration from robust convex optimization and Lyapunov theory to define robust control Lyapunov barrier functions that generalize despite model uncertainty. We demonstrate our approach in simulation on problems including car trajectory tracking, nonlinear control with obstacle avoidance, satellite rendezvous with safety constraints, and flight control with a learned ground effect model. Simulation results show that our approach yields controllers that match or exceed the capabilities of robust MPC while reducing computational costs by an order of magnitude. We provide source code at github.com/dawsonc/neural_clbf/.

Safe Nonlinear Control Using Robust Neural Lyapunov-Barrier Functions
Charles Dawson', Zengyi Qin', Sicun Gao², Chuchu Fan!
Massachusetts Institute of Technology, {cba, qinzy, chuchujemis edu University of California, San Diego, sicungeuesd. edu
Keywords: Certified control, learning for control
Safety and stability are common requirements for robotic control sys- however, designing safe, stable controllers remains difficult for nonlinear and uncertain models. We develop model-based learning approach to synthesize robust feedback controllers with safety and stability guarantees. We take inspira- tion from robust convex optimization and Lyapunov theory to define robust control Lyapunov barrier functions that generalize despite model uncertainty. We demon- strate our approach in simulation on problems including car trajectory tracking, nonlinear control with obstacle avoidance, satellite rendezvous with safety con- straints, and flight control with learned ground effect model. Simulation results show that our approach yields controllers that match or exceed the capabilities of robust MPC while reducing computational costs by an order of magnitude. We hclpr//c/uauaa/bb/...o
Abstract:
tems;
provide source code at gi thub.
1 Introduction
Robot control systems are challenging to design, not least be- cause of the problems of task complexity and model uncer- tainty. Robotics control problems like those in Fig. 1 often involve both safety and stability requirements, where the con- troller must drive the system towards goal state while avoid- ing unsafe regions. Complicating matters, the model used to design the controller is seldom a perfect representation of the physical plant, and so controllers must account for uncertainty in any parameters (e.g. mass, friction, or unmodeled effects) that vary between the engineering model and true plant. Auto- matically synthesizing safe, stable, and robust controllers for nonlinear reach-avoid tasks is a long-standing open problem in controls. In this paper, we address this problem with novel approach robust model-basee -ased learning. Our work presents unified framework for handling both model uncertainty and complex safety and stability specifications.
UAI Navigation
Trijectary
Figure 1:
Safe control problems considered in Section 6.
Over the years, several approaches have been proposed to solve this problem. In one view, reach- avoid can be treated as an optimal control problem and solved using model predictive control (MPC) schemes and their robust variants. Robust MPC promises method for general-purposs controller synthesis, finding an optimal control signal given only a model of the system and a specification of the task. However, there are number of recognized disadvantages of robust MPC. First, there are currently no techniques for guaranteeing the safety, stability, or recursive feasibility robust MPC beyond the linear case [1]. Second, model uncertainty (e.g. mass or friction) is often multiplicative in the dynamics, but robust MPC is typically limited to additive uncertainty [1, 2]. Finally, MPC is computationally expensive, making it difficult to achieve high control frequencies in practice [3].
An alternative method for synthesizing safe, stable controllers comes from Lyapunov theory, through the use of control Lyapunov and control barrier functions (resp., CL.Fs and CBFs, [4] certificates that prove the stability and safety of control system, respectively. CLFs and CBFs are similar to standard Lyapunov and barrier functions, but they can be used to synthesize controller rather than
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

just verifying the performance of a closed-loop system. Unfortunately, CLF and CBF certificates are very difficult construct in general. particularly for systems with nonlinear dynamics [5].
The most recent set of methods promising general-purpose controller synthesis come from the field of learning for control; for instance, using reinforcement learning [6, 7] or supervised learning [8, 9, 10, 11]. However, the introduction of learning-enable components into safety-critical control tasks raises questions about soundness, robustness, and generalization. Some learning-based control techniques incorporate certificates such as Lyapunov functions [8], barrier functions [12, 10, 13], and contraction metries [9, 11] to prove the soundness of learned controllers. Unfortunately. these certificates' guarantees are sensitive to uncertainties in the underlying model. In particular, if the model used during training differs from that encountered during deployment, then guarantees on safety and stability may no longer hold.
Our main contribution is a learning-based framework for synthesizing robust nonlinear feedback controllers from safery and stability specifications. This contribution has two parts. First, we pro- vide a novel extension of control Lyapunov barrier functions robust control, defining robust control Lyapunov barrier function (robust CLBF). Second, we develop a model-based approach to learning robust CLBFs, which we use derive a safe controller using techniques from robust con- vex optimization. Other methods for learning Lyapunov and barrier certificates exist, but a key advantage of our approach that we learn certificates with explicit robustness guarantees, enabling generalization beyond the system parameters seen during training. We demonstrate our approach on range of challenging control problems, including trajectory tracking, nonlinear control with obstacle avoidance, flight control with learned model of ground effect, and satellite rendezvous problem with non-convex safety constraints, comparing our approach with robust MPC. In all of these experiments, we find that our method either matches or exceeds the performance of robust MPC while reducing computational cost at runtime by at least a factor of 10. We provide source code at gi thub. com/ daws wwwoonnnn//nunnaeeeeeeeeeeaaaaae aeeural_cllb and video at youtu be 4WWVL
2 Related Work
This work builds on a rich history of crrtficate-basee control theory. including classical Lyapunov functions well as more recent approaches such as control Lyapunov functions (CLFs [14, 15]) and control barrier functions (CBFs [16], a generalization of artificial potential fields [17). The majority of classical certificate-bast bassed controllers rely hand-designed certificates [18, 19], but these can be difficult to obtaín for nonlinear or high-dimensional systems. Some automated techniques exist for synthesizing CLFs and CBFs; however, many of these techniques (such as finding a Lyapunov function as the solution of a partial differential equation) are computationally intractable for many practical applications [5]. Other automated synthesis techniques are based on convex optimization, particularly sum-of-squares programming (SOS, (20), but are limited to systems with polynomial dynamics and do not scale favorably with the dimension of the system.
A promising line of work in this area is to use neural networks to learn certificate functions. These techniques range in complexity from verifying the stability of a given control system [21, 22] to simultaneously learning a control policy and certificate [9, 8, 10]. Most of these works do not explicitly consider robustness model uncertainty, although contraction metries may be used to certify robustness to bounded additive disturbance [9].
Most approaches to handling model uncertainty in the context of certificate-guided learning for control involve online adaptation. For example, [18, 23] assume that a CLF or CBF are given and learn the unmodeled residuals in the CLF and CBF derivatives. When combined with QP- based CLF/CBF controller, this technique enables adaptation to model uncertainty but relies on a potentially unsafe exploration phase. Although safe adaptation strategies exist, the main drawback with these techniques is their reliance on a hand-designed CLF and CBF, which are non-trivial to synthesize for nonlinear systems. Additionally, combined CLF/CBF controllers are prone to getting stuck when the feasible sets of the CLF and CBF no longer intersect.
Online optimization-ba based control techniques such as model-predictive control (MPC) are also rel- evant as a general-purpor control synthesis strategy. However, the computational complexity of MPC, and particularly robust MPC, is a widely-recognized issue, particularly when considering de- ployment to resource-consttaiied robotic systems such as UAVs [1,3]. 3]. We revisit the computational cost of robust MPC, particularly as compared with the cost of our proposed method, in Section 6.

---

Some approaches apply learning to characterize uncertainty in system dynamics and augment a ro- bust MPC scheme [24], but these methods do not fundamentally change the computational burden of MPC. Other methods rely on imitation learning to recreate an MPC-hased policy online [25], but these methods can encounter difficulties in generalizing beyond the training dataset.
A number of techniques from classical nonlinear control also deserve mention, such as sliding mode and adaptive controllers. These methods do not directly support state constraints and must be paired with a separate trajectory planning layer [26]. Another drawback is that these techniques require significant effort manually derive appropriate feedback control laws, and we are primarily interested in automated techniques for controller synthesis.
3 Preliminaries and Background
We consider continuous- time,, control-affine dynamical systems of the form à fz) + go (z)),, where E X' C RR, u E IR', and Se : R - IR" and ge R + Raxé are smooth functions modeling control-afiine nonlinear dynamics. We assume that fe and go depend on model parameters 0 E θ C '' and are affine in those parameters for any fixed z. This assumption on the dynamies is not restrictive; it covers many physical systems with uncertainty in inertia, damping, or friction (e.g. rigid-body dynamics additive and multiplicative disturbance as a special case. We also assume that fo and go are Lipschitz but make no further assumptions, allowing us to consider cases when components of fo and go learned from experimental data. For concision, we will use f and g (without subscript) to refer to the dynamics evaluated with nominal parameters 0g E O. In this paper, we consider the following control synthesis problem:
systems described by the manipulator equations), and it includes bounded
Definition 1 (Robust Safe Control Problem). Given a control-affine system with uncertain parameters 0 E O, goal configuration Tgoal, a set of unsafe states Kunsato S X, and a set of safe states Nsato C. X (such that Xeate Xunsate Reate). find a control policy u = π(x) such that all trajectories x(t) satisfying á
0 and Tgoal
fo(x) + go(x))(z) and z(0) E Xeate have the following properties for any parameters A: Reachability of Igoal with tolerance
Safety: xxt1)= E Xaate implies x(t2) E
o: lime- [|=[(]) - Igoail S
Xunsate V Vl2 2 41
Simply put, we wish to reach the goal dggoal while avoiding the unsafe states Xunsafe- We use the notion of reachability instead of asymptotic stability to permit (small) steady-state error; in the following we will use "stable" as shorthand for reachability. Note that we do not require Xaate U
Xunsafe X, as it will be made clear in the following discussion that need non-empty boundary layer X' \ (Katte U Xunsate) to allow for flcxibility in finding a safcty certificate.
Lyapunov theory provides tools that are naturally suited to reach-avoid problems: control Lyapunov functions (for stability) and control barrier functions (for safety [4]). To avoid issues arising from learning two separate certificates, we rely on single, unifying certificate known as a control Lya- punov barrier function (CLBF). Our definition of CLBFs is related those in [27] and [28] (differing from the formulation in [27] by a constant offset c, and differing from [28] where safety and reacha- bility are proven using two separate CLBFs). We begin by providing a standard definition a CLBF in the non-robust case, but in the next section we provide novel, robust extension of CLBF theory before demonstrating how neural networks may be used to synthesize these functions for a general class of dynamical system. In the following, we denote Ly V as the Lie derivative of V along f. Definition (CLBF). A function V : X > IR is a CLBF if, for some C, A >> O.
(irgoal) = 0 ((x) > VV x E X' \ Ignal
(1a)
V(x) ScVx E Xanto
(1c)
(Ib)
V(x) > cV E Xunsafe
(Id)
inf T.y V + V. (V 50V X Tgoal
(le)
Intuitively, we can think of CLBF as special case of a control Lyapunov function where the safe and unsafe regions are contained in sub- and super-level sets, respectively. If we define a set of
admissible controls K K(x) {u T. V + Vu + AV < 0). then we arrive at a theorem proving
the stability and safety of any controller that outputs elements of this set (the proof is included in the
supplementary material).
Theorem 1. (x) is a CLBF then any control policy x(x) E K(x) V E X will be both safe and stable, in the sense of Definition! 1.
3

---

Based on these results, we can define a CLBF-based controller, analogous to the CLF/CBF-based controller in [18] but without the risk of conflicts between the CLF and CBF conditions, relying on the CLBF V and some nominal controller nomiinal (e.g. the LQQR policy):
TCLBF = arg min
||u - Tnominal (x)ll²
(CLBF-QP)
L,V + Lg + AV 0
It should be clear that (C) E ((x) v E X Tggaal so this controller will result in system that is certifiably safe and stable (with the CLBF V actinn as the certificate). The nominal control signal Tnominal is included to encourage smoothness in the solution TCLBr (z)), particularly near the desired fixed point at Tgoul where V becomes small. CLBFs provide single, unified certificate of safety and stability; however, some significant issues remain. In particular, how do we guarantee that CLBF will generalize beyond the nominal parameters?
4 Robust CLBF Certificates for Safe Control
In this section, we extend the definition of CLBFs to provide explicit robustness guarantees, and we present key theorem proving the soundness robust CLBF- based control.
Definition 3 (Robust CLBF, ICLBF). A function V X R is robust CLBF for bounded
parametric uncertainty E O where θ is the convex hull of scenarios 01, 01,02,--.0 02,,0.,000 On, if the standard
CLBF conditions (1d) hold, the dynamics f and g are affine with respect 0, and V
X \ Tgoal there exist c, X > such that
inf Lgg, V Lo,, Vu + (V SO l. "
(3)
As in the non non-robust case, we define the set of admissible controls for a robust CLBF, (x) = [u] Lis, V + Lg, Vu Vu+ =ivi= O...., ny } and the corresponding Qa-based controller, the soundness of which is given by Theorem 2:
#CCLB = arg min || ग nomiinal
(rCLBF-QP)
s.t. Lio, V Lag, Vu XV 50;
(4)
Theorem 2. (x) a robust CLBF then any control policy T(x) E KK(x)) V X will be both safe and stable, in the sense of Definition 1, when executed on system fo. 9o with uncertain parameters (where the convex hull of scenarios Bu...., )..).
Proof. See the supplementary materials.
This result demonstrates the soundness and robustness of an rCLBF-based controller, but does not provide means to construct valid rCLBF. In the next section, we will present an automated model- based learning approach to rCLBF synthesis, yielding a general framework for solving robust safe control problems even for systems with complex, nonlinear, partially-Ieann dynamics.
5 Learning Robust CLBFs
persistent challenge in using of certificate-hasel controllers is the difficulty of finding valid cer- tificates. especially for systems with nonlinear dynamics and complex specifications of Xsate and Xunsade (e.g. obstacle avoidance). Taking inspiration from recent advances in certificate-guide learning for control [8, 10], we employ a model-based supervised learning framework to synthesize an (CLBF-based controller. The controller architecture is comprised of three main parts: the rCLBF V. a proof controller nNN, and the Qa-based controller (rCLBF-QP)) We parameterize V and nN X + R as neural networks. These networks are trained offline, where NNN is used to prove that the feasible set of (ICLBF-QP) is non-empty, then is evaluated online provide the parameters of (rCLBF-QP), which is solved to find the control input. In the offline training stage, our primary goal is finding an rCLBF ((x) such that the conditions of Definition 3 are satisfied. To ensure (1b), we define V(x) = w (r))(()) 2 0, where w is the activation vector of the last
IR

---

hidden layer of the V neural network. To train V such that conditions (la), (Ic), (Id), and (3) are satisfied over the domain of interest, we sample Nirain points uniformly at random from X to yield a population of training points then define the empirical loss:
C.CLF (a 2 01 Nsate
1
Σ
-cl_ + ag Nunsafe
1
Σ (-+c-V(x)]
zER
Byy
03
Σ
r((x)
Σ
E
Lio, V(x) + Lge, +
(5)
Fly NayNur train
where d1-03 are positive tuning parameters, > 0 is small parameter (typically 0.01) that allows
us to encourage strict inequality satisfaction and enables generalization claims, Neate and Nunsafe are
the number of points in the training sample in Xeaae and Xunssfe. respectively, and [o]+ max(o, 0)
is the ReLU function. The terms in this empirical loss are directly linked to conditions (Ia), (Ic), (Id), and (3) such that each term is zero if the corresponding condition is satisfied at all Nersin train- ing points. For example, the final term in this loss is designed to encourage satisfaction of the robust CLBF decrease condition (3). The factor r(z) in the final term computed by solving (rCLBF-QP) at each training point and computing the maximum violation of constraint (4), such that r(x) 0 when the QP has feasible solution and r(z) > otherwise. This loss optimized using stochastic gradient descent, alternating epochs between training the V and NNN networks. During training, we rely on NN to compute the time derivative of V(x) the final term of the loss. To provide a
training signal for TNN, we define an additional loss C = ||TNN - Tnominatll where noominal is nominal controller (e.g. a policy derived from LQR approximation). The parameters of V and TNN are optimized using the combined loss L = LCLBF + (10-5) The small weight applied to f ensures that the training process prioritizes satisfying the CLBF conditions.
An important detail of our control architecture is that the learned control policy TNN is used pri- marily to demonstrate that the feasible set of (rCLBF-QP) is non-empty. We are not required to use TNN at execution time; we can choose any control policy from the admissible set K, (z). In the on- line stage, we rely on an optimization based controller (rCLBF-QP), which solves small quadratic program with n, constraints and variables (one for each element of u). To ensure that this QP is feasible at execution, we permit relaxation of the CLBF constraints (4) and penalize relaxation with a large coefficient in the objective. Once trained, V can be verified using neural-network verifica- tion tools [29], sampling [30], a generalization error bound [10]. More details data collection, training, implementation, and verification strategies are included in the supplementary materials.
It is important to note that this training strategy encourages satisfying (3) only on the finite set of training points sampled uniformly from the state space: there is no learning mechanism that enforces dense satisfaction of (3). In the supplementary materials, we include plots of 2D sections of the state space showing that (3) is satisfied at the majority of points, but there is a relatively small violation on sparse subset of the state space. Because these violation regions are sparse, the theory of almost Lyapunov functions applies [31]: small violation regions may induce temporary overshoots (requiring shrinking the certified invariant set), but they do not invalidate the safety and stability assurances of the certificate. Strong empirical results on controller performance in Section 6 support this conclusion, though we admit that good empirical performance is not substitute for guarantees based on rigorous verification, which we hope to revisit in future work.
6 Experiments
To evaluate the performance of our learned rCLBF-QP controller, we compare against min-max robust model predictive control (as described in [2, 32]) on a series of simulated benchmark prob- lems representing safe control problems with increasing complexity. The first two concern trajectory tracking, where we wish to limit the tracking error despite uncertainty in the reference trajectory. The next two benchmarks are UAV stabilization problems that add additional safety constraints and increasingly nonlinear dynamics. The last three benchmarks involve highly non-convex safety constraints. The first four benchmarks provide solid basis for comparison between proposed method and robust MPC, while the last three demonstrate the power of our approach to generalize to maintain safety even in complex environments.
In each experiment, we vary model parameters randomly in θ simulate the performance of the controller, and compute the rate of safety constraints violations and average error relative to the
5

---

goal || Zgool| across simulations. These data are reported along with average evaluation time for each controller in Table 1. To examine the effect of control frequency on MPC performance, we include results for two different control periods dt for all robust MPC experiments (we also report the horizon length N). In some cases we observed that the evaluation time for MPC exceeds the control period; in practice this would lead to the controller failing, but in our experiments we simply ran the simulation slower than real-time. Our robust MPC comparison supports only linear models with bounded additive disturbance; we linearize the systems about the goal point and select an ad- ditive disturbance to approximate the disturbance from uncertain model parameters. The following sections will present results from each benchmark separately, and more details are provided in the supplementary materials, including the dynamics and constraints used for each benchmark, as well as the hardware used for training and execution.
Table 1: Comparison of controller performance under parameter variation
Task
Algorithm
Safety rate
||x Fanml
Evaluation time (ms)
Car trajectory tracking' Kinematic model
rCLBF-QP Robust MPC (dt = s, N =
0.7523
10.4
1.5148
194.6
(n = 5, = n. 2)
Robust MPC (dt - 0.25 N 6)
12.4438
172.8
Car trajectory tracking' Sideslip model
xCLBI--QP
1.0340
9.6
Robust MPC (dt 0.1ss. N 5)
0.1560
336.5
(n = 7,8 n, 2)
Robust MPC = 0.25 N 5)
18.1939
316.9
3D Quadrotor
rCLBF-QP
100%
0.4647
9.7
(n = 9,8 = 4, 2)
Robust MPC (d) = 0.10s, N = 5)
100%
0.0980
316.2
Robust MPC (dt = 0.25 N = 5)
100%
63.6303
291.0
Neural Lander (n = 6, = n, 1)
iCLBF-QP
100%
0,1332
13.1
Robust MPC (dt 0.10s, N 5)
100%
0.2086
247.2
Robust MPC (dt 0.25 N
100%
0.3267
253.2
Segway (n = 4,8= 1,m. 4)
iCI.FFQP
100%
0.0447
4,4
Robust MPC (dt = 0.10 N = 5)
21%
1.3977
214.8
Robust MPC (dt = 0.25,, N = 5)
11%
1.9725
239.1
2D Quadrotor (n = 6,6 = n,n, 4)
ICLBF-QP
83%
18.6
Robust MPC (d) = 0.10s, N = 5)
5.3%
276.9
Robust MPC (dt = 0.25 s, N
0%
265.2
Satellite Rendezvous
rCLBF-QP
100%
0.1369
8.2
(n 4,
Robust MPC 0. N
39%
6,3751
187.3
Robust MPC 0.25 N =
15%
9.0592
197.4
For car trajectory tracking, we compute maximum tracking error over the trajectory.
For 2D quadrvior, we compute % of trials reaching the goal with volerance 5 = 0.3 without collision.
Note: We also implemented SOS optimization to search for CLBF and controller, but bilinear optimizatiinn (as in (33])
did not converge with maximum polynomial dogree 10 and Taylor expansion of the nonlincar dynamics.
6.1 Car trajectory tracking
First, we consider the problem of tracking an a priori unknown trajectory using two different car models. In the first model (the kinematic model), the vehicle state [Te. Be. 8, Ue, vee, representing error relative to the reference trajectory (5 is the steering angle). The second model (the sideslip model) has state [xe, Ve, ô, Ve, ve, Vie 3]. where 3 is the sideslip angle [34]. Both models have control inputs for the rate of change of ō and ve. We assume that the reference trajectory is parame- terized by an uncertain curvature: at any point the angular velocity of the reference point can vary on [-1.5, 1.3]. The goal point zero error relative to the reference, and the safety constraint requires maintaining bounded tracking error.
The performance of our controller is shown in Fig. 2. We see that for both models, both our controller and robust MPC are able to track the reference trajectory. However, robust MPC was only successful when run at slower than real-time speeds (with a control period dt 0.1 s roughly twice as fast the average evaluation time), MPC became unstable when run at a slower control frequency dt = 0.25 s. In contrast, our rCLBF-QP controller runs in real-time with control period of s 10 ms on a laptop computer. This significant improvement in speed is due primarily to the reduction in the size of (rCLBF-QP) relative to that of the QPs used by robust MPC. For example, for the sideslip model, our controller solves QP with 2 variables and 2 constraints, whereas the robust MPC controller solves a QP with 35 variables and 23 constraints (after pre-compiling using YALMIP [32]). Because the learned rCLBF encodes long-term safety and stability constraints into local constraints the rCLBF derivative, the rCLBF controller requires only single-step horizon (as opposed to the receding horizon used by MPC).

---

By comparing performance between these two models, we can discern an important feature of our approach. Increasing the state dimension when moving between models does not substantially crease the evaluation time for our controller (as it does for robust MPC), but it does degrade the tracking performance, suggesting that the number of samples required to train the CLBF to any given level of performance increases with the size of the state space. These examples also highlight a potential drawback our approach, which relies on a parameter-invariun robust CLBF. Because it attempts to find common rCLBF for all possible parameter values, controller exhibits some small steady-state error near the goal. This oceurs because there no single control input that ren- ders the goal fixed point for all possible parameter values and motivates use of goal-reaching tolerance in Definition
Caltsiin Thoddng Elsor (n)
XC18--GP
Virro
ICLBF OP HMPC (4P 0.1]
IMPC 0.1)
(.25
125
1MPC
TU
25
Figure 2: Trajectory tracking on kinematic (left) and sideslip (right) vehicle models, with contour plots of V. Blue shows the e-level set.
6.2 UAV stabilization
The next two examples involve stabilizing quadrotor near the ground while maintaining minimum altitude. Relative to the previous examples, these benchmarks increase the complexity of the state constraints, and we consider two models with increasingly challenging dynamics. The first model (referred to as the "3D quadrotor") has 9 state dimensions for position, velocity, and orientation, with control inputs for the net thrust and angular velocities [9]. The second model (the "neural lander") has lower state dimension, including only translation and velocity, with linear acceleration as an input, but its dynamics include neural network trained to approximate the acrodynamic ground effect, which is particularly relevant to this safe hovering task [35]. The mass of both models is uncertain, but assumed to lie on [1.0, 1.5] for the 3D quadrotor and [1.47, 2.0] for the neural lander.
Fig. shows simulation results on these two models. The trend from the previous benchmarks continues: our controller maintains safety while reducing evaluation time by factor of 10 relative to MPC. Moreover, while robust MPC method can achieve low error relative the goal for the 3D quadrotor model, the nonlinear ground effect term prevents MPC from driving the neural lander to the goal. In contrast, the iCLBF-QP method can consider the full nonlinear dynamics of the system, including the learned ground effect, and achieves much lower error relative to the goal.
13 RF-GF
Craal
1.00
IVPC 0:3) WVCC (ut Butb)
GLLP OF
VX--0
Unahs
10
1.2
0.50
30
56%
-0.25
Figure 3: Controller performance for the 3D quadrotor (left) and neural lander (right), with contour plots of V. Blue shows the c-level set, white shows regions where condition (3) violated.
6.3 Navigation with non-convex safety constraints
The preceding benchmarks all include convex safety constraints that can be easily encoded in a linear robust MPC scheme. Our next set of examples demonstrate the ability of our approach to generalize to complex environments. These problems are commonly solved by combining planning and robust tracking control, so in our comparisons we use robust MPC to track a safe reference path through each environment. In contrast, our rCLBF-QP controller is not provided with a reference path and instead synthesizes a safe controller using only the model dynamics and (non-convex) safety con- straints, which is more challenging problem than the tracking problem as in Section 6.1. 1. The three

---

navigation problems we consider are: (a) controlling a Segway to duck under an obstacle to reach goal [36], (b) navigating a 2D quadrotor model around obstacles [9], and (c) completing a satellite rendezvous that requires approaching the target satellite from specific direction [37]. For (a) and (c), we conducted additional comparisons with Hamilton-Jacobi-baae policy trained via constrained policy optimization reinforcement learning (CPO, [39]). Simulated trajectories are shown in Fig. 4. Note that the Segway and satellite examples, robust MPC fails to track the reference path, while the rCLBF controller successfully navigates the environment. HJ preserves safety in the satellite example but fails to reach the goal (which is positioned near the bor- der of the unsafe region), while HJ controller synthesis failed in the Segway example (the backwards reachable set did not reach the start location with a 5s horizon). Note that the HJ satellite controller requires different initial conditions, since it will fail if started outside of the safe region. The policy trained using CPO navigates to the goal in the satellite example, but it is not safe. In the Segway example, CPO does not learn stable controller (details are given in the appendix).
controller (HJ, [38]) and
KLLF-OP
NPC (.11
HMP (
CNC
Inerta
Masa 1.1
(b) Mass 1.5
ABC E-X
(M-C 1ST 2-35
Hegoo
143
0.15
075
035
032
-0.5
30
Icl Mean-motion 1.2
(d) Mean-motion 1.5
-20
-70
20
18
400
-13
-10
Figure 4: Navigation problems solved using our rCLBF-QP controller, compared with robust MPC. Clockwise from right: satellite rendezvous, planar quadrotor, and Segway.
7 Discussion & Conclusion
These results demonstrate two clear trends. First, the performance of our controller (in terms of both safety rate and error relative the goal) comparable to that of MPC when the MPC con- troller is stable. In some cases, our method achieves lower steady-state error due to its ability to consider highly nonlinear dynamics, as the neural lander example. In other cases, the dynamics are well-approximated by the linearization and robust MPC achieves better steady-state error, but our approach still achieves a comparable safety rate. Second, we observe that the performance of the robust MPC algorithm is highly sensitive to the control frequency, and these controllers are only stable at control frequencies that cannot run in real-time on a laptop computer. This highlights one benefit of our method over traditional MPC, which trades increased offline computation for an order of magnitude reduction evaluation time. In all cases, we find that our proposed algorithm finds a controller that satisfies the safety constraints despite variation in model parameters, validating our claim of presenting a framework for robust safe controller synthesis.
In summary, we present novel, learning-based approach to synthesizing robust nonlinear feedback controllers. Our approach is guided by robust extension to the theory of control Lyapunov bar- rier functions that explicitly accounts for uncertainty in model parameters. Through experiments in simulation, we successfully demonstrate the performance of our approach on range of challenging safe control problems. A number of interesting open questions remain, including scalable verifica- tion strategies for V. the sample complexity of this learning method, and the relative convergence rates of V. TNN, and the QP controller derived from V, which we hope to revisit in future work. We also plan on exploring application hardware systems, including considerations of delay and state estimation uncertainty.
8

---

Acknowledgments
The NASA University Leadership Initiative (grant #80NSSC20M01633 and Defense Science and Technology Agency in Singapore provided funds to assist the authors with their research, but this article solely reflects the opinions and conclusions of its authors and not any NASA entity, DSTA Singapore, the Singapore Govermment. C. Dawson is supported by the NSF Graduate Research Fellowship under Grant No. 1745302.
References
[1] A. Bemporad and M. Morari. Robust model predictive control: A survey. In A. Garulli and A. Tesi, editors, Robustness in identification and control, pages 207-226, London, 1999. Springer London. ISBN 778-1-84628-53--1
[2] J. Löfberg. Approximations of closed-loop mpc. In Proceedings of the 42nd IEEE Conference on Decision and Control, pages 1448-1442, Maui, Hawaïi, 2003.
[3] W. S. Levine and S. V. Rakovic. Handbook of Model Predictive Control. Birkhäuser, Cham, 2019. ISBN 978-3-319-77488-6 401:10011077997833311777777888 springer. com 10. 9 101-11000077---1---7------
URL http://link
[4] A. D. Ames, X. Xu, J. W. Grizzle, and P. Tabuada. Control Barrier Function Based Quadratic Programs for Safety Critical Systems. IEEE Transactions on Automatic Control, 62(8):3861-
3876, Aug 2017. ISSN 00189286. 000:10.109//AAC22112266668
[5] Giesl and S. Hafstein. Review on computational methods for Lyapunov functions. Discrete and Continuous Dynamical Systems Series B, 20(8): Oct 2015. ISSN 15313492. doi: (00.3994/dcdsbb2215522222222 doi /101/1..0 333.333344/// /dcsbb 2015 20 22.91.
URL https:/ /tt::://wwwww www. w aimsciincsssssseee wwt../miieeeee...o/rrrii.....e..
[6] R. Cheng, G. Orosz, R. M. Murray, and J. W. Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In 33rd AAAI Conference on Artificial Intelligence, AAA/ 2019, volume 33, pages 3387-3399 AAAI Press, Jul 2019.
ISBN 9781577358091. 10i100. wttaaaaiivv33...... v33001.330000000 3330111.33
URL www aaai org.
[7] M. Han, L. Zhang, J. Wang, and W. Pan. Actor-Critic Reinforcement Learning for Control with Stability Guarantee. IEEE Robotics and Automation Letters, 5(4): Apr 2020. URL http: http://rxxvvv./vv org/ abss/0044 14288.
[8] Y.-. Chang. N. Roohi, and S. Gao. Neural Lyapunov Control. In Advances in Neural Infor- mation Processing Systems, volume 32, pages 3245-3254, 2019.
[9] D. Sun, S. Jha, and C. Fan. Learning Certified Control using Contraction Metric. In Conference on Robot Learning. Conference on Robot Learning, Nov 2020. URL abs / 22111 12569.
[10] Z. Qin, K. Zhang, Y. Chen, J. Chen, and C. Fan. Learning Safe Multi-Agent Control with Decentralized Neural Barrier Certificates. Conference on Learning Representations. Con- ference on Learning Representations, Jan 2021. URL http: /ttpivaxiiiv//.////ii//.a./a/.aa aogg/abbb/111.11 05436.
[11] H. Tsukamoto and S.J. Chung. Neural Contraction Metrics for Robust Estimation and Control: A Convex Optimization Approach. IEEE Control Systems Letters, 5(1):211-216, Jun 2020.
20000000000666 URL http: http//axiii././//////e/////.a../ wwtp:/i.org/abb/////// 2020. 3001646.
04361http: /x doi.
[12] Dean, Taylor, R. K. Cosner, B. Recht, and D. Ames. Guaranteeing Safety of Learned Perception Modules via cnasurement-Robuse Control Barrier Functions. In Conference Robot Learning. Conference on Robot Learning, Oct 2020. URL. http://tp/// //taaaivvvorrr abs/ 2bss2211... 16001.

---

[13] A. Peruffo, D. Ahmed, and A. Abate. Automated and Formal Synthesis of Neural Barrier Certificates for Dynamical Models. arXiv, Jul 2020. URL http://azxiv.commm www.wvo.rrrreeee /bs// 2007.00 03251.
[14] Z. Artstein. Stabilization with relaxed controls. Nonlinear Analysis, 7(11):1163-1173, Jan 1983. ISSN 0362546X.
[15] A. D. Ames. K. Galloway, K. Sreenath, and J. W. Grizzle. Rapidly exponentially stabiliz- ing control дуарunov functions and hybrid zero dynamics. IEEE Transactions on Automatic Control, 8944)8776-81, 2014. ISSN 00189286. 1001000.11999AC2211422999999
[16] A. D. Ames, S. Coogan, M. Egerstedt, G. Notomista, K. Sreenath, and P. Tabuada. Control barrier functions: Theory and applications. In 2019 18th European Control Conference, ECC 2019, pages 3420-3431, Institute of Electrical and Electronics Engineers Inc., Jun 2019. ISBN 9783907144008
[17] A. Singletary, K. Klingebiel, J. Bourne, A. Browning, P. Tokumaru, and A. Ames. Comparative Analysis of Control Barrier Functions and Artificial Potential Fields for Obstacle Avoidance. arXiv, oct 2020. URL hitps: http:://aaxiv..iie/e org. httgg//bs/. abs 2010.00000000 098.9998.9l.
[18] Choi, F. Castañeda, C. J. Tomlin, and K. Sreenath. Reinforcement Learning for Safety- Critical Control under Model Uncertainty, using Control I Lyapunov Functions and Control Barrier Functions. Robotics: Science and Systems. Robotics: Science and Systems, Apr 2020. URL http: .rgg htt///bs/200000 07584.
[19] Castañeda, J. Choi, B. Zhang, C. Tomlin, and K. Sreenath. Gaussian Process-based Min- norm Stabilizing Controller for Control-Affine Systems with Uncertain Input Effects. arXiv, Nov 2020. URL http:
/ttpxvaxivvv..///////..///..a... wttg/br/////.....a/ 07183.
[20] A. A. Ahmadi and A. Majumdar. Some applications of polynomial optimization in opera- tions research and real-time decision making. Optimization Letters, 10(4):709-729, Apr 2016. ISSN 18624480. doi: 11211..1007//115100055550-0000
URL /ttp://////...o//111100000/
s11590- 015 0894 3--3-.
[21] A. Abate, D. Ahmed, M. Giacobbe, and A. Peruffo. Formal Synthesis of Lyapumov Neural Networks. IEEE Control Systems Letters, 5(3):733-778, Mar 2020. URL http://///aaxii erg/abs/205 2003, 08910.
[22] S. M. Richards, F. Berkenkamp, and A. Krause. The Yyарuпov neural network: Adaptive stability certification for safe learning of dynamical systems. In Conference on Robot Learning. arXiv, Aug 2018. URL http: : arxi www..i..........ee....oeeeeeeeee org/aas//////eeeeeee//e/eeeee 1808.88000 00924.
[23] A. J. Taylor, D. Dorobantu, H. M. Le. Y. Yue, and A. D. Ames. Episodic Learning with Con- trol Lyapunov Functions for Uncertain Robotic Systems. IEEE International Conference on Intelligent Robots and Systems, pages 6878 6884. Institute of Electrical and Electronies Engi- neers Inc., Mar 2019. doi: 10000000100//0044402222000226600 abs/ / 1 19933... 01 5 www17177ppp:: /xx doi org RRA.0//IRR044998000 000.009972229990 2019 8.8668600.
URL /
[24] D. Fan, A. Agha, and T. Evangelos. Deep Tubes for Tube MPC. In Robotics: Science and Systems, 2020. URL papers/81 1.merr......1...
[25] G. Kahn, T. Zhang, S. Levine, and P. Abbeel. PLATO: Policy Learning using Adaptive Trajec- tory Optimization. Proceedings IEEE International Conference on Robotics and Automation, pages 3342-3349, mar 2016. URL http::////a/.....oo http://axxiiv//// abs 1603. 60622.
[26] J.JJ. E. Slotine and W. Li. Applied nonlinear control: an introduction. Prentice-Hoi Hall, 1991.
[27] M. Z. Romdlony and B. Jayawardhana. Stabilization with guaranteed safety using Control Lyapunov-Barde Function. In Automatica, volume 66, pages 39-47, Elsevier Ltd, Apr 2016.
doi: 10. 101.0..11...auioo 1
10

---

[28] W. Xiao, C. A. Belta, and C. G. Cassandras. High Order Control Hrapunov-Barrics Functions for Temporal Logic Specifications. In 2019 American Controls Conference, ACC 2019, Feb 2021. URL https::////.mm. / /ttp///axvvvvoo/ abs / 2102.. 06787v1.
[29] C. Liu, T. Armon, Lazarus, C. Strong. C. Barrett, and M. J. Kochenderfer. Algorithms for verifying deep neural networks. Foundations and Trends in Optimization, 43-44))2444444 2021. doi: 10.1 1561/2400000335. URL //tp://////vvi// orgg abs 903 06758.
[30] R. Bobiti and M. Lazar. Automated -aaimattdd-amlling---- Based Stability Verification and DOA Estima- tion for Nonlinear Systems. IEEE Transactions on Automatic Control, 63(11):3659-367, Nov 2018. ISSN 15582523. doi:10. 10001100//AA...1.8..79999
[31] S. Liu, D. Liberzon, and V. Zharnitsky. Almost Iyapunov functions for nonlinear systems. Automatica, 113:108758, 2020.
[32] J. Löfberg. Automatic robust convex programming. Optimization methods and software, 27 (1):115-129, 2012.
[33] A. Majumdar, A. A. Ahmadi, and R. Tedrake. Control design along trajectories with sums of squares programming. In Proceedings IEEE International Conference on Robotics and Au- tomation, pages 4054-4061 2013. ISBN 978 1463366411.
149.
[34] M. Althoff, M. Koschi, and S. Manzinger. Commonroad: Composable benchmarks for mo- tion planning roads. In Proc. the IEEE Intelligent Vehicles Symposium, 2017. ISBN 9781 7781500 509048045. doi:10. 1109/vv..011..99558
[35] A. Liu, G. Shi, S.-J. Chung, A. Anandkumar, and Y. Yuc. Robust regressinn for safe exploration in control, 2020.
[36] K. J. Áström and R. M. Murray. Feedbbckk systems: an introduction for scientists and engi- neers. Princeton university press, 2021.
[37] C. Jewison and R. S. Erwin. A spacecraft benchmark problem for hybrid control and estima- tion. 2016 IEEE 55th Conference on Decision and Control (CDC), pages 3300-3305. IEEE, 2016.
[38] M. Mitchell and J. A. Templeton. toolbox of hamilton-jacohi solvers for analysis of non- deterministic continuous and hybrid systems. In Proceedings of the 8th International Confer- ence on Hybrid Systems: Computation and Control, HSCC'05, page 480-494 Berlin, Heidel- berg, 2005. Springer- Verlag. ISBN 3540251081. doi: 10. 111.11..000000---11 100000998----54003955422101 ht https: /ooi. org/oogg///.oooo.ooo 1007/100000000 9778- 3- 000---00-55-1 31444----00 3_.
URL
[39] J.
Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained policy optimization. In Proceedings the 34th International Conference on Machine Learning- Volume 70, ICML' page 22-31. JMLR.org, 2017.
[40] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. Detioo, M. Rai- son, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang. J. Bai, and S. Chintala. Py- torch: An imperative style, high-performance deep learning library.
In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, ed- itors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019. URL sctp:://papee 9015 5-55---trcch-ar http-soooooo-------iiiieeeee---e onppeetteeeeeeeee..o...rrreeeeee deep htt.r:/deeetttooommaaaaaaaeeeeaa pdf.
hwwppppppppp.ppppppppppppppp.ppp nww.ips wwtppppeeeccppppe//////e
[41]
Falcon, WA. highhign 3, 2019.
Pytorch lightning.
GitHub.
Note:
[42] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018. URI.
https: httpsss//ppenree/// ew net/ forum? d=B1 QRgziT-
11

---

[43] N. M. Boffi, S. Tu, N. Matni, J. E. Slotine, and V. Sindhwani. Learning stability certificates from data. Conference on Robot Learning. arXiv, Aug 2020. URL wttattpp//aaaa.. .rrg/ abs/ 2208/ 05952.
[44] L. Gurobi Optimization. Gurobi optimizer reference manual, 2021. URL http: / /www. gurobi.. com.
[45] S. Bansal, M. Chen, S. Herbert, and C. J. Tomlin. Hamilton-jacon reachability: A brief overview and recent advances. In 2017 IEEE 56th Anmual Conference Decision and Control (CDC), pages 2242-2253, 2017. doi: 10. 109/CCC..011..8666666
[46] W. Clohessy and R. Wiltshire. Terminal guidance system for satellite rendezvous. Journal of the Aerospace Sciences, 27(9):653-658 1960.
12


Reviews:
[{"review_id": "pcAJP-v0hLI", "reviewer": "Reviewer_o7Dq", "summary": "The paper formulates a novel approach for robust model-based learning based on control lyapunov functions (CLF) and control barrier function (CBF) in a unified framework that allows to produce a  robust lyapunov-like function that certifies stability and safety for a system with unknown but bounded parameters. The construction of the Lyapunov-like function is automatically generated from a supervised learning framework. The authors show the performance of the controller compared against robust MPC formulations on 7 different plants. They show that the robust control lyapunov function based QP (rCLBF-QP) offers more reliability on the safety rate and it outperforms robust MPC in most of the cases. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper has an incremental and very clear presentation of the rCLBF, expressing theorems that provide a sound formulation of the controller. \nThe idea of providing a lyapunov-like function that produces a certificate for stability and safety together provides a better framework for learning and optimization.\nMoreover, the conditions of the theorems are used to construct a Loss function for the neural network approximator of the lyapunov-like function, leaving less design work to hand tuned parameters or functions. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposes a novel formulation of a unifying lyapunov-like function that provides a robust certificate for stability and safety. The formulation allows an automatic way to produce such functions via supervised learning. ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "The paper might benefit with more discussion about the implementation details of the neural network, like the number of train points, or the parameters a1 - a3, for instance.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "QXdtBKQ3dJ2", "reviewer": "Reviewer_mZgd", "summary": "The main idea of the paper is to design feedback control systems with safety and stability guarantees by fitting robust Lyapunov/Barrier functions and policies, and then applying a standard pointwise min-norm control. The resulting controller is compared in simulation to robust linear MPC on a number of interesting examples.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) The authors are clearly well-versed in control theory. Material in Sections 1-4 provides a well-written overview of the main ideas around control Lyapunov functins (CLFs) and their close cousins the control barrier functions (CBFs). \n2) The main technical novelty is a proposed method for computing CLFs, CBFs, and policies as a supervised-learning problem, given in Section 5, although this builds fairly naturally on the contributions of citation [8] in the paper. \n3) The simulation results are interesting and demonstrate clear potential for the method. Experiments would be nice of course, but in the year of COVID it is understood that this was extremely difficult.\n\nWeaknesses:\n1) This is minor, but I think it could be more clearly stated in Definition 3 or Theorem 2 that the dynamics must be affine in \\theta for this approach to work. This is briefly mentioned at the start of Section 3 and is dismissed as not restrictive, however in many cases of interest in robotics, in order to achieve an affine parameterisation of uncertainty it is necessary to use quite conservative overparameterisations.\n2) Similarly, for parameter-dependent problems it is known that have parameter *independent* Lyapunov functions can be very conservative, however this is required for the proposed approach. The conservatism of this is not discussed.\n3) I think the main contribution of the paper is really the sampling and fitting scheme in Section 5, but the discussion of sampling is not very deep, it just states that N_train points are sampled uniformly at random from X. \n- Is X bounded? If not, how does one sample uniformly? If it is, how do we know X is robustly forward-invariant? Usually a Lyapunov function would be used to verify such a property, but this is assumed before the search for a Lyapunov function begins. \n- Why uniform sampling? The method proposed is analogous to an actor-critic algorithm in RL with the \"critic\" in this case being a CLBF, and the Lyapunov condition replacing the Bellman equation. In RL uniform sampling of the state space is rare, and sampling design is indeed fundamental to the problem of \"exploration vs exploitation\", as well as classical experimental design.\n4) I'm a little uneasy about claiming safety based on condition (3a) being satisfied only on a finite collection of sample points. The paper says \"once trained, V can be verified using...\" and gives some references. But these are quite general in nature, it is not entirely clear how they would be adapted to the present problem. Would they prove that (3a) holds globally?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall I enjoyed reading the paper and think it has the beginnings of a very useful approach. I have some reservations about points of detail, as discussed above, at least if the objective is for feedback systems that are really provably safe.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "As above, I think the paper would be significantly improved if:\n- The technical requirements of affine parameterisation and parameter-independent Lyapunov functions were discussed more clearly\n- The question of sampling training points was clarified and explored in more detail\n- It was clarified how the learned CLBF could be rigorously verified."}}, {"review_id": "72PSVXG9smm", "reviewer": "Reviewer_JDXV", "summary": "This paper proposes a somewhat novel definition of a Control Lyapunov-Barrier function. Such a function is approximately synthesized by learning it as a neural network using samples drawn from a robot's state space as training data. The method is demonstrated successfully on a variety of examples of increasing complexity.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS\n- The first three sections are really well-written and easy to follow, much better than many other papers I've read\n- The core idea is clear and well-founded in controls theory; I love seeing this effort to merge controls and learning literature.\n\nWEAKNESSES\n- The proposed CLBF is not very surprising, and the paper doesn't really address the hardest parts of synthesizing such an object; I don't feel that the idea is novel enough on its own to warrant such a \"light\" discussion of the challenges (which are, to the paper's credit, clearly stated in the conclusion!)\n- The experiments section is severely lacking in explanation of how exactly the CLBF is synthesized; since this is a paper about learning a CLBF, it should explain more about learning, not just about application.\n- The proposed method could probably be compared against a couple other approaches, besides just the rather naive robust MPC in the current comparison. I'd recommend comparing against an HJB PDE method and, if possible, a GP-based safe RL method.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I personally don't see how this is especially different from learning a potential field or other generic barrier function style approach. Given that the core idea is the CBLF, it was surprising to me that, just using the paper, I had no clear idea of how I'd implement the proposed method. Perhaps that is just my own ignorance! But I think there are many clarifications that could be added about the proposed method to make it a much more convincing paper.\n\nUpdate (I'm leaving my previous comments unedited). The authors have addressed my concerns both in their comments and in the paper. I think the paper is much stronger now, and demonstrates an interesting advance in RL. I also must admit that I didn't carefully read the supplementary material while originally reviewing the paper! So now I will switch from \"reject\" to \"accept.\"", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "Here I've listed notes that I took while reading, most of which are \"issues\" but some of which are also just comments about what I enjoyed when reading the paper :)\n\nAbstract\n- Are the safety and stability guarantees DURING training, or only AFTER training?\n- Does the proposed method consider actuator limits? How does one ensure there is always a feasible safe control action available?\n\nIntroduction / Related Work\n- Fig. 1 would benefit from a bit of organization (maybe create a grid?) and some additional subtitles or labels explaining what each problem is (e.g., drone, segway, car)\n- The discussion of robust MPC is really helpful and clear\n- It's a bit odd to have just one subsection in Sec. 1\n- An additional challenge to note with CBFs and CLFs is that it's unclear if a feasible safe control is always available (i.e., is the QP always feasible?)\n\nPreliminaries and Background\n- Thank you for clearly stating the problem! So many papers fail to do this!\n- The CLBF definition is nice, but sounds very non-convex and hard to satisfy when synthesizing $V$ -- I'm really curious to see how you do it!\n- Minor typo: the subscript \"CLBF\" in the paragraph under (2) shouldn't be italicized\n- Give that CLBFs are basically a type of potential field, how do you actually ensure reachability? That is, how do you ensure the controlled system doesn't get, e.g., \"stuck\" behind an obstacle when the gradients cancel out in (1e)?\n\nRobust CLBF Certificates for Safe Control / Learning rCBLFs\n- I've had lots of reviewers complain when I put core proofs in supplementary materials. I personally think this is fine, but just be prepared for other reviewers to complain ;)\n- Is it really reasonable to consider a *convex hull* of parameters? The idea of convexity here needs some further explanation to guide reader intuition and justify this idea.\n- Fig. 2 is... not really helpful at all. I mean, there's a state, a neural network or two, and some outputs.\n- Ah, I love this idea that $\\pi_{\\mathrm{NN}}$ is used to prove that the feasible set of the QP is nonempty! Very curious to see how that works.\n- How would this method ever sample enough points in a high-dimensional state space to enable making strong claims that $V$ satisfies (1) (besides the \"easy\" claim of satisfying (1b))?\n- It might be better to use \"split\" rather than \"subequations\" for (5), since it's a single equation continued onto a second line.\n- I prefer the convention of using \"\\cdot\" instead of \"\\circle\" to represent an arbitrary argument (e.g., in where the paper explains the ReLU notation)\n- OK, so it seems sneaky that $\\pi_{\\mathrm{NN}}$ is supposed to ensure the QP is always feasible, but, at the end of the day, you have to relax the QP to make this possible.\n- It seems terrible to have to verify $V$ after training it. Was this verification actually implemented? I guess I'll find out in the next section!\n\nExperiments\n- What does \"steadily\" increasing complexity mean? Just say \"increasing complexity\" and avoid empty words!\n- I really appreciate that you also tried to implement a SOS method. They're really hard to use!\n- What hardware did you use to train your neural networks? How long did training take? Did you apply a verification method afterwards to check that the Lyapunov conditions hold? It's really odd to write a neural network training paper and then include no training results or discussion!\n- For the simpler models, it would also be good to compare against, e.g., a grid-based HJB PDE solver approach, since such methods would have similar online running rates as your proposed method.\n- Fig. 5 is fun and helpful\n\nDiscussion / Conclusion\n- Cool, the method works!\n- There are huge open questions, though! I think the conclusion does a good job of stating these questions -- but they aren't very surprising. I'm a little sad this paper doesn't do a better job of trying to address these questions, since the underlying CLBF idea isn't incredibly new or surprising."}}]

Author rebuttal:

Meta-review: This paper proposes a novel learning-based approach to synthesizing robust nonlinear feedback controllers. While the reviewers agree that this paper contains some interesting ideas, they also raise a number of important concerns. The authors should carefully address the reviewers' comments in their rebuttal. 

UPDATE POST DISCUSSION PHASE: I would like to thank the authors for their comments and clarifications during the discussion phase. The reviewers agree that this paper provides a valuable contribution, and I concur with this assessment. 

Decision: Accept (Poster)