Neural Lyapunov Model Predictive Control
Authors: Mayank Mittal
Marco Gallieri
Alessio Quaglino
Seyed Sina Mirrazavi Salehian
Jan Koutnik
Venue:  (2021)
Keywords: optimal control
mpc
lyapunov neural networks
safe-learning
safety

Abstract: With a growing interest in data-driven control techniques, Model Predictive Control (MPC) provides a significant opportunity to exploit the surplus of data reliably, particularly while taking safety and stability into account. In this paper, we aim to infer the terminal cost of an MPC controller from transitions generated by an initial \emph{unknown} demonstrator. We propose an algorithm to alternatively learn the terminal cost and update the MPC parameters according to a stability metric. We design the terminal cost as a Lyapunov function neural network and theoretically show that, under limited approximation error, our proposed approach guarantees that the size of the stability region (region of attraction) is greater than or equal to the one from the initial demonstrator. We also present theorems that characterize the stability and performance of the learned MPC in the presence of model uncertainties and sub-optimality due to function approximation. Empirically, we demonstrate the efficacy of the proposed algorithm on non-linear continuous control tasks with soft constraints. Our results show that the proposed approach can improve upon the initial demonstrator also in practice and achieve better task performance than other learning-based baselines. 

Under review as a conference paper at ICLR 2021
NEURAL LYAPUNOV MODEL PREDICTIVE CONTROL
Anonymous authors
Paper under double-blind review
ABSTRACT
With growing interest in data-driven control techniques, Model Predictive Control (MPC) provides a significant opportunity to exploit the surplus of data reliably, particularly while taking safety and stability into account. In this paper, we aim infer the terminal cost of an MPC controller from transitions generated by an initial unknown demonstrator. We propose an algorithm to alternatively learn the terminal cost and update the MPC parameters according to stability metric. We design the terminal cost as a Lyapunov function neural network and theoretically show that, under limited approximation error, our proposed approach guarantees that the size of the stability region (region of attraction) is greater than or equal to the one from the initial demonstrator. We also present theorems that characterize the stability and performance of the learned MPC in the presence of model uncertainties and sub-optimality due to function approximation. Empirically, we demonstrate the efficacy of the proposed algorithm on non-linear continuous control tasks with soft constraints. Our results show that the proposed approach can improve upon the initial demonstrator also in practice and achieve better task performance than other learning- based baselines.
INTRODUCTION
Control systems comprise of safety requirements that need to be considered during the controller design process. In most applications, these in the form of state/input constraints and convergence to an equilibrium point, specific set or a trajectory. Typically, a control strategy that violates these specifications can lead unsafe behavior. While learning- based methods are promising for solving challenging non-linear control problems, the lack of interpretability and provable safety guarantees impede their use in practical control settings (Amodei al., 2016). Model-based reinforcement learning (RL) with planning uses a surrogate model to minimize the sum of future costs plus learned value function terminal cost (Moerland et al., 2020; Lowrey et al., 2018). Approximated value functions, however, do not offer safety guarantees. By contrast, control theory focuses these guarantees but it is limited by its assumptions. Thus, there is a gap between theory and practice.
A feedback controller stabilizes system if local Control Lyapunov Function (CLF) function exists for the pair. This requires that the glosed-loop response from any initial state results a smaller value of the CLF the next state. The existence of such function is necessary and sufficient condition for showing stability and convergence (Khalil, 2014). However, finding an appropriate Lyapunov function is often cumbersome and can be conservative. By exploiting the expressiveness of neural networks (NNs), Lyapunov NNs have been demonstrated as a general tool to produce stability (safety) certificates (Bobiti, 2017: Bobiti & Lazar, 2016) and also improve an existing controller (Berkenkamp et al., 2017: Gallieri et al., 2019; Chang et al., 2019). In most of these settings, the controller is parameterized through a NN as well. The flexibility provided by this choice comes the cost of increased sample complexity, which is often expensive in real-world safety-critical systems. In this work, we aim to overcome this limitation by leveraging an initial set of one-step transitions from an wnknown expert demonstrator (which may be sub-optimal) and by using a learned Lyapunov function and surrogate model within an Model Predictive Control (MPC) formulation.
Our key contribution is an algorithmic framework, Neural Lyapunov MPC (NLMPC), that obtains a single-step horizon MPC for Lyapunov- based control of non-linear deterministic systems with constraints. By treating the learned Lyapunov NN estimate of the value function, we provide theoretical results for the performance of the MPC with an imperfect forward model. These results complement the ones by Lowrey et al. (2018), which only considers the case of perfect dynamics

---

Under review as a conference paper at ICLR 2021
model. In proposed framework, alternate learning is used to train the Lyapunov NN in supervised manner and to tune the parameters of the MPC. The learned Lyapunov NN is used as the MPC's terminal cost for obtaining closed-loop stability and robustness margins to model errors. For the resulting controller, we show that the size of the stable region can be larger than that from an MPC demonstrator with longer prediction horizon. To empirically illustrate the efficacy of approach, we consider constrained non-linear continuous control tasks: torque-limited inverted pendulum and non-holonomic vehicle kinematics. We show that NLMPC can transfer between using an inaccurate surrogate and nominal forward model, and outperform several baselines in terms of stability.
2 PRELIMINARIES AND ASSUMPTIONS
Controlled Dynamical System Consider a discrete-time, time-invariant, deterministic system:
+) f(=(t), y(t)= x()), f(0, 0,
(1)
where t E N is the timestep index, xr(t) E R u(t) E R and (t) E - are, respectively, the state, control input, and measurement at timestep t. We assume that the states and measurements are equivalent and the origin is the equilibrium point. Further, the system (1) is subjected to closed and bounded, convex constraints over the state and input spaces:
x(()= E X C R.",, u(I) E E EUCR", A4 0.
(2)
The system is to be controlled by a feedback policy, K R.. + R",, The policy K is considered safe if there exists invariant set, X S X, for the closed-loop dynamics, inside the constraints. The set X2 is also referred to as the safe-set under K. Namely, every trajectory for the closed-loop system that starts at some X, remains inside this set. If asymptotically reaches the target, , By X,, then X. is a Region of Attraction (ROA). In practice, convergence often occurs to a small set, Xr.
Lyapunov Conditions and Safety We formally assess the safety of the closed-loop system in terms of the existence of the positively invariant-set, X,, inside the state constraints. This is done by means of learned CLF, (xx,, given data generated under (initially unknown) policy, K(x).
The candidate CLF needs to satisfy certain properties. First, it needs to be upper and lower bounded by strictly incressing, unbounded, positive (Koo) functions (Khalil, 2014). We focus on optimal control with quadratic stage cost and assume the origin as the target state:
[x, u) x Qr + u Ru,
Q Z 0, R > 0.
(3)
For above, possible choice for Ko function is the scaled sum-of ou-uqes of the states:
1e 1 S V(x) S Lv |
(4)
where le and Ly are the minimum eigenvalue Q and a Lipschitz constant for V respectively.
Further for safety, the convergence to a set, Xr C X., can be verified by means of the condition:
Vx E X. Xe, u == (x) = (f (a. u)) (v < O, with X E [0,
(5)
This means that to have stability (ix) must decrease along the closed-loop trajectory in the annulus.
The sets X., X7, satisfying (5), are (positively) invariant. If they are inside constraints, i.e. X, C X, then they are safe. For a valid Lyapunov function V, the outer safe-set can be defined as a level-set:
X, = {x E X (x) < (.).
(6)
For further definitions, refer the reader to Blanchini & Miani (2007); Kerrigan (2000). If condition (5), holds everywhere in X,, then the origin is a stable equilibrium (X₂- = {0)). If (most likely) this holds only outside non-empty inner set, XT {a E X V(x) < br) C X.. with XT [0). then the system converges to a neighborhood of the origin and remains there in the future.
Approach Rationale
We aim to match or enlarge the safe region of unknown controller, K₃(x).
For perfect model, f. and a safe set x(( there exists an a > 1, such that the one-step MPC:
K K(x)= =
arg min aV(f(z, u)) + {(x u), uEU, Exx0)
2

---

Under review as a conference paper at ICLR 2021
results in new safe set, x ((++1) c(x!)). the one-step controllable set of X and the feasible region of (7), x((+1) 2 x soften the state constraints in (7) and use it recursively to estimate x! j > i. We formulate algorithm that learns the parameter as well the safe set. We train neural network via SGD to approximate V, hence the ROA estimate will not always increase through iterations. To aim for max ROA and minimum MPC horizon, we use voss-validation and verification. We motivate our work by extending theoretical results on MPC stability and a sub-optimality bound for approximate f and V. Finally, we provide an error bound the learned f for having stability.
Learning and Safety Verification We wish to learn V(x) and X from one-step on-policy rollouts, as well as a forward model f (a, u). After the learning, the level I, defining set X will be refined and its safety formally verified a posterion. This is done by evaluating (5) using the model, starting from a large set of states sampled uniformly within X, Ax. We progressively decrease the level I, starting from the learned one, and also increase the inner level ly, starting from zero, such that condition (5) holds for n 2 na samples in X. \XT The number of verifying samples, T4. provides a probability detailed in (Bobiti & Lazar, 2016). The algorithm detailed in appendix and based on (Bobiti & Lazar, 2016). For our theoretical results, we
lower bound on safety, namely Paafe(X, (x) 2 p(nz),
assume the search terminates with Pau fe (X, Xx) R 1 and consider the condition deterministic.
NN-based dynamics model In some MPC applications, it might not be possible to gather sufficient data from demonstrations in order to be able to learn a model that predicts over long sequences. One-step few-steps dynamics learning based on NNs longer time. For instance, errors can accumulate through the horizon due to small instabilities either from the physical system as artifacts from short sequence learning. Although some mitigations are possible for specific architectures or longer sequences (Armenio et al., 2019; Doan et al., 2019; Pathak et al., 2017; Ciccone al., 2018), we formulate MPC to allow for very short horizon and unstable dynamics. Since we learn surrogate NN forward model, f(x, u), from one-step trajectories, we will assume this to have a locally bounded mne-step-ahead prediction error, w(t), where:
suffer when the model is unrolled for
w = f(x, u) - f(z,, u), ||l||2 S 14, V(z, u) E X x U,
(8)
for some compact set of states, X 3 X. We also assume that both and f are locally Lipschitz in this set, with constants Lyz, Lia. and L fx- Liju respectively. A conservative value of can be inferred from these constants as the input and state sets are bounded. It can also be estimated from test set.
3 NEURAL LYAPUNOV MPC
In the context of MPC, function V. which satisfies the Lyapunov property (5) for some local controller Ko. is instrumental to formally guarantee stability (Mayne et al., 2000; Limon et al., 2003). We use this insight and build a general Lyapunov function terminal cost for our MPC, based on neural networks. We discuss the formulation of the Lyapunov network and the MPC in Section 3.1 and Section 3.2 respectively. In order extend the controller's ROA while maintaining a short prediction horizon, an alternate optimization scheme is proposed to tune the MPC and re- re-train the Lyapunov NN. We describe this procedure in Section 3.3 and provide a pseudocode in Algorithm
3.1
LYAPUNOV NETWORK LEARNING
We use the Lyapunov function network introduced by Gallieri et al. (2019):
V(x)= = x" (leI + Vnec(æ)" Vner(z)) (aer x,
(9)
where Vzer(x) is a (Lipschitz) feedforward network that produces ny n. matrix. The scalars my and l > 0 are hypee-parameters It is easy to verify that (9) satisfies the condition mentioned in (4). In our algorithm, we learn the parameters of the network, Vner(r), and safe level, 4,. Note that equation (5) allows to learn V from demonstrations without explicitly knowing the current policy.
Loss function Suppose Dx denotes a set of state-action-e tratisiittnniiiion tuples of the form (æ, u, x²)), where x is the next state obtained from applying the policy K(x). The Lyapunov network is
3

---

Under review as a conference paper at ICLR 2021
trained using the following loss:
Ix (z)
(s u, x+) + Jualls, u, at)
min ((z, JD Vaets ba
(10)
p
where,
Ix, (2) = 0.5 (sign [1. - V(x)] + 1); Jz, u, xt)= max (Vx)) 0
V(z)ter
[uo u, x+) = sign [V(=] 11. - -V(x),,,
Av(x) = (x+) AV(x).
In (10), Zx, is the indicator function for the safe set Xs, which is multiplied to Ja a function that penalises the instability. The term Jeal is classification loss that tries to compute the correct boundary between the stable and unstable points. is also instrumental in increasing the safe set volume. The scalars Ey 0, X [0, 1), and 0 < P < 1, are hyper-paramee parameters, where the latter trades off volume for stability (we take p = 10-3 3 as in Richards et al. (2018); Gallieri et al. (2019)). To make sure that X S X, we scale-down the learned I, a-posteriori. The loss (10) extends the one proposed by Richards et al. (2018) in the sense that we only use one-step transitions, and safe trajectories are not explicitly labeled before training. This loss is then also used to tune the MPC cost scaling factor for V, namely, a 2 1, to guarantee stability. This is discussed next.
3.2 NEURAL LYAPUNOV MPC
We improve stability of the initial controller, used to collect data, by replacing it with an MPC solving the following input-limited, soft-constrained, discounted optimal control problem:
Jare (xt)) min
7 (Va((((N) + Em' )'(()()), a(1) (x(s)))
u
s.t.
f(i+1) (((((),(())) a(1) + s(i) E X, Ei [0,N],
(11)
(x(s) = n s + 12l1s|1, m > 0, 1/2 > 0,
a(1) E U. Vi E -11, 20)) (t).
where a(i) and û(i) are the predicted state and the input at i-steps in the future, s(1) are slack variables, u = N== the stage cost is given by (3), 7 E (0. 1] is discount factor, the function is a forward model, the function V is the terminal cost, in our case a Lyapunov NN from (9), scaled by a factor a 2 1 to provide stability, and rrt) is the measured system state at the current time. The penalty &x is used for state constraint violation, see Kerrigan & Maciejowski (2000).
Problem (11) is solved online given the current state x(t); then, the first element of the optimal control sequence, u (0), provides the action for the physical system. Then, new state is measured, and (11) is again solved, in a receding horizon. The implementation details are given in Appendix.
Stability and safety
We extend results from Limon al. (2003; 2009) to the discounted case and to the X-contractive V from (5). In order to prove them, we make use of the uniform continuity of the model, the SQP solution and the terminal cost, V, as done by Limon al. (2009). Consider the set:
Y YN { ( : <
1- 1-2 + NN Nalall ola) where d= in E(x, 0). xg%.
(12)
The following are obtained for system (1) in closed loop with the MPC defined by problem (11).
Results are stated for Xx = {0). For Xr # (0), convergence would occur to set instead of O.
Theorem 1. Stability and robustness Assume that Vxx) satisfies (5), with X E 10, Xr {0).
Then, given N 2 1, for the MPC (11) there exist a constant â 2 O, a discount factor 7 E (0, 1]. and a model error bound p such that, if a 2 ã. μ $ p and y 2 5, then, Vz( C(X.):
1. If N 1, μ 0, then the system is usymptotically stable for any y > Vz((0) TN
4

---

Under review as a conference paper at ICLR 2021
2. N > 00 then the system reaches set By that is included in X.. This set increases with decreasing discount factors, 7, (z) E TN. y= 1==1= 1B, = {0}.
3. If (z) is the optimal value function in for the problem, = 0, and if C(xa) of %s. then the system is asymptotically stable, Vx((0) E TN....
4. If μ O, then a 2 â implies that (V 2 (*(x), E X. where V is the optimal value function for the infinite horizon problem with cost (3) and subject to (2).
5. The MPC has a stability margin. If the MPC uses surrogate model satisfying (8). with one-step ervor bound ||ll|
3 < ft² = Lv
1-À Llwle. a. then the system is Input-to-Stant to-tte (practically)
Stable (ISpS) and there exists set NN xxt) - BN (z B 8 $ 1.
Theorem states that for given horizon length N and contraction factor A, one can find minimum scaling of the Lyapunov function V and a lower bound on the discount factor such that the system under the MPC is stable. Hence, if the model is perfect, then the state would converge to the origin as time progresses. the model is not perfect, then the safety the system depends on the size of the model error. If this error is less than the maximum tolerable error, μ S ju, then the system is safe: the state converges to a bound, the size of which increases with the size of the model error, the prediction horizon N, and is inversely proportional to In other words, the longer the predictions with an incorrect model, the worse the outcome. Note that the ROA also improves with larger a and r. The proof of the theorem is provided in Appendix. Results hold with the verified probability, (X,,)
Performance with surrogate models In order to further motivate for the search of V giving the largest x,, notice that a larger X, can allow for shortening the MPC horizon, yielding the same ROA. Contrary to Lowrey et al. (2018), we demonstrate how model mismatch and longer horizons can decrease performance with respect to an udinite-horizon oracle with same cost and perfect model.
Let ED|Jv- (K*)) define the expected fffnite-horizon performance of the optimal policy K", evalu- ated by using the expected infinite- horizon performance (value function), V*, for the stage cost (3) infinite-horizon
and subject to (2). Similarly, let EgED [ (x)) define the MPC's expected performance with the learned V. when a surrogate model is used and EzeD (PP f'] when f is known.
Theorem 2. Performance Assume that the value function error is bounded for all x, namely, and that the model error satisfies (8), for some μ > O. Then. for any 8 > 0:
EgED| - (x) S 27 1- + (1 +1+ 110122 EN- 78 (Ej+bi) 11)² 12
+ (1 2)) 7 aLv (ENwo Lig)
2 122 + ((H)
+6 EXED [2;
where L, = min(Ljj:,4a) and vb is a Koneftinttinn representing the constraint penalty terms.
Theorem 2 is related to Asadi et al. (2018) for value-based RL. However, here we do not constrain the system and model be stable, nor assume the MPC optimal cost be Lipschitz. Theorem shows that a discount or a shorter horizon N can mitigate model errors. Since y < 1 can limit stability (Theorem 1) we opt for the shortest horizon, hence 7 1. Proof of Theorem is in Appendix.
MPC auto-tuning The stability bounds discussed in Theorem can be conservative and their computation is non-trivial. Theoretically, the bigger the the larger is the ROA (the safe region) for the MPC, up to its maximum extent. Practically, for very high a, the MPC solver may not converge due to ill- ill-conditioning. scheme, we tried to tune the parameters through gradient- based optimization of the loss (10). These attempts not successful, expected from the considerations in Amos et al. (2018). Therefore, for practical reasons, in this work, we perform a grid search over the MPC parameter a. Note that the discount factor 7 is mainly introduced for Theorem 2 and analysed in Theorem allow for future combination of stable MPC with value iteration.
Initially, by using the tool from Agrawal et al. (2019) within an SQP
5

---

Under review as a conference paper at ICLR 2021
3.3 LEARNING ALGORITHM
Our alternate optimization of the Lyapunov NN,
Algorithm 1 Neural Lyapunov MPC learning
(xx), and the controller is similar to Gallieri et al. (2019). However, instead of training a NN policy, we tune the scaling a and learn V(r)
In: Damos f.A E [0, (le, > 0, (0, 1). N 2 1, Nazts Ny.ev, Vinits f(z, Out: Vnets la a²
used by the MPC (11). Further, approach by using a dataset of demonstrations, Deemo, instead of explicitly defined initial policy. These are one-step transition tuples, = ...,, M. generated by (possibly sub-optimal) stabilizing policy, Ko. Unlike in the approach by Richards al. (2018), our V is a piece-wise quadratic, and it is learned without labels. We in fact produce our own psuedo-labed using the sign of (V(x) in (10) in order to estimate 1,. The latter means that we don't 't require episode-terminalte rollouts, which aren't always available from data accurate when using surrogate. Also, there is no ambiguity on how to label rollouts.
extend their
Vact +- Vinit j Ov..Nv (Vnet, La, X.) +- Adam step on (10)
(ar(0), (((), 2: (1))
for O...N do
Daam (1 Cen) for a am do
4r + sPC(Vanrs f. D; a). from (11)
Darc (a) 4- one.atep sp.ep.imff.......... D.U,) ((a) Evaluate (10) on (a)
end
a
arg min(C(a))
Dare Vnet Vinit
(long)
for j U...Ny
(Vnet, lu X.} Adam (10)
end
end
Once the initial V, X, are learned from the demonstrations, we use V and a learned model, f. within the MPC. We tune the MPC parameter a to minimize the loss defined in (10), using (1 + )x new enlarged target safe set instead of X.. This is done to push the safe set to extend. We propose Algorithm 1, which runs multiple iterations where after each of them the tuned MPC serves demonstrator for training the next V and X, to verify the MPC in closed-loop with the model. Since is not guaranteed that the ROA will increase during learning, we select the Lyapunov function and the MPC using the criteria that the percentage of stable points (AV < () increases and that of unstable points decreases while iterating over and i when evaluated on a validation set.
In Algorithm 1, MP C denotes the proposed Neural Lyapunov MPC, while one_step_st sipmsiimm denotes one-step propagation
the MPC action into the system surrogate model. To train the parameters of V and the level-set I.. Adam optimizer is used Kingma & Ba (2014). A grid search over the MPC parameter a is performed. thorough tuning of all MPC parameters also possible, for instance, by using black-box optimisation methods. This is left for future work.
4 NUMERICAL EXPERIMENTS
Through our experiments, show the following: 1) increase in the safe for the learned controller by using our proposed alternate learning algorithm, 2) robustness of the one-step NLMPC compared longer horizon MPC (used demonstrator) when surrogate model is used for predictions, and 3) effectiveness of our proposed NLMPC against the demonstrator and various RL baselines.
Constrained inverted pendulum In this task, the pendulum starts near the unstable equilibrium (0 09). The goal to stay upright. We bound the input so that the system cannot be stabilized if (0) > 60% We use an MPC with horizon 4 as demonstrator, with terminal cost, 500x Pi.qR
Table Inverted Pendulum: Learning on nominal model. With iterations, the number of points verified by the controller increases.
(a) Demon- strator
(b) NLMPC (nominal)
(c) NLMPC (surrogate)
ITER.
Loss
VERIF.
Nor VeriF.
log(1 x))
%
%
Figure 1: Inverted Pendulum: Testing learned controller on nominal system. Lyapunov func- tion with safe trajectories. NLMPC learns cessfully and also transfers to surrogate model.
1
3.21
13.25
0.00
2
1.08
13.54
0.00

---

Under review as a conference paper at ICLR 2021
Table 2: Car: Learning on nominal model.
Table 3: Car: Learning on surrogate model.
ITER.
Loss
VERIF.
NOT VERIE.
ITER.
Loss
VERIF:
NOT VERIE.
1
1.55
92.20
4.42
1.84
91.74
8.26
2
0.87
93.17
4.89
2
1.43
92.26
7.74
3
0.48
94.87
3.89
1.65
91.61
8.39
116
610
010
(a) Short-horizon (b) MPC demonstrator using surrogate
(c) Lyapunov MPC using surrogate (ours)
(d) MPC demonstrator using nominal
(e) Lyapunov MPC using nominal (ours)
MPC using surrogate
Figure Car kinematics: Transfer from surrogate to a nominal model. Top: Lyapunew function contours with trajecturies for 40 steps. Bottom: Lyapumov function evaluated for specific policy on several initial states (decreasing means more stable).
where PI.qR is the LQR optimal cost matrix. This evaluated on 10K equally spaced initial states to generate the dataset demo- We train grey-box NN model, f using 10) random transition tuples. More details are in Appendix. The learned and a, obtained from Algorithm produce one-step MPC that stabilizes both the surrogate and the actual system. Table shows that the loss and pereentage of verified points improve across iterations. The final ROA estimate is ncarly maximal and is depicted along with the safe trajectorics. produced by the MPC whilc using predictions from the nominal and surrogate models, in Figure The performance matches that of the haseline and the transfer is successful due to the accuracy of the learned model. A full ablation study is in Appendix.
Constrained car kinematics The goal is to steer the car to the (0, 0) position with zero orientation. This is only possible through non-linear control. The vehicle canmot move sideways, bence policies such as LQR is not usable to generate demonstrations. Thus to create Datmn: an MPC with horizon 5 is evaluated over 10K random initial states. The surrogate, f is grey-box NN trained using 10K random transition tuples. More details are in Appendix. Higure 3 shows the learning curves, training of the Lyapumov function over iterations and line-scarch for the MPC auto-tuning. Table summarises the metrics improvenient across the iterations. indicating an increase in the ROA when perfect model is used. With an imperfect model, the second iteration gives the best results, as shown in Table 3.
We the transfer capability of the approach in two ways. First, we learn using the nominal model and test using the surrogate model for the MPC predictions. This is reported in Appendix for the sake of space. Second, the learning is performed using the surregate model in Algorithm and the MPC is then tested on the nominal model while still using the surrogate for prediction. This is depicted in Figure 2. Our MPC works better than the demonstrator when using the incorrect model. The learned MPC transfers successfully and completes the task safely.
Comparison to haselines et al., 2017) provide safety guarantees in terms of constraint satisfaction that hold in expectation. However, due to unavailability of working implementation, against it. Instead to enforee safety constraints during training of the RL algorithms,, we use two different strategics: v1) carly episode termination; v2) reward shaping with constraint penalty. The v2 formulation is simila lar to the one used in Ray et al. (2019), which demonstrated its practical equivalence to CPO when uned. We compare approach against model-fres and model-hased
Prior works such as constrained policy optimization (CPO) (Achiam
are unable to compare our approach

---

Under review as a conference paper at ICLR 2021
100
25
80
20
60
15
10
2-
(a) Lyapunov Loss
(b) Verified Points
(c) Not Verified Points
(log + x))
(%)
(%)
1.0
1.0
1.0
0.24
1.08
1.21
0.5
0.22
0.19
0.96
0.84
0.5
0.94
1.08
0.72
0.8]
0.0
0.1
0.13
0.0
0.60
0.48
0.0
0.54
0.67
0.08
0.36
0.40
0.5
0.05
00.5
0.24
005
0.27
0.02
0.12
0.13
0.00
0,00
0.00
-1.0
-1.0
-1.0
0
4
5
10
20
25
30
15
20
35
15
30
35
0f
Iteration 1
Iteration 2 (best)
Iteration 3
Figure 3: Car kinematics: Alternate learning on surrogate model. After every Nv = 800 epochs of Lyapunov learning, the learned Lyapunov function is used to tume the MPC: parameters. Top: The training curves for Lyapunov function. Vertical lines separate iterations. Middle: The resulting Lyapunov function V ø 0 with the best performance. Bottom: Line-Search for the MPC parameter a to minimize the Lyapunov loss (10) with V as terminal cost. The loss is plotted on the y-axis in a log(1 x) scale. The point marked red the parameter which minimizes the loss.
baseline algorithms. For the model-free baselines, we consider the on-policy algorithm proximal policy optimization (PPO) (Schulman et al., 2017) and the off-policy algorithm soft actor-critic (SAC) (Haarnoja et al.. 2018), 8). For model-based basclines. we consider model-based policy opti- mization (MBPO) (Janner et al., 2019) and the demonstrator MPC. Further details about the reward shaping and learning curves are in Appendix.
We consider the performance of learned controllers in terms stability and safety. Stability performance is analogous to the success rate in performing the sct-point tracking task. We consider a task is completed when [|æ(T)]|2 0.2 where T is the final time of the trajectory. For the car, we exclude the orientation from this index. The safety performance combines the former with state constraints satisfaction over the entire trajectory. As shown Table lead to some safe trajectories. Note that the demonstrator (which bas LQR terminal cost) is an optimal controller and the maximum performance that can be achieved. In terms of stability performance, approach performs good the demonstrator MPC: 'Tee RI. trained policies give sub-optimal behaviors, Le. sometimes the system goes to the other equilibria. For the car, the demonstrator MPC is sub-optimal policy. NLMPC improves upon it in performance and it is on par with it in terms of safety. NLMPC also significantly outperforms all of the considered RL baselines while using lesser number of samples for learning
the inverted pendulum, the policies
For all our experiments, training datapoints: PPO: 4 x 10 SAC: x 10th MBPO: 2.4 X 10S NLMPC: 40 (random) 10* (demonstrations).
8

---

Under review as a conference paper at ICLR 2021
Table Comparison with baselines. We compare our NLMPC (with surrogate model for predic- tions) with baselines. In the pendulum, our approach is second to the demonstrator for less than 1% margin. In the car task, NLMPC performs better than all baselines and improves convergence from the demonstrator, while it is nearly on par with the latter on constraints.
CONSTRAINED INVERTED PENDULUM
CONSTRAINED CAR KINEMATICS
ALGORITHM
STABILITY (%)
SAFETY (%)
STABILITY (%)
SAFETY (%)
PPO (v1)
14.67
14.66
15.17
0.50
PPO (v2)
26.33
26.33
8.16
0.83
SAC (V1)
29.99
29.99
12.33
0.00
SAC (V2)
27.17
27.17
8.00
0.00
MBPO (V1)
12.67
12.67
6.00
0.00
MBPO (v2)
26.00
26.00
6.00
0.00
MPC (DEMO)
36.00
36.00
81.33
73.33
NLMPC
35.33
35.33
86.00
72.67
RELATED WORK
Stability and robustness of MPC and of discounted optimal control have been studied in several prior works Mayne al. (2000); Rawlings & Mayne (2009); Limon al. (2009; 2003): Raković al. (2012); Gaitsgory et al. (2015). Numerical stability verification was studied in Bobiti (2017); Bobiti Lazar (2016) and, using neural network Lyapunov functions in Berkenkamp al. (2017); Gallieri et al. (2019). Neural Lyapunov controllers were also trained in Chang et al. (2019). MPC solvers based on iterative LQR (iLQR) were introduced in Tassa et al. (2012). Sequential Quadratic Program (SQP) was studied in Nocedal & Wright (2006). NNs with structural priors have been studied in Quaglino et al. (2020); Yıldız et al. (2019); Pozzoli et al. (2019). Value functions for planning were learned in Lowrey et al. (2018); Deits et al. (2019); Buckman et al. (2018), Gallieri al. (2019) learned a NN Lyapunov function and an NN policy with an alternating descent method, initialized using known stabilizing policy. We remove this assumption and use MPС. Suboptimality was analysed in Grune & Rantzer (2008) for MPC and in Janner et al. (2019) for policies. Differently from NNs, non-parametric models have been largely studied for control, see for instance Koller et al. (2018); Hewing et al. (2020) and references therein for closed-form results using Gaussian processes.
CONCLUSIONS
We presented Neural Lyapunov MPC, framework to train a stabilizing non-linear MPC based on learned neural network terminal cost and surrogate model. After extending existing theoretical results for MPC and value-based reinforcement learning, we have demonstrated that the proposed framework can incrementally increase the stability region the MPC through offline RL and then safely transfer on simulated constrained non-linear control scenarios. Through comparison of our approach with existing RI. baselines, we showed how NNs can leveraged to achieve policies that perform at par with these methods while also having provable safety guarantees.
Future work could address the reduction of the proposed optimally bound, for instance through the integration of value learning with Lyapunov function learning as well as the optimal selection of MPC prediction horizon. broader class of stage costs and rewards could also be investigated.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. arXiv preprint arXiv: 10025,,, 2017.
Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and Zico Kolter. Differentiable Convex Optimization Layers. 19Xiv 12430 [cs, math, stat], October 2019.
URL /tttt:://rxiv..... org/ awt.aasss.111// 12430. arXiv: 1910.12430.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete Problems in AI Safety, 2016.
Brandon Amos, Ivan Jimenez, Jacob Sacks, Byron Boots, and Zico Kolter. Differentiable MPC for End-to-end Planning and Control. In Advances in Neural Information Processing Systems 31, pp. 8289 8300. Curran Associates, Inc., 2018.
Luca Bugliari Armenio, Enrico Terzi, Marcello Farina, and Riccardo Scattolini. Echo state networks: analysis, training and predictive control. In 2019 18th European Control Conference (ECC). IEEE, June 2019. doi: 2223199/cc.22199877788 8795677.
URI. https::///// http//i..ogggoo/// 2391//11.111111 9/ecc.
Kavosh Asadi, Dipendra Misra, and Michael L. Littman. Lipschitz Continuity in Model-based Reinforcement Learning. orXiv:/804.07/99 [cs, stat), July 2018. URL http:// /ttp///xxvvvooo/o// org/ abs/1804 07193.
arXiv: 1804.07193.
A. Bemporad, M. Morari, V. Dua, and E.N. Pistikopoulos. The explicit solution of model predictive control via multiparametric quadratic programming. In Proceedings of the American Control Conference. IEEE, 2000. URL https: //tpi.//oi../// /0g/10.... 1ew1111/////. htc./cc..2000 2000. 876624.
Felix Berkenkamp, Matteo Turchetta, Angela P. Schoellig, and Andreas Krause. Safe Model-based Reinforcement Learning with Stability Guarantecs. arXiv:170S 08551 [cs, stat], May 2017. URL abs/ 10..0...5111111
http:://rxte org/ab/g////
arXiv: 1705.08551.
Franco Blanchini and Stefano Miani. Set-Theoretic Methods in Control (Systems & Control: Founda- tions & Applications). Birkhäuser, 2007. ISBN 08 186322557.
R. V. Bobiti. Sampling driven stability domains computation and predictive control of constrained non- linear systems. PhD thesis, 2017. URL httph: /ppree.:////reeeeeeeeee tuw.erre..ue.e... tue /tt./////E1ee//4554335 20171022 Bobiti pdf.
Ruxandra Bobiti and Mireca Lazar Sampling-based verification of Lyapunov's incquality for piecewise continuous nonlinear systems. arXiv:1609. 10X.000009000022 00300 [cs], September 2016. URL http:
arXiv: 1609.00302.
Jacob Buckman. Danijar Hafner. George Tucker, Eugene Brevdo, and Honglak Lec. Sample- efficient reinforcement learning with stochastic ensemble value expansion. In S. Ben- gio, H. Wallach, H. Larcchllle, K. Grauman, N. Cisa-Bianchi, and R. Garnet (eds.). Ad- vances in Neural Information Processing Systems. volume 31, pp. 8224-8234. Curran Asso- neurips oc http:/ppppe////////e/2 ciates, Inc., 2018. URL https:: Paper.pdls (0220aa07760eee11cc//ffffffddddd
Ya-Chien Chang, Nima Roohi, and Sicun Gao. Neural Lyapunov Control. In Advances in Neural Information Processing Systems, pp. 3240 3249. 2019. URL http: /ttp://papprrra nips cc/ paper/ wwwraaa--aa------eeeeeaa www.urral-yaauuoooooottooo pdf.
Marco Ciccone, Marco Gallieri, Jonathan Masci, Christian Osendorfer, and Faustino Gomez. NAIS- Net: Stable Deep Networks from Non-Autonomous Differential Equations. 1xXvv:880.002200 [cs. stat], April 2018. URL http: http//axxvvv//// wwwp:/iv.oorraaaao
wwtp://.//////....o......o.sss/ abs/ /804. 07209. arXiv: 1804.07209.
Robin Deits, Twan Koolen, and Russ Tedrake. Lvis: Learning from value function intervals for contact-aware robot controllers. pp. 7762-7768, 05 2019. doi: 10.1109/ 2019.88993322
10

---

Under review as a conference paper at ICLR 2021
Nguyen Anh Khoa Doan, Wolfgang Polifke, and Luca Magri. Physics-informet echo state networks for chaotic systems forceasting. In Lecture Notes in Computer Science, pp. 192-198. 198. Springer International Publishing, 2019. doi: 10. 10.1007/978-33000-277777 15. URL https /doi.
org/10. 0te/110.000//9788 0011/9988-33000 3-000--00-0-00000 2000022277-0_155
Thor Fossen. Handbook of marine craft hydrodynamics and motion control. John Wiley Sons, 2011.
Vladimir Gaitsgory, Lars Gruene, and Neil Thatcher. Stabilization with discounted optimal control. volume 82, pp. 91-98, 07 2015. doi: httoo///ssscoole220115
Marco Gallieri, Seyed Sina Mirrazavi Salehian, Nihat Engin Toklu, Alessio Quaglino, Jonathan Masci, Jan Koutník, and Faustino Gomez. Safe Interactive Model- Based Learning. arXiv: [cs, eess/, November 2019. URL http: org/ abs/ 1911.11. 6655. arXiv: 1911.06556
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yec Whyc Teh and Mike Titterington (cds.). Proceedings of the International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 256, 2010.
L. Grune and A. Rantzer. On the infinite horizon performance of receding horizon controllers. IEEE Transactions on Automatic Control, 53(0):2100-2111, 2008. doi: 10. 109/TACC.0008.9779999
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:/801.012000 2018.
Dylan Hadfield-Menell, Christopher Lin, Rohan Chitnis, Stuart Russell, and Pieter Abbeel. Sequential quadratic programming for task plan optimization. In Proceedings of IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 5040-5047. IEEE, 2016.
Lukas Hewing, Kim P. Wabersich, Marcel Menner, and Melanie N. Zeilinger. Learning- based model predictive control: Toward safe learning in control. Annual Review of Control, Robotics, and Autonomous Systems, 3(1):269-296, 2020. doi: 10.1 1tt////auure--ooooo0000000000000
URL
https://doi. org 10 1 11t.:/11///////eee////// www.mreewoooooooooooooooooooooo- https///......ooo/ooo//////eerra
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to Trust Your Model: Model- Based Policy Optimization, /rXiv:/906.000 08253 [cs, stat), November 2019. URL htRp:h/t////xxii
org/ab hta//bb/19900 08253. arXiv: 0925..
EC Kerrigan, Robust constraint satisfaction: Invariant sets and predictive control. Technical report, 2000. URL httht:::/// //hd1 http:://hhd.....n.iaeeeea hww.la..dee................eea.. wttp10044//////eee//
hric C. Kerrigan and Jan M. Maciejowski. Soft constraints and exact penalty functions in model
predictive control. Proceedings of UKACC International Conference, 2000.
Hassan K. Khalil. Nonlinear Control. Pearson, 2014. ISBN 9780133499261
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv:1412.6990 [cs], December 2014. URL http: /atpiva/aiivvvv/aa/////ee. wwt. abs 1412 6980. arXiv: 1412.6980.
T. Koller, F. Berkenkamp, M. Turchetta, and A. Krause. Learning- bbaed model predictive control for safe exploration. In 2018 IEEE Conference on Decision and Control (CDC), pp. 6059-6066. 2018. doi: 00.109/CDC..001.88119555
D. Limon. T. Alamo, and E. F. Camacho. Stable constrained MPC without terminal constraint. Proceedings of the American Control Conference, 2003.
D. Limon, T. Alamo, D. M. Raimondo, D. Muñoz de la Peña, J. M. Bravo, A. Ferramosca, and E. F. Camacho. Input-to-Stacte Stability: A Unifying Framework for Robust Model Predictive Control. In Nonlinear Model Predictive Control, pp. 1-26. Springer Berlin Heidelberg, 2009.
11

---

Under review as a conference paper at ICLR 2021
Kendall Lowrey, Aravind Rajeswaran Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan Online, Leam Offlinc: Efficient Learning and Exploration via Model-Based Control. arXiv:1811.0184 [cs, stat]. November 2018. URL http://arxa atgg/bb////l 01848. arXiv: 1811.01848.
D.Q. Mayne, J.B. Rawlings, C.V. Rao, and P.O.M. Scokaert. Constrained model predictive control: Stability and optimality. Automatica,, 36(6):789 814, 2000.
Thomas M. Moerland, Joost Broekens, and Catholijn M. Jonker. Model-based reinforcement learning: A survey, 2020.
Jorge Nocedal and Stephen Wright. Numerical optimization. Springer Science & Business Media, 2006.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer. James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, segh-performane deep learning library, 2019. URL http: / arxiv. org/ /tabs//9910 1912. 01703.
Jaideep Pathak, Zhixin Lu, Brian R. Hunt, Michelle Girvan, and Edward Ott. Using machine learning to replicate chaotic attractors and calculate Iyapunov exponents from data. Chaos: An Interdisciplinary Journal of Nonlinear Science, 27(12)112110,, December 2017. doi: 10.1063/1.
5010300. URL https: // wrw.orgg/////// /00 10w3/166111111 10t...1111111111000000000 5010300.
Simone Pozzoli, Marco Gallieri, and Riccardo Scattolini. Tustin neural networks: a class of recurrent nets for adaptive MPC of mechanical systems. arXiv:1911.013/1 [cs, eess), November 2019. URL
http:// /ttp:i//axivvv/// org/ abs/ 1911. 01310. arXiv: 1911.01310.
Alessio Quaglino, Marco Gallieri, Jonathan Masci, and Jan Koutník. SNODE: Spectral Discretization of Neural ODEs for System Identification. arXiv:1906.0777 07038 [cs], January 2020. URL http:
wwwxiv.org/aai.... 9 07038. arXiv: 1906.07038.
S. V. Raković, B. Kouvaritakis, R. Findeisen, and M. Cannon. Homothetic tuhe model predictive control. Automatica, 1861688- 08 2012. doi: 10. aut.:/aaautmmmcaa222222222
J. B. Rawlings and D. Q. Mayne. Model Predictive Control Theory and Design. Nob Hill Pub, Llc, 2009. ISBN 0975937707.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking Safe Exploration in Deep Reinforce- ment Learning. 2019.
Spencer M. Richards, Felix Berkenkamp, and Andreas Krause. The Lyapunov Neural Network: Adaptive Stability Certification for Safe Learning of Dynamical Systems. arXiv: 00994 [cs], August 2018. URL http: /ttp:::////vvv../eeeeeeeeeeeeee/ org/ abbs/////////// / 1808.080 C0924. arXiv: 1808.00924.
John Salvatier, Thomas V. Wiecki, and Christopher Fonnesbeck. Probabilistic programming in python using PyMC3. Peerl Computer Science, 2:e55, apr 2016. doi: 10.7717/peerr-cs.555. URL
https::/dol org/ol.g............ http:/....771//11 7 55.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint a7Xiv:1707.06347, 2017.
Yuval Tassa, Tom Frez, and Emanuel Todorov. Synthesis and stabilization of complex behaviors
through online trajectory optimization. In Proceedings of the IEEF/RSI International Conference on Intelligent Robots and Systems., pp. 4906-1913, 10 2012. ISBN 978-1-4673--17777 doi:
N0.1109/ROS.0012.63386 2012.638602..
Cağatay Yıldı. Markus Heinonen, and Harri Lähdesmäki. VDE: Deep generative second order ODES with Bayesian neural networks. vrXiv:/905, 10994 [es, stat], Octoher 2019. URI. http::/aat /tapxi///xxv... v. www//bs.//////ooo/eo/..ooeee abs/190s/ 10994. arXiv: 1009.1
12


Reviews:
[{"review_id": "eK-Vvpg17SP", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper the author proposed an MPC algorithm in which both the dynamics function and the Lyapunov function are parameterized with neural networks.. Specifically leveraging the results of Lyapunov networks (2018 CORL paper: https://arxiv.org/abs/1808.00924) for learning Lyapunov functions, the authors derived an MPC algorithm for quadratic cost/reward problems and also proved the stability, robustness, and sub-optimality performance. To demonstrate the effectiveness of the algorithms, the authors also evaluated this approach on the simple inverted pendulum and car kinematics tasks. \n\nIn general I find this paper presents a comprehensive results of a model-based control method that is very popular in the control theory community. To justify their algorithms they also proved several standard properties (stability, sub-optimality performance) in control, which I appreciate their efforts. However, I do have severals questions/concerns regarding the details of their approach:\n\n1) The presentation of the loss function of Lyapunov network is not easy to parse, especially there are couple terms that contain specific mathematical operators (sign, ReLU). Can the authors explain each term in the loss and why such choices of loss terms are necessary. Is this loss function identical to the Lyapunov network 2018 CORL paper?\n\n2) From the main paper it is unclear how the NN-dynamics model \\hat f is learned. Does it just train based on prediction loss? More importantly, while the MPC algorithm uses the learned model how does the dynamics model error affect the stability/robustness/performance bounds of the control algorithm? I cannot immediately find this information in lemma 1 and lemma 2, which makes me worried about the correctness of these results. (Unfortunately I haven't had a chance to check the appendix for proofs)\n\n3) Having sub-optimality performance for MPC algorithms is a nice result, as not many MPC algorithms have performance guarantees. However these kind of results are also not new (for example, see https://ieeexplore.ieee.org/document/4639448).  How does the MPC performance result here compared with the ones by Grune and Rantzer? \n\n4) Among various safe MPC papers, how does the proposed one in this paper compared with this safe MPC algorithm: https://arxiv.org/pdf/1803.08287.pdf, which is also proposed by Andreas Krause's group (that proposed the Lyapunov network)? At least experimentally how does the proposed algorithm compare with other safe MPC baselines (such as the one above) on the standard benchmark tasks (for example the above work also tested the algorithms on the pendulum task).\n\nOn the overall, I find this paper's algorithm interesting. However, there are several technical question listed above, and one high-level concern is its novelty. Without further discussions, it appears to me that the work combines several existing results on Lyapunov network and MPC, for which the contribution is rather incremental.", "extra_scores": {}, "extra_text": {}}, {"review_id": "I8TsjF8osRW", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the question of how to stabilize a system in a vicinity of an equilibrium. While the majority of reinforcement learning algorithms rely on trial and error, which may damage the system, the authors introduce an algorithm for safe exploration and control. A traditional approach in model-based RL is to use MPC with a surrogate forward model to minimize a planning objective comprising a sum of stage costs along with a terminal cost, often chosen as an approximated value function -i.e. the optimal expected cost-to-go- which can be learned by a Bellman equation. Instead, this work is placed in the framework of Robust MPC, where this value function is replaced by a Luyapunov function $V$, which is related to the notion of stability and is only constrained to decrease along trajectories. Such a Luyapunov function, when available, provides both a safe region, defined as a level-set of V, and a MPC policy for which stability analyses have been developed: the authors extend a result from Limon et al. (2003; 2009) to show that this MPC policy enjoys asymptotic stability in general, and input-to-state stability in the presence of small enough model errors. Accordingly, the authors propose a scheme allowing to learn a Lyapunov function $V$ from demonstration data only, through a loss function that penalizes increments of $V$ along one-step transitions. A regularization parameter $\\alpha$ of this MPC, which balances stability with constraints satisfaction and stage costs, is also learned jointly by an alternative training procedure. This approach is evaluated empirically on two standard constrained non-linear continuous control tasks.\n\n\nStrong points:\n1. This paper is clearly written, well motivated, honest about its place in the literature, and all derivations seem technically correct.\n2. By bringing together existing results and techniques (MPC stability analyses, value-based RL analyses, LuyapunovNet), the authors manage to relax several assumptions of prior works (no need for access to the perfect dynamics or a stabilizing policy, but only to a demonstration dataset) which makes the approach more practical.\n3. Empirically, the learned Luyapunov functions seem to effectively capture useful stability information, since the proposed approach outperforms a standard MPC with a longer planning horizon. Even better, this observation is theoretically justified by Lemma 2: errors of the surrogate model compound when used in an MPC, which is detrimental for long-term planning. Conversely, if $V$ contains long-term information, it can directly be used for short-sighted planning, similarly to being greedy with respect to a value function.\n\n\nWeak points:\n1. The part which was the least clear to me is the *Performance with surrogate models* paragraph, with Lemma 2. The authors draw a parallel between Luyapunov functions in control theory and value functions in RL, but the latter are not really defined clearly in the text. The authors state in their introduction that they treat \"the learned Lyapunov NN as an estimate of the value function\" and later they mention a \"correct\" value function $V^*$ for optimal \"expected infinite-horizon performance\", but this quantity is nowhere defined. I suppose $V^*$ is an expected infinite sum of discounted stage costs, but which costs? The same as in equation (3)? If so, I find it hard to believe that the assumption of Lemma 2 should be satisfied ($V^*$ close to $\\alpha V$), given that the stage cost $l$ of (3) used to define $V^*$ does not appear in the loss (10) used to learn the Lyapunov function $V$.\n2. I find it difficult to assess the novelty of the theoretical results. The authors are honest in stating that they are extending/adapting known results, but it is not precisely stated what their added value is. Moreover, the abstract mentions that \"we also present theorems\" but in the article these results are presented as lemmas, which to me usually suggests that they are either instrumental to another result (which they are not), or of minor importance.\n3. One of the main claim of the paper is the ability of the method to expand a demonstrated safe region. However, this is not really observed consistently across tasks and iterations. For instance, it is not the case in Figure 3. Likewise, the Table 1 states that \"With iterations, the number of points verified by the controller increases\", but only a two iterations is provided (i.e. a single opportunity to increase/decrease). This seems a bit suspicious and suggests that the ratio of verified points may actually decrease on subsequent iterations, as it does on iteration 3 of Table 3.\n\n\nTo conclude, I lean toward recommending acceptance, but I am ready to increase my score provided that the authors improve the clarity on both the analogy between Lyapunov and value functions, and state more clearly the novelty of their theoretical contributions.\n\n\nMinor remarks and typos:\n* I do not really see the relevance of the safety performance metric in the Inverted Pendulum experiment (Fig. 3): the state constraints are loose enough that every successful trajectory is also considered safe, so this safety plot (right) does not really bring any new information to the table.\n* p2, Equation (5): X_s\\ **X_T**\n* p3, Learning and Safety Ver**i**fication\n* p6, to minimize the loss defined in **(11)**", "extra_scores": {}, "extra_text": {}}, {"review_id": "3J_u0t9-3tX", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an MPC algorithm based on a learned (neural network) Lyapunov function. In particular, they learn both the Lyapunov function and the forward model of the dynamics, and then control the system using an MPC with respect to these models.\n\nCons\n- Poorly written\n- Unclear connections to related work\n- Weak experiments\n\nIt is unclear exactly what problem the authors are attempting to solve. In general, the authors introduce a large amount of notation and theory, but very little of it appears to be directly related to their algorithm. For example, they refer to the stability guarantees afforded by Lyapunov functions, but as far as I can tell, they never prove that their algorithm actually learns a Lyapunov function (indeed, Lemma 1 starts with “Assume that V(x) satisfies (5) [the Lyapunov condition] ...”).\n\nSimilarly, they allude to “robustness margins to model errors”, but nothing in the algorithm actually takes into account model errors. Is the point of these margins just to show that they exist? If so, it’s not clear the results (either theoretical or empirical) are very meaningful, given that they depend on the unknown model error (which they assume to be bounded).\n\nIn addition, the different loss functions they use (e.g., (10)) are poorly justified. Why is this loss the right one to use to learn a Lyapunov function?\n\nFurthermore, the authors’ approach is closely related to learning the value function and planning over some horizon using the value function as the terminal cost (indeed, the value function is a valid Lyapunov function, but not necessarily vice versa). For instance;\n\nBuckman et al., Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion. In NeurIPS, 2018.\n\nThe most closely related work I’m aware of is the following:\n\nDeits et al., Lvis: Learning from value function intervals for contact-aware robot controllers. In ICRA, 2019.\n\nThe authors should clarify their contributions with respect to these papers. More importantly, the authors should discuss their motivation for indirectly learning a Lyapunov function instead of simply learning the value function (which appears to be more natural and potentially more effective).\n\nNext, the authors’ experiments are very weak. They only consider two environments, the inverted pendulum and car, both of which are very simple. The inverted pendulum starts near the unstable equilibrium, which further trivializes the problem. In addition, they do not even appear to give the dynamics model of the car they are using (or the state space).\n\nFinally, this paper is poorly written and hard to follow. They provide a lot of definitions and equations without sufficient explanation or justification, and introduce a lot of terminology without giving sufficient background.\n\n------------------------------------------------------------------------------------------------------------------------------------------------\n\nPost rebuttal: While I appreciate the authors' comments, they do not fundamentally address my concerns that the paper is too unclear in terms of the meaning of its technical results to merit acceptance. As a concrete example, in their clarification, the authors indicate that they obtain \"probabilistic safety guarantees\" by checking the Lyapunov condition (5) using sampling. However, at best, sampling can ensure that the function is \"approximately\" Lypaunov (e.g., using PAC guarantees) -- i.e., satisfies (5) on all but 1-\\epsilon of the state space.\n\nUnfortunately, an \"approximately\" Lyapunov function (i.e., satisfies the Lyapunov condition (5) on 1-\\epsilon of the state space) provides *zero* safety guarantees (not even probabilistic safety at any confidence level). Intuitively, at each step, the system has a 1-\\epsilon chance of exiting a given level set of the Lyapunov function. These errors compound as time progresses; after time horizon T, only 1 - T * \\epsilon of the state space is guaranteed to remain in the level set, so eventually the safety guarantee is entirely void.\n\nOne way to remedy this is if the Lyapunov function is Lipschitz continuous. However, then, the number of samples required would still be exponential in the dimension of the state space. At this point, existing formal methods tools for verifying Lyapunov functions would perform just as well if not better, e.g., see:\n\nSoonho Kong, Sicun Gao, Wei Chen, and Edmund Clarke. dReach: δ-Reachability Analysis for Hybrid Systems. 2015.\n\nThis approach was recently applied to synthesizing NN Lyapunov functions (Chang et al. 2019). My point isn't that the authors' approach is invalid, but that given the current writing it is impossible for me to understand the theoretical properties of their approach.\n\nOverall, I think the paper may have some interesting ideas, but I cannot support publishing it in its current state", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose an MPC based approach for learning to control systems with continuous state and actions - the dynamics, control policy and a Lyapunov function are parameterized as neural networks and the authors claim to derive stability certificates based on the Lyapunov function.

The reviewers raised several serious technical issues with the paper as well as the lack of clarity in the presentation of the main technique in the initial version of the paper. While the clarity concerns were partially addressed during the rebuttal, the technical concerns (in particular those raised by reviewer 1) remain unaddressed - the stability certificate derived is questionable due to the fact that sampling based approaches to certifying that a function is a valid Lyapunov function are insufficient to derive any stability guarantee. Further, the experimental results are only demonstrated on relatively simple dynamical systems. Hence I recommend rejection. 

However, all reviewers agree that the ideas presented in the paper are potentially interesting - I would suggest that the authors consider revising the paper to address the feedback on technical issues and submit to a future venue.


Decision: Reject