Neural Lyapunov Model Predictive Control

1 INTRODUCTION . Control systems comprise of safety requirements that need to be considered during the controller design process . In most applications , these are in the form of state/input constraints and convergence to an equilibrium point , a specific set or a trajectory . Typically , a control strategy that violates these specifications can lead to unsafe behavior . While learning-based methods are promising for solving challenging non-linear control problems , the lack of interpretability and provable safety guarantees impede their use in practical control settings ( Amodei et al. , 2016 ) . Model-based reinforcement learning ( RL ) with planning uses a surrogate model to minimize the sum of future costs plus a learned value function terminal cost ( Moerland et al. , 2020 ; Lowrey et al. , 2018 ) . Approximated value functions , however , do not offer safety guarantees . By contrast , control theory focuses on these guarantees but it is limited by its assumptions . Thus , there is a gap between theory and practice . A feedback controller stabilizes a system if a local Control Lyapunov Function ( CLF ) function exists for the pair . This requires that the closed-loop response from any initial state results in a smaller value of the CLF at the next state . The existence of such a function is a necessary and sufficient condition for showing stability and convergence ( Khalil , 2014 ) . However , finding an appropriate Lyapunov function is often cumbersome and can be conservative . By exploiting the expressiveness of neural networks ( NNs ) , Lyapunov NNs have been demonstrated as a general tool to produce stability ( safety ) certificates ( Bobiti , 2017 ; Bobiti & Lazar , 2016 ) and also improve an existing controller ( Berkenkamp et al. , 2017 ; Gallieri et al. , 2019 ; Chang et al. , 2019 ) . In most of these settings , the controller is parameterized through a NN as well . The flexibility provided by this choice comes at the cost of increased sample complexity , which is often expensive in real-world safety-critical systems . In this work , we aim to overcome this limitation by leveraging an initial set of one-step transitions from an unknown expert demonstrator ( which may be sub-optimal ) and by using a learned Lyapunov function and surrogate model within an Model Predictive Control ( MPC ) formulation . Our key contribution is an algorithmic framework , Neural Lyapunov MPC ( NLMPC ) , that obtains a single-step horizon MPC for Lyapunov-based control of non-linear deterministic systems with constraints . By treating the learned Lyapunov NN as an estimate of the value function , we provide theoretical results for the performance of the MPC with an imperfect forward model . These results complement the ones by Lowrey et al . ( 2018 ) , which only considers the case of a perfect dynamics model . In our proposed framework , alternate learning is used to train the Lyapunov NN in a supervised manner and to tune the parameters of the MPC . The learned Lyapunov NN is used as the MPC ’ s terminal cost for obtaining closed-loop stability and robustness margins to model errors . For the resulting controller , we show that the size of the stable region can be larger than that from an MPC demonstrator with a longer prediction horizon . To empirically illustrate the efficacy of our approach , we consider constrained non-linear continuous control tasks : torque-limited inverted pendulum and non-holonomic vehicle kinematics . We show that NLMPC can transfer between using an inaccurate surrogate and a nominal forward model , and outperform several baselines in terms of stability . 2 PRELIMINARIES AND ASSUMPTIONS . Controlled Dynamical System Consider a discrete-time , time-invariant , deterministic system : x ( t+ 1 ) = f ( x ( t ) , u ( t ) ) , y ( t ) = x ( t ) , f ( 0 , 0 ) = 0 , ( 1 ) where t ∈ N is the timestep index , x ( t ) ∈ Rnx , u ( t ) ∈ Rnu and y ( t ) ∈ Rny are , respectively , the state , control input , and measurement at timestep t. We assume that the states and measurements are equivalent and the origin is the equilibrium point . Further , the system ( 1 ) is subjected to closed and bounded , convex constraints over the state and input spaces : x ( t ) ∈ X ⊆ Rnx , u ( t ) ∈ U ⊂ Rnu , ∀t > 0 . ( 2 ) The system is to be controlled by a feedback policy , K : Rnx → Rnu . The policy K is considered safe if there exists an invariant set , Xs ⊆ X , for the closed-loop dynamics , inside the constraints . The set Xs is also referred to as the safe-set under K. Namely , every trajectory for the closed-loop system that starts at some x ∈ Xs remains inside this set . If x asymptotically reaches the target , x̄T ∈ Xs , then Xs is a Region of Attraction ( ROA ) . In practice , convergence often occurs to a small set , XT . Lyapunov Conditions and Safety We formally assess the safety of the closed-loop system in terms of the existence of the positively invariant-set , Xs , inside the state constraints . This is done by means of a learned CLF , V ( x ) , given data generated under a ( initially unknown ) policy , K ( x ) . The candidate CLF needs to satisfy certain properties . First , it needs to be upper and lower bounded by strictly increasing , unbounded , positive ( K∞ ) functions ( Khalil , 2014 ) . We focus on optimal control with a quadratic stage cost and assume the origin as the target state : � ( x , u ) = xTQx+ uTRu , Q � 0 , R � 0 . ( 3 ) For above , a possible choice for K∞-function is the scaled sum-of-squares of the states : l��x�22 ≤ V ( x ) ≤ LV �x�22 , ( 4 ) where l� and LV are the minimum eigenvalue of Q and a Lipschitz constant for V respectively . Further for safety , the convergence to a set , XT ⊂ Xs , can be verified by means of the condition : ∀x ∈ Xs\XT , u = K ( x ) ⇒ V ( f ( x , u ) ) − λV ( x ) ≤ 0 , with λ ∈ [ 0 , 1 ) . ( 5 ) This means that to have stability V ( x ) must decrease along the closed-loop trajectory in the annulus . The sets Xs , XT , satisfying ( 5 ) , are ( positively ) invariant . If they are inside constraints , i.e . Xs ⊆ X , then they are safe . For a valid Lyapunov function V , the outer safe-set can be defined as a level-set : Xs = { x ∈ X : V ( x ) ≤ ls } . ( 6 ) For further definitions , we refer the reader to Blanchini & Miani ( 2007 ) ; Kerrigan ( 2000 ) . If condition ( 5 ) , holds everywhere in Xs , then the origin is a stable equilibrium ( XT = { 0 } ) . If ( most likely ) this holds only outside a non-empty inner set , XT = { x ∈ X : V ( x ) ≤ lT } ⊂ Xs , with XT ⊃ { 0 } , then the system converges to a neighborhood of the origin and remains there in the future . Approach Rationale We aim to match or enlarge the safe region of an unknown controller , Ki ( x ) . For a perfect model , f , and a safe set X ( i ) s , there exists an α � 1 , such that the one-step MPC : K ( x ) = argmin u∈U , f ( x , u ) ∈X ( i ) s αV ( f ( x , u ) ) + � ( x , u ) , ( 7 ) results in a new safe set , X ( i+1 ) s = C ( X ( i ) s ) , the one-step controllable set of X ( i ) s and the feasible region of ( 7 ) , X ( i+1 ) s ⊇ X ( i ) s . We soften the state constraints in ( 7 ) and use it recursively to estimate X ( j ) s , j > i . We formulate an algorithm that learns the parameter α as well as the safe set . We train a neural network via SGD to approximate V , hence the ROA estimate will not always increase through iterations . To aim for max ROA and minimum MPC horizon , we use cross-validation and verification . We motivate our work by extending theoretical results on MPC stability and a sub-optimality bound for approximate f and V . Finally , we provide an error bound on the learned f for having stability . Learning and Safety Verification We wish to learn V ( x ) and Xs from one-step on-policy rollouts , as well as a forward model f̂ ( x , u ) . After the learning , the level ls defining set Xs will be refined and its safety formally verified a posteriori . This is done by evaluating ( 5 ) using the model , starting from a large set of states sampled uniformly within Xs\XT . We progressively decrease the level ls starting from the learned one , and also increase the inner level lT , starting from zero , such that condition ( 5 ) holds for n ≥ ns samples in Xs\XT . The number of verifying samples , ns , provides a probability lower bound on safety , namely Psafe ( Xs\XT ) ≥ p ( ns ) , as detailed in ( Bobiti & Lazar , 2016 ) . The algorithm is detailed in appendix and based on ( Bobiti & Lazar , 2016 ) . For our theoretical results , we assume the search terminates with Psafe ( Xs\XT ) ≈ 1 and consider the condition deterministic . NN-based dynamics model In some MPC applications , it might not be possible to gather sufficient data from demonstrations in order to be able to learn a model that predicts over long sequences . One-step or few-steps dynamics learning based on NNs can suffer when the model is unrolled for longer time . For instance , errors can accumulate through the horizon due to small instabilities either from the physical system or as artifacts from short sequence learning . Although some mitigations are possible for specific architectures or longer sequences ( Armenio et al. , 2019 ; Doan et al. , 2019 ; Pathak et al. , 2017 ; Ciccone et al. , 2018 ) , we formulate our MPC to allow for a very short horizon and unstable dynamics . Since we learn a surrogate NN forward model , f̂ ( x , u ) , from one-step trajectories , we will assume this to have a locally bounded one-step-ahead prediction error , w ( t ) , where : w = f ( x , u ) − f̂ ( x , u ) , �w�2 ≤ µ , ∀ ( x , u ) ∈ X̃× U , ( 8 ) for some compact set of states , X̃ ⊇ X . We also assume that both f and f̂ are locally Lipschitz in this set , with constants Lfx , Lfu , and Lf̂x , Lf̂u respectively . A conservative value of µ can be inferred from these constants as the input and state sets are bounded . It can also be estimated from a test set .

Summary: This paper addresses the question of how to stabilize a system in a vicinity of an equilibrium. While the majority of reinforcement learning algorithms rely on trial and error, which may damage the system, the authors introduce an algorithm for safe exploration and control. A traditional approach in model-based RL is to use MPC with a surrogate forward model to minimize a planning objective comprising a sum of stage costs along with a terminal cost, often chosen as an approximated value function -i.e. the optimal expected cost-to-go- which can be learned by a Bellman equation. Instead, this work is placed in the framework of Robust MPC, where this value function is replaced by a Luyapunov function $V$, which is related to the notion of stability and is only constrained to decrease along trajectories. Such a Luyapunov function, when available, provides both a safe region, defined as a level-set of V, and a MPC policy for which stability analyses have been developed: the authors extend a result from Limon et al. (2003; 2009) to show that this MPC policy enjoys asymptotic stability in general, and input-to-state stability in the presence of small enough model errors. Accordingly, the authors propose a scheme allowing to learn a Lyapunov function $V$ from demonstration data only, through a loss function that penalizes increments of $V$ along one-step transitions. A regularization parameter $\alpha$ of this MPC, which balances stability with constraints satisfaction and stage costs, is also learned jointly by an alternative training procedure. This approach is evaluated empirically on two standard constrained non-linear continuous control tasks.