Enforcing robust control guarantees within neural network policies

1 INTRODUCTION . The field of robust control , dating back many decades , has been able to provide rigorous guarantees on when controllers will succeed or fail in controlling a system of interest . In particular , if the uncertainties in the underlying dynamics can be bounded in specific ways , these techniques can produce controllers that are provably robust even under worst-case conditions . However , as the resulting policies tend to be simple ( i.e. , often linear ) , this can limit their performance in typical ( rather than worst-case ) scenarios . In contrast , recent high-profile advances in deep reinforcement learning have yielded state-of-the-art performance on many control tasks , due to their ability to capture complex , nonlinear policies . However , due to a lack of robustness guarantees , these techniques have still found limited application in safety-critical domains where an incorrect action ( either during training or at runtime ) can substantially impact the controlled system . In this paper , we propose a method that combines the guarantees of robust control with the flexibility of deep reinforcement learning ( RL ) . Specifically , we consider the setting of nonlinear , time-varying systems with unknown dynamics , but where ( as common in robust control ) the uncertainty on these dynamics can be bounded in ways amenable to obtaining provable performance guarantees . Building upon specifications provided by traditional robust control methods in these settings , we construct a new class of nonlinear policies that are parameterized by neural networks , but that are nonetheless provably robust . In particular , we project the outputs of a nominal ( deep neural network-based ) controller onto a space of stabilizing actions characterized by the robust control specifications . The resulting nonlinear control policies are trainable using standard approaches in deep RL , yet are guaranteed to be stable under the same worst-case conditions as the original robust controller . We describe our proposed deep nonlinear control policy class and derive efficient , differentiable projections for this class under various models of system uncertainty common in robust control . We demonstrate our approach on several different domains , including synthetic linear differential inclusion ( LDI ) settings , the cart-pole task , a quadrotor domain , and a microgrid domain . Although these domains are simple by modern RL standards , we show that purely RL-based methods often produce unstable policies in the presence of system disturbances , both during and after training . In contrast , we show that our method remains stable even when worst-case disturbances are present , while improving upon the performance of traditional robust control methods . 2 RELATED WORK . We employ techniques from robust control , ( deep ) RL , and differentiable optimization to learn provably robust nonlinear controllers . We discuss these areas of work in connection to our approach . Robust control . Robust control is concerned with the design of feedback controllers for dynamical systems with modeling uncertainties and/or external disturbances ( Zhou and Doyle , 1998 ; Başar and Bernhard , 2008 ) , specifically controllers with guaranteed performance under worst-case conditions . Many classes of robust control problems in both the time and frequency domains can be formulated using linear matrix inequalities ( LMIs ) ( Boyd et al. , 1994 ; Kothare et al. , 1996 ) ; for reasonably-sized problems , these LMIs can be solved using off-the-shelf numerical solvers based on interior-point or first-order ( gradient-based ) methods . However , providing stability guarantees often requires the use of simple ( linear ) controllers , which greatly limits average-case performance . Our work seeks to improve performance via nonlinear controllers that nonetheless retain the same stability guarantees . Reinforcement learning ( RL ) . In contrast , RL ( and specifically , deep RL ) is not restricted to simple controllers or problems with uncertainty bounds on the dynamics . Instead , deep RL seeks to learn an optimal control policy , represented by a neural network , by directly interacting with an unknown environment . These methods have shown impressive results in a variety of complex control tasks ( e.g. , Mnih et al . ( 2015 ) ; Akkaya et al . ( 2019 ) ) ; see Buşoniu et al . ( 2018 ) for a survey . However , due to its lack of safety guarantees , deep RL has been predominantly applied to simulated environments or highly-controlled real-world problems , where system failures are either not costly or not possible . Efforts to address the lack of safety and stability in RL fall into several main categories . The first tries to combine control-theoretic ideas , predominantly robust control , with the nonlinear control policy benefits of RL ( e.g. , Morimoto and Doya ( 2005 ) ; Abu-Khalaf et al . ( 2006 ) ; Feng et al . ( 2009 ) ; Liu et al . ( 2013 ) ; Wu and Luo ( 2013 ) ; Luo et al . ( 2014 ) ; Friedrich and Buss ( 2017 ) ; Pinto et al . ( 2017 ) ; Jin and Lavaei ( 2018 ) ; Chang et al . ( 2019 ) ; Han et al . ( 2019 ) ; Zhang et al . ( 2020 ) ) . For example , RL has been used to address stochastic stability in H∞ control synthesis settings by jointly learning Lyapunov functions and policies in these settings ( Han et al. , 2019 ) . As another example , RL has been used to address H∞ control for continuous-time systems via min-max differential games , in which the controller and disturbance are the “ minimizer ” and “ maximizer ” ( Morimoto and Doya , 2005 ) . We view our approach as thematically aligned with this previous work , though our method is able to capture not only H∞ settings , but also a much broader class of robust control settings . Another category of methods addressing this challenge is safe RL , which aims to learn control policies while maintaining some notion of safety during or after learning . Typically , these methods attempt to restrict the RL algorithm to a safe region of the state space by making strong assumptions about the smoothness of the underlying dynamics , e.g. , that the dynamics can be modeled as a Gaussian process ( GP ) ( Turchetta et al. , 2016 ; Akametalu et al. , 2014 ) or are Lipschitz continuous ( Berkenkamp et al. , 2017 ; Wachi et al. , 2018 ) . This framework is in theory more general than our approach , which requires using stringent uncertainty bounds ( e.g . state-control norm bounds ) from robust control . However , there are two key benefits to our approach . First , norm bounds or polytopic uncertainty can accommodate sharp discontinuities in the continuous-time dynamics . Second , convex projections ( as used in our method ) scale polynomially with the state-action size , whereas GPs in particular scale exponentially ( and are therefore difficult to extend to high-dimensional problems ) . A third category of methods uses Constrained Markov Decision Processes ( C-MDPs ) . These methods seek to maximize a discounted reward while bounding some discounted cost function ( Altman , 1999 ; Achiam et al. , 2017 ; Taleghan and Dietterich , 2018 ; Yang et al. , 2020 ) . While these methods do not require knowledge of the cost functions a-priori , they only guarantee the cost constraints hold during test time . Additionally , using C-MDPs can yield other complications , such as optimal policies being stochastic and the constraints only holding for a subset of states . Differentiable optimization layers . A great deal of recent work has studied differentiable optimization layers for neural networks : e.g. , layers for quadratic programming ( Amos and Kolter , 2017 ) , SAT solving ( Wang et al. , 2019 ) , submodular optimization ( Djolonga and Krause , 2017 ; Tschiatschek et al. , 2018 ) , cone programs ( Agrawal et al. , 2019 ) , and other classes of optimization problems ( Gould et al. , 2019 ) . These layers can be used to construct neural networks with useful inductive bias for particular domains or to enforce that networks obey hard constraints dictated by the settings in which they are used . We create fast , custom differentiable optimization layers for the latter purpose , namely , to project neural network outputs into a set of certifiably stabilizing actions . 3 BACKGROUND ON LQR AND ROBUST CONTROL SPECIFICATIONS . In this paper , our aim is to control nonlinear ( continuous-time ) dynamical systems of the form ẋ ( t ) ∈ A ( t ) x ( t ) +B ( t ) u ( t ) +G ( t ) w ( t ) , ( 1 ) where x ( t ) ∈ Rs denotes the state at time t ; u ( t ) ∈ Ra is the control input ; w ( t ) ∈ Rd captures both external ( possibly stochastic ) disturbances and any modeling discrepancies ; ẋ ( t ) denotes the time derivative of the state x at time t ; and A ( t ) ∈ Rs×s , B ( t ) ∈ Rs×a , G ( t ) ∈ Rs×d . This class of models is referred to as linear differential inclusions ( LDIs ) ; however , we note that despite the name , this class does indeed characterize nonlinear systems , as , e.g. , w ( t ) can depend arbitrarily on x ( t ) and u ( t ) ( though we omit this dependence in the notation for brevity ) . Within this class of models , it is often possible to construct robust control specifications certifying system stability . Given such specifications , our proposal is to learn nonlinear ( deep neural network-based ) policies that provably satisfy these specifications while optimizing some objective of interest . We start by giving background on the robust control specifications and objectives considered in this work . 3.1 ROBUST CONTROL SPECIFICATIONS . In the continuous-time , infinite-horizon settings we consider here , the goal of robust control is often to construct a time-invariant control policy u ( t ) = π ( x ( t ) ) , alongside some certification that guarantees that the controlled system will be stable ( i.e. , that trajectories of the system will converge to an equilibrium state , usually x = 0 by convention ; see Haddad and Chellaboina ( 2011 ) for a more formal definition ) . For many classes of systems,1 this certification is typically in the form of a positive definite Lyapunov function V : Rs → R , with V ( 0 ) = 0 and V ( x ) > 0 for all x 6= 0 , such that the function is decreasing along trajectories – for instance , V̇ ( x ( t ) ) ≤ −αV ( x ( t ) ) ( 2 ) for some design parameter α > 0 . ( This particular condition implies exponential stability with a rate of convergence α.2 ) For certain classes of bounded dynamical systems , time-invariant linear control policies u ( t ) = Kx ( t ) , and quadratic Lyapunov functions V ( x ) = xTPx , it is possible to construct such guarantees using semidefinite programming . For instance , consider the class of norm-bounded LDIs ( NLDIs ) ẋ = Ax ( t ) +Bu ( t ) +Gw ( t ) , ‖w ( t ) ‖2 ≤ ‖Cx ( t ) +Du ( t ) ‖2 , ( 3 ) where A ∈ Rs×s , B ∈ Rs×a , G ∈ Rs×d , C ∈ Rk×s , and D ∈ Rk×a are time-invariant and known , and the disturbance w ( t ) is arbitrary ( and unknown ) but obeys the norm bounds above.3 For these systems , it is possible to specify a set of stabilizing policies via a set of linear matrix inequalities ( LMIs , Boyd et al . ( 1994 ) ) : [ AS + SAT + µGGT +BY + Y TBT + αS SCT + Y TDT CS +DY −µI ] 0 , S 0 , µ > 0 , ( 4 ) where S ∈ Rs×s and Y ∈ Ra×s . For matrices S and Y satisfying ( 4 ) , K = Y S−1 and P = S−1 are then a stabilizing linear controller gain and Lyapunov matrix , respectively . While the LMI above is specific to NLDI systems , this general paradigm of constructing stability specifications using LMIs applies to many settings commonly considered in robust control ( e.g. , settings with normbounded disturbances or polytopic uncertainty , or H∞ control settings ) . More details about these types of formulations are given in , e.g. , Boyd et al . ( 1994 ) ; in addition , we provide the relevant LMI constraints for the settings we consider in this work in Appendix A . 1In this work , we consider sub-classes of system ( 1 ) that may indeed be stochastic ( e.g. , due to a stochastic external disturbance w ( t ) ) , but that can be bounded so as to be amenable to deterministic stability analysis . However , other settings may require stochastic stability analysis ; please see Astrom ( 1971 ) . 2See , e.g. , Haddad and Chellaboina ( 2011 ) for a more rigorous definition of ( local and global ) exponential stability . Condition ( 2 ) comes from Lyapunov ’ s Theorem , which characterizes various notions of stability using Lyapunov functions . 3A slightly more complex formulation involves an additional term in the norm bound , i.e. , Cx ( t ) +Du ( t ) + Hw ( t ) , which creates a quadratic inequality in w. The mechanics of obtaining robustness specifications in this setting are largely the same as presented here , though with some additional terms in the equations . As such , as is often done , we assume that H = 0 for simplicity .

Summary: In this paper, a neural control method is proposed with stability guarantees. The control is assumed to be from a neural network that takes in the state. Stability is guaranteed by projecting the control to the set that satisfies the Lyapunov stability condition for the LQR problem. In particular, minimizing the cost of LQR cost subject to stability constraints can be cast as an SDP for norm-bouned linear differential inclusions. Through making use of the convex optimization layers proposed in Agrawal et al. (2019), the SDP can be added as a layer after the neural policy and efficient projections can be derived such that implicit function theorem can be utilized to differentiate through the fixed point (the optimal conditions of the SDP), such that end to end learning is possible. The proposed approach is compared with the unconstrained method on various tasks.  Both model-based and model-free RL algorithms are used as the neural policy for comparison. The stability-guaranteed approach is able to remain stable even under bounded adversarial dynamics. In comparison, the non-robust methods fail to maintain stability.