Stabilizing Dynamical Systems via Policy Gradient Methods

1 Introduction . Stabilizing an unknown control system is one of the most fundamental problems in control systems engineering . A wide variety of tasks - from maintaining a dynamical system around a desired equilibrium point , to tracking a reference signal ( e.g a pilot ’ s input to a plane ) - can be recast in terms of stability . More generally , synthesizing an initial stabilizing controller is often a necessary first step towards solving more complex tasks , such as adaptive or robust control design [ Sontag , 1999 , 2009 ] . In this work , we consider the problem of finding a stabilizing controller for an unknown dynamical system via direct policy search methods . We introduce a simple procedure based off policy gradients which provably stabilizes a dynamical system around an equilibrium point . Our algorithm only requires access to a simulator which can return rollouts of the system under different control policies , and can efficiently stabilize both linear and smooth , nonlinear systems . Relative to model-based approaches , model-free procedures , such as policy gradients , have two key advantages : they are conceptually simple to implement , and they are easily adaptable ; that is , the same method can be applied in a wide variety of domains without much regard to the intricacies of the underlying dynamics . Due to their simplicity and flexibility , direct policy search methods have become increasingly popular amongst practitioners , especially in settings with complex , nonlinear dynamics which may be challenging to model . In particular , they have served as the main workhorse for recent breakthroughs in reinforcement learning and control [ Silver et al. , 2016 , Mnih et al. , 2015 , Andrychowicz et al. , 2020 ] . Despite their popularity amongst practitioners , model-free approaches for continuous control have only recently started to receive attention from the theory community [ Fazel et al. , 2018 , Kakade et al. , 2020 , Tu and Recht , 2019 ] . While these analyses have begun to map out the computational and statistical tradeoffs that emerge in choosing between model-based and model-free approaches , ∗Correspondence to jcperdomo @ berkeley.edu 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . they all share a common assumption : that the unknown dynamical system in question is stable , or that an initial stabilizing controller is known . As such , they do not address the perhaps more basic question , how do we arrive at a stabilizing controller in the first place ? 1.1 Contributions . We establish a reduction from stabilizing an unknown dynamical system to solving a series of discounted , infinite-horizon LQR problems via policy gradients , for which no knowledge of an initial stable controller is needed . Our approach , which we call discount annealing , gradually increases the discount factor and yields a control policy which is near optimal for the undiscounted LQR objective . To the best of our knowledge , our algorithm is the first model-free procedure shown to provably stabilize unknown dynamical systems , thereby solving an open problem from Fazel et al . [ 2018 ] . We begin by studying linear , time-invariant dynamical systems with full state observation and assume access to inexact cost and gradient evaluations of the discounted , infinite-horizon LQR cost of a state-feedback controller K. Previous analyses ( e.g. , [ Fazel et al. , 2018 ] ) establish how such evaluations can be implemented with access to ( finitely many , finite horizon ) trajectories sampled from a simulator . We show that our method recovers the controller K ? which is the optimal solution of the undiscounted LQR problem in a bounded number of iterations , up to optimization and simulator error . The stability of the resulting K ? is guaranteed by known stability margin results for LQR . In short , we prove the following guarantee : Theorem 1 ( informal ) . For linear systems , discount annealing returns a stabilizing state-feedback controller which is also near-optimal for the LQR problem . It uses at most polynomially many , ε-inexact gradient and cost evaluations , where the tolerance ε also depends polynomially on the relevant problem parameters . Since both the number of queries and error tolerance are polynomial , discount annealing can be efficiently implemented using at most polynomially many samples from a simulator . Furthermore , our results extend to smooth , nonlinear dynamical systems . Given access to a simulator that can return damped system rollouts , we show that our algorithm finds a controller that attains near-optimal LQR cost for the Jacobian linearization of the nonlinear dynamics at the equilibrium . We then show that this controller stabilizes the nonlinear system within a neighborhood of its equilibrium . Theorem 2 ( informal ) . Discount annealing returns a state-feedback controller which is exponentially stabilizing for smooth , nonlinear systems within a neighborhood of their equilibrium , using again only polynomially many samples drawn from a simulator . In each case , the algorithm returns a near optimal solution K to the relevant LQR problem ( or local approximation thereof ) . Hence , the stability properties of K are , in theory , no better than those of the optimal LQR controller K ? . Importantly , the latter may have worse stability guarantees than the optimal solution of a corresponding robust control objective ( e.g . H∞ synthesis ) . Nevertheless , we focus on the LQR subroutine in the interest of simplicity , clarity , and in order to leverage prior analyses of model-free methods for LQR . Extending our procedure to robust-control objectives is an exciting direction for future work . Lastly , while our theoretical analysis only guarantees that the resulting controller will be stabilizing within a small neighborhood of the equilibrium , our simulations on nonlinear systems , such as the nonlinear cartpole , illustrate that discount annealing produces controllers that are competitive with established robust control procedures , such as H∞ synthesis , without requiring any knowledge of the underlying dynamics . 1.2 Related work . Given its central importance to the field , stabilization of unknown and uncertain dynamical systems has received extensive attention within the controls literature . We review some of the relevant literature and point the reader towards classical texts for a more comprehensive treatment [ Sontag , 2009 , Sastry and Bodson , 2011 , Zhou et al. , 1996 , Callier and Desoer , 2012 , Zhou and Doyle , 1998 ] . Model-based approaches . Model-based methods construct approximate system models in order to synthesize stabilizing control policies . Traditional analyses consider stabilization of both linear and nonlinear dynamical systems in the asymptotic limit of sufficient data [ Sastry and Bodson , 2011 , Sastry , 2013 ] . More recent , non-asymptotic studies have focused almost entirely on linear systems , where the controller is generated using data from multiple independent trajectories [ Fiechter , 1997 , Dean et al. , 2019 , Faradonbeh et al. , 2018a , 2019 ] . Assuming the model is known , stabilizing policies may also be synthesized via convex optimization [ Prajna et al. , 2004 ] by combining a ‘ dual Lyapunov theorem ’ [ Rantzer , 2001 ] with sum-of-squares programming [ Parrilo , 2003 ] . Relative to these analysis our focus is on strengthening the theoretical foundations of model-free procedures and establishing rigorous guarantees that policy gradient methods can also be used to generate stabilizing controllers . Online control . Online control studies the problem of adaptively fine-tuning the performance of an already-stabilizing control policy on a single trajectory [ Dean et al. , 2018 , Faradonbeh et al. , 2018b , Cohen et al. , 2019 , Mania et al. , 2019 , Simchowitz and Foster , 2020 , Hazan et al. , 2020 , Simchowitz et al. , 2020 , Kakade et al. , 2020 ] . Though early papers in this direction consider systems without pre-given stabilizing controllers [ Abbasi-Yadkori and Szepesvári , 2011 ] , their guarantees degrade exponentially in the system dimension ( a penalty ultimately shown to be unavoidable by Chen and Hazan [ 2021 ] ) . Rather than fine-tuning an already stabilizing controller , we focus on the more basic problem of finding a controller which is stabilizing in the first place , and allow for the use of multiple independent trajectories . Model-free approaches . Model-free approaches eschew trying to approximate the underlying dynamics and instead directly search over the space of control policies . The landmark paper of Fazel et al . [ 2018 ] proves that , despite the non-convexity of the problem , direct policy search on the infinite-horizon LQR objective efficiently converges to the globally optimal policy , assuming the search is initialized at an already stabilizing controller . Fazel et al . [ 2018 ] pose the synthesis of this initial stabilizing controller via policy gradients as an open problem ; one that we solve in this work . Following this result , there have been a large number of works studying policy gradients procedures in continuous control , see for example Feng and Lavaei [ 2020 ] , Malik et al . [ 2019 ] , Mohammadi et al . [ 2020 , 2021 ] , Zhang et al . [ 2021 ] just to name a few . Relative to our analysis , these papers consider questions of policy finite-tuning , derivative-free methods , and robust ( or distributed ) control which are important , yet somewhat orthogonal to the stabilization question considered herein . The recent analysis by Lamperski [ 2020 ] is perhaps the most closely related piece of prior work . It proposes a model-free , off-policy algorithm for computing a stabilizing controller for deterministic LQR systems . Much like discount annealing , the algorithm also works by alternating between policy optimization ( in their case by a closed-form policy improvement step based on the Riccati update ) and increasing a damping factor . However , whereas we provide precise finite-time convergence guarantees to a stabilizing controller for both linear and nonlinear systems , the guarantees in Lamperski [ 2020 ] are entirely asymptotic and restricted to linear systems . Furthermore , we pay special attention to quantifying the various error tolerances in the gradient and cost queries to ensure that the algorithm can be efficiently implemented in finite samples . 1.3 Background on stability of dynamical systems . Before introducing our results , we first review some of the basic concepts and definitions regarding stability of dynamical systems . In this paper , we study discrete-time , noiseless , time-invariant dynamical systems with states xt ∈ Rdx and control inputs ut ∈ Rdu . In particular , given an initial state x0 , the dynamics evolves according to xt+1 = G ( xt , ut ) where G : Rdx × Rdu → Rdx is a state transition map . An equilibrium point of a dynamical system is a state x ? ∈ Rdx such that G ( x ? , 0 ) = x ? . As per convention , we assume that the origin x ? = 0 is the desired equilibrium point around which we wish to stabilize the system . This paper restricts its attention to static state-feedback policies of the form ut = Kxt for a fixed matrix K ∈ Rdu×dx . Abusing notation slightly , we conflate the matrix K with its induced policy . Our aim is to find a policy K which is exponentially stabilizing around the equilbrium point . Time-invariant , linear systems , whereG ( x , u ) = Ax+Bu are stabilizable if and only if there exists aK such thatA+BK is a stable matrix [ Callier and Desoer , 2012 ] . That is if ρ ( A+BK ) < 1 , where ρ ( X ) denotes the spectral radius , or the largest eigenvalue magnitude , of a matrix X . For general nonlinear systems , our goal is to find controllers which satisfy the following general , quantitative definition of exponential stability ( e.g Chapter 5.2 in Sastry [ 2013 ] ) . Throughout , ‖ · ‖ denotes the Euclidean norm . Definition 1.1 . A controllerK is ( m , α ) -exponentially stable for dynamicsG if there exist constants m , α > 0 such that if inputs are chosen according to ut = Kxt , the sequence of states xt+1 = G ( xt , ut ) satisfy ‖xt‖ ≤ m · exp ( −α · t ) ‖x0‖ . ( 1.1 ) Likewise , K is ( m , α ) -exponentially stable on radius r > 0 if ( 1.1 ) holds for all x0 such that ‖x0‖ ≤ r. For linear systems , a controller K is stabilizing if and only if it is stable over the entire state space , however , the restriction to stabilization over a particular radius is in general needed for nonlinear systems . Our approach for stabilizing nonlinear systems relies on analyzing their Jacobian linearization about the origin equilibrium . Given a continuously differentiable transition operator G , the local dynamics can be approximated by the Jacobian linearization ( Ajac , Bjac ) of G about the zero equilibrium ; that is Ajac : = ∇xG ( x , u ) ∣∣ ( x , u ) = ( 0,0 ) , Bjac : = ∇uG ( x , u ) ∣∣ ( x , u ) = ( 0,0 ) . ( 1.2 ) In particular , for x and u sufficiently small , G ( x , u ) = Ajacx+Bjacu+ fnl ( x , u ) , where fnl ( x , u ) is a nonlinear remainder from the Taylor expansion of G. To ensure stabilization via state-feedback is feasible , we assume throughout our presentation that the linearized dynamics ( Ajac , Bjac ) are stabilizable .

Summary: This paper proposes a model-free algorithm for stabilizing linear and nonlinear dynamical systems. The algorithm solves a series of discounted LQR problems using policy gradient, and provably converges to a stabilizing controller for LTI systems; for nonlinear systems, the algorithm recovers a stabilizing controller in a small ball around the origin (assuming origin is the equilibrium point). This work overcomes a significant limitation of prior work on policy gradient methods for control, which requires a stabilizing controller a priori. Experiments on a simulated nonlinear system suggest that the algorithm can produce a controller in a few iterations that is stabilizing in a ball of reasonable radius around the origin.