Stabilizing Dynamical Systems via Policy Gradient Methods

1 Introduction . Stabilizing an unknown control system is one of the most fundamental problems in control systems engineering . A wide variety of tasks - from maintaining a dynamical system around a desired equilibrium point , to tracking a reference signal ( e.g a pilot ’ s input to a plane ) - can be recast in terms of stability . More generally , synthesizing an initial stabilizing controller is often a necessary first step towards solving more complex tasks , such as adaptive or robust control design [ Sontag , 1999 , 2009 ] . In this work , we consider the problem of finding a stabilizing controller for an unknown dynamical system via direct policy search methods . We introduce a simple procedure based off policy gradients which provably stabilizes a dynamical system around an equilibrium point . Our algorithm only requires access to a simulator which can return rollouts of the system under different control policies , and can efficiently stabilize both linear and smooth , nonlinear systems . Relative to model-based approaches , model-free procedures , such as policy gradients , have two key advantages : they are conceptually simple to implement , and they are easily adaptable ; that is , the same method can be applied in a wide variety of domains without much regard to the intricacies of the underlying dynamics . Due to their simplicity and flexibility , direct policy search methods have become increasingly popular amongst practitioners , especially in settings with complex , nonlinear dynamics which may be challenging to model . In particular , they have served as the main workhorse for recent breakthroughs in reinforcement learning and control [ Silver et al. , 2016 , Mnih et al. , 2015 , Andrychowicz et al. , 2020 ] . Despite their popularity amongst practitioners , model-free approaches for continuous control have only recently started to receive attention from the theory community [ Fazel et al. , 2018 , Kakade et al. , 2020 , Tu and Recht , 2019 ] . While these analyses have begun to map out the computational and statistical tradeoffs that emerge in choosing between model-based and model-free approaches , ∗Correspondence to jcperdomo @ berkeley.edu 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . they all share a common assumption : that the unknown dynamical system in question is stable , or that an initial stabilizing controller is known . As such , they do not address the perhaps more basic question , how do we arrive at a stabilizing controller in the first place ? 1.1 Contributions . We establish a reduction from stabilizing an unknown dynamical system to solving a series of discounted , infinite-horizon LQR problems via policy gradients , for which no knowledge of an initial stable controller is needed . Our approach , which we call discount annealing , gradually increases the discount factor and yields a control policy which is near optimal for the undiscounted LQR objective . To the best of our knowledge , our algorithm is the first model-free procedure shown to provably stabilize unknown dynamical systems , thereby solving an open problem from Fazel et al . [ 2018 ] . We begin by studying linear , time-invariant dynamical systems with full state observation and assume access to inexact cost and gradient evaluations of the discounted , infinite-horizon LQR cost of a state-feedback controller K. Previous analyses ( e.g. , [ Fazel et al. , 2018 ] ) establish how such evaluations can be implemented with access to ( finitely many , finite horizon ) trajectories sampled from a simulator . We show that our method recovers the controller K ? which is the optimal solution of the undiscounted LQR problem in a bounded number of iterations , up to optimization and simulator error . The stability of the resulting K ? is guaranteed by known stability margin results for LQR . In short , we prove the following guarantee : Theorem 1 ( informal ) . For linear systems , discount annealing returns a stabilizing state-feedback controller which is also near-optimal for the LQR problem . It uses at most polynomially many , ε-inexact gradient and cost evaluations , where the tolerance ε also depends polynomially on the relevant problem parameters . Since both the number of queries and error tolerance are polynomial , discount annealing can be efficiently implemented using at most polynomially many samples from a simulator . Furthermore , our results extend to smooth , nonlinear dynamical systems . Given access to a simulator that can return damped system rollouts , we show that our algorithm finds a controller that attains near-optimal LQR cost for the Jacobian linearization of the nonlinear dynamics at the equilibrium . We then show that this controller stabilizes the nonlinear system within a neighborhood of its equilibrium . Theorem 2 ( informal ) . Discount annealing returns a state-feedback controller which is exponentially stabilizing for smooth , nonlinear systems within a neighborhood of their equilibrium , using again only polynomially many samples drawn from a simulator . In each case , the algorithm returns a near optimal solution K to the relevant LQR problem ( or local approximation thereof ) . Hence , the stability properties of K are , in theory , no better than those of the optimal LQR controller K ? . Importantly , the latter may have worse stability guarantees than the optimal solution of a corresponding robust control objective ( e.g . H∞ synthesis ) . Nevertheless , we focus on the LQR subroutine in the interest of simplicity , clarity , and in order to leverage prior analyses of model-free methods for LQR . Extending our procedure to robust-control objectives is an exciting direction for future work . Lastly , while our theoretical analysis only guarantees that the resulting controller will be stabilizing within a small neighborhood of the equilibrium , our simulations on nonlinear systems , such as the nonlinear cartpole , illustrate that discount annealing produces controllers that are competitive with established robust control procedures , such as H∞ synthesis , without requiring any knowledge of the underlying dynamics . 1.2 Related work . Given its central importance to the field , stabilization of unknown and uncertain dynamical systems has received extensive attention within the controls literature . We review some of the relevant literature and point the reader towards classical texts for a more comprehensive treatment [ Sontag , 2009 , Sastry and Bodson , 2011 , Zhou et al. , 1996 , Callier and Desoer , 2012 , Zhou and Doyle , 1998 ] . Model-based approaches . Model-based methods construct approximate system models in order to synthesize stabilizing control policies . Traditional analyses consider stabilization of both linear and nonlinear dynamical systems in the asymptotic limit of sufficient data [ Sastry and Bodson , 2011 , Sastry , 2013 ] . More recent , non-asymptotic studies have focused almost entirely on linear systems , where the controller is generated using data from multiple independent trajectories [ Fiechter , 1997 , Dean et al. , 2019 , Faradonbeh et al. , 2018a , 2019 ] . Assuming the model is known , stabilizing policies may also be synthesized via convex optimization [ Prajna et al. , 2004 ] by combining a ‘ dual Lyapunov theorem ’ [ Rantzer , 2001 ] with sum-of-squares programming [ Parrilo , 2003 ] . Relative to these analysis our focus is on strengthening the theoretical foundations of model-free procedures and establishing rigorous guarantees that policy gradient methods can also be used to generate stabilizing controllers . Online control . Online control studies the problem of adaptively fine-tuning the performance of an already-stabilizing control policy on a single trajectory [ Dean et al. , 2018 , Faradonbeh et al. , 2018b , Cohen et al. , 2019 , Mania et al. , 2019 , Simchowitz and Foster , 2020 , Hazan et al. , 2020 , Simchowitz et al. , 2020 , Kakade et al. , 2020 ] . Though early papers in this direction consider systems without pre-given stabilizing controllers [ Abbasi-Yadkori and Szepesvári , 2011 ] , their guarantees degrade exponentially in the system dimension ( a penalty ultimately shown to be unavoidable by Chen and Hazan [ 2021 ] ) . Rather than fine-tuning an already stabilizing controller , we focus on the more basic problem of finding a controller which is stabilizing in the first place , and allow for the use of multiple independent trajectories . Model-free approaches . Model-free approaches eschew trying to approximate the underlying dynamics and instead directly search over the space of control policies . The landmark paper of Fazel et al . [ 2018 ] proves that , despite the non-convexity of the problem , direct policy search on the infinite-horizon LQR objective efficiently converges to the globally optimal policy , assuming the search is initialized at an already stabilizing controller . Fazel et al . [ 2018 ] pose the synthesis of this initial stabilizing controller via policy gradients as an open problem ; one that we solve in this work . Following this result , there have been a large number of works studying policy gradients procedures in continuous control , see for example Feng and Lavaei [ 2020 ] , Malik et al . [ 2019 ] , Mohammadi et al . [ 2020 , 2021 ] , Zhang et al . [ 2021 ] just to name a few . Relative to our analysis , these papers consider questions of policy finite-tuning , derivative-free methods , and robust ( or distributed ) control which are important , yet somewhat orthogonal to the stabilization question considered herein . The recent analysis by Lamperski [ 2020 ] is perhaps the most closely related piece of prior work . It proposes a model-free , off-policy algorithm for computing a stabilizing controller for deterministic LQR systems . Much like discount annealing , the algorithm also works by alternating between policy optimization ( in their case by a closed-form policy improvement step based on the Riccati update ) and increasing a damping factor . However , whereas we provide precise finite-time convergence guarantees to a stabilizing controller for both linear and nonlinear systems , the guarantees in Lamperski [ 2020 ] are entirely asymptotic and restricted to linear systems . Furthermore , we pay special attention to quantifying the various error tolerances in the gradient and cost queries to ensure that the algorithm can be efficiently implemented in finite samples . 1.3 Background on stability of dynamical systems . Before introducing our results , we first review some of the basic concepts and definitions regarding stability of dynamical systems . In this paper , we study discrete-time , noiseless , time-invariant dynamical systems with states xt ∈ Rdx and control inputs ut ∈ Rdu . In particular , given an initial state x0 , the dynamics evolves according to xt+1 = G ( xt , ut ) where G : Rdx × Rdu → Rdx is a state transition map . An equilibrium point of a dynamical system is a state x ? ∈ Rdx such that G ( x ? , 0 ) = x ? . As per convention , we assume that the origin x ? = 0 is the desired equilibrium point around which we wish to stabilize the system . This paper restricts its attention to static state-feedback policies of the form ut = Kxt for a fixed matrix K ∈ Rdu×dx . Abusing notation slightly , we conflate the matrix K with its induced policy . Our aim is to find a policy K which is exponentially stabilizing around the equilbrium point . Time-invariant , linear systems , whereG ( x , u ) = Ax+Bu are stabilizable if and only if there exists aK such thatA+BK is a stable matrix [ Callier and Desoer , 2012 ] . That is if ρ ( A+BK ) < 1 , where ρ ( X ) denotes the spectral radius , or the largest eigenvalue magnitude , of a matrix X . For general nonlinear systems , our goal is to find controllers which satisfy the following general , quantitative definition of exponential stability ( e.g Chapter 5.2 in Sastry [ 2013 ] ) . Throughout , ‖ · ‖ denotes the Euclidean norm . Definition 1.1 . A controllerK is ( m , α ) -exponentially stable for dynamicsG if there exist constants m , α > 0 such that if inputs are chosen according to ut = Kxt , the sequence of states xt+1 = G ( xt , ut ) satisfy ‖xt‖ ≤ m · exp ( −α · t ) ‖x0‖ . ( 1.1 ) Likewise , K is ( m , α ) -exponentially stable on radius r > 0 if ( 1.1 ) holds for all x0 such that ‖x0‖ ≤ r. For linear systems , a controller K is stabilizing if and only if it is stable over the entire state space , however , the restriction to stabilization over a particular radius is in general needed for nonlinear systems . Our approach for stabilizing nonlinear systems relies on analyzing their Jacobian linearization about the origin equilibrium . Given a continuously differentiable transition operator G , the local dynamics can be approximated by the Jacobian linearization ( Ajac , Bjac ) of G about the zero equilibrium ; that is Ajac : = ∇xG ( x , u ) ∣∣ ( x , u ) = ( 0,0 ) , Bjac : = ∇uG ( x , u ) ∣∣ ( x , u ) = ( 0,0 ) . ( 1.2 ) In particular , for x and u sufficiently small , G ( x , u ) = Ajacx+Bjacu+ fnl ( x , u ) , where fnl ( x , u ) is a nonlinear remainder from the Taylor expansion of G. To ensure stabilization via state-feedback is feasible , we assume throughout our presentation that the linearized dynamics ( Ajac , Bjac ) are stabilizable .

Summary: This work gives a method to compute stabilizing controllers given access to noisy gradient evaluations with respect to the policy. The challenge is that if a controller does not stabilize the system, then the cost is infinite and the gradient is undefined. They key insight is to use the discount factor as  a free parameter. Namely, for sufficiently small discount factors, the cost is finite and the gradient can be evaluated. So, the algorithm alternates between taking gradient steps and increasing the discount factor. When a discount factor of 1 is reached, stabilization has been obtained and the problem reduces to a policy gradient method. Bounds on convergence and applications to nonlinear systems are given.