Model-free Learning Control of Nonlinear Stochastic Systems with Stability Guarantee

1 INTRODUCTION . Control of discrete-time nonlinear stochastic systems is an important topic in both control theory and reinforcement learning . In the past decades , the advancement of nonlinear control theory in the control community has been successfully applied in aircraft , automobiles , advanced robots and space systems ( Slotine et al. , 1991 ; Isidori , 1995 ) . Concurrently , reinforcement learning was developed in the machine learning community to address similar nonlinear control problems ( Sutton et al. , 1992 ; Tesauro , 1995 ; Bertsekas & Tsitsiklis , 1996 ) . Until recently , significant progress has been made by combining advances in deep learning ( LeCun et al. , 2015 ) with reinforcement learning . Impressive results are obtained in a series of high-dimensional continuous nonlinear control problems ( Duan et al. , 2016 ; Zhang et al. , 2016 ; Zhu et al. , 2017 ; Gu et al. , 2017 ) in which control-theoretic approach is typically difficult to apply . Given a control system , regardless of which controller design method is used , control theory or reinforcement learning , the first and most important property of a system needs to be guaranteed is stability , because an unstable control system is typically useless and potentially dangerous ( Slotine et al. , 1991 ) . Qualitatively , a system is described as stable if starting the system in the neighborhood of its desired operating point implies that it will stay around the point ever after . For aircraft control systems , a typical stability problem is intuitively related to the following question : will a trajectory perturbation caused by a gust result in a significant deviation in the later flight trajectory ? Here , the desired operating point of the system is the flight trajectory in the absence of disturbance . Every control system , whether linear or nonlinear , involves a stability problem which should be carefully studied . 1https : //www.dropbox.com/sh/j9mhvi0vydu7x7c/AACwJbqU5MCLcKPGgcOv0zrHa ? dl=0 The most useful and general approach for studying the stability of control systems is Lyapunov method ( Lyapunov , 1892 ) , which is dominant in control engineering ( Åström & Wittenmark , 1989 ; Mayne et al. , 2000 ) . In Lyapunov method , a scalar “ energy-like ” function called Lyapunov function is constructed to analyze the stability of the system . For a linear dynamical system , a quadratic function is typically chosen as Lyapunov function in some classic controller design method , such as linear quadratic regulator ( LQR ) and model predictive control ( MPC ) . Unfortunately , there is no universal method for constructing Lyapunov functions . In this paper , we propose a stability guaranteed reinforcement learning framework to jointly learn the controller or policy2 and a Lyapunov function both of which are parameterized by deep neural networks , with a focus on stabilization and tracking problems in discrete-time nonlinear stochastic systems modeled by Markov decision process . The contribution of our paper can be summarized as follows : 1 ) a novel data-based approach for analyzing the stability of the closed-loop system is proposed by constructing a Lyapunov function parameterized by deep neural network ; 2 ) a practical learning algorithm is designed to search the stability guaranteed controller ; 3 ) the learned controller is able to stabilize the system when interfered by uncertainties such as unseen disturbance and system parameters variations of certain extent . In our experiment , we can show that the stability guaranteed controller is more capable of handling uncertainties compared to those without such guarantees in nonlinear control problems including classic CartPole stabilization tasks , control of 3D legged robots and manipulator and reference tracking tasks for synthetic biology gene regulatory networks . 1.1 RELATED WORKS . In model-free reinforcement learning ( RL ) , stability is rarely addressed due to the formidable challenge of analyzing and designing the closed-loop system dynamics in a model-free manner ( Buşoniu et al. , 2018 ) , and the associated stability theory in model-free RL remains as an open problem ( Buşoniu et al. , 2018 ; Gorges , 2017 ) . Recently , Lyapunov analysis is used in model-free RL to solve control problems with safety constraints ( Chow et al. , 2018 ; 2019 ) . In Chow et al . ( 2018 ) , Lyapunov-based approach for solving constrained Markov decision process is proposed with a novel way of constructing the Lyapunov function through linear programming . In Chow et al . ( 2019 ) , the above results were further generalized to continuous control tasks . Even though Lyapunov-based methods were adopted in these results , neither of them addressed the stability of the system . As a basic tool in control theory , the construction/learning of Lyapunov function is not a trivial issue and many works are devoted to this problem . In Perkins & Barto ( 2002 ) , the RL agent controls the switch between controllers designed using Lyapunov domain knowledge , so that any policy is safe and reliable . Petridis & Petridis ( 2006 ) proposes a straightforward approach for constructing Lyapunov function for nonlinear systems using neural networks . Richards et al . ( 2018 ) proposes a learning-based approach for constructing Lyapunov neural networks with maximized region of attraction . Results on learning and construction of Lyapunov functions are referred to Noroozi et al . ( 2008 ) ; Prokhorov ( 1994 ) ; Serpen ( 2005 ) ; Prokhorov & Feldkamp ( 1999 ) . Other interesting results on the stability of learning-based control systems are reported in recent years . In Postoyan et al . ( 2017 ) , an initial result is proposed for the stability analysis of deterministic nonlinear systems with optimal controller for infinite-horizon discounted cost , based on the assumption that discount is sufficiently close to 1 . In Berkenkamp et al . ( 2017 ) , a learning model-based safe RL approach with safety guarantee during exploration is introduced but limited to Lipschitz continuous nonlinear systems such as Gaussian process model . In addition , the verification of stability condition requires the discretization of state space , which limits its application to tasks with low-dimensional finite state space . 2 PROBLEM STATEMENT . We consider discrete-time nonlinear stochastic systems modeled by the Markov decision process ( MDP ) . A MDP is defined by the tuple ( S , A , c , P , ρ ) , where S ⊆ Rn is the set of states , A ⊆ Rm is the set of actions , c ( s , a ) : S ×A → R+ is the cost function , P ( s′|s , a ) is the transition probability function , and ρ ( s ) is the starting state distribution . 2Controller and policy will be used interchangeably throughout the paper . In this paper , we focus on the stabilization and tracking problems for discrete-time nonlinear stochastic systems modeled by MDP . For both problems , the goal is to find a policy π which can bring the cost c to zero . In stabilization problems , the cost function is defined as the norm of states ‖s‖ where ‖ · ‖ denotes the Euclidean norm . In tracking problems , we divide the state s into two vectors , s1 and s2 , where s1 is composed of elements of s that are aimed at tracking the reference signal r while s2 contains the rest . For tracking , ‖s1 − r‖ is chosen to be the cost function . From a control theoretic perspective , the task of stabilization and tracking could be addressed as ensuring the closed-loop system or error system to be asymptotically stable , i.e. , starting from an initial point , the trajectories of state always converge to a single point or the reference trajectory . Let cπ ( s ) , Ea∼πc ( s , a ) denote the cost function under policy π , the definition of stability studied in this paper is given as follows . Definition 1 The stochastic system is said to be stable in mean cost if limt→∞ Estcπ ( st ) = 0 holds for any initial condition s0 ∈ { s0|cπ ( s0 ) ≤ b } . If b is arbitrarily large then the stochastic system is globally stable in mean cost . Remark 1 Form of the cost is strictly ruled as the Euclidean norm of state or partial state , while other forms are not considered in this paper . The stability studied in this paper is a type of local asymptotic stochastic stability , which is different to the definition of mean square stability ( MSS ) that extensively studied on stochastic systems in control theory ( Shaikhet , 1997 ; Huang , 2012 ) . Before proceeding , some notations are to be defined . The closed-loop transition probability is denoted as Pπ ( s′|s ) , ∫ A π ( a|s ) P ( s ′|s , a ) da . We also introduce the closed-loop state distribution at certain instant t as P ( s|ρ , π , t ) , which could be defined in an iterative way : P ( s′|ρ , π , t + 1 ) =∫ S Pπ ( s ′|s ) P ( s|ρ , π , t ) ds , ∀t ∈ Z+ and P ( s|ρ , π , 0 ) = ρ ( s ) . 3 MAIN RESULTS . In this section , we propose the main assumptions and a new theorem . Assumption 1 The stationary distribution of state qπ ( s ) , limt→∞ P ( s|ρ , π , t ) exists . Assumption 2 There exists a positive constant b such that ρ ( s ) > 0 , ∀s ∈ { s|cπ ( s ) ≤ b } . Our approach is to construct/find a Lyapunov function which can be used to analyze the stability of the closed-loop system . The Lyapunov method has long been used for stability analysis and controller design in control theory ( Boukas & Liu , 2000 ) , but mostly exploited along with a known model , whether deterministic or probabilistic ( Corless & Leitmann , 1981 ; Thowsen , 1983 ; Huang et al. , 2011 ) . The Lyapunov function is a class of semi-positive definite functions L : S → R+ . The general idea of exploiting Lyapunov function is to ensure that the difference ( or derivative , if the system is in continuous time ) of Lyapunov function along the state trajectory is semi-negative definite , so that the state goes in the direction of decreasing the value of Lyapunov function and eventually converges to the origin or a sub-level set of Lyapunov function . Next , we give sufficient conditions for a system to be stable in mean cost in the following . Theorem 1 The stochastic system is stable in mean cost if there exists a function L : S → R+ and positive constants α1 , α2 and α3 , such that α1cπ ( s ) ≤ L ( s ) ≤ α2cπ ( s ) ( 1 ) Es∼µπ ( Es′∼PπL ( s′ ) − L ( s ) ) ≤ −α3Es∼µπcπ ( s ) ( 2 ) where µπ ( s ) , limN→∞ 1N ∑N t=0 P ( st = s|ρ , π , t ) is the sampling distribution . Due to space limitations , we will include the detailed proof in Appendix A. Eq . ( 2 ) is called the energy decreasing condition , i.e. , requiring the expectation of Lyapunov function to be decreasing between two consecutive instants . Eq . ( 1 ) is the constraint for Lyapunov function , though a rather broad range of parameterization is covered . The sum of quadratic polynomials , e.g. , L ( s ) = sTQs where Q is a positive definite matrix , are extensively used in the control theory . Such Lyapunov functions can be efficiently discovered by the semi-definite programming solvers and bring in limited conservatism for the control tasks where the cost are also of a quadratic form . In ( Richards et al. , 2018 ) , a neural network φθ ( · ) is designed to construct the Lyapunov function , L ( s ) = φθ ( s ) Tφθ ( s ) . As explored in ( Chow et al. , 2018 ) and ( Berkenkamp et al. , 2017 ) , the value function could be exploited as a Lyapunov function as well . Additionally , the sum of cost over a limited time horizon could also be employed as Lyapunov function , i.e. , L ( s ) = ∑t+N t Ecπ ( st ) , which is a valid Lyapunov candidate in model predictive control literature ( Mayne & Michalska , 1990 ; Mayne et al. , 2000 ) . The choice of Lyapunov function candidate plays an important role in learning a policy . Value function evaluates the infinite time horizon and thus offers a better performance in general , but is rather difficult to approximate because of significant variance and bias ( Schulman et al. , 2015 ) . On the other hand , the finite horizon sum of cost provides an explicit target for learning a Lyapunov function , thus inherently reduces the bias and enhances the learning process . However , as the model is unknown , predicting the future costs based on the current state and action inevitably introduces variance , which grows as the prediction horizon extends . In principle , for tasks with simple dynamics , the sum-of-cost choice enhances the convergence of learning and robustness of the trained policies , while for complicated systems the choice of value function generally produces better performance . In this paper , we use both value function and sum-of-cost over various horizons as Lyapunov function candidates in different tasks and compare their strength and weakness respectively . Now we would like to give the following two remarks . Remark 2 This remark is on Assumption 1 and sampling distribution µπ . If an MDP is ergodic then the existence of qπ is naturally assured , but all states have to be positive recurrent and aperiodic ( Papoulis & Pillai , 2002 ) . The existence of sampling distribution µπ ( s ) is guaranteed by the existence of qπ ( s ) . Since the sequence { P ( s|ρ , π , t ) , t ∈ Z+ } converges to qπ ( s ) as t approaches∞ , then by the Abelian theorem , the sequence { 1N ∑N t=0 P ( s|ρ , π , t ) , N ∈ Z+ } also converges and µπ ( s ) = qπ ( s ) . Thus we use µπ to approximate the qπ since the evaluation of qπ requires data to be sampled after infinite instants the episode begins . Even if the sampling period N < < ∞ , one can still assure that cπ converges to a neighborhood of zero , which is related to the initial state distribution and length of N , i.e. , EsN cπ ( sN ) ≤ α2α1+α3Eρcπ ( s0 ) − α3 α1+α3 ∑N−1 t=0 Estcπ ( st ) . Remark 3 This remark is on the connection to previous results concerning the stability of stochastic systems . It should be noted that the stability conditions of Markov chains have been reported in ( Shaikhet , 1997 ; Meyn & Tweedie , 2012 ) , however , of which the validation requires the full knowledge of the model , i.e. , the transition probability P ( s′|s , a ) . On the contrary , our approach solely depends on data to analyze the stability of the closed-loop system , which further enables the model-free learning algorithms with stability guarantee . However , the validation of stability through a sample-based approach theoretically requires tremendous , if not infinite , amount of samples to thoroughly estimate the distributions , which is the drawback of our approach . We would demonstrate empirically that the algorithm built upon this theorem is reliable though only limited sample is collected .

Summary: In this paper, the authors introduce an algorithm to learn a stable controller using deep NN actor-critic method. They define the stability in the mean cost criteria,  which is used to constrain the critic network as a Lyapunov function. In addition, the semi-positive definiteness of the Lyapunov function is enforced by constructing the critic.