Model-free Learning Control of Nonlinear Stochastic Systems with Stability Guarantee

1 INTRODUCTION . Control of discrete-time nonlinear stochastic systems is an important topic in both control theory and reinforcement learning . In the past decades , the advancement of nonlinear control theory in the control community has been successfully applied in aircraft , automobiles , advanced robots and space systems ( Slotine et al. , 1991 ; Isidori , 1995 ) . Concurrently , reinforcement learning was developed in the machine learning community to address similar nonlinear control problems ( Sutton et al. , 1992 ; Tesauro , 1995 ; Bertsekas & Tsitsiklis , 1996 ) . Until recently , significant progress has been made by combining advances in deep learning ( LeCun et al. , 2015 ) with reinforcement learning . Impressive results are obtained in a series of high-dimensional continuous nonlinear control problems ( Duan et al. , 2016 ; Zhang et al. , 2016 ; Zhu et al. , 2017 ; Gu et al. , 2017 ) in which control-theoretic approach is typically difficult to apply . Given a control system , regardless of which controller design method is used , control theory or reinforcement learning , the first and most important property of a system needs to be guaranteed is stability , because an unstable control system is typically useless and potentially dangerous ( Slotine et al. , 1991 ) . Qualitatively , a system is described as stable if starting the system in the neighborhood of its desired operating point implies that it will stay around the point ever after . For aircraft control systems , a typical stability problem is intuitively related to the following question : will a trajectory perturbation caused by a gust result in a significant deviation in the later flight trajectory ? Here , the desired operating point of the system is the flight trajectory in the absence of disturbance . Every control system , whether linear or nonlinear , involves a stability problem which should be carefully studied . 1https : //www.dropbox.com/sh/j9mhvi0vydu7x7c/AACwJbqU5MCLcKPGgcOv0zrHa ? dl=0 The most useful and general approach for studying the stability of control systems is Lyapunov method ( Lyapunov , 1892 ) , which is dominant in control engineering ( Åström & Wittenmark , 1989 ; Mayne et al. , 2000 ) . In Lyapunov method , a scalar “ energy-like ” function called Lyapunov function is constructed to analyze the stability of the system . For a linear dynamical system , a quadratic function is typically chosen as Lyapunov function in some classic controller design method , such as linear quadratic regulator ( LQR ) and model predictive control ( MPC ) . Unfortunately , there is no universal method for constructing Lyapunov functions . In this paper , we propose a stability guaranteed reinforcement learning framework to jointly learn the controller or policy2 and a Lyapunov function both of which are parameterized by deep neural networks , with a focus on stabilization and tracking problems in discrete-time nonlinear stochastic systems modeled by Markov decision process . The contribution of our paper can be summarized as follows : 1 ) a novel data-based approach for analyzing the stability of the closed-loop system is proposed by constructing a Lyapunov function parameterized by deep neural network ; 2 ) a practical learning algorithm is designed to search the stability guaranteed controller ; 3 ) the learned controller is able to stabilize the system when interfered by uncertainties such as unseen disturbance and system parameters variations of certain extent . In our experiment , we can show that the stability guaranteed controller is more capable of handling uncertainties compared to those without such guarantees in nonlinear control problems including classic CartPole stabilization tasks , control of 3D legged robots and manipulator and reference tracking tasks for synthetic biology gene regulatory networks . 1.1 RELATED WORKS . In model-free reinforcement learning ( RL ) , stability is rarely addressed due to the formidable challenge of analyzing and designing the closed-loop system dynamics in a model-free manner ( Buşoniu et al. , 2018 ) , and the associated stability theory in model-free RL remains as an open problem ( Buşoniu et al. , 2018 ; Gorges , 2017 ) . Recently , Lyapunov analysis is used in model-free RL to solve control problems with safety constraints ( Chow et al. , 2018 ; 2019 ) . In Chow et al . ( 2018 ) , Lyapunov-based approach for solving constrained Markov decision process is proposed with a novel way of constructing the Lyapunov function through linear programming . In Chow et al . ( 2019 ) , the above results were further generalized to continuous control tasks . Even though Lyapunov-based methods were adopted in these results , neither of them addressed the stability of the system . As a basic tool in control theory , the construction/learning of Lyapunov function is not a trivial issue and many works are devoted to this problem . In Perkins & Barto ( 2002 ) , the RL agent controls the switch between controllers designed using Lyapunov domain knowledge , so that any policy is safe and reliable . Petridis & Petridis ( 2006 ) proposes a straightforward approach for constructing Lyapunov function for nonlinear systems using neural networks . Richards et al . ( 2018 ) proposes a learning-based approach for constructing Lyapunov neural networks with maximized region of attraction . Results on learning and construction of Lyapunov functions are referred to Noroozi et al . ( 2008 ) ; Prokhorov ( 1994 ) ; Serpen ( 2005 ) ; Prokhorov & Feldkamp ( 1999 ) . Other interesting results on the stability of learning-based control systems are reported in recent years . In Postoyan et al . ( 2017 ) , an initial result is proposed for the stability analysis of deterministic nonlinear systems with optimal controller for infinite-horizon discounted cost , based on the assumption that discount is sufficiently close to 1 . In Berkenkamp et al . ( 2017 ) , a learning model-based safe RL approach with safety guarantee during exploration is introduced but limited to Lipschitz continuous nonlinear systems such as Gaussian process model . In addition , the verification of stability condition requires the discretization of state space , which limits its application to tasks with low-dimensional finite state space . 2 PROBLEM STATEMENT . We consider discrete-time nonlinear stochastic systems modeled by the Markov decision process ( MDP ) . A MDP is defined by the tuple ( S , A , c , P , ρ ) , where S ⊆ Rn is the set of states , A ⊆ Rm is the set of actions , c ( s , a ) : S ×A → R+ is the cost function , P ( s′|s , a ) is the transition probability function , and ρ ( s ) is the starting state distribution . 2Controller and policy will be used interchangeably throughout the paper . In this paper , we focus on the stabilization and tracking problems for discrete-time nonlinear stochastic systems modeled by MDP . For both problems , the goal is to find a policy π which can bring the cost c to zero . In stabilization problems , the cost function is defined as the norm of states ‖s‖ where ‖ · ‖ denotes the Euclidean norm . In tracking problems , we divide the state s into two vectors , s1 and s2 , where s1 is composed of elements of s that are aimed at tracking the reference signal r while s2 contains the rest . For tracking , ‖s1 − r‖ is chosen to be the cost function . From a control theoretic perspective , the task of stabilization and tracking could be addressed as ensuring the closed-loop system or error system to be asymptotically stable , i.e. , starting from an initial point , the trajectories of state always converge to a single point or the reference trajectory . Let cπ ( s ) , Ea∼πc ( s , a ) denote the cost function under policy π , the definition of stability studied in this paper is given as follows . Definition 1 The stochastic system is said to be stable in mean cost if limt→∞ Estcπ ( st ) = 0 holds for any initial condition s0 ∈ { s0|cπ ( s0 ) ≤ b } . If b is arbitrarily large then the stochastic system is globally stable in mean cost . Remark 1 Form of the cost is strictly ruled as the Euclidean norm of state or partial state , while other forms are not considered in this paper . The stability studied in this paper is a type of local asymptotic stochastic stability , which is different to the definition of mean square stability ( MSS ) that extensively studied on stochastic systems in control theory ( Shaikhet , 1997 ; Huang , 2012 ) . Before proceeding , some notations are to be defined . The closed-loop transition probability is denoted as Pπ ( s′|s ) , ∫ A π ( a|s ) P ( s ′|s , a ) da . We also introduce the closed-loop state distribution at certain instant t as P ( s|ρ , π , t ) , which could be defined in an iterative way : P ( s′|ρ , π , t + 1 ) =∫ S Pπ ( s ′|s ) P ( s|ρ , π , t ) ds , ∀t ∈ Z+ and P ( s|ρ , π , 0 ) = ρ ( s ) . 3 MAIN RESULTS . In this section , we propose the main assumptions and a new theorem . Assumption 1 The stationary distribution of state qπ ( s ) , limt→∞ P ( s|ρ , π , t ) exists . Assumption 2 There exists a positive constant b such that ρ ( s ) > 0 , ∀s ∈ { s|cπ ( s ) ≤ b } . Our approach is to construct/find a Lyapunov function which can be used to analyze the stability of the closed-loop system . The Lyapunov method has long been used for stability analysis and controller design in control theory ( Boukas & Liu , 2000 ) , but mostly exploited along with a known model , whether deterministic or probabilistic ( Corless & Leitmann , 1981 ; Thowsen , 1983 ; Huang et al. , 2011 ) . The Lyapunov function is a class of semi-positive definite functions L : S → R+ . The general idea of exploiting Lyapunov function is to ensure that the difference ( or derivative , if the system is in continuous time ) of Lyapunov function along the state trajectory is semi-negative definite , so that the state goes in the direction of decreasing the value of Lyapunov function and eventually converges to the origin or a sub-level set of Lyapunov function . Next , we give sufficient conditions for a system to be stable in mean cost in the following . Theorem 1 The stochastic system is stable in mean cost if there exists a function L : S → R+ and positive constants α1 , α2 and α3 , such that α1cπ ( s ) ≤ L ( s ) ≤ α2cπ ( s ) ( 1 ) Es∼µπ ( Es′∼PπL ( s′ ) − L ( s ) ) ≤ −α3Es∼µπcπ ( s ) ( 2 ) where µπ ( s ) , limN→∞ 1N ∑N t=0 P ( st = s|ρ , π , t ) is the sampling distribution . Due to space limitations , we will include the detailed proof in Appendix A. Eq . ( 2 ) is called the energy decreasing condition , i.e. , requiring the expectation of Lyapunov function to be decreasing between two consecutive instants . Eq . ( 1 ) is the constraint for Lyapunov function , though a rather broad range of parameterization is covered . The sum of quadratic polynomials , e.g. , L ( s ) = sTQs where Q is a positive definite matrix , are extensively used in the control theory . Such Lyapunov functions can be efficiently discovered by the semi-definite programming solvers and bring in limited conservatism for the control tasks where the cost are also of a quadratic form . In ( Richards et al. , 2018 ) , a neural network φθ ( · ) is designed to construct the Lyapunov function , L ( s ) = φθ ( s ) Tφθ ( s ) . As explored in ( Chow et al. , 2018 ) and ( Berkenkamp et al. , 2017 ) , the value function could be exploited as a Lyapunov function as well . Additionally , the sum of cost over a limited time horizon could also be employed as Lyapunov function , i.e. , L ( s ) = ∑t+N t Ecπ ( st ) , which is a valid Lyapunov candidate in model predictive control literature ( Mayne & Michalska , 1990 ; Mayne et al. , 2000 ) . The choice of Lyapunov function candidate plays an important role in learning a policy . Value function evaluates the infinite time horizon and thus offers a better performance in general , but is rather difficult to approximate because of significant variance and bias ( Schulman et al. , 2015 ) . On the other hand , the finite horizon sum of cost provides an explicit target for learning a Lyapunov function , thus inherently reduces the bias and enhances the learning process . However , as the model is unknown , predicting the future costs based on the current state and action inevitably introduces variance , which grows as the prediction horizon extends . In principle , for tasks with simple dynamics , the sum-of-cost choice enhances the convergence of learning and robustness of the trained policies , while for complicated systems the choice of value function generally produces better performance . In this paper , we use both value function and sum-of-cost over various horizons as Lyapunov function candidates in different tasks and compare their strength and weakness respectively . Now we would like to give the following two remarks . Remark 2 This remark is on Assumption 1 and sampling distribution µπ . If an MDP is ergodic then the existence of qπ is naturally assured , but all states have to be positive recurrent and aperiodic ( Papoulis & Pillai , 2002 ) . The existence of sampling distribution µπ ( s ) is guaranteed by the existence of qπ ( s ) . Since the sequence { P ( s|ρ , π , t ) , t ∈ Z+ } converges to qπ ( s ) as t approaches∞ , then by the Abelian theorem , the sequence { 1N ∑N t=0 P ( s|ρ , π , t ) , N ∈ Z+ } also converges and µπ ( s ) = qπ ( s ) . Thus we use µπ to approximate the qπ since the evaluation of qπ requires data to be sampled after infinite instants the episode begins . Even if the sampling period N < < ∞ , one can still assure that cπ converges to a neighborhood of zero , which is related to the initial state distribution and length of N , i.e. , EsN cπ ( sN ) ≤ α2α1+α3Eρcπ ( s0 ) − α3 α1+α3 ∑N−1 t=0 Estcπ ( st ) . Remark 3 This remark is on the connection to previous results concerning the stability of stochastic systems . It should be noted that the stability conditions of Markov chains have been reported in ( Shaikhet , 1997 ; Meyn & Tweedie , 2012 ) , however , of which the validation requires the full knowledge of the model , i.e. , the transition probability P ( s′|s , a ) . On the contrary , our approach solely depends on data to analyze the stability of the closed-loop system , which further enables the model-free learning algorithms with stability guarantee . However , the validation of stability through a sample-based approach theoretically requires tremendous , if not infinite , amount of samples to thoroughly estimate the distributions , which is the drawback of our approach . We would demonstrate empirically that the algorithm built upon this theorem is reliable though only limited sample is collected .

Summary: In this work the authors studied the model-free RL approach for learning a policy with stability guarantees. Leveraging the Lyapunov stochastic stability criterion, instead if minimizing the cumulative cost (plus a soft entropy), they propose optimizing an objective function with a specific Lyapunov critic, which is a specific critic function that satisfies the Lyapunov criterion to guarantee stability. They also show in several Cartpole, Mujoco, and Repressilator experiments that this approach is more robust to perturbations (such as sinusoids), where the agent are more robust to dynamic uncertainties and disturbances.