Reinforcement Learning for Control with Probabilistic Stability Guarantee

1 INTRODUCTION . Reinforcement learning ( RL ) has achieved superior performance on some complicated control tasks ( Kumar et al. , 2016 ; Xie et al. , 2019 ; Hwangbo et al. , 2019 ) for which the traditional control engineering methods can be hardly applicable ( Åström and Wittenmark , 1973 ; Morari and Zafiriou , 1989 ; Slotine et al. , 1991 ) . The dynamical system to be controlled is often highly stochastic and nonlinear which is typically modeled by Markov decision process ( MDP ) , i.e. , st+1 ∼ P ( st+1|st , at ) , ∀t ∈ Z+ ( 1 ) where s ∈ S ⊂ Rn denotes the state , a ∈ A ⊂ Rm denotes the action and P ( st+1|st , at ) is the transition probability function . An optimal controller can be learned from samples through “ trial and error ” by memorizing what has been experienced ( Kaelbling et al. , 1996 ; Bertsekas , 2019 ) . However , there is a major caveat that prevents the real-world application of learning methods for control engineering applications . Without using a mathematical model , the current sample-based RL methods can not guarantee the stability of the closed-loop system , which is the most important property of any control system as in control theory . The most useful and general approach for studying the stability of a dynamical system is Lyapunov ’ s method Lyapunov ( 1892 ) , which is dominant in control engineering Jiang and Jiang ( 2012 ) ; Lewis et al . ( 2012 ) ; Boukas and Liu ( 2000 ) . In Lyapunov ’ s method , a suitable “ energy-like ” Lyapunov function L ( s ) is selected and its derivative along the system trajectories is ensured to be negative semi-definite , i.e. , L ( st+1 ) − L ( st ) < 0 for all time instants and states , so that the state goes in the direction of decreasing the value of Lyapunov function and eventually converges to the origin or a sub-level set of the Lyapunov function . In the traditional control engineering methods , a mathematical model must be given , i.e. , the transition probability function in ( 1 ) is known . Thus the stability can be analyzed without the need to assess all possible trajectories . However , in learning methods , as the dynamic model is unknown , the “ energy decreasing ” condition has to be verified by trying out all possible consecutive data pairs in the state space , i.e. , to verify infinite inequalities L ( st+1 ) − L ( st ) < 0 . Obviously , the “ infinity ” requirement makes it impractical to directly exploit Lyapunov ’ s method in a model-free framework . In this paper , we show that the mean square stability of the system can be analyzed based on a finite number of samples without knowing the model of the system . The contributions of this paper are summarized as follows : 1 . Instead of verifying an infinite number of inequalities over the state space , it is possible to analyze the stability through a sampling-based method where only one inequality is needed . 2 . Instead of using infinite sample pairs { st+1 , st } , a finite-sample stability theorem is proposed to provide a probabilistic stability guarantee for the system , and the probability is an increasing function of the number M and length T of sampled trajectories and converging to 1 as M and T grow . 3 . As an independent interest , we also derive the policy gradient theorem for learning stabilizing policy with sample pairs and the corresponding algorithm . We further reveal that the classic REINFORCE algorithm ( Williams , 1992 ) is a special case of the proposed algorithm for the stabilization problem . We also conclude two takeaways for the paper : • Samples of a finite number M and length T of trajectories can be used for stability analysis with a certain probability . The probability is monotonically converging to 1 when M and T grow . • There is a lower bound on T and the probability is much more demanding for M than T . • The REINFORCE like algorithm can learn the controller and Lyapunov function simultane- ously . The paper is organized as follows : In Section 2 , related works are introduced . In Section 3 , the definition of mean-square stability ( MSS ) and the problem statement is given . In Section 4 , the samplebased MSS theorem is proposed . In Section 5 , we propose the probabilistic stability guarantee when only a finite number of samples are accessible and the probabilistic bound in a relation to the number and length of sampled trajectories is derived . In Section 6 , based on the stability theorems , the policy gradient is derived and a model-free RL algorithm ( L-REINFORCE ) is given . Finally , a simulated Cartpole stabilization task is considered to demonstrate the effectiveness of the proposed method.In Section 7 , the vanilla version of L-REINFORCE is tested on a simulated Cartpole stabilization task to demonstrate the effectiveness ; it is further incorporated with the maximum entropy framework to control the more high-dimensional and stochastic systems , including a legged robot , HalfCheetah , and the molecular synthetic biological gene regulatory networks ( GRN ) corrupted by the additive and multiplicative uniform noises . 2 RELATED WORKS . Lyapunov ’ s Method As a basic tool in control theory , the construction/learning of the Lyapunov function is not trivial and many works are devoted to this problem ( Noroozi et al. , 2008 ; Prokhorov , 1994 ; Serpen , 2005 ; Prokhorov and Feldkamp , 1999 ) . In Perkins and Barto ( 2002 ) , the RL agent controls the switch between designed controllers using Lyapunov domain knowledge so that any policy is safe and reliable . Petridis and Petridis ( 2006 ) proposes a straightforward approach to construct the Lyapunov functions for nonlinear systems using neural networks . Richards et al . ( 2018 ) proposes a learning-based approach for constructing Lyapunov neural networks with the maximized region of attraction . However , these approaches require the model of the system dynamics explicitly . Stability analysis in a model-free manner has not been addressed . In Berkenkamp et al . ( 2017 ) , local stability is analyzed by validating the “ energy decreasing ” condition on discretized points in the subset of state space with the help of a learned model , meaning that only a finite number of inequalities need to be checked . This approach is further extended by using a Noise Contrastive Prior Bayesian RNN in Gallieri et al . ( 2019 ) . Nevertheless , the discretization technique may become infeasible as the dimension and space of interest increases , limiting its application to rather simple and low-dimensional systems . Reinforcement Learning In model-free reinforcement learning ( RL ) , stability is rarely addressed due to the formidable challenge of analyzing and designing the closed-loop system dynamics by solely using samples Buşoniu et al . ( 2018 ) , and the associated stability theory in model-free RL remains as an open problem Buşoniu et al . ( 2018 ) ; Gorges ( 2017 ) . Recently , Lyapunov analysis is used in model-free RL to solve control problems with safety constraints Chow et al . ( 2018 ; 2019 ) . In Chow et al . ( 2018 ) , a Lyapunov-based approach for solving constrained Markov decision processes is proposed with a novel way of constructing the Lyapunov function through linear programming . In Chow et al . ( 2019 ) , the above results were further generalized to continuous control tasks . It should be noted that even though Lyapunov-based methods were adopted in these results , neither of them addressed the stability of the system . In Postoyan et al . ( 2017 ) , an initial result is proposed for the stability analysis of deterministic nonlinear systems with optimal controller for infinite-horizon discounted cost , based on the assumption that discount is sufficiently close to 1 . However , in practice , it is rather difficult to guarantee the optimality of the learned policy unless certain assumptions on the system dynamics are made Murray et al . ( 2003 ) ; Abu-Khalaf and Lewis ( 2005 ) ; Jiang and Jiang ( 2015 ) . Furthermore , the exploitation of multi-layer neural networks as function approximations Mnih et al . ( 2015 ) ; Lillicrap et al . ( 2015 ) only adds to the impracticality of this requirement . Given certain information on the model , Adaptive dynamic programming ( ADP ) can guarantee convergence to the optimal solution , and thus stability is naturally ensured Balakrishnan et al . ( 2008 ) . For nonlinear systems with input-affine structure , model-free ADP algorithms can guarantee the stability of the closed-loop system Murray et al . ( 2003 ) ; Abu-Khalaf and Lewis ( 2005 ) ; Shih et al . ( 2007 ) ; Jiang and Jiang ( 2015 ) ; Deptula et al . ( 2018 ) . This paper steps beyond the scope of controlaffine systems and are devoted to learning a controller with a stability guarantee for the general stochastic nonlinear system . To the best of the author ’ s knowledge , the finite sample-based approach for the stability analysis of stochastic nonlinear systems considered in this paper is still missing . For the model-based approaches , promising results on stability analysis are reported but generally based on certain model assumptions . Model predictive control ( MPC ) has long been studying the issue of optimal control of various dynamical systems without violating state and action constraints , and Lyapunov stability is naturally guaranteed ( Mayne and Michalska , 1990 ; Michalska and Mayne , 1993 ; Mayne et al. , 2000 ) . Favorable as it may seem , the nice properties above are built upon the accurate and concise modeling of the dynamics , which narrows its scope to certain fields . In Ostafew et al . ( 2014 ) , a learning-based nonlinear MPC algorithm is proposed to learn the disturbance model online and improve the tracking performance of field robots , but first , a priori model is required . Aswani et al . ( 2013 ) proposed a new learning-based MPC scheme that can provide deterministic guarantees on robustness while performance is improved by identifying a richer model . However , it is limited to the case that a linear model with known uncertainty bound is available . Other results concerning learning-based MPC are referred to Aswani et al . ( 2011 ) ; Bouffard et al . ( 2012 ) ; Di Cairano et al . ( 2013 ) . In Bobiti ( 2017 ) ; Bobiti and Lazar ( 2018 ) , a sampling-based approach for stability analysis and domain of attraction estimation is proposed for deterministic nonlinear systems . The reliability of the estimation is addressed with a probabilistic bound on the number of samples , however , based on the assumption that all the samples are independently distributed . This infers that given multiple state trajectories , only the first-step data are applicable for the stability analysis , which is inefficient in a model-free framework and will be improved in this paper . Nevertheless , the aforementioned approach can be favorable in a model-based setup ( Gallieri et al. , 2019 ) , given that 1-step predictions can be performed in parallel . It should also be noted that this paper is to address the stability analysis and control of stochastic systems , while the results above are focused on the deterministic nonlinear systems .

Summary: **General overview:** The paper studies guaranteeing the closed-loop stability of a Markov decision process (MDP) using a given a policy, based on a finite number of trajectories each containing finite number of steps. Both the state and the action spaces are assumed to be subsets of finite dimensional Euclidean spaces. Particularly, the concept of mean square stability (MSS) is used which is guaranteed based on the properties of certain Lyapunov functions. Theoretical results on probabilistic MSS guarantees are provided based on finite samples. A variant of the standard REINFORCE policy gradient method is also presented which searches for a policy having MSS guarantees. Finally, numerical experiments on a simulated cart-pole problem comparing the suggested L-REINFORCE method with the soft actor-critic (SAC) off-policy RL algorithm are shown.