Lyapunov-based Safe Policy Optimization

1 INTRODUCTION . In many real-world reinforcement learning ( RL ) problems , the agent must satisfy a number of constraints while minimizing its expected cumulative cost . In particular , these constraints might be related to safety , i.e. , the policy learned by the agent should not take it to certain undesirable parts of the state and/or action space . In some problems , it is not only important that the policy learned by the agent ( that will be deployed ) be safe ( Amodei et al. , 2016 ) , but also crucial that all the policies generated during training ( with which the agent interacts with the environment to collect samples ) be safe ( Achiam et al. , 2017 ) . For example , a robot should always avoid taking actions that irrevocably harm its hardware . We often formulate the agent ’ s interaction with the environment in constrained sequential decision-making problems as a constrained Markov decision process ( CMDP ) . CMDPs are extensions of MDPs in which , in addition to the original cost function , there exists a constraint cost function , whose expected cumulative value should remain bounded . The additional constraint cost function gives more flexibility to CMDPs in modeling problems with trajectorybased constraints compared to approaches that artificially re-cast the cost of MDPs to enforce the constraints ( Regan & Boutilier , 2009 ) . Under the CMDP framework , we consider a policy to be safe if it satisfies the ( expected ) cumulative cost constraints . A common approach to solve CMDPs is to use the Lagrangian method ( Altman , 1998 ; Geibel & Wysotzki , 2005 ) that augments the original objective function with a penalty on constraint violation and computes the saddle-point of the constrained policy optimization via primal-dual method ( Chow et al. , 2017 ) . Although safety is ensured when the policy converges asymptotically , a major drawback of this approach is that it makes no guarantee with regards to the safety of the policies generated during training . To address this issue ( safety during training ) , heuristic algorithms for policy search with safety constraints have been proposed ( e.g. , Uchibe & Doya 2007 ) . While some of them work reasonably well in practice , none is theoretically grounded to generate constraint-satisfying policies . Achiam et al . ( 2017 ) recently proposed the constrained policy optimization ( CPO ) method that extends the trust-region policy optimization ( TRPO ) algorithm ( Schulman et al. , 2015a ) to handle CMDP constraints . Although there is no theoretical guarantee that the policies generated by CPO are all safe , the empirical results reported in the paper are promising in terms scalability , performance , and constraint satisfaction , both during training and after convergence . Moreover , CPO is based on a principled approach to generate constraint-satisfying policies , and not just on heuristics . However , the CPO methodology is closely connected to TRPO . While adopting this methodology to PPO for constrained optimization is straight-forward ( this is exactly how we derive our SPPO algorithm in this paper ) , it is not clear how it can be combined with algorithms that do not belong to the family of proximal PG algorithms ( PG algorithms that are regularized with relative entropy ) , such as DDPG . Another recent approach to solve CMDPs , while remaining safe during training , is by Chow et al . ( 2018 ) . This work is based on the notion of Lyapunov function that has a long history in control theory to analyze the stability of dynamical systems ( Khalil , 1996 ; Neely , 2010 ) . Using Lyapunov functions in RL was first studied by Perkins & Barto ( 2002 ) , where they were used to guarantee closed-loop stability of an agent . In a recent work , Berkenkamp et al . ( 2017 ) used Lyapunov functions to guarantee that a model-based RL agent can be brought back to a “ region of attraction ” during exploration . Using the theoretical underpinnings of the Lyapunov approach , Chow et al . ( 2018 ) proposed two dynamic programming ( DP ) algorithms to solve a CMDP and provided theoretical analyses for the feasibility and performance of the resulted policies . The proposed algorithms correspond to the two celebrated DP algorithms : policy and value iteration . They then extended their algorithms to learning ( the scenario in which the MDP model is unknown ) and proposed RL algorithms that correspond to approximate policy and value iteration . However , since their algorithms are all value-function-based , applying them to continuous action problems is not straightforward . In this paper , we extend the Lyapunov-based approach to solving CMDPs of Chow et al . ( 2018 ) to continuous action problems that play an important role in control and robotics . Our contributions can be summarized as follows : 1 ) We formulate the problem of safe RL as a CMDP and propose a Lyapunov-function-based policy optimization framework that can handle continuous action CMDPs . 2 ) By leveraging the theoretical underpinnings of the Lyapunov-based approach to CMDPs in Chow et al . ( 2018 ) , we present two classes of safe policy optimization algorithms that can work with any standard policy gradient algorithm such as deep deterministic policy gradient ( DDPG ) ( Lillicrap et al. , 2015 ) and proximal policy optimization ( PPO ) ( Schulman et al. , 2017 ) . The first class of our algorithms is based on constrained optimization w.r.t . policy parameter ( similar to what is done in constrained policy optimization ( CPO ) ) . The second class hinges on the concept of a safety layer introduced by Dalal et al . ( 2018 ) and transforms the constrained policy optimization problem into an unconstrained one , by integrating the Lyapunov constraints into the policy network via safety-layer augmentation . 3 ) We evaluate our algorithms and compare them to two baselines , CPO ( Achiam et al. , 2017 ) and the Lagrangian method , on several robot locomotion tasks , in which the agent must satisfy certain safety constraints while minimizing its expected cumulative cost . Our results show that our algorithms outperform the baselines in terms of balancing the performance and constraint satisfaction at every policy update . 2 PRELIMINARIES . We consider the RL problem in which the agent ’ s interaction with the environment is modeled as a Markov decision process ( MDP ) . A MDP is a tuple ( X , A , γ , c , P , x0 ) , where X is the state space ; A is the action space ; γ ∈ [ 0 , 1 ) is the discounting factor ; c ( x , a ) ∈ [ 0 , Cmax ] is the immediate cost function ( negative reward ) ; P ( ·|x , a ) is the transition probability distribution ; and x0 ∈ X is the initial state . Our results easily generalize to random initial states and random costs , but for simplicity we will focus on the case of deterministic initial state and immediate cost . In a more general setting where cumulative constraints are taken into account , we define a constrained Markov decision process ( CMDP ) , which extends the MDP model by introducing additional costs and associated constraints . A CMDP is defined by ( X , A , γ , c , d , P , x0 , d0 ) , where the components X , A , γ , c , P , x0 are the same as in the unconstrained MDP ; d ( x ) ∈ [ 0 , Dmax ] is the immediate constraint cost ; and d0 ∈ R≥0 is an upper-bound on the expected cumulative ( through time ) constraint cost . To formalize the optimization problem associated with CMDPs , let ∆ be the set of Markov stationary policies , i.e. , ∆ ( x ) = { π ( ·|x ) : X → R≥0s : ∑ a π ( a|x ) = 1 } for any state x ∈ X . For notational convenience , at each state x ∈ X , we define the generic Bellman operator w.r.t . policy π ∈ ∆ and generic cost function h as Tπ , h [ V ] ( x ) = ∑ a π ( a|x ) [ h ( x , a ) +γ ∑ x′∈XP ( x ′|x , a ) V ( x′ ) ] . Given a policy π ∈ ∆ and an initial state x0 , the expected cumulative cost is defined as Cπ ( x0 ) : = E [ ∑∞ t=0 γ tc ( xt , at ) | x0 , π ] , and the safety constraint is defined asDπ ( x0 ) ≤ d0 , whereDπ ( x0 ) : = E [ ∑∞ t=0 γ td ( xt ) | x0 , π ] is the safety constraint function ( expected cumulative constraint cost ) . The goal in CMDPs is to solve the constrained optimization problem π∗ ∈ min π∈∆ { Cπ ( x0 ) : Dπ ( x0 ) ≤ d0 } . ( 1 ) Using the discounting property of a CMDP , Theorem 3.1 in Altman ( 1999 ) shows that if the feasibility set is non-empty , then there exists an optimal policy in the class of stationary Markovian policies ∆ . To motivate the CMDP formulation , we refer the reader to Chow et al . ( 2018 ) , which presents two real-world examples in modeling safety using ( i ) the reachability constraint , and ( ii ) the constraint that limits the agent ’ s visits to undesirable states . 3 A LYAPUNOV APPROACH FOR SOLVING CMDPS . In this section , we revisit the Lyapunov approach to solving CMDPs proposed by Chow et al . ( 2018 ) and report the mathematical results that are important in developing our safe policy optimization algorithms . To start , without loss of generality , we assume that we have access to a baseline feasible policy of Equation 1 , πB ; i.e . πB satisfies DπB ( x0 ) ≤ d0 . We define a set of Lyapunov functions w.r.t . initial state x0 ∈ X and constraint threshold d0 as LπB ( x0 , d0 ) = { L : X → R≥0 : TπB , d [ L ] ( x ) ≤ L ( x ) , ∀x ∈ X ; L ( x0 ) ≤ d0 } , and call the constraints in this feasibility set Lyapunov constraints . For any arbitrary Lyapunov function L ∈ LπB ( x0 , d0 ) , we denote by FL ( x ) = { π ( ·|x ) ∈ ∆ : Tπ , d [ L ] ( x ) ≤L ( x ) } the set of L-induced Markov stationary policies . Since Tπ , d is a contraction mapping ( Bertsekas , 2005 ) , any L-induced policy π has the property Dπ ( x ) = limk→∞ T kπ , d [ L ] ( x ) ≤ L ( x ) , ∀x ∈ X . Together with the property that L ( x0 ) ≤ d0 , they imply that any L-induced policy is a feasible policy of Equation 1 . However , in general , the set FL ( x ) does not necessarily contain an optimal policy of Equation 1 , and thus it is necessary to design a Lyapunov function ( w.r.t . a baseline policy πB ) that provides this guarantee . In other words , the main goal is to construct a Lyapunov function L ∈ LπB ( x0 , d0 ) such that L ( x ) ≥ Tπ∗ , d [ L ] ( x ) , L ( x0 ) ≤ d0 . ( 2 ) Chow et al . ( 2018 ) show in their Theorem 1 that 1 ) without loss of optimality , the Lyapunov function can be expressed as L ( x ) : = E [ ∑∞ t=0 γ t ( d ( xt ) + ( xt ) ) | πB , x ] , where ( x ) ≥ 0 is some auxiliary constraint cost uniformly upper-bounded by ∗ ( x ) : = 2DmaxDTV ( π∗||πB ) ( x ) / ( 1 − γ ) , and 2 ) if the baseline policy πB satisfies the condition maxx∈X ∗ ( x ) ≤ Dmax · min { ( 1− γ ) ( d0 −DπB ( x0 ) ) /Dmax , Dmax − ( 1− γ ) D/Dmax + ( 1− γ ) D } , where D = maxx∈X maxπ Dπ ( x ) is the maximum constraint cost , then the Lyapunov function candidate L ∗ also satisfies the properties of Equation 2 , and thus , its induced feasible policy set FL ∗ contains an optimal policy . Furthermore , suppose that the distance between the baseline and optimal policies can be estimated effectively . Using the set of L ∗ -induced feasible policies and noting that the safe Bellman operator T [ V ] ( x ) = minπ∈FL ∗ ( x ) Tπ , c [ V ] ( x ) is monotonic and contractive , one can show that T [ V ] ( x ) = V ( x ) , ∀x ∈ X has a unique fixed point V ∗ , such that V ∗ ( x0 ) is a solution of Equation 1 , and an optimal policy can be constructed via greedification , i.e. , π∗ ( ·|x ) ∈ arg minπ∈FL ∗ ( x ) Tπ , c [ V ∗ ] ( x ) . This shows that under the above assumption , Equation 1 can be solved using standard dynamic programming ( DP ) algorithms . While this result connects CMDP with Bellman ’ s principle of optimality , verifying whether πB satisfies this assumption is challenging when a good estimate of DTV ( π∗||πB ) is not available . To address this issue , Chow et al . ( 2018 ) propose to approximate ∗ with an auxiliary constraint cost ̃ , which is the largest auxiliary cost satisfying the Lyapunov condition L̃ ( x ) ≥ TπB , d [ L̃ ] ( x ) , ∀x ∈ X and the safety condition L̃ ( x0 ) ≤ d0 . The intuition here is that the larger ̃ , the larger the set of policies FL̃ . Thus , by choosing the largest such auxiliary cost , we hope to have a better chance of including the optimal policy π∗ in the set of feasible policies . Specifically , ̃ is computed by solving the following linear programming ( LP ) problem : ̃ ∈arg max : X→R≥0 { ∑ x∈X ( x ) : d0 −DπB ( x0 ) ≥ 1 ( x0 ) > ( I − γ { P ( x′|x , πB ) } x , x′∈X ) −1 } , ( 3 ) where 1 ( x0 ) represents a one-hot vector in which the non-zero element is located at x = x0 . When πB is a feasible policy , this problem has a non-empty solution . Furthermore , according to the derivations in Chow et al . ( 2018 ) , the maximizer of Equation 3 is an indicator function of the form ̃ ( x ) = ( d0 −DπB ( x0 ) ) · 1 { x = x } /E [ ∑∞ t=0 γ t1 { xt = x } | x0 , πB ] ≥ 0 , where x ∈ arg minx∈X E [ ∑∞ t=0 γ t1 { xt = x } | x0 , πB ] . They also show that by further restricting ̃ ( x ) to be a constant function , the maximizer is given by ̃ ( x ) = ( 1−γ ) · ( d0−DπB ( x0 ) ) , ∀x ∈ X . Using the construction of the Lyapunov function L̃ , Chow et al . ( 2018 ) propose the safe policy iteration ( SPI ) algorithm ( see Algorithm 1 in Appendix A ) in which the Lyapunov function is updated via bootstrapping , i.e. , at each iteration L̃ is recomputed using Equation 3 w.r.t . the current baseline policy . This algorithm has the following properties : 1 ) Consistent Feasibility , i.e. , if the current policy πk is feasible , then πk+1 is also feasible ; 2 ) Monotonic Policy Improvement , i.e. , Cπk+1 ( x ) ≤ Cπk ( x ) for any x ∈ X ; and 3 ) Asymptotic Convergence . Despite all these nice properties , SPI is still a valuefunction-based algorithm , and thus it is not straightforward to use it in continuous action problems . The main reason is that the greedification step becomes an optimization problem over the continuous set of actions that is not necessarily easy to solve . In Section 4 , we show how we use SPI and its nice properties to develop safe policy optimization algorithms that can handle continuous action problems . Our algorithms can be thought as combinations of DDPG or PPO ( or any other on-policy or off-policy policy optimization algorithm ) with a SPI-inspired critic that evaluates the policy and computes its corresponding Lyapunov function . The computed Lyapunov function is then used to guarantee safe policy update , i.e. , the new policy is selected from a restricted set of safe policies defined by the Lyapunov function of the current policy .

Summary: In this paper, authors compare different ways to enforce stability constraints on trajectories in dynamic RL problems. It builds on a recent approach by Achiam et al on Constrained Policy Optimization (oft- mentioned "CPO") and an accepted NIPS paper by Chow which introduces Lyapunov constraints as an alternative method. While this approach is reasonable indeed, the novelty of the approach is questionable, not only in light of recent papers but older literature: inference of Markov Decision Processes under constraints is referred to and has been known a long time. Furthermore, the actual tasks chosen are quite simple and do not present complex instabilities. Also, actually creating a Lyapunov function and weighing the relative magnitude of its second derivative (steep/shallow) is not trivial and must influence the behavior of the optimizer. Also worth mentioning that complex nonlinearities might imply that instabilities in the observed dynamics are not seen and learned unless the space exploration is conservative. That is, comparison of CPO and Lagrangian constraint based RL with Lyapunov based method proposed depends on a lot of factors (such as those just mentioned) that are not systematically explored by the paper.