Learning Safe Policies with Cost-sensitive Advantage Estimation

1 INTRODUCTION . In recent years , Reinforcement Learning ( RL ) has achieved remarkable success in learning skillful AI agents in various applications ranging from robot locomotion ( Schulman et al. , 2015a ; Duan et al. , 2016 ; Schulman et al. , 2015c ) , video games ( Mnih et al. , 2015 ) and the game of Go ( Silver et al. , 2016 ; 2017 ) . These agents are either trained in simulation or in risk-free environments , and the deployed RL algorithms can focus on maximizing the cumulative return by exploring the environment arbitrarily . However , this is barely workable for real-world RL problems where the safety of the agent is important . For example , a navigating robot can not take the action of crashing into a front obstacle even if the potential return on reaching the target faster is higher . Actually , in reality , some states or actions might be unsafe and harmful to the system , and the agent should learn to avoid them in deployment when performing certain tasks . Conventional RL algorithms do not particularly consider such safety-constrained environments , which limits their practical application . Recently , Safe Reinforcement Learning ( Garcıa & Fernández , 2015 ; Mihatsch & Neuneier , 2002 ; Altman , 1999 ) has been proposed and drawn increasing attention . Existing safe RL algorithms generally fall into two categories based on whether or not the agents are required to always stay safe during learning and exploration . The algorithms with exploration safety ( Dalal et al. , 2018 ; Pecka & Svoboda , 2014 ) insist that safety constraints never be violated even during learning , and thus they usually require certain prior knowledge of the environment to be available , e.g. , in the form of human demonstrations . Comparatively , deployment safety ( Achiam et al. , 2017 ; Chow et al. , 2018 ) RL algorithms train the agents from interaction with the environment and allow safety constraints violation during learning to some extent . This is reasonable since whether a state is safe will not be clear until the agent visits that state . Since human demonstrations are too difficult or expensive to collect in some cases and may not cover the whole state space , we focus on deployment safety in this work . RL problems with deployment safety are typically formulated as Constrained Markov Decision Process ( CMDP ) ( Altman , 1999 ) that extends MDP by requiring the agent to satisfy cumulative cost constraints in expectation in the meanwhile of maximizing the expected return . Leveraging the success of recent deep learning powered policy optimization methods ( Schulman et al. , 2015b ) , Constrained Policy Optimization ( CPO ) ( Achiam et al. , 2017 ) makes the first attempt on highdimensional control tasks in continuous CMDPs . However , CPO only considers the total cost of a trajectory of a sequence of state-action pairs during policy optimization . It does not differentiate the safe state-action pairs from the unsafe ones in the trajectories . Due to such incapability of exploiting the intrinsic structure of environments and trajectories , CPO sacrifices too much on the expected return for learning the safety policy . In this work , we propose Cost-sensitive Advantage Estimation ( CSAE ) which generalizes the conventional advantage estimation for safe RL problems by differentiating safe and unsafe states , based on the cost information returned by the environment during training . CSAE depresses the advantage value of unsafe state-action pairs but controls effects upon their adjacent safe state-actions in the trajectories . Thus , the learned policy can maximally gain rewards from the safe states . Based on CSAE , we develop a new safe RL algorithm with proved monotonic policy performance improvement in terms of both safety and return from safe states , showing superiority over other safe RL algorithms . Moreover , to further enhance the agent ’ s ability of enforcing safety constraints , we propose Worst-case Constrained Markov Decision Process ( WCMDP ) , an extension of CMDP by constraining the cumulative cost in worst cases through the Conditional Value-at-Risk ( Tamar et al. , 2015 ) , instead of that in expectation . This augmentation makes the learned policy not only safer but also better , both experimentally and theoretically . With CSAE and WCMDP , we develop a new safe RL algorithm by relating them to trust region methods . We conduct extensive experiments to evaluate our algorithm on several constrained robot locomotion tasks based on Mujoco ( Todorov et al. , 2012 ) , and compare it with well-established baselines . The results demonstrate that the agent trained by our algorithm can collect a higher reward , while satisfying the safety constraints with less cost . 2 RELATED WORK . Safe Reinforcement Learning has drawn growing attention . There are various definitions of ‘ safety ’ in RL ( Garcıa & Fernández , 2015 ; Pecka & Svoboda , 2014 ) , e.g. , the variance of return ( Heger , 1994 ; Gaskett , 2003 ) , fatal transitions ( Hans et al. , 2008 ) and unknown states ( Garcıa et al. , 2013 ) . In this paper , we focus on the RL problems with trajectory-based safety cost , under the constrained MDP ( CMDP ) framework . Through Lagrangian method , Geibel & Wysotzki ( 2005 ) propose to convert CMDP into an unconstrained problem to maximize the expected return with a cost penalty . Though such a problem can be easily solved with well-designed RL algorithms , e.g . ( Schulman et al. , 2015b ; 2017 ) , the trade-off between return and cost is manually balanced with a fixed Lagrange multiplier , which can not guarantee safety through learning . To address this , inspired by trust region methods ( Schulman et al. , 2015b ) , Constrained Policy Optimization ( Achiam et al. , 2017 ) ( CPO ) establishes linear approximation to the safety constraint and solves the corresponding optimization problem in the dual form . Compared with previous CMDP algorithms , CPO scales well to highdimensional continuous state-action spaces . However , CPO does not distinguish the safe states from the unsafe ones in the training process , limiting its performance in the return . Besides developing various optimization algorithms , some recent works also explore other approaches to enhance the safety constraints , e.g. , adopting the Conditional Value-at-Risk ( CVaR ) of the cumulative cost as the safety constraint ( Tamar et al. , 2015 ) . Along this direction , Tamar et al . ( 2015 ) develop a gradient estimator through sampling to optimize CVaR with gradient descent . Prashanth ( 2014 ) further applies this estimator to CVaR-Constrained MDP to solve the stochastic shortest path ( SSP ) problem . Our work considers a similar framework to CPO ( Achiam et al. , 2017 ) , but it treats states differently by extending Generalized Advantage Estimation ( Schulman et al. , 2015c ) to be safety-sensitive . Our proposed CSAE can boost the policy performance in terms of the return while ensuring the safety property . Moreover , our algorithm with WCMDP is safer than CPO in terms of constraint violation ratio during learning . There are also some non-CMDP based algorithms for safe RL that are not in the scope of this work . In ( Dalal et al. , 2018 ) , a linear safety-signal model is built to estimate per-step cost from state-action pairs and rectify the action into a safe one . However , this method requires a pre-collected dataset to fit the linear cost estimation model , which limits its application . Similarly , Cheng et al . ( 2019 ) augment the model-free controller to enforce safety per step by designing a modle-based controller with control barrier functions ( CBFs ) . Some works introduce Lyapunov functions to build safe RL algorithms . For example , Berkenkamp et al . ( 2017 ) apply Lyapunov functions for safely recovering from exploratory actions , while Chow et al . ( 2018 ) construct Lyapunov functions that explicitly model constraints . 3 PRELIMINARIES . A standard Markov Decision Process ( MDP ) ( Sutton et al. , 1998 ) is defined with a tuple ( S , A , P , R , γ , µ ) , where S andA denote the set of states and actions respectively , P : S×A×S → [ 0 , 1 ] is the transition dynamics modeling the probability of transferring from state s to s′ after taking action a , R ( s , a , s′ ) represents the reward function during this transition , γ ∈ [ 0 , 1 ] is the discount factor and µ : S 7→ [ 0 , 1 ] denotes the starting state distribution . An MDP agent is usually equipped with a policy π ( a|s ) , which denotes the probability distribution over actions a given a state s. The performance of a policy π is measured with the expected discounted total reward J ( π ) = Eτ∼π , s0∼µ [ ∑∞ t=0 γ tR ( st , at , st+1 ) ] , where τ = ( s0 , a0 , s1 , . . . ) is a trajectory generated by following policy π. RL algorithms for MDPs try to find the policy π∗ that achieves the highest reward , i.e. , π∗ = arg maxπ J ( π ) . They commonly use the value function Vπ ( s ) = Eτ∼π [ ∑∞ t=0 γ tR ( st , at , st+1 ) |s0 = s ] , the action value function Qπ ( s , a ) = Eτ∼π [ ∑∞ t=0 γ tR ( st , at , st+1 ) |s0 = s , a0 = a ] and the advantage function Aπ ( s , a ) = Qπ ( s , a ) − Vπ ( s ) . The discounted future state distribution will also be useful , which is defined as dπ ( s ) = ( 1− γ ) ∑ t=0 γ tP ( st = s|π ) . Constrained Markov Decision Process ( CMDP ) ( Altman , 1999 ) extends MDP to environments with safety cost that could harm the agent when undesired actions are taken . As various safety costs may exist in a single CMDP , we relate them with m cost functions { C1 ( s , a , s′ ) , . . . , Cm ( s , a , s′ ) } , each of which denotes the cost an agent receives for each transition ( s , a , s′ ) ( similar to reward functions ) . Let Ci ( τ ) = ∑∞ t=0 γ tCi ( st , at , st+1 ) denote the cumulative cost along a trajectory τ generated from policy π . We consider a trajectory-based cost constraint in CMDP , which limits the cumulative cost in expectation JCi = Eτ∼π , s0∼µ [ Ci ( τ ) ] with value di . Then safe RL aims to learn the policy π under CMDP by solving the following problem , π∗ = arg max J ( π ) , s.t . JCi = Eτ∼π , s0∼µ [ Ci ( τ ) ] ≤ di , i = 1 , . . . , m. ( 1 ) Safe RL algorithms search for the policy π∗ that achieves the maximal cumulative reward and meanwhile does not violate the imposed safety constraints on the costs . In the following , analogous to the definition of value functions ( i.e. , Vπ , Qπ and Aπ ) , we use V Ciπ , Q Ci π and A Ci π to denote the cost-value functions w.r.t . cost function Ci .

Summary: In this paper, the authors proposed a new constrained policy optimization algorithm and a worst-case version of the constrained MDP framework. Tho proposed constrained policy optimization algorithm is based on CPO, and a novel advantage function (CSAE) based on the concept of a "safe" state. Experiments in control simulation tasks are provided.