Learning Safe Policies with Cost-sensitive Advantage Estimation

1 INTRODUCTION . In recent years , Reinforcement Learning ( RL ) has achieved remarkable success in learning skillful AI agents in various applications ranging from robot locomotion ( Schulman et al. , 2015a ; Duan et al. , 2016 ; Schulman et al. , 2015c ) , video games ( Mnih et al. , 2015 ) and the game of Go ( Silver et al. , 2016 ; 2017 ) . These agents are either trained in simulation or in risk-free environments , and the deployed RL algorithms can focus on maximizing the cumulative return by exploring the environment arbitrarily . However , this is barely workable for real-world RL problems where the safety of the agent is important . For example , a navigating robot can not take the action of crashing into a front obstacle even if the potential return on reaching the target faster is higher . Actually , in reality , some states or actions might be unsafe and harmful to the system , and the agent should learn to avoid them in deployment when performing certain tasks . Conventional RL algorithms do not particularly consider such safety-constrained environments , which limits their practical application . Recently , Safe Reinforcement Learning ( Garcıa & Fernández , 2015 ; Mihatsch & Neuneier , 2002 ; Altman , 1999 ) has been proposed and drawn increasing attention . Existing safe RL algorithms generally fall into two categories based on whether or not the agents are required to always stay safe during learning and exploration . The algorithms with exploration safety ( Dalal et al. , 2018 ; Pecka & Svoboda , 2014 ) insist that safety constraints never be violated even during learning , and thus they usually require certain prior knowledge of the environment to be available , e.g. , in the form of human demonstrations . Comparatively , deployment safety ( Achiam et al. , 2017 ; Chow et al. , 2018 ) RL algorithms train the agents from interaction with the environment and allow safety constraints violation during learning to some extent . This is reasonable since whether a state is safe will not be clear until the agent visits that state . Since human demonstrations are too difficult or expensive to collect in some cases and may not cover the whole state space , we focus on deployment safety in this work . RL problems with deployment safety are typically formulated as Constrained Markov Decision Process ( CMDP ) ( Altman , 1999 ) that extends MDP by requiring the agent to satisfy cumulative cost constraints in expectation in the meanwhile of maximizing the expected return . Leveraging the success of recent deep learning powered policy optimization methods ( Schulman et al. , 2015b ) , Constrained Policy Optimization ( CPO ) ( Achiam et al. , 2017 ) makes the first attempt on highdimensional control tasks in continuous CMDPs . However , CPO only considers the total cost of a trajectory of a sequence of state-action pairs during policy optimization . It does not differentiate the safe state-action pairs from the unsafe ones in the trajectories . Due to such incapability of exploiting the intrinsic structure of environments and trajectories , CPO sacrifices too much on the expected return for learning the safety policy . In this work , we propose Cost-sensitive Advantage Estimation ( CSAE ) which generalizes the conventional advantage estimation for safe RL problems by differentiating safe and unsafe states , based on the cost information returned by the environment during training . CSAE depresses the advantage value of unsafe state-action pairs but controls effects upon their adjacent safe state-actions in the trajectories . Thus , the learned policy can maximally gain rewards from the safe states . Based on CSAE , we develop a new safe RL algorithm with proved monotonic policy performance improvement in terms of both safety and return from safe states , showing superiority over other safe RL algorithms . Moreover , to further enhance the agent ’ s ability of enforcing safety constraints , we propose Worst-case Constrained Markov Decision Process ( WCMDP ) , an extension of CMDP by constraining the cumulative cost in worst cases through the Conditional Value-at-Risk ( Tamar et al. , 2015 ) , instead of that in expectation . This augmentation makes the learned policy not only safer but also better , both experimentally and theoretically . With CSAE and WCMDP , we develop a new safe RL algorithm by relating them to trust region methods . We conduct extensive experiments to evaluate our algorithm on several constrained robot locomotion tasks based on Mujoco ( Todorov et al. , 2012 ) , and compare it with well-established baselines . The results demonstrate that the agent trained by our algorithm can collect a higher reward , while satisfying the safety constraints with less cost . 2 RELATED WORK . Safe Reinforcement Learning has drawn growing attention . There are various definitions of ‘ safety ’ in RL ( Garcıa & Fernández , 2015 ; Pecka & Svoboda , 2014 ) , e.g. , the variance of return ( Heger , 1994 ; Gaskett , 2003 ) , fatal transitions ( Hans et al. , 2008 ) and unknown states ( Garcıa et al. , 2013 ) . In this paper , we focus on the RL problems with trajectory-based safety cost , under the constrained MDP ( CMDP ) framework . Through Lagrangian method , Geibel & Wysotzki ( 2005 ) propose to convert CMDP into an unconstrained problem to maximize the expected return with a cost penalty . Though such a problem can be easily solved with well-designed RL algorithms , e.g . ( Schulman et al. , 2015b ; 2017 ) , the trade-off between return and cost is manually balanced with a fixed Lagrange multiplier , which can not guarantee safety through learning . To address this , inspired by trust region methods ( Schulman et al. , 2015b ) , Constrained Policy Optimization ( Achiam et al. , 2017 ) ( CPO ) establishes linear approximation to the safety constraint and solves the corresponding optimization problem in the dual form . Compared with previous CMDP algorithms , CPO scales well to highdimensional continuous state-action spaces . However , CPO does not distinguish the safe states from the unsafe ones in the training process , limiting its performance in the return . Besides developing various optimization algorithms , some recent works also explore other approaches to enhance the safety constraints , e.g. , adopting the Conditional Value-at-Risk ( CVaR ) of the cumulative cost as the safety constraint ( Tamar et al. , 2015 ) . Along this direction , Tamar et al . ( 2015 ) develop a gradient estimator through sampling to optimize CVaR with gradient descent . Prashanth ( 2014 ) further applies this estimator to CVaR-Constrained MDP to solve the stochastic shortest path ( SSP ) problem . Our work considers a similar framework to CPO ( Achiam et al. , 2017 ) , but it treats states differently by extending Generalized Advantage Estimation ( Schulman et al. , 2015c ) to be safety-sensitive . Our proposed CSAE can boost the policy performance in terms of the return while ensuring the safety property . Moreover , our algorithm with WCMDP is safer than CPO in terms of constraint violation ratio during learning . There are also some non-CMDP based algorithms for safe RL that are not in the scope of this work . In ( Dalal et al. , 2018 ) , a linear safety-signal model is built to estimate per-step cost from state-action pairs and rectify the action into a safe one . However , this method requires a pre-collected dataset to fit the linear cost estimation model , which limits its application . Similarly , Cheng et al . ( 2019 ) augment the model-free controller to enforce safety per step by designing a modle-based controller with control barrier functions ( CBFs ) . Some works introduce Lyapunov functions to build safe RL algorithms . For example , Berkenkamp et al . ( 2017 ) apply Lyapunov functions for safely recovering from exploratory actions , while Chow et al . ( 2018 ) construct Lyapunov functions that explicitly model constraints . 3 PRELIMINARIES . A standard Markov Decision Process ( MDP ) ( Sutton et al. , 1998 ) is defined with a tuple ( S , A , P , R , γ , µ ) , where S andA denote the set of states and actions respectively , P : S×A×S → [ 0 , 1 ] is the transition dynamics modeling the probability of transferring from state s to s′ after taking action a , R ( s , a , s′ ) represents the reward function during this transition , γ ∈ [ 0 , 1 ] is the discount factor and µ : S 7→ [ 0 , 1 ] denotes the starting state distribution . An MDP agent is usually equipped with a policy π ( a|s ) , which denotes the probability distribution over actions a given a state s. The performance of a policy π is measured with the expected discounted total reward J ( π ) = Eτ∼π , s0∼µ [ ∑∞ t=0 γ tR ( st , at , st+1 ) ] , where τ = ( s0 , a0 , s1 , . . . ) is a trajectory generated by following policy π. RL algorithms for MDPs try to find the policy π∗ that achieves the highest reward , i.e. , π∗ = arg maxπ J ( π ) . They commonly use the value function Vπ ( s ) = Eτ∼π [ ∑∞ t=0 γ tR ( st , at , st+1 ) |s0 = s ] , the action value function Qπ ( s , a ) = Eτ∼π [ ∑∞ t=0 γ tR ( st , at , st+1 ) |s0 = s , a0 = a ] and the advantage function Aπ ( s , a ) = Qπ ( s , a ) − Vπ ( s ) . The discounted future state distribution will also be useful , which is defined as dπ ( s ) = ( 1− γ ) ∑ t=0 γ tP ( st = s|π ) . Constrained Markov Decision Process ( CMDP ) ( Altman , 1999 ) extends MDP to environments with safety cost that could harm the agent when undesired actions are taken . As various safety costs may exist in a single CMDP , we relate them with m cost functions { C1 ( s , a , s′ ) , . . . , Cm ( s , a , s′ ) } , each of which denotes the cost an agent receives for each transition ( s , a , s′ ) ( similar to reward functions ) . Let Ci ( τ ) = ∑∞ t=0 γ tCi ( st , at , st+1 ) denote the cumulative cost along a trajectory τ generated from policy π . We consider a trajectory-based cost constraint in CMDP , which limits the cumulative cost in expectation JCi = Eτ∼π , s0∼µ [ Ci ( τ ) ] with value di . Then safe RL aims to learn the policy π under CMDP by solving the following problem , π∗ = arg max J ( π ) , s.t . JCi = Eτ∼π , s0∼µ [ Ci ( τ ) ] ≤ di , i = 1 , . . . , m. ( 1 ) Safe RL algorithms search for the policy π∗ that achieves the maximal cumulative reward and meanwhile does not violate the imposed safety constraints on the costs . In the following , analogous to the definition of value functions ( i.e. , Vπ , Qπ and Aπ ) , we use V Ciπ , Q Ci π and A Ci π to denote the cost-value functions w.r.t . cost function Ci .

Summary: The authors propose to improve a safe RL algorithm, constrained policy optimizaiton, that can learn the optimal safe policy while exploring unsafe states less often during the training process. In particular, they dampen the estimated advantage associated with unsafe states, which encourages the RL algorithm to explore safe states more often during the learning process. In addition, the authors aim to find a policy that satisfies the constraints with high probability, rather than only in expectation, by considering the worst-case constraints. The empirical results show that a safe RL algo that dampens the advantage and respects worst-case constraints are able to learn policies with large returns and avoid unsafe states.