CUP: A Conservative Update Policy Algorithm for Safe Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) ( Sutton & Barto , 1998 ) has achieved significant successes in many fields ( Mnih et al. , 2015 ; Silver et al. , 2017 ; OpenAI , 2019 ; Afsar et al. , 2021 ) , robotics ( Deisenroth et al. , 2013 ) , playing Go ( Silver et al. , 2016 ; 2017 ) , Starcraft ( Vinyals et al. , 2019 ) , Dota ( OpenAI , 2019 ) , and recommendation system ( Afsar et al. , 2021 ) . However , most RL algorithms improve the performance under the assumption that an agent is free to explore any behaviors . In real-world applications , only considering return maximization is not enough , and we also need to consider safe behaviors . For example , a robot agent should avoid playing actions that irrevocably harm its hardware , and a recommender system should avoid presenting offending items to users . Thus , it is crucial to consider safe exploration for RL , which is usually formulated as constrained Markov decision processes ( CMDP ) ( Altman , 1999 ) . It is challenging to solve CMDP since traditional approaches ( e.g. , Q-learning ( Watkins , 1989 ) & policy gradient ( Williams , 1992 ) ) usually violate the safe exploration constraints , which is undesirable for safe RL . Recently , Achiam et al . ( 2017 ) ; Yang et al . ( 2020 ) ; Bharadhwaj et al . ( 2021 ) suggest to use some surrogate functions to replace the objective and constraints . However , their implementations involve some convex approximations to the non-convex objective and safe constraints , which leads to many error sources and troubles . Concretely , Achiam et al . ( 2017 ) ; Yang et al . ( 2020 ) ; Bharadhwaj et al . ( 2021 ) approximate the non-convex objective ( or constraints ) with first-order or second Taylor expansion , but their implementations still lack a theory to show the error difference between the original objective ( or constraints ) and its convex approximations . Besides , their approaches involve the inverse of a high-dimension Fisher information matrix , which causes their algorithms to require a costly computation for each update when solving high-dimensional RL problems . Our Main Work . To address above problems , we propose the conservative update policy ( CUP ) algorithm with a theoretical safety guarantee . We derive the CUP bases on some new proposed surrogate functions with respect to objective and constraints and provide a practical implementation of CUP that does not depend on any convex approximation to adapt high-dimensional safe RL . Concretely , in Section 3 , Theorem 1 shows generalized difference bounds between two arbitrary policies for the objective and constraints . Those bounds provide principled approximations to the objective and constraints , which are theoretical foundations for us to use those bounds as surrogate functions to replace objective and constraints to design algorithms . Although using difference bound to replace objective or constraints has appeared in some existing works ( e.g. , ( Kakade & Langford , 2002 ; Schulman et al. , 2015 ; Achiam et al. , 2017 ) ) , Theorem 1 improves their bounds at least two aspects : ( i ) Firstly , our rigorous theoretical analysis extends the bound with respect to generalized advantage estimator ( GAE ) ( Schulman et al. , 2016 ) . GAE significantly reduces variance while maintains a tolerable level of bias , which is one of the critical steps for us to design efficient algorithms in the later section . Although Zhang et al . ( 2020 ) ; Kang et al . ( 2021 ) have applied GAE to solve safe RL problems , their approaches are empirical and lack a theoretical analysis with respect to GAE . Thus , our result provides a theory to illustrate the effectiveness of the work ( Zhang et al. , 2020 ; Kang et al. , 2021 ) . ( ii ) Our new bounds refine classic difference bounds . For example , our bounds are more compact than Achiam et al . ( 2017 ) , i , e. , using our new bounds as surrogate functions are better local approximations to the objective and constraints . Besides , the surrogate functions with respect to our new bounds are more accessible to be estimated from the samples than the approaches appears in ( Kakade & Langford , 2002 ; Schulman et al. , 2015 ) ) , for more discussions , please see Remark 1 . In Section 4 , we provide the necessary details of the proposed CUP . The CUP contains two steps : it performs a policy improvement at first , then it projects the policy back onto the safe region to reconcile the constraint violation . Theorem 2 shows a lower bound on policy improvement and an upper bound on constraint violation for CUP at each update . Notably , the result in Theorem 2 shows the bound of CUP is more compact than state-of-the-art safe RL algorithms : CPO ( Achiam et al. , 2017 , Proposition 1-2 ) , PCPO ( Yang et al. , 2020 , Theorem 1 ) and FOCOPS ( Zhang et al. , 2020 ) , which provides a partial explanation for why CUP is so good in practice . For more discussions , please refer to Remark 2 . Finally , we provide a practical implementation of sample-based CUP . Such an implementation allows us to use deep neural networks to train a model . Mainly , CUP does not depend on any convex approximation for objective and constraints , and it optimizes the objective according to the first-order optimizer . Extensive high-dimensional experiments on continuous control tasks show the effectiveness of CUP where the agent satisfies safe constraints . 2 PRELIMINARIES . Reinforcement learning ( RL ) ( Sutton & Barto , 1998 ) is often formulated as a Markov decision process ( MDP ) ( Puterman , 2014 ) that is a tupleM = ( S , A , P , r , ρ0 , γ ) . Here S is state space , A is action space . P ( s′ |s , a ) is probability of state transition from s to s′ after playing a. r ( · ) : S ×S ×A → R , and r ( s′|s , a ) denotes the reward that the agent observes when state transition from s to s′ after it plays a. ρ0 ( · ) : S → [ 0 , 1 ] is the initial state distribution and γ ∈ ( 0 , 1 ) . A stationary parameterized policy πθ is a probability distribution defined on S ×A , πθ ( a|s ) denotes the probability of playing a in state s. We use Πθ to denote the set of all stationary policies , where Πθ = { πθ : θ ∈ Rp } , and θ is a parameter needed to be learned . Let Pπθ ∈ R|S|×|S| be a state transition probability matrix , and their components are : Pπθ [ s , s ′ ] = ∑ a∈A πθ ( a|s ) P ( s′|s , a ) = : Pπθ ( s ′ |s ) , which denotes one-step state transformation probability from s to s′ by executing πθ . Let τ = { st , at , rt+1 } t≥0 ∼ πθ be a trajectory generated by πθ , where s0 ∼ ρ0 ( · ) , at ∼ πθ ( ·|st ) , st+1 ∼ P ( ·|st , at ) , and rt+1 = r ( st+1|st , at ) . We use Pπθ ( st = s ′ |s ) to denote the probability of visiting the state s ′ after t time steps from the state s by executing πθ . Due to the Markov property in MDP , Pπθ ( st = s ′ |s ) is ( s , s′ ) -th component of the matrix Ptπθ , i.e. , Pπθ ( st = s ′ |s ) = Ptπθ [ s , s ′ ] . Finally , let ds0πθ ( s ) = ( 1 − γ ) ∑∞ t=0 γ tPπθ ( st = s|s0 ) be the stationary state distribution of the Markov chain ( starting at s0 ) induced by policy πθ . We define dρ0πθ ( s ) = Es0∼ρ0 ( · ) [ d s0 πθ ( s ) ] as the discounted state visitation distribution on initial distribution ρ0 ( · ) . The state value function of πθ is defined as Vπθ ( s ) = Eπθ [ ∑∞ t=0 γ trt+1|s0 = s ] , where Eπθ [ ·|· ] denotes a conditional expectation on actions which are selected by πθ . Its state-action value function is Qπθ ( s , a ) = Eπθ [ ∑∞ t=0 γ trt+1|s0 = s , a0 = a ] , and advantage function is Aπθ ( s , a ) = Qπθ ( s , a ) − Vπθ ( s ) . The goal of reinforcement learning is to maximize J ( πθ ) : J ( πθ ) = Es∼dρ0πθ ( · ) [ Vπθ ( s ) ] . ( 1 ) 2.1 POLICY GRADIENT AND GENERALIZED ADVANTAGE ESTIMATOR ( GAE ) . Policy gradient ( Williams , 1992 ; Sutton et al. , 2000 ) is widely used to solve policy optimization , which maximizes the expected total reward by repeatedly estimating the gradient g = ∇J ( πθ ) . Schulman et al . ( 2016 ) summarize several different related expressions for the policy gradient : g = ∇J ( πθ ) = E [ ∞∑ t=0 Ψt∇ log πθ ( at|st ) ] , ( 2 ) where Ψt can be total discounted reward of the trajectory , value function , advantage function or temporal difference ( TD ) error . As stated by Schulman et al . ( 2016 ) , the choice Ψt = A ( st , at ) yields almost the lowest possible variance , which is consistent with the theoretical analysis ( Greensmith et al. , 2004 ; Wu et al. , 2018 ) . Furthermore , Schulman et al . ( 2016 ) propose generalized advantage estimator ( GAE ) ÂGAE ( γ , λ ) t ( st , at ) to replace Ψt : for any λ ∈ [ 0 , 1 ] , Â GAE ( γ , λ ) t ( st , at ) = ∞∑ ` =0 ( γλ ) ` δVt+ ` , ( 3 ) where δVt = rt+1 + γV ( st+1 ) − V ( st ) is TD error , and V ( · ) is an estimator of value function . GAE is an efficient technique for data efficiency and reliable performance of reinforcement learning . 2.2 SAFE REINFORCEMENT LEARNING . Safe RL ( Ray et al. , 2019 ) is often formulated as a constrained MDP ( CMDP ) M∪C ( Altman , 1999 ) , which is a standard MDPM augmented with an additional constraint set C. The set C = { ( ci , bi ) } mi=1 , where ci are cost functions : ci : S ×A → R , and limits are bi , i = 1 , · , m . The cost-return is defined as : Jci ( πθ ) = Eπθ [ ∑∞ t=0 γ tci ( st , at ) ] , then we define the feasible policy set ΠC as : ΠC = ∩mi=1 { πθ ∈ Πθ and Jci ( πθ ) ≤bi } . The goal of CMDP is to search the optimal policy π ? such that π ? = arg max πθ∈ΠC J ( πθ ) . ( 4 ) Furthermore , we define value functions , action-value functions , and advantage functions for the auxiliary costs in analogy to Vπθ , Qπθ , and Aπθ , with ci replacing r respectively , we denote them as V ciπθ , Q ci πθ , and Aciπθ . For example , V ci πθ ( s ) = Eπθ [ ∑∞ t=0 γ tci ( st , at ) |s0 = s ] . Without loss of generality , we will restrict our discussion to the case of one constraint with a cost function c and upper bound b . Finally , we extend the GAE w.r.t . auxiliary cost function c : Â GAE ( γ , λ ) C , t ( st , at ) = ∞∑ ` =0 ( γλ ) ` δCt+ ` , ( 5 ) where δCt = rt+1 + γC ( st+1 ) − C ( st ) is TD error , and C ( · ) is an estimator of cost function c .

Summary: This paper considers reinforcement learning, a common model which has seen success recently in many areas (e.g. games, robotics, autonomous vehicles).  Prototypical reinforcement learning algorithms explore the action space in order to maximize their rewards as much as possible, potentially ignoring impacts of the chosen actions or safety constraints.  Typical to many real-world scenarios, though, are constraints on the selected actions allowing the algorithm designer or practitioner to enforce certain constraints on the selected actions in the environment (e.g. ensure that the autonomous vehicle stays within the dictated lines on the road, etc).  This is typically done by introducing a cost-function and adding an additional constraint that the resulting policies have long-run discounted cost upper bounded by a given constant.  While many RL algorithms have been designed in this setting, they mostly focus on either taking lagrangian relaxations of the given policy constraints, or constructing convex approximations to the non-convex objectives to optimize (which also in turns adds to additional complexity in constructing higher order moments of the objective for optimization).  In contrast, the authors in this paper take a different view, via the following: 1. Construct tight upper and lower bounds on the difference in value functions for arbitrary policies with respect to arbitrary functions $\phi$, a given additional discount parameter $\lambda$ 2. Instantiate the upper bound with $\phi$ taken as the value function to get a lower bound on the difference in value between any two policies 3. Instantiate the lower bound with $\phi$ taken as the cost-value function to get an upper bound on the cost-difference between any two policies  Once these bounds are established, the algorithmic approach is simple.  In the first step they perform a performance improvement step to improve the performance of the policy by maximizing the lower bound on the difference in value between the two policies.  The second step is a projection step, where after lagrangianizing the constraint set they minimize the lower bound on the difference in costs between the two policies.  The authors show a guarantee on per-step improvements on the rewards and cost for this proposed approach assuming exact maximization / minimization of these objectives.  To be more specific, the authors consider the typical RL model with an MDP characterized via $(S, A, P, r, \gamma)$.  They consider the goal of maximizing the expected long term rewards $r$.  However, there is an additional cost function $c$ and they add an additional constraint that the long-run discounted cost is upper bounded by a given constant $b$.  The hope is that the learned policy picks actions which satisfy the long-run average cost constraint while simultaneously maximizing the expected rewards.  The authors first start-off by proposing a novel bound, generalizing the policy performance difference $J(\pi_\theta) - J(\pi_{\theta'})$ between any two policies.  In particular the difference is upper and lower bounded by two terms.  The first term can be interpreted as the expectation between the TD errors of $\pi_\theta$ and $\pi_{\theta'}$ where TD errors are computed for an arbitrary function $\phi$.  The second part is the discounted distribution difference between the two different policies.  These bounds are then instantiated with different functions $\phi$ and parameters in order to get lower and upper bounds on the performance difference between the policies with respect to the costs and rewards of the problem.  Once these lower and upper bounds on the performance difference are established, the algorithm falls naturally by maximizing the reward lower bound and minimizing the cost upper bound.  This is then theoretically justified by providing a per-step improvement with respect to the two different objectives.    To complement the algorithmic framework, the authors present a set of synthetic experiments to compare the efficiency and constraint violation of the resulting policies of their method and others in the literature.  In particular, they test the policies on three different environments and compare to several other existing algorithms in the literature (more on this later).  They show that their algorithm and approach dominates others in terms of performance while simultaneously ensuring the cost constraints.