CUP: A Conservative Update Policy Algorithm for Safe Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) ( Sutton & Barto , 1998 ) has achieved significant successes in many fields ( Mnih et al. , 2015 ; Silver et al. , 2017 ; OpenAI , 2019 ; Afsar et al. , 2021 ) , robotics ( Deisenroth et al. , 2013 ) , playing Go ( Silver et al. , 2016 ; 2017 ) , Starcraft ( Vinyals et al. , 2019 ) , Dota ( OpenAI , 2019 ) , and recommendation system ( Afsar et al. , 2021 ) . However , most RL algorithms improve the performance under the assumption that an agent is free to explore any behaviors . In real-world applications , only considering return maximization is not enough , and we also need to consider safe behaviors . For example , a robot agent should avoid playing actions that irrevocably harm its hardware , and a recommender system should avoid presenting offending items to users . Thus , it is crucial to consider safe exploration for RL , which is usually formulated as constrained Markov decision processes ( CMDP ) ( Altman , 1999 ) . It is challenging to solve CMDP since traditional approaches ( e.g. , Q-learning ( Watkins , 1989 ) & policy gradient ( Williams , 1992 ) ) usually violate the safe exploration constraints , which is undesirable for safe RL . Recently , Achiam et al . ( 2017 ) ; Yang et al . ( 2020 ) ; Bharadhwaj et al . ( 2021 ) suggest to use some surrogate functions to replace the objective and constraints . However , their implementations involve some convex approximations to the non-convex objective and safe constraints , which leads to many error sources and troubles . Concretely , Achiam et al . ( 2017 ) ; Yang et al . ( 2020 ) ; Bharadhwaj et al . ( 2021 ) approximate the non-convex objective ( or constraints ) with first-order or second Taylor expansion , but their implementations still lack a theory to show the error difference between the original objective ( or constraints ) and its convex approximations . Besides , their approaches involve the inverse of a high-dimension Fisher information matrix , which causes their algorithms to require a costly computation for each update when solving high-dimensional RL problems . Our Main Work . To address above problems , we propose the conservative update policy ( CUP ) algorithm with a theoretical safety guarantee . We derive the CUP bases on some new proposed surrogate functions with respect to objective and constraints and provide a practical implementation of CUP that does not depend on any convex approximation to adapt high-dimensional safe RL . Concretely , in Section 3 , Theorem 1 shows generalized difference bounds between two arbitrary policies for the objective and constraints . Those bounds provide principled approximations to the objective and constraints , which are theoretical foundations for us to use those bounds as surrogate functions to replace objective and constraints to design algorithms . Although using difference bound to replace objective or constraints has appeared in some existing works ( e.g. , ( Kakade & Langford , 2002 ; Schulman et al. , 2015 ; Achiam et al. , 2017 ) ) , Theorem 1 improves their bounds at least two aspects : ( i ) Firstly , our rigorous theoretical analysis extends the bound with respect to generalized advantage estimator ( GAE ) ( Schulman et al. , 2016 ) . GAE significantly reduces variance while maintains a tolerable level of bias , which is one of the critical steps for us to design efficient algorithms in the later section . Although Zhang et al . ( 2020 ) ; Kang et al . ( 2021 ) have applied GAE to solve safe RL problems , their approaches are empirical and lack a theoretical analysis with respect to GAE . Thus , our result provides a theory to illustrate the effectiveness of the work ( Zhang et al. , 2020 ; Kang et al. , 2021 ) . ( ii ) Our new bounds refine classic difference bounds . For example , our bounds are more compact than Achiam et al . ( 2017 ) , i , e. , using our new bounds as surrogate functions are better local approximations to the objective and constraints . Besides , the surrogate functions with respect to our new bounds are more accessible to be estimated from the samples than the approaches appears in ( Kakade & Langford , 2002 ; Schulman et al. , 2015 ) ) , for more discussions , please see Remark 1 . In Section 4 , we provide the necessary details of the proposed CUP . The CUP contains two steps : it performs a policy improvement at first , then it projects the policy back onto the safe region to reconcile the constraint violation . Theorem 2 shows a lower bound on policy improvement and an upper bound on constraint violation for CUP at each update . Notably , the result in Theorem 2 shows the bound of CUP is more compact than state-of-the-art safe RL algorithms : CPO ( Achiam et al. , 2017 , Proposition 1-2 ) , PCPO ( Yang et al. , 2020 , Theorem 1 ) and FOCOPS ( Zhang et al. , 2020 ) , which provides a partial explanation for why CUP is so good in practice . For more discussions , please refer to Remark 2 . Finally , we provide a practical implementation of sample-based CUP . Such an implementation allows us to use deep neural networks to train a model . Mainly , CUP does not depend on any convex approximation for objective and constraints , and it optimizes the objective according to the first-order optimizer . Extensive high-dimensional experiments on continuous control tasks show the effectiveness of CUP where the agent satisfies safe constraints . 2 PRELIMINARIES . Reinforcement learning ( RL ) ( Sutton & Barto , 1998 ) is often formulated as a Markov decision process ( MDP ) ( Puterman , 2014 ) that is a tupleM = ( S , A , P , r , ρ0 , γ ) . Here S is state space , A is action space . P ( s′ |s , a ) is probability of state transition from s to s′ after playing a. r ( · ) : S ×S ×A → R , and r ( s′|s , a ) denotes the reward that the agent observes when state transition from s to s′ after it plays a. ρ0 ( · ) : S → [ 0 , 1 ] is the initial state distribution and γ ∈ ( 0 , 1 ) . A stationary parameterized policy πθ is a probability distribution defined on S ×A , πθ ( a|s ) denotes the probability of playing a in state s. We use Πθ to denote the set of all stationary policies , where Πθ = { πθ : θ ∈ Rp } , and θ is a parameter needed to be learned . Let Pπθ ∈ R|S|×|S| be a state transition probability matrix , and their components are : Pπθ [ s , s ′ ] = ∑ a∈A πθ ( a|s ) P ( s′|s , a ) = : Pπθ ( s ′ |s ) , which denotes one-step state transformation probability from s to s′ by executing πθ . Let τ = { st , at , rt+1 } t≥0 ∼ πθ be a trajectory generated by πθ , where s0 ∼ ρ0 ( · ) , at ∼ πθ ( ·|st ) , st+1 ∼ P ( ·|st , at ) , and rt+1 = r ( st+1|st , at ) . We use Pπθ ( st = s ′ |s ) to denote the probability of visiting the state s ′ after t time steps from the state s by executing πθ . Due to the Markov property in MDP , Pπθ ( st = s ′ |s ) is ( s , s′ ) -th component of the matrix Ptπθ , i.e. , Pπθ ( st = s ′ |s ) = Ptπθ [ s , s ′ ] . Finally , let ds0πθ ( s ) = ( 1 − γ ) ∑∞ t=0 γ tPπθ ( st = s|s0 ) be the stationary state distribution of the Markov chain ( starting at s0 ) induced by policy πθ . We define dρ0πθ ( s ) = Es0∼ρ0 ( · ) [ d s0 πθ ( s ) ] as the discounted state visitation distribution on initial distribution ρ0 ( · ) . The state value function of πθ is defined as Vπθ ( s ) = Eπθ [ ∑∞ t=0 γ trt+1|s0 = s ] , where Eπθ [ ·|· ] denotes a conditional expectation on actions which are selected by πθ . Its state-action value function is Qπθ ( s , a ) = Eπθ [ ∑∞ t=0 γ trt+1|s0 = s , a0 = a ] , and advantage function is Aπθ ( s , a ) = Qπθ ( s , a ) − Vπθ ( s ) . The goal of reinforcement learning is to maximize J ( πθ ) : J ( πθ ) = Es∼dρ0πθ ( · ) [ Vπθ ( s ) ] . ( 1 ) 2.1 POLICY GRADIENT AND GENERALIZED ADVANTAGE ESTIMATOR ( GAE ) . Policy gradient ( Williams , 1992 ; Sutton et al. , 2000 ) is widely used to solve policy optimization , which maximizes the expected total reward by repeatedly estimating the gradient g = ∇J ( πθ ) . Schulman et al . ( 2016 ) summarize several different related expressions for the policy gradient : g = ∇J ( πθ ) = E [ ∞∑ t=0 Ψt∇ log πθ ( at|st ) ] , ( 2 ) where Ψt can be total discounted reward of the trajectory , value function , advantage function or temporal difference ( TD ) error . As stated by Schulman et al . ( 2016 ) , the choice Ψt = A ( st , at ) yields almost the lowest possible variance , which is consistent with the theoretical analysis ( Greensmith et al. , 2004 ; Wu et al. , 2018 ) . Furthermore , Schulman et al . ( 2016 ) propose generalized advantage estimator ( GAE ) ÂGAE ( γ , λ ) t ( st , at ) to replace Ψt : for any λ ∈ [ 0 , 1 ] , Â GAE ( γ , λ ) t ( st , at ) = ∞∑ ` =0 ( γλ ) ` δVt+ ` , ( 3 ) where δVt = rt+1 + γV ( st+1 ) − V ( st ) is TD error , and V ( · ) is an estimator of value function . GAE is an efficient technique for data efficiency and reliable performance of reinforcement learning . 2.2 SAFE REINFORCEMENT LEARNING . Safe RL ( Ray et al. , 2019 ) is often formulated as a constrained MDP ( CMDP ) M∪C ( Altman , 1999 ) , which is a standard MDPM augmented with an additional constraint set C. The set C = { ( ci , bi ) } mi=1 , where ci are cost functions : ci : S ×A → R , and limits are bi , i = 1 , · , m . The cost-return is defined as : Jci ( πθ ) = Eπθ [ ∑∞ t=0 γ tci ( st , at ) ] , then we define the feasible policy set ΠC as : ΠC = ∩mi=1 { πθ ∈ Πθ and Jci ( πθ ) ≤bi } . The goal of CMDP is to search the optimal policy π ? such that π ? = arg max πθ∈ΠC J ( πθ ) . ( 4 ) Furthermore , we define value functions , action-value functions , and advantage functions for the auxiliary costs in analogy to Vπθ , Qπθ , and Aπθ , with ci replacing r respectively , we denote them as V ciπθ , Q ci πθ , and Aciπθ . For example , V ci πθ ( s ) = Eπθ [ ∑∞ t=0 γ tci ( st , at ) |s0 = s ] . Without loss of generality , we will restrict our discussion to the case of one constraint with a cost function c and upper bound b . Finally , we extend the GAE w.r.t . auxiliary cost function c : Â GAE ( γ , λ ) C , t ( st , at ) = ∞∑ ` =0 ( γλ ) ` δCt+ ` , ( 5 ) where δCt = rt+1 + γC ( st+1 ) − C ( st ) is TD error , and C ( · ) is an estimator of cost function c .

Summary: The paper studies the problem of learning under constrained Markov decision processes. It proposes to solve this problem by an RL algorithm based on generalized advantage estimator (GAE), which they call  Conservative Update Policy (CUP). Although existing works have already proposed GAE-based algorithms, the current paper is the first to provide theoretical foundations for this type of algorithm. Specifically, it proves bounds on performance difference for GAE-based algorithms and one-step improvement for CUP. Numerical experiments have been conducted to showcase the performance of the proposed algorithm.