A Primal Approach to Constrained Policy Optimization: Global Optimality and Finite-Time Analysis

√ T ) convergence rate to the global optimal policy in the constrained policy set and an O ( 1/ √ T ) error bound on constraint satisfaction . This is the first finite-time analysis of SRL algorithms with global optimality guarantee . Our empirical results demonstrate that CRPO can outperform the existing primal-dual baseline algorithms significantly . 1 INTRODUCTION . Reinforcement learning ( RL ) has achieved great success in solving complex sequential decisionmaking and control problems such as Go Silver et al . ( 2017 ) , StarCraft DeepMind ( 2019 ) and recommendation system Zheng et al . ( 2018 ) , etc . In these settings , the agent is allowed to explore the entire state and action space to maximize the expected total reward . However , in safe RL , in addition to maximizing the reward , an agent needs to satisfy certain constraints . Examples include self-driving cars Fisac et al . ( 2018 ) , cellular network Julian et al . ( 2002 ) , and robot control Levine et al . ( 2016 ) . One standard model for safe RL is constrained Markov Decision Process ( CMDP ) Altman ( 1999 ) , which further requires the policy to satisfy the constraints on a number of accumulated costs . The global optimal policy in this setting is the one that maximizes the reward and at the same time satisfies the cost constraints . In general , it is very challenging to find the global optimal policy in CMDP , as both the objective and constraints are nonconvex functions . A commonly used approach to solve CMDP is the primal-dual method Chow et al . ( 2017 ) ; Tessler et al . ( 2018 ) ; Ding et al . ( 2020a ) ; Stooke et al . ( 2020 ) , in which the constrained problem is converted to an unconstrained one by augmenting the objective with a sum of constraints weighted by their corresponding Lagrange multipliers . Usually , Lagrange multipliers are updated in the dual space concurrently Tessler et al . ( 2018 ) . Although it has been observed that primal-dual methods always converge to the feasible set in the end Ray et al . ( 2019 ) , such an approach is sensitive to the initialization of Lagrange multipliers and the learning rate , thus can incur extensive cost in hyperparameter tuning Achiam et al . ( 2017 ) ; Chow et al . ( 2019 ) . Another baseline approach is constrained policy optimization ( CPO ) , in which a linearized constrained problem is solved from scratch at each iteration to obtain the policy in the next step . However , a successful implementation of CPO requires a feasible initialization , which by itself can be very difficult especially with multiple constraints Ray et al . ( 2019 ) . Other approaches such as Lyapunov method Chow et al . ( 2018 ; 2019 ) , safety layer method Dalal et al . ( 2018a ) and interior point method Liu et al . ( 2019b ) have also been proposed recently . However , these methods do not have clear guidance in hyperparameter tuning , and thus suffer from nontrivial cost to implement in practice Stooke et al . ( 2020 ) . Thus , one goal here is to design an easy-to-implement SRL algorithm that enjoys the ease as uncontrained problems and readily approaches feasible points from random initialization . In contrast to the extensive empirical studies of SRL algorithms , theoretical understanding of the convergence properties of SRL algorithms is very limited . Tessler et al . ( 2018 ) provided an asymptotic convergence analysis for primal-dual method and established a local convergence guarantee under certain stability assumptions . Paternain et al . ( 2019 ) showed that the primal-dual method achieves zero duality gap , which can imply the global optimality under certain assumptions . Recently , Ding et al . ( 2020a ) proposed a primal-dual type proximal policy optimization ( PPO ) and established the regret bound for linear CMDP . The convergence rate of primal-dual method is characterized in a concurrent work Ding et al . ( 2020b ) . So far , there exist no primal-type SRL algorithms that have been shown to enjoy global optimality guarantee under general CMDP . Further , the finite-time performance ( convergence rate ) has not been characterized for any primal-type SRL algorithm . Thus , the second goal here is to establish global optimality guarantee and the finite-time convergence rate for the proposed algorithm under general CMDP . 1.1 MAIN CONTRIBUTIONS . We propose a novel Constraint-Rectified Policy Optimization ( CRPO ) approach for CMDP , where all updates are taken in the primal domain . CRPO applies unconstrained policy maximization update w.r.t . the reward on the one hand , and if any constraint is violated , momentarily rectifies the policy back to the feasible set along the descent direction of the violated constraint also by applying unconstrained policy minimization update w.r.t . the constraint function . Hence , CRPO can be implemented as easy as unconstrained policy optimization algorithms . It does not introduce heavy hyperparameter tuning to enforce constraint satisfaction , nor does it require initialization to be feasible . CRPO provides a primal-type framework for solving SRL problems , and its optimization update can adopt various well-developed unconstrained policy optimization methods such as natural policy gradient ( NPG ) Kakade ( 2002 ) , trust region policy optimization ( TRPO ) Schulman et al . ( 2015 ) , PPO , etc . To provide the theoretical guarantee for CRPO , we adopt NPG as a representative optimizer and investigate the convergence of CRPO in two settings : tabular and function approximation , where in the function approximation setting the state space can be infinite . For both settings , we show that CRPO converges to a global optimum at a convergence rate ofO ( 1/ √ T ) . Furthermore , the constraint satisfaction error converges to zero at a rate of O ( 1/ √ T ) . To the best of our knowledge , CRPO is the first primal-type SRL algorithm that has provably global optimality guarantee . This work also provides the first finite-time analysis for SRL algorithm without restrictive assumptions on CMDP . Our experiments demonstrate that CRPO outperforms the baseline primal-dual algorithm with higher return reward and smaller constraint satisfaction error . 1.2 RELATED WORK . Safe RL and CMDP : Algorithms based on primal-dual methods have been widely adopted for solving constrained RL problems , such as PDO Chow et al . ( 2017 ) , RCPO Tessler et al . ( 2018 ) , OPDOP Ding et al . ( 2020a ) and CPPO Stooke et al . ( 2020 ) . The effectiveness of primal-dual methods is justified in Paternain et al . ( 2019 ) , in which zero duality gap is guaranteed under certain assumptions . Constrained policy optimization ( CPO ) Achiam et al . ( 2017 ) extends TRPO to handle constraints , and is later modified with a two-step projection method Yang et al . ( 2019a ) . Other methods have also been proposed . For example , Chow et al . ( 2018 ; 2019 ) leveraged Lyapunov functions to handle constraints . Yu et al . ( 2019 ) proposed a constrained policy gradient algorithm with convergence guarantee by solving a sequence of sub-problems . Dalal et al . ( 2018a ) proposed to add a safety layer to the policy network so that constraints can be satisfied at each state . Liu et al . ( 2019b ) developed an interior point method for safe RL , which augments the objective with logarithmic barrier functions . This paper proposes a CRPO algorithm , which can be implemented as easy as unconstrained policy optimization methods and has global optimality guarantee under general CMDP . Finite-Time Analysis of Policy Optimization : The finite-time analysis of various policy optimization algorithms have been well studied . The convergence rate of policy gradient ( PG ) and actor-critic ( AC ) algorithms have been established in Shen et al . ( 2019 ) ; Papini et al . ( 2017 ; 2018 ) ; Xu et al . ( 2020a ; 2019 ) ; Xiong et al . ( 2020 ) ; Zhang et al . ( 2019 ) and Xu et al . ( 2020b ) ; Wang et al . ( 2019 ) ; Yang et al . ( 2019b ) ; Kumar et al . ( 2019 ) ; Qiu et al . ( 2019 ) , respectively , in which PG or AC algorithm is shown to converge to a local optimal . In some special settings such as tabular and LQR , PG and AC can be shown to convergence to the global optimal Agarwal et al . ( 2019 ) ; Yang et al . ( 2019b ) ; Fazel et al . ( 2018 ) ; Malik et al . ( 2018 ) ; Tu & Recht ( 2018 ) ; Bhandari & Russo ( 2019 ; 2020 ) . Algorithms such as NPG , NAC , TRPO and PPO explore the second order information , and achieve great success in practice . These algorithms have been shown to converge to a global optimum in various settings , where the convergence rate has been established in Agarwal et al . ( 2019 ) ; Shani et al . ( 2019 ) ; Liu et al . ( 2019a ) ; Wang et al . ( 2019 ) ; Cen et al . ( 2020 ) ; Xu et al . ( 2020c ) . However , all the above studies only consider unconstrained MDP . A concurrent and independent work Ding et al . ( 2020b ) established the global convergence rate of primal-dual method for CMDP under weak Slater ’ s condition assumption . So far the finite-time performance of primal-type policy optimization in general CMDP settings has not been studied . Our work is the first one that establishes such a result . 2 PROBLEM FORMULATION AND PRELIMINARIES . 2.1 MARKOV DECISION PROCESS . A discounted Markov decision process ( MDP ) is a tuple ( S , A , c0 , P , ξ , γ ) , where S and A are state and action spaces ; c0 : S×A×S → R is the reward function ; P : S×A×S → [ 0 , 1 ] is the transition kernel , with P ( s′|s , a ) denoting the probability of transitioning to state s′ from previous state s given action a ; ξ : S → [ 0 , 1 ] is the initial state distribution ; and γ ∈ ( 0 , 1 ) is the discount factor . A policy π : S → P ( A ) is a mapping from the state space to the space of probability distributions over the actions , with π ( ·|s ) denoting the probability of selecting action a in state s. When the associated Markov chain P ( s′|s ) = ∑ A P ( s ′|s , a ) π ( a|s ) is ergodic , we denote µπ as the stationary distribution of this MDP , i.e . ∫ S P ( s ′|s ) µπ ( ds ) = µπ ( s′ ) . Moreover , we define the visitation measure induced by the police π as νπ ( s , a ) = ( 1− γ ) ∑∞ t=0 γ tP ( st = s , at = a ) . For a given policy π , we define the state value function as V 0π ( s ) = E [ ∑∞ t=0 γ tc0 ( st , at , st+1 ) |s0 = s , π ] , the state-action value function as Q0π ( s , a ) = E [ ∑∞ t=0 γ tc0 ( st , at , st+1 ) |s0 = s , a0 = a , π ] , and the advantage function as A0π ( s , a ) = Q 0 π ( s , a ) − V 0π ( s ) . In reinforcement learning , we aim to find an optimal policy that maximizes the expected total reward function defined as J0 ( π ) = E [ ∑∞ t=0 γ tc0 ( st , at , st+1 ) ] = Eξ [ V 0π ( s ) ] = Eξ·π [ Q0π ( s , a ) ] .

Summary: This paper considers safe RL through solving a constrained MDP in (1). The authors proposed a primal method which alternates between maximizing the reward and minimizing the constraint violation. The convergence rate of the algorithm is also provided under standard settings, e.g., bounded reward, iid samples, etc.. Interestingly, the authors also analyzed the case for using a 2-layer neural network (NN) function approximator for the policy and the actor.