A Primal Approach to Constrained Policy Optimization: Global Optimality and Finite-Time Analysis

√ T ) convergence rate to the global optimal policy in the constrained policy set and an O ( 1/ √ T ) error bound on constraint satisfaction . This is the first finite-time analysis of SRL algorithms with global optimality guarantee . Our empirical results demonstrate that CRPO can outperform the existing primal-dual baseline algorithms significantly . 1 INTRODUCTION . Reinforcement learning ( RL ) has achieved great success in solving complex sequential decisionmaking and control problems such as Go Silver et al . ( 2017 ) , StarCraft DeepMind ( 2019 ) and recommendation system Zheng et al . ( 2018 ) , etc . In these settings , the agent is allowed to explore the entire state and action space to maximize the expected total reward . However , in safe RL , in addition to maximizing the reward , an agent needs to satisfy certain constraints . Examples include self-driving cars Fisac et al . ( 2018 ) , cellular network Julian et al . ( 2002 ) , and robot control Levine et al . ( 2016 ) . One standard model for safe RL is constrained Markov Decision Process ( CMDP ) Altman ( 1999 ) , which further requires the policy to satisfy the constraints on a number of accumulated costs . The global optimal policy in this setting is the one that maximizes the reward and at the same time satisfies the cost constraints . In general , it is very challenging to find the global optimal policy in CMDP , as both the objective and constraints are nonconvex functions . A commonly used approach to solve CMDP is the primal-dual method Chow et al . ( 2017 ) ; Tessler et al . ( 2018 ) ; Ding et al . ( 2020a ) ; Stooke et al . ( 2020 ) , in which the constrained problem is converted to an unconstrained one by augmenting the objective with a sum of constraints weighted by their corresponding Lagrange multipliers . Usually , Lagrange multipliers are updated in the dual space concurrently Tessler et al . ( 2018 ) . Although it has been observed that primal-dual methods always converge to the feasible set in the end Ray et al . ( 2019 ) , such an approach is sensitive to the initialization of Lagrange multipliers and the learning rate , thus can incur extensive cost in hyperparameter tuning Achiam et al . ( 2017 ) ; Chow et al . ( 2019 ) . Another baseline approach is constrained policy optimization ( CPO ) , in which a linearized constrained problem is solved from scratch at each iteration to obtain the policy in the next step . However , a successful implementation of CPO requires a feasible initialization , which by itself can be very difficult especially with multiple constraints Ray et al . ( 2019 ) . Other approaches such as Lyapunov method Chow et al . ( 2018 ; 2019 ) , safety layer method Dalal et al . ( 2018a ) and interior point method Liu et al . ( 2019b ) have also been proposed recently . However , these methods do not have clear guidance in hyperparameter tuning , and thus suffer from nontrivial cost to implement in practice Stooke et al . ( 2020 ) . Thus , one goal here is to design an easy-to-implement SRL algorithm that enjoys the ease as uncontrained problems and readily approaches feasible points from random initialization . In contrast to the extensive empirical studies of SRL algorithms , theoretical understanding of the convergence properties of SRL algorithms is very limited . Tessler et al . ( 2018 ) provided an asymptotic convergence analysis for primal-dual method and established a local convergence guarantee under certain stability assumptions . Paternain et al . ( 2019 ) showed that the primal-dual method achieves zero duality gap , which can imply the global optimality under certain assumptions . Recently , Ding et al . ( 2020a ) proposed a primal-dual type proximal policy optimization ( PPO ) and established the regret bound for linear CMDP . The convergence rate of primal-dual method is characterized in a concurrent work Ding et al . ( 2020b ) . So far , there exist no primal-type SRL algorithms that have been shown to enjoy global optimality guarantee under general CMDP . Further , the finite-time performance ( convergence rate ) has not been characterized for any primal-type SRL algorithm . Thus , the second goal here is to establish global optimality guarantee and the finite-time convergence rate for the proposed algorithm under general CMDP . 1.1 MAIN CONTRIBUTIONS . We propose a novel Constraint-Rectified Policy Optimization ( CRPO ) approach for CMDP , where all updates are taken in the primal domain . CRPO applies unconstrained policy maximization update w.r.t . the reward on the one hand , and if any constraint is violated , momentarily rectifies the policy back to the feasible set along the descent direction of the violated constraint also by applying unconstrained policy minimization update w.r.t . the constraint function . Hence , CRPO can be implemented as easy as unconstrained policy optimization algorithms . It does not introduce heavy hyperparameter tuning to enforce constraint satisfaction , nor does it require initialization to be feasible . CRPO provides a primal-type framework for solving SRL problems , and its optimization update can adopt various well-developed unconstrained policy optimization methods such as natural policy gradient ( NPG ) Kakade ( 2002 ) , trust region policy optimization ( TRPO ) Schulman et al . ( 2015 ) , PPO , etc . To provide the theoretical guarantee for CRPO , we adopt NPG as a representative optimizer and investigate the convergence of CRPO in two settings : tabular and function approximation , where in the function approximation setting the state space can be infinite . For both settings , we show that CRPO converges to a global optimum at a convergence rate ofO ( 1/ √ T ) . Furthermore , the constraint satisfaction error converges to zero at a rate of O ( 1/ √ T ) . To the best of our knowledge , CRPO is the first primal-type SRL algorithm that has provably global optimality guarantee . This work also provides the first finite-time analysis for SRL algorithm without restrictive assumptions on CMDP . Our experiments demonstrate that CRPO outperforms the baseline primal-dual algorithm with higher return reward and smaller constraint satisfaction error . 1.2 RELATED WORK . Safe RL and CMDP : Algorithms based on primal-dual methods have been widely adopted for solving constrained RL problems , such as PDO Chow et al . ( 2017 ) , RCPO Tessler et al . ( 2018 ) , OPDOP Ding et al . ( 2020a ) and CPPO Stooke et al . ( 2020 ) . The effectiveness of primal-dual methods is justified in Paternain et al . ( 2019 ) , in which zero duality gap is guaranteed under certain assumptions . Constrained policy optimization ( CPO ) Achiam et al . ( 2017 ) extends TRPO to handle constraints , and is later modified with a two-step projection method Yang et al . ( 2019a ) . Other methods have also been proposed . For example , Chow et al . ( 2018 ; 2019 ) leveraged Lyapunov functions to handle constraints . Yu et al . ( 2019 ) proposed a constrained policy gradient algorithm with convergence guarantee by solving a sequence of sub-problems . Dalal et al . ( 2018a ) proposed to add a safety layer to the policy network so that constraints can be satisfied at each state . Liu et al . ( 2019b ) developed an interior point method for safe RL , which augments the objective with logarithmic barrier functions . This paper proposes a CRPO algorithm , which can be implemented as easy as unconstrained policy optimization methods and has global optimality guarantee under general CMDP . Finite-Time Analysis of Policy Optimization : The finite-time analysis of various policy optimization algorithms have been well studied . The convergence rate of policy gradient ( PG ) and actor-critic ( AC ) algorithms have been established in Shen et al . ( 2019 ) ; Papini et al . ( 2017 ; 2018 ) ; Xu et al . ( 2020a ; 2019 ) ; Xiong et al . ( 2020 ) ; Zhang et al . ( 2019 ) and Xu et al . ( 2020b ) ; Wang et al . ( 2019 ) ; Yang et al . ( 2019b ) ; Kumar et al . ( 2019 ) ; Qiu et al . ( 2019 ) , respectively , in which PG or AC algorithm is shown to converge to a local optimal . In some special settings such as tabular and LQR , PG and AC can be shown to convergence to the global optimal Agarwal et al . ( 2019 ) ; Yang et al . ( 2019b ) ; Fazel et al . ( 2018 ) ; Malik et al . ( 2018 ) ; Tu & Recht ( 2018 ) ; Bhandari & Russo ( 2019 ; 2020 ) . Algorithms such as NPG , NAC , TRPO and PPO explore the second order information , and achieve great success in practice . These algorithms have been shown to converge to a global optimum in various settings , where the convergence rate has been established in Agarwal et al . ( 2019 ) ; Shani et al . ( 2019 ) ; Liu et al . ( 2019a ) ; Wang et al . ( 2019 ) ; Cen et al . ( 2020 ) ; Xu et al . ( 2020c ) . However , all the above studies only consider unconstrained MDP . A concurrent and independent work Ding et al . ( 2020b ) established the global convergence rate of primal-dual method for CMDP under weak Slater ’ s condition assumption . So far the finite-time performance of primal-type policy optimization in general CMDP settings has not been studied . Our work is the first one that establishes such a result . 2 PROBLEM FORMULATION AND PRELIMINARIES . 2.1 MARKOV DECISION PROCESS . A discounted Markov decision process ( MDP ) is a tuple ( S , A , c0 , P , ξ , γ ) , where S and A are state and action spaces ; c0 : S×A×S → R is the reward function ; P : S×A×S → [ 0 , 1 ] is the transition kernel , with P ( s′|s , a ) denoting the probability of transitioning to state s′ from previous state s given action a ; ξ : S → [ 0 , 1 ] is the initial state distribution ; and γ ∈ ( 0 , 1 ) is the discount factor . A policy π : S → P ( A ) is a mapping from the state space to the space of probability distributions over the actions , with π ( ·|s ) denoting the probability of selecting action a in state s. When the associated Markov chain P ( s′|s ) = ∑ A P ( s ′|s , a ) π ( a|s ) is ergodic , we denote µπ as the stationary distribution of this MDP , i.e . ∫ S P ( s ′|s ) µπ ( ds ) = µπ ( s′ ) . Moreover , we define the visitation measure induced by the police π as νπ ( s , a ) = ( 1− γ ) ∑∞ t=0 γ tP ( st = s , at = a ) . For a given policy π , we define the state value function as V 0π ( s ) = E [ ∑∞ t=0 γ tc0 ( st , at , st+1 ) |s0 = s , π ] , the state-action value function as Q0π ( s , a ) = E [ ∑∞ t=0 γ tc0 ( st , at , st+1 ) |s0 = s , a0 = a , π ] , and the advantage function as A0π ( s , a ) = Q 0 π ( s , a ) − V 0π ( s ) . In reinforcement learning , we aim to find an optimal policy that maximizes the expected total reward function defined as J0 ( π ) = E [ ∑∞ t=0 γ tc0 ( st , at , st+1 ) ] = Eξ [ V 0π ( s ) ] = Eξ·π [ Q0π ( s , a ) ] .

Summary: This paper proposes a new method for constrained MDP. The proposed method does not require primal-dual formulation and is easy to implement given the availability of state-of-the-art policy optimization solvers.  When the natural policy gradient is used as the  policy optimizer,  a sublinear global convergence rate is proved for both the tabular setting and the function approximation case.