Projection-Based Constrained Policy Optimization

1 INTRODUCTION . Recent advances in deep reinforcement learning ( RL ) have demonstrated excellent performance on several domains ranging from games like Go ( Silver et al. , 2017 ) and StarCraft ( AlphaStar , 2019 ) to robotic control ( Levine et al. , 2016 ) . In these settings , agents are allowed to explore the entire state space and experiment with all possible actions during training . However , in many realworld applications such as self-driving cars and unmanned aerial vehicles , considerations of safety , fairness and other costs prevent the agent from having complete freedom to explore . For instance , an autonomous car , while optimizing its driving policies , must not take any actions that could cause harm to pedestrians or property ( including itself ) . In effect , the agent is constrained to take actions that do not violate a specified set of constraints on state-action pairs . In this work , we address the problem of learning control policies that optimize a reward function while satisfying predefined constraints . The problem of policy learning with constraints is more challenging since directly optimizing for the reward , as in Q-Learning ( Mnih et al. , 2013 ) or policy gradient ( Sutton et al. , 2000 ) , will usually violate the constraints . One approach is to incorporate constraints into the learning process by forming a constrained optimization problem . Then perform policy updates using a conditional gradient descent with line search to ensure constraint satisfaction ( Achiam et al. , 2017 ) . However , the base optimization problem can become infeasible if the current policy violates the constraints . Another approach is to add a hyperparameter weighted copy of the constraints to the objective function ( Tessler et al. , 2018 ) . However , this incurs the cost of extensive hyperparameter tuning . To address the above issues , we propose projection-based constrained policy optimization ( PCPO ) . This is an iterative algorithm that performs policy updates in two stages . The first stage maximizes reward using a trust region optimization method ( e.g. , TRPO ( Schulman et al. , 2015a ) ) without 1 For code see the project website : https : //sites.google.com/view/iclr2020-pcpo constraints . This might result in a new intermediate policy that does not satisfy the constraints . The second stage reconciles the constraint violation ( if any ) by projecting the policy back onto the constraint set , i.e. , choosing the policy in the constraint set that is closest to the selected intermediate policy . This allows efficient updates to ensure constraint satisfaction without requiring a line search ( Achiam et al. , 2017 ) or adjusting a weight ( Tessler et al. , 2018 ) . Further , due to the projection step , PCPO offers efficient recovery from infeasible ( i.e. , constraint-violating ) states ( e.g. , due to approximation errors ) , which existing methods do not handle well . We analyze PCPO theoretically and derive performance bounds for the algorithm . Specifically , based on information geometry and policy optimization theory , we construct a lower bound on reward improvement , and an upper bound on constraint violations for each policy update . We find that with a relatively small step size for each policy update , the worst-case constraint violation and reward degradation are tolerable . We further analyze two distance measures for the projection step onto the constraint set . We find that the convergence of PCPO is affected by the smallest and largest singular values of the Fisher information matrix used during training . By observing these singular values , we can choose the appropriate projection best suited to the problem . Empirically , we compare PCPO with state-of-the-art algorithms on four different control tasks , including two Mujoco environments with safety constraints introduced by Achiam et al . ( 2017 ) and two traffic management tasks with fairness constraints introduced by Vinitsky et al . ( 2018 ) . In all cases , the proposed algorithm achieves comparable or superior performance to prior approaches , averaging more reward with fewer cumulative constraint violations . For instance , across the above tasks , PCPO achieves 3.5 times fewer constraint violations and around 15 % more reward . This demonstrates the ability of PCPO robustly learn constraint-satisfying policies , and represents a step towards reliable deployment of RL in real problems . 2 PRELIMINARIES . We frame our policy learning as a constrained Markov Decision Process ( CMDP ) ( Altman , 1999 ) , where policies will direct the agent to maximize the reward while minimizing the cost . We define CMDP as the tuple < S , A , T , R , C > , where S is the set of states , A is the set of actions that the agent can take , T : S ×A×S → [ 0 , 1 ] is the transition probability of the CMDP , R : S ×A → R is the reward function , and C : S × A → R is the cost function . Given the agent ’ s current state s , the policy π ( a|s ) : S → A selects an action a for the agent to take . Based on s and a , the agent transits to the next state ( denoted by s′ ) according to the state transition model T ( s′|s , a ) , and receives the reward and pays the cost , denoted by R ( s , a ) and C ( s , a ) , respectively . We aim to learn a policy π that maximizes a cumulative discounted reward , denoted by JR ( π ) . = Eτ∼π [ ∞∑ t=0 γtR ( st , at ) ] , while satisfying constraints , i.e. , making a cumulative discounted cost constraint below a desired threshold h , denoted by JC ( π ) . = Eτ∼π [ ∞∑ t=0 γtC ( st , at ) ] ≤ h , where γ is the discount factor , τ is the trajectory ( τ = ( s0 , a0 , s1 , · · · ) ) , and τ ∼ π is shorthand for showing that the distribution over the trajectory depends on π : s0 ∼ µ , at ∼ π ( at|st ) , st+1 ∼ T ( st+1|st , at ) , where µ is the initial state distribution . Kakade & Langford ( 2002 ) give an identity to express the performance of policy π′ in terms of the advantage function over another policy π : JR ( π′ ) − JR ( π ) = 1 1− γ E s∼dπ ′ a∼π′ [ AπR ( s , a ) ] , ( 1 ) where dπ is the discounted future state distribution , denoted by dπ ( s ) .= ( 1 − γ ) ∑∞ t=0 γ tP ( st = s|π ) , and AπR ( s , a ) is the reward advantage function , denoted by AπR ( s , a ) . = QπR ( s , a ) − V πR ( s ) . Here Q π R ( s , a ) . = Eτ∼π [ ∑∞ t=0 γ tR ( st , at ) |s0 = s , a0 = a ] is the discounted cumulative reward obtained by the policy π given the initial state s and action a , and V πR ( s ) . = Eτ∼π [ ∑∞ t=0 γ tR ( st , at ) |s0 = s ] is the discounted cumulative reward obtained by the policy π given the initial state s. Similarly , we have the cost advantage function AπC ( s , a ) = QπC ( s , a ) − V πC ( s ) , where QπC ( s , a ) . = Eτ∼π [ ∑∞ t=0 γ tC ( st , at ) |s0 = s , a0 = a ] , and V πC ( s ) . = Eτ∼π [ ∑∞ t=0 γ tC ( st , at ) |s0 = s ] . 3 PROJECTION-BASED CONSTRAINED POLICY OPTIMIZATION . To robustly learn constraint-satisfying policies , we develop PCPO – a trust region method that performs policy updates corresponding to reward improvement , followed by projections onto the constraint set . PCPO , inspired by projected gradient descent , is composed of two steps for each update , a reward improvement step and a projection step ( This is illustrated in Fig . 1 ) . Reward Improvement Step . First , we optimize the reward function by maximizing the reward advantage function AπR ( s , a ) subject to a Kullback-Leibler ( KL ) divergence constraint . This constraints the intermediate policy πk+ 1 2 to be within a δ-neighbourhood of πk : πk+ 1 2 = argmax π E s∼dπ k a∼π [ Aπ k R ( s , a ) ] s.t . E s∼dπk [ DKL ( π||πk ) [ s ] ] ≤ δ . ( 2 ) This update rule with the trust region , { π : E s∼dπk [ DKL ( π||πk ) [ s ] ] ≤ δ } , is called Trust Region Policy Optimization ( TRPO ) ( Schulman et al. , 2015a ) . It constraints the policy changes to a divergence neighborhood and guarantees reward improvement . Projection Step . Second , we project the intermediate policy πk+ 12 onto the constraint set by minimizing a distance measure D between πk+ 1 2 and π : πk+1 = argmin π D ( π , πk+ 1 2 ) s.t . JC ( πk ) + E s∼dπ k a∼π [ Aπ k C ( s , a ) ] ≤ h. ( 3 ) The projection step ensures that the constraint-satisfying policy πk+1 is close to πk+ 1 2 . We consider two distance measures D : L2 norm and KL divergence . In contrast , using KL divergence projection in the probability distribution space allows us to provide provable guarantees for PCPO . 3.1 PERFORMANCE BOUND FOR PCPO WITH KL DIVERGENCE PROJECTION . In safety-critical applications such as autonomous cars , one cares about how worse the performance of a system evolves when applying a learning algorithm . To this end , for PCPO with KL divergence projection , we analyze the worst-case performance degradation for each policy update when the current policy πk satisfies the constraint . The following theorem provides a lower bound on reward improvement , and an upper bound on constraint violation for each policy update . Theorem 3.1 ( Worst-case Bound on Updating Constraint-satisfying Policies ) . Define π k+1 R . = max s ∣∣Ea∼πk+1 [ AπkR ( s , a ) ] ∣∣ , and πk+1C .= maxs ∣∣Ea∼πk+1 [ AπkC ( s , a ) ] ∣∣ . If the current policy πk satisfies the constraint , then under KL divergence projection , the lower bound on reward improvement , and upper bound on constraint violation for each policy update are JR ( πk+1 ) − JR ( πk ) ≥ − √ 2δγ π k+1 R ( 1− γ ) 2 , and JC ( πk+1 ) ≤ h+ √ 2δγ π k+1 C ( 1− γ ) 2 , where δ is the step size in the reward improvement step . Proof . See the supplemental material . Theorem 3.1 indicates that if δ is small , the worst-case performance degradation is tolerable . Due to approximation errors or the random initialization of policies , PCPO may have a constraintviolating update . Theorem 3.1 does not give the guarantee on updating a constraint-violating policy . Hence we analyze worst-case performance degradation for each policy update when the current policy πk violates the constraint . The following theorem provides a lower bound on reward improvement , and an upper bound on constraint violation for each policy update . Theorem 3.2 ( Worst-case Bound on Updating Constraint-violating Policies ) . Define π k+1 R . = max s ∣∣Ea∼πk+1 [ AπkR ( s , a ) ] ∣∣ , πk+1C .= maxs ∣∣Ea∼πk+1 [ AπkC ( s , a ) ] ∣∣ , b+ .= max ( 0 , JC ( πk ) − h ) , and αKL . = 1 2aTH−1a , where a is the gradient of the cost advantage function and H is the Hessian of the KL divergence constraint . If the current policy πk violates the constraint , then under KL divergence projection , the lower bound on reward improvement and the upper bound on constraint violation for each policy update are JR ( πk+1 ) − JR ( πk ) ≥− √ 2 ( δ + b+2αKL ) γ πk+1 R ( 1− γ ) 2 , and JC ( πk+1 ) ≤ h+ √ 2 ( δ + b+2αKL ) γ πk+1 C ( 1− γ ) 2 , where δ is the step size in the reward improvement step . Proof . See the supplemental material . Theorem 3.2 indicates that when the policy has greater constraint violation ( b+ increases ) , its worstcase performance degradation increases . Note that Theorem 3.2 reduces to Theorem 3.1 if the current policy πk satisfies the constraint ( b+ = 0 ) . The proofs of Theorem 3.1 and Theorem 3.2 follow from the fact that the projection of the policy is non-expansive , i.e. , the distance between the projected policies is smaller than that of the unprojected policies . This allows us to measure it and bound the KL divergence between the current policy and the new policy .

Summary: This paper proposes a new algorithm - Projection based Constrained Policy Optimization, that is able to learn policies with constraints, i.e., for CMDPs. The algorithm consists of two stages: first an unconstrained update for maximizing reward, and the second step for projecting the policy back to the constraint set.  The authors provide analysis in terms of bounds for reward improvement and constraint violation.  The authors characterize the convergence with two projection metrics: KL divergence and L2 norm.  The new algorithm is tested on four control tasks: two mujoco environments with safety constraints, and two traffic management tasks, where it outperforms the CPO and lagrangian based approaches.