Reward Constrained Policy Optimization

1 Introduction . Applying Reinforcement Learning ( RL ) is generally a hard problem . At each state , the agent performs an action which produces a reward . The goal is to maximize the accumulated reward , hence the reward signal implicitly defines the behavior of the agent . While in computer games ( e.g . Bellemare et al . ( 2013 ) ) there exists a pre-defined reward signal , it is not such in many real applications . An example is the Mujoco domain ( Todorov et al. , 2012 ) , in which the goal is to learn to control robotic agents in tasks such as : standing up , walking , navigation and more . Considering the Humanoid domain , the agent is a 3 dimensional humanoid and the task is to walk forward as far as possible ( without falling down ) within a fixed amount of time . Naturally , a reward is provided based on the forward velocity in order to encourage a larger distance ; however , additional reward signals are provided in order to guide the agent , for instance a bonus for staying alive , a penalty for energy usage and a penalty based on the force of impact between the feet and the floor ( which should encourage less erratic behavior ) . Each signal is multiplied by it ’ s own coefficient , which controls the emphasis placed on it . This approach is a multi-objective problem ( Mannor and Shimkin , 2004 ) ; in which for each set of penalty coefficients , there exists a different , optimal solution , also known as Pareto optimality ( Van Moffaert and Nowé , 2014 ) . In practice , the exact coefficient is selected through a time consuming and a computationally intensive process of hyper-parameter tuning . As our experiments show , the coefficient is not shared across domains , a coefficient which leads to a satisfying behavior on one domain may lead to catastrophic failure on the other ( issues also seen in Leike et al . ( 2017 ) and Mania et al . ( 2018 ) ) . Constraints are a natural and consistent approach , an approach which ensures a satisfying behavior without the need for manually selecting the penalty coefficients . In constrained optimization , the task is to maximize a target function f ( x ) while satisfying an inequality constraint g ( x ) ≤ α . While constraints are a promising solution to ensuring a satisfying behavior , existing methods are limited in the type of constraints they are able to handle and the algorithms that they may support - they require a parametrization of the policy ( policy gradient methods ) and propagation of the constraint violation signal over the entire trajectory ( e.g . Prashanth and Ghavamzadeh ( 2016 ) ) . This poses an issue , as Q-learning algorithms such as DQN ( Mnih et al. , 2015 ) do not learn a parametrization of the policy , and common Actor-Critic methods ( e.g . ( Schulman et al. , 2015a ; Mnih et al. , 2016 ; Schulman et al. , 2017 ) ) build the reward-to-go based on an N-step sample and a bootstrap update from the critic . In this paper , we propose the ‘ Reward Constrained Policy Optimization ’ ( RCPO ) algorithm . RCPO incorporates the constraint as a penalty signal into the reward function . This penalty signal guides the policy towards a constraint satisfying solution . We prove that RCPO converges almost surely , under mild assumptions , to a constraint satisfying solution ( Theorem 2 ) . In addition ; we show , empirically on a toy domain and six robotics domains , that RCPO results in a constraint satisfying solution while demonstrating faster convergence and improved stability ( compared to the standard constraint optimization methods ) . Related work : Constrained Markov Decision Processes ( Altman , 1999 ) are an active field of research . CMDP applications cover a vast number of topics , such as : electric grids ( Koutsopoulos and Tassiulas , 2011 ) , networking ( Hou and Zhao , 2017 ) , robotics ( Chow et al. , 2015 ; Gu et al. , 2017 ; Achiam et al. , 2017 ; Dalal et al. , 2018 ) and finance ( Krokhmal et al. , 2002 ; Tamar et al. , 2012 ) . The main approaches to solving such problems are ( i ) Lagrange multipliers ( Borkar , 2005 ; Bhatnagar and Lakshmanan , 2012 ) , ( ii ) Trust Region ( Achiam et al. , 2017 ) , ( iii ) integrating prior knowledge ( Dalal et al. , 2018 ) and ( iv ) manual selection of the penalty coefficient ( Tamar and Mannor , 2013 ; Levine and Koltun , 2013 ; Peng et al. , 2018 ) . Novelty : The novelty of our work lies in the ability to tackle ( 1 ) general constraints ( both discounted sum and mean value constraints ) , not only constraints which satisfy the recursive Bellman equation ( i.e , discounted sum constraints ) as in previous work . The algorithm is ( 2 ) reward agnostic . That is , invariant to scaling of the underlying reward signal , and ( 3 ) does not require the use of prior knowledge . A comparison with the different approaches is provided in Table 1 . 2 Preliminaries . 2.1 Markov Decision Process ( MDP ) . A Markov Decision ProcessesM is defined by the tuple ( S , A , R , P , µ , γ ) ( Sutton and Barto , 1998 ) . Where S is the set of states , A the available actions , R : S × A × S 7→ R is the reward function , P : S × A × S 7→ [ 0 , 1 ] is the transition matrix , where P ( s′|s , a ) is the probability of transitioning from state s to s′ assuming action a was taken , µ : S 7→ [ 0 , 1 ] is the initial state distribution and γ ∈ [ 0 , 1 ) is the discount factor for future rewards . A policy π : S 7→ ∆A is a probability distribution over actions and π ( a|s ) denotes the probability of taking action a at state s. For each state s , the value of following policy π is denoted by : V πR ( s ) = Eπ [ ∑ t γtr ( st , at ) |s0 = s ] . 1A mean valued constraint takes the form of E [ 1 T ∑T−1 t=0 ct ] ≤ α , as seen in Section 5.2 . 2Under the appropriate assumptions . 3Algorithms such as PPO are not intended to consider or satisfy constraints . An important property of the value function is that it solves the recursive Bellman equation : V πR ( s ) = Eπ [ r ( s , a ) + γV πR ( s′ ) |s ] . The goal is then to maximize the expectation of the reward-to-go , given the initial state distribution µ : max π∈Π JπR , where J π R = Eπs∼µ [ ∞∑ t=0 γtrt ] = ∑ s∈S µ ( s ) V πR ( s ) . ( 1 ) 2.2 Constrained MDPs . A Constrained Markov Decision Process ( CMDP ) extends the MDP framework by introducing a penalty c ( s , a ) , a constraint C ( st ) = F ( c ( st , at ) , ... , c ( sN , aN ) ) and a threshold α ∈ [ 0 , 1 ] . A constraint may be a discounted sum ( similar to the reward-to-go ) , the average sum and more ( see Altman ( 1999 ) for additional examples ) . Throughout the paper we will refer to the collection of these constraints as general constraints . We denote the expectation over the constraint by : JπC = Eπs∼µ [ C ( s ) ] . ( 2 ) The problem thus becomes : max π∈Π JπR , s.t . J π C ≤ α . ( 3 ) 2.3 Parametrized Policies . In this work we consider parametrized policies , such as neural networks . The parameters of the policy are denoted by θ and a parametrized policy as πθ . We make the following assumptions in order to ensure convergence to a constraint satisfying policy : Assumption 1 . The value V πR ( s ) is bounded for all policies π ∈ Π . Assumption 2 . Every local minima of JπθC is a feasible solution . Assumption 2 is the minimal requirement in order to ensure convergence , given a general constraint , of a gradient algorithm to a feasible solution . Stricter assumptions , such as convexity , may ensure convergence to the optimal solution ; however , in practice constraints are non-convex and such assumptions do not hold . 3 Constrained Policy Optimization . Constrained MDP ’ s are often solved using the Lagrange relaxation technique ( Bertesekas , 1999 ) . In Lagrange relaxation , the CMDP is converted into an equivalent unconstrained problem . In addition to the objective , a penalty term is added for infeasibility , thus making infeasible solutions sub-optimal . Given a CMDP ( 3 ) , the unconstrained problem is min λ≥0 max θ L ( λ , θ ) = min λ≥0 max θ [ JπθR − λ · ( J πθ C − α ) ] , ( 4 ) where L is the Lagrangian and λ ≥ 0 is the Lagrange multiplier ( a penalty coefficient ) . Notice , as λ increases , the solution to ( 4 ) converges to that of ( 3 ) . This suggests a twotimescale approach : on the faster timescale , θ is found by solving ( 4 ) , while on the slower timescale , λ is increased until the constraint is satisfied . The goal is to find a saddle point ( θ∗ ( λ∗ ) , λ∗ ) of ( 4 ) , which is a feasible solution . Definition 1 . A feasible solution of the CMDP is a solution which satisfies JπC ≤ α . 3.1 Estimating the gradient . We assume there isn ’ t access to the MDP itself , but rather samples are obtained via simulation . The simulation based algorithm for the constrained optimization problem ( 3 ) is : λk+1 = Γλ [ λk − η1 ( k ) ∇λL ( λk , θk ) ] , ( 5 ) θk+1 = Γθ [ θk + η2 ( k ) ∇θL ( λk , θk ) ] , ( 6 ) where Γθ is a projection operator , which keeps the iterate θk stable by projecting onto a compact and convex set . Γλ projects λ into the range [ 0 , λmax 4 ] . ∇θL and ∇λL are derived from ( 4 ) , where the formulation for ∇θL is derivied using the log-likelihood trick ( Williams , 1992 ) : ∇θL ( λ , θ ) = ∇θEπθs∼µ [ log π ( s , a ; θ ) [ R ( s ) − λ · C ( s ) ] ] , ( 7 ) ∇λL ( λ , θ ) = − ( Eπθs∼µ [ C ( s ) ] − α ) , ( 8 ) η1 ( k ) , η2 ( k ) are step-sizes which ensure that the policy update is performed on a faster timescale than that of the penalty coefficient λ . Assumption 3 . ∞∑ k=0 η1 ( k ) = ∞∑ k=0 η2 ( k ) =∞ , ∞∑ k=0 ( η1 ( k ) 2 + η2 ( k ) 2 ) < ∞ and η1 ( k ) η2 ( k ) → 0 . Theorem 1 . Under Assumption 3 , as well as the standard stability assumption for the iterates and bounded noise ( Borkar et al. , 2008 ) , the iterates ( θn , λn ) converge to a fixed point ( a local minima ) almost surely . Lemma 1 . Under assumptions 1 and 2 , the fixed point of Theorem 1 is a feasible solution . The proof to Theorem 1 is provided in Appendix C and to Lemma 1 in Appendix D .

Summary: The paper introduces RCPO, a model-free deep RL algorithm for learning optimal policies that satisfy some per-state constraint on expectation. The derivation of the algorithm is quite straightforward, starts from the definition of constrained optimization problem, and proceed by forming and optimizing the Lagrangian. Additionally, a value function for the constraint is learned. The algorithm is only compared to a baseline optimizing the Lagrangian directly using Monte-Carlo sampling.