Reward Constrained Policy Optimization

1 Introduction . Applying Reinforcement Learning ( RL ) is generally a hard problem . At each state , the agent performs an action which produces a reward . The goal is to maximize the accumulated reward , hence the reward signal implicitly defines the behavior of the agent . While in computer games ( e.g . Bellemare et al . ( 2013 ) ) there exists a pre-defined reward signal , it is not such in many real applications . An example is the Mujoco domain ( Todorov et al. , 2012 ) , in which the goal is to learn to control robotic agents in tasks such as : standing up , walking , navigation and more . Considering the Humanoid domain , the agent is a 3 dimensional humanoid and the task is to walk forward as far as possible ( without falling down ) within a fixed amount of time . Naturally , a reward is provided based on the forward velocity in order to encourage a larger distance ; however , additional reward signals are provided in order to guide the agent , for instance a bonus for staying alive , a penalty for energy usage and a penalty based on the force of impact between the feet and the floor ( which should encourage less erratic behavior ) . Each signal is multiplied by it ’ s own coefficient , which controls the emphasis placed on it . This approach is a multi-objective problem ( Mannor and Shimkin , 2004 ) ; in which for each set of penalty coefficients , there exists a different , optimal solution , also known as Pareto optimality ( Van Moffaert and Nowé , 2014 ) . In practice , the exact coefficient is selected through a time consuming and a computationally intensive process of hyper-parameter tuning . As our experiments show , the coefficient is not shared across domains , a coefficient which leads to a satisfying behavior on one domain may lead to catastrophic failure on the other ( issues also seen in Leike et al . ( 2017 ) and Mania et al . ( 2018 ) ) . Constraints are a natural and consistent approach , an approach which ensures a satisfying behavior without the need for manually selecting the penalty coefficients . In constrained optimization , the task is to maximize a target function f ( x ) while satisfying an inequality constraint g ( x ) ≤ α . While constraints are a promising solution to ensuring a satisfying behavior , existing methods are limited in the type of constraints they are able to handle and the algorithms that they may support - they require a parametrization of the policy ( policy gradient methods ) and propagation of the constraint violation signal over the entire trajectory ( e.g . Prashanth and Ghavamzadeh ( 2016 ) ) . This poses an issue , as Q-learning algorithms such as DQN ( Mnih et al. , 2015 ) do not learn a parametrization of the policy , and common Actor-Critic methods ( e.g . ( Schulman et al. , 2015a ; Mnih et al. , 2016 ; Schulman et al. , 2017 ) ) build the reward-to-go based on an N-step sample and a bootstrap update from the critic . In this paper , we propose the ‘ Reward Constrained Policy Optimization ’ ( RCPO ) algorithm . RCPO incorporates the constraint as a penalty signal into the reward function . This penalty signal guides the policy towards a constraint satisfying solution . We prove that RCPO converges almost surely , under mild assumptions , to a constraint satisfying solution ( Theorem 2 ) . In addition ; we show , empirically on a toy domain and six robotics domains , that RCPO results in a constraint satisfying solution while demonstrating faster convergence and improved stability ( compared to the standard constraint optimization methods ) . Related work : Constrained Markov Decision Processes ( Altman , 1999 ) are an active field of research . CMDP applications cover a vast number of topics , such as : electric grids ( Koutsopoulos and Tassiulas , 2011 ) , networking ( Hou and Zhao , 2017 ) , robotics ( Chow et al. , 2015 ; Gu et al. , 2017 ; Achiam et al. , 2017 ; Dalal et al. , 2018 ) and finance ( Krokhmal et al. , 2002 ; Tamar et al. , 2012 ) . The main approaches to solving such problems are ( i ) Lagrange multipliers ( Borkar , 2005 ; Bhatnagar and Lakshmanan , 2012 ) , ( ii ) Trust Region ( Achiam et al. , 2017 ) , ( iii ) integrating prior knowledge ( Dalal et al. , 2018 ) and ( iv ) manual selection of the penalty coefficient ( Tamar and Mannor , 2013 ; Levine and Koltun , 2013 ; Peng et al. , 2018 ) . Novelty : The novelty of our work lies in the ability to tackle ( 1 ) general constraints ( both discounted sum and mean value constraints ) , not only constraints which satisfy the recursive Bellman equation ( i.e , discounted sum constraints ) as in previous work . The algorithm is ( 2 ) reward agnostic . That is , invariant to scaling of the underlying reward signal , and ( 3 ) does not require the use of prior knowledge . A comparison with the different approaches is provided in Table 1 . 2 Preliminaries . 2.1 Markov Decision Process ( MDP ) . A Markov Decision ProcessesM is defined by the tuple ( S , A , R , P , µ , γ ) ( Sutton and Barto , 1998 ) . Where S is the set of states , A the available actions , R : S × A × S 7→ R is the reward function , P : S × A × S 7→ [ 0 , 1 ] is the transition matrix , where P ( s′|s , a ) is the probability of transitioning from state s to s′ assuming action a was taken , µ : S 7→ [ 0 , 1 ] is the initial state distribution and γ ∈ [ 0 , 1 ) is the discount factor for future rewards . A policy π : S 7→ ∆A is a probability distribution over actions and π ( a|s ) denotes the probability of taking action a at state s. For each state s , the value of following policy π is denoted by : V πR ( s ) = Eπ [ ∑ t γtr ( st , at ) |s0 = s ] . 1A mean valued constraint takes the form of E [ 1 T ∑T−1 t=0 ct ] ≤ α , as seen in Section 5.2 . 2Under the appropriate assumptions . 3Algorithms such as PPO are not intended to consider or satisfy constraints . An important property of the value function is that it solves the recursive Bellman equation : V πR ( s ) = Eπ [ r ( s , a ) + γV πR ( s′ ) |s ] . The goal is then to maximize the expectation of the reward-to-go , given the initial state distribution µ : max π∈Π JπR , where J π R = Eπs∼µ [ ∞∑ t=0 γtrt ] = ∑ s∈S µ ( s ) V πR ( s ) . ( 1 ) 2.2 Constrained MDPs . A Constrained Markov Decision Process ( CMDP ) extends the MDP framework by introducing a penalty c ( s , a ) , a constraint C ( st ) = F ( c ( st , at ) , ... , c ( sN , aN ) ) and a threshold α ∈ [ 0 , 1 ] . A constraint may be a discounted sum ( similar to the reward-to-go ) , the average sum and more ( see Altman ( 1999 ) for additional examples ) . Throughout the paper we will refer to the collection of these constraints as general constraints . We denote the expectation over the constraint by : JπC = Eπs∼µ [ C ( s ) ] . ( 2 ) The problem thus becomes : max π∈Π JπR , s.t . J π C ≤ α . ( 3 ) 2.3 Parametrized Policies . In this work we consider parametrized policies , such as neural networks . The parameters of the policy are denoted by θ and a parametrized policy as πθ . We make the following assumptions in order to ensure convergence to a constraint satisfying policy : Assumption 1 . The value V πR ( s ) is bounded for all policies π ∈ Π . Assumption 2 . Every local minima of JπθC is a feasible solution . Assumption 2 is the minimal requirement in order to ensure convergence , given a general constraint , of a gradient algorithm to a feasible solution . Stricter assumptions , such as convexity , may ensure convergence to the optimal solution ; however , in practice constraints are non-convex and such assumptions do not hold . 3 Constrained Policy Optimization . Constrained MDP ’ s are often solved using the Lagrange relaxation technique ( Bertesekas , 1999 ) . In Lagrange relaxation , the CMDP is converted into an equivalent unconstrained problem . In addition to the objective , a penalty term is added for infeasibility , thus making infeasible solutions sub-optimal . Given a CMDP ( 3 ) , the unconstrained problem is min λ≥0 max θ L ( λ , θ ) = min λ≥0 max θ [ JπθR − λ · ( J πθ C − α ) ] , ( 4 ) where L is the Lagrangian and λ ≥ 0 is the Lagrange multiplier ( a penalty coefficient ) . Notice , as λ increases , the solution to ( 4 ) converges to that of ( 3 ) . This suggests a twotimescale approach : on the faster timescale , θ is found by solving ( 4 ) , while on the slower timescale , λ is increased until the constraint is satisfied . The goal is to find a saddle point ( θ∗ ( λ∗ ) , λ∗ ) of ( 4 ) , which is a feasible solution . Definition 1 . A feasible solution of the CMDP is a solution which satisfies JπC ≤ α . 3.1 Estimating the gradient . We assume there isn ’ t access to the MDP itself , but rather samples are obtained via simulation . The simulation based algorithm for the constrained optimization problem ( 3 ) is : λk+1 = Γλ [ λk − η1 ( k ) ∇λL ( λk , θk ) ] , ( 5 ) θk+1 = Γθ [ θk + η2 ( k ) ∇θL ( λk , θk ) ] , ( 6 ) where Γθ is a projection operator , which keeps the iterate θk stable by projecting onto a compact and convex set . Γλ projects λ into the range [ 0 , λmax 4 ] . ∇θL and ∇λL are derived from ( 4 ) , where the formulation for ∇θL is derivied using the log-likelihood trick ( Williams , 1992 ) : ∇θL ( λ , θ ) = ∇θEπθs∼µ [ log π ( s , a ; θ ) [ R ( s ) − λ · C ( s ) ] ] , ( 7 ) ∇λL ( λ , θ ) = − ( Eπθs∼µ [ C ( s ) ] − α ) , ( 8 ) η1 ( k ) , η2 ( k ) are step-sizes which ensure that the policy update is performed on a faster timescale than that of the penalty coefficient λ . Assumption 3 . ∞∑ k=0 η1 ( k ) = ∞∑ k=0 η2 ( k ) =∞ , ∞∑ k=0 ( η1 ( k ) 2 + η2 ( k ) 2 ) < ∞ and η1 ( k ) η2 ( k ) → 0 . Theorem 1 . Under Assumption 3 , as well as the standard stability assumption for the iterates and bounded noise ( Borkar et al. , 2008 ) , the iterates ( θn , λn ) converge to a fixed point ( a local minima ) almost surely . Lemma 1 . Under assumptions 1 and 2 , the fixed point of Theorem 1 is a feasible solution . The proof to Theorem 1 is provided in Appendix C and to Lemma 1 in Appendix D .

Summary: This work tackles the difficult problem of solving Constrained Markov Decision Processes. It proposes the RCPO algorithm as a way to solving CMDP. The benefits of RCPO is that it can handle general constraints, it is reward agnostic and doesn't require prior knowledge. The key is that RCPO trains the actor and critic using an alternative penalty guiding signal.