Balancing Constraints and Rewards with Meta-Gradient D4PG

1 INTRODUCTION . Reinforcement Learning ( RL ) algorithms typically try to maximize an expected return objective ( Sutton & Barto , 2018 ) . This approach has led to numerous successes in a variety of domains which include board-games ( Silver et al. , 2017 ) , computer games ( Mnih et al. , 2015 ; Tessler et al. , 2017 ) and robotics ( Abdolmaleki et al. , 2018 ) . However , formulating real-world problems with only an expected return objective is often sub-optimal when tackling many applied problems ranging from recommendation systems to physical control systems which may include robots , self-driving cars and even aerospace technologies . In many of these domains there are a variety of challenges preventing RL from being utilized as the algorithmic solution framework . Recently , Dulac-Arnold et al . ( 2019 ) presented nine challenges that need to be solved to enable RL algorithms to be utilized in real-world products and systems . One of those challenges is handling constraints . All of the above domains may include one or more constraints related to cost , wear-and-tear , or safety , to name a few . Hard and Soft Constraints : There are two types of constraints that are encountered in constrained optimization problems ; namely hard-constraints and soft-constraints ( Boyd & Vandenberghe , 2004 ) . Hard constraints are pairs of pre-specified functions and thresholds that require the functions , when evaluated on the solution , to respect the thresholds . As such , these constraints may limit the feasible solution set . Soft constraints are similar to hard constraints in the sense that they are defined by pairs of pre-specified functions and thresholds , however , a soft constraint does not require the solution to hold the constraint ; instead , it penalizes the objective function ( according to a specified rule ) if the solution violates the constraint ( Boyd & Vandenberghe , 2004 ; Thomas et al. , 2017 ) . Motivating Soft-Constraints : In real-world products and systems , there are many examples of soft-constraints ; that is , constraints that can be violated , where the violated behaviour is undesirable but not catastrophic ( Thomas et al. , 2017 ; Dulac-Arnold et al. , 2020b ) . One concrete example is that of energy minimization in physical control systems . Here , the system may wish to reduce the amount of energy used by setting a soft-constraint . Violating the constraint is inefficient , but not catastrophic to the system completing the task . In fact , there may be desirable characteristics that can only be attained if there are some constraint violations ( e.g. , a smoother/faster control policy ) . Another common setting is where it is unclear how to set a threshold . In many instances , a product * indicates equal contribution . manager may desire to increase the level of performance on a particular product metric A , while ensuring that another metric B on the same product does not drop by ‘ approximately X % ’ . The value ‘ X ’ is often inaccurate and may not be feasible in many cases . In both of these settings , violating the threshold is undesirable , yet does not have catastrophic consequences . Lagrange Optimization : In the RL paradigm , a number of approaches have been developed to incorporate hard constraints into the overall problem formulation ( Altman , 1999 ; Tessler et al. , 2018 ; Efroni et al. , 2020 ; Achiam et al. , 2017 ; Bohez et al. , 2019 ; Chow et al. , 2018 ; Paternain et al. , 2019 ; Zhang et al. , 2020 ; Efroni et al. , 2020 ) . One popular approach is to model the problem as a Constrained Markov Decision Process ( CMDP ) ( Altman , 1999 ) . In this case , one method is to solve the following problem formulation : max⇡ J⇡R s.t . J ⇡ C , where ⇡ is a policy , J⇡ R is the expected return , J⇡ C is the expected cost and is a constraint violation threshold . This is often solved by performing alternating optimization on the unconstrained Lagrangian relaxation of the original problem ( e.g . Tessler et al . ( 2018 ) ) , defined as : min 0 max⇡ J⇡R+ ( J⇡C ) . The updates alternate between learning the policy and the Lagrange multiplier . In many previous constrained RL works ( Achiam et al. , 2017 ; Tessler et al. , 2018 ; Ray et al. , 2019 ; Satija et al. , 2020 ) , because the problem is formulated with hard constraints , there are some domains in each case where a feasible solution is not found . This could be due to approximation errors , noise , or the constraints themselves being infeasible . The real-world applications , along with empirical constrained RL research results , further motivates the need to develop a soft-constrained RL optimization approach . Ideally , in this setup , we would like an algorithm that satisfies the constraints while solving the task by maximizing the objective . If the constraints can not be satisfied , then this algorithm finds a good trade-off ( that is , minimizing constraint violations while solving the task by maximizing the objective ) . In this paper , we extend the constrained RL Lagrange formulation to perform soft-constrained optimization by formulating the constrained RL objective as a nested optimization problem ( Sinha et al. , 2017 ) using meta-gradients . We propose MetaL that utilizes meta-gradients ( Xu et al. , 2018 ; Zahavy et al. , 2020 ) to improve upon the trade-off between reducing constraint violations and improving expected return . We focus on Distributed Distributional Deterministic Policy Gradients ( D4PG ) ( Barth-Maron et al. , 2018 ) as the underlying algorithmic framework , a state-of-the-art continuous control RL algorithm . We show that MetaL can capture an improved trade-off between expected return and constraint violations compared to the baseline approaches . We also introduce a second approach called MeSh that utilizes meta-gradients by adding additional representation power to the reward shaping function . Our main contributions are as follows : ( 1 ) We extend D4PG to handle constraints by adapting it to Reward Constrained Policy Optimization ( RCPO ) ( Tessler et al. , 2018 ) yielding Reward Constrained D4PG ( RC-D4PG ) ; ( 2 ) We present a soft constrained meta-gradient technique : Meta-Gradients for the Lagrange multiplier learning rate ( MetaL ) 1 ; ( 3 ) We derive the meta-gradient update for MetaL ( Theorem 1 ) ; ( 4 ) We perform extensive experiments and investigative studies to showcase the properties of this algorithm . MetaL outperforms the baseline algorithms across domains , safety coefficients and thresholds from the Real World RL suite ( Dulac-Arnold et al. , 2020b ) . 2 BACKGROUND . A Constrained Markov Decision Process ( CMDP ) is an extension to an MDP ( Sutton & Barto , 2018 ) and consists of the tuple hS , A , P , R , C , i where S is the state space ; A is the action space ; P : S ⇥ A ! S is a function mapping states and actions to a distribution over next states ; R : S ⇥ A ! R is a bounded reward function and C : S ⇥ A ! RK is a K dimensional function representing immediate penalties ( or costs ) relating to K constraints . The solution to a CMDP is a policy ⇡ : S ! A which is a mapping from states to a probability distribution over actions . This policy aims to maximize the expected return J⇡ R = E [ P1 t=0 trt ] and satisfy the constraints J⇡ Ci = E [ P1 t=0 tci , t ] i , i = 1 . . .K . For the purpose of the paper , we consider a single constraint ; that is , K = 1 , but this can easily be extended to multiple constraints . Meta-Gradients is an approach to optimizing hyperparameters such as the discount factor , learning rates , etc . by performing online cross validation while simultaneously optimizing for the overall RL optimization objective such as the expected return ( Xu et al. , 2018 ; Zahavy et al. , 2020 ) . The goal is to optimize both an inner loss and an outer loss . The update of the ✓ parameters on the inner 1This is also the first time meta-gradients have been applied to an algorithm with an experience replay . loss is defined as ✓0 = ✓ + f ( ⌧ , ✓ , ⌘ ) , where ✓ 2 Rd corresponds to the parameters of the policy ⇡✓ ( a|s ) and the value function v✓ ( s ) ( if applicable ) . The function f : Rk ! Rd is the gradient of the policy and/or value function with respect to the parameters ✓ and is a function of an n-step trajectory ⌧ = hs1 , a1 , r2 , s2 . . . sni , meta-parameters ⌘ and is weighted by a learning rate ↵ and is defined as f ( ⌧ , ✓ , ⌘ ) = ↵ dJ ⇡✓ obj ( ✓ , ⌧ , ⌘ ) d✓ where J ⇡✓ obj ( ✓ , ⌧ , ⌘ ) is the objective being optimized with respect to ✓ . The idea is to then evaluate the performance of this new parameter value ✓0 on an outer loss – the meta-gradient objective . We define this objective as J 0 ( ⌧ 0 , ✓0 , ⌘̄ ) where ⌧ 0 is a new trajectory , ✓0 are the updated parameters and ⌘̄ is a fixed meta-parameter ( which needs to be selected/tuned in practice ) . We then need to take the gradient of the objective J 0 with respect to the meta-parameters ⌘ to yield the outer loss update ⌘0 = ⌘ + ↵⌘ @ J 0 ( ⌧ 0 , ✓0 , ⌘̄ ) @ ⌘ . This gradient is computed as follows : @ J 0 ( ⌧ 0 , ✓0 , ⌘̄ ) @ ⌘ = @ J 0 ( ⌧ 0 , ✓0 , ⌘̄ ) @ ✓0 @ ✓ 0 @ ⌘ . The outer loss is essentially the objective we are trying to optimize . This could be a policy gradient loss , a temporal difference loss , a combination of the two etc ( Xu et al. , 2018 ; Zahavy et al. , 2020 ) . Meta-gradients have been previously used to learn intrinsic rewards for policy gradient ( Zheng et al. , 2018 ) and auxiliary tasks ( Veeriah et al. , 2019 ) . Meta-gradients have also been used to adapt optimizer parameters ( Young et al. , 2018 ; Franceschi et al. , 2017 ) . In our setup , we consider the continuous control setting , provide the first implementation of metagradients for an algorithm that uses an experience replay , and focus on adapting meta-parameters that encourage soft constraint satisfaction while maximizing expected return . D4PG is a state-of-the-art continuous control RL algorithm with a deterministic policy ( BarthMaron et al. , 2018 ) . It is an incremental improvement to DDPG ( Lillicrap et al. , 2015 ) . The overall objective of DDPG is to maximize J ( ✓a , ✓c ) = E [ Q✓c ( s , a ) |s = st , a = ⇡✓a ( st ) ] where ⇡✓a ( st ) is a deterministic policy with parameters ✓a and Q✓c ( s , a ) is an action value function with parameters ✓c . The actor loss is defined as : Lactor = kSG ( raQ✓c ( st , at ) |at=⇡✓a ( s ) + a✓a , t ) a✓a , tk2 where SG is a stop gradient . The corresponding gradient update is defined as r✓aJ ( ✓a ) = E [ raQ✓c ( s , a ) r✓a⇡✓a ( st ) ] . The critic is updated using the standard temporal difference error loss : Lcritic = ( r ( s , a ) + QT ( s0 , ⇡T ( s0 ) ) Q✓c ( s , a ) ) 2 where QT , ⇡T are the target critic and actor networks respectively . In D4PG , the critic is a distributional critic based on the C51 algorithm ( Bellemare et al. , 2017 ) and the agent is run in a distributed setup with multiple actors executed in parallel , n-step returns and with prioritized experience replay . We will use the non-distributional critic update in our notation for ease of visualization and clarity for the reader2 .

Summary: The paper focuses on soft-constrained RL techniques and proposes a meta-gradient approach for the same. It first extends the RCPO (Tessler et al)  algorithm using the methodology of DDPG (Lillicarp et al) to propose an off-policy version of RCPO (called RC-D4PG). The main contribution of the work is the proposal of two new meta-gradients based algorithms for the soft-constrained RL problem that are able to find a good trade-off between constraint violation and maximizing returns. The first proposed algorithm - Meta-L - is based on a meta-learning based adaptive update rule for the Lagrange multiplier's learning rate. The second algorithm is based on similar principles but instead focuses on adapting the reward-shaping update in a meta manner. The author's show the empirical evidence of their method's strengths on a bunch of continuous control based simulator tasks.