Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecification

1 INTRODUCTION . Reinforcement Learning ( RL ) has had a number of recent successes in various application domains which include computer games ( Silver et al. , 2017 ; Mnih et al. , 2015 ; Tessler et al. , 2017 ) and robotics ( Abdolmaleki et al. , 2018a ) . As RL and deep learning continue to scale , an increasing number of real-world applications may become viable candidates to take advantage of this technology . However , the application of RL to real-world systems is often associated with a number of challenges ( Dulac-Arnold et al. , 2019 ; Dulac-Arnold et al. , 2020 ) . We will focus on the following two : Challenge 1 - Constraint satisfaction : One such challenge is that many real-world systems have constraints that need to be satisfied upon deployment ( i.e. , hard constraints ) ; or at least the number of constraint violations as defined by the system need to be reduced as much as possible ( i.e. , soft-constraints ) . This is prevalent in applications ranging from physical control systems such as autonomous driving and robotics to user facing applications such as recommender systems . Challenge 2 - Model Misspecification ( MM ) : Many of these systems suffer from another challenge : model misspecification . We refer to the situation in which an agent is trained in one environment but deployed in a different , perturbed version of the environment as an instance of model misspecification . This may occur in many different applications and is well-motivated in the literature ( Mankowitz et al. , 2018 ; 2019 ; Derman et al. , 2018 ; 2019 ; Iyengar , 2005 ; Tamar et al. , 2014 ) . There has been much work on constrained optimization in the literature ( Altman , 1999 ; Tessler et al. , 2018 ; Efroni et al. , 2020 ; Achiam et al. , 2017 ; Bohez et al. , 2019 ) . However , to our knowledge , the effect of model misspecification on an agent ’ s ability to satisfy constraints at test time has not yet been investigated . ⇤indicates equal contribution . Constrained Model Misspecification ( CMM ) : We consider the scenario in which an agent is required to satisfy constraints at test time but is deployed in an environment that is different from its training environment ( i.e. , a perturbed version of the training environment ) . Deployment in a perturbed version of the environment may affect the return achieved by the agent as well as its ability to satisfy the constraints . We refer to this scenario as constrained model misspecification . This problem is prevalent in many real-world applications where constraints need to be satisfied but the environment is subject to state perturbations effects such as wear-and-tear , partial observability etc. , the exact nature of which may be unknown at training time . Since such perturbations can significantly impact the agent ’ s ability to satisfy the required constraints it is insufficient to simply ensure that constraints are satisfied in the unperturbed version of the environment . Instead , the presence of unknown environment variations needs to be factored into the training process . One area where such considerations are of particular practical relevance is sim2real transfer where the unknown sim2real gap can make it hard to ensure that constraints will be satisfied on the real system ( Andrychowicz et al. , 2018 ; Peng et al. , 2018 ; Wulfmeier et al. , 2017 ; Rastogi et al. , 2018 ; Christiano et al. , 2016 ) . Of course , one could address this issue by limiting the capabilities of the system being controlled in order to ensure that constraints are never violated , for instance by limiting the amount of current in an electric motor . Our hope is that our methods can outperform these more blunt techniques , while still ensuring constraint satisfaction in the deployment domain . Main Contributions : In this paper , we aim to bridge the two worlds of model misspecification and constraint satisfaction . We present an RL objective that enables us to optimize a policy that aims to be robust to CMM . Our contributions are as follows : ( 1 ) Introducing the Robust Return Robust Constraint ( R3C ) and Robust Constraint ( RC ) RL objectives that aim to mitigate CMM as defined above . This includes the definition of a Robust Constrained Markov Decision Process ( RC-MDP ) . ( 2 ) Derive corresponding R3C and RC value functions and Bellman operators . Provide an argument showing that these Bellman operators converge to fixed points . These are implemented in the policy evaluation step of actor-critic R3C algorithms . ( 3 ) Implement five different R3C and RC algorithmic variants on top of D4PG and DMPO , ( two state-of-the-art continuous control RL algorithms ) . ( 4 ) Empirically demonstrate the superior performance of our algorithms , compared to various baselines , with respect to mitigating CMM . This is shown consistently across 6 different Mujoco tasks from the Real-World RL ( RWRL ) suite1 . 2 BACKGROUND . 2.1 MARKOV DECISION PROCESSES . A Robust Markov Decision Process ( R-MDP ) is defined as a tuple hS , A , R , , Pi where S is a finite set of states , A is a finite set of actions , R : S ⇥ A ! R is a bounded reward function and 2 [ 0 , 1 ) is the discount factor ; P ( s , a ) ✓ M ( S ) is an uncertainty set where M ( S ) is the set of probability measures over next states s0 2 S. This is interpreted as an agent selecting a state and action pair , and the next state s0 is determined by a conditional measure p ( s0|s , a ) 2 P ( s , a ) ( Iyengar , 2005 ) . We want the agent to learn a policy ⇡ : S ! A , which is a mapping from states to actions that is robust with respect to this uncertainty set . For the purpose of this paper , we consider deterministic policies , but this can easily be extended to stochastic policies too . The robust value function V ⇡ : S ! R for a policy ⇡ is defined as V ⇡ ( s ) = infp2P ( s , ⇡ ( s ) ) V ⇡ , p ( s ) where V ⇡ , p ( s ) = r ( s , ⇡ ( s ) ) + p ( s0|s , ⇡ ( s ) ) V ⇡ , p ( s0 ) . A rectangularity assumption on the uncertainty set ( Iyengar , 2005 ) ensures that “ nature ” can choose a worst-case transition function independently for every state s and action a . This means that during a trajectory , at each timestep , nature can choose any transition model from the uncertainty set to reduce the performance of the agent . A robust policy optimizes for the robust ( worst-case ) expected return objective : JR ( ⇡ ) = infp2P Ep , ⇡ [ P1 t=0 trt ] . The robust value function can be expanded as V ⇡ ( s ) = r ( s , ⇡ ( s ) ) + infp2P ( s , ⇡ ( s ) ) Ep [ V ⇡ ( s0 ) |s , ⇡ ( s ) ] . As in ( Tamar et al. , 2014 ) , we can define an operator infP ( s , a ) v : R |S| ! R as infP ( s , a ) v = inf { p > v|p 2 P ( s , a ) } . We can also define an operator for some policy ⇡ as inf⇡ : R|S| ! R|S| where { inf⇡ v } ( s ) = inf P ( s , ⇡ ( s ) ) v. Then , we have defined the Robust Bellman 1https : //github.com/google-research/realworldrl_suite operator as follows T⇡RV ⇡ = r⇡ + inf⇡ V ⇡ . Both the robust Bellman operator T⇡R : R|S| ! R|S| for a fixed policy and the optimal robust Bellman operator T ⇤Rv ( s ) = max⇡ T ⇡ Rv ( s ) have previously been shown to be contractions ( Iyengar , 2005 ) . A Constrained Markov Decision Process ( CMDP ) is an extension to an MDP and consists of the tuple hS , A , P , R , C , i where S , A , R and are defined as in the MDP above and C : S⇥A ! RK is a mapping from a state s and action a to a K dimensional vector representing immediate costs relating to K constraint . We use K=1 from here on in and therefore C : S⇥A ! R. We refer to the cost for a specific state action tuple hs , ai at time t as ct ( s , a ) . The solution to a CMDP is a policy ⇡ : S ! A that learns to maximize return and satisfy the constraints . The agent aims to learn a policy that maximizes the expected return objective J⇡R = E [ P1 t=0 trt ] subject to J⇡C = E [ P1 t=0 tct ] where is a pre-defined constraint threshold . A number of approaches ( Tessler et al. , 2018 ; Bohez et al. , 2019 ) optimize the Lagrange relaxation of this objective min 0 max✓ J⇡R ( J⇡C ) by optimizing the Lagrange multiplier and the policy parameters ✓ using alternating optimization . We also define the constraint value function V ⇡ , pC : S ! R for a policy ⇡ as in ( Tessler et al. , 2018 ) where V ⇡ , pC ( s ) = c ( s , ⇡ ( s ) ) + p ( s 0|s , ⇡ ( s ) ) V ⇡ , pC ( s0 ) . 2.2 CONTINUOUS CONTROL RL ALGORITHMS . We address the CMM problem by modifying two well-known continuous control algorithms by having them optimize the RC and R3C objectives . The first algorithm is Maximum A-Posteriori Policy Optimization ( MPO ) . This is a continuous control RL algorithm that performs policy iteration using an RL form of expectation maximization ( Abdolmaleki et al. , 2018a ; b ) . We use the distributional-critic version in Abdolmaleki et al . ( 2020 ) , which we refer to as DMPO . The second algorithm is Distributed Distributional Deterministic Policy Gradient ( D4PG ) , which is a state-of-the-art actor-critic continuous control RL algorithm with a deterministic policy ( BarthMaron et al. , 2018 ) . It is an incremental improvement to DDPG ( Lillicrap et al. , 2015 ) with a distributional critic that is learned similarly to distributional MPO . 3 ROBUST CONSTRAINED ( RC ) OPTIMIZATION OBJECTIVE . We begin by defining a Robust Constrained MDP ( RC-MDP ) . This combines an R-MDP and C-MDP to yield the tuple hS , A , R , C , , Pi where all of the variables in the tuple are defined in Section 2 . We next define two optimization objectives that optimize the RC-MDP . The first variant attempts to learn a policy that is robust with respect to the return as well as constraint satisfaction - Robust Return Robust Constrained ( R3C ) objective . The second variant is only robust with respect to constraint satisfaction - Robust Constrained ( RC ) objective . Prior to defining these objectives , we add some important definitions . Definition 1 . The robust constrained value function V ⇡C : S ! R for a policy ⇡ is defined as V ⇡C ( s ) = supp2P ( s , ⇡ ( s ) ) V ⇡ , p C ( s ) = supp2P ( s , ⇡ ( s ) ) E⇡ , p P1 t=0 tct . This value function represents the worst-case sum of constraint penalties over the course of an episode with respect to the uncertainty set P ( s , a ) . We can also define an operator supP ( s , a ) v : R |S| ! R as supP ( s , a ) v = sup { p > v|p 2 P ( s , a ) } . In addition , we can define an operator on vectors for some policy ⇡ as sup⇡ : R|S| ! R|S| where { sup⇡ v } ( s ) = sup P ( s , ⇡ ( s ) ) v. Then , we can defined the Supremum Bellman operator T⇡sup : R|S| ! R|S| as follows T⇡supV ⇡ = r⇡ + sup⇡ V ⇡ . Note that this operator is a contraction since we get the same result if we replace T⇡inf with T ⇡ sup and replace V with V . An alternative derivation of the sup operator contraction has also been derived in the Appendix , Section A.3 for completeness . 3.0.1 ROBUST RETURN ROBUST CONSTRAINT ( R3C ) OBJECTIVE The R3C objective is defined as : max ⇡2⇧ inf p2P Ep , ⇡ X t tr ( st , at ) s.t . sup p02P Ep 0 , ⇡ X t tc ( st , at ) ( 1 ) Note , a couple of interesting properties about this objective : ( 1 ) it focuses on being robust with respect to the return for a pre-defined set of perturbations ; ( 2 ) the objective also attempts to be robust with respect to the worst case constraint value for the perturbation set . The Lagrange relaxation form of equation 1 is used to define an R3C value function . Definition 2 ( R3C Value Function ) . For a fixed , and using the above-mentioned rectangularity assumption ( Iyengar , 2005 ) , the R3C value function for a policy ⇡ is defined as the concatenation of two value functions V⇡ = f ( hV ⇡ , V ⇡C i ) = V ⇡ V ⇡C . This implies that we keep two separate estimates of V ⇡ and V ⇡C and combine them together to yield V ⇡ . The constraint threshold term offsets the value function , and has no effect on any policy improvement step2 . As a result , the dependency on is dropped . The next step is to define the R3C Bellman operator . This is presented in Definition 3 . Definition 3 ( R3C Bellman operator ) . The R3C Bellman operator is defined as two separate Bellman operators T⇡R3C = hT⇡inf , T⇡supi where T⇡inf is the robust Bellman operator ( Iyengar , 2005 ) and T⇡sup : R|S| ! R|S| is defined as the sup Bellman operator . Based on this definition , applying the R3C Bellman operator to V⇡ involves applying each of the Bellman operators to their respective value functions . That is , T⇡R3CV = T ⇡ infV T⇡supVC . It has been previously shown that T⇡inf is a contraction with respect to the max norm ( Tamar et al. , 2014 ) and therefore converges to a fixed point . We also provided an argument whereby T⇡sup is a contraction operator in the previous section as well as in Appendix , A.3 . These Bellman operators individually ensure that the robust value function V ( s ) and the constraint value function VC ( s ) converge to fixed points . Therefore , T ⇡R3CV also converges to a fixed point by construction . As a result of the above argument , we know that we can apply the R3C Bellman operator in value iteration or policy iteration algorithms in the policy evaluation step . This is achieved in practice by simultaneously learning both the robust value function V ⇡ ( s ) and the constraint value function V ⇡C ( s ) and combining these estimates to yield V ⇡ ( s ) .

Summary: The standard Reinforcement Learning framework is limited in many ways, and numerous variants have been introduced to deal with aspects such as partial observability, temporal abstraction, safety, domain transfer, etc. Yet, these issues are often studied separately and it is often unclear how to combine them together. This is the ambitious challenge taken by this paper, which attempts to bridge the two separate settings of Robust MDPs, which aim at considering ambiguity in the dynamics, and Constrained MDPs, which aim at enforcing the satisfaction of a constraint on an expected cost signal. The authors propose the formulation of two objectives, that merge the two aspects and include both a worst-case evaluation over the ambiguity set and a constraint violation penalty term. The ways of dealing with both issues are fairly standard (Lagrangian relaxation of the constraints with alternating optimization, and worst-case evaluation over a finite set of simulated transitions in practice), but their combination seems novel and relevant. These objectives come with the corresponding Bellman Expectation operators, which allow to evaluate the current policy (critic) and provide a feedback (gradient) for the actor to ensure robust constraint satisfaction. The applicability of the proposed approaches is demonstrated on a benchmark of Mujoco tasks, where they are shown to compare favorably to several baselines.