Constrained Markov Decision Processes via Backward Value Functions

1 INTRODUCTION . Reinforcement Learning ( RL ) provides a sound decision-theoretic framework to optimize the behavior of learning agents in an interactive setting ( Sutton & Barto , 2018 ) . Recently , the field of RL has found success in many high-dimensional domains , like video games , Go , robot locomotion and navigation . However , most of the success of RL algorithms has been limited to simulators , where the learning algorithm has the ability to reset the simulator . In the physical world , an agent will need to avoid harmful behavior ( e.g . damaging the environment or the agent ’ s hardware ) while learning to explore behaviors that maximize the reward . A few popular approaches for avoiding undesired behaviors for high-dimensional systems include reward-shaping ( Moldovan & Abbeel , 2012 ) , reachability-preserving algorithms ( Mitchell , 2003 ; Eysenbach et al. , 2017 ) , state-level surrogate constraint satisfaction algorithms ( Dalal et al. , 2018 ) , risk-sensitive algorithms ( Tamar et al. , 2013 ; Chow et al. , 2015 ) and apprenticeship learning ( Abbeel & Ng , 2004 ) . There also exists model-based Bayesian approaches that are focused on imposing the constraints via the dynamics ( such as classifying parts of state space as unsafe ) and then using model predictive control to incorporate the constraints in the policy optimization and planning ( Turchetta et al. , 2016 ; Berkenkamp et al. , 2017 ; Wachi et al. , 2018 ; Koller et al. , 2018 ) . A natural way to model safety is via constraint satisfaction . A standard formulation for adding constraints to RL problems is the Constrained Markov Decision Process ( CMDP ) framework ( Altman , 1999 ) , wherein the environment is extended to also provide feedback on constraint costs . The agent must then attempt to maximize its expected return while also satisfying cumulative constraints . A few algorithms have been proposed to solve CMDPs for high-dimensional domains with continuous action spaces - however they come with their own caveats . Reward Constrained Policy Optimization ( Tessler et al. , 2018 ) and Primal Dual Policy Optimization ( Chow et al. , 2015 ) do not guarantee constraint satisfaction during the learning procedure , only on the final policy . Constrained Policy Optimization ( Achiam et al. , 2017 ) provides monotonic policy improvement but is computationally expensive due to requiring a backtracking line-search procedure and conjugate gradient algorithm for approximating the Fisher Information Matrix . Lyapunov-based Safe Policy Optimization ( Chow et al. , 2019 ) requires solving a Linear Program ( LP ) at every step of policy evaluation , although they show that there exists heuristics which can be substituted for the LP at the expense of theoretical guarantees . In this work , we propose an alternate formulation for solving CMDPs that transforms trajectory-level constraints into localized state-dependent constraints , through which a safe policy improvement step can be defined . In our approach , we define a notion of Backward Value Functions , which act as an estimator of the expected cost collected by the agent so far and can be learned via standard RL bootstrap techniques . We provide conditions under which this new formulation is able to solve CMDPs without violating the constraints during the learning process . Our formulation allows us to define state-level constraints without explicitly solving a LP or the Dual problem at every iteration . Our method is implemented as a reduction to any model-free on-policy bootstrap based RL algorithm , both for deterministic and stochastic policies , and discrete and continuous action spaces . We provide the empirical evidence of our approach with Deep RL methods on various safety benchmarks , including 2D navigation grid worlds ( Leike et al. , 2017 ; Chow et al. , 2018 ) , and MuJoCo tasks ( Achiam et al. , 2017 ; Chow et al. , 2019 ) . 2 CONSTRAINED MARKOV DECISION PROCESSES . We write P ( Y ) for the set of probability distributions on a space Y . A Markov Decision Process ( MDP ) ( Puterman , 2014 ) is a tuple ( X , A , P , r , x0 ) , where X is a set of states , A is a set of actions , r : X × A → [ 0 , RMAX ] is a reward function , P : X × A → P ( X ) is a transition probability function , and x0 is a fixed starting state . For simplicity we assume a deterministic reward function and starting state , but our results generalize . A Constrained Markov Decision Process ( CMDP ) ( Altman , 1999 ) is a MDP with additional constraints that restrict the set of permissible policies for the MDP . Formally , a CMDP is a tuple ( X , A , P , r , x0 , d , d0 ) , where d : X → [ 0 , DMAX ] is the cost function1 and d0 ∈ R≥0 is the maximum allowed cumulative cost . The set of feasible policies that satisfy the CMDP is the subset of stationary policies ΠD : = { π : X → P ( A ) ∣∣ E [ ∑Tt=0 d ( xt ) | x0 , π ] ≤ d0 } . We consider a finite time horizon T after which the episode terminates . The expected sum of rewards following a policy π from an initial state x is given by the value function V π ( x ) = E [ ∑T t=0 r ( xt , at ) | π , x ] . Analogously , the expected sum of costs is given by the cost value function V πD ( x ) = E [ ∑T t=0 d ( xt ) | π , x ] . The RL problem in the CMDP is to find the feasible policy which maximizes expected returns from the initial state x0 , i.e . π∗ = arg max π∈ΠD V π ( x0 ) An important point to note about CMDPs is that , in the original formulation , the cost function depends on immediate states but the constraint is cumulative and thus depends on the entire trajectory . In the case of MDPs , where a model of the environment is not known or is not easily obtained , it is still possible for the agent to find the optimal policy using Temporal Difference ( TD ) methods ( Sutton , 1988 ) . Broadly , these methods update the estimates of the value functions via bootstraps of previous estimates on sampled transitions ( we refer the reader to Sutton & Barto ( 2018 ) for more information ) . In the on-policy setting , we alternate between estimating the state-action value function Qπ for a given π and updating the policy to be greedy with respect to the value function . 3 SAFE POLICY ITERATION VIA BACKWARD VALUE FUNCTIONS . Our approach proposes to convert the trajectory-level constraints of the CMDP into single-step state-wise constraints in such a way that satisfying the state-wise formulation will entail satisfying the original trajectory-level problem . The advantages of this approach are twofold : i ) working with single-step state-wise constraints allows us to obtain analytical solutions to the optimization problem , and ii ) the state-wise constraints can be defined via value-function-like quantities and can thus be estimated with well-studied value-based methods . The state-wise constraints are defined via Backward Value Functions , in Section 3.2 , and in Section 3.3 we provide a safe policy iteration procedure which satisfies said constraints ( and thus the original problem ) . 1Here the cost only depends on states and not state-action pairs . 3.1 BACKWARD MARKOV CHAIN . Unlike in traditional RL , in the CMDP setting the agent needs to take into account the constraints which it has accumulated so far in order to plan accordingly for the future . Intuitively , the accumulated cost so far can be estimated via the cost value function VD running “ backward in time ” . Before giving the details of our approach and formally introducing the Backward Value Functions , we review the main ideas , which are built upon the work of Morimura et al . ( 2010 ) , who also considered time-reversed Markov chains but from the standpoint of estimating the gradient of the log stationary distribution ; we extend these ideas to TD methods . Assumption 3.1 ( Stationarity ) . The MDP is ergodic for any policy π , i.e. , the Markov chain characterized by the transition probability Pπ ( xt+1|xt ) = ∑ at∈A P ( xt+1|xt , at ) π ( at|xt ) is irreducible and aperiodic . Let M ( π ) denote the Markov chain characterized by transition probability Pπ ( xt+1|xt ) . The above assumption implies that there exists a unique stationary distribution ηπ associated with π , such that it satisfies : ηπ ( xt+1 ) = ∑ xt∈X P π ( xt+1|xt ) ηπ ( xt ) . We abuse the notation and denote Pπ ( xt+1 , at|xt ) = P ( xt+1|xt , at ) π ( at|xt ) . According to Bayes ’ rule , the probability q ( xt−1 , at−1|xt ) of a previous state-action pair ( xt−1 , at−1 ) leading to the current state xt is given by : q ( xt−1 , at−1|xt ) = P ( xt|xt−1 , at−1 ) Pr ( xt−1 , at−1 ) ∑ xt−1∈X ∑ at−1∈A P ( xt|xt−1 , at−1 ) Pr ( xt−1 , at−1 ) . From Assumption 3.1 , we have that Pr ( xt−1 , at−1 ) = ηπ ( xt−1 ) π ( at−1|xt−1 ) , and∑ xt−1∈X ∑ at−1∈A P ( xt|xt−1 , at−1 ) Pr ( xt−1 , at−1 ) = η π ( xt ) . We denote the posterior q ( xt−1 , at−1|xt ) as backward ( or time-reversed ) probability # » P π ( xt−1 , at−1|xt ) , and we have : # » P π ( xt−1 , at−1|xt ) = P ( xt|xt−1 , at−1 ) ηπ ( xt−1 ) π ( at−1|xt−1 ) ηπ ( xt ) = Pπ ( xt , at−1|xt−1 ) ηπ ( xt−1 ) ηπ ( xt ) . ( 1 ) The forward Markov chain , characterized by the transition matrix Pπ ( xt+1|xt ) , runs forward in time , i.e. , it gives the probability of the next state in which the agent will end up . Analogously , a backward Markov chain is denoted by the transition matrix # » P π ( xt−1|xt ) = ∑ at−1∈A # » P π ( xt−1 , at−1|xt ) , and describes the state and action the agent took to reach the current state . Definition 3.1 ( Backward Markov Chain ) . A backward Markov chain associated with M ( π ) is denoted by # » B ( π ) and is characterized by the transition probability # » P π ( xt−1|xt ) . 3.2 BACKWARD VALUE FUNCTION . We define the Backward Value Function ( BVF ) to be a value function running on the backward Markov chain # » B ( π ) . A BVF is the expected sum of returns or costs collected by the agent so far . We are mainly interested in maintaining estimates of the cumulative cost incurred at a state in order to express the total constraint state-wise . We note that , since every Markov chainM ( π ) is ergodic by Assumption 3.1 , the corresponding backward Markov chain B ( π ) is also ergodic ( Morimura et al. , 2010 , Prop . B.1 ) . In particular , every policy π can reach the initial state via some path in the transition graph of the backward Markov chain . Thus , the backwards Markov chain are also finite-horizon for some TB , with x0 corresponding to the terminal state . We define a finite-horizon Backward Value Function for cost as : # » V π D ( xt ) = E # » B ( π ) [ TB∑ k=0 d ( xt−k ) |xt ] . ( 2 ) Proposition 3.1 ( Sampling ) . Samples from the forward Markov chainM ( π ) can be used directly to estimate the statistics of the backward Markov chain # » B ( π ) ( or the Backward Value Function ) . We have : E # » B ( π ) [ K∑ k=0 d ( xt−k ) |xt ] = EM ( π ) [ K∑ k=0 d ( xt−k ) |xt , ηπ ( xt−K ) ] , ( 3 ) = EM ( π ) [ K∑ k=0 d ( xt+k ) |xt+K , ηπ ( xt ) ] , where EM ( π ) and E # » B ( π ) are expectations over the forward and backward chains respectively . The Equation ( 3 ) holds true even in the limit K →∞ . The proof is given in Appendix B.1 . Using the above proposition , we get an interchangeability property that removes the need to sample from the backward chain . We can use the traditional RL setting and draw samples from the forward chain and still estimate the BVFs . Equation ( 2 ) can be written recursively as : # » V π D ( xt ) = E # » B ( π ) [ d ( xt ) + # » V π D ( xt−1 ) ] . In operator form , the above equation can also be written as : ( # » T π # » V π D ) ( xt ) = E xt−1∼ # » P π [ d ( xt ) + # » V π D ( xt−1 ) ] . ( 4 ) Proposition 3.2 ( Fixed point ) . For a policy π , the associated Backward Value Function vector , # » V π , satisfies limk→∞ ( # » T π ) k # » V = # » V π for every vector # » V , and # » V π is the unique solution of the equation # » V π = # » T π # » V π . The proof is given in Appendix B.2 . The above proposition allows us to soundly extend the RL methods based on Bellman operators for the estimation of BVFs .

Summary: The paper approaches the CMDP problem, in which one wishes to learn a max return policy subject to trajectory-based constraints.  The paper proposes a technique based on the introduced concept of "backward value functions".  These functions satisfy a sort of Bellman equation.  The paper proposes a safe policy improvement step based on these value functions, with theoretical guarantees on the safety of the resulting policy.  The method is evaluated on gridworlds and mujoco tasks, showing good performance.