DESTA: A Framework for Safe Reinforcement Learning with Markov Games of Intervention

1 INTRODUCTION . Reinforcement learning ( RL ) is a framework that enables autonomous agents to learn complex behaviours from interactions with the environment ( Sutton & Barto , 2018 ) . Inspired by concepts within neuroscience , RL has had notable successes in a number of practical domains such as robotics and video games ( Deisenroth et al. , 2011 ; Peng et al. , 2017 ) . During its training phase , an RL agent explores using a trial and error approach in order to determine the best actions . This process can lead to the agent selecting actions whose execution in some states may result in critical damage ; for example , an aerial robot attempting to fly at high velocities can result in the helicopter crashing and subsequent permanent system failure . Consequently , a major challenge in RL is to produce methods that solve the task and ensure safety both during and after training . A common approach to tackle the problem of safe exploration in RL is to use a constrained Markov decision process ( c-MDP ) formulation ( Altman , 1998 ) . In this framework , the agent seeks to maximise a single objective subject to various safety constraints . Although c-MDP can be solved if the model is known , extending this formalism to settings in which the model is unknown remains a significant challenge ( Chow et al. , 2019a ) . One of the main tools for tackling the c-MDP problem setting in RL is the Lagrangian approach for solving a constrained problem , that is , solving maxθ minλ f ( θ ) − λg ( θ ) by gradient descent in λ and ascent in θ instead of solving maxθ f ( θ ) subject to g ( θ ) ≤ 0 . This approach introduces a trade-off between competing objectives : maximising the task reward and minimising safety violation . This trade-off which is calibrated by λ , undermines the ability of the agent to optimise both criteria simultaneously ( Srinivasan et al. , 2020 ; Kahn et al. , 2017 ) . Consequently , the resulting policies do not offer guarantees of safety during training Platt & Barr ( 1987 ) ; Song & Leland ( 1998 ) and can lead to safety violations both during training and at convergence ( Chow et al. , 2018 ; Stooke et al. , 2020 ) . In contrast , to ensure their safety , animals exhibit a vast number of safety reflexes : reactive intervention systems designed to override and assume control in dangerous situations to prevent injury . One such example is the diving reflex , a sequence of physiological responses to the threat of oxygen deprivation ( asphyxiation ) that overrides the body ’ s basic behavioural and regulatory ( homeostatic ) systems . This reflex is fundamental that it has been found in all air-breathing vertebrates ( Butler & Jones , 1997 ) . Inspired by such naturally occurring systems , in this paper , we tackle the challenge of learning safely in RL with a new two-agent framework for safe exploration and learning , DESTA . The framework entails an interdependent interaction between an RL agent , Task Agent whose goal is to maximise the set of environment rewards and an additional RL agent , Safety Agent whose goal is to ensure the safety of the system . The Safety Agent has the authority to override the Task Agent and assume control of the system and , using a deterministic policy to avoid unsafe states while Task Agent can perform its actions everywhere else to maximise rewards . As such , the goals of completing the task set by the environment and ensuring safety are decoupled and each is delegated to an individual agent . Transforming these components into a workable framework requires a formalism known as Markov games ( MGs ) . MGs augment MDPs to tackling strategic interactions between agents ( Littman , 1994 ) . To bridge the gap between safety solutions offered in nature and machine learning , we develop a new type of MG namely a nonzero-sum MG of interventions . In this novel type of MG , two agents with distinct goals cooperate and exchange control of the system to minimise safety violations meanwhile maximising the task objective . Our framework , which consists of a strategic interaction between two independent RL agents confers several key advantages : • Decoupled Objectives & Safety Planning : The tasks of maximising the environment reward and minimising safety violations are fully decoupled . This means Safety Agent pursues its safety objectives without trading-off safety for environment rewards and , performs safe planning to minimise safety violations at future system states . Moreover , since the agents are trained in a shared system , each agent learns a policy which acts in response to the behaviour of the agent . In particular , Safety Agent anticipates Task Agent ’ s future actions ( after and during training ) and acts to reduce safety violations ( see Experiment 1 ) . • Selective ( deterministic ) interventions : Safety Agent acts only at states in which its action ensures the safety criteria . At states in which the safety criterion is irrelevant only actions that are relevant to the task are played . Moreover Safety Agent uses a deterministic policy which eliminates inadvertent actions that may lead to safety violations ( see Lunar Lander experiment ) . • Plug & play : DESTA is highly flexible and accommodates various notions of safety within Safety Agent ’ s objective . Unlike various approaches in which safety contingencies are manually embedded into the policy e.g . Schulman et al . ( 2015 ) , DESTA accommodates any RL policy . This also enables safe policies to be ( re ) used and applied only at states where necessary . 2 RELATED WORK . Recent works in safe RL include assumptions from knowing the set of safe states and access to a safe policy ( Koller et al. , 2018 ; Berkenkamp et al. , 2017 ) , knowing the environment model ( Fisac et al. , 2019 ; Dean et al. , 2019 ) , having access to the cost function ( Chow et al. , 2019b ) , having a continuous safety cost function ( Cowen-Rivers et al. , 2020 ) and using reversibility as a criterion for safety ( Eysenbach et al. , 2018 ) . Constrained Policy Optimization ( CPO ) ( Achiam et al. , 2017 ) extends trust region policy optimisation ( TRPO ) ( Schulman et al. , 2015 ) with the aim of ensuring that a feasible policy stays within the constraints in expectation . Similarly in the context of multi-agent RL ( Yang & Wang , 2020 ) , Gu et al . ( 2021 ) develops MACPO by extending multi-agent TRPO ( Kuba et al. , 2021 ) with safe constraints . However , the convergence of these methods is still challenging ; the learning dynamics tend to oscillate Stooke et al . ( 2020 ) , and the methodology does not readily accommodate general RL algorithms ( Chow et al. , 2018 ) . In Tessler et al . ( 2018 ) , a reward-shaping approach is used to guide the learned policy to satisfy the constraints , however their approach provides no guarantees during the learning process . In Dalal et al . ( 2018 ) , a safety layer is introduced that acts on top of an RL agent ’ s possibly unsafe actions to prevent safety violations though their framework does not deal with negative long-term consequences of an action . In Bharadhwaj et al . ( 2021 ) , a similar framework to CPO is used with a sparse binary safety signal where the Q function is overestimated to provide tuneable safety . Recently , Stooke et al . ( 2020 ) investigated the oscillation issue from a dynamical system point of view and introduced a treatment by applying a PID controller on the dual variable . To combat the limitations of the c-MDP formulation , several methods transform the original constraint to a more conservative one to ease the problem . For example El Chamie et al . ( 2016 ) replace the constraint cost with a conservative step-wise surrogate constraint . A significant drawback of these approaches is their conservativeness undermines task performance ( the extent of the sub-optimality has yet to be characterised ) . Other approaches manually embed engineered safety-responses that are executed near safety-violating regions ( Eysenbach et al. , 2018 ; Turchetta et al. , 2020 ) . For example , in Turchetta et al . ( 2020 ) , a safe teacher-student framework in which the teacher ’ s objective is the value of the student ’ s final policy , the agent is endowed with a pre-specified library of reset controls that it activates close to danger . These approaches can require time-consuming human input contrary to the goal of autonomous learning . Moreover , these approaches do not allow the safety response to anticipate future the behaviour of the RL agent and do not perform safety planning . 3 PRELIMINARIES . Reinforcement Learning ( RL ) . In RL , an agent sequentially selects actions to maximise its expected returns . The underlying problem is typically formalised as an MDP 〈S , A , P , R , γ〉 where S ⊂ Rp is the set of states , A ⊂ Rk is the set of actions , P : S ×A× S → [ 0 , 1 ] is a transition probability function describing the system ’ s dynamics , R : S × A → R is the reward function measuring the agent ’ s performance and the factor γ ∈ [ 0 , 1 ) specifies the degree to which the agent ’ s rewards are discounted over time ( Sutton & Barto , 2018 ) . At time t ∈ 0 , 1 , . . . , the system is in state st ∈ S and the agent must choose an action at ∈ A which transitions the system to a new state st+1 ∼ P ( ·|st , at ) and produces a reward R ( st , at ) . A policy π : S × A → [ 0 , 1 ] is a probability distribution over state-action pairs where π ( a|s ) represents the probability of selecting action a ∈ A in state s ∈ S. The goal of an RL agent is to find a policy π̂ ∈ Π that maximises its expected returns given by the value function : vπ ( s ) = E [ ∑∞ t=0 γ tR ( st , at ) |at ∼ π ( ·|st ) ] where Π is the agent ’ s policy set . Safety in RL . A key concern for RL in control and robotics settings is the idea of safety . This is handled in two main ways ( García et al. , 2015 ) : using prior knowledge of safe states to constrain the policy during learning or modifying the objective to incorporate appropriate penalties or safety constraints . The constrained MDP ( c-MDP ) framework ( Altman , 1999 ) is a central formalism for tackling safety within RL . This involves maximising reward while maintaining costs within certain bounds which restricts the set of allowable policies for the MDP . Formally , a c-MDP consists of an MDP 〈S , A , P , R , γ〉 and C = { ( Li : S × A → R , di ∈ R ) |i = 1 , 2 . . . n } , which is a set of safety constraint functions L : = ( L1 , . . . Ln ) that the agent must satisfy and { di } which describe the extent to which the constraints are allowed to be not satisfied . Given a set of allowed policies ΠC : = { π ∈ Π : vπLi ≤ di ∀i = 1 , . . . , n } where v π Li ( s ) = E [ ∑∞ t=0 γ tLi ( st , at ) |s0 = s ] , the c-MDP objective is to find a policy π ? such that π ? ∈ arg maxπ∈ΠC vπ ( s ) , for all s ∈ S. The accumulative safety costs can be represented using hard constraints , this captures for example avoiding subregions U ⊂ S. When Li is an indicator function i.e . takes values { 0 , 1 } , it is easy to see that each vLi represents the accumulated probability of safety violations since v π Li ( s ) = E [ ∑∞ t=0 γ t1 ( st ) |s0 = s ] = P ( violation ) . Safe exploration in RL seeks to address the challenge of learning an optimal policy for a task while minimising the occurrence of safety violations ( or catastrophic failures ) during training ( Hans et al. , 2008 ) . Since in RL , the model dynamics and reward function are a priori unknown , the aim is to keep the frequency of failure in each training episode as small as possible . Markov games . Our framework involves a system of two agents each with their individual objectives . Settings of this kind are formalised by MGs , a framework for studying self-interested agents that simultaneously act over time ( Littman , 1994 ) . In the standard MG setup , the actions of both agents influence both each agent ’ s rewards and the system dynamics . Therefore , each agent i ∈ { 1 , 2 } has its own reward function Ri : S × ( ×2i=1Ai ) → R and action set Ai and its goal is to maximise its own expected returns . The system dynamics , now influenced by both agents , are described by a transition probability P : S × ( ×2i=1Ai ) × S → [ 0 , 1 ] . Unlike classical MGs , in our MG , Safety Agent does not intervene at each state but is allowed to assume control of the system at certain states which it decides using a form of control known as impulse control ( Mguni , 2018 ) .

Summary:  The paper presents an alternate approach to the problem of minimizing constraint violations during learning in CMDPs (safe exploration). They formulate the problem using the framework of Markov games, where three policies are learned: (i) a policy that maximized just the primary reward objective for CMDP, (ii) a safety or override policy that is deterministic and focuses on minimizing the constraint violation, and (iii) an intervention policy that switches between (i) and (ii). The authors then test their approach on toy environments: T-maze, Lunar-lander and Safety Gym Benchmark.