DESTA: A Framework for Safe Reinforcement Learning with Markov Games of Intervention
Authors: David Henry Mguni
Joel Jennings
Taher Jafferjee
Aivar Sootla
Changmin Yu
Usman Islam
Ziyan Wang
Yaodong Yang
Jun Wang
Venue: ICLR 2022 Submitted (2022)
Keywords: Safe reinforcement learning
safety
Markov games
stochastic games

Abstract: Exploring in an unknown system can place an agent in dangerous situations,
exposing to potentially catastrophic hazards. Many current approaches for tackling
safe learning in reinforcement learning (RL) lead to a trade-off between safe
exploration and fulfilling the task. Though these methods possibly incur fewer
safety violations they often also lead to reduced task performance. In this paper, we
take the first step in introducing a generation of RL solvers that learn to minimise
safety violations while maximising the task reward to the extend that can be
tolerated by safe policies. Our approach uses a new two-player framework for safe
RL called DESTA. The core of DESTA is a novel game between two RL agents:
Safety Agent that is delegated the task of minimising safety violations and Task
Agent whose goal is to maximise the reward set by the environment task. Safety
Agent can selectively take control of the system at any given point to prevent
safety violations while Task Agent is free to execute its actions at all other states.
This framework enables Safety Agent to learn to take actions that minimise future
safety violations (during and after training) by performing safe actions at certain
states while Task Agent performs actions that maximise the task performance
everywhere else. We demonstrate DESTA’s ability to tackle challenging tasks and
compare against state-of-the-art RL methods in Safety Gym Benchmarks which
simulate real-world physical systems and OpenAI’s Lunar Lander.


Under review as a conference paper at ICLR 2022
DESTA: A FRAMEWORK FOR SAFE REINFORCEMENT LEARNING WITH MARKOV GAMES OF INTERVENTION
Anonymous authors Paper under double-blind review
ABSTRACT
Exploring in an unknown system can place an agent in dangerous situations, exposing to potentially catastrophic hazards. Many current approaches for tackling safe learning in reinforcement learning (RL) lead a trade-off between safe exploration and fulfilling the task. Though these methods possibly incur fewer safety violations, they often also lead to reduced task performance. In this paper, we take the first step in introducing generation of RL solvers that learn to minimise safety violations while maximising the task reward to the extent that can be tolerated by safe policies. Our approach uses a new two-player framework for safe RL called Distributive Exploration Safety Training Algorithm (DESTA). The core of DESTA is novel design of game between two RL agents: Safety Agent that is delegated the task of minimising safety violations and Task Agent whose goal is to maximise the reward set by the environment task. Safety Agent can selectively take control of the system at any given point to prevent safety violations while Task Agent is free to execute its actions at all other states. This framework enables Safety Agent to learn to take actions that minimise future safety violations (during and after training) by performing safe actions at certain states while Task Agent performs actions that maximise the task performance everywhere else. We demonstrate DESTA's ability to tackle challenging tasks and compare against state-of-the-arr RI. methods in Safety Gym Benchmarks which simulate real-world physical systems and OpenAl's s Lunar Lander.
INTRODUCTION
Reinforcement learning (RL) is framework that enables autonomous agents to learn complex behaviours from interactions with the environment (Sutton & Barto, 2018). Inspired by concepts within neuroscience, RL has had notable successes in number of practical domains such as robotics and video games (Deisenroth et al., 2011; Peng et al., 2017). During its training phase, an RL agent explores using trial and error approach in order determine the best actions. This process can lead to the agent selecting actions whose execution in some states may result in critical damage; for example, aerial robot attempting to fly high velocities subsequent permanent system failure. Consequently, a major challenge in RI. is to produce methods that solve the task and ensure safety both during and after training.
result in the helicopter crashing and
A common approach to tackle the problem of safe exploration in RL is to use a constrained Markov decision process (c-MDP) formulation (Altman, 1998). In this framework, the agent seeks to max- imise a single objective subject to various safety constraints. Although c-MDP can be solved if the model known, extending this formalism to settings in which the model is unknown remains a significant challenge (Chow et al., 2019a). One of the main tools for tackling the c-MDP prob- lem setting in RL is the Lagrangian approach for solving a constrained problem, that is, solving maxo mIn (0) )g(#) by gradient descent in À and ascent in 0 instead of solving max, (0) subject to g(0) < This approach introduces a trade-off between competing objectives: maximising the task reward and minimising safety violation. This trade-off which is calibrated by X, undermines the ability of the agent to optimise both criteria simultaneously (Srinivasan et al., 2020; Kahn et al., 2017). Consequently, the resulting policies do not offer guarantees of safety during training Platt & Barr (1987); Song & Leland (1998) and can lead to safety violations both during training and at convergence (Chow et al., 2018 8; Stooke et al., 2020).

---

Under review as a conference paper at ICLR 2022
In contrast, to ensure their safety, animals exhibit a vast number of safety reflexes: reactive intervention systems designed to override and assume control in dangerous situations to prevent injury. One such example is the diving reftex, a sequence of physiological responses to the threat of oxygen deprivation (asphyxiation) that overrides the body's basic behavioural and regulatory (homeostatic) systems. This reflex is fundamental that it has been found in all air-breathing vertebrates (Butler & Jones, 1997).
Inspired by such naturally occurring systems, in this paper, we tackle the challenge of learning safely in RL with a new two-agent framework for safe exploration and learning, DESTA. The framework entails an interdependent interaction between an RL agent, Task Agent whose goal is to maximise the set of environment rewards and an additional RL agent, Safety Agent whose goal to ensure the safety of the system. The Safety Agent has the authority to override the Task Agent and assume control of the system and, using a deterministic policy to avoid unsafe states while Task Agent can perform its actions everywhere else to maximise rewards. As such, the goals of completing the task set by the environment and ensuring safety are decoupled and each is delegated to an individual agent.
Transforming these components into a workable framework requires a formalism known as Markov games (MGs). MGs augment MDPs to tackling strategic interactions between agents (Littman, 1994). To bridge the gap between safety solutions offered in nature and machine learning. we develop new type of MG namely a monzero-sum MG of interventions. In this novel type of MG, two agents with distinct goals cooperate and exchange control of the system to minimise safety violations meanwhile maximising the task objective. Our framework, which consists of a strategic interaction between two independent RL agents confers several key advantages:
The tasks of maximising the environment reward and minimising safety violations are fully decoupled. This means Safety Agent pursues its safety objectives without trading-off safety for environment rewards and, performs safe planning to minimise safety violations at future system states. Moreover, since the agents are trained in a shared system, each agent learns policy which acts in response to the behaviour of the agent. In particular, Safety Agent anticipates Task Agent's 's future actions (after and during training) and acts to reduce safety violations (see Experiment 1).
Decoupled Objectives & Safety Planning:
Safety Agent acts only at states in which its action ensures the safety criteria. At states in which the safety criterion is irrelevant only actions that are relevant to the task are played. Moreover Safety Agent uses a deterministic policy which eliminates inadvertent actions that may lead to safety violations (see Lunar Lander experiment).
Selective (deterministic) interventions:
Plug & play: DESTA is highly flexible and accommodates various notions of safety within Safety Agent's objective. Unlike various approaches in which safety contingencies are manually embedded into the policy e.g. Schulman et al. (2015), DESTA accommodates any RL policy. This also enables safe policies to be (re)used and applied only at states where necessary.
RELATED WORK
Recent works in safe RL include assumptions from knowing the set of safe states and access to a safe policy (Koller et al., 2018; Berkenkamp et al., 2017), knowing the environment model (Fisac et al., 2019; Dean et al., 2019), having access to the cost function (Chow et al., 2019b), having a continuous safety cost function (Cowen-Rine Rivers et al., 2020) and using reversibility as a criterion for safety (Eysenbach et al., 2018). Constrained Policy Optimization (CPO) (Achiam al., 2017) extends trust region policy optimisation (TRPO) (Schulman et al., 2015) with the aim of ensuring that a feasible policy stays within the constraints in expectation. Similarly in the context of multi-agent RL (Yang & Wang, 2020), Gu et al. (2021) develops MACPO by extending multi-agent TRPO (Kuba et al., 2021) with safe constraints. However, the convergence of these methods is still challenging: the learning dynamics tend to oscillate Stooke et al. (2020), and the methodology does not readily accommodate general RL algorithms (Chow et al., 2018). 8). In Tessler al. (2018), a reward-shaping approach is used to guide the learned policy to satisfy the constraints, however their approach provides no guarantees during the learning process. In Dalal et al. (2018), a safety layer is introduced that acts on top of an RL agent's possibly unsafe actions to prevent safety violations though their framework does not deal with negative long-term consequences of an action. In Bharadhwaj et al. (2021), a similar framework to CPO is used with a sparse binary safety signal where the Q function is overestimated to provide tuneable safety. Recently, Stooke et al. (2020) investigated the oscillation issue from a dynamical system point of view and introduced a treatment by applying a PID controller the dual variable.
2

---

Under review as a conference paper at ICLR 2022
To combat the limitations of the c-MDP formulation, several methods transform the original constraint to a more conservative one to ease the problem. For example El Chamie et al. (2016) replace the constraint cost with conservative step-wist surrogate constraint. A significant drawback of these approaches is their conservativeness undermines task performance (the extent of the sub-optimality has yet be characterised). Other approaches manually embed engineered safety-responses that executed near safety-violating regions (Eysenbach et al., 2018; Turchetta et al., 2020). For example, in Turchetta et al. (2020), safe teacher-student framework in which the teacher's objective is the value of the student's ss final policy, the agent is endowed with pre-specified library of reset controls that it activates close to danger. These approaches can require time-consuming human input contrary to the goal of autonomous learning. Moreover, these approaches do not allow the safety response to anticipate future the behaviour of the RI. agent and do not perform safety planning.
PRELIMINARIES
Reinforcement Learning (RL). In RL, an agent sequentially selects actions to maximise its expected returns. The underlying problem is typically formalised as an MDP (S. A. P, 7) where S RR is the set of states, A C IR* is the set of actions, P S x A S - [0, 1] is transition probability function describing the system's 's dynamics, R x A + R the reward function measuring the agent's performance and the factor 7 [0, 1) specifies the degree to which the agent's 's rewards are discounted over time (Sutton & Barto, 2018). At time E 0, 1...., the system is in state 8, E S and the agent must choose action ag E A which transitions the system
new state 8t+1 ~ P(-|s,, (z) and produces a reward R(s,, at). A policy π : S x A + [0. 1] is a probability distribution over state-action pairs where π (als) represents the probability of selecting action a E A in state s E S. The goal of an RL agent is to find a policy Ω E II that maximises its expected returns given by the value function: vn (s) E|E==0 R(s,, )t) rT-1s:)] where II is the agent's policy set.
Safety in RL. A key concern for RL in control and robotics settings is the idea of safety. This is handled in two main ways (Garefa et al., 2015): using prior knowledge of safe states to constrain the policy during learning or modifying the objective to incorporate appropriate penalties or safety constraints. The constrained MDP (c-MDP) framework (Altman, 1999) is a central formalism for tackling safety within RL. This involves maximising reward while maintaining costs within certain bounds which restricts the set of allowable policies for the MDP. Formally, a c-MDP consists of an MDP (S, AS P. R, 7) and C = {L: S x A - R, de E R) = 1, ..2....., nn.. which is set of safety constraint functions == (L1.... . Ln) that the agent must satisfy and {d+) which describe the extent to which the constraints
allowed to be not satisfied. Given set of allowed policies
Ilc == {I E II : L. I d, Vi = I,....nIJ n where o7 (s) = EIEN= LL ac)ls0 = s]. the cDMD objective is to find a policy x" such that '- E arg max v (s). for all s E S. The accumulative safety costs can be represented using hard constraints, this captures for example avoiding subregions 4 C S. When L, is an indicator function i.e. takes values {0, 1}, it is easy to see that each vr., represents the accumulated probability of safety violations since vI, (8) s] P(violation).
of
Safe exploration in RL seeks to address the challenge of learning an optimal policy for a task while minimising the occurrence of safety violations (or catastrophic failures) during training (Hans et al., 2008). Since in RL, the model dynamics and reward function are priori unknown, the aim is to keep the frequency of failure in each training episode as small as possible.
Markoy games. Our framework involves system of two agents each with their individual objectives. Settings of this kind are formalised by MGs, a framework for studying self--nterested agents that simultaneously act over time (Littman, 1994). In the standard MG setup, the actions of both agents influence both each agent's rewards and the system dynamics. Therefore, each agent i E {1, 2} has its own reward function R; S x (A) + I and action set A; and its goal is to maximise its own expected returns. The system dynamics, now influenced by both agents, are described by a transition probability P. S x Ax) x S + Jo, 1]. Unlike classical MGs, in our MG, Safety Agent does not intervene at each state but is allowed to assume control of the system at certain states which it decides using form of control known impulse control (Mguni, 2018).
3

---

Under review as a conference paper at ICLR 2022
4 OUR FRAMEWORK
We now describe our framework which consists of two core components: firstly MG between two agents, Task Agent and a second agent, the Safety Agent and, an impulse control component which used by the Safety Agent. The impulse control component allows the Safety Agent to be selective about the set of states that assumes control (and in doing so influence the transition dynamics and reward) so that actions geared towards safety concerns are performed only at relevant states. This leaves Task Agent to maximise the environment reward everywhere else. Unlike the c-MDP formulation, the goal of minimising safety violations and maximising the task reward are now delegated to two individual agents that now have distinct objectives.
4.1 A MARKOV GAME OF INTERVENTIONS ON ONE SIDE
We introduce our new MG of interventions on one side. Formally, our MG is defined by a tuple
= (W, S, A, A2.sale P. R1, R2, 7) where the new elements the set of agents V 21, A2,safe C A which is the action set for Safety Agent and the functions RG : S x A x A2 - IR
the one-step reward for agent E {1. 2]. The transition probability P S x A x A2,safe x x + [0. 1] takes the state and action of both agents as inputs. Task Agent and Safety Agent use the Markov policies π. S + [0. 1] and 2,39afe S A2.safo - [0. 1] respectively each of which is contained in the sets II and safe C II which are stochastic policy set and deterministic policy subset respectively. Therefore, whenever Safety Agent assumes control, random exploratory actions are switched off. Lastly, Safety Agent also has (categorical) policy $2 S - {0, 1) which it uses to determine whether or not should intervene.
Denote by {Tx 1=20 the intervention times or points at which the Safety Agent decides to take an action so for example if the Safety Agent chooses to take an actions at state 86 and again at state sg. then Ti 6 and T2 8 (we will shortly describe these in more detail). At any instance the transition dynamics are affected by only the Safety Agent whenever it decides to act (Task Agent influences the dynamics all other times). With this in mind, define the function P: S x A 22safe x S [0, 1] by P(8+11, 0t Eogage ai
2.sale 6T 8t) where the function &g. is the Kronecker-delta function (so
Enckee af 2:safing 57 is a2.sate whenever TI, ...... and null otherwise). The transition dynamics are determined by the probability transition function given by
P(s', a, a,a22atte a2aaaee s) P(s',a,
Ar.cata (a (a2:sale)) + a(s',a2.saee 1 (a (a2ssufe). (1)
Note that if Safety Agent plays fixed policy then the MG reduces to MDP.
4.2 THE TASK AGENT OBJECTIVE
The goal of Task Agent is to maximise its expected cumulative reward set by the environment (note that this does not include safety which is delegated to Safety Agent). To construct the objective for Task Agent, we begin by defining the function Rg S x A x A2,safe + R by R₁ (St, ae G2 2.safe,
R(s,, a1)(1 - ar.ant (a, ( + R(s,, a, (.?.safo)11 1 (a₂ (a,-safe) where Ly(v) is the indicator function which is 1 whenever E y and otherwise. The objective that Task Agent seeks to maximise is:
( (2)
ΣΣ 12005k
(st, r'R (sc, aes az
2,salo 87 Su S
where a, ~O (-1s:) is Task Agent's action and a
2.aae 2 is an action chosen by the Safety Agent. Therefore, the reward received by Task Agent is R(s,, a, 2,safe) when t = Tke h: 0, 1........ i.e. whenever the Safety Agent decides to take an action and R(st, a,) otherwise.
4.3 THE SAFETY AGENT OBJECTIVE
The goal of the Safety Agent is to minimise safety violations both during and after training. Unlike Task Agent whose actions incur cost, for the Safety Agent, each intervention incurs cost applied
4

---

Under review as a conference paper at ICLR 2022
by a cost function : A2,safe - R20. The cost ensures any Safety Agent interventions are warranted by an increase in safety. The objective that Safety Agent seeks to maximise is:
ey u5 0z) (8) -
ΣΣ 12005k
7' (-E(se, ae; aie (2anler - c(a=, 2aanll ((22))1107)))
(3)
at a?.eate)- L(se, a,)( 110--112200 A2 (a? ((?)ale)) + where SS A A2sale + Ri is defined by L(se, a2
L(s, (2 Aa.uue (al) and L := (Li.... Ln) is a set of constraint functions that indicate how much a given constraint has been violated and is provided by the environment. Therefore to maximise its objective, the Safety Agent must determine the sequence of points [T&} at which the benefit of performing precise action overcomes the cost of doing so. We specialise the case c(a2,aae)- K a2e where is positive constant. Each function L, can represent a (possibly binary) signal that indicates a visitation to an unsafe state.
We now mention some key aspects of the framework. In particular, now the task of ensuring safety is delegated to Safety agent whose sole objective is minimise safety violations throughout the course of the problem. Secondly, the presence of the (strictly negative) intervention cost induces selection process for Safety agent whereby it seeks only to intervene at the set of states for which the reduction in cumulative safety violations is sufficiently high merit an intervention. This means that at all other states, task agent can freely act and in doing so learn to play actions that deliver task rewards whenever there is no potential for safety violations. Lastly, since the agents learn how to respond to one another, the strategic interaction between the two agents leads to policies in which task agent anticipates the actions of the task agent and vice-versa.
4.4 THE SAFETY AGENT IMPULSE CONTROL MECHANISM
The problem for the Safety Agent is to determine at which states it should assume control and what actions should perform. We now describe how a given state Safety Agent decides to intervene and overwrite Task Agent or not and the magnitudes of such interventions. In our setup at each state the Safety Agent first makes a binary decision to decide to assume control. Therefore, the intervention times } are rules that depend on the state that are given by тæe inf Tk-1188 E T, (2())) = 1} where T the or sequence of states induced by the joint actions of Task Agent and the Safety Agent and the probability kernel P. Therefore, by learning an optimal g2, the Safety Agent learns the useful states to perform an intervention. As we later explain, these intervention times are determined by a condition on the state which is easy evaluate (see Prop. 1).
Unlike (Eysenbach et al., 2018; Turchetta et al., 2020), our approach enables learning a safe interven- tion policy during training without the need to preprogram manually engineered safety responses and minimises safety violations during training unlike (Tessler et al., 2018; Dalal et al., 2018). Unlike (Fisac et al., 2019; Dean al., 2019; Chow et al., 2019b; Fisac et al., 2019), which require access to information which is not available without a priori knowledge of the environment, our framework does not require priori knowledge of the model of the environment or the unsafe states.
Learning to solve an MG involves finding a stable point in the independent agents' policies. In our MG, this means the Safety Agent learns assume control at a subset of states and minimise safety violations given Task Agent's 's policy and, Task Agent learns to execute actions that maximise the task objective at all other states (given the actions of Safety Agent).
THE LEARNING METHOD
A key aspect of our framework is the presence of two RL agents that each adapt their play according to each other's behaviour. This produces two concurrent learning processes each designed to fulfill distinct objectives. At stable point of the learning processes the Safety Agent minimises safety violations while Task Agent maximises the environment reward. Additionally, Safety Agent learns the set of states in which perform an action to maintain safety the current future states. Therefore, central to this process is the decision of where Safety Agent should intervene. The intervention times characterised by 'obstacle condition' which can be evaluated online.
We now derive the condition that characterises Safety Agent's intervention times. We begin by first defining key object in preparation for this characterisation:
5

---

Under review as a conference paper at ICLR 2022
Algorithm 1: Distributive Exploration Safety Training Algorithm (DESTA)
1: Initialise replay buffers D (0). D2
{0)) Dint = (0).
2: for Nepisodes do
3: State 50
4: for = 0, 1....do do
5: Sample task action a, ~ IT (-(st), ) a safe action of aaaee ~ x7 (-16e), and an intervention action at ~ g2(-1s;) (E {0.1})
6:
if a = 0 then
7:
Apply task action a, so 81+1 P( 8t). Set dt
8:
else if attt = 1 then
9:
2.smfe Apply safe action dt so 81+1 end if
P(.la, 2,aat 81). Set dz
2,safe
10:
11:
Receive reward R(st, a) and cost L(s a)
12:
Set R = R(s,, a), R2 = - (s,,, a) da). Rint - a) c(a)ajue
13:
Add the sample (st, a, 81+1, R1) to D, the sample (8t, a, St+1, R2) to D the sample
(st, at S1+1, Rint) to Dint end for
15:
// Learn the individual policies
Update the policy æ using D, the policy 2,ssf using D2,safi, the policy g2 using Dint
17: end for
We first introduce the following object which is required for constructing the Bellman operator:
Definition Let E II and 222sale II2.sale be a Task Agent and a Safety Agent policy re-
spectively, then for any STY E S and any TI, .,.., define by 2 Q7..7" 3iaate
(STu, 22 ==
-((ST TT 2.sale) c(Ti2aale) + 7 Ss P(s; misale, 8ra )u
(s). We define the intervention
operator M by the following: M 2
0/2
J. 2ante
(STL):
max a' a.A.sae
27
2,aie a
The quantity Muz measures the expected future stream of rewards for Safety Agent after an immedi- ate (optimal) intervention minus the cost of intervening. The following result characterises the Safety Agent policy d2 and the times that Safety Agent must perform an intervention.
Proposition 1 For any S. the Safety Agent intervention policy g2 is given by the following aa daate
expression (2( = H (T FRawn [M² U2
(s) Q Q77 (s) (s) where, Q
(8, ()
a() + 7 I ds' ds' P(s'; ((65:an s)ug aime (8') and H is the Heaviside function, moreover Safety Fent
Agent's intervention times are given by = inl {r> M|
2,sain
02
U2
Hence, Prop. 1 also characterises the (categorical) distribution g2- Moreover, the times {Tk} can be determined by evaluating if Mu² = b2 holds. The result yields a key aspect of our algorithm for executing Safety agent's activations.
While implementing the intervention policy appears to be straightforward by comparing value functions, it requires the optimal value functions in question. Furthermore, learning the intervention policy and these value functions simultaneously resulted in an unstable procedure. As a solution we propose to learn 22 using off-the-shelf policy gradient algorithm (such as TRPO, PPO SAC). This policy is categorical with values {0, }} and has a reward Rint equal to - L(st, a) + c(a) if
the safety agent intervenes with an action a, 2,safe and -((s;, if the safety agent does not intervene, i.c., the task agent applics an action a = a,.
Implementing both the safety policy, which maximise the safety associated reward, and the task policy, which maximises the task reward, can be done using any off-the-shelf policy gradient methods without modifications. To summarise, we will learn three policies: the first one learning the actions of the task agent, the second learning the action of the safety agent, if the safety agent intervenes, and the final learning to intervene.

---

Under review as a conference paper at ICLR 2022
Figure 1: Diagram of the T-Junction environment
We take an off-policy approach and every policy is an instance of SAC with appropriate action spaces and rewards. Since we learn off-policy every agent can collect the same triplets St, St41, where is the current action (i.e., either the task action a the safety action de d. ). The rewards that the policies will receive will differ though. The task policy will receive R₁ = R(s:, a). the safety policy will get Rg - L(s,, a) a)-c(a)), and the intervention policy rewards are Rint -L(s a) -(aaaa)) c(a)a where a a ~ (-(+)))) Note that we chose to use the same triplets for all policies since we use off- off-policy algorithms, which allows to use the acquired data more efficiently.
6 EXPERIMENTS
We performed series of challenges to see if DESTA 1) learns to perform safe planning 2) learns to select the appropriate state to perform safe override interventions which avoids a trade-off between safety and task performance 3) learns to deterministic controls to ensure precise actions. In these tasks, we compared the performance of DESTA with state-of-the-art RL methods for safe learning: PPO (Schulman al., 2017), Lagrangian PPO, TRPO (Schulman et al., 2015), CPO (Achiam et al., 2017), and WCPG (Tang et al., 2019). We then compared DESTA against these baselines on performance benchmarks in challenging high dimensional problems in safety gym benchmarks (Yang et al., 2021) which simulate real-world physical environments and OpenAi's Lunar Lander.
Having an individual agent that is responsible for safety enables our method learn to plan ahead to minimise safety violations. In particular, it is able to learn to take actions to avoid safety violations at future states. To demonstrate this, we designed a T-Junction environment with two goal states. The environment is T-shaped grid of three corridors. The agent starts at one end of the neutral corridor and must advance to the junction at the other end. From here it must choose to go left or right into a safe or an unsafe corridor respectively. They each have a reward at the far end but the unsafe corridor additionally has a safety cost. In our experiments, the safe goal has a reward of 50, the unsafe goal reward of 100 and all other squares one-shot reward of 10. Each square in the unsafe corridor has a 10% chance of giving a safety cost of 100. Therefore, the agent must learn forego larger reward to stay safe, with Safety Agent directing it down the safe corridor. The maximum possible safe return per episode is 170, any higher and the unsafe corridor was breached.
Experiment 1. Safe Planning:
We compared our algorithm to PPO, Lagrangian PPO, TRPO, Lagrangian TRPO, SAC and CPO. DESTA-SAC (ours), Lagrangian PPO and Lagrangian TRPO were able to solve the environment by going down the correct corridor, with the others turning into the unsafe one. Ours trains faster and is more stable across seeds than the other baselines that solve the environment.
Experiment 2. Safe precision control using the Lunar Lander (Brockman et al., 2016): Since Safety Agent uses determinstic controls to perform its actions, Safety Agent able perform precise actions to ensure the safety of the system. This allows our method to avoid safety violations in instances where precision is required. To verify this claim, we used the Lunar Lander environment within OpenAl gym (Brockman et al., 2016). Given that there is no strict definition of safety violation as in the safety-gym environments (Brockman et al., 2016), we manually defined that a safety violation is incurred whenever the spacecraft transitions outside fixed horizontal threshold radius from the origin. By introducing such safety violation our goal was to test if DESTA can exploit the interplay between the reward maximisation and cost minimisation (i.e., ensure the spacecraft is always on the near near-optimal path), hence enabling stronger and more efficient learning. We include additional experiment details in the Appendix.
7

---

Under review as a conference paper at ICLR 2022
Fvaluation Returrn
Fealuaaion
500
250
2200
100
Maximum possible
100:
60
10
30
40
53
Stops 1008]
10
(b)
Training Rate
Evaliaation Average Correct Giaals Eplsode
1.0-
DESTA (9.5)
PPO
TEPO
SAC
PPO-1 AGGRANGIAN
TRPO LAGRANGIAN
0.2
CPO
Steps 1005)
Stepa 1000)
(c)
Figure 2: Empirical evaluation of DESTA and baselines on T-Junction environment. 32 evaluation episodes
every 1000 steps and 5 seeds were used per algorithm. Intervention costs of both 0.1 and 0.5 were used and learning rates were 0.001. Note that 20 SAC gradient steps were required achieve the reported performance. (a) Average return of evaluation episodes. On this environment, 170 is the maximum safe return per episode, obtained by moving through all centre squares, the junction square and all squares in the safe corridor without turning into the unsafe corridor. Any higher means the unsafe corridor was entered. (b) Average cost of evaluation episodes. (c) Average training cost per step so far. (d) Average number of correct (left) goals per evaluation episode.
Figure 3: The lander must land on the pad between two flags.
In Fig. 4(a) we observe that the DESTA agent outperforms all the baseline agents, in terms both the sample efficiency and the asymptotic score. Moreover, observe that DESTA enables more stable training, possibly due to the reduced action space given the intervention of the Safety Agent. In Fig. 4(b) show the accumulative safety violation throughout the course of training, and we again observe that DESTA yields the lowest accumulative cost amongst all the evaluated agents, whilst maintaining the stability of the cost across different random seeds. Such behaviour indicates that the Safety Agent is well-trained such that the safety-violation states are much more rarely encountered comparing to the baseline agents.
We note that the consistent low variance across various independent training runs indicates that DESTA enables low sensitivity with respect the randomness given different random seeds that plagues many of the existing RI algorithms (Colas et al., 2018). We also examined the behaviours of the intervention through the course of training (Fig. 8), which nicely corresponds with our intuition that the interventions mostly occur during the early phase of each episode. See the Appendix 10 for further discussion.
Safety Gym Benchmark Experiments: We compared DESTA with the baselines in challenging environments within a Safety Gym Benchmark in (Yang et al., 2021). These environments, each representing simulated physical systems vary in the number and size of hazards, and generation of goal and hazards' locations. In the first experiment, the hazards are static. We then tested setting with moving hazards. In terms of performance, DESTA outperforms the best competing baselines.
8

---

Under review as a conference paper at ICLR 2022
Figure 4: Empirical evaluation of DESTA on the Lunarl task. (a) Evaluation of the agents on the LunarLander task. (b) Accumulative safety violation cost during training.
Evaluation Return
Evaluation Cost
Training Cost Rate
1000
0.10
Ours
Botum
TRPO
40
50.04
TRPO
Lagrangion
-22
20
0.02
WCSAC
0
100
200
300
600
500
600
100
200
300
1000)
400
5do
600
100
200
200
400
500
600
Figure 6: StaticEnv-v0
Evaluation Return
Evaluation Cost
Training Cost Rate
200
20
0.10
Durs
10
Rotar
100
TRPOO
50.04
Lagrangian
50
WCSAC
-20
0.02
0
100
200
Stage's (000)
300
000
100
200
Steps's 1000)
300
600
100
200
300
1000)
000
600
Figure 7: DynamicEnv-v0
In terms of safety violations, DESTA incurs lower number of violations compared to the best competing baselines.
CONCLUSION
In this paper, we presented a novel game theoretical framework to solve the problem of learning safely. Our Markov game framework of Task agent and a second Safety agent de- couples the tasks of ensuring safety and maximising the task reward. This enables one of the agents learn com- plex behaviours at subsets of states to ensure the goal of minimising safety violations and at the same time ex- hibiting adaptive behaviour to the Task agent that maximises the environment reward at all other states. By presenting theoreti- cally solid and empirically robust approach to solving the safe learning problem, our method opens up the applicability of RL to range of real-world control problems with sophisticated safety constraints. In future, we believe our approach of marrying RL, MARL and game can adopted to solve other open challenges in RI.
DynamicEny
Geal
Apem:
Boundary
Boundary
Figure 5: Safety Gym Benchmark environments. Left: Static environment with hazard. Right: Dynamic environment with multiple hazards.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization, 2017.
Eitan Altman, Constrained markov decision processes with total cost criteria: Lagrangian approach and dual linear program. Mathematical methods of operations research, 48(3):387-477, 1998.
Eitan Altman. Constrained Markov Decision Processes. CRC Press, 1999.
Felix Berkenkamp, Matteo Turchetta, Angela P. Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. In Neural Information Processing Systems (NeurIPS), 2017. URL https: /tt///axxivv// org/ abtp//55.111.//. 08551.
Homanga Bharadhwaj, Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Animesh Garg. Conservative safery crities for exploration. In International Conference Learning Representations, 2021. URL https: / wwwpppereeieeeeee.eeeee formm ?id-ia08 (aA.a/00BDDUKKK 6DUuKi.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.00 01540, 2016.
Patrick J Butler and David R Jones. Physiology of diving of birds and mammals. Physiological reviews, 77(3):837-899, 1997.
Yinlam Chow, Ofir Nachum, Edgar Duemez-Guzman, and Mohammad Ghavamzadeh, A Iyapunov- based approach safe reinforcement learning. arXiv preprint /rXiv:/805.0770%. 2018.
Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. Lyapunov-based safe policy optimization for continuous control. arXiv preprint arXiv:1901.10031 2019a.
Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Gazezn and Mohammad Ghavamzadeh. Lyapunov- safe policy optimization for continuous control, 2019b.
C. Colas, O. Sigaud, and P. Y. Oudeyer. How many random seeds? statistical power analysis in deep reinforcement learning experiments. arXiv preprint arXiv:/806.088: 08295, 2018.
Alexander Nowen-Rivers Daniel Palenicek, Vincent Moens, Mohammed Abdullah, Aivar Sootla, Jun Wang, and Haitham Ammar. Samba: Safe model-based & active reinforcement learning, 2020.
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.0875: 2018.
S. Dean, S. Tu, N. Matni, and B. Recht. Safely learning to control the constrained linear quadratic regulator. In 2019 American Control Conference (ACC), pp. ACC.2019,88 1888198888886....
2019. doi: 10.23919/
Marc Peter Deisenroth, Carl Edward Rasmussen, and Dieter Fox. Learning to control low-cost manipulator using data-efficient reinforcement learning. Robotics: Science and Systems VII, pp. 57-64, 2011.
Mahmoud El Chamie, Yue Yu, and Behçet Açıkmeşe. Convex synthesis of randomized policies for controlled markov chains with density safety upper bound constraints. In 2016 American Control Conference (ACC), pp.
IEEE, 2016.
Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave no trace: Learning to reset for safe and autonomous reinforcement learning. In International Conference on Learning Representations, 2018. URL https: //tp:::ve//pppeeviiieee net /
J. F. Fisac, A. K. Akametalu, M. N. Zeilinger, S. Kaynama, J. Gillula, and J. Tomlin. A general safety framework for learning- based control in uncertain robotic systems. IEEE Transactions on Automatic Control, 64(7):2737-2752, 2019. doi: 10. ST..TAAAA2.11.88666563
Jaime F Fisac, Neil F Lugovoy, Vicenç Rubies-Royo Shromona Ghosh, and Claire J Tomlin. Bridging hamilton-jacok jacobi safety analysis and reinforcement learning. 2019 International Conference on Robotics and Automation (ICRA), pp. 8550-8556. IEEE, 2019.
10

---

Under review as a conference paper at ICLR 2022
Javier García, Femn, and Fernández. A comprehensive survey safe reinforcement learning. Journal of Machine Learning Research, 16(42):1437-1480, 2015. URL http:// /o/g papers/ http///////aaaaaaaaaaaaaa html.
Shangding Gu, Jakub Grudzien Kuba, Munning Wen, Ruiqing Chen, Ziyan Wang, Zheng Tian, Jun Wang. Alois Knoll, and Yaodong Yang. Multi-agent constrained policy optimisation. arXiv preprint 1rXiv221 2021.
Alexander Hans, Daniel SchneegaB, Anton Maximilian Schäfer, and Steffen Udluft. Safe exploration for reinforcement learning. In ESANN, pp. 143-148. Citeseer, 2008.
Gregory Kahn, Adam Villaflor, Vitchyr Pong, Pieter Abbeel, and Sergey Levine. Uncertainty-awant reinforcement learning for collision avoidance. arXiv preprint arXiv:1702.01182, 2017.
Torsten Koller, Felix Berkenkamp, Mattco Turchetta, and Andreas Krause. Learning-base based model predictive control for safe exploration. Proc. of the Conference on Decision and Control (CDC), 2018. URL http: / rxxi.. org/ abbs//// / 602.02.. 04450.
Jakub Grudzien Kuba, Ruiqing Chen, Mumning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning. arXiv preprint arXiv:2109. 1251, 2021.
Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pp. 157-163. Elsevier, 1994.
David Mguni. Optimal selection of transaction costs in a dynamic principal-agent problem. arXiv preprint arXiv:1805. 000620 2018.
Peng Peng, Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent difirretioonallycoordnnaaee nets: Emergence of human-level coordination in learning to play starcraft combat games. arXiv preprint urXiv::703: 10069, 2017.
John C Platt and Alan H Barr. Constrained differential optimization. In Proceedings of the 1987 International Conference on Neural Information Processing Systems, pp. 612-621 621, 1987.
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1889-1897, Lille, France, 07-09 Jul 2015. PMLR. URL http:ht///// http:///poceeediig// mlr. press /veess/v3/// http///3//ccchllc swcsshchalnnn1.mmmm 1atmmmmmmmmmmm........m
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 .06477, 2017.
Qiang Song and Robert P Leland. An optimal control model of neural networks for constrained optimization problems. Optimal Control Applications and Methods. 19(5):371-376, 1998.
Krishnan Srinivasan, Benjamin Eysenbach, Sehoon Ha, Jie Tan, and Chelsea Finn. Learning to be safe: Deep rl with safety critic. arXiv preprint 2XXi:20100116003 2020.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. International Conference on Muchine Learning, pp. 9133-9143. PMLR, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Yichuan Charlie Tang, Jian Zhang, and Ruslan Salakhutdinov. Worst cases policy gradients. arXiv preprint 4rX.v,191,.03618, 2019.
Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint 1805 11074, 2018.
Matteo Turchetta, Andrey Kolobov, Shital Shah, Andreas Krause, and Alekh Agarwal. Safe rein- forcement learning via curriculum induction. arXiv preprint arXiv:2006. 020006//2130. /2/6, 2020.
11

---

Under review as a conference paper at ICLR 2022
Qisong Yang, Thiago D Simão, Simon H Tindemans, and Matthijs TJ Spaan. Wesac: Worst-case soft actor critic for safety-constraing reinforcement learning. In Proceedings of the Thirty-Fifth AAAI Conference Artificial Intelligence. AAAI Press, online, 2021.
Yaodong Yang and Jun Wang. An overview of multi-agent reinforcement learning from game theoretical perspective, arXiv preprint Xiix:2011100055 2020.
12

---

Under review as a conference paper at ICLR 2022
Part I
Appendix
Table of Contents
8 Hyperparameter Settings
14
9 Proof of Technical Results
15
LunarLander experiment histogram
13

---

Under review as a conference paper at ICLR 2022
8 HYPERPARAMETER SETTINGS
Table 1: Hyperparameters for DESTA.
T junction
Lunar lander
StaticEnv
DynamicEnv
gradient steps
0
g
Runner
environment steps
0
600k
600k
Agent frequency update
Agent batch size
0
0
64
64
Policy network dimensions Policy networks activations
(0)
(0)
(256, 256}
(256, 256)
ReLU
ReLU
ReLU
ReLU
Agents
Value network layer dims
(0)
{0}
[256, 256}
(256, 256)
Value networks activations
ReLU
ReLU
ReLU
ReLU
Discount factor
0.99 0.005
0.99
Polyak update scale
0.005
Intervention cost
0
0.1
0.5
Optimiser
Opt Algorithm
Adam
Adam
Adam
Adam
Policy learning rate
0
0
10-4
10-1
Value function learning
0
10
10 10-4 4
Temperature learning rate
10-4
10-4
14

---

Under review as a conference paper at ICLR 2022
PROOF OF TECHNICAL RESULTS
PROOF OF PROPOSITION
Proof: [Proof of Prop. The proof given by establishing a contradiction. Therefore suppose that M= (8 ((TA)) S (b( (Txx) and suppose that the intervention time oi 71 an optimal intervention time. Construct the safety agent x²2 II2 and 5² policy switching times by (To- ......) and x² E II? policy by (T .I....) respectively. Define by inf 0: (st. Io) (s,, and m = sup < T/.. By construction we have that
u. 2 (s, 1o) 0/2
E R(so: ao) + E
... + 7
1-11 EE R( + - +ym-111B E -1;5;11 7MMm0 (5,TG)))
E R(s0, ao) + B
E 1( (1,8-11,47-11 + 7M T.=
U22 (0n,
We now use the following observation
[R(s, -1 01 + 7M Txx
U2
(st (st:
S Smax
{
02 1z.0 (071 [(T))), max en agEA EA
R(oT), an) + Eves P(slaa,, 8n )n (4,T6))))
Using this we deduce that
U/ T2
10) S E R(so, un) +E
+
E
-1
GrT max M x, (st) ((r)),
max or EA
R(STa, + Σ P(s': 8r, ) -. (s',
s'ES
= [[(so ao)
E [..... +
R(sn + y To 7,5
(ST3: [m))]]] =12 (s, To))
T
where the first inequality is true by assumption on M.. This is contradiction since x²2 is an optimal policy for safety agent. Using analogous reasoning, we deduce the same result for 7k < т after which deduce the result. Moreover, by invoking the same reasoning, we can conclude that it must be the case that (To, Ti Tk-1, Tk: Te+l....,) are the optimal intervention times.
15

---

Under review as a conference paper at ICLR 2022
Intervention Density Plot
2.00
1.75
1.50
1.25
1.00
0.75
0.50
0.25
0.00
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
Figure 8: Histogram of intervention location during training.
10
LUNARLANDER EXPERIMENT HISTOGRAM
16


Reviews:
[{"review_id": "t0uuPZ4SDUP", "reviewer": "Reviewer_7tNg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Safe RL problems are very important in many applications. The paper considers the c-MDP formulation for safe RL and proposes a novel Markov game approach, called DESTA, to solve c-MDP. The idea of DESTA is to decompose a c-MDP agent into two agents: the task agent and the safety agent. The action of the c-MDP agent is selected by either the task agent or the safety agent depending on who is currently in control, and the safety agent also determines which agent is in control at each time. With these two agents, the c-MDP is transformed into a Markov game between the two agents. Given the safety agent's policy, the task agent try to maximize rewards. Given the task agent's policy, the safety agent try to determine when to take over control and what actions to take in order to minimize costs.\n\nThis idea of formulating a Markov game between two agents to tackle c-MDP is very interesting. However, despite making some intuitive sense, the paper has no analysis on why the \"solution\" of the Markov game may be good for the c-MDP. There are two major issues with this Markov game framework.\n\nFirst, the paper never discusses what is the \"solution\" of the game or what is the goal the algorithm is trying to achieve after learning from the game. Since the two agents have different objective, typically one may seek to find an equilibrium of the game. But since the game is none zero-sum, there could be multiple equilibria in general. It is even more complicated when the policy of the safety agent is further decomposed in to an action selection policy and an intervention policy. So in fact there are three agents with three different objective functions. Does the policies generated by the algorithm converge to some kind of equilibrium of the game? Does the proposed training method has some convergence properties? \n\nSecond, even if the algorithm converges to some equilibrium policies, it's not clear if there is any connection between these policies and the c-MDP performance. Does the two separate policies for the safety agent trained by different objectives minimize the cost for the safety agent? Does the resulting overall policy satisfies the safety constraints? Are there reward guarantees using the equilibrium task and safety policies? \n\nAnother issue of the paper is the confusing notations. In many parts of the paper the interventions are represented by a sequence of intervention times. These intervention times creates many unnecessary lengthy equations with summations over mostly zero terms. In fact, the intervention could be represented by whether $a^{int}_t$ is zero or one, as described in the algorithm pseudo code. The current notations create several readability issues, and some of the equations do not make sense. For example, $Q_2^{\\pi, \\pi^{2, safe}}$ has two different definitions, one in Definition 1 and one in Proposition 1. \n\nFor the experiments, although the paper claims that the proposed method performs the best, it actually performs worse than some baselines in Figure 7. The abrupt changes in the return in Figure 6 and in the cost in Figure 7 also indicates some stability concerns.", "summary_of_the_paper": "The paper proposes Markov game based framework for safe reinforcement learning. The proposed method consists of two agents: the task agent selecting actions to maximize the reward, while the other safety agent determining whether to take over control of actions to minimize the cost. Experimental results are shown in three simulated environments.", "summary_of_the_review": "Pros\n*  Novel and interesting idea to use the two agents Markov game to tackle c-MDP.\n\nCons\n* Some issues with the Markov game framework and no analysis for them.\n* Confusing notations and problematic formulas.\n* Experimental results are mixed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hsXPpqtXJc3", "reviewer": "Reviewer_BFBF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n(1). The paper proposes a Markov game-based method for dealing with two competing objects: reward maximization and constraint satisfaction. This method incorporates the idea of the Markov game into the constrained Markov decision processes. In my view, it is a new and interesting idea. \n\n(2). The proposed method is flexible in many aspects, e.g., dealing with reward/cost separately, using a deterministic policy, and incorporating different safety criteria.\n\n(3). The paper also provides comparison experiments to demonstrate the performance of that the proposed method.  \n\n\nWeaknesses:\n\n(1). It is important to specify the meaning: reward maximization and the safety violation minimization are fully decoupled. Can you provide any evidence on this in the proposed framework?\n\n(2). I believe the framework in Section 3 is a key contribution. Is it always true that we can transform an expectation constraint into the objective for the safety agent? How exactly is it?\n\n(3). Do you have any theory that supports the proposed method? I feel the authors may devote some efforts to analyzing the proposed general-sum Markov games. ", "summary_of_the_paper": "The authors use a Markov game-based approach to study constrained Markov decision processes, propose a safety training algorithm, and show better performance than baseline methods in computational experiments.     \n", "summary_of_the_review": "Overall, I like the idea in the paper. However, I won't be able to support the acceptance. My major concern is theoretical support and clarity. Hopefully, the authors can supplement some progress during the rebuttal. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "JyMKXwodoBz", "reviewer": "Reviewer_vp9k", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths:\n\nThe idea of using Markov games to model the problem of safe-exploration is particularly interesting and novel to the best of my knowledge. \n\n### Concerns:\n\n- *Missing related-work:* The idea of having decoupled task objectives and safety planning has been explored in the community, although not from the lens of Markov games. Some of the works that come to mind are: [1] where barrier functions take the role of safety planner (Safety Agent) and can override the Task Agent policy,  [2] where pessimism is used for the Safety planning, and even in CPO based approached where there is a fallback policy when the agent violates the safety constraint that overrides the original task policy [3].  I would expect some more comparisons or details on the trade-offs of the proposed approach with the related work.\n\n- *Intervention cost origins:* In the safety objective each intervention occurs a  cost $c$ (or $\\kappa$) that is not part of the original CMDP formulation. How do we get this cost function? Is it a hyper-parameter or design-time decision? If so, we will have the same limitations as the Lagrangian methods. What’s the benefit of the proposed approach then? \nThe authors mention that unlike other work based on interventions they don’t need to design manually engineered safety responses, but I will argue that designing a cost function for intervention is just another form for that. \n\n- *Confusing writing:* I found the writing for Section 3 particularly confounding. The authors add on notations and terms in a very casual manner. I would expect formal definitions for each of the introduced new terms. \n\n- *Implication of Proposition 1?:* Proposition just tells us there exists a solution policy corresponding to the intervention reward. If we consider that the intervention reward function is a linear combination of different reward functions (with the same underlying transition) then isn’t this expected? I guess I’m failing to understand the importance of Proposition 1. \n\n- *Baselines:* Note that there are approaches like [4] that are not based on Lagrangian formulation and still can be used in the context of CMDPs during the learning, with even better performance than Lagrangian formulations. They would have made stronger baselines and would have strengthened the claim. \n\n- *Environments:* I would like to know why the authors decided to not use any of the environments that are used in context with the baselines (for instance OpenAI Safety Gym for Lagrangian-PPO, CPO etc.). Only the simpler environment (T-maze) is stochastic where both the lagrangian and proposed approaches perform similarly. The other environments considered in this work seem deterministic and not sufficient to support the claims. \n\n- *Reproducibility:* As all the claims are empirical, providing the accompanying code would have helped to strengthen them. There is also no mention of the number of seeds. Fig 6,7 have no captions or info about the axis. \n\n", "summary_of_the_paper": " The paper presents an alternate approach to the problem of minimizing constraint violations during learning in CMDPs (safe exploration). They formulate the problem using the framework of Markov games, where three policies are learned: (i) a policy that maximized just the primary reward objective for CMDP, (ii) a safety or override policy that is deterministic and focuses on minimizing the constraint violation, and (iii) an intervention policy that switches between (i) and (ii). The authors then test their approach on toy environments: T-maze, Lunar-lander and Safety Gym Benchmark. ", "summary_of_the_review": "Although the paper has some interesting ideas, it needs significantly more work in strengthening the supporting claims (weak experiments as well as no code), placing the work in context to the related work, as well as writing and presentation. I think the paper in its current state will be marginally useful to the community, and as such vote for rejection. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "E9UiD7D5d3z", "reviewer": "Reviewer_mFYr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is easy to read, and the key ideas are clearly presented.\n\nAs far as the reviewer knows, it is a new idea to formulate safe RL problems into Markov games with two players (i.e., Task Agent and Safety Agent). This framework enables us to train an agent more easily compared with Lagrangian methods since reward and safety signals are decoupled.\n\nI have several concerns below.\n\nFirst concern lies in Experiment. The authors conducted experiments in three different environments, but I do not think that the effectiveness and merits of the proposed method (framework) are not shown. For example, in Figure 2, how do the authors convince the readers that their proposed method over-performs the other baselines. If the authors are concerned about many constraint violations in the early training phase, then DESTA's results presented in Figure 7 is also problematic.\nGiven there is no theoretical result regarding optimality or safety guarantees, I think that merits of the proposed method are neither presented theoretically nor empirically.\n\nSecond concern is advantages over Lagrangian methods (e.g., TRPO-Lagrangian), which is also related to the first concern.\nIn the Section 1, the authors criticizes Lagrangian-based method saying \"Consequently, the resulting policies do not offer guarantees of safety during training and can\nlead to safety violations both during and after training\". However, seeing the experimental results, the proposed method also violates safety constraints in a similar level compared with Lagrangian-based RL algorithm. I am not fully convinced whether the proposed method is actually better than existing methods.\n\nThe last concern is reproducibility. Since there is no source-code attached and only little information is provided in the appendix, I do not think the experimental results presented in this paper is sufficiently reproducible. For example, how did you implement baselines and what is the parameters?\n\nMinor comments\n- Baselines are different for each experiment (for example, WCSAC is not tested in T-junction environment and TRPO-Lagrangian are not tested in LunarLander). Are there any reasons why such baselines are omitted?\n- In section 2, the action space for agent $i$ is defined as $\\mathcal{A}_i$, but in Section 3A, the it is denoted as $\\mathcal{A}^i$. Also, the action space for Task Agent is somewhat abbreviated into $\\mathcal{A}$ ($i$ is omitted). Please make such definitions consistent throughout the paper.\n- In the main text (Section 5), the authors say that WCPG is a baseline, but in Figures 4 and 5, results for WCSAC is presented.\n- Please add shade (e.g., standard deviation) in Figure 2.", "summary_of_the_paper": "This paper proposes a new framework for safe RL which is called DESTA. DESTA is a framework where two players called Safety Agent and Task Agent interact with the environment. The task of Safety Agent is to minimize the violations of the safety constraints and tat of Task Agent is to maximize the cumulative reward. Safety of the (safety-agnostic) Task Agent is encouraged by Safety Agent's interventions. The authors compare the performance of their proposed method with state-of-the-art RL baselines in three tasks.", "summary_of_the_review": "Though DESTA framework is new and interesting, its merits are neither presented theoretically nor empirically. In addition, although there is no theoretical results supporting the authors' claims, reproducibility is rather low.\nHence, I vote for rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigated online safe reinforcement learning problem in the constraint MDP setting. By introducing Safety Agent and Task Agent, the authors translate the RL problem into a Markov game. The AC agrees with all reviewers that there is a lack of theoretical analysis and experimental comparisons with existing benchmarks. It has not reached the bar of ICLR papers.
Decision: Reject