Towards Safe Reinforcement Learning with a Safety Editor Policy
Authors: Haonan Yu
Wei Xu
Haichao Zhang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Safe RL
constrained Markov decision process
safety layer
first-order safety method
model-free RL

Abstract: We consider the safe reinforcement learning (RL) problem of maximizing utility with extremely low constraint violation rates. Assuming no prior knowledge or pre-training of the environment safety model given a task, an agent has to learn, via exploration, which states and actions are safe. A popular approach in this line of research is to combine a model-free RL algorithm with the Lagrangian method to adjust the weight of the constraint reward relative to the utility reward dynamically. It relies on a single policy to handle the conflict between utility and constraint rewards, which is often challenging. We present SEditor, a two-policy approach that learns a safety editor policy transforming potentially unsafe actions proposed by a utility maximizer policy into safe ones. The safety editor is trained to maximize the constraint reward while minimizing a hinge loss of the utility state-action values before and after an action is edited. SEditor extends existing safety layer designs that assume simplified safety models, to general safe RL scenarios where the safety model can in theory be arbitrarily complex. As a first-order method, it is easy to implement and efficient for both inference and training. On 12 Safety Gym tasks and 2 safe racing tasks, SEditor obtains much a higher overall safety-weighted-utility (SWU) score than the baselines, and demonstrates outstanding utility performance with constraint violation rates as low as once per 2k time steps, even in obstacle-dense environments. On some tasks, this low violation rate is up to 200 times lower than that of an unconstrained RL method with similar utility performance. Code is available at https://github.com/hnyu/seditor.

Towards Safe Reinforcement Learning with a Safety Editor Policy
Haonan Yu, Wei Xu, and Haichao Zhang
Horizon Robotics Cupertino, CA 95014
(haonan. yu, wei xu, wwwhaaihhaa chao zhang))horrizon. ai
S
UM
a
Env
SE
Aa
Figure SEditor's framework with two policies utility maximizer (UM) and safety editor (SE). Given state s, UM proposes preliminary action à that aims at maximizing the utility reward. Then SE outputs an adjustment Aa to it based on and itself, to ensure a low constraint violation rate. The action proposal à and edit Aa are fed to an action editing function h that outputs final action interact with the environment. Both UM and SE are stochastic policies modeled as neural networks and are trained via SGD.
Abstract
We consider the safe reinforcement learning (RL) problem of maximizing utility with extremely low constraint violation rates. Assuming no prior knowledge or pre-training of the environment safety model given task, an agent has to learn, via exploration, which states and actions are safe. A popular approach in this line of research is to combine model-free RL algorithm with the Lagrangian method to adjust the weight
constraint reward relative to utility reward dynamically. It relies on a single policy to handle the conflict between utility and constraint rewards, which is often challenging. We present SEditor, a two-policy approach that learns safety editor policy transforming potentially unsafe actions proposed by utility maximizer policy into safe ones. The safety editor is trained maximize the constraint reward while minimizing hinge loss of the utility state-action values before and after an action is edited. SEditor extends existing safety layer designs that assume simplified safety models, to general safe RL scenarios where the safety model can theory be arbitrarily complex. As first- order method, it easy implement and efficient for both inference and training. On 12 Safety Gym tasks and safe racing tasks, SEditor obtains much a higher overall safety-weighhed-uriike (SWU) score than the baselines, and demonstrates outstanding utility performance with constraint violation rates as low as once per 2k time steps, even obstacle-dense environments. On some tasks, this low violation rate is up to 200 times lower than that of an unconstrained RL method with similar utility performance. Code is available at wtthttts//// //gi thub. www.cooooomccaiaeeeecce sedittrrrrrrrrrr
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

1 Introduction
Safety has been one of the major roadblocks in the way of deploying reinforcement learning (RL.) to the real world. Although RL for strategy and video games has achieved great successes (Silver et al.) 2016; Vinyals et al. 2019 Berner et al. 2019), the cost of executing actions that lead to catastrophic failures in these cases is low: at most losing a game. On the other hand, RL for control has also mostly been studied in virtual simulators (Brockman et al., 2016 Tassa et al. 2020). Aside from the data collection consideration, to circumvent the safety issue (e.g. damages to real robots and environments) is another important cause of using these simulators. When RL is sometimes applied to real robots (Kim et al. 2004 Levine et al. 2016 OpenAl et al., 2019), data collection and training configured in restrictive settings to ensure safety. Thus to promote the deployment of RL to more real-world scenarios, safety is a critical topic. In this paper, we study a safe RL problem of maximizing utility with extremely low constraint violation rates.
There are two general settings of safe RL. Some existing works make the assumption that the environment safety model is a known prior. The agent is able to query an oracle function to see if any state is safe or not, withoul actually visiting that state. This includes having access to a well calibrated dynamics model (Berkenkamp et al. 2017, Chow et al., 2018 Thomas et al., [021]) or the set of safe/unsafe states (Turchetta et al. 2020, Luo and Ma| 2021: Li et al. 2021), Or they assume that a pre-training stage of safe/unsafe states or policies is performed on offline data, and then the learned safety knowledge is transferred to main tasks (Dalal et al. 2018: Miret et al. 2020; Thananjeyan et al. 2021). With these assumptions, they are able to achieve few even zero constraint violations during online exploration. However, such assumptions also put restrictions on their applicable scenarios.
This paper focuses on another setting where no prior knowledge or pre-training of safety models is assumed. The agent only gets feedback on which states are unsafe from its exploration experience. In other words, it has to learn the safety model (implicitly or explicitly) from scratch and the constraint budget can only be satisfied asympiotically. Although we cannot completely avoid constraint violations during online exploration, this setting can be especially helpful for sim-to-real transfer (Zhao et al.) 2020). or when the training process can be protected from damages. Prior works combine model-frec RL (Bhatnagar and Lakshmanan [012 Ray et al. [2099) Tessler et al.)
Bohez et al.) 2019 Stooke et al. | et al. |020]] Qin et al. 2021) or model-based
RL (As et al. 2022) with the Lagrangian method (Bertsekas 1999). This primal-dual optimization dynamically adjusts the weight of the constraint reward relative to the utility reward, depending on how well the violation rate target is being met. A single policy is then trained from weighted combination of the utility and constraint rewards. However, reconciling utility maximization with constraint violation minimization usually poses great challenges to this single policy.
In this paper, we present SEditor, a general safe RL approach that decomposes policy learning across two polices (Figure []. The utility maximizer (UM) policy is only responsible for maximizing the utility reward without considering the constraints. Its output actions are potentially unsafe. The safety editor (SE) policy then transforms these actions into safe ones. It is trained to maximize the constraint reward while minimizing a hinge loss of the utility state-action values before and after an action is edited. Both UM and SE are trained in an off- off-policy manner for good sample efficiency. Our two- policollcy paradigm is largely inspired by existing safety layer designs (Dalal et al.) 2018) Pham et al.) 2018) Cheng et al., 2019 Li et al. (02T) which simplify (e.g., to linear or quadratic) environment safety models. The high-level idea is the same though: modifying a utility- mixlit-maxiiiziin action only when necessary.
We evaluate SEditor on 12 Safety Gym (Ray ct al. 2019 tasks and safe car racing tasks adapted from Brockman et al. (2016), targeting very low violation rates. SEditor obtains much higher overall waiitty- outstanding utility performance with constraint violation rates as low once per 2k time steps, even in obstacle-dense environments. Our results reveal that the two-policy cooperation is critical, while simply doubling the size of a single policy network will not lead to comparable results. The choices of the action distance function and editing function are also important in certain circumstances. In summary, our contributions are:
(SWU) score (defined in Section than four baselines. It demonstrates
a) We extend the existing safety layer works to more general safe RL scenarios where the environment safety model can in theory be arbitrarily complex.
2

---

b) We present SEditor, first-order, -osy-to-implement approach that is trained by SGD like most model-free RL methods. It efficient during both inference and training, as it does not solve multi-step inner-level optimization problem when transforming
unsafe action into safe one. When measuring the distance between an action and its edited version, we show that in some cases the hinge loss of their state-action values is better than the usual L.2 distance (Dalal et al. 2018; Pham et al. 2018 Li et al., 2021) in the action space, We further show that an additive action editing function introduces effective inductive bias for the safety editor.
d) We propose the -afety-wiighted-ue method. The score is soft indicator of the dominance defined by Ray et al. (2019).
(SWU) score for quantitatively evaluating a safe RL
e) Finally, we achieve outstanding utility performance even with an extremely low constraint violation rate (5 x 10 +) in dense-obstacke environments. On some tasks, this low violation rate is up to 200 times lower than that of unconstrained RI. method with similar utility performance.
2 Preliminaries
The safe RL problem can be defined as policy search in a constrained Markov decision process (CMDP) (S, A. p. r). The state space S and action space A are both assumed to be continuous. The environment transition function p( (5,, ae) determines the probability density of reaching $141 E S after taking action mines the probability density of an episode starting at state 8p- Both p($t+1 [8t, dz) and μ(50) are usually unknown to the agent. For every transition (st, ae. St+1), the environment outputs scalar r(st, Ct, St+1) which we call the utility reward. Sometimes one uses the expected reward of taking at at 8t as r(8t, at) - Eecyr (pes,ep( r(8e, ae; 8t+1) for a simpler notation. Similarly, the environment also outputs a scalar c(St, ae) as the cost. To unify the reward and cost notations, we define the constraint reward ra(8), ax) - -c(st, a+) S O, and the CMDP becomes (S, A. p. re). For both utility and constraint rewards, the value is the higher the better. Finally, we denote the agent's policy as π (ax]ss) which dictates the probability density of taking ag at 8t-
A at state E S. The initial state distribution ju( (5)) deter-
For each 8t, the utility state value of following π is denoted by V (st) = Ex.p Er= 70 "(((s de), and the utility state-action value is denoted by Q" (st, az) r(t, (z) + EaIIi V* ) where E [0, 1) is the discount for future rewards. Similarly, we can define Vir and Q for the constraint reward. Then consider the safe RL objective:
E V (5o),
s.t.
E V.T C220
where C 2 O is the constraint violation budget. It might be unintuitive to specify C' for a discounted return, so one can rewrite
E
(r a.) +c)
(2)
M.T-P
and specify the per-step budget e instead, relating to by Ei x - C. Note that is not strictly imposed on every step. Instead, is only in the average sense (by a discount factor), treated as the violation rate target.
The Lagrangian method (Bertsckas 1999) converts the constrained optimization problem Eq. [ an unconstrained one by introducing a multiplier A:
into
V. (8o) + +c)
min max X
E -000p
(*((s) +
EE <00=
(3)
Intuitively, it dynamically adjusts the weight À according to how well the constraint state value satisfies the budget, by evaluating (approximately) the difference
A 2 E
VT (so)
(so))
+C= E
(re(se az) + +c,,
(4)
Sprup
P.T.P
which is the gradient of À given π in Eq. 3.
When optimizing ι given X, Eq. becomes
max
E (V*(s)) + AV. AV = max E
of (r(se, ar) + Arr(st,
x
1=0
3

---

Thus À can be seen as the weight of the constraint reward when it is combined with the utility reward to convert multi-objective RL into single-objective RL. This objective can be solved by typical
model-free RL algorithms.. For practical implementations, previous works (Ray et al. 2019] [Tsssler jet al. 2019; Bohee et al. 2019) usually perform gradient ascent on the parameters of n and gradient descent on À simultaneously, potentially with different learning rates.
3 Approach
We consider a pair of cooperative policies. The first policy utility maximizer (UM) denoted by Tg. optimizes the utility reward by proposing preliminary action T-(sss) which is potentially unsafe. The second policy safety editor (SE) denoted by Th. edits the preliminary action by Aa ~ (-1ss â) ensure safery, and the result action a h(ā, Aa) is output to the environment, where h represents an editing function. Note that we condition SE on UM's output à. Together the two policies cooperate to maximize the agent's 's utility while maintaining a safe condition (Figure D For simplicity, we will denote the overall composed policy by
Motivation. SEditor decomposes difficult policy learning task that maximizes both utility and safety, into two easier subtasks that focus on either utility or safety, based on the following considerations:
a) Different effective horizons. In most scenarios, safety requires either responsive actions (Dalal et al, 2018), or planning a number of steps ahead to prevent the agent entering non-recoverable states. Thus SE's actual decision horizon could be short depending on the nature of the safety constraints. This is in contrast to UM's decision horizon which is usually long for goal achieving. In other words, could expect the optimization problem of SE to be casier than that of UM, and SE has a chance of being learned faster if separated from UM.
b) Guarded exploration. From the perspective of UM, its MDP (precisely, action space) is altered by SE. UM's actions are guarded by the barriers set up by the SE. Instead of UM being discouraged for an unsafe action (i.e., punished with negative signal), SE gives suggestions UM by redirecting the unsafe action to a safe but also utility-high action to continue its exploration. This guarded exploration leads to better overall exploration strategy because safety constraints are less likely to hinder UM's exploration (Figure fillustration..
Objectives. We employ an off-policy actor-critic setting for training the two policies. Given overall (s, a) parameterized
policy Tyoo. we can use typical TD backup to learn QT (s, a) and Q Q as Q(s, a; 8) and Qe(s, a; 0) respectively, where we use @ to collectively represent the network parameters of the two state-action values. Given 8t+1 ~ p(-1s;, at) and ac+1 ~ x-00(-18(11), the Bellman backup operator (point estimate) for the utility state-action value is
(5) with the backup operator for the constraint state-action value Qe defined similarly with re. Both ((s, a; 0) and Qo(s, a, 0) can be learned on transitions (St, ars St+1) sampled from replay buffer.
TRacs ((s ae; 0) = r(st, +
For off- off-policy training of and vh, we first transform Eq. 3 intt bi-level optimization surrogate as:
max 6.10
E 8~~ D.a.Daaaaa (-1e)
(Q(s, (Q((s,a;0))
(b)
min XA AArooa X20
D denotes replay buffer and A A is defined by Fq. We basically have a historical marginal state distribution for training the policies, but still use the mitial state distribution P for training X. The motivation for this difference is, when tuning A, we should always care about how well the policy satisfies our constraint budget starting with μ but not with some historical state distribution.
We continue transforming the off-policy objective (Eq, 6 a) into two,
[Q(s, 0)). (b)
(a)
max
E B (-1e) (anto(-199,)) a=h(a,Aa)
max B (- Sanas(-asd a=((ā,An)
E
where d(aa, à) is a distance function measuring the change from to a. It is not necessarily propor- tional to Aa because the editing function hi could be nonlinear. The role of SE ne is to maximize the constraint reward while minimizing some distance between the actions before and after the modification. The role of UM To is to only maximize the utility reward. However, it only can only do

---

i
día a.
@a a.
AQu(s. a;8)
The computational graph of Eq. 7 Nodes denote variables and edges denote operations. The green blocks are negative losses, the blue paths are the (reversed) gradient paths of o, and the orange paths are the (reversed) gradient paths of tb. Paths in black or blue are detached for t, and paths in black or orange are detached for Φb
Figure 2:
so through the lens of SE T.. In other words, SE has actually changed the action space (also MDP) of UM. This property corresponds the motivation of guarded exploration mentioned earlier.
We would like to emphasize that, unlike the safety layer (Dalal et al. 2018) which projects unsafe actions based on instantancous safety costs, the training objective (Eq. 7 b) for SE relies on safety critic Qe which is learned as the expected future constraint return. Therefore, maximizing this critic will take into account long-tern term safety behaviors. In other words, if there are non-recoverabbe states where SE can't 't do anything to ensure safety, it will edit the agent's 's actions long before that to avoid entering those states.
Action editing function. We choose the editing function be mainly additive and non-parametric. Without loss of generality, we assume a bounded action space [- A, AJM and that both à and Aa are already in this space. Then we define a hh Aa) = min (max + 2da, A); A), where min and max are element-wise The multiplication by 2 and the clipping make sure that E [-A, A|M namely, SE has a full control over the final action in that it can overwrite UM's action if necessary. Because SE can be arbitrarily complex, its output Aa could depend on the current state and the action proposal à in an arbitrarily complex way. Thus even though the additive operation is simple, the overall editing process is already general enough to represent any modification.
This additive editing function motivated by constraint sparsity. Usually, constraint violations are only triggered for some states. Most often, the action proposal by UM is already safe if the agent is far away from obstacles. To explicitly introduce this inductive bias, we use the additive editing function which ensures that the majority of SE's modifications are close to O. This makes the optimization landscape of SE easier (see Figure 12 Appendix F for some empirical observations).
Distance function. Prior works on safety layer design, such as Dalal et al. (2018); Pham et al. (0118) Li et al. (202I), set the distance function d(.,-) the L2 distance, Later will show that L2 not always the best option. Instead, we use the hinge loss of the utility state-action values of â and a:
d(a, à) _ max Q(s, à; 0) (s, a; 0))
This loss is zero if the edited action a already obtains a higher utility state-action value than the preliminary action à. In this case, only the constraint Q is optimized by gT. Otherwise, the inner part of Eq. (b) is recovered as Q(s, a; 0) + (s, a; 0), as we can drop the term -Q(s, a; 0) due to its gradient w.z.. 2/ being zero. Our distance function in the utility Q space is more appropriate than the L.2 distance in the action space, because eventually we care about how the utility changes after the action is edited. The L2 distance between the two actions is only an approximation to the change, from the perspective of the Taylor series of ((,,, a; 0).
Evaluating ATvOS' Given batch of rollout experiences ((n,, N following Tooo, we approxi- mate the gradient of X (Eq. as
N
1
re(Sn; an) +
N
where e is the violation rate target defined in Eq. Namely, after every rollout, we collect batch of constraint rewards, compare each of them to cc and use the mean of differences to adjust X. This approximation allows us to update À using mini-batches of data instead of having to wait for whole episodes to finish, or having to rely on estimated V which is usually not accurate. To reduce temporal correlation (which might affect the constraint evaluation) in the rollout batch data, we use multiple parallel environments (Appendix H). This rollout batch is also the one that will be put into the replay buffer.
5

---

Training, To practically train the objectives, we apply SGD to Eq. (b) and Eq. simultaneously resulting in first-order method for approximated bi-level optimization (Likhosherstov et al. (202T) SGD is made possible by applying the re- ee-parameterization trick (Appendix G to both à ~ a and Aa ~ Tigh computational graph of Hq. [ is illustrated in Figure 2 We would like to highlight that a big difference between our SE and the previous safety layers (Dalal et al., 2018 Pham ct al. 2018) is that SE directly uses a feedforward prediction Aa ~ T (Eq. 7 a) to replace the optimization result of the objective of transforming unsafe actions into safe ones. In contrast, previous safety layers construct slosed-form solutions or inner-level differentiable optimization steps (e.g., quadratic programming (Amos and Kolter, 2017)) for simplified safety models at every inference step. Our overall approach generalizes to various constraint rewards (safety models) and action distance functions. Finally, to encourage exploration, we add the entropy terms of n and T.. into Eq. 7 with their weights dynamically adjusted according to two entropy targets following HHarnoja et al. (2018).
Both UM and SE are trained from scratch in our experiments. Here we discuss an alternative situation where an existing policy pre-trained to maximize utility, and we use it as the initialization for UM. In this case, UM cannot be frozen because its MDP (and hence its optimum) will be changed by the evolving SE. Instead, it should be fine-tuned to adapt to the changing behaviors of SE. Potentially, a pre-trained UM will speed up the convergence of SEditor. We will leave this to our future work.
4 Experiments
Baselines. We compare SEditor with four baselines.
PPO-Lag combines PPO (Schulman et al. 2017) with the Lagrangian method, as done in Ray et al.) (2019). Since PPO is an on-policy algorithm, we expect its sample efficiency to be much lower than off-policy algorithms.
FOCOPS (Zhang et al., (020) is analogous to PPO-Lag with two differences: 1) there is no clipping of the importance ratio, and 2) a KL divergence regularization term with fixed weight is added to the policy improvement loss, with an early stopping when this term averaged over rollout batch data violates the trust region constraint.
SAC-actor2x-Lag combines SAC (Haarnoja et al. 2018) with the Lagrangian method. Similar to SEditor, SAC-actor2x L-aaa trains its policy on states sampled from the replay buffer but trains states generated by the rollout policy. Its gradient of À is also estimated by Eq. 9 The policy network size is doubled. This is to match the model capacity of having two actors in SEditor. SAC serves as unconstrained optimization baseline to calibrate the utility return.
We choose not to compare with second-order CMDP approaches such as CPO (Achiam al. 2017) or PCPO YYang et al. (2200) because they require (approximately) computing the inverse of the Fisher information matrix, which is prohibitive when the parameter space is large. Especially for our CNN based policies, second-order methods are impractical.
To analyze the key components of SEditor, also evaluate two variants of for ablation studies.
SEditor-L2 defines d(a, à) ف ||a all as done in most prior works of safety layer. SEditor-overwrite makes SE directly overwrite UM's action proposal by a = h(āa, Da) = Aa.
All compared approaches including the variants of SEditor, share common training configuration (e.g., replay buffer size, mini-batch size, learning rate, etc) as much as possible. Specific changes are made to accommodate to particular algorithm properties (Appendix HH).
Evaluation metric. Following Ray al. 2211)) one method dominates another "it strictly improves on either return or cost rate and does at least as well on the other". Accordingly, we emphasize that the utility reward constraint violation rate should never be compared in isolation, as one could easily find a method that optimizes either metric very well. Particularly in the experiments, we set the learning rate of the Lagrangian multiplier À much larger than that of the remaining parameters. This ensures that any continued constraint violation will lead to quick increase of X and drive the violation rate back to the target level. This conservative strategy makes some compared methods have similar constraint violation curves, but vastly different utility performance. For quantitative comparison of their final performance, we compute the weighreewwttttuttiiin (SWU) scores of the compared methods towards the end of training. The score is a soft indicator of dominance, and it is a product of two ratios:
SWU e min {1. ConstrainnVVollationRaan
UtilityScore
ConstraintViolationn.aa
UtilityScore UncomstrainedRL

---

SAC
PPO-Lag
FOCOPS
SAC-actor2x-Lag
SEditor
PointFush1
PointGoal1
1.
0.
18
0
10"
12
0.
0
03
03
0.8
0.
PointButton1
PointPush2
0.6
PointGool2
PointButton2
13
15
0.00
0.55
0.58
0.15
1.00
0.00
0.5
0.58
0.15
1.00
0.00
0.25
350
3.75
1.00
0.30
025
050
0.5
100
Environment steps (56+
SAC
PPO Lag
FOCOPS
SAC www.aactor2xxe Lag
SEditor
CarRush1
Cartooa12
135
15"4
13
15
18
12"
124
3.00
0.00
1.00
0.2
1.00
1.2
100
10"
0.8
189
0.
10-1
13-
0.6
0.
0.3
03
0.8
0.0
CarGoal2
CarButton2
Environment steps (5e+006)
Figure 3: The training curves of the 12 Safety Gym tasks. Odd columns: 1 success rate. Even columns: constraint violation rate (log scale). Red dashed horizontal lines: violation rate target c= 5 10-4. Shaded areas: 95% confidence interval (CI).
The reason for having min(1, -} is because we only care when the violation rate is higher than the target. In practice, UtilityScore can be the episodic utility return (assuming positive) or the success rate. We choose StirityScoreeac as UtllityScorr and ConstraintViolationRe are averaged over the last TO of the training steps to reduce variances.
our experiments, both UtilityScore
Safety Gym tasks. In Safety Gym (Ray et al. 2019), robot with lidar sensors navigates through cluttered environments to achieve goals. We use the POINT and CAR robots in our experiments. Either of them has three tasks: GOAL, BUTTON, and PUSH. Each task has two levels, where level has more obstacles and larger map size than level 1. In total we have 3 x Whenever an obstacle is in contact with the robot, constraint reward of -1 is given. Thus the constraint violation rate can be calculated as the negative average constraint reward. The utility reward is calculated as the decrement of the distance between the robot (GOAL and BUTTON) box
= 12 tasks.

---

Salety Gym
Sale
Racing SRO
Overall
Improvementt
CP1
CGI
CB1
CP2
CG2
CB2
PP1
PG1
PB1
PP2
PG2
PB2
SR
SAC
0.02
0.02
001
0.01
001
0.01
0.05
0.02
0.01
0.04
0.01
0.11
0.14
0.03
0.03
3567%
0.01
0.03
0.00
0,04
2650%
POCOPS
0.03
0.28
0.03
0.03
0.20
0.05
0.02
0.32
0.10
0.01
0.26
0.09
0.04
0.04
0.11
9006
AcC-actor2x--ag
0.74
0.63
0.70
0.59
1.00
0.81
0.60
1.00
0.89
0.37
0.94
0.64
0.37
0.08
0.67
64%
SEditor
1
0,94
0.78
1.49
0.99
0.95
1.55
1.00
1,00
1.78
1.00
1.00
1.28
0.57
1.10
Table 1: The SWU scores of different methods. Task name abbreviations: CP- CARPUSH, CG
CARGOAL, CB CARBUTTON, PP- POINTPUSH, PG POINTGOAL, PB POINTBUTTON, SR
SAFERACING, and SRO- SAFERACINGOBSTALLI
BAC
Figure 4: Visualization of rollout trajectories different training stages. For each method from left to right, the trajectories are generated by checkpoints at 10%, 20%, 30%, and 50% of the training process. On each trajectory, time flows from cyan to purple. A red cross denotes failed trajectory while a green cross + denotes a successful trajectory. Each bird s-eye map is a sketch of the environment layout. The small black dot the robot's initial position. The big green circle the goal location and the other shapes are obstacles. In summary, SEditor has better exploration strategy because the barriers set up by SE can redirect UM's unsafe actions continue its exploration, while S (--actor2x-Lag' exploration will be hindered in when encountering similar obstacles.
(PUSH) and the goal at every step. We define a success as finishing a task within time limit of 1000. We emphasize that the agent has no prior knowledge of which states are unsafe. The map layout is randomized at the beginning of each episode. This randomization poses a great challenge to the agent because it has to generalize the safety knowledge to new scenarios.
We customize the environments to equip the agent with a more advanced lidar sensor in two ways:
i) is a natural lidar instead of a pscudo one used by Ray et al. (019). The natural lidar simulates ray intersecting with an object from origin. The pseudo lidar can only detect object centers, while the natural lidar reveals shape information of objects, which is required to achieve a very low constraint threshold.
ii) Our lidar has 64 bins while the original lidar has only 16 bins. The 16-bin pseudo lidar's 's low precision is bottleneck of achieving much fewer safety violations. Without perception precision, the problem is ill ill-posed for a robot suffering from blind spots to avoid touching objects at all.
We refer the reader to Appendix A.I for more details of the environment and tasks. Our customized Safety Gym is available at htttps://gttiie
wwwm http///yyusaaftt-eeeeee gym
We set the constraint violation target 5 x 10-4, meaning that the agent is allowed to violate any constraint only once every 2k steps on average. This threshold is only 5n of the threshold e = 0.025 used by the original Safety Gym experiments (Ray et al.) 2019), highlighting the difficulty of our task. Each compared approach trained with 9 random seeds. The training curves are in Figure 3 and the SWU scores are in Table We see that SEditor obtains much higher SWU scores than the baselines on the 12 tasks, while being comparable on the rest. While SAC-actor2x-Las performs well among the baselines, its double-size policy network does not lead to results comparable to SEditor. This suggests that SEditor does not simply rely on the large combined capacity of two policy networks to improve the performance, instead, its framework in Figure I FOCOPS missed all the constraint violation rate targets. The unconstrained baseline SAC obtains better success rate sample efficiency than SEditor with the CAR robot, but violates constraints much more. Surprisingly, SAC is worse than SEditor with the POINT robot regarding success rates, even without constraints. Finally, we highlight that towards the end of training IA CARPUSH2, and CARGOAL2, SEditor violates constraints up to 200 times less than SAC does, while achieving success rates on par with SAC (black curves vs. wy.dashed blue curves)!
matters. Both PPO-Lag and
To analyze the exploration behaviors of SAC, SAC-actor2x-Lag PPO-Lag, and SEditor, we visualize their rollout trajectories on example map of POINTGOAL2, at different training stages (the
8

---

training is done on randomized maps but here we only evaluate on one map). For each approach, the checkpoints at 10%, 20%, 30%, and 50% of the training process are evaluated and 100 rollout trajectories are generated for each of them. The trajectories are then drawn on bird's s-eye sketch map (Figure F. We can see that although SAC is able to find the goal location in the very beginning. it tends to explore more widely regardless of the obstacles, and its final paths ignore obstacles. SAC-actor2x-Lag learns to respect the constraints after 10% of training, but the obstacles greatly hinder its exploration: the trajectories are confined in small region. As result, it takes some time for it to find the goal location. PPO-Lag is slower at learning the constraints and it even hits obstacles at 50%. SEditor is able to quickly explore regions between obstacles and refine the navigation paths passing them instead of being blocked.
Our ablation studies (Appendix Figure 8) show that, in this particular safe RL scenario, SEditor not sensitive to the choice of distance function: SEditor-L22 achieves similar results with SEditor. However, the editing function does make a difference: SEditor-overwrice is clearly worse than SEditor in terms of utility performance. This shows that the inductive bias of the edited action a being close to the preliminary action à is very effective (see Appendix F. Our two-stage "propose-and-ede" strategy is more efficient than outputting an action in one shot.
Safe racing tasks. Our next two tasks, SAFERACING and from the unconstrained car racing task in Brockman et al. (016). The goal of either task is to finish a racetrack as fast as possible. The total reward of finishing a track is 1000, and it is evenly distributed on the track tiles. In SAFERACING, the car has to stay on the track and receives constraint reward the car receives a constraint
are adapted
of 11 whenever driving outside of it. In SAFER reward of 1 if it hits an obstacle on the track, but there is no penalty for being off-track. An episode finishes after every track tile is visited by the car, or after 1000 time steps. The track (length and shape) and obstacles (positions and shapes) are randomly generated for each episode. As in Safety Gym, this randomization could expose a never-experienced safety scenario to the agent at any time. The agent's observations include a sird's-eye view image and a car status vector. Note that this high-dimensional input space usually poses great challenges to second-order safe RL methods. Again, the constraint violation rate can be computed as the negative average constraint reward. We set the undiscounted episode return as the UtilityScore. Each compared approach is trained with 9 random seeds. The training curves shown in Figure and SWU scores in Table [ SEditor has much higher SWU scores than all baselines. Moreover, it is the only one that satisfies the harsh violation rate target towards the end of training. Surprisingly, even with constraints SEditor gets much better utility return than SAC on SAFERACING. One reason is that without the out-of-track penalty as racing guidance, the car easily gets lost on the map and collects lots of meaningless timesteps. Although SAC gets much higher return on SAFER AENEEGRCIIOOBSTTA studies show results that are complementary to those on the Safety Gym tasks (Appendix Figure (). Now the action distance function makes big difference. Changing it to the L2 distance greatly impacts the utility return, especially on SAFERACINGOBSTAAR made. This demonstrates that the closeness of two actions does not necessarily reflect the closeness of their state-action values (Appendix H
constraint violation rate target 5 x 10 4. For evaluation,
it greatly violates the constraint budget. Interestingly, the ablation
where almost no improvement is
5 Related Work
Safe RL is closely related to multi-objective RL (Roijers et al. 2013), where the agent optimizes a scalarization of multiple rewards given a preference (Van Moffaert et al. 2013), or finds a set of policies covering the Pareto front if no preference is provided (Moffaert and Nowé 2014). In our
case, the preference for the constraint reward is always changing, because we try to maintain it to a certain level instead of maximizing it.
Trust-region methods (Achiam al. 2017 Chow et al. 2019; Yang et al., 2020 Liu et al.) 2022) for solving CMDPs put constraint on the KL divergence between the new and old policies, where the KL divergence second-orde -oderr approximated. For solving the surrogate objective at each iteration, the inverse of the Fisher information matrix is approximated.
Liu et al. (202)) decomposes the policy update step of CPO (Achiam et al., (017)) into two steps, where the first E-step computes a non-parametric form of the optimal policy and the second M-step distills it to parametric policy. These methods often involve complex formulations and high computational costs. In contrast, SEditor is a conceptually simple and computationally efficient first-order method.
9

---

SAC
PPO Lag
FOCOPS
SAC actor2x Lag
SEditor
SafeRacing
SafeRacingO::lade
0.00
0.2
0:50
813
3.0
0.00
923
0.50
3.00
025
355
575
1.00
0.00
4.2
6:50
975
1.00
Environment steps (5e+06)
Environment steps (1e+07)
Figure 5: The training curves of the safe racing tasks. Odd columns: t undiscounted episode return. Even columns: &. constraint violate rate (log scale). Red dashed horizontal lines: violation rate target c = 5 x 10-4. Shaded areas: 95% confidence interval (CI).
The line of works closest to ours is safety layer (Dalal et al. 20188] Pham et al. 2018 Cheng et al., 2019; [i et al., 202T). Although they also transform unsafe actions into safe ones, their safety layers assume simple safety models which can be solved in closed forms by inner-level optimization (e.g., quadratic programming (Amos and Kolter 2017 We extend this idea to more general safe RL scenarios without assuming particular safety model. Our SE can be also treated as teacher under the teacher-student framework, where UM is the student whose actions are corrected by SE. However, existing works (Turchetta et al.) [220 Langlois and Everitt 2021) under this framework more less rely on various heuristics and task-specific rules, or have access to the simulator's s internal state. In complex realistic environments, such assumptions are invalidated.
Recovery RL (Thananjeyan et al. 202T) also decomposes policy learning across two policies. makes a hard switch between task policy and a recovery policy by comparing a pre-trained constraint critic with a manual threshold. requires an offline dataset of demonstrations to learn the constraint critic and fix it during online exploration. This offline training requirement and the hard switch scheme reduce the flexibility of Recovery RI. DESTA (Mguni al.) 202T) fully decouples utility from safety by introducing two separate agents. The safety agent can decide at which states it takes control of the system while the task agent can only maximize utility at the remaining states. There is hardly any communication or cooperation between the two agents. In contrast, SEditor lets UM and strategy, instead of switching between them. Flet-Berliac
SE cooperate with a "propose-and and Basu (2022) also adopts two-policy design. However, their two policies are adversarial. More specifically ly, one of their policies tries to intentionally maximize risks by behaving unsafely, to shrink the feasibility region of the agent's value function. Their overall policy will bias towards being conservative. Our SE minimizes risks, and UM and SE always cooperate to resolve the conflict between utility and safety.
Limitations and Conclusions
While SEditor is able to maintain an extremely low constraint violation rate in expectation, oscillation does happen occasionally on the training curves around the target threshold (Figure 33. In general, if no oracle safety model can be accessed by the agent, avoiding this oscillation issue would be very difficult. On one hand, our tasks always generate new random environment layouts for new episodes, and in theory the agent has to generalize its safety knowledge (e.g., one obstacle being dangerous at this location usually implies it being also dangerous even when the background changes). On the other hand, neural networks have the notorious issue of catastrophic forgetting, meaning that while the agent learns some new safety knowledge, it might forget that already learned. Finally, from practical point of view, as the violation target reaches O, it takes an exponential number of samples to evaluate if the agent truly can meet the target. These challenging issues are currently ignored by SEditor and would be interesting future directions.
In summary, we have introduced SEditor, for general safe RL scenarios, that makes a safety editor policy transform preliminary actions output by utility maximizer policy into safe actions. In problems where utility maximizing action is often safe already, this architecture leads to faster policy learning. We train this two-policy framework with simple primal-dual optimization, resulting in an efficient first-order approach. On 14 safe RL tasks with very harsh constraint violation rates, SEditor achieves an outstanding overall SWU score. We hope that SEditor can serve as a preliminary step towards safe RL with harsh constraint budgets.
10

---

References
Achiam, J., Held, D., Tamar, A., and Abbeel, P. (2017). Constrained policy optimization. ICML.
Amos, B. and Kolter, J. Z. (2017). Optnet: Differentiable optimization as a layer in neural networks. In ICMI..
As, Y., Usmanova, I., Curi, S., and Krause, A. (2022). Constrained policy optimization via bayesian world models. In International Conference on Learning Representations.
Berkenkamp, F., Turchetta, M., Schoellig, A. P., and Krause, A. (2017). Safe model-based reinforce- ment learning with stability guarantees. In NeurIPS.
Berner, C.,, Brockman, G., Chan, B., Cheung, V., Debiak, P., Dennison, C., Farhi, D., Fischer, Q. Hashme, S., Hesse, C., Józefowicz, R., Gray, S., Olsson, C., Pachocki. J., Petrov, M., de Oliveira Pinto, H. P., Raiman, J., Salimans, T., Schlatter, J., Schneider, J., Sidor, S., Sutskever, I., Tang, J., Wolski, F., and Zhang, S. (2019). Dota 2 with large scale deep reinforcement learning. arXiv.
Bertsekas, D. (1999). Nonlinear Programming. Athena Scientific.
Bhatnagar, S. and Lakshmanan, K. (2012). An online actor-criti critic algorithm with function approxima- tion for constrained markov decision processes. Journal of Optimization Theory and Applications, 153.
Bohez, S., Abdolmaleki, A., Neunert, M., Buchli, J., Heess, N., and Hadsell, R. (2019). Value constrained model-free continuous control. arXiv.
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). Openai gym.
Cheng, R., Orosz, G., Murray, R. M., and Burdick, J. W. (2019). End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In AAA!
Chou, P.W.., Maturana, D., and Scherer, S. (2017). Improving stochastic policy gradients in con- tinuous control with deep reinforcement learning using the beta distribution. In ICML, pages 834-843 843.
Chow, Y., Nachum, O., Guzman, E., and Ghavamzadeh, M. (2018). A lyapunov-based approach safe reinforcement learning. NeurIPS.
Chow, Y., Nachum, O., Faust, A., Duenez-Guzman, E., and Ghavamzadeh, M. (2019). Lyapunov- based safe policy optimization for continuous control. arXiv.
Dalal, G., Dvijotham, K., Vecerík, M., Hester, T., Paduraru, C., and Tassa, (2018). Safe exploration in continuous action spaces. CoRR.
Flet-Berliac, Y. and Basu, D. (2022). SAAC: safe reinforcement learning as an adversarial game of actor-crities. In Conference on Reinforcement Learning and Decision Making.
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., and Levine, S. (2018). Soft actor-critic algorithms and applications. arXiv, abs/1812.05905.
Kim, H., Jordan, M., Sastry, S., and Ng. A. (2004). Autonomous helicopter flight via reinforcement learning. In Advances in Neural Information Processing Systems, volume 16.
Langlois, E. D. and Everitt, T. (2021). How agents behave when their actions are modified. In AAAI.
Levine, S., Pastor, P., Krizhevsky, A., and Quillen, D. (2016). Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. arXiv.
Li, Y., Li, N., Tseng, F.,, Girard, A. R., Filev, D., and Kolmanovsky, (2021). Safe reinforcement learning using robust action governor. In L4DC.

---

Likhosherstov, V., Song, X., Choromanski, K., Davis, J., and Weller, A. (2021). Debiasing first-order heuristic for approximate bi-level optimization. In ICML.
Liu, Z., Cen, Z., Isenbaev, V., Liu, W., Wu, Z., Li, B., and Zhao, D. (2022). Constrained variational policy optimization for safe reinforcement learning. In ICML.
Luo, Y. and Ma, (2021). Learning barrier certificates: Towards safe reinforcement learning with zero training-time time violations. In NeurIPS.
Mguni, D., Jennings, J., Jafferjee, T., Sootla, A., Yang, Y., Yu, C., Islam, U., Wang. Z., and Wang. J. (2021). DESTA: A framework for safe reinforcement learning with markov games of intervention. arXiv.
Miret, S., Majumdar, S., and Wainwright, C. (2020). Safety aware reinforcement learning (sarl). arXiv.
Moffaert, K. V. and Nowé, A. (2014). Multi-objectivv reinforcement learning using sets of pareto dominating policies. JAIR.
OpenAI, Akkaya, I., Andrychowicz. M., Chociej. M., Litwin, M., McGrew, B.. Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., Schneider, J.. Tezak, N., Tworek, J.. Welinder, P., Weng. L.. Yuan, Q., Zaremba, W., and Zhang, L. (2019). Solving rubik's s cube with robot hand. arXiv.
Pham, T.-H., Magistris, G. D., and Tachibana, R. (2018). Optlayer practical constrained optimization for deep reinforcement learning in the real world. In ICRA.
Qin, Z., Chen, Y., and Fan, C. (2021). Density constrained reinforcement learning. In ICML.
Ray, A., Achiam, J., and Amodei, D. (2019). Benchmarking Safe Exploration in Deep Reinforcement Learning.
Roijers, D. M., Vamplew, P., Whiteson, S., and Dazeley, R. (2013). A survey of multi-objective sequential decision-making. JAIR.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv.
Silver, D., Huang, A., Maddison, J. Guez, A., Sifre, L., den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanetot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap. T., Leach, M.. Kavukeuoglu, K., Graepel, and Hassabis. D. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):444489)
Stooke, A., Achiam, J., and Abbeel, P. (2020). Responsive safety in reinforcement learning by pid lagrangian methods. In JCML.
Tassa, Y., Tunyasuvunakool, S., Muldal, A., Doron, Y., Liu, S., Bohez, S., Merel, J., Erez, T., Lillicrap, T., and Heess, N. (2020). dm_control: Software and tasks for continuous control.
Tessler, C., Mankowitz, D. J., and Mannor, (2019), Reward constrained policy optimization. In ICLR.
Thananjeyan, B., Balakrishna, A., Nair, S., Luo, M., Srinivasan, K., Hwang. M., Gonzalez, J. E., Ibarz, J., Finn, C., and Goldberg, K. (2021). Recovery RL: safe reinforcement learning with learned recovery zones. ICRA.
Thomas, G., Luo, Y., and Ma, T. (2021). Safe reinforcement learning by imagining the near future. In NeurIPS.
Turchetta, M., Kolobov, A., Shah, S., Krause, A., and Agarwal, A. (2020). Safe reinforcement learning via curriculum induction. In NeurIPS.
Van Moffaert, K., Drugan, M. M., and Nowé, A. (2013). Scalarized multi-objective reinforcement learning: Novel design techniques. In 2013 IEFE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL).
12

---

Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H.,
Powell, R., Ewalds, T., Georgiev, P., Oh, J., Horgan, D., Kroiss, M., Danihelka, I., Huang. A., Sifre. L.., Cai, T., Agapiou, J. P., Jaderberg. M., Vezhnevets, A. S., Leblond, R., Pohlen, T., Dalibard, V., Budden, D., Sulsky, Y., Molloy, J., Paine, T. I.., Gulcehre, C., Wang. Z., Pfaff, T., Wu, Y., Ring, R., Yogatama, D., Wünsch, D., McKinney, K., Smith, O., Schaul, T., Lillicrap, T. P., Kavukcuoglu, K., Hassabis, D., Apps, C., and Silver, D. (2019). Grandmaster level starcraft ii using multi-agent reinforcement learning. Nature, pages 1-5.
Yang. T.-Y., Rosca, J., Narasimhan, K., and Ramadge, P. J. (2020). Projection- based constrained policy optimization. In ICLR.
Zhang. Y. Vuong, Q., and Ross, W. (2020). First order constrained optimization in policy space. In NeurlPS.
Zhao, W., Queralta, P., and Westerlund, (2020). Sim-to-real transfer deep reinforcement learning for robotics: a survey. In 2020 JEEE Symposium Series on Computational Intelligence (SSCI).
NeurIPS Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Section 6
(c) Did you discuss any potential negative societal impacts of your work? [N/A] The paper is about RL safety which strives to provide positive societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] No theoretical results.
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Appendix
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)? [Yes] For example, Figure 3 and Figure 5
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Appendix H
4. If you are using existing assets (e.g., code, data, models) or curating/releasnng new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 4 Appendix(A.) and Appendix A..).
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent
obtained from people whose data you'ne using/curating? [N/A] The data are open-sourced.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] We only use virtual simulators.
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
13

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "af54Y9gmC10", "reviewer": "Reviewer_iJnb", "summary": "This paper studies the problem of safe reinforcement learning in which the learning agent seeks to optimize the reward while minimally reducing the number of cost violations. The paper points out that there are two main commonly used assumptions in the field: the oracle function for verifying whether the state is safe or not, and the pre-trained safe policy that brings the agent to a safe state. However, these approaches make the algorithm hard to generalize to other applications. In this paper, the assumption is that the agent only sees whether the state is safe or not “only after” it visits that state. This makes the problem challenging and the learning agent needs to learn to be safe from scratch. As a result, the paper says that we can only satisfy the constraint “asymptotically”. The paper proposes a two-policy structure safe reinforcement learning algorithm: a utility maximizer (UM) that only optimizes the reward function without considering the safety constraints, and a safe editor (SE) that transforms the action proposed by UM into a safe action. SE is also a reinforcement learning agent with the goal to maximize the “constraint” reward while forcing the state-action values to be similar before and after editing. There are several contributions: (1) the proposed algorithm does not require an inner loop to edit the action to the safer one, which makes the algorithm easy to use; (2) they show that hinge loss is better than L2 when constraining the distance of actions before and after editing, and (3) they verify the algorithm in the experiment. The experiment results are as follows: (1) Table 1 and Figure 3 show the results of SWU scores and the learning curves over the number of steps. The paper points out that SEditor has a higher score than the baselines on 7 out of 12 tasks. (2) Figure 4 shows the visualization of rollout trajectories at different training stages. They show that SEditor is able to find the feasible path without being blocked, and finally (3) Figure 5 shows the result in the racing task.", "questions": "(1) In Figure 4, could you give the reader more intuition on why SEditor has a better exploration strategy? \n(2) Other than empirical observations, could you explain why hinge loss is better than L2 loss theoretically? \n(3) After we train UM, do we need SE during deployment. The reader thinks the answer is yes since UM never learns how to be safe. If this is the case, what is the absolute value of the $\\delta a$ along the training? I expect the $|\\delta a|$ would be the same across training. The reader is curious to see this plot.\n(4) How to apply the proposed algorithm in real-world application? For example, learning a policy for self-driving cars.\n(5) Please address the comment on number 2.\n\nI am happy to increase the score if the author answers the question well. Thank you.\n ", "limitations": "See the cons in the previous section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n(1) The paper is well-written.\n(2) The assumption presented in the paper is clear. For example, it clearly says that we can only learn to be safe asymptotically due to the lack of the oracle that tells whether the state is safe or not. This point is often missed in the literature.\n(3) The literature review is good. It covers the work from safe reinforcement learning, especially the work with a two-policy structure. \n\nCons:\n(1) As pointed out by the paper, the proposed approach has some similarities to recovery RL. The difference seems to be that recovery RL switches hardly from the learner policy to the safe policy whereas, in this paper, SE edits the action from UM by adding. \n(2) I generally agree that the method here is simple and effective based on the experiment. But there is no discussion about the failure case of the proposed algorithm. One possible failure case of the proposed algorithm is that the action proposed by UM may exceed the editing capability of SE. For example, if the learning agent enters the state called “near-termination state”, meaning that the agent has not died, but every possible action is no longer safe enough to recover the agent from this state. Then the proposed approach could fail.\n(3) There is no theory in the paper to support the claim that the agent will learn safely.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MwyCSmgfbrK", "reviewer": "Reviewer_MUd3", "summary": "This paper proposes a new method for tackling reinforcement learning with safety constraints with a two-policy approach. The first policy learns to maximize the overall return while the second policy edits actions coming from the first policy to ensure safety. The new algorithm was evaluated on the SafetyGym tasks and two additional safe racing tasks.", "questions": "One additional question I have aside from the points I mentioned in the main review. In discussing the limitations of your work, you mentioned that oscillation is still an issue that you observe, I wonder if you have considered or whether it's possible to integrate some of the ideas from Stooke et al. (2020) (referenced in your paper) which addresses the oscillation issue but for Lagrangian methods.", "limitations": "Yes, see above comment.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I like this paper a lot, the idea presented by the authors is simple and elegant yet appears to be quite powerful in practice. The paper is very clearly written and a pleasure to read. The authors clearly explained their methodology and motivated the different design choices they employed (e.g. Hinge loss vs. L2, additive edit function etc.). Experiments as well as detailed ablation studies are very well done and strongly support the authors' arguments. I think this paper makes a strong contribution to the field of safe RL and recommend this paper for acceptance at this venue. Additional detailed comments are as follows:\n- The paper argues why the additive edit function is preferred and the authors provide some solid observations for why this is the case. I was wondering if you could give a bit more intuition on why this is the case (especially why additive is better than overwrite).\n- Also a related question to the above, have you considered other forms of edit functions aside from additive and overwrite?\n- My biggest concern is with how $\\Lambda$ is evaluated. My understanding is that $\\Lambda$ is evaluated in an on-policy Monte Carlo fashion even though the overall algorithm is off-policy. I agree with the author's argument since off-policy evaluation is quite challenging and generally inaccurate, however I do have several questions:\n  - When you use a batch of size $N$, you mentioned that you don't necessarily wait for the episode to finish when doing evaluation. Does this mean each batch could contain multiple episodes or a partial unfinished episode?\n  - In general, larger $N$ is preferred from an accuracy perspective but is problematic in terms of sample efficiency. From a theoretical perspective you could get away with a smaller $N$ if the underlying MDP has a small mixing time. However I wonder in practice how did you chose $N$?\n  - Also for evaluating $\\Lambda$, did you have to collect a separate batch of data or was it the same data you collect to put into the data buffer? Since in the former case this does need to be taken into account when comparing to other baselines in terms of sample efficiency.\n- One suggestion in terms of presentation, Section 3 quite clearly explains the proposed approach, but since it does contain many different pieces, I believe it would be helpful to include some kind of pseudocode in either the main text or appendix to summarize Section 3.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AuHNqcEOtOA", "reviewer": "Reviewer_AoPc", "summary": "The authors consider the problem of safe reinforcement learning which has attracted a lot of attention in the recent years. While there are many algorithms to address the problem, they do typically have some limitations. One of the major concerns is that the algorithms are quite brittle and require significant tuning  as well as many interactions with the environment to solve the problem. The authors propose a modification of a recently proposed safety layer to solve some of the issues with safe RL. \n\nThe learning process is decomposed into three independent learning processes: a) scheduling the Lagrangian multiplier weighing  the estimated cost b) learning the utility policy maximizing the reward, and c) learning the projection onto the safety state. While the learning process a) is identical to the classical procedure for learning the  Lagrangian multiplier, the process b) and c) are novel.  Albeit, there were similar approaches in the past (e.g., [Mguni et al 2021])\n\nThe authors then conduct a number of experiments showcasing the performance of their approach in the case of small cost limits. ", "questions": "1. While the motivation for the hinge loss makes sense, estimates of the value functions can be quite noisy. I wonder if the authors made some steps to alleviate this issue. Further, would it make sense to consider the $l_2$ distance between the $Q$ functions instead of the hinge loss? Does it work worse?\n\n2. I suggest expanding the motivation behind learning the policy $\\pi_\\psi(\\cdot | s, a)$ predicting the update $\\delta a$ instead of learning the policy predicting the corrected action $\\hat a$, i.e.,  $\\hat a \\sim \\pi_\\psi(\\cdot | s, a)$.  That is, I recommend moving parts of the Appendix D into the main text.\n\n3. The choice of nonlinearity $h$ is a bit confusing, why $\\delta a$ is multiplied by two?\n\n4. Is it possible to elaborate if the correction policy takes into account the long term behavior of safe actions. This could be important since the safety layer by Dalal et al makes the correction decisions based on the instantaneous information (costs, states and actions)\n\n5. Please add absolute values for the returns in order to compare the performance to other works. Normalizing the returns unfortunately makes such a comparison impossible.\n\n6. While having an almost zero cost limit is an important problem to consider, it is important to compare the performance on the benchmarks as they were intended. Hence adding some experiments (say point push and/or car push) with the cost limit of $25$ is very important.\n\n7. It seems that the proposed improves sample efficiency of the Lagrangian SAC, but doesn't really improve the performance in many tasks. Hence, I suggest running the experiments for longer steps for SAC to make sure that the performance is indeed worse. \n\n8. The baselines are not unfortunately the latest in safe RL. I suggest comparing to [Stooke et al 2020]  and [As et al 2022]. The latter reference published numerical values of their experiments with the code. One could also consider [Liu et al 2022], but it was just recently published. \n\n9. Could the approach work with a pre-trained utility policy? That is, making an already existing policy safe. \n\nMissing references:\n* [Stooke et al 2020] Stooke, Adam, Joshua Achiam, and Pieter Abbeel. \"Responsive safety in reinforcement learning by PID Lagrangian methods.\" International Conference on Machine Learning. PMLR, 2020.\n* [As et al 2022] As, Yarden, et al. \"Constrained Policy Optimization via Bayesian World Models.\" ICLR 2022 \n* [Mguni et al 2021] Mguni, David, et al. \"DESTA: A Framework for Safe Reinforcement Learning with Markov Games of Intervention.\" arXiv preprint arXiv:2110.14468 (2021)\n* [Liu et al 2022] Liu, Zuxin, et al. \"Constrained variational policy optimization for safe reinforcement learning.\" arXiv preprint arXiv:2201.11927 (2022) \n\t\nNB: I would consider raising my score if some of these questions are answered, especially the concerns relating to the experiments section.", "limitations": "\n1. Providing some intuition would help to understand the approach better. \n2. I am not sure if anything can be done for the theoretical analysis, but it would be a quite welcome addition as well.\n3. It is necessary to consider the case of large cost limits.\n4. As discussed above, comparison to more baselines and the presentation of the existing results should be improved", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written and the presentation is great and has only a few minor problems. The experiments look impressive as well. Overall, I enjoyed reading this paper, but there are a few weaknesses in my point of view:\n \n1. Some of the design choices are not very clear. For example. it is not clear why separating the policy learning process into two makes sense. There's naturally some empirical evidence, but the splitting process is not very intuitive. I suggest providing an intuition why this approach works and how. \n\n2. Theoretical analysis is limited. In particular, I am a bit concerned that there's a an implicit assumption that transition between a policy maximizing the reward and a safe policy is continuous. \n\n3. Presentation of the experiments is quite impressive, but a few improvements are necessary (see questions).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5Y0fp1w6eAS", "reviewer": "Reviewer_hrVA", "summary": "This paper proposes a new and general approach for safe RL with relatively strong empirical evidence. The proposed approach leverages two policies, the first one which is a utility maximizer (UM), and the second one which is a safety editor (SE) in charge of adjusting the UM action to ensure a sufficiently low constraint violation rate. The method is compared against existing safe RL algorithms in several continuous control tasks. The experiment section also shows some ablation studies (in Appendix).", "questions": "I find a few issues or have some comments with the paper, please see below:\n\n1. Have the authors considered using a parametric action editing (h) function? To be clear, I think this is a reasonable choice, but could the authors elaborate on what motivated the non-paremetric choice?\n2. The proposed approach seem to introduce structure into the policy search by decomposing the safe RL task into two subtasks, the reward maximisation subtask (governed by \\phi) and the safety subtask (governed by /psi). Could the authors elaborate on the differences with the mechanism introduced in [2] where the safety subtask is given to an adversary and the actor is given the subtask to maximize future expected rewards while also maximizing its discrepancy with the adversary. To what extent is the policy search space different? I note that the paper [2] is quite recent (April 20th 2022). That being said, a short discussion highlighting the similarities and differences may help to give further context.\n3. Additional relevant work: [3].\n4. I understand that the number of tasks is high but I would kindly encourage the authors to use resource constraints on presenting more seeds with less tasks than the contrary. Unfortunately, using 3 seeds in deep RL cannot be statistically conclusive.\n\nMainly based the last point, my score is what it is, but I will significantly increase it if the above concerns/comments (in particular the last one) are addressed.\n\n[1] Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018, April). Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence (Vol. 32, No. 1).  \n[2] Flet-Berliac, Y., & Basu, D. (2022). SAAC: Safe Reinforcement Learning as an Adversarial Game of Actor-Critics. In 5th Conference on Reinforcement Learning and Decision Making.  \n[3] Bhatnagar, S., & Lakshmanan, K. (2012). An online actor–critic algorithm with function approximation for constrained markov decision processes. Journal of Optimization Theory and Applications, 153(3), 688-708.", "limitations": "Some limitations of the work have been addressed in the conclusion with potential interesting future work directions.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The contribution of this paper is original and appears relevant to the community. Overall the paper is mostly well written and develops its idea clearly. The paper presents an idea that is simple and leads to favorable empirical results. However, the very low number of seeds considered in the “Safety Gym tasks” is very low and does not allow to statistically draw any empirical conclusion [1] for this part. We develop more points below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept