Enhancing Safe Exploration Using Safety State Augmentation
Authors: Aivar Sootla
Alexander Imani Cowen-Rivers
Jun Wang
Haitham Bou Ammar
Venue: NeurIPS 2022 Accept (2022)
Keywords: safe reinforcement learning
safety during training

Abstract: Safe exploration is a challenging and important problem in model-free reinforcement learning (RL). Often the safety cost is sparse and unknown, which unavoidably leads to constraint violations - a phenomenon ideally to be avoided in safety-critical applications. We tackle this problem by augmenting the state-space with a safety state, which is nonnegative if and only if the constraint is satisfied. The value of this state also serves as a distance toward constraint violation, while its initial value indicates the available safety budget. This idea allows us to derive policies for scheduling the safety budget during training. We call our approach Simmer (Safe policy IMproveMEnt for RL) to reflect the careful nature of these schedules. We apply this idea to two safe RL problems: RL with constraints imposed on an average cost, and RL with constraints imposed on a cost with probability one. Our experiments suggest that "simmering" a safe algorithm can improve safety during training for both settings. We further show that Simmer can stabilize training and improve the performance of safe RL with average constraints. 

Effects of Safety State Augmentation on Safe Exploration
Aivar Sootla Byju's Lab
Alexander Cowen-Rivers Technische Universität Darmstadt mc rirvers@eccoou.. com
Jun Wang
University College London jun. wangdcs ucl ac uk
aivar. s0otta@@aaall com
Haitham Bou Ammar
Huawei R&D haitham ammar@huawei. com
Abstract
Safe exploration is a challenging and important problem in model-free reinforce- ment learning (RL). Often the safety cost sparse and unknown, which unavoidably leads to constraint violations
phenomenon ideally to be avoided in safety- critical applications. We tackle this problem by augmenting the state-space with a safety state, which is nonnegative if and only if the constraint is satisfied. The value this state also serves as a distance toward constraint violation, while its initial value indicates the available safety budget. This idea allows to derive policies for scheduling the safety budget during training. We call our approach Simmer (Safe policy MEtt for RI. reflect the careful nature of these schedules. We apply this idea to two safe RI. problems: RI with constraints imposed on an average cost, and RL with constraints imposed experiments suggest that "simmering" safe algorithm can improve safety during training for both settings. We further show that Simmer can stabilize training and improve the performance of safe RL with average constraints.
cost with probability one. Our
Introduction
Reinforcement learning (RL) is a framework for sequential decision decision-making that makes minimal prior assumptions about the environment where the agent has to act or make the decisions [45]. The
policy for taking actions is learned through interactions with the environment over time. RL has seen
recent successes in playing video games with computer [32), board games with human [4 and on a path toward real-life applications such as video compression [46], and plasma control (20]
There are still, however, some unsolved challenges for successful deployment of RL such as efficient learning of constrained or safe Markov Decision Processes (MDPs) [4. The constraints
typically modeled by discounted sum of nonnegative costs that have be smaller than some pre-defined value we call the safety budget.
40 34). In the context of safe RL, while exploring, we do not want to incur the safety cost and constraint violations, making exploration harder task. We can distinguish two main research directions for minimizing constraint violation in safe RL: a model-based approach (which includes partially and fully known en- vironments) and model-free approach. In the model-frn free case, which focus on, the agent would al- most certainly visit unsafe regions to learn the safe policy, and therefore it is next to impossible to avoid constraint violations completely. The general goal, in this case, minimize the number of violations during training. For example, [10] use a conservative safety critic and rejection sampling to choose
Exploration is crucial component of RL and is still active area of research 25
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

a "safer" action, [4] design a curriculum learning approach, where the teacher resets the student violating safety constraints, and [2 learns reset the policy if the safety constraint is violated 2TT.
In this work, we aim to improve model-free safe reinforcement learning by augmenting the state-space with state encapsulating the safety information. This safety state is initialized with the safety budget and the value of the safety state can serve as a measure of distance to the unsafe region. This form of the safety state was proposed in [88] and introduced in RL in 43), where a safe RL algorithm with probability one constraints was derived. In this paper, we take a closer look at the purposes of the safety state broader context. First, we claim that safety state augmentation often crucial for the averaged constrained problem as well and provide examples of such occurrences. In some particular problems, the use of safety state augmentation may potentially be avoided, however, this can be said about any state in the environment. Second, our main suggestion is that scheduling the safety budget at different training epochs can improve the algorithm's performance. Our claim extends equally over averaged and almost surely constrained reinforcement learning problems. In particular, we achieve Mate-of-the-are performance in some environments in the safety gym benchmark. Further, we claim that scheduling the initial safety budget can lead to reducing safety constraint violations during training for safe RL with probability one constraints. We design two algorithms that automatically tune the safety budget, one is based on a classical control engineering PI controller, while the other uses Q learning to decide on the safety budget.
Related work. Many of the current safe RL methods are extensions of the most successful RL algorithms: Trust region policy optimization (TRPO) [8), proximal policy optimization (PPO) [39) soft actor-critic (SAC) [23 etc. Safe versions of TRPO, PPO, and SAC with a I Lagrangian approach
were first presented in 37 which still considered to be one of the major baselines. direct extension of TRPO by adding constraints to the trust region update was proposed in 22. Model-based
approaches were also considered cf. [36] I6 27 30) and most of them took a Bayesian approach to model one-step transitions. To our best knowledge, the most successful approaches for safe reinforcement learning to date are PID-Lagrangian [44] and LAMBDA (5). The förmer views the Lagrangian multiplier update as another control problem and employs a PID controller to solve it (cf. [6). Specifically, the authors link the multiplier update to integral control and add proportional and derivative controllers to achieve superior behavior. On the other hand, [5] is a model-based approach that uses Bayesian world models to enhance safety. A recent work (2] formulated safe RL as inference resulting in a sample efficient off- off-policy approach.
Other formulations of safe RL were considered in the literature. For example, 14 16 51 proposed
to use cnnditional-value-e risk (CVaR) constraints, while [4] [2 proposed to enforce constraints with probability one. Further, as we discussed above eliminating the number of constraint violations
typically requires strong assumptions, e.g., finite state space [47. (5 42), knowledge of partial
model (28) or initial safe policy [8]. These results are the spirit of safe RL with control-theoretic
22 which make significant prior assumptions to guarantee safety. Finally, the closest algorithm in the literature to our method is the curriculum learning approach to Safe RL from [48.. Due to space limitations we discuss this approach in detail in Appendix. We finally mention that 31 proposed a two-player framework with the cooperating task agent and safety agents, [17 proposed safety layer that would be applied after the action is computed using a classical policy, (4 defined a probabilistic shield for safety. We also note [52 26 that considered safe deployment of RL policies in real-life settings.
notions [15 9 33
13
3
19
2 Simmer: Safe policy improvement for reinforcement learning
2.1 MDP with safety state augmentation
We consider constrained reinforcement learning setting defined for a Markov Decision Process (MDP) M = (S, P,r, x+) with transition probability P S A x S + [0, 1] acting on state S and A spaces, with the reward r: 5 x A x S + RR and the discount factor 7r E (0, 1]. The MDP is endowed with the following optimization problem
I-
rsr a,, 8141), subject to: g (a- E-Enilee ri ar, s(+1) 2 0,
T-1
Σ max x(-18)
{d-E 1
t=0
with the time horizon T > O, the safety discount factor 71 E (0, 1). the safety cost I S A x S + [0, too). The statistic g(-) R + R is design choice (c.g., Mean, CVaR, chance
2

---

constraints etc [4). In this paper, we consider two most relevant options, from our point of view: a) a constraint with probability one, i.e. 9po P( 2 0) 1, constraint on average Jav (2) Ez.
Similarly to 43 we augment the safety state information into the state-space by introducing
the state 24
(dd Li of (8s, ag2 81++)), which has the following update 2 = (z+
1(8,, ag $t+1))/76 with Zy augmented into the MDP. The variable 2, has the interpretation of the remaining safety budget, and by definition enforcing the constraint on 2T is equivalent to enforcing the constraint on the accumulated cost. Now we can rewrite the safe RL problem as follows:
Noting that the update is Markovian this state can be easily
T-1
max (T118.0)
E
Σ 2=
(s ae; 8t41), subject to: g(2T) 2 O.
(2)
While 43] considered only the case (=), we argue that the case of (z) deserves additional
attention in the context of safety state augmentation. For completeness, review the main points of
the approach [43] in Appendix. Note that the policy now also depends on the safety state z and this feature deserves more thorough discussion.
2.2 Do we need the safety state?
As simple demonstration consider the cartoon in Figure T A robot needs to reach the goal while crossing the hazard region, which is marked by the red circle, and the safety cost is acquired for every time unit spent in the region. Both green and blue paths are safe, i.e., satisfy the constraint. However, at the crossing of these paths robot needs to know which path it took to this state. Switching from the green path to the blue one will lead to a constraint violation. Standard safe RL algorithms will have trouble with such scenarios while adding the safety state solves the issue. We confirm this observation in our experiments.
??
Figure I: A robot needs reach the goal while crossing the hazards region (marked by the red circle) and the safety cost is acquired for every time unit spent in the region. Both green and blue paths are safe, i... satisfy the constraint. However, at the path crossing robot needs know which path it took to this state. Switching from the green path to the blue one will lead to constraint violations.
Let us now discuss how this logic can be mathematically formalized. In the deterministic case, this problem is well studied in the optimal control literature in the context of problems with known dynamics and with terminal or end-point constraints [49]. Further, the authors 43 showed that the policy dependence probability one constraints. The problem was also studied in the context of stochastic diffusions [35) where the authors derived the representation of the optimal value function. In both cases, it was shown that the optimal policy depends on the whole state including the safety state z. While in some cases may not need to augment the safety state, in many situations it is critical.
the remaining safety budget is crucial for safe reinforcement learning with
2.3 Simmering Safe Reinforcement Learning
In this paper, we propose to use the initial safety budget as another tuning dial for the algorithms. Specifically, we will exploit the link between the safety budget d with the initial state of the safety state. We argue that adjustment of d during training from some initial value start to the target value daarett can lead improved exploration in terms of safety and performance.
Let us now present the mathematical formulation. At every epoch we pick a test safety budget de. collect the data set DR, compute the returns ED, E(D 7fr(s,, az) and the costs åD, (2T), where the function 9r. (-) is the empirical mean or the maximum for the averaged and the probability one
3

---

constrained problems, respectively, Using this information we aim at solving the following problem at every epoch (but we only apply a pre-determined number of gradient steps):
T
max (-1e,))
EDg
Σ t=0
(fr(st, az), subject to: 9D, (2r) 2 0, 20 dç
(3)
where so So and 20 = d2 the initial states of the augmented MDP. Note that for off-policy algorithms the data set Dj can potentially grow with epochs, while for on-policy algorithms the data set D3 will be emptied every epoch.
The key assumptions of our approach are as follows:
There is a finite number of test safety budgets, i.e., we assume that de can take the values {dtef] with di .... S d with d = disff and dtarget = -
We assume that the values {d(e) } are such that the task can be solved;
At for every we perform only one epoch of optimization solving Problem
The intuition behind our formulation is based on our empirical observations, where lower safety budgets usually caused lower bursts in accumulated safety costs. We hypothesize that the policy can quickly learn "extremely unsafe" actions thus providing low safety cost bursts for low safety budgets. Therefore if we start with very strict safety budget datart, then by gradually increasing the safety bud- get d from datart to d'arget, we can reduce the number of constraint violations during training. We will show that the problem of safety during training can be formalized as two-level recision-making prob- lem, however, even naive approach of scheduling dy can lead to improvement in performance and safety. We see this formulation the first step toward eliminating safety violations during training.
2.4 Application: Simmering for safety during training
Exploration can often lead to constraint violation during training due to the inherent stochasticity of exploration. While there is significant effort in research for safe exploration it typically requires significant prior assumptions on MDPs. We propose to embrace the philosophy of classical reinforce- ment learning and proceed with minimal assumptions. We propose to choose the test safety budget dy using another decision- -akinn problem:
Σ ReLU ReLU (cfaaget))
max U 5d]
(4)
d++1 clip (d& + ug, dtart dlarget) do = ctart
where (ddaget) is the violation of the target constraint obtained as a result of solving one epoch of Problem 3 the action set is 1-ōd, 5d] and 5d is pre-determined, clip(z; (z, y. 2) clips the value to lower bound y and an upper bound 2. i.e., the function returns y if r S y, 2 if I x and x otherwise.
We formalized our problem as reinforcement learning over a partially observable process with the non-stationary observations of the learning process. This two-level RI. problem allows us to use a broad spectrum of tools available in sequential decision-making differing our approach from the existing in the literature. However, solving Problem 4 appears to be a daunting task, especially in the online setting as we intend. Hence, we will employ heuristic solutions, which may prove more effective than the quest to find an optimal solution. Intuitively, a gradual increase in de by assigning scheduling increasing reference values (denoted by d:ef) would less likely lead to constraint violation due to stricter exploration constraints. However, fixing the schedule a priori limits the ability of the algorithm to react to constraint violations. To alleviate this issue propose two approaches: Pl Simmer (PI-controlled safety budget) and Q Simmer (Online Q-learning with non- non-stationary rewards). In both algorithms the intuition is again is based on our empirical observations that lower safety budgets cause lower bursts in accumulated safety costs. In particular, if the current accumulated costs well below the safety budget dref, then are very safe and the safety budget can be further increased. If the accumulated costs are around the safety budget, then we could stay the same level increase the safety budget. If the current accumulated costs are well above the safety budget dref, then the safety budget should be decreased.
PI Simmer. The idea for the Pl controller is quite intuitive it takes the error term e = dref e and uses it for action computation. P stands for proportional control and links the error terms with actions

---

by multiplying the error term by the gain K. The proportional part delivers brute force control by having a large control magnitude for large errors, but it is not effective if the instantaneous error values become small. Proportional control cannot achieve zero error e tracking. This happens since zero error results in a zero proportional action and hence the control over the error lost. To deliver zero error, integral control is typically used, which sums up previous error terms and uses this sum to determine actions instead the error. The integral term can be seen as an "action accelcration" (or "momentum") term which is large if the past errors are large. This can potentially cause unwanted behaviors (such as oscillations) if the corresponding gain K; is too large. Both gains K, and K; usually chosen by tuning, but there rules of thumb for tuning and choosing the gains, which we discuss in Appendix. PI controller can solve many hard control problems, but there are some implementation and engineering tricks and improvements. We introduce simplified version in Algorithm| and provide a full version of the algorithm and our ablation studies in Appendix.
Algorithm 1: Pl SIMMER (basic version)
Inputs: {dk [d k=0 safett budget schedule, hyper-parametenr Kp. Ki, öd.
Set dg d0
for k = . - do
Perform one epoch of learning for the safe policy with dy and get the statistic (zT (dx)))
Compute the error term 6₁ dhe (Zr(dd) (d&));
Compute action uk
KpEK
+ Ki Σ - 66
P-Patt
I-Part
Set da+1 = Sip -d, 8d) + dui
end
Q Simmer. Consider MDP with the states {dp ..... d K- J. which for simplicity of notation we denote (0..... K- - 1), with the actions {a-1, 4o, 0+1 J. where the action 4+1 moves the state s = i to the state s = + 1, a-1 moves the state = i to the state s = - 1 and the action ag does not transfer the state. Note that the action a-1 is defined for all i < K 1 and the action a-1 is defined for all i 0. Our design for the rewards of this MDP is guided by the following intuition.
We are not safe
We are borderline safe
We are very safe if
if
<
-
if ss
5 o
2
8:
2
0 -1
a = a 0, 1,
0 -1,
(5)
r
a = 0,
r
r
0 = 0,
1,
2
0. = 1.
where o = å (2)( is safety violation statistic (over an epoch) of the safe RL algorithm, 8 is significance violation threshold. We use Q-learning update to learn the Q function:
Q(si, (1 ((t, az) + (.(rz + max (($++1,)) b))
(6)
where 4 is the learning rate and get the action with e-greedy exploration strategy:
argmax, Q(se,)) with probability E,
Q random
(7)
with probability 1 1-6 E.
We summarize the approach as Algorithm
3 Experiments
3.1 Baselines, environments and code base
Environments: We use the safe pendulum environment defined in [66] and we also use the custom-made safety gym environment with deterministic constraints, which we call static point goal [51]. In this environment with 46 states and 2 actions, large hazard circle placed before the goal and forces the agent to go around it to reach the goal similarly to our cartoon in Figure] [
5

---

Algorithm 2: Q-SIMMER
Input: { [oef K-1
priori chosen state space; hyper-parameters N, Ir, ó,
k==0
Initialize do = d
for k O.....
do
Perform one epoch of learning for the safe policy with dN and get the statistic (zr(de));
Update the function in Equation G while computing the reward as Equation [
Compute action dg as in Equation and compute d accordingly;
end
Unsafe region
Safe pendulum swing-up.
(h) Custom Safety gym
(c) Point Goal
(d) Point Button
(e) Point Push
(a): the safe pendulum environment ((16)) ). 0 action, and the angle defines the unsafe region position where the safety cost the angle difference and is incurred only in the red area. (b): the custom safety gym environment ((51): robot needs to reach the goal while avoiding the unsafe region. (c)-(e): Safety Gym Goal, Button and Push tasks for the robot Point. Car robot environments are depicted Appendix. The illustrations are from 43 37
Figure 2:
is the angle from the upright position, is the
We provide additional details Figure 2 and Appendix. The rest of our tests are performed on the safety gym benchmarks [37].
Code base: Our code based on two repositories: safety starter agents and PID Lagrangian The code for PI Simmer and Simmer is available at //gi thub com huawei -noahhohhhhha tree/mastter/SIMER We use default parameters for both code bases unless stated otherwise.
Computational resources: We performed all computations on a PC equipped with 512GB of RAM, two Intel Xeon E5 CPUs, and four 16GB NVIDIA Tesla V100 GPUs.
Baselines (+) and our algorithms
CPO, Lagrangian PPO (L-PPPO), and TRPO (L-TRRO); Standard baselines from 37
PID-Lagrangian. An algorithm stabilizing I.-PPO learning 44
LAMBDA. A model- method showing great performance on Safety Gym 55]
PO-PPO PPO-based al algorithm with probability one constraints from [55
PI-Simmer Scheduling safety budget using PI controller for PO-PPO;
Q-Simmer Scheduling safety budget using Q learning for PO-PPO:
L-PPO (PID-L) SA L-PPO (PID-L) solving Problem with safety state augmentation;
Simmer L-PPO (PID-L) L-PPO (PID-L) SA and safety budget scheduling.
3.2 Improving Safety During Training for Pendulum Swing-Up
Improving safety during training is suited for almost surely safe RL and will take PO PPO as our baseline. In this setting, we can aim to reduce the number of individual trajectories that violate the constraints, and thus we can avoid estimating the statistic g. For PI Simmer we chose the following hyper-paree parameters K = 0.01, K; = 0.005, Ka = 0.01 and = 0.995. The parameter Kaw resets the integral term avoiding the accumulation of error the higher the value the more aggressive reset, The parameter is the Polyak update for the error term, which pre- rre-processes the error term for the PI controller. We discuss these parameters in detail in Appendix. We also note that except for an initial burst of violations both our approaches manage to keep the number of violations quite low. Overall we found that low values for K and K, are beneficial to avoid overreaction to constraint violations. this case, keeping the value of Kaw low is advisable as the action saturation does not occur too often. Fi- nally, keeping close to one will force the controller react most the constraint violations, For Q Simmer we chose g 1, 0.995, l, 0.05, and 0.95. It appears that fast learning rate here

---

can allow for learning, sufficiently fast forgetting of past rewards, but also to avoid catastrophic forget- ting. As we consider finite state MDP we can avoid using sophisticated techniques for online learning and use the simplest one tuning learning rate. We chose and to avoid frequent state transitions.
Task return
Number of violations
Safety budget schedule
200
35
80
180
60
30
160
140
PO PPO
40
25
P1-Simmer PO PPO
120
D-Smmeer
20
100
0-
200
400 epoch
600
800
200
400
600
800
200
400
600
800
epoch
epoch
(a) Returns
(b) Number violations
(c) Safety budgets
Figure 3: Reducing safety violations during training for Safe RL with constraints almost surely.
We compare our algorithm to PO PPO in terms of the number of trajectories with constraint violations, and returns, and compare the progression of the schedule d:sst Naturally, some individual trajectories still violate the constraint, but the number can be significantly reduced using Simmer RL as Figure 3 suggests. While PI Simmer outperforms Simmer in these runs, it is worth mentioning that PI Simmer uses more prior information than Q Simmer. Indeed, while composing schedule is not hard, we still have identify the switch points, which are learned by Q Simmer. We perform ablation studies on the parameters and discuss their choice in more detail in Appendix.
3.3
Guiding Exploration by Scheduling Safety Constraints
Task return
Number of violations
Safety cost
200
10000
60
50-
40
160
6000
L-PPO
30
140
4000
L-PPO w
20
120
Simmer L-PPO
2000
10
100
400
600
400
600
800
400
600
epoch
epoch
epoch
(a) Returns
(b) Number of violations
(c) Safety costs
Figure 4: Comparison of Simmer L-PPO, L-PPO with and without safety state augmentation. Mean returns and cost are computed over hundred different trajectories obtained for three different seeds. Shaded areas represent standard deviations.
We now turn our attention safe RL with constraints imposed on average costs. We test the performance of Simmer L-PPO and L-PPO SA on the swing-up pendulum environment and present training results in Figure Here we use safety starter agents as base learner for all algorithms. The major observations are that L-PPO w SA delivers almost constraini violations with respect to the mean cost estimate, and therefore using Q Simmer and PI Simmer is not necessary. In the meantime, L-PPO has even trouble converging. Note that we have used the same hyper-purameten for all algorithmss, which are default parameters safety starter agents and the learning rate 0.03. While the behavior of the L-PPO algorithm certainly be improved with tuning, we note that simply augmenting the safety state leads to improved performance as well as stability of the algorithm. Further, we observe that Simmer L-PPO leads to fewer number of violations, however, the rate of violations for the safety budget of 35 is fairly similar to L-PPO w SA.
Interestingly, similar picture occurs with more advanced baselines such as PID Lagrangian 44] and more complicated environments. Here, we used the static point goal environment designed where a large static hazard region is placed in front of the goal, which is similar to our motivational example Figure The results for PID-I. and PID-L with state augmentation are depicted Fig- ures 5a and 5b respectively, suggest that the presence of the safety state stabilizes training and leads
51
7

---

Return for PID-Lagrangien
Return PID-Lagrangian w
Return
3-
2-
6-1.0
1-
6-10.00
1-
d= 10.0
PID-L
d=220.0
d=20.0
PID w
0-
0
100
200
300
600
epoch Safety cost for PID-Lagrangian
epoch
opoch
Safety cost for PD-Lagrangien SA
Safety cost
50
6=1.0
PID-
6-00.00
0=100.0
40
2020.0
6-20.0
Safety budget
100
200
300
100
200
300
200
400
600
(a) PID-L.
(b) PID-L w SA.
(c) Simmer RL.
Figure 5: PID-L with and without safety state augmentation. d is the safety budget used in training. The curves are means and shaded areas are standard deviations computed over runs. These results suggest that safety state augmentation can stabilize training and deliver safer solutions.
to more consistent constraint satisfaction for different safety budgets d. Note that syper-parametent for all the rums are the same for both algorithms. We further apply the naive simmer approach to both baselines with results depicted in Figure Sc In both cases, the safety budget takes values of 1, 5, 10, 13, and 20, and increased after equal time intervals. Note that in both cases now training curves are quite stable, although state augmentation delivers an extra boost. In all our experiments used the same hyper-parameters for all versions of PID-L, ie., = 0.1, K, = 0.01, n 0.99.
Overall, our experiments suggest that safety during training with constraints imposed on average costs becomes much easier problem with safety state augmentation. Indeed, even intuitively every outlier trajectory with large constraint violation should bias the average cost and should instruct the algorithm not to follow this path. We note that in both cases above we have sparse costs, i.c., the agent encounters unsafe regions and incurs costs while navigating toward the goal. While an algorithm without the safety state will receive information on constraint violation after an episode, the safety state would constantly inform the algorithm of the distance toward a violation. This is one of the reasons why safety state augmentation can learn the task better. Note that simmering additionally offers fewer constraint violations while training. These results suggest that simmering safe RL together with state augmentation delivers overall safer solutions with stable training curves.
3.4 Tests on safety gym benchmarks
We finally test approach more challenging environments from the safety gym benchmark: PointPush1, PointGoall, PointButtonI Figures 6a
6b and [c respectively, and CarPush1, 1, respectively. The hyper-parameters and the
CarGoall and CarButton] in Figures 7a 7b and 7c
tuning specifics can be found Appendix. Again we use a naïve schedule increasing the safety budget from 10 to 25 in 5 unit increments. We observe that Simmer PID-L and PID-L outperform other baselines significantly in terms of the return while delivering safe policies. Simmer PID-L does perform very similarly to PID-L in terms of returns but generally outperforms PID-L in terms of safety costs and cost rates. We note that our results for PointGoall are consistent with the results from 44
4 Conclusion
We augment the safety information into the state space and show how we can effectively use it to improve safe exploration. The safety state is nonnegative if and only if the constraint is satisfied and therefore it can serve as distance toward constraint violations. We argue that the optimal policy must depend on this information to achieve safe performance. We validate this argument using

---

Retums
Safety costs
Cost Rates
Simmer PID-L
PID-L
0.05
Safety budget
1.0
L-PPO
L-TRPO
0.02
0.00
Lambda
1+7
1=7
(a) 1ointPush1
Safely
Simmer PID-.
30
PID-L
0.06
Safety budget
2.04
L-PPO
100
L-TRPO
CPO
0.00
Lambda
2
steps
stepp
steps
(b) PointGoall
Retums
Safaty costs
Cost Rates
Simmer PID-.
1.5
PID-L
0.05
3.0
Safety budget
0.04
L-PPO
L-TRPO
0.02
Lambda
3
2 steps
2
2
107
steps
167
steps
167
(c) PointButton
Figure 6: Results for the point robot in the goal, button, and push environments. Simmer PID-L outperforms all baselines in terms of the cost rate, which is measure of safety during training. At the same time, Simmer PID-I is competitive with PID-L in terms of the returns or safety costs, while significantly outperforming other baselines. The curves are means and shaded areas are variances computed over runs with random seeds.
intuitive examples, references to theoretical results, and experiments. We do not doubt that safety state augmentation needed for effective safe RL.
Safety state augmentation and simmering show superior performance on pendulum swing-up and static point goal tasks for average constrained problems. We further tuned PID Lagrangian to show very strong performance safety gym benchmarks, which has not been previously reported. Simmer PID Lagrangian shows competitive performance in terms of returns to PID Lagrangian and outperforms all baselines (including PID Lagrangian) in terms of the cost rate and the costs. Further, using state augmentation for the average constrained problems appears to be quite beneficial as well. Scheduling the safety budget can stabilize safe algorithms without state augmentation, but using both state augmentation and simmering improves performance and safety. We achieve this performance and safety boost at the expense of sample efficiency since effectively learn series of safe policies.
Simmering RL algorithms with probability one constraints can significantly reduce safety violations during training. We illustrate this feature by developing two algorithms for safe learning. The first one is based on a PI controller and allows for the adjustment of pre-defined learning schedule depending the estimated training cost. The second one is based on online Q learning and learns to adjust the safety budget automatically. Both approaches have some advantages and limitations. PI simmering is more effective if there is reasonable safety budget schedule, while Q simmering requires less prior knowledge to learn. We have tested PI-Simmer and Q-Simmer on a rather simple in terms of the state and action spaces environments. We foresee that in a more complex setting the main problem would be learning the agent's 's behavior, but our algorithmic development would remain valid. Nevertheless, it would be interesting extend this approach
more complex setting.
Our algorithmic development focused on probability one constraints for two reasons. First, since probability constraints have to be imposed on all trajectories the empirical estimate of violations

---

Retums
Safety costs
Cost Rates
PID-L
0.05
Safety budget
0.02
0.00
1w7
1+7
1=7
(a) CarPush1
Safely
Cost
Simmer PID-L
30
PID+L
Safety budget
2.04
L-PPO
L-TRPO
1.0
Lambda
0.00
2
steps
stwpp
steps
(b) CarGoall
Raturns
Safaty costs
Cost Rates
Simmer PID-L
PID-L
0.05
Safety budget
0.02
-2
0
2
2
steps
107
steps
167
steps
167
(c) CarButtonl
Figure 7: Results for the car robot in the goal, point and push environments. Simmer PID-L outperforms all baselines in terms of the cost rate, which is measure of safety during training. At the same time, Simmer PID-I is competitive with PID-L in terms of the returns or safety costs, while significantly outperforming other baselines. The curves are means and shaded areas are variances computed over five runs with random seeds. Note that we have baseline results only for the CarGoall environment.
is simpler than in the average constrained case (i.e., maximum vs empirical average). Second, appears that the average constrained case with "simple" environments (such as pendulum swing-up and static point goal), we can track the constraint quite efficiently after the initial (and unavoidable) burst in constraint violations. Our experiments suggest that PI Simmer and Simmer can be redundant for the average constrained RL.
Augmenting the safety state and scheduling the safety budget does not solve all the problems in safe exploration. First, an initial burst of constraint violations an inevitable reality of using our model-free approach. It appears that lowering the initial safety budget lowers the initial burst of constraint violations, but does not completely solve the problem. We do not see how to address this limitation without making further assumptions about the environment, Second, it noticeable that the performance on the PointPush! environment is quite noisy for both PID-L and Simmer PID-L. This is because some seeds achieve very good performance (return of 15) and some seeds do not (return of 5), while the learning curves appear to be stable. This suggests that the learning procedure finds local maxima. our point of view, this calls for more sophisticated algorithm for exploration maximizing return, which may help achieve stable performance in this environment. Beyond these limitations, one can also list safety violations caused by variance and disturbances, lack of provable safety guarantees, etc. Studying these limitations is outside of the scope of this paper.
5 Acknowledgement
This work was performed while the first two authors were with Huawei R&D UK. The authors thank Dr. Zimmer from Huawei R&D UK for helpful discussions and suggestions, as well as, the anonymous referees whose suggestions and comments significantly improved the paper.
10

---

References
[1] Anti-windup control using PID controller, 2022. https://www.nathworkks com colp
html (p.
[2] Joshua Achiam, David Held, Aviv Tamar, and Picter Abbeel. Constrained policy optimization.
In International Conference Machine Learning, pages 22-31, 2017. (p. 2)
[3] Anayo K Akametalu, Jaime Fisac, Jeremy Gillula, Shahab Kaynama, Melanie Zeilinger, and Claire J Tomlin. Reachability--asee safe learning with Gaussian processes. In IEEE Conference on Decision and Control, pages 1424-1433 1431, 2014. (p.2)
[4] Eitan Altman. Constrained Markov decision processes, volume 7. CRC Press, 1999. (p.
[5] Yarden As, Ilnura Usmanova, Sebastian Curi, and Andreas Krause. Constrained policy opti- mization via bayesian world models. International Conference on Learning Representations, 2022. (p.
(p-
[6] Karl Johan Aström. Advanced PID control, volume 461. The Instrumentation, Systems, and Automation Society, 2006. (p. 2). (p. a3)
[7] Karl Johan Áström and Richard M Murray, Feedback systems. Feedback Systems. Princeton university press, 2010. (p. a3
[8] Felix Berkenkamp, Riccardo Moriconi, Angela P Schoellig, and Andreas Krause. Safe learning of regions of attraction for uncertain, nonlinear systems with Gaussian processes. In 2016 IEEE 55th Conference on Decision and Control (CDC), pages
IEEE, 2016. (p.
[9] Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. In Advances in Neural Information Processing (p.2)
Systems, pages 908-918, 2017.
[10] Homanga Bharadhwaj. Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Animesh Garg. Conservative safety crities for exploration. In International Conference on Learning Representations, 2021. (p.
[11] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAl gym. arXiv preprint arXiv:1606.01 540, 2016. (p. pp.(5)
[12] Agustin Castellano, Hancheng Min, Enrique Mallada, and Juan Andrés Bazerque. Reinforce- ment learning with almost sure constraints. In Learning for Dynamics and Control Conference,
pages 559-570. PMLR, 2022. (p. 2
[13] Richard Cheng, Gábor Orosz, Richard M Murray, and Joel W Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In AAAI, 2019, (p. (p.2)
[14] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18(1):6070-6120, 2017. (p.
(p.
[15] Yinlam Chow, Ofir Nachum, Edgar Duenez- Guzzaa,, and Mohammad Ghavamzadeh. A Lyapumov-based approach safe reinforcement learning. In Advances in Neural Information Processing Systems, pages 8092-8101, 2018. (p.2)
[16] Alexander Cowen-Rivers, Daniel Palenicek, Vincent Moens, Mohammed Amin Abdullah, Aivar Sootla, Jun Wang, and Haitham Bou-Ammar. SAMBA: Safe model-based & active reinforcement learning. Machine Learning, pages 1-31, 2022. (p.2). (p. 5). 6 (p. a5)) (p.
a6
[17] Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint arXiv: 1801.08777 2018.
(p.
2
11

---

[18] AN Daryin and AB Kurzhanski. Nonlinear control synthesis under double constraints. IFAC Proceedings Volumes, 38(1:2247-252, 2005. (p.2)
[19] S. Dean, S. Tu, N. Matni, and B. Recht. Safely learning to control the constrained linear quadratic regulator. In American Control Conference, pages 5582-5588, 2019. (p.
[20] Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Mag- netic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):414- 419, 2022. (p. D
[21] Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave trace: Learning to reset for safe and autonomous reinforcement learning. In International Conference on Learning Reprexentations, 2018. (p. 2). (p. a2)
[22] F. Fisac, A. K. Akametalu, M. N. Zeilinger, Kaynama, Gillula, and C. Tomlin. A general safety framework for learning-based control in uncertain robotic systems. IEEE Trunsuctions on Automatic Control, 64(7):2737-2752, 2019. (p.2)
[23] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870, 2018. (p. 2
[24] Nils Jansen, Bettina Könighofer, Sebastian Junges, Alex Serban, and Roderick Bloem. Safe reinforcement learning using probabilistic shields. In 31st International Conference on Con- currency Theory, pages 31-316. Schloss Dagstuhl--Lcibniz-zeentee fur Informatik GmbH,
Dagstuhl Publishing, 2020. (p. 2
[25] Chi Jin, Akshay Krishnamurthy, Max Simchowitz, and Tiancheng Yu. Reward-free exploration for reinforcement learning. In International Conference on Machine Learning, pages 4870-4879,
PMLR, 2020. (p. 1
[26] Gregory Kahn, Adam Villatlor, Vitchyr Pong, Picter Abbeel, and Sergey Levine. Uncertainty- aware reinforcement learning for collision avoidance. arXiv preprint arXiv:1702.011 2017.
(p.
[27] Sanket Kamthe and Mare Deisenroth. Data-efficient reinforcement learning with probabilistic model predictive control. In International Conference on Artificial Intelligence and Statistics, pages 1701-1710. PMLR, 2018. (p- 2
[28] Torsten Koller, Felix Berkenkamp. Mattco Turchetta, and Andreas Krause. Learning-bai based model predictive control for safe exploration. In IEEE Conference Decision and Control, pages 6069,6066, 2018. (p.
[29] Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo Li, and Ding Zhao. Constrained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning, pages 13644-1360 PMLR, 2022. (p.
[30] Yecheng Jason Ma, Andrew Shen, Osbert Bastani, and Jayaraman Dinesh. Conservative and adaptive penalty for model-based safe reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, pages 5404-5412 2022. (p.2)
[31] David Mguni, Joel Jennings, Taher Jafferjee, Aivar Sootla, Yaodong Yang, Changmin Yu, Usman Islam, Ziyan Wang, and Jun Wang. DESTA: A framework for safe reinforcement learning with markov games of intervention. arXiv preprint arXiv:2/ /4468, 2021. (p.
[32] Volodymyr Mnih Koray Kavukcuoglu, David Silver, Alex Graves, Joannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 132.v:1312.5602 2013. (p.
[33] Motoya Ohnishi, Li Wang, Gennaro Notomista, and Magnus Egerstedt. Barrier-certified adaptive reinforcement learning with applications to brushbot navigation. IEFE Transactions on Robotics, 35(5):1186-122 2019. (p.2)
12

---

[34] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-die -diveen exploration by sulf-supe prediction. In International Conference on Machine Learning (ICML), pages
2778 2787, 2017.
(p. I
[35] Laurent Pfeiffer. Two approaches to stochastic optimal control problems with final-time expectation constraint. Applied Mathematics & Oppimization, 77(2):377-404, 2018. (p.
[36] Kyriakos Polymenakos, Nikitas Rontsis, Alessandro Abate, and Stephen Roberts. SafePILCO: software tool for safe and data-efficient policy synthesis. In Marco Gribaudo, David N. Jansen, and Anne Remke, editors, Quantitative Evaluation of Systems, pages 18-26. Springer International Publishing, 2020. (p.2)
[37] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep rein- forcement learning, 2019. (p. 2 (p. O (p. (p. (p.
[38] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pages 1889-1897, 2015. (p.
2
[39] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0634/ 2017. (p.
[40] Philipp Schwartenbeck, Thomas FitzGerald, Ray Dolan, and Karl Friston. Exploration, novelty, surprise, and free energy minimization. Frontiers in psychology, 4:710, 2013. (p. D
[41] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifree, George Van Den Driess- che, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016. (p. [1]
[42] Thiago D Simão, Nils Jansen, and Matthijs TJ Spaan. Alwayssafe: Reinforcement learning without safety constraint violations during training. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems. International Foundation for Autonomous Agents and Multiagent Systems, 2021. (p.2
[43] Aivar Sootla, Alexander Cowen-Rivers, Taher Jafferjee, Ziyan Wang, David H Mguni, Jun Wang, and Haitham Ammar. Sauté rl: Almost surely safe reinforcement learning using state augmentation. In International Conference on Machine Learning, pages 20423 204433. PMLR, 2022. (p.
2 (p- 3). (p. 6
43
a9
[44] Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pages 9133-99 99143. PMLR, 2020. (p. 2). (p. 6 (p. 7 (p- 8 (p.
(p.
[45] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018. (p. !
[46] MuZero Applied Team. Muzero's first step from research into the real world. In DeepMind blog. Deepmind, Feb 2022. https:/wwww.youuube. wwaelbbe..oowwww/e wwt.w/owo/wwwwtecceeec Soomm//wcch/-ZTyySSaii
SFuGG.M. [p.
[47] Matteo Turchetta, Felix Berkenkamp, and Andreas Krause. Safe exploration in finite Markov decision processes with Gaussian processes. In Advances in Neural Information Processing Systems, pages 4312-4320, 2016. (p.
[48] Matteo Turchetta, Andrey Kolobov. Shital Shah, Andreas Krause, and Alekh Agarwal. Safe reinforcement learning via curriculum induction. Advances in Neural Information Processing Systems, 33:12151-12162, 2020. (p.
2 (p.
alp.
(p. a2)
[49] Richard B Vinter. Optimal control. Springer, 2010.
(p-3)
[50] Akifumi Wachi, Yanan Sui, Yisong Yue, and Masahiro Ono. Safe exploration and optimization of constrained MDPs using Gaussian processes. AAAI Conference on Artificial Intelligence,
2018. (p. 2
13

---

[51] Qisong Yang, Thiago D Simão, Simon H Tindemans, and Matthijs TJ Spaan. WCSAC: Worst- case soft actor critic for safety-constraing reinforcement learning. In AAAI Conference on Artificial Intelligence, 2021. (p. 2.
(p. (p.
a5
(p. a6)
[52] Jesse Zhang. Brian Cheung, Chelsea Finn, Sergey Levine, and Dinesh Jayaraman. Cautious adaptation for reinforcement learning in safety-critical settings. In International Conference on Machine Learning, pages 11055-11066 1065. PMLR, 2020. (p. 2)
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proof's of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We reported the error bars over all the runs across different seeds
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] We use PC equipped with 512GB of RAM, two Intel Xeon E5 CPUs, and four 16GB NVIDIA Tesla V100 GPUs
4. If you using existing assets (e.g., code, data, models) curating/releusing
assets...
(a) your work uses existing assets, did you cite the creators? (es]] We use the code from safety starter agents, safety gym by open Al (37), PID Lagrangian method [4
(b) Did you mention the license of the assets? [No] All the libraries are under MIT Licence.
(c) Did you include any new assets either in the supplemental material or URL? Yes] We published the code at /
master/SIMMER
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "sgNyVKvpeOE", "reviewer": "Reviewer_vED7", "summary": "This paper presented a novel state augmentation technique to enhance safe exploration of the RL problem with safety budget constraints. Both probability-one constraint and average constraint are considered in the problem formulation. A safety budget schedule is designed to reduce constraint violations and guide safe exploration in the simmering safe RL algorithm. The experimental results well supported the claimed contributions. I will consider further improving my rating if the author could explain the connection and difference between this paper and the reference [37].", "questions": "1. The major question I am interested in is what is the difference between this paper and  [37]?\n\n   Other minor questions about technical details:\n\n2. For the probability one constraints, the notion $P(z\\geq 0) - 1 \\leq d$ is confusing because the left-hand side must be lower than zero. However, the budget is always positive in the experiment section. Could you explain that?\n\n3. Can the proposed methods handle safety constraints other than the safety budget?", "limitations": "The paper title starts with *enhancing safe exploration*, which might be too strong for the contemporary model-free safe RL algorithms. I agree that the proposed algorithm has significantly improved safety during training with a solid and novel solution. However, many other challenges still remain unfixed when referring to the **safe exploration** problem, such as violation bursts in the initial training stages, violations caused by variance and disturbances during training, lack of provable safety guarantees, etc. I suggest the author explain more the contributions and limits of this paper in the context of the safe exploration problem.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n1. This paper is well-written, and the solution is novel and clearly presented. \n2. The experimental results well supported the claimed contribution. The contribution of safety enhancement is solid.\n\nWeakness\n\n1. This motivation of this paper is similar to the paper [37] in the reference. The author has not thoroughly explained the connection and difference between this paper and [37].", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dDIPq0vSkq7", "reviewer": "Reviewer_KpTK", "summary": "The paper considers the problem of safety reinforcement learning (model-free) in which the goal is to train an agent to not only maximize the average return but also satisfy an average safety violation cost along the trajectories. The approach of augmenting the state space plus shaping the reward in a 'saute' way (which can be applied to any RL algorithm) is being explored. The intuition behind the augmented state is the safety budget that is being diminished during the policy deployment, the paper introduces the SIMMER approach for scheduling the safety budget during the policy evaluation. ", "questions": "* I do not get why the pure SAUTE method (without planning the budget) is not included in the baselines?\n* Why the full suite of Safety-Gym benchmarks is not presented ?\n* I think it is necessary to clearly show what are the advantages of the method except for marginal performance gains (w.r.t. average cost).\n* Algorithms / introduced modifications should be presented in a standard way (using a pseudo-code) for better clarity.\n\n## Minor remarks\n* Eq. (4) the optimal control should be denoted by (e.g. $u^\\star$),\n* Eq. (6), (10) and others the symbols utilized are not defined formally,", "limitations": "Limitations and potential negative social impact of their work were properly addressed in the paper.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper builds upon the SAUTE a state augmentation safety-RL method introduced in A. Sootla \"SAUTE RL: Almost surely safe reinforcement learning using state augmentation\" . The idea is based on switching from a constrained MDP into a standard MDP with augmented state space. The interpretation of the additional state variable is the safety budget depleted by each safety violation event. \n\nThe original SAUTE algorithm worked for a fixed safety budget.  The current paper as I understand proposes and evaluates an improved SAUTE by introducing two possible safety budget planning during deployment approaches. The resulting approach of SAUTE enhanced with safety budget planning is denoted by authors 'SIMMER'.  The two introduced methods for safety budget planning are based on 1. PI control, 2. Q-learning. The algorithm works for both settings - with the probability one safety constraint and the average constraints.\n\nThe paper is experimental the benchmarks include a safe Pendulum Swing-Up environment and the Safety-Gym benchmark suite. The baselines are: the Lagrangian PPO and Lagrangian TRPO methods introduced in the Safety-Gym paper, Constrained Policy Optimization (CPO), and PID Lagrangian from A Stooke et al, \"Responsive safety in reinforcement learning by PID lagrangian methods\". \n\n The approach is heavily based on the previously introduced method. It is not enough to fine-tune an existing approach, introduce a new trick, and coin a fancy-sounding acronym to obtain a new algorithm. \n\nThe Pendulum Swing-Up experiment shows an improvement over the baselines in terms of the safety constraint violations, over the Lagrangian PPO. However, this is a simple environment, and all of the methods reach a large task return pretty quickly. \n\nIn the Safety-Gym benchmarks restricted to the Point set of environments, an improvement is less clear. In cases where the average cost is less for SIMMER than those of the baseline PID Lagrangian method, the average return is significantly smaller for SIMMER. Only at the final stages of training (10 mln steps) does the average return of those methods equalize. For the PointGoal environment, it is visible that SIMMER has a lower variance than PID Lagrangian for the same average threshold at around 25. For me, it looks like it is a minuscule improvement over the PID Lagrangian approach. Appreciate that the PID Lagrangian baseline was carefully fine-tuned by the authors.\n\nIt is a pity that the probability one constraint setting was not examined in the safety environments (only the average cost setting is considered).\n\nI do not understand why other benchmarks from Safety-Gym (Car and Doggo-based environments are not included), would show a more complete picture of the capabilities of the studied Safe-RL approaches. I just realized that 2 out of 3 total Car based environments are actually included in the appendix, and the results for Car-Push are looking much better than for Point-Push so I do not see what was author's motivation in making such a choice.\n\nConceptually the ideas are not well presented in the paper. I do not think that tabular form like (6) is clear for the reader. I understand that these are computational steps of an algorithm, so why not presenting them properly as an algorithm pseudo-code.\n\nOn the positive side, the experiments are well documented (ablation study and hyper-parameters are provided).\n\n### Summary\nI am not convinced that the introduced SIMMER algorithm is novel enough, and show a significant improvement over existing baselines to meet the publication bar on the NeurIPS conference.\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9sVQddcZrK-", "reviewer": "Reviewer_UM3D", "summary": "This paper explores learning a safety-state conditioned policy, along with methods for controlling the safety input adaptively to give extra slack to the policy for being more unsafe when it can afford to do so. I mainly have issues with parts of the writing of the paper, and some unanswered questions, that if fixed would make me recommend this paper for acceptance.", "questions": "- In 2.2, how come the example given is only relevant for safe RL with average safety cost constraints, and not for probability 1 constraints?\n- How would you extend PI/Q-simmer for safety selection to more complex state and action spaces?\n- It seems [37] also highlights the importance of the safety state. Can the authors clarify what is different about this work that makes the safety state seem more important?", "limitations": "Not really addressed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": " ## Strengths\n\n**Results:** Strong results in RL SafetyGym benchmark tasks\n\n**Writing:** Explanations are generally good and intuitive, and writing is generally clear.\n\n**Method:** An intuitive yet pretty simple method of selecting the safety constraint on-demand.\n\n## Weaknesses\n\n**Related Works:** Some missing safety papers taking different approaches the authors could cite and compare: [https://arxiv.org/pdf/1702.01182.pdf](https://arxiv.org/pdf/1702.01182.pdf), [https://arxiv.org/abs/2008.06622](https://arxiv.org/abs/2008.06622),  [https://www.aaai.org/AAAI22Papers/AAAI-5810.MaY.pdf](https://www.aaai.org/AAAI22Papers/AAAI-5810.MaY.pdf)\n\n**Writing/Clarity:** One issue I have with the writing is in the presentation of the baselines. There are quite a few baselines and ablations (Simmer PID-L, PID-L, L-PPo, Lambda, PID-L w SA,… etc). The experiments section would be a lot easier to read and think about if each baseline was explicitly written out (bullet pointed list? with bolded names for each baseline and a quick description?) in some subsection.\n\nSimilarly, paragraph 3 of the introduction describes the method in a bit too much detail and is hard to follow when reading for the first time without prior knowledge about the method, safety states, etc.\n\nAlso, “SAUTE MDP” is never defined but just used in the paper (you cite [37] but don’t mention the word saute). \n\nA custom environment is made, but there isn’t a picture of it in either the main paper or appendix, making it hard to think about and contextualize the results on it. It would also make sense to just have an overview figure of all environments (including some example safety gym ones) for clarity.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3v_6xperbhZ", "reviewer": "Reviewer_n7QX", "summary": "The authors present a method for safe RL by augmenting the original MDP with an additional state which functionally serves as a \"distance\" to constraint violation. A method is presented to learn the initial value of the cost function (referred to a as a safety budget). The authors study the case where the constraints are imposed on the average cost, and where the constraints are imposed on the cost with probability 1. The authors compare their approach against some benchmarks and show reduced constraint violations. ", "questions": "Have the authors considered the use of temporal logics and synthesis in order to specify the safety requirement? For example the following paper: (Jiang, Y., Bharadwaj, S., Wu, B., Shah, R., Topcu, U., & Stone, P. (2021). Temporal-Logic-Based Reward Shaping for Continuing Reinforcement Learning Tasks). \n\nThese methods, in essence results in an augmented MDP with additional safety states. While the linked paper uses the \"distance to the safety state\" as a reward shaping function, a separate cost function may be applied instead. \n\nIs it correct to assume that assigning a cost function to these states will result in the same outcome as the work presented in this paper? If not, perhaps a comparison would help strengthen the authors' contribution. ", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is presented well and explained. However, the writing could be further enhanced with the use of toy examples to illustrate concepts. \n\nIt seems one of the main contributions of this paper is learning the safety budget. There are certain aspects regarding the tuning of the safety budget that seems ad-hoc or could do with further explanations and concrete testing. For example, the authors state: \"If the current accumulated costs are well above the safety budget then the safety budget should be decreased to ensure that the policy is incentivized to be safer\". It is not clear why decreasing the safety budget would help here. If the policy is already causing significant safety violations above the safety budget, it seems that the safety requirement just may not feasible in that setting? \n\nThe authors compare against related algorithms in RL, but there are several different approaches to safe RL that could still be compared against for more empirical evidence. For example the paper \"Safe Reinforcement Learning by Probabilistic Shields\" is another approach to safe RL, but allows the user to specify a probability of safety violation. In contrast the safety budget approach doesn't have a direct relationship on the amount of violation a user is able to accept.\n\nOverall, while this paper does present an interesting idea, it seems to be a somewhat incremental contribution. It relies on a lot of tuning and doesn't compare enough against other safe RL approaches. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept