Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors present a method for safe RL by augmenting the original MDP with an additional state which functionally serves as a "distance" to constraint violation. A method is presented to learn the initial value of the cost function (referred to a as a safety budget). The authors study the case where the constraints are imposed on the average cost, and where the constraints are imposed on the cost with probability 1. The authors compare their approach against some benchmarks and show reduced constraint violations. 
strengths_and_weaknesses: The paper is presented well and explained. However, the writing could be further enhanced with the use of toy examples to illustrate concepts. 

It seems one of the main contributions of this paper is learning the safety budget. There are certain aspects regarding the tuning of the safety budget that seems ad-hoc or could do with further explanations and concrete testing. For example, the authors state: "If the current accumulated costs are well above the safety budget then the safety budget should be decreased to ensure that the policy is incentivized to be safer". It is not clear why decreasing the safety budget would help here. If the policy is already causing significant safety violations above the safety budget, it seems that the safety requirement just may not feasible in that setting? 

The authors compare against related algorithms in RL, but there are several different approaches to safe RL that could still be compared against for more empirical evidence. For example the paper "Safe Reinforcement Learning by Probabilistic Shields" is another approach to safe RL, but allows the user to specify a probability of safety violation. In contrast the safety budget approach doesn't have a direct relationship on the amount of violation a user is able to accept.

Overall, while this paper does present an interesting idea, it seems to be a somewhat incremental contribution. It relies on a lot of tuning and doesn't compare enough against other safe RL approaches. 
questions: Have the authors considered the use of temporal logics and synthesis in order to specify the safety requirement? For example the following paper: (Jiang, Y., Bharadwaj, S., Wu, B., Shah, R., Topcu, U., & Stone, P. (2021). Temporal-Logic-Based Reward Shaping for Continuing Reinforcement Learning Tasks). 

These methods, in essence results in an augmented MDP with additional safety states. While the linked paper uses the "distance to the safety state" as a reward shaping function, a separate cost function may be applied instead. 

Is it correct to assume that assigning a cost function to these states will result in the same outcome as the work presented in this paper? If not, perhaps a comparison would help strengthen the authors' contribution. 
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presented a novel state augmentation technique to enhance safe exploration of the RL problem with safety budget constraints. Both probability-one constraint and average constraint are considered in the problem formulation. A safety budget schedule is designed to reduce constraint violations and guide safe exploration in the simmering safe RL algorithm. The experimental results well supported the claimed contributions. I will consider further improving my rating if the author could explain the connection and difference between this paper and the reference [37].
strengths_and_weaknesses: Strength

1. This paper is well-written, and the solution is novel and clearly presented. 
2. The experimental results well supported the claimed contribution. The contribution of safety enhancement is solid.

Weakness

1. This motivation of this paper is similar to the paper [37] in the reference. The author has not thoroughly explained the connection and difference between this paper and [37].
questions: 1. The major question I am interested in is what is the difference between this paper and  [37]?

   Other minor questions about technical details:

2. For the probability one constraints, the notion $P(z\geq 0) - 1 \leq d$ is confusing because the left-hand side must be lower than zero. However, the budget is always positive in the experiment section. Could you explain that?

3. Can the proposed methods handle safety constraints other than the safety budget?
limitations: The paper title starts with *enhancing safe exploration*, which might be too strong for the contemporary model-free safe RL algorithms. I agree that the proposed algorithm has significantly improved safety during training with a solid and novel solution. However, many other challenges still remain unfixed when referring to the **safe exploration** problem, such as violation bursts in the initial training stages, violations caused by variance and disturbances during training, lack of provable safety guarantees, etc. I suggest the author explain more the contributions and limits of this paper in the context of the safe exploration problem.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper considers the problem of safety reinforcement learning (model-free) in which the goal is to train an agent to not only maximize the average return but also satisfy an average safety violation cost along the trajectories. The approach of augmenting the state space plus shaping the reward in a 'saute' way (which can be applied to any RL algorithm) is being explored. The intuition behind the augmented state is the safety budget that is being diminished during the policy deployment, the paper introduces the SIMMER approach for scheduling the safety budget during the policy evaluation. 
strengths_and_weaknesses: The paper builds upon the SAUTE a state augmentation safety-RL method introduced in A. Sootla "SAUTE RL: Almost surely safe reinforcement learning using state augmentation" . The idea is based on switching from a constrained MDP into a standard MDP with augmented state space. The interpretation of the additional state variable is the safety budget depleted by each safety violation event. 

The original SAUTE algorithm worked for a fixed safety budget.  The current paper as I understand proposes and evaluates an improved SAUTE by introducing two possible safety budget planning during deployment approaches. The resulting approach of SAUTE enhanced with safety budget planning is denoted by authors 'SIMMER'.  The two introduced methods for safety budget planning are based on 1. PI control, 2. Q-learning. The algorithm works for both settings - with the probability one safety constraint and the average constraints.

The paper is experimental the benchmarks include a safe Pendulum Swing-Up environment and the Safety-Gym benchmark suite. The baselines are: the Lagrangian PPO and Lagrangian TRPO methods introduced in the Safety-Gym paper, Constrained Policy Optimization (CPO), and PID Lagrangian from A Stooke et al, "Responsive safety in reinforcement learning by PID lagrangian methods". 

 The approach is heavily based on the previously introduced method. It is not enough to fine-tune an existing approach, introduce a new trick, and coin a fancy-sounding acronym to obtain a new algorithm. 

The Pendulum Swing-Up experiment shows an improvement over the baselines in terms of the safety constraint violations, over the Lagrangian PPO. However, this is a simple environment, and all of the methods reach a large task return pretty quickly. 

In the Safety-Gym benchmarks restricted to the Point set of environments, an improvement is less clear. In cases where the average cost is less for SIMMER than those of the baseline PID Lagrangian method, the average return is significantly smaller for SIMMER. Only at the final stages of training (10 mln steps) does the average return of those methods equalize. For the PointGoal environment, it is visible that SIMMER has a lower variance than PID Lagrangian for the same average threshold at around 25. For me, it looks like it is a minuscule improvement over the PID Lagrangian approach. Appreciate that the PID Lagrangian baseline was carefully fine-tuned by the authors.

It is a pity that the probability one constraint setting was not examined in the safety environments (only the average cost setting is considered).

I do not understand why other benchmarks from Safety-Gym (Car and Doggo-based environments are not included), would show a more complete picture of the capabilities of the studied Safe-RL approaches. I just realized that 2 out of 3 total Car based environments are actually included in the appendix, and the results for Car-Push are looking much better than for Point-Push so I do not see what was author's motivation in making such a choice.

Conceptually the ideas are not well presented in the paper. I do not think that tabular form like (6) is clear for the reader. I understand that these are computational steps of an algorithm, so why not presenting them properly as an algorithm pseudo-code.

On the positive side, the experiments are well documented (ablation study and hyper-parameters are provided).

### Summary
I am not convinced that the introduced SIMMER algorithm is novel enough, and show a significant improvement over existing baselines to meet the publication bar on the NeurIPS conference.
 
questions: * I do not get why the pure SAUTE method (without planning the budget) is not included in the baselines?
* Why the full suite of Safety-Gym benchmarks is not presented ?
* I think it is necessary to clearly show what are the advantages of the method except for marginal performance gains (w.r.t. average cost).
* Algorithms / introduced modifications should be presented in a standard way (using a pseudo-code) for better clarity.

## Minor remarks
* Eq. (4) the optimal control should be denoted by (e.g. $u^\star$),
* Eq. (6), (10) and others the symbols utilized are not defined formally,
limitations: Limitations and potential negative social impact of their work were properly addressed in the paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper explores learning a safety-state conditioned policy, along with methods for controlling the safety input adaptively to give extra slack to the policy for being more unsafe when it can afford to do so. I mainly have issues with parts of the writing of the paper, and some unanswered questions, that if fixed would make me recommend this paper for acceptance.
strengths_and_weaknesses:  ## Strengths

**Results:** Strong results in RL SafetyGym benchmark tasks

**Writing:** Explanations are generally good and intuitive, and writing is generally clear.

**Method:** An intuitive yet pretty simple method of selecting the safety constraint on-demand.

## Weaknesses

**Related Works:** Some missing safety papers taking different approaches the authors could cite and compare: [https://arxiv.org/pdf/1702.01182.pdf](https://arxiv.org/pdf/1702.01182.pdf), [https://arxiv.org/abs/2008.06622](https://arxiv.org/abs/2008.06622),  [https://www.aaai.org/AAAI22Papers/AAAI-5810.MaY.pdf](https://www.aaai.org/AAAI22Papers/AAAI-5810.MaY.pdf)

**Writing/Clarity:** One issue I have with the writing is in the presentation of the baselines. There are quite a few baselines and ablations (Simmer PID-L, PID-L, L-PPo, Lambda, PID-L w SA,… etc). The experiments section would be a lot easier to read and think about if each baseline was explicitly written out (bullet pointed list? with bolded names for each baseline and a quick description?) in some subsection.

Similarly, paragraph 3 of the introduction describes the method in a bit too much detail and is hard to follow when reading for the first time without prior knowledge about the method, safety states, etc.

Also, “SAUTE MDP” is never defined but just used in the paper (you cite [37] but don’t mention the word saute). 

A custom environment is made, but there isn’t a picture of it in either the main paper or appendix, making it hard to think about and contextualize the results on it. It would also make sense to just have an overview figure of all environments (including some example safety gym ones) for clarity.
questions: - In 2.2, how come the example given is only relevant for safe RL with average safety cost constraints, and not for probability 1 constraints?
- How would you extend PI/Q-simmer for safety selection to more complex state and action spaces?
- It seems [37] also highlights the importance of the safety state. Can the authors clarify what is different about this work that makes the safety state seem more important?
limitations: Not really addressed.
review_title: null

Meta-review: metareview: The authors develop a novel method for safe RL extending the work of Sootla et al (https://proceedings.mlr.press/v162/sootla22a/sootla22a.pdf) to deal with both expected and probability one constraints, demonstrating the utility of safety state aggregation in both scenarios. The authors validate their approach empirically and conduct careful experiments on several benchmark domains, showing gains from their approach relative to prior work.

Reviewers pointed out several issues in presentation and novelty relative to prior work that the authors addressed adequately in the rebuttal phase. Hence, I recommend acceptance.
confidence: Less certain
Decision: Accept