Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new method for tackling reinforcement learning with safety constraints with a two-policy approach. The first policy learns to maximize the overall return while the second policy edits actions coming from the first policy to ensure safety. The new algorithm was evaluated on the SafetyGym tasks and two additional safe racing tasks.
strengths_and_weaknesses: I like this paper a lot, the idea presented by the authors is simple and elegant yet appears to be quite powerful in practice. The paper is very clearly written and a pleasure to read. The authors clearly explained their methodology and motivated the different design choices they employed (e.g. Hinge loss vs. L2, additive edit function etc.). Experiments as well as detailed ablation studies are very well done and strongly support the authors' arguments. I think this paper makes a strong contribution to the field of safe RL and recommend this paper for acceptance at this venue. Additional detailed comments are as follows:
- The paper argues why the additive edit function is preferred and the authors provide some solid observations for why this is the case. I was wondering if you could give a bit more intuition on why this is the case (especially why additive is better than overwrite).
- Also a related question to the above, have you considered other forms of edit functions aside from additive and overwrite?
- My biggest concern is with how $\Lambda$ is evaluated. My understanding is that $\Lambda$ is evaluated in an on-policy Monte Carlo fashion even though the overall algorithm is off-policy. I agree with the author's argument since off-policy evaluation is quite challenging and generally inaccurate, however I do have several questions:
  - When you use a batch of size $N$, you mentioned that you don't necessarily wait for the episode to finish when doing evaluation. Does this mean each batch could contain multiple episodes or a partial unfinished episode?
  - In general, larger $N$ is preferred from an accuracy perspective but is problematic in terms of sample efficiency. From a theoretical perspective you could get away with a smaller $N$ if the underlying MDP has a small mixing time. However I wonder in practice how did you chose $N$?
  - Also for evaluating $\Lambda$, did you have to collect a separate batch of data or was it the same data you collect to put into the data buffer? Since in the former case this does need to be taken into account when comparing to other baselines in terms of sample efficiency.
- One suggestion in terms of presentation, Section 3 quite clearly explains the proposed approach, but since it does contain many different pieces, I believe it would be helpful to include some kind of pseudocode in either the main text or appendix to summarize Section 3.

questions: One additional question I have aside from the points I mentioned in the main review. In discussing the limitations of your work, you mentioned that oscillation is still an issue that you observe, I wonder if you have considered or whether it's possible to integrate some of the ideas from Stooke et al. (2020) (referenced in your paper) which addresses the oscillation issue but for Lagrangian methods.
limitations: Yes, see above comment.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new and general approach for safe RL with relatively strong empirical evidence. The proposed approach leverages two policies, the first one which is a utility maximizer (UM), and the second one which is a safety editor (SE) in charge of adjusting the UM action to ensure a sufficiently low constraint violation rate. The method is compared against existing safe RL algorithms in several continuous control tasks. The experiment section also shows some ablation studies (in Appendix).
strengths_and_weaknesses: The contribution of this paper is original and appears relevant to the community. Overall the paper is mostly well written and develops its idea clearly. The paper presents an idea that is simple and leads to favorable empirical results. However, the very low number of seeds considered in the “Safety Gym tasks” is very low and does not allow to statistically draw any empirical conclusion [1] for this part. We develop more points below.
questions: I find a few issues or have some comments with the paper, please see below:

1. Have the authors considered using a parametric action editing (h) function? To be clear, I think this is a reasonable choice, but could the authors elaborate on what motivated the non-paremetric choice?
2. The proposed approach seem to introduce structure into the policy search by decomposing the safe RL task into two subtasks, the reward maximisation subtask (governed by \phi) and the safety subtask (governed by /psi). Could the authors elaborate on the differences with the mechanism introduced in [2] where the safety subtask is given to an adversary and the actor is given the subtask to maximize future expected rewards while also maximizing its discrepancy with the adversary. To what extent is the policy search space different? I note that the paper [2] is quite recent (April 20th 2022). That being said, a short discussion highlighting the similarities and differences may help to give further context.
3. Additional relevant work: [3].
4. I understand that the number of tasks is high but I would kindly encourage the authors to use resource constraints on presenting more seeds with less tasks than the contrary. Unfortunately, using 3 seeds in deep RL cannot be statistically conclusive.

Mainly based the last point, my score is what it is, but I will significantly increase it if the above concerns/comments (in particular the last one) are addressed.

[1] Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018, April). Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence (Vol. 32, No. 1).  
[2] Flet-Berliac, Y., & Basu, D. (2022). SAAC: Safe Reinforcement Learning as an Adversarial Game of Actor-Critics. In 5th Conference on Reinforcement Learning and Decision Making.  
[3] Bhatnagar, S., & Lakshmanan, K. (2012). An online actor–critic algorithm with function approximation for constrained markov decision processes. Journal of Optimization Theory and Applications, 153(3), 688-708.
limitations: Some limitations of the work have been addressed in the conclusion with potential interesting future work directions.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the problem of safe reinforcement learning in which the learning agent seeks to optimize the reward while minimally reducing the number of cost violations. The paper points out that there are two main commonly used assumptions in the field: the oracle function for verifying whether the state is safe or not, and the pre-trained safe policy that brings the agent to a safe state. However, these approaches make the algorithm hard to generalize to other applications. In this paper, the assumption is that the agent only sees whether the state is safe or not “only after” it visits that state. This makes the problem challenging and the learning agent needs to learn to be safe from scratch. As a result, the paper says that we can only satisfy the constraint “asymptotically”. The paper proposes a two-policy structure safe reinforcement learning algorithm: a utility maximizer (UM) that only optimizes the reward function without considering the safety constraints, and a safe editor (SE) that transforms the action proposed by UM into a safe action. SE is also a reinforcement learning agent with the goal to maximize the “constraint” reward while forcing the state-action values to be similar before and after editing. There are several contributions: (1) the proposed algorithm does not require an inner loop to edit the action to the safer one, which makes the algorithm easy to use; (2) they show that hinge loss is better than L2 when constraining the distance of actions before and after editing, and (3) they verify the algorithm in the experiment. The experiment results are as follows: (1) Table 1 and Figure 3 show the results of SWU scores and the learning curves over the number of steps. The paper points out that SEditor has a higher score than the baselines on 7 out of 12 tasks. (2) Figure 4 shows the visualization of rollout trajectories at different training stages. They show that SEditor is able to find the feasible path without being blocked, and finally (3) Figure 5 shows the result in the racing task.
strengths_and_weaknesses: Pros:
(1) The paper is well-written.
(2) The assumption presented in the paper is clear. For example, it clearly says that we can only learn to be safe asymptotically due to the lack of the oracle that tells whether the state is safe or not. This point is often missed in the literature.
(3) The literature review is good. It covers the work from safe reinforcement learning, especially the work with a two-policy structure. 

Cons:
(1) As pointed out by the paper, the proposed approach has some similarities to recovery RL. The difference seems to be that recovery RL switches hardly from the learner policy to the safe policy whereas, in this paper, SE edits the action from UM by adding. 
(2) I generally agree that the method here is simple and effective based on the experiment. But there is no discussion about the failure case of the proposed algorithm. One possible failure case of the proposed algorithm is that the action proposed by UM may exceed the editing capability of SE. For example, if the learning agent enters the state called “near-termination state”, meaning that the agent has not died, but every possible action is no longer safe enough to recover the agent from this state. Then the proposed approach could fail.
(3) There is no theory in the paper to support the claim that the agent will learn safely.

questions: (1) In Figure 4, could you give the reader more intuition on why SEditor has a better exploration strategy? 
(2) Other than empirical observations, could you explain why hinge loss is better than L2 loss theoretically? 
(3) After we train UM, do we need SE during deployment. The reader thinks the answer is yes since UM never learns how to be safe. If this is the case, what is the absolute value of the $\delta a$ along the training? I expect the $|\delta a|$ would be the same across training. The reader is curious to see this plot.
(4) How to apply the proposed algorithm in real-world application? For example, learning a policy for self-driving cars.
(5) Please address the comment on number 2.

I am happy to increase the score if the author answers the question well. Thank you.
 
limitations: See the cons in the previous section.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors consider the problem of safe reinforcement learning which has attracted a lot of attention in the recent years. While there are many algorithms to address the problem, they do typically have some limitations. One of the major concerns is that the algorithms are quite brittle and require significant tuning  as well as many interactions with the environment to solve the problem. The authors propose a modification of a recently proposed safety layer to solve some of the issues with safe RL. 

The learning process is decomposed into three independent learning processes: a) scheduling the Lagrangian multiplier weighing  the estimated cost b) learning the utility policy maximizing the reward, and c) learning the projection onto the safety state. While the learning process a) is identical to the classical procedure for learning the  Lagrangian multiplier, the process b) and c) are novel.  Albeit, there were similar approaches in the past (e.g., [Mguni et al 2021])

The authors then conduct a number of experiments showcasing the performance of their approach in the case of small cost limits. 
strengths_and_weaknesses: The paper is well-written and the presentation is great and has only a few minor problems. The experiments look impressive as well. Overall, I enjoyed reading this paper, but there are a few weaknesses in my point of view:
 
1. Some of the design choices are not very clear. For example. it is not clear why separating the policy learning process into two makes sense. There's naturally some empirical evidence, but the splitting process is not very intuitive. I suggest providing an intuition why this approach works and how. 

2. Theoretical analysis is limited. In particular, I am a bit concerned that there's a an implicit assumption that transition between a policy maximizing the reward and a safe policy is continuous. 

3. Presentation of the experiments is quite impressive, but a few improvements are necessary (see questions).
questions: 1. While the motivation for the hinge loss makes sense, estimates of the value functions can be quite noisy. I wonder if the authors made some steps to alleviate this issue. Further, would it make sense to consider the $l_2$ distance between the $Q$ functions instead of the hinge loss? Does it work worse?

2. I suggest expanding the motivation behind learning the policy $\pi_\psi(\cdot | s, a)$ predicting the update $\delta a$ instead of learning the policy predicting the corrected action $\hat a$, i.e.,  $\hat a \sim \pi_\psi(\cdot | s, a)$.  That is, I recommend moving parts of the Appendix D into the main text.

3. The choice of nonlinearity $h$ is a bit confusing, why $\delta a$ is multiplied by two?

4. Is it possible to elaborate if the correction policy takes into account the long term behavior of safe actions. This could be important since the safety layer by Dalal et al makes the correction decisions based on the instantaneous information (costs, states and actions)

5. Please add absolute values for the returns in order to compare the performance to other works. Normalizing the returns unfortunately makes such a comparison impossible.

6. While having an almost zero cost limit is an important problem to consider, it is important to compare the performance on the benchmarks as they were intended. Hence adding some experiments (say point push and/or car push) with the cost limit of $25$ is very important.

7. It seems that the proposed improves sample efficiency of the Lagrangian SAC, but doesn't really improve the performance in many tasks. Hence, I suggest running the experiments for longer steps for SAC to make sure that the performance is indeed worse. 

8. The baselines are not unfortunately the latest in safe RL. I suggest comparing to [Stooke et al 2020]  and [As et al 2022]. The latter reference published numerical values of their experiments with the code. One could also consider [Liu et al 2022], but it was just recently published. 

9. Could the approach work with a pre-trained utility policy? That is, making an already existing policy safe. 

Missing references:
* [Stooke et al 2020] Stooke, Adam, Joshua Achiam, and Pieter Abbeel. "Responsive safety in reinforcement learning by PID Lagrangian methods." International Conference on Machine Learning. PMLR, 2020.
* [As et al 2022] As, Yarden, et al. "Constrained Policy Optimization via Bayesian World Models." ICLR 2022 
* [Mguni et al 2021] Mguni, David, et al. "DESTA: A Framework for Safe Reinforcement Learning with Markov Games of Intervention." arXiv preprint arXiv:2110.14468 (2021)
* [Liu et al 2022] Liu, Zuxin, et al. "Constrained variational policy optimization for safe reinforcement learning." arXiv preprint arXiv:2201.11927 (2022) 
	
NB: I would consider raising my score if some of these questions are answered, especially the concerns relating to the experiments section.
limitations: 
1. Providing some intuition would help to understand the approach better. 
2. I am not sure if anything can be done for the theoretical analysis, but it would be a quite welcome addition as well.
3. It is necessary to consider the case of large cost limits.
4. As discussed above, comparison to more baselines and the presentation of the existing results should be improved
review_title: null

Meta-review: metareview: Reviews and responses make sense. The authors made a lot of improvements during the review process. The updated version could be an accepted.
confidence: Less certain
Decision: Accept