Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents a method to avoid potential future catastrophes in a reinforcement learning setting.
It uses the framework STEVE because it explicitly estimates the uncertainty for the dynamics but any other MBRL with uncertainty quantification could be used. 

The method works by checking for failures in the imaginary rollouts and then replacing the faulty actions with an MPC controller. The catastrophe prediction is done with an ensemble of networks trained by supervised learning with human labels, called the Blocker Network.
The loss for model learning consists of:
- the loss used in STEVE
- an additional cross entropy loss for their binary network which classifies whether the current transition is safe or not (supervised learning with human labels)

The expected reward optimized by the policy network consists of three terms:
- the standard reward from the environment 
- an exploration term (model entropy) that is proportional to the probability of *not* being in a dangerous region
- a safe-learning reward that explicitly penalizes unsafe areas

strengths_and_weaknesses: ### Strengths

This method is able to avoid non-local catastrophes, meaning that the decision to avoid failure is done many steps ahead rather than right before the critical point. 
The single components are not novel but their combination is novel and works better than the presented baselines.

### Weaknesses
The method consists of many components and it is not very easy to put everything in place, the presentation is a bit confusing and it could be improved. 


I believe the authors could compare their work to other baselines that also employ active exploration.
By reading the paper it seems that this work is the first one to have both active exploration and safety awareness. I would rephrase the introduction/related work and add more comparisons to existing methods. The presented model-based ones are more an ablation of MBHI.
The authors could better summarize the necessary additions - that are now scattered in the paper - and present some take home messages. For example, if separating the buffer is an important addition (which is briefly mentioned in line 244).

Additional comments listed in Issues.

summary_of_recommendation: I believe the paper could be better organized and optimized for clarity. Some details are missing and should be included. Some baselines are reported in the results but nothing that specifically includes active exploration and risk awareness. 
I think the paper has potential: the idea of avoiding future catastrophes in advance and to include human intervention is valid. I will modify my score upon improvement of the paper.

Edit: Updated to Strong Accept after rebuttal
issues: Regarding Figure 9 in the Appendix: what we see are the replaced actions or the original ones? What are the unsafe areas in these cases? 


You don’t really show how accurate the blocker network is. It could be interesting to add a plot with the labeled areas and compare to the regions where actions are replaced. As a reader, it is not clear what the labels are. Are only the obstacles labeled as unsafe?

It is not clear to me if the obstacles are modeled by the dynamics.

Instead of using a blocker network, did you try explicitly adding a penalty cost as a distance to the obstacles?

What is the final difference between your modification of HIRL and MBHI? Is it the blocker network and the buffer split? This information should be made explicit.

Line 158, what do you mean by “asymptotically consistent”?
Line 162, did you try to plot lambda over time? It could be interesting to see.
Line 189, it is mostly clear what is the use of the KL divergence term, namely to sample away from unsafe regions, but it could be better explained.


Many typos, please correct.
e.g.:
line 43: the catastrophe of locally avoidable
or line 260: sampling samples
Line 263: and the model only be pre-trained
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a way to perform model-based RL with catastrophe action blocking and active correction. It builds on the MBRL algorithm STEVE (which comes with uncertainty quantification), and also human-in-the-loop method HIRL (which trains a human blocker model), with the additional technique of unrolling imaginary trajectories and optimize the future trajectory. The algorithm is able to outperform several baselines in catastrophe prevention and reward maximization. 
strengths_and_weaknesses: I think the main strengths of the paper are: 
- Novel technique of unrolling imagined trajectory
- Good literature review of the field
- Promising experimental results

The weaknesses:
- Experiment details: There could be more details on the algorithm and experiment settings. For example, it is unclear what's the human workload during the human supervision phase.
- Baseline comparison: Since this method involves human blocking, there could be other types of human-in-the-loop algorithms serving as baseline comparisons. For example, instead of naive DDPG, DDPGfD* might also serve as a fair comparison since that involves human demonstrations rather than just RL from scratch.

* Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, Martin Riedmiller
summary_of_recommendation: I would recommend the paper because of its insight and novelty, the introduction of human supervision and blocking in MBRL and extensive experimental results. It could potentially benefit from more algorithmic details and baseline comparisons as mentioned above but overall I think it is an insightful work.
issues: - Has the authors considered other types of human-in-the-loop algorithms besides HIRL for catastrophe detection? Would be curious for any other existing work in the field
- More details could be given in the Appendix / supplementary materials section on algorithm and experiment details.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work deals with the safe RL problem.  The authors propose a method where a model of the environment is learned and used to simulate fixed-length rollouts from a current state.  With these rollouts, a supervised learning framework is derived where it is predicted if the rollout is dangerous (based on human-defined labels).  If the rollout is dangerous, MPC control is used to avoid the dangerous state (select safe actions).  The work is highly relevant to robot learning and is validated on the standard suite of MuJoCo safe RL environments.  Compared to other models such as HIRL, this work deals with longer simulated trajectories rather than a single transition so it can reason about non-local danger.
strengths_and_weaknesses: This method is well validated on highly relevant tasks where safety is an important aspect and results are promising.  This method performs well in terms of avoiding catastrophes, but performance in terms of return is not highly significant in most tasks compared with a baseline model-based approach (especially in the non-local catastrophe environment).  There is an absence of real robotic simulations, but I do not think this diminishes the quality of the work. The ablation experiments are useful and highlight the specific impact of the method, though more could be added on parameter sensitivity.

The work appears to be sound in terms of the method. The improvements are simple/incremental over HIRL but the work is shown to be empirically useful for the problem it attacks, avoiding non-local danger.  It is validated on the non-local danger tasks explicitly.  The novelty of the method overall is lacking, but this work combines many different models to form a useful and relevant method.

More comparisons against traditional safe RL algorithms could be present, such as a policy optimization method with conditional value at risk objective (CVaR). Comparing against CVaR would make a useful comparison against a method not requiring human intervention, which is difficult to obtain.  I would also have liked to see some comparisons against methods with more constrained updates such as TRPO or PPO as opposed to DDPG.  

The text is generally polished.  There are a few minor details to point out: 
- The authors alternate between the notation s’ and s_{t+x} between the method and related work section.  
- \alpha in (9) is superscript and appears to be a power instead of a weight.
- In (14) tau is written instead of the symbol.
- The parameters for each part of the dynamics model could be combined into a single set with its own symbol?
- Notation could be simplified at many steps such as (13) where the bound is not stated.  
- It is unclear how many random seeds are used.

Overall, the visuals are very useful in understanding the method. The paper reads clearly.

summary_of_recommendation: I think this work is borderline.  The method is relevant to CoRL but is lacking some key evaluation baselines and novelty in general. The authors should be able to make minor adjustments to the text and overall, the paper is relatively easy to understand.  

**Post Response**:  While I still think this work is borderline, due to the revisions from the authors I am now leaning towards acceptance.  I believe the additional baseline and ablation analysis solidify the usefulness of the method and believe the paper should be "accepted if there is room".
issues: See above
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper proposes a method which utilizes a policy guided by dynamics and value-function predictions and uncertainties over unsafe states for safer exploration and learning.

strengths_and_weaknesses: **Paper Strengths**
Technical contribution: The method is an amalgamation of different ideas that results in a well-performing algorithm which demonstrates improvements over a number of baselines. While no part itself is particularly novel, the combination of everything makes the method a novel contribution.

Experiments: Experimental evaluation was solid with respect to the baselines and environments tested. The method performs well in terms of catastrophe avoidance, and offers mixed to strong improvements over baselines on final reward.

Paper Writing: Given the complexity of the method, the method is generally described well.

**Paper Weaknesses**
Missing related work: Cautious Adaptation for RL in Safety-Critical Settings (Zhang et al 2020) also targets state-based safety through an ensemble of dynamics models with MPC planning, and SAVED (Thananjeyan et al 2020) performs safe RL from human demonstrations, with a similar value-function and dynamics-model based cost function. These are important recent works the authors should cite and contrast their method with in the Related Works/intro section.

Claim: “However, existing state-based approaches only consider the catastrophe of locally avoidable, that is, catastrophes can be avoided by choosing the appropriate action just before the catastrophe would have happened.” I don’t think this can be stated as fact. Model-based methods with MPC planning will be able to avoid catastrophes over longer horizons as long as the horizon of the planner is long enough. This is even more true for Dynamics + Value-function based const functions which exist in both STEVE and the aforementioned SAVED work.

Clarity: $T_{STEVE}^H$ seems to come out of nowhere (Eq 5) and its purpose is not explained immediately, and $T_h^\mu$ also should be mentioned.

Ablations: The authors introduce a method that has many different design choices, such as the leave-one-out estimation, gradient updates for CEM, having c_l and c_h, the safety bound, etc. There should be more ablation studies and sensitivity analysis on these parameters in general.

Experiments: I can’t find where the paper mentions the number of random seeds the results are reported over. And as hinted to in the related works weaknesses, there are baselines that can be extended to work on the scenario in the paper that would be nice to compare against.


**Minor Issues**
Typo: Safe Active Explorationm (4.2), Puck-Workd (L214),

summary_of_recommendation: I recommend acceptance of this paper due to the solid performance and the technical contribution of the method. However, I do have some issues listed in the paper weaknesses section that I would like to be addressed during the rebuttal.

issues: See weaknesses/minor issues section.
review_title: null

Meta-review: This paper proposes a safe reinforcement learning method, which switches to a safe MPC controller if unsafe events are predicted to happen. Reviewers generally agree that the paper makes good technical contributions to the important problem of safe learning. While most of the components of this algorithm are not novel, reviewers acknowledge that the combination of them all is novel and useful. The original concerns from reviewers about presentation, literature reviews, and baselines comparisons are well addressed in the rebuttal.
Decision: Accept (Poster)