Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a safe reinforcement learning algorithm, CRABS, which can achieve zero training-time safety violations. The key idea is to iteratively learn the dynamic model, the policy and the barrier certificates. The barrier certificates are learned via adversarial training and can constrain the exploration of the policy in the safe set. The proposed algorithm is evaluated in four simple classical control problems (variants of inverted pendulum and cartpole). The evaluations demonstrate zero training-time violations and comparable learning performance compared to other safe learning baselines.
main_review: Safe learning with zero training-time violations is critical for applying reinforcement learning to real-world systems, such as robotics. This paper proposes a method that nicely combines reinforcement learning (SAC) with control theory (barrier certificates) to tackle this exact challenge. The paper is well written. The method is novel. The results are promising.

Although I like the proposed method and I believe that it has good technical contributions, I did not give the paper a high score mainly due to the evaluations. I am disappointed that the method is only evaluated on simple control problems based on inverted pendulum and cartpole. Although it seems that CRABS can beat baselines (CPO, recovery RL) in these environments, it is not evaluated in more complicated robotic environments, such as locomotion, navigation, and manipulation, where the baselines work well. To make CRABS useful and impactful, experiments that show CRABS can work equally well or better in complex environments are essential.

Here are a few detailed suggestions how the paper can be improved:
1) It is important to include more evaluations on challenging environments and show the advantages of CRABS. I understand that the model-learning part is the bottleneck to scale up the method to high-dimensional environments. Since for many control problems, the model is well known (e.g. the unicycle model for wheeled robots), it is still valuable to show that if the ground-truth or an approximate model is given, CRABS can guarantee safety during the training process. A navigation task with wheeled robots (safety gym) seems to be a good example to add.

2) It appears to me that CRABS learns a model and uses that model in barrier certificate, but does not leverage the model in policy learning. The policy learning is a modified SAC, a model-free method. As we all know, model-based RL is more data efficient than its model-free counterparts. I wonder whether the policy update of CRABS can use the learned model, and then becomes more data efficient. This is just a suggestion for the long-term research direction, not a criticism for the current paper.

3) The second paragraph of 2.2 seems incorrect to me. According to the text, If h(s)>=0 and h(s')>=0, and then h is a barrier certificate. If this is the case, finding a barrier certificate would be easy: h = |s|^2. Did I miss anything?

4) Throughout of the paper, there are a few equations that have argmax_{s:h_\phi(s)<=1} U. I am not sure where "h_\phi(s)<=1" comes from. Shouldn't it be h_\phi(s)<=0?

5) Since the effectiveness of CRABS heavily depends on the model accuracy, it would be nice to have an ablation study to understand the relation between the number of training-time violations and the model accuracy.

6) It would be great to give more details about "Handcrafted barrier function" paragraph in Section 6. Since how the barrier certificate is handcrafted would significantly impact the performance of the algorithm, the conclusion that "this variant does not perform well" is less convincing without details of handcrafting.

7) Typo: Line 352: rseults->results
limitations_and_societal_impact: Both the limitation and the potential societal impact are discussed.

A suggestion: I like Appendix E, which is a comprehensive summary of the limitations. It would be great to make limitation more explicit in the main text, instead of "hidden" in the Appendix.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work proposes an algorithm of iteratively learning dynamical system model, barrier certificate and policies, for safe RL.
main_review: The overall idea of co-learning of dynamics, barrier certificate and policy are interesting and important.
And the paper is clearly written.
However the reviewer has the following concerns.

Major concerns:
1) The author(s) mention that the proposed method potentially works for higher dimensional systems; however the experiment is only considering rather simple models.
Also, the work stresses that the proposed method aims at zero training-time violations and compared it against existing safe RL work; however, as the author(s) also mention in the experiment section that more data are necessary to achieve zero violations, safety violations ultimately depend on data, representational capacity, disturbance, convergence of algorithm, approximation of confidence interval etc etc. in practice.  Note theoretically it is easy to say zero-training violation if one makes certain assumptions.  Therefore, it is very important to clearly discuss why the proposed method outperforms others in terms of safety violations from several perspectives.

2) There is already an existing work that does co-learning of dynamics, Lyapunov function (through value function learning), (and policy). (e.g. Berkenkamp)  This work basically replaced Lyapunov function by barrier certificate, and generalized the problem setting.
(please consider the nominal known dynamical system model as that can be obtained by initial safe policies assumed in this work)
Therefore, it is again very important to discuss why the use of barrier certificate with the proposed adversarial training may be better than that. 

If the above concerns are addressed, the reviewer is happy to raise the score.

----------
review updated after the initial response
limitations_and_societal_impact: Limitations are well discussed.  However, as long as the author(s) stress zero training-time violaton, the limitation is crucial.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes an RL algorithm that starts from an initial safe but low return policy that is iteratively improved while ensuring no safety violation occurs during the rollouts required by the training. The paper achieves this by learning barrier functions as safety certificates.  The two "simpler" barrier conditions are directly enforced in the structure of the learned function. In contrast, the "closed under a policy+environment step" condition R3 is checked heuristically on a probabilistic model of the environment with a Metropolis-adjusted Langevin algorithm. 
Finally, the authors evaluate the achieved return and number of safety violations of their RL algorithm and other RL algorithms on variations of the Pendulum and CartPole gym environments.
main_review: I think the paper is well written.
Nonetheless, I noticed a few unclear parts of the paper that I would like to authors to clarify.

To check R3 requires solving the non-convex/concave global optimization problem in Eq. (1) and its derived forms. As MALA works only with a set of local maxima, no guarantees about the global optimality can be made (it seems to be only a heuristic). I think this design choice is okay but needs to be communicated more clearly, i.e., as it is very well done in section 4.3 that there is a tradeoff between feasibility and correctness of the transition model (in the sense that a flawed dynamics model can lead to incorrect safety guarantees of the barrier function).
 

The section about the handcrafted barrier function is a bit imprecise. As a barrier function is policy dependent, I suppose the "handcrafted" function is a barrier function of the initial safe policy. Therefore, I would then expect that the achievable return when training the initial policy very much depends on the specific barrier certificate, i.e., if by the initial policy non-reachable but actually by a better policy reachable safe states are considered safe or unsafe.
Moreover, how does the scenario with a handcrafted barrier function lead to safety violations? This shouldn't be the case? Please elaborate.

In Figure 2 top row, it seems that CRABS learns faster than unconstraint SAC. This is counter-intuitive and suggests that the hyperparameters are inadequately chosen, i.e., either SAC hyperparameters are non-optimal or CRABS hyperparameters received a much larger tuning budget.

I like that the authors discuss the limitations of their approach with respect to higher dimensional problems, i.e., that learning a dynamics model becomes challenging and leads to safety violations.

I positively acknowledge that the authors provide the code for their experiments.


I am looking forward to reading the author's response.
limitations_and_societal_impact: I think most of the limitations of the paper are sufficiently communicated (except my first concern in the main review).
review_title: null

Meta-review: I agree with the reviewers that this paper deserves to be accepted; it has a novel foundational contribution and is well written. 

The paper presents a new approach that iteratively generates a barrier certificate, an environment model and a policy, in a way that ensures that there are no safety violations even during training. The algorithm requires an initial safe (but possibly suboptimal) policy, and optimizes it while ensuring that there will not be any safety violations during training. 

As pointed out by one of the reviewers, the main downside of the paper is that the experiments are not very ambitious; the approach is evaluated on only a few problems with a very low-dimensional state space. However, the task of training with no safety violations is already challenging even for these relatively simple environments. There are also a few places where the explanations could be a little clearer, for example, the way algorithm 2 is presented, it is not entirely clear where the initial barrier certificate comes from. But overall, this is a very strong paper with a solid contribution to the state of the art in safe reinforcement learning.


Decision: Accept (Poster)