Venue: ICLR 2022 Conference

Reviews:

final_score: rating: null
review_content: Thanks to the authors for providing a lot of additional explanations and modifications. Based on this and the reviews, I am still positive about the paper. ICLR restricts the positive (accept) evaluations to include something equivalent to "just above the threshold" and a firm "accept". I chose the latter, since I think that the first is too low. However, my evaluation should be read as something in between the two. I share the many concerns of the other reviews, for example when it comes to the matter of "having safe/unsafe demonstrations", and the experimental evaluation. Compared to the safety literature, there are some questions to ask as whether the approach is realistic enough on this aspect, for example. Overall though, I do like the approach and the paper, and the authors have provided extensive and relevant reactions to all reviewers.
review_title: General evaluation

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new safe RL method, SAFER, that uses a behavioral prior learning algorithm to accelerate policy learning under safety constraints. SAFER learns to extract a safety variable from offline data and the safe primitive skills over abstract actions in different scenarios via contrastive training. The authors demonstrate the effectiveness of the methods with several complex safety-critical robotic grasping tasks inspired by game Operation. Results show that SAFER no only outperforms baseline methods in learning successful policies but also enforces safety more effectively. 


main_review: Strength:
The problem is important and interesting.
The paper was well written and easy to understand. 
The experimental results also show the methods have better performance than the benchmarks.

Weakness:
The key components of the methods, including using conditional normalizing flow to represent the behavioral prior, using chance constraints to represent unsafe constraints, using ELBO to optimize the chance constraints are all well-known standard treatments in the field of reinforcement learning. The authors provided heuristics on the choices of these tools but lack rigorous analysis on the proposed methods. Given the authors only one environment to evaluate the proposed method, it is unknown how well the methods could generalize to other problems. 

It is suggested that authors add more theoretical analysis to unveil the novelty and insight of the method and more diverse experiments to show the strength and limits of the methods in a more comprehensive way.
summary_of_the_review: It is an interesting paper but lacks insight into the novelty and the application scope of the proposed method
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper is an extension of "PARROT" by Singh et al. (2021), which does a form of hierarchical pretaining from offline datasets to a setting where preferences need to be expressed on the kinds of behaviors that ought to be generalized. The authors use the expression "safety" to refer to what their work tries to accomplish, but I think that a more direct description of their work is in terms of preference specification via binary variables. 

My reference point to understand this work is to think about hierarchical reinforcement learning, especially the probabilistic viewpoint laid out by Daniel et al. (2016). In this perspective, we think of the "termination functions" (or initiation) in the options framework in terms of the sequence of Bernoulli variables that they induce. Similarly, the proposed SAFER approach introduces an auxiliary "safety" variable "c" which needs to be inferred at execution time.  In the proposed framework, it is as if we were to learn a mixture model where instead of having a categorical distribution over options (discrete), we would instead be using a continuous (Gaussian) distribution to represent our distribution over latent options. The particularity of this line of work (PARROT) is that this Gaussian latent variable is learned in an unsupervised manner over a dataset of successful solutions across different tasks, rather than a purely downstream-oriented manner.

The new component that this paper adds to PARROT is this additional latent "safety" latent variable. (Note: this paper is not very self-contained: I had to read carefully the PARROT, which I didn't know beforehand, to make sense of the proposal.) One complication that arises from this modification is to now enforce that this new "safety" latent behaves the way that is intended to: that is, to steer/control the induced behavior to act "safely" when told to do so. The solution proposed by the authors is to view this as a chance-constrained optimization problem. This allows them to write the safety satisfaction requirement in probabilistic terms (constraint violation), which they then turn into a constrained program where the probabilistic constraint becomes a deterministic one via Markov's inequality. The authors then *do not solve* the resulting problem as a constrained problem, but rather choose to work with a regularized unconstrained approximation coming from the Lagrangian. 

A last important conceptual idea of the paper is that of calibration (which they call "safety assurance") of that safety variable. The problem arises from the fact that in adding this safety knob, we need to ensure that the constraint violation coefficient specified in the chance-constrained relaxation does indeed coincide empirically with the level set by the designer. Here, the authors leverage the properties of normalizing flows to estimate this threshold from historical data. 
main_review: > We rewrite the optimization 5 into its equivalent

Regarding the expression "equivalent": Eq. 5 is a mathematical program, while eq 6 is Lagrangian-like term. Eq. 6 is not a mathematical program per-se. You may want to write it as a saddle-point problem if you want to talk about "equivalent optimization", or say something related to the KKT conditions.  

> Equation 8 and 9 

You use $\theta$ while I think you mean to use $\phi$. 

> we propose the specific policy structure µψ = fφ ◦ πθ, where fφ ◦ πθ(a|s) := R z∈Z fφ(z, s)dπθ(z|s), and ψ = (φ, θ)

I think that talking about function composition is appropriate here because the underlying model is inherently probabilistic. All this is saying is that you have a mixture model, which is a clear concept when explained in those terms. 

Also, in your math expression, I don't understand why you have "a" on the lhs but not the rhs. 

# Actionable feedback 

Be clear about the problem setup. "Safety" can mean a lot of things depending on the framework, and the flavor considered in this work is not a usual one of the kind that we have in CMDPs for example. I was personally struggling to grasp the motivation and the real-world grounding of the problem setup. The operation game for the robot is concrete, but arguably a rather artificial safety-critical setup. What would be a real-real world scenario where your framework would make sense (compared to the alternatives)? I suggest you expand on this in the intro.
summary_of_the_review: My main criticism of this paper is that it presents a fairly specific idea, that completely hinges on very recent prior work by Singh et al. 2021: to the point that the submitted paper is undecipherable unless you read Singh et al. (2021) first (which I had to do). That being said, the paper presents an interesting new perspective and builds on chance-constrained optimization and constrained optimization (but the authors choose not to go deeper in this direction, perhaps because of technical difficulties that are not mentioned in the paper).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes an approach for safe policy learning with a behavioral prior conditioned policy. The approach relies on learning a behavior prior from a dataset of safe and unsafe trajectories with chance-constrained variational inference, and using this prior as a latent skill distribution to condition the policy during reinforcement learning. The proposed approach is evaluated against prior skill learning approaches on simulated robot manipulation tasks.
main_review: The main contribution of the paper is a modification of existing latent skill-learning algorithms, in particular PARROT (Singh et al.) to incorporate safe learning at test-time by learning a behavioral prior from safe and unsafe offline data. The approach is intuitive, easy to understand, and performs better than two external baselines - PARROT and SAC in terms of exhibiting safer behavior after training. 

There are several crucial issues with the paper, that I've listed below:

1. The main drawback of the approach is that it requires an offline data of *both* safe and unsafe behavior for training the latent skill prior. This is fundamentally limiting and goes against the entire principle of safe RL - collecting such a dataset requires multiple failures and hence requires being *unsafe* multiple times during data collection. If the aim of safety is to prevent the robot + it's environment from unintended consequences, implementing this approach on a real robot would lead to multiple failures during data collection and thus by definition be unsafe. 

2. Several prior works (both old and new) on safe RL and skill learning are not cited / discussed. Some relevant papers are listed below [a-e].   The papers [a], [b], [c], and those cited in the paper in section 4 emphasize the non-trivial nature of safe RL by describing how it is difficult to collect data while being safe (i.e. safety during training). This paper completely omits discussion of this point and proposes a safe RL method by collecting an offline dataset while being unsafe. 

3. In light of the points in 1 and 2, the results on safety violations in Table 1 need to be updated to account for all the safety violations during offline data collection. 

4. The paper does not compare with *any* external safe RL baselines. Why are PARROT and SAC the only external baselines compared against? It is important to compare with safe RL papers in the experiments since the main claim of the paper is number of safety violations. Again, for fairness of comparison, the number of safety violations for SAFER should include the violations during data collection.

5. There are no guarantees of any sort on safe behavior. It is important to provide some theoretical guarantees on safe behavior - either during or after convergence. In the current framework, during training would not make sense as the offline dataset collection necessarily involves multiple failures. So, some guarantees on the behavioral prior would help understand the generalization of the method beyond the environments evaluated in the paper. 

6. In equation 9, it is unclear how the objective corresponds to a form of contrastive learning. It will be helpful to clarify what is being contrasted with what and why is that helpful, compared to other possible lower bounds on the ELBO. 

7. How is it ensured that the policy conditioned on the skill prior, simply doesn't ignore the prior during learning? Is there any constraint on the optimization process to ensure that this does not happen?


There are interesting ideas in the paper around learning a safe behavior prior for latent skill-conditioned policy learning. However, as I pointed out above, there are fundamental issue and assumptions on offline data collection that require multiple safety violations in the data collection process, thereby defeating the purpose of the proposed safe RL approach.

[a] Srinivasan, K., Eysenbach, B., Ha, S., Tan, J. and Finn, C., 2020. Learning to be safe: Deep rl with a safety critic. arXiv preprint arXiv:2010.14603.

[b] Bharadhwaj, H., Kumar, A., Rhinehart, N., Levine, S., Shkurti, F. and Garg, A., 2020, September. Conservative Safety Critics for Exploration. In International Conference on Learning Representations.

[c] Brunke, L., Greeff, M., Hall, A.W., Yuan, Z., Zhou, S., Panerati, J. and Schoellig, A.P., 2021. Safe learning in robotics: From learning-based control to safe reinforcement learning. arXiv preprint arXiv:2108.06266.

[d] Xie, K., Bharadhwaj, H., Hafner, D., Garg, A. and Shkurti, F., 2020, September. Latent Skill Planning for Exploration and Transfer. In International Conference on Learning Representations.

[e] Konidaris, G. and Barto, A., 2009. Skill discovery in continuous reinforcement learning domains using skill chaining. Advances in neural information processing systems, 22, pp.1015-1023.
summary_of_the_review: There are interesting ideas in the paper around learning a safe behavior prior for latent skill-conditioned policy learning. However, as I pointed out above, there are fundamental issue and assumptions on offline data collection that require multiple safety violations in the data collection process, thereby defeating the purpose of the proposed safe RL approach.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper builds on previous ideas of behavioral priors in reinforcement learning (RL), more in particular the paper by Singh et al 2021. The main idea is to have a behavioral prior BP function Z x S -> A and a policy over an abstract space Z such that the composition of both will deliver actions that are somehow constrained by the likelihood of actions occurring in a dataset. This dataset is used to obtain the BP in an offline learning sessions, after which standard RL algorithms can learn a policy over Z. Overall this speeds up RL. In this paper this approach is extended with methods that learn the BP in the context of safety conditions (by learning and using an extra safety indicator variable), such that the overall behavior is learned more quickly again but also makes better distinction between safe and unsafe action choices. Much of the paper is about the learning setting (and loss function) to accomplish this. The paper also shows a mechanism to balance performance and safety violations. The approach is tested on a continuous/robotic task of manipulating objects and shown effective, also against some baselines.
main_review: This paper is a clear extension of behavior priors to safety domains. The paper argues that the typical BP setup is not adequate for safety contexts because of how they are typically trained vs. out of distribution samples. One question I have here is about the relation between the quality of the dataset (safety of the demonstrated behaviors, and completeness of the demonstrations wrt the full behavioral repertoire) and how large the problem of OOP actually is in this context. I wish more discussion would appear in the paper about this. Conceptually, one could think of creating a better dataset such that all OOP actions would be really outside the "safe" distribution in the dataset. Here the authors choose for a more explicit solution of labeling (automatically) safe and unsafe behaviors and predicting a separate safety variable as a useful signal to learn the difference between safe and unsafe behaviors. I think it makes a lot of sense, although I am not sure a single binary (or graded, but in the end only the mean of a distribution is chosen) will be sufficient for all safety scenarios. Some discussion on the limits of this approach would be appreciated. I am not knowledgeable enough about the specifics of the learning approach (and the contrastive way of optimizing the desired loss) whether there is a contribution here, or whether this is the "smartest" way of doing it. Nevertheless, it seems this is the first time this (BP + safety) is considered so any solution would be sufficient.

I have another issue with the paper when it comes to the "abstract" action space in the function composition of the policy and the BP. I think some of the terminology is confusing a bit, and maybe too close to the deep learning formulation. What I mean by this, is that abstractions in the action space are used very often in MDP contexts, but typically such abstractions are given/modeled/learned (e.g. an abstraction putON(X,Y) abstracting real actions like putON(a,b), putON(b,c), putON(d,a) etc.) or they are part of a behavioral hierarchy (such as in hierarchical RL, with move-to-roomX, with low-level actions in the MDP such as north, south, west, etc). The paper mentions terms like "downstream task", and hierarchical, and abstract, but the definition of Z wrt the MDP (or POMDP for that matter) is not made expliciet (enough). Also, it is unclear how the paper (and BPs in general) relate to many other works in continuous spaces where abstract actions are used which have to be "instantiated" or "grounded" when actually used (e.g. to choose parameters and specific values). To make the paper better, a better description of BPs and how they relate to the problem ((PO)MDP) would be needed, and only after that one should move to a deep learning solutions where Z happens to be positioned in a (relatively random) latent space. This would also make things more clear in terms of learning high-level behavioral choices (Z) and "mappings" from Z to actual actions in the underlying problem space. I checked Singh et al 2021 and the description of BPs there is longer and better, but also lacks a better grounding of concepts in the underlying problem. To make the paper better, and to connect better to related literature, these things need to be described better and more in full.

Overall, a well-done paper with a clear contribution and several good experiments showing the benefits of the approach. The description of the approach can be improved (see issues above, and below, related work, overall story, bibliography, typos, etc).

- "inspired by the game Operation"; I was disappointed it was not about the real game itself :)
- p1:par2:line3 "use a generic offline datasets" (grammar)
- (see also above): explain the non-standard "downstream"
- Figure 2 (and especially the caption text) is not useful at all for me.
- The safety variable can be motivated better
- title section 3.2: "learning THE safety variable"?
- Maybe the chance constrained optimization needs more explanation (technical, especially the relaxation)
- I like the approach in section 3.4. although it also sounds like a practical hack (no offense)
- The related work is ok, but could say more about the safety in RL literature (beyond safe exploration)
- sec5.1:par2:line1 "whose" -> "which"(?)
- sec5.1:par2:line3 "the PyBullet" (grammar, and sentence ends strangely)
- Experiment:table1 I think it is strange that all the baselines perform so badly on the task (success rate). This table says that all methods are not capable at all, and only SAFER works (but only sometimes). This sounds weak. Can you elaborate?
- Experiment:table1 The results for safety violations show large variations over methods and tasks. Is there any pattern here (other than SAFER is more robust)?
- I like the experiments in Fig 5, since they show how safety can hinder raw performance in various ways. Maybe this could be discussed more in full, and earlier in the paper. I think it is relevant.
- Conclusions/future work are quite short, and the bibliography needs polishing.

summary_of_the_review: Overall, a well-done paper with a clear contribution and several good experiments showing the benefits of the approach. The description of the approach can be improved (see issues above, and below, related work, overall story, bibliography, typos, etc).
review_title: null

Meta-review: The paper provides an algorithmic framework to accelerate RL through Behavioral Priors, while having some notion of safety incorporated. The reviewers are divided about this paper:

On the positive side, some of the reviewers consider the problem important, and the experimental results reasonable and promising.

On the negative side, reviewers raised issues such as
1) The paper is on a heuristic side.
2) No formal guarantee on the safety is provided.
3) The paper is not as self-contained as it should be, as it relies much on Singh et al. (2021).
4) The algorithm requires access to unsafe offline data.

I do not give the same weights to all these concerns. For example, even though (d) is an issue in some applications, it is alright for others. What concern me most are (1) and (2). 

A method for safety that is only evaluated empirically and does not have any formal guarantee cannot be used for safety critical tasks. I realize that some other published papers may have the same issue. But given that this is a real concern, and that two out of four reviewers believe that the paper should not be accepted, unfortunately I cannot recommend acceptance of this paper, especially given the competitiveness of this conference.

P.S: I also noticed that in the proof of Proposition 3.1, an expectation term $E[p_\phi(a|s,c)]$ in Eq. (9) is replaced by a $\log p_\phi(a|s,c)$. This requires more justifications.
Decision: Reject