Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: Authors introduce a safety-oriented policy-refinement method for reinforcement learning. Authors use Bayesian optimization to find safety-violation traces from a policy generated by a vanilla RL algorithm. Those traces are latter use to produce a new trained policy that satisfies the requirements. The method is meant for continuous-action environments
main_review: Authors present a novel method for generating safe RL agents as far as I am aware, both the process of finding counterexamples through Bayesian optimization and  their policy refinement method are interesting contributions. However, authors claim two times that their are first to refine learned RL policies through counterexamples. This is not the case since [1] already did that. However the proposed method is significantly different from that previous approach. Moreover, the proposed method is applicable to a wider range of applications since it can handle larger state spaces than the model checking approach from [1].

The paper is clear and well written, authors introduce various images and running examples that makes the approach intuitive and easy to follow. They provide sufficient proof and explanation with definitions and various experimental benchmarks. I only saw a handful minor things that needed correction.

Overall is a good contribution and I recommend acceptance (upon that authors correct the two times they say to be first to present a counter-example refinement method for RL and include [1] in their references).

Additional comments:
- Error intervals are barely visible  in figure 3, I strongly suggest to draw the plots with a different tool than tensorboard
- What do the error intervals represent, quartiles? std?
- Table titles should be above the table according to neurips style, also in the caption, II suggest recalling what n is

[1] Zhou, W., & Li, W. (2018, July). Safety-aware apprenticeship learning. In International Conference on Computer Aided Verification (pp. 662-680). Springer, Cham.
limitations_and_societal_impact: Authors explicitly state their limitations in the conclusions
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a counterexample-guided policy refinement method to check and correct an already trained RL policy. The proposed algorithm (i) uses Bayesian Optimization to find failure cases in a policy, (ii) then learns a new sub-policy to locally overcome the failed cases, and (iii) finally uses a slightly modified PPO algorithm to refine/update the initial policy to eliminate the failure cases in a proximal manner. The paper is evaluated on several OpenAI Gym environments and shows they are able to find and fix several failure cases with a small mean variation distance between the policies. 

main_review: Strengths:
- Using a counterexample guided approach in RL to ensure safety is an interesting approach
- The algorithm to proximally refine the policy is neat and well done 

Weakness:

This paper needs a more thorough evaluation including comparing to a few baselines, evaluating on more metrics, and on harder problems 
- With the lack of baselines, it is hard to appreciate the numbers in the evaluation. One obvious baseline would be training/fine-tuning with the combined reward function (reward + loss penalty, the total reward used for training $\pi_c$) on all initial states (essentially eliminating the Bayesian Optimization step. 
- Also, please report the initial reward, final reward, # failure cases in the final policy.
- It will also be interesting to see how this algorithm affects the sample complexity (which needs to take into account the sample complexity for both the Bayesian optimization and the retraining steps) and compare to other baselines/approaches' sample complexity.
- A tricky part of any counter-example guided algorithm is that it requires multiple back and forth iterations and sometimes it does not converge. When you fix a policy for some failure cases, more and newer failure cases creep up. However, in the current evaluation, it seems like all problems are solved with just 1 iteration. Can the authors comment on this? Is this some special property of your algorithm or are the tasks too simple?


limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: this paper studies the problem of how to train a policy \pi which maximize rewards while retaining certain safety properties, stated as logical constraints. a BO algorithm discovers failure trajectories, which are then added to the PPO learning algorithm to steer the policy away from constraint violations.
main_review: ### Originality and Significance:
I am a fan of this line of research, I think incorporating constraints as a way of specifying safety properties might be more clear to end-users than reward engineering or preference learning. As constraints are something (presumably) easy to understand by both algorithms and humans, it is a natural avenue to explore in human-machine communications in the realm of safety specifications.

### Clarity:
I do not think this paper is clearly written. Specifically, I spent a long time trying to understand what exactly is the query space for the BO. 

To put it in lunar lander terms, is BO trying to discover a failure trajectory by:
1. put the lander into a weird initial location (x,y) with an odd angle so that it fails?
2. varying the mass / size of the lunar lander so that it fails?
3. forcing the policy \pi to make a low probability choice so it is more likely to fail?

The key phrase "Parameter Bound" seems to denote the space of query points for the BO, but is present in the paper only twice, and later in table1 there's a column named "domain of uncertainty". Are they the same thing? I'm assuming they are, but I am very lost.

I would suggest use lunar lander as the motivating example, as example 3.1 is way too detached from the main body of work to be the "opener example". I would also make the space of BO very explicit, if my understanding is correct, I would love to see a phrase of the form "randomly sampling mass and size of the lunar lander is unlikely to uncover a failure trajectory, but with BO we can actively query for a failure case of mass and size, and reliably sample failure trajectory with high probability. This is assuming the space of BO in lunar lander is this "parameter bound" or "domain of uncertainty", which could be wrong.

### Quality:
[edit : with the added random sampling baseline I think it is sufficient quality]

This work is (not yet) solid.

It is unclear to me from the experiments that BO is warranted for the domain in question. There should be some kind of metric showcasing how BO out performs just random sampling

It is also unclear to me that if the proposed method "proves" that no failure can occur, as the BO is working over a stochastic policy \pi, it may be possible that under some roll-outs, the policy will exhibit unsafe behaviour but simply "got lucky". At the very least, there should be an experiment that empirically justifies that your proposed algorithm gives a policy that is safer over many roll-outs.

I would suggest inclusion of a comprehensive table where, for each task (lander, walker, etc) you train three different policies:
\pi_a) without safety corrections, only maximizing rewards
\pi_b) with corrections, but sampled randomly without BO
\pi_c) with corrections, and sampled with BO
Then, evaluate how \pi_a, \pi_b, \pi_c performs on 10k new roll outs, notice the percentage of states that have failures. Hopefully, you'll discover that \pi_c have a very low number of failure states (this isn't going to be 0, due to the stochastic nature of \pi).

Overall, this work does not "prove" the resulting policy \pi_c is absolutely safe, and does not justify it empirically either.
limitations_and_societal_impact: I do not see a "limitation and negative impact" section in the paper.

I find the check-list item "Did you discuss any potential negative societal impacts of your work? [N/A] paper presents works on safe RL with no foreseeable direct societal implications" incorrect, as safety in RL is (at least it is touted to be) very important for societal implications as we need to trust the AI we built. In fact, the first paragraph of the intro makes a very strong case that this line of work has huge safety implications. This is _inconsistent_ with how the paper handled the checklist item with [N/A].
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a training methodology to incorporate constraints into a trained RL policy. It uses Bayesian optimization to select parameters for the MDP that are most likely to produce trajectories violating the constraints and then performs PPO updates to steer the policy away from these trajectories while remaining close to the original one. The experiments are conducted on popular OpenAI gym environments.
main_review: I'm not an expert on safe RL and I found the presentation of the topic given in the paper rather confusing. In particular it's not clear to me what exactly is the specific problem with directly penalizing constraint violation in the reward function that the authors are trying to solve. The authors claim that their method avoids such an approach, yet the proposed PPO updates look very much like adding negative reward for constraint violation. Those updates also seem to be largely perpendicular to the choice of using BO for MDP parameter selection and it's unclear to me whether the authors claim each of those separately as novel contributions or only their combination.

For me the fatal flaw of this paper is the underwhelming experimental evaluation, in particular not including any baselines. I don't quite understand how the original policy in the experiments was obtained, but my impression is that it was trained without any constraint violation penalties and I would very much like to see a comparison with a policy being trained in such a setting with normal PPO. On the BO side, at the very least we should see a comparison with random sampling. Without those experiments, I don't think the paper can be accepted.

EDIT:
raised score after rebuttal, see the rationale in the discussion below
limitations_and_societal_impact: N/A
review_title: null

Meta-review: This paper has considerable support from the reviewers but not does not seem to rise to the level of an oral.

For me the formal adversarial set up is strange.  Some form importance sampling under an "unsafe" state-transition bias would yield an estimate of the probability of a safety violation. Optimizing an estimate of the probability of a safety violation, possibly with stochastically drawn model parameters, seems like a cleaner conceptual set up to me.  But the reviewers are happy and I will go along.
Decision: Accept (Poster)