Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 
### Contributions
* This paper proposes a safe model-based deep RL approach where
    * No simulator is needed. The dynamic model is learned from data.
    * No constraint is specified.
* This work is an extension of reward query synthesis via trajectory optimization (ReQueST).

* ReQueSt (previous work)
    * Algorithm
        * Dynamics learned from (potentially unsafe) random exploration.
        * Reward learned from binary human feedback that is generated by a procedural reward function.
        * Demonstrated in State-based 2D navigation (non-pixel-based), Image-based Car Racing (pixel-based, 64x64×3).
    * Regarding safety
        * ReQueST avoids the problem of safe exploration by allowing agents to explore these states in a simulated model without having to visit them in the real environment.
* This work
    * Algorithm
        * Dynamics learned from safe demonstration provided by humans (160 person-hours).
          * Similar to ReQueSt, `except that we use a larger encoder network, a deconvolutional decoder network, and train using a simple mean-squared error loss between ground-truth pixel observations and predicted pixel observations.`
        * Reward learned from
            * Type of feedback from humans: reward sketch (previous work) (10 person-hours).
            * The proxies for value of information: maximization and minimization of reward as predicted by the current reward model.
            * The algorithm for reward learning: MPC.
        * Demonstrated in 3D env
          * Task goal = eat all the 3 apples
          * Env 1 = Cliff edge environment
            * Safety = not fall off the end of the world
          * Env 2 = Dangerous blocks environment
            * Safety = bumping into blocks
          * Each env has 3 Sized subvariants.
          * Each env is pixel-based, 96×72×3.
    * Regarding safety
        * Safety is achieved by focusing the learned dynamic model in the safe scenarios, with only the safe data. `Given models of sufficient fidelity, this should allow us to train an agent with close to zero instances of unsafe behaviour in the real environment.`

### Results
* Env 1 = Cliff edge environment (Fig5)
    * Safety violations during training: proposed method << model-free RL
    * Safety violations during testing: proposed method = model-free RL << random policy
    * Apples eaten: random policy <= proposed method < model-free RL
    * => Proposed method vs model-free RL => tradeoff between safety violations during training and apples eaten
* Env 2 = Dangerous blocks environment (Fig6)
    * Safety violations during training: Proposed method << model-free RL
    * Safety violations during testing: Proposed method << model-free RL = random policy
    * Apples eaten: random policy = model-free RL < proposed method
    * => Proposed method is better than model-free RL and random

main_review: 
### Concerns
1. This method ensures safety by learning the dynamics of safe human demonstrations. However, if some of the human demonstrations are unsafe, would there be a way to know? Or would this method be sensitive to those cases?
2. This method requires a lot of human data, which is ok considering the challenge of the task. One concern is that it seems unclear how to determine when the training for dynamics (and reward) are "completed." Especially for safety-critical tasks like the ones suggested by the authors, what would be the indicators in the proposed method that can suggest: "ok, now the learned model is safe enough, please feel free to deploy."
3. A programmatic reward bonus is applied to the proposed method, not baselines. Would this mean that the reward learning component in this method is insufficient to recover the reward function?

### Minor points
1. Env2 has blocks in the world, where if the agent moves the block, the reward it will receive will change. This is the main reason why in the results, model-free RL performs much worse than the proposed method.
    * I think this is a good example of reward hacking, where the defined reward - apple picking and avoiding touching blocks, does not match the actual goal of the task - apple picking. Hence, model-free RL performed poorly. The proposed method leverage human demonstration to avoid suffering from the misspecified reward.
    * This indicates the power of the proposed method to handle misaligned rewards. However meanwhile, this might also indicate that the proposed method cannot completely outperform model-free RL when the reward is specified well, such as in the cliff env.

summary_of_the_review: This paper is an extension of the previous work ReQueSt, with the focus on learning safe policy from human demonstrations only, without simulators or specifications. There are some concerns about the approach. It has a very nice demonstration in challenging pixel-based 3D tasks.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper shows that  ReQueST can be used to train an agent in a 3D environment with an order-of-magnitude reduction in instances of unsafe behaviour than typically required with reinforcement learning. No procedural specification of safe behaviour is required with minimal assumptions other than unsafe or near-unsafe states being recognisable by humans.


main_review: The paper shows that ReQueST is plausibly a general purpose solution to  the safe exploration problem: where safe behaviour is learned from humans, rather than given by a procedural function, and the simulator can be learned from data.

There are several weakness, for example, the task of apple picking, is this sufficient to show the challenge of RL or more difficult tasks should be used?

The paper also does not do a detailed comparison with existing approaches. If no existing approaches can achieve what the paper proposed to do, then please clearly say so. Otherwise please do a comparison to strengthen the  claims of novelty of the paper.
summary_of_the_review: Overall it is an interesting paper with some good results shown. I am not sure the results are sufficient enough. I would also like to see more detailed comparisons with the state-of-the-art.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes an extension to ReQueST, which learn learns a neural simulator of the environment from safe human trajectories and then learns a reward model from human feedback. This work extends ReQueST by dense reward sketches on imagines trajectories and evaluates the idea on a visually-complex 3D environment. The paper also discusses the amount of training data that is needed for ReQueST to work in such a setting. The authors collect 160 person-hours of "safe" human exploratory trajectories and 10 person-hours of reward sketches. Their results show that the application of ReQuest results in "3 to 20" times less constraint violations compared to a non-safe baseline.
main_review: I generally like the idea of the paper to bring ReQueST together with human feedback through reward sketches. The paper is well-written and easy to follow. The experimental setup is also well-defined and described. However, the second environment should be better justified as the setting looks very unusual.
The authors clearly state their contributions and delineate it from previous work. The use of real human data and reward sketches is laudable, and an important contribution. Experiments are replicated 10 times, which is a sensible amount.

However, this only seems to be a very incremental work over the existing algorithm. Additionally benchmarking ReQueST on 3D environments is valuable but not a contribution that is sufficient for publication in ICLR. The paper is better suited as a workshop contribution. 

Also, I'm not convinced about the numerical results. Their main safety result shows that ReQuest is unable to outperform a simple baseline (more on this baseline in a bit) during the evaluation, performing less safe and gathering fewer apples in the "Cliff Edge" environment. The authors should identify the problems encountered by ReQuest and their modifications here and propose a solution that improves the performance to be better than the baseline. I would also actually argue that having constraint violations in the gathered dataset should improve the performance for an algorithm (not necessarily for ReQuest), since the agent should exploit the knowledge on constraints this dataset contains.

Some further points:
-	In Fig. 2 you could more explicitly show the difference to original ReQueST
-	While the authors use a large amount of replications, they only include a classical RL baseline. This baseline uses a simplistic, sparse reward, rather then the richer reward sketches used by ReQuest. Also, it penalizes constraint violations by the same amount that it rewards successes. This kind of baseline method will never perform safely during training.
-	The paper should also compare itself against well-known SafeRL algos and/or against classifier-based approaches (such as the original version, why don’t you compare to vanilla-ReQueST in order to show the benefits?). Also, while the authors argue against imitation and offline learning, such approaches could also serve as baselines and provide experimental evidence.
-	Since they claim to provide superior safety, the authors should compare their method to recent Safe RL methods, like 2020 Stooke: Responsive Safety in Reinforcement Learning by PID Lagrangian Methods, or 2020 Srinivasan: Learning to be Safe: Deep RL with a Safety Critic. The latter method uses a pretraining step to bootstrap the safety critic before training, which could use a similar dataset than the one used in this study. Rather than only evaluating ReQuest, it could be a more interesting paper if multiple Safe RL approaches based on the dataset are evaluated.

summary_of_the_review: I generally like the idea of the original paper (and hence also this paper) but in my opinion this paper is not enough for publication in ICLR. The contribution is very limited over the original ReQueST.
review_title: null

Meta-review: This work takes ReQuest, an approach for safe deep reinforcement learning utilizing human feedback, and studies it's feasibility in pixel-based 3D environments (previously it was only shown to work in simple 2D environments). In order to apply ReQuest in this much more challenging settings, this novel instantiation of ReQuest learns a pixel-based dynamics model from a lot of human demonstration data, and a different (as compared to the "base" ReQuest) reward sketching approach to infer the reward function from human feedback. 

**Strengths**
globally a well motivated and well written/structured manuscript
Adresses an important problem, and shows promising results

**Weaknesses**
on the more detailed level, there are some clarity concerns (even with the lengthy appendix)
evaluation was missing some more relevant comparison (partially fixed after rebuttal/revision)
lack of technical novelty, and lack of in depth analysis of results
motivation of algorithmic choices : why did you choose the reward sketching approach that you chose? How is it different, and does it improve performance?

**Rebuttal**
The authors addressed most questions/things that were unclear and updated the paper to include an additional baseline.  

Additional baseline: First, it's great that you added this additional baseline! Yet, to me it's unclear what that additional baseline really represents (Request + sparse rewards). The original ReQuest paper also learns reward from human feedback, is that what you did for this paper? If yes then what does the sparse reward mean? Why does this version of request perform worse than your proposed version?

**Summary**
I agree with the reviewers and authors that this is a promising direction. However, in it's current form this manuscript is not ready yet for publication. My  concern are centered around motivation of algorithmic choices: The reward sketching part (while not novel in itself) is a novel component of the ReQuest pipeline, but you do not evaluate what it adds, and neither do you motivate that choice. Furthermore, the additional baseline is not clearly described, it's unclear how it's different for your proposed approach and why we see the performance improvement of your ReQuest version vs the baseline ReQuest version.
Decision: Reject