Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes EXPAND - a method for allowing human input and semantic understanding to guide policy learning for RL agents. Humans are asked to provide binary feedback (good/bad) on actions taken by the agent, as well as highlight visual features which helped guide their decision (e.g. stop sign tells human to stop). The binary feedback is used to train an advantage function by leveraging a large margin classification loss. The visual highlighting is used to supervise feature learning by way of a consistency loss on perturbed data (i.e. blur segments not highlighted by human).
main_review: It seems the primary difference between EXPAND and previous work DQN-TAMER [1] is that human input is integrated into an Advantage Function (which is also tuned w/ RL), rather than linearly blended with a separate Q function. This change does seem novel and results in a more elegant method with less hyper-parameters. The experimental results show EXPAND significantly outperforms DQN-TANNER in all environments. However, I do wish the authors more precisely described DQN-TAMER, and made these points clearer in the paper.

It's important to note that most experiments in this paper used an oracle to simulate human input likely to make the study less time consuming. The authors do one experiment using human input on the Enduro-1000 environment, but its difficult to tell if the performance matches the oracle Enduro-1000 runs due to separate y-axis scales. Furthermore, DQN-TAMER is only run with oracle input and not human. I hope the authors fix these bugs, so it becomes more clear if EXPAND can compete with DQN-TAMER when using human input.

Finally, the ablations do a good job justifying visual feature annotation helps with downstream RL task performance. I'm not convinced by the author's claim that Gaussian blurring somehow uniquely doesn't change state content. After all, what if high frequency information is important in the state? However, I do agree that it seems to work fine for the chosen environments.

[1] Arakawa, Riku, et al. "Dqn-tamer: Human-in-the-loop reinforcement learning with intractable feedback." arXiv preprint arXiv:1810.11748 (2018).
limitations_and_societal_impact: I would ask the authors to more thoroughly discuss limitations of their method. In particular, if performance is very different between oracle and human input then that is important to discuss. Furthermore, the human input seems far more time consuming to produce. For example, the Enduro-1000 human experiment is only run to 60K steps v.s. 100K+ for oracle. The authors claim this takes total of 30 minutes, but if that's the case why not run for as many steps as the oracle? 

Regarding social impacts, the authors claim EXPAND would help humans align the agent's behavior with our own value system. However, this paper does not show experiments where the RL reward is necessarily at odd with human values. Thus, this claim does not seem substantiated, and I would recommend the authors re-write the social impacts section accordingly.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a way of data augmentation based on human feedback. Namely a human experts is tasked with marking on relevant areas of the state space (visual input, frames) with bounding boxes. The rest of the input is then blurred, which, presumably, should lead to the learning algorithms "focusing" on relevant areas and learning to ignore irrelevant (blurred) ones. Experiments on 5 simulated environment show that applying this technique leads to somewhat lower sample complexity and better final performance.

In my estimation this is an interesting idea on using human-in-the-loop for RL training and would be useful for RL practitioners working on such visual environment where data samples are expensive. Real-world robotics would be one such example.
main_review: Overall I do not have any specific questions or criticism or this work. The idea is valuable, the description is clear and the execution is valid and allows me to trust the results reported in the paper.

Originality: The idea to use humans to annotate the important regions of the input sample is quite original and in my estimation is well executed in this work, allowing the community to evaluate the benefits of the proposed trick. In general, human in the loop framework offers, in my opinion, a multitude of ways of enhancing the RL process and this work explores an interesting aspect of this area of study.

Quality: The claims are well-formulated and are supported by the empirical results provided. 

Clarity: The proposed method is presented clearly and with a good level of technical detail allowing to understand the method down to the low-level technical details and mathematical formulation. Experiments and ablations are well-justified and address the right questions.

Significance: RL is known to be notoriously data hungry which is the main obstacle for its mass adoption in real physical systems where data samples are expensive. The proposed method add a potentially useful tool to the toolbox of RL practitioner working on application where sample complexity is crucially important. I find the contribution of this work to be significant enough to deserve being seen by the RL community. Also, while the result is presented in the context of human-in-the-loop, the Oracle experiments (Section 5.1) demonstrate the benefit of the proposed method also in a fully automated training process. As long as the environment allows to create a useful oracle, this method can be used to speed up learning as an additional module to any vision-based RL process.

limitations_and_societal_impact: The limitations of the method are adequately addressed in the text of the manuscript.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: They propose methods to better learn (off-policy) Q functions using binary human feedback, and also using visual explanations from humans.
main_review: (disclosure:  I had briefly seen a previous version of this as a poster under a different name.  I do not know the authors and don't think it affects my review)

I like the direction of this work.  It seems good generally for humans to try to indicate subsets of observations that most affect actions as a form of feedback.  Overall, I am somewhat convinced it is a promising method, although I think the presentation of the results could be a lot better.  I would've liked to see experiments in a harder environment where I think the gap between EXPAND and DQN-feedback could've been even larger.  Also would've been nice to see at least a picture of each environment and the oracle behavior, given the nature of the work.

Figure 2: I kinda feel looking at this picture that cutout would be more natural.  would that have been worth trying?

Baselines:
- DQN-TAMER is never really explained.  It would be nice to explain the difference between it and DQN-feedback, so the reader doesn't need to look at other papers and think about it
- Not super important, but could context-aware data augmentation be applied to DQN-tamer as another baseline?
- another baseline to get an understanding of how much augmentation helps, is to run DQN-feedback, but training on an extra frames for each would've-been-augmented frame.  presumably this is a strong baseline, and it would be more impressive (but plausible) to beat it.  if I understand correctly, this would also control better for amount of compute used

Improvements on Environment and Feedback Sample Efficiency
- it would be good to know what percentage of frames get augmented.  

Comparison to explanatory interactive learning methods:  this section overall felt unconvincing to me
- Attention-Align seems a little complex and the results seem very poor, making me wonder if there were implementation issues or if it was otherwise a poor choice of baseline for some unknown reason.  I'm guessing there's a hyperparameter to tune to balance the attention prediction objective, for example?
- The comparison of EXPAND with Ex-AGIL seems like it could easily come down to number of oracle/human samples + efficiency of learning the mask, since in one case we have an additional learned model.  If the model is good, then it seems like you get to do a lot more augmentation, right?
- why were only two environments used in figure 5?  also line 374 says "all three tasks" when there are just two
- might be nice for Figure 5 to have a no-aug baseline

EXPAND versus Context-Agnostic Data Augmentations
- why not use the augmentations on all states, if there is an oracle?
- it would be good to know what percentage of frames get augmented.  it would be interesting to see, e.g. 100% gaussian blur vs. gaussian blur except when there's human data
- overall comparisons here look solid, although again I wonder why there are only 2 environments

5.2 
- the human trainer seems to do worse than the synthetic oracle.  is this just because the oracle is quite good?  it might be interesting to find a harder task where it's not as easy to write a synthetic oracle - I think results would be more compelling in such a setting.
- why was DQN-tamer not included in this?

5.3 
- i assume these were done with the oracle again.  it would be good to explicitly state this
- again, I wonder why the other environments weren't included here
limitations_and_societal_impact: not much here
review_title: null

Meta-review: The paper presents a method for incorporating richer feedback than a reward signal from humans for training RL agents. Humans provide feedback by (i) indicating which actions are good/bad and (ii) highlighting salient areas in visual observations. All reviewers unanimously agree that the paper should be accepted. The experiments provided by the authors during the rebuttal phase further strengthen the paper. At the same time, I recommend the authors to improve the clarity of the paper, explanation of prior work (e.g., DQN-TAMER) and address other minor concerns raised by the reviewers in the camera-ready version.

If I were to simply go by the technical contributions of the paper, I would recommend it as a Poster. However, the line of research on providing human feedback beyond reward functions is worth highlighting to the research community. I therefore recommend the paper to be presented as a spotlight presentation. 


Decision: Accept (Spotlight)