Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper approaches the problem of imitation gap. In situations where the teacher (an expert) has access to privileged information, the student may not have such comfort. Therefore, blindly imitating the teacher may lead to inefficient, unwanted or incorrect behavior of the student. The paper provides examples of such situations. Actually, sometimes the student may not even be able to imitate the teacher in some states as not all the information is available. 

This leads to an imitation gap. Performing classic imitation learning in states where the imitation gap is significant can be pointless. The authors propose to try existing RL methods in such situations.

Specifically, the imitation loss and the RL loss are weighted by a normalized coefficient. One of the main contributions of the paper is how to dynamically determine the value of the coefficient. The idea is to in some sense 'measure' or 'quantify' the imitation gap in a given state. The authors follow the intuition that for states where the imitation gap is substantial, learning well the teacher's distribution over actions may not be achievable. It means the divergence between the teacher's distribution and the imitation distribution of actions will be relatively large. This quantity is used as an estimation of imitation gap.

To compute that quantity an approximation to the imitation policy is introduced. The divergence between the teacher and this approximation is used to estimate the imitation gap.

main_review: The paper is very clear to read and understand. The flow is logical and smooth. The problem is well stated and supported by a clear example. 

The approach is precisely described. The notations is consistent and easy to comprehend. 

The method is practical in the sense that it takes under consideration that computing the exact imitation policy is not possible. Instead, it offers a simple and efficient way of approximating it. On the other hand it is flexible as not-critical elements of the algorithm are abstracted away. For example, there are multiple possible distance functions between the teacher policy and the imitation policy (or its approximation).

The paper introduces a novel approach to estimating the coefficient between imitation learning and reinforcement learning for states that have a significant imitation gap.

The results are on the example problems and on more challenging environments. The proposed approach performs well in the basic and more complex circumstances.

My rating is based on the proposed approach but also on the significance of the problem itself.

Of course it would be great to see the method evaluated on even more complex environments and challenges. Nevertheless, that probably will be the topic of future work. At this point the presented experiments illustrate the problem and the presented approach is able to make significant progress. 

One comment is that it would be beneficial to include more analysis of the intuitions behind the imitation gap and the suggested way of quantifying it. For example, where does the quantification method perform well and what are the cases where it is imperfect? Why? What should be considered to improve over the proposed algorithm? What are the edge cases?





   
 
limitations_and_societal_impact: The paper has a concrete chapter addressing limitations and social impact. It is clear that authors had these in their mind and addressed potential concerns.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This work introduces ADVISOR, a simple yet effective approach to adaptively combining RL with supervision from imitation learning. The approach is motivated by the issue of *information gaps,* which describes the case where the policy generating the supervision has more information about the state than does the policy being trained. The authors motivate their approach as a way to address this issue and contribute a thorough set of empirical analyses to demonstrate its effectiveness. In addition, this work provides a concrete demonstration that the information gap is an obstacle.
main_review: I consider this a strong paper. It clearly motivates the problem as well as the solution it proposes (which itself is attractively simple), making use of concrete demonstration throughout. It is one of the rare papers that seems to get *better* as you read it.

I lack the specific expertise to fully evaluate the originality of the work, but, as best I can tell, it should be considered novel. At the very least, the combination of contributions makes the aggregate work in the paper original.

The ideas in the paper are presented quite clearly. The authors use examples well and provide clear formalizations where useful. The paper does not rely on the appendix, but it contains plenty of clarifying information nevertheless. The main results figure (Fig 5) is a bit hard to parse.

A major strength of the paper is its experimental rigor and use of illustrative "toy" settings. The adherence to best practices makes the results more meaningful and impactful. The analysis presented in Figure 6 is particularly nice and helps to establish the significance of the information gap.

One issue with the paper is that it seems to be particularly focused on the settings where it would be possible to query the teacher policy at every state. Often, I would think, that is not the case. There is at least one experiment using ADVISOR with a fixed set of teacher demonstrations, so the method can handle this case. My sense is that it would strengthen the potential impact of the paper if deeper analysis were presented of how ADVISOR fares in these demonstration settings.
limitations_and_societal_impact: Yes, adequately addressed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper points out the existence of "imitation gap" when a teacher and student policy has different observations. Due to this imitation gap, the student policy cannot determine which action leads to a higher reward, and thus ends up outputting a uniformly random action or average actions of the teacher policy.

The proposed method, ADVISOR, tackles such imitation gap by balancing between learning from reward and expert. When the limited observation is enough to predict the optimal action, the student policy imitates the demonstrations; otherwise, it learns from reward. As the weight function, ADVISOR uses the divergence between the teacher policy and the auxiliary policy, which solely learns from the demonstrations.
main_review: **Originality**: This paper identifies a new concept of "imitation gap", which can easily happen and degrade imitation learning performance, and proposes a simple method to get around the imitation gap.

**Clarity**: The paper is clearly written with intuitive examples.

**Quality**: The exhaustive experiments on diverse environments demonstrate the existence of the imitation gap and ADVISOR's superior performance under the imitation gap. 

**Significance**: Overall, the paper proves the importance of dealing with "imitation gap" and the proposed method effectively learns toy tasks under the imitation gap. This tells us that when we design an expert policy or collect expert demonstrations, we must be aware of the effect of such imitation gaps and try to minimize the gaps. However, it is not very clear how important this imitation gap is in complex environments, such as PointGoal and ObjectGoal navigation tasks, as the performance gap is not significant.


Here are a few questions regarding the method and experiments.

- The only imitation gap considered in this paper is about the observational difference between the teacher and student. However, the gap between the teacher and student could be often aroused from different dynamics, embodiments, and environment stochasticity. Can this proposed method handle such gaps as well, especially under stochastic environments?
- The experiments are mostly focused on navigation. A few experiments on common continuous control benchmarks (locomotion or manipulation) could make the paper stronger.
- The key of ADVISOR is to ignore the expert policy on states which are not reproducible in the student's observation space. This highly relies on the accurate auxiliary policy. To learn a more accurate auxiliary policy, use of a replay buffer can be more appropriate. What is the rationale of using on-policy data, which can make the auxiliary policy biased toward recent trajectories?
- The weight function used in ADVISOR is somewhat complicated and requires two additional hyperparameters to be examined. Is there any benefit or explanation of the use of this complex function compared to simple linear, quadratic, or exponential ones?

### Minor comments and questions

- In some places, the white spaces between lines and figures are too compact to read comfortably.
- In Table 2, "e" is added to the second column.
limitations_and_societal_impact: Adequately addressed.
review_title: null

Meta-review: There was a strong consensus among reviewers that this paper should be accepted. The paper proposes a method to address the imitation gap - the setting where the expert has access to privileged information making imitation impossible for the student. The paper shows both the efficacy of the method and the importance of the imitation gap for performance. They only point of improvement would be to look at more diverse environment. Currently the paper is focused on visual navigation.
Decision: Accept (Poster)