Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors propose a novel imitation learning method, regularized by object-aware encoding and masking during the training phase. The improvements over the performance of behavioral cloning baselines on Atari games indicate the effectiveness of the proposed framework.
main_review: - The proposed method is easy to follow. The overall writing looks well-organized to me. Both big picture and implementation details are discussed intensively. I believe the readers could catch up with the most critical observations in an easy way. 
- The illustrations are helpful for the readers to realize the main motivations and contributions of the proposed method. One possible suggestion is to highlight the masked area in figure 1. It takes me a long time to realize where the masked area is.
limitations_and_societal_impact: The proposed OREO method heavily utilizes VQ-VAE to disentangle the raw RGB input into discrete codes for each semantic object. So the accuracy of such a grouping could be crucial. Even though VQ-VAE works well on Atari games, the real-world application, such as autonomous driving simulation CARLA, typically has much more complicated scenes with higher resolution. I wonder if the proposed OREO could still help when VQ-VAE generated discrete codebook becomes fuzzier. Also, as mentioned in section 5, the application of learning from a sequence of observations could be interesting to study.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper considers a recently identified problem in imitation learning, that the trained policy learns relationships to nuisance variables, which do not have a casual relationship to the main goal of the action. 

For instance, when learning pong, the learned policy becomes dependent on the score display and the learned policy is difficult to transfer to an environment where the score display does not exist. On the other hand, a policy learned on an environment where the score display was masked out transfers better to an environment where it is present. 

The authors are proposing a technique to regularize the imitation learning process by masking out certain objects from the environment. They achieve this by training a VQ-VAE, and assuming that units of a feature map corresponding to the same objects are mapped to similar discrete codes. The authors then randomly drop the units of a feature map that share the same discrete code during training. 

The authors test the proposed approach on a number of games from the Atari environment using the "confounded input model", where the action from the previous time step is encoded into the current input. Previous work had shown that such a confounded input, although it contains more information than the original form of the input, is in practice harder to learn. 
main_review: The paper describes a regularization approach from the general class of approaches initiated by dropout, for the specific problem of nuisance variables in imitation learning. The approach is based on randomly dropping certain objects from the environment, with the objects being identified by the fact that they map to similar discrete codes when encoded by a VQ-VAE. 

The paper shows improvement in the performance of the Atari games considered. 

There is a certain degree of concern about the significance of the proposed approach. The approach does not strictly speaking identify causal relationships. It seems that the primary impact is that it prevents the policy to focus too much on one particular variable. It is also not necessarily clear why training on the confounded input changes the learned policy, and why the proposed regularization improves it (eg Figure 8). While I am aware that this technique of confusing the input was not proposed in this paper, its practical significance is unclear. Would this problem appear in a real world setting? Would the proposed approach work with real world images?
limitations_and_societal_impact: The paper correctly identifies that the proposed work only focuses on the nuisance features on the current state, and does not actually infer the causal relationships. 

The authors also discuss some hypothetical negative impacts about self-driving cars and imitation learning, however these are hypothetical and not specific to the contributions of this paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the causal confusion problem in imitation learning (IL) is considered. To tackle the confusion that resulted from not knowing the causal relationship, a two-stage approach is proposed. At the first stage, a VQ-VAE model is trained to generate semantic codes for different objects in the state space. At the second stage, a dropout-based approach is adopted to decouple the semantic confusion among the objects. The contributions of the paper are:
>- Propose a practical approach to solve the causal confusion problem for IL;
>- Conduct extensive experiments under Atari to verify the performance of the proposed approach.
main_review: Originally: Good

Addressing the causal confusion problem for IL receives significant attention in recent years. The paper proposes a novel and practical approach to deal with this problem. 

Quality: Overall good

The paper proposes an interesting approach, which is quite practical in my view. The experiments show significant improvement over existing baselines. While I still think the following limitations exist:
>- The proposed approach seems to work only in vision-based tasks. 
>- The effectiveness of the proposed approach seems to rely heavily on the performance of the VQ-VAE model. I am curious to see how the proposed approach is robust to the quality of the semantic codes learned by VQ-VAE.
>- The experiments include only Atari tasks.
>- Only behavior cloning is discussed. While I think applying the approach to other IL methods is also possible. I am curious to see whether this is true.

Clarity: The paper is overall well-written. Both the technical and the experimental parts are clear to be understood.

Significance: To my understanding, the paper provides an interesting approach to address the causal confusion problem in IL, which may be of interest to researchers in this area. I suggest releasing the code to make the results easier to reproduce by other researchers.



limitations_and_societal_impact: They are adequately addressed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces a policy regularization method that randomly masks objects which share discrete codes inferred from VQ-VAE, to prevent learners from focusing on non-essential parts of expert actions. OREO, the suggested model, is trained without environment interaction but learns how to evenly attend to all objects in confounded Atari games. Consequently, OREO aims to solve the causal confusion problem which is an innate weakness of behavior cloning from observation.
main_review: The paper is well written, but it has limited novelty. 
1. (originality) OREO can be considered as BC with VQ-VAE, with no other contributions.
It has a better performance compared to CCIL, but the poor performance of CCIL seems due to insufficiency of beta-VAE in a high-dimensional environment, and the better results of OREO are likely to result from the ability of VQ-VAE that infers well discrete latent representations in a high-dimensional environment.
2. (significance) Also, if the authors wanted to claim the suggestion of the dropping technique as a contribution, I think the paper should have included the baseline like DropOut and DropBlock within VQ-VAE architecture, rather than pure convolutional encoder feature maps. To elaborate, the authors suggest the dropping of feature map units that share the same discrete codes, but I think dropping feature map units of random discrete codes or random contiguous discrete codes like in DropOut and DropBlock would be better baselines for the dropping technique.

#### Minor Questions
1. (clarity/ quality) Is there any reason you choose DRIL and GAIL as IRL baselines? GAIL has already been mentioned in CCIL as “GAIL fails to converge to above-chance performance on any of the (confounded) Atari environments.” I wonder what insights that the evaluation result of DRIL and GAIL each can give to the readers.
2. (clarity) Please explain more about Figure 3. For people who don’t have prior knowledge of Atari games, it is hard to know the intention and meaning of even column images.  

limitations_and_societal_impact: Yes, the authors pointed out agreeable limitations and potential negative societal impacts. 
However, it would be interesting to give the difference with CCIL in terms of constructing a causal model(graph), because CCIL proposed the way to deduce the cause-effect relations as a causal graph while OREO did not. I think the dropping method does not lessen the need to know cause-effect relation in causal confusion problems, rather the method can provide a better way to utilize the inferred cause-effect relation.
review_title: null

Meta-review: This submission targets the problem of "causal confusion" in imitation learning, where spurious correlates in expert data hurt environmental performance. The approach proposed is simple: train a vector-quantized VAE representation, and apply dropout over it during imitation learning. The motivation is: VQ-VAEs can learn to disentangle objects in images, and this can help coherently omit semantic information. 

The key strengths of this submission are the simplicity of the proposed approach, the breadth of experimental evaluation, and the strong empirical results. In particular, the evaluation uses 27 Atari environments, aside from new rebuttal experiments in simple CARLA settings. These may become valuable contributions to studying this problem. 

This submission is very much borderline, and I only lightly lean towards acceptance. My main reasons for hesitation are:
- Like reviewers have also commented, the paper relies on learning not just good quality discrete codes, but disentangled *objects* through VQ-VAEs (or at least, it is motivated in this way). While this works well enough on Atari settings such as in the submission, I'm not convinced that it will in more complex settings. 
- The CARLA settings tested in new rebuttal experiments ("straight", and "one turn") are simple, and not the standard town driving environments commonly used in the imitation learning literature. 
- Dropout has previously been proposed as an approach to learn robust imitation policies that can tackle these problems: see [1]. This means that the proposed approach is effectively a combination of VQ-VAE with an existing technique 
- Aside from new somewhat preliminary rebuttal experiments on 2 Atari environments in the "stacked inputs" setting, all experiments are performed in the more contrived setting in which past actions are overlaid on the input images. Other works in this literature have now moved away from this assumption, such as [1, 2].

[1] "Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst."
[2] "Fighting Copycat Agents in Behavioral Cloning from Observation Histories"
Decision: Accept (Poster)