Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose V-MAIL, a variational model-based imitation learning algorithm. The transition dynamics model is learned by optimizing a relevant variational lower bound under the assumption of partial observability, while discriminators for imitation learning are trained over the latent-action space. The empirical results show that V-MAIL outperforms baselines including SQIL and DAC (where the data-augmentation technique is applied for both baselines). The authors additionally check the generalization ability (transfer learning) for V-MAIL. 
main_review: Although the proposed idea outperforms their baselines, I think more baselines and related works should have been considered, which makes me vote for rejection of this work.
- The submission is not the first work considering a latent representation in imitation learning, e.g., Gangwani et al., "Learning Belief Representations for Imitation Learning in POMDPs". In Gangwani et al., the inequalities similar to Theorem 1 of this submission have been considered. 
- The submission is not the first work on image-based imitation learning, e.g., Pathak et al., "Zero-shot visual imitation", Torabi et al., "Imitation Learning from Video by Leveraging Proprioception", Liu et al., "Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation", etc. I couldn't write all here, but I could easily find out vision-based imitation learning algorithms that refer to GAIL paper in Google Scholar. (FYI, my keywords were "visual based learning", "pixel", "image" under the papers citing GAIL.) While most of them didn't consider model-based imitation, I think comparing V-MAIL with model-free vision-based imitation learning is necessary. (If not possible, authors need to state why some of those submissions cannot be compared with V-MAIL.)
- The contribution seems marginal. The proposed method combines well-known approaches in vision-based RL to improve the performance of imitation learning. 
limitations_and_societal_impact: No societal impact was concerend.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Motivated by several challenges that face the use of adversarial imitation learning (AIL) in practice, the authors propose V-MAIL—a model-based algorithm for visual AIL. V-MAIL addresses the challenges of sample inefficiency and difficult numerical optimization by learning and leveraging both a variational observation model and a variational forward dynamics model. The authors provide/prove interesting theoretical performance bounds with respect to the proposed formulation, and propose V-MAIL as a practical solution to that formulation.
main_review: STRENGTHS:

(S1) The proposed approach is novel and proven effective empirically in standard benchmark domains.
        
(S2) The authors have provided a relevant and interesting theoretical analysis that motivates the proposed algorithm.
    
WEAKNESSES:

(W1) The paper is missing discussion of important related literature in imitation from observation (IfO). While the authors should draw connection to the area as a whole, they also need to discuss the similarities/differences with BCO [Torabi et al., 2018] in particular, since it’s extremely similar. For example, BCO _also_ contains a model-learning phase followed by an imitation phase. To be clear, the proposed algorithm is unique in its own right, the authors just need to address the close relationship here.

POST-DISCUSSION COMMENTS:

Thanks to the authors for agreeing to revise the paper to make the connection to IfO more explicit. I'm happy to continue recommending acceptance.
limitations_and_societal_impact: I found the authors’ treatment of these topics in the paper to be sufficient.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors present a novel method for imitation learning from a limited set of expert demonstrations. The method is heavily based on the DREAMER model-based RL framework and extends this with a discriminator to match a learner policy to an expert rollout.
I find the paper overall well-written and a worthwhile contribution but there are some technicalities holding the paper back from being a top submission. I am overall still in favor of getting this work accepted since I believe most things can be addressed easily and I'm happy to change my rating if the authors can address some of my concerns.
main_review: **Originality:** 

The work combines a GAIL-like discriminator between expert and agent trajectory with a much more sample-efficient world model-style approach, for which the authors used DREAMER. There are many other trajectory embedding approaches to imitation learning, most of which are cited or even used as a baseline but one noteworthy omission is VIRL [1], which has a similar network structure but lower sample efficiency. I think it would be good to at least acknowledge that work in the related works section.

**Quality:** 

The method is explained in great detail but to a fault. In my personal opinion, the POMDP formulation is not necessary at all. While technically, visual control tasks in 3D should always be considered POMDPs, in most contemporary works, they aren't. And accordingly, I don't think you gained anything by including section 4.2 in the main body of the paper (rather than the appendix), and instead I would prefer more ablation studies. 
- Specifically, I'd be curious to see the influence of the number of expert samples on the training performance. 
- I also would like to see more than 3 seeds in the result plots since there is such an incredibly high variance. Please see this tutorial on picking a number of seeds that is statistically significant: https://arxiv.org/pdf/1806.08295.pdf
- I think there is some issue in the implementation of the baseline methods: Neither SQIL nor DAC are nearly as unstable in their original publications as they are depicted here, which leads me to believe that the author-added DrQ modification may have had negative impacts on stability. This needs to be addressed urgently.

**Clarity:** 

The paper is clearly written and the authors get huge bonus points from me for including working code with instructions for running it.

**Significance:** 

I think this work is a very meaningful extension of DREAMER and definitely a valuable new imitation learning method that does not need excessive environment interactions while learning. I didn't think about using model-based RL methods in this way, but I really like the idea.

**Nitpicks and questions:**
1. line 16 "better stability compared to prior work" - not at any statistical significance with only 3 seeds and not with the unexplained performance deterioration of the baselines
2. line 26 ...circumventing exploration... - it doesn't really circumvent exploration. If the expert demonstration only covers a tiny part of the state space, the imitation learning has no chance of generalizing outside of that.
3. Figure 1 right side is great and communicates all major points! The bold and dash lines on the left side are probably more confusing than anything. Maybe something like Fig 3a from the dreamer paper would be better.
4. About the zero-shot/few-shot: I think this is only valid when both tasks share mostly the same dynamics, so the forward rollout model is identical given a state and action. But if you're going from one task to the other, e.g. driving as fast as possible while avoiding obstacles on CarRacing vs. making a lot of sharp turns in CarRacing, then I don't think this holds up. This wasn't really explored in the transfer section and probably needs a sentence. In your examples, the environment dynamics of walker-walk are included in walker-run and the environment dynamics of the 4-pronged D'Claw clockwise are too similar to the other tasks.
5. line 237 "scale" - scale from what?
6. line 241 1000 GPU hours total or each or how are they distributed?
7. line 254 CarRacing-v0 is not terribly hard to solve and for example Ilya Kostrikov's PPO implementation (https://github.com/ikostrikov/pytorch-a2c-ppo-acktr-gail/tree/master/a2c_ppo_acktr) can get to a mean episodic reward of ~550 and 700 within 500k and 1M steps respectively. 
8. Figure 3: "consistently outperforms" - Not sure this is correct, looking at the top performance, that's not true for Cheetah Run, and looking at the early performance, that's not true for Claw, Car Racing, and Walker.
9. line 295 how does that work if you have 2 different action spaces - 3-pronged D'Claw has a smaller action space than 4-pronged D'Claw, right?
10. line 334 "Finally, the experiments suggest that this algorithm is efficient enough to be applied to real robots," - I don't know how this sentence is justified.


**References:**

- [1] Berseth, Glen, and Christopher J. Pal. "Visual Imitation Learning with Recurrent Siamese Networks." arXiv preprint arXiv:1901.07186 (2019).), https://arxiv.org/abs/1901.07186
limitations_and_societal_impact: I would like to commend the authors for being the only one in my 6 reviews that actually followed instructions and put the limitations into their own little subsection in the conclusion. 
One additional limitation that I can see is resilience to visual distractors. Since your state is based on image to image reconstruction, all pixels, including meaningless but moving background and other actors will be reconstructed and therefore have to be captured in the latent state. Just a single distractor agent that's doing something task-independent could throw this method off, no?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces Variational model-based adversarial imitation learning (V-MAIL). V-MAIL considers a model-based setup for adversarial imitation learning: a latent dynamic model is learned and used to generate on-policy trajectories for training the RL policy generally used in the GAIL setup. It manages to achieve state-of-the-art performance on multiple tasks from visual observations, including MuJoCo, classic car racing, and two other robot manipulation tasks.

main_review: Overall the paper is well-written and easy to follow. Combining the model-based learning with GAIL-like imitation learning has indeed given extra benefit: it works like a regulizer for the policy, and provides nice on-policy rollouts for the policy learning. The experiment results also look quite promising. I think the interesting part of the paper is that it combines several interesting ideas and manages to prove that they can significantly improve the performance of imitation learning in the challenging scenarios. 

However, I still have certain concerns about the paper, which make me feel that the paper is not ready for publication yet. Please find the detailed comments below.

limitations_and_societal_impact: I have certain concerns about the paper.

1/ I think the contribution of the paper is a bit limited. V-MAIL combines several existing ideas, e.g., latent imagination, GAIL, variational state space model, etc., and achieves a good performance. However, how these components affect each other and how they contribute to the final performance are not clear. An ablation study is also missing from the paper. In this case, it would be hard to get inspiration from reading it.

2/ Although the paper has provided theoretical analysis about the model-based adversarial imitation learning (sect. 4.1 and sect. 4.2), they are disconnected from the practical implementations (sect. 4.3). In particular, Theorem 1 shows that the divergence of the visitation distributions in a MDP can be upper bounded by a divergence of the visitation distributions in a POMDP. However, in the practical implementation, a variational state space model captures the belief, rather than the visitation distribution of the belief. In addition, it feels difficult to compute the visitation frequency of a belief, whose size is exponential to the size of the history and the state space. I believe the proposed algorithm indeed has its merit, but I don’t think Theorem 1 provides a correct justification of the optimization objective used in this paper.

3/ I feel the author should be careful when making certain claims. For example, from line 39 to line 48, the authors are analyzing the limitations of the existing IRL methods and adversarial imitation learning methods. “These approaches explicitly train a GAN-based classifier [17] to distinguish the visitation distribution of the agent from the expert, and use it as a reward signal for training the agent with RL….” However, not all IRL methods are adversarial imitation learning. In fact, most of them don’t train a GAN-based classifier and do RL afterwards. Instead, a lot of them recover the reward and do planning instead.

4/ The authors claimed that V-MAIL achieves zero-shot transfer to novel environments. However, the policy is fine-tuned with additional expert demonstrations, as shown in Alg. 2. Why is this zero-shot? In addition, I guess the transferability might be limited by the real difficulty of the source task / target task. Walker-run is clearly harder than walker-walk, so the policy transfer here is possible. Also, for the manipulation scenario, 3-prong task with both clockwise / counter clockwise rotations, together with one 4-prong task, actually provides sufficient information about the target task. I guess it might be difficult to transfer a policy from simpler tasks to more complex tasks. This has to be made clear in the paper. Otherwise, it is quite misleading.
review_title: null

Meta-review: This paper presents V-MAIL, a model-based algorithm for visual Adversarial Imitation Learning. V-MAIL learns both a variational observation model and a variational forward dynamics model. The algorithm is motivated by theoretical performance bounds, proposing the algorithm as a practical solution to the formulation. All the reviewers evaluated positive about the paper, and voted for acceptance. 
Decision: Accept (Poster)