Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper studies the effect of representation learning and data augmentation on imitation learning.  A range of methods were considered on both the representation learning and the data augmentation side, including both pre-training and joint training.  The biggest take-away from the paper is that data augmentatoin seems to be the single most impactful thing one can do to improve behavioral cloning.  This is a reasonable finding.  
strengths: The setup of the benchmark is nice.  The wide range of design decisions studied is a good template for future work.  The setup enables finding conclusions such as the value of data augmentation.
weaknesses: I didn't seen any major weaknesses.  

Perhaps the biggest question is how the results generalize to other imitation learning approaches beyond behavioral cloning, but I don't count that as a reason for rejecting the paper.

Another issue is a lack more thorough discussion of how these findings compare with analogous studies in the reinforcement learning context.  It's a natural question to raise, and there isn't much commentary in the paper.

I think the paper leaves a lot of tantalizing questions unanswered, but that's as much the effect of clearly articulating the follow-up questions & speculations raised by the study, rather than anything else.  So I wouldn't hold that against the paper.

Minor stuff:

-- Typo on Line 242

-- Figure 2 is too small
correctness: The claims see correct.
clarity: The paper is well written.
relation to prior work: There isn't much direct prior work in this setting for imitation learning.  The closest area is reinforcement learning and there could be more discussion there.
additional feedback: None.
review_title: solid paper

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents a well-thought design (and well-documented implementation) of a modular benchmarking framework for a broad investigation into whether representation learning benefits imitation learning.

The framework and software is organized according to major axes of variation in representation learning and end-to-end algorithm design, including choice of representation learning targets, type of loss function, presence of augmentation, specifics of encoder and decoder networks, as well as pre-/joint training schemes.

A primary takeaway from the implemented study is a bit of a "negative result"---that as far as behavioral cloning is concerned (as evaluated across 10 selected tasks from 3 popular environments), representation learning does not---on average---appear to add value above and beyond what staple image-based augmentation methods can already accomplish.
strengths: 1. *Significance*. It is refreshing to see an honest "negative result" of this sort. In the recent trend towards "smart" representation learning in all domains, the claim is invariably that such auxiliary objectives can improve efficiency of learning higher-level objectives, with accompanying experiments picked to align with prior knowledge. However, especially in reinforcement learning and---more recently---imitation learning, it is clear that the heterogeneity of inputs can hardly guarantee that these methods work well across the board. This paper is significant in that it is one of few "large-scale" studies that attempt to measure how things actually work in a mixed, agnostic setting.

2. *Code Quality*. The structure if very well-designed, and the code is modular and reusable---something quite rare in esp. reinforcement learning and imitation learning research. Something like this conceivably could be the basis for a lot of future benchmarking work (it would almost appear irresponsible for new methods/papers not to at least run the comparison already implemented using this framework, with it appearing to be quite easy to employ).
weaknesses: 1. *Imitation Learning vs. Classification*: A major empirical point is that while representation learning can benefit image classification, the same cannot be said for image-based imitation learning. However, the only imitation baseline is behavioral cloning, which is essentially a naive supervised classification algorithm. Is this not correct? Of course, in a regular classification task the performance evaluation at test time is usually different from in imitation learning. But apart from this, why should we expect BC to be different from classification?

2. *Predicting Actions vs. Rewards*: It is stated, "One possible reason is that, to the extent that visual state mostly conveys progress through a trajectory, that state will contain more information about rewards (which are typically related to trajectory progress) than actions (which are often repeated at divergent points within a trajectory)." But a basic fact from the Bellman equation is that there is a 1-1 correspondence between the space of rewards and the space of Q-functions. So if something conveys information about rewards, it also conveys information about Q-functions, which gives everything you need to predict the action (hard-max, or soft-max of those Q-functions). So it is not clear that this hypothesis is convincing. Perhaps without further structure in the loss functions, it would be "difficult" for a black-box model to automatically infer Bellman-related relationships?

3. *Other Imitation Algorithms*: Why is BC the only method used?

4. *Confounding Variables*: It is stated, "RepL pretrained policy operating on Procgen can rely heavily on background information, but this is rarely the case in DMC". Why would this be the case, if the training set is sufficiently large (so that there should ideally be no distribution shifts in background information between train and test time)? If this is impossible to guarantee, perhaps we could use a simple form of invariant representation learning to solve this problem (i.e. to decompose the observed image into noise independent of actions, and causal factors that influence actions).
correctness: Broadly, the evaluation methods and experiment design seem appropriately designed.

- See above question on why BC is the only method used.

- Ideally, for this sort of "deep dive" there would be even more environments/games/tasks. But the code appears very extensible, so this is not a major gripe, just a suggestion for future development.
clarity: In general the paper is well-organized and well-written.

Typo:

- Line 265: "alignged" -> "aligned"
relation to prior work: To the best of my knowledge, the relationship to prior work is covered satisfactorily.
additional feedback: The implementation and design is great. My primary concern (and I would really appreciate some thought into this) relates to my first question. The authors propose some "hypotheses" for why things turned out the way they did (i.e. no consistent improvement using representation learning over image augmentation, for BC). However, I found the analysis and diagrams to be explanations of the result itself, but not an explanation of **why** such a result is obtained. Any further insights/clarity would be good.
review_title: Refreshing analysis and implementation, but intuition for results unclear

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This work aims to study the utility of using representation learning in the process of learning control behaviors using imitation learning. The contributions of this work are two-fold:
1. It identifies different elements of variation in a standard representation learning (RepL) approach used in adjacent ML fields like NLP, computer vision, etc., and designs a generic framework for coming up with customized RepL approaches by tuning the elements of variation, for imitation-based control. The authors have built a modular codebase to that extent and open-sourced it.

2. The paper compares sophisticated RepL approaches generated using the generic framework against simple well-tuned baselines that use data augmentation, and shows that across multiple tasks and environments, data augmentation often outperforms both representation learning based pre- or joint training. It further tries to analyze the learned representations and shows a lack of correlation among representations for similar actions, thus showing that using ground truth action as the learning signal in imitation doesn't really benefit from RepL. 
strengths: 1. Interesting idea: I found the idea of testing the usefulness of RepL in imitation learning pretty interesting. Although deep RL has found a lot of traction over the past few years and still offers a lot of scope for improvement and progress,  imitation learning is well-known to work much better and scale in real-world scenarios for control tasks. In my view, trying to improve policies learned through imitation by training good representations is an important line of work and deserves more attention in current and future research.

2. Generality of proposed framework: The codebase for designing customized RepL algorithms looks pretty generic and easily extendable. It could serve as a useful tool for designing more advanced imitation learning specific RepL algos. in the future. 

3. Somewhat comprehensive experimentation: The paper does a decent job at comparing different RepL-based imitation techniques across multiple tasks and environments of varying complexity. That helps the authors make observations about how simple data augmentation fares against more sophisticated RepL algorithms in such settings. It also helps the authors identify the kind of choices that one should generally make in designing their own RepL algorithm. 
weaknesses: Lack of experiments on more realistic datasets: At several points in the paper, the authors note why RepL could be expected to work not as well as in the case of more realistic vision and text data. They mention that the inherent simplicity of the data in the settings considered in the paper allows the models to solve the task without learning high-level semantic concepts. Such behavior reduces the ability of RepL to improve the task performance of the learned policies. If that's the case, it's important to extend the evaluation to more photo-realistic settings like embodied control, self-driving, etc., or even better, to robotic control in real-world environments.

P.S. -- I am aware that doing a comprehensive evaluation with an actual robot in real-world settings could be very hard during a short rebuttal period, but it would be good to see some results in more realistic simulators like CARLA, Thor, Habitat, etc.


Edit: rating updated 
correctness: The claims made in the paper and the supporting experiments/evaluation methods look correct to me.
clarity: The paper is written clearly.
relation to prior work: The work compares with prior work and tries to look at a direction that's not been previously looked at. However, some more evaluation would make the work even stronger. I have listed such possible evaluation under 'Weakness'.
additional feedback: Other questions/suggestions:
1. L52-54, 'We argue ... Fig. 1' -- the 'axes of variation' is not very clear; either change the statement or change the figure.

2. L149-150, 'We have implemented ... a RecurrentEncoder' -- as far as I understand, even momentum encoders could be single image based encoders. If that's the case, then this statement isn't entirely correct?

3. L219-220, 'Using our modular ... learning algorithms': any particular reason why you used just these 5? Also related to this, maybe report the missing experiments mentioned in L228-230 ('Note that we ... doing so') in the Supp.

4. L242, 'Note that mot' -- 'Note that most'?

5. Table 4 and 5: the table captions aren't self-explanatory. As a reader, it's not very pleasing to go back to the text to understand a table.
review_title: Interesting work in representation learning for imitation but could benefit from more experimentation

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors argued that most common representation learning algorithms for imitation learning vary over the choice of target selection, loss type, use of augmentation, neural network encoder/decoder type, and whether it is used for pretraining or joint training. From this framework  a code base was constructed that can be used for building Representation Learning algorithms. This was used to evaluate the utility of representation learning over 10 tasks from 3 benchmarks (MAGICAL, Procgen, DeepMind Control Suite) compared with Behavioral Cloning with and without data augmentation. Representation learning was able to outperform data BC with augmentation on the DMC benchmarks but had little impact on MAGICAL or Procgen.

strengths: A Modular framework from which varying implementations of representation learning can be constructed, allowing for effects of various design decisions to be analyzed.
An analysis of various representation learning algorithms compared to behavioral cloning with and without augmentation, that reinforce previous work that shows little additional improvement in using representation learning over augmentation.
weaknesses: All the benchmarks that were used relied on synthetic data. In Sections 4.2 and 4.3 the variation in the datasets and the importance of evaluating across different benchmarks is discussed and STL10 is mentioned. Including a benchmark that uses real world data would add value.

correctness: The evaluation methods and experiments were appropriate and performed correctly. 
clarity: The paper was clearly written and easy to follow.

relation to prior work: The paper provides an analysis of different representation learning approaches for imitation learning in image-based environments and the design choices that were made (target selection, loss type, use of augmentation, neural network encoder/decoder type, and whether it is used for pretraining or joint training).

additional feedback: I'm not sure if including the implementation sections were needed.
Figures 2 and 4 were a little hard to read and in Figure 4 showing the original scenes would help give more context.
In the conclusion the variation in datasets is discussed and, having figure 3 show samples from the DMC task would be helpful.
It would be useful to see the results of the preliminary experiments in the Appendix that showed momentum, projection heads, and compression did not show any advantage.
review_title:  A modular framework for the construction and evaluation of representation learning algorithms

Meta-review: This work provided a systematic investigation of the efficacy of representation learning for imitation learning. The experiment settings are well thought of and the benchmark is designed in a modular fashion with great extensibility for future work. The results have indicated that several recent advanced representation learning methods have led to marginal gains over the simple baseline of image augmentation when used with an imitation learning algorithm (behavioral cloning). While this is a bit of a negative result, such evaluations could potentially benefit the research community for validating the real merit of representation learning for policy learning. At the end of the discussion period, all four expert reviewers expressed positive impressions of this work. The AC agreed with the reviewers that this work could be further strengthened with 1) expanded discussions on the experimental results, 2) evaluations with other imitation learning algorithms, and 3) testing on more realistic decision-making domains (e.g., autonomous driving and robot manipulation). Nonetheless, these suggestions should not be considered as a ground for rejection. The AC believes that this kind of systematic empirical evaluation, albeit with negative results, should be encouraged in our community and thus recommends the acceptance of this paper.
Decision: Accept