Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper considers using offline datasets for improving representation learning for subsequent imitation learning. 
main_review: The paper is in general well written and the theoretical contributions are nicely explained with important differences from the similar field of bisimulation metric based methods. The authors also validate the approach on toy environment and Atari domain, although I would suggest the authors to also report percentile performance improvements for individual Atari games which seem to be missing in the report. Overall the paper is a decent contribution to the field of imitation learning.
limitations_and_societal_impact: I think the authors need to better address the limitation of the work so that its more informative for practical scenarios, this could for instance include some paragraphs about the specific shortcomings of the method complemented with the theoretical results.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors study the impact of representation learning when applied to behavioral cloning (BC), and present a series of interesting theoretical results that establish its utility in this context. In particular, the authors show that there is a direct relationship between the quality of the state representation and the performance of the policy derived using BC over that representation. The authors then present a practical representation learning algorithm designed using the insight gained from their theoretical results and show that using it in combination with BC results in an improvement over comparison approaches.
main_review: STRENGTHS:

(S1) The overall thrust of the work, i.e., the studying the connection between representation learning and imitation learning, is excellent and extremely important to the community.

(S2) The paper contains interesting and seemingly well-founded theoretical insights that have practical implications.

(S3) The authors have proposed a practical algorithm based on their analysis that appears to achieve good empirical results.
    
WEAKNESSES:

(W1) My major concern with the paper is that it seems to ignore a major portion of the literature that has provided the field with an understanding of behavioral cloning (BC) for the better part of a decade, i.e., the work of Stephane Ross and Drew Bagnell as far back as 2010 (e..g, [Ross and Bagnell, AISTATS 2010]). In the submission under review here, the authors seem to identify the major problem with BC to be that of aliasing, whereas in the referenced work above—and throughout the community, to the best of my own knowledge—the major issue is commonly understood to be the quadratic growth in error with respect to the task duration. While both issues seem to be intrinsically linked to a lack of demonstration data, it’s not clear the extent to which the ignorance of this work impacts the presented technique. Is it simply a hole in the related work which the authors can easily address, or is it a more fundamental issue? For example, in the error bounds derived by the authors, I do not see any hint of an error term that grows quadratically with the duration of the task—where have these terms gone? I am very curious to read the authors’ response.


POST-DISCUSSION COMMENTS:

Thanks to the authors for addressing (W1) above. As mentioned in the discussion, I’d like to see some text added to the next version of the paper acknowledging this connection and explicitly drawing the distinction between distribution shift and aliasing.
limitations_and_societal_impact: I found the authors’ treatment of these topics in the paper to be sufficient.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper tackles the problem of learning representations for imitation learning, in the context of offline data (with demonstrations). Firstly, given some low dimensional paramterization of the state, the authors provide a performance bound on the learnt policy over the representation matching the ground truth policy, regardless of the structure of the ground truth, as well as bound on the sample efficiency. The authors show that it is possible to learn the model and representations using a contrastive learning objective, and the final representation is output by applying random fourier features to the features learnt via contrastive learning. The authors provide some toy experiments in tabular decision tree settings to verify this hypothesis as well as on offline Atari dataset. 
main_review: Strengths: This paper tackles representation learning in offline data and 1) presents a thorough proof of the optimal representations needed for imitation learning, 2) the contrastive objective based representation learning insights seem very useful as well. The bounds derived seem quite thorough and the experiments provided are informative. The paper is overall clear and well written. 


Weaknesses: While the paper provides potentially useful insights, the fact that most cases consider linear dynamics makes it hard to be applicable to realistic settings. It would be good for the authors to discuss which aspects/assumptions make it difficult to scale to the real world, and potential down-stream uses of the learnt representations/policies, especially since the authors provide experiments on a non-standard Atari setting. It would also be good if the authors provided an exact training procedure used to learn the representation from the contrastive objective.

Clarifications: 

What term in theorem 3 shows the dependency on the quality of the representation? 

Why is theorem 4 only in the tabular case? How does learning happen in tabular cases? 


limitations_and_societal_impact: the authors do not address limitations of their work in detail and do not have a section on potential negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes an approach for learning a low-dimensional state representation that favors the training procedure of a policy with imitation learning. One of the main challenges is that the representation learning strategy is run on offline data, and is possibly not informed about the expert policy to imitate. The paper presents a theoretical framework to bound the performance of the student policy in function of the expressiveness of the representation and the distance to the distance between expert and student on low-dimensional states. The theoretical findings are then put into practice by deriving an algorithm to learn representations via contrastive learning and Fourier features. The algorithm is then evaluated on both toy tabular examples and on more complex high-dimensional tasks (Atari).
main_review: The main findings of this work are, to the best of my knowledge, novel. The paper does a very good job of covering the (large) field of representation learning for control. The most important related work is adequately cited, and some interesting connections are drawn between prior empirical evidence and the proposed method. However, it would have been nice to also mention and discuss existing work on learning representation by maximizing information for the task [1].

I find the paper presentation very good, well organized, and easy to follow (except for some minor comments that I report below). However, I am a bit concerned that very few technical details are given for the key experiments of the paper, i.e., the one on Atari games. Indeed, only there learning a representation for the states (i.e., images) actually makes sense (the other demonstrator is just a toy tabular MDP). No mention is made of the architecture, the learning rate, the training time, etc., which makes the experiment basically not reproducible. I have checked in the supplementary code, but I only found code related to the toy tabular MDP and not of the Atari games.

Regarding the technical quality of the submission, I think that the derivations are sound and supported by empirical evidence. On the negative side, I believe this work's biggest weakness is the lack of a clear discussion about the limitations of the proposed method. More specifically, Eq. 11, which presents the central finding of the paper, shows how the student performance is bound in terms of the representation power and the distance expert-student in representation space. However, it is completely unclear what are, in the general settings, the dominant terms of this bound. My concern is that the state observations probability is pretty different between an expert and offline demonstrations, giving a large  $D_{\chi^2}(d^{\pi} || d^{off})$. Such component will probably also be amplified by the loss of representations $\epsilon_{R, T}$ (which I would doubt to be below one). This probably becomes even worse as the representation and policies are forced to be log-linear. As a result, I feel that the bound is completely dominated by its first term and de facto loose for most cases. Note that such a remark does not diminish the derived practical algorithm, which is interesting and with good performance. However, I think that the paper should be more open about this potential issue and discuss how to address it. On a similar note, it would be nice to clarify the effect of realizability of the expert policy on the low-level representation z (the so-called $const(\pi_*, \phi)$, which could actually also pretty large if the demonstrations and expert policy are very different. 

Other minor comments to improve the presentation and strengthen the contribution:

1. Include stronger baselines for representation learning (e.g. [39]  or several other methods mentioned in the related work).
2. After each lemma/theorem report in which part of the appendix the proof can be found.
3. Section 4.3 is titled sample efficiency, but it has nothing to do with the number of samples required to training the policy. It only specializes the bound of Eq.11 to a tabular setting, as far as I understood.
4. It would be nice to clarify that the low-level representation needs to encode enough information to solve the downstream task. Otherwise, some trivial solutions could be found (e.g., putting everything to zero).

Overall, I think this is a nice and interesting paper. The findings could be useful to the control learning community and open new ways to learn policies from high-level observations in a simple and (possibly) provable fashion. However, the contribution could be strongly enhanced by openly discussing the limitations of the theory and algorithm, ensuring reproducibility, and empirically comparing with stronger (or simply more related) baselines. 

[1] A Separation Principle for Control in the Age of Deep Learning, Achille, et al.
limitations_and_societal_impact: No remark at all was done about the possible negative societal impacts of this work. I think this is acceptable since the contribution is mainly theoretical, but I would have appreciated it if the paper would have at least mentioned it (for example, can such work be used for military applications?).
review_title: null

Meta-review: After discussion, all reviewers believe that the rebuttal has addressed their concerns and agree on an acceptance. 
Decision: Accept (Poster)