Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors study the problem of generalizing out of distribution for RL.  They conduct an extensive empirical study, using simulation and real robotics, and observed properties that predict how well an agent will generalize.

main_review: Primary confusion/clarity concern: I had trouble understanding the rank-correlation results.  I think the clarity issues begin at the following quote in Section 2: “As factor prediction mode...as GS-OOD1, GS-OOD2-sim, and GS-OOD2-real accordingly.”  After reading these 4 sentences many times, I still think they are very difficult to understand (primarily on my first and second readthrough, but even now that I think I mostly understand the approach, I still think these sentences are confusing).  Similarly, sentences like “A simple supervised metric to evaluate a representation is how well a small downstream model can predict the ground-truth factors of variation” and “we use the MLP10000 and GBT10000 metrics (simply MLP and GBT in the following), where MLPs and GBTs are trained to predict the FoVs from 10,000 samples” and “rank correlations between representation metrics and training reward” need to be explained better.  (If space is a factor, then pointing to an appendix for more detailed/intuitive explanations is a good strategy.)

Minor problem: “Correlations with p<0.05 are color-coded.”  How are they color-coded?  This is probably obvious to a reader without any of the primary confusion above (I think it’s red, but even then, which shade of red?), but making it explicit would be better.

Minor grammar edit: “As factor prediction model, we will likewise...” missing article adjective?

Strengths:
- Outstanding limitations discussion; this frank discussion of limitations strengthens the paper.
- Thorough overall results with some fascinating insights.
- The sim-to-real experiments were very nice.

Questions:
- MLP stands for multi-layer perceptron, correct?  This should be defined.  GLB should be defined.
- What are the differences between MLP vs MLP(1-7)?
- Can you give any intuition about why MLP(1) might have a negative correlation in Figure 2?  Is this just noise?

**Update**:  After reading the authors' responses and other reviews, most of my concerns have been addressed (assuming the two changes I suggest are made).  I am updating my score to an 8; I feel that this is a strong paper. Regarding other reviewers' concerns:

I understand reviewer BEPU's concerns and agree with some of them; but do not think they are deal-breakers.  The paper has its limitations, but I think it is extremely strong despite these.  Robotics papers do tend to have more limitations than papers based exclusively on simulated/game environments, since real-world robotics experiments are much more time-consuming to perform (but they are also often much more convincing, since they are much closer to real-world problems of interests, and not games or simplistic simulations).

I respectfully disagree with many of Reviewer YvcV's concerns, many of which amount to "the authors should have done [some other nice thing]".  I think we can always ask authors to do more, and think the many of the reviewer's suggestions would be nice to do, but I think the contribution is sufficient in this case, and that rejecting because the authors did not do even more is not the correct response.  This paper's results are impressive and extensive despite the limitations that Reviewer YvcV points out.  I think part of the disconnect may be non-robotics RL researchers forgetting how time-consuming and worthwhile real-world robotics experiments are (compared to simulated or game experiments).
summary_of_the_review: This empirical paper has some clarity issues, but is exceptionally strong otherwise.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper discusses a study of different learned autoencoder based representations in the context of generalisation (in and out of distribution) in reinforcement learning. It focuses on two types of VAE models and a robotic block reaching/pushing setting. Comparisons are performed to separately test OOD data for the representation and learned policy as well as generalisation to a real world setting after training in simulation. The analysis shows correlations between various properties and metrics and final downstream RL performance


main_review: The paper is well written and intuitive. Further studies of generalisation in robot learning are overall highly relevant and this study contributes to better understanding. That being said, it seems to be very close to prior existing work Dittadi et al 2021 (citation in paper) and some insights are close to common knowledge. 

The evaluation of generalisation also remains limited. After discussing the 7 different factors of variation in the domain, the full investigation purely focuses on color which limits the meaningfulness of the evaluation. 

On the positive side, it’s great to see the focus on investigating proxy metrics which could be evaluated as a replacement to full reinforcement learning and this part of the paper is highly relevant.

It’s interesting that a sparsity inducing regularisation on the first policy layer has no effect on the usefulness of disentangled representations. Maybe it would be useful here to evaluate with sparsity regularisation across the whole policy.

Minor:
The initial section only cites papers starting 2019 regarding the problem of generalisation. This question has a much longer history in ML research which should be taken into account.
Table 1 on page 2 includes acronyms which are explained many pages later. Please make sure that an acronym is explained before using it on its own.
Figure 1 likely presents representative view points and not agent inputs. Additionally providing these would be more helpful in understanding the sim to real gap.
Why is there no OOD1 in real?
Please add an argument for the specific choice of the 2 types of VAEs. What is the benefit of adding Ada-GVAE in addition to beta-VAE?
Why not evaluate with beta=0 to include the regular autoencoder case?
The dimensionality of the latent space could have a big impact. I’d recommend at least a minor ablation with the best known parameters and sweeping over the size of the latent space.
Why are the correlations ‘milder’ on the more complex pushing task? It does not seem directly intuitive and it would be useful to add a hypothesis.
Some related work on representation learning for reinforcement learning is missing [1,2,3]

[1] Kulkarni, Tejas D. et al. “Unsupervised Learning of Object Keypoints for Perception and Control.” ArXiv abs/1906.11883 (2019): n. pag.
[2] Wulfmeier, Markus, et al. "Representation matters: Improving perception and exploration for robotics." 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.
[3] Higgins, Irina et al. “Towards a Definition of Disentangled Representations.” ArXiv abs/1812.02230 (2018): n. pag.

summary_of_the_review: Overall a relevant investigation into the effect of representations on generalisation in RL. While some insights are trivial and the only investigated factor of variation for generalisation is color, there are aspects in the paper worth distributing and discussing (in particular the evaluation of proxy metrics for downstream performance).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents an extensive empirical evaluation of the VAE-based pretrained representations for OoD generalization of RL agents in a robotic setup.

From the reported experiment results, it draws two conclusions:
1) agents build on top of the pretrained representations can generalize to some challenging OoD scenarios; 
2) the prediction error of the ground truth values of the underlying factors of variations can be used as a proxy metric for selecting representation for OoD agents.
main_review: Strengths 
- OoD generalization of RL agents is a very active area of research. The paper provides a systematic and thorough empirical study on the impact of pretrained representations.
- The correlation analysis between different proxy metrics and the OoD generalization performance is extensive and sound.

Weakness
- While the key claim of the paper lies in OoD generalization, a large part of experiments are essentially not OoD. Among the four scenarios (IID, OoD1, OoD2, and real-world), only the OoD2 presents a clear domain shift to the pre-trained encoder. 
- It seems not very clear to which degree the *agents generalize OoD*. A more direct comparison between IID, OoD1 and OoD2 would perhaps make claim 1 about OoD generalization more convincing.  
- The core of claim 2, i.e., the GS metric can be used for representation selection, looks over-stretched to me since:
  - it requires labeled samples from the OoD domain, which is often not realistic
  - it requires ground truth values of the underlying factors of variation, which is often not realistic neither
- Finally, the experiment design overall looks quite incremental to [A Dittadi et al. ICLR'21].
summary_of_the_review: While the empirical evaluation is extensive and rigorous, the two key claims made in the paper are debatable. I, therefore, recommend reject. I'd be happy to change my rating if the author can better support the two claims during rebuttal.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper is an analysis paper that uses well-known existing pretraining techniques to study the effect of pretrained representations on out-of-distribution generalization performance. The paper mainly uses the VAEs to train the visual representation network. While I believe this is an important topic, the paper only analyzed one pretraining technique and in a visually simple environment.
main_review: Strengths:

* The paper studies the effect of pretraining visual representation on the downstream out-of-distribution task performance.
* The paper conducted a relatively thorough analysis on this issue. More specifically, the paper studies the effect of VAE pretraining on the TriFinger environment. The studied factors include the measuring metric, network regularization, input noise, different OOD distributions.
* While most of the conclusions are intuitive to understand, it's good to see that the paper has many experiments to support them.

Weaknesses:

* While the paper did a thorough analysis on VAE and in the TriFinger environment, I found these two are limiting. First, there are many different pretraining techniques such as contrastive learning and using auxilliary loss. This paper only studies VAE. Therefore, the paper title is a bit over-claimed. Second, TriFinger is a challenging control task but not necessarily a difficult task in terms of perception. As we can see in the example environment figures (Figure 1), the image input to the policy is relatively simple and has a clean background. Therefore, it is possible that the perception network can easily learn to only focus on the object in the image and ignore other things during training. And such a network can be more robust than expected and can transfer to the real world better. Therefore, the conclusions drawn in this paper only apply to VAE pre-training and TriFinger environments. It remains unclear whether the conclusions still hold in visually more complex environments such as ProcGen environments.

* Even just in the TriFinger environments, a more systematic analysis should include the testings on the TriFinger environments with different backgrounds. As we can see in Figure 1, the bowl environments are very clean and look similar in simulation and in the real world. Since the pretraining is mainly about training the perception module, OOD tests should also change the bowl texture, color during the testing.

summary_of_the_review: I believe the paper is analyzing an important issue in reinforcement learning. However, more experiments on TriFinger with different visual backgrounds and other visually complex environments such as Procgen, and experiments with different pretraining techniques are needed to make the analysis more thorough and convincing.
review_title: null

Meta-review: It can be prohibitively expensive to train a reinforcement learner from scratch &mdash; particularly in cases where experience is expensive to obtain, such as with a physical robot. So, we might hope to speed up RL in a couple of ways: first, by pre-training a representation that makes subsequent RL need less data; and second by running our RL on a cheaper proxy environment such as a simulator. For pre-training, we hope to be able to take advantage of available pre-collected data, and we hope to be able to use supervised learning or reconstruction tasks since they can be cheaper than RL. For either pre-training or a proxy environment, we have to deal with distribution shifts: the properties of the environment may change between pre-training and RL, and between RL and testing the learned policy.

The paper presents an empirical study of how different pre-trained representations and different distribution shifts affect RL performance. It evaluates a number of representations trained by different VAEs (differing in aspects such as loss and hyperparameter settings) under various scenarios of distribution shift. It also asks whether we can predict the performance of the learned policies from properties of the representations, before going to the expense of training and evaluating our reinforcement learner.

The paper concludes that it is possible to significantly reduce RL data requirements using pre-trained representations, even in the presence of significant distribution shifts &mdash; including demonstrating zero-shot sim2real transfer. And, the paper concludes that inexpensive measurements of OOD performance on supervised tasks can at least partially predict success in generalization.

The reviewers praised the extensive experimental evaluation, including a large number of experiments on a physical robot, as well as the investigation of less-expensive ways to predict generalization.

Some reviewers were concerned that the choice of environments was limiting &mdash; e.g., that the distributional distance between in-distribution and out-of-distribution tests was limited, or that the results might not generalize to other related robotic environments. However, in the end there was support for the conclusion that the experiments cover a sufficiently general and interesting question.
Decision: Accept (Poster)