Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: Focusing on the performance of an RL agent given a finite number of samples from ATARI in the regime where it is able to pretrain on a offline dataset, the paper shows that the final performance of the agent is improved by the addition of several additional auxiliary losses during pretraining, along with adjustments to the finetuning step.
main_review: Broadly speaking I believe this paper makes a strong contribution by showing that the proposed techniques improve the finetuned performance of the model, bringing it to substantially higher performance relative to other models in similar regimes. 

The evaluation of the models is detailed, the ablations and variations that are considered are reasonable and executed well, and the presentation is good. I believe that work like this which clearly sets out the benefit (and weaknesses) of pretraining approaches is worthy of acceptance. For that reason I am giving a recommendation of 7.

However I have one significant comment, and a set of more minor ones.

1) A substantial part of the improvement in mean/median scores on Atari, as shown in Table 4, comes from the particular finetuning strategy employed (specifically, the reduced LR setting). Without that trick, the performance of the network drops substantially, roughly equivalent to removing all but the Inverse Modeling head from the finetuning tasks (comparing to Table 3), or reverting back to the behavioral cloning baseline (comparing to Table 2). This has substantial impact, and while it is discussed in Section 5.4, I believe it should be give a substantially more central role to the paper, including asking some further questions to probe exactly the cause of the effect (for instance, is performance recovered with a frozen encoder if you make the Q-Learning network deeper?).

Additionally, I have a set of more minor comments:

* [Line 73] You're claiming these questions are policy-indepenent, but isn't the inverse modelling question (line 70) policy-dependent for non-trivial environments? Similarly for the question in line 71, it seems like that is policy-dependent as well.

* [Line 111] It seems worthwhile to me to cite some earlier examples of inverse dynamics modeling here.

* [Line 113] re: "it is naturally not prone to representational collapse". That seems like a broad claim which depends a lot on the situation, can you cite or explain?

* [Figure 3] This plot strikes me as begging to be extended both further to the right and in model size. Especially because you bring up the comparison to [Kaplan, 2020], it would be interesting to see these points (especially if they were extended) plotted as log-logs to try to see whether there are visible patterns in the scaling properties.

* [Line 389] You refer to an All Losses row in Table 5, but no such row exists.

* [Line 391] nit: Can you make clear that Figure 5d is in the appendix?


Finally, I would like to note that while I am familiar with the subject space, I am not fully up to date on recent developments in this area and am not a strong judgement of novelty past the references given in the paper (relative to which I think this submission is sufficiently novel). I will take a look at the other reviews to ensure that I didn't miss anything in my survey.
limitations_and_societal_impact: Yes, the societal impacts relevant to the paper are tangential and I believe the authors sufficiently describe them.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a combination of three existing self-supervised objectives (self predictive representations, goal conditioned rl and inverse dynamics modeling) to improve the performance of RL agents, after pre-training on unsupervised data and then fine tuning on a small supervised (i.e. task specific) dataset. The authors analyzed the significance of each one of these objectives as well as the importance of the number of parameters and the impact of pre-training data quality.
main_review: ==== Writing

The paper is clearly written and is easy to read and understand.  The paper does not introduce any new model or objective but rather analysis a combination of already existing objectives for pre-training and the effect of each one on the performance of the agent on the downstream task. Due to the high number of possible combinations of self-supervised objectives, I really enjoyed the notations used by the paper to make it accessible and easy to read and understand. However, these notations and abbreviations can become overwhelming and I suggest the authors to include an additional summary table which describes them all in one glance.

==== Methodology

In short, the paper is an empirical study of the importance of the pre-training objectives on the performance of the rl agent. And such studies are hard to do by nature because the number of possible combinations can grow exponentially, making experimentations harder. The authors, did a good job keeping the number of experiments at a (relatively) low number by fixing many of variables e.g. the rl algorithm itself which is fixed to Q-learning and the environment which is fixed at Atari. 

At the same time, this is a major shortcoming of the paper. For example, one may argue that the gains in Table 2 vs SimPLe with no pre-training can be described by using a fundamentally more efficient rl method (Q-learning vs PPO) just because the paper does not include any study to demonstrate the same performance gains for PPO. Or ATC vs SGI comparison can be "explained away" by insignificance of augmentation for Atari. Or one may argue that SGI is observing better performances because learning a representation of Atari games is easy and SGI will perform the same when the environment is hard to compress (at least naively) such as the The Distracting Control Suite. To summarize, I believe the paper suffers from overfitting on its fixed assets e.g. environment (Atari) and rl method (Q), and in lack of any experiment to cover other cases, this is hard to support or oppose. Ofcourse, it is impossible to run all possible variations but a selective few can substantially support the main claims of the paper. At the same time, the claims can be rewritten in a less generalize form to limit it on the scope of the experiments.

Another major issue of the paper is how it represents the representation learning (no pun intended). The paper claims "stronger" representations however this claim is only support by demonstrating stronger performance on a single downstream task (this is another reason why Atari is not a good environment for this study vs multi-task environments). I encourage the authors to use other representation learning metrics such as Mutual I(z;x) or Linear Regression of reward from z or the number of latent active units.

Overall, I find a disparity between the scope of the claims and the provided empirical evidence which can be improved by toning down the claims and adding more empirical evidence.


limitations_and_societal_impact: Minimal description of the limitation and and social impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new approach for pretraining an encoder using unlabeled data which is then used for a downstream RL task, leading to large improvements in sample efficiency. The approach is based on latent dynamics modeling and unsupervised goal-conditioned RL. The approach shows significant gains in performance and sample efficiency on Atari 100k and scales well with model size and better data. 
main_review: This paper introduces a rather simple yet effective approach for learning good representations in an unsupervised way which can be helpful for downstream RL tasks. This problem is very important as it can lead to more sample efficient algorithms and thus enable more real-world applications of RL where interactions with an environment are expensive. The paper is clear overall, the experiments are well designed, and the approach is thoroughly evaluated and compared with many relevant baselines and ablations. The results also look quite promising and the method clearly improves performance and sample efficiency relative to prior art. 

I particularly liked the analysis of the algorithm's dependence on the quality of the dataset and found the insights very valuable even if not necessarily surprising. The paper also contains many ablations thus resulting in a better understanding of how each component contributes to the final results. 

However, I think there are a few additional baseline variants / ablations which could further improve the paper and our understanding of how the various algorithms compare. Would it be possible to add experiments with BC, VISR, APT, CPT, ATC, SGI using the all of the four different datasets considered, namely R, E, W, and M? Some of these are already included in Table 2 but completing this space would be useful to gain a full picture. For Table 5, could you add results with G only and I only?

From Table 4, it looks like Frozen is comparable with BC-M so it seems that reducing the learning rate during finetuning is quite important to maximize performance. Hence, it would be interesting to see the effect of reducing the learning rate on the other baselines which perform finetuning on downstream tasks. Is the right conclusion that reducing the learning rate during finetuning plays a more important role than the objective function used for pretraining the encoder? I think further analyzing this question would be of interest for the readers. 

I think it would also be valuable to add experiments in a different domain in order to better understand whether the conclusions drawn from the Atari experiments generalize to other domains as well. A few suggestions are evaluating the methods on continuous control tasks from MuJoCo or DMC. 

Given the above reasons, I tend to recommend acceptance of this paper, but encourage the authors to take into account the feedback to further improve this paper. 
limitations_and_societal_impact: None
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The proposed method combines 3 unsupervised representation learning objectives (latent dynamics modelling, goal-conditioned RL, and inverse dynamics modeling) in order to pre-train representations for RL. The experiments on the Atari 100K benchmark show that SGI improves performance over previous representation pretraining baselines (ATC, CPT), and is competitive with exploration-based approaches (APT, VISR) which require more pretraining data and interactions with the environment.

Summary of experimental results:

(1) Data-efficiency: Table1 shows that SGI achieves comparable performance as APT and CPT (unsupervised exploration methods) but using far fewer data and environment interactions (250M steps for APT, 4B steps for CPT, 6M for SGI).

(2) Varying the quality of the offline pre-training data (collected by a random policy, SGI-R; an exploratory policy, SGI-E; suboptimal DQN actions, SGI-W; and DQN agent at different checkpoints, SGI-M): SGI’s performance increases as data comes from better-performing or more-exploratory policies.

(3) Varying the size of the encoder used in SGI-M: SGI-M/S (small), SGI-M/L (large): Larger model size results in increased performance.

(4) Pretraining vs. size of the network (Section 5.2): Model size and amount of pretraining data should increase in parallel.

(5) Ablation study (Section 5.3, Table 3): Inverse dynamics modeling objective seems to be the most critical to performance. Performance increases as more objectives are used.

(6) How to finetune pre-trained representations properly: Reducing the finetuning LRs improves performance.

main_review: Originality & Significance: The largest contribution of the paper are the extensive experiments and empirical analyses (Section 4), which I believe are valuable for understanding how pre-trained representations can help RL with learning efficiency. (See “Summary” above for a list of empirical findings reported in the paper.)

The proposed method itself (SGI) is less novel -- it is a combination of three unsupervised representation learning objectives that operate in latent space provided by a shared encoder. However, the ablation analysis (Section 5.3) is helpful in understanding how the combination of the three objectives contributes to SGI’s performance.

Quality: The experiments are thoughtfully designed and well-explained (evaluation metrics, datasets, empirical analyses and discussion of results).

Clarity: The paper is very well-written; the problem motivation, experimental design, related works, evaluation metrics, etc. are all clearly explained.

Minor nit suggestions that did not affect my score:
- L71: Missing question mark “?”
- L293: “settingss” -> “settings”

limitations_and_societal_impact: Yes
review_title: null

Meta-review: Taking as evaluation setting Atari 100k, the paper demonstrates the effectiveness of pretraining an agent torso/encoder on a collection of auxiliary self supervised losses, greatly improving data efficiency and final performance of the agent when training on a small of amount of experience on the downstream tasks (in conjunction with careful tuning of fine-tuning learning step).

Whilst the proposed unsupervised losses are not novel (a combination of latent dynamics modelling, goal-conditioned RL, and inverse dynamics modeling), the most impactful contribution of the submission is the thorough evaluation of the method, its extensive, carefully designed experiments, ablations and empirical analyses, as well as the solid choice of baselines. Although limited to Atari 100k, these experiments are well described and discussed, forming a very solid foundation for eventual follow-up work investigating if these results can generalize to other domains or training regimes,

I believe the paper will be of great interest to the broader community, as it sheds light on viable training regimes for agents in real-world application, where access to real environment experience is scarce.
Decision: Accept (Poster)