Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents an study  on the effect of multi-tasks pretraining in the context of Atari 2600 with deep reinforcement learning agents. It does so by pretraining in multiple variants of 4 games and testing in unseen variants of the game for zero-shot generalization and fine-tuning -for 200 million frames- to see if pre-training helps the agent.
strength_and_weaknesses: The paper pursues a computationally challenging and ambitious goal. However I believe it fails short to be considered for publication at ICLR since the contributions and outcomes don't really provide any new information about RL or deep learning. The paper also fails to contrast with their results with relevant literature in generalization. Specifically:

* The fact that agents don't do well in the new variants of the task with zero-shot is not surprising since the variant could mean that an optimal policy in the previous variants don't necessarily translate into the new variant. It would have been good to track from which variants the agent was learning and what is the new thing being asked. For instance if I have a variant in the test set that for the first time requires the agent to shoot to air balloons it is more meaningful to evaluate one/two-shot learning to see how the agent performs in the new task. This would explain the contradiction between the results from this work, were authors suggest that DRL is weak at zero-shot generalisation, while there are plenty of examples [1-4] to name a few, of the opposite. 

* That pre-training helps to up to certain point is quite known already. For example e.g. ALOE workshop at ICLR 2022 and other workshops mixing unsupervised learning and reinforcement learning have plenty of works on the topic already [5,6] are examples of it

* That larger networks generalize better and that are able to handle multi-task learning better it is also well known [1,7,8]. Specially relevant for this size of networks are transformers architectures, which aren't part of this work either

In general I am afraid that this work is too far below the bar of significance and novelty required for ICLR. One possible direction for this work would strength the study to include different kind of networks, e.g. relational networks, transformers. GNNs... and also use some additional benchmarks different to Atari. Current results suggest that there is a moment that pretraining in those variants is not useful anymore, finding an automatic way to find relevant variants / automatic curriculum may be appealing.

[1] Hill, Felix, et al. "Grounded Language Learning Fast and Slow." International Conference on Learning Representations. 2021.

[2] Hill, Felix, et al. "Environmental drivers of systematicity and generalization in a situated agent." International Conference on Learning Representations. 2020.

[3] León, Borja G., Murray Shanahan, and Francesco Belardinelli. "In a Nutshell, the Human Asked for This: Latent Goals for Following Temporal Specifications." International Conference on Learning Representations. 2021.

[4] Vaezipoor, Pashootan, et al. "Ltl2action: Generalizing ltl instructions for multi-task rl." International Conference on Machine Learning. PMLR, 2021.

[5]Seo, Younggyo, et al. "Reinforcement learning with action-free pre-training from videos." International Conference on Machine Learning. PMLR, 2022.

[6] Yi, Mingyang, et al. "Improved ood generalization via adversarial training and pretraing." International Conference on Machine Learning. PMLR, 2021.

[7] Brown, Tom, et al. "Language models are few-shot learners." Advances in neural information processing systems 33 (2020): 1877-1901.

[8] Chan, Stephanie CY, et al. "Data Distributional Properties Drive Emergent Few-Shot Learning in Transformers." arXiv preprint arXiv:2205.05055 (2022).
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written although there are some points that need rework:
*Figure 1, the numbers within the images are never explained
* First paragraph section 3, it misses an explanation about what are sticky actions
* The end of that paragraph, explain what is that pre-processing (you can do it in the appendix but then refer it)
* Third paragraph, you mention you restrict the variants of the games, how?
* fourth paragraph, how many seeds you used for Impala normalized scores?
*fifth one, it is not evident why 15 billion frames assures that agents interact with each mode for at least 250 million frames

Since the paper is all about the Atari experiment quality is not good given that many details about reproducibility are missing and that it doesn't contrast the results with existing literature. Still I believe this is fixable in the rebuttal time.

Novelty and significance is the biggest issue here since the contributions of this work are all well known things within the current literature
summary_of_the_review: The paper presents a large experiment focused on Impala with varied residual nets Atari games variants. However the outcomes of the experiment are all well abundant in existing literature.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper analyses an RL training regime consisting of multi-task pretraining on a set of atari game variants followed by fine-tuning on an unseen variant. They show that while multi-task pretraining often doesn't help zero-shot performance (in particular scaling training time and number of variants doesn't affect zero-shot transfer performance), it improves the fine-tuning performance and sample efficiency significantly, and these improvements scale as pretraining dataset size and number of variants increases. They further show that in this multi-task pretraining regime, larger models than previously used can provide strong benefits to both pretraining and fine-tuning performance.
strength_and_weaknesses: Strengths: The questions asked and answered by the paper are interesting and unanswered, and the rigorous experiments provide robust conclusions in this setting, which I'd expect to at least somewhat generalise to settings other than Atari. The paper is well-written and easy to follow, and all the results are clearly presented. I think the results are novel and perhaps counter to some prevailing wisdom in the RL community about the appropriate model size and the value of pretraining. The use of fine-tuning to evaluate the impact of pretraining is good, as it shows conclusions that previously we didn't have.

Weaknesses: There are few weaknesses that I can see. Obviously it would be better if the experiments were demonstrated on more environments, perhaps ones which are different from atari games in some way, but I think the current experiments provide sufficient evidence for the conclusions.

Some suggestions to improve the completeness of the paper: I'd like to see the normalised reward curves during pretraining for the various model sizes, to see if increasing model size also improves pre-training performance as well as fine-tuning performance. Also, given the amount of data gathered on both pretraining dataset size and model size, it would be interesting to see if a functional form for transfer performance similar to https://arxiv.org/abs/2102.01293 can be found, which would allow us to answer questions such as "how much pretraining data do we need to reduce the amount of fine-tuning data by 5x while maintaining the same performance?". If the functional form was similar for the different games, then it could provide guidance for how to trade off pretraining and fine-tuning compute.
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is easy to read and understand, and all the results are presented clearly.

Novelty: The experiments presented are novel, and produce conclusions which were unknown beforehand.

Quality: I think the work is of high quality, as mentioned in the strengths section.

Reproducibility: Unfortunately, due to the high computational costs of running all the experiments, the results aren't that reproducible without a similar level of compute. However, if the authors released the data from all their experiments, then that would enable others to build on and analyse that data without having to rerun expensive experiments.
summary_of_the_review: Overall, I think the paper is worthy of acceptance, and hence I'm recommending an 8. I think the paper's experiments are novel and conclusions interesting and in some cases surprising. In general I'm in favour of more empirical scientific work of this kind, as I think it's something the RL community is lacking.

I think it would be difficult for my score to rise from an 8 to a 10, and it would probably require either: similar experiments on another qualitatively different class of environments to make the results even more robust; the discovery of a robust functional form for fine-tuning performance which holds across environments; or experiments for different algorithm types, to see if these results also generalise to Q-learning based and/or model-based methods.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper investigates how model size and pretaining on similar environments/more data affect the zero-shot and fine-tuning generalisation capabilities of RL agents. 

strength_and_weaknesses: ### Strengths

* The paper is a good methodical study of an effect of the amount of data/diff environments and the model size on the generalisation properties of an RL agent.
* The paper does not invent a flashy algorithm to push some curves higher but does a thorough study of a phenomenon which might be useful for the researchers and practitioners.
* The effect of number of environments/model size on the fine-tuning performance is important and novel, to the best of my knowledge.

### Weaknesses

* Most of the shown results are known in the community. They have been scattered around different works though, and having them in one place is a plus to the paper in review. However, IMPALA paper has already shown a positive effect of model size on the performance, while the ProcGen paper has shown that training on many seeds (similar to many variants in the current work) improves the zero-shot generalisation performance
* The paper positions itself as the multi-task learning paper, and I would expect it to have a more methodological 'related work' section pinpointing the main literature on the topic.

### Requests/Questions

* Could you, please, add a separate 'related work' section to the paper? Before the additional page is allowed, I am fine with having it in the appendix to be moved to the main text afterwards.
* Figure 6 in log-scale looks a bit confusing to me with most of the curves flattened out. Can you, please, replot them without the log scale in the appendix?
* I think that the paper should be more explicit about the previous works showing similar results: IMPALA’s experiment showing that model size positively affects the performance, and procgen’s number of seeds and model scaling experiments. I don’t think that procgen’s model scaling was discussed in the paper at all. I think these should be added somewhere in the introduction.
* What do you think the limitations of your paper are? Can you add a paragraph on these to the paper?
* You mention that prior IMPALA’s multitask implementation uses a separate critic per task. Can you give any pointers to either the paper or specific implementation?
* “By restricting the different tasks variants of the same game we observe less negative interference between tasks”. Can you provide any evidence for this?
* Figure 6 shows quite a big jump from no pretraining to 4-mode pretraining. Can I see the same curve, but for 1,2 and 3 modes? Is there a huge qualitative jump from 1 or 2 modes pretraining?

### Nits

* Why do you include 1 to the gamma range in Section 2?
* You cite Puterman’s book for MDP, but include \gamma to your MDP tuple. In Puterman’s book, \gamma is not included to the MDP definition.
* Can you plot Figure 3 with shared y-axis? I want to see patterns across the games, and, especially, one subplot in log-scale just makes it confusing.


clarity,_quality,_novelty_and_reproducibility: The paper is mostly clearly written, the evaluation is done is a methodological way, and the paper is of high quality overall. My thoughts on the novelty of the work are in the section above.
summary_of_the_review: I preliminary choose '6: marginally above', but I am ready to bump the score up if my requests are addressed throughout the discussion period.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper is an empirical study of the generalization abilities of the deep RL algorithm IMPALA pre-trained on multiple tasks. Pre-training is done on variants of a single game from the Atari benchmark and fine-tuning/testing is done on held-out variants. The authors show that by limiting to variants of a single game, negative interference is reduced enough that the pre-training improves sample complexity (and sometimes final performance as well). However, zero-shot performance does not seem to be affected. In addition, the paper suggests that larger networks, such as those from the ResNet family, can be used in the policy/value function in this multi-task pre-training setting.
strength_and_weaknesses: # Strengths
- The paper is clear and simple to follow. The authors are clear about the set-up and assumptions.
- The experimental set-up is sound. In particular, the same number of time-steps is used for fine-tuning as for training a single environment on the ALE benchmark. 
- The paper shows that we can use a ResNet for policy/value function architecture in some cases. 

# Weaknesses
- The settings considered in the paper are fairly limited. Only the IMPALA algorithm is considered, and only variants of the same game are pre-trained on.
- The finding that pre-training helps is not surprising, given the fact that only variants of the same game are pre-trained on and other variants are tested on.
clarity,_quality,_novelty_and_reproducibility: # Clarity
The paper is written clearly, and the graphs are easy to understand. 

# Quality
Overall, the paper appears to be technically sound. The experimental set-up is valid and the analysis of the results is fairly complete. However, the set-up seems to be a bit limited; only the IMPALA algorithm is studied. It would also be interesting to study how similar the tasks have to be for multi-task pre-training to be useful. For example, could we use variants of two similar games such as Pong and Tennis be used?

# Novelty
In my opinion this work has moderate novelty at best. There have been many works on generalization in deep RL using similar environments and algorithms; Machado et al. (2018) uses the same environments and Cobbe et al. (2020) uses IMPALA. As stated in the paper, the main difference is that fine-tuning (with the same number of time steps as the usual single environment Atari benchmark) is considered. The conclusion that pre-training helps is not surprising, given that the diversity of environments is restricted. The use of a ResNet for image-based multi-task deep RL is novel, as far as I know.

# Reproducibility
Code is not provided. However, the authors link to the environments/repositories used and list the hyperparameters, so I think reproducibility is not an issue.
summary_of_the_review: The paper is well written, the experimental set-up is sound, and the analysis is clear. However, the setting is somewhat limited, and the results for the most part are not significantly novel.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
This paper studies multi-task pretraining on a set of atari game variants followed by fine-tuning on an unseen variant. The paper shows that while multi-task pretraining often doesn't help zero-shot performance, it improves the fine-tuning performance and sample efficiency significantly, and these improvements scale as pretraining dataset size and the number of variants increases. Furthermore, it was shown that in this multi-task pretraining regime, larger models than previously used can provide strong benefits to both pretraining and fine-tuning performance.
The reviewers raised concerns about the claims and experiment setups. The AC thanks the authors and reviewers for extensive and engaging discussions. 
The AC believes the paper presents rigorous studies on several phenomena about pertaining, which are valuable to the community. 
justification_for_why_not_higher_score: The experiments are limited in Atari.
justification_for_why_not_lower_score: The experimental results on pretraining are valuable to the community
Decision: Accept: poster