Investigating Multi-task Pretraining and Generalization in Reinforcement Learning
Authors: Adrien Ali Taiga
Rishabh Agarwal
Jesse Farebrother
Aaron Courville
Marc G Bellemare
Venue: ICLR 2023 poster (2023)
Keywords: generalization
transfer
atari

Abstract: Deep reinforcement learning~(RL) has achieved remarkable successes in complex single-task settings. However, designing RL agents that can learn multiple tasks and leverage prior experience to quickly adapt to a related new task remains challenging. Despite previous attempts to improve on these areas, our understanding of multi-task training and generalization in RL remains limited. To fill this gap, we investigate the generalization capabilities of a popular actor-critic method, IMPALA. Specifically, we build on previous work that has advocated for the use of modes and difficulties of Atari 2600 games as a challenging benchmark for transfer learning in RL. We do so by pretraining an agent on multiple variants of the same Atari game before fine-tuning on the remaining never-before-seen variants. This protocol simplifies the multi-task pretraining phase by limiting negative interference between tasks and allows us to better understand the dynamics of multi-task training and generalization. We find that, given a fixed amount of pretraining data, agents trained with more variations are able to generalize better. Surprisingly, we also observe that this advantage can still be present after fine-tuning for 200M environment frames than when doing zero-shot transfer. This highlights the potential effect of a good learned representation. We also find that, even though small networks have remained popular to solve Atari 2600 games, increasing the capacity of the value and policy network is critical to achieve good performance as we increase the number of pretraining modes and difficulties. Overall, our findings emphasize key points that are essential for efficient multi-task training and generalization in reinforcement learning.

Published as a conference paper at ICLR 2023
INVESTIGATING MULTI-TASK PRETRAINING AND GEN- ERALIZATION IN REINFORCEMENT LEARNING
Adrien Ali Taiga
Rishabh Agarwal
MILA, Université de Montréal
MILA, Université de Montréal
Google Brain
Google Brain
alitai gaß goog 1 e. com
rishabhagarwa 1 @goog ggoogle.com com
Jesse Farebrother
Aaron Courville
Mila, McGill University
MILA, Université de Montréal aaron. courvil 1lle @umontreal. ca
Google Brain
farebroojtmiaa, quebec
Marc G. Bellemare *
Google Brain
be dellemare@@ooggee.com com
com
ABSTRACT
Deep reinforcement learning (RL) has achieved remarkable successes in complex single-task settings. However, designing RL agents that can learn multiple tasks and leverage prior experience to quickly adapt to related new task remains chal- lenging. Despite previous attempts to improve on these areas, our understanding multi-task training and generalization in RL remains limited. To fill this gap, in- vestigate the generalization capabilities of a popular actor-critic method, IMPALA (Espeholt et al. 2018). Specifically, build on previous work that has advocated for the use of modes and difficulties of Atari 2600 games as a challenging bench- mark for transfer learning RL (Farebrother et al.) 2018 Rusu al. 2022). We do so by pretraining an agent on multiple variants of the same Atari game before one-tuning on the remaining rever-before-sent variants. This protocol simplifies the multi-task pretraining phase by limiting negative interference between tasks and allows us better understand the dynamics of multi-task training and gener- alization. We find that, given fixed amount of pretraining data, agents trained with more variations are able to generalize better. Surprisingly, we also observe that this advantage can still be present after fine-tuning for 200M environment frames than when doing zero-shot transfer. This highlights the potential effect of good learned representation. We also find that, even though small networks have remained popular to solve Atari 2600 games, increasing the capacity of the value and policy network is critical to achieve good performance as increase the number of pretraining modes and difficulties. Overall, our findings emphasize key points that are essential for efficient multi-task training and generalization in reinforcement learning.
INTRODUCTION
Deep RL has achieved remarkable results in recent years, from surpassing human-leves performance on challenging games (Silver et al. 2017 Berner et al.] 2019 Vinyals et al. 2019 to learning complex control policies that be deployed in the real world (Levine al. [2016] Bellemare et al. 22220). However, these successes were attained with specialized agents trained to solve a single task and with every new task requiring a new policy trained from scratch. On the other hand, high-capaciry models trained on large amounts of data have remarkable generalization abilities in other deep learning
"CIFAR Fellow

---

Published as a conference paper at ICLR 2023
domains such vision and NLP (Brown al. 2020: He et al. 2022) Such models can solve multiple tasks simultaneously (Chowdhery et al. 2022) show emergent capabilities (Srivastava et al. (022). and quickly adapt unseen but related tasks by leveraging previously acquired knowledge (Brown lt al. 2020). While recent work has succeeded in learning such broadly generalizing policies using
supervised learning (Reed al. 2022 Lee et al. (022), despite many attempts, deep RI. agents have not been able to achieve the same kind of broad generalization.
RL policies pretrained on multiple tasks are often unable to leverage information about previous tasks to accelerate learning on related tasks (Kirk et al. (202)) Furthermore, multi-task agents are reported to perform worse on individual tasks than a single-task agent trained on that task (Espeholt et al., 2018). The limited performance of multi-task policies is believed to be due to negative interference between tasks (Schaul et al. 2019). Recent work has tried to address this issue, |Hessel et al. (2019) proposes reward rescaling scheme that normalizes the effect of each task on the learning dynamics. Similarly Guo al. (2000) introduces self-supervised learning objective to improve representation learning during multi-task training. Other works have shown that multi-task pretraining can lead to better representations that are useful for downstream tasks when the policy only needs to generalize to new reward (Borsa al.,, 2016; Yang et al.) 2020; (odhani et al., 2021). Nevertheless, in complex multi-task multi-task problems, such as the Arcade Learning Environment (ALE: Bellemare et al.) 2013), generalization of pretrained policies to unseen tasks remains unsolved problem.
In this paper, we propose to take a closer look at multi-task RI. pretraining and generalization on the ALE, one of the most widely used deep RL benchmark. Though, one might hope that an agent could benefit from multi-task training to learn some high level concepts such as affordances (Khetarpal al., 2020 or contingency awareness (Bellemare al., 2012). the lack of common ground between tasks makes joint training on multiple Atari games a difficult problem. To circumvent these issues, we make use of the modes and difficulties of Atari 2600 games (Figure [.. which we call variants (Machado et al. 2018). These variants were developed by game designers to make each game progressively more challenging for human players. Previous work has advocated for their use to study generalization in reinforcement learning (Farebrother et al. 2018 Rusu et al. (022). though these works did not go beyond pretraining on a single task. Notably, Farebrother et al. (2018) argued that the representation learned by DQN agent (Mnih et al., (013) after pretraining is brittle and is not able transfer well to variants of the same game. They find that during pretraining, the representation tends to overfit to the task it is being trained on, decreasing its generalization capabilities.
Our goal in this work is to revisit these results under a new light, leveraging advances in algorithms, architectures and computation that have happened since. We
more efficient algorithm, IMPALA (Espeholt et al. 2018) instead of DQN, much larger networks than the decade old 3-layer convolutional neural networks used by DQN (Mnih et al. 2013) and pretrain our agents using multiple variants of a game as opposed to just a single one. By limiting the pretraining tasks to different variants of the same game, we facilitate multi-task transfer between variants, which in-turn allows us study under what conditions RL algorithms are able to generalize better when training on multiple tasks. Our contribution are as follow:
We find that pretrained policies can achieve zero-shot transfer on variants of the same game. If we then fine-tune these polices using interactions in the unseen variant, the fine-tuned policies generalize quite well and learn significantly faster than a randomly initialized policy.
We observe that a good representation can be learned using pretraining on a relatively small number of modes. Fine-tuning performance from these representations improve as we increase the amount of the pretraining data, as opposed to overfitting on pretraining tasks.
Finally, we demonstrate that it is possible to train high capacity networks such as residual networks (He et al. 2016),, with tens of millions of parameters, using online RL. We find that increased representation capabilities from such networks are essential to reach peak performance in the multi-task regime.
BACKGROUND
Reinforcement learning: We consider Markov decision process (MDP) (Sutton & Barto 2018) M represented by a tuple AS, P,r,y, with the state space, A the finite set of actions, P S x A > (0, i| the transition probability distribution, r S x A + R the reward function, 7 E [0,
2

---

Published as a conference paper at ICLR 2023
Space Invaders
Asteroids
Breakout
Bank Heist
880
Figure 1: Examples game variants of Atari 2600 games. Each column represents two different variants of game, where show their visual differences using the numbered rectangles. In SPACE INVADERS shields move from static to mobile, ASTEROIDS the shape of the spaceship changes, in BREAKOUT the ball becomes steerable by the paddle, in BANK HEIST banks appear different locations. While there are only subtle changes in the visual observations or agent embodiment, such changes crucial for the optimal policy, which makes generalization these variants difficult.
the discount factor and p the initial state distribution. At each timestep the agent observe the current state 8t, then its chooses an action sampled according to its policy receive reward Ft and observe the subsequent state $1+1- Given policy
A - [0, 1] before it can define the value function at time f to be the discounted sum of expected rewards to be collected from the current state when following policy T: ((*) == E Σ 7're4s/8,, ag ~ x8t)]. The agent's goal to find a policy that maximizes its value in each state.
Transfer learning / Generalization: In transfer learning, we are interested in designing learning methods that are able to leverage previous interactions with task environment solve tasks more efficiently. There are many possible ways to quantify transfer in reinforcement learning. The first one we study is zero-shot where performance is evaluated on the target tasks without any adaptation (e.g.. Cobbe et al., [2200a] Agarwal et al. 202 la Zhang et al. 2021). We also evaluate performance after the agent is allowed to interact with the target environment for fixed number of online environment interactions. We refer the reader to Zhu et al. (2000;; Kirk et al. (2011 for in-depth overview on transfer and generalization in deep reinforcement learning.
Arcade Learning Environment: The Arcade Learning Environment (Bellemare al. 2013) provides an interface to Atari 2600 games and was first proposed as a platform to evaluate the abilities of reinforcement learning agents across wide variety of environments. It has since become cornerstone reinforcement learning research. Practitioners have focused mostly on training agents individually each game to achieve the highest possible reward and current techniques far exceed human performance (Mnih et al.) 2015 Van Hasselt et al.) 2016 Bellemare et al. 2017
Kapturowski et al., 2018 Badia et al. 2020). Recent work has also looked the ALE in the context of generalization, transfer learning and continual learning (Farebrother et al. 2018 Rusu et al. 2022). The major benefit of the ALE compared other generalization benchmarks (Nichol et al.) 2013 Yu et al. (020) is that it proposes a diverse set of games that is free of researcher bias, as these games were designed to be challenging for human players, rather than for improving deep RL agents. The latest version of the ALE (Machado et al., (2018) introduces different modes and difficulties for Atari 2600 games. These flavours were part of the original titles and were designed to offer players more entertaining experience by offering variations of the default game. Each variant offers different alteration of the base game across some discrete modifications while most of the game remains the same (see Figure] D). Because the underlying dynamics of the environment do not change much can expect that playing some modes can lead to faster learning on others. Faaebrotherr et al. (2018) and Rusu al. 2022 have both previously investigated the transfer and generalization capabilities of value based agents using modes and difficulties of Atari 2600 games. Our work goes further by looking the impact of pretraining multiple variants improve generalization.
3

---

Published as a conference paper at ICLR 2023
3 EMPIRICAL METHODOLOGY
Environment Protocol: We follow the latest ALE recommendations proposed by (Machado et al. 2018). We use sticky actions to introduce stochasticity in the environment, with probability 0.25 the environment will not execute the action specified by the agent and will repeat the previous action instead. The agent also does not receive a signal when a life is lost within the game. Even when not all actions are available, agents must act with the full set of 18 possible actions. We use the default pre-processing (Mnih et al., 2015) for environment frames.
ALE Games & Variants: We will be using four Atari games, SPACE INVADERS, ASTEROIDS, BREAKOUT and BANK HEIST. We chose them because of their relatively large number of game variants, 32 for SPACE INVADERS, 66 for ASTEROIDS, 24 for BREAKOUT and 32 for BANK HEIST (see the Appendix for detailed overview of each game). For each game we split the available variants into a train and a test set, where the test set contains unseen variants. This split is done randomly, except for SPACE INVADERS where we reuse the test set introduced by Farebrother et al. (2018).
Pre-training Agent: All our experiments use IMPALA (Espeholt et al. 2018 model-free actor- critic method. IMPALA introduces V-traces, an off-policy learning algorithm that corrects updates that become off- policc due to the lag between actors and learner in distributed setup. Prior work using IMPALA in the multi-task regime always used an separate head of value function for each task. Here, we use a single policy and value network play every flavour of game. By restricting the different tasks variants of the same game we observe less negative interference between tasks which allows us share parameters between the policy and value function. For experiments that analyze the impact of network capacity, we also use deeper ResNets (He et al. 2016) than those used by Espeholt al. (2018). Further details regarding hyperparameters and the training procedure may be found in the Appendix.
Reporting & Evaluation: Given that human scores are not available for Atari 2600 game modes and difficulties, we cannot report nomaan--ommllzedd scores. Instead, we provide results using an score, where we assign a normalized score of to the average score obtained by an IMPALA agent trained for 200 million frames and a normalized score of 0 to the random agent. For reliable evaluation, we report aggregate metrics and performance profiles, with 90% stratified bootstrap confidence intervals (Agarwal et al. (2021b) In particular, we the interquartile mean (IQM) recommended by Agarwal et al. (2021b). which computes the mean of normalized scores over the middle 50% of the runs, combined across all the test variants. IQM is more robust to outliers than mean and results in smaller CIs than median. We report learning curves collected by a separate evaluation process during.
Pre-training procedure: Unless otherwise specified, the amount of pretraining data is kept fixed for each agent 15 billion environment frames. As a result, each agent interacts with each mode and difficulty for at least 200 million frames. The different tasks are split among multiple actors and the agent must play all variants of game in the training set at the same time. The agent is not provided the mode and difficulty identities with the observation. Though passing this information may improve training performance, it would negatively impact generalization unseen variants.
Fine-tuning on unseen variants: A common way to use learned representation is to reuse its parameters as initialization to solve subsequent task. We follow this procedure and use pretrained models to initialize the policy and value network of a new IMPALA agent to play the test set games. Previous work has usually limited the length of the fine-tuning process to 10 million frames, here we choose the standard limit of 200 million frames to quantify the impact of the learned initialization using the standard amount of online interactions used by ALE agents (Machado et al. 2018). Every fine-tuning experiment is run using 10 seeds.
4 EXPERIMENTS
Using the above methodology, we now present our results on multi-tase multi-task pretraining and generalization using the game variants in ALE.
4

---

Published as a conference paper at ICLR 2023
Zero-shot Performance
Fine-tuning 200M Performance
Asteroids
Impala-Normalized IQM
Bank Heist
300%
100%
200%
50%
100%
0%
0%
0
16
32
63
8
22
29
Number of Pretraining modes
Number of Pretraining modes
Impala-Normalizaee IQM
Space Invaders
Impala-Normalized IQM
Breakout
600%
100%
400%
50%
200%
0%
0%
01 4
15
29
11 16 21
Number of Pretraining modes
Number of Pretraining modes
Figure 2: Zero-shot vs. Fine-tuning performance on unseen modes using interquartile mean (IQM) scores. Error bars show 90% bootstrap confidence intervals. Notably, increasing the number of modes for pre-training consistently improves zero-shot and fine-tuning performance after 200M frames of online interactions for most games. These results indicate that the pre-trained representations learn relevant features that can transfer to variants of the same game.
4.1 PRETRAINING ON MULTIPLE VARIANTS
RL agents are often trained starting from randomly initialized policy, as it is unclear whether they leverage their experience during pretraining to generalize well subsequent tasks. To understand this behavior we are interested in quantifying the impact of pretraining and number of pretraining variants the transfer and generalization capabilities of IMPALA agent (Espeholi) et al. 2018). We first compare agents pretrained on subsets containing different numbers of variants of game using the same budget of 15B environment frames. We evaluate zero-shot transfer by using a pretrained policy directly on target variant. We also analyze fine-tuning performance by allowing the pretrained policy to interact with the test variant for 200 million environment frames. Given the large number of variants for selected games, it precludes pretraining on all possible subsets of variants as this number grows exponentially with the number of variants. Instead, we randomly subsample multiple subsets of different sizes and require that each subset be a superset of previously picked smaller subsets. This was necessary to limit the effect particular variants that might have a larger impact generalization due to their higher similarity with unseen variants (Rusu et al. 2022)
4.1.1 ZERO-SHOT
Zero-shot generalization is the most desirable but also most difficult kind of generalization to achieve as it does not allow for any interaction with the target environment. In the context of RL generalization benchmarks with procedurally generated environments, such as Procgen (Cobbe et al.) (020a)). zero- shot transfer only emerges after interacting with hundreds of training environments. Here, we assess the zero-shot performance of pretrained models on the test game variants after pretraining on relatively much smaller number of environments. Results are shown in Figure (orange). Pretraining highly beneficial on all games- on ASTERIOIDS and SPACE INVADERS pretrained agents achieve higher than the non-pretrained IMPALA baseline. We also observe that policies pretrained with more modes generally perform better and benefit from the additional diversity in the pretraining data. While results contrast with Farebrother al. (2018), who did not observe much zero-shot
5

---

Published as a conference paper at ICLR 2023
Asteroids: Performance Profiles
Impala-Normal mazlizzed IQM (Log Scale)
Fraction runs with score
1.00
400%
0.75
Pretraining Modes
300%
0.50
32
200%
0.25
100%
0 0.4.8 8 16 32
63
0%
100%
200%
300%
400% 500%
Number of Pretraining modes
Impala-Normalized Score (r)
Impala-Normalized IQM
Fraction of runs with score >
Bank Performance Profiles
1.00
100%
0.75
0.50
50%
Num Pretraining Modes
0.25
22
0%
0.00
29
8
15
0%
20%
40%
60%
80%
100% 120%
Number Pretraining modes
Impala-Nore lodela--orallzzedd Score
Breakout: Performance Profiles
Impala-Normal IQM
200%
Fraction of runs with score
1.00
Num Pretraining Modes
0
16
0.75
150%
0.50
100%
0.25
50%
0.00
11
16
21
50%
100%
150%
200%
Number Pretraining modes
Impala-Normalized Score (r)
Fraction of runs with score >t
Space Invaders: Performance Profiles
Impala llommalized IQM
1.00
Pretraining
15
0.75
22
(Log Scale)
1000%
0.50
0.25
100%
0.00
15 Number of Pretraining modes
4
0%
250% 500% 750% 1000% 1250% Impala-Normalized Score (r)
Figure 3: Fine-tuning performance unseen modes after 200M frames. On all games except BREAKOUT, pretrained policies significantly outperform the baseline that was randomly initialized (0 modes). On BREAKOUT, we observe a huge boost in sample efficiency from pretraining but final performance saturates for agents. Left. Aggregate IQM as a function pretraining modes. Crosses show performance on the 3 individual variants, where each color corresponds to specific variant. Right. Performance profiles are quite useful for qualitative summarization: Higher is better. Area under the profiles correspond to the mean performance, the median can be read by the intersection of the profile curves with the horizontal line at y = 0.5. Similarly, we can easily check how many score above certain score threshold using the intersections of the curves with the vertical line x 7. Shaded region shows 90% stratified bootstrap Cls.

---

Published as a conference paper at ICLR 2023
Impala-Normalizer IQM
Asteroids
Bank Heist
400%
100%
Num Pretraining Modes
200%
50%
Num Pretraining Modes
0%
0%
0
50
100
150
200
0
50
100
150
200
Impala-Normalizee IQM
Space Invaders
Breakout
Num Pretraining Modes
100%
29
500%
50%
Num Pretraining Modes
0
21
0%
0%
50
100
150
200
50
100
150
200
Number of Frames (in millions)
Number of Frames (in millions)
Figure 4: Sample efficiency IQM evaluation curves for IMPALA with no pretraining (0 modes), a small number of pretraining modes and a large number of pretraining modes. Shaded regions show 90% bootstrap confidence intervals. Pretrained agents on a large number of modes achieve a much score than agents trained from scratch, in some cases like BREAKOUT and BANK HEIST they do need to be finetuned.
transfer Atari after pretraining single mode, the two results do not contradict each other as we see that SPACE INVADERS pretrained using single mode does not generalize well zero-shot. Surprisingly, we find that pretraining on just 4 modes enables good zero-shot transfer.
4.1.2 FINE-TUNING
A common approach for transfer is to fine-tune a pretrained model on the test variant, giving time for the policy to adapt to the new task. In this section we focus on quantifying the performance of fine-tuning models after pretraining with different numbers of variants. We report results in Figure [ We observe that fine-tuning is valuable for games such as SPACE INVADERS and ASTEROIDS where some models had not achieved the highest score they could attain. On ASTEROIDS and SPACE INVADERS, fine-tuned models significantly outperform the IMPALA baseline trained from scratch and reach x 6x higher scores.
Our result corroborates previous results demonstrating the benefits of pretraining and fine-tuning, in particular, with domain randomization (Tobin al., 2017 Cobbe et al./ (2020b). What is surprising that, in our setup, these effects are largely captured with a relatively small number of pretraining modes and difficulties, possibly due to the fact that these variants may be sufficient to cover most factors of variations present in the test set. We see little additional improvements when increasing the number of pretraining modes. On the other hand, domain randomization will often generate hundreds of thousands of variations of an environment; Cobbe et al. (2020b)'s 's experiments required thousands of levels see an improvement terms of zero-shot generalization. These results indicate that few-shot transfer is much more difficult to achieve and requires a large amount of diversity in the pretraining data. Nonetheless, representations learned with a handful of modes result in sample-efficient fine-tuning even when they have limited zero-shot transfer capabilities. For example, despite not performing better than a random agent in zero shot transfer, the agent pretrained on a single SPACE INVADERS see its performance increasing rapidly in the first 25 million frames after adapting its representation to the new task at hand. Somewhat surprisingly. the pretrained agents still significantly outperform the same agent trained from scratch (0 modes) even after 200 million frames, whereas we could have imagined that the effect of initialization had been washed away by now. This in line with recent work (Igl et al. 2020 initialization in deep RL.
Nikishin et al. 2022) highlighting the lasting effects of
7

---

Published as a conference paper at ICLR 2023
Asteroids
Impala-Normaliied IQM
1000%
750%
500%
Num Pretraining Modes
250%
1
16
2
63
0%
IMPALA
ResNet-18
ResNet-34
ResNet-50
ResNet-101
Network Architecture
Space Invaders
Impala- Impala-Norme -Noomalizedd IQM
2000%
1500%
1000%
Num Pretraining Modes
500%
4
15
0%
2
29
IMPALA
ResNet-18 18
ResNet-34
ResNet-50
ResNet-101
Network Architecture
Figure 5: Impact of network capacity fine-tuning performance unseen modes. 0 modes corresponds to no pretraining. When networks are randomly initialized, ResNets underperform the IMPALA architecture and performance decreases the network capacity is increased. With pretraining ResNets significantly outperform the IMPALA architecture.
4.2 SCALING MODEL SIZE
Our results so far hold for the network introduced with the IMPALA algorithm (Espeholt et al. 2018) Though it an improvement over the DQN network architecture, the IMPALA network has only 15 layers and less than 2 million parameters. This is relatively small in comparison with models used in vision and NLP where large networks trained on large datasets have been shown to be superior and display impressive generalization capabilities. Additionally, larger capacity models in single-task deep RL settings often lead to instabilities or performance degradation (Van Hasselt et al. 2018)) Ota al., 2021 Bjorck et al. 2021), making it unclear whether multi-task RL leverage such high-capacity models. We may wonder if increasing the network size is beneficial in the multi-task setting, where there diversity in the training data and hence more to learn.
In this section we investigate this idea and take a closer look at the generalization capabilities of the IMPALA algorithm combined with larger networks from the ResNet family (He al. 2016). In our experiments we use ResNet-18, ResNet-34, ResNer-50 and ResNet- 101 (respectively 11.4, 21.5, 23.9 and 42.8 million parameters). As before, we pretrain these architectures for 15B frames with varying numbers of game variants before fine-tuning on the test flavours for 200 million frames. Due to the huge computational cost of these experiments we limit our results ASTEROIDS and SPACE INVADERS. Results are displayed in Figure 5] When training only on test environments, increasing the network size noticeably degrades performance in both games. This result is aligned with prior findings and may explain the dearth of reported results using very large networks in the RL literature. Inversely, pretraining on multiple variants unlocks the potential larger networks in deep RL. As such, observe significant increase in performance (note the logarithmic scale) and these architectures achieve a score 3-4x higher than the IMPALA network trained in the same conditions. Moreover, most of these benefits attained with ResNet-18 18, with small benefits in performance
8

---

Published as a conference paper at ICLR 2023
IQM Impala Normalized Score
IMPALA
IMPALLA X3
IMPAAA Reshw:118
IMPAALS 24
Restier-3 38
100%
1000%
0%
500%
0
50
100
150
200
50
100
150
200
(a) ASTEROIDS without pretraining
(b) ASTEROIDS with pretraining on modes
Figure 6: Sample efficiency IQM evaluation curves for IMPALA with pretraining (0 modes) and a large number of pretraining modes. Shaded regions show 90% bootstrap confidence intervals. IMPALA x2 and IMPALA x4 achieve higher returns with and without pretraining. However they remain far below the performance of fine-tuned residual networks showing that these networks generalize better.
when further increase model capacity. Altogether, it is unexpected have such a dramatic impact on the performance of larger networks. In recent years, researchers have attempted to train multi-task policies all Atari games using slight variations of the IMPALA architecture (Hessel et al., (2019; Guo et al. (020). Our results indicate that these approaches would likely benefit from higher capacity networks.
that pretraining can
4.3 SCALING IN WIDTH
Prior work has looked into scaling IMPALA using wider networks by increasing the number of channels in convolutional layers. Cobbe al. (2020a) multiplied the number of channels in the IMPALA architecture by a factor and 4, they observed better sample efficiency and generalization with wider networks. In this section attempt understand how scaling the network in width compares with scaling in depth with residual networks. We evaluate the wider architectures used by Cobbe et al. (2020a), referred as IMPALA x2 and IMPALA x4 (respectively 2.4 and 5.5 million parameters) with the baseline IMPALA architecture and ResNet-18 18. Figures 6 and 9 show learning curves for these architectures on ASTEROIDS and SPACE INVADERS test variants when networks trained from scratch and after pretraining. We observe that IMPALA x2 and IMPALA x4 provide meaningful benefits over the IMPALA baseline. These models achieve higher returns when they are trained from scratch and after pretraining which is line with Cobbe al. (2020a)'s 's results. Nonetheless after pretraining their performance still remains far ResNet- 18 which achieves almost twice their performance.
CONCLUSION
Deep reinforcement learning has made lot of strides in recent years, yet real world applications remain seldom. Better generalization in reinforcement learning could be one of the missing pieces unlock real-world reinforcement learning. Having access generalist agents that can perform multiple tasks and are capable of leveraging their prior knowledge to learn new tasks efficiently would take RL new heights. Nevertheless, prior work has shown that RL remains far from achieving such ambitions and that generalization in RL should not be relied upon. Yet our work offers glimmer of hope: in our controlled setup, RI. agents pre-trained on a small number of game variants generalize to never-before-seed variants and considerably improve sample efficiency. Overall, our work calls for using more diverse data and bigger networks in reinforcement learning. which now given in other machine learning communities. Though there remains a lot of work to be done to fully take advantage of these two elements, our work offers a glimpse at the potential of large scale multi-task RL. Finally, in line with Agarwal al. (2022). researchers to reproduce our findings as well as study zero-shot generalization and teeeituingg in RL, without the excessive cost of multi-task pretraining from scratch.
would open-source our pretrained models allow

---

Published as a conference paper at ICLR 2023
6 ACKNOWLEDGEMENTS
The authors would like to thank Robert Dadashi, Andrei Rusu and anonymous ICLR's reviewers for providing valuable feedback on an earlier draft of this work.
REFERENCES
Rishabh Agarwal, Marlos C Machado, Pablo Samuel Castro, and Mare G Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. 2021a.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304 29320, 2021b.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Beyond tabula rasa: Reincamating reinforcement learning. Advances in neural information processing systems, 2022.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pp. 507-517. PMLR, 2020.
Mare G Bellemare, Joel Veness, and Michael Bowling. Investigating contingency awareness using atari 2600 games. In Twenty-Sixth AAAI Conference on Artificial Intelligence, 2012.
Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Marc Bellemare, Will Dabney, and Remi Munos. distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference Machine Learning- Volune 70, pp. 449-458, 2017.
Marc G Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C Machado, Sub- hodeep Moitra, Sameera Ponda, and Ziyu Wang. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 588(7836):77-82 2020.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large seale deep reinforcement learning. arXiv preprint arXi:: 2019.
Johan Bjorck, Carla P Gomes, and Kilian Q Weinberger. Towards deeper deep reinforcement learning. arXiv preprint ar S1v:2106.011555 2021.
Diana Borsa, Thore Graepel, and John Shawe-Taylor. Learning shared representations in multi-task reinforcement learning. arXiv preprint arXiv:1603.0200 2016.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Johan Samir Obando Ceron and Pablo Samuel Castro. Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research. In International Conference on Machine Learning, pp. 1373-1383. PMLR, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.03311, 2022.
10

---

Published as a conference paper at ICLR 2023
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pP- 2048-2055 PMLR, 2020a.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. pp. 2048-2056. PMLR, 13-18 Jul 2020b.
Jacob Devlin, Ming-Wor Wiig Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint
2018.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning, pp. 1407-1416. PMLR, 2018.
Lasse Espeholt, Raphaël Marinier, Piotr Stanczyk, Ke Wang, and Marcin Michalski. Seed rl: Scalable and efficient deep-rl with accelerated central inference. arXiv preprint arXiv:1910.06591, 2019.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint
2018.
Jesse Farebrother, Marlos C Machado, and Michael Bowling. Generalization and regularization in dqn. arXiv preprint arXiv:/810.00/23. 2018.
Zhaohan Daniel Guo, Bernardo Avila Pires, Bilal Piot, Jean-Bastien Grill, Florent Altché, Rémi Munos, and Mohammad Gheshlaghi Azar. Bootstrap antent-predictive representations for multitask reinforcement learning. International Conference on Machine Learning, pp. 3875-3886. PMLR, 2020.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729 9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Matteo Hessel, Hubert Soyer. Lasse Espeholt, Wojeiech Czarnecki, Simon Schmitt, and Hado van Hasselt. Multi-task deep reinforcement learning with popart. pp. 3796-3803, 2019.
Matteo Hessel, Manuel Kroiss, Aidan Clark, Iurii Kemaev, John Quan, Thomas Keck, Fabio Viola, and Hado van Hasselt. Podracer architectures for scalable reinforcement learning. arXiv preprint arXiv:2/04.06272 2021.
Maximilian Igl. Gregory Farquhar, Jelena Luketina, Wendelin Bochmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. arXiv preprint 20Xi::2006.08826, 2020.
Dmitry Kalashnkov, Jake Varley, Yevgen Chebotar, Ben Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv, 2021.
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018
Khimya Khetarpal, Zafarali Ahmed, Gheorghe Comanici, David Abel, and Doina Precup. What can do here? theory of affordances reinforcement learning. In International Conference on Machine Learning, pp. 5243-5253. PMLR, 2020.

---

Published as a conference paper at ICLR 2023
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of generalisation in deep reinforcement learning, arXiv preprint urXiv:2111.09794 2021.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022. URL https
//openreview net/
Kuang-Huei Lee, Ofir Nachum, Mengjia Yang. Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, lan Fischer, Eric Jang. Henryk Michalewski, et al. Multi-game decision transformers. arXiv preprini arXiv:2205. 2022.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-1375) 2016.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0 20000000000000.,... 2020.
Yao Lu, Karol Hausman, Yevgen Chebotar, Mengyuan Yan, Eric Jang, Alexander Herzog, Ted Xiao, Alex Irpan, Mohi Khansari, Dmitry Kalashnikov, and Sergey Levine. AW-opt: Learning robotic skills with imitation andreinforcement scale. In 5th Annual Conference Robot Learning, 2021. URL
net, forum? Xa
Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-500 562, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1rXiv:1312.5602, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540))529) -333339993333- 2015.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09333 2020.
Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta learn fast: A new benchmark for generalization rl. arXiv preprint arXiv:1804.03721 2018
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Lue Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning. pp. 16828-16847. PMLR, 2022.
Kei Ota, Devesh K Jha, and Asako Kanezaki. Training larger networks for deep reinforcement learning, arXiv preprint arXiv:2102.07920, 2021.
Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsurprising effectiveness of pre-trained vision models for control. arXiv preprint arXiv:2203.03580. 2022.
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and
transfer reinforcement learning. arXiv preprint 15/ 2015.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06177 2022.
Andrei Rusu, Sebastian Flennerhaga, Dushyant Rao, Razvan Pascanu, and Raia Hadsell. Probing transfer in deep reinforcement learning without task engineering. Conference on Lifelong Learning Agents, 2022.
12

---

Published as a conference paper at ICLR 2023
Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukeuoglu, and Raia Hadsell. Policy distillation. arXiv preprint arXiv:1511.06295 2015.
Tom Schaul, Diana Borsa, Joseph Modayil, and Razvan Pascanu. Ray interference: source of plateaus in deep reinforcement learning. arXiv preprint arXivv:904.111 2019.
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R De- von Hjelm, Philip Bachman, and Aaron C Courville. Pretraining representations for data-efficient reinforcement learning. Advances in Neural Information Processing Systems, 34:12686-1200 12699, 2021.
Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International Conference on Machine Learning, pp. 8583-8592. PMLR, 2020.
Younggyo Seo, Kimin Lee, Stephen L James, and Pieter Abbeel. Reinforcement learning with action- free pre-training from videos. In International Conference on Machine Learning, pp. 19561-19579. PMLR, 2022.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 1134-1141, IEEE, 2018.
Rutav Shah and Vikash Kumar. Rrl: Resnet as representation for reinforcement learning, arXiv preprint arXiv:2107.00 0033000 2021.
David Silver, Julian Schritwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676)3554, 2017.
Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context- based representations. In International Conference on Machine Learning, pp. 9767-9779, PMLR, 2021.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonse et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.046155 2022.
Richard S Sutton and Andrew G Barto. Reinforcemení learning: An introduction. MIT press, 2018.
Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. Advances in neural information processing systems, 30, 2017.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 20/7 IEEE/RSJ international conference intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Ahmed Touati and Yann Ollivier. Learning one representation to optimize all rewards. pp. 13- 23. Curran Associates, Inc., 2021. URL WtR/attpsp//prreeddnng www.oeeeeedddinn............ cc/paper
Paper. .ddf
Frederik Träuble, Andrea Dittadi, Manuel Wuthrich, Felix Widmaier, Peter Vincent Gehler, Ole Winther, Francesco Locatello, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. The
role of pretrained representations for the OOD generalization of RL agents. In International Conference Learning Representations, 2022. htRpsttt::/// ptattps://ppereev. net. forum?
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. 2016.
13

---

Published as a conference paper at ICLR 2023
Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint arXiv: 1812.02644, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Oriol Vinyals, Igor Babuschkin, Junyoung Chung. Michael Mathieu, Max Jaderberg, Wojtek Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, Timo Ewalds, Dan Horgan, Manuel Kroiss, Ivo Danihelka, John Agapiou, Junhyuk Oh, Valentin Dal- ibard, David Choi. Laurent Sifre, Yury Sulsky, Sasha Vezhnevets, James Molloy, Trevor Cai, David Budden, Tom Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Toby Pohlen, Dani Yogatama, Julia Cohen, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lil- licrap, Chris Apps, Koray Kavukcuoglu, Demis Hassabis, and David Silver. AlphaStar: Mastering the Real-Time Strategy Game StarCraft
com.://...cc//lo////// /log/ teal-timee -il--tiimm-stttteeyy strategy game- www.sssatrrrtttaaa 1i/ 2019.
alphastar www.aaarmmattiinnnnn
Zhihui Xie, Zichuan Lin, Junyou Li, Shuai Li, and Deheng Ye. Pretraining in deep reinforcement learning: A survey. arXiv preprint arXiv:2211.03955 2022.
Ruihan Yang. Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. Advances in Neural Information Processing Systems, 33:4767-4777, 2020.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning. pp. 1094-1100. PMLR, 2020.
Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. 2021.
Zhuangdi Zhu, Kaixiang Lin, and Jiayu Zhou. Transfer learning in deep reinforcement learning: A survey. Xrviv preprint htt:::2000007888. 2020.
14

---

Published as a conference paper at ICLR 2023
A RELATED WORK
Pretraining in reinforcement learning: The most common workflow in reinforcement learning is to learn tabula rasa by training a randomly initialized agent. However, this often computationally inefficient, it requires many interactions between the learning agent and the environment and does not make use of knowledge that may be available. The hope of pretraining is to leverage large amounts of training data to accelerate learning on downstream tasks. Recently reinforcement learning practitioners have started to adopt the pretrain then finetune paradigm that has become prevalent in other domains like vision (He et al. 2020 Chen et al., 2020) or NLP (Brown et al.) 2020 Devlin let al. 2018). Due to the large number of reinforcement learning tasks that rely on perception, many works have advocated for using visual representations pretrained on images or videos. A policy or value function is then learned on top the pretrained representation (Sermanet et al., 2018 (Shah &
Kumar 202T Parisi et al., 2022. Seo et al. 2022 Trauble et al. 2022). Another common strategy for pretraining in reinforcement learning is to use an offline dataset of human or agent interactions to learn a policy using offline RL (Levine
al. 2020). The pretrained policy is then fine-tuned
using online RL (Nair et al., 2020 Lu et al., 2021| Kostrikov et al. 2022). Instead of optimizing policy directly other works have focused on learning a world model that can later used by an agent to improve its policy in a sample efficient manner (Sekar et al. 2020 [Seo et al., 2022). Another approach is to learn lask-agnostic features through interactions with an environment in the absence of reward function. These features can be reused in the tinettunig stage when the reward function becomes available to improve sample efficiency (Eysenbach et al., 2018 Schwarzer et al. 2021
Touati & Ollivier 2021). For more detailed overview of the current state of pretraining in RL we refer to Xie et al. 20022)
Current successes in RL have been limited to agents trained to solve a single task and designing generalist agents that can solve multiple tasks at the same time remains an open challenge. A popular approach to multi-task reinforcement learning has been to concurrently learn multiple tasks at the same time (Espeholt et al. 2018 Hessel et al. 2019 Guo al., 2020). Nonetheless, because of negative interference between tasks these models usually perform worse than agents trained on individual tasks. To circumvent those issues previous work have proposed to distill single tasks expert into a multi-task model Parisotto et al. 2015); Rusu et al. (2015): Teh et al. (2017). In the offline setting, recent works have attempted to frame the RL problem as a sequence prediction problem and build on high capacity models based on the Transformer architecture /Vaswani et al. [017). Gato (Reed et al. (222)) and Multi-Game Decision Transformer (Lee et al.) 2022) are multi-task agents trained on expert data that are able to play multiple Atari games. Though one of the ambitions of multi-task reinforcement learning is to take advantage of information from previously learned tasks to learn new tasks faster this is not always observed as negative interference between tasks can impact downstream learning. Yet recent contributions have shown that large scale multi task training on a set of diverse tasks can lead to generalization to new unseen tasks (Kalashnkov et al. 2021
Multi task reinforcement learning:
Lee al. 2022).
B TRAINING PROTOCOL
B.1 HYPERPARAMETERS
In this section, the specific parameter settings that used throughout our experiments given in detail. We kept all the original IMPALA hyperparameters fixed except the optimizer where we used Adam (Kingma & Ba 2014) instead of RMSProp as it was shown be superior in the context of reinforcement learning (Ceron & Castro 2021). We did not do any hyperparameter tuning.
We an efficient implementation of IMPALA Espeholt et al. 2019 Hessel al. 2021) that runs on TPUs. Experiment specific hyperparameters are as follow
The pretraining phase is carried on 32 TPU v3 cores using 6400 actors.
Fine-tuning is done 8 TPU v2 cores using 900 actors.
15

---

Published as a conference paper at ICLR 2023
Table I: Hyperparameters for Atari experiments.
Parameter
Value
Image Width
Image Height
84
Grayscaling
Yes
Action Repetitions
4
Max-pool over last N action repeat frames
Frame Stacking
4
End of episode when life lost
No
Sticky actions
Yes
Action set
Full (18 actions)
Reward Clipping
[-1, 1]
Unroll Length (n)
19
Batch size
128
Discount (?)
0.99
Baseline loss scaling
0.5
Entropy Regularizer
0.01
Adam 31
0.9
Adam B2
0.999
Adam e
1c-8
Learning rate
3e44
Clip global gradient norm
40
B.2 ATARI VARIANTS: GAME MODES AND DIFFICULTIES
We briefly review modes and difficulties of Atari games used throughout the main paper. We refer to the original manuals for more in depth information. Rusu et al. (2022) also provides brief overview of game modes and difficulties for the games BREAKOUT and SPACE INVADERS. To identify individual flavours we use the notation m. dY where X refers to the game mode setting and Y to the difficulty setting:
Space Invaders is shooting game where the player moves laser cannon across the bottom the screen and fires at aliens. The aliens begin on the upper half of the screen and move left and right group. The players wins by eliminating all the aliens. There two difficulty switches and sixteen different game modes leading to a total of 32 flavours. We keep m1d0, mld1 and m9d0 for the test set and keep the other modes for training.
Breakout is game where rows of bricks layered the top half of the screen. Using single ball the player must hit the bricks using the paddle it controls at the bottom of the sereen. However the player let the ball fall below the bottom the screen it looses life. There are two difficulty switches and twelve game modes for total of 24 flavours. We use ml m2d0, m22811 m336d for the test set and use the remaining modes for training.
Asteroids, in this game the player controls ship in space that must destroy asteroids. The ship must avoid getting hit by asteroids, saucers will also appear regularly and try shoot the ship. There are two difficulty switches and 33 game modes for total of 63 flavours. m4d0, ml 1d33 m27d3 saved for the test set while the remaining modes part of the training set.
Bank Heist is maze game similar to Pac-Man where the objective is to rob as many banks as possible while avoiding police cars that appear everytime bank robbed. For this game there are four difficulty switches and eight game modes for total of 32 different flavours. The training set contains all the flavours except mSd3, m20d1, m24d0 that are used for the test set.
https://atariage.. com/
htt.://ww.....oo.ststeoteoe system_ssms..... itw.ttttmmmm.m. tts.spppp2sstteem (0-26006 ter ID-MANUAL
16

---

Published as a conference paper at ICLR 2023
B.3 SETS OF VARIANTS USED FOR PRETRAINING
We use the following subsets of modes during the pretraining phase. For each game we randomly subsampled multiple subsets of different sizes and required that each subset be superset of previously picked smaller subsets to limit the effect of particular variants that might have a larger impact on generalization.
Asteroids: We used the following modes to pretrain on ASTEROIDS
mode: m9d3
2 modes: m9d3, m17d3
4 modes: m9d3, m10d3, m16d0, m17d3
8 modes: m6d3, m9d3, m10d3, m12d0, m16d0, m17d8, m18d0, m25d3
m2d3, m4d3, m5d0, m6d3, m8d0, m9d3, m10d3, m12d0, m16d0, m17d3, m18d0, m20d3, m23d0, m25d3, m28d0, m29d3
16 modes:
m0d3, m1d0, mld3, m2d3, m4d3, m5d0, m6d0, m6d3, m7d0, m7d3, m8d0, m9d3, m10d3, m12d0, m16d0. m16d3, m17d3, m18d0, m20d3, m21d0, m2240, m23d0, m23d3, m24d0, m22d3, m2640, m28d0, m29d0, m29d3, m31d0, m31d3, m128d0
32 modes:
modes: 32 modes + m0000 m2d0, m3d0, m3d3, m5d3, m8d3, m9d0, m10d0, m m11d0, mlld3 m 12d3, m13d0, ml3d3, m14d0, m14d3, ml m15d3, m17d0, m19d0, m19d3, m20d0, m21d3, m22d3, m24d3, m25d0, m26d3, m27d0, m28d3, m30d0, m30d3, m128d3
Space Invaders: We used the following modes to pretrain on SPACE INVADERS
mode: m5d0
2 modes: m5d0, m 12d1
4 modes: m5d0, m8d1, m12d1, m15d0
8 modes:
m5d0, m5d1, m661, m7d1, m8d1, m m10d1, m12d1, m15d0
m0d1, m220, m3d1, m4d0, mAd1, m5d0, m5d1, m660, m6d1, m7d1, m8d1, m10d1, m12d1, 141400 m15d0
15 modes:
m0d0, m0d1, m2d0, m3d0, m3d1, m4d0, m4d1, m5d0, m5d1, m6d0, m6d1, mTdl, m&d1, m9d1, m10011 m11d0, m12d1, m13d1, m14d0, m14d1, m15d0, m15d1
22 modes:
m0d0, m0d1, m2d0, m2d1, m3d0, m3d1, m4d0, m4d1, m5d0, m5d1, m6d0, m6dl, m7d0, m7d1, m&d0, m&d1, m9d1, m1000, m10d1, m11d0, mlldl, m12d0, m12d1, m13d0, m13d1, m14d0, m14d1, m15d0, m15d1
29 modes:
Bank Heist: We used the following modes to pretrain on BANK HEIST
mode: m8d0
4 modes: m&d1, m12d2, m20d0, m28d1
8 modes: m0d2, m0d3, m4d2, m8d1, m&d2, m12d2, m20d0, m28d1
15 modes: m0000, m002, m0d3, m4d2, m4d3, m8d0, m8d1, m882, m1222, m1662, m20d0, m20d2, m24d1, m24d2, m28d1
m0d0, m0d1, m0d2, m0d3, m4d0, m4d2, m4d3, m8d0, m8d1, m8d2, m12d1, m12d2, m12d3, m16d1, m16d2, m16d3, m20d0, m20d2, m24d1, m24d2, m24d3, m28d1
22 modes:
29 modes: m0d0, m0d1, m0d2, m0d3, m4d0, m4d1, m4d2. m4d3, m8d0, m8d1, m8d2, m12d0, m12d1, m12d2, m12d3, m16d0, m1661, m1662, m1 m20d0, m20d2, m20dd3, m24d1, m2242, m24d3, m28d0, m28d1, m28d2, m28d
Breakout: We used the following modes pretrain BREAKOUT
modes: m0d0, m4d0, m4d1, m8d00 m8d1, m20d0, m20d1, m32d1, m36d0, m40d1, m44d1
17

---

Published as a conference paper at ICLR 2023
Asteroids
Impala-Normalizer IQM
Num Pretraining Modes
32
400%
8
63
16
300%
200%
850M
1.9B
2.8B
3.75B
7.5B
15B
Amount of Pretraining Data
Spacelnvaders
Impala-Normalizee IQM
500%
Num Pretraining Modes
22
8
29
400%
15
300%
200%
850M
1.9B
2.8B
3.75B
7.5B
15B
Amount of Pretraining Data
Figure 7: Impact of pretraining data on fine-tuning performance on unseen variants. Error bars show 90% bootstrap Cls. Increasing the number of environment interactions during pretraining is beneficial to fine-tuning performance.
16 modes: m0d0, m0d1, m4d0, m4d1, m8d0, m8d1, m16d0, m16d1, m20d0, m20d1, m28d0, m32d1, m33d0, m40d0, m40d1, m44d1
m0d0, m0d1, m4d0, m4d1, m8d0, m8d1, m12d1, m m16d0, m16d1, m20d0, m20d1, m24d0, m24d1, m28d0, m32d0, m32d1, m36d0, m40d0, m40d1, m44d0, m44d1
21 modes:
C ADDITIONAL RESULTS
C.I INFLUENCE OF PRETRAINING DATA
Farebrother et al. 2018)*s experiments showed that the limited generalization capabilities of the DQN agent they studied is in part due to an overfitting phenomenon. Their agent is pretrained on single task and learns a specialized representation for this task that does not generalize well. They show that the zero shot performance of a DQN agent can increase then decrease during pretraining as the representation becomes more specialized. This is highly undesirable and shows how fragile the representation learned by reinforcement learning agent can be. In this section we investigate whether representations learned using large amounts of pretraining data still overfit to variants encountered during pretraining. To do so we use intermediate checkpoints of the previous pretrained IMPALA models saved 850M, 1.9B, 2.8B, 3.75B and 7.5B frames. They are then fine-tuned for 200 million frames on the test set variants. Results are shown in Figure 7 We observe that fine-tuning performance improves with more pretraining data and we do not observe the overfitting phenomenon. This likely due to the fact that joint pretraining several variants acts as regularizer the representation and prevents over specialization. This is akin to how training on large amounts of diverse data from multiple tasks in supervised learning leads representations that generalize broadly.
18

---

Published as a conference paper at ICLR 2023
Qbert
Seaquest
16000
14000
2500
12000
2000
10000
8000
6000
4000
500
2000
100
150
200
50
100
150
200
Asterix
Ms Pacman
IMPALA
ResNet-18
300000
ResNet-34
4000
ResNet-50
ResNet-101
3000
2000
100000
1000
Figure 8: Training curves over 200 million frames on the games QBERT, SEAQUEST, ASTERIX, Ms PACMAN with different network architectures. As the model of the model increase the return of the algorithm proportionally decreases. This highlights the difficulty of IMPALA to scale with bigger network architectures.
C.2 SCALING WITH DEEP NETWORKS
Figure 8 provides additional evidence that the decrease in performance observed using residual networks without pretraining is not limited ASTEROIDS and SPACE INVADERS. Figure shows evaluation curves on the games QBERT, ASTERIX, SEAQUEST and Ms PACMAN with the IMPALA- CNN and ResNets architectures. results are average over 5 seeds. We observe once again that deeper architectures do not benefit from their increased capacity and perform worse than the smaller IMPALA architecture.
C.3 ADDITIONAL. FIGURES
19

---

Published as a conference paper at ICLR 2023
IQM Impala Normalized Score e
IMPALA IMPALA
IMPALA IMPNA
IMPALA 24
Rendee:15
100%
1000%
0%
500%
0
50
100
150
200
50
100
150
200
(a) ASTEROIDS without pretraining
ASTEROIDS with pretraining on modes
IQM Impala Normalized Score
NPALLA IPALLA
2000%
IMPALA INPALA INPALA Restiet
100%
1000%
0%
0%
50
100
150
200
100
150
200
Number of Frames (in millions)
Number Frames (in millions)
(c) SPACE INVADERS without pretraining
SPACE INVADERS with pretraining 29 modes
Figure 9: Sample efficiency IQM evaluation curves for IMPALA with no pretraining (0 modes) and large number of pretraining modes. Shaded regions show 90% bootstrap confidence intervals. IMPALA x2 and IMPALA x4 achieve higher returns with and without pretraining. However they remain far below the performance of fine-tuned residual networks showing that these networks generalize better.
20

---

Published as a conference paper at ICLR 2023
3000
3130
3540
15.
13
m201
m.300
m301
m440
m4d1
4000
3000
4000
2000
2006
m1500
mm301
mm600
m661
m77d5
30000
190
35600
3000
m771
280
2000
1000
m330d
m3001
m11.00
m1141
m12.00
3000
3300
200
3000
m11500 15d1
IMPALA
ResNet-18
ResNet-34
100
ResNet-50
ResNet-101
Figure 10: Training curves over 15 billions frames during joint multi-task pretraining 29 variants of the game SPACE INVADERS with different network architectures. The IMPALA-CNN architecture does not have enough capacity to do well different tasks during multi task training, the architecture chooses instead to focus on a single task m0d0 and underperform the remaining tasks, This is likely due to a lack of capacity.
21

---

Published as a conference paper at ICLR 2023
3740
1200
33400
7056
4000
4080
D00
12
m540
m600
m643
m700
m:74
13000
4000
3080
1000
15
mildn
m931
m1.043
m12d0
m1.660
17400
4000
6000
4.79
3008
4200
4000
1100
15
m22.0
m2300
m23d4
m2460
m25e51
1000
1.5
m254D
m2800
m22980
m29d3
m31.90
1000
6000
20000
1500
m12860
IMPALA
ResNet-18
6000
ResNet-34
ResNet-50
ResNet-101
Figure 11: Training curves over billions frames during joint multi-task pretraining on variants of the game ASTEROIDS with different network architectures. We observe again that the IMPALA architectures cannot match residual networks in the multi-task regime. Here ResNet- 18 and ResNet- 34 appear to perform much better than the higher capacity model.
22

---

Published as a conference paper at ICLR 2023
IQM Impala Normalized Score
200%
IMPALA
ResNet-18
ResNet-34
ResNet-50
150%
ResNet-101
100%
50%
0%
0
5
10
15
Number of Frames (in billions)
Figure 12: IQM training curves for different network architectures jointly trained for 15 billions frames during multi-task pretraining on 32 variants of the game SPACE INVADERS. Normalization for each variant done using the score achieved by the IMPALA-CNN architecture after multi task pretraining.
IQM Impala Normalized Score
IMPALA
300%
ResNet-18
ResNet-34
ResNet-50
ResNet-101
200%
100%
0%
0
5
10
15
Number of Frames (in billions)
Figure 13: IQM training curves for different network architectures jointly trained for 15 billions frames during multi-task pretraining on 32 variants of the game ASTEROIDS. Normalization for each variant done using the score achieved by the IMPALA-CNN architecture after multi task pretraining.
23


Reviews:
[{"review_id": "bxC1eXFzqh_", "reviewer": "Reviewer_eTGE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an study  on the effect of multi-tasks pretraining in the context of Atari 2600 with deep reinforcement learning agents. It does so by pretraining in multiple variants of 4 games and testing in unseen variants of the game for zero-shot generalization and fine-tuning -for 200 million frames- to see if pre-training helps the agent.", "summary_of_the_review": "The paper presents a large experiment focused on Impala with varied residual nets Atari games variants. However the outcomes of the experiment are all well abundant in existing literature. ", "strength_and_weaknesses": "The paper pursues a computationally challenging and ambitious goal. However I believe it fails short to be considered for publication at ICLR since the contributions and outcomes don't really provide any new information about RL or deep learning. The paper also fails to contrast with their results with relevant literature in generalization. Specifically:\n\n* The fact that agents don't do well in the new variants of the task with zero-shot is not surprising since the variant could mean that an optimal policy in the previous variants don't necessarily translate into the new variant. It would have been good to track from which variants the agent was learning and what is the new thing being asked. For instance if I have a variant in the test set that for the first time requires the agent to shoot to air balloons it is more meaningful to evaluate one/two-shot learning to see how the agent performs in the new task. This would explain the contradiction between the results from this work, were authors suggest that DRL is weak at zero-shot generalisation, while there are plenty of examples [1-4] to name a few, of the opposite. \n\n* That pre-training helps to up to certain point is quite known already. For example e.g. ALOE workshop at ICLR 2022 and other workshops mixing unsupervised learning and reinforcement learning have plenty of works on the topic already [5,6] are examples of it\n\n* That larger networks generalize better and that are able to handle multi-task learning better it is also well known [1,7,8]. Specially relevant for this size of networks are transformers architectures, which aren't part of this work either\n\nIn general I am afraid that this work is too far below the bar of significance and novelty required for ICLR. One possible direction for this work would strength the study to include different kind of networks, e.g. relational networks, transformers. GNNs... and also use some additional benchmarks different to Atari. Current results suggest that there is a moment that pretraining in those variants is not useful anymore, finding an automatic way to find relevant variants / automatic curriculum may be appealing.\n\n[1] Hill, Felix, et al. \"Grounded Language Learning Fast and Slow.\" International Conference on Learning Representations. 2021.\n\n[2] Hill, Felix, et al. \"Environmental drivers of systematicity and generalization in a situated agent.\" International Conference on Learning Representations. 2020.\n\n[3] León, Borja G., Murray Shanahan, and Francesco Belardinelli. \"In a Nutshell, the Human Asked for This: Latent Goals for Following Temporal Specifications.\" International Conference on Learning Representations. 2021.\n\n[4] Vaezipoor, Pashootan, et al. \"Ltl2action: Generalizing ltl instructions for multi-task rl.\" International Conference on Machine Learning. PMLR, 2021.\n\n[5]Seo, Younggyo, et al. \"Reinforcement learning with action-free pre-training from videos.\" International Conference on Machine Learning. PMLR, 2022.\n\n[6] Yi, Mingyang, et al. \"Improved ood generalization via adversarial training and pretraing.\" International Conference on Machine Learning. PMLR, 2021.\n\n[7] Brown, Tom, et al. \"Language models are few-shot learners.\" Advances in neural information processing systems 33 (2020): 1877-1901.\n\n[8] Chan, Stephanie CY, et al. \"Data Distributional Properties Drive Emergent Few-Shot Learning in Transformers.\" arXiv preprint arXiv:2205.05055 (2022).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "ZX1cY-IuoK2", "reviewer": "Reviewer_t4n9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper analyses an RL training regime consisting of multi-task pretraining on a set of atari game variants followed by fine-tuning on an unseen variant. They show that while multi-task pretraining often doesn't help zero-shot performance (in particular scaling training time and number of variants doesn't affect zero-shot transfer performance), it improves the fine-tuning performance and sample efficiency significantly, and these improvements scale as pretraining dataset size and number of variants increases. They further show that in this multi-task pretraining regime, larger models than previously used can provide strong benefits to both pretraining and fine-tuning performance.", "summary_of_the_review": "Overall, I think the paper is worthy of acceptance, and hence I'm recommending an 8. I think the paper's experiments are novel and conclusions interesting and in some cases surprising. In general I'm in favour of more empirical scientific work of this kind, as I think it's something the RL community is lacking.\n\nI think it would be difficult for my score to rise from an 8 to a 10, and it would probably require either: similar experiments on another qualitatively different class of environments to make the results even more robust; the discovery of a robust functional form for fine-tuning performance which holds across environments; or experiments for different algorithm types, to see if these results also generalise to Q-learning based and/or model-based methods.", "strength_and_weaknesses": "Strengths: The questions asked and answered by the paper are interesting and unanswered, and the rigorous experiments provide robust conclusions in this setting, which I'd expect to at least somewhat generalise to settings other than Atari. The paper is well-written and easy to follow, and all the results are clearly presented. I think the results are novel and perhaps counter to some prevailing wisdom in the RL community about the appropriate model size and the value of pretraining. The use of fine-tuning to evaluate the impact of pretraining is good, as it shows conclusions that previously we didn't have.\n\nWeaknesses: There are few weaknesses that I can see. Obviously it would be better if the experiments were demonstrated on more environments, perhaps ones which are different from atari games in some way, but I think the current experiments provide sufficient evidence for the conclusions.\n\nSome suggestions to improve the completeness of the paper: I'd like to see the normalised reward curves during pretraining for the various model sizes, to see if increasing model size also improves pre-training performance as well as fine-tuning performance. Also, given the amount of data gathered on both pretraining dataset size and model size, it would be interesting to see if a functional form for transfer performance similar to https://arxiv.org/abs/2102.01293 can be found, which would allow us to answer questions such as \"how much pretraining data do we need to reduce the amount of fine-tuning data by 5x while maintaining the same performance?\". If the functional form was similar for the different games, then it could provide guidance for how to trade off pretraining and fine-tuning compute.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "YnjmSoRy91", "reviewer": "Reviewer_JzdD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates how model size and pretaining on similar environments/more data affect the zero-shot and fine-tuning generalisation capabilities of RL agents. \n", "summary_of_the_review": "I preliminary choose '6: marginally above', but I am ready to bump the score up if my requests are addressed throughout the discussion period.", "strength_and_weaknesses": "### Strengths\n\n* The paper is a good methodical study of an effect of the amount of data/diff environments and the model size on the generalisation properties of an RL agent.\n* The paper does not invent a flashy algorithm to push some curves higher but does a thorough study of a phenomenon which might be useful for the researchers and practitioners.\n* The effect of number of environments/model size on the fine-tuning performance is important and novel, to the best of my knowledge.\n\n### Weaknesses\n\n* Most of the shown results are known in the community. They have been scattered around different works though, and having them in one place is a plus to the paper in review. However, IMPALA paper has already shown a positive effect of model size on the performance, while the ProcGen paper has shown that training on many seeds (similar to many variants in the current work) improves the zero-shot generalisation performance\n* The paper positions itself as the multi-task learning paper, and I would expect it to have a more methodological 'related work' section pinpointing the main literature on the topic.\n\n### Requests/Questions\n\n* Could you, please, add a separate 'related work' section to the paper? Before the additional page is allowed, I am fine with having it in the appendix to be moved to the main text afterwards.\n* Figure 6 in log-scale looks a bit confusing to me with most of the curves flattened out. Can you, please, replot them without the log scale in the appendix?\n* I think that the paper should be more explicit about the previous works showing similar results: IMPALA’s experiment showing that model size positively affects the performance, and procgen’s number of seeds and model scaling experiments. I don’t think that procgen’s model scaling was discussed in the paper at all. I think these should be added somewhere in the introduction.\n* What do you think the limitations of your paper are? Can you add a paragraph on these to the paper?\n* You mention that prior IMPALA’s multitask implementation uses a separate critic per task. Can you give any pointers to either the paper or specific implementation?\n* “By restricting the different tasks variants of the same game we observe less negative interference between tasks”. Can you provide any evidence for this?\n* Figure 6 shows quite a big jump from no pretraining to 4-mode pretraining. Can I see the same curve, but for 1,2 and 3 modes? Is there a huge qualitative jump from 1 or 2 modes pretraining?\n\n### Nits\n\n* Why do you include 1 to the gamma range in Section 2?\n* You cite Puterman’s book for MDP, but include \\gamma to your MDP tuple. In Puterman’s book, \\gamma is not included to the MDP definition.\n* Can you plot Figure 3 with shared y-axis? I want to see patterns across the games, and, especially, one subplot in log-scale just makes it confusing.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DTxLhQ-OXMM", "reviewer": "Reviewer_9aV5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is an empirical study of the generalization abilities of the deep RL algorithm IMPALA pre-trained on multiple tasks. Pre-training is done on variants of a single game from the Atari benchmark and fine-tuning/testing is done on held-out variants. The authors show that by limiting to variants of a single game, negative interference is reduced enough that the pre-training improves sample complexity (and sometimes final performance as well). However, zero-shot performance does not seem to be affected. In addition, the paper suggests that larger networks, such as those from the ResNet family, can be used in the policy/value function in this multi-task pre-training setting.", "summary_of_the_review": "The paper is well written, the experimental set-up is sound, and the analysis is clear. However, the setting is somewhat limited, and the results for the most part are not significantly novel.", "strength_and_weaknesses": "# Strengths\n- The paper is clear and simple to follow. The authors are clear about the set-up and assumptions.\n- The experimental set-up is sound. In particular, the same number of time-steps is used for fine-tuning as for training a single environment on the ALE benchmark. \n- The paper shows that we can use a ResNet for policy/value function architecture in some cases. \n\n# Weaknesses\n- The settings considered in the paper are fairly limited. Only the IMPALA algorithm is considered, and only variants of the same game are pre-trained on.\n- The finding that pre-training helps is not surprising, given the fact that only variants of the same game are pre-trained on and other variants are tested on.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster