Understanding the Generalization Gap in Visual Reinforcement Learning
Authors: Anurag Ajay
Ge Yang
Ofir Nachum
Pulkit Agrawal
Venue: ICLR 2022 Submitted (2022)
Keywords: Visual Reinforcement Learning
Transfer in Reinforcement Learning
Generalization in Reinforcement Learning

Abstract: Deep Reinforcement Learning (RL) agents have achieved superhuman performance on several video game suites. However, unlike humans, the trained policies fail to transfer between related games or even between different levels of the same game. Recent works have attempted to reduce this generalization gap using ideas such as data augmentation and learning domain invariant features. However, the transfer performance still remains unsatisfactory. In this work, we use procedurally generated video games to empirically investigate several hypotheses to explain the lack of transfer. We also show that simple auxiliary tasks can improve the generalization of policies. Contrary to the belief that policy adaptation to new levels requires full policy finetuning, we find that visual features transfer across levels, and only the parameters, that use these visual features to predict actions, require finetuning. Finally, to inform fruitful avenues for future research, we construct simple oracle methods that close the generalization gap. 

Under review as a conference paper at ICLR 2022
UNDERSTANDING THE GENERALIZATION GAP IN VI- SUAL REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Deep Reinforcement Learning (RL) agents have achieved superhuman performance on several video game suites. However, unlike humans, the trained policies fail to transfer between related games or even between different levels of the same game. Recently several techniques have been proposed close the generalization gap such as data augmentation, domain invariant feature learning, separation of actor and critic networks, etc. However, the transfer performance of RL agents still remains unsatisfactory. In this work, conduct large scale empirical study using procedurally generated video games to understand why the generalization gap still exists. We also show that simple auxiliary tasks improve generalization of policies. Furthemore, contrary to the belief that adaptation to new levels requires finetuning all layers of the policy network, we find that features in the visual trunk can be kept fixed and only the parameters that use visual features to predict actions require finetuning. Finally, to inform fruitful avenues for future research, we construct simple oracle methods that close the generalization gap.
INTRODUCTION
Deep Reinforcement Learning (RL) has achieved tremendous success on several video game suites (Mnih et al., 2013; Vinyals al., 2019; Berner al., 2019). However, otate-of-the-art (SOTA) RL agents are usually trained and tested on the same game. It turns out that the learned policy fails transfer to different games or even unseen levels of the same game (Cobbe et al., 2018; 2019). In contrast, humans are remarkable at transferring to new tasks and environments. Behavior studies (Dubey et al., 2018) show that humans make extensive use of visual priors such as object permanence, sub-goals, and intuitive physics when learning new games. Without these priors, data efficiency of humans drops an order of magnitude (Dubey et al., 2018). (Sax et al., 2018) learns visual priors with supervised learning on datasets with manual annotations of several semantically meaningful features. In practice however, acquiring such datasets challenging.
In the absence of labelled datasets and human supervision, past works have explored three main ways to incorporate visual priors to improve generalization. The first paradigm is data augmenta- tion (Michael et al., 2020; Kostrikov et al., 2020) where invariance in the learned representations is induced by training the agent on a large set of task-irrelevant variations of the observed data. Since the distinction between what is relevant irrelevant is task-dependent, the optimal choice of data augmentation is also task dependent. To avoid manual selection of data augmentations, UCB-DrAC (Raileanu et al., 2020) used upper confidence bound (UCB) algorithm (Auer, 2002) for choosing optimal augmentations. However, it remains unclear whether choosing one out of many task-agnostic data augmentations, such as cropping, helps closing the generalization gap. this paper, we show that set of transformations used in oate-offthheann methods are insufficient at closing the generalization gap. On the other hand, we also show that task-informed data augmentations close the generalization gap, but require task variation information during training. How to construct task sensitive data augmentations remains an open question and an avenue of future research. We discuss this finding in Section 4.
The second paradigm for transfer learning is to leverage domain confusion (Tzeng et al., 2014; 2015; Hoffman et al., 2013), technique commonly used in computer vision to make the source data distribution indistinguishable from the target distribution. The main idea is to discourage learning of spurious domsiif/evel-specfffc features by enforcing the constraint that it should not be possible

---

Under review as a conference paper at ICLR 2022
(a) Jumper Levels
(b) Climber Levels
Figure I: Visualization of different levels (a) Jumper and (b) Climber. both the games, the first and second level differ in theme whereas the first and third level differ in both theme and layout.
predict the identity of the domain/level from the policy features. iDAAC (Raileanu & Fergus, 2021) used this technique to improve policy generalization. However, our investigation reveals that policies that generalize well contain domain/level specifie information. In hindsight, this is not surprising. On image classification tasks, the identity of the object doesnot change based on factors such background clutter, size, etc. However, small changes in the size of obstacles or platforms can drastically change the optimal policy. Therefore, the policy must encode information about level-specific game layout, making application of domain invariance to policy learning challenging. Details of this investigation are provided in Section 5.
The third paradigm of improving generalization involves using auxiliary prediction tasks (such as depth prediction, image reconstruction, contrastive learning) to regularize the policy features to prevent learning spurious features (Jaderberg et al., 2016; Srinivas et al., 2020). In this work, we choose the auxiliary task of inverse model prediction (Agrawal et al., 2016) due to its simplicity. This improves policy generalization. We detail this finding in Section 6.
It is well-known that choice of hyper parameters has substantial impact the performance of RL agents. When it comes to policy generalization, discussions around hyperparameter selection have been limited. We show that state-of-the- art (SOTA) methods that improve policy generalization require careful hyperparameter selection for each task. When these methods are constrained to choose one set of hyperparameters across tasks, they donot outperform the base PPO algorithm. We detail this finding in Section 7.
Finally, we probe if deep RL algorithms learn spurious features that deters performance of the same task new environments. Our experiments reveal that visual features learned by training on a limited set of levels of one game donot require adaption achieve good performance on new levels, Infact, only the layers of the policy network that convert visual features into actions require finetuning. Details of this investigation are provided in Section
In summary, our experiments on Procgen (Cobbe et al., 2019), suite of procedurally generated games reveal that (i) task- tusk-informed data augmentation closes the generalization gap; (ii) learning level- invariant (or domain-invariant) features is not necessary for good generalization; (iii) regularizing the policy features using simple auxiliary tasks can improve generalization; (iv) SOTA methods that improve policy generalization rely on careful hyperparameter selection; (v) training on limited number of levels of the same game doesnot result in learning of sevel-specific spurious features. When operating new levels, good performance can be achieved by only fine-tuning two layers that transform these features into actions.
2 RELATED WORK
Prior works (Cobbe et al., 2018: 2019; Justesen et al., 2018; Zhang et al., 2018a;;; Juliani et al., 2019; Rajeswaran et al., 2017; Raileanu & Rocktäschel, 2020; Grigsby & Qi, 2020; Kuttler et al., 2020; Farebrother et al., 2018) have established the problem of overfitting reinforcement learning. In response, multiple methods have been proposed to mitigate overfitting. (Cobbe et al., 2018; 2019) showed that classical techniques such as dropout (Srivastava et al., 2014), L2 regularization, and batch normalization (Toffe & Szegedy, 2015)- originally developed for supervised learning reduce the generalization gap in RL. Other works (Michael et al., 2020; Cobbe et al., 2018; Ye et al., 2019; Raileanu et al., 2020; Wang al., 2020) have made of data augmentation to learn policies that generalize well. Another popular approach to use representation learning techniques such as variational information bottleneck (Igl et al., 2019) and bisimulation metries (Agarwal al., 2021;
2

---

Under review as a conference paper at ICLR 2022
Zhang et al., 2020) to help with the problem of overfitting in RL. (Cobbe al., 2020; Raileanu & Fergus, 2021) learned a separate network for policy and value function and showed that helps reduce the generalization gap in RL. (Sax et al., 2018) showed that using mid-level visual features optimized for segmentation, depth, keypoints, surface normal prediction and others improve transfer. In our work, rather than proposing novel approach to solve overfitting, we analyze the limitations of the current techniques and propose avanues for improvements.
PRELIMINARIES
We consider distribution of Partially Observable Markov Decision Process (POMDPs) p(M) such that M, = (S,, O,, A,, Ti, S2, RR, 7) p(M). We can think of different M, as instances of same task in different environments. S, the state space, O₂ is the observation space, A, is the action space, (s 8s a) is the transition function, (o'|s,,, |s, a) is the observation function, (s, a) is the reward function and 7 is the discount factor. The goal is to find a policy Ty that maximizes the expected sum of discounted rewards over the distribution of POMDPs, (TT) = Px.M(M) - - 7 (st, at)
During training, we only have access to a limited number of POMDPs . = {M,)) Our goal is to find policy To that generalizes to new POMDPs sampled from p(M).
3.1 ENVIRONMENT SETUP
We conduct experiments using the ProcGen suite (Cobbe et al., 2019), collection of 16 procedurally generated video games. Each game contains multiple levels wherein the agent needs to perform the same task, but the environment varies due to changes in position of objects, their textures and the background texture. We refer to variation in levels resulting solely from changes in texture of object, theme variation. We collectively refer variation in levels resulting
agent and the background from changes in positions of the objects as layout variation. Figure illustrates theme and layout variation for two games, Jumper and Climber. For each game, the first and the second level differ in theme whereas the first and the third level differ in both the theme and the layout.
3.2 METHODS
We use standard neural network architecture to represent the policy, a ~ To(als), consisting of: a visual encoder क, =s))(s) and a policy head a w0, (a|z) with parameters 01, 0g respectively. Collectively, they are referred as 0 [01, 0g]. The architecture of the visual encoder is borrowed from IMPALA (Espeholt et al., 2018) and is a ResNet with 15 convolutional layers. It outputs flattened feature map (z) that we refer to as the visual feature. These features are passed into a two 2-layer fully-comn connected neural networks with 256 hidden units for predicting the actions (i.e., the policy head) and the value function (i.e., the value head, (z)).
We analyze the generalization performance of PPO (Schulman et al., 2017) and three state-of-the-arr (SOTA) methods:
UCB-DrAC (Raileanu et al., 2020) uses Upper Confidence Bound (UCB) to automatically select game specific data augmentations out of eight possible ones: crop, grayscale, cutout, culout-color, flip, rotate, random convolution and color jilter. For brevity, we sometimes refer to UCB-DrAC as simply DrAC.
DAAC (Raileanu & Fergus, 2021) trains two separate networks for representing the policy and the value function. The intuition is that in POMDPs, the value network will overfit, but the policy can still generalize. The seperation of networks prevents interference in visual features learned for representing the value function and the policy. PPG (Cobbe et al., 2020) also trains separate networks for actor and critic but mainly focuses on improved training of policies and not their generalization performance.
iDAAC (Raileanu & Fergus, 2021) adds an auxiliary loss to the training objective of DAAC to encourage domain invariance.
3

---

Under review as a conference paper at ICLR 2022
30
Reward
20-
DICC
10
DrAC
PPO PPO-Exed train
PPO-Eaed eva.
167
Steps
167
167
167
(a) BigFish
(b) StarPilot
(c) FruitBot
(d) BossFight
Reward
Staps
Steps
Stegs
Steps
Steps
(e) Ninja
(l) Plunder
(g) CaveFlyer
(h) CoinRun
Jumper
Roward
Steps
le7
Steps
Steps
Steps
Steps
1e7
(j) Chaser
(k) Climber
(I) DodgeBall
(m) Heist
(n) Leaper
0.75
Reward
10
Narmalizdd
6.50
DrAC DT
DeAC TR
0.25
DeVC
0.00
Steps
Steps
(0) Maze
(p) Miner
(q) Mean Normalized Score
Figure 2: UCB-DrAC with theme randomization (DrAC-TR) gets close to the test performance of DrAC-FT on games with fixed themes indicating that learned theme invariant features. Thus, outperforms UCB-DrAC implying the need for better domain randomization and data augmentation. Black horizontal lines are the performance of a PPO baseline trained with fixed texture. Dashed line is training performance.
We use PPO's pytorch implementation (Kostrikov, 2018), the official implementation for UCB- DrAC (Raileanu et al., 2020), and the official implementation for DAACIDAAC (Raileanu Fergus, 2021), See Appendix B for the hyperparameters used.
3.3 EVALUATION METRICS
Different procgen games have different score ranges which makes it difficult for to average the scores across games. Recent works (Raileanu et al., 2020; Raileanu & Fergus, 2021) used percentage games. However, this metric doesn't 't properly take
improvement over PPO for averaging reward range of a game into account and is also biased by PPO's performance on that game, thereby being more susceptible large variations. To remedy this, suggested in (Cobbe et al., 2019), R-Rmin_ use the normalized score Rmas - Russ to compute mean scores (MNS) across games. Here, Rmoo and Rmin are game-dependent constants provided in (Cobbe et al., 2019). To calculate the final train and test performances of a policy, we evaluate the policy 100 times on train levels (1-200) and test levels for each game. We report the mean and standard deviation of final train and test performances computed across different seeds. We train the policy on the first two hundred levels and evaluate subset of remaining levels (i.e. approximately two million unique levels).
4

---

Under review as a conference paper at ICLR 2022
Table Performance of level classifier trained on the encoder features of policies learned with PPO on 200 levels, with PPO on 100k levels, with DAAC on 200 levels, and with iDAAC on 200 levels. averaged across all procgen games.
PPO (200)
PPO (100k)
DAAC
iDAAC
Train
0.95 + 0.1
0.96 + 0.1
0.95 0.1
0.94
Test
0.91 + 0.16
0.87
0.89 + 0.12
0.87
4 INVESTIGATING POLICY GENERALIZATION WITH DATA AUGMENTATION
A policy can fail to generalize to new level due to inability to deal with either: (i) layout variation or (ii) theme variation. To tackle the second issue, prior work (Raileanu et al., 2020; Michael et al., 2020) has employed data augmentation. If these methods then their performance on test-levels should be same as a policy that is trained and evaluated on ProcGen games with fixed themes. To test if this is true, data augmentation method on ProcGen games with fixed themes. We refer this to oracle method as DrAC+FT, where FT stands for fixed theme. Results in Figure show significant test performance difference between UCB-DrAC and DrAC+FT which suggests that UCB-DrAS does not achieve theme invariance. Moreover, as shown in Figure 3, UCB-DrAC does not outperform a well-tuned PPO baseline when averaged across all the games.
successful at achieving theme in mnrvaiiaeee,
trained UCB-DrAC,
of-te-of-the-art
The above results raise question: is possible to close the generalization gap by adopting better data augmentation scheme? To answer this, we constructed game levels that had fixed layout, but varied only in theme. We refer to a PPO agent trained on this baseline as DrAC-TR for theme randomization. Our intent is to use this domain randomization scheme, that makes use of privileged access to the game engine, as a proxy of better data augmentation schemes. Figure shows that DrAC+TR greatly improves over UCB-DrA' and gets closer to the test performance of DrAC+FT. Our result demonstrates the viability of pushing for stronger data augmentation schemes for producing theme invariance and thus policy generalization. We hope to encourage the development towards this venue of research with our reported findings.
5 INVESTIGATING POLICY GENERALIZATION WITH DOMAIN CONFUSION
Another way to learn generalizable policies is by using domain confusion (Tzeng et al., 2014; 2015; Hoffman et al., 2013) which discourages policies from predicting task irrelevant properties of the environment, thereby encouraging them to focus on task relevant properties. In addition to separating the policy and the value network, (Raileanu & Fergus, 2021) used this technique to prevent the policy from learning level-specific features and obtained SOTA results. Given these findings, we might think that we should make policies level invariant to obtain better generalization. However, level (in a game) is not only defined by its theme but also by its layout which contains task relevant information. Therefore, learning level-invariant policies might remove layout features important for solving the task and thus hurt the performance of these policies. Hence, it's not clear the policies, that generalize well, are independent of level-specific features.
To answer this, we first take the policies trained by PPO on 100k levels, by DAAC on 200 levels and by iDAAC on 200 levels on different games of procgen. All these policies have good generalization capabilities. We want to test if the visual features coming from these policies contain level-specific features. Therefore, we collect 1 trajectory with random exploration for each level from 200 (per game) and label each state in the trajectory with the corresponding level id. We then mix all the (state, level id) tuples in the collected trajectories and call the resulting dataset DD The first 80% of 'D is labelled as train set and the remaining as the test set. For each of the policies described above, we use the train set to learn a linear level classifier on top their visual features and evaluate the performance of the learned classifier the test set. Table shows that each of the classifier learned obtained a good train and test performance averaged across all games. We further provide the performance of these classifiers on individual games in the Appendix. This implies that all the above mentioned policies, with good generalization capabilities, contain level-specific features. To further bolster our claim, we use the train set to learn linear classifier top of visual features of
5

---

Under review as a conference paper at ICLR 2022
10
Reward
PPO Inv
DAAC
10-
DAAC
PRO-fxed train
0
Stops
Steps
Stege
Steps
PRO-laed eval
(a) BigFish
(b) StarPilot
(c) FruitBot
(d) BossFight
Reward
1.5
7.5
5.-
Steps
Steps
Steps
107
Steps
Steps
(c) Ninja
(f) Plunder
(g) CaveFlyer
(h) CoinRun
Jumper
7.5
1.
Rowird
5.0-
1.5
Steps
Steps
Steps
Steps
Steps
Chaser
(k) Climber
(I) DodgeBall
(m) Heist
(n) Leaper
10
Normelized Seore
Reward
P90 ins
6:25
DAAC
IDAAC
Steps
Steps
le7
Steps
107
(0) Maze
(p) Miner
(q) Mean Normalized Score
Figure 3: PPO with inverse model regularization (PPO inv m) is competitive (in terms of test performance) against UCB-DrAC, DAAC and iDAAC in the ProcGen domain despite being a much simpler method. Black horizontal lines are the test performance of a PPO baseline. Dashed line is training performance.
policies trained by PPO on 200 levels and evaluate performance on the test set. We see that its performance is similar to that of the previous classifiers. This shows that policies have level specific features regardless whether it generalizes or not.
DAAC/DAAC (Raileanu & Fergus, 2021) performs well due to less overfitting a result of separation of policy and value networks and the regularization of policy features from auxillary losses. However, the insight that these methods don't learn level-specific features and discouraging the policies to learn level-specific features leads to better generalization, isn't empirically grounded.
INVESTIGATING POLICY GENERALIZATION WITH AUXILIARY TASKS
Data augmentation and domain confusion can be thought of as two separate methods of constraining the learned visual features to potentially improve generalization. A third way to constrain features is to optimize them for auxiliary objectives in addition the policy and value prediction losses. (Jaderberg et al., 2016) showed that depth prediction improves both data efficiency and the generalization of learned policies on first-person navigation tasks. However, depth an additional sensory modality that is not always available.
Therefore, we propose a simpler auxiliary task of encouraging the visual features coming from two consecutive states to predict the corresponding action. This inverse model regularization has been shown to be an effective regularization technique prior works (Agrawal et al., 2016; Pathak et al., 2017; 2018). Let 2 m. (st). ). If
is the inverse model and (st, 4t 8t+1) is the state action state

---

Under review as a conference paper at ICLR 2022
Nomullezd Score
1.00
No Score
0.75
P20
P70
0,50
DAC
0.25
DAAC
IDAN
Steps
Steps
(a)
(b)
Figure We compare PPO, PPO with inverse model regularization (PPO inv m), UCB-DrAC, DAAC, and iDAAC on test levels (a) when using different hyperparameters for each game and (b) when using a single set of hyperparameters across all games
tuple, then this regularization minimizes CE(a,, (Z 2+4-1)) where CE is the cross-entropy loss. Figure shows that this regularization is competitive with UCB-DrAC in terms of test performance despite heing simpler.
EFFECTS OF HYPERPARAMETER SELECTION ON POLICY GENERALIZATION
Until now we have investigated several methods for learning policies that generalize across levels. However, with exception of UCB-DrAC, we found that all the other methods (PPO inv m, DAAC, iDAAC) require careful hyperparameter selection to improve over the base PPO algorithm. To quantify the impact of careful hyperparameter selection, we chose single set of hyperparameters for each method that maximizes the mean normalized score (MNS) evaluated on validation levels 201 400. We then compare these policies on the test levels. Figure 4 shows that average test performance across games of all methods is similar to PPO. Additional details about hyperparameter selection are provided in Appendix B. These results suggest that improvements over PPO heavily depends on careful hyperparameter selection on each game. One could argue that different games require different hyperparameters due to their unique properties. However, careful hyperparameter tuning on each game is compute-intensive, and there's no guidance on how to set these hyper- parameters other than performing a grid search. Hence, automated tuning of hyperparameter to save compute costs is an important avenue for future research.
8 INVESTIGATING POLICY ADAPTATION TO NEW LEVELS
Until now we have investigated zero-shot evaluation of policies on test levels. The other possibility is finetune the policies on new levels.
First, we train a policy on limited number of levels (i.e., levels 1 - 200).. However, rather than directly evaluating the trained policy, we finetune the trained policy on new levels (i.e., levels 201 400) to achieve the train-time performance quickly. In this setting, it is natural to wonder if finetuning of the entire policy is required to achieve the train-time performance on new levels. To answer this question, we fix the visual encoder Th of policy trained using PPO on levels 1 - 200 of game and finetune only the policy head T/ (as well the value head Vv2) on levels 201 400 of the game. If finetuning the entire policy on levels 201 - 400 is required to achieve the train-time performance, then finetuning only the policy head on those levels should lead a sub-optimal performance.
Figure 5 shows that the average returns of the policy on levels 201 - 400 become similar to the average returns of the policy on levels 1 200 after finetuning the policy head. Furthermore, finetuning the entire policy on levels 201 - 400 levels gives a similar average return. These observations imply that finetuning only the policy head (on levels 201 400) sufficient to recover the average return of levels 1 200. Additionally, we see that end-to-end training of policy from scratch on levels 201 400 is less sample-efficient than finetuning the policy head as well as finetuning the entire policy. Moreover, training of policy head with randomly initialized visual encoder as well as imagenet (Krizhevsky et al., 2012) trained visual encoder on levels 201 400 leads to poor average returns. These observations show that the visual encoder features learned from levels 1 200 are meaningful and help finetune both the policy head and the entire policy. Overall, this implics that the visual features learned on training levels (1 200) meaningful and can be kept fixed when
7

---

Under review as a conference paper at ICLR 2022
10
Reward
30
20
Head only
Pull
10-
From Scraich
10
Head (Randor))
Head (InugeNer)
147
Steps
1x7
107
[7-200]
(a) BigFish
(b) StarPilot
(c) FruitBot
(d) BossFight
16
Rewar
Steps
Steps
tex
Steps
Steps
Steps
107
(e) Ninja
(f) Plunder
(g) CaveFlyer
(h) CoinRun
ii) Jumper
10.0
Reound
5.D-
25
Steps
Steps
Steps
1e7
Steps
Steps
i Chaser
(k) Climber
(1) DodgeBall
(m) Heist
(n) Leaper
Noemalized Score
1..
Full
19
Hest
Reward
0.c
Heal (Randon)
(EmageNet)
0.
[1-200
00
0
Steps
127
Steps
Steps
107
(0) Maze
(p) Miner
(q) Mean Normalized Score
Figure 5: Transfer: We compare fin-tuning the policy head (Head-only) versus the entire policy network (Full) on new levels (201 - 400). The policy is trained on 200 levels. Fine tuning just the policy head allows us to recover this performance on the training levels (I1 - 200J), showing that the visual features only require small number of training levels to generalize. We additionally include the learning curve of a policy trained from scratch (From Scratch), a policy head trained on randomly initialized visual encoder (Head (Random)), and a policy head trained imagenet pretrained visual encoder (Head (ImageNet)) to show the importance using learned visual features.
Normiheed Score
100
1.00
1,50
1.00
0.55
0.75
Haudorly
0.30
0.50
0.50
0.50
From Scrach
Head (Randor))
0.00
0.00
0.00
0.00
Head (ImageNee))
Steps
Steps
Steps
Steps
[1-200]
(a) 200 Levels
(b) 1k Levels
(c) 10k Levels
(d) 100k Levels
Figure 6: We compare fine-tuning the policy head (Head-only) versus the entire policy network (Full). The policy is trained on 200 training levels ([1 200J), and fine-tuned on (a) 200, (b) 1k, (c) 10k, and (d) 100k new test levels. We additionally include the learning curve a policy trained from scratch (From Scratch), policy head trained on a randomly initialized visual encoder (Head (Random)), and a policy head trained on a imagenet pretrained visual encoder (Head (ImageNet)) to show the ability for the visual features to transfer. The plots show the normalized reward averaged across all games (see results on individual games in Appendix C).
8

---

Under review as a conference paper at ICLR 2022
adapting new levels (201 400). Instead, it is the policy head feature that fails to transfer to new levels and requires further finetuning.
However, the strategy of training a policy training levels (1 200) and then finetuning the policy head on new levels has limitations. As the number of new levels increases, it becomes harder to finetune the policy head those levels and recover the average policy returns of the training levels. From Figure 6, we can see that finetuning the policy head on 200 and 1k new levels recovers the average policy returns of the training levels. But, it fails to do the same for 10k and 100k new levels.
DISCUSSION
How to visual features? Most of the work visual transfer in RL has focused on learning generalizable visual features. However, another related problem setting is policy adaptation to new levels. Section 8 shows that visual features learned by PPO can be kept fixed when adapting to new levels but the policy head, tasked to combine visual features to predict actions, requires finetuning on new levels. This suggests fruitful avenue for future work: instead of learning better visual features, focus on how to leverage the learned features to solve a new task.
Towards better data augmentation. UCB-DrAC performs similar to PPO baseline when averaged across all the games and doesn't I fully address the problem of learning invariant features. Specifically, is significantly worse as compared to training PPO with fixed texture. This indicates that better data augmentation techniques are required to make the policy invariant to irrelevant visual appearances. We show that one way to overcome this challenge by combining UCB-DrAC with theme randomization. However, theme randomization requires task-specific knowledge of how the theme varies across different variants of the task. In general, such knowledge may not available to the agents. An exciting area of future investigation is to develop data augmentation method that can automatically discover task-relevant augmentations.
Further investigation of auxiliary losses. We show PPO, with inverse model regularization, signifi- cantly improves upon PPO despite being a simple change. Similarly, iDAAC made use of auxiliary losses to focus task-relevant features of the environment. This indicates the need for further investigation into auxiliary losses for better policy generalization.
Towards compute-efficient hyperparameter tuning. We show that state-of-the-att (SOTA) meth- ods that improve policy generalization heavily rely on careful hyperparameter selection for each game. However, hyperparameter tuning via grid search each game is compute-intensive. Therefore, automatically tuning hyperparameters in a compute-efficient manner is fruitful avenue for future works.
REFERENCES
Rishabh Agarwal, Marlos C. Machado, P. S. Castro, and Marc G. Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. 2021.
Pulkit Agrawal, Ashvin Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. arXiv preprint arXiv:1606.0744 2016.
Peter Auer. Using confidence bounds for exploitation-exploratoo trade-offs. Journal of Machine Learning Research, 3(Nov):397-422, 2002.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv: 2019.
K. Cobbe, J. Hilton, O. Klimov, and John Schulman. Phasic policy gradient. ArXiv, Ab4/2009.04416, 2020.
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. arXiv preprint arXiv: 18/2.0234, 2018.
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. arXiv preprint arXiv: 1912,01588 2019.

---

Under review as a conference paper at ICLR 2022
Rachit Dubey, Pulkit Agrawal, Deepak Pathak, Thomas L Griffiths, and Alexei A Efros. Investigating human priors for playing video games. arXiv preprint arXiv: 182.111802..127,,, 2018.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint 1rxiv: 2018.
Jesse Farebrother, Marlos C. Machado, and Michael H. Bowling. Generalization and regularization in dqn. ArXiv, abs/1 8 10.00123, 2018.
Jake Grigsby and Yanjun Qi. Measuring visual generalization in continuous control from pixels. ArXiv, abs/2010.06740, 2020.
Judy Hoffman, Erik Rodner, Jeff Donahue, Trevor Darrell, and Kate Saenko. Efficient learning of domain-invariant image representations. arXiv preprint arXiv:1301.322 2013.
Maximilian Igl, Kamil Ciosek, Yingzhen Li, Sebastian Tschiatschek, Cheng Zhang, Sam Devlin, and Katja Hofmann. Generalization in reinforcement learning with selective noise injection and information bottleneck. In Advances in Neural Information Processing Systems, pp. 13956-13968, 2019.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ArXiv, abs/ 500.03167, 2015..
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv: 1. .03397, 2016.
Arthur Juliani, Ahmed Khalifa, Vincent-Piiee -Petre Berges, J. Harper, Hunter Henry, Adam Crespi, J. Togelius, and D. Lange. Obstacle tower: A generalization challenge in vision, control, and planning. ArXiv, abs/1902.01378, 2019.
Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi. Illuminating generalization in deep reinforcement learning through procedural level generation. arXiv preprint 1806. 2018.
Ilya Kostrikov. Pytorch implementations of reinforcement learning algorithms. httm://kkstrkko/pyeeeee a2c ppo acckr-ggall. 2018
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 13649, 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Heinrich Kuttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. ArXiv, abs/2006.13760, 2020.
Laskin Michael, Lee Kimin, Stooke Adam, Pinto Lerrel, Abbeel Pieter, and Srinivas Aravind. Reinforcement learning with augmented data. ArXiv, abs/2004. 14990, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 11Xiv:13/2.5602, 2013.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning, pp. 2778-2787 PMLR, R, 2017.
Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A Efros, and Trevor Darrell. Zero-shot visual imitation. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 2050-2053 2018
10

---

Under review as a conference paper at ICLR 2022
Roberta Raileanu and Rob Fergus. Decoupling value and policy for generalization in reinforcement learning. arXiv preprint arXiv:2102.10330, 2021.
Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for procedurally- generated environments. ArXiv, abs/2002.12292. 2020.
Roberta Raileanu, M. Goldstein, Denis Yarats, Ilya Kostrikov, and R. Fergus. Automatic data augmentation for generalization in deep reinforcement learning. ArXiv, abs/2006. 12862, 2020.
Aravind Rajeswaran, Kendall Lowrey, Emanuel Todorov, and Sham M. Kakade. Towards generaliza- tion and simplicity in continuous control. ArXiv, abs/ 0bb//703300260, 2017.
Alexander Sax, Bradley Emi, Amir R Zamir, Leonidas Guibas, Silvio Savarese, and Jitendra Ma- lik. Mid-level visual representations improve generalization and sample efficiency for learning visuomotor policies. arXiv preprint arXiv:1812.11977. 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 170770637777 2017.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. ArXiv, abs/2004.04136, 2020.
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting, J. Mach. Learn. Res., 15: 1929-1958, 2014.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell, Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv: 1412.3444 2014.
Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. Simultancous deep transfer across domains and tasks. In Proceedings of the IEEE international conference on computer vision, pp. 4068 80764 2015.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-354 2019.
K. Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. Improving generalization reinforcement learning with mixture regularization. ArXiv, ttos/20101081..0 2020.
Mang Ye, Xu Zhang, Pong C. Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. 2019 JEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6203-6212, 2019.
Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. ArXiv, abs/ abss/880.079955 07997, 201 8a.
Amy Zhang, Clare Lyle, Shagun Sodhani, Angelos Filos, Marta Kwiatkowska, Joelle Pineau, Yarin Gal, and Doina Precup. Invariant causal prediction for block mdps. arXiv preprint 0000:20003.0606,, 2020.
Chiyuan Zhang, Oriol Vinyals, Rémi Munos, and Samy Bengio. A study overfitting in deep reinforcement learning. ArXiv, abs/1 818.0089333.. 2018b.
11

---

Under review as a conference paper at ICLR 2022
A PPO WITH INVERSE MODEL REGULARIZATION
PPO (Schulman et al., 2017) is an actor-critic algorithm that alternates between sampling data with environment interaction and optimizing objective function with stochastic gradient descent. Let the policy being optimizes be тhe and the associated value function be Ve.v- PPO maximizes
Jpro = Jna - Jv + a2S
where ST, is the entropy for aiding exploration,
S.o EsETeN
Σ
log
aEA
Jy is the value function loss,
Jv Egen (VVg.(s) Ve
and Jug the policy objective term
Jz. = (s Anin(roA,clip(rar. 1 c, + c)
Here, rg
is the importance weight for estimating the advantage function and À Ae... as is estimated advantage function for Te.. using returns and value function (s)
(sls)
To [a]s)
We make use of inverse models to regularize policy features for improved performance and better generalization. Let 2, TT (st). If fa is the inverse model and (s,, aq st+1) is the state action state tuple coming from Taa then this regularization minimizes Jnn = (E (Z z++1) where CE is the cross- cross-entropy loss. Overall, PPO with inverse model regularization maximizes
JpPo Jr, Jy 02 Oinv Jin
where Oiiw is the regularization constant associated with inverse model regurlarization loss.
B HYPERPARAMETERS
We use the default hyperparameters from PPO (Schulman al., 2017). UCB-DrAC (Raileanu et al., 2020) and DAACEDAAC (Raileanu & Fergus, 2021). DrAC-FT and DrAC-TR uses same hyperparameters as UCB-DrAC (DrAC). PPO with inverse model regularization only introduces an extra hyperparameter Ciiv- We write all the hyperparameters in table and 3. While table 2 describes the best single set of hyperparameters for different methods selected using validation levels 201 400, table 3 details game-specific changes to some of those hyperparameters for improved performance.
C PROCGEN RESULTS
Table 4 and describe scores on train levels (1-200) and test levels across all procgen games for PPO, UCB-DrAC, DrAC-FT (UCB-DrAS with fixed theme games), DrAC-TR (UCB-DrAC with randomized theme games). DAAC, iDAAC, and PPO inv-m (PPO with inverse model regularization). Figure places games next the RL algorithm excluded as they perform the best on two or more RL algorithm. Figures 8, 9, and 10 show the learning curves for finetuning the policy head as well as the full policy on 1k, 10k and 100k new levels respectively. The policy was originally trained on levels 1-200 with PPO. End-to-end training of policy and training of policy head with randomly initialized visual encoder on new levels are also included as baselines. Finally, table 6 and 7 give train and test accuracy of level classifier trained on the encoder features of policies learned with PPO on 200 levels, with PPO on 100k levels, with DAAC on 200 levels, and with iDAAC on 200 levels on all procgen games.
which they perform the best. Some games are
12

---

Under review as a conference paper at ICLR 2022
DAAGADACC
Biglish
Boesfight
PPO
Cavellyer
Leaper
Miner
PIOEZI
T
UCB-DrAC
Dodgebeil
Made
Figure 7: Visualization of games placed next to the RL algorithm on which they perform the best. Some games excluded as they perform the best on two more RL algorithms.
13

---

Under review as a conference paper at ICLR 2022
10
Head-oolly
Rewand
20
700
10
From Sernich
2.5
Heal (Semich)
Head (ImageNer)
Stops
Staps
165
Stegs
Steps
[1-200]
(a) BigFish
(b) StarPilot
(c) FruitBot
(d) BossFight
Reward
Steps
Steps
165
Steps
Steps
Steps
16t
(c) Ninja
(f) Plunder
(g) CaveFlyer
(h) CoinRun
Jumper
Reward
Steps
Steps
165
Steps
Steps
Steps
( Chaser
(k) Climber
(I) DodgeBall
(m) Heist
(n) Leaper
Full
Nonnelized State
Head-nly
From Senoch
Roward
0.c
(Scnsch)
(IragsNar)
0.2
[1-200]
0.
Steps
167
167
167
(0) Maze
(p) Miner
(q) Mean Normalized Score
Figure 8: Transfer: We compare tie-ttnnig the policy head (Head-only) versus the entire policy network (Full) on 1K new levels. The policy is trained 200 levels. Fine tuning just the policy head allows us to recover this performance on the training levels ([1 200]). We additionally include the learning curve of a policy trained from scratch (From Scratch), a policy head trained randomly initialized visual encoder (Head (Random)), and policy head trained on imagenet pretrained visual encoder (Head (ImageNet)) to show the importance of using learned visual features.
14

---

Under review as a conference paper at ICLR 2022
30
Head only
Rewand
70
10
From Sernich
2.-
Heal (Semich)
[ead (ImageNer)
Stops
Staps
165
Stegs
Steps
[1-200]
(a) BigFish
(b) StarPilot
(c) FruitBot
(d) BossFight
Reward
Steps
Steps
165
Steps
Steps
Steps
16t
(c) Ninja
(f) Plunder
(g) CaveFlyer
(h) CoinRun
Jumper
Reward
7.5
3.0
5.5
Steps
Steps
Steps
Steps
Steps
i Chaser
(k) Climber
(I) DodgeBall
(m) Heist
(n) Leaper
Full
Nornelized Seate
Head-nly
From Senoch
Roward
(Scnsch)
(IragsNar)
[1-200]
0.0
Steps
167
167
107
(0) Maze
(p) Miner
(q) Mean Normalized Score
Figure 9: Transfer: We compare fine-tumin tie-ttnnng the policy head (Head-only) versus the entire policy network (Full) on 10k new levels. The policy is trained 200 levels. Fine tuning only the policy head fails to recover the performance on the training levels. We additionally include the learning curve of policy trained from scratch (From Scratch), policy head trained randomly initialized visual encoder (Head (Random)), and policy head trained on a imagenet pretrained visual encoder (Head (ImageNet)) to show the importance of using learned visual features.
15

---

Under review as a conference paper at ICLR 2022
30-
Rewand
Head-orlly
20
5.0-
10
From Sernich
2.-
Heal (Semich)
Head (ImageNer)
Stops
Staps
165
Stegs
Steps
[1-200]
(a) BigFish
(b) StarPilot
(c) FruitBot
(d) BossFight
Reward
Steps
Steps
165
Steps
Steps
Steps
16t
(c) Ninja
(f) Plunder
(g) CaveFlyer
(h) CoinRun
Jumper
Reward
7.5
30
5.0
255
Steps
Stegs
Steps
Steps
Steps
i Chaser
(k) Climber
(I) DodgeBall
(m) Heist
(n) Leaper
Full
Nornelized Seate
Head-nly
From Senoch
Roward
0.c
(Scnsch)
(IragsNar)
[1-200]
0.0
Steps
167
107
167
(0) Maze
(p) Miner
(q) Mean Normalized Score
Figure 10: Transfer: We compare fine-tuning the policy head (Head-only) versus the entire policy network (Full) 100k new levels. The policy is trained on 200 levels. Fine tuning only the policy head fails to recover the performance on the training levels. We additionally include the learning curve of policy trained from scratch (From Scratch), policy head trained randomly initialized visual encoder (Head (Random)), and policy head trained on a imagenet pretrained visual encoder (Head (ImageNet)) to show the importance of using learned visual features.
16

---

Under review as a conference paper at ICLR 2022
Table 2: Best single of hyperparameters for PPO, UCB-DrAC, DAAC and iDAAC selected using
validation levels 201 400.
Hyperparameter
Value
0.999
0.95
# timesteps per rollout
256
# epochs per rollout # minibatches per epoch entropy bonus (o2) value loss coefficient (a1)
3
8
0.01
0.01
clip range reward normalization
0.2
yes
learning rate
5e-4
workers # environments worker # total timesteps optimizer
1
64
25M
Adam
LSTM
no
frame stack
Oiizv
0.4
UCB augmentation coefficient (a..)
0.1
UCB exploration coefficient (c)
0.1
UCB window length (K)
10
DAACADAAC actor epochs per rollout
1
DAACADAAC value epochs per rollout (o.)
9
# DAACADAAC value update frequency (vy)
1
DAACIDAAC advantage loss coefficient (a.)
0.25
iDAAC order loss coefficient (ar)
0.001
Table 3: Game-specific changes to some hyperparameters for improved performance
Hyperparameter
V
1ff
Oa
Oigv
plunder
1
8
0.3
0.1
0.4
chaser
1
1
0.15
0.001
0.1
miner
9
32
0.25
climber
9
0.05
0.001
0.01
bigfish
32
0.05
0.01
0.1
dodgeball
9
32
0.25
0.001
0.1
maze
9
1
0.25
0.001
0.01
leaper
9
0.25
0.001
0.4
fruitbot
9
0.25
0.001
0.04
bossfight
9
0.25
0.001
0.1
jumper
9
0.25
0.001
ninja
9
0.25
0.001
starpilot
9
0.25
0.001
0.1
coinrun
9
1
0.25
0.001
0.004
heist
9
1
0.25
0.001
0.4
caveflyer
9
1
0.25
0.001
1
17

---

Under review as a conference paper at ICLR 2022
Table 4: Procgen scores on train levels (1-200) after training on 25M environment steps. MNS is shorthand for median normalized score across games. The mean and standard deviation are computed using 5 different seeds.
Game
PPO
UCB-DrAC
DIAC-FT
DrAC-TR
DAAC
iDAAC
PPO inv-m
bigfish
1.2
19.3
1.5
2.0
= 1.6
1.
17.3 6.9
bossfight
0.4
8.1
0.7
0.5
1..0
1..4
9.0 0.6
caveflyer
0.4
0.4
9.211.1
1.3
5.8
6.6
9.4
chaser
6.0
7.1
6.9
0.6
6.2
7.5
6.7
climber
8.4
8.3
11.9
9.5
10.0
10.2
8.3
coinrun
9.5
84#14
8.0
9.3
9.8
9.8
9.7
dodgeball
4.4
9.0
12.1
1.3
5.2
4.9
8.8
fruitbot
0.5
29.2
31.0
30.3
29.7
29.1
0.4
heist
7.0
0.6
7.0
1.1
5.2
4.5
7.1
jumper
8.9
8.1
9.2
0.3
8.6
8.7
9.1
leaper
6.1
4.1
6.1
2.7
8.0
8.3
9.1
maze
9.2
8.
9.0
9.4
6.6
6.4
9.2
miner
1..7
1.2
1..8
1..
1.93
1.55
12.8 0.1
ninja
7.7
6.7
9.7
9.2
8.8
0.3
9.2 0.3
plunder
0.8
7.1
16.0
9.5
2..5
2.66
9.0 3.8
starpilot
3.5
3.9
4.0
3.9
3..0
38.6 2.2
36.6 2.1
MNS
0.57 0.05
0.55 0.09
0.72 0.07
0.66 1 0.08
0.62 1 0.06
0.63 1 0.05
0.69 0.06
Table Procgen scores on test levels after training 25M environment steps. MNS shorthand for median normalized score across games. The mean and standard deviation are computed using different seeds.
Game
PPO
UCB-DI ICB-DrAC
DTAC-FT
DrAC- TR
DAAC
iDAAC
PPO inv-m
bigfish
1.7 + 0.3
15.1 15.133 3.8
26.2 2.2
20.9 + 2.7
17.8 1.4
18.5 1.2
12.3 + 6.2
bossfight
7.8
0.2
0.6
9.7
9.5
9.5
8.6
caveflyer
6.7
5.0
7.0
6.9
4.2
5.0
7.0
chaser
1.0
7.1
6.400.8
1.2
6.6
6.2112
2.7
climber
5.7
6.9
11.2.0.22
8.2
7.8
8.3
5.9
coinrun
8.9
7.6
7.3+14
8.4
9.2
9.4
9.1
dodgeball
1.5
6.0
9.4
7.7
3.3
3.2
3.8
fruitbot
2..1
25.6++0.2
27.8+13
27.9
28.6
27.9
27.0
heist
2.7
3.2111
3.611.3
3.1
3.3
3.5102
3.6
jumper
5.940.44
5.5
7.7
7.140.4
6.5
6.3
6.3
leaper
6.4111
1.0116
6.0
5.4424
7.3111
7.7111
8.7
maze
6.2
6.9
8.1
1.3
5.5
5.6
6.0
miner
9.0
9.140.3
11.20..4
1.90
8.6
9.5
10.1
ninja
6.2
5.0
9.3
8.6
6.8
6.8
6.9
plunder
0.5
6.0
13.6#211
8.6
2..7
23.3#33
8.3
starpilot
28.8
31.5+33
42.1
3.60
36.4
3..0+2.
36.7
MNS
0.36 0.06
0.3
0.19
0.11
0.48
0.50 0.06
0.47
18

---

Under review as a conference paper at ICLR 2022
Table 6: Train accuracy of level classifier trained the encoder features of policies learned with PPO on 200 levels, with PPO on 100k levels, with DAAC on 200 levels, and with iDAAC on 200 levels all proegen games.
PPO (200)
PPO (100k)
DAAC
iDAAC
bigfish
1.00 + 0.01
1.00 0.00
1.00 0.00
1.00 0.00
bossfight
1.00 + 0.01
0.99 + 0.01
1.00 0.00
1.00 + 0.00
caveflyer
1.00 0.00
1.00
0.08
1.00 +.000
chaser
1.00
0.00
1.00 1.00
climber
0.97 0.02
0.05
0.07 0.00
coinrun
0.67
0.27
1.00
0.98 = 0.02
dodgeball
1.00 0.00
1.00 0.00
1.00
1.00 0.00
fruitbot
0.00
0.90 1:0000.0 0.01
0.70
0.66 + 0.03
heist
1.00
1.00 100+0000 0.00
1.00 0.00
1.00 + 0.00
jumper
1.00
1.00 0.00
1.00
0.99 + 0.01
leaper
1.00
1.00 : 0.00
1.00
1.00 0.00
maze
0.99
1.00
1.00 1.00
miner
0.92
1.00
0.78 0.02
ninja
0.75
0.04
0.73 0.04
plunder
1.00
0.00
0.99
0.99
starpilot
1.00
1.00
1.00
1.00
Table 7: Test accuracy of level classifier trained the encoder features of policies learned with PPO on 200 levels, with PPO on 100k levels, with DAAC on 200 levels, and with iDAAC on 200 levels on all proegen games.
PPO (200)
PPO (100k)
DAAC
iDAAC
bigfish
0.94 + 0.04
0.74 0.12
0.97 0.01
0.97 0.01
bossfight
0.78 + 0.03
0.69 0.02
0.78 - 0.04
0.76
caveflyer
1.00 0.00
0.94
0.80 0.02
chaser
1.00
0.06
0.89
0.02
climber
0.93 + 0.02
0.72
0.05
0.79
coinrun
= 0
0.22
0.95
0.92 + 0.03
dodgeball
+.00
1.00 0.00
1.00
0.99 + 0.01
fruitbot
4.79
0.37 0.06
0.54 + 0.03
heist
1.00
1.00 + 0.00
0.99 0.00
0.99 + 0.00
jumper
1.00
0.99 0.00
0.88
0.89 0.04
leaper
1.00
1.00
0.00
0.98 + 0.00
maze
0.02
0.99
1.00 1.00
miner
0.02
+.00
0.78 0.001
ninja
0.70
0.04
0.71
0.69
plunder
1.00
1.00 + 0.00
0.99+0.00
0.90
starpilot
1.00
+.00
+.00
1.00
19


Reviews:
[{"review_id": "yIhL7Dp29Ce", "reviewer": "Reviewer_SFef", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\nThe paper attempts to address why policies learned for video games have generalization issues, which is an important problem and helps us design better models and learning algorithms in the future.  \n\n**Weakness**\n\nThe main issue is that there is almost no novel finding in the paper. Most of the conclusions are already known. See details below:\n\n- Section 4: I am wondering why the policy should not have level-specific features when it is trained for it. Also, the experiment is not flawless. Two states in train and test might be very close to each other. This doesn't mean that the policy has level-specific features.\n\n- Section 5: The paper shows an inverse model (inferring the action between two states) achieves good performance. That is also already known, and several others papers have shown that. Some of these papers have been cited in the paper as well.\n\n- Section 6: The conclusion of this section is that the performance drops if a single set of parameters is used for all games, which is expected.\n\n- Section 7:  It is mentioned that \"we see that end-to-end training of policy on levels 201 − 400 is less sample-efficient than finetuning the policy head as well as finetuning the entire policy\". This statement is not correct. The blue and black curves in Figure 5 are almost identical or have a large overlap for most games. \n\n- Some of the conclusions are not aligned with the observations of other works. For example, the paper recommends that \"This suggests a fruitful avenue for future work: instead of learning better visual features, focus on how to leverage the learned features to solve a new task\". However, works such as [A] show better visual encoders are important for generalization to unseen environments.\n\n**Comments about the details:**\n\n- It is mentioned that \"Figure 5 shows that the average returns of the policy on levels 201−400 become similar to the average\nreturns of the policy on levels 1 − 200\". I am wondering if average returns are comparable. These are two different sets of layouts.\n\n- It is mentioned that  \"Furthermore, we implemented a variant of the ProcGen suite that uses a separate randomization\nprocess for the theme versus the game layouts\". I am not sure what this means. It is unclear how the same randomization process was used for both theme and layout in the other scenario.\n\n[A] Wijmans et al., How to Train PointGoal Navigation Agents on a (Sample and Compute) Budget, 2020.", "summary_of_the_paper": "The paper studies generalization of policies trained for video games. More specifically, it analyzes transfer to new game situations via changing the theme (appearance) or the level (layout) of the game. The Procgen framework is used to procedurally generate new levels or themes to evaluate a set of hypotheses. The paper investigates (1) data augmentation, (2) policy generalization via domain confusion (3) generalization via auxiliary tasks (4) effects of hyperparameter selection and (5) adaptation to new levels.", "summary_of_the_review": "First, I do not see any novel conclusion in the paper:\n\n(1) It is already shown by several papers that inverse models are helpful.\n\n(2) It is already known that data augmentation techniques are required for better transfer.\n\n(3) It is expected that using the same parameters for all games results in inferior performance compared to using game-specific parameters.\n\nSecond, the conclusion that improving visual encoders is not that important contradicts other works.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "xDL_d_ffvsl", "reviewer": "Reviewer_sNjW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes an empirical study of different mechanisms often set in place to prevent overfitting in Deep RL.\nAlthough the empirical findings are interesting and well-conducted, they don't fully cover the span of regularization methods. More importantly, the authors point out the challenges and weaknesses but do not address them, limiting the impact of the contribution.\n\n## Writing and clarity\n\nThis article appears to have been written in a hurry, without any proof-reading or work on the structure.\nThere are lots of typos, many articles or words missing, acronyms are undefined, one bibliographical reference is replaced with a question mark, the ordering of the bibliography is shady, etc.\nSeeing the related work section after the conclusion is an unorthodox practice at best, and looks more as if it was written and just dropped there. Additionally, it is mostly redundant with the introduction section and most citations already appear in the paper.\nMany ideas are repeated several times (e.g. the last paragraph of the introduction just repeats the previous ones, Section 8 is a repetition of the main take-away messages from Sections 3 to 7).\nDespite these aspects and the fact that it lacks content (see next comments), the paper is rather clear, well-organized and pleasant to read.\n\n## Lack of perspective on regularization for RL\n\nThe idea of preventing overfitting and fostering generalization is often tackled as regularization. As the authors point out, many recent work have concentrated on this aspect, including the three papers by Raileanu et al, which serve as a basis for this work. iDAAC being somewhat state-of-the-art, it is justified, but other approaches such as IBAC-SNI, random convolutions, or the \"historical\" RAD algorithm would be legitimate candidates to verify the empirical study conducted here. Additionally, as noted in (Song et al, 2019) regularization can be separated as explicit regularization (explicit penalization of the empirical risk) or implicit regularization on the architecture and optimizer (dropout, batchnorm, momentum, early stopping) or on the data (data augmentation). This categorization applies both to supervised and to reinforcement learning. The only place where regularization is covered as a whole is the very last paragraph of the paper. And a natural question is then to assess whether the techniques evaluated in the paper actually cover a significant span of the possible regularization techniques overall.\n\n## Insufficient discussion\n\nMost take-away messages take the form of a challenge and some perspectives for future work. Besides the raw results, the reader does not get much insight from the reading.\n\nIn Section 4, I think the discussion is very incomplete. It seems to me that it is necessary for the policy to rely on level-specific features and that, ideally, an algorithm that generalizes well should disentangle (I didn't see any reference to the topic of disentanglement in the paper) the theme information from the layout information and only base the policy on the latter. Therefore, I think the problem should not be to evaluate whether \"policies have level-specific features\" but rather to assess whether these features are theme-independent. the discussion on this question is too shallow (almost inexistent).\nMore anecdotically, the claim that \"policies have level-specific features regardless of whether it generalizes or not\" is not backed up by Table 1 since the policies that have been used in this table \"have good generalization capabilities\".\n\nIn Section 5, the discussion on auxiliary tasks deserves a lot more details. The presented results display a lot of variability and the actual benefit of the proposed auxiliary task is very dependent on the game tackled. This section would benefit a lot from discussing the reasons of this variability. For example, auxiliary tasks could be expected to bring in game-dependent benefits since it depends on whether the auxiliary tasks features help disambiguate between equivalent sets of features for the policy. Unfortunately, the discussion stops after the plain presentation of results.\nAlso, data augmentation was mixed up with domain randomization. I assume it is a moment of inattention coupled with a lack of proof-reading. \n\nSection 6 (unfortunately) is nothing new. Hyperparameter sensitivity is (unfortunately) a recurrent, known problem in RL (even without the problem of generalization). And (unfortunately), no new insights are provided in this section.\n\nFigure 6 (Section 7) deserves a much longer discussion. Why does training on more levels hinder the overall performance? Is the training done sequentially or in parallel (in the latter case, is it only a matter of number of samples to process before moving on to a next step?)? Even the raw results are not sufficient here to inform the reader on the mechanisms at hand.", "summary_of_the_paper": "This paper proposes an empirical study of different mechanisms often set in place to prevent overfitting in Deep RL. It draws conclusions at what the challenges are and suggests future work directions.", "summary_of_the_review": "This paper proposes an empirical study of different mechanisms often set in place to prevent overfitting in Deep RL.\nAlthough the empirical findings are interesting and well-conducted, they don't fully cover the span of regularization methods. More importantly, the authors point out the challenges and weaknesses but the discussion is very shallow and they do not address them, limiting the impact of the contribution.\nThe paper itself needs an important effort of editing (it is still a very preliminary version, assembled in a rush), and a significant contribution to improve the generalization capability of RL algorithms, to have a meaningful impact for the community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "mjNWTJoTmL7", "reviewer": "Reviewer_185P", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths \n\nThe paper's overarching goal is to understand why current RL algorithms (aimed at improving generalization) still fall short in certain settings. This is an important problem and answers to this question would be valuable for the community. The paper contains some interesting insights regarding some recently proposed algorithms.\n\nI also liked the findings related to the fact that simple models e.g. using a simple inverse model as auxiliary task can be quite effective at improving generalization in RL and even comparable with other SOTA (aiming to learn domain invariant features) methods like DrAC or DAAC. This suggests a fruitful direction for future work might be understanding which types of auxiliary losses or self-supervised training may lead to good generalization. I think these insights would be valuable for the community and those experiments seemed reasonably well designed for the conclusions to hold. \n\n### Weaknesses\n\nHowever, the paper does not seem to provide a clear and definite answer to the question. It seems to contain a number of disconnected experiments that answer slightly different questions than the main question.\n\nIn addition, some of the claims do not seem to be supported by the experiments. Some of the experiments are lacking important details and are not very well designed to answer the questions they are supposed to. \n\nThe paper seems to look at both the setting of zero-shot generalization and the setting of few-shot / fast adaptation to new tasks. However, the authors should clarify which setting they are referring to and use more specific nomenclature throughout the paper since the writing can be confusing at the moment (especially at the beginning of the paper when readers may not know you are investigating multiple settings). In addition, referring to a level as a separate task can be misleading since that typically refers to a different reward and / or transition function, which is not typically the case in Procgen where the levels differ in layout / theme but the overall goal is the same. \n\nIn many places throughout the paper, including the figures, it is not clear whether you are referring to the training or test set performance, which makes it difficult to assess the results and conclusions. \n\n1. I am not convinced that the best way to test whether a method is theme invariant is to see if its test performance is the same when training it on levels with the same theme because this may confound better optimization with better generalization (train and test performance)? A better way to test whether a method is theme invariant is to test it on levels that have the same layout but different themes and see if the performance differs from the training levels. However, this does not seem to be the case here, but I found the description to be quite unclear. For DrAC-FT, do you use the exact same layout for the training levels as those used to train DrAC, with the only difference being that the theme is the same for all levels?\n\n2. Does DrAC-TR use observations with same layout but different themes as augmentations or does it simply add more levels to the training set and uses the same types of augmentations as DrAC? Again, I found this section to be missing important details and clarifications. The comparison between DrAC-TR and DrAC isn’t fair since DrAC-TR essentially gets to train on many more levels of a game and because it sees the exact same levels in different themes it has the opportunity to learn theme invariance (perhaps even without the need to explicitly regularize the policy and value function). \n\n3. It would be interesting to compare with PPO-TR to see how it compares to DrAC. You can also apply DrAC but with augmentations given by different themes of the game. An interesting question is whether you need to explicitly regularize the policy and value when you have this type of data and perhaps how many themes do you need to capture this invariance (to any other theme)?\n\n4. In Figure 2, the aggregated results don't seem to differ very much. However, there are games where the differences are significant, and others where they are not. I don't think the text reflects this very well and some of the claims can be misleading. \n\n5. Can you provide some intuition on why DrAC-FT would be better than DrAC? Are they being trained and tested on the exact same dataset? \n\n6. While your experiments seem to suggest that the methods still capture level specific features, while having good generalization, and thus those features are needed to learn generalizable policies, I am not sure the claim is correct. There are level specific features which you need in order to take reasonable actions and others which you do not (so you want to be invariant to these). Perhaps a better test is whether the theme can be predicted (instead of the level), which is typically not useful for solving the task. In contrast, the layout typically is needed to both solve the task and predict the level of the game. It also depends how different the test set is from the train set. If you collected this using a random policy you might end up with essentially identical observations in the train and test set and thus it will be very easy to predict the level (due to memorization). I suggest collected the dataset using a pretrained policy or checking that the test set is completely different from the train set. \n\nThe problem might still be that the visual features, while containing the necessary elements to make good decisions, they also contain elements that are superfluous, level-specific, and thus making the policy overfit and fail on certain levels at test time (similar to spurious correlations in SL). I encourage the authors to design experiments aimed at teasing out the two to better understand whether those features are beneficial or not (and in what settings). \n\n7. I think Procgen may not be the best benchmark for this study since it can be difficult to control all the elements of variation across the different levels of a task. I would suggest designing simpler experiments in domains where you can change one variable at a time (e.g. the theme or layout, or agent-goal distance etc.). Perhaps you can then use Procgen to see if the conclusions hold in more complex / different domains. \n\n### Minor \n1. Sometimes you refer to UCB-DrAC as UCB-DrAC while other times as DrAC. I think it's better to decide on one of these and mention the convention at the beginning of the paper and then use consistent notation throughout the draft.\n2. The related work section should be moved earlier in the paper and definitely before the discussion and conclusion \n", "summary_of_the_paper": "This paper looks into why some of the recent methods for improving generalization in deep RL still do not perfectly generalize to new levels of a game. For data augmentation methods, they show that better augmentations which make use of the underlying process that generates the data / ground truth task variation. For approaches that learn domain invariant features, the authors claim that you actually need domain-dependent features to generalize well and that these methods may work well for different reasons (i.e. the additional signal and regularization from the auxiliary losses). They investigate approaches that use data augmentation and domain invariant features and provide some new insights. The authors also show that simple auxiliary tasks can improve generalization policies and that other approaches. ", "summary_of_the_review": "Overall, I think this paper is the beginning of some interesting investigations. However, I am not convinced the experiments described in this paper support the claims made by the authors. In addition, I think it would be better to focus on a one or two questions, clearly state them, and try to answer them as clearly as possible and provide as much insight and depth as possible. In its current form, it is not clear what questions the authors are trying to answer and what the conclusions are. \n\nGiven all the above, I do not think the paper is ready for publication. However, I do think the subject of the paper is important and would be of interest for the community. I encourage the authors to take the feedback into account in order to improve the paper, and resubmit at a future conference. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dw0-6MT_-kj", "reviewer": "Reviewer_eZnU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper asks important questions about generalization theme and task wise and answers them but I am not sure (not saying that they are not, just cannot wrap my head around it) if generalization of the conclusions is justified. Especially considering results showing that hyperparameters must be finetuned for every game separately. \n\n- In abstract starting from \"Contrary to...\" it's not clear what you meant. Introduction part which states the same info is clear.\n- Introduction is sturdy, clean and clear.\n- Probably would drop \"investigating\" in headers and separate motivation/setup/results/discussion parts  of each subsection in more clear way\n- Some figures could use better comments (e.g. Figure 2 is ok while Figure 3 lacks similar details like what is the dotted line).\n- Figure 2: isn't it an overstatement to say that TR matched FT? on 8-12 out of 15 it look like significantly or slightly  worse. \n- The paper is not fully explicit in terms of experimental part but it's usually \"like in the <link provided>\" so I assume that the sources are explicit enough. Similar approach is taken to descriptions of algos and such - no elaboration and just reference (usually I feel that there is too much but here is too little perhaps). It surely benefits the brevity and cleanness but requires more jumps in other papers than usually if not acquainted with previous work. However I'm not sure whether to count it as positive or negative.\n\n", "summary_of_the_paper": "The paper studies generalization of several previously published visual RL SOTA algorithms with PPO as a baseline in ProcGen environment (2d platformed style, procedurally generated maps). Generalization is studied in terms of theme (colors and styles of objects and background) and tasks (levels with different layouts and specifics things to do). DrAC is shown to not be theme-invariant by comparing result from vanilla version, a version where themes are fixed and version with randomized themes (vanilla version performs worse). For PPO, DAAC and iDAAC, linear classifiers on top of visual feature extractor layers (pretrained on the rl task) are trained to classify states (images from level) based on the level they belong to. Authors claim that accuracy around 90% hints that tested policies contain level specific information regardless of any generalization capabilities. Moreover authors propose a simple auxiliary task \"inverse model\" for ppo and show that it's comparable with UCB-DRAC. Finally it is mentioned that all methods need careful on per-game basis hyperparameter tuning which requires further research", "summary_of_the_review": "The whole paper is not groundbreaking but makes its point in a clear and understandable way and seems to be natural consequence of previous research. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an empirical study of generalization in visual reinforcement learning. This study is carried out in the domain of video games and it addresses the benefits of techniques such as regularization, augmentation and training with auxiliary tasks. The reviewers for this submission were positive about the goal and setups in this paper. They agreed that understanding why present day methods that attempt to improve generalization continue to fall short, is an important problem. However, most reviewers were underwhelmed by the findings presented in the submission. As examples: Reviewer 185P mentions that "the paper does not seem to provide a clear and definite answer to the question" and " I am not convinced the experiments described in this paper support the claims made by the authors." and Reviewer SFef mentions that "Most of the conclusions are already known". Some reviewers also found a lack of clarity and several typos in the initial submission. The authors have provided detailed responses to the reviewers. In particular they have fixed most writing issues. They also detailed why certain algorithms and techniques were benchmarked in this submission and others were left out. I think this is reasonable. One cannot expect a paper to benchmark every algorithm out there, and choosing promising and representative ones is sufficient. My takeaway from detailed discussions about this paper are that: The paper is much improved from a writing point of view and the rebuttal addresses some concerns well. However, I do agree with the reviewers that the findings presented in the paper are for the most part expected. This reduces the value of the paper to readers. When this is the case, it may be beneficial to dig deeper into these findings and present a narrow but deep analysis. Please see Reviewer 185P's suggestions in this regard. Given the above, I am recommending rejection for this conference, but I encourage the authors to take into the reviewers suggestions and resubmit.
Decision: Reject