Local Feature Swapping for Generalization in Reinforcement Learning
Authors: David Bertoin
Emmanuel Rachelson
Venue: ICLR 2022 Poster (2022)
Keywords: Reinforcement learning
Generalization
Regularization

Abstract: Over the past few years, the acceleration of computing resources and research in Deep Learning has led to significant practical successes in a range of tasks, including in particular in computer vision. Building on these advances, reinforcement learning has also seen a leap forward with the emergence of agents capable of making decisions directly from visual observations. Despite these successes, the over-parametrization of neural architectures leads to memorization of the data used during training and thus to a lack of generalization.
Reinforcement learning agents based on visual inputs also suffer from this phenomenon by erroneously correlating rewards with unrelated visual features such as background elements. To alleviate this problem, we introduce a new regularization layer consisting of channel-consistent local permutations (CLOP) of the feature maps. The proposed permutations induce robustness to spatial correlations and help prevent overfitting behaviors in RL. We demonstrate, on the OpenAI Procgen Benchmark, that RL agents trained with the CLOP layer exhibit robustness to visual changes and better generalization properties than agents trained using other state-of-the-art regularization techniques.

Published as a conference paper at ICLR 2022
LOCAL FEATURE SWAPPING FOR GENERALIZATION IN REINFORCEMENT LEARNING
David Bertoin
Emmanuel Rachelson
IRT Saint-Exupéry
ISAE-SUPAERO
ISAE-SUPAERO
Université de Toulouse
ANITI
ANITI
Toulouse, France
Toulouse, France
david. bertoiniit www.brooonnditt-siiineeeppp.eee saintexupere.coo
emmanuel wammanuel..aheesooroooo www.heeonnissee
ABSTRACT
Over the past few years, the acceleration of computing resources and research in deep learning has led to significant practical successes in range of tasks, in- cluding in particular in computer vision. Building on these advances, reinforce- ment learning has also seen leap forward with the emergence of agents capable making decisions directly from visual observations. Despite these successes, the rerepaaramttizttoonn of neural architectures leads memorization of the data used during training and thus to a lack of generalization. Reinforcement learning agents based visual inputs also suffer from this phenomenon by erroneously correlating rewards with unrelated visual features such as background elements. To alleviate this problem. we introduce a new regularization technique consist- ing of channel-consisten local permutations (CLOP) of the feature maps. The proposed permutations induce robustness to spatial correlations and help prevent overfitting behaviors in RL. We demonstrate, on the OpenAl Procgen Benchmark, that RL agents trained with the CLOP method exhibit robustness to visual changes and better generalization properties than agents trained using other state-of-the-art regularization techniques. We also demonstrate the effectiveness of CLOP as a general regularization technique in supervised learning.
INTRODUCTION
Advances made in deep learning have opened the way to many applications in computer vision such as classification, object recognition, or image segmentation. The powerful representation capabili- ties of deep neural networks paved the way for many successes in deep reinforcement learning with the design of agents able take decisions directly from pixels (Mnih et al., 2013; 2015). However, the sensitivity of neural networks to the distribution of training data strongly affects their generaliza- tion abilities. Neural networks are intrinsically designed to memorize the data they are trained upon, since their fitting implies empirical risk minimization (Vapnik, 1992) (they minimize the empirical average prediction error over a large training dataset). Therefore, they are prone to prediction errors on unseen samples. RL agents also suffer from this handicap and tend memorize training tra- jectories, rather than general skills and features leading to transferable policies. This phenomenon, usually known as overfitting, takes double sense in RL. Generalization in RL implies the ability to generalize across states (as in supervised learning), but also across environments. It is only recently that several environments with different configurations for training and testing have emerged and received a lot of attention (Nichol et al., 2018; Justesen et al., 2018: Zhang et al., 201 Sa; Cobbe al., 2019; 2020), shedding light on the generalization issue which remained mostly overlooked, and confirming the poor generalization ability of current algorithms.
Strategies to achieve good generalization and avoid overfitting in deep learning fall into three cat- egories of regularization: explicit regularization (e.g., loss penalization, weight decay), implicit regularization via the architecture and optimization (e.g.. dropout, natch-normalization, batch size selection, momentum, early stopping). or implicit regularization by enhancement of the input data (data augmentation). Direct application of these strategies to deep RL agents has demonstrated some improvements in agent generalization abilities in some environments, but much progress remains to

---

Published as a conference paper at ICLR 2022
be made to integrate RL systems in real-world applications. In this work, we address the observa- tional overfitting issue introduced in Song et al. (2019) which considers a zero-shot generalization RL setting. An agent is trained on specific distribution of environments (for example some levels of a platform video game) and tested on similar environments sharing the same high-level goal and dynamics but with different layouts and visual attributes (background or assets). We argue that structured stochastic permutation of features of an RL agent during training leads to state-of-the- art generalization performance for vision-based policies. We introduce an efficient regularization technique based on Channel-consistent LOcal Permutations (CLOP) of the feature maps that mit- igates overfitting. We implement it as an intermediate layer in feed-forward neural networks, and demonstrate its effectiveness on several reinforcement and supervised learning problems.
This paper is organized as follows. Section presents the necessary background on generalization in supervised learning and reinforcement learning. Section 3 reviews recent work in the literature
that allows for critical look at our contribution and put it in perspective. Section 4 introduces the CLOP technique and the corresponding layer. Section 5 empirically evaluates agents using CLOP against state-of-the-art generalization methods and discusses their strengths, weaknesses, and variants. Section 6 summarizes and concludes this paper.
2 WHAT IS GENERALIZATION?
Generalization in supervised learning (SI.). Let X' be an input space of descriptors and ) an output space of labels. A SL problem is defined by distribution p(z, y) of elements of X' x y, and a loss C(g. y) which measures how different 5 y and y E y are. Then, for given function intended to capture the mapping from X' to y' underlying the p distribution, one defines the (expected) risk as R(f) = Ep (x)) y)]. Since the true p is generally unknown, one cannot directly minimize the risk in search for the optimal f. Given training set S = {(1;, )=) of items in X' x y drawn i.i.d. according to p(r, y). empirical risk minimization (Vapnik, 1992) seeks to minimize R&(f) = 1/n Ii by the generalization gap R(f) Rf) which is often evaluated by approaching R(f) as the empirical risk Rf) over a test set T = { J: also drawn i.i.d. from p(z, y). Closing the generalization gap can be attempted through structural risk minimization, which modifies R.s so as to include a regularization penalty into the optimization, or more generally by the introduction of inductive biases (Mitchell, 1980).
v)). The ability of f to generalize to unseen samples is then defined
Reinforcement learning (RL). RL (Sutton & Barto, 2018) considers the problem of learning a de- cision making policy for an agent interacting over multiple time steps with a dynamic environment. At each time step, the agent and environment are described through state s E S, and an action a E performed; then the system transitions to a new state s' according to probability T(s') s, a), while receiving reward R(ss a). The tuple M = (S, A, T, R) forms a Markov Decision Process (Puterman, 2014, MDP), which is often complemented with the knowledge of an initial state distri- bution (s). A decision making policy parameterized by 0 is a function re(a|s) mapping states to distributions over actions. Training a reinforcement learning agent consists in finding the policy that maximizes the discounted expected return: J((o)) E 70 ( a.)).
Generalization in RL. Departing from the rather intuitive definition of generalization in SL, the idea of generalization in RL may lead to misconceptions. One could expect policy that general- izes well to perform well across environments. It seems important to disambiguate this notion and stress out the difference between generalization in the SL sense (which is single MDP problem), and domain generalization or robustness. For instance, one could expect policy that learned to play a certain level of platform game be able to play on another level, provided the key game features (c.g. platforms, enemies, treasures) remain similar enough and the game dynamics are the same. This type of generalization benchmark typically captured by procedurally generated environments, such as Procgen (Cobbe et al., 2020). Finding policy that yields a guaranteed min- imal performance among a set of MDPs (sharing common state and action spaces) is the problem of solving Robust MDPs (Iyengar, 2005). Similarly, that of policy optimization over a distribution over MDPs (sharing common states and actions too) is that of domain generalization (Tobin et al., 2017). We argue that these problems are unrelated and much harder than vanilla generalization in RL. Robust MDPs induce a max, min7, problem, domain generalization is a max, ET,R one, while vanilla generalization as understood in SL remains max, problem that we try to solve for
2

---

Published as a conference paper at ICLR 2022
all states within single MDP (and not only the ones in the training set). Consequently, the ability to generalize in RL is a broader notion than structural risk minimization, which, unfortunately, uses the same name of generalization. In given game, finding a policy that plays well on all levels is still solving the very same MDP. The underlying transition and reward models are the same and only the parts of the state space explored in each level are different. Note that these explored subsets of the state space may have non-empty intersections (e.g. the final part of two different levels might be common) and the optimal distribution over actions for given state is unique. In this work we focus on generalization as the problem of preventing overfitting (and thus reducing the generalization gap) within a single MDP, which we call observational overfitting (Song et al., 2019).
Observational overfitting (00) in RL. The inability to generalize might be caused by overfitting to the partially explored environment dynamics (Rajeswaran et al., 2017), or to some misleading signal that is correlated with progress but does not generalize to new levels (Machado et al., 2018; Song et al., 2019). Therefore, preventing OO in RL. remains the ability to generalize across states, within the same MDP. Beyond the ability of RL agents to memorize good actions in explored states, it boils down to their ability to capture rules that extend to unencountered states, just as in structural risk minimization. As in SL, many policies might fit the observed data, but few might generalize to the true mechanisms of the underlying MDP. Song et al. (2019) propose to capture OO through framework where unique latent state space is transformed into variety of observation spaces. Observation functions are built by combining useful features with purely decorative and unimportant ones which vary from one observation function to the next. They suppose the observation functions are drawn from certain distribution (over procedural generation parameters for instance) and define the corresponding distribution over MDPs. In turn, they define the risk and the generalization gap with respect to this distribution over MDPs. We slightly depart from their derivation and argue this distinction is unnecessary: what is being solved the unique MDP defined by the underlying dynamics and the projection of latent states into observations. The risk is thus defined with respect to the distribution over observations induced by the distributions over initial states and observation functions. Overall, this allows capturing the problem of generalization gap minimization in RL and underpins the developments proposed in the remainder of this paper.
RELATED WORK
In the following paragraphs, we cover essential works which aim to improve the generalization abilities of neural networks in both supervised learning and reinforcement learning.
In supervised learning, the process of modifying learning algorithm with the objective to reduce its test error while preserving its train error is known as regularization (Goodfellow et al., 2016). Direct, or explicit, regularization can be achieved by adding regularizer term into the loss function, such as an L2 penalty on networks parameters (Plaut et al., 1986; Krogh & Hertz, 1992). A second, implicit, regularization strategy consists in feature-level manipulations like dropout (Srivastava et al., 2014), drop-connect (Wan et al., 2013) or natch-normallzation (loffe & Szegedy, 2015). Regularization can also be achieved implicitly by directly augmenting the training data with perturbations such as adding Gaussian noise (Bishop, 1995), or, in the case of visual inputs, random cropping and flipping (Krizhevsky et al., 2012; Szegedy et al., 2017) or removing structured parts of the image (DeV (reVsies & Taylor, 2017). Another efficient augmentation strategy, called label smoothing (Szegedy et al., 2016), consists in penalizing overconfident predictions of neural networks by perturbing the labels. Combining perturbation of both inputs and outputs, Zhang al. (2018c); Yun et al. (2019), produce synthetic data and labels using two different samples and their corresponding labels.
Many studies have highlighted the limited ability of RL agents to generalize to new scenarios (Fare- brother et al., 2018; Packer et al., 2018: Zhang et al., 2018b; Song et al., 2019; Cobbe et al., 2019). Using saliency maps, Song al. (2019) exhibit that RI. agents trained from pixels in environments with rich and textured observations, such as platform video games (e.g. Sonic (Nichol et al., 2018)), focus on elements of the scenery correlated with in-game progress but which lead to poor gener- alization in later situations. One of the identified counter-measures to overfitting in RL consists in applying standard methods in supervised learning. Cobbe et al. (2019; 2020) demonstrated the contribution of classical supervised learning regularization techniques like weight decay. dropout, or Batch-normalizen to generalization in procedurally generated environments. Similar to data- augmentation in supervised learning, Raileanu et al. (2020); Laskin et al. (2020); Yarats et al. (2020)
3

---

Published as a conference paper at ICLR 2022
apply -aggllddttttuggttttttttiio the observations provided by the environment to train robust agents. Igl et al. (2019) use a selective noise injection and information bottleneck to regularize their agent. Wang et al. (2020) propose mixreg, direct application of mixup (Zhang et al., 2018c) in RI., combining two randomly sampled observations, and training the RL agent using their interpo- lated supervision signal. Lee al. (2020) use random convolution layer ahead of the network architecture, to modify the color and texture of the visual observations during training. Tobin et al. (2017) tackle the sim-to-real problem, using domain randomization on visual inputs to bridge the gap between simulation and reality in robotics. Raileanu & Fergus (2021) dissociate the optimiza- tion process of the policy and value function represented by separate networks and introduce an auxiliary loss that encourages the representation to be invariant to task-irrelevant properties of the environment. Another recent strategy consists in learning representations that are invariant to vi- sual changes. Higgins et al. (2017) use two-stage learning process: first they extract disentangled representations from random observation and then they exploit these representations to train an RL agent. Zhang al. (2020) use bisimulation metrics quantify behavioral similarity between states and learn robust task-relevant representations. Wang et al. (2021b) extract, without supervision, the visual foreground to provide background invariant inputs to the policy learner.
Our work focuses on the 00 situations, where an agent overfits visual observations features that irrelevant the latent dynamics of the MDP. Our method approaches the augmentation of data by noise injection at the feature level, thus avoiding the computationally costly operations of high- dimensional image transformation in regular data augmentation. By directly modifying the encoun- tered features' spatial localization, the CLOP layer aims to remove the correlations between spurious features and rewards.
4 CHANNEL-CONSISTENN LOCAL PERMUTATION LAYER
Three key intuitions underpin the proposal of the channel-consistent local permutation (CLOP) layer. 1) In many image-based decision making problems, the important information is often spatially scarce and very localized. The example of platform games is typical: most of the pixels are decor- related from the MDP's objective. But this also applies to other games (c.g., driving, shooting) or environments such image-based object manipulation. Consequently, most pixels don't ton't convey useful information for an optimal policy and data augmentation should help disambiguate hetween informative and uninformative pixels. Specifically, we can expect that image and its altered counterpart share the same reward value for the same action. Even if perturbing an image might mistakenly associate the wrong action it, in most cases we expect noise injection in images to be beneficial for generalization since it creates synthetic images where most of the perturbed features were unimportant in the first place.
2) We refer to the process of modifying C' x H x W image as data augmentation, whether this image is an input image or set of feature maps produced by the C' filters of convolutional layer. We argue that data augmentation in the latent space is more efficient for generalization in de- cision making than in the input space. For example, seems more useful for generalization to leam rule stating that an enemy sprite's 's position is important for the action choice, than to learn this same rule based on independent pixels, which might lead to O0. The successive convolutional layers provide greater levels of abstraction of abstraction and forms a latent description space of the input images, with abstract features, like whole-object positions. Consequently, to achieve generalization in RL, data augmentation should probably be done in the latent space, on the feature maps after the deepest convolutional layer, rather than in the imput space.
each image: the deepest layer is the highest level
3) Good actions in state probably remain good actions in closeby states in the latent space. For example, if one has determined that jumping a good action in a specific image, then it is likely that if the image's 's objects move very locally, jumping remains good action in the corresponding new images. This idea is related to that of Rachelson & Lagoudakis (2010) who study the locality of action domination in the state space. Consequently, we conjecture that local shifts of features in the deepest feature maps (hence representing high-level concepts and objects, rather than raw pix- els) might help generate synthetic samples that help the optimization process disambiguate between useful and unimportant information. In turn, we expect this data augmentation to help prevent the bare memorization of good trajectories and instead generalize to good decision making rules.
4

---

Published as a conference paper at ICLR 2022
input: of shape (NN, C, H, W) if training mode then
P shuffle w))
for (h, w) in
(h' w")
draw a direct neighbor of (h, w)
With proba a, (: h, w], [ w())
return X
Figure 1: Channel-consistent I.Ocal Permutation Layer (CI.O))
0
0.2
0.4
0.6
0.8
Figure 2: Examples of CLOP layer outputs with different values for
The CLOP technique transposes these intuitions into convolutional networks. We implement it as regularization layer that we introduce after the deepest convolutional layer's feature maps. During training, it swaps the position of pixels in these feature maps, while preserving the consistency across channels, so as to preserve the positional consistency between feature maps. This swapping is performed sequentially in random order among pixels, locally (each pixel is swapped with one of its neighbors), and with probability a. At evaluation time, the CLÖP layer behaves like the identity function. Figure illustrates this process and presents the pseudo-code. Since pixels taken a random order, a given pixel value might be "transported" far from its original position, with small probability. Figure 2 illustrates the effect of varying values of on the output of the layer. Since CLOP is applied after the deepest convolutional layer, each of the pixels in this image should be interpreted to different filters. CLOP shuffles these descriptors while preserving channel consistency (no new colors are created in Figure 2) and neighborhoods (objects remain close each other).
salient high-level feature in the input image, and the channels correspond
5 EXPERIMENTAL RESULTS AND DISCUSSION
This section first exhibits the CLOP method's effect on regularization in supervised learning and then demonstrates its efficiency in reinforcement learning. Hyperparameters, network architectures, and implementation choices are summarized in Appendix A and B.
5.1 SUPERVISED LEARNING
To assess the contribution of the CLOP layer in supervised learning, we first train simple net- work (three convolutional layers followed by three linear layers) on the MNIST dataset (LeCun et al., 1998) and evaluate its generalization performance the USPS (Hull, 1994) dataset, slightly different digit classification dataset. Then, to confirm this performance evaluation on large-scale images, train VGG11 network (Simonyan & Zisserman, 2015) using the Imagenette dataset,' a subset of ten classes taken from Imagenet (Deng et al., 2009). In both experiments, the net- works are trained using four configurations: with dropout in the dense part of the network, with batch-normalization between convolutions, with CLOP layer after the last convolutional layer, and without any regularization. Results reported in Table 1 show that the CLOP layer allows the net- works to generalize considerably better on USPS data than the unregularized network or when using dropout batch-normalizztton. Note that testing on USPS implies both generalization as defined in Section 2, and domain generalization (Wang et al., 2021a), a). that is the ability to generalize to other input data distributions at test time. Likewise, applying the CLOP layer to VGG11 improves gen-

---

Published as a conference paper at ICLR 2022
Method
MNISTimain)
USPS(test)
Imagenelie2 (crain)
Imagenette (teat)
network
0.0
Drupout
99.7 1 0.0
77.7
1.4
100 1: 0.0
81.4
0.6
Batch-nerm
100 = 0.0
65.6
= 7.7
100 + 0.0
52. 4 = 0.5
(ours)
91.2 =
Table 1: Comparison of train and test accuracies in supervised learning
(a) Coinrun
(b) Leaper
Figure 3: Example of levels heterogeneity two Proegen environments
eralization performance compared the other methods, thus confirming the intuition that localized permutations at the feature level during training may improve generalization at testing. Appendix D features an extended discussion on the application of CLOPP other datasets (including Imagenet) and comparison with mixup (Zhang et al., 2018c).
5.2 REINFORCEMENT LEARNING
We assess the regularization capability of CLOP on the Procgen benchmark, commonly used to test the generalization of RL agents. Procgen is set of 16 visual games, each allowing procedural gener- ation of game levels. All environments use discrete set of 15 actions and provide x 64 x 64 image observations the agent. Since all levels are procedurally generated, Procgen training and testing levels differ in many visual aspects like backgrounds, assets, intrinsic level design (Figure 3).
CLOP within PPO. In the following experiments, we train all agents with PPO (Schulman et al., 2017) and the hyperparameters recommended by Cobbe et al. (2020). As in many PPO implementations, our actor and critic share common stem of first convolutional layers, extracting shared features. Since the critic's 's role is only to help estimate the advantage in states that are present the replay buffer (in other words: since the critic won't ever be used to predict values elsewhere than on the states it has been trained on), overfitting of the critic is not critical issue in PPO. An overfitting critic will provide an oracle-like supervision signal, which is beneficial to the actor optimization process without inducing overfitting in the policy. Therefore, a CLOP layer only to the actor part of the agent's 's network, located immediately after the shared feature extractor part (Appendix) discusses the application of CLOP of both actor and critic).
choose to append
Comparison with classical supervised learning regularization methods. We first compare the performance of the data augmentation in the latent space performed by the CLOP layer with clas- sical data augmentation methods that directly enhance inputs. Following the setup of Cobbe et al. (2020), evaluate the zero-shot generalization of RL agents using the CLOOP layer three Procgen
15.0
dop
8
cuf_out_cotrr
7.5
6
motation
50
4
diopout
2.5
2
0.0
1.0
1.5
2,0
0.0
05
1.0
L5
2,0
0.0
3.5
1.0
15
(a) Miner
(b) Chaser
(c) Dodgeball
Figure 4: Average of rewards test environments, versus time steps.

---

Published as a conference paper at ICLR 2022
Game
PPO
Mixreg
Rand + FM
UCB-DraC
IBAC-
RAD
IDAAC
CLOPP
Bigtish
4.3
1.6
0.6
9.2 2.0
0.8
9.7
18.5 1.2
4.6
BossFight
9.1
8.2
1.7
7.8
1.7
0.9
9.8
9.7
CaveFlyer
5.4
5.0
8.0
5.0
Chaser
0.8
5.8
1.7
6.6
1.3
3.9
6.
8.2
Climber
6.3 0.4
6.9 0.7
5.3+0.7
6.3+0.6
3.4
6.9
8.3
7.4003
CoinRun
9.0
0.3
9.
8.8
8.7
9.0
9.4
9.1
Dodgeball
0.4
1.7
0.4
1.2
14#04
2.8
3.2
7.2
FruitBot
28.5
27.3
2..
07
24.7
27.3+1.8
27.9
Heist
0.7
2.4
0.6
3.1
9.8
11=10
3.2
45
Jumper
5.4
0.0
5.3
6.2-0.33
3.6
6.5
63+002
5.6 0.2
Leaper
6.1
5.1
0.5
18
6.6
13
7.7
9.2
Maze
5.1-+0.2
6.2
8.0+0.77
6.3
1..0
6.1
5.6
5.9 0.2
Miner
8.4
94404
7.7
9.2
8.4
9.
9.
9.8
Ninja
6.5 0.1
0.5
6.1 0.8
6.6
9.2
0.9
6.8
5.8 + 0.4
Plunder
6.1 111::0.. 0.8
5.9 0.5
3.0
0.6
8.3 1.1
2.
1.2
23.31.4
5.4
StarPilot
32.4
0.7
30.0
33.4
37.0
40.9 +
Table 2: Average returns on Procgen games. Bold: best agent; underlined: second best.
environments, setting the difficulty parameter to hard. All agents were trained during 200M steps on set of 500 training levels and evaluated 1000 different test levels. Figure 4 shows the average sum of rewards on the test levels obtained along training for all agents: the CLOP agents outperform classical data augmentation techniques (full results and discussion in Appendix G). CLOP offers three significant advantages: asymptotic generalization performance is better, sample complexity is smaller (the agent needs fewer interactions steps reach high performance levels) and computa- tional complexity is negligible (see Appendix C). Interestingly, other regularization methods based on pixel-block disturbance, such as cutout or cutout-color, despite being somehow related to the CLOP layer but applied at the input image level, often even decrease the baseline performance. The CLOP layer also outperforms dropout, another feature-level manipulation method. Compared to dropout, the CLOP layer randomly shifts features from their original position instead of completely removing them. The persistence of these features in the feature map but other positions appears to help to prevent spurious correlations with the reward signal. Overall, empirical validation confirms that the CLOP layer has significant impact on generalization, helping prevent OO by decorrelating features that are irrelevant to the task (such as background elements) from the policy's output.
Comparison with RL regularization methods. We also compare the CLOP layer with current state-of-the-art regularization methods specifically designed for RL: mixreg (Wang al., 2020), Rand+FM (Lee et al., 2020), UCB-DraC (Raileanu et al., 2020), IBAC-SNI (Igl et al., 2019), RAD (Laskin al., 2020), and IDAAC (Raileanu & Fergus, 2021). Following the protocol proposed by Raileanu et al. (2020), we conduct set of experiments on the easy setting of all environments available in Procgen. All agents were trained during 25M steps on 200 training levels and their performance is compared on the entire distribution of levels (Appendix G reports results on the hard setting). We report the average sum of rewards for all environments in Table 2. The CLOP layer outperforms other state-of-the-aie methods on 7 out of the 16 Procgen games, sometimes by large margin. Compared to these methods, the CLOP layer offers a direct and easy way to augment the RL agent's 's ability generalize to unseen environments (Appendix E reports results on the combination with IDAAC).
CLOP improves both performance and generalization gap. Table 3 reports the gains obtained by the CLOP layer in terms of performance on training levels, testing levels and generalization gap, compared to PPO, after 25M steps of training on all Proegen games. Complete results reported Appendix F. The CLOP layer systematically improves the generalization gap (of 30% average and up to 50% StarPilot). It also has notable effect on both the training and the generalization absolute performance. On all games but three (CaveFlyer, Ninja, and Plunder), the training performance is at least equivalent (10 games), and sometimes much better (BigFish, Chaser, Leaper) with the CLOP layer, both in training speed and asymptotical performance. The CLOP layer's generalization absolute performance is also better on all but the same three games," 2 with significantly greater advantage in number of games. It reaches significantly better generalization performance on 11 games and equivalent performance on 2 games (Coinrun, Jumper).
sinnee the generalization gap is still reduced, conjecture it is matter of convergence of the training process, rather than of generalization ability
7

---

Published as a conference paper at ICLR 2022
Game
PPO train
PRO test
gap
CLOPP tmin
CLOP test
CLOP
biglish
18.1
13.8
0%
314 4%
14.2%
hosslight
10.3 1 0.6
9.1
1.2
+ -..9%
+6.2%
25.0%
caveltyer
7.8
0.4
2.3
15.4%
-7.9%
30.4%
chaser
8.0 1.2
6.8
1.1
+1888%
+26.7%
18.2%
climber
10.2 2.220..4 0.4
6.4
3.9
5.9%
16.6%
43.6%
coinrun
10.0
0.0
1.0
-0.4%
+1.6%
20.0%
dodgeball
10.8 10.8
3.4
7.5
+8.2%
+117.7%
10.0%
Iruitbot
28.5
+4.3%
40.0%
heist
8.8
2.7
6.1
+5.1%
+66.4%
21.3%
jumper
8.4
5.4
3.5
0.6%
+3.4%
55.7%
leuper
1.6
6.1
0.6
+39.1%
+41.1%
+16.7%
maze
9.1
5.1
4.8
1.5%
+4.99
20.8%
miner
1.2
8.4
4.3
+0.9%
+17.4%
-00.2%
ninja
9.6 0.3
6.1
3.1
16.6%
100.4%
29.0%
plunder
1.7
2.8
starpilor
44.7 + 2.4
36. 1.6
8.6
+2.1%
+3.3%
- -4.3%
Table 3: Performance training and testing levels, and generalization gap after 25M steps.
10
0.0
0.4
0.0
0.8
1.0
(a) Bigfish
(b) Dodgeball
(c) Chaser
Figure 5: Influence of the parameter on test performance.
Influence of the permutation rate. The CLOP layer has single tunable hyperparameter a, sponding to the probability of accepting a candidate permutation between pixels during the forward pass. Figure reports the effect of varying this hyperparameter on three environments where the CLOP layer improved generalization. Interestingly, large values of a do not prevent the agent from learning. some cases (Figure 5c), it might even lead improved performance. Since permuta- tions occur locally, a high probability of swapping does not imply the risk of generating latent states that are too far off from the original latent state (and would thus harm overall training performance). Figures 6c and 6d (see next paragraph) illustrate this risk. Another desirable property of this param- eter is that performance does seem to be too sensitive to Any non-zero value seems to permit generalization. Consequently, using the CLOP layer does not require fine hyperparameter tuning.
Locality and channel-consistenccy are crucial features for generalization. Figure illustrates the effect of forward pass through the CLOPP layer without these components on handcrafted, 3- channel (RGB) feature map on the Leaper game (Figure 6b). Although there is a non-zero probabil- ity that CLOP sends pixel's content arbitrarily far from its original position, the output of the layer retains an interpretable shape from a human perspective (Figure 6c). In contrast, Figure 6d illustrates not local anymore and can send any pixel's content anywhere in
the output if the permutations the image in a single permutation. Similarly, Figure 6e shows how loosing the channel-consistengy feature creates new colors in the image by recombining unrelated features together. Both these be- haviors seem undesirable for proper generalization. We performed an ablation study to assess the influence of these two components. Figure 7 shows that both the locality and channel-consistency properties of CLOP are beneficial to generalization (full results in Appendix H). Interestingly, non- local permutations and Channel-inconsisterr permutations still provide better generalization than plain PPO. Such image modifications are agnostic to the feature maps spatial structure. But since irrelevant features (e.g., background) can be swapped without consequences, and since these fea- tures concern majority of pixels, with high probability these modifications still generate samples that preserve the useful semantic information. However, whenever useful feature pixels are swapped without locality, this semantic information is made ambiguous between actual experience samples and augmented data, that might induce different action choices. This confirms the general idea that data augmentation helps generalization in RL, but that improved performance comes from preserv-
8

---

Published as a conference paper at ICLR 2022
(a) Original input
(h) Feature map
(c) CL.OP
(d) No locality
(c) No consistency
Figure 6: Applying the CLOP layer (a = 0.5). Ablation the locality and consistency properties.
clop
1st block
clop block
dlpp on actor and cribic
no chanel consistencyy
no locality
(a) Bigfish
(b) Dodgeball
(c) Chaser
Figure Ablation study, effect on the score
15
(a) CLOP
(b) No locality
(c) CLOP
(d) No locality
CLOP (e)
(f) No locality
Figure 8: Saliencies induced by the locality property (a = 0.5)
ing spatial knowledge (length, width, and channel-wise) in the latent space. We also used Grad-Cam (Selvaraju al., 2017) to produce visual explanations of the importance of these two factors (Fig- ure 8).3 An agent trained using a CLOP layer stripped of the local permutation feature focuses on a spread-out portion of the image, while the full CLOP agent displays very focused saliency areas. The policies of the PPO agents, or those of the CLOP agents with local but channel-inconsister per- mutations, display much more focused saliency maps too, despite having lower performance than the full CLOP agents. This underlines limit of saliency map interpretation: focussing specific pixels is important but it is the combination of these pixel values that makes a good decision and this process more intricate than solely concentrating important parts of the image.
CLOP layer's position in the network. Figure 7 also displays the loss in generalization perfor- mance when applying CLOP to earlier layers in the IMPALA architecture used, confirming that CLOP is most beneficial on the deepest feature maps (more results in Appendix I). It also reports the effect of applying CLOP of both the actor and critic of PPO (discussion in Appendix J).
CONCLUSION
In this work, we introduce the Channel-consistent LOcal Permutations (CLOP) technique, intended improve generalization properties convolutional neural networks. This stems from an original perspective on observational overfitting in image-based reinforcement learning, We discuss why observational overfitting structural risk minimization problem that should be tackled the level of abstract state representation features. To prevent RL agents from overfitting their decisions to in- game progress indicators and, instead, push informative, causal variables into policy features, we
Fuul videos available https: /bi 11/39999

---

Published as a conference paper at ICLR 2022
implement CLOP a feeward layer and apply it to the deepest feature maps. This corresponds to performing data augmentation at the feature level while preserving the spatial structure these features. The CLOP layer implies very simple operations that induce negligible overhead cost. Agents equipped with a CLOP layer set new state-of-the-ard reference on the Procgen generaliza- tion benchmark. Although generalization in RL is broad topic that reaches out well beyond obser- vational overfitting, the approach taken here endeavors to shed new light and proposes a simple and low-cost original solution this challenge, hopefully contributing useful tool the community.
REFERENCES
Christopher M Bishop. Neural networks for pattern recognition. Oxford university press, 1995.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215-223. 2011.
Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman. Quantifying generaliza- tion reinforcement learning. International Conference on Machine Learning. pp. 1282 1289. PMLR, 2019.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pp. 2048- 2056. PMLR, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 245-255. leee, 2009.
Terrance DeVries and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint /0t8:1770.04555 2017.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dumning, et Impala: Scalable distributed deep-rl with im- portance weighted actor-learner architectures. In International Conference Machine Learning, pp. 1407-1416. PMLR, 2018.
Jesse Farebrother, Marlos C Machado, and Michael Bowling. Generalization and regularization in dqn. arXiv preprint arXiv: /8/000///0.0//23, 2018.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning, 2016.
Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot trans- fer in reinforcement learning. In International Conference on Machine Learning, pp. 1480-1490. PMLR, 2017.
Jonathan J. Hull. A database for handwritten text recognition research. IEEE Transactions Pattern Analysis and Machine Intelligence, 16(5):550-554, 1994.
Maximilian Igl, Kamil Ciosek, Yingzhen Li, Sebastian Tschiatschek, Cheng Zhang, Sam Devlin, and Katja Hofmann. Generalization in reinforcement learning with selective noise injection and information bottleneck. Advances in Neural Information Processing Systems, 32:13978-13990, 2019.
Sergey loffe and Christian Szegedy. Batch normalization: reducing internal covariate shift. In International conference on machine learning, pp. 448-456 456. PMLR, 2015.
Accelerating deep network training by
Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2): 257-280, 2005.
Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi. Illuminating generalization in deep reinforcement learning through procedural level generation. In NeurlPS Workshop Deep Reinforcement Learning, 2018.
10

---

Published as a conference paper at ICLR 2022
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1097-1105 2012.
Anders Krogh and John A Hertz. A simple weight decay can improve generalization. In Advances in neural information processing systems, pp. 950 950-957, 1992.
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. Advances in Neural Information Processing Systems, 33, 2020.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-233) 1998.
Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: A simple tech- nique for generalization in deep reinforcement learningg. International Conference on Learning Representations, 2020.
Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
Tom M Mitchell. The need for biases in learning generalizations. Department of Computer Science, Laboratory for Computer Science Research .... 1980.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:13/2.5602 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 518(7540):529-533 2015.
Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta learn fast: A new benchmark for generalization in rl. arXiv preprint arXiv: 2018.
Charles Packer, Katelyn Gao, Jemnj Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning. arXiv preprint arXiv: 1810.1222, 2018.
David C Plaut, Steven Nowlan, and Geoffrey E Hinton. Experiments on learning by back propa- gation. Technical Report CMU-CS-8K 18-126, Carnegie University, 1986.
Martin I. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Emmanuel Rachelson and Michail G. Lagoudakis. On the locality of action domination in sequential decision making. In 11th International Symposium on Artificial Intelligence and Mathematics, 2010.
Roberta Raileanu and Rob Fergus. Decoupling value and policy for generalization in reinforcement learning, arXiv preprint arXiv:2102.10330 2021.
Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikow, and Rob Fergus. Auto- matic data augmentation for generalization in deep reinforcement learning. arXiv preprint arXiv:2006. /2862, 2020.
Aravind Rajeswaran, Kendall Lowrey, Emanuel Todorov, and Sham Kakade. Towards generalization and simplicity in continuous control. arXiv preprint 1rXiv.1703.02664 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (000:0777007777 2017.

---

Published as a conference paper at ICLR 2022
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations, 2015.
Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational overfit- ting in reinforcement learning. In International Conference on Learning Representations, 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, lly Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15()):1229-1988 2014.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-first AAAI con- ference on artificial intelligence, 2017.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba. and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Vladimir Vapnik. Principles risk minimization for learning theory. In Advances neural infor- mation processing systems, pp. 831-838, 1992.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference on machine learning, pp. (058-1066. PMLR, 2013.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, and Tao Qin. Generalizing to unseen domains: survey on domain generalization. In Proceedings of the Thirtieth International Joint Conference Artificial Intelligence, pp. 4627-4635, 2021a.
Kaixin Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. Improving generalization in reinforcement learning with mixture regularization. In NewrIPS, 2020.
Xudong Wang, Long Lian, and Stella X Yu. Unsupervised visual attention and invariance for rein- forcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6677-6687, 2021b.
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representa- tions, 2020.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.
Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pp. 6023-6032, 2019.
Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint arXiv:1806.07937 2018a.
Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learn ing invariant representations for reinforcement learning without reconstruction. In International Conference Learning Representations, 2020.
12

---

Published as a conference paper at ICLR 2022
Chiyuan Zhang, Oriol Vinyals. Remi Munos, and Samy Bengio. A study overfitting in deep reinforcement learning. arXiv preprint
2018b.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018c.
13

---

Published as a conference paper at ICLR 2022
A NETWORK ARCHITECTURE AND TRAINING HYPER-PARAMETERS
A.1 SUPERVISED LEARNING
We used a simple network for the MNIST-USPS experiment, composed of convolutional feature extractor:
Con kemel=ō 5, stride=1, padding-,, activation=Rel.)
Max Pooling(filters=2 stride=2)
4 kemel=5 x 5, stride=1, padding=2, activation=Rel.U)
Max Pooling(filter=22 x 2, stride=2)
Conv2D(fittrs=118,, kernel=3 x 3, stride=1, padding=2, activation=Rel.U)
followed by two hidden layer MLP:
(0treemnbbeeroos-51,,,2 activation=RelU)
mannsenn_neuroos=1000 activation=Rel.U)
Dense(nb_neuroneen aciaaiainiaiia
The dropout and the CLOP layers where added between the feature extractor and the classifier. For the natch-normalization configuration, we added batch-normalization layers after each convolutional layer. We trained all networks for 30 epochs with Adam, a learning rate = 10 and a cosine annealing schedule.
For the Imagenette experiment (and Imagewoof experiment of Appendix D). we used Pytorch's implementation of VGG11, replacing the output layer match the ten classes. For the STL-10 experiment of Appendix D, since the images are already downscaled, we used Pytorch's s implemen- tation of VGG9 network, replacing the output layer to match the ten classes. For the Imagenet a VGG16 network, replacing the
experiment of Appendix D we used Pytorch's 's implementation method,
output layer match the ten classes. When testing from the fully connected part of the network and added a CLOP layer between the last convolutional layer and the first fully connected one. As is common on these benchmarks, we applied horizon- tal stochastic flipping function to preprocess the input basic data augmentation. We trained all networks on 90 epochs with Adam, learning rate Ir 5 10 44 and cosine annealing schedule.
removed the dropout layers
A.2 REINFORCEMENT LEARNING
based our implementation of PPO on the one proposed by Hojoon Lee available at: https: // gi situub. com/ wwt.:/oooooooooyeee taeskkyttrring http://tttiiiipoggger.ee prroogen--ytorchh pytorch. We used IMPALA (Espeholt et al., 2018) architecture on all agents, with a CLOP layer added to the actor part only for our CLOP agents (except for the ablation study reported in Figure 7 and Appendix J where it is also included in the critic). Figure 9 illustrates CLOP agent's 's architecture. Values a used for each environment reported in Table 4.
14

---

Published as a conference paper at ICLR 2022
Experiment
Number of epochs/steps
Time by experiment
Number of experiment repetitions
Sec. 5.1 MINSTT and USPS
30
2 minutes
20
Sec. 5.1 Imagenette/ Imagewoof
90
40 minutes
10
ST
Sec. 5.2 Procgen Hard (for each enviroument)
200M
22 hours
Sec. 3.2 Proegen Easy (Ior ench envirmment)
23M
2h41
Table 5: Experimental setup
FC 15
1C
ReLU
Game
alpha
Bigfish
0.6
ReLU
StarPilot
0.3
Layer
FruitBot
0.3
BossFight
0.3
ReLU
Ninja
0.3
Residual Block
Plunder
0.3
CaveFlyer
0.3
Residual Block
[16.32.52] channels
CoinRun
0.3
stride
3x3,
Jumper
0.3
Chaser
0.8
Conv 3x3, stride 1
RaLU
Climber
0.3
Cony 3x3, stride
Dodgeball
0.6
ReLU
Heist
0.3
Leaper
0.8
Maze
0.3
Miner
0.3
Figure 9: CLOP layer within the IMPALA architecture
Table 4: Values a
B REPRODUCIBILITY
All the experiments from Section 5 were run desktop machine (Intel i9, 10th generation pro- cessor, 32GB RAM) with a single NVIDIA RTX 3080 GPU. Details about each experiments are reported in Table 5.
In a general context of data efficient machine learning research, it seems relevant to underline that each subfigure in Figure 4 evaluated 7 regularization methods, over 3 repetitions. Overall, this required an order of 7 x x 22 = 462 computing hours (around 20 days) for a single subfigure on high-end desktop machine. The experimental protocol used select the three games reported in Figure 4 relies on the results obtained on the easy mode of Procgen (Figure 10 and Table 2), on random sampling of hard mode games, and limited training length results on hard mode games.
Besides this information, we provide the full source code of our implementation and experiments, along with the data files of experimental results we obtained.
C COMPUTATIONAL COST OF THE CLOP LAYER
The CLOP layer has process each element of the intermediate feature map. but only width and height-wise. This results in a O(H'W") cost, where H' and W are the height and width of the feature maps, which can much smaller than that of the input image. important feature the independence on the channel depth, which can grow significantly in deep layers. Comparatively, methods that perform data augmentation in the input space generally suffer from complexities at least in O(CHW) (for input images of size C' H x W), and sometimes more for costly aug- mentation methods like rotations. Table reports two examples of the computational overhead of introducing the CLOP layer in PPO training. Table 7 performs similar experiment comparing the running times of various standard data augmentation techniques with CLOP on MNIST and Imagenette. This confirms the negligible overhead induced by the CL.OP layer.
15

---

Published as a conference paper at ICLR 2022
Without CLOP
With CLOP
Jumper easy mode Jumper hard mode Caveflyer easy mode Caveflyer hard mode
2h 40min
2h 41min
22h 55min
23h Imin
2h 40min
2h 41min
22h 25min
22h 34min
Table Computation time with and without clop on two Progen games
MNIST
Imagenette
No augmentation
6,6s
41.6s
Random crop
16.8s
41.7s
Rotation Cutout
18.1s
54.1s
6.9s
41.7s
Random convolutions
N/A
42.3s
CLOP
6.6s
41.7s
Table 7: Computation time of one forward pass on the entire dataset
D ADDITIONAL EXPERIMENTS IN SUPERVISED LEARNING
Following the experiments of Section 5.1, evaluated the benefit of the CLOP layer when training a VGG16 network on the full Imagenet database over 90 epochs, and compared with the reported improvement brought by mixup (Zhang al., 2018c) in the same setting. Mixup improves the generalization performance by 0.2 to 0.6% across network architectures (generalization accuracies without mixup between 76 and 80% depending negligible 0. 1.15% improvement (71.6% top-1 generalization accuracy without CLOP, versus 71.75% with CLOP). We conjecture the sample variety and diversity in Imagenet actually cancels the need for data augmentation altogether. To confirm this conjecture and refine the understanding of when CLOP is beneficial not, we ran the same experiment on several datasets with less variety in the images and less data. STL-10 (Coates et al., 2011) is a subset of Imagenet inpired by CIFAR- 10 (Krizhevsky, 2009), where each class has even fewer labeled training examples than in CIFAR- 10 (500 training images, 800 test images per class). Imagewoof is the subset of 10 breed classes from ImageNet. Training on Imagewoof or on Imagenette does not benefit from the diversity of image features present in the whole Imagenet dataset. In these cases, CLOP contributed most to close the generalization gap. as shown in Table 8. These results, along with the ones on the Imagenette dataset indicate that the CLOP layer becomes really beneficial when data is scarce and one needs to avoid overfitting.
the network used). Similarly, CLOP brings
the method that
E COMBINATION OF CLOP AND IDAAC
One advantage of CLOP is its simplicity and the possibility to combine it with other regularization methods. Since, at the time of writing. IDAAC is one the most efficient algorithms on Proc- gen, it is tempting evaluate its combination with CLOP. Table 9 reports this, all Procgen games, easy setting, after 25M training steps. CLOP was included in the IDAAC code provided in http:///tthbb.ommmraaemmmidaaa
STL-10
Imagewoof
Imagenette
train
test
train
train
without regularization
100 1 0.0
66.8 1.4
0.1
51.2 1 1.2
100
0.0
74.7 1.7
with CLOP with Mixup
100 0.0
100+000
71.9 = 1.6
100 - 0.0
83.8 0.6
99.9
65.1 =
46.1
25.1 - 19.2
100 - 0.0
76.6
with Dropout
100
70.6 1.4
0.1
63.2 1.5
100
- 0.0
0.5
with Batch norm
100 0.0
70.7 + 0.6
99.3 0.5
72.1
1.6
100 4 0.0
82.4
Table 8: Training and testing accuracy (in %) various scarce-data supervised learning tasks
16

---

Published as a conference paper at ICLR 2022
Game
IDAAC
CLOP
IDAAC
Bigfish
18.5 4 1.2
19.2 4.6
2..7
BossFight
0.6
9.7
9.5
CaveFlyer
0.6
0.3
5.0 0.4
Chaser
6.8
8.7
6.0 0.5
Climber
8.3
0.3
8.3
CoinRun
9.4
9.1
9.2 0.5
Dodgeball
3.3
1.2
3.4 0.4
FruitBot
2.5
2.3
26.8
Heist
3.5
4.2
3.6 0.8
Jumper
0.2
0.2
5.8 0.2
Leaper
7.7
9.2
6.5 2.4
Maze
0.3
0.2
5.6 0.2
Miner
9.5
0.3
9.9 0.1
Ninja
6.4
5.8
6.4 0.5
Plunder
2..3
5.4
3.2
StarPilot
2.3
10.9
36.3 1 3.5
Table 9: IDAAC+CLOP on testing levels
Overall, the combination neither seems beneficial or detrimental and the effects are rather game dependent. Being able to combine methods together has "cocktail effect" that does not nec- essarily imply that the combination will be systematically more efficient than the base elements. IDAAC+CLOP substantially improves IDAAC only in the Bigfish game. Conversely, it seems to degrade its performance on Plunder. On all other games, the improvement or degradation with respect IDAAC seems marginal. IDAAC's auxiliary tasks the policy network aim pre- dicting the advantage and not being able to predict the ordering between frames (which is strongly connected with predicting the value). This focuses the features essential information for the pol- icy, thus creating an inductive bias and, maybe removing the need for data augmentation. Depending on the game, swapping pixels that contain policy-relevan relevant information might thus have positive or negative impact on the generalization performance. On the other hand, our implementation of CLOP based on standard PPO, where policy and value networks are not separated and thus the fea- ture maps contain both policy and value-relevant features, and thus the improvement brought by CLOP is more important than on IDAAC. This discussion also sheds light on the cases when IDAAC features seemed insufficient to provide good policy, i.c. when PPO+CLOP better than IDAAC: in these cases, adding CLOP to IDAAC cannot help recover the performance of PPO+CLOP (e.g. Chaser, Dodgeball, Fruitbot, Heist, Leaper, StarPilot).
already
F FULL RESULTS ON PROCGEN'S 'S EASY MODE
Table 3 reported the gain in generalization gap between plain PPO and PPO+CLOP in percentages, for better readability. Specifically, it reported the gain in training accuracy, testing accuracy and generalization gap with respect those of the PPO baseline. We emphasize it gain in generalization gap with respect the PPO baseline, not the generalization gap itself.
In order to assess the generalization gap itself, we calculate it as the difference between training and testing levels, as percentage of the training performance. Table 11 reports the performance on training levels of all methods, borrowing the values mentioned in the work of Raileanu et al. (2020); Raileanu & Fergus (2021). This information is then combined with the performance on testing levels in order to assess the generalization gap of all methods, reported in Table 12. This generalization gap should read with the absolute generalization performance mind: a small generalization gap alone is not sufficient to characterize method.
Figure 10 illustrates the evolution both the performance and generalization gap along training, for PPO and PPO+CLOP.
In order to follow the protocol of (Cobbe et al., 2020) and for a comparison between CLOP and IDAAC, we also report normalized scores in Table 13, both training and testing levels. Note that
17

---

Published as a conference paper at ICLR 2022
Game
PPO train
PPO test
PPO gap
CLOP train
CLOP test
CLOP gap
bigfish
18.1
4.3 1.2
13.8
26.9
19.2 + 1.6
7.7
bossfight
10.3
9.1
1.2
0.6
0.1
0.9
caveflyer
1.
0.4
2.3
0.5
5.3
1.6
chaser
1.2
0.8
1.1
9.6
8.7
0.9
climber
10.2
6.3
3.9
9.2
7.3
2.2
coinrun
1.0
9.0
1.0
9.1
9.1
0.8
dodgeball
1.7
3.4
7.5
11.7
1.2
4.5
fruitbot
0.5
2.2
3.0
1.0
2.3
1.8
heist
8.3
2.7
9.3
4.2
4.8
jumper
8.4
5.4
3.5
0.3
5.2
3.3
leaper
1.1
1.1
0.6
9.2
9.2
0.7
maze
0.1
0.2
4.8
9.7
5.2
3.8
miner
1.2
8.4
4.3
1..8
9.8
3.0
ninja
9.6
6.1
3.1
8.0 0.5
5.8 0.4
2.2
plunder
8.9
6.1 0.8
2.8
0.8+1. 1.0
5.4
1.4
starpilot
2.4
36.1
8.6
2.8
40.9 1.7
4.7
Table 10: Performance on training and testing levels, and generalization gap after 25M steps
these scores should be taken with grain of salt since the normalizing Rmaz can be quite far from the actual top performance among algorithms (e.g. Starpilot, CaveFlyer, Dodgeball, BigFish). This has an effect of shrinking the apparent performance of some methods on average since it returns (somehow artificially) low normalized score on these games. These normalized scores are multiplied by 100 for better readability. Figure 11 displays the histograms of the difference between CLOPP and IDAAC on training and testing levels. The specific case of the Plunder game seems to be outlier where IDAAC vastly outperforms CLOP (and all other methods from the literature) on testing levels (by 70.2 normalized points). For all other games, on testing levels, IDAAC dominates by 0 to 15.4 normalized points on games, and CLOP dominates by to 22.9 normalized points on the remaining 10 games. The average on training levels also show an advantage for CLOP (+13.5 on average, without specific outliers). Again,
these 15 games shows CLOP being marginally better (+3.7). Results
against over-interpretation of these figures, that depend the normalization factors. We would also like to emphasize that these results compare the two top-performing methods (which both bring significant improvements with respect other baselines). Given these results, it also seems relevant to note that both IDAAC and CLOP bring different insights on what is important for regularization in RL, therefore, beyond the raw comparison, these figures should mainly open new questions as to what makes a method better in each case.
Tables 14, 15 and 16 follow the outline of Table and report the improvement of each method from the literature, with respect to the PPO baseline in terms of (respectively) training levels perfor- mance, testing levels performance, and generalization gap. We report them mainly for the sake of completeness.
G FULL RESULTS ON PROCGEN'S HARD MODE
CLOP's direct competitor, IDAAC, does not report results Procgen's hard mode. As indicated in Appendix B, each subfigure Figure 4 required around 20 days of computation on high-end machine.
The experimental protocol the Procgen paper implies testing on the full distribution of levels, including those used during training. Our choice of using only the testing levels makes the bench- mark even harder since no level seen during training is among our testing distribution. We argue our protocol has two benefits. We test only the generalization property, without any influence from the performance on training levels. Also, the levels selected for testing along the training process are precisely the same across algorithms.
18

---

Published as a conference paper at ICLR 2022
bigfish
bossfight
caveflyer
chaser
lest train
15
1,0
20
15 dodgeball
10
20
climber
coinrun
fruitbot
02
0.5
1.0
1.5
20
2.
05
10
15
20
25
0.
05
1.0 leaper
15
20
25
800
05
10
25 maze
15
28
heist
jumper
Score
10 plunder
miner
hinja
starpilot
timesteps
107
timesteps
107
timesteps
107
timesteps
1e7
Figure 10: Train and test learning curves in easy mode
3.0
2.5
3
Count
Count
2.0
2
1.5
1.0
0.5
0.0
400
-20
20
40
60
-60
-20
20
Difference
Difference
(a) Training levels
(b) Testing levels
Figure 11: Comparison of CLOP IDAAC's performance (data from Table 13)
19

---

Published as a conference paper at ICLR 2022
Game
PPO
Mixneg
Rand FM
UCB-DraC
IBAC-SNI
RAD
IDAAC
CLOPP
Bighish
18.1
1.30
0.3
12.8
19.1 0.8
13.2228
21.8
2.0
BossFight
10.3 0..
7.9 T.900.
0.7
8.1
7.7
81111
104101
10.6
CaveFlyer
7.8
6.6
Chaser
8.0
0.9
0.7
7.0
3.1
6.
7.5
9.6
Climher
10.2 - 0.4
7.5
7.5
8.6
7.1
9.3+11
10.2
9.6
CoinRun
10.0 0.0
9.5
9.6
9.4
9.6
9.6
9.1
9.9
Dodgeball
9.1
0.3
7.3
9.1
3.0
4.
11.7
FruitBot
0..5
2..
2.72-0.7
293=005
2.84
26.1
29.1
31.6
Heist
0.3
1.
0.0
6.2
0.7
6.2
1.3
0.3
Jumper
0.9
8.4
8.9
8.2+0.1
0.6
8.6
87+
8.9 0.3
Leaper
1.
322
3.2
5.0
2.7
1.
8.
9.9
Maze
9.9
8.7
8.9
8.5
8.2
8.4
6.4
9.7
Miner
1..
0.9
1.5
12.0=03
0.7
12.6
11.5=055
0.1
Ninja
8.9 0.3
8.2
0.6
8.0
8.3
8.9
8,9
8.0
Plunder
6.2
5.5 0.7
10.2
6.0
8.4
24.6
6.8
StarPilot
26.7
Table 11: Average returns on Procgen training levels. Bold: best agent; underlined: second best.
Game
РРО
Mixreg
Rand FM
UCB-DraC
IHAC-SNI
RAD
IDACC
CLOP (Ours)
Bigfish
76.24%
67%
90, 00%00%
28, 13%
81%
00%
15. 14%
28, 62%
BoshTight
11, 11.65%
+3,80% 80%
69, 69.64%
-3, 3.7%
87. 34%
2,,47%
5, 77%
8. 4.4%
CaveFlyer
49%
-1.61%
16, 92%
13. 79%
+39.03%
-0%
-19,
24, 24%
Chaser
75%
59%
50, 00%
10.00%
06%
-7.81%
33%
3.%8
Climber
38, 24%
8, 5,00%
29.33%
26, 74%
53%
81%
63%
22, 92%
CoinRun
10, 00%
-9, 17%
-3, 12%
-6.38%
99.38%
25.5%
4. 08%
8,08%
Dodgeball
44%
31%
88.37%
42.47%
85,11%
44, 00%
69%
46%%
FruitBot
52%
-8, 70%
-16 16, 10%
80.80%
- -15,99%
+4.60%
-4, 12%
70%
Heist
69. 32%
40, 91%
-0, 00%
53%
104, 17%
33,
-22 22%
51.61%
Jumper
33.
29, 41%
40, 45%
24.30%
65%
24, 24,42%
27.50%
37,
Leaper
45%
+65, 63%
73%
-4.00%
+151, 85%
12,21%
23%
-7.07%
Maze
48, 48%
40, 23%
10.11%%
88%
21, 95%
27,38%
50%
39, 18%
Miner
33, 86%
-5, 62%
-34, 19%
23. 33%
-5,88
-5, 40%
-17399%
-23, 44%
Ninja
32.29%
-17, 07%
-8%
17.50%
+10, 84%
42%
-3.
50%
Plunder
31, 46%
4, 84%
45.4
18.631%
65, 00%
+1. 19%
5,28%
20, 59%
StarPilot
19.24%%
+12, 89%
66.54%
9, 9,37%
65%
8, 8.49%
4, 15%
10,31%
Table 12: Generalization gap (difference between test and train performance)
Figure 12 shows the training curves along with the testing performance for CLOP and PPO. Overall, the main conclusions remain: CLOP dominates in a majority games and often improves the convergence speed, both in terms of training and testing performance.
Although was not possible to repeat the comparison the regularization methods of Figure (or of other methods) on all Procgen games, mixreg (Wang et al., 2020) does report results the hard setting. Wang et al. (2020) follow the protocol of Cobbe et al. (2020) that evaluate the agents the full distribution
on the full distribution of levels. Table 17 reports the compared performance of testing levels of PPO, mixreg, and CLOP in the hard setting, after 200M training steps. Here again, CLOP dominates (by a variable margin) mixreg 10 out of the 16 Procgen games.
H FULL RESULTS ON THE ABLATION STUDY
Figure 13 reports the effect of turning off the channel consistency constraint, the locality con- straint, on all Procgen games. On a majority of games (10 out of 16: Bigfish, Chaser, Climber, Dodgeball, Fruitbot, Jumper, Leaper, Maze. Miner, Starpilot), the trend shown Section 5.2 is con- fimmed: both locality and Shannel-consistenn are important for generalization performance. Three games stand-out as outliers (Bossfight, Coinrun, Heist), where turning off one of these constraints seemed to sometimes improve generalization performance. First of all, we note this improvement small compared to the improvement of CLOP upon PPO. So this does not question the interest of using CLOP overall. Secondly, we conjecture these improvements are rather game or training- dependent. Finally, it seems illegitimate to draw conclusions on the three last games (CaveFlyer, Ninja, Plunder) since PPO outperforms CLOP on them, with without these constraints. Interest- ingly, IDAAC also performs rather poorly on two games among these three (CaveFlyer and Ninja, where (BAC-SNI is the leading algorithm), while it dominates Plunder (Table 2).
20

---

Published as a conference paper at ICLR 2022
bigfish
bossfight
caveflyer
chaser
20.0
17.5
p20
2.5
0.0
climber
coinrun
loa
dodgeball
fruitbot
1.0
20
helst
jumper
leaper
maze
Score
miner
ninja
les
plunder
les
starpilot
Score
30.0
2.5
10
00
05
10 timesteps
15
22
00
05
10 timesteps
15
22
00
0.5
10
15
25
00
23
1.0
15
timesteps
timesteps
Figure 12: CLOP and PPO performance in hard mode.
21

---

Published as a conference paper at ICLR 2022
Training levels
Testing levels
Game
IDAAC
CLOP
Difference
IDAAC
CLOP
Difference
bigfish
53.3
66.4
+13.1
44.9
46.7
+1.8
bossfight
79.2
80.8
+1.6
74.4
73.6
-0.8
caveflyer
31.8
36.5
+4.7
17.6
17.6
+0.0
chaser
56.0
72.8
+16.8
50.4
65.6
+15.2
climber
77.4
71.7
-5.7
59.4
50.9
-8.5
coinrun
96.0
98.0
+2.0
88.0
82.0
-6.0
dodgeball
19.4
58.3
+38.9
9.7
32.6
+22.9
fruitbot
90.3
97.6
+7.4
86.7
92.3
+5.6
heist
15.4
89.2
+73.8
0.0
15.4
+15.4
jumper
81.4
84.3
+2.9
47.1
37.1
-10.0
leaper
75.7
98.6
+22.9
67.1
88.6
+21.4
maze
28.0
94.0
+66.0
12.0
18.0
+6.0
miner
87.0
98.3
+11.3
69.6
72.2
+2.6
ninja 83.1
69.2
13.8
50.8
35.4
15.4
plunder 46.7
9.0
37.6
73.7
3.5
-70.2
starpilot
58.7
70.1
+11.4
56.1
62.4
+6.3
Table 13: Normalized score (100x) Procgen Easy mode (graphical representation of the same data on Figure 11)
bigfish
bossfight
caveflyer
chaser
6.00
score
15
5.50
10
5.25
5.00
climber
coinrun
dodgeball
fruitbot
30.5
30.0
29.0
6.0
28.5
heist
jumper
leaper
maze
6.00
5.0
5.8
5.75
4.0
5.50
5.25
5.4
5.00
2.5
miner
ninja
plunder
starpilot
10.5
10.0
42
6.2
6.5
6.0
5.5
38
5.6
5.0
5.4
dop
10. I.C.CC
dop
no. C_C. no,
dop
no.5 D. no Jocal
dop
no. N0,C.C no local
Figure 13: Ablation study on all games.
22

---

Published as a conference paper at ICLR 2022
Game
PPO
Mixreg
Rand FM
UCB-DraC
IBAC-SNI
RAD
IDAAC
CLOP (Ours)
Righih
17.1%
66.8%
3%
+5.5%
27.1%
+20.4%
+48.6%
Bight
10.3 0.6
23.3%
45.6%
21.4%
23.3%
21.4%
+1.0%
+2.9%
CaveFlyer
7.8 1.4
20.5%
16.7%
25.6%
20.5%
23.1%
20.5%
16.4%
Chaser
8.0
-57.5%
65.0%
12.5%
-61.2%
20.0%
-6.2%
20.0%
Climber
10.2
26.5%
26.5%
15.7%
30.4%
+0.0%
CoinRun
0.0
5.0%
4.0%
6.0%
4.0%
4.0%
2.0%
110%%
Dodgeball
10.8
16.7%
60.2%
32.4%
13.0%
53.7%
51.6%
+8.3%
FruitBot
0.5
5.1%
7.3%
7.0%
6.7%
17.1%
7.6%
+0.3%
Heist
8.3
50.0%
31.8%
29.5%
45.4%
29.5%
48.8%
+5.7%
Jumper
0.4
-4.5%
+0.0%
7.9%
4.5%
33.%%
-2.2%
+0.0%
Leuper
7.1
54.9%
54.9%
29.6%
62.0%
31.0%
16.9%
39.4%
Maze
9.9
-2.1%
10.1%
14.1%
17.1%
15.1%
35.3%
-2.%
Miner
1.2
29.9%
7.9%
-5.5%
33.1%
0.8%
-9.%
+0.8%
Ninja
-14.6%
25.0%
16.7%%
13.5%
773%
77.%
-16.7%
Plunder
8.9
30.3%
38.2%
+14.%
33.6%
5.6%
+ +166.4%
23.6%
StarPilot
44.7
-35.8%
41.2%
25.9%
-003%
18.3%
13.6%
+2.0%
Table 14: Training levels performance versus PPO
Game
PPO
Mixreg
Rand FM
UCB-DraC
IBAC-SNI
RAD
DDAC
CLOP (Ours)
Bighih
1.311.2
103.12%
86.05%
113.93%
-1.10%
130%
330 23%
336.51
BissHight
9.1
-989%%
81.32%
14.29%
89.01%
13%1
+7.69%
+-6.59%
CaveFlyer
5.5
+10.91%
-182%
9..9%
+5.45%
-7.27%
-9.09%
-9.09%
Chaser
6.9
15.94%
79.71%
-870%%
81.16%
14.499%
-1.55%
+ 26.09%
Climber
6.3 0.4
+9.52%
15.87%
+0.00%
47..2%%
+9.52%
+31.75%
+17.66%
CoinRun
9.0
4.4%%
+3.33%
2.22
3333%
+0.00%
+4.4%%
+1.11%
Dodgeball
3.3 3.3::0..
48.48%
84.85%
+27.27%
-5.58%
15.15%%
-3.03%
+118.18%
FruitBot
2.2
-1.11%
-1.04%
-3.6%%
13.33%
-121%
-211%
+4.56%%
Heist
2.7
-3.70%
11..11%
+22.63%%
262.96%
+51.85%
+22.63%
66.67%
Jumper
5.4
+11.11%
-1.85%
+14.81%
33.33%
+20337%
+16.67%
+3.70%
Leuper
-18.46%
-..22%
26.155
+4.622%
-3385%
+41.54%
Maze
5.1
+1.9%%
56.886%
+23.53%
96.088
+19.61%
99.80%
+15.09%
Miner
6.4
+11.90%
8.33%
+9.52%
4.76%
+11.90%
+13.10%
+16.67%
Ninja
+4.62%
-6.5%
+1.4%
+41.54%
+6.15%
-4.22%
10.77%
Plunder
6.1
-3.88%
50.82%
36.07%
-6..77%
+39.31%
281.97%
-1.48%
StarPilot
36.1 1.111.6. 1.6
10.25%
- 75.62%
16.90%
86.43%
7.48%
2.49%
+ +1.300%
Table 15: Testing levels performance versus PPO
I APPLYING CLOP AT VARIOUS DEPTHS WITHIN THE NETWORK
The CLOP layer be applied after any convolutional pooling layer of neural network. Fig- ure 7 shows the impact of the CLOP layer's position, after different Resnet blocks, in the IMPALA architecture, evaluated on three Procgen games. We can observe that the CLOP layer has a more substantial effect when applied on the most deepest feature map. To verify whether this is also con- firmed in supervised learning, we trained two networks (architectures Appendix A) with CLOP layer at various depths, MNIST and STL-10. The results, reported in Figure 14 show that the position of the CLOP layer also has an impact on supervised learning. Similarly to the phenomenon observed in RL, in supervised learning, applying the CLOOP layer on the deepest features maps in- creases the gencralization capabilities of the networks in both experiments. This goes to confirm that data augmentation over abstract features (deep convolutional feature maps) is key to efficient generalization.
J APPLYING CLOP ON BOTH THE ACTOR AND THE CRITIC OF PPO
The results reported in Section 5.2 made use of CLOP only within the actor of PPO. It legitimate to wonder whether there is any interest in applying the CLOP layer also to the critic.
useful to note that PPO is a "true" policy gradient method, contrarily to SAC or TD3 that are approximate value iteration methods which include gradient ascent the policy to solve the greed- iness search (the arg max operator in the Bellman equation). Because this, it collects samples at every round to estimate the critic and cannot rely persistent replay buffer. Intrinsically, the critic is only used to compute advantage estimates states that have been encountered during re- cent exploration. Consequently, there no need for the critic to actually generalize unseen states
23

---

Published as a conference paper at ICLR 2022
Game
PPO
Mixseg
Rand FM
UCB-DraC
IBAC-SNI
RAD
DAAC
CTOP (Ours)
Bigtish
13.8
42.8%
60.9%
73.9%
+32,
76.1%
76.1%
44.2%
BossFight
1.2
125.0%
+225.0%
-75.0%
+475,0% 0%
83.3%
50.0%
25.0%
CaveFlyer
2.3
-9.7%%
-5..2%
66.5 -6.2%
178,3%
60.9%
-47.8%
30.4%
Chaser
1.1
318.2%
+27.3%
- -36.4%
+63. 6%
54.5%
-66.4%
-8.22%
Climber
3.9
84.6%
43.6%
-11.0%
-2,6%
388.5%
-51.3%
43.6%
CoinRun
1.0
10.0%
70.0%
40.0%
-00
40.0%
-0.0%
20.0%
Dodgeball
7.5
49.3%
58.77%
+6, 7%
-0..7%
77.3%
40.0%
FruitBot
3.0
13.3%
+ 55.77%
43.3%
+56. 7%
140.0%
60.0%
40.0%
Heist
6.1
70.5%
41.0%
55.7%
182, 0%
65 65.6%
83.6%
21.3%
Jumper
3.5
28. 28.6%
+.99%
42.9%
40, 0%
40.0%
31.4%
5.7%
Leaper
0.6
450.0%
600.0%
66.7%
783,3% 3%
+0.0%
+0.0%
+16.7%
Maze
4.8
27.1%%
-8.3%
-54.2%
137,5%
52.1%
-3.3%
20.8%
Miner
4.3
111.6%
-7.0 -7.0%
34.9%
-8, 4%
25.6%
53.5%
30.2%
Ninja
3.1
-4.8%
-4.5%
64.8%
129,0%
35.5%
33.3%
29.0%
Plunder
2.8
89.3%
-00.7%
-32.1%
+39, 3%
103.6%
-33.6%
-50.0%
StarPilot
8.6
143.0%
103.55%
64.0%
+153, 5%
64.0%
-814%%
-45.3%
Table 16: Generalization gap versus PPO
Game
PPO
Mixreg
CLOP
bigfish
1.7 4.0
1.2
13.14.4
bossfight
1..
1..1
11.7
caveflyer
4.6
5.2
4.8
chaser
9.4 0.2
7.7
9.9
climber
6.0
0.5
6.7
coinrun
6.9 0.3
6.6
7.2
dodgeball
4.6
0.4
10.9
fruitbot
5.4
15.5
15.7
heist
1.3
1.3
2.9
jumper
3.2
4.3
2.9
leaper
8.0
6.7
8.4
maze
2.7
4.0
2.3
miner
1.2
14.5
1.5
ninja
5.9
6.6
4.1
plunder
4.3
4.4
2.8
starpilot
0.6
1.2
8.3 1.4
Table 17: CLOP and mixreg test performance hard mode.
and letting the critic overfit the value function is actually beneficial. One could conjecture that this phenomenon of overfitted critic benefiting PPO, is also one reason for the success of DAAC (and IDAAC), since their critic is constrained share a common stem of convolutional layers with the actor and is thus free to better fit the values.
0.925
Test accuracy
Test accuracy
0.900
0.70
0.875
0.68
0.850
0.825
0.66
0.800
1st
2nd
3rd
4th
Sth
6th after
2nd
3rd
last pooling
(a) MNIST -USPS
(b) STL-10
Figure 14: Test accuracy for different positions of the CLOP layer
24

---

Published as a conference paper at ICLR 2022
Figure 7 reports the difference in performance between the two versions of PPO, confirming this analysis on a limited set Procgen games.
K WHAT WOULD BE THE EQUIVALENT OF CLOP IN NON-IMAGE-BASED RL?
This short discussion was motivated by comments from the ICLR reviewers.
CLOP is essentially image-based method that exploits the spatial structure of images. Local permutations are a way of performing local "fuzzying". When such fuzzying is done the input space it aims invariance with respect measurement noise. The rationale behind the CLOP layer is that performing such a fuzzying in feature map space is a principled way to aim at object positions invariance. Consequently, one could see the encoding of the feature maps isomorphic to a latent space of positional variables. In this latent space, the equivalent of our permutations would correspond (at least in the intention) the application of small, centered noise on positional variables. so as to reach positional invariance. From a theoretical perspective, one could interpret this process margin maximization problem (the margin being defined the feature space rather than in the input space). This is an interesting avenue for future research and this short discussion applies more generally to any noise addition process in data augmentation.
25


Reviews:
[{"review_id": "zzvBEkBCG-l", "reviewer": "Reviewer_2gnP", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS\n\nI want to start by acknowledging that the authors provided a version of the code used in their experiments which was very easy to navigate through and understand.\n\nThe main strength of the regularization method introduced in this work is in its simplicity. Swapping dimensions of an intermediate feature map while keeping channel-wise consistency does not require heavily engineered functions  and is potentially applicable besides images to other visual inputs like videos. The only introduced parameter in the method, alpha, is the one that controls the probability of randomly swapping an element of the feature map with a random neighbour. Generalization seems not to be sensitive to the value of alpha as long as it is non-zero, implying the lack of necessity for heavy hyperparameter tuning.\n\nAnother important contribution of this work is in the improved generalization performance obtained in a supervised learning as well as a RL setup. In the former, CLOP improves accuracy on two vision dataset. For the latter, the authors validate CLOP on a set of procedurally generated RL environments. Compared to other commonly used regularizations, their method ranked first or second in terms of generalization performance on 10 out of 16 environments.\n\n\nWEAKNESSES\n\n\nThroughout the paper, CLOP computational complexity is defined as negligible without any explanation. Does CLOP have to process each element of the intermediate feature map, thus scaling with a quadratic cost assuming a constant cost for a swap operation, and is considered a negligible overhead given that features maps usually have low dimensions?\nWithout a proper cost analysis it is difficult to judge the tradeoff of using CLOP and I am not in favor of accepting this work without any discussion of the reportedly negligible cost.\n\nI agree with the claim that the type of spatial regularization that CLOP tries to inject works at higher layers of neural networks. However, I’m questioning the robustness of using CLOP across different layers. Someone wishing to use CLOP in an RL setup would greatly benefit from an analysis of its performance across different parts of the model. Did the authors try applying CLOP on a deep layer that is not the last one in the model architecture? \nWith evidence of the robustness of the proposed method I am in favor of vouching for the acceptance of the paper.\n\nIn this supervised learning results (section 5.1, table 1), CLOP seems to be similarly beneficial when using MNIST as well as Imagenette, with the latter being a visually richer dataset. \nMoreover, Imagenet(te) pictures are iconic, with the target object often clearly visible at the center of the image. For this reason, being able to achieve things like background invariance, possibly induced by CLOP, should be more beneficial for Imagenette compared to MNIST/USPS. This makes me wonder about the scalability of CLOP to richer vision datasets. Did the authors try to validate their results on a larger-scale experiment (e.g. ImageNet or CIFAR100). While this is not a fundamental weakness, I think the paper would benefit by the inclusion of this experiment. On this topic, why is the paper presented mainly as a reinforcement learning regularization method if supervised learning results are also reported?\n\n----\n\nMinor Suggestions or enhancements:\n\n\nThe generalization taxonomy in section 2 is rather confusing, especially given that the paper later focuses on the “simplest” type of generalization among those presented. This is the problem of generalizing within a single MDP and is defined as observational overfitting (OO) at the end of section 3. However, when first introduced quoting Song et al. (2019), OO is defined as the generalization to a distribution over MDPs and, even if presented in the text, it is not clear how the generalization phenomenon studied in this work departs from Song’s formulation. \n\nIn the code, I noticed that the Imagenette experiments used a stochastic flipping function to preprocess the input. This should be mentioned in the paper, possibly in a footnote.\n\nWhile I appreciate the authors explicitly reporting standard deviation values in their results, I did not find mentions of the number of repeated runs for each experiment until table 6 in the appendix. I suggest the authors briefly report this in the main text or point to the appendix for a better interpretation of the presented results.\n\nIn the “observational overfitting in RL paragraph”, OO is mentioned (“Therefore, preventing OO in RL remains the ability etc.)” without being properly defined until later in the text (“Song et al. call the problem of OO etc.”).\n\nThere could be more consistency in the choice image encoding: in section 4, an image is described as C x H x W, then in section 5.2 an image is described as 64 x 64 x 3.\n\nAcross plots and tables, except for the supervised learning results, the “no regularization” setup is defined as PPO and compares to CLOP and several others regularization/augmentation strategies. I find it clearer to define said setup as something like “no regularization” or “plain setup” or something of the sort.\n\nI was able to grasp the main message behind table 3 but I found the results on the CLOP gap slightly confusing. It might be worth mentioning in the caption how the gap is computed.\n", "summary_of_the_paper": "In this work, a novel regularization method for deep neural networks is introduced. By locally swapping dimensions of intermediate feature maps, the authors report generalization improvements in supervised learning and on several reinforcement learning benchmarks. The method is evaluated against several other common regularization techniques and was found to lead to better results.", "summary_of_the_review": "Overall, the paper is clearly written and easy to follow and I find the idea behind CLOP simple and effective. However, concerns regarding scalability and robustness of the proposed method should be addressed by the authors. I am happy to raise my score and vote for the inclusion of this work at ICLR provided that authors report evidence of a deeper empirical analysis.\n\n\n===============\n\nUPDATE: I raised my score from 5 to 8. I think that the analysis conducted by the authors have drastically improved the paper. I believe it meets the criteria to be included in the conference. The main contributions of this work are the introduction of a new regularization method (CLOP) for image-based RL and supervised learning tasks as well as a detailed empirical analysis and evaluation on several standard benchmarks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "iZQmVe5_1vK", "reviewer": "Reviewer_VitZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n* As the authors have mentioned, this is a very simple regularization method which achieves good empirical results. From the point of view of judging it alongside other image-based regularization techniques (e.g. cutout, data augmentation) and even RL-specific upgrades, this method seems particularly effective.\n* The paper is quite comprehensive in its showing of results (over classification, Procgen + baselines, saliency, visualizations), and via ablations over locality of permutations, show that the method does take advantage of the locality of objects in images strongly. I have a good empirical understanding over the method.\n\nWeaknesses:\n* One incredibly small detail that I caught (but I think may matter a lot) in Figure 9, Appendix A.2 is the fact that the CLOP layer isn't used in the image torso shared by both the value and policy heads, but rather only the policy head itself. This makes me somewhat suspicious that one hypothesis is maybe CLOP just affecting the policy's action outputs, leading to random actions during training and thus an exploratory behavior/\"data augmentation\" that's leading to the higher test performance. This is even more confounded because of the separation between forward passes (for data collection) vs backward passes (where the value head is still affected by CLOP since the image encoder's gradient is affected by CLOP via the policy head). Could you please provide an explanation for why you only used CLOP on the policy head, or add a few extra ablations to refute my proposed hypothesis? For example, what if CLOP is applied at the very end of the image encoder torso, but still shared by both heads? It is not a fatal issue if my proposed hypothesis is true, but analyzing this direction would substantially clarify why CLOP is effective. Apologies if I may be overreacting.\n \n* Since this is a vision-based method, it is naturally difficult to provide more fine grained conceptual analysis on the CLOP regularization method, and thus this paper might not be as impactful to say, someone in the theoretical field as someone in the computer vision field. From the point of view of computer vision, this is a good contribution, but it seems slightly unsatisfying from a theoretical perspective. The method relies on the inductive bias where objects in images are localized, which is true for a large distribution of images. How would this method work on scenarios where this is not the case, especially ones where data other than RGB is used? What would happen if the data also involves e.g. [1] 2D xy coordinates appended into the depth channel, or standard positional embeddings for images sent to a Vision Transformer [2]? It is not obvious to me if CLOP would work here. I wonder if it may be possible to learn the distribution of random permutations during training instead for these non-RGB scenarios.\n\n[1] https://eng.uber.com/coordconv/\n\n[2] https://arxiv.org/abs/2010.11929", "summary_of_the_paper": "This paper proposes a new regularization scheme, which involves randomly permuting nearby (in terms of spatial locations in the downsized feature tensor) depth blocks. The authors show its effectiveness in supervised learning tasks and establish a SOTA-like result over Procgen generalization, and also provide several empirical ablations and visualizations over the method to understand its inner workings. The paper shows that regularization which involves the locality of objects is particularly effective.", "summary_of_the_review": "The proposed CLOP method is simple and effective, although I'm afraid it can (to some readers) seem like a (somewhat motivated) \"hack\", which isn't a huge flaw, but might be limited in impact scope. It is strong to the eyes of an experimentalist, but raises many questions to someone who may be more theoretical.\n\nI am willing to increase the score if the authors answer my main question about the policy head, which made me somewhat alert/suspicious.\n\n\nEDIT: I have increased my score to an 8 after the rebuttal. The authors have answered my questions thoroughly. Previously, I was concerned that the CLOP method might be a bit too \"hacky\" (for lack of a better word), but I think the paper's simplicity and strong empirical results may also lead to how we redefine RL generalization. Perhaps Procgen tests too much on \"visual generalization\" and thus this paper might be providing some evidence for new benchmarks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fe0HnZZRijb", "reviewer": "Reviewer_8Mbh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n\n* The proposed technique is simple and effective.\n* The paper is easy to follow.\n\n**Weaknesses:**\n\n* The paper aims at generalization in RL. However, the proposed method seems more like a general data augmentation technique for visual input and CNN. The motivations in section 4 are only loosely connected to RL. For example, \"the important information is often spatially scarce and very localized\" also applies to image classification.\n* It would be better to use the same experiment protocol as in the Procgen paper (full distribution of levels for testing instead of 1000, for hard difficulty). Also, the choice of the three environments (Dodgeball, Miner, Chaser) seems arbitrary. In addition, it would be better to include comparison with other methods that also use hard difficulty such as mixreg.\n* The results of IDAAC and CLOP in table 2 seem comparable. Though the improvement may be complementary, it is better to justify with experiments (i.e., IDAAC + CLOP)\n* To enable direct comparison, it would be better to compute a normalized score, as suggested by the Procgen paper.\n* Why table 3 only includes the comparison of CLOP and PPO? The training and testing results for IDAAC / mixreg / UCB-DraC are available in IDAAC paper.\n* To make the results more convincing, it would be better to conduct ablation experiments (Figure 5 and Figure 7) on all environments.", "summary_of_the_paper": "This paper introduces an augmentation technique: channel-consistent local permutations (CLOP), to help address observational overfitting issue in RL. The proposed method is evaluated on Procgen benchmark and outperforms other methods.", "summary_of_the_review": "Overall, I think the proposed augmentation technique is simple and effective when compared to original PPO and other data augmentation techniques, but it does not significantly outperforms state-of-the-art IDAAC. I have some concerns on its connections to RL and some experiment details. Therefore, at this moment I lean towards rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Fyz374A6FM1", "reviewer": "Reviewer_rG3k", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n- The empirical results are comprehensive with great ablation studies. The presentation of the method and the intuition behind it is clear and easy to follow.\n\n## Weaknesses\n- Presentation issues\n  - The acronyms for the baselines in Table 2 need to be connected to the paper references.\n  - Figure 1: it is unclear to me how many pairs of $(h, w)$ are being sampled from $P$. As far as I could tell, this piece of information is also not provided in the rest of the paper. By looking at Figure 2, it seems likely that the for loop is looping over all spatial location of the feature map. It would be good to get clarification from the authors on this. \n\n- Weak experimental results\n  - Compared to previous baselines, the proposed method is only better on 7 out of 16 Procgen games. IDAAC is quite competitive as it performs the best on 4 of 16 Procgen games. The gap is even smaller when we are looking at the number of tasks that these approaches get top 2 on. IDAAC is top 2 for 9 out of 16 games and CLOP is top 2 for 10 out of 16 games. \n  - The authors mentioned that \"Compared to these methods, the CLOP layer offers a direct and easy way to augment the RL agent’s ability to generalize to unseen environments, with the benefit of being entirely complementary with each of them.\" The combination of the CLOP layer and the previous approach could potentially be quite promising but unfortunately there is no experiment that demonstrates whether CLOP can *actually* bring benefits to the previous approaches. \n\n## Other comments\n- In Section 3 last paragraph, the authors stated that \"Our method approaches the augmentation of data by noise injection at the feature level, thus avoiding the computationally costly operations of high-dimensional image transformation in regular data augmentation\" - From my understanding, data augmentations usually cost a negligible amount of time compared to running the image through the network itself. \n\n", "summary_of_the_paper": "The paper introduces the Channel-consistent LOcal Permutations (CLOP), which randomly swap adjacent features spatially after the last convolution layer in a neural network. Empirical evaluations show that CLOP can improve the generalization of convolutional networks on both supervised and reinforcement learning settings. ", "summary_of_the_review": "The empirical evaluations of the paper are thorough and informative. Although the experimental results are not the strongest (marginal improvements over baselines), they do reflect some effectiveness of the approach despite its simplicity. Because of that, I believe the paper deserves a spot in the conference.\n\n=======================================================================\nUPDATE:\n\nThanks the authors for their response. All of my concerns have been thoroughly addressed. Even though I still think that the improvements of CLOP over previous baselines are not that large, the insights provided by additional experiments and empirical analyses can be quite valuable to the community. Because of that, I have raised my score to 8.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5sBMp9_XDK", "reviewer": "Reviewer_JpKh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- A novel technique (to my knowledge) for CNN regularization, a topic that is well-studied for many years now\n- Strong experimental design and results, spanning supervised and reinforcement learning settings, particularly, generalization across levels.\n\nWeaknesses and clarification questions:\n\n- The supervised learning results are a bit superficial, and don't evaluate against state-of-the-art approaches like Mixup, and on more standard tasks like the full ImageNet task. I do appreciate that that is not the aim of the authors, but since the proposed method itself doesn't have much to do with RL, and the impact of the paper would be broader if it added more thorough supervised learning experiments too, to establish the proposed approach as the state of the art for CNN regularization in general vision tasks.\n\n- Fundamentally, augmentation is a way of expressing known invariances to \"label-preserving transformations\" in the data. We normally perform augmentation in the input space because this is where we know the invariances. In a latent feature space many layers deep into the CNN, we don't know what is represented. What does it mean to do augment the data here? How do we know that moving the features won't change the optimal action? For example, in a driving scenario, if CLOP moves a pedestrian in a scene from the pavement onto the road, shouldn't the driving agent respond significantly differently? Have the authors observed systematic ill effects from CLOP in some settings?\n\n- How does the layer to which CLOP is applied affect performance? Besides the prescription of applying it to the last convolutional layer, some empirical results addressing this would be useful.\n\n- The writing is sometimes a bit convoluted. The text in Sec 2 Generalization in RL, I found more confusing than illuminating. ", "summary_of_the_paper": "This submission proposes a simple new regularization technique for convolutional neural networks: \"channel-consistent local permutations\" (CLOP): at a deep layer in a CNN, it randomly swaps some pixels with their neighbors in the forward pass during training. This is shown to outperform other approaches for regularization, both in supervised learning and in reinforcement learning settings, showing particular benefits for generalizing across levels of procedurally generated games.", "summary_of_the_review": "An interesting and neat approach to regularize CNNs, motivated by the need to learn task-relevant representations invariant to background and other distractions for visual RL tasks. Strong results, well-designed experiments, and well presented. I am in favor of accepting this paper, but await the author responses and other reviewers' comments to form a stronger opinion.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a novel regularization technique for CNNs based on swapping feature vectors in the final layer. It is demonstrated that this simple technique helps with generalization in supervised learning and RL with image inputs.
Following the author rebuttal, all reviewers agreed that the simplicity of this method and the nice empirical performance it obtains is important to report to the community. In this respect, I agree with the reviewers, and recommend acceptance.

One important issue that came up during the discussion is how much this work is related to RL, and the authors SL experiments helped to put the contribution in a broader context. Indeed, one way to see the results of this work is that if such performance improvement is obtained in the Procgen benchmark with just image-based regularization, perhaps this benchmark is not very suitable for studying generalization in RL (where we expect that more sophisticated techniques would be required). In addition, I can think of RL domains (e.g., Tetris, which was mentioned in the discussion) where I would not expect the proposed method to help. It would be good if the authors discuss these issues in some capacity in their final version.

Please take all reviewer comments into account when preparing the final version.
Decision: Accept (Poster)