Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning
Authors: Kimin Lee
Kibok Lee
Jinwoo Shin
Honglak Lee
Venue:  (2020)
Keywords: Deep reinforcement learning
Generalization in visual domains

Abstract: Deep reinforcement learning (RL) agents often fail to generalize to unseen environments (yet semantically similar to trained agents), particularly when they are trained on high-dimensional state spaces, such as images. In this paper, we propose a simple technique to improve a generalization ability of deep RL agents by introducing a randomized (convolutional) neural network that randomly perturbs input observations. It enables trained agents to adapt to new domains by learning robust features invariant across varied and randomized environments. Furthermore, we consider an inference method based on the Monte Carlo approximation to reduce the variance induced by this randomization. We demonstrate the superiority of our method across 2D CoinRun, 3D DeepMind Lab exploration and 3D robotics control tasks: it significantly outperforms various regularization and data augmentation methods for the same purpose.

Published as a conference paper at ICLR 2020
NETWORK RANDOMIZATION: A SIMPLE TECHNIQUE FOR GENERALIZATION IN DEEP REINFORCEMENT LEARNING
Kimin Leelxt, Kibok Lee"*, Jinwoo Shin', Honglak Leea2 KAIST, UUniversity of Michigan, *Google Brain
ABSTRACT
Deep reinforcement learning (RL) agents often fail to generalize to unseen envi- ronments (yet semantically similar to trained agents), particularly when they are trained on high-dimensional state spaces, such as images. In this paper, pro- pose simple technique to improve a generalization ability deep RL agents by introducing randomized (convolutional) neural network that randomly perturbs input observations. It enables trained agents to adapt to new domains by learning robust features invariant across varied and randomized environments. Further- more, we consider an inference method based on the Monte Carlo approxima- tion reduce the variance induced by this randomization. We demonstrate the superiority of our method across 2D CoinRun, 3D DeepMind Lab exploration and 3D robotics control tasks: it significantly outperforms various regulariza- tion and data augmentation methods for the same purpose. Code is available at github. com/ ttt.ai.ccom/ppaaappko/a////////a wwwpppokaaaae....e.oppp nerdd...
INTRODUCTION
Deep reinforcement learning (RL) has been applied to various applications, including board games (e.g., Go (Silver et al., 2017) and Chess (Silver et al., 2018)), video games (e.g., Atari games (Mnih et al., 2015) and StarCraft (Vinyals et al., 2017)), and complex robotics control tasks (Tobin et al., 2017; Ren et al., 2019). However, it has been evidenced in recent years that deep RL agents of- ten struggle to generalize to new environments, even when semantically similar to trained agents (Farebrother al., 2018; Zhang et al., 2018b; Gamrian & Goldberg, 2019; Cobbe et al., 2019). For example, RL. agents that learned a near-optimal policy for training levels in a video game fail to perform accurately in unseen levels (Cobbe et al., 2019), while human can seamlessly generalize across similar tasks. Namely, RL agents often overfit to training enviromments, thus the lack of gen- eralization ability makes them unreliable in several applications, such health care (Chakraborty & Murphy, 2014) and finance (Deng et al., 2016).
The generalization of RL agents be characterized by visual changes (Cobbe et al., 2019; Gam- rian & Goldberg, 2019), different dynamics (Packer et al., 2018), and various structures (Beattie al., 2016; Wang et al., 2016). In this paper, we focus on the generalization across tasks where the trained agents take various unseen visual patterns at the test time, e.g., different styles of back- grounds, floors, and other objects (see Figure 1). We also found that RL agents completely fail due to small visual changes! because it is challenging to learn generalizable representations from high-dimensional input observations, such as images.
To improve generalization, several strategies, such as regularization (Farebrother et al., 2018; Zhang et al., 2018b; Cobbe et al., 2019) and data augmentation (Tobin et al., 2017; Ren et al., 2019), have been proposed in the literature (see Section 2 for further details), In particular, Tobin et al. (2017) showed that training RL agents in various environments generated by randomizing rendering in a simulator improves the generalization performance, leading better performance in real environments. This implies that RL agents can learn invariant and robust representations diverse * Equual contribution.
Work done while at University of Michigan. Demonstrations on the CoinRun and DeepMind Lab exploration tasks available at [1 11ik-videe]]

---

Published as a conference paper at ICLR 2020
Seen
Seen
Seen
Unseen
Unseen
Unseen
Random
Layer
(a) Outputs of random layer
(b) 2D CoinRun
(c) 3D DeepMind Lab
(d) 3D Surreal robotics
Figure 1: (a) Examples of randomized inputs (color values in each channel normalized for visualization) generated by re-initializing the parameters of a random layer. Examples of seen and unseen environments on (b) CoinRun, (c) DeepMind Lab, and (d) Surreal robotics control.
input observations are provided during training. However, their method is limited by requiring physics simulator, which may not always be available. This motivates our approach of developing a simple and plausible method applicable to training deep RL agents.
The main contribution of this paper is to develop a simple randomization technique for improving the generalization ability across tasks with various unseen visual patterns. Our main idea is to utilize random (convolutional) networks to generate randomized inputs (see Figure (a)), and train RI. agents (or their policy) by feeding them into the networks. Specifically, by re-initializing the parameters of random networks at every iteration, the agents are encouraged to be trained under a broad range of perturbed low-level features, e.g-, various textures, colors, or shapes. We discover that the proposed idea guides RL agents to learn generalizable features that are more invariant in unseen environments (see Figure 3) than conventional regularization (Srivastava et al., 2014; Ioffe & Szegedy, 2015) and data augmentation (Cobbe et al., 2019; Cubuk et al., 2019) techniques. Here, we also provide an inference technique based on the Monte Carlo approximation, which stabilizes the performance by reducing the variance incurred from our randomization method at test time.
We demonstrate the effectiveness of the proposed method on the 2D CoinRun (Cobbe et al., 2019) game, the 3D DeepMind Lab exploration task (Beattie et al., 2016), and the 3D robotics control task (Fan et al., 20.8). For evaluation, the performance of the trained agents is measured in unseen environments with various visual and geometrical patterns (e.g.. different styles of backgrounds, ob- jects, and floors), guaranteeing that the trained agents encounter unseen inputs at test time. Note that learning invariant and robust representations against such changes is essential generalize unseen environments. In our experiments, the proposed method significantly reduces the generalization gap in unseen environments unlike conventional regularization and data augmentation techniques. For example, compared to the agents learned with the cutout (De & Taylor, 2017) data augmenta- tion methods proposed by Cobbe al. (2019), our method improves the success rates from 39.8% to 78% under 2D CoinRun, the total score from 55.4 to 358.2 for 3D DeepMind Lab, and the total score from 31.3 to 356.8 for the Surreal robotics control task. Our results can be influential to study other generalization domains, such as tasks with different dynamics (Packer et al., 2018), as well as solving
problems, such sim-to-real transfer (Tobin et al., 2017).
RELATED WORK
Generalization in deep RL. Recently, the generalization performance of RL agents has been in- vestigated by splitting training and test environments using random seeds (Zhang et al., 2018a) and distinct sets of levels in video games (Machado al., 2018: Cobbe et al., 2019). Regularization is one of the major directions to improve the generalization ability of deep RL algorithms. Fare- brother et al. (2018) and Cobbe et al. (2019) showed that regularization methods can improve the generalization performance of RL agents using various game modes of Atari (Machado et al., 2018) and procedurally generated arcade environments called CoinRun, respectively. On the other hand, data augmentation techniques have also been shown to improve generalization. Tobin et al. (2017) proposed a domain randomization method to generate simulated inputs by randomizing rendering in the simulator. Motivated by this, Cobbe et al. (2019) proposed a data augmentation method by modifying the cutout method (DeVsies & Taylor, 2017). Our method can be combined with the prior methods further improve the generalization performance.
2

---

Published as a conference paper at ICLR 2020
Random networks for deep RL. Random networks have been utilized in several approaches for different purposes in deep RL. Burda et al. (2019) utilized a randomly initialized neural network to define an intrinsic reward for visiting unexplored states in challenging exploration problems. By learning to predict the reward from the random network, the agent can recognize unexplored states. Oshand et al. (2018) studied method to improve ensemble-based approaches by adding randomized network to each ensemble member to improve the uncertainty estimation and efficient exploration in deep RL. Our method is different because we introduce random network improve the generalization ability of RL agents.
Transfer learning. Generalization is also closely related to transfer learning (Parisotto al., 2016; Rusu et al., 201 2016a;b), which is used to improve the performance on a target task by transferring the knowledge from a source task. However, unlike supervised learning, has been observed that fine- tuning model pre-trained on the source task for adapting to the target task is not beneficial in deep RL. Therefore, Gamrian Goldberg (2019) proposed a domain transfer method using generative adversarial networks (Goodfellow al., 2014) and Farebrother et al. (2018) utilized regularization techniques to improve the performance of fine-tuning methods. Higgins et al. (2017) proposed a multi-stage RL, which learns to extract disentangled representations from the input observation and then trains the agents on the representations. Alternatively, we focus on the zero-shot performance of each agent test time without further fine-tuning of the agent's parameters.
NETWORK RANDOMIZATION TECHNIQUE FOR GENERALIZATION
We consider standard reinforcement learning (RL) framework where agent interacts with an environment in discrete time. Formally, at each timestep f, the agent receives a state 8; from the environment and chooses action de based on its policy The environment returns reward r: and the agent transitions to the next state 8t+1- The return R² = Ek= 7*re+* is the total accumulated rewards from timestep t with discount factor 7 [0, 1). RL then maximizes the expected return from each state St.
3.1 TRAINING AGENTS USING RANDOMIZED INPUT OBSERVATIONS
We introduce random network f with its parameters & initialized with a prior distribution, e.g.- Xavier normal distribution (Glorot & Bengio, 2010). Instead of the original input s, train an agent using a randomized input = f(;; Φ). For example, in the case of policy-based methods," the parameters 0 of the policy network are optimized by minimizing the following policy gradient objective function:
Crandon R.)D [- log (a:1f (st; 6) ;0) Re],
where D = {(s,, ae, R,)} is set of past transitions with cumulative rewards. By re-initializing the parameters of the random network per iteration, the agents are trained using varied and random- generated with various visual
ized input observations (see Figure (a)). Namely, environments patterns, but with the same semantics by randomizing the networks. Our agents are expected to adapt to new environments by learning invariant representation (see Figure 3 for supporting experiments).
To learn more invariant features, the following feature matching (FM) loss between hidden features from clean and randomized observations also considered:
Crandon = E,,ED [l]b (f( 0):
0) - h (sx; [13]
where ((-) denotes the output of the penultimate layer of policy म. The hidden features from clean and randomized inputs are combined learn more invariant features against the changes in the input observations.* Namely, the total loss is:
Crandon
crandon
+ Crandon
(3)
policy
where 3 > 0 is hyper-parameter The full procedure is summarized in Algorithm 1 in Appendix M.
Throughout this paper, focus high-dimensional state spaces, e.g., images.
Our method is applicable to the value-hased methods as well.
*FM loss has also been investigated for various purposes: semi-supervised learning (Miyato et al., 2018)
and unsupervised learning (Salimans et al., 2016; Xie al., 2019).
3

---

Published as a conference paper at ICLR 2020
Classification
Accuracy (%)
Method
Train (seen)
Test (unseen)
ResNet-18
95.0 24
40.3 + 1.2
ResNet-18 GR
96.4 + 1.8
70.9 + 1.7
ResNet-18 CO
95.9 + 23
41.2 1.7
ResNet-18 IV
91.0 2.0 2.0
47.1 + 15.1
ResNet-18 + CJ
95.2 + 0.6
43.5 + 0.3
ResNet-18 + ours
95.9 + 1.6
84.4 + 4.5
Training Data
Test Data
The classification accuracy (%) on dogs vs. cats dataset. The results show the mean and standard deviation averaged over three runs and the best result is indicated in bold.
Table I:
Figure 2: Samples of dogs vs. cats dataset. The training set consists of bright dogs and dark cats, whereas the test set consists of dark dogs and bright cats.
Details of the random networks. We propose to utilize single-layer convolutional neural net- work (CNN) as a random network, where its output has the same dimension with the input (see Appendix D for additional experimental results on the various types of random networks). To re- initialize the parameters of the random network, we utilize the following mixture of distributions:
P(ø) al(Φ I) a)N 0: V tia "entttonn where I is an identity kernel, a E [0. 1] is a positive constant, / denotes the normal distribution, and Rin, Yuut are the number of input and output channels, respectively. Here, clean inputs are used with the probability a because training only randomized inputs can complicate training. The Xavier normal distribution (Glorot & Bengio, 2010) is used for randomization because it maintains the variance of the input and the randomized input S. We empirically observe that this distribution stabilizes training.
Removing visual bias. To confirm the desired effects of our method, we conduct an image classi- fication experiment on the dogs and cats database from Kaggle. Following the same setup Kim et al. (2019), we construct datasets with an undesirable bias as follows: the training set consists of bright dogs and dark cats while the test set consists of dark dogs and bright cats (see Appendix H for further details). A classifier is expected to make a decision based on the undesirable bias, (e.g., brightness and color) since CNNs are biased towards texture or color, rather than shape (Geirhos et al., 2019). Table 1 shows that ResNet-18 (He et al., 201 does not generalize effectively due to overfitting to an undesirable bias in the training data. To address this issue, several image processing methods (Cubuk et al., 2019), such as grayout (GR), cutout (CO; DeVries & Taylor 2017), inversion (IV), and color jitter (CJ), can be applied (see Appendix C for further details). However, they are not effective in improving the generalization ability, compared to our method. This confirms that our approach makes DNNs capture more desired and meaningful information such as the shape by changing the visual appearance of attributes and entities in images while effectively keeping the semantic information. Prior sophisticated methods (Ganin et al., 2016; Kim et al., 2019) require additional information eliminate such an undesired bias, while our method does not. Although we mainly focus on RL applications, our idea can also be explorable in this direction.
3.2 INFERENCE METHODS FOR SMALL VARIANCE
Since the parameter of random networks is drawn from a prior distribution P(ø), our policy is modeled by stochastic neural network: [[aas; 0) E& π (alf (s; 0)] Based on this inter- pretation, our training procedure (i.e., randomizing the parameters) consists of training stochastic models using the Monte Carlo (MC) approximation (with one sample per iteration). Therefore, at the inference or test time, an action a is taken by approximating the expectations as follows: (a|s; 0) ~ N EM (al.f f (s; ((m)) :). where (((m) (6) and A is the number of MC samples. In other words, we generate M random inputs for each observation and then aggregate their decisions. The results show that this estimator improves the performance of the trained agents
by approximating the posterior distribution more accurately (see Figure 3(d)).
'https: / /www. k kaggl e. com/ c/ Eescogs-s--aar Usiig the known bias information (i.e., (dark, bright)) and ImageNet pre-trained model, Kim et al, (2019) achieve 90.3%, while our method achieves 84.4% without using both inputs.
4

---

Published as a conference paper at ICLR 2020
4 EXPERIMENTS
In this section, we demonstrate the effectiveness of the proposed method on 2D CoinRun (Cobbe al., 2019), 3D DeepMind Lab exploration (Beattic et al., 2016), and 3D robotics control task (Fan et al., 2018) 8). To evaluate the generalization ability, we measure the performance of trained agents in unseen environments which consist of different styles of backgrounds, objects, and floors. Due to the space limitation, we provide more detailed experimental setups and results in the Appendix.
4.1 BASELINES AND IMPLEMENTATION DETAILS
For CoinRun and DeepMind Lab experiments, similar to Cobbe et al. (2019), we take the CNN architecture used in IMPALA (Espeholt et al., 2018) as the policy network, and the Proximal Policy Optimization (PPO) (Schulman et al., 2017) method to train the agents." 7 At each timestep, agents given an observation frame of size 64 x 64 input (resized from the raw observation of size 320 x 240 as in the DeepMind Lab), and the trajectories are collected with the 256-step rollout for training. For Surreal robotics experiments, similar to Fan et al. (2018), the hybrid of CNN and long short-term memory (LSTM) architecture is taken as the policy network, and distributed version of PPO (i.e., actors collect a massive amount of trajectories, and the centralized learner updates the model parameters using PPO) is used to train the agents. 8 We measure the performance in the unseen environment for every 10M timesteps and report the mean and standard deviation across three runs.
Our proposed method, which augments PPO with random networks and feature matching (FM) loss (denoted PPO + ours), compared with several regularization and data augmentation methods. As regularization methods, we compare dropout (DO; Srivastava et al. 2014), L2 regularization (L2), and batch normalization (BN; loffe & Szegedy 2015). For those methods, we use the hyperpa- rameters suggested in Cobbe et al. (2019), which empirically shown be effective: dropout probability of 0.1 and a coefficient of 10-4 for L2 regularization. We also consider various data augmentation methods: variant of cutout (CO; DeVries & Taylor 2017) proposed in Cobbe et al. (2019), grayout (GR), inversion (IV), and color jitter (CJ) by adjusting brightness, contrast, and sat- uration (see Appendix C for more details). As an upper bound, we report the performance of agents trained directly on unseen environments, dented PPO (oracle). For our method, we use 8 = 0.002 for the weight of the FM loss, a 0.1 for the probability of skipping the random network, A = 10 for MC approximation, and single-layer CNN with the kernel size of 3 as random network.
4.2 EXPERIMENTS ON COINRUN
Task description. In this task, an agent is located at the leftmost side of the map and the goal is to collect the coin located at the rightmost side of the map within 1,000 timesteps. The agent observes its surrounding environment in the third-person point of view, where the agent is always located the center of the observation. CoinRun contains an arbitrarily large number of levels which are generated deterministically from a given seed, In each level, the style of background, floor, and obstacles is randomly selected from the available themes (34 backgrounds, 6 grounds, 5 agents, and 9 moving obstacles). Some obstacles and pitfalls are distributed between the agent and the coin, where a collision with them results in the agent's 's immediate death. We measure the success rates, which correspond to the number of collected coins divided by the number of played levels.
Ablation study on small-scale environments. First, we train agents on one level for 100M timesteps and measure the performance in unseen environments by only changing the style of the background, as shown in Figure 3(a). Note that these visual changes are not significant to the game's dynamics, but the agent should achieve high success rate if it can generalize accurately. However, Table 2 shows that all baseline agents fail to generalize to unseen environments, while they achieve near-optimal performance in the seen environment. This shows that regularization techniques have no significant impact on improving the generalization ability. Even though data augmentation tech- niques, such as cutout (CO) and color jitter (C.), slightly improve the performance, our proposed method is most effective because it can produce a diverse novelty in attributes and entities. Train- ing with randomized inputs can degrade the training performance, but the high expressive power
7 We used a reference implementation in https : //gi /ttpttttbbb/b//////////// thub. com/openai /cainrun. 'e used reference implementation with two actors in https::/ //g/ thub. www./ooreeeaeeeeeeeeeeeee 1A1/ surreal.
5

---

Published as a conference paper at ICLR 2020
PPO
PPO
PPO
PPO
PPO
PPO
PPO
PPO
PPO
DO
L2
BNN
+CO
+IV
GR
+ +C
Rand
Rand
FM
100
100
98.3
93.3
100
95.0
100
100
95.0
100
Seen
Success
+00
+ 7.1
34.6
25.3
34.1
31.5
41.9
37.5
26.9
43.1
76.7
78.1
Unseen
+
5.5
13.1
+
18.9
13.3
24.4
25.5
27.8
17.8
17.7
32,2
64.7
67.8
Cycle- consistency
2-way
+ 10.9
+ 2.2
+ 1.1
+ 6.6
+ 10.6
+ 15.6
+ 1.1
+ 3.1
+ 4.4
+ 6.2
4,4
4.4
8.9
7.4
9.6
5.6
2.2
15.6
39.3
43.3
3-way
+ 2.2
+ 22
+ 3.8
+ 1.2
+ 56
+: 4.7
+ 38
+ 3.1
+: 8.5
d:
Table 2: Success rate (%) and cycle-consistency (%) after 100M timesteps in small-scale CoinRun. The results show the mean and standard deviation averaged over three runs and the best results are indicated in bold.
80
Seen
79
78-
Average Succosss Reies
77-
78-
Unseen A
75-
74-
73-
Unseen B
Seon
Seen Unswen
70-
Unseen
10
Number of MC Samples
(a) Trajectories
(b) PPO
PPO + ours
(d) Effects of MC samples
Figure 3: (a) We collect multiple episodes from various environments by human demonstrators and visualize the hidden representation of trained agents optimized by (b) PPO and (c) PPO constructed by 1-SNE, where the colors of points indicate the environments of the corresponding observations. (d) Average success rates for varying number of MC samples.
of DNNs prevents from it. The performance in unseen environments can be further improved by optimizing the FM loss. verify the effectiveness of MC approximation at test time, we measure the performance unseen environments by varying the number of MC samples. Figure 3(d) shows the mean and standard deviation across 50 evaluations. The performance and its variance can be improved by increasing the number of MC samples, but the improvement is saturated around ten samples. Thus, we use ten samples for the following experiments.
Embedding analysis. We analyze whether the hidden representation of trained RL agents exhibits meaningful abstraction in the unseen environments. The features on the penultimate layer of trained agents visualized and reduced to two dimensions using t-SNE (Maaten & Hinton, 2008). Figure shows the projection of trajectories taken by human demonstrators in seen and unseen environments (see Figure 17 in Appendix N for further results). Here. trajectories from both seen and unseen environments are aligned on the hidden space of our agents, while the baselines yield scattered and disjointed trajectories. This implies that our method makes RL agents capable of learning the invariant and robust representation.
To evaluate the quality of hidden representation quantitatively, the secle-oonsistency proposed in Aytar et al. (2018) is also measured. Given two trajectories V and U, v; E V first locates its nearest neighbor in the other trajectory u, arg min [l((e;) A(u)ll², where ft-)) denotes the output of the penultimate layer of trained agents. Then, the nearest neighbor of u, in V is located,
i.c., = arg min.ev [|<(v) (u,))]? and v; is defined as cycle-consistent if j kl < 1. i.c., can return to the original point. Note that this cycle-consistency implies that two trajectories are accurately aligned in the hidden space. Similar to Aytar et al. (2018), we also evaluate the three-way cycle-consistency by measuring whether v; remains cycle-consistent along both paths, V V - + V and V + U > V. where J is the third trajectory. Using the trajectories shown in Figure 3(a), Table 2 reports the percentage of input observations in the seen environment (blue curve) that cycle- vycle-consisten with unseen trajectories (red and green curves). Similar to the results shown in Figure 3(c), our method significantly improves the cycle-consistency compared to the vanilla PPO agent.

---

Published as a conference paper at ICLR 2020
PPO
PPO + ours
Unseen Seen
Figure 4: Visualization of activation maps via Grad-CAM in seen and unseen environments in the small-scale CoinRun. Images are aligned with similar states from various episodes for comparison.
Success Ratce (9 Unsoon Enstrrnens S.00ss Parns (50 on Unspen Enstanmatts
80
PPD
P50+15
PPO DUR
650
600
PPQ
FPD
RD
Average Rotume on Unsoee Enveeerrt
500
PRS LE PRQ
BO
400
450-
400-
200
350
900
250
20
150
10
100
50-
100
50:
200
40
6,000
15050 20000
25,000 30,000
Tencsaga
89
70
191 (oreer
PPG cmm
600
600
PPO
PROS car
fresngg Rooumes Unpoon Envrrmoee
450
500
9P0
PPO
PRO +
P80 paocke
600
400
8
450
400-
250
0:0
800
250
250
200
150
100
10-
20-
0:
(a) Large-scale CoinRun
(b) DeepMind Lab
(c) Surreal robotics
Figure 5: The performances of trained agents in unseen environments under (a) large-scale CoinRun. (b) DeepMind Lab and (c) Surreal robotics control. The solid/dashed lines and shaded regions represent the mean and standard deviation, respectively.
Visual interpretation. To verify whether the trained agents can focus on meaningful and high- level information, the activation maps are visualized using Grad-CAM (Selvaraju et al., 2017) by averaging activations channel-wise in the last convolutional layer, weighted by their gradients. As shown in Figure 4, both vanilla PPO and our agents make decision by focusing on essential objects, such as obstacles and coins in the seen environment. However, the unseen environment, the vanilla PPO agent displays a widely distributed activation map in some cases, while our agent does not. As a quantitative metric. we measure the entropy of normalized activation maps. Specifically, we first normalize activations Tthh.w E [0, such that it represents 2D discrete probability distribution at
timestep L, i.e., Σ Σ 1==1 Ot.h.w
Then, we measure the entropy averaged over the timesteps as follows: - - ET Σ==1 Σw -==11 Tthh, log f.A.w. Note that the entropy of the activation map quantitatively measures the frequency an agent focuses on salient components in its observation. Results show that our agent produces a low entropy on both seen and unseen environments (i.e., 2.28 and 2.44 for seen and unseen, respectively). whereas the vanilla PPO agent produces a low entropy only in the seen environment (2.77 and 3.54 for seen and unseen, respectively).
W
Results on large-scale experiments. Similar to Cobbe et al. (2019), the generalization ability by training agents is evaluated on fixed set of 500 levels of CoinRun. To explicitly separate seen and unseen environments, half of the available themes are utilized (i.e., style of backgrounds, floors, agents, and moving obstacles) for training, and the performances of unseen themes are measured." " As shown in Figure 5(a), our method outperforms all baseline methods by large margin. In particular, the success rates are improved from 39.8% 58.7% compared to the PPO with cutout (CO) augmentation proposed in Cobbe et al. (2019), showing that our agent learns generalizable representations given a limited number of seen environments.
1,000 different levels consisting
Oracle agents are trained on same map layouts with unseen themes to measure the optimal generalization performances unseen visual patterns.
7

---

Published as a conference paper at ICLR 2020
PPO
PPO + ours
# of Seen
Total Rewards
# of Seen
Total Rewards
# of Seen
Total Rewards
Environments
Environments
Environments
DeepMind Lab Surreal Roboties
55,4 - 332
16
218.3 = 992
358.2 + 81.5
59.2 + 319
25
168.8 + 1558
356.8 + 154
Table Comparison with domain randomization. The results show the mean and standard deviation averaged over three runs and the best results are indicated in bold.
4.3 EXPERIMENTS ON DEEPMIND LAB AND SURREAL ROBOTICS CONTROI.
Results on DeepMind Lab. We also demonstrate the effectiveness of our proposed method on DeepMind Lab (Beattie al., 2016), which is 3D game environment in the first-person point of view with rich visual inputs. The task is designed based on the standard exploration task, where a goal object is placed in one the rooms 3D maze. In this task, agents aim to collect as many goal objects as possible within 90 seconds to maximize their rewards. Once the agent collects the goal object, it receives ten points and relocated to a random place. Similar to the small-scale CoinRun experiment, agents are trained to collect the goal object in fixed map layout and tested in unseen environments with only changing the style of the walls and floors. We report the mean and standard deviation of the average scores across ten different map layouts, which are randomly selected. Additional details are provided in Appendix G.
Note that simple strategy of exploring the map actively and recognizing the goal object achieves high scores because the maze size is small in this experiment. Even though the baseline agents achieve high scores by learning this simple strategy in the seen environment (see Figure 6(c) in Appendix A for learning curves), Figure 5(b) shows that they fail to adapt to the unseen environ- ments. However, the agent trained by our proposed method achieves high scores in both seen and unseen environments. These results show that our method can learn generalizable representations from high-dimensional and complex input observations (i.c., 3D environment).
Results on Surreal robotics control. We evaluate our method in the Block Lifting task using the Surreal distributed RL framework (Fan et al., 2018): the Sawyer robot receives a reward if succeeds to lift a block randomly placed on table. We train agents on a single environment and test on five unseen environments with various styles of tables and blocks (see Appendix for further details). Figure 5(c) shows that our method achieves a significant performance gain compared to all baselines in unseen environments while maintaining performance in the seen environment (see Figure 13 in Appendix 1), implying that our method can maintain essential properties, such as structural spatial features of the input observation.
with domain randomization. To further verify the effectiveness of our method, the vanilla PPO agents are trained by increasing the number of seen environments generated by ran- domizing rendering in simulator, while agent is still trained in a single environment (see Ap- pendices G and for further details). Table 3 shows that the performance of baseline agents can be improved with domain randomization (Tobin et al., 2017). However, our method still outperforms the baseline methods trained with more diverse environments than ours, implying that our method is more effective in learning generalizable representations than simply increasing the (finite) number of seen environments.
Comparison
CONCLUSION
In this paper, we explore generalization in RL where the agent is required to generalize to new envi- ronments in unseen visual patterns, but semantically similar. To improve the generalization ability, we propose to randomize the first layer of CNN to perturb low-level features, e.g., various textures, colors, or shapes. Our method encourages agents to learn invariant and robust representations by producing diverse visual input observations. Such invariant features could be useful for several other related topics, like an adversarial defense in RL (see Appendix B for further discussions), sim-to-real transfer (Tobin et al., 2017; Ren et al., 2019), transfer learning (Parisotto et al., 2016; Rusu et al., 20;b),,,, and online adaptation (Nagabandi et al., 2019). We provide the more detailed discussions on an extension to the dynamics generalization and failure cases our method in Appendix J and K, respectively.
8

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGEMENTS
This work was supported in part by Kwanjeong Educational Foundation Scholarship and Sloan Research Fellowship. We also thank Sungsoo Ahn, Jongwook Choi, Wilka Carvalho, Yijie Guo, Yunseok Jang, Lajanugen Logeswaran, Sejun Park, Sungryull Sohn, Ruben Villegas, and Xinchen Yan for helpful discussions.
REFERENCES
Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching youtube. In NeurIPS, 2018.
Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprini arXiv:1612.03881, 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In ICLR, 2019.
Bibhas Chakraborty and Susan A Murphy, Dynamic treatment regimes. Annual review of statistics and its application, 1:447-464, 2014.
Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman. Quantifying generaliza- tion in reinforcement learning. In ICML, 2019.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation policies from data. In CVPR, 2019.
Yue Deng, Feng Bao, Youyong Kong, Zhiquan Ren, and Qionghai Dai. Deep direct reinforcement learning for financial signal representation and trading. IEEE transactions on neural networks and learning systems, 28(3):653 664, 2016.
Terrance DeVries and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint arXiv: /ww.:/70...52222 2017.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. ICML, 2018.
Linxi Fan, Yuke Zhu, Jiren Zhu, Zihua Liu, Orien Zeng, Anchit Gupta, Joan Creus-Costa, Silvio Savarese, and Li Fei-Fei. Surreal: Open-source reinforcement learning framework and robot manipulation benchmark. In CoRL, 2018.
Jesse Farebrother, Marlos C Machado, and Michael Bowling. Generalization and regularization in dqn. arXiv preprint arXiv:1810.00123, 2018.
Shani Gamrian and Yoav Goldberg. Transfer learning for related reinforcement learning tasks via image-to-image translation. In ICML. 2019.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. The Journal of Machine Learning Research, 17(1):2096-0300 2016.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias im- proves accuracy and robustness. ICLR, 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.

---

Published as a conference paper at ICLR 2020
Ian J Goodffllow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Irina Higgins, Arka Pal. Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot trans- fer in reinforcement learning. In ICML, 2017.
Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Picter Abbeel. Adversarial attacks on neural network policies. arXiv preprint arXiv:1702.02288 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not to learn: Training deep neural networks with biased data. In CVPR, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Yen-Chen Lin, Zhang-Wa Wei Hong, Yuan- Liao, Meng-Li Shih, Ming- Yu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learning agents, In IJCAI, 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, (Nov)):577 2205, 2008
Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi- superviee learning. IEEE transactions on pattern analysis and machine intelligence, 1999:1197911993, 2018
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep online learning via meta-learning: Continual adaptation for model-based rl. In ICLR, 2019.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. NeurIPS, 2018.
Charles Packer, Katelyn Gao, Jemej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization deep reinforcement learning. arXiv preprint urXiv:18100 12282, 2018.
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic:
Deep multitask and
transfer reinforeement learning. In ICLR, 2016.
Xue Bin Peng, Marein Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In ICRA, 2018.
Xinyi Ren, Jianlan Luo, Eugen Solowjow, Juan Aparicio Ojea, Abhishek Gupta, Aviv Tamar, and Pieter Abbeel. Domain randomization for active pose estimation. In ICRA, 2019.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3):211-253 2015.
Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre. Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distil- lation. In ICLR, 2016a.
10

---

Published as a conference paper at ICLR 2020
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04677, 2016b. 2016b.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, In NeurlPS, 2016.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, 2015.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Picter Abbeel. High- dimensional continuous control using generalized advantage estimation. In ICLR, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In ICCV, 2017.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676)3354, 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140- 1144, 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-1955 2014.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In IROS, 2017.
Oriol Vinyals, Timo Ewalds. Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani. Heinrich Küttler, John Agapiou, Julian Schrittwicser, et al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint 1708.0/70000478:0 2017.
Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv:161..05660 2016.
Ronald J Williams and Jing Peng. Function optimization using comectionnstttiiiinrrrreen reinforrcmm learning algorithms. Connection Science, 3(3):241-268, 1991.
Qizhe Xie. Zihang Dai, Eduard Hovy, Minh- Kinh-Thang Luong, and Quoe V Le. Unsupervised data augmentation. arXiv preprint arXiv: 12848, 2019.
Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning, arXiv preprint arXiv:1806.07937 2018a.
Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. study on overfitting in deep reinforcement learning. arXiv preprint arXiv: /0X.vv/804.0689 2018b.
Wenxuan Zhou, Lerrel Pinto, and Abhinav Gupta. Environment probing interaction policies. In ICLR, 2019.

---

Published as a conference paper at ICLR 2020
Appendix: Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning
A
LEARNING CURVES
Sucowes Seees coreeette
S,0000 Paats 8con Emfrrreor morts
Tork Scones Steen Enverteeeens
P50
D0
FPO
FRO
PP0
2.10
6'10
5432
2+10
10410
5-"0""
10h10
15-10'
20H10
25-10"
2+107
4x10
2-10
10
Timosapa
Timesoopa
602
Ducowes Seee Eivenrreee
B,0000 Paaos (S) Boor Eniirrmrte
Than Sconet Been Endtorments
191 P00
PRO
5P0
2+10"
Extoy
8+10
10<10'
10x10" 16x107
20H10" 25-10
6x107 Beld"
ax10'
10.10"
(a) Small-scale CoinRun
(b) Large-scale CoinRun
(c) DeepMind Lab
Figure 6: Learning curves on (a) small-scale, (b) large-scale CoinRun and (c) Deep Lab. The
solid line and shaded regions represent the mean and standard deviation, respectively, across three runs.
Sucosss Ratas (Ng Unssen Ervitonment 18
120
PPQ
PPG +2 RPO DO
RPO (oreci)
Succoss Ratee on Uneeen Environments
PPO DO PPO OR
PPO +V PPO CJ
RPO (prada) PPO + cura
PPO BN
FPO curs
100
100
40-
20
20-
10
20
30
40
50
80
70
60
90
100
10
20
30
40
50
60
70
80
90
100
Timestops (M)
Timestess (M)
(a) Comparison with regularization techniques
(b) Comparison with data augmentation techniques
Figure 7: The performance in unseen environments in small-scale CoinRun. The solid/dashed line
and shaded regions represent the mean and standard deviation, respectively, across three runs.
B ROBUSTNESS AGAINST ADVERSARIAL ATTACKS
The adversarial (visually imperceptible) perturbation (Szegedy et al., 2014) to clean input obser- vations can induce the DNN-based policies to generate incorrect decision at test time (Huang et al., 2017; Lin et al., 2017). This undesirable property of DNNs has raised major security con- cerns. In this section, attacks. Our method is expected to improve the robustness against such adversarial attacks because the agents are trained with randomly perturbed inputs. To verify that the proposed method can improve the robustness to adversarial attacks, the adversarial samples are generated using FGSM (Goodfellow et al., 2015) by perturbing inputs the opposite direction the most probable action initially predicted by the policy:
evaluate if the proposed method can improve the robustness adversarial
Sadv -ssin V V log (a (a 0)),
12

---

Published as a conference paper at ICLR 2020
where is the magnitude of noise and a' arg [aass: 8) is the action from the policy. Table shows that our proposed method can improve the robustness against FGSM attacks with = 0.01,
which implies that hidden representations of trained agents are more robust.
Small- Scale CoinRun
Large-Scale CoinRun
DeepMind Lab
Clean
FGSM
Clean
FGSM
Clean
FGSM
61.5
77.4
163.5
PPO
100
96.2
352.5
(-38.5)
(-19.5)
(-53.6)
88.0
84.4
184.0
PPO +
100
99.6
368.0
(-12.0)
(-15.3)
(-50.0)
Table 4: Robustness against FGSM attacks on training environments. The values in parentheses represent the relative reductions from the clean samples.
C DETAILS FOR TRAINING AGENTS USING PPO
Policy optimization. For all baselines and our methods, PPO is utilized to train the policies. Specif- ically, we use discount factor y = 0.999, generalized advantage estimator (GAE) Schulman al. (2016) parameter À 0.95, and an entropy bonus (Williams & Peng, 1991) of 0.01 to ensure sufficient exploration. We extract 256 timesteps per rollout, and then train the agent for 3 epochs with 8 mini-batches. The Adam optimizer (Kingma & Ba, 2015) is used with the starting learning rate 0.0005. We run 32 environments simultaneously during training. As suggested in Cobbe et al. (2019), two boxes are painted the upper-left corner, where their color represents the x- and y-axis velocity to help the agents quickly learn to act optimally. In this way, the agent does not need to memorize previous states, so a simple CNN- based policy without LSTM can effectively perform in experimental settings.
Data augmentation methods. In this paper, we compare a variant cutout (DeVrics & Taylor, 2017) proposed in Cobbe al. (2019), grayout, inversion, and color jitter (Cubuk al., 2019). Specifically, the cutout augmentation applies a random number of boxes in random size and color to the input, the grayout method averages all three channels of the input, the inversion method inverts pixel values by 50% chance, and the color jitter changes the characteristics of images commonly used for data augmentation in computer vision tasks: brightness, contrast, and saturation. For every timestep in the cutout augmentation, we first randomly choose the number of boxes from zero to five, assign them a random color and size, and place them in the observation. For the color jitter, the parameters for brightness, contrast, and saturation are randomly chosen in [..5, For each episode, the parameters of these methods are randomized and fixed such that the same image pre- processing is applied within episode.
D DIFFERENT TYPES OF RANDOM NETWORKS
In this section, we apply random networks to various locations in the network architecture (see Figure 9) and measure the performance in large-scale CoinRun without the feature matching loss. For all methods, single-layer CNN is used with kernel size of 3, and its output tensor is padded in order to be in the same dimension as the input tensor. As shown in Figure 8, the performance of unseen environments decreases as the random network is placed in higher layers. On the other hand, the random network in residual connections improves the generalization performance, but does not outperform the case when random network is placed at the beginning of the network, meaning that randomizing only the local features of inputs can be effective for a better generalization performance.
Foror additional details, see https: wwwpe/tooooorrreeeeeeeoooeeeeeee dwcs/ /ttss//ssbllee e/ nodul 1 eeleee///rrre sonocciiiiiiooooo trans wwtsssootttstassaiooeeo html #ooloriitter.
13

---

Published as a conference paper at ICLR 2020
Success Rates (%) on Sean Envionments
100
Success Fales (%) onjonn Unseen Environments
70
PPO
PPO
60
40-
20-
PPO
PPO
5x10
10"10"
15-10"
20<10'
100
150
200
250
Timestaps (M)
Timestops (M)
(a) Performance in environments
(b) Performance in environments
Figure 8: The performance of random networks in various locations in the network architecture on (a) seen and (b) unseen environments large-scale CoinRun. We show the mean performances averaged over three different runs, and shaded regions represent the standard deviation.
randombred
randdomlzdd during
rindomiee aed
Action a
Action a
Action
50860000000 www.0000000000000o
Dish-Dimmnsionall Input
High- Dimensional Input
Dige--immnsionall Input
Block
(b) Block 2
(c) Block
randomined towning
randomtted durlng training Action
Action
NO400000 0000000000000000000
Digh-Dimenssonal Input
High Dimensional Input 8
(d) Residual
(c) First
Figure 9: Network architectures with random networks in various locations. Only convolutional layers and the last fully connected layer are displayed for conciseness.
E ENVIRONMENTS IN SMALL-SCALE COINRUN
For small-scale CoinRun environments, consider fixed map layout with two moving obstacles and measure the performance the trained agents by changing the style of the backgrounds (see Figure 10). Below is the list of seen and unseen backgrounds in this experiment:
Seen background:
14

---

Published as a conference paper at ICLR 2020
Seen environment
(b) Unseen environment
(c) Unseen environment
Figure 10: Examples of seen and unseen environments in small-scale CoinRun.
kemns//Backgronnd//blie cicllue_deerrtppe
Unseen backgrounds:
sents//aackkoonntt//ooooorsssggs
(4) pn)
bettgrooutt//gggneeaaccgggoooeee (5) .nng
backgppuutt//onneecaaccgggog/aep (2) .png
backgpouttt/yannnaaaaarrruusssar (3) .nng
F ENVIRONMENTS IN LARGE-SCALE COINRUN
In CoinRun, there are themes for backgrounds, for grounds, agents, and for obstacles. For the large-scale CoinRun experiment, we train agents a fixed set of 500 levels of CoinRun using half of the available themes and measure the performances on 1,000 different levels consist- ing of unseen themes. Specifically, the following is list of seen and unseen themes used in this experiment:
Seen backgrounds:
kenner/Beckgroond//bbeee..edee png
kenee//Bekkrrond//buuuuggzssss png
Benner/Baokkronnt//leeeeaaaaaa.p
15

---

Published as a conference paper at ICLR 2020
(a) Seen environment
(b) Seen environment
Unseen environment
Unseen environment
Figure 11: Examples of seen and unseen environments in large-scale CoinRun.
kenter//ackkronng//oooooodssssse
kente//Baacgpoondd///oooesesssgp
background.ppg.
aecepponni/onnnnaaaagtppossssssp (4)
bacegroooutk.ggemmmaaaagggeg/eee ( .nng
(2) .ngg
battrromms/ggeeeaaaarceemnaagggn (3) ppng
Unseen backgrounds:
16

---

Published as a conference paper at ICLR 2020
Seen grounds:
Dirt
Grass
Planet
Unseen grounds:
Sand
Snow
Stone
Seen player themes:
Beige
Blue
Unseen player themes:
Green
Pink
Yel ow
Figure 12: The top-down view of the trained map layouts.
G ENVIRONMENTS ON DEEPMIND LAB
Dataset. Among the styles (textures and colors) provided for the 3D maze in the DeepMind Lab, we take different styles floors and walls, respectively (see the list below). We construct training dataset by randomly choosing a map layout and assigning a theme among ten floors and walls, respectively. The domain randomization method compared Table 3 uses four floors and four wall themes (16 combinations in total). Trained themes are randomly chosen before training and their combinations are considered to be seen environments. To evaluate the generalization ability, we measure the performance of trained agents on unseen environments by changing the styles of walls and floors. Domain randomization has more seen themes than the other methods, so all methods are compared with six floors and six walls (36 combinations in total), which unseen for all methods. The mean and standard deviation of the average scores across ten different map layouts are reported in Figure 12.
17

---

Published as a conference paper at ICLR 2020
Floor themes:
ggsstyle_01_flloororraagg
1g.styyle01..floorbbblee
4g_styye_00_ffloorbbuu
1g.styll__0__floot_greee
1g_styyl__03_floorggreee
1e_style__0__floorbbbee
1g_styye__0__loooobbbee
1g_ettyle_0__floororraagg
1g_sstya__0__lloorbbbbl
1g_etyyle_0__floororrragg
Wall themes:
kg_style_01_wall_gree
4g_styya_0i_wall_zee
1g_style__0__aal_yelloo
1g_style_00__aal_bluee
1a.styyle_03_aallorragg
"g_styyle_03_wal__rra
1gletyyl__0__vallggree
Ig_styyle_04_wall_reel
1g_style_00__wll__ree
1g_styyle_0_wwal_yaall.
Action space. Similar to IMPALA (Espeholt al., 2018), the agent can take eight actions from the DeepMind Lab native action samples: {Forward, Backward, Move Left, Move Right, Look Left, Look Right, Forward Look Left, and Forward + Look Right). Table 5 describes the detailed mapping.
Action
DeepMind Lab Native Action
Forward
D, 0, 0,
1
0, 0, 0]
Backward
0, 0, 0,
0, 0]
Move Left
0, 0, 1,
0, 0, 0, 0]
Move Right
0, 0,
1,
0, 0, 0, 0]
Look Left
[-20, 0,
0,
0,
0, 0, 0]
Look Right
20, 0,
0,
0, 0, 0, 0]
Forward + Look Left
[-20,
0,
0,
1, 0, 0, 0]
Forward + Look Right
20, 0,
0,
1, 0, 0, 0]
Table 5: Action set used in the DeepMind Lab experiment, The DeepMind Lab native action set consists of seven discrete actions encoded in integers ([L.,U indicates the lower/upper bound of the possible values): 1) yaw (left/right) rotation by pixel [-512,512], 2) pitch (up/down) rotation by pixel [-512,512], 3) horizontal move [-1,1], 4) vertical move [-1,1], 5) fire [0,1], 6) jump [0.1], and 7) crouch [0.1].
H EXPERIMENTS ON DOGS AND CATS DATABASE
Dataset. The original database is a set of 25,000 images of dogs and cats for training and 12,500 images for testing. Similar to Kim et al. (2019), the data is manually categorized according the color of the animal: bright or dark. Biased datasets are constructed such that the training set consists of bright dogs and dark cats, while the test and validation sets contain dark dogs and bright cats. Specifically, training, validation, and test sets consist of 10,047, 1,000, and 5,738 images, respec- tively." ResNet- 18 (He et al., 2016) trained with an initial learning rate chosen from {0.05, 0.1)
'Teee code is available at: https: / werithu..comm fei woffoeeeeeeee leaeeeeeearinnene Maarring-ee not not to- notetto-eee////eeee basterr/aataeet/dggge
18

---

Published as a conference paper at ICLR 2020
and then dropped by 0.1 at 50 epochs with total of 100 epochs. We the Nesterov momentum of 0.9 for SGD, à mini-batch size chosen from {32, 64), and the weight decay set to 0.0001. We report the training and test set accuracies with the hyperparameters chosen by validation. Unlike Kim et al. (2019), we do not use ResNet-18 18 pre-trained with ImageNet (Russakovsky et al., 2015) in order to avoid inductive bias from the pre-trained dataset.
I EXPERIMENTAL RESULTS ON SURREAL ROBOT MANIPULATION
Our method is evaluated in the Block Lifting task using the Surreal distributed RL framework (Fan et al., 2018). In this task, the Sawyer robot receives a reward it successfully lifts a block randomly placed table. Following the experimental setups in (Fan al., 2018), the hybrid CNN-LSTM architecture (see Figure 13(a)) is chosen as the policy network and a distributed version of PPO (i.e., actors collect massive amount of trajectories and the centralized learner updates the model parameters using PPO) is used to train the agents.* 12 Agents take 84 x 84 observation frames with proprioceptive features (e.g., robot joint positions and velocities) and output the mean and log of the standard deviation for each action dimension. The actions are then sampled from the Gaussian distribution parameterized by the output. Agents are trained on a single environment and tested on five unseen environments with various styles of table, floor, and block, as shown in Figure 14. For the Surreal robot manipulation experiment, the vanilla PPO agent trained on 25 environments generated by changing the styles of tables and boxes. Specifically, we use (blue, gray, orange, white, purple} and fred, blue, green, yellow, cyan} for table and box, respectively.
Value
Policy
450
450
400
400
LSTM
150
250
300
500
CNN
250
260
200
200
Random Network
180
100
1002
FFD
FRO DO
FPO CD
PPO CI
FPO mmX
POO
Proprioceptive Features
6,000
10,000
16,000 20,000
25,000 30,000
5,000
10,000
25,000 30,000 Timestaps
15,000
20,000
Timestape
(a) Network architecture
(b) Regularization
(c) Data augmentation
Figure 13: (a) An illustration of network architectures for the Surreal robotics control experiment, and learning curves with (b) regularization and (c) data augmentation techniques. The solid line and shaded regions represent the mean and standard deviation, respectively, across three runs.
J EXTENSION TO DOMAINS WITH DIFFERENT DYNAMICS
In this section, we consider an extension to the generalization on domains with different dynamics. Similar to dynamics randomization (Peng et al., 2018), one can expect that our idea can be useful for improving the dynamics generalization. To verify this, we conduct an experiment on CartPole and Hopper environments where an agent takes proprioceptive features (e.g.. positions and velocities). The goal of CartPole is to prevent the pole from falling over, while that of Hopper is to make one- legged robot hop forward as fast as possible, respectively. Similar to the randomization method we applied to visual inputs, we introduce random layer between the input and the model. As natural extension of the proposed method, we consider performing the convolution operation by multiplying d d diagonal matrix d-dimensional input states. For every training iteration, the elements of the matrix are sampled from the standard uniform distribution U (O.88, 1.2). One can note that this method can randomize the amplitude of input states while maintaining the intrinsic information (c.g.. sign of inputs). Following Packer et al. (2018); Zhou et al. (2019), we measure the performance of the trained agents on unseen environments with a different set of dynamies parameters, such as mass, length, and force. Specifically, for CartPole experiments, similar to Packer et al. (2018), the policy
We used reference implementation with two actors in: https: / //g/ thub. com Surr/uurraal 1AT/ surreal.
19

---

Published as a conference paper at ICLR 2020
(a) Seen environment
(b) Unseen environment
(c) Unseen environment
(d) Unseen environment
(e) Unseen environment
(l) Unseen environment
Figure Examples of seen and unseen environments the Surreal robot manipulation.
2500
400
TRFO
Neaig Rogar
2000
Aweeogs Donant
1000
1000
12
500
1910
11:0
400
600
252
500
Timestopo
Tamnssrape (1)
Timoste
Timestopi 90)
(a) CartPole (seen)
(b) CartPole (unseen)
(c) Hopper (seen)
(d) Hopper (unseen)
Figure 15: Performances of trained agents in seen and unseen environments under (a/b) CartPole and (c/d) Hopper. The solid/dashed lines and shaded regions represent the mean and standard deviation, respectively.
and value functions are multi-layer perceptrons (MLPs) with two hidden layers of 64 units each and hyperbolic tangent activation and the Proximal Policy Optimization (PPO) (Schulman et al., 2017) method is used to train the agents. The parameters of the training environment are fixed at the default values in the implementations from Gym, while force, length, and mass of environments are sampled from [1, 5] U [15. 20), [0.05, 0.25] U [0.75, 1.0]. [0.01, 0.05] U [0.5, 1.0] that the policy has never seen in any stage of training. For Hopper experiments, similar Zhou al. (2019), the policy is a MI.P with two hidden layers of 32 units each and ReI.U activation and value function is a linear model. The trust region policy optimization (TRPO) (Schulman al., 2015) method used to train the agents. The mass of the training environment is sampled from {1.0, 2.0, 3.0, 4.0, 5.0). while is sampled from (6.0, 7.0, 8.0) during testing." Figure 15 reports the mean and standard deviation across runs. Our simple randomization improves the performance of the agents in unseen environments, while achieving performance comparable to seen environments. We believe that this evidences wide applicability of our idea beyond visual changes.
We used reference implementation in https::/bal berker bharkeleyy eettteeley.euuubbb ht/e
r1 general zation.
"We used reference implementation with two actors in: https: / t/tbt/b..///tt///////g/e
thub. ctt. Won//Wnxuan---- /hou/
EPI.
20

---

Published as a conference paper at ICLR 2020
SucoNs toree (h6
Sucoes Rapes
60
10
270
1000
2000
1000
2000
08 03 05 07
CoinRun with good (h) Performance in (c) Performance in (d) Performance in and bad coins
environments
seen environments
seen environments
Figure 16: (a) Modified CoinRun with good and bad coins. The performances on (b) seen and (c) unseen environments. The solid line and shaded regions represent the mean and standard deviation, respectively, across three runs. (d) Average success rates on large-scale CoinRun for varying the fraction of clean samples during training. Noe that corresponds to vanilla PPO agents.
K FAILURE CASE OF OUR METHODS
In this section, we verify whether the proposed method can handle color (or texture)-condittioee RL tasks. One might expect that such RL tasks can be difficult for our methods to work because of the randomization. For example, our methods would fail if consider an extreme seek-avoid object gathering setup, where the agent must learn to collect good objects and avoid bad objects which have the same shape but different color. However, we remark that our method would not always fail for such tasks if other environmental factors (e.g., the shape of objects in Collect Good Objects in DeepMind Lab (Beattie et al., 2016)) are available to distinguish them. verify this, we consider modified CoinRun environment where the agent must learn to collect good objects (c.g., gold coin) and avoid bad objects (c.g., silver coin). Similar to the small-scale CoinRun experiment, agents trained to collect the goal object in fixed map layout (see Figure 16(a)) and tested in unseen environments with only changing the style of the background. Figure 16(b) shows that our method can work well for such color-conditioned RL tasks because trained agent can capture the other factors such as a location to perform this task. Besides, our method achieves a significant performance gain compared vanilla PPO agent in unseen environments as shown Figure 16(c).
As another example, in color-matching tasks such as the keys doors puzzle in DeepMind Lab (Beat- tie al., 2016). the agent must collect colored keys open matching doors. Even though this task is color-conditioned, a policy trained with our method can perform well because the same colored objects will have the same color value even after randomization, i.e., our randomization method still maintains the structure of input observation. This evidences the wide applicability of our idea. We also remark that our method can handle more extreme corner cases by adjusting the fraction clean samples during training. In summary, we believe that the proposed method covers broad scope of generalization across low-level transformations in the observation space features.
L ABLATION STUDY FOR FRACTION OF CLEAN SAMPLES
We investigate the effect of the fraction of clean samples. Figure 16(d) shows that the best unseen performance is achieved when the fraction of clean samples is 0,1 on large-scale CoinRum.
M TRAINING ALGORITHM
Algorithm PPO random networks, Actor-Critic Style
for iteration= 1, 1,2, ,, do
Sample the parameter & of random networks from prior distribution P(ő)
for actor= 1, ... N do
Run policy (al (s; Φ) 0) in the given environment for T timesteps Compute advantage estimates
end for Optimize Crandon in equation (3) with respect to 0 end for
21

---

Published as a conference paper at ICLR 2020
N
VISUALIZATION OF HIDDEN FEATURES
Soon Unseen A Unssen
Seen Unseon
Seen
Unseen
Unsoon
(a) PPO
(b) PPO + L.
(c) PPO + BN
Seen
Seen Unseen
Seen Unseen
Unseen Unseen
(d) PPO DO
(c) PPO + CO
(f) PPO + CJ
Soan
Seen Unseen
Seen
Unszon
Unseen A
Unseen
(g) +PO IV
(h) PPO + GR
(i) PPO ours
Figure 17: Visualization of the hidden representation of trained agents optimized by (a) PPO, (b) PPO L2, (c) PPO BN, (d) PPO + DO, (e) PPO CO, (f) PPO GR, (g) PPO IV, (h) PPO CJ, and (I) PPO + ours using I-SNE. The point colors indicate the environments of the corresponding observations.
22


Reviews:
[{"review_id": "Hyl49oREcH", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes using a randomly parameterized convolutional layer as additional processing of the input observation to provide data augmentation to make policies more robust to environments with different observation spaces. The empirical results are thorough, comparing with other regularization techniques, including dropout, L2 regularization, and batch normalization with the same policy gradient method, PPO on a variety of generalization in RL benchmarks. There are additional experiments of this method to check that it actually removes visual bias in a computer vision problem better than other methods. \n\nThey also incorporate a feature matching loss that explicitly forces the learned representations of equivalent states to be close in L2 distance. While the empirical results are impressive, it is hard to feel excited about this work, which relies on the inductive bias of a randomly parameterized convolution layer to modify the texture of the observation and show it works in certain settings. I'd like more discussion and showcasing of failure modes, it seems that this wouldn't work for settings where the train and test environments are different in ways beyond texture and changes in small objects, and additional analysis in terms of the dogs and cats database about why it performs so much better than other methods. What exactly is the desired and meaningful information in images that a random convolution layer can keep while removing something that is able to generalize to different shades of cats and dogs? Why would this perform better than grayscaling?  What about grayscaling and additive Gaussian noise?\n\nThe comparison of PPO is also unfair in that the author's method uses an ensemble of policies to act, which other methods do not. A more fair comparison would use ensembles in all other baselines as well or results showing how their method performs without this ensemble.\n\nOverall, the presentation, analysis, and writing can all be improved to match the strong empirical results produced. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Hke62XQY9H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes applying random convolutions to the observation space to improve the ability of deep RL agents to generalize to unseen environments. To encourage the learning of invariant features, the authors further include a loss term to align features of perturbed and unperturbed observations. Thorough experiments on multiple generalization benchmarks show that this method outperforms many previously used regularization and data augmentation techniques.\n\nAlthough the proposed method is simple, it represents a useful contribution. The need to generalize across low level transformations in the observation space features prominently in several environments, including DeepMind Lab and CoinRun. The clear need for agents to be invariant to these low level transformations well motivates the proposed approach, as does the failure of many existing methods to provide this invariance.\n\nThe authors could more explicitly discuss the main drawbacks of this approach. As with any data augmentation, there is an assumption that the applied transformation generally won’t destroy information pertinent to the task. While this is true for the MDPs investigated here, it is easy to imagine slight variants of these MDPs for which this approach would fail. If an optimal policy must condition on color or texture information from observations, then using these random convolutions would render training impossible. Encountering such MDPs is not farfetched, so this weakness seems worth acknowledging.\n\nIn Figure 5 it would be useful to visualize performance of an agent trained directly on these unseen environments, as this presumably serves as an upper bound for the zero-shot performance of “PPO + ours”. How close does “PPO + ours” come to closing this gap? Without any context on the reward scale, it’s hard to infer how well this method is generalizing, beyond seeing that it beats some (possibly weak) baselines. Admittedly some closely related curves can be found in Appendix Figures 9 and 14, though they’re a bit out of the way.\n\nSection 3.1 mentions that using alpha = 0 complicates training. It is somewhat surprising that using alpha > 0 is necessary or significant and yet the value used (alpha = .1) is relatively small. Any further comments on this choice?\n\nI appreciate the discussion in Appendix F. It’s natural to wonder about alternative injection sites for the random network, and it’s good to see how the proposed method compares to these alternatives.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BJlOAwOpYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes methods to improve generalization in deep reinforcement learning with an emphasis on unseen environments. The main contribution is essentially a data augmentation technique that perturbs the input observations using a noise generated from the range space of a random convolutional network. The empirical results look impressive and demonstrate the effectiveness of the method. The experiments are thorough (includes even adversarial attack) and the core method is novel as far as I am aware.\n\nThat said, I have a couple of concerns regarding this paper and I would be willing to change my score if authors can address these.\n\n1) Feature matching loss (Eq 2) is presented as a novel contribution without referring to related work in semisupervised learning literature. This is essentially consistency training. See:\na) Miyato, Takeru, et al. \"Virtual adversarial training: a regularization method for supervised and semi-supervised learning.\" IEEE transactions on pattern analysis and machine intelligence 41.8 (2018): 1979-1993.\nb) Xie, Qizhe, et al. \"Unsupervised data augmentation.\" arXiv preprint arXiv:1904.12848 (2019).\n\n2) The main contribution appears to be a data augmentation technique where we add a random neural net based perturbation to the state. My question is:\n\n*Why don't you first evaluate this on computer vision tasks given that the core idea is data augmentation for images?*\n\nIf this technique is so powerful, shouldn't this do a great job in CIFAR10, Imagenet etc? Instead authors only provide a niche example (bright vs dark cat/dogs).\n\nIf this can compete with top augmentation techniques on Imagenet (e.g. autoagument), then it can explain the RL performance. Otherwise, please provide some intuition on why this works so well on RL but not as well on computer vision tasks. Is it the unseen environment diversity of RL challenges?\n\n3) While proposed method performs well on the benchmarks, it is not clear whether authors compare to the state-of-the-art algorithms. For each task (CoinRun, DeepMind Lab, etc), please explicitly state the best prior result (e.g. Espeholt et al, Tobin et al, Cobbe et al etc) so that proposed method's performance can be better assessed.\n\n-------------------------\n\nAfter rebuttal: Authors addressed most of my comments. I also found the new experimental results (Fig 5 and 7) very insightful. I increase my score to Weak Accept.\n\nFor future improvement: More realistic experiments on computer vision tasks (besides cats and dogs) would be welcome. Otherwise, please justify why proposed strategy is particularly good for RL (rather than traditional computer vision benchmarks) in boosting robustness to new domains.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission proposes an RL method for learning policies that generalize better in novel visual environments. The authors propose to introduce some noise in the feature space rather than in the input space as is typically done for visual inputs. They also propose an alignment loss term to enforce invariance to the random perturbation.

Reviewers agreed that the experimental results were extensive and that the proposed method is novel and works well.

One reviewer felt that the experiments didn’t sufficiently demonstrate invariance to additional potential domain shifts. AC believes that additional experiments to probe this would indeed be interesting but that the demonstrated improvements when compared to existing image perturbation methods and existing regularization methods is sufficient experimental justification of the usefulness of the approach.

Two reviewers felt that the method should be more extensively compared to “data augmentation” methods for computer vision tasks. AC believes that the proposed method is not only a data augmentation method given that the added loss tries to enforce representation invariance to perturbations as well. As such comparisons to feature adaptation techniques to tackle domain shift would be appropriate but it is reasonable to consider this line of comparison beyond the scope of this particular work.

Ac agrees with the majority opinion that the submission should be accepted.
Decision: Accept (Poster)