Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning

1 INTRODUCTION . Deep reinforcement learning ( RL ) has been applied to various applications , including board games ( e.g. , Go ( Silver et al. , 2017 ) and Chess ( Silver et al. , 2018 ) ) , video games ( e.g. , Atari games ( Mnih et al. , 2015 ) and StarCraft ( Vinyals et al. , 2017 ) ) , and complex robotics control tasks ( Tobin et al. , 2017 ; Ren et al. , 2019 ) . However , it has been evidenced in recent years that deep RL agents often struggle to generalize to new environments , even when semantically similar to trained agents ( Farebrother et al. , 2018 ; Zhang et al. , 2018b ; Gamrian & Goldberg , 2019 ; Cobbe et al. , 2019 ) . For example , RL agents that learned a near-optimal policy for training levels in a video game fail to perform accurately in unseen levels ( Cobbe et al. , 2019 ) , while a human can seamlessly generalize across similar tasks . Namely , RL agents often overfit to training environments , thus the lack of generalization ability makes them unreliable in several applications , such as health care ( Chakraborty & Murphy , 2014 ) and finance ( Deng et al. , 2016 ) . The generalization of RL agents can be characterized by visual changes ( Cobbe et al. , 2019 ; Gamrian & Goldberg , 2019 ) , different dynamics ( Packer et al. , 2018 ) , and various structures ( Beattie et al. , 2016 ; Wang et al. , 2016 ) . In this paper , we focus on the generalization across tasks where the trained agents take various unseen visual patterns at the test time , e.g. , different styles of backgrounds , floors , and other objects ( see Figure 1 ) . We also found that RL agents completely fail due to small visual changes1 because it is challenging to learn generalizable representations from high-dimensional input observations , such as images . To improve generalization , several strategies , such as regularization ( Farebrother et al. , 2018 ; Zhang et al. , 2018b ; Cobbe et al. , 2019 ) and data augmentation ( Tobin et al. , 2017 ; Ren et al. , 2019 ) , have been proposed in the literature ( see Section 2 for further details ) . In particular , Tobin et al . ( 2017 ) showed that training RL agents in various environments generated by randomizing rendering in a simulator improves the generalization performance , leading to a better performance in real environments . This implies that RL agents can learn invariant and robust representations if diverse ∗Equal contribution . †Work done while at University of Michigan . 1Demonstrations on the CoinRun and DeepMind Lab exploration tasks are available at [ link-video ] . input observations are provided during training . However , their method is limited by requiring a physics simulator , which may not always be available . This motivates our approach of developing a simple and plausible method applicable to training deep RL agents . The main contribution of this paper is to develop a simple randomization technique for improving the generalization ability across tasks with various unseen visual patterns . Our main idea is to utilize random ( convolutional ) networks to generate randomized inputs ( see Figure 1 ( a ) ) , and train RL agents ( or their policy ) by feeding them into the networks . Specifically , by re-initializing the parameters of random networks at every iteration , the agents are encouraged to be trained under a broad range of perturbed low-level features , e.g. , various textures , colors , or shapes . We discover that the proposed idea guides RL agents to learn generalizable features that are more invariant in unseen environments ( see Figure 3 ) than conventional regularization ( Srivastava et al. , 2014 ; Ioffe & Szegedy , 2015 ) and data augmentation ( Cobbe et al. , 2019 ; Cubuk et al. , 2019 ) techniques . Here , we also provide an inference technique based on the Monte Carlo approximation , which stabilizes the performance by reducing the variance incurred from our randomization method at test time . We demonstrate the effectiveness of the proposed method on the 2D CoinRun ( Cobbe et al. , 2019 ) game , the 3D DeepMind Lab exploration task ( Beattie et al. , 2016 ) , and the 3D robotics control task ( Fan et al. , 2018 ) . For evaluation , the performance of the trained agents is measured in unseen environments with various visual and geometrical patterns ( e.g. , different styles of backgrounds , objects , and floors ) , guaranteeing that the trained agents encounter unseen inputs at test time . Note that learning invariant and robust representations against such changes is essential to generalize to unseen environments . In our experiments , the proposed method significantly reduces the generalization gap in unseen environments unlike conventional regularization and data augmentation techniques . For example , compared to the agents learned with the cutout ( DeVries & Taylor , 2017 ) data augmentation methods proposed by Cobbe et al . ( 2019 ) , our method improves the success rates from 39.8 % to 58.7 % under 2D CoinRun , the total score from 55.4 to 358.2 for 3D DeepMind Lab , and the total score from 31.3 to 356.8 for the Surreal robotics control task . Our results can be influential to study other generalization domains , such as tasks with different dynamics ( Packer et al. , 2018 ) , as well as solving real-world problems , such as sim-to-real transfer ( Tobin et al. , 2017 ) . 2 RELATED WORK . Generalization in deep RL . Recently , the generalization performance of RL agents has been investigated by splitting training and test environments using random seeds ( Zhang et al. , 2018a ) and distinct sets of levels in video games ( Machado et al. , 2018 ; Cobbe et al. , 2019 ) . Regularization is one of the major directions to improve the generalization ability of deep RL algorithms . Farebrother et al . ( 2018 ) and Cobbe et al . ( 2019 ) showed that regularization methods can improve the generalization performance of RL agents using various game modes of Atari ( Machado et al. , 2018 ) and procedurally generated arcade environments called CoinRun , respectively . On the other hand , data augmentation techniques have also been shown to improve generalization . Tobin et al . ( 2017 ) proposed a domain randomization method to generate simulated inputs by randomizing rendering in the simulator . Motivated by this , Cobbe et al . ( 2019 ) proposed a data augmentation method by modifying the cutout method ( DeVries & Taylor , 2017 ) . Our method can be combined with the prior methods to further improve the generalization performance . Random networks for deep RL . Random networks have been utilized in several approaches for different purposes in deep RL . Burda et al . ( 2019 ) utilized a randomly initialized neural network to define an intrinsic reward for visiting unexplored states in challenging exploration problems . By learning to predict the reward from the random network , the agent can recognize unexplored states . Osband et al . ( 2018 ) studied a method to improve ensemble-based approaches by adding a randomized network to each ensemble member to improve the uncertainty estimation and efficient exploration in deep RL . Our method is different because we introduce a random network to improve the generalization ability of RL agents . Transfer learning . Generalization is also closely related to transfer learning ( Parisotto et al. , 2016 ; Rusu et al. , 2016a ; b ) , which is used to improve the performance on a target task by transferring the knowledge from a source task . However , unlike supervised learning , it has been observed that finetuning a model pre-trained on the source task for adapting to the target task is not beneficial in deep RL . Therefore , Gamrian & Goldberg ( 2019 ) proposed a domain transfer method using generative adversarial networks ( Goodfellow et al. , 2014 ) and Farebrother et al . ( 2018 ) utilized regularization techniques to improve the performance of fine-tuning methods . Higgins et al . ( 2017 ) proposed a multi-stage RL , which learns to extract disentangled representations from the input observation and then trains the agents on the representations . Alternatively , we focus on the zero-shot performance of each agent at test time without further fine-tuning of the agent ’ s parameters . 3 NETWORK RANDOMIZATION TECHNIQUE FOR GENERALIZATION . We consider a standard reinforcement learning ( RL ) framework where an agent interacts with an environment in discrete time . Formally , at each timestep t , the agent receives a state st from the environment2 and chooses an action at based on its policy π . The environment returns a reward rt and the agent transitions to the next state st+1 . The return Rt = ∑∞ k=0 γ krt+k is the total accumulated rewards from timestep t with a discount factor γ ∈ [ 0 , 1 ) . RL then maximizes the expected return from each state st . 3.1 TRAINING AGENTS USING RANDOMIZED INPUT OBSERVATIONS . We introduce a random network f with its parameters φ initialized with a prior distribution , e.g. , Xavier normal distribution ( Glorot & Bengio , 2010 ) . Instead of the original input s , we train an agent using a randomized input ŝ = f ( s ; φ ) . For example , in the case of policy-based methods,3 the parameters θ of the policy network π are optimized by minimizing the following policy gradient objective function : Lrandompolicy = E ( st , at , Rt ) ∈D [ − log π ( at|f ( st ; φ ) ; θ ) Rt ] , ( 1 ) where D = { ( st , at , Rt ) } is a set of past transitions with cumulative rewards . By re-initializing the parameters φ of the random network per iteration , the agents are trained using varied and randomized input observations ( see Figure 1 ( a ) ) . Namely , environments are generated with various visual patterns , but with the same semantics by randomizing the networks . Our agents are expected to adapt to new environments by learning invariant representation ( see Figure 3 for supporting experiments ) . To learn more invariant features , the following feature matching ( FM ) loss between hidden features from clean and randomized observations is also considered : LrandomFM = Est∈D [ ||h ( f ( st ; φ ) ; θ ) − h ( st ; θ ) ||2 ] , ( 2 ) where h ( · ) denotes the output of the penultimate layer of policy π . The hidden features from clean and randomized inputs are combined to learn more invariant features against the changes in the input observations.4 Namely , the total loss is : Lrandom = Lrandompolicy + βLrandomFM , ( 3 ) where β > 0 is a hyper-parameter . The full procedure is summarized in Algorithm 1 in Appendix M. 2Throughout this paper , we focus on high-dimensional state spaces , e.g. , images . 3Our method is applicable to the value-based methods as well . 4FM loss has also been investigated for various purposes : semi-supervised learning ( Miyato et al. , 2018 ) and unsupervised learning ( Salimans et al. , 2016 ; Xie et al. , 2019 ) . Details of the random networks . We propose to utilize a single-layer convolutional neural network ( CNN ) as a random network , where its output has the same dimension with the input ( see Appendix D for additional experimental results on the various types of random networks ) . To reinitialize the parameters of the random network , we utilize the following mixture of distributions : P ( φ ) = αI ( φ = I ) + ( 1 − α ) N ( 0 ; √ 2 nin+nout ) , where I is an identity kernel , α ∈ [ 0 , 1 ] is a positive constant , N denotes the normal distribution , and nin , nout are the number of input and output channels , respectively . Here , clean inputs are used with the probability α because training only randomized inputs can complicate training . The Xavier normal distribution ( Glorot & Bengio , 2010 ) is used for randomization because it maintains the variance of the input s and the randomized input ŝ . We empirically observe that this distribution stabilizes training . Removing visual bias . To confirm the desired effects of our method , we conduct an image classification experiment on the dogs and cats database from Kaggle.5 Following the same setup as Kim et al . ( 2019 ) , we construct datasets with an undesirable bias as follows : the training set consists of bright dogs and dark cats while the test set consists of dark dogs and bright cats ( see Appendix H for further details ) . A classifier is expected to make a decision based on the undesirable bias , ( e.g. , brightness and color ) since CNNs are biased towards texture or color , rather than shape ( Geirhos et al. , 2019 ) . Table 1 shows that ResNet-18 ( He et al. , 2016 ) does not generalize effectively due to overfitting to an undesirable bias in the training data . To address this issue , several image processing methods ( Cubuk et al. , 2019 ) , such as grayout ( GR ) , cutout ( CO ; DeVries & Taylor 2017 ) , inversion ( IV ) , and color jitter ( CJ ) , can be applied ( see Appendix C for further details ) . However , they are not effective in improving the generalization ability , compared to our method . This confirms that our approach makes DNNs capture more desired and meaningful information such as the shape by changing the visual appearance of attributes and entities in images while effectively keeping the semantic information . Prior sophisticated methods ( Ganin et al. , 2016 ; Kim et al. , 2019 ) require additional information to eliminate such an undesired bias , while our method does not.6 Although we mainly focus on RL applications , our idea can also be explorable in this direction .

Summary: This work proposes using a randomly parameterized convolutional layer as additional processing of the input observation to provide data augmentation to make policies more robust to environments with different observation spaces. The empirical results are thorough, comparing with other regularization techniques, including dropout, L2 regularization, and batch normalization with the same policy gradient method, PPO on a variety of generalization in RL benchmarks. There are additional experiments of this method to check that it actually removes visual bias in a computer vision problem better than other methods.