Dropout's Dream Land: Generalization from Learned Simulators to Reality

1 INTRODUCTION . Reinforcement learning ( Sutton & Barto , 2018 ) ( RL ) has experienced a flurry of success in recent years , from learning to play Atari ( Mnih et al. , 2015 ) to achieving grandmaster level performance in StarCraft II ( Vinyals et al. , 2019 ) . However , in all these examples , the target environment is a simulator that can be directly trained in . Reinforcement learning is often not a practical solution without a simulator of the environment . Sometimes the target environment is expensive , dangerous , or even impossible to interact with . In these cases , the agent is trained in a simulated source environment . Approaches that train an agent in a simulated environment with the hopes of generalization to the target environment experience a common problem referred to as the reality gap ( Jakobi et al. , 1995 ) . One approach to bridge the reality gap is domain randomization ( Tobin et al. , 2017 ) . The basic idea is that an agent which can perform well in an ensemble of simulations will also generalize to the real environment ( Antonova et al. , 2017 ; Tobin et al. , 2017 ; Mordatch et al. , 2015 ; Sadeghi & Levine , 2016 ) . The ensemble of simulations is generally created by randomizing as much of the simulator as possible without fundamentally changing the task at hand . Unfortunately , this approach is only applicable when a simulator is provided and the simulator is configurable . A recently growing field , World Models ( Ha & Schmidhuber , 2018 ) , focuses on the side of this problem when the simulation does not exist . World Models offer a general framework for optimizing controllers directly in learned simulated environments . The learned dynamics model can be viewed as the agent ’ s dream environment . This is an interesting area because it removes the need for an agent to operate in the target environment . Some related approaches ( Łukasz Kaiser et al. , 2020 ; Hafner et al. , 2019 ; 2020 ; Sekar et al. , 2020 ; Sutton , 1990 ; Kurutach et al. , 2018 ) focus on an adjacent problem which allows the controller to continually interact with the target environment . Despite the recent improvements ( Łukasz Kaiser et al. , 2020 ; Hafner et al. , 2019 ; Sekar et al. , 2020 ; Kim et al. , 2020 ; Hafner et al. , 2020 ) of World Models , none of them address the issue that World Models are susceptible to the reality gap . The learned dream environment can be viewed as the source domain and the true environment as the target domain . Whenever there are discrepancies between the source and target domains the reality gap can cause problems . Even though World Models suffer from the reality gap , none of the domain randomization approaches are directly applicable because the dream environment does not have easily configurable parameters . In this work we present Dropout ’ s Dream Land ( DDL ) , a simple approach to bridge the reality gap from learned dream environments to reality . Dropout ’ s Dream Land was inspired by the first principles of domain randomization , namely , train a controller on a large set of different simulators which all adhere to the fundamental task of the target environment . We are able to generate a nearly infinite number of different simulators via the insight that dropout ( Srivastava et al. , 2014 ) can be understood as learning an ensemble of neural networks ( Baldi & Sadowski , 2013 ) . Our empirical results demonstrate the advantage of Dropout ’ s Dream Land over baseline ( Ha & Schmidhuber , 2018 ; Kim et al. , 2020 ) approaches . Furthermore , we perform an extensive set of ablation studies which indicate the source of generalization improvements , requirements for the method to work , and when the method is most useful . 2 RELATED WORKS . 2.1 DROPOUT . Dropout ( Srivastava et al. , 2014 ) was introduced as a regularization technique for feedforward and convolutional neural networks . In its most general form , each unit is dropped with a probability p during the training process . Recurrent neural networks ( RNNs ) initially had issues benefiting from Dropout . Zaremba et al . ( 2014 ) suggests not to apply dropout to the hidden state units of the RNN cell . Gal & Ghahramani ( 2016b ) shortly after show that the mask can also be applied to the hidden state units , but the mask must be fixed across the sequence during training . In this work , we follow the dropout approach from Gal & Ghahramani ( 2016b ) when training the RNN . More formally , for each sequence , the boolean masks mxi , mxf , mxw , mxo , mhi , mhf , mhw , and mho are sampled , then used in the following LSTM update : it = Wxi ( xt mxi ) +Whi ( ht−1 mhi ) + bi , ( 1 ) ft = Wxf ( xt mxf ) +Whf ( ht−1 mhf ) + bf , ( 2 ) wt = Wxw ( xt mxw ) +Whw ( ht−1 mhw ) + bw , ( 3 ) ot = Wxo ( xt mxo ) +Who ( ht−1 mho ) + bo , ( 4 ) where xt , ht , and ct are the input , hidden state , and cell state , respectively , Wxi , Wxf , Wxw , Wxo ∈ Rd×r Whi , Whf , Whw , Who ∈ Rd×d are the LSTM weight matrices , and bi , bf , bw , bo ∈ Rd are the LSTM biases . The masks are fixed for the entire sequence , but may differ between sequences in the mini-batch . 2.2 DOMAIN RANDOMIZATION . The goal of domain randomization ( Tobin et al. , 2017 ; Sadeghi & Levine , 2016 ) is to create many different versions of the dynamics model with the hope that a policy generalizing to all versions of the dynamics model will do well on the true environment . Figure 1 illustrates many simulated environments ( êj ) overlapping with the actual environment ( e∗ ) . Simulated environments are often far cheaper to operate in than the actual environment . Hence , it is desirable to be able to perform the majority of interactions in the simulated environments . Randomization has been applied on observations ( e.g. , lighting , textures ) to perform robotic grasping ( Tobin et al. , 2017 ) and collision avoidance of drones ( Sadeghi & Levine , 2016 ) . Randomization has also proven useful when applied to the underlying dynamics of simulators ( Peng et al. , 2018 ) . Often , both the observations and simulation dynamics are randomized ( Andrychowicz et al. , 2020 ) . Domain randomization generally uses some pre-existing simulator which then injects randomness into specific aspects of the simulator ( e.g. , color textures , friction coefficients ) . Each of the simulated environments in Figure 1 can be thought of as a noisy sample of the pre-existing simulator . To the best of our knowledge , domain randomization has yet to be applied to entirely learned simulators . 2.3 WORLD MODELS . The world model ( Ha & Schmidhuber , 2018 ) has three modules trained separately : ( i ) vision module ( V ) ; ( ii ) dynamics module ( M ) ; and ( iii ) controller ( C ) . A high-level view is shown in Algorithm 1 . Algorithm 1 World Models : Training in dreams . 1 : Initialize parameters of V , M , and C 2 : Collect N trajectories o , d , and a from e∗ 3 : Optimize V on observations o 4 : Generate embeddings z for o with V 5 : Optimize M on z and d 6 : Generate dream environment ê from M 7 : for iteration=1 , 2 , . . . do 8 : Optimize C via interactions with ê The vision module ( V ) is a variational autoencoder ( VAE ) ( Kingma & Welling , 2013 ) , which maps image observation ( o ) to a lower-dimensional representation z ∈ Rn . The dynamics model ( M ) is a mixture density network recurrent neural network ( MDN-RNN ) ( Ha & Schmidhuber , 2018 ; Graves , 2013 ) . It is implemented as an LSTM followed by a fully-connected layer outputting parameters for a Gaussian mixture model with k components . Each feature has k different π parameters for the logits of multinomial distribution , and ( µ , σ ) parameters for the k components in the Gaussian mixture . At each timestep , the MDN-RNN takes in the state z and action a as inputs and predicts π , µ , σ . To draw a sample from the MDN-RNN , we first sample the multinomial distribution parameterized by π , which indexes which of the k normal distributions in the Gaussian mixture to sample from . This is then repeated for each of the n features . Depending on the experiments , Ha & Schmidhuber ( 2018 ) also include an auxiliary head to the LSTM which predicts whether the episode terminates ( d ) . The controller ( C ) is responsible for deciding what actions to take . It takes features produced by the encoder V and dynamics model M as input ( not the raw observations ) . The simple controller is a single-layer model which uses an evolutionary algorithm ( CMA-ES ( Hansen & Ostermeier , 2001 ) ) to find its parameters . Depending on the problem setting , the controller ( C ) can either be optimized directly on the target environment ( e∗ ) or on the dream environment ( ê ) . This paper is focused on the case of optimizing exclusively in the dream environment . 3 DROPOUT ’ S DREAM LAND . In this work we introduce Dropout ’ s Dream Land ( DDL ) . Dropout ’ s Dream Land is the first work to offer a strategy to bridge the reality gap between learned neural network dynamics models and reality . Traditional domain randomization generates many different dynamics models by randomizing configurable parameters of a given simulation . This approach does not apply to neural network dynamics models because they generally do not have configurable parameters ( such as textures and friction coefficients ) . In Dropout ’ s Dream Land , the controller can interact with billions1 of dream environments , whereas previous works ( Ha & Schmidhuber , 2018 ; Kim et al. , 2020 ) only use one dream environment . A naive way to go about this would be to train a large collection of neural network world models . However , this would be computationally expensive . To keep the computational cost low , we go about this by applying dropout to the dynamics model in order to form different dynamics models . Crucially , dropout is applied at both training time and inference time ofM . Each unique dropout mask applied to the dynamics modelM can be viewed as a different environment . Similar to the spirit of domain randomization , an agent is expected to perform well in the real environment if it can perform well in all the different simulated environments . 3.1 LEARNING THE DREAM ENVIRONMENT . The Dropout ’ s Dream Land environments are built around the dynamics model M . During training time the controller interactions are described by Figure 2 . In Figure 2 r̂ , d̂ , and ẑ are generated entirely by M . In this work M is an LSTM where x = [ z > , a > ] > from equations ( 1 ) - ( 4 ) . The LSTM is followed by multiple heads for predictions of the latent state ( ẑ ) , reward ( r̂ ) and termination ( d̂ ) . The reward and termination heads are simple fully-connected layers . Latent state prediction is 1In practice we are bounded by the total number of steps instead of every possible environment . done with a MDN-RNN ( Ha & Schmidhuber , 2018 ; Graves , 2013 ) , but this could be replaced by any other neural network that supports dropout ( e.g. , GameGAN ( Kim et al. , 2020 ) ) .

Summary: The paper introduces the use of dropout for World Models. The argument put forward is that different dropout masks essentially lead to different “dream” environments. The authors compare their approach to the original World Models paper and the recent Game GAN work in the two original World Model environments (Doom and Car Racing). The authors appear to show results that outperform these baselines, and they then perform ablation studies to fully dig into the implications of this application of dropout.