Dropout's Dream Land: Generalization from Learned Simulators to Reality

1 INTRODUCTION . Reinforcement learning ( Sutton & Barto , 2018 ) ( RL ) has experienced a flurry of success in recent years , from learning to play Atari ( Mnih et al. , 2015 ) to achieving grandmaster level performance in StarCraft II ( Vinyals et al. , 2019 ) . However , in all these examples , the target environment is a simulator that can be directly trained in . Reinforcement learning is often not a practical solution without a simulator of the environment . Sometimes the target environment is expensive , dangerous , or even impossible to interact with . In these cases , the agent is trained in a simulated source environment . Approaches that train an agent in a simulated environment with the hopes of generalization to the target environment experience a common problem referred to as the reality gap ( Jakobi et al. , 1995 ) . One approach to bridge the reality gap is domain randomization ( Tobin et al. , 2017 ) . The basic idea is that an agent which can perform well in an ensemble of simulations will also generalize to the real environment ( Antonova et al. , 2017 ; Tobin et al. , 2017 ; Mordatch et al. , 2015 ; Sadeghi & Levine , 2016 ) . The ensemble of simulations is generally created by randomizing as much of the simulator as possible without fundamentally changing the task at hand . Unfortunately , this approach is only applicable when a simulator is provided and the simulator is configurable . A recently growing field , World Models ( Ha & Schmidhuber , 2018 ) , focuses on the side of this problem when the simulation does not exist . World Models offer a general framework for optimizing controllers directly in learned simulated environments . The learned dynamics model can be viewed as the agent ’ s dream environment . This is an interesting area because it removes the need for an agent to operate in the target environment . Some related approaches ( Łukasz Kaiser et al. , 2020 ; Hafner et al. , 2019 ; 2020 ; Sekar et al. , 2020 ; Sutton , 1990 ; Kurutach et al. , 2018 ) focus on an adjacent problem which allows the controller to continually interact with the target environment . Despite the recent improvements ( Łukasz Kaiser et al. , 2020 ; Hafner et al. , 2019 ; Sekar et al. , 2020 ; Kim et al. , 2020 ; Hafner et al. , 2020 ) of World Models , none of them address the issue that World Models are susceptible to the reality gap . The learned dream environment can be viewed as the source domain and the true environment as the target domain . Whenever there are discrepancies between the source and target domains the reality gap can cause problems . Even though World Models suffer from the reality gap , none of the domain randomization approaches are directly applicable because the dream environment does not have easily configurable parameters . In this work we present Dropout ’ s Dream Land ( DDL ) , a simple approach to bridge the reality gap from learned dream environments to reality . Dropout ’ s Dream Land was inspired by the first principles of domain randomization , namely , train a controller on a large set of different simulators which all adhere to the fundamental task of the target environment . We are able to generate a nearly infinite number of different simulators via the insight that dropout ( Srivastava et al. , 2014 ) can be understood as learning an ensemble of neural networks ( Baldi & Sadowski , 2013 ) . Our empirical results demonstrate the advantage of Dropout ’ s Dream Land over baseline ( Ha & Schmidhuber , 2018 ; Kim et al. , 2020 ) approaches . Furthermore , we perform an extensive set of ablation studies which indicate the source of generalization improvements , requirements for the method to work , and when the method is most useful . 2 RELATED WORKS . 2.1 DROPOUT . Dropout ( Srivastava et al. , 2014 ) was introduced as a regularization technique for feedforward and convolutional neural networks . In its most general form , each unit is dropped with a probability p during the training process . Recurrent neural networks ( RNNs ) initially had issues benefiting from Dropout . Zaremba et al . ( 2014 ) suggests not to apply dropout to the hidden state units of the RNN cell . Gal & Ghahramani ( 2016b ) shortly after show that the mask can also be applied to the hidden state units , but the mask must be fixed across the sequence during training . In this work , we follow the dropout approach from Gal & Ghahramani ( 2016b ) when training the RNN . More formally , for each sequence , the boolean masks mxi , mxf , mxw , mxo , mhi , mhf , mhw , and mho are sampled , then used in the following LSTM update : it = Wxi ( xt mxi ) +Whi ( ht−1 mhi ) + bi , ( 1 ) ft = Wxf ( xt mxf ) +Whf ( ht−1 mhf ) + bf , ( 2 ) wt = Wxw ( xt mxw ) +Whw ( ht−1 mhw ) + bw , ( 3 ) ot = Wxo ( xt mxo ) +Who ( ht−1 mho ) + bo , ( 4 ) where xt , ht , and ct are the input , hidden state , and cell state , respectively , Wxi , Wxf , Wxw , Wxo ∈ Rd×r Whi , Whf , Whw , Who ∈ Rd×d are the LSTM weight matrices , and bi , bf , bw , bo ∈ Rd are the LSTM biases . The masks are fixed for the entire sequence , but may differ between sequences in the mini-batch . 2.2 DOMAIN RANDOMIZATION . The goal of domain randomization ( Tobin et al. , 2017 ; Sadeghi & Levine , 2016 ) is to create many different versions of the dynamics model with the hope that a policy generalizing to all versions of the dynamics model will do well on the true environment . Figure 1 illustrates many simulated environments ( êj ) overlapping with the actual environment ( e∗ ) . Simulated environments are often far cheaper to operate in than the actual environment . Hence , it is desirable to be able to perform the majority of interactions in the simulated environments . Randomization has been applied on observations ( e.g. , lighting , textures ) to perform robotic grasping ( Tobin et al. , 2017 ) and collision avoidance of drones ( Sadeghi & Levine , 2016 ) . Randomization has also proven useful when applied to the underlying dynamics of simulators ( Peng et al. , 2018 ) . Often , both the observations and simulation dynamics are randomized ( Andrychowicz et al. , 2020 ) . Domain randomization generally uses some pre-existing simulator which then injects randomness into specific aspects of the simulator ( e.g. , color textures , friction coefficients ) . Each of the simulated environments in Figure 1 can be thought of as a noisy sample of the pre-existing simulator . To the best of our knowledge , domain randomization has yet to be applied to entirely learned simulators . 2.3 WORLD MODELS . The world model ( Ha & Schmidhuber , 2018 ) has three modules trained separately : ( i ) vision module ( V ) ; ( ii ) dynamics module ( M ) ; and ( iii ) controller ( C ) . A high-level view is shown in Algorithm 1 . Algorithm 1 World Models : Training in dreams . 1 : Initialize parameters of V , M , and C 2 : Collect N trajectories o , d , and a from e∗ 3 : Optimize V on observations o 4 : Generate embeddings z for o with V 5 : Optimize M on z and d 6 : Generate dream environment ê from M 7 : for iteration=1 , 2 , . . . do 8 : Optimize C via interactions with ê The vision module ( V ) is a variational autoencoder ( VAE ) ( Kingma & Welling , 2013 ) , which maps image observation ( o ) to a lower-dimensional representation z ∈ Rn . The dynamics model ( M ) is a mixture density network recurrent neural network ( MDN-RNN ) ( Ha & Schmidhuber , 2018 ; Graves , 2013 ) . It is implemented as an LSTM followed by a fully-connected layer outputting parameters for a Gaussian mixture model with k components . Each feature has k different π parameters for the logits of multinomial distribution , and ( µ , σ ) parameters for the k components in the Gaussian mixture . At each timestep , the MDN-RNN takes in the state z and action a as inputs and predicts π , µ , σ . To draw a sample from the MDN-RNN , we first sample the multinomial distribution parameterized by π , which indexes which of the k normal distributions in the Gaussian mixture to sample from . This is then repeated for each of the n features . Depending on the experiments , Ha & Schmidhuber ( 2018 ) also include an auxiliary head to the LSTM which predicts whether the episode terminates ( d ) . The controller ( C ) is responsible for deciding what actions to take . It takes features produced by the encoder V and dynamics model M as input ( not the raw observations ) . The simple controller is a single-layer model which uses an evolutionary algorithm ( CMA-ES ( Hansen & Ostermeier , 2001 ) ) to find its parameters . Depending on the problem setting , the controller ( C ) can either be optimized directly on the target environment ( e∗ ) or on the dream environment ( ê ) . This paper is focused on the case of optimizing exclusively in the dream environment . 3 DROPOUT ’ S DREAM LAND . In this work we introduce Dropout ’ s Dream Land ( DDL ) . Dropout ’ s Dream Land is the first work to offer a strategy to bridge the reality gap between learned neural network dynamics models and reality . Traditional domain randomization generates many different dynamics models by randomizing configurable parameters of a given simulation . This approach does not apply to neural network dynamics models because they generally do not have configurable parameters ( such as textures and friction coefficients ) . In Dropout ’ s Dream Land , the controller can interact with billions1 of dream environments , whereas previous works ( Ha & Schmidhuber , 2018 ; Kim et al. , 2020 ) only use one dream environment . A naive way to go about this would be to train a large collection of neural network world models . However , this would be computationally expensive . To keep the computational cost low , we go about this by applying dropout to the dynamics model in order to form different dynamics models . Crucially , dropout is applied at both training time and inference time ofM . Each unique dropout mask applied to the dynamics modelM can be viewed as a different environment . Similar to the spirit of domain randomization , an agent is expected to perform well in the real environment if it can perform well in all the different simulated environments . 3.1 LEARNING THE DREAM ENVIRONMENT . The Dropout ’ s Dream Land environments are built around the dynamics model M . During training time the controller interactions are described by Figure 2 . In Figure 2 r̂ , d̂ , and ẑ are generated entirely by M . In this work M is an LSTM where x = [ z > , a > ] > from equations ( 1 ) - ( 4 ) . The LSTM is followed by multiple heads for predictions of the latent state ( ẑ ) , reward ( r̂ ) and termination ( d̂ ) . The reward and termination heads are simple fully-connected layers . Latent state prediction is 1In practice we are bounded by the total number of steps instead of every possible environment . done with a MDN-RNN ( Ha & Schmidhuber , 2018 ; Graves , 2013 ) , but this could be replaced by any other neural network that supports dropout ( e.g. , GameGAN ( Kim et al. , 2020 ) ) .

Summary: The paper proposes a method for improving the generalisation of model-based  RL algorithms. The paper proposes that one should learn a distribution of transition models, which they do by training with Dropout to improve generalization in model-based RL. While the paper is addressing an important problem, sadly it lacks in novelty, it is unclear if the results are significant and there is a lack of discussion and comparison to related work in this direction.