Dropout's Dream Land: Generalization from Learned Simulators to Reality
Authors: Zac Wellmer
James Kwok
Venue:  (2021)
Keywords: Reinforcement Learning

Abstract: A World Model is a generative model used to simulate an environment. World Models have proven capable of learning spatial and temporal representations of Reinforcement Learning environments. In some cases, a World Model offers an agent the opportunity to learn entirely inside of its own dream environment. In this work we explore improving the generalization capabilities from dream environments to reality (Dream2Real). We present a general approach to improve a controller's ability to transfer from a neural network dream environment to reality at little additional cost. These improvements are gained by drawing on inspiration from domain randomization, where the basic idea is to randomize as much of a simulator as possible without fundamentally changing the task at hand. Generally, domain randomization assumes access to a pre-built simulator with configurable parameters but oftentimes this is not available. By training the World Model using dropout, the dream environment is capable of creating a nearly infinite number of \textit{different} dream environments. Our experimental results show that Dropout's Dream Land is an effective technique to bridge the reality gap between dream environments and reality. Furthermore, we additionally perform an extensive set of ablation studies. 

Under review as a conference paper at ICLR 2021
DROPOUT'S DREAM LAND: GENERALIZATION FROM LEARNED SIMULATORS TO REALITY
Anonymous authors Paper under double-blind review
ABSTRACT
A World Model is a generative model used to simulate an environment. World Models have proven capable of learning spatial and temporal representations of Reinforcement Learning environments. In some cases, World Model offers agent the opportunity to learn entirely inside of its own dream environment. this work we explore improving the generalization capabilities from dream envi- ronments to reality (Dream2Real). We present a general approach to improve a controller's 's ability to transfer from a neural network dream environment to reality at little additional cost. These improvements are gained by drawing inspiration from domain randomization, where the basic idea to randomize as much of a simulator possible without fundamentally changing the task at hand. Generally, domain randomization assumes access to a pre-built simulator with configurable parameters but oftentimes this is not available. By training the World Model using dropout, the dream environment is capable of creating nearly infinite number different dream environments. Our experimental results show that Dropout's s Dream Land is effective technique to bridge the reality gap between dream en- vironments and reality. Furthermore, we additionally perform an extensive set of ablation studies.
INTRODUCTION
Reinforcement learning (Sutton & Barto, 2018) (RL) has experienced a flurry of success in recent years, from learning to play Atari (Mnih al., 2015) achieving grandmaster level performance in StarCraft II (Vinyals et al., 2019). However, in all these examples, the target environment is simulator that can be directly trained in. Reinforcement learning is often not a practical solution without a simulator of the environment.
Sometimes the target environment is expensive, dangerous, or even impossible to interact with. In these cases, the agent is trained in a simulated source environment. Approaches that train an agent in a simulated environment with the hopes of generalization to the target environment experience common problem referred to the reality gap (Jakobi et al., 1995). One approach to bridge the reality gap is domain randomization (Tobin et al., 2017). The basic idea is that an agent which can perform well in an ensemble of simulations will also generalize to the real environment (Antonova al., 2017: Tobin et al., 2017: Mordatch et al., 2015; Sadeghi Levine, 2016). The ensemble of simulations is generally created by randomizing as much of the simulator as possible without fundamentally changing the task hand. Unfortunately, this approach is only applicable when simulator provided and the simulator is configurable.
A recently growing field, World Models (Ha & Schmidhuber, 2018), focuses on the side of this problem when the simulation does not exist. World Models offer a general framework for optimizing controllers directly in learned simulated environments. The learned dynamics model can be viewed as the agent's dream environment. This is an interesting area because it removes the need for an agent to operate in the target environment. Some related approaches (Lukasz Kaiser et al., 2020; Hafner et al., 2019; 2020; Sekar et al., 2020; Sutton, 1990; Kurutach et al., 2018) focus on an adjacent problem which allows the controller continually interact with the target environment.
Despite the recent improvements (Lukasz Kaiser et al., 2020; Hafner et al., 2019; Sekar et al., 2020; Kim et al., 2020; Hafner et al., 2020) of World Models, none of them address the issue that World Models are susceptible to the reality gap. The learned dream environment can be viewed as the source domain and the true environment as the target domain. Whenever there are discrepancies be- tween the source and target domains the reality gap cause problems. Even though World Models

---

Under review as a conference paper at ICLR 2021
suffer from the reality gap, none of the domain randomization approaches are directly applicable be- cause the dream environment does not have easily configurable parameters.
In this work we present Dropout's Dream Land (DDL), a simple approach to bridge the reality gap from learned dream environments reality. Dropout's Dream Land was inspired by the first principles of domain randomization, namely, train a controller on large set of different simulators which all adhere to the fundamental task of the target environment. We are able to generate a nearly infinite number of different simulators via the insight that dropout (Srivastava et al., 2014) can be understood as learning an ensemble of neural networks (Baldi & Sadowski, 2013).
Our empirical results demonstrate the advantage of Dropout's Dream Land over baseline (Ha & Schmidhuber, 2018; Kim et al., 2020) approaches. Furthermore, we perform an extensive set of ablation studies which indicate the source of generalization improvements, requirements for the method to work, and when the method is most useful.
2
RELATED WORKS
2.1 DROPOUT
Dropout (Srivastava et al., 2014) was introduced as regularization technique for feedforward and convolutional neural networks. In its most general form, each unit is dropped with a probability p during the training process. Recurrent neural networks (RNNs) initially had issues benefiting from Dropout. Zaremba et al. (2014) suggests not apply dropout to the hidden state units of the RNN cell. Gal & Ghahramani (2016b) shortly after show that the mask can also be applied to the hidden state units, but the mask must be fixed across the sequence during training.
In this work, we follow the dropout approach from Gal Ghahramani (2016b) when training the RNN. More formally, for each sequence, the boolean masks myi, Mzj, Mrw: Mgo mbo m
Mhor and mão are sampled, then used in the following LSTM update:
i, W zi (Xt O
mai) + Wh: @ mai) + bi,
(1)
f; Wa Wayle (x, mzf) + War 2 mmy) + by:
(2)
W. W ww (X, m Whw (h--1 m + bw
O: W (x, O mgo) + Wno (h,-1 ② maa) bo,
(4)
where X2. h2 and c₁ are the input, hidden state, and cell state, respectively, W... W Waf. zf W zws Wzo Rdxr Whs. Way. W Mee Who E Raxd the LSTM weight matrices, and by by. bws bo E Rd are the LSTM biases. The masks are fixed for the entire sequence, but may differ between sequences the mini-batch.
2.2 DOMAIN RANDOMIZATION
The goal of domain randomization (Tobin et al., 2017: Sadeghi & Levine, 2016) is to create many different versions of the dynamics model with the hope that a policy generalizing to all versions of the dynamies model will do well on the true environment. Figure illustrates many simulated environments (éd) overlapping with the actual environment (e"). Simulated environments are often far cheaper to operate in than the actual environment. Hence, it is desirable to be able perform the majority of interactions in the simulated environments.
Randomization has been applied on observations (e.g., lighting. textures) to perform robotic grasp- ing (Tobin et al., 2017) and collision avoidance of drones (Sadeghi & Levine, 2016). Randomization has also proven useful when applied to the underlying dynamics of simulators (Peng et al., 2018). Often, both the observations and simulation dynamics are randomized (Andrychowicz et al., 2020).
Domain randomization generally uses some pre-existing simulator which then injects randomness into specific aspects of the simulator (e.g., color textures, friction coefficients). Each of the simulated environments in Figure can be thought of as a noisy sample of the pre-existing simulator. To the best of our knowledge, domain randomization has yet to be applied to entirely learned simulators.
2.3
WORLD MODELS
The world model (Ha & Schmidhuber, 2018) has three modules trained separately: (i) vision module (V); (ii) dynamics module (M); and (iii) controller (C). A high-level view is shown in Algorithm 1.
2

---

Under review as a conference paper at ICLR 2021
Algorithm World Models: Training in dreams.
1: Initialize parameters of V. M, and C
2: Collect N trajectories o, and a from e*
3: Optimize V on observations o
4: Generate embeddings for with V
5: Optimize M on z and d
6: Generate dream environment é from M
7: for iteration= 2, ... do
Figure 1: e" is the actual environment, and é 's are randomized variants of the simulated environment.
8: Optimize C' via interactions with è
The vision module (V) is a variational autoencoder (VAB) (Kingma & Welling, 2013), which maps image observation (o) to liver-dimensional representation R".
The dynamics model (M) is mixture density network recurrent neural network (MDN-RNN) (Ha & Schmidhuber, 2018; Graves, 2013). It is implemented as an LSTM followed by folle-connected layer outputting parameters for Gaussian mixture model with & components. Each feature has different л parameters for the logits of multinomial distribution, and (μ,, σ) parameters for the k components in the Gaussian mixture. At each timestep, the MDN-RN takes in the state and action a as inputs and predicts π, μr, a.. To draw a sample from the MDN-RNN, we first sample the multinomial distribution parameterized by æ, which indexes which of the k normal distributions in the Gaussian mixture to sample from. This is then repeated for each of the features. Depending on the experiments, Ha & Schmidhuber (2018) also include an auxiliary head to the LSTM which predicts whether the episode terminates (d).
The controller (C) is responsible for deciding what actions to take. It takes features produced by the encoder V and dynamics model M as input (not the raw observations). The simple controller is single-layer model which uses evolutionary algorithm (CMA-ES (Hansen & Ostermeier, 2001)) to find its parameters. Depending on the problem setting, the controller (C) can either be optimized directly on the target environment (e*) or on the dream environment (é). This paper is focused on the case of optimizing exclusively in the dream environment.
3 DROPOUT'S DREAM LAND
In this work we introduce Dropout's Dream Land (DDL). Dropout's Dream Land is the first work to offer a strategy to bridge the reality gap between learned neural network dynamics models and reality. Traditional domain randomization generates many different dynamics models by randomiz- ing configurable parameters of a given simulation. This approach does not apply to neural network dynamics models because they generally do not have configurable parameters (such as textures and friction coefficients). In Dropout's Dream Land, the controller can interact with billions! ! of dream environments, whereas previous works (Ha & Schmidhuber, 2018; Kim et al., 2020) only use one dream environment. naive way to go about this would be to train large collection of neural network world models. However, this would be computationally expensive.
To keep the computational cost low, go about this by applying dropout to the dynamics model in order to form different dynamics models. Crucially, dropout is applied at both training time and in- ference time of M. Each unique dropout mask applied to the dynamics model M can be viewed as different environment. Similar the spirit of domain randomization, an agent is expected to perform well in the real environment it can perform well in all the different simulated environments.
3.1 LEARNING THE DREAM ENVIRONMENT
The Dropout's Dream Land environments are built around the dynamies model M. During training time the controller interactions are described by Figure 2. In Figure 2 f, d, and å are generated entirely by M. In this work M is LSTM where LSTM is followed by multiple heads for predictions of the latent state (2), reward (f) and termination (d). The reward and termination heads are simple fully fully-connected layers. Latent state prediction is
[z Tz__ a 1 from equations (1)-(4). The
In practice bounded by the total number of steps instead of every possible environment.
3

---

Under review as a conference paper at ICLR 2021
2
M
M
d
a
M
h
Figure 3: Interactions with the real environment. The controller being optimized only interacts with the real environment during the final testing phase.
Figure 2: Interactions with the dream environ- ment. dropout mask is sampled at every step yielding new MP
done with MDN-RNN (Ha & Schmidhuber, 2018; Graves, 2013), but this could be replaced by any other neural network that supports dropout (e.g., GameGAN (Kim et al., 2020)).
3.1.1 Loss FUNCTION
The dynamies model M jointly optimizes all three heads where the loss of single transition is defined as:
L La a.L aaL..
log(E)= t1 N2, R3g)) is a mixture density loss for the latent state predictions, where n is the size of the latent feature vector 2, ãt. is the jth component probability for the ith feature, Ar.. Ô., are the corresponding mean and standard deviation. = (r 8)2 is the square loss on rewards, where and are the true and estimated rewards, respectively. L log - (1 - d) log(1 dd) is the cross-entropy loss for termination prediction, where and d are the true and estimated probabilities of the episode ending, respectively. Constants ord and o, in (5) are for trading off importance of the termination and reward objectives. The loss (CM) aggregated over each sequence and averaged across the mini-batch.
Here,
3.1.2 TRAINING DYNAMICS MODEL M WITH DROPOUT
At training time of M (Algorithm Line 5), we apply dropout (Gal & Ghahramani, 2016b) to the LSTM to simulate different random environments. For each input and hidden unit, first sample a boolean indicator with probability Prain- If the indicator is I, the corresponding input/hidden unit is masked. Masks mai. myf. Myw. Mao- mis- m MAss and m/ are sampled independently (Equations (1)-(4)). When training the RNN, each mini-batch contains multiple sequences. Each sequence uses an independently sampled dropout mask. We fix the dropout mask for the entire sequence this previously found to be critically important (Gal & Ghahramani, 2016b).
Training the RNN with many different dropout masks critical in order to generate multiple differ- ent dynamics models. At the core of domain randomization is the requirement that the randomiza- tions do not fundamentally change the task. This constraint is violated if we do not train the RNN with dropout but apply dropout inference time (explored further Scction 4.4). After optimiz- ing the dynamics model M, we can use to construct dream environments for controller training (Section 3.2).
In this work, we never sample masks to apply to the action (a). We do not zero out the action because in some environments this could imply the agent taking an action (ex: moving to the left). This design choice could be changed depending on the environment, for example, when zero'd action corresponds to no-op or sticky action.
3.2 TRAINING THE CONTROLLER
3.2.1
INTERACTING WITH DROPOUT'S DREAM LAND
Interactions with the dream environment (Algorithm Line 8) can be characterized as training time for the controller (C) and inference time of the dynamics model (M). An episode begins by gen- erating the initial latent state vector by either sampling from standard normal distribution or sampling from the starting points of the observed trajectories used to train M (Ha & Schmidhuber, 2018). The hidden cell (c) and state (h) vectors initialized with zeros.
4

---

Under review as a conference paper at ICLR 2021
The controller (C) decides the action take based 2 and h. In Figure 2 the controller also observes f and d but these are exclusively used for the optimization process of the controller. The controller then performs an action a on a dream environment.
A new dropout mask is sampled (with probability Pinter) and applied to M. We refer to the masked dynamics model as MM and the corresponding Dropout's Dream Land environment as é) The current latent state and action a are concatenated, and passed to M to perform a forward pass. The episode terminates based on a sample from a Bernoulli distribution parameterized by d. The dream environment then outputs the latent state, LSTM's hidden state, reward, and whether the episode terminates.
It is crucial to apply dropout at inference time (of the dynamics model M) in order to create different versions of the dream environment for the controller C. In addition to randomizing masks every step, we will also consider several other variants of how to apply dropout in Sections 4.3 and 4.4.
DDL's use of dropout is different from traditional applications of dropout. In related works (Kahn al., 2017) Monte-Carlo (MC) Dropout (Gal & Ghahramani, 2016a) has been used to approximate the mean and variance of output predictions from an ensemble. We emphasize that DDL does not use MC Dropout. The purpose of DDL's approach dropout is generate many different versions of the dynamics model. More explicitly, the controller is trained to maximize expected returns across many different dynamics models in the ensemble as opposed maximizing expected returns on the ensemble average.
Dropout has also traditionally been used as a model regularizer. Dropout as a model regularizer is only applied training time but at inference time. 'he usual trade-off is lower test loss at the cost of higher training loss (Srivastava et al., 2014; Gal & Ghahramani, 2016b). However, DDL's ultimate goal is not lower test loss. In fact, we expect applying dropout at inference time will consistently make the test loss worse (this will be experimentally verified Section 4.2). The ultimate goal is providing dream environments to a controller so that the optimal policy in Dropout's Dream Land also maximizes expected returns in the target environment (e*).
3.2.2 TRAINING WITH CMA-ES
We follow the same controller optimization procedure was done in World Models (Ha & Schmid- huber, 2018) and GameGAN (Kim et al., 2020) on their Doom TakeCover experiments. We train the controller with CMA-ES (Hansen & Ostermeier, 2001). Each controller in the population (of size Npop) reports their mean returns on a set of Nuials episodes generated in Section 3.2.1. As controllers in the population do not share a dream environment, the probability of controllers interacting with the same sequence of dropout masks is vanishingly small. Let N ep len be the maximum number of steps in an episode. In a single CMA-ES iteration, the population as a whole can interact with Noop X Nuials X Nmax_ep_enn different environments. In our experiments, Npop 64, Nirials 16, and Nmax len 1000 for CarRacing and 2100 for PoomTakeCover. This potentially results in > 1000000 different environments at each generation.
3.2.3 DREAM LEADER BOARD
After every fixed number of generations (25 in our experiments), the best controller in the population (which received the highest average returns across its respective Nural episodes) is selected for evaluation (Ha & Schmidhuber, 2018; Kim et al., 2020). This controller evaluated for another Npop X Nrials episodes in the Dropout's Dream Land environments. The controller's mean across Npop X Nirials trials is logged to the Dream Leader Board. After 2000 generations, the controller at the top of the Dream Leader Board is evaluated in the real environment.
3.2.4 INTERACTING WITH THE REAL ENVIRONMENT
In Figure 3 we illustrate the controller's interaction with the real environment (e"). The controller only interacts with the target environment during testing. These interactions are never used modify parameters of the controller. At test time r. d, and o are generated by the target environment (c") and is the embedding of from the VAE (V). The only use of M when interacting with the target environment producing h as a feature for the controller.
Interactions with e" do not apply dropout to the input/hidden units of M. The purpose of applying dropout when training (Sections 3.2.1 and 3.2.2) and selecting the controller (Section 3.2.3) was to
5

---

0.1 if a new Under review as a conference paper at ICLR 2021
CarKacingFinedN
random policy
NIA
210
108
dream
real
CarRacing-s0
GameGAN
NIA
763
482
rundom policy
NIA
50 I 38
53 1 41
Action LSTM
NIA
280
104
WM
641
351
309
= 135
388 + 137
WM
1465 + 633
849 = 499
DDL
881
214
267
1221
933
Table 2: Returns from a random policy, World Mod-
Table I: Returns from baseline methods and els and DDL (Prain 0.05 and Pinfer 0.1)) on the
DDL (Puain 0.05 and Pinter 0.1) on the CarRacingFixedN and the original CarRacing-ve vO en- vironments.
@bomTakeCover-vl environment.
force the controller to do well on a variety of environments at the cost of dynamics model accuracy (detailed experimental results are in Section 4.2). At the time of interaction with e* the controller's
parameters are frozen and thus there is benefit to trading dynamics model accuracy.
EXPERIMENTS
In this section, we perform experiments on the DoomTakeCoverr (Paquette, 2017) and CarRacing- vO (Klimov, 2016) environments from OpenAl Gym (Brockman et al., 2016). These have also been used in related works (Kim et al., 2020; Ha & Schmidhuber, 2018). Architecture details of V, M, and C' are in Appendix A.1.
Doom' TakeCover is control task in which the goal is to dodge fireballs for as long possible. The controller receives a reward of +1 for every step it is alive. The maximum number of frames is limited to 2100.
CarRacing is a continuous control task to learn from pixels. The race track is split up into "tiles". The goal make it all the way around the track (i.e., crossing every tile). We terminate an episode when all tiles are crossed or when the number of steps exceeds 1000. Let Nutes be the total number of tiles. The simulator (Klimov, 2016) defines the reward rt at each timestep as 100 Nus
tile is crossed, and -0.1 otherwise. The number of tiles is not explicitly set by the simulator. We generated 10000 tracks and observed that the number of tiles in the track appears to follow normal distribution with mean 289. To simplify the reward function, we fix Nutes 289 in the randomly generated tracks, and call the modified environment CarRacingFixedN.
For all experiments the controller is trained exclusively in the dream environment (Section 3,2.1) for 2000 generations. The controller only interacts with the target environments for testing (See- tion 3.2.4). The target environment is never used to update parameters of the controller. Broadly speaking, our experiments are focused on either evaluating the dynamics model (M) or the controller (C). Accuracy of the dynamics model is evaluated against a training and testing set of trajectories (Appendix A.2). The controller is evaluated by returns in the dream and real environments (Ap- pendix A.2).
4.1 COMPARISON WITH BASELINES
We compare the controller trained in Dropout's Dream Land (DDL) with the baseline World Mod- els (WM) controller and a random policy. On the Doom environment we also compare with GameGAN (Kim et al., 2020) and Action-LSTM (Chiappa et al., 2017) All controllers are trained entirely in dream environments. Results on the dream and real environments are in Tables 1 and 2. The CarRacing-v0 results appear different from those found in World Models (Ha & Schmidhuber, 2018) because we are not performing the same experiment. In this paper we train the controller entirely in the dream environment and only interact with the real environment during testing. In World Models (Ha & Schmidhuber, 2018) the controller was trained directly in the CarRacing-vd environment.
In Table we observe that DDL offers performance improvements The DoomTakeCovve-VV returns from DDL are lower than the returns reported by the temperature- regulated variant in Ha & Schmidhuber (2018). 8). Even though the temperature-regulate variant increases uncertainty of the dream environment it is still only capable of creating one dream en- vironment. Furthermore, we emphasize that adjusting temperature is only useful for a limited set of dynamies models. For example, it would not be straightforward to apply temperature any dynamics model which does not produce probability density function (ex: GameGAN); whereas
all the baseline approaches.
Results GameGAN and Action-I.STM returns are from Kim et al. (2020)

---

Under review as a conference paper at ICLR 2021
1.0
Vocel Locs
22-
Vock
1.4
15-
1.0
0.2
0.2
(a) DoonTakeCover.
(b) CarRacingFixedN.
Figure 4: Loss of DDL dynamics model (Purin = 0.05) at different inference dropout rates.
900
800
500
Retume
Rohume
900
500
230
400
130
200
0.0
0.1
0-2
0.3
0.4
0.5
0.0
0.1
0.2
0.2
0.5
Inference Dropout
Inference Dropoun
(a) DoomTakeeovea
(b) FarRacingFixedN,
Figure 5: DDL (Puain = 0.05) returns at different inference dropout rates in the real environments.
the DDL approach of generating many different dynamics models useful to any learned neural network dynamics model.
In Table 2 observe that DDL achieves significant performance boost over WM in both the real CarRacing-v0 and CarRacingFixedds environment variants. This because the WM dream environment was easier for the controller to exploit and overfit to errors between the simulator and reality. Forcing the controller to succeed in many different dropout environments makes difficult to overfit to discrepancies between the dream environment and reality.
4.2 DOES DROPOUT AFFECT ACCURACY OF THE RNN?
In this experiment, compare the losses in (5) achieved by standard RNN and an RNN with dropout (Puain 0.05). The same training and test sets described in Appendix A.2 are used.
Standard use cases of dropout generally observe a larger training loss but lower test loss relative to the same model trained without dropout (Srivastava et al., 2014; Gal & Ghahramani, 2016a). In Table 3, we do not observe any immediate performance improvements of the RNN trained with dropout. In fact, observe worse results the test set. It is possible that the combination of the RNN's small size and dropout are causing the dynamics model underfit.
The poor performance of both DDL RNNs indicates a clear conclusion about the results from Ta- bles and 2. The improved performance of DDL relative to World Models comes from forcing the controller to operate in many different environments and not from a single more accurate dynamics model M.
Doom Toke
CarRacingFixedN
training loss
test loss
training loss
best loss
without dropout
0.89
0.91
2.36
3.10
with dropout
0.93
0.91
3.19
3.57
Table 3: RNN's loss with and without dropout (Prriin 0.05 and Pinfer 0) during training.
4.3 DOES THE INFERENCE DROPOUT RATE AFFECT DREAM2REAL GENERALIZATION?
uncommon to evaluate test loss while still applying dropout. In Dropout's Dream Land this is important because we still apply dropout at inference time of the RNN, In this experiment we evaluate the relationship between the inference dropout rate (Dinter), model loss, and returns in the target environment. Model loss is measured by the loss in (5) on the test sets for varying levels of inference dropout (Piater). Returns in the target environment are reported based on the best controller (Section 3.2.3) trained with varying levels of inference dropout.
Model loss results are shown in Figure 4. As expected, as the inference dropout rate is increased our model loss increases. However, we also observe in Figure that increasing the inference dropout
7

---

Under review as a conference paper at ICLR 2021
rate improves generalization to the target environment. We believe that the increase in returns comes from an increase in capacity to distort the dynamies model. Figures and 5 suggest that we can trade accuracy of the dream environment for better generalization on the target environment. However, this should only be useful up to the point where the lask at hand is fundamentally changed. Figure suggests this point is somewhere between 0.1 and 0.2 for Pinfer: though we suspect in practice this will be highly dependent on network architecture and the environment.
In Figure 5 we observe relatively weak returns on the real CarRacingFixedN environment when the inference dropout rate is zero. Recall from Table 3 that the dropout variant has a much higher test loss than the non-dropout variant on CarRacingFixedN, This means that when Pinter = 0 the single environment DDL is able to create is relatively inaccurate. It is easier for the controller to exploit any discrepancies between the dream environment and target environment because only a single dream environment exists. However, as we increase the inference dropout rate it becomes harder for the controller to exploit the dynamics model, suggesting that DDL is especially useful when it is difficult to learn accurate World Model.
4.4 WHEN SHOULD DROPOUT MASKS BE RANDOMIZED DURING CONTROLLER TRAINING?
In this ablation study we evaluate when the dropout mask should be randomized during training of C. We consider two possible approaches of when to randomize the masks. The first case only randomizes the mask at the beginning of an episode (episode rondomization). The second case samples new dropout mask at every step (step randomization). We also consider if it is effective to only apply dropout at inference time but not during RNN training (i.e., Pinter 0, Purin = 0).
Table 4 shows the results. As can be seen, randomizing the mask at each step offers better returns on both real environments. Better returns in the real environment when applying step randomization comes from the fact that the controller is exposed to much larger number (> 1000x) of dream environments. We also observe that applying step randomization without training the dynamics model with dropout yields a weak policy the real environment. This is due to the randomization fundamentally changing the task. Training the RNN with dropout ensures that at inference time the masked RNN is meaningful.
Doom TakeCorre
FirRacingFixedN
dream
real
dream
episode randomization
505 160
780 +
346 287
601 # 197
(Pean 0.05, Pistes
0.1)
step ramdomization
1221 664
933 1 552
881 214
625 1:
(Pean 0.05, Pirtar = 0.1)
step randomization
320 = 185
339 + 90
331 256
-43 - 52
(Prmin = O, Pirke
0.1)
Table 4: Returns of the controller with different frequencies to randomize the dropout mask.
4.5 DOES THE NUMBER OF DREAM ENVIRONMENTS AFFECT RETURNS?
1000
Motivated by similar experiment from Cobbe et al. (2019), we measure the relationship be- tween the number of inference dropout masks available to the dream environments and returns of the controller in the dream and real CarRac- ingFixedN environments. Limiting the number of masks the same as controlling for the num- ber of possible environments.
800
600
Returns
400
200
Real Returss
At the beginning of controller training we sam- ple m sets' of masks according to Pinfer = 0.1, yielding dream environments. While opti- mizing the controller (Sections 3.2.2 and 3.2.3) randomize the dream environment at every step by uniformly sampling from the m possi- ble dream environments (è) where 0 < < m). Increasing m forces the controller to be capable of maximizing expected returns across a larger number of dream environments.
Dream Returns
1
10"1
10"2
100h
10^4
10"5
Number
Figure 6: Returns achieved by the controller when varying the number of dropout masks in the real and dream CarRacingFixedh environment.
As can be seen from Figure 6, interestingly, a relatively small number of masks offers substantial improvements to the controller's 's ability to generalize to the target environment. We also notice that
Fach set has mai maf Mag. Mo Mai Maf, Maw, and Mio-
8

---

Under review as a conference paper at ICLR 2021
using more masks does not necessarily imply better generalization. This suggests that there exists a smarter way to sample masks, aligned with similar observations in domain randomization (Mehta et al., 2020).
4.6 Is DROPOUT'S 'S DREAM LAND MORE THAN A REGULARIZER?
In this experiment compare Dropout's Dream Land with three regularization methods. First, we consider applying the standard use case of dropout (0 < Purann < and Pinfer = 0). Second, we consider a noisy variant of M, where every adds a small amount of noise (e ~ (0, 10 10-))) Third, we apply L2 regularization to the weights (with L2 penalty À 0.001) during the training process of M.
In Table 5, we do not observe any of the standard regularization methods (L2, Dropout, Noise) to improve World Models generalization from dream environments to target environments. L2 Regu- larized World Models and Dropout World Models can both be viewed as regularizers on M. Noisy World Models be viewed regularizer on the controller C. The strong returns on the real environment by DDL suggest that it is doing more than standard regularization techniques.
CarRacingFixedN
dream
real
CarRacing-vv0
World Models
641 351
399
135
388 + 137
12
Regularized World Models
760 1. 419
- 76 1:
-90
Dropout World Models
1585 + 268
-36 19
-36 + 20
Nuisy World Modllls
639 350
277 135
270 1. 167
Dropout's s Dream Land
881 2 214
625 1 289
610 1 267
Table 5: Returns from World Models, -2-ueuulr World Models
0.001). Dropout World Models (Prainn = 0.05 and Pinfer = 0.0), Noisy (0, 10-$)) World Models, and DDL (Ptrinn = 0.05 and Pinter = 0.1) on the CarRacingFixedN and the original CarRacing- vO environments.
4.7 COMPARISON TO EXPLICIT ENSEMBLE METHODS
In this experiment we compare Dropout's Dream Land with two other approaches for randomizing the dynamics of the dream environment. We consider using an explicit ensemble of a population of dynamics models. Each environment in the population was trained with different initialization and different mini-batches. With the population of World Models we train a controller with Step Ran- domization and controller with Episode Randomization. Note that the training cost of dynamics models and RAM requirements at inference time scale linearly with the population size. Due the large computational cost we consider a population size of 2.
In Table 6, we observe that neither Population World Models (PWM) Step Randomization or Episode Randomization help close the Dream2Sim gap. Episode Randomization does not help because the controller is forced to understand the hidden state (h) representation of every M in the population. Step Randomization performs even worse than Episode Randomization because ontop of the previously state limitations, each of the dynamics models in the population is also forced to be compatible with the hidden state (h) representation of all other dynamics models in the population. Even though we use a modest population size, PWM is plagued with issues that will not be fixed by larger population size. DDL does not suffer from any of the previously stated issues and is also computationally cheaper because only one M must be trained as opposed to an entire population.
CarRacingFixedN real
CarRacing-v0
World Mudels
641 351
399
135
388 157
PWM Fgiisdde Kandomization
724 491
308
= 128
4122 4: 142
PWM Step Randomization
10
78
= 14
-77 13
Dropout's 's Dream Land
881 1: 214
625
289
610 +: 267
Table 6: Returns from Population World Models (PWM) Episode Randomization, PWM Step Ran- domization, Noisy World Models, and DDL (Purinn 0.05 and Pinter 0.1) on the CarRacingFixedN and the original CarRacing-vo environments.
5 CONCLUSION
In this work we introduce approach to improve controller generalization from dream environ- ments to reality at little cost. To the best of our knowledge this is the first work to bridge the reality gap between learned simulators and reality. Future direction for this work could be modifying the dynamics model parameters simple approach to generating different versions of a model could also be useful in committee based methods (Settles, 2009; Sekar et al., 2020).
targeted manner (Wang et al., 2019; 2020: Such al., 2019). This

---

Under review as a conference paper at ICLR 2021
REFERENCES
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, al. Learning dexterous in-hand manipulation. International Journal of Robotics Research, 39(1);3-20, 2020.
Rika Antonova, Silvia Cruciani, Christian Smith, and Danica Kragic. Reinforcement learning for pivoting task. Preprint a 703.00472, 2017.
Pierre Baldi and Peter Sadowski. Understanding dropout. Advances Neural Information Processing Systems, pp. 2814 2822, 2013.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. Preprint
2016.
Silvia Chiappa, Sébastien Racaniere, Daan Wierstra, and Shakir Mohamed. Recurrent environment simulators. Preprint arX:i: 170.02244, 2017.
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman, Quantifying generaliza- tion in reinforcement learning. In International Conference on Machine Learning, pp. 1282-1289. PMLR, 2019.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, pp. 1059, 2016a.
Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 1019-1021 2016b.
Alex Graves. 2013.
Generating sequences with recurrent neural networks. Preprint arXiv: urXiv:1308.06: 130..0800,
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems, pp. 2-62, 2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning. pp. 2555-2565 2019.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.
Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary computation, 9(2):159-195, 2001.
Nick Jakobi, Phil Husbands, and Inman Harvey. Noise and the reality gap: The use of simulation in evolutionary robotics. In European Conference on Artificial Life, pp. 704-720. Springer, 1995.
Gregory Kahn, Adam Villaflor, Vitchyr Pong. Picter Abbecl, and Sergey Levine, Uncertainty-awar reinforcement learning for collision avoidance. arXiv preprint arXiv:/702.011 182, 2017.
Seung Wook Kim, Yuhao Zhou, Jonah Philion, Antonio Torralba, and Sanja Fidler. Learning to simulate dynamic environments with gamegan. IEEE Conference on Computer Vision and Pattern Recognition, pp. 1231-1240, 2020.
Diederik Kingma and Max Welling. Auto-encoding variational Bayes. Preprint arXiv: arXiv:1312.61111 2013.
Oleg Klimov. Carracing-vO, 2016. URLhttps:/ openai conm/envs//araacingggoo
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint arXiv:1802. 10592, 2018.
Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J Pal, and Liam Paull, Active domain randomization. In Conference Robot Learning, pp. 1162-1176, 2020.
10

---

Under review as a conference paper at ICLR 2021
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, Nature, 518(7540):529-555 533, 2015.
Igor Mordatch, Kendall Lowrey, and Emanuel Todorov. Ensemble-cio: Full-body dynamic motion planning that transfers physical humanoids. In IEEE/RSJ International Conference on Intelli- gent Robots and Systems, pp- 5307-5314, 2015.
Philip Paquette. Doomtakeccove---ooo... 2017. URL https::/p//. /ttp::://///..oooo httppp:/ggg...geaaaaccooo/oaaa/a http://ww......co/////////atttta
Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In IEEE International Conference Robotics and Automation, pp- 1-8, 2018.
Fereshtch Sadeghi and Sergey Levine. Cad2rl: Real single-image flight without a single real image. Preprint 1tXiv:1611.04220 0020011 2016.
Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via supf-supe world models. Preprint arXiv:2005.0500 005960, 2020.
Burr Settles. Active learning literature survey. Technical report, University of Misconsin-MMddion Department of Computer Sciences, 2009.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting, Journal of Machine Learning Research. (5()):1929-1958 2014.
Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Genera- tive teaching networks: Accelerating neural architecture search by learning to generate synthetic 2019.
training data. Preprint arXiv:1912.077
Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approxi- mating dynamic programming. In International Conference on Machine learning, pp. 216-224, 1990.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 23-30, 2017.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czameeki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev. et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782):350-3354 2019.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (POET): Endlessly generating increasingly complex and diverse learning environments and their solutions. Preprint arXiv:1901 01153, 2019.
Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeff Clune, and Kenneth O Stanley. Enhanced POET: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. Preprint arXiv:2003.085: 08556, 2020.
Wojeiech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. Preprint Xiv: 140.2229 2014.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Mitos, Blažej Osiński, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, and Hemryk Michalewski. Model based reinforeement learning for atari. In International Conference on Learning Representations, 2020.
11

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 ARCHITECTURE DETAILS
We follow the same architecture setup World Models. We adopt the following notation (Kim al., 2020) to describe the VAE architecture.
(any2D(a, b, c): 2D-Convolution layer with output channel size a, kernel size b, and stride c. All valid padding and relu activations.
T.Conv2D(aa b, c): Transposed 2D-Convolution layer with output channel size a, kernel size b stride The final layer uses sigmoid activation but every other layer uses relu activations.
LSTM(a): LSTM layer with units.
Dense(a): Fully Connected layer with output size a followed by relu activation.
Linear(a): Linear layer with output size a.
Reshape(a): Reshape input to output size
DoomTakeCome
CarRacing Conv2D(32, 4, 2)
Conv2D(32, 4,
Conv2D(64, 2)
Conv2D(64, 4, 2)
Conv2I((128, 4.
Conv2D(128, 4, 2)
Conv2I(256, 4, 2)
Conv2D(256, 4, 2)
Reshape 10224)
Reshape 1024)
Linear(32), Linear(32)
Linear(64), Linear(64)
Dense (024)
(ensc(1024))
Reshape 1, 1024)
Reshape(1, 1, 1024)
T.Conv2D(128,, 5, 2)
T.Conv2D(128, 5, 2)
T.Conv2D(64, 5,
T.Conv2D 5,
T.Conv2D(32, 6, 2)
T.Conv2D(32, 6, 2)
TCConv2(33, 6, 2)
T.Conv2D(3, 6, 2)
Table 7: Architecture for VAE (V) in DooneTakeCover and CarRacing.
DoomTakeCCn
CarRacing
LSTM(512)
LSTM(256)
(661s(9661
Dense(482)
Table 8: Architecture for the dynamics model (M) in Doom TaveCorrr and CarRacing.
Doom love CarRacing Linear(1)
Linear(3)
Table 9; Architecture for the controller (C) in Doom TakeCovrr and CarRacing.
A.2 ENVIRONMENT DETAILS
Means and standard deviations of returns achieved by the best controller (Section 3.2.3) in the target environment (Section 3.2.4) are reported based on 100 trials for CarRacing and 1000 trials for Doom- TakeCover." 4 Returns in the dream environment are reported based on 1024 trials (Section 3.2.3) for both CarRacing and Doom TokeCover.
Doom Tove Environment For all tasks on this environment, we collect training set of 10000 trajectories and test set of 100 trajectories. A trajectory is sequence of state (z), action (a), reward (r), and termination (d) tuples. Both datasets generated according to random policy. Following the same convention as World Models (Ha & Schmidhuber, 2018), on the DoomTakeCover envi- ronment we concatenate h. and input the controller. In (5), set as 1 and a, 0 because the Doom reward function is determined entirely based off whether the controller lives or dies.
$100 trials are used for the baselines GameGAN and Action-I.STM.
12

---

Under review as a conference paper at ICLR 2021
CarRacing Environment For all tasks this environment, the training contains 5000 trajec- tories and the test set contains 100 trajectories. Both datasets are collected by following an expert policy with probability 0.9, and random policy with probability 0.1. The expert policy was trained directly the CarRacing-vll environment and received average return of 885 + 63 across 100 trials. In comparison, the performance of the random policy is -53 41. This is similar to the setup in GameGAN (Kim et al., 2020) on the Pacman environment which also used an expert policy. For this environment, we set Otd ar in (5).
13


Reviews:
[{"review_id": "bEUH-v3zLsR", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method for improving the generalisation of model-based  RL algorithms. The paper proposes that one should learn a distribution of transition models, which they do by training with Dropout to improve generalization in model-based RL. While the paper is addressing an important problem, sadly it lacks in novelty, it is unclear if the results are significant and there is a lack of discussion and comparison to related work in this direction.\n\nQuestions and comments:\n1. There is a series of work in model-based RL focused on ensemble methods which leverage a population of transition models throughout training to improve generalization. For example: \n  * [Kurutach et al., 2017](https://arxiv.org/abs/1802.10592)  which trains a model-free algorithm on purely imaginary data and uses an ensemble of transition models to avoid overfitting to errors made by the individual models.\n  * [Buckman et al. 2018](https://arxiv.org/abs/1807.01675) which trains an ensemble of transition models to capture the uncertainty over next step predictions. \n2. Work by [Kahn et. al., 2017](https://arxiv.org/abs/1702.01182) seems to be also using dropout as an approximation to an ensemble of transition models to reduce overfitting in real-world navigation tasks. It would be helpful to discuss how your work differs from this work?\n3. It is quite unclear why the latest results of [Ha & Schmidhuber, 2018](https://arxiv.org/abs/1809.01999) with adjusted temperature weren’t used as baseline. They are better across the board, and the arguments provided in Section 4.1 do not seem highly convincing. Did you try to combine your method with a temperature-variant of World Model?\n4. While the effects of the dropout probability during inference has been studied in the paper, there is no discussion of why the dropout probability during training has been fixed to 0.05 in all the experiments. Was this as a result of a hyperparameter tuning?\n5. In Ablation 4.4, it appears that step-based dropout is performing better than fixed episode dropout. Why wasn’t step-based dropout selected for the rest of the results? Also, doesn’t this suggest that the original motivation for using dropout as a way to leverage an ensemble of transition models isn’t strictly true and instead dropout is doing some kind of regularisation here instead?\n\nOverall, the problem being addressed by the paper is interesting, however, the contribution isn’t particularly novel. At the same time, the experiments are slightly lacking and it is hard to assess the significance of the results, for example seed variability seems high when/if present (it is actually unclear what the reported variability corresponds to in Table 1 & 2) and most figures only show a single curve without any error bars. \n\nHence, I believe this work is not ready for publication at this time. However, I encourage the authors to improve their work by a) discuss the relevant related work e.g. ensemble methods b) discuss differences to [Kahn et. al., 2017](https://arxiv.org/abs/1702.01182),  c) compare results to an explicit model ensemble (e.g. different initialization of the model) d) report results across multiple training runs,  e) explain the choice of hyperparameters and experimental setup & f) improve the clarity of the paper overall. \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "aWo1xZLMNP", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n# Summary:\n\nThe paper builds on the \"World Models\" (Ha & Shmidhuber 2018) framework with ideas inspired by Domain Randomization (Tobin et. al. 2017 and others). \n\n# Strengths:\n \n - I find the problem well motivated and clear. \n - The solution is intuitive and simple \n\n# Weaknesses:\n\n - Overall I find the contribution over the original World Models work relatively minor. This strikes me as little more than World Models + MC Dropout (Gal and Ghahramani 2015) in the VAE. Perhaps I'm missing something about why this is challenging or difficult. \n\n - Related to the above, there are few statements that I find I bit puzzling. There is a high emphasis put on the fact that we should also apply dropout at test time. It is argued for example that \"DDL’s use of dropout is different from traditional applications of dropout. Generally dropout is only applied at training time but not inference time\" and similar things are repeated other places. Application of dropout at test time is exactly MC Dropout (Gal and Ghahramani) and is very commonly done. I agree that randomizing over the model changes the underlying MDP that the RL agent using for learning. This same problem is known in the domain randomization literature. \n \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "QkME06EEvEJ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces the use of dropout for World Models. The argument put forward is that different dropout masks essentially lead to different “dream” environments. The authors compare their approach to the original World Models paper and the recent Game GAN work in the two original World Model environments (Doom and Car Racing). The authors appear to show results that outperform these baselines, and they then perform ablation studies to fully dig into the implications of this application of dropout. \n\nThis is a relatively simple approach, but that is not necessarily a bad thing and has the potential to be quite valuable. The evaluation shows positive results (though I have concerns I’ll get to in a moment), and the ablation studies offer valuable insights. \n\nI am concerned with the authors evaluation and the baseline performance. From the original World Models paper Ha and Schmidhuber’s approach achieved an average return of 1092 +/- 556, compared to the value reported in this paper as 849 +/- 499. Similarly for Car Racing the original World Models paper reports 906 +/- 21 whereas this paper reports 388 +/- 157. Both of the scores reported by the World Models paper notably outperform the dropout approach from this paper. It is unclear why this is the case. Potentially the authors of this paper may have used a different optimization method besides CMA-ES. Regardless, it is concerning and casts a shadow on the most important part of this paper. \n\nI am willing to be convinced otherwise but based on the evaluation potentially having major flaws and this being the crux of the paper, I lean towards rejecting it. \n\nQuestions: \n-There’s a claim of the size of possible different environments in section 3.2.2, but how different are each of these environments from one another? A visualization or some summarizing statistics would be helpful here.\n-I don’t understand the leaderboard structure used in the evaluation or why it was necessary, could the authors explain this?\n-It may be obvious from up above, but can the authors account for the discrepancy in the performance of the original World Models approach?\n\nSmall note: There’s a couple places in the paper with odd language. The final sentence of the abstract is a bit awkward and the beginning of the related works section rapidly changes tenses when describing prior work. Overall though the paper is very well-written.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "FSu2Ve-X7hx", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\n**Summary**:\n\nThis paper presents a novel idea of using Dropout in the learned dynamics model for domain randomization. While domain randomization has been extremely popular in the field of sim-to-real transfer, it has been relatively less explored when the dynamics model itself is learned. The paper applies Dropout on the recurrent dynamics model (LSTM) and shows that it helps narrow the reality gap between the imagined (dream) environments and the reality.  Overall, I find the idea to be intuitive and simple. The paper also provides good results in a few environments. But the environments used in the paper are not strong enough to support the claim that using Dropout on the learned dynamics model can bridge the reality gap as all the environments are in simulation.\n\n##########################################################################\n\n**Strengths**:\n\nThe idea of applying Dropout on a learned dynamics model is new. The paper is mainly built upon the Recurrent World Model [1]. And it is simple to implement the idea in the Recurrent World Model.\n\nThe paper is well motivated, and it is clear that domain randomization requires extra efforts to be applied on a learned dynamics model, and Dropout is one method to incorporate domain randomization into the recurrent dynamics model.\n\n\n##########################################################################\n\n**Weaknesses**:\n\nThe paper claims that adding Dropout to the dynamics model can bridge the reality gap. However, both environments used in the paper are simulated video game environments. The paper does not really show the capability to narrow the sim-to-**real** gap but rather the sim-to-**sim** gap. It remains unclear whether such an approach will improve the performance of transferring a policy trained in simulation to the real world (for example, transfer a navigation skill from a simulated agent to the real robot).\n\n\nSuggestions on more baselines:\n* A baseline where other regularization techniques (such as weight decay, entropy maximization) are used to prevent the dynamics model M from being overfitted. In section 4.2, the paper shows that adding Dropout leads to higher training and testing loss for the dynamics model `M`, and argues that the reason why adding Dropout helps improve the final performance is that it forces the controller to operate in many different environments created by a less accurate `M`. If we use other regularization techniques such as adding weight decay and maximizing the entropy of the distribution to make sure the dynamics model `M` is underfitted or less overfitted, does the final performance also get improved?\n\n* Another way to add randomness to the dynamics model is adding noise to the prediction. Even though the paper uses a Gaussian Mixture, it could be that after supervised training, the variance of each component becomes small without Dropout. So one can add noise to the output to add more randomness to the prediction. For example, suppose the dynamics model makes a prediction `z`, we can either add a gaussian noise `\\delta` on top of it to get the output `z_o` (`z_o=z+\\delta`) or interpolate between `z` and a uniformly sampled vector `z_u` (`z_o=\\lambda * z+ (1-\\lambda) * z_u`).\n\n\n\nQuestions:\n\nIn Section 2.2, “This is then repeated for each of the n features”. `n` is the dimension of the latent vector `z`. So do you sample a component from the Gaussian mixture for each dimension separately? Why not sample a component and then sample a vector `z` from the selected multivariate Gaussian distribution directly?\n\nIn Figure 2: should there be an arrow that goes from `z` into `M` as the dynamics model `M` takes as input the latent state `z` and action `a`.\n\n\n\n##########################################################################\n\n**Minor points**:\n\nPlease make Figure 6 more legible.\n\nIt would provide more insights if the paper can provide some visualization on diverse simulated trajectories generated from the recurrent model. For example, start from a state `o_0`, apply the same sequence of actions `a_0, a_1, …, a_n`, then use the recurrent model with dropout to generate many sequences of future observations (reconstructed by the VAE decoder). We would expect using the dropout will yield many visually different trajectories with the same initial state and action sequence.\n\n[1] Ha, David, and Jürgen Schmidhuber. \"Recurrent world models facilitate policy evolution.\" Advances in Neural Information Processing Systems. 2018.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Dropout's Dream Land: Generalization from Learned Simulators to Reality

This work explores the use of dropout inside a learned dynamics model, so that when an agent is trained inside an environment generated by this model (rather than the actual environment), the policy learned would do better in the actual environment. In a sense, this is a form of domain randomization in a learned simulator. They show that their approach has better transfer capabilities compared to the baseline World Models method, where an entropy injection via temperature adjustment is used to make transfer more effective, and this is particularly evident in the CarRacing "learn in latent simulation" experiment.

While this work is interesting to me, and I believe it has something to offer to the ICLR community, after reading the reviews and also the detailed author / reviewer discussion (and after understanding and clarifying all of the nuanced points in the experiments), the reviewers and myself believe that this paper needs more work before meeting the bar of ICLR conference acceptance. I believe the author clarified many issues and misunderstandings with the reviewers, and we have made sure the reviewers took that into consideration. Based on the reviews, I have summarized recommendations below to help the authors improve this work. There are 2 dimensions of the work that can be improved:

1) Novelty and Connections with prior works that used dropout in RL

While this work is not using MC dropout (it applies dropout inside the LSTM M), there has been sufficient work (as listed by R2) using MC dropout with RL, and the reviewers' impression that while they may not be exactly the same, it does bring to question of the novelty in this work. It is recommended to discuss not only that the approach is not MC dropout, but also connections with previous work that used MC dropout (or even other forms of dropout) dynamics models to prevent overfitting, and also discuss why this particular approach is needed (or is better, via experiments), over MC dropout, if that can also be used to generate novel dream environments. As R3 mentioned, the idea of applying dropout (of whichever form) on a learned dynamics model *is* new, and this point should be emphasized very clearly to the reader, so I recommend the authors improve the writing to incorporate discussions and relations to previous work (R2 provides a good list), and emphasize what is considered novel in this particular work.

2) Experimental design

The authors show that the proposed method offers a clear advantage over the baseline WM approach for CarRacing, where they show that DDL can do the "train in dream / deploy in real env" transfer much better than the original baseline can. For the Doom experiment, I'm less concerned about the exact score compared to the baseline (as noted by R1), given the high variance of these results, and also the high randomness in the process used to collect data via a random policy, and see that their result is within the margin of statistical error. Just noting the replication effort that went in as a footnote and citing existing results, noting the high variance, should suffice.

What would really improve the work, IMO, is to compare to GameGAN on PacMan environment. Would DDL offer improvements vs GameGAN on PacMan? This will be a strong data point for the proposed method's effectiveness, compared to more trivial tasks such as DoomTakeCover. R3 also brings out a good point that the paper offers better sim-to-sim transfer, rather than sim-to-real transfer. That is another avenue to explore, if this work is to be improved.

The authors have cited PlaNet / Dreamer / SimPle papers, but mentioned that these works don't deal with the issue of reality gap, but I would argue that the iterative learning (data gathering / retraining) aspect of these algorithms is actually one method to address the gap. The tasks studied in these more recent papers, such as DM Control from Pixels, or Atari, have more datapoints, or "leaderboard" participants, using this paper's terminology, so they can also be considered if the authors wish to try DDL to see if this method can help improve the performance of these newer approaches which are based on world models with iterative training and data collection. One can explore whether this approach can lead to better sample efficiency gains, in addition to absolute performance after training, when combined with iterative training in PlaNet / Dreamer type approaches.

Overall, the work in its current form would make a good workshop paper, but I look forward to seeing more work done in the experiments to see better convincing results, in addition to clarifying the writing on related approaches and making contributions / novelty more clear, which I believe will really improve the work for a future submission.
Decision: Reject