Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: The paper proposes a method for improving the generalisation of model-based  RL algorithms. The paper proposes that one should learn a distribution of transition models, which they do by training with Dropout to improve generalization in model-based RL. While the paper is addressing an important problem, sadly it lacks in novelty, it is unclear if the results are significant and there is a lack of discussion and comparison to related work in this direction.

Questions and comments:
1. There is a series of work in model-based RL focused on ensemble methods which leverage a population of transition models throughout training to improve generalization. For example: 
  * [Kurutach et al., 2017](https://arxiv.org/abs/1802.10592)  which trains a model-free algorithm on purely imaginary data and uses an ensemble of transition models to avoid overfitting to errors made by the individual models.
  * [Buckman et al. 2018](https://arxiv.org/abs/1807.01675) which trains an ensemble of transition models to capture the uncertainty over next step predictions. 
2. Work by [Kahn et. al., 2017](https://arxiv.org/abs/1702.01182) seems to be also using dropout as an approximation to an ensemble of transition models to reduce overfitting in real-world navigation tasks. It would be helpful to discuss how your work differs from this work?
3. It is quite unclear why the latest results of [Ha & Schmidhuber, 2018](https://arxiv.org/abs/1809.01999) with adjusted temperature weren’t used as baseline. They are better across the board, and the arguments provided in Section 4.1 do not seem highly convincing. Did you try to combine your method with a temperature-variant of World Model?
4. While the effects of the dropout probability during inference has been studied in the paper, there is no discussion of why the dropout probability during training has been fixed to 0.05 in all the experiments. Was this as a result of a hyperparameter tuning?
5. In Ablation 4.4, it appears that step-based dropout is performing better than fixed episode dropout. Why wasn’t step-based dropout selected for the rest of the results? Also, doesn’t this suggest that the original motivation for using dropout as a way to leverage an ensemble of transition models isn’t strictly true and instead dropout is doing some kind of regularisation here instead?

Overall, the problem being addressed by the paper is interesting, however, the contribution isn’t particularly novel. At the same time, the experiments are slightly lacking and it is hard to assess the significance of the results, for example seed variability seems high when/if present (it is actually unclear what the reported variability corresponds to in Table 1 & 2) and most figures only show a single curve without any error bars. 

Hence, I believe this work is not ready for publication at this time. However, I encourage the authors to improve their work by a) discuss the relevant related work e.g. ensemble methods b) discuss differences to [Kahn et. al., 2017](https://arxiv.org/abs/1702.01182),  c) compare results to an explicit model ensemble (e.g. different initialization of the model) d) report results across multiple training runs,  e) explain the choice of hyperparameters and experimental setup & f) improve the clarity of the paper overall. 



review_title: Official Blind Review #2

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
# Summary:

The paper builds on the "World Models" (Ha & Shmidhuber 2018) framework with ideas inspired by Domain Randomization (Tobin et. al. 2017 and others). 

# Strengths:
 
 - I find the problem well motivated and clear. 
 - The solution is intuitive and simple 

# Weaknesses:

 - Overall I find the contribution over the original World Models work relatively minor. This strikes me as little more than World Models + MC Dropout (Gal and Ghahramani 2015) in the VAE. Perhaps I'm missing something about why this is challenging or difficult. 

 - Related to the above, there are few statements that I find I bit puzzling. There is a high emphasis put on the fact that we should also apply dropout at test time. It is argued for example that "DDL’s use of dropout is different from traditional applications of dropout. Generally dropout is only applied at training time but not inference time" and similar things are repeated other places. Application of dropout at test time is exactly MC Dropout (Gal and Ghahramani) and is very commonly done. I agree that randomizing over the model changes the underlying MDP that the RL agent using for learning. This same problem is known in the domain randomization literature. 
 

review_title: ICLR 2021

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper introduces the use of dropout for World Models. The argument put forward is that different dropout masks essentially lead to different “dream” environments. The authors compare their approach to the original World Models paper and the recent Game GAN work in the two original World Model environments (Doom and Car Racing). The authors appear to show results that outperform these baselines, and they then perform ablation studies to fully dig into the implications of this application of dropout. 

This is a relatively simple approach, but that is not necessarily a bad thing and has the potential to be quite valuable. The evaluation shows positive results (though I have concerns I’ll get to in a moment), and the ablation studies offer valuable insights. 

I am concerned with the authors evaluation and the baseline performance. From the original World Models paper Ha and Schmidhuber’s approach achieved an average return of 1092 +/- 556, compared to the value reported in this paper as 849 +/- 499. Similarly for Car Racing the original World Models paper reports 906 +/- 21 whereas this paper reports 388 +/- 157. Both of the scores reported by the World Models paper notably outperform the dropout approach from this paper. It is unclear why this is the case. Potentially the authors of this paper may have used a different optimization method besides CMA-ES. Regardless, it is concerning and casts a shadow on the most important part of this paper. 

I am willing to be convinced otherwise but based on the evaluation potentially having major flaws and this being the crux of the paper, I lean towards rejecting it. 

Questions: 
-There’s a claim of the size of possible different environments in section 3.2.2, but how different are each of these environments from one another? A visualization or some summarizing statistics would be helpful here.
-I don’t understand the leaderboard structure used in the evaluation or why it was necessary, could the authors explain this?
-It may be obvious from up above, but can the authors account for the discrepancy in the performance of the original World Models approach?

Small note: There’s a couple places in the paper with odd language. The final sentence of the abstract is a bit awkward and the beginning of the related works section rapidly changes tenses when describing prior work. Overall though the paper is very well-written.

review_title: Interesting but major concerns with the evaluation

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################################################################

**Summary**:

This paper presents a novel idea of using Dropout in the learned dynamics model for domain randomization. While domain randomization has been extremely popular in the field of sim-to-real transfer, it has been relatively less explored when the dynamics model itself is learned. The paper applies Dropout on the recurrent dynamics model (LSTM) and shows that it helps narrow the reality gap between the imagined (dream) environments and the reality.  Overall, I find the idea to be intuitive and simple. The paper also provides good results in a few environments. But the environments used in the paper are not strong enough to support the claim that using Dropout on the learned dynamics model can bridge the reality gap as all the environments are in simulation.

##########################################################################

**Strengths**:

The idea of applying Dropout on a learned dynamics model is new. The paper is mainly built upon the Recurrent World Model [1]. And it is simple to implement the idea in the Recurrent World Model.

The paper is well motivated, and it is clear that domain randomization requires extra efforts to be applied on a learned dynamics model, and Dropout is one method to incorporate domain randomization into the recurrent dynamics model.


##########################################################################

**Weaknesses**:

The paper claims that adding Dropout to the dynamics model can bridge the reality gap. However, both environments used in the paper are simulated video game environments. The paper does not really show the capability to narrow the sim-to-**real** gap but rather the sim-to-**sim** gap. It remains unclear whether such an approach will improve the performance of transferring a policy trained in simulation to the real world (for example, transfer a navigation skill from a simulated agent to the real robot).


Suggestions on more baselines:
* A baseline where other regularization techniques (such as weight decay, entropy maximization) are used to prevent the dynamics model M from being overfitted. In section 4.2, the paper shows that adding Dropout leads to higher training and testing loss for the dynamics model `M`, and argues that the reason why adding Dropout helps improve the final performance is that it forces the controller to operate in many different environments created by a less accurate `M`. If we use other regularization techniques such as adding weight decay and maximizing the entropy of the distribution to make sure the dynamics model `M` is underfitted or less overfitted, does the final performance also get improved?

* Another way to add randomness to the dynamics model is adding noise to the prediction. Even though the paper uses a Gaussian Mixture, it could be that after supervised training, the variance of each component becomes small without Dropout. So one can add noise to the output to add more randomness to the prediction. For example, suppose the dynamics model makes a prediction `z`, we can either add a gaussian noise `\delta` on top of it to get the output `z_o` (`z_o=z+\delta`) or interpolate between `z` and a uniformly sampled vector `z_u` (`z_o=\lambda * z+ (1-\lambda) * z_u`).



Questions:

In Section 2.2, “This is then repeated for each of the n features”. `n` is the dimension of the latent vector `z`. So do you sample a component from the Gaussian mixture for each dimension separately? Why not sample a component and then sample a vector `z` from the selected multivariate Gaussian distribution directly?

In Figure 2: should there be an arrow that goes from `z` into `M` as the dynamics model `M` takes as input the latent state `z` and action `a`.



##########################################################################

**Minor points**:

Please make Figure 6 more legible.

It would provide more insights if the paper can provide some visualization on diverse simulated trajectories generated from the recurrent model. For example, start from a state `o_0`, apply the same sequence of actions `a_0, a_1, …, a_n`, then use the recurrent model with dropout to generate many sequences of future observations (reconstructed by the VAE decoder). We would expect using the dropout will yield many visually different trajectories with the same initial state and action sequence.

[1] Ha, David, and Jürgen Schmidhuber. "Recurrent world models facilitate policy evolution." Advances in Neural Information Processing Systems. 2018.

review_title: Official Blind Review #3

Meta-review: Dropout's Dream Land: Generalization from Learned Simulators to Reality

This work explores the use of dropout inside a learned dynamics model, so that when an agent is trained inside an environment generated by this model (rather than the actual environment), the policy learned would do better in the actual environment. In a sense, this is a form of domain randomization in a learned simulator. They show that their approach has better transfer capabilities compared to the baseline World Models method, where an entropy injection via temperature adjustment is used to make transfer more effective, and this is particularly evident in the CarRacing "learn in latent simulation" experiment.

While this work is interesting to me, and I believe it has something to offer to the ICLR community, after reading the reviews and also the detailed author / reviewer discussion (and after understanding and clarifying all of the nuanced points in the experiments), the reviewers and myself believe that this paper needs more work before meeting the bar of ICLR conference acceptance. I believe the author clarified many issues and misunderstandings with the reviewers, and we have made sure the reviewers took that into consideration. Based on the reviews, I have summarized recommendations below to help the authors improve this work. There are 2 dimensions of the work that can be improved:

1) Novelty and Connections with prior works that used dropout in RL

While this work is not using MC dropout (it applies dropout inside the LSTM M), there has been sufficient work (as listed by R2) using MC dropout with RL, and the reviewers' impression that while they may not be exactly the same, it does bring to question of the novelty in this work. It is recommended to discuss not only that the approach is not MC dropout, but also connections with previous work that used MC dropout (or even other forms of dropout) dynamics models to prevent overfitting, and also discuss why this particular approach is needed (or is better, via experiments), over MC dropout, if that can also be used to generate novel dream environments. As R3 mentioned, the idea of applying dropout (of whichever form) on a learned dynamics model *is* new, and this point should be emphasized very clearly to the reader, so I recommend the authors improve the writing to incorporate discussions and relations to previous work (R2 provides a good list), and emphasize what is considered novel in this particular work.

2) Experimental design

The authors show that the proposed method offers a clear advantage over the baseline WM approach for CarRacing, where they show that DDL can do the "train in dream / deploy in real env" transfer much better than the original baseline can. For the Doom experiment, I'm less concerned about the exact score compared to the baseline (as noted by R1), given the high variance of these results, and also the high randomness in the process used to collect data via a random policy, and see that their result is within the margin of statistical error. Just noting the replication effort that went in as a footnote and citing existing results, noting the high variance, should suffice.

What would really improve the work, IMO, is to compare to GameGAN on PacMan environment. Would DDL offer improvements vs GameGAN on PacMan? This will be a strong data point for the proposed method's effectiveness, compared to more trivial tasks such as DoomTakeCover. R3 also brings out a good point that the paper offers better sim-to-sim transfer, rather than sim-to-real transfer. That is another avenue to explore, if this work is to be improved.

The authors have cited PlaNet / Dreamer / SimPle papers, but mentioned that these works don't deal with the issue of reality gap, but I would argue that the iterative learning (data gathering / retraining) aspect of these algorithms is actually one method to address the gap. The tasks studied in these more recent papers, such as DM Control from Pixels, or Atari, have more datapoints, or "leaderboard" participants, using this paper's terminology, so they can also be considered if the authors wish to try DDL to see if this method can help improve the performance of these newer approaches which are based on world models with iterative training and data collection. One can explore whether this approach can lead to better sample efficiency gains, in addition to absolute performance after training, when combined with iterative training in PlaNet / Dreamer type approaches.

Overall, the work in its current form would make a good workshop paper, but I look forward to seeing more work done in the experiments to see better convincing results, in addition to clarifying the writing on related approaches and making contributions / novelty more clear, which I believe will really improve the work for a future submission.
Decision: Reject