Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors build on the Dreamer architecture, that is able to learn models of an environment, to build DreamerV2, which learns a model of an environment in latent space. The authors then train their agent in this latent space. DreamerV2 was evaluated on the Atari learning environment and results showed that it was comparable to Rainbow and better, under certain metrics.

I am unclear on the motivation of this paper. As with previous papers on model-based learning for Atari (i.e. Kaiser et. al (2019)), the goal of learning a model has been to reduce the number of environment steps. However, the authors use the same number of environment steps with the only difference being the model is trained in latent space. Training the model in latent space can speed up learning. Is this the main contribution of the paper?

There is no analysis as to why using a world model for training might lead to better results than training in the real-world if the same number of environment steps are used. What is the authors' perspective on this? Did DreamerV2 use more steps in the world-model environment than in the real-world environment?

Given that the latent space is trained based on some reconstruction error (instead of only being useful to a value function as with value prediction networks) it is not immediately obvious that the latent space will be a better place to learn a policy. Have the authors tried training their method on the reconstructions? Perhaps this will be slower, but I think it is still relevant since the authors claim that it may be easier to learn a model in latent space: "Predicting compact representations instead of images can reduce the effect of accumulating errors"

In the introduction, the authors say: Several attempts at learning accurate world models of Atari games have been made, without achieving competitive performance (Oh et al., 2015; Chiappa et al., 2017; Kaiser et al., 2019)." I do not think this is a fair statement because papers such as Kaiser et al., 2019 intentionally use fewer environment steps.

---After rebuttal---

The authors have partially addressed my concerns, however, I am still not quite sure why their method would be better than SimPLE. The authors' ablation study of removing image gradients does not address my main question about where the performance benefit is coming from. I am assuming that the architecture and hyperparameters that the authors use are different than SimPLE. I think one must instead replace predicting the latent state with the image as SimPLE did to see if that makes a difference in performance. Therefore, I will be keeping my review the same.
review_title: An application of world models to Atari with an unclear motivation

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The authors introduce DreamerV2, a modification of the influential Dreamer RL agent (hereafter refered to as DreamerV1). The primary changes from DreamerV1 are a discrete latent space and a modified loss function (and with it, a modified optimization scheme). As in DreamerV1, the agent trains a world model with environment experience, and the policy is learned by "imagining" within the learned latent space using the world model to simulate transitions and rewards. They demonstrate superior performance over a variety of successful benchmarks that use similar compute (1 GPU, 1 environment -- e.g. MuZero, which requires vastly more, is not considered) on Atari. Further, they analyze several ways of aggregating Atari scores, and (while their algorithm performs best of those tried in each aggregation), they recommend one aggregation method (along with several other choices made for benchmarking) going forward.

This is impressive work. The modification over DreamerV1 is simple (simple enough that they can describe important optimization details within the main body of the paper -- great!) and the results are a convincing demonstration of its utility. The methods are detailed and well-described. Further, I think the effort spent on exactly how to benchmark (and in particular, to report scores) is very welcome and useful to communicate (though I'll be interested to hear if others object to portions of the recommendations!).

The main weakness: I wish we gained a better understanding of why the discrete latent space works well. Interpretability might be difficult (and I don't necessarily consider that bad -- the results speak for themselves) but we are left with a mix of several plausible hypotheses. Perhaps some visualization might be useful. 

Regardless, I strongly advocate for acceptance. What they propose is relatively simple (and the paper describes it well) and appears to work well in this setting. I'll be recommending students try it in other settings. Further, the benchmarking discussion is very useful for the community.

On the use/interpretation of the discrete latent, a question for the authors: do we have a good sense of what sorts of environments DreamerV2 does poorly on, relative to the continuous latent ablation? DreamerV1 does well on a variety of continuous control tasks -- does DreamerV2 do poorly in continuous control? (Apologies if I missed this described somewhere.) Having some delineation of where it does well vs poorly could help me get a better sense of use of the discrete latent.

Appendix A is most welcome.

----------------------------------

Edit after reading other conversations.

I do think the other reviewers make some fair points. I've adjusted my score to a 9, though, and I'd very much like to see the paper accepted. Here is my thinking on a couple points that led to this score adjustment.

AnonReviewer2's SimPLE-like ablation request: If I'm understanding this correctly, the reviewer would like to see the policy model trained on reconstructions instead of the latent space (or maybe would like the world model trained to future predict in pixel space? not completely clear to me). To me, this would be a potentially insightful ablation, but I think not a dealbreaker that they do not. Two points on this:

(1) If I am understanding the paper and conversations, SimPLE significantly underperforms in the metric (Atari "end performance" under certain normalizations) that the authors care about (and is a fairly established metric). I, then, don't see a *particularly* strong motivation for careful ablations of the method.

(2) Pixel-based future predictions generally perform poorly, and this is I think fairly widely thought to be a strong contributor to the failure of model-based approaches. Again, it would be nice to see that happen here (and it might be insightful to see the quality of the frame predictions) but I think there is a reasonable expectation that this would work poorly.

 
AnonReviewer1's thoughts on the value of this sort of work in ICLR: I see your point, but I personally think there's great value to this sort of work in this sort of venue. End performance on Atari has been (for better or worse) an important baseline for the field, and the creation of performant model-based approaches has been a central question for several years, now. The proposed improvements over DreamerV1 (which has seen fruitful applications in other work) might be simple, but DreamerV1 did not work well on this baseline, and this does. I think that we far too often get excited by complex new methods, often evaluated with novel and poorly understood baselines and metrics, only to drop them as time goes on. That the innovation is a simple one should make us excited to try it in other applications.

I also think that their discussion of evaluation metrics is very useful -- we continually need more careful conversations about the right ways to measure success. So, in short, the paper might be a technically simple innovation, but it puts forth strong evidence that the method is useful.




review_title: Simple and impactful innovation

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary:
The focus of this paper is to extend DreamerV1 to perform harder control tasks such as Atari instead of easier control tasks which were demonstrated in DreamerV1. In doing so, this paper proposes a model-based RL approach which is
- the first Atari agent that achieves human-level performance on the Atari benchmark of 55 tasks by learning behaviors inside a separately trained world model. Human-level performance historically required model-free agents.
- appears to be the first Atari agent which uses a stochastic recurrent state to model the observations and train the policy purely via latent imagination. Consequently, it is much more efficient in terms of training computation in comparison to previous model-based RL for Atari approaches (e.g. SimPLE). 
- outperforms model-free single-GPU Atari approaches such as Rainbow and IQN which rest on years of research. This suggests great promise for model-based approaches.

Novelty of the problem: 
- Cons: The claims of the paper mainly apply in the context of Atari + single-GPU setup. 
- Pros: That being said, it is an important setup from the perspective of model-based RL.

Contributions in terms of analysis: 
- Cons: There could be more analysis about what aspects of Atari games make it different from DreamerV1 tasks, and in this context, why Categorical latents provide a better world model. What are the current limitations of DreamerV2 in terms of Atari? What lessons can we learn about applying it outside Atari / single-GPU setting.

Contribution of the experiments: 
- Pros: Experiments provide valuable proof of concept, showing that model-based RL can outperform top model-free algorithms on Atari benchmark despite years of research and engineering effort. 2) DreamerV2 is also computationally efficient. 3) Provides a good analysis of which metric is appropriate for comparing performance i.e. Mean Clipped Record Normalized.

Novelty of the conceptual idea for the solution: 
- Pros: The model is elegant. For Atari, it appears to be the first model-based approach that uses a recurrent state space model (RSSM) for modeling observations and reward in a stochastic framework. This makes it computationally much faster than previous model-based approaches. The closed-form straight-through gradients for training the policy is also a nice characteristic.
- Cons: Much of this is already proposed in DreamerV1.

Novelty of the low-level implementation of the conceptual idea: 
- Pros: The use of Discrete latents versus Gaussian is interesting. The argument for KL balancing appears convincing.
- Cons: However other innovations are small and not analyzed in detail. 

Presentation:
- Pros: Well-written
- Suggestions: Pseudo-code of the training loop may be helpful. Qualitative examples of imagination may be helpful.

---- After Rebuttal ----
After reading other reviews and the rebuttal, I stay at my current score. Given that the paper does not contain much analysis about "why", the paper is about an empirical discovery of a mode that is critical in getting good performance. I think this kind of discovery paper is also important to share with the community.

review_title: Impressive empirical results. More analysis would make it better.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes DreamerV2, a set of modifications to the existing
model-based RL system Dreamer, and shows via an ablation study that
these modifications improve performance on the standard Atari
benchmark over Dreamer. It is shown that DreamerV2 performs well
compared to model-free RL methods as well, especially when using a new
reporting scheme proposed by the authors that normalizes against the
world record on each game rather than the more standard performance of
a strong human player.

In my view, model-based RL is an extremely important and useful
direction to pursue, and so I really appreciate that the authors are
pushing forward in this direction, trying to prove that model-based RL
can be better than model-free RL. However, there are two major reasons
why I lean toward voting reject on this paper. (1) There is simply not
enough technical novelty to warrant publication. (2) The results
are not particularly compelling to me.

(1) It seems that basically, this paper proposes a couple of small
ideas for how to improve Dreamer, the largest one being the use of
categorical, rather than Gaussian, latent states. While it's nice that
these yield improvements (although the categorical latent doesn't help much
by the authors' suggested reporting scheme, looking at the right-most plot in Figure
3), there is ultimately very little in the way of an interesting
technical/scientific contribution in this paper.

(2) Figure 1 greatly surprises me: although it appears on the first
page, the results do not seem commensurately compelling. On the
standard reporting metric shown in the left plot (which I understand
the authors later suggest not to focus on, for various reasons),
DreamerV2 and Rainbow are indistinguishable, and IQN actually
outperforms both in the low-data regime. Looking at the right plot, I
can hardly say that DreamerV2 outperforms Rainbow. On about 50-60% of the games, it outperforms
Rainbow, but on the others, it performs worse. It would be useful to understand *what*
characteristics of the games DreamerV2 is leveraging, on the games
where it outperforms Rainbow. For instance, are these games perhaps
more amenable to model learning for some reason? (The authors do talk
about why Video Pinball doesn't do well, but I'm curious why do some of the
other domains do very well?)

Some other questions:

1. The authors often mention that DreamerV2 uses only a single
environment instance for each game. Some clarification would be useful
surrounding this. How many instances do the other evaluated approaches
use, like Rainbow? Does this only affect training time, or the learning curves
as well?

2. Since you're only focusing on Atari experiments, where we actually
do have a low-dimensional ground-truth model in the form of the ALE
RAM state, I am curious if the authors have tried ablating away the
world model learning phase of their approach, and instead just use
the ground truth ALE transition model? I ask because the authors often
mention that planning in image space is prohibitively expensive, which I totally
agree with, but planning in RAM space (with the known model) may not
be so bad?
review_title: In my opinion, not enough technical novelty to merit acceptance

Meta-review: The main contribution of the paper is showing that a model-based approach can be competitive with (and even outperform) strong model-free methods on the 200M Atari benchmark. This is achieved through a set of improvements over the original Dreamer algorithm.

Reviewers have been polarized over this submission (4,5,8,9). After reading the paper, reviews, rebuttals, and engaging with all reviewers in private conversations, I am recommending acceptance as a poster. I agree with R3 and R4 that « this is impressive work », « results are a convincing demonstration of its utility », « it is an important setup from the perspective of model-based RL », « the model is elegant », and « the benchmarking discussion is very useful for the community ».

Although it is true, as R1 puts it, that this work can be seen as « an incremental set of tricks over a prior published approach », these tricks are not obvious and lead to very substantial empirical performance gains. Since the authors described them in details and have also committed to sharing their code, I expect them to be quite valuable to other researchers.

Finally, although I respect R2’s choice to stick to their rating of 4, I believe that their main concern, related to not fully understanding why this work improves on the existing SimPLE algorithm, is indeed justified, but is not enough for rejection. DreamerV2 has a lot of differences compared to SimPLE and it would be very costly to investigate in details the impact of each of them. Hopefully, this work will motivate further research in model-based RL that will shed more light on such questions. I would encourage the authors, however, to elaborate a bit more on the differences vs. SimPLE in the « Related work » section (or Appendix, if there is not enough room in the main text).
Decision: Accept (Poster)