Mastering Atari with Discrete World Models
Authors: Danijar Hafner
Timothy P Lillicrap
Mohammad Norouzi
Jimmy Ba
Venue: ICLR 2021 Poster (2021)
Keywords: Atari
world models
model-based reinforcement learning
reinforcement learning
planning
actor critic

Abstract: Intelligent agents need to generalize from past experience to achieve goals in complex environments. World models facilitate such generalization and allow learning behaviors from imagined outcomes to increase sample-efficiency. While learning world models from image inputs has recently become feasible for some tasks, modeling Atari games accurately enough to derive successful behaviors has remained an open challenge for many years. We introduce DreamerV2, a reinforcement learning agent that learns behaviors purely from predictions in the compact latent space of a powerful world model. The world model uses discrete representations and is trained separately from the policy. DreamerV2 constitutes the first agent that achieves human-level performance on the Atari benchmark of 55 tasks by learning behaviors inside a separately trained world model. With the same computational budget and wall-clock time, Dreamer V2 reaches 200M frames and surpasses the final performance of the top single-GPU agents IQN and Rainbow. DreamerV2 is also applicable to tasks with continuous actions, where it learns an accurate world model of a complex humanoid robot and solves stand-up and walking from only pixel inputs.

Published as conference paper at ICLR 2021
MASTERING ATARI WITH DISCRETE WORLD MODELS
Danijar Hafner
Timothy Lillierap DeepMind
Mohammad Norouzi
Jimmy Ba University of Toronto
Google Research
Google Research
ABSTRACT
Intelligent agents need to generalize from past experience to achieve goals in complex environments. World models facilitate such generalization and allow learning behaviors from imagined outcomes to increase sample-efficiency. While learning world models from image inputs has recently become feasible for some tasks, modeling Atari games accurately enough to derive successful behaviors has remained an open challenge for many years. We introduce DreamerV2, a reinforcement learning agent that learns behaviors purely from predictions in the compact latent space of powerful world model. The world model uses discrete representations and is trained separately from the policy. DreamerV2 constitutes the first agent that achieves human-level performance on the Atari benchmark of 55 tasks by learning behaviors inside separately trained world model. With the same computational budget and wall-clock time, Dreamer V2 reaches 200M frames and surpasses the final performance the top single-GPU agents IQN and Rainbow. DreamerV2 is also applicable to tasks with continuous actions, where it learns an accurate world model of a complex humanoid robot and solves stand-up and walking from only pixel inputs.
INTRODUCTION
Atari Performance
To successfully operate in unknown environments, re- inforcement learning agents need to learn about their environments over time. World models are an explicit way to represent agent's knowledge about its environ- ment. Compared to model-free reinforcement learning that learns through trial and error, world models facilitate generalization and can predict the outcomes of potential actions enable planning (Sutton, 1991). Capturing gen- eral aspects of the environment, world models have been shown be effective for transfer novel tasks (Byravan et al., 2019), directed exploration (Sekar et al., 2020), and generalization from offline datasets (Yu al., 2020). When the inputs are sigh--imenssonal images, latent dy- namics models predict ahead an abstract latent space (Watter et al., 2015; Ha and Schmidhuber, 2018; Hafner et al.. 2018; Zhang et al., 2019). Predicting compact representations instead of images has been hypothesized to reduce accumulating errors and their small memory footprint enables thousands of parallel predictions single GPU (Hafner et al., 2018; 2019). Leveraging this approach, the recent Dreamer agent (Hafner et al., 2019) has solved a wide range of continuous control tasks from image inputs.
2.0
Model-based Model-free
1.6
1.2
Human Gamer
0.8
0.4
0.0
DreamerV2
Rainbow
ION
DON
DreamerV1
SimPLe
Figure Gamer normalized median score on the Atari benchmark of 55 games with sticky actions at 200M steps. DreamerV2 is the first agent that learns purely within a world model to achieve human-level Atari performance, demon- strating the high accuracy of its learned world model. DreamerV2 further outper- forms the top single-GPU agents Rain- bow and IQN, whose scores are provided by Dopamine (Castro et al., 2018). Ac- cording its authors, SimPLe (Kaiser et al., 2019) was only evaluated easier subset of 36 games and trained for fewer steps and additional training does not further increase its performance.
Despite their intriguing properties, world models have far not been accurate enough to compete with the state- of-the-art model-free algorithms on the most competi- tive benchmarks. The well-established Atari benchmark
an
"Correspondence Danijar Hafner <mail danijar.com>.

---

Published as conference paper at ICLR 2021
(Bellemare et al., 2013) historically required model-free algorithms to achieve human-level per- formance, such as DQN (Mnih et al., 2015), A3C (Mnih et al., 2016), or Rainbow (Hessel et al., 2018). Several attempts at learning accurate world models of Atari games have been made, without achieving competitive performance (Oh et al., 2015; Chiappa et al., 2017: Kaiser et al., 2019). On the other hand, the recently proposed MuZero agent (Schrittwieser et al., 2019) shows that planning can achieve impressive performance on board games and deterministic Atari games given extensive engineering effort and vast computational budget. However, its implementation not available to the public and it would require over 2 months of computation to train even one agent on a GPU, rendering impractical for most research groups.
In this paper, we introduce DreamerV2, the first reinforcement learning agent that achieves human- level performance on the Atari benchmark by learning behaviors purely within a separately trained world model, as shown in Figure 1. Learning successful behaviors purely within the world model demonstrates that the world model learns to accurately represent the environment. To achieve this, we apply small modifications to the Dreamer agent (Hafner et al., 2019), such as using discrete latents and balancing terms within the KI. loss. Using a single GPU and a single environment instance, D2eamerV2 outperforms top single-GPU Atari agents Rainbow (Hessel et al., 2018) and IQN (Dabney et al., 2018), which rest upon years of model-free reinforcement learning research (Van Hasselt et al., 2015; Schaul et al., 2015; Wang al., 2016; Bellemare et al., 2017; Fortunato et al., 2017). Moreover, aspects of these algorithms are complementary to our world model and could be integrated into the Dreamer framework in the future. To rigorously compare the algorithms, we report scores normalized by both a human gamer (Mnih et al., 2015) and the human world record (Toromanoff et al., 2019) and make a suggestion for reporting scores going forward.
2 DREAMERV2
We present Dreamer V2, an evolution of the Dreamer agent (Hafner et al., 2019). We refer to the original Dreamer agent as DreamerV1 I throughout this paper. This section describes the complete DreamerV2 algorithm, consisting of the three typical components of model-based agent (Sutton, 1991). We learn the world model from a dataset of past experience, learn an actor and critic from imagined sequences of compact model states, and execute the actor in the environment to grow the experience dataset. In Appendix C, we include list of changes that we applied to DreamerV1 and which of them we found to increase empirical performance.
2.1 WORLD MODEL LEARNING
World models summarize an agent's ' experience into a predictive model that can be used in place of the environment to learn behaviors. When inputs are high-dimensional images, it is beneficial to learn compact state representations of the inputs to predict ahead in this learned latent space (Watter et al., 2015; Karl et al., 2016; Ha and Schmidhuber, 2018). These models are called latent dynamics models. Predicting ahead in latent space not only facilitates long-term predictions, it also allows to efficiently predict thousands of compact state sequences in parallel in a single batch, without having to generate images. DreamerV2 builds upon the world model that introduced by PlaNet (Hafner et al., 2018) and used in DreamerV1, by replacing its Gaussian latents with categorical variables.
The world model is trained from the agent's growing dataset of past experience that contains sequences of images 1, actions d1.T. rewards TI.T, and discount factors Yi... The discount factors equal fixed hyper parameter 7 = 0.999 for time steps within an episode and are set to zero for terminal time steps. For training,
Experience dataset
batches of B 50 sequences of fixed length L = 50 that are sampled randomly within the stored episodes. To observe enough episode ends during training, we sample the start index of each training sequence uniformly within the episode and then clip it to not exceed the episode length minus the training sequence length.
Model components The world model consists of an image encoder, a Recurrent StateSSpace Model (RSSM; Hafner et al., 2018) to learn the dynamics, and predictors for the image, reward, and discount factor. The world model is summarized in Figure 2. The RSSM uses a sequence of deterministic recurrent states he, from which it computes two distributions over stochastic states at each step. The posterior state 2 incorporates information about the current image 2,, while the prior state & aims to predict the posterior without access the current image. The concatenation of deterministic and
2

---

Published as conference paper at ICLR 2021
a
2
h,
h2
hg
32 categoricals
min KL
min KL
min KL
X
32 classes each
Figure 2: World Model Learning. The training sequence of images 2₁ is encoded using the CNN. The RSSM uses a sequence of deterministic recurrent states h4 At each step, it computes a posterior stochastic state 2; that incorporates information about the current image stochastic state & that tries to predict the posterior without access to the current image. Unlike in PlaNet and DreamerV1, 1, the stochastic state of Dreamer V2 is a vector of multiple categorical variables. The learned prior is used for imagination, as shown in Figure 3. The KI. loss both trains the prior and regularizes how much information the posterior incorporates from the image. The regularization increases robustness to novel inputs. It also encourages reusing existing information from past steps predict rewards and reconstruct images, thus learning long-term dependencies.
as well as prior
stochastic states forms the compact model state. From the posterior model state, we reconstruct the current image v, and predict the reward r; and discount factor "/. The model components are:
Recurrent model:
h4 1
11
RSSM
Representation model:
(z
Transition predictor:
&. p (
h+)
(1)
Image predictor:
R, ps (Er
hu, 2t) hes 2t.
Reward predictor:
A. ~ po (f:
Discount predictor:
f. (f he, 2t)-
All components are implemented as neural networks and & describes their combined parameter vector. The transition predictor guesses the next model state only from the current model state and the action but without using the next image, so that we can later learn behaviors by predicting sequences of model states without having to observe or generate images. The discount predictor lets us estimate the probability of an episode ending when learning behaviors from model predictions.
Neural networks The representation model is implemented as Convolutional Neural Network (CNN; LeCun et al., 1989) followed by Multi-Layer Perceptron (MLP) that receives the image embedding and the deterministic recurrent state. The RSSM uses Gated Recurrent Unit (GRU; Cho et al., 2014) to compute the deterministic recurrent states. The model state is the concatenation of deterministic GRU state and a sample the stochastic state. The image predictor is a transposed CNN and the transition, reward, and discount predictors MLPs. We down-scale the 84 x 84 grayscale images to 64 x 64 pixels so that we can apply the convolutional architecture of DreamerV1.
Algorithm 1: Straight- Gradients with Automatic Differentiation
sample one ooe_hot (draw (logits))
I sample has no gradient want gradient of this has gradient probs
probs softmax (logits)
sample sample probs stop stop_grad (probs)
3

---

Published as conference paper at ICLR 2021
We use the ELU activation function for all components of the model (Clevert al., 2015). The world model uses a total of 20M trainable parameters.
Distributions The image predictor outputs the mean of a diagonal Gaussian likelihood with unit variance, the reward predictor outputs univariate Gaussian with unit variance, and the discount predictor outputs Bernoulli likelihood. In prior work, the latent variable in the model state was a diagonal Gaussian that used reparameterization gradients during backpropagation (Kingma and Welling, 2013; Rezende et al., 2014). In DreamerV2, we instead use vector of several categorical variables and optimize them using straight-throug gradients (Bengio et al., 2013), which are easy to implement using automatic differentiation shown in Algorithm We discuss possible benefits of categorical over Gaussian latents in the experiments section.
Loss function All components of the world model are optimized jointly. The distributions produced by the image predictor, reward predictor, discount predictor, and transition predictor are trained to maximize the log-likeliing lig-liiklihood of their corresponding targets. The representation model is trained to produce model states that facilitates these prediction tasks, through the expectation below. Moreover, it is regularized to produce model states with high entropy, such that the model becomes robust to many different model states during training. The loss function for learning the world model is:
( = Egglsuer (ur,))))))
ΣL
po h2,2) 21) In In he, 2c) In ( he 2t) reward log loss
imago log
discount log loss
+ B KL ( he, m2)
|| (o ha)l
KL loss
We jointly minimize the loss function with respect to the vector & that contains all parameters of the world model using the Adam optimizer (Kingma and Ba, 2014). We scale the loss by 0.1 for Atari and by 3 = 1.0 for continuous control (Higgins et al., 2016).
KL balancing The world model loss function in Equation 2 is the ELBO or variational free energy of a hidden Markov model that is conditioned on the action sequence. The world model can thus be interpreted as sequential VAE, where the representation model is the approximate posterior and the transition predictor is the temporal prior. In the ELBO objective, the KL. loss serves two purposes: it trains the prior toward the representations, and it regularizes the representations toward the prior. However, learning the transition function is difficult and we want to avoid regularizing the representations toward poorly trained prior. To solve this problem, we minimize the KL loss faster with respect to the prior than the representations by using different learning rates, = 0.8 for the prior and 1 a for the approximate posterior. We implement this technique as shown in Algorithm 2 and refer to it as KL balancing. KL. balancing encourages learning an accurate prior over increasing posterior entropy, so that the prior better approximates the aggregate posterior. KL balancing is different from and orthogonal to beta-VAEs (Higgins et al., 2016).
2.2 BEHAVIOR LEARNING
DreamerV2 learns long-horizon behaviors purely within its world model using an actor and a critic. The actor chooses actions for predicting imagined sequences of compact model states. The critic accumulates the future predicted rewards take into account rewards beyond the planning horizon. Both the actor and critic operate on top of the learned model states and thus benefit from the representations learned by the world model. The world model is fixed during behavior learning, so the actor and value gradients do not affect its representations. Not predicting images during behavior learning lets us efficiently simulate 2500 latent trajectories in parallel on a single GPU.
Imagination MDP To learn behaviors within the latent space of the world model, we define the imagination MPD follows. The distribution of initial states ån in the imagination MDP is the distribution of compact model states encountered during world model training. From there, the transition predictor (&, (--1,ū=1) outputs sequences Ê1.1 of compact model states up to the
Algorithm 2: KL Balancing with Automatic Differentiation
k1 loss
alpha
compute kl (s (rppoxx_x__ostteeric prior) gaap_grardddii (prior))
+ (1 alpha)
compute kl
4

---

Published as conference paper at ICLR 2021
à
h2
h
hg
h
Figure 3: Actor Critic Learning. The world model learned in Figure 2 is used for learning policy from trajectories imagined the compact latent space. The trajectories start from posterior states computed during model training and predict forward by sampling actions from the actor network. The critic network predicts the expected sum of future rewards for each state. The critic uses temporal difference learning the imagined rewards. The actor is trained to maximize the critic prediction, via reinforce gradients, straight-through gradients of the world model, or combination of them.
imagination horizon H = 15. The mean the reward predictor Po 2x) is used as reward sequence fI.H. The discount predictor Pa(Sx &)) outputs the discount sequence 91. that is used to down-weight rewards. Moreover, we weigh the loss terms of the actor and critic by the cumulative predicted discount factors softly account for the possibility of episode ends.
Model components To learn long-horizon behaviors in the imagination MDP, we leverage a stochastic actor that chooses actions and a deterministic critic. The actor and critic are trained cooperatively, where the actor aims to output actions that lead to states that maximize the critic output, while the critic aims to accurately estimate the sum of future rewards achieved by the actor from each imagined state. The actor and critic use the parameter vectors v and ç, respectively:
Actor:
ā Pul (āe. (x))
(3)
Critic:
uy (x( ~ Epsipe
In contrast to the actual environment, the latent state sequence is Markovian, that there is no need for the actor and critic to condition on more than the current model state. The actor and critic are both MLPs with ELU activations (Clevert et al., 2015) and use 1M trainable parameters each. The actor outputs categorical distribution over actions and the critic has a deterministic output. The two components are trained from the imagined trajectories but optimize separate loss functions.
Critic loss function The critic aims to predict the discounted sum of future rewards that the actor achieves in given model state, known the state value. For this, leverage -inporal-differenee learning, where the critic trained toward a value target that is constructed from intermediate rewards and critic outputs for later states. A common choice is the 1-step target that sums the current reward and the critic output for the following state. However, the imagination MDP lets us generate on on-policy trajectories of multiple steps, suggesting the use of n-step targets that incorporate reward information into the critic more quickly. We follow DreamerV1 in using the more general A-target (Sutton and Barto, 2018; Schulman et al., 2015) that is defined recursively follows:
V = A. fe (1 A)ve (8 AVA if t< H. (0g(2H)
if t=
Intuitively, the X-target is a weighted average of n-step returns for different horizons, where longer horizons are weighted exponentially less. We set À 0.95 in practice, focus long horizon

---

Published as conference paper at ICLR 2021
targets than short horizon targets. Given trajectory of model states, rewards, and discount factors, we train the critic to regress the X-return using a squared loss:
((g) = F.p..P EE 3 (vg(E)) sg((A)))
We optimize the critic loss with respect to the critic parameters & using the Adam optimizer. There no loss term for the last time step because the target equals the critic at that step. We stop the gradients around the targets, denoted by the sg(.) function, as typical in the literature. We stabilize value learning using target network (Mnih al., 2015), namely, we compute the targets using a copy of the critic that is updated every 100 gradient steps.
Actor loss function The actor aims to output actions that maximize the prediction of long-term future rewards made by the critic. To incorporate intermediate rewards more directly, we train the actor to maximize the same A-return that was computed for training the critic. There are different gradient estimators for maximizing the targets with respect to the actor parameters. Dreamer combines unbiased but high-variance Reinforce gradients with biased but low-variance straight- through gradients. Moreover, we regularize the entropy of the actor to encourage exploration where feasible while allowing the actor to choose precise actions when necessary.
Learning by Reinforce (Williams, 1992) maximizes the actor's 's probability of its own sampled actions weighted by the values those actions. The variance of this estimator can be reduced by subtracting the state value as baseline, which does not depend on the current action. Intuitively, subtracting the baseline centers the weights and leads to faster learning. The benefit of Reinforce that it produced unbiased gradients and the downside is that it can have high variance, even with baseline.
DreamerV1 1 relied entirely reparameterization gradients (Kingma and Welling, 2013; Rezende et al., 2014) to train the actor directly by backpropagating value gradients through the sequence of sampled model states and actions. DreamerV2 uses both discrete latents and discrete actions. To backpropagate through the sampled actions and state sequences, we leverage straight-through gradients (Bengio et al., 2013). This results in biased gradient estimate with low variance. The combined actor loss function is:
E# (-plnpa(aa sg) Dg (2+)) (1 (1-p) P)V Hay )]
((b) = Epp.Po
(6)
reinforce
dynamics backprop
entropy regularizer
We optimize the actor loss with respect to the actor parameters &i using the Adam optimizer. We consider both Reinforce gradients and ttraight-throug gradients, which backpropagate directly through the learned dynamics. Intuitively, the vor-aariance but biased dynamics backpropagation could learn faster initially and the unbiased but high-variance could converge better solution. For Atari, we find Reinforce gradients to work substantially better and use p and 10-3 For continuous control, we find dynamics backpropagation work substantially better and use p and = 10-4, Annealing these hyper parameters can improve performance slightly but to avoid the added complexity we report the scores without annealing.
3 EXPERIMENTS
We evaluate DreamerV2 on the well-established Atari benchmark with sticky actions, comparing to four strong model-free algorithms. DreamerV2 outperforms the four model-free algorithms in all scenarios. For extensive comparison, report four scores according to four aggregation protocols and give a recommendation for meaningfully aggregating scores across games going forward. We also ablate the importance of discrete representations in the world model. Our implementation of DreamerV2 reaches 200M environment steps in under 10 days, while using only a single NVIDIA V100 GPU and single environment instance. During the 200M environment steps. Dreamer learns its policy from 468B compact states imagined under the model, which is 10,000x more than the 50M inputs received from the real environment after action repeat. Refer to the project website for videos, the source code, and training curves in JSON format.
"htsps://dani jar.

---

Published as conference paper at ICLR 2021
Gamer Median
Gamer Mean
Record Mean
Clipped Record Mean
0.45
2.4
40
0.24
1.8
30
0.30
0.16
1.2
20
0.15
0.08
0.6
10
0.0
0.00
0.00
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
DreamerV2
IQN
Rainbow
C51
DQN
1e8
Figure 4: Atari performance over 200M steps. See Table 1 for numeric scores. The standards in the literature aggregate over tasks are shown the left two plots. These normalize scores by professional gamer and compute the median mean over tasks (Mnih et al., 2015; 2016). In Section 3, we point out limitations of this methodology. As a robust measure of performance, we recommend the metric in the right-most plot. We normalize scores by the human world record (Toromanoff et al., 2019) and then clip them, such that exceeding the record does not further increase the score, before averaging over tasks.
Experimental setup We select the 55 games that prior works in the literature from different research labs tend agree (Mnih al., 2016; Brockman al., 2016; Hessel et al., 2018; Castro et al., 2018; Badia et al., 2020) and recommend this set of games for evaluation going forward. We follow the evaluation protocol of Machado al. (2018) with 200M environment steps, action repeat of 4, time limit of 108,000 steps per episode that correspond to 30 minutes of game play, no access life information, full action space, and sticky actions. Because the world model integrates information over time, DreamerV2 does not use frame stacking. The experiments use single-task setup where a separate agent is trained for each game. Moreover, each agent uses only single environment instance. We compare the algorithms based on both human gamer and human world record normalization (Toromanoff et al., 2019).
Model-free baselines We compare the learning curves and final scores of DreamerV2 to four model-free algorithms, IQN (Dabney et al., 2018), Rainbow (Hessel al., 2018), C51 (Bellemare et al., 2017), and DQN (Mnih et al., 2015). We use the scores these agents provided by the Dopamine framework (Castro et al., 2018) that sticky actions. These may differ from the reported results in the papers that introduce these algorithms the deterministic Atari setup. The training time of Rainbow was reported 10 days on single GPU and using one environment instance.
3.1 ATARI PERFORMANCE
The performance curves of DreamerV2 and four standard model-free algorithms are visualized in Figure 4. The final scores at 200M environment steps are shown in Table and the scores on individual games included in Table K1. There are different approaches for aggregating the scores across the 55 games and we show that this choice can have a substantial impact on the relative performance between algorithms. To extensively compare DreamerV2 the model-free algorithms, we consider the following four aggregation approaches:
Agent
Gamer Median
Gamer Mean
Record Mean
Clipped Record Mean
DreamerV2
2.15
42.26
0.44
0.28
DreamerV2 (schedules)
2.64
31.71
0.43
0.28
IMPALA
1.92
16.72
0.34
0.23
IQN
1.29
11.27
0.21
0.21
Rainbow
1.47
9,95
0.17
0.17
C51
1.09
8.25
0.15
0.15
DQN
0.65
3.28
0.12
0.12
Table 1: Atari performance at 200M steps. The scores of the 55 games are aggregated using the four different protocols described Section 3. To overcome limitations of the previous metrics, we recommend the task mean of clipped record normalized scores robust measure of algorithm performance, shown the right-most column. DreamerV2 outperforms previous single-GPU agents across all metrics. The baseline scores taken from Dopamine Baselines (Castro et al., 2018).
7

---

Published as conference paper at ICLR 2021
Latent Variables
KL Balancing
Image Gradients
Reward Gradients
0.24
0.24
0.24
0.24
0.18
0.18
0.16
0.18
0.12
0.12
0.12
0.08
0.06
0.06
0.06
0.00
0.00
0.00
0.00
0.0
0.5 1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Categorical Gaussian
Enabled Disabled
Enabled Disabled
Enabled 1e8 Disabled
Figure 5: Clipped record normalized scores of various ablations of the DreamerV2 agent. This experiment uses slightly earlier version of DreamerV2. The score curves for individual tasks are shown in Figure H1. The ablations highlight the benefit of using categorical over Gaussian latent variables and of using KL balancing. Morcover, they show that the world model relies image gradients for learning its representations. Stopping reward gradients even improves performance on some tasks, suggesting that representations that not specifically trained to predict previously e experienced rewards may generalize better to new situations.
Gamer Median Atari scores commonly normalized based on random policy and professional gamer, averaged over seeds, and the median over tasks is reported (Mnih et al., 2015; 2016). However, if almost half of the scores would be zero, the median would not be affected. Thus, we argue that median scores are not reflective of the robustness of an algorithm and results wasted computational resources for games that will not affect the score.
Gamer Mean Compared to the task median, the task mean considers all tasks. However, the gamer performed poorly on small number of games, such as Crazy Climber, James Bond, and Video Pinball. This makes it easy for algorithms to achieve a high normalized score on these few games, which then dominate the task mean so it is not informative of overall performance.
Record Mean Instead of normalizing based on the professional gamer, Toromanoff et al. (2019) suggest to normalize based on the registered human world record of each game. This partially addresses the outlier problem but the mean is still dominated by games where the algorithms easily achieve superhuman performance.
Clipped Record Mean To overcome these limitations, we recommend normalizing by the human world record and then clipping the scores to not exceed a value of 1, so that performance above the record does not further increase the score. The result is robust measure of algorithm performance on the Atari suite that considers performance across all games.
From Figure 4 and Table 1, see that the different aggregation approaches let us examine agent performance from different angles. Interestingly. Rainbow clearly outperforms IQN in the first aggregation method but IQN clearly outperforms Rainbow in the remaining setups. DreamerV2 outperforms the model-free agents in all four metrics, with the largest margin in record normalized mean performance. Despite this, we recommend clipped record normalized mean as the most meaningful aggregation method, as it considers all tasks to similar degree without being dominated by small number of outlier scores. In Table 1, we also include DreamerV2 with schedules that anneal the actor entropy loss scale and actor gradient mixing over the course of training, which further increases the gamer median score of DreamerV2.
Individual games The scores on individual Atari games at 200M environment steps are included in Table K1, alongside the model-free algorithms and the basclines of random play, human gamer, and human world record. We filled reasonable values for the 2 out of 55 games that have no registered world record. Figure El compares the score differences between DreamerV2 and each model-free algorithm for the individual games. DreamerV2 achieves comparable or higher performance on most games except for Video Pinball. We hypothesize that the reconstruction loss of the world model does not encourage learning a meaningful latent representation because the most important object in the game, the ball, occupies only single pixel. One the other hand, DreamerV2 achieves the strongest improvements over the model-free agents on the games James Bond, Up N Down, and Assault.
8

---

Published as conference paper at ICLR 2021
Agent
Gamer Median
Gamer Mean
Record Mean
Clipped Record Mean
DreamerV2
1.64
13.39
0.36
0.25
No Layer Norm
1.66
11.29
0.38
0.25
No Reward Gradients
1.68
14.29
0.37
0.24
No Discrete Latents
0.85
3.96
0.24
0.19
No KL Balancing
0.87
4.25
0.19
0.16
No Policy Reinforce
0.72
5.10
0.16
0.15
No Image Gradients
0.05
0.37
0.01
0.01
Table 2: Ablations to DreamerV2 measured by their Atari performance at 200M frames, sorted by the last column. The this experiment uses a slightly earlier version of DreamerV2 compared to Table 1. Each ablation only removes one part of the DreamerV2 agent. Discrete latent variables and KL balancing substantially contribute to the success of DreamerV2. Moreover, the world model relies on image gradients to learn general representations that lead to successful behaviors, even the representations are not specifically learned for predicting past rewards.
3.2 ABLATION STUDY
To understand which ingredients of DreamerV2 are responsible for its success, we conduct an extensive ablation study. We compare equipping the world model with categorical latents, in DreamerV2, to Gaussian latents, as DreamerV1. Moreover, we study the importance of KL balancing. Finally, we investigate the importance of gradients from image reconstruction and reward prediction for learning the model representations, by stopping one of the two gradient signals before entering the model states. The results of the ablation study are summarized in Figure and Table Refer to the appendix for the score curves of the individual tasks.
Categorical latents Categorical latent variables outperform than Gaussian latent variables on 42 tasks, achieve lower performance on 8 tasks, and are tied on 5 tasks. We define tie as being within 5% of another. While we do not know the reason why the categorical variables are beneficial, we state several hypotheses that can investigated in future work:
categorical prior can perfectly fit the aggregate posterior, because a mixture of categoricals is again categorical. In contrast, a Gaussian prior cannot match mixture of Gaussian posteriors, which could make difficult to predict multi-modal changes between one image and the next.
The level of sparsity enforced by vector of categorical latent variables could be beneficial for generalization. Flattening the sample from the 32 categorical with 32 classes each results in sparse binary vector of length 1024 with 32 active bits.
Despite common intuition, categorical variables may be easier to optimize than Gaussian variables, possibly because the traight-through gradient estimator ignores a term that would otherwise scale the gradient. This could reduce exploding and vanishing gradients.
Categorical variables could be a better inductive bias than unimodal continuous latent variables for modeling the non-smooth aspects of Atari games, such as when entering new room, when collected items or defeated enemies disappear from the image.
KL balancing KL balancing outperforms the standard KL regularizer on 44 tasks, achieves lower performance on 6 tasks, and is tied on tasks. Learning accurate prior dynamics of the world model is critical because it is used for imagining latent state trajectories using policy optimization. By scaling up the prior cross entropy relative the posterior entropy, the world model encouraged to minimize the KI. by improving its prior dynamics toward the more informed posteriors, opposed to reducing the KL by increasing the posterior entropy. KI. balancing may also be beneficial for probabilistic models with learned priors beyond world models.
Model gradients Stopping the image gradients increases performance on 3 tasks, decreases perfor- mance on 51 tasks, and tied on task. The world model of Dreamer V2 thus heavily relies on the learning signal provided by the nigh-dimensiioal images. Stopping the reward gradients increases performance on 15 tasks, decreases performance on 22 tasks, and is tied on 18 tasks. Figure HI further shows that the difference in scores is small. In contrast to MuZero, DreamerV2 thus learns general representations of the environment state from image information alone. Stopping reward gradients improved performance number of tasks, suggesting that the representations that are not specific to previously experienced rewards may generalize better to unseen situations.

---

Published as conference paper at ICLR 2021
Reward Modeling
Image Modeling
Latent Transitions
Single GPU
Trainable Parameters
Atari Frames
Accelcrator
Algorithm
Days
DreamerV2
22M
200M
SimPLe
/
74M
4M
40
MuZero
X
x
40M
20B
80
MuZero Reanalyze
X
X 40M
200M
80
Table 3: Conceptual comparison of recent RL algorithms that leverage planning with a learned model. DreamerV2 and SimPLe learn complete models of the environment by leveraging the learning signal provided by the image inputs, while MuZero learns its model through value gradients that are specific an individual task. The Monte-Carlo tree search used by MuZero is effective but adds complexity and is challenging to parallelize. This component is orthogonal to the world model proposed here.
Policy gradients Using only Reinforce gradients to optimize the policy increases performance on 18 tasks, decreases performance on 24 tasks, and is tied on 13 tasks. This shows that DreamerV2 relies mostly on Reinforce gradients to learn the policy. However, mixing Reinforce and straight-through gradients yields substantial improvement on James Bond and Seaquest, leading to a higher gamer normalized task mean score. Using only arraigh-tthrough gradients to optimize the policy increases performance on 5 tasks, decreases performance on 44 tasks, and is tied on 6 tasks. We conjecture that straight-through gradients alone are not well suited for policy optimization because of their bias.
4 RELATED WORK
Model-free Atari The majority of agents applied to the Atari benchmark have been trained using model-free algorithms. DQN (Mnih et al., 2015) showed that deep neural network policies can be trained using Q-learning by incorporating experience replay and target networks. Several works have extended DQN to incorporate bias correction as in DDQN (Van Hasselt et al., 2015), prioritized experience replay (Schaul et al., 2015), architectural improvements (Wang et al., 2016), and distri- butional value learning (Bellemare al., 2017; Dabney al., 2017: 2018). Besides value learning, agents based on policy gradients have targeted the Atari benchmark, such as ACER (Schulman et al., 2017a), PPO (Schulman al., 2017a). ACKTR (Wu al., 2017), and Reactor (Gruslys et al., 2017). Another line of work has focused on improving performance by distributing data collection, often while increasing the budget of environment steps beyond 200M (Mnih et al., 2016; Schulman et al., 2017b; Horgan et al., 2018; Kapturowski et al., 2018; Badia et al., 2020).
World models Several model- -ased agents focus on proprioceptive inputs (Watter et al., 2015; Gal al., 2016; Higuera et al., 2018: Henaff et al., 2018; Chua al., 2018; Wang al., 2019; Wang and Ba, 2019), model images without using them for planning (Oh et al., 2015; Krishnan et al., 2015; Karl et al., 2016; Chiappa et al., 2017: Babaeizadeh al., 2017: Gemici et al., 2017; Denton and Fergus, 2018; Buesing et al., 2018; Doemr et al., 2018: Gregor and Besse, 2018), or combine the benefits of model-based and model-free approaches (Kalweit and Boedecker, 2017: Nagabandi et al., 2017; Weber et al., 2017; Kurutach et al., 2018: Buckman al., 2018; Ha and Schmidhuber, 2018; Wayne et al., 2018; Igl et al., 2018; Srinivas et al., 2018; Lee et al., 2019). Risi and Stanley (2019) optimize discrete latents using evolutionary search. Parmas et al. (2019) combine reinforce and reparameterization gradients. Most world model agents with image inputs have thus far been limited relatively simple control tasks (Watter al., 2015; Ebert et al., 2017; Ha and Schmidhuber, 2018: Hafner et al., 2018; Zhang et al., 2019; Hafner et al., 2019). We explain the two model-base -ased approaches that were applied to Atari in detail below.
SimPLe The SimPLe agent (Kaiser et al., 2019) learns a video prediction model in pixel-space and uses its predictions to train PPO agent (Schulman et al., 2017a), as shown in Table 3. The model directly predicts each frame from the previous four frames and receives an additional discrete latent variable as input. The authors evaluate SimPLe on subset of Atari games for 400k and 2M environment steps, after which they report diminishing returns. Some recent model-free methods have followed the comparison at 400k steps (Srinivas et al., 2020; Kostrikov al., 2020). However, the highest performance achieved in this data-efficient regime a gamer normalized median score of 0.28 (Kostrikov et al., 2020) that is far from human- level performance. Instead, we focus on the well-established and competitive evaluation after 200M frames, where many successful model-free algorithms are available for comparison.
10

---

Published as conference paper at ICLR 2021
MuZero The MuZero agent (Schrittwieser et al., 2019) learns sequenee model of rewards and values (Oh et al., 2017) to solve reinforcement learning tasks via Monte-Carlo Tree Search (MCTS; Coulom, 2006; Silver et al., 2017). The sequence model is trained purely by predicting task-specific information and does not incorporate explicit representation learning using the images, as shown in Table 3. MuZero shows that with significant engineering effort and vast computational budget, planning can achieve impressive performance on several board games and deterministic Atari games. However, MuZero is not publicly available, and it would require over 2 months train an Atari agent on one GPU. By comparison, DreamerV2 is a simple algorithm that achieves human-leve performance on Atari on a single GPU 10 days, making reproducible for many researchers. Moreover, the advanced planning components of MuZero are complementary and could be applied to the accurate world models learned by DreamerV2. DreamerV2 leverages the additional learning signal provided by the input images, analogous to recent successes by semi-supervised image classification (Chen et al., 2020; He et al., 2020; Grill al., 2020).
5 DISCUSSION
We present DreamerV2, model-based agent that achieves human-level performance the Atari 200M benchmark by learning behaviors purely from the latent-space predictions of separately trained world model. Using single GPU and single environment instance, DreamerV2 outperforms top model-free single-GPU agents Rainbow and IQN using the same computational budget and training time. To develop DreamerV2. apply several small modifications to the Dreamer agent (Hafner et al., 2019). We confirm experimentally that learning a categorical latent space and using KL balancing improves the performance of the agent. Moreover, we find the Dreamer V2 relies on image information for learning generally useful representations - its performance is not impacted by whether the representations are especially learned for predicting rewards.
DreamerV2 serves as proof of concept, showing that model- based RL can outperform top model-free algorithms on the most competitive RL benchmarks, despite the years of research and engineering effort that modern model-free agents rest upon. Beyond achieving strong performance on individual tasks, world models open avenues for efficient transfer and multi-task learning, sample-efficient learning on physical robots, and global exploration based on uncertainty estimates.
Acknowledgements We thank our anonymous reviewers for their feedback and Nick Rhinchart for an insightful discussion about the potential benefits of categorical latent variables.

---

Published as conference paper at ICLR 2021
REFERENCES
M Babaeizaach, C Finn, D Erhan, RH Campbell, S Levine. Stochastic Variational Video Prediction. ArXiv Preprint ArXiv:1710.11252, 2017.
AP Badia, Piot, S Kapturowski, P Sprechmann, A Vitvitskyi, D Guo, C Blundell. Agent57: Outperforming the Atari Human Benchmark. ArXiv Preprint /33350 2020.
MG Bellemare, Y Naddaf, Veness, M Bowling. The Arcade Learning Environment: An Evaluation Platform for General Agents. Journal of Artificial Intelligence Research, 47, 2013.
MG Bellemare, W Dabney, R Munos. A Distributional Perspective on Reinforcement Learning.
ArXiv Preprint 170iv:1707.06660 2017.
Bengio, N Léonard, A Courville. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. ArXiv Preprint 3083v:/308.3432, 2013.
Brockman, V Cheung, Pettersson, Schneider, J Schulman, Tang, W Zaremba. Openai Gym, 2016.
J Buckman, D Hafner, G Tucker, E Brevdo, H Lee. Sample-Efficient Reinforcement Learning With Stochastic Ensemble Value Expansion. Advances in Neural Information Processing Systems, 2018.
Buesing. Weber, S Racaniere, Eslami, D Rezende, DP Reichert, Viola, F Besse, K Gregor, D Hassabis, et al. Learning and Querying Fast Generative Models for Reinforcement Learning. ArXiv Preprint 1xXXvv/802..00000 2018.
A Byravan, JT Springenberg, A Abdolmaleki, R Hafner, M Neunert, T Lampe, N Siegel, N Heess, M Riedmiller. Imagined Value Gradients: Model-Basee Policy Optimization With Transferable Latent Dynamics Models. ArXiv Preprint ArXiv:1910.04142 2019.
PS Castro, S Moitra, C Gelada, S Kumar, MG Bellemare. Dopamine: A Research Framework for Deep Reinforcement Learning. ArXiv Preprint ArXiv:1812.061 10, 2018
Chen, S Kornblith, M Norouzi, G Hinton. A Simple Framework for Contrastive Learning of Visual Representations, ArXiv Preprint ArXiv:2002.0577 057099 2020.
Chiappa, S Racaniere, D Wierstra, S Mohamed. Recurrent Environment Simulators. ArXiv Preprint
ArXiv:1704.02254, 2017.
K Cho, B Van Merriënboer, C Gulcehre, D Bahdanau, F Bougares, H Schwenk, Y Bengio. Learning Phrase Representations Using Rnn Snnoder-Deeooeer for Statistical Machine Translation. ArXiv Preprint ArXiv:1406.1078 2014.
K Chua, R Calandra, R McAllister, S Levine. Deep Reinforcement Learning in Handful of Trials Using Probabilistic Dynamics Models. Advances in Neural Information Processing Systems, 2018.
DA Clevert, T Unterthiner, S Hochreiter. Fast and Accurate Deep Network Learning by Exponential Linear Units (Elus). ArXiv Preprint 17.029,5,,.07289, 2015.
Coulom. Efficient Selectivity and Backup Operators in Monte-Carle Tree Search, International Conference Computers and Games. Springer, 2006.
W Dabney, M Rowland, MG Bellemare, R Munos. Distributional Reinforeement Learning With Quantile Regression. ArXiv Preprint ArXiv: 10044, 2017.
W Dabney, G Ostrovski, D Silver, R Munos. Implicit Quantile Networks for Distributional Reinforcement Learning, ArXiv Preprint ArXiv:1806.06923: 2018.
Denton Fergus. Stochastic Video Generation With Learned Prior. ArXiv Preprint
ArXiv: 10XI::1802.0687 2018.
A Doerr, C Daniel, M Schiegg, D Nguyen-Tuong, S Schaal, M Toussaint, S Trimpe. Probabilistic Recurrent SSace Models. ArXiv Preprint ArXiv:1801.10395, 2018.
12

---

Published as conference paper at ICLR 2021
F Ebert, Ciimn, AX Lee, S Levine. Self-Supervised Visual Planning With Temporal Skip Connections. ArXiv Preprint ArX1vv1710.05266, 2017.
M Fortunato, MG Azar, Piot, J Menick, Osband, A Graves, Mnih, R Munos, D Hassabis, O Pietquin, et al. Noisy Networks for Exploration. ArXiv Preprint 1xXi::7706.10290 2017.
Y Gal, R McAllister, CE Rasmussen. Improving Pilco With Bayesian Neural Network Dynamics Models. Data- Aata-Efficient Machine Learning Workshop, ICML, 2016.
M Gemici, CC Hung, A Santoro, G Wayne, Mohamed, DJ Rezende, D Amos, Lillicrap. Generative Temporal Models With Memory. ArXiv Preprint 1Rt2::7702.0400 2017.
K Gregor F Besse. Temporal Difference Variational Auto-Encoder. ArXiv Preprint ArXiv: 18xv:1806.03107, 2018.
JB Grill, F Strub, F Altché, C Tallle,, PH Richemond, E Buchatskaya, C Doersch, BA Pires, ZD Guo, MG Azar, al. Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning. ArXiv Preprint ArXiv:2006.07733, 2020.
A Gruslys, W Dabney, MG Azar, B Piot, M Bellemare, R Munos. The Reactor: A Fast and Sample- Efficient Actor-Critic Agent for Reinforcement Learning. ArXiv Preprint ArXiv:1704.00 04651, 2017.
D Ha J Schmidhuber. World Models. ArXiv Preprint 100.vv:v003//1222 2018.
D Hafner, T Lillicrap, Fischer, R Villegas, D Ha, H Lee. Davidson. Learning Latent Dynamics for Planning From Pixels. ArXiv Preprint ArXiv: /8/ 0,455,, 2018.
Hafner, Lillicrap, J Ba, M Norouzi. Dream to Control: Learning Behaviors by Latent Imagination. 2019.
ArXiv Preprint 1912
K He, H Fan, Y Wu, S Xie, R Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
M Henaff, WF Whitney, Y LeCun. Model-Based Planning With Discrete and Continuous Actions. ArXiv Preprint ArXiv: 705.077177 2018.
M Hessel, J Modayil, H Van Hasselt, T Schaul, G Ostrovski, W Dabney, D Horgan, Piot, Azar, D Silver. Rainbow: Combining Improvements in Deep Reinforcement Learning. Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
I Higgins, L Matthey, A Pal, C Burgess, X Glorot, M Botvinick, S Mohamed, A Lerchner. Beta- Vae: Learning Basic Visual Concepts With a Constrained Variational Framework. International Conference on Learning Representations, 2016.
JCG Higuera, D Meger, G Dudek. Synthesizing Neural Network Controllers With Probabilistic Model Based Reinforcement Learning ArXiv Preprint
2018.
Horgan, J Quan, D Budden, G Barth-Maron, M Hessel, H Van Hasselt, D Silver. Distributed /80888000000333 2018.
Prioritized Experience Replay. ArXiv Preprint
M Igl, L Zintgraf. TA Le, F Wood, S Whiteson. Deep Variational Reinforcement Learning for Pomdps. ArXiv Preprint ArXiv: 1806. 024602426, 2018
Kaiser, M Babaeizadeh, Milos, B Osinski, RH Campbell, Czechowski, D Erhan, C Finn, P Kozakowski, S Levine, et al. Model-Based Reinforcement Learning for Atari. ArXiv Preprint 19Xiv:/033.00 2019.
Kalweit Boedecker. Uncertainty-Driven Imagination for Continuous Deep Reinforcement Learning. Conference Robot Learning, 2017.
S Kapturowski, G Ostrovski, J Quan, R Munos, W Dabney. Recurrent Experience Replay in Distributed Reinforcement Learning. International Conference on Learning Representations, 2018.
13

---

Published as conference paper at ICLR 2021
M Karl, M Soelch, Bayer, P van der Smagt. Deep Variational Bayes Filters: Unsupervised Learning of State Space Models From Raw Data. ArXiv Preprint 1rXiv:1605.064333 2016.
DP Kingma J Ba. Adam: A Method for Stochastic Optimization. ArXiv Preprint ArXiv:1412.6980, 2014.
DP Kingma M Welling. Auto-Encodiing Variational Bayes, ArXiv Preprint ArXiv:1312.6114 2013.
I Kostrikov, D Yarats, R Fergus. Image Augmentation Is All You Need: Regularizing Deep Reinforeement Learning From Pixels. ArXiv Preprint ArXiv:2004.13649 2020.
RG Krishnan, U Shalit, D Sontag. Deep Kalman Filters. ArXiv Preprint
2015.
T Kurutach, I Clavera, Y Duan, A Tamar, P Abbeel. Model-Ensemble Trust-Region Policy Optimization. ArXiv Preprint ArXiv: 10592, 2018.
LeCun, B Boser, JS Denker, Henderson, RE Howard, Hubbard, LD Jackel. Backpropagation Applied to Handwritten Zip Code Recognition. Neural Computation, 1(4), 1989.
AX Lee, A Nagabandi, P Abbeel, Levine. Stochastic Latent Actor-Critic: Deep Reinforcement Learning With a Latent Variable Model. ArXiv Preprint ArXiv: 1907000577, 2019.
MC Machado, MG Bellemare, Talvitie, J Veness, M Hausknecht, M Bowling. Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents.
Journal of Artificial Intelligence Research, 61, 2018.
V Mnih, K Kavukcuoglu, D Silver, AA Rusu, J Veness, MG Bellemare, A Graves, M Riedmiller, AK Fidjeland, G Ostrovski, al. Human-Level Control Through Deep Reinforcement Learning. Nature, 518(7540), 2015.
V Mnih, AP Badia, M Mirza, A Graves, T Lillicrap, T Harley, D Silver, K Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learning. International Conference on Machine Learning, 2016.
A Nagabandi, G Kahn, RS Fearing, S Levine. Neural Network Dynamics for Model-Based Deep Reinforcement Learning With Model-Free Fine-Tuning. ArXiv Preprint ArXiv:/708.025500 2017.
J Oh, X Guo, H Lee, RI. Lewis, S Singh. Action-Conditional Video Prediction Using Deep Networks in Atari Games. Advances in Neural Information Processing Systems, 2015.
J Oh, Singh, Lee. Value Prediction Network. Advances in Neural Information Processing Systems, 2017.
P Parmas, CE Rasmussen, J Peters, K Doya. Pipps: Flexible Model-Based Policy Search Robust to the Curse of Chaos. ArXiv Preprint 1rXiv:190..01244 2019.
Pathak, P Agrawal, AA Efros, T Darrell. Curiosity-Driven Exploration by Self-Supervised Prediction. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, 2017.
DJ Rezende, S Mohamed, D Wierstra. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. ArXiv Preprint 1:Xiv./401.444 2014.
S Risi KO Stanley. Deep Neuroevolution of Recurrent and Discrete World Models. Proceedings of the Genetic and Evolutionary Computation Conference, 2019.
T Schaul, J Quan, I Antonoglou, D Silver. Prioritized Experience Replay. ArXiv Preprint ArXhv:1511.05952 2015.
J Schritwieser, Antonoglou, Hubert, K Simonyan, Sifre, Schmitt, A Guez, E Lockhart, D Hassabis, T Graepel, et al. Mastering Atari, Go, Chess and Shogi by Planning With Learned Model. ArXiv Preprint ArXiv: 2019.
J Schulman, P Moritz, Levine, M Jordan, Abbeel. High-Dimensional Continuous Control Using Generalized Advantage Estimation. ArXiv Preprint 1006.055500024:: 2015.
14

---

Published as conference paper at ICLR 2021
J Schulman, F Wolski, P Dhariwal, A Radford, O Klimov. Proximal Policy Optimization Algorithms. ArXiv Preprint ArX1v:1707.06347, 2017a.
J Schulman, Wolski, P Dhariwal, A Radford, O Klimov, Proximal Policy Optimization Algorithms.
ArXiv Preprint ArXiv:/707.06347, 2017b.
Sekar, O Rybkin, K Daniilidis, P Abbeel, Hafner, D Pathak. Planning to Explore via Self- Supervised World Models. ArXiv Preprint ArXiv:2005.0596 2020.
D Silver, J Schrittwieser, K Simonyan, Antonoglou, A Huang, A Guez, T Hubert, L Baker, M Lai, A Bolton, et al. Mastering the Game of Go Without Human Knowledge. Nature, 550(7676), 2017.
A Srinivas, A Jabri, P Abbeel, S Levine, C Finn. Universal Planning Networks. ArXiv Preprint UrXiv:1804.00645, 2018.
A Srinivas, M Laskin, P Abbeel. Curl: Contrastive Unsupervised Representations for Reinforcement Learning. ArXiv Preprint ArXiv:2004.04 2020.
RS Sutton. Dyna, an Integrated Architecture for Learning, Planning, and Reacting. ACM SIGART Bulletin, 2(4), 1991.
RS Sutton AG Barto. Reinforcement Learning:
An Introduction. MIT press, 2018.
AA Taiga, W Fedus, MC Machado, Courville, MG Bellemare. On Bonus Based Exploration Methods in the Arcade Learning Environment. International Conference on Learning Representations, 2019.
M Toromanoff, E Wirbel, F Moutarde. Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the Playing Field. ArXiv Preprint ArXiv: 1908.0633,, 2019.
Van Hasselt, A Guez, Silver. Deep Reinforcement Learning With Double Q-Learning ArXiv Preprint ArXiv:1509.06400 2015.
Wang J Ba. Exploring Model-Based Planning With Policy Networks. ArXiv Preprint ArXiv: 11906608649 2019.
T Wang, X Bao, I Clavera, J Hoang, Y Wen, E Langlois, S Zhang, G Zhang, P Abbeel, J Ba. Benchmarking Model-Based Reinforcement Learning. CoRR, abss/1907.020571 2019.
Z. Wang, T Schauu,, M Hessel, H Hasselt, M Lanetot, N Freitas. Dueling Network Architectures for Deep Reinforcement Learning. International Conference on Machine Learning, 2016.
M Watter, Springenberg. Boedecker, M Riedmiller. Embed to Control: A Locally Linear Latent Dynamics Model for Control From Raw Images. Advances in Neural Information Processing Systems, 2015.
G Wayne, CC Hung, D Amos, M Mirza, Ahuja, A Grabska-Barwinska. J Rae, P Mirowski, JZ Leibo, A Santoro, et al. Unsupervised Predictive Memory in a Goal-Directed Agent. ArXiv Preprint ArXiv: 10760, 2018.
T Weber, S Racanière, DP Reichert, L Buesing, A Guez, DJ Rezende, AP Badia, O Vinyals, N Heess, Li, et al. deseiiattio-AAummenn Agents for Deep Reinforcement Learning. ArXiy Preprint ArXh::1007.0620: 2017.
RJ Williams. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning, 8(3-4), 1992.
Y Wu, E Mansimov, RB Grosse, S Liao, J Ba. Scalable Trust-Region Method for Deep Reinforeement Learning Using Kronecker-Factored Approximation. Advances in Neural Information Processing Systems, 2017.
T Yu, G Thomas, L Yu, S Ermon, J Zou, S Levine, C Finn, T Ma. Mopo: Model-Based Offline Policy Optimization. ArXiv Preprint ArXiv:2005.132 13239, 2020.
M Zhang, S Vikram, L Smith, P Abbeel, M Johnson, S Levine. Solar: Deep Structured Representations for Model-Based Reinforcement Learning. International Conference on Machine Learning, 2019.
15

---

Published as conference paper at ICLR 2021
A HUMANOID FROM PIXELS
Behavior learned by DreamerV2 on the Humanoid Walk task from pixel inputs only. The task is provided by the DeepMind Control Suite and uses continuous action space with 21 dimensions. The frames show the agent inputs.
Figure Al:
While the main experiments of this paper focus on the Atari benchmark with discrete actions, Dream- erV2 is also applicable to control tasks with con- tinuous actions. For this, we the actor outputs truncated normal distribution instead of categor- ical distribution. demonstrate the abilities of DreamerV2 for continuous control, we choose the challenging humanoid environment with only im- age inputs, shown in Figure A1. We find that for continuous control tasks, dynamics backpropagation substantially outperforms reinforce gradients and thus set = 0. We also set 7 10-5 and B 2 and leave all other hyper parameters at their defaults. We find that DreamerV2 reliably solves both the stand-up motion required the beginning of the episode and the subsequent walking. The score is shown in Figure A2. To the best of our knowledge, this constitutes the first published result of solving the humanoid environment from only pixel inputs.
Humanoid Walk
800
600
400
200
DreamerV2
0
1e7
Figure A2: Performance on the humanoid walking task from only pixel inputs.
16

---

Published as conference paper at ICLR 2021
B MONTEZUMA'S REVENGE
Figure Bl: Behavior learned by DreamerV2 on the Atari game Montezuma's 's Revenge, that poses hard exploration challenge. Without any explicit exploration mechanism, DreamerV2 reaches about the same performance as the exploration method ICM.
While our main experiments use the same hyper pa- rameters across all tasks, find that DreamerV2
Montezuma Revenge
3000
achieves higher performance on Montezuma's 's Re- venge by using lower discount factor of y 0.99, possibly to stabilize value learning under sparse rewards. Figure B2 shows the resulting perfor- mance, with all other hyper parameters left their defaults. Dreamer V2 outperforms existing model- free approaches on the hard-exploratiin game Mon- tezuma's 's Revenge and matches the performance of the explicit exploration algorithm ICM (Pathak et al., 2017) that was applied on top of Rainbow by Taiga al. (2019). This suggests that the world model may help with solving sparse reward tasks, for example due to improved generalization, efficient policy opti- mization in the compact latent space enabling more actor critic updates, or because the reward predictor generalizes and thus smooths out the sparse rewards.
2000
1000
0
0.0
0.5
1.0
1.5
2.0
1e8
DreamerV2 (y=0.99)
IQN
Rainbow + Curiosity
C51
Rainbow
DQN
Figure B2: Performance on the Atari game Montezuma's Revenge.
17

---

Published as conference paper at ICLR 2021
C SUMMARY OF MODIFICATIONS
To develop DreamerV2, used the Dreamer agent (Hafner et al., 2019) starting point. This subsection describes the changes that we applied to the agent to achieve high performance on the Atari benchmark, well the changes that were tried but not found to increase performance and thus were not not included DreamerV2.
Summary of changes that were tried and were found to help:
Categorical latents Using categorical latent states using straight-through gradients in the world model instead of Gaussian latents with reparameterized gradients.
Separately scaling the prior cross entropy and the posterior entropy in the KL loss to encourage learning an accurate temporal prior, instead of using free nats.
KL balancing
Reinforce only
Reinforce gradients worked substantially better for Atari than dynamics back- propagation. For continuous control, dynamics backpropagation worked substantially better.
Model size Increasing the number of units or feature maps per layer of all model components, resulting in a change from 13M parameters to 22M parameters.
Policy entropy Regularizing the policy entropy for exploration both in imagination and during data collection, instead of using external action noise during data collection.
Summary of changes that were tried but were found
help substantially:
Binary latents
Using larger number of binary latents for the world model instead of categor- ical latents, which could have encouraged more disentangled representation, was worse.
Long-term entropy Including the policy entropy into lemporal-diffrenne loss of the value function, so that the actor seeks out states with high action entropy beyond the planning horizon.
Mixed actor gradients Combining Reinforce and dynamics backpropagation gradients for learning the actor instead of Reinforce provided marginal or no benefits.
Scheduling Scheduling the learning rates, KL scale, actor entropy loss scale, and actor gradient mixing (from 0.1 to 0) provided marginal or no benefits.
Layer norm Using layer normalization in the GRU that is used part of the RSSM latent transition model, instead of no normalization, provided no or marginal benefits.
Due the large computational requirements, comprehensive ablation study this list of all changes is unfortunately infeasible for us. This would require 55 tasks times 5 seeds for 10 days per change run, resulting over 60,000 GPU hours per change. However, we include ablations for the most important design choices in the main text of the paper.
18

---

Published as conference paper at ICLR 2021
D HYPER PARAMETERS
Name
Symbol
Value
World Model
Dataset size (FIFO)
2 106
Batch size
B
50
Sequence length
50
Discrete latent dimensions
32
Discrete latent classes
32
RSSM number of units
600
KI. loss scale
8
0.1
KL balancing
a
0.8
World model learning rate
2 10-4
Reward transformation
tanh
Behavior
Imagination horizon
H
15
Discount
7
0.995
X-target parameter
0.95
Actor gradient mixing
Actor entropy loss scale
7
1- 10
Actor learning rate
4 4-10-5
Critic learning rate
1-10
Slow critic update interval
100
Common
Environment steps per update
MPL number of layers
MPL number of units
400
Gradient clipping
100
Adam epsilon
10 10-5
Weight decay (decoupled)
10
Table DI: Atari hyper parameters of DreamerV2. When tuning the agent for new task, we recommend searching the KI. loss scale 3 {0.1, 0.3. 1,3), actor entropy loss scale rj {3. 10-5 10-1,3 3. 10-4, 10-3), and the discount factor y E (0.99, 0.999). The training frequency update should be increased when aiming for higher data-efficiency.
19

---

Published as conference paper at ICLR 2021
E AGENT COMPARISON
DreamerV2 vs IQN
10
0
-10
-100
James Bond
Up N Down
Krull
Gopher
Deman Attack
Ascault
Road Runner
Time Pilot
Broakout
Asteni
Phoenix
Atlantis
Zaxonn
Ice Hockey
Wizarr Of Wor
Name Gama
Kung Mastor
Robatank tank
Grazy CrazyCllmt Climber
Asteroids
Frostijve
Centpedee
Gravitar
Beam Rider
Kangaroo
Fishing Derby
Skiing
Arridar
Bank Helst
Tuhanicam
Riverraid
Bowling
Pacman
Berzerk
Pittall
Freeway
Pong
Battle Zone
Solaris
Montezumy Rev.
Allen
Seaquest
Hero
Tennis
Enduro
Chopper Com.
Venture
Space Dunk
Star Gunner
video Pinball
Private
Yars
DreamerV2 vs Rainbow
100
10
0
-100
Up Down
Krail
Assault
Copher
Demon Attack
Road Runner
Time Pilot
Atlantts
Breakout
Asterix
Phocenl
Qbert
Zaxxon
Ice Hackey
Yars Revenge
Robotank
Wiaad Of War
Name This Game
Tennie
Asteroids
Beam Rider Gravitar
Frost ite
Crazy Climber
Cantipade
Fiching Dorby
Ms Pacman
Turanich am
Alien
Bank Heist
Amidar
Bettle Zone
Pitrall
Ereway
Berzerk
Seaquest
Solars
Montezuma Rev.
Private Eye
Riverraid
Boxing
Enduro
Herr
Space Invaders
Venture
Star Gunner
Video Pinball
Deuble
Kung
DreamerV2 C51
100
10
-10
-100
Up Down
Assault
Deman Attack
Kruil
Gopher
Rurner
Time Pllct
stlantis
Dauble Dunk
Asterix
Phoenix
Qbort
Zaxxan
Broakout
Yars Revenge
lce Hockey
Wor
Kangaroo
Robotank
Kung Ru Master
Frastbite
Climber
Derby
Skiing
Gravitar
Boxing
Asteroids
Amidar
Bank Helst
Enduro
Centipede
Battle Zone
Game
Pacman
Alien
Riveirald
Tutankham
Bowling
Berzerk
Pong
Pitaali
Freeway
Private Eye
Montezuma Rev.
Hero
Chapper Cam-.
Tenili
Seaquest
Invaders
Venture
Star Gunner
video Pinball
James
DreamerV2 DQN
100
10
0
-10
-100
James Bond
Up N Down
Assault
Demon Attack
Kruil
Copter
Runner
Tiiee Pilok
Double Dunk
Asterix
Allambs
preakkut
Phoenix
Qbert
Zaxzon
Ice Hackey
Frostiite
Revenge
Wizard Wor
Crazy Climber
Robatank
Kung Fu Mastar
Name Fisting Derpy
Enduro
Boxing
Gravitar
Tutankhamm
Tennis
Contipede
Asteroles
Kangaroo
Amildar
Beam Rider
Bottie Zone
Skiing
Space invaders
Me Pacman
Riverraid
Freeway
Alion
Hero
Seaquest
Rowling
Berzeric
Pong
Privae Eye
Chopper Com.
Montezuma Rev.
Pitzall
Venture
star Gunner
video Pinball
Figure E1: Atari agent comparison. The bars show the difference in gamer normalized scores at 200M steps. DreamerV2 outperforms the four model-free algorithms IQN, Rainbow, C51, and DQN while learning behaviors purely by planning within separately learned world model. DreamerV2 achieves higher or similar performance on all tasks besides Video Pinball, where we hypothesize that the reconstruction loss does not focus the ball that makes up only pixel the screen.
20

---

Published as conference paper at ICLR 2021
F MODEL-FREE COMPARISON
Alien
Amidar
Assault
Asterix
Asteroids
1 antis
s0c0
40000
120000
1.00
60c0
3000
30000
90000
4000
20000
60000
40000
0.50
2000
1000
0.25
40000
0.0 05 1.0
1.5 2.0
0.0 0.5 1.0 1.5 2.0
0.0
0.5 1.0 1.5
0.0
0.5
1.0
1,5 2.0
0.0
0.5
1.0
1.5 2.0
0.0
1.0 15 2.5
Bank Heist
Battle Zone
Beam Rider
Berzerk
Bowling
Boxing
1200
900
15000
830
60
600
30000
10000
600
15000
200
1.5
15
1.
Breakout
Centipede
Chopper Com.
Crazy Climber
Demon Attack
Double Dunk
300
15000
12000
9000
12000
9000
220000
80000
160000
120000
aoc00
150
3000
3000
40000
-15
0.0 0. 1.0 1.0
0.0.5 1.0 155 2.0
0.0 0.5 1.5
0.0
0.5
1.0
1:5 2.0
0.0 0.5
10
1:5 2.0
0.0 0.5 1.0 15 2.0
Enduro
Fishing Derby
Freeway
Frostbite
Gopher
Gravitar
2400
40
30
24000
18000
90000
60000
-40
6000
30000
6.5 1.0 1.5
0.3 0.6.5 1.5 2.0
0.0 0.5 1.5
0.0
0.5 1.0
1.5 2.0
0.0 0.5 1.0 1.5 2.0
0.0 0.5 1.5 15 2.0
Hero
Ice Hockey
James Bond
Kangaroo
Krull
Kung Fu Master
60000
16000
100000
80000
12000
sonto
50000
40000
15000
15000
-15
1.0
Montezuma Rev.
Ms Pacman
Name This Game
Phoenix
Pitfall
Pong
8000
16000
80000
6000
12000
60000
880
500
6000
8000
2000
20000
-10
-20
0.0 05 1.0 1.5 2.0
0.0
05 1.0
135 2.0
0.0 0.5
1.0 1.5
0.0 0.5 1.0
1.5 2.0
0.0
0.5
1.0
1.5 2.0
0.0
0.5
1.0 2.5
Private Eye
Qbert
Riverraid
Road Runner
Robotank
Seaquest
3000
300000
20000
450000
60
24000
20000
100000
10000
40
10000
6000
100000
1.5
15
1.5
0.5 1.0
10 1.5
1.0
Skiing
Solaris
Space Invaders
Star Gunner
Tennis
Time Pilot
-17000
2400
6000
1600
4000
30000
-2400
800
15000
0.0 1.0
0.0 1.0 1.5 200
0.0 05 1.0 1.2
0.0 0.5 1.0 1.5 2.0
00 0.5 10 1.5 2.0
0.0 0.5 1.0 15 2.0
Tutankham
Up N Down
Venture
Video Pinball
Wizard Of Wor
Yars Revenge
600000
1500
600000
200000
300000
1000
500
300000
18000
12000
150000
150000
s000
50000
0.0 05 1.5
0. 1.0 1.515. 2.0
0.0
05
1.0
0.0
0.5 1.0 2.0
0.0 0.5 1.0 1.5 2.0
0.0 0.5 1.0 155 2.0
Zaxxon
Gamer Median
Gamer Mean
Record Mean
Clip Record Mean
80000
2.4
0.45
45000
0.30
30000
0.15
15000
0.0
15
Rainbow Figure F1: Comparison of DreamerV2 to the top model-free RL methods IQN and Rainbow. The curves show mean and standard deviation over seeds. IQN and Rainbow additionally average each point over 10 evaluation episodes, explaining the smoother curves. DreamerV2 outperforms IQN and Rainbow in all four aggregated scores. While IQN and Rainbow tend to succeed on the same tasks, DreamerV V2 shows a different performance profile.
DreamerV2
IQN
21

---

Published as conference paper at ICLR 2021
G LATENTS AND KL BALANCING ABLATIONS
Alien
Amidar
Assault
Asterix
Asteroids
Atlantis
1200
12000
32000
36000
1000000
2400
3000
14000
acon
500000
800
3000
8000
2500000
% 05 1.5 2.0
0.0
0.5 1.0 1.5 2.0
0.0 0. 1.0 1.5 2.0
0.0
0.5 1.0
1.5 2.0
0.0
0.5 1.0 1.5 2.0
0.0
0.5
1.0
15 2.5
Bank Heist
Battle Zone
Beam Rider
Berzerk
100 Bowling
Boxing
40000
12000
800
800
8000
630
60
400
4000
400
40
3:5
0.6
150
200
-30
Breakout
Centipede
Chopper Com.
Crazy Climber
Demon Attack
Double Dunk
450
10000
7500
4500
120000
160000
12000
20
5000
8000
150
2500
1500
a
-0000
-20
0.0 05 1.0 1.200
0.0.5 1.0 1.5 2.0
0.0
0.5 1.5
0.0
0.5
1.0
2:5 2.0
0.0
0.5
10
1:5
2.0
0.0
0.5
1.0
15 2.0
Enduro
Fishing Derby
Freeway
Frostbite
Gopher
Gravitar
1800
80000
75000
3000
800
-40
16-
40000
20000
50000
25000
1500
0.0 6.5 1.0 1.5
0.3 0.055
05
1.5
0.0
0.5 1.0 1.5 2.0
0.0 0.5 1.0 1.5 2.0
0.0 0.5 1.5 3.5 2.0
Hero
Ice Hockey
James Bond
Kangaroo
Krull
Kung Fu Master
24000
12000
30
12000
18000
12000
20000
75000
16000
6000
4000
6000
50000
25000
-15
1.0
Montezuma Rev.
Ms Pacman
Name This Game
Phoenix
Pitfall
Pong
3200
16000
20
2400
6000
30000
-80
3000
160
4000
-240
-20
0.0
05 1. 15
0.0 005110 1.0
135 2.0
0.0 D5 1.0
1.5
0.0 D.5 1.0 1.5 2.0
0.0
0.5
1.0
1.5 2.0
0.0
0.5
1.0
15 2.5
Private Eye
Qbert
Riverraid
Road Runner
Robotank
Seaquest
3000
4500
450000
150000
300000
1200e
160000
120000
RonD0
160000
80000
4000
40000
1.5
1.5
0.5.
1.0
1.
Skiing
Solaris
Space Invaders
Star Gunner
Tennis
Time Pilot
-6000
6000
4000
45000
-17000
4500
3000
3000
-24000
1000
10000
15000
-30000
0.0
1.5.0
0.0.5 1.0
0.0 05 1.0 1.2
0.0 0.5 1.0 1.5 2.0
00 0.5 10
1.5
2.0
0.0 0.5 1.0
15 2.0
Tutankham
Up N Down
Venture
Video Pinball
Wizard Of Wor
Yars Revenge
80000
600000
240
450000
18000
60000
180
30000o
12000
40000
150000
s000
20000
0.0
2.0
0.0
1.0
0.0 05 1.5
0.0
0.5
1.0 15.2.
0.0 0.5 1.0
0.0 0.5 1.0 155 2.0
Zaxxon
Gamer Median
Gamer Mean
Record Mean
Clip Record Mean
20000
0.24
15000
15
0.3
10000
1.0
0.12
0.0
0.0
0.00
1.0
No KL Balance Figure G1: Comparison of DreamerV2 Gaussian instead of categorical latent variables, and no KI. balancing. The ablation experiments use a slightly carlier version of the agent. The curves show mean and standard deviation across two seeds. Categorical latent variables and KL balancing both substantially improve performance across many of the tasks. The importance of the two techniques is reflected in all four aggregated scores.
Dreamerv2
Gaussian Latents.
22

---

Published as conference paper at ICLR 2021
H REPRESENTATION LEARNING A ABLATIONS
Alien
Amidar
Assault
Asterix
Asteroids
Atlantis
5000
1200
12000
6000
1000000
2400
750000
40co
3000
500000
800
250000
0.0 0. 1.0 1.5 2.0
0.0 0.5 1.0 1.5 2.0
0.0
0. 1.0 1.5 2.0
0.0
0.5
1.0
1.5 2.0
0.0
0.5
1.0 1.5 2.0
0.0.0.3 1.0
15 2.5
Bank Heist
Battle Zone
Beam Rider
Berzerk
Bowling
Boxing
1200
40000
900
12000
m00
50
600
20000
8000
40
300
10000
4000
400
20
-50
-100
1.5
0.0
0.0
Breakout
Centipede
Chopper Com.
Crazy Climber
Demon Attack
Double Dunk
10000
160000
18000
8000
6000
120000
12000
300
4000
BODDO
150
4000
2000
a0000
-15
-0000
0.0 05 1.0 1.2
0.0.5 1.0 155 2.0
0.0 0.
1.5
0.0
0.5
1.0
1:5 2.0
0.0
0.5
10
1:5 2.0
0.0 0.0 2.0
Enduro
Fishing Derby
Freeway
Frostbite
Gopher
Gravitar
1800
24000
75000
3000
50000
600
-40
8000
25000
1500
0.0 6. 1.0 1.5
0.3 0.655 2.0
0.0 05 1.0 1.5
0.0
0.5 1.0 1.5 2.0
0.0 0.5 1.0 1.5 2.0
0.0 0.5 1.0 155 2.0
Hero
Ice Hockey
James Bond
Kangaroo
Krull
Kung Fu Master
12000
20
18000
20000
24000
12000
7500
75000
16000
-15
6000
4000
2500
5000
25000
50000
1.0
Montezuma Rev.
Ms Pacman
Name This Game
Phoenix
Pitfall
Pong
3200
20
2400
6000
12000
30000
250
40c0
8000
100
2000
-600
-20
0.0 05 1.0 15
0.0
05 1.0 13 2.0
0.0
D5 7.0
1.5
0.0 0.5 1.0 1.5 2.0
0.0
0.5
1.0
1.5
2.0
0.0
1.5 2.0
Private Eye
Qbert
Riverraid
Road Runner
Robotank
Seaquest
20000
320000
2000
15000
240000
160000
1000
80000
150000
a0000
20
-1000
2.0
1.5
1:5
0.5
1.0
1.
Skiing
Solaris
Space Invaders
Star Gunner
Tennis
Time Pilot
-12000
-6000
6000
4000
45000
4000
2000
20000
30000
-24000
2000
1000
10000
-15
15000
0.0 05 1.0 1.500
0.0 1.0 155 200
0.0
05
1.0
1.5 2,0
0.0
0.5 1.0
1.5 2.0
00 0.5 10 1.5 2.0
0.0 0.5 1.0
15 2.0
Tutankham
Up N Down
Venture
Video Pinball
Wizard Of Wor
Yars Revenge
240
320
600000
450000
80000
18000
75000
160
40000
50000
150000
20030
6000
25000
0.0 05 1.0 1.5
0305 1.0 2.0
0.0 05 2.0.1. 2.5
0.0
1.0 1.0 15.2. 2.0
0.0 05 1.0
1.5 2.0
0.0 0.5 1.0 1.555.55 20
Zaxxon
Gamer Median
Gamer Mean
Record Mean
Clip Record Mean
20000
2.0
0.26
15000
0.3
0.16
10000
1.0
0.1
0.08
0.0
0.00
0.0.5
No Image Gradients Figure HI: Comparison of leveraging image prediction, reward prediction, or both for learning the model representations. While image gradients are crucial, reward gradients are not necessary for world model to succeed and their gradients can be stopped. Representations learned purely from images are not biased toward previously encountered rewards and outperform eeward-specffic representations number of tasks, suggesting that they may generalize better to unseen situations.
DreamerV2
No Reward Gradients
23

---

Published as conference paper at ICLR 2021
POLICY LEARNING A ABLATIONS
Alien
Amidar
Assault
Asterix
Asteroids
Atlantis
10000
1200
12000
32000
6000
1000000
7500
2400
24000
750000
5000
3000
2500
800
3000
8000
250000
0.0 0. 1.0
1.5 2.0
0.0 0.5 1.0 155 2.0
0.0 0. 1.0 1.5 2.0
0.0
0.5
1.0 1.5 2.0
0.0
0.5 1.0 1,5 2.0
0.0
0.5
1.0
15
2.5
Bank Heist
Battle Zone
Beam Rider
Berzerk
Bowling
Boxing
1200
900
48000
16000
12000
1250
1000
90
600
20000
mnno
750
300
10000
250
30
3.5
1.5
0.5
Breakout
Centipede
Chopper Com.
Crazy Climber
Demon Attack
Double Dunk
450
10000
6000
160000
7500
4500
5000
3000
80030
150
2500
1500
40000
-20
0.0 05 1.0 1.0
0. 1.0 1:5 2.0
0.0 0.5 1.5
0.0
0.5
1.0
1:5 2.0
0.0
0.5
10
1:5 2.0
0.0
0.5
1.0
15 2.0
Enduro
Fishing Derby
Freeway
Frostbite
Gopher
Gravitar
1800
24000
50000
3000
600
-00
8000
25000
1500
0.0 6.5 1.0 2.0
0.9
2.0
05
1.5
0.0
0.5 1.0 1.5 2.0
0.0 0.5 1.0 1.5 2.0
0.0 0.5 1.5 155 2.0
Hero
Ice Hockey
James Bond
Kangaroo
Krull
Kung Fu Master
24000
16000
12000
8000
30
12000
18000
8000
12000
4000
20000
5000
7500
75000
50000
-15
2500
1.0
1.0
Montezuma Rev.
Ms Pacman
Name This Game
Phoenix
Pitfall
Pong
3200
16000
20
2400
6000
12000
30000
20000
-80
3000
8000
-160
1500
-240
0.0 05 1. 15 2.0
0.0
05 1.0
135 2.0
0.0 D. 1.0 1.5
0.0 0.5 1.0 1.5 2.0
0.0
0.5
1.0
1.5
2.0
0.0
1.0 15 2.5
Private Eye
Qbert
Riverraid
Road Runner
Robotank
Seaquest
500
750
10000
20000
15000
300000
450000
45
160000
80000
5000
-250
2.0
0.5
1.0 1.5
1.
Skiing
Solaris
Space Invaders
Star Gunner
Tennis
Time Pilot
11000
-6000
4500
4500
20000
30n20
45000
30000
-24000
1500
1500
15000
-30000
0.0 05 1.2000
0.0 1.0 20
0.0
05 1.0 1.5 2.0
0.0 0.5 1.0 1.5 2.
00 0.5 10 1.5 2.0
0.0 0.5 1.0
15 2.0
Tutankham
Up N Down
Venture
Video Pinball
Wizard Of Wor
Yars Revenge
600000
60000
80000
240
450000
18000
160
12000
40000
150000
15000
s000
20000
0
0.0
1.0
0.05 1.0
0.0 05 1.5
0.0
0.5
2.0
0.0 05 1.0 2.0
0.0 0.5
1.0 15 2.0
Zaxxon
Gamer Median
Gamer Mean
Record Mean
Clip Record Mean
32000
0.24
24000
1.
0.3
16000
1.0
0.12
s0c0
0.0
0.0
0.00
Figure I1:
Compariseaa plekarpraaananeeeegggegggggggg RRiigfSERSi haapua/rarnggiiiiiioooooaaaaaaaa seettt both for
training the actor. While Reinforce gradients are crucial, ttrighhttthouge gradients are not important for most of the tasks. Nonetheless, combining both gradients yields substantial improvements small number of games, most notably on Seaquest. We conjecture that straight-through gradients have low variance and thus help the agent start learning, whereas Reinforce gradients unbiased and help converging to better solution.
24

---

Published as conference paper at ICLR 2021
ADDITIONAL ABLATIONS
Alien
Amidar
Assault
Asterix
Asteroids
Atlantis
3200
12000
6000
1000000
24000
3000
3000
500000
800
3000
8000
250000
0.0 05 1.0 1.5 2.0
0.0 0.5 1.0 1.5 2.0
0.0 0. 1.0 1.5
0.0
0.5
1.0
1.5 2.0
0.0
0.5 1.0 1.5 2.0
0.0
0.5
1.0
15 2.5
Bank Heist
Battle Zone
Beam Rider
Berzerk
Bowling
Boxing
1200
900
45000
12000
1000
90
8000
750
300
15000
4800
30
30
0
1:5
250
0.6.1.
160
Breakout
Centipede
Chopper Com.
Crazy Climber
Demon Attack
Double Dunk
450
10000
7500
4000
5000
160000
120000
16000
24000
150
5000
2000
80000
8000
40000
0.0 05 1.0 1.2
0.0.5 1.5 2.0
0.0 0. 1.5
0.0
0.5
1.0
1:5 2.0
0.0
0.5
10
1:5
2.0
0.0
0.5
1,0 15 2.0
Enduro
Fishing Derby
Freeway
Frostbite
Gopher
Gravitar
4500
1800
24000
50000
75000
3000
800
-40
8000
25000
0.0 65 1.0 1.5
0.3 0. 2.0
05 1.0 2.5
0.0
0.5 1.0 1.5 2.0
0.0 0.5 1.0 1.5 2.0
0.0 0.5 1.5 155 2.0
Hero
Ice Hockey
ammes Bond
Kangaroo
Krull
Kung Fu Master
40000
32000
120000
10000
30000
12000
50000
16000
8C00
6000
10000
4000
1.0
1.0
Montezuma Rev.
Ms Pacman
Name This Game
Phoenix
Pitfall
Pong
3200
16000
20
2400
6000
30000
-80
3000
-160
4000
-240
-20
0.0 05 1. 15 2.0
0.0 05 1.0 13 2.0
0.0 D. 1.0
1.5
0.0 0.5 1.0 1.5 2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5 1.0
15 2.5
Private Eye
Qbert
Riverraid
Road Runner
Robotank
Seaquest
450000
15000
150000
300000
12000
180000
120000
45
160000
80000
5000
60000
4000
2.5
1.5
1.0
1.0
1.0
Skiing
Solaris
Space Invaders
Star Gunner
Tennis
Time Pilot
-6000
4000
40000
45000
-17000
3000
30000
20000
24000
1000
10000
-15
15000
330000
0.0 05 1.0 1.000
0.0
2.0
0.0
05 1.0 1.5 2.0
0.0 0.5 1.0
1.5 2.0
00 0.5 10 1.5 2.0
0.0 0.5 1.0
15 2.0
Tutankham
Up N Down
Venture
Video Pinball
Wizard Of Wor
Yars Revenge
600000
32000
450000
30000
18000
24000
300000
12000
150000
15000
5000
8000
-20
05 1.5
03 05 1.0 15.24
0.0
05
1.5
0.0
0.5 1.0
0.0 0S 1.0 1.5 2.0
0.0 0.5 1.0 20
Zaxxon
Gamer Median
Gamer Mean
Record Mean
Clip Record Mean
20000
0.26
15000
1.
0.3
0.16
10000
1.0
0.1
0.08
0.0
0.00
0.0
No Layer Norm Figure J1: Comparison of DreamerV2 to version without layer norm in the GRU and to training from experience collected over time by uniform random policy. We find that the benefit of layer norm depends the task at hand, increasing and decreasing performance of tasks. The comparison to random data collection highlights which of the tasks require non-trivial exploration, which help guide future work directed exploration using world models.
DreamerV2
Random Data
roughly equal number
25

---

Published as conference paper at ICLR 2021
K
ATARI TASK SCORES
Baselines Gamer
Algorithms
Task
Random
Record
Rainbow
IQN
Dreamer V2
Alien
229
7128
251916
3457
4961
3967
Amidar
6
1720
104159
2529
2393
2577
Assault
222
742
8647
3229
4885
23625
Asterix
210
8503
1000000
18367
10374
72311
Asteroids
719
47389
10506650
1484
1585
41526
Atlantis
12850
29028
10604840
802548
890214
978778
Bank Heist
753
82058
1075
1052
1126
Battle Zone
2360
37188
801000
40061
40953
40325
Beam Rider
364
16926
999999
6290
7130
18646
Berzerk
124
2630
1057940
833
648
810
Bowling
23
161
300
43
39
49
Boxing
0
12
100
99
98
92
Breakout
2
30
864
120
79
312
Centipede
2091
12017
1301709
6510
3728
11883
Chopper Command
811
7388
999999
12338
9282
2861
Crazy Climber
10780
35829
219900
145389
132738
161839
Demon Attack
152
1971
1556345
17071
15350
82263
Double Dunk
-19
-16
22
22
21
17
Enduro
0
860
9500
2200
2203
1656
Fishing Derby
-92
-39
71
42
45
65
Freeway
0
30
38
34
34
33
Frostbite
65
4335
454830
8208
7812
11384
Gopher
258
2412
355040
10641
12108
92282
Gravitar
173
3351
162850
1272
1347
3789
Hero
1027
30826
1000000
46675
36058
21868
Ice Hockey
-11
36
-5
26
James Bond
7
29
45550
1097
3166
40445
Kangaroo
52
3035
1424600
12748
12602
14064
Krull
1598
2666
104100
4066
8844
50061
Kung Fu Master
258
22736
1000000
26475
31653
62741
Montezuma Revenge
0
4753
1219200
500
500
81
Ms Pacman
307
6952
290090
3861
5218
5652
Name This Game
2292
8049
25220
9026
6639
14649
Phoenix
761
7243
4014440
8545
5102
49375
Pitfall
2229
6464
114000
-20
-13
0
Pong
-21
15
21
20
20
20
Private Eye
25
69571
101800
21334
4181
2198
Obert
164
13455
2400000
17383
16730
94688
Riverraid
1338
17118
1000000
20756
15183
16351
Road Runner
12
7845
2038100
54662
58966
203576
Robotank
2
12
66
66
78
Seaquest
68
42055
999999
9903
17039
7480
Skiing
17798
-4337
-3272
-27708
11162
-9999
Solaris
1236
12327
111420
1583
1684
922
Space Invaders
148
1669
621535
4131
4530
2474
Star Gunner
664
10250
77400
57909
80003
7800
Tennis
-24
-8
21
23
14
Time Pilot
3568
5229
65300
12051
11666
37945
Tutankham
11
168
5384
239
251
264
Up N Down
533
11693
82840
34888
59944
653662
Venture
0
1188
38900
1529
1313
2
Video Pinball
16257
17668
89218328
466895
415833
41860
Wizard Of Wor
564
4756
395300
7879
5671
12851
Yars Revenge
3093
54577
15000105
45542
84144
156748
Zaxxon
32
9173
83700
14603
11023
50699
Table K1: Atari individual scores. We select the 55 games that are common among most papers in the literature. We compare the algorithms DreamerV2. IQN, and Rainbow to the baselines of random actions, DeepMind's human gamer, and the human world record. Algorithm scores are highlighted bold when they fall within 5% of the best algorithm. Note that these scores are already averaged across seeds, whereas any aggregated scores must be computed before averaging across seeds.
26


Reviews:
[{"review_id": "liCjv8y4851", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe focus of this paper is to extend DreamerV1 to perform harder control tasks such as Atari instead of easier control tasks which were demonstrated in DreamerV1. In doing so, this paper proposes a model-based RL approach which is\n- the first Atari agent that achieves human-level performance on the Atari benchmark of 55 tasks by learning behaviors inside a separately trained world model. Human-level performance historically required model-free agents.\n- appears to be the first Atari agent which uses a stochastic recurrent state to model the observations and train the policy purely via latent imagination. Consequently, it is much more efficient in terms of training computation in comparison to previous model-based RL for Atari approaches (e.g. SimPLE). \n- outperforms model-free single-GPU Atari approaches such as Rainbow and IQN which rest on years of research. This suggests great promise for model-based approaches.\n\nNovelty of the problem: \n- Cons: The claims of the paper mainly apply in the context of Atari + single-GPU setup. \n- Pros: That being said, it is an important setup from the perspective of model-based RL.\n\nContributions in terms of analysis: \n- Cons: There could be more analysis about what aspects of Atari games make it different from DreamerV1 tasks, and in this context, why Categorical latents provide a better world model. What are the current limitations of DreamerV2 in terms of Atari? What lessons can we learn about applying it outside Atari / single-GPU setting.\n\nContribution of the experiments: \n- Pros: Experiments provide valuable proof of concept, showing that model-based RL can outperform top model-free algorithms on Atari benchmark despite years of research and engineering effort. 2) DreamerV2 is also computationally efficient. 3) Provides a good analysis of which metric is appropriate for comparing performance i.e. Mean Clipped Record Normalized.\n\nNovelty of the conceptual idea for the solution: \n- Pros: The model is elegant. For Atari, it appears to be the first model-based approach that uses a recurrent state space model (RSSM) for modeling observations and reward in a stochastic framework. This makes it computationally much faster than previous model-based approaches. The closed-form straight-through gradients for training the policy is also a nice characteristic.\n- Cons: Much of this is already proposed in DreamerV1.\n\nNovelty of the low-level implementation of the conceptual idea: \n- Pros: The use of Discrete latents versus Gaussian is interesting. The argument for KL balancing appears convincing.\n- Cons: However other innovations are small and not analyzed in detail. \n\nPresentation:\n- Pros: Well-written\n- Suggestions: Pseudo-code of the training loop may be helpful. Qualitative examples of imagination may be helpful.\n\n---- After Rebuttal ----\nAfter reading other reviews and the rebuttal, I stay at my current score. Given that the paper does not contain much analysis about \"why\", the paper is about an empirical discovery of a mode that is critical in getting good performance. I think this kind of discovery paper is also important to share with the community.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "WyIK2qTc1H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors build on the Dreamer architecture, that is able to learn models of an environment, to build DreamerV2, which learns a model of an environment in latent space. The authors then train their agent in this latent space. DreamerV2 was evaluated on the Atari learning environment and results showed that it was comparable to Rainbow and better, under certain metrics.\n\nI am unclear on the motivation of this paper. As with previous papers on model-based learning for Atari (i.e. Kaiser et. al (2019)), the goal of learning a model has been to reduce the number of environment steps. However, the authors use the same number of environment steps with the only difference being the model is trained in latent space. Training the model in latent space can speed up learning. Is this the main contribution of the paper?\n\nThere is no analysis as to why using a world model for training might lead to better results than training in the real-world if the same number of environment steps are used. What is the authors' perspective on this? Did DreamerV2 use more steps in the world-model environment than in the real-world environment?\n\nGiven that the latent space is trained based on some reconstruction error (instead of only being useful to a value function as with value prediction networks) it is not immediately obvious that the latent space will be a better place to learn a policy. Have the authors tried training their method on the reconstructions? Perhaps this will be slower, but I think it is still relevant since the authors claim that it may be easier to learn a model in latent space: \"Predicting compact representations instead of images can reduce the effect of accumulating errors\"\n\nIn the introduction, the authors say: Several attempts at learning accurate world models of Atari games have been made, without achieving competitive performance (Oh et al., 2015; Chiappa et al., 2017; Kaiser et al., 2019).\" I do not think this is a fair statement because papers such as Kaiser et al., 2019 intentionally use fewer environment steps.\n\n---After rebuttal---\n\nThe authors have partially addressed my concerns, however, I am still not quite sure why their method would be better than SimPLE. The authors' ablation study of removing image gradients does not address my main question about where the performance benefit is coming from. I am assuming that the architecture and hyperparameters that the authors use are different than SimPLE. I think one must instead replace predicting the latent state with the image as SimPLE did to see if that makes a difference in performance. Therefore, I will be keeping my review the same.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2rVyGMguhoK", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors introduce DreamerV2, a modification of the influential Dreamer RL agent (hereafter refered to as DreamerV1). The primary changes from DreamerV1 are a discrete latent space and a modified loss function (and with it, a modified optimization scheme). As in DreamerV1, the agent trains a world model with environment experience, and the policy is learned by \"imagining\" within the learned latent space using the world model to simulate transitions and rewards. They demonstrate superior performance over a variety of successful benchmarks that use similar compute (1 GPU, 1 environment -- e.g. MuZero, which requires vastly more, is not considered) on Atari. Further, they analyze several ways of aggregating Atari scores, and (while their algorithm performs best of those tried in each aggregation), they recommend one aggregation method (along with several other choices made for benchmarking) going forward.\n\nThis is impressive work. The modification over DreamerV1 is simple (simple enough that they can describe important optimization details within the main body of the paper -- great!) and the results are a convincing demonstration of its utility. The methods are detailed and well-described. Further, I think the effort spent on exactly how to benchmark (and in particular, to report scores) is very welcome and useful to communicate (though I'll be interested to hear if others object to portions of the recommendations!).\n\nThe main weakness: I wish we gained a better understanding of why the discrete latent space works well. Interpretability might be difficult (and I don't necessarily consider that bad -- the results speak for themselves) but we are left with a mix of several plausible hypotheses. Perhaps some visualization might be useful. \n\nRegardless, I strongly advocate for acceptance. What they propose is relatively simple (and the paper describes it well) and appears to work well in this setting. I'll be recommending students try it in other settings. Further, the benchmarking discussion is very useful for the community.\n\nOn the use/interpretation of the discrete latent, a question for the authors: do we have a good sense of what sorts of environments DreamerV2 does poorly on, relative to the continuous latent ablation? DreamerV1 does well on a variety of continuous control tasks -- does DreamerV2 do poorly in continuous control? (Apologies if I missed this described somewhere.) Having some delineation of where it does well vs poorly could help me get a better sense of use of the discrete latent.\n\nAppendix A is most welcome.\n\n----------------------------------\n\nEdit after reading other conversations.\n\nI do think the other reviewers make some fair points. I've adjusted my score to a 9, though, and I'd very much like to see the paper accepted. Here is my thinking on a couple points that led to this score adjustment.\n\nAnonReviewer2's SimPLE-like ablation request: If I'm understanding this correctly, the reviewer would like to see the policy model trained on reconstructions instead of the latent space (or maybe would like the world model trained to future predict in pixel space? not completely clear to me). To me, this would be a potentially insightful ablation, but I think not a dealbreaker that they do not. Two points on this:\n\n(1) If I am understanding the paper and conversations, SimPLE significantly underperforms in the metric (Atari \"end performance\" under certain normalizations) that the authors care about (and is a fairly established metric). I, then, don't see a *particularly* strong motivation for careful ablations of the method.\n\n(2) Pixel-based future predictions generally perform poorly, and this is I think fairly widely thought to be a strong contributor to the failure of model-based approaches. Again, it would be nice to see that happen here (and it might be insightful to see the quality of the frame predictions) but I think there is a reasonable expectation that this would work poorly.\n\n \nAnonReviewer1's thoughts on the value of this sort of work in ICLR: I see your point, but I personally think there's great value to this sort of work in this sort of venue. End performance on Atari has been (for better or worse) an important baseline for the field, and the creation of performant model-based approaches has been a central question for several years, now. The proposed improvements over DreamerV1 (which has seen fruitful applications in other work) might be simple, but DreamerV1 did not work well on this baseline, and this does. I think that we far too often get excited by complex new methods, often evaluated with novel and poorly understood baselines and metrics, only to drop them as time goes on. That the innovation is a simple one should make us excited to try it in other applications.\n\nI also think that their discussion of evaluation metrics is very useful -- we continually need more careful conversations about the right ways to measure success. So, in short, the paper might be a technically simple innovation, but it puts forth strong evidence that the method is useful.\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0e3V9V8fpLF", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes DreamerV2, a set of modifications to the existing\nmodel-based RL system Dreamer, and shows via an ablation study that\nthese modifications improve performance on the standard Atari\nbenchmark over Dreamer. It is shown that DreamerV2 performs well\ncompared to model-free RL methods as well, especially when using a new\nreporting scheme proposed by the authors that normalizes against the\nworld record on each game rather than the more standard performance of\na strong human player.\n\nIn my view, model-based RL is an extremely important and useful\ndirection to pursue, and so I really appreciate that the authors are\npushing forward in this direction, trying to prove that model-based RL\ncan be better than model-free RL. However, there are two major reasons\nwhy I lean toward voting reject on this paper. (1) There is simply not\nenough technical novelty to warrant publication. (2) The results\nare not particularly compelling to me.\n\n(1) It seems that basically, this paper proposes a couple of small\nideas for how to improve Dreamer, the largest one being the use of\ncategorical, rather than Gaussian, latent states. While it's nice that\nthese yield improvements (although the categorical latent doesn't help much\nby the authors' suggested reporting scheme, looking at the right-most plot in Figure\n3), there is ultimately very little in the way of an interesting\ntechnical/scientific contribution in this paper.\n\n(2) Figure 1 greatly surprises me: although it appears on the first\npage, the results do not seem commensurately compelling. On the\nstandard reporting metric shown in the left plot (which I understand\nthe authors later suggest not to focus on, for various reasons),\nDreamerV2 and Rainbow are indistinguishable, and IQN actually\noutperforms both in the low-data regime. Looking at the right plot, I\ncan hardly say that DreamerV2 outperforms Rainbow. On about 50-60% of the games, it outperforms\nRainbow, but on the others, it performs worse. It would be useful to understand *what*\ncharacteristics of the games DreamerV2 is leveraging, on the games\nwhere it outperforms Rainbow. For instance, are these games perhaps\nmore amenable to model learning for some reason? (The authors do talk\nabout why Video Pinball doesn't do well, but I'm curious why do some of the\nother domains do very well?)\n\nSome other questions:\n\n1. The authors often mention that DreamerV2 uses only a single\nenvironment instance for each game. Some clarification would be useful\nsurrounding this. How many instances do the other evaluated approaches\nuse, like Rainbow? Does this only affect training time, or the learning curves\nas well?\n\n2. Since you're only focusing on Atari experiments, where we actually\ndo have a low-dimensional ground-truth model in the form of the ALE\nRAM state, I am curious if the authors have tried ablating away the\nworld model learning phase of their approach, and instead just use\nthe ground truth ALE transition model? I ask because the authors often\nmention that planning in image space is prohibitively expensive, which I totally\nagree with, but planning in RAM space (with the known model) may not\nbe so bad?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main contribution of the paper is showing that a model-based approach can be competitive with (and even outperform) strong model-free methods on the 200M Atari benchmark. This is achieved through a set of improvements over the original Dreamer algorithm.

Reviewers have been polarized over this submission (4,5,8,9). After reading the paper, reviews, rebuttals, and engaging with all reviewers in private conversations, I am recommending acceptance as a poster. I agree with R3 and R4 that « this is impressive work », « results are a convincing demonstration of its utility », « it is an important setup from the perspective of model-based RL », « the model is elegant », and « the benchmarking discussion is very useful for the community ».

Although it is true, as R1 puts it, that this work can be seen as « an incremental set of tricks over a prior published approach », these tricks are not obvious and lead to very substantial empirical performance gains. Since the authors described them in details and have also committed to sharing their code, I expect them to be quite valuable to other researchers.

Finally, although I respect R2’s choice to stick to their rating of 4, I believe that their main concern, related to not fully understanding why this work improves on the existing SimPLE algorithm, is indeed justified, but is not enough for rejection. DreamerV2 has a lot of differences compared to SimPLE and it would be very costly to investigate in details the impact of each of them. Hopefully, this work will motivate further research in model-based RL that will shed more light on such questions. I would encourage the authors, however, to elaborate a bit more on the differences vs. SimPLE in the « Related work » section (or Appendix, if there is not enough room in the main text).
Decision: Accept (Poster)