Dream to Control: Learning Behaviors by Latent Imagination
Authors: Danijar Hafner
Timothy Lillicrap
Jimmy Ba
Mohammad Norouzi
Venue:  (2020)
Keywords: world model
latent dynamics
imagination
planning by backprop
policy optimization
planning
reinforcement learning
control
representations
latent variable model
visual control
value function

Abstract: Learned world models summarize an agent's experience to facilitate learning complex behaviors. While learning world models from high-dimensional sensory inputs is becoming feasible through deep learning, there are many potential ways for deriving behaviors from them. We present Dreamer, a reinforcement learning agent that solves long-horizon tasks from images purely by latent imagination. We efficiently learn behaviors by propagating analytic gradients of learned state values back through trajectories imagined in the compact state space of a learned world model. On 20 challenging visual control tasks, Dreamer exceeds existing approaches in data-efficiency, computation time, and final performance.

Published as a conference paper at ICLR 2020
DREAM TO CONTROL: LEARNING BEHAVIORS BY LATENT IMAGINATION
Danijar Hafner"
Timothy Lillicrap DeepMind
Jimmy Ba
Mohammad Norouzi
University of Toronto
University of Toronto
Google Brain
Google Brain
Abstract
Learned world models summarize agent's experience to facilitate learning complex behaviors. While learning world models from high-dimensional sensory inputs is becoming feasible through deep learning, there are many potential ways for deriving behaviors from them. We present Dreamer, a reinforcement learning agent that solves long-horizon tasks from images purely by latent imagination. We efficiently learn behaviors by propagating analytic gradients of learned state values back through trajectories imagined in the compact state space of a learned world model. On 20 challenging visual control tasks, Dreamer exceeds existing approaches data-efficiency, computation time, and final performance.
INTRODUCTION
Intelligent agents can achieve goals in complex environments even though they never encounter the exact same situation twice. This ability requires building representations of the world from past experience that enable generalization to novel situations. World models offer an explicit way to represent an agent's knowledge about the world in a parametric model that can make predictions about the future.
Dataset of Experience
When the sensory inputs are high- gigh--imensional images, latent dynamics models can abstract observations to predict forward in compact state spaces (Watter et al., 2015; Oh et al., 2017: Gregor et al., 2019). Compared to predictions in image space, latent states have small memory footprint that enables imagining thousands of trajectories in parallel. Learning effective latent dynamics models is becoming feasible through advances in deep learning and latent variable models (Krishnan et al., 2015; Karl et al., 2016; Doerr et al., 2018; Buesing et al., 2018).
Learned Latent Dynamics
Behaviors can be derived from dynamics models in many ways. Often, imagined rewards are maximized with parametric policy (Sutton, 1991; Ha and Schmidhuber, 2018; Zhang et al., 2019) or by online planning (Chua et al., 2018; Hafner et al., 2018). However, considering only rewards within a fixed imagination horizon results in shortsighted behaviors (Wang et al., 2019). Moreover, prior work commonly resorts to derivative-free optimization for robustness to model errors (Ebert et al., 2017; Chua et al., 2018; Parmas et al., 2019), rather than leveraging analytic gradients offered by neural network dynamics (Henaff et al., 2019; Srinivas et al., 2018).
Value and Action Learned by Latent Imagination
We present Dreamer, an agent that learns long-howe mor-zorizon behaviors from images purely by latent imagination. A novel actor critic algorithm accounts for rewards beyond the imagination horizon while making efficient use of the neural network dynamics. For this, predict state values and actions in the learned latent space as summarized in Figure 1. The values optimize Bellman consistency for imagined rewards and the policy maximizes the values by propagating their analytic gradients back through the dynamics. In comparison to actor critic algorithms that learn online or by experience replay (Lillicrap et al., 2015; Mnih et al., 2016; Schulman et al., 2017; Haarnoja al., 2018; Lee et al., 2019), world models can interpolate past experience and offer analytic gradients of multi-step returns for efficient policy optimization.
Figure 1: Dreamer
learns a world model from past experience and efficiently learns farsighted behaviors in its latent space by backpropagating value estimates back through imagined trajectories.
"Correspondence Danijar Hafner <mail @ @danjjar.com>.

---

Published as a conference paper at ICLR 2020
(a) Cup
(b) Acrobot
(c) Hopper
(d) Walker
(e) Quadruped
Figure 2: Image observations for 5 of the 20 visual control tasks used in our experiments. The tasks pose a variety challenges including contact dynamics, sparse rewards, many degrees of freedom, and 3D environments. Several of these tasks could previously not be solved through world models.
The key contributions of this paper are summarized as follows:
Learning long-horizon behaviors by latent imagination Model-based agents can be short- sighted if they use finite imagination horizon. We approach this limitation by predicting both actions and state values. Training purely by imagination in a latent space lets us efficiently learn the policy by propagating analytic value gradients back through the latent dynamics.
Empirical performance for visual control We pair Dreamer with existing representation learning methods and evaluate it on the DeepMind Control Suite with image inputs, illustrated in Figure Using the same hyper parameters for all tasks, Dreamer exceeds previous model-based and model-free agents in terms of data-efficiency, computation time, and final performance.
2 CONTROL WITH WORLD MODELS
Reinforcement learning We formulate visual control as a partially observable Markov decision process (POMDP) with discrete time step E []; continuous vector-valued actions ag p(ar oct, (.et) generated by the agent, and high-dimensional observations and scalar rewards or, rt. p(o,,r: Vt Oct, (.et) generated by the unknown environment. The goal is to develop an agent that maximizes the expected sum of rewards Ep Ei= rt). Figure shows a selection of our tasks.
Agent components The classical components of agents that learn in imagination are dynamics learning, behavior learning. and environment interaction (Sutton, 1991). In the case of Dreamer, the behavior is learned by predicting hypothetical trajectories in the compact latent space of the world model. As outlined Figure and detailed in Algorithm 1, Dreamer performs the following operations throughout the agent's life time, either interleaved in parallel:
Learning the latent dynamics model from the dataset of past experience to predict future re- wards from actions and past observations. Any learning objective for the world model can be incorporated with Dreamer. We review existing methods for learning latent dynamics Section 4.
Learning action and value models from predicted latent trajectories, described in Section 3. The value model optimizes Bellman consistency for imagined rewards and the action model is updated by propagating gradients of value estimates back through the neural network dynamics.
Executing the learned action model in the world to collect new experience for growing the dataset.
Latent dynamics Dreamer a latent dynamics model that consists of three components. The representation model encodes observations and actions to create continuous vector-valued model states 8 with Markovian transitions (Watter et al., 2015: Zhang al., 2019; Hafner al., 2018). The transition model predicts future model states without seeing the corresponding observations that will later cause them. The reward model predicts the rewards given the model states,
Representation model:
p( 8t 0t-1,
Transition model:
g(s, 1-.10
Reward model:
q(r: st).
We p for distributions that generate samples in the real environment and for their approximations that enable latent imagination. Specifically, the transition model lets us predict ahead in the compact latent space without having observe imagine the corresponding images. This results in a low memory footprint and fast predictions of thousands of imagined trajectories in parallel.
The model mimics non-linear Kalman filter (Kalman. 1960), latent state space model, or HMM with real-valued states. However, it is conditioned on actions and predicts rewards, allowing the agent imagine the outcomes of potential action sequences without executing them in the environment.
2

---

Published as a conference paper at ICLR 2020
a
F
a
a
3,
0g
9,
(a) Learn dynamics from experience
(b) Learn behavior in imagination
(c) Act the environment
Figure 3: Components of Dreamer. (a) From the dataset of past experience, the agent learns encode
observations and actions into compact latent states ), for example via reconstruction, and predicts environment rewards (o). (b) In the compact latent space, Dreamer predicts state values and actions (1) that maximize future value predictions by propagating gradients back through imagined trajectories. (c) The agent encodes the history of the episode to compute the current model state and predict the next action to execute in the environment. See Algorithm for pseudo code of the agent.
3 LEARNING BEHAVIORS BY LATENT IMAGINATION
Dreamer learns long-horizon behaviors the compact latent space of a learned world model by efficiently leveraging the neural network latent dynamics. For this, we propagate stochastic gradients of multi-step returns through neural network predictions of actions, states, rewards, and values using reparameterizztion. This section describes the main contribution of our paper.
Imagination environment The latent dynamics define a Markov decision process (MDP; Sutton, 1991) that is fully observed because the compact model states 8, Markovian. We denote imagined quantities with as the time index. Imagined trajectories start at the true model states of observation sequences drawn from the agent's past experience. They follow predictions of the transition model 8, ~ 4(,, br 87-1, 41 reward model ry q(r; sr), and policy a, g(a, 37). The objective is to maximize expected imagined rewards E.g (Er -r-)r) with respect to the policy.
Algorithm 1: Dreamer
Initialize dataset D with random seed episodes.
Model components
Initialize neural network parameters 0, Φ, v randomly.
Representation
Po(s: $11, 0.1, 0.)
while not converged do for update step 1..C do
Transition
(s: St1, ax11)
Reward
8t)
Dynamics learning
Action
q
st))
Draw B data sequences {(a), 0t, r1))k D.
Value
(p()))
Compute model states 8t Po( 94-1, Update using representation learning.
Hyper parameters
Seed episodes
S
// Behavior learning
Collect interval
C'
Imagine trajectories {(T, ar) 1++ from each 8t-
Batch size
B
Predict rewards E (g 87)) and values ey (s,))
Sequence length
L
Compute value estimates Vx(s;) ) via Equation 6.
Update + a Et+ ++# Vx(s)).
Imagination horizon
H
Update v + a
++
7=t 2 ( -x>ss)]1
Learning rate
a
// Environment interaction 01 + env rewete () for time step = 1..7 do
Compute s, (s( St-1, 41-1, 0t) from history.
Compute a₂ ~ (a; se) with the action model. Add exploration noise to action.
rt, oc+1 6- env. env.step step (a+).
Add experience to dataset D + DU ((o 0e, rt)
3

---

Published as a conference paper at ICLR 2020
Cartpole Swingup
Chectah Run
Quadraped Walk
W Walker Walk
1000
1000
1000
1000
Eptcodk Return
800
800
800
800
600
600
600
600
400
400
400
400
Dreamer (1)
200
200
200
200
PlaNet
No value (3%)
(P))
10
30
40
10
20
30
40
10
20
30
40
10
20
30
Imaagination Hérrzonn
Horizon
Insaginatiin Horizon
Imagination Horizon
Figure 4: Imagination horizons. We compare the final performance of Dreamer, learning an action model without value prediction, and online planning using PlaNet. Learning a state value model to estimate rewards beyond the imagination horizon makes Dreamer more robust to the horizon length. The agents use pixel reconstruction for representation learning and an action repeat of R = 2.
Action and value models Consider imagined trajectories with finite horizon H. Dreamer uses an actor critic approach to learn behaviors that consider rewards beyond the horizon. We learn an action model and value model the latent space of the world model for this. The action model implements the policy and aims to predict actions that solve the imagination environment. The value model estimates the expected imagined rewards that the action model achieves from each state s,.
Action model:
a, ~ ( | 17) (s(s)) Eg(()s.) ( Et=H - (r-)
Value model:
The action and value models are trained cooperatively as typical in policy iteration: the action model aims to maximize an estimate of the value, while the value model aims match an estimate of the value that changes as the action model changes.
We use dense neural networks for the action and value models with parameters á and th respectively. The action model outputs tanh-transformed Gaussian (Haarnoja al., 2018) with sufficient statisties predicted by the neural network. This allows for reparameterized sampling (Kingma and Welling, 2013; Rezende al., 2014) that views sampled actions as deterministically dependent on the neural network output, allowing us backpropagate analytic gradients through the sampling operation,
a = tanh((p(ss)) + co( Normal(O. I).
Value estimation To leamn the action and value models, need to estimate the state values of imagined trajectories {Sy, ay rr 14+H )+ sequence batches drawn from the agent's dataset of experience and predict forward for the imagination horizon H using actions sampled from the action model. State values can be estimated in multiple ways that trade off bias and variance (Sutton and Barto, 2018),
These trajectories branch off of the model states s, of
++ Σ
VR(S;)= = Ego.go
I'
(4)
(&+) = Eggu.go
7 n-T rn - (=(TT(((s)))
with min(r k,t H),
(5)
8-1
Va(+,) = (1
Σ n=1
'm (2, " -
(6)
where the expectations are estimated under the imagined trajectories. VR simply sums the rewards from t until the horizon and ignores rewards beyond This allows learning the action model without a value model, ablation we compare to in our experiments. Vš estimates rewards beyond k steps with the learned value model. Dreamer uses VX. an sxponentiallyyweighhe for different i to balance bias and variance. Figure shows that learning value model in imagination enables Dreamer to solve long- horizon tasks while being robust to the imagination horizon. The experimental details and results all tasks are described in Section 6.
average of the estimates
4

---

Published as a conference paper at ICLR 2020
Context
6
10
15
20
25
30
35
40
45
50
Model True Model True
4
4
5
{
/
H
M
S
5
4
Figure 5: Reconstructions of long-term predictions. We apply the representation model to the first images of two hold-out trajectories and predict forward for 45 steps using the latent dynamics, given only the actions. The recurrent state space model (RSSM; Hafner et al., 2018) performs accurate long-term predictions, enabling Dreamer to learn successful behaviors in compact latent space.
Learning objective To update the action and value models, we first compute the value estimates (s(s) for all states s, along the imagined trajectories. The objective for the action model (a(a, s,) to predict actions that result in state trajectories with high value estimates. The objective for the value model o (sr), ), in turn, is to regress the value estimates,
I+H Σ (v(()))
++ Σ (oloo( - va(++))
max Equ.go
(7)
min
Eige To:Rs
The value model is updated to regress the targets, around which we stop the gradient as typical (Sutton and Barto, 2018). The action model uses analytic gradients through the learned dynamics to maximize the value estimates. To understand this, we note that the value estimates depend on the reward and value predictions, which depend on the imagined states, which in turn depend on the imagined actions. Since all steps implemented neural networks, V Vg |E= Vx(s)) by stochastic backpropagation (Kingma and Welling, 2013; Rezende al., 2014). We reparameterization for continuous actions and latent states and straight-through gradients (Bengio et al., 2013) for discrete actions. The world model fixed while learning behaviors. In tasks with early termination, the world model also predicts the discount factor from each latent state to weigh the time steps in Equations and 8 by the cumulative product the predicted discount factors, so terms are weighted down based on how likely the imagined trajectory would have ended. Comparison actor critic methods Agents using Reinforce gradients (Williams, 1992), such as A3C and PPO (Mnih et al., 2016; Schulman et al., 2017), employ value baselines to reduce gradient variance, while Dreamer backpropagates through the value model. This is similar to deterministic or reparameterized actor critics (Silver et al., 2014), such as DDPG and SAC (Lillicrap et al., 2015; Haarnoja et al., 2018). However, these do not leverage gradients through transitions and only maximize immediate Q-values. MVE and STEVE (Feinberg et al., 2018; Buckman et al., 201 8) extend them to multi-step Q-learning with learned dynamics to provide more accurate Q-value targets. We predict state values, which is sufficient for policy optimization since we backpropagate through the dynamics. Refer to Section 5 for a more detailed comparison to related work.
analytically compute
4 LEARNING LATENT DYNAMICS
Learning behaviors in imagination requires world model that generalizes well. We focus latent dynamics models that predict forward in a compact latent space, facilitating long-term predictions and allowing the agent to imagine thousands of trajectories in parallel. Several objectives for learning representations for control have been proposed (Watter et al., 2015; Jaderberg et al., 2016; Oord al., 2018; Eslami et al., 2018). We review three approaches for learning representations to with Dreamer: reward prediction, image reconstruction, and contrastive estimation.
Reward prediction Latent imagination requires representation model p(st 2 oz).
transition model g(s; 5 (t-1, and reward model g(r; 5r), as described in Section 2. In principle, this could be achieved by simply learning predict future rewards given actions and past observations (Oh et al., 2017: Gelada et al., 2019; Schrittwieser et al., 2019). With a large and diverse dataset, such representations should be sufficient for solving control task. However, with finite dataset and especially when rewards are sparse, learning about observations that correlate with rewards is likely improve the world model (Jaderberg al., 2016; Gregor al., 2019).
5

---

Published as a conference paper at ICLR 2020
Dreamer (5e6 steps)
PlaNet (5c6 steps)
D4PG (1c9 steps)
A3C (1c9 steps, proprio)
Episode Return
1000
800
400
200
Run RR
Carppole
Balance
Walker
Stand
Cup
Cateb
Walker
Walk
Camppoe
Bai. Sparse
Reacher
Easy
Quadruped
Walk
Hopper
Stand
Cheetab
Run
Finger
Turn Hord
Quadruged
Cartpole Swingup
Peodulum
Swingup
Finger
Walker
Run
Reacher
Hord
Cutpols
Swi. Sparse
Finger
Spin.
Happ
Turn
Figure 6: Performance comparison to existing methods. Dreamer inherits the data-efficieney of
PlaNet while exceeding the asymptotic performance of the best model-free agents. After 5 106 environment steps, Dreamer reaches an average performance of 823 across tasks, compared to PlaNet 332 and the top model-free D4PG agent at 786 after 10 steps. Results are averages over 5 seeds.
Reconstruction We first describe the world model used by PlaNet (Hafner et al., 2018) that learns latent dynamics by reconstructing images as shown in Figure 3a. The world model consists of the following components, where the observation model is only used to provide learning signal,
Representation model:
(s; S1-1,
Observation model:
go(or 8t)
Reward model:
(r(rz |s)
Transition model:
qo(st St-1, Qz-1).
The components optimized jointly increase the variational lower bound (ELBO; Jordan et al., 1999) or more generally the variational information bottleneck (VIB; Tishby et al., 2000; Alemi et al., 2016). As derived in Appendix B, the bound includes reconstruction terms for observations and rewards and KL regularizer. The expectation is taken under the dataset and representation model,
JREC = Ep
Σ
Jk (R+TT) Jo
Ib)
+ const To = Inq( st)
(10)
Jh = Inq 82)
JB - KKL (p(s: 81-1, 01-100))
q(se s1-1,
We implement the transition model as a recurrent state space model (RSSM: Hafner et al., 2018), the representation model by combining the RSSM with convolutional neural network (CNN; LeCun al., 1989) applied the image observation, the observation model as a transposed CNN, and the reward model as dense network. The combined parameter vector 0 is updated by stochastic backpropagation (Kingma and Welling, 2013; Rezende et al., 2014). Figure shows video predictions of this model. We refer to Appendix A and Hafner al. (2018) () model details.
Contrastive estimation Predicting pixels can require high model capacity. We can also encourage mutual information between model states and observations by instead predicting the states from the images (Guo et al., 2018). This replaces the observation model with state model,
State model:
(s oz).
(11)
While the reconstruction objective used the fact that the observation marginal is constant, we now face the state marginal. As shown Appendix B, this can be estimated noise contrastive estimation (NCE; Gutmann and Hyvärinen, 2010; Oord et al., 2018) 8) by averaging the state model over observations of the current sequence batch. Intuitively, qs oz) makes the state predictable from the current image while In Ev q(sc. o') keeps it diverse to prevent collapse.
JNCE E Σ
(
Th + +b))
Th Ing(s: ot) In E E9(se 10))
(12)
We implement the state model as a CNN and again optimize the bound with respect to the combined parameter vector using stochastic backpropagation. While avoiding pixel prediction, the amount of information this bound can extract efficiently limited (McAllester and Statos, 2018). We empirically compare reward, reconstruction, and contrastive objectives in experiments in Figure 8.

---

Published as a conference paper at ICLR 2020
Acrobot Swingup
Caripole Swingup Spurse
Hopper Hop
Hopper Stand
1000
400
Egtoode Return
300
600
750
400
200
500
100
200
250
0
0.5
1.5
0.0
0.5
1.0
1.5
2.0
Pendulum Swingup
Quadruped Walk
Walker Run
Walker Walk
1000
800
1000
Eprioode Return
750
600
750
500
400
500
250
250
200
250
0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Environment Staps
lefi
Eavironment Staps
le6
Environment Steps
lc6
Environment Steps
Dreamer
No value
PlaNet
--
D4PG (1e9 steps)
--
ASC (Ie9 steps, propris)
Figure 7: Dreamer succeeds at visual control tasks that require long-horizon credit assignment, such as the acrobot and hopper tasks. Optimizing only imagined rewards within the horizon via an action model or by online planning yields shortsighted behaviors that only succeed in reactive tasks, such as in the walker domain. The performance on all 20 tasks summarized in Figure and training curves are shown in Appendix D. See Tassa et al. (2018) for performance curves of D4PG and A3C.
RELATED WORK
Prior works learn latent dynamics for visual control by derivative-fres policy learning or online planning, augment model-free agents with multi-step predictions, or use analytic gradients of Q- values or multi-step rewards, often for low-dimensional tasks. comparison, Dreamer uses analytic gradients to efficiently learn long-horizon behaviors for visual control purely by latent imagination.
Control with latent dynamics E2C (Watter et al., 2015) and RCE (Banijamali et al., 2017) embed images to predict forward a compact space to solve simple tasks. World Models (Ha and Schmid- huber, 2018) learn latent dynamics in two-stage process to evolve linear controllers in imagination. PlaNet (Hafner et al., 2018) learns them jointly and solves visual locomotion tasks by latent online planning. SOLAR (Zhang al., 2019) solves robotic tasks via guided policy search in latent space. 12A (Weber et al., 2017) hands imagined trajectories to a model-free policy, while Lee et al. (2019) and Gregor et al. (2019) learn belief representations to accelerate model-free agents.
Imagined multi-step returns VPN (Oh et al., 2017), MVE (Feinberg al., 2018), and STEVE (Buckman et al., 2018) learn dynamics for multi-step Q-learning from a replay buffer. AlphaGo (Silver et al., 2017) combines predictions of actions and state values with planning, assuming access to the true dynamics. Also assuming access to the dynamics, POLO (Lowrey et al., 2018) 8) plans to explore by learning a value ensemble. MuZero (Schrittwieser al., 2019) learns -see-ipe reward and value models to solve challenging tasks but requires large amounts of experience. PETS (Chua al., 2018), VisualMPC (Ebert et al., 2017), and PlaNet (Hafner et al., 2018) plan online using derivative- freee optimization. POPLIN (Wang and Ba, 2019) improves over online planning by self-imitation. Piergiovanní et al. (2018) learn robot policies by imagination with latent dynamics model. Planning with neural network gradients Henaff et al., 2018) but has been challenging to scale (Parmas et al., 2019).
shown small problems (Schmidhuber, 1990;
Analytic value gradients DPG (Silver et al., 2014), DDPG (Lillicrap et al., 2015), and SAC (Haarnoja et al., 2018) leverage gradients of learned immediate action values to learn policy by experience replay. SVG (Heess et al., 2015) reduces the variance of model-free on-policy algorithms by analytic value gradients of one-step model predictions. Concurrent work by Byravan et al. (2019) uses latent imagination with deterministic models for navigation and manipulation tasks. ME-TRPO (Kurutach et al., 2018) accelerates otherwise model-free agent via gradients of predicted rewards for proprioceptive inputs. DistGBP (Henaff et al., 2017; 2019) uses model gradients for online planning in simple tasks.
7

---

Published as a conference paper at ICLR 2020
Acroboo Swingup
Cheetah Run
Cup Catch
Finger Spin
1000
1000
Egloode Return
750
500
750
300
500
500
100
250
250
250
1.5
2.0
0.5
1.5
2.0
Hopper Stund
Pendulum Swingup
Quadruped Run
Walker Stand
1000
1000
1000
1000
Eptooks Return
750
750
500
600
250
400
200
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Environment Steps
lc6
Environment Steps
Environment Steps
166
Environment Steps
Dreamer + Reconstructio
Dreamer + Contrastive
Decomer + Rolard
DAPG (1c9 skeps)
A3C (le9 steps, proprio)
Figure 8: Comparison of representation learning objectives to be used with Dreamer. Pixel recon- struction performs best for the majority of tasks. The contrastive objective solves about half of the tasks, while predicting rewards alone was not sufficient in our experiments. The results suggest that future developments in learning representations are likely to translate into improved task performance for Dreamer. The performance curves for all tasks are included in Appendix E.
6 EXPERIMENTS
We experimentally evaluate Dreamer on variety of control tasks. We designed the experiments to compare Dreamer to current best methods in the literature, and to evaluate its ability to solve tasks with long horizons, continuous actions, discrete actions, and early termination. We further compare the orthogonal choice of learning objective for the world model. The source code for all our experiments and videos of Dreamer are available https: / dan jar com/ dreamer.
Control tasks We evaluate Dreamer on 20 visual control tasks of the DeepMind Control Suite (Tassa al., 2018), illustrated in Figure 2. These tasks pose variety of challenges, including sparse rewards, contact dynamics, and 3D scenes. We selected the tasks on which Tassa et al. (2018) report non-zero performance from image inputs. Agent observations are images of shape 64 x 64 x actions range from to 12 dimensions, rewards range from 0 to 1, episodes last for 1000 steps and have randomized initial states. We use fixed action repeat of R = 2 across tasks. We further evaluate the applicability of Dreamer to discrete actions and early termination on a subset of Atari games (Bellemare al., 2013) and DeepMind Lab levels (Beattie et al., 2016) as detailed Appendix
Implementation Our implementation uses TensorFlow Probability (Dillon et al., 2017). We use single Nvidia V100 GPU and 10 CPU cores for each training run. The training time for our Dreamer implementation is below 5 hours per 10th environment steps on the control suite, compared to 11 hours for online planning using PlaNet, and the 24 hours used by D4PG to reach similar performance. We use the same hyper parameters across all continuous tasks, and similarly across all discrete tasks, detailed in Appendix A. The world models are learned via reconstruction unless specified.
Baseline methods The highest reported performance on the continuous tasks is achieved by D4PG (Barth-Maron et al., (018), an improved variant of DDPG (Lillicrap et al., 2015) that uses distributed collection, distributional Q-learning, multi-step returns, and prioritized replay. We include the scores for D4PG with pixel inputs and A3C (Mnih et al., 2016) with state inputs from Tassa et al. (2018). PlaNet (Hafner et al., 2018) learns the same world model Dreamer and selects actions via online planning without an action model and drastically improves over D4PG and A3C in data efficiency. We re-run PlaNet with R for unified experimental setup. For Atari, show the final performance of SimPLe (Kaiser et al., 2019), DQN (Mnih et al., 2015) and Rainbow (Hessel et al., 2018) reported by Castro al. (2018), and for DeepMind Lab that of IMPALA (Espeholt et al., 2018) as a guideline.
8

---

Published as a conference paper at ICLR 2020
Performance To evaluate the performance of Dreamer, compare it state-of-the-at art reinforce- ment learning agents. The results are summarized in Figure With an average score 823 across tasks after 5 x 10t environment steps, Dreamer exceeds the performance of the strong model-free D4PG agent that achieves an average of 786 within 10 environment steps. the same time, Dreamer inherits the data-efficiency of PlaNet, confirming that the learned world model can help generalize from small amounts of experience. The empirical success of Dreamer shows that learning behaviors by latent imagination with world models can outperform top methods based on experience replay.
Long horizons To investigate its ability to learn long-horizon behaviors, we compare Dreamer to alternatives for deriving behaviors from the world model at various horizon lengths. For this, we learn an action model to maximize imagined rewards without value model and compare to online planning using PlaNet. Figure 4 shows the final performance for different imagination horizons, confirming that the value model makes Dreamer more robust to the horizon and performs well even for short horizons. Performance curves for all 19 tasks with horizon of 20 are shown in Appendix D, where Dreamer outperforms the alternatives 16 of 20 tasks, with 4 ties.
Representation learning Dreamer can be used with any differentiable dynamics model that pre- dicts future rewards given actions and past observations. Since the representation learning objective is orthogonal to our algorithm, we compare three natural choices described in Section 4: pixel recon- struction, contrastive estimation, and pure reward prediction. Figure 8 shows clear differences in task performance for different representation learning approaches, with pixel reconstruction outperform- ing contrastive estimation on most tasks. This suggests that future improvements in representation learning are likely translate to higher task performance with Dreamer. Reward prediction alone was not sufficient in our experiments. Further ablations are included in the appendix of the paper.
7 CONCLUSION
We present Dreamer, an agent that learns long-horizon behaviors purely by latent imagination. For this, we propose an actor critic method that optimizes parametric policy by propagating analytic gradients of multi-step values back through learned latent dynamics. Dreamer outperforms previous methods in data-efficiency, computation time, and final performance on variety of challenging continuous control tasks with image inputs. We further show that Dreamer is applicable to tasks with discrete actions and early episode termination. Future research representation learning can likely scale latent imagination to environments of higher visual complexity.
Acknowledgements We thank Simon Kornblith, Benjamin Eysenbach, Ian Fischer, Amy Zhang, Geoffrey Hinton, Shane Gu, Adam Kosiorek, Jacob Buckman, Calvin Luo, and Rishabh Agarwal, and our anonymous reviewers for feedback and discussions. We thank Yuval Tassa for adding the quadruped environment to the control suite.

---

Published as a conference paper at ICLR 2020
REFERENCES
A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410 2016.
E. Banijamali, R. Shu, M. Ghavamzadeh, H. Bui, and A. Ghodsi. Robust locally-linear controllable embedding. arXiv preprini urXiv:1710.05377 2017.
G. Barth-Maron, M. W. Hoffman, D. Budden, W. Dabney, D. Horgan, A. Muldal, N. Heess, and T. Lil- licrap. Distributed distributional deterministic policy gradients. arXiv preprint arXiv:1804.08111 2018.
C. Beattie, J. Z. Leibo, D. Teplyashin, T. Ward, M. Wainwright, H. Küttler, A. Lefrancg, S. Green, V. Valdés, A. Sadik, al. Deepmind lab. arXiv preprint arXiv: 1iX.vv6612.008000 0.00000,,,,,,. 2016.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
Y. Bengio, N. Léonard, and A. Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint
2013.
J. Buckman, D. Hafner, G. Tucker, E. Brevdo, and H. Lee. Sample-efficient reinforcement learning with stochastic ensemble value expansion. Advances in Neural Information Processing Systems, pages 8224 8234, 2018
L. Buesing, Weber, Racaniere, Eslami, D. Rezende, Reichert, F. Viola, F. Besse, K. Gregor, D. Hassabis, et al. Learning and querying fast generative models for reinforcement learning. arXiv preprint
000000000000. 2018.
A. Byravan, J. T. Springenberg, A. Abdolmaleki, R. Hafner, M. Neunert, T. Lampe, N. Siegel, N. Heess, and M. Riedmiller. Imagined value gradients: Model-based policy optimization with transferable latent dynamics models. arXiv preprint arXiv:1910.04142. 2019.
P. S. Castro, S. Moitra, C. Gelada, S. Kumar, and M. G. Bellemare. Dopamine: A research framework for deep reinforcement learning. arXiv preprint arXiv:1812.061100 2018.
K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pages 4754 4765, 2018
D.-A. Clevert, T. Unterthiner, and S. Hochreiter. Fast and accurate deep network learning by exponential linear units (clus), arXiv preprint arXiv:1511.07289 2015.
V. Dillon, I. Langmore, Tran, E. Brevdo, Vasudevan, D. Moore, B. Patton, A. Alemi, M. Hoffman, and R. A. Saurous. Tensorflow distributions. arXiv preprint 1et.17111116004 2017.
A. Doemr C. Daniel, M. Schiegg, D. Nguyen-Tuong, S. Schaal, M. Toussaint, and S. Trimpe. Probabilistic recurrent state-space models. arXiv preprint arXiv: 10395, 2018
F. Ebert, C. Finn, A. X. Lee, and S. Levine. Self-supervised visual planning with temporal skip connections. arXiv preprint arXiv:1710.05266, 2017.
S. A. Eslami, D. J. Rezende, F. Besse, F. Viola, A. S. Morcos. M. Garnelo, A. Ruderman, A. A. Rusu, Danihelka, K. Gregor, et Neural scene representation and rendering. Science, 360(6394): 1204-1218 2018.
L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley I. Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.0150 2018.
V. Feinberg, A. Wan, Stoica, M. I. Jordan, J. Gonzalez, and S. Levine. Model-based value estimation for efficient model-free reinforcement learning. arXiv preprint / 00/01, 2018.
10

---

Published as a conference paper at ICLR 2020
C. Gelada, S. Kumar, J. Buckman, O. Nachum, and M. G. Bellemare. Deepmdp: Learning continuous latent space models for representation learning. arXiv preprint arXiv:1906.020 02736, 2019.
K. Gregor, D. J. Rezende, Besse, Y. Wu, H. Merzic, and A. V. Oord. Shaping belief states with generative environment models for rl. arXiv preprint arXiv: 09237, 2019.
Z. D. Guo, M. G. Azar, Piot, B. A. Pires, I. Pohlen, and R. Munos. Neural predictive belief 2018.
representations. arXiv preprint
M. Gutmann and A. Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 297-304, 2010.
D. Ha and J. Schmidhuber. World models. arXiv preprint 1803::1803.11222 2018.
T. Haarnjja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:/801.01290 2018.
2018.
D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. arXiv preprint / 1..055551 2018.
N. Heess, G. Wayne, D. Silver, T. Lillicrap, T. Erez, and Y. Tassa. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems, pages 2944-2952, 2015.
M. Henaff, W. F. Whiteey and Y. LeCun. Model- planning discrete action spaces. CoRR. Moss/705.07111 2017.
M. Henaff, W. F. Whitney, and Y. LeCun. Model-based planning with discrete and continuous actions. arXiv preprint arXiv: TXXXv:/705.071177 2018.
M. Henaff, A. Canziani, and Y. LeCun. preecctree policy learning with uncertainty regulariza- tion for driving in dense traffic. arXiv preprint arXiv: 02705, 2019.
M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. In Secody AAAI Conference on Artificial Intelligence. 2018.
M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, Silver, and K. Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397 2016.
M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. An introduction to variational methods for graphical models. Machine learning, 37(2):183-2)
1999.
L. Kaiser, M. Babaeizadeh, P. Milos, B. Osinski, R. H. Campbell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, S. Levine, al. Model-based reinforcement learning for atari. arXiv preprint arXiv:1903.00374. 2019.
R. E. Kalman. A new approach to linear filtering and prediction problems. Journal of basic Engineering, 82(1):35 1960.
M. Karl, M. Soelch, J. Bayer, and P. van der Smagt. Deep variational bayes filters: Unsupervised learning of state space models from raw data. arXiv preprint 1xXiv:1605.04432 2016.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
D. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 132.6111 2013.
R. G. Krishnan, U. Shalit, and D. Sontag. Deep kalman filters. arXiv preprint arXiv:1511.0510 2015.
Kurutach, I. Clavera, Y. Duan, A. Tamar, and P. Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint / 2018

---

Published as a conference paper at ICLR 2020
Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.
A. X. Lee, A. Nagabandi, Abbeel, and Levine. Stochastic latent actor-critic: Deep reinforcement learning with latent variable model. arXiv preprint arXiv: 2019.
T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint
2015.
K. Lowrey, A. Rajeswaran, S. Kakade, E. Todorov, and I. Mordatch. Plan online, learn offline: Efficient learning and exploration via model-based control. arXiv preprint arXiv: 11/111//1111. 2018.
M. C. Machado, M. G. Bellemare, E. Talvitie, J. Veness, M. Hauskmecht, and M. Bowling. Revisiting the areade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
D. McAllester and K. Statos. Formal limitations the measurement of mutual information. arXiv preprint /8/ 1.04251, 2018.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Ried- miller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, pages 1928-19377 2016.
J. Oh, S. Singh, and H. Lee. Value prediction network. In Advances in Neural Information Processing Systems, pages 61 18-6128, 2017.
A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 180.:03700337 2018
Parmas, C. E. Rasmussen, Peters, and K. Doya. Pipps: Flexible model-based policy search robust to the curse of chaos. arXiv preprint arxiv: 2019.
A. Piergiovanni, A. Wu, and M. S. Ryoo. Learning real-world robot policies by dreaming. arXiv preprint arXiv:1805.0788 07.113,, 2018.
B. Poole, S. Ozair, A. d. Oord, A. A. Alemi, and G. Tucker. On variational bounds of mutual information. arXiv preprint arXiv: 100::9055000 009206 2019.
D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint 1rXiv: 2014.
Schmidhuber. Making the world differentiable: On using self-supervised fully recurrent neural networks for dynamic reinforcement learning and planning in non-stationary environments. 1990.
J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, I.. Sifre, S. Schmitt, A. Guez, F. Lockhart, D. Hassabis, T. Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. arXiv preprint arXiv: 2019.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint
D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. Deterministic policy gradient algorithms. Proceedings of the 31 International Conference on Machine Learning, 2014.
D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676): 354, 2017.
12

---

Published as a conference paper at ICLR 2020
A. Srinivas, A. Jabri, P. Abbeel, S. Levine, and C. Finn. Universal planning networks. arXiv preprint arXiv:1804.00645 2018.
R. S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting, ACM SIGART Bulletin, 264):160-163, 1991.
R. Sutton and A. Barto. Reinforcement learning: An introduction. MIT press, 2018.
Y. Tassa, Y. Doron, Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, Merel,
A. Lefrancq, et al. Deepmind control suite. arXiv preprint arXi::1801.00690 2018.
N. Tishby. F. C. Pereira, and W. Bialek. The information bottleneck method. physics/0004057, 2000.
arXiv preprint
T. Wang and J. Ba. Exploring model-based planning with policy networks. arXiv preprint
arXiv:1906.0866 08649, 2019.
Wang, X. Bao, I. Clavera, J. Hoang, Y. Wen, Langlois, Zhang. G. Zhang, P. Abbeel, and Ba. Benchmarking model-based reinforcement learning. CoRR, abs/1907 0000000077777.. 2019.
M. Watter, J. Springenberg, J. Boedecker, and M. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances in neural information processing systems, pages 2754,, 2015.
Weber, S. Racanière, D. P. Reichert, L. Buesing, A. Guez, D. Rezende, A. Badia, O. Vinyals, agents for deep reinforcement learning, arXiv
N. Heess, Y. Li, et al.
preprint arXiv: 0007/77070000. 2017.
R. Williams. Simple statistical gradient-following algorithms connectionist reinforcement learning. Machine learning, 8(3-4):229 2262226.-..6 1992.
M. Zhang, S. Vikram, L. Smith, P. Abbeel, M. Johnson, and S. Levine. Solar: representations for model-based reinforcement learning, In International Conference on Machine Learning, 2019.
deep structured
13

---

Published as a conference paper at ICLR 2020
A HYPER PARAMETERS
Model components We use the convolutional encoder and decoder networks from Ha and Schmid- huber (2018), the RSSM of Hafner et al. (2018), and implement all other functions as three dense layers of size 300 with ELU activations (Clevert et al., 2015). Distributions in latent space 30-dimensional diagonal Gaussians. The action model outputs a tanh mean scaled by factor of 5 and softplus standard deviation for the Normal distribution that is then transformed using tanh (Haarnoja et al., 2018). The scaling factor allows the agent to saturate the action distribution.
Learning updates We draw batches of 50 sequences length 50 to train the world model, value model, and action model models using Adam (Kingma and Ba, 2014) with learning rates 6 x 10-1 x 10-5, 8 x 10-5, respectively and scale down gradient norms that exceed 100. We do not scale the KL regularizers (3 1) but clip them below 3 free nats as in PlaNet. The imagination horizon is H = and the same trajectories are used to update both action and value models. We compute the Va targets with y 0.99 and À 0.95. We did not find latent overshooting for learning the model, an entropy bonus for the action model, or target networks for the value model necessary.
Environment interaction The dataset initialized with S = 5 episodes collected using random actions. We iterate between 100 training steps and collecting episode by executing the predicted mode action with Normal(O,, 0.3) exploration noise. Instead of manually selecting the action repeat for each environment as in Hafner et al. (2018) and Lee et al. (2019), we fix it to for all environments. See Figure 12 for an assessment of the robustness different action repeat values.
Discrete control For experiments Atari games and DeepMind Lab levels, the action model predicts the logits of a categorical distribution. We use step during latent imagination. The action noise is epsilon greedy where is linearly scheduled from 0.4 + 0.1 over the first 200, 000 gradient steps. To account for the higher complexity of these tasks, we use imagination horizon of H 10, scale the KL regularizers by 3 0.1, and bound rewards using tanh. We predict the discount factor from the latent state with a binary classifier that is trained towards the soft labels of and 7
gradients for the sampling
14

---

Published as a conference paper at ICLR 2020
B DERIVATIONS
We define the information bottleneck objective (Tishby et al., 2000) for latent dynamics models,
max [88::T; (O1:T, (1:T) (1:)) - / (81:T f1:T (1:T),
(13)
where 3 is scalar and i, are dataset indices that determine the observations p(or (n) = 8(o, σt) as in Alemi et al. (2016).
Maximizing the objective leads to model states that can predict the sequence of observations and rewards while limiting the amount of information extracted at each time step. This encourages the model reconstruct each image by relying on information extracted preceeding time steps the extent possible, and only accessing additional information from the current image when necessary. As result, the information regularizer encourages model to learn long-term dependencies.
For the generative objective, we lower bound the first term using the non-negativity of the KL divergence and drop the marginal data probability does not depend on representation model,
1(81:75 (OLTT, T1:T) (1:T)
In phorr, T1: (1rr,Orr) Implorg, 01.T
cuma
ApploiT,TI: s1.,,1T)
In p(ooorssTer ser, KL (plorsiri S1r,,rr))
IIq(o a(o (t)a((r 1s))
Σ In I st) + Ing(re si)).
(14)
For the contrastive objective, we subtract the constant marginal probability of the data under the variational encoder, apply Bayes rule, and use the InfoNCE mini-batch bound (Poole et al., 2019),
E(In s.) In st))
= g(In (x.) (n:((:)) + In g(rz ss))
E(In o,)) In(():) + +Ing(r: st))
(15)
2 E In o.) In Σ q(sa 1o' Inq(ra s.)
For the second term, the non-negativity of the KL divergence to obtain upper bound,
1(s17ςirr air)
Σ p(s st-1, a-1. p(s st-1, 01-1)
E
Σ Inp(s: (--10 Imp(s: | (1-1,
(16)
E In p(s s(-1, - In / (1-1,
F
Σ KL. (p(s: 80-1 g((t 9t-1,0t
This lower bounds the objective.
15

---

Published as a conference paper at ICLR 2020
C DISCRETE CONTROL
We evaluate Dreamer on a subset of tasks with discrete actions from the Atari suite (Bellemare et al., 2013) and DeepMind Lab (Beattie et al., 2016). While agents that purely learn through world models are not yet competitive in these domains (Kaiser et al., 2019), the tasks offer a diverse test bed with visual complexity. sparse rewards, and early termination. Agents observe 64 x 64 3 images and select one of between 3 and 18 actions. For Atari, we follow the evaluation protocol of Machado al. (2018) with sticky actions. Refer to Figure 9 for these experiments.
Boxing
Choppercommund
Douhledunk
Fishingderhy
20
Epioande Retun
50
5000
20
167
167
Ice Hockey
Kamgaroo
Krull
Kungfiumaster
Egiiade Return
7500
-10
5000
-15
2500
-20
1c7
167
167
167
Mspacman
N Namethisgame
Pong
Turankham
1000
10000
Episode Ream
10
200
3000
7500
5000
100
1000
2500
167
167
1c7
1c7
Up N Down
Zaxxon
Collect Good Objects
W Watermaze
200000
15000
10.0
Egtinol Rofurn
7.5
10000
100000
5.0
M
50000
2.5
0.0
125 Enviranment Steps lc7
Environment Steps 1c7
Environment Steps l67
Environment Steps 107
Drcamer
SimPLe (lv5
DQN (2c8 Rainbow (2c8
IMPALA (1e10
Random
Figure 9; Performance of Dreamer in environments with discrete actions and early termination. Dreamer learns successful behaviors on this subset of Atari games and the object collection level of DMLab. We highlight representation learning for these environments that could enable competitive performance across all Atari games and DMLab levels using Dreamer.
direction of future work
16

---

Published as a conference paper at ICLR 2020
D BEHAVIOR LEARNING
Acrobot Swingup
Caripole Balance
Curpple Balance Sparse
Cartpole Swingup
1000
1000
1000
1000
Egisonte Reharn
500
900
300
500
250
250
2:0
250
2
Cartpole Swingup Sparse
Cheetah Run
Cup Catch
Pinger Spin
1000
1000
1000
1000
Eptocde Reim
500
500
500
500
250
250
250
250
Finger Turn
Finger Turn Hard
Hopper Hop
Hopper Stand
1000
1000
1000
1000
Egtondi Retern
750
500
500
500
500
250
250
250
0
Pendulum Swingup
Quadruped Run
Quadruped Walk
Reacher Easy
1000
1000
1000
1000
Epivode Retur
750
750
500
500
500
500
250
0
0
Rencher Hard
Walker Run
Walker Stand
Walker Walk
Epriedee Reiura
750
750
750
750
500
500
500
500
0
Enviromment Steps
1c6
Envisonment Steps 166
Steps
lc6
Enstimement Steps
(1e9 steps)
stege, progeio)
steps)
Figure Comparison of action selection schemes on continuous control tasks of the DeepMind Control Suite from pixel inputs. The lines show mean scores over environment steps and the shaded areas show the standard deviation across 5 seeds. We compare Dreamer that learns both actions and values in imagination, only lcarning actions in imagination, and Planet that selects actions by online planning instead of learning a policy. The baselines include the top model-free algorithm D4PG, the well- A3C agent, and the hybrid SLAC agent.
17

---

Published as conference paper at ICLR 2020
E REPRESENTATION LEARNING
Acrobot Swingup
Curppole Balance
Curpple Balance Sparse
Cartpole Swingup
1000
1000
1000
1000
Egisnde
500
900
300
500
250
250
250
250
2
Cartpole Swingup Sparse
Cheetah Run
Cup Catch
Pinger Spin
1000
1000
1000
1000
Epraode Remum
500
500
500
500
250
250
250
250
Finger Turn
Finger Turn Hard
Hopper Hop
Hopper Stand
1000
1000
1000
1000
Epiiok Reim
750
500
500
500
500
250
250
0
Pendulum Swingup
Quadruped Run
Quadruped Walk
Reacher Easy
1000
1000
1000
1000
Episok Retum
750
750
500
500
500
500
250
0
Reacher Hard
Walker Run
Walker Stand
Walker Walk
1000
1000
1000
1000
750
750
750
750
Episnds
500
500
500
500
0
Stops 1c6
Steps 166
Steps lc6
Steps
Dreamer Recoustmecton
Dreamer
Dreamer Reward
(1c9 steps)
A3C (1e9 steps, propdo)
Figure 11: Comparison of representation learning methods for Dreamer. The lines show mean scores and the shaded areas show the standard deviation across 5 seeds. We compare generating both images and rewards, generating rewards and using a contrastive loss to learn about the images, and only predicting rewards. Image reconstruction provides the best learning signal across most of the tasks, followed by the contrastive objective. Learning purely from rewards was not sufficient in our experiments and might require larger amounts of experience.
18

---

Published as conference paper at ICLR 2020
F ACTION REPEAT
Acrobot Swingup
Curpole Balance
Carppoee Balance Sparse
Carpple Swingup
Egtooolc Rotum
800
800
800
X00
400
400
400
400
04
0.6
0.0
0.2
0.8
1.0
0.4
0.6
0.0
0.2
0.8
1.0
Cartpole Swingup Sparse
Cheetah
Cup Catch
Finger Spin
1000
1000
1000
1000
800
800
600
600
600
Egtends
400
200
200
200
200
0.0
Finger Tum Easy
Finger Turn Hard
Hopper Hop
Hopper Stand
1000
1000
1000
1000
Egisnnle Rehen
600
600
600
600
200
200
200
200
0.0
0.2
0.1
0.6
0.0
0.2
0.6
0.8
1.0
0.2
0.1
0.6
0.0
0.2
0.1
0.6
0.8
1.0
Pendulum Swingup
Quadruped Run
Quadruped Walk
Reacher Easy
1000
1000
1000
1000
Ephocie Beturn
600
600
600
600
400
400
400
200
200
200
200
0.0
0.2
0.1
0.6
0.8
1.0
0.0
0.2
0.1
0.6
0.8
1.0
0.0
0.2
0.1
0.6
08
1.0
0.0
0.2
0.1
0.6
0.8
1.0
Reacher Hard
Walker Run
Walker Stand
Walker Walk
1000
1000
1000
1000
Epioodd Retum
800
800
800
800
400
400
400
400
200
0.2
0.1
0.6
0.0
0.2
0.8
1.0
0.4
0.6
0.8
0.0
0.2
0.1
0.6
0.8
1.0
Environment Steps
Environment Steps
Environment Steps
Environment Stops
Repeat
Repeat 2
Repeat 4
A3C (1c9 steps, proprio)
D4PG (1c9 steps)
PlaNet (te6steps)
SLAC (3e6 steps)
Robustness of Dreamer different control frequencies. Reinforcement learning methods can be sensitive to this hyper parameter, which could be amplified when learning dynamics models the control frequency of the environment. For this experiment, amounts of action repeat. The areas show one standard deviation across seeds. We used a previous hyper parameter setting for this experiment. We find that a value of R works best across tasks.
Figure 12:
train Dreamer with different
19

---

Published as a conference paper at ICLR 2020
G CONTINUOUS CONTROL SCORES
A3C
D4PG
PlaNet!
Dreamer
Modality
proprio
pixels 109
pixels
pixels
Steps
109
5 x 10
5 10
Acrobot Swingup
41.90
91.70
3.21
365.26
Cartpole Balance
951.60
992.80
452.56
979.56
Cartpole Balance Sparse
857.40
1000.00
164.74
941.84
Cartpole Swingup
558.40
862.00
312.56
833.66
Cartpole Swingup Sparse
179.80
482.00
0.64
812.22
Cheetah Run
213.90
523.80
496.12
894.56
Cup Catch
104.70
980.50
455.98
962.48
Finger Spin
129.40
985.70
495.25
498.88
Finger Turn Easy
167.30
971.40
451.22
825.86
Finger Turn Hard
88.70
966.00
312.55
891.38
Hopper Hop
0.50
242.00
0.37
368.97
Hopper Stand
27.90
929.90
5.96
923.72
Pendulum Swingup
48.60
680.90
3.27
833.00
Quadruped Run
280.45
888.39
Quadruped Walk
238.90
931.61
Reacher Easy
95.60
967.40
468.50
935.08
Reacher Hard
39.70
957.10
187.02
817.05
Walker Run
191.80
567.20
626.25
824.67
Walker Stand
378.40
985.20
759.19
977.99
Walker Walk
311.00
968.30
944.70
961.67
Average
243.70
786.32
332.97
823.39
We re-run PlaNet with fixed action repeat of R 2 to not tune the this value for each of the 20 tasks. As result, the scores differ from Hafner al. (2018).
20


Reviews:
[{"review_id": "r1lsMBCdFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work is clearly the work of a large team. the paper clearly defines what is being done. I have spent a lot of effort with MCTS. I can not find the corresponding allowance for stochastic jumps in the latent space long horizon learning. \n\nYou have the phrase \"allowing to imagine thousands of trajectories in parallel\". I would like some elaboration on this. I think you have ideas of what is happening in the latent space that I am not following. \n\nYou are heavy on the machinery and math. I find the learning in the latent space the important part and there are things like how much simulation is done in the latent learning not clearly spelled out. How does the effort compare to the 1E9 steps of the base line your refer to? \n\nYour team is highly competent your style is distinct. Now may be the time to move you to understanding what structures get learned in latent space, are the in fact compact, diverse?\n\nPerhaps there is room for memory/memories in the latent space? \n\nMassive effort, nice results. Now for learning on our part (the humans). ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "r1lYlete9S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a world model-based approach in which behaviours are optimised by rollouts (i.e. imagination) in latent space. The paper achieves impressive results across a large selection of tasks, both in terms of sample efficiency and final performance.\n\nI found the paper interesting to read and well written. The main contribution (backpropagating analytic gradients through imagined trajectories?) could potentially be highlighted more but otherwise the paper was clear. I wonder if the authors ever looked at how much the size of the latent vector determines the performance of the system? Is there an optional latent vector size across domains or is that optimal size task dependent?  \n\nAdditionally, how much variance is there in the imagined trajectories from a certain starting state? In other words, are the endpoints of most imagined trajectories similar or very different? \n\nThere is actually not too much for me to critique and I would suggest this paper should be accepted. \n\n\nMinor comment:\n- On page 2 it says “We approach this limitation in latenby”, which I assume is a typo? \n\n####After rebuttal####\nThe authors' response addressed my remaining questions. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "SJeRQb-oFH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper summary.\nThe paper proposes Dreamer, a model-based RL method for high-dimensional inputs such as images. The main novelty in Dreamer is to learn a policy function from latent representation-and-transition models in an end-to-end manner. Specifically, Dreamer is an actor-critic method that learns an optimal policy by backpropagating re-parameterized gradients through a value function, a latent transition model, and a latent representation model. This is unlike existing methods which use model-free or planning methods on simulated trajectories to learn the optimal policy. Meanwhile, Dreamer learns the remaining components, namely a value function, a latent transition model, and a latent representation model, based on existing methods (the world models and PlaNet). Experiments on a large set of continuous control tasks show that Dreamer outperforms existing model-based and model-free methods. \n\nComments. \nEfficiently learning a policy from visual inputs is an important research direction in RL. This paper takes a step in this direction by improving existing model-based methods (the world models and PlaNet) using the actor-critic approach. I am leaning towards weak accepting the paper. \n\nI am reluctant to give a higher score due to its incremental contribution. Specifically, the policy update in Dreamer resembles that of SVG (Heess et al., 2015), which also backpropagates re-parameterized gradients through a value function and a transition model. The main difference between Dreamer and SVG is that Dreamer incorporates a latent representation model. From this viewpoint, the actor-critic component in Dreamer is an incremental contribution. Since the latent models are learned based on existing techniques, the paper presents an incremental contribution. \n\nBesides the above comments, I have these additional comments. \n- Effectiveness on very long horizon trajectories: \nSimulating long-horizon trajectories with a probabilistic model is known to be unsuitable for model-based RL due to accumulated errors. This is an open issue in model-based RL. The paper attempts to solve this issue by backpropagating policy gradients through the transition model, which is known to be more robust against model errors (see e.g., PILCO (Deisenroth et al., 2011)). However, the issue still exists in Dreamer, since there seems to be an upper limit of effective horizon length (perhaps around 40, according to Figure 4). This horizon length is still short compared to the entire horizon length of many MDPs (e.g., 1000). I think this point should be discussed in the paper. That is, the issue still exists, and Dreamer is less effective with very long horizon.\n\n- Inapplicability to discrete controls: \nOne restriction of re-parameterized gradients is that the technique is not applicable to discrete random variables. This restriction exists in Dreamer, and the method cannot be applied to discrete control tasks unless approximation techniques such as Gumbel-softmax are used. Still, such approximations would make learning more challenging, especially with long-horizon backpropagation. This restriction should be noted in the paper. \n\n- There is no mention about variance of policy gradient estimates. Dreamer does not use any variance reduction technique, so the gradient estimates could have very large variance. \n\n- q_theta was introduced in Eq. (8) before it is defined in Eq. (11). Also, I suggest moving Section 4 to be right after Section 2, since Section 4 presents existing techniques similarly to Section 2, while Section 3 presents the main contribution. \n\n\nUpdate after authors' response.\nI read the response. The paper is more clear after authors' clarification. Though, I still think the contribution is incremental, since back-propagating gradients through values and dynamics has been studied in prior works (albeit with less empirical successes compared to Dreamer). Nonetheless, I am keen to acceptance. I would increase the rating from 6 to 7, but I will keep the rating of 6 since the rating of 7 is not possible.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1xSQUW2tS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduced a latent space model for reinforcement learning in vision-based control tasks. It first learns a latent dynamics model, in which the transition model and the reward model can be learned on the latent state representations. Using the learned latent state representations, it used an actor-critic model to learn a reactive policy to optimize the agent's behaviors in long-horizon continuous control tasks. The method is applied to vision-based continuous control in 20 tasks in the Deepmind control suite. \n\nPros:\n1. The method used a latent dynamics model, which avoids reconstruction of the future images during inference.\n2. The learned actor-critic model replaced online planning, where actions can be evaluated in a more efficient manner.\n3. The model achieved better performances in challenging control tasks compared to previous latent space planning methods, such as PlaNet.\n\nCons:\n1. The work has limited novelty: the learning of the world model (recurrent state-space model) closely follows the prior work of PlaNet. In contrast to PlaNet, the difference is that this work learns an actor-critic model in place of online planning with the cross entropy method. However, I found the contribution of the actor-critic model is insufficient and requires additional experimental validation (see below).\n\n2. Since the actor-critic model is the novel component in this model (propagating gradients through the learned dynamics), I would like to see additional analysis and baseline comparisons of this method to previous actor-critic policy learning methods, such as DDPG and SAC training on the (fixed) latent state representations, and recent work of MVE or STEVE that use the learned dynamics to accelerate policy learning with multi-step updates.\n\n3. The world model is fixed while learning the action and value models, meaning that reinforcement learning of the actor-critic model cannot be used to improve the latent state model. It'd be interesting to see how optimization of the actions would lead to better state representations by propagating gradients from the actor-critic model to the world model.\n\nTypos:\nReward prediction along --> Reward prediction alone\nthis limitation in latenby?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an approach to model-based reinforcement learning in high-dimensional tasks. The approach involves learning a latent dynamics model, and performing rollouts thereof with an actor-critic model to learn behaviours. This is extensively evaluated on 20 visual control tasks.

This paper was favourably received, but there were concerns around it being incremental (relative to PlaNet and SVG). The authors highlighted the differences in the rebuttal, clarifying the novelty of this work. 

Given the interesting ideas presented, and the convincing results, this paper should be accepted.
Decision: Accept (Spotlight)