Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper presents a world model-based approach in which behaviours are optimised by rollouts (i.e. imagination) in latent space. The paper achieves impressive results across a large selection of tasks, both in terms of sample efficiency and final performance.

I found the paper interesting to read and well written. The main contribution (backpropagating analytic gradients through imagined trajectories?) could potentially be highlighted more but otherwise the paper was clear. I wonder if the authors ever looked at how much the size of the latent vector determines the performance of the system? Is there an optional latent vector size across domains or is that optimal size task dependent?  

Additionally, how much variance is there in the imagined trajectories from a certain starting state? In other words, are the endpoints of most imagined trajectories similar or very different? 

There is actually not too much for me to critique and I would suggest this paper should be accepted. 


Minor comment:
- On page 2 it says “We approach this limitation in latenby”, which I assume is a typo? 

####After rebuttal####
The authors' response addressed my remaining questions. 

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper introduced a latent space model for reinforcement learning in vision-based control tasks. It first learns a latent dynamics model, in which the transition model and the reward model can be learned on the latent state representations. Using the learned latent state representations, it used an actor-critic model to learn a reactive policy to optimize the agent's behaviors in long-horizon continuous control tasks. The method is applied to vision-based continuous control in 20 tasks in the Deepmind control suite. 

Pros:
1. The method used a latent dynamics model, which avoids reconstruction of the future images during inference.
2. The learned actor-critic model replaced online planning, where actions can be evaluated in a more efficient manner.
3. The model achieved better performances in challenging control tasks compared to previous latent space planning methods, such as PlaNet.

Cons:
1. The work has limited novelty: the learning of the world model (recurrent state-space model) closely follows the prior work of PlaNet. In contrast to PlaNet, the difference is that this work learns an actor-critic model in place of online planning with the cross entropy method. However, I found the contribution of the actor-critic model is insufficient and requires additional experimental validation (see below).

2. Since the actor-critic model is the novel component in this model (propagating gradients through the learned dynamics), I would like to see additional analysis and baseline comparisons of this method to previous actor-critic policy learning methods, such as DDPG and SAC training on the (fixed) latent state representations, and recent work of MVE or STEVE that use the learned dynamics to accelerate policy learning with multi-step updates.

3. The world model is fixed while learning the action and value models, meaning that reinforcement learning of the actor-critic model cannot be used to improve the latent state model. It'd be interesting to see how optimization of the actions would lead to better state representations by propagating gradients from the actor-critic model to the world model.

Typos:
Reward prediction along --> Reward prediction alone
this limitation in latenby?
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Paper summary.
The paper proposes Dreamer, a model-based RL method for high-dimensional inputs such as images. The main novelty in Dreamer is to learn a policy function from latent representation-and-transition models in an end-to-end manner. Specifically, Dreamer is an actor-critic method that learns an optimal policy by backpropagating re-parameterized gradients through a value function, a latent transition model, and a latent representation model. This is unlike existing methods which use model-free or planning methods on simulated trajectories to learn the optimal policy. Meanwhile, Dreamer learns the remaining components, namely a value function, a latent transition model, and a latent representation model, based on existing methods (the world models and PlaNet). Experiments on a large set of continuous control tasks show that Dreamer outperforms existing model-based and model-free methods. 

Comments. 
Efficiently learning a policy from visual inputs is an important research direction in RL. This paper takes a step in this direction by improving existing model-based methods (the world models and PlaNet) using the actor-critic approach. I am leaning towards weak accepting the paper. 

I am reluctant to give a higher score due to its incremental contribution. Specifically, the policy update in Dreamer resembles that of SVG (Heess et al., 2015), which also backpropagates re-parameterized gradients through a value function and a transition model. The main difference between Dreamer and SVG is that Dreamer incorporates a latent representation model. From this viewpoint, the actor-critic component in Dreamer is an incremental contribution. Since the latent models are learned based on existing techniques, the paper presents an incremental contribution. 

Besides the above comments, I have these additional comments. 
- Effectiveness on very long horizon trajectories: 
Simulating long-horizon trajectories with a probabilistic model is known to be unsuitable for model-based RL due to accumulated errors. This is an open issue in model-based RL. The paper attempts to solve this issue by backpropagating policy gradients through the transition model, which is known to be more robust against model errors (see e.g., PILCO (Deisenroth et al., 2011)). However, the issue still exists in Dreamer, since there seems to be an upper limit of effective horizon length (perhaps around 40, according to Figure 4). This horizon length is still short compared to the entire horizon length of many MDPs (e.g., 1000). I think this point should be discussed in the paper. That is, the issue still exists, and Dreamer is less effective with very long horizon.

- Inapplicability to discrete controls: 
One restriction of re-parameterized gradients is that the technique is not applicable to discrete random variables. This restriction exists in Dreamer, and the method cannot be applied to discrete control tasks unless approximation techniques such as Gumbel-softmax are used. Still, such approximations would make learning more challenging, especially with long-horizon backpropagation. This restriction should be noted in the paper. 

- There is no mention about variance of policy gradient estimates. Dreamer does not use any variance reduction technique, so the gradient estimates could have very large variance. 

- q_theta was introduced in Eq. (8) before it is defined in Eq. (11). Also, I suggest moving Section 4 to be right after Section 2, since Section 4 presents existing techniques similarly to Section 2, while Section 3 presents the main contribution. 


Update after authors' response.
I read the response. The paper is more clear after authors' clarification. Though, I still think the contribution is incremental, since back-propagating gradients through values and dynamics has been studied in prior works (albeit with less empirical successes compared to Dreamer). Nonetheless, I am keen to acceptance. I would increase the rating from 6 to 7, but I will keep the rating of 6 since the rating of 7 is not possible.
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: This work is clearly the work of a large team. the paper clearly defines what is being done. I have spent a lot of effort with MCTS. I can not find the corresponding allowance for stochastic jumps in the latent space long horizon learning. 

You have the phrase "allowing to imagine thousands of trajectories in parallel". I would like some elaboration on this. I think you have ideas of what is happening in the latent space that I am not following. 

You are heavy on the machinery and math. I find the learning in the latent space the important part and there are things like how much simulation is done in the latent learning not clearly spelled out. How does the effort compare to the 1E9 steps of the base line your refer to? 

Your team is highly competent your style is distinct. Now may be the time to move you to understanding what structures get learned in latent space, are the in fact compact, diverse?

Perhaps there is room for memory/memories in the latent space? 

Massive effort, nice results. Now for learning on our part (the humans). 
review_title: Official Blind Review #2

final_score: rating: null
review_content: In performance comparison you mention 2 million environment steps. I can not tell how many latent imagined steps if any? Is it zero? Or many?

You talk about representation learning, I think there are several ways to think about representation. There is the way Paul Cisek of The University of Montreal talks about feedback loops that maintain homeostasis via interaction with the environment. These have no explicit representation they are simple their own meaning based on their outcome. Then there is abstract thinking with internal visualization, planning, goal setting. humm maybe abstract representations are just more [across several variables/modes] of the same. 

Can your agent learn compact representations of long horizon latent trajectories? Does it just memorize long paths?

Is there a memory? I am guessing no. Should there be a memory? How much memory is needed I wonder. Your learned  long paths do they just exist as vague imprints in the weights? Can they be made more explicit? 

It is nice to read a paper where the author have a good command of English.  
review_title: data efficiency 

Meta-review: This paper presents an approach to model-based reinforcement learning in high-dimensional tasks. The approach involves learning a latent dynamics model, and performing rollouts thereof with an actor-critic model to learn behaviours. This is extensively evaluated on 20 visual control tasks.

This paper was favourably received, but there were concerns around it being incremental (relative to PlaNet and SVG). The authors highlighted the differences in the rebuttal, clarifying the novelty of this work. 

Given the interesting ideas presented, and the convincing results, this paper should be accepted.
Decision: Accept (Spotlight)