Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Summary:
This paper proposes a novel algorithm for planning on specific domains through latent reward prediction. The proposed model uses an encoder to learn embedding the state to the latent state, a forward dynamics function to learn dynamical system in latent state space, and a reward function to estimate the reward given a latent state and an action. Using these functions, the authors define the objective using the mean-squared error between true and multi-step prediction of rewards. To justify the proposed method, the authors provide a theoretical analysis and experimental results on specific RL domains, multi-pendulum and multi-cheetah, which contain irrelevant aspects of the state.

Comments:
This paper is well-written and easy to understand.
- In this paper, the authors assume deterministic transition and use deterministic function for latent transition. It seems to be the authors want to use MPC, which is a powerful planning algorithm. However, many RL tasks are modeled with stochastic transition. In stochastic transition cases, is the proposed algorithm still valid?
- As shown in Figure 3, even proposed method shows better performance than SAC in early episode but table 1 says that SAC shows the best convergence results in any number of pendulums except the single pendulum case. It seems to be different results from intuition, because the authors emphasize that the strength of the proposed method is efficiency of learning in RL tasks with irrelevant information. 

Questions and minor comments:
- What objective is used to learn the latent model of the state-prediction model algorithm? 
- Providing detailed experimental settings, like detailed settings for three deterministic feed-forward neural networks, and results such as consumed CPU time will help the comparison algorithms.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper presents a technique for model based RL/planning with latent dynamics models, which learns the latent model only using reward prediction. This is in contrast to existing work which generally use a combination of reward prediction and state reconstruction to learn the latent model. The paper suggests that by removing the state reconstruction loss, the agent can learn to ignore irrelevant parts of the state, which should enable better performance in settings where state reconstruction is challenging. 

Overall the motivation for this work is good, and the idea is promising. Difficulty in reconstructing high dimensional states is a challenge for learning latent dynamics models. The paper is also very well written and easy to follow.

My concerns are centered around the experimental evaluation. Specifically, I see the following issues: (1) the experimental environments seem artificial, and hand tailored for this method, (2) given that the proposed method is a minor modification to the PlaNet paper, it seems that PlaNet should be included as a comparison (especially because it has been shown to work on high dimensional states), and (3) the proposed method seems very prone to overfitting to the given task, and there should be an analysis of how the proposed change affects generalization and robustness.

(1): The testing environments contain many distractor pendulums/cheetahs, which makes state reconstruction especially challenging. While this does seem to be the point the authors are trying to show, the environments are an extreme, almost artificial, case of difficult state reconstruction. Would the same results hold in more realistic settings, for example, visual robot manipulation in a cluttered scene? Model based RL with video prediction models has been shown to work in such real cluttered robot manipulation environments. Showing that the proposed method can outperform such approaches in robot manipulation settings would be a powerful result. The results on images in the appendix seem to show a delta between the true and predicted reward, suggesting that the proposed method does not yet work on images. Why might this be the case?

(2): From what I can see, the proposed method is very similar to the PlaNet algorithm with state reconstruction loss removed. Given the similarity, PlaNet should be included as a comparison in both the pendulum and cheetah environments. Similarly why was DeepMDP performance not shown in the Cheetah environment?

(3): One of the strengths of model based reinforcement learning is the ability to plan to reach unseen goals with a model trained via self-supervision or different goals. Does the proposed approach lose some of this, by overfitting to only the task reward? I suspect that in generalizing to unseen tasks, a model trained with state prediction would potentially perform much better. If trained on many tasks, could this method achieve similar generalization? 

Due to some of these questions which remain unanswered by the experimental evaluation my current rating is Weak Reject. If the authors are able to clarify some of the questions above I may adjust my score.

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper claims that one only needs a reward prediction model to learn a good latent representation for model-based reinforcement learning. They introduce a method that learns a latent dynamics model exclusively from multi-step reward prediction, then use MPC to plan directly in the latent space. They claim this is sample efficient in the model-based way, and is more useful than predicting full states. They learn a model that predicts only current and future rewards conditioned on action sequences, and that observation reconstruction is unnecessary to learn a good latent space. They provide planning performance guarantees for approximate latent reward prediction models. 

I tend to reject this work, because although I support the premise and believe it is very important, and like the style of experiments run with the use of distractors, I believe it is not impactful if only looking at the dense reward setting. The type of environment they describe that requires using lossy representations is also likely to only have sparse reward, so to only note in the conclusion as future work is not enough. The contributions consist only of learning a multi-step reward model for planning, and only provide results in two dense reward environments. In the second experiment with more difficult, high-dimensional observation and action space setting, two of the 3 baselines are left out, namely the state model and DeepMDP. I think it is crucial to include DeepMDP, as it is the one most likely to perform competitively with the proposed method. 

The justification for Table 1 vs. Figure 3 are also very unclear, as to why SAC is trained with 10^6 samples while DeepMDP is trained under a random policy, whereas the original paper utilizes samples collected by the policy as its trains. SAC is off policy, and can therefore be evaluated with random data, rather than being used as an "upper bound baseline". The final evaluation performance in dashed line in Figure 3 also doesn't include standard deviation across the 5 seeds, which it should. The final results for SAC also do not match the performance in Figure 3, although it is hard to tell since the final performance in Table 1 is written in terms of number of environment steps while Figure 3 the axis is in terms of episodes. 

Including sparse reward experiments would vastly help support the claims in the paper, as well as including the DeepMDP results for HalfCheetah and additional explanation of the difference in performance of SAC in Table 1 and Figure 3.
review_title: Official Blind Review #1

Meta-review: The authors propose a model-based RL algorithm, consisting of learning a                                                           
deterministic multi-step reward prediction model and a vanilla CEM-based MPC                                                       
actor.                                                                                                                             
In contrast to prior work, the model does not attempt to learn from observations                                                   
nor is a value function learned.                                                                                                   
The approach is tested on task from the mujoco control suit.                                                                       
                                                                                                                                   
The paper is below acceptance threshold.                                                                                           
It is a variation on previous work form Hafner et al.                                                                              
Furthermore, I think the approach is fundamentally limited: All the learning                                                       
derives from the immediate, dense reward signal, whereas the main challenges in RL                                                 
are found in sparse reward settings that require planning over long horizons, where value                                          
functions or similar methods to assign credit over long time windows are                                                           
absolutely essential.
Decision: Reject