Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: 
# Rebutal Respons:

Planning Horizon:
- I agree small horizons can speed up learning. However, if we want to drastically reduce the sample complexity, we need longer planning horizons. Therefore, we should be looking further in this direction but this is out of scope for this paper.

Model Learning:
- Good idea to update the conclusion and treat the online model learning as future work.

Figure 3:
You could plot axes a - f in a single row and remove the colorbar for each individual plot. Just do one colorbar for all plots. This would also improve the comparability of the different methods. Currently, the color-coding is different for each axis, which is bad practice. Axes g - i can be reshaped in a separate figure. Furthermore, figure 2 is not necessary as the 4 room environment is depicted in Fig. 3 a - f and and one could reference these axes. 

=> I keep my rating as weak accept.

# Review:
Summary:  
The paper proposes an adaptive scheme to adapt the horizon of the value function update. Therefore, the sample efficiency should be increased and the value function should be learned faster. 

Conclusion: 
The problem of learning good policies from partially correct models is very interesting and important. The proposed approach is technically sound and reasonable. The experiments highlight the qualitative as well as quantitative performance. Furthermore, the quantitative performance is compared to state-of-the-art methods. I cannot comment on the related work as I am not familiar with the baselines MVE & STEVE. 

My Main Concerns are:

- The roll-out horizon is 3-5 timesteps. This horizon is really short especially for problems with strong non-linearities and high sampling frequencies. For such systems 5 timesteps would only correspond to 0.5s (10Hz) or 0.05s (100Hz) and I am uncertain whether these short horizons really help for such problems. 

- The specialized online model learning algorithm seems quite hacky. It feels like it was introduced last minute to make it work. The overall question of how should we learn a model optimally is super important and should be addressed within a separate paper (and more thoroughly). I would even propose to remove the online learning section from this paper as it is just too hacky and without relation to prior work or context. 

- Could the authors please update figure 3 as the figure has too much whitespace. This whitespace could be used to enlarge the individual axis when the axis are rearranged.  

review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: #rebuttal responses
 
The authors' reply does not convince me, and I still think the paper has some problems:
(1) I do not believe that the cumulative model-error can not be learned efficiently;
(2) Experimental results are weak as some baselines do not converge! 

Thus I keep my rating as reject.

#review 
This paper proposes a new adaptive model-based value-expansion method, AdaMVE, that decides the planning horizon of the learned model by learning the model-error. The model-error is learned by temporal difference methods.
Experimental results show that AdaMVE beats MVE, STEVE, and DDPG in several environments.  

Overall the paper is well written.  The paper proposes an interesting question: how to adaptively change the planning horizon based on the state-dependent model-error? Firstly, The authors upper bound the cumulative target error by the cumulative model-error.  Then the cumulative model-error is learned by the temporal difference method. With the learned cumulative model-error function over different rollout steps, the planning horizon is decided by a softmax policy.  

However, I do not think that learning an upper bound of the target error helps to determine the value of H, as there is no justification that the gap between the target error and the model error is small theoretically.  I also doubt that the cumulative model-error can be learned without large loss, as there are no plots of W in this paper. 

The authors claim that it is expensive to retrain the model error for the current policy at every step, thus they use some reference policy. I think it is ok, but I want to see the results of AdaMVE using the updated current policy, or updating the Q function before improving the policy. Adding a figure showing the change of H in the training helps to motivate this paper.

Finally, AdaMVE is only compared in two MuJoCo environments. Baselines in other environments are only trained in 1e5 steps, thus the experimental results are not convincing.

I am happy to change my opinion on this paper if authors give better motivation and the detail of the learning.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: Learning to Combat Compounding-Error in Model-Based Reinforcement Learning

The authors study the problem of choosing the appropriate planning horizon in model-based reinforcement learning. We want to have a balance between the efficiency of planning into the future and while avoiding the compounding error brought by the learnt model.
And the authors propose to learn an error estimation model that is a function of current state, policy and horizon by estimating the compounding error as a new reward using policy evaluation.
Pros -
I think the novelty in this paper is most appealing for me to vote for a clear acceptance. The growth of model-based reinforcement learning is in a big need of a framework to work with the planning horizon.
The proposed method is a very elegant way of estimating the errors and choose the appropriate horizon.
Still, I think there is a lot of room for improvement, as summarized below.

Cons
- One thing that concerns me is that the maximum horizon H looks very small. A max horizon of 3, 5 or even 10 is still much smaller than the value we usually use in planning based reinforcement learning (MPC in your case). A common choice can be up to 30 or 50 for PETS based algorithms [1].

Does that indicate that the proposed framework is not stable or robust when the horizon is big? Or maybe that scale of horizon is not common in Steve and thus not tried in the experimental section?

- Experiments
For the high dimensional control tasks, only Half-Cheetah and Swimmer was tried. It will generalize the conclusion if the results of other benchmarking environments are tested, including complex tasks such as Humanoid, and easier tasks such as Cart-pole.

- Algorithms
The used baselines in the paper are generally not considered to be state-of-the-art. I guess it is sort-of fair comparison since you add the proposed component on top of the baselines. But extending the current state-of-the-art such as TD3 [2] / PETS

I also suggest adding this benchmarking paper into the related work, which empirically studies the compounding error in different state-of-the-art model-free and model-based reinforcement algorithms.

[1] Chua, K., Calandra, R., McAllister, R., & Levine, S. (2018). Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems (pp. 4754-4765).
[2] Fujimoto, Scott, Herke van Hoof, and David Meger. "Addressing function approximation error in actor-critic methods." arXiv preprint arXiv:1802.09477 (2018).
[3] Wang, Tingwu, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel and Jimmy Ba. “Benchmarking Model-Based Reinforcement Learning.” ArXiv abs/1907.02057 (2019): n. pag.
review_title: Official Blind Review #1

Meta-review: The paper received mixed reviews: R (R3), WA (R2), A (R1). AC has read the reviews, rebuttal and paper. AC is concerned about the short planning horizon, which seems like a major issue: (i) as R1 notes, most MPC algorithms use much longer horizons as they find it helps performance and (ii) the claim of the approach to be able to pick the planning horizon is moot if its dynamic range is small.  Overall, the paper is very borderline. The idea is interesting but without addressing longer horizons, the contribution is limited. Under guidance from the PCs, the AC feels that the paper just falls below the acceptance threshold and thus cannot be accepted unfortunately. The work is definitely interesting however and should be revised for a future submission. 


Decision: Reject