Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Paper Summary:

This paper performs a detailed ablation study over different mechanisms for predicting dynamics for model-based control. The paper proposes its own metrics for models, evaluates how different types of uncertainty impact predictions, and measures control performance with random shooting MPC. By implementing a new hyper parameter schedule, the paper shows new SOTA performance (in terms of sample efficiency) on the acrobat task.

----- 

Rebuttal phase:
I am updating my score based on the numerous clarifications of experiments and improvements of framing the paper.
-----


Score Summary:

The analysis in this paper is very warranted, but the methodology executed raises many questions as to if the claims will be generalizable. The paper evaluates models on only one task and on their own metrics, so there is difficulty in matching the paper to the literature. This difficulty of placement for a paper making broad claims, leads to a currently unpublishable form.



-----


Comments:
0) This type of work is absolutely critical to advancing MBRL. I have a lot of criticisms below, but I want to say that I think this could be a very good paper when all is said and done. The metrics proposed are relevant and the breadth of models tested is encouraging.

1) the authors refer to micro-data in the title and multiple times in the introduction, but never define this. What regime of data do the authors consider micro data? Is it a cap at 1000 points or depend on the environment?
1b) Where did the number of training points at 5000 come from in 2.3?

2)The authors make some strange claims in the paper, that vary from interesting to unsupported and too strong. Many of these claims need to be backed by citations.
2a) "Unlike computers, physical systems do not get faster with time" is playing loose and fast with the fact that the computational power of computers is increasing rapidly in recent years. This does not really have much of a baring on the rest of the paper and should be tied in better.

3) The evaluation of only fixed, random-shooting MPC is okay, but limited. For a paper that is titled "what are the critical model properties to choose" I would like to see more acknowledgement that the model one chooses is heavily dependent on the method it is being used for. 
3a) PETS (Chua et. al 2018) as a paper is cited, can the authors comment why the trajectory propagation method or an informed sampling like CEM is not considered? For example, in appendix A.8, the PETS paper shows performance is dramatically better with CEM, and other methods such as POPLIN (Wang et. al 2019) and PDDM (Nagabandi et. Al 2019) have built on this, it seems like a step back to not consider these advances.

4) The model hyper-parameters section is hard to read (A.D) and is crucial to being able to trust the results. What search method was performed over the ranges given per model? And, how extensive was said search?
4b) The authors claim "system modeling for MBRL is essentially a supervised learning problem with AutoML." To my knowledge there are no detailed examples of MBRL using AutoML?

5) the paper claims SOTA sample efficiency on acrobat, but I am not sure. Can the authors provide more information. For example, I am looking at a MBRL benchmarking paper and the reward takes on a vastly different range (https://arxiv.org/pdf/1907.02057v1.pdf). Acrobat is not the most well studied environment in recent years, so this discrepancy is hard to reason with. 

6) While the model requirements and evaluation metrics are insightful, it could be useful to distill them into slightly more compact analysis. What are the most important takeaways for each is interesting, but more useful is "which type of model does each person want." It seems unlikely that every problem should use the same model.

7) The number of action samples used in the random shooting is extremely low at n=100 sequences. Was this validated against? It seems at least an order of magnitude lower than many other papers in the area.

8) In figure 1, the paper shows "how different model types deal with uncertainty" which types of models were actually used, or is this an illustration?

9) The claim that random-shooting on the real environment is optimal control on the real environment is very troubling (especially with the number of action samples). Random shooting is only guaranteed to be optimal when the number of samples goes to infinity. If this metric was meant as "matches the state of the art for control of the acrobat" this should be clarified.

10) A more complicated environment is needed to make many of these claims more believable. Were any experiments done on hopper or half cheetah? 

11) What does "starting from an approximately hanging position" mean for acrobat, and is that the standard used in other control papers? Please classify this numerically.

12) the claim that model accuracy is the bottleneck of MBRL should be cited in the introduction. I think there is some preliminary work showing that direction, but sampling based control is also sub-optimal.

13) Are certain requirements correlated with certain metrics? This would be an interesting analysis, but there wasn't much of it in the paper (yes low on space, I know :/)

14) was the likelihood bound of 1.47E-6 taken from somewhere or how was this number computed?

15) The models and results section was confusing. I think the reader would benefit from a discussion of the results. There are interesting findings, such as what the authors discussed on training with uncertainty, that are not always super clear.

-----

Minor Points:
- Paragraph 2 of second page, "MBLR"
- inconsistency in hyphenating one-step "one step ahead"
- "Robust and computationally efficient probabilistic generative models are the crux of many machine" needs some commas or punctuation.
- Fragment section 2 "So our goal is to learn p()"
- Section 3 starts with "A commonly held belief," just say what the results show.
- Footnote 2, I agree it is annoying at times, but most papers report the trial length, no?
- The use of our in the introduction is slightly strange
- Generally accepted use fo quotes at the end of a sentence is "hi."
- The analogy to Yann Lecun is strange and un-cited.
review_title: Interesting methods proposed to evaluate dynamics models, but confusing paper and not enough evaluation

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper discusses the quality of surrogate models for model-based RL.  The two main contributions are first: defining the a set of evaluation metrics + experimental design and secondly performing a comparative analysis over existing methods.

There are several things I like about this paper:
 
- There is lack of existing work in throroughly comparing surrogate models in model-based RL
- The authors do a great job defining and arguing for evaluation metrics  (Section 2 is excellent)
- Defining requirements to the model that are falsifiable (2.1) are good to augment quantitative comparisons with qualitative ones

On the other hand there a several things I think that could be improved:

- Introduction:  The introduction  was quite confusing: Not only was it intermixed with conclusions that seem out of place. Also the discussion about uncertainty/probability was unclear.  Especially when talking about both uncertainty in a Bayesian sense (uncertainty over parameters or functions, epistemic) and "randomness" (stochastic effects of the environment) it is very easy
to be confusing.  E.g  "Probabilistic models are needed when the system benefits from multimodal predictive uncertainty."  It is not clear here what you mean by "probabilistic":  Parametric uncertainty (like in GPs) or models that can model multimodalities (like MDN)?

- Relevant work:  There is relevant work missing here.  Methods dealing with Bayesian deep learning[1,2], contextual auto-encoders[3]. 

   - [1] Gal, Yarin, Rowan McAllister, and Carl Edward Rasmussen. "Improving PILCO with Bayesian neural network dynamics models." Data-Efficient Machine Learning workshop, ICML. Vol. 4. 2016
   - [2] Depeweg, Stefan, et al. "Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning." International Conference on Machine Learning. PMLR, 2018.
   -[3] Moerland, Thomas M., Joost Broekens, and Catholijn M. Jonker. "Learning multimodal transition dynamics for model-based reinforcement learning." arXiv preprint arXiv:1705.00470 (2017).


- Limited scope: Only considering a single benchmark (in the sense of a single environment) is too limited. Chances are the "surprising results" are due to the specifics of this particular problem.  In general:   Methods that can model stochastic effects (e.g. neural networks that also predict a heteroscedastic Gaussian noise) should perform well when there is true stochasticity in the benchmark. Bayesian approaches (that model some form of parametric/functional uncertainty) should perform well when data is limited. 

Overall I think this paper is written quite well,  deals with a topic that is relevant. While it is rigorous in terms of defining metrics and experimental set-up, it would greatly improve if additional benchmarks (and possibly methods) are considered.
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: This paper tackles a key problem in model-based RL that is to identify the essential properties a predictive model needs to have to achieve good control performance. They solve this problem by systematically accessing the performance of a family of autoregressive mixtures learned by deep neural nets (DARMDN) using a fixed control strategy (i.e., random shooting). An important finding from this investigation is that deterministic models, when trained with a likelihood score in a generative setup, has comparable performance with the probabilistic modes. Furthermore, they proposed several static metrics (LR, OR, R2, KS, long-horizon R2, and long-horizon KS) which can be used to assess the performance of the models before having to run long experiments on the dynamic systems. These metrics can help to debug the models when evaluated dimension-wise. They found out that R2 with a 10-steps horizon correlates the most with dynamic performance. The findings lead to state-of-art sample complexity on the Acrobot system by applying an aggressive training schedule. 

Strength: 
1) The paper is well-written and structured. 
2) The paper provides some good practices for evaluating model-based RL which will help to advance the field. 
Weakness: 
1) The paper only used one simple problem (Acrobot) and a random shooting control strategy to evaluate the different predictive models of the dynamic system, it'll be more interesting to see if the findings stills hold in more realistic and complex RL problems 

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Update: 
After going through all the discussion, I'd be happy to raise my score to 7. The authors did a great job in clarifying all the concerns raised by the reviewers. This make the paper a much stronger publication. 
review_title: A well written paper that provides great insights of essential properties of good models for model-based RL 

final_score: rating: 7: Good paper, accept
review_content: ##########################################################################

Summary:
This is a nice experimental paper on comparing various methods for MBRL. They listed properties that are desirable for MBRL, and defined multiple static metrics and dynamic metrics. They discussed popular methods within this framework and performed experiments in one task (two conditions) with the same control policy. Overall, the paper provides a good benchmark for MBRL. 

##########################################################################

Reasons for score: 
I think this paper provides a good benchmark and a good discussion and comparison between existing methods. This is important for future progress in MBRL. The code is promised to be publicly available, which I think it’s very crucial for an experimental paper. Bonus points if the authors can discuss how the results in this task might generalize to other different tasks (or actually performed some simple experiments). I think the current task and conditions do serve its purpose, but it just leaves the reader wondering about how the result might generalize. 
 
##########################################################################

Pros: 

1. The writing is very clear and easy to follow, also without redundancy. The paper is also well-structured. I particularly like that the authors clearly indicated which direction is better for the metrics in the tables. 
2. The seven properties and metrics are mostly well-motivated and well-defined. 
3. The authors discussed the results clearly with implications, not just stating which one is better, but also discussed when and why it is better.
4. The two important findings seem very interesting to me. I think the result of the necessity of probabilistic vs. deterministic models in different scenarios is a good contribution to this field.
5. The authors noted in the appendix that the code will be made public if accepted.
 
##########################################################################

Cons: 
1. The authors provided motivation to the metrics they proposed. It would be great to see a brief description of how different metrics related to each other, what (different) aspects they capture, and whether/how they correspond to the seven desirable properties. Also, it seems not sufficient enough to filter out methods just because of the impossibility of computing the defined metrics. 
2. Generalization to other tasks and different control policies. The authors did point this out as future work, but it’s also good to briefly discuss how it might or might not generalize.

#########################################################################

typo: 
Introduction 5th paragraph: MBLR -> MBRL 

review_title: A nice experimental papers

final_score: rating: 7: Good paper, accept
review_content: This paper analyses the random shooting control strategy in combination with
various generative models \( p(\by_{t+1} | \mathcal T_t) \), which model
observations \(\by\) conditioned on the history of observation-action pairs.
The authors select two variants of the Acrobot environment to make
requirements like multimodal posteriors more explicit. Both statistics on the
distribution associated with the generative model under a fixed policy and
reward-dependent metrics were defined and analysed for a range of models.

The paper's contribution are twofold. First, the suggested experimental
protocol for model evaluation and benchmarking extends the usual evaluation
process in reinforcement learning, which often focuses purely on the
cumulative reward. The framework described introduces a range of ``static''
likelihood-based metrics. These static metrics are evaluated under a fixed
(potentially stochastic) policy and allow the separation of the model-based
control strategy and the underlying model for evaluation purposes.
Reward-based ``dynamic'' metrics are evaluated under a random-shooting
control mechanism. This framework heavily simplifies model evaluation by
providing evaluation metrics and fixing the environment and control strategy.
Under these assumptions this approach allows direct comparison and even
visualisation of various quantities of interest, including trajectories of
the one-step forward model. Second, this work applies the conceived framework
to evaluate a range of models on two different environments. These
environments are intended to make requirements like probabilistic posteriors
explicit. The authors conclude by claiming that 1. ``Probabilistic models are
needed when the system benefits from multimodal predictive uncertainty'', and
2. Deterministic models are sufficient if trained ``with a loss allowing
heteroscedasticity''.

There's an inherent trade-off between simplicity of the study and generality
its conclusion. While some of the simplifying assumptions made make this kind
of study possible in the first place, they also raise a range of questions:
- How appropriate are these metrics for problems with higher observation
  spaces? Can we expect the variance of estimates to increase and ratios and
  likelihoods to diminish by multiple orders of magnitude?
- Do the claims presented as ``important findings'' generalise to other
  environments?
- Does the correlation of the explained variance with the dynamic metrics
  hold on other environments?

A small note: a clear definition of micro-data reinforcement learning is
missing, and MBRL is introduced twice with conflicting definitions.

The future directions outlined by the authors of extending the results to
larger systems and other planning strategies are very relevant to reduce the
concerns of generalisability of the results and applications to problems of
higher dimensions. At the same time these modification will increase the
experimental complexity. This paper serves as a suitable baseline for
reference for future work to answer these questions. Therefore I consider
this paper a valid contribution to ICLR.

review_title: Random shooting and generative models

Meta-review: **Overview** This paper performs detailed ablation studies over different dynamics prediction methods for MBRL. It proposes metrics for models to evaluate how different types of uncertainty impact predictions. The paper also measures control performance with random shooting MPC. The paper further implements a new hyper parameter schedule to achieve new SOTA performance on the acrobat task.

**Pro** 
- The paper is well-written.
- The analysis in this paper is very warranted. 
- The paper provides a very detailed ablation study.
- The authors do a great job defining and arguing for evaluation metrics.
- The seven properties and metrics are mostly well-motivated and well-defined.
- The authors discussed the results clearly with implications.
- The result of the necessity of probabilistic vs. deterministic models in different scenarios is a good contribution to this field.

**Con**
- The methodology might be hard generalizable, i.e., there is difficulty in matching the paper to the literature based on its own defined metric.
- The scope might be limited.

**Recommendation** The paper provides a significant contribution to MBRL by providing a detailed empirical study. During the rebuttal phase, the authors addresses many reviewers' concerns in a satisfactory way.  The paper is well-written and easy to read. The recommendation is an accept.

Decision: Accept (Poster)