Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors compare a number of recent and commonly used learning strategies for model-based planning on four typical RL control environments. The findings provides some useful insights for practitioners as well as for future research directions.
main_review: I enjoyed reading this paper. The main weakness are that only four benchmarks were used to arrive at these conclusions, and some elements of the methodology are not clear. 

I have some questions:
1) Data Used: If I understand sec 4.1 correctly, you are actually using fixed data sets to evaluate the rewards of MPC. While I understand you intentionally did this to ease comparison, you remove a key factor exploration of MBRL: exploration. This weakens your comparisons for stochastic models in particular, since the uncertainty of a model can interact with the planning algorithm to learn more efficiently (or not). I also do not really understand how this works, what are the "Updates" on the x-axis in figures, number of data points accessed from this fixed data set? I think it's important to be clear here, there is no connection with what the planner explores? 

2) Are you drawing enough samples for the stochastic models? Stochastic models are very vulnerable to drawing too few samples. Some data on this would be useful. 

3) Multi-step: Fig. 3 is curious, why are the results for deterministic, steps = 1 are worse (e.g. finger-spin, ball-in-cup) than those for deterministic 1-step prediction in Fig. 2? Are these 95% CI?

4) The multi-step results were the most surprising to me, the optimal number of steps also seems to vary considerably between tasks, from 5-40 steps. Any insight why this happens?

5) Ensembles in Fig. 4: Again, is not 1 ensemble the same as a regular NN in earlier figures? Finally, since 10 ensembles performs betst, what would happen if you used 15? Additionally, you basically get more samples (and uncertainty) by adding ensembles, so there could be a sampling factor impact results here as well.

6) Input Noise: Is this for the planner or the model learning? If you use a fixed data set, I don't see how this could be for the model? If it's for the planner, it seems more like state noise than *observation* noise (c.f. section title) since the future states will all be affected? Finally, shouldn't ball-in-cup noise=0 match earlier figures for stochastic model?

Minor:

- The figures are displaced 1-2 pages from the text, which is bit confusing.

- "The obtained reward is marginally below the reward obtained by planning with the true model." - The gap in mean reward seems to be >10% in most of these example, which is at least mathematically not marginal, even though their behavior may visually seem very close.

- Normalized MSE vs. standardization, did you try with state and actions that weren't centered around zero in your benchmarks? What you are doing here is basically the variance compensation of standardization, but not the mean one.

- Why in-text [Link] tags instead of /url clickable footnotes?

- It might be interesting to see if these model-based approaches actually could converge to near-optimal rewards if you let them run longers. 

- Note: "Instead of using a deterministic model, Chua et al. (2018) proposed to use stochastic models. This
model predicts the change of the system using a normal distribution.". I would phrase it as "learns the variance of the predictions". MSE is of course mathematically identical to neg log-likelihood for Gaussian noise with isotropic covariance, it's just that the variance is never estimated (because parameters are invariant of this). What is new here is learning state-dependent noise, presumably modelled as an extra output of the network (often called aleatoric uncertainty as you note). I am not entirely sure what you mean by the lower entropy bound, perhaps just that the planner will tend to avoid poorly modelled parts of the state space since the noise is presumably higher?

- Typos: "rollouts of the provide", "significant more data", " the modeling the "

summary_of_the_review: While some design choices made in the study could be clearer, and I would have preferred to have more benchmark environments, the paper is well written and the experiments appear carefully designed. I believe ablative studies like this are very useful for the research community and practitioners alike.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigates different choices of designing the planning model which predict the next state on the observation space. The paper compares (1) prediction with deterministic or stochastic models, (2) using 1-step forward prediction loss or multi-step forward prediction loss to train the model, (3) prediction with single network or network ensemble, and (4) using perfect observation or add gaussian noise to the observation. 
Through the above experiments, the paper suggests that deterministic models should be trained with multi-step loss, while stochastic models work better when the 1-step log-likelihood loss is used and noise is added to the observation space. In both cases, ensemble networks tend to show better performance.

main_review: The paper shows the following advantage:

(+) This work tries to provide the community a deeper understanding of the predictive planning model on the observation space, by comparing several commonly used model designing choices and providing a conclusion on the choices that generally work better. The comparison takes the network structure, loss function, and stochasticity of models into consideration.

In the meanwhile, there remain several concerns.

(-) The stochastic and deterministic models are both applied to deterministic environments, while the stochastic model may lose part of its advantage in this case. Considering the comparison between the stochastic and deterministic model is one of the important experiments in this work, it may be worth classifying the environments into 2 corresponding types, then looking into how stochastic and deterministic models perform in each of them. It can also verify whether the current conclusion still holds in stochastic environments.

(-) Another concern is about the random seeds. There are 5 random seeds in total. The average and confidence interval based on 5 random seeds may not be accurate enough, therefore it is hard to justify whether some conclusions are valid or not. For example, in Section 4.4 and Figure 4, the result indicates that network ensemble improves the performance more significantly in stochastic models, while in the deterministic model, ensemble only matters in Ball-in-cup. However, when reading the plots, both Finger and Walker have overlapped areas when indicating the 20/80 percentile. It is not clear to me if the above conclusion still holds when adding more random seeds. As a paper empirically compares different model designing choices, I believe showing reliable averaged performance is important. Thus, using more random seeds (ideally above 10) might make the conclusion more clear.

(-) It is mentioned in the paper that the experimental results obtained in this work are partially inconsistent with previous work (Chua et al. 2018), however, there is a lack of discussion on what may cause this inconsistency, which makes it hard to justify whether the result provided in this work is reliable or not.

(-) When comparing the stochastic and deterministic models, the networks have different structures. It causes differences including capacities and numbers of parameters to train. These differences can affect the learning efficiency of the agent. Thus it is not clear to me whether the conclusion about deterministic and stochastic models is accurate.

summary_of_the_review: Overall, I think the paper is not good enough to get accepted yet. On one hand, the paper focuses on understanding and comparing different settings of designing predictive models in the literature, which offers the community a better understanding of planning. On the other hand, there remain several concerns. My major concern is that the experiment setting may affect the accuracy of the conclusion provided in the paper. Focusing too much on deterministic environments makes me wonder if the conclusion still holds when there exists stochasticity in the environment. This may cause an unfair comparison between the deterministic model and stochastic model, which is a major part of this work. Furthermore, the overlapping area when showing the 20/80 percentile suggests more than 5 random seeds are needed. Moreover, using different network architectures can affect the learning efficiency thus causing inaccuracy when comparing these models.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work ablates some of the design choices that go into learning a dynamics model for control-based environments. They ablate 4 choices: use of deterministic vs. stochastic models, multistep losses, network ensembles, and input noise. The authors study this in a few of the DeepMind control suite environments.
main_review: Strengths: 
* Interesting problem setting of understanding the role of design choices for model learning in model-based RL.

Weaknesses:
* Takeaways from the paper don’t seem to be conclusive (rf. “No single design choice works significantly better”). It doesn’t satisfactorily answer the question the work sets out. It might be helpful to frame the paper as informing future researchers who want to learn accurate dynamics models: How can this work help inform them which methods to try out vs. be eliminated?
* Investigation is limited in terms of environments explored. Specifically, the study only focuses on four control environments which are not very representative of settings where you would want to do model-based RL (e.g. the related works section motivates the use of accurate model prediction for robotics, but only applies it to simulated, deterministic environments). I would recommend the authors to extending their analysis in a robotics-based environment, e.g. RoboSuite [1], PlayRoom [2], Meta-World [3].
* Investigation is limited in terms of design choices explored. For example, memory-based architectures are omitted, despite playing a key role in domains that are partially observable (e.g. velocity of agents cannot be inferred from their setup which makes model-learning hard). It would be nice if their analysis could be expanded to cover memory-based architectures. 

[1] robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
[2] Learning Latent Plans from Play
[3] Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
summary_of_the_review: --- methodology ---

I find the question of the paper interesting, but the methodology doesn’t feel satisfactory to me. A few things appear odd to me:
* Running a stochastic model in a deterministic setting seems like a weird setting for evaluating the benefits of stochastic models. Feels like an unfair comparison. Could the authors try running this on environments that elicit stochasticity, and where this choice might seem more natural? 
* The setup of the paper seems odd: The data collection assumes that we already have a well-trained model-free agent that already does the task we care about...We then learn a dynamics model from this policy and deploy the model-based version. It seems that this setup doesn’t even need a model-based method in the first place. So why should we care about these domains that the authors evaluate on? Plus, the most challenging domain - humanoid - doesn’t seem to lend itself to model analysis in this work.

--- paper narrative & motivation --- 

In some sense, the takeaway of the paper feels negative to me: there isn’t really a consistent way of learning accurate / useful dynamics models, so future researchers might just need to empirically test all the design choices for themselves and see what works best. I wish the work could do more to explore a bit more into what kinds of errors these dynamics models incur, and why these approximation errors are fatal - e.g. humanoid completely fails and so all of its results are omitted, but … why are those model errors that bad compared to other domains? How can this failure case inform us in how to better learn dynamics models? 

There are several areas in the paper that come unmotivated or are not well contextualized. For example, I don’t really understand the dynamics manifold argument in Section 4.5 when the authors say that long time horizon predictions lead to inputs being outside the desired manifold, yet claim that adding noise also “causes the inputs to lie outside the manifold” but the model is more robust to predict solutions returning to the manifold. It seems like noise will only, up to a certain extent, help you in the short run but long horizon settings will still be difficult to predict accurately. 

Why would we consider having a model-based method in the first place? It would have been nice to see explicit sample gains in having model-based methods - e.g. number of data samples saved for these settings [or considering some form of computational budget]. 

The work also seems to assume the best case scenario of having an optimal (model-free) policy which is oftentimes not available. It would have been interesting to investigate the approximation tradeoffs when a well-trained policy is not available as a data source for the to-be-learned dynamics model. 

--- experiments ---

* What does the MSE graph look like for Humanoid? 
* Observation that MSE is not a good predictor of planning performance when using a learned model seems non intuitive and potentially an interesting thing to further investigate. 
E.g. Figure 3: Why is it that for longer horizons, MSE goes down and reward goes down? Since you only execute the first action anyway, how frequently does the plan deviate from what an optimal policy would on average do? 
If you were to enforce the proper physical constraints in the prediction, does that resolve the exploitation problem? Since you have access to the simulator and you’re only predicting observation joints, this feels like a conjecture that can be easily verified. This would also disentangle whether physical constraints are the issue vs. the prediction being wrong at critical states of the trajectory.
* Why did the authors choose to use MPO policy as the base policy? 
* Could the authors elaborate why the multi-step loss helps up to a certain number of steps for those domains? 
* Is state dimensionality the core issue in preventing humanoid from working? Could this hypothesis be testing on 4k+1 observation-space swimmer environment where k varies? 
* It’s impossible to have complete coverage of previous methods, but I feel like this work is missing a key class of dynamics models which are a function of history. E.g. in robotics settings (already cited by the authors), this includes concatenating states or having a recurrent neural network. This helps in settings where the environment is partially observable. Another thing is that it’s common practice to predict the difference in states. Authors should consider including experiments there.
* Related work is tied to robotics (and this is where a good dynamics model is super critical, not the synthetic simulated environments used here), but no robotics-like environment is used for evaluation.
* It’s surprising to see that ball-in-cup can (a) tolerate that much noise and (b) improves with increasing noise. Why is that the case for this domain, and not for the others? Eg. I could imagine Finger-Spin to also somewhat tolerate noise, but the fact that it’s sensitive makes me think that the observation ranges matter a lot in this study. Could the authors elaborate on this? 

Nits: 
* Title is not reflective of the paper goal. Currently reads quite broad - doesn’t indicate that this paper is about understanding the design choices for dynamics models.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper systematically studies a range of design considerations in model learning: 1) deterministic versus stochastic models, 2) 1-step versus multi-step targets, 3) the number of instances in an ensemble model, and 4) the effect of input noise to make the model more robust. They find 1) both deterministic an stochastic models work, 2) a medium horizon (3-5 steps) works best & model MSE is a poor indicator of planning performance, 3) ensembles improve performance up to about 5 components, and 4) the benefit of input noise varies over tasks. 
main_review: Strong:
* I like the overall idea of the paper, not to propose another new method, but instead make a systematic comparison of some important aspects of model-based RL. I think we could use more papers like this. 
* The paper is clearly written, has many results, and good additional videos online. 

Weak:
* One of my doubts is the diversity of the tasks. Since the Humanoid does not give any results, they essentially compare performance on 4 tasks. Given that this is a benchmarking paper, I think this is a somewhat narrow range. I think the results partially illustrate this, since on multiple occasions the results are somewhat inconclusive (on one or two it works better, on the other two on parr or worse).
* I think you missed some important related work [1], which already extensively studied the effect of multi-step losses in RL tasks.
* I doubt whether it makes full sense to study stochastic models on deterministic environments only (Sec 4.1). I see that they may still be beneficial, but to really say something about the necessity of stochastic predictions, I think you would need to at least have a few stochastic environments as well. 
* You decide to not propagate uncertainty over multiple steps because it cannot be done analytically through non-linear neural networks (Sec 4.3), which is true, but you can easily use a particle method for this right, i.e., use sampling? This is a common solution. 
* While is the paper is generally well written, you incidentally have some strange sentences with missing words. For example: “Therefore, the modeling the contact is not the main problem but the model exploitation in the vicinity of strong non-linear changes”. I think the paper needs one more careful proofreading. 

[1] Chiappa, Silvia, et al. "Recurrent environment simulators." arXiv preprint arXiv:1704.02254 (2017).
summary_of_the_review: This paper has positive and negative sides for me. As mentioned above, I like a comparison/benchmarking effort, which I believe the field indeed needs, and the authors have taken a nice, systematic approach, and written a well-structured paper. On the downside, I kind of doubt whether I learned too much from the results, since 1) the results vary quite a bit over tasks, which I think might be due to the relatively small set of tasks, 2) the results on stochasticity are limited by the set-up, which excludes stochastic environments, and 3) some other results were already partially known (like the multi-step prediction effects and poor performance of MSE). I am therefore a bit in doubt, because the paper does still have its merit in systematically comparing different element of model learning in deterministic control tasks.
review_title: null

Meta-review: The paper studies the effect of different design choices related to learning a dynamics model. The reviewers uniformly agree that the topic of the paper, systematically studying different design choices, is important. Furthermore, the paper is very well written. However, there are a number of weaknesses as well, that limit the relevance of this work. Arguably, the main weakness is that the results are inconclusive: there is no single design choice that is better, a conclusion that provides little guidance for researchers working in this space. Another weakness is that the study focuses on only 4 domains. And while performing such a study on a much broader set of domains can be prohibitively expensive, that doesn't take away from the fact that it is hard to draw strong conclusions from such a small set of tasks. For these reasons, I recommend rejection.
Decision: Reject