Learning Dynamics Models for Model Predictive Agents
Authors: Michael Lutter
Leonard Hasenclever
Arunkumar Byravan
Gabriel Dulac-Arnold
Piotr Trochim
Nicolas Heess
Josh Merel
Yuval Tassa
Venue: ICLR 2022 Submitted (2022)
Keywords: Model Learning
Model Based Reinforcement Learning
Control

Abstract: Model-Based Reinforcement Learning involves learning a dynamics model from data, and then using this model to optimise behaviour, most often with an online planner. Much of the recent research along these lines presents a particular set of design choices, involving problem definition, model learning and planning. Given the multiple contributions, it is difficult to evaluate the effects of each. This paper sets out to disambiguate the role of different design choices for learning dynamics models, by comparing their performance to planning with a ground-truth model -- the simulator. First, we collect a rich dataset from the training sequence of a model-free agent on 5 domains of the DeepMind Control Suite. Second, we train feed-forward dynamics models in a supervised fashion, and evaluate planner performance while varying and analysing different model design choices, including ensembling, stochasticity, multi-step training and timestep size. Besides the quantitative analysis, we describe a set of qualitative findings, rules of thumb, and future research directions for planning with learned dynamics models. Videos of the results are available at https://sites.google.com/view/learning-better-models.

Under review as a conference paper at ICLR 2022
LEARNING DYNAMICS MODELS FOR MODEL PREDICTIVE AGENTS
Anonymous authors Paper under double-bling review
ABSTRACT
Model-Based Reinforcement Learning involves learning a dynamics model from data, and then using this model to optimise behaviour, most often with an online planner. Much of the recent research along these lines (Chua et al. (0188) Janner et al. 2019 Yu et al. 2020) presents a particular set of design choices, involving problem defiiiton,, model learning and planning. Given the multiple contribu- tions, it is difficult to evaluate the effects of each. This paper sets out to disam- biguate the role of different design choices for learning dynamics models, by com- paring their performance to planning with ground-truit model the simulator. First, collect rich dataset from the training sequence of model-free agent 5 domains of the Deep.innd Control Suite. Second, we train feed-fore cord-forward dynam- ies models in supervised fashion, and evaluate planner performance while vary- ing and analysing different model design choices, including ensembling, stochas- ticity, multi-step training and timestep size. Besides the quantitative analysis, we describe a set of qualitative findings, rules of thumb, and future research directions for planning with learned dynamics models. Videos of the results are available at https: /ttesstess.s..///ee///eee///eees google com / /ivvvvewwwwe http://arrnnnggn bettamiinn-eette-mmmddeeeee
INTRODUCTION
Recently reinforcement learning (RL) (Sutton & Barto 2018 in particular actor-critic ap- proaches (Lillicrap al.] 2015) were shown to successfully solve variety of continuous control problems (Schulman et al. 2017 Lillicrap et al. 2015 Fujimoto et al. 2018 Haarnoja al.) 2018). The simplicity this approach has led to an explosion of research demonstrating the effectiveness of these methods (Vecerik et al.) 2019 Kalashnikov et al. 2018). However, model-free RI. suffers from two key disadvantages (Dulac-Armol
et al. 2021). First, model-free RL is sample inefficient, often requiring millions or billions of environment interactions. Second, the learned policies are tied to specific task, making transfer of learned knowledge in multi-task settings or across related tasks difficult (Finn al. 2017 Andrychowicz et 2017).
Model-Bused RL holds the promise of overcoming these drawbacks while maintaining the benefits of model-free methods. Model-based RL involves learning dynamics model from data, and then ing this model to optimise behaviour, most often with an online planner. Since the dynamics model ideally independent of the reward and state-action distribution, multi-task settings are simple: the
reward function can be changed at will (Rajeswaran et al., 2020; Argenson & Dulac-Arnold 2021
Belkhale et al. 202T). For the same reason, offline learning with off-policy data is possible Yu
et al. 2020 Kidambi et al. 20200 Argenson &
2021). Moreover, mrodel-based
RL
can achieve better sample efficieccy (Janner al.] Nagabandi et al. 2018 2020 Byravan
et al. 2020 Buckman et al. 2018 Hatner et al. 2019b] Argenson & Dulac-Amold
2021
In this paper we investigate the impact of the different model learning design choices on the planner performance. For this evaluation we focus on learning models using feed-forward neural networks as these are the dominant models in the recent model-based RL literature. A large portion these papers mainly proposes different methods to train the networks and variations the planner. Un- fortunately, each new paper proposes a series of evolutions, and so individual design choices are rarely ablated fully. This is the goal of our work. We look specifically at the effects of various model learning design choices on the planning performance. We consider 4 different design choices, deter- ministic stochastic models, multi-step vs. 1-step losses, of ensembles, and input noise. These

---

Under review as a conference paper at ICLR 2022
U
(c)
Figure I: Domains of the DM Control Suite Tassa et al. ball in cup, (d) walker, and (e) humanoid. These domains include various challenges like high-dimensionality non-rigid strings multiple contacts.
2018 used in this paper. (a) cart-pole, (b) finger, (c)
design choices have been proposed to obtain better long term prediction (Abbeel al. 2005 Chua
et al. 20118] Hafner et al., 2019b) and planning performance (Venkatraman et al. 20155] Sanchez- Gonzalez et 2020 Pfaff et al. 2020). To derive best practices for each phoice-parametes perform ablation studies evaluating the impact of the different choices. In addition we investigate the qualitative performance of the learned models, by evaluating their ability describe complex trajectories with reasonable fidelity, and to be consistent.
The contributions of this paper are
We perform systematic comparison evaluating the planning performance different design choices for model learning, i.e., stochastic models, multi-step loss, network ensembles and input noise, on five DeepMind Control Suite environments.
We observe that mean squared error (I-step multi-step) not good predictor of planning performance when using learned model.
We find that multiple model learning design choices can obtain high reward and consistent long-term predictions for the evaluated systems excluding the humanoid. The differences in reward between the different combinations is not significant.
We characterise best practices for learning models using feed-forward networks. For experiments, deterministic models need to be trained using the multi-step loss and combined with ensembles. Stochastic models require more ensemble components and input noise.
The paper is structured follows. First, cover related work model-based RL for continuous control in Sec 2. Afterwards, we review the naive approach to learning dynamics models (Sec. 3 Section 4 introduces the design choices of model learning and evaluates the learned models using model- preeiccpreeittive control. The subsequent discussion (Sec. 5 summarizes the insights, results and highlights general challenges of model learning for planning.
RELATED WORK
Learning dynamics model has a long tradition in robotics. The seminal work Atkesoo et al. (196))
proposed to learn the dynamics parameters of rigid-body kinematic chain from data. To remove the
limitations of analytic rigid-boo models (Atkeson et al. 1986; Lutter et al. 2020 2021). black- box
function approximation was leveraged (Schaal et al. 2002; Choi al. 2007] Calinon et al., 2010
Nguyen-Tuong et al. 2009 Nguyen-Tuong & Peters 2010 Lutter et al.) 2019 Lutter & Peters
2019) to learn system models able learn the peculiarities real-world robotic systems.
The majority of model-based RI. focuses on learning dynamics models, which predict the next step using the current state and action. The model learning for control literature refers to these models as forward models (Nguyen-Tuong & Peters 2011). Model-based RL algorithms use the model as a simulator to generate additional data (Sutton 1991: Janner et al., 2019 Morgan et al. 2021), T), to evaluate the reward of an action sequence (Chua al. 2018 Nagabandi et al. [018 Lambert al.
2019 Nagabandi et al. 2020), to improve the value function estimate (Feinberg et al. (0018] Buck-
man et al. 2018) or backpropagate the policy gradients through time (Miller al. 1995 Deisenroth
2

---

Under review as a conference paper at ICLR 2022
& Rasmussen 2011 Heess al. 2013 Byravan al. 2020 Amos al.) 2021). Current model- based RL methods use 1-step predictions to compute the optimal trajectory. Most current model- based RL approaches use different architectures of deep networks to approximate the model, e.g.,
deterministic networks (Nagabandi et al. 2018: Byravan et al. 2020 Feinberg et al. 2018: Kuru-
tach et al. 2018), stochastic networks (Heess al. 2015 Lambert al. 2019), recurrent networks
(Hafner et al. 2019al b 2020; Ha & Schmidhuber 2018). 8), stochastic ensembles (Chua et al., 2018
Nagabandi al. 2020 Janner al. al.] 2019 Kidambi et al. 2020; Yu al. 2020 Buckman al.
2018 Rajeswaran et al. 2020 Lambert et al. 2020b) and graph neural networks (Sanchez-Gonzall
ct al. 2018).
In this work we focus on feed-forward neural networks for learning dynamics models with contin- uous states and actions. The learned models are used with model predictive control (MPC) (Garcia et al. 1989: Allgöwer & Zheng 2012 Pinneri et al. 2020) to solve continuous control tasks. Sim- ple forward neural networks are the standard of most model-based RL. and already offer many different variations that affect the planning performance. For the considered environments, MPC enables the direct evaluation of the learned models without requiring policy or value function approximation. Therefore, the model performance for planning be directly measured and com- pared to planning with the ground-trun toouh model.
3 LEARNING DYNAMICS MODELS
We concentrate on 1-step forward models and review this setup in more detail. Dynamics models predict the next observation I441 using the current observation I; and action u, These models can be used as a simulator for planning without interacting with the real system. For example, some sample-based planning algorithms sample action sequences close the current plan, evaluates the reward of each action sequence by simulating the trajectories using the dynamics model and updates the plan using the action sequences with the highest reward. The simplest approach learn such a forward model is to use a deterministic model that minimizes the prediction error. This objective can be achieved by minimising the 1-step mean squared error between the predicted and observed next state. This optimization loss is described by
0- arg min
Σ (a,+1 x041)² (#++1 xi+1) with Li+1 f(x;i u,;0), 0.uED
ND
0
with the model parameters the dataset 'D with ND samples.
Dataset Current model-based RL algorithms re-use the data collected by the agent to learn the model. Therefore, the data is acquired in the vicinity of the optimal policy. This exploration leads to a specialized model for the considered task as only regions of high reward are explored. Hence, the current learned models are not independent of the state-action distribution and the reward function cannot be interchanged at will. In contrast to model-based RL, system identification focuses purely on obtaining the optimal data for learning the model cover most of the state space.
Integrator Instead of predicting the next observation, one predict the change of the observation. Then the system dynamics are modelled as an integrator described by i;+1 2t; + Al J f(x;, u,; 0) with the time step At. This technique has proven to result in more stable long term prediction compared to predicting the next state and has been widely adapted (Chua et al. 2018 Janner et al.) 2019
Yu et al. 2020 Amos et al. 202T Lambert et al. 2020a).
Normalization Optimizing the MSE can be problematic if the scales of the different observations different. In this case, the model overfits to the dimensions with larger scales. For forward models this results in overfitting to the velocities as the position is usually an order of magnitude smaller. To mitigate this overfitting the MSE can be normalized using the diagonal variance of the dataset D.D The normalized MSE is described by
0+ arg min
Σ R.
(x++1 - 2.41)" DD' (x1+1 2.41).
B
NO
The normalized MSE is comparable standardization of the model inputs and outputs. In our periments we did not observe any significant benefits using standardization instead of the normalized MSE.
3

---

Under review as a conference paper at ICLR 2022
Cartpolle
Swingup
Finger Spin
Ball
In
Catch
Walker Malk
Hunanoid
Stand
1869
1860
1863-
1600
1016
Berattoo Roward
408
393-
593-
582
910
Suterninist Mocel
Stochastic Model
Hotel
469
403
403-
405-
416
200
200
202
202
220
Updates [3a-05]
(36H05]
Figure 2: The learning curves comparing deterministic and stochastic models for five seeds on the five test domains. The shaded area marks the 20/80 percentile. For all systems except the humanoid, the stochastic and deterministic ensembles learn good model that enables successful planning. The obtained reward is marginally below the reward obtained by planning with the model. For the humanoid both approaches fail.
4 RESULTS
In the following we present our quantitative and qualitative evaluation of the model learning design choices. We evaluate the planning performance for 4 different design choices, i.e., stochastic mod- els (Sec. 4.2). multi-step loss (Sec. 4.3 network ensembles (Sec. 4.4) and input noise (Sec. 4.5
These different design choices have been previously proposed within the literature improve the model for planning (Abbeel et al. 2005] et al. 22118: Hafner et al. (2019b) and generate better long-term predictions (Venkatraman et al. 2015| Sonzaez- et al. 2020 Pfaff et al. 2020). For each of these proposed models we first introduce the approach, summarize the results and con- clude the best practices. The best hyperparameter for model learning were identified first using random search and the ablation studies compare to the optimal hyperparameter. The humanoid is only used for the initial ablation study as no model-learning approach learns a model that could be used for planning. A more detailed experimental setup. the hyperparameters and additional ablation studies are contained in the appendix. The links within the results point to the referenced section contained in htttss://ssiette googl e com/ vi ew/ Learninnn ng better mondeterrrmmedeee s/
4.1 EXPERIMENTAL SETUP
Data Sets For the experiments the models are learned using a fixed data set. The data sets are recorded using five snapshots of an MPO policy (Abdolmaleki et al. 2018) during the training process. Therefore, the data set has diverse trajectories including random movements and optimal trajectories. The combined data set contains 500 episodes which corresponds approximately to hour of data for the cartpole and hours of data for the humanoid.
Evaluation Criteria The models are evaluated using the normalized task reward and the MSE. The normalized reward is rescaled such that the maximum reward per episode is 1000. The MSE is computed using the test set with an 80/20 split and normalized using the data set variance. If not specifically mentioned, the MSE is computed between a predicted and observed trajectory of 0.5s.
Model Predictive Control The planning performance of the models is evaluated using CEM- based MPC agent (Garcia et al., 1989 Allgöwer & Zheng 2012 Pinneri et al. 2020). Instead of maximizing the raw actions, we optimize the control points and interpolate linearly to obtain the actions at each simulation step. This approach simplifies the planning as the search space is decreased. We do not use terminal value function. The MPC hyperparameters are optimized w.r.t. the true model and held fixed for planning with all learned models. For ensembles, we use fixed ensemble component per rollout during forward prediction and do not switch between different components per rollout step. The reward is averaged across the rollouts of the ensemble (Chua et al.) 2018 Lambert et al. 2019 Nagabandi et al. 2020 Argenson & Dulac-Arnold 2021
4.2 DETERMINISTIC vs. STOCHASTIC MODELS
Instead of using a deterministic model, Chua et al. (2018) proposed to use stochastic models. This model predicts the change of the system using a normal distribution. If the covariance matrix is
4

---

Under review as a conference paper at ICLR 2022
Cartpole
Swingup
Finger
Spin
Ball In Cup
Catchh
Walker Malk
2016
1860
1640
1860
True Model
Detormintstt Hodel
808
369
820
200
Steps
Steps
406
460
450
400
Steps
200
220
Steps
Steps
Ste
Desurmintatz Nodal
Steps
Squarez
20
Steps
Steps
20-2
Steps
Figure 3: The learning and loss curves averaged across five seeds for different horizons the multi-step loss. The MSE is computed between an observed and predicted trajectory of length 0.5s. The shaded area visualizes the 20/80 percentile. For all systems, the multi-step loss for up to 5 steps improves the model performance for planning. For longer horizons, the MSE keeps decreasing but the obtained reward also decreases.
assumed to be diagonal, the next state is described by
L++1 = At (ú( ue; 0) ( u,; 0)
with & ~ (O, 1), the mean f and standard deviation à. Instead of minimizing the MSE, the stochastic model maximizes the 1-step log-likelihood described by
0 arg
1
Σ BNED
(Ax
pa) (Ax (t) + log((E))= k log((2))
0 2ND
with A = (#.+1 x.) At, the diagonal covariance matrix Σ containing 6² and the state space dimensionality k. To make the optimization numerically stable the standard deviation bounded by maii max]] This clipping can be obtained using the differentiable transformation
min
S (a
min S(O - -))
with the softplus funetion S. Depending on the implementation, the network can either predict the standard deviation (Hafner et al.) 2019a) or the logarithm of the standard deviation (Chua et al.) 2018). The common motivation for this stochastic model the estimation of aleatoric uncertainty, which arises from the fundamental stochasticity of the environment. However, most performed experiments use a deterministic environment, which has no alcatoric uncertainty. The main benefits of the stochastic model for deterministie environments is the lower entropy bound and the adaptive selection of precision. The lower entropy bound reduces the risk model exploitation by increasing the variance of the model predictions. The state-dependent variance enables model with limited capacity to trade-off precision with higher variance at specific states. Prior work (Nagabandi et al.) 2020 Hatner al. 2019b) has also shown that fixing the variance for all states can achieve the same performance as the deteedee variance.
Results The learning curves comparing stochastic and deterministic models are shown Figure Videos of the model performance are available at [[ink]] Both the deterministic and the stochastic ensemble learn a model that can be used for planning for most environments. For the cartpole, finger, ball in cup and walker the learned models are able to solve the task. The differences in reward are not large and the qualitative differences the rewards are not visible. Compared to the true model, the obtained rewards are marginally lower and exhibit higher variance. The differences between models are not clearly visible in the videos. Visualising the plans [Link] of the approximated models shows that the plans are plausible. Only for the ball in cup system the model exploitation is obvious as the ball can pass through the top of the cup. When transferring the models to a different task (Ap- pendix Table. 2). increasing the replan intervals (Appendix Fig. 8 and comparing the learning
5

---

Under review as a conference paper at ICLR 2022
Cartpole
Swingup
Finger
Spin
Ball In Cup
Cattch
Walker Malk
2016
1860
1640
1860
True Model
Nurmitien Decormintsrt Hodel
828
369
820
300
Enseeble
Enseeble
Ensemble
Ensemble
406
460
4.6
400
Enseeble
Enseeble 18
200
220
1800
1.000
1800
True Model
Sto Podes
0103
8:00
Enseeble
Enseeble
Ensemble
Enseeble
460-
4:0
Enseeble 10
200
220
Figure 4: The learning curves averaged over five seeds for different number of ensembles. The shaded area highlights the 20/80 percentile. The obtained reward increases when increasing the number of ensembles. For stochastic ensembles the increase performance is higher compared to deterministic ensembles. Without an ensemble both models fail the task and only obtain lower reward.
speed, the performance of the deterministic ensembles is better. For the humanoid both variants fail to learn good model. The MPC agent is able to easily exploit the approximated model. The visu- alized plans show that the optimization finds an action sequence that lets the humanoid magically float midair [Link]]
The results are partially different compared to CoChua al. (018), which shows that the stochastic ensembles perform better than deterministic ensembles. We also found that stochastic ensembles are better, when the deterministic model was trained using the 1-step loss. However, if the deterministic model is trained using the multi-step loss (Sec. 4.3) the deterministic models achieve the same or better performance than the stochastic models trained on the 1-step log- likeliikdoood.
Conclusion Both deterministic and stochastic ensembles capable learn models for planning for most evaluated systems and fail for the humanoid. However, both model types require different design choices to learn good model for planning.
4.3 1-STEP Loss vs. MULTI-STEP Loss
The previously described losses optimized the 1-step loss that uses the information of two consec- utive observations to learn the parameters. While this information is theoretically sufficient to learn the model accurately, various approaches (Abbeel et al. 20055; Venkatraman et al. [0055] Hafner al., 2019b) have used the multi-step loss learn better models. For the deterministic model the multi-step loss is described by
H HND ΣΣ
0" arg min
(0++1 2 EET' (I++1 2141)
with the horizon H. the prediction 2:+1 24 + At f(i;, uN 0) and 2 Zg. The computational complexity of the multi-step loss grows linearly with the horizon the model needs be evaluated H times for each gradient update. Extending the multi-step loss to stochastic models is non-trivial. To compute the log-likelihood conditioned on the previous prediction one would need to propa- gate the uncertainty through the non-linear dynamics which is not possible analytically. Various approximations to propagate the uncertainty have been used for optimizing the policy (Deisenroth & Rasmussen, 2011 Watson al. 2021), but these approaches have not been adapted to model learning, yet. Therefore, we do not consider these approximations within this comparison and leave it for future work.
Results The learning curves and loss curves on the test set are shown in Figure 3 reward for multi-step loss using up to 10 steps is higher than the reward of the 1-step optimization
The obtained

---

Under review as a conference paper at ICLR 2022
Cartpole
Swingup
Finger Spin
Ball
In Cup
Cattch
Walker Malk
2016
1860
1640
1860
True Model
Decormintsst Hodel
800
369
810
200
Noise 8exd6
Noise 5e-63
Noise 1e-02
406
460
4.6
460
5e-02
Noise 1e-81
200
220
1820
1.000
1000
True Model
Sto Mades de
800
0.03
8:0
Noise 0c-68
Noise Se-63
Noise le-82
436
460-
450
Noise 1e-81
200
220
[3e405]
Figure 5: The learning curves averaged over 5 seeds for different levels of input noise. The shaded area highlights the 20/80 percentile. For deterministic models the input noise is not required to achieve high reward. For stochastic models the input noise significantly improves the reward for walker and ball in cup. Without the noise ball in cup cannot solved using the stochastic ensemble.
loss. For very long horizons the obtained reward drops for finger and ball in cup. The increase of the reward due to multi-step loss varies by system. For the walker and cartpole the increase is not large. For the finger and ball in cup the multi-step loss is required to achieve high reward. For all systems an horizon of 3 to steps works best, which much lower compared to the planning horizon of 50+ steps. In addition, we observed that increasing the horizon using schedule results in a lower reward (Figure 9). The systems that benefit the most from the multi-step objective only obtain reward comparable the 1-step optimization loss. In contrast the reward, the MSE keeps improving when increasing the horizon. This observation highlights that a lower MSE does necessarily imply better model for planning. We elaborate the implications of the MSE on the planning performance in section
Conclusion For deterministic ensembles the multi-step objective increases the reward. Increasing the horizon using a schedule performs worse. A horizon between 3-5 steps performs the best for these systems. Despite the lower multi-step MSE for long horizons, the planning starts to fail for too long horizons. This observation highlights that the MSE (single multi-step) is not good indicator for the planning performance. A similar observation has been made for the 1-step log- likeehood by Lambert et. al. (Lambert et al. 2020a).
4.4 SINGLE NETWORK vs. NETWORK ENSEMBLE
To prevent the planner to exploit the model, various authors used the ensemble disagreement as a measure of epistemic uncertainty to regularize for the policy optimization. For example, PETS (Chua et al. 2018) averages over the reward the predicted ensemble trajectories to pre- vent over-fitting to too optimistic predictions. MOReL (Kidambi et al.) 2020) and MOPO Yuu et al.) 2020) add a reward penalty to state with high epistemic uncertainty to prevent the policy optimiza- tion venturing into uncertain regions. The epistemic uncertainty can be approximated using boot- strapped network ensembles. The ensembles stack deep networks with different initialisation and train the networks with different minibatches. Therefore, all ensemble components are trained using the same data but the order and composition of the mini-batches is different. The computational and memory complexity for training the models and planning increases linearly with the number of ensemble components.
Results The learning curves for different ensemble sizes are shown in figure 4 Videos highlighting the model entropy [Link] and open-loop rollouts [Link] are available. More ensemble components increase the obtained reward. The increase in reward depends on the model type and specific system. For deterministic models the ensemble size only significantly matters for ball in a cup. For this task, the reward doubles when using model ensemble. For the other systems the reward increases by up

---

Under review as a conference paper at ICLR 2022
Cartpole-
Swiingup
Finger
Spin
Bal.1
Catchh
Malker
Humanoid
Stand
18
10a
169
18'
10t
Density
10"
10
10"
10:3
10
10
18
10t
-25
25
-25
-25
-25
Discrepancy
Biscrepancy
Revard riscrepancy
Discresancy
Discrepancy
Figure 6: The reward discrepancy measuring the difference between the expected and observed reward. the discrepancy is positive the planner is too optimistic. If the value negative the planner is too pessimistic. The planner with the true model rarely over-estimates the future reward. The learned model frequently estimates the future reward highlighting the model exploitation. Especially pects a too high reward in close to all time steps. For the other systems the model exploitation is not so drastic as the maximum mode ist at O.
humanoid the planner ex-
to 20%, which qualitatively does not significantly affect the task completion. For stochastic models, the advantages of ensembles is more significant compared to deterministic models. For all systems except the cartpole, the ensembles significantly outperform the single deep network. It is important to note that the computational complexity grows linearly with the number of ensembles. Therefore, future work needs to evaluate whether the increased performance of ensembles persists when the computational budget is fixed.
Qualitatively, the videos show that each ensemble component provides a stable long-term prediction of 1 second for all systems except the humanoid. The open-loop rollouts of the provide diverse set of trajectories that are all physically plausible and not obviously wrong. However, the predictions not perfect and have minor errors. For example, the walker marginally sinks into the floor and the edges of the cup are not modelled accurately. Only for the humanoid the predictions yield diverging trajectories that are obviously wrong. Furthermore, the entropy describing the epistemic uncertainty increases at intuitive state space configurations. For example the entropy increases when the finger touches the spinner or the walker is in contact with the floor.
Conclusion Adding more ensemble components to the model increases the planning performance. For the considered systems, the stochastic models benefit more using ensembles. The reward gain beyond five ensemble components is marginal. It is important to note that single deep network does not completely fail the task and only obtains a lower reward.
4.5 PERFECT OBSERVATIONS VS. INPUT NOISE
For dynamics models the data commonly lies a manifold due system constraints and over- parametrized observations. For example the cartpole positions with sine/cosine features lie on the cylinder. When predicting long time horizons using the approximate model, one will inevitably deviate from the manifold. This deviation is inevitable as even simulators with perfect assumptions and knowledge frequently violate constraints. Simulators use forces orthogonal to the constraint to return to the manifold. For learned models returning to the manifold is not straight forward as the dynamics are random outside manifold. Therefore, is very common that the policy optimization exploits these random dynamics during planning to obtain high reward. [Sanchez- Gonzalez et al. (2020) proposed to add noise to the inputs to learn dynamics model that remains on the manifold for long horizons. Adding noise the inputs causes them to lic outside the manifold. Using these inputs the model learns to predict solutions that return to the manifold. In this case the next step of the stochastic model is described by
Lt+1 T. + Al [Ā + L w Ue; 0) + ò + X Lo w, un; 0) E]
with w (O, the Cholesky decomposition Lp of the variance Ep. and noise amplitude X.
Results The learning curves for different noise amplitudes are shown in figure Adding noise to the inputs can be beneficial, if the noise amplitude is not chosen too large for the specific environment. the amplitude too large the obtained reward drops significantly. For the deterministic models trained using the multi-step loss the input noise is not required achieve good performance. For
8

---

Under review as a conference paper at ICLR 2022
the stochastic models the input noise significantly improves the obtained reward for ball in cup and walker. Especially for ball in cup the adding inputs noise is required to obtain a high reward. Without the input noise, the stochastic model fails on this task.
Conclusion Adding input noise during the training can improve the model performance, if the noise amplitude is not too large. For some models this data augmentation is even required to solve the task, e.g., ball in cup with a stochastic model.
CONCLUSION
Feed-forward networks can learn models that enable planning for most of the considered systems. The obtained reward is only slightly lower than the reward when planning with the true model. Qualitatively, the models generate feasible plans and stable open-loop predictions for more than 100 steps. No single design choice works significantly better. Multiple combinations achieve comparable reward. Deterministic models must be trained using the multi-step loss and combined with ensembles to achieve good performance. Stochastic models trained using the 1-step log- likelihood perform well when ensembles are used and noise is added to the inputs.
High Dimensional Systems All tested models fail for the humanoid. The learned models predict diverging trajectories for open-loop sequences. The planner is too optimistic (Fig. ( and finds an action sequence that lets the humanoid float midair [Link] The results for the humanoid are expected as to the best of our knowledge only DreamerV2 (Hafner et al. (020)) obtained a
humanoid model that is able to generate stable open-loop rollouts by using significant more data, online learning and a recurrent latent space model.
Model Exploitation The model exploitation is visible when comparing the difference between ex- pected and obtained reward (Fig. (6. The approximate model frequently overestimates the reward, while the true model does not. Especially, systems that involve contacts of two moving bodies, e.g., finger and ball in cup, are more prone to exploitation. One common failure mode for ball in cup that the planner tries to tunnel the ball through the bottom of the cup [Link]] The collisions of ball and cup are modelled for the open-loop trajectories. Only during planning, action sequence found that enables to bypass this collision. Therefore, the modeling the contact is not the main problem but the model exploitation in the vicinity of strong non-linear changes.
Mean Squared Error The experiments show that the MSE (multi-step or I-step) is not a good indicator for planning performance. A similar observation was described by Lambert et al. (2020a) for the 1-step log-likeli It is important to note that this observation does not suggest that one desires higher MSE model. The result only suggests that MSE is not sufficient to estimate the planning performance of the model. During training, the reward increases and the MSE decreases with the number of updates. However, the MSE does not provide information about the planning performance when comparing two trained models. A particular peculiar example of this effect is shown in figure I0] For ball in cup only coarse time step obtains a high reward, even though the MSE is an order of magnitude higher compared to smaller time steps. It is intuitive that the accuracy not that important for this task. The trajectory after the collision does not need to be modelled precisely. The model must only ensure that the ball can not tunnel into the cup.
Future Work To advance model learning for planning, the existing techniques must be (1) extended to high-dimensional observations and (2) made more robust to avoid model exploitation. As data fit is not the main indicator for planning performance and ensembles are not sufficient to prevent model exploitation, a promising research direction is adversarial model learning. This approach can explicitly incorporate the planner into the model learning objective to minimize the reward discrepancy and promises to learn a robust model. One should specifically focus on dynamics with strong non-linearities as these are more susceptible to model exploitation.
REFERENCES
Pieter Abbeel, Varun Ganapathi, and Andrew Ng. Learning vehicular dynamics, with application to modeling helicopters. Advances in Neural Information Processing Systems, 2005.
Abbas Abdolmalcki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum posteriori policy optimisation. arXiv preprint arXiv: 1808880060620

---

Under review as a conference paper at ICLR 2022
2018.
Frank Allgöwer and Alex Zheng. Nonlinear model predictive control. Birkhäuser, 2012.
Brandon Amos, Samuel Stanton, Denis Yarats, and Andrew Gordon Wilson. On the model-based stochastic value gradient for continuous reinforcement learning. In Learning for Dynamics and Control, 2021.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. arXiv preprint arXiv: 0tte:11777..19.... 2017.
Arthur Argenson and Gabriel Dulac-Amold, Model-based offline planning. International Confer- on Learning Representations (ICLR), 2021.
Christopher G Atkeson, Chae H An, and John M Hollerbach. Estimation of inertial parameters of manipulator loads and links. The International Journal of Robotics Research (IJRR), 1986.
Suncel Belkhale, Rachel Li, Gregory Kahn, Rowan McAllister, Roberto Calandra, and Sergey Levine. Model-based meta-reinforcement learning for flight with suspended payloads. IEEE Robotics and Automation Letters, 2021.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample- efficient reinforcement learning with stochastic ensemble value expansion. arXiv preprint arXiv:1807.01675, 2018.
Arunkumar Byravan, Jost Tobias Springenberg, Abbas Abdolmaleki, Roland Hafner, Michael Ne- unert, Thomas Lampe, Noah Siegel, Nicolas Heess, and Martin Riedmiller. Imagined value gra- dients: Model-based policy optimization with tranferable latent dynamics models. Conference on Robot Learning (CoRL), 2020.
Sylvain Calinon, Florent D'halluin, Erie L Sauser, Darwin G Caldwell, and Aude G Billard. Learn- ing and reproduction of gestures by imitation. IEEE Robotics & Automation Magazine, 2010.
Younggeun Choi, Shin- Young Cheong, and Nicolas Schweighofer. Local online support vector regression for learning control. In International Symposium on Computational Intelligence in Robotics and Automation, 2007.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in a handful of trials using probabilistic dynamics models. Neural Information Processing, 2018.
Marc Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In International Conference on machine learning (ICML-I1). 2011.
Gabriel Dulac-Arnold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, 2021.
Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I Jordan, Joseph E Gonzalez, and Sergey Levine. Model-based value estimation for efficient model-free reinforcement learning. arXiv preprint arXiv:1803.0010 00101, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leant leta-learnnig for fast adaptation of deep networks. In International Conference Machine Learning. PMLR, 2017.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, 2018. 8.
Carlos E Garcia, David M Prett, and Manfred Morari. Model predictive control: Theory and prac- tice-a survey. Automatica, 1989.
David Ha and Jürgen Schmidhuter. World models. arXiv preprint 1803. 10/22, 2018.
10

---

Under review as a conference paper at ICLR 2022
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforeement learning with a stochastic actor. In International Confer- on Machine Learning (ICML), 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint VrXiv:/922.01600 2019a.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning (ICML), 2019b.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. arXiv preprint arXiv:2010.02192 2020.
Nicolas Heess, Greg Wayne, David Silver, Timothy Lillicrap. Yuval Tassa, and Tom Frez. Learning continuous control policies by stochastic value gradients. Neural Information Processing Systems (Neurips), 2015.
Michael Janner. Justin Fu. Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. Neural Information Processing Systems, 2019.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprini urXiv: 10293, 2018.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. arXiv preprint urXiv:2005.05951 2020.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint 1rXiv:1802.10590 2018.
Nathan Lambert, Brandon Amos, Omry Yadan, and Roberto Calandra. Objective mismatch in model-based reinforcement learning. arXiv preprint arXiv:2002.04523 2020a.
Nathan O Lambert, Daniel S Drew, Joseph Yaconelli, Sergey Levine, Roberto Calandra, and Kristofer SJ Pister. Low-level control of a quadrotor with deep model-based reinforcement learn- ing. IEEE Robotics and Automation Letters, 2019.
Nathan O Lambert, Albert Wilcox, Howard Zhang, Kristofer SJ Pister, and Roberto Calandra. Learning accurate long-term dynamics for model-based reinforcement learning. arXiv preprint arXiv:2012.09156. 2020b.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 1509. 150.:00999909..... 2015.
M. Lutter and J. Peters. Deep lagrangian networks for end-to-end learning of energy-based con- trol for under-actuated systems. In International Conference Intelligent Robots and Systems (IROS), 2019.
Michael Lutter, Christian Ritter, and Jan Peters. Deep lagrangian networks: Using physics as model prior for deep learning. In International Conference on Learning Representations (ICLR), 2019.
Michael Lutter, Johnannes Silberbauer, Joe Watson, and Jan Peters. A differentiable newton euler algorithm for multi-body model learning. In Robotics: Science and Systems Conference (RSS), Workshop on Structured Approaches to Robot Learning for Improved Generalization, 2020.
Michael Lutter, Johannes Silberbauer, Joe Watson, and Jan Peters. Differentiable physics models for real-world offline model- reinforcement learning. International Conference Robotics and Automation (ICRA), 2021.
W Thomas Miller, Paul Werbos, and Richard S Sutton. Neural networks for control. MIT press, 1995.
11

---

Under review as a conference paper at ICLR 2022
Andrew S Morgan, Daljeet Nandha, Georgia Chalvatzaki, Carlo D D'Eramo, Aaron M Dollar, and Jan Peters. Model predictive actor-critic: Accelerating robot skill acquisition with deep reinforcement learning. arXiv preprint arXiv:2103.13842 2021.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning, IEEE International Conference on Robotics and Automation, 2018.
Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep dynamics models for learning dexterous manipulation. Conference on Robot Learning, 2020.
Duy Nguyen-Tuong and Jan Peters. Using model knowledge for learning inverse dynamics. In International Conference on Robotics and Automation (ICRA), 2010.
Duy Nguyen-Tuong and Jan Peters, Model learning for robot control: a survey, Cognitive process- ing. 2011.
Duy Nguyen- Tuong, Matthias Seeger, and Jan Peters. Model learning with local gaussian process regression. Advanced Robotics, 2009.
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter W Battaglia. Learning mesh- based simulation with graph networks. arXiv preprint arXiv:2010.03409, 2020.
Cristina Pinneri, Shambhuraj Sawant, Sebastian Blaes, Jan Achterhold, Joerg Stueckler, Michal Rolinek, and Georg Martius. Sample-efficient cross-entropy method for real-time planning. arXiv preprint 111i::2008.063888 2020.
Aravind Rajeswaran, Igor Mordatch, and Vikash Kumar. A game theoretic framework for model based reinforcement learning. In International Conference Machine Learning, 2020.
Alvaro Sanchez-Gonzaler Nicolas Heess, Jost Tobias Springenberg. Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. In International Conference on Machine Learning (ICML), 2018.
Alvaro Janchez-Gonzalee Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning simulate complex physics with graph networks. In International Confer- ence on Machine Learning, 2020.
Stefan Schaal, Christopher G Atkeson, and Sethu Vijayakumar. Scalable techniques from nonpara- metric statistics for real time robot learning. Applied Intelligence, 2002.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 1991.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint arxi: 000690, 2018
Mel Vecerik, Oleg Sushkov, David Barker, Thomas Rothörl, Todd Hester, and Jon Scholz. A prac- tical approach to insertion with variable socket position using deep reinforcement learning. In International Conference on Robotics and Automation (ICRA), 2019.
Arun Venkatraman, Martial Hebert, and Andrew Bagnell. Improving multi-step prediction of learned time series models. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Joe Watson, Hany Abdulsamad, Rolf Findeisen, and Jan Peters. Stochastic control through approx- imate bayesian input inference. arXiv preprint arXiv:2105. 07693, 2021.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005. 13239, 2020.
12

---

Under review as a conference paper at ICLR 2022
APPENDIX
EXPERIMENTAL SETUP
Systems To evaluate the model learning approaches, we apply these algorithms to Cartpole, Finger, Ball in Cup, Walker and Humanoid of the Deepmind Control suite (Figure These systems cover wide range of tasks, include contacts and include nigh-dimensional systems. For example the finger environment requires learning the interactions between the finger and the spinner. The ball in cup environment requires learning the collision shapes such that the ball does not tunnel into the cup. The walker and humanoid require to learn high dimensional dynamics and contacts with the floor.
Data Sets The data sets are recorded using five snapshots of an MPO policy during the training process. Therefore, the data set has diverse trajectories including random movements and optimal trajectories, The combined data set contains 500 episodes which corresponds approximately to hour of data for the cartpole and hours of data for the humanoid.
Observations The observations are over weer-arameterized and can include Cartesian coordinates or sensors. For example, the finger contains touch sensors on the tip. The humanoid observations contain the ego-centric positions of hands and feet. Therefore, the data does not correspond to minimal coordinates of the simulator. Due to the overparameterized representation, the data lies on lin-dimensional manifold defined by the kinematic structure. Therefore, not all points in IR" correspond to physically possible configuration. The system fully observed. Hence, the simulator state except the x-y world coordinates of the floating base systems can be reconstructed from the observations.
Reward Function For environments with sparse rewards the reward function is smoothed to provide a dense function. Therefore, the reward corresponds to the euclidean distance to the goal state. This smoothing is necessary provide good search signal for the planner. The reward per step is bounded to be between 0 and 1. The reported normalized reward scales the reward such that the maximum obtainable reward for single episode is 1000.
Mean Squared Error The reported MSE is computed using the test set with an 80/20 split and normalized using the data variance. If specifically mentioned the MSE is computed between predicted and the observed trajectory of 0.5s.
Model Predictive Control The planning performance of the models is evaluated using CEM- based MPC agent (Garcia et al. 1989, Allgöwer & Zheng 2012 Pinneri et al.) 2020). Instead of maximizing the raw actions, we optimize the control points and interpolate linearly to obtain the actions at each simulation step. This approach simplifies the planning as the search space is decreased. We do not use terminal value function. The MPC hyperparameters are optimized w.r.t. the true model and held fixed for planning with all learned models. For ensembles, we use fixed ensemble component per rollout during forward prediction and do not switch between different components per rollout step. The reward is averaged across the rollouts of the ensemble (Chua et al. 2018 Lambert et al. 2019 Nagabandi et al. 2020 Argenson Arnodd 2021).
7 ADDITIONAL EXPERIMENTS & ABLATION STUDIES
In addition the experiments presented in the main paper. we provide additional experiments that vary the dataset (Sec. 7.I). the task for the trained model (Sec. 7.2) and the replan fre- quency (Sec. 7.3). We also provide ablation studies evaluating the impact of a schedule for the multi-step loss (Sec. 7.4 and the impact of the time discretization (Sec. 7.5).
7.1 MODEL GENERALIZATION W.R.T. DATASET
To test the performance across different datasets, we compare the planning performance when vary- ing the dataset. We consider 5 different datasets that contain epsiodes with different average reward. For example dataset 0 contains only random actions while dataset 4 contains the converged MPO policy. The average and maximum reward of the datasets that the approach of learning a model and utilizing the model for planning achieves in most cases higher reward than the average and maximum reward of the dataset. For the most domains the
shown in figure Z The results show
13

---

Under review as a conference paper at ICLR 2022
Table 1: The hyperparameters for the deep networks and the model predictive control. The deep network parameters were identified using random search to obtain parameters with the highest reward for each system. Ablation studies comparing different network parameters did not show any coherent trend for all systems. Therefore, these studies are omitted. The MPC parameters were optimized using random search using the ground-truth dynamics models.
Network Parameters
Cartpole- Swing Up Stechustic
Finger
Spin
Ball in Cup Catch
Walker Walk
Humanoid Sland
Detemmnnistic
Deterministic
Detemmnisiic
Determinittic
Stuehartic
Time Step [s]
0,01
0.01
0.01
0.01
0.04
0.04
0.01
0.025
0.02
0,025
Netwark
(12
x2225
[22
256]
[3x5121
512]
5122
Activation
Swish
ReLu
Swish
Ela
ReLu
Swish
Swish
Elu
Elu
Elu
Leaming
Se-4
le-
le4
Ratch Size
256
512
512
178
128
512
178
256
128
Hurizoon
a Frsemble
Noise
5e-2
5e-2
1e-5
le-3
Model Predictive Control
Cartpole- - Swing Up
Finger Spin
Ball in Cup- Catch
Walker Walk
Humaseid SSann
Bonirom
1.25%/
0,254 254
0.75%
0.50%
0.50s
Replan Interval [s]
1/100s
1/100s
17100s
12100.
1/50x
Control Points
1/50x/ / 634
1/75s
1/25s/
1/20s/
1/50x/25
a Puriicess
200
500
200
200
500
Explontion Noise
0.5
1.0
0.5
1.0
0.75
Explonatio
arg in.
True
True
True
True
True
model-based planning approach obtains high reward for dataset 1 or datasei 2, which is far from containing the data of the optimal policy. Therefore, the learned models are capable to generalize the beyond the state-action distribution of the dataset. The generalization is not perfect as for the random dataset, the performance is commonly worse. However, this be expected as the random dataset often does not contain large portions of the state space. For example, the random actions dataset for the cartpole do not contain states close to the balancing of the cartpole. Therefore, the planner with the learned model can swingup the pendulum but not balance the pendulum. When comparing deterministic and stochastic models, the deterministic models achieve most cases a better or identical performance than the stochastic model. The only exception is dalaset 2 for ball in a cup.
Cartpole
Swingup
Finger Spin
Cup
walker
1800
1000
1000
1010
Nerallzzd hoard
800
032
300
810
000
8:0
2.0
2:0
2
2
2
3
Daliss
Dolaio
Datusel
Dla 00000
Figure 7: The obtained reward for the four different DM control suite environments and 5 different datasets with different average reward. most cases, the model-based planning approach obtains a higher reward than the average dataset reward. Even when the dataset contains only random actions, i.e, dataset 0, the planner obtains a much higher reward than random actions.
7.2 MODEL GENERALIZATION W.R.T. TASK
To test the transferability of the learned models, we transfer the approximate models between the different walker tasks. For example, the model is trained using walker walk data and applied to solve walker stand and walker run. The results are summarized table 2 The deterministic model per- forms well in the transfer tasks as the learned models generalize to the other two tasks. The obtained reward is marginally lower than training the model for the specific task. Only when transferring the deterministic model from walker stand to walker model on walker run. This difference is expected as the state- state-action distribution between the stand and run task are very different. Furthermore, the experiments show that the transfer from the
the reward is much lower than training the
14

---

Under review as a conference paper at ICLR 2022
Table The obtained reward when transferring the model between tasks. The model is trained using the train- ing domain and then applied to the evaluation domain. The deterministic model exhibits a good performance and obtains comparable reward than the model trained on the test domain for most transfers. The stochastic model performs worse than the deterministic model when the model is transferred between tasks.
Walker Stand
Walker Walk
Walker Run
Train Deterministic Ensemble
Model Type
Avg.
80th Per
Avg
80th Per
Per
Avg
820.1
897.8
976.4
787.8
853,5,5
949.5
344,4
394.6
446.2
Walker Stand
Stochustic Ensemble
468.4
690.0
905.5
296.6
426.0
565.6
87.0
140.9
190.9
Deterministic Frsemble
890.0
931.3
993.2
822.9
869.0
962.3
463.5
503.6
563.6
Walker Walk
Stechastic Ensemble
685.4
782.9
886.3
672.8
750.2
930.6
212.5
300.7
411.5
Deterministic Enserbble
799.7
865.2
985.5
817.7
839.9
954.5
422.7
464,6
559.7
Walker Run
Stechastic Ensemble
334.2
668.9
909.7
70.5
328.1
595.5
53.7
158.0
268.3
Model
927.7
883.8
920.9
563.1
590.4
the stand task is simpler than transferring the model from stand to the run task. This difference is intuitive as the run datasets contains configurations of the standing walker, while the stand task does not contain the data of the fast moving walker.
When comparing the transfer of the deterministic and stochastic model, the deterministic model performs much better than the stochastic model. When transferred the 20th percentile of the de- terministic model better than the 80th percentile of the stochastic model for most task transfer. Therefore, the deterministic model is better when transferring models between tasks.
7.3 REPLAN FREQUENCY
An additional experiment to test the model performance for planning is reduce the replanning frequency and observe the decrease in reward, As the replanning frequency is reduced, the planner evaluates fewer action sequences, is more likely to overfit the approximate model and takes more steps before adapting the plan to the observed model mismatch. Therefore, the obtained reward can decrease for the true and approximate model. The results are shown in figure g For the cartpole and finger domain, the stochastic and deterministic model perform comparable. For the ball in cup and walker domain, the deterministic model performs better than the stochastic model. Especially for the walker domain, the deterministic model performance does not decrease while the reward of the stochastic model decreases by lot. The obtained reward decreases slower does not decrease for the deterministic model while the reward of the stochastic model decreases faster.
Car spole
SwingUp
Fingerr
Spin
Ball in
Catch
Walker Wa Lk
Hunanoid
Stand
1800
1303
1207
1000
1030
Norest tzes Reward
800-
102
0:0-
020
820
542
600
408
102
402
402
410
Frequency [MC]
RepTan Frequency [Nc]
Seplan Frequency [Nz]
Reptan Frequency [Hz]
Replan Frequescy [Hzl
Figure 8: The obtained reward when decreasing the replan frequency for the deterministic, stochastic and true model. The shaded area highlights the 20/80 percentile. For the different replan frequencies, the deterministic model obtains better or comparable reward compared to the stochastic model.
7.4 MULTI-STEP LOSs WITH SCHEDULE
Instead of training directly the multi-step loss, one can increase the horizon of the multi-step loss using a schedule. One starts to train on the 1-step loss and increases the horizon with the number of updates. In this work we increase the horizon linear w.r.. the number of updates. The motivation for this schedule is that the multi-step loss is too ambiguous hard to optimize this loss directly. Therefore, one uses curriculum that starts with the simple 1-step loss and makes
15

---

Under review as a conference paper at ICLR 2022
the optimization slowly harder. The results for the multi-step loss using schedule is shown in figure In our experiments a schedule leads worse results than directly training on the multi-step loss directly. The MSE curves clearly show the effect of the horizon the loss curves start identical. After some updates, the loss curves spread out according to the horizon length. Similar to training without schedule, the longer horizon lead better MSE. However, the MSE becomes unstable for some very long horizons. This instability was not observed without the schedule. Furthermore, the obtained rewards are comparable or worse compared to not using the schedule. This especially visible for the Finger spin task. Even for the optimal horizon length of 5 steps, the schedule causes the model obtain reward comparable training without the schedule. Therefore, the model seems to over-fit fit to the 1-step and cannot obtain the same performance as training on the multi-step loss without schedule.
1-step loss, which significantly worse compared to
Cartpole
Swingup
Finger
Spin
Ball In Cup
Catch
Wol Walker
Malk
1000
1860
1800
1.60
True Model
Decorninssts Modet
806-
369
810
360-
Steps
Steps
Steps
416
468
400
Steps
206
208
210
280
St
44
[1er65]
Cartpole
Swingup
Finger
Ball Cup
Catcch
Walker
10
18a
12a
10a
Steps
Datermintsts Not
Steps
Phan: Squraal
26-1-
Steps
12a
12--
10a
Steps
Steps
Steps 26
St
46
12
120
Updates [1ee0]]
Uptates 13e-10 05
Updatex [1++05]
Updates
3
Figure 9: The learning curves and loss curves for the multi-step loss using schedule to linearly in- crease the horizon. The schedule cause the loss curve to start identical and spread once the schedule increases the horizon. Compared to training on the multi-step loss without a schedule, the training with the schedule performs comparable or worse
7.5 TIME DISCRETIZATION
frequently neglected design choice for model learning is the choice of the time step. The time step can be chosen multiple of the observation frequency. The common choice of model-based RL approaches is to use the time step of the model-free RL agent, which uses control frequencies of 30 100Hz and up-scales the actions using action repeats. These control frequencies are much lower than the simulation time steps ranging from 100 500Hz for the considered environments. However, the time step can be chosen differently to simplify learning the model. The learning and loss curves for different time steps are shown in figure 10 and figure
Deterministic Model If the time step is chosen coarse, the MSE is much higher than finer time step. Despite the lower accumulation error the MSE is higher. If the time step is too small, i.e., 1/100s, the MSE starts increasing again due the higher accumulation error. For most evaluated systems, the deterministic model obtained the smallest MSE for model time step between 1/40- 1/100s. terms of reward, the reward is lower for very coarse time steps as the time step lower bounds the control points distance for the planner. For example, the reward for cartpole swingup and finger spin the reward decreases for time steps greater than 1 1/100s. A particular peculiar example is ball in cup. For this domain, the planner only obtains high reward when using a very coarse model with time step of 1/20- 1/25s, despite that the MSE is about an order magnitude larger than the MSE for timestep of 1/50s. This result shows that even the multi-step MSE is not sufficient criteria to evaluate the model for planning. The optimal time step to obtain the highest reward depends the domain. Cartpole swingup and finger spin require fine time step of 1/ 1/100s,
16

---

Under review as a conference paper at ICLR 2022
ball in cup catch only solves the task using very coarse time step of 1/25s and walker walk performs well for time steps between 1/40 - 1/100s.
Stochastic Model The optimal time steps for the stochastic models are identical to the optimal discretization of the deterministic model. Therefore, the optimal time step depends on the domain but not the model representation. In contrast deterministic models, stochastic models perform slightly better terms of reward for more coarse time steps. However, for very fine time steps the stochastic models perform solves the task for all time steps < 1/10s while the stochastic model fails for time steps < 1/50s. Furthermore, the MSE of ball in a cup shows that the stochastic models can diverge for very fine time steps.
than deterministic models. For example, the deterministic model
Cartpole
Swingup
Finger Spin
Cup
1916
1860
Daternintstts Model
258H
Free 20042
606
669
610
660-
Free 1001/
Free 50H2
400
Free 4092
2507
260
200
3
Updates [1ee0:]
Updates [3e-0h]
Updates [Leett]
Updates [1eth)]
Cartpole Swingup
Finger
Spin
Ball In Cup
Catch
Walker Malk
Free 566Hz
Deterntnistic Nodel
Free 40092
Free 25842
2009
Square
16662
20-4
2002
Free 1817
2
Updates [1ee05]
Updates [3e-05]
Updates Ele+053
[3ex05]
Figure 10: The learning and loss curves for different time discretization averaged over seeds. The shaded regions highlight the 20/80 percentile. For too fine and too coarse time steps, the MSE increases. The lowest MSE obtained for time steps between 1/40- - 1/100s. The optimal time step to obtain the highest reward depends on the domain, while caripole swingup and finger spin require a fine time step, ball in cup catch only solves the task using a very coarse time step.
17

---

Under review as a conference paper at ICLR 2022
Cartpote
Swingup
Finger Spin
Balt In Cup
Catch
Wat Walk
Fres 566H2
Free 40012
Stochastic Noda
torratize towrr
820
803
n50
300
Freq 2581/
Free 26042
600
600
10M9
5002
430
200
260
230
200-
1017
Updates [1ee05]
Optates 13e-051
Updates [lee05]
1ex051
Cartpolt Swingup
Finger
Spin
Balt Cup
Catch
Wal Kere
Malk
10a
Free 568H2
Stochastic Nodel Maa squared EIrOF
25082
Frea 2512
Free 2012
Fres 1811
[1c+85]
Updatos [3c-65]
Updates [1:+85]
[2c+65)
The learning and loss curves for the stochastic model with different time discretization averaged over 5 seeds. The shaded area highlights the 20/80 percentile. Similar the deterministic model, the optimal time step depends on the domain. The optimal time steps are identical to the optimal time steps of the deterministic model.
Figure 11:
18


Reviews:
[{"review_id": "adj2MsWa_W5", "reviewer": "Reviewer_nmSn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper shows the following advantage:\n\n(+) This work tries to provide the community a deeper understanding of the predictive planning model on the observation space, by comparing several commonly used model designing choices and providing a conclusion on the choices that generally work better. The comparison takes the network structure, loss function, and stochasticity of models into consideration.\n\nIn the meanwhile, there remain several concerns.\n\n(-) The stochastic and deterministic models are both applied to deterministic environments, while the stochastic model may lose part of its advantage in this case. Considering the comparison between the stochastic and deterministic model is one of the important experiments in this work, it may be worth classifying the environments into 2 corresponding types, then looking into how stochastic and deterministic models perform in each of them. It can also verify whether the current conclusion still holds in stochastic environments.\n\n(-) Another concern is about the random seeds. There are 5 random seeds in total. The average and confidence interval based on 5 random seeds may not be accurate enough, therefore it is hard to justify whether some conclusions are valid or not. For example, in Section 4.4 and Figure 4, the result indicates that network ensemble improves the performance more significantly in stochastic models, while in the deterministic model, ensemble only matters in Ball-in-cup. However, when reading the plots, both Finger and Walker have overlapped areas when indicating the 20/80 percentile. It is not clear to me if the above conclusion still holds when adding more random seeds. As a paper empirically compares different model designing choices, I believe showing reliable averaged performance is important. Thus, using more random seeds (ideally above 10) might make the conclusion more clear.\n\n(-) It is mentioned in the paper that the experimental results obtained in this work are partially inconsistent with previous work (Chua et al. 2018), however, there is a lack of discussion on what may cause this inconsistency, which makes it hard to justify whether the result provided in this work is reliable or not.\n\n(-) When comparing the stochastic and deterministic models, the networks have different structures. It causes differences including capacities and numbers of parameters to train. These differences can affect the learning efficiency of the agent. Thus it is not clear to me whether the conclusion about deterministic and stochastic models is accurate.\n", "summary_of_the_paper": "This paper investigates different choices of designing the planning model which predict the next state on the observation space. The paper compares (1) prediction with deterministic or stochastic models, (2) using 1-step forward prediction loss or multi-step forward prediction loss to train the model, (3) prediction with single network or network ensemble, and (4) using perfect observation or add gaussian noise to the observation. \nThrough the above experiments, the paper suggests that deterministic models should be trained with multi-step loss, while stochastic models work better when the 1-step log-likelihood loss is used and noise is added to the observation space. In both cases, ensemble networks tend to show better performance.\n", "summary_of_the_review": "Overall, I think the paper is not good enough to get accepted yet. On one hand, the paper focuses on understanding and comparing different settings of designing predictive models in the literature, which offers the community a better understanding of planning. On the other hand, there remain several concerns. My major concern is that the experiment setting may affect the accuracy of the conclusion provided in the paper. Focusing too much on deterministic environments makes me wonder if the conclusion still holds when there exists stochasticity in the environment. This may cause an unfair comparison between the deterministic model and stochastic model, which is a major part of this work. Furthermore, the overlapping area when showing the 20/80 percentile suggests more than 5 random seeds are needed. Moreover, using different network architectures can affect the learning efficiency thus causing inaccuracy when comparing these models.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XRr3hCyCsxq", "reviewer": "Reviewer_PwSp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong:\n* I like the overall idea of the paper, not to propose another new method, but instead make a systematic comparison of some important aspects of model-based RL. I think we could use more papers like this. \n* The paper is clearly written, has many results, and good additional videos online. \n\nWeak:\n* One of my doubts is the diversity of the tasks. Since the Humanoid does not give any results, they essentially compare performance on 4 tasks. Given that this is a benchmarking paper, I think this is a somewhat narrow range. I think the results partially illustrate this, since on multiple occasions the results are somewhat inconclusive (on one or two it works better, on the other two on parr or worse).\n* I think you missed some important related work [1], which already extensively studied the effect of multi-step losses in RL tasks.\n* I doubt whether it makes full sense to study stochastic models on deterministic environments only (Sec 4.1). I see that they may still be beneficial, but to really say something about the necessity of stochastic predictions, I think you would need to at least have a few stochastic environments as well. \n* You decide to not propagate uncertainty over multiple steps because it cannot be done analytically through non-linear neural networks (Sec 4.3), which is true, but you can easily use a particle method for this right, i.e., use sampling? This is a common solution. \n* While is the paper is generally well written, you incidentally have some strange sentences with missing words. For example: “Therefore, the modeling the contact is not the main problem but the model exploitation in the vicinity of strong non-linear changes”. I think the paper needs one more careful proofreading. \n\n[1] Chiappa, Silvia, et al. \"Recurrent environment simulators.\" arXiv preprint arXiv:1704.02254 (2017).", "summary_of_the_paper": "This paper systematically studies a range of design considerations in model learning: 1) deterministic versus stochastic models, 2) 1-step versus multi-step targets, 3) the number of instances in an ensemble model, and 4) the effect of input noise to make the model more robust. They find 1) both deterministic an stochastic models work, 2) a medium horizon (3-5 steps) works best & model MSE is a poor indicator of planning performance, 3) ensembles improve performance up to about 5 components, and 4) the benefit of input noise varies over tasks. ", "summary_of_the_review": "This paper has positive and negative sides for me. As mentioned above, I like a comparison/benchmarking effort, which I believe the field indeed needs, and the authors have taken a nice, systematic approach, and written a well-structured paper. On the downside, I kind of doubt whether I learned too much from the results, since 1) the results vary quite a bit over tasks, which I think might be due to the relatively small set of tasks, 2) the results on stochasticity are limited by the set-up, which excludes stochastic environments, and 3) some other results were already partially known (like the multi-step prediction effects and poor performance of MSE). I am therefore a bit in doubt, because the paper does still have its merit in systematically comparing different element of model learning in deterministic control tasks. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XH9WHbeNOSb", "reviewer": "Reviewer_xfZq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n* Interesting problem setting of understanding the role of design choices for model learning in model-based RL.\n\nWeaknesses:\n* Takeaways from the paper don’t seem to be conclusive (rf. “No single design choice works significantly better”). It doesn’t satisfactorily answer the question the work sets out. It might be helpful to frame the paper as informing future researchers who want to learn accurate dynamics models: How can this work help inform them which methods to try out vs. be eliminated?\n* Investigation is limited in terms of environments explored. Specifically, the study only focuses on four control environments which are not very representative of settings where you would want to do model-based RL (e.g. the related works section motivates the use of accurate model prediction for robotics, but only applies it to simulated, deterministic environments). I would recommend the authors to extending their analysis in a robotics-based environment, e.g. RoboSuite [1], PlayRoom [2], Meta-World [3].\n* Investigation is limited in terms of design choices explored. For example, memory-based architectures are omitted, despite playing a key role in domains that are partially observable (e.g. velocity of agents cannot be inferred from their setup which makes model-learning hard). It would be nice if their analysis could be expanded to cover memory-based architectures. \n\n[1] robosuite: A Modular Simulation Framework and Benchmark for Robot Learning\n[2] Learning Latent Plans from Play\n[3] Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning", "summary_of_the_paper": "This work ablates some of the design choices that go into learning a dynamics model for control-based environments. They ablate 4 choices: use of deterministic vs. stochastic models, multistep losses, network ensembles, and input noise. The authors study this in a few of the DeepMind control suite environments.", "summary_of_the_review": "--- methodology ---\n\nI find the question of the paper interesting, but the methodology doesn’t feel satisfactory to me. A few things appear odd to me:\n* Running a stochastic model in a deterministic setting seems like a weird setting for evaluating the benefits of stochastic models. Feels like an unfair comparison. Could the authors try running this on environments that elicit stochasticity, and where this choice might seem more natural? \n* The setup of the paper seems odd: The data collection assumes that we already have a well-trained model-free agent that already does the task we care about...We then learn a dynamics model from this policy and deploy the model-based version. It seems that this setup doesn’t even need a model-based method in the first place. So why should we care about these domains that the authors evaluate on? Plus, the most challenging domain - humanoid - doesn’t seem to lend itself to model analysis in this work.\n\n--- paper narrative & motivation --- \n\nIn some sense, the takeaway of the paper feels negative to me: there isn’t really a consistent way of learning accurate / useful dynamics models, so future researchers might just need to empirically test all the design choices for themselves and see what works best. I wish the work could do more to explore a bit more into what kinds of errors these dynamics models incur, and why these approximation errors are fatal - e.g. humanoid completely fails and so all of its results are omitted, but … why are those model errors that bad compared to other domains? How can this failure case inform us in how to better learn dynamics models? \n\nThere are several areas in the paper that come unmotivated or are not well contextualized. For example, I don’t really understand the dynamics manifold argument in Section 4.5 when the authors say that long time horizon predictions lead to inputs being outside the desired manifold, yet claim that adding noise also “causes the inputs to lie outside the manifold” but the model is more robust to predict solutions returning to the manifold. It seems like noise will only, up to a certain extent, help you in the short run but long horizon settings will still be difficult to predict accurately. \n\nWhy would we consider having a model-based method in the first place? It would have been nice to see explicit sample gains in having model-based methods - e.g. number of data samples saved for these settings [or considering some form of computational budget]. \n\nThe work also seems to assume the best case scenario of having an optimal (model-free) policy which is oftentimes not available. It would have been interesting to investigate the approximation tradeoffs when a well-trained policy is not available as a data source for the to-be-learned dynamics model. \n\n--- experiments ---\n\n* What does the MSE graph look like for Humanoid? \n* Observation that MSE is not a good predictor of planning performance when using a learned model seems non intuitive and potentially an interesting thing to further investigate. \nE.g. Figure 3: Why is it that for longer horizons, MSE goes down and reward goes down? Since you only execute the first action anyway, how frequently does the plan deviate from what an optimal policy would on average do? \nIf you were to enforce the proper physical constraints in the prediction, does that resolve the exploitation problem? Since you have access to the simulator and you’re only predicting observation joints, this feels like a conjecture that can be easily verified. This would also disentangle whether physical constraints are the issue vs. the prediction being wrong at critical states of the trajectory.\n* Why did the authors choose to use MPO policy as the base policy? \n* Could the authors elaborate why the multi-step loss helps up to a certain number of steps for those domains? \n* Is state dimensionality the core issue in preventing humanoid from working? Could this hypothesis be testing on 4k+1 observation-space swimmer environment where k varies? \n* It’s impossible to have complete coverage of previous methods, but I feel like this work is missing a key class of dynamics models which are a function of history. E.g. in robotics settings (already cited by the authors), this includes concatenating states or having a recurrent neural network. This helps in settings where the environment is partially observable. Another thing is that it’s common practice to predict the difference in states. Authors should consider including experiments there.\n* Related work is tied to robotics (and this is where a good dynamics model is super critical, not the synthetic simulated environments used here), but no robotics-like environment is used for evaluation.\n* It’s surprising to see that ball-in-cup can (a) tolerate that much noise and (b) improves with increasing noise. Why is that the case for this domain, and not for the others? Eg. I could imagine Finger-Spin to also somewhat tolerate noise, but the fact that it’s sensitive makes me think that the observation ranges matter a lot in this study. Could the authors elaborate on this? \n\nNits: \n* Title is not reflective of the paper goal. Currently reads quite broad - doesn’t indicate that this paper is about understanding the design choices for dynamics models. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "3o4K9Uj6PW2", "reviewer": "Reviewer_sY5Q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I enjoyed reading this paper. The main weakness are that only four benchmarks were used to arrive at these conclusions, and some elements of the methodology are not clear. \n\nI have some questions:\n1) Data Used: If I understand sec 4.1 correctly, you are actually using fixed data sets to evaluate the rewards of MPC. While I understand you intentionally did this to ease comparison, you remove a key factor exploration of MBRL: exploration. This weakens your comparisons for stochastic models in particular, since the uncertainty of a model can interact with the planning algorithm to learn more efficiently (or not). I also do not really understand how this works, what are the \"Updates\" on the x-axis in figures, number of data points accessed from this fixed data set? I think it's important to be clear here, there is no connection with what the planner explores? \n\n2) Are you drawing enough samples for the stochastic models? Stochastic models are very vulnerable to drawing too few samples. Some data on this would be useful. \n\n3) Multi-step: Fig. 3 is curious, why are the results for deterministic, steps = 1 are worse (e.g. finger-spin, ball-in-cup) than those for deterministic 1-step prediction in Fig. 2? Are these 95% CI?\n\n4) The multi-step results were the most surprising to me, the optimal number of steps also seems to vary considerably between tasks, from 5-40 steps. Any insight why this happens?\n\n5) Ensembles in Fig. 4: Again, is not 1 ensemble the same as a regular NN in earlier figures? Finally, since 10 ensembles performs betst, what would happen if you used 15? Additionally, you basically get more samples (and uncertainty) by adding ensembles, so there could be a sampling factor impact results here as well.\n\n6) Input Noise: Is this for the planner or the model learning? If you use a fixed data set, I don't see how this could be for the model? If it's for the planner, it seems more like state noise than *observation* noise (c.f. section title) since the future states will all be affected? Finally, shouldn't ball-in-cup noise=0 match earlier figures for stochastic model?\n\nMinor:\n\n- The figures are displaced 1-2 pages from the text, which is bit confusing.\n\n- \"The obtained reward is marginally below the reward obtained by planning with the true model.\" - The gap in mean reward seems to be >10% in most of these example, which is at least mathematically not marginal, even though their behavior may visually seem very close.\n\n- Normalized MSE vs. standardization, did you try with state and actions that weren't centered around zero in your benchmarks? What you are doing here is basically the variance compensation of standardization, but not the mean one.\n\n- Why in-text [Link] tags instead of /url clickable footnotes?\n\n- It might be interesting to see if these model-based approaches actually could converge to near-optimal rewards if you let them run longers. \n\n- Note: \"Instead of using a deterministic model, Chua et al. (2018) proposed to use stochastic models. This\nmodel predicts the change of the system using a normal distribution.\". I would phrase it as \"learns the variance of the predictions\". MSE is of course mathematically identical to neg log-likelihood for Gaussian noise with isotropic covariance, it's just that the variance is never estimated (because parameters are invariant of this). What is new here is learning state-dependent noise, presumably modelled as an extra output of the network (often called aleatoric uncertainty as you note). I am not entirely sure what you mean by the lower entropy bound, perhaps just that the planner will tend to avoid poorly modelled parts of the state space since the noise is presumably higher?\n\n- Typos: \"rollouts of the provide\", \"significant more data\", \" the modeling the \"\n", "summary_of_the_paper": "The authors compare a number of recent and commonly used learning strategies for model-based planning on four typical RL control environments. The findings provides some useful insights for practitioners as well as for future research directions.", "summary_of_the_review": "While some design choices made in the study could be clearer, and I would have preferred to have more benchmark environments, the paper is well written and the experiments appear carefully designed. I believe ablative studies like this are very useful for the research community and practitioners alike.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the effect of different design choices related to learning a dynamics model. The reviewers uniformly agree that the topic of the paper, systematically studying different design choices, is important. Furthermore, the paper is very well written. However, there are a number of weaknesses as well, that limit the relevance of this work. Arguably, the main weakness is that the results are inconclusive: there is no single design choice that is better, a conclusion that provides little guidance for researchers working in this space. Another weakness is that the study focuses on only 4 domains. And while performing such a study on a much broader set of domains can be prohibitively expensive, that doesn't take away from the fact that it is hard to draw strong conclusions from such a small set of tasks. For these reasons, I recommend rejection.
Decision: Reject