Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose?
Authors: Balázs Kégl
Gabriel Hurtado
Albert Thomas
Venue: ICLR 2021 Poster (2021)
Keywords: model-based reinforcement learning
generative models
mixture density nets
dynamic systems
heteroscedasticity

Abstract: We contribute to micro-data model-based reinforcement learning (MBRL) by rigorously comparing popular generative models using a fixed (random shooting) control agent. We find that on an environment that requires multimodal posterior predictives, mixture density nets outperform all other models by a large margin. When multimodality is not required, our surprising finding is that we do not need probabilistic posterior predictives: deterministic models are on par, in fact they consistently (although non-significantly) outperform their probabilistic counterparts. We also found that heteroscedasticity at training time, perhaps acting as a regularizer, improves predictions at longer horizons. At the methodological side, we design metrics and an experimental protocol which can be used to evaluate the various models, predicting their asymptotic performance when using them on the control problem. Using this framework, we improve the state-of-the-art sample complexity of MBRL on Acrobot by two to four folds, using an aggressive training schedule which is outside of the hyperparameter interval usually considered.

Published as a conference paper at ICLR 2021
MODEL-BASED MICRO-DATA REINFORCEMENT LEARN- ING: WHAT ARE THE CRUCIAL MODEL PROPERTIES AND WHICH MODEL TO CHOOSE?
Balázs Kégl, Gabriel Hurtado, Albert Thomas Huawei Noah's Ark Lab, Paris, France [balazs. kegi, gabriel.... www.eaddohraooooo albert. thomas @hhaawe..com com
ABSTRACT
We contribute to micro-data model-based reinforcement learning (MBRL) by rig- orously comparing popular generative models using fixed (random shooting) control agent. We find that on an environment that requires multimodal posterior predictives, mixture density nets outperform all other models by large margin. When multimodality is not required, our surprising finding is that we do not need probabilistic posterior predictives: deterministic models are on par, in fact they consistently (although non-significantly) outperform their probabilistic counter- parts. We also found that heteroscedasticity at training time, perhaps acting as a regularizer, improves predictions at longer horizons. At the methodological side, we design metrics and an experimental protocol which can be used to evaluate the various models, predicting their asymptotic performance when using them the control problem. Using this framework, we improve the state-of-the-arr sample complexity of MBRL on Acrobot by to four folds, using aggressive training schedule which is outside of the hyperparameter interval usually considered.
INTRODUCTION
Unlike computers, physical systems do not get faster with time (Chatzilygerouut et al., 2020). This is arguably one of the main reasons why recent beautiful advances in deep reinforcement learning (RL) (Silver al., 2018; Vinyals al., 2019; Badia et al., 2020) stay mostly in the realm of simulated worlds and do not immediately translate to practical success in the real world. Our long term research agenda is to bring RL to controlling real engineering systems. Our effort is hindered by slow data generation and rigorously controlled access to the systems.
Micro-data RL is the term for using RL on systems where the main bottleneck or source of cost is access to data (as opposed to, for example, computational power). The term was introduced in robotics research (Mouret, 2016; Chatzilygeroudis et al., 2020). This regime requires performance metrics that put as much emphasis on sample complexity (learning speed with respect to sample size) as on asymptotic performance, and algorithms that are designed make efficient use of small data. Engineering systems are both tightly controlled for safety and security reasons, and physical by nature (so do not get faster with time), making them a primary target of micro-data RL. At the same time, engineering systems the backbone of today's 's industrial world: controlling them better may lead to multi-billion dollar savings per year, even we only consider energy efficiency."
Model-based RL (MBRL) builds predictive models of the system based on historical data (logs, trajectories) referred to here as traces. Besides improving the sample complexity of model-free RL by orders of magnitude (Chua et al., 2018), these models can also contribute adoption from the human side: system engineers can "play" with the models (data-driven generic "neural" simulators) and build trust gradually instead of having to adopt black-box control algorithm at once (Argenson & Dulac-Arnold, 2020). Engineering systems suit MBRL particularly well in the sense that most system variables that are measured and logged are relevant, either to fed to classical control or to a human operator. This means that, as opposed to games in which only few variables (pixels) are relevant for winning, learning a forecasting model in engineering systems for the full set of logged variables is arguably an efficient use of predictive power. It also combines well with the micro-data learning principle of using every bit of the data to learn about the system.
1 % of the yearly energy of the US manufacturing sector roughly hillion dollar [link, link].

---

Published as a conference paper at ICLR 2021
Robust and computationally efficient probabilistic generative models are the erux of many machine learning applications. They are especially one of the important bottlenecks in MBRL (Deisenroth & Rasmussen, 2011; Ke et al., 2019; Chatzilygeroudis et al., 2020). System modelling for MBRL is essentially a supervised learning problem with AutoMI. (Zhang et al., 2021): :): models need to be retrained and, if needed, even retuned hundreds of times, on different distributions and data sets whose size may vary by orders of magnitude, with little human supervision. That said, there is little prior work on rigorous comparison of system modelling algorithms. Models are often part of a larger system, experiments are slow, and it is hard to know if the limitation or success comes from the model or from the control learning algorithm. System modelling hard because i) data sets are non-i.i.d., and ii) classical metrics on static data sets may not be predictive of the performance on the dynamic system. There is no canonical lata-generating distribution as assumed in the first page of machine learning textbooks, which makes it hard to adopt the classical train/test paradigm. At the same time, predictive system modelling is a great playground and it can be considered as an instantiation of self-supervised learning which some consider the "greatest challenge in ML and Al of the next few years".
We propose to compare popular probabilistic models the Acrobot system study the model properties required to achieve state-of-thh-art performances. We believe that such ablation studies are missing from existing "horizontal" benchmarks where the main focus is on state-of-the-ar com- binations of models and planning strategies (Wang et al., 2019). We start from a family of flexible probabilistic models, autoregressive mixtures learned by deep neural nets (DARMDN) (Bishop, 1994; Uria et al., 2013) and assess the performance of its models when removing autoregressivityy multimodality, and heteroscedasticity, We favor this family of models it is easy i) to compare them on static data since they come with exact likelihood, ii) to simulate from them, and iii) to incorporate prior knowledge on feature types. Their greatest advantage is modelling flexibility: they can be trained with loss allowing heteroscedasticity and, unlike Gaussian processes (Deisenroth & Rasmussen, 2011 I; Deisenroth et al., 2014), deterministic neural nets (Nagabandi et al., 2018; Lee et al., 2019), multivariate Gaussian mixtures (Chua et al., 2018), variational autoencoders (VAE) (Kingma& & Welling, 2014; Rezende et al., 2014), and normalizing flows (Rezende & Mohamed, 2015), deep (autoregressive) mixture density nets can naturally and effortlessly represent a multi- modal posterior predictive and what we will call y-interdependence (dependence among system observables even after conditioning the history).
We chose Acrobot with continuous rewards (Sutton, 1996; Wang et al., 2019) which we could call the "MNIST of MBRL" for three reasons. First, it is simple enough answer experimental questions rigorously yet it exhibits some properties of more complex environments so we believe that our findings will contribute to solve higher dimensional systems with better sample efficiency as well as better understand the existing state-of-ther of-thh-arttttttt solutions. Second, Acrobot is one of the systems where i) random shooting applied on the real dynamics is state of the art in an experimental sense and ii) random shooting combined with good models is the best approach among MBRL (and even model-free) techniques (Wang et al., 2019). This means that by matching the optimal performance, we essentially "solve" Acrobot with a sample complexity which will be hard to beat. Third, using a single system allows both a deeper and simpler investigation of what might explain the success of popular methods. Although studying scientific hypotheses precedence (Abbas et al., 2020), we leave open the possibility that our findings are valid only on Acrobot (in which case we definitely need to understand what makes Acrobot special).
single system is not without
There are three complementary explanations why model limitations lead to suboptimal performance in MBRL (compared to model-free RL). First, MBRL learns fast, but it converges suboptimal models because of the lack of exploration down the line (Schaul et al., 2019; Abbas et al., 2020). We argue that there might be second reason: the lack of the approximation capacity of these models. The two reasons may be intertwined: not only do we require from the model family to contain the real system dynamics, but we also want it to be able represent posterior predictive distributions. which i) are consistent with the limited data used to train the model, ii) are consistent with (Icarnable) physical constraints of the system, and iii) allow efficient exploration. This is not the "classical" notion of approximation, it may not be alleviated by simply adding more capacity to the function representation; it needs to be tackled by properly defining the output of the model. Third, models are trained to predict the system one step ahead, while the planners need unbiased multi-step predictions
Phttps://www fwc.back.ccc.... com/ cote:/26770c1111110 7226772267777 10.677142//osts//// /0t.//00015590000220
2

---

Published as a conference paper at ICLR 2021
which often do not follow from one-step optimality. Our two most important findings nicely comment on these explanations.
Probabilistic models are needed when the system benefits from multimodal predictive uncertainty. Although the real dynamics might be deterministic, multimodality seems to be crucial to properly handle uncertainty around discrete jumps in the system state that lead to qualitatively different futures.
When systems do not exhibit such discontinuities, we do not need probabilistic predic- tions at all: deterministic models are on par, in fact they consistently (although non- significantly) outperform their probabilistic versions. We also found that heteroscedas- ticity at training time, perhaps acting as regularizer, improves predictions at longer horizons (compared to classical regressors trained to minimize the mean squared error one step ahead).
Note that while our hypotheses and experimental findings are related to the grand debate on how to represent and categorize uncertainties (Deisenroth & Rasmussen, 2011; Gal, 2016; Gal al., 2016; Depeweg et al., 2018; Osband et al., 2018; Hullermeier & Waegeman, 2019; Curi et al., 2020), we remain agnostic about which the right representation by concentrating on posterior predictives on which the different approaches (e.g., Bayesian or not) are directly empirically comparable. We contribute to the debate by providing empirical evidence on noiseless system, demonstrating unexplained phenomena even when uncertainties purely epistemic.
We also contribute to good practices in micro-data MBRL by building an extendable experi- mental protocol in which we design static data sets and measure various metrics which may correlate with the performance of the model on the dynamic system. We instantiate the protocol by a simple setup and study models systematically in fast experimental loop. When comparing models, the control agent or learning algorithm is part of the scoring mechanism. We fix it to a random shooting model predictive control agent, used successfully by (Nagabandi et al., 2018 8), for fair comparison and validation of the models. Our reproducible and extensible benchmark is made publicly available https:://// / / /git hub. rrmpp kits/ /r s simulator.
THE FORMAL SETUP
Let Ta = ((V1, a1). .... (Ur, )) be system trace consisting of T steps of observable-action pairs (y:, a.): given an observable y, of the system state at time I, an action a, was taken, leading
to a new system state observed as 31+1- The observable vector y, = (vl (ye...., numerical or categorical variables, measured on the system at time (. The action vector a, contains d, numerical or categorical action variables, typically set by a control function a, = т (T+-1,9)) of the history T-1 and the current observable yr (or by a stochastic policy a (Tr 9.)).
g/ dy contains dy
The objective of system modelling is to predict y/++1 given the system trace Te. There are applications where point predictions Ut+1 fT.) sufficient, however, in most control applications (e.g..
reinforcement learning or Bayesian optimization) we need to access the full posterior distribution of Ve+1 1 TT take into consideration the uncertainty of the prediction and/or to model the randomness of the system (Deisenroth & Rasmussen, 2011; Chua et al., 2018). Thus, our goal is to learn 1( (T.).
To convert the variable length input (condition) T = (()), a1), (1),...,,(1 ax)) into a fixed length state vector 8, we use fixed feature extractor s; = fIz((TTT). After this step, the modelling simpli- fies to classical learning of (conditional) multi-variate density p(y++1 8;) (albeit non-i.i.d. data). In the description of autoregressive models will the notation x! 8, and a/ (Vi41....,, w/41 8:) for j >> for the input (condition) of the jth autoregressive predictor py (v/411)) See Appendix A for more details on the autoregressive setup.
2.1 MODEL REQUIREMENTS
We define seven properties of the model p that are desirable if to used MBRL, These restrict and rank the family of density estimation algorithms to consider. Req (R1) is absolutely mandatory for rajectory-sampling controllers, and Req (R2) is mandatory in this paper for using our experimental toolkit to its full extent. Reqs (R3) to (R7) are softer requirements which i) qualitatively indicate the potential performance of generative models in dynamic control, and/or ii) favor practical usability on real engineering systems and benchmarks, Table I provides summary on how the different models
3

---

Published as a conference paper at ICLR 2021
satisfy (or not) these requirements. We note that depending the application and the desired control frequency of the system, one may also require models with fast prediction times.
(RI) It should be computationally easy to properly simulate observables Y++1 p(-IT;) given the system trace to interface with popular control techniques that require such simula- tions. Note that it is then easy to obtain random traces of arbitrary length from the model by applying p and π alternatel..
(R2) Given 91+1 and Tr, should be computationally easy to evaluate p(y 1 (T)) to obtain a likelihood score in order to compare models on various traces. This means that p(y TT) > 0 and f p(yl (T = should be assured by the representation of p, without having to go through sampling, approximation, or numerical integration.
(R3) We should be able to model y-interdependennce dependence among the dy elements of Ve+1 = -Vi v(-11) given Tr. In our experiments we found that the MBRL perfor- mance was not affected by the lack of this property, however, favor since the violation of strong physical constraints in telccommunication or robotics may hinder the acceptance of the models (simulators) by system engineers. See Appendix B for further explanation.
(R4) Heteroscedastic models are able to vary their uncertainty estimate function of the state or trace Tr. Abbas et al. (2020) show how to use input-dependent variance to improve the planning. We found that even when using the deterministic prediction at planning time, allowing heteroscedasticity at training time alleviates error accumulation down the horizon.
(R5) Allowing multi-modal posterior predictives seems be crucial to properly handle uncer- tainty around discrete jumps in the system state that lead to qualitatively different futures.
(R6) We should be able to model different observable types, for example discrete/continuous, finite/infinite support, positive, heavy tail, multimodal, etc. Engineers often have strong prior knowledge on distributions that should be used in the modelling, and the popular (multivariate) Gaussian assumption often leads to suboptimal approximation.
(R7) Complex multivariate density estimators rarely work out of the box on new system. We are aiming at reusability of our models (not simple reproducibility of our experimental results). In the system modelling context, density estimators need to be retrained and retuned automatically. Both of these require robustness and debuggability: self-tuning and gray-box models and tools that can help the modeler to pinpoint where and why the model fails. This requirement is similar to what is often imposed on supervised models by application constraints, for example, in health care (Caruana et al., 2015).
2.2 EVALUATION METRICS
We define a set of metrics to compare system models both on fixed static traces T (Section 2.2.1) 1) and on dynamic systems (Section 2.2.2). We have triple aim. First, contribute moving the RL community towards a supervised-lcarmin be made more precise. Second, we define an experimental process where models can be evaluated rapidly using static metrics before having to run long experiments on the dynamic systems. Our methodological goal is identify static metrics that predict the performance of the models on the dynamic system. Third, we provide diagnostics tools to the practical modeller to debug the models and define triggers and alarms when something goes wrong on the dynamical system (e.g., individual outliers, low probability traces).
like rigorous evaluation process where claims can
2.2.1 STATIC METRICS
We use four metrics on our static "supervised" experiment to assess the models p(y,-1lst). We define all metries formally in Appendix C. First we compute the (average) log-likelihood of p on a test trace Tr- for those models that satisfy Req (R2). Log-likelihood is unitless metrics which is hard to interpret and depends on the unit in which its input is measured. To have better interpretation, we normalize the likelihood with a baseline likelihood of a multivariate independent unconditional Gaussian, to obtain the likelihood ratio (LR) metrics. LR is between 0 (although LR < 1 usually indicates bug) and oo, the higher the better. We found that LR works well in i.i.d. setup but distribution shift often causes "misses": test points with extremely low likelihood. Since these points dominate LR, decided to clamp the likelihood and compute the rate of test points with likelihood
4

---

Published as a conference paper at ICLR 2021
less tham' Pmin 10-47x10-, This outlier rate (OR) measures the "surprise" of model on trace T. OR is between 0 and 1, the lower the better. Third, we compute the explained variance (R2) to quantify the precision of the predictors. We prefer using this metrics over the MSE because it is normalized so it can be aggregated over the dimensions of y. R2 is between and 1, the higher the better. Fourth, for models that provide marginal CDFs, compute the Kolmogorov-Smirnor (KS) statistics between the uniform distribution and the quantiles of the test ground truth (under the model CDFs). Well-calibrated models have been shown to improve the performance of MBRL algorithms (Malik et al., 2019). KS is between O and 1, the lower the better.
All our density estimators are trained to predict the system one step ahead yet arguably what matters is their performance at longer horizon L specified by the control agent. Our models do not provide explicit likelihoods steps ahead, but we can simulate from them (following ground truth actions) and evaluate the metrics by Monte-Carlo Carlo estimate, obtaining long horizon metrics KS(T.) and R2(... In all our experiments we use L, = 10 with 100 Monte Carlo traces, and, for computational reasons, sample the test set at 100 random positions, which explains the high variance on these scores.
2.2.2 DYNAMIC METRICS
Our ultimate goal is to develop good models for MBRL so we also measure model quality in terms of the final performance. For this, we fix the control algorithm to random shooting (RS) (Richards, 2005; Rao, 2010) which performs well on the true dynamics of Acrobot as well as many other systems (Wang et al., 2019). RS consists a random search of the action sequence maximizing the expected cumulative reward over a fixed planning horizon L. The agent then applies the first action of the best action sequence. We use L = 10 and generate n = 100 random action sequences for the random search. For stochastic models we average the cumulative rewards of 5 random trajectories obtained for a same action sequence. We note that one could achieve better results by using a larger n or the cross entropy method (CEM) (de Boer et al., 2004; Chua et al., 2018). One could also consider more complex planning strategies (Wang& & Ba, 2020; Argenson & Gulac-Arolod 2020). However we judge RS with 100 to be sufficient for our study (see Appendix D for more details). We present here the MBRL loop and notations which will be needed to define the dynamic metrics.
1. Run random policy (11) for T = 200 steps, starting from an initial "seed" trace T%
(typically a single-step state ( = (yo:)) .)) to obtain a random initial trace Tf Let the epoch index be 7 1.
2. Learn p!r) on the full trace TxxT= = 1
3. Run RS policy (T) using model p(7), (re)starting from T/ to obtain trace Tf (T+1)
4. If < N. let = T + and go to Step 2, otherwise stop.
Given the formal algorithm, we can now elaborate what we mean by system modelling for MBRL being essentially a supervised learning problem with AutoML (and why (R7) is important). Zhang et al. (2021) make similar argument in paper that came out independently of ours. In Step 2. the chosen model needs to be retrained and, if needed, retuned, on data sets TxxT of different distribution whose size may vary by orders of magnitude, with little human supervision. This does not mean we need to do full hyperopt in every episode T, rather that p(T) should be robust: trainable without human babysitting over a range of different distributions and data sizes. A single catastrophic learning failure (e.g. getting stuck in initial random function) means the full MBRL loop goes off the rail. Models that need to be retuned (because of sensitivity to hyperparameters) must have the retuning (AutoML) feature encapsulated into their training. The models that ended up on the top were not sensitive to the choice of hyperparameters,
we did not need to retune them in every iteration.
MEAN ASYMPTOTIC REWARD (MAR) AND RELATIVE MAR (RMAR). Given a trace TT and a reward F; obtained at each step i, we define the mean reward as R(Tr) = + Σi rt. The mean reward in iteration 7 is then MR(T) (T) = (T{ Our measure of asymptotic performance, the mean asymptotic reward, is the mean reward in the second half of the epochs (after convergence; we set N
As a salute to 5-sigma, using the analogy of the MBRL loop (Section 2.2.2) as the iterated scientific method. A 'The common practice is not to normalize the cumulative reward by the (maximum) episode length T, which makes it difficult to immediately compare results across papers and experiments. In micro-data RI,, where T is hyperparameter (vs. part of the experimental setup), we think this should be the common practice.
5

---

Published as a conference paper at ICLR 2021
in such way that the algorithms converge after less than N/2 epochs) MAR = N Σ MR( To normalize across systems and to make the measure independent of the control algorithm use on top of the model, we define the relative mean asymptotic reward RMAR = (MAR MAR ( (MARup MA where MAR is the mean asymptotic reward obtained by running the same control algorithm on the dynamies oM 2.104 in our experiments on Acrobot"), and Mm is the mean asymptotic reward obtained by running the initial random policy on the true dynamics (MAR 0.12 in our experiments Acrobot). This puts RMAR between 0 and (the higher the better).
MEAN REWARD CONVERGENCE PACE (MRCP(70)). To assess the speed of conver- gence, we define the mean reward convergence pace MRCP(p%) as the number of steps needed to achieve p% of (MARppt () using the running average of epochs MRCP(p%) T arg min, { [ MR(T) MAR > p% x (MARopt MAR The unit of MRCP()%) is system access steps, not epochs, first to make it invariant to epoch length, and second because in micro-data RL the unit of cost is system access step. We use 70 in our experiments.
2.3 THE EVALUATION ENVIRONMENT
The Acrobot benchmark system has four observables y = [02, 02, 01, 0g]: 0, the angle to the vertical axis of the upper link; 8g the angle of the lower link relative to the upper link, both being normalized to [- m]: 0, and O2 the corresponding angular momenta. The action is a discrete torque on the lower link 0, 1). We only y, as the input to the models but augment it with the sines and cosines of the angles, so s; [01, sin 01, cos f1, 822 sin 82, cos 02, 01 621. The reward the height of the tip of the lower link over the hanging position r(y) = 2 cos 01 cos(0; + 02) E [0, 4].
We use two versions of the system to test various properties of the system models we describe in Section 3. In the "raw angles" system we keep the prediction target which means that models have deal with the noncontinuous angle trajectories when the links roll over at ो... This requires multimodal posterior predictives illustrated in Figure and in Appendix F. In the "sincos" system we change the target to y [sin 0,, cos 0,, sin 02, cos 02, 01, 02] which are the observables of the Acrobot system implementation in OpenAI Gym (Brockman et al., 2016). This smoothes the target but introduces a strong nonlinear dependence between sin 0 and cos i(+1, even given the state st.
12
12
20
24
28
12
12
26
(a) deterministic DARNN&
(c) heteroscedastic (b) homoscedastic DARNN,
(d) multimodal
unimodal DARMDN(I)
DARMDN(1O)
Figure 1: How different model types deal with uncertainty and chaos around the non-continuity I on the Acrobot "raw angles" system. The acrobot is standing up at step 18 and hesitates whether to stay left (0, 0) or go right (0, < O with a jump of 2T). Deterministic and homoscedastic models underestimate the uncertainty so a small one-step error leads to picking the wrong mode and huge errors down the horizon. heteroscedastic unimodal model correctly determines the large uncertainty but represents it as a single Gaussian so futures are not sampled from the modes. The multimodal model correctly represents the uncertainty (two modes, each with small sigma) and leads to reasonable posterior predictive after ten steps. The thick curve is the ground truth, the red segment is past, the black segment is future, and the orange curves are simulated futures. See Section 3 for the definition of the different models and Appendix FF for more insight.
Our aim of predicting dynamic performance static experiments will require not only score design but also data set design. In this paper we evaluate our models on two data sets. The first is generated by running a random policy (1)) on Acrobot. We found that this was too easy to learn, so scores hardly predicted the dynamic performance of the models (Schaul et al., 2019). To create a more
"See Table in Appendix D for more discussion on MAR

---

Published as a conference paper at ICLR 2021
"skewed" data set, execute the MBRL loop (Section 2.2.2) for one iteration using the linear ARLin, model (see Section 3), and generate traces using the resulting policy T AARLin, On both data sets we use ten-fold cross validation on 5K training points and report test scores on a held-out test set of 20K points. All sets comprise of episodes of length 500, starting from an approximately
(2)
hanging position: all state variables (the angles and the angular velocities) are uniformly sampled in [-0.1, 0.1].
3 MODELS AND RESULTS
commonly held belief (Lee al., 2019; Wang et al., 2019) is that MBRL learns fast but cannot reach the usymptotic performance of model-free RL. It presumes that models either "saturate" (their approximation error cannot be climinated even when the size of the training set grows high) and/or they get stuck in local minima (since sampling and learning are coupled). Our research goal is to design models that alleviate these limitations. The first step is to introduce and study models that are learnable with small data but are flexible enough to represent complicated functions (see the summary in Table I). Implementation details are given in Appendix
Table 1: Summary of the different models satisfying (or not) the various requirements from Sec- tion 2.1. (R1): efficient simulation; (R2): explicit likelihood; (R3): g-interdependence (yellow means "partially"): (R4); heteroscedasticity (yellow means "at training"); (R5): multimodality (yellow means "in principle, yes, in practice, no"); (R6): ability model different feature types; (R7): robustness and debuggability. The last two columns indicate whether the model is among the optimal ones on the Acrobot sincos and raw angles systems (Section 2.3 and Table 2; yellow means significantly worse than the best model but within 5% of the optimum).
Model
(R1)
(R2)
(R3)
(R4)
(R5)
(R6)
(R7)
sincos
raw angles
ARLing
<<
11
DARNN,
GP
DMDN(1)
DMDN(10)
DARMDN(I)
DARMDN(1O)
PETS (bagged DMDN(L))
VAE
RealNVP
DARNN
<SS
DMDN(1 I )aee
DARMDN(1)) Dae:
AUTOREGRESSIVE DETERMINISTIC REGRESSOR + FIXED VARIANCE. We learn dy determinis- tie regressors fi ,,. (x*) by minimizing MSE and estimate a uniform residual variance
of = Σ - (x)))) for each output dimension j = 1, 1,... dy. The probabilistic model is then Gaussian (v' (x/) = (yyss (a (ar)). a.;) The two baseline models of this type are linear regression (ARLin,,) and neural net (DARNN.). These models are easy to train, they can handle y y-interdependence (since they are autoregressive), but they fail (R5) and (R4): they cannot handle multimodal posterior predictives and heteroscedasstictyy
GAUSSIAN PROCESS (GP) is the method of choice in the popular PILCO algorithm (Deisenroth & Rasmussen, 2011). On the modelling side, it cannot handle non-Gaussian (multimodal or het- eroscedastic) posteriors and j-interdependdnce, failing Req (R6). More importantly, similarly to Wang et al. (2019) and Chatzilygeroudis et al. (2020), we found it very hard to tune and slow to simulate from. We have reasonable performance on the sincos data set which we report, however GPs failed the raw angles data set (as expected due to angle non-continuity) and, more importantly, the hyprpaarameters tuned lead to suboptimal dynamical performance, so we decided not report these results. We believe that generative neural nets that can learn the same model family are more robust, faster to train and sample from, and need less babysitting the MBRI loop.
7

---

Published as a conference paper at ICLR 2021
Table 2: Model evaluation results on the dynamic environments using random shooting MPC agents. RMAR is the percentage of the optimum reward achieved asymptotically, and MRCP(70) is the number of system access steps needed to achieve 70% of the optimum reward (Section 2.2.2). & and t mean lower and higher the better, respectively. Unit is given after the sign.
MIXTURE DENSITY NETS. A classical deep mixture density net DMDN(D) (Bishop,
1994) is a feed-forme weeddforward neural net outputting
D(1+ + 2dg) parameters [wo ue,, , - n²
|05]ā=1 of multivari- ate independent Gaussian mixture p(yls) (s) (1r (s), diag( (s)))) Its autoregressive counterpart DARMDN(D) learns dy independent neural nets outputting the 3Dy parameters [wy, p.. of of dy mixtures Pl....., Pd, (2). Both models are trained to max- imize the log likelihood (3). They can both represent heteroscedasticity and, for D > 1, multimodal posterior predictives. In engineer- ing systems we prefer DARMDN for its better handling of g-interdependence and its ability to model different types of system variables. DARMDN(D) is similar to RNADE (Uria et al., 2013) except that in system modelling we do not need to couple the dy neural nets. While RNADE was used for anomaly detection (Iwata & Yamanaka, 2019), acoustic modelling (Uria et al., 2015), and speech synthesis (Wang et al.,
[u}]ī=1, o'
Method
RMAR/10 "t
MRCP170N
Acrobot raw angles system
ARLia,
215:177
NaN=NNN
DARNN,
612:19
14070-:3350
DARNN&
703-1:7
5660=980
DMDN(10)
968:18
2200:::40
DARMDN(1)
730-1:7
3120==60
DARMDN(10)
963-17
1680=100
DARMDN 10)as
709+
3960=570
PETS
715:17
7260=:2200
VAE
668.111
15100
Acrobot sincos system
ARLin,
11:13
NaN NaN
DARNN
947:
1600= 170
DARNN
963
1440==80
DMDN(10)
980:18
1670=990
DARMDN())
982:17
1400=50
DARMDN 10)
977:188
1340=100
DARMDN(10Mr
986.1.7
1300=1000
DARMDN(1) lace
987-1.7
1300=
2017), our knowledge, neither DARMDN nor RNADE have been used in the context of MBRL. DMDN has been used in robotics by Khansari-Zadeh & Billard (2011) 1) and it is an important brick in the world model of Ha & Schmidhuber (2018). Probabilistic Ensembles with Trajectory Sampling (PETS) (Chua et al., 2018) is an important contribution to MBRL that trains a DMDN(D) model by bagging D DMDN(1) models. In our experiments we also found that bagging can improve the LR score (4) significantly, and bagging seems to accelerate learning by being more robust for small data sets (MRCP(70) score in Table 2 and learning curves Appendix E); however bagged single Gaussians are not multimodal (all bootstrap samples will pick instances from every mode) so PETS fails on the raw angles data.
PETS
992-1-7
1040=110
PETSan
995:177
840=40
VAB
952:1100
1770=1100
ReaINVP
536:227
NaN=NNN
DETERMINISTI MODELS are important baselines, used successfully by Nagabandi et al. (2018) and Lee et al. (2019) in MBRL. They fail Req (R2) but can be alternatively validated using R2. On the other hand, when used in an autoregressive setup, if the mean prediction represents the posterior predictives well (unimodal distributions with small uncertainty), they work well. In fact, in our experiments found that deterministic models are consistently (although non-significantly) better than their probabilistic versions, possibly because the mean prediction is more precise. We implemented deterministic models by "sampling"" the mean of the DARNN, and DARMDN(-) models, obtaining DARNN det and DARMDN( )eet. respectively.
VARIATIONAL AUTOENCODERS AND FLOWS. We tested two other popular techniques, variational autoencoders (VAE) (Kingma& & Welling, 2014; Rezende et al., 2014 4) and the flow-hased RealNVP (Dinh et al., 2017). VAE does not provide exact likelihood (R2); RealNVP does, but the R2 and KS scores are harder to compute. In principle they can represent multimodal posterior predictives, but in practice they do not seem to be flexible enough to work well on the raw angles system. A potential solution would be to enforce a multimodal output as done by Moerland et al. (2017). VAE performed well (although significantly worse than the mixture models) on the sincos system.
Our results are summarized in Tables 2 and 3. We show mean reward learning curves in Appendix E. We found that comparing models solely based on their performance on the random policy data is a bad choice: most models did well in both the raw angles and sincos systems. Static performance on the linear policy data is a better predictor of the dynamic performance; among the scores, not surprisingly, and also noted by Nagabandi al. (2018), the R2(10) score correlates the most with dynamic performance.
8

---

Published as a conference paper at ICLR 2021
Table 3: Model evaluation results on static data sets. and mean lower and higher the better, respectively. Unit is given after the sign.
Method
LRt
OR/ 04/10-14
R2/10-"1
KS KS/10-$1
R2/10/110*1
KS110//10"$1
tt/minn
tst/sec]
Acrobor raw angles, data generated by random policy
ARLin,
27:11
44:7
9763+
177#33
8308
1574:11
0::00
0=:0
DARNN,
54:18
171::37
9829:19
1711:36
8711::491
212:1-48
2::0
1::0
DMDN(10)
430:226
0=:0
9790:
124+:0
8973
129::29
15::0
2=:0
DARMDN(1)
424:1:88
10:22
9784:122
126:1:
9267::269
106:1:17
19:1:
2:10
DARMIN(1O)
410.18
311
9782.12
9049.1.35
18:10
Acrohot raw angles, data generated by lincar policy after one epoch
ARLin,
3=:0
20:55
6832499
85#11
398
87::14
0::0
0=:0
DARNN,
25:11
1764311
9374:1
193::16
4844
139:1:23
24:0
1=0
DMDN(1O)
137+10
40=111
8449:t-443
72::
5659 1086
135=:19
15:00
2=:0
DARMDN(D
120-222
56:112
5677:16
47::5
1291 ::66
114:::20
20::11
2::0
DARMIN(1O)
143.66
9571.1
1.363
100.111
20.10
Acrobot sincos, data generated by randorm policy
ARLine
6.10
47:110
8976.11
1181:3
5273 1:300
110.111
0.10
0.:0
DARNN,
50:14
188:120
998711
176:122
9249-1633
257:1:44
4::0
2::0
GP
88:22
0=:0
9999
224+:11
9750=
168:t29
0::0
DMDN(10)
361::22
0::0
9957
139::15
8963-1338
146:::35
21:11
1=0
281:1.
DARMDN(1O)
288:17
1:0
998311
10
9296-1233
140:1225
28:11
4
Acrobot sinces, data generated by lineur policy after one spoch
ARLin,
2:100
11:14
6652,1
46:1
35411304 304
127:1188
0:10
0.10
DARNN,
32:1-2
166:234
9986:t2
16
7944 1061
194:1:99
41-0
2=:0
GP
56:2:
6::1
9995:
113:2
8334=1885
133:2:15
0::00
9=1
DMDN(10)
95:1-5
29:16
21::0
DARMDN())
125--4
1244
9991#11
80:24
8603=286
89::13
32#2
3=0
DARMDN(10)
119:44
9:55
9991:22
68:144
8655
95:115
30::1
4:20
Our most counter-intuitive result (although Wang al. (2019) and Wang & Ba (2020) observed similar phenomenon) is that DARMDN(() and PETSda are tied for winning on the sincos system, which suggests that deterministic model can be on par with (or even slightly better than) the best probabilistic models if the system requires no multimodality. What is even more surprising is that classical neural net aARNNaa det is slightly but significantly worse, suggesting that the optimal model, even if it is deterministic, needs to be trained for likelihood score in generative setup. The lower R2(10) score DARNN (and the case study in Appendix F) suggest that classical regression optimizing MSE leads eπror accumulation and thus subpar performance down the horizon. Our hypothesis that heteroscedasticity at training time acts as a regularizer, leading somehow less accumulation longer horizon.
On the sincos system PETS reaches the optimum MARapt within statistical uncertainty which means that this setup of the Acrobot system is essentially solved. We improve the convergence pace MCPR(70) of the PETS implementation of Wang & Ba (2020) by two to four folds (Figure 3 in Appendix E) by using a more ambitious learning schedule (short epochs and frequent retraining). The real forte of D D(AR)MDN(10) is the 95% RMAR score on the raw angles system that requires multimodality, beating the other models by more than 20 %. It suggests remarkable robustness that makes it the method of choice for larger systems with more complex dynamics.
4 CONCLUSION AND FUTURE WORK
Our study was made possible by developing toolbox of good practices for model evaluations and debuggability model-based reinforcement learning, particularly useful when trying to solve real world applications with domain engineers. We found that heteroscedasticity at truining time alleviates error accumulation down the horizon. Then at planning time, we not need stochastic models: the deterministic mean prediction suffices. That is, unless the system requires multimodal posterior predictives, in which case deep (autoregressive or not) mixture density nets are the only current generative models that work. Our findings lead to state-of-the-art sample complexity (by far) on the Acrobot system by applying an aggressive training schedule. The most important future direction is extend the results to complex systems requiring larger planning horizons and to planning strategies beyond random shooting.

---

Published as a conference paper at ICLR 2021
REFERENCES
Zaheer Abbas, Samuel Sokota, Erin J. Talvitie, and Martha White. Selective Dyna-style planning under limited model capacity. In Proceedings of the 37th International Conference on Machine Learning, 2020.
Arthur Argenson and Gabriel Aulac-Amold. Model-based offline planning. CoRR, abs/2008.05554 2020. URL https::/pp....c...t....tt /ttp:i////aavvvaa/eeeeeeeeesaava wttp:/vv..rga/aa/////aaa http:///aaaii..oog//a//////00000 05556.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, and Charles Blundell. Agent57: Outperforming the Atari human benchmark. ArXiv, abs/2003. 13350, 2020.
Christopher M. Bishop. Mixture density networks. Technical report, 1994.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAl gym, 2016.
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Mare Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the st International Conference on Knowledge Discovery and Data Mining, pp. 1721-1730.
ACM, 2015.
Konstantinos Chatzilygeroudis, Vassilis Vassiliades, Freek Stulp. Sylvain Calinon, and Yean-Bapties Mouret. A survey on policy search algorithms for learning robot controllers in a handful trials. IEEE Transactions Robotics, 36(2):328-34) -47888347444 2020.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems 31, pp. 4765- Curran Associates, Inc., 2018.
Sebastian Curi, Felix Berkenkamp, and Andreas Krause. Efficient model-based reinforcement learning through optimistic policy search and planning. In Neural Information Processing Systems (NeurIPS), 2020. URL https: /ttp/vaaavvv//////e//eee/ee.a... org/ abs/ 2006 08684.
Pieter-Tjert de Boer, Dirk P. Kroese, Shie Mannor, and Reuven Y. Rubinstein. A tutorial on the cross-entropy method. Annals of Operations Research, 134, 2004.
Marc Peter Deisenroth and Carl Edward Rasmussen. PILCO: A model-based and data-efficient approach to policy search. In Proceedings of the International Conference on Machine Learning, 2011.
Marc Peter Deisenroth, Dieter Fox, and Carl Edward Rasmussen. Gaussian processes for data- efficient learning in robotics and control. IEEE Transactions Pattern Analysis and Machine Intelligence, 2014.
Stefan Depeweg, Jos Miguel Hernndez-Lobato, Finale Doshi-Velez, and Steffen Udluft. Decom- position of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning. In Proceedings of the 35th International Conference on Machine Learning, pp. 1192-1200 PMLR, 2018.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In 5th International Conference Learning Representations, ICLR 2017, 2017.
Yarin Gal. Uncertainty in Deep Learning. PhD thesis, University of Cambridge, 2016.
Yarin Gal, Rowan McAllister, and Carl E. Rasmussen. Improving PILCO with Bayesian neural network dynamics models. In Data-E Effficifii Machine Learning workshop. ICML, April 2016.
Jacob R. Gardner, Geoff Pleiss, David Bindel, Kilian Q Weinberger, and Andrew Gordon Wilson. GPyTorch: Blackbox matrix-matric Gaussian process inference with GPU acceleration. In Advances in Neural Information Processing Systems, 2018.
10

---

Published as a conference paper at ICLR 2021
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianahi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 2450-2462. Curran Associates, Inc., 2018.
E. Hullermeier and W. Waegeman. Alcatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. arXiv: Learning, 2019.
Tomoharu Iwata and Yuki Yamanaka. Supervised anomaly detection based on deep autoregressive density estimators. arXiv preprint arXiv:1904.0603: 2019.
Nan Rosemary Ke, Amanpreet Singh, Ahmed Touati, Anirudh Goyal, Yoshua Bengio, Devi Parikh, and Dhruv Batra. Modeling the long term future in model-basee based reinforcement learning. In International Conference on Learning Representations, 2019, URL htttt::://pperev/// net/ wtt.//oouuussssee Forrum?id=skggBnneeFF.
Balázs Kégl, Alexandre Boucaud, Mehdi Cherti, Akin Kazakei, Alexandre Gramfort, Guillaume Lemaitre, Joris Van den Bossche, Djalel Benbouzid, and Camille Marini. The RAMP framework: from reproducibility to transparency the design and optimization of scientific workflows. In ICML workshop on Reproducibility in Machine Learning, 2018.
S. Mohammad Shansari-Zadeh and Aude Billard. Learning stable nonlinear dynamical systems with Gaussian mixture models. IEEE Transactions on Robotics, 27(5):943 997799977,99 2011.
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Alex X. Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent Actor-Critic Deep reinforcement learning with a latent variable model. arXiv preprint 11Xiv:997700055. 2019.
Ali Malik, Volodymyr Kuleshov, Jiaming Song, Danny Nemer, Harlan Seymour, and Stefano Ermon. Calibrated model-based deep reinforcement learning. Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp.
4314 4323. PMLR, 2019.
Thomas M. Moerland, Joost Broekens, and Catholijn M. Jonker. Learning multimodal transition dynamics for model-based reinforcement learning. In Scaling Up Reinforcement Learning (SURL) Workshop European Machine Learning Conference (ECML), 2017.
Jean-Baptiste Mouret. Micro-Data Learning: The Other End of the Spectrum. ERCIM News, (107):2,
September 2016. URL https:t/////// //ha:///alaaa. /hal. inria. wtt.fffo1h11111700000
Anusha Nagabandi, Gregory Kahn, Ronald Fearing, and Sergey Levine. Neural network dy- namics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference on Robotics and Automation, ICRA 2018, pp. 7559-7566. IEEE, 2018.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. In Advances in Neural Information Processing Systems 31, pp. 8617-8629. Curran Associates, Inc., 2018.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems 30, pp. 3338-2347. Curran Associates, Inc., 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorhh: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. Curran Associates, Inc., 2019.
11

---

Published as a conference paper at ICLR 2021
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg. Jake Vanderplas, Alexandre Passos, David Coumapeaa,, Matthieu Brucher, Matthieu Perrot, and Edouart Duchesnay. Scikit-leam: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2838 2011.
Anil Rao. A survey of numerical methods for optimal control. Advances in the Astronautical Sciences, 135, 01 2010.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1530-1538, Lille, France, 07-09 Jul 2015. PMLR.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp.
1278-1286. PMLR, 2014.
Arthur George Richards. Robust constrained model predictive control. PhD) thesis, Massachusetts Institute of Technology, 2005.
Tom Schaul, Diana Borsa, Joseph Modayil, and Razvan Pascanu, Ray interference: source of plateaus in deep reinforcemen learning. arXiv preprint arXiv:1904.1 11455, 2019.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap. Karen Simonyan, and Demis Hassabis. general reinforcement learning algorithm that masters Chess, Shogi, and Go through self-play. Science, 362(6419):1 1100-1144,,1, 2018. ISSN 0036-8075. doi: 10.1126/science.aarr/// a4464440
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 4883-199.. Curran Associates, Inc., 2015.
Richard S Sutton.
Generalization in reinforcement learning: Successful examples using sparse coarse coding. In S. Touretzky, M. Mozer, and M. E. Hasselmo (eds.), Advances Neural Information Processing Systems 8, pp. 1048-1044. MIT Press, 1996.
R. Ueda and T. Arai. Dynamic programming for global control of the Acrobot and its chaotic aspect. In 2008 IEEE International Conference on Robotics and Automation, pp. 2416-2422, 2008.
Benigno Uria, lain Murray, and Hugo Larochelle. RNADE: The real-valued neural autoregressive density-estimator. In Advances in Neural Information Processing Systems 26, pp. 2175-2183. Curran Associates Inc., 2013.
Benigno Uria, lain Murray, Steve Renals, Cassia Valentini-Botinhan and John Bridle. Modelling acoustic feature dependencies with artificial neural networks: Trajectory-RNADD In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4465-4469, 2015.
Oriol Vinyals, Igor Babuschkin, Wojeiech Czarnecki, Michal Mathieu, Andrew Dudzik, Junyoung Chung. David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John Agapiou, Max Jaderberg, and David Silver. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575, 11 2019. doi: 10. 00008//4556--099172200
Tingwu Wang and Jimmy Ba. Exploring model-based planning with policy networks. In 8th International Conference on Learning Representations, ICLR 2020, 2020.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model- based reinforcement learning. arXiv preprint arXiv: 1r07v 1907000077 2019.
12

---

Published as a conference paper at ICLR 2021
Xin Wang, Shinji Takaki, and Junichi Yamagishi. An autoregressive recurrent mixture density network for parametric speech synthesis. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4895-4899, 2017.
Baohe Zhang, Raghu Rajan, Luis Pineda, Nathan Lambert, André Biedenkapp, Kurtland Chua, Frank Hutter, and Roberto Calandra. On the importance of hyperparameter optimization for model-based reinforcement learning. A/STATS, 2021.
13

---

Published as a conference paper at ICLR 2021
A AUTOREGRESSIVE MIXTURE DENSITIES
The multi-variate density p(yc+1 (e) is decomposed into a chain of one-dimensional densities
d, II ==2
dy
p(14+118:) = Pı (Vé4 ( (st)
pj (v44 16+1 N.l. 8t) = pr (1 ax)) II j=2
Py (3 12/)) (1)
where, for simplicity, we denote the input (condition) of the jth autoregressive predictor by
of = -V/+ 1 s1). First, p a proper dy-diimnsiionl density as long as the components Pj
are valid one-dimensional densities (Req (R2)). Second, if it is easy to draw from the components Py, it is easy to simulate ++ following the order of the chain (1) (Req (R1)). Third, Req (R3) is satisfied by construction. But the real advantages are on the logistics of modelling. Unlike in computer vision (pixels) or NLP (words), engincering systems often have inhomogeneous features that should be modeled differently. There exists plethora of different mieddimensionl density models which we can use in the autoregressive setup, whereas multi-dimensional extensions are rare, especially when feature types are different (Req (R6). At the debuggability side (Req (R7)) the advantage is the availability of one-dimensional goodness of fit metrics and visualization tools which make it easy pinpoint what goes wrong if the model is not working. On the negative side, autoregression breaks the symmetry of the output variables by introducing an artificial ordering and, depending on the family of the component densities Pj. the modelling quality may depend on the order.
To preserve these advantages and alleviate the order dependence we found that we needed rich family of one-dimensional densities so we decided to use mixtures
Ps P (x0)
Σ
(1):
where component types 3 component parameters 0f. and component weights wf can all depend on L, and the input x², In general, the modeller has large choice of easy-to-fit component types to choose from given the type variable y² (Req (R6)); this paper all our variables were numerical so only use Gaussian components with free mean and variance. Contrary the widely held belief (Papamakarios et al., 201 in our experiments we found no evidence that the ordering of the variables matters, arguably because of the flexibility of the one-dimensional mixture models that can pick up non-Gaussian features such as multimodality (Req (R5)). Finally a computational advantage: given test point æ, do not need carry around (density) functions: our representation of p(y (x) is numerical vector concatenating [w]. Pf. 09] j.
B g-INTERDEPENDENCE
cose
DMDN(5)
DARMDN(1)
GP
sin@
Figure 2: How different models handle g-interdependence. GP (and DMDN(1)) "spreads" the uncertainty in all directions, leading to non-physical predictions. DMDN(D > 1) may "tile" the nonlinear y- y-interdependence with smaller Gaussians, and the limit of - oo it can handle --interdependence for the price of large number of parameters to learn. DARMDN, with its autoregressive function learning, can put the right amount of dependent uncertainty on y² learning for example the noiseless functional relationship between cos 0 and sin O.
14

---

Published as a conference paper at ICLR 2021
y p-interdependencee is the dependence among the dy elements of Vt+1 (3l 1::+1:-... Vi+11 given Ti- Some popular algorithms such as PILCO (Deisenroth & Rasmussen, 2011) suppose that elements of Ve+1 independent given Tz. It reasonable assumption when modelling aleatoric uncertainty in stochastic systems with independent noise. but it is clearly wrong when the posterior predictive has structure due to functional dependence. It happens even in the popular AI Gym benchmark systems (Brockman et al., 2016) (think about usual representation of angles: cos 0x+1 is clearly dependent of sin 41+1 even given Te: see Figure 2), let alone systems with strong physical constraints telecom- munication robotics. Generating non-physical traces due to not modelling y-interdependence may lead not only to subpar performance but also to reluctance to accept the models (simulators) by system engineers.
de
C STATIC METRICS
We define our static metrics from the decomposition of the multivariate density p(3++1 (s)) into the product mie-immnniionll densities (see Appendix for details):
p(yr+11s:) = Pı (V241 |==)) II (1 lx,))
where x. = (1++1 vi st).
j=2
LIKELIHOOD RATIO TO SIMPLE BASELINE (LR) is our "master" metrics. The (average) log-likelihood
[(TriP) J Σ dy
Σ lo (xt+
can be evaluated easily on any trace TI thanks to Req (R2). Log-likelihood is a unitless metrics which is hard to interpret and depends on the unit in which its input is measured (this variability particularly problematic when Py is a mixed continuous/discrete distribution). To have better
interpretation, we normalize the likelihood
(T(:.)
LR(T:p) eLa(T)
with a baseline likelihood (T) which can be adapted to the feature types. In our experiments (T(T) is multivariate independent unconditional Gaussian. LR is between 0 (although LR < usually indicates a bug) and oo, the higher the better.
OUTLIER RATE (OR). We found that LR works well in an i.i.d. setup but distribution shift often causes "misses": test points with extremely low likelihood. Since these points dominate C and LR, we decided clamp the likelihood at Pmin 1.47 10-6 Given a trace T and model p, we define 'I (pp Pmin) {y a.) E T p(:1 > Pmin report LR (T(p; p instead of LR(T:p). and measure the "surprise" of model trace T by the outlier rate (OR)
[T Pmvin)
OR OR (T:P))
(5)
OR is between O and 1, the lower the better. EXPLAINED VARIANCE (R2) assesses the mean performance (precision) of the methods. For- mally
R2((T:P)
1
MSE (Tr:P) of
with MSE,
T (1-=5(
dy 1=1
where f5(x,) = p, (: w!) {rr) is the expectation of v given ax. under the model Py (point prediction), and of is the sample variance of ..)...,)/). We prefer using this metrics over the
As salute to five sigma, using the analogy of the MBRL loop (Section 2.2.2) being the iterated scientific method.
15

---

Published as a conference paper at ICLR 2021
MSE because it is normalized so it can be aggregated over the dimensions of y. R2 is between 0 and 1, the higher the better.
CALIBRATEDNESS (KS). Well-calibrated models have been shown to improve the performance of algorithms (Malik et al., 2019). A well-calibrated density estimator has the property that the quantiles of the (test) ground truth uniform. To assess this, we compute the Kolmogorov-Smirno
(KS) statistics. Formally, let (v ( - Pj Ps(''' (/(1=))))))))))) |xy))dy' be the cumulative distribution function
(CDF) of Pj. and let the order statistics of Ff F3 (vÊ [2 1-1 the th largest quantile in Fy. Then we define
) T-1
be 85, that is, Fj ( (v², (m2,,) is
d
KS(TT; P)
Σ dy j=1
Fj ( 1ods)) [[IT-I]]
max
T 5j
Computing KS requires that the model can provide conditional CDFs, which further filters the possible models we can use. On the other hand, the aggregate KS and especially the one-dimensional CDF plots (1,, 2 vs. sT/(T 1)) great debugging tools. KS is between 0 and 1, the lower the better.
All four metries (LR, OR, R2, KS) are averaged over the dimensions, but for debugging we can also evaluate them dimension-wise.
LONG HORIZON METRICS KS(L) AND R2(L). All our density estimators are trained to predict the system one step ahead yet arguably what matters is their performance at longer horizon L specified by the control agent. Our models do not provide explicit likelihoods L steps ahead, but we can simulate from them (following ground truth actions) and evaluate the metrics by Monte-Carlo estimate. Given n random estimates V/_- = 1ū++,, we can use fz())) = 1 Egey, ÿ' in (6) to obtain an unbiased R2(L) estimate. To obtain a (S(L) estimate, we order V2 and approximate F2 (v² (x') by 1 l10 D E V'L 3" < y'H in (7). LR and OR would require approximate techniques so we omit them. In all our experiments we use L = 10, n = 100, and, for computational reasons, sample the test set at 100 random positions, which explains the high variance on these scores.
All six metrics (LR, OR, R2, KS, R2(10), KS(10)) are averaged over the dimensions to obtain single scores for the environment/mode pair, but for debugging we can also evaluate them dimension- wise. LR is the "master" score that combines precision (R2) and calibratedness (KS). R2 is good single measure to assess the models, especially when iterated to obtain R2(L). OR and KS are excellent debugging tools. The single-target KS and quantile plots especially useful to spot how the models are miscalibrated: e.g.. points accumulating in the middle indicate that we overestimate the tails, leading to nonphysical simulations, and vice versa, accumulation at the edges means our model is missing modes. OR is great to detect catastrophic failures or distribution shifts, so monitoring it on the deployed system is crucial. Finally, correlating these metrics the dynamic performance (Section 2.2.2) for the given system can form the basis of a comprehensive monitoring system which as important model performance in practice.
D IMPLEMENTATION DETAILS
Note that all experimental code is publicly available at pettps://githh: com/ramp kits/ /att////kkiis////// rl_s ss dis.imulatt or.. In this section we give enough information so that all models can be reproduced by a moderately experienced machine learning expert.
The sincos and raw angles Acrobot systems are based on the OpenAI Gym implementation (Brockman et al., 2016). The starting position of each episode is the one obtained from the default reset function of this implementation: all state variables (the angles and the angular velocities) are uniformly sampled in 0.11, 0.1]. For the linear regression model we use the implementation of Scikit-learn (Pedregosa et al., 2011) without regularization. We use Pytorch (Paszke et al., 2019) for the neural network based models (DARNN, DMDN and DARMDN) and Gpytorch (Gardner et al., 2018) for the GP models. The hyperparameter search for these models was done in two steps: first using random search over a coarse hyperparameter grid, then using a second step of random search over a finer grid around values of interest. The steps of the coarse grid were defined to contain five values of each
16

---

Published as a conference paper at ICLR 2021
hyperparameters (or less where applicable), the finer grid defined to contain five values of each hyperparameter (or less where applicable) between two interesting spots close in the hyperparameter space. The selected hyperparameters
given in Table 4.
"Nb layers" corresponds to the number of fully connected layers, except for the two following models:
RealNVP (Dinh et al., 2017): it is the number of coupling layers.
CVAE (Sohn et al., 2015): it the total number of layers (encoder plus decoder).
"Nb components" is the number of components in the outputted density mixture. In the GP and deterministic NN cases, it trivially one.
Table 4: Model hyperparameters.
Method
Learning rate
Neurons per layer
Nb layers
Nb components
Validation size
Nb epochs
Tried values
DARNN
[1o-4 Ie-1]
[20. 300]
[1.4] 4]
[0.05, 0.4]
[10. 300]
DMDN
[1e:5, 1e-21
[100, 600)
12, 51
12. 20]
10.05, 0.41
150. 500]
DARMDN
[1e:5, 1e:21
[20, 300]
[1.4]
[0.05, 0.4]
[50. 500]
CVAR
[le-5, le:2]
[20. 300]
[4, 10]
NaN
[0.05, 0.4]
[50, 500]
ReaINVP
[1o-5, Io-21
[10,30] 3003
[2.
NaN
[0.05,
[50, 500]
GP
[le-3, le-1]
NaN
NaN
[0.05, 0.4]
[10, 300)
Best values
DARNN
4e-3
200
0.05
100
DMDN(1O)
Se-3
200
3
10
300
DARMDN(1)
DARMDN(1O)
le.3
100
3
10
0.1
300
CVAE
NaN
ReaINV
50-3
20
3
NaN
0.15
200
GP
56-2
NaN
NaN
0.15
50
For PETS we use the code shared by Wang et al. (2019) for the Acrobot sincos system. Following Chua et al. (2018), the size of the ensemble is set to 5. For the Acrobot angles system same PETS neural network architecture as the one available for the original sincos system. Although the default number of epochs was set to 5 in the available code reached better results with 100 epochs and use this value in our results. Finally, the RS agent is configured to be the same as the one 100 and particles.
the
use: planning horizon I, 10, search population size
We selected the planning strategy (random shooting with search population size n = 100) by evaluating the performance of random shooting and the cross entropy method (CEM) on the true dynamics for different values of n. Results are presented in Table 5. Although for both RS and CEM with 500 leads to better performance, n 100 is already sufficient achieve more than decent mean rewards and outperform the result of Wang et al. (2019) while reducing the total computational cost of the study. CEM was implemented with a learning rate of 0.1, an elite size equal to 50 and iterations. For a fair comparison between RS and CEM n means the total number of sampled action sequences. This means that, for CEM, n means a search population size of n/5 for each the iterations.
Table 5: Comparison of RS and CEM on the true dynamics. The + values are 90% Gaussian confidence intervals based on 100 random repetitions of a 200-step rollout.
500
RS
1000
CEM
CEM with 1000
2. + 0.035
2.27 0.025
2.29 + 0.024
2.32 0.021
2.30 + 0.021
We implemented reusable system models and static experiments within the RAMP framework (Kégl et al., 2018).
All + values in the results tables 90% Gaussian confidence intervals based on i) 10-fold cross- validation for the static scores in Table 3, ii) 50 epochs and two to ten seeds in the RMAR column, and iii) ten seeds in the MRCP(70) column of Table 2.
17

---

Published as a conference paper at ICLR 2021
E MEAN REWARD LEARNING CURVES
Figure 3 shows the mean reward learning curves on the Acrobot raw angles and sincos systems. The top models PETS and DARMDN(10)a) converge close the optimum at around the same pace on the sincos system. PETS converges slightly faster than the other models in the early phase. Our hypothesis is that bagging creates more robust models in the extreme low data regime (100s of training points). Our models were tuned using 5000 points which seems to coincide with the moment when the bagging advantage disappears.
On the raw angles system DARMDN(110) and DMDN(10) separate from the pack indicating that this setup requires non-dcterministic predictors and mixture densities to model multimodal posterior predictives. The reward is between 0 (hanging) and 4 (standing up). Each epoch starts at hanging position and it takes about 100 steps to reach the stationary regime where the tip of acrobot is above the horizontal line most of the time. This means that reaching average reward above needs excellent control policy.
2.2
method
2.0
(dashed) previous SOTA [Wang et 2019] DARMDN(1)
1.8
DARMDN(1). (OMdDN(1)__ett
mean reward
DARMDN(10)
1.6
DARMDN(10)_de))
DARNN det
1.4
DARNN sigma
1.2
DMDN(10)
PETS-RS
1.0
RS on real system (optimum)
VAE
0.8
8,000
16,000
system access step
2.2
method
2.0
(dashed) previous SOTA [Wang et al. 2019]
DARMDN(1)
1.8
DARMDN(1). (e)dddd
mean reward
DARMDN(10)
1.6
DARMDN(10)_des
DARNN det
1.4
DARNN sigma
1.2
DMDN(10)
PETS-RS
1.0
RS on real system (optimum)
VAE
0.8
0
8,000
16,000
system access step
Figure 3: Acrobot learning curves on the raw angles (top) and sincos (bottom) systems, Reward is between (hanging) and (standing up). Episode length is T' 200, number of epochs N 100 with one episode per epoch. Mean reward curves are averaged across three to ten seeds and smoothed using a running average of five epochs, plotted at the middle of the smoothing window (so the first point is at step 600).
F THE POWER OF DARMDN: PREDICTING THROUGH CHAOS
Acrobot is a chaotic system (Ueda Arai, 2008): small divergence in initial conditions may lead to large differences down the horizon. This behavior is especially accentuated when the acrobot slowly approaches the unstable standing position, hovers, "hesitates" which way to go, and "decides" to fall back left right. Figures 4 and 5 depict this precise situation (from the test file of the "linear" data,
18

---

Published as a conference paper at ICLR 2021
Section 2.3): around step 18 both angular momenta close to zero and 01 To make the modelling even harder, = т is the exact point where the trajectory is non-continuous the raw angles data, making it hard to model by predictive densities that cannot handle non-smooth traces.
In both figures we show the ground truth (red: past, black: future) and hundred simulated traces (orange) starting at step 18. There is "correct" solution here since can imagine several plausible "beliefs" learned using limited data. Yet is rather indicative about their performance how the different models handle this situation.
First note how diverse the models are. On the sincos data (Figure 4) most posterior predictives after steps unimodal. GP and DARMID((10) whether Acrobot falls left or right, DARMDN(10) bets more on the ground truth mode. Among the deterministic models, both DARNN and DARMDN(10)dr work well one step ahead (on average, according to their R2 score in Table 3), but ten steps ahead DARMDN(10)se is visibly better, illustrating its excellent R2(10) score.
not, but while GP predicts coin toss
On the raw angles data (Figure 5) we see very different picture. The deterministic DARNNaa picks one of the modes which happens to be the wrong one, generating a completely wrong trajectory. DARMDN(10))er predicts average of two extremem modes (around and -TT) resulting in non-physica prediction (0,) which has in fact zero probability under the posterior predictive of DARMDN(10). The homoscedastic DARNN, has constant sigma which, in this situation is too small: it cannot "cover" the two modes, so the model picks one, again the wrong one. The heteroscedastic DARMND(1) correctly outputting huge uncertainty, but since it is single unimodal Gaussian, generates a lot of non- non-physical predictions between and outside of the modes. This shows that heteroscedasticity without multimodality may be harmful in these kinds of systems. Finally, DARMDN(10)) (10) has a higher variance than on the sincos data, especially on the mode not validated by the ground truth, but it is the only model which puts high probability on the ground truth after steps, and whose uncertainty is what a human would judge reasonable.
19

---

Published as a conference paper at ICLR 2021
DARNN
M
BALLOK
11
M
SWOLAR
20
12
20
20
DARNN
det
M
12
16
28
DMDN(10)
M
UNLLONL
1M6.3
DARMDN(1)
M
DARMDN(10)
M
twt.3
12
16
75
76
10
70
34
12
70
34
26
-12
0
95
70
34
76
DARMDN(10). det
dea ML1
140.
1311 316
1971 3080
11mw stap
Figure 4: Ground truth and simulation of "futures" " by the models trained on the sincos system. The thick curve is the ground truth, the red segment is past, the black segment is future. System models start generating futures from their posterior predictives at step 18. We show a sample of hundred trajectories and histogram after time steps (orange).
20

---

Published as a conference paper at ICLR 2021
DARNN
LM
5 5.3 O
10
20
16
28
12
24
20
DARNN det
NM
3H0_3
DMDN(10)
M
3 b.b
12
34
1ime Kxp
DARMDN(1)
M
Eine wtap
Eis x5p
Siw stpp
DARMDN(10)
M
INALI
19me 15p
U1ne slop
13re s1op
130e slop
DARMDN(10., (10)
8
100
24
20
stop
1071 xtop
Figure 5: Ground truth and simulation of "futures" by the models trained the angles system. The thick curve is the ground truth, the red segment is past, the black segment is future. System models start generating futures from their posterior predictives at step 18. We show sample of hundred trajectories and a histogram after ten time steps (orange).
21


Reviews:
[{"review_id": "fby9jcGNyTn", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper discusses the quality of surrogate models for model-based RL.  The two main contributions are first: defining the a set of evaluation metrics + experimental design and secondly performing a comparative analysis over existing methods.\n\nThere are several things I like about this paper:\n \n- There is lack of existing work in throroughly comparing surrogate models in model-based RL\n- The authors do a great job defining and arguing for evaluation metrics  (Section 2 is excellent)\n- Defining requirements to the model that are falsifiable (2.1) are good to augment quantitative comparisons with qualitative ones\n\nOn the other hand there a several things I think that could be improved:\n\n- Introduction:  The introduction  was quite confusing: Not only was it intermixed with conclusions that seem out of place. Also the discussion about uncertainty/probability was unclear.  Especially when talking about both uncertainty in a Bayesian sense (uncertainty over parameters or functions, epistemic) and \"randomness\" (stochastic effects of the environment) it is very easy\nto be confusing.  E.g  \"Probabilistic models are needed when the system benefits from multimodal predictive uncertainty.\"  It is not clear here what you mean by \"probabilistic\":  Parametric uncertainty (like in GPs) or models that can model multimodalities (like MDN)?\n\n- Relevant work:  There is relevant work missing here.  Methods dealing with Bayesian deep learning[1,2], contextual auto-encoders[3]. \n\n   - [1] Gal, Yarin, Rowan McAllister, and Carl Edward Rasmussen. \"Improving PILCO with Bayesian neural network dynamics models.\" Data-Efficient Machine Learning workshop, ICML. Vol. 4. 2016\n   - [2] Depeweg, Stefan, et al. \"Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning.\" International Conference on Machine Learning. PMLR, 2018.\n   -[3] Moerland, Thomas M., Joost Broekens, and Catholijn M. Jonker. \"Learning multimodal transition dynamics for model-based reinforcement learning.\" arXiv preprint arXiv:1705.00470 (2017).\n\n\n- Limited scope: Only considering a single benchmark (in the sense of a single environment) is too limited. Chances are the \"surprising results\" are due to the specifics of this particular problem.  In general:   Methods that can model stochastic effects (e.g. neural networks that also predict a heteroscedastic Gaussian noise) should perform well when there is true stochasticity in the benchmark. Bayesian approaches (that model some form of parametric/functional uncertainty) should perform well when data is limited. \n\nOverall I think this paper is written quite well,  deals with a topic that is relevant. While it is rigorous in terms of defining metrics and experimental set-up, it would greatly improve if additional benchmarks (and possibly methods) are considered.", "extra_scores": {}, "extra_text": {}}, {"review_id": "TzdsshA04eZ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyses the random shooting control strategy in combination with\nvarious generative models \\( p(\\by_{t+1} | \\mathcal T_t) \\), which model\nobservations \\(\\by\\) conditioned on the history of observation-action pairs.\nThe authors select two variants of the Acrobot environment to make\nrequirements like multimodal posteriors more explicit. Both statistics on the\ndistribution associated with the generative model under a fixed policy and\nreward-dependent metrics were defined and analysed for a range of models.\n\nThe paper's contribution are twofold. First, the suggested experimental\nprotocol for model evaluation and benchmarking extends the usual evaluation\nprocess in reinforcement learning, which often focuses purely on the\ncumulative reward. The framework described introduces a range of ``static''\nlikelihood-based metrics. These static metrics are evaluated under a fixed\n(potentially stochastic) policy and allow the separation of the model-based\ncontrol strategy and the underlying model for evaluation purposes.\nReward-based ``dynamic'' metrics are evaluated under a random-shooting\ncontrol mechanism. This framework heavily simplifies model evaluation by\nproviding evaluation metrics and fixing the environment and control strategy.\nUnder these assumptions this approach allows direct comparison and even\nvisualisation of various quantities of interest, including trajectories of\nthe one-step forward model. Second, this work applies the conceived framework\nto evaluate a range of models on two different environments. These\nenvironments are intended to make requirements like probabilistic posteriors\nexplicit. The authors conclude by claiming that 1. ``Probabilistic models are\nneeded when the system benefits from multimodal predictive uncertainty'', and\n2. Deterministic models are sufficient if trained ``with a loss allowing\nheteroscedasticity''.\n\nThere's an inherent trade-off between simplicity of the study and generality\nits conclusion. While some of the simplifying assumptions made make this kind\nof study possible in the first place, they also raise a range of questions:\n- How appropriate are these metrics for problems with higher observation\n  spaces? Can we expect the variance of estimates to increase and ratios and\n  likelihoods to diminish by multiple orders of magnitude?\n- Do the claims presented as ``important findings'' generalise to other\n  environments?\n- Does the correlation of the explained variance with the dynamic metrics\n  hold on other environments?\n\nA small note: a clear definition of micro-data reinforcement learning is\nmissing, and MBRL is introduced twice with conflicting definitions.\n\nThe future directions outlined by the authors of extending the results to\nlarger systems and other planning strategies are very relevant to reduce the\nconcerns of generalisability of the results and applications to problems of\nhigher dimensions. At the same time these modification will increase the\nexperimental complexity. This paper serves as a suitable baseline for\nreference for future work to answer these questions. Therefore I consider\nthis paper a valid contribution to ICLR.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "CkmLxg1zemC", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper Summary:\n\nThis paper performs a detailed ablation study over different mechanisms for predicting dynamics for model-based control. The paper proposes its own metrics for models, evaluates how different types of uncertainty impact predictions, and measures control performance with random shooting MPC. By implementing a new hyper parameter schedule, the paper shows new SOTA performance (in terms of sample efficiency) on the acrobat task.\n\n----- \n\nRebuttal phase:\nI am updating my score based on the numerous clarifications of experiments and improvements of framing the paper.\n-----\n\n\nScore Summary:\n\nThe analysis in this paper is very warranted, but the methodology executed raises many questions as to if the claims will be generalizable. The paper evaluates models on only one task and on their own metrics, so there is difficulty in matching the paper to the literature. This difficulty of placement for a paper making broad claims, leads to a currently unpublishable form.\n\n\n\n-----\n\n\nComments:\n0) This type of work is absolutely critical to advancing MBRL. I have a lot of criticisms below, but I want to say that I think this could be a very good paper when all is said and done. The metrics proposed are relevant and the breadth of models tested is encouraging.\n\n1) the authors refer to micro-data in the title and multiple times in the introduction, but never define this. What regime of data do the authors consider micro data? Is it a cap at 1000 points or depend on the environment?\n1b) Where did the number of training points at 5000 come from in 2.3?\n\n2)The authors make some strange claims in the paper, that vary from interesting to unsupported and too strong. Many of these claims need to be backed by citations.\n2a) \"Unlike computers, physical systems do not get faster with time\" is playing loose and fast with the fact that the computational power of computers is increasing rapidly in recent years. This does not really have much of a baring on the rest of the paper and should be tied in better.\n\n3) The evaluation of only fixed, random-shooting MPC is okay, but limited. For a paper that is titled \"what are the critical model properties to choose\" I would like to see more acknowledgement that the model one chooses is heavily dependent on the method it is being used for. \n3a) PETS (Chua et. al 2018) as a paper is cited, can the authors comment why the trajectory propagation method or an informed sampling like CEM is not considered? For example, in appendix A.8, the PETS paper shows performance is dramatically better with CEM, and other methods such as POPLIN (Wang et. al 2019) and PDDM (Nagabandi et. Al 2019) have built on this, it seems like a step back to not consider these advances.\n\n4) The model hyper-parameters section is hard to read (A.D) and is crucial to being able to trust the results. What search method was performed over the ranges given per model? And, how extensive was said search?\n4b) The authors claim \"system modeling for MBRL is essentially a supervised learning problem with AutoML.\" To my knowledge there are no detailed examples of MBRL using AutoML?\n\n5) the paper claims SOTA sample efficiency on acrobat, but I am not sure. Can the authors provide more information. For example, I am looking at a MBRL benchmarking paper and the reward takes on a vastly different range (https://arxiv.org/pdf/1907.02057v1.pdf). Acrobat is not the most well studied environment in recent years, so this discrepancy is hard to reason with. \n\n6) While the model requirements and evaluation metrics are insightful, it could be useful to distill them into slightly more compact analysis. What are the most important takeaways for each is interesting, but more useful is \"which type of model does each person want.\" It seems unlikely that every problem should use the same model.\n\n7) The number of action samples used in the random shooting is extremely low at n=100 sequences. Was this validated against? It seems at least an order of magnitude lower than many other papers in the area.\n\n8) In figure 1, the paper shows \"how different model types deal with uncertainty\" which types of models were actually used, or is this an illustration?\n\n9) The claim that random-shooting on the real environment is optimal control on the real environment is very troubling (especially with the number of action samples). Random shooting is only guaranteed to be optimal when the number of samples goes to infinity. If this metric was meant as \"matches the state of the art for control of the acrobat\" this should be clarified.\n\n10) A more complicated environment is needed to make many of these claims more believable. Were any experiments done on hopper or half cheetah? \n\n11) What does \"starting from an approximately hanging position\" mean for acrobat, and is that the standard used in other control papers? Please classify this numerically.\n\n12) the claim that model accuracy is the bottleneck of MBRL should be cited in the introduction. I think there is some preliminary work showing that direction, but sampling based control is also sub-optimal.\n\n13) Are certain requirements correlated with certain metrics? This would be an interesting analysis, but there wasn't much of it in the paper (yes low on space, I know :/)\n\n14) was the likelihood bound of 1.47E-6 taken from somewhere or how was this number computed?\n\n15) The models and results section was confusing. I think the reader would benefit from a discussion of the results. There are interesting findings, such as what the authors discussed on training with uncertainty, that are not always super clear.\n\n-----\n\nMinor Points:\n- Paragraph 2 of second page, \"MBLR\"\n- inconsistency in hyphenating one-step \"one step ahead\"\n- \"Robust and computationally efficient probabilistic generative models are the crux of many machine\" needs some commas or punctuation.\n- Fragment section 2 \"So our goal is to learn p()\"\n- Section 3 starts with \"A commonly held belief,\" just say what the results show.\n- Footnote 2, I agree it is annoying at times, but most papers report the trial length, no?\n- The use of our in the introduction is slightly strange\n- Generally accepted use fo quotes at the end of a sentence is \"hi.\"\n- The analogy to Yann Lecun is strange and un-cited.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7jQ4oxpG6lr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles a key problem in model-based RL that is to identify the essential properties a predictive model needs to have to achieve good control performance. They solve this problem by systematically accessing the performance of a family of autoregressive mixtures learned by deep neural nets (DARMDN) using a fixed control strategy (i.e., random shooting). An important finding from this investigation is that deterministic models, when trained with a likelihood score in a generative setup, has comparable performance with the probabilistic modes. Furthermore, they proposed several static metrics (LR, OR, R2, KS, long-horizon R2, and long-horizon KS) which can be used to assess the performance of the models before having to run long experiments on the dynamic systems. These metrics can help to debug the models when evaluated dimension-wise. They found out that R2 with a 10-steps horizon correlates the most with dynamic performance. The findings lead to state-of-art sample complexity on the Acrobot system by applying an aggressive training schedule. \n\nStrength: \n1) The paper is well-written and structured. \n2) The paper provides some good practices for evaluating model-based RL which will help to advance the field. \nWeakness: \n1) The paper only used one simple problem (Acrobot) and a random shooting control strategy to evaluate the different predictive models of the dynamic system, it'll be more interesting to see if the findings stills hold in more realistic and complex RL problems \n\n-------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n\nUpdate: \nAfter going through all the discussion, I'd be happy to raise my score to 7. The authors did a great job in clarifying all the concerns raised by the reviewers. This make the paper a much stronger publication. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "6optgf_7jJz", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\nThis is a nice experimental paper on comparing various methods for MBRL. They listed properties that are desirable for MBRL, and defined multiple static metrics and dynamic metrics. They discussed popular methods within this framework and performed experiments in one task (two conditions) with the same control policy. Overall, the paper provides a good benchmark for MBRL. \n\n##########################################################################\n\nReasons for score: \nI think this paper provides a good benchmark and a good discussion and comparison between existing methods. This is important for future progress in MBRL. The code is promised to be publicly available, which I think it’s very crucial for an experimental paper. Bonus points if the authors can discuss how the results in this task might generalize to other different tasks (or actually performed some simple experiments). I think the current task and conditions do serve its purpose, but it just leaves the reader wondering about how the result might generalize. \n \n##########################################################################\n\nPros: \n\n1. The writing is very clear and easy to follow, also without redundancy. The paper is also well-structured. I particularly like that the authors clearly indicated which direction is better for the metrics in the tables. \n2. The seven properties and metrics are mostly well-motivated and well-defined. \n3. The authors discussed the results clearly with implications, not just stating which one is better, but also discussed when and why it is better.\n4. The two important findings seem very interesting to me. I think the result of the necessity of probabilistic vs. deterministic models in different scenarios is a good contribution to this field.\n5. The authors noted in the appendix that the code will be made public if accepted.\n \n##########################################################################\n\nCons: \n1. The authors provided motivation to the metrics they proposed. It would be great to see a brief description of how different metrics related to each other, what (different) aspects they capture, and whether/how they correspond to the seven desirable properties. Also, it seems not sufficient enough to filter out methods just because of the impossibility of computing the defined metrics. \n2. Generalization to other tasks and different control policies. The authors did point this out as future work, but it’s also good to briefly discuss how it might or might not generalize.\n\n#########################################################################\n\ntypo: \nIntroduction 5th paragraph: MBLR -> MBRL \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Overview** This paper performs detailed ablation studies over different dynamics prediction methods for MBRL. It proposes metrics for models to evaluate how different types of uncertainty impact predictions. The paper also measures control performance with random shooting MPC. The paper further implements a new hyper parameter schedule to achieve new SOTA performance on the acrobat task.

**Pro** 
- The paper is well-written.
- The analysis in this paper is very warranted. 
- The paper provides a very detailed ablation study.
- The authors do a great job defining and arguing for evaluation metrics.
- The seven properties and metrics are mostly well-motivated and well-defined.
- The authors discussed the results clearly with implications.
- The result of the necessity of probabilistic vs. deterministic models in different scenarios is a good contribution to this field.

**Con**
- The methodology might be hard generalizable, i.e., there is difficulty in matching the paper to the literature based on its own defined metric.
- The scope might be limited.

**Recommendation** The paper provides a significant contribution to MBRL by providing a detailed empirical study. During the rebuttal phase, the authors addresses many reviewers' concerns in a satisfactory way.  The paper is well-written and easy to read. The recommendation is an accept.

Decision: Accept (Poster)