Deep autoregressive density nets vs neural ensembles for model-based offline reinforcement learning
Authors: Abdelhakim Benechehab
Albert Thomas
Balázs Kégl
Venue: Submitted to ICLR 2023 (2023)
Keywords: Offline reinforcement learning
batch reinforcement learning
ensemble
autoregressive
D4RL
model-based

Abstract: We consider the problem of offline reinforcement learning where only a set of system transitions is made available for policy optimization. Following recent advances in the field, we consider a model-based reinforcement learning algorithm that infers the system dynamics from the available data and performs policy optimization on imaginary model rollouts. This approach is vulnerable to exploiting model errors which can lead to catastrophic failures on the real system. The standard solution is to rely on ensembles for uncertainty heuristics and to avoid exploiting the model where it is too uncertain. We challenge the popular belief that we must resort to ensembles by showing that better performance can be obtained with a single well-calibrated autoregressive model on the D4RL benchmark. We also analyze static metrics of model-learning and conclude on the important model properties for the final performance of the agent.

Under review as a conference paper at ICLR 2023
DEEP AUTOREGRESSIVE DENSITY NETS VS NEURAL ENSEMBLES FOR MODEL-BASED OFFLINE REINFORCE- MENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We consider the problem of offline reinforcement learning where only set of system transitions is made available for policy optimization. Following recent ad- vances in the field, we consider a model- -ased reinforcement learning algorithm that infers the system dynamics from the available data and performs policy opti- mization on imaginary model rollouts. This approach is vulnerable exploiting model errors which can lead to catastrophic failures on real system. The stan- dard solution is to rely ensembles for uncertainty heuristics and to avoid ex- ploiting the model where it is too uncertain. We challenge the popular belief that we must resort to ensembles by showing that better performance with a single well-calibrated autoregressive model on the D4RL benchmark. We also analyze static metrics of model-leaming and conclude on the important model properties for the final performance of the agent.
be obtained
INTRODUCTION
Reinforcement learning consists in learning a control agent (policy) by interacting with dynamical system (environment) and collecting feedback (rewards). This learning paradigm turned out to be able to solve some of the world's most difficult problems (Silver et al., 2017; 2018; Mnih et al., 2015: Vinyals et al., 2019). However, the scope of the systems that RL is capable of solving remains restricted to the simulated world and does not extend to real engineering systems. Two of the main reasons are i) small data due to operational constraints and ii) safety standards of such systems. In an attempt to bridge the gap between RL and enginecring systems, motivate the setting of offline reinforcement learning (Levine et al., 2020).
Offline reinforcement learning removes the need to query dynamical system by using previously collected dataset of controller-system interactions. In this optic, we view this setting as a supervised learning problem where tries to approximate the underlying distribution of the data at hand, and hopefully be able to generalize to out-of-distribution samples. This turns out to be a difficult task for classical RI algorithms because of the distribution shift that occurs between the dataset and the learned policy during the learning process (Fujimoto et al., 2019; Levine et al., 2020). Thus we need design algorithms that are well-suited for offline reinforcement learning. A common idea in this field is conservatism where one would only consider the learned agent when the input states are close to the support of the offline dataset. Depending on the algorithm, conservatism can take multiple forms, ranging from penalized Q-targets (Kumar et al., 2020) to uncertainty penalized Markov decision processes (Kidambi et al., 2020; Yu et al., 2020). To develop further into this direction, we make the distinction between model-free and model-based RL (MBRL) algorithms.
peeeaaainttyppplizeee
Model-free algorithms learn a policy and/or a value function by observing the reward signal realiza- tions and the underlying dynamics of the system, which in most environments requires significant number of interactions for achieving good performance (Haarnoja et al., 2018). In this category, way to incorporate conservatism is to penalize the value targets of data points that distant from the offline dataset (Kumar et al., 2020). Other methods include behavior regularized policy optimization (Wu et al., 2020).
Model-based algorithms are composed two independent (and often alternating) steps: i) model learning: a supervised learning problem of learning the dynamics (and sometimes also the reward

---

Under review as a conference paper at ICLR 2023
function) of the system of interest; and ii) policy optimization, where we sample from the learned dynamics to learn a policy and/or a value function. MBRL is known to be sample-efficient, since policy/value learning is done (completely or partially) from imaginary model rollouts (also called background planning) that are cheaper and more accessible than rollouts in the true dynamics (Janner al., 2019). Furthermore, a predictive model with good out-of-disributioon performance affords easy transfer of the true model to new tasks or areas not covered in the offline dataset (Yu et al., 2020). Conservatism in MBRL is frequently achieved by uncertainty-bast penalization of the model predictions. This relies on well-calibrated estimation of the epistemic uncertainty of the learned dynamics, which is a limitation of this approach.
It is of great interest to build models that know when (and how much) they do not know, thus uncer- tainty estimation remains a central problem MBRL. Many recent works have made progress in this direction (Osband et al., 2021). The most common approach to date is bootstrap ensemblex: we construct a population of predictive models (most often probabilistic neural networks) and consider disagreement metrics as uncertainty measurement. The source of randomness in this case is the random initialization of the parameters of neural networks and the subset of the training data that each model sees. When the environment is stochastic, ensembles help separate the aleatory uncertainty (intrinsic randomness of the environment) and the epistemic uncertainty (Chua et al., 2018). When the environment is deterministic (which is the case of the D4RL Mujoco benchmark environments considered in most of the offline RI. literature (Fu et al., 2021a)), the error is fully epistemic: consists of the estimation error (due lack of training data) and the approximation er- ror (mismatch between the model class and the true distribution) (Hüllermeier & Waegeman, 2021), I). This highlights the need of well-calibrated probabilistic models whose posterior variance can be used as an uncertainty measurement in conservative MBRL.
In this work, we propose to compare autoregressive dynamics models (Uria al., 2016) to ensem- bles of probabilistic feedforward models, both in terms of static evaluation (supervised learning metries on the task of learning the system dynamics) and dynamic evaluation (final performance of the MBRL agent that uses the model). Autoregressive models learn conditional distribution of each dimension of the next state conditioned on the input of the model (current state and action) and the previously generated dimensions of the next state. Mcanwhile, probabilistic feedforward models learn a multivariate distribution of the next state conditioned on the current state and action. We argue that autoregressive models learn the implicit functional dependence between state dimensions, which makes them well-calibrated, leading to good uncertainty estimates suitable for conservatism in MBRL.
Our key contributions are the following.
We apply autoregressive dynamics models in the context of offline model-based reinforce- ment learning and show that they improve over neural ensembles in terms of static evalua- tion metrics and the final performance of the agent.
We introduce an experimental setup that decouples model selection from agent selection to reduce the burden of hyperparameter optimization in offline RI.
We study the impact of static metrics the dynamic performance of the agents, and con- clude on the importance of single-step calibratedness in model-based offline RI.
2 RELATED WORK
Offline RL has been an active area of research following its numerous applications in domains such as robotics (Chebotar et al., 2021), healthcare (Gottesman et al., 2018), recommendation systems (Strehl et al., 2010), and autonomous driving (Kiran et al., 2022). Despite outstanding advances in online RL (Haarnoja et al., 2018; Silver et al., 2017; Mnih et al, 2015) and iterated offline RL (Wang et al., 2019; Wang & Ba, 2020; Matsushima et al., 2021; Kégl et al., 2021), offline RL remained a challenging problem due to the dependency on the data collection procedure and its potential lack of exploration (Levine et al., 2020).
Although any off-policy model-free RL agent can theoretically be applied to offline RL (Haarnoja et al., 2018: Degris et al., 2012; Lillicrap et al., 2016; Munos et al., 2016), it has been shown that these algorithms suffer from distribution shift and yield poor performance (Fujimoto et al., 2019;
2

---

Under review as a conference paper at ICLR 2023
Levine et al., 2020). To alleviate the problem of distribution shift, conservatism was introduced successfully by several techniques, such as BEAR (Kumar et al., 2019), AlgaeDICE (Nachum et al., 2019), AWR (Peng et al., 2020), BRAC (Wu et al., 2020), and CQL Kumar et al. (2020). The general objective of these methods is to keep the model-free policy close to the behavioral policy, in other words, to avoid wandering into regions of the state/action space where data collected.
Model-based RL has been successfully applied to the online RL setting by alternating model learning and planning (Deisenroth & Rasmussen, 2011: Hafner et al., 2021; Gal et al., 2016; Levine & Koltun, 2013; Chua et al., 2018; Janner et al., 2019; Kégl et al., 2021). Planning is done either decision-time via moddl--predictie et al., 2020; Kégl et al., 2021)), or Dyna style by learning a model-free RL agent on imagined model rollouts (Janner et al., 2019; Sutton, 1991; Sutton et al., 1992; Ha & Schmidhuber, 2018). For instance, MBPO (Janner et al., 2019) trains an ensemble of feed-forward models and generates imaginary rollouts to train a soft actor-critic, which policy is then used to generate new data for model learning. MBPO has been showed to achieve state of the art in continuous control task with the smallest sample efficiency. An adaptation of MBPO to the offline setting is MOPO (Yu et al., 2020). MOPO incorporates conservatism via surrogate MDP where the rewards are penalized with the uncertainty of the model. While MOPO relies on disagreement metrics between the members of the learned ensemble, variance is good proxy to the model estimation error. Similar uncertainty penalized policy search is used in a number of other works (Kidambi et al., 2020; Lee et al., 2021; Shen et al., 2021; Swazinna et al., 2021; Depeweg et al., 2018), while others explore pessimism-bar based decision time planning (Argenson & Dulac-Arold, 2021; Zhan et al., 2021), conservative value learning (Yu et al., 2021; Liu et al., 2021).
control (Draeger et al., 1995; Chua et al., 2018: Hafner al., 2019; Pinneri
suggest the of well-calibrated autoregressive models whose learned
Autoregressive models have been studied in number of previous works for generative modeling in general (Uria et al., 2016; 2013; Papamakarios et al., 2017; Van Den Oord et al., 2016). However, only a handful of papers use them in the context of MBRL (Kégl et al., 2021; Zhang et al., 202 lb; Zhan et al., 2021). Zhang et al. (2021b) used autoregressive models for model-based off- off-pooicy eval- uation, while we focus our study on the important model properties for offline policy optimization. We also adapt metrics from Kégl et al. (2021) to provide complete guide model selection for offline MBRL.
Previous works have tackled hyperparameter selection in online RL (Andrychowicz et al., 2021; Engstrom et al., 2020), MBRL (Zhang et al., 2021a), and offline RL (Paine et al., 2020). showing the sensibility of existing algorithms to hyperparameter choices. La et al. (2022) perform similar analysis to this work. Similarly to us, they base their analysis on MOPO, but they focus on the uncertainty-related hyperparameters while we revisit the model design and architecture
3 PRELIMINARIES
The standard framework of RL is the finite-horizon Markoy decision process (MDP) M (S. A. p. r. Mu, where represents the state space, A the action space, p S x (possibly stochastic) transition dynamics, r S x A + R the reward function, no the initial state distribution, and 7 E [0, 1] the discount factor. The goal of RL is to find, for each state s E S, distribution Ts)) over the action space A, called the policy, that maximizes the expected sum of discounted rewards J(R := Esg porpo.aaem..e.. sispoup Σn1 " 4t)], where H is the MDP hori- zon. Under policy we define the state-action value function (Q-fumction) at an (s, a) E S x A pair as the expected sum of discounted rewards, starting from the state s, taking the action a, and fol- lowing the policy π afterwards until termination:
S the
E 7'r(s,, a;) 80 = 8, 4g = a). We can similarly define the state value function by taking the expectation with respect to the initial action ag: V (s) = (a E "r((s,, 80 = s].
Q a) = -B
In offline RL, we are given a set of transitions D = ((s), at. rf. (141) where N is the size of the set, generated by an unknown behavioral policy T²_ The difficulty of offline RL comes from the fact that optimize the objective J(TT M) with æ * x² In practice the current offline RL algorithms are still provided with online evaluation budget, a setting we will follow in the rest of the paper.
not allowed to interact further with the environment M even though
aim to
3

---

Under review as a conference paper at ICLR 2023
The question of offline policy evaluation (or budget-limited policy evaluation) is an active research direction (see, e.g., Fu et al. (2021b)) and is beyond the scope of this paper.
Model-based RL algorithms use an offline dataset D to solve the supervised learning problem of estimating the dynamics of the environment p and/or the reward function é. For various reasons (stochastic environment, ability to represent the uncertainty of the predictions), the loss function usually the log-likelihook p():
af). The learned model can then be used for policy search under the MDP M = (S,A, f, Mu. 7), which has the same state and action spaces S,A as the true environment .M, but which has the transition probability á and the reward function i that are learned from the offline data D. The obtained optimal policy f = argmax, [[t; MM) is not guaranteed to be optimal under the true MDP M due to distribution shift and model bias. [[T, .M) and MM) are somewhat analogous to training and test scores in supervised learning. with two fundamental differences: i) they are only loosely connected to the actual supervised loss L(D: p) that we can optimize and measure on a data set, and ii) because we are not allowed to collect data using π,, there is a distribution shift between training and test.
Ez
Regarding the type of model, the usual choice is probabilistic model that lears the parameters of a multivariate Gaussian over the next state and reward, conditioned on the current state and action: S141, ~ ( a,) .N (Ho(s,, dz), ( a,)). where 0 represents the parameters of the predictive model. In practice, we use fully connected neural networks as they are proved to be powerful function approximators (Nagabandi et al., 2018; Chua al., 2018), and for their suitability to high-dimensional environments over simpler non-parametric models such as Gaussian processes. Following previous work (Chua et al., 2018), we assume a diagonal covariance matrix for which learn the logarithm of the diagonal entries: σg- = Diag(exp((o)) with lo output by the neural network.
One of the conditions of such a joint model is the conditional independence of the dimensions of the predicted state, which is a strong assumption, especially in the case of functional (or physical) de- pendency. y y-interdependence (Kégl et al., 2021) happens, for example, when angles are represented by sine and cosine. For this purpose, we study autoregressive models that learn a single model per dimension, conditioned on the input of the model (st, a:) and the previously generated dimensions. $141, 8c; at), where d, is the
Formally, Po (8r+1 (t, ar) = po, (s|+1 (e, at) 2 (0, dimension of the state space S.
Conservatism in MBRL requires an uncertainty estimate i(s, a) reflecting the quality of the model in different regions of the state/action space. For this purpose, probabilistic models provide an uncer- tainty estimate by learning the variance of the predictions (in this case under a Gaussian distribution). In noisy environments, this uncertainty estimate represents both the aleatory uncertainty (intrinsic randomness of the environment) and the epistemic uncertainty (model estimation and approxima- tion errors). Conservative MBRL uses the epistemic uncertainty only, so, in practice. the problem of separating the alcatory uncertainty and the epistemic uncertainty is addressed through the use of bootstrap ensembles (Chua et al., 2018). Ensembling consists in having D E N - [1} models, each initialized randomly and trained a set De for L E {1,...: D} generated by sampling with replacement from a common dataset D. Using ensembles, we can compute a disagreement metric to capture the epistemic uncertainty, as opposed to the aleatory uncertainty learned by each member of the ensemble. A detailed discussion about these uncertainty heuristics is provided in Section 4.
4 A BASELINE: MODEL-BASED OFFLINE POLICY OPTIMIZATION (MOPO)
Models á in MBRL are not used in isolation. Their likelhood ratio, precision, and calibratedness (LR, R2, and KS in Section 5.1 and Appendix C) are good proxies, but ultimately their quality is judged when they are used in policy. To compare the dynamic performance of the models ý. we fix the policy to MOPO (Yu et al., 2020), a conservative agent-learning algorithm. MOPO uses a pessimistic MDP (P-MDP) to ensure that the performance of the policy with the model will be lower bound of the performance of the policy on the real system. Yu et al. (2020) show a theoretical lower bound on the true return based on the estimation error of the learned dynamics J [(T, M) 2 Ea-x, s [r(s a) >/ (s, a)l]. In this formula, G% (s,a) is defined by IV ( ) VM (s') which quantifies the effect of the model error on the return. However, this
4

---

Under review as a conference paper at ICLR 2023
requires access the value function of the policy under the true MDP M. which is not given in practice.
To derive algorithm based this theoretical bound, MOPO relies upper uppe bound of G (s, based the integral probability metric: GM (s, a) S sup E [B>="[((s'] - (s') where FF is an arbitrary set of functions. In practice, the authors use ensemble-base uncertainty heuristics to set an upper bound on the true error of the model. The maximum standard deviation among the ensemble members (labeled max aleatory or MA) is considered define penalized reward F(s, a) = f(s, a) Xs(s, a), where i(s,a) = ma..= | a)(l and X is penalty hyperparameter. Yu et al. (2020) then define the associated P-MDP M (S, F, Po: 7) on which a soft actor-critic (SAC) (Haarnoja et al., 2018) agent is trained until convergence (Algo- rithm 1). This algorithm based on Model-based policy optimization (MBPO) (Janner al., 2019) which alternates between model learning and agent learning. MOPO can be described as one it- eration of MBPO, which learns the dynamics model (a bootstrap ensemble of probabilistic neural networks) from the offline dataset and then learns the off-policy agent on a buffer' of rollouts in the P-MDP M. Using this P-MDP prevents the agent from exploiting rewards of highly uncertain regions.
Data: Dataset D, penalty coefficient X, rollout horizon h, Number of SAC training batches B, conservatism penalty i(s,a), u).
Train dynamics model و on offline dataset D; Initialize SAC policy and empty replay buffer D modelss
for 1....... do
Sample initial state So from D;
for i = 1, h do
Sample an action a; (s;):
Sample the next state from the dynamics model 8;-1, p(s;. Compute the penalized reward = ri Aū(s;,
Add sample (s,, a,, P,. 11+1) to Doddet
end
Draw a batch from Moddet update π following SAC schema; end
Algorithm 1: MOPO pseudocode. Yu al. (2020) uses a(s, a) max (o)
we also experimented with two other penalty heuristics by Lu et al. (2022).
While Yu et al. (2020) only tried the max aleatory estimator for the uncertainty heuristic, Lu et al. (2022) introduced concurrent ensemble-based uncertainty heuristics from recent works and de- ployed them in MOPO. Among these, we chose the following two, showing competitive perfor- mance in benchmarks.
Max pairwise difference (MPD) (Kidambi et al., 2020): 0(s, a) = max,, |mb (s, a) " (s, a)ll2 for + l' L....,.. D. This metric captures the largest disagreement among ensemble members as an indicator of model error.
Ensemble standard deviation (ESD) (Lakshminarayanaal et al., 2017):
a(s,a)
V E ((ob, (s, + (ub, a)))) (ā(s, a))² with ((s,
1. Σi 1bg (s, a). is the standard deviation of the ensemble, i.e., the standard deviation of the equally-weighted mixture of the Gaussian densities.
5 EXPERIMENTAL SETUP
We implement our MOPO baseline based on the MBRL library released by Kégl et al. (2021) which is built on top of the RAMP framework (Kégl et al., 2018). We run our experiments with the following models:
Initially, MOPO selected 5% of the batch from the real system dataset D. and 95% of model rollouts. However, Yu et al. (2020) show that this does influence the performance of the algorithm.
5

---

Under review as a conference paper at ICLR 2023
DARMDN(D): Deep autoregressive mixture density net. d, E No feed-forward neural network that learn the parameters (mean and log-standard deviation), and the weights of D E I' univariate Gaussian distributions (dg being the dimension of the state space S). Although our implementation is general, for the rest of the paper we only consider DAR- MDN((1) due to runtime bottleneck, refer to it simply DARMDN.
DMDN(D): Deep mixture density net. A feed-forward neural network that learns the parameters (mean and log-standard deviation) and the weights of D E N multivariate Gaussian distributions. For similar reasons as DARMDN, we only consider DMDN(L) and refer to it as DMDN.
ENS: Ensemble of D E N DMDN models. We implement a vectorized version that is optimized to run on Graphical Processing Units (GPUs). Notice that ENS is equivalent to the original model MOPO used, modulo architectural choices.
For the single models (DARMDN and DMDN), we consider their learned standard deviation (or)) as the uncertainty heuristic to use for reward penalization, which is equivalent to the max aleatory heuristic for an ensemble of a single member. For ENS, we follow the schema by Lu et al. (2022) additional hyperparameter among MA, MPD, and ESD,
and tune the uncertainty heuristic defined in Section 4.
In a typical MBRL loop, the experimental setup consists of alternating model learning and agent learning until the potential convergence of the dynamic performance (episodic return) of the agent on the real environment. For computationally limited hyperparameter optimization, this setup pro- vides continuous feedback on the return of a given model, which helps to early-stop unpromising experiments. This not possible in single-iteration offline RL as we only have access to a static dataset for model learning, and we have to run all the pipeline to compute the evaluation score of a given model. For this purpose, we suggest to decouple model selection and agent selection in an attempt to reduce the overall computational budget of the approach. The experimental setup will then be separated to two independent parts:
Static evaluation of the models: Starting from dataset D, we evaluate the different models by computing supervised-leaming evaluation metrics (Sections 5.1 and C) on held-out validation set. We then select the best model hyperparameters based on these metrics.
Dynamic evaluation of the agents: After selecting the best model p. we train agents by interacting with the P-MDP defined on the learned dynamics of the model. During training, we evaluate the agents by repeatedly rolling-out trajectories in the real environment and computing their average episodic return. For this purpose, we assume access to the true simulator at evaluation time, although the recorded episodes are not made available to training.
A limitation of this approach comes from the fact that static supervised learning metrics do not necessarily reflect the quality of the model for agent learning. We thus investigate how these static metrics predict the overall dynamic performance in Section
5.1 STATIC METRICS
We use metrics introduced by Kégl et al. (2021) in the context of iterated offline reinforcement learning These metrics are designed to assess different aspects of model quality: precision, cali- bratedness, and sensitivity to compounding errors via long- ang-horizon metrics.
Precision is evaluated using the explained variance (R2) which we prefer over the standard Mean- squared error (MSE) because it is normalized and can be aggregated over multiple dimensions. Calibratedness is measured using the Kolmogorov-Smiroov statistics (KS) between the ground truth validation quantiles and a uniform distribution. This metrics indicates if the ground truth values are distributed following the predicted distributions. In the Gaussian case, it is equivalent to the predicted standard deviation being in the order of magnitude of the true model error (although a bad KS may also indicate that the model errors not Gaussian). We also use the likelihood ratio with respect to a baseline score (LR), and the outlier ratio (OR), the rate of data points on which the likelihood is close zero. For the impact of compounding errors, sample population of

---

Under review as a conference paper at ICLR 2023
trajectories (following ground truth actions) and compute Monte- Monte-Carlo estimates of the long-horizig metrics (R2(L) and KS(L) for E {l,...,20) 20 The formal definition of these metrics can be found in Appendix C.
5.2 DYNAMIC METRICS
Similarly to Kidambi et al. (2020); Wu et al. (2020), we compute the average episodic re- turn (undiscounted sum of rewards) of the agent on the real system during training, formally R({(s;, de. rt. $441)](=) = Σm= Pt of the agent, where H is the episode size. We then keep track of the agent with the highest return for the final evaluation. This is not what we should do if the goal was develop a standalone offline RI. algorithm (we could not use the real return to select the agent), but our goal in this paper is to compare models P of the system dynamics, so as long as the agent is selected the same way for all the models, the comparison fair.'
We use the normalized scores introduced in the D4RL benchmark. This metric is linear transfor- mation of the episodic return and takes values between and 100 with corresponding to the score of randomly initialized SAC agent, and 100 to a SAC agent that is trained until convergence on the real system.
EXPERIMENTS & RESULTS
All our experiments are conducted in the continuous control environment Hopper. We use the implementation of OpenAl Gym (Brockman et al., 2016) that based on the Mujoco physies simulator (Todorov et al., 2012). A description of this environment can be found in Appendix B.
For static datasets, we use the D4RL Hopper benchmark that provides four static sets generated by different behavior policies (random: IM steps generated by a randomly initialized SAC agent, medium: 1M steps generated by SAC agent trained until half the score at convergence, medium-replay: All the traces collected by a SAC agent trained until half the score convergence, medium-expert: 2M steps consisting of the medium dataset and 1M steps generated by an expert SAC agent).
Figure 1: Hopper
The results of the static evaluation of the models are summarized in Table 1. The reported scores validation scores on a held-out 10% validation set from the D4RL datasets.
Table 1: Model evaluation results on static datasets. + and mean lower and higher the better, respectively. Unit is given after the / sign.
Model
LRT
OR/10 41 R2/10 "t
KS/10 $1
R2(10/100 4t
KS/10/100 $1
Hopper v2, D4RL random datasel
DMDN
976
0
9986
146
8017
190
DARMDN
1141
D
9957
134
5011
377
ENS
304
9984
217
9442
190
Hopper-v2, D4RL medium datasct
DMDN
1322
9998
117
9938
84
DARMDN
1473
9996
9586
112
ENS
341
9953
233
9296
143
Hogper v2, D4RL medium replay datasel
DMDN DARMDN
361
3
9990
120
9817
141
141
ENS
219
9928
190
6982
115
Ho D4RL medium--xpprt dataset
DMDN
DARMDN
1675
ENS
452
9976
227
9727
108
As related remark, we consider the giant variance of the return both across seeds and across training iterations of the agent crucial, arguably the most important problem of offline RI., but outside the scope of this paper.
7

---

Under review as a conference paper at ICLR 2023
One-step metrics (LR, OR, R2, and KS). We first ob- served that single models are consistently better than the en- semble in terms of one-step metrics. To better understand this result, we propose to use the ground truth test quantiles debugging tool the calibratedness of the models. Fig- ure 2 and Appendix E show that the ensemble model over- estimates its error because the ground truth values are con- centrated around 0.5. We suggest that this is because each DMDN ensemble member has well-calibrated variance, but when we treat the ensemble as mixture model, the vari- ance of the mean adds to the individual variances, "diluting" the uncertainty. Regarding the comparison between DMDN and the autoregressive DARMDN, we observe that although they have similar R2 scores, DARMDN is consistently beat- ing DMDN in terms of KS and LR which depend also on accurate and well-calibrated sigma estimates, important property for conservative MBRL.
120
60
20-
0.9
0.6
Caanthee
Figure 2: Histogram of Hopper's thigh ground truth quantiles, un- der the model distribution (D4RL medium dataset). The legend also in- cludes the value of the KS calibrated- ness metric. The dotted red line indi- cates the ideal case when the quan- tiles follow a uniform distribution.
To push the analysis further, we suggest to look at the dimension-wise static metrics, reported in Appendix D. The results depend on the different datasets, yet some results are consistent and help explain the im- provement that autoregressive models bring over their counterparts. For instance in three out of four datasets, the LR score of the thigh and thigh dot dimensions is an order of magnitude higher for the autoregressive model. We suggest that this is due to the functional dependence that might exist between the different observables, which is easily captured by the autoregressive model as it uses the previously predicted dimensions as input to the next ones.
Long-horizon metrics. Unlike in single-step metrics, here we observe significant degradation in the perfor- mance of DARMDN, both in terms of R2(L) and KS(L) for (I..... 20} (Figure 3 and Appendix F). We sug- gest that this is the due to optimizing the models for single- step likelihood. Outliers (last bin of Figure 2) count little in the single-step likelihood, but may compound when cursing the model through L steps.
randern dataset 1221)
OMON
30
thd
Figure 3: Long horizon explained
variance R2(L) in the D4RL random dataset.
Dynamic evaluation. Table 2 shows the episodic return achieved by the best agent throughout one million steps of SAC training. SAC agents that were trained using DARMIN models performed better on the real system despite their suboptimal long-horizon performance. We suggest that for an agent that trained by one-step Q-learning, such as SAC, only one-step errors matter. Ensemble models improve over DMDN in the random dataset, but scores are comparable or worse in the remaining tasks, although none of the differences are highly significant (they depend on a couple of lucky seeds; phenomenon that muddies the offline RL field). One result seems remarkable: DARMDN models seem to be able to consistently generate agents that go beyond Hopper simply standing up (score of about 30).
Table Model dynamic evaluation: mean + std over 3 seeds of the hyperoptimal SAC agents. The reported score is the D4RL normalized score explained in Section 5.
D4RL dataset
DARMDN
DMDN
ENS
random
31.34 +: 0.50
17.54 9.80
31,97 0.26
medium
66.96 + 6,33
33.12 + 5.96
14.12 + 13.25
medium_replay
70.57 +: 24.45
33.16 2 1.90
28.96
1587
medium. expert
64.18 25.10
32.10 0.67
Correlating static metrics and dynamic scores. The experimental setup we introduce has the advantage of reducing the combinatorics of the hyperparameter optimization process. However, the best agents do not necessarily come from the models with the best static metrics, since these are
"The medium- dataset contains 2M timesteps which is costly in compute and memory. We therefore, omit this experiment.
8

---

Under review as a conference paper at ICLR 2023
measured static data not representative of the distribution which they are applied in the dy- namic run. In an attempt to optimize model selection, we investigate the model properties (static metrics) that most important for dynamic scores. For this, we compute Spearman rank correla- tion (p) and Pearson bivariate correlation (r) between the static score obtained for all models and their respective dynamic scores.
A value of p = 1 indicates that the static metric conserves the same ranking observed in the dynamic evaluation (sufficient for model selection) while r 1 tells that the gap observed in the static metric is in the same scale of the one observed in the dynamic performance (linear correlation). The results in Fig- ure 4 and Appendix G show that in most datasets, the two most correlating metrics are LR (p = 1.0 and = 0.93) and (S( (p 1.0 and r 0.83), metries that evaluate the calibratedness of the mod-
DHANEN
DARPEN
DNC
DNG
303
726
3000
1250
1502
-250
-150
-200
-50
Figure 4: The Spearman and Pearson corre- lations between the episodic return and LR/- KS metrics on the D4RL medium dataset.
els. This underlines the fact that autoregressive models yield the best agent because of their ability to learn one-step uncertainty estimates that represent well their true errors.
D4RL benchmark. We compare the scores obtained with our best agent (based on an autoregres- sive model) with existing literature in the D4RL benchmark and include the results in Table 3.
Table 3: Results the D4RL benchmark. The scores indicate the mean + standard deviation across 3 seeds (6 seeds for MOPO) of the normalized episodic return. We take the scores of MBRL algorithms from their respective papers, and the scores of the model free algorithms and Behavior cloning (BC) from the D4RL paper (Fu et al., 2021a).
D4RI dataset
BC
Ours
MOPO
COMBO
MOREL
SAC
BEAR
BRAC-v
CQI.
random
1.6
0.5
11.7 1 044
17.9
53.6
11.3
9.5
12.0
10.8
medium
29.0
66.9
6.3
28.0 + 12.4
94.9
95.4
0.8
47.6
32.3
86.6
medium.replay
11.8
70.5
+ 24.4
67.5 24.7
73.1
93.6
1.9
10.8
0.9
48.6
medium_expper
111.9
64.1 + 25.1
23.7 1 6.0
111.1
108.7
1.6
4.0
0.8
111.0
Our algorithm achieves better or similar (medium.replay) performance than MOPO, suggesting that potentially the improvement is brought by autoregressive models over neural ensembles, which sup- ports the case of single well-calibrated models. However, we would like to emphasize that there may be other potential reasons behind such differences of performance. For instance Kidambi et al. (2020) append the observations with the unobserved x_velocity to get access the full state of the true MDP. The D4RL dataset version (v0 or v2) has also been criticized as providing different qual- ities for the same dataset (we use v2 similar to Kidambi et al. (2020) and Yu al. (2021) while Yu et al. (2020) uses vO)?. Another important point is the evaluation protocol that sometimes assumes access to the real system for policy evaluation (Kidambi et al., 2020; Wu et al., 2020; Fujimoto et al., 2019; Kumar et al., 2019), and sometimes only reports the online evaluation score of the pol- icy at the last agent- ggent-training iteration (Yu et al., 2020; 2021). Finally the architectural choices of the model design and the chosen policy optimization algorithm can also impact the performance. Consequently, we believe that, beyond designing benchmark data set, providing a unified evaluation framework for offline RL is highly necessary. We plan to explore this direction in future work.
7 CONCLUSION
In this paper, ask what are the best neural networks based dynamic system models, estimating their own uncertainty, for conservativism--ase. MBRL algorithms. We build on a previous work by Yu et al. (2020) (MOPO: model- offline policy optimization) who use bootstrap ensem- bles. Throughout a rigorous empirical study incorporating metrics that assess different aspects of the model (precision, calibratedness, long-horizon performance), we show that deep autoregressive models can improve upon the baseline in Hopper, one of the D4RL benchmark environments. Our results exhibit the importance of calibratedness when the learned variance is used as an uncertainty heuristic for reward penalization. Future work includes confirming our results on other benchmarks and designing a unified offline RI. evaluationn protocol.
Some issues have been raised about this in prior work: Lu al. (2022), Issue 1, Issue 2.
9

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
In order to ensure reproducibility we will release the code at <URL hidden for review>, once the paper has been accepted.
Finally, the hyperparameters of the algorithms are listed in Appendix A and the pseudocode is shown in Section 4.
REFERENCES
Marcin Andrychowiez, Anton Raichuk, Piotr Stañczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What matters for on-policy deep actor-critic methods? large-scale study. In International Conference on Learning Representations, 2021.
Arthur Argenson and Gabriel Dulac-Amold. Model-based offline planning. In International Con- ference on Learning Representations, 2021.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl gym, 2016.
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, and Sergey Levine. Actionable models: Un- supervised offline reinforcement learning of robotic skills. CoRR, abs/2104.07779, 2021. URL https:: /arpps////xxvv./////// org/ abs /2044. 07749.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems 31, pp. 4754-4765. Curran Associates, Inc., 2018.
Thomas Degris, Martha White, and Richard Sutton. Off-policy actor-critic. Proceedings of the 29th International Conference Machine Learning, ICML' 12, pp. 179-186, Madison, WI, USA, 2012. Omnipress. ISBN 978 12803.
Mare Peter Deisenroth and Carl Edward Rasmussen. PILCO: A model-based and data-efficient approach policy search. In Proceedings of the International Conference Machine Learning, 2011.
Stefan Depeweg, José Miguel Lerbandez-Lobato, Finale Deshi-Velez, and Steffen Udluft. Decom- position of uncertainty bayesian deep learning for efficient and risk-sensitive learning. In ICML, 2018.
Andreas Draeger, Sebastian Engell, and Horst Ranke. Model predictive control using neural net- works. IEEE Control Systems, 15:61-66, 1995. ISSN 1066033X. doi: 10.1109/37.46621..
Logan Engstrom, Andrew Nyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study on ppo and trpo. In International Conference on Learning Representations, 2020. URL https: //opnrevieww
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. 2021a. URL. https: /opperveeeeeeeeeeeeeeeeeeeeeeeee net forum? d=
3xC-N3_KjA.
Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Ziyu Wang, Alexander Novikov, Mengjiao Yang, Michael Zhang, Yutian Chen, Aviral Kumar, Cosmin Paduraru, Sergey Levine, and Tom Le Paine. Benchmarks for deep off- off-policy evaluation, 2021b.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2062-2062. PMLR, 09-15 Jun 2019. press/v97 fuj motol 9a html.
URL https: : /tttpps://rrceeeeee h/pp///eeeinngs..//// mmr..
10

---

Under review as a conference paper at ICLR 2023
Yarin Gal, Rowan McAllister, and Carl Edward Rasmussen. Improving PILCO with Bayesian neural network dynamics models. In Data-Efficiie Machine Learning workshop, International Confer- on Machine Learning. 2016.
Omer Gottesman, Fredrik D. Johansson, Joshua Meier, Jack Dent, Donghun Lee, Srivatsan Srini- vasan, Linying Zhang, Yi Ding, David Wihl, Xuefeng Peng, Jiayu Yao, Isaac Lage, Christopher Mosch, wei H. Lehman, Matthieu Komorowski, A. Aldo Faisal, Leo Anthony Celi, David A. Sontag, and Finale Doshi-Velez. Evaluating reinforcement learning algorithms in observational health settings. ArXiv, Ptns/1805.12299 2018.
David Ha and Jürgen Schmidhuber, Recurrent world models facilitate policy evolution. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 3/, pp. 2450-2462. Curran Associates, Inc., 2018.
Tuomas Haaroja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870. PMLR, 10-15 Jul 2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lec, and James Davidson. Learning latent dynamics for planning from pixels. In Proceedings of the 36th In- ternational Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2555-2565, 2019.
Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference Learning Representations, 2021. URL
https: /ettps://pperevviie
net/ aosrumm?dd0oabbwyy.
Eyke Hüllermeier and Willem Waegeman. Alcatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning. 1103))457-500 2021.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on In- ternational Conference on Machine Learning Volume 37, ICML'15, pp. 448-456. JMLR.org, 2015.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alchéé Buu, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Balázs Kégl, Alexandre Boucaud, Mehdi Cherti, Akin Kazakci, Alexandre Gramfort, Guillaume Lemaitre, Joris Van den Bossche, Djalel Benbouzid, and Camille Marini. The RAMP framework: from reproducibility to transparency in the design and optimization scientific workflows. In ICML workshop on Reproducibility in Machine Learning, 2018.
Balázs Kégl, Gabriel Hurtado, and Albert Thomas. Model-based micro-data reinforcement learn- ing: what are the crucial model properties and which model to choose? In International Confer- ence on Learning Representations, 2021. URL https: oppprevieei net / forum?ior iid= p5uy1G94S68.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Bal- can, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 21810-21823. Curran Associates, Inc., 2020. URL cc/ pcc/paper/2020 hat///ile///// htw.a///eeefaaf5444 w
neurips. pdf.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. URL http: 6980.
//arxiv.org/aae
11

---

Under review as a conference paper at ICLR 2023
B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A. Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A sur- vey. IEEE Transactions on Intelligent Transportation Systems, (3(6):499-44222 2022. doi: 241109/TTSS.021.3336
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Sys- tems, volume 32. Curran Associates, Inc., 2019. URL https: /ttp://ecocceceeceeeeeeeeooocoee neurips. sc/paper/2019 http:///2221//11eee22//21/12 htte//iieecc20/3//3///
7b53 A3
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Proceedings of the 34th International Conference on Neural Infor- mation Processing Systems, NIPS'20 20, Red Hook, NY, USA, 2020, Curran Associates Inc. ISBN 9781713829546.
Balaji Lakshminarayanan. Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 6405-66666 6416, Red Hook, NY, USA, 2017.
Curran Associates Inc. ISBN 9781510860964.
Byung-Jun Lee, Jongmin Lee, and Kee-Fung Kim. Representation balancing offline model-based reinforcement learning. In International Conference on Learning Representations, 2021. URL https: /tttp:://ppeneeveww wwweeeeeouuueee forum?id 1atpmmmd-----2-_ee--0 Ri2Y.
Sergey Levine and Vladlen Koltun. Guided policy search. In Sanjoy Dasgupta and David McAllester (eds.), Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 1-9, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR. URL http::::t////....... /ttes://prroeee proceed perooceedings ngs ml . press/ / v22// / evinel 3. html.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprini arXiv:2005.01645; 2020.
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR (Poster), 2016. URL http: /awvvv.vvvv///////////.//////avl org/ abs////////// 00///159.0000 02.91.
Ruizhen Liu, Zhicong Chen, and Dazhi Zhong. Dromo: Distributionally robust offline model-based policy optimization. 2021.
Cong Lu, Philip Ball, Jack Parker-Holder, Michael Osborne, and Stephen J. Roberts. Revisiting de- sign choices in offline model based reinforcement learning. In International Conference on Learn- ing Representations, 2022. URL https: wwwpnevieeeeeeeeeeee/eeeee wwt.rreeieeweeeeeeeeeeeeeeeeeeee forum ?id= 9
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. In International Confer- ence on Learning Representations, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattic, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 51 518(7540):529-533 2015.
Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang. William Paul, Michael Jordan, and Ion Stoica. Ray: A distributed framework for emerging ai applications. In Proceedings of the 13th USENIX Conference on Operating Systems Design and Implementation, OSDI' 18, pp. 561-577, USA, 2018. USENIX Association. ISBN 9781931971478.
Rémi Munos, Thomas Stepleton, Anna Harutyunyan, and Mare G. Bellemare. Safe and efficient off-pc reinforcement learning. In Proceedings of the 30th International Conference Neural Information Processing Systems, NIPS' 16, pp. 1054-1062, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 978151000 9781510838819.
12

---

Under review as a conference paper at ICLR 2023
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Al- gaedice: Policy gradient from arbitrary experience. CoRR, abs/1912.02074, 2019. URL http://arxiv org/ babs/1912 02074.
Anusha Nagabandi, Gregory Kahn, Ronald S. Fearing, and Sergey Levine. Neural network dy- namics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference Robotics and Automation, ICRA 2018, pp. 7559-7566. IEEE, 2018.
Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xi- uyuan Lu, and Benjamin Van Roy. Epistemic neural networks, 2021. URL https://arxix: org/ org/abs// 2107. 08924.
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Çaglar Gülçehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. CoRR, abs/2007.00005, 2020. URL https:://rxx: www.aawwww................oooooe brrg/bb// 2007 .03000.09050 09055.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. In Proceedings of the 31 International Conference Neural Information Process- ing Systems, NIPS' 17, pp. 2335- 2344, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN
9781510860964.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage weighted regression: Simple and scalable off-policy reinforcement learning, 2020. URL htthttsss///ppereeeee
4FVS...
Luis Pineda, Brandon Amos, Amy Zhang, Nathan O. Lambert, and Roberto Calandra. Mbrl-lib: A modular library for model-based reinforcement learning. Arxiv, 2021. URL https:
org/ab abs//bb/// 10159.
Cristina Pinneri, Shambhuraj Sawant, Sebastian Blaes, Jan Achterhold, Joerg Stueckler, Michal Rolinek, and Georg Martius. Sample-efficint cross-entropy method for real-time planning. In Conference on Robol Learning 2020, 2020. URL https:/ corl conf .eanf.gittuul io/ cor12020/ 2a00200ppper_22_1
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1 2021. URL http:://mll. org/ vapers/v2// 20 10-1136..ht html.
Jian Shen, Mingcheng Chen, Zhicheng Zhang, Zhengyu Yang. Weinan Zhang, and Yong Yu. Model- based offline policy optimization with distribution correcting regularization. volume 12975 LNAL, pp. 174-189. Springer Science and Business Media Deutschland GmbH, 2021. ISBN
9783030864859, doi:
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap. Karen Si- monyan, and Demis Hassabis. Mastering chess and shogi by self-play with general reinforce- ment learning algorithm, 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Si- monyan, and Demis Hassabis. A general reinforcement learning algorithm that masters Chess, Shogi, and Go through self-play. Science, 162(64199:1140-1111 2018. ISSN 0036-8075. doi: 10.1 http///eeeee.aareee....
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 1999 2014. URL http: /jmmm werllr.orrggggee wwt..//ppppp//vs...... srivastavalee
Alexander L. Strehl, John Langford, Lihong Li, and Sham M. Kakade. Learning from logged im- plicit exploration data. In Proceedings of the 23rd International Conference Neural Informa- tion Processing Systems- Volume 2, NIPS' 10, pp. 2217-2225, Red Hook, NY, USA, 2010. Curran Associates Inc.
13

---

Under review as a conference paper at ICLR 2023
Richard S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM
SIGART Bulletin, 2:160-163, 7 1991. ISSN 0163-5719. doi: 10.1145/123344122277. URL https:: wtt.aaa.......ooooo http://...111112224.... 1223777
Richard S Sutton, Csaba Szepesvári, Alborz Geramifard, and Michael Bowling. Dyna-style planning with linear function approximation and prioritized sweeping. Moore and Atkeson, 1992.
Phillip Swazinna, Steffen Udluft, and Thomas Runkler. Overcoming model bias for robust offline deep reinforcement learning. Engineering Applications of Artificial Intelligence. 104:104366, 2021. ISSN 1976. doi: http///iinncaaaicceepppp..ee/ // www. www.ssccieeeeeereeooeee.ee. www/
URL https: 620002141
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 5026-5033, 2012. doi:
Benigno Uria, lain Murray, and Hugo Larochelle. RNADE: The real-valued neural autoregressive eensity- In Advances in Neural Information Processing Systems 26, pp. 2175-2183. Curran Associates Inc., 2013.
Benigno Uria, Marc-Alexandre Côté, Karol Gregor, lain Murray, and Hugo Larochelle. Neural autoregressive distribution estimation. Mach. Learn. Res., (7(1):7184-723) jan 2016. ISSN 1532-4435.
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning- Volume 48, ICML pp. 1747-1756. JMIR.org. 2016.
Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang. L. Sifre. Trevor Cai. John P. Agapiou. Max Jaderberg. Alexander Sasha Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine, Caglar Gulcehre, Ziyun Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, pp. 1-5, 2019.
Tingwu Wang and Jimmy Ba. Exploring model-based planning with policy networks. In 8th Inter- national Conference on Learning Representations, ICLR 2020, 2020.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforce- ment learning. arXiv preprint arXiv: 907 907.02057, 2019.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. 2020. URL https: wwwp//ppwneeeeeee////e/e. www.:eee//eeeee wwwpeeooomm.ddeegg........
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 14129-14142. Curran Asso- wttppappe///2//Eiii/ieee ciates. Inc., 2020. URL attps://proceeeeee newproceeeiigg..neriisesse neurips...... Paplea pdf..
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 28954 Curran Asso- ciates, Inc., 2021. URL htttps::/proceeeaiiie setprediiinnn.uuuisspeee. oc cctppappe//22//1eeee Paper.pe pdf.
Xianyuan Zhan, Xiangyu Zhu, and Haoran Xu. Model-based offline planning with trajectory prun- ing. 2021.
14

---

Under review as a conference paper at ICLR 2023
Bangnig Zhang, Raghunandan Rajan, Luis Pineda, Nathan Lambert, André Biedenkapp, Kurtland Chua, Frank Hutter, and Roberto Calandra. On the importance of hyperparameter optimization for model-based reinforcement learning. In AISTATS, 2021a.
Michael R Zhang, Thomas Paine, Ofir Nachum, Cosmin Paduraru, George Tucker, ziyu wang, and Mohammad Norouzi. Autoregressive dynamics models for offline policy evaluation and opti- mization. In International Conference on Learning Representations, 2021b. URL hotps: // /popeneeview.... net for um? kmgjgggg
15

---

Under review as a conference paper at ICLR 2023
A IMPLEMENTATION DETAILS
MOPO Implementation details. Following MBPO, MOPO uses bootstrap ensemble of proba- bilistic neural networks po = -N of) trained independently by log-likelihood maximization. The dynamics model is a four-layer neural network with 200 units each, swish activation functions and ridge regularization with different weight decays on each hidden layer. During the model roll- out generation phase, MOPO first samples initial states from the offline dataset, then performs short rollouts on the learned dynamics (with the horizon h E {1,55).
Our Implementation details. For all the models, we use a neural network composed of a com- mon number of hidden layers and two output heads (with Tanh activation functions) for the mean and standard deviation of the learned probabilistic dynamics. We use batch normalization (loffe & Szegedy, 2015), Dropout layers (Srivastava et al., 2014), and set the learning rate of the Adam optimizer (Kingma & Ba, 2015), the number of common layers, and the number of hidden units as hyperparameters that we tune using the built-in hyperoptimization engine in the RAMP frame- work (Kégl et al., 2018). For the ensemble implementation, optimal hyperparameters and train them by shuffling the training set (a practical variation to boot- strapping (Chua et al., 2018; Pineda et al., 2021)). In all experiments, we use an ensemble of three models. Table 4 shows the grid search ranges for the hyperparameters of our models.
replicate the DMDN model with the
Table 4: Model hyperparameters Grid search range.
Model
DARMEN
DMDN
Learning rate (L.
10 x 10-4
10-3.3 3 10
Number of hidden units (Nbu)
50, 100. 200
100, 200,
Number of common layers (Nel)
1.2
2.3.4
Using the one-million-tin timeessloo-ttmseepppppp D4RL data sets, we first determine the best model hyperparameters (in terms of the aggregate validation static metrics) on subset of 50K training points (and 500K validation points), then we train the best models on 90% of the whole data sets.
For the dynamic scores, we use Ray-tune (Moritz. et al., 2018) to find the optimal hyperparameters (short rollouts horizon h E {1,5, 50, 100), uncertainty penalty heuristic for ensembles u E {Max aleatory (MA), Max pairwise difference (MPD), Ensemble stan- dard deviation (ESD)) on each model/data pair. We use the implementation of the open-source library StableBaselines3 (Raffin et al., 2021) for the SAC agents.
{0.1, 1, 5, 25), and uncertainty
We give the best hyperparameters for each model/data pair in Table 5.
Table 5: The optimal hyperparameters for all model/data pair.
Model
Lr
Nhu
Nel
h
A
D4RI. random dataset
DMDN DARMDN
3 10 4
500
3
5
0.1
10-3
200
2
1.0
ENS
3x 10
500
3
5
5
ESD
D4RL medium dataset
DMDN
3x 10-4
500
3
5
0.1
DARMON
10
200
5
0.1
ENS
3 10
500
3
50
25
ESD
D4RL medium-ruplay dataset
DMDN
x 10
500
3
5
0.1
DARMDN
10-3
200
2
100
0.1
ENS
10-4
3
MPD
D4RL medium expert datasel
DARMDN
10-3
1
100
0.1
ENS
3 10-3
200
3
5
0.1
ESD
B CHARACTERISTICS OF THE BENCHMARK ENVIRONMENT: HOPPER
The hopper environment consists of robot leg with 11 observations (rootz, rooty, thigh. leg. foot, rootx dot, rootz dot, rooty dot, thigh dot, leg dot, foot dot) including the angular positions and ve- locities of the leg joints, except for the position of the root joint. The action is control signal
16

---

Under review as a conference paper at ICLR 2023
applied by three actuators located in the three joints. The goal of the system is to hop forward as fast as possible (maximizing the velocity in the direction x)) while applying the smallest possi- ble control (measured by ||o,12)), and without falling into unhealthy states (terminal states where the position of the leg is physically unfeasible). We detail the characteristics of the environment in Table 6.
Table 6: Hopper characteristics.
dimension of the observable space
dimension of the action space
horizon
roward function
1000
0.1 ||aell + { healthy }
C
STATIC METRICS
define our static metrics based on the marginal one-dimensional densities of each predicted feature. For autoregressive models, these densities are learned separately while con-autoregressive models learn multivariate density that is separated using the product rule:
d. Ps (p((5/+12z)
p($441/81 (t, = Pı (#141 (a))) II J=2
where = - 141 8t, at).
All metries will be evaluated on data set D of size N, consisting of transitions in the real system. D stands for a held-out validation set on the offline training datasets.
EXPLAINED VARIANCE (R2): Measures the precision of the mean predictions.
R2(D;0;j E (I,...,,d)) dd)) NV Σin (SLe 166(5).,
E (s1.41 - 8(/11)
where 0 are the model parameters and 8/41 the sample mean of the jth dimension of 8141. R2
between 0 and 1, the higher the better.
LIKELIHOOD RATIO (LRR): The average log- aigllikelhoodd evaluated on D is defined
N
L(D; d,)) =
N Σ
1
log po (sc,4118s,s ai,))
where Po is the PDF of the Gaussian distribution induced by the learned parameters: N (Hu(st, a,), at)). The log-likelihood is uninterpretable unitless measure that ide- ally want to maximize. Following Kégl et al. (2021), we normalize L with the log-likelihooe of multivariate unconditional Gaussian distribution (Chaseline) whose parameters estimated from the dataset D.
((D:::je(),......)
LR(D;0:
(3)
Creccine (D:jE(1....d.))
OUTLIER RATE (OR): practice, the log- likeliiihoodd estimator dominated by out-of-distribution test points where the likelihood tends to zero. For this reason, omit the data points that have likelihood smaller or equal to Puni 1.47 10-6 when computing the LR. The OR metric the proportion of data points that fall in this category. Formally:
K(s,, ar, S411) D (g a4) > Pmin
D (1,
d.)) =
(4)
OR is between 0 and 1, the lower the better.
17

---

Under review as a conference paper at ICLR 2023
005 reward
10
CALIBRATEDNESS (KS): This metric is computed using the quantile (under the model distribution) of the ground truth val- ues. Hypothetically, these quantiles are uniform if the error we make on the ground truth is a random variable distributed Gaussian having the predicted standard devi- ation, a property we characterize as calibratedness. To as- sess this, compute the Kolmogorov-Smirno (KS) statis- tics. Formally, starting from the model cumulative distribu- tion function (CDF) (s az), we define the empirical CDF of the quantiles of ground truth values by Fag(2) Kisaous
0.
according
0.2
0.
Quantiies
(s (1))))))) for E [0, 1]. We denote by (x) the CDF the uniform distribution over the interval [0, 1], and define the KS statistics the largest absolute difference between the two CDFs across the data set D:
Figure 5: Kolmogorov- (KS) statistic (in red) of the pre- dicted reward.
KS(D;0;j E (I,...,da)) dd.)
(S an,)) - - (slet1ls (5)
max (E.....,N)
The KS score is between zero and one, the lower the better.
LONG HORIZON METRICS KS(L) AND R2(L): Although the models trained optimize the one-step prediction log-likelihood score, we want to assess their precision and calibratedness longer horizon. Indeed, during the agent learning phase we sample trajectories of multiple steps which can lead to uncertain regions in the case of significant compounding errors down the horizon. For this purpose. we use ground truth actions from a system trace generate a population of n E N trajectories of length Lmax: YL 1:t compute a Monte-Carlo estimate of the R2(L) metric, for L = O,.... Lmax. using the sample mean ( St, az) A E>Ey, S14L as approximate prediction. For the KS(L) metric, we estimate the model CDF with the order statistic Fo(s4t()s,,a)) I Sec+L among the population of trajectories.
and use the mean predictions to
n
D PER-DIMENSION STATIC METRICS
In all plots, as in Table 1, the KS score is multiplied by 1000, and the OR and R2 scores are multi- plied by 10000,
18

---

Under review as a conference paper at ICLR 2023
D.1
RANDOM DATASET
dAtirrandom 12 score dÁíi random dalasel
10000
5050
submission
DARMON
9843
DMON
ENE
9900
10012
roely
thigh
192
foot
roots, dot rootz, dot rooty, dct thigh, dol log, dot foot, dot cbs roward
obs dim
ks score dAri random dalaset
230
submission
150
DARMON
DMON
EME
90
10002
100ty
tlih
log
toot
noo, cot rootz, cot rooty dot thigh det log, dof
toot dot dbs reward
cas dim
Ir acorr did random dataset
5000
3030
eubmission DMDN
2030
ENS
1030
10:02
rodly
thigh
lgg
Tool
mobt dot rostz del moty, dot thigh, dot leg, dot obs, dim
foot dof cbs. roward
scor0 d41 random dataset
4.0
3.6
3.0
submission
DMDN
EMS
00
1038z
rcoly
Brihh
Ing
fool
roeb__dol obs_ dim
roete_ctll rodly_dot
thigh_dto
Ieg_dot
foot_dol obe_ reeward
Figure 6: Per-dimension static metrics in the random dataset. The metrics include: R2, KS, LR, and OR. They are computed for all Hopper observables, addition to the predicted reward (labeled obs obs-reward). The dots show the mean + the standard deviation among the training and the validation scores for each metric.
19

---

Under review as a conference paper at ICLR 2023
D.2
MEDIUM DATASET
12 score déd medium dataset
13000
1980
submission
DARMON
8800
ENS
07550
8700
rootz
roply
Ehihh
lgg
tool
rocbr_ott rootr_dot rooly_dot thight log_tol foot_dot obs_reward
obs_dlm
ks score ddri medium dataset
360
300
200
submission
DARMON
150
ENS
m1007
Inny
thigh
leg
mol maiz_dot motty_dtt thigh_dot obs_ dim
Ingudat
foot_dott the rewerd
9coro 64d medium dataset
10000
submission
DARMON
8000
EKS
4000
2000
100t2
r00ty
900
to0l
Root__00 dot rootz_dot rooty_dot thigh_dot leg_dol foot_oct obs_leward
005 dim
score d4H medium datasot
submission
DARMON
3
DNDN
2
ENS
motz
modty
100l7 rodyy.de triph obs, dim
leg, Gol dol
Figure 7: Per-dimension statie metrics in the medium dataset. The metrics include: R2, KS, LR, and OR. They computed for all Hopper observables, in addition to the predicted reward (labeled obs reward). The dots show the mean the standard deviation among the training and the validation scores for each metric.
20

---

Under review as a conference paper at ICLR 2023
D.3 MEDIUM-REPLAY DATASET
22 score c4r medium replay datasst
13000
9600
submission
6700
DARMON
8600
ENS
8500
9400
rootz
roply
Eighh
lgg
tool
roctr_oot rootr_dot rooly_dol thigh log_tol foot_ obs_reward
obs_dlm
ks score d4ri h_a__mediu_mm__e paaplnrrppaaaaaaaa datasat
250
200
150
submission
DARMON
100
ENS
60
mo/27
innly
thigh
leg
mol maiz_dot moity_dt thigh_dot obs dim
lngudat
foot_dot the rewerd
Ir score d4ri medium replay dacasat
5000
4000
3000
submission
DARMON
2000
EKS
1000
10012
1001y
trigh
160
toct
roax dot rodtz_dot inoty_dot #igh_dot I8g foot_dot obs_leward
0bs 3m
score d4d, madium, roplay dataset
12
submission
DARMON
DMDN
ENS
mot7
10xty
t4gh
18g
100lK dot rootz, mony thigh det obs_dm
leg det
dor reward
Figure 8: Per-dimension static metries in the medium-replay dataset. The metries include: R2, KS. LR, and OR. They and are computed for all Hopper observables, in addition to the predicted reward (labeled obs_reward). The dots show the mean the standard deviation among the training and the validation scores for each metric.
21

---

Under review as a conference paper at ICLR 2023
D.4
MEDIUM-REPLAY DATASET
12 scora d4ri mediu dataset
13000
1980
8960
9840
8920
submissioo
DARMDN
ENS
198U
8860
8840
rootz
roply
Eighh
lgg
tool
roctr_oot rootr_dot rooly_dol
thightdot log_tol foot_dot obw_reward
obs_dlm
ks score - d4ri mwdiummdmmmmmm export datasot
300
150
DARMDN
ENS
60
m1077
innly
thigh
leg
moit_dat maiz_det motty_or thigh_dot obs_dii dim
legudat
foot_dttt rhe rewerd
Ir score ddd madium, expert dataset
12000
10000
8000
8000
submission NARMON
2000
10ot2
rooty
th.hn
90
tool
Rooty rootz_dot rooty_oot thigh_dot leg_dol
foot_oct obs_reward
0bs_ dim
or scoro - d4r medium expert datasa!
submission
3
nerMoN
2
motz
modty
100l7, mony dct triph dor leg, Gol obs, dim
dol raward
Figure 9: Per-dimension static metrics in the medium-expert dataset. The metrics include: R2, KS, LR, and OR. They computed for all Hopper observables, in addition to the predicted reward (labeled obs_reward). The dots show the mean + the standard deviation among the training and the validation scores for each metric.
22

---

Under review as a conference paper at ICLR 2023
E ERROR QUANTILE HISTOGRAMS
E.1
RANDOM DATASET
histagram rootz
histagram rosty
histogram thigh
100
DAMRE ledt
100
(68%
80
Coart
60
COLET
60
40
20
Quantiles
Quantiees
QuartBees
histogram- - leg
histogram - foot
histogram rootx dot
2864 kw0
100
2405, 10-
80
(M0% bv1M
C0
Caunt
60-
80
20
20
20
0.00
0.25
0.50
0.75
1.00
6.00
0.25
0.50
075
1.00
0.00
0.25
0.50
0.75
1.00
Quantiles
Quantles
Quar Guartiles
histegram rootz dot
histagram- rooty dot
histogram thigh dot
04rees b--544
100
Darabe h-10
DC 0-224
De: 6-307
80
06 9-330
Caunt
60
Czunt
60
40
40
20
20
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Quanbles
Quantles
histogram leg_dot
histogram am
histogram
obs
100
120
Daraee bvi1 ard
80
Qoant
80
40
20
20
0.00
Quantiles
Quanties
QuartBee
Figure 10: Per-dimension Error quantile histograms in the random dataset. The plot shows the ground truth validation quantiles under the model distribution. The legend includes the value of the KS calibratedness metric, and the dotted red line indicates the ideal case when the quantiles follow uniform distribution. The histograms are computed for all Hopper observables, in addition to the predicted reward (labeled obs reward).
23

---

Under review as a conference paper at ICLR 2023
E.2
MEDIUM DATASET
histogram - rootz
histogram - rooty
histogram thighh
In-34
(ACJEe 1n-30
148. Go-302
100
Chunt
60
Cbunnt
40
20
20
20
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Quantles
Quantles
histogrem leg
histogram
histogram rootx,
CARDE MoxT
100
100
(M0%
100
Count
80
Caantt
40
20
20
20
Quantiles
Quantles
Quentiles
histegram- rootz dot
histogram rooty dot
histogram thigh dot
100
Eede twi28
(406, leids
(40% Iu-15
Count
60
40
0.00
0.25
0.50
0.75
1.00
0
0.00
0.25
0.50
0.5
1.00
0.00
0.25
0.50 Quantiles
0.75
1.00
Quantiles
Quantles
histogram - leg_dot
histogram- foot dot
histogram lawooobsseeaaaaeeeeeee reward
120
100
Cauitt
Dhint
60
80
20
20
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Figure 11: Per-dimension Error quantile histograms in the medium dataset. The plot shows the ground truth validation quantiles under the model distribution. The legend includes the value of the KS calibratedness metric, and the dotted red line indicates the ideal case when the quantiles follow a uniform distribution. The histograms are computed for all Hopper observables, addition to predicted reward (labeled obs reward).
24

---

Under review as a conference paper at ICLR 2023
E.3 MEDIUM-REPLAY DATASET
histogram - rootz
histogram - rooty
histogram thigh
In:30
100
120
65 1i-308
198 Go-d12
100
Chunt
60
Chimn
60
80
80
40
20
20
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Quantles
Quantles
histogrem leg
histogram
histogram rootx,
100
(M0%
80
Count
40
20
20
Quantiles
Quantles
Quentiles
histegram- rootz dot
histogram rooty dot
histogram thigh dot
800
Enta twv15
00
(406, Io-00
(40%) bu-12)
60
60
40
20
0.00
0.25
0.50
0.75
1.00
0
0.00
0.25
0.50
075
1.00
0.00
0.25
0.50 Quantiles
0.75
1.00
Quantiles
Quantles
histogram - leg_dot
histogram- - foot dot
histogram obs reward
100
100
Ino docu
Caunt
60
Caint
60
40
20
20
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Quantles
Figure 12: Per-dimension Error quantile histograms in the medium-replay dataset. The plot shows the ground truth validation quantiles under the model distribution. The legend includes the value of the KS calibratedness metric, and the dotted red line indicates the ideal case when the quantiles follow a uniform distribution. The histograms are computed for all Hopper observables, addition to the predicted reward (labeled obs obw_rew)rd).
25

---

Under review as a conference paper at ICLR 2023
E.4
MEDIUM-EXPERT DATASET
histogram rootz
histogram - rooty
histogram thigh
100
100
80
Quant
Chim
60
60
40
40
20
0.00
0.25
0.50
0.75
1,00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50 Quantiles
0.75
1.00
Quantles
Quantles
histagrem leg
histogram
histogram rootx,
80
60
Cotut
40
20
20
Quantiles
Quantles
Quentiles
histegram- rootz dot
histogram rooty dot
histogram thigh dot
04 tar
Be: 6w-17
06 mnd13
80
80
80
Count
40
20
0.00
0.25
0.50 Quantiles
0.75
1.00
0
0.00
0.25
0.50 Quantles
0.5
1.00
0.00
0.25
0.50
0.75
1.00
Quantiles
histogram - leg_dot
histogram- foot dot
histogram lawoooosssaaaaaeeeeeee reward
DMACK budz)
120
100
100
00
Caunt
60
Daint
60
40
80
20
20
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Quantles
Figure 13: Per-dimension Error quantile histograms in the medium-expert dataset. The plot shows the ground truth validation quantiles under the model distribution. The legend includes the value of the KS calibratedness metric, and the dotted red line indicates the ideal case when the quantiles follow a uniform distribution. The histograms are computed for all Hopper observables, addition to the predicted reward (labeled obs obw_rew)rd).
26

---

Under review as a conference paper at ICLR 2023
F
LONG HORIZON METRICS
20
0.4
0.3
Harfon
dé1 medium dotaset (2(L)
d4rl medium dotaset keil)
0.20
0.15
DARITH DNUR
0.02
2.5
10.0
12.5
15.0
115
20.0
2.5
5,0
75
10.0
12.5
15.0
175
20.0
dari medim replay dataset (26))
dAri dataser
ke0
1.0
12
15.0
Horton
34d medium expert dacaset
(eddiu expert
1.0
0.00
0.8
0.6
1.0
12.5
15.0
5,0
12.5
15.0
Figure 14: Long horizon explained variance R2(L) and calibratedness KS(L). The metric is aggre- gated by averaging Hopper's 's observables and predicted reward.
27

---

Under review as a conference paper at ICLR 2023
G STATIC AND DYNAMIC METRICS CORRELATIONS
G.1 RANDOM DATASET
Ir(1) -0.50,
30
30
opisodic return
cpisotic roturn
25
DARMON
DARMON
DMDN
DMDN
600
800
1000
1200
1400
2.0
1.8
1..
-144
-1.2
1..
k(1)
on(11
(2(1) rho -1.00,
-0.57
-ks(1) -0.50,
-0.21
opisodic return
opisodic raturn
25
DARMON
DARMON DMDN
DMDN
ENS
ENS
9960
9970
9980
9990
10000
-255
-200
-175
-100 kS(1)
-125
-100
-75
-50
(2(1)
r2(10) -1.00,
ks(10) rho -1.00,
DARMON DMDN ENS
cpisadic return
opisadic roturn
DARMON DMDN
ENS
9300
9400
9500
9600
9700
9800
9900
-140
-130
-120
-110
-100
-90
(2(10)
ks/10)
r2(20)
-0.50,
-0.79
-ks|20)
-1.00,
DARMON
30
30
DMDN
opisodic return
cpisotic roturn
25
DARMON
OMON
8600
8800
9000
9200 (2(20)
9400
9600
9800
115
150
145
133
1300
ks/20)
Figure 15: The Spearman and Pearson correlations between the episodic return and the statie metries (LR, negative OR, R2(1), negative KS(1), R2(10), negative KS(10), R2(20), negative KS(20)) in the random dataset. To uniformly evaluate the metrics' positive correlation with the episodic return, we take the negative of the metrics where the smaller is the better (KS(L) and OR).
28

---

Under review as a conference paper at ICLR 2023
G.2 MEDIUM DATASET
Ir(1) rho 1.00, r= 0.85
-or(1) tho -0.87, -0.93
DARMON
DARMON
DMDN
DMDN
spbcdiic return
ENS
spisodic return
ENS
30
20
20
400
600
800
1000
1200
1400
-2.0
-1.8
-1.6
-1.4
-1.2
-1.0
r2(1) rho 0.50, r 0.75
kk((1) tho 1.00, r= 0.94
DMDN
DMDN
apisodic return
ENS
apisadic return
ENS
30
30
20
9960
9970
9980
9990
10000
-225
-200
-175 -150
-125
-100
-75
-50
r2(10) rha 0.50, = 0.30
-s/100 rho 0.50, r 0.40
DARMON
DARMON
DMDN
DMDN
episodic return
ENS
opisodic return
ENS
30
30
20
20
9300
9400
9500
9600
9700
9800
9900
-140
-130
-120
110
-100
-00
r2(20) rho -0.50, -0.71
-ks(20) rho 0.50, -0.04
DARMON DMDN
DARMON
DMDN
Sooodll return
ENS
spisodic return
ENS
20
20
8600 8800
9000
9200
9400
9600
9600
-155
-150
-145
-140
-135
-130
Figure 16: The Spearman and Pearson correlations between the episodic return and the static metrics (LR, negative OR, R2(1), negative KS(1), R2(10), negative KS(10), R2(20), negative KS(20)) in the medium dataset. To uniformly evaluate the metrics' positive correlation with the episodic return, take the negative of the metrics where the smaller the better (KS(L) and OR).
29

---

Under review as a conference paper at ICLR 2023
G.3
MEDIUM-REPLAY DATASET
Ir(1) rho 1.00, r 0.67
-or(1) tho -0.87, -1.00
DARMON
DARMON
OMDN
DMDN
upioddic return
60
ENS
spisodic return
60
ENS
30
30
400
600
800
1000
1200
1400
-2.0
-1.8
-1.6
-1.4
-1.2
-1.0
r2(1) rho 0.50, r 0.54
-k((1) rho 1.00, r= 0.82
apbsoicc return
60
DMDN
ENS
apisodic return
60
DMDN
ENS
30
30
9960
9970
9980
9990
10000
-225
-200
-175 -150
-125
-100
-75
-50
r2(10) the 0.50, = 0.04
ks/100 rho 0.50, r 0.14
DARMON DMON ENS
DARMON
DMDN
egisodic return
60
cpisodic return
60
ENS
30
30
9300
9400
9500
9600
9700
9800
9900
144
133
-120
1100
-100
-90
r2(20) rho -0.50, -0.87
-ks(20) rho 0.50, -0.31
DARMON
DARMON
DMDN
DMDN
spisodil return
60
ENS
opisodic return
60
ENS
30
30
8600
8800
9000
9200
9400
9600
9800
-155
-150
-145
-140
-135
-130
Figure 17: The Spearman and Pearson correlations between the episodic return and the static metrics
(LR, negative OR, R2(1), negative KS(1), R2(10), negative KS(10), R2(20), negative KS(20)) in the medium-replay dataset. To uniformly evaluate the metrics' positive correlation with the episodic return, take the negative of the metries where the smaller is the better (KS(L) and OR).
30

---

Under review as a conference paper at ICLR 2023
G.4 MEDIUM-EXPERT DATASET
lr(1) rho 1.00, r= 1.00
-or(1) tho -1.00, -1.00
DARMON
DARMON
ENS
EN5
upbbddic return
spisodic return
50
50
40
400
600
800
1000
1200
1400
-2.0
-1.8
-1.6
-1.4
-1.2
-1.0
r2(1) rho 1.00, r 1.00
ks(() tho 1.00, r= 1.00
(AARMDN
ENS
ENS
apbsdic return
apisadic return
50
50
40
9960
9970
9980
9990
-225
-200
-175
-150
-125
-100
-75
-50
r2(10) the 1.00, == 1.00
ks/100 rho 1.00, r 1.00
DARMON
ENS
ENS
episodic return
opisodic return
50
80
40
9300
9350
9400
9450
9500
9550
9600
144
144
-135
-130
125
-120
115
-ks10)
r2(20) rho -1.00. = -1.00
-ks200 rho 1.00, r 1.00
DARMON
DARMON
ENS
EN5
spibddic return
opisodic return
50
50
40
8600
8800
9000
9200
-158
-157
-196
-155
-154
Figure 18: The Spearman and Pearson correlations between the episodic return and the static metrics (LR, negative OR, R2(1), negative KS(1), R2(10), negative KS(10), R2(20), negative KS(20)) in the medium-expert dataset. To uniformly evaluate the metrics' positive correlation with the episodic return, take the negative of the metries where the smaller is the better (KS(L) and OR).
31


Reviews:
[{"review_id": "jMCom2vBg1X", "reviewer": "Reviewer_eNAT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper deals with uncertainty quantification of models in model based offline RL. \nThe paper uses an autoregressive dynamic model and shows that a single model that can provide an uncertainty estimate can be used as an alternative to uncertainty estimation using ensembles. Different methods are compared on the hopper benchmark from the D4RL benchmark collection. ", "summary_of_the_review": "The paper makes an interesting contribution to the discussion of uncertainty quantification of dynamic models.", "strength_and_weaknesses": "**Strengths**\n* The paper is well written.\n* The paper makes an interesting contribution to the discussion of uncertainty quantification of dynamic models.\n\n**Weaknesses**\n* The empirical investigation relies on only one environment (Hopper). \n\n**Further suggestions for improvement**\n* The statement \"Our algorithm achieves better performance than MOPO in all datasets\" should be softened, e.g. \"Our algorithm achieves equal or better performance than MOPO in all datasets\", because in medium_replay the performance is not significantly better.\n\n* In Table 2 and Table 3 standard deviations are given after $\\pm$. Correctly, the $\\pm$ sign is used to indicate the uncertainty of the measurement. So a confidence interval or a standard error.\nThis serves to ensure that the statistical significance of differences in the measured values can be easily grasped.\n\n* Uncertainties should be stated with one or at most two valid digits.\n\n* The number of decimal places in A $\\pm$ B must match. E.g. \"31.34 ± 0.5\" -> \"31.3 ± 0.5\".\n\n* Using only three repetitions (seeds) leads to unreliable results. If it is somehow possible, there should be more, e.g. 10 or 50.\n\n* In \"Other methods include behavior regularized policy optimization\", also (Fujimoto et al., 2019) should be cited.\n\n* Similar to the present paper, (Depeweg et al., Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning 2018) estimates uncertainty without ensemble and uses the uncertainty for conservatives in offline RL, it should, therefore, be cited. However, long roll-outs are used for the return estimation and no Q-function is used, so that the approach is structurally clearly different from MOPO.\n\n* It is claimed \"Conservatism in MBRL is achieved by uncertainty-based penalization of the model predictions.\" however, this is only one possible way, another possibility is also in model-based the behavior regularized policy optimization, e.g. (Swazinna et al, Overcoming model bias for robust offline deep reinforcement learning, 2021). Presumably other techniques exist, so it is probably better to write, e.g., \"Conservatism in MBRL is frequently achieved by uncertainty-based penalization of the model predictions.\"\n\n* It would be interesting to study the behavior in a stochastic environment, because only in stochastic environments the problem exists to separate aleatory and epistemic uncertainty (future work).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EjF0EiK8y6", "reviewer": "Reviewer_xTqk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper uses a standard off-policy moder-based reinforcement learning algorithm and uses a set of dynamics models (autoregressive models, ensembles & mixture density models) on a single benchmark. \n\nThey then test the algorithms using a set of metrics and found that auto-regressive models appear to give improved performance\n\n", "summary_of_the_review": "Overall, I cannot recommend acceptance because I unfortunately do not see a clear contribution and significance of the presented work.", "strength_and_weaknesses": "Strengths:\n- the use of auto-reggressive models and its comparison to  standard ensembling methods appears novel\n\nWeaknesses:\n- Significance : The methodology the authors present (offline model-based RL with a parameterized policy) is standard. The algorithms for model learning (deep ensembles, auto-regressive models) are known and testing on a single benchmark does not provide any insight into their advantages and disadvantages \n- Limited experiments: The authors say in the conclusion \"In this paper, we ask what are the best dynamic system models, estimating their own uncertainty,\"  -- but all tests are done on a single benchmark.\n- Limited comparisons: There is more work on using uncertainty-aware models in offline model-based RL, such as Gaussian Processes, Bayesian Deep Learning or using Dropout-mechanisms ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "DjTCaTs4NX", "reviewer": "Reviewer_ArJf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers using autoregressive models to predict the different state dimensions when learning a transition model. The authors argue that autoregressive models are able to capture correlations between state dimensions and therefore work better than standard neural nets in the context of model-based offline RL. Experiments support these claims.", "summary_of_the_review": "This paper proposes using autoregressive models for offline RL, arguing that these are better able to capture correlations between state dimensions. Experiments show a clear advantage of single neural nets and ensembles. The authors look at various performance metrics and check their correlation with final control performance. The contributions are worthy of acceptance, though the paper is weighed down by its narrow experimental scope. Repeating these experiments in a few more mujoco tasks would greatly strengthen this paper.", "strength_and_weaknesses": "**strengths**\n\n* The paper questions the use of ensembles in certain scenarios. Given the amount of attention that ensembles have received over the last years, the results of this paper are quite interesting.\n* The experiments are in favor of the approach.\n* Looking at different metrics and their correlation with RL performance is quite helpful and interesting.\n\n**weaknesses**\n\n* The experiments feature a single mujoco environment. This is my main reason for giving the paper a 6 over an 8.\n* Autoregressive models are more computationally expensive than single neural nets and ensembles (though this might depend on the implementation).\n* The autoregressive models are clearly outperformed when one considers L-step metrics. This would have ramifications about the applicability of the results. In works like PETS [1], predictive performance over a longer horizon is essential. The results in this paper suggest that autoregressive models are not a good fit for that. While I can understand that 1-step performance and L-step performance might not necessarily agree (e.g. one can sacrifice one to improve the other), it would be a little odd that one model category is better in the offline setting, while the other is better in the other.\n* Given that the authors' initial pitch for autoregressive models is about correlations between state dimensions, I think the experiments section should check if there is indeed a difference between the two model classes in this regard. Perhaps a toy experiment involving a pendulum, where one could check the same sine-cosine scenarion that was described in the paper.\n\n----\n\n[1] Deep Reinforcement Learning in a Handful of Trials, https://arxiv.org/pdf/1805.12114.pdf", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject