Revisiting Design Choices in Offline Model Based Reinforcement Learning
Authors: Cong Lu
Philip Ball
Jack Parker-Holder
Michael Osborne
Stephen J. Roberts
Venue: ICLR 2022 Spotlight (2022)
Keywords: Model-Based Reinforcement Learning
Offline Reinforcement Learning
Uncertainty Quantification

Abstract: Offline reinforcement learning enables agents to leverage large pre-collected datasets of environment transitions to learn control policies, circumventing the need for potentially expensive or unsafe online data collection. Significant progress has been made recently in offline model-based reinforcement learning, approaches which leverage a learned dynamics model. This typically involves constructing a probabilistic model, and using the model uncertainty to penalize rewards where there is insufficient data, solving for a pessimistic MDP that lower bounds the true MDP. Existing methods, however, exhibit a breakdown between theory and practice, whereby pessimistic return ought to be bounded by the total variation distance of the model from the true dynamics, but is instead implemented through a penalty based on estimated model uncertainty. This has spawned a variety of uncertainty heuristics, with little to no comparison between differing approaches. In this paper, we compare these heuristics, and design novel protocols to investigate their interaction with other hyperparameters, such as the number of models, or imaginary rollout horizon. Using these insights, we show that selecting these key hyperparameters using Bayesian Optimization produces superior configurations that are vastly different to those currently used in existing hand-tuned state-of-the-art methods, and result in drastically stronger performance.

Published as a conference paper at ICLR 2022
REVISITING DESIGN CHOICES IN OFFLINE MODEL-BASED REINFORCEMENT LEARNING
Cong Lu", Philip J. Ball", Jack Parker-Holdets Michael A. Osborne, Stephen J. Roberts
Department of Engineering University of Oxford
ABSTRACT
Offline reinforcement learning enables agents to leverage large pre-collected datasets of environment transitions to learn control policies, circumventing the need for potentially expensive or unsafe online data collection. Significant progress has been made recently in offline model-basee reinforcement learning, approaches which leverage a learned dynamics model. This typically involves constructing a probabilistic model, and using the model uncertainty penalize rewards where there is insufficient data, solving for a pessimistic MDP that lower bounds the true MDP. Existing methods, however, exhibit a breakdown between theory and practice, whereby pessimistic return ought to be bounded by the total variation distance of the model from the true dynamics. but is instead implemented through a penalty based on estimated model uncertainty. This has spawned a variety of uncer- tainty heuristics, with little to no comparison between differing approaches. In this paper, compare these heuristics, and design novel protocols to investigate their interaction with other hyperparameters, such as the number of models, or imaginary rollout horizon. Using these insights, show that selecting these key hyperparam- eters using Bayesian Optimization produces superior configurations that are vastly different to those currently used in existing hand-tunce state-of state-oftthe-eee of-the-art art methods, and result in drastically stronger performance.
INTRODUCTION
In offline (or batch) reinforcement learning (RL) (Ernst et al., 2005; Levine al., 2020), the goal is to leverage offline datasets of transitions in an environment to train a policy that transfers to an online task. This could have vast implications for using RL in real-world settings, agents can make use of ever-increasing amounts of data without the need for an accurate simulator, while also avoiding expensive and potentially even unsafe exploration in the environment.
Model-based reinforcement learning (MBRL) has recently shown promise in this paradigm, obtaining state-of-the-e the-arr performance on offline RL benchmarks (Kidambi al., 2020; Yu et al., 2021), improving upon powerful model-free approaches (e.g. Kumar et al. (2020)). MBRL works by training a dynamics model from the offline data, then optimizing a policy using imaginary rollouts from the model. This allows the agent to learn from on-policy experience, as the model is agnostic to the policy used to generate data, making possible to achieve high returns using data collected from even a random policy. Furthermore, recent work has demonstrated the utility of world models beyond maximizing return, such as generalizing to unseen variations in an environment (Ball et al., 2021), transferring to tasks (Yu et al., 2020), and learning with safety constraints (Argenson & Dulac-Arnodd, 2021). Therefore, the case for MBRL in offline RL is clear: not only does it represent faate-of-the-asd in terms of performance, but it also provides the opportunity in the offline data to generalize onto tasks beyond those encoded by the behavior policy. This is crucial for offline RL to be useful for real-world tasks (Dulac-Arnal Arnold al., 2021), where there will inevitably be differences between the data and desired task.
maximize the signal
However, common failure mode of MBRL is when policies exploit the model in parts of the state-action space where the model is inaccurate. Thus, naïve application of MBRL to offline data
"Joint first authors. Correspondence: cong lungtattttt....oooo ball wall@rooots.oocooo ac.uc.uk

---

Published as a conference paper at ICLR 2022
Norm. Scale
1.0
Dynamics Error
Max Alealoric
Adroit
Max Pairwise Diff.
10"
Ensemble Std.
MujoCo
Ensemble Var.
0.4 0.5 0.6 0.7 0.8 0.9 1.0
25
50
100
P[Optimized > MOPO]
Time Steps in Model
(a) Uncertainty vs. True Errors
(b) Probability of Improvement
Figure 1: a) The variation of different uncertainty penalties against dynamics enror during model rollout of Hopper Medium-Expert. The canonical ensemble variance penalty most closely fits the true dynamics emrrr b) Tuning key hyperparameters (an approach we call Optimized) can lead to large gains over state-of-the-ar methods (MOPO) on the D4RL benchmark, as we show in this summary using 11 i1lia iable (Agarwal et al., 202 I).
can result in suboptimal performance. To prevent this, concurrent works (Yu et al., 2020; Kidambi al., 2020) have approached the problem by training a policy pessimistic MDP (P-MDP). The P-MDP lower bounds the true MDP, and discourages the policy from regions where there is large discrepancy between the true and learned dynamics; this often provides a theoretical guarantee of improvement over cloning the behavior policy that generated the offline data. This is made practically possible by adding a penalty correlated with the uncertainty in the dynamics model. However, while these recent successes are similar in principle, in practice they differ in series of design choices. First and foremost, they make use of different heuristics to measure model uncertainty, in some cases deviating from simpler metrics which are more consistent with the theory.
In this paper, we conduct a rigorous investigation into a series of these design choices. We begin by focusing the choice of uncertainty metric, comparing both recent state-of-the-art offline approaches (Kidambi et al., 2020; Yu et al., 2020; Rafailov et al., 2020) with additional metrics used in the online setting (Ball et al., 2020; Pan et al., 2020; Cowen- Riveers al., 2022). We also explore the interaction with a series of other hyperparameters, such as the number of models and imaginary rollout length. Interestingly, the relationship between these variables and model uncertainty varies significantly depending on the choice of uncertainty penalty. Furthermore, we compare these uncertainty heuristics under new evaluation protocols that, for the first time, capture the specific covariate shift induced by model-basee RL. This allows us to assess calibration to model exploitation in MBRL, observing that some existing penalties are surprisingly successful capturing the errors in predicted dynamics, as seen in Fig. la (see App. D for details). Then, using the insights gained from sections 4 and 5, we then achieve a 43% gain over a previously grid-searched method by using a single hyperpurameter value across all environments. We then jointly fine-tune our identified key variables using powerful Bayesian Optimization algorithm (Wan et al., 2021) and find the simpler uncertainty measures can provide state-of-the-as results in continuous control offline benchmarks, and that the chosen optimal hyperparameters continue to align with our analysis. Finally, we rigorously confirm the aggregate improvement of results using the rliable framework (Agarwal et al., 2021) in Fig. 1b, and show that the improvements over existing methods are significant (see App. H for details). This work intended benefit both researchers and practitioners offline RL. Our main findings include:
Longer horizon rollouts with larger penalties can improve existing methods. Contrary to common wisdom, conducting significantly longer rollouts inside the model, coupled with larger uncertainty penalties, typically improves performance.
Penalties that use canonical forms of uncertainty estimation achieve better correlation with OOD measures. The uncertainty estimation approach of Lakshminarayanan et al. (2017) often outperforms the penalty from state-of-the-an methods (Yu et al., 2020; Kidambi et al., 2020). We observe that the ensemble standard deviation is statistically strikingly similar to that used in Kidambi et al. (2020), but has improved correlation and scaling behavior.
Uncertainty more correlated with dynamics error than distribution shift. We find that suc- cessful penalties measure the discrepancy in dynamics, and can in fact assign high certainty to data far away from the offline data.
2 RELATED WORK
Two recent works concurrently demonstrated the effectiveness of model-based reinforcement learning (MBRL) in the offline setting. MOPO (Yu et al., 2020) follows the successful online RL algorithm MBPO (Janner et al., 2019) but trains inside a conservative MDP, penalizing the reward based on the maximum aleatoric uncertainty the ensemble members. MOReL. (Kidambi et al., 2020) achieves
2

---

Published as a conference paper at ICLR 2022
even stronger performance, penalizing the rewards by a penalty based on the maximum pair-wiss difference in ensemble member predictions. For pixel-based tasks, LOMPO (Rafailov et al., 2020) also proposed novel penalty, using the variance of ensemble log- ligelliiihoods.. Outside the offline setting, probabilistic dynamics models leveraging uncertainty have underpinned a series of successes (Chua al., 2018; 8;; Kurutach et al., 2018; Buckman et al., 2018; Pan et al., 2020; Pacchiano et al., 2021). Uncertainty can also be measured in MBRL without the use of neural networks (Deisenroth & Rasmussen, 201 1), although these methods tend be harder scale and thus lack widespread use.
Effective hyperparameter selection in RL has been shown to be crucial to the success of popular algorithms (Engstrom et al., 2020; Andrychowicz al., 2021). This becomes even more challenging in MBRL with additional supprparanccersddeeii-cchoie choiiis for the dynamics model. Recent work has shown that carefully optimizing these hyperparameters for online MBRI. can significantly improve performance, with the tuned agent breaking the MuloCo simulator (Zhang et al., 2021). In contrast, we foeus on the offline setting, and investigate parameters specifically related to uncertainty estimation. Previous work studied the impact of hyperparameters in offline RL (Paine et al., 2020), finding offline RL algorithms to be brittle to hyperparameter choices. However, unlike our work they only consider model-free approaches, whereas we specifically investigate model-based offline algorithms. Abbas et al. (2020) investigates the impact of different uncertainty estimation methods in online MBRL; they too find penalizing with combined aleatoric and epistemic uncertainty improves performance.
Our work also relates to the rich literature on deep ensembles (Lakshminarayanan et al., 2017), which train multiple deep neural networks with different initializations and dataset orderings, and generally outperform variational Bayes methods (Mackay, 1992; Blundell et al., 2015). Achieving effective calibration with neural networks notoriously difficult (Guo et al., 2017; Kuleshov et al., 2018; Maddox et al., 2019), and furthermore we require calibration under co-variate shift (Ovadia et al., 2019), as the policy learned in the model will likely deviate from the behavior policy that generated the offline data. Recent work has highlighted this issue in offline RL (Kumar et al., 2020; Yu et al., 2021) and has reported superior performance when eschewing model uncertainty entirely, and instead performing "conservative" Q-updates. However, it is unclear if this improvement is due to poor uncertainty calibration, implementation details, or a limitation in the pessimistic- MDDP formulation.
3 BACKGROUND
All of the methods we investigate in this paper model the environment as a Markov Decision Process (MDP), defined a tuple M AS P. Rt po: 7), where S and A denote the state and action spaces respectively, P(s'|s a) the transition dynamics, R(ss a) the reward function, po the initial state distribution, and 7 E (0, 1) the discount factor. The goal is to optimize a policy क (als) that maximizes the expected discounted return PxP 1E ? R(s,, a.)).
In offline RL, the policy is not deployed in the environment until test time. Instead, the algorithm only has access to a static dataset Dene {($ Gj r., s;+1)) collected by one or more behavioral policies ng. Following the notation in Yu et al. (2020) we refer to the distribution from which D Dens was sampled as the behavioral distribution. The canonical approach in offline MBRL is to train an ensemble of N probabilistic dynamics models (Nix & Weigend, 1994). These usually learn to predict both the next state 81+1 and reward from a state-action pair, and are trained on Den using supervised learning. Concretely, each of the N models output a Gaussian PA ( rr|st, (V az), Ej(st,, a:)) parameterized by Φ. The resulting learned dynamics model i and reward model R define model MDP M (S,A, P, R. po: 7). To train the policy, we use k-step rollouts inside M to generate trajectories (Sutton, 1991).
To prevent policy exploitation in model, pessimistic MDP (P-MDP) is constructed by lower bounding the true-cxpected return, 7 (T). using some error between the true and estimated models. For instance, in Yu et al. (2020), the authors show that lower bound on the return can be established by penalizing the reward by a measure that corresponds to estimated model error:
(M( Z (5,a)~p3
[R(s. 7|G (s.a)]]
where 07. represents transitioning under the dynamics model P and policy x. Several potential choices for |G Nr (s) are proposed, including an upper bound based on the total variation distance
3

---

Published as a conference paper at ICLR 2022
between the learned and true dynamics. However, for their practical algorithm, the authors elect to use a heuristic based on impressive empirical results. Concurrent to MOPO, MOReL (Kidambi et al., 2020) in theory constructs P-MDP by augmenting standard MDP with negative valued absorbing state that is transitioned to when total variation distance between true and learned dynamics is exceeded. They show that policy learned in this P-MDP exceeds simple behavior cloning. However, while dynamics-based total variation distance has desirable theoretical properties, the practical algorithm relies on another heuristic to approximate this quantity. This motivates the study of penalties used, as well as other under-used used candidates, and their overall effectiveness.
4 UNCERTAINTY PENALTY
The key idea underpinning recent success in offline MBRL is the introduction of a P-MDP, penalized by some uncertainty penalty. The theory dictates this should be some distance measure between the true and predicted dynamics. Of course, this cannot be truly estimated without access to an oracle, so a proxy for this quantity is constructed instead based on uncertainty heuristics. In this paper, we compare the following uncertainty heuristics, from recent works in both offline and online MBRL:
Max Aleatoric (Yu al., 2020): max;=1,....N .v I|2 (s, a)lle. which corresponds to the maximum aleatoric error, computed over the variance heads of the model ensemble.
P' (s, a)ll.. which corresponds to the pairwise maximum difference of the ensemble predictions.
Max Pairwise Diff (Kidambi et al., 2020): max,, (ils (s, a)
Variance) (Rafailov et al., 2020)): (logg (s'ls,, a), ,,...,,NJ NN), where s' is next state sampled from single ensemble member. We evaluate its log- regllikllhood under each ensemble member and take the variance.
LL Var LLog-Likelihen
LOO KL (Leave-One-Out KL Divergence (Pan et al., 2020): D (-ss, a) a) (-ss, a)], which corresponds to the KL divergence between the Gaussian parameterized by a randomly selected ensemble member, and the aggregated Gaussian of the remaining ensemble members.
(Lakshminarayanan et al., 2017): The variance is given as: (s, = N EN (EE) (s, a))² + (s)) (u*(s, a))² where jı" is the mean of the means (x = N Σ; a).. This corresponds to a combination of epistemic and aleatoric model uncertainty. This is surprisingly under-utilized in offline MBRL, and is a canonical method of uncertainty estimation used in the Bayesian inference literature (Ovadia et al., 2019; Filos et al., 2019; Scalia et al., 2020). We choose to evaluate both standard deviation (the square root of the above) and variance, as this will provide intuition about the importance of penalty distribution shape.
Ensemble Standard Deviation/Variant
These can all be computed using the output from an ensemble of probabilistic dynamics models (Lakshminarayanan et al., 2017), so we are able to compare them in controlled manner.
4.1 How WELL Do ENSEMBLE PENALTIES DETECT OUT OF DISTRIBUTION ERRORS?
We begin by assessing how well uncertainty penalties correlate with next state MSE (we justify the MSE under deterministic dynamics in App. A.2). This is crucial in penalizing the policy from visiting parts of the state-action space where the model is inaccurate, and therefore exploitable. Using D4RL (Fu et al., 2021a), train models on each dataset, then evaluate them other datasets from the same environment, but collected under different policies. These form our "Transfer" experiments as they directly measure the ability of uncertainty penalties detecting errors on unseen data, We compare the penalties against true MSE for a variety of settings in App. A.3, and summarize this in the "Transfer" column of Table 1. We measure Spearman rank (p) and Pearson bivariate (r) correlations, and justify their use in App. A. Full details of all experiments and hyperparameters are given App. G. We will analyze these results in detail in the next section, after introducing a novel protocol for assessing our penalties under the out-of out-of-distribution (OOD) data induced by model exploitation.
4.2 HOW DO THESE PERFORM DURING AN IMAGINARY ROLLOUT?
We additionally design experiment aimed at capturing the OOD data generated by the actual offline MBRL process, which we call our "True Model-Basedd experiments. First, we train a set of policies with 4 different starting seeds without penalty inside the model for 500 iterations. We
4

---

Published as a conference paper at ICLR 2022
Table I: Correlation statistics of penalties against true mean-sq. model error, averaged over all datasets (i.c., Random through to Expert) showing = SD over 12 seeds. The best in cach column bolded. The ensemble penalties generally perform best.
Transfer
True Model-Based
HalfCheetah
Hopper
HallCheetah
Hopper
Penalty
P
P
Max Aleatoric
0.78=0000
0.55+0.01
0.71=0001
0.41+00.01
0.58++0.01
0.42+0001
0.73=0.03
0.48++0.01
Max Pairwise Dift.
0.79..0.01
0.62.10.00
0.77.10.00
0,57100.00
0.58.10.01
0.52.10.01
0.75.00.22
0.55
Ens. Std.
0,82=0.01
0.64:0.01
0.79:10.00
0,56+00.0
0.61::0011
0.52:10.00
0.79=0002
0.55:0002
Ens, Var.
0.82=0.01
0.67=0.00
0.79=0000
0.59=0.00
0.60+0.01
0.49:0.01
0.77=0002
0.55+00.02
I.I. Var:
0.13::0.05
0.14::0.02
0.36:10.04
0.12::0.02
0.04
0.07:40.06
0.50=00.22
0.16::0.02
LOO KL.
0.03:0002
0.11
0.11:00.2
0.081:0.02
-0.02.0.12
0.06:10.06
0.22:0002
0.10
then measure the difference between the return predicted by the model over a rollout, and the true return the real environment. We define policy to be "exploitative" if the model significantly over-extimates the return compared to the true return. It is these exploitative policies that induce the types of extrapolation errors which cause MBRL methods to fail in the offline setting. It is therefore important that the penalty is able to accurately determine when the model is being exploited in this way. We use a subset of the most exploitative policies to generate trajectories in the model, and record the uncertainty predicted by each penalty every time step. To generate the True Model- -Bssed data, we then "replay" these trajectories in the true environment, loading the state and action taken in the model into the environment, and record the "true" next state according to the MuJoCo simulator (Todorov et al., 2012). True Model-Basee therefore calculates the MSE between the predicted and actual next states. Table summarizes the results from both the Transfer and True Based-Based experiments. Additional details are provided in App. D along with full correlation plots in App. A.3.
We are now in a position analyze the results in Table I. It is immediately obvious that the LOO KL and LL Var penalties have very weak correlation with MSE. We believe this is because LL Var relies on likelihood statistics, which are notoriously sensitive; it was designed for use with KL-regularized latent state space model which has well-behaved dynamics. Regarding LOO KL, we note that this penalty was designed for the online setting with significantly less data, and becomes quite uncorrelated in this larger data setting. This advocates penalties that are less reliant on distributional information concerning the separate Gaussians in the ensemble, as such penalties appear sensitive to the quality of their estimated distributions. We observe that Max Alcatoric, Max Pairwise Diff and the Ensemble penalties perform broadly similarly despite their different analytical forms; Ensemble measures do however exhibit noticeably higher rank correlation. We also observe significant performance loss between the Transfer and True Model-Based HalfCheetah settings, with the latter being relatively poor. This implies further work is needed develop penalties that can successfully detect the type of dynamics discrepancies that actually arise in offline MBRL. Finally, we observe that despite the similar rank correlations p. the bivariate correlations r can vary considerably, and observe from the scatter plots that Max Alcatoric exhibits low kurtosis, having large penalty values "bunched" at its extreme; we provide 3rd and 4th order moment statistics to facilitate shape comparisons in App. C.
5
KEY HYPERPARAMETERS IN OFFLINE MBRL
5.1 How MANY MODELS DO WE NEED?
At present the number of models used has not been discussed since MBPO, which trains seven probabilistic dynamics models of the same architecture (with different initializations), using only the top five models based on validation accuracy (referred to as "Elites" in the Evolutionary community, e.g. Mouret & Clune (2015)). The reason or justification for this is not discussed, but it has seemingly been adopted in the wider MBRL setting (Shen et al., 2020; Omer et al., 2021; Pineda et al., 2021). However, offline RI. is totally different paradigm, where it is possible that access to compute is less of a bottleneck and it may be preferable to use more models to extract the most signal possible from the static dataset. Inevitably, many of the ensemble penalties are dependent on the number of models; for example, it is easy to see that the Max Aleatoric value could scale poorly with more models.
How Does Penalty Distribution Change with Model Count? We now vary the number of models used in the calculation of the penalties and plot their respective distributions; an illustrative example is shown in Fig. 2 with full results in App. B. The scaling of penalties relying on max over sets is most affected with increasing the number of models due to admitting more extreme values, and we observe that the distribution shape of Max Aleatoric changes significantly as we admit more models, which
5

---

Published as a conference paper at ICLR 2022
Max Aleatoric
Max Pairw. Dift.
Ensemble Std.
Ensemble Var.
LL Var.
LOO KL
Number of Models
10
100 200 300
10
Panalty
Figure 2: Box Plots showing D4RL Medium transferred to Random. We show IQR limits and the median value denoted by the black vertical line. Green = HalfCheetah, Blue = Hopper. Max Aleatoric, Max Pairw. Diff. and LOO KL are unstable w.r.t. ensemble member count. In contrast, ensemble variance and std. far more stable.
we validate in App. C. This impacts the tuning of this hyperparameter, as we have to contend with changing distribution along with calibration quality (which we explore in the next section). Finally, observe that the Ensemble penalties change the least with differing model count, highlighting their ease of tuning; this is clearly a desirable property for designing such metrics going forward.
How does Penalty Performance Scale with Model Count? Empirically, there exists an optimal number of models to use in an ensemble for model-based RL (Kurutach et al., 2018; Matsushima et al., 2021). ). Up now, heuristics have been used select how many models we use for uncertainty estimation, despite it being possible use a different number of models for dynamics prediction and uncertainty estimation. For instance, MOPO, transitions are generated with five Elite models, but all seven models are used to calculate the penalty. MOReL, four models are used for both transitions and penalty prediction. Therefore, we wish to understand if there is merit to using a larger number of models for uncertainty estimation compared with next state prediction. We provide snapshot in Fig. 3. showing the aggregated results on the True Model- Based data in Hopper, with full results in App. B. We see there is no clear consensus, and that the optimal number of models highly dependent environment, the behavior data, and penalty type, with some settings showing improved calibration with model count and vice-versa. This clearly justifies treating the number of models as a hyperparameter that is important to tune, especially on transfer tasks. Interestingly, we observe that it is possible to simultaneously improve rank (p) correlation, but reduce bivariate (r) correlation, especially with the MOPO penalty. This again suggests that the number of models not only affects the quality of the estimation, but also its distributional shape.
Spearman:
Pearson:
02
02-
15
10
Number of Models
Number of Models
Max Aleatoric Max DIt:.
Ensemble Std. LL Var.
Ensemble Var.
LOO
Figure 3: changes with model number in Hopper across all
Plot how error and penalty correlation
datasets (i.e., Random through to Expert).
5.2 THE WEIGHT OF UNCERTAINTY X
To weight penalty against reward, MOPO introduces a parameter that trades off between the two terms. In their paper, the authors sweep over À {1, 5} for each environment. However, the optimal values may lie outside this region. Furthermore, we have shown this value will need to drastically change account for using different penalty even number of models.
5.3 THE ROLLOUT HORIZON h
The horizon h of the rollouts plays a crucial role in offline RL. Longer horizon rollouts increase the likelihood of errors in the transitions (we verify this intuition App. D), but conversely can improve performance when errors are properly managed (Janner et al., 2019; Pan et al., 2020). Furthermore, as highlighted in Fig. 1a, the model can generalize, and dynamics error does not necessarily increase with drift away from the offline dataset. Instead, observe spikes, and note it is possible to recover from these to valid states and transitions. is therefore imperative that a penalty captures these spikes over the course of entire model rollout with horizon h, and down-weights the reward accordingly.
Using this observation, we design a novel experiment that treats these spikes as "positive" labels, and normalize each penalty to [0. 1]. This converts the penalties into probabilistic classifier, and we evaluate how well they classify these events that occur increasingly under longer h. This is precisely the intuition behind the LOO KL and L.I Var approaches, whereby the penalty acts as

---

Published as a conference paper at ICLR 2022
Table 2: Performance of different penalties as OOD event detectors averaged over all datasets in Hopper and HalfChectah (i.c., Random through to Expert) showing + 1 SD over 12 seeds. AUC is "Area Under Curve" and AP is "Average Precision". The best (highest) in each column is highlighted in bold.
Percentiee
90th
95th
99th
Dynamics
Distribution
Dynamics
Distribution
By Dynamics
Distribation
Penaliy
AUC
AP
AUC
AP
AUC
AP
ACC
AP
ADC
AP
ADC
AP
Alcatoric
0:59:.0.11
0.501:003
0.76:001
0.351001
0.89:000
0.3510:0
0.80.0001
0.27:001
8.92:0.00
0.201004
0.8210.00
0:16:000
Max Painwiae DVT..
0.90:00.0
0.54::01
0.7::001
0.341001
0.11:000
0:40:002
081-0.01
0.28:004
8.33:0.00
0.281001
0.8920.02
0:13:002
Enecmble Sid.
0.90:0000
0.55x001
8.745001
0.38:0m1
0.11:000
0,000002
033-0.01
0.31:001
B.3:0000
0.254002
0,90:0.02
0.38:002
Ersemble Vaz.
0.90=0000
0.56=001
8.7850001
0.254001
0.1:0000
0,420002
022-0.01
0.295001
0.3300.0
0.27:001
0.89:0.02
0.16=0022
0:00=0005
0.33=000
0.74=0002
0.33+000
0.670012
0.21=000
0.6-0002
0.256002
0.73=0007
0,024001
0.81=000
0.11=000
LOO
0.59:0005
0.21=001
0.0k0000
0.246002
0.00-012
0.126000
0.70:0.11
0.3440022
0.05=0002
0.0100000
0.72=0.00
0.0000000
anomaly detector, removing detrimental transitions that lie above a threshold. This is the regime we focus on here, where binary detection is more important than correlation. Finally, assess two "True Model-Based" errors: the dynamics error as before, and introduce the distance from the offline distribution trained on, which we calculate as the 2-norm between state-action tuple and its nearest point in the offine data (Dadashi et al., 2021); these are called "Dynamics" and "Distribution" respectively. We provide precision-recatl curves and more details on this experiment in App. D and E.
We observe in Table 2 that the penalties are powerful at identifying dynamics discrepancy, but not as accurate at identifying when the world-model data is out-of-distribution with respect to the offline data. This is well-known phenomenon in deep neural networks and has been recently investigated in terms of feature collapse (Van Amersfoort et al., 2020), where latent representations of points far away in the input space get mapped close together. On the other hand, this shows an important distinction between the regularization induced by MBRL uncertainty and explicit state-action regularization in model-free approaches, such as Kumar et al. (2020); Wu et al. (2021). In the latter approaches, policies are penalized for taking out of distribution actions w.r.t the offline dataset, but this is not always the case with policies trained under MBRL and uncertainty penalties. The success of MBRL methods in RL. may therefore lie in the generation of state-action samples that are OOD but represent accurate dynamics, thus facilitating dynamics generalization in policies; recent work has shown that augmenting dynamics improves offline RL policy generalization (Ball et al., 2021). We believe future work understanding the implications of this property is vitally important.
TESTING THE LIMITS OF CURRENT APPROACHES
Given our previous analysis, in this section we seek to answer the following question: how well can existing methods perform with more optimal selection of the discussed hyperparameters? To answer this, we consider, first, naïve selection of one hyperparameter set across all environments (based on our previous analysis), and then more definitively, tuning the configuration for each individual D4RL MuJoCo environment using a state-of-the-as Bayesian Optimization (BO) algorithm (Wan et al., 2021). Our first set of results show that following our analysis can provide significant gains over existing baselines, whilst the second beats the current SOTA. Note, previous analysis focused on HalfCheetah and Hopper environments, so we extend our evaluation to Walker2d as a held-out test.
General applicability of our insights. Two of our main takcaways in Sections 4 and 5 are that should favor the canonical Ensemble penalties and longer rollout horizons. To test these claims, we design an experiment where we fix h = 20 for the horizon (c.f. h = 5 in MOPO at most), and only use Ensemble Std. as our penalty (see App. G for details). Since tuning the penalty weight À per environment is unrealistic, we employ an automatic penalty tuning scheme, analogous to the automatic entropy tuning used in Haarnoja al. (2018). We tune the penalty weight on-the-fly to constraint value of A = 1, meaning we use only a single hyperparameter across all environments. Full details on the penalty weight tuning are provided in App. I. With this approach, we get an average reward of 49.0 in the D4RL. locomotion test suite (Fu et al., 2021a), an increase of 43% over MOPO, which was grid-searched per environment.
This clearly shows that applying the findings from our analysis provides large performance gains generally. This result is the best we know for a single hyperparameter setup, and is particularly significant other offline MBRL algorithms tune many hyperparameters per environment. This 'zero-shot' hyperparameter restriction is also the most realistic application of offline RL to real world problems. If we were to allow ourselves to take the maximum over just 2 hyperparameter setups (the second setup being h = 10, A 0.5), we achieve an average reward of 57.8, an increase of 69 % over MOPO. We show the full results in Table 8 in App. I with improvement probabilities.
7

---

Published as a conference paper at ICLR 2022
Table 3: Best hyperparameters discovered by our BO algorithm, followed by a comparative evaluation the D4RL benchmark suite against other model- RL algorithms. We use D4RL v0 datasets. The raw score for Optimized! and MOPO was taken to be the average over the last 10 iterations of policy training, averaged over 4 seeds and showing = SD. Results of MOPO and COMBO were taken from the COMBO paper. Results for MOReL were taken from its paper. indicates p 0.05 for Welch's t-test for gain over MOPO, RRnn on codebase. AAuthors' reported scores. Authors used D4RL v2, which has more performant offline data.
Discovered Hy Hyperparameters
Enviromment
Optimized
MOPO
MOPO:
MOReL
COMBO
N
Penalty
random
10
6.64
Ensemble Std
31.7 1.5
32.7
35.4
25.6
38.8
HalfCheetah
mixed
11
0.96
37
Fasemble Var
58.0 +:2.5
$2.8 +1,1
53.1
40.2
55.1
medium
12
5,92
Ensemble Var
45.7 12266
46.5 1.0.7
42.3
42.1
54,2
med.-cxp.
4.56 7
Max Alcatoric
104.2 #
67.6 423.6
63.3
53.3
90.0
random
6
4.46
47
Ensemble Std
12.1 1.0.2
4.2 11.5
11.7
53.6
17.8
Hopper
mixed
7
5.90
Max Aleatoric
90.8 +11.1 *
66.7 +:218
67.5
93.6
73.1
medium
7
37.28
Ensemble Std
69.3 1:15.2
17.3
28.0
95.4
med. exp.
12.
39.08
43
Max Alcatoric
105.8 +11.2
24.9 +:5.5
23.7
108.7
111.1
random
10
0.21
12
Ensemble Var
21.7 1.0.1
13.6 1:1.
13.6
37.3
7.0
Walker2d
mixed
13
2.48
Ensemble
+
37.6 +:2066
49.8
medium
8
5,28
14
Ensemble Std
79.7 1.2.3
-0.1 =0.0
17.8
77.8
75.5
med.
12
0.99
37
Ensemble Std
97.1 +4.9
46.2 +27.0
44.6
95.6
96.1
Averuge Score
65.2 +54
34.2 19.8
36.7
64.4
64.1
Testing the limits of current approaches. Next, we wish to further validate that our earlier theoret- ical analysis can correspond to strong empirical performance gains by performing BO over the key hyperparameters. Details on the BO algorithm are listed in App. G. We define our search space over hyperparameterss most related to uncertainty quantification:
Penalty type (categorical): taking values over {Max Aleatoric, Max Pairwise Diff, LOO KL, LL Var, Ensemble Std, Ensemble Variance).
Penalty scale À (continuous): taking values over [1, 100).
h (integer): taking values over {1, 2..,, .03.
Models N (integer): taking values over .,,2,...11).....
Table 3 shows the optimal hyperparameters under BO. We note that Ensemble penalties are mainly selected, corroborating the findings in our analysis that these are most correlated with model error. We observe that Max Pairwise Diff is not chosen, likely because ensemble penalties are better correlated with true dynamics error, and are more stable under tuning since their scaling changes less with model number; we know that Max Pairwise Diff has very similar shape statistics to Ensemble Std. (App. C). Finally, we also observe these solutions have lower performance variance than MOPO.
The selection of Max Aleatoric is also explainable; we observe it displays significantly lower skew and kurtosis than all other metrics (App. C), while still maintaining strong rank correlation. We also found that in all Hopper experiments, Ensemble Var. never achieved high performance, despite the only difference with Ensemble Std. being its distributional shape. Interestingly, in HalfCheetah, the opposite is true, with Ensemble Var. delivering significant performance gains. This implies that distributional shape may play as important role as calibration, and advocates the learning of meta- meeaa-praaeeees that control for this. Finally, in Walker2d, the well-grounded ensemble penalties win in all cases. We note that values of the rollout horizon h and penalty weight À differ greatly from those chosen in the original MOPO paper, which chooses both from 5]. Notably, the Hopper and Walker2d environments can prefer much longer rollout length and higher penalty weight, even accounting for penalty magnitudes. Again this is backed up by our analysis; along a single rollout, dynamics errors do not necessarily accumulate, they simply become more likely to occur. Therefore, as long as we penalize errors appropriately, we can handle longer rollouts and, as a result, generate more on-policy data. The number of models used to compute the uncertainty estimates can also differ greatly from the standard 7. This again aligns with our findings that using more models for uncertainty estimation can be beneficial, but is dependent on environment, data, and penalty.
Table 3 also demonstrates how these unconventional hyperparameter choices fare against of the-art offline MBRL algorithms. We spent considerable effort ensuring that our implementation of MOPO matched the authors' results using the same hyperparameters. We note the two are very similar!, thereby allowing us to make a faithful comparison when modifying hyperparameters. Our approach, labeled "Optimized!", achieves statistically significant improvements over MOPO on
'There was a disparity in Walker2d-medium, but this was also noted in Ball et al. (2021)
8

---

Published as a conference paper at ICLR 2022
Environment
MOPO
Optimized!
CQL
Normmllzzd Score
183
cloned
+10.8
23.0 #:.22
39.2
dpdinized MORO (Oun))
pen
human
6.2 #7.8
19.0 +7.9
37.5
MOPO (Aurhero)
expert
15.1 +:9.7
50.6 =10.5
107.0
cloned
0.2 =0.1
5.2 1.5
2.1
hammer
human
0.2 =0.0
0.5 +0.8
4.4
Offine Epochs
200
expert
6.2 +8.4
23.3 +:4.1
86.7
Figure 4: MOPO performance on the Hopper medium-expee expert environment.
Table 4:
Comparative evaluation on the D4RL Adroit
vθ dataset against Model Free CQL
out of 12 environments, validating our prior analysis over the key design choices. As an additional bonus, and it is not the stated aim of this work, our approach achieves state-of-the-an performance on five HalfCheetah and Walker2d environments by a considerable margin. Further notable results include the Hopper mixed and Hopper medium-expert environments, in which we show we able to tune the MOPO-like method up to the performance of COMBO (Yu et al., 2021) and MOReL. The importance of good uncertainty estimation and hyperparameter selection shown visually in Fig. where we improve MOPO performance by over 5x whilst obtaining a stable solution.
As aforementioned, we found our policies are more stable than previous works and consequently do not need to cherry-pick high performing checkpoints Instead, we report the average performance over our final 10 colecy-impovvement iterations. should be noted that stability during training (Chan et al. 2020) is paramount for successful policy deployment in offline RL, and we should therefore prioritize hyperparameters that ensure this. We further confirm the reliability of our evaluation using the rriabbal le framework (Agarwal et al., 2021) in Fig. 1b, showing that the improvement over MOPO (with 95% bootstrap CIs shaded) is clear in both MuJoCo and Adroit.
Results on Adroit dexterous hand manipulation tasks. We present results in Table 4 on the Adroit Pen and Hammer environments which, as far as we are aware, have not previously been used in offline MBRL, and present very different challenges to the locomotion tasks. These tasks feature sparse rewards, real human demonstrations and narrow data distributions. We compare against the current state-of-the-ae model-free algorithm (CQL, Kumar al. (2020)) and find that offiine MBRL can learn useful policies in the Adroit domains, providing the best performance seen so far on the cammer-cloned setting. Best found penalties and hyperparameters are listed in App. and mirror the findings in the locomotion experiments. We believe issues with the world model not accurately capturing sparse rewards may account for any major performance difference. Our work is therefore an important step towards bridging the gap between model-based and model-free methods for sparse reward tasks, especially in the offline setting where exploration is not possible. We define MOPO to be the best performance with the Max Aleatoric penalty, searching X, h in {1,5}²
7 CONCLUSION
In this paper, we rigorously evaluated the impact of various key design choices on offline MBRL, comparing for the first time a number of different uncertainty penalties used the literature. By proposing novel evaluation protocols, we have also gained key insights into the nature of uncertainty in offline MBRL that we believe benefits the RL community. We demonstrated the impact of this analysis by significantly improving upon existing offline MBRL by using vastly different key hyperparameters, obtaining statistically significant performance improvements in almost all benchmarks.
Going forward, 2021b) to accurately assess agent performance without querying the environment. This would open the door for population-based training methods (Jaderberg et al., 2017; Parker-Holder al., 2020), which have shown great success in online MBRL (Zhang et al., 2021). Furthermore, throughout the paper we have highlighted potential areas of interest, from better understanding the generalization provided by world models, through to the development of meta- mnea--paaaeetes controlling penalty distribution shape. We also highlight key issues implementation in App. F as we strongly believe this is vital frontier for disentangling the effect of algorithmic innovations from code-level details. Finally, Offline MBRL so far has only focused on determinstic environments; the ensemble penalties we investigate support the modeling of stochastic dynamics, and the novel tools for analysis we develop here can be readily applied to such settings.
excited by developments in offline evaluation (Chen et al., 2021; Fu et al.,
II is unclear what procedure is used in some prior work (indeed issues have been raised about this).
9

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
The authors would like acknowledge Rishabh Agarwal for helpful feedback during the project. We would also like to thank the anonymous reviewers for their constructive feedback, which helped to improve the paper. Cong Lu is funded by the Engineering and Physical Sciences Research Council (EPSRC). Philip J. Ball is funded through the Willowgrove Studentship.
REFERENCES
Zaheer Abbas, Samuel Sokota, Erin Talvitie, and Martha White. Selective Dyna-style planning under limited model capacity. In ICML, pp. 1-10, 2020. URL http:://proceeeingg mml.. press/vi abbas20a html.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning the edge of the statistical precipice. In Advances in Neural Information Processing Systems, volume 34. 2021.
Marcin Andrychowicz, Anton Raichuk, Piotr Stanezyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What matters for on-policy deep actor-critic methods? a large-scale study. In International Conference Learning Representations, 2021. URL https: http///ppepeeeeeeeceee net fforum?? id=nIAxjsniDzg;
Arthur Argenson and Gabriel Dulac-Arnold. Model-based offline planning. In International Confer- ence on Learning Representations, 2021. URL
net/ weet/forure forum??n di
OMNB1G5xzd4.
Philip Ball, Jack Parker-Holder, Aldo Pacchiano, Krzysztof Choromanski, and Stephen Roberts. Ready policy one: World building through active learning. In Proceedings of the 37th International Conference on Machine Learning, ICML. 2020.
Philip Ball, Cong Lu, Jack Parker-Holder, and Stephen Roberts. Augmented world models facilitate zero-shot dynamics generalization from a single offline environment. In Marina Meila and Tong Zhang (eds.), Proceedings the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 619-629. PMLR, 18-24 Jul 2021. URL http:/ http://ppccceeieeeeeeeme mlr prwss wttpesss/w1///aal2aaaata
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning. volume 37 of Proceedings of Machine Learning Research, pp. 1613-1622, Lille, France, 07-09 Jul 2015. PMLR. URL htttt:://prcceeinngs. mlr. press/v37/ http://33///uudde11111 html.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample- efficient reinforcement learning with stochastic ensemble value expansion. In Advances in Neural Information Processing Systems. 07 2018
Stephanie C.Y. Chan, Samuel Fishman, Anoop Korattikara, John Canny, and Sergio Guadarrama. Mea- suring the reliability of reinforcement learning algorithms. International Conference on Learning Representations, 2020. URL https: /ttp:////pe///eeeee//e/ee net/ forum htt::/oormiddddds0.. SI
Yutian Chen, Liyuan Xu, Caglar Gulcehre, Tom Le Paine, Arthur Gretton, Nando de Freitas, and Arnadd Doucet. On instrumental variable regression for deep offline policy evaluation, 2021.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems 31, pp. 4755-4765. 2018.
Alexander Cowen-Riven, Daniel Palenicek, Vincent Moens, Mohammed Abdullah, Aivar Sootla, Jun Wang, and Haitham Bou Ammar. Samba: safe model-based & active reinforcement learning. Machine Learning, pp. 1-31, 01 2022. doi: 10. 1007//1009940-11000000
10

---

Published as a conference paper at ICLR 2022
Robert Dadashi, Leonard Hussenot, Matthieu Geist, and Olivier Pietquin. Primal Wasserstein imitation learning. International Conference on Learning Representations, 2021. URL httpa:
//oppenrevew.weee net mattt/ooooriii--T1t/33333332223
Marc Peter Deisenroth and Carl Edward Rasmussen. PILCO: A model-based and data-efficient approach policy search. In Proceedings of the 28th International Conference on International Conference on Machine Learning, pp. 465-472, 2011.
Gabriel Dulac-Aroold, Nir Levine, Daniel Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, pp. 1-50, 2021.
Logan Engstrom, Andrew llyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep RL: A case study on PPO and TRPO, In International Conference Learning Representations, 2020.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Leurning Research, 6(18):003-556, 2005. URL http: httt:::/mmmr.rrr html.
Angelos Filos, Sebastian Farquhar, Aidan N. Gomez, Tim G. J. Rudner, Zachary Kenton, Lewis Smith, Milad Alizadeh, Amoud de Kroon, and Yarin Gal. A systematic comparison of bayesian deep learning robustness in diabetic retinopathy tasks, 2019.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4(RL): Datasets for deep data-driven reinforcement learning, 2021a.
Justin Fu, Mohammad Norouzi, Ofrr Nachum, George Tucker, ziyu wang, Alexander Novikov, Mengjiao Yang. Michael R Zhang, Yutian Chen, Aviral Kumar, Cosmin Paduraru, Sergey Levine, and Thomas Paine Benchmarks for deep off-policy evaluation. In International Confer- on Learning Representations, 2021b. URL https: /ttp///ppp//eeee////e//e/eeeeeet net watt/foouuee forum?id- kWSeCEeHvFF.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pp. 1321-1330. PMLR, 06-11 Aug 2017. URL htttt::/prrceeedeie guol 7a html.
mlr. ptl...press/v0//
Tuomas Haaroja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Picter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. CoRR, abs/18122 112.05905, 2018
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojeiech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population based training of neural networks, 2017.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems. 2019.
Sham M Kakade. A natural policy gradient. In T. Dietterich, S. Becker, and
Z. Ghahramani (eds.), Advances in Neural Information Processing Systems, volume 14. neurips cc/papprr/201/filee 43GG
MIT Press, 2002, URL
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model- based offline reinforcement learning. In Advances in Neural Information Processing Systems. 2020.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q-learning, 2021.
11

---

Published as a conference paper at ICLR 2022
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learn- ing Research, pp. 2796-2804, PMLR, 10-15 Jul 2018. URL http::ttp::////eeee http:///rocceeeeee
press/ v80 / /kul eshovl Ba Bttml.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Qllearning for offline reinforcement learning. Advances in Neural Information Processing Systems. 2020.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. In International Conference on Learning Representations, 2018.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of the 31sl International Conference on Neural Information Processing Systems, NIPS' 17, pp. 6405-6416, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510866 11086094.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020.
David John Cameron Mackay. Bayesian Methods for Adaptive Models. PhD thesis, USA, 1992. UMI Order No. GAX92-32200.
Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. In H. Wallach, Larochelle, A. Beygelz- imer, F. d' A'Aché-Bus Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: http:://ccp...ee........oos..e neurips. Sc/ppppr/22099 9 221_ee/8822222222222222222222200
3fc2
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. In International Confer- ence on Learning Representations, 2021. URL https: /ttp//pppneeee.eeeeeee/ee
net / met/forum?id+
IhGN
Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites, 2015.
D. A. Nix and S. Weigend. Estimating the mean and variance of the target probability distribution. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN 194, volume 1, pp. 55-60 1, 1994.
Muhammad Omer, Rami Ahmed, Benjamin Rosman, and Sharief F. Babikir. Model predictive-actor critic reinforcement learning for dexterous manipulation. In 2020 International Conference on Computer, Control, Electrical, and Electronics Engineering (ICCCEEE), pp. 1-6, 2021. doi: 10.1 MA119/CCCCEFF49999 2021:9999992221.00 9429677.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evalu- ating predictive uncertainty under dataset shift.
Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: http:ppss//prooeeeingg...o wtwp::/ewweeeeeeeee.eeeeeeeeeeee neurips cc/ paper/201 9/ file/ l 8558c55488888 08 66662166661116 371 (111188677deebddPappppe
Aldo Pacchiano, Philip Ball, Jack Parker-Holder, Krzysztof Choromanski, and Stephen Roberts. Towards tractable optimism in model-based reinforcement learning. In Uncertainty in Artificial Intelligence. 2021.
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Çaglar Gülçehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. CoRR, abs/2007 .09055. 2020. URL https: : www.wwwwwww.wwaaaaaaeoo/re.r..ae
090555.
Feiyang Pan, Jia He, Dandan Tu, and Qing He. Trust the model when it is confident: Masked model--ased actor. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 10537-10546. Cur- ran Associates, Inc., 2020. URL https: http:///coccceeeeeeeeeeeoo neurips....p...... httpps.cc//peee httprrippsccpppppe///2222 3.w.133eeeeeee. ftl1:
12

---

Published as a conference paper at ICLR 2022
Jack Parker-Holder, Vu Nguyen, and Stephen J Roberts. Provably efficient online hyperparameter optimization with population based bandits. In Advances in Neural Information Processing Systems, volume 33. pp. 17211. Curran Associates, Inc., 2020.
Luis Pineda. Brandon Amos, Amy Zhang. Nathan O. Lambert, and Roberto Calandra. MBRL-
Lib: A modular library for model-based reinforcement learning. Arxiv, 2021. URL https:
///aaxii/// otg. abs 2tt..//21..............
Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offlie reinforcement learning from images with latent space models. In Offline Reinforcement Learning Workshop at Neural Information Processing Systems, 2020.
Binxin Ru, Ahsan Alvi, Vu Nguyen, Michael A Osborne, and Stephen Roberts. Bayesian optimisation over multiple continuous and categorical inputs. In International Conference on Machine Learning, pp. 8276-8285. PMLR, 2020.
Gabriele Scalia, Colin A Grambow, Barbara Pernici, Yi-Pei Li, and William H Green. Evaluating scalable uncertainty estimation methods for deep learning-based molecular property prediction. Journal of chemical information and modeling, 60(6):297--2717 2020.
Jian Shen, Han Zhao, Weinan Zhang, and Yong Yu. Model-based policy optimization with unsu- pervised model adaptation. H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 2882322334. Curran As- sociates, Inc., 2020. URL https: http://oooceeceneeeeeeeeeeecesee neurips wttpp/ppppp/////////e...// 1dc3a 8900 40ba 31 72 9b0ba 7 4b9 Pap.a333ppPpp........m
Richard S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. SIGART Bull., 244):160-163, July 1991. ISSN 0163-5719. doi: 10. (0.1445/123344112277 URL https:
//doi orgg10 11////1..111111 1145/ 122142222.4. 11.....111111224.12222
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-basee control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2033,
2012. doi: 10.1 200//ROS..002..63866666
Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using a single deep deterministic neural network. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 9690-9700, PMLR, 13-18 Jul 2020.
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Ma- chine Learning Research, 9(86):2579-2608 2008. URL http: /mwrr.///r//////////////////// org/ httppppprrs///////// vandermaaten088 html.
Xingehen Wan, Vu Nguyen, Huong Ha, Binxin Ru, Cong Lu, and Michael A. Osborne. Think global and act local: Bayesian optimisation over high-dimensional categorical and mixed search spaces. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 10663-10674. PMLR, 18-24 Jul 2021.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. In To Appear: The International Conference on Learning Representations (ICLR). 2021.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. In Advances in Neural Information Processing Systems, 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO: Conservative offline model-based policy optimization. In Advances in Neural Informa- tion Processing Systems, 2021.
Baohe Zhang, Raghu Rajan, Luis Pineda, Nathan Lambert, André Biedenkapp. Kurtland Chua, Frank Hutter, and Roberto Calandra. On the importance of hyperparameter optimization for model- based reinforcement learning. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, 2021.
13

---

Published as a conference paper at ICLR 2022
A CALIBRATION
A.1 CHOICE OF CALIBRATION METRICS
We consider both the Spearman rank (p) correlation and Pearson bivariate (r)) correlation. We believe that the former better represents the actual statistical power of the metric compared to the true distributional shift value, it robust to outliers and isn't impacted by distributional shape (i.e., skewness, kurtosis). After all, we do not know if some 'true' [G% (s, a)l is even linearly correlated with the MSE values that report, so naïvely comparing based on bivariate correlation may result in incorrect assessment of penalty efficacy. However, we do also include the Pearson bivariate correlation gain insight into how the penalty distribution shape changes with design choices. For instance, consider two metrics that have identical Spearman coefficients, but vastly different Pearson coefficients this implies they have significantly different distributional shapes whilst having the same statistical ranking power. The two correlation coefficients have the further advantage that they are unaffected by the scale of the uncertainty penalty, which can vary widely. Furthermore, algorithms such as MOPO and MOReL will often scale the penalty by some coefficient and thus the raw unscaled value is hard to interpret.
A.2 THE USE MSE AS THE GROUND TRUTH FOR DETERMINISTIC DYNAMICS
Following Yu et al. (2020), it is possible upper bound the expected performance и/M of a policy æ in the true MDP M under training in a world model MDP M as follows:
TM (M( 2 (s.a)rup
|R(s a) y|G (s, a)]
(2)
where :) is the reward function, PT represents transitioning under the world model dynamics P and policy ıT. The quantity |G it (s, a) can be upper-bounded by an integral probability metric (IPM):
G Gh (s, a)l S sup E. P [f()) P. =: (P(s) P(s,a)
(3)
JEF
where F is some set of functions mapping S to IR, and the dynamies under the true MDP M. noted in Yu et al. (2020), making assumptions over the functional form of F induces different distance measures. Restricting F to the set I-Lipschitz functions results an IPM with the following form:
(s, a)l S cWi(P(s, a), P(s.a)
(4)
which is the Wasseersteiin distance, where the constant is the Lipschitz constant of the value function Vž with respect norm || ||- Recalling that the environments we evaluate have deterministic dynamics (Todorov al., 2012), this means the dynamics distributions P and P Eq. are Dirac delta functions. In this case, the 1- I-Wasserstein distance simply reduces to the 2-norm between some 'true dynamics' T(s, a) and the 'estimated dynamics' f(s, a). This justifies the use of MSE between the oracle dynamics (as detailed in Sec. 4.1 and App. D.1) and the world model dynamics as the ground truth measure under which assess calibration.
14

---

Published as a conference paper at ICLR 2022
A.3 OFFLINE DATASET TRANSFER CALIBRATION
We present the full calibration scatter plots described in Sec. 4.2. Concretely, we plot penalty values on the y-axis, and ground-truth MSE the x-axis. First, we present the transfer performance of training sets onto offline datasets. Then, we present the results for all training datasets under the True Model-Basee experiment under the adversarial policies.
A.3.1 HALFCHEETAM
Aleatoric
Diff.
Ensemblee
le7
30
1.5
200
1.0
100
F.
7.44455
0.5
1:0.57
1. 0.343
0.003
7. 0.263
0.0
0-
20
20
40
20
MSE
(a) Random transferred to Expert
Max. Alesttoiic
Max Painwise Dift.
Ensemble Std.
Ensemble Var.
LL Var.
KL LDD
50
200
Purelty
0' 2.007
20
0.404
/.0.655
30
0.565
100
120
(b) Medium transferred to Expert
30
30
Penalty
20
20
50000
11551
10
3 719
0.0
F:0.208
- 034
10
10
15
10
5
10
15
30
15
10
15
MSE
Medium transferred to Random
Max Aleatoric
Max Pairwise Diff.
Ensern bie Std.
Ensemble Var.
LL Var.
KL LOO
30000
Panaty
20000
5.733
10000
0.0355
12-
# 0.005
6.3400
0/922
7 0.11
MSC
(d) Expert transferred to Random
Figure Scatter Plots showing HalfCheetah D4RL transfer tasks.
15

---

Published as a conference paper at ICLR 2022
A.3.2 HOPPER
Max Alestoric
Max Pairwis Diff
Ensemblee Std.
Ensemble Var
2012
Var.
LOD
1.5
2.0
150
1.0
202
01-3
2:2.503
0.0332
5-
0.3
0/028
0.372
0.0
60
MSE
(a) Random transferred to Expert
Max Aleatoiic
Max Psirwise
Ensemble Std.
Ensemble Vir.
LL Var.
KL.
1.20
403000
0.50
6.10
0.25
277773
2. 8.826
0.423
2:
8 9-483
0.107
0.30
0.00
0.2
0.2
5.4
0.0
3.2
0.2
0.4
5.0
(b) Medium transferred to Expert
Aleatoric
Palrwise
Enser Std.
Ensemble
108
LOO
15-
1.0
20
7: 0.518
6.7755
0.756
7.0.7MM
2.0.25
0.0
MSE
(c) Medium transferred to Random
Mixx Aleatzriic
Max Pa Paiiiis Diff.
Ensemb ble Std.
Ensemble Var.
LL Var
KL. LGO
0.6-
0.4
0.4
550
50
<t 533
0,509 0.3.55
2/ 576
3,051
0.444
5.0
(d) Expert transferred to Random
Figure 6: Scatter Plots showing Hopper D4RL transfer tasks.
16

---

Published as a conference paper at ICLR 2022
A.4
TRUE MODEL-BASED ERROR CALIBRATION
A.4.1 HALFCHEETAH
Max Aleatoric
Max Pairwise Dif..
Ensemble Std.
Ensemble Ver.
LL Var.
KL,000
2000
Panatty
2-790
7.0 10001
2,000
1600
1-
0.22
50
0.788
0.0
2.3
1.5
0.0
1.3
1.5
0.0
1.0
1.5
0.0
0.3
1.0
1.5
0.0
2.0
1.5
0.0
0.5
1.0
1.3
MSL
(a) Random
Max Aleatoric
Max Pairwise
Ensemble
Enserble Var.
LL Var.
KL LGO
30203
30-
1.-
20203
6-3-366
0.762
of CL0S1
10
10
(b) Mixed
Max Aleatoric
Max Pairwise DVf.
Ensemble Std.
Ensemble Var
1.0
30-
Heralty
20-
10
10
5.46
0.555
7.0.01
PI-8 DOR
0.0
10
10
10
MSE
(c) Medium
Max Alcatoric
Max Painwise Dirt.
Ensemble Std.
Ensemble Var.
LL Var.
KL LDO
12
3002
Pernlty
(5 /X.0.32)
1363
2,520
1.0
10
25
20
10
20
(d) Medium Expert
Aleatoric
Ensemble
LOO
0.8
7.5
40
Pensky
5,5
0.4
20
2.415
1.0415
0.2
2 0.141
55
0.244
0.01%
0,0
0.
J0
10
20
20
10
20
10
MSE
(e) Expert
Figure 7: Scatter Plots showing HalfCheetah D4RL. true model-based error calibration.
17

---

Published as a conference paper at ICLR 2022
A.4.2
HOPPER
Ensembble Std.
Ensemble
1:11
2.0
Ponaty
1.8
30
1.5
10-
600
1.0
400
7. 0.460
7: 7.05
15 6.3.40
66666
F. F.c.005
0.095
0.2
0-
0.0
MSE
(a) Random
Max Aleatoric
Max Pairwise Dift.
Encemble Std.
Ensemble Var.
Var.
KL LOD
6020.0
1.5
Penalty
1.0-
402020
0.5
0.003
6-2778
0.12
202030
2.0.212
5-332
0.75
5.25
0.00
0.50
5.50
(b) Mixed
ln7
Var.
LOO
5.20
1.0
Penalty
50
0.5
7:0.40
F.
6.000
0:5.5
7.0.588
2:
7-0.201
J'0.100
0.0
0-
0.0
0.00
0.0
0.0
0.2
0.4
0.0
3.2
0.4
3.5
0.2
2.4
0.0
0.2
0.4
5.0
0.2
0.4
0.0
0.2
0.4
MSE
(c) Medium
Max Aleatoric
Max Patrwise Dift.
Ersemble Std.
Ensemble Var.
LL Var.
KL LO0
800000
500000
02
402000
0.505
0.1
125
0.500
2117
0.5.7
2.380
1:2:587
MSE
(d) Medium Expert
Max
Max Painwise
Enserble Std.
Ensemble
0.15
0.100
80
Penalty
0.10
0.075
100000
60
0.030
45-
0072
8:754
0.05
0.936
0.025
8.0.33 015
S0000
0.455
22
0.265
0.0
0.000
9.2
0.0
0.2
0.0
02
(e) Expert
Figure 8: Scatter Plots showing Hopper D4RL true model-based error calibration.
A.5 ADDITIONAL LOG PROBABILITY CORRELATION ANALYSIS
Table 5 shows correlation between reward penalties and the negative log-likelihood of the true data under the model in the Transfer experiments.
Table 5:
Correlation statistics of penalties against model negative log- likelihhood of the true data, averaged over all datasets (i.e., Random through Expert) showing + SD over seeds. The best in each column bolded.
Transfer
HalrCheetah
Hopper
Penalty
P
p
Max Aleatoric
0.87:10.00
0.82
0.8111001
0.57
Max Pairwise Diff.
0.79:10.01
0.62+0.00
0.79:1001
0.51=0.00
Ens. Std.
0.93++0.00
0.86:10011
0.89=0.01
0.61=0001
Ens. Var.
0.90:1001
0.74::0.1
0.82:1.00
0.59=0.00
LL Var.
0.04++0.07
0.07++0.03
0.25++0.33
0. 0.10==0.1
LOO KL.
-0.04-0.06
-0.02.1..04
0.08.10.03
0.00
18

---

Published as a conference paper at ICLR 2022
B FULL RESULTS INCREASING MODELS
B.1 PENALTY DISTRIBUTION
In this section we provide the full set of results showing the impact of increasing model count on the distribution quantile statistics as introduced in Sec. 5.1. We show inter-quartile range and the median (the latter being denoted by black vertical line) of each penalty as function of increasing model number across all training domains and test settings. First, we present the transfer performance of all training sets onto all offline datasets. Then, we present the results for all training datasets under True Model -asedd experiment under the adversarial policies.
B.1.1
OFFLINE DATASET TRANSFER DISTRIBUTION
Max Aleatoric
Max Painwise Diff.
Ensemble Std.
Ensemble Var.
LL
LOO
Number Models
10
20
3.0
3.5
10
15
0.75
1.00
1.25
5eg
25
50
Penalty
(a) Random transferred to Expert
Max Aleatoric
Max Pairwise Diff.
Ensemble Std.
Ensemble Var.
LL Var.
KL LOO
Number of Models
3
5-
20
100
167
Penalty
(b) Medium transferred to Expert
Max Aleatoric
Max Pairwise Diff.
Ensemble Std.
Ensemble Var.
LL Var.
KL LOO
Number of Models
10
15
20
Penalty
(c) Medium transferred to Random
Max Aleatoric
Max Pairwise Diff.
Ensemble Std.
Ensemble Var.
LL
LOO
Number of Models
10
15
20
15
1.25
100
200
300
10
Penalty
(d) Expert transferred to Random
Figure 9: Box Plots showing HalfCheetah D4RL transfer tasks.
19

---

Published as a conference paper at ICLR 2022
Aleatoric
Painwise Diff.
Ensemble
Ensemble
Number of Models
10-
15
20
1.40
10
5.0
100
Penalty
(a) Random transferred to Expert
Max Aleatoric
Max Pairwise Diff.
Ensemble Std.
Ensemble Var.
LL Var.
KL LOD
Number of Models
3
5-
10-
20
0.2
0.4
0.025 0.050 0.075 0.00
0.02
1000
2000
10
20
30
Penalty
(b) Medium transferred to Expert
Max Aleatoric
Max Pairwise Diff.
Ensemble Std.
Ensemble Var.
LL Var.
KL LOO
Number of Models
3
15
20
1.0
1.5
5
10
0.5
1.0
200000 400000
Penalty
(c) Medium transferred to Random
Aleatoric
Max Diff.
Ensemble Std.
Ensemble Var.
Number of Models
10
15
20
Penalty
(d) Expert transferred to Random
Figure 10: Box Plots showing Hopper D4RL transfer tasks.
20

---

Published as a conference paper at ICLR 2022
B.1.2
TRUE MODEL-BASED ERROR DISTRIBUTION
Max Aleatoric
Max Pairwise Difr.
Ensemble Std.
Ensemble Var.
KL LOD
Number of Models
10
20
10
0.2
0.4
0.6 Penalty
0.5
1.0
1000
10
20
(a) Random
Max Aleatoric
Max Pairwise Diff.
Ensemble Std.
Ensemble Var.
LL Var.
KL LOO
Number of Models
3
7-
10
15
20
0.25
0.50
0.75
0.5
1.0
1.5
500 1000 1500
10
20
Penalty
(h) Mixed
Aleatoric
Max Painwise
Ensemble Std.
Ensemble Var.
Number of Modell
10
15
20
Penalty
(c) Medium
Max Aleatoric
Max Pairwise Diff.
Ensemble Std.
Ensemble Var.
LL
KL
Number of Models
10
15
20
2
0.1
0.2
0.0
0.1
0.2
100
200
Penalty
(d) Medium Expert
Max Aleatoric
Max Pairwise Diff.
Ensemble Std.
Ensemble Var.
LL Var.
KL LOO
Number Models
3
7
10
15
20
0.5
1.0
1.5
2
0.05
0.10
0.15
0.0250.0500.075
250 500
750
10
20
Penalty
(e) Expert
Figure 11: Boxplots showing HalfCheetah D4RL true model-based error penalty distributions.
21

---

Published as a conference paper at ICLR 2022
Aleatoric
Painwise Diff.
Ensemble
Ensemble Var.
Number of Models
5
0.25 0.75
0.05
0.10
0.15 0.00
250000 500000
25
Penalty
(a) Random
Max Aleatoric
Max Pairwise Diff.
Ensemble Std.
Ensemble Var.
LL Var.
LOO
Number of Models
3
10
20
0.2
0.4
0.5
1.0
0.02
0.04
0.000
0.005 0.010
100 200
300
10
Penalty
(b) Mixed
Max Aleatoric
Max Pairwise Dift.
Ensemble Std.
Ensemble Var.
LL Var.
KL LOO
Number of Models
3
15
20
0.1
0.2
0.3
0.25
0.50
0.75
0.01
0.02
0.03
00.50
250
500
10
20
Penalty
(c) Medium
Aleatoric
Pairwise
Std.
Var.
Var.
Number of Models
10
15
00000000025000050
Penalty
(d) Medium Expert
Max Aleatoric
Max Pairwise Diff.
Ensemble Std.
Ensemble Var.
Var.
LOO
Number of Models
5
10
15
20
0.1
0.2
0.25
0.50
0.01
0.02
0.03
0.000
0.002
0.004
250
500
750
10
20
Penalty
(c) Expert
Figure 12: Boxplots showing Hopper D4RL true model-based error penalty distributions.
22

---

Published as a conference paper at ICLR 2022
B.2 PENALTY PERFORMANCE
In this section, we provide the full set of results showing the impact of increasing model count on the correlation statistics of each penalty, as described in Sec. 5.1. We show the Spearman and Pearson correlation between penalty and ground truth MSE for all training datasets. First, we present the transfer performance of all training sets onto all offline datasets. Then, we present the results for all training datasets under the True Model-Basee experiment under the adversarial policies.
B.2.1
HALFCHEETAH D4RL: TRANSFER
Random Random
Random Mixed
Random Med.
Random Med.-Exp.
Random Exp.
0.75
0.50
0.25
0.00
Mixed Random
Mixed Mixed
Mixed Med.
Mixed Med.-Exp.
Mixed Exp.
0.75
0.50
0.25
0.00
Med. Random
Med. Mixed
Med. Med.
Med. Med.-Exp
Med. Exp.
0.75
Spearman:
0.50
0.25
0.00
Med.-Exp. Random
Med. Eed.-Exp.-v Mixed
Med.-Exp. Med.
Med.-Exp. Med.-Exp
Med.-Exp. Exp.
0.75
0.50
0.25
0.00
Exp. Randon
Exp. Mixed
Exp. Med.
Exp. Med.Exp.
Exp. Exp.
0.75
0.50
0.25
0.00
Number Models
Aleatoric
Painwise Dift.
Ensamble Ensemble
Figure 13: HalfCheetah Spearman Statisties
23

---

Published as a conference paper at ICLR 2022
Random
Random
Random
Random
Random Exp.
0.75
0.25
Mixed
Mixed Exp.
0.75
0.25
Random
Mixed
Med.
Med. Med.-Exp.
Med. Exp.
0.75
Pearson:
0.25
Random
Med.-Exp. Mixed
Med.-Exp. Med.
Med.-Exp. Med.-Exp
Med.-Exp. Exp.
0.75
0.25
Random
Mixed
Med.
Med.-Exp
Exp. Exp.
0.75
0.50
0.25
10
20
15
10
20
15
10
15
20
Number of Models
Dift.
Ensemble
Figure 14: HalfCheetah Pearson Statistics
24

---

Published as a conference paper at ICLR 2022
B.2.2 HOPPER D4RL: TRANSFER
Random Random
Random Mixed
Random Med.
Random Med.-Exp.
Random Exp.
0.75
0.50
0.25
0.00
Mixed Random
Mixed Mixed
Mixed Med.
Mixed Med.-Exp.
Mixed
0.75
0.50
0.25
0.00
Med. Random
Med. Mixed
Med. Med.
Med. Med.-Exp.
Med. Exp.
0.75
Spearman:
0.50
0.25
0.00
Med.-Exp, Random
Med. Exp. Mixed
Med.-Exp. Med.
Med.-Exp Med.-Exp
Med.-Exp. Exp.
0.75
0.50
0.25
0.00
Exp. Randonm
Exp. Mixed
Exp. Med.
Exp. Med.Exp.
Exp. Exp.
0.75
0.50
0.25
0.00
Number Models
Aleatoric
Figure 15: Hopper Spearman Statistics
25

---

Published as a conference paper at ICLR 2022
Random
Random
Random
Random
Random Exp.
0.75
0.25
Mixed
Mixed Exp.
0.75
0.25
Random
Mixed
Med.
Med. Med.-Exp.
Med. Exp.
0.75
Pearson:
0.25
Random
Med.-Exp. Mixed
Med.-Exp. Med.
Med.-Exp. Med.-Exp
Med.-Exp. Exp.
0.75
0.25
Random
Mixed
Med.
Med.-Exp
Exp. Exp.
0.75
0.50
0.25
10
20
15
10
20
15
10
15
20
Number of Models
Dift.
Ensemble
Figure 16: Hopper Pearson Statistics
26

---

Published as a conference paper at ICLR 2022
B.2.3
HALFCHEETAH D4RL: TRUE MODEL-BASER ERROR
Random
Mixed
Med.
Med.-Exp.
Exp.
Spearman:
0.25
0.00
-0.25
10
15
10
20
30
15
20
10
35
20
10
15
20
Number of Models
Max Alcatoric
Painwise
Ensemble Std.
Ensemble Var.
L00
Figure 17: HalfCheetah Spearman Statistics
Random
Mixed
Med.
Med. Mxp.EEx.
Exp.
0.73
Pearson:
0.00
10
15
20
10
15
5
10
15
20
20
10
15
20
Number Models
Max Alcatoric
Max Pairwise Dift
Ensemble Std.
Ensemble Var.
11. Var.
KL LOO
Figure 18: HalfCheetah Pearson Statistics
B.2.4 HOPPER D4RL: TRUE MODEL-BASED ERROR
Random
Miked
Med.
Med.-Exp
Exp.
Spearman:
0.6
0.4
10
15
20
20
15
20
10
15
20
10
15
20
Max Aleatoric
Max Pairwise Dift
Ensemble Std.
Ensemble Var.
L1. Vic.
KL LD0
Figure Hopper Spearman Statistics
Random
Mixed
Med.
Med.-Exp.
Exp.
Pearson:
0.4
10
15
20
Number Models
Max Aleatoric
Max Pairwise DFf
Ensemble Std.
Ensemble Var.
11 Var.
KL. LDO
Figure 20: Hopper Pearson Statisties
27

---

Published as a conference paper at ICLR 2022
B.2.5 ALL AGGREGATED
HalfCheetah
Hopper
0.8
Spearn Spearmant:
Pearson:
Spearmam:
0.2
0.4
0.2
10
10
15
Models
Number
Number
Ensemble
Ensemble
LOO
Figure 21: Aggregated True Model-Basse correlation statistics over all datasets (i.e., Random through to
Expert); Left: HalfCheetah; Right: Hopper
C SKEWNESS AND KURTOSIS COMPARISONS
C.I SKEWNESS AND KURTOSIS OVERALL
Here we present the 3rd and 4th order statistics (skew and kurtosis respectively) of each penalty, illustrating that even with identical model counts, the shape statistics between penalties are vastly different.
Table 6: Skew (71) and Kurtosis (72) statistics all experiments averaged over datasets (i.c., Random through to Expert) using the MOPO Default of 7 models.
Transfer
True Model-
HalfCheetah
Hopper
HalfCheetah
Hopper
Penalty
7/2
7/2
7/2
Max Aleatoric
-0.010
0.580
0.689
1.377
0.671
0.920
1.873
2.864
Max Pairwise Diff.
0.919
0.957
1.967
4.578
1.661
3.081
2.571
7.465
Ensemble Std.
0.794
0.806
2.136
6.560
1.656
3.178
2.739
9.061
Ensemble Var.
1.823
4.830
3,436
15.983
2.612
8.800
4.517
25.380
I.I. Var.
6.893
114.843
10,920
180.716
5.100
37,865
14.415
251,705
1.00 KI.
1.778
5.729
3.729
29.606
1.840
4.600
4.008
28.089
C.2 SKEW AND KURTOSIS SCALING WITH MODEL COUNT
We omit LL Var. and LOO KL due to the fact that their changes were so significant as to obfuscate the changes of the more performant penalties.
We choose 7 models, as in Table 6, to act as our 'baseline' (following the default MOPO setting). and we measure the change in the skew and kurtosis relative to this, hence 7 models always has 0% change in our graphs. For brevity. in the transfer experiments, we average over all 'transferred to' environments, e.g., Random, Medium, etc.; the graph title refers to the data that the model was trained on.
Again, we observe the environment and setting dependency of these metrics, sometimes having increasing skewness and kurtosis with model count, and other times decreasing. This further justifies using ranking metric to compare penalties, as the overall penalty shape can vary hugely and unpredictably w.r.t. co-dependent hyperparameters. We do observe however in the True Model-Bused experiments that ensemble standard deviation appears to be most robust to scaling with models. We also observe that the Max Aleatoric penalty change shape significantly w.r.t. model count, and no penalties are fully immune to this. This further advocates the use of shape for changing distribution properties when adjusting the number of models as a hyperparameter, as well as selecting penalties that are relatively invariant to model count to make tuning easier.
to control
28

---

Published as a conference paper at ICLR 2022
Random
Mixed
Med.
Mod. Exp.
Exp.
Skewneess: % Change
40
50
-20
-100
Number of Models
Skewness Scaling
Exp.
Kurbosls:: Change
300
200
100
-100
20
-60
10
20
10
20
Number of Models
Max Aleatoric
Max Pairwise Dift.
Ensemble Std.
Ensembbee Vir:
(b) Kurtosis Scaling
Figure 22: HalfCheetah Transfer.
Skewness Change
Random
Mixed
Med.
Med.-Exp.
Exp.
0
50
-20
-100
-1.00
-40
20
Number of Models
Skewness Scaling
Mixed
Kürtosis: Change
300
200
100
-100
-20
0
10
20
10
20
30
20
Number of Models
Max Alcatoric
Max Pairwise Dift.
Ensemble Std.
Ensemble Var:
(b) Kurtosis Scaling
Figure 23: Hopper Transfer.
29

---

Published as a conference paper at ICLR 2022
Random
Mixed
Midd.
Med. Exp.
Exp.
Skewnness; % Change
200
20
100
50
20
-20
-200
10
Number of Models
(a) Skewness Scaling
Exp.
Exp.
Kurbosis: 12 Change
100
400
20
-20
0-
-40
-20-
-60
-200
440
10
20
20
10
10
10
20
Number of Models
Max Alnatoric
Max Pairwise Dift.
Ensemble Std.
Ensemble Vir.
(b) Kurtosis Scaling
Figure 24: HalfCheetah True Model-Based.
Random
Mixed
Med.
Med.-Exp.
5kowness: in Change
20
20
10
-20
-10
-20
10
20
10
20
10
20
10
20
10
20
Max Aleatoric
Nax Pairwise Dift.
Ensemble Std.
Ensemble Var.
(a) Skewness Scaling
Random
Mixed
Med.
Med. Exp.
Exp.
Kurtosis: % Change
-2
50
-40
-25
-100
-50
Number of Models
Max Aleatoric
Nax Pairwise Dift.
Ensomble Std.
Ensemble Var
(b) Kurtosis Scaling
Figure 25: Hopper True Model-Based.
30

---

Published as a conference paper at ICLR 2022
D FURTHER DETAILS ON TRUE MODEL-BASED EXPERIMENTS
D.1 METHIODOLOGICAL DETAILS
We leverage the MuJoCo (Todorov al., 2012) simulator provide with ground truth dynamics that we can use to compare against our model predictions and penalties. This is done by providing the state and action inputs given to the model the simulator through the set_st state method in the simulator API. It must be noted that this method also requires addition "displacement" value which is not modelled by the world models (nor it provided in the D4RL data), however we found in practice this did not affect the dynamics predicted by the simulator, and simply setting this to 0 was sufficient to generate ground truth predictions.
This makes it possible provide the simulator the hallucinated model states, and provide true proxy to the dynamics discrepancy. We note that since the states are 'hallucinated" by the model, might be the case that they may not be admissible under the true environment, but in reality the simulator was able to process almost any combination of state and action, barring settings that featured anomalously large magnitudes. To handle such cases, we found it necessary clip the model states the range [-10, 10].
In order assess the permissibility of states, well measure the accuracy of the penalties as OOD input detectors, we provide an alternative distance measure based on the distance away from the training set. We use this measure for our analysis in Section 5.3, and is calculated as the distance from the offline training dataset, which we define to be the 2-norm between given state-action tuple and nearest point in the offline data, a similar metric to those used in recent works on imitation learning (Dadashi et al., 2021). We describe this quantity henceforth as 'Distribution Error'.
D.2 ON THE NATURE OF OOD DATA ALONG HALLUCINATED TRAJECTORIES
Here discuss the nature of OOD data along single hallucinated trajectory (in the model) in offline MBRL, analyzing the inductive bias that some 'error' increases with increasing rollout length in the model. We find that there is merit to this assumption, and show this in Fig. 26 for all HalfCheetah and Hopper environments in D4RL.. Here, we plot the median error at each time-step across 30, 000 aggregated trajectories in the model. Note that for all plots, we re-normalize all penalties by subtracting their mean and dividing by their standard deviation to facilitate comparison; this normalization was also applied in the analysis performed in Fig. la. Concretely, each time step corresponds to the normalized median value of 30, 000 data-points.
Random
Mixed
Med.
Med. Exp.
Exp.
HarChuutah
Normalized Medians
Hupper
50
100
50
100
0
50
100
50
100
50
100
Steps Model
Dynamics
Distribution
Figure 26: Median True Model-Basee Errors as a function of rollout timestep
We observe indeed that both median dynamics and distribution errors increase with increasing time step in the model. The only real exception is HalfCheetah Bediimm-Expers which we believe to be due to trained policy not being able to successfully exploit this environment.
The above analysis captures overall trends in the error over a large number trajectories. However, the way errors manifest during individual rollout is not so straightforward. To illustrate this,
31

---

Published as a conference paper at ICLR 2022
observe Fig. 27, where we plot random subset of 5 individual rollouts from the Hopper Medium- Expert data we generated.
Normalized Medians
10.0
5.0
2.5
0.0
Time Steps in Model
Dynamics
Distribution
Figure 27: Several Individual Ground Truth Rollouts in Hopper Medium Medium-Expert
We observe that errors along any single trajectory tend manifest as 'spikes', and that it is entirely possible to recover from these, returning to either admissible dynamics, or parts of the state-action space that have been seen in the data. This speaks to the nature of how we ought to penalize policies for accessing regions of inaccuracy/uncertaintyy and may justify hybrid MOPO/MOReL. approach, whereby we penalize individual transitions along a trajectory, but do not stop rollouts early. Indeed, this is similar to the approach taken in M2AC (non-stop), albeit they choose to 'mask' uncertain transitions, not penalize them. We leave the design of such an algorithm to future work.
Finally, we address the issue of comparing OOD dynamics and inputs. As already observed in Fig. 27, these two errors are not necessarily always the same, and oftentimes is possible that one quantity is large, whilst the other is small. We revisit Fig. Ia to explore this, now also plotting the Distribution Error in Fig. 28.
Noralloe ritesurr
6
Dynamics Distribution Ener Max Aleatoric
2-
Ense mole Std.
Ensemblc Var.
Steps
Figure 28: Comparing OOD dynamics and inputs on Hopper Medium- Eeaiuum-Experttt trajectory
We first speak to the inset annotated '1'. Here we observe that the transitions generated in fact closely resemble the data that our model was trained on, however the predicted dynamics are incorrect, and cause an aforementioned 'spike". This is the opposite of what is observed in inset annotated '2'; where we actually predict accurate dynamics, however the resultant state-action tuples do not closely resemble the data that our model was trained on. We generally observe that regions high Distribution Error tend to be preceded by 'spikes' pertaining to high Dynamics Error, and this present an exciting avenue for future work understanding how these quantities are related.
D.3 ON THE DIVERSITY OF EXPLOITATIVE POLICIES
is possible that training policies purely to exploit the world models may result generating state- action tuples that are low in diversity, the policy could discover "pockets" in the model that provide consistently high return. To prevent this, we train multiple policies inside the model from different seeds, with the aim of inducing different modes of exploitation. To validate this induces diverse trajectories in the world models, we visualize the state-action manifold using a t-SNE projection (van der Maaten & Hinton, 2008) of: 1) the D4RL Hopper Mixed- -eelly (which contains diverse samples); 2) the imagined rollouts inside the model from the exploitative policies Fig. 29. The policies were trained exploit model that itself trained the Hopper Mixed- Replay data.
32

---

Published as a conference paper at ICLR 2022
t-SNE Plot of State-Action Tuples
Medium-Replay
Imaginee
-25
-50
-15
Figure I-SNE projection of 10,000 Hopper D4RL Medium-Replay and 10,000 exploitative Imagined policy state-action tuples
We observe that the induced policies inside the model displays some overlap with the D4RL data, but also resides in parts of the manifold where there is little coverage from the D4RL data, likely representing regions of exploitation. Importantly, the exploitative WM trajectories display strong state-action tuple diversity, comparable to that of the offline data it was trained on.
E USING METRICS AS OOD EVENT DETECTORS
E.1 MEASURING STATISTICS
1.0
0.6
0.4
95th
0.25
0.75
20
60
100
Time Steas
Ground Fuuhh
Aleatoric
Max
Ensemmtee
Ensambte
Figure 30: Hopper Medium-Expert True Model-Based Experiments: Left: Precision v.s. Recall against Ground Truth; Middle: Higher Performing Penalties v.s. Ground Truth MSE in Imagined Rollout; Right: Lower Performing Penalties Ground Truth MSE in Imagined Rollout
As noted previously, different penalties have varying scales and distribution profiles, so we need way of standardizing the method of assessment. Using our observation that errors manifest as 'spikes' during rollout, we propose treating each penalty as a classifier. Concretely, our test set consists of the ground truth data labeled by whether not they exceed certain percentile at particular time step. Each penalty may be then be treated as a 'classifier' by normalizing its range to lie in [0, 1]. We then use standard classification quality measures. such AUC, determine the effectiveness of these penalties at capturing these spikes, whilst sidestepping the issue of the different distributional profiles identified previously.
Fig. 30 shows how our proposed method may be used to compare the effectiveness of each metric at capturing OOD events. the figure. plot single rollout the model, and the resultant ground truth MSE between the predicted next state and the true next state in black. We then superimpose the 90th, 95th and 99th percentile MSEs across the entire imagined trajectories onto the figure in gray dashed lines. To construct our OOD labels, we label any point below the percentile line as being "False", and any point above that line as being 'True', Finally, we normalize the uncertainty metrics as previously described into values in the range (0, 1]. allowing to construct precision-recall graphs and calculate classifier statistics.
33

---

Published as a conference paper at ICLR 2022
E.2
PRECISION RECALL CURVES
In this section we present the Precision-Recall curves described in App. E.
Happer
Hopper Hopper Hopper Med.-Exp.
Happer Exp.
1.00
0.75
0.25
Precision
0.00
ICheethh Random
HallCheetah Mixed
Cheecah Med.
HalfCheetah Med.EEpp
HalfCheetah Exp.
1.00
0.75
0.25
0.00
1.0
0.0
0.0
0.0
0.5
1.0
1.0
Recall
Ensemble Std.
Ensemble
(a) 90th Percentile
Happer
Hopper
Hopper
Hopper Med.-Exp.
Happer Exp.
1.00
0.50
0.25
Precicion
0.00
HalfCheetah Random
HaltCheetah Mixed
HaleChh Med.
HalfCheetah Med-Exp.
HaltCheetah Exp.
1.00
0.75
0.25
0.00
1.0
0.0
0.5
1.0
Recall
(b) Percentile
Hopper Random
Hopper
Hopper
Hopper Med.-Exp.
Hopper Exp.
1.00
0.50
0.25
Precision
0.00
Hal [Cheetah Random
HallCheetah Mixed
HallCheetah Med.
HallCheetah Med. Mxp.EExp
HalfCheetah Exp.
1.00
0.75
0.25
0.00
0.0
0.5
1.0
Recall
(c) Percentile
Figure 31: Precision Recall curves ground truth data.
34

---

Published as a conference paper at ICLR 2022
F IMPLEMENTATION DETAILS
This paper extensively discusses key hyperparameters specific to current offline MBRL algorithms. However, there are significant code-level implementation details which are often critical for strong performance and make it hard to disambiguate between algorithmic and implementation improve- ments. Worryingly, many of these details are not mentioned in their respective papers, or are different between the authors' code and paper. We detail clear examples of this below. We believe further investigation of these code-level implementation details represents important future work, as has already been done for policy gradients (Engstrom al., 2020; Andrychowicz al., 2021), Indeed is unclear if the improvement of MOReL over MOPO is due to its different P-MDP formulation, or is successful in spite of this formulation, due superior policy optimization strategy or dynamics model design. We believe that this paper takes a significant first step in tackling this issue by directly comparing a number of key design choices, and understanding their individual impact. Now we summarize key differences between the paper and code for the MOPO and MOReL algorithms which we compare against that are crucial to achieve the same reported performance.
In MOPO,
Each layer in the model neural network has different level of weight-decay
The authors' code uses different objectives for training (log- (oggliikeihhod) and validation (MSE).
The authors elites, but only for state prediction (as discussed previously). In MOReI,,
There is difference in the authors' code about how the penalty threshold is calculated and tuned, and isn't provided as hyperparameter in the appendix.
The absorbing HALT state does not appear in the authors' code.
The negative halt penalty appears significantly different between code and paper.
There a minimum trajectory steps parameter (hard-coded to 4) not mentioned in the paper.
The reward function appears to be hard-coded in the authors' implementation, not learned as stated the paper.
The policy architecture is different in the authors' code (64,64 hidden layers) and the paper (32,32 hidden layers)
It is not clear when the optional behavior cloning initialization step is applied.
G HYPERPARAMETERY AND EXPERIMENT DETAILS
The D4RL (Fu et al., 2021a) codebase and datasets used for the empirical evaluation is available under the CC BY 4.0 Licence. As stated in the main text, choose to 'v0' experiments these are more challenging for Hopper due to having low return trajectories (Kostrikov et al., 2021), and we clearly state when other benchmarks use the 'v2" experiments, which have offline trajectories with higher returns on Hopper.
The remaining hyperparameters for the MOPO algorithm that we do not vary by Bayesian Opti- mization were taken from the original MOPO paper (Yu et al., 2020), apart from we fix the number of policy epochs/iterations to 1,000 for experiments. This means our implementation uses the same probabilistic dynamics models (with unchanged hyperparameters) and policy optimizer (SAC, Haarnoja et al. (2018)) as MOPO, differing from MOReL, which uses Natural Policy Gradient (Kakade, 2002).
The hyperparameters used for the BO algorithm, CASMOPOLITAN, are listed in Table 7. We use the batch-mode of CASMOPOLITAN, where multiple hyperparameters settings proposed and evaluated concurrently.
Each BO iteration is rum for 300 epochs single seed, and the full optimization dataset took -200 hours on a NVIDIA GeForce GTX 1080 Ti GPU taken up predominantly by MOPO training.
offline
35

---

Published as a conference paper at ICLR 2022
Table CASMOPOLITAN Hyperparameters
Parameter
Value
Number of parallel trials Number of random initializing points
20
ARD
False
Acquisition Function
Thompson Sampling
Global BO Kernel
True
CoCaBo Kernel (Ru al., 2020)
Unless specified otherwise, plots and reported statistics are completed with 7 models in the ensemble, as this the number chosen in the original MOPO paper used with the Max Aleatoric penalty.
H RLIABLE FRAMEWORK FOR PERFORMANCE EVALUATION
Throughout this work, we choose to adopt the rliable framework introduced in Agarwal al. (2021) to evaluate the performance of our approaches. r cliable advocates for computing aggregate performance statistics and probability of improvement across many tasks benchmark suite; indeed we take this approach when reporting the values in the analysis performed in Sections 4 and 5. This important when the number of tasks become large, and also prevents outliers from dominating mean statistics. Furthermore, this allows us make clear statements about improvements given the relatively low number of seeds that are used in deep RL; indeed, Agarwal et (2021) show that using high seed counts does not ameliorate the variance issues experienced when training such algorithms. For normalization, we use the standard D4RL return scaling.
I EVALUATION USING AUTOMATIC CONSTRAINT TUNING
We show the full tabulated results from Sec. with statistical significance using the rliable framework in Table 8, using the 'Probability of Improvement" metric in Agarwal et al. (2021).
To evaluate our claims in Sections 4 and 5 without needing to laboriously tune the penalty weight per environment, employ automatic penalty tuning scheme, analogous to the automatic entropy tuning used in Haarnoja et al. (2018). Concretely, given a constraint value A, at each epoch we minimize:
((A) = L6, a [log A(A A u(s:, a:)]]
We start from an initial weight À = 1. We observe that the penalty weight found by automatic tuning tends converge within the first 50 epochs and then remains stable throughout training.
Table Improvement over grid-searched MOPO through restricted hyperparameter choices (e.g., one single choice, or an arg max between two) on the D4RL MuJoCo benchmark. The single and two setup approaches both use the Ensemble Std. penalty and N 10.
Algorithm
Average Score
Pilmprovmmen over MOPO)
MOPO (default hyperparameters)
34,2
Single setup: (h = 20, A 1)
49,0 (+43%)
73.96%
Two setups: arg max ((h 10. A = 0.5). (h 20. A 1))
57.8 (+69%)
80.20%
Optimized MOPO (ours, Table 3)
65.2 (+91%)
89.06%
36

---

Published as a conference paper at ICLR 2022
J BEST FOUND ADROIT HYPERPARAMETERS
We present the best found hyperparameters under our BO procedure for the D4RL Adroit tasks in Table We see similar trends as our main evaluation in Table favoring higher rollout lengths and the Ensemble penalties.
Table 9: Best discovered hyperparameters using BO for Adroit
Discovered Hyperparameters
Environment
N
Penalty
cloned
10
6.64
12
Ensemble Std
pen
human
11
0.96
37
Ensemble Var
expert
4.56
Max Aleatoric
cloned
10
0.21
12
Ensemble Var
hammer
human
2.48
47
Ensemble Std
expert
0.99
Ensemble Std
37


Reviews:
[{"review_id": "yfOp35kKfgD", "reviewer": "Reviewer_eM3K", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution of this paper is to systematically compare several uncertainty estimate methods. Perhaps surprisingly, such comparison has not been conducted before, even in those papers that originally proposes to implement a pessimistic MDP. To this end, this paper fills an important gap in the literature. The paper is also well written. The experiment setups are clearly defined. \n\nHowever, my major concern is that the technical contribution of this paper is not strong enough. Other than designing several interesting experiments to compare model uncertainty estimation methods, the paper does not contribute enough new insights that can potentially be interesting for a broader research community. \n\nFurthermore, I think a very related paper is missing (Abbas et al., 2020). This paper also compare several uncertainty estimation methods in model based RL, including the ensemble based approaches (Lakshminarayanan et al., 2017). The main conclusion is very similar: a combination of epistemic and aleatoric uncertainty is the best choice. \n\nAbbas, Zaheer, et al. \"Selective Dyna-style planning under limited model capacity.\" International Conference on Machine Learning. PMLR, 2020.\n\n", "summary_of_the_paper": "Model-based offline reinforcement learning algorithms typically involve constructing a pessimistic MDP, which is implemented based on an uncertainty estimation of the learned model. This paper conducts empirical analysis to compare different design choices of the uncertainty estimation in practice. In more details, the authors compare different approaches in terms of the correlation between the estimated uncertainty and ground truth model error. They also use bayesian optimization to search the best hyperparameter configuration that achieves strong empirical performance. ", "summary_of_the_review": "I recommend rejecting this paper because the technical contribution is not strong enough. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pCMrUs15dnj", "reviewer": "Reviewer_TWSZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this work is well written and presents a large range of empirical results on the utility of different uncertainty heuristics. The wide scope of these experiments, evaluating the uncertainty penalties in relation to both model prediction error and downstream MBRL performance is a strength of the approach. The approach highlights that the predictive variance or standard deviation of the ensemble predictions can perform well as an uncertainty penalty, even though past work in offline MBRL has not considered this. In addition, it highlights the important role that hyperparameters play in controlling the performance of the MOPO algorithm.\n\nMy main concern with this work is that does not make any actionable claims. They claim that they achieve SOTA results by choosing different hyperparameters than previous work, but this hyperparameter selection is chosen via Bayesian optimization that is specific to the same environment. To me, this is the equivalent of \"training on the test set,\" in that it uses interaction with the true system to select hyperparameters for the same system. In that sense, the result can no longer be called offline RL. Of course, hyperparameter selection is an under addressed topic in the community, and there are many papers which manually select hyperparameters that yield good performance. The fact that explicitly optimizing these hyperparameters through interaction on the test environment can yield better performance is not very surprising, but also not very useful for the target applications of offline RL where interaction with the true system is expensive or safety-critical. Indeed, these results do not even paint a consistent picture where one uncertainty penalty is clearly preferred to the others. The work would be significantly improved if the authors used the results of their empirical experimentation to propose a general recommendation for penalty selection and hyperparameter selection. \n\nOther comments:\n- Experiments are limited to deterministic environments where the distinction between aleatoric and epistemic uncertainty is unimportant. In stochastic environments, I don't see why aleatoric uncertainty should factor into a reward penalty, so the ensemble variance and standard deviation penalties proposed may not work well in such environments.\n- The claim that the ensemble standard deviation or variance is \"arguably the most principled\" choice is not supported. Why is the variance of the ensemble predictions match the total variation distance term? What makes it \"more closely aligned with the theory\"?\n- The AUC / AP are hard to compare across percentile choices because the relative proportion of positive and negative samples is also changing, so the AUC and AP of a random classifier changes across this task. It would be interesting to report these results on a subsample with even proportions of low-error and high-error samples such that the AUC of a random classifier would be 0.5 in all cases. Furthermore, the precision recall curves in the appendix don't make sense to me, as recall should vary all the way from 0 to 1 as the threshold changes. Perhaps the axes are flipped?\n- It is unclear what is meant by \"latent dynamics that are KL-regularized to a spherical Gaussian\" (last paragraph of section 4.2)", "summary_of_the_paper": "The authors present an empirical study of several uncertainty quantification heuristics applied to model learning in offline model-based reinforcement learning. Specifically, they consider the basic architecture of MOPO, in which an uncertainty based state-action penalty function is applied on top of the standard reward to construct a pessimistic MDP. Within this set up, the authors perform a empirical study of several different uncertainty penalties, exploring their correlation to the prediction error of the model, as well as in terms of their ability to detect individual transitions with high-percentile prediction errors. Finally, the authors perform Bayesian optimization over the choice of the uncertainty penalty as well as other hyperparameters such as number of ensemble elements, planning horizon, penalty weights, etc. They present these results showing that the optimal choice of penalty and penalty weight can vary significantly, not only between environments, but also within an environment as a function of offline dataset.", "summary_of_the_review": "Overall, the paper presents a large amount of empirical results, but is lacking a well supported, clear actionable message beyond using longer rollout horizon and ensemble standard deviation as an uncertainty penalty. Even this claim is lacking empirical support, as there is no evaluation of a fixed, guiding principle for hyperparameter selection that is held constant over a wide range of domains. As such, I'm hesitant to recommend acceptance. If the manuscript is updated to have clearer empirical evidence for the main claims in the paper, including a fair evaluation of the recommended strategy across a variety of domains without optimizing hyperparameters separately for each environment, I would increase my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "L1E57BlqSOY", "reviewer": "Reviewer_uPzV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points:\n\n\nThis is a quite thorough evaluation which compares several of the recent popular offline MBRL works, and investigates critical design choices made in each of them. I feel that this is a valuable contribution to research in offline RL, since it’s generally quite difficult to understand where performance improvements between various methods come from due to so many differences in hyperparameters.\nThe comparison metrics for the uncertainty penalty are quite insightful and reasonable.\n\nWeak points:\n\n\nThe “optimized” version presented in the paper only optimizes and presents statistically significant results compared to MOPO, so it is unclear if similar trends hold for the other methods compared to in the paper and there may not be sufficient evidence to argue that hyperparameter selection affords large improvements for other algorithms.\n\n\nThe “Optimized” numbers are reported by optimizing over each D4RL task. With that, it seems unsurprising that with a single setting and an optimization run over a single seed, the hyperparameters can be tuned better than the original implementations. It seems important to this evaluation to also determine how stable these hyperparameter choices are across random seeds.\n\nQuestions/clarifications/nitpicks:\n\n\nIt’s a bit confusing to me to present “optimized” as “our method” in section 6, when the difference (as far as I am aware) is in the hyperparameter selection of MOPO (which I think could be made more clear). \n\nAdditionally, in Figure 1, “Optimized” has not yet been introduced, so while in hindsight it is clear that it’s an optimized version of MOPO, it may not be immediately obvious.\n\nFigure 1(b) and the methodology used in this claim is not clearly described in the text\n", "summary_of_the_paper": "The paper provides an evaluation of many of the design choices and hyperparameter decisions made in offline model-based reinforcement learning methods which have emerged recently. Particularly, the empirical study looks at uncertainty penalties used in these methods, as well as hyperparameters such as ensemble size, penalty weighting, and rollout horizon. The authors find that offline MBRL methods are quite sensitive to each of these parameters. They compare the “optimized” version of MOPO with hyperparameters tuned using Bayesian optimization, and find that it leads to statistically significant performance improvements over the version of MOPO presented in the original paper.", "summary_of_the_review": "I recommend accepting this paper. I think that more rigorous evaluations of the design and choices made in RL and particularly offline RL methods are important contributions to help drive future research. I think the analysis of the uncertainty quantification methods is particularly interesting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KkJogqpzEwc", "reviewer": "Reviewer_PM7Z", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- the paper is well written and overall easy to read and understand.\n- such work, summarizing different contributions and comparing the different choices that are made in each contribution, are important for the research community as well as practitioners\n- I also like the different messages and perspectives that are made in the paper\n- The related work seems rather complete for model-based offline RL.\n\nWeaknesses\n- I would need a better explanation of Figure 1a and section 5.3. (see detailed comments below).\n- for such an empirical work error bars are needed in Table 1, 2, 3, 4 to judge the statistical significance of the results. Table 3 identifies algorithms that are significantly better but it would still be nice to have corresponding std or error bars for the different performances.\n- What about existing calibration metrids such as the expected calibration error (ECE)?\n\nOther comments:\n- Figure 1:\n    * the figure should be better documented, even if this has to be in the appendix. A precise definition of the y-axis of FIgure 1a should be given and a precise description of how this figure was computed so that someone can reproduce it. What's the environment? Is the error and uncertainty the ones for the reward or another dynamics variable (or the mean error over all the dynamics state variables)\n    * I would especially like the authors to clarify whether the errors are computed step by step comparing the outputs f(s,a) (obtained with the true dynamics) and g(s,a) (obtained with the model) assuming same inputs (s,a) or for multi-step predictions i.e. considering a fixed sequence of actions and a first initial state s_0 look at the error between each state of the sequence (s_1, ..., s_t) predicted by a rollout with the model and its corresponding state in the sequence (s'_1, ..., s'_t) obtained for the same action sequence with the true dynamics when starting from the same initial state s_0. Is it what is being computed in Figure 1a and in Figure 27? What is the difference between figure 26 and figure 27: Figure 26 is just the median overall several rollouts whereas figure 26 shows individual rollout?\n    * I think this is a bit more clear in appendix D, but I would still like the authors to make this clear in their replies and to make the effort of making this more clear in the main paper as well.\n    * Appendix D: Can the authors elaborate on the \"displacement\" variable? Is this a variable needed when setting the state of the Mujoco Simulator?\n    * for Figure 1b: the authors should clarify the definition of the probability of improvement and how this is computed\n- For figure 1 and table 1: what if the error used is the log likelihood instead of the MSE?\n- Can the authors confirm that the hyperparameter optimization is done by evaluating the performance of the different hyperparameters on the real environment? My comment here is that this is impossible in practice. I would not let this influence my review because I think this is the current practice for most papers in RL.\n- Table 2: \"the penalties are powerful at identifying dynamics discrepancy but not as accurate at identifying when the world-model data is out-of-distribution\" -> it seems that AUCs of 0.9 are reachable which is not too bad in my opinion?\n\nMinor Comments\n- Figure 1a legend: during single a model rollout -> during a single model rollout?\n- Introduction: \"Furthermore, using the insights gained from this section\" -> it is not clear at all what \"this section\" is refering to here.\n- Section 3. reward notation \"r\" is also used for the Pearson bivariate correlation.\n- equation (1) -> I think a capital \"R\" should be used for the reward function.\n- I should maybe have a look at the paper by Yu et al, 2020 but how do they justify the use the aleatoric uncertainty as a penalty? Aleatoric uncertainty is specific to the system and even the true dynamics have an aleatoric uncertainty. I would expect better results by considering epistemic uncertainty which is what is attempted when using the ensembles.", "summary_of_the_paper": "Recent successful model-based offline RL techniques have relied on heuristics to penalize rewards according to the uncertainty of the estimated MDP. This paper reviews the different penalties that have been designed in the literature. The impact and importance of associated hyperparameters such as the planning horizon and the number of models in the ensemble are also evaluated. The author show that the selection of the best penalty and best hyperparameters lead to stronger performance.", "summary_of_the_review": "I like the paper and the work done by the authors. I think it is valuable for the community (researchers and practitioners). I would just like to have some clarifications about the points I raised in my review. I will change my score accordingly.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7QJWP1izpM4", "reviewer": "Reviewer_DXYY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the research is well-motivated and interesting, while I have some concerns about the experiments. I give the comments in the following.\n\nPros:\n\n1. The research is well-motivated.  The gap between the true model and the learned model is estimated by uncertainty measurements. Nevertheless, in the MOPO and Morel papers, different uncertainties are not rigorously studied. I think this problem is important and should be studied in-depth.\n\n2. The use of Spearman rank and Pearson bivariate to identify the correlation between uncertainty and true MSE is novel. The experiments verify such measurements are closely related to the performance.\n\nCons:\n\n1. The experiments in section 4.2 are a bit confusing. Why should we use exploitative policies that are generated by non-penalty methods to generate trajectories? The non-penalty methods cause overestimation and poor performance, and the resulting policy may be similar to a random policy. The trajectories generated by such policies may lie in a small range of state-action space. I suggest using the trajectories from the medium-replay dataset to evaluate since it covers a large state-action space. You can point it out if I don't understand correctly.\n\n2. The aim of section 5.3 is to study the effect of rollout horizon. It is unclear how the results presented in table 2 relate back to the rollout horizon, since I do not find the rollout horizon is used as a variable in Table 2. The dynamics discrepancy should be related to the horizon length. Can you explain more? \n\n3. In Table 3, the results mix the D4RL v0 and v2 datasets. It is not valid since these two datasets are quite different. The v2 dataset is much larger than the v0 dataset. I suggest unifying all results in a single dataset. Maybe you can refer to other offline RL papers released recently to get the scores of MOPO and Morel in v2 dataset.\n\n4. Considering different tasks have different optimal hyper-parameters, I suggest adding additional analysis about the sensitiveness of N, lambda, and h in Table 3. E.g. Should lambda be larger in single-policy datasets and be smaller in mixed datasets?  Does a larger N perform better? Does N affect the performance significantly if N>10? Maybe h is sensitive to different tasks?\n\n5. The author should explain why “Ensemble Standard Deviation” measures both the epistemic uncertainty and aleatory uncertainty. The paper should be self-contained. What about other methods considered in section 4 (in measuring the epistemic uncertainty and aleatory uncertainty)? Can we draw the conclusion that both the aleatoric and epistemic are important to offline RL?\n\nMinor:\n\n1. The method to obtain true dynamics error should be given in discussing Figure.1.  Although it has been discussed later in the paper.\n\n2. At the end of section 5.1, the paper writes “This again suggests that the number of models not only affects the quality of the estimation, but also its distributional shape. ” Can you explain more?", "summary_of_the_paper": "The paper provides detailed analysis of different uncertainty quantifications in Model-based offline RL, from both statistical and empirical perspectives. Further, the paper performs Bayesian optimization to find hyper-parameters and the optimized hyper-parameters perform better empirically. \n", "summary_of_the_review": "The paper is well motivated while it needs to be improved in clarity. (1) The dataset used in evaluation should be unified. (2) The experimental setup in Table 1 and Table 2 should be explained more clearly. (3) Additional ablation or explanation of Table 3 is suggested to be added. (4) Some discussion about the epistemic and aleatoric uncertainty.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper empirically studies various design choices in offline model-based RL algorithms, with a focus on MOPO (Model-based Offline Policy Optimization). Among the key design choices is the uncertainty measure used in MOPO that provides an (approximate) lower bound on the performance, the horizon rollout length, and the number of model used in ensemble.

The reviewers are positive about the paper, found the experiments thorough, and the results filling a gap in the current literature. They have raised several issues in their reviews, many of which are addressed in the rebuttal and the revised paper. I would like to recommend acceptance of the paper. Also since the results of this work might be of interest to many researchers working on model-based RL, I also recommend a spotlight presentation for this work.

I have some additional comments:

(1) The paper studies the correlation of uncertainty measures with the next-state MSE, with the aim of showing which one has a higher correlation. The underlying assumption is that the next-state MSE is the gold standard that we should aim for.

If we go back to the MOPO paper, we see that to define an uncertainty-penalized reward, we need an upper bound on the absolute value of G(s, a), which is the difference between the expected value of the value function at the next-state according to the true model and the estimated model.

If we assume that the value function belongs to the Lipschitz function class w.r.t. a metric d, this upper bound is proportional to the 1-Wasserstein distance between the true next-state distributions and the model's distribution.
If the dynamics is deterministic, 1-Wasserstein distance becomes the $d( T(s, a), \hat{T}(s,a) )$. If the distance d is the Euclidean distance, this becomes the squared error.

Therefore, the squared error makes sense for deterministic dynamics, and it only provides an upper bound of $|G(s, a)|$.
If the environment is not deterministic, the squared error may not be a reasonable gold standard anymore to compare the correlation of various uncertainty measures with.

The paper introduces a generic MDP framework, but does not mention anything about its focus on MBRL for deterministic environments until the last sentence of its conclusion. Please clarify this in your camera ready paper.

(2) The experiments are conducted using 3 or 4 seeds. Although this is the common practice in the deep RL community, it is too small. Standard deviations in Tables 1, 2, ... are computed with 3 seeds, which would be cringeworthy to statisticians and empirical scientists. I encourage the authors to increase the number of independent random experiments to make their results more powerful.
Decision: Accept (Spotlight)