Revisiting Design Choices in Offline Model Based Reinforcement Learning
Authors: Cong Lu
Philip Ball
Jack Parker-Holder
Michael Osborne
S Roberts
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Model-Based Reinforcement Learning
Offline Reinforcement Learning
Uncertainty Quantification

Abstract: Offline reinforcement learning enables agents to make use of large pre-collected datasets of environment transitions and learn control policies without the need for potentially expensive or unsafe online data collection. Recently, significant progress has been made in offline RL, with the dominant approach becoming methods which leverage a learned dynamics model. This typically involves constructing a probabilistic model, and using it to penalize rewards in regions of high uncertainty, solving for a pessimistic MDP that lower bounds the true MDP. Recent work, however, exhibits a breakdown between theory and practice, whereby pessimistic return ought to be bounded by the total variation distance of the model from the true dynamics, but is instead implemented through a penalty based on estimated model uncertainty. This has spawned a variety of uncertainty heuristics, with little to no comparison between differing approaches. In this paper, we show these heuristics have significant interactions with other design choices, such as the number of models in the ensemble, the model rollout length and the penalty weight. Furthermore, we compare these uncertainty heuristics under a new evaluation protocol that, for the first time, captures the specific covariate shift induced by model-based RL. This allows us to accurately assess the calibration of different proposed penalties. Finally, with these insights, we show that selecting these key hyperparameters using Bayesian Optimization produces drastically stronger performance than existing hand-tuned methods.

Revisiting Design Choices in Offline Model Based Reinforcement Learning
Anonymous Author(s)
Affiliation Address
email
Abstract
Offline reinforcement learning enables agents to leverage large pre-collected datasets of environment transitions to learn control policies circumventing the need for potentially expensive or unsafe online data collection. In recent times there has been significant progress in offline RL, with the dominant approach be- coming methods which leverage a learned dynamics model. This typically involves constructing a probabilistic model, and using the model uncertainty penalize rewards where there is insufficient data, solving for pessimistic MDP that lower bounds the true MDP, Recent work, however, exhibits breakdown between theory and practice, wherehy pessimistic return ought to be bounded by the total variation distance of the model from the true dynamics, but instead implemented through a penalty based on estimated model uncertainty. This has spawned variety of uncertainty heuristics, with little to no comparison between differing approaches. In this paper, we compare these heuristics, and design novel protocols to investigate their interaction with other hyperparameters such as the number of models, or imaginary rollout horizon. Using these insights, show that selecting these key hyperparameters using Bayesian Optimization produces optimal configurations that are vastly different those currently used in existing hand-tuned state-of-the-art methods, often resulting in drastically stronger performance.
10
11
12
14
16
18
Introduction
20 In offline (or batch) reinforcement learning (RL) 15. 26 the goal is to learn policies that perform 21 well in an environment given a fixed data of pre-collected experiences. This could have vast 22 implications for using RL real-world settings, as agents can make use of ever increasing amounts of data without the need for accurate simulator, while also avoiding expensive and potentially even unsafe exploration in the environment.
24
Model-based reinforcement learning (MBRL) has recently shown promise in this paradigm, obtaining state-of-the-ard performance on offline RL benchmarks [21 48 improving upon powerful model-free approaches (i.e., (33). MBRL works by training dynamics model from the offline data, then optimizing a policy using imaginary rollouts from the model. This allows the agent to learn from on-policy experience, the model is agnostic to the policy used to generate data. Furthermore, recent work has demonstrated the utility world models beyond maximizing return, such as generalizing to unseen environments 44, transferring to new tasks in the environment [49 and learning with 32 safety constraints 2 Therefore, the case for MBRL offline RL clear: not only does it represent the-art in terms of performance, but it also provides the opportunity maximize the signal 34 in the offline data to generalize onto tasks beyond those encoded by the behavior policy.
26
28
30
However, a common failure mode of MBRL is when the policy exploit the model in parts of the 36 state-action space where the model is inaccurate. Thus, naive application of MBRL to offline data can
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

37 result in sub-optimal performance. To prevent this, concurrent recent works 49 21 have approached the problem by training a policy pessimistic MDP (P-MDP). The P-MDP lower bounds the true MDP, and discourages the policy from regions where there is large discrepancy between the true and learned dynamics; this often provides a theoretical guarantee of improvement over simply cloning the behavior policy. This is made practically possible by adding penalty proportional to the uncertainty in the dynamics model. However, while these recent successes are similar in principle, in practice they differ in series of design choices. First and foremost, they make use of different heuristics to measure model uncertainty, in some cases deviating from simpler metrics which are more consistent with the theory. Indeed, these decisions are justified by superior performance, given limited amount of hyperparameter tuning or analysis.
38
39
40
41
42
43
44
46
47
In this paper we conduct rigorous investigation into a series of these design choices. We begin focusing on the choice of un- certainty metric, comparing both recent state-of-the-ar offline ap-
Nomalozd Meature
Dymamics Erer MOPO
48
MORaL
49
Ensemble Std.
50
proaches 21
49
41
with additional metrics used in the online
4-
Ensemble Var.
51
We also explore the interaction with a series of other hyperparameters. such as the number of models and imagi- nary rollout length. Interestingly, the relationship between these variables and the model uncertainty varies significantly depending on the choice of metric. Furthermore, we compare these uncertainty heuristics under new evaluation protocols that, for the first time, capture the specific covariate shift induced by model-based RL. This allows us to assess calibration to model exploitation in MBRI,, observe that some existing penalties surprisingly successful at capturing the errors in predicted dynamics, seen in Fig- Finally, using the insights gained from this section, we test the capability of existing methods given an optimal choice over all variables, modeled jointly using a powerful Bayesian Optimization algorithm 40. We find that simple and intuitive uncertainty measure can provide state-of-the results in continuous control benchmarks when properly tuned, and the chosen hyperparameters align with our analysis.
setting B
7
52
60
80
100
54
Time Step Model
How penalty and true error vary over model rollout
Figure 1:
56
58
59
60
61
62
63
64
We believe this work will contain a variety of interesting insights for researchers and practitioners in offline RL. Below we highlight some of the main findings:
66
Longer horizon rollouts with larger penalities can improve existing methods. We see that conducting significantly longer rollouts inside the model, coupled with larger uncertainty penalities, typically improves performance.
67
68
69
Penalties that are more closely aligned with the theory achieve better correlation with OOD measures. The deep ensembles approach of [25] often outperforms the penalty from MOPO 4g
70
71
and MOReL 21 We observe that the ensemble standard deviation is statistically strikingly similar to the MOReL penalty, but has improved correlation and scaling behavior.
72
73
Uncertainty is more correlated with dynamics error than distribution shift. We find that suc- cessful penalties measure the discrepancy in dynamics, and can in fact assign high certainty to regions far away from the offline data.
74
75
76
2
Related Work
77
Two recent works concurrently demonstrated the effectiveness of model based reinforcement learning (MBRL) in the offline setting. MOPO 49 follows MBPO [19] but trains inside conservative
78
79
MDP which penalizes the reward based on the maximum aleatoric uncertainty over the ensemble members. MORel. achieves even stronger performance, penalizing the rewards by penalty based on
80
81
the maximum pair-wise difference in ensemble member predictions. For pixel-based tasks, LOMPO 41 also proposed novel penalty, using the variance of ensemble log- likelihoods. Outside of the offline setting, probabilistic dynamics models leveraging uncertainty have underpinned a series of Uncertainty can also be measured in MBRL without the use of neural networks 10 although these methods tend to be harder to scale and thus lack widespread use.
83
84
successes g 35
37
85
Effective hyperparameter selection in RL has been shown to be crucial to the success of commonly used algorithms [1 12]. This becomes even more challenging in MBRL with additional hyperpa- 88 rameters for the dynamics model and model architecture needing to be selected. Recent work has
86
87
89 shown that carefully optimizing these hyperparameters for online MBRL can significantly improve performance, with the tuned agent breaking the MuJoCo simulator [50] In contrast, we focus on the

---

91 offline setting, and investigate parameters specifically related to uncertainty estimation. Previous work
92 studied the impact of hyperparameterss in offline RL [36], finding offline RL algorithms to be brittle
93 to hyperparameter choices. However, unlike our work they only consider model-free approaches, whereas we specifically investigate model-based offline algorithms.
94
95
Our work also relates to the rich literature on deep ensembles [25], which train multiple deep neural networks with different initializations and different dataset orderings, and generally outperform variational Bayesian methods [27 11 Achieving effective uncertainty calibration with neural networks
96
97
96
notoriously difficult [6
22 28 and furthermore we require good calibration in the face of co-
variate shift [34] as the policy we learn in the model will likely deviate from the behavior policy that generated the offline data. Indeed, recent work has highlighted this issue in offline RL 223 48
99
100
101
and has reported superior performance despite eschewing model uncertainty entirely. However, it unclear if this performance improvement is due to poor uncertainty calibration, implementation details, or fundamental limitation of the pessimistic-MDD formulation.
102
103
Background
104
105
All of the methods we investigate in this paper model the environment as a Markov Decision Process (MDP), defined tuple M (S. P. R. pu: Y), where S and A denote the state and action spaces respectively. P(s'|s, a) the transition dynamics, R(s, a) the reward function, Po the initial state distribution, and y E (0, 1) the discount factor. The goal is to optimize policy л (als) that maximizes the expected discounted return Po |Ei= ? R(st, 0:)]-
107
108
109
110 In offline RL, the policy is not deployed in the environment until test time. Instead, the algorithm only has access to a static dataset Denv = {(ss, a, r, 8'),. collected by one or more behavioral policies To. Following the notation in [49] we refer to the distribution from which Dene was sampled as the behavioral distribution. The most prominent offline MBRL methods all train an ensemble of N probabilistic dynamics models B2] These usually learn predict both the next state s' and reward from a state- satteaatioon pair, and are trained on Dene using supervised learning. Concretely, each of the N models output Gaussian P5 ax) (u a.), as), parameterized by ø. The resulting learned dynamics model P and reward model R define model MDP M A, R, po: Y).
111
113
115
117
118
To train the policy, we use k-step rollouts inside M to generate trajectories
43
119
To prevent policy exploitation in a model, pessimistic MDP (P-MDP) is constructed by lower bounding the true-expected return using some error between the true and estimated models. For instance, in 2y the authors show that a lower bound on the return can be established by penalizing the reward by a measure that corresponds to estimated model error:
120
121
122
TM S E
[r(s, a) - > >| (s, a)]
(I)
(s,a)=p7
123
Ay Several potential choices for |G (s, a)) are proposed, including an upper bound based the total variation distance between the learned and true dynamics. However, for their practical algorithm the authors elect to use an alternative, based on impressive empirical results. Concurrent to MOPO,
125
126
MOReL (21] constructs P-MDP by augmenting standard MDP with negative valued absorbing
127
state that is transitioned to when total variation distance between true and learned dynamics is exceeded. They show that a policy learned in the P-MDP exceeds simple behavior cloning. Whilst dynamics-based total variation distance has desirable theoretical properties, the practical algorithm
128
129
130
relies on heuristic approximate this quantity. Next, we investigate the penalties used in these works, as well as other under-used candidates, and explore their effectiveness.
131
Uncertainty Penalty
132
133
As we have discussed, the key idea underpinning recent success in offline MBRL is the introduction of a conservative MDP, penalized by some uncertainty penalty. The theory dictates this should be
134
135
some distance measure between the true and predicted dynamics. Of course, this cannot be truly
136
estimated without access to an oracle, so instead proxy for this quantity is constructed instead. In
137
this paper, we compare the following uncertainty heuristies, from recent works in both offline and online MBRL:
3

---

139
m I / puted over the variance heads of the model ensemble.
MОPO 49
a)IlF. which corresponds to the maximum aleatoric error, com-
140
141
MOReL [2]]:
maxi,illd (s, a) - (s, (),a))l which corresponds to the pairwise maximum differ-
142
ence of the ensemble predictions.
143
LOMPO 41l: Var(flog (s s, a). i = 1, .T). where s' is next state sampled from a single
144
ensemble member. We evaluate its log-likelihood under each ensemble member and take the variance.
M2AC 371: D (-Is, a)]l P (-ss (lss.)),, a)]. which corresponds to the KL divergence between the Gaussian parameterized by the randomly selected ensemble member we generate the next state from,
145
146
147
and the aggregated Gaussian of the remaining ensemble members.
Ensemble Standard Deviation/Variann
[25 (s. (EE + (s)())
149
(I" (s, a))² where Je" is the mean of the means (1*
N Σ (s)ss This corresponds to combination of epistemic and alcatoric model uncertainty. This is surprisingly under-utilized
150
151
in offline MBRL, and is arguably the most principled uncertainty penalty. We choose evaluate both standard deviation and variance as this will provide intuition about the importance of penalty distribution shape.
152
153
154
Each of these penalties can be computed using the output from an ensemble of probabilistic dynamics models 25
thus, are able to compare them in controlled manner.
156
4.1
How Do These Perform on Fixed Offlie Datasets?
157
We begin by assessing how well uncertainty penalties correlate with next state prediction error. This crucial in order to correctly penalize the policy from visiting parts of the state-action space where the model is inaccurate, and therefore exploitable. We use the datasets from D4RL [14,, train models on each dataset, then evaluate them on other datasets from the same environment, but collected under different policies. This is important we may change the task we train on in the model (such as the Ant-direction experiment in 59 ) so require good calibration on unseen data. As a result, we call these our 'Transfer' experiments. We compare the penalty and MSE for variety of settings in the Appendix (see: Section A.2), with a snapshot in Fig. 2 We measure Spearman rank (p) and
158
159
160
162
164
Pearson bivariate (r) correlations, and discuss this in App.
Full details of all experiments and
186
hyperparameters are given in App.
NOPO
MOROL
Ensomble
Lat Ensombic
LOMPO
JAl
MIAC
2,2
15-
3.3-
3.3
0.5
F 0223
A 9.254
1.065
3.5-
3.3
1.0
0.5
1.005
500
0.928
1.012
0.042
30
23
23
MIE
Figure 2: Scatter Plots showing models trained on D4RL Medium being tested on data from Random. Green HalfChectah, Blue= Hopper.
167
Before we begin analyzing these results in detail, we now introduce a novel approach to assessing penalties under the OOD data induced by model exploitation by policy.
169
4.2
How Do These Perform During an Imaginary Rollout?
170
We now design an experiment aimed at capturing the OOD data generated by the actual offline MBRL process, which we call our 'Ground Truth' experiments. First, we train set of policies without a penalty inside the model. We then measure the difference between the return predicted by the model over a rollout, and the true return the real environment. We define policy to be 'exploitative" the model significantly over-estimates the return compared to the true return. It is vital that we train exploitative policies as these precisely induce the extrapolation errors which cause MBRL methods to
171
172
175
174
175
fail in the offline setting. It is therefore important that the penalty is able to accurately determine when

---

177
the model is being exploited in this way. We use a subset of the most exploitative policies to generate trajectories in the model, and record the uncertainty predicted by each penalty at each time step. To generate the ground truth data, we then 'replay" these trajectories in the true environment, loading the state and action taken in the model into the environment, and record the 'true" next state according to
178
179
180
the MuloCo simulator A4. The 'Ground Truth' is therefore the MSE between the predicted next
182
state and actual next state. Additional details are provided in App. Dalong with plots in App. A.2
Table I summarizes the results from both the 'Transfer' and 'Ground Truth' experiments.
Table 1: Statistics of all experiments averaged over different test settings.
Transfer
Ground Truth
HalfCheetah
Hopper
HalfCheetah
Hopper
Penalty
r
p
MOPO
0.780
0.545
0.710
0.411
0.581
0,419
0.732
0.484
MOReL.
0.789
0.624
0.772
0.571
0.581
0.518
0.750
0.546
Ensemble Std.
0,820
0,644
0.789
0.556
0.608
0.521
0.789
0.545
Ensemble Var.
0,821
0.671
0.786
0.589
0.604
0.493
0.767
0.545
LOMPO
0.126
0.141
0.361
0.122
0.035
0.067
0.496
0.161
M2AC
0.029
0.107
0.111
0.082
-0.019
0.062
0.220
0.095
We immediately notice that the LOMPO and M2AC penalties have very weak correlation with MSE for the examples in Fig. 2 We believe this is the case because LOMPO relies on likelihood statisties, which are notoriously sensitive, and has been designed for use in scenarios involving 'well-behaved' latent dynamics that are KI.rregularized to a spherical Gaussian. Regarding M2AC, we note that this penalty was designed for the online setting with significantly less data, and becomes quite uncorrelated in this larger data setting. We believe this advocates for the design of penalties that are less reliant on distributional information concerning the separate Gaussians in the ensemble,
185
186
187
188
189
190
191
as these penalties appear sensitive to the quality of their estimated distributions. We observe that MOPO, MOReL and the ensemble penalties perform broadly similarly despite their analytically different forms. We do observe, however, the ensemble measures display noticeable improvement
192
193
194
as a ranking statistic. We also observe a significant loss in performance between the Transfer and Ground Truth HalfCheetah settings, with the latter being relatively poor. This implies further work needed to develop penalties that can successfully detect the type of dynamics discrepancies that actually occur in offline MBRL. Finally, we observe that despite the similar rank correlations p. the bivariate correlations r can vary considerably, and observe from the scatter plots that MOPO exhibits low kurtosis, having large penalty values 'bunched' at its extreme; we provide 3rd and 4th order moment statistics to facilitate comparison in App. C
195
196
197
198
200
Key Hyperparameters in Offline MBRL
201
In order to design effective search space for penalty comparison experiments, need to understand the impact of different hyperparameters on the uncertainty estimation process itself. Furthermore, this analysis will prove useful in understanding what is important when designing these penalties in the first place.
203
205
206
5.1
How Many Models Do We Need?
207
Since we may have a larger compute budget due to zero experience collection in the environment, it may not make sense to copy the existing approach, originally developed for the online case where online runtime may be an issue; for instance, we can choose to train many more ensemble members. Concretely, MBPO (and subsequently MOPO) trains 7 identical probabilistic dynamies models (with
208
209
210
different initializations). Then, when training the policy, it generates trajectories using the top 5
212
models based on validation accuracy, referred to as "Elites" the Evolutionary community [3I]. The reason or justification for this is not discussed in either paper, but it has seemingly been adopted in the wider MBRL setting 42 B3 39.. In this section we seek to understand what the impact of varying this away from the default values has on the performance of the penalties discussed above.
214
216
5.1.1
How Does Penalty Distribution Change with Model Count?
217
We now vary the number of models used in the calculation of the penalties and plot their respective distributions; an illustrative example is shown in Fig. B with full results in App. B The scaling of
5

---

219
the penalties relying on max over sets (i.e., MOPO and MOReL) is most affected as we increase the number of models due to admitting more extreme values, and we observe that the distribution shape of MOPO changes significantly as we admit more models, which we validate in App. C
220
This clearly impacts the ease by which we can tune this hyperparameter, as we have to contend with a changing
222
223
metric distribution along with calibration quality (something we explore in the next section). Finally,
224
we observe that simple ensemble deviation and variance change the least with differing numbers of models, highlighting their ease in tuning; this is clearly a desirable property for designing such
226
metries going forward.
MOPO
MOReL
Ensemble Std.
Ensemble Var.
LOMPO
M2AC
Number of Models
10-
15-
20
0.8
100
300
31
10
1.5
1.0
200000
40
Penalty
Figure 3: Box Plots showing D4RL Medium transferred to Random. We show the IQR limits and the median value denoted by the black vertical line. Green = HalsCheetah, Blue = Hopper.
227
5.1.2
How does Penalty Performance Scale with Model Count?
Empirically, there exists an optimal number of models to use in an ensemble for -base RL
229
Up to now, heuristics have been used to select how many models we use for uncertainty estimation, despite it being possible to use a different number of models for dynamics prediction
30
231
and uncertainty estimation. For instance, MOPO transitions are generated with Elite models, but all 7 models are used to calculate the penalty. In MOReL, 4 models are used for both transitions
233
and penalty prediction. We therefore wish to understand if there is merit to using a larger number of models for uncertainty estimation compared with next state prediction.
HaifCheetah
Hopper
6.5-
Seammn
0.4
0.G
0.2
0.0
15
0.2
0.2
Number Mocele
Numbber Modte
Numbbr Mocelt
Nurbber Modee
MCPO
Ensemblle
Cnaamble
LONPO
M3AC
Figure 4: All Ground Truth tasks aggregated; Left: HalfCheetah; Right:
Hopper
235
We provide a snapshot in Fig. showing the aggregated results on the Ground Truth data, with full results in App. [ We there is no clear consensus, and that the optimal number of models is highly dependent on environment, the behavior data, and penalty type, with some settings showing improving calibration with model count and vice-versa. This clearly justifies treating the number of models as a hyperparamter that is important to tune, especially on transfer tasks. Interestingly, we observe that it is possible to simultaneously improve rank (p) correlation, but reduce bivariate (r) correlation, especially with the MOPO penalty. This again suggests that the number of models not
237
238
239
240
242
only affects the quality of the estimation, but also its distributional shape.
243
5.2
The Weight of Uncertainty X
244
To weight penalty against reward, MOPO introduces a parameter that trades off between the two terms. In their paper, the authors sweep over À E {1,5} for each environment. However, the optimal values may lie outside of this region, and furthermore, we have shown this value will need to drastically change to account for using different penalty or even number of models. Clearly, this is a crucial hyperparameter for offline MBRI. that needs to be tuned alongside other hyperparameters of interest.
245
246
248

---

250
5.3
The Rollout Horizon h
251
The horizon h of the rollouts plays a crucial role in offline RL. Longer horizon rollouts increase the likelihood of errors in the transitions (we verify this intuition App. D. but conversely can improve performance when errors are properly managed 19 37 Furthermore, as highlighted in Fig. D errors do not always accumulate during single rollout in the model. Instead, we observe spikes, and note is possible to recover from these to valid states and transitions. It is therefore imperative that a penalty idettifies these spikes over the course of a model rollout and flonn-wiigt the reward accordingly.
252
253
254
255
256
257
Using this observation, we propose a novel experiment that treats these spikes as 'positive' labels, and normalize each metric [0, 1j. This converts each penalty into a probabilistic classifier, and we evaluate how well they classify OOD events that occur increasingly under longer h. This is precisely the intuition behind the MOReL and M2AC approaches, whereby the penalty acts as an 'anomaly' detector, removing detrimental transitions that lie above a threshold. The analysis in this section can
258
259
260
281
262
also be viewed as assessing the efficacy of penalties under these schemes, where binary detection is more important than correlation. Finally, we assess two ground truth errors: the dynamics discrepancy
263
264
(as before), and also introduce the distance from the offline distribution trained on, which we measure
265
as the 2-norm between a state-action tuple and its nearest point in the offline data; these are called
286
'Dynamics' and 'Distribution' respectively. We provide precision- recall curves and more details on this experiment in App. D and
Table 2: Performance of different penalties as OOD event detectors averaged over all datasets in Hopper and HalfChectah. AUC is 'Area Under Curve' and AP is 'Average Precision' (higher is better for both).
Percentile
90th
95th
99th
Dynamics
Distribution
Dynamics
Distribution
Dynamics
Distribution
Penalty
AUC
AP
AUC
AP
AUC
AP
AUC
AP
AUC
AP
AUC
AP
MOPO
0.886
0,503
0.759
0.345
0.893
0.351
0.800
0.273
0,921
0,200
0.885
0.157
MOReL
0.897
0.537
0.774
0.343
0.905
0.403
0.814
0.279
0,931
0,260
0.886
0.148
Ensemble Std.
0.902
0.551
0.794
0.378
0.907
0.401
0.834
0.309
0.929
0.251
0.904
0.177
Ensemble Var.
0.903
0.559
0.777
0.352
0,910
0.419
0.817
0.287
0.933
0.270
0.891
0.158
LOMPO
0.662
0.328
0.735
0.326
0,673
0,211
0.760
0.250
0.731
0.088
0.805
0.111
M2AC
0.585
0,206
0.676
0,235
0.597
0.115
0.696
0.140
0.650
0.039
0.717
0.048
266 We observe that the penalties are powerful at identifying dynamics discrepancy, but not as accurate at identifying when the world-model data is out-of -o-mann with respect to the offline data. This is a well known phenomenon in deep neural networks and has been recently investigated in terms of feature collapse 45), where latent representations of points far away in the input space get mapped close together. On the other hand, this shows an important distinction between the regularization induced by MBRL uncertainty and explicit state-action regularization in model-free approaches, such as 47 23). In the latter approaches, policies are penalized for taking out of distribution actions w.r.t. the offline dataset, but this is not always the case with policies trained under MBRL. and uncertainty penalties. The success of MBRL methods in RL may therefore lie in the generation of state-action samples that are OOD but represent accurate dynamics, thus facilitating dynamics generalization in policies; recent work has shown that specifically augmenting dynamics without taking into account state-action shift can improve offline RL policy generalization OOD 4 We believe future work understanding the implications of this property is vitally important.
270
272
273
274
275
278
277
278
279
280
5,4 Implementation Details
282
The above discussion captures many of the key hyperparameters specific to current offline MBRL algorithms. However, there are significant code-level implementation details which are often critical for strong performance and make it hard to disambiguate between algorithmic and implementation improvements. Worryingly, many of these details are not mentioned in their respective papers, or are different between the authors' code and paper. We detail clear examples of this in App. F We believe further investigation of these code-level implementation details represents important future work,
283
284
285
286
287
as has already been done for policy gradients 12 M.. Indeed - it is unclear if the improvement of MOReL over MOPO is due to its P-MDP formulation or it is successful spite of this formulation,
289
due to superior policy optimizer or dynamics model design. We believe that this paper takes
7

---

291
significant first step in tackling this issue by directly comparing a number of proposed penalties along with other important implementation factors and understanding their individual impact.
292
Testing the Limits of Current Approaches
293
In this section we seek to answer the following question: how well can existing methods perform, given optimal selection of the discussed hyperparameters? To answer this question, we use state- of-the-art Gaussian Process- Bayesian Optimization (GP-BO) algorithm, CASMOPOLITAN [6. and tune the configuration for each individual environment. Each BO iteration is run for 300 epochs on a single seed. CASMOPOLITAN uses tailored kernels and trust regions to handle mixed categorical and continuous hyperparameter search spaces. The hyperparameters are listed in App. G We define our search space over:
235
297
298
301
Penalty type (categorical); taking values over {MOPO, MOReL, LOMPO, M2AC, Ensemble Std, Ensemble Variance].
302
Penalty scale X (continuous): taking values over [1, 100).
304
h (integer): taking values over {1, .........,,.
305
Models (integer): taking values over (I,2,...,15], 15).
Our implementation mimics MOPO in that we use the same probabilistic dynamics models (with unchanged hyperparameters) and policy optimizer (SAC, (17), which differs from MORel. which uses Natural Policy Gradient (20). The focus of our experiment is explore parameters relating to uncertainty quantification, and we helieve this is sufficiently fair set up.
307
308
309
Table 3 shows the optimal discovered hyperparameters, We note that the only penalties chosen are the MOPO and ensemble penalties, corroborating the findings in our analysis that these are often the most effective. We observe that MOReL is not chosen, likely because ensemble penalties are generally better correlated with true dynamics error, and are easier to tune since their scaling changes less with increasing model number; also observe that MOReL has very similar shape statistics to Ensemble Std. (App. C
311
313
315
Table 3: Optimal discovered hyperparameters using BO
Discovered Hyperparameters
Environment
N
h
Penalty
random
10
6.64
Ensemble Std
mixed
11
0.96
Ensemble Variance
HalfCheetuh
medium
12
5.92
Ensemble Variance
medium-expert
7
4.56
MOPO
random
6
4.46
47
Ensemble Std
mixed
7
5.90
MOPO
Hopper
medium
7
20.03
31
Ensemble Std
medium-expert
12
39.08
43
MOPO
316
The selection of MOPO also explainable; we observe it displays significantly lower skew and kurtosis than all other metrics (App. ( whilst still maintaining competitive rank correlation. We also found that in all Hopper experiments, Ensemble Var. never achieved high performance, and its only major difference to Ensemble Std. lies in its distributional shape. Interestingly. in HalfCheetah, the opposite is true, with Ensemble Var. delivering significant performance gains. This implies that distributional shape may play as important role as overall calibration, and advocates for the learning of meta- peraamerrssssess that control for these.
317
318
320
322
323
We note that values of the rollout horizon h and penalty weight À differ greatly from those chosen in the original MOPO paper, which chooses from (1,5). Notably, the Hopper environments prefer much longer rollout length and higher penalty weight, even accounting for the magnitude of the penalty used. Again this is backed up by our analysis: along a single rollout dynamics errors do not necessarily accumulate, they simply become more likely to occur. As long as we penalize the aforementioned spikes appropriately, we can handle longer rollouts, and generate more on-policy data. The number of models used to compute the uncertainty estimates can also differ greatly from the standard 7. This again aligns with our findings that using more models for uncertainty estimation can be beneficial, but is dependent on environment, data, and penalty.
324
325
326
327
328
330
8

---

Table 4: Comparative evaluation on the D4RL benchmark suite against other model-based RL algorithms. The raw score for Optimized (Ours) and MOPO (Ours) was taken be the average over the last 10 iterations of policy learning averaged over 4 seeds. Results MOPO and COMBO were taken from the COMBO paper.
Results for MOReL were taken from its paper.
Environment
Optimized (Ours)
MOPO (ours)
MOPO (authors)
MOReL
COMBO
random
31.7
32.7
35.4
25.6
38.8
mixed
58.0
52.8
53.1
40.2
55.1
HalfCheetah
medium
45.7
46.5
42.3
42.1
54.2
mediurn-xxpet
104.2
67.6
63.3
53.3
90.0
random
12.1
4.2
11.7
53.6
17.8
mixed
90.8
66.7
67.5
93.6
73.1
Hopper
medium
46.5
17.3
28.0
95.4
94.9
medium-cxpert
105.8
24.9
23.7
108.7
111.1
Table 4 how these unconventional hyperparameter choices fare against state-of-the-art offline model- based RL algorithms. We include a comparison of our implementation of MOPO v.s. the authors' reported performance using the same hyperparameters. We note the two are relatively similar and thus we are able to make a faithful comparison. Our method, which we label as "Optimized (Ours)", is state-of-the-arr on the Halfchectah mixed and Halfcheetah medium-expert environments by a strong margin. Further notable results include the hopper mixed and hopper medium-exper environments which show
335
335
337
able to tune MOPO-like method up to the performance of COMBO and MOReL. The importance of good uncertainty quantification and hyperparameter selection for MOPO is illustrated in Fig. where we show can improve MOPO performance by over 5x whilst obtaining a stable solution.
339
341
I Limitations of our work include the fact that we solely performed BO over the hyperparameters which directly had an influence on uncertainty quantification. Other hyperparameters which have a sig- nificant general impact on MBRL performance include the number of Elites and the model training hyperparameters [50] (i.e., learning rate, weight decay). Each BO iteration evaluated a hyperparameter setting single seed which could introduce stochasticity; we do
Hopper Med-Exp Comparison
343
Normalizea Score
345
NGPO (uuthors)
347
349
however expect the Gaussian Process surrogate model to account for this aleatoric uncertainty. We also note that individually fine-tuning hy perparameters for each environment is not tractable; due to this
Figure 5: Comparison of MOPO performance on the Hop- per medium-expert environment.
Orfline Epochs
1p0
350
351
352
we only performed BO over 2 environment types in the D4RL suite. However, the same method could be used to find an optimal single configuration for all environments. We also use true environment reward as BO feedback, whereas in reality we may be forced to use offline/off policy evaluation (OPE) p 5
353
354
355
However we do note that our solutions can be more stable over policy training iterations than previous works, and we believe that metries useful for
356
357
training will also be useful for direct method OPE.
The primary goal of our work is to improve understanding of existing methods, the majority of which we believe will be used for good. Indeed, offline RL promises to be beneficial in variety of weal-world settings, such healthcare (40) and robotics [1I. However, we note that it is of course possible our findings aid those looking into applying these methods for malicious use.
359
360
361
Conclusion
362
In this paper, we rigorously evaluated the impact of various key design choices on offline MBRL, comparing for the first time a number of different uncertainty penalties used the literature. By proposing novel evaluation protocols, we have also gained key insights into the nature of uncertainty in offline MBRL that we believe will be of benefit to the wider RL community. We demonstrated the impact of this analysis by improving upon existing offline MBRL methods in performance with significant changes to key hyperparameters compared to prior work, obtaining significantly improved performance in almost all benchmarks.
364
386
368
370
Going forward, we are particularly excited by developments in offline/off-policy evaluation [1] to facilitate accurate assessment of agent performance without querying the environment. This would then open the door for population-based training methods 18 38) which have shown great success in online MBRI [50] Furthermore, throughout the paper we have highlighted potential areas of interest, from better understanding the role of implementation details, through to the development of
371
372
374
meta- meta-paramett parameters controlling penalty distribution shape attributes.

---

References
378
377 [1] M. Andrychowicz, A. Raichuk, P. Stanczyk, M. Orsini, S. Girgin, R. Marinier, L. Hussenot, M. Geist, O. Pietquin, M. Michalski, Gelly, and O Bachem. What matters for on-policy deep actor-critic methods? a large-scale study. In International Conference on Learning Representations, 2021.
378
379
380
[2] A. Argenson and G. Dulac-Amold. Model-based offline planning. In International Conference on Learning Representations, 2021.
382
383
[3] P. Ball, J. Holder, A. Pacchiano, K. Choromanski, and S. Roberts. Ready policy one: World building through active learning. In Proceedings of the 37th International Conference on Machine Learning, /CML. 2020.
384
386
[4] P. J. Ball, C. Lu, J. Parker-Holder, and S. J. Roberts. Augmented world models facilitate zero-shot dynamics generalization from a single offline environment. In The International Conference on Machine Learning, 2021.
388
[5] C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural network. In F. Bach and D. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1613-1622, Lille, France, 07-09 Jul 2015. PMLR.
390
392
[6] J. Buckman, D. Hafner, G. Tucker, E. Brevdo, and H. Lee. Sample-efficient reinforcement learn- ing with stochastic ensemble value expansion. In Advances in Neural Information Processing Systems. 07 2018
394
395
396
[7] Y. Chen, L. Xu, C. Gulcehre, T. L. Paine, A. Gretton, N. de Freitas, and A. Doucet. On instrumental variable regression for deep offline policy evaluation, 2021.
397
398
[8] K. Chua, R. Calandra, R. McAllister and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems 31, pages 4754 4765. 2018.
389
400
401
[9] A. I. -ovensRivers, Palenicek, V. Moens, Abdullah, A. Sootla, J. Wang, and H. Ammar. Samba: Safe model-based & active reinforcement learning, 2020.
402
403
[10] M. P. Deisenroth and C. E. Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on International Conference on Machine Learning, page 465 2011.
405
[11] F. Ebert, C. Finn, S. Dasari, A. Xie, A. Lee, and S. Levine. Visual foresight: Model-based deep reinforcement learning for rision-based robotic control, 2018.
407
[12] L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, F. Janoos, L. Rudolph, and A. Madry. Imple- mentation matters in deep rl: A case study on ppo and trpo. In International Conference on Learning Representations, 2020.
409
410
411
[13] D. Ernst, P. Geurts, and L. Wehenkel. Tree-based batch mode reinforcement learning, Journal of Machine Learning Research, 6(18):003-556, 2005.
412
413
[14] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4(rl): Datasets for deep data-driven
414
reinforcement learning, 2021.
415
[15] J. Fu. M. Norouzi, O. Nachum, G. Tucker, ziyu wang, A. Novikov, M. Yang, M. R. Zhang, Y. Chen, A. Kumar, C. Paduraru, S. Levine, and T. Paine. Benchmarks for deep off-policy evaluation. In International Conference on Learning Representations, 2021.
416
417
418
[16] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1321-1330. PMLR, 06-11 Aug 2017.
420
10

---

422
[17] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, Abbeel, and S. Levine. Soft actor-critic algorithms and applications. CoRR, abs/18 1as.05982. 2018.
423
425
[18) M. Jaderberg, V. Dalibard, S. Osindero, W. M. Czarnecki, J. Donahue, A. Razavi, O. Vinyals, T. Green, I. Dunning, K. Simonyan, C. Fermando, and K. Kavukcuoglu. Population based training of neural networks, 2017.
427
428
[19] M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems. 2019.
429
430
[20] S. M. Kakade. A natural policy gradient. In T. Dietterich, S. Becker, and Z. Ghahramani, editors, Advances in Neural Information Processing Systems, volume 14. MIT Press, 2002.
432
[21] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel Model-based offline reinforcement learning. In Advances in Neural Information Processing Systems. 2020.
433
[22] V. Kuleshov, N. Fenner, and S. Ermon. Accurate uncertainties for deep learning using calibrated regression. J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2796-2804. PMLR, 10-15 Jul 2018.
435
436
437
438
[23] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems. 2020.
440
[24] T. Kurutach, I. Clavera, Y. Duan, A. Tamar, and P. Abbeel. Model-ensemble trust-region policy optimization. In International Conference on Learning Representations, 2018.
442
[25] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, page 6405 061666616, Red Hook, NY, USA, 2017. Curran Associates Inc.
443
444
445
[26] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020.
447
448
[27] D. J. C. Mackay. Bayesian Methods for Adaptive Models. PhD thesis, USA, 1992. UMI Order No. GAX92-32200.
450
[28] W. J. Maddox, P. Izmailov, T. Garipov, D. P. Vetrov, and A. G. Wilson. A simple baseline for bayesian uncertainty in deep learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, F. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
452
454
[29] T. Mandel, Y.-F. Liu, E. Brunskill, and Z. Popović. Offlin evaluation of online reinforcement learning algorithms. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI 16, page 1926-1933. AAAI Press, 2016.
455
457
[30] T. Matsushima, H. Furuta, Y. Matsuo, O. Nachum, and S. Gu. Deployment-efficien reinforce- ment learning via model- offline optimization. In International Conference on Learning Representations, 2021.
458
459
460
[31] J.BB. Mouret and J. Clune. Illuminating search spaces by mapping elites, 2015.
[32] D. A. Nix and A. S. Weigend. Estimating the mean and variance of the target probability distribution. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94), volume I, pages 55-60 vol.1, 1994.
462
463
[33] M. Omer, R. Ahmed, B. Rosman, and S. F. Babikir. Model predictive-actor critic reinforcement learning for dexterous manipulation. In 2020 International Conference on Computer, Control, Electrical, and Electronics Engineering (ICCCEEE), pages 1-6, 2021.
465
11

---

487
[34] Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. Dillon, B. Lakshminarayanan, and J. Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, H. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
488
470
[35] A. Pacchiano, P. Ball, J. Parker-Holder, K. Choromanski, and S. Roberts. On optimism in model-based reinforcement learning, 2020.
473
[36] T. L.. Paine, C. Paduraru, A. Michi, Ç. Gülçehre, K. Zolna, A. Novikov, Z. Wang. and N. de Fre- itas. Hyperparameter selection for offline reinforcement learning- CoRR, abs/2007 0.00550 2020.
475
477
[37] Pan, J. He, D. Tu, and Q. He. Trust the model when it is confident: Masked model-based actor-critic. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 10537-10546. Curran Associates, Inc., 2020.
478
479
480
[38] J. Parker-Holder, V. Nguyen, and S. J. Roberts. Provably efficient online hyperparameter optimization with population-baaser bandits. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 17211-17211. Curran Associates, Inc., 2020.
482
484
485
[39] L. Pineda, B. Amos, A. Zhang, N. O. Lambert, and R. Calandra. Mbrl-lib: A modular library for model-based reinforcement learning. Arxiv, 2021.
486
487
[40] N. Prasad, L. Cheng, C. Chivers, M. Draugelis, and B. E. Engelhardt. reinforcement learning approach to weaning of mechanical ventilation in intensive care units. In G. Elidan, K. Kersting, and A. T. Ihler, editors, Proceedings of the Thirty- Third Conference on Uncertainty in Artificial Intelligence, UAI 2017, Sydney, Australia, August 11-15, 2017. AUAI Press, 2017.
489
491
[41] R. Rafailov, T. Yu, A. Rajeswaran, and C. Finn. Offline reinforcement learning from images with latent space models. In Offline Reinforcement Learning Workshop at Neural Information Processing Systems, 2020.
492
493
494 [42] J. Shen, H. Zhao, W. Zhang, and Y. Yu. Model-based policy optimization with unsupervised model adaptation. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 2823-2834. Curran Associates, Inc., 2020.
486
487
498
[43] R. Sutton. Bull., 2(4): July 1991.
Dyna, an integrated architecture for learning, planning, and reacting. SIGART
[44] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033, 2012.
501
502
[45] J. Van Amersfoort, L. Smith, Y. W. Teh, and Y. Gal. Uncertainty estimation using a single deep deterministic neural network. In H. D. IIII and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 9690-9700. PMLR, 13-18 Jul 2020.
503
505
506
[46] X. Wan, V. Nguyen, H. Ha, B. Ru, C. Lu, and M. A. Osborne. Think global and act local: Bayesian optimisation over high-dimensional categorical and mixed search spaces. In The International Conference on Machine Learning, 2021.
507
508
[47] Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning. In To Appear: The International Conference on Learning Representations (ICLR). 2021.
510
511
[48] T. Yu, A. Kumar, R. Rafailov, A. Rajeswaran, S. Levine, and C. Finn. Combo: Conservative
512
offline model-based policy optimization, 2021.
513
[49] Yu, G. Thomas, I.. Yu, S. Ermon, Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optimization. In Advances in Neural Information Processing Systems. 2020.
12

---

515
[50]
B. Zhang, R. Rajan, L. Pineda, N. Lambert, A. Biedenkapp. K. Chua, F. Hutter, and R. Calandra.
516
On the importance of hyperparameter optimization for model-based reinforcement learning. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, 2021.
13

---

518
1. For all authors...
519
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The paper seeks to investigate the claims introduced in the introduction
520
521
522
(b) Did you describe the limitations of your work?[Yes, Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
524
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
525
526
2. If you are including theoretical results...
527
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
628
(b) Did you include complete proofs of all theoretical results? [N/A]
529
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] supplementary material.
531
532
533
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] In Appendix G
534
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
536
537
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] In Appendix G
538
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
540
(a) If your work uses existing assets, did you cite the creators? Yes, D4RL.
(b) Did you mention the license of the assets? [Yes] In Appendix
542
(c) Did you include any new assets either in the supplemental material or as URL? [No]
543
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
546
547
5. If you used crowdsoureing or conducted research with human subjects...
548
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
549
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
551
552
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "rPUnjRSCIAt", "reviewer": "Reviewer_mit8", "summary": "Offline RL has seen an explosion of interest in the past year or so, with model-based RL methods being the state of the art (or close to it) on commonly studied benchmark tasks (like D4RL and RL Unplugged). This paper takes a retrospective view on model-based offline RL to benchmark/evaluate different design choices in recent papers. Based on experimental evaluations, the paper makes a number of interesting observations and suggestions for improvements such as longer rollout horizons and penalties more closer to OOD detection rather than uncertainty estimation, among others.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- Important domain for retrospective study considering broad recent interest.\n- Very interesting choice of research questions and design methodology for studying them.\n\n**Weaknesses/Limitations**\n- While the questions studied are interesting, I find that many major design factors were not studied. In particular, the choice of the policy optimization between MOReL (NPG/TRPO) and MOPO (SAC) are different. In my mind, this is a major point of departure between the methods and can explain much of the differences -- NPG/TRPO in general works well on Hopper environments, while SAC substantially outperforms it in HalfCheetah. We see the same trend reflected in MOReL vs MOPO too.\n- MOReL also has a penalty term when truncating rollouts due to OOD detection. I don't see this as part of the implementation or study in Sections 5 and 6. Can the authors clarify?\n- I disagree with lines 306-309 in Section 6. In my opinion the policy optimization algorithm and uncertainty estimation are inseparably intertwined. More concretely, in MOReL, the policy is initialized with behavior cloning and updated by taking small NPG/TRPO steps that ensures the policy does not deviate too much. Thus, the space over which the policy searches may not stray too far from the data distribution by design enabling certain types of penalty schemes to work better than others. MOReL also implements much of the final suggestions in the paper -- it uses a much longer rollout horizon of 500-1000, uses discrepancy more for OOD detection than uncertainty estimation etc. By making fundamental design changes to how MOReL is actually implemented, the retrospective study is unfair in its comparisons.\n- Section 6 appears to be a bit misleading. Ensemble std/var was not used in any prior work, somewhat creating a departure from a pure retrospective study (it mixes up comparing older algorithms with providing a newer one). This leads to a skewed results in favor of other methods due to high correlation between MOReL and ensemble std/var. I would suggest the authors to alter the presentation -- present the results/scores for the best hyperparameters for each penalty category. This will likely provide a more comprehensive and unbiased retrospective. \n- [Minor] The appendix mostly just contains plots and tables without providing much context on the experiments. Adding more text and clearly explaining each figure and table can be helpful.\n\n**Final summary** Overall, the paper studies a very important and timely topic given the explosion of interest in offline RL and proliferation of model-based methods. That said, for me, retrospective papers need to achieve a high bar in terms of being thorough, unbiased, and having a very clean presentation, because it is very hard to write a retrospective paper on another retrospective paper. While some of the limitations are acknowledged in the paper, they are nevertheless limitations and must be addressed (and can be addressed with more time and experiments).\n\nUltimately, in my opinion the shortcomings of the study are more than the strengths, and thus I am unable to recommend acceptance at this stage. I would recommend the authors to incorporate suggestions from my review (and others), expand the set of design choices studied, and improve the submission for a future conference cycle.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There are quite a few limitations, some of which were acknowledged in the paper. I have provided more detailed comments in the main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "jB6ksyN8NpJ", "reviewer": "Reviewer_hd7x", "summary": "The authors introduce several assesment strategy to evaluate risk-sensitive criterions in model-based RL. In particular the authors focus on more recent proposal to incorporate risk into model-based RL and how these criteria interact with other hyper-parameter of model-based RL such as the roll-out horizon.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThere are several things I like about this:\n\n1. The paper addresses a relevant issue: How to incorporate risk in model-based (particularly, offline (or batch) model-based RL is still an open question\n\n2. The experiment make sense given the research question.  This is the first paper I am aware that tries to systematically evaluate these different risk criterions\n\n3. The experiments are through, while only two benchmarks are considered (see my cons) the authors investigate the different risk-sensitive criterions on different, and interesting aspects.\n\nCons:\n1. Some evaluations, such as 4.2  do not appear to be a novel approach. To my knowledge evaluating predicted returns in the model with the ground truth is a standard process in model-based RL.  This is hard to asses, as such evaluation are often done in practice but may have not been formulated in clarity and as a reproducible performance indicator in literature so far.\n\n2. The choice of environments in particular is limited to HalfCheetah and Hopper which make it difficult to asses the universality of the finding. Additionally, these benchmarks to my knowledge do not exhibit any intrinsic stochastic dynamics.  Risk-sensitive criterions that include aleatoric uncertainty may not be suited for these type of problems. If they are, these improvements may be due to heuristic reasons (aleatoric models may be more smooth regressors which generalize better?)\n\n3. Relevant literature is missing. E.g on risk-sensitive criterions [1], on batch (offline) RL [2]  on uncertainty-based risk-sensitive RL [3,4,5,6].\n\n- [1] Garcıa, Javier, and Fernando Fernández. \"A comprehensive survey on safe reinforcement learning.\" Journal of Machine Learning Research 16.1 (2015): 1437-1480.\n- [2] Lange, Sascha, Thomas Gabel, and Martin Riedmiller. \"Batch reinforcement learning.\" Reinforcement learning. Springer, Berlin, Heidelberg, 2012. 45-73.\n- [3] Berkenkamp, Felix, et al. \"Safe model-based reinforcement learning with stability guarantees.\" arXiv preprint arXiv:1705.08551 (2017).\n- [4] Depeweg, Stefan, et al. \"Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning.\" International Conference on Machine Learning. PMLR, 2018.\n- [5] Sun, Yi, Faustino Gomez, and Jürgen Schmidhuber. \"Planning to be surprised: Optimal bayesian exploration in dynamic environments.\" International Conference on Artificial General Intelligence. Springer, Berlin, Heidelberg, 2011.\n- [6] Bagnell, J. Andrew, Andrew Y. Ng, and Jeff G. Schneider. \"Solving uncertain Markov decision processes.\" (2001).\n\n\nOverall, I believe this paper serves as nice addition to the literature, as it develops reproducible performance indicators for offline model-based RL, and connects these to the relevant hyper-parameters of this approach, such as the roll-out horizon. Therefore I am leaning towards acceptance, however,  the limited (and deterministic) choice of benchmarks makes me have doubts about the universality of the findings.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes.", "review_text": "", "extra_scores": {"ethical_concerns": "not applicable", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "K30sxo5leNC", "reviewer": "Reviewer_Zw3Q", "summary": "This paper considers the problem of offline, model-based reinforcement learning, wherein a dynamics model is learned from offline data, and subsequently, simulated trajectories in this learned model are used to train a policy. A major challenge in model-based offline RL is how to handle the inevitable distributional shift that occurs when the learned policy is deployed on the true system. Previous works have proposed to mitigate this issue by considering a pessimistic MDP, wherein the reward function is augmented by a measure of the model mismatch between the estimated and true dynamics to obtain an MDP on a which a policies expected return is a lower bound for its expected return on the true dynamics.\n\nThis paper compares a collection of prior work following this general approach, but differing in hyperparameters such as the formulation of the model mismatch penalty, rollout horizon, and number of models used the ensemble dynamics model. The paper applies Bayesian optimization to offer insights as to which choices are most effective on two test domains, the Hopper and Half-Cheetah.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper addresses an important topic in the model-based RL literature by comparing several related techniques on a common implementation, and performing a suite of experiments to highlight the importance of various hyperparameters common to these algorithms. Work to this end is significant, as it provides empirical insights to inform future algorithm development. Overall, I think this work presents a good set experimental results, but I feel that there are some limitations in the clarity of the message / presentation of the results as well as limitations in the scope of the experiments that hold this paper back.\n\nFirst, while the authors do a good job of covering related work and background, I found that some sections were hard to follow, including the description and discussion of the experiments. This left me with several questions:\n\t- 4.2: How exactly was the degree of exploitation of the policy controlled? The specific generation of this policy should be explained in the main paper.\n\t- 4.2: The name \"ground-truth\" for this experiment is confusing, as I'm not sure how this experiment is any more \"ground-truth\" than the \"transfer\" experiment. \n\t- 5.1.2: The experiment measures the quality of a particular choice in terms of correlation coefficients, but this choice of optimality measure is not well motivated. The authors should either plot MBRL performance with all hyperparameters fixed apart from the number of models, or provide more theoretical justification why higher correlation should lead to better performance.\n\t- 5.3: The section is titled Rollout Horizon, but the bulk of it is spent discussing an interpretation of the dynamics penalty as an OoD detection scheme, investigating the utility of these penalties as binary classifiers. The details of this experiment are unclear, however: what data was used to train the model, and what data was used to evaluate the penalties? (other sampled transitions or the transitions from rollouts of a learned policy?) What does \"percentile\" indicate in Table 2? From the appendix, I see that this corresponds to the error threshold used to deem points as OoD, but this should be explicitly detailed in the body of the paper.\n\t- What are the differences the different experimental domains (\"random\", \"mixed\", etc.)? The authors should describe the experimental domains clearly in the body of the paper.\n\nOverall, the authors should present the experiments with greater clarity, making it clear how each experiment was carried out, motivating the choice of metric for each experiment, and clearly defining terms and notation.\n\nSecond, the experiments were limited, making it hard to understand how well these results would generalize. The authors perform these studies on only two domains, both MuJoCo based environments with dense reward functions. The penalty function as motivated by the PMDP formulation has a key connection to the value function of the MDP, and not just the dynamics, so it could be worthwhile to perform these studies with different reward functions. Furthermore, the Bayesian optimization experiments only show results of performance on the same environment that was used to optimize the hyperparameters. While this answers the question that the authors pose -- \"how well can we do if we choose optimal hyperparameters?\" -- this does not provide any actionable insights into the more practical question of \"what hyperparameters should I use on a new environment?\" Results on how well the optimized parameters transfer between environments would provide more insight and strengthen the paper.\n\nOverall, this paper represents a good step towards the important goal of strong empirical comparisons of recent offline MBRL techniques, but it is hindered by the limitations in exposition and the discussion of the experiments.\n\n\nMinor comments:\n\t- Notation in equation 1 is not defined (what is $eta_M$?). If $eta_M$ corresponds to the expected discounted return, as it does in [49], then this equation gives an upper bound, rather than a lower bound which is written in the text. However, it appears that in [49] the inequality is flipped, so perhaps this is a typo.\n\t- The best performing methods should be bolded in all tables. Confidence intervals should be included for tabular results as well.\n\t- The inline table in sections 4.2  is confusing, as it contains results from the experiment described in section 4.1. This should be a floating element and not inline.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The discussion of societal impacts and limitations is reasonable given the content of this work. The limitations are clearly discussed, and, as this work is focused on experimental comparisons of existing techniques in model-based offline RL, the short discussion of potential negative societal impact was adequate.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall the reviews were thorough and pointed to a number of limitations of the original submission. The authors provided a series of responses, which included a significantly larger set of experimental results, adjustments to the paper text including fundamental adjustments to make the paper focus clear, and promises of additional paper adjustments. The reviewers and AC took all of this into account. 

In the end, we are looking at a very different paper than what was originally submitted. It is likely a nice improvement, which is the intent of the review process. However, the delta between the original and new content is quite large and arguably points to the fact that a new submission is needed. Indeed, there are many new experiments that have been produced in a very compressed amount of time (this even includes new experimental domains). It seems most appropriate that these be used as the basis for an upcoming submission, where more time can be spent verifying and analyzing the results. 
 
Decision: Reject