Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper provides detailed analysis of different uncertainty quantifications in Model-based offline RL, from both statistical and empirical perspectives. Further, the paper performs Bayesian optimization to find hyper-parameters and the optimized hyper-parameters perform better empirically. 

main_review: I think the research is well-motivated and interesting, while I have some concerns about the experiments. I give the comments in the following.

Pros:

1. The research is well-motivated.  The gap between the true model and the learned model is estimated by uncertainty measurements. Nevertheless, in the MOPO and Morel papers, different uncertainties are not rigorously studied. I think this problem is important and should be studied in-depth.

2. The use of Spearman rank and Pearson bivariate to identify the correlation between uncertainty and true MSE is novel. The experiments verify such measurements are closely related to the performance.

Cons:

1. The experiments in section 4.2 are a bit confusing. Why should we use exploitative policies that are generated by non-penalty methods to generate trajectories? The non-penalty methods cause overestimation and poor performance, and the resulting policy may be similar to a random policy. The trajectories generated by such policies may lie in a small range of state-action space. I suggest using the trajectories from the medium-replay dataset to evaluate since it covers a large state-action space. You can point it out if I don't understand correctly.

2. The aim of section 5.3 is to study the effect of rollout horizon. It is unclear how the results presented in table 2 relate back to the rollout horizon, since I do not find the rollout horizon is used as a variable in Table 2. The dynamics discrepancy should be related to the horizon length. Can you explain more? 

3. In Table 3, the results mix the D4RL v0 and v2 datasets. It is not valid since these two datasets are quite different. The v2 dataset is much larger than the v0 dataset. I suggest unifying all results in a single dataset. Maybe you can refer to other offline RL papers released recently to get the scores of MOPO and Morel in v2 dataset.

4. Considering different tasks have different optimal hyper-parameters, I suggest adding additional analysis about the sensitiveness of N, lambda, and h in Table 3. E.g. Should lambda be larger in single-policy datasets and be smaller in mixed datasets?  Does a larger N perform better? Does N affect the performance significantly if N>10? Maybe h is sensitive to different tasks?

5. The author should explain why “Ensemble Standard Deviation” measures both the epistemic uncertainty and aleatory uncertainty. The paper should be self-contained. What about other methods considered in section 4 (in measuring the epistemic uncertainty and aleatory uncertainty)? Can we draw the conclusion that both the aleatoric and epistemic are important to offline RL?

Minor:

1. The method to obtain true dynamics error should be given in discussing Figure.1.  Although it has been discussed later in the paper.

2. At the end of section 5.1, the paper writes “This again suggests that the number of models not only affects the quality of the estimation, but also its distributional shape. ” Can you explain more?
summary_of_the_review: The paper is well motivated while it needs to be improved in clarity. (1) The dataset used in evaluation should be unified. (2) The experimental setup in Table 1 and Table 2 should be explained more clearly. (3) Additional ablation or explanation of Table 3 is suggested to be added. (4) Some discussion about the epistemic and aleatoric uncertainty.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present an empirical study of several uncertainty quantification heuristics applied to model learning in offline model-based reinforcement learning. Specifically, they consider the basic architecture of MOPO, in which an uncertainty based state-action penalty function is applied on top of the standard reward to construct a pessimistic MDP. Within this set up, the authors perform a empirical study of several different uncertainty penalties, exploring their correlation to the prediction error of the model, as well as in terms of their ability to detect individual transitions with high-percentile prediction errors. Finally, the authors perform Bayesian optimization over the choice of the uncertainty penalty as well as other hyperparameters such as number of ensemble elements, planning horizon, penalty weights, etc. They present these results showing that the optimal choice of penalty and penalty weight can vary significantly, not only between environments, but also within an environment as a function of offline dataset.
main_review: Overall, this work is well written and presents a large range of empirical results on the utility of different uncertainty heuristics. The wide scope of these experiments, evaluating the uncertainty penalties in relation to both model prediction error and downstream MBRL performance is a strength of the approach. The approach highlights that the predictive variance or standard deviation of the ensemble predictions can perform well as an uncertainty penalty, even though past work in offline MBRL has not considered this. In addition, it highlights the important role that hyperparameters play in controlling the performance of the MOPO algorithm.

My main concern with this work is that does not make any actionable claims. They claim that they achieve SOTA results by choosing different hyperparameters than previous work, but this hyperparameter selection is chosen via Bayesian optimization that is specific to the same environment. To me, this is the equivalent of "training on the test set," in that it uses interaction with the true system to select hyperparameters for the same system. In that sense, the result can no longer be called offline RL. Of course, hyperparameter selection is an under addressed topic in the community, and there are many papers which manually select hyperparameters that yield good performance. The fact that explicitly optimizing these hyperparameters through interaction on the test environment can yield better performance is not very surprising, but also not very useful for the target applications of offline RL where interaction with the true system is expensive or safety-critical. Indeed, these results do not even paint a consistent picture where one uncertainty penalty is clearly preferred to the others. The work would be significantly improved if the authors used the results of their empirical experimentation to propose a general recommendation for penalty selection and hyperparameter selection. 

Other comments:
- Experiments are limited to deterministic environments where the distinction between aleatoric and epistemic uncertainty is unimportant. In stochastic environments, I don't see why aleatoric uncertainty should factor into a reward penalty, so the ensemble variance and standard deviation penalties proposed may not work well in such environments.
- The claim that the ensemble standard deviation or variance is "arguably the most principled" choice is not supported. Why is the variance of the ensemble predictions match the total variation distance term? What makes it "more closely aligned with the theory"?
- The AUC / AP are hard to compare across percentile choices because the relative proportion of positive and negative samples is also changing, so the AUC and AP of a random classifier changes across this task. It would be interesting to report these results on a subsample with even proportions of low-error and high-error samples such that the AUC of a random classifier would be 0.5 in all cases. Furthermore, the precision recall curves in the appendix don't make sense to me, as recall should vary all the way from 0 to 1 as the threshold changes. Perhaps the axes are flipped?
- It is unclear what is meant by "latent dynamics that are KL-regularized to a spherical Gaussian" (last paragraph of section 4.2)
summary_of_the_review: Overall, the paper presents a large amount of empirical results, but is lacking a well supported, clear actionable message beyond using longer rollout horizon and ensemble standard deviation as an uncertainty penalty. Even this claim is lacking empirical support, as there is no evaluation of a fixed, guiding principle for hyperparameter selection that is held constant over a wide range of domains. As such, I'm hesitant to recommend acceptance. If the manuscript is updated to have clearer empirical evidence for the main claims in the paper, including a fair evaluation of the recommended strategy across a variety of domains without optimizing hyperparameters separately for each environment, I would increase my score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper provides an evaluation of many of the design choices and hyperparameter decisions made in offline model-based reinforcement learning methods which have emerged recently. Particularly, the empirical study looks at uncertainty penalties used in these methods, as well as hyperparameters such as ensemble size, penalty weighting, and rollout horizon. The authors find that offline MBRL methods are quite sensitive to each of these parameters. They compare the “optimized” version of MOPO with hyperparameters tuned using Bayesian optimization, and find that it leads to statistically significant performance improvements over the version of MOPO presented in the original paper.
main_review: Strong points:


This is a quite thorough evaluation which compares several of the recent popular offline MBRL works, and investigates critical design choices made in each of them. I feel that this is a valuable contribution to research in offline RL, since it’s generally quite difficult to understand where performance improvements between various methods come from due to so many differences in hyperparameters.
The comparison metrics for the uncertainty penalty are quite insightful and reasonable.

Weak points:


The “optimized” version presented in the paper only optimizes and presents statistically significant results compared to MOPO, so it is unclear if similar trends hold for the other methods compared to in the paper and there may not be sufficient evidence to argue that hyperparameter selection affords large improvements for other algorithms.


The “Optimized” numbers are reported by optimizing over each D4RL task. With that, it seems unsurprising that with a single setting and an optimization run over a single seed, the hyperparameters can be tuned better than the original implementations. It seems important to this evaluation to also determine how stable these hyperparameter choices are across random seeds.

Questions/clarifications/nitpicks:


It’s a bit confusing to me to present “optimized” as “our method” in section 6, when the difference (as far as I am aware) is in the hyperparameter selection of MOPO (which I think could be made more clear). 

Additionally, in Figure 1, “Optimized” has not yet been introduced, so while in hindsight it is clear that it’s an optimized version of MOPO, it may not be immediately obvious.

Figure 1(b) and the methodology used in this claim is not clearly described in the text

summary_of_the_review: I recommend accepting this paper. I think that more rigorous evaluations of the design and choices made in RL and particularly offline RL methods are important contributions to help drive future research. I think the analysis of the uncertainty quantification methods is particularly interesting.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Model-based offline reinforcement learning algorithms typically involve constructing a pessimistic MDP, which is implemented based on an uncertainty estimation of the learned model. This paper conducts empirical analysis to compare different design choices of the uncertainty estimation in practice. In more details, the authors compare different approaches in terms of the correlation between the estimated uncertainty and ground truth model error. They also use bayesian optimization to search the best hyperparameter configuration that achieves strong empirical performance. 
main_review: The main contribution of this paper is to systematically compare several uncertainty estimate methods. Perhaps surprisingly, such comparison has not been conducted before, even in those papers that originally proposes to implement a pessimistic MDP. To this end, this paper fills an important gap in the literature. The paper is also well written. The experiment setups are clearly defined. 

However, my major concern is that the technical contribution of this paper is not strong enough. Other than designing several interesting experiments to compare model uncertainty estimation methods, the paper does not contribute enough new insights that can potentially be interesting for a broader research community. 

Furthermore, I think a very related paper is missing (Abbas et al., 2020). This paper also compare several uncertainty estimation methods in model based RL, including the ensemble based approaches (Lakshminarayanan et al., 2017). The main conclusion is very similar: a combination of epistemic and aleatoric uncertainty is the best choice. 

Abbas, Zaheer, et al. "Selective Dyna-style planning under limited model capacity." International Conference on Machine Learning. PMLR, 2020.


summary_of_the_review: I recommend rejecting this paper because the technical contribution is not strong enough.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Recent successful model-based offline RL techniques have relied on heuristics to penalize rewards according to the uncertainty of the estimated MDP. This paper reviews the different penalties that have been designed in the literature. The impact and importance of associated hyperparameters such as the planning horizon and the number of models in the ensemble are also evaluated. The author show that the selection of the best penalty and best hyperparameters lead to stronger performance.
main_review: Strengths
- the paper is well written and overall easy to read and understand.
- such work, summarizing different contributions and comparing the different choices that are made in each contribution, are important for the research community as well as practitioners
- I also like the different messages and perspectives that are made in the paper
- The related work seems rather complete for model-based offline RL.

Weaknesses
- I would need a better explanation of Figure 1a and section 5.3. (see detailed comments below).
- for such an empirical work error bars are needed in Table 1, 2, 3, 4 to judge the statistical significance of the results. Table 3 identifies algorithms that are significantly better but it would still be nice to have corresponding std or error bars for the different performances.
- What about existing calibration metrids such as the expected calibration error (ECE)?

Other comments:
- Figure 1:
    * the figure should be better documented, even if this has to be in the appendix. A precise definition of the y-axis of FIgure 1a should be given and a precise description of how this figure was computed so that someone can reproduce it. What's the environment? Is the error and uncertainty the ones for the reward or another dynamics variable (or the mean error over all the dynamics state variables)
    * I would especially like the authors to clarify whether the errors are computed step by step comparing the outputs f(s,a) (obtained with the true dynamics) and g(s,a) (obtained with the model) assuming same inputs (s,a) or for multi-step predictions i.e. considering a fixed sequence of actions and a first initial state s_0 look at the error between each state of the sequence (s_1, ..., s_t) predicted by a rollout with the model and its corresponding state in the sequence (s'_1, ..., s'_t) obtained for the same action sequence with the true dynamics when starting from the same initial state s_0. Is it what is being computed in Figure 1a and in Figure 27? What is the difference between figure 26 and figure 27: Figure 26 is just the median overall several rollouts whereas figure 26 shows individual rollout?
    * I think this is a bit more clear in appendix D, but I would still like the authors to make this clear in their replies and to make the effort of making this more clear in the main paper as well.
    * Appendix D: Can the authors elaborate on the "displacement" variable? Is this a variable needed when setting the state of the Mujoco Simulator?
    * for Figure 1b: the authors should clarify the definition of the probability of improvement and how this is computed
- For figure 1 and table 1: what if the error used is the log likelihood instead of the MSE?
- Can the authors confirm that the hyperparameter optimization is done by evaluating the performance of the different hyperparameters on the real environment? My comment here is that this is impossible in practice. I would not let this influence my review because I think this is the current practice for most papers in RL.
- Table 2: "the penalties are powerful at identifying dynamics discrepancy but not as accurate at identifying when the world-model data is out-of-distribution" -> it seems that AUCs of 0.9 are reachable which is not too bad in my opinion?

Minor Comments
- Figure 1a legend: during single a model rollout -> during a single model rollout?
- Introduction: "Furthermore, using the insights gained from this section" -> it is not clear at all what "this section" is refering to here.
- Section 3. reward notation "r" is also used for the Pearson bivariate correlation.
- equation (1) -> I think a capital "R" should be used for the reward function.
- I should maybe have a look at the paper by Yu et al, 2020 but how do they justify the use the aleatoric uncertainty as a penalty? Aleatoric uncertainty is specific to the system and even the true dynamics have an aleatoric uncertainty. I would expect better results by considering epistemic uncertainty which is what is attempted when using the ensembles.
summary_of_the_review: I like the paper and the work done by the authors. I think it is valuable for the community (researchers and practitioners). I would just like to have some clarifications about the points I raised in my review. I will change my score accordingly.
review_title: null

Meta-review: This paper empirically studies various design choices in offline model-based RL algorithms, with a focus on MOPO (Model-based Offline Policy Optimization). Among the key design choices is the uncertainty measure used in MOPO that provides an (approximate) lower bound on the performance, the horizon rollout length, and the number of model used in ensemble.

The reviewers are positive about the paper, found the experiments thorough, and the results filling a gap in the current literature. They have raised several issues in their reviews, many of which are addressed in the rebuttal and the revised paper. I would like to recommend acceptance of the paper. Also since the results of this work might be of interest to many researchers working on model-based RL, I also recommend a spotlight presentation for this work.

I have some additional comments:

(1) The paper studies the correlation of uncertainty measures with the next-state MSE, with the aim of showing which one has a higher correlation. The underlying assumption is that the next-state MSE is the gold standard that we should aim for.

If we go back to the MOPO paper, we see that to define an uncertainty-penalized reward, we need an upper bound on the absolute value of G(s, a), which is the difference between the expected value of the value function at the next-state according to the true model and the estimated model.

If we assume that the value function belongs to the Lipschitz function class w.r.t. a metric d, this upper bound is proportional to the 1-Wasserstein distance between the true next-state distributions and the model's distribution.
If the dynamics is deterministic, 1-Wasserstein distance becomes the $d( T(s, a), \hat{T}(s,a) )$. If the distance d is the Euclidean distance, this becomes the squared error.

Therefore, the squared error makes sense for deterministic dynamics, and it only provides an upper bound of $|G(s, a)|$.
If the environment is not deterministic, the squared error may not be a reasonable gold standard anymore to compare the correlation of various uncertainty measures with.

The paper introduces a generic MDP framework, but does not mention anything about its focus on MBRL for deterministic environments until the last sentence of its conclusion. Please clarify this in your camera ready paper.

(2) The experiments are conducted using 3 or 4 seeds. Although this is the common practice in the deep RL community, it is too small. Standard deviations in Tables 1, 2, ... are computed with 3 seeds, which would be cringeworthy to statisticians and empirical scientists. I encourage the authors to increase the number of independent random experiments to make their results more powerful.
Decision: Accept (Spotlight)