Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors introduce several assesment strategy to evaluate risk-sensitive criterions in model-based RL. In particular the authors focus on more recent proposal to incorporate risk into model-based RL and how these criteria interact with other hyper-parameter of model-based RL such as the roll-out horizon.
main_review: 
There are several things I like about this:

1. The paper addresses a relevant issue: How to incorporate risk in model-based (particularly, offline (or batch) model-based RL is still an open question

2. The experiment make sense given the research question.  This is the first paper I am aware that tries to systematically evaluate these different risk criterions

3. The experiments are through, while only two benchmarks are considered (see my cons) the authors investigate the different risk-sensitive criterions on different, and interesting aspects.

Cons:
1. Some evaluations, such as 4.2  do not appear to be a novel approach. To my knowledge evaluating predicted returns in the model with the ground truth is a standard process in model-based RL.  This is hard to asses, as such evaluation are often done in practice but may have not been formulated in clarity and as a reproducible performance indicator in literature so far.

2. The choice of environments in particular is limited to HalfCheetah and Hopper which make it difficult to asses the universality of the finding. Additionally, these benchmarks to my knowledge do not exhibit any intrinsic stochastic dynamics.  Risk-sensitive criterions that include aleatoric uncertainty may not be suited for these type of problems. If they are, these improvements may be due to heuristic reasons (aleatoric models may be more smooth regressors which generalize better?)

3. Relevant literature is missing. E.g on risk-sensitive criterions [1], on batch (offline) RL [2]  on uncertainty-based risk-sensitive RL [3,4,5,6].

- [1] Garcıa, Javier, and Fernando Fernández. "A comprehensive survey on safe reinforcement learning." Journal of Machine Learning Research 16.1 (2015): 1437-1480.
- [2] Lange, Sascha, Thomas Gabel, and Martin Riedmiller. "Batch reinforcement learning." Reinforcement learning. Springer, Berlin, Heidelberg, 2012. 45-73.
- [3] Berkenkamp, Felix, et al. "Safe model-based reinforcement learning with stability guarantees." arXiv preprint arXiv:1705.08551 (2017).
- [4] Depeweg, Stefan, et al. "Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning." International Conference on Machine Learning. PMLR, 2018.
- [5] Sun, Yi, Faustino Gomez, and Jürgen Schmidhuber. "Planning to be surprised: Optimal bayesian exploration in dynamic environments." International Conference on Artificial General Intelligence. Springer, Berlin, Heidelberg, 2011.
- [6] Bagnell, J. Andrew, Andrew Y. Ng, and Jeff G. Schneider. "Solving uncertain Markov decision processes." (2001).


Overall, I believe this paper serves as nice addition to the literature, as it develops reproducible performance indicators for offline model-based RL, and connects these to the relevant hyper-parameters of this approach, such as the roll-out horizon. Therefore I am leaning towards acceptance, however,  the limited (and deterministic) choice of benchmarks makes me have doubts about the universality of the findings.

limitations_and_societal_impact: yes.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: Offline RL has seen an explosion of interest in the past year or so, with model-based RL methods being the state of the art (or close to it) on commonly studied benchmark tasks (like D4RL and RL Unplugged). This paper takes a retrospective view on model-based offline RL to benchmark/evaluate different design choices in recent papers. Based on experimental evaluations, the paper makes a number of interesting observations and suggestions for improvements such as longer rollout horizons and penalties more closer to OOD detection rather than uncertainty estimation, among others.
main_review: **Strengths**
- Important domain for retrospective study considering broad recent interest.
- Very interesting choice of research questions and design methodology for studying them.

**Weaknesses/Limitations**
- While the questions studied are interesting, I find that many major design factors were not studied. In particular, the choice of the policy optimization between MOReL (NPG/TRPO) and MOPO (SAC) are different. In my mind, this is a major point of departure between the methods and can explain much of the differences -- NPG/TRPO in general works well on Hopper environments, while SAC substantially outperforms it in HalfCheetah. We see the same trend reflected in MOReL vs MOPO too.
- MOReL also has a penalty term when truncating rollouts due to OOD detection. I don't see this as part of the implementation or study in Sections 5 and 6. Can the authors clarify?
- I disagree with lines 306-309 in Section 6. In my opinion the policy optimization algorithm and uncertainty estimation are inseparably intertwined. More concretely, in MOReL, the policy is initialized with behavior cloning and updated by taking small NPG/TRPO steps that ensures the policy does not deviate too much. Thus, the space over which the policy searches may not stray too far from the data distribution by design enabling certain types of penalty schemes to work better than others. MOReL also implements much of the final suggestions in the paper -- it uses a much longer rollout horizon of 500-1000, uses discrepancy more for OOD detection than uncertainty estimation etc. By making fundamental design changes to how MOReL is actually implemented, the retrospective study is unfair in its comparisons.
- Section 6 appears to be a bit misleading. Ensemble std/var was not used in any prior work, somewhat creating a departure from a pure retrospective study (it mixes up comparing older algorithms with providing a newer one). This leads to a skewed results in favor of other methods due to high correlation between MOReL and ensemble std/var. I would suggest the authors to alter the presentation -- present the results/scores for the best hyperparameters for each penalty category. This will likely provide a more comprehensive and unbiased retrospective. 
- [Minor] The appendix mostly just contains plots and tables without providing much context on the experiments. Adding more text and clearly explaining each figure and table can be helpful.

**Final summary** Overall, the paper studies a very important and timely topic given the explosion of interest in offline RL and proliferation of model-based methods. That said, for me, retrospective papers need to achieve a high bar in terms of being thorough, unbiased, and having a very clean presentation, because it is very hard to write a retrospective paper on another retrospective paper. While some of the limitations are acknowledged in the paper, they are nevertheless limitations and must be addressed (and can be addressed with more time and experiments).

Ultimately, in my opinion the shortcomings of the study are more than the strengths, and thus I am unable to recommend acceptance at this stage. I would recommend the authors to incorporate suggestions from my review (and others), expand the set of design choices studied, and improve the submission for a future conference cycle.
limitations_and_societal_impact: There are quite a few limitations, some of which were acknowledged in the paper. I have provided more detailed comments in the main review.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper considers the problem of offline, model-based reinforcement learning, wherein a dynamics model is learned from offline data, and subsequently, simulated trajectories in this learned model are used to train a policy. A major challenge in model-based offline RL is how to handle the inevitable distributional shift that occurs when the learned policy is deployed on the true system. Previous works have proposed to mitigate this issue by considering a pessimistic MDP, wherein the reward function is augmented by a measure of the model mismatch between the estimated and true dynamics to obtain an MDP on a which a policies expected return is a lower bound for its expected return on the true dynamics.

This paper compares a collection of prior work following this general approach, but differing in hyperparameters such as the formulation of the model mismatch penalty, rollout horizon, and number of models used the ensemble dynamics model. The paper applies Bayesian optimization to offer insights as to which choices are most effective on two test domains, the Hopper and Half-Cheetah.
main_review: This paper addresses an important topic in the model-based RL literature by comparing several related techniques on a common implementation, and performing a suite of experiments to highlight the importance of various hyperparameters common to these algorithms. Work to this end is significant, as it provides empirical insights to inform future algorithm development. Overall, I think this work presents a good set experimental results, but I feel that there are some limitations in the clarity of the message / presentation of the results as well as limitations in the scope of the experiments that hold this paper back.

First, while the authors do a good job of covering related work and background, I found that some sections were hard to follow, including the description and discussion of the experiments. This left me with several questions:
	- 4.2: How exactly was the degree of exploitation of the policy controlled? The specific generation of this policy should be explained in the main paper.
	- 4.2: The name "ground-truth" for this experiment is confusing, as I'm not sure how this experiment is any more "ground-truth" than the "transfer" experiment. 
	- 5.1.2: The experiment measures the quality of a particular choice in terms of correlation coefficients, but this choice of optimality measure is not well motivated. The authors should either plot MBRL performance with all hyperparameters fixed apart from the number of models, or provide more theoretical justification why higher correlation should lead to better performance.
	- 5.3: The section is titled Rollout Horizon, but the bulk of it is spent discussing an interpretation of the dynamics penalty as an OoD detection scheme, investigating the utility of these penalties as binary classifiers. The details of this experiment are unclear, however: what data was used to train the model, and what data was used to evaluate the penalties? (other sampled transitions or the transitions from rollouts of a learned policy?) What does "percentile" indicate in Table 2? From the appendix, I see that this corresponds to the error threshold used to deem points as OoD, but this should be explicitly detailed in the body of the paper.
	- What are the differences the different experimental domains ("random", "mixed", etc.)? The authors should describe the experimental domains clearly in the body of the paper.

Overall, the authors should present the experiments with greater clarity, making it clear how each experiment was carried out, motivating the choice of metric for each experiment, and clearly defining terms and notation.

Second, the experiments were limited, making it hard to understand how well these results would generalize. The authors perform these studies on only two domains, both MuJoCo based environments with dense reward functions. The penalty function as motivated by the PMDP formulation has a key connection to the value function of the MDP, and not just the dynamics, so it could be worthwhile to perform these studies with different reward functions. Furthermore, the Bayesian optimization experiments only show results of performance on the same environment that was used to optimize the hyperparameters. While this answers the question that the authors pose -- "how well can we do if we choose optimal hyperparameters?" -- this does not provide any actionable insights into the more practical question of "what hyperparameters should I use on a new environment?" Results on how well the optimized parameters transfer between environments would provide more insight and strengthen the paper.

Overall, this paper represents a good step towards the important goal of strong empirical comparisons of recent offline MBRL techniques, but it is hindered by the limitations in exposition and the discussion of the experiments.


Minor comments:
	- Notation in equation 1 is not defined (what is $eta_M$?). If $eta_M$ corresponds to the expected discounted return, as it does in [49], then this equation gives an upper bound, rather than a lower bound which is written in the text. However, it appears that in [49] the inequality is flipped, so perhaps this is a typo.
	- The best performing methods should be bolded in all tables. Confidence intervals should be included for tabular results as well.
	- The inline table in sections 4.2  is confusing, as it contains results from the experiment described in section 4.1. This should be a floating element and not inline.
limitations_and_societal_impact: The discussion of societal impacts and limitations is reasonable given the content of this work. The limitations are clearly discussed, and, as this work is focused on experimental comparisons of existing techniques in model-based offline RL, the short discussion of potential negative societal impact was adequate.
review_title: null

Meta-review: Overall the reviews were thorough and pointed to a number of limitations of the original submission. The authors provided a series of responses, which included a significantly larger set of experimental results, adjustments to the paper text including fundamental adjustments to make the paper focus clear, and promises of additional paper adjustments. The reviewers and AC took all of this into account. 

In the end, we are looking at a very different paper than what was originally submitted. It is likely a nice improvement, which is the intent of the review process. However, the delta between the original and new content is quite large and arguably points to the fact that a new submission is needed. Indeed, there are many new experiments that have been produced in a very compressed amount of time (this even includes new experimental domains). It seems most appropriate that these be used as the basis for an upcoming submission, where more time can be spent verifying and analyzing the results. 
 
Decision: Reject