Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper uses a standard off-policy moder-based reinforcement learning algorithm and uses a set of dynamics models (autoregressive models, ensembles & mixture density models) on a single benchmark. 

They then test the algorithms using a set of metrics and found that auto-regressive models appear to give improved performance


strength_and_weaknesses: Strengths:
- the use of auto-reggressive models and its comparison to  standard ensembling methods appears novel

Weaknesses:
- Significance : The methodology the authors present (offline model-based RL with a parameterized policy) is standard. The algorithms for model learning (deep ensembles, auto-regressive models) are known and testing on a single benchmark does not provide any insight into their advantages and disadvantages 
- Limited experiments: The authors say in the conclusion "In this paper, we ask what are the best dynamic system models, estimating their own uncertainty,"  -- but all tests are done on a single benchmark.
- Limited comparisons: There is more work on using uncertainty-aware models in offline model-based RL, such as Gaussian Processes, Bayesian Deep Learning or using Dropout-mechanisms 
clarity,_quality,_novelty_and_reproducibility: The paper is written quite clear The insights from the paper do not provide much novelty compared to previous work, mainly based on the limited evaluation. The comparison between models is interesting, but only done on a single benchmark.
summary_of_the_review: Overall, I cannot recommend acceptance because I unfortunately do not see a clear contribution and significance of the presented work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers using autoregressive models to predict the different state dimensions when learning a transition model. The authors argue that autoregressive models are able to capture correlations between state dimensions and therefore work better than standard neural nets in the context of model-based offline RL. Experiments support these claims.
strength_and_weaknesses: **strengths**

* The paper questions the use of ensembles in certain scenarios. Given the amount of attention that ensembles have received over the last years, the results of this paper are quite interesting.
* The experiments are in favor of the approach.
* Looking at different metrics and their correlation with RL performance is quite helpful and interesting.

**weaknesses**

* The experiments feature a single mujoco environment. This is my main reason for giving the paper a 6 over an 8.
* Autoregressive models are more computationally expensive than single neural nets and ensembles (though this might depend on the implementation).
* The autoregressive models are clearly outperformed when one considers L-step metrics. This would have ramifications about the applicability of the results. In works like PETS [1], predictive performance over a longer horizon is essential. The results in this paper suggest that autoregressive models are not a good fit for that. While I can understand that 1-step performance and L-step performance might not necessarily agree (e.g. one can sacrifice one to improve the other), it would be a little odd that one model category is better in the offline setting, while the other is better in the other.
* Given that the authors' initial pitch for autoregressive models is about correlations between state dimensions, I think the experiments section should check if there is indeed a difference between the two model classes in this regard. Perhaps a toy experiment involving a pendulum, where one could check the same sine-cosine scenarion that was described in the paper.

----

[1] Deep Reinforcement Learning in a Handful of Trials, https://arxiv.org/pdf/1805.12114.pdf
clarity,_quality,_novelty_and_reproducibility: The ideas and arguments are clear. The quality of writing is good. The idea of using the predicted variance of an autoregressive model to penalize the policy is novel as far as I am aware. The authors say they will publish the code for reproducibility.
summary_of_the_review: This paper proposes using autoregressive models for offline RL, arguing that these are better able to capture correlations between state dimensions. Experiments show a clear advantage of single neural nets and ensembles. The authors look at various performance metrics and check their correlation with final control performance. The contributions are worthy of acceptance, though the paper is weighed down by its narrow experimental scope. Repeating these experiments in a few more mujoco tasks would greatly strengthen this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper deals with uncertainty quantification of models in model based offline RL. 
The paper uses an autoregressive dynamic model and shows that a single model that can provide an uncertainty estimate can be used as an alternative to uncertainty estimation using ensembles. Different methods are compared on the hopper benchmark from the D4RL benchmark collection. 
strength_and_weaknesses: **Strengths**
* The paper is well written.
* The paper makes an interesting contribution to the discussion of uncertainty quantification of dynamic models.

**Weaknesses**
* The empirical investigation relies on only one environment (Hopper). 

**Further suggestions for improvement**
* The statement "Our algorithm achieves better performance than MOPO in all datasets" should be softened, e.g. "Our algorithm achieves equal or better performance than MOPO in all datasets", because in medium_replay the performance is not significantly better.

* In Table 2 and Table 3 standard deviations are given after $\pm$. Correctly, the $\pm$ sign is used to indicate the uncertainty of the measurement. So a confidence interval or a standard error.
This serves to ensure that the statistical significance of differences in the measured values can be easily grasped.

* Uncertainties should be stated with one or at most two valid digits.

* The number of decimal places in A $\pm$ B must match. E.g. "31.34 ± 0.5" -> "31.3 ± 0.5".

* Using only three repetitions (seeds) leads to unreliable results. If it is somehow possible, there should be more, e.g. 10 or 50.

* In "Other methods include behavior regularized policy optimization", also (Fujimoto et al., 2019) should be cited.

* Similar to the present paper, (Depeweg et al., Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning 2018) estimates uncertainty without ensemble and uses the uncertainty for conservatives in offline RL, it should, therefore, be cited. However, long roll-outs are used for the return estimation and no Q-function is used, so that the approach is structurally clearly different from MOPO.

* It is claimed "Conservatism in MBRL is achieved by uncertainty-based penalization of the model predictions." however, this is only one possible way, another possibility is also in model-based the behavior regularized policy optimization, e.g. (Swazinna et al, Overcoming model bias for robust offline deep reinforcement learning, 2021). Presumably other techniques exist, so it is probably better to write, e.g., "Conservatism in MBRL is frequently achieved by uncertainty-based penalization of the model predictions."

* It would be interesting to study the behavior in a stochastic environment, because only in stochastic environments the problem exists to separate aleatory and epistemic uncertainty (future work).

clarity,_quality,_novelty_and_reproducibility: **Clarity** very good

**Quality** good

**Novelty** good

**Reproducibility** fair

summary_of_the_review: The paper makes an interesting contribution to the discussion of uncertainty quantification of dynamic models.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper investigates the effectiveness of autoregressive models versus ensemble models in estimating the transition dynamics and the uncertainty of the model in offline reinforcement learning. The paper's result shows that autoregressive models can be more effective than ensemble models for offline policy optimization. The reviewers note that the paper provides interesting results, delivering insights into how each model quantifies uncertainty and how model error correlates with optimized policy performance. They also note that some experiments need to be included to show why one model class can better model the dynamics or uncertainty than the other. Additionally, the reviewers point to the need for more than three trials to claim that one method is better. This paper has the potential for a solid contribution to the community. Still, it needs experiments to understand further why or when one model class will outperform the other and properly account for the performance metrics' uncertainty. 

Side note: a deep investigation of a single environment is acceptable for publication. The important thing is to demonstrate new insights that are likely to generalize beyond that single environment. 
justification_for_why_not_higher_score: The paper could be considered for a poster, but the experiments do not properly quantify their uncertainty and thus should not be accepted.
Decision: Reject