Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces a Conservative Offline Model-Based policy Optimization RL algorithm, called COMBO, that learns a pessimistic model by enabling a lower bound optimization on the policy performance without requiring uncertainty quantification. COMBO employs an actor-critic method to learn the value function on both offline and synthetic datasets. The idea is to penalize Q-values learned from state-action pairs that are out-of-support of the offline dataset. 

main_review: $\textbf {Overview}$:

This paper proposes a new technique for conservative offline model-based policy optimization. Theoretical guarantees are provided to demonstrate that: 1) COMBO achieves less conservative Q-values compared to prior model-free offline RL methods, 2) COMBO's learned Q-function is the lower bound of the true Q-function that eliminates the need to access a model error or uncertainty oracle in model-based RL, and 3) it guarantees safe policy improvement. Empirical results show the outstanding generalization performance of COMBO and its adaptability to unseen behaviors. Moreover, it can be applied in vision-based tasks and achieve comparable results. 

$\textbf {Major concerns}$:

- Despite the outstanding performance of the COMBO in most of the standard benchmarks compared to the state-of-the-art model-based and model-free algorithms, and the theoretical guarantees discussed in the paper, COMBO is still dependent on the sub-optimality of the data (offline + synthetic). This issue can also be inferred from the occasional inferiority of COMBO comparing to model-based methods when dealing with more diverse state-action distributions. 

- To evaluate how well COMBO generalizes to the new tasks, experimental results are averaged over only three random seeds. This can introduce some bias in the final results due to the low number of runs. Lack of insight into the standard deviation values and the inadequate number of independent runs damage the reliability of comparing different methods.


$\textbf {Minor concern}$:

- Regarding the results demonstrated in D4RL tasks, it is not adequately explained how they have normalized return values of COMBO and other methods neither in the paper nor in the appendix. I would like the authors to explain how they have chosen the expert scores for every three environments? How did they fine-tune the trained policy with behavioral cloning [1]?   

- Since source code availability is not mentioned in the paper (but included in supplementary), I would invite the authors to release their code.

Overall, I found the paper is well written, the proposed technique interesting, and the motivation clear. To the best of my knowledge, what is proposed in this article is novel. 

[1] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
limitations_and_societal_impact: The authors adequately addressed the limitations and future directions in the conclusion of the paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work investigates model-based reinforcement learning with offline dataset. In particular, most of the work on mbrl with offline datasets requires uncertainty estimation to determine the out-of-distribution states and actions. This work eliminates the need to have uncertainty estimation by using the existing Conservative Q-Learning(CQL) algorithm for states and actions from the dataset as well as the rollouts generated from the dynamics using the policy being learned. They show that inclusion of new dataset from the generated rollouts and by being pessimistic about their values, they tend to be better than existing Model-based offline Reinforcement Learning which requires uncertainty estimation
main_review: Originality and Significance:  Uncertainty estimation for model based offline reinforcement learning has been required for determining out of distribution state and actions. This is done either using an oracle estimator or by learning an ensemble of models(dynamics) and measuring the divergence between the states generated by these ensembles. This work eliminates the need to have this uncertainty metric by simply minimizing the value estimates for all states and actions out of the given dataset; as done in the CQL algorithm. They supplement CQL by addition of new data generated using the model(dynamics) and they treat the  states and actions in the generated rollout as out of distribution. 

Quality:  They support their approach via a theoretical analysis to indicate the learned critic estimates would be lower-bounded than the actual values given by the bellman operator with the oracle transition model. Irrespective of inaccuracies in the learned model which can give overestimated reward to states generated in rollouts, they simply depend on the \beta in eq. 2 for suppression of Q-values. I am wondering, how do they choose this value in practice?  And, did this choice affected their results for random-hopper? And medium-reply hopper in table 3.  

They experimented with both image-based environments as well as the d4rl dataset. It was not clear to me if they were using latent dynamics or observation-based dynamics for the image-based experiments.

Clarity: Yes, the ideas are conveyed in a clear manner. And, the given equations are enough for implementation; given the proposed method would only require a minimal change in existing open-source implementations of the baseline algorithms.

limitations_and_societal_impact: Yes, the authors talk about limitations of their work in terms of potential miss-reward specification which could be harmful for real-world. This especially arises due to learning of parametric dynamics models
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a new model-based offline RL algorithm, COMBO. The algorithm applies model-based learning to be able to generate random rollouts, and the generated data is combined with the given fixed dataset for value function training. The paper shows by using the generated rollouts as data for Conservative Q-learning provides better constraint and better empirical performance compared to the orignal CQL algorithm. 
main_review: This paper proposed algorithm COMBO, which is a combination of model-based RL and offline RL algorithm CQL. Instead of using random generation to explore (s, a) space for Q value constraining, the algorithm used model-based learning to learn the environment dynamics and then applied auxiliary policy to generate random rollouts for the use of constraining Q value. Theoretical proof is provided to show that the new Q update is conservative and the policy improvement is guaranteed. There are no new ideas proposed, but combining these two and making the algorithm work in most scenarios are precious work. Extensvie experiments are done to show COMBO performs well in generalization tasks, image-based tasks and OpenAI Gym tasks. 

The papar is well written and well structured. After reading, I have the following questions for the authors: 
1. Do you have a ablation study of using which kind of rollout policy \mu(a|s) to generate addtional data points is the best for improving the performance? 

2. Experiment details: for D4RL experiments, do you use the datasets in version 'v0' or 'v2' for performance comparison? 

3. It seems that the hyperparameter tuning is based on each single task, which is not a standard way for performance comparison in RL? Are the table of results shown in the paper based on the same set of hyperparameters or several set of hyperparameters? There is no validation dataset for RL task. If you are tunning hyperparameters on real rollouts, that is a really dangerous thing in practice, for example in medical and financial areas. 

Minor:

In proposition 4.2, the \delta_{COMBO} has the same (s, a) input distribution as \delta_{CQL}, which I think it should be different based on your notations in the paper. 

In proposition 4.1, Equation (*), I think the subscript \pi(a|s) is not needed in the second term. 

Line 68: under which the this?

limitations_and_societal_impact: Yes!
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper presents a new approach for model-based offline RL. Here, the model is not used to estimate the return directly using roll-outs, but to generate synthetic data that is then used to train the Q-function. In a way, the method is a combination of CQL and MOPO. What is new is the idea of using this synthetic data to, penalize extrapolation instead of using uncertainty estimation.
This modification performs much better on the selected benchmarks.
As benchmarks a selection of D4RL benchmarks is used (3 benchmarks) and two vision based benchmarks.

main_review: Originality:\
Very good.
On the one hand, it is a combination of existing methods (CQL and MOPO), but on the other hand, the idea of using the generated synthetic data to penalize the extrapolation seems to me to be a significant idea. Simple, elegant, but seemingly functional.

Quality:\
High quality.

Clarity:\
Very clear.

Significance:\
I think the idea to use the synthetic data as means for behavior regularization might be very influential.

Further remarks:\
The results would be even more convincing if the experiments had been performed with more than just three trials.

**Update after feedback:** \
Based on the feedback, discussion, and very competent responses and explanations from the authors, I have raised my score to 8.

The present results show statistically significantly that COMBO performs best in the experiments conducted.
A simple t-test (student-t with 5 degrees of freedom) or a rank test clearly show this.

Furthermore, I think it is clear that the algorithm is not plagiarism, but a new development, even if it is essentially a combination of two known methods.
My opinion is that it is absolutely common to create something new by combining what is known. The fact that this combined algorithm works recognizably better is an interesting finding worth presenting to the specialized public.

Regarding the treatment of hyperparameters, I want to state my conviction from many years of practice that on the one hand the best comparison between different methods is the one where each method is near 1) its respective optimum in the hyperparameter space (otherwise one compares the suitability of the hyperparameters rather than the methods) and that in the applications of offline RL the data efficiency and performance were always in the foreground and not the effort for finding suitable hyperparameters. In short, if it is POSSIBLE, it will be done. On this topic, I find the authors' response remarkably thoughtful and coherent.

1\) by spending the same effort to come near the respective optimum in hyper-parameter space for each method, if the difficulty of finding the optimum differs significantly between the different methods





limitations_and_societal_impact: Yes
review_title: null

Meta-review: The reviewers all agreed that this work is promising. The authors also provided important clarifications in the response, including increasing the number of runs and clarifying that hyperparameter selection was actually done with an approach that they developed for this paper suitable for the offline setting. This improvements undoubtedly make the paper stronger. However, the original omission of the strategy to select hyperparameters offline is a large omission, as it is a critical part of the algorithm. This algorithm should be discussed and contrasted to other approaches. 

The theory was not extensively discussed (unfortunately), and relies heavily on the CQL theory. The CQL theory already has some constants that make for relatively loose bounds. Here, there are additional issues with the introduction of beta. The first main result relies on having a potentially very large beta, especially if you look at the proof and see the terms that beta*(small positive constant) is overcoming. The second result, on policy improvement, relies again a potentially large beta and on a term being positive that is not guaranteed to be positive (there is simply a discussion in the paper on why it reasonably could be positive in settings of interest). Theory for offline RL is hard, and so progress should be acknowledged. But, at the same time, given the complexity of these results, it is important to more clearly explain the limitations and what this theory can truly guarantee. 

As a note, the concern about per-environment hyperparameter tuning was resolved in the discussion. It is reasonable to pick hyperparameters per environment, since you have an automated algorithm to do so. This is very different from optimizing (tuning) hyperparameters with sweeps, which is not an algorithm to be used in practice but rather an approach to evaluate methods. 

As it stands, this work is borderline, and would highly benefit from another round to incorporate these changes and be re-reviewed. Some benefit of the doubt can be given that these changes may be done for the final paper, and so if there is space in the program, this paper could be accepted.
Decision: Accept (Poster)