Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose RAMBO to offline RL which imposes conservatism by adversarially modifying the transition dynamics of a learned model. RAMBO is built on a formulation for which PAC bounds are guaranteed. The experiments are conducted in the D4RL benchmark and several SOTA baselines are taken for comparison.
strengths_and_weaknesses: 
Strengths
1. The article is well-written and easy to follow.
2. The solution is simple to implement and sounds reasonable.


Weaknesses

1. RAMBO is similar to COMBO: COMBO learns a conservative Q through regularization based on the policy while RAMBO reduces unreasonable V through regularization (Adversarial training) based on the model. The results in Table 1 also show the similarity in performance. It is OK to give a novel regularization from other perspectives. But I think further comparison should be considered to clarify the advantages/disadvantages/scopes of the two regularizations.
questions: The article is well-written and soundness to me. I have no further questions.
limitations: 
As mentioned above, I think further comparison should be considered to clarify the advantages/disadvantages/scopes of the two regularizations.If it is done and we can find more insights in the two regularizations after that,  I will consider increasing the score
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: To remedy with the extrapolation error in offline setting, this paper formulates the offline RL as an adversarial process between the agent and the environment. The environment is modeled as a neural network and trained with both MLE objective and conservative objective which is essentially minimizing the state value of the agent. Previous research [1] proved that this formulation enjoys a PAC performance guarantee, and the experiments do demonstrate some performance improvement on certain D4RL tasks. 

[1] Masatoshi Uehara and Wen Sun. Pessimistic Model-Based Offline Reinforcement Learning under Partial Coverage. ICLR 2022.
strengths_and_weaknesses: Overall I enjoyed reading this paper. 
+ This paper is well organized and comprehensive in presentation. The authors manage to convey their idea in concise but informative language.
+ The main argument that adversarial training can relieve value overestimation and model defect is supported by an illustrative example in the appendix. This is a persuasive argument to justify the effectiveness of the proposed method. 
+ The authors provide the detail about their hyper-parameter configuration and strategy in the appendix. I think the strategy of deciding on RAMBO's hyper-parameters is acceptable, for no excessive online fine-tuning is involved in training stage. 

Some weaknesses:
+ The baselines, no matter model-based ones or model-free ones, seem to be old at this time. For model-free ones, there are [2] and [3]; for model-based algorithms, there is [4] as far as I can recall. 
+ The analysis of RAMBO is too brief. From table 2 I can tell that RAMBO does better on medium-replay datasets while (comparatively) worse on medium-expert datasets, but the authors did not provide in-depth analysis of this phenomenon. More discussions, such as whether this phenomenon is related to adversarial training and how, is expected. 

[2] Gaon An, Seungyong Moon, Jang-Hyun Kim, Hyun Oh Song. Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble. NeurIPS 2021. 

[3] Ilya Kostrikov, Rob Fergus, Jonathan Tompson, Ofir Nachum. Offline Reinforcement Learning with Fisher Divergence Critic Regularization. ICML 2021.

[4] Yijun Yang, Jing Jiang, Tianyi Zhou, Jie Ma, Yuhui Shi. Pareto Policy Pool for Model-based Offline Reinforcement Learning. ICLR 2022. 


questions: Just one question about COMBO baseline. In the original paper of COMBO, it was not clarified which version of the dataset (v0 or v2) was used in the experiment, so I suppose COMBO used v0 datasets as CQL did. So did you re-implement COMBO and tested it with v2 dataset? 

Also, it is interesting that the performance of COMBO reported in this paper is precisely identical to the original paper, except for walker2d-medium-expert, hopper-medium-replay, hopper-medium and walker2d-medium. So would you elaborate more on how did you obtain the result of COMBO? 

Please correct me if there is any mis-understanding. 

limitations: The authors do mention some limitations of their work at the conclusion section, such as the computational cost. However I don't think this is a major concern.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes RAMBO, an offline model-based algorithm that formulates the optimization of models and policies as a two-player zero-sum game. Specifically, RAMBO trains models with an extra objective that minimizes the value function to achieve conservative policy optimization. The authors provide a PAC performance guarantee and prove that the learned value function lower bounds the true value function. Experiments demonstrate that RAMBO achieves state-of-the-art performance.
strengths_and_weaknesses: Originality: 

The idea of RAMBO is novel. This paper introduces the idea of RARL into offline model-based RL, which can provide a pessimistic value estimation and alleviate the distribution-shift issue. 

Quality: 

The theory part is solid.
1. The authors provide a PAC bound that guarantees the accuracy of the learned value function.
2. The author proves that the learned value function is a lower bound of the value function in a real environment.

Nevertheless, the authors may want to provide more details on the algorithm and experiment parts.
1. I can not understand how the gradient is computed according to (5), especially if $s^\prime$ is a terminal state. 
2. Appendix shows that RAMBO is sensitive to hyperparameters, especially in Hopper and Walker2D. I wonder whether the gradient in (5) has high variance if $s^\prime$ is a terminal state, and then the optimization of models becomes unstable.
3. I wonder whether the model training phase in RAMBO leads to high computational cost. The authors may want to compare RAMBO and COMBO in wall-clock time. 
4. I wonder whether RAMBO learns a lower bound of the true value function in authors' empirical studies. If the authors can provide analyses similar to those in TD3 [1] and REDQ [2], the authors' claims would be better supported.

Clarity:

This paper is well written and easy to follow.

Significance:

This paper provides a new idea for learning a pessimistic value function. However, the performance improvement is insignificant, as COMPO outperforms RAMBO^OFF as shown in Appendix. 

[1] Fujimoto S, Hoof H, Meger D. Addressing function approximation error in actor-critic methods[C]//International conference on machine learning. PMLR, 2018: 1587-1596.

[2]Chen X, Wang C, Zhou Z, et al. Randomized Ensembled Double Q-Learning: Learning Fast Without a Model[C]//International Conference on Learning Representations. 2020.
questions: Please refer to the "Strengths And Weaknesses" part.
limitations: The authors have addressed parts of the limitations of RAMBO. Nevertheless, I still have concerns about the sensitivity of RAMBO to the hyperparameters.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors tackle the generic Offline RL problem and propose a model-based approach to address it. They introduce a novel algorithm (RAMBO), consisting in adversarially training the model in order to minimize the trained policy performance estimate. The justification of the work builds on a previously published theoretical paper. The empirical validation of the algorithm is limited to MuJoCo-v2 domain where it performs very well, and the AntMaze-v0 where it achieves better than other model-based baselines, but much worse that model-free baselines.
strengths_and_weaknesses: Strengths: 
* The authors propose a novel algorithm for model-based Offline RL, which makes a lot of sense and is potentially high impact.
* RAMBO performs reasonably well in the experiments in the benchmark.
* The paper is clear overall.

Weaknesses:
* The formalization/theory is not rigorous enough:
  * The theoretical claims are vaguely implying an overstatement of what they are. Near optimality is intractable in Offline RL [Foster2021,Xiao2022] so lines 48-50 makes claims that are too strong. The PAC bounds of [69] are loose in the general case. Similarly, lines 189-190 are much too strong. The constants are not discussed at all, which prevents the reader to understand the true nature of the offered guarantees.
  * Equation 1: what does it mean to have 2 policies in your setting? It is laudable to anchor your work into the literature, but I find it very confusing here, while Problem 1 is much clearer right after.
  * Equation 3: $\widehat{T}_{MLE}$ has been defined, and the concept of it is not well defined in large/continuous environments. I understand that the authors, once again, try to make connection with the literature, but this is again confusing. I would have expected to see here the TV with respect to the dataset, instead, which we learn much later to be what the practical algorithm is doing (section 5.2).
  * Theorem 1: $(1-\gamma)^2$ should be $(1-\gamma)^{-2}$.
  * Proposition 2 is wrongly formalized: $s$ is used both in $V_\phi^\pi(s)$ and in the sampling in the expectation. I have looked into the proof in Appendix A and found that the same imprecision is in the proof. It should actually be instead $z\sim d_\phi^\pi(s)$ where $d_\phi^\pi(s)$ is the state distribution starting from $s$.
* The empirical validation is limited:
  * Some simpler problems (multi-arm bandits, gridworld with finite state-action space) might be useful for a better analysis of what it does better than other model-based Offline RL algorithms.
  * Some harder problems with visual input (Atari for instance) would be useful too as it is where model-based approaches typically fail.

[Foster2021] Offline reinforcement learning: Fundamental barriers for value function approximation, Foster, Dylan J and Krishnamurthy, Akshay and Simchi-Levi, David and Xu, Yunzong (NeurIPS, Offline RL workshop 2021)

[Xiao2022] The Curse of Passive Data Collection in Batch Reinforcement Learning, Xiao, Chenjun and Lee, Ilbin and Dai, Bo and Schuurmans, Dale and Szepesvari, Csaba (AISTATS 2022).
questions: In addition to addressing the weakness points above, I would be appreciative that the authors answer the following questions:
* Equation 9: why use the *standard* MLE loss instead of the TV loss as the theory prescribes?
* Remark 1: I think that this point is important to make stronger. I fail to see why Problem 1 does not reduce to pessimistic reward modification. My intuition is that the worst possible transition outcome consists in transiting to a state with minimal value, which is equivalent to applying a pessimistic reward modification of this amplitude.

Minor points:
* Line 1: find near-optimal => near-optimality is intractable in general in Offline RL, so this is not the right objective for Offline RL.
* Line 6: achieve conservatism => enforce conservatism.
* I don't like the use of conservatism to refer to any kind of regularisation with respect to the dataset, including pessimism. Generally, conservatism is the regularization of the policy to remain close to that of the behavioural policy, which is generally made in opposition to pessimism which intends to find a lower bound to the value function and does not guarantee to remain close to the behavioural.

limitations: As I have said before, I am worried that over-stating theoretical findings may lead to improper use of them in the future. Other than that, I do not have any concern with respect to the societal impact of this submission.
review_title: null

Meta-review: metareview: This paper introduces the idea of Robust Adversarial RL for offline model-based RL, which could have a high impact. It is well organized and the writing is very comprehensive; the authors manage to convey their idea in concise but informative language. The proposed RAMBO approach performs reasonably well in the presented experiments, although it was pointed out that the paper would benefit from more scenarios showing the necessity of RAMBO compared with the current baseline (COMBO). Questions and issues related to the theory that were raised during the reviewing process have been addressed in the rebuttal.


confidence: Certain
Decision: Accept