Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes an Alternating Markov Game formulation of offline RL which induces a MDP with pessimistic dynamics. The authors derive an approximate solution to this game which requires no uncertainty penalization and show strong results on the D4RL MuJoCo benchmark.
strengths_and_weaknesses: Strengths:
- Strong theoretical background including derivation of connection between AMG and the equivalent MDP, well-related to standard offline MBRL in the $N=k=1$ case.
- Strong empirical results on the D4RL MuJoCo benchmark. The offline evaluation is thorough uses a single hyperparameter setup across all environments, thus avoiding using online samples to tune performance for future deployment.

Weaknesses:
- Presentation is sometimes dense, related work should be discussed in the main paper.

Minor:
- Typo on line 22, 345: trial instead of trail.
- The version of D4RL dataset used in the main evaluation should be highlighted, there are performance differences between v0-v2.
- Section link in line 498 broken.
- [1] considers a related formulation which also induces pessimistic dynamics. It would be valuable to compare these works, but I appreciate this work appeared after the ICML deadline.

[1] Rambo-rl: Robust adversarial model-based offline reinforcement learning. M Rigter, B Lacerda, N Hawes.

questions: - How does the slow-evolving policy update relate to TRPO/PPO style KL-constrained policy optimisation?
- It would be interesting to understand more the motivation behind the design of the optimization procedure in Section 4. How inefficient is solving the original problem, what is the overall gain in training speed by using the reference policy?

limitations: Efficiency of the algorithm should be detailed in the main paper, as well as discussion of related work.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper points out that characterizing the impact of dynamics uncertainty through reward penalty may incur unexpected over-conservative behaviors. To overcome this problem in existing model-based offline RL works, the paper proposes to maintain a belief distribution over dynamics belief and optimize policy via biased sampling from the distribution. The sampling procedure is derived based on an AMG formulation of offline RL. For practical consideration, an offline RL approach named PMDB is further designed. Results on the D4RL benchmark show the state-of-the-art performance of PMDB. 
strengths_and_weaknesses: Strengths: 
* Motivation: The paper points out an important while has been ignored problem in model-based offline RL, that it’s unsuitable to directly characterize the dynamic uncertainty by reward penalty. 
* Theoretical Contribution: To solve the problem, the paper formulates the offline RL problem as an AMG and discusses the relationship between AMG and robust MDP. Throughout the formulation, several interesting theoretical results are provided. The illustration that AMG is a successive interpolation between model-based RL and robust MDP provides an intuitive explanation of the theoretical results.
* Empirical Contribution: The paper verifies the proposed PMDB method on D4RL and shows the SOTA performance without the need to tune hyperparameters for each task. 

Weaknesses:
* It may be due to the complexity of the overall framework, the paper is somewhat hard to follow. It took me a very long time to understand the symbols and formulations defined in Section 3.1. 
* Although the framework is very complicated, it seems the core parts that work are: (1) dynamically adjusting the game transitions in each training iteration; (2) Updating the Q-function based on the K-percentile dynamics instead of the worst one. While with the theoretical proof, it would be better to do ablations on the above two parts to further illustrate the effectiveness of the designs.

questions: 1. I’m not sure if I actually understand the updating of game transitions/dynamics beliefs. Could you please explain it to me in detail?
2. PMDB relies on the initial dynamics belief and the paper uses a uniform distribution over dynamics as the initial belief. Do you mean inserting additional knowledge into the initial dynamics belief is to pre-define an initial dynamic belief? If so, could you give some examples of the scenarios in which we obtain priors on dynamic belief? If involved the expert knowledge, how much will the performance of the algorithm 
3. I’m concerned about whether the training of PMDB is stable considering the design of this framework?


limitations: The paper discussed the limitation that expert knowledge is not always available thus it’s hard to manually pre-define the dynamics belief. Besides, I think another limitation is the lack of ablations on several key components.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a mechanism for offline RL based on a two player game with a Bayesian environment, in which an RL actor picks a policy and a environment actor picks environment transitions from a likely set pessimistically. The authors show general convergence and existence results in their framework and empirical results based on the established D4RL framework.
strengths_and_weaknesses: == Strengths ==
- The core theory is well laid out and supported with theoretical results as well as empirical investigation.
- The structure is overall well laid out and easy to follow. Some minor points of confusion and suggestions are discussed below, but in general, the paper is presented in a clear legible fashion.

== Weaknesses ==
- The paper is missing a clear related work section. This makes it harder to position it in the relevant literature and makes it difficult to assess whether concurrent similar strands of the literature were reviewed. Since I am not an expert in the concrete field the authors are tackling, some of the following concerns might be wrong or missing relevant details, but I will list them for discussion:
  - My biggest concern is that no other works on Bayesian Robust RL were cited. The formulation the authors present seems novel, but to the best of my knowledge several other works have tackled Bayesian Robust RL and a clear comparison would greatly improve the paper. I.e. how does the proposed method relate to https://arxiv.org/abs/1905.08188 and similar works.
  - The other strand of research I am missing comparison to is CVaR style algorithms, which seem similar in its setup in targeting a quantile of returns. For example, this paper from NeurIPS 2021 seems very related https://papers.nips.cc/paper/2021/hash/08f90c1a417155361a5c4b8d297e0d78-Abstract.html
- The set generating mechanism and definitions in (3) could be explained better. Big $\tau$ seems to refer to a set of transitions, but $\bar{s}$ is a state (or a single transition)? This seems to be explained in the text under the equations, I think it might be clearer for readers if the state and action spaces of the 2nd player was fully explained in the introduction of the formalism.

Unclear lines/grammatical errors:
- Line 51: with entirely different implication compared to the reward or Q-value?
- Line 140: To differentiate with uncertainty set we call it candidate set, but to avoid redundancy we reuse the notation of T?
- Line 24: in vast of applications?
questions: - 295: if you assume online-access here, do you break offline RL assumptions? In that case, the core assumptions of the scenario should be clarified here. This would make the comparison between this algorithm and fully offline methods slightly less convincing, if you assume i.e. a few-shot scenario.
- Theorem 2: is that MDP stationary? It depends on pi, so that might make things more complicated? If the policy has to be learned in a changing MDP, how does that affect learning? Theorem 1 states that the operator is a contractive mapping, but 
- (14) why not KL as difference of distributions, or Wasserstein? Is there a specific reason for this concrete formulation?
limitations: A brief discussion is given, without mentioning any major drawbacks in depth.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Existing studies on model-based offline RL often rely on the quantification of the  uncertainty in the learned dynamics and utilize it to regularize the reward.  One drawback of this approach  is that it may result in unreliable evaluation on the impact of the uncertainty and incur unexpected tradeoff between model utilization and risk avoidance. To resolve this issue, the authors introduced a novel model-based offline RL algorithm with pessimism-modulated dynamics belief, where the uncertainty is not explicitly quantified but  a belief distribution over system dynamics is used for sampling instead, when updating the policy evaluation and policy update.  The sampling procedure is cleverly modeled as an alternating Markov game and the degree of pessimism can be determined by the hyper parameters in the sampling procedure. The proposed offline RL algorithm solves the AMG by iterative regularized policy optimization with monotonous improvement guarantee. The empirical results on D4RL shows the efficiency and performance improvement of the proposed algorithm. 
strengths_and_weaknesses: Strength:
+ The proposed methodology is explained in detail and the paper is easy to follow. The empirical results and ablation studies shwocase the efficiency of the proposed approach on the standard benchmarks. 

+ The formulation of offline RL as AMG is interesting and leads to the development of pessimism-modulated dynamics belief derivation.

+ The theoretical results on the pessimism-modulated dynamics belief is promising and serves good guidance on the empirical offline RL algorithm design.  

Weakness:
- The experimental results can be made more comprehensive. In Figures 2 and 3, what is the possible cause of the large dip in half-cheetah-medium large dip in halfcheetah-medium task?

- How to tune the hyper parameter $k$ and $N$ when processing a new task? 

- How to determine the weights in Eqn. (11)? The optimization over the Bellman residual of both the AMF and the empirical MDP ia applied with the same weight (both are 1).

- Line 194,   the reference to Eqn. (18) is undefined.

questions: How to construct a sufficient belief without expert knowledge  for an arbitrary task? How to tune the hyper parameter $k$ and $N$ when processing a new task? 

How to determine the weights in Eqn. (11)? The optimization over the Bellman residual of both the AMF and the empirical MDP ia applied with the same weight (both are 1).
limitations: One limitation of this work is that is is unclear a prior how to obtain a sufficient belief without expert knowledge. It will be of interest to shed light on how to construct a valuable (initial) dynamic belief for arbitrary tasks.
review_title: null

Meta-review: metareview: I went through the manuscript, reviews and authors' responses. I think this paper is qualified for NeurIPS publication.
confidence: Certain
Decision: Accept