Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work proposes a new algorithm for offline model-based reinforcement learning (MBRL). Based on a lower bound of the cumulative reward objective (Eq.(5)), the proposed method alternatively optimizes the policy and the transition model using a fixed dataset. To bypass the difficulties in the objective, the part of the lower bound is further approximated by the Jensen-Shannon divergence (JSD) so that GAN training can be applied. In addition, the objective includes a marginal importance weight (MIW) that is also being estimated during training. Experiments on the D4RL benchmark show that the proposed method can achieve better performance than existing methods in the literature.
strengths_and_weaknesses: Strengths
- A unified objective for training both the model and the policy
- Solid experiments

Weaknesses
- Lacking theoretical justification
- Some of the approximation steps are questionable
questions: Although it is nice to have a unified objective, there are several questions/concerns regarding the method.

1. The stability of the proposed method becomes a question when it alternates between learning several models (policy, action-value function, MIW, transition kernel, and the GAN discriminator). Each of them requires some sort of stability tricks, so the overall procedure seems very fragile as a whole.

2. L121: Why implicit policy is preferred? Gaussian or Beta policies are common in the literature.

3. When optimizing (13), 1. When optimizing (13), does some of the experience in the augmented dataset have fake rewards? Algorithm 1 does not mention anything about the learning of \hat{r} and how it is used.

4. The learning of MIW is interesting. Eq.(9) in the current paper is Eq.(4) in Wen et al., (2020) [44] when expanded. Such connection should be discussed and referenced. The main difference is that the current paper uses a test function. However, the motivation for using the action-value function as the test function is unsubstantiated and requires further discussion/evaluation. It seems more natural to use the reward function instead when it is motivated by the primal-dual relation (Eq.(1)), because we care more about learning accurate MIW. In this way, the LHS of (10) will be exactly the cumulative reward objective.

5. Experiments
- It would be more readable if the best model-based method is italicized/underlined instead of also bold in Table 1
- How are the MIWs normalized in Fig.2? It is surprising that the median of DualDICE can be so far away from 1 after normalization.
- The description in Sec.4.2(c) is unclear. The first sentence suggests that the regularization uses KL as in Thm.1, but the second sentence immediately suggests using JSD instead, which seems to be the “Main” variant. Specifically, how is WPR different from Main?

Minor
- L63: \pi_b is presumably the behaviour policy?
- L163: does not undermine the convergence property (under mild conditions on Q).
limitations: The authors have sufficiently addressed the limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper focuses on the problem of offline reinforcement learning, where the authors propose a novel model based algorithm. Specifically, the authors start from the general framework of bounding the difference between the return on the true model and a learned model, and then derive the objective for model learning and policy improvement. For the learning dynamics model, the authors propose a maximum likelihood objective weighted by the importance ratio of the policy state action visitation distribution and the dataset distribution. Specifically, the authors introduce a Bellman equation-like objective to estimate this important ratio. For policy improvement, the authors adopt an actor-critic style algorithm and penalizes the divergence between the policy and the dataset during actor updates. The divergence value is estimated using a generative adversarial network.

The authors evaluate the proposed method in the D4RL benchmark, and show that the proposed method outperforms prior model based and model free methods.

strengths_and_weaknesses: Overall I think the paper presents an interesting idea of doing model based RL from offline dataset. However I do have some concerns.

Pros

The empirical results of the paper are pretty strong. From table 1, it seems that the proposed method achieves better performance than pior method in most of the tasks from D4RL, which is a clear indication that the proposed method provides performance benefits. I also find the ablation study quite useful in showing that the MIW weighted model learning is important.

The paper is well written and the presentation is very clear. Building on top of the general framework of model based RL, the derivations and analysis of the proposed method are intuitive and easy to follow. 


Cons

It seems to me that the practical implementation of the algorithm is somewhat disconnected from the derivation of the algorithm, so it is hard to tell if the method works because of the reasons justified by theoretical analysis or because of the choices made in practical implementation. For example, the authors start from the standard approach to bound the gap between the return between the true and learned dynamics model (theorem 1) and propose to estimate the importance ratio (MIW). However, the derived method for estimating MIW is not implementable since it uses quantities not available to the algorithm, so the authors propose to replace the unknown density with the learned Q function. It is not clear to me why this is a good choice and hence I believe that more theoretical analysis is necessary.


During policy training, the authors used a GAN to estimate the divergence, which is one of many ways to estimate the divergence or quantify the uncertainty. Prior work [1] suggests that the performance difference of model based offline RL algorithms mainly come from these practical choices. Hence I believe the paper could benefit from a comparison of different ways of penalizing the policy during policy improvement. 


While the paper presents an interesting idea and the empirical results are strong, I do believe that some more analysis and justification is needed for the proposed method. Therefore, I’m neutral between acceptance and rejection for this paper. I highly encourage the authors to include them during the author response period.



References

[1] Lu, Cong, et al. "Revisiting design choices in offline model based reinforcement learning." International Conference on Learning Representations. 2021.


----
The authors' response addressed most of my concerns about the paper and therefore I'm recommending acceptance. I've increased my rating from 5 to 6.

questions: Prior work [1] suggests that some baseline methods used in this paper can be tuned much better. I wonder if the authors could include these improvements in the baseline comparison as well as incorporate them into the proposed method if applicable.


References

[1] Lu, Cong, et al. "Revisiting design choices in offline model based reinforcement learning." International Conference on Learning Representations. 2021.

limitations: The authors have sufficiently addressed the limitations and societal impacts.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: 

The authors propose a unified framework for alternating model training and policy learning in the offline RL setting. The model and the policy are trained to maximize a lower bound of the actual return. The experiments are conducted in the D4RL benchmark and several SOTA baselines are taken for comparison.
strengths_and_weaknesses: 
Strengths:

1. The motivation for building a unified framework for alternating model training and policy learning is in time and reasonable. Although current offline model-based RL algorithms have learned policies considering the inaccuracy of the models, the objective of model learning is still separated from the offline model-based RL pipeline. It is valuable as a research topic to investigate; 

2. The article is overall well-written and easy-to-follow to me;

3. The experiment results in the benchmark datasets seem great.

Weaknesses:

With respect to the theoretical modeling,  there still have large gaps between Theorem 1 and the algorithm designs in Sec. 3.2, 3.3, and 3.4. And thus, I prefer to evaluate the value of the article with respect to the novelty of the proposed techniques. 

In Section 3.2 and 3.2, the authors proposed a new regularizer for policy learning and a new technique for importance weight estimation.  Both of the techniques are interesting: the regularizer not only evaluates the difference between two policies but the effect difference of the policies in different models, which makes sense for formulating a unified training framework. Importance weight estimation is also a challenging task. The proposed method utilize a test function to identify and correct the weight estimation. However, there are not enough experiments and descriptions to demonstrate the actual mechanism and the relation of the proposed techniques (See details in the following).
questions: 
1. In Section 3.2, to learn a policy, the authors optimize a surrogate objective of Eq.(7) via estimate D_\pi. However, If we have an approximation of D_\pi, why don’t we optimize the model through D_\pi too? Similarly, If we have an approximation of \omega, why don’t we use \omega to estimate D_pi directly for policy optimization? 

2. The results in Figure 2 seem great but the details of experiment settings are still unclear to me. Can you give a detailed description (which can be in the Appendix)?  For example, how to normalize the MIWs; What’s the difference among the 1~9 policies for testing.

3. Can the authors provide more evidence to demonstrate the mechanism of the regularizer and MIW training techniques such as conducting experiments in some interpretable environments or recording some intermediate  metrics in some specially designed experiment like (b) in Section 4.2
limitations: 
I think the overall idea of the article is interesting but needs some refinements before publication. The authors may be able to further analyze from a theoretical perspective; Or, from an experimental perspective, I think the mechanism of regularization and omega estimation can be more fully experimentally verified. I will consider increasing the score if the authors improve the article in either of the aspect.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this work, the authors investigate a new approach to the problem of Model-Based Reinforcement Learning in the off-line RL setting.
The paper proposes a number of new ideas. First, instead of first learning a model and subsequently using it to obtain a policy, their approach alternates between updating the model and improving the policy. The MLE model learning is aware of the discrepancy between the stationary distribution of the behavior policy and of the learned one through the use of marginal importance weighting. These weights are learned using a novel Bellman backup operator. At the same time the authors propose a new regularization loss in their policy updates to account for the difference between the true transition probabilities and the ones produced by the learned model. This encourages the imaginary rollouts produced by the model and policy to stay close to the true distribution observed in the offline dataset. 

strengths_and_weaknesses: The authors propose a theoretically and empirically principled algorithm. Their method introduces a number of novel ideas, in a clear way, which are in the right direction of coupling model and policy learning in offline RL. 

A possible weakness would be the stability of the learning rules. 
The model learning updates depend on the values of the importance weights. At the same time the importance weights are learned through a Bellman backup which depends on the Q values learned using the model. Moreover, the policy is regularized through a GAN mechanism which can further destabilize learning. The authors show empirically that their method is stable which removes some of these concerns. However, since there are many hyperparameters involved it is not clear how important these are in achieving stability. 
questions: The authors should look at the work proposed in [1] as another simpler method where model and policy learning are coupled to achieve state-of-the-art performance. Is there any benefit in using transition models instead of value equivalent ones ? 

Minor suggestions:
Cite and clarify the idea of target networks. 
Cite and clarify the use of a Huber loss in the Q-function update rule. 

[1] Schrittwieser J, Hubert TK, Mandhane A, Barekatain M, Antonoglou I, Silver D (2021) Online and Offline Reinforcement Learning by Planning with a Learned Model. In: Beygelzimer A, Dauphin Y, Liang P, Vaughan JW (eds) Advances in Neural Information Processing Systems
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: 
In this paper, the authors motivated from the observation that the learning objectives of model and the polices in offline model-based RL are mismatched, and established a lower bound of the true expected return, which includes both the model and policy learning. The authors designed a tractable approximation to the lower bound, building upon which an algorithm is derived with practical implementation. The algorithm is then justified empirical to demonstrate the superior. 

Most of the reviewers appreciate the proposed method. The paper can be further improved:

>All reviewers believes there are some gaps in the derivation of the practical algorithm from the original optimization. The rationale behind the approximation and parametrization should be clearly discussed to avoid the possible confusion. I understand there is page limit, but I believe the paper can be reorganized to leave the important discussion, especially the comparison w.r.t. DICE family and VPM, in the maintext, not in the Appendix. \


In sum, the paper indeed provided an interesting method which interactively learn the policy and the model in offline setting by optimizing the lower bound of the true expected return, and a practical approximation scheme with superior performance. I recommend for acceptance. 
confidence: Less certain
Decision: Accept