Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work outlines a model-based approach for off-policy evaluation. It revolves around learning (from a static, offline dataset) a recurrent variational latent model of the environment. They outline two "tricks" to improve the approach. One is termed "recurrent state alignment", an additional loss is used to encourage the hidden state representations in the decoder and the encoder to be similar. Secondly, they use multiple decoders and when using the model for OPE average over the predictions of multiple decoders. They test their approach on some off-policy benchmarks on control tasks and compare with other OPE baselines as well as try ablations.

strength_and_weaknesses: Strengths:
- The paper is well-written.

- Off-policy evaluation is an important topic for practical use of RL and of interest to the community.

- Source code is provided.

- The empirical results are compelling.

Weaknesses:
- The environments used (control tasks from D4RL) are typically more used for off policy optimization rather than evaluation. It would be helpful to test the approach on other kinds of OPE tasks such as recommender systems / advertising e.g. [1, 2].

- It would be helpful to discuss (and ideally report experiments) on using this approach for policy optimization rather than just evaluation.

Minor:
- Spelling mistake in section heading 2.

[1] Chen, Minmin, et al. "Off-Policy Actor-critic for Recommender Systems." Proceedings of the 16th ACM Conference on Recommender Systems. 2022.

[2] Saito, Y., Udagawa, T., Kiyohara, H., Mogi, K., Narita, Y., & Tateno, K. (2021, September). Evaluating the robustness of off-policy evaluation. In Fifteenth ACM Conference on Recommender Systems (pp. 114-123).

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is overall well-explained. However, I believe it would be helpful to provide a little more detail (e.g. pseudocode algorithm) on how the model is used for OPE.

Quality: The paper is of high quality.

Novelty: The use of variational autoencoders for models in RL has a long history (as cited here). However, the specific tricks introduced here to make the approach work off-policy is novel and of interest to the community. I would assign this work a moderate degree of novelty (not ground breaking, but certainly not just a reproduction of earlier work).

Reproducibility: The code is made available by the authors (but I did not try running it), so I assume you would be able to reproduce their results. The paper describes the experiments in reasonable detail.
summary_of_the_review: Interesting paper providing some tricks to make variational autoencoders as models work better on offline datasets. It would be of wider interest if it also considered policy optimization.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to tackle the Off-Policy Evaluation problem in a model-based fashion, i.e., contrary to importance sampling-based methods, a model is trained to directly generate future trajectories based on past information. The proposed model is a Recurrent Variational Auto-encoder, where both the encoder and decoder are embodied by LSTMs. The main contribution, besides leveraging variational inference for model-based OPE, is the addition of two features: Recurrent State Alignment (RSA) and branching architecture for the decoder.
RSA adds a term in the loss so that the hidden LSTM states of the encoder and the decoder stay close to each other. The authors claim that this term augments the information flow and helps the decoder generate more realistic trajectories.
The branching architecture is a kind of ensemble of decoders, where each branch is assigned a weight (summing to 1 over all branches) that is trained jointly with the parameters of the model. The authors claim that it helps alleviate the effect of initialization.
The authors present results in the DOPE benchmark against model-based and model-free baselines and achieve superior performance overall.
strength_and_weaknesses: Strengths:
- The results on the DOPE benchmark are compelling
- The ablation study is well conducted
- Writing and presentation are pleasant and help get to the point quickly.

Weaknesses:
- The claims and motivation for adding RSA and the branching architecture lack theoretical or empirical justifications, beyond the final results: would it be possible to characterize the behavior observed when MSE is used / RSA is not used, and explain why decoder initialization can lead to such largely decreased performance
- The intuitive statements made by the authors are sometimes vague: would it be possible t clarify terms such as "structure", "information flow" and "impact of initialization artifacts" in this particular context?
- The results might be poorly reliable, as they are given on only 3 random seeds.


clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well-written overall but motivation is sometimes eluded.
Quality: Results on DOPE are compelling, but evaluating only 3 seeds is not enough especially considering the sometimes unexplained variations between variants.
Novelty: This paper is the first to leverage variational inference, and additional features seem novel.
Reproducibility: the model is evaluated on the open-source DOPE benchmark and sufficient implementation details are given, but there seems to be no plan for code release.


summary_of_the_review: This paper proposes an Off-Policy Evaluation problem in a model-based fashion using a generative model. This is in contrast to policy correction methods like IPS, and the Doubly Robust method. The paper is well-written and easy to understand. 

However, the claims and motivation for adding RSA and the branching architecture lack justifications, with some ambiguity in writing at places.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes the application of recurrent world model to the task of off-policy evaluation (OPE) in offline RL setting. As contributions, the paper identifies two empirical techniques to help improve performance in the offline ODE setting: (1) recurrent state alignment (RSA); (2) branching architecture. Finally, experiment results show improvement over baseline methods.
strength_and_weaknesses: === Strengths ===

The paper seems to propose an interesting application of learning recurrent world model for the purpose of offline ODE. Intriguingly, applying such world model technique does not work optimally out-of-the-box, and as the authors suggest in the paper, would require additional techniques to work better. The paper identifies such important elements as the RSA loss and proposes the novel branching architecture, which can be useful for future investigation in the model-based RL for OPE in general. Experiment results also seem solid.

=== Weakness ===

I think a major weakness of the paper lies in its conceptual contribution. One central question is: since all the tasks being evaluated are Mujoco continuous controls or Adroit tasks, arguably the environments are almost fully Markovian, then what is the principal reason why using a recurrent model works better than just using a Markovian world model? This is a key motivational question not addressed enough by the paper. I will expand more in the following.
clarity,_quality,_novelty_and_reproducibility: === Clarity ===

The paper is written quite clearly.

=== Quality ===

The paper has good writing quality and has clear presentations of experiment results. The paper also has good technical quality in terms of empirical contributions, but may have flaws regarding its motivational and conceptual contributions.

=== Novelty ===

The paper is novel in its new branching architecture and provides another application of recurrent world model to offline RL OPE. Otherwise it feels like a fairly plain application of recurrent world model to OPE that seems to obtain better performance. Overall, the novelty is mediocre.

=== Reproduce ===

Maybe reproducible since the authors have provided code.


summary_of_the_review: A few questions regarding the paper.

=== **Motivation** ===

Motivation. Since evaluated envs are Markovian, why is a Markovian world model not enough for offline OPE? I think providing a more clear and detailed investigation into this will greatly boost the motivation for using recurrent models for offline OPE in general. It can just be the case that in practice, using recurrent model is much more superior empirically, but conceptually, it remains unclear why it is the case. Is this because of statistical efficiency? Is this because of these envs are in fact non-Markovian and recurrent model is in fact being useful in a general POMDP setting? 

Overall, the paper has shown good empirical gains over prior methods. But the "why" is not clear. Thus far, I feel general readers cannot gain a better understanding as to why recurrent model is useful in OPE in general. If the paper is meant to just leverage a recurrent model training scheme, most part of which has existed in public, and show it delivers better performance, then the contributions are not novel nor insightful enough.

=== **RSA** ===

The paper proposes that RSA is very important for practical benefits, and has shown evidence through ablation. However, what's not clear to me is why RSA is important specifically in the case of OPE? What is the implication of the RSA loss function that makes the OPE more accurate? 

For example, if RSA benefits recurrent world model in general, what makes the loss function novel in the OPE case? Is this because in addition to the usual training stability benefits provided by RSA in the dreamer setting, extra benefits are present in the OPE case. Or maybe for the offline OPE case, it is even more important to carry out RSA than the dreamer case because now the data is offline.

A clear explanation and motivation for the above question will help elucidate the novelty of the paper's contribution. If RSA does not provide benefits beyond how it helps dreamer, it is not clear if RSA is a novel contribution of the current paper.

=== **Markovian world model** ===

Comparison to Markovian world model. In Fig 3 the paper compares VLBM with auto-regressive world model, obtained from prior work. An experiment to address the conceptual question in Q1 would be to compare with a Markovian world model, and maybe borrow results from prior work if there is any.

=== **Why VLBM is better than AR** ===

Given VLBM and AR are both recurrent world models, a natural question is why one is better than another? It would be nice to have visualizations like in Fig 8 for the AR model too, to see whether it is because AR model fails to distinguish policies with different performance in the representation space.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This papar focuses on the model-based off-policy evaluation(OPE) method, where there exist both limited coverage of state-action space and sensitive initialization during training the transition model. To address these, this paper presents the variational latent branching model(VLBM) to obtain as much information from the limited offline data and improve its robustness, through variational inference framwork with the Recurrent State Alignment and Branching Architecture of decoders respectively.
strength_and_weaknesses: Strength:
1. The proposed method is very detailed and sounds reasonable, and the empirical results and ablation studies are sufficient to verify its effectiveness;

Weaknesses:
1. technical novelty is limited: the proposed method is basically a straightforward extention to the VAE model in the sense that the latent variable is extended to a sequence of latent variables modeled, where both encoder and decoder is implemented with LSTMs.

2. The main concern of this paper is how to train a better transition model, but why this is the core issue of OPE needs further explanation. I think that  it's better to evaluate the presented method across more model-based RL applications rather than OPE.

3. typos: In formula (4), the left side doesn't contain a_{t}, but the right side contain a_{t}. 

4. In Fig.6, one can observe that only two weights are relatively large - what this means?  how does this affect performance?

5. it is claimed that "From Fig. 7, one can observe that it tends to significantly under-estimate", why? it seems that there has over-estimate and under-estimate in Fig.7.

6. in eq.(3), why a diagnal Gaussian is chosen to model the latent transition model?  It could be over-simplified in some complex enironment.


clarity,_quality,_novelty_and_reproducibility: fair
summary_of_the_review: This work proposed a variant of VAE in which both encoder and decoder are LSTM to model the trajectory of a RL agent and use this for OPE. I think that this is less related to the problem of OPE than other model based RL problems, and wish to see more results on those applications.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new model-based offline policy evaluation methods. It use a recurrent latent model to learn the world model from offline dataset as a VAE. In order to obtain good performance, it introduces two modifications from vanilla VAE: (1) recurrent state alignment (RSA) that introduces additional regularization to align the latent states in the encoder and decoder; (2) branching architecture that trains multiple models with different parameter initializations. Experiments on the DOPE benchmark shows compelling performance compared to baselines.

Strengths:
- Good empirical performance
- Moderate novelty in the two modifications: RSA and Branching

Weaknesses:
- Not well explained why a recurrent architecture is required for a Markovian environment
- Limited novelty under VAE framework. Not sufficient justification / explanation why the two modifications would help
- (minor) Would be better to consider the proposed recurrent world model in the more general context of offline RL instead of restricted to OPE


justification_for_why_not_higher_score: There are remaining concerns that are not well addressed by the authors' rebuttal.
justification_for_why_not_lower_score: Experiments are conducted convincingly and the results are compelling.
Decision: Accept: poster