Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the model learning problem in sequential decision making. Traditional methods may learn a model that fails under counterfactual samples (unseen in offline training dataset). To imporve the model's generalizability, the authors introduce adversarial weighted empirical risk minimization (AWRM) which learns a model with adversarially selected test-time policies. Although the original formulation of AWRM is intractable, the paper proposes a surrogate objective with several approximations. The effectiveness of the proposed method is verified by experiments in a synthetic environment, 3 MuJoCo environments, as well as a real-world food-delivery platform.
strength_and_weaknesses: Strengths:

1. The main idea of the paper are well-motivated and interesting.
2. I am not sure how novel and original the relaxation of the adversarial objective is since I am not familiar with the related work. But the technical insights are sound.
3. The paper is in general well-written with much details well-organized in the appendix.
4. The experimental results are convincing in terms of showing the major messages.

Weaknesses:

1. It is not clear how the entire algorithm works and how many networks are learned in total. A high-level summarization or even an architecture figure would be helpful.
2. Many implementation details are omitted in the main paper. How is $H_{M^*}(x,a)$ implemented?
3. The presentation could be improved. The notations are sometimes not very clear. Are $\kappa$ and $\hat{\mu}$ (in appendix) the same? The notation $a$ is used several times for different meanings. Also, the layout is too tight. It seems that a negative vspace is used very aggresively.
4. The experiments only focus on the model learning performance in the form of prediction errors. However, it would be better to demonstrate how such a model can help model-based RL (learn a policy from the fitted model).
clarity,_quality,_novelty_and_reproducibility: The paper is mostly clear. The idea and technical contributions look novel to me.
summary_of_the_review: This paper addresses the generalization challenge of model learning in sequential decision making problems. The proposed method is reasonable and sound, with theoretical analysis and empirical evidence.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces an approach for learning a model of the environment for model-based RL (MBRL) based on counterfactual data. This enables more stable model performance outside of the training distribution. The added generalization is driven by adversarial training to mitigate selection bias influencing the learned relationships between observations and subsequent actions.
strength_and_weaknesses: **Strengths**

This is a very complete and thoroughly written paper. The extensive efforts made by the authors to clearly justify the steps they took in the development of the proposed GALILEO is appreciated. I do wish there was a more detailed "related work" section with more concrete discussion about how the proposed approach builds or differs from prior work. However, this concern is mitigated mostly by how well cited the rest of the paper is. Each addition and concept throughout the paper is properly attributed.

I found the step-by-step derivation of GALILEO through Section 4 to be really easy to follow. The writing was very clear in the development of the final use of AWRM. This was capped off with a great discussion about how the theoretical components of the method are implemented (Section 4.3). Unfortunately, this level of detail (and accessibility) was not carried through the experiments (more on that below). 

The extent of the experiments provide some clear signals that the proposed method carries some "general" significance outside of the simple synthetic domains many papers use as a proof of concept. It's clear that the models learned with GALILEO are more accurate than those trained with other approaches.

**Weaknesses**

To quickly jump to the empirical justification. It's disappointing that the improved models learned with GALILEO aren't shown to improve the overall task performance of the individual domains the models were learned to assist with. I can understand that this is not the major focus of the paper but the conceptual justification for needing generalizable models for MBRL is to provide more robust policy learning. As such the reported results feel only partial in nature. It would be great to see what the knock-on effects of an improved model are when learning a policy in the environment. Is there an improved nature of sample efficiency? How about the performance in more diverse test scenarios?

It's also disappointing that there aren't very thorough baseline comparisons to contemporary MBRL approaches in the accuracy of the learned models. Aside from the simple synthetic tasks, there are not really any baseline comparisons with "competing" approaches. I suppose that the SL baseline is better than nothing but the experiments overlook contributions made by the prior work in learning adequate models. Without a discussion about how these approaches devolve into SL, it's hard to place any significance on the experiments.

At times the discussion is cast in very formal language that distracts from the actual usage (or motivation) of the concepts. It's technically correct to cast "error" as "risk" (especially since the paper pursues "risk minimization") but it comes across as unnecessarily formal in places. Especially when MSE is used as a metric... This isn't a major concern but it does belie some aspects of where the writing in the paper could be improved to facilitate greater clarity. 

The "real world" example is too vague in the initial sections of the paper. There is a lot of weight placed on the contributions of GALILEO working on a real domain but there's not sufficient framing or discussion about why this experiment works or is necessarily best solved by GALILEO (esp. when not compared to relevant baselines). 

There are missing references to prior work using counterfactuals to improve environment modeling. In particular, I would interested to hear from the authors why they omit contributions from prior work such as Sontakke, et al (ICML 2021).

```Sontakke, S. A., Mehrjou, A., Itti, L., & Schölkopf, B. (2021, July). Causal curiosity: RL agents discovering self-supervised experiments for causal representation learning. In International Conference on Machine Learning (pp. 9848-9858). PMLR.```


clarity,_quality,_novelty_and_reproducibility: *Clarity*: Great for the most part (detailed concerns above)

*Quality*: Very Good

*Novelty*: There are some minor concerns about significance since the experimental results barely touch on how the improved models contribute to better policies (and only for the "real world" example). The lack of discussion about relevant causal environment modeling papers is of some concern.

*Reproducibility*: There is an extensive section in the Appendix that outlines how the methods were implemented with corresponding pseudocode. 

summary_of_the_review: This is a good paper and one that warrants publication. I however reserve my enthusiastic recommendation because there are points where the discussion could be improved, specifically around the empirical justifications used to validate the development of GALILEO.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors address the age old offline RL problem of counterfactual prediction, i.e. what would have happened if an unobserved action would have happened at an unobserved state. To do this, they use a weighted empirical risk minimization framework, which is extended with an adversarial framework to learn a model under a worst-case assumption. 

They verify that their model makes accurate counterfactual predictions on unseen data by evaluating on D4RL and a propriety real-world food delivery task. 
strength_and_weaknesses: Strengths:
- addresses an important problem
- applies model based RL to offline RL, a promising avenue of research
- applies adversarial worst-case ideas, which should in theory result in better 

Major Concerns:

Clarity: It took a lot of reading and re-reading for me to understand what this paper is about. The main thing that made it hard for me to understand is that it's sold by the authors as a RL paper, when in fact their method is more a Domain Adaptation method, where they take data from one domain (observations of a transition function under one policy) and try to make predictions in another domain (behavior of a transition function under another policy). This is confusing, since what one is actually interested in is off policy evaluation, knowing how well a an new policy will work based on observation of an old policy.

Quality: I do not thing the authors claims are supported by experimental evidence. The problem with the presented experimental evidence is three-fold: First, they don't compare to other state of the art domain adaptation methods, even though it's a domain adaption paper. Second, their experiments only talk about prediction accuracy of the transition function. It's totally possible to predict accurately enough an environment, but be very inaccurate at predicting a policy's performance in the environment. Last, the authors claim their method's "policy significantly improves performance" but only show it for a non-reproducible setup. In RL, the whole point is to improve a policies performance, and the authors can't demonstrate that their method indeed improves performance.

To be frank, since the authors were unable, in my view, to support their claims with evidence I didn't carefully study every mathematical detail of their method.


clarity,_quality,_novelty_and_reproducibility: Reproducibility: As an experiment, the authors deploy their policy in a "real-world large-scale food-delivery platform", which is clearly not reproducible. 
summary_of_the_review: The authors fail to show their method is relevant for reinforcement learning, as the method merely learns a prediction of the transition function, but they don't show this leads to better policies or better off-policy evaluation.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors propose a new way for learning models within the model-based RL framework. They propose an adversarially weighted IPS ERM framework, that tries to find models that perform well under multiple target distributions. Even though the paper offers a promising approach with good empirical performance, the initial submission offered weak experimental evidence and was not clear on the positioning of the paper. The authors improved their submission on both fronts, but still seems the paper would be more appropriate for a re-submission.
justification_for_why_not_higher_score: Clarity of exposition
Decision: Reject