Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary:
The present paper proposes a novel approach for model selection for individual treatment effect (ITE) estimation in the unsupervised domain adaptation (UDA) setting. The motivation for this approach, called interventional causal model selection (ICMS), is to exploit causal invariance to choose a model that has both good predictive power and fits the a priori belief for causal relationships.
More precisely, the authors prove a necessary condition for optimality of an ITE model and combine this condition with the classical target risk minimization from UDA model selection.
This necessary condition states preservation of all conditional independencies of the causal DAG by the interventional distribution of the target domain.

Recommendation:
Clear accept. The framework proposed in this work constitutes an interesting new approach to model selection exploiting ideas from causal invariance and also allows to adapt existing ITE estimation methods to tackle covariate shift problems.

Strong points:
 - This framework is theoretically motivated, using the structural causal model framework, and allows to improve various methods by allowing for model selection in UDA setting, as illustrated on a wide range of examples (simulations and real data).
 - The flexibility also extends to the quality of the data or prior knowledge, as this framework allows to incorporate as much expert structure knowledge as possible, or to specify its importance in the model selection via the hyperparameter $\lambda$.

Weak points:
 - The role of the causal DAG is key to this approach, it is however difficult to assess from the experiments and discussion, how much the proposed framework relies on the correctness of the used DAG, or, put differently, how sensitive this framework is to mis-specifications of the DAG.
 - It would be interesting to see the code for the presented examples and also to test it on other applications. Will it be made public at some point?
 - The presented work is considerable and it many experiments are provided. However I think that the experiments section is too long compared to the main section (Section 4). For instance, (parts of) the appendix A could be moved into the main part while shortening the Section 5.2.

Questions/Issues:
 - Sec. 5.2.: Reference for FGES is missing in the main part of the article (it's in the appendix but should also be in cited in Section 5.2)
 - Appendix B (additional related work) should be moved to the main part (Section 2).
 - Sec. 5.1: What is the reason for only shifting one ancestor of $Y$ in $G$? And can it also be, at the same time, an ancestor of the treatment, i.e., a confounder in the observational data (I think yes, since the intervention on the treatment variable cuts of incoming arrows, but has this been tested in practice as well?)?
 - Sec. 5.1: How do the authors choose the 30 candidate models for each model architecture? Is the variability across the different models similar for all comparable for all compared architecture?

Minor comments (that did not impact the score):
 - p. 3: unconfoundness $>>$ unconfoundedness
 - p. 6: i.e $>>$ missing comma (i.e.,)
 - p. 6: return $>>$ returned
 - p. 7: For completeness, add reference for domain shift between rural and urban populations.
 - p. 7: hospital admission $>>$ hospital admissions
review_title: Interesting novel approach combining treatment effect estimation and domain adaptation, leveraging the concept of causal invariances.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper introduces a model selection metric for ITE models under the unsupervised domain adaptation setting. By modeling the data from a source domain, the metric is able to select optimal ITE models that perform well on a target domain, where only unlabeled data available. Experiments are conducted on both synthetic and real-world datasets.  

+ The problem is significant in machine learning and the idea is incrementally innovative: causal inference under UDA setting. 
+ The idea flow is logical. The paper is very well-written.

Questions:
1. How to evaluate the generated causal graph? The causal structure learned from the source domain is used to estimate the causal risk on the target domain. How to evaluate the goodness of a learned causal structure? Especially in the real-world dataset, the causal graph could be much more complex than the synthetic setting. How to evaluate the dependencies among each pair of nodes? 

2. How to evaluate the parameter lambda? In equation (8), a parameter lambda is used for controlling the weight of NCI and set to 1. Additional experiments should be conducted to evaluate the performance under different values of lambda. 

3. Some notations are not clear.  
- In equation (7), the definition of NCI is not clear. A graphic illustration of NCI results in the causal graph should be provided. 
review_title: A very well-written paper with incremental innovation

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################

Summary:

This paper proposes a novel interventional causal model selection (ICMS) score to select individualized treatment effects (ITE) models under the unsupervised domain adaption (UDA) setting. The problem is fundamentally challenging as counterfactual outcomes cannot be observed. The authors make an assumption that the underlying causal structure across domains remains unchanged when adapting the ITE models from one domain to another. The authors propose Theorem 1 that the conditional independence relationships in the interventional DAG are equal to that in the interventional distribution for the target domain, followed by augmenting the target domain data with the model's prediction of the potential outcomes. Then the model that generates the best augmented target data in the sense that matches best with the interventional DAG is selected. To access this fitness, authors use the negative log-likelihood of the interventional DAG given the augmented data on the target domain. Finally, results with both synthetic data and real-world COVID-19 dataset show improvement for all ITE models.



##########################

Pros:
 
1. This paper tackles an important and challenging task: adapting ITE models unsupervisedly with unobserved counterfactual outcomes.
2. This paper is overall theoretically sound. The core assumption of causal structure invariance makes sense to me and the experimental results show significant improvement for all SOTA ITE models used.
3. This paper is well organized and easy to follow.

Based on these, I tend to vote for acceptance.


##########################

Cons: 

1. In Eq (9) and Fig. 2, is the validation risk $v_r$ computed using the source dataset? This seems unjustified given that the risk computed in the source domain might not be a good approximation of that computed in the target domain.
2. In Eq. (9) and the paragraph ``Assessing causal graph ﬁtness'', $c_r$ is computed by the negative log-likelihood of G given the augmented dataset Dhat_{tgt}. This fitness makes sense to me: the model that generates the augmented dataset in target domain that best align with $G_{\bar{T}}$ should be selected. However, what is the relationship with Eq. (7-8)? It does not seem straightforward to see how Eq (9) and the NCI term are linked up.


review_title: Theoretically sound and good experimental results

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: the paper attacks the problem of model selection for individual treatement effect (ITE) models when the domain of learning and prediction differ. Proposal is to use causal consistency as an additional "regularizer" in existing domain adaptation (DA) model selection methods. The "regularizer" would be scoring to which extent replacing factual outcomes by their counterfactual predictions would preserve conditional independence relations (induced by the causal graph) in the prediction domain. Experiments on a variety of datasets are conducted to show the added performance induced by using the "regularizer".

Good points:
- novel and very creative idea
- excellent writing that introduces concepts as they are needed + illustrative figures (Fig. 1 and 2)
- rigorous exposition of assumptions - quite important in this kind of problems
- interesting, illustrative use-case on covid

Questions:
- Is the dataset on covid openly accessible ? 
- It seems that all base models are deep i.e. involving a non-convex optimization problem that makes the model prone to initial solution. Have you taken the "choice" of the random seed into account in the same manner as the hyper-parameters to select the model ?

Points currently limiting the relevance of the paper:
- [impact] model selection is here studied in isolation to other related techniques, notably causal feature selection (e.g. https://arxiv.org/abs/1911.07147) - it seems to me that it thus limits notably the generality of the conclusions as I don't see why practitioners would use one technique without the other. It is a big question mark for me and I'd like to read the point of view of authors on that point.
- [relevance of baselines] In related works authors mention two methods that they place as "state of the art" for ITE model selection: Causal Assurance and Influence Functions. Even if they are not designed specifically for the DA use case why not include them in the experiments ? It is often the case in practice that methods not designed for a specific case are unexpectedly robust to more specific setting. In particular I think it would be most telling to compare an augmented/"regularized" method (e.g. DEV+ICMS) vs generic SOTA methods (CA or IF). Disclaimer: I don't know enough of these methods to evaluate how easy it would be to perform such a comparison.
- [metrics] The standard PEHE metric seems to have been replaced by PEHE-10. I can't seem to find the definition of this metric in the paper ? why not use the standard ?
- [reproducibility] I can't seem to find the hyper-parameter grid that was used for the different base models and among which the competing methods would choose the best candidate - even in supplementary ? To allow other researchers to independently reproduce your results this is just mandatory. 
- [reproducibility2] I can't find details on your strategy for seeding the different models. How many random seeds ? are they considered as hyper-params or part of an inner optimization loop ? 
- [insights on results] The provided result in Table 1 indicates that adding ICMS to DEV or IWCV uniformly improves model selection. It seems in a sense "too good to be true". Can you provide intuition on why it is so ? Could it be possible to check that using ICMS alone is not already a very strong baseline - or is it in combination only that it works ?

Points that could improve the paper (and my score):
- provide a self-contained formula of NLL directly when introducing it in Sec. 4
- give full hyper-param grid for all models in supplementary
- define PEHE-10
- clarify setting for random seeds
- add an experiment with the Causal Assurance and Influence Functions baselines
- add an experiment comparing best model selected by proposal (e.g. DEV ICMS) vs best model found by causal feature selection (e.g. Rojas-Carulla 2018 or Magliacane 2018)

Overall, it seems to me the work deserves a more rigorous evaluation part as the underlying idea is very interesting and seems to be a very strong addition to the causal toolbox for ITE modeling.
review_title: Very interesting, novel idea yet maybe needs to be linked more to other methods and evaluated accordingly

Meta-review: This paper considers the problem of identification of causal effects under the unsupervised domain adaptation setting. The authors assume the invariance of the causal structure and use it to regularize the predictor of causal effects. The method is interesting and looks effective, although this assumption may not hold always true (e.g., in some domains, some causal influences may disappear, leading to extra conditional independence relations). Hope the authors will update the paper to address the concerns raised by the reviewers, especially to conduct a sensitivity analysis of the framework to misspecification of the causal structure and make the motivation for the used evaluation metrics clear, and also provide a more thorough review of related work.
Decision: Reject