Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a framework for embedding interventional distributions into RKHS that accounts for uncertainty in causal datasets especially when the confounders, treatments, mediators, and outcomes are not all observed in a single observational dataset. The main problem that is addressed here is quantifying the uncertainty in the estimation of causal effects when the treatment and the outcome are measured in two separate sources (potentially under different data-generating conditions) and the aim is to estimate the effect from a causal data fusion perspective. The paper introduces Bayesian Interventional Mean Processes for representing interventional distributions in the reproducing kernel Hilbert space. The method is applied to the task of Causal Bayesian Optimization where it is shown that the method converges faster as compared to other baseline methods. The main contribution of the paper is obtaining the uncertainty estimates for $E\[T\mid do(X) = x\]$.
main_review: The idea of quantifying uncertainty about causal estimates especially when the treatment and the outcome are not observed in the same setting is of significant importance but relatively understudied. The proposed approach definitely builds in this direction. Relation to prior work is discussed and cited when the approach builds on ideas developed therein.

The claims are well-supported to the best of my knowledge.  The write-up is generally clear. While the example on estimating the effect of statin drug dosage on cancer volume is helpful, it would also help the reader to present some of the real-world settings where the approach may not succeed as expected, for example, in the case of recommender systems with high-dimensional confounding and higher uncertainty about the causal graph as well.

The results will aid practitioners in quantifying the uncertainties with respect to the causal effect estimates as well as guide in better data-collection practices if the uncertainties span over a larger region and estimating the effects with data from two different sources may not be accurate. It would help to address/analyze with the setting where the data is from a single source but the causal graph is uncertain, this can be an interesting future direction as well. 



limitations_and_societal_impact: Given that the proposed approach aims to combine causal information from multiple datasets that may not all be collected under the same settings, it can have several potential impacts in fields like healthcare. The paper discusses this to some extent.

The limitations are discussed with respect to assuming complete knowledge of the causal graph. However, there could also be challenges with obtaining enough samples across the two sources, in which case there would be additional uncertainty because of representation bias. It would be helpful to elucidate the specific challenges and extensions in this case.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper tackles the problem of computing the average causal effect given data generated by distinct (fragmented) graphs which does not contain the same set of variables. It uses kernel mean embeddings to represent interventional distributions, marginalize out the common variable Y and account for statistical uncertainties. 
main_review: Firstly the problem of fusing datasets consisting of different variables  from distinct data sources and using them to answer questions is a non-trivial  and important task. So this is an important and interesting work. 

My main concern here is the assumption T || do(X)|Y. Note that T and Y are variables and do() is an operator. As such it is strange to see an independence claim expressed between  a variable and an operator. Normally, we rely on rules of do-calculus to remove or add do(x) operators to factors in an estimand. In this case the variables do not  belong to the same  causal model (but to two distinct graphs) and as such it is not clear how to directly apply do-calculus. 

A) By T || do(X)|Y, do you mean that in the true data generating model G that generated P(X,Y,Z,T) (which is unavailable), X is not a direct cause of T and that all causal paths from X to T are mediated by Y? If so, explicitly state this. If not, please explain this to me. 

B) Wouldn’t a minor modification of your results make them applicable to the case where Z is not a common cause of X and Y but related to them in the following way: X-->Z-->Y? 

C) Also in the model you have considered, why are there no confounders between Y and T? Realistically, wouldn't that be the case?

     
More importantly, I think the authors need to put in a bit more effort to make their results more accessible to the general audience. Even in the case of causal inference researchers, not everyone can be expected to be familiar with RKHS and hence might not be able to make use of the results presented here. It is important therefore that the authors add a sentence or two after each proposition describing it in english (without RKHS jargon). 

limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper studies the problem of uncertainty quantification for causal data fusion.  They introduce Bayesian Interventional Mean Processes, a framework that represents the interventional distributions using kernel mean embeddings and probabilistic integration. The framework is able to measure the uncertainty within the causal graph in each dataset. In experiments, the author demonstrates the performance improvement of the proposed method on the causal Bayesian optimization task.
main_review: The setup is clearly related to the problem of instrumental variable (IV) regression. Besides the difference that the regression at different stages are based on different datasets, the connection with IV can be seen by changing the treatment variable to an instrumental variable and changing the mediating treatment variable to a treatment variable. 

If this is the case, I think the method may be better suited to be presented in the potential outcome (PO) framework rather than the framework of structural causal models. Under the PO framework, you should provide the same assumptions for instrumental variable (IV) regression. For example, the treatment variable (the instrumental variable in IV) has no direct effect on the target variable; there are no unmeasured confounders for the treatment variable (the instrumental variable in IV) and the target variable. Because the paper is about uncertainty quantification, I found putting down these formal assumptions becomes more important. For instance, the biggest uncertainty in causal inference can be due to some unmeasured confounders of the treatment variable (the instrumental variable in IV) and the target variable. Later on, I see you provide two assumptions in lines 155-157. I do believe these assumptions should be made earlier, e.g., in Section 2.1.

In the introduction, I think the author should provide more detailed explanations of epistemic uncertainties and aleatoric uncertainties. Bayesian methods are not mainstream in causal inference. I don’t think causal inference researchers are familiar with these two concepts. In line 74, you mention “[17] considers a frequentist approach, which does not account for epistemic uncertainties”.  Not all the frequentist approaches measure the uncertainty using the asymptotic confidence intervals. For example, bootstraps or subsampling can be used to quantify uncertainties. It is unclear why we should use the Bayesian approach to quantify the uncertainty instead of the frequentist approach. It is possible that the user chooses a wrong prior on the Gaussian process models, then the Bayesian uncertainty is also not very well calibrated, i.e., the epistemic uncertainties are not quantified in the right class of models.

The proposed method is presented in a clear way. However, I don’t think making something Bayesian is very novel technically, given what has been done on Interventional mean embeddings and the connection between kernel ridge regression and Gaussian process regression. Is there any technical challenge due to using multiple datasets your method overcome but I didn’t realize?

In experiments, the ablation study should report the coverage rate at a range of sample sizes, dimensions and simulated functions. It is not convincing to see just the performance of one function, dimension and sample size. Causal Bayesian optimization is an interesting application but may be too complicated. I understand fast convergence in Bayesian optimization means the method estimates the function mean and variance very well. I expect to see some simple experiments on treatment effect estimation, in which you can disentangle the gain from estimating mean and variance. Also, Bayesian optimization is not effective at every iteration; it just improves the value to optimize in some steps. Overall, I think doing experiments on treatment effect estimation directly is more transparent. For your healthcare experiments, learning dosage in practice is usually not that straightforward. Increasing the dose level also increases the toxicity. There should be an unknown safety constraint on your intervention. Bayesian optimization in your healthcare example should take this into account for the method to be used in real-world applications.

There are many recent works (see below) in causal inference that try to combine multiple datasets. They are not directly very relevant to your paper. But your paper title and abstract are too general without specifying the problem setup you focus on. 

Shu Yang, Peng Ding. Combining multiple observational data sources to estimate causal effects. https://arxiv.org/abs/1801.00802

Katherine Evans, BaoLuo Sun, James Robins, Eric J. Tchetgen Tchetgen. Doubly Robust Regression Analysis for Data Fusion. https://arxiv.org/abs/1808.07309

Wenshuo Guo, Serena Wang, Peng Ding, Yixin Wang, Michael I. Jordan.
Multi-Source Causal Inference Using Control Variates. https://arxiv.org/abs/2103.16689

E Rosenman, AB Owen, M Baiocchi, H Banack. Propensity score methods for merging observational and experimental datasets. arXiv preprint arXiv:1804.07863




 
limitations_and_societal_impact: Yes, they do.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a Bayesian approach for estimating the treatment effect by combining two data sources. The main advantage of the Bayesian approach over the frequentist counterpart [17] is its ability to quantify uncertainty. The main machinery was GP which replaces KRR used in existing works. 
main_review: The paper is very well written, experiments are adequate to illustrate the strength of the approaches, and the proposed idea has some interesting contributions (i.e., causal uncertainty quantification in the data fusion context) to the existing literature. In terms of potential applications of the proposed methods, I believe there are plenty in medicine and other fields where data are collected from multiple sources.  Therefore, I believe overall the quality, clarity, and significance of this work are high.

One concern is, however, regarding the originality. The novel elements of this paper are all based on established techniques, e.g., the IMP extends [24] and IME extends [25-27]. The idea of using RKHS is also not particularly new. However, the utilization of these various existing techniques in a meaningful way in this particular context is, in my opinion, novel. 

======update=====
Thanks for the authors' responses. I keep my original score. 
limitations_and_societal_impact: The limitations are adequately discussed and the potential negative societal impact is briefly discussed.
review_title: null

Meta-review: All four reviewers advocate acceptance. I also recommend accepting the paper for its contributions to the emerging field of Bayesian causal inference.
Decision: Accept (Poster)