Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors propose an approach to estimating causal effect for populations for which only observational data exists, given that experimental data exists for a related population.
strengths_and_weaknesses: **Strengths**

* The authors focus on how to exploit situations in which multiple data sets (both experimental and observational) are given, and increasingly realistic situation for problems of interest in social science, medicine, and other areas.

* The authors focus not just on obtaining point estimates, but on obtaining confidence intervals on those effects.

* The paper is well-grounded theoretically but also features an empirical demonstration of the approach on a well-known data set.

* The paper is clear about its assumptions.

**Weaknesses**

* The authors refer to observational estimates as "valid" or "invalid". Any estimate will have error due to bias and variance. More precise language would focus on the bias and variance of a given estimator, rather than a binary determination of "valid" or "invalid".

* The experiments in Section 4 are more of a demonstration than a convincing empirical evaluation of the proposed approach. The experiments employ only a single data set (IHDP), a practice which recently has been strongly critiqued (e.g., Curth et al. 2021). The approach used by the authors on IHDP could be applied to other RCT data (see Gentzel et al. 2021), but is not.  The result is that readers are left with little empirical evidence that the method works in practice, and theoretical treatment that makes a large set of assumptions that may (or may not) be valid.

* The homogeneity of causal effects is a standard (though highly suspect) assumption of most methods for estimating average treatment effect from observational studies. The authors authors assume that average treatment effect varies among subgroups, but that enough homogeneity exists that valid extrapolations can be made. However, RCTs and observational studies often differ in ways that go far beyond the randomization of treatment assignment. Randomized experiments often involve other artificial conditions that are not replicated in observational studies. Meanwhile observational studies are often substantially different from each other (and very different from experimental settings).

* For most of the paper, the authors oddly refrain from naming the proposed algorithm. This leads to odd linguistic constructions such as a section headers that read "Implementation and Evaluation of Meta-Algorithm" and "Meta-algorithm produces confidence intervals that cover the true GATE with nominal probability". Then, in section 4.2, the authors name the algorithm (Extrapolated Pessimistic Confidence Sets (ExPCS)). Use the name early and throughout the paper.

* The authors use of the term "meta-algorithm" seems unnecessary.

* The use of hypothesis tests, particularly given the large literature on the limitations of this basic framework, is open to criticism.
questions: Why is it more useful to describe to describe the proposed algorithm as a "meta-algorithm" rather than just a simple "algorithm"? Is the "meta" prefix intended to indicate that the procedure that you describe "is an algorithm about algorithms", that it is "an algorithm about meta-analysis", or something else?
limitations: To their credit, the authors are very clear about many of the limitations of their proposed approach. Clearly, the authors' approach requires that both experimental and observational data are available for a given (super) population of interest. The authors also assume that at least one observational estimate (among several) is "valid" across all subpopulations. This is a large assumption, given that any given estimate may have high bias or variance for any given subpopulation, and that those error properties are likely to vary substantially across different subpopulations. Finally, the authors assume that "every observational dataset has support for all groups." This seems unlikely. Furthermore, as the authors state clearly: "...we may reject an observational estimator due to failures in transportability, even if it yields unbiased estimates of the extrapolated effects."

Again, the authors are fairly clear about the limitations and assumptions of their proposed approach. This makes an extensive empirical evaluation all the more important as a demonstration that, even with these assumptions and limitations, the approach can produce accurate estimates of causal effect. Unfortunately, the empirical evaluation is limited to a single data set with known issues.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a meta-analysis for reliably extrapolating group level causal effects from multiple observational datasets when experimental data is available for some subgroups. The first part involves falsifying effect estimates from observational data that are biased. This is based on hypothesis testing where the statistic developed compares effect estimates from observational data to that of the RCT for groups that have experimental data. Under the assumption that at least one observational data exists for each group providing a consistent estimator (and assuming that all estimates are pointwise asymptotically normal), the statistic allows to reject biased estimates efficiently. Following this confidence intervals are generated using a simple algorithm that conservatively estimates the intervals based on the intervals of the observational data. Proposed method is evaluated on semi-synthetic IHDP data compared to simple meta-analysis and baselines that do not consist of falsification. 

------------------------------------ Post rebuttal update ---------------------------------------------------------------------------------------------

I have read the full author response and I believe they address my major concerns with the paper. I have updated the score based on the response. 
strengths_and_weaknesses: Strengths:
1. The proposed falsification method is interesting although feels impractical due to concerns/clarifications I mention below.

2. The paper is well written, assumptions are clearly stated.

3. The simplicity of the approach is appealing.

4. Interesting experimental results.

Weaknesses:
1. It is really unclear how often RCT data could be available that could provide consistent group level effect estimates. This implies that the design of the RCT itself needs to be explicitly targeted for estimating group effects. Hence I am not sure how practical this approach really is.

2. Although the IHDP results are interesting, and evaluation specific to the data is thorough, I believe just one semi-synthetic evaluation is fairly limited to be convincing.
questions: 1. Regarding my concern on practicality, can the authors suggest situations/RCTs that are explicitly designed to provide group level effects (this is different from subgroup analysis, which is not something one should do either way).

2. I would strongly encourage adding additional experiments, I have briefly seen the code and it will genuinely strengthen the paper.

3. Relating methods used for effect estimation in observational data to quality of falsification. This is clear to me based on the main assumption of asymptotic normality, it is however unclear if this is the only assumption needed? I am assuming all results use the DML estimators since they satisfy the assumption? 

4. Conceptual clarification: Based on falsification, why can't a stronger statement be made about an estimator that aggregates the extrapolated effects?

5. Not sure why but some proofs in the appendix are made unnecessarily concise. Please expand and make everything clear.
limitations: I believe the authors have adequately described limitations of their work. Based on my clarification questions, please consider adding comments on practicality in the limitations section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Randomized controlled trials are high-standard with inclusion criteria in recruiting patients but may fail to include some heterogeneous patient subgroups in the full population. On the contrary, large-scale observational studies are likely to contain more diverse patient subgroups but they can be invalid to use due to hidden bias from some unmeasured confounders. The idea of this paper is to first validate the estimators from observational studies by comparing them with the estimator based on RCTs. This comparison is made on the patient subgroups that are observed in the RCTs. This step filters out the observational studies and their estimators that are inconsistent with the RCTs. After that, the authors use the non-rejected estimators and construct conservative confidence intervals to extrapolate the treatment effects for the subgroups that are not observed in the RCT.
strengths_and_weaknesses: Strengths
1. The paper recognizes the advantages and disadvantages of RCTs and observational studies and lets them complement each other in the method proposed to estimate group average treatment effects. The idea is interesting and well motivated!
2. I am convinced by Assumption 2.3 that at least one observational estimator is asymptotically normal and consistent for both the validation and extrapolated effects, and Assumption 2.2 which says that the RCT estimator is also consistent.  I think the assumptions are necessary so that at least in large samples, we can pick up the consistent observational estimator to estimate both effects.
3. The experiments uncover both the power and conservativeness of the proposed method. I still think the proposed idea is useful in practice.

Weaknesses
1. The groups $I_R$ and $I_O$ are given instead of being learnt from the data. I don't think this is often not the case in practice. We may know something about the effect heterogeneity but it is too strong to assume our knowledge is close to the ground truth.
2. The proposed method (Algorithm 1)  is not very novel and a bit trivial, which includes some standard techniques we often do in practice, e.g. constructing asymptotic normal estimators and doing a t-test. The author does not do much to make the method powerful, i.e., less conservative. I feel all the theoretical results are expected and follow from the DML literature.
3. The method looks quite conservative in Figures 3 and 4. The interval width is similar to the simple Union, which reports the union bounds of the confidence intervals of all observational studies, with no falsification procedure.
questions: Why only consider group average treatment effects? could we extend the method to other causal parameters?
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary:  Randomized controlled trials (RCTs) are considered the gold standard for studying the causal relationship between treatments and outcomes, and  Clinical Practice Guideline (CPG) policy recommendations are based on experimental results from RCTs. However, due to the cost (time and money) and ethical and methodological considerations, the populations in RCTs are narrow. Hence, the treatment effects outside of the support are missing. An alternative approach is to estimate treatment effects using observational data. However, the estimated treatment effects from these historical data might be biased due to the failure to control confounding effects or the existence of selection bias in the data. This work considers the problem of providing unbiased treatment effects, along with the confidence intervals, outside of the support of RCTs from the existing observational datasets. Given RCT data and multiple observational studies, this work first provides a hypothesis-testing technique to remove the observational estimates that are not consistent with the estimate of RCT. Under the assumption (Assumption 2.3) that there is a least one observational estimator that is a consistent estimator of RCT, i.e., the strong ignorability assumption holds, and the confidence intervals on the extrapolated treatment effects outside of the support of RCT is then can be provided. Throughout the experimental validation of the IHDP dataset, this work shows that their meta-algorithm can provide the confidence interval that covers the true GATE and with narrow width.
strengths_and_weaknesses: Strengths:

1. The paper is very written: Problem formulation, notations, assumptions, and derivations are clearly provided.
2. Empirical results are compared with existing meta-analysis and the comparison is comprehensive.

Weakness:

1. A lack of a real-world dataset is provided.
questions: 
This work provides a very solid meta-algorithm that is able to provide unbiased causal effects with confidence intervals from RCT and multiple observational datasets. However, the major question that comes to my mind is whether the use case of this approach is limited or not. I am not an expert in this area. Is it the case that many of the observational studies (or RCTs ) have a function mapping that $$ G: \mathcal{X} \mapsto \{1,\dots, I\}.$$ and the function mappings across studies are consistent so that the subsequent analysis can be provided from the meta-algorithm developed by the authors. 

Perhaps it is challenging, but would it be possible to provide a real-world experimental validation using the publicly available clinical database, such as the MIMIV-IV database.

limitations: This work provides a meta-algorithm that can report unbiased causal effects outside of the support of RCT with confidence intervals. This work's only potential negative social impact might happen when they falsely accept the biased estimator from an observational study that is not consistent with the estimator of RCT. The author may provide more examples and cases of what would happen when this case is true.
review_title: null

Meta-review: metareview: The authors propose an approach for estimating causal effects when both observational and limited experimental data exists. The authors propose falsifying effect estimates from observational data before using the effect estimate on other populations. This is an important idea that may improve reliability of causal inference. The authors provide confidence intervals for the proposed procedure. The considered problem is of clear importance; and the simplicity of its approach is appealing (cPQd). There have been some concerns about the limited empirical evaluation (icYd). The authors provide additional numerical evidence during the rebuttal period. This evidence should be added to the appendix for the camera-ready version. 

Note: The reviewer most critical of the paper (rating 4, icYd) does not seem to have updated their score post-rebuttal.
confidence: Certain
Decision: Accept