Constructing Confidence Intervals for Average Treatment Effects from Multiple Datasets

Authors: Yuxin Wang, Maresa Schröder, Dennis Frauen, Jonas Schweisthal, Konstantin Hess, Stefan Feuerriegel
Primary area: causal reasoning
Keywords: Causality machine learning, Average treatment effects, Confidence intervals, Prediction-powered inference

Abstract:
Constructing confidence intervals (CIs) for the average treatment effect (ATE) from patient records is crucial to assess the effectiveness and safety of drugs. However, patient records typically come from different hospitals, thus raising the question of how multiple observational/experimental datasets can be effectively combined for this purpose. In our paper, we propose a new method that estimates the ATE from multiple observational/experimental datasets and provides valid CIs. Our method makes little assumptions about the observational datasets and is thus widely applicable in medical practice. The key idea of our method is that we leverage prediction-powered inferences and thereby essentially `shrink' the CIs so that we offer more precise uncertainty quantification as compared to na{\"i}ve approaches. We further prove the unbiasedness of our method and the validity of our CIs. We confirm our theoretical results through various numerical experiments.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729156368100

thread_type:
official_review

<thread_object>
creator: reviewer_pijf

<content>
summary:
The authors present a method, based on prediction-powered inference, for estimating confidence intervals from two datasets, one with observed confounders and one with additionally hidden confounders. Theoretical and empirical results are provided.

strengths:
- The work addresses a clear research gap with respect to existing work.
- The provided theoretical results support the method.
- The method is presented clearly, and the work is easy to follow.

weaknesses:
- The current empirical analysis is limited, with mostly quantitative results and only linear models.

- Due to the required assumptions, the practical relevance of the work is not entirely clear to me.

- The conclusion is an afterthought in the current version, with no discussion of limitations or directions for future work.

questions:
1. **Empirical results:**
  - Mostly qualitative results are provided for the synthetic data. I would be interested in seeing the coverage and RMSE of each method with respect to ground truth, which should be known as the data is simulated. It seems like the proposed method has a higher bias, but lower variance.
  - In my opinion, it would be interesting to experiment with settings with more data in $\mathcal{D}_1$ to see when only using $\mathcal{D}_1$ becomes better.
  - How can the RMSE be calculated for the real data, when the true ATE is unknown? 
  - Unless I am mistaken, it seems that only simple learners are considered, based on linear and logistic regression (Appendix E3)? If so, it would be insightful to also compare with more advanced ML algorithms (e.g. gradient boosting).

2. **Practical significance:**
Although the work is interesting from a theoretical perspective, I wonder at its practical significance. How realistic are your assumptions? How can a practitioner verify them and trust your method?

3. **Conclusion:** A more detailed discussion of limitations and directions for future work would be appreciated.

___


**Minor points:**
- Brackets missing in the citations on lines 48 and 49.
- Gray text in Table 1 is not applied to all columns.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729176855091

thread_type:
official_review

<thread_object>
creator: reviewer_sim7

<content>
summary:
The authors propose a new method to compute confidence intervals of the Average Treatment Effect (ATE) of the Risk Difference based on two observational data sets, the first one of small size in which all confounders are observed, the second one being larger with potential unobserved confounders. They use the largest data set to debias the estimation of the ATE computed on the first data set. The proposed confidence interval is shown to have the correct asymptotic coverage. Experiments on a toy data set and on real-world data sets show that the proposed method leads to smaller unbiased confidence intervals.

strengths:
The paper focuses on a very important topic, which is uncertainty quantification of point estimate for the ATE, based on observational data sets. Obtaining accurate confidence intervals for the ATE helps to understand the effect of a treatment (e.g., drugs). Observational data are becoming more and more common in causal inference. The main contribution of the paper is to propose a new way of computing confidence intervals based on two observational data sets. The formula is rather simple and easy to use, while it can be applied to a wide variety of estimators (in the paper, AIPW or IPW are used depending on the context). 
Experiments tend to show that the proposed method is better in terms of coverage and IC length.

weaknesses:
There is a potential practical interest for the confidence interval and the point estimate developed in this paper. I have two majors concerns with the paper: 
- the experimental results on the toy data set are inconclusive : the input variable is of dimension one, which does not correspond to a real-world problem. For the conclusion to hold, I would expect more experiments on more complex settings (more input variables, with or without dependence, how relaxing the assumption of all observed confounders in the first data set may impact the estimation performances). Experimenting on one single input variable does not allow us to obtain a clear comprehension of the performance of the proposed method. 
- Theorem 5.2 is the main theoretical result of the paper. It is claimed below that it does not directly result from the Prediction-powered Inference (PPI) framework. However, going in detail through the proof, I could not find an argument specific to the causal inference framework or the AIPW estimator. The proof relies on Central Limit Theorems of two independent quantities.
Thus, I find the novelty of the contribution mild from a theoretical perspective. Besides, in my opinion, more experiments should be carried out to assess the performance of the method.

questions:
- Position of the work: it is assumed that the small data set contains all confounders. The major difference with a RCT is that the propensity score is unknown and depends on the covariate (as discussed in page 4, last paragraph). This justifies the use of IPW with the true propensity score (step B, page 7). However, numerous work have shown that even in a RCT scenario where the true propensity score is known, estimating it can reduce the variance of the ATE estimate (see, e.g., references in https://arxiv.org/pdf/2303.17102). Thus, in a RCT, one would likely use an estimation of the propensity score inside the IPW estimate. This point should be discussed and may lead to qualify statements about differences between RCT and the small data set you consider. In particular, could we apply methods developed for RCT (van der Laan et al., 2024) to the first experimental setting? Paragraph l.119-121 could be modified to make explicit the differences (if any) between RCT+observational and the considered setting observational+observational. 
- Regarding the work of van der Laan et al., 2024, it is mentioned that their method is unstable due to a matrix inversion (l.509-510). I went quickly through their paper and did not find such an operation. Could you be more specific, and maybe describe with more details how this method works, as it is the main competitor?
- l.151-152 ``PP CI s smaller than the classical CI when the model $f$ is sufficiently accurate.'' Is there a specific reference for this fact?
- l.203-204: It is usually required that $0< \pi(x) < 1$.
- l.269: "The estimation needs to be both valid and unbiased to later yield valid CIs.'' What does ``valid'' mean in these contexts?
- Lemma 5.1 Assumptions are not complete/clear: there are constraints on $\alpha_{\mu}$ and $\alpha_{\pi}$, the assumption on $e(\cdot)$ is about $1/\hat{e}(\cdot)$ and not $e$ and the proof of Wager holds for crossfitted estimators. The asymptotic variance involves theoretical quantities and not their estimators. Please correct the statement. Some remarks also hold for Theorem 5.2.
- l.298  $\tilde{Y}_{\hat{\eta}}$ is built using $D_1$ and then its empirical mean (subtracted by $\hat{\tau}_2$) is computed over 
$D_1$. 

In the proof, it is argued that $\hat{\Delta}_{\tau}$ verifies a Central Limit Theorem which is not clear here, as the term inside the sum depends on the whole sample $\mathcal{D}_1$. 

I believe the proof holds for a fixed function $\tilde{Y}_{\eta}$ but not for the estimated function. Or it needs at least clarifications. 
- Equation 5, how do this new confidence interval compares to a classical one (using only $\mathcal{D}_1$). Can you show that it is stricly smaller in some specific settings (and describe them)? 
- Lemma 6.1 A factor $1/n$ is missing in the definition of $\hat{\tau}_1$, $\hat{\sigma}_1^2$ cannot depend on the sample size $n$. Its expression can be made explicit as a function of the nuisance components. In the proof, $\hat{\pi}$ should be replaced by $\pi$.  
- l488 In the real-world application, how the RMSE is computed, since we do not have access to the true ATE value? Please define ``factual outcome''. 
- l.499 ``Takeaway: Our PPI-based method is effective for medical data.'' Please qualify this statement, as you evaluate your method on two datasets only. 

Typos: 
- open parenthesis l.124
- l.162 : a for is missing
- l.177 and l.334 : $p$ is used for the number of input variables and the ratio between the data set sizes. 
- l.363 ``This steps computes analogous the above''
- l.409 ``We now evaluate our the effectiveness''
- l469 ``Hence, our method performs thus best.''
</content>
</thread_object>

</thread>
<thread>
creation date:
1730653684716

thread_type:
official_review

<thread_object>
creator: reviewer_frum

<content>
summary:
This work presents a method for estimating average treatment effects (ATE) and constructing valid confidence intervals (CIs) from multiple observational datasets. It uses minimal assumptions and leverages prediction-powered inferences for more precise CIs. The approach is validated through theoretical proofs and numerical experiments, with extensions for mixed data sources.

strengths:
The article is well written, and technically sound. The proposed approach is interesting, with great potential for practical application. The previous literature is well integrated, and the paper's novelty is well explicited.

weaknesses:
1. It appears that the novelty of this work is incremental. Use the previously proposed estimator for causal effects and use it in a PPI framework to construct confidence intervals. 

2. Lemma 6.1 is a theoretical property of the inverse probability weighting (IPW) estimator and should not be considered a contribution of this paper. Furthermore, the IPW estimator can also be interpreted as an estimation method based on the influence function, while the augmented inverse probability weighting (AIPW) estimator is derived from the effective influence function. Therefore, the statements made in lines 357-360 are inaccurate.

3. The proof of Theorem 5.2 appears to depend on the asymptotic normality of $\tau_2$. However, the $\mathcal{D}_2$ does not seem to satisfy ignorability, suggesting that it is a biased estimator. I have reservations about whether it is indeed asymptotically normal about $\hat{\tau}_2$ and $\tau_2$ at this point.

questions:
This paper is written towards an audience of applied statistics researchers, and is more suited for a statistical journal such as Biometrika. Although the reviewer acknowledges the significance of rigorously discussing the construction of confidence intervals, the methodology presented may not be highly relevant to the broader community of ICLR. For example, it does not discuss neural network or representation learning at all.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730658983162

thread_type:
official_review

<thread_object>
creator: reviewer_k7su

<content>
summary:
This paper considers the problem of estimating confidence intervals for average treatment effects when given access to a small, unconfounded observational dataset and a larger, confounded observational dataset. Specifically, the problem setting assumes that the covariates for both datasets come from the same population, but the propensity score may differ and the potential outcomes are only independent of treatment assignment in the smaller unconfounded dataset.

The paper's method builds on the prediction-powered inference (PPI) framework, which creates an estimate of the ATE via the confounded dataset and then uses the unconfounded dataset to adjust it and build the appropriate confidence intervals. The paper proves that their method asymptotically results in valid confidence intervals. The method is evaluated on both synthetic and real data, showing its gains over using only the unconfounded dataset.

strengths:
The paper addresses an important issue that has not been covered in the literature before. The paper also does a good job of placing itself within the literature. The paper also does a good job of precisely backing up their claims with theorems.

weaknesses:
On the theoretical side, the paper only shows the asymptotic validity of the proposed method. There are no results demonstrating the width of the resulting confidence intervals. One would like to know under what circumstances the given method will produce a confidence interval that converges on exactly 1-$\alpha$. 

Moreover, the paper does not give any theoretical insight into when it is beneficial to use both datasets as opposed to just the unconfounded dataset. There should be some result showing that the confidence intervals are tighter for proposed method than for the naive method.

questions:
Under what circumstances can it be proved that the proposed method is actually better than operating on just the unconfounded dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732159452353

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to all reviewers

comment:
Thank you very much for the constructive evaluation of our paper and your helpful comments\! We addressed all of them in the comments below. We also uploaded our **revised version of the paper**.

Our main **improvements** are the following:

* **Additional theoretical analysis:** We added a very detailed and technical illustration in our **new Appendix H** showing the key differences between our method and the work from van der Laan et al., 2024\. The method by van der Laan et al involves the matrix inversion problem, which is one of the key steps while reweighting the canonical gradient of the beta-component which is important since ATMLE needs the canonical gradient to iteratively estimate the targeted estimand. This is the reason why their method is unstable and often breaks. We provide further details about the technical problems of the method as well as an in-depth comparison in our **new Appendix H**.   
* **Additional experimental results** We expanded our experimental results in our **new** **Appendix G.** Specifically, we performed the following new experiments:  
  * **New neural instantiations of our method:** We instantiated our method with neural networks. Thereby, we show how our method contributes to representation learning and demonstrate that our method can effectively handle neural base methods (see our **new** **Appendix G.1**).  
  * **New base learners:** We replaced the regression model for nuisance parameters from linear regression to XGBoost  (see our **new** **Appendix G.2**). Thereby, we show the flexibility of our method to other base learners.  
  * **High-dimensional covariates:** We generated a synthetic dataset with high-dimensional covariates. This allows us to study how our method behaves in settings with increasing complexity (see **our new Appendix G.3**).  
  * **Strength of dependence:** We generated an additional covariate as the mean of all other covariates to construct the collinearity in input space. That is, we here examine how our method when the dependence on the covariates varies (see our **new** **Appendix G.4**).   
  * **Different strengths of (un)confounding in $\\mathcal{D}\_1$:** We relaxed the “Unconfoundedness” assumption with three different confounding scenarios while fixing the confounding in $\\mathcal{D}\_2$. Thereby, we can study the performance of our method in settings with different strengths of (un)confounding (see our **new** **Appendix G.5**).   
  * **AIPW to RCT+obs datasets and A-TMLE to observational datasets:** We apply the AIPW to the RCT dataset (see our **new** **Appendix G.6**) and A-TMLE to the observational datasets (see our **new** **Appendix G.7**). Thereby, we show the robustness of our method to RCT and observational datasets.  
  * **Increasing sample size in $\\mathcal{D}\_1$:** We varied the size of the small dataset $\\mathcal{D}\_1$ from 100 to 2500\. Thereby, we further assess the role of the size of $\\mathcal{D}\_1$ (see our **new** **Appendix G.8**).

  We find that **our method is always better than the na{\\”i}ve method under all settings**. This confirms our theoretical contributions and verifies the effectiveness of our method.

We incorporated all changes (indicated with the label **Action** in our individual responses) into the revised version of our paper and highlighted all key changes in **blue color**. Given these improvements, we are confident that our paper provides valuable contributions to the causal machine learning literature and is a good fit for ICLR 2025\.
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary**: This paper proposes a new method to generate reliable confidence intervals for average treatment effects using multiple datasets where additional data set samples may have residual unobserved confounding. Authors build on the prediction-powered inference (PPI) framework to estimate ATE on confounded data, rectifying the bias using the smaller unconfounded data, and using sample statistics of the bias for generating confidence intervals. The procedure uses CATE estimates from the larger dataset to obtain a DR CATE estimate using sample splitting, followed by rectifying using the efficient influence function and subsequently deriving confidence intervals from the summary statistics. The framework is flexible by allowing ML methods used for nuisance estimation in high dimensions and applicable to AIPW, IPW estimators. Theoretical results show asymptotic coverage of the confidence intervals. 

**Strengths**: 
1. Reliable uncertainty quantification of ATE is a critical problem for improving causal inference. 
2. The paper further attempts to tighten interval estimates by leveraging potentially confounded data thereby leveraging the power of additional samples. The contribution is significant and novelty is clear and easy to follow. 
3. Theoretical claims while straightforward are well presented and justified.
4. Empirical evaluation clearly shows better coverage compared to prior work.

**Weaknesses**:
1. Multiple reviewers pointed out that the empirical evaluation is relatively weak for the standards of ICLR, but the focus of the contribution
2. Considering that the contribution is more statistical, i.e., targeted toward better inference without heavy focus on the representation learning of the nuisances, fit to ICLR was not clear for multiple reviewers

**Justification**: Overall all reviewers agree that the contribution is valuable. Multiple reviewers had clarifying questions on the details. There were many clarity issues in the writing in the initial version which were subsequently addressed, proof statements clarified, additional empirical evaluation added, including updates to the written text to add pertinent details e.g., on use of sample splitting. Overall after multiple iterations of discussions between reviewers and authors, it is clear that clarity issues where addressable and not fundamental issues with the contribution. Overall the novelty of the contribution is explicit, demonstration of empirical benefit clear, and multiple reviewers raised their scores post rebuttal. Considering the novelty of the contribution, the quality of the writing, and potential interest to a small but significant subset of ICLR readers, I am recommending an accept.