Constructing Confidence Intervals for Average Treatment Effects from Multiple Datasets
Authors: Yuxin Wang
Maresa Schröder
Dennis Frauen
Jonas Schweisthal
Konstantin Hess
Stefan Feuerriegel
Venue: ICLR 2025 Poster (2025)
Keywords: Causality machine learning
Average treatment effects
Confidence intervals
Prediction-powered inference
Primary area: causal reasoning
TL;DR: Our paper introduces a novel method for estimating average treatment effects with valid confidence intervals by combining multiple datasets through prediction-powered inferences.

Abstract: Constructing confidence intervals (CIs) for the average treatment effect (ATE) from patient records is crucial to assess the effectiveness and safety of drugs. However, patient records typically come from different hospitals, thus raising the question of how multiple observational/experimental datasets can be effectively combined for this purpose. In our paper, we propose a new method that estimates the ATE from multiple observational/experimental datasets and provides valid CIs. Our method makes little assumptions about the observational datasets and is thus widely applicable in medical practice. The key idea of our method is that we leverage prediction-powered inferences and thereby essentially `shrink' the CIs so that we offer more precise uncertainty quantification as compared to na{\"i}ve approaches. We further prove the unbiasedness of our method and the validity of our CIs. We confirm our theoretical results through various numerical experiments.

Published as a conference paper at ICLR 2025
CONSTRUCTING CONFIDENCE INTERVALS FOR AVERAGE TREATMENT EFFECTS FROM MULTIPLE DATASETS
Yuxin Wang, Maresa Schröder, Dennis Frauen, Jonas Schweisthal, Konstantin Hess & Stefan Feuerriegel
LMU Munich
Munich Center for Machine Learning (MCMI.) Yuxin. Wangl @ @lmu.cc
ABSTRACT
Constructing confidence intervals (Cls) for the average treatment effect (ATE) from patient records is crucial to assess the effectiveness and safety of drugs. However, patient records typically come from different hospitals, thus raising the question of how multiple observational/exxerimnnnal datasets can be effectively combined for this purpose. In our paper, we propose a new method that estimates the ATE from multiple observationalleeperiieee Our method makes little assumptions about the observational datasets and is thus widely applicable medical practice. The key idea of our method is that we lever- age prediction- rrddetiionpowerrd inferences and thereby essentially 'shrink' the CIs so that we offer more precise uncertainty quantification as compared to naïve approaches. We further prove the unbiasedness of our method and the validity of our Cls. We confirm our theoretical results through various numerical experiments.
datasets and provides valid CIs.
INTRODUCTION
Estimating the average treatment effect (ATE) together with confidence intervals (CIs) is relevant in many fields, such as medicine, where the ATE used to assess the effectiveness and safety of drugs (Glass et al. 2013 Feuerriegel et al. 2024). Nowadays, there is growing interest in using observational datasets for this purpose, for example, electronic health records (EHRs) and clinical registries (Johnson et al. 2016 Corrigan- Curay et al. 2018 Hong 2021). Importantly, such observational datasets typically originate from different hospitals, different health providers, or even different countries (Colnet et al. 2024), thus raising the question of how to construct CIs for ATE estimation from multiple observational datasets.
Motivating example: During the COVID-19 pandemic, the effectiveness and safety of potential drugs and vaccines were often assessed from electronic health records that originated from different hospitals to rapidly generate new evidence with treatment guidelines (Tacconelli et al. 2022 For example, one study (Wong et al.] 2024) estimated the effect of nirmatrelvir/iionavii (also known under the commercial name "paxlovid") in patients with COVID-19 diagnosis on 28-day all-cause hospitalizations from data obtained through retrospective, multi-center study. The study eventually reported only the ATE but also the corresponding CIs to allow for uncertainty quantification, which is standard in medicine (Kneib et al. 2023). However, the question of how one could combine the predictions and confidence intervals from multiple studies to provide better estimates remains.
Existing works for estimating ATES from multiple datasets can be loosely categorized by (a) which datasets are used and (b) the underlying objective as follows (see Fig. D (a) The underlying patient data can come either from experimental datasets (i.e., randomized controlled trials; RCTs) and/or observational datasets (Feuerriegel et al. 2024). Both require tailored methods as the propensity score is known in RCTs but not in observational data and must thus be estimated (Rubin) 1974). We later focus on a setting where the ATE is estimated from multiple observational datasets, but we also provide extension for combinations of RCT and observational datasets. (b) Much of the literature focused on estimating ATEs from multiple datasets focuses on point estimates (Kallus et al.
2018
John et al. 2019 Yang & Ding 2020
Guo al. 2022 Hatt et al. 2022 Demirel et al.
2024

---

Published as a conference paper at ICLR 2025
but not uncertainty quantification. However, valid Cls are needed in medicine to ensure reliable decision-makingg because of which the existing methods are not applicable for medical applications.
Our method: In this paper, we propose novel method to construct valid CIs for ATE estimation from multiple observational datasets. Specifically, we consider the following setting: we have one (potentially small) unbiased observational dataset DD' for which we assume unconfoundedness (i.e., all confounders observed) and another large-scale observational dataset D for which we allow for unobserved confounders. Then, the key idea of our method is that we tailor prediction-powered inference to our task so that we can essentially 'shrink' the CI and thus offer more precise uncertainty quantification as compared to naive approach. We further present an extension of our method where we 'shrink' CIs in settings with a combination of RCT and observational data.
Why are naïve approaches precluded? One may think that one can simply concatenate both datasets to compute pooled ATE, yet this is not possible as the second dataset DD2 may be confounded and, hence, the overall ATE estimate would be biased. A different, naïve approach is to simply construct finite-sample CIs from D' to obtain unbiased ATE with valid Cls. Yet, the additional power of the second dataset D2 (e.g., the information about the treatment assignment and thus the propensity score) would be ignored so that the CIs are too conservative (Aronow et al. 2021).
Intuition behind our method: Even though the second dataset D2 large and employing it in addition to D may help shrink the estimation variance, it may be confounded, leading to biases in the downstream estimation. Therefore, using the second dataset D directly for inference could lead to biased CATE estimates. As remedy, we derive we decompose the variance of the population-level estimate of the CATE into two parts: one part comes from the estimation variance of the CATE on dataset D2 while the second part is due the difference in estimators of ATES across both datasets D and D2 The estimation variance of the first part can be significantly decreased with access to a large-scale dataset D2 Interestingly, the second part allows us to account for potential confounding bias in 2 and thus still derives valid CIs (+ Theorem |
cnddiction-powerss
inference estimator where
Our main contributions are 2 (1) We propose new method to construct CIs for the ATE from multiple observational datasets. We further extend our method to combinations of RCTs and observational datasets. (2) We prove that our method is consistent estimator and gives valid Cls. (3) We perform experiments with medical data to demonstrate the effectiveness of our method.
2 RELATED WORK
We give an overview of three literature streams relevant to our work: (i) methods for constructing Cls for the ATE that solely rely single dataset; (ii) methods that estimate the ATE from multiple datasets; and (iii) prediction-powered inference.
Estimating CIs for the ATE: Several works focus on constructing Cls for the ATE [Bang & Robins 2005; van der Laan & Rubin (2066). One literature stream ad- dresses asymptotically normal data, which typically results in Vnn-consistent, asymptoti- cally unbiased, normally distributed estima- tors. Another literature stream focuses on finite-sample settings, yet these works im- pose strong assumptions, such as that the data come from an RCT (Aronow et al. (2021) or assume unconfoundedness with relaxed overlap as- sumptions (Armstrong & Kolesár 2021). However, this literature stream focuses on ATE estimation from a single dataset, which is unlike our work.
Methods
setting
Uncertainty quantificatio
Iatuueraa (2016), Har (2002) Deneal g204
RCT + Obt..
X
Obs, Obos.
X
Obs.
Ours
Obs. Obs. |RCT + Obs.)
Doearr
Figure I: Key works aimed at ATE estimation from multiple datasets.
'We refer to Barnard 1949) and refer confidence interval as "valid" when the interval achieves its stated coverage probability. For example, 95% confidence interval valid if, under repeated sampling, it contains the true parameter value approximately 95% of the time. Validity ensures the interval accurately reflects the level of uncertainty about the estimate. "Code and data are available via httttpps///gtthtt..ooo///
2

---

ATE estimation from multiple datasets: Existing methods can be grouped by (a) the underlying dataset setting and (b) the objective, that 18, whether the method provides point estimates or uncer- tainty quantification (see Fig. I: (a) Some methods focus settings with RCT observational data Other methods Published as a conference paper at ICLR 2025
(e.g, Kallus et al. 2018 Chen et al. 2021 Hatt et al. 2022 Demirel et al. 2024 F
focus on multiple observational datasets (e.g.. (e.g..Yang & Ding 2020; Guo al.. 2002 which is also our focus. The former is typically easier because the propensity score is known. In contrast, in the latter, the propensity score unknown and must be estimated account for the covariate shift across treated and non-trean treated patients. (b) Most works focus on only point estimation (e..g, Kallus al. 2018 Chen et al., [021] Hatt et al. [2022] Yang & Ding 2020
Guo al., 2022 Demirel al.
2024
but not uncertainty quantification.
Closest to our method is the work by yan der Laan et al. (2024). Yet, there are crucial differences: the method focuses on (i) RCT+obbervational data, (ii) has different ATE estimation process that can lead to numerical instabilities, and (ii) has limited flexibility
how D2 is leveraged (e.g., the estimators for 2 and D must be identical, despite that different estimators may be beneficial due to the different size and nature of the datasets). We discuss the differences to our method in Appendix H
Prediction-powered inference (PPI); Angelopoulos et al. 2203a proposed the PPI framework for performing valid statistical inference from given dataset when the dataset is supplemented with predictions from a machine-learning model (a brief overview is in Section C.. Several extensions have been proposed recently (e.g..fFisch al.. 2024 Zrnic & Candès 2024). So far, PPI was derived mostly for traditional statistical quantities (e.g., mean, median, quantile). For example, Demirel al. (2024) propose method based on PPI to generalize point estimates of causal effects from one population to a target population. However, they do not provide uncertainty quantification. To the best of our knowledge, there is no work that has tailored PPI to construct CIs for ATE estimation, which is our novelty.
PROBLEM SETUP
We consider the standard setting for ATE estimation from observational data (e.g.. Imbens 2004
Rubin 2006 Shalit et al. 2017 Hatt et al. 2022), which we extend multiple datasets.
Setting: We consider a setting with a small observational dataset D = {(.), at, ( and a large-scale observational dataset D = {(2] az. yy) (see left part in Fig. 2). We use a discrete variable d E D = 21 to refer to the datasets and thus indicate from which dataset variables are (e.g., Xd E De for Dd E (D)). We omit the superscripts for x and use x; and 2, (in color) instead to denote that patients belong to DL and D², respectively, in order avoid misunder- standings with squared values. Furthermore, we use and N' to denote the size of the datasets D and D2, respectively, with < N'. Without loss of generality, is straightforward extend our method to more than two datasets simply by concatenating them into 2
Both datasets have patient information about treatments, outcomes (e-g., tumor size, length of hospi- tal stay), and covariates (e.g., the age or sex of a patient). Formally, both datasets consist of assigned treatments af E A = {0, 1}, outcomes y E y C R. and covariates 2 X' C RR for dataset d E 21 and with i 1... (for d = 1) and j 1...N N' (for d = 2). We denote random variables by capital letters X with realizations 2d, Later, we use sample splitting, and let N denote the sample size of one half of D2, i..., N N Our setting is relevant for variety of practical applications in medicine where electronic health records are collected from different environments, such from different hospitals or different countries.
"Several methods have also aimed at estimating heterogeneous treatment effects (HTEs) from multiple datasets (e.g.. Dohansson than estimating the ATE because of the variation across subpopulations and the larger risk of overlap violations. Importantly, using HTEs for computing ATES is suboptimal, which is well-established in efficiency theory for ATE estimation (Kennedy 2016) and would lead to so-called plug-in in bias (Curth & Van der Schaar 2021)
Joha.ssonnnsonn et al. [0166] Schweisthal et al. 2024). However, estimating HTEs is more challenging
Hence, methods for HTE estimation are orthogonal to our work.
'There are further specialized settings, yet which are different from ours. For example, some works estimate
long-term outcomes by combining RCT Co data (Athey et al. 2020 Ghassami al. [2022 Imbens]
al. 2024.
Even others aim to increase the efficiency of trial analyses (Schuler et al. 2022 Jao et 2023)
3

---

Published as a conference paper at ICLR 2025
We assume that datasets are sampled i..dd from populations (X), ~ P and (X4, A yd) AX I D = d) ~ Ipd with d E {1,2), but with the same marginal distribution of X i.e., P PP and X = -X X² in IP. We later also generalize our theory settings with distribution shifts and finite populations in Appendix B. and Appendix [8.2] respectively. Given that we focus on observational datasets, the treatment assignment rule may vary, and thus define the dataset- specific propensity score via (x) = IP (A* = X = x), d E {1,2}. Formally, we assume that the propensity score may differ across the two datasets (i.e., x * T²). This is common in medical practice, where different hospitals or countries have different treatment guidelines.
and
Target estimand:
We adopt the potential outcomes framework (Neyman 1923, Rubin 2005 to formalize our causal inference task. Let (a) E V. d E [1,2] denote the potential outcome in different distribution Td for treatment intervention A = a. In this paper, we are interested in estimating the ATE on the target population (i.e., where the small dataset is sampled from), which is given by T- = E (1) Y (0)J. and then constructing corresponding CIs for
Assumptions: We make the following assumptions necessary for ATE identification and estimation. Of note, the following assumptions are standard in ATE estimation (Imbens] [004] [Rubin] 2005 Shalit et al., 2017). We distinguish the assumptions for the small dataset D and the large dataset D
Assumption 3.1. For dataset D', it holds: (i) (Consistenccy) A = == y = Y (a): (ii) (Overlap) 0 < (z) < 1, Vo E (iū) (Unconfoundeduess) Y (0), (1)) 11 A X
Assumption 3.2. For dataset D2, it holds: (i) (Consistency) A = a = Y = (a): (ii) (Overlap) 0 < (x)) < 1, Vo E X.
The above assumptions are the standard assumptions for estimating ATES from observational data and are widely used for any underlying estimation method (Imbens] 2004; Rubin 2006) (Shalit et al.
2017). Consistency usually holds long health information accurately and systematically recorded. Overlap can be ensured through preprocessing (e.g., clipping). Unconfoundedness is plausible in digital health settings due to the growing availability of rich electronic health records F
The above assumptions are consistent with the literature studying multiple observational datasets YYang& & Ding. 2020 Guo al. 2022). Note that the assumptions for dataset D are weaker as compared to dataset Dr For D=, we assume that there is no unobserved confounding, but the propensity score is unknown. This is often the case in specialized medical facilities where patients receive close supervision and where thus all critical health measurements are reported, which is typ- ically the case cancer care (Castellanos et al., 2024) and in intensive care units (Johnson et al.) 2016). Needless to say, our assumption is still considerably weaker than assuming an RCT be- cause we allow that the treatment assignment mechanism varies greatly across subpopulations, is unknown, and must thus be estimated. Nevertheless, RCTs are special case of the setting consid- ered in our general framework in which the propensity score n is known. For D2, we do not make the latter assumption but instead allow for unobserved confounding. This is often the case when data are recorded by general practitioners where the need for documentation is typically not as strictly enforced as in other medical facilities.
In sum, D would naturally lead to unbiased ATE estimation but suffers from a large estimation variance due to the small sample size. In contrast, D2 has larger size and thus more statistical power but could lead to biased estimates due to unobserved confounding.
4 OUR METHOD FOR ATE ESTIMATION FROM MULTIPLE DATASETS
The general idea of our approach is shown in Fig. A Measure of fit: We first use sample splitting to compute a measure of fit, ma, to estimate the ATE on the large, observational dataset D2. Here, we use refer to the estimate as fz)) where T2 = (1) - Y²(0) biased due to unobserved confounding, because of which we later need to adjust for this via the so-called rectifier (see below). B Influence function estimation: We compute the non-centered
Overview:
saute-of-the-ane art method based on the DR-learner (Waeer] 2024). We X2 x). Yet, (2) can be
FFurthemorre, advances in sensitivity analysis (Frauen et al. 2024 Oprescu et al. 2023 and partial iden- tification (Duarte et al.) 2024) offer complementary pathways to relax this assumption. The existing works from Fig. [or causal inference from multiple datasets generally make this assumption. In that sense, our work makes weaker assumptions that are more realistic as allow for unobserved confounders in D'.
4

---

Published as a conference paper at ICLR 2025
Fyte)
V
Emit (o/2)- Pa(or)
fula), & D
%
Zapar Faleyi
Lourre
Figure 2: Overview our method. construct CIs for the ATE with two observational datasets from the same population but with different assumptions. we leverage ceddiction-powere inferences: we decompose our task into computing a measure of fit (i.e., estimating the ATE on the large dataset 2> via the DR RR-leareer given by (z)) and a rectifier Á, (i.e., that measures the differences in ATE estimates across both datasets D and Dd). However, finding rectifier for our task is non non-trivial and requires a careful derivation in order to ensure asymptotically valid Cls (-+ our Theorem 4.2).
influence function score (x) for the observational dataset D This is later used in the rectifier. C Rectifier: We compute the rectifier A,, which use to adjust for the bias between datasets D and D2 This allows us to transform the biased estimates f2 (z) into unbiased estimates of the ATE in population. D Constructing Cls: Eventually, we compute the CIs CPP for significance level a.
Why is the above task challenging? First, there is an information gap between the datasets D and D2 This means that different datasets come from different distributions, and we do not have any prior knowledge of the relationship between the datasets, except that the marginal distributions of X are the same. In particular, there can be distribution shift due to various reasons, such as unobserved effect modifiers (i.c., variables that change the treatment effect even if they are no con- founders) and different treatment assignment mechanisms, because of which the propensity scores may be different across both datasets. We later propose a rectifier that accounts for such distribution shifts through an augmented inverse-propense scores must be estimated, which introduces another source of uncertainty. Second, due to the fun- damental problem of causal inference (Rubin 1974). the ATES are not directly observed but must be estimated while considering the aforementioned distribution shift. Further, such estimates must be asymptotically valid so that we can later derive CIs that are also asymptotically valid (-> see our Theorem 4.2
(AIPW) based estimation. Further, the propensity
Below, we describe the steps A
D in detail. Pseuocode Algorithm
samplesplitting and estimate the conditional average treatment effect (CATE) (2)) = (1) - (²) X r on the half of the large-scale dataset D2 of size N (after sample splitting). Let i (x) denote an arbitrary CATE estimator (which may be biased due to, e-g., unobserved confounding). For example, we could choose the DR-learner due to its fast convergence rate and several favorable theoretical properties (Kennedy 2023). Needless to say, our method is also applicable to other estimators. Formally, we train f2(xx and then yield the measure of fit on D2 via i2 1 En= (2,,) by sample splitting.
Step A Measure of fit. The first step is to use sample splitting
Step B: Influence function estimation. For our proposed rectifier, we later need the non-centered influence function (IF) score on D Ideally, one would directly compute the difference in ATES across both datasets for the rectifier, but this is impossible since the ATES are not directly observed but rather need to be estimated. The estimation needs to be both valid and unbiased to later yield valid Cls.
We observe that the average of the non-centered IF score of the AIPW estimator is an unbiased estimation of ATE and is asymptotically normally distributed. This is beneficial for two reasons: (i) we get an unbiased estimate, which allows us to later obtain an unbiased ATE in population, and (ii) the estimate asymptotically normal that we later can derive valid Cls.
Formally, the non-centered IF score of the AIPW estimator (Rbiins& & Rotnitzky 1995) is given by
{((#:) a
1- a! I- + (1.) wl
af (x.) # (x.) (1 # (2:))
[[1 - #' (a:)) i (2:) + + (z: Ao(2:))
(I)
where the nuisance functions (xx) (e (1)) # (x)) plug-in in estimators from D.. while (a) are estimated response functions for Ma (a) y
x xr, Al a] from D Then,
5

---

Published as a conference paper at ICLR 2025
the AIPW estimator is J-IlPW = - (z,)).. Leveraging results from causal inference literature
Wager 2024), have that FAIPW is asymptotically normally distributed with vn (FAIPW 7) N
00 VAIPW W.I, J. where
(A. 11 1x=''(((xx)
21 + 1 110001)9/
(1--))
VAIPWW = (x) - [X(X)] +B
T'())
1-
Hence, the above procedure allows us to estimate the ATE for D and construct the corresponding CI with non-centered influence function scores, which we then use in the rectifier to assess the bias between both datasets D and D²
Step C: Rectifier. We now introduce our proposed rectifier to quantify the difference in ATE across both datasets. Formally, we define the rectifier A, as the expectation of the difference between
Ys (x) and (x) on D'_ For individual observations i, we write A; (r,) t2 (z;). Note that our rectifier carefully tailored to our task and is non-trivial because, due to the fundamental
problem causal inference (Rubin) 1974), the ATES are never observed, but we need to leverage the influence functions score to be able to compute a valid and unbiased estimate. Formally, we have
A -
(xs))) (2x)
= Σ
af
1-a! I- +
# (T.)
Vi
1=1
at - # (x:)
[(1 (m:) (m) + * (a) [o( F2(2.)
# (zx, ) (1 (x:))
We now present Lemma
where we leverage results from the causal inference literature (Cher-
nozhukov et al., 2018 Wager 2024).
Lemma 4.1 (follows from Wager (2024)). Let D' and D2 be sampled i.i.d under the assumptions above. Assume that we have estimated CATE estimator (2(x) with sample splitting on D2, and have consistent estimated nuisance functions ii(x) = (oo(z). # (x)) trained using cross-fitting with converge rate O(n-ar) and O(n on DJ, ie., a (faa())) - Ma(x:) 2x a = 0, and
n (z) 2 O. Then, we have that
Vū A 7 - (Aaaa - (
Proof. See Appendix A.I
Then, the prediction- estimate of the ATE DD is computed via
PP
+
+
r2(x,)
= 1 Σ (((a.)) - =a(x:)) + 1
N Σ
(2(z))
(4)
7
N
a
1 at
(
(5)
N
(I:)
#2 (1:)
#=1
[[1 (1- (sx) A (xx) + A (x: (ooo)) ta(m:))
Step DD: Constructing Cls. We now use the above PPI-based ATE estimate to construct our Cls. Let 0² denote the empirical variance of fz),,, and let 03 denote the empirical variance of Á,. Then, for significance level a E (0. our predictionpoweeet confidence interval is
03
02 N
C
7P
(6)
2
where 03 = Σ. (v,) - fz (x:) - A and 02
Ex (f2(1;) - 22) We show N
1
theoretically that CPP is asymptotically valid in Theorem 4.2
Equation 6 has several implications for how method 'shrinks' Cls. (i) The width of the Cls depends on the size of the different datasets (which we later evaluate empirically as part of our
*Strong double robustness holds here due to the following reason. We use that the estimated nuisance functions are both consistent and that the RMSE of (z) and # (x) decays fast enough. Then, the AIPW estimation is asymptotically normal around the oracle ATE.

---

Published as a conference paper at ICLR 2025
sensitivity analyses). Hence, the width shrinks with larger dataset D' and/or larger dataset D² (ii) The width of the CIs depends on the estimation variance 02, from the dataset Dd. This is desired because our method is particularly designed for using large-scale but confounded datasets D2, so this term should shrink the Cls. (iii) The CIs further depend on the estimation variance of the rectifier 83. This term becomes smaller the less confounding the observational dataset DD² has.
Theorem 4.2 (Validity of our cenfiction-poweree Cls). Let 'D' and D are sampled ii.dd un- der the assumptions from above. Further, assume that we have consistent and cross-fitted esti- mated nuisance functions f(x)) = (po(x), Pi (2), (x)) with converge rates O(n D(n=ax) ax), O(n ar) on D', i.e. n n"= (Pa(x) - Ma(x)) E 0.
-1/T' (x)) 4 and On + 0x 2 1/2. We further assume a CATE estimator F2(x)) trained on D². For some p E [0. 1).
0,1 and n n= "1//// (x)
Nim,,N+oc N p. Fix a E (0. 1). and let C.
( (P 22
63 +
62
N Then, it holds that
lim sins P(T E 2 Proof. See Appendix where we leverage Lemma 4.1
Why is our method better than using the unconfounded dataset only? As shown in Equation 6
the width of our proposed CIs is mainly determined by the variance term + a2 (2) is sufficiently accurate, the rectifier is almost equal to zero, i..., Â s Then, the variance of the rectifier is significantly smaller than the variance of estimated non-centered IF scores, i.e., 83 S 8²,-- Given the large size of D2, the variance of the estimated CATE goes to zero since the estimated variance should be divided by the sample size of D2, i..., N (after sample splitting). As a result, the variance (and thus the CI width) is smaller when using our method than when using only the unconfounded dataset, which means that our CIs are more narrow than such naïve baseline.
N. When the
The above theorem is crucial because it ensures that our PPI-based CIs are asymptotically valid. Further, note that the above theorem is our contribution: it does not directly follow from the PPI- based framework. Rather, we must carefully leverage theoretical guarantees for the estimand of interest and the chosen rectifier, which is one of our contributions.
EXTENSION OF OUR METHOD FOR RCT + OBSERVATIONAL DATASETS
We now extend our PPI-based method to combinations of RCT+observational data. Using an RCT dataset is special case of D As a result, the propensity score is known, which simplifies the underlying task. Yet, the information gap between the datasets remains in that both come from different distributions (e.g., different populations X, different effect modifiers, etc.).
straightforward way to extend our method would be apply our AIPW-base method directly with the known propensities. However, this may have disadvantages as it still requires the estimation of nuisance functions (response functions). Below, we describe the alternative method based on the inverse-propensse propensity weighting (IPW) estimator, which makes necessary changes for the steps A A Note that we no longer need the step estimating the influence functions because the propensity score known, meaning that we can directly estimate via the IPW estimator.
Step A: Measure of fit. First, we compute the CATES analogous to the above approach by using sample splitting to train i2 on D22 We thus yield the measure of fit, i.e., 22 N Ey f2(zy).
Step B: IPW estimator. Given the RCT dataset DJ and known propensity scores, compute the prverse-propensity weighted estimation of ATE via n
n
(1 (1-a))yl
y
n
1=1
which we later use in the rectifier (instead of the influence function score as in our method for multiple observational datasets).
Remark 5.1. Let ft denote the IPW estimator for the ATE of the RCT dataset (D*), A is asymp- (0,
totically normally distributed, iee. (n( -)) 2 N (0. o7), where ?n = = Em (2)
6f = n Σt (x.)
TThe proof is standard and follows from, e.g., Wager 2024

---

Published as a conference paper at ICLR 2025
The above Remark 5.1 ensures that the estimate is asymptotically normal, which allows us later obtain valid Cls.
Step C: Rectifier. We now introduce our rectifier Ar, which denotes the average difference be-
tween = (x) and (z( on D Here, (2(x)) is trained CATE estimators on D based on sample splitting. Our rectifier then given by A, = E (x)) -2(()
Then, the prediction-poweree estimate of the ATE on D is computed via
N
N
PP
M (1))
V..))) -xa(e) fz()))
(8)
N
N
Step D:
Constructing Cls. We now use the above PPI-based ATE estimate to construct our Cls. Let o² denote the empirical variance of t2(x), and let 03 denotes empirical variance rectifier.
Then, for significance level (0, 1), our prediction-powered CI is given by
02
CPP
PP
+ 21 2
72
N
1 E E (f2(2;) - f2). The following theorem shows that our above prediction-powered CI is valid.
where
A,
0
1. Σ
and fz2 E
6%
1 N
Theorem 5.2 (Validity of our prediction-powered Cls in RCT+observational setting). Let D and DD are sampled ii.d under the assumptions from above, and lim,,N-+s N = for some p E 10, Then, the prediction-powert confidence interval has valid coverage: lim infn, P ( CCT 2 a
a
Proof. See Appendix A.3 where we leverage Remark 5.1
EXPERIMENTS
We now evaluate the effectiveness of our proposed method by examining the faithfulness and width of the constructed Cls. To this end, follow prior research and perform experiments with both synthetic and medical data (e.g.. Schröder et al.) 2024 Schweisthal et al. 2024). Synthetic data has the advantage that we have access the ground-truth CATES and thereby can make comparisons
against oracle estimates. Further, medical data allows us to demonstrate both the applicability and relevance of our method in practice.
6.1 SYNTHETIC DATA
Data: Inspired by Demirel al. (2024), we simulate samples from data-generating pro- cess with confounder X
1, and an unobserved confounder U2 E U C [-1,1], binary treatment A E 1), and a real-valued outcome y E R for d E [1, We generate the potential outcomes (d). E {1,2] con- ditioned on X x and U from a Gaussian process GP [|1 + R with mean function m(x, u) O and kernel function & ((x, (,', u²)). We choose a com- posite kernel by adding squared-exponentie (SE) kernel to model the local variation and a linear kernel to model trends in the outcome. We thus have le ((2, (x', u()) ayIxx +
by sampling
Performance for synthetic data. Left: We show the estimated Cls for five random seeds. The red line is the oracle ATE. Ideally, the CIs should be narrow but still overlap with the or- acle ATE. Right: Shows in the width of the CIs av- eraged over five different seeds (a = 0.05). Here, we vary the size of the different datasets given by (D') and N (D²). Note that AIPP (D² only) shown in intentionally shown gray: it is not faithful as seen in the left plot and therefore not valid baseline. + Our method yields faithful CIs, and Cls are shorter as desired.
Figure 3:
with con- figuration parameters = {a.s au Lz (u e R.. We simulate different confounding strengths by varying the value of t.
au uu' + exp
(a-xx) (u-w')
22
2
We generate the covariates of observational datasets D and D2 by sampling X Uniform[- -1, 1] for d E {1,2] independently. For each patient, treatments assignments are sam- pled via A, Bernuuli(P(A
I4, U))) where the probability of treatment assignment is
8

---

Published as a conference paper at ICLR 2025
via logit function Lz, u) sampled from GP&, (I, 2), where 0₂ = {a.; a2 1 ]. A larger value of a and a smaller value of l, implies stronger confounding. The observed outcomes are computed via Y (1 A) GG (z. u) + A GP uz)
generated similarly to Equation 6.1
We generate π = 200 (D)) and = 5000 (D²) samples. For D', we set O2 = and ly = 10 to prevent confounding. For D2, use different values of 0 generate different confounding scenarios: * Scenario 1: little confounding (o,, = 0, L, = 10%). Scenario 2: medium confounding (a., - 0. Lw - 0.5). Scenario 3: heavy confounding (ax,, 10, l, = 0.5). Further details and illustrations about the data-generation process are given in Appendix F.T Altogether, we generate over 60 different datasets under varying configurations for the evaluations below.
Baselines: We compare our PPI-based method +PP for constructing CIs against the following base- line: (1) we estimate the ATE via the AIPW estimator AIPP only on the small dataset, named &ATPW (D* only) which is the naive baseline; (2) we estimate the ATE via the AIPW estimator on the large, confounded dataset, named GAIPW (D² only); and (3) we report the oracle value for T in D
(1) The CIs from our method overlap with the oracle ATE (in red), which shows that our method is faithful. (2) In contrast, the baseline 4AIPW (Da only) rarely covers the oracle ATE and is thus not faithful. This
Main results: Fig. (left). We observe the following:
be expected since the dataset computes the CIs based on the confounded dataset and, hence, yields biased estimates. The unfaithfulness becomes especially evident in Scenario 3 where data under large confounding is gen- erated. (3) Our method generates CIs that are more narrow as compared to the naive baseline &AIPW (D) only). For example, in the left plot, our CIs are, on average, smaller by 49.99% (Scenario 1), 55.37% (Scenario 2), and 55.35% (Scenario 3). + Takeaway: Our PPI-based method yields faithful Cls but where the width of the Cls is clearly shorter. Hence, our method performs the best.
Sensitivity to dataset size: Fig. 3 (right) compares the sensitivity across different dataset sizes. (1) Our method generates CIs that are again more narrow and, therefore, superior. We observe that our method performs better in terms of widths of CIs than the naïve baseline. (2) The advantages of our method become pronounced for setting where N >> n as expected (see right plot, top row).
6.2 MEDICAL DATA
Dataset: We now provide case study that demonstrates the applicability of our method to medi- cal datasets. We chose two common datasets: the MIMIC-III dataset (Johnson et al.) 2016) and a Brazilian COVID-19 dataset (Baqui et al.] (020). MIMIC-III contains health records from pa- tients admitted to intensive care units at large hospitals. We aim to estimate the average red blood cell count of all patients after being treated with mechanical ventilation. Our estimation is based on 8 confounders from medical practice (e.g., respiratory rate, hematocrit). The COVID-19 dataset contains health records of hospitalizations in Brazil across different regions and from patients with different socio-economic backgrounds. We are interested in predicting the effect of comorbidities on the mortality of COVID-19 patients. We created two different splits of the original dataset into D and D2: (i) we split by regions of the hospitals in Brazil (i.c., North and Central-Soutte and (ii) by ethnicity of participants (i.e., White and others). Further details are in Appendix F.2
Table I: Results for different medical datasets. Results: The results are in Table D We again We report the RMSE of the ATE estimator and the width of the Cls. The results for fAPPW (D²
compare the CIs of our estimator against the baselines from above. We further report the root means squared error for the factual outcomes. We find: (1) Our method achieves the smallest RMSE, which indicates that underlying patterns in the data are well captured. (2) Our method obtains the smallest yet valid Cls. Compared to f AIPW (D) only), our methods leads 33.5x re- duction in the width of Cls. (3) AAIPW (D² only) known to be biased. This explains why the RMSE is sometimes considerably larger than the
only) are shown in gray because the estimator is
not faithful and therefore also not a viable base-
line. Reported is the average performance over 5 random seeds.
Dataact
MIMIC-I0
COVID-19 regins)
COVID-19 chincity)
RMSE
Walh
RMSE
Widh
Walh
JAPPA (D) only)
RU87
0.077
2.591
8.499
39.90
0.811
JAPH (Dly))
0.058
0.005
17.125
0.311
30.909
0.004
(Ours)
IJ57
1003
39.48
IU126
Smaller better. Best value
RMSE for the other methods, which again corroborates our findings that fAIPW is not faithful.
Takeaway: Our PPI-based method is effective for medical data.
6.3 RESULTS FOR *OT+EEERVVAIIN DATA
Data: We use the same data gentraatnnn process from above. However, now mimic an RCT for D by setting the unobserved confounder to zero and corresponding a = 0 and L, = 10 Details are in Appendix F.

---

Published as a conference paper at ICLR 2025
Baselines: We now report our method based on IPW (instead of AIPW). We additionally implement the method by yan der Laan et al. (2024), which allows to estimate the ATE from both datasets. We refer to this method by f However, we note that the method is often unstable: their method involves a matrix inversion, yet where the matrix is often singular, so that no CIs can be computed (see Appendix H for detailed explanation). This later explains the fairly noisy performance of the baseline. For a fair comparison, we simply set the output in these cases to D
Results: Fig. (left) shows the results. We find: (1) The CIs from our method cover the oracle ATE (in red), which shows that our method is faithful for the RCT+observational dataset. (2) AAIMLE is faithful in the settings with little and medium confounding (Scenarios and 3). but it fails in Scenario 3 where it is not faithful. (3) Our method generates CIs that are consistently more narrow compared to the baselines. => Takeaway: Our method performs best.
Sensitivity to dataset size: In Fig. (right), we analyze the role of dataset sizes. The results confirm our findings from above: Compared to fATMLE, our method is much more stable. Further, our method generates CIs that are consistently more narrow and thus superior.
6.4 ADDITIONAL EXPERIMENTS We provide further experiments to corroborate our above takeaways in (Appendix. G).
Variations of our method: (1) We performed experiments where we in- stantiated our method using neu- ral networks regression models for estimating nuisance parameters in AIPW to offer more flexibility in learning representations of the co- variate space (see Appendix [..T).
Figure 4: Performance for synthetic data. Left: We show the estimated CIs for five different seeds in RCT and observa- tional datasets. Right: We show the width of the CIs averaged over five different seeds (a 0.05). Our method is both stable and leads to Cls that are faithful and narrow, as desired.
(2) We performed experiments with XGBoost to show the applicability of our method to underlying base mod- els for estimating nuisance parameters in AIPW (see Appendix G.2) Different settings: (3) We varied the size of the covariate space to demonstrate the effectiveness of our method in settings with a digeedimensional covariate space (see Appendix G.3). (4) We varied the covariate dependence by increasing the collinearity in the input space (see Appendix G.4). (5) We varied the strength of confounding in D (sce Appendix G.5). We found that our method performs best across all settings. (6) Oftentimes, estimates of treatment effects in RCT settings benefit when the propensity score is estimated (Su et al. 2023 Cai & van der Laan, 2018). Motivated by this, we applied our AIPW- based method to combinations of RCT and observational datasets (see Appendix G.6). Here, find that we can improve the CI width further using our AlPW-based method. (7) We performed refutation check in which we applied A-TMLE to combinations of two observational datasets (see Appendix G.7p, but remind that this violates the assumptions of A-TMLE. As expected, A-TMLE underperforms, and our method remains clearly superior.
(8) We expanded the sample size of D) from 100 to 2500 to further assess the role of the size of 'D' and the robustness of our method regard- ing the size of D1. We found that our method shows a clear margin and the results are as expected (see Appendix G.8
DISCUSSION
Relevance: In this paper, we developed a new method for ATE estimation from multiple observa- tional datasets. Our method is highly relevant to medical practice as it helps assess the effectiveness and safety of drugs. To this end, we perform rigorous uncertainty quantification by deriving and re- porting valid Cls. Limitations & future work: One improvement is extending our method to other estimands like the CATE or to causal survival analysis. Future research may combine our method with pre-trained large language models (LLMs)) or develop tailored neural network architectures on top of our method for baxtd representations. However, as with any method of causal inference, the assumptions must be carefully assessed to ensure safe and reliable use.
10

---

Published as a conference paper at ICLR 2025
Acknowledgments. Our research was supported by the DAAD program Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the Federal Ministry of Education and Research.
REFERENCES
Ahmed M Alaa and Mihacla Van Der Schaar. Bayesian inference of individualized treatment effeets using multi-task Gaussian processes. NewrlPS, 2017.
Anastasios N Angelopoulos, Stephen Bates, Clara Fannjiang. Michael I Jordan, and Tijana Zmic. Prediction-powered inference. Science, 382(6671):669-677777 2023a.
Anastasios Angelopoulos, John Duchi, and Tijana Zrnic. PPI++: Efficient ceddiction-powerts inference. arXiv preprint arXiv:2311.01453, 2023b.
Timothy B Armstrong and Michal Kolesár. Finite-sample optimal estimation and inference on aver- age treatment effects under unconfoundedness. Econometrica, 141-11171 2021.
PM Aronow, James M Robins, Theo Saarinen, Fredrik Sävje, and Jasjeet Sekhon. Nonparametric identification enough, but randomized controlled trials are. In Workshop on the Neglected Assumptions in Causal Inference (NACI) at the 38 th International Conference on Machine Learn- ing, 2021.
Susan Athey, Raj Chetty, and Guido Imbens. Combining experimental and observational data to estimate treatment effects on long term outcomes. arXiv preprint arXiv:2006.0661 2020.
Heejung Bang and James M Robins. Doubly robust estimation in missing data and causal inference models. Biometrics, 61(4):962-9)) 97799,22-73922 2005.
Pedro Baqui, Ioana Bica, Valerio Marra, Ari Ercole, and Mihaela Der Schaar. Ethnic and regional variations in hospital mortality from covid-19 in brazil: a cross-sectional observational study. The Lancet Global Health, 808)e1018-e1026 2020.
George A Barnard. Statistical inference. Journal of the Royal Statistical Society: Series B (Method- ological), 11(2):155-149, 1949.
Weixin Cai and Mark J van der Laan. One-step targeted maximum likelihood for time-to-event outcomes. arXiv preprint arXiv:1802.09473 2018
Emily H Castellanos, Brett K Wittmershaus, and Sheenu Chandwani. Raising the bar for real- world data in oncology: Approaches to quality across multiple dimensions. JCO Clinical Cancer Informatics, 8, 2024.
Shuxiao Chen, Bo Zhang, and Ting Ye. Minimax rates and adaptivity in combining experimental and observational data. arXiv preprint (XXi:2/09.110222 2021.
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Dutlo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural pa- rameters. The Econometrics Journal, 21(1):C1-C66 2018.
Bénédicte Colnet, Imke Mayer, Guanhua Chen, Awa Dieng, Ruohong Li, Gaël Varoquaux, Jean- Philippe Vert, Julie Josse, and Shu Yang. Causal inference methods for combining randomized trials and observational studies: a review. Statistical Science, 39(1):165-191, 2024.
Jacqueline Corrigan- Corrigan-Curay, Leonard Sacks, and Janet Woodcock. weal-world evidence and real- world data for evaluating drug safety and effectiveness. JAMA, 320(9):867-886) 2018.
Alicia Curth and Mihaela Van der Schaar. effects: From theory to learning algorithms. In AISTATS, 2021.
Nonparametric estimation of heterogeneous treatment
llker Demirel, Ahmed Alaa, Anthony Philippakis, and David Sontag. Prediction Prewietion-powered general- ization of causal inferences. In ICMI,, 2024.
11

---

Published as a conference paper at ICLR 2025
Guilherme Duarte, Noam Finkelstein, Dean Knox, Jonathan Mummolo, and Ilya Shpitser. An au- tomated approach to causal inference in discrete settings. Journal of the American Statistical Association, 119(547):1778-177) 2024.
Stefan Feuerriegel, Dennis Frauen, Valentyn Melnychuk, Jonas Schweisthal, Konstantin Hess, Ali- cia Curth, Stefan Bauer, Niki Kilbertus, Isaac S Kohane, and Mihaela van der Schaar. Causal machine learning for predicting treatment outcomes. Nature Medicine, 30(4):958 968, 2024.
Adam Fisch, Joshua Maynez, R Alex Hofer, Bhuwan Dhingra, Amir Globerson, and William W Cohen. Stratified prediction-poweet inference for hybrid language model evaluation. arXiv preprint arXiv:2406..44 04291, 2024.
Dennis Frauen, Tobias Hatr, Valentyn Melnychuk, and Stefan Feuerriegel. Estimating average causal effects from patient trajectories. In AAAI, 2023.
Dennis Frauen, Valentyn Melnychuk, and Stefan Feuerriegel. Sharp bounds for generalized causal sensitivity analysis. In NeurIPS, 2024.
AmirEmad Ghassami, Alan Yang. David Richardson, Ilya Shpitser, and Eric Tchetgen Tchetgen. Combining experimental and observational data for identification and estimation of long-term causal effects. arXiv preprint arXiv:2201. 2022.
Thomas A Glass, Steven N Goodman, Miguel A Hernán, and Jonathan M Samet. Causal inference in public health. Annual Review of Public Health, 34(1):61-75, 2013.
Wenshuo Guo, Serena Lutong Wang. Peng Ding, Yixin Wang, and Michael Jordan. Multi-source causal inference using control variates under outcome selection bias. Transactions Machine Learning Research, 2022.
Tobias Hatt and Stefan Feuerriegel. Estimating average treatment effects via orthogonal regulariza- tion. In CIKM, 2021.
Tobias Hatt, Jeroen Berrevoets, Alicia Curth, Stefan Feuerriegel, and Mihaela van der Schaar. Com- bining observational and randomized data for estimating heterogeneous treatment effects. arXiv preprint arXiv:2202.1289) 2022.
Konstantin Hess, Valentyn Melnychuk, Dennis Frauen, and Stefan Feuerriegel. Bayesian neural controlled differential equations for treatment effect estimation. In ICLR, 2024.
Julian C Hong. Strategies to turn real-world data into real-world knowledge. JAMA Network Open, 4(10):e2128045, 2021.
Shunsuke Horii and Yoichi Chikahara. Uncertainty quantification in heterogeneous treatment effect estimation with Gaussian-process- based partially linear model. In AAAI, 2024.
Guido Imbens, Nathan Kallus, Xiaojic Mao, and Yuhao Wang. Long-term causal inference under persistent confounding via data combination. Journal of the Royal Statistical Society Series B: Statistical Methodology, 2024.
Guido Imbens. Nonparametric estimation of average treatment effects under exogeneity: A review. Review of Economics and Statistics, 86(1):4-29) 29, 2004.
Andrew Jesson, Sören Mindermann, Uri Shalit, and Yarin Gal. Identifying causal-effect inference failure with uncertainty- aware models. In NeurIPS, 2020.
Fredrik Johansson, Uri Shalit, and David Sontag. Learning representations for counterfactual infer- ence. In /CML, 2016.
ER John, Keith R Abrams, Christopher Brightling, and Nuala A Shechan. Assessing causal treat- ment effect estimation when using large observational datascts. BMC Medical Research Method- ology, 19(1):207, 2019.
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger Mark. MIMIC-III: freely accessible critical care database. Scientific Data, 3(1), 2016.
12

---

Published as a conference paper at ICLR 2025
Nathan Kallus, Aahlad Manas Puli, and Uri Shalit. Removing hidden confounding by experimental grounding. In NeurlPS, 2018.
Edward H Kennedy, Semiparametric theory and empirical processes in causal inference. Springer, Cham, Switzerland, 2016.
Edward H Kennedy. Towards optimal doubly robust estimation of heterogeneous causal effects. Electronic Journal of Statistics, 17(2):3088-3499, 2023.
Thomas Kneib, Alexander Silbersdorff, and Benjamin Säfken. Rage against the mean: review of distributional regression approaches. Econometrics and Statistics, 26:99-123, 2023.
Lihua Lei and Emmanuel Candès. Conformal inference of counterfactuals and individual treat- ment effects. Journal of the Royal Statisfical Society Series B: Statistical Methodology, 83(5): 911-999 911-938, 2021.
Lauren D Liao, Emilie MajijjerreFFrnnssen Alan E Hubbard, and Alejandro Schuler. Prognostic adjustment with efficient estimators to unbiasedly leverage historical data randomized trials. arXiv preprint arXiv:2305.19180 2023.
Valentyn Melnychuk, Dennis Frauen, and Stefan Feuerriegel. Causal transformer for estimating counterfactual outcomes. In ICML,, 2022.
Valentyn Melnychuk, Stefan Feuerriegel, and Mihaela van der Schaar. Quantifying aleatoric uncer- tainty of the treatment effect: A novel orthogonal learner. In NeurlPS, 2024.
Jerzy Neyman. Sur les applications de thar des probabilities aux experiences agaricales: Essay des principle. excerpts reprinted (1990) in english. Statistical Science, 5(463-472):4, 1923.
Miruna Oprescu, Jacob Dorn, Marah Ghoummaid, Andrew Jesson, Nathan Kallus, and Uri Shalit. B-learner: Quasi-oracle bounds on heterogeneous causal effects under hidden confounding. In ICML, 2023.
Sky Qiu, Lars van der Laan, and Mark J van der Laan. atmle: Adaptive Targeted Minimum Loss- based Estimation for Augmenting Randomized Controlled Trial with Real-World Data., 2024. R package version 0.1.0.
James M Robins and Andrea Rotnitzky. Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Statistical Association, 90(429): 122-129, 1995.
Donald B Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5)6888-)) -0188,7--111 1974.
Donald B Rubin. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 300(469) 2005.
Donald B Rubin. Matched sampling for causal effects. Cambridge University Press, Canbridge, UK, 2006.
Maresa Schröder, Dennis Frauen, Jonas Schweisthal, Konstantin Hess, Valentyn Melnychuk, and Stefan Feuerriegel. Conformal prediction for causal effects of continuous treatments. arXiv preprint urXiv:2407.03094 2024.
Alejandro Schuler, David Walsh, Diana Hall, Jon Walsh, and Fisher. Increasing the efficiency of randomized trial estimates via linear adjustment for a prognostic score. The International Journal of Biostatistics, 18(2):329-33 356. 2022.
Jonas Schweisthal, Dennis Frauen, Mihacla van der Schaar, and Stefan Feuerriegel. Meta-learners for partialll-identiifii treatment effects across multiple environments. In ICML, 2024.
Uri Shalit, Fredrik Johansson, and David Sontag. Estimating individual treatment effect: gener- alization bounds and algorithms. In ICML., 2017.
13

---

Published as a conference paper at ICLR 2025
Fangzhou Su, Wenlong Mou, Peng Ding, and Martin J Wainwright. When is the estimated propensity score better? High-dimensional analysis and bias correction. arXiv preprint arXiv:2303.17102. 2023.
Evelina Tacconelli, Anna Gorska, Elena Carrara, Ruth Joanna Davis, Mare Bonten, Alex W Friedrich, Corinna Glasner, Herman Goossens, Jan Hasenauer, Josep Maria Haro Abad, et al.
Challenges of data sharing in European Covid-19 19 projects: learning opportunity for advancing pandemic preparedness and response. The Lancet Regional Health: Europe. 21: 100467, 2022.
Mark J van der Laan and Daniel Rubin. Targeted maximum likelihood learning. The International
Journal of Biostatistics, 2(1):11, 2006.
Mark J van der Laan, David Benkeser, and Weixin Cai. Efficient estimation pathwise differen- tiable target parameters with the undersmoothed highly adaptive lasso. The International Journal of Biostatistics, 19(1):261-25) 2023.
Mark J van der Laan, Sky Qiu, and Lars van der Laan. Adaptive-TMLE for the average treat- ment effect based on randomized controlled trial augmented with real-world data. arXiv preprint 2.Xiv:2400507180 2024.
Stefan Wager. Causal Inference: Statistical Learning Approach. Preprint, 2024.
Shirly Wang. Matthew BA McDermott, Geeticka Chauhan, Marzyeh Ghassemi. Michael C Hughes, and Tristan Naumann. MIMIC-extract: A data extraction, preprocessing, and representation pipeline for MIMIC-III. In ACM CHII., 2020.
Carlos KH Wong, Kristy TK Lau, Ivan CH Au, Sophelia HS Chan, Eric HY Lau, Benjamin J Cowling, and Gabriel M Leung. Effectiveness of hurmatreliirriitoaviii in children and adolescents aged 12-17 years following SARS-CoV-2 Omicron infection: A target trial emulation. Nature Communications, 15(1):4917) 2024.
Shu Yang and Peng Ding. Combining multiple observational data sources to estimate causal effects. Journal of the American Statistical Association, 115(531):1440-1555 2020.
Tijana Zrnic and Emmanuel Candès. Cros-ppeediction-poweerd inference. Proceedings of the National Academy of Sciences, 121((55::232003112), 2024 2024.
14

---

Published as a conference paper at ICLR 2025
A PROOFS
A.1 1 SUPPORTING LEMMA
Below, we restate Lemma in a more detailed way and provide the proof.
Lemma 4.1 Let D and D be sampled i.i.d. under the assumptions above. We assume sample split- ting; i... let D' be split into D and D and let D2 be split into into D2.1 and D2.2, Let us assume assume that we have esti- mated nuisance functions n/' (x: (f)(2)) (ar), # # (x)) and ri(xx) = (1 (x)) that were estimated separately on D and D with converge rates
that we have estimated CATE estimator (2)
D Further,
1
1
D1.1
(1) - (x)) 2x O,
a= 0, 1,
(10)
1
(x) 1/ 2 0,
D1.1
for some constants with az Ogg 2 0 and 02 + ax 2 1/2. We assume that Equation 10 also holds when the roles of Di.. and D1,2 swapped. Then, we construct the rectifier on D given by
Å
fAIPW 72
=D!
(11)
where the cross-fitted estimator is
D
DD22
DD
FAIPW
(12)
n
AD11
1
Yeu (z;)
(13)
D1.1
iEDI.
1
(i(z)) (Z.)))) + y1 (xx)
Vi /
(1 a,)
D1.1
11 (1,(2z)
1 # (x.)
iEDI.
#D1.2
1
Yg (z;)
(14)
1D1.2
iEDI.
1
y1 - put (34)
811 Ad
(+(a))) - (x()))) + ai 12,1
(1 as)
D
1 - 2 (x,)
iEDI.
and the evaluated CATE estimator on DD1 is
>D'
(2(1)).
(15)
EDD
All x; the above and following equations are a:. so we ignore the superscript for simplicity and use colors to distinguish. Then, we have that
v A - - = (0,
(16)
we show it in three steps: (1) decompasition of the rectifier Á (2) the central limited theorem for oracle nuisance functions and CATE estimator; and (3) error due to nuisance estimation. These are below. We use the to refer to oracle estimates.
Proof. To prove Equation 16
(1) Decomposition of Á, First, let us define a new auxiliary random variable Z, (x(xx)
We then separate the rectifier Â (i.e., the average of Zj) into two parts:
Zi
Y2 (2:)
Σ IED
-
71
71
7
(17)
1=1
IED!
error due to muisance estimation
oracle nuisance functions
15

---

Published as a conference paper at ICLR 2025
The first term in Equation 17 denotes the error introduced by using estimated nuisance functions, and the second term in Equation 17 denotes the mean of the differences between pseudo-outcomes based on the oracle nuisance function and the CATE estimator.
(2) The central limited theorem for oracle nuisance functions and CATE estimator. Note that, for the second term in Equation 17 (xs) are pseudo-outcomes based oracle nuisance functions any estimated functions on D or DD), while (-(-) is estimated on D2.1 (particularly, it is independent of D.). Thus, (z)xx) and f2(2) are independent functions, which means that Y, (2.) (2())) are i..d. random variables. Then, following the central limit theorem (CLT), we have
(with no dependency
n
(x;) (2:)) - T + E[rz] = N((,,
(18)
7
IED!
(3) Error due to nuisance estimation. We follow the proof in Wager (2024) to show that the estimation error is negligible. First, we rewrite the first term in Equation 17
(a(,)) Y, (v.)
1 Σ (x,))
1 Σ (,,))) = -AIPW AIPW-
(19)
IED
RD
EED
Second, following Equation
12
we can rewrite the oracle estimation
D
2 2D12,
D
AIPW
n
DD111
DD.2
(20)
Σ
(,())).
n
DDI1
n
DD1.2
IED
iEDI.2
and #D01. as
Moreover, we can decompose
fD
D
D1.1
mi
mg
(21)
FD
Di,
m D',,
mi
where
mp D
j (x;))
Y1 p (TT))
D1.1
a
$ (1.)
iEDLI
Vi - (z.
m0
D
D1.1
A (
+ (1 a:)
1 11 (z.)
iEDL1
(22)
DI.1
1
F1 (z.) a Y) M1 (2:)
m1
D1.1
π (22
IEDL1
+ (1 a;) y/ - (1(x))
D1.1
- π (1;)
IEDLI
Then, we have
FAPPW
AIPW
D1.1
11.2
AD1.
D
+D111
D1.2
pD1.2
+
n
n
D2.22 n
n
D1.1
2
n D1.1
n
12.1
D1.1
D1.a +
D1.
m
m
-11
+ ml
(23)
n
D1.2
D1.2
D2.2
1.2
D 1.2
1i61
m
m
m
11,1
D2.1
D1.1
D1.1
D1.1
Dl.1
m
m
mg
-mo
n
n
D1.2
D1.2
Di.2
D1.2
pl.
D1.2
D1.2.0)
m
m
mm
mn
n
n
16

---

Published as a conference paper at ICLR 2025
To verify vn (fAIPw - (XIPw) -tp O, suffices to show that vn (5D1i - - -p O. Hence,
is further suffices to show that vn (mpl. m
23.1
7p The proof can then extended by
carrying out the same argument for different cross fotts- folds and treatments.
For the first cross-fitted fold and treatment (w = 1), we decompose the error term as follows:
D
Di.
mi
m
y1 ((. #12((r.)
14 (z;)
(1:)
MI (2:) +
DI.1
π (244
iEDI
1
(1i(1:) - -M)
(1-
DII
x x(x;))
(24)
iED1.
+
1
Σ
- (Mi())
D1.1
(x,)
iED!A
1
Σ
D1.1
(I2((1)
π (xra)
IEDI
For the first term, be treated as deterministic when considering terms on D).7, We further observe that, conditional on D and observed covariate values, these terms can be treated as the average of independent mean-zero terms. We thus yield
the fact that, thanks to cross-fitting construction, ji effectively
E
Σ
(ni((x)) (z:))
(1-m2)) ((x))
D1.2
DI.
iEDI.
Var
(i(((;)) - mi (x:)))
(1- m((o)))
( ))
Dii2
D1,1
iEDL.
2 T(x;)) )
1
E
(p(((t)) (1())))
D1.2 zi
D
(25)
EEDI.
Txx)
- -mi
D1.1
(()))
iEDI.
Σ (d((x))-iian) mi(z))
DL. 1
iED
The three equalities above follow from cross-fitting, while the inequalities follow from the overlap assumption and consistency. The second summand in Equation 24 can also be bounded by similar argument. Finally, for the last summand in Equation 24 we use the Cauchy-Schwarl inequality:
Σ
(pR(())) (21(z)) ().((T)))
T(2.))
D1.1
iEDII
1
2
(2((2.)-A - (((x)))²
(26)
D1.1
D1.1
(x;))
(xx))
iEDI.
iEDia
=
natt
which follows from the stated converge rate. Hence, find that this term is also Op mean- ing that it is negligible in probability on the 1/Vri-ccale claimed.
n
17

---

Published as a conference paper at ICLR 2025
A.2
PROOF OF THEOREM 4.2
Proof of Theorem 4.2
We show that T & C. with probability of at most a; that is,
sup P (1 (A F2 21-0/21
0
S a.
(27)
n.N-+00
72
N
Following Lemma 4.1
we obtain that
vū
= N (0,
(28)
Then, we can apply the central limit theorem and obtain that
VN (fz E(72)) = N (0,
(29)
where o is the variance of Å, Yg(x,) - (z,) and of is the variance of f2(2). Therefore, by Slutsky's theorem, we yield
VN A - 12 EJA, + F21) =VR Â - E(Á.1)
V/=H" (N E(f2))
N
(30)
1
*
0, 0 + 0
This, in turn, implies
1 E + + 21-0//11
Ô N
lim P No00
Saa
(31)
where ē is consistent estimate of the variance 1 + Let = N o + of which is a consistent estimate since the two terms are individually consistent estimates of the respective variances. We notice that
P +
12
=
E
Σ
(x:))) - Σ f2(x)))
+ Σ (2(2))
=
Σ Yg(ax) = (32)
REDI
iED!
JED2
LIED!
The last step is combine Equation arrive at
31 and Equation
32 and then apply union bound. We then
a2
lim sup P | + fr2 - B(A, + +al > 21-1/21
+
S
a.
(33)
N.N +00
n
N
Therefore, we have lim No00 P (10, + > 21-0/21
82 +
N
S a.
A.3 PROOF OF THEOREM 5.2
Proof of Theorem 5.2 We show that T & CPP with probability at most at that is,
lim P N.N-
21 2 (1
| 22
02
à2
a.
(34)
n
N
First, let use define the new auxiliary random variable Z, = (xx)) fz (a:). Given the fact that D1 is RCT dataset and known propensity score, Y=(x) and Rz (xx) are independent functions, which means 7, i.i.d random variables. Then, following the CLTT, obtain that
Vn م + B(rz)) N 20
(35)
Then, apply the central limit theorem 2 and obtain that
VN E(72)) + N' (0,
(36)
18

---

Published as a conference paper at ICLR 2025
where o is the variance of Å, (x;))) - (244 and oz2 is the variance of f2(2)) Therefore, by Slutsky's theorem, we yield
V (N B((el)
I
VN
A
72
B|A, + r2l) =V (A, -[(A,)
(37)
- 0, - + +
1
This, in turn, implies
lim sup P ( Noo0
REA, T2] > 21-0/211
Sa, N
(38)
where ē is consistent estimate of the variance 1 + oo Let ő estimate since the two terms are individually consistent estimates of the respective variances. We notice that
N No + o which is a consistent
N > (2. + Σ f2(
E E
=E
T2
Σ
(x.))
E
(x;)
= T. (39)
EEDI
The last step is combine Equation 38 and Equation 39 and then apply a union bound. We then arrive at
(1 f2 -T
02
lim sup P
t
> 21-0/2
I
(40)
Noo
N
Therefore, have lim supn, N0000 P (15. a//
03
82
S a.
N
19

---

Published as a conference paper at ICLR 2025
B ADDITIONAL THEORETICAL RESULTS
In this section, we present further theoretical results regarding our method. Specifically, we show how our method can be generalized deal with distribution shifts (see (.pendixBB.I). finite sample settings (see Appendix B.2), and average potential outcomes (see Appendix B.3
B.1 DISTRIBUTION SHIFT
In our main paper, we focus on computing redicction-owereee confidence intervals when the D and Dd come from the same distribution. We now extend the setting to the case where D comes from and the D2 comes from Q, and two distributions are related by a distribution shift of covariates.
Setting under distribution shift: First, we assume that Q is characterized by covariate shift of P. That is, if we denote by Q = Qx Qaxx QYJA. and Pxx PAIx Py the relevant marginal and conditional distributions, assume that Q PYJA.X. and Caix PAJx- As in main paper, we calculate the measure of fit from our target population Q via
1 N
9 = Eq
F2(2s)
(41)
The estimand from Equation 41 can be transferred to form on P using the Radon-Nikodym deriva- tive. In particular, suppose that Qx is dominated by Px and assume that the Radon-Nikodym derivative w(X) Qx (X) is known. Then, we can rewrite Equation Fx
as
T2
Ep
Σ F2 (x(x))((()))
(42)
In sum, the estimation of f Q be written reweighted function. This allows to perform inference using the rectifier based on data sampled from P as before. For concreteness, we explain the estimation approach in detail. Let
n
Ep
Yn(a:)u(1s)) - ((r))
(43)
Then, the confidence interval for the above rectifier suffices for prediction-powertt inference
Confidence interval covariate shift: Let oz.g denote empirical variance of 73 (X), a denote empirical variance of Â,w. Then, for the significance level E (0, 1), the poddiction-powerss confidence interval is
2
CPP
PPP + 31
A
(44)
N
where
n
1
APP
+
AW
- -
+
Σ
(45)
N
j=1
j=1
2
(Z))))((a))) 2r (x:))u()))) ATn]? -
(46)
2
(+((2,)o0r5)) - 2212
(47)
N
B.2 INFERENCE IN FINITE POPULATION
Our method developed can be directly translated to pinite-population settings.
20

---

Published as a conference paper at ICLR 2025
Setting under finite-population settings: Here, treat D' and D fixed finite populations consisting of n confounder-outcomee pairs, without imposing distributional assumptions on the data points. The only assumption required to apply the latter is that Y2,(x) - f(x) has a known bound, i.e. (a,, b.], valid for all [n)]
In the finite-popie cotte-population setting, we still follow the same way of constructing the centiction-oowered estimates of ATE via
N
PPP A + 12
(n:))) - f(2i))
Σ P2(z))
(48)
N
j=1
2
Confidence interval finite-pppulation settings: Let o denotes empirical variance of f2(X), 03 denotes empirical variance of Á, Then, for significance level a E (0, by Hoeffding's inequality, the prediction-powered confidence interval
-i=1 (b: )2
à2
CPP
PP +
log + 21 $
(49)
2n
N
where 03 = 12 Ei=1 (y((x))-a((1)) (2(1)) -A) and 6
Ei (f2 1000-fo)) F2)²
B.3 INFERENCE OF AVERAGE POTENTIAL OUTCOMES
In this section, we show how to generalize our method to the average potential outcome (APO). We define the mean outcome function in 2 as (X) == [E|((a) X].
APO estimation. Let (z) be the estimated potential outcome function on D' and (z) be the estimated potential outcome function D2 Let the rectifier A, denotes the difference between fr (z) and f2 (z) on D i.e., A. = E (z) fz and let A; = f(())) (r;). Then, the prediction-powered estimation of the APO D is given by as
1 " = + Na.2 n
1
(t)) + N
Σ (z2))
(50)
Confidence interval for APO: Let ø2,2 denote empirical variance of fx). and let 0 denote
empirical variance of Å.. Then, for significance level a E (0, 1), the prediction-poweredd interval is
confidiin confidence
CPP
APP + 21-
0
0,
(51)
n
where 03 = - Em (ff(z.) ((( _a.) and 02
Σラ (fe(2:) (aa2))
21

---

Published as a conference paper at ICLR 2025
C MATHEMATICAL BACKGROUND
We offer brief overview of PPI (Angelopoulos et al. 2023a). In the following of standard PPI framework, one assumes a labeled dataset S, = {(X) Yi)..... (Xn, Yn)3 of i.i.d. samples drawn from some unknown, but fixed distribution P. where X, X is input and Y2 E y is the outcome. One further assumes larger sample SN {(X1, f(x1),,........... ...., (XN, f(Xx)) where
n & N, for which the outcome not available, but where one has access to a pre-trained function f: X +.
PPI protocol The objective is then to estimate a statistical quantity of interest given by the esti- mand lr (e.g.. the mean). In PPI, one then constructs a prediction-powee
estimate @PP through a decompisition OeP = mg + FA. where my is called 'measure of fit' and As is called 'rectifier'. Of note, me typically defined by the statistical quantity of interest (e.g.. my computes the sample av- erage when 6* is the mean). while the rectifier is a measure of the prediction accuracy of f. Yet, the rectifier not given 'out-of-the-box' but it needs to be carefully derived for the statistical quantity of interest. Finally, the cnnfictiio-powerre CI is constructed via CPP [0 |mm + Aol S (wa(a)) where un (a) a constant that depends on the confidence level. Then, C. guaranteed to contain the true parameter 0* with probability at least - a% (Angelopoulos et al. 2023a)). Crucially, the prediction-powered CI smaller than the classical CI when the model / is sufficiently accurate.
Example: Let us focus on the mean, i.e., 0" [[Y.) The classical estimate of 0* is the sample average of the outcomes in Sn i... geluss 1 Σ 1= Y. Then, one can derive the inddiction-powered
estimate of the mean via
@PP
Σ Y f(X)),
(52)
N
m73
Do
so that 95% confidence intervals for 0* are given by 95%
where 6} and o3 are the estimated variances of the f(X) and f(X). respectively. Then, the ceddiction-powered CI is smaller than the classical CI, since, for < N, the width of the prediction- powered CI is primarily determined by the term o² y. Furthermore, when the model has only small errors, have - < .5, which thus helps significantly shrink the Cls.
(0ee + 1.96
69 +
83 N
Of note, the rectifier must be carefully tailored for the estimand, and the derivation is typically non- trivial, especially in order to obtain further theoretical guarantees (e.g., show that the CIs are asymptotically valid).
FFor formal derivation of the Cls, refer Angelopoulos al. 2023a
22

---

Published as a conference paper at ICLR 2025
D PSEUDOCODE
In our main paper, we presented the algorithm for computing the prediction eeddiction-powered estimation of ATE and confidence interval from multiple observational datasets. Here, provide the pseudocode in Algorithm [ We further provide the pseudocode for the extension of our method that aims at RCT+observational data (Algorithm 2)
Algorithm Prediction-powered CIs for ATE estimation from multiple observational datasets
Input: small dataset { (mi, at, vt) ) large dataset D = { (11, a5. No signifi-
cance level E (0, 1)
I: (z) <- estimate CATE estimator D by sample splitting
2: (x) 6 non-centered IF score with estimated nuisance functions on D by cross-fitinng
3: Á, 1 -EE1
(x.),) f2(2.) for E D
rectifier D
4: f2 + N
Σソ= (2;;) for æ2 E D
> measure of fit D
5: p 22 Å.
predictiio-powertte estimator empirical variance of CATE estimation in D
6: 0, N Σ
(f2(2:)) }2)²
7: 03 + Σ
Á Á-)
empirical variance of rectifier in D
$1 22
8: Wa + 21-
B normal approximation
Output: prediction-poweree confidence interval C = (7"H 1 wa)
Algorithm& 2 Prediction-powerees ATE estimation with RCT observational datasets
Input: small RCT dataset D = {(x, at, vi i large dataset D = (I) a3. j No signif-
icance level a E (0, 1)
1: (²()) < estimate CATE estimator from D
2: (1(x) estimate non-centered influential function score from D by cross-fitting
3: Á, Pg(x:) (2.)))
4: f2 +- & +EN (25))) and Á +- EELA
5: 4 + - -Å,
prediction-powee powered estimator
6: o + x Ei=1 (fz(2;) - F2)²
& empirical variance of CATE estimation in
7: 03 + E A Â.)
b empirical variance of rectifier in D
2
8: W. + 21
B normal approximation
Output: Iediction-poweree confidence interval c. = -7 + wa)
23

---

Published as a conference paper at ICLR 2025
E EXTENDED LITERATURE REVIEW
In this section, we present additional related work on uncertainty quantification in causal treatment
effect estimation.
Various approaches exist for quantifying uncer- tainty in causal estimates, with many relying Bayesian methods (e.g.. Alaa & Van Der Schaar 2017 Hess et al.) 2024 Jesson et al.) 2020 Horii & Chikabara 2024). While effective, Bayesian methods require prior distributions informed by domain knowledge. making them less robust to model misspecification and unsuitable for model-agnostic machine learning frameworks. Other works quantify the uncertainty in treatment effect estimation through estimating and sampling from the conditional distribution of the treatment effect (Melnychuk et al.) 2024). Even other techniques offer finite-sample uncertainty guarantees through conformal prediction (e.g.. Lei & Candès) [ Schröder et al. 2024) for potential outcomes. However, uncertainty intervals for treatment effects constructed from conformal prediction intervals around the potential outcomes commonly tend to be very wide. In contrast, in work, aim to provide narrow confidence intervals for the average treatment effect.
Uncertainty quantification for causal quantities:
The classical way of constructing confidence intervals by making use of obser- vational dataset is utilizing the TMLE estimator and the AIPW estimator (Bang & Robins (0055 van der Laan & Rubin 2006). This idea is based on the property of unbiasedness and bounded variance, which provide the theoretical support for valid Cls. Hatt & Feuerriegel (2021) propose novel regularization framework for estimating ATES that exploits unconfoundedness.
Cls for ATE:
Cls for ATE from multiple datasets: Other works aim at combining observational datasets to estimate ATES Yanng& Ding 2020; Guo al., 2022). However. these works make assumptions that the small dataset needs to be sampled from the same distribution as the observational dataset. Also, these studies aim to create more efficient point estimators. They use bootstrap way build confidence intervals, yet which leads to more uncertainty in the estimates.
Another important stream of combining multiple datasets Kallus al. (2018) proposed a method that combined the RCT dataset and observational dataset to obtain an estimate of CATE, which could be seen as special case our method Section [5] Demirel al. (2024) also applied prediction- powered inference but focused average potential outcomes and did not consider the uncertainty quantification of estimation as we do.
24

---

Published as a conference paper at ICLR 2025
F EXPERIMENTAL DETAILS
F.1 SYNTHETIC DATASET
Following the setup in Section 6.1 we consider the three different scenarios of confounding in D>. As shown in Equation 6.1] larger value of au and smaller value of l. imply stronger confounding components. For scenario 1, we set au = and la = 10 which is a scenario that almost no confounding. For scenario 2, we set On, - 0 and l, 0.5, which means that we still do not consider the linear component of U, but let the unobserved confounder play more important role in the exponential term. For scenario 3, set au 10 and Lu 0.5, where the unobserved confounder both influences linear and exponential terms; thus, it presents a scenario with strong confounding. For better understanding, state three kernels of the different settings in the following:
(X - X')
(U-U")
Kscenariol (XX))) exp
(53)
2 106
2 x 106
(X- -X-X" X')
(U - U²)
Kscenario2 ((X,U), exp
(54)
x 106
(X - X²)
(U U*)
Kscenario3 (XX, =10 UU exp
(55)
2 106
Here, we also need to make clear that the unobserved confounder only plays a role in the data- generating process of treatment, which means that does not have direct relationship with with the difference in the means across DD and D
F.2 MEDICAL DATASET
We follow the prior research (Melnychuk et al. 2002 Frauen et al. 2023) to demonstrate our method based on the MIMIC-III dataset (Johnson et al. (0166). which includes electronic health records (EHRs) from patients admitted to intensive care units. We extract confounders (heart rate, sodium, red blood cell count, glucose, hematocrit, respiratory rate, age, gender) and a binary treatment (me- chanical ventilation) using an open-source preprocessing pipeline (Wang et al.] (020).. We define the outcome variable as the red blood cell count after treatment. To extract features from the patient trajectories in the EHRs, we sample random time points and average the value of each variable over the ten hours prior to the sampled time point. All samples with missing values and outliers are re- moved from the dataset. Our final dataset contains 14719 samples, which we separate the samples now into two datasets D and 2 with constant ratio n / = 1/50 and add noise on D²
For the second semi-synthetic dataset, we study COVID-19 hospitalizations in Brazil across different regions (Baqui et al.) (2020). We are interested in predicting the effect of comorbidity on the mortality of COVID-19 19 patients. To model two different observational datasets,
the regions of the hospitals in Brazil, which are split into North and Central-South. As observed confounders, we include age, sex, and ethnicity. Further, we exclude patients younger than 20 or older than 80 years. To model comorbidity as binary variable, we define comorbidity as 1 if at least one of the following conditions were diagnosed for the patient: cardiovascular diseases, asthma, diabetes, pulmonary disease, immunosuppression, obesity, liver diseases, neurological disorders, and renal disease. We then use the same data mata-generating process Section 6.1 to generate A; and Y, while using the second confounding scenario and keeping the ratio of sample size n/N = 1/50 and + N = 6881.
F.3 IMPLEMENTATION DETAILS
We choose the DR-learner to compute îg in D2 together with linear regression and logistic regression models for estimating the nuisance functions. We also use linear regression and logistic regression models for the nuisance function regression when estimating the AIPP We use the default set-
tings fo their regression models, and we did not perform any hyperparameter optimization, our method aims to provide an agnostic confidence interval that applies to all CATE estimators. All our experiments are based on average results from runs across five random seeds.
25

---

Published as a conference paper at ICLR 2025
G ADDITIONAL EXPERIMENTAL RESULTS
G.1 NEURAL INSTANTIATIONS OF OUR METHOD
We follow the same experiment setting and data generation process Section 6.1 but replace the regression model for the nuisance regression model with multi-layer perception (MLP) in Figure Compared with the simple linear regression, our method achieves CIs that have a shorter width (as desired). Further, the Cls from our method consistently cover the oracle ATE, which again confirms the superiority of our method.
Scenario
Scenano
Scenario
Scerario
Scenario
Scenario
5000
3150
283
289
230
1.0
200
g" (Ours)
giie tol
[Ourst
pe
gann 12 only
Figure 5: Results for MLP regression method.
26

---

Published as a conference paper at ICLR 2025
G.2 INSTANTIATION WITH OTHER MACHINE LEARNING MODELS
Figure 6 we follow the same data generated setting as experiments in the Figure 3 but replace the regression method used for the nuisance parameter estimation from linear regression XGBoost. Again, our method is highly effective, which demonstrates the flexibility of our method beyond simple regression model.
Scenario
Scenario 2
Sceraric 3
Scenario
5000
0-0.05
420
1/50
1.00
333
200
3.00
2000
4500
5 (Oum)
gain (2F onlyl
9009 1,P only)
(True
7" 10urst
gare (XP only
para [2P only)
Figure 6: Results for using XGBoost as nuisance parameter regression model.
27

---

Published as a conference paper at ICLR 2025
G.3 LIGH-DIMENSIONLL COVARIATES
We repeated our experiments with more input variables to show that our method is robust in settings with high-dimensional covariate spaces. For this, used data-generating mechanism similar to that in the main paper. In Figure we generate 5 covariates, x E [J,, In Figure we generate 50 covariates, a' [-1, 1] 55 In Figure 9
we generate 500 covariates, [-1, 1. 1500 The results
show that the Cls from our method consistently cover the oracle ATE and that our method reduces
the width of Cls (as desired).
Scenario
Scenatio 2
Sceeario
5000
0-0.0
428
1,50
1.00
3.2h
200
1.00
2000
400
2009
4500
9" (Ouns)
gain (DF only!
dea- gii Jas only)
(True in
2" 10urt
powe txt oniyt
parn (2t only)
Figure 7: Results for 5 variables.
Sconario
Sconario 2
Scanario
Scenerio
Scenario 2
Scenario
5000
60.05
3:00
1,50
1.00
3235
d4
200
2003
1006
2049
4990
4 (Dum))
ginn col only
Tol only)
(True of1
5u EXpt mnyt
gitn cor onay
(Dun)
Figure 8: Results for 50 variables.
Scenario
Sconario 2
Scorario 3
Scanario
Scenario 2
Scénario 3
5000
0.005
0:00
Cic
100c
610
3000
900
000
oN 1,50
1.00
0.25
00
1.0
60
0.2%
199
000
1,00
200
0.50
08
1804
4CK
100c
4000
3000
2 (Dun))
gann (1P only)
[1P only)
(Tru in
pana (2P anly)
para (7P only)
(Oun))
Figure 9: Results for 500 variables.
28

---

Published as a conference paper at ICLR 2025
G.4 STRENGTH OF DEPENDENCE
As extended experiments based on Section G.3
we simulated the I E [-1, 1J and let as
Ei= which leads to collinearity in the input space in Figure 10 Thereby, we can assess the sensitivity of our method to a varying strength of dependence the input space. Compared with ii.d. high-dimensional covariates, notice that the dependence does not affect our method. Our method still outperforms the other baselines and achieves the best width.
Scenario
Scenano
Scenano
Scerario
Scenario
Scenario
a=0.0
3/50
289
289
400
200
ACE
200
(Oun))
per
gain
Figure 10: Results for varying dependence strength in input space.
29

---

Published as a conference paper at ICLR 2025
G.5 DIFFERENT STRENGTHS OF (UNJCONFOUNDING IN D
We aim to show the experiment setting when relaxing 'unconfoundedness" assumption for DJ. We fixed the confoundedness in D as in Scenario 2 but varied the confoundedness in D from Scenario to 3 in Figure [T] We noticed that, while the strength of confounding becomes larger, our method performs better. The results again confirm that our method performs best.
Scenario 1
Scenari 2
Scenario
Scenario
5000
0-0.05
42e
NNN 1/50
1:00
330
200
200
4000
2 (Oun)
git (2F onlyl
gion only)
(Tree is
7" 10urt
gare px onlt
para (2P only
Figure 11: Results for relaxing unconfoundedness assumptions in D
30

---

Published as a conference paper at ICLR 2025
G.6 ROBUSTNESS CHECK OF APPLYING OUR AIPW METHOD TO RCT+OBSERVATIONAL DATASETS
Data: We adopt the same data-generating process as outlined in the main paper while applying our proposed AIPW method described in Section to to the RCT+observational setting.
Main results: In Figure 12 we demonstrates that, when replacing the known propensity score with the estimated propensity score, the performance difference is small. Both methods consistently cover the oracle ATE (in the left figure) and show a large gain compared the naive baseline (in the right figure).
Somario
Screars
Scaue
Sconaric
Scerarto
P. Duat
Figure 12: Applying our AIPW method RCT+observational datasets.
31

---

Published as a conference paper at ICLR 2025
G.7 REFUTATION CHECK OF APPLYING THE A- A-TMLE METHOD TO OBSTRVATIONALLOOBEEETTTTOAA DATASETS
We apply the A-TMLE method to the synthetic datasets with note, this violates the assumptions that underly A-TMLE, so we expect that the method leads to large errors.
data. Of
In Figure [3] we notice that, when applying the A-TMLE method to the synthetic dataset, the A- TMLE performs not that well. Although it constructs short CIs, A-TMLE barely covers the oracle ATE in the left figure. In the right figure, the A-TMLE method shows a large instability in the estimation process again. These findings highlight that A-TMLE leads to CIs that are not faithful in RCT++obeevaatioal settings. Again, this is expected.
Scweeris
Sceewrnn
Sc
Scenano 1
Scerarn
3oatara 3
1.00
1:00
gie
Figure 13: Applying the A-TMLE to multiple observational datasets.
32

---

Published as a conference paper at ICLR 2025
G.8 INCREASING SAMPLE SIZE IN D
Data: To provide more comprehensive evaluation of our method, we increased the sample size in D to enable further comparisons under varying conditions. In Figure [4 the sample size in D is fixed at 5000 (N = 5000), while the sample size in DD varies from T00 to 2500 across three distinct scenarios. This setup allows us systematically assess the performance of our method under different data regimes.
Figure [4 reveals that our method consistently outperforms the naive method across all scenarios. Notably, as the sample size in D increases, the performance gap gradually narrow,, indicating diminishing returns in improvement as more data becomes available D These results are expected and, therefore, further validate the robustness of our method. Scenario
Main results:
Scenario
Scenario
1.8
5000
0.2
3.0
1050
1960
1960
1000
(Ours)
pupa only)
pa only)
Figure 14: Performance for increasing sample size of DJ. The figure shows the width of the Cls averaged over five different seeds (a 0.05). Here, we vary the size D datasets given constant sample size N (D²) from 100 to 2500. Note that AAPPW (D² only) is shown in intentionally shown in gray: it is not faithful as seen the left plot and therefore not a valid baseline. = Our method continually performs better than the AIPP (D2 only).
33

---

Published as a conference paper at ICLR 2025
G.9 RMSE AND COVERAGE FOR THE EXPERIMENTS WITH SYNTHETIC DATA
In Table 2 we report the RMSE of our point estimation and the width of the Cls in Section 6. Table 2: We report the RMSE of the ATE estimator and the width of the Cls. We use the synthetic dataset. The results for FAIPW (D² only) are shown in gray because the estimator not faithful and therefore also not viable baseline. Reported is the average performance over random seeds.
Dataset
RMSE
Width
fAIPW only)
0.298/0. 298/0.298 0,442/0.478/0.470
2241100.40/0.237
j.IPW (Da only)
0.217/0.144/0.131
GPP (Ours)
0.276/0.27410.271
0.241/0.24400.237
Smaller is better. Best value in bold.
34

---

Published as a conference paper at ICLR 2025
H COMPARISON TO A-TMLE
In this section, we compare our method against A-TMLE (van der Laan et al. 2024) and thereby highlight key differences as well as why the training in A-TMLE is unstable.
About A-TMLE: A-TMLE is method that combines an RCT observational dataset to estimate the ATE and can also construct valid confidence intervals. In van der Laan et al. (2024), the au- thors prove that the A-TMLE estimator is /n-consistent and asymptotically normal and gives valid confidence intervals. As a result, A-TMLE achieves smaller mean-squared errors and narrower confidence intervals.
The A-TMLE method proceeds as follows. First, in A-TMLE, the author decomposed targeted ATE estimand as the difference of (a) the pooled-ATE estimand & and (b) a bias estimand y = T $ At a high level, A-TMLE constructs two separated TMLE estimators for the T and T Then, A-TMLE calculates the difference of TMLE outcomes the targeted estimand.
More specifically, for the bias estimand W the estimation process can be decomposed into two steps: (i) learning parametric working model, and (ii) constructing efficient estimator for the targeted estimands. In the first step (i), the method applies the highly adaptive lasso minimum estimator (HAL-ML metric regression working model. Here, the method uses the 'atmle' R-package (Qiu et al. 2024). Given the above definition, one can define the
der Laan al., 2023) with the HAL basis functions for the semipara-
projection parameter
* (P) = (0 W. W).) m(P)) (W, 0) - 10 W, (T..,. 1),
(56)
where P denotes the distribution and where W denotes the covariates. We refer to van der Laan et al. (2024) for more details about the notation.
After that, we need the canonical gradient of the 3(P) ((P))-cmmpnnett to construct the canonical gradient of the working-model-spedee model-specific projection parameter V (P. at P. However, when calculating the canonical gradient of the 3(P)-componens
one of the important things to observe is that I, [1 T) (1 W, A) (W, A), which measures the variance-covarianee structure across basis functions. The expression for T adjusts for variability in different directions, reducing weights for directions with high variance (overrepresented in data) and increasing weights where variance is low (underrepresented). Hence, A-TMLE essentially performs an adaptive weighting to make the patients both datasets more similar for the final estimate.
The reason for why A-TMLE is unstable: The computation of Ip has an important shortcoming: when (a) the dimension of the covariate space is low or (b) when collinearity among the covariates exists, the computation of I is challenging due to the matrix inversion. Eventually, this can lead to numerical instabilities, which can cause the entire A-TMLE method to break down.
Differences to our method: In addition to the drawbacks of A-TMLE, two key differences exist as follows: (1) differences in ATE estimation processes and (2) differences in the flexibility of estimating f2. In the following, we discuss the differences (I) and (2) in detail:
(1) Differences in the ATE estimation process. One of the key differences between our method and A-TMLE is that our method is based on two different ATE estimations when computing the rectifier. In our method, we define the rectifier A, as the difference of fAIPw and fo on D Formally, we have
A
(T())) - =2(x))
(57)
n
A (x;))
A- (xx)
#(x;) Y; (I,, # (2x))
A
[[1 #(z:)) (z.) + A(x.) - (2:))
In contrast, A-TMLE defines the target estimand by applying bias correction V which can be viewed as the expectation of a weighted combination of the conditional effect of the treatment indicators the treatment effect of the two treatment arms, where the weights are the probabilities of enrolling in the RCT of the two arms. Then, the highly adaptively lasso minimum- -losssum--ssssssso estimator (HAL-MMLB) is used to learn the semi-parametric regression model.
35

---

Published as a conference paper at ICLR 2025
(2) Flexibility. Another key difference is that our method is more flexible, allowing to any approach to estimate f2 in D In contrast, the process in A-TMLE is more rigid: A-TMLE con- structs TMLE for the pooled-ATE and bias correction term estimation. This can limit the flexibilityy for computing f2, especially when we want to use different modeling approaches for both datasets (which is likely given that one dataset is probably larger than the other!).
Instead, our method supports a variety of approaches, allowing end-users of our method to better adapt to the underlying data-generating process. For example, we can use various meta-learners like the S-learner, T-learner, R-learner, and DR-learner, where each comes with unique strengths in practice. The S-Jearnrr, for instance, works well when there are fewer treatment interactions, while the T-learner and R-learner handle more complex treatment effect patterns.
Additionally, our method allows to use pre-trained models directly (which is unlike A- A-TMLE!). This allows us - in our method - to calculate the ATE from model predictions without needing to re-fit or modify the model. Alternatively, one can even use large language models or foundation models to generate the predictions of f2. The flexibility to use various models integrate pre- trained models makes our approach more flexible to handle broad variety different settings and data structures. We believe that this makes our method powerful tool for accurate ATE estimation in range of applications. For example, we are given a pre-trained machine learning model f f(x), then we have access to the predictions on D as f(r). Formally, we then yield the measure of fit and the rectifier via
N
f2
f125),
(58)
N
j=1
(o(o.)) - f(x.)) Σ
A 1- Y.
*-)())) )
(59)
((x))
t=1
A, - ax;;)
[(1 - --(x)))) (1(())) (:: -(a f(zz)
#(x:) (1 ((x))
N
PP 1
Σ ((x,
(o(x:))(aa))
- f(x.).
(60)
j=1
According to the central limited theorem of the predictions f(x) and the asymptotical normality of the AIPW estimator, we can construct valid CI as we mentioned in the main paper. This means, we have C.P of fit respectively.
PP - 21
62
where 03 and o2, variance of the rectifier and measure
36


Reviews:
[{"review_id": "4y9P5S4qzJ", "reviewer": "Reviewer_k7Su", "summary": "This paper considers the problem of estimating confidence intervals for average treatment effects when given access to a small, unconfounded observational dataset and a larger, confounded observational dataset. Specifically, the problem setting assumes that the covariates for both datasets come from the same population, but the propensity score may differ and the potential outcomes are only independent of treatment assignment in the smaller unconfounded dataset.\n\nThe paper's method builds on the prediction-powered inference (PPI) framework, which creates an estimate of the ATE via the confounded dataset and then uses the unconfounded dataset to adjust it and build the appropriate confidence intervals. The paper proves that their method asymptotically results in valid confidence intervals. The method is evaluated on both synthetic and real data, showing its gains over using only the unconfounded dataset.", "questions": "Under what circumstances can it be proved that the proposed method is actually better than operating on just the unconfounded dataset?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses an important issue that has not been covered in the literature before. The paper also does a good job of placing itself within the literature. The paper also does a good job of precisely backing up their claims with theorems.", "weaknesses": "On the theoretical side, the paper only shows the asymptotic validity of the proposed method. There are no results demonstrating the width of the resulting confidence intervals. One would like to know under what circumstances the given method will produce a confidence interval that converges on exactly 1-$\\alpha$. \n\nMoreover, the paper does not give any theoretical insight into when it is beneficial to use both datasets as opposed to just the unconfounded dataset. There should be some result showing that the confidence intervals are tighter for proposed method than for the naive method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9GWJAffCdY", "reviewer": "Reviewer_frUm", "summary": "This work presents a method for estimating average treatment effects (ATE) and constructing valid confidence intervals (CIs) from multiple observational datasets. It uses minimal assumptions and leverages prediction-powered inferences for more precise CIs. The approach is validated through theoretical proofs and numerical experiments, with extensions for mixed data sources.", "questions": "This paper is written towards an audience of applied statistics researchers, and is more suited for a statistical journal such as Biometrika. Although the reviewer acknowledges the significance of rigorously discussing the construction of confidence intervals, the methodology presented may not be highly relevant to the broader community of ICLR. For example, it does not discuss neural network or representation learning at all.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The article is well written, and technically sound. The proposed approach is interesting, with great potential for practical application. The previous literature is well integrated, and the paper's novelty is well explicited.", "weaknesses": "1. It appears that the novelty of this work is incremental. Use the previously proposed estimator for causal effects and use it in a PPI framework to construct confidence intervals. \n\n2. Lemma 6.1 is a theoretical property of the inverse probability weighting (IPW) estimator and should not be considered a contribution of this paper. Furthermore, the IPW estimator can also be interpreted as an estimation method based on the influence function, while the augmented inverse probability weighting (AIPW) estimator is derived from the effective influence function. Therefore, the statements made in lines 357-360 are inaccurate.\n\n3. The proof of Theorem 5.2 appears to depend on the asymptotic normality of $\\tau_2$. However, the $\\mathcal{D}_2$ does not seem to satisfy ignorability, suggesting that it is a biased estimator. I have reservations about whether it is indeed asymptotically normal about $\\hat{\\tau}_2$ and $\\tau_2$ at this point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e7Q8pOHl76", "reviewer": "Reviewer_sim7", "summary": "The authors propose a new method to compute confidence intervals of the Average Treatment Effect (ATE) of the Risk Difference based on two observational data sets, the first one of small size in which all confounders are observed, the second one being larger with potential unobserved confounders. They use the largest data set to debias the estimation of the ATE computed on the first data set. The proposed confidence interval is shown to have the correct asymptotic coverage. Experiments on a toy data set and on real-world data sets show that the proposed method leads to smaller unbiased confidence intervals.", "questions": "- Position of the work: it is assumed that the small data set contains all confounders. The major difference with a RCT is that the propensity score is unknown and depends on the covariate (as discussed in page 4, last paragraph). This justifies the use of IPW with the true propensity score (step B, page 7). However, numerous work have shown that even in a RCT scenario where the true propensity score is known, estimating it can reduce the variance of the ATE estimate (see, e.g., references in https://arxiv.org/pdf/2303.17102). Thus, in a RCT, one would likely use an estimation of the propensity score inside the IPW estimate. This point should be discussed and may lead to qualify statements about differences between RCT and the small data set you consider. In particular, could we apply methods developed for RCT (van der Laan et al., 2024) to the first experimental setting? Paragraph l.119-121 could be modified to make explicit the differences (if any) between RCT+observational and the considered setting observational+observational. \n- Regarding the work of van der Laan et al., 2024, it is mentioned that their method is unstable due to a matrix inversion (l.509-510). I went quickly through their paper and did not find such an operation. Could you be more specific, and maybe describe with more details how this method works, as it is the main competitor?\n- l.151-152 ``PP CI s smaller than the classical CI when the model $f$ is sufficiently accurate.'' Is there a specific reference for this fact?\n- l.203-204: It is usually required that $0< \\pi(x) < 1$.\n- l.269: \"The estimation needs to be both valid and unbiased to later yield valid CIs.'' What does ``valid'' mean in these contexts?\n- Lemma 5.1 Assumptions are not complete/clear: there are constraints on $\\alpha_{\\mu}$ and $\\alpha_{\\pi}$, the assumption on $e(\\cdot)$ is about $1/\\hat{e}(\\cdot)$ and not $e$ and the proof of Wager holds for crossfitted estimators. The asymptotic variance involves theoretical quantities and not their estimators. Please correct the statement. Some remarks also hold for Theorem 5.2.\n- l.298  $\\tilde{Y}_{\\hat{\\eta}}$ is built using $D_1$ and then its empirical mean (subtracted by $\\hat{\\tau}_2$) is computed over \n$D_1$. \n\nIn the proof, it is argued that $\\hat{\\Delta}_{\\tau}$ verifies a Central Limit Theorem which is not clear here, as the term inside the sum depends on the whole sample $\\mathcal{D}_1$. \n\nI believe the proof holds for a fixed function $\\tilde{Y}_{\\eta}$ but not for the estimated function. Or it needs at least clarifications. \n- Equation 5, how do this new confidence interval compares to a classical one (using only $\\mathcal{D}_1$). Can you show that it is stricly smaller in some specific settings (and describe them)? \n- Lemma 6.1 A factor $1/n$ is missing in the definition of $\\hat{\\tau}_1$, $\\hat{\\sigma}_1^2$ cannot depend on the sample size $n$. Its expression can be made explicit as a function of the nuisance components. In the proof, $\\hat{\\pi}$ should be replaced by $\\pi$.  \n- l488 In the real-world application, how the RMSE is computed, since we do not have access to the true ATE value? Please define ``factual outcome''. \n- l.499 ``Takeaway: Our PPI-based method is effective for medical data.'' Please qualify this statement, as you evaluate your method on two datasets only. \n\nTypos: \n- open parenthesis l.124\n- l.162 : a for is missing\n- l.177 and l.334 : $p$ is used for the number of input variables and the ratio between the data set sizes. \n- l.363 ``This steps computes analogous the above''\n- l.409 ``We now evaluate our the effectiveness''\n- l469 ``Hence, our method performs thus best.''", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper focuses on a very important topic, which is uncertainty quantification of point estimate for the ATE, based on observational data sets. Obtaining accurate confidence intervals for the ATE helps to understand the effect of a treatment (e.g., drugs). Observational data are becoming more and more common in causal inference. The main contribution of the paper is to propose a new way of computing confidence intervals based on two observational data sets. The formula is rather simple and easy to use, while it can be applied to a wide variety of estimators (in the paper, AIPW or IPW are used depending on the context). \nExperiments tend to show that the proposed method is better in terms of coverage and IC length.", "weaknesses": "There is a potential practical interest for the confidence interval and the point estimate developed in this paper. I have two majors concerns with the paper: \n- the experimental results on the toy data set are inconclusive : the input variable is of dimension one, which does not correspond to a real-world problem. For the conclusion to hold, I would expect more experiments on more complex settings (more input variables, with or without dependence, how relaxing the assumption of all observed confounders in the first data set may impact the estimation performances). Experimenting on one single input variable does not allow us to obtain a clear comprehension of the performance of the proposed method. \n- Theorem 5.2 is the main theoretical result of the paper. It is claimed below that it does not directly result from the Prediction-powered Inference (PPI) framework. However, going in detail through the proof, I could not find an argument specific to the causal inference framework or the AIPW estimator. The proof relies on Central Limit Theorems of two independent quantities.\nThus, I find the novelty of the contribution mild from a theoretical perspective. Besides, in my opinion, more experiments should be carried out to assess the performance of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hB6CdjYPne", "reviewer": "Reviewer_pijF", "summary": "The authors present a method, based on prediction-powered inference, for estimating confidence intervals from two datasets, one with observed confounders and one with additionally hidden confounders. Theoretical and empirical results are provided.", "questions": "1. **Empirical results:**\n  - Mostly qualitative results are provided for the synthetic data. I would be interested in seeing the coverage and RMSE of each method with respect to ground truth, which should be known as the data is simulated. It seems like the proposed method has a higher bias, but lower variance.\n  - In my opinion, it would be interesting to experiment with settings with more data in $\\mathcal{D}_1$ to see when only using $\\mathcal{D}_1$ becomes better.\n  - How can the RMSE be calculated for the real data, when the true ATE is unknown? \n  - Unless I am mistaken, it seems that only simple learners are considered, based on linear and logistic regression (Appendix E3)? If so, it would be insightful to also compare with more advanced ML algorithms (e.g. gradient boosting).\n\n2. **Practical significance:**\nAlthough the work is interesting from a theoretical perspective, I wonder at its practical significance. How realistic are your assumptions? How can a practitioner verify them and trust your method?\n\n3. **Conclusion:** A more detailed discussion of limitations and directions for future work would be appreciated.\n\n___\n\n\n**Minor points:**\n- Brackets missing in the citations on lines 48 and 49.\n- Gray text in Table 1 is not applied to all columns.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The work addresses a clear research gap with respect to existing work.\n- The provided theoretical results support the method.\n- The method is presented clearly, and the work is easy to follow.", "weaknesses": "- The current empirical analysis is limited, with mostly quantitative results and only linear models.\n\n- Due to the required assumptions, the practical relevance of the work is not entirely clear to me.\n\n- The conclusion is an afterthought in the current version, with no discussion of limitations or directions for future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)