Falsification before Extrapolation in Causal Effect Estimation
Authors: Zeshan Hussain
Michael Oberst
Ming-Chieh Shih
David Sontag
Venue: NeurIPS 2022 Accept (2022)
Keywords: Causal Inference
Randomized Trial
Hypothesis Testing
Causal Effects

Abstract: Randomized Controlled Trials (RCTs) represent a gold standard when developing policy guidelines. However, RCTs are often narrow, and lack data on broader populations of interest.  Causal effects in these populations are often estimated using observational datasets, which may suffer from unobserved confounding and selection bias.  Given a set of observational estimates (e.g., from multiple studies), we propose a meta-algorithm that attempts to reject observational estimates that are biased. We do so using validation effects, causal effects that can be inferred from both RCT and observational data. After rejecting estimators that do not pass this test, we generate conservative confidence intervals on the extrapolated causal effects for subgroups not observed in the RCT. Under the assumption that at least one observational estimator is asymptotically normal and consistent for both the validation and extrapolated effects, we provide guarantees on the coverage probability of the intervals output by our algorithm. To facilitate hypothesis testing in settings where causal effect transportation across datasets is necessary, we give conditions under which a doubly-robust estimator of group average treatment effects is asymptotically normal, even when flexible machine learning methods are used for estimation of nuisance parameters. We illustrate the properties of our approach on semi-synthetic experiments based on the IHDP dataset, and show that it compares favorably to standard meta-analysis techniques.

Falsification before Extrapolation in Causal Effect Estimation
Zeshan Hussain" MIT CSAIL & IMES Cambridge, MA zeshanmh@mit, edu
Michael Oberst* MIT CSAIL & IMES Cambridge, MA moberst@mit.ed edu
Ming- Chigh Shih'
David Sontag
National Dong Hwa University
MIT CSAIL & IMES
Hualien, Taiwan mcshih0gms. ndhu, edu. tw
Cambridge, MA
dsontagOcsail mim. edt
Abstract
Randomized Controlled Trials (RCT's) represent gold standard when developing policy guidelines. However, RCTs are often narrow, and lack data on broader populations of interest. Causal effects in these populations using observational datasets, which may suffer from unobserved confounding and selection bias. Given set of observational estimates (e.g. from multiple studies), we propose meta-algoritee allorithmm that attempts to reject observational estimates that are biased. We do so using validation effects, causal effects that can be inferred from both RCT and observational data. After rejecting estimators that do not pass this test, we generate conservative confidence intervals on the extrapolated causal effects for subgroups not observed in the RCT. Under the assumption that least one observational estimator is asymptotically normal and consistent for both the validation and extrapolated effects, provide guarantees the coverage probability of the intervals output by our algorithm. To facilitate hypothesis testing in settings where causal effect transportation across datasets is necessary, we give conditions under which doubly-robust estimator of group average treatment effects is asymptotically normal, even when flexible machine learning methods are used for estimation nuisance parameters. We illustrate the properties our approach semi-synthetic and real world datasets, and show that it compares favorably to standard meta-analysis techniques.
often estimated
Introduction
Policy guidelines often rely on conclusions from Randomized Controlled Trials (RCTs), whether considering treatment decisions in healthcare, classroom interventions in education, or social pro- grams in economics [I 1E < In healthcare, when a target population has reasonable overlap with the inclusion criteria of RCTs, current clinical treatment guidelines rely primarily RCTs 23 22 For target populations not well-represented in RCTs, observational studies are often used to infer treatment effects. However, different observational estimates give conflicting conclusions. We give an example of this tension when looking at new chemotherapy for multiple myeloma.
Example 1.1 (Carfilzomib- based Combination Therapy for Newly Diagnosed Multiple Myeloma (NDMM)). Until 2020, the effect of Carfilzomib-based combination therapy in the NDMM subpopu-
"Equal contribution. Order determined alphabetically, For code and instructions data access, please visit: https://github.com com/ cl www.l//mnnnmmmmt...commm obs- extrapolation
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

lation had not been studied via an RCT. However, a trial (ASPIRE) in 2015 measured the effect of Carfilzomib-base therapy on survival in Relapsed & Refractory Multiple Myeloma (RRMM) patients
[58] The CoMMpass trial, an observational dataset, was also available in which the Carfilzomib regimen was given to both NDMM and RRMM patients [3).. Several analyses on the CoMMpass
dataset to estimate the effect of Carfilzomib-base
therapy on NDMM patients led to different,
sometimes opposing, conclusions on the benefit of the therapy in this subpopulation
+ 32
A traditional meta-analysis approach would combine observational estimates under the assumption that differences arise only due to random variation, and not e.g. differences in confounding bias 24 Section 10.10.4.1J. This is unlikely to be true in practice. For instance, in Example I.I. the two studies in question made different choices in e.g., how to adjust for confounders. In this paper, we relax the assumption that all observational estimates are valid. Instead, observational estimate is valid across all subpopulations. In the context of Example I.] we might assume that at least one of the candidate observational studies yields consistent and asymptotically normal estimates of the effects in both the NDMM and RRMM populations. While we cannot verify that any given estimator is valid for all subpopulations, we can falsify this claim of validity if an estimator is inconsistent for the causal effects identified by the RCT (e.g., RRMM). Hence, we use the term validation effects to refer to causal effects in subpopulations that overlap between the observational and randomized datasets (e.g., RRMM), and use the term extrapolated effects to refer to those only covered by observational datasets (e.g.. NDMM).
assumee that at least one
We propose a meta-algorithm that combines two key ideas: falsification of estimators, and pessimistic combination of confidence intervals. We first aim to falsify candidate estimators using hypothesis testing, rejecting those that fail to replicate the RCT estimates of validation effects. In Section 2.2 motivate this approach with examples of observational estimates based on different causal assumptions, showing that hypothesis tests based on asymptotic normality can be applied even when causal assumptions fail to hold. Then, we combine accepted estimators get confidence intervals on the extrapolated effects. Since failure to reject does not imply validity we return an interval that contains every confidence interval of the accepted estimators. We demonstrate theoretically that if at least one candidate estimator is consistent for both the validation and extrapolated effects, then the intervals returned by our algorithm provide valid asymptotic coverage of the true effects.
In scenarios where the covariate distribution differs across datasets, estimators that "transport" the causal effect should be used 42 [ (55). Furthermore, in the case of digeenimeniio covariates, flexible machine learning methods are required to estimate nuisance functions, which can affect the hypothesis tests due to their slower convergence rates. In light of this, we adapt estimators of the average treatment effect in this setting to provide estimates of group-wise treatment effects, and show (via the framework of double machine learning 112 (55) that this estimator enjoys asymptotic normality under mild conditions on convergence rates of the nuisance function estimators, Our conclusions are supported by semi-synthetic experiments, based on the IHDP dataset, as well as real-world experiments, based on clínical trial and observational data from the Women's 's Health Initiative (WHI), that demonstrate various characteristies of our meta-algorithm.
2 Setup and Motivating Examples
2.1 Notation and Assumptions
Let Y )' denote an outcome of interest, and A {0, 13 denote binary treatment. We use Ya to denote the potential outcome of an individual under treatment A = a. We use X E X' to denote all other covariates. To distinguish between different sampling distributions (i.e., datasets), we use the random variable D E [O.....), where J 2 1 is the number of observational datasets, and D is reserved for the sampling distribution of the randomized trial. We let IP, Yo, Y, X, D) denote the joint distribution over all variables, including unobserved potential outcomes. For instance, P(Y, Yo, X D = 0) denotes the distribution of potential outcomes and covariates the RCT.
We seek to estimate conditional average treatment effects for a finite set of subgroups ( We assume subgroups are defined a-priori by a function G X' + {1, ...... 1}. such that G = indicates that X E G. We use observational data precisely because not all groups are supported on the RCT
For instance, we could fail to reject due to low power, or because falsification is impossible, due to differences in causal structure across subpopulations, as discussed in Appendix

---

dataset. To this end, we use In = {i supported on the RCT dataset, and we let Io denote the complement {I..... .I3 \ IR. We use [I to denote the cardinality of a set, and assume that every observational dataset has support for all groups. Assumption 2.1 (Support), We assume that PPG i, D = j) > for all E {l..... / and J E [I...., J/. i.e., all observational datasets (D 2 1) have support for all groups.
> 0} to denote the set of subgroups
Definition 2.1 (Validation and Extrapolated Effects). We define the group average treatment effect (GATE as
Y - Yo G -, = 0], if In B - G - I], if To
(I)
and refer to Tæ for E In as a validation effect, and Tú for E To as an extrapolated effect.
Here, we focus on discrete subgroups, in part to reflect the practical reality of comparing RCTs to observational studies, where we may have large observational datasets with rich covariates but only have access to the published results of the RCT, which often provides estimates (with confidence intervals) for subgroup effects but not the raw data itself (56) Figure 4, for example]. In Def. 2.T allow for the fact that different datasets may have different distributions of effect modifiers. To have a well-defined effect of interest, we have chosen the reference dataset D 1 arbitrarily, but in principle we could choose any of the observational datasets. We discuss further nuances of this definition under Assumption 2.3 By Def. 2.1 we often write these effects as vector r IR We use (k) R to denote an estimator, where k E (o..... } with F(0) reserved to denote the estimator derived from the RCT data. The remainder are observational estimators D In general, we use "hat" notation refer to estimators, and refer to their population quantities without a hat. We use N₂ to denote the number of samples used by each estimator. Throughout, we will assume that the RCT estimator is consistent.
A ssuumption 2.2. The RCT estimator +0) is consistent estimator of the (supported) dimensions of T, such that for each i IR. (0) is consistent for Tr.
Below, our central assumption states that at least one observational estimator also enjoys consistency, We discuss examples of specific observational estimators in Section 22
Assumption 2.3. There exists at least one observational estimator f(k) E R k 2 1 that is a consistent estimator of T E R,, such that for each i E {1,.... .). i. (k) is consistent for TJ.
Remark 2.1. Assumption 2.3 is our primary non-trivial assumption, and in Appendix B we give one example of causal assumptions (for given observational study) under which the entire GATE vector 7 is identifiable from observational data, and give an estimator of the resulting observational
quantity which is asymptotically normal 4l 42
40 15 [6]. In order to compare observational estimates with experimental ones, Assumption 2.3 requires not only that the observational data is free of confounding, but also that the causal effect can be transported to the RCT population. This can be done so long as relevant effect modifiers are observed in both the RCT and observational study, but the latter requirement is satisfied automatically (without requiring RCT data) if e.g.. treatment effects are constant within each subgroup G, or if the distribution of effect modifiers is the same between the RCT and observational study, in which case E[ Yo D Y Y. | G]. This represents one (conservative) failure mode of our approach, in which we may reject an observational estimator due to failures in transportability, even if it yields unbiased estimates of the extrapolated effects.
Assumptions 2.2 and 2.3 imply that there exists observational estimator +kk) such that both (k)
and the RCT estimate To (0) are both consistent for the validation effects tı, Vi E IR. To validate this implication in finite samples, we will construct statistical test to compare iz (k) and (0). Our
general approach could be modified to use any valid test, but to facilitate further analysis, as well as explicit construction of confidence intervals, we additionally assume the following:
Assumption 2.4. All GATE estimators are pointwise asymptoticall normally distributed.
(2) for all k E (O,,,..,K,, and for all in IR if k 0 (the RCT estimator), and otherwise for all i In U Uo- Here, 4 denotes convergence in distribution, and o? (l)) is estimate of the variance that converges in probability to of (k)). the asymptotic variance of (N)(ff( (k)).
(N(((.()) - (())))/ (0,
3 We use this term in line with the literature II 28
3x 55 and to distinguish it from the CATE function. 4 We define (0) as a vector in R for simplicity of notation, allowing the entries (0), E To to he arbitrary.
Here, "pointwise" refers to the fact that each subgroup effect estimate is asymptotically normal.
3

---

Assumption 2.4 requires each estimator ((k) to be consistent and asymptotically normal for some rkk), which may not be equal to r. This is not a particularly strong assumption, as we discuss below.
2.2 Asymptotic Normality of Biased Estimators
In this section, we give two simple examples to illustrate the principle that multiple estimators (kk) may be asymptotically normal, even if they are asymptotically biased (i.e., r(k) + T). In both cases, there is a distinction between the stutistical assumptions required to obtain asymptotic normality, and the causal assumptions required for ((k) to identify the causal effect т. For simplicity in both examples, estimate the GATE, 7, in a single group = 1 covered by all datasets. The statistical claims here also extend to GATE estimation with multiple groups (55
restrict to the setting of comparing onedimensional estimates 7(k) E R, which
Example 2.1 (Variation in confounding across datasets). Suppose that there is one estimator of the GATE per observational dataset, and each estimator seeks to estimate the population quantity, T(k) = E[gg(11 (g) 9N (0, Xx) = D = Al,. where X2 denotes the controls used in each study, and gs (A, Xx) = E A, Xx, D = k] and my (Xx) = P(A = Xx: D = k). We assume that 1 < (z) < 1 -) for some 7) > O for all æ, k. Note that r(k) is only a statistical quantity: identifying this with the causal quantity (the GATE) requires additional assumptions like unconfoundedness,, that Y. I A Xg for the given dataset D. This assumption may hold for some datasets, but not others, particularly if the set of observed confounders X₁ differs across datasets.
Regardless of the interpretation of r(k), one can construct estimators of it that are consistent and asymptotically normal using flexible machine learning estimators One approach, given in Chernozhukov et al. [12.. is to use double machine learning (DML), which employs cross- fitting to produce estimates +kk) based on the doubly-robust score 45 while using plug-in esti- mates OR, rit based on machine learning models. This approach achieves asymptotic normality,
(N (())) 3 (0, 1), under regularity conditions that allow for flexible machine learning estimators that converge at slower than parametric rates, and where (²k) converges in probability to the variance of the doubly robust score [See Theorem 5.1 of 12] for additional details].
These results hold whether or not (k)) т. as discussed in Footnote 9 of Chernozhukov et al.
For simplicity. we have focused on the case where E[ Y. G 1] is constant across datasets. When this does not hold, certain conditions enable valid transportation of treatment effects across datasets [o] with the use of transported estimators [ (see Appendix B for details).
Example 2.2 (Selection of Adjustment Strategy). Consider the two causal graphs given in Figure D and assume that all variables are binary. Each graph suggests different identification strategy for the causal effect, IE[Y do(A- a), G 1]. In Figure Ia this is identified by the (observational) quantity ET a, G 1) P(Y A', 1))P(AAA I)
a, G = 1], and in Figure Ib by front-door adjustment R as Em AM P(M
These observational quantities will typi- cally differ: the one that represents the true interventional effect depends on which graph reflects the true causal structure. However, in the case where all variables discrete and dimenimniional, still construct asymptotically normal esti- mators for both observational quantities | For more complex settings (e.g., requiring regularized MI. models for estimating con- ditional distributions) asymptotic normality has been established under certain condi- tions for general graphs 7 29
A
M
Y
A
M
Y
G
(b)
Figure 1: (Ex. 2.2 In 4 M and Y are confounded by unobservables (bi-directional dotted arrow). In bb. A and Y are confounded, but the causal effect is identified via front- door adjustment.
Remark 2.2. In each example, there are multiple estimators available, each asymptotically normal under basic statistical assumptions, but potentially biased in the sense that r(k) * т. In the first example, this bias occurs if X is not sufficient to control for confounding in all observational
'A rich literature focuses on establishing such results, beyond the approach this example 2 20 60 R
'This follows from the use of maximum likelihood (i.e., empirical counts) for estimating each conditional distribution, and applying the delta method to the front-door estimator.

---

datasets. In the second, this bias arises in given estimator if the causal graph is incorrectly specified. Assumption 2.3 corresponds assuming that both the statistical assumptions and causal assumptions hold for one of the candidate estimators, e.g.. X is sufficient to control for confounding in at least one study (Example 2.T.. or that one of the causal graphs is correct (Example 2.2
2.3 Asymptotic Normality of GATE Estimators with Transportation
assumes that E[y, Yo G = i] is constant across datasets. In practice, it may be necessary correct for differences (not captured by group indicators) between the observational and RCT populations. There exist estimators for the ATE in this setting under mild additional assumptions These extend in straightforward way to estimators of the GATE, but proving asymptotic normality is nuanced in high-dimensional settings when using flexible machine learning methods to estimate nuisance functions. For completeness, inspired by Semenova and Chernozhukov [55], we demonstrate that doubly-robust GATE estimator for this setting is asymptotically normal under reasonable conditions (Assumption C to C.Sp. Details on the estimator, and the corresponding proof of normality, are given in Appendix and may be of independent interest.
Example
2.1
15 [E
2.4 Testing for Bias under Asymptotic Normality
Under Assumption 2.4 each observational estimate (k(:)) be compared to the estimate from the randomized trial (0) for i E In, the groups with common support. Since the observational and randomized datasets are distinct, we can conclude that each (kk is independent of (0), and use this to test for the hypothesis that rk(k)= = Ti.
Proposition 2.1. For an observational estimator +kk), assume Assumptions 2.2 and 2.4 hold.
Furthermore, let N = Ng + No with fixed proportions, where N₂ pN,, No (1 P)N for p E (0. Define the test statistic
Tx(K, (k(k) - 7. (0)
- (k(k)
(3)
where 8² = 6P(M) N2 +
0710) No
is the estimated variance, and (k(c)) = (k)) T₁. This test statistic
converges in distribution to a normal distribution as N - oc, (k. 1).
We present the proof for Proposition the construction of simple hypothesis tests. For instance, one can construct Wald test for Ho Tr(k) Ti. with asymptotic level a by setting A4 (A) = 0 in Equation (3) and rejecting Ho whenever, [T' (ks i)|> 2 where Ra/2 is the 1 ax/2 quantile of the normal CDF, Moreover, the asymptotic power this test (the probability correctly rejecting Ho) is given by
in Appendix
This asymptotic normality allows for
(k)k)
[[((k)
1-
+ Fa/2
+ +
Ea/2
(4)
0k.0
where or. == No + (0) asymptotic confidence interval for (kk)
o² (R).
No [see Theorems 10.4, 10.6 of 62 Likewise, Assumption 2.4 implies an
((()))) = (k(()) 2o/2 di(k)
Ea/2 (k))
(k(k)) +
(5)
VNr
/Nx
3 Meta-Algorithm for Conservative Extrapolation
In this section, we more formally introduce our algorithm (Algorithm TT. There are two primary steps: falsification of estimators, and combination of confidence intervals. First, we attempt falsify candidate estimators via hypothesis testing, rejecting estimator & whenever we are able to reject the null hypothesis Ho 7a (k))) Vi Vi E IR- We use Bonferroni correction to control the false positive rate of the test. For the combination of confidence intervals, while we are unlikely to reject the "correct" estimator if one exists (Assumption 2.3), we may be unable to reject all "incorrect" (i.e., biased) estimators. This motivates the combination of confidence intervals (for the extrapolated effects) of the accepted estimators by taking the maximum and minimum bounds over all such intervals. Our main result characterizes the properties of our procedure, with proof in Appendix
5

---

Algorithm Extrapolated Pessimistic Confidence Sets
For each i In. RCT estimate (0) and variance (0). For each i E Zn UTo. K candidate estimators i; (k) and variances o? (k). Sample sizes No..... NR- Initialize: Empty candidate set é 2 for
Input: Desired coverage
to K do Compute (w i), Vi E In. with pi(k: 0 (Eq. if E ZR. (w(kk i) S #a/1Inn. then < Cu(K. (k)
end for for i To do
L + min (a()((a/2) and U, + max 0,(k)(a/2) (Fq. end for Return: L, U, for each i To.
Theorem 3.1 (Properties of Algorithm D. Under Assumptions 2. and 2.2 the output of Algorithm
has the following asymptotic properties as N - oc, where N denotes the total sample size, and the samples used for all estimators are of the same order N₂ = No, vk Z 1, for some ps > O.
1. Under Assumptions
2.3 and 2.4 for each Io.
lim P(T [L., Ú,)) 2 1
(6)
N-+00
2. Under Assumption 2.4 for each estimator where Tz # Ti for some IR.
lim P(k: E = N-000
The first point says that for each extrapolated effect T₁, the coverage of the final confidence interval [L,, U.] is at least 1- a in the limit. It follows from Assumption 2.3 and 2.4 that at least one estimator provides intervals [L, (a:)). (a))(a2))) that achieve asymptotic coverage of 1 - a/2. The result follows from our choice of threshold for the significance test well as application of union bounds. The second point says that we will reject estimators that are not consistent for the validation effects, in the limit. Assumption 2.4eessrres that Proposition 2. holds for all estimators, that this rejection a consequence of the asymptotic power in Equation
going to 1 for a fixed bias as - co.
Remark 3.1. Equations 4 and 5 are useful for building further intuition. All of the candidate confidence intervals shrink at rate of O(I/VN) as the overall sample size increases. For sufficiently large N, the width of our generated intervals will depend largely our power to reject biased estimators, which will be higher for observational estimates with larger biases for validation effects.
4
Semi-Synthetic Experiments
4.1 Setup of Simulation
We generate semi-synthetic RCTs and observational datasets with covariates from the Infant Health and Development Program (IHDP), a randomized experiment on premature infants assessing the
effect of home visits from a trained provider on the future cognitive performance [8] The outcomes
are simulated. Our data generation based on the partial IHDP dataset used in (25), which includes Yln = 985 observation, 28 covariates, and binary treatment variable. We construct a scenario where there are four subgroups, defined by the infant's birth weight and maternal marital status: (high [2 2000g], married), (low [< 2000g], married), (high, single) and (low, single), which shorthand as HM, LM, HS and LS. We include all subgroups in the observational studies, but exclude the latter two subgroups for the simulated RCT (i.e. only infants with married mothers are in the RCT).
For each simulated dataset, we generate RCT and observational studies. For the observational studies, we resample the rows of the IHDP dataset to the desired sample size n = r 7o- We performed weighted sampling to induce a different covariate distribution for observational studies, such that male infants, infants whose mothers smoked, and infants whose mothers worked during pregnancy are less prevalent. Then, we introduce confounding in the observational data, generating m. continuous confounders and my binary confounders. Finally, we simulate outcomes in each

---

dataset, modifying the response surface given in Hill [25] In our experiments, we may choose to conceal some confounders in each observational study to mimic unobserved confounding, denoting the number of concealed variables across the K studies as c₂ = (Cz1, C22, ..., Cz ). For further details on confounder generation, outcome simulation, and confounder concealment, see Appendix F]] Data generation parameters include K, r. me. my. Cz. and the significance level a. By default, we set K = 5, r = 10, m, = 4, mn = 3, C2 = (0, 0, 2, 4, 6), and a 0.05. The full hyperparameter search is provided in Appendix and details of hyperparameter tuning can be found in Appendix
4.2 Implementation and Evaluation of Meta-Algorithm
To implement Algorithm 1 we first obtain GATE estimates for the four subgroups and their estimated
variances in each observational study, combining techniques from the DML and trasportabilityy
literature [55] 15I. Estimation details are shown in Appendices B and C For the RCT, we stratify the data into the subgroups HM, LM and estimate the GATES as the difference of mean outcomes between the treated and untreated. The x tests in Algorithm| [ are applied to both GATE estimates in
the HM and LM subgroups ( (IZ - 2), and the significance level of the tests is set at a/4.
We evaluate performance using two main metrics: (1) the coverage probability of the output confi- dence intervals (ideally at least ax), and (2) the width of the confidence intervals (narrower is better). In addition to assessing the intervals produced by Algorithm I which we call Extrapolated Pes- simistic Confidence Sets (ExPCS), we will evaluate intervals produced by a variant of our algorithm, called Extrapolated Optimistic Confidence Sets (EXOCS). In ExOCS, after falsifying estimators, we combine confidence intervals using random-effects meta-analysis the non-falsified observational studies. We compare ExPCS and ExOCS against two baselines. Meta-Analysis is a random-effects meta-analysis on all observational studies, described in Section 6 with heterogeneity variance estimated via the DerSimonian-Laii for aggregating observational study results. The second baseline, Simple Union, uses the maximum upper bound and minimum lower bound of the 1 - a confidence intervals across all observational studies, with no falsification procedure
moment method [17] This baseline is the current standard
4.3 Results
We perform three semi-synthetic experiments to assess the performance of our proposed meta- algorithm under different scenarios. The first experiment applies our algorithm under the default settings given in Section 4.1 In the second experiment, we vary the sample size ratio between the observational studies and the original RCT, r. from 1 to 10. In the third experiment, we vary the proportion of biased observational studies by setting C₂ to be (0, 0, O, 0, 0), (0, 0,0, 0, 0,3). (0, 0, 0, 3, 3) or (0, 3, 3, 3, 3). corresponding to 0, 1, 2, 4 studies being biased out of a total of 5 observational studies. Results for the latter two experiments are shown over 100 simulations of the datasets. Results for all experiments are shown in Figures 2 3 and Figure 5 in Appendix G respectively. We observe the following:
Meta-algorithm produces confudence intervals that cover the true GATE with nominal probubilit: We demonstrate in Figure g the application of our meta-algorithm (ExPCS), variant of it (ExOCS), and two other baselines on one datasct. Our goal is to produce narrow confidence intervals that still cover the true GATEs in the extrapolated subgroups. The confidence intervals of ExPCS cover the true GATES in the extrapolated subgroups with reasonable widths. In contrast, intervals produced by Meta- Analysis fail cover the true GATE in both extrapolated subgroups due to the false assumption of unbiasedness across all studies. The ExOCS approach produces narrow intervals for the extrapolated effects, though it barely covers the true effect in the HS subgroup. This hints at the need for conservative combination of non-falsified studies. However, an overly conservative approach (e.g. Simple Union) produces wide intervals that may be of little use for meaningful inference.
Although Meta-Analysis produces confidence intervals with inadeuuate coverage, its intervals for the married subgroups still have considerable overlap with the intervals produced by the RCT. This suggests that testing the meta-analyzed GATE estimates against the RCT GATE estimates may not be enough to demonstrate their validity. Compared to our ExPCS intervals, the lower bounds of the
*Note that Simple Union combines 1 - a contidence intervals, while our approach combines 1 - a/2 confidence intervals to account for the probability of rejecting the "correct" estimator, if one exists. As result, Simple Union intervals do not always strictly cover the intervals produced by ExPCS.
7

---

15
Validation Effects
Estrapolated Effects
10
RCT
EPPCS
GATE
EHOCS
Mett-Arulplk
Simple
LM
HM
LS
HS
Figure 2: The confidence intervals for group average treatment effects (GATE) within the four subgroups output by our algorithm (ExPCS), our algorithm variant (ExOCS), random-effects meta-analysis on all observational studies (Meta-Analysis), simple union bound on all observational studies (Simple), and RCT, for one dataset generated using the default parameter settings laid out in Section 4. defined in Section
LM, HI,, LS, HIS represent four subgroups
1.0
Mean width of 95% confidence intervals
Coverge Probobitty
0.5
Upsampling ratio
EdPCS
Group Estimator
10
3.
ExOKS
06-
Sipple
ExPCS
21.00 9.51
5.53 3.58
8.4
0.4
L.
Simple
21.00
10.20
6.95
5.69
0.2
0.3
ExPCS
25.00
8.29
5.36 4.34
HS
Simple
26.00
9.15
6.90 6.33
Upsampling Ratio
Upsempling Ratio
Figure 3: Coverage probabilities of confidence intervals shown function of the size of the observational studies relative to the RCT. Dotted red lines stand for 95% coverage. Vertical bars are the 95% confidence intervals of the coverage probabilities. LS / HS stand for groups with low / high birth weight and single mother. Between ExPCS and Simple which have adequate coverage. ExPCS generally has narrower intervals.
Simple Union intervals are higher in several subgroups, since we use higher confidence level for the candidate intervals corresponding to each study account for probable error in study falsification.
An analysis of increasing observational study size: Figure 3 we find that the coverage of the Meta- Analysis intervals is quite low across all sample sizes and particularly decreases at higher sample sizes. This result is intuitive, as three out five studics are biased, meaning that meta-analysis will converge to biased estimate as the amount of data increases. One could attempt to fix this issue through ExOCS, which does meta-analysis after falsification. However, EXOCS has poor coverage when the sample size of the observational studies is small, since the falsification tests are underpowered (evidenced by the high probability of selecting biased studies in Appendix and the Simple Union intervals have adequate coverage across all sample sizes. However, the widths of the intervals reported at the bottom of Figure 3 show that ExPCS intervals are narrower when there is adequate power, i.e. at higher sample sizes. Ultimately, ExPCS will tend to provide intervals that cover the true effect regardless sample size, and in the case we have sufficient power, these intervals will both have good coverage and narrower width, allowing for more meaningful inference.
G Table 2 Both EXPCS
Women's Health Initiative (WHI) Experiments
In order to assess our approach in a real-world setting, we use clinical trial and observational data available from the WHI. Each subgroup is supported in both RCT and observational data, which proves useful for evaluation. At high level, we "hide" some number of subgroups from the RCT, estimate a confidence interval of the effect estimate using our algorithm on the remaining data, and compare the result to the hidden RCT estimate. We this over a large set of possible "held- out" subgroups, yielding 2000 different scenarios on which to test our approach. Because the original observational datasets replicate the RCT results fairly well using standard methods, we create additional "biased" datasets by sub-selecting the original observational dataset in way that induces
8

---

selection bias. We evaluate each method, for each held-out subgroup. according to the length of the intervals as well as coverage the RCT point estimates. Below, we describe the specifics of the data, the experimental setup, and the main results of the analysis. For additional details on data preprocessing, setup, and evaluation, see Appendix F
5.1 Setup
The Postmenopausal Hormone Therapy (PHT) trial, i.e. the RCT used in this analysis, was run on postmenopausal women aged 50-79 years who had an intact uterus. It studied the effect of hormone combination therapy on several types of cancers, cardiovascular events, and fractures. The observational study (OS) was run in parallel, had a similar follow-up time to the RCT, and tracked similar outcomes. In our analysis, we use composite outcome, where Y if any of the following events are observed to occur in the first 7 years of follow-up. and Y = 0 otherwise: coronary heart disease, stroke, pulmonary embolism, endometrial cancer, colorectal cancer, hip fracture, and death due to other causes. This represents binarization of the "global index" time-to-event outcome from the original study, where = 0 could also occur due to censoring. We establish treatment and control groups in the OS based on explicit confirmation or denial of usage of both estrogen and progesterone in the first three years. We use only covariates measured in both the RCT and OS to simplify analysis.
5.2 Evaluation
Our empirical evaluation consists of several steps. In the first step. we replicate the principal results from the PHT trial, given in Table 2 of [51]. by fitting a doubly robust estimator (of the style given in Appendix C) on the WHI OS data. Then, while treating the WHI OS dataset as the "unbiased" observational dataset, we simulate additional "biased" observational datasets by inducing selection bias into the WHI OS. The exact mechanism of selection bias and its clinical intuition is given in Appendix B Importantly, this is the only part of the evaluation that involves any simulation.
The second step to construct a large suite of tasks on which evaluate our method, by considering different sets of validation validation-extrapoliuee extrapolation subgroups. To construct the subgroups. we consider all pairs of a selected set of binary covariates (see Appendix E. where each pair defines four subgroups. For example, one covariate pair is ('current smoker" "currently drinks alcohol"). We treat two of the subgroups as validation subgroups and two as extrapolated subgroups. For the latter groups, we apply our algorithm without access to the RCT data, and only use the RCT data for final evaluation. The total number of covariate pairs is 592, leading For each task, we evaluate ExPCS (our method), ExOCS, Simple, and Meta-Analysis (described in Section 4.2). Additionally, we evaluate an "oracle" method, which is identical to ExPCS, except that it always selects only the original observational study (i.e. the base WHI OS to which we have not added any selection bias). For each method, we compute the following metrics, averaged across all tasks- Length: length of the confidence interval, Coverage: percentage of tasks where the interval covers the RCT point estimate. In addition, we report the Unbiased OS Percentage: the percentage of tasks where the ExPCS approach retains the unbiased study after the falsification step.
1184 distinct "tasks" (i.e., extrapolated groups).
5.3 Results
Table repoots the metrics above, averaged across all extrapolated subgroups.
Coverage
Length
OS %
Simple
0.39
0.416
Compared to the "simple" baseline, our approach has better coverage with much shorter confidence intervals. Our falsifi- cation procedure retains the unbiased ob- servational study 99% of the time, yield- ing near-oracle coverage rates, but produces substantially shorter intervals than the "sim- ple" baseline. Recall that the simple base- line takes a union over all 1 - intervals estimated from each observational dataset, while ExPCS takes union of smaller number of slightly wider (1 a/2) confidence intervals.
Meta-Analysis
0.03
0.260
FxOCS ExPCS (ours)
0.28
0.058
0.45
0.081
0.99
Oracle
0.44
0.068
Table Coverage. length, and unbiased OS % of ExPCS and baselines. ExPCS achieves comparable coverage to the oracle method with highly efficient intervals. Additionally, we do not reject the unbiased OS in 99% of the tasks.

---

Compared to the Meta-Analysis and ExOCS baselines, we achieve comparable (or much better) length with substantially better coverage. In particular, compared to meta-analysis, we achieve tighter intervals and also cover the RCT estimate with higher frequency. This result is intuitive, since one will get biased estimate biased observational studies are included in the meta-analysis. Additionally, conservatively combining the non-falsified estimates (as opposed to ExOCS, which does a meta-analysis on the non-falsified estimates) is important to achieve good coverage (0.45 vs 0.28).
We get comparable coverage and interval lengths to the oracle method. Our coverage rate is nearly identical (0.45) to that of the oracle method (0, (0.44), with intervals that are marginally wider (0.081 vs. 0.068). Our slightly improved coverage is possible due to the wider intervals. Note that our measure of "coverage" may be pessimistic, because we track coverage of the RCT point estimate, as opposed to the true causal effect (which is unknown), and the confidence intervals are designed to cover the latter. Indeed, we report the oracle method precisely as a means of providing a more suitable comparison. Overall, our real-world results suggest that our method of falsification followed by conservative combination of intervals may be useful for biostatisticians and clinicians when doing meta-analyses.
6 Related Work
Meta-analysis for combining observational estimates Among the quantitative approaches for meta-analysis to account for potential bias, our Meta-Analysis baseline is standard for meta-analysis of observational data [24] to account for heterogeneity. Allowing for heterogeneity of treatment
effects among studies produces wider confidence intervals and thus more conservative inference. If additional study-level covariates are available (e.g. study designs, drop-out rate), several approaches
aim to adjust for potential bias, either by modeling the bias magnitude [9] 64 2T), down- down-weighting
studies with higher risk of bias 126 35
or using Bayesian hierarchical regression to account for
difference between subgroups of studies 4
63 Our work differs from these approaches. in that (1) we use information from outside the population of interest to assess bias, and (2) we do not place any assumptions on the patterns of bias across studies.
Partial identification and sensitivity analysis These methods seek to place bounds on causal effects
when they cannot be point-identified. Our method can be seen as alternative way of doing so, with fundamentally different type of assumption. Methods for partial identification rely on having discrete variables and known causal graph (typically including unobserved confounders) [8 Section 9]. Methods for sensitivity analysis, on the other hand, translate assumptions about the strength and nature of unobserved confounding into bounds causal effects not make any such assumptions, c.g., we allow for continuous variables, and when some candidate estimators are biased due to unmeasured confounding, we do not place any limit a-priori on the bias. An extended related work is given in Appendix
47
48 55
In contrast, we do
7 Discussion and Limitations
We have presented a meta-algorithm that constructs conservative confidence intervals for group average treatment effects of subgroups that not represented in RCTs, but are represented in observational studies. Under the assumption that there exists at least one candidate estimator that is asymptotically normal and consistent for both the validation and extrapolated effects, these intervals will achieve the correct asymptotic coverage of the true effect. However, our method is not without limitations. Most notably, may fail to reject the null hypothesis due to low power, e.g., when an observational estimate f(k) has high variance. In practice, we expect that our approach will be most useful when the observational studies in question have large sample sizes, leading to higher-precision estimates of potential bias, and smaller confidence intervals on the extrapolated effects. Our hope is that methods such ours will lead to higher confidence in observational estimates when RCT data is available to falsify observational studies that do not replicate known causal relationships. Finally, great care should be taken appropriately validate and soundly interpret the results of our method in practice, especially with more sensitive subgroups (e.g. with respect to race or gender).
10

---

Acknowledgments: We would like to thank Ahmed Alaa, Hunter Lang, Christina Ji. Hussein Mozannar, and other members of the Clincal Machine Learning group for helpful discussions and valuable feedback on the manuscript. 7.H was supported by an ASPIRE award from The Mark Foundation for Cancer Research and by the National Cancer Institute of the National Institutes of Health under Award Number F30CA268631. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes Health. MO and DS were supported in part by Office of Naval Research Award No. N00014-21- 1-2807. MCS was supported by the LEAP program from the Ministry of Science and Technology in Taiwan.
References
[1] Andrew Anglemyer, Hacsi T Horvath, and Lisa Bero. Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials. Cochrane database of systematic reviews, (4):MR000034, April 2014.
[2] Susan Athey, Guido W Imbens, and Stefan Wager. Approximate residual balancing: De-Biased inference of average treatment effects in high dimensions. arXiv preprint (1604.07125), April 2016.
[3] Susan Athey, Julie Tibshirani, and Stefan Wager. Generalized random forests. The Annals of Statistics, 47(2):1148 148-1178, April 2019.
[4] Hailey R Banack, Jay S Kaufman, Jean Thetawski-Wende Bruce R Troen, and Steven D Stovitz. Investigating and remediating selection bias in geriatrics research: the selection bias toolkit. Journal of the American Geriatrics Sociery, 67(9):197011970 2019.
[5] Alexandre Belloni, Victor Chernozhukov, and Christian Hansen. Inference for high-dimensional sparse econometric models. arXiv preprint 1xXiv:1201.022)) 2011.
[6] Alexandre Belloni, Victor Chernozhukov, and Lie Wang. Square-root lasso: pivotal recovery of sparse signals via conic programming. Biometrika, 98(4):791-806, 2011.
[7] Rohit Bhattacharya, Razich Nabi, and Ilya Shpitser. Semiparametric inference for causal effects in graphical models with hidden variables. arXiv preprint (2003.12659) March 2020.
[8] Jeanne Brooks-Gume Fong-ruey Liaw, and Pamela Kato Klebanov. Effects of early intervention on cognitive function of low birth weight preterm infants. The Journal of pediatrics, 120(3): 350-359, 1992.
[9] Peter Bühlmann and Sara Van De Geer. Statistics for high-dimensional data: methods, theory and applications. Springer Science & Business Media, 2011.
[10] Xiaohong Chen and Halbert White. Improved rates and asymptotic normality for nonparametric neural network estimators. IEEE Transactions on Information Theory, 45(2))6882-6) 191, 1999.
[11] Victor Chernozhukov, Mert Demirer, Esther Duflo, and Iván Fernández- Val. Generic machine learning inference on heterogenous treatment effects in randomized experiments. arXiv preprint (17 (1712,04802), December 2017.
[12] Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. The econometrics journal, 21(1)C1--668, January 2018
[13] Jordan M Cloyd, Victor Heh, Timothy M Pawlik, Aslam Ejaz, Mary Dillhoff, Allan Tsung, Terence Williams, Laith Abushahin, John FP Bridges, and Heena Santry. Neoadjuvant therapy for resectable and borderline resectable pancreatic cancer: a meta-analysis of randomized controlled trials. Journal of clinical medicine, 9(4):1129, 2020.
[14] Issa Dahabreh, Sarah Robertson, Lucia Petito, Miguel A Hernán, and Jon Steingrimsson. Efficient and robust methods for causally interpretable meta-analysis: transporting inferences from multiple randomized trials to a target population. arXiv preprint (1908.09230) August 2019.
11

---

[15] Issa J Dahabreh, Sarah E Robertson, Jon A Steingrimsson, Elizabeth A Stuart, and Miguel A Hernán. Extending inferences from randomized trial a new target population. Statistics in medicine, 19911):19999201 June 2020.
[16] Irina Degtiar and Sherri Rose. A review of generalizability and transportability, arXiv preprint (2104. February 2021.
[17] Rebecca DerSimonian and Nan Laird. Meta-analyse in clinical trials. Controlled clinical trials, 113):177-188, 1986.
[18] Guilherme Duarte, Noam Finkelstein, Dean Knox, Jonathan Mummolo, and Ilya Shpitser. An automated approach to causal inference discrete settings. September 2021.
[19] David M Eddy, Vie Hasselblad, and Ross Shachter. An introduction to bayesian method for meta-analysis: the confidence profile method, 1990.
[20] Max Farrell. Robust inference on average treatment effects with possibly more covariates than observations. arXiv preprint (1309.4686), February 2018.
[21] Sander Greenland. Multiple-bias modelling for analysis of observational data. Journal of the Royal Statistical Society: Series A (Statistics in Society), 168(2):267-306, 2005.
[22] Gordon H Guyatt, Andrew Oxman, Regina Kunz, Gunn E Vist, Yngve Falck- Yiter, and Holger J Schünemann. What is "quality of evidence" and why is it important to clinicians? Bmj, 336(7631))))))) (:10551)):9999999 9989, 2008.
[23] Gordon Guyatt, Andrew D Oxman, Gunn E Vist, Regina Kunz, Yngve Falck- Ytter, Pablo Alonso-Coello, and Holger Schünemann. Grade: emerging consensus rating quality of evidence and strength of recommendations. Bmj, 336(7650):924-926, 2008.
[24] Julian PT Higgins, James Thomas, Jacqueline Chandler, Miranda Cumpston, Tianjing Li, Matthew J Page, and Vivian Welch. Cochrane handbook for systematic reviews of interven- tions. John Wiley & Sons, 2019.
[25] Jennifer L Hill. Bayesian nonparametric modeling for causal inference. Journal of Computa- tional and Graphical Statistics, 20(1):217-240 2011 1.
[26] Joseph G Ibrahim and Ming- Chen. Power prior distributions for regression models. Statistical Science, pages 46-60, 2000.
[27] Guido W Imbens and Donald B Rubin. Causal Inference in Statistics, Social, and Biomedical Sciences. Cambridge University Press, April 2015.
[28] Daniel Jacob. Group average treatment effects for observational studies. arXiv preprint (191 1.02688), November 2019.
[29] Yonghan Jung, Jin Tian, and Elias Bareinboim. Estimating identifiable causal effects through double machine learning. AAAI, aaai.org. 2021.
[30] Nathan Kallus, Aahlad Manas Puli, and Uri Shalit. Removing hidden confounding by exper- imental grounding. In S. Bengio and H. Wallach and H. Larochelle and K. Grauman and N. Cesa-Bianchi and R. Garnett, editor, Advances in Neural Information Processing Systems, 31. Curran Associates, Inc., 2018
[31] Keum, DH Lee, DC Greenwood, JE Manson, and E Giovannucci. Vitamin d supplementation and total cancer incidence and mortality: meta-analysis of randomized controlled trials. Annals of Oncology, 30(5):733-744 2019.
[32] Ola Landgren, David S Siegel, Daniel Auclair, Ajai Chari, Michael Boedigheimer, Tim Welliver, Khalid Mezzi, Karim Iskander, and Andrzej Jakubowiak. sarfizzomib-leaaiddomiie dexamethasone versus http:/omi--enaiiimmideeeeeeeeeee nosed multiple myeloma: results from the prospective, longitudinal, observational commpass study. Blood, 132:799, 2018.
in patients with newly diag-
12

---

[33] Bin Li, Kaili Ren, Lei Shen, Peijie Hou, Zhenqiang Su, Alessandra Di Bacco, Jin-Liern Hong. Aaron Galaznik, Ajeeta B Dash, Victoria Crossland, et al. Comparing rrnezzombb-lnnaddomiid dexamethasone (vrd) with http://miweeeallddddddddeeeeeeee newly diagnosed multiple myeloma (ndmm) in two observational studies. Blood, 132:3298. 2018.
(krd) in the patients with
[34] Mare Lipsitch, Eric Tchetgen Tchetgen, and Ted Cohen. Negative controls: a tool for detecting confounding and bias in observational studies. Epidemiology, 2133):33--388 May 2010.
[35] Beat Neuenschwander, Michael Branson, and David Spiegelhalter. A note on the power prior. Statistics in medicine, 28(28):3562-3566 2009.
[36] NIH. Relating clinical outcomes in multiple myeloma to personal assessment of genetic profile (com-mpass). Climical Trials website. https://climicattrials, gow/c2/stow/ACr004554999/ 2016.
[37] Miruna Oprescu, Vasilis Syrgkanis, and Zhiwei Steven Wu. Orthogonal random forest for causal inference. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 4932-4941. PMLR, 2019.
[38] Chan Park and Hyunseung Kang. A groupwise approach for inferring heterogeneous treatment effects in causal inference. arXiv preprint 04427), August 2019.
[39] Judea Pearl. Causal diagrams for empirical research. Biometrika, 82(4):669 688, 1995.
[40] Judea Pearl. Generalizing experimental findings. Journal of Causal Inference, 3(2):259-266 September 2015.
[41] Judea Pearl and Elias Bareinboim. Transportability of causal and statistical relations: A formal approach. Proceedings of the AAAI Conference on Artificial Intelligence, 25(1):247-25 August 2011.
[42] Judea Pearl and Elias Bareinboim. External Validity: From Do-Calculus to Transportability Across Populations. Statistical Science, 29(4):579-595, November 2014.
[43] Francesco Paolo Prete, Angela Pezzolla, Fernando Prete. Mario Testini. Rinaldo Marzaioli, Alberto Patriti, Rosa Maria Jimenez- Rodrigue,, Angela Gurrado, and Giovanni FM Strippoli. Robotic versus laparoscopic minimally invasive surgery for rectal cancer: a systematic review and meta-analysis of randomized controlled trials. Annals of surgery, (67 1046, 2018.
[44] Teresa C Prevost, Keith R Abrams, and David R Jones. Hicrarchical models in generalized synthesis of evidence: example based on studies of breast cancer screening, Statistics in medicine, 19(24))3359-3377 2000.
[45] James M Robins and Andrea Rotnitzky. Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Statistical Association, 90(429):122-129, March 1995.
[46] James M Robins, Andrea Rotnitzky, and Lue Ping Zhao Estimation of regression coefficients when some regressors are not always observed. Journal of the American statistical Association, 89(427)8466886 1996, 1994.
[47] P R Rosenbaum and D B Rubin. Assessing sensitivity unobserved binary covariate in an observational study with binary outcome. Journal of the Royal Statistical Society. Series B, Statistical methodology, 45(2):212-22 1983.
[48] Paul R Rosenbaum, PR Rosenbaum, and Briskman. Design of observational studies, volume 10. Springer, 2010.
[49] Evan Rosenman, Guillaume Basse, Art Owen, and Michael Baiocchi. Combining observational and experimental datasets using shrinkage estimators. arXiv preprint 2xXi::0002.06708 2020.
[50] Evan TR Rosenman, Art Owen, Mike Baiocchi, and Hailey R Banack. Propensity score methods for merging observational and experimental datasets. Statistics in Medicine, 2021.
13

---

[51] Jacques E Rossouw, Gamet L Anderson, Ross L Prentice, Andrea Z LaCroix, Charles Kooper- berg, Marcia Stefanick, Rebecca Jackson, Shirley AA Beresford, Barbara V Howard. Karen C Johnson, et al. Risks and benefits of estrogen plus progestin in healthy postmenopausal principal results from the women's health initiative randomized controlled trial. Jama, 288(3):321-333, 2002.
women:
[52] Peter F Schnatz. Xuezhi Jiang, Aaron K Aragaki, Matthew Nudy, David M O'Sullivan, Mark Williams, Erin LeBlanc, Lisa W Martin, JoAnn Manson, James M Shikany, et al. Effects of calcium, vitamin d, and hormone therapy on cardiovascular disease risk factors in the women's 's health initiative: a randomized controlled trial. Obstetrics and gynecology. 129(1):121, 2017.
[53] Martijn J Schuemie, Patrick B Ryan, William DuMouchel, Mare A Suchard, and David Madigan. Interpreting observational studies: why empirical calibration is needed to correct p-values. Statistics in medicine, 33(2)200--118, January 2014.
[54] Martijn Schuemie, George Hripcsak, Patrick B Ryan, David Madigan, and Marc A Suchard. Empirical confidence interval calibration for population-level effect estimation studies in obser- vational healthcare data. Proceedings of the National Academy of Sciences of the United States of America, 1 115(1 ():111):257725777 March 2018
[55] Vira Semenova and Victor Chernozhukov. Debiased machine learning of conditional average treatment effects and other causal functions. The econometrics journal, 24(2):264-285 2021,
[56] SPRINT Research Group, Jackson T Wright, Jr, Jeff D Williamson, Paul K Whelton, Joni K Snyder, Kaycee M Sink, Michael V Rocco, David M Reboussin, Mahboob Rahman, Suzanne Oparil, Cora E Lewis. Paul L Kimmel, Karen C Johnson, David C Goff, Jr, Lawrence J Fine, Jeffrey A Cutler, William Cushman, Alfred Cheung, and Walter T Ambrosius. randomized trial of intensive versus standard Blood-Pressure Pressure control. The New England journal of medicine, (73(22):2103-2 16, November 2015.
[57] Leonard A Stefanski and Dennis D Boos. The calculus of M-Estimation. The American statistician, 56(1):29-38, 2002.
[58] A Keith Stewart, S Vincent Rajkumar, Meletios A Dimopoulos, Tamás Masszi, Ivan Spička, Albert Oriol, Roman Hájek, Laura Rosiñol, David Siegel, Georgi G Mihaylov, et al. Carfil- zomib, lenalidomide, and dexamethasone for relapsed multiple myeloma. New England Journal of Medicine, 372(2):142 152, 2015.
[59] Marc A Suchard, Martijn J Schuemie, Harlan M Krumholz, Seng Chan You, Ruijun Chen, Nicole Pratt, Christian G Reich, Jon Duke, David Madigan, George Hripcsak, and Patrick B Ryan. Comprehensive comparative effectiveness and safety of first-line antihypertensive drug classes: a systematic, multinational, large-scale analysis. The Lancet, 394(102111111116-18866 November 2019.
[60] Stefan Wager and Susan Athey. Estimation and inference of heterogeneous treatment effectt using random forests. Journal of the American Statistical Association, 113(523): July 2018.
[61] Stefan Wager and Guenther Walther. Adaptive concentration of regression trees, with application random forests. arXiv preprint arXiv: 1503: 1003::/533.0688.. 2015.
[62] Larry Wasserman. All of Statistics: A Concise Course in Statistical Inference. Springer, New York, NY, 2004.
[63] Nicky Welton, Anthony E Ades, JB Carlin, DG Altman, and JAC Sterne. Models for potentially biased evidence in meta-analysis using empirically based priors. Journal of the Royal Statistical Society: Series A (Statistics in Society), 172(1):119-136 2009.
[64] Robert L Wolpert and Kerrie L Mengersen. Adjusted likelihoods for synthesizing empirical evidence from studies that differ in quality and design: effects of environmental tobacco smoke. Statistical Science, 4903)-50--71, 2004.
[65] Steve Yadlowsky, Hongseok Namkoong, Sanjay Basu, John Duchi, and Lu Tian. Bounds the conditional and average treatment effect with unobserved confounding factors. August 2018.
14


Reviews:
[{"review_id": "kBrayJWsRg0", "reviewer": "Reviewer_Gg3m", "summary": "Randomized controlled trials are high-standard with inclusion criteria in recruiting patients but may fail to include some heterogeneous patient subgroups in the full population. On the contrary, large-scale observational studies are likely to contain more diverse patient subgroups but they can be invalid to use due to hidden bias from some unmeasured confounders. The idea of this paper is to first validate the estimators from observational studies by comparing them with the estimator based on RCTs. This comparison is made on the patient subgroups that are observed in the RCTs. This step filters out the observational studies and their estimators that are inconsistent with the RCTs. After that, the authors use the non-rejected estimators and construct conservative confidence intervals to extrapolate the treatment effects for the subgroups that are not observed in the RCT.", "questions": "Why only consider group average treatment effects? could we extend the method to other causal parameters?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The paper recognizes the advantages and disadvantages of RCTs and observational studies and lets them complement each other in the method proposed to estimate group average treatment effects. The idea is interesting and well motivated!\n2. I am convinced by Assumption 2.3 that at least one observational estimator is asymptotically normal and consistent for both the validation and extrapolated effects, and Assumption 2.2 which says that the RCT estimator is also consistent.  I think the assumptions are necessary so that at least in large samples, we can pick up the consistent observational estimator to estimate both effects.\n3. The experiments uncover both the power and conservativeness of the proposed method. I still think the proposed idea is useful in practice.\n\nWeaknesses\n1. The groups $I_R$ and $I_O$ are given instead of being learnt from the data. I don't think this is often not the case in practice. We may know something about the effect heterogeneity but it is too strong to assume our knowledge is close to the ground truth.\n2. The proposed method (Algorithm 1)  is not very novel and a bit trivial, which includes some standard techniques we often do in practice, e.g. constructing asymptotic normal estimators and doing a t-test. The author does not do much to make the method powerful, i.e., less conservative. I feel all the theoretical results are expected and follow from the DML literature.\n3. The method looks quite conservative in Figures 3 and 4. The interval width is similar to the simple Union, which reports the union bounds of the confidence intervals of all observational studies, with no falsification procedure.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cL3_3qwEs64", "reviewer": "Reviewer_icYd", "summary": "The authors propose an approach to estimating causal effect for populations for which only observational data exists, given that experimental data exists for a related population.", "questions": "Why is it more useful to describe to describe the proposed algorithm as a \"meta-algorithm\" rather than just a simple \"algorithm\"? Is the \"meta\" prefix intended to indicate that the procedure that you describe \"is an algorithm about algorithms\", that it is \"an algorithm about meta-analysis\", or something else?", "limitations": "To their credit, the authors are very clear about many of the limitations of their proposed approach. Clearly, the authors' approach requires that both experimental and observational data are available for a given (super) population of interest. The authors also assume that at least one observational estimate (among several) is \"valid\" across all subpopulations. This is a large assumption, given that any given estimate may have high bias or variance for any given subpopulation, and that those error properties are likely to vary substantially across different subpopulations. Finally, the authors assume that \"every observational dataset has support for all groups.\" This seems unlikely. Furthermore, as the authors state clearly: \"...we may reject an observational estimator due to failures in transportability, even if it yields unbiased estimates of the extrapolated effects.\"\n\nAgain, the authors are fairly clear about the limitations and assumptions of their proposed approach. This makes an extensive empirical evaluation all the more important as a demonstration that, even with these assumptions and limitations, the approach can produce accurate estimates of causal effect. Unfortunately, the empirical evaluation is limited to a single data set with known issues.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* The authors focus on how to exploit situations in which multiple data sets (both experimental and observational) are given, and increasingly realistic situation for problems of interest in social science, medicine, and other areas.\n\n* The authors focus not just on obtaining point estimates, but on obtaining confidence intervals on those effects.\n\n* The paper is well-grounded theoretically but also features an empirical demonstration of the approach on a well-known data set.\n\n* The paper is clear about its assumptions.\n\n**Weaknesses**\n\n* The authors refer to observational estimates as \"valid\" or \"invalid\". Any estimate will have error due to bias and variance. More precise language would focus on the bias and variance of a given estimator, rather than a binary determination of \"valid\" or \"invalid\".\n\n* The experiments in Section 4 are more of a demonstration than a convincing empirical evaluation of the proposed approach. The experiments employ only a single data set (IHDP), a practice which recently has been strongly critiqued (e.g., Curth et al. 2021). The approach used by the authors on IHDP could be applied to other RCT data (see Gentzel et al. 2021), but is not.  The result is that readers are left with little empirical evidence that the method works in practice, and theoretical treatment that makes a large set of assumptions that may (or may not) be valid.\n\n* The homogeneity of causal effects is a standard (though highly suspect) assumption of most methods for estimating average treatment effect from observational studies. The authors authors assume that average treatment effect varies among subgroups, but that enough homogeneity exists that valid extrapolations can be made. However, RCTs and observational studies often differ in ways that go far beyond the randomization of treatment assignment. Randomized experiments often involve other artificial conditions that are not replicated in observational studies. Meanwhile observational studies are often substantially different from each other (and very different from experimental settings).\n\n* For most of the paper, the authors oddly refrain from naming the proposed algorithm. This leads to odd linguistic constructions such as a section headers that read \"Implementation and Evaluation of Meta-Algorithm\" and \"Meta-algorithm produces confidence intervals that cover the true GATE with nominal probability\". Then, in section 4.2, the authors name the algorithm (Extrapolated Pessimistic Confidence Sets (ExPCS)). Use the name early and throughout the paper.\n\n* The authors use of the term \"meta-algorithm\" seems unnecessary.\n\n* The use of hypothesis tests, particularly given the large literature on the limitations of this basic framework, is open to criticism.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "H11Et8n7KxM", "reviewer": "Reviewer_cPqd", "summary": "This paper proposes a meta-analysis for reliably extrapolating group level causal effects from multiple observational datasets when experimental data is available for some subgroups. The first part involves falsifying effect estimates from observational data that are biased. This is based on hypothesis testing where the statistic developed compares effect estimates from observational data to that of the RCT for groups that have experimental data. Under the assumption that at least one observational data exists for each group providing a consistent estimator (and assuming that all estimates are pointwise asymptotically normal), the statistic allows to reject biased estimates efficiently. Following this confidence intervals are generated using a simple algorithm that conservatively estimates the intervals based on the intervals of the observational data. Proposed method is evaluated on semi-synthetic IHDP data compared to simple meta-analysis and baselines that do not consist of falsification. \n\n------------------------------------ Post rebuttal update ---------------------------------------------------------------------------------------------\n\nI have read the full author response and I believe they address my major concerns with the paper. I have updated the score based on the response. ", "questions": "1. Regarding my concern on practicality, can the authors suggest situations/RCTs that are explicitly designed to provide group level effects (this is different from subgroup analysis, which is not something one should do either way).\n\n2. I would strongly encourage adding additional experiments, I have briefly seen the code and it will genuinely strengthen the paper.\n\n3. Relating methods used for effect estimation in observational data to quality of falsification. This is clear to me based on the main assumption of asymptotic normality, it is however unclear if this is the only assumption needed? I am assuming all results use the DML estimators since they satisfy the assumption? \n\n4. Conceptual clarification: Based on falsification, why can't a stronger statement be made about an estimator that aggregates the extrapolated effects?\n\n5. Not sure why but some proofs in the appendix are made unnecessarily concise. Please expand and make everything clear.", "limitations": "I believe the authors have adequately described limitations of their work. Based on my clarification questions, please consider adding comments on practicality in the limitations section. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed falsification method is interesting although feels impractical due to concerns/clarifications I mention below.\n\n2. The paper is well written, assumptions are clearly stated.\n\n3. The simplicity of the approach is appealing.\n\n4. Interesting experimental results.\n\nWeaknesses:\n1. It is really unclear how often RCT data could be available that could provide consistent group level effect estimates. This implies that the design of the RCT itself needs to be explicitly targeted for estimating group effects. Hence I am not sure how practical this approach really is.\n\n2. Although the IHDP results are interesting, and evaluation specific to the data is thorough, I believe just one semi-synthetic evaluation is fairly limited to be convincing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8CQqW8IxvKU", "reviewer": "Reviewer_hHvj", "summary": " Randomized controlled trials (RCTs) are considered the gold standard for studying the causal relationship between treatments and outcomes, and  Clinical Practice Guideline (CPG) policy recommendations are based on experimental results from RCTs. However, due to the cost (time and money) and ethical and methodological considerations, the populations in RCTs are narrow. Hence, the treatment effects outside of the support are missing. An alternative approach is to estimate treatment effects using observational data. However, the estimated treatment effects from these historical data might be biased due to the failure to control confounding effects or the existence of selection bias in the data. This work considers the problem of providing unbiased treatment effects, along with the confidence intervals, outside of the support of RCTs from the existing observational datasets. Given RCT data and multiple observational studies, this work first provides a hypothesis-testing technique to remove the observational estimates that are not consistent with the estimate of RCT. Under the assumption (Assumption 2.3) that there is a least one observational estimator that is a consistent estimator of RCT, i.e., the strong ignorability assumption holds, and the confidence intervals on the extrapolated treatment effects outside of the support of RCT is then can be provided. Throughout the experimental validation of the IHDP dataset, this work shows that their meta-algorithm can provide the confidence interval that covers the true GATE and with narrow width.", "questions": "\nThis work provides a very solid meta-algorithm that is able to provide unbiased causal effects with confidence intervals from RCT and multiple observational datasets. However, the major question that comes to my mind is whether the use case of this approach is limited or not. I am not an expert in this area. Is it the case that many of the observational studies (or RCTs ) have a function mapping that $$ G: \\mathcal{X} \\mapsto \\{1,\\dots, I\\}.$$ and the function mappings across studies are consistent so that the subsequent analysis can be provided from the meta-algorithm developed by the authors. \n\nPerhaps it is challenging, but would it be possible to provide a real-world experimental validation using the publicly available clinical database, such as the MIMIV-IV database.\n", "limitations": "This work provides a meta-algorithm that can report unbiased causal effects outside of the support of RCT with confidence intervals. This work's only potential negative social impact might happen when they falsely accept the biased estimator from an observational study that is not consistent with the estimator of RCT. The author may provide more examples and cases of what would happen when this case is true. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is very written: Problem formulation, notations, assumptions, and derivations are clearly provided.\n2. Empirical results are compared with existing meta-analysis and the comparison is comprehensive.\n\nWeakness:\n\n1. A lack of a real-world dataset is provided.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept