Detecting hidden confounding in observational data using multiple environments
Authors: Rickard Karlsson
JH Krijthe
Venue: NeurIPS 2023 poster (2023)
Keywords: causal inference
hidden confounding
multiple environments
independent causal mechansisms
independence testing
TL;DR: We show that, when assuming independent causal mechanisms, there exist testable conditional independencies in multi-environment data that are only violated in the presence of hidden confounders.

Abstract: A common assumption in causal inference from observational data is that there is no hidden confounding. Yet it is, in general, impossible to verify the presence of hidden confounding factors from a single dataset. Under the assumption of independent causal mechanisms underlying the data-generating process, we demonstrate a way to detect unobserved confounders when having multiple observational datasets coming from different environments. We present a theory for testable conditional independencies that are only absent when there is hidden confounding and examine cases where we violate its assumptions: degenerate & dependent mechanisms, and faithfulness violations. Additionally, we propose a procedure to test these independencies and study its empirical finite-sample behavior using simulation studies and semi-synthetic data based on a real-world dataset. In most cases, the proposed procedure correctly predicts the presence of hidden confounding, particularly when the confounding bias is large.

Detecting Hidden Confounding In Observational Data Using Multiple Environments
Rickard K.A. Karlsson
Jesse H. Krijthe
Department of Intelligent Systems Delft University of Technology
Department of Intelligent Systems
Delft University of Technology
The Netherlands
The Netherlands
I. r.k. a. karlsssonttudel.t....
j J. ....krijthh@tuueee
Abstract
A common assumption in causal inference from observational data is that there is no hidden confounding. Yet it is, in general, impossible to verify this assumption from a single dataset. Under the assumption of independent causal mechanisms underlying the data-generating process, we demonstrate way to detect unobserved confounders when having multiple observational datasets coming from different environments. We present theory for testable conditional independencies that are only absent when there is hidden confounding and examine cases where we violate its assumptions: degenerate & dependent mechanisms, and faithfulness violations. Additionally, we propose a procedure to test these independencies and study its empirical finite-sample behavior using simulation studies and semi-synthetic data based on real-world dataset. In most cases, the proposed procedure correctly predicts the presence of hidden confounding, particularly when the confounding bias is large.
Introduction
Estimating the causal effect of a treatment outcome is fundamental challenge in many areas of science and society. While this is straightforwardly done using data from randomized studies, using observational data for this task is appealing since they often more feasible collect while also being more representative such data is often assumed there no hidden confounding. When this untestable assumption is violated we run the risk of confusing causal relationships with spurious correlations. This can have serious consequences such unknowingly suggesting non-effective or, even worse, potentially harmful treatment. Therefore, detecting the presence of hidden confounding is an important problem.
the population of interest [Pearl 2009]. To identify causal effects using
Data collected from different sources often tend to be heterogeneous due to changing circum- stances time shifts. In this work, we show how such heterogeneity can be exploited make hidden confounding testable soleły from observational data. We consider a setting where observational data has been collected from different environments treatment T and outcome Y, as well as covariates X that are known confounders between T and Y. Further, we assume that the data heterogeneous across these environments under the principle of Independent Causal Mechanisms [Peters et al. 2017 (Schölkopf et al. [022], which states that a causal system consists of autonomous modules that not inform or influence each other. The question ask is whether there exists further hidden confounding between T and Y after having adjusted for X. that is the case, the causal effect of I on Y not identifiable in general. Perhaps surprisingly, demonstrate way to decide if the causal effect would be identifiable by deriving
each environment, we observe the same
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Causal mechianisms
X
Obenaal conteender
Hilken
(A)
A
Sumples/ N,
Environments 1,..,
Environments
Observed
Unobserved
(a)
(b)
Figure We have multiple observations 1...., .N of treatment T,(), outcome Y and confounder x in different environments E(*), The dashed bi- bi-directed edge between X (A) and
((()) allows for any causal relationship, or lack thereof, between observed and hidden confounder.
(a): The hierarchical structure of the multi-enviromment data; the causal mechanisms are unobserved we know indicator E(k for what environment observations belong to. (b): By unrolling the graph in (a) same environment (when not conditioning on the mechanisms). This can be exploited to detect the presence of the hidden confounder.
that dependencies exist between any pairs of observations (i.j) from the
testable implications for whether hidden confounding is present or not. We achieve this by exploiting the hierarchical structure of the problem, shown in Figure [
As an illustration of setting where this might be applied, there are many existing multi-level studies in which individuals are nested in clusters and non-randomly assigned to treatment/contro) on can have pre-defined clusters in multi-level observational
an individual level. For instance, study that investigate specific treatment and outcome from multiple hospitals [Goldstein et al. 22022] or schools [Leite al.) 2015] that care for patients/pupils from different demographics. Here the clusters constitute different environments. Often we might suspect the existence of potential individual-level confounders such as socio-economic status. Now, if these confounding factors have different distributions at each cluster our work proposes way to statistically test the presence of confounding between the treatment and outcome even when we do not observe the confounding factors directly.
Another example where method is suitable is when there multiple observational studies where no randomized control trials are available, a problem area where systematic procedures are still lacking [Mueller et al.] 2018). In particular, individual participant data aeta-analyses are type of analysis that uses all individual-level data from multiple studies instead of aggregating summary statistics (Riley al. 2010, Di Angelantonio et al. 20166. With this information and given that observe the same treatment and outcome across all studies, our proposed algorithm can also be used detect if there is common hidden confounding among the studies.
Contributions We prove that there exists, under the principle independent causal mechanisms, testable independencies that are only violated in the presence of unobserved confounding between treatment and outcome (Sec. [ Theorem [ Further, we explore the effect of changes and violations of our assumptions while most assumptions are necessary, we find that some can be relaxed (Sec. Ip. We then introduce statistical testing procedure that uses any suitable conditional independence test to detect the presence of hidden confounding in observational datasets from multiple environments (Sec. (..2p. Lastly, we perform an empirical snite-sample analysis of using both synthetic and semi-synthetic data generated with real-world covariates from the Twins

---

dataset [Almond et al. 2005 Louizos et al. 2017]. We observe that our proposed procedure correctly predicts the presence of hidden confounding most cases, particularly when the confounding bias is large (Sec.
Problem setting
We start with some preliminaries of the causal terminology used in this paper.
Definition 1 (Causal Graphical Model (CGM)). A causal graphical model M = (g. P) over d random variables V = (V1, .2..... Vd) comprises (i) a directed acyelic graph (DAG) with vertices V and edges Vi + V3 V; is a direct cause of V,, and (ii) joint distribution P such that it has the following Markov or causal
factoraaat factorization over G:
P(V1,V2,.... V2,..., Va) II P(V. Pa(V))
where Pa(V;) denotes the parents (direct causes) of V, in G and P(V; Pa((V)) is the causal mechanism of Vs.
The DAG 9 encodes various conditional independencies between the variables also known as d-separations in the DAG, see Pearl [1988] Chapter 3.3] which we write as A 1 d over some disjoint sets of variables A, B and C. We shall assume that condi- tional independencies in G imply the same conditional independencies in P, and vice versa:
Assumption 1 (Faithfulness & Causal Markov Property). For and G we have = A
E
(i) the faithfulness property that A IL B C causal Markov property that A 1 p A
AAA a B C, and (ii) the
X
U
We consider setting with the following variables our causal graphical model: one-dimensional treatment T E T and outcome Y E V, in addition to some observed covariates X E X' and unobserved covariates U E U. We not restrict the dimensionality of X and U. Additionally, in this setting, the environment E has a direct effect on all other variables, making it a root node in G. We say that a variable is confounder between T and Y it is cause of both T and Y in G. We assume that is known confounder between and Y, while the relationship between U and the other variables is unknown. Hence, U could be an unobserved hidden confounder (as illustrated in Figure 2 or, for instance, completely unrelated the other variables.
T
Y
Figure 2: The
setting where we want detect the presence of a hid- den confounder U in G.
Expressed in the framework of Pearl 2009), the goal of causal inference is estimate the probability P(Y dT = ()) where do(7 = represents an intervention on the treatment. Without any further assumptions, P(Y do(T = t)) not identifiable from an observational dataset; that data where we have observed the choice of treatment without influencing it (Pearl 2009]. In particular, in the setting we consider here, the interventional effect remains unidentifiable if the unobserved U is confounder between T and Y. Unfortunately, there is no way to check whether such unobserved confounders are present in single dataset. We will show, however, that things are different when we have access to observational datasets from multiple environments. In this setting, we present way to detect confounding even it is not observed, hence demonstrating a novel and valuable approach for verifying essential prerequisite for causal inference from observational data. In the rest of this section, we present the main assumptions that enable us to do this.
First, we have data from multiple environments U E). We shall use PE(-) to denote P(- E), and use small letters for the random vari- ables whenever they take particular values. In our setting, we have datasets D = (k) (k) (k) Nv.
E E with different joint distributions
'There exist other procedures that could cireumvent this issue, but these alternatives seldom avoid the unconfoundedness assumption completely. As an example, instrumental variable estimation is applicable when there is unobserved confounding between 7" and Y, but only when the relationship between the instrumental variable and I is unconfounded [Angrist al. 1996].
3

---

e(k); each has N& observations which are assumed to be i.dd within the environment. N2 is fixed but can be different for each environment. The environments are related to each other through the following assumption.
from multiple environments e()) _e?2
Assumption (Shared Causal Graph). All environments share the same underlying causal DAG g.
Next, we specify how changes in (T X, U) arise between the different environments. We shall assume that the conditional probabilities in ① which we refer to causal mechanisms vary independently per environment. This is known as the independent causal mechanism principle.
Assumption 3 (Independent Causal Mechanism (ICM) Principle [Peters al., 2017). The causal generative process of a system's 's variables is composed of autonomous modules that do not inform or influence each other. In the probabilistic case, this means that the conditional distribution of each variable given its causes (i.e., its parents in the causal graph) does not inform or influence the other mechanisms.
The above assumption covers two aspects: one concerning informing and the other about inftuencing. That the mechanisms do not inform each other can be interpreted as that knowing the conditional probability of one variable does not tell us anything about the conditional probabilities of other variables [Janzing and Schölkopfi 2010 Guo et al. 2022]. Further, we assume that changing (or performing an intervention upon) one mechanism has no influence on other mechanisms (Schölkopf et al. 2012). While this notion of independence between mechanisms can be described through non-stochastic, algorithmic mutual information [Janzing and Schölkopf 2010), we focus in this work explicitly on statistically independent mechanisms.
To model changes between environments with independent causal mechanisms, we parameterize each causal mechanism with Oy E Oy for V E {T, Y, X, U.. In each environment, these parameters are fixed and determine the distribution
PE X,U)
II
Pe, (V PPa(V))
VE(TYXX
Note that while we need to know which observations come from which environment, we do not assume to know the particular values of the individual parameters (θr, Oy, O θτ) in any environment. We shall assume that environments are randomly sampled from distribution over mechanisms by defining non-degenerate probability measures for each causal mechanism.
Assumption degennrgee Probabilistic Independent Causal Mechanisms). The independent causal mechanisms are non -egg-eeeeeetee random variables with probability measures P(ey) for all V E (T, X, U} such that Or, Oy, O and Oy are pairwise independent random variables.
With the above assumption, when we now say independent causal mechanisms, we refer to statistical independence between them. We argue that the above assumption is not particularly strong if we already have Assumption 3] the mechanisms are now allowed to change across environments in a probabilistic manner. Guo et al. (2022) proved the existence of such probability measures for the causal mechanisms when the data comprises an infinitely exchangeable sequence of random variables, drawing parallels to de Finetti's theorem de Finetti 1937).
As final note on the assumptions we have made: these assumptions should not be taken for granted and it is crucial to also understand how violations of them will influence our theory. For this reason, will cover this topic in Section 4.1
Hierarchical model of the environments Using these assumptions, we can now express the distribution of the datasets (Dx K HK as hierarchical model [Gelman et al. 2013 Chapter 5], wherein we first sample the mechanisms i.i.d. (M)) P(Ov) for = 1,..., K and V E Y,,Y, U}
and then, for each environment k obtain (Ir"",, (k) xx()), ((())) by repeatedly sampling N₂ times according to (2). Using plate notation, we can compactly represent this hierarchical model in the
augmented DAG g" shown in Figure la The edges in G* between (T((4) ((k) ,x(()) ,U(!)) are the
same as those between (T, Y, X, U) 9 and e (A)) E <Pa(V((k)) where V E (T,Y, NYY,UU7 U. for all i and k.

---

In the next parts of the paper, we will prove how the structure of g' implies novel observable constraints in the enttromenii data distribution that can be exploited to statistically test the presence of hidden confounding between T and Y after having adjusted for X. But first, we discuss the main literature related to our work.
3
Related work
This paper contributes to the growing body of research based on the principle of Independent Causal Mechanisms (Peters et al.] [017 which has inspired further research integrating machine learning and causality [Schölkopf et al. 2012 Peters et al. 2016 von Kügelgen et al. 2020 [Schlkkoff et al. 2021]. Multiple works have demonstrated how the independent causal mechanism principle could improve causal structure learning when data comes from heterogeneous environments that share the same causal model (Zhang et al., 2017 Ghassami al. 2018, Guo al. 2022]. In particular, Guo et al. [2022] demonstrated how independent causal mechanisms imply independence constraints similar to ours when the data is exchangeable but they assume there exist no unobserved latent
variables in contrast to our work where we detect the presence of such variables.
Detecting hidden confounding is hard, and often we can only reason about the plausibility of having unmeasured confounders using some sort of sensitivity analysis (Rosenbaum and Rubin 1983b VanderWeele and Ding 2017 Cinelli et al.) 2019]. Other approaches check whether treatment effect estimate is robust to changes in our assumptions by varying the adjustment set [Lu and White 2014 Oster 2019 Su and Henckel, 2022]. However, the guarantees are elusive for whether this type of robustness implies unconfoundedness. Similarly, one could test for heterogeneity of the treatment effect estimates from multiple environments and conclude that if they are different, then it is due to unobserved confounding; this idea bears resemblance to the pscudo-treatment approach discussed by Imbens and Rubin [2055] for assessing unconfoundedness. But testing heterogeneity detect confounding only works if the treatment effect is assumed to be fixed across all environments, which Lastly,Danzing
excludes many real-world settings. Lastly, Janzing and Schölkopf [2018] proposed a method to detect hidden confounding which is restricted to settings with linear models.
In the setting with data from multiple environments, various approaches have been proposed to deal with hidden confounding, typically by combining both experimental and observational data
[Bareinboim and Pearl 20166 Kallus et al. 2018 Athey et al., 2020 Hatt et al., 2022 Ilse et al.)
2022] Imbens et al./ 2002]. In contrast, we consider setting combining only observational data from multiple environments. Some works make parametric assumptions in this case, such as Huang ct al. 2020). assuming linearity with non- Gaussian noise. Since we want to avoid strong parametric assumptions, we consider approaches that avoid these assumptions. The principled Joint Causal Inference (JCI) framework [Mooij et al., [2020] is one such approach.
demonstrates how to apply traditional constraint-based methods for causal discovery [Glymour et al.) 2019] with multi- environment data. In the simpler setting with observed variables (T, Y, E) excluding X, the JCI framework informs us that Y 1 p E T is violated in the presence of hidden confounder U if E is an instrumental variable. But this means, once again, that the treatment effect is fixed across environments we assume E has direct effect on Y. Variants of this type of test have also been mentioned by others, for instance Athey et al. [2020] Lemma 3] and Dahabreh et al. [2020). We demonstrate the limitations of using this approach in our experiments, and provide a more in- depth explanation using graph-based arguments in Appendix C. Our contribution is a more general non-parametric test that works even if E is an invalid instrument that influence any of the other variables.
Detecting hidden confounding in multi-environment data
Our goal is to detect the presence of hidden confounding between treatment T and outcome Y after having adjusted for some observed confounders X. Graphically, this corresponds to detecting the existence of both edges U + T and U - Y in the causal DAG G. In this section, we demonstrate testable conditional independencies between the observed variables that are only violated when both those edges exist hence providing testable implications for hidden confounding.
5

---

While we do not assume to know the complete causal DAG & between our variables, we put two restrictions on (i) that Y not an ancestor of T and (ii) that X is a confounder both T and Y in contrast to, for instance, being a mediator or only a cause to either one of them. These restrictions are relatively weak as (i) holds in all practical causal inference settings as a treatment T happens before outcome Y in time and (ii) can sometimes be verified by checking that both T and Y depend on X. Under this setting, we prove the following.
Theorem 1. Let T(k) = -(T(A) "..,...III!! be the vector of all observed treatments in envi- ronments E(k); define Y(k), X and U similarly. We consider the data distribution ((T()), ( x()), , with No 2 under assumption
2 and 4 Furthermore, assume an underlying causal DAG G where Y is not an ancestor of and that X is a known common cause to T and Y. Then, for any k = I..... K, there exists hidden confounding between T and Y in 9 if and only if
T L
T L,Y ((()) ((()) x(() = M..j=l...NN:445.
(3)
Proof sketch. To prove the statement, look d-separations in the extended causal graphical model 9" and show that (3) only is true for corresponding graphs 9 where the unobserved U is a confounder
between T and Y. Figure illlustrates how open paths may exist between pairs of observations (i.
going through e θ (k)
x or ( by unrolling the augmented graph g" These paths are open
Je)
because of Assumption This technique resembles the twin network method used for counterfactual
inference [Balke and Pearl [994] but the results we obtain from using this approach are distinctly
different. The complete proof can be found in the Appendix.
The variables Ty(h) and y:(A) the treatment and outcome of different observations in the same environment. Intuitively, the theorem states that after having adjusted for (T!(A, x,(*,, ((()), ).
would expect under the ICM principle that T not provide any information about how y,(k) behaves. Thus, if it still does, then this can only be due to unobserved confounding. Testing this independence hence provides us with a testable implication in our observed data distribution on whether the unobserved U is a confounder or not.
Two-variable case without observed confounders We can drop the observed confounder X in Theorem and, interestingly, in that case, obtain even stronger results for detecting the presence of a hidden confounder. This setting is interesting
the two-variable case is notoriously difficult in causal discovery [Reichenbach] [956] [Peters et al.) (017)]. Unlike in the more general setting, we no longer need to know the direction of the causal re relationship between T and Y.
Theorem 2. Let (k) (IT()) (.)) be the vector of all observed treatments in environments F define (k) and Uj(k) similarly. We consider the data distribution P(T(k), Y(k), U](k) without any observed confounders and N₃ 2 2 under assumption 22 and Then, for any k 1,... K. there exists hidden confounding between T and Y in G if and only if
(Kk))
(i) Tpt))
d p Y: Tr( and (ii) T/ L Y2(k)
((k) Vi.j = 1, No :i
Guo et al. [0022] studied a similar setting to Theorem and demonstrated how to decide the causal direction between 7 and Y in this case when there is no latent variable. Our results extend theirs as we now also show how to exclude the possibility of latent common cause in this setting. The proof is similar to that of Theorem [ but the conditional independencies are different. Firstly, we have
TT(+) 1 p y-(k) (.(K) which is the conditional independence in Theorem without conditioning on
xx(a) and (K)) Secondly, we have (.(K)) y y,(") y((A) This one is necessary as we no longer assume anything about the ancestral relationship between treatment and outcome. If we had assumed (i) the theorem necessary. Similarly, condition (ii) is only necessary when Y could not be a descendant of T'
that T could not be a descendant of Y, we can show that only condition

---

4.1 Influence of the assumptions
Our theory shows how to test for hidden confounding, but it now relies other untestable assump- tions: namely non non-degenerate independent causal mechanisms and the faithfulness & causal Markov property. Due this, we investigate the necessity of these assumptions and identify various failure cases when they are violated. On a more positive note, we also demonstrate that the assumption of non-degenerate mechanisms can be weakened. We present here the main conclusions regarding violations on two of the assumptions while more elaborate explanations can be found in Appendix [ together with a demonstration of how our procedure can fail due to faithfulness violations as well as a discussion on assumptions about positivity and selection bias.
Violation of Assumption 3 dependent causal mechanisms What happens if any of the pair-wise independencies between θr; ry O or Oy are violated? To investigate this, we go through the same procedure for proving Theorem where we allow any of these mechanisms to be dependent.
We find that T7,4) IL p y-(k) TT!h, x (x(() can be violated even when there no confounding in all but one case with dependent mechanisms, meaning that it no longer works for detecting hidden confounding. The only case where our theory still works when Ox IL p Qu i.e. the mechanisms
of the observed and unobserved confounders are allowed to co-vary across environments.
Violation of Assumption 4 degencrate causal mechanisms What happens if one more of the distributions P(Or), P(@r), x(0x) and P(Qy) are degenerate, meaning that some mechanisms are fixed across all environments? In the most extreme case, if all mechanisms are fixed then the distribution PE would be identical in each environment. We investigate these scenarios by first adding check Or, Oy, O and/or θy to the conditioning set of the independence in Theorem I Then, whether this independence still is violated in the presence of hidden confounding using the same procedure used for proving the theorem. We find that the theorem fails only when we condition on both Or and Ov. In other words, it is only strictly necessary for our theory that changes in Peo (U Pa(U)) occur between environments.
Per (T) Pa(TT)
Remark 1. We may now identify more conservative interpretation of our proposed procedure. First, one can verify the assumption of non-degenerate causal mechanisms by checking from data whether PE(T X) varies across environments; if it does, then that is likely because OT and/or through potential downstream effects- -y are non non-degenerate. Next, we would run our proposed procedure. Now if the null is rejected then we can be conservative by concluding that this is either because we have hidden confounding and/or dependent mechanisms. But in the case of no rejection, can only be interpreted as having no hidden confounders present. This is because having dependent causal mechanisms (violation of assumption 3 can only cause false positives.
4.2 Testing the independence
Here, explain how test the conditional independence T 1 p y-(k) Theorem [siig multi-environment data; the full procedure is summarized in Algorithm where we have defined 12-11 = (t 1 and similarly for y , 2 and 2:2 a.
T .x .x (k) from
To test 17.M)) I (,()) 11((), xx(1) ((,,). we need simulate sampling from the joint distribution P(T!()) (k) y,(k), xx(1) .T, ((((), xx())). Note here that we do not condition on EAk), The idea is as follows: we select two different observations i and from all environments such that we get a vector
of observed treatments t, (t(2), (!2) {(K)) outcomes y; = (u(²) y/ (2) y/ (k)): and so on for z1, ty and zy. Then, we can use any suitable method for conditional independence testing with
to Vis .i, ty and Iz. Note that the choice of observations within each environment is arbitrary long as we do not pick the same observation for i and j, this is a consequence of observations being i.i.d. within each environment.
Increasing power of test with Fisher's method In essence, we perform a conditional independence test where the "sample size" of the test is the number of available environments. Thus, for small
7

---

Algorithm 1: Algorithm for statistically testing the presence of hidden confounding
Input:
Datasets D2 {(t, 3i Iz ) from environments k 1,..., .,.,..,, K; significance level a; minimum number of environments required for hypothesis test Kmin
(k)
Lmax <
ceiling(maxx /2/
The maximum number of possible hypothesis tests
for i ....., Limxx do
T2l + {k E [K] N2 - 20} size((T2) < K min then
Retrieve all environments with at least 2i samples
Limax f-
break
Stop for-loop we run out of a sufficient number of environments
Pi cond _ndei_ndepp__ test (t2r-1 IL y2
22 22
22
Get p-value from independence test
8 2 f- 2 "i log(p;)
Aggregate p-values with Fisher's method Compute global p-value
f- 1 cdf x3ema (2)
return p S
number of environments, our test might have low power (probability of detecting hidden confounding when present). To alleviate this issue. we recognize that we can perform this test multiple times if we have many samples N& per environment. Then, we select new observations from every environment for each hypothesis test until all observations have been used up. It is important to note that we only select from environments where there still are observations that have not yet been used for the hypothesis testing. Since each hypothesis test is independent and has the same null, we can aggregate the p-values from all tests using Fisher's method to obtain global hypothesis test [Fisher, 1925]. As we show our experiments, using Fisher's method drastically improves the power of our method, thus reducing the number of environments needed to detect the presence of hidden confounding. Having a different number of samples per environment N₂ also necessitates specifying the hyperparameter Kmin- which determines the minimum observations required in each hypothesis test. This parameter should be chosen to ensure that the used independent testing method works properly if it is provided with at least Kmin samples.
5 Experiments
To evaluate and investigate the theory for testing hidden confounding in multi-environment data, we perform a series of simulation studies with synthetic data in addition to experiments with semi- synthetic data generated using the Twins dataset [Almond et al. 2005 Louizos et al. 2017 As data where the ground-truth
want to evaluate causal graph is known. Unless otherwise stated, each experiment repeated 50 times where we use significance level 0.05. Depending the variable types in the experiment, state what suitable conditional independence testing method is used by our algorithm.
method's ability detect confounding,
5.1 Synthetic data
For the synthetic data experiments, we generate data as follows: we have the confounder U(()) N (k) 1); treatment 27((k) Ber(Sigma(;" + (l())); and outcome y:( Ber(Sigm(XU + r + e(4))) Note Sigm(x) = 1/(1 +e(1 is the logistic function and @(k) Normal ) for V E (T,Y,U) Unless otherwise stated, we use de, = dey der 1. We control the strength of confounding by varying X, where À = 0 corresponds to no confounding.
"Code available hug.ttibww..ckkkkaaaaaaeeeeeeeea

---

10
0.8
Parameter
Detectie
06
T
04
4.
K-200
0.-
1000
1500
2000
600
40
60
100
0.10
Number of enstrunmentss
Number of samples per envirarment
(a) Varying
(b)
(c) X = (no confounding)
Figure 3: Synthetic data (a): Detecting confounding with N& across range of confounder ef-
feet sizes and numbers of environments K. (500 repetitions) (b): Simulations with fixed confounding strength À 5 for N& with small number of environments K. (c): Comparing the proposed procedure and an alternative testing procedure by varying the standard deviation of By in the absence of confounding. The black dashed line corresponds the desired type error control 0.05. The shaded area shows the standard error from 50 repetitions.
1.00
1.00
200
0.75
Deterhonnak
6.75
n5-50
150
100
0.00
0.00
Average pmmiteed bias
Number cunfounders adjucced
Number confounders adjasted
p
p
Figure 4: Twins dataset (a): Effect of bias from omitting confounders on the test statistic of our hypothesis test. (b), (c): Performance when adjusting for observed confounders with either a total of 3 or 5 confounders in the data. The different curves correspond to combining numbers of hypothesis tests. The black dashed line corresponds to the rejection threshold desired type error a 0.05 in all figures, and the error bars / shaded area shows standard deviation (figure or standard error (figures and c) from 50 repetitions.
Larger confounder effect sizes increase the probability of detection We investigate how the effect size of the confounding variable influences our proposed testing procedure. We vary between 0 (no confounding) and 10 while also varying the number of environments. We perform this experiment with Ni = 2 for & and use the G-test for conditional independence testing [McDonald 2014). The results are shown in Figure 3al We note two things: the probability of detection grows for larger confounder effect size and also grows when the number of environments increased.
The growth rate in detection depends on the number of environments We investigate the probability of detecting confounding when varying both the number of environments and the number of samples per environment for fixed confounding strength À = 5. We use a permutation- -based [T'smarddinos and Borboudakis 2010
method for the conditional independence to rely only on asymptotic validity (such as in the G-test) due to the limited number of environments. The results show that the performance of the testing procedure is highly dependent the number of environments K, see 3b) The probability of detection grows we increase the number of samples. Noticeably, the rate of growth increases with the number of environments K.
do not want
Robustness environmental changes We compare our proposed procedure the alternative approach of testing Y L E T to detect hidden confounding, the latter being valid when E is an instrumental variable [Mooij et al. 2020). Here we test the sensitivity to violating one of its conditions, namely that (Y T) is nxxed under the null. We vary dey between 0 and when there is no confounding by setting
500, and we use the G-test for conditional independence testing [McDonald] 2014). As shown in Figure 3d the probability of false detection using Y p E. increases when dey starts to increase. Meanwhile, the false detection rate (type error) remains bounded by 0.05 for our procedure as desired. In Appendix F
with N 100 and

---

also include the same comparison when confounding is present to confirm that our method is able to detect confounding in this case.
5.2 Twins dataset
We use data from twin births in the USA between 1989-1991 [Almond et al., 2005 Louizos et al. 2017] to construct an observational dataset with continuous treatment/outcome and non-linear relationships. Here the environments are different states, and notable element of our dataset that all variations between environments stem solely from the real-world distribution shifts of the covariates between birth states. The strength of confounding is controlled by parameter X, where À = 0 corresponds to no confounding. The full procedure for data generation is described in use the Kernel Conditional Independence Test (Zhang (et al. 2012 in our algorithm due to having continuous variables and, unless otherwise stated, combine 50 hypothesis tests using Fisher's method.
Appendix For the following experiments,
Detection rate increases with bias from unobserved confounding We perform an experiment having p unobserved confounders, where we vary confounding strength À between 0 and 5. We compute the bias from omitting the unobserved confounders when estimating the average treatment effect of T on Y in each environment. We then compare the average bias to the test statistic computed by our algorithm averaged over multiple iterations. As observed in Figure 4a the test statistic increases together with the bias. The black dashed line in the figure represents the rejection threshold at 0.05, hence can that for sufficient bias the method will detect it.
Adjusting for observed confounders In the last experiments, we attempt to detect hidden confound- ing while also adjusting for observed confounders. We go from observing none to all confounders while having confounding strength of À 5. We do this for the case with either a total of 3 or p 5 confounders, shown in Figure 46 and 4c respectively. In addition, we investigate the influence of combining multiple hypothesis tests (n. denotes the number of tests) using Fisher's ' method. We observe first that adjusting for more confounders leads to decrease detection rate, and that our desired type 1 eтror of a = 0.05 is controlled when we have adjusted for all confounders. Secondly, the performance deteriorates when the total number of confounders increases, as indicated by the detection rate, which is lower when adjusting for 4 confounders when 5 than adjusting for 2 confounders when p This likely because the conditional independence test loses power as the conditioning set becomes larger (Zhang et al. 2012). Thirdly, see that the combination of multiple hypothesis tests using Fisher's 's method does improve the power of our algorithm. We did, however, not see any significant benefit in combining more than 50 hypothesis tests in these experiments.
6 Discussion
In this work, we studied a setting where observational data has been collected from different het- crogencous environments in which the same treatment T, outcome Y, and covariates X have been observed. We showed that assuming independent causal mechanisms, there exist testable conditional independencies that are violated in the presence of hidden confounders, for which we also proposed a statistical procedure to test these independencies from observed data. In many cases, with sufficient number of environments, we show that we are able to detect confounding when it is present. While our main goal to derive testable implications of hidden confounding, open questions remain how to improve sample efficiency and tackle loss of power when adjusting for many observed confounders. Addressing these can lead to better tools for researchers to validate their causal assumptions and move towards making safer causal inferences.
Socictal impact Causal inference has a big influence on real-world decision-making as lies at the core of many sciences, ranging from medicine to public policy. While our work has the potential to improve the soundness and safety of causal inference methodology, this research still in its infancy and we caution careful use of this work, particularly in high-stakes settings.
10

---

Acknowledgements
The authors thank Marco Loog, Stephan Bongers, Alexander Mey, and Frans Olichoek, and the members of the Pattern Recognition lab for their invaluable feedback on earlier versions of this manuscript. We also thank our anonymous reviewers for their helpful comments and input.
References
Douglas Almond, Kenneth Y Chay, and David S Lee. The costs of low birth weight. The Quarterly Journal of Economics, 120(3):0031-108; 2005.
Joshua Angrist, Guido W Imbens, and Donald Rubin. Identification of causal effects using instrumental variables. Journal of the American statistical Association, 91(434):444-555 1996.
Susan Athey, Raj Chetty, and Guido Imbens. Combining Experimental and Observational Data to Estimate Treatment Effects on Long Term Outcomes. arXiv preprint arXiv:2006.09676, 2020.
Kunihiro Baba, Ritei Shibata, and Masaaki Sibuya. Partial correlation and conditional correlation measures of conditional independence. Australian & New Zealand Journal of Statistics, 46(4): 657-664, 2004.
Alexander Balke and Judea Pearl. Counterfactual probabilities: Computational methods, bounds and applications. Uncertainty Proceedings 1994, pages 46-54. Elsevier, 1994.
Elias Bareinboim and Judea Pearl. Causal inference and the data-fusion problem. Proceedings of the National Academy of Sciences, 113(27):345--352, 2016.
Carlos Cinelli, Daniel Kumor, Bryant Chen, Judea Pearl, and Elias Bareinboim. Sensitivity analysis of linear structural causal models. In International conference on machine learning, pages 1252-1261. PMLR, 2019.
Issa J Dahabreh, James M Robins, and Miguel A Hernán. Benchmarking observational methods by comparing randomized trials and their emulations. Epidemiology (Cambridge, Mass.), 31(5): 614-619, 2020.
Bruno de Finetti. La prévision lois logiques, sources subjectives. Annales de l'institut Henri Poincaré, 7(1):1-68, 1937.
Emanuele Di Angelantonio, Shilpa N Bhupathiraju, David Wormser, Pei Gao, Stephen Kaptoge, Amy Berrington De Gonzalez, Benjamin J Caims, Rachel Huxley, Chandra L Jackson, Grace Joshy, al. Body-mass index and all-cause mortality: artividual-partiiipaaa-ee 239 prospective studies in four continents. The Lancet, 388(10046):776-78, 2016.
meta-analysis of
Ronald A Fisher. Statistical Methods for Research Workers. Oliver and Boyd, 1925.
Andrew Gelman, John B Carlin, Hal S Stern, David B Dunson, Aki Vehtari, and Donald B Rubin. Bayesian data analysis. CRC press, 2013.
AmirEmad Ghassami, Negar Kiyavash, Biwei Huang, and Kun Zhang. Multi-domain causal structure learning in linear systems. Advances in neural information processing systems, 31, 2018.
Clark Glymour, Kun Zhang, and Peter Spirtes. Review of causal discovery methods based on graphical models. Frontiers Genetics, 10, 2019.
Harvey Goldstein, William Browne, and Jon Rasbash. Multilevel modelling of medical data. Statistics in medicine, 21221):391-33115 2002.
Siyuan Guo, Viktor T6th, Bernhard Schölkopf, and Ferenc Huszár. Causal de Finetti: On the Iden- tification of Invariant Causal Structure in Exchangeable Data. arXiv preprint arXiv:2203.1577 2022.

---

Tobias Hatt, Jeroen Berrevoets, Alicia Curth, Stefan Feuerriegel, and Mihaela van der Schaar. Combining Observational and Randomized Data for Estimating Heterogeneous Treatment Effects. arXiv preprint arXiv:2202.11 2891, 2022.
Miguel A Hernan and James M Robins. Causal Inference. Chapman & Hall/CRC Monographs Statistics & Applied Probab. CRC Press, 2023.
Biwei Huang, Kun Zhang, Mingming Gong, and Clark Glymour. Causal discovery from multiple data sets with non-identical variable sets. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 10153-10161, 2020.
Maximilian llse, Patrick Forré, Max Welling, and Joris M. Mooij. Combining Interventional and Observational Data Using Causal Reductions. arXiv preprint arXiv:2103.04786 2022.
Guido Imbens, Nathan Kallus, Xiaojie Mao, and Yuhao Wang. Long-term Causal Inference Under Persistent Confounding via Data Combination. arXiv preprint arXiv:2202.07234, 2022.
Guido Imbens and Donald B Rubin. Causal inference statistics, social, and biomedical sciences. Cambridge University Press, 2015.
Dominík Janzing and Bernhard Schölkopf. Causal inference using the algorithmic markov condition. IEEE Trans, Inf. Theor., 56(10):5168-55) 2010.
Dominik Janzing and Bernhard Schölkopf. Detecting confounding in multivariate linear models via spectral analysis. Journal Causal Inference, 6(1):20170013, 2018.
Nathan Kallus, Aahlad Manas Puli, and Uri Shalit. Removing Hidden Confounding by Experimental Grounding. In Advances in Neural Information Processing Systems, volume 31, 2018.
Manabu Kuroki and Judea Pearl. Measurement bias and effect restoration in causal inference. Biometrika, 101(2):423-377, 2014.
Walter Leite, Francisco Jimenez, Yasemin Kaya, Laura M Stapleton, Jann W MacInnes, and Robert Sandbach. An evaluation of weighting methods based on propensity scores to reduce selection bias in multilevel observational studies. Multivariate behavioral research, 50(3):265-284, 2015.
Christos Louizos, Uri Shalit, Joris Mooij, David Sontag, Richard Zemel, and Max Welling. Causal effect inference with deep latent-variable models. Advances in neural information processing systems, 30, 2017.
Xun Lu and Halbert White. Robustness checks and robustness tests in applied economics. Journal of econometrics, 178:194-206, 2014.
Edwin P Martens, Wiebe R Pestman, Anthonius de Boer, Svetlana V Belitser, and Olaf H Klungel. Instrumental variables: application and limitations. Epidemiology, pages 260-267, 2006.
John H. McDonald. Handbook of Biological Statistics. Sparky House Publishing, 3rd edition, 2014.
Christopher Meek. Strong completeness and faithfulness in bayesian networks. In Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence, page 411-418, 1995.
Wang Miao, Zhi Geng, and Eric J Tchetgen Tchetgen. Identifying causal effects with proxy variables of an unmeasured confounder. Biometrika, 105(4))987-993, 2018.
Joris M. Mooij. Sara Magliacane, and Tom Claassen. Joint causal inference from multiple contexts. Journal of Machine Learning Research, 21099))1-108, 2020.
Monika Mueller, Maddalena D' Addario, Matthias Egger, Myriam Cevallos, Olaf Dekkers, Catrina Mugglin, and Pippa Scott. Methods systematically review and meta-analyse observational studies: a systematic scoping review of recommendations, BMC medical research methodology, 18(1):1-18, 2018.
Emily Oster. Unobservable selection and coefficient stability: Theory and evidence. Journal of Business & Economic Statistics, 3772):187-204, 2019.
12

---

Judea Pearl. Probabilistic reasoning in intelligent systems: networks of plausible inference. Morgan kaufmann, 1988.
Judea Pearl. Causality. Cambridge University Press, 2nd edition, 2009,
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 7865):947-1012 2016.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of Causal Inference: Foundations and Learning Algorithms, MIT Press, 1st edition, 2017.
Hans Reichenbach. The Direction of Time. Dover Publications, 1956.
Richard Riley, PC Lambert, and Ghada Abo-Zaid. Meta-analysis of individual participant data: rationale, conduct, and reporting. British Medical Journal (International edition), 340:c221, 2010.
Paul R Rosenbaum and Donald B Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55, 1983a.
Paul. R. Rosenbaum and Donald B. Rubin. Assessing sensitivity to an unobserved binary covariate in observational study with binary outcome. Journal of the Royal Statistical Society. Series B (Methodological), 45(2):212-218, 1983b.
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pages 459-466, 2012.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke. Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of IEEE, 109(5) 2-432, 2021.
Zehao Su and Leonard Henckel. A robustness test for estimating total effects with covariate adjust- ment. In The 38th Conference on Uncertainty in Artificial Intelligence, 2022.
Johannes Textor, Benito van der Zander, Mark S Gilthorpe, Maciej Liskiewicz, and George TH Ellison. Robust causal inference using directed acyclic graphs: the R package 'dagitty'. International Journal of Epidemiology, 4586):188711844 2017.
Joannis Tsamardinos and Giorgos Borboudakis. Permutation testing improves bayesian network learning. José Luis Balcázar, Francesco Bonchi, Aristides Gionis, and Michèle Sehag, editors, Machine Learning and Knowledge Discovery in Databases, pages 322-337, Berlin, Heidelberg, 2010. Springer Berlin Heidelberg.
Tyler VanderWeele and Peng Ding. Sensitivity analysis in observational research: introducing the e-value. Annals of internal medicine, 167(4):268 274, 2017.
Julius Kügelgen, Alexander Mey, Marco Loog, and Bernhard Schölkopf. Semi-supervised learning, causality, and the conditional cluster assumption. In Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI), volume 124, pages 1-10. PMLR, 2020.
Kun Zhang, Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Kased conditional independence test and application in causal discovery. arXiv preprint 1eXiv:1202.3777 2012.
Kun Zhang, Biwei Huang, Jiji Zhang, Clark Glymour, and Bernhard Schölkopf. Causal dis- data: Skeleton estimation and orientation determina-
covery from nonstationarrhheeerggenooo tion. In Proceedings of the Twenty-Sixe International Joint Conference on Artificial Intelligence, UJCAI-17, pages 1347-1353, 2017.
13

---

Appendix
The appendix contains the following sections.
A. Future Work: This section contains a discussion on open questions that we have identified in this work.
B. Proofs: This section provides the full proof of Theorem and
C. Alternative Test for Detecting Hidden Confounding in Two-variable Case: We discuss the alternative approach of testing Y E T detect hidden confounding in the two-variable case without any observed confounder. We demonstrate conditions when and when not would work.
D. Further Analysis the Influence of the Assumptions: We elaborate on the examples of violating the assumptions behind our theory, discussed in Section 4.1 First, we provide concrete failure cases when we have dependent causal mechanisms. Secondly, we demonstrate how we can relax the assumptions on non-degenerate mechanisms. Thirdly, we also give an example of faithfulness violations in linear- Gaussian structural causal models. Finally, we discuss how both positivity violations and the presence of selection bias could influence our procedure.
E. Generation of Twins Semi-synthetic Dataset: We explain how generate the semi- synthetic observational data using the Twins dataset.
F. Additional Experiments: We perform additional simulation studies, extending the results in the main paper with results with both continuous and binary data.
14

---

A Future Work
We have identified a set of open questions deriving from our work. Firstly, our theory applies to the common setting in causal inference with treatment T, outcome Y, and a possibly high-dimensional confounder X, in which we want to detect the presence of additional hidden confounding U (that also can be high-dimensional), While this is arguably the most typical setting in causal inference, it is of interest to consider other scenarios with more variables and interactions between them. We conjecture that other testable implications exist for confounding in these settings that could be found with similar arguments as we use. A particularly interesting setting is when we observe proxy to hidden confounder, which can be used for adjustment instead [Kuroki and Pearl 2014 Miao et al. 2018]. In this case, is longer straightforward say whether there could be a hidden confounder that is unrelated to the proxy. We also believe that our techniques might be applicable to scenarios with instrumental variables (IV) [Angrist et al., 1996, Martens et al.) 2006), to test whether there exists any confounding between the IV and treatment which is requirement for valid IV estimation.
Secondly, our theorem fundamentally relies on a set of untestable assumptions: independent & non-degenerate causal mechanisms and the faithfulness & causal Markov property. Although we investigated various violations. these results raised new questions. In particular, the effect of faith- fulness violations, perhaps surprisingly, had a large influence on our procedure. Therefore, it is important to understand whether similar observations can be made in more realistic settings.
Thirdly, an interesting direction for future research would be to investigate how our approach can be used to estimate confounding strength, to be used in well-studied approaches sensitivity analysis [Rosenbaum and Rubin 1983b Cinelli et al. 2019
Lastly, while our main goal was to derive testable implications of hidden confounding, there are opportunities to improve the way we test these from data. We observed that our proposed procedure sometimes requires large number of environments. While it is unclear whether this is property of the theory or the lack of efficiency in the test procedure we used, we note that combining multiple hypothesis tests using Fisher's method helped with performance. A possible research direction could be to investigate how to refine this approach further. Further, we observed performance deteriorating we adjusted for more observed confounders, likely due to the curse of dimensionality [Zhang et al.) techniques from
2012]. A promising solution here could be to use popular dimensionnllit-rrduccii causal inference such as the propensity score [Rosenbaum and Rubin 1983a].
B Proofs
T(P) be the vector of all observed treatments in environments ((k). Define (k), (1k), and U(k) similarly.
In this section, we present the proof for Theorem and 2 Let (R) (T((A)
Theorem We consider the data distribution P(T(k), (k), x(k), U(K)) with N₂ 2 under
assumption 2 3 and A
Furthermore, assume an underlying causal DAG G where Y is not
ancestor of T, and that X a known common cause to T and Y. Then, for any I,..., K....,K. K, there exists hidden confounding between T and Y in if and only if
I Y, (k)
(k) .xx(x) Vi,j L..... No j
(5)
Proof. We constrain ourselves to DAGs G with variables (T. Y, XU) where X is a known common cause both T and Y. and Y not an ancestor of T in G. Under the assumption of non-degenerate, independent causal mechanisms (Assumption and D we can introduce the mechanisms Oy for each variable V E (T, X,X,UU U}. Further, we can augment G with a hierarchical structure [Gelman
et al. 2013 Chapter 5], wherein we first sample the mechanisms i.i.d. @(²) P(Ov) for k =
1, 1...., and V E (T, Y, X, U} and then, for each environment k. obtain (T,(*) y((k) (
x((k) U,!0)
by repeatedly sampling N₂ times conditioned on the mechanisms. We denote this augmented graph
with the hierachical structure as g", where edges between (T((K) y-(k) x((k) U(())) are the same as UJ,
for (T, X, U) and ((²) (()) where V UX,,,, for all i and k. An example of
15

---

T:(0)
Y!(A)
X!k)
(
+ (k)
+ (k)
(()
+ (A)
U
Y
X (k)
T
((k)
Environments = ..2,...... K
Figure 5: Example of unrolling the augmented causal DAG G* for some pair of observations (i. The samples are generally not independent due to the shared mechanisms (Or, θx Uu, θy), unless we condition them. Confounding is present, and the red edges mark open paths such that TT(K) 1 y.!*) ( .x, .x(x), xx(²) in this graph. Note that these paths go through the outgoing edges
from (=(1) (ha)) and y((h) (and similarly for j), and that the paths are closed if either of the edges is removed.
such augmented graph G" is shown in Figure 5 Notably, this augmentation can be done for all k because we assume that environments share the same causal graph 9 (Asssumption 2h.
Now, given the constraints that we have defined, we consider every combination of the edges between (T', Y, X, U) that are DAGs. In total, there 40 different DAGs that encompass all these combinations edges. We say that U is a confounder one of these DAGs if both the edges - T and U + Y exist. For each of these graphs G, shall investigate the d d-separations in its augmented version g- Notably, due to the assumption of non-degenerate mechanisms (Assumption 4 we allow open paths in G" that go through e (kk) (k)
or (%)) This means that two different observations (T!( (((k) x((A), (T(h) and (T(A), (-)k) (R) UT(*)) can be dependent when i *
for i,j = N,. These paths are best illustrated by unrolling the augmented graph G as in the example in Figure 5] However, such dependencies can only happen if we do not condition on the mechanisms (that is, the environment) as we know that the observations are sampled i.i.d. within each environment. This demonstrates the need for multiple environments, the randomness from sampling ( ( (k) e(()) allows us to treat them as ordinary random variables. To capture this randomness, we need observe multiple environments though. Note that we also need N₂ for * j to hold.
We will pay attention to particular d-separation in G*, namely
T (k)
1 a
Y
(k)
T (k)
x X
(k)
(k)
16

---

We automatically iterate over our list of DAGs using the dagitty package in R Textor et al. 2017] and check whether this d-separation holds; the results are displayed in Table Il We note that the shaded rows in the table are the cases where U is confounder, and these are the only cases where is violated in g". In other words, checking this d-separation is sufficient to determine whether Ü is confounder in G. Assuming the faithfulness and causal Markov property (Assumption T. we have that:
(33 IL d y,(k) T X!!*, X
T, L y!!h) (TT(A) x x(:
T/ L PY (k) x(
Consequently, follows that
(k) for i - Us confounder to T and Y. T// 1 p x(()) 17/h) xx!)) _
This result holds for any & since we assume that all environments share the same causal DAG (Assumption 2
Remark 2. ((4) can be removed from the conditioning set in the independence of Theorem 1 Theen we assume that the observed and unobserved confounders are independent of each other. In practice, however, we most likely would not like to make this assumption which is why we recommend to condition on both x!!) and X,(h.
Theorem 2. We consider the data distribution ((T(), Y()), U(k)) without any observed con- founders and N& 2 under assumption [2 confounding between T and Y in 9 if and only if
3 and
7
for any I..... ,, there exists hidden
(i) TT(h) I PpY. Y!!
T((h) and (ti) T PY
y Vi.j = 1,.... No Ni:
(6)
Proof. Using the same arguments as in the proof of Theorem we can show that T. 1
rT(*) and (r(a)) p y y,!h) exclusively holds for those where there exists no common cause
between T and when we only have the variables (T, Y, U), excluding any observed confounder X
The corresponding table to check this is demonstrated in Table
C Alternative Test for Detecting Hidden Confounding in Two-variable Case
In this section, we explain when (and when not) testing Y 1 E T is valid strategy for detecting hidden confounding between T' and Y. We require that Y p E ST, when there is no confounding meaning that there can not exist any direct causal relationships E - Y or E - U; this does not hold then testing Y 1 P E T would not be informative. This demonstrated with the dashed edges in Figure ba Mcanwhile, in Figure 6b we display the case where violation of Y r E T correctly detects hidden confounding being present. The conclusion is that this alternative test, which is easy to test, only works if the changes between environments happen for the conditional distribution PE Pa(T)), as only T may be directly caused by F. if this test shall work. This corresponds to E being an instrumental variable [Angrist et al., 1996]. meaning that T L p E, p E, and the environment E can not have any direct effect on the outcome Y under the null (that is no confounding).
D Further Analysis on the Influence of the Assumptions
In this section, we present more elaboration on the examples of violating the assumptions behind our theory, as discussed in Section 4.] First, we provide a counter-example that shows how Theorem [ fails when we have dependent causal mechanisms. Secondly, we demonstrate how we got to our conclusions on having degenerate mechanisms. Thirdly, we provide insights into when faithfulness violations could occur in an example with inear-Gaussian structural causal model. Finally, we discuss the analogy to the positivity assumption for our theory as well as how our procedure could be influenced by the presence of selection bias.
17

---

E
U
U
Y
Figure 6: Examples of using Y 1. P E T to detect confounding. Gray corresponds to unobserved variables. (a) Violations of 1 E T despite there being no confounding between and Y. Removing the two dashed edges would make Y 1 p E' hold. (b) Case with correct violation of E T when unobserved confounding is present.
D.1 Dependent Mechanisms
We wish to demonstrate examples of DAGs where the main condition
(8) L
Theorem iss violated despite that there is no confounding when some of the causal mechanisms pairwise dependent. We able find examples of violations with all pairwise dependencies, except for Ox L Oy. Hence, we conjecture that a dependency between the mechanism of the observed and unobserved confounder does not influence the test proposed by our theory. Figures 7a Sa and 8b show found examples where violations occur between the mechanisms.
76
7
D.2
Degenerate Mechanisms
What happens if one more of the distributions P(OT), P(Oy), P(Ox), P(Ov) are constant across all environments?
investigate these scenarios by first adding θr, Oy, Ox and/or O₃ to the conditioning set in T.(A) IL y:(h) ! x xx(x)
xx( the testable implication for confounding. We then follow the same procedure as for proving Theorem [ and noting whether this conditional independence still discriminates between cases when U is and is not a confounder. Out of 15 possible cases with degenerate mechanisms, we identify that theorem fails every time we condition on both Or and 0v, as demonstrated in Table 3
D.3 Faithfulness Violation
What happens if the conditional independencies test do correspond the dependencies in the underlying causal graph? If that is the case, we would fail to detect dependencies implying the presence of confounders. Knowing when faithfulness violation occurs is not possible, we only reason about its plausibility. In this section, we first showcase an example where a faithfulness violation occurs in Gaaessinn structural causal models for particular configurations as well as when the confounder effect sizes that is and in become very large.
It was proved by Meek 1995 that all graphs with discrete and linear-Gaussen Gaussian data distributions fulfill faithfulness measure- theoretic measure-theoretic zero.
sense; distributions that violate faithfulness have measure
However, even if we restrict 7 and Y to be categorical we might not want to assume that U or the causal mechanisms follow a discrete distribution. The following example also demonstrates the practical issues stemming from faithfulness violations even when the data is jointly Gaussian.
18

---

Example 1. Let k = I....,K
K and = ,,...... Ng. Consider the structural causal model
U/40] = =6/ + EU.i
Etr ~ (0,
T((b) = -U,1h) (k) + o + ETi
Er, ~ Normal(O, 0Z,),
(7)
y (U!h) BT, + +oy +Ey
(k)
EY3 Normal(O, ,
where ( N0 ) and ev V for V (T.Y.U). Further, subscript for the noise variables indicates that they are sampled independently for each observation i. Then, despite the presence of confounding, T 1L p Y (T(h) for any + j when dee = _2 de,- In the saitee setting, it noticeably influences our ability to detect confounding when the distribution parameters come close to this equality, as illustrated in Figure
To create Figure 9 we generate data according to 7 with the following parameters fixed: & = = À 1, Jay σy du = 1 and dı 5.2 Meanwhile, vary dar and dey between 0 and 5. We test (k) L P Y (T(A) using the partial correlation [Baba et al. 2004). The experiment is repeated 1000 times with 1000 environments and significance level 0.05.
D.4 Derivation of Example
We consider the structural causal model in 7 Now, we want to prove that TTAa) 1 Y, (h for any + j when de, au de,. For ease of notation, we will drop the superscript (k), as the results hold for any k.
Crucially, we note that the partial correlation
PT,.Y.TR = PT PT.,T.PT.YS
(8)
Pi / p7..
is zero if and only T, 1 Y; T; when the data is jointly Gaussian Baba et al. 2004], which the case for (7) because p(T,, Y,, U.) P(Y, T., P(T. P(D,) where each factor is Gaussian density.
To check when the partial correlation is zero, we need to find out when
PT;Y PT,,T, 0.
Since PX,Y
Cov(X.Y) for some random variables X and Y. we can write this as Var())
Cov Y.)
Cov(T;, Ti)
Cov Y.)
PT;Y; PTy.TPPT.M
(9)
VVr((T) Var(())
Var((T) Var(T:) v Var(T:)
Cor(T;) Yi) To
(10)
(Tr(TT))
where we used the fact that Var(T.) = Var(T)) for any samples and j.
First, we need to determine all the (co)variances, for which we need to know
T: = 70u + 7EU, + Or + ET,
Y. 10u XEU, By + ByEu, + 30, BeT, + Oy + EY,
Note that E[T] [Y) = 0. Consequently, write out the covariances, for any i, j. follows:
19

---

Cov(T,, V.) B[T (7) +
Cov(T,, = E[T,T] = -
(Tv, Yi) = E[T.Y;] = (>)
Var(T;) = B[T;T; - - -
Var(Y;) B[УY]] =2(2² + + + +
B + 3 B + + BlsE]
Now, we look the numerator in the partial correlation zero:
10 and we want to know when it could be since that makes
Var (ar( (T3, Y)
CTv To)Co Y.)
[(()))
The solution is given by
OU der deu
(11)
where the square root of the second moments are equal the standard deviations. This is the same equality as demonstrated in the example.
D.5 Asymptotic Behavior of Partial Correlation
We also look at the partial correlation and ask what happens when the confounder effect sizes y or become very large. The numerator in (10) grows linearly with respect to both y and X, and the other variances can be rewritten as
Cov T3) " 0
Cov(2), Y.) (7^ -
Var(T,) =
Var(Y:) = 2( + (E))
where O(1) is constant with respect to y and X.
We rewrite the partial correlation B
PTy.Y. (Var(Ti)Cov (T, Yi) Cov(T) Cov Y:)) / Var (arr(T))VVar(X
Cov(Ty,T,) Var(?.]
V Cov(T, Y.)a Var())
Assuming that all second moments are non-zero and finite, it is possible to show that
7 for |y| >> 1, for | >>>>> 1
Pry,Y.- x
Hence, when either hy or | goes infinity have
PI,.Y. 17
PT PI..Y.T. 1N|+000
C' for some C [-1,1]
Note that C could be zero, for instance, when de or 0 although demonstrate with simulation studies in Appendix Fl case where C' is non-zero as well.
au
Interestingly, in this case, the bias from estimating the causal effect without adjusting for the confounder U
B| EY -
-
(12)
We note that when co the bias goes to zero, similar to the partial correlation. Meanwhile, the bias increases with which also is consistent with the asymptotic behavior of the partial correlation À oc.
20

---

D.6 Positivity violations in the sampling of mechanisms
In this section, we discuss another potential issue that can come up in our problem setting, namely that there could be positivity violations in the sampling of the mechanisms. Particularly unique to our setting is that the support for (Ør, yy, Ox Q²) must be the same for different environments, not then this would be direct violation of Assumption
We start by pointing out that there exist two categories of positivity violations: structural violations and random violations [Herman and Robins 2023]. Structural violations occur for instance when certain range of values of a variable never will be observed, and they may restrict the population for which we can draw causal conclusions. Meanwhile, random violations are due to having a finite number of samples. Random violations are perhaps also less problematic, as they can go away we collect more data.
In our setting with random positivity violations. For instance, structural violation could occur if we have only collected data from multiple hospitals in country A but there is also another country such that P(Or, ry xx Oy | country A) does not overlap with P(Or, Qy, Ox Ou country B). Then, we have a structural violation between environments in countries A and B. Meanwhile, a random positivity violation could come from observing enough environments in country A, assuming that we are only interested in studying data from that country.
data, we can make the same analogy with structural and
So based on what we have discussed so far, one could ask how to reason about positivity violations when trying detect hidden confounding. In this case, we think it is important to first ask ourselves: In what population are we trying to detect hidden confounding? If the answer is the population in country A, then we should not include data from country B, as there a structural positivity violation; and vice versa. For random violations, it is harder to anticipate what problems can come up, but these issues are mainly avoided by ensuring that we have sampled enough environments. This is what, in the end, will affect the quality of the conditional independence test in our method.
D.7 Selection bias
In our theory, we assume there is no selection bias. That is. for in- stance, that there are no unobserved colliders that we have conditioned on in our causal DAG. In principle, we can study such scenarios by adding colliders in our original problem setup to introduce different types of selection mechanisms. While we leave this topic for future work, we present here a quick illustration how selection bias can (or can not) hurt our procedure.
U
T
Y
Example Consider the graph G in Figure 10 where C. is a collider between treatment T and outcome Y. Now we consider the corresponding augmented DAG G*, add the causal mechanism Oc P(Oc) as parent to C, and check how (unknowingly) con- ditioning on C influences our ability to detect the presence of the unobserved U. The fact thut Oc is a random variable would reflect that we have different selection mechanisms different environments. We note this case that the (k)) c(()) will be violated regardless of whether U is present or not. In other words, selection bias would in this case lead to false positives of our procedure. This is because there will always be open path between T/()) and y. through Oc in G" But if. on the other hand, the selection mechunism remains fixed across all environments (meaning Oe is constant), this would close that path. That means we do not have this problem of false detection anymore, and the proposed approach would still work.
Figure 10: Grey variables are unobserved.
conditional independence TT(A) L ((k) (C(")
21

---

E
Generation of Twins Semi-synthetic Dataset
We use data from twin births in the USA between 1989-1991 Almond et al. 2005 Louizos et al. 2017] to construct an observational dataset with known causal structure. The dataset contains 46 covariates related to pregnancy, birth, and parents, out of which we select ten as potential confounders in our experiments. Many covariates are highly imbalanced and have low variance, some even on the border of being constant. Due to this, we wish to exclude those in data generation. The selection performed by first excluding any binary variables from the list of covariates, and then further removing the remaining ones that have empirical variance smaller than one.
In the end, the following covariates are used from the Twins dataset for our data generation: birth month, father's s age, mom's age, mom's 's education, mom's ' place of birth, number of prenatal visits, number of live births before twins, the total number of births before twins, period of gestation, state of birth occurrence, and state of registered residence. Note that all of these are reported as categorical//iscrete variables in the dataset.
Among the covariates, we use the state in which the birth took place as environment label, hence obtaining 51 different environments. We simulate a continuous treatment and outcome by randomly selecting p E [I...., 10] features . and a set of functions to generate the data as follows:
T Σ aafa(Xa +er and Y = Σ Baga(Xa + 8T + d=1
(13)
d=1
For each d, Qa is sampled from a uniform distribution Uni 8a is sampled from a uniform distribution (1)5( and fa and Ja are sampled from the set of functions (tanh(z), x. 2²) with equal probability. The treatment effect, 8, is also randomly sampled from a uniform distribution U1i 2). and we have noise variables ÊT ~ Normal(O, 1/4) and By ~ Normal 1/4). The features from the Twins dataset are also scaled
sd.od = 5 * (Xa
min(X.)),
(14)
where mean/max/min are taken over all observed values of the covariate X.. Note that the functions are fixed across all environments in this setup, and variations between environments only stem from the real-world distribution shifts of the covariates between birth states.
F Additional Experiments
We present additional simulation studies, mainly replicating the experiments synthetic data from Se. with continuous data. In addition, we further investigate the asymptotic behavior of the partial correlation from Appendix D.3 with both the binary and continuous data.
D.3
The continuous data is generated from a Gaussinn DAG described in (7). Unless otherwise = dy = 1, de, dey and de,
stated, we use 3 = 1, or the hidden confounder, we can adjust either 7 or A. We test T: p (k) (T( using the partial correlation (Baba et al. 2004] with N₂ samples per environment.
5. To vary the influence of
vary the number of environments and the confounder influence by setting A. A. In the main part of the paper, we only considered what happens when varying À with = Similarly, we do the same experiment with the binary data for comparison. The results are seen in Figure [1] Notably, the probability of detecting hidden confounding starts decreasing when goes above a certain threshold. This is consistent with our previous conclusions from Appendix D.3 where we noted that partial correlation is proportional to y for >> 1 while remaining constant for X >> 1. Hence, we would expect the partial correlation shrink as both y and À grow. Notably, the effect is more pronounced with the continuous data although it can also be seen with the binary data.
For the first experiment,
Secondly, we perform the experiment with continuous data where we only vary while fixing 7 The results are shown in Figure 12 we note that the probability of detection no longer decreases the
22

---

confounder effect size increases. The results with binary data are also shown again for comparison. Once again, this is predicted by the asymptotic behavior of the partial correlation.
Finally, we compare our statistical testing procedure to testing Y r E. T with continuous data in Figure 14] We run the experiment with 10000 environments, 100 samples per environment, and deg 10 to avoid the issues faithfulness violations which we have discussed before. Similar to the case with binary data, in Figure 13b we see that the probability of false detection when testing P E T grows as the standard deviation of Oy increases. Meanwhile, we not observe this problem in testing procedure. The case when confounding is present is shown in Figure 13a to confirm that our method is able to detect confounding, For completion, we also include a case where 10 (confounding is present) for the binary data experiment presented in the main part of the paper, this shown in Figure 14a
23

---

T
(A)
T
Y; (k)
X:1)
X (k)
(k)
+ (k)
θ (R)
θ (R)
θ (k) x
O (k)
(A)
X
(r(k)
x,(0
Ur
(y(k)
y(k)
k
y-(k)
j
Environments & 1, 1.2,.... K
Environments k 1, 1,2,.... .... K
(a) θr Y. Dr
(b) θr A. Ox
(!kk)
(R)
x-(a)
U.A (k)
+ (k)
+ (k)
)(R) (K)
X (k)
((()) (k)
Tr(R)
(k)
Environments 1, 2, .....
(c) Or 1 p Ou
Figure 7: Violations of (R)0)) 1 y2(k)
(T(()) x (R) with dependent mechanisms despite that
X is a valid adjustment set for estimating P(Y dT())). Open paths between conditioning on Tú, X, and X, are marked in red.
T7(0) and Y; (*) after

---

r ()
((()
1:(h)
(k)
(k)
x (k)
(k)
θ (k)
(k) T
(
O
+ (k)
r
X
x
X
x(k)
(*(k)
(()))
(>(k)
y(
Environments & = 1,2, K
Environments k = 1,2, K
(a) Oy L Ox
(b) Or 1 Oy
Figure 8: Violations of (T(()) 1 Y,(k) T xx xx( with dependent mechanisms despite that X is valid adjustment set for estimating P(Y do())). Open paths between TT( and y:(M) after conditioning on T;, X, and X, are marked in red.
5
1.0
Standard deviation dea
Probability of detection
8
3
2
0.4
1
0.2
0.0
2
3
5
Standard deviation de,
Figure 9: Faithfulness violation from Example 1
Varying de, and de, with or and du zu der (red line).
the probability for detecting confounding goes to zero as we get closer to de.
25

---

10
1.0
10
1.0
0.8
0.8
Parameters =
Detection rate
Detection rate
6-
0.6
Parameters
0.6
4-
0.4
4
0.4
2-
0.2
0.2
0.0
0
0.0
500
1000
1500
2000
200 400 600
800
1000
Number environments
Number environments
(a) Binary data
(b) Continuous data
Figure 11: Probability detecting hidden confounding for varying the number of environments K and confounder effect sizes where À ? are varied jointly. 500 repetitions.
10
1.0
10
1.0
0.8
8-
0.8
Detection rate
Detection rate
Parameter
Parameter
6-
0.6
6-
0.6
4-
0.4
Dete
4-
0.4
0.2
2
0.2
0.0
0.0
500
1000
1500
2000
200
400
600
800
1000
Number environments K
Number environments K
(a) Binary data
(b) Continuous data
Figure 12:
Probability of detecting hidden confounding for varying the number of environments K and confounder effect sizes where À is varied while is fixed. 500 repetitions
1.0
1.0
T II T, (ours) YLEIT
False detection rate
Detection rate
0.8
0.8
0.6
0.6
0.4
0.4
0.2-
0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
og
d7
(a) Confounding is present (A 10)
(b) No confounding is present (A
Figure 13: Comparison on continuous linear-Gaussian data between the proposed procedure and alternative testing procedure by varying the standard deviation of Qy in both the presence and absence of confounding. The black dashed line corresponds to the desired type error control 0.05. The shaded area shows the standard error from 50 repetitions.
26

---

Detection rate
0.8
detection
0.8
0.6
0.6
T,LYIT (ours)
YLEIT
0.4
0.4
0.2
0.2
0.0
0.0
0.00
0.05
0.10
0.15
0.20
0.25
0.00
0.05
0.10
0.15
0.20
0.25
03,
03
(a) Confounding is present (A 10)
(b) No confounding is present (A 0)
Figure 14: Comparison on binary data between the proposed procedure and an alternative testing procedure varying the standard deviation of Oy in both the presence and absence of confounding. The black dashed line corresponds to the desired type error control a = 0.05. The shaded area shows the standard error from 50 repetitions.
27

---

Table I: Conditional d-separations in combinations of DAGs with variables (T, Y, X, U). For the d-separation, (V) indicates that it holds and (X) otherwise. The shaded rows are the cases where U is a confounder to and Y in G.
T 1. Y |T ,x(00, ,x/x)
U is confounder
X 1
X -
T T-Y
U-T
0 U-Y
U-X
id
4 4
t
X
2
4
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
T
T
19
20
21
22
23
24
+
25
26
27
28
29
30
31
32
33
34
35 +
36 +
37
38 + 4
39 + 4
40 4
28

---

Table 2: Conditional d-separations in combinations of DAGs with variables (T, V,Y.U) (this excludes any observed confounder X). For the d-separation, (V) indicates that it holds and (X) otherwise. The shaded rows are the cases where U is a confounder to and Y g.
T Lي y,o !
Triy da y(()) (
U is confouneer
y ancestor T
id
X
X
3
7
X
7
5
6
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
29

---

Table 3: Conditional d-separations in combinations of DAGs with variables Y, (,, U)). We test T)ke 1 y T x x Z where Z contains the mechanisms that are degenerate, as noted column. For the d-separation, (v) indicates that it holds and (X) otherwise. The shaded rows are the cases where U is confounder T and in g.
Deg. or, 0r,0y, By 0x 0x,0xx00 Or
U is confounder
Deg. Or, By, Ox
Deg Or. Oy Ou
Deg. Or, 0u
Deg. dy 0x. Ou
Deg. Or. Oy
Deg. 0r, 8x
Deg. 0r. 0u
Deg. Oy 0x
Deg. Oy: Ou
Deg. xx, Ou
X Y
7 T-Y Y
U U-T T
U-Y
U-X X
Deg. 0r
Deg. By
Deg. 0x
Deg. Ou
X
id
144
X
X
X
X
X
X
2
3
4
4
5
7
8
10
11
12
13
14
15
16
11
17
18
19
20
XXXXX
21
22
+
1
7
23
24
7
25
26
27
28
29
30
31
4
32
33
34
35
36 +
37
38
4 +
39
4
40 7 +
30


Reviews:
[{"review_id": "BCLP74uhjp", "reviewer": "Reviewer_1FZJ", "summary": "This paper presents a method for detecting hidden confounding in observational data.  Without further assumptions, detecting hidden confounding is known to be impossible.  This paper takes advantage of a common scenario where one has access to observational data collected from multiple environments.  Given particular assumptions that some of the underlying mechanisms and/or exogenous factors are varying across environments, this paper presents a method to detect hidden confounding between a pair of target variables <T,Y>.  The paper presents the theory, a discussion of its assumptions and which can be weakened, synthetic experiments and one experiment in a real-world dataset.\n", "questions": "\n- The current causal graph representation of the problem seems like it cannot represent situations where selection bias is occurring.  I.e., where we only observe X,Y,T when certain conditions hold and essentially throw away the data if those conditions do not hold.  Is my understanding correct?\n\n- Assuming that the parameters $\\theta_{T,Y,X,U}$ vary across environments is a strong assumption.  I appreciate the discussion of a violation of this assumption starting in line 237.  I'd recommend adding a forward-reference to this discussion weakening assumptions.\n\n- There are various positivity assumptions here (e.g., in sampling causal mechanisms and in sampling data within an environment).  In practice, such positivity assumptions are violated often.  E.g., we may gather many sets of data in country A where some variable may take only a small range of values.  Later, we may realize there is hidden confounding when we gather new data sets in country B and the given variable takes on a larger range values and, for example, passes some threshold.  I am concerned that a naive user of your method will over rely on your method's analysis of data from country A and believe that there is no hidden confounding that will threaten their inferred causal inference mechanism even in country B.  Note that this is distinct from the discussion section 4.1 in lines 237-245 --- even if P(T|Pa(T)) is varying --- even if an RCT is performed in country A --- it would not identify that effects will be different in our hypothetical country B.   It would be good to highlight this limitation, e.g., as another paragraph in 4.1\n", "limitations": "\nThe authors have largely addressed limitations and broader impacts.  I appreciate the note of caution regarding use of this work in high-stakes settings.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- detecting hidden confounding under reasonable assumptions is a useful contribution.\n\n- the paper is well written and easy to understand.  I appreciate the additional detail on assumptions in sec 4.1.\n\n- experiments demonstrate implications in synthetic settings and 1 real-world dataset", "weaknesses": "- I don't think this approach will handle detection of hidden confounding under selection bias (i.e., where a data collection mechanism means not all data samples are visible in an environment).  This is a common challenge in practice, and it is worth mentioning this as a limitation and/or noting it in the initial problem statement.\n\n- the paper addresses a scenario where causal mechanisms are parameterized by environment variables $\\theta_{T,X,U,V,..}$.  If the causal mechanisms themselves are varying, I'm not sure I understand what the motivating causal inference scenario is.  That is, if the causal inference question is to identify $P(Y|T)$, how is that a well-defined question if the causal mechanism defining $Y|Pa(Y)$ is changing across environments?\n\n- the paper would be improved with additional real world datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 1FZJ for their insightful feedback and questions. Based on their recommendation, to help future readers of our paper, we have added an earlier reference to our discussion on the violation of our assumptions when first presenting the assumptions in the paper.\n\n## The influence of selection bias in our theory\n\n1FZj raises an important point that we have not commented on what would happen under selection bias. Indeed, in our current setup, we assume there is no selection bias, and have not discussed this further in the paper. We will clarify this in our problem description to avoid confusion for future readers. Still, it is interesting to reason how our procedure would be influenced by selection bias. To do so, we can add colliders in our original graph in the paper to introduce different types of selection mechanisms. As a quick illustration of how selection bias can (or can not) hurt our procedure, we provide an example: say there is a selection mechanism that acts as a collider between treatment and outcome. Further, let this selection mechanism be changing across environments – maybe we have different conditions for when data is thrown out in each environment. Then, we expect our procedure to falsely detect hidden confounding even when it is not present. But if, on the other hand, the selection mechanism remains fixed across all environments, we do not have this problem of false detection anymore, and the proposed approach would still work. We will add this example to the appendix, where we also discuss the influence of other assumption violations.\n\n## What is the causal inference question if $P(Y\\mid Pa(Y))$ is changing across environments?\n\nThis is an attentive question, as it is always good to carefully think about the causal estimand of interest. First of all, as discussed in section 4.1, it should be noted that it is not necessary for $P(Y\\mid Pa(Y))$ to change across environments for our procedure to work. Meaning, it works whether $P(Y\\mid Pa(Y))$ changes or not. Therefore, one could very well assume that $P(Y\\mid Pa(Y))$ (or at least $E[Y\\mid Pa(Y)]$) is fixed – as in a standard causal inference setting. \n\nWe can however also give an example where  $P(Y\\mid Pa(Y))$ changes with the environment $E$ and we still have well-defined estimands. Say that our environments are the different states of the US, like in the Twins dataset we use in our experiments. One possible causal estimand here would be state-specific interventions, that is $P(Y\\mid do(T=t), E)$. This goes more in the direction of attempting to capture treatment effect variation, which is natural to do in a hierarchical model (Feller and Gelman, 2015). We could however also country-wide interventions $P(Y\\mid do(T=t))$ – meaning we aggregate the state-specific effects into one country-wide effect. For both of these estimands, our method still helps us determine if the estimand is identifiable. Because in both cases we require that there is no unmeasured confounding within each environment.\n\n## Violation of positivity assumption\n\nWe thank reviewer 1FZJ for bringing up this discussion on positivity violations. Related to this discussion, we think it also highlights the need to define a population of interest. In practice, positivity violations are common and can be divided into two categories: structural violations and random violations (Hernán and Robins, 2020). Structural violations occur for instance when a certain range of values of a variable never will be observed, and they may restrict the population for which we can draw causal conclusions. Meanwhile, random violations are due to having a finite number of samples. Random violations are perhaps also less problematic, as they can go away as we collect more data.  \n\nIn our setting with multi-environment data, we can make the same analogy with structural and random positivity violations. If we understand the setting you are describing correctly, you are describing a structural violation where some types of environments are never observed in country A. Thus, before using our method, we think it is important to ask ourselves: in what population are we trying to detect if there is hidden confounding? If the answer is the population in country A, then we should not use data from country A to also make conclusions about country B, and vice versa. For random violations, it is hard to anticipate what problems can come up, but we can avoid them by having enough environments from the population of interest. \n\nWhile positivity violations are not a unique limitation to our method, we agree that it is important to highlight how to reason about them in our problem setting. This would encourage responsible usage of our method. We are adding a paragraph on this in the appendix, in the same section as where we discuss the other assumption violations.\n\n## References\n- Feller, Avi, and Andrew Gelman. \"Hierarchical models for causal effects.\" Emerging Trends in the Social and Behavioral Sciences: An interdisciplinary, searchable, and linkable resource (2015): 1-16.\n- Hernán Miguel A., Robins James M. Causal Inference: What If. Boca\nRaton: Chapman & Hall/CRC. (2020)\n\n"}, {"review_id": "giPtEz4C5A", "reviewer": "Reviewer_ZLeH", "summary": "The authors proposed a detection method for hidden confounders, with only observational data in multiple environments. They, in particular, designed a testable independence condition for it, under the assumptions of (i) Faithfulness & Causal Markov Property, (ii)Shared Causal Graph, (iii) Independent Causal Mechanism Principle, and (iv) Non-degenerate Probabilistic Independent Causal Mechanisms. They also performed synthetic and semi-synthetic experiments to verify the effectiveness of their proposed method.", "questions": "Regarding the problem setting, there are some confusing notations. For example, since the $E^{(k)}$consists of four causal mechanisms shown in Fig.1(a), how could it serve as an indicator variable for what environment observations belong to? Btw, why $E^{(k)}$ is observed while the $(\\theta_T^{(k)} , \\theta_Y^{(k)}, \\theta_X^{(k)}, \\theta_U^{(k)} )$ are all unobserved, whereas $E = (\\theta_T^{(k)} , \\theta_Y^{(k)}, \\theta_X^{(k)}, \\theta_U^{(k)} )$.\n\nIn line 116, the authors said, “We do not assume to know anything about the individual parameters $(\\theta_T, \\theta_Y, \\theta_X, \\theta_U )$, however in their method, they sampled the mechanisms i.i.d. for $(\\theta_T, \\theta_Y, \\theta_X, \\theta_U )$.\n\nIn line 113, ${T, Y, X, Y}$ might be ${T, Y, X, U}$.\n\nIn Figure 1, $\\theta_T, \\theta_Y, \\theta_X, \\theta_U$ are parents of the ${T, Y, X, U}$.respectively. So I suggest removing the subscript $\\theta_V$ in Eq.2 since it’s easily misleading for the readers that $\\theta_V$ is the parameter of conditional distributions.\n\nRegarding the theory, Theorems 1 and 2 seem to derive the independence condition from a fixed environment $k$ other than using the advantages from multiple environments, as shown in Eq.(3). In contrast, the authors used multiple environments for the independent testing.\n\nSince the goal is to detect latent confounders, I wonder if some other baselines for a single environment could obtain better results. We might see multiple environments as one environment for this verification.\n\nIn line 289, the distribution notation for the $\\theta_V^{(k)}$ should have reflected the heterogeneity.\n", "limitations": "The needed number of environments is so high.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper is well written with clear motivation and a description of their method.\n\nWhat the authors focused on is indeed an interesting yet challenging research topic in causal inference. And the idea of using conditional independence conditions, especially between observations, is interesting to me.", "weaknesses": "The problem setting has some confusing notations.\n\nThe experiments cannot verify better performances than other methods for detecting latent confounders.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer ZLeH for their helpful feedback. We have corrected the typo on line 113 pointed out by them. For clarity, we will address their questions in a Q&A format below.\n\n**Q: Since the $E^{(k)}$ consists of four causal mechanisms shown in Fig.1(a), how could it serve as an indicator variable for what environment observations belong to? Btw, why is $E^{(k)}$ observed while the $(\\Theta_T^{(k)}, \\Theta_Y^{(k)}, \\Theta_X^{(k)}, \\Theta_U^{(k)})$ are all unobserved, whereas $E^{(k)}=(\\Theta_T^{(k)}, \\Theta_Y^{(k)}, \\Theta_X^{(k)}, \\Theta_U^{(k)})$.**\n\nA: In each environment, denoted with $E^{(k)}$, we have the parameters $(\\Theta_T^{(k)}, \\Theta_Y^{(k)}, \\Theta_X^{(k)}, \\Theta_U^{(k)})$ for the conditional distributions (i.e. causal mechanisms) in eq. (2) in the paper. Importantly, we are saying that for a specific environment, the distribution $P(T,Y,X,U \\mid \\Theta_T^{(k)}, \\Theta_Y^{(k)}, \\Theta_X^{(k)}, \\Theta_U^{(k)})$ is fixed as the parameters are fixed. These parameters themselves are not observed, in the sense that their values are unknown. For simplicity we wrote $E^{(k)}=(\\Theta_T^{(k)}, \\Theta_Y^{(k)}, \\Theta_X^{(k)}, \\Theta_U^{(k)})$ to underline their connection. But we understand how using an equality sign here might be confusing. For this reason, we have removed this equality statement in the text and Figure 1a, and included a better explanation of their relationship in the original paragraph (lines 112 - 118).\n\n**Q: In line 116, the authors said, “We do not assume to know anything about the individual parameters $(\\Theta_T^{(k)}, \\Theta_Y^{(k)}, \\Theta_X^{(k)}, \\Theta_U^{(k)}$”. However in their method, they sampled the mechanisms i.i.d. for $(\\Theta_T^{(k)}, \\Theta_Y^{(k)}, \\Theta_X^{(k)}, \\Theta_U^{(k)})$.**\n\nA: We will clarify what we mean by “not knowing anything” in this sentence. In particular, our approach does not require that we observe the values of these parameter values $(\\Theta_T^{(k)}, \\Theta_Y^{(k)}, \\Theta_X^{(k)}, \\Theta_U^{(k)})$. We do not even try to infer them in our method. The only assumption we make about them is that they are pairwise independent random variables with non-degenerate distributions. We will edit line 116 to make this clearer.\n\n**Q: In Figure 1,  $(\\Theta_T^{(k)}, \\Theta_Y^{(k)}, \\Theta_X^{(k)}, \\Theta_U^{(k)}$ are parents of the $(T,Y,X,U)$ respectively. So I suggest removing the subscript $\\Theta_V$ in Eq. (2) since it’s easily misleading for the readers that $\\Theta_V$ is the parameter of conditional distributions.**\n\nA: We believe this is a misunderstanding, as this is exactly how we intend them to be interpreted.\n\n**Q: Regarding the theory, Theorems 1 and 2 seem to derive the independence condition from a fixed environment $k$ other than using the advantages from multiple environments, as shown in Eq.(3). In contrast, the authors used multiple environments for the independent testing.**\n\nA: Let’s clarify what the independence condition in Theorem 1 says (the same explanation also applies to Theorem 2). Within a given environment $E^{(k)}$ if there is hidden confounding, Theorem 1 states: the outcome of individual $i$ should be dependent on the treatment given to another individual $j$ even after adjusting for all other available information. This condition, while relating two individuals in the same environment, holds for all $k$. To better understand why multiple environments are necessary, we can look at the following example. If we only had a single environment, then the data would always be i.i.d. as the parameters are fixed within a single environment. Thus, the condition in Theorem 1 would fail because individuals always would be independent of each other – regardless of whether there is hidden confounding or not. \n\n**Q: Since the goal is to detect latent confounders, I wonder if some other baselines for a single environment could obtain better results. We might see multiple environments as one environment for this verification.**\n\nA: It is well known that without additional assumptions in a single environment, it is impossible to detect hidden confounding (see, for instance, Peters et al, 2017). Our key research question is whether access to data from multiple heterogeneous environments allows us to detect hidden confounding. Keeping this question in mind, the strongest baseline we found was the Joint Causal Inference framework by Mooij et al (2020) which, similar to ours, also uses conditional independence testing with multi-environment data and makes no further parametric assumptions. Still, we do mention other methods in the related work section, such as Janzing and Schölkopf (2018) that can be applied in a single environment setting. Their approach, however, assumes linear relations with a one-dimensional real-valued confounder. These are strong assumptions that deviate significantly from our setting, making it difficult to construct a fair comparison. With that said, we are happy to hear if the reviewer has specific suggestions for other suitable baselines.\n\n**Q: In line 289, the distribution notation for the $\\Theta_V^{(k)}$ should have reflected the heterogeneity.**\n\nA: We believe this is already reflected as $\\Theta_V^{(k)}$ is Normal-distributed across environments. We are happy to further discuss how to clarify this for the reviewer if they can elaborate on their question. \n\n## References\n- Janzing, Dominik, and Bernhard Schölkopf. \"Detecting confounding in multivariate linear models via spectral analysis.\" Journal of Causal Inference 6.1 (2018): 20170013.\n- Mooij, Joris M., Sara Magliacane, and Tom Claassen. \"Joint causal inference from multiple contexts.\" The Journal of Machine Learning Research 21.1 (2020): 3919-4026.\n- Peters, Jonas, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.\n"}, {"review_id": "t01GEOBuMQ", "reviewer": "Reviewer_7Ey5", "summary": "The authors focus on a setting where we observe treatments, outcomes, and a set of confounders across multiple environments.  While it's generally not possible to identify unobserved confounders in a single data set, when we have multiple data sets and assume the mechanisms operate in the same way across those environments (even if the parameters may differ), we can test for the presence of unobserved confounders.  The authors propose an independence test between a treatment in one environment and an outcome in another, and show that, if they are not independent (conditioned on the observed confounders and the other treatment), then there must be at least one unobserved confounder that's producing that dependence.", "questions": "While being able to detect whether or not there is hidden confounding is useful, it would be more useful to then make some statement about the strength of hidden confounding, or how much we might expect it to affect our estimates.  I suspect in many realistic situations, there is going to be /some/ hidden confounding.  If care is taken when choosing which potential confounders to measure, though, the effect of this hidden confounding is hopefully small. Do you think your approach could be used or modified to answer questions about the strength of hidden confounding?", "limitations": "The authors' approach is confined to a fairly specific setting, where the same variables, operating in the same manner (with different parameters), are measured across different environments.  The approach also only outputs a binary yes or no for whether hidden confounding exists.  If the test comes back positive for hidden confounding, it's unclear if this means causal analysis is impossible, or what the next steps should be.\n\nIn terms of societal impact, I don't see any real potential for negative impact.  This work can be used to test for one potential hurdle for accurate causal inference (hidden confounders), but there are many other assumptions that need to be met as well, and this paper does a good job at making those clear.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The presentation of this paper is excellent, and I found the explanation throughout very clear.  The idea comes across as fairly simple, which speaks to the clarity of the explanation.   The graphics in particular (Figures 1 and 2) go a long way towards making the approach understandable\n\nI appreciate the attention paid to assumptions, including the discussion in 4.1 about assumption relaxation.", "weaknesses": "The main weakness I see of this paper is the lack of motivation for the setting the authors consider.  This work is focused on a situation where we have observational data from multiple environments where we have collected the same variables from all environments and, while parameters are allowed to differ between them, the actual mechanisms at play need to have the same structure.  The authors bring up the example in the intro of different hospitals administering the same treatment but serving different populations, but otherwise, I feel like this work would be served by a discussion of how realistic this sort of setting is, or how common it is in the real world.\n\nSimilarly, while I do like Section 4.1 and the authors' attention to the assumptions, the lack of clear real-world ties extends here.  What would it look like in real data if we had pair-wise dependent mechanisms?  Is this something we would expect to see frequently?  This lack of discussion makes it hard to assess how important these assumptions actually are, and how useful the relaxations are in turn.\n\nConsidering that Section 4.2 is the actual core of the approach, I wish more attention were paid to it.  I understand that space is limited, but including the actual algorithm you're proposing seems worth moving something else to the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 7Ey5 for their valuable feedback on our work. As suggested by them, in the final version of the paper, we intend to bring the algorithm back to the main paper if space allows in the camera-ready version. Similarly, we will also extend the introduction with two additional examples of use cases for our method.\n\n## The real-world practical utility of our approach (*see global rebuttal reply*)\n\nAs a very similar question also was raised by reviewer g1qg on the real-world motivation of our problem setting, we have written a joint reply found in the global author rebuttal reply. We kindly ask reviewer 7Ey5 to read it there.\n\n## Quantifying the strength of hidden confounding\n\nWe agree that it would be a great addition to go beyond a yes/no answer on the presence of hidden confounding and instead quantity the magnitude of the bias incurred by hidden confounding. We mention this possibility in our future works section in the appendix. While doing so is a much more difficult task, one empirical observation makes us believe our method can be modified to quantity the strength of the hidden confounding. Figure 4a in the paper shows that the test statistic from the conditional independence test increases if the bias from hidden confounding goes up. Thus, perhaps this test statistic can act as a proxy for the size of the bias from hidden confounding. For instance, a simple heuristic procedure to gauge the effect of hidden confounding would involve eliminating one or more observed confounders from the analysis and seeing the resulting variations in both the test statistic and the estimated causal effect. This would give us a calibrated guess on how the magnitude of the statistic relates to the bias. By then evaluating the test statistic when all observed confounders are accounted for, we might get a clue what the corresponding remaining bias could be. Turning such ideas into a principled method requires significantly more research, which we are currently engaged in.  \n"}, {"review_id": "q6sLIY2gMM", "reviewer": "Reviewer_g1qg", "summary": "The authors develop a test for hidden confounding using only observational data from multiple environments with the same DAG under the independent causal mechanisms assumption.", "questions": "I do not have any question beyond some points in the weaknesses part.", "limitations": "The authors acknowledge some important limitations page 6.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and covers the related work well.\nTheir setting, assumptions, and contributions are clear and they meaningfully discuss the profound limitations of their work with potential solutions.\nUnder new (but strong) assumptions, they propose a test for no unmeasured confounding and provide an algorithm using only observational data. The proofs in the construction of the test and the test itself is novel and can be counted as original contributions.\nThey support their claims through adequate (semi)-synthetic experiments.", "weaknesses": "1. My main concerns about the paper is about its practical utility.\n    - It is not often a practitioner has access to multiple observational datasets\n    - The observational datasets may be highly heterogeneous, e.g., the type of data recorded, the way they are recorded, etc., making the preprocessing task for those datasets to be used in a way that is proposed in this manuscript very hard, if not impossible.\n    - Analyzing observational studies is hard for a lot of reasons. As we increase the number of studies, the chances of introducing bias through other and often overlooked ways increase such as non-adherence to treatment assignments, or selection bias introduced while defining the treatment assignments and where the follow-up begins etc. Those can all be reasons behind rejection beyond confounding. Please correct me if I am wrong in that regard.\n\n2. I find it hard to believe that different observational datasets will have the same DAG.\n    - Different environments (e.g. hospitals) may have very different mechanisms (e.g. institutional practices) that result in different DAGs.\n\n3. In summary, although the paper formalizes a framework where we can test the no unmeasured confounding assumption, I think the approach would only work in very controlled settings.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful questions, as well as pointing out the novelty of designing the test in our paper. We further agree that it is important to keep in mind that our test relies on new strong assumptions that should not be taken for granted.\n\n## The real-world practical utility of our approach (*see global rebuttal reply*)\n\nAs a very similar question also was raised by reviewer 7Ey5 on the real-world motivation of our problem setting, we have written a joint reply found in the global author rebuttal reply. We kindly ask reviewer g1qg to read it there.\n\n## Having the same DAG in multiple observational datasets\n\nAn important assumption is that each environment shares the same DAG. We gave an example of this with the multi-level study from Leite et al. (2015) where a shared structure is assumed across clusters. In this case, such an assumption does not come across as particularly controversial. Looking more broadly, having an invariant structure is a common assumption in causal inference literature, see for instance Barenboim and Pearl (2016) and Rothenhäusler et al. (2021). In these works, the DAG stays the same across different datasets/environments, and the influence of changing environment is modeled by adding it as a variable in the shared DAG. We do agree with the reviewer that it is good to think about this assumption carefully, but we also remain positive that this assumption can oftentimes be made, since the structure of the graph often encodes invariant properties, such as which variable precedes which other variable in time, while the exact mechanism itself is likely to change across environments. It is these variations in the mechanisms that we exploit to construct our method.\n\n## References\n- Bareinboim, Elias, and Judea Pearl. \"Causal inference and the data-fusion problem.\" Proceedings of the National Academy of Sciences 113.27 (2016): 7345-7352.\n- Leite, W. L., Jimenez, F., Kaya, Y., Stapleton, L. M., MacInnes, J. W., & Sandbach, R. (2015). An evaluation of weighting methods based on propensity scores to reduce selection bias in multilevel observational studies. Multivariate behavioral research, 50(3), 265-284.\n- Rothenhäusler, Dominik, et al. \"Anchor regression: Heterogeneous data meet causality.\" Journal of the Royal Statistical Society Series B: Statistical Methodology 83.2 (2021): 215-246.\n"}]

Author rebuttal:
This is a joint reply to reviewers 7Ey5 and g1qg for their questions on the real-world practical utility of our proposed method. For the camera-ready version, if space allows, we will include the below examples in the introduction of our paper to help future readers understand our method’s practical utility.

## The real-world practical utility of our approach

We believe reviewers 7Ey5 and g1qg both bring up important points of addressing how realistic or common our proposed setting is. Furthermore, as mentioned by g1qg, analyzing multiple observational datasets can be very hard. Especially when it comes to pre-processing and taking into account other sources of bias. Yet, there are many settings where such analyses are done and below we give some examples; these also describe scenarios where we believe our method would be suitable to use. 

Firstly, note that environments don't need to come from completely different studies: they could also be pre-existing clusters within a study. Particularly, we believe our method is applicable in many existing multi-level studies in which individuals are nested in clusters and nonrandomly assigned to a treatment/control on an individual level. In this type of setting it is natural to introduce a hierarchical model, like the one we consider in our paper. One example is found in Leite et al. (2015): they are interested in estimating the effect of giving special education services to first-graders on their reading performance in the third grade. They analyze data with 7,783 students (i.e. individuals) clustered across 1454 schools (i.e. environments). In this case, they use a hierarchical model that shares the same structure for all schools, and they include random parameters to allow changing causal mechanisms between schools. 

Another example setting is individual participant data meta-analyses. This is a type of analyses that uses all individual-level data from multiple studies instead of aggregating summary statistics (Riley et al. 2010). They are resource intensive but also do enable us to draw stronger conclusions. While individual participant data meta-analyses can vary in size, Di Angelantonio (2016) offers one example where 239 prospective studies are jointly analysed. This showcases a setting where many observational datasets are analyzed jointly and where our method could be used. 

We hope these two examples draw a better picture of the real-world use cases of our proposed method. For the camera-ready version, we will refer to both of these examples in the introduction of our paper to help future readers understand our method’s practical utility.

## References
- Di Angelantonio, Emanuele, et al. "Body-mass index and all-cause mortality: individual-participant-data meta-analysis of 239 prospective studies in four continents." The Lancet 388.10046 (2016): 776-786.
- Leite, W. L., Jimenez, F., Kaya, Y., Stapleton, L. M., MacInnes, J. W., & Sandbach, R. (2015). An evaluation of weighting methods based on propensity scores to reduce selection bias in multilevel observational studies. Multivariate behavioral research, 50(3), 265-284.
- Riley, Richard D., Paul C. Lambert, and Ghada Abo-Zaid. "Meta-analysis of individual participant data: rationale, conduct, and reporting." Bmj 340 (2010).


Meta-review: The paper proposes a new approach for detecting hidden confounding when multiple datasets are available, sharing some of their causal structure. This is done without performing full causal discovery, a daunting and often impossible task. The paper gives some compelling use cases, and good empirical evidence for the performance of the method. This is a novel and useful research approach, and I expect substantial future work will build on the results presented here.
Decision: Accept (poster)