Falsification of Unconfoundedness by Testing Independence of Causal Mechanisms
Authors: Rickard Karlsson
JH Krijthe
Venue: ICML 2025 poster (2025)
Keywords: causal inference
observational data
falsification
unmeasured confounding
independent causal mechanisms
Primary area: general_machine_learning->causality
TL;DR: We develop a novel algorithm to falsify the assumption of no unmeasured confounding using multi-environment observational data.

Abstract: A major challenge in estimating treatment effects in observational studies is the reliance on untestable conditions such as the assumption of no unmeasured confounding. In this work, we propose an algorithm that can falsify the assumption of no unmeasured confounding in a setting with observational data from multiple heterogeneous sources, which we refer to as environments. Our proposed falsification strategy leverages a key observation that unmeasured confounding can cause observed causal mechanisms to appear dependent. Building on this observation, we develop a novel two-stage procedure that detects these dependencies with high statistical power while controlling false positives. The algorithm does not require access to randomized data and, in contrast to other falsification approaches, functions even under transportability violations when the environment has a direct effect on the outcome of interest. To showcase the practical relevance of our approach, we show that our method is able to efficiently detect confounding on both simulated and semi-synthetic data.

Falsification of Unconfoundedness by Testing
Independence of Causal Mechanisms
Rickard K.A. Karlsson
Jesse H. Krijthe
Abstract
tion is that we assume there is unmeasured confounding, meaning that there are unobserved factors that have both an influence on the treatment and on the outcome of interest that are not accounted for by the method. If unmeasured confounders are present, our treatment effect estimates are likely to be biased and inconsistent (Greenland al., 1999). This can have serious downstream consequences such as unknowingly recommending non-effective even worse, potentially harmful treatment policy. Unfortunately, without making further assumptions, it is in general impossible to verify all assumptions needed to identify treatment effects from observational data.
A major challenge in estimating treatment ef- fects in observational studies is the reliance on untestable conditions such as the assumption of no unmeasured confounding. In this work, we propose an algorithm that can falsify the assump- tion of no unmeasured confounding in a setting with observational data from multiple heteroge- sources, which refer to environments. Our proposed falsification strategy leverages a key observation that unmeasured confounding can cause observed causal mechanisms to appear de- pendent. Building on this observation, we develop a novel two-stage procedure that detects these de- pendencies with high statistical power while con- trolling false positives. The algorithm does not require access to randomized data and, in con- trast to other falsification approaches, functions even under transportability violations when the environment has a direct effect on the outcome of interest. showcase the practical relevance our approach, we show that our method is able to efficiently detect confounding on both simulated and synteetnt data.
In this work, we investigate novel strategy for falsifying unconfoundedness. Specifically, focus on the common scenario where observational datasets are collected from different heterogeneous sources, which we refer to envi- romments. Each environment corresponds to distinct study populations, due to factors such as geographical differences that results in distribution shifts between the environments. We propose falsification strategy based on the assump- tion that these distribution shifts stem from independent changes in the underlying causal mechanisms. This idea is grounded in the principle of independent causal mechanisms (ICM) (Janzing et al., 2012; Peters et al., 2017), which posits that causal system comprises autonomous modules that do not inform or influence each other. Assuming indepen- dent causal mechanisms has been leveraged to, for instance, improve causal structure learning (Huang et al., 2020; Guo et al., 2024a) and understand model behavior in statistical machine learning (Schölkopf al., 2012). However, the implications of assuming independent causal mechanisms to treatment effect estimation problems has received far less attention.
1. Introduction
Using observational studies to estimate treatment effects a ubiquitous yet challenging task in many disciplines, such as medicine (Hernán & Robins, 2006) or social sci- ences (Athey & Imbens, 2017). Whereas there exists rich literature of methods for treatment effect estimation in the observational setting (Bang & Robins, 2005; Wager & Athey, 2018; Chernozhukov et al., 2018), all methods have in common that before a causal effect can be estimated, often untestable conditions need to hold. One such condi-
Our proposed falsification strategy leverages key observa- tion that unmeasured confounding cause observed mech- anisms to appear dependent (Janzing & Schölkopf, 2018; Karlsson & Krijthe, 2023; Mameche et al., 2024b; Reddy & Balasubramanian, 2024). If we assume that the underlying causal mechanisms should be independent, contrary to what is observed, follows that any apparent dependencies could be the result of unmeasured confounding. This observation motivates the central research question of this paper: How
Department of Intelligent Systems, Delft University of Tech- nology, the Netherlands. Correspondence to: Rickard Karlsson k.r..a.kallsonnee @atissoonuueeft......
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Falsification of Unconfoundedness by Testing Independence of Causal Mechanisms
can we efficiently test causal mechanism independencies to falsify the conditions required for treatment effect estimation in settings with multi-enviromment data?
while sensitivity analysis probes 'what-if" scenarios regard- ing potential unmeasured confounding (a process that can always be undertaken), falsification aims to empirically test whether assumptions are violated, based on the observable implications of those assumptions (which is not always fea- sible). For instance, falsification may involve testing the validity of instrumental variables (Pearl, 1995) or evaluating the compatibility of learned causal structures with observed data (Faller et al., 2024). In this way, sensitivity analysis and falsification are complementary: the former explores possible scenarios, while the latter seeks direct empirical evidence for these scenarios. Despite this, falsification has received comparatively less attention in the literature.
Contributions By formalizing the problem using Neyman-Rubin causal model for multi-environment data, we show that falsification of unconfoundedness is possi- ble by testing dependencies between causal mechanisms directly by combining the principle of independent causal mechanisms with functional assumptions on the mecha- nisms. In this model, we prove that the presence of un- measured confounding has testable implications in the form of dependencies between the model's 's observed parameters. Using our theoretical results, we introduce new algorithmic ideas that can be used for falsification: in particular, we propose a two-stage algorithm that statistically tests statisti- cal dependencies between learned model parameters of the treatment assignment and outcome mechanism. We show that our algorithm performs favorably compared to alterna- tive approaches on both simulated and semi-synthetic data.. To showcase the potential applications of our algorithm and clarify what constitutes an "environment", we provide two illustrative examples where we envision our algorithm being used.
One line of research on falsification in observational causal inference assumes that certain transportability conditions hold, allowing causal effects to be transferred between dif- ferent environments (Dahabreh et al., 2020b; Hussain et al., 2022; 2023). The basic premise is that, under transportabil- ity conditions, comparing treatment effect estimates from multiple observational studies, or from a single observa- tional study and a randomized one, should yield consistent results. If inconsistencies are found, this can be used fal- sify the identifiability conditions, assuming the transporta- bility assumptions hold. This idea has been further extended to time-to-event outcomes with censoring (Demirel et al., 2024), as well as for quantifying bias from unmeasured confounding (De Bartolomeis et al., 2024a;b). In contrast, our approach assumes independence of causal mechanisms, which does not require transportable treatment effects or access to randomized data.
In a meta-analysis of multiple observational studies with individual participant data (Riley et al., 2010), our algorithm can jointly test whether an unmeasured con- founder is present between the treatment and outcome across all studies. Here, each observational study serves as dis- tinct environment.
Example 1
Testing for independence of causal mechanisms has been applied in previous work to falsify causal assumptions, such as detecting hidden confounding (Karlsson & Krijthe, 2023) or testing the validity of instrumental variables (Burauel, 2023; Karlsson et al., 2023). Most similar to our work is that of Karlsson & Krijthe (2023), though their method relies conditional independence testing which is notoriously difficult statistical problem in itself (Shah & Peters, 2020). To avoid the challenges of conditional independence testing- for instance, losing statistical power as the adjustment set becomes larger-we we address this problem by proposing an alternative method that does not rely on conditional inde- pendence testing.
Example 2 In a single observational analysis involving lulti-level structure in which individuals are nested in clusters and randomld assigned to a treatment/control on an individual level, such as students from different schools (Leite et al., 2015) or patients from different hospi- tals (Goldstein et al., 2002), our algorithm can test whether the conditions necessary to identify treatment effects are vio- lated within each cluster due to unmeasured confounding, In this context, the environment refers the sub-popun poob-popllatios within each cluster of the same observational study.
2. Related Works
Parallel to our ideas on falsification, other approaches have been proposed for detecting or addressing unmeasured con- founding, under various assumptions on the setting and data- generating process. For example, when multiple causes are observed (Wang & Blei, 2019; D' Amour, 2019) or when a negative control is available (Lipsitch et al., 2010).
When discussing the validity of causal assumptions, sensi- tivity analysis might come to mind. In sensitivity analysis, one hypothesizes departures from the assumption of no un- measured confounding and investigates how different biases would arise depending on the hypothesized confounder's relationship with treatment and outcome (Cornfield et al., 1959; Tan, 2006; VanderWeele & Ding, 2017). This typi- cally results in bounds on the treatment effect, which is an instance of partial identification (Manski, 2003). However,
Finally, our work investigates the implications of the princi- ple of independent causal mechanisms, which has a rich liter- ature in causal discovery, particularly in culti-ennvioommnt
2

---

Falsification of Unconfoundednes by Testing Independence of Causal Mechanisms
settings (Huang et al., 2020; Perry et al., 2022: Guo al., 2024a; Mameche al., 2024a). A closely related line of research assumes the existence of invariant mechanisms across environments (Peters et al., 2016). In contrast, our approach explicitly allows these mechanisms to vary-and, as we will show, such variation is sometimes necessary to enable falsification. Rather than aiming to learn the en- tire causal structure as typically done in causal discovery, our approach focuses on verifying specific aspects of a par- tially known structure that is relevant for treatment effect estimation. Recently, Guo et al. (2024b) examined how independent causal mechanisms can lead to identification of certain treatment effects, though they did not address sce- narios where causal assumptions are violated, such as in the presence of unmeasured confounders, which we study here.
3.2. Assumptions for identification of causal effects
To define causal effects of interest, we use potential (coun- terfactual) outcomes (Rubin, 1974). For an individual i, we posit the potential outcome Y for E AA which denotes the outcome under an intervention to set treatment A; to a. For the typical causal analysis in a non-nested study design, the goal is often to estimate the average treatment effect or conditional average treatment effect between two different treatments a, a' E A in the underlying population from an environment {S 8). that is Ts E y S s), resp. To (z) E - y X x, s]. It is well- known that under certain conditions and T, (x) are identi-
fied from the observations in environment {S = =}.
Assumption 3.1. We assume the following conditions for each environment & = l,..., K. Consistency: if A, a, then Y, = Y,, for every individual i and every treat- ment a E A. Positivity: for each treatment a E A, if [e, S = 8) + O, then Pr(A = a|x- = S = 8) > 0. Unconfoundedness: for each E A. 1 A ==s) s).
3. Setup
3.1. Notation & data structure
For each individual = I..... n. we observe bascline co- variates X, in X' C Rd, treatment A; in A S I and outcome Y; in y C R. We allow the treatment and out- come to be binary or continuous; but to simplify exposi- tion, we will mainly show our results for the continuous case and then discuss how to modify our theory for binary treatments and outcomes when appropriate. We consider observations to be collected from K different environments KJ where K 2 2. We denote n
Consistency is satisfied when the treatment is clearly de- fined, ensuring that no hidden treatment variation exist and that there is no interference between individuals. Positivity requires that every possible covariate pattern in the envi- ronment S = has a nonzero probability of receiving each possible treatment option. Unconfoundedness, also referred to as conditional exchangeability, implies there is no unmea- sured confounding. That is, the covariates X are sufficient to adjust for in order to identify the causal effect of A on Y. In observational studies, assuming unconfoundedness is often considered controversial, requiring strong domain expertise to justify its validity.
labeled with S; E as the number of observations from environment {S = }} and we define = Σ n2 as the total number of obser- vations. Each observation therefore consists of the tuple O, (X,, S2, A., Y,). Throughout the paper, we will use capitalized letters to denote random variables and small letters to denote their realized values.
When the conditions in Assumption 3.1 are met, both the average treatment effect and the conditional average treat- ment effect can be identified from the observed data (Her- nan & Robins, 2020). Let (X) = EE X,A A a, S s). then a statistical estimand for the ATE is T, E (X) (X) S s) and for the CATE is T. (X) Ma,s( pa (X)), Rather than focusing on how to estimate these estimands from data, we will con- centrate on how to assess the validity of the conditions that allow us to identify them in the first place. Specifically, in the context of data from multiple environments, we will demonstrate that Assumption 3.1 can be falsified under cer- tain conditions related to distributional shifts across the different environments.
We are considering a setting with a composite dataset of observations from separate environments. Each environ- ment represent a different study population where the sam- pling probability of individual i belonging to environment {5 =} can be unknown; this setting is referred to as a non-nested study design (Dahabreh et al., 2020a). Formally, we consider observations within an environment {S = s} to be sampled independently and identically (i.i.)) according to some distribution (X, A,YY ~ P(X, Y,Y S = s). This distribution may vary across the different environ- ments s E {l,.... K}. Importantly, observations are not assumed to be i.i.d. if we consider the marginal distribu- tion P(X, A, Y) over all environments. Furthermore, we assume that the environments are related to each other by having a shared, albeit unknown, causal structure, that is: the causal directed acyclic graph (DAG) between the vari- ables (X, S, A, Y) is the same for all S E (1,...,K) K].
4. A novel falsification strategy
4.1. Assumptions on environment changes
We consider a general class of models of the treatment and potential outcomes, namely: all linear functions of the
3

---

Falsification of Unconfoundednes by Testing Independence of Causal Mechanisms
feature representations (X): X + IR= and o(X, A) : interventions on the distribution P(X, A, S = s). In X' x A
the broader statistical context, Assumption 4.1 can also be related to hierarchical regression models with prior inde- pendence assumption, see e.g. Gelman (2007, Chapter 11), Here, the independent sampling of the parameters (as,, Ps) resembles the way hierarchical models account for variabil- ity between environments.
A a (Xx EA
(1)
y 8. oxX, A = aa) +
where the noise variables fulfill ElEA X,S = x| 0 and Blsy X, A, = s] = 0. Additionally, under Assump- tion 3.1, the noise variables are independent A T EY | S.
Finally, we will contrast our approach with falsification strategies based transportabilitty, which for instance would assume that the outcome mechanism E[y X, = s] remains invariant across environments S. This assump- tion can be violated when unmeasured effect modifiers differ in distribution across environments, causing E |X,S= = s] to vary. contrast, Assumption 4.1 does not require such invariance and explicitly allows this causal mechanism to vary. As result, even when transportability fails to hold, Assumption 4.1 may still hold. We will later show that this makes our proposed falsification robust to violations of transportability, whereas transportability-baa -based strategies may yield false positives: that is, incorrectly rejecting uncon- foundedness despite the absence of unmeasured confound- ing. For more detailed discussion of transportability-bae falsification strategies, see Appendix A.
The function class described in (1) encompasses wide range of complex models, particularly because the repre- sentations (X)) and ((X, .A) can involve nonlinear trans- formations of the variables A). Although we focus on continuous treatment and outcome values to illustrate the core ideas of our falsification strategy, this framework can be extended to generalized linear models that accom- can modates binary/categorical values. For instance, include binary treatment by defining P(A 1 S = s) = (a, vb (X)), where h(p)= = In(p/(1 (1 - p)) is the logit link function (Mccullagh & Nelder, 1989).
Distributional changes between envirouments are accounted for by (1) through allowing the parameters as E IR² and B. E IR= to change for different environments s E {I.... K}. in addition to changes in the covariate distri- bution P(X| | S = s). Changes in the parameters (a,, B.) correspond to shifts in the treatment assignment mechanism E[A| X, S = s] = a__ (X) and outcome mechanism E|Y" X, S = s) = B, o(X, = a): the feature repre- sentations (X) and o(X, A) are considered to be fixed across environments. In practice, changes in the treatment assignment and outcome mechanisms
4.2. A testable implication under the independence assumption
To focus on the core ideas and limits of our falsification strategy, we assume the feature representations 6 and &b are known up to some permutation and element-wise scal- ing. Moreover, to allow the use of standard estimation techniques, we will require the dimensionality of the feature representations to not be larger than any of the individual sample sizes among the different environments. We formal- ize these two conditions as follows.
often expected. For example, if an unmeasured effect modifier exists, the outcome mechanism E|y X = s] will vary if the distribution of unmeasured effect modifiers differs across environments (Dahabreh et al., 2020a). Similarly, variation in the treatment assignment B[A| S, = s] can be ex- pected due to factors like differences in treatment policies across environments.
Assumption 4.2. We have access to ((X) = CO(X) and (XX,) = (e A) where C' E g=x= and D R are invertible matrices. The dimensionality of the feature representations 2, z' E N is finite and lower than the small- est sample size across environments, i.e, 2, z' < ming Rag
We will now pose the main assumption on how changes in the mechanism parameters (a., B.) occur. Specifically, we assume there exists an unknown prior distribution P(a, B) that fulfills the following condition.
Our proposed falsification strategy will rely on estimating E[A X, S = s] and E|Y X, A, S = s) which under As- sumption 3.1 corresponds to the true treatment and outcome mechanism. To estimate these conditional expectations, we employ two statistical working models e, (X) = w Ø(X) and h2 (X, = 7, (X, A), respectively. Since we re- placed the unknown feature representations {Ø. }/} with the observed feature representations (ő, Ø) the mechanism pa- rameters (a,, are replaced by Yv). Our falsification strategy will test a statement equivalent to Assumption 4.1 but, again, substituting (a,, B.) with (we, 7e) as follows,
Assumption 4.1. The parameters (a., 3a) ~ P(a, B) are drawn independently for each s = =1,... 1,..., K. Furthermore, the parameters are independent from each other such that P(a, B) = P(B).
Following the principle of independent causal mechanisms, Assumption 4.1 states that the parameters (a., B.) are unin- formative of each other as they are sampled independently, and furthermore, that changing a has no influence on 3, and vice versa. In the language of structural causal mod- els, sampling (a., (.)) should be seen as independent soft
Ho P(w,7) P(w)
(2)

---

Falsification of Unconfoundedness by Testing Independence of Causal Mechanisms
To understand how our falsification strategy will be centered around testing this null hypothesis, we begin by establishing the following key result (see Appendix 1.1 for the proof),
and standard deviations o ((A) E ER+ and o o()) E RR. Con- sider the well-specified working models e (x) = wi
o(X)
and (x, A) = 7 (X,, A) with ((X) = [1, and (X,, = AX, AP].. Under the model in (3) with U being an unmeasured confounder, we then have that
Theorem 4.3. Under the functional class described in (1), assumptions 3.1, 4.1 and 4.2, and with e, (X) and h2 (X, A)
being correctly specified models for [[AA] x,5 = s] and B||| A, = s), we have that Ho in (2) is true.
the observable parameters are w. a, + [0% 1 and 7s = 13.: O] + -T, where
(U) (0) ,0]
The above theorem suggests that if we reject the null hypoth- esis Hg. it is likely because at least one of the conditions in the theorem is violated. While rejecting Ho does not tell which condition in the theorem could be false, it still provides valuable information about the validity of the con- ditions in Assumption 3.1 which are necessary for treatment effect estimation. Before introducing our algorithm to sta- tistically test Hy, we first explore a setting where violating Assumption 3.1 provably leads to the falsity of Ho-
-
10)
10
(V) (o!A))
(1)
u
(
2
r. 6,
( (a A(
(U)
(4) 2
8 (
and ó, = -(o! (o!0))2 + (0)0))
2
4.3. Unmeasured confounding leads to mechanism dependencies
The lemma, which holds for any P(X | S = s). states that if U is an unmeasured confounder then the observable parameters (7)) have shared dependencies on the true parameters of the underlying data-generating process: the parameters (ag _ag") .. (V) )) appear in both the ex-
We examine a setting involving a linear causal model that includes both a main effect of treatment and interaction effects between treatment and covariates. While linearity may not always hold in real-world scenarios, this setting offers valuable insights into the conditions necessary to falsify causal assumptions in a multi-environment context.
pressions of wis and Ys--
The above results hold for both single-environment data (K = 1) and multi-environment data (K > 1), and does not rely on Assumption 4.1. Next, we show that our proposed falsification strategy allows us to detect the presence of the unmeasured confounder U under certain conditions on the multi-environment structure when invoking Assumption 4.1 (see Appendix B.3 for the proof).
To understand what effect an unmeasured confounder has on the independence of mechanisms, we introduce another unmeasured covariate U E IR as follows,
A a. (XX) + 00 U +EA y = B. oX - + (B!0) + asssurt) U t
(3)
Theorem 4.5. Under the assumptions stated in Lemma 4.4 and Assumption 4.1, if at least one of the following pa- (U), rameters (a (0) as ,,)0) , (() are i.i.d. sampled from a non-degenerate distribution for 8 = 1, 1,..., K, then Ho is false if and only U a confounder for all s E (1...., K}.
We let (X)= = [1, and A(X. = [1, X, A such that we can define the parameters a, = [a: (
and B. = [ xx( ,3(4),
, 3 {(44))) Throughout this exam- ple, we assume that X 1 U SS.
The above causal model is partially observed because U an unmeasured covariate. If U is a common cause to both the treatment A and potential outcome ya that is
The theorem establishes that Ho can be false due to viola- tions of Assumption 3.1. which can be understood in terms of following statement: unmeasured confounding can create dependencies between observable parameters. The reason (0) (X)
(U) + 0, ( * 0} and/or (a( + 0, (.{{A A 0}, then we say that U is an unmeasured confounder.
at least one of the parameters (a Ore .re ./ss _)(5)) must be sampled from non-degenerate distribution is that this creates a statistical dependence between w, and Ys in the presence of an unmeasured confounder. Detecting this de- pendence becomes crucial for falsifying unconfoundednesss.
Whereas it in general impossible to determine the pres- ence of U. if we have correctly specified working models for E[A] X,S = s) and E|y X, S = s), we note that there exists dependencies between the observable parame- ters W. and % when U is an unmeasured confounder (see Appendix B.2 for the proof).
The parameters (a (X) (x) (U)
0i .1ss .)( are related to the distributions P(A| X,S s) and P(U | 5==5=0 55=55 s). Thus, the non-degeneracy condition implies that falsifying As- sumption 3.1 requires changes in either the treatment as- signment or the distribution of the unmeasured confounder
Lemma 4.4. Assume U has a normal distribution with mean L E R and standard deviation (0) e Rt, and the noise
voriables (c ex) are normally distributed with mean zero
5

---

Falsification of Unconfoundednes by Testing Independence of Causal Mechanisms
across environments. This observation motivates the require- ment of having multi-environment data: that is, without multiple environments there are no distributional changes that enable falsification to happen.
where wi K-- == We,t and 7, - K K- -==1 To..
Lastly, we need to calibrate rejection threshold Rt such that we reject Ho if [ 7) > R while ensuring guarantees on for some
the Type error Pr Pr(T(a,)) > R Ho) a E (0, 1). While this can be done using permutation- based procedure with M resamples, we have to take into account the uncertainty of the estimates from the model fitting in the first step of our algorithm.
The same non-degeneracy condition was observed by Karls- son & Krijthe (2023) despite using a different formalism based on causal graphs. Their approach relies on identifying specific d-separation via a conditional independence test between the treatment and outcome variables, which also allows for falsification of unconfoundedness across multiple environments. Their test can be interpreted as an indirect test of independence of causal mechanisms, as it operates solely on observed variable relationships. In contrast, our approach directly tests independence at the level of mecha- nism parameters. As a consequence, further key difference is that their approach needs to make additional structural assumptions on the covariate distribution P(X (S), while our theoretical results impose no such constraint.
To address this problem, we introduce an additional modifi- cation in the permutation- calibration. Specifically, in the first step, use bootstrapping and resample M datasets with replacement to obtain estimates {(5(m)) 77(m)) ) m== Then, for each m = =I.... M, we compute Tm T T(a(()) s(()) where ((m) is a random permutation of @(m), Finally, we determine the rejection threshold as
M
R = arg max {1: M (0,000))
1(T > 1) a).
m=
5. Algorithm
where 1(T... > 1) equals 1 if Tm t and otherwise 0. Throughout the remainder of the paper, we let M 1000. To highlight the importance of bootstrapping in the calibra- tion, we present an ablation study in Appendix D.5 where we show that bootstrapping is essential for ensuring Type I errors remain below a.
We now introduce the Mechanism INdependent Test (MINT) algorithm, which operationalizes our falsification strat- egy for testing mechanism independence using data from multiple environments. We will use the following no- tation: for all environments s = 1,..., K, we denote the observed data matrices as A, = [AI,..., An,
Y. -Y,,, [Y,,..,, Yn,,_ V. (XI). (X,,,,]] and
6. Experiments
4. =o((X, A1), ................................ o(x,, "T
We conducted a series of experiments to compare the pro- posed MINT algorithm with alternative baseline approaches. First, we investigate efficiency with respect to number of samples and number of environments. Next, we validate our theoretical findings by investigating necessary mecha- nism changes that allow for falsification. We then assessed the sensitivity of our algorithm (mis)specificatioo in its working models. Lastly, evaluated all methods under more realistic conditions using semi-synthetic data based on the real-world Twins dataset (Almond et al., 2005), which includes birth data across different geographical locations used as environment labels.
The MINT algorithm can be divided into two steps: In the first stage, for all 8 = 1 1...., K, we estimate the parame- ters (war 7a). The estimates are obtained through solving the least-squares problems Wg arg min ||A,- V.wll and % arg min, YY, 7.2 where -1 denotes the Pnnormm We denote all estimated parameters w [k] and &- Fi..... r.]. In the second stage, we perform statistical independence test for the null hy- pothesis Ho : P(w, 7)- = P(w) using the estimated parameters @ and 5. If we accept Ho then we should con- sider Assumption 3.1 and 4.1 to hold. On the other hand, if we reject Ho then both assumptions are falsified jointly.
We measured performance using the falsification rate (proba- bility of falsification) and set the significance level 0.05 to control Type 1 errors. In the absence of unmeasured con- founding, the falsification rate reflects the Type I error rate and should remain below the significance level a = 0.05. Conversely, in the presence of unmeasured confounding. the falsification rate corresponds to the statistical power of the algorithms, and thus, a higher rate desirable. The code for reproducing our experiments is available at GitHub repository.'
For the statistical independence test in the second stage, we study the co-variability of W2) across all environments by analyzing the covariance matrix Σ- = Cov(w, We propose using the test statistic T = V Σ EEyy which is the Frobenius norm of the covariance matrix; crucially, this test statistic is always non-negative and T under Hy- The estimated test statistic becomes
K (a,s - -a.(Ges -75)
- 1
T(0, 7) K
pttspa://github... ckardkarr1 falsificat wtw.iiiaaaoonnnnoooooonneeeeee
i=1 1=1
6

---

Falsification of Unconfoundedness by Testing Independence of Causal Mechanisms
Fixed (V, d) (25,3)
Fxxe (f,, d) (100,
Fixed (K, di (10,1)
Fixed (K, 5) (50.1)
Fixed (K. NI (50,
1.0
Falufcation rata
0.8
0.8-
0.8
0.8-
0.5
6.6
0.0
0.0
0.0
0.-
0.4-
0.0
04-
20
80
20
80
100
150
200
50
100
150
200
10
Number of unvironmmnnss
Number of environments
Number
Number samples
Number of covariates
MNT (Linar))
(Pearson)
(Pearson)
Figure 1: Comparison of falsification rate when varying either the number of environment K, the number of samples per environment N, the number of observed covariates d. The error bars show the standard error over 250 repetitions.
No unmensured corfounder
Unmessurec confounder present
1.0
6,9-
5.8
0.6
0.4
6.3-
0.0
2:91
8/9
g(4
8/90
1100
(12)
gell gino g!s gins (27) (23)
Degree paymomeal mode Urmassured confounder
Number sarroles
HINT
Transporullll tast
HGI
(h)
Figure 2: (a): Comparison of falsification rate when differrnt mechanisms vary across the environment. The parameters on the x-axis correspond to those of the luta-generating process in (3). (b): Our algorithm's 's performance is evaluated using polynomial basis functions as feature representation. The falsification rate is plotted against polynomial degree, with the true data-gent settaageneratinggg process including polynomials up to degree 2. The black dotted line in both figures correspond to the chosen significance level a = 0.05. Average falsification rate and standard standard errors are reported over 250 repetitions. In the absence of unmeasured confounding, the falsification rate reflects the Type error rate and should remain below the significance level 0.05. Conversely, in the presence of unmeasured confounding, the falsification rate corresponds to the power of the test, and thus, higher rate is desirable.
6.1. Baselines
test (KCIT) (Zhang et al., 2011) with radial basis function kernel, which is better suited for nonlinear data. For HGIC, we encountered some issues with KCIT that required minor modifications to the original implementation used by Karls- son & Krijthe (2023). These issues and the differences between implementation and the original are discussed in detail in Appendix D.4.
We compare the proposed MINT algorithm to two baselines. The first, referred to as the transportability test, evaluates whether the independence Y IL 5 A, X holds, allowing for the joint falsification of Assumption 3.1 and the trans- portability condition y 1. X (Dahabreh et al., 2020b). detailed overview of transportability-boe -based falsification strategies is provided in AppendixA. The second baseline is the hierarchical graph independence constraint (HGIC) approach (Karlsson & Krijthe, 2023). This approach tests a conditional independence statement based on hierarchi- cal description of the data. Unlike the transportability test, HGIC remains valid even when transportability conditions are violated, as it relies on an independence of causal mech- anisms assumption similar to ours. This makes HGIC strong candidate for comparison.
6.2. Synthetic data
6.2.1. WHICH METHOD IS MOST EFFICIRNT?
In the first experiment, we aimed evaluate the efficiency of each method in well-specified linear setting (see Ap- pendix D. for more details data generation). To make our method well-specified to the underlying data gener- ating process, we used linear feature representations for MINT, and for the two basclines methods we used the par- tial Pearson correlation test which suitable for conditional independence testing with linear data. Additionally, follow- ing Karlsson & Krijthe (2023), tested HGIC using KCIT,
Since both baselines require selecting a conditional indepen- dence testing method, we evaluated them using either the Pearson partial correlation test, which suitable for linear data, or the non-parametric kernel conditional independence

---

Falsification of Unconfoundednes by Testing Independence of Causal Mechanisms
as it is also well-specified in this context.
unmessured confounder
Five confounders present
We evaluated the falsification rate of each method under an unmeasured confounder while varying the number of environments K, the number of samples per environment N, or the number of observed covariates d, keeping the other factors fixed. The results are shown in Figure When varying the number of environments K, our proposal MINT consistently outperformed HGIC. The transportabilitty test was most effective when K was small, though both MINT and HGIC showed higher falsification rates as K increased. HGIC performed better with the Pearson test than with KCIT, highlighting the advantage of a parametric test in a well-speciffed setting. Increasing the number of samples N improved falsification rates for all methods except HGIC with KCIT, although the gains were slower compared to increasing K. Finally, when varying d, HGIC with KCIT lost power the fastest, followed by MINT, while Pearson- based methods remained robust up to d = 10.
0.8
Palsitcattoo
0.4-
0.0
Number of observed confounders
Number observed confounders
-... HNT [0uadrand)
Tarsp Test (ROI))
Han HSC (KOT)
Figure 3: Comparison on the Twins semi-synthetic dataset. Average falsification rate and standard errors are reported over 250 repetitions; the black dotted line correspond to the chosen significance level a = 0.05.
6.2.3. MODEL SPECIFICATION & PERFORMANCE
Additionally, in Appendix D.5, we confirm that all methods controlled Type 1 error in the absence of an unmeasured confounder, with the falsification rate remaining below the significance level 0.05.
In the third experiment, we sampled data from process with polynomial basis function (see Appendix D.I for more details on the data-generating process). We examined how changing the specification of the working models e, (X) and (x)) in MINT affected its performance. The true polynomials in the data-generating process had a degree of 2, while MINT used a representation with polynomial basis functions with degrees ranging from 1 to 10. If the degree was set to 1, this introduced misspecification. For degrees of 2 or higher, the model was well-specified but became increasingly flexible as the polynomial degree increased.
6.2.2. WHAT ARE NECESSARY MECHANISM CHANGES TO DETECT CONFOUNDING?
In the second experiment, we validated the theory behind our proposed MINT algorithm by generating various types of independent mechanism changes across the environments, following the model in (3) (see Appendix D.2 for details on data generation). We also applied the baseline methods to the same data to provide further insights into the necessary conditions for them to serve valid falsification strategy.
As shown in Figure 2b, misspecified models led to an ele- vated false positive rate in the absence of unmeasured con- founding. However, once the models were well-specified, false positives dropped below the nominal level a 0.05 even as model flexibility increased. When an unmeasured confounder was present, MINT successfully detected it, though its power (i.e., true positive rate) declined with higher model flexibility. We observed, however, that this reduction in power could be mitigated by increasing the number of samples per environment.
The different parameters on the x-axis in Figure 2a represent which of the parameters in (3) are varied across different environments, while all other parameters are kept fixed. This done under both the absence and presence of unmeasured confounding. We observed that environmental changes in the parameters (a. (0) (X) (U) .u (,))), which influence either the treatment mechanism E[T X,5 s] or the distribution of the unmeasured confounder P(U S = s), were suffieient for MINT to falsify unconfoundedness. This observation supports the claim we proved in Theorem 4.5.
We also compared the transportability test and HGIC under both well-specified and misspecified settings. Using the Pearson partial correlation test on nonlinear data allowed us to assess their performance under misspecification. Similar to MINT, both exhibited higher false positive rates in the absence of unmeasured confounding when misspecified. Full results are provided in Table 2 in Appendix D.5.
Furthermore, both HGIC and the transportability test fal- sified under the same conditions when unmeasured con- founders were present. However, the transportability test showed a notable issue with false positives in the absence of unmeasured confounding. The most likely explanation is that these false positives result from mechanism changes that violate the transportability condition, a key assumption for applying the transportability test.
6.3. Twins data
In the final experiment, we used data from twin births in the USA between 1989-1991 (Almond et al., 2005) to construct a multi-environment observational dataset with a known
8

---

Falsification of Unconfoundedness by Testing Independence of Causal Mechanisms
causal structure. The environment corresponds to the birth state of each pair of twins. We generated treatment and outcome variables using the covariates from this dataset, providing a ground-truth causal structure to validate our methods while emulating realistic distributions with real- world covariates (see Appendix D.3 for details on dataset construction). The outcomes and treatment were modeled using a quadratic polynomial. and all methods were well- specified through either quadratic polynomial feature rep- resentation or KCIT for conditional independence testing.
ficiently expressive to capture the underlying relationships in the data. However, increasing the richness of the fea- ture representation introduces a trade-off: while it reduces the risk of misspecification, it can also decrease statistical power due to increased model complexity. This trade-off was evident our experiments (Figure 2). where increasing model complexity lead to a reduction in power of the test.
Hence, key future direction is to address the case where {v, o} are unknown and attempt to learn them from data. In this case, we have shown that it would be sufficient to learn them up to some permutation and clement-wise scal- ing. Alternatively, we could attempt to use more flexible (implicit) feature representations through the use of kernel methods (Schölkopf & Smola, 2002). Whereas further work is needed to adapt our theory to a kernelized algorithm, we provide sketch as starting point for such an approach in Appendix C.
We examined a scenario with five confounders, varying the number of observed covariates from one to five. When all five confounders were observed, no unmeasured con- founders remained; otherwise, some confounders were un- measured. As a control, we repeated the experiment while varying the number of observed confounders but ensuring no unmeasured confounders. The results, shown in Figure 3, indicate that MINT outperforms both the transportabilityy test and HGIC in terms of power. Additionally, when all five confounders were observed, MINT achieved the nominal falsification error below a 0.05.
8. Conclusion
We propose novel algorithmic ideas to directly exploit ob- served dependencies in causal mechanisms for falsification of the assumptions necessary for causal effect identification. Specifically, we propose a two-stage algorithm that can be applied to multi-ennvionment data. Although there are no universal solutions for addressing untestable assumptions in causal inference, we believe that our proposal has an important place in evaluating the necessary conditions to enable more trustworthy causal conclusions.
7. Discussion
Our falsification strategy is not silver bullet to detect un- measured confounding, As we have demonstrated, our pro- posed algorithm is a joint falsification test that assesses both the conditions necessary for causal identification and the assumption of independent causal mechanisms. Thus, the limit to how informative this falsification test be will depend the plausibility of the independent causal mechanism assumption.
Impact Statement
Causal inference plays crucial role in real-world decision- making, underpinning fields from medicine to public policy. While our work aims to enhance the reliability and safety of causal inference methods, it remains an early-stage develop- ment. We stress the importance of careful implementation in collaboration with domain experts, partieularly in high- stakes settings.
One reason our proposed algorithm performs well, espe- cially compared to HGIC with KCIT, could be because of the parametric nature of our approach. While parametric assumptions can be incorporated into both HGIC and the transportability test by selecting an appropriate conditional independence test, such as the Pearson test used in our experiment, our algorithm encodes these assumptions differ- ently. Specifically, it explicitly incorporates the parametric assumptions for both the treatment and outcome models. Interestingly, this approach aligns more closely with the common practice of specifying both models when estimat- ing treatment effects in observational studies.
Acknowledgements
Research reported in this work was was facilitated by the computational resources and support of the Delft AI Clus- ter (DAIC) at TU Delft. We also thank our anonymous reviewers for their helpful comments and input.
A drawback of relying on parametric assumptions for the treatment and outcome models is the increased Type error under misspecification. This happened to our algorithm when { }} were misspecified. So far, we have assumed these representations are known a priori. To mitigate the risk of misspecification, one strategy is to construct feature rep- resentations that apply broad set of transformations to the observed covariates. This ensures the representation is suf-
References
Almond, D., Chay, K. Y., and Lee, D. The costs of low birth weight. The Quarterly Journal of Economics, 120 00)103311108,, 2005.
Athey, S. and Imbens, G. W. The state of applied econo-

---

Falsification of Unconfoundednes by Testing Independence of Causal Mechanisms
metrics: Causality and policy evaluation. Journal of Economic perspectives, (1()33332, 2017.
cases and a promising alternative. In The 22nd interna- tional conference on artificial intelligence and statistics, pp. 3478-3486. PMLR, 2019.
Bang, H. and Robins, J. M. Doubly robust estimation in missing data and causal inference models. Biometrics, 61 (4):962-973, 000622997,,000 2005.
Faller, M.M. Vankadara, C., Mastakouri, A. A., Locatello, F., and Janzing, D. Self-compatibiltyy: Evaluating causal discovery without ground truth. In Dasgupta, S., Mandt, S., and Li, Y. (eds.), Proceedings of The 27th Interna- tional Conference on Artificial Intelligence and Statis- lics, volume 238 of Proceedings of Machine Learning Research, pp. 4132-4140, PMLR, 02-04 May 2024.
Burauel, F. Evaluating instrument validity using the prin- ciple of independent mechanisms. Journal of Machine Learning Research, 24(176):1-50 2023.
Cheroozuuovv V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 3111:C1-C63 2018.
Fisher, R. A. Statistical Methods for Research Workers. Oliver and Boyd, 1925.
Gelman, A. Data analysis using regression and multi- level/hierarchical models. Cambridge university press, 2007.
Colnet, B., Mayer, I., Chen, G., Dieng, A., Li, R., Varo- quaux, G., Vert, J.-P., Josse, J., and Yang, S. Causal inference methods for combining randomized trials and observational studies: review. Statistical science, 39(1): 165-191, 2024.
Goldstein, H., Browne, W., and Rasbash, J. Multilevel modelling of medical data. Statistics in medicine, (112: 3291-3315, 2002.
Cornfield, J., Haenszel, W., Hammond, E. C., Lilienfeld, A. M., Shimkin, M. B., and Wynder, E. L. Smoking and lung cancer: recent evidence and a discussion of some questions. Journal of the National Cancer institute, 22 (1):173-203, 1959.
Greenland, S., Pearl, J., and Robins, J. M. Confounding and collapsibility in causal inference. Statistical science, 14 (1):29-46, 1999.
Gretton, A., Herbrich, R., Smola, A., Bousquet, O., and Schölkopf, B. Kernel methods for measuring indepen- dence. Journal of Machine Learning Research, 6:2075- 2129, 2005.
Dahabreh, J., Robertson, S. E., Steingrimsson, J. A., Stu- art, E. A., and Hernan, M. A. Extending inferences from a randomized trial to a new target population. Statistics in medicine, (9(14):199) (999))999922119 2014, 2020a.
Grimmett, G. and Stirzaker, D. Probability and random processes. Oxford university press, 2020.
Dahabreh, I. J., Robins, M., and Hernán, M. A. Bench- marking observational methods by comparing random- ized trials and their emulations. Epidemiology, 31): 614-619, 2020b.
Guo, S., Tóth, V., Schölkopf, B.,, and Huszár, F. Causal de finetti: On the identification of invariant causal structure in exchangeable data, Advances in Neural Information Processing Systems, 36, 2024a.
De Bartolomeis, P., Abad, J., Donhauser, K., and Yang, F. Detecting critical treatment effect bias small subgroups. In Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence, pp. 943-99999 965. PMLR, 2024a.
Guo. S., Zhang, C., Mohan, K., Huszár, F., and Schölkopf, B. Do finetti: On causal effects for exchangeable data. arXiv preprint arXiv:2405. 18836, 2024b.
Heard, N. and Rubin-Delanchy, P. Choosing between methods of combining-valle -vauues.. Biometrika, 105(1):239- 246, 2018.
De Bartolomeis, P., Martinez, J. A., Donhauser, K., and Yang, F. Hidden yet quantifiable: A lower bound for confounding strength using randomized trials. In Interna- tional Conference on Artificial Intelligence and Statistics, pp. 1045-1053. PMLR, 2024b.
Hernan, M. and Robins, J. Causal Inference: What If. Chap- man & Hall/CRC Monographs on Statisties & Applied Probab. CRC Press, 2020.
Demirel, I., De Brouwer, E., Hussain, Z. M., Oberst, M., Philippakis, A. A., and Sontag, D. Benchmarking ob- servational studies with experimental data under right- censoring. In International Conference on Artificial Intel- ligence and Statistics, pp. 4285-4293, PMLR, 2024.
Hernán, M. A. and Robins, M. Estimating causal effects from epidemiological data. Journal of Epidemiology Community Health, 60(7):578 -56, 2006.
Huang, B., Zhang, K., Zhang, J., Ramsey, J., Sanchez- Romero, R., Glymour, C., and Schölkopf, B. Causal discovery from heerrgeneous/honstatonyy data. The
D' Amour, A. On multi-cause approaches to causal inference with unobserved counfounding: Two cautionary failure
10

---

Falsification of Unconfoundednes by Testing Independence of Causal Mechanisms
Journal of Machine Learning Research, 21(():3482-3544 2020.
Pearl, J. On the testability of causal models with latent and instrumental variables. In Proceedings of the Eleventh conference on Uncertainty in artificial intelligence, pp. 435 403,443 1995.
Hussain, Z., Shih, M.-C., Oberst, M., Demirel, I., and Son- tag. D. Falsification of internal and external validity in observational studies via conditional moment restrictions. In International Conference on Artificial Intelligence and Statistics, pp. 5869-5898. PMLR, 2023.
Perry, R., Von Kügelgen, J., and Schölkopf, B. Causal dis- covery in heterogeneous environments under the sparse mechanism shift hypothesis. Advances in Neural Infor- mation Processing Systems, 35:10904-1000 2022.
Hussain, Z. M., Oberst, M., Shih, M.-C., and Sontag, D. Fal- sification before extrapolation in causal effect estimation. Advances in Neural Information Processing Systems, 35: 6161-611 $161-6174, 2022.
Peters, J., Bühlmann, P., and Meinshausen, N. Causal in-
ference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical So- ciety Series B: Statistical Methodology, 7865):4771002, 2016.
Janzing. D. and Schölkopf, B. Detecting confounding in multivariate linear models via spectral analysis. Journal of Causal Inference, (1):201700133 2018.
Peters, J., Janzing, D., and Schölkopf, B. Elements of Causal Inference: Foundations and Learning Algorithms. MIT Press, 1st edition, 2017.
Janzing, D., Mooij. J., Zhang, K., Lemeire, J., Zscheis- chler, J., Daniušis, P., Steudel, B., and Schölkopf, B. informatiiongeommette approach to inferring causal direc- tions. Artificial Intelligence, 182:1-31, 2012.
Reddy, A. G. and Balasubramanian, V. N. Detecting and measuring confounding using causal mechanism shifts. Advances in Neural Information Processing Systems, 2024.
Karlsson, R. and Krijthe, J. Detecting hidden confound- ing in observational data using multiple environments. Advances in Neural Information Processing Systems, 36, 2023.
Riley, R. D., Lambert, P. C., and Abo-Zaid, G. Meta- analysis individual participant data: rationale, conduct, and reporting. Bmj. 340, 2010.
Karlsson, R., Creastä, S., and Krijthe, J. Putting causal identification to the test: Falsification using multi- environment data. In Causal Representation Learning Workshop at NeurlPS 2023, 2023.
Rubin, D. B. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of edu- cational Psychology, 66(5):688, 1974.
Leite, W. L., Jimenez, F, Kaya, Y., Stapleton, L. M., MacInnes, J. W., and Sandbach, R. An evaluation of weighting methods based on propensity scores to reduce selection bias in multilevel observational studies. Multi- variate behavioral research, 5063):265-284, 2015.
Schölkopf, B. and Smola, A. J. Learning with kernels: support vector machines, regularization, optimization, and beyond. 2002.
Schölkopf, B., Janzing, D., Peters, J., Sgouritsa, E., Zhang, K., and Mooij. J. On causal and anticausal learning. In Proceedings of the 29th International Coference Inter- national Conference Machine Learning, ICML'12, pp. 459-466, Madison, WI, USA, 2012. Omnipress. ISBN 078145031288
Lipsitch, M., Tchetgen, E. T., and Cohen, T. Negative controls: tool for detecting confounding and bias in ob- servational studies. Epidemiology, 21(3):383-385 2010.
Mameche, S., Kaltenpoth, D., and Vreeken, J. Learning causal models under independent changes. Advances in Neural Information Processing Systems, 36, 2024a.
Shah, R. D. and Peters, J. The hardness of conditional inde- pendence testing and the generalised covariance measure. The Annals of Statistics, 48(3):1514 1538, 2020.
Mameche, S., Vreeken, J., and Kaltenpoth, D. Identifying confounding from causal mechanism shifts. In Interna- tional Conference Artificial Intelligence and Statistics, pp. 4897-4905. PMLR, 2024b.
Tan, Z. A distributional approach for causal inference using propensity scores. Journal of the American Statistical Association, 101(476):6619-6633 2006.
Manski, C. F. Partial identification of probability distribu- tions. Springer Science & Business Media, 2003.
Tippett, L. H. C. The methods of statistics. 1931.
VanderWeele, T. J. and Ding, P. Sensitivity analysis in observational research: introducing the e-value. Annals of internal medicine, 167(4):268-2)) 274, 2017.
Mccullagh, and Nelder, Generalized linear models. CRC press, 1989.

---

Falsification of Unconfoundedness by Testing Independence of Causal Mechanisms
Wager, S. and Athey, S. Estimation and inference of hetero- geneous treatment effects using random forests. Journal of the American Statistical Association, 113(523):1228- 1242, 2018.
Wang. Y. and Blei, D. M. The blessings of multiple causes. Journal of the American Statistical Association, 114(528):
1574-1596, 2019.
Zhang, K., Peters, J., Janzing, D., and Schölkopf, Kernel- based conditional independence test and application in causal discovery. In Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence. pp. 804-8 813, 2011.
Zheng. Y., Huang, B., Chen, W., Ramsey, J., Gong, M., Cai, R., Shimizu, S., Spirtes, P., and Zhang, K. Causal-learn: Causal discovery in python. Journal of Machine Learning Research, 25(60):1-8, 2024.
12

---

Falsification of Unconfoundednes by Testing Independence of Causal Mechanisms
A. Falsification with transportability conditions
common way of using data from multiple environments to falsify the validity of Assumption 3.1 is to assume a transportabilityy condition that relates the different environments each other. One of the most common way of formalizing the transportability condition is as follows.
Assumption A.1 (Conditional exchangeability between environments). We assume for all a E A, Y²
1 S X.
Other variations of the transportability condition is to assume conditional mean exchangeability Eey X, S = =]= B X] or that an effect measure such as the conditional average treatment effect TE. [ - yo x,5 = s) = EF - yo x] is transportable (Colnet et al., 2024).
well-known that Assumption 3.1 and A.1 together have a testable implication in the law of the observed data, see e.g. Dahabreh et al. (2020b). More specifically, Assumption 3.1 and A. together imply that the following conditional independence must be true
YLS X,A.
(4)
Testing (4) can be done with any suitable conditional independence test and efficient procedures also exists for testing implications from the other variations of the transportabilityy condition, see e.g. Hussain et al. (2023). However, the underlying premise is always the same: if one would conclude that (4) does not hold, then this could be due either a violation of Assumption 3.1 or A.I. Thus, if one believes that Assumption A.1 must hold yet observes (4) to be false, that means that Assumption 3.1 must be violated in at least one of the environments. This argument becomes particularly strong if treatment has been randomized in one of the environments since any difference between the environments is more likely to be explained by an unmeasured confounder in the remaining environments with observational data. However, Assumption A.1 itself can be controversial as it is also untestable. More specifically, it would be violated if there are unmeasured so-called effect modifiers which are covariates that differ distribution between environments and modulate treatment heterogeneity. Effect modifiers are distinet from confounders as effect modifiers only need to be cause of the outcome of interest. Thus, confounders can be effect modifiers but not vice versa, meaning that we often might expect to have unmeasured effect modifiers present even where are unmeasured confounders.
The primary distinction between our falsification strategy and a transportabilitty -based falsification strategy lies in the assumption our strategy relies on: instead of using Assumption A.I, our strategy employs Assumption 4. to derive an alternative jointly testable implication. This comparison also highlights their underlying similarity: both strategies aim to combine two untestable assumptions to generate a testable implication, enabling the joint falsification of these otherwise untestable assumptions.
B. Proofs
B.1. Proof of Theorem 4.3
Proof. Using the conditions from Assumption 3.1 and that hx) = =1 T(X, A = a) is a correctly specified model for B| A, X, = s), we can write
TTo(X,A= a)= = E|Y X, = s] =]
=E|y" X = a, S.S=1 s]
ya 1 4 1 = A a= y"=y
= =E(Y| X,A = S,S
Because we assumed 6(X,= = a) = CO(X, = a) for some invertible matrix C (Assumption 4.2), it follows for s I,... K that 7. = (C. Furthermore, using that (X) = W (xX A = a) is a correctly specified model for EjA X. s) and ((X,A)= Div(X, A) for some invertible matrix D (again, Assumption 4.2), it follows using similar arguments that w. = =(D-1) TTa.) Ta, for 1,.... K. To conclude the proof, using Assumption 4.1 which states that there exists a distribution P(a, 3) = P(3), we observe that (w,, Ys) are distributed according to distribution defined as P(w, 7) == P((C-I)Ta o, (D 1)T B).. It is well-known that if a, and B3 are independent random variables, then their transformations (C-1)Ta, a, and (D 10))T 3, are also independent; see Grimmett & Stirzaker (2020, Chapter 4.2). Thus, we have P(a, B) = P(a)) = P(w, 7) = P(r)
13

---

Falsification of Unconfoundednes by Testing Independence of Causal Mechanisms
B.2. Proof of Lemma 4.4
Before we can prove the lemma, we need the following auxiliary result.
Lemma B.1. Consider two Normal probability densities [1(x)
V/na??
exp
(-z03 (a 11)))) and f2(Ø)
exp (- (ar 12)²) with 02 We then have that the product the densities fi f1())) f2(1) is proportional
2
1o Normal density 1 exp (-
1 (x
m12)² (1²))²²²
where
V2n032
2012
B1li + u207 M1
0.02 of + 02
and
i
of + +
Proof. Note that
(I 11)2
(x: 122 2
f(()) f2(x)=
exp
2T0102
o
02
where the expression inside the exponential function can be written
Q (0 -0²) x² 2 (o103 (20f) 1903 u220
0i02
x2 2 (N103 t
(+)+
(0)+0))
(r]+a3)
x (
(m10)+2207) (ofto3)
p (o)+
+
(of+03)
of
oi
(T++o))
oftt))
As the second term on the last line independent of r, we finish the proof by observing that fi(x) f2(x) is up to some constant proportional to
(mloî+ (07+o2)
1
exp
V 2T
77t07
(ap- -3
Next, we proceed with the proof of lemma 4.4.
Proof. To simplify notation, we will drop the subscript s for all parameters. We start with E|A X x, S s). which can be written
E[A X,S = s| = Eja (0) + a (x) + (U) + EA X
(0) + (x)) E X.S [] +] X.
(4) 0 ()) + (x) + (1)) (U)
00
+ aa (1l0)11071
ax
X
where in (a) we use that U (S 8) meaning that EEU
X, S =| S (1())
14

---

Falsification of Unconfoundedness by Testing Independence of Causal Mechanisms
Next, we continue with B| A,S. s]. which can be expressed
E B g(x) x + GLUTU A (B((A) 3(Ax) + ((A0)T) + her X,A,S =
= 3 + g(x) + (3l4) + x((xx)x)
(5)
+ +30) + AB((1)) EiU X, S = +|+ X, = =
To evaluate the conditional expectation BUU X, A,S = s), we use Bayes rule to rewrite the probability density function
S(U X,A,S) = f(A SUU, X. f(A |X,S)
Firstly, note that [(U X) = S(U S) follows from that X 1. U (S=s), The density S(U S) is corresponds to the density of (( (a(())?). Secondly, we can express f(A X, U, differently by exploiting that = a + ax) + (U) + E as follows,
f(A = a xx= x, U- u,
= f(EA= = a- (0) (xx) - (0)) L0luISS s== s)
(b)
2((o()))2 (a - a (0) a (2) x/ - a u1))
1
exp
v 2T (o(A)))
1
(0) (all)) (a -a(d))) - 2
(o((1) exp
c(a) atty
where (b) follows from that A (S= =) ~ ((, (o(4)))). In (c) we reshuffle terms to explicitly break out u inside the exponential function. From inspecting the expression on the last line, we note that it looks like an unnormalized probability density function w.r.t u for Normal distribution. we rescale f(A
X = x, u, S = 8) by ate we obtain a0 - (x))x)
(4)
an probability density function w.r.t w for the Normal distribution N
This
ater
observation together with the results from lemma B. allows us to show that the product of densities f(A X, S)U, X, S) also corresponds to an unnormalized, scaled probability density function of a Normal distribution with mean equal to
(a( (0)) - (x)) (o(())2 (U)) (a(A)) alv)
(6)
(o(0))2 +
a(a)
Since we can re-normalize | X,A| X,S) with 1/f(A X)S), we have that the conditional expectation
= S =) = X, K.S= 8) du
E
x, A,A,S = s)
f(A x,S=s)
equal (6).
Plugging (6) back into (5), have that
E|| X,A,S = s) = 30)) + ((x)x A (B(A) -((x)x)
(arl)))- (A (0 -x(())) (o110)22 + ml0
((())
+ +(100) + Ag(10))
axoo
(o(1))2 +
(g(a)) ((1))
15

---

Falsification of Unconfoundedness by Testing Independence of Causal Mechanisms
To conclude the proof, we simplify the above expression to the form EE
A, = AX,, A2IT
where
- -B(U)
(0(0))
10
((14)
a00)
glo)
3(x)
-B(
alu)
7
B(a)
+
B(0) (a 12
(((U))
a (o alU)
(1))
( (((a) acoy
2)
g(
(1))
-B(AV)
( att) ((A0) x(t)
and [all1)]+ + (x1o)
((()) 2
B.3. Proof of Theorem 4.5
Before proving the theorem, we present the following result that we will need later.
Lemma B.2. Under the data generating process in (3) and (all)) 0}. we have that E X, A, x],
and E X, A, s) 7s [1, X, A, AX, A2TT where
3!00
(
us
(
(0)
ag
((
o (AU (C)
02
a2 (X)
Ys
On the other hand, if instead of (U 0} we have that (5(0) = 0, ((U)) = 0} then
x(x)
(U)
(U)
+ Os
3(4)
02
0 (X)
7/8
S(AX)
Proof. For the first case with (U) 0}. we have that E[A X,S s] E(a. (0) +x{
(X) X+EA S s) a. +a,
(X) X.
Further, we can show
B| A, s]
B ((x) AX +
AU X, s]
- + ((x) X x ((4))) A+ 8( (A)) AX + 30 11
(8) + (U)) (U) A
where the second equality from that U 1 X A,S holds in (3) if (V) 0). This concludes the first case.
For the second case with {B(U)) 0, (0, 0}. it follows from the above equation that
E|Y X, s] 5 + x X + ((4) A+ (((x) AX
Meanwhile, for the treatment mechanism we now instead have that
E[A X,S = s] [. + a (X) X + B(a U S s]
a + a
(x) x a 11
(U)
Now, we can proceed with the proof of the Theorem 4.5.
16

---

Falsification of Unconfoundednes by Testing Independence of Causal Mechanisms
A, A, AX, A We will show that regardless % (X, A) for some (Wa, (a) and only if and only if U is confounder will wi I % under some conditions on the parameters (a!", Ozs Ots 11s ).
Proof. Throughout the proof, we define d(X) X and ( [1, of the presence of the confounder U, we can write E[A X, S s) = W3 (XX) and E X, A,
(X) (U)
Note that under Assumption 4.1, it follows that there exists a distribution (wi, ~ P(w, since the parameters (Wa, (4) directly dependent on the parameters (a., 83)) ~ P(a, B). for s....... K. To determine if p(w. = P(w) we have to determine whether Wa and % both depend on the same parameters from the underlying data-generating process and under what conditions this can create statistical dependencies between Wa and 7/--
No unmeasured confounding present First, consider the condition that U is not confounder. There are three cases for which this happens: (1) we have {a(C) 0}. (2) we have { 0, B(AU) = 0}, and (3) we have both {a = 0} and (3(0) = 0, (LT) = 0}. For case (1) and (2), it follows immediately from lemma B.2 that wa and % have no shared
parameters. For the final case (3), it is easy to see that Wa [a. a x and 7 [ p(x), 2/4)) 3(AX), 0] where,
again, there are no shared parameters between wa and 7s- Thus, we can conclude that under all of the cases when U is not confounder, (Wg, Ys) have no shared parameter and thus w, 1 7'--
Unmeasured confounding present Next, consider the condition that U is confounder. It follows from lemma 4.4 that if Ms (10) Thus, if any of the parameters (a."), .i aa Mus ) vary across different environments, which happens if we assume there exist non-degenerate
U is confounder, then both wa and 7/2 depend on the parameters (as (0) , (X) (E)) (X0 (U) distribution for at least of one these parameters, it follows that wis L 7s- This concludes the proof.
as
C. Extension to implicit feature representations
In this section, we provide sketch for replacing the features representation {Ø, }h in our falsification algorithm with an implicit feature representation through the use of kernel methods (Schölkopf & Smola, 2002). More specifically, we let ((x) be the implicit feature representation whose inner product is given by the kernel k(2, x) = (Ø(x), ((x)) defined on X' with corresponding RKHS H and, similarly, let (22 a) be the implicit feature representation with inner product given by h ((ar,a) (Ir a)) = (1, a), ((,,,)) o))g defined on X @ A with corresponding RKHS g.
With some minor modifications, we can run our falsification algorithm without having to compute x(x) and (z). To illustrate this, we impose the restriction that we use the same number of observations from each enviromment, denoted with n. We shall focus estimators given by W, = arg min,, | F,w _ - >1||W|la and 9. arg min...
I|Y, V y + dallyl2 for some constants Å1, A2 > O. The above optimization problems correspond to kernel ridge regression problem, for which it is well-known that the estimators have closed form solution, namely
a. (K, + ndi (n) A, and 9. (H, + -
(X.,, where K, = I((,,, X and H, M(X,, (X.,, )) are the Gram matrices. The above estimators are however not always be computable since they can, depending on the choice of kernel, be diminie-dimesiional. This also makes it infeasible directly compute the covariance matrix Σ = Cov(w, 7). However, it is luckily still possible to compute the Frobenius norm [2 Using results from lemma 1 in Gretton et al. (2005), can rewrite ||i|| E Ep(,,))w" ( w yT > + EP(w) [w E/ 7] 2k)(,,)) (Er Ww T "w]]Ep(>> T >]]. From this equality, it follows that we can compute ||20||2 by inspecting the inner products W. Ber and 77 7er for all 8, E KY. Interestingly, these inner products can be computed as follows
6. T = AT + +nAr4,)-1(K + -
Ror = (H, + InAAT))) + +nAIT)--
which means that ||V||| can be computed and we can in principle statistically test for independence of w and y with some implicit feature representations {Ø, }) For future work, remains unknown how to best implement this algorithm and investigate how our theory needs to be modified for it.
17

---

Falsification of Unconfoundednes by Testing Independence of Causal Mechanisms
D. Experimental details
D.I. Sampling from data-generating process with polynomial basis functions
We generated observational datasets as follows. For each environment & = -,..... K, we obtain = ,,..., .N individuals by first sampling set of d-dimensional covariates according to X; ~ N(us (X) - =2²z) with the mean μ15 (x) E I ~ N(, 5 TT)) where La was d x d identity matrix and the covariance matrix M of shape d had its diagonal elements set to 2 and its off-diagonal elements set to 0.1. Thereafter, we sampled the treatment T; and outcome Y, according to (1) with the features representations 1/( (X) = [1, X1,... Xd, XP..... X2) and @Xx, A) [1, XI..... Xd, XP:.... d: A] being polynomial basis functions of degree The noise variables εл and Ey were mean-zero zero Normal distributed with their standard deviation set to 0.5. Each element in a, where sampled uniformly from the set {-1,1} while each element in 3. was set to 1, except for the elements in a, corresponding to the intercepts which were sampled according to (0, 1). Only the intercept elements were resampled for each new environment, whereas the remaining coefficients in (as. B.) where kept fixed for all environments. When introducing an unmeasured confounder, we additionally sampled an one-dimensional covariate U: ~ N(us (0) .2 with its mean وuر ( ~ N(,, 1). Then, we added U, directly to 7; and Y, For simplicity, we let each environment have the same number of samples N ny = nkK even though all methods also work the number of samples per enviromment differ.
D.2. Sampling from data-generating process in (3)
We sampled a multi-environment dataset with K = 250 environments and 1000 samples per environment according to the data- generatinn process described in Section 4.3:
A = ag (X) + a U +EA
Y = 3._ (xx A = a) + (3!U) + ag!(uU)) U tey
where we let (x(X) = T) and o(x, A)= = X1 A, AX the noise variables were sampled according to E ~ N(0, )) and Ey ~ (0, }). and the covariates were sampled according X ~ N(ux 1) and U ~ N NNw,,), 1). By default, we set the parameters as a2 = [ JT 3. = 1 2: 3. 11x 1, and pay = 1. To impose the presence of an unmeasured confounder, we would set the remaining parameters (a _ B
a(4))) to 1 and otherwise set them to To introduce changes in the parameters among environments, we would select one of the parameter values and overrule the above default values by sampling from uniformly from the range [0.1, 3.0] for each new environment.
D.3. Generating Twins semi-synthetic dataset
We use data from twin births in the USA between 1989-1991 (Almond et al., 2005) to construct an multi-environment observational dataset with known causal structure. The dataset contains 46 covariates related pregnancy, birth, and parents. As many covariates are highly imbalanced and have low variance, we select subset of the covariates for generating the semi-synthetic dataset.
As the environment label we used the birth state and as covariates we used the following variables (variable names from the dataset documentation are shown in parenthesis): birth month (birmon), father's 's age (dfageq), number of live births before twins (dlivord min), total number of births before twins (dtotord min), gestation type (gestat10), mom's 's age (mager8), mom's education (meduc6), mom's place of birth (mplbir), and number of prenatal visits (nprevistq).
The treatment and outcome were generated using the same procedure described in Section D. with one key difference: the synthetic covariates were replaced by real-world covariates from the Twins dataset. Prior to generating the treatment and outcome, the covariates were standardized. Each time a semi-synthetic dataset was created using the Twins dataset covariates, five of the chosen covariates were randomly selected as the confounders. A polynomial degree of p = 2 was consistently used throughout all experiments.
D.4. Kernel conditional independence testing with the HGIC approach
When using the original implementation of the hierarchical graph independence constraint (HGIC) approach described in Karlsson & Krijthe (2023) as a baseline in our experiments, we noted that their implementation sometimes would not be properly calibrated (i.e., elevated Type error above a 0.05). For this reason, we introduced some modifications to their
18

---

Falsification of Unconfoundednes by Testing Independence of Causal Mechanisms
Table I: Comparison of the new and old HGIC implementation using the data-generating process with polynomial basis
functions. The average falsification rate and standard error (in parenthesis) is reported from 250 repetitions.
Method
No unmeasured confounder
Unmeasured confounder present
Modified HGIC implementation
0.04 (.01)
0.88 (.02)
Original HGIC implementation
0.28 (.03)
0.80 (.03)
method that resolved this issue. Note that the modification we describe below were only necessary when using HGIC with the kernel conditional independence test, and not the Pearson conditional independence test used in some other experiments.
The elevated Type errors was caused by that HGIC combines p- p-values from multiple independence tests using Fisher's method, which employs the test statistic T Σx log pı: where ps where are the p-values from the tests (Fisher, 1925). This modification allowed HGIC falsification test's power. However, noticed in own simulations that poorly calibrated conditional independence test can cause the combination of p-values to amplify type I errors. So to address this issue, we made two modifications to the original HGIC implementation.
all observations in the multi-environment dataset and was observed to help increase the
First, we improved calibration by switching to permutation based calibration, replacing the original Gamma distribution approximation that is also commonly used for KCIT. Furthermore, we adopted the KCIT implementation from the causal- learn Python package (Zheng al., 2024) which allowed for optimizing the kernel width hyperparameter in the test using Gaussian process regression.
Second, although the above changes improved KCIT's calibration, Fisher's method still sometimes amplified type errors. To mitigate this, we replaced it with Tippett's 's method, which uses T = min Pk as the test statistic (Tippett, 1931). Like Fisher's method, Tippett's method emphasizes the smallest p-values (Heard & Rubin-Delanchy, 2018) but we found Tippett's method to be more conservative under the null. Testing on a simple conditional independence scenario confirmed that Tippett's s method worked better with KCIT, while retaining the benefits of increased power in combining p-values.
To illustrate the difference between modified implementation and the original implementation described in Karlsson & Krijthe (2023), we include experiment using the tettagennrating process with polynomial basis functions described in Appendix D. We used K = 100 environments with N = 50 samples per environment and d = 1 observed confounder, and set the polynomial degree to 2. Here, both implementations combine 25 p-values. As the results in Table I show, our implementation achieved a better Type 1 error while retaining similar power to the original implementation.
D.5. Additional experiments
We have included three additional experiments in this section to complement the experiments in the main paper.
First, repeating the same setup as in the experiment presented in Section 6.2. 1, we include results that confirmed that all methods have controlled Type errors in the well-specified setting. These results are shown in Figure 4.
Secondly, we conducted an ablation study to highlight the importance of using bootstrapping on top of the permutation-basee based test in procedure. We repeated the same setup as in Section 6.2.3, but implemented permutation- based testing without bootstrapping. As shown in Figure 5, bootstrapping is crucial for maintaining Type I errors below 0.05, even with an increased sample size.
Lastly, we compared all methods under misspecification across several scenarios in the data- data-generating process described in Appendix D. These scenarios included the presence or absence of an unmeasured confounder, whether transportability holds by sampling the intercept term in 8, from (0, 1) across environments, and whether the underlying data-generating process (DGP) was linear (p- 1) or nonlinear (p = 3) The results in Table 2 show that misspecification led to elevated Type 1 errors (falsifying without an unmeasured confounder) for all methods. Additionally, transportability violations caused higher Type errors for the transportability test, while our proposed algorithm remained unaffected.
19

---

Falsification of Unconfoundedness by Testing Independence of Causal Mechanisms
Fixed (N, d25,25
Fixed (N, d)0(100.
Fixed (K. (3000
Fixed di d50050
Fixed (K. -K0))(50222
1.0
1.0
Falsiictioo rate
0.8-
0.6-
0.4
40
60
100
150
50
environments
environments
Number samples
Number samp
covaristes
-.- NINT (Linnar)
-.-
Transp. teet (Pwarsnn)
HGG. (hwanon)
HIgHt
HGG (KEIT)
Figure 4: Same experiment as in Figure but with no unmeasured confounding being present. Comparison falsification rate when varying either the number of environment K, the number of samples per environment N, or the number of observed covariates d. The error bars show the standard error over 250 repetitions.
10-
1.0
0.9
0.9
0.8
0.8
0.7
0.7
Sisikcation
Galtttattoon
0.0
Degree of polynomial model
Degrec polynomial model
Unmeasuree confounder
Number of samples
Unmeasured canfounder Number of samples
(a) With bootstrapping
(b) Without bootstrapping
Figure 5: An ablation study showing the falsification rate our proposed algorithm using permutation-bast testing with bootstrapping versus without bootstrapping. We resample 1000 times when using bootstrapping. The error bars show the standard error over 250 repetitions. The black dotted lines correspond the chosen significance level a 0.05.
Table 2: Comparison of different approaches under various scenarios with K = 100 environments and N = 100 samples per environment. The average falsification rate and standard error (in parenthesis) is reported from 250 repetitions.
No unmeasured confounder
Unmeasured confounder present
Transportability
Holds
Violated
Holds
Violated
DGP
Cubic
Linear
Cubic
Linear
Cubic
Linear
Cubic
Linear
MINT (Linear)
0.62 (.03)
0.01 (.01)
0.65 (.03)
0.05 (.01)
0.60 (.03)
1.00 (.00)
0.53 (.03)
1.00 (.00)
MINT (Cubie)
0.00 (.00)
0.02 (.01)
0.04 (.01)
0.06 (.01)
1.00 (.00)
1.00 (.00)
1.00 (.00)
1.00 (.00)
Transp. test (Pearson)
0.69 (.03)
0.04 (.01)
0.68 (.03)
0.81 (.02)
0.65 (.03)
0.75 (.03)
0.71 (.03)
0.86 (.02)
Transp. test (KCIT)
0.05 (.01)
0.07 (.02)
0.38 (.03)
0.43 (.03)
0.16 (.02)
0.24 (.03)
0.34 (.03)
0.42 (.03)
HGIC (Pearson)
0.66 (.03)
0.03 (:01)
0.58 (.03)
0.04 (.01)
0.44 (.03)
1.00 (.00)
0.43 (.03)
1.00 (.00)
HGIC (KCIT)
0.02 (.01)
0.02 (.01)
0.11 (.02)
0.02 (.01)
0.76 (.03)
0.99 (.01)
0.35 (.03)
0.70 (:03)
20


Reviews:
[{"review_id": "67Tli4T52p", "reviewer": "Reviewer_mtnu", "summary": "This manuscript presents an algorithm for falsifying the assumption of no unmeasured confounding in a setting of observational data from multiple environments. To this end, the authors employ the Rubin potential outcome causal model, assume positivity, consistency and no unmeasured confounding; subsequently they model the functional relationships between (a) the confounders and the treatment and (b) treatment, confounders and the outcome. Finally, they introduce the key assumption that the parameters associated with functions (a) and (b) are independent from each other for all environments. The main theoretical result of the paper follows: given all the aforementioned assumptions, the learned parameters associated with (a) and (b) are independent. Then a Gaussian linear model is assumed which makes it possible to equate the dependence of the learned parameters with unobserved confounding. The paper concludes with a series of experiments (3 on synthetic data and one on real data) comparing the proposed approach in terms of efficiency (falsification rates) to two benchmarks.\n\n\n\n##### After rebuttal #####\n\n\nI would like to thank the authors for their rebuttal.\n\n\nI acknowledge the algorithmic contribution of the paper. The authors' response has shed a little more light on the difference to the theoretical results in [KK23], but a closer examination and establishing a formal relationship between them (showing whether the difference only stems from switching paradigms from causal DAGs to potential outcomes) could improve the paper. All in all, I am willing to raise my rating by a notch.\n\n\nMy current understanding is that there is no clear relationship between the theoretical results (the algorithmic ones differ of course) in [KK23] and this submission, but I still have a feeling that they are related and the relationship could be formalized by specifying e.g. additional conditions for some sort of equivalence.\n\nThe lack of conditional independence testing is a clear difference, and maybe the biggest contribution of the manuscript.\n\nI would like to clarify that I am not in any way associated with the authors of (Wang, Blei, 2018) or of any other follow-up paper I mentioned, nor have ever I published in the post 2018 “deconfounder” line of research. I simply thought the high level intuition “if the model (or causal mechanisms) do not factorize perfectly, this might be because of unobserved confounding” is something the current submission and (Wang, Blei, 2018) have in common and was wondering whether the authors had any thoughts on this. I think that citing any of the works only makes sense if accompanied by a detailed discussion.\n\nI acknowledge the algorithmic contribution of the paper. The theoretical discussion has shed a little more light on the difference to [KK23], but a closer examination of the results could improve the paper. All in all, I am willing to raise my rating by a notch.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and for spotting a number of typos which now have been addressed. We have provided answers and proposed changes related to your questions below. \n\n# 1: Connection between our theory and that of of Karlsson and Krijthe (2023)\nWhile the possibility of testing for unconfoundedness may be less surprising in light of [KK23], it is reassuring that we obtain similar identifiability guarantees under different assumptions and proof techniques. Our problem formulation not only leads to a novel test that directly targets the dependence between mechanisms but also introduces different conditions under which the results hold. In this context, we would kindly ask you to also read our reply to reviewer sWrJ, where we emphasize a key conceptual difference between our approach and that of [KK23]. Specifically, our framework *directly* tests the independence of causal mechanisms (ICM), necessitating a new theoretical treatment of the problem.\n\nWe now go over two key differences in the assumptions between our theory and [KK23] to clarify why our findings do not trivially follow from [KK23], even under a linear-Gaussian model:\n1. Most importantly, there is a difference in the formulation of the ICM assumption. [KK23] assumes that, in the absence of unmeasured confounding, variations in the conditional distributions $P(A\\mid X,S=s)$, $P(Y\\mid X,A,S=s)$ and $P(X\\mid S=s)$ are independent of each other. In contrast, our formulation in Assumption 4.1 only requires changes in $P(A\\mid X, S=s)$ and $P(Y^a\\mid X, S=s)$ to be unrelated, without making any assumptions about $P(X\\mid S=s)$. As [KK23] highlights in their Section 4.1 (\"Influence of assumptions\"), this additional condition on $P(X\\mid S=s)$ is necessary for their method to work. Meanwhile, our approach achieves the same falsification guarantees without requiring this additional condition because the results in Lemma 4.3 holds for any $P(X\\mid S=s)$.\n2. Also, [KK23] uses a constraint-based causal discovery approach relying on assuming the faithfulness and causal Markov property. In contrast, by instead leveraging functional assumptions, our approach does not require these assumptions. \n\nBased on the above discussion, and in addition to the changes mentioned in our response to reviewer sWrJ to clarify our contributions relative to [KK23], we will add a paragraph in Section 4 of our manuscript highlighting the key differences in assumptions between [KK23] and our work.\n\n# 2: Relationship to de-confounding literature\nWe thank the reviewer for highlighting the work of Wang and Blei (2018) and the surrounding discussion on it. As suggested, we will add references to these papers in our manuscript.\n\nWe are aware of this line of work, but we think the setting and motivation considered by these papers do not reflect our work. First, the argument used in Wang and Blei does not apply to our setting: we do not have multiple observed causes, but rather multiple environments. Second, they focus on identifiability about the interventional distribution under unmeasured confounding. In contrast, our goal is slightly less ambitious, we just want to determine whether there is an unmeasured confounder in the first place. \n\nThat said, it would be interesting to explore if there is a similarity between the factorization exploited by Wang and Blei and the ICM assumption, which also assumes a type of factorization. This could help us understand if there are implications for identifying interventional distributions in our setting. Here, the key question becomes what the access to data from multiple environments will add and whether that will allow for identifiability. \n\n# Do the environments facilitate reasoning about statistical efficiency?\nThe number of environments plays a crucial role in the success of falsification. As a thought experiment, consider a scenario where $(\\omega_s,\\gamma_s)$ are directly observed, allowing us to bypass estimating them in the first stage of our algorithm. In this case, the number of environments effectively determines the sample size available for testing independence between these two random variables.\n\nIf $\\omega_s$ and $\\gamma_s$ are only weakly dependent – such as when unmeasured confounding is weak – detecting their dependence would require a large number of environments. Conversely, if the confounding is strong, fewer environments may suffice to reveal dependence in these model parameters. \n\nOnce we add back the fact that we need to estimate $\\omega_s$ and $\\gamma_s$, this introduces additional sampling variance into the independence test. As a result, with fewer samples per environment, detecting a dependence becomes more challenging, and a larger number of environments will likely be required to achieve a high statistical power of our test. This is also what we observe in our experiments shown in Figure 1: decreasing number of environments or the sample size leads to lower falsification rate.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I read the three proofs in Appendix B (Theorem 4.2, Lemma 4.3, Theorem 4.4). I did not check the math details in the proof of Lemma 4.3, but it seems straightforward (multiplying two normal densities and plugging in to the assumed model). I also skipped the proof of Lemma B.2.\n\n\nThe proofs appear to be OK.", "experimental_designs_or_analyses": "The experimental design presented in the main body of the paper seems OK (Chapter 6; two synthetic experiments and a real-data one). I did not check the details in Appendix D.", "relation_to_broader_scientific_literature": "The paper’s theoretical results are very similar to (Karlsson, Krijthe, 2023), and while it is cited, I feel that a more thorough discussion of the overlap (especially concerning theory) would improve the manuscripts’ clarity.\n\n\nA review of de-confounding, (e.g. Louizos et al. 2017), dealing with ignorability or propensity scores is missing, but this is not fatal and can be explained with limited space.", "essential_references_not_discussed": "The paper’s theoretical results are very similar to (Karlsson, Krijthe, 2023), and while it is cited, I feel that a more thorough discussion of the overlap (especially concerning theory) would improve the manuscripts’ clarity.", "other_strengths_and_weaknesses": "Strengths\n\n\nThe paper tackles an important problem of testing the existence of unmeasured confounding.\nThe main contribution of the paper is a performance improvement over a similar method (Karlsson, Krijthe, 2023) on synthetic and real data.\n\n\n\nWeaknesses\n\n\nThe main weakness of the paper is its limited novelty. The theoretical results mirror those of (Karlsson, Krijthe, 2023) (Theorem 1 and Theorem 2). The main difference is that the current paper adopts the potential outcome framework instead of a Pearlian graphical model (for Thm 4.2) and a Gaussian linear model (Lemma 4.3, Thm 4.4).\n\n\nThe paper would be easier to follow if a discussion of (Karlsson, Krijthe, 2023) were held along with enunciating the current paper’s relation to it in terms of theory. It would also improve clarity if the paper’s focus on experimental improvement over (Karlsson, Krijthe, 2023) was stated.\n\n\nI feel that the role of environments is not discussed clearly enough. It seems to me that the proposed method beats (Karlsson, Krijthe, 2023) more clearly for a small number of environments (Figure 1). Do the environments facilitate reasoning about statistical efficiency?\n\n\nIt would be interesting to compare the paper with another approach for unmeasured confounding modelling, namely (Wang, Blei, The Blessings of Multiple Causes, 2018). The paper caused a lot of response leading to a number of limiting results for unmeasured confounding modelling (D’Amour 2018 Comment: Reflections on the Deconfounder; D'Amour 2019 On Multi-Cause Causal Inference with Unobserved Confounding:\nCounterexamples, Impossibility, and Alternatives; Ogburn et al. 2020 Counterexamples to \"The Blessings of Multiple Causes\" by\nWang and Blei). The underlying intuition of (Wang, Blei, 2018) is similar to the one presented here: if the model for the confounded treatment-response factorizes into independent parts, there has to be (multiple) unmeasured confounding. Do any of the model limitations apply here? Do the environments play a role?", "other_comments_or_suggestions": "Typos:\n\n\nl. 112 refereed -> referred\n\n\nl. 162 accounted -> accounted for\n\n\nl. 324 refereed -> referred", "questions_for_authors": "1. Given the equivalence of the Pearl model and the potential outcome model (Pearl 2009), does Theorem 4.2 follow from Theorem 1 in (Karlsson, Krijthe, 2023)? Under the assumption of a Gaussian linear model, do Lemma 4.3 and Thm 4.4 follow from Theorems 1 and 2 in?\n\n\n2. What is the relation of the unmeasured confounding falsification to (Wang, Blei, 2018) and the following discussion?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gilKQjIWNB", "reviewer": "Reviewer_sWrJ", "summary": "This paper addresses the problem of unmeasured confounding in observational data.  Most causal estimation methods assume that there are no unmeasured confounders, an assumption that is hard to test in practice.  The authors propose a method for testing this assumption in situations where data from multiple environments is present.  Previous work in this area relied on unreliable conditional independence testing, so this work proposes a novel approach based on hypothesis testing of statistical covariance instead.  If the model is well-specified, the authors show that this approach consistently outperforms other approaches.\n\nI appreciate the authors' comments.  The other reviewers seemed to share similar concerns around the clarity of the connection/difference from Karlsson & Krijthe 2023, and it seems like the authors are aware of the need to explain this in more detail.  With that and an explicit discussion of assumptions are feature representation, I'm happy to raise my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the invaluable feedback and questions. We have provided answers and proposed changes related to your questions below. In addition, we also do the following changes for the camera-ready version:\n- update Section 4 with more explicit assumptions on the feature representations\n- name our method to MINT (Mechanism INdependence Test)\n- add references on identifiability in related works section, specifically on partial identification under unmeasured confounding\n\n# 1, 2: Differences in our work and Karlsson and Krijthe (2023)\nWhile we adopt the same setting as [KK23], a key conceptual difference in our approach makes our falsification strategy more sample-efficient. The framework of [KK23], which is based on DAGs, leads to a constraint-based causal discovery approach that relies on faithfulness and the causal Markov property. Importantly, they test independent causal mechanisms *indirectly* via a conditional independence statement involving treatment, outcome, and covariates in their assumed DAG. In contrast, one of our main contributions is the insight that it is possible to *directly* test the independence of causal mechanisms by inspecting the parameters of the treatment mechanism $E[A\\mid X,S=s]$ and outcome mechanism $E[Y^a\\mid X,S=s]$, which allows us to leverage functional assumptions. Here we prove the same theoretical guarantees for falsification remain when going from the indirect testing to the direct testing of ICM.\n\nThis new conceptual shift results in a substantially different algorithm. While [KK23] performs conditional independence tests on the data, our algorithm uses a two-stage approach: first estimating nuisance models (parameters of the treatment and outcome mechanism) before applying an unconditional independence test. Our approach appears to be more efficient, provided these models are well specified, even in comparison to HGIC when a well-specified conditional independence test was used.\n\nThere are two potential reasons for this efficiency. First, an unconditional independence test is statistically easier than a conditional one. Second, and more subtly, HGIC does not naturally allow for the efficient use of all available data in each environment because the conditional independence test in HGIC can only use 2 samples per environment, so it aggregates results from multiple tests to utilize all samples in each environment. Our algorithm avoids this issue entirely, as it leverages all data within a single independence test.\n\nBased on this, we will make the following changes to the manuscript:  \n- add a paragraph in Section 4 highlighting the above-mentioned differences between our work and [KK23].\n- reformulate first part of our contributions to: *“by formalizing the problem using a Neyman-Rubin causal model for multi-environment data, we show that falsification of unconfoundedness is possible by testing dependencies between causal mechanisms directly, rather than indirectly, by combining the principle of independent causal mechanism with functional assumptions on the mechanisms.”*\n\n# 3: Transportability versus ICM\nTo argue why ICM can be a more reasonable assumption than transportability, we will illustrate why we think that ICM is typically a weaker assumption than transportability. We will add this explanation to our paper when introducing the ICM assumption. We also want to point the reviewer to Appendix A in the supplementary materials where we discussed transportability in more depth.\n\nSpecifically, transportability violations occur (according to most definitions as we mention in Appendix A) when the outcome mechanism $E[Y^a\\mid X, S=s]$ varies across environments. A reason why this can happen is due to the presence of unmeasured effect modifiers, which often is a plausible scenario in many real-world settings. Very importantly, our Assumption 4.1 on ICM specifically allows $E[Y^a\\mid X, S=s]$ to change across environments. Thus, we can establish that even when transportability is violated, it is clear that ICM may still hold. What this means in practice is that our method is insensitive to whether transportability holds or not, whereas transportability-based tests will exhibit false positives (i.e. falsify unconfoundedness in the absence of unmeasured confounding) under violations of transportability. \n\n# 4: Clarification on Figure 2\nThanks to your comment, we understand the confusion. The falsification rate is the probability that a method will falsify unconfoundedness, regardless of whether this is correctly (there is an unmeasured confounder) or incorrectly done (there is no unmeasured confounder). It should have been more clear that under the absence of unmeasured confounding, we want a falsification rate below the significance level $\\alpha$. Under the presence of unmeasured confounding, a higher falsification rate is better.\n\nBased on this, we will update the figure caption of Figure 2b to clarify how to interpret the falsification rate on the y-axis.", "claims_and_evidence": "The claims made by this paper appear well supported.  The problem they're seeking to address is well-motivated and defined, and the approach seems sound.", "methods_and_evaluation_criteria": "The synthetic data used for the experiments appears reasonable.  I do think any detail at all about the synthetic data used should be included in the main body of the paper and not relegated to Appendix D.1, but it appears like a reasonable choice for synthetic data.  The twins data set is a nice choice for an empirical evaluation.  I wish more than 2 other methods were available for comparison, but given the specific nature of the problem being addressed, it's understandable.", "theoretical_claims": "While I did not dig into the proofs in the supplementary material, the theoretical claims in the main paper appear correct", "experimental_designs_or_analyses": "The baselines used for the experiments make sense, and the experimental design overall is sound.  I do wish the authors had done more experimentation with model-misspecification, since the need for a correctly-specified model seems like a major weakness of this approach, so more robust testing of the consequences of model-misspecification would help reduce those concerns.", "relation_to_broader_scientific_literature": "This is probably my biggest open question.  From the related work, it seems like your approach is quite similar, at least in problem setup and high-level approach, to Karlsson & Krijthe 2023.  The related work, in fact, makes it seem almost incremental (i.e., same exactly approach but they did CI tests and you did a different sort of test), but I'm not sure if that's actually the case.  It would be helpful to more clearly describe how this paper extends, and differs from, Karlsson & Krijthe 2023.  For example, your first two listed contributions are: \"we formalize a Neyman-Rubin causal model for multi-environment data under the principle of independence causal mechanisms\" and \"we prove that the presence of unmeasured confounding has testable implications in the form of dependencies between the model's observed parameters\".  Skimming through Karlsson & Krijthe 2023, both of those already seem to be covered by that work.  Is there some subtle difference between their version and this one that would qualify both of these as novel contributions?", "essential_references_not_discussed": "I'm not sure how essential this is, but in the Related Work, I was surprised that there was no discussion of identifiability, since that seems like a closer analogue to falsification than sensitivity analysis.", "other_strengths_and_weaknesses": "Overall, I think this is a solid paper.  The problem is well-motivated, the paper layout and descriptions are very clear, and the approach seems reasonable and effective.\n\nI wish there were more discussion around the feature representations psi and phi.  The need for correctly-specified feature representations isn't really discussed in the paper until the experimental results in 6.2.3 and the post-experiment discussion.  While the narrative of the paper as a whole flows well, Section 4.1 seems to be missing at least a sentence or two of lead-in.  An assumption of this approach seems to be that we have access to these feature representations, but it's not actually directly called out as an assumption anywhere that I can see.  As you mention in the discussion, we could learn these from data, so it's not a fatal flaw.  But some discussion in Section 4 about how we get psi and phi, and experimental results showing that it's at least somewhat possible with empirically learned psi and phi, would help a lot.\n\nI think some clarity around what assumptions are made by your algorithm, and which assumptions are ones you'll be attempting to falsify, would help.  Assumption 3.1 is a common assumption and one you'll falsify, but that's not directly stated in Section 3.2 until the final sentence, making it a bit unclear.  Similarly, Assumption 4.1 is another one that your algorithm attempts to falsify, but just a little bit before that in the body of the text (lines 170-171), you state another assumption (\"the feature representations psi(X) and phi(X,A) are considered to be fixed across environments\"), but I think this assumption is one you're /actually/ making, not one you're falsifying.\n\nThe first time psi-~ and phi-~ appear, I believe, is lines 210-211 in Section 4.2, but you don't actually define them at this point (you just say that we assume we have access to them).  In the next paragraph, you discuss them more and it seems like they are estimates of phi and psi, but actually defining them before you use them would help.\n\nAre you using the wrong y-axis label in Figure 2?  Figures 1 and 2 both have the y-axis labeled \"Falsification rate\", but in Figure 1, higher is clearly better (which makes sense since falsification is the goal), while in Figure 2, lower is better.  Figure 2 seems to actually be the p-value from the hypothesis test? (hence why you show the alpha=0.05 significance level)", "other_comments_or_suggestions": "You should name your algorithm so it's not just called \"Ours\".  If it's largely an extension to HGIC, naming it HGIC-XXX (for some relevant XXX acronym) would be fine, but it's clunky to not actually have a name in the paper.", "questions_for_authors": "My first two are primarily what I went over under \"Relation to Broader Scientific Literature:\n\n1. How does your proposed algorithm differ from HGIC in Karlsson & Krijthe 2023?\n2. In what ways are our first two listed contributions different from the similar formalisms/proofs in Karlsson & Krijthe 2023?\n\n3. Could you discuss the transportability assumption, and how it relates to your work, a bit more?  In Related Work, you contrast your work with prior work in transportability and point out the key difference as being that your approach \"assumes independence of causal mechanisms\" while this work \"require[s] transportable treatment effect or access to randomized data.\"  In what ways is the 'independence causal mechanisms' assumption more reasonable/achievable in practice than the 'transportable treatment effect' assumption?\n\n4. Unless I'm misunderstand what's being shown in Figure 2 (see my confusion about the y-axis label for Figure 1 vs Figure 2), it appears as though, in Figure 2b, in the presence of unmeasured confounding, the falsification rate is actually significantly worse as the number of samples increases.  For example, for a 6-degree polynomial, only using 50 samples has a falsification rate (p-value??) of nearly 0 (well below the 0.05 line) while 500 samples is over .9!  Why is that?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EwrpDU33qP", "reviewer": "Reviewer_33m1", "summary": "The paper proposes a test for unconfoundedness based on the usual assumptions of causality from the potential outcomes perspective, but more critically assumptions about the independence of causal mechanisms and a specific functional form (including specifying functional form well). The test requires observing more than one environment. The test statistic is based on a Frobenius norm of the covariance matrix of the parameters of the causal mechanisms. The test statistic is calibrated through permutation. Finally, the authors investigate the falsification rate of the method and baselines with synthetic data, modifying the number of environments, number of observations per environment, number of covariates, and violations to the specification of the model. They also test it on a semi-synthetic dataset.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the response, in particular their excitement for our proposal for extending our method to implicit feature representations. Whereas it felt as to be outside the scope of the current manuscript, we agree this is one of the more promising directions of our work and we also hope to encourage others to see this possibility. Below you can find answers to your comments. In addition, based on your feedback, we are doing the following changes to our camera-ready version:\n- we state more explicitly in the abstract and experimental section that we use semi-synthetic data.\n- we add the references you suggested to the related works section. \n\n# Clarification on results in section 6.2.2\nThe primary objective of the experiment described in Section 6.2.2 is to empirically validate the theoretical results presented in Section 4.3. To achieve this, we simulate data based on the model specified in Equation (3). The different parameters on the x-axis in Figure 2a represent which of the parameters in Equation (3) are allowed to vary across different environments. This is done under both the absence and presence of unmeasured confounding.\n\nTheorem 4.2 predicts that any changes associated with the $\\alpha$ parameters should enable the detection of an unmeasured confounder, if one is present. Our experimental results confirm this prediction. The underlying reason is that, unlike the $\\beta$ parameters, the $\\alpha$ parameters appear in both the treatment and outcome mechanisms, as established in Lemma 4.3.\n\n# Using other test statistics\nThere are indeed alternative test statistics for testing the independence between two random variables: the main requirement is that the test statistic works with multivariate random variables (due to $\\omega$ and $\\gamma$ being multivariate). Our reasoning for using the Frobenius norm of the covariance matrix as a test statistic stems from the fact that it only considers linear dependencies, which are typically the easiest to identify. One could also consider other test statistics that consider nonlinear dependencies, but this extra flexibility could lead to the method being more data hungry. For instance, we also considered the Hilbert-Schmidt Independence Criterion (HSIC) which allows for testing nonlinear dependencies using an appropriate kernel (when using a linear kernel one would obtain the test statistic which we already use). However, some initial experiments showed that HSIC was also not particularly robust. Another alternative test statistic that we did not explore but could be considered is distance correlation, which can also detect both linear and nonlinear dependencies.\n\n# Connection with ICP\nWe agree that Invariant Causal Prediction (ICP), as introduced by Peters et al. (2016), should be acknowledged for its role in highlighting the premise of utilizing data from multiple environments. We have thus added a reference to their paper in the related works. While both ICP and our approach consider multiple environments, they are based on fundamentally different principles. Our method explicitly exploits changes across environments, whereas ICP is designed to identify what remains invariant.\n\nTo further illustrate this contrast, Peters et al. (2016) state on page 2:\n\n*“We exploit, in other words, that the conditional distribution of the target variable of interest (often also termed “response variable”), given the complete set of corresponding direct causal predictors, has to remain identical under interventions on variables other than the target variable”*\n\nThis stands in stark contrast to our framework. Unlike ICP, which believes the conditional distribution of the target variable to remain unchanged across environments, we explicitly state in Assumption 4.1 that the conditional distributions of the target variables (in this case outcome and treatment variables) can vary across environments rather than remain invariant.\n\n# Comment on our method being “data hungry”\nWhile improving sample efficiency is an important goal for future work, we emphasize that our method already outperforms the baselines across different numbers of environments and samples per environment. Even in the most challenging small-sample case we tested – 10 environments with 25 samples each – our method achieved a falsification rate of 0.8, compared to 0.6 and 0.4 for the baselines. Having said that, access to multiple environments is critical for any falsification method based on variation between environments to work, though the number of required environments will likely depend on how much variation there is between the environments.", "claims_and_evidence": "In general the claims are supported by evidence, both theoretically and empirically. Maybe a minor complaint about claims vs. evidence is the fact that the authors claim they tried their method in real-world data whereas the closest they get to real world data is semi-synthetic data.", "methods_and_evaluation_criteria": "Yes, the methods and the evaluation criteria make sense for the proposed problem.", "theoretical_claims": "I didn’t check the proofs of any of the theoretical results, but they all seem natural to me (of course with the exception of Lemma 4.3., which requires a lot of computation.", "experimental_designs_or_analyses": "The experimental design and analysis seem very reasonable to me. Both the test statistic and the permutation based calibration make sense. Would be interesting to know whether the authors tried other test statistics or whether they think that other tests would work and potentially would be more efficient in some way. Additionally, testing for the number of environments, sample size, number of covariates and misspecification is something I would expect to see.", "relation_to_broader_scientific_literature": "I think the ideas presented in the paper are interesting. In the end, confounding is arguably one of the biggest problems in causality, and proposing a method for detecting confounding based on heterogeneous data is a valuable addition to the research on causality.", "essential_references_not_discussed": "The authors cite several related research. I would personally add a couple more references, for example in page 2, column 2, line 97 when they talk about multi-environment data and looking only at the relevant portion for causal effect estimation, that’s precisely the idea of Invariant Causal Prediction (ICP) in Peters et.al. (2016) (from JRSSSB) which does the same task and were one of the first to explicitly include environment information for causal tasks. Other research that could have been included is the more recent work on falsifiability of causal discovery algorithms like Faller et.al. (2024) in AISTATS.", "other_strengths_and_weaknesses": "Strengths:\n- The theoretical ideas of the paper are interesting and seem reasonable to me. \n- The empirical tests make sense with respect to their claims.\n\nWeaknesses:\n- I think the empirical evaluation reveals what I consider to be one of the biggest weaknesses of the paper. Particularly how data hungry the method is. I can imagine that because of the way the test statistic is designed –and the permutation based calibration– one needs to have several environments and several observations per environment to get a reasonable falsification rate.", "other_comments_or_suggestions": "None.", "questions_for_authors": "The results of section 6.2.2. are really difficult to interpret. What I understand is that you change the parameters as changes in the mechanisms to see whether you can detect unconfoundedness using your test. But why do we expect changes in the alphas to give us falsification while changes in betas don’t?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "No concerns"}}, {"review_id": "zIbYnPrLVu", "reviewer": "Reviewer_GbFw", "summary": "Addressing the common assumption of causal sufficiency, this work proposes an approach to falsify this assumption in causal effect estimation settings. It relies on datasets from multiple environments to test for violations of the Independence of Causal Mechanisms (ICM) principle. The authors study a causal model where treatment and outcome are linear transformations of (known) feature representations and show theoretically that unobserved confounding creates dependencies in certain model parameters. They propose an algorithm to test for unconfoundedness along with a permutation-based calibration scheme to set a test rejection threshold. Besides conceptual improvements over existing approaches, such as avoiding CI testing and having milder assumptions, the method shows promising empirical improvements over existing algorithms in synthetic settings and convincing results on realistic data.\n\n### update after rebuttal ###\n\nI thank the authors for the rebuttal. My assessment remains positive.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their invaluable feedback, suggestions for improvement and questions. Below are answers to your questions. \n\n# 1: Fixed feature representations with linear parameter shifts\nWe agree that it is important to consider what it entails to assume a model class with fixed feature representation across environments with linear parameters shifts. Our key idea -- testing independence of causal mechanisms -- benefits from the fixed feature representation, as it enables efficient testing by focusing solely on the linear parameters. However, a fixed feature representation with linear parameter shifts is not fundamental to our proposed falsification strategy for testing mechanism independence. As explored in  Appendix C, one could, in principle, replace the fixed feature representation with a potentially infinite-dimensional implicit representation using kernel methods.\n\nThat said, we do not consider the assumption of a fixed feature representation to be particularly restrictive. To illustrate this, suppose a feature $X_1$ is highly predictive of treatment assignment in environment A but not in environment B. The fixed representation for environment A and B can still accommodate this by including $X_1$ (and its nonlinear transformations), and then only the linear parameter in environment A will utilize this feature whereas the linear parameter in environment B will not. This suggests having a fixed representation is not overly restrictive, as long as it is “rich enough” to accommodate different environments. In practice, of course, the richness of the representation will affect the efficiency of the test, making it a trade-off between expressiveness and statistical power. Hence, methods to learn a common representation or implicit representations are interesting future research directions.\n\n# 1: Unable to falsify under changes of $\\beta$\nIn an ideal world, we would hope that our method could also falsify under solely changes on $\\beta$. However, based on our current understanding, this may represent a fundamental limitation of this type of falsification approach, as both our findings and those of Karlsson and Krijthe (2023) suggest the same conclusion. But from a practical view-point, a key characteristic of observational studies is that treatment assignment is non-randomized. In our setting, where we look at multiple observational studies, it would seem plausible that each study has its own unique treatment assignment mechanism. Thus, in practical use cases one could expect $\\alpha$ (if not $\\mu$) to likely vary across environment. \n\n# 2: Degradation with more covariates\nThe decline in falsification rate as the number of covariates increases in the right-most plot of Figure 1 can be explained as follows: As the number of covariates grows, estimating the linear parameters $(\\omega,\\gamma)$ becomes more challenging, leading to greater sample variance in the estimates. This, in turn, increases the uncertainty in estimating the covariance matrix $\\text{Cov}(\\omega,\\gamma)$  which is crucial for computing our test statistic. Consequently, the test loses power as the number of covariates rises. \n\nFor HGIC, the choice of conditional independence test significantly impacts its response to increasing covariates. For example, we found that using HGIC with the kernel conditional independence test (KCIT) performed poorly as the number of covariates increased; an expected outcome since large conditioning sets are known to harm KCIT (see Zhang et al., 2011). Meanwhile, HGIC remained effective when using the Pearson partial correlation test, even as the number of covariates grew. But this robustness was not unique to HGIC: the transportability test with Pearson’s test also remained unaffected by additional covariates. Thus, we could only speculate that the test statistic used in the Pearson test is very well-suited for the linear synthetic data we generated.\n\n# 3: Large sample sizes in data-generating process in Appendix D.2\nIn the experiment shown in Figure 2a, where we used data as described in Appendix D.2, our goal was to explore the theoretical limits of our approach in light of the results from Section 4.3. Specifically, the identifiability guarantees for falsifying unmeasured confounding under various linear parameter shifts. To minimize finite-sample effects, we increased the sample size and the number of environments. However, due to computational constraints, we could not scale them indefinitely. Notably, this was the only experiment where we took this approach. In our other experiments, our method remains effective even with much smaller sample sizes and fewer environments. For instance, in the left-most plot of Figure 1, our approach was able to successfully falsify unconfoundedness with just 10 environments and 25 samples per environment.\n\n# References\nZhang, Kun, et al. \"Kernel-based conditional independence test and application in causal discovery.\" UAI, 2011.", "claims_and_evidence": "All theoretical statements are supported and the proofs are accessible. The authors also made efforts to support their claims empirically, for example illustrating the effect of shifts in different model parameters (Fig. 2).", "methods_and_evaluation_criteria": "The authors evaluate their approach against suitable competitors for falsification of unconfoundedness in a treatment effect setting, using sensible evaluation criteria (falsification rate). The method was also tested in a realistic setting using a twin study proposed in earlier work (Karlsson and Krijthe, 23).", "theoretical_claims": "Thm. 4.2, 4.3, 4.4, where I did not find any issues.", "experimental_designs_or_analyses": "Figs. 1-5. The synthetic data generation is reasonable and there are no apparent issues with the twin dataset setup.", "relation_to_broader_scientific_literature": "The idea of detecting confounding from violations of the ICM principle has been explored before and the approach is somewhat close to existing work by Karlsson and Krijthe (2023). However, it takes a new perspective with different algorithmic ideas, adding sufficient originality. Overall, there is only a small literature addressing untestable assumptions in causal inference and discovery, and the paper adds useful new insights.", "essential_references_not_discussed": "The relevant related work has been cited and put into context.", "other_strengths_and_weaknesses": "The problem of addressing untestable assumptions in causal inference, such as sufficiency, is of high significance to the literature and, therefore, a strength of this work. The paper is clearly written, and the presentation of both theory and experiments is well structured. As a potential weakness, the assumptions are rather strong. In particular, the true feature representations $\\varphi$ and $\\psi$ in the model need to be known. The authors, however, illustrate the implications of this in experiments under misspecification (Fig. 2) and also sketch a kernelized version of their method (Appendix C), looking to remove this assumption in future work.  According to the authors (l. 401 right) the parametric nature also brings certain empirical advantages over existing approaches. Given these points, it seems worthwhile to investigate the given model for the scope of this work.", "other_comments_or_suggestions": "- ln. 602, left: \"then so are the transformations also independent r.v.s\"\n- ln. 687, left: \"where (b) equality follows from that\"", "questions_for_authors": "1. Can you comment on how constraining you consider the assumption that the feature representations $\\varphi$ and $\\psi$ remain the same across environments while only the linear parameters shift? Related to this, how limiting is the fact that your method only works under changes of $\\alpha$ and $\\mu$ but not $\\beta$ (e.g. Fig. 2)?\n2. Do you have an explanation for the degrading performance with more covariates (Fig. 1 right) and why this is not an issue for HGIC? \n3. The generating process in (3) as described in Appendix D.2. uses quite more samples and environments (1000, 250) than the experiments in D.1. Is such a large number of environments necessary here for the approach to work?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose a simple method - leveraging multiple observational data sets to test independence of the coefficient parameters that are assumed to be independently drawn under the null (ICM assumption) - that rejects the null hypothesis only if one of their assumptions (unconfoundedness, standard assumptions such as consistency, positivity, independence of mechanism parameters, or well-specification of the treatment effect and outcome models) do not hold. That means that, if the hypothesis is rejected, it may be due to unmeasured confounding - although it is not possible to know for sure that confounding was the "culprit" or one of the other many possible reasons, it is still valuable information to know what not to do with the data for treatment effect estimation. 

Apart from the reviewer comments you have already addressed, from a quick reading, it seems to me that a very strong assumption that may jeopardize the usefulness of the method in practice is that the null may either be rejected if the feature representations $\tilde{\phi}, \tilde{\psi}$ are not correct (i.e. misspecification of A|... and Y|...) or the model not expressive enough unless $\alpha, \beta$ are made very large - in which case testing independence becomes harder and that power suffers (even in the case of dependence, it is not spottable when d>>n). This is implicitly suggested in Figure 2b) - it would be nice to plot "power" as well in that case (I couldn't find a plot). Hence I wonder about the usefulness of this method for real-world data (i.e. when the polynomial model is not of degree 2 but known to be quite complicated - and hence enforcing the dimensionality of $\alpha,\beta$ to be large. In this case, how does this new method compare with HGIC/Pearson tests? This seems to be reflected in the last plot in Figure 1. I wish the authors could provide a better power analysis in (semi-)synthetic data (beyond just table 1) and a more experiments and discussion of the effectiveness of the new test on real-world data.
Decision: Accept (poster)