Detecting and Measuring Confounding Using Causal Mechanism Shifts
Authors: Abbavaram Gowtham Reddy
Vineeth N. Balasubramanian
Venue: NeurIPS 2024 poster (2024)
Keywords: Causality
Confounding
Mechanisms
Measure
Primary area: causal_inference
TL;DR: We propose methods to detect and measure confounding using data from causal mechanism shifts.

Abstract: Detecting and measuring confounding effects from data is a key challenge in causal inference. Existing methods frequently assume causal sufficiency, disregarding the presence of unobserved confounding variables. Causal sufficiency is both unrealistic and empirically untestable. Additionally, existing methods make strong parametric assumptions about the underlying causal generative process to guarantee the identifiability of confounding variables. Relaxing the causal sufficiency and parametric assumptions and leveraging recent advancements in causal discovery and confounding analysis with non-i.i.d. data, we propose a comprehensive approach for detecting and measuring confounding. We consider various definitions of confounding and introduce tailored methodologies to achieve three objectives: (i) detecting and measuring confounding among a set of variables, (ii) separating observed and unobserved confounding effects, and (iii) understanding the relative strengths of confounding bias between different sets of variables. We present useful properties of a confounding measure and present measures that satisfy those properties. Our empirical results support the usefulness of the proposed measures.

Detecting and Measuring Confounding Using Causal Mechanism Shifts
Abbavaram Gowtham Reddy Indian Institute of Technology Hyderabad cs19resch110020itth. in
Vineeth N Balasubramanian Indian Institute of Technology Hyderabad vineethnbeiith ac.
Abstract
Detecting and measuring confounding effects from data is key challenge in causal inference. Existing methods frequently assume causal sufficiency, disregarding the presence of unobserved confounding variables. Causal sufficiency is both unrealistic and empirically untestable. Additionally, existing methods make strong parametric assumptions about the underlying causal generative process to guarantee the identifiability of confounding variables. Relaxing the causal sufficiency and parametric assumptions and leveraging recent advancements in causal discovery and confounding analysis with non-i.idd. data, we propose comprehensive ap- proach for detecting and measuring confounding. We consider various definitions of confounding and introduce tailored methodologies achieve three objectives: (i) detecting and measuring confounding among a set of variables, (ii) separating observed and unobserved confounding effects, and (iii) understanding the relative strengths of confounding bias between different sets of variables. We present useful properties of confounding measure and present measures that satisfy those properties. Our empirical results support the usefulness of the proposed measures.
1 Introduction
Understanding the underlying causal generative process of a set of variables is crucial in many scientific studies for applications in treatment and policy designs [44]. While randomized controlled trials (RCTs) and causal inference through active interventions are ideal choices for understanding the underlying causal model [19, 12, 13, 55), RCTs and/or active interventions ble/infeasible, and some times unethical [50, 6]. Research efforts in causal inference hence rely on observational data to study causal relationships [44, 59, 65, 18, 41J. However, recovering the underlying causal model purely from observational data is challenging without further assumptions; this challenge is further exacerbated in the presence of unmeasured confounding variables.
often impossi-
confounding variable is a variable that causes two other variables, resulting in a spurious association between those two variables. As exemplified with Simpson' paradox [58] and many other studies [20, 31], the presence of confounding variables is an important quantitative explanation for why correlation does not imply causation. It is challenging to observe and measure all confounding variables in a scientific study [60, 44). Identifying latent or unobserved confounding variables is even more challenging, and misinterpretation presents various challenges in downstream applications, such as discovering causal structures from observational data. Numerous methods operate under the assumption of causal sufficiency [45, 60, 8, 51, 65], implying the non-existence of unobserved confounding variables. Causal sufficiency presupposes that all pertinent variables required for causal inference have been observed. However, this may not be practical or testable assumption.
The study of confounding has various applications, chief among them being causal discovery identi- fying the causal relationships among variables [38, 40, 63]. It is also useful for determining whether a set of observed confounding variables is sufficient to adjust for estimating causal effects [29], measuring the extent to which statistical correlation between variables can be attributed to confound-
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

ing [24, 25, 62], and verifying the comparability of treatment and control groups in non-randomized interventional studies [16].
A fundamental problem in causal inference tasks lies in detecting hidden confounding variables from observational data alone. However, this is non-trivial and poses various challenges. For example, key issue is that given marginal distribution over observed variables, there are infinitely many joint distributions corresponding to causal graphs involving unobserved variables [56]. To tackle such challenges, recent endeavors show that using data from different environments helps in improved causal discovery [40, 38, 33, 45, 23], detecting causal mechanism shifts [36], and detecting unobserved confounding [29, 38]. However, such recent efforts often subsume confounding detection under causal discovery, focusing primarily on identifying confounding factors while overlooking other useful information, such as the relative strength of confounding between variable sets and the distinction between observed and unobserved confounding within a variable set. We seek to address these gaps in this work.
We focus exclusively on the problem of studying confounding from multiple perspectives, including (i) detecting and measuring confounding among a set of variables, (ii) assessing the relative strengths of confounding among different sets of observed variables, and (iii) distinguishing between observed and unobserved confounding among a set of variables. The primary focus of causal inference often lies in verifying the presence or absence of confounding rather than determining the exact value of the measured confounding. However, we leverage the measured confounding to assess the relative strengths of confounding between sets of variables. To achieve the above objectives, we utilize data from various contexts, where each context results from shifts in the causal mechanisms of a set of variables [38, 45). This allows us to propose different measures of confounding based on the available context information. Our contributions can be summarized as follows.
For various definitions of confounding, we propose corresponding measures of confounding and present useful properties of the proposed measures. To our knowledge, this is the first comprehensive study that examines various aspects of observed and unobserved confounding using data from multiple contexts without making parametric causal sufficiency assumptions.
We study pair-wise confounding, confounding among multiple variables, how to separate unob- served confounding from overall confounding, and present ways to assess relative confounding.
We present an algorithm for detecting and measuring confounding using data from multiple contexts. Experimental results are performed to verify theoretical analysis.
2 Related Work
The study of confounding has typically been embedded as part of causal discovery algorithms in most existing work. Causal discovery methods can be categorized according to several criteria, parametric whether they relax causal sufficiency assumptions [65, 59]. Considering our focus in this work on studying confounding comprehensively by going beyond observed confounding variables, we discuss literature that are directed towards methods that relax the causal sufficiency assumption and rely on experimental data.
including the type of data utilized (observational versus interventional/expeimennt versus non non--oonnnnnn -arrmmraettiic approaches,
Causal Discovery via Observational Data, Relaxing Causal Sufficiency: Constraint based causal discovery algorithms produce equivalence class of graphs that satisfy set of conditional independence constraints [60, 11, 9, 42]. Other methods such as [2, 28, 27] reduce the problem complexity by assuming a parametric form of the underlying causal model (e.g., variables are jointly Gaussian in Chandrasekaran et al. [7]), thereby returning unique causal graphs. Nested Markov Models (NMMs) [56, 57, 49, 14] allow identifiability of causal models with latent factors by using (pairwise) Verma constraints. A recent approach using differentiable causal discovery [2] combines NMMs with the differentiable constraint [66] to discover a partially directed causal network and likely confounded nodes. Unlike these methods, our focus in this work is on detecting and measuring confounding under various settings, instead of recovering the entire causal graph or equivalence class.
Causal Discovery Using Data From Multiple Environments: Given access to a set of observed confounding variables, very recent work [29] presented testable conditional independence tests that are violated only when there is unobserved confounding. However, their analysis is focused towards the downstream causal effect estimation. We aim to provide a unified framework for studying and measuring confounding under different types of contextual information available.

---

Other methods [33, 23] learn an equivalence class of graphs when data from observational and interventional distribution are available. Confounding has also shown to be detected in linear models with non-Gaussian variables [20]. In linear models, a spectral analysis method was proposed in [25] to understand to what extent the statistical correlation between a set of variables on target variable can be attributed to confounding. See Tab. 4 of [40] for an overview of causal discovery methods that use data from multiple environments or contexts. Under the specific assumptions of causal sufficiency and sparse mechanism shift, a method was proposed in [45] to reduce the size of a given Markov equivalence class using mechanism shift score. A differentiable causal discovery method was proposed in [4] to use interventional data to recover interventional Markov equivalence class. While these methods use data from different contexts, they assume the absence of unobserved confounding variables; we instead focus capturing both observed and unobserved confounding.
Measuring and Interpreting Confounding: Earlier efforts in the field have studied different measures for observed confounding, each tailored to address specific challenges [44, 15, 35, 3, 39, 30, 43, 34). Such measures have also been refined to address specific issues [24, 54]; for e.g., a method to correct the non- non-linearity effect present in confounding estimates via the seposure-outcome association with and without adjustment for confounding was proposed in [24]. In contrast, we measure the effects of both observed and unobserved confounding. Motivated from the ignorability property in potential outcomes framework [61, 26], the divergence between nominal and complete propensity density has been considered an indicative of hidden confounding [26]. To the best of our knowledge, the efforts closest to ours are [38, 40], which study confounding using data from multiple contexts without the causal sufficiency assumption. However, they do not measure confounding and detect confounding only as a step to discover the causal graph. Ours is a more general framework for studying and measuring confounding from multiple perspectives.
In regression models, certain difference thresold between the coefficients of treatment variable before and after adjusting for the possible confounding is considered as the indication for the presence of confounding. This process of choosing a threshold is also called change-in-estimate criterion. Typical threshold used in literature is 10% [54, 32, 5].
3 Background and Problem Setup
Let X be set of observed variables and Z be a set of unobserved or latent variables. The values of X, Z can be real, discrete, or mixed. Let G be the underlying directed acyelic graph (DAG) among the variables V = X U Z. Directed edges among the variables in V indicate direct causal influences. Assume that the set of unobserved variables Z are jointly independent and are exogenous to X (i.e.,
Zi 1 Z, and X₁ A Zj ji,j. k). In this setting, any two nodes X;, X, E X sharing a common parent 2/p E are said to be confounded, and Zμ is said to be a confounding variable. For node X, E X, PA, (X; E X X.) U (Z₁ E 2 Z - X;) denotes the set of parents of X,,
For node X., P(X, is called the causal mechanism of X,. The causal mechanism P( encodes how the variable X; is influenced by its parents PA, Following earlier work [22, 38, 45, 21, 46, 52], we make the following general assumption about the underlying causal mechanisms of data. Assumption 3.1. (Independent Causal Mechanisms [44, 471) A change in PPX.((A.)) has no effect on and provides no information about ((X))PA)) Vj # i.
Identifying confounding from only observational data is challenging without further assumptions [28]. Hence, following earlier work [38, 29, 40], we assume that the data over the variables X is observed over multiple contexts or environments. While there are various ways of formulatin/cconstrutinng contexts, in this paper, we assume that each context is created as result of either hard (a.k.a. structural) interventions or seft (a.ka. parametric) interventions on a subset Vs S V of variables where 5 is a set of indices. Performing hard intervention on variable V. is the same as setting the value of V4 to a value e, Hard intervention on a variable VÚ removes the influence of its parents PA, variable V; is the same as changing the causal mechanism of
on V2. Performing soft intervention V,, (PA), with a new causal mechanism PP,)). Soft intervention on variable Vú does not remove the influence of its parents PA, on V,. The idea of explicitly considering context information and using different contexts as context variables to create extended causal graphs has been studied in the literature. Context variables are also called as policy variables, decision variables, regime variables, domain variables, environment variables, etc. [40, 45, 17, 22].
Let C- = {c1, .,, be the set of n contexts and let Pe P((X), E C, denotes the probability distribution of the observed variables X in the context c. Let CSAR, where S, R are sets of indices, be
3

---

the set of contexts in which we observe mechanism changes for the set of variables XSuR. Similarly, let CSA-R be the set of contexts in which we observe mechanism changes for the set of variables Xs but not for the variables X We say that the causal mechanism of variable X, changes between two contexts e' if (X; (PA.)
+ p (P,). Given the data over observed variables in each context, there exist methods for detecting mechanism shifts of each variable between the contexts [36, 38, 45, 37J. For example, the p-value(IP (X, (PA3) * J (X, (XA)) where PA; is the set of observed parents of X; can be used to detect mechanism change for X, between the contexts c, c' [38, 36]. Hence, we focus on detecting and measuring confounding among a set of variables, assuming that the causal mechanism shifts are observed among that set of variables.
Context information is not very useful if there no restriction on how causal mechanisms are changed between the contexts [45, 38]. For example, the causal mechanisms of X, and Xj both differing across all (or no) contexts would trivially satisfy Assumption 3.1, but reveal no information about the underlying causal mechanisms [10, 38]. Hence, following earlier work [45, 38, 17], we make the following assumptions.
Assumption 3.2. (Sparse Causal Mechanism Shift (53]) Causal mechunisms of variables change sparsely across contexts, ie., if == (P, (PA.) + Pe (X.)). then 0 < p < 0.5; Ve, d' E C.
Assumption 3.2 implies that the causal mechanisms change infrequently across contexts. This assumption is more general because, in many scientific studies, for any given context, interventions typically affect only few variables [53].
Assumption 3.3. (Markov Property under Mechanism Shifts [17[) The distribution P(V) is given by P(V)= = p (V)dP(C) = I (Vi[A, (C)). In other words, variables V are assumed to be conditionally exchangeable, so that the same graph 9 applies in every context E C.
Assumption 3.4. (Causal Sufficiency Over X U Z) All common parents of any pair of observed nodes belong to the set X U Z. In other words, all relevant variables for detecting confounding and the unobserved confounding variables are already present in X U ZZ
Given data over the observed variables X in multiple contexts, each context resulting from a sparse causal mechanism shift of variables in V, (i) can we identify which pairs or sets of variables are confounded and can we measure the confounding strength? (ii) can we isolate the confounding effects of observed and unobserved confounding variables? and (iii) can we study the relative strengths of confounding among different sets of variables?
Problem Statement:
To address the above problem, in the next section, consider various definitions of confounding and present appropriate confounding measures depending on the context information available.
4 Detecting and Measuring Confounding
In this section, we present meth- ods for detecting and measur- ing confounding for various sce- narios in which shifts in causal mechanisms are observed. Con- sidering any three observed vari- ables X,, Xy. X. E X and an unobserved confounding vari- able E Z. we present mea- sures of confounding depending on the information about mechanism shifts of X;, X,, Xa, Z. Each of the following subsections includes: (i) a definition of confounding, (ii) corresponding definition of the confounding measure, (iii) method for isolating the unobserved confounding measure from the overall confounding, (iv) an extension of the confounding measure to more than two variables, and (v) key properties of the proposed confounding measures. See Tab. I and Fig. for an overview.
Settings
Confounding Definition Based
Required Context Information
Type of Intervention
Dirccted Information [48] & Noncollapsibiltty 115,, 43, 54]
Cajaing in CupaoPa
/ /trrctural
2 2 3
Mutual Information
Soft / Parametric
Table I: Summary of the various settings for detecting and measuring confounding between X., X,, Here P:ş is the set of node indices that belong to path from X; to X₃ including j.
4.1 Setting 1: Measuring Confounding Using Directed Information Between X,, Xz-
In this setting, we use the fact that directed information does not vanish in the presence of confounding variable [64, 48]. To this end, we leverage the interventional effects of X,, X, on each other to define measure of confounding.
4

---

Setting
Setting 2
Setting 3
Figure I: Setting When contexts C. and C())--P, are known where P.y is the set of node indices that belong to a path from X, to X3 including J. we leverage directed information from X; to X, and from X, to X; to define a measure of confounding (Defn. 4.4), Setting 2: Causal mechanism changes in Z introduces dependencies on the observed distributions of X,, X,. We leverage such dependencies to measure confounding when contexts (()))()) are known (Defn. 4.6). Setting If we know that there causal path from X; x,, we leverage dependencies hetween the pairs (X,, X3) and (Z, X;) to measure confounding. Similarly, we know that there is a causal path from X2 to X. we leverage dependencies between the pairs (X,, Xy) and (Z, X;) to measure confounding (Defn. 4.7). Dashed arrows from Z indicate that Z is unobserved.
Definition 4.1. (Directed Information [48]). The directed information ((x; + X3) X; E X to X3 E X is defined as the conditional Kullback-Leite Leibler divergence between the distributions IPX) ())x((x))))))))))))))))))))))))) That is:
= X,(,,,,,, log P(X:JX,) (o()))
T(X; + x,) DEL
(I)
Definition 4.2. (No Confounding [44]) When measuring the causal effect of a (treatment) variable X; on a (target) variable X.. the ordered pair Xç) is unconfounded if and only if the directed information from Xj to X4: 1(X X:) is zero. Equivalently, Xx)) = P(X, ((X))do(X.)).
A similar definition of confounding that relates the conditional distribution P(X,)X)) and interven- tional distribution ((o))))(X)) is defined as follows.
Definition 4.3. (Noncollapsibility) [15, 43, 54J The statistical association between two variables X; and Xj is said to be noncollapsible if the association strength differs in each level/strata of other variable Xg. That is, if X is a confounding variable between X5, Xz, we have P P(X)(X;) + ((X,ldo(X,))
Ex, (X,
From Defns. 4.1 and 4.2, for a pair of variables (X;,X)) observing -X) > 0 and [(;; + X²) implies that (XX] (X)) + P(X,IX,) and hence the presence of confounding (see Tab. 2). Using the above properties of directed information, we measure confounding as follows.
Definition 4.4. (Confounding Measure 1) When causal mechanism shifts of two variables X,, X3 E X are observed, resulting in different contexts, under the Assumptions 3.2-3.4, the measure of confounding CN F-N-,1,,,, XX) between X; and X3 is defined as follows.
CNF-1(X,, = e mx/
(2)
For all the confounding measures, we use expo- nential transformation to limit the range of the measure between and 1. Note that DAG,
Graph
[(X; - +x))
I(X, - X
Confounded Unenf.
X X,
>0
=
one of T(X; -> x,), T(X; + X;) is zero under no confounding (see Tab. for simple exam- ple with two and three node graphs). Hence (NI, outputs zero when there is no confounding between X;, Xj- Similarly
Xx Xi
=0
> 0
Ix: X;;
> 0
> 0
2 X 2 +2
Xx X1
>0 0
- X,, 7 +,
CN C-F-1 Xj) outputs positive real value in the range (0, 1] when there is confounding. We leverage data from multiple contexts to evaluate F(X,|X,) and P(X,|do(X,
Table 2: Directed information values in two and three node graphs. If X,,X, are confounded by Z, we ob- serve positive directed information from both directions.
follows. In this setting, we assume each context is generated as a result of hard interventions on a subset of variables. Let Pty be the set of node indices that belong to path from X, X, including j. we the contexts C(j)AP,, to evaluate PPX,, do(X))) as P (dx())(X,)) = EEECC [P (X,) Intuitively, to compute the interventional effects of X, on Xj. we need to observe mechanism changes only for X; to account for the potential causal influence from X; to Xj. In addition, none of the nodes in a causal path from X, to X, should be intervened. We use observational data to evaluate P(X, (X,).
5

---

Proposition 4.1. (Identifiability of P(X,) (do(X:))) P(X)) is identifiable from the set of contexts Pay To detect and measure confounding between a pair of nodes X, is enough to observe two sets of contexts (s) and Cjj)-P, Thus, sets of contexts are needed to detect and measure confounding between (2) distinct pairs of nodes in a causal DAG with nodes.
When a confounding variable X, between X;, ,, is observed, and there may exist an unobserved confounding variable Z, it is crucial to detect and measure unobserved confounding effect [29]. We utilize conditional directed information to define the measure of unobserved confounding.
Definition 4.5, (Conditional Directed Information [48]). The conditional directed information I(X; + X,lXo from X, to X, conditioned on Xo is defined as the conditional Bellbacck-eebbler divergence between the distributions IP [ X X.), P(X, [d)(X)),, Xo) as follows.
P,, X.) X,X.I.,.X.) P(X, (do()), Xo)
I(X, X,1X,) Dww(P((X,X,,p....e X.))
log
(3)
This measure can trivially be extended to the case where there exist multiple observed and unob- served confounding variables. The expression P(X, [dx]]do(,,, (X,) X.) means conditioning on X, in the interventional distribution P(X, ())). Now, the conditional confounding can be measured as:
CNF-1(X,, x,(x.) = e min (Xx)).(Xx
Intuitively, by conditioning on an observed confounding variable X.. we control the association between X, flowing via X, and measure the influence via the unobserved confounding variables.
Beyond Pairwise Confounding: We now study when a set Xs of variables where [xl > are jointly confounded i.e., share a common confounding variable and how to measure the joint confounding among the variables Xs.
Theorem 4.1. A set of observed variables X are jointly unconfounded if and only if there exists three variables X,,X,
x,, X2 Xx such that I(X; + x,)x,) I([X, Xx} - X2).
We now define the measure of confounding among the variables in Xs as follows.
CNF-I(Xs) = ECNP-I(Xs(():.. X.)
(5)
iES
Conditional confounding among a set of variables can be defined similarly to Eqn. 4. We now study some useful properties of the measure CNF-1.
Theorem 4.2. For any three observed variables X,, Xo and an unobserved confounding variable Z, the following statements are true for the measure CN F-1.
1. (Reflexivity and Symmetry.) CNFF-1(X, (X,,,x,,,,
CN X((o.)
2. (Positivity.) CNF-1(X,,X,)) > 0 if and only if X,, X2 are confounded. Given an observed confounding variable X. between X, CNF-1(X,; X,,,xyKKK XzlXx) > O if and only if there exists an unobserved confounding variable Z between X,,
3. (Monotonicity.) CNF-1 X2) CNF-1(X,, Xi) implies that the pair of variables X,, X; are more strongly confounded than the pair of variables Xx, in the sense of Defns. 4.2 and 4.3.
4.2 Setting 2: Detecting and Measuring Confounding Using the Mechanism Shifts of 7.
The previous setting utilizes the interventional effects of Xx(X)) on XX,),) to define a measure of confounding between X,, Xz-- In this setting, we utilize the association between the observed marginal distributions of X,, X, under causal mechanism shifts of to measure confounding. To this end, similar to [38], we make the following assumption.
A Assumption 4.1. (Shift Faithfulness [38]) Let Z be a common parent for a set of variables Xs C X. Then each causal mechanism shift in 7. between two contexts c, c' entails a causal mechanism change in each X; E Xs between the same contexts c, c'.
One consequence of the Assumption 4.1 is that change in the causal mechanism of Z induces correlations between the expectations of X,, X, in different contexts. To understand this, consider the following structural equations.
Z ~ N(((c).
X aZ +
x, BX, + 7Z +
(6)

---

Where denotes the context and Eg and Ey are noise variables with zero mean and have no additional restriction on the underlying probability distribution. The causal graph corresponding to this model has the nodes X; Xj. Z and edges: Z + X,, 7 - Xj, X; - Xj. It is easy to see that R(X,) = ay((c) and (E(X,) = (a. + 7))(c). Following Assumption 4.1, whenever there is change in causal mechanism of Z (e.g., changes to ē in Eqn. 6), there is a change in both E(X,) E(X,) Additionally, since Z is a common cause of both X., Xy, there is a spurious association between E(X,), E(X)). Subsequently, in the set of contexts (( the values EX(X)), EE(2)) are spuriously associated. Under Assumptions 3.2 and 4.1, restricting our analysis to C(J)() ensures that with high probability, the association between EE (X,),, IE(x.) due to the confounding variable Z. In this example, the association between ()),), B(x;) exists even if / = 0, i.e., X; A Xj- To define confounding measure, we create two random variables Et, Eg which we define as Ff (xxx)=((x)) (Xi), Ff (x,xP=(x,)(x) respectively where ((g)a)) Relying on the context information C(aA1 and utilizing the association between E and E we define a confounding measure as follows.
Proposition 4.2. (Confounding Based on Mutual Information) If two variables X,, X are con- founded by a variable Z. the induced random variables Eς, ES as described above have non zero mutual information T T :.))-
Definition 4.6. (Confounding Measure 2) When the causal mechanism shifts are observed for X,, X3 in different contexts and the contexts C(syn1)) are known, under the Assumptions 3.2-4.1, I, the measure of confounding CN F-FF-,X,, X3) between X; and X is defined as
(N, = 11 -
To measure the unobserved confounding strength when we already observe a confounding variable X.. we condition on the observed confounding variable X. to define XX, as follows.
CNF-2(XX, X,(X,)= = 1 - e T(E : (x,)
(8)
Following earlier work [38]. we utilize total correlation among triplets EE. E of random variables in { hes to verify whether a set of variables Xs are jointly confounded. By Assumption 4.1, we know that the variables in X5 jointly confounded only each pair XJ; i.j E S is pairwise confounded. If all three variables share the same latent confounding variable Z, then knowing about one of E.C ET. E explains away some of the associa- tion between the other two, so that we have T(EE 1 < <T(FF F..). However, for a triplet X,, Xx). it is possible that, rather than jointly confounded, there may be three disjoint confound- ing variables Zi2, Zia, Z23 confounding each of the individual pairs: (X,, (X,, Xx), (Xx, X,). In general, for a set of variables of size to permit such equivalent explanation, we would need to have a total of (-) confounding variables with 8(8 - 1) outgoing edges to obtain the same structure of pairwise confounding [38]. While this may plausibly occur for small sets of variables that appear to be pairwise correlated, we assume the true graph G to be causally minimal in the following sense.
Beyond Pairwise Confounding:
Assumption 4.2. (Confounder Minimality [38]) For every subset Xs of at least |S| 2 4 variables, there are at most 2|S] edges incoming into Xs from latent confounding variables with at least three children in Xs-
Assumption 4.2 ensures that variables that appear to be jointly confounded are indeed confounded. In other words, when a small number of latent variables suffice to explain the observed correlations, there should indeed exist only few confounding variables. With this assumption, we can guarantee that joint confounding can be identified from the total correlation.
Theorem 4.3. Let X be set of variables such that all X;, Xj E Xs are pairwise confounded. Then Xs is jointly confounded if and only if for each triple X,, Xj, Xx E Xx we have I(E;; ; E (EE) < ((E;ED))
Now, the measure of joint confounding among a set of variables Xs can be defined using total correlation T ..... E(s)) as follows. To evaluate the following expression, we need to use the contexts C(1ju- (iSs)) to ensure that with high probability, the association among the variables in Xs is due to the joint confounding variable Z.
CNF = =1- F.r,,,,.,....
(9)
7

---

Theorem 4.4. For any three observed variables X, Z, the following statements are true for the measure CN F-2.
x, Xo and an unobserved confounding variable
1. (Reflexivity and Symmetry.) ) C = 1- -- (x, Vi where H (.l.) denotes conditional entropy and CNF- - X; X.))
(Positivity.)) CN,-2(XX,,,) > 0 if and only if Xx, X, are confounded. Given an observed confounding variable X. between X,, X,. CNF-2( (,,, Xj(X,) > 0 if and only if there exists an unobserved confounding variable Z between X;, X3-
3. (Monotonicity.) CN X2) > CN x(,, Xi) implies that the pair of variables X,, X, more strongly confounded than the pair of variables Xg, Xt in the sense of Defn.. 4.2.
4.3 Setting Observing the Causal Mechanism Shifts in and Known Causal Path Direction Between X, and X
Similar to the previous settings, we utilize marginal and conditional distributions of X,, X, to define measure of confounding. By prior knowledge, if we know the direction of causal path between X,, X,, we can utilize the causal direction to measure confounding as explained below. In addition to the notations E E. introduced in the previous setting, let us denote for each e E C()) (x,xP X, (X,), xx,-P=xXXX (x,,,x,XXX,,,,,XXXXXXXX (X, with F.G E respectively. We now leverage dependency among these variables define the measure of confounding. Intuitively, if S
X; + Xj and if we observe change in the causal mechanisms of both X, X; due to the causal mechanism changes in Z, we also observe change in the causal mechanism (X))
Definition 4.7. (Confounding Measure 3) When the causal mechanism shifts are observed for X,, X3 and the causal direction between the nodes Xi, X; is known, under the Assumptions 3.2-4.1, the measure of confounding CN3 (X,, Xj) between X E X and X; X is defined
e ((5San)) is X; X X3
CN,, = 1- 1(- if -, X₃
(10)
CNE-2(XX,X))) Otherwise
To measure the unobserved confounding strength in the presence of an observed confounding variable X., similar to setting 2, we can modify Eqn. 10 to condition on the variable X..
Beyond Pairwise Confounding: Using the Assumption 4.2, have the following.
Theorem 4.5. Let X5 be set of variables such that all X;, X2 E Xg pairwise confounded and the causal relationships among each pair X,, X Then X is jointly confounded if and only if for each triple X4, Xx E Xg we have I E E < I(E E..)
Since we have access to random variables EG in addition to E L.C it is not straightforward to use all of them to measure joint confounding. To keep the measure simple, we let the measure of joint confounding among the variables Xs be the same as CNF-2(Xs). That is, CN F-3(s)) CN F-2((s).. Setting 3 is an alternative to Setting 2 when we know the direction of the causal path between X,, X,. Settings and act as complementary to each other in validating the correctness of our analysis.
Theorem 4.6. For any three observed variables X,, X,, Xo and an unobserved confounding variable Z, the following statements are true for the measure CN F-3.
1. (Reflexivity and Symmetry.) CNF-3(X,, = conditional entropy and CNF-3(X,, X,Xx)) CNF-3 X,)).).
(X,))x))))))))))))) Vi where H().)) denotes
(Positivity.) CN (1,, > 0 if and only if X1, Xy are confounded. Given an observed confounding variable X, between X,, X,, CNF-3(X, (X,, X, Xx)X > 0 if and only if there exists an unobserved confounding variable Z between X,, X,,
(Monotonicity.) CNFF3(X,, X,) > CNF-3(Xx, Xi) implies that the pair of variables X,, X; more strongly confounded than the pair of variables Xx, X₁ in the sense of Defn. 4.2.
8

---

5 Algorithm
Algorithm I outlines the procedures to measure confounding in all three settings and can be extended to the case where we evaluate conditional confounding and evaluating confounding among multiple variables. We present two real-world examples where our methods can be applied in Appendix & B.
Algorithm 1: Algorithm for evaluating pairwise CNF-L,
Data: Context information C(gAAPPys CLAA-PHs Cuya(), Contextual Datasets (D) JEE.
Result: CNF-1 X,), CNNF2 X,), CNF-3(X,
Step Evaluate P():(X;), P(X
P(X)) Xx)) using observational data;
Step 2:
Evaluate (((x)) using (De YOA .ry.
Step 3:
Evaluate ((X))ld((.)) using (Dr JoECCIAN
Step 4:
Evaluate I(X; X;), I(X; X:);
Step 5:
CNF-I(X,,,
1 e min(((XX XX)
Step 6:
Evaluate E. Ej using (D) DeEC
Step CN 1 :EEE)
Step Evaluate E.j, Efi using (Dr Jecd
compute CN Xj) according Defn. 4.7; return
Step 9:
CX) CNF-2 CNF-3(X,, Xz)
6 Experiments and Results
We perform simulation studies to verify the correctness of the proposed measures. All the experiments are run on a CPU. We report the mean and standard deviation of results taken over five random seeds. Code to reproduce the results is presented in the supplementary material. Code is available at https://g gi thub. com/ htt///auttnnn gautam0707/CC CNC..
Measuring Confounding: In this set of experiments, we consider the following four causal structures made of three nodes X;, X,, Z: G1 : Empty graph over Z, X;, Xj i.e., nodes are isolated in the graph, G2 : :x; + Xj. Ga 7 - X, Z + X, G4 2 + X,, - X,, X; - X,. In G1, G2, there is no confounding between X,, X3 and in G3, 94 there is confounding effect of Z on X; and Xz. Results in Fig. 2 show that our measures output zero when there is no confounding between X,, X, and output positive values when X,, X, are confounded by confounding variable Z.
Setting
Setting
Setting 3
CNr 10K, Xt
0.115
(CnD
CUr 2M,
0.2
&
300000
0.2
0.10
(Cnf) 9
(Cnf) 9.
(Cnf) Gi
(Uncat)
0.1
(Uncnt)
0.05
(Uncpf) di...
(Uncnt) c.
(Uncnf) &a
(Uncnf) 9.
0.00
1000
2000
3000 4000 5000
1000
2000 Number of Samples
3000
4000 5000 1006 2000
3000 4000 5000 Number of Samples
Number of Samples
Figure 2: Mcasure of confounding between pair of variables X,, X,. Our measures output zero when there is no confounding between X,, X; and output positive values when X,, X, are confounded.
Measuring Conditional Confound- ing: We consider the following two causal structures.
0.25
0
95
Z1
X,, 7 + X, 7-2 4 X, 2g
g 024
PUX X|el
0015
CWF- X(X
Cio
2XX 1141
CWF 2XX
X, + Xj. Go
7 - X1, 2
0.10
X, + X, In Gs. X; and X, are confounded by two variables Z,, Zz. We measure conditional confounding between X, X, conditioned on 0. Zi. and Z₂ respectively. Since confound- ing still exists all of the above con- ditioning settings, C-F-2 correctly returns positive confounding value in all three cases (see Fig. 3 left). On the other hand, in Go. we measure conditional confounding
022
1000
2000
3000
4000
5000
Number Samples
Number Samples
Conditioning one of 0, Zı, Zu will not remove confounding between X.,X, in G. Hence CN returns posi- tive values. Right: In Go. conditioning on 0 does not remove the confounding effect of Z on Xi, Xy. Hence, we observe a positive value for CNF-2(X, (X,, (6). Conditioning on Z will block the confounding hetween X,, X,. Hence CN F-2 is closer to zem.
Figure 3: Left:

---

between X, X, conditioning on empty set and Z. Since conditioning on Z will block the confound- ing association between X,, X,, CNF-2 returns confounding value closer to zero. However, the unconditioned confounding (conditioning on empty set) value is still large. These results empirically validate the correctness of the proposed measures.
Downstream Causal Er- feet Estimation: For the causal graphs G3, G4, we ex- amine the impact of control- ling for nodes identified us- ing our method. We mea-
Causal Gruph
Not Controlling Confounding
Controlling Confounding
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Go
0.55
0.57
0.55
0.52
0.52
0.06
0.02
0.007
0.03
0.009
94
0.24
0.26
0.23
0.24
0.23
0.04
0.05
0.06
0.02
0.05
Table 3: Downstream application causal effect estimation.
sure the causal effect of X;
on X; with and without controlling for the detected confounding variable and report the absolute difference between the true and estimated causal effects in Tab. 3. The results show that controlling for the variables identified by our method reduces the bias in the estimated causal effects.
To verify the performance of our method on large scale, similar [38], generate causal graphs of various number nodes using Erdös-Rényi model. In these experiments, each context is a result of intervention on one node. This is the reason for having the same value for number of nodes N and number of contexts |C|. Sample size denotes the number of data points used in each context. We detect and measure whether each pair of nodes confounded not. We then calculate the Precision, Recall, and Fl scores. Our confounding measures obtain good results across all settings.
Binary Data Erdös-Reee Rényii Causal Graphs:
Setting
Setting
Setting
[C]
Sample Size
Precision
Recall
F1
Precision
Recall
F1
Precision
Recall
F1
100
0.64
0.97
0.77
0.67
0.83
0.74
0.64
0.72
0.68
200
0.64
1.0
0.78
0.83
0.74
0.79
0.74
300
0.64
1.0
0.78
0.67
0.83
0.74
0.65
0.76
0.70
10
400
0.64
1.0
0.78
0.67
0.83
0.74
0.67
0.83
0.74
10
500
0.64
1.0
0.78
0.67
0.83
0.74
0.67
0.83
0.74
15
100
0.81
0.95
0.88
0.80
0.85
0.82
0.80
0.79
0.80
15
200
0.82
1.0
0.90
0.80
0.85
0.82
0.80
0.85
0.82
15
300
0.82
1.0
0,90
0.80
0.85
0.82
0.80
0.85
0.82
400
0.82
1.0
0.90
0.85
0.82
0.80
0.85
0.82
500
0.82
0.90
0.85
0.82
0.84
0.82
100
0.68
0.95
0.80
0.88
0.77
0.69
0.84
0.76
200
0.69
0.82
0.68
0.88
0.77
0.87
0.76
20
300
0.69
1.0
0.82
0.68
0.88
0.77
0.67
0.86
0.75
20
400
0.69
1.0
0.82
0.68
0.88
0.77
0.68
0.87
0.76
20
500
0.69
1.0
0.82
0.68
0.88
0.77
0.68
0.87
0.76
25
100
0.83
0.96
0.89
0.83
0.91
0.87
0.83
0.89
0.86
25
200
0.83
1.0
0.91
0.83
0.91
0.87
0.82
0.90
0.86
300
0.83
1.0
0.91
0.83
0.91
0.87
0.83
0.91
0.87
400
0.83
0.92
0.87
0.87
500
0.83
0.91
0.83
0.91
0.87
0.83
0.91
0.87
Table 4: Results on synthetic datasets for settings 1,2,3.
7 Conclusions, Limitations, and Future Work
In this paper, based on the known causal mechanism shifts of observed variables, we propose three measures of confounding along with their conditional and multivariate variants. We also study key properties of these measures. Our measures complement each other depending on the available context information. We propose algorithms to compute the proposed measures and empirically verify their correctness. However, for the same confounded pair of variables, our metrics may yield different results depending on the chosen measure. As discussed in the introduction, the measures are intended to assess the relative strengths of confounding rather than for point-to point-to-point comparison. The number of contexts required to evaluate the measure can be large because many contexts without changes in particular mechanisms are discarded. Identifying appropriate real-world datasets and applying the proposed measures to those datasets an interesting area for future work, as is developing measures that efficiently use context information. Additionally, devising new definitions for confounding and proposing corresponding confounding measures is also an interesting future direction. We aim to pursue these ideas.
10

---

Acknowledgments
This work was partly supported by the Prime Minister's Research Fellowship (PMRF) program and Google Research Scholar Award. We are grateful to the anonymous reviewers for their valuable feedback, which improved the presentation of the paper.
References
[1] John Aldrich. Correlations genuine and spurious in pearson and yule. Statistical science, pages 1995.
[2] Rohit Bhattacharya, Tushar Nagarajan, Daniel Malinsky, and llya Shpitser. Differentiable causal discovery under unmeasured confounding. In International Conference on Artificial Intelligence and Statistics, pages 2314-2322, 2021.
[3] Norman E. Breslow, Nicholas E Day, and Elisabeth Heseltine. Statistical methods in cancer research. 1980.
[4] Philippe Brouillard, Sébastien Lachapelle, Alexandre Lacoste, Simon Lacoste-Julien, and Alexandre Drouin. Differentiable causal discovery from interventional data. In Advances in Neural Information Processing Systems, volume 33, pages 21865-2188 2020.
[5] Esben Budtz-Jergensen, Niels Keiding, Philippe Grandjean, and Pal Weihe. Confounder selection in environmental epidemiology: Assessment health effects of prenatal mercury exposure. Annals of Epidemiology, 17(1):27-35, 2007.
[6] Timothy A Carey and William Stiles. Some problems with randomized controlled trials and some viable alternatives. Clinical Psychology & Psychotherupy, 23(1):87-95, 2016.
[7] Venkat Chandrasekaran, Pablo A Parrilo, and Alan S Willsky. Latent variable graphical model selection via convex optimization. 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 1610-1613. IEEE, 2010.
[8] David Maxwell Chickering. Learning equivalence classes of bayesian-network structures. The Journal of Machine Learning Research, 2:445-498, 2002.
[9] Diego Colombo, Marloes Maathuis, Markus Kalisch, and Thomas S Richardson. Learning high-dimensional directed acyclic graphs with latent and selection variables. The Annals of Statistics, pages 294-321, 2012.
[10] Shai Ben David, Tyler Lu, Teresa Luu, and Dávid Pál. Impossibility theorems for domain adaptation. In A/STATS, pages 129-136, 2010.
[11] Mirthe Maria Van Diepen, Ioan Gabriel Bucur, Tom Heskes, and Tom Claassen. Beyond the markov equivalence class: Extending causal discovery under latent confounding. In 2nd Conference on Causal Learning and Reasoning, 2023.
[12] Frederick Eberhardt and Richard Scheines. Interventions and causal inference. Philosophy of science, (4(5):981-995 2007.
[13] Frederick Eberhardt, Clark Glymour, and Richard Scheines. On the number of experiments sufficient and in the worst case necessary to identify all causal relations among variables. arXiv preprint arXiv: 207. 1389, 2012.
[14] Robin J Evans and Thomas S Richardson. Smooth, identifiable supermodels discrete dag models with latent variables. Bernoulli, 18848-876, 2019.
[15] Sander Greenland and Hal Morgenstem. Confounding in health research. Annual review of public health, 1891:289-212, 2001.
[16] R.H.H. Groenwold, F. Hak, and A. Hoes. Quantitative assessment of unobserved confounding mandatory in nonrandomized intervention studies. Journal of Clinical Epidemiology, 62(1): 22-28. 2009.

---

[17] Siyuan Guo, Viktor Tóth, Bernhard Schölkopf, and Ferenc Huszár. Causal de finetti: On the identification of invariant causal structure in exchangeable data. In Thirty-sevents Conference on Neural Information Processing Systems, 2023.
[18] Gemma Hammerton and Marcus R Munafò. Causal inference with observational data: the need for triangulation of evidence. Psychological medicine, 51(())563-577 2021.
[19] Alain Hauser and Peter Bühlmann. Two optimal strategies for active learning of causal models from interventional data. International Journal of Approximate Reasoning, 55(4):926-939, 2014.
[20] Patrik O Hoyer, Shohei Shimizu, Antti J Kerminen, and Markus Palviainen. Estimation of causal effects using linear non con-gaussian causal models with hidden variables. International Journal of Approximate Reasoning, 49(2):362-378 2008.
[21] Biwei Huang, Kun Zhang. Jiji Zhang, Ruben Sanchez- Romero, Clark Glymour, and Bernhard Schölkopf. Behind distribution shift: Mining driving forces of changes and causal arrows, In 2017 IEEE International Conference on Data Mining (ICDM), pages 913-918. IEEE, 2017.
[22] Biwei Huang, Kun Zhang, Jiji Zhang, Joseph Ramsey, Ruben Sanchez-Romero Clark Glymour, and Bernhard Schölkopf. Causal discovery from haterogeneous/noonstttionny data. Journal of Machine Learning Research, 21 2020.
[23] Amin Jaber, Murat Kocaoglu, Karthikeyan Shanmugam, and Elias Bareinboim. Causal discovery from soft interventions with unknown targets: Characterization and learning. Advances in neural information processing systems, 33:9551-9566 2020.
[24] Holly Janes, Francesca Dominici, and Scott Zeger. On quantifying the magnitude of confound- ing. Biostatixtics, 1 (3)55522-588, 2010.
[25] Dominik Janzing and Bernhard Schölkopf. Detecting confounding in multivariate linear models via spectral analysis. Journal of Causal Inference, 6(1):20170013, 2018.
[26] Andrew Jesson, Alyson Rose Douglas, Peter Manshausen, Maëlys Solal, Nicolai Meinshausen, Philip Stier, Yarin Gal, and Uri Shalit. Scalable sensitivity and uncertainty analyses for causal- effect estimates of continuous-valuee interventions. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[27] David Kaltenpoth and Jilles Vreeken. Nonlinear causal discovery with latent confounders. In International Conference on Machine Learning, pages 15639-15654, 2023.
[28] David Kaltenpoth and Jilles Vreeken. Causal discovery with hidden confounders using the algorithmic Markov condition. In Proceedings of the Thittt-Ninth Conference Uncertainty in Artificial Intelligence, pages 1016-1026, 2023.
[29] Rickard Karlsson and Jesse Krijthe. Detecting hidden confounding in observational data using multiple environments. Advances in Neural Information Processing Systems, 36, 2023.
[30] David G Kleinbaum, Kevin M Sullivan, and Nancy D Barker. pocket guide to epidemiology. Springer, 2007.
[31] Charles Ksir and Carl L Hart. Correlation still does not imply causation. The Lancet Psychiatry, 3(5):401, 2016.
[32] Paul H Lee. Is a cutoff of 10% appropriate for the change-in-estimane criterion of confounder identification? Journal of epidemiology, 24(2):161-167, 2014.
[33] Adam Li, Amin Jaber, and Elias Bareinboim. Causal discovery from observational and interven- tional data across multiple environments. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[34] George Maldonado and Sander Greenland. Simulation study of confounder--ellectoon strategies. American journal of epidemiology, (38(11):923996, 1993.
12

---

[35] George Maldonado and Sander Greenland. Estimating causal effects. International journal of epidemiology, 31(2):422 429, 2002.
[36] Sarah Mameche, David Kaltenpoth, and Jilles Vreeken. Discovering invariant and changing mechanisms from data. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1242-1252, 2022.
[37] Sarah Mameche, David Kaltenpoth, and Jilles Vreeken. Learning causal models under indepen- dent changes. Advances in Neural Information Processing Systems, 36, 2023.
[38] Sarah Mameche, Jilles Vreeken, and David Kaltenpoth. Identifying confounding from causal mechanism shifts. In Proceedings of the 27th International Conference Artificial Intelligence and Statistics (AISTATS). PMLR, 2024.
[39] Olli S Miettinen and E Francis Cook. Confounding: essence and detection. American journal of epidemiology, 114(4):593-60) 6039, 1981.
[40] Joris M Mooij, Sara Magliacane, and Tom Claassen, Joint causal inference from multiple contexts. Journal of Machine Learning Research, 21:1-108, 2020.
[41] Austin Nichols. Causal inference with observational data. The Stata Journal, 7(4):507-541, 2007.
[42] Juan Miguel Ogarrio, Peter Spirtes, and Joe Ramsey. A hybrid causal search algorithm for latent variable models. In Proceedings of the Eighth International Conference on Probabilistic Graphical Models, pages 368-379, 2016.
[43] Menglan Pang, Jay S Kaufman, and Robert W Platt. Studying noncollapsibility of the odds ratio with marginal structural and logistic regression models. Statistical methods in medical research, 25(5): 1925-1997771 2016.
[44] Judea Pearl. Causality. Cambridge university press, 2009.
[45] Ronan Perry, Julius Von Kügelgen, and Bernhard Schölkopf. Causal discovery in heterogeneous environments under the sparse mechanism shift hypothesis. In Advances in Neural Information Processing Systems, pages 10904-10917, 2022.
[46] Jonas Peters, Joris M Mooij, Dominik Janzing, and Bernhard Schölkopf. Causal discovery with continuous additive noise models. The Journal of Machine Learning Research, 15(1): 2009 2000,2053,200 2014.
[47] Jonas Peters, Dominik Janzing, and Bernhard Schlkopf. Elements of Causal Inference: Founda- tions and Learning Algorithms. The MIT Press, 2017.
[48] Maxim Raginsky. Directed information and pearl's causal caleulus. In 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 958-965, 2011.
[49] Thomas S Richardson, Robin J Evans, James M Robins, and Ilya Shpitser. Nested markov properties for acyclic directed mixed graphs. The Annals of Statistics, 51(1):334-361 2023.
[50] Robert William Sanson-Fished. Billie Bonevski, Lawrence W. Green, and Cate D' D'Este. Limi- tations of the randomized controlled trial in evaluating population based health interventions. American Journal of Preventive Medicine, 33(2):155- 161, 2007.
[51] Mauro Scanagatta, Cassio de Campos, Giorgio Corani, and Marco Zaffalon. Learning bayesian networks with thousands of variables. Advances in neural information processing systems, 28, 2015.
[52] B Schölkopf. D Janzing, J Peters, E Sgouritsa, K Zhang, and J Mooij. On causal and anticausal learning. 29th International Conference on Machine Learning (ICMI. 2012), pages 1255- 1262. International Machine Learning Society, 2012.
13

---

[53] Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5)x612-634, 2021.
[54] Noah A Schuster, Jos WR Twisk, Gerben Ter Riet, Martijn W Heymans, and Judith JM Rijnhart. Noncollapsibility and its role in quantifying confounding bias in logistic regression. BMC medical research methodology, 21:1-9, 2021.
[55] Karthikeyan Shanmugam, Murat Kocaoglu, Alexandros G Dimakis, and Sriram Vishwanath. Learning causal graphs with small interventions. Advances in Neural Information Processing Systems, 28, 2015.
[56] lya Shpitser, Robin Evans, Thomas Richardson, and James M Robins. Introduction to nested markov models. Behaviormetrika, 41:3-39, 2014.
[57] llya Shpitser, Robin J Evans, and Thomas S Richardson. Acyclic linear sems obey the nested markov property. In Uncertainty artificial intelligence: proceedings of the... conference. Conference Uncertainty in Artificial Intelligence, volume 2018. NIH Public Access, 2018.
[58] Edward H Simpson. The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society: Series B (Methodological), 13(2):238-22 241,, 1951.
[59] Peter Spirtes and Kun Zhang. Causal discovery and inference: concepts and recent methodolog- ical advances. In Applied informatics, volume 3, pages 1-28. Springer, 2016.
[60] Peter Spirtes, Clark N Glymour, and Richard Scheines. Causation, prediction and search. 2000.
[61] Zhiqiang Tan. A distributional approach for causal inference using propensity scores. Journal of the American Statistical Association, 1016476) 2006.
[62] Tyler VanderWeele and Ilya Shpitser. On the definition of confounder. Annals of statistics, 41(1):196, 2013.
[63] Y. Samuel Wang and Mathias Drton. Causal discovery with unobserved confounding and non-gaussian data. Journal of Machine Learning Research, 24(271):1-61, 2023.
[64] Aleksander Wieczorek and Volker Roth. Information theoretic causal effect quantification. Entropy, 21(10), 2019.
[65] Alessio Zanga, Elif Ozkirimli, and Fabio Stella. survey on causal discovery: Theory and practice. International Journal of Approximate Reasoning, 151:101-129, 2022.
[66] Xun Zheng, Bryon Aragam, Pradcep K Ravikumar, and Eric P Xing. Dags with no tears: Continuous optimization for structure learning. Advances in neural information processing systems, 31, 2018
14

---

Appendix
A Proofs
Proposition 4.1. (Identifiability of P(X, (do((X)))) P(X)) is identifiable from the set of contexts Cay--Pe To detect and measure confounding between a pair of nodes X,, X,, enough to observe two sets of contexts (s)n-Pss and (j)--ps Thus, sets of contexts are needed to detect and measure confounding between distinct pairs of nodes in causal DAG with nodes.
Proof. Since the set of contexts CHAA,, consist of data with all possible interventions X;, context e is generated by performing intervention on X; with the value Iú, the expression z4)) is equal to the expression P(Xy|X, = z;) in that context c.
From Defn. 4.4, to detect and measure confounding between the pair of variables X,, Xy. we need to evaluate P(Xy [do[] Xx)) and PPXX, (X))))) To this end, from the previous paragraph, we need two sets of contexts (.) and C()) Following these observations, is enough have n sets of contexts to detect and measure confounding between
(2) distinct pairs of nodes.
Theorem 4.1. A set of observed variables X are jointly unconfounded if and only if there exists three variables X,, x, Xx Xg such that I(X; X, I(IX, Xx) + X,).
Proof. Consider three variables x, Xj, X in the underlying causal graph. Consider the conditional directed information between X;, X, given X₂ and the subsequent manipulations as follows.
X,, X&)
((, + X,(x,) X,, log
F(X,Ido(X (x,)) Xx))
P(X,, Xx(X,)
P(Xuldo(X)))
XE.(X.,X,X.)) log
x
P(X;, xx))do(x)))
P(X+|X,)
P(X;,, Xx(X,)
(BF(K, log P(Xx(X;)
= Xpp( log
P(X, Xx xx|oo(X)))
P(X
=]([X,Xa) + X,) - 1(xx - X,)
Since T(Xx - x,) 2 0, we have T(X; X,(Xa) S T([X,X)) + X2). Equality holds only when Xg. X, are unconfounded.
Theorem 4.2. For any three observed variables X,, X,, X, and an unobserved confouuddig variable 7, the following statements are true for the measure C-F-1.
1. (Reflexivity and Symmetry.) CNFF-(XX,
CNF-I (NV X.(X.).
(Positivity.) CN,- if and only if X;, Xj are confounded. Given an observed confounding variable X. between X., Xj, CNF-1(X. XjlX,) > 0 if and only if there exists an unobserved confounding variable Z between X;, Xi,
(Monotonicity.) CNF-1(X,, Xz) > CNF-1 X1) implies that the pair of variables X,, X are more strongly confounded than the pair of variables Xk, Xt in the sense of Defns. 4.2 and 4.3.
Proof. Reflexivity: From the definition of directed information, 1(X, + -x(Xx)) =
[X.. = 0 and hence CN, = 1 c 0. X,, P(X,
Symmetry: Even if I(X; X,)X)), I(X, + X,IX.))' is symmetric and hence CNF-11XX,,
X,)X,) is not symmetrie, the expression 'min(I(X;
symmetric.
Positivity: If X, X, are confounded, irrespective of the direction of the causal path between X, and X,, we have P((,)|,)) + (do
and P (X, o IP(X;: dx))).). Hence I(X; - X,) > and T(X, + X,) > O. We now have CNF-1(X, X₂) > O. The above statement is true even if there no causal path between the nodes X,, X,. The above statements are valid even after conditioning on an observed confounding variable X, if there is unobserved confounding between X,,
15

---

Monotonicity: Without loss of generality, assume that the inequality CN,-I(XX,,,, >
CN;,-1(X;,,) X1) is result of [X,; +xX,,,,) I(X; ++ Xi). That is, the KL divergence between PX);|X;) and P(X;/d(( x))) is greater than the kl divergence between [P(xa] Xt) and P(Xa (x)).
That is, the pair of distributions (P( and (P(xxldo(X) (x)) are closer to each other compared to the pair p)) and PXX, xxIoo((x))) (x)). As a result, Xa: X₁ are closer to being not confounded in the sense of Defns. 4.2 and 4.3.
Proposition 4.2. (Confounding Based on Mutual Information) two variables X Xj are con- founded by a variable Z, the induced random variables E F as described above have non zero mutual information T T FG))
Proof. There are two sources of dependency between E DJ. If X;, are causally related in the underlying causal model generating the data, there will be a dependency between E,, EES in the context (()))))) as the interventions are soft. On the other hand, as per the Assumption 4.1, any shift in the causal mechanism of Z leads change in both the mechanisms of X, X, leading to dependency. Hence the random variables Ef. Ey have non-zero mutual information.
Theorem 4.3. Let X be a set of variables such that all X,, X, E X are pairwise confounded. Then Xs is jointly confounded if and only if for each triple X,, X,, Ny E Xg we have I ( <
T F
Proof. Following the Assumption 4.2, when three variables Xi, Xj, Xx are confounded by as single confounding variable Z, conditioning on one E. E.. Eg explains away some of the dependency between other two. Hence we have T[E;; ( < ; for all triples i, k.
Theorem 4.4. For any three observed variables X,, X,, X. and an unobserved confounding variable 2, the following statements are true for the measure F-F-2.
1. (Reflexivity and Symmetry.) 2X,
1- H(E()x) Vi where H.I..) denotes conditional entropy and (XN = CNF-2 X,
(Positivity.) C-, > O if and only if X,, X, are confounded. Given an observed confounding variable X. between X,, CNF-2(X, X, Xx)) >> 0 if and only if there exists an unobserved confounding variable 7 between X,, Xj.
3. (Monotonicity.) C2,, X,) > CNF-2(XF;; Xi) implies that the pair of variables X,, X3 are more strongly confounded than the pair of variables Xx, X1 in the sense of Defn. 4.2.
Proof. Reflexivity: from the definition of mutual information, I E Xx)) ((E(1X))
(E EP, X.) = (EE Substituting in the definition of CN CNF-2(X,, Xy), result follows.
Symmetry: The result follows from the 'symmetry' property of mutual information.
Positivity: If X,, Xj are confounded, from the Assumption 4.1, E.C , Ee are dependent random variables. Hence the mutual information is positive. The result follows after substituting some positive value for TEE;; E in the definition of CNF-2 Xz). The same argument goes for conditional confounding.
Monotonicity: from the definition of C- X,), CNF-2(X, X,) > CNF-2(X,,; Xt) im- plies T F29) > T(FE; F.F). From the Defn. 4.2, X,, X, have higher mutual information than the pair Xg, X1 and hence X, are more strongly confounded than Xx, X1.
L
Theorem 4.5. Lei X be set of variables such that all X X2 E Xg are pairwise confounded and the causal relationships among each pair X, X,, Then X. is jointly confounded if and only for each triple X,, X,,X, E X we have 1 LEE; E 19 < <IE E.))
Proof. Following the Assumption 4.2, when three variables X,, X,, Xx are confounded by single confounding variable Z, conditioning on E explains away some of the dependency between EG EC. Hence we have [EE;; E. ( < I(E5; Eg)) ) for all triples i j. k.
16

---

Theorem 4.6. For any three observed variables X,, x, X. and an unobserved confounding variable Z, the following statements are true for the measure CN F-3.
(Reflexivity and Symmetry.) CNF-3(X, X conditional entropy and CN X,lX.) CNF-3 X,
(X)))X)) Vi where H(-..) denotes
2. (Positivity.)) CNF-3(X,,X,) > if and only if X,, X, confounded. Given observed confounding variable X. between X,, X,, CN ENNF-3XX, ;,, XylX,) > 0 if and only if there exists an unobserved confounding variable 7 between X;, X--
3. (Monotonicity.) CNF-3(X,, Xj) CNF-3(X;; X() implies that the pair of variables X,, Xj are more strongly confounded than the pair of variables Xg. X, in the sense of Defnn 4.2.
Proof. Reflexivity: from the definition of mutual information, I(E,: 1X I(E;; E x H (Xx) H X.) H (X)). Substituting in the definition of CNF-3(X; Xj). result follows.
Symmetry: Since we rely on the direction of the causal path between X,, X,, for a given pair of nodes X,, X,, we have C-F-3( X;, Xj) CN F-,(X,, X,) from Defn. 4.7.
Positivity: If X,, Xj confounded and X; + Xj. from the Assumption 4.1, E,, E. are dependent random variables. Hence the mutual information Egg, F. is positive. The result follows after substituting positive value for / (E jis E in the definition of (N;,,(X;,X,), The same argument goes for conditional confounding.
Monotonicity: from the definition of (X, Xy). without loss of generality,
CN F-3 X2) CNF,33(Xx,X, implies [ Ef) > [ Ef). From the Defn. 4.2, X, have higher mutual information and hence are more strongly confounded than X Xt. O
B Real-world Examples
Pro
Edu
Exp
Lab
Wag
Inv
Figure 4: Two real-world examples where our method can applied. Here Pro: Production Volume, Exp: Exports, Lab: Total Labor Required, Edu: Education, Wag: Wages, Inv: Investments. We can perform interventions on the above variables and any combination thereof obtain context-specific data. We can use such data to identify and measure confounding by applying our methods.
17

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All the claims made in the abstract and introduction are supported with theory and experiments in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See 7 for the discussion on the limitations of this work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification:
For each of the theoretical results, proofs and the assumptions used are presented in the Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Experimental setup is presented in main paper and code to reproduce the results is made public. See $ 6.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
19

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Code reproduce the results is made public. See $
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/nips. cttpsscccppbllcccccccccc http:/pullcggguideeeeeeepp guides/ CoodeSummis ssion agionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental setup is presented main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
Each experiment is repeated for several random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
20

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Experimental setup is presented in main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httppr///////.....//e htt.biic/ethicsuudeeleeeeee
Answer: [Yes]
Justification:
Our research confirms with NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
To best of our knowledge, there are no detrimental impacts of our work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All papers, data, and code used are cited.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Code is documented and provided in supplementary material. Code will be made publicly available.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "BudjZLHs6V", "reviewer": "Reviewer_n1WD", "summary": "This paper presents methods that: \n1. define a measure of confounding between sets of variables,\n2. separate the effects of observed and unobserved confounders, and\n3.  assess the relative strengths of confounding between sets of variables,\n\nin three different settings. In each setting they assume that data from several different contexts is available, and that the changes in causal mechanisms between the different contexts are known.", "questions": "**line 128:** unclear notation, what information is $P_x(V), X \\subseteq V$ adding here? If $P_*$ is a set, why not write the set that defines it instead?\n**line 128:** (i.e., $X = \\emptyset$). why isn't $X$ bold?\n\n**Line 154:** \"Let $C_{S \\wedge \\neg R}$ be the set of contexts in which we observe mechanism changes for the set of variables $X_S$ but not for the variables $X_R$.\" Changes relative to what? The interventional distribution? Just between two contexts? Is the shift the same across each of the contexts in this set?\n\n**Line 158:** This does not look like a p-value. This looks like an indicator taking a value of true or false, not a probability. Is there a missing $P$ outside of the brackets? If so, what is the relevant random variable? $c$? If $c$, confusing to have it lower-case while a potential value $X_i$ is upper case. Same questions for **Assumption 3.2**.\n\n**Theorem 4.1:** missing comma? $\\\\{X_iX_j\\\\}$ -> $\\\\{X_i, X_j\\\\}$\n\n**Line 286:** Are the $\\epsilon$ normally distributed? or is zero mean and any distribution ok?\n\n**Definition 4.7:** mutual information is defined between random variables. Are these random variables with respect to contexts? If so lower case $c$ is a bit confusing.", "limitations": "The authors adequately discuss the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper provides thorough theoretical justification for their methods. The analyses given for the latter two settings seem particularly novel to me.", "weaknesses": "The assumption \"that the causal mechanism changes are known for each variable across different contexts\" seems to be quite strong, but is not mentioned until relatively late in the paper (line 160).\n\nThe formal definition of a mechanism shift comes quite late in the paper. It would be nice to have intuitive explanations in the abstract and introduction. I also wonder if the formal notation could start with the notion of multiple environments baked in. As it is written now, it is unclear what the mechanism shift is relative to. Is it a shift between an environmental distribution and the interventional distribution, or is it only necessary to know the relative shift between environments?\n\nThe relationship between definitions 4.2 and 4.3 and ignorability and conditional ignorability as independence and conditional independence assumptions needs to be added to the related works. There is also perhaps a connection to [Scalable Sensitivity and Uncertainty Analyses for Causal-Effect Estimates of Continuous-Valued Interventions](https://arxiv.org/pdf/2204.10022) that could be discussed. Notably, the relationship between the likelihood ratios and the KL divergence between the interventional and observational distribution mentioned in Appendix A.3.1.\n\nWhile the theoretical results are thorough, the empirical results are very limited. A major concern with hidden confounding is that it can arbitrarily bias estimated treatment effects. The magnitude of the bias should be straightforward to calculate in your synthetic experiments. It might be more demonstrative to show how the proposed measures correlate to induced biases.\n\n## Points of confusion\n\n**Definition 3.1**\n- Looks like $X$ is being redefined here as to include any variable in $V$ rather than the subset that excludes $Z$. Mildly confusing.\n- line 131: $P(X_i \\mid \\mathrm{PA}_i)$ is not defined in definition 3.1\n\n**Assumption 3.1**\n- talks about changes in causal mechanisms, but this concept hasn't been formalized yet which makes this assumption hard to understand.\n\n**Line 147:** what is a context node? what is an extended causal graph? Might help to define context specific distributions before rather than after.\n\n**Line 156:** two two\n\n**Line 156:** $P^c$ notation could be introduced earlier to formalize a mechanism shift.\n\n**Assumption 3.3:** $dPC$ is missing brackets\n\n**Line 182.** shifts -> shift\n\n**Line 183:** \"extent of hidden confounding.\" word choice is a bit vague. maybe strength, or the effect of the confounder on each variable, or how much it may bias some estimand.\n\n**Definition 4.1:** The notation is a little awkward with the condition on $P(X_j)$ within $D_{\\mathrm{KL}}$. \n\n**Theorem 4.2:** strongly -> more strongly\n\n**Definition 4.6:** missing definition of mutual information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The assumption \"that the causal mechanism changes are known for each variable across different contexts\" ..\n\nTo measure confounding between **all pairs** of nodes in a causal graph, we need to know the mechanism changes for **each variable** across contexts. However, if the number of nodes among which confounding is to be identified is small, **we only need to know that the causal mechanisms are changed for that small set of nodes, but not for all nodes**. This assumption has been made in [35, 37]. We will edit line 160 as follows to make it clear that the assumption is not too strong.\n\n\"*Hence, we focus on detecting and measuring confounding among a set of variables, assuming that the causal mechanism shifts are observed among that subset of variables.*\"\n\n>The formal definition of a mechanism shift..?\n\nWe thank you for the suggestions. We will ensure that the intuitive explanations of mechanism shifts appear early in the paper. In particular, we will move lines 131-133 and lines 163-169 to introduction line 48 to introduce causal mechanisms and how the shift in causal mechanisms are useful in detecting confounding. Also, it is sufficient to know relative shift between environments, where each environment is a result of either a soft or hard intervention to a set of variables.\n\n>The relationship between definitions 4.2 and 4.3 and ignorability....\n\nThis is an interesting insight. We thank you for bringing up this point. The above paper presents an intriguing way of defining confounding as the difference between nominal and complete propensity scores in the potential outcomes framework. This is inherently connected to the ignorability assumption. Since directed information also relies on the KL-divergence between conditional and interventional distributions, we can leverage this definition to define relative confounding strength between sets of variables. We will include all of these points in the revised manuscript and discuss them in the related work section.\n\n> While the theoretical results are thorough...\n\nOur primary focus in this work was on effectively identifying confounding variables, which is non-trivial by itself. Once we effectively identify observed confounding variables, we can control for them to reduce bias in estimated causal effects. We appreciate your interest in the results on bias in estimated causal effects. We conducted experiments to study the impact of the proposed confounding measures on reducing bias. Table 2 in the uploaded rebuttal PDF demonstrates that controlling for the variables detected as confounders by our method helps reduce the bias in estimated causal effects.\n\n> Definition 3.1 Looks like X is being redefined here ...\n\n**Definition 3.1:** We undestand the concern. We will use a different variable in Defintion 3.1, say $\\mathbf{W}$, in place of $\\mathbf{X}$. With this change, none of the other contents gets affected.\n\n**Line 131:** We will edit line 131 as follows.\n\n*\"For a node $X_i$, $\\mathbb{P}(X_i|\\mathbf{PA}_i)$ is called the *causal mechanism* of $X_i$.\"*\n\n> Assumption 3.1 talks about changes in causal mechanisms, but this concept hasn't been formalized...\n\nAssumption 3.1 discusses independent causal mechanisms but not changes in causal mechanisms. We introduce the concept of causal mechanism shifts in lines 145-146 and later in Assumption 3.2, we discuss sparse causal mechanism shifts.\n\n> Line 147: what is a context node?...\n\nThank you for your suggestions. A context node can be viewed as an exogenous parent to the set of nodes on which an intervention is performed. Our analysis does not depend on this extended causal graph. However, we will define context specific distributions as you suggested to make this clear. \n\n> Definition 4.1: The notation is a little awkward with the condition on $\\mathbb{P}(X_{j})$ within $D_{KL}$.\n \nWe followed [60] to use this particular notation of conditioning on $\\mathbb{P}(X_{j})$ within $D_{KL}$. We believe this is to differentiate between conditional and interventional probabilities. Since it is clear from the context, we will exclude conditioning on $\\mathbb{P}(X_{j})$ in the revised manuscript.\n \n> Line 128: unclear notation, what information is $P_{x}(V), X\\subseteq V$ adding here?... \n \nWe use $P_x (V), X \\subseteq V$ and $P_*$ to formally define causal Bayesian networks. They do not have any relation to our method. $X$ should be bold here. We will fix this typo.\n\n> Line 154: \"Let $C_{S\\wedge \\neg R}$ be the set of contexts in which we observe mechanism changes for the set of variables $X_S$ but not for the variables $X_R$.\" Changes relative to what?...\n\nIn your question, if $i\\in S$, we have $P^c (X_i|\\mathbf{PA}_i) \\neq P^{c'}(X_i|\\mathbf{PA}_i)$. If $i \\in R$, the inequality becomes equality.  The shifts do not have to be the same across all contexts. Contexts are a result of either soft or hard intervention on the nodes. Hence, the changes between contexts are relative to interventional distributions.\n\n>Line 158: This does not look like a p-value. This looks like an indicator taking a value of true or false, not a probability... Same questions for Assumption 3.2.\n\n Following [35], we will edit line 158 as follows: \n\n*\"For example, the $p\\text{-value}(\\mathbb{P}^c(X_i|\\mathbf{PA}_i^o)\\neq\\mathbb{P}^{c'}(X_i|\\mathbf{PA}_i^o))$ where $\\mathbf{PA}_i^o$ is the set of...\"*\n\nThe above p-value tests whether the causal mechanisms of $X_i$ are different in two different contexts $c,c'$. We will remove $p$ from line 158 to avoid confusion. In assumption 3.2, similar to [35], we use $p$ to indicate the probability of two distributions being different. \n\n>Line 286: Are the $\\epsilon$ normally distributed? or is zero mean and any distribution ok?\n \nThe only restriction on $\\epsilon$ is that it has zero mean with no other restriction on the underlying probability distribution.\n\n> Definition 4.7: Lower case $c$ is a bit confusing.\nWe will replace $E^c_i$ to $E^C_i$ to avoid confusion in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xgQBkQi7U3", "reviewer": "Reviewer_3wgM", "summary": "The authors mainly introduce capturing both observed and unobserved confounding using data from multiple contexts. Leveraging experimental data proposes a comprehensive approach for detecting and measuring confounding effects from three different settings that don't need the parametric assumptions and relaxes the causal sufficiency assumption. The authors provide the measure for detecting confounding effects (relative strengths of confounding).\nFor each of the proposed measures, this article presents key properties and an algorithm for detecting and measuring confounding using data from multiple contexts.", "questions": "See the above Weaknesses.", "limitations": "The authors provide a detailed discussion of the limitations and applicability of their method.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The entire article primarily introduces the detection and quantification of confounding effects without making parametric or causal sufficiency assumptions. It presents a well-structured logical framework to discuss this concept.", "weaknesses": "I think overall the authors did interesting research, but my main concerns are listed below.\n\n1. When the environment changes, for example, if $c$ changes to $c'$, will the original causal relationship change?\n\n2. If the length of the environment node is only 1, will the method described in the article fail?\n\n3. Why do settings 2 and 3 introduce mutual information to define confounding? Would there be any difficulties in using the KL divergence?\n\n4. In line 104, the author said \"an approach we also leverage in this work\", but the author did not introduce the method in which article in detail.\n\n5. In line 111, the article describes \"we measure the effects of both observed and unobserved confounding\". However, the subsequent sections do not appear to provide a direct method for measuring causal effects.\n\n\n6. In fact, the confounding effect is not always a number between 0 and 1. Converting it to a number between 0 and 1 only measures its relative strength, but does not directly calculate its confounding effect.\n\n7. In Setting 1: When there is confounding between two variables, CNF-1 is used to calculate it. If multiple variables have common confounding, CNF-1 is calculated for each variable with the others and then summed. Will this lead to the repeated accumulation of confounding?\n\n8. The detect confounding method proposed in the article, but this measure is relatively rarely used in real life. In fact, it is unrealistic to only conduct simulation experiments. In this part of the experiments, the author did not specifically introduce the settings, so why were only the CNF-2 results output, and not the CNF-3 results? Is there a significant difference between these two results? For the section on \"Measuring Conditional Confounding,\" the author should add experiments involving observed confounding, such as replacing the unobserved confounding with observed confounding in Experiment 1. \n\n9. The authors repeat write “decision variables” in lines 149 or 150, and \"two\" in line 156. \n\n10. The authors miswrite “$\\mathcal{D^{C}}$” as “$\\mathcal{D}$” in the third line of ALgorithm 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> When the environment changes, for example, if c changes to c', will the original causal relationship change?\n\nIt depends on the type of intervention. If context change is a result of soft intervention on a variable, the underlying causal relationships do not change. If the context change is a result of hard intervention, the causal relationships can change. A similar discussion can be found in Section 2 of [35]. In this paper, we consider both types of interventions on  variables as highlighted in Table 1.\n\n> If the length of the environment node is only 1, will the method described in the article fail?\n\nWhen there is only one context available, there is a fundamental limitation: confounding cannot be uniquely identified, as explained in lines 43-45. Rather than calling it a limitation, we state that our method is not applicable when there is only one environment.\n\n> Why do settings 2 and 3 introduce mutual information to define confounding? Would there be any difficulties ..?\n\nWe build upon earlier works that utilize mutual information to measure confounding. According to our formulation, mutual information is among the most suitable choices for measuring the dependency between random variables induced by changes in the mechanism, as explained in lines 295-297. See [35] for a similar approach using mutual information for identifying confounding. In our experiments, we did not encounter any difficulties using KL divergence.\n\n> In line 104, the author said \"an approach we also leverage in this work\", but the author did not introduce the method in which article in detail.\n\nApologies for the confusion. By \"an approach,\" we refer to the idea of using different contexts to measure confounding. We will either modify or remove the text in brackets to clarify this point and avoid any confusion.\n\n> In line 111, the article describes \"we measure the effects of both observed and unobserved confounding\". However, the subsequent...\n\nWe believe the reason for confusion is the word 'effect'. We aim to study and quantify the confounding **bias** instead of its effect on causal **effects** as a downstream application. Our goal in this paper is to provide a unified framework for confoudning. We will replace **effect** with **bias** to remove the ambiguity. \n \nWe appreciate your interest in the results on causal effects. We conducted experiments to study the impact of the proposed confounding measures on reducing bias in estimated causal effects. Table 2 in the uploaded rebuttal PDF demonstrates that controlling for the variables detected as confounders by our method helps reduce the bias in estimated causal effects.\n \n> In fact, the confounding effect is not always a number between 0 and 1. Converting it to a number between 0 and 1...\n\nWe provide a reason for studying relative strength in lines 57-62 of the introduction section. Also, it is trivial to evaluate the actual confounding effect that is not between 0 and 1 by ignoring the exponential transformation used in our definition of confounding measure.\n\n> In Setting 1: When there is confounding between two variables, CNF-1 is used to calculate it. If multiple variables have common confounding...\n\nSince this is a first effort to define a measure of confounding among a set of variables, our focus herein was on the stated settings. It is possible however to accumulate the confounding. It is easy to see that our current definition of measuring joint confounding  satisfies positivity and monotonicity properties. We believe that exploring other ways of defining joint confounding can be interesting directions of future work.\n\n> The detect confounding method proposed in the article, but this measure is relatively rarely ...\n\nWe included two simple real-world applications of our method in the uploaded rebuttal PDF, as shown in Figure 1. (The purpose of these examples is only to show its practical value, and not intended to be comprehensive or complex.) For the conditional confounding experiments, we observe similar results for Settings 2 and 3; therefore, we report results for Setting 2 only. Additionally, our experiments on conditional confounding assume that the confounding variables are observed. We have provided additional experimental results in the uploaded rebuttal PDF, detailed in Tables 1 and 2, which demonstrate the usefulness of our methods.\n\n> The authors miswrite $D^c$ as $D$ in the third line of ALgorithm 1.\n\nAs explained in line 245, we combine data from all contexts to evaluate conditional probability. We will update third line of Algorithm 1 to include $\\mathcal{D} = \\cup_{c}\\{\\mathcal{D}^c\\}$ to make it clear.\n\nReferences:\n[35] Mameche, Sarah, Jilles Vreeken, and David Kaltenpoth. \"Identifying Confounding from Causal Mechanism Shifts.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2024.\n\nWe hope our responses address your concerns. We will update the manusript accordingly. We are happy to discuss further if you have any additional questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uTQKJfzXyM", "reviewer": "Reviewer_K6Rd", "summary": "The paper addresses the challenge of identifying and quantifying (unobserved) confounding in causal inference. They propose a more comprehensive approach by relaxing the classic assumption of causal sufficiency and leveraging the sparse causal mechanism shifts assumption. The authors introduce methods to detect and measure confounding effects, distinguish observed and unobserved confounding, and evaluate the relative strengths of confounding among variable sets.  An empirical validation supports their theoretical analysis.", "questions": "(Q1) I got confused at lines 103-106: \"While these methods use data from different contexts (an approach we also leverage in this work), they assume the absence of unobserved confounding variables; we instead focus on capturing both observed and unobserved confounding using data from multiple contexts.\". [27] proposes a test for the presence of hidden confounding and hence allows for unobserved variables. Can the author briefly comment on how their measures compare with the approach proposed in [27]? \n\n(Q2) I am confused slightly confused with Assumption 3.2 (line 169):  $$ P^c( X_i | \\text{PA}_i) \\neq P^{c'}(X_i | \\text{PA}_i) $$\nwhat does it mean for conditional distributions to be different?\n \n(Q3) Can the authors comment with one motivating example of how their proposed confounding measure could be used in practice? (See W2)\n\n[27] Rickard Karlsson and Jesse Krijthe. Detecting hidden confounding in observational data using multiple environments. Advances in Neural Information Processing Systems, 36, 2023.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(S1) The paper provides a thorough study of confounding from several perspectives: detecting and measuring confounding between pairs of variables and among multiple variables; distinguishing between observed and unobserved confounding; assessing the relative strengths of confounding among different sets of observed variables. To my knowledge, this is the first study exploring all these aspects in a unified framework.\n\n(S2) The paper is well written and easy to follow.", "weaknesses": "(W1) I found the paper to be lacking in experimental evaluations. It is not clear how hard (both statistically and computationally) it is to compute the measures of confounding that were proposed, especially the ones measuring confounding among multiple variables. \n\n(W2) I am somehow skeptic about the practical relevance of the results presented in the paper (and the lack of experiments reinforces this point, see W1). In particular, people focused a lot on the marginal sensitivity model (and variations) as a measure of confounding strength because sensitivity analysis bounds can be easily derived from it. I am not sure the same would be possible under the proposed measure of confounding, and hence I am not sure how it would be useful in practice. \n\n\n(W3) I think some relevant related works are missing. In particular, when interventional data is available (e.g. RCTs) [1] and [2] propose testable implications for detecting hidden confounding. Further, [3] and [4] propose to lower bound the strength of hidden confounding (as measured by the marginal sensitivity model). \n\n[1] Falsification of Internal and External Validity in Observational Studies via Conditional Moment Restrictions. Hussain et al. AISTATS 2023.\n\n[2] Benchmarking Observational Studies with Experimental Data under Right-Censoring. Demirel et al. AISTATS 2024. \n\n[3] Hidden yet quantifiable: A lower bound for confounding strength using randomized trials. De Bartolomeis et al. AISTATS 2024.\n\n[4] Detecting critical treatment effect bias in small subgroups. De Bartolomeis et al. UAI 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> I found the paper to be lacking in experimental evaluations. It is not clear how hard (both statistically and computationally) it is to compute the measures of confounding that were proposed, especially the ones measuring confounding among multiple variables.\n\nDue to space constraints, we included experiments in Appendix Section B (our paper required significant space for discussing the considered 3 settings and our formulations therein). We have now included additional experimental results in the uploaded rebuttal PDF. If accepted, we will use the additional page available in the final version to include the results in the main paper itself. \n\nAs stated in the appendix, our experiments were conducted on a CPU and are straightforward to run. We have provided the code in the supplementary material, and it can be executed quickly.\n \n> I am somehow skeptic about the practical relevance of the results presented in the paper ... people focused a lot on the marginal sensitivity model ... I am not sure the same would be possible under the proposed measure of confounding, and hence I am not sure how it would be useful in practice.  I think some relevant related works are missing...\n\nIn the uploaded rebuttal PDF, Figure 1 demonstrates two simple real-world examples where our methods can be applied (the purpose of these examples is only to show its practical value, and not intended to be comprehensive or complex). As explained in the previous response, we have included experiments in the Appendix and new experiments in the uploaded rebuttal PDF.\n\nWe appreciate your suggestion regarding sensitivity analysis. We will incorporate the following points in the revised manuscript for completeness.\n\n1. Our method aims to estimate the exact value of confounding rather than approximate it using bounds obtained via sensitivity analysis.\n2. The difference between the total confounding and the conditional confounding by conditionining on observed confounding variables can be interpreted as the unobserved confounding estimate obtained via sensitivity analysis. \n3. As discussed with reviewer n1WD, we can connect marginal sensitivity definition of confounding and the confounding based on directed information used in our paper. We will include this connection in the revised manuscript.\n\nWe also thank you for providing the relevant papers. We will discuss them in the revised manuscript and include a discussion on sensitivity analysis as mentioned before for completeness.\n\n> Q1) I got confused at lines 103-106: \"While these methods use data from different contexts... [27] proposes a test for the presence of hidden confounding and hence allows for unobserved variables... compare with the approach proposed in [27]?\n\n\nApologies for the oversight. The study of hidden confounding detection in [27] primarily focuses on downstream causal effect estimation. In contrast, our work aims to provide a unified framework for studying and measuring both observed and unobserved confounding across different types of contextual information. This framework supports various downstream applications beyond causal effect identification, including assessing the relative strengths of confounding and measuring confounding between pairs and sets of variables. We will add these points to the related work in the revised manuscript. We also study downstream causal effect estimation tasks. As demonstrated in the results in the rebuttal PDF, confounding detection using our method contributes to improved causal effect estimation.\n\n> Q2) I am slightly confused with Assumption 3.2 (line 169): what does it mean for conditional distributions to be different?\n \nConsider a variable $X_i$ and its parents $PA_i$. If $\\mathbb{P}(X_i=x_i|PA_i = pa_i)$ is different in two contexts/environments $c,c'$ for atleast one pair $(x_i, pa_i)$, we say that the causal mechanisms are different in two contexts $c,c'$. According to Assumption 3.2, such causal mechanism shifts are rare/sparse for a variable $X_i$.\n \n> (Q3) Can the authors comment with one motivating example of how their proposed confounding measure could be used in practice? (See W2)\n\nWe have included two simple real-world examples in the uploaded rebuttal PDF (Figure 1) demonstrating the applicability of our methods. Additionally, since the benchmark datasets from the BNLearn repository come with known data-generating processes, we can generate different contexts by performing interventions and subsequently applying our method, similar to [35,37].\n\nReferences:\n[27] Rickard Karlsson and Jesse Krijthe. Detecting hidden confounding in observational data using multiple environments. Advances in Neural Information Processing Systems, 36, 2023.\n[35] Mameche, Sarah, Jilles Vreeken, and David Kaltenpoth. \"Identifying Confounding from Causal Mechanism Shifts.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2024.\n[37] Mooij, Joris M., Sara Magliacane, and Tom Claassen. \"Joint causal inference from multiple contexts.\" Journal of machine learning research 21.99 (2020): 1-108.\n\n\nWe hope our responses address your concerns. We will update the manusript accordingly. We are happy to discuss further if you have any additional questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MoSxvH7eY1", "reviewer": "Reviewer_e4Fq", "summary": "This paper introduces some measures of (conditional) confounding, based on information theoretic quantities. Some of their properties and an algorithm to estimate the measures using data from different environments. Although not included in the main text, the authors test their theory on some synthetic data.", "questions": "I would like to get a feeling for why the properties of their measure of confoundedness are valuable (Theorems 4.2, 4.4 and 4.6). The only really essential property that I can think of such a measure is that it is 0 if the variables are non confounded and not 0 otherwise. Why for example do we want positivity? One could even be interested in a negative measure if the variables have a negative correlation under confoundedness, for example.", "limitations": "This research does not contain any obvious negative societal impact. The authors do include a couple of sentences at the end of the paper where they state that there is a challenge to find real-world data to test their theory. Although I value the author's honesty about this limitation, I think it is too strong of a limitation.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The ideas on the paper are original as far as I know and I believe that the question regarding the measurement of confounding is an important one. The authors differentiate their work from previous work.\n- Beyond the errors outlined in the weakness section, the paper is well-written and presented. It is a bit unfortunate that the experiments were not included on the main paper.", "weaknesses": "- There are several typos/mistakes on the paper:\n\n-- On line 158 and 169: $p$ is defined as the inequality between two distributions, this definition doesn’t make sense. I checked their reference 33 and 35 and it says in there that it is the probability of the distributions being different. Furthermore, p-value is a well-defined and known term, why overloading it?\n\n-- On definition 4.1 what they call directed information is not an -expected- KL divergence, it is simply a KL divergence.\n\n-- On Table 2, on rows 1 and 2 the equality with 0 is inverted. For example, if $X_i\\to X_j$, then $P(X_i\\mid X_j)=P(X_i\\mid do(X_j))$ so that the log of such quantity is 0 for all values of $X_i$ and $X_j$, that is $I(X_i\\to X_j)=0$.\n\n-- Is definition 4.6 really a definition? That sounds like a proposition to me.\n\n- I’m unsure of how large is the significance of the contribution being made (see questions and limitations). Which paired with the errors and typos above make me recommend for rejection of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> On line 158 and 169: p is defined as the inequality between two distributions, this definition doesn’t make sense... why overloading it?\n\nWe undestand the concern. To fix this, following [35], we will edit line 158 in the revised manuscript as follows.\n\n*\"For example, the $p\\text{-value}(\\mathbb{P}^c(X_i|\\mathbf{PA}_i^o)\\neq\\mathbb{P}^{c'}(X_i|\\mathbf{PA}_i^o))$ where $\\mathbf{PA}_i^o$ is the set of...\"*\n\n> On definition 4.1 what they call directed information is not an -expected- KL divergence, it is simply a KL divergence.\n\nWe agree with you. We will change **\"expected KL divergence\"** to **\"conditional KL-divergence\"** in the revised manuscript.\n \n>  On Table 2, on rows 1 and 2 the equality with 0 is inverted...\n \nWe beg to differ. If $X_i\\rightarrow X_j$, then we have $\\mathbb{P}(X_i|X_j) \\neq \\mathbb{P}(X_i|do(X_j))$ and hence $I(X_i\\rightarrow X_j)>0$. This is because, in $X_i\\rightarrow X_j$, conditioning on $X_j$ does not make $X_i, X_j$ independent. However, intervening on $X_j$ makes $X_i, X_j$ independent when there is no confounding between $X_i, X_j$. Formally, when there is no confounding, $\\mathbb{P}(X_i|do(X_j)) = \\mathbb{P}(X_i)$. On the other hand, $\\mathbb{P}(X_i|X_j)\\neq \\mathbb{P}(X_i)$.\n \n> Is definition 4.6 really a definition? That sounds like a proposition to me.\n\nWe agree with you, we will change the definition to proposition. The proof is trivial and follows from the definition of mutual information, which we will include in the revised manuscript.\n\n> I would like to get a feeling for why the properties of their measure of confoundedness are valuable...\n\nWe believe that these properties are essential to ensure the correctness of the proposed metrics from a measurement perspective. The monotonicity property is crucial for understanding which set of variables are more confounded than others. Since our metrics are derived from positive quantities like directed information and mutual information, they return positive values. Studying negative confounding is a potential future direction.\n\n> The authors do include a couple of sentences at the end of the paper where they state... I think it is too strong of a limitation.\n\nWe'd like to point out that we only didn't focus on real-world application, and rather deferred it to future work to apply our methods to real-world datasets. We have included two simple real-world examples in the uploaded rebuttal PDF (Figure 1) demonstrating the applicability of our methods (the purpose of these examples is only to show its practical value, and not intended to be comprehensive or complex). For example, we can study settings 2 and 3 using the SACHS dataset [35,37]. To study setting 1, it is enough to check for datasets that are obtained from randomized control trials where a set of variables are intervened. \n\nWe will modify the line in conclusions to reflect this point.\n\nReferences:\n[35] Mameche, Sarah, Jilles Vreeken, and David Kaltenpoth. \"Identifying Confounding from Causal Mechanism Shifts.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2024.\n[37] Mooij, Joris M., Sara Magliacane, and Tom Claassen. \"Joint causal inference from multiple contexts.\" Journal of machine learning research 21.99 (2020): 1-108.\n\nWe hope our responses address your concerns. We will update the manusript accordingly. We are happy to discuss further if you have any additional questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Common response to all reviewers
We thank all reviewers for their thoughtful feedback. We are pleased to see the following encouraging comments from the reviewers.

1. The problem addressed in this paper is of significant importance (e4Fq).
2. The ideas presented are both original (e4Fq) and novel (n1WD).
3. The theoretical justification for our methods is thorough (n1WD).
4. We are the first to study various aspects of confounding (K6Rd) within a unified framework (K6Rd, 3wgM).
5. The paper is well-written and well-presented (K6Rd, e4Fq).

As can be seen from the reviews, most of the reviewers' concerns are about clarifications, which we have addressed below. We have also uploaded a PDF with additional results as suggested by the reviewers. Our paper required significant space for discussing the considered 3 settings and our formulations therein; for purposes of clarity, the experimental results were moved to Appendix. If accepted, we will use the additional page available in the final version to include the results in the main paper itself. The additional results, included in the rebuttal, further demonstrate the usefulness of our methods. We will update the manuscript as per the suggestions, and will release our code publicly on acceptance.

Meta-review: This paper introduces a novel approach for detecting and quantifying confounding effects by leveraging known causal mechanism shifts in observed variables, without relying on parametric assumptions or causal sufficiency. The idea is recognized as novel and original (n1WD, e4Fq), the paper is well-written (K6Rd, e4Fq), and the work is presented within a unified framework (K6Rd, 3wgM). However, some concerns remain regarding the clarity of certain descriptions (n1WD), the depth of experimental evaluations, and the demonstration of real-world applicability (3wgM, K6Rd, e4Fq). Additionally, the importance of the proposed measure for confoundedness requires further consideration (e4Fq). Nonetheless, the authors have adequately addressed the key issues in their responses, leading to the decision to accept the paper.
Decision: Accept (poster)