Title: Detecting and Measuring Confounding Using Causal Mechanism Shifts
Venue: NeurIPS 2024
Accepted: yes
Keywords: Causality

Confounding

Mechanisms

Measure
Abstract: Detecting and measuring confounding effects from data is a key challenge in causal inference. Existing methods frequently assume causal sufficiency, disregarding the presence of unobserved confounding variables. Causal sufficiency is both unrealistic and empirically untestable. Additionally, existing methods make strong parametric assumptions about the underlying causal generative process to guarantee the identifiability of confounding variables. Relaxing the causal sufficiency and parametric assumptions and leveraging recent advancements in causal discovery and confounding analysis with non-i.i.d. data, we propose a comprehensive approach for detecting and measuring confounding. We consider various definitions of confounding and introduce tailored methodologies to achieve three objectives: (i) detecting and measuring confounding among a set of variables, (ii) separating observed and unobserved confounding effects, and (iii) understanding the relative strengths of confounding bias between different sets of variables. We present useful properties of a confounding measure and present measures that satisfy those properties. Our empirical results support the usefulness of the proposed measures.

Detecting and Measuring Confounding Using Causal Mechanism Shifts

Abstract

Detecting and measuring confounding effects from data is a key challenge in causal inference. Existing methods frequently assume causal sufficiency, disregarding the presence of unobserved confounding variables. Causal sufficiency is both unrealistic and empirically untestable. Additionally, existing methods make strong parametric assumptions about the underlying causal generative process to guarantee the identifiability of confounding variables. Relaxing the causal sufficiency and parametric assumptions and leveraging recent advancements in causal discovery and confounding analysis with non-i.i.d. data, we propose a comprehensive approach for detecting and measuring confounding. We consider various definitions of confounding and introduce tailored methodologies to achieve three objectives: (i) detecting and measuring confounding among a set of variables, (ii) separating observed and unobserved confounding effects, and (iii) understanding the relative strengths of confounding bias between different sets of variables. We present useful properties of a confounding measure and present measures that satisfy those properties. Our empirical results support the usefulness of the proposed measures.

1 Introduction

Understanding the underlying causal generative process of a set of variables is crucial in many scientific studies for applications in treatment and policy designs [44]. While randomized controlled trials (RCTs) and causal inference through active interventions are ideal choices for understanding the underlying causal model [19, 12, 13, 55], RCTs and/or active interventions are often impossible/infeasible, and some times unethical [50, 6]. Research efforts in causal inference hence rely on observational data to study causal relationships [44, 59, 65, 18, 41]. However, recovering the underlying causal model purely from observational data is challenging without further assumptions; this challenge is further exacerbated in the presence of unmeasured confounding variables.

Aconfounding variable is a variable that causes two other variables, resulting in a spurious association between those two variables. As exemplified with Simpson's paradox [58] and many other studies [20, 1, 31], the presence of confounding variables is an important quantitative explanation for why correlation does not imply causation . It is challenging to observe and measure all confounding variables in a scientific study [60, 44]. Identifying latent or unobserved confounding variables is even more challenging, and misinterpretation presents various challenges in downstream applications, such as discovering causal structures from observational data. Numerous methods operate under the assumption of causal sufficiency [45, 4, 60, 8, 51, 65], implying the non-existence of unobserved confounding variables. Causal sufficiency presupposes that all pertinent variables required for causal inference have been observed. However, this may not be a practical or testable assumption.

The study of confounding has various applications, chief among them being causal discovery - identifying the causal relationships among variables [38, 40, 63]. It is also useful for determining whether a set of observed confounding variables is sufficient to adjust for estimating causal effects [29], measuring the extent to which statistical correlation between variables can be attributed to confound-

ing [24, 25, 62], and verifying the comparability of treatment and control groups in non-randomized interventional studies [16].

A fundamental problem in causal inference tasks lies in detecting hidden confounding variables from observational data alone. However, this is non-trivial and poses various challenges. For example, a key issue is that given a marginal distribution over observed variables, there are infinitely many joint distributions corresponding to causal graphs involving unobserved variables [56]. To tackle such challenges, recent endeavors show that using data from different environments helps in improved causal discovery [40, 38, 33, 45, 23], detecting causal mechanism shifts [36], and detecting unobserved confounding [29, 38]. However, such recent efforts often subsume confounding detection under causal discovery, focusing primarily on identifying confounding factors while overlooking other useful information, such as the relative strength of confounding between variable sets and the distinction between observed and unobserved confounding within a variable set. We seek to address these gaps in this work.

We focus exclusively on the problem of studying confounding from multiple perspectives, including (i) detecting and measuring confounding among a set of variables, (ii) assessing the relative strengths of confounding among different sets of observed variables, and (iii) distinguishing between observed and unobserved confounding among a set of variables. The primary focus of causal inference often lies in verifying the presence or absence of confounding rather than determining the exact value of the measured confounding. However, we leverage the measured confounding to assess the relative strengths of confounding between sets of variables. To achieve the above objectives, we utilize data from various contexts, where each context results from shifts in the causal mechanisms of a set of variables [38, 45]. This allows us to propose different measures of confounding based on the available context information. Our contributions can be summarized as follows.

For various definitions of confounding, we propose corresponding measures of confounding and present useful properties of the proposed measures. To our knowledge, this is the first comprehensive study that examines various aspects of observed and unobserved confounding using data from multiple contexts without making parametric or causal sufficiency assumptions.

We study pair-wise confounding, confounding among multiple variables, how to separate unobserved confounding from overall confounding, and present ways to assess relative confounding.

Wepresent an algorithm for detecting and measuring confounding using data from multiple contexts. Experimental results are performed to verify theoretical analysis.

2 Related Work

The study of confounding has typically been embedded as part of causal discovery algorithms in most existing work. Causal discovery methods can be categorized according to several criteria, including the type of data utilized (observational versus interventional/experimental), parametric versus non-parametric approaches, or whether they relax causal sufficiency assumptions [65, 59]. Considering our focus in this work on studying confounding comprehensively by going beyond observed confounding variables, we discuss literature that are directed towards methods that relax the causal sufficiency assumption and rely on experimental data.

Causal Discovery via Observational Data, Relaxing Causal Sufficiency: Constraint based causal discovery algorithms produce equivalence class of graphs that satisfy a set of conditional independence constraints [60, 11, 9, 42]. Other methods such as [2, 28, 27] reduce the problem complexity by assuming a parametric form of the underlying causal model (e.g., variables are jointly Gaussian in Chandrasekaran et al. [7]), thereby returning unique causal graphs. Nested Markov Models (NMMs) [56, 57, 49, 14] allow identifiability of causal models with latent factors by using (pairwise) Verma constraints. A recent approach using differentiable causal discovery [2] combines NMMs with the differentiable constraint [66] to discover a partially directed causal network and likely confounded nodes. Unlike these methods, our focus in this work is on detecting and measuring confounding under various settings, instead of recovering the entire causal graph or equivalence class.

Causal Discovery Using Data From Multiple Environments: Given access to a set of observed confounding variables, very recent work [29] presented testable conditional independence tests that are violated only when there is unobserved confounding. However, their analysis is focused towards the downstream causal effect estimation. We aim to provide a unified framework for studying and measuring confounding under different types of contextual information available.

Other methods [33, 23] learn an equivalence class of graphs when data from observational and interventional distribution are available. Confounding has also shown to be detected in linear models with non-Gaussian variables [20]. In linear models, a spectral analysis method was proposed in [25] to understand to what extent the statistical correlation between a set of variables on a target variable can be attributed to confounding. See Tab. 4 of [40] for an overview of causal discovery methods that use data from multiple environments or contexts. Under the specific assumptions of causal sufficiency and sparse mechanism shift, a method was proposed in [45] to reduce the size of a given Markov equivalence class using mechanism shift score. A differentiable causal discovery method was proposed in [4] to use interventional data to recover interventional Markov equivalence class. While these methods use data from different contexts, they assume the absence of unobserved confounding variables; we instead focus on capturing both observed and unobserved confounding.

Measuring and Interpreting Confounding: Earlier efforts in the field have studied different measures for observed confounding, each tailored to address specific challenges [44, 15, 35, 3, 39, 30, 43, 34]. Such measures have also been refined to address specific issues [24, 54]; for e.g., a method to correct the non-linearity effect present in confounding estimates via the exposure-outcome association with and without adjustment for confounding was proposed in [24]. In contrast, we measure the effects of both observed and unobserved confounding. Motivated from the ignorability property in potential outcomes framework [61, 26], the divergence between nominal and complete propensity density has been considered as an indicative of hidden confounding [26]. To the best of our knowledge, the efforts closest to ours are [38, 40], which study confounding using data from multiple contexts without the causal sufficiency assumption. However, they do not measure confounding and detect confounding only as a step to discover the causal graph. Ours is a more general framework for studying and measuring confounding from multiple perspectives.

In regression models, certain difference thresold between the coefficients of treatment variable before and after adjusting for the possible confounding is considered as the indication for the presence of confounding. This process of choosing a threshold is also called change-in-estimate criterion. Typical threshold used in literature is 10% [54, 32, 5].

3 Background and Problem Setup

Let X be a set of observed variables and Z be a set of unobserved or latent variables. The values of X , Z can be real, discrete, or mixed. Let G be the underlying directed acyclic graph (DAG) among the variables V = X ∪ Z . Directed edges among the variables in V indicate direct causal influences. Assume that the set of unobserved variables Z are jointly independent and are exogenous to X (i.e., Z i ⊥ ⊥ Z j and X k ̸→ Z j ∀ i, j, k ). In this setting, any two nodes X i , X j ∈ X sharing a common parent Z k ∈ Z are said to be confounded, and Z k is said to be a confounding variable. For a node X i ∈ X , PA i = { X j ∈ X | X j → X i } ∪ { Z j ∈ Z | Z j → X i } denotes the set of parents of X i .

̸

For a node X i , P ( X i | PA i ) is called the causal mechanism of X i . The causal mechanism P ( X i | PA i ) encodes how the variable X i is influenced by its parents PA i . Following earlier work [22, 38, 45, 21, 46, 52], we make the following general assumption about the underlying causal mechanisms of data. Assumption 3.1. (Independent Causal Mechanisms [44, 47]) A change in P ( X i | PA i ) has no effect on and provides no information about P ( X j | PA j ) ∀ j = i .

Identifying confounding from only observational data is challenging without further assumptions [28]. Hence, following earlier work [38, 29, 40], we assume that the data over the variables X is observed over multiple contexts or environments . While there are various ways of formulating/constructing contexts, in this paper, we assume that each context is created as a result of either hard (a.k.a. structural) interventions or soft (a.k.a. parametric) interventions on a subset V S ⊆ V of variables where S is a set of indices. Performing hard intervention on a variable V i is the same as setting the value of V i to a value v i . Hard intervention on a variable V i removes the influence of its parents PA i on V i . Performing soft intervention on a variable V i is the same as changing the causal mechanism of V i , P ( V i | PA i ) , with a new causal mechanism ˜ P ( V i | PA i ) . Soft intervention on a variable V i does not remove the influence of its parents PA i on V i . The idea of explicitly considering context information and using different contexts as context variables to create extended causal graphs has been studied in the literature. Context variables are also called as policy variables, decision variables, regime variables, domain variables, environment variables, etc. [40, 45, 17, 22].

Let C = { c 1 , c 2 , . . . , c n } be the set of n contexts and let P c ( X ) , c ∈ C , denotes the probability distribution of the observed variables X in the context c . Let C S ∧ R , where S, R are sets of indices, be

̸

̸

the set of contexts in which we observe mechanism changes for the set of variables X S ∪ R . Similarly, let C S ∧¬ R be the set of contexts in which we observe mechanism changes for the set of variables X S but not for the variables X R . We say that the causal mechanism of a variable X i changes between two contexts c, c ′ if P c ( X i | PA i ) = P c ′ ( X i | PA i ) . Given the data over observed variables in each context, there exist methods for detecting mechanism shifts of each variable between the contexts [36, 38, 45, 37]. For example, the p -value ( P c ( X i | PA o i ) = P c ′ ( X i | PA o i )) where PA o i is the set of observed parents of X i can be used to detect mechanism change for X i between the contexts c, c ′ [38, 36]. Hence, we focus on detecting and measuring confounding among a set of variables, assuming that the causal mechanism shifts are observed among that set of variables.

Context information is not very useful if there is no restriction on how causal mechanisms are changed between the contexts [45, 38]. For example, the causal mechanisms of X i and X j both differing across all (or no) contexts would trivially satisfy Assumption 3.1, but reveal no information about the underlying causal mechanisms [10, 38]. Hence, following earlier work [45, 38, 17], we make the following assumptions.

̸

Assumption 3.2. (Sparse Causal Mechanism Shift [53]) Causal mechanisms of variables change sparsely across contexts, i.e., if p := ( P c ( X i | PA i ) = P c ′ ( X i | PA i )) , then 0 < p < 0 . 5; ∀ c, c ′ ∈ C .

Assumption 3.2 implies that the causal mechanisms change infrequently across contexts. This assumption is more general because, in many scientific studies, for any given context, interventions typically affect only a few variables [53].

Assumption 3.3. (Markov Property under Mechanism Shifts [17]) The distribution P ( V ) is given by P ( V ) = ∫ P C ( V ) d P ( C ) = ∫ Π i P C ( V i | PA i ) d P ( C ) . In other words, variables V are assumed to be conditionally exchangeable, so that the same graph G applies in every context c ∈ C .

Assumption 3.4. (Causal Sufficiency Over X ∪ Z ) All common parents of any pair of observed nodes belong to the set X ∪ Z . In other words, all relevant variables for detecting confounding and the unobserved confounding variables are already present in X ∪ Z .

Problem Statement: Given data over the observed variables X in multiple contexts, each context resulting from a sparse causal mechanism shift of variables in V , (i) can we identify which pairs or sets of variables are confounded and can we measure the confounding strength? (ii) can we isolate the confounding effects of observed and unobserved confounding variables? and (iii) can we study the relative strengths of confounding among different sets of variables?

To address the above problem, in the next section, we consider various definitions of confounding and present appropriate confounding measures depending on the context information available.

4 Detecting and Measuring Confounding

In this section, we present methods for detecting and measuring confounding for various scenarios in which shifts in causal mechanisms are observed. Considering any three observed variables X i , X j , X o ∈ X and an unobserved confounding variable Z ∈ Z , we present mea-

sures of confounding depending on the information about mechanism shifts of X i , X j , X o , Z . Each of the following subsections includes: (i) a definition of confounding, (ii) a corresponding definition of the confounding measure, (iii) a method for isolating the unobserved confounding measure from the overall confounding, (iv) an extension of the confounding measure to more than two variables, and (v) key properties of the proposed confounding measures. See Tab. 1 and Fig. 1 for an overview.

4.1 Setting 1: Measuring Confounding Using Directed Information Between X i , X j .

In this setting, we use the fact that directed information does not vanish in the presence of a confounding variable [64, 48]. To this end, we leverage the interventional effects of X i , X j on each other to define a measure of confounding.

Setting 1

Setting 2

Setting 3

Definition 4.1. (Directed Information [48]). The directed information I ( X i → X j ) from X i ∈ X to X j ∈ X is defined as the conditional Kullback-Leibler divergence between the distributions P ( X i | X j ) , P ( X i | do ( X j )) . That is:

Definition 4.2. (No Confounding [44]) When measuring the causal effect of a (treatment) variable X i on a (target) variable X j , the ordered pair ( X i , X j ) is unconfounded if and only if the directed information from X j to X i : I ( X j → X i ) is zero. Equivalently, P ( X j | X i ) = P ( X j | do ( X i )) .

A similar definition of confounding that relates the conditional distribution P ( X i | X j ) and interventional distribution P ( X i | do ( X j )) is defined as follows.

̸

Definition 4.3. (Noncollapsibility) [15, 43, 54] The statistical association between two variables X i and X j is said to be noncollapsible if the association strength differs in each level/strata of other variable X k . That is, if X k is a confounding variable between X i , X j , we have P ( X j | X i ) = P ( X j | do ( X i )) = E X k ( P ( X j | X i , X k )) .

̸

From Defns. 4.1 and 4.2, for a pair of variables ( X i , X j ) , observing I ( X j → X i ) > 0 and I ( X i → X j ) > 0 implies that P ( X j | do ( X i )) = P ( X j | X i ) and hence the presence of confounding (see Tab. 2). Using the above properties of directed information, we measure confounding as follows.

Definition 4.4. (Confounding Measure 1) When causal mechanism shifts of two variables X i , X j ∈ X are observed, resulting in different contexts, under the Assumptions 3.2-3.4, the measure of confounding CNF -1( X i , X j ) between X i and X j is defined as follows.

For all the confounding measures, we use exponential transformation to limit the range of the measure between 0 and 1 . Note that in a DAG, one of I ( X i → X j ) , I ( X j → X i ) is zero under no confounding (see Tab. 2 for a simple example with two and three node graphs). Hence CNF -1( X i , X j ) outputs zero when there is no confounding between X i , X j . Similarly CNF -1( X i , X j ) outputs positive real value in the range (0 , 1] when there is confounding. We leverage data from multiple contexts to evaluate P ( X i | X j ) and P ( X i | do ( X j )) as follows. In this

setting, we assume each context is generated as a result of hard interventions on a subset of variables. Let P ij be the set of node indices that belong to a path from X i to X j including j , we use the contexts C { i }∧¬ P ij to evaluate P ( X j | do ( X i )) as P ( X j | do ( X i )) = E c ∈ C { i }∧¬ P ij [ P c ( X j | X i )] . Intuitively, to compute the interventional effects of X i on X j , we need to observe mechanism changes only for X i to account for the potential causal influence from X i to X j . In addition, none of the nodes in a causal path from X i to X j should be intervened. We use observational data to evaluate P ( X j | X i ) .

Proposition 4.1. (Identifiability of P ( X j | do ( X i )) ) P ( X j | do ( X i )) is identifiable from the set of contexts C { i }∧¬ P ij . To detect and measure confounding between a pair of nodes X i , X j , it is enough to observe two sets of contexts C { i }∧¬ P ij and C { j }∧¬ P ji . Thus, n sets of contexts are needed to detect and measure confounding between ( n 2 ) distinct pairs of nodes in a causal DAG with n nodes.

When a confounding variable X o between X i , X j is observed, and there may exist an unobserved confounding variable Z , it is crucial to detect and measure unobserved confounding effect [29]. We utilize conditional directed information to define the measure of unobserved confounding.

Definition 4.5. (Conditional Directed Information [48]). The conditional directed information I ( X i → X j | X o ) from X i to X j conditioned on X o is defined as the conditional Kullback-Leibler divergence between the distributions P ( X i | X j , X o ) , P ( X i | do ( X j ) , X o ) as follows.

This measure can trivially be extended to the case where there exist multiple observed and unobserved confounding variables. The expression P ( X i | do ( X j ) , X o ) means conditioning on X o in the interventional distribution P ( X i | do ( X j )) . Now, the conditional confounding can be measured as:

Intuitively, by conditioning on an observed confounding variable X o , we control the association between X i , X j flowing via X o and measure the influence via the unobserved confounding variables.

Beyond Pairwise Confounding: We now study when a set X S of variables where | X S | > 2 are jointly confounded i.e., share a common confounding variable and how to measure the joint confounding among the variables X S .

Theorem 4.1. A set of observed variables X S are jointly unconfounded if and only if there exists three variables X i , X j , X k ∈ X S such that I ( X i → X j | X k ) = I ( { X i , X k } → X j ) .

We now define the measure of confounding among the variables in X S as follows.

Conditional confounding among a set of variables can be defined similarly to Eqn. 4. We now study some useful properties of the measure CNF -1 .

Theorem 4.2. For any three observed variables X i , X j , X o and an unobserved confounding variable Z , the following statements are true for the measure CNF -1 .

( Reflexivity and Symmetry. ) CNF -1( X i , X i | X o ) = 0 , CNF -1( X i , X j | X o ) = CNF -1( X j , X i | X o ) .

( Positivity. ) CNF -1( X i , X j ) > 0 if and only if X i , X j are confounded. Given an observed confounding variable X o between X i , X j , CNF -1( X i , X j | X o ) > 0 if and only if there exists an unobserved confounding variable Z between X i , X j .

( Monotonicity. ) CNF -1( X i , X j ) > CNF -1( X k , X l ) implies that the pair of variables X i , X j are more strongly confounded than the pair of variables X k , X l in the sense of Defns. 4.2 and 4.3.

4.2 Setting 2: Detecting and Measuring Confounding Using the Mechanism Shifts of Z .

The previous setting utilizes the interventional effects of X i ( X j ) on X j ( X i ) to define a measure of confounding between X i , X j . In this setting, we utilize the association between the observed marginal distributions of X i , X j under causal mechanism shifts of Z to measure confounding. To this end, similar to [38], we make the following assumption.

Assumption 4.1. (Shift Faithfulness [38]) Let Z be a common parent for a set of variables X S ⊆ X . Then each causal mechanism shift in Z between two contexts c, c ′ entails a causal mechanism change in each X i ∈ X S between the same contexts c, c ′ .

One consequence of the Assumption 4.1 is that a change in the causal mechanism of Z induces correlations between the expectations of X i , X j in different contexts. To understand this, consider the following structural equations.

Where c denotes the context and ϵ x and ϵ y are noise variables with zero mean and have no additional restriction on the underlying probability distribution. The causal graph corresponding to this model has the nodes X i , X j , Z and edges: Z → X i , Z → X j , X i → X j . It is easy to see that E ( X i ) = αµ ( c ) and E ( X j ) = ( αβ + γ ) µ ( c ) . Following Assumption 4.1, whenever there is a change in causal mechanism of Z (e.g., c changes to ˜ c in Eqn. 6), there is a change in both E ( X i ) , E ( X j ) . Additionally, since Z is a common cause of both X i , X j , there is a spurious association between E ( X i ) , E ( X j ) . Subsequently, in the set of contexts C { i }∧{ j } the values E ( X i ) , E ( X i ) are spuriously associated. Under Assumptions 3.2 and 4.1, restricting our analysis to C { i }∧{ j } ensures that with high probability, the association between E ( X i ) , E ( X i ) is due to the confounding variable Z . In this example, the association between E ( X i ) , E ( X j ) exists even if β = 0 , i.e., X i ̸→ X j . To define confounding measure, we create two random variables E C i , E C j which we define as E C i = E X i ∼ P c ( X i ) ( X i ) , E C j = E X j ∼ P c ( X j ) ( X j ) respectively where c ∈ C { i }∧{ j } . Relying on the context information C { i }∧{ j } and utilizing the association between E C i and E C j , we define a confounding measure as follows.

Proposition 4.2. (Confounding Based on Mutual Information) If two variables X i , X j are confounded by a variable Z , the induced random variables E C i , E C j as described above have non zero mutual information I ( E C i ; E C j ) .

Definition 4.6. (Confounding Measure 2) When the causal mechanism shifts are observed for X i , X j in different contexts and the contexts C { i }∧{ j } are known, under the Assumptions 3.2-4.1, the measure of confounding CNF -2( X i , X j ) between X i and X j is defined as

To measure the unobserved confounding strength when we already observe a confounding variable X o , we condition on the observed confounding variable X o to define CNF -2( X i , X j | X o ) as follows.

Beyond Pairwise Confounding: Following earlier work [38], we utilize total correlation among triplets ( E C i , E C j , E C k ) of random variables in { E C i } i ∈ S to verify whether a set of variables X S are jointly confounded. By Assumption 4.1, we know that the variables in X S jointly confounded only if each pair X i , X j ; i, j ∈ S is pairwise confounded. If all three variables share the same latent confounding variable Z , then knowing about one of E C i , E C j , E C k explains away some of the association between the other two, so that we have I ( E C i , E C j | E C k ) < I ( E C i , E C j ) . However, for a triplet ( X i , X j , X k ) , it is possible that, rather than jointly confounded, there may be three disjoint confounding variables Z 12 , Z 13 , Z 23 confounding each of the individual pairs: ( X i , X j ) , ( X j , X k ) , ( X k , X i ) . In general, for a set of variables of size s to permit such an equivalent explanation, we would need to have a total of ( s 2 ) confounding variables with s ( s -1) outgoing edges to obtain the same structure of pairwise confounding [38]. While this may plausibly occur for small sets of variables that appear to be pairwise correlated, we assume the true graph G to be causally minimal in the following sense.

Assumption 4.2. (Confounder Minimality [38]) For every subset X S of at least | S | ≥ 4 variables, there are at most 2 | S | edges incoming into X S from latent confounding variables with at least three children in X S .

Assumption 4.2 ensures that variables that appear to be jointly confounded are indeed confounded. In other words, when a small number of latent variables suffice to explain the observed correlations, there should indeed exist only few confounding variables. With this assumption, we can guarantee that joint confounding can be identified from the total correlation.

Theorem 4.3. Let X S be a set of variables such that all X i , X j ∈ X S are pairwise confounded. Then X S is jointly confounded if and only if for each triple X i , X j , X k ∈ X S we have I ( E C i ; E C j | E C k ) < I ( E C i ; E C j ) .

Now, the measure of joint confounding among a set of variables X S can be defined using total correlation T ( E C i , . . . , E C | S | ) as follows. To evaluate the following expression, we need to use the contexts C { 1 }∪···∪{| S |} to ensure that with high probability, the association among the variables in X S is due to the joint confounding variable Z .

Theorem 4.4. For any three observed variables X i , X j , X o and an unobserved confounding variable Z , the following statements are true for the measure CNF -2 .

( Reflexivity and Symmetry. ) CNF -2( X i , X i | X o ) = 1 -e -H ( E C i | X o ) ∀ i where H ( . | . ) denotes conditional entropy and CNF -2( X i , X j | X o ) = CNF -2( X j , X i | X o ) .

( Positivity. ) CNF -2( X i , X j ) > 0 if and only if X i , X j are confounded. Given an observed confounding variable X o between X i , X j , CNF -2( X i , X j | X o ) > 0 if and only if there exists an unobserved confounding variable Z between X i , X j .

( Monotonicity. ) CNF -2( X i , X j ) > CNF -2( X k , X l ) implies that the pair of variables X i , X j are more strongly confounded than the pair of variables X k , X l in the sense of Defn. 4.2.

4.3 Setting 3: Observing the Causal Mechanism Shifts in Z and Known Causal Path Direction Between X i and X j

Similar to the previous settings, we utilize marginal and conditional distributions of X i , X j to define a measure of confounding. By prior knowledge, if we know the direction of causal path between X i , X j , we can utilize the causal direction to measure confounding as explained below. In addition to the notations E C i , E C j introduced in the previous setting, let us denote for each c ∈ C { i }∧{ j } , E X i ∼ P c ( X i | X j ) ( X i | X j ) , E X j ∼ P c ( X j | X i ) ( X j | X i ) with E C ij , E C ji respectively. We now leverage dependency among these variables to define the measure of confounding. Intuitively, if X i → X j and if we observe a change in the causal mechanisms of both X i , X j due to the causal mechanism changes in Z , we also observe a change in the causal mechanism P ( X j | X i ) .

Definition 4.7. (Confounding Measure 3) When the causal mechanism shifts are observed for X i , X j and the causal direction between the nodes X i , X j is known, under the Assumptions 3.2-4.1, the measure of confounding CNF -3( X i , X j ) between X i ∈ X and X j ∈ X is defined as

To measure the unobserved confounding strength in the presence of an observed confounding variable X o , similar to setting 2, we can modify Eqn. 10 to condition on the variable X o .

Beyond Pairwise Confounding: Using the Assumption 4.2, we have the following.

Theorem 4.5. Let X S be a set of variables such that all X i , X j ∈ X S are pairwise confounded and the causal relationships among each pair X i , X j . Then X S is jointly confounded if and only if for each triple X i , X j , X k ∈ X S we have I ( E C ij ; E C jk | E C j ) < I ( E C ij ; E C jk ) .

Since we have access to random variables E C ij in addition to E C i , E C j , it is not straightforward to use all of them to measure joint confounding. To keep the measure simple, we let the measure of joint confounding among the variables X S be the same as CNF -2( X S ) . That is, CNF -3( X S ) = CNF -2( X S ) . Setting 3 is an alternative to Setting 2 when we know the direction of the causal path between X i , X j . Settings 2 and 3 act as complementary to each other in validating the correctness of our analysis.

Theorem 4.6. For any three observed variables X i , X j , X o and an unobserved confounding variable Z , the following statements are true for the measure CNF -3 .

( Reflexivity and Symmetry. ) CNF -3( X i , X i | X o ) = 1 -e -H ( E C i | X o ) ∀ i where H ( . | . ) denotes conditional entropy and CNF -3( X i , X j | X o ) = CNF -3( X j , X i | X o ) .

( Positivity. ) CNF -3( X i , X j ) > 0 if and only if X i , X j are confounded. Given an observed confounding variable X o between X i , X j , CNF -3( X i , X j | X o ) > 0 if and only if there exists an unobserved confounding variable Z between X i , X j .

( Monotonicity. ) CNF -3( X i , X j ) > CNF -3( X k , X l ) implies that the pair of variables X i , X j are more strongly confounded than the pair of variables X k , X l in the sense of Defn. 4.2.

5 Algorithm

Algorithm 1 outlines the procedures to measure confounding in all three settings and can be extended to the case where we evaluate conditional confounding and evaluating confounding among multiple variables. We present two real-world examples where our methods can be applied in Appendix § B.

Data: Context information C { i }∧¬ P ij , C { j }∧¬ P ji , C { i }∧{ j } , Contextual Datasets {D c } c ∈ C . Result: CNF -1( X i , X j ) , CNF -2( X i , X j ) , CNF -3( X i , X j ) Step 1: Evaluate P ( X i | X j ) , P ( X j | X i ) using observational data; Step 2: Evaluate P ( X i | do ( X j )) using {D c } c ∈ C { j }∧¬ P ji ; Step 3: Evaluate P ( X j | do ( X i )) using {D c } c ∈ C { i }∧¬ P ij ; Step 4: Evaluate I ( X i → X j ) , I ( X j → X i ) ; Step 5: CNF -1( X i , X j ) = 1 - e - min( I ( X i → X j ) ,I ( X j → X i )) ; Step 6: Evaluate E C i , E C j using {D c } c ∈ C { i }∧{ j } ; Step 7: CNF -2( X i , X j ) = 1 - e - I ( E c i ; E c j ) ; Step 8: Evaluate E C ij , E C ji using {D c } c ∈ C { i }∧{ j } ; Step 9: compute CNF -3( X i , X j ) according to Defn. 4.7; return CNF -1( X i , X j ) , CNF -2( X i , X j ) , CNF -3( X i , X j )

6 Experiments and Results

We perform simulation studies to verify the correctness of the proposed measures. All the experiments are run on a CPU. We report the mean and standard deviation of results taken over five random seeds. Code to reproduce the results is presented in the supplementary material. Code is available at https://github.com/gautam0707/CD_CNF .

Measuring Confounding: In this set of experiments, we consider the following four causal structures made of three nodes X i , X j , Z : G 1 : Empty graph over Z, X i , X j i.e., nodes are isolated in the graph, G 2 : X i → X j , G 3 : Z → X i , Z → X j , G 4 : Z → X i , Z → X j , X i → X j . In G 1 , G 2 , there is no confounding between X i , X j and in G 3 , G 4 there is confounding effect of Z on X i and X j . Results in Fig. 2 show that our measures output zero when there is no confounding between X i , X j and output positive values when X i , X j are confounded by a confounding variable Z .

1000

2000

3000

4000

5000

Number of Samples

0.00

0.05

0.10

0.15

CNF

1(

,

Xj

)

Setting 1

(Cnf)

4

(Cnf)

3

(Uncnf)

2

(Uncnf)

1

1000

2000

3000

4000

5000

Number of Samples

0.0

0.1

0.2

CNF

2(

,

Xj

)

Setting 2

(Cnf)

4

(Cnf)

3

(Uncnf)

2

(Uncnf)

1

1000

2000

3000

4000

5000

Number of Samples

0.0

0.1

0.2

CNF

3(

,

Xj

)

Setting 3

(Cnf)

4

(Cnf)

3

(Uncnf)

2

(Uncnf)

1

Measuring Conditional Confound-

ing: We consider the following two causal structures. G 5 : Z 1 → X i , Z 1 → X j , Z 2 → X i , Z 2 → X j , X i → X j . G 6 : Z → X i , Z → X j , X i → X j . In G 5 , X i and X j are confounded by two variables Z 1 , Z 2 . We measure conditional confounding between X i , X j conditioned on ∅ , Z 1 , and Z 2 respectively. Since confounding still exists in all of the above conditioning settings, CNF -2 correctly returns positive confounding value in

1000

2000

3000

4000

5000

Number of Samples

0.22

0.24

0.26

CNF

2

5

CNF

2(

,

Xj

|

)

CNF

2(

,

Xj

|

Z

1)

CNF

2(

,

Xj

|

Z

2)

1000

2000

3000

4000

5000

Number of Samples

0.10

0.15

0.20

0.25

CNF

2

6

CNF

2(

,

Xj

|

)

CNF

2(

,

Xj

|

Z

)

all three cases (see Fig. 3 left). On the other hand, in G 6 , we measure conditional confounding

between X i , X j conditioning on empty set and Z . Since conditioning on Z will block the confounding association between X i , X j , CNF -2 returns confounding value closer to zero. However, the unconditioned confounding (conditioning on empty set) value is still large. These results empirically validate the correctness of the proposed measures.

Downstream Causal Effect Estimation: For the causal graphs G 3 , G 4 , we examine the impact of controlling for nodes identified using our method. We measure the causal effect of X i

on X j with and without controlling for the detected confounding variable and report the absolute difference between the true and estimated causal effects in Tab. 3. The results show that controlling for the variables identified by our method reduces the bias in the estimated causal effects.

Binary Data - Erdös-Rényi Causal Graphs: To verify the performance of our method on a large scale, similar to [38], we generate causal graphs of various number nodes using Erdös-Rényi model. In these experiments, each context is a result of intervention on one node. This is the reason for having the same value for number of nodes N and number of contexts | C | . Sample size denotes the number of data points used in each context. We detect and measure whether each pair of nodes is confounded or not. We then calculate the Precision, Recall , and F1 scores. Our confounding measures obtain good results across all settings.

7 Conclusions, Limitations, and Future Work

In this paper, based on the known causal mechanism shifts of observed variables, we propose three measures of confounding along with their conditional and multivariate variants. We also study key properties of these measures. Our measures complement each other depending on the available context information. We propose algorithms to compute the proposed measures and empirically verify their correctness. However, for the same confounded pair of variables, our metrics may yield different results depending on the chosen measure. As discussed in the introduction, the measures are intended to assess the relative strengths of confounding rather than for point-to-point comparison. The number of contexts required to evaluate the measure can be large because many contexts without changes in particular mechanisms are discarded. Identifying appropriate real-world datasets and applying the proposed measures to those datasets is an interesting area for future work, as is developing measures that efficiently use context information. Additionally, devising new definitions for confounding and proposing corresponding confounding measures is also an interesting future direction. We aim to pursue these ideas.

Acknowledgments

This work was partly supported by the Prime Minister's Research Fellowship (PMRF) program and a Google Research Scholar Award. We are grateful to the anonymous reviewers for their valuable feedback, which improved the presentation of the paper.

References

John Aldrich. Correlations genuine and spurious in pearson and yule. Statistical science , pages 364-376, 1995.

Rohit Bhattacharya, Tushar Nagarajan, Daniel Malinsky, and Ilya Shpitser. Differentiable causal discovery under unmeasured confounding. In International Conference on Artificial Intelligence and Statistics , pages 2314-2322, 2021.

Norman E Breslow, Nicholas E Day, and Elisabeth Heseltine. Statistical methods in cancer research. 1980.

Philippe Brouillard, Sébastien Lachapelle, Alexandre Lacoste, Simon Lacoste-Julien, and Alexandre Drouin. Differentiable causal discovery from interventional data. In Advances in Neural Information Processing Systems , volume 33, pages 21865-21877, 2020.

Esben Budtz-Jørgensen, Niels Keiding, Philippe Grandjean, and Pal Weihe. Confounder selection in environmental epidemiology: Assessment of health effects of prenatal mercury exposure. Annals of Epidemiology , 17(1):27-35, 2007.

Timothy A Carey and William B Stiles. Some problems with randomized controlled trials and some viable alternatives. Clinical Psychology & Psychotherapy , 23(1):87-95, 2016.

Venkat Chandrasekaran, Pablo A Parrilo, and Alan S Willsky. Latent variable graphical model selection via convex optimization. In 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 1610-1613. IEEE, 2010.

David Maxwell Chickering. Learning equivalence classes of bayesian-network structures. The Journal of Machine Learning Research , 2:445-498, 2002.

Diego Colombo, Marloes H Maathuis, Markus Kalisch, and Thomas S Richardson. Learning high-dimensional directed acyclic graphs with latent and selection variables. The Annals of Statistics , pages 294-321, 2012.

Shai Ben David, Tyler Lu, Teresa Luu, and Dávid Pál. Impossibility theorems for domain adaptation. In AISTATS , pages 129-136, 2010.

Mirthe Maria Van Diepen, Ioan Gabriel Bucur, Tom Heskes, and Tom Claassen. Beyond the markov equivalence class: Extending causal discovery under latent confounding. In 2nd Conference on Causal Learning and Reasoning , 2023.

Frederick Eberhardt and Richard Scheines. Interventions and causal inference. Philosophy of science , 74(5):981-995, 2007.

Frederick Eberhardt, Clark Glymour, and Richard Scheines. On the number of experiments sufficient and in the worst case necessary to identify all causal relations among n variables. arXiv preprint arXiv:1207.1389 , 2012.

Robin J Evans and Thomas S Richardson. Smooth, identifiable supermodels of discrete dag models with latent variables. Bernoulli , 25:848-876, 2019.

Sander Greenland and Hal Morgenstern. Confounding in health research. Annual review of public health , 22(1):189-212, 2001.

R.H.H. Groenwold, E. Hak, and A.W. Hoes. Quantitative assessment of unobserved confounding is mandatory in nonrandomized intervention studies. Journal of Clinical Epidemiology , 62(1): 22-28, 2009.

Siyuan Guo, Viktor Tóth, Bernhard Schölkopf, and Ferenc Huszár. Causal de finetti: On the identification of invariant causal structure in exchangeable data. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Gemma Hammerton and Marcus R Munafò. Causal inference with observational data: the need for triangulation of evidence. Psychological medicine , 51(4):563-578, 2021.

Alain Hauser and Peter Bühlmann. Two optimal strategies for active learning of causal models from interventional data. International Journal of Approximate Reasoning , 55(4):926-939, 2014.

Patrik O Hoyer, Shohei Shimizu, Antti J Kerminen, and Markus Palviainen. Estimation of causal effects using linear non-gaussian causal models with hidden variables. International Journal of Approximate Reasoning , 49(2):362-378, 2008.

Biwei Huang, Kun Zhang, Jiji Zhang, Ruben Sanchez-Romero, Clark Glymour, and Bernhard Schölkopf. Behind distribution shift: Mining driving forces of changes and causal arrows. In 2017 IEEE International Conference on Data Mining (ICDM) , pages 913-918. IEEE, 2017.

Biwei Huang, Kun Zhang, Jiji Zhang, Joseph Ramsey, Ruben Sanchez-Romero, Clark Glymour, and Bernhard Schölkopf. Causal discovery from heterogeneous/nonstationary data. Journal of Machine Learning Research , 21(89):1-53, 2020.

Amin Jaber, Murat Kocaoglu, Karthikeyan Shanmugam, and Elias Bareinboim. Causal discovery from soft interventions with unknown targets: Characterization and learning. Advances in neural information processing systems , 33:9551-9561, 2020.

Holly Janes, Francesca Dominici, and Scott Zeger. On quantifying the magnitude of confounding. Biostatistics , 11(3):572-582, 2010.

Dominik Janzing and Bernhard Schölkopf. Detecting confounding in multivariate linear models via spectral analysis. Journal of Causal Inference , 6(1):20170013, 2018.

Andrew Jesson, Alyson Rose Douglas, Peter Manshausen, Maëlys Solal, Nicolai Meinshausen, Philip Stier, Yarin Gal, and Uri Shalit. Scalable sensitivity and uncertainty analyses for causaleffect estimates of continuous-valued interventions. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

David Kaltenpoth and Jilles Vreeken. Nonlinear causal discovery with latent confounders. In International Conference on Machine Learning , pages 15639-15654, 2023.

David Kaltenpoth and Jilles Vreeken. Causal discovery with hidden confounders using the algorithmic Markov condition. In Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence , pages 1016-1026, 2023.

Rickard Karlsson and Jesse Krijthe. Detecting hidden confounding in observational data using multiple environments. Advances in Neural Information Processing Systems , 36, 2023.

David G Kleinbaum, Kevin M Sullivan, and Nancy D Barker. A pocket guide to epidemiology . Springer, 2007.

Charles Ksir and Carl L Hart. Correlation still does not imply causation. The Lancet Psychiatry , 3(5):401, 2016.

Paul H Lee. Is a cutoff of 10% appropriate for the change-in-estimate criterion of confounder identification? Journal of epidemiology , 24(2):161-167, 2014.

Adam Li, Amin Jaber, and Elias Bareinboim. Causal discovery from observational and interventional data across multiple environments. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

George Maldonado and Sander Greenland. Simulation study of confounder-selection strategies. American journal of epidemiology , 138(11):923-936, 1993.

George Maldonado and Sander Greenland. Estimating causal effects. International journal of epidemiology , 31(2):422-429, 2002.

Sarah Mameche, David Kaltenpoth, and Jilles Vreeken. Discovering invariant and changing mechanisms from data. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1242-1252, 2022.

Sarah Mameche, David Kaltenpoth, and Jilles Vreeken. Learning causal models under independent changes. Advances in Neural Information Processing Systems , 36, 2023.

Sarah Mameche, Jilles Vreeken, and David Kaltenpoth. Identifying confounding from causal mechanism shifts. In Proceedings of the 27th International Conference on Artificial Intelligence and Statistics (AISTATS) . PMLR, 2024.

Olli S Miettinen and E Francis Cook. Confounding: essence and detection. American journal of epidemiology , 114(4):593-603, 1981.

Joris M Mooij, Sara Magliacane, and Tom Claassen. Joint causal inference from multiple contexts. Journal of Machine Learning Research , 21:1-108, 2020.

Austin Nichols. Causal inference with observational data. The Stata Journal , 7(4):507-541, 2007.

Juan Miguel Ogarrio, Peter Spirtes, and Joe Ramsey. A hybrid causal search algorithm for latent variable models. In Proceedings of the Eighth International Conference on Probabilistic Graphical Models , pages 368-379, 2016.

Menglan Pang, Jay S Kaufman, and Robert W Platt. Studying noncollapsibility of the odds ratio with marginal structural and logistic regression models. Statistical methods in medical research , 25(5):1925-1937, 2016.

Judea Pearl. Causality . Cambridge university press, 2009.

Ronan Perry, Julius Von Kügelgen, and Bernhard Schölkopf. Causal discovery in heterogeneous environments under the sparse mechanism shift hypothesis. In Advances in Neural Information Processing Systems , pages 10904-10917, 2022.

Jonas Peters, Joris M Mooij, Dominik Janzing, and Bernhard Schölkopf. Causal discovery with continuous additive noise models. The Journal of Machine Learning Research , 15(1): 2009-2053, 2014.

Jonas Peters, Dominik Janzing, and Bernhard Schlkopf. Elements of Causal Inference: Foundations and Learning Algorithms . The MIT Press, 2017.

Maxim Raginsky. Directed information and pearl's causal calculus. In 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 958-965, 2011.

Thomas S Richardson, Robin J Evans, James M Robins, and Ilya Shpitser. Nested markov properties for acyclic directed mixed graphs. The Annals of Statistics , 51(1):334-361, 2023.

Robert William Sanson-Fisher, Billie Bonevski, Lawrence W. Green, and Cate D'Este. Limitations of the randomized controlled trial in evaluating population-based health interventions. American Journal of Preventive Medicine , 33(2):155-161, 2007.

Mauro Scanagatta, Cassio P de Campos, Giorgio Corani, and Marco Zaffalon. Learning bayesian networks with thousands of variables. Advances in neural information processing systems , 28, 2015.

B Schölkopf, D Janzing, J Peters, E Sgouritsa, K Zhang, and J Mooij. On causal and anticausal learning. In 29th International Conference on Machine Learning (ICML 2012) , pages 12551262. International Machine Learning Society, 2012.

Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE , 109(5):612-634, 2021.

Noah A Schuster, Jos WR Twisk, Gerben Ter Riet, Martijn W Heymans, and Judith JM Rijnhart. Noncollapsibility and its role in quantifying confounding bias in logistic regression. BMC medical research methodology , 21:1-9, 2021.

Karthikeyan Shanmugam, Murat Kocaoglu, Alexandros G Dimakis, and Sriram Vishwanath. Learning causal graphs with small interventions. Advances in Neural Information Processing Systems , 28, 2015.

Ilya Shpitser, Robin J Evans, Thomas S Richardson, and James M Robins. Introduction to nested markov models. Behaviormetrika , 41:3-39, 2014.

Ilya Shpitser, Robin J Evans, and Thomas S Richardson. Acyclic linear sems obey the nested markov property. In Uncertainty in artificial intelligence: proceedings of the... conference. Conference on Uncertainty in Artificial Intelligence , volume 2018. NIH Public Access, 2018.

Edward H Simpson. The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society: Series B (Methodological) , 13(2):238-241, 1951.

Peter Spirtes and Kun Zhang. Causal discovery and inference: concepts and recent methodological advances. In Applied informatics , volume 3, pages 1-28. Springer, 2016.

Peter Spirtes, Clark N Glymour, and Richard Scheines. Causation, prediction, and search . 2000.

Zhiqiang Tan. A distributional approach for causal inference using propensity scores. Journal of the American Statistical Association , 101(476):1619-1637, 2006.

Tyler J VanderWeele and Ilya Shpitser. On the definition of a confounder. Annals of statistics , 41(1):196, 2013.

Y. Samuel Wang and Mathias Drton. Causal discovery with unobserved confounding and non-gaussian data. Journal of Machine Learning Research , 24(271):1-61, 2023.

Aleksander Wieczorek and Volker Roth. Information theoretic causal effect quantification. Entropy , 21(10), 2019.

Alessio Zanga, Elif Ozkirimli, and Fabio Stella. A survey on causal discovery: Theory and practice. International Journal of Approximate Reasoning , 151:101-129, 2022.

Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. Dags with no tears: Continuous optimization for structure learning. Advances in neural information processing systems , 31, 2018.

Appendix

A Proofs

Proposition 4.1. (Identifiability of P ( X j | do ( X i )) ) P ( X j | do ( X i )) is identifiable from the set of contexts C { i }∧¬ P ij . To detect and measure confounding between a pair of nodes X i , X j , it is enough to observe two sets of contexts C { i }∧¬ P ij and C { j }∧¬ P ji . Thus, n sets of contexts are needed to detect and measure confounding between ( n 2 ) distinct pairs of nodes in a causal DAG with n nodes.

Proof. Since the set of contexts C { i }∧ P ij consist of data with all possible interventions on X i , if a context c is generated by performing intervention on X i with the value x i , the expression P ( X j | do ( X i = x i )) is equal to the expression P ( X j | X i = x i ) in that context c .

From Defn. 4.4, to detect and measure confounding between the pair of variables X i , X j , we need to evaluate P ( X j | do ( X i )) and P ( X i | do ( X j )) . To this end, from the previous paragraph, we need two sets of contexts C { i }∧ P ij and C { j }∧ P ji . Following these observations, it is enough to have n sets of contexts to detect and measure confounding between ( n 2 ) distinct pairs of nodes.

Theorem 4.1. A set of observed variables X S are jointly unconfounded if and only if there exists three variables X i , X j , X k ∈ X S such that I ( X i → X j | X k ) = I ( { X i , X k } → X j ) .

Proof. Consider three variables X i , X j , X k in the underlying causal graph. Consider the conditional directed information between X i , X j given X k and the subsequent manipulations as follows.

Since I ( X k → X j ) ≥ 0 , we have I ( X i → X j | X k ) ≤ I ( { X i X k } → X j ) . Equality holds only when X k , X j are unconfounded.

Theorem 4.2. For any three observed variables X i , X j , X o and an unobserved confounding variable Z , the following statements are true for the measure CNF -1 .

( Reflexivity and Symmetry. ) CNF -1( X i , X i | X o ) = 0 , CNF -1( X i , X j | X o ) = CNF -1( X j , X i | X o ) .

( Positivity. ) CNF -1( X i , X j ) > 0 if and only if X i , X j are confounded. Given an observed confounding variable X o between X i , X j , CNF -1( X i , X j | X o ) > 0 if and only if there exists an unobserved confounding variable Z between X i , X j .

( Monotonicity. ) CNF -1( X i , X j ) > CNF -1( X k , X l ) implies that the pair of variables X i , X j are more strongly confounded than the pair of variables X k , X l in the sense of Defns. 4.2 and 4.3.

Proof. Reflexivity: From the definition of directed information, I ( X i → X i | X o ) = E P ( X i ,X j ,X o ) log P ( X i | X o ) P ( X i | X o ) = 0 and hence CNF -1( X i , X j | X o ) = 1 -e 0 = 0 .

Symmetry: Even if I ( X i → X j | X o ) is not symmetric, the expression ' min( I ( X i → X j | X o ) , I ( X j → X i | X o )) ' is symmetric and hence CNF -1( X i , X j | X o ) is symmetric.

̸

̸

Positivity: If X i , X j are confounded, irrespective of the direction of the causal path between X i and X j , we have P ( X i | X j ) = P ( X i | do ( X j )) and P ( X j | X i ) = P ( X j | do ( X i )) . Hence I ( X i → X j ) > 0 and I ( X j → X i ) > 0 . We now have CNF -1( X i , X j ) > 0 . The above statement is true even if there is no causal path between the nodes X i , X j . The above statements are valid even after conditioning on an observed confounding variable X o if there is an unobserved confounding between X i , X j .

Monotonicity: Without loss of generality, assume that the inequality CNF -1( X i , X j ) > CNF -1( X k , X l ) is a result of I ( X i → X j ) > I ( X k → X l ) . That is, the KL divergence between P ( X i | X j ) and P ( X i | do ( X j )) is greater than the kl divergence between P ( X k | X l ) and P ( X k | do ( X l )) . That is, the pair of distributions P ( X k | X l ) and P ( X k | do ( X l )) are closer to each other compared to the pair P ( X i | X j ) and P ( X i | do ( X j )) . As a result, X k , X l are closer to being not confounded in the sense of Defns. 4.2 and 4.3.

Proposition 4.2. (Confounding Based on Mutual Information) If two variables X i , X j are confounded by a variable Z , the induced random variables E C i , E C j as described above have non zero mutual information I ( E C i ; E C j ) .

Proof. There are two sources of dependency between E C i , E C j . If X i , X j are causally related in the underlying causal model generating the data, there will be a dependency between E C i , E C j in the context C { i }∧{ j } as the interventions are soft. On the other hand, as per the Assumption 4.1, any shift in the causal mechanism of Z leads to a change in both the mechanisms of X i , X j leading to a dependency. Hence the random variables E C i , E C j have non-zero mutual information.

Theorem 4.3. Let X S be a set of variables such that all X i , X j ∈ X S are pairwise confounded. Then X S is jointly confounded if and only if for each triple X i , X j , X k ∈ X S we have I ( E C i ; E C j | E C k ) < I ( E C i ; E C j ) .

Proof. Following the Assumption 4.2, when three variables X i , X j , X k are confounded by as single confounding variable Z , conditioning on one of E C i , E C j , E C k explains away some of the dependency between other two. Hence we have I ( E C i ; E C j | E C k ) < I ( E C i ; E C j ) for all triples i, j, k .

Theorem 4.4. For any three observed variables X i , X j , X o and an unobserved confounding variable Z , the following statements are true for the measure CNF -2 .

( Reflexivity and Symmetry. ) CNF -2( X i , X i | X o ) = 1 -e -H ( E C i | X o ) ∀ i where H ( . | . ) denotes conditional entropy and CNF -2( X i , X j | X o ) = CNF -2( X j , X i | X o ) .

( Positivity. ) CNF -2( X i , X j ) > 0 if and only if X i , X j are confounded. Given an observed confounding variable X o between X i , X j , CNF -2( X i , X j | X o ) > 0 if and only if there exists an unobserved confounding variable Z between X i , X j .

( Monotonicity. ) CNF -2( X i , X j ) > CNF -2( X k , X l ) implies that the pair of variables X i , X j are more strongly confounded than the pair of variables X k , X l in the sense of Defn. 4.2.

Proof. Reflexivity: from the definition of mutual information, I ( E C i ; E C i | X o ) = H ( E C i | X o ) -H ( E C i | E C i , X o ) = H ( E C i | X o ) . Substituting in the definition of CNF -2( X i , X j ) , result follows.

Symmetry: The result follows from the 'symmetry' property of mutual information.

Positivity: If X i , X j are confounded, from the Assumption 4.1, E C i , E C j are dependent random variables. Hence the mutual information is positive. The result follows after substituting some positive value for I ( E C i ; E C j ) in the definition of CNF -2( X i , X j ) . The same argument goes for conditional confounding.

Monotonicity: from the definition of CNF -2( X i , X j ) , CNF -2( X i , X j ) > CNF -2( X k , X l ) implies I ( E C i ; E C j ) > I ( E C k ; E C l ) . From the Defn. 4.2, X i , X j have higher mutual information than the pair X k , X l and hence X i , X j are more strongly confounded than X k , X l .

Theorem 4.5. Let X S be a set of variables such that all X i , X j ∈ X S are pairwise confounded and the causal relationships among each pair X i , X j . Then X S is jointly confounded if and only if for each triple X i , X j , X k ∈ X S we have I ( E C ij ; E C jk | E C j ) < I ( E C ij ; E C jk ) .

Proof. Following the Assumption 4.2, when three variables X i , X j , X k are confounded by as single confounding variable Z , conditioning on E C k explains away some of the dependency between E C ij , E C jk . Hence we have I ( E C ij ; E C jk | E C j ) < I ( E C ij ; E C jk ) for all triples i, j, k .

Theorem 4.6. For any three observed variables X i , X j , X o and an unobserved confounding variable Z , the following statements are true for the measure CNF -3 .

( Reflexivity and Symmetry. ) CNF -3( X i , X i | X o ) = 1 -e -H ( E C i | X o ) ∀ i where H ( . | . ) denotes conditional entropy and CNF -3( X i , X j | X o ) = CNF -3( X j , X i | X o ) .

( Positivity. ) CNF -3( X i , X j ) > 0 if and only if X i , X j are confounded. Given an observed confounding variable X o between X i , X j , CNF -3( X i , X j | X o ) > 0 if and only if there exists an unobserved confounding variable Z between X i , X j .

( Monotonicity. ) CNF -3( X i , X j ) > CNF -3( X k , X l ) implies that the pair of variables X i , X j are more strongly confounded than the pair of variables X k , X l in the sense of Defn. 4.2.

Proof. Reflexivity: from the definition of mutual information, I ( E C ii ; E C i | X o ) = I ( E C i ; E C i | X o ) = H ( E C i | X o ) -H ( E C i | E C i , X o ) = H ( E C i | X o ) . Substituting in the definition of CNF -3( X i , X j ) , result follows.

Symmetry: Since we rely on the direction of the causal path between X i , X j , for a given pair of nodes X i , X j , we have CNF -3( X i , X j ) = CNF -3( X j , X i ) from Defn. 4.7.

Positivity: If X i , X j are confounded and X i → X j , from the Assumption 4.1, E C ji , E C j are dependent random variables. Hence the mutual information E C ji , E C j is positive. The result follows after substituting positive value for I ( E C ji ; E C j ) in the definition of CNF -3( X i , X j ) . The same argument goes for conditional confounding.

Monotonicity: from the definition of CNF -3( X i , X j ) , without loss of generality, CNF -3( X i , X j ) > CNF -3( X k , X l ) implies I ( E C ji ; E C j ) > I ( E C lk ; E C l ) . From the Defn. 4.2, X i , X j have higher mutual information and hence are more strongly confounded than X k , X l .

B Real-world Examples

Edu

Inv

Wag

Pro

Lab

Exp

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All the claims made in the abstract and introduction are supported with theory and experiments in the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See § 7 for the discussion on the limitations of this work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: For each of the theoretical results, proofs and the assumptions used are presented in the Appendix § A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Experimental setup is presented in main paper and code to reproduce the results is made public. See § 6.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code to reproduce the results is made public. See § 6.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experimental setup is presented in main paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Each experiment is repeated for several random seeds.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Experimental setup is presented in main paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research confirms with NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: To the best of our knowledge, there are no detrimental impacts of our work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All papers, data, and code used are cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Code is documented and provided in supplementary material. Code will be made publicly available.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces some measures of (conditional) confounding, based on information theoretic quantities. Some of their properties and an algorithm to estimate the measures using data from different environments. Although not included in the main text, the authors test their theory on some synthetic data.\n\nSTRENGTHS:\n- The ideas on the paper are original as far as I know and I believe that the question regarding the measurement of confounding is an important one. The authors differentiate their work from previous work.\n- Beyond the errors outlined in the weakness section, the paper is well-written and presented. It is a bit unfortunate that the experiments were not included on the main paper.\n\nWEAKNESSES:\n- There are several typos/mistakes on the paper:\n\n-- On line 158 and 169: $p$ is defined as the inequality between two distributions, this definition doesn’t make sense. I checked their reference 33 and 35 and it says in there that it is the probability of the distributions being different. Furthermore, p-value is a well-defined and known term, why overloading it?\n\n-- On definition 4.1 what they call directed information is not an -expected- KL divergence, it is simply a KL divergence.\n\n-- On Table 2, on rows 1 and 2 the equality with 0 is inverted. For example, if $X_i\\to X_j$, then $P(X_i\\mid X_j)=P(X_i\\mid do(X_j))$ so that the log of such quantity is 0 for all values of $X_i$ and $X_j$, that is $I(X_i\\to X_j)=0$.\n\n-- Is definition 4.6 really a definition? That sounds like a proposition to me.\n\n- I’m unsure of how large is the significance of the contribution being made (see questions and limitations). Which paired with the errors and typos above make me recommend for rejection of the paper.\n\nQUESTIONS:\nI would like to get a feeling for why the properties of their measure of confoundedness are valuable (Theorems 4.2, 4.4 and 4.6). The only really essential property that I can think of such a measure is that it is 0 if the variables are non confounded and not 0 otherwise. Why for example do we want positivity? One could even be interested in a negative measure if the variables have a negative correlation under confoundedness, for example.\n\nLIMITATIONS:\nThis research does not contain any obvious negative societal impact. The authors do include a couple of sentences at the end of the paper where they state that there is a challenge to find real-world data to test their theory. Although I value the author's honesty about this limitation, I think it is too strong of a limitation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the challenge of identifying and quantifying (unobserved) confounding in causal inference. They propose a more comprehensive approach by relaxing the classic assumption of causal sufficiency and leveraging the sparse causal mechanism shifts assumption. The authors introduce methods to detect and measure confounding effects, distinguish observed and unobserved confounding, and evaluate the relative strengths of confounding among variable sets.  An empirical validation supports their theoretical analysis.\n\nSTRENGTHS:\n(S1) The paper provides a thorough study of confounding from several perspectives: detecting and measuring confounding between pairs of variables and among multiple variables; distinguishing between observed and unobserved confounding; assessing the relative strengths of confounding among different sets of observed variables. To my knowledge, this is the first study exploring all these aspects in a unified framework.\n\n(S2) The paper is well written and easy to follow.\n\nWEAKNESSES:\n(W1) I found the paper to be lacking in experimental evaluations. It is not clear how hard (both statistically and computationally) it is to compute the measures of confounding that were proposed, especially the ones measuring confounding among multiple variables. \n\n(W2) I am somehow skeptic about the practical relevance of the results presented in the paper (and the lack of experiments reinforces this point, see W1). In particular, people focused a lot on the marginal sensitivity model (and variations) as a measure of confounding strength because sensitivity analysis bounds can be easily derived from it. I am not sure the same would be possible under the proposed measure of confounding, and hence I am not sure how it would be useful in practice. \n\n\n(W3) I think some relevant related works are missing. In particular, when interventional data is available (e.g. RCTs) [1] and [2] propose testable implications for detecting hidden confounding. Further, [3] and [4] propose to lower bound the strength of hidden confounding (as measured by the marginal sensitivity model). \n\n[1] Falsification of Internal and External Validity in Observational Studies via Conditional Moment Restrictions. Hussain et al. AISTATS 2023.\n\n[2] Benchmarking Observational Studies with Experimental Data under Right-Censoring. Demirel et al. AISTATS 2024. \n\n[3] Hidden yet quantifiable: A lower bound for confounding strength using randomized trials. De Bartolomeis et al. AISTATS 2024.\n\n[4] Detecting critical treatment effect bias in small subgroups. De Bartolomeis et al. UAI 2024\n\nQUESTIONS:\n(Q1) I got confused at lines 103-106: \"While these methods use data from different contexts (an approach we also leverage in this work), they assume the absence of unobserved confounding variables; we instead focus on capturing both observed and unobserved confounding using data from multiple contexts.\". [27] proposes a test for the presence of hidden confounding and hence allows for unobserved variables. Can the author briefly comment on how their measures compare with the approach proposed in [27]? \n\n(Q2) I am confused slightly confused with Assumption 3.2 (line 169):  $$ P^c( X_i | \\text{PA}_i) \\neq P^{c'}(X_i | \\text{PA}_i) $$\nwhat does it mean for conditional distributions to be different?\n \n(Q3) Can the authors comment with one motivating example of how their proposed confounding measure could be used in practice? (See W2)\n\n[27] Rickard Karlsson and Jesse Krijthe. Detecting hidden confounding in observational data using multiple environments. Advances in Neural Information Processing Systems, 36, 2023.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors mainly introduce capturing both observed and unobserved confounding using data from multiple contexts. Leveraging experimental data proposes a comprehensive approach for detecting and measuring confounding effects from three different settings that don't need the parametric assumptions and relaxes the causal sufficiency assumption. The authors provide the measure for detecting confounding effects (relative strengths of confounding).\nFor each of the proposed measures, this article presents key properties and an algorithm for detecting and measuring confounding using data from multiple contexts.\n\nSTRENGTHS:\nThe entire article primarily introduces the detection and quantification of confounding effects without making parametric or causal sufficiency assumptions. It presents a well-structured logical framework to discuss this concept.\n\nWEAKNESSES:\nI think overall the authors did interesting research, but my main concerns are listed below.\n\n1. When the environment changes, for example, if $c$ changes to $c'$, will the original causal relationship change?\n\n2. If the length of the environment node is only 1, will the method described in the article fail?\n\n3. Why do settings 2 and 3 introduce mutual information to define confounding? Would there be any difficulties in using the KL divergence?\n\n4. In line 104, the author said \"an approach we also leverage in this work\", but the author did not introduce the method in which article in detail.\n\n5. In line 111, the article describes \"we measure the effects of both observed and unobserved confounding\". However, the subsequent sections do not appear to provide a direct method for measuring causal effects.\n\n\n6. In fact, the confounding effect is not always a number between 0 and 1. Converting it to a number between 0 and 1 only measures its relative strength, but does not directly calculate its confounding effect.\n\n7. In Setting 1: When there is confounding between two variables, CNF-1 is used to calculate it. If multiple variables have common confounding, CNF-1 is calculated for each variable with the others and then summed. Will this lead to the repeated accumulation of confounding?\n\n8. The detect confounding method proposed in the article, but this measure is relatively rarely used in real life. In fact, it is unrealistic to only conduct simulation experiments. In this part of the experiments, the author did not specifically introduce the settings, so why were only the CNF-2 results output, and not the CNF-3 results? Is there a significant difference between these two results? For the section on \"Measuring Conditional Confounding,\" the author should add experiments involving observed confounding, such as replacing the unobserved confounding with observed confounding in Experiment 1. \n\n9. The authors repeat write “decision variables” in lines 149 or 150, and \"two\" in line 156. \n\n10. The authors miswrite “$\\mathcal{D^{C}}$” as “$\\mathcal{D}$” in the third line of ALgorithm 1.\n\nQUESTIONS:\nSee the above Weaknesses.\n\nLIMITATIONS:\nThe authors provide a detailed discussion of the limitations and applicability of their method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents methods that: \n1. define a measure of confounding between sets of variables,\n2. separate the effects of observed and unobserved confounders, and\n3.  assess the relative strengths of confounding between sets of variables,\n\nin three different settings. In each setting they assume that data from several different contexts is available, and that the changes in causal mechanisms between the different contexts are known.\n\nSTRENGTHS:\nThis paper provides thorough theoretical justification for their methods. The analyses given for the latter two settings seem particularly novel to me.\n\nWEAKNESSES:\nThe assumption \"that the causal mechanism changes are known for each variable across different contexts\" seems to be quite strong, but is not mentioned until relatively late in the paper (line 160).\n\nThe formal definition of a mechanism shift comes quite late in the paper. It would be nice to have intuitive explanations in the abstract and introduction. I also wonder if the formal notation could start with the notion of multiple environments baked in. As it is written now, it is unclear what the mechanism shift is relative to. Is it a shift between an environmental distribution and the interventional distribution, or is it only necessary to know the relative shift between environments?\n\nThe relationship between definitions 4.2 and 4.3 and ignorability and conditional ignorability as independence and conditional independence assumptions needs to be added to the related works. There is also perhaps a connection to [Scalable Sensitivity and Uncertainty Analyses for Causal-Effect Estimates of Continuous-Valued Interventions](https://arxiv.org/pdf/2204.10022) that could be discussed. Notably, the relationship between the likelihood ratios and the KL divergence between the interventional and observational distribution mentioned in Appendix A.3.1.\n\nWhile the theoretical results are thorough, the empirical results are very limited. A major concern with hidden confounding is that it can arbitrarily bias estimated treatment effects. The magnitude of the bias should be straightforward to calculate in your synthetic experiments. It might be more demonstrative to show how the proposed measures correlate to induced biases.\n\n## Points of confusion\n\n**Definition 3.1**\n- Looks like $X$ is being redefined here as to include any variable in $V$ rather than the subset that excludes $Z$. Mildly confusing.\n- line 131: $P(X_i \\mid \\mathrm{PA}_i)$ is not defined in definition 3.1\n\n**Assumption 3.1**\n- talks about changes in causal mechanisms, but this concept hasn't been formalized yet which makes this assumption hard to understand.\n\n**Line 147:** what is a context node? what is an extended causal graph? Might help to define context specific distributions before rather than after.\n\n**Line 156:** two two\n\n**Line 156:** $P^c$ notation could be introduced earlier to formalize a mechanism shift.\n\n**Assumption 3.3:** $dPC$ is missing brackets\n\n**Line 182.** shifts -> shift\n\n**Line 183:** \"extent of hidden confounding.\" word choice is a bit vague. maybe strength, or the effect of the confounder on each variable, or how much it may bias some estimand.\n\n**Definition 4.1:** The notation is a little awkward with the condition on $P(X_j)$ within $D_{\\mathrm{KL}}$. \n\n**Theorem 4.2:** strongly -> more strongly\n\n**Definition 4.6:** missing definition of mutual information.\n\nQUESTIONS:\n**line 128:** unclear notation, what information is $P_x(V), X \\subseteq V$ adding here? If $P_*$ is a set, why not write the set that defines it instead?\n**line 128:** (i.e., $X = \\emptyset$). why isn't $X$ bold?\n\n**Line 154:** \"Let $C_{S \\wedge \\neg R}$ be the set of contexts in which we observe mechanism changes for the set of variables $X_S$ but not for the variables $X_R$.\" Changes relative to what? The interventional distribution? Just between two contexts? Is the shift the same across each of the contexts in this set?\n\n**Line 158:** This does not look like a p-value. This looks like an indicator taking a value of true or false, not a probability. Is there a missing $P$ outside of the brackets? If so, what is the relevant random variable? $c$? If $c$, confusing to have it lower-case while a potential value $X_i$ is upper case. Same questions for **Assumption 3.2**.\n\n**Theorem 4.1:** missing comma? $\\\\{X_iX_j\\\\}$ -> $\\\\{X_i, X_j\\\\}$\n\n**Line 286:** Are the $\\epsilon$ normally distributed? or is zero mean and any distribution ok?\n\n**Definition 4.7:** mutual information is defined between random variables. Are these random variables with respect to contexts? If so lower case $c$ is a bit confusing.\n\nLIMITATIONS:\nThe authors adequately discuss the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}