SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Drago Plecko and Elias Bareinboim
dp3144@columbia.edu, eb@cs.columbia.edu
One of the fundamental challenges found throughout the data sciences is to explain why things happen in specific ways, or through which mechanisms a certain variable  $X$  exerts influences over another variable  $Y$ . In statistics and machine learning, significant efforts have been put into developing machinery to estimate correlations across variables efficiently. In causal inference, a large body of literature is concerned with the decomposition of causal effects under the rubric of mediation analysis. However, many variations are spurious in nature, including different phenomena throughout the applied sciences. Despite the statistical power to estimate correlations and the identification power to decompose causal effects, there is still little understanding of the properties of spurious associations and how they can be decomposed in terms of the underlying causal mechanisms. In this manuscript, we develop formal tools for decomposing spurious variations in both Markovian and Semi-Markovian models. We prove the first results that allow a non-parametric decomposition of spurious effects and provide sufficient conditions for the identification of such decompositions. The described approach has several applications, ranging from explainable and fair AI to questions in epidemiology and medicine, and we empirically demonstrate its use.
Understanding the relationships of cause and effect is one of the core tenets of scientific inquiry and the human ability to explain why events occurred in the way they did. Hypotheses on possible causal relations in the sciences are often generated based on observing correlations in the world, after which a rigorous process using either observational or experimental data is employed to ascertain whether the observed relationships are indeed causal. One common way of articulating questions of causation is through the average treatment effect (ATE), also known as the total effect (TE), given by
$$
\mathbb {E} [ y \mid d o (x _ {1}) ] - \mathbb {E} [ y \mid d o (x _ {0}) ], \tag {1}
$$
where  $do(\cdot)$  symbolizes the do-operator [9], and  $x_0, x_1$  are two distinct values attained by the variable  $X$ . Instead of just quantifying the causal effect, researchers are more broadly interested in determining which causal mechanisms transmit the change from  $X$  to  $Y$ . Such questions have received much attention and have been investigated under the rubric of causal mediation analysis [3, 12, 10, 14].
Often, however, the causal relationship may be entirely absent or account only for a part of the initially observed correlation. In these cases, the spurious (or confounded) variations between  $X$  and  $Y$  play a central role in explaining the phenomenon at hand. Interestingly, though, tools for decomposing spurious variations are almost entirely missing from the literature in causal inference<sup>1</sup>.
<sup>1</sup>The only previous work which considers decompositions of spurious effects is [16]. However, this work considers the specific case of the covariance operator, and no claims are made about the general setting.
Phenomena in which spurious variations are of central importance are abundant throughout the sciences. For instance, in medicine, the phenomenon called the obesity paradox signifies the counterintuitive association of increased body fat with better survival chances in the intensive care unit (ICU) [6]. While the full explanation is still unclear, evidence in the literature suggests that the relationship is not causal [5], i.e., it is explained by spurious variations. Spurious variations also play a central role in many epidemiological investigations [13]. In occupational epidemiology, for example, the relationship of exposure to hazardous materials with cancer is confounded by other hazardous working conditions and lifestyle characteristics [4], and such spurious variations themselves may be the target of scientific inquiry. Quantities that measure such spurious variations (or a subset thereof) are called spurious effects in this paper.
Spurious variations are key in applications of fair and explainable AI as well. For instance, consider the widely recognized phenomenon in the literature known as redlining [15, 7], in which the location where loan applicants live may correlate with their race. Applications might be rejected based on the zip code, disproportionately affecting certain minority groups. Furthermore, in the context of criminal justice [8], the association of race with increased probability of being classified as high-risk for recidivism may in part be explained by the spurious association of race with other demographic characteristics (we take a closer look at this issue in Sec. 5). Understanding which confounders affect the relationship, and how strongly, is an important step of explaining the phenomenon, and also determining whether the underlying classifier is deemed as unfair and discriminatory.
These examples suggest that a principled approach for decomposing spurious variations may be a useful addition to the general toolkit of causal inference, and may find its applications in a wide range of settings from medicine and public health all the way to fair and explainable AI. For concreteness, in this paper we will consider the quantity
$$
P (y \mid x) - P (y \mid d o (x)),
$$
Figure 1: Exp-SE representation.
which we will call the experimental spurious effect (Exp-SE, for short). This quantity, shown graphically in Fig. 1, captures the difference in variations when observing  $X = x$  vs. intervening that  $X = x$ , which can be seen as the spurious counterpart of the total effect. Interestingly, the Exp-SE quantity is sometimes evoked in the causal inference literature, i.e.,
$$
P (y \mid x) - P (y \mid d o (x)) = 0 \tag {2}
$$
is known as the zero-bias condition [2, 9, Ch. 6]. This condition allows one to test for the existence of confounding between the variables  $X$  and  $Y$ . A crucial observation is that, in many cases, the quantity itself may be of interest (instead of only its null), as it underpins the spurious variations.
Against this background, we note that tools that allow for decomposing the Exp-SE quantity currently do not exist in the literature. Our goal in this manuscript is to fill in this gap, and provide a formalism that allows for non-parametric decompositions of spurious variations. Specifically, our contributions are the following:
(i) We introduce the notion of a partially abducted submodel (Def. 1), which underpins the inference procedure called Partial Abduction and Prediction (Alg. 2) (akin to Balke & Pearl 3-step procedure [9, Ch. 7]). Building on this new primitive, we prove the first non-parametric decomposition result for spurious effects in Markovian models (Thm. 1),
(ii) Building on the insights coming from the new procedure, we prove the decomposition result for settings when unobserved confounding is present (Semi-Markovian models) (Thm. 3).
(iii) We develop sufficient conditions for identification of spurious decompositions (Thm 2, 4).
2 Preliminaries
We use the language of structural causal models (SCMs) as our basic semantical framework [9]. A structural causal model (SCM) is a tuple  $\mathcal{M} \coloneqq \langle V, U, \mathcal{F}, P(u) \rangle$ , where  $V, U$  are sets of endogenous (observables) and exogenous (latent) variables respectively,  $\mathcal{F}$  is a set of functions  $f_{V_i}$ , one for each  $V_i \in V$ , where  $V_i \gets f_{V_i}(\mathrm{pa}(V_i), U_{V_i})$  for some  $\mathrm{pa}(V_i) \subseteq V$  and  $U_{V_i} \subseteq U$ .  $P(u)$  is a strictly positive probability measure over  $U$ . Each SCM  $\mathcal{M}$  is associated to a causal diagram  $\mathcal{G}$  [9] over
Figure 2: Graphical representations of the SCM in Ex. 1.
the node set  $V$  where  $V_{i}\rightarrow V_{j}$  if  $V_{i}$  is an argument of  $f_{V_j}$ , and  $V_{i}\leftrightarrow V_{j}$  if the corresponding  $U_{V_i},U_{V_j}$  are not independent [2]. A model with no bidirected edges is called Markovian, while a model with bidirected edges is called Semi-Markovian. An instantiation of the exogenous variables  $U = u$  is called a unit. By  $Y_{x}(u)$  we denote the potential response of  $Y$  when setting  $X = x$  for the unit  $u$ , which is the solution for  $Y(u)$  to the set of equations obtained by evaluating the unit  $u$  in the submodel  $\mathcal{M}_x$ , in which all equations in  $\mathcal{F}$  associated with  $X$  are replaced by  $X = x$ . In a slight abuse of notation, we also replace  $Y = y$  with just  $y$  whenever the former is clear from the context. We next introduce an important inferential procedure for solving different tasks in causal inference.
2.1 Abduction, Action and Prediction
The steps of the abduction-action-prediction method can be summarized as follows:
Algorithm 1 (Abduction, Action and Prediction [9]). Given an SCM  $\langle \mathcal{F}, P(u) \rangle$ , the conditional probability  $P(Y_C \mid E = e)$  of a counterfactual sentence "if it were  $C$  then  $Y$  ", upon observing the evidence  $E = e$ , can be evaluated using the following three steps:
(i) Abduction - update  $P(u)$  by the evidence  $e$  to obtain  $P(u \mid e)$ ,
(ii) Action - modify  $\mathcal{F}$  by the action  $do(C)$ , where  $C$  is an antecedent of  $Y$ , to obtain  $\mathcal{F}_C$ ,
(iii) Prediction - use the model  $\langle \mathcal{F}_C, P(u \mid e) \rangle$  to compute the probability of  $Y_C$ .
In the first step, the probabilities of the exogenous variables  $U$  are updated according to the observed evidence  $E = e$ . Next, the model  $\mathcal{M}$  is modified to a submodel  $\mathcal{M}_C$ . The action step allows one to consider queries related to interventions or imaginative, counterfactual operations. In the final step, the updated model  $\langle \mathcal{F}_C, P(u \mid e) \rangle$  is used to compute the conditional probability  $P(y_C \mid e)$ . There are two important special cases of the procedure. Whenever the action step is empty, the procedure handles queries in the first, associative layer of the Pearl's Causal Hierarchy (PCH, [2]). Whenever the abduction step is empty, but the action step is not, the procedure handles interventional queries in the second layer of the PCH. The combination of the two steps, more generally, allows one to consider queries in all layers of the PCH, including the third, counterfactual layer. In the following example, we look at the usage of the procedure on some queries.
Example 1 (Abduction, Action, Prediction). Consider the following SCM:
$$
\mathcal {F}: \left\{ \begin{array}{l} X \leftarrow f _ {X} \left(U _ {X}, U _ {X Z}\right) \\ Z \leftarrow f _ {Z} \left(U _ {Z}, U _ {X Z}\right) \\ Y \leftarrow f _ {Y} \left(X, Z, U _ {Y}\right), \end{array} \right. \tag {3}
$$
with  $P(U_{X}, U_{XZ}, U_{Z}, U_{Y})$  the distribution over the exogenous variables. The causal diagram of the model is shown in Fig. 2a, with an explicit representation of the exogenous variables in Fig. 2b.
We are first interested in the query  $P(y \mid x)$  in the given model. Based on the abduction-prediction procedure, we can simply compute that:
$$
P (y \mid x) = \sum_ {u} \mathbb {1} (Y (u) = y) P (u \mid x) = \sum_ {u} \mathbb {1} (Y (u) = y) P \left(u _ {z}, u _ {y}\right) P \left(u _ {x}, u _ {x z} \mid x\right). \tag {6}
$$
where the first step follows from the definition of the observational distribution, and the second step follows from noting the independence  $U_Z, U_Y \perp U_X, U_{XZ}, X$ . In the abduction step, we can compute the probabilities  $P(u_x, u_{xz} \mid x)$ . In the prediction step, query  $P(y \mid x)$  is computed based on Eq. 6.
Based on the procedure, we can also compute the query  $P(y_{x})$  (see Fig. 2c):
$$
P \left(y _ {x}\right) = \sum_ {u} \mathbb {1} \left(Y _ {x} (u) = y\right) P (u) = \sum_ {u} \mathbb {1} \left(Y \left(x, u _ {x z}, u _ {z}, u _ {y}\right) = y\right) P (u). \tag {7}
$$
where the first step follows from the definition of an interventional distribution, and the second step follows from noting that  $Y_{x}$  does not depend on  $u_{x}$ . In this case, the abduction step is void, since we are not considering any specific evidence  $E = e$ . The value of  $Y(x,u_{xz},u_z,u_y)$  can be computed from the submodel  $\mathcal{M}_x$ . Finally, using Eq. 7 we can perform the prediction step. We remark that
$$
\mathbb {1} (Y (x, u _ {x z}, u _ {z}, u _ {y}) = y) = \sum_ {u _ {x}} \mathbb {1} (Y (u _ {x}, u _ {x z}, u _ {z}, u _ {y}) = y) P (u _ {x} \mid x, u _ {x z}, u _ {z}, u _ {y}), \tag {8}
$$
by the law of total probability and noting that  $X$  is a deterministic function of  $u_{x}, u_{xz}$ . Thus,  $P(y_{x})$  also admits an alternative representation
$$
\begin{array}{l} P \left(y _ {x}\right) = \sum_ {u} \mathbb {1} \left(Y \left(u _ {x}, u _ {x z}, u _ {z}, u _ {y}\right) = y\right) P \left(u _ {x} \mid x, u _ {x z}, u _ {z}, u _ {y}\right) P \left(u _ {x z}, u _ {z}, u _ {y}\right) (9) \\ = \sum_ {u} \mathbb {1} (Y (u) = y) P \left(u _ {x} \mid x, u _ {x z}\right) P \left(u _ {x z}, u _ {z}, u _ {y}\right), (10) \\ \end{array}
$$
where Eq. 10 follows from using the independencies among  $U$  and  $X$  in the graph in Fig. 2b. We revisit the representation in Eq. 10 in Ex. 2.
3 Foundations of Decomposing Spurious Variations
After getting familiar with the abduction-action-prediction procedure, our next task is to introduce a new procedure that allows us to decompose spurious effects. First, we define the concept of a partially abducted submodel:
Definition 1 (Partially Abducted Submodel). Let  $U_{1}, U_{2} \subseteq U$  be a partition of the exogenous variables. Let the partially abducted (PA, for short) submodel with respect to the exogenous variables  $U_{1}$  and evidence  $E = e$  be defined as:
$$
\mathcal {M} ^ {U _ {1}, E = e} := \langle \mathcal {F}, P (u _ {1}) P (u _ {2} \mid u _ {1}, E) \rangle . \tag {11}
$$
In words, in the PA submodel, the typically obtained posterior distribution  $P(u \mid e)$  is replaced by the distribution  $P(u_2 \mid u_1, e)$ . Effectively, the exogenous variables  $U_1$  are not updated according to evidence. The main motivation for introducing the PA model is that spurious variations arise whenever we are comparing units of the population that are different, a realization dating back to Pearson in the 19th century [11]. To give a formal discussion on what became known as Pearson's shock, consider two sets of differing evidence  $E = e$  and  $E = e'$ . After performing the abduction step, the variations between posterior distributions  $P(u \mid e)$  and  $P(u \mid e')$  will be explained by all the exogenous variables that precede the evidence  $E$ . In a PA submodel, however, the posterior distribution  $P(u_1)P(u_2 \mid u_1, e)$  will differ from  $P(u_1)P(u_2 \mid u_1, e')$  only in variables that are in  $U_2$ , while the variables in  $U_1$  will induce no spurious variations. Note that if  $U_1 = U$ , then the PA submodel will introduce no spurious variations, a point to which we return in the sequel.
We now demonstrate how the definition of a PA submodel can be used to obtain partially abducted conditional probabilities:
Proposition 1 (PA Conditional Probabilities). Let  $P(Y = y \mid E = e^{U_1})$  denote the conditional probability of the event  $Y = y$  conditional on evidence  $E = e$ , defined as the probability of  $Y = y$  in the PA submodel  $\mathcal{M}^{U_1, E = e}$  (i.e., the exogenous variables  $U_1$  are not updated according to the evidence). Then, we have that:
$$
P (Y = y \mid E = e ^ {U _ {1}}) = \sum_ {u _ {1}} P \left(U _ {1} = u _ {1}\right) P (Y = y \mid E = e, U _ {1} = u _ {1}). \tag {12}
$$
3.1 Partial Abduction and Prediction
Based on the notion of a PA submodel, we can introduce the partial-abduction and prediction procedure:
Figure 3: Graphical representations of the SCM in Ex. 1.
Algorithm 2 (Partial Abduction and Prediction). Given an SCM  $\langle \mathcal{F}, P(u) \rangle$ , the conditional probability  $P(Y = y \mid E = e^{U_1})$  of an event  $Y = y$  upon observing the evidence  $e$ , in a world where variables  $U_1$  are unresponsive to evidence, can be evaluated using the following two steps:
(i) Partial Abduction - update  $P(u)$  by the evidence  $e$  to obtain  $P(u_{1})P(u_{2} \mid u_{1}, e)$ , where  $(u_{1}, u_{2})$  is a partition of the exogenous variables  $u$ ,
(ii) Prediction - use the model  $\langle \mathcal{F}, P(u_1)P(u_2 \mid u_1, e) \rangle$  to compute the probability of  $Y = y$ .
In the first step of the algorithm, we only perform partial abduction. The exogenous variables  $U_{2}$  are updated according to the available evidence  $E = e$ , while the variables  $U_{1}$  retain their original distribution  $P(u_{1})$  and remain unresponsive to evidence. This procedure allows us to consider queries in which only a subset of the exogenous variables respond to the available evidence. We next explain what kind of queries fall within this scope, beginning with an example:
Example 2 (Partial Abduction and Prediction). Consider the model in Eq. 3-5. We are interested in computing the query:
$$
\begin{array}{l} P (y \mid x ^ {U _ {x z}, U _ {z}}) = \sum_ {u} \mathbb {1} (Y (u) = y) P \left(u _ {x z}, u _ {z}\right) P \left(u _ {x}, u _ {y} \mid u _ {x z}, u _ {x}, x\right) (13) \\ = \sum_ {u} \mathbb {1} (Y (u) = y) P \left(u _ {x z}, u _ {z}\right) P \left(u _ {y}\right) P \left(u _ {x} \mid u _ {x z}, u _ {x}, x\right) (14) \\ = \sum_ {u} \mathbb {1} (Y (u) = y) P \left(u _ {x z}, u _ {z}, u _ {y}\right) P \left(u _ {x} \mid u _ {x z}, u _ {x}, x\right), (15) \\ \end{array}
$$
where the first step follows from Prop. 1, and the remaining steps from conditional independencies between the  $U$  variables and  $X$ . Crucially, the query yields the same expression as in Eq. 10 that we obtained for  $P(y_{x})$  in Ex. 1. Therefore, the conditional probability  $P(y \mid x^{U_{xz}}, U_z)$  in a world where  $U_{XZ}, U_Z$  are unresponsive to evidence is equal to the interventional probability  $P(y_x)$ .
As the example illustrates, we have managed to find another procedure that mimics the behavior of the interventional  $(do(X = x))$  operator in the given example. Interestingly, however, in this procedure, we have not made use of the submodel  $\mathcal{M}_x$  that was used in the abduction-action-prediction procedure. We next introduce an additional example that shows how the new procedure allows one to decompose spurious variations in causal models:
Example 3 (Spurious Decomposition). Consider an SCM compatible with the graphical representation in Fig. 3b (with exogenous variables  $U$  shown explicitly in red), and the corresponding Semi-Markovian causal diagram in Fig. 3a. We note that, based on the partial abduction-prediction procedure, the following two equalities hold:
$$
P (y \mid x) = P (y \mid x ^ {\emptyset}) \tag {16}
$$
$$
P \left(y _ {x}\right) = P \left(y \mid x ^ {U _ {x z _ {1}}}, U _ {x z _ {2}}\right), \tag {17}
$$
which shows that
$$
E x p - S E _ {x} (y) = P \left(y \mid x ^ {\emptyset}\right) - P \left(y \mid x ^ {U _ {x z _ {1}}, U _ {x z _ {2}}}\right). \tag {18}
$$
The experimental spurious effect can be written as a difference of conditional probabilities  $y \mid x$  in a world where all variables  $U$  are responsive to evidence vs. a world in which  $U_{XZ_1}, U_{XZ_2}$  are
(a)  $\operatorname{Exp - SE}_x^{\emptyset ,U_{1X}}(y)$
(b)  $\operatorname{Exp} - \operatorname{SE}_x^{U_1X, \{U_{1X}, U_{2X}\}}(y)$ .
Figure 4: Graphical representation of how the Exp-SE effect is decomposed in Ex. 3.
Procedure	SCM	Queries
Abduction-Prediction	⟨F, P(u | E)⟩	Layer 1
Action-Prediction	⟨Fx, P(u)⟩	Layer 2
Abduction-Action-Prediction	⟨Fx, P(u | E)⟩	Layers 1, 2, 3
Partial Abduction-Prediction	⟨F, P(u1)P(u2 | E)⟩	Layers 1, 2, 3
Table 1: Summary of the different procedures and the corresponding probabilistic causal models.
unresponsive to evidence. Furthermore, we can also consider a refinement that decomposes the effect
$$
E x p - S E _ {x} (y) = \underbrace {P \left(y \mid x ^ {\emptyset}\right) - P \left(y \mid x ^ {U _ {x z _ {1}}}\right)} _ {\text {v a r i a t i o n s o f} U _ {x z _ {1}}} + \underbrace {P \left(y \mid x ^ {U _ {x z _ {1}}}\right) - P \left(y \mid x ^ {U _ {x z _ {1}} , U _ {x z _ {2}}}\right)} _ {\text {v a r i a t i o n s o f} U _ {x z _ {2}}}, \tag {19}
$$
allowing for an additive, non-parametric decomposition of the experimental spurious effect.
The first term in Eq. 19, shown in Fig. 8a, encompasses spurious variations explained by the variable  $U_{XZ_1}$ . The second term, in Fig. 4b, encompasses spurious variations explained by  $U_{XZ_2}$ .
For an overview, in Tab. 1 we summarize the different inferential procedures discussed so far, indicating the structural causal models associated with them.
4 Non-parametric Spurious Decompositions
We now move on to deriving general decomposition results for the spurious effects. Before doing so, we first derive a new decomposition result for the TV measure, not yet appearing in the literature (due to space constraints, all proofs are given in Appendix A):
Proposition 2. Define the total variation (TV) measure as  $TV_{x_0,x_1}(y) = P(y \mid x_1) - P(y \mid x_0)$ , and the total effect TE as  $TE_{x_0,x_1}(y) = P(y_{x_1}) - P(y_{x_0})$ . The total variation measure can be decomposed as:
$$
T V _ {x _ {0}, x _ {1}} (y) = T E _ {x _ {0}, x _ {1}} (y) + \left(E x p - S E _ {x _ {1}} (y) - E x p - S E _ {x _ {0}} (y)\right). \tag {20}
$$
The above result clearly separates out the causal variations (measured by the TE) and the spurious variations (measured by Exp-SE terms) within the TV measure. The seminal result from [10] can be used to further decompose the TE measure. In the sequel, we show how the Exp-SE terms can be further decomposed, thereby reaching a full non-parametric decomposition of the TV measure.
4.1 Spurious Decompositions for the Markovian case
When using the definition of a PA submodel, the common variations between  $X,Y$  can be attributed to (or explained by) the unobserved confounders  $U_{1},\ldots ,U_{k}$ . In order to do so, we first define the notion of an experimental spurious effect for a set of latent variables:
Definition 2 (Spurious effects for Markovian models). Let  $\mathcal{M}$  be a Markovian model. Let  $Z_{1},\ldots ,Z_{k}$  be the confounders between variables  $X$  and  $Y$  sorted in any valid topological order, and denote the corresponding exogenous variables as  $U_{1},\ldots ,U_{k}$ , respectively. Let  $Z_{[i]} = \{Z_1,\dots,Z_i\}$  and
$Z_{-[i]} = \{Z_{i + 1},\dots ,Z_k\}$ $U_{[i]}$  and  $U_{-[i]}$  are defined analogously. Define the experimental spurious effect associated with variable  $U_{i + 1}$  as
$$
E x p - S E _ {x} ^ {U _ {[ i ]}, U _ {[ i + 1 ]}} (y) = P \left(y \mid x ^ {U _ {[ i ]}}\right) - P \left(y \mid x ^ {U _ {[ i + 1 ]}}\right). \tag {21}
$$
The intuition behind the quantity  $\mathrm{Exp - SE}_x^{U_{[i]},U_{[i + 1]}}(y)$  can be explained as follows. The quantity  $P(y\mid x^{U_{[i]}})$  captures all the variations in  $Y$  induced by observing that  $X = x$  apart from those explained by the latent variables  $U_{1},\ldots ,U_{i}$ , which are fixed a priori and not updated. Similarly, the quantity  $P(y\mid x^{U_{[i + 1]}})$  captures the variations in  $Y$  induced by observing that  $X = x$ , apart from those explained by  $U_{1},\ldots ,U_{i},U_{i + 1}$ . Therefore, taking the difference of the two quantities measures the variation in  $Y$  induced by observing that  $X = x$  that is explained by the latent variable  $U_{i + 1}$ .
Based on this definition, we can derive the first key non-parametric decomposition of the experimental spurious effect that allows the attribution of the spurious variations to the latent variables  $U_{i}$ :
Theorem 1 (Latent spurious decomposition for Markovian models). The experimental spurious effect  $\text{Exp-SE}_x(y)$  can be decomposed into latent variable-specific contributions as follows:
$$
E x p - S E _ {x} (y) = \sum_ {i = 0} ^ {k - 1} E x p - S E _ {x} ^ {U _ {[ i ]}, U _ {[ i + 1 ]}} (y) = \sum_ {i = 0} ^ {k - 1} P \left(y \mid x ^ {U _ {[ i ]}}\right) - P \left(y \mid x ^ {U _ {[ i + 1 ]}}\right). \tag {22}
$$