SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hengrui Cai
hengrc1@uci.edu
Yixin Wang
yixinw@umich.edu
Rui Song
songray@gmail.com
The causal revolution has stimulated interest in understanding complex relationships in various fields. Most of the existing methods aim to discover causal relationships among all variables within a complex large-scale graph. However, in practice, only a small subset of variables in the graph are relevant to the outcomes of interest. Consequently, causal estimation with the full causal graph—particularly given limited data—could lead to numerous falsely discovered, spurious variables that exhibit high correlation with, but exert no causal impact on, the target outcome. In this paper, we propose learning a class of necessary and sufficient causal graphs (NSCG) that exclusively comprises causally relevant variables for an outcome of interest, which we term causal features. The key idea is to employ probabilities of causation to systematically evaluate the importance of features in the causal graph, allowing us to identify a subgraph relevant to the outcome of interest. To learn NSCG from data, we develop a necessary and sufficient causal structural learning (NSCSL) algorithm, by establishing theoretical properties and relationships between probabilities of causation and natural causal effects of features. Across empirical studies of simulated and real data, we demonstrate that NSCSL outperforms existing algorithms and can reveal crucial yeast genes for target heritable traits of interest.
Causal discovery has gained significant attention in recent years for disentangling complex causal relationships in various fields. Building upon the causal graphical model [see e.g., 23], many causal structural learning algorithms have been developed [see e.g., 35; 7; 34; 14; 4; 27; 46; 44; 48; 5] to infer the causal knowledge (e.g., causal graphs) from observed data. These algorithms are based on the assumption of causal sufficiency (the absence of unmeasured confounders). In real-world applications, to satisfy such an assumption, we strive to learn large-scale causal graphs [see e.g., 20; 6; 38; 21], in the hope of sufficiently describing how an outcome of interest depends on its relevant variables.
In addition to sufficiency, it is also crucial to account for the concept of necessity by excluding redundant variables in explaining the outcome of interest. Failure to do so can result in the inclusion of spurious variables in the learned causal graphs, which are highly correlated but have no causal impact on the outcome. These variables can impede causal estimation with limited data and lead to falsely discovered spurious relationships, leading to poor generalization performance for downstream prediction [31]. For example, it might be observed that men aged 30 to 40 who buy diapers are also likely to buy beer. However, beer purchase is a spurious feature for diaper purchases: their correlation is not necessarily causal, as both purchases might be confounded by a shared cause, such as new
Figure 1: Left: Illustration of the causal relationship between the customer being a new father or not, beer purchasing, and diaper purchasing, where solid lines represent the true model, and the dashed line corresponds to the spurious correlation between beer purchasing and diaper purchasing. Right: Relationship between various causal structures. The nodes  $A$ ,  $B$ , and  $C$  belong to the necessary and sufficient causal graph for the desired result  $Y$  and are represented within the solid green square. Among them, nodes  $B$  and  $C$  are members of the Markov blanket of  $Y$ , enclosed by the blue square. The node  $S$  is the spurious variable for  $Y$ , while the nodes  $N$  and  $M$  are not related to the target.
fathers buying diapers for childcare while also buying beer to alleviate stress. Therefore, simply increasing the availability of diapers or beer will not causally improve the demand for the other (see also Fig. 1(left)).
Furthermore, the number of variables causally relevant to the outcome of interest is often considerably smaller than the number of variables included in estimating a causal graph (see Fig. 1(right)). For example, while an individual's genome may encompass 4 to 5 million single nucleotide polymorphisms (SNPs), only a limited number of non-spurious genes or proteins are found to systematically regulate the expression of the phenotype of interest [e.g., 6]. Similarly, in natural language processing tasks, excluding spurious embeddings such as writing style and dialect can enhance model accuracy and downstream prediction performance [e.g., 10]. Thus, a more parsimonious causal graph is required to unveil the necessary and sufficient causal dependencies.
In this work, we focus on learning necessary and sufficient causal graphs (NSCG) that only contain causally relevant variables (which we term causal features) for an outcome of interest, offering a compact representation of causal graphs for a target outcome. Our contributions are three-fold.
- We propose the notion of NSCG (see an illustration inside the green solid square in the right panel of Fig. 1). The key idea is to leverage the marginal and conditional probabilities of causation (POC) to systematically characterize the importance of variables (a.k.a. features).
- We establish theoretical properties and relationships between POC and the natural causal effects of features, and derive the conditions under which they are equivalent, with lower bounds provided for identification. The natural causal effects of features have explicit forms under parametric models such as the linear structural equation model, enabling convenient estimation of the POC from observed data.
- To select necessary and sufficient features in causal graphs, we propose a necessary and sufficient causal structural learning algorithm (NSCSL) to learn an NSCG containing all necessary and sufficient causes without unnecessary spurious features. This enables feature selection for causal discovery.
The proposed method provides concise explanations of causal relationships with high-dimensional data (i.e., with a large number of variables). Empirical studies in simulated datasets show that NSCSL outperforms existing algorithms in distilling relevant subgraphs for outcomes of interest; NSCSL can also identify important quantitative trait loci for the yeast and the causal protein signaling network for single cell data, as demonstrated in real data analyses.
The literature on causal structural learning can be broadly classified into three classes. The first class of methods focuses on using local conditional independence tests to identify the causal skeleton and determine the direction of the edges, such as the PC algorithm [35; 14; 36]. The second class of methods uses functional causal models with additional assumptions about the data distribution, including ICA-LiNGAM [34] and the causal additive model (CAM) [4]. The third class, the score-based methods, includes greedy equivalence search (GES) [7; 27; 11] and acyclicity optimization methods [46]. Refer to [44; 48; 17; 5; 47; 41] for additional cutting-edge causal structural learning methods. Yet, these works do not consider the necessity of the variables incorporated in the causal graph, i.e., whether the variables are causally relevant to the outcome. Consequently, such algorithms can often produce a redundant or potentially misleading graph, as depicted in Fig. 1 (right).
Our work also links to feature selections [see an overview in 16]. Despite the extensive literature, only a few studies have examined variable selection in causal graphs. One notable exception is Aliferis et al. [1], which uses the concept of the Markov blanket to construct a local causal graph for the target variable of interest. In this context, a Markov blanket of a variable  $Y$  is the minimal variable subset conditioned upon which all other variables become probabilistically independent of  $Y$ . Consequently, their algorithm uncovers only direct parents or children in the identified causal graph (such as the blue dotted square in the right panel of Fig. 1) and thereby overlooks the ancestors that contain atavistic information and indirectly influence the outcome. Recent works [18; 19] consider a minimal sufficient action set in bandits. Yet, these methods [also see 12; 13] rely on a true or known graph. We instead propose to simultaneously learn the causal graph and select the causal features.
Lastly, our work is connected to the body of research on probability of causation [e.g., 22; 39; 45], which delineates the necessity and sufficiency of features for the outcome of interest. Recently, Wang & Jordan [42] introduced this concept into representation learning, formulating the non-spuriousness and efficiency of representations by generalizing the probabilities of causation to accommodate low-dimensional representations of high-dimensional data. However, these works primarily concentrate on the identification of probabilities of causation, assuming that the causal graph among the variables under consideration is known with causally independent features. We address this gap in our work by incorporating the notion of probabilities of causation into learning complex causal graphs.
Graph terminology. Consider a graph  $\mathcal{G} = (\pmb{X},\pmb{D}_{\pmb{X}})$  with a node set  $\pmb{X}$  and an edge set  $D_{X}$  that encompasses all edges in  $\mathcal{G}$  for nodes  $\pmb{X}$ . A node  $X_{i}$  is said to be a parent of  $X_{j}$  if there is a directed edge from  $X_{i}$  to  $X_{j}$ , i.e.,  $X_{i}$  is a direct cause of  $X_{j}$ . A node  $X_{k}$  is said to be an ancestor of  $X_{j}$  if there is a directed path from  $X_{k}$  to  $X_{j}$  regulated by at least one additional node  $X_{i}$  for  $i\neq k$  and  $i\neq j$ , i.e.,  $X_{k}$  is an indirect cause of  $X_{j}$ . Let the set of all parents/ancestors of node  $X_{j}$  in  $\mathcal{G}$  as  $\mathrm{PA}_{X_j}(\mathcal{G})$ . A directed graph  $\mathcal{G}$  that does not contain directed cycles is called a directed acyclic graph (DAG). The structural causal model (SCM) characterizes the causal relationship among  $|\pmb {X}| = d$  nodes via a DAG  $\mathcal{G}$  and noises  $\pmb {e}_{\pmb{X}} = [e_{X_1},\dots ,e_{X_d}]^\top$  such that  $X_{i}\coloneqq h_{i}\{\mathrm{PA}_{X_{i}}(\mathcal{G}),e_{X_{i}}\}$  for some unknown  $h_i$  and  $i = 1,\dots ,d$
Notations and assumptions. Denote  $\mathcal{O} = (Z, Y)$  as a collection of nodes that contains features  $Z = [Z_1, \dots, Z_p]^\top \in \mathcal{Z} \subset \mathbb{R}^p$  and a discrete outcome of interest as  $Y \in \mathcal{L} = \{y_1, \dots, y_l\}$  for  $l$  different values. Here, the features can be intervened, such as treatment and mediators. Let  $Y(\mathbf{Z} = \mathbf{z})$  be the potential value of  $Y$  that would be observed after setting variable  $\mathbf{Z}$  as  $\mathbf{z}$ . This is equivalent to the value of  $Y$  by imposing a 'do-operator' of  $do(\mathbf{Z} = \mathbf{z})$  as in Pearl et al. [23]. Similarly, one can define the potential outcome,  $Y(Z_i = z_i)$ , by setting an individual variable  $Z_i$  as  $z_i$ , while keeping the rest of the model unchanged. Suppose there exists an SCM that characterizes the causal relationship among  $O$ , with its DAG as  $\mathcal{G}_O$ . A notation and abbreviation table is provided in App. A. Following the causal inference literature [see e.g., 29; 22; 23; 42], we assume:
(A1). Consistency:  $Z = z \leftrightarrow Y(Z = z) = Y, \forall z \in \mathcal{Z}$ .
(A2). Ignorability: (i)  $Y(\mathbf{Z} = \mathbf{z}) \perp \mathbf{Z}, \forall \mathbf{z} \in \mathcal{Z}$ ; (ii)  $Y(Z_{i} = z_{i}) \perp Z_{i} | \mathrm{PA}_{Z_{i} \cup Y}(\mathcal{G}_{\mathbf{O}}), \forall z_{i} \in \mathcal{Z}_{i}$ .
Here, (A1) implies that the outcome observed for each unit under study with features as  $z$  is identical to the outcome we would have observed had that unit been set with features  $Z = z$ . In addition, since we include as many confounders as possible, the ignorability assumption in (A2), also known as the no unmeasured confounderness assumption, is satisfied.
3 Necessary and Sufficient Causal Graphs
We care about a subset or a function of  $Z$ , denoted as  $X = [X_1, \dots, X_d]^\top$  (of  $d$  dimension with possibly  $d \ll p$ ), which indeed captures the causal relationship between  $Z$  and  $Y$ . To be specific, let an SCM for causal nodes  $V = (X, Y)$  with its DAG as  $\mathcal{G}_V = (V, D_V)$  and  $e_V$  as a  $d + 1$  dimensional independent noise, to characterize the causal relationship between  $X$  and  $Y$ . Let  $\mathbb{P}_{\mathcal{G}}$  be the mass/density function for an SCM with its DAG  $\mathcal{G}$ . Following the causal (or disentangled) factorization in the causal graphical model [23], we define the sufficient causal graph as follows.
Definition 3.1. (Sufficient Graph) The graph  $\mathcal{G}_V$  is a sufficient causal graph to capture the causal relationship among  $Z$  and  $Y$  with  $X\subset Z$  or  $X = f(Z)$  (where  $f$  is within a countable or Vapnik-Chervonenkis (VC) class) if  $\mathbb{P}_{\mathcal{G}_V}\{Y|\mathrm{PA}_Y(\mathcal{G}_V)\} \prod_{X_i\in \mathrm{PA}_Y(\mathcal{G}_V)}\mathbb{P}_{\mathcal{G}_V}\{X_i|\mathrm{PA}_{X_i}(\mathcal{G}_V)\} = \mathbb{P}_{\mathcal{G}_O}\{Y|\mathrm{PA}_Y(\mathcal{G}_O)\} \prod_{Z_i\in \mathrm{PA}_Y(\mathcal{G}_O)}\mathbb{P}_{\mathcal{G}_O}\{Z_i|\mathrm{PA}_{Z_i}(\mathcal{G}_O)\} .$
Here, Def. 3.1 refers to a sub-structure  $\mathcal{G}_V$  (from the whole graph  $\mathcal{G}_O$ ) containing all directed edges or paths towards  $Y$ , making it sufficient to describe how  $Y$  depends on all its ancestors. Then, the causal graph  $\mathcal{G}_V$  is said to be necessary and sufficient if it satisfies the following definition.
Definition 3.2. (Necessary and Sufficient Graph) Suppose  $\mathcal{G}_V$  satisfies Def. 3.1, then  $\mathcal{G}_V$  is a necessary and sufficient causal graph to capture the causal relationship among  $Z$  and  $Y$  if for any true subset  $W$  of  $X$ , i.e.,  $W \subset X$  or  $W = g(X)$  (where  $g$  is within a countable or VC class), with  $U = (W, Y)$ , we have  $\mathbb{P}_{\mathcal{G}_V}\{Y|\mathrm{PA}_Y(\mathcal{G}_V)\} \prod_{X_i \in \mathrm{PA}_Y(\mathcal{G}_V)} \mathbb{P}_{\mathcal{G}_V}\{X_i|\mathrm{PA}_{X_i}(\mathcal{G}_V)\} \neq \mathbb{P}_{\mathcal{G}_U}\{Y|\mathrm{PA}_Y(\mathcal{G}_U)\} \prod_{W_i \in \mathrm{PA}_Y(\mathcal{G}_U)} \mathbb{P}_{\mathcal{G}_U}\{W_i|\mathrm{PA}_{W_i}(\mathcal{G}_U)\}$ , where  $\mathcal{G}_U$  is the causal graph for  $U$ .
Therefore, by Def. 3.2, we can further identify the minimal sub-structure  $\mathcal{G}_V$  which includes only all directed edges or paths leading to  $Y$ . The goal is to learn such a necessary and sufficient causal graph (NSCG)  $\mathcal{G}_V$  from the observed data denoted as  $\{\pmb{o}^{(j)} = (\pmb{z}^{(j)},y^{(j)})\}_{1\leq j\leq n}$  with sample size  $n$ , by identifying the latent causal features  $\pmb{X}$ . Denote the resulting estimated graph as  $\widehat{\mathcal{G}}_V$ .
4 Probability of Causation and Causal Effects
Obtaining an NSCG  $\mathcal{G}_V$  directly based on Def. 3.2 poses several challenges, as the latent causal features  $X$  driving the causal graph remain unknown. A naive approach is to search all different combinations of  $Z$  for a candidate of  $X$  such that Def. 3.2 holds, which yields a complexity of  $\mathcal{O}(p^p)$ . This motivates us to assess the necessity and sufficiency of features in determining the outcome by introducing the concepts of probabilities of causation and causal effects, which will be elaborated on and interconnected in this section.
4.1 Probabilities of Causation and Lower Bounds
The probabilities of a feature being necessary and sufficient, known as the probability of causation (POC), have been proposed and studied [see 22; 39; 42]. Specifically, the probability of necessity and sufficiency (PNS) of feature  $Z$  is first defined in Tian & Pearl [39] as follows.
Definition 4.1. PNS in Tian & Pearl [39] with a univariate binary feature  $Z$ :
$$
P N S \equiv \mathbb {P} \{Y (Z \neq z) \neq y, Y (Z = z) = y \} \underset {b y (A 1)} {=} \mathbb {P} (Z = z, Y = y) \cdot P N + \mathbb {P} (Z \neq z, Y \neq y) \cdot P S,
$$
where the probability of necessity (PN) is  $PN = \mathbb{P}\{Y(Z \neq z) \neq y|Z = z, Y = y\}$ , and the probability of sufficiency (PS) is  $PS = \mathbb{P}\{Y(Z = z) = y|Z \neq z, Y \neq y\}$ .
The second equation in Def. 4.1 holds under (A1) [see details in 22; 39]. The PN score reflects the necessity of  $Z$  by evaluating the probability of the outcome becoming worse if revising the features given the good outcome observed. Similarly, the PS score indicates the sufficiency of  $Z$  by evaluating the probability of the outcome becoming better if changing the features given the bad outcome observed. Therefore, the PNS score shows the causal importance of the features by combining necessary and sufficient properties. The above definition can be generalized to multivariate cases for nonbinary features [see e.g., 42] to quantify the POC of an individual feature  $Z_{i}$ . Let  $Z_{-i} \equiv Z \setminus Z_{i}$  be the set of complementary variables of  $Z_{i}$ . In the following, we consider two different POCs for  $Z_{i}$  by extending the work of Wang & Jordan [42].
Definition 4.2. Marginal POC (M-POC) for  $Z_{k}$ :
$$
\mathrm {M - P O C} _ {i} (y) \equiv \mathbb {P} \{Y (Z _ {i} \neq z _ {i}) \neq y, Y (Z _ {i} = z _ {i}) = y \}.
$$
Definition 4.3. Conditional POC (C-POC) for  $Z_{i}$ :
$$
\mathrm {C} - \operatorname {P O C} _ {i} (y) \equiv \mathbb {P} \left\{Y \left(Z _ {i} \neq z _ {i}, Z _ {- i} = z _ {- i}\right) \neq y, Y \left(Z _ {i} = z _ {i}, Z _ {- i} = z _ {- i}\right) = y \right\}.
$$
We introduce the marginal POC (M-POC) as a novel quantity in the literature to summarize the overall causal importance of an individual feature in determining the outcome's value. The conditional POC (C-POC) in Def. 4.3 corresponds to the conditional PNS in Wang & Jordan [42], which quantifies the likelihood of an individual feature being a direct and significant cause of the outcome while holding other features constant. As per Section 9.2.3 in Pearl et al. [22], the PNS in Def. 4.1 is not estimable unless additional conditions (monotonicity) are specified. To alleviate such a condition for identifying Defs. 4.2 and 4.3, we derive the lower bounds for the proposed POCs as follows.
Theorem 4.4. (Lower Bound of Probabilities of Causation) Suppose (A1) and (A2) hold. Then
$$
M - P O C _ {i} (y) \geq \mathbb {P} (Y = y | Z _ {i} = z _ {i}) - \mathbb {P} (Y = y | Z _ {i} \neq z _ {i}),
$$
$$
C - P O C _ {i} (y) \geq \mathbb {P} (Y = y | Z _ {i} = z _ {i}, \boldsymbol {Z} _ {- i} = \boldsymbol {z} _ {- i}) - \mathbb {P} (Y = y | Z _ {i} \neq z _ {i}, \boldsymbol {Z} _ {- i} = \boldsymbol {z} _ {- i}).
$$
Table 1: Causal effects from the customer being a new father or not  $(X_{F})$  and diaper purchasing  $(X_{D})$  on beer purchasing  $(X_{B})$ , where  $\omega_{1}, \omega_{2} \in (0,1)$  with  $\omega_{1} + \omega_{2} = 1$  and  $c \to 1$ , in the estimated causal graph  $(X_{F} \xrightarrow{\omega_{1}} X_{B}$ , and  $X_{F} \xrightarrow{c} X_{D} \xrightarrow{\omega_{2}} X_{B})$ .
Variable Name	Direct Effect	Total Effect
New Father (XF)	ω1	ω1 + ω2 → 1	(> ω2)
Diaper (XD)	ω2	ω2
The proofs of Thm. 4.4 are in App. D. The lower bound equality holds when an additional monotonicity condition is imposed, with details in App. D. The results in Thm. 4.4 allow us to estimate the lower bound of POC from observed data by learning the conditional probability of  $Y$  given various combinations of confounders. This, in turn, aids in evaluating the significance of features, with details provided in App. B. Yet, estimating these conditional probabilities of  $Y$  based on high-dimensional features is very challenging [e.g., 32; 42], which motivates us to consider the corresponding expected mean outcome given different combinations of the confounders.
4.2 Causal Effects and Connection to POCs
To connect the proposed POCs and facilitate the empirical estimation, we introduce the natural total effect (TE) and natural direct effect (DE) for  $Z_{i}$  by extending definitions in Pearl et al. [22].
Definition 4.5. Natural Causal Effects for  $Z_{i}$ :
$$
T E _ {i} = \mathbb {E} \{Y (Z _ {i} = z _ {i} + 1) \} - \mathbb {E} \{Y (Z _ {i} = z _ {i}) \},
$$
$$
D E _ {i} = \mathbb {E} \{Y (Z _ {i} = z _ {i} + 1, Z _ {- i} = z _ {- i} ^ {(z _ {i})}) \} - \mathbb {E} \{Y (Z _ {i} = z _ {i}) \},
$$
where  $\pmb{z}_{-i}^{(z_i)}$  is the value of  $Z_{-i}$  if setting  $do(Z_i = z_i)$ .
The natural total effect  $(TE_{i})$  can be understood as the marginal change in the outcome when increasing  $Z_{i}$  by one unit. Similarly, the natural direct effect  $(DE_{i})$  represents the conditional change in the outcome when  $Z_{i}$  is increased by one unit, with all other features held constant. Indeed, the natural total and direct causal effects delineated in Pearl et al. [22] emerge as particular instances of Def. 4.5 when examining a solitary treatment subject to intervention. By comparing Def. 4.5 with Defs. 4.2 and 4.3, it is natural to establish the relationship between POCs and causal effects below.
Theorem 4.6. (Relation between POCs and Causal Effects) Define  $\delta_M(z_i) \equiv \mathbb{E}\{Y|Z_i = z_i\} - \mathbb{E}\{Y|Z_i \neq z_i\}$  and  $\delta_C(z_i) \equiv \mathbb{E}\{Y|Z_i = z_i, Z_{-i} = z_{-i}\} - \mathbb{E}\{Y|Z_i \neq z_i, Z_{-i} = z_{-i}\}$ . Suppose (A1)-(A2) hold, then
$$
\sum_ {y \in \mathcal {L}} y M - P O C _ {i} (y) \geq \delta_ {M} (z _ {i}), \quad \sum_ {y \in \mathcal {L}} y C - P O C _ {i} (y) \geq \delta_ {C} (z _ {i}),
$$
if  $Y$  is nonnegative. Further, if  $Z_{i}$  is binary, we have
$$
\min  \{\sum_ {y \in \mathcal {L}} y M - P O C _ {i} (y), | T E _ {i} | \} \geq \delta_ {M} (z _ {i}), \quad \min  \{\sum_ {y \in \mathcal {L}} y C - P O C _ {i} (y), | D E _ {i} | \} \geq \delta_ {C} (z _ {i}).
$$
The proofs of Thm. 4.6 can be found in App. D, with the lower bound equality holds when an additional monotonicity condition is imposed. We can summarize the findings of Thms. 4.4 and 4.6 in two key aspects. First, the marginal and conditional POCs assess the likelihood of a feature being spurious, while the absolute values of natural causal effects quantify the size of such a spurious effect based on the magnitude of the outcome of interest. Both are lower bounded by the same quantity, that is, the differences in expectations based on the corresponding POC, given non-negative outcomes and binary features. With appropriate data processing, we can transform the outcome to be nonnegative, and thus causal effects become a suitable substitute for POCs. Second, these two approaches exhibit consistency under the monotonicity condition. The natural causal effects of features have explicit forms under parametric models (see details in § 5.1), such as the linear structural equation model, enabling convenient estimation of the necessity and sufficiency of features from observed data. This section concludes with a toy example illustrating the identification of spurious features through the proposed causal effects and the distinction between total and direct causal effects.
Example 4.7. (Recall: Beer and Diaper) Given the causal graph in Fig. 1(left), consider a linear SCM for the customer being a new father or not  $(X_{F})$ , diaper purchasing  $(X_{D})$ , and beer purchasing  $(X_{B})$ :  $X_{D} = X_{F} + e_{D}$  and  $X_{B} = X_{F} + e_{B}$ , where  $e_{D}$  and  $e_{B}$  are independent mean zero noises.