SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {P} (y | d o (t = t)) = \sum_ {\boldsymbol {s}} \left(\sum_ {t ^ {\prime}} \mathbb {P} (y | \boldsymbol {s}, t ^ {\prime}) \mathbb {P} \left(t ^ {\prime}\right)\right) \mathbb {P} (\boldsymbol {s} | t). \tag {10}
$$
Remark 1. Consider the case where  $y$  is a child of  $t$ . Then, the sufficient conditions in Theorems 3.1 and 3.2 do not hold because the conditional independence in (7) will not pass as  $y \in \mathbf{b}$  (which follows from Assumption 3). However, in this case, the causal query itself is not identifiable as there exists a bi-directed edge between  $y$  and  $t$  (from Assumption 2) (see Tian and Pearl [2002, Theorem 4]).
Algorithm 1: ATE estimation using subset search.
Input:  $n_r, t, y, \mathbf{b}, \mathcal{Z}, p_v$   
Output:  $\mathrm{ATE}_z, \mathrm{ATE}_s$   
Initialization:  $\mathrm{ATE}_z = 0, \mathrm{ATE}_s = 0, c_1 = 0$   
for  $r = 1, \dots, n_r$  do // Use a different train-test split in each run  
 $\mathrm{ATE}_z^r = 0, \mathrm{ATE}_s^r = 0, c_2 = 0; //$  These are used to average over different subsets that satisfy our conditions for a specific train-test split  
for  $\mathbf{z} \in \mathcal{Z}$  do // Perform an exhaustive search over  $\mathcal{Z}$   
if  $CI(\mathbf{b} \perp_p y|\mathbf{z}, t) > p_v$  then // Check for (7) where  $CI$  stands for conditional independence  
 $\begin{array}{l} \mathbf{z}^{(i)} = \mathbf{z} \setminus \mathbf{z}^{(o)}; \\ \text{if min}\{CI(\mathbf{z}^{(i)} \perp_p t), CI(\mathbf{z}^{(o)} \perp_p t|\mathbf{b}, \mathbf{z}^{(i)})\} > p_v\} \\ c_2 = c_2 + 1, \mathbf{s} = (\mathbf{b}, \mathbf{z}^{(i)}); \\ \mathrm{ATE}_z^{\mathrm{r}} = \mathrm{ATE}_z^{\mathrm{r}} + \frac{\sum_{j:t_j=1} \sum_{t'} \mathbb{E}[y|\mathbf{z}_j,t']\mathbb{P}(t')}{|\{j:t_j=1\}|} - \frac{\sum_{j:t_j=0} \sum_{t'} \mathbb{E}[y|\mathbf{z}_j,t']\mathbb{P}(t')}{|\{j:t_j=0\}|}; \\ \mathrm{ATE}_s^{\mathrm{r}} = \mathrm{ATE}_s^{\mathrm{r}} + \frac{\sum_{j:t_j=1} \sum_{t'} \mathbb{E}[y|\mathbf{s}_j,t']\mathbb{P}(t')}{|\{j:t_j=1\}|} - \frac{\sum_{j:t_j=0} \sum_{t'} \mathbb{E}[y|\mathbf{s}_j,t']\mathbb{P}(t')}{|\{j:t_j=0\}|}; \\ \end{array}$   
if  $c_2 > 0$  then  
 $\begin{array}{rl} & {\mathrm{ATE}_z = \mathrm{ATE}_z + \mathrm{ATE}_z^r / c_2, \mathrm{ATE}_s = \mathrm{ATE}_s + \mathrm{ATE}_s^r / c_2, c_1 = c_1 + 1;} \end{array}$
if  $c_{1} > 0$  then  $\mathrm{ATE}_z = \mathrm{ATE}_z / c_1, \mathrm{ATE}_s = \mathrm{ATE}_s / c_1; // c_1$  is used to average over different train-test splits.
else Failed to find  $\mathbf{z} = (\mathbf{z}^{(i)},\mathbf{z}^{(o)})$  satisfying (7) and (8);
3.2 Algorithm for ATE estimation
The ATE can be computed by taking the first moment version of (9) or (10). In Algorithm 1, we provide a systematic way to estimate the ATE using Theorem 3.2 by searching for a set  $\mathbf{z} \in \mathcal{Z} \triangleq \mathcal{V} \setminus \{t, \mathbf{b}, y\}$  such that (a) p-value of conditional independence in (7) passes a threshold  $p_v$  and (b) there exists a decomposition  $\mathbf{z} = (\mathbf{z}^{(i)}, \mathbf{z}^{(o)})$  such that p-values of conditional independencies in (8) pass the threshold  $p_v$ . Then, for every such  $\mathbf{z}$ , the algorithm computes the ATE using the first moment version of (9), and averages. The algorithm produces another estimate by using (10) instead of (9).
3.3 Relation to PAG-based algorithms
Now, we exhibit how our approach can recover the causal effect in certain scenarios where PAG-based methods are not suitable. PAGs depict ancestral relationships (not necessarily direct) with directed edges and ambiguity in orientations (if they exist across members of the equivalence class) by circle marks. Figure 4(c) shows the PAG consistent with SMCM in Figure 4(a). While we formally define PAGs in Appendix B, we refer interested readers to Triantafillou and Tsamardinos [2015]. The IDP algorithm of Jaber et al. [2019] is sound and complete for identifying causal effect from PAGs.
Consider SMCM in Figure 4(a) where our approach recovers the causal effect as  $(i)$  Assumptions 1 to (3),  $(ii)$  (7), and  $(iii)$  (8) hold (where  $(ii)$  and  $(iii)$  can be tested from observational data). However, the IDP algorithm fails to recover the effect from the PAG. To see this, consider SMCM in Figure 4(b) which is Markov equivalent to SMCM in Figure 4(a), i.e., the PAG in Figure 4(c) is also consistent with SMCM in Figure 4(b). Intuitively, when the strength of the edge between  $t$  and  $\mathbf{b}$  is very small but the strength of the edge between  $t$  and  $y$  is very high for both Figure 4(a) and Figure 4(b), causal effect in Figure 4(b) remains high while the causal effect in Figure 4(a) goes to zero.
We note that Assumptions 1 and 3, and (7) do not hold for the SMCM in Figure 4(b).
Figure 4: (a) An SMCM satisfying (7) and (8). (b) An SMCM obtained from (a) by modifying the edges between  $t$  and  $\mathbf{b}$  and between  $t$  and  $y$ . (c) The PAG corresponding to SMCM in (a) and (b).
Remark 2. Obtaining a PAG requires a large number of sequential conditional independence tests where the choice of the next test depends on the previous ones [Claassen et al., 2013]. The erroneous tests and orientation steps can potentially alter the structure of the PAG non-locally (see Strobl et al. [2016] for an example). This makes it difficult to control the false discovery rate for PAG based methods. Moreover, incorporating arbitrary side information into a PAG in a systematic way is still an open problem. In contrast, our approach does not rely on constructing a graphical object such as a PAG and the conditional independence tests could be carried out in parallel. Thus, our method can be viewed as a way to mitigate the issues associated with sequential testing by using structural side information. However, in scenarios where the structural side information is not available, we may have to resort to PAG-based methods.
We evaluate our approach empirically in 3 ways:  $(i)$  we demonstrate the applicability of our method on a class of random graphs,  $(ii)$  we assess the effectiveness of our method in estimating the ATE using finite samples, and  $(iii)$  we showcase the potential of our method for causal fairness analysis.
4.1 Applicability to a class of random graphs
In this experiment, we create a class of random SMCMs, sample 100 SMCMs from this class, and check if (7) and (8) hold by checking for corresponding d-separations in the SMCMs.
Creation of random SMCMs. Let  $p \triangleq |\mathcal{V}|$  denote the dimension of observed variables including  $x$ ,  $t$ , and  $y$ . Let  $\nu_{1}, \dots, \nu_{p}$  denote a causal ordering of these variables. Our random ensemble depends on two parameters: (i)  $d \leq p/2$  which is the expected in-degree of variables  $\nu_{2d}, \dots, \nu_{p}$  and (ii)  $q \leq p$  which controls the number of unobserved features. For  $1 \leq i < j \leq p$ , we add  $\nu_{i} \rightarrow \nu_{j}$  with probability 0.5 if  $j \leq 2d$  and with probability  $d/(j-1)$  if  $j > 2d$ . We note that this procedure is such that the expected in-degree of variables  $\nu_{2d}, \dots, \nu_{p}$  is same (and equal to  $d$ ) which is consistent with other recent work (e.g., Addanki et al. [2020]). Next, for  $1 \leq i < j \leq p$ , we add  $\nu_{i} \leftrightarrow \nu_{j}$  with probability  $q/p$  such that the expected number of unobserved features is  $q(p-1)/2$ . Then, we choose  $\nu_{p}$  as  $y$ , any variable that is ancestor of  $y$  but not its parent or grandparent as  $t$ , and all children of  $t$  as  $b$ . Finally, we add  $t \leftrightarrow y$  if missing.
Results. We compare the success rate of two approaches:  $(i)$  exhaustive search for  $\mathbf{z}$  satisfying (7) and (8) which is exponential in  $p$  and  $(ii)$  search for a  $\mathbf{z}$  of size at-most 5 satisfying (7) and (8) which is polynomial in  $p$ . We provide the number of successes of these approaches as a tuple in Table 1 for various  $p, d,$  and  $q$ . We see that the two approaches have comparable performances. We also compare with the IDP algorithm by providing it the true PAG. However, it gives 0 successes across various  $p, d,$  and  $q$ . We provide results for another random ensemble in Appendix G.
Table 1: Number of successes out of 100 random graphs for our methods shown as a tuple. The first method searches a  $\mathbf{z}$  exhaustively and the second method searches a  $\mathbf{z}$  with size at-most 5.
p=10	p=15
d=2	d=3	d=4	d=2	d=3	d=4
q=0.0	(43,43)	(20,20)	(21,21)	(27,26)	(9,9)	(4,2)
q=0.5	(23,23)	(16,16)	(7,7)	(18,17)	(4,3)	(0,0)
q=1.0	(6,6)	(4,4)	(5,5)	(9,9)	(10,9)	(0,0)
4.2 Estimating the ATE
In this experiment, we generate synthetic data using the 6 random SMCMs in Section 4.1 for  $p = 10$ ,  $d = 2$ , and  $q = 1.0$  where our approach was successful indicating existence of  $\mathbf{z} = (\mathbf{z}^{(i)}, \mathbf{z}^{(o)})$  such that the conditional independence statements in Theorem 3.2 hold. Then, we use Algorithm 1 to compute the error in estimating ATE and compare against a Baseline which uses the front-door adjustment in (5) with  $\mathbf{z} = \mathbf{b}$  given the side information in Assumption 3. We provide the results for the same experiment for specific choices of SMCMs including the one in Figure 2 in Appendix G. We also provide the 6 random SMCMs in Appendix G. We use RCoT hypothesis test [Strobl et al., 2019b] for conditional independence testing from finite data.
Data generation. We use the following procedure to generate data from every SMCM. We generate unobserved variables independently from Unif[1,2] which denotes the uniform distribution over [1,2]. For every observed variable  $\nu \in \mathcal{V}$ , let  $\pi(\nu) \triangleq (\pi^{(o)}(\nu), \pi^{(u)}(\nu)) \in \mathbb{R}^{d_v \times 1}$  denote the set of observed and unobserved parents of  $\nu$  stacked as a column vector. Then, we generate  $\nu \in \mathcal{V}$  as
$$
v = \mathbf {a} _ {v} ^ {\top} \pi (v) + 0. 1 \mathcal {N} (0, 1) \text {f o r} v \in \mathcal {V} \backslash \{t \} \quad \text {a n d} \quad t = \operatorname {B e r n o u l l i} (\operatorname {S i g m o i d} \left(\mathbf {a} _ {t} ^ {\top} \pi (t)\right)) \tag {11}
$$
where the coefficients  $\mathbf{a}_v\in \mathbb{R}^{d_v\times 1}$  with every entry sampled independently from Unif[1,2]. Also, to generate the true ATE, we intervene on the generation model in (11) by setting  $t = 0$  and  $t = 1$
Results. For every SMCM, we generate  $n$  samples of every observed variable in every run of the experiment. We average the ATE error over 10 such runs where the coefficients in (11) vary across runs. We report the average of these averages over the 6 SMCMs in Figure 5 for various  $n$ . While the error rates of Baseline and Algorithm 1 are of the similar order for  $n = 100$ , Algorithm 1 gives much lower errors for  $n = 1000$  and  $n = 10000$  showing the efficacy of our method.
Figure 5: Average ATE for Algorithm 1 and Baseline vs. number of samples.
4.3 Experiments with real-world fairness benchmarks
Next, we describe how our results enable finding front-door-like adjustment sets in fairness problems. In a typical fairness problem, the goal is to ensure that the outcome variable  $y$  does not unfairly depend on the sensitive/protected attribute, e.g., race or gender (which we define to be treatment variable  $t$ ), which would reflect undesirable biases. Often, the outcome is a descendant of the sensitive attribute (as per Assumption 1), and both outcome and sensitive attribute are confounded by unobserved variables (as per Assumption 2). Furthermore, there are a multitude of measured post-sensitive-attribute variables that can affect the outcome. This stands in contrast to the usual settings for causal effect estimation, where pre-treatment variables are primarily utilized.
Fairness problems are typically evaluated using various fairness metrics, such as causal fairness metrics or observational metrics. Causal metrics require knowing the underlying causal graph, which can be a challenge in practice. Observational criteria can be decomposed into three types of effects [Zhang and Bareinboim, 2018, Plecko and Bareinboim, 2022]: spurious effects, direct effects, and indirect effects (through descendants of sensitive attribute). In some scenarios, capturing the sum of direct and indirect effects is of interest, but even this requires knowing the causal graph.
Now, we demonstrate the application of our adjustment formulae in Theorem 3.2 to compute the sum of direct and indirect effects of the sensitive attribute on the outcome, while separating it from spurious effects. The sum of these effects is indeed the causal effect of sensitive attribute on the outcome. In other words, we consider the following fairness metric:  $\mathbb{E}[y|do(t = 1)] - \mathbb{E}[y|do(t = 0)]$ . We assume that all the children of the sensitive attribute are known, which may be easier to justify compared to the typical assumption in causal fairness literature of knowing the entire causal graph.
German Credit Dataset. The German Credit dataset [Hofmann, 1994] is used for credit risk analysis where the goal is to predict whether a loan applicant is a good or bad credit risk based on applicant's 20 demographic and socio-economic attributes. The binary credit risk is the outcome  $y$  and the applicant's age (binarized by thresholding at 25 [Kamiran and Calders, 2009]) is the sensitive attribute  $t$ . Further, the categorical attributes are one-hot encoded.
We apply Algorithm 1 with  $n_r = 100$  and  $p_v = 0.1$  where we search for a set  $\mathbf{z} = (\mathbf{z}^{(o)}, \mathbf{z}^{(i)})$  of size at most 3 under the following two distinct assumptions on the set of all children  $\mathbf{b}$  of  $t$ :
1. When considering  $\mathbf{b} = \{\# \text{ of people financially dependent on the applicant, applicant's savings, applicant's job}\}$ , Algorithm 1 results in  $\mathbf{z}^{(i)} = \{\text{purpose for which the credit was needed, indicator of whether the applicant was a foreign worker}\}$ ,  $\mathbf{z}^{(o)} = \{\text{installment plans from providers other than the credit-giving bank}\}$ ,  $\mathrm{ATE}_z = 0.0125 \pm 0.0011$ , and  $\mathrm{ATE}_s = 0.0105 \pm 0.0018$ .
2. When considering  $\mathbf{b} = \{\# \text{ of people financially dependent on the applicant, applicant's savings}\}$ , Algorithm 1 results in  $\mathbf{z}^{(i)} = \{\text{purpose for which the credit was needed, applicant's checking account status with the bank}\}$ ,  $\mathbf{z}^{(o)} = \{\text{installment plans from providers other than the credit-giving bank}\}$ ,  $\mathrm{ATE}_z = 0.0084 \pm 0.0008$ , and  $\mathrm{ATE}_s = -0.0046 \pm 0.0021$ .
Under the first assumption above, the causal effect using the adjustment formulae in (9) and (10) have same sign and are close in magnitude. However, under the second assumption, the effect flips sign. The results suggest that the second hypothesis regarding  $\mathbf{b}$  is incorrect, implying that applicant's job may indeed be a direct child of applicant's age, which aligns with intuition.
The dataset has only 1000 samples, which increases the possibility of detecting independencies in our criterion by chance, even with the size of  $\mathbf{z}$  constrained. To address this issue, we use 100 random bootstraps with a sample size equal to half of the training data and evaluate the p-value of our conditional independence criteria for all subsets returned by our algorithm. We select the subset  $\mathbf{z}$  with the highest median p-value (computed over the bootstraps) and use it in our adjustment formulae on a held out test set. To assess the conditional independencies associated with the selected  $\mathbf{z}$ , we plot a histogram of the corresponding p-values for all these bootstraps. If the conditional independencies hold, we expect the p-values to be spread out, which we observe in the histograms in Figure 6 for the first choice of  $\mathbf{b}$ . We report similar results for the second choice of  $\mathbf{b}$  in Appendix G.
Adult Dataset: We perform a similar analysis on the Adult dataset [Kohavi and Becker, 1996]. With suitable choices of  $\mathbf{b}$ , Algorithm 1 was unable to find a suitable  $\mathbf{z}$  satisfying  $\mathbf{b} \perp_{p} y|\mathbf{z}, t$ . This suggests that in this dataset, there may not be any non-child descendants of the sensitive attribute, which is required for our criterion to hold. More details can be found in Appendix G.
Figure 6: Histograms of p-values of the conditional independencies in (9) and (10) over 100 bootstrap runs for  $\mathbf{b} = \{\# \text{ of people financially dependent on the applicant, applicant's savings, applicant's job}\}$ .
In this work, we proposed sufficient conditions for causal effect estimation through a generalized front-door adjustment given structural side information irrespective of the number or complexity of the latent confounders. Our approach can identify causal effect in graphs where known Markov equivalence classes do not allow identification. However, our approach relies primarily on two assumptions: Assumption 2 and Assumption 3.
Assumption 2 plays a crucial role in Theorem 3.2 (see the discussion in Section E.3) as it requires the presence of an unobserved confounder between the treatment variable and the outcome variable. This assumption is necessary for the applicability of our approach. If Assumption 2 does not hold, it implies that there is a set that satisfies the back-door criterion, and existing methods for finding back-door adjustment sets [Entner et al., 2013, Cheng et al., 2020, Shah et al., 2022] can be utilized. This suggests that our results could be derived under the weaker condition that there is an unblockable back-door path between  $t$  and  $y$ . However, in many real-world scenarios, the presence of unobserved variables that potentially confound the treatment and the outcome is common, and we expect Assumption 2 to be true.
Assumption 3 is the requirement of knowing the entire set of children of the treatment variable within the causal graph. While this is strictly less demanding than specifying the entire causal graph, it may still present practical challenges in real-world scenarios. For instance, in large-scale observational studies or domains with numerous variables, exhaustively identifying all the children may be computationally demanding. It remains unclear whether one can estimate the causal effect using front-door-like adjustment with even less side information, e.g., knowing only one child of the treatment variable or knowing any subset of children of the treatment variable. An important future direction could be to approximate the causal effect when only the children corresponding to weak edges are unknown. Such variations around our condition are promising directions for future work. However, until then, one could seek input from domain experts. These experts possess valuable knowledge and insights about the specific domain under study, which can aid in identifying all the relevant variables that serve as children of the treatment.
The time complexity of Algorithm 1 is exponential due to its search over all possible subsets of observed variables (except  $t, \mathbf{b}, y$ ). While this is inherent in the general case, recent work by Shah et al. [2022] proposed a scalable approximation for conditional independence testing using continuous optimization by exploiting the principle of invariant risk minimization, specifically for back-door adjustment without the need for the causal graph. However, extending this approach to multiple conditional independence tests, as required in (7) and (8), remains an open challenge. Therefore, exploring the development of continuous optimization-based methods for scalability of front-door adjustment in the absence of the causal graph is crucial. Further, Algorithm 1 could potentially be augmented with ideas from double machine learning and inverse variance weighting for bias and variance reduction.
Lastly, it is important to note that Theorem 3.2 provides only sufficient conditions for causal effect estimation. This means that there may be cases where front-door-like adjustment is possible, but the conditions stated in Theorem 3.2 do not hold. Specifically, there could be scenarios, such as when the outcome variable is a child of the children of the treatment variable, i.e.,  $y$  is a child of  $\mathbf{b}$ , where conditions (7) and (8) are not satisfied.
We thank the anonymous reviewers for their comments and suggestions. Murat Kocaoglu acknowledges the support of NSF Grant CAREER 2239375.
J. Zhang. On the completeness of orientation rules for causal discovery in the presence of latent confounders and selection bias. Artificial Intelligence, 172(16-17):1873-1896, 2008.
J. Zhang and E. Bareinboim. Fairness in decision-making—the causal explanation formula. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
A Broader impact
In this work, we propose an algorithm for estimating causal effects from observational studies without relying on expert knowledge of the causal model. Our approach is particularly valuable in scenarios where conducting randomized control trials (RCTs) is challenging or unethical, such as in healthcare settings where consensus treatment protocols are often determined based on observational data. While our algorithm shows promise in providing accurate causal effect estimates, it is crucial to address the potential negative impact of incorrect results that may arise from our work.
One significant concern is the possibility of miscalculating the treatment effect due to limitations in testing power at finite sample sizes or the misidentification of certain features as direct children of the treatment variable. This introduces the risk of inaccurate estimations, which could have detrimental consequences when making decisions or establishing treatment protocols based on the conclusions derived from our algorithm. It is essential to approach the interpretation of our algorithm's results with caution and subject them to critical evaluation. It is worth noting that the potential for incorrect results is not unique to our algorithm but is inherent in most observational studies and effect estimation algorithms. Acknowledging these potential negative impacts emphasizes the need for further research to improve the reliability and accuracy of causal effect estimation in observational studies.
B Preliminaries about ancestral graphs
In this section, we provide the definition of partial ancestral graphs (PAGs). PAGs are defined using maximal ancestral graphs (MAGs). Below, we define MAGs and PAGs based on their construction from directed acyclic graphs (DAGs).
A MAG can be obtained from a DAG as follows: if two observed nodes  $x_{1}$  and  $x_{2}$  cannot be d-separated conditioned on any subset of observed variables, then  $(i)x_{1}\longrightarrow x_{2}$  is added in the MAG if  $x_{1}$  is an ancestor of  $x_{2}$  in the DAG,  $(ii)x_{2}\longrightarrow x_{1}$  is added in the MAG if  $x_{2}$  is an ancestor of  $x_{1}$  in the DAG, and  $(iii)x_{1}\leftrightarrow x_{2}$  is added in the MAG if  $x_{1}$  and  $x_{2}$  are not ancestrally related in the DAG.  $(iv)$  After the above three operations, if both  $x_{1}\leftrightarrow x_{2}$  and  $x_{1}\longrightarrow x_{2}$  are present, we retain only the directed edge. In general, a MAG represents a collection of DAGs that share the same set of observed variables and exhibit the same independence and ancestral relations among these observed variables. It is possible for different MAGs to be Markov equivalent, meaning they represent the exact same independence model.
A PAG shares the same adjacencies as any MAG in the observational equivalence class of MAGs. An end of an edge in the PAG is marked with an arrow  $(> \text{or} <)$  if the edge appears with the same arrow in all MAGs in the equivalence class. An end of an edge in the PAG is marked with a circle  $(o)$  if the edge appears as an arrow  $(> \text{or} <)$  and a tail  $(-)$  in two different MAGs in the equivalence class.
C Rules of do-calculus
In this section, we provide the do-calculus rules of Pearl [1995] that are used to prove our main results in the following sections. We build upon the definition of semi-Markovian causal model from Section 2.