SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note that, by restricting each layer such that  $\nabla_{\mathbf{z}^l}\tau (\mathbf{z}^l)\equiv \mathbf{I} + \mathbf{A}$ , there cannot exist shortcuts at the optima. For example, in Fig. 4, the (indirect) information of  $\mathrm{u}_1$  to generate  $\mathbf{x}_3$  by first generating
3The order has been reversed w.r.t. the causal NF definition from Eq. 3.
$\mathbf{x}_2$  needs to go through the middle nodes in Fig. 4b. However, as shown by Sánchez-Martin et al. [34], we need at least  $L = \mathrm{diam}(\mathbf{A})$  layers in our causal NF to avoid shortcuts and thus differentiate between the different direct and indirect causal paths connecting a pair of observed variables.
In contrast, if we only know the causal ordering, then the causal NF will need to rule out the spurious correlations by learning during training the necessary zeroes to fulfil causal consistency (Cor. 2), i.e., such that  $\nabla_{\mathbf{x}}T_{\boldsymbol{\theta}}(\mathbf{x})\equiv \mathbf{I} + \mathbf{A}$  and  $\nabla_{\mathbf{u}}T_{\boldsymbol{\theta}}^{-1}(\mathbf{u})\equiv \mathbf{I} + \sum_{n = 1}^{\mathrm{diam}(A)}\mathbf{A}^{n}$ .
Abductive model Reminiscent to the abduction step [30], another natural choice is to model the inverse equations of the SCM as in Fig. 4d, hence building a causal NF from  $\mathbf{x}$  to  $\mathbf{u}$ . Under a known causal graph, we can again use extra masking in the causal NF to force each layer  $l$  to be structurally equivalent to  $I - A$ , such that
$$
\mathbf {z} _ {i} ^ {l} = \tau_ {i} \left(\mathbf {z} _ {i} ^ {l - 1}; \mathbf {h} _ {i} ^ {l}\right), \quad \text {w h e r e} \quad \mathbf {h} _ {i} ^ {l} = c _ {i} \left(\mathbf {z} _ {\mathrm {p a} _ {i}} ^ {l - 1}\right). \tag {6}
$$
Remarkably, this architecture is capable of capturing all indirect dependencies of  $\mathbf{u}$  on  $\mathbf{x}$ , even with a single layer. This is a result of the autoregressive nature of the ANFs used here to build causal NFs, as they compute the inverse sequentially. In the example of Fig. 4, the indirect influence of  $\mathbf{u}_1$  on  $\mathbf{x}_3$  via  $\mathbf{x}_2$  has to necessarily generate  $\mathbf{x}_2$  first (Fig. 4a). Similar to the previous architecture, in the absence of a causal graph (i.e., when only the causal ordering is known), the causal NF will need to rely on optimization to discard all spurious correlations.
4.1 Necessary conditions
We next analyse the necessary conditions for the design of a causal NF to be able to accurately approximate and manipulate an SCM. A summary of the analysis can be found in Tab 1.
Expressiveness The least restrictive condition is that the causal NF should be able to reach the optima and, as mentioned in §3, a single ANF layer (Eq. 3) is a universal TMI approximator [25].
Identifiability In order to perform interventions as we describe later in §5, we need the causal NF to isolate the exogenous variables, so that we can associate them with their respective endogenous variables. As we saw in §3, if the causal NF is expressive enough, and if it follows a valid causal ordering w.r.t. the true causal graph  $\mathbf{A}$ , then Thm. 1 ensures that we can isolate the exogenous variables up to elementwise transformations.
Causal consistency As stated in Cor. 2, the causal NF needs to share the causal dependencies of the SCM at the optima, meaning that their Jacobian matrices need to be structurally equivalent, i.e.,  $\nabla_{\mathbf{x}}T_{\theta}(\mathbf{x})\equiv I - A$  (Fig. 4d), and  $\nabla_{\mathbf{u}}T_{\theta}^{-1}(\mathbf{u})\equiv \sum_{n = 1}^{\mathrm{diam}(A)}A^{n} + I$  (Fig. 4c). Given the (partial) causal graph  $A$ , the generative model in Eq. 5 by design holds  $\nabla_{\mathbf{u}}T_{\theta}^{-1}(\mathbf{u})\equiv \sum_{n = 1}^{\mathrm{diam}(A)}A^{n} + I$  for any sufficient number of layers  $L\geq$  diam  $A$  (see [34, Prop. 1]), however, there might still exist spurious paths from  $\mathbf{x}$  to  $\mathbf{u}$ . Similarly, the abductive model in Eq. 6, while may not remove all spurious paths from  $\mathbf{x}$  to  $\mathbf{u}$  if  $L > 1$ , ensures causal consistency when  $L = 1$ . In cases where the selected architecture for the causal NF does not ensure causal consistency by design, but we have access to the causal graph, we can use this extra information to regularize our MLE problem as
$$
\underset {\theta} {\text {m i n i m i z e}} \quad \mathrm {E} _ {\mathbf {x}} \left[ - \log p \left(T _ {\boldsymbol {\theta}} (\mathbf {x})\right) + \| \nabla_ {\mathbf {x}} T _ {\boldsymbol {\theta}} (\mathbf {x}) \odot (\mathbf {1} - \mathbf {A}) \| _ {2} \right], \tag {7}
$$
where  $\mathbf{1}$  is a matrix of ones, thus penalizing spurious correlations from  $\mathbf{x}$  to  $T_{\theta}(\mathbf{x})$
Tab 1 summarizes the discussed properties of the considered design choices. Remarkably, the abductive model with a single layer (similar to Fig. 4d) enjoys all the necessary properties of a causal  $NF$  by design. That is, the abductive model with  $L = 1$  is expressive, and causally consistent w.r.t. the provided causal graph  $A$ , greatly simplifying the optimization process.
Remark. It is not straightforward why abductive models can be causally consistent by design, but generative models cannot. The answer lies on the structure of the problem. Intuitively, this is a consequence of the mapping  $\mathbf{x} \to \mathbf{u}$  being structurally sparser ( $\mathbf{u}_i$  depends on  $\mathrm{pa}_i$ , see Fig. 4d) than that of  $\mathbf{u} \to \mathbf{x}$  ( $\mathbf{x}_i$  depends on  $\mathrm{an}_i$ , see Fig. 4c). Therefore, ensuring causal consistency from  $\mathbf{u}$  to  $\mathbf{x}$  does not necessarily imply causal consistency from  $\mathbf{x}$  to  $\mathbf{u}$ .
Table 1: Summary of the considered design choices, their induced properties, and their time complexity for density evaluation and sampling. Generative models design their forward pass as  $\mathbf{u}\rightarrow \mathbf{x}$ , and abductive models as  $\mathbf{x}\rightarrow \mathbf{u}$ . See §4 for an in-depth discussion.
Design Choices	Model Properties	Time Complexity
Network Type	Causal Assumption	Causal Consistency	Sampling	Evaluation
u → x	x → u
u → x	Generative	Ordering	x	x	O(L)
Generative	Graph A	✓	x	O(L)
Abductive	Ordering	x	x	O(dL)
x → u	Abductive (L > 1)	Graph A	x	x	O(dL)
Abductive (L = 1)	Graph A	✓	✓	O(dL)
5 Do-operator: enabling interventions and counterfactuals
In this section, we propose an implementation of the do-operator well-suited for causal NFs, such that we can evaluate the effect of interventions and counterfactuals [28]. The do-operator [30], denoted as  $do(\mathbf{x}_i = \alpha)$ , is a mathematical operator that simulates a physical intervention on an SCM  $\mathcal{M}$ , inducing an alternative model  $\mathcal{M}^{\mathcal{I}}$  that fixes the observational value  $\mathbf{x}_i = \alpha$ , and thus removes any causal dependency on  $\mathbf{x}_i$ . Usually, the do-operator is implemented by yielding an SCM  $\mathcal{M}^{\mathcal{I}} = (\tilde{\mathbf{f}}^{\mathcal{I}}, P_{\mathbf{u}})$  result of replacing the  $i$ -th component of  $\tilde{\mathbf{f}}$  with a constant function,  $\tilde{f}_i^{\mathcal{I}} := \alpha$ . Unfortunately, this implementation of the do-operator only works for the recursive representation of the SCM (Fig. 4a), thus not generalizing to the different architecture designs of causal NFs discussed in §4 the previous section.
We instead propose to manipulate the SCM by modifying the exogenous distribution  $P_{\mathbf{u}}$ , while keeping the causal equations  $\tilde{\mathbf{f}}$  untouched. Specifically, an intervention  $do(\mathbf{x}_i = \alpha)$  updates  $P_{\mathbf{u}}$ , restricting the set of plausible  $\mathbf{u}$  to those that yield the intervened value  $\alpha$ . We define the intervened SCM as  $\mathcal{M}^{\mathcal{I}} = (\tilde{\mathbf{f}}, P_{\mathbf{u}}^{\mathcal{I}})$ , where the density of  $P_{\mathbf{u}}^{\mathcal{I}}$  is of the form
$$
p ^ {\mathcal {I}} (\mathbf {u}) = \delta \left(\left\{\tilde {f} _ {i} \left(\mathbf {x} _ {\mathrm {p a} _ {i}}, \mathbf {u} _ {i}\right) = \alpha \right\}\right) \cdot \prod_ {j \neq i} p _ {j} \left(\mathbf {u} _ {j}\right), \tag {8}
$$
and where  $\delta$  is the Dirac delta located at the unique value of  $\mathbf{u}_i$  that yields  $\mathbf{x}_i = \alpha$  after applying the causal mechanism  $\tilde{f}_i$ . This approach resembles the one proposed for soft interventions [10] and backtracking counterfactuals [37]. Moreover, as shown in App. C, it can be generalized for non-bijective causal equations. Note also that Eq. 8 is well-defined only if the set  $\{\mathbf{u} \sim P_{\mathbf{u}} \mid \tilde{\mathbf{f}}_i(\mathbf{x}, \mathbf{u}) = \alpha\}$  is non-empty, i.e., if the intervened variable takes a plausible value (i.e., with positive density in the original causal model). Remarkably, this implementation works directly on the distribution of the exogenous variables and can be applied to any SCM representation (see Fig. 4), and therefore any of the architectures for the causal NF in §4.
Implementation details We take advantage of the autoregressive nature of causal NFs, and generate samples from Eq. 8 by: i) obtaining the exogenous variables  $\mathbf{u} \coloneqq T_{\theta}(\mathbf{x})$ ; ii) replacing the observational value by its intervened value,  $x_{i} \coloneqq \alpha$ ; and iii) by setting  $u_{i}$  to the value of the  $i$ -th component of  $T_{\theta}(\mathbf{x})$ . If the causal NF has successfully isolated the exogenous variables ( $\S 3$ ), and it preserves the true causal paths ( $\S 4$ ), then the causal NF ensures that  $x_{i} = \alpha$  independently of the value of its ancestors, since  $u_{i}$  can be seen in Eq. 8 as a deterministic function of the given  $\alpha$  and the value of its parents (and therefore its ancestors). We provide further details and the step-by-step algorithms to compute interventions and counterfactuals with causal NFs in App. C.
6 Empirical evaluation
In this section, we empirically validate the insights from §4, and compare causal NFs with previous works. Additional results and in-depth descriptions can be found in App. D.
(a) Ablation study on a 4-node chain SCM, as we vary the number of layers  $L$ . Top: generative models ( $\mathbf{u} \rightarrow \mathbf{x}$ ). Bottom: abductive models ( $\mathbf{x} \rightarrow \mathbf{u}$ ).
(b) Comparison of the two best models of each row.
Figure 5: Ablation of different choices of the causal NF to be causally consistent, and capture the observational and interventional distributions. The use of regularization on the Jacobian (Eq. 7) is indicated with the  $\star$  superscript. The abductive causal NF with information on  $A$  and  $L = 1$  outperforms the rest of models across all metrics, demonstrating its efficacy and simplicity.
6.1 Ablation study
Experimental setup We evaluate every network combination described in Tab 1 on a 4-chain SCM (which has diameter 3 and a very sparse Jacobian) and assess the extent to which these models: i) capture the observational distribution, using  $\mathrm{KL}(p_{\mathcal{M}}\| p_{\theta})$ ; ii) remain causally consistent w.r.t. the original SCM, measured via  $\mathcal{L}(\nabla_{\mathbf{x}}T_{\theta}(\mathbf{x})) \coloneqq \| \nabla_{\mathbf{x}}T_{\theta}(\mathbf{x}) \cdot (\mathbf{1} - \mathbf{A})\|_2$  from Eq. 7; and iii) perform at interventional tasks, such as estimating the Average Treatment Effect (ATE) [29] and computing counterfactuals, both of which we measure with the RMSE w.r.t. the original SCM. Every experiment is repeated 5 times, and every causal NF uses Mask Autoregressive Flows (MAFs) [24] as layers.
Results Fig. 5 shows the result for different design choices of causal NFs. Specifically, we show: network design (generative  $\mathbf{u} \rightarrow \mathbf{x}$  vs. abductive  $\mathbf{x} \rightarrow \mathbf{u}$ ), causal knowledge (ordering vs. graph), number of layers  $L$ , and whether to use MLE with regularization (Eq. 4 vs. 7).
First, we see in Fig. 5a (top) that, as expected, the generative models  $(\mathbf{u} \rightarrow \mathbf{x})$  using the causal graph cannot capture the SCM with  $L < \mathrm{diam}A = 3$ . Furthermore, we observe that abductive models  $\mathbf{x} \rightarrow \mathbf{u}$  (Fig. 5a, bottom) accurately fit the observational distribution, and that embedding the causal graph in the architecture significantly improves the ATE estimation.
Second, we now compare the two network designs in Fig. 5b, and observe that in general abductive models results in more accurate estimates of the observational distribution, as well as of interventional and counterfactual queries. Finally, we observe that regularization works well in all cases, yet it renders useless for the abductive model with  $L = 1$  and knowledge on the graph, since it is causally consistent by design. In summary, our experiments confirm that, despite its simplicity, the causal abductive model with  $L = 1$  outperforms the rest of design choices. As a consequence, in the following sections we will stick to this particular design choice, and refer to it as causal NF.
6.2 Non-linear SCMs
Experimental setup We compare our causal NF (causal, abductive, and with  $L = 1$ ) with two relevant works: i) CAREFL [16], an abductive NF with knowledge on the causal ordering and affine layers; and ii) VACA [34], a variational auto-encoding GNN with knowledge on the graph. For fair comparison, every model uses the same budget for hyperparameter tuning, our causal NF uses affine layers, and CAREFL has been modified to use the proposed do-operator from §5 (as the original implementation only works in root nodes). We increase the complexity of the SCMs and consider: i) TRIANGLE, a 3-node SCM with a dense causal graph; ii) LARGEBD [11], a 9-node SCM with non-Gaussian  $P_{\mathbf{u}}$  and made out of two chains with common initial and final nodes; and iii) SIMPSON [11], a 4-node SCM simulating a Simpson's paradox [33], where the relation between two variables changes if the SCM is not properly approximated.
Results The results are summarized in Tab 2. In a nutshell: the proposed causal NF outperforms both CAREFL and VACA in terms of performance and computational efficiency. VACA shows poor performance, and is considerably slower due to the complexity of GNNs. Our causal NF outperforms
Table 2: Comparison, on three non-linear SCMs, of the proposed causal NF, VACA [34], and CAREFL [16] with the do-operator proposed in §5. Results averaged over five runs.
Dataset	Model	Performance	Time Evaluation (μs)
KL	ATERMSE	CFRMSE	Training	Evaluation	Sampling
TRIANGLE	Causal NF	0.000.00	0.120.03	0.130.02	0.520.07	0.580.07	1.070.12
NLIN	CAREFL†	0.000.00	0.120.03	0.170.03	0.570.18	0.830.26	1.680.62
[34]	VACA	7.710.60	4.780.01	4.190.04	28.821.21	23.000.55	70.653.70
LARGEBD	Causal NF	1.510.04	0.020.00	0.010.00	0.520.10	0.600.17	3.050.66
NLIN	CAREFL†	1.510.05	0.050.01	0.080.01	0.840.47	1.180.17	8.251.29
[11]	VACA	53.662.07	0.390.00	0.820.02	164.9211.10	137.8815.72	167.9425.75
SIMPSON	Causal NF	0.000.00	0.070.01	0.120.02	0.590.17	0.600.11	1.510.30
SYMPROD	CAREFL†	0.000.00	0.100.02	0.170.04	0.490.15	0.810.19	1.910.33
[11]	VACA	13.850.64	0.890.00	1.500.04	49.264.09	37.783.41	79.2014.60
Table 3: Accuracy, F1-score, and counterfactual unfairness of the audited classifiers. Causal NFs enable both fair classifiers and accurate unfairness metrics. Results are averaged on five runs.
Logistic classifier	SVM classifier
full	unaware	fair x	fair u	full	unaware	fair x	fair u
f1	72.28616	72.37490	59.66857	73.08438	76.04286	76.80582	68.28574	77.39152
accuracy	67.00383	66.75263	54.75591	66.50370	69.50311	71.00383	59.25299	69.75126
unfairness	5.84293	2.81072	0.00000	0.00000	6.65245	2.78040	0.00000	0.00000
CAREFL in counterfactual estimation tasks with identical observational fitting, showing once more the importance of being causally consistent. Even more, our causal NF is also quicker than CAREFL, as best-performing CAREFL architectures have in general more than one layer.
7 Use-case: fairness auditing and classification
To show the potential practical impact of our work, we follow the fairness use-case of Sánchez-Martin et al. [34] on the German Credit dataset [8]—a dataset from the UCI repository where the likelihood of individuals repaying a loan is predicted based on a small set of features, including sensitive attributes such as their sex. Extra details and results appear in App. E.
Experimental setup As proposed by Chiappa [3], we use a partial graph which groups the 7 discrete features of the dataset in 4 different blocks with known causal relationships, putting in practice the results from §3.1. For the causal NF, we use the abductive model with a single non-affine neural spline layer [9]. Our ultimate goal is to train a causal NF that captures well the underlying SCM, and use it to train and evaluate classifiers that predict the (additional) binary feature credit risk, while remaining counterfactually fair w.r.t. the binary variable sex,  $\mathbf{x}_S$ .
In this setting, we call a binary classifier  $\kappa : \mathbb{X} \to \{0,1\}$  counterfactually fair [21] if, for all possible factual values  $\mathbf{x}^{\mathrm{f}} \in \mathbb{X}$ , the counterfactual unfairness remains zero. That is, if we have that  $\operatorname{E}_{\mathbf{x}^{\mathrm{f}}} \left[ P(\kappa(\mathbf{x}^{\mathrm{cf}}) = 1 \mid d o(\mathbf{x}_S = 1), \mathbf{x}^{\mathrm{f}}) - P(\kappa(\mathbf{x}^{\mathrm{cf}}) = 1 \mid d o(\mathbf{x}_S = 0), \mathbf{x}^{\mathrm{f}})\right] = 0$ , where  $\mathbf{x}^{\mathrm{cf}}$  is a counterfactual sample coming from the distribution  $P(\mathbf{x}^{\mathrm{cf}} \mid d o(\mathbf{x}_S = s), \mathbf{x}^{\mathrm{f}})$ , for  $s = 0,1$ .
Following Sánchez-Martin et al. [34], we audit: a model that takes all observed variables (full); an unaware model that leaves the sensitive attribute  $\mathbf{x}_S$  out; a fair model that only considers non- descendant variables of  $\mathbf{x}_S$  (fair  $\mathbf{x}$ ); and, to demonstrate the ability to learn a counterfactually fair classifier, we include a classifier that takes  $\mathbf{u} = T_{\theta}(\mathbf{x})$  as input, but leaves  $\mathbf{u}_S$  out (fair  $\mathbf{u}$ ).
Results Tab 3 summarizes the performance and unfairness of the classifiers, using logistic regression [6] and SVMs [5]. Here, we observe that by taking the non-sensitive exogenous variables from the causal NF, the obtained classifiers achieve comparable or better accuracy than the rest of the classifiers, while at the same time being counterfactually fair. Moreover, the estimations of unfairness obtained with the causal NF match our expectations [21], with full being the most unfair, followed by aware and the two fair models. With this use-case, we demonstrate that Causal NFs may indeed be a valuable asset for real-world causal inference problems.
8 Concluding remarks
In this work, we have shown—both theoretically and empirically—that causal NFs are a natural choice to learn a broad class of causal data-generating processes in a principled way. Specifically, we have proven that causal NFs can match the observational distribution of an underlying SCM, and that in doing so the ANF needs to be causally consistent. However, as limited data availability and local optima may hamper reaching these solutions in practice, we have explored different network designs, exploiting the available knowledge on the causal graph. Moreover, we have provided causal NFs with a do-operator to efficiently solve causal inference tasks. Finally, we have empirically validated our findings, and demonstrated that our causal NF framework: i) outperforms competing methods; and ii) can deal with mixed data and partial knowledge on the causal graph.
Practical limitations Despite considering a broad class of SCMs, we have made several assumptions that, while being standard, may not hold in some application scenarios. With regard to our causal assumptions, the presence of unmeasured hidden confounders may break our causal sufficiency assumption; mismatches between the true causal graph (e.g., it may contain cycles) and our assumed graph/ordering may lead to poor estimates of interventional and counterfactual queries; and the non-bijective true causal dependencies may invalidate our theoretical and thus practical findings. Besides, we have focused on MLE estimation for learning the causal NF. However, MLE does not test the independency of the exogenous variables during training, which would also break our causal sufficiency assumption.
Future work We firmly believe that our work opens a number of interesting directions to explore. Naturally, we would like to address current limitations by, e.g., using interventional data to address the existence of hidden confounders [14, 23], explore alternative losses other than MLE (e.g., flow matching [22]). Moreover, it would be exciting to see causal NFs applied to other problems such as causal discovery [11], fair decision-making [21], or neuroimaging [12], among others. However, we would like to stress that, in the above contexts, it would be essential to validate the suitability of our framework (e.g., using experimental data) to prevent potential harms.
We would like to thank Batuhan Koyuncu and Jonas Klesen for their invaluable feedback, as well as the anonymous reviewers whose feedback helped us improve this manuscript. This project is funded by DFG grant 389792660 as part of TRR 248 – CPEC. Pablo Sánchez Martín thanks the German Research Foundation through the Cluster of Excellence “Machine Learning – New Perspectives for Science”, EXC 2064/1, project number 390727645 for generous funding support. The authors also thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Pablo Sánchez Martín.
[1] Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006. (page 3)
[2] Patrick Chao, Patrick Blobaum, and Shiva Prasad Kasiviswanathan. Interventional and counterfactual inference with diffusion models. ArXiv preprint, abs/2302.00860, 2023. URL https://arxiv.org/abs/2302.00860. (page 1, 2, 29)
[3] Silvia Chiappa. Path-specific counterfactual fairness. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 7801-7808. AAAI Press, 2019. doi: 10.1609/aaai.v33i01.33017801. URL https://doi.org/10.1609/aaai.v33i01.33017801. (page 9)
[4] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). In Proc. of ICLR, 2016. URL http://arxiv.org/abs/1511.07289. (page 27)
[5] Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20(3): 273-297, 1995. (page 9)