SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\tilde {f} _ {3} \left(\mathrm {x} _ {1}, \mathrm {x} _ {2}, \mathrm {u} _ {3}\right) = 0. 2 5 \cdot \mathrm {x} _ {2} - 0. 5 \cdot \mathrm {x} _ {1} + 0. 5 \cdot \mathrm {u} _ {3} \tag {49}
$$
FORKLIN:
$$
\tilde {f} _ {1} \left(\mathrm {u} _ {1}\right) = \mathrm {u} _ {1} \tag {50}
$$
$$
\tilde {f} _ {2} \left(\mathrm {u} _ {2}\right) = 2 - \mathrm {u} _ {2} \tag {51}
$$
$$
\tilde {f} _ {3} \left(\mathrm {x} _ {1}, \mathrm {x} _ {2}, \mathrm {u} _ {3}\right) = 0. 2 5 \cdot \mathrm {x} _ {2} - 1. 5 \cdot \mathrm {x} _ {1} + 0. 5 \cdot \mathrm {u} _ {3} \tag {52}
$$
$$
\tilde {f} _ {4} \left(\mathrm {x} _ {3}, \mathrm {u} _ {4}\right) = \mathrm {x} _ {3} + 0. 2 5 \cdot \mathrm {u} _ {4} \tag {53}
$$
FORKNLIN:
$$
\tilde {f} _ {1} \left(\mathrm {u} _ {1}\right) = \mathrm {u} _ {1} \tag {54}
$$
$$
\tilde {f} _ {2} \left(\mathrm {u} _ {2}\right) = \mathrm {u} _ {2} \tag {55}
$$
$$
\tilde {f} _ {3} \left(\mathrm {x} _ {1}, \mathrm {x} _ {2}, \mathrm {u} _ {3}\right) = \frac {4}{1 + e ^ {- \mathrm {x} _ {1} - \mathrm {x} _ {2}}} - \mathrm {x} _ {2} ^ {2} + 0. 5 \cdot \mathrm {u} _ {3} \tag {56}
$$
$$
\tilde {f} _ {4} \left(\mathrm {x} _ {3}, \mathrm {u} _ {4}\right) = \frac {2 0}{1 + e ^ {0 . 5 \cdot \mathrm {x} _ {3} ^ {2} - \mathrm {x} _ {3}}} + \mathrm {u} _ {4} \tag {57}
$$
LARGEBDNLIN: Let us define
$$
\mathrm {L} (x, y) = s (x + 1) + s (0. 5 + y) - 3. 0, \tag {58}
$$
and let us call  $\mathrm{CDF}^{-1}(\mu, b, x)$  the quantile function of a Laplace distribution with location  $\mu$ , scale  $b$ , evaluated at  $x$ . Then the structural equations are
$$
\tilde {f} _ {1} \left(\mathrm {u} _ {1}\right) = s \left(1. 8 \cdot \mathrm {u} _ {1}\right) - 1 \tag {59}
$$
$$
\tilde {f} _ {2} \left(\mathrm {x} _ {1}, \mathrm {u} _ {2}\right) = 0. 2 5 \cdot \mathrm {u} _ {2} + \mathrm {L} \left(\mathrm {x} _ {1}, 0\right) \cdot 1. 5 \tag {60}
$$
$$
\tilde {f} _ {3} \left(\mathrm {x} _ {1}, \mathrm {u} _ {3}\right) = \mathrm {L} \left(\mathrm {x} _ {1}, \mathrm {u} _ {3}\right) \tag {61}
$$
$$
\tilde {f} _ {4} \left(\mathrm {x} _ {2}, \mathrm {u} _ {4}\right) = \mathrm {L} \left(\mathrm {x} _ {2}, \mathrm {u} _ {4}\right) \tag {62}
$$
$$
\tilde {f} _ {5} \left(\mathrm {x} _ {3}, \mathrm {u} _ {5}\right) = \mathrm {L} \left(\mathrm {x} _ {3}, \mathrm {u} _ {5}\right) \tag {63}
$$
$$
\tilde {f} _ {6} \left(\mathrm {x} _ {4}, \mathrm {u} _ {6}\right) = \mathrm {L} \left(\mathrm {x} _ {4}, \mathrm {u} _ {6}\right) \tag {64}
$$
$$
\tilde {f} _ {7} \left(\mathrm {x} _ {5}, \mathrm {u} _ {7}\right) = \mathrm {L} \left(\mathrm {x} _ {5}, \mathrm {u} _ {7}\right) \tag {65}
$$
$$
\tilde {f} _ {8} \left(\mathrm {x} _ {6}, \mathrm {u} _ {8}\right) = 0. 3 \cdot \mathrm {u} _ {8} + (s \left(\mathrm {x} _ {6} + 1\right) - 1) \tag {66}
$$
$$
\tilde {f} _ {9} \left(\mathrm {x} _ {7}, \mathrm {x} _ {8}, \mathrm {u} _ {9}\right) = \mathrm {C D F} ^ {- 1} \left(- s \left(\frac {\mathrm {x} _ {7} \cdot 1 . 3 + \mathrm {x} _ {8}}{3} + 1\right) + 2, 0. 6, \mathrm {u} _ {9}\right) \tag {67}
$$
SIMPSONNLIN:
$$
\tilde {f} _ {1} \left(\mathrm {u} _ {1}\right) = \mathrm {u} _ {1} \tag {68}
$$
$$
\tilde {f} _ {2} \left(\mathrm {x} _ {1}, \mathrm {u} _ {2}\right) = s \left(1 - \mathrm {x} _ {1}\right) + \sqrt {3 / 2 0} \cdot \mathrm {u} _ {2} \tag {69}
$$
$$
\tilde {f} _ {3} \left(\mathrm {x} _ {1}, \mathrm {x} _ {2}, \mathrm {u} _ {3}\right) = \tanh  \left(2 \cdot \mathrm {x} _ {2}\right) + 1. 5 \cdot \mathrm {x} _ {1} - 1 + \tanh  (\mathrm {u} _ {3}) \tag {70}
$$
$$
\tilde {f} _ {4} \left(\mathrm {x} _ {3}, \mathrm {u} _ {4}\right) = \frac {\mathrm {x} _ {3} - 4}{5} + 3 + \frac {1}{\sqrt {1 0}} \cdot \mathrm {u} _ {4} \tag {71}
$$
SIMPSONSYMPROD:
$$
\tilde {f} _ {1} \left(\mathrm {u} _ {1}\right) = \mathrm {u} _ {1} \tag {72}
$$
$$
\tilde {f} _ {2} \left(\mathrm {x} _ {1}, \mathrm {u} _ {2}\right) = 2 \cdot \tanh  \left(2 \cdot \mathrm {x} _ {1}\right) + \frac {1}{\sqrt {1 0}} \cdot \mathrm {u} _ {2} \tag {73}
$$
$$
\tilde {f} _ {3} \left(\mathrm {x} _ {1}, \mathrm {x} _ {2}, \mathrm {u} _ {3}\right) = 0. 5 \cdot \mathrm {x} _ {1} \cdot \mathrm {x} _ {2} + \frac {1}{\sqrt {2}} \cdot \mathrm {u} _ {3} \tag {74}
$$
$$
\tilde {f} _ {4} \left(\mathrm {x} _ {1}, \mathrm {u} _ {4}\right) = \tanh  \left(1. 5 \cdot \mathrm {x} _ {1}\right) + \sqrt {\frac {3}{1 0}} \cdot \mathrm {u} _ {4} \tag {75}
$$
TRIANGLELIN:
$$
\tilde {f} _ {1} \left(\mathrm {u} _ {1}\right) = \mathrm {u} _ {1} + 1 \tag {76}
$$
$$
\tilde {f} _ {2} \left(\mathrm {x} _ {1}, \mathrm {u} _ {2}\right) = 1 0 \cdot \mathrm {x} _ {1} - \mathrm {u} _ {2} \tag {77}
$$
$$
\tilde {f} _ {3} \left(\mathrm {x} _ {1}, \mathrm {x} _ {2}, \mathrm {u} _ {3}\right) = 0. 5 \cdot \mathrm {x} _ {2} + \mathrm {x} _ {1} + \mathrm {u} _ {3} \tag {78}
$$
TRIANGLE $_{NLIN}$ :
$$
\tilde {f} _ {1} \left(\mathrm {u} _ {1}\right) = \mathrm {u} _ {1} + 1 \tag {79}
$$
$$
\tilde {f} _ {2} \left(\mathrm {x} _ {1}, \mathrm {u} _ {2}\right) = 2 \cdot \mathrm {x} _ {1} ^ {2} + \mathrm {u} _ {2} \tag {80}
$$
$$
\tilde {f} _ {3} \left(\mathrm {x} _ {1}, \mathrm {x} _ {2}, \mathrm {u} _ {3}\right) = \frac {2 0}{1 + e ^ {- \mathrm {x} _ {2} ^ {2} + \mathrm {x} _ {1}}} + \mathrm {u} _ {3} \tag {81}
$$
(a) 3-CHAIN
(b) 4-CHAIN
(c) 5-CHAIN
(d) TRIANGLE
(e) COLLIDER
(f) FORK
(g) SIMPSON [NLIN]
(h) SIMPSON [SYMPROD]
(i) LARGEBD
Figure 9: Causal graph of the different SCMs considered in §6 and App. D.
D.1 Ablation: Time complexity
As the first additional ablation study, we evaluate the time complexity of the design choices introduced in §4. Fig. 10 summarizes the results, where the x-axis represents the number of nodes in the dataset,
Figure 10: Time complexity comparison between the different design choices discussed in  $\S 4$ .
$d$ , and the y-axis indicates the time (in microseconds) required for a single forward pass of the normalizing flow during training. We can find several interesting insights:
Results First, abductive models ( $\mathbf{x} \to \mathbf{u}$ , solid lines) train significantly faster compared to generative models ( $\mathbf{u} \to \mathbf{x}$ , dotted lines). This emphasizes the computational cost associated with inverting autoregressive flows during training. Furthermore, it is worth noting that the time complexity of abductive models remains constant regardless of the number of nodes  $d$ , whereas the complexity of generative models increases linearly with  $d$ , as indicated in the right-most column of Tab 1. Note that we should see the exact opposite behaviour when sampling.
Second, the inclusion of Jacobian regularization, represented by a star marker, introduces a significant time overhead. This observation is clearly depicted in Fig. 10, where plot lines with the star marker are consistently above their counterparts.
Finally, leveraging causal graph information or relying solely on ordering (represented in orange and green, respectively) has minimal impact on computational time. This is once again clear in Fig. 10, as green and orange pairs largely coincide.
D.2 Ablation: Base distribution
We now assess to which extent a mismatch of the distribution  $P_{\mathbf{u}}$  between the SCM and the causal NF negatively affects performance. To this end, we consider two more complex SCMs—SIMPSON [11] and TRIANGLE [34]—and distributions—Normal and Laplace—for which we either fix or learn their parameters during training. Both SCMs use a standard Normal distribution for  $P_{\mathbf{u}}$ .
Hyperparameter tuning While we fixed the flow to have a single MAF [24] layer with ELU [4] activation functions, we determined through cross-validation the optimal number of layers and hidden units of the MLP network within MAF. Specifically, we considered the following combinations  $([a,b]$  represents two layers with  $a$  and  $b$  hidden units): [16, 16, 16, 16], [32, 32, 32], [16, 16, 16], [32, 32], [32], [64]. As discussed at the start of the section, we report test results for the configuration with the best validation performance at the last epoch.
Results The results, shown in Fig. 11a, reveal a notable distinction between Normal and Laplace distributions in terms of density estimation. However, this discrepancy appears to have minimal implications for Average Treatment Effect (ATE) and counterfactual estimation. We hypothesize that this disparity originates from dissimilarities in their tails, as it can be inferred by the slight edge of Normal over Laplace on the last column—which measures per sample differences—where bigger errors happen at the outliers elements which are, by definition, scarce. Interestingly, with this particular architecture of causal NF, every model struggles to model the denser TRIANGLE SCM.
D.3 Ablation: Flow architecture
Considering the observed challenges faced by the Masked Autoregressive Flow (MAF) [24] layer in accurately modelling the TRIANGLE SCM in the previous experiment, we further investigate the potential impact of flow architecture on performance.
(a) Ablation on the base distribution
(b) Ablation on the flow architecture
Figure 11: Performance on the SIMPSON $_{\text{NLIN}}$  and TRIANGLE $_{\text{LIN}}$  datasets of causal NFs with a) different base distributions (Normal and Laplace), where  $\theta$  indicates that we learn the parameters of the base distribution; and b) flow architectures. Differences in base distribution affect KL divergence, while the choice of flow architecture influences the overall performance.
Hyperparameter Tuning We cross-validate again the optimal number of layers and hidden units of the MLP internally used by the unique layer of the Causal NF. We consider the following values ([a, b] represents two layers with  $a$  and  $b$  hidden units): [32, 32, 32], [16, 16, 16], [32, 32], [32], and [64]. As before, test results are reported for the configuration that achieved the best performance on the validation set at the final epoch.
Results Fig. 11b summarizes our results, where we consider a Causal NF with one MAF [24] layer (depicted in orange), and a Causal NF with a Neural Spline Flow (NSF) [9] as layer (depicted in blue). Note that, NSFs are built on top of MAFs and abandoned the realm of affine ANFs, and are thus expected to outperform MAFs in general. We employed the same set of SCMs as in App. D.2.
Our empirical analysis reveals that the NSF consistently outperforms the MAF across the three metrics: observational distribution (measured by the KL divergence), ATE estimation, and counterfactual estimation. Whilst expected, these findings highlight the practical implications of selecting an appropriate flow architecture, which should be taken into consideration by practitioners.
D.4 Comparison: Extra non-linear SCMs
In this section, we complement the results from §6.2 and provide a more extensive comparison of the proposed Causal NF, along with CAREFL [16] and VACA [34], on additional datasets.
Hyperparameter Tuning For VACA, we cross-validated the dropout rate with values  $\{0.0, 0.1\}$ , the GNN layer architecture with  $\{\mathrm{GIN}, \mathrm{PNA}, \mathrm{PNADisjoint}\}$ , (see [34] for details), and the number of layers in the MLP prior to the GNN with choices  $\{1, 2\}$ . For CAREFL, we cross-validated the number of layers in the flow,  $\{1, \mathrm{diam} A\}$ , and the number of layers and hidden units in the MLP composing the flow layers (same format as before),  $\{[16, 16, 16], [32, 32], [32], [64]\}$ . For Causal NF, we used the abductive model with a single layer, and cross-validated the number of layers and hidden units in the MLP composing the layer of the flow with values  $\{[16, 16, 16], [32, 32, 32], [16, 16, 16], [32, 32], [32], [64]\}$ . We report test results for the configuration with the best validation performance at the final epoch.
Results Tab 4 shows the performance of each model for all the considered datasets, further validating the conclusions drawn in the main manuscript: the proposed Causal NF consistently outperforms both CAREFL and VACA in terms of performance and computational efficiency. The performance of VACA is notably inferior, and its computation time is significantly longer, primarily due to the complexity of graph neural networks (GNNs). Our Causal NF achieves similar performance to CAREFL in terms of observational fitting, while surpassing it on interventional and counterfactual estimation tasks. Additionally, Causal NF outperforms CAREFL in computational speed. This is to be expected since the optimal CAREFL architectures often have multiple layers, resulting in increased computation time. In contrast, Causal NF has a single layer, reducing computational complexity.
Fig. 12 qualitative proofs the effectiveness of the proposed Causal NF in accurately modelling both observational and interventional distributions for the SIMPSON $_{\mathrm{NLIN}}$  dataset. In this plot, blue represents the real distribution/samples, while orange represents the ones generated by Causal NF. Fig. 12a clearly shows that the model successfully captured the correlations among all variables in the observational distribution. Furthermore, Fig. 12b displays the interventional distribution
Table 4: Comparison, on different SCMs, of the proposed Causal NF, VACA [34], and CAREFL [16] with the do-operator proposed in §5. Results averaged over five runs.
Dataset	Model	Performance	Time Evaluation (μs)
KL	ATERMSE	CFRMSE	Training	Evaluation	Sampling
3-CHAIN	Causal NF	0.000.00	0.050.01	0.040.01	0.410.06	0.480.10	0.760.06
LIN	CAREFL†	0.000.00	0.200.13	0.200.09	0.680.24	0.970.33	1.940.77
[34]	VACA	4.441.03	5.760.07	4.980.10	36.191.54	28.330.72	75.344.58
3-CHAIN	Causal NF	0.000.00	0.030.01	0.020.01	0.520.06	0.560.03	1.020.05
NLIN	CAREFL†	0.000.00	0.050.02	0.040.02	0.600.22	0.840.22	1.660.41
[34]	VACA	12.821.00	1.540.03	1.320.02	39.454.12	30.932.30	84.369.60
4-CHAIN	Causal NF	0.000.00	0.070.02	0.040.01	0.560.08	0.620.15	1.540.40
LIN	CAREFL†	0.000.00	0.160.07	0.140.04	0.700.28	0.990.20	2.850.54
VACA	13.140.73	3.820.01	3.720.05	61.855.06	49.314.11	92.067.93
5-CHAIN	Causal NF	0.010.00	0.120.02	0.080.01	0.620.19	0.690.15	1.910.44
LIN	CAREFL†	0.000.00	0.470.23	0.460.22	0.790.41	1.190.25	4.210.87
VACA	17.310.84	5.950.05	6.060.08	103.7510.04	80.8111.06	124.5220.86
COLLIDER	Causal NF	0.000.00	0.020.01	0.010.00	0.460.12	0.560.11	0.950.19
LIN	CAREFL†	0.000.00	0.020.01	0.010.00	0.390.07	0.450.05	0.740.07
[34]	VACA	13.450.43	0.220.01	0.860.02	37.223.55	28.774.22	71.216.73
FORK	Causal NF	0.000.00	0.030.01	0.010.00	0.520.05	0.590.08	1.570.57
LIN	CAREFL†	0.000.00	0.040.01	0.020.00	0.600.17	0.780.16	2.391.06
[2]	VACA	8.750.73	0.870.02	1.430.02	45.844.64	34.662.39	73.294.70
FORK	Causal NF	0.000.00	0.070.02	0.070.00	0.630.16	0.740.31	1.840.84
NLIN	CAREFL†	0.010.01	0.110.04	0.180.07	0.570.17	0.770.08	1.960.17
[2]	VACA	5.090.60	2.010.03	3.190.06	49.225.48	42.132.95	101.0218.94
LARGEBD	Causal NF	1.510.04	0.020.00	0.010.00	0.520.10	0.600.17	3.050.66
NLIN	CAREFL†	1.510.05	0.050.01	0.080.01	0.840.47	1.180.17	8.251.29
[11]	VACA	53.662.07	0.390.00	0.820.02	164.9211.10	137.8815.72	167.9425.75
SIMPSON	Causal NF	0.290.01	0.040.01	0.020.00	0.580.18	0.630.26	1.570.64
NLIN	CAREFL†	0.290.01	0.040.01	0.030.00	0.690.32	1.020.26	2.950.79
[11]	VACA	18.970.66	0.600.00	1.190.02	54.767.46	43.697.92	87.0116.33
SIMPSON	Causal NF	0.000.00	0.070.01	0.120.02	0.590.17	0.600.11	1.510.30
SYMPROD	CAREFL†	0.000.00	0.100.02	0.170.04	0.490.15	0.810.19	1.910.33
[11]	VACA	13.850.64	0.890.00	1.500.04	49.264.09	37.783.41	79.2014.60
TRIANGLE	Causal NF	0.000.00	0.240.05	0.210.05	0.540.05	0.560.04	1.050.07
LIN	CAREFL†	0.000.00	0.150.06	0.140.03	0.600.20	0.750.05	1.500.10
[34]	VACA	3.820.69	7.490.07	7.220.17	27.461.53	21.611.00	67.006.23
TRIANGLE	Causal NF	0.000.00	0.120.03	0.130.02	0.520.07	0.580.07	1.070.12
NLIN	CAREFL†	0.000.00	0.120.03	0.170.03	0.570.18	0.830.26	1.680.62
[34]	VACA	7.710.60	4.780.01	4.190.04	28.821.21	23.000.55	70.653.70
obtained when we do  $do(\mathrm{x}_3 = -1.09)$ , i.e., when we intervene on the 25-th empirical percentile of  $\mathbf{X}_3$ . Remarkably, Causal NF accurately learns the distribution of descendant variables, i.e.,  $\mathbf{X}_4$ , and effectively breaks any dependency between the ancestors of the intervened variable and  $\mathbf{X}_4$ . Additionally, Fig. 13 shows a similar analysis for  $5\text{-CHAIN}_{\text{LIN}}$ , when we perform  $do(\mathrm{x}_3 = 2.18)$  which corresponds to intervening on the 75-th percentile of  $\mathbf{X}_3$ —clearly showing that the correlations not involving the intervened path ( $\mathrm{x}_1 \to \mathrm{x}_2$  and  $\mathrm{x}_4 \to \mathrm{x}_5$ ) are preserved.
E Details on the fairness use-case
In this section, we provide additional details on the use-case of fairness auditing and classification using the German dataset [8], whose causal graph is shown in Fig. 14.
Training For this section, we performed minimal hyperparameter tuning, and only tested a few combinations by hand. We decided to use a Neural Spline Flow (NSF) [9] for the single layer of the Causal NF, which internally uses an MLP with 3 layers, and 32 hidden units each. We use Adam [17] as the optimizer, with a learning rate of 0.01, along with a plateau scheduler with a decay factor of
(a) Observational distribution.
(b) Interventional distribution  $d o(x_{3} = -1.09)$
Figure 12: Pair plot of real (in blue) and generated (in orange) data of SIMPSON $_{\mathrm{NLIN}}$ . On the left are samples from the true and learnt observational distribution. On the right are samples from the true and learnt interventional distribution when  $do(\mathbf{x}_3 = -1.09)$ . The plot illustrates that the dependency of  $\mathbf{x}_4$  on the ancestors of  $\mathbf{x}_3$ , namely  $\mathbf{x}_1$  and  $\mathbf{x}_2$ , is effectively broken.
(b) Interventional distribution  $do(\mathrm{x}_3 = 2.18)$ .
Figure 13: Pair plot of real (in blue) and generated (in orange) data of  $5\text{-CHAIN}_{\text{LIN}}$ . On the left are samples from the true and learnt observational distribution. On the right are samples from the true and learnt interventional distribution when  $do(\mathrm{x}_3 = 2.18)$ . The plot illustrates that the dependency of  $\mathbf{x}_4$  and  $\mathbf{x}_5$  on the ancestors of  $\mathbf{x}_3$ , namely  $\mathbf{x}_1$  and  $\mathbf{x}_2$ , is effectively broken.
0.9 and a patience parameter of 60 epochs. The training is performed for 1000 epochs, and the results are reported using 5-fold cross-validation with a  $80 - 10 - 10$  split for train, validation, and test data.
Results On addition to the results from §7, Fig. 15 shows two pair plots from one of the 5 runs, chosen at random. The true empirical distribution is shown in blue, and the learnt distribution by Causal NF is depicted in orange. Specifically, Fig. 15a illustrates the observational distribution, and Fig. 15b the interventional distribution, obtained when we intervene on the sex variable and set it to 1, i.e.,  $do(\mathrm{x}_1 = 1)$ . We can observe that Causal NF achieves a remarkable fit in both cases, demonstrating its capability to handle discrete data, and partial knowledge of the causal graph.
Figure 14: Partial causal graph used for the German Credit dataset [8]. Rectangles show the strongly connected components (SCCs) grouping different variables. Solid arrows represent causal relationships between SCCs, and dashed arrows represent an arbitrary order picked to learn the joint distribution of each SCC with an ANF. See App. A.2.2 for an in-depth explanation on the proposed method to deal with partial causal graphs using causal NFs.
(a) Observational distribution.
(b) Interventional distribution  $do(\mathbf{x}_1 = 1)$ .
Figure 15: Pair plot of real (in blue) and generated (in orange) data of German dataset. Above, the samples from the true and learnt observational distributions. Below, the samples from the true and learnt interventional distributions when  $do(\mathrm{x}_1 = 1)$ . The plot illustrates that Causal NF is able to handle discrete data and correctly intervene.