SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our goal is to show that  $\partial_{x_l}\log p_l(X_l|\mathrm{PA}_l) = \partial_{x_l}\log p_l(U_l)$ , with  $U_{l} = g^{-1}(X_{l}) - f_{l}(\mathrm{PA}_{l})$  and  $g^{-1}$  the inverse of the postnonlinear function  $g$  (which is invertible by modeling assumption). As a notational remark, in what follows we will drop any sub-index on the distribution of the random variables, which we distinguish by their argument. Also, we denote realizations of random variables (or random vectors) with lowercase letters (e.g.  $x_{l}$  is the value of the random variable  $X_{l}$ ). We rewrite the distribution of  $X_{l}$  conditional on its parents by marginalizing over all values of  $U_{l}$ :
$$
\begin{array}{l} p \left(x _ {l} \mid \mathrm {p a} _ {l}\right) = \int_ {u _ {l}} p \left(x _ {l} \mid \mathrm {p a} _ {l}, u _ {l}\right) p \left(u _ {l}\right) d u _ {l} (24) \\ = \int_ {u _ {l}} p \left(x _ {l} \mid \operatorname {p a} _ {l}, u _ {l}\right) p \left(u _ {l}\right) \mathbb {1} \left(x _ {l} = g \left(f _ {l} \left(\operatorname {p a} _ {l}\right) + u _ {l}\right)\right) d u _ {l} (25) \\ = \int_ {u _ {l}} p \left(x _ {l} \mid \operatorname {p a} _ {l}, u _ {l}\right) p \left(u _ {l}\right) \mathbb {1} \left(u _ {l} = g ^ {- 1} \left(x _ {l}\right) - f _ {l} \left(\operatorname {p a} _ {l}\right)\right) d u _ {l}, (26) \\ \end{array}
$$
with  $\mathbb{1}$  being the indicator function. Being  $g$  an invertible function, the value of  $u_{l}$  equals to  $g^{-1}(x_l) - f_l(\mathrm{pa}_l)$  is unique, which implies that  $p(x_{l}|\mathrm{pa}_{l},u_{l}) = 0$  if  $u_{l}\neq g^{-1}(x_{l}) - f_{l}(\mathrm{pa}_{l})$ , else  $p(x_{l}|\mathrm{pa}_{l},u_{l}) = 1$ . Let us denote  $u_{l}^{*}\coloneqq g^{-1}(x_{l}) - f_{l}(\mathrm{pa}_{l})$ . Then, the integral in Equation 26 simply becomes:
$$
p \left(x _ {l} \mid \mathrm {p a} _ {l}\right) = \int_ {u _ {l}} d p \left(u _ {l}\right) \mathbb {1} \left(u _ {l} = u _ {l} ^ {*}\right) = p \left(u _ {l} ^ {*}\right). \tag {27}
$$
Thus,  $\partial_{x_l}\log p(X_l|\mathrm{PA}_l) = \partial_{x_l}\log p(U_l)$
E.2 Discussion
Proposition 1 derives a connection between Lemma 2 defined by Montagna et al. [25] for identifiable additive noise models to the case of a PNL model. Note that the authors define a consistent estimator of  $s_l$  score function of a leaf node  $X_l$  from the residual  $R_l \coloneqq X_l - \mathbf{E}[X_l \mid \mathbf{X} \setminus X_l]$ , which satisfies  $R_l = U_l$  in the case of an ANM with noise terms centered at zero. In general, the latter equality does not hold for a post nonlinear model, meaning that regression of a leaf variable against all the other variables of  $\mathbf{X}$  does not guarantee a consistent estimation of the disturbance on the leaf structural equation. This implies that, as is, NoGAM doesn't provide theoretical guarantees of consistent estimation of the topological order of a PNL model.
F Proof of Proposition 2
We define two lemmas preliminary to the proof of Proposition 2.
Lemma 4. Let  $\mathbf{X} \in \mathbb{R}^d$  be generated according to an SCM  $\mathcal{M}$  that satisfies score-sortability, and let  $\mathcal{G}$  be the graph induced by the model. Then, there exists a leaf node of  $\mathcal{G}$  that is score-identifiable.
Proof. By contradiction, let's say that the node  $X_{l}$  with  $l = \operatorname{argmin}_{i}\operatorname{Var}[s_i(\mathbf{X})]$  is not a leaf node. Then, the causal order  $\pi$  where  $X_{l}$  is a successor of all other nodes in the graph, is not a correct ordering, implying that the model is not score-sortable.
Lemma 5. Let  $\mathbf{X} \in \mathbb{R}^d$  be generated according to an SCM  $\mathcal{M}$  that satisfies score-sortability, and let  $\mathcal{G}$  be the graph induced by the model. Let  $\mathcal{M}_{\backslash \{l\}}$  the model defined removing the leaf node  $X_{l}$  from the set of structural equations of  $\mathcal{M}$ . Then, the model  $\mathcal{M}_{\backslash \{l\}}$  is score-sortable.
Proof. By contradiction, let's assume that  $\mathcal{M}_{\backslash \{l\}}$  does not satisfy score-sortability, such that the node  $X_{m}$  with  $m = \operatorname{argmin}_{i=1,\ldots,l-1,l+1,\ldots,d} \operatorname{Var}[s_{i}(\mathbf{X})]$  is not a leaf node in the graph  $\mathcal{G}_{\backslash \{l\}}$  induced by  $\mathcal{M}_{\backslash \{l\}}$ . Then, any topological order  $\pi$  with  $X_{m}$  successor of all nodes  $X_{i}, i = 1,\dots,l-1,l+1,\dots,d$ , is a wrong topological ordering of the graph  $\mathcal{G}$ . This implies that  $\mathcal{M}$  is not score-sortable.
Now, we present the proof of Proposition 2.
Proof. (Proof of Proposition 2.) By Lemma 4, being  $\mathcal{M}$  a score-sortable model, there exists a leaf  $X_{l}$  such that  $l\coloneqq \mathrm{argmin}_i\mathrm{Var}[s_i(\mathbf{X})]$ . Then,
$$
\operatorname {V a r} \left[ \partial_ {l} \log p _ {l} \left(X _ {l} \mid \mathrm {P A} _ {l}\right) \right] \leq \operatorname {V a r} \left[ \partial_ {i} \log p _ {i} \left(X _ {i} \mid \mathrm {P A} _ {i}\right) \right] + \sum_ {k \in \mathrm {C H} _ {i}} \operatorname {V a r} \left[ \partial_ {k} \log p _ {k} \left(X _ {k} \mid \mathrm {P A} _ {k}\right) \right] + C,
$$
for all  $i = 1, \ldots, d$ . Moreover, by previous Lemma 5, the model  $\mathcal{M} \backslash \{l\}$  is score-sortable. Thus there exists an index  $m \in \{1, \ldots, l-1, l+1, \ldots, d\}$  such that  $X_m$  is a leaf and  $\operatorname{Var}[s_m(\mathbf{X} \backslash \{l\})] \leq \operatorname{Var}[s_i(\mathbf{X} \backslash \{l\})], \forall i = 1, \ldots, l-1, l+1, \ldots, d$ . Then, the topological ordering defined by iterative identification of leaf nodes with Lemma 4 and Lemma 5 on the subgraphs resulting by removal of a leaf node, is correct with respect to the model  $\mathcal{M}$ .
G Tuning of the hyperparameters in the experiments
The methods included in the benchmark require the tuning of several hyperparameters for the inference procedure. In particular, PC, DAS, SCORE, NoGAM, RESIT, GraN-DAG, CAM, and DiffAN require a threshold  $\alpha$  over the p-value of the statistical test used for the edge selection procedure. Instead, GES applies a regularization term weighted by  $\lambda$  to its score, which penalizes the number of edges included in the inferred graph: the higher the value of  $\lambda$ , the sparser the solution. Given that the tuning of both  $\alpha$  and  $\lambda$  requires prior knowledge about the sparsity of the ground truth, there is no established procedure for finding their optimal values in real-world settings, where the ground truth is not accessible. Thus, in order to enable a fair comparison between all the methods, we always select the optimal value of  $\alpha \in \{0.001, 0.01, 0.05, 0.1\}$  and  $\lambda \in \{0.05, 0.5, 2, 5\}$  over each benchmark dataset. In Section H we discuss the stability of the algorithms with respect to choices of these hyperparameters.
Figure 4: The violin plots in the figure represent the difference between the F1 score of a method running inference with hyperparameters optimized using the ground truth, versus the F1 score of the same method using a default value of the hyperparameters. We denote this difference with  $|\mathrm{f1}_{\mathrm{diff}}|$ . In the case of GES, we define as default  $\lambda = 0.5$ . For all the remaining methods, the default alpha threshold is defined as  $\alpha = 0.05$ . The violin plots refer to the inference performance on datasets and graphs generated according to 20 different random seeds. Results in the table are on data generated from the vanilla scenario, and we consider Erdos-Renyi graphs with the number of nodes in  $\{5, 10, 20, 50\}$  in the dense and sparse settings.
GraN-DAG and DiffAN both define a learning procedure over the data, which requires the tuning of several training hyperparameters, the most important of which is the learning rate  $\eta$ . For each dataset, this is optimized over the loss function on a held-out validation set, without accessing the ground truth graph.
H Stability with respect to hyperparameters choices
Most causal discovery methods come with hyperparameters that alleviate minor assumption violations (e.g. sparsity regularization or higher thresholds on p-values in statistical tests). In the absence of background knowledge, tuning these hyperparameters is an art that often relies on pre-conceptions about reasonable solutions. In this section, we investigate the impact of these hyperparameters on the accuracy of the output graph. GES penalizes dense solution with a regularization term in its score, weighted by a hyperparameter  $\lambda$  that can not be tuned in the absence of the ground truth. Similarly, an  $\alpha$  threshold on p-values for statistical tests for edge selection is required by all the benchmarked methods (excluding GES and DirectLiNGAM) and can not be tuned without knowledge of the ground truth. In this section, we analyze the inference F1 score by fixing  $\alpha$  and  $\lambda$  to the commonly accepted default values of 0.05 and 0.5 respectively. In Figure 4 we summarise the absolute value of the difference between the F1 score obtained with hyperparameters optimized on the ground truth of each dataset, against the F1 score yielded by inference with the default  $\alpha$  and  $\lambda$  values (denoted with  $|\mathrm{f}1_{\mathrm{diff}}|$  in the plots). According to our empirical findings, in the case of graphs with at least 10 nodes, the median of this difference is in general lower than 0.1, and most of the time close to 0. Sparse graphs seem to be more affected in their performance by the hyperparameters choice: this means that using the default  $\alpha$  and  $\lambda$  causes an increase of false positives in the output graph. Buhlmann et al. [46] shows that under correct topological order, a graph whose set of edges is a superset of the ground truth still provides consistent estimates of the causal effects, such that increasing the false positives doesn't affect the outcome of downstream tasks, but only the statistical efficiency of the inference. Given that estimation of the topological ordering is not affected by the choice of  $\alpha$  and  $\lambda$  values, we suggest that the role of hyperparameters value is in this respect marginal with respect to the task of interest.
Implications. We observe remarkable stability of the benchmarked methods with respect to the choice of their hyperparameters. The biggest drops in F1 score are observed on sparse graphs, meaning that the default parameters cause an increase of false positives, which nevertheless does not affect the downstream task of interest of consistent estimation of causal effects.
I Other experimental results on Erdos-Renyi graphs
In this section, we present additional experimental results on Erdos-Renyi graphs.
I.1 The effect of non-iid distribution of the data
Figure 1f (right) illustrates that all the methods included in our benchmark do not perform well on data sampled from a non-iid distribution generated according to the autoregressive model of Equation (7): F1 score and FNR-  $\hat{\pi}$  are indeed similar to that of the random baseline. It clearly appears that none of the presented algorithms provide guarantees of good empirical performance in the setting of non-iid distribution of the data.
I.2 Experiments under arbitrary distribution of the noise terms
In Section 2.2 we discussed the effect of the distribution of the noise terms on the identifiability of the causal graph underlying an SCM. Given that the assumption of Gaussian distribution of the disturbances is often not satisfied in real datasets, it is important to provide empirical evidence on the performance of the benchmarked methods on data generated with an arbitrary distribution of the noise. In this section, we discuss experiments on data generated with the noise terms that are iid samples from the distribution of Figure 2c. Similar to Section 4, we analyze results on ER graphs with 20 nodes, with experiments repeated over 20 random seeds. In this section, we include results of DirectLiNGAM, on both linear and nonlinear SCMs.
Figure 5 illustrates the FNR- $\pi$  score of the inferred topological order on data generated according to the vanilla model with non-Gaussian noise terms. Under these conditions, NoGAM and RESIT provide theoretical guarantees of consistent estimate of the causal ordering. Similarly, PC and GES do not make explicit assumptions on the distribution of the noise terms (despite the fact that GES optimizes a Gaussian likelihood). SCORE, DiFFAN, DAS, and CAM instead are limited by restrictions on the noise terms, which are required to be normally distributed. However, Figure 5 (right) shows that, except for CAM, they can estimate the order with accuracy comparable to that achieved in the case of vanilla generated data, with Gaussian distribution of the disturbances. These observations are in line with the experimental findings in Montagna et al. [25], which shows how the structure of the score entries of leaf nodes can still be exploited by SCORE for inference on data generated under arbitrary noise distribution. Our experimental results agree with this intuition: surprisingly, SCORE ordering ensures better FNR- $\hat{\pi}$  accuracy than RESIT, despite the latter being explicitly designed to be insensitive to the distribution of the noise terms. Interestingly, we notice that the median of the violin plot referred to DirectLiNGAM in Figure 5 (right) is close to that of RESIT and CAM: this suggests that in the realistic scenario of mixed linear and nonlinear mechanisms with non-Gaussian additive disturbances, we can expect DirectLiNGAM to give performance significantly better than several methods designed to perform on nonlinear ANM. Figure 5 (left), shows that the in the case of methods whose ordering accuracy is comparable to the Gaussian case, the F1 score after pruning is also comparable to that on Gaussian data. This means that CAM-pruning is robust with respect to arbitrary distributions of the noise terms. Additional experimental results on data generated according to the misspecified scenarios of Section 3.1.1 with non-Gaussian distribution of the disturbances, are presented in Figure 6.
Implications. Most of the benchmarked methods are capable of robust inference on datasets generated by an ANM with non-Gaussian noise terms. DirectLiNGAM shows remarkable performance, comparable to that of several methods designed for inference on nonlinear additive noise models.
I.3 Experiments with score-sortability
In this section, we present the experimental results of a simple ordering algorithm, that we name ScoreSort, based on the score-sortability criterion defined in Section 4.1.2. Given the random vector
Figure 5: F1 score and FNR-  $\hat{\pi}$  on data generated with non-Gaussian distribution of the noise terms (c.f. Figure 2c). For each method, we also display the violin plot of its performance on the vanilla scenario with Gaussian noise terms, with transparent color. F1 score (the higher the better) and FNR-  $\hat{\pi}$  (the lower the better) are evaluated over 20 seeds on Erdos-Renyi dense graphs with 20 nodes (ER-20 dense). FNR-  $\hat{\pi}$  is not computed for GES and PC methods, whose output is a CPDAG.
(e) Unfaithful distribution
(d) Measurement error
progressive model (non-Gaussian) - ER-20 dense
(f) Non-  $i$  id data distribution
Figure 6: Experimental results on the misspecified scenarios with non-Gaussian distribution of the noise terms. For each method, we also display the violin plot of its performance on the same misspecified scenario under Gaussian distribution of the noise terms, with transparent color. F1 score (the higher the better) and FNR-  $\hat{\pi}$  (the lower the better) are evaluated over 20 seeds on Erdos-Renyi dense graphs with 20 nodes (ER-20 dense). FNR-  $\hat{\pi}$  is not computed for GES and PC, methods whose output is a CPDAG.
(a) Vanilla
(b)PNL
(c) LiNGAM
(d) Latent confounders
(e) Measurement error
(f) Unfaithful distribution
(g) Non-  $i\bar{d}$  distribution
Figure 7: Experiments on score-sortability. We compare the FNR-  $\hat{\pi}$  accuracy of the simple ScoreSort baseline (c.f. Algorithm 1) with SCORE and NoGAM algorithms performance. The violin plots are evaluated over 20 seeds on Erdos-Renyi dense graphs with 20 nodes (ER-20 dense).
$\mathbf{X} \in \mathbb{R}^d$  generated according to a structural causal model, the ScoreSort baseline identifies the index of a leaf node  $l$  as the  $\mathrm{argmin}_i \operatorname{Var}[s_i(\mathbf{X})]$ . Then it removes the leaf node  $X_l$  from the set of vertices of the graph, and identifies the next leaf with the argmin of the variance of the score vector of the remaining set of nodes. Identification of leaf nodes according to this procedure over the  $d$  (sub)graphs obtained by the iterative leaves removal yields a topological ordering  $\pi$  for the graph  $\mathcal{G}$  underlying the SCM. If the model is score-sortable, then, according to Proposition 2, the causal order  $\pi$  is correct with respect to the graph. Details on ScoreSort are presented in the Algorithm box 1. In practice, ScoreSort estimates the score vector  $\hat{s}$  according to the same score-matching algorithm used by SCORE and NoGAM, which is based on the Stein identity [22].
Figure 7 compares ScoreSort performance with NoGAM and SCORE ordering algorithms, on data generated according to the vanilla and misspecified scenarios of Section 3.1.1. In line with our considerations in the discussion on score matching robustness in Section 4.1.2, we observe that ScoreSort FNR- $\hat{\pi}$  accuracy is comparable, with statistical significance, to that of SCORE and NoGAM. This is true both in the case of data generated under vanilla and misspecified scenarios.
Algorithm 1 ScoreSort algorithm for inference of the causal order
Input: data matrix  $X\in \mathbb{R}^{n\times d}$ $\pi \gets []$    
nodes  $\leftarrow [1,\dots ,d]$    
for  $i = 1,\ldots ,d$  do  $\hat{\mathbf{s}}\gets$  score-matching(X)  $l_{index}\gets$  argmin  $\hat{\mathrm{Var}} [\hat{\mathbf{s}} ]$ $l\gets$  nodes[liindex]  $\pi \gets [l,\pi ]$  Remove  $l_{index}$  -th column from  $X$  ; Remove  $l$  from nodes   
end for   
 $\pi \gets$  reverse(  $\pi$  ) (first node is a source, last node is a leaf)   
return  $\pi$
J Experiments on SF, GRP, and FC graphs
In this section, we analyze the F1 score and FNR-  $\hat{\pi}$  accuracy of the benchmarked methods in the case when the ground truth graph is generated with Scale-free, Fully Connected, and Gaussian Random Partitions algorithms.
J.1 Experiments on Scale-free graphs
Figure 8 illustrates the F1 score and FNR-  $\hat{\pi}$  on SF graphs. We see that similar to the case of ER networks, score matching-based methods show remarkable robustness in the inferred order in the case of several misspecified scenarios, particularly, on data generated by the PNL (Figure 8a right), measurement error (Figure 8d right), and unfaithful models (Figure 8e right). However, we notice two significant differences with respect to the conclusions that we derived in the case of ER graphs in Section 4, Figure 1: in the case of the LiNGAM model, SCORE, DAS and NoGAM display FNR-  $\hat{\pi}$  accuracy that is remarkably close to that on vanilla data (Figure 8b right), whereas their decrease in performance in the case of latent confounders effects (Figure 8c right), is worse than that observed on ER graphs. Interestingly, the results on the F1 score show that DAS, SCORE, NoGAM, and DiffAN performance is surprisingly good (with respect to the random baseline) across all the misspecified scenarios, which suggests good performance of CAM-pruning on SF graphs. Moreover, we see that GraN-DAG and RESIT inference procedure is close to that of the random baseline in almost all the misspecified scenarios: this is also explained by the poor performance of these two methods on vanilla data and SF graphs (illustrated in the transparent violin plots of Figure 8).
Implications. Score-matching based approaches show remarkable robustness even in the case of SF graphs. Interestingly, CAM-pruning performance on SF graphs is generally better than the one relative to ER-generated ground truths, such that the observed F1 score is often better than random. We also observe that RESIT and GraN-DAG ordering ability is negatively affected by the SF ground truth, in comparison to the case of ER graphs.
J.2 Experiments on fully connected graphs
In the case of fully connected graphs, the ground truth admits a unique topological ordering. This means that we expect to observe an increase in the false negative rate FNR-  $\hat{\pi}$ , with respect to the results on ER graphs of Figure 1. This is in line with our empirical evidence, as illustrated in Figure 9. However, we see that score matching-based approaches still show robust performance in the inference of the ordering with respect to misspecified scenarios, except for the case of data generated according to the LiNGAM ground truth model. Notably, the F1 score accuracy of GES is consistently better than that of all the other methods, across every scenario. This is to be understood with the fact that the unpenalized BIC score optimized by GES always improves by increasing the number of edges in the graph. Given that we optimize the regularizer term  $\lambda$  on each dataset, the optimal  $\lambda$  value will naturally privilege the densest solutions. Different is the case for methods that rely on CAM-pruning, which display an F1 score consistently lower than the random baseline, except for the case of data generated by the unfaithful and LiNGAM models.
Implications. Score matching-based approaches are in general robust to misspecifications of the scenario in the case of a fully connected ground truth. GES shows a remarkable performance, that is partly explained by the optimization of the loss penalization term directly on the ground truth. Finally, we observe that the CAM-pruning step is negatively affected by the large density of the graphs.
J.3 Experiments on GRP graphs
In Figure 10 we see that score matching-based methods and CAM algorithm display better robustness in the inference of the order than the remaining approaches, in reference to all of the benchmarked scenarios. The FNR-  $\hat{\pi}$  of RESIT, GraN-DAG, and DiffAN are significantly close to the random baseline for data generated according to most of the ground truth models (with the exception of DiffAN on the LinGAM model and GraN-DAG on unfaithful samples). In terms of F1 score, most of the methods show good capability of inferring the ground truth graph, even in the case of data generated under assumption violations. Note that the F1 score of the random ground truth is remarkably bad, if compared to the case of SF, FC, and ER graphs. This is in line with the cluster structure of GRP graphs: given that the random baseline connects pair of nodes all with the same
PNL model - SF-20 dense
LiNGAM-SF-20 dense
Confounded model - SF-20 dense
Measurement error model - SF-20 dense
(d) Measurement error
Figure 8: Experimental results on the misspecified scenarios. For each method, we also display the violin plot of its performance on the vanilla scenario with transparent color. F1 score (the higher the better) and FNR-  $\hat{\pi}$  (the lower the better) are evaluated over 20 seeds on Scale-free dense graphs with 20 nodes (SF-20 dense). FNR-  $\hat{\pi}$  is not computed for GES and PC methods, whose output is a CPDAG. Note that DirectLiNGAM performance does not appear, as both the linear mechanisms and non-Gaussian noise assumptions are violated.
probability 0.5, we expect a large number of false positives due to edges between nodes of different clusters.
Implications. Score matching-based approaches and CAM algorithm are remarkably robust to model misspecification both in terms of F1 score and FNR-  $\hat{\pi}$  accuracy.
K Other results