SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In order to simulate nonlinear causal mechanisms of an additive noise model, we sample functions from a Gaussian process, such that  $\forall i = 1,\dots ,d,f_{i}(X_{PA_{i}}) = \mathcal{N}(\mathbf{0},K(X_{PA_{i}},X_{PA_{i}}))$ , a multivariate normal distribution centered at zero and with covariance matrix as the Gaussian kernel  $K(X_{PA_i},X_{PA_i})$ , where  $X_{PA_i}$  are the observations of the parents of the node  $X_{i}$ .
B.2 Non-Gaussian distribution of the noise terms
We generate data with non-Gaussian noise terms as follows: for each node  $i \in \{1, \dots, d\}$ , we model noise terms following a Gaussian distribution  $U_i \sim \mathcal{N}(0, \sigma_i)$  with variance  $\sigma_i \sim U(0.5, 1.0)$ . Those noise terms are then transformed via a random nonlinear function  $t$ , s.t.  $\tilde{U}_i = t(U_i)$ . In our experiments, we sampled three different functions  $t$ , modeled as multilayer perceptrons (MLPs) with 100 nodes in the single hidden layer, sigmoid activation functions, and weights sampled from  $U(-\alpha, \alpha), \alpha \in \{0.5, 1.5, 3.0\}$ , respectively (c.f. Fig. 2).
B.3 Algorithms for random graphs simulation
We use four random graph generation algorithms for sampling the ground truth causal structure of each dataset. In particular, we consider the Erdos-Renyi (ER) model, which allows specifying the number of nodes  $d$  and the average number of connections per node  $m$  (or, alternatively, the probability  $p$  of connecting each pair of nodes). In ER graphs, pair of nodes have the same probability of being connected. Scale-free graphs are generated under a preferential attachment procedure [41], such that nodes with a higher degree are more likely to be connected with a new node, allowing for the presence of hubs (i.e. high degree nodes) in the graphs. The Gaussian Random Partition model (GRP) [42] is created by connecting  $k$  subgraphs (i.e. partitions) generated by an ER model. A parameter  $p_{in}$  specifies the probability of connecting a pair of nodes in the same partition, while  $p_{out}$  defines the probability of connections among distinct partitions. Clusters appear when  $p_{in} >> p_{out}$  (e.g. in our experiments we consider  $p_{in} = 0.4$ ,  $p_{out} = 0.05$ ). Finally, we consider Fully Connected graphs, generated by sampling a topological order  $\pi$  and connecting all nodes in the graph to their successors with a directed edge. Given a ground truth fully connected graph, the accuracy of the inference procedure is maximally sensitive to errors in the order.
(a)  $U(-0.5, 0.5)$
(b)  $U(-1.5, 1.5)$
(c)  $U(-3.0, 3.0)$
Figure 2: Gaussian noise (left) transformed via random nonlinear functions (center) to non-Gaussian iid noise (right). Weights of the MLP are sampled from either (a)  $U(-0.5, 0.5)$ , (b)  $U(-1.5, 1.5)$ , or (c)  $U(-3.0, 3.0)$ .
B.4 Modeling of unfaithful distributions
Given a ground truth causal graph, we model an unfaithful distribution of the data by enforcing the cancellation of directed causal effects between pairs of nodes. In practice, we identify the fully connected triplets of nodes  $X_{i} \to X_{k} \gets X_{j} \gets X_{i}$  in the ground truth, and we adjust the causal mechanisms such that the direct effect of  $X_{i}$  on  $X_{k}$  cancels out. In order to clarify the implementation details of our model, we consider a graph  $\mathcal{G}$  with vertices  $X_{1}, X_{2}, X_{3}$  and with the set of edges corresponding to the fully connected graph with trivial topological order  $\pi = \{X_{1}, X_{2}, X_{3}\}$ . We allow for mixed linear and nonlinear edges with liner effect on the nodes, such the set of structural equations is defined as:
$$
X _ {1} := U _ {1},
$$
$$
X _ {2} := f \left(X _ {1}\right) + U _ {2}, \tag {17}
$$
$$
X _ {3} := f \left(X _ {1}\right) - X _ {2} + U _ {3},
$$
with  $f$  nonlinear function. This definition of the mechanisms on  $X_{3}$  cancels out  $f(X_{1})$  in the structural equation. In the case of large graphs with the number of nodes  $d \in \{5, 10, 20, 50\}$  that we use in our experiments, we verify the unfaithful independencies in the data via kernel-based test of conditional independence [52], in correspondence of the pairs of nodes whose causal effect cancels out. We use a threshold of 0.05 for the conditional independence testing.
B.5 Dataset configurations
In this section, we extend the discussion of Section 3.1.2 which presents an overview of the parameters that define the different configurations for the generation of the synthetic datasets of our benchmark. We sample the ground truth structures according to four different algorithms for random graph
5 nodes	10 nodes	20 nodes	50 nodes
Sparse	p = 0.1*	m = 1	m = 1	m = 2
Dense	p = 0.4*	m = 2	m = 4	m = 8
* Graphs are re-sampled such that they have at least 2 edges.
Table 2: Density schema for Erdos-Renyi graphs. The parameter  $p$  denotes the probability of an edge between each pair of nodes in the graph, and  $m$  denotes the average number of edges for each node in the graph. We scale the parameter  $m$  with the number of nodes, such that the relative density (sparsity) is similar for all graph dimensions.
generation, and according to different specifications of density, number of nodes, and distribution of the noise terms. In the case of Erdos-Renyi (ER) generated graphs, we define the density of the edges relative to the number of nodes, according to the schema defined in Table 2. For the Scale-free (SF) model, we define the edge density in the graphs according to the same values of Table 2, but we do not generate SF graphs of 5 nodes. Similarly, we generate fully connected (FC) and Gaussian Random Partition (GRP) graphs only for  $\{10,20,50\}$  nodes. FC generation does not require specifying any parameter for the density. In the case of GRP graphs, we use  $p_{in} = 0.4$  as the probability of edges between a pair of nodes inside the same cluster, and  $p_{out} = 0.1$  as the probability of edges between a pair of nodes belonging to different clusters.
For each of the graph configurations, we sample a ground truth and a dataset of observations generated according to one of the following scenarios (described in detail in Section 3.1.1):
- Vanilla additive noise model.
- PNL model, with invertible post nonlinear function  $g(x) = x^3$  for each speficied structural equation.
- LiNGAM model, where the number of structural equations with linear mechanisms is parametrized by  $\delta \in \{0.33, 0.66, 1.0\}$ . The first two values of  $\delta$  allow modeling mixed linear and nonlinear causal mechanisms, with respectively  $33\%$  and  $66\%$  of the structural equations being linear. Unless differently specified, we consider  $\delta = 1$  when referring to the LiNGAM model.
- Confounded model, where the number of confounded pairs is parametrized by  $\rho \in \{0.1, 0.2\}$ , denoting the probability of two nodes having a common cause. Unless differently specified, we consider  $\rho = 0.2$  when referring to the confounded model.
- Measurement error model, where the amount of variance explained by the additive error is parametrized by  $\gamma \in \{0.2, 0.4, 0.6, 0.8\}$ , denoting the inverse signal to noise ratio  $\gamma := \frac{\operatorname{Var}[\epsilon_i]}{\operatorname{Var}[X_i]}$ , with  $X_i$  and  $\epsilon_i$  defined in the structural equation (6). Unless differently specified, we consider  $\gamma = 0.8$  when referring to the measurement error model.
- Unfaithful model, as discussed in the Appendix B.4.
- Autoregressive model, defined according to the structural equation (7) in order to simulate non-iid samples in the data.
For each scenario, and for each parametrization that it admits, we generate a ground truth  $\mathcal{G}$  according to each of the graph configurations specified at the beginning of the section, and a corresponding pair of datasets  $\mathcal{D}$  of size 100 and 1000. The dataset generation is repeated under four possible distributions of the noise terms. In particular, each dataset has exogenous variables that are either normally distributed, or following a randomly generated non-Gaussian distribution, as discussed in Appendix B.2. Finally, in order to ensure statistically significant results, for each pair of graph and dataset configurations we generate  $\mathcal{G},\mathcal{D}$  according to 20 different random seeds.
C Benchmark methods
C.1 CAM
CAM algorithm [46] infers a causal graph from data generated by an additive Gaussian noise model. First, it infers the topological ordering by finding the permutation of the graph nodes corresponding
to the fully connected graph that maximizes the log-likelihood of the data. After inference of the topological ordering, a pruning step is done by variable selection with regression. In particular, for each variable  $X_{j}$  CAM fits a generalized additive model using as covariates all the predecessor of  $X_{j}$  in the ordering, and performs hypothesis testing to select relevant parent variables. This is known as the CAM-pruning algorithm. For graphs with size strictly larger than 20 nodes, the authors of CAM propose an additional preliminary edge selection step, known as Preliminary Neighbours Search (PNS): given an order  $\pi$ , variable selection is performed by fitting for each  $j = 1, \dots, d$  an additive model of  $X_{j}$  versus all the other variables  $\{X_{i}: X_{j} \succ X_{i} \text{ in } \pi\}$ , and choosing the  $K$  most important predictor variables as possible parents of  $X_{j}$ . This preliminary search step allows scaling CAM pruning to graphs of large dimensions. In our experiments, CAM-pruning is implemented with the preliminary neighbours search only for graphs of size 50, with  $K = 20$ .
C.2 RESIT
In RESIT (regression with subsequent independence test) [13] the authors exploit the independence of the noise terms under causal sufficiency to identify the topological order of the graph. For each variable  $X_{i}$ , they define the residuals  $R_{i} = X_{i} - \mathbf{E}\left[X_{i} \mid \mathbf{X} \setminus \{X_{i}\}\right]$ , such that for a leaf node  $X_{l}$  it holds that  $R_{l} = U_{l} - \mathbf{E}[U_{l}]$ . The method is based on the property that under causal sufficiency, the noise variables are independent of all the preceding variables: after estimating the residuals from the data, it identifies a leaf in the graph by finding the residual  $R_{l}$  that is unconditionally independent of any node  $X_{i}, \forall i \neq l$  in the graph. Once an order is given, they select a subset of the edges admitted by the fully connected graph encoding of the ordering. We implement this final step with CAM-pruning.
C.3 GraN-DAG
GraN-DAG [38] defines a continuous constrained optimization problem to infer the causal graph from an ANM with Gaussian noise terms. For each variable  $X_{i}$  in the graph, the authors estimate the parameters of the conditional distribution  $p(X_{i}\mid \mathbf{X}\setminus \{X_{i}\})$  with a neural network  $\phi_{i}$ . They define an adjacency matrix  $A\in \mathbb{R}^{d\times d}$  representation of the causal DAG, by finding inactive paths in the neural network computations, where a path is defined as the sequence of weights of the network from the input  $j$  to the output  $k$ : if zero weights are encountered in the path, then the output  $k$  is independent of the input  $j$ . If this is repeated for all paths from  $j$  to  $k$  and for all outputs  $k$ , then all paths are inactive and  $X_{i}$  is independent of  $X_{j}$  conditional on the remaining variables, meaning that  $A_{ij} = 0$ . Note that GraN-DAG in principle does not require a post-processing consisting of an edge selection procedure, given that its output is not a fully connected encoding of a topological order, but an arbitrarily sparse graph. However, in practice, it is the case that GraN-DAG output approximates a fully connected graph, with a large number of false positives with respect to the ground truth edges (see Table 5 of Appendix A.3 in Lachapelle et al. [38] for quantitative results). To account for this, the authors of the method apply the CAM-pruning step on top of their neural network graph output. In our experiments, in order to compare the goodness of the ordering encoded by the GraN-DAG output before applying the CAM-pruning step, we sample one order at random between those admitted by the output adjacency matrix and compute its FNR-  $\hat{\pi}$ . Given that the order is selected at random, we consider an unbiased solution, as we show in Figure 3.
C.4 DirectLiNGAM
ICA-LiNGAM [44] formulates a causal discovery algorithm for the identifiable LiNGAM model, assuming linear mechanisms and non-Gaussian noise terms. The idea is that solving for  $\mathbf{X}$  the system defined in Equation (3), one obtains
$$
\mathbf {X} = \mathbf {A U}, \tag {18}
$$
where  $\mathbf{A} = (\mathbf{I} - \mathbf{B})^{-1}$ . By standard linear ICA (independent component analysis) it is possible to find  $\mathbf{A}$ , which is equivalent to finding the weighted adjacency matrix  $\mathbf{B}$ . This intuition lies at the base of the DirectLiNGAM algorithm [44], a variation of ICA-LiNGAM that uses pairwise independence measures to find the topological order of the graph, and covariance-based regression to find the connection strengths in the matrix  $\mathbf{B}$ .
Figure 3: In order to evaluate the goodness of the inferred ordering of GraN-DAG, we sample one topological order at random between those admitted by the adjacency matrix before the CAM-pruning step. In this figure, we compare the empirical FNR-  $\hat{\pi}$  of an order randomly sampled between those admitted by the output, against the average of the FNR-  $\hat{\pi}$  computed on the set of all possible orderings admitted by the output. We see that selecting an order at random gives an unbiased representation of the average order accuracy, between those admitted by GraN-DAG output before the CAM-pruning. The violin plots refer to the FNR-  $\hat{\pi}$  evaluated on ER graphs with 10 nodes over 20 different random seeds.
C.5 PC
PC algorithm (Section 5 of Spirtes et al. [4]) is a causal discovery method based on conditional independence testing that finds a CPDAG from the data. First, it starts from a fully connected undirected graph, and estimates the skeleton of the graph by removing edges between each pair of nodes  $X_{i}, X_{j}$  if it finds a subset  $\mathbf{S} \subset \mathbf{X} \setminus \{X_{i}, X_{j}\}$  such that  $X_{i} \perp X_{j} \mid \mathbf{S}$ . Then, it finds all the v-structures  $X_{i} \to X_{j} \gets X_{k}$  along with their directions. Finally, additional orientation rules are applied to direct as many edges as possible in the output CPDAG. In our experiments, we use a kernel-based test of conditional independence [52].
C.6 GES
The GES algorithm [9] (Greedy Equivalent Search) defines a discrete optimization problem over the space of all CPDAGs, and outputs the graph that maximizes the fit measure according to some score (e.g. the Bayesian Information Criterion (BIC) score). The algorithm is defined as a two steps greedy procedure. It starts from an empty graph, and in the forward step it adds directed edges one by one, each time selecting the directed edge that most increases the fit score. When edge addition doesn't improve the score any further, in the backward step it removes edges one by one until the score stops increasing. The DAG defined by this procedure is then transformed into a CPDAG, such that GES final output is a Markov equivalence class.
C.7 SCORE
Rolland et al. [23] defines a formal criterion for the identification of the causal order of a graph underlying an additive noise models with Gaussian distribution of the noise terms  $U_{i} \sim \mathcal{N}(0, \sigma_{i})$ . Under these assumptions, the score entry of a leaf node  $X_{l}$  is  $s_{l}(\mathbf{X}) = -\frac{X_{l} - f_{l}(\mathrm{PA}_{l})}{\sigma_{l}^{2}}$ . It is then easy to verify that  $\partial_{X_{l}} s_{l}(\mathbf{X}) = -\frac{1}{\sigma_{j}^{2}}$ , such that the diagonal entry of the score's Jacobian associated to a leaf node is a constant. Based on this relation, a formal criterion identifying leaf nodes holds:
Lemma 1 (Lemma 1 of [23]). Let  $\mathbf{X}$  be a random vector generated according to an identifiable ANM with exogenous noise terms  $U_{i} \sim \mathcal{N}(0, \sigma_{i}^{2})$ , and let  $X_{i} \in \mathbf{X}$ . Then
$$
\operatorname {V a r} \left[ \partial_ {X _ {i}} s _ {i} (\mathbf {X}) \right] = 0 \Longleftrightarrow X _ {i} \text {i s a l e a f}, \forall i = 1, \dots , d. \tag {19}
$$
The Lemma is exploited by SCORE algorithm for estimation of the topological order, given a dataset of i.i.d. observations  $X \in \mathbb{R}^{n \times d}$ : first it estimates the diagonal elements of the Jacobian matrix of the score  $J(s(\mathbf{X}))$  via score matching [22]. Then, it identifies a leaf in the graph as the  $\mathrm{argmin}_i \operatorname{Var}[\partial_{X_i} s(\mathbf{X})]$ , which is removed from the graph and assigned a position in the order vector. By iteratively repeating this two steps procedure up to the source nodes, all variables in  $\mathbf{X}$  end up
being assigned a position in the causal ordering. Finally, SCORE applies the CAM-pruning algorithm to select a subset of the edges in the fully connected DAG encoding of the inferred topological order.
C.8 NoGAM
Montagna et al. [25] proposes a generalization of SCORE, defining a formal criterion for the identification of leaf nodes in a graph induced by an additive noise model without restrictions on the distribution of the noise terms. After some manipulations, it can be shown that the score entry of a leaf  $X_{l}$  defined in Equation (9) satisfies
$$
s _ {l} (\mathbf {X}) = \partial_ {U _ {l}} \log p _ {l} (U _ {l}), \tag {20}
$$
such that observations of the pair  $(U_{l},s_{l}(\mathbf{X}))$  can be used to learn a predictor of the score entry. For an additive noise model, the authors show that the noise term of a leaf is equal to the residual defined as:
$$
R _ {l} := X _ {l} - \mathbf {E} \left[ X _ {l} \mid \mathbf {X} \backslash X _ {l} \right]. \tag {21}
$$
Then, it is possible to find a consistent approximator of the score entry of a leaf node using  $R_{l}$  as the only predictor.
Lemma 2 (Lemma 1 of [25]). Let  $\mathbf{X}$  be a random vector generated according to an identifiable ANM, and let  $X_{i} \in \mathbf{X}$ . Then
$$
\mathbf {E} \left[ \left(\mathbf {E} \left[ s _ {i} (\mathbf {X}) \mid R _ {i} \right] - s _ {i} (\mathbf {X})\right) ^ {2} \right] = 0 \Longleftrightarrow X _ {i} \text {i s a l e a f}.
$$
Similarly to SCORE, NoGAM algorithm defines a procedure for estimation of the topological order by iterative identification of leaf nodes, which are found as the  $\mathrm{argmin}_i\mathbf{E}\left[(\mathbf{E}[s_i(\mathbf{X})\mid R_i] - s_i(\mathbf{X}))^2\right]$ . In practice, the residuals  $R_{i}, i = 1,\dots ,d$  can be estimated by any regression algorithm, whereas the score is approximated by score matching with Stein identity [22].
C.9 DAS
Montagna et al. [24] defines a condition on the Jacobian of the score function that identifies the edges of the graph induced by an additive noise model with Gaussian distribution of the noise terms, given a valid causal order.
Lemma 3 (Lemma 1 of [24]). Let  $\mathbf{X}$  be a random vector generated according to an identifiable ANM with exogenous noise terms  $U_{i} \sim \mathcal{N}(0, \sigma_{i}^{2})$ , and let  $X_{l} \in \mathbf{X}$  be a leaf node. Then:
$$
\mathbf {E} \left[ \left| \partial_ {X _ {j}} s _ {l} (\mathbf {X}) \right| \right] \neq 0 \Longleftrightarrow X _ {j} \in \mathrm {P A} _ {l} (\mathbf {X}), \forall j \in \{1, \dots , d \} \backslash \{l \} \tag {22}
$$
In practice, off-diagonal elements of the Jacobian matrix contain information about conditional independencies of the variables in the DAG, such that they define a condition of identification of the graph edges. Given the ordering procedure of SCORE, DAS (acronym for Discovery At Scale) defines an algorithm that can map the score function to a unique causal graph with directed edges: in practice, condition (22) of Lemma 3 is verified via hypothesis testing for the mean equals to zero. Note that, despite the fact that this approach provides a consistent estimator of the causal graph, the authors of DAS retain a CAM-pruning step on top of their edge selection procedure based on Lemma 3, in order to reduce the number of false positives of the inferred output. The benefit of DAS preliminary edge detection is that it reduces the computational costs of CAM-pruning, which is cubic in the number of nodes in the graph, such that it doesn't scale well to high dimensional graphs. Overall, given an input dataset  $X \in \mathbb{R}^{n \times d}$ , with  $n$  number of samples and  $d$  number of nodes in the graph, DAS computational complexity is  $\mathcal{O}(dn^3 + d^2)$ , whereas, for the SCORE algorithm this is  $\mathcal{O}(dn^3 + nd^3)$ .
C.10 Random Baseline
In our experimental analysis of Section 4, we consider the performance of a random baseline in terms of F1 score and FNR-  $\hat{\pi}$  accuracy of the order (Figure 1). Our random baseline is defined as follows. Given a graph with  $d$  variables, we sample a random topological order  $\pi$  as a permutation of the vector of elements  $X_{1},\ldots ,X_{d}$ . Then, given the fully connected graph admitted by the order, with the set of edges  $\mathcal{E}_{\pi} = \{X_{\pi_i}\to X_{\pi_j}:X_{\pi_i}\prec X_{\pi_j},\forall i,j = 1,\dots ,d\}$ , for each pair of connected nodes we sample a Bernoulli random variable  $Y$  with parameter  $p = 0.5$ , such that the edge is removed for  $Y = 0$ .
D Metrics definition
For the evaluation of the experimental results of our benchmark, we consider the F1 score, the false positive (FP) and false negative (FN) rates of the inferred graph, and the false negative rate  $\mathrm{FNR - \hat{\pi}}$  of the fully connected encoding of the output topological order. In order to specify the F1 score, we need a definition of FP, FN, and true positive (TP), that applies to both undirected and directed edges, given that we evaluate both DAGs and CPDAGs.
- We define as TP any predicted edge that is in the skeleton of the ground truth graph (i.e. the set of edges that doesn't take direction into account).
- We define the FPs as the edges in the skeleton of the predicted graph that are not in the skeleton of the ground truth graph. Note that this definition of FP doesn't penalize undirected edges or edges inferred with reversed direction.
- We define as FN a pair of nodes that are disconnected in the predicted skeleton while being connected in the ground truth. Additionally, we count as false negatives inferred edges whose direction is reversed with respect to the DAG ground truth.
Then, the F1 score is defined as the ratio  $\frac{TP}{TP + 0.5(FN + FP)}$ .
E Possible generalisation of NoGAM to the PNL model
Proposition 1 of Section 3.2.1 suggests that it is possible to generalize Lemma 2 and, accordingly, the NoGAM algorithm, to the case of the post nonlinear model.
E.1 Proof of Proposition 1
Proof. Let  $\mathbf{X} \in \mathbb{R}^d$  be a random vector generated by the post nonlinear model of Equation (5). Given the Markov factorization of Equation (2), the logarithm of the joint distribution  $p_{\mathbf{X}}$  satisfies the following equation:
$$
\log p _ {\mathbf {X}} (\mathbf {X}) = \sum_ {i = 1} ^ {d} \log p _ {X _ {i}} (X _ {i}).
$$
Then, for a node  $X_{i}$  in the graph the score entry  $s_i$  is defined according to Equation (8), whereas given a leaf node  $X_{l}$  in the graph,  $s_l$  satisfies the following:
$$
s _ {l} (\mathbf {X}) := \partial_ {x _ {l}} \log p _ {\mathbf {X}} (\mathbf {X}) = \partial_ {x _ {l}} \log p _ {l} \left(X _ {l} | \mathrm {P A} _ {l}\right). \tag {23}
$$