SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Statistical efficiency. Figure 11 shows F1 score and FNR-  $\hat{\pi}$  accuracy on datasets with sample size equals to 100 and 1000. Comparing the relative difference in performance with respect to different sample sizes, we get an empirical idea of the statistical efficiency of the inference methods. In line with our expectations, the experimental results show that both metrics are negatively affected by the reduction in sample size. Interestingly, in the case of SCORE, DAS, NoGAM, and DirectLiNGAM, we observe better stability of the FNR-  $\hat{\pi}$ , compared to the other methods, with the score matching-based approaches that are in general significantly better than the random baseline also with datasets of size 100.
The effect of the graph size and density. Figure 13 illustrates the F1 score and the FNR-  $\hat{\pi}$  accuracy on datasets generated according to the vanilla scenario and ground truth graphs that differ in size and density. In particular, we consider the case of dense and sparse graphs, with  $\{5,10,20,50\}$  nodes. Interestingly, we see good stability of the F1 score across different graph dimensions in the sparse case. The decrease in performance due to larger graph sizes is more evident in the case of dense graphs: this is particularly true for dense graphs with 50 nodes, where the preliminary neighbours search step (described in Section C.1) before the CAM-pruning reduces the ability to
Figure 9: Experimental results on the misspecified scenarios. For each method, we also display the violin plot of its performance on the vanilla scenario with transparent color. F1 score (the higher the better) and FNR-  $\hat{\pi}$  (the lower the better) are evaluated over 20 seeds on fully connected graphs with 20 nodes (FC-20). FNR-  $\hat{\pi}$  is not computed for GES and PC methods, whose output is a CPDAG. Note that DirectLiNGAM performance does not appear, as both the linear mechanisms and non-Gaussian noise assumptions are violated.
infer true positives for most of the methods. Considering the FNR- $\hat{\pi}$  of the inferred orders, we see that, similarly to what we observed in the analysis of the F1 score, in the case of sparse ground truths most of the methods display stable results across different graph dimensions. Indeed, score matching-based approaches, as well as CAM, DiffAN, GraN-DAG, and DirectLiNGAM do not display any clear evidence of degraded performance for larger graphs. In the dense setting, instead, we see that CAM and DirectLiNGAM accuracy in the inference of the order is negatively affected by larger dimensionality.
The balanced scoring function. Figure 12 illustrates the inference accuracy of misspecified scenarios in terms of the Balanced Scoring Function (BSF), proposed by Constantinou [53]. This is defined as:
$$
\frac {1}{2} \left(\frac {T P}{a} + \frac {T N}{i} - \frac {F P}{i} - \frac {F N}{a}\right),
$$
where:  $TP$  and  $FP$  denote the true and false positives, respectively, and  $TN$  and  $FN$  are the true and false negatives;  $a$  and  $i$  represent the number of arcs and independencies in the true graph respectively. The key difference between the BSF and the F1 score is given by the fact that the balanced scoring function accounts for the whole confusion matrix (i.e. TP, FP, TN, FN), whereas in the F1 score the true negatives are not included in the computation. The BSF ranges from  $-1$  (worst) to 1 (best), with a value of 0 corresponding to the score of an empty graph. In general, we expected the BSF to be correlated to the F1 score. The main difference that we see is that the F1 score in general penalizes PC and GES performance more than the BSF (when comparing their accuracy to the random baseline), meaning that they tend to infer graphs with low true positive rates and large true negative rates.
PNL model - GRP-20
LiNGAM - GRP-20
Confounded model - GRP-20
Measurement error model - GRP-20
(d) Measurement error
Unfaithful model - GRP-20
Figure 10: Experimental results on the misspecified scenarios. For each method, we also display the violin plot of its performance on the vanilla scenario with transparent color. F1 score (the higher the better) and FNR-  $\hat{\pi}$  (the lower the better) are evaluated over 20 seeds on Gaussian Random Partitions graphs with 20 nodes (GRP-20). FNR-  $\hat{\pi}$  is not computed for GES and PC methods, whose output is a CPDAG. Note that DirectLiNGAM performance does not appear, as both the linear mechanisms and non-Gaussian noise assumptions are violated.
L FCI experiments on confounded graphs
In this section, we describe the experimental setting for the FCI algorithm (Fast Causal Inference) [4]. Given that the method can handle latent confounders, we focus our experiments on data generated from graphs admitting latent common causes.
PAG. The output graphical object of FCI is a Partial Ancestral Graph (PAG) [47]. It admits six types of edges. We denote the two ends of an edge as marks. The possible marks are a tail  $(-)$ , a circle  $(\circ)$ , and an arrowhead  $(>)$ , which combined allow for six edges. These graphs represent an equivalence class for Maximal Ancestral Graphs, which are graphical objects that represents the presence of confounders effects and selection bias [2].
Metrics. For the evaluation of the FCI inferred output, we adopt the strategy proposed by Heinze-Deml et al. [17] (see their Section 4.2). We define true positives, false positives, and false negatives over three possible adjacency matrices, each one defined by a specific query.
- IsPotentialParent query: the estimated adjacency matrix has  $A_{ij} = 1$  if there is an edge between  $X_i - X_j$ ,  $X_i \circ X_j$ ,  $X_i \to X_j$ ,  $X_i \circ X_j$ ,  $X_i \circ X_j$ ,  $X_i \circ X_j \to X_j$  in the estimated PAG, else  $A_{ij} = 0$ .  $A_{ij} = 1$  denotes the case in which  $X_i$  is a potential parent of  $X_j$ .
- IsAncestor query: the estimated adjacency matrix has  $A_{ij} = 1$  if there is a path from  $X_i$  to  $X_j$  with edges of type  $X_i \circ X_j$ ,  $X_i \to X_j$ ,  $X_i \circ X_j$  in the estimated PAG, else  $A_{ij} = 0$ .  $A_{ij} = 1$  denotes the case in which  $X_i$  is an ancestor of  $X_j$ .
- IsPotentialAncestor query: the estimated adjacency matrix has  $A_{ij} = 1$  if there is a path from  $X_i$  to  $X_j$  with edges of type  $X_i - X_j$ ,  $X_i \circ X_j$ ,  $X_i \to X_j$ ,  $X_i \circ -X_j$ ,  $X_i \circ -\circ X_j$ ,
Figure 11: Experiments on the effect of the sample size. We compare the F1 score and FNR-  $\hat{\pi}$  accuracy on datasets generated under the vanilla scenario with Gaussian noise, with different sample sizes. We remark that in the case of DirectLiNGAM, in order to provide meaningful results, we report the performance on datasets with non-Gaussian noise terms. Violin plots filled with color refer to datasets of size 100, and transparent violin plots refer to datasets of size 1000. The metrics are reported on Erdos-Renyi graphs of size  $\{5,10,20,50\}$  both in the sparse and dense case (PC and GES are not included for graphs of 50 nodes, as their computational demand is too high). Experiments are repeated over 20 different random seeds.
Figure 12: Experimental results on the misspecified scenarios. For each method, we also display the violin plot of its performance on the vanilla scenario with transparent color. BSF and F1 score (the higher the better) are evaluated over 20 seeds on Erdos-Renyi dense graphs with 20 nodes (ER-20 dense).
$X_{i} \circ \rightarrow X_{j}$  in the estimated PAG, else  $A_{ij} = 0$ .  $A_{ij} = 1$  denotes the case in which  $X_{i}$  is a potential ancestor of  $X_{j}$ .
For each adjacency matrix defined by one of the three queries, we define true positives, false negatives, and false positives as follows:
- A true positive (TP) is a pair  $i, j$  with  $A_{ij} = 1$  in both the inferred and ground truth adjacency matrices (with the ground truth DAG converted to a PAG).
- A false negative (FN) is a pair  $i, j$  with  $A_{ij} = 0$  in the inferred matrix, and  $A_{ij} = 1$  in the ground truth (with the ground truth DAG converted to a PAG).
- A false positive (FP) is a pair  $i, j$  with  $A_{ij} = 1$  in the inferred matrix, and  $A_{ij} = 0$  in the ground truth (with the ground truth DAG converted to a PAG).
Given these definitions of TP, FN, FP, we define the F1 score as  $F1 = \frac{TP}{TP + 0.5(FP + FN)}$ , which we use to present our empirical results in Figure 14.
M Deep-dive in PC and GES experimental results
In this section, we analyze GES and PC performance in terms of false positive and false negative rates on graphs characterized by different numbers of nodes and edge density. In Section 4.1.1, we discuss the case of inference with PC and GES on Erdős-Renyi dense and large graphs (20 nodes): Figure 1 in the main text reports PC and GES F1 score to be consistently and significantly worse than random across all of the tested scenarios. We argue that this is in line with previous findings in the literature
Figure 13: Experiments on the effect of the graph size and graph density. We compare the F1 score and FNR-  $\hat{\pi}$  accuracy on datasets generated under the vanilla scenario with Gaussian noise, on ground truth graphs with the number of nodes  $\{5,10,20,50\}$  both in the sparse and dense case (PC and GES are not included for graphs of 50 nodes, as their computational demand is too high). We remark that in the case of DirectLiNGAM, in order to provide meaningful results, we report the performance on datasets with non-Gaussian noise terms. The metrics are reported on Erdos-Renyi graphs. Experiments are repeated over 20 different random seeds.
das cam score nogam diffan grandag resit lingam pc ges
Figure 14: FCI performance on dense and sparse ER graphs, on datasets generated under latent confounders effects.
(d) Measurement error
(f) Non-  $i$  id data distribution
Figure 15: FNR (False Negative Rate) and FPR (False Positive Rate) of the experiments on the misspecified scenarios, on Erdos-Renyi dense graphs with 20 nodes (ER-20 dense). For each method, we also display the violin plot of its performance on the vanilla scenario with transparent color. The noise terms are normally distributed, except for the LiNGAM model, in which case we generate disturbances according to a non-Gaussian distribution.
by Uhler et al. [7], which shows that the set of distributions that are not strong-faithful ([54]) has non-zero Lebeasgue measure, in contrast to the set of unfaithful distributions that has zero measure (which justifies the common assumption of faithful causal models). In particular, the measure of the set of not strong-faithful distributions tends to increase for large and dense causal graphs, which we argue is key to explain the degraded performance of PC in the ER-20 dense graphs of Figure 1. Coherently with our claim, Figures 19 and 20 show FNR and FPR consistently better than random for both PC and GES, respectively on sparse and dense graphs of 5 nodes.
Figure 16: FNR (False Negative Rate) and FPR (False Positive Rate) of the experiments on the misspecified scenarios, on Erdos-Renyi sparse graphs with 20 nodes (ER-20 sparse). For each method, we also display the violin plot of its performance on the vanilla scenario with transparent color. The noise terms are normally distributed, except for the LiNGAM model, in which case we generate disturbances according to a non-Gaussian distribution.
(f) Non-iid data distribution
Figure 17: FNR (False Negative Rate) and FPR (False Positive Rate) of the experiments on the misspecified scenarios, on Erdos-Renyi dense graphs with 10 nodes (ER-10 dense). For each method, we also display the violin plot of its performance on the vanilla scenario with transparent color. The noise terms are normally distributed, except for the LiNGAM model, in which case we generate disturbances according to a non-Gaussian distribution.
Figure 18: FNR (False Negative Rate) and FPR (False Positive Rate) of the experiments on the misspecified scenarios, on Erdos-Renyi sparse graphs with 10 nodes (ER-10 sparse). For each method, we also display the violin plot of its performance on the vanilla scenario with transparent color. The noise terms are normally distributed, except for the LiNGAM model, in which case we generate disturbances according to a non-Gaussian distribution.
(f) Non-iid data distribution
Figure 19: FNR (False Negative Rate) and FPR (False Positive Rate) of the experiments on the misspecified scenarios, on Erdos-Renyi dense graphs with 5 nodes (ER-5 dense). For each method, we also display the violin plot of its performance on the vanilla scenario with transparent color. The noise terms are normally distributed, except for the LiNGAM model, in which case we generate disturbances according to a non-Gaussian distribution.
Figure 20: FNR (False Negative Rate) and FPR (False Positive Rate) of the experiments on the misspecified scenarios, on Erdos-Renyi sparse graphs with 5 nodes (ER-5 sparse). For each method, we also display the violin plot of its performance on the vanilla scenario with transparent color. The noise terms are normally distributed, except for the LiNGAM model, in which case we generate disturbances according to a non-Gaussian distribution.