SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lastly, exact GP regression is in  $\mathcal{O}(N^c)$  for  $N$  data points and  $c\approx 2.7$ , while the RFF approximation is in  $\mathcal{O}(\min \{N^3,M^3\})$  for  $M$  features [34].
In this section, we include additional information on the experiments we present in the main paper. In addition, we illustrate how LINC and its variants introduced in the previous section scale in practice.
C.1 Causal Discovery Beyond MEC
First, we revisit our main experiment on causal discovery beyond MEC in Fig. 2(a). As we show F1 scores in this figure, we also include the Precision and Recall in Fig. 5.
We can observe that LINC performs well in terms of edge recall in all settings, while maintaining precision comparably high to MSS. Only for large edge densities, LINC is surpassed by MSS, while we outperform competitors for small DAG sizes, small edge density, and under few mechanism changes respectively for a small number of contexts. In particular, for no mechanism changes  $(|S| = 0$  and  $|C| = 1$ ) and when few variables are involved in the causal relationships  $(|G| = 3$  and  $p = 0.3$ ), we can see that the recall values are close to 0.5 (dashed lines) suggesting that the DAGs recovered by the competitors are close to the known base MEC, while LINC learns additional causal directions.
LINC  $\mathbf{G}_{\mathbb{P}}$  -LINC  $\mathrm{R_{FF}}$  MSs [13]MC [12] Pooled PC [4]
Figure 5: LINC discovers causal models beyond Markov Equivalence. We report Recall and Precision on discovering causal edges within a MEC. As in the main paper, we consider synthetic data from different contexts, with the base settings of  $|\mathcal{C}| = 5$  contexts,  $|G| = 6$  variables,  $|S| = 2$  intervened variables, edge density  $p = 0.3$ , and  $|c| = 500$  samples.
C.2 Scalability Experiments
Given that we suggested more efficient variants of LINC, we evaluate their performance and scalability here. We are interested in whether  $\mathrm{LINC}_{\mathrm{CLUS}}$  can efficiently discover partitions for increasing  $|\mathcal{C}|$ , whether  $\mathrm{LINC}_{\mathrm{RFF}}$  improves runtimes for increasing sample sizes  $|c|$ , and whether our method and competitors are applicable to larger DAG sizes  $|G|$ .
Throughout this section, we consider the main experimental setup of discovering causal models beyond Markov Equivalence, and use the settings  $|S| = 1, \mathcal{C}| = 3, |c| = 500$  and  $|G| = 3, p = 0.3$ .
C.2.1 LINC with Clustering
In Fig. 6, we compare the variant of LINC that exhaustively enumerates all context partitions (Alg. 1) to  $\mathrm{LINC}_{\mathrm{CLUS}}$  which relies on clustering the contexts by compression gain (Alg. 2). As the clustering procedure in  $\mathrm{LINC}_{\mathrm{CLUS}}$ , we consider Integer Linear Programming (ILP) as well as agglomerative clustering (aggl.).
As an exhaustive search over all partitions did not finish running for  $|\mathcal{C}| = 20$  in a reasonable time (one day), we include partial results (gray).
For simplicity, we instantiated LINC with RFFs in all cases. While an enumeration of all partitions of the contexts is not feasible for more than 20 contexts, we found the clustering approaches to scale easily to at least 50 contexts. In particular, both result in high F1 scores for directing edges in a MEC given enough contexts, with perfect F1 scores when given more than 15 contexts. This suggests that in practice, rather than computing the full MDL score for each partition, we can consider using pairwise MDL compression gains to apply LINC to many contexts.
Figure 6: LINC can be combined with clustering to discover partitions. We compare exhaustive enumeration of partitions (gray, Alg. 1) to clustering (Alg. 2).
C.2.2 LINC with Random Fourier Features
Next, we turn to the kernel approximation through Random Fourier Features (RFFs) described in the previous section.
(a) Context Size.
(b) DAG size.
Figure 8: LINCs variants are accurate while being scalable. We compare LINC with RFFs and clustering to the baselines, for (a) an increasing numbers of contexts, and (b) increasing DAG size.
We also included RFFs in our main evaluation in Fig. 5 to confirm that they work well in practice. To illustrate the computational benefit, we also compare the runtimes of GPs and RFFs in Fig. 7. We again use the settings  $|S| = 1$ ,  $|c| = 500$  and  $p = 0.3$  and consider  $|C| = 3$ . We confirm that RFFs mitigate the sample complexity of the GP regression without notably affecting performance.
C.2.3 LINC and Competitors
To conclude, we compare the efficient variants of LINC to its competitors. We are interested in whether the clustering variant of LINC combined with RFFs is scalable while being accurate.
In Fig. 8, we show our experiment on discovering causal edges for up to 50 contexts, resp. variables. As we found no noticeable difference between the clustering approaches that we instantiate LINC with, we show only the ILP version to avoid clutter.
LINC scales to at least 50 contexts, while Pooled PC was not applicable to more than 30 contexts in a reasonable time, even on 3-variable DAGs (Fig. 8(a)). We also find that LINC scales to DAGs with at least 50 variables with comparable runtimes to
the nonlinear baselines (Fig. 8(b)). Regarding performance, in Fig. 8(a) we can see that MC discovers only the base MEC due to the small number of variables. Pooled PC and particularly MSS need a larger number of contexts in this case, whereas the efficient version of LINC discovers all causal directions given five contexts. We conclude from our experiments that the efficient versions of LINC reliably perform in practice.
Figure 7: LINC can be combined with RFFs to improve sample complexity. We compare LINC with the GP MDL score (solid) to the RFF approximation (dashed).
[1] B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):612-634, 2021. doi: 10.1109/JPROC.2021.3058954. URL https://ieeexplore.ieee.org/stamp/stamp.jsp? arnumber=9363924.
[2] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant Risk Minimization. CoRR, abs/1907.02893, 2019.
[51] S.L. Lauritzen. Graphical Models. Oxford Statistical Science Series. Clarendon Press, 1996. ISBN 9780191591228. URL https://books.google.de/books?id=mGQWkx4guhAC.
[52] J Bernardo, J Burger, and ADEM Smith. Information-theoretic characterization of bayes performance and the choice of priors in parametric and nonparametric problems. In Bayesian statistics, volume 6. Oxford Univ. Press, 1998.
[53] Atsushi Miyauchi, Tomohiro Sonobe, and Noriyoshi Sukegawa. Exact clustering via integer programming and maximum satisfiability. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018. doi: 10.1609/aaai.v32i1.11519. URL https://ojs.aaaai.org/index.php/AAAI/article/view/11519.