SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Without any assumptions on the functional form of structural models, observational data cannot generally distinguish between any two DAGs within the same Markov equivalence class [34], since the joint distribution always factorizes according to either DAG. Scoring functions like the BGe that assign the same score to all DAGs in the same class satisfy score equivalence. Imposing parametric models on the local probability distributions of a Bayesian network may however break score equivalence. Specifically for GPNs, an alternative factorization may not admit a representation that follows the structural equation model in (8). Consequently, DAGs belonging to the same Markov equivalence class may display different scores and become identifiable beyond their equivalence class. For this reason, the GP score generally differentiates models according to the direction of any of their edges.
Importantly, the identifiability of otherwise score equivalent DAGs is a direct consequence of the assumptions underpinning the GPN model. Because the functions  $f_{i}$  are not generally invertible, the likelihood in (7) will assign higher values to functional dependencies that admit the GPN structural equation model. Further, Peters et al. [35] demonstrated that the asymmetry distinguishing between Markov equivalent factorizations holds beyond the case of non-invertible functions. Indeed, with the exception of the linear case, all structural models that take the form in (8) with additive Gaussian noise violate score equivalence and may identify a DAG beyond its Markov equivalence class. GPNs may encompass cases where equivalent DAGs are indistinguishable, such as when the functions  $f_{i}$  are linear. In this case, the computed scores of equivalent DAGs will be similar, as long as the GP prior allows learning sufficiently linear relations. The numerical experiments in supplementary Section A.1 show that the GP-based score function displays near score equivalent behaviour when the data come from a joint Gaussian distribution, coherently with theoretical considerations [35].
4 Experimental Results
We evaluate the Bayesian GP network inference scheme on data generated from known random networks with  $n = 10$  nodes. The DAG structures are constructed from an Erdős-Rényi model, where each node has an independent probability of 0.2 to be connected with another with a higher topological ordering. For every node  $X_{i}$  in each randomly generated network, we then sample 100 observations as a non-linear function of its parents. The nonlinear data are generated by transforming the parents' instances using a weighted combination of six Fourier components
$$
X _ {i} = \sum_ {j \mid X _ {j} \in \operatorname {P a} _ {X _ {i}}} \beta_ {i, j} \left\{w _ {i, j, 0} X _ {j} + \sum_ {k = 1} ^ {6} \left[ v _ {i, j, k} \sin (k X _ {j}) + w _ {i, j, k} \cos (k X _ {j}) \right] \right\} + \epsilon_ {i}. \tag {15}
$$
The weights  $v_{i,j,k}$  and  $w_{i,j,k}$  are sampled from a Dirichlet distribution with concentration parameters equal to seven positive exponentially decreasing values  $\gamma_k = \frac{e^{-k / \lambda}}{\sum_k e^{-k / \lambda}}$ , for  $k = \{0, \dots, 6\}$ . The
parameter  $\lambda$  controls the rate of exponential decay, with values of  $\lambda$  close to zero providing mostly linear effects between variables and higher values resulting in increasingly non-linear relationships.
The edge coefficients  $\beta_{i,j}$  determine the strength of the dependencies between  $X_{i}$  and its parents and are sampled from a uniform distribution on  $(-2, -\frac{1}{2}) \cup (\frac{1}{2}, 2)$ ; the noise variable  $\epsilon_{i}$  has a standard normal distribution. Instances of root nodes (nodes without parents) are also sampled from a standard normal. The linear-Gaussian case corresponds asymptotically to  $\lambda = 0$ ; in this case, we set the weights  $v_{i,j,k}$  and  $w_{i,j,k}$  to zero except for  $w_{i,j,0}$ .
We compare all structure learning algorithms in terms of structural Hamming distance (SHD) [48], which compares estimated graphs  $\mathcal{G}$  with the ground truth graph  $\mathcal{G}^*$ . Following Lorch et al. [29], we compute the average SHD of the samples weighted by the posterior:
$$
\mathbb {E} - \operatorname {S H D} (p, \mathcal {G} ^ {*}) := \sum_ {\mathcal {G}} \operatorname {S H D} (\mathcal {G}, \mathcal {G} ^ {*}) p (\mathcal {G} | D). \tag {16}
$$
The  $\mathbb{E}$  SHD summarizes how close the estimated posterior is to the true DAG; lower values are therefore desirable. It is however not a direct measure of the similarity of the estimated posterior to the true posterior.
4.1 Choice of Priors
Different choices of kernels for the GP prior result in different behaviours of the conditional expectation  $f$  in equation (9) of a variable  $X$  given its parents  $\mathrm{Pa}_X$ . A simple model employs an additive kernel
$$
k (.,.) = \sum_ {i = 1} ^ {| \mathrm {P a} _ {X} |} k _ {\theta_ {i}} (.,.) \tag {17}
$$
which corresponds to modeling each variable  $X$  as a sum of the individual contributions of each of its parents. The additive model serves to reduce the computational burden of calculating the scores by keeping the number of parameters as small as possible while preserving non-linearity in the model. The approach can however easily accommodate more complex relationships at a higher computational cost, such as an additive kernel with all first-order interactions [5]:
$$
k (.,.) = \tau_ {1} \sum_ {i = 1} ^ {| \mathrm {P a} _ {X} |} k _ {\theta_ {i}} (.,.) + \tau_ {2} \sum_ {i = 1} ^ {| \mathrm {P a} _ {X} |} \sum_ {j = i + 1} ^ {| \mathrm {P a} _ {X} |} k _ {\theta_ {i}} (.,.) k _ {\theta_ {j}} (.,.). \tag {18}
$$
For each parent  $Z \equiv (\mathrm{Pa}_X)_i$  we used a squared exponential kernel function  $k_{\theta_i}(z,z') = \exp \left(-\frac{\|z - z'\|^2}{2\theta_i^2}\right)$ , with each  $\theta_i$  measuring the degree of non-linearity along the  $Z$ -th dimension. The kernel function has unit variance since the data are always normalized in the structure learning process. We assign the following independent prior distributions to the hyperparameter set  $\Theta = \{\mu, \sigma, \theta_i : i \in 1,\dots,|\mathrm{Pa}_X|\}$  of the GPN model (9):
$$
\theta_ {i} \sim \operatorname {I G} (2, 2), \quad \mu \sim \mathcal {N} (0, 1), \quad \sigma \sim \operatorname {I G} (1, 1). \tag {19}
$$
The inverse-gamma priors for each lengthscale  $\theta_{i}$  and the noise variance suppress values near zero, which in either case would result in overfitting and degenerate behaviour of samples of  $f$  [15]. The additional parameters  $\tau_{1}$  and  $\tau_{2}$  of the kernel function (18) were assigned independent IG(1, 1) priors.
4.2 Results
We compare two different versions of the GP-based structure learning scheme: the first one employs the order MCMC algorithm for sampling DAGs, and the second uses partition MCMC. We then compare both versions of the GPN sampling scheme with order and partition MCMC with the BGe score [28]. To reduce the computational burden of the simulations, we employ the simpler additive kernel [17] for the GP score.
As an additional benchmark, we include the DiBS+ algorithm [29], which models the adjacency matrix probabilistically, using particle variational inference to approximate a posteriori over structures. In the simulations, we parameterize DiBS+ by a neural network with one hidden layer with 5 nodes. We also consider methods which rely on the constraint-based PC algorithm [42], which
GP, partition GP, order
B
Figure 1: Distribution of  $\mathbb{E}$  -SHD values for all the different algorithms.  $\lambda = 0$  corresponds to linear-Gaussian data while higher values increase the degree of non-linearity of the relations among variables.
Ge, partition
BGe, order
kPC-HSIC
kPC-DC
DiBS+
learns a network by testing for conditional independence among the variables. Since these are not Bayesian methods, we bootstrap the data to obtain different graph estimates via the PC algorithm [10]. To account for the non-linear dependencies between the variables we apply the kernel PC (kPC) algorithm [17] to the resampled data using two different independence tests: HSIC [16] and distance correlation [46].
Figure 1 shows, for three values of  $\lambda$ , the distribution of  $\mathbb{E}-\mathrm{SHD}$  values for the different algorithms over the 100 generated structures. For non-linear data ( $\lambda = 1$ ), both versions of the GP samplers outperform existing methods: the median  $\mathbb{E}-\mathrm{SHD}$  for either of our algorithms is equal or lower than the bottom quartile of the BGe samplers. For linear ( $\lambda = 0$ ) and slightly non-linear data ( $\lambda = 0.5$ ), the GPN sampler performs competitively with the state-of-the-art BGe score-based sampling algorithms. All  $\mathbb{E}-\mathrm{SHD}$  values are computed with respect to DAGs; the results comparing completed partially directed acyclic graphs (CPDAGs) are available in figure A.3 in the supplementary material.
Figure 2 displays the run-times for the same methods as in figure 1. The run-times are stable across the different levels of non-linearity parameterized by  $\lambda$ , with the k-PC algorithms being the most computationally expensive, followed by the GP score-based approach, DiBS+ and finally the BGe score-based MCMC methods which are the fastest.
GP, partition GP, order
Figure 2: Distribution of run-times for all the different algorithms.  $\lambda = 0$  corresponds to linear-Gaussian data while higher values increase the degree of non-linearity of the relations among variables.
Figure 3: Left: reverse K-L divergence between the true posterior and the BGe posterior (green), the Laplace approximate posterior (blue) and the posterior obtained via importance sampling (red) as a function of the number of sampled DAGs. Right: the true posterior (gray) together with the BGe posterior (green), the Laplace approximate posterior (blue) and the posterior obtained via importance sampling (red). The majority of DAGs have a very low true posterior probability and are therefore never sampled by the MCMC algorithms (see inset).
Besides estimating accurately the structure, our approach can quantify the uncertainty in its estimates via the sampled graphs from the posterior distribution over GPNs. To evaluate the accuracy of the sampling approach in estimating the general posterior over structures, we compare its estimated posterior distribution over DAGs with the "true" posterior, obtained by enumerating every possible structure and computing its score directly with equation (12). Due to the exceedingly large number of DAGs, this approach is only feasible with small structures. The left panel of figure 3 shows the reverse Kullback-Leibler (K-L) divergence between the estimated and true posteriors for a given network with  $n = 4$  nodes, as a function of the number of samples  $M$  in the partition MCMC algorithm. The estimated ("GP, weighted") posterior probability  $p(\mathcal{G} \mid D)$  for a generic DAG  $\mathcal{G}$  is obtained by setting  $p(\Psi \mid \mathcal{G}_j) = \mathbb{1}_{(\mathcal{G}_j = \mathcal{G})}$  in equation (14). Reverse K-L divergence was chosen as a metric since the algorithms assign a probability of zero to DAGs that were not sampled.
The plot includes the divergence between the Laplace approximate posterior  $q(\mathcal{G} \mid D)$  in equation (14) and the true posterior, as well as between the posterior obtained with the BGe score and the true posterior. The divergence of the Laplace approximation is reduced by roughly one order of magnitude by weighting the samples via importance sampling. For reference, the DiBS+ algorithm yields a reverse K-L divergence of 27.8 with 1000 samples, i.e. two orders of magnitude higher than our approach, despite also being allocated longer run-times (see figure A.4 in the supplementary material). Sampling a higher number of graphs with DiBS+ quickly becomes infeasible since its run-time scales quadratically with the number of samples [29], while MCMC sampling scales linearly with  $M$ . The right panel of figure 3 shows the posterior distributions over the 543 possible DAGs, for  $M = 10^4$ , obtained after complete enumeration and by sampling with different methods. The plots confirm that as the number of sampled DAGs increases, our approach can accurately reflect the full posterior uncertainty. The results also underline the importance of sampling from the hyperparameters' priors to obtain an accurate representation of the posterior, as the Laplace approximation of the marginal likelihood results in a highly biased approximation even when sampling a large number of DAGs.
5 Application on Protein Signaling Networks
We also applied the GP score-based structure inference to the flow cytometry dataset of Sachs et al. [39] to learn protein signaling pathways. The authors provide an accepted consensus network, which we used as reference. We considered the first experimental condition, consisting of 853 single-cell observations of  $n = 11$  phosphoproteins and phospholipids in human T cells. The first three columns of table [1] show the performances of all the different algorithms in reconstructing the consensus network. The results include the GP model using the additive kernel (18) with all first-order interactions, denoted as  $\mathrm{GP}^2$ . We measure the different algorithms in terms of the E-SHD,
Table 1: Performance of the different algorithms in reconstructing the consensus network from flow cytometry data. The last two columns show the posterior probabilities of the two features experimentally validated by Sachs et al. [39], where the edge on the left should be present and the one on the right absent; up/down arrows indicate higher/ lower values are better.
E-SHD ↓	E-TP ↑	E-FP ↓	Erk → Akt ↑	Erk → PKA ↑
GP, partition	14.5	7.2	4.7	1	0.75
GP, order	14.6	6.8	4.4	1	0.42
GP2, partition	16.7	6.6	6.3	1	0
GP2, order	16.7	6.7	6.4	1	0.34
BGe, partition	15.9	4.3	3.2	0	1
BGe, order	15.3	4.4	2.7	0.17	0.98
kPC-HSIC	17	5.5	5.5	0.69	0.26
kPC-DC	16.9	6	5.8	0.72	0.28
DiBS+	16.1	5	4.2	0.45	0.7
as well as the  $\mathbb{E}$  -TP and  $\mathbb{E}$  FP, the absolute number of TP and FP edges in the DAGs weighted by the posterior, obtained by replacing the SHD in equation (16) by TP or FP, respectively.
One of the benefits of Bayesian structure inference is the possibility of deriving posterior probabilities of specific edges of interest in the network. In their work, Sachs et al. [39] experimentally tested two relationships between the proteins by intervening directly on the cells. By means of small interfering RNA inhibition, they concluded that the inhibition of Ekt has a direct effect on Akt, while there was a weaker and non-significant effect on PKA. We therefore expect an edge  $\mathrm{Erk} \rightarrow \mathrm{Akt}$  but no directed edge (or path) from Erk to PKA in the learned networks. The last two columns of table [1] display the posterior probabilities of these two features according to the different algorithms. GP- and BGe-based methods perform best, with the GP learner correctly assigning the highest probability to the edge  $\mathrm{Erk} \rightarrow \mathrm{Akt}$ , while the lack of the edge  $\mathrm{Erk}$  to PKA is predicted with a lower probability. The sparser BGe score-based methods assign a lower probability to the first edge, but correctly predict the absence of the second edge.
6 Conclusions
In this work, we have proposed a procedure that efficiently performs Bayesian structural inference on Gaussian process networks and estimates the posterior of generic features of interest. Building on the original GPN idea by Friedman and Nachman [9], we now embed it in a fully Bayesian framework. In particular, our approach involves placing priors on the hyperparameters of the score function and sampling DAGs via MCMC. Although more computationally expensive than a greedy search over the DAG space for a high-scoring network, the Bayesian approach allows one to accurately quantify the uncertainty in features of the network. This is made feasible by minimizing the number of scores to compute in an MCMC chain via importance sampling, as well as harnessing the advances in MC and MCMC methods [19, 28, 43, 45] that have been made since the introduction of GP networks.
The flexible nature of GPs allows the corresponding BNs to model a large range of functional dependencies between continuous variables. In the case of linear dependencies, our method remains competitive with state-of-the-art methods and exhibits desirable properties such as (approximate) score equivalence. The versatility of the model is particularly useful in domains where the nature of relations among variables is unknown and strict parametric assumptions on the distributions of the variables want to be avoided. Based on the promising simulation results and the convenient properties of the proposed method, we believe that it holds potential for making accurate inference on the underlying structure of BNs in complex domains.
The authors are grateful to acknowledge partial funding support for this work from the two Cantons of Basel through project grant PMB-02-18 granted by the ETH Zurich.
[37] Felix L. Rios, Giusi Moffa, and Jack Kuipers. Benchpress: a scalable and platform-independent workflow for benchmarking structure learning algorithms for graphical models. arXiv:2107.03863, 2021.
[38] Robert W. Robinson. Counting labeled acyclic digraphs. In New Directions in Graph Theory, pages 239-273. New York: Academic Press, 1973.
[39] Karen Sachs, Omar Perez, Dana Pe'er, Douglas A. Lauffenburger, and Garry P. Nolan. Causal protein-signaling networks derived from multiparameter single-cell data. Science, 308:523-529, 2005.
[40] Charupriya Sharma and Peter van Beek. Scalable Bayesian network structure learning with splines. In Proceedings of The 11th International Conference on Probabilistic Graphical Models, volume 186, pages 181-192, 2022.
[41] Tomi Silander, Teemu Roos, and Petri Myllymäki. Locally minimax optimal predictive modeling with Bayesian networks. In Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics, volume 5, pages 504-511, 2009.
[42] Peter Spirtes, Clark Glymour, and Richard Scheines. Causation, prediction, and search. Springer New York, 1993.
[43] Stan Development Team. RStan: the R interface to Stan, R package version 2.21.8, 2023. URL https://mc-stan.org/.
[44] Chengwei Su and Mark E. Borsuk. Improving structure MCMC for Bayesian networks through Markov blanket resampling. Journal of Machine Learning Research, 17:1-20, 2016.
[45] Polina Suter, Jack Kuipers, Giusi Moffa, and Niko Beerenwinkel. Bayesian structure learning and sampling of Bayesian networks with the R package BiDAG. Journal of Statistical Software, 105:1-31, 2023.
[46] Gábor J. Székely, Maria L. Rizzo, and Nail K. Bakirov. Measuring and testing dependence by correlation of distances. The Annals of Statistics, 35:2769-2794, 2007.
[47] Topi Talvitie, Aleksis Vuoksenmaa, and Mikko Koivisto. Exact sampling of directed acyclic graphs from modular distributions. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, pages 965-974, 2020.
[48] Ioannis Tsamardinos, Laura Brown, and Constantin Aliferis. The max-min hill-climbing Bayesian network structure learning algorithm. Machine Learning, 65:31-78, 2006.
[49] Jussi Viinikka, Antti Hyttinen, Johan Pensar, and Mikko Koivisto. Towards scalable Bayesian learning of causal DAGs. In Advances in Neural Information Processing Systems, volume 33, pages 6584-6594, 2020.
[50] Jackie S.T. Wong, Jonathan J. Forster, and Peter W.F. Smith. Properties of the bridge sampler with a focus on splitting the MCMC sample. Statistics and Computing, 30:799-816, 2020.
[51] Yue Yu, Jie Chen, Tian Gao, and Mo Yu. DAG-GNN: DAG structure learning with graph neural networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 7154-7163, 2019.
[52] Xun Zheng, Bryon Aragam, Pradeep K. Ravikumar, and Eric P. Xing. DAGs with NO TEARS: continuous optimization for structure learning. In Advances in Neural Information Processing Systems, volume 31, pages 9492-9503, 2018.