Title: A Bayesian Take on Gaussian Process Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: Bayesian networks

structure learning

graphical models

gaussian processes

Bayesian inference

MCMC sampling

importance sampling
Abstract: Gaussian Process Networks (GPNs) are a class of directed graphical models which employ Gaussian processes as priors for the conditional expectation of each variable given its parents in the network. The model allows the description of continuous joint distributions in a compact but flexible manner with minimal parametric assumptions on the dependencies between variables. Bayesian structure learning of GPNs requires computing the posterior over graphs of the network and is computationally infeasible even in low dimensions. This work implements Monte Carlo and Markov Chain Monte Carlo methods to sample from the posterior distribution of network structures. As such, the approach follows the Bayesian paradigm, comparing models via their marginal likelihood and computing the posterior probability of the GPN features. Simulation studies show that our method outperforms state-of-the-art algorithms in recovering the graphical structure of the network and provides an accurate approximation of its posterior distribution.

A Bayesian Take on Gaussian Process Networks

Abstract

Gaussian Process Networks (GPNs) are a class of directed graphical models which employ Gaussian processes as priors for the conditional expectation of each variable given its parents in the network. The model allows the description of continuous joint distributions in a compact but flexible manner with minimal parametric assumptions on the dependencies between variables. Bayesian structure learning of GPNs requires computing the posterior over graphs of the network and is computationally infeasible even in low dimensions. This work implements Monte Carlo and Markov Chain Monte Carlo methods to sample from the posterior distribution of network structures. As such, the approach follows the Bayesian paradigm, comparing models via their marginal likelihood and computing the posterior probability of the GPN features. Simulation studies show that our method outperforms state-of-the-art algorithms in recovering the graphical structure of the network and provides an accurate approximation of its posterior distribution.

1 Introduction

Bayesian networks (BNs) are a powerful tool for compactly representing joint distributions and the underlying relationships among a large set of variables [33]. These relationships are described via a directed acyclic graph (DAG), with each node in the graph representing a random variable. The joint distribution of a set of random variables X = { X 1 , ... , X n } factorizes into conditional distributions for each variable given its parents in the DAG:

The DAG provides a visual description of the dependency structure among the variables, where missing edges encode conditional independence relations among pairs of variables. Thanks to their inherent flexibility and their ability to combine expert knowledge with data, Bayesian networks have been applied to a large range of domains [2].

In the absence of full prior knowledge of the underlying graph, inference on the structure is necessary. For this purpose, a plethora of structure learning algorithms have been proposed, which involve either learning the structure from the conditional independence relations in the data (i.e. constraint-based) or assigning a score to each DAG and searching the graph space for high-scoring networks. A recent overview of current algorithms appears in Kitson et al. [23] while a reproducible workflow for

benchmark studies is available in Rios et al. [37]. Hybrid methods combining both constraint- and score-based aspects generally offer current state-of-the-art performance.

Typical scoring functions rely on measures of goodness of fit of the graphical model for the given data, such as the Bayesian Information Criterion (BIC) and the Mutual Information Test (MIT) [24, 3]. Another common choice is to use the graph marginal likelihood that we can derive by integrating over the parameter prior on a graphical model. The vast majority of score-based methods implement searches aiming to maximise the scoring function and return one single optimal structure. By complementing the marginal likelihood with a graph prior, one can score graphs according to their posterior distribution and sample from it. By fully characterizing the posterior distribution over graphs it is possible to go beyond optimal point estimates and enable Bayesian model averaging. In this context, approaches have emerged to capture both network and parameter uncertainties across a variety of research areas [27, 1, 32].

Likewise, we focus on Bayesian methods which sample DAGs according to their posterior distribution. Unlike approaches based on maximizing the score function, which return a single graph estimate, Bayesian approaches provide a full posterior distribution over DAGs, which we can use to perform inference on the network's features of interest. However, Bayesian structure inference typically requires a parametric model for the conditional distributions of each variable given its parents in the network to compute the posterior. Variations of the BDe score [21] are the usual choice for discrete-variable networks. Other information theory-based score functions [41, 3] are not Bayesian since they do not estimate a marginal likelihood but rather provide a measure of fitness of a DAG to the data based on the minimum description length principle.

In the case of continuous-variable networks, most of the current research has focused on the linearGaussian case, due to the availability of the closed-form BGe score [12, 26]. When relaxing the linear-Gaussian assumption, current approaches fall outside of the Bayesian framework since they do not target a posterior probability for DAGs but rather search the DAG space for a high-scoring network by minimizing a generic loss function. For example, Elidan [6] employs rank correlation as a goodness-of-fit measure between a graph and the observed data; Sharma and van Beek [40] propose modeling the relations among variables via regression splines and scoring the networks via a cross-validated score. Using the formulation by Zheng et al. [52] of the structure learning problem as a constrained continuous optimization, Yu et al. [51] model the conditional distributions via a graph neural network.

In this work, we develop a sampling scheme to perform fully Bayesian structure inference on generic continuous-variable networks with potentially non-linear relations among variables. We follow the strategy of Friedman and Nachman [9] and model the functional dependencies between each variable and its parents via Gaussian process priors. We extend their approach to the Bayesian framework by making structural inference based on the posterior distribution, which also involves treating the hyperparameters of the model as random. After a short review in Section 2 of Bayesian structure inference, Gaussian processes and how they can be used to parameterize BNs, Section 3 proposes a sampling scheme to perform fully Bayesian inference on nonlinear, continuous networks. In Sections 4 and 5 we evaluate and compare our algorithm to existing approaches on simulated and real data.

2 Background

2.1 Bayesian Structure Inference

When modeling a set of variables X with a Bayesian network, a typical objective is to estimate the probability of a generic feature of interest Ψ . Examples of such a feature are the presence and direction of certain edges, the topological ordering of nodes or conditional independence relations among variables. Given a complete set D of observations of X , we can obtain the posterior probability of Ψ by integrating the probability or presence of the feature over the posterior distribution of graphs:

wherein a key component is the likelihood of the data integrated over the prior distribution of the parameters θ of the conditional distributions:

The marginal likelihood p ( D | G ) for a given DAG G is available in closed form for specific combinations of priors on parameters and likelihood functions [11, 12].

The number of DAGs grows super-exponentially in the number of nodes [38], hence exact posterior inference is still exponentially complex, making it effectively intractable for large networks [47]. Approximate inference for p ( Ψ | D ) is possible if one can obtain samples G 1 , ... , G M from the posterior distribution over graphs p ( G | D ) .

Markov Chain Monte Carlo (MCMC) methods have successfully tackled the problem of posterior inference in the large and complex space of DAGs. The rationale consists of defining a Markov chain whose stationary distribution is the posterior distribution of interest p ( G | D ) . Madigan et al. [30] suggested a Metropolis-Hastings sampler: at each step, the algorithm proposes a new DAG G ′ and it accepts or rejects it according to a probability α . The value of α depends on the relative posterior probability p ( G ′ | D ) of the proposal graph compared to that of the current DAG:

where Q ( · , · ) are the transition probabilities from one DAG to another. In its original formulation, the algorithm proposes a new graph by adding or deleting one edge from the current DAG, with Q uniformly distributed in the set of neighbouring DAGs (including the current one).

Several refinements to the original method have been made over the years, providing for example more efficient proposal distributions [14, 20, 44]. The order MCMC approach by Friedman and Koller [8] introduced an important variation by sampling in the space of node orderings, which is smaller and smoother than the space of DAGs. Because of these attractive properties of the sample space, the order MCMC algorithm achieves superior mixing and convergence results compared to regular structure MCMC. Sampling orders however introduces a bias in the resulting posterior since node orders do not induce a uniform coverage in the DAG space [7]. The partition MCMC algorithm [25] corrects this bias in order MCMC by sampling in the larger space of ordered partitions of nodes to achieve unbiased samples from the posterior.

Current state-of-the-art methods rely on conditional independence tests to obtain a first coarse estimate of the structure to use as a starting point for a score-based method [48]. More sophisticated approaches involve iteratively expanding the search space to correct for errors in its initial estimate [28]. To achieve efficient posterior inference [49, 28] in the selected search space it helps to precompute all combinations of scores potentially needed in the chain so it can run at a very low computational cost.

Key to enabling efficient DAG sampling is decomposability , a property ensuring that we can express the posterior probability of a DAG G as a product of local scores that depend only on each variable and its parents in G :

Decomposability guarantees that we need to recompute only those scores whose parent sets have changed, or we can precompute all needed combinations of scores efficiently [28].

2.2 Gaussian Processes

Gaussian processes are a flexible tool used in machine learning for regression and classification tasks. Formally, a Gaussian Process (GP) is a distribution over functions such that every finite collection of its function values { f ( x 1 ) , f ( x 2 ) , ... , f ( x k ) } has a multivariate Gaussian distribution [36]. A GP is therefore fully specified by its mean function m ( x ) = E [ f ( x )] and covariance function k ( x, x ′ ) = Cov [ f ( x ) , f ( x ′ )] . Due to their ability to model a large range of functional behaviours, GPs find common use as priors over regression functions f ( X ) = E ( Y | X ) . A common GP regression model assumes independent Gaussian additive noise:

Notably, GP models admit a closed-form expression of the marginal likelihood, in this case, the likelihood of the N observations y marginalised over the prior distribution of f :

2.3 Gaussian Process Networks

Gaussian process networks (GPNs) refer to Bayesian networks whose conditional distributions are modeled via Gaussian process priors [9]. The structural equation model defining the distribution of each variable X i given its parents in a GPN is

where ε i has a normal distribution N ( µ, σ 2 ) independent of the data and a Gaussian process prior is placed on the function f i . Thanks to the nonparametric nature of GPs, the model in (8) can capture a wide range of functional dependencies between variables while maintaining the closed-form expression (7) for the marginal likelihood.

The covariance function k of the Gaussian process is typically parameterized by a set of hyperparameters θ which determine the behaviour of samples of f i such as smoothness, shape or periodicity. When these hyperparameters (together with the noise mean µ and variance σ 2 ) are unknown they are typically learned by maximizing the marginal likelihood. Since the marginal likelihood and its derivatives are available in closed form, the optimization can be performed efficiently via gradient ascent [36]. The GPN model enables structure learning of networks on continuous variables without the need to make strict parametric assumptions on the distributions of the variables. In scenarios even with low sample sizes, Friedman and Nachman [9] have shown that searching for the highest-scoring structure can accurately reconstruct the underlying DAG under different functional relationships.

When estimating the score of a GPN, the common approach to learning the hyperparameters by simple maximization can however lead to problematic estimates, since many local optima may exist [4]. In addition, the resulting plug-in estimate of the marginal likelihood would not correctly reflect the uncertainty in the data of the full posterior. Bayesian model averaging provides a natural solution by integrating over the prior distribution of the hyperparameters to obtain a true marginal likelihood p ( D | G ) , which we can then use to perform posterior inference on the structure.

3 Bayesian Structure Inference for GPNs

In this section, we describe a method to sample from the posterior distribution p ( Ψ | D ) of a generic feature Ψ of a GPN for continuous, non-linear data. To implement a fully Bayesian approach, we place priors over the hyperparameters θ of the kernel function and the Gaussian noise.

The priors ensure that the uncertainty in the functional relationships between each variable and its parents is fully accounted for. On the other hand, a maximum likelihood approach to estimating the hyperparameters could yield overly confident score functions and in turn misrepresent the posterior p ( G | D ) . Under the GPN model, the score remains decomposable, a necessary condition for its efficient evaluation. As in Section 2.1, making inference about network features Ψ of interest is possible by sampling graphs from the posterior:

Sampling graphs hinges on computing the scores of all variables given different combinations of their possible parent sets (see equation 5).

Let Θ = { µ, σ , θ } be the d -dimensional set of hyperparameters for a given node X and its parents Pa X . Unless stated otherwise, throughout the rest of the text we assume a uniform prior over all structures p ( G ) ∝ 1 . The score function is then the likelihood (7) of the observations x marginalized with respect to the hyperparameter priors:

If a variable X has no parents then the Gram matrix of the kernel is zero and the score function reduces to a Gaussian marginal likelihood. Since the above score function is generally intractable,

one option is to use Monte Carlo (MC) approaches such as bridge sampling [31] to approximate it. Bridge sampling employs a Gaussian proposal distribution g and a bridge function h chosen to minimize the MSE of the resulting estimator. The bridge sampling estimator of (11) is then defined as

The estimator is also a function of samples from the posterior of the hyperparameters p ( Θ | x, Pa X ) , which can easily be obtained via MCMC sampling. One can show that other approaches such as importance sampling or harmonic mean are special cases of bridge sampling [18]. MC methods based on bridge sampling provide consistent estimators for the marginal likelihood but may be biased for finite samples [50]. Nonetheless, such methods can become computationally expensive in high dimensions (i.e. for large parent sets).

Since sampling DAGs via MCMC requires computing a large number of scores of potential parent sets, which may not all be represented in the final sample, we avoid computing these expendable scores by first running the MCMC algorithm using a Laplace approximation of the score (11) around the MAP value of Θ :

We denote the resulting posterior probability of a DAG G from this Laplace approximated score as q ( G | D ) to distinguish it from the true posterior p ( G | D ) .

The Laplace approximate score provides an approximation of the posterior at a lower computational cost, speeding up considerably the running time of the MCMC algorithm used to sample graphs. After sampling M graphs in the first step with the Laplace approximate score, we can make inference with respect to the true posterior by re-computing the scores and performing importance sampling. To estimate the posterior probability of a feature of interest Ψ via importance sampling we evaluate where p ( G j | D ) and q ( G j | D ) are the posterior probabilities of DAG G j computed respectively with the bridge sampling MC estimate (12) and the Laplace approximation (13) for the score. The two posteriors do not need to be normalized for valid posterior inference on Ψ . This is because the normalizing constants in the importance sampling weights (14) cancel out. The procedure is summarized as pseudo-code in Algorithm 1 .

Re-computing the score functions in a second step and implementing importance sampling is computationally advantageous compared to running the MCMC algorithm directly with the MC estimates of the scores. The advantage is simply due to the number of unique scores in the final chain being much lower than those evaluated or needed during the chain itself (see figure A.5 in the supplementary material). Regardless of the MCMC algorithm used, we expect a substantial improvement in run-time compared to using the MC scores directly.

3.1 Implementation Details

Bayesian inference and optimization of the hyperparameters were performed via the Stan interface RStan [43]. The library offers a highly efficient C++ implementation of the No U-turn sampler [22], providing state-of-the-art posterior inference on the hyperparameters. We performed MC estimation of the marginal likelihood via the bridgesampling package [19], which can easily be combined with fitted Stan models.

In the implementation of Algorithm 1 we use order or partition MCMC to generate the network samples q ( G j | D ) ; the procedure can however accommodate a variety of sampling methods, as long as they result in samples from the posterior. Given its good performance in benchmarking studies [37], we use the recent BiDAG [45] hybrid implementation for MCMC inference on the graph. The hybrid sampler requires an initial search space which is then improved upon to correct for possible estimation errors [28]. As initial search space we take the output of the dual PC algorithm [13]. For the bridge sampling estimator (12) of the marginal likelihood we used N 1 = N 2 = 300 particles from the proposal and posterior distribution over the hyperparameters. The proposal function g was set to a normal distribution, with its first two moments chosen to match those of the posterior distribution. R code to implement Algorithm 1 and reproduce the results in Sections 4 and 5 is available at https://github.com/enricogiudice/LearningGPNs .

3.2 Score Equivalence

Without any assumptions on the functional form of structural models, observational data cannot generally distinguish between any two DAGs within the same Markov equivalence class [34], since the joint distribution always factorizes according to either DAG. Scoring functions like the BGe that assign the same score to all DAGs in the same class satisfy score equivalence. Imposing parametric models on the local probability distributions of a Bayesian network may however break score equivalence. Specifically for GPNs, an alternative factorization may not admit a representation that follows the structural equation model in (8). Consequently, DAGs belonging to the same Markov equivalence class may display different scores and become identifiable beyond their equivalence class. For this reason, the GP score generally differentiates models according to the direction of any of their edges.

Importantly, the identifiability of otherwise score equivalent DAGs is a direct consequence of the assumptions underpinning the GPN model. Because the functions f i are not generally invertible, the likelihood in (7) will assign higher values to functional dependencies that admit the GPN structural equation model. Further, Peters et al. [35] demonstrated that the asymmetry distinguishing between Markov equivalent factorizations holds beyond the case of non-invertible functions. Indeed, with the exception of the linear case, all structural models that take the form in (8) with additive Gaussian noise violate score equivalence and may identify a DAG beyond its Markov equivalence class. GPNs may encompass cases where equivalent DAGs are indistinguishable, such as when the functions f i are linear. In this case, the computed scores of equivalent DAGs will be similar, as long as the GP prior allows learning sufficiently linear relations. The numerical experiments in supplementary Section A.1 show that the GP-based score function displays near score equivalent behaviour when the data come from a joint Gaussian distribution, coherently with theoretical considerations [35].

4 Experimental Results

We evaluate the Bayesian GP network inference scheme on data generated from known random networks with n = 10 nodes. The DAG structures are constructed from an Erd˝ os-R´ enyi model, where each node has an independent probability of 0 . 2 to be connected with another with a higher topological ordering. For every node X i in each randomly generated network, we then sample 100 observations as a non-linear function of its parents. The nonlinear data are generated by transforming the parents' instances using a weighted combination of six Fourier components

The weights v i,j,k and w i,j,k are sampled from a Dirichlet distribution with concentration parameters equal to seven positive exponentially decreasing values γ k = e -k/ λ ∑ k e -k/ λ , for k = { 0 , ... , 6 } . The

parameter λ controls the rate of exponential decay, with values of λ close to zero providing mostly linear effects between variables and higher values resulting in increasingly non-linear relationships.

The edge coefficients β i,j determine the strength of the dependencies between X i and its parents and are sampled from a uniform distribution on ( -2 , -1 2 ) ∪ ( 1 2 , 2) ; the noise variable /epsilon1 i has a standard normal distribution. Instances of root nodes (nodes without parents) are also sampled from a standard normal. The linear-Gaussian case corresponds asymptotically to λ = 0 ; in this case, we set the weights v i,j,k and w i,j,k to zero except for w i,j, 0 .

We compare all structure learning algorithms in terms of structural Hamming distance (SHD) [48], which compares estimated graphs G with the ground truth graph G ∗ . Following Lorch et al. [29], we compute the average SHD of the samples weighted by the posterior:

The E -SHD summarizes how close the estimated posterior is to the true DAG; lower values are therefore desirable. It is however not a direct measure of the similarity of the estimated posterior to the true posterior.

4.1 Choice of Priors

Different choices of kernels for the GP prior result in different behaviours of the conditional expectation f in equation (9) of a variable X given its parents Pa X . A simple model employs an additive kernel

which corresponds to modeling each variable X as a sum of the individual contributions of each of its parents. The additive model serves to reduce the computational burden of calculating the scores by keeping the number of parameters as small as possible while preserving non-linearity in the model. The approach can however easily accommodate more complex relationships at a higher computational cost, such as an additive kernel with all first-order interactions [5]:

For each parent Z ≡ ( Pa X ) i we used a squared exponential kernel function k θ i ( z, z ′ ) = exp ( -|| z -z ′ || 2 2 θ 2 i ) , with each θ i measuring the degree of non-linearity along the Z -th dimension. The kernel function has unit variance since the data are always normalized in the structure learning process. We assign the following independent prior distributions to the hyperparameter set Θ = { µ, σ , θ i : i ∈ 1 , ... , | Pa X |} of the GPN model (9):

The inverse-gamma priors for each lengthscale θ i and the noise variance suppress values near zero, which in either case would result in overfitting and degenerate behaviour of samples of f [15]. The additional parameters τ 1 and τ 2 of the kernel function (18) were assigned independent IG (1 , 1) priors.

4.2 Results

We compare two different versions of the GP-based structure learning scheme: the first one employs the order MCMC algorithm for sampling DAGs, and the second uses partition MCMC. We then compare both versions of the GPN sampling scheme with order and partition MCMC with the BGe score [28]. To reduce the computational burden of the simulations, we employ the simpler additive kernel (17) for the GP score.

As an additional benchmark, we include the DiBS+ algorithm [29], which models the adjacency matrix probabilistically, using particle variational inference to approximate a posterior over structures. In the simulations, we parameterize DiBS+ by a neural network with one hidden layer with 5 nodes. We also consider methods which rely on the constraint-based PC algorithm [42], which

0

5

10

15

GP, partition

GP, order

BGe, partition

BGe, order

kPC

-

HSIC

kPC

-

DC

E

-

SHD

λ

=

1

0

5

10

15

GP, partition

GP, order

BGe, partition

BGe, order

kPC

-

HSIC

kPC

-

DC

E

-

SHD

λ

=

0.5

0

5

10

15

GP, partition

GP, order

BGe, partition

BGe, order

kPC

-

HSIC

kPC

-

DC

E

-

SHD

λ

=

0

GP, partition

GP, order

BGe, partition

BGe, order

kPC

-

HSIC

kPC

-

DC

learns a network by testing for conditional independence among the variables. Since these are not Bayesian methods, we bootstrap the data to obtain different graph estimates via the PC algorithm [10]. To account for the non-linear dependencies between the variables we apply the kernel PC (kPC) algorithm [17] to the resampled data using two different independence tests: HSIC [16] and distance correlation [46].

Figure 1 shows, for three values of λ , the distribution of E -SHD values for the different algorithms over the 100 generated structures. For non-linear data ( λ = 1 ), both versions of the GP samplers outperform existing methods: the median E -SHD for either of our algorithms is equal or lower than the bottom quartile of the BGe samplers. For linear ( λ = 0 ) and slightly non-linear data ( λ = 0 . 5 ), the GPN sampler performs competitively with the state-of-the-art BGe score-based sampling algorithms. All E -SHD values are computed with respect to DAGs; the results comparing completed partially directed acyclic graphs (CPDAGs) are available in figure A.3 in the supplementary material.

Figure 2 displays the run-times for the same methods as in figure 1. The run-times are stable across the different levels of non-linearity parameterized by λ , with the k-PC algorithms being the most computationally expensive, followed by the GP score-based approach, DiBS+ and finally the BGe score-based MCMC methods which are the fastest.

1

10

100

1000

GP, partition

GP, order

BGe, partition

BGe, order

kPC

-

HSIC

kPC

-

DC

Time (s)

λ

=

1

1

10

100

1000

GP, partition

GP, order

BGe, partition

BGe, order

kPC

-

HSIC

kPC

-

DC

Time (s)

λ

=

0.5

1

10

100

1000

GP, partition

GP, order

BGe, partition

BGe, order

kPC

-

HSIC

kPC

-

DC

Time (s)

λ

=

0

GP, partition

GP, order

BGe, partition

BGe, order

kPC

-

HSIC

kPC

-

DC

0

1

2

3

300

1000

3000

10000

Samples

Reverse K

-

L divergence

0.0

0.1

0.2

0

50

100

150

200

DAG

Posterior probability

0e+00

1e

-

09

2e

-

09

3e

-

09

4e

-

09

200

300

400

500

DAG

Posterior probability

BGe

GP, Laplace

GP, weighted

Besides estimating accurately the structure, our approach can quantify the uncertainty in its estimates via the sampled graphs from the posterior distribution over GPNs. To evaluate the accuracy of the sampling approach in estimating the general posterior over structures, we compare its estimated posterior distribution over DAGs with the 'true' posterior, obtained by enumerating every possible structure and computing its score directly with equation (12). Due to the exceedingly large number of DAGs, this approach is only feasible with small structures. The left panel of figure 3 shows the reverse Kullback-Leibler (K-L) divergence between the estimated and true posteriors for a given network with n = 4 nodes, as a function of the number of samples M in the partition MCMC algorithm. The estimated ('GP, weighted') posterior probability p ( G | D ) for a generic DAG G is obtained by setting p ( Ψ | G j ) = ( G j = G ) in equation (14). Reverse K-L divergence was chosen as a metric since the algorithms assign a probability of zero to DAGs that were not sampled.

The plot includes the divergence between the Laplace approximate posterior q ( G | D ) in equation (14) and the true posterior, as well as between the posterior obtained with the BGe score and the true posterior. The divergence of the Laplace approximation is reduced by roughly one order of magnitude by weighting the samples via importance sampling. For reference, the DiBS+ algorithm yields a reverse K-L divergence of 27 . 8 with 1000 samples, i.e. two orders of magnitude higher than our approach, despite also being allocated longer run-times (see figure A.4 in the supplementary material). Sampling a higher number of graphs with DiBS+ quickly becomes infeasible since its run-time scales quadratically with the number of samples [29], while MCMC sampling scales linearly with M . The right panel of figure 3 shows the posterior distributions over the 543 possible DAGs, for M = 10 4 , obtained after complete enumeration and by sampling with different methods. The plots confirm that as the number of sampled DAGs increases, our approach can accurately reflect the full posterior uncertainty. The results also underline the importance of sampling from the hyperparameters' priors to obtain an accurate representation of the posterior, as the Laplace approximation of the marginal likelihood results in a highly biased approximation even when sampling a large number of DAGs.

5 Application on Protein Signaling Networks

We also applied the GP score-based structure inference to the flow cytometry dataset of Sachs et al. [39] to learn protein signaling pathways. The authors provide an accepted consensus network, which we used as reference. We considered the first experimental condition, consisting of 853 single-cell observations of n = 11 phosphoproteins and phospholipids in human T cells. The first three columns of table 1 show the performances of all the different algorithms in reconstructing the consensus network. The results include the GP model using the additive kernel (18) with all first-order interactions, denoted as GP 2 . We measure the different algorithms in terms of the E -SHD ,

as well as the E -TP and E -FP , the absolute number of TP and FP edges in the DAGs weighted by the posterior, obtained by replacing the SHD in equation (16) by TP or FP, respectively.

One of the benefits of Bayesian structure inference is the possibility of deriving posterior probabilities of specific edges of interest in the network. In their work, Sachs et al. [39] experimentally tested two relationships between the proteins by intervening directly on the cells. By means of small interfering RNA inhibition, they concluded that the inhibition of Ekt has a direct effect on Akt, while there was a weaker and non-significant effect on PKA. We therefore expect an edge Erk → Akt but no directed edge (or path) from Erk to PKA in the learned networks. The last two columns of table 1 display the posterior probabilities of these two features according to the different algorithms. GP- and BGe-based methods perform best, with the GP learner correctly assigning the highest probability to the edge Erk → Akt, while the lack of the edge Erk to PKA is predicted with a lower probability. The sparser BGe score-based methods assign a lower probability to the first edge, but correctly predict the absence of the second edge.

6 Conclusions

In this work, we have proposed a procedure that efficiently performs Bayesian structural inference on Gaussian process networks and estimates the posterior of generic features of interest. Building on the original GPN idea by Friedman and Nachman [9], we now embed it in a fully Bayesian framework. In particular, our approach involves placing priors on the hyperparameters of the score function and sampling DAGs via MCMC. Although more computationally expensive than a greedy search over the DAG space for a high-scoring network, the Bayesian approach allows one to accurately quantify the uncertainty in features of the network. This is made feasible by minimizing the number of scores to compute in an MCMC chain via importance sampling, as well as harnessing the advances in MC and MCMC methods [19, 28, 43, 45] that have been made since the introduction of GP networks.

The flexible nature of GPs allows the corresponding BNs to model a large range of functional dependencies between continuous variables. In the case of linear dependencies, our method remains competitive with state-of-the-art methods and exhibits desirable properties such as (approximate) score equivalence. The versatility of the model is particularly useful in domains where the nature of relations among variables in unknown and strict parametric assumptions on the distributions of the variables want to be avoided. Based on the promising simulation results and the convenient properties of the proposed method, we believe that it holds potential for making accurate inference on the underlying structure of BNs in complex domains.

References

Federico Castelletti and Guido Consonni. Discovering causal structures in Bayesian Gaussian directed acyclic graph models. Journal of the Royal Statistical Society: Series A (Statistics in Society) , 183:1727-1745, 2020.

R´ on´ an Daly, Qiang Shen, and Stuart Aitken. Learning Bayesian networks: approaches and issues. The Knowledge Engineering Review , 26:99-157, 2011.

Luis M. de Campos. A scoring function for learning Bayesian networks based on mutual information and conditional independence tests. Journal of Machine Learning Research , 7: 2149-2187, 2006.

David Duvenaud, James Lloyd, Roger Grosse, Joshua Tenenbaum, and Ghahramani Zoubin. Structure discovery in nonparametric regression through compositional kernel search. In Proceedings of the 30th International Conference on Machine Learning , volume 28, pages 1166-1174, 2013.

David K. Duvenaud, Hannes Nickisch, and Carl Rasmussen. Additive Gaussian processes. In Advances in Neural Information Processing Systems , volume 24, pages 226-234, 2011.

Gal Elidan. Lightning-speed structure learning of nonlinear continuous networks. In Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics , volume 22, pages 355-363, 2012.

Byron Ellis and Wing Hung Wong. Learning causal Bayesian network structures from experimental data. Journal of the American Statistical Association , 103:778-789, 2008.

Nir Friedman and Daphne Koller. Being Bayesian about network structure: a Bayesian approach to structure discovery in Bayesian networks. Machine Learning , 50:95-125, 2003.

Nir Friedman and Iftach Nachman. Gaussian process networks. In Proceedings of the Sixteenth Conference on Uncertainty in Artificial Intelligence , pages 211-219, 2000.

Nir Friedman, Moises Goldszmidt, and Abraham Wyner. Data analysis with Bayesian networks: a bootstrap approach. In Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence , pages 196-205, 2013.

Dan Geiger and David Heckerman. Learning Gaussian networks. In Proceedings of the Tenth Conference on Uncertainty in Artificial Intelligence , pages 235-243, 1994.

Dan Geiger and David Heckerman. Parameter priors for directed acyclic graphical models and the characterization of several probability distributions. The Annals of Statistics , 30:1412-1440, 2002.

Enrico Giudice, Jack Kuipers, and Giusi Moffa. The dual PC algorithm and the role of Gaussianity for structure learning of Bayesian networks. International Journal of Approximate Reasoning , 161:108975, 2023.

Paolo Giudici and Robert Castelo. Improving Markov chain Monte Carlo model search for data mining. Machine Learning , 50:127-158, 2003.

Robert B. Gramacy and Herbert K. H. Lee. Bayesian treed Gaussian process models with an application to computer modeling. Journal of the American Statistical Association , 103: 1119-1130, 2008.

Arthur Gretton, Kenji Fukumizu, Choon Teo, Le Song, Bernhard Sch¨ olkopf, and Alex Smola. A kernel statistical test of independence. In Advances in Neural Information Processing Systems , volume 20, pages 585-592, 2007.

Arthur Gretton, Peter Spirtes, and Robert Tillman. Nonlinear directed acyclic structure learning with weakly additive noise models. In Advances in Neural Information Processing Systems , volume 22, pages 1847-1855, 2009.

Quentin F. Gronau, Alexandra Sarafoglou, Dora Matzke, Alexander Ly, Udo Boehm, Maarten Marsman, David S. Leslie, Jonathan J. Forster, Eric-Jan Wagenmakers, and Helen Steingroever. A tutorial on bridge sampling. Journal of Mathematical Psychology , 81:80-97, 2017.

Quentin F. Gronau, Henrik Singmann, and Eric-Jan Wagenmakers. Bridgesampling: An R package for estimating normalizing constants. Journal of Statistical Software , 92:1-29, 2020.

Marco Grzegorczyk and Dirk Husmeier. Improving the structure MCMC sampler for Bayesian networks by introducing a new edge reversal move. Machine Learning , 71:265-305, 2008.

David Heckerman and Dan Geiger. Learning Bayesian networks: a unification for discrete and Gaussian domains. Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence , pages 274-284, 1995.

Matthew D. Homan and Andrew Gelman. The No-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research , 15:1593-1623, 2014.

Neville Kitson, Anthony Constantinou, Guo Zhigao, Yang Liu, and Kiattikun Chobtham. A survey of Bayesian network structure learning. Artificial Intelligence Review , 56:1-94, 2023.

Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques adaptive computation and machine learning . The MIT Press, 2009.

Jack Kuipers and Giusi Moffa. Partition MCMC for inference on acyclic digraphs. Journal of the American Statistical Association , 112:282-299, 2017.

Jack Kuipers, Giusi Moffa, and David Heckerman. Addendum on the scoring of Gaussian directed acyclic graphical models. The Annals of Statistics , 42:1689-1691, 2014.

Jack Kuipers, Thomas Thurnherr, Giusi Moffa, Polina Suter, Jonas Behr, Ryan Goosen, Gerhard Christofori, and Niko Beerenwinkel. Mutational interactions define novel cancer subgroups. Nature Communications , 9:4353, 2018.

Jack Kuipers, Polina Suter, and Giusi Moffa. Efficient sampling and structure learning of Bayesian networks. Journal of Computational and Graphical Statistics , 31:639-650, 2022.

Lars Lorch, Jonas Rothfuss, Bernhard Sch¨ olkopf, and Andreas Krause. DiBS: differentiable Bayesian structure learning. In Advances in Neural Information Processing Systems , volume 34, pages 24111-24123, 2021.

David Madigan, Jeremy York, and Denis Allard. Bayesian graphical models for discrete data. International Statistical Review , 63:215-232, 1995.

Xiao-Li Meng and Wing Hung Wong. Simulating ratios of normalizing constants via a simple identity: a theoretical exploration. Statistica Sinica , 6:831-860, 1996.

Giusi Moffa, Jack Kuipers, Giuseppe Carr` a, Cristina Crocamo, Elizabeth Kuipers, Matthias Angermeyer, Traolach Brugha, Mondher Toumi, and Paul Bebbington. Longitudinal symptomatic interactions in long-standing schizophrenia: a novel five-point analysis based on directed acyclic graphs. Psychological Medicine , pages 1-8, 2021.

Judea Pearl. Probabilistic reasoning in intelligent systems: networks of plausible inference . Morgan Kaufmann Publishers Inc., 1988.

Judea Pearl. Causality: Models, reasoning, and inference . Cambridge University Press, 2000.

Jonas Peters, Joris Mooij, Dominik Janzing, and Bernhard Sch¨ olkopf. Causal discovery with continuous additive noise models. Journal of Machine Learning Research , 15:2009-2053, 2013.

Carl Edward Rasmussen. Gaussian processes in machine learning . Springer Berlin Heidelberg, 2004.

Felix L. Rios, Giusi Moffa, and Jack Kuipers. Benchpress: a scalable and platformindependent workflow for benchmarking structure learning algorithms for graphical models. arXiv:2107.03863 , 2021.

Robert W. Robinson. Counting labeled acyclic digraphs. In New Directions in Graph Theory , pages 239-273. New York: Academic Press, 1973.

Karen Sachs, Omar Perez, Dana Pe'er, Douglas A. Lauffenburger, and Garry P. Nolan. Causal protein-signaling networks derived from multiparameter single-cell data. Science , 308:523-529, 2005.

Charupriya Sharma and Peter van Beek. Scalable Bayesian network structure learning with splines. In Proceedings of The 11th International Conference on Probabilistic Graphical Models , volume 186, pages 181-192, 2022.

Tomi Silander, Teemu Roos, and Petri Myllym¨ aki. Locally minimax optimal predictive modeling with Bayesian networks. In Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics , volume 5, pages 504-511, 2009.

Peter Spirtes, Clark Glymour, and Richard Scheines. Causation, prediction, and search . Springer New York, 1993.

Stan Development Team. RStan: the R interface to Stan, R package version 2.21.8, 2023. URL https://mc-stan.org/ .

Chengwei Su and Mark E. Borsuk. Improving structure MCMC for Bayesian networks through Markov blanket resampling. Journal of Machine Learning Research , 17:1-20, 2016.

Polina Suter, Jack Kuipers, Giusi Moffa, and Niko Beerenwinkel. Bayesian structure learning and sampling of Bayesian networks with the R package BiDAG. Journal of Statistical Software , 105:1-31, 2023.

G´ abor J. Sz´ ekely, Maria L. Rizzo, and Nail K. Bakirov. Measuring and testing dependence by correlation of distances. The Annals of Statistics , 35:2769-2794, 2007.

Topi Talvitie, Aleksis Vuoksenmaa, and Mikko Koivisto. Exact sampling of directed acyclic graphs from modular distributions. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference , pages 965-974, 2020.

Ioannis Tsamardinos, Laura Brown, and Constantin Aliferis. The max-min hill-climbing Bayesian network structure learning algorithm. Machine Learning , 65:31-78, 2006.

Jussi Viinikka, Antti Hyttinen, Johan Pensar, and Mikko Koivisto. Towards scalable Bayesian learning of causal DAGs. In Advances in Neural Information Processing Systems , volume 33, pages 6584-6594, 2020.

Jackie S.T. Wong, Jonathan J. Forster, and Peter W.F. Smith. Properties of the bridge sampler with a focus on splitting the MCMC sample. Statistics and Computing , 30:799-816, 2020.

Yue Yu, Jie Chen, Tian Gao, and Mo Yu. DAG-GNN: DAG structure learning with graph neural networks. In Proceedings of the 36th International Conference on Machine Learning , volume 97, pages 7154-7163, 2019.

Xun Zheng, Bryon Aragam, Pradeep K. Ravikumar, and Eric P. Xing. DAGs with NO TEARS: continuous optimization for structure learning. In Advances in Neural Information Processing Systems , volume 31, pages 9492-9503, 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGaussian Process (GP) networks are directed graphical models for continuous data, where the function mapping from parent node values to parameters of child node is a Gaussian process. Given the graphical network structure and a dataset of observations, learning the GP model for each node simply reduces to a standard supervised GP regression using parents nodes as input $x$ values and child node as output $y$ values. Likewise marginal likelihood evaluation and hyperparameter optimization/marginalization can use the standard methods for GP regression.\n\nIn this work, as I understand, the authors goal is to marginalise over network structures, each network structure yields a different set of parents-child links and each one requires marginalizing GP hyperparameters again, if done naively, this is very very expensive. The authors thus propose methods to speed up this process hence making network structure marginalisation much more computationally efficient. These methods include\n- using Metropolis Hasting like algorithms to make small steps through the space of graphs thereby removing the need for excessive recomputation.\n- for each graph in the Metropolis Hastings proposal, the authors use a Laplace approximation to marginalize GP hyperparameters instead of full MCMC marginalisation.\n- once a dataset of sampled graphs is collected, the expensive MCMC marginalisation is used to weight each sampled graph with importance sampling.\n\n\n\nExperiments on both synthetic toy example and a protein interaction network example are presented.\n\nSTRENGTHS:\n- I feel that the proposed methods are very standard approaches and I feel are well justified in this use case and the overall model and algorithm is is well designed.\n- personally coming from a traditional GP background, I felt this paper was accessible and easy to digest.\n\nWEAKNESSES:\n- Everything from line 101 up until line 159 is focused on model fitting and hyperparameter marginalization for a single GP and, as far as I can tell, has nothing to do with networks, hence the section title felt a bit misleading and rearranging material might make this easier. Equations 8 - 12 could hypothetically be written using $x, y$ instead of $Pa_{X}, X$ and moved to Section 2. Then section 3 could contain a description about how graph structure determines each nodes \"local GP training set\" and how the model is a summation over GPs over individual training sets. and the fact that there are so many justifies a hyper-parameter approximation.\n- Equation (10) appears to be a standard marginal likelihood with an extra layer of marginalisation over kernel hyperparameters (with or without hyperpriors), this is standard practice in GP regression [Code here](https://github.com/rmgarnett/gpml_extensions/) and [this paper](https://proceedings.neurips.cc/paper/2010/hash/4d5b995358e7798bc7e9d9db83c612a5-Abstract.html). The authors cite bridge sampling which I am not personally familiar with, but there exist many, many algorithms to solve this problem (this is not a weakness of the paper as any MCMC algorithm will suffice, but at first glance seems a rather strange choice)\n\n- In Section 4, the authors repeatedly state that $\\lambda =0$ removes higher frequencies and is a linear case and so $w_{i,0}$ is the only non-zero coefficient. If I take equation 14 and make the suggested substitution, I have a constant with noise\n$$\nX_i = \\sum_{Z\\in Pa_{X_i}} \\beta_i w_{i,0} + \\epsilon_i\n$$\nWhich implies no dependency between $X$ and $Z$. (The only way I could see a linear relationship between $X$ and $Z$ is if $\\sin(jZ)$ for very small $j$ where $\\sin()$ is approximately linear near the origin, but given normalised $Z$ and $j=1$ this isn't likely). Hence the $\\lambda=0$ case is generating data that ignores graph structure, and the hamming distance is meaningless as the models will learn the empty graph from the data.\n\nQUESTIONS:\n- as mentioned above regarding $\\lambda=0$, is my understanding correct? Can the authors comment on this?\n- Figure 1: I assume the graph hamming distance shows high \"tight\" the posterior over graphs is clustered around the true graph, tighter is better, whereas with 100 observations in 10D space, this seems very sparse dataset to learn any complex model and I would expect the posterior to be rather broad. What is the hamming distance of uniformly randomly generated graphs? What is the average hamming distance of the \"true\" posterior from the true generating graph?\n\nLIMITATIONS:\n- the authors provide an explanation of the Score equivalence issue (which hadn't occurred to me until it was mentioned)", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Bayesian structure learning of the GPNs framework that is claimed to be less computationally. To address this, the approach presented in this work utilizes Monte Carlo and importance sampling to sample from the posterior distribution of network structures. This approach compares models using their marginal likelihood and computes the posterior probability of GPN features. Some simulations have been discussed.\n\nSTRENGTHS:\n- This paper studies a rather important problem.\n- It is easy to follow the thoughts.\n\nWEAKNESSES:\n- The contribution of the paper is not clearly justified and seems moderate at best. It is a combination of a few approaches (Gaussian processes, Laplace approximation, and Importance sampling).\n- The claims of the paper are not completely justified. It provides an approach but does not discuss how the main concerns are addressed. It is not clear how efficient this method is relative to others. \n- Some references are missing, for instance: \n     * Efficient and scalable structure learning for Bayesian networks: algorithms and applications by Zhu 2020\n     * Efficient structure learning of Bayesian networks using constraints by de Campos \n     * MCMC algorithms for Bayesian analysis of phylogenetic trees\n- Lack of comparison/validation to other state-of-the-art algorithms and also lack of discussion on how efficient this framework is by providing computational intensity analysis. \n- Lack of theoretical discussion regarding the convergence and issues rising from Laplace approximation and Importance sampling. \n- Lack of validation, sensitivity analysis and comparison to alternative models.\n\nQUESTIONS:\nA few questions that come to mind: \n- Laplace approxiamtion is sensitive to the MAP estiamte. as the score s are being approximated this may cause poor approxiamtion and bias and thus it may impact the perfomrance of the subsequenct imporatnce sampling and causing high variance of imporance weights and thus inefficient sampling and inaccurate estmation. could the paper elaborate on this issue. \n- Imporatnce sampling can introduce bias if the proposal does not adequetly cover the suppot of the target posterior. Also, high variance of the weights can lead to unstable estiamtes. \n- Althogh use of Lapace approxiamtion and imporatance sampling may in general reduce the computaional intensity  but overall performance dependes on model and the size of dataset (specifically when approxiamting the score and so on). It is well-known that as dimension of the netwrok increases or dataset grows larger the computational cost of Laplace approxiamtion and imporatnece samling can be significant. How is the relationship between effective sample size, variabaility of the weights and growth of the netwrok, dataset, and dimentionality justified?\n\nLIMITATIONS:\n- Laplace approximation is only valid if the distribution is Gaussian or close to Gaussian. This assumption can simply be violated. \n- Lack of discussion on computational intensity and how this framework overcomes those known computational issues. \n- Lack of validation, sensitivity analysis and comparison to alternative models.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes methodology to perform Bayesian inference on the (hyper) parameters of a so-called Gaussian Process Networks (GPNs), which are sets of functional equations with Gaussian Process (GP) priors on the functions relating a variable to its parents, as well as inference on the graph structure and graph posterior expectations.\nPrevious works in the GPN literature that consider a posterior over graphs did not put priors over hyperparameters of the GP kernels and perform associated posterior inference. \nTo address the challenges, the authors crucially exploit the decomposability property of p(G | D): \" scores \" (conditionals of X_i given PA_i ) only need to be recomputed if the parents have changed, so that scores can be reused across different graphs. \nSince scores are intractable integrals over (hyper) parameters of the GPs, the authors use bridge sampling to approximate the integrals, and make use of Laplace approximations  in a two-step procedure to reduce computational cost compared to running MCMC directly with the estimated scores. \nThe methodology is evaluated on a good set of synthetic and real-data examples.\n\nSTRENGTHS:\nOverall a technically solid paper on an interesting topic for the NeurIPS community. Main strengths are as follows: \n- GPNs are gaining more attention in particular due to their application in causal structure learning (von Kügelgen et al 2019), so the authors address a relevant problem of performing \"fully\" Bayesian inference with these models \n- The scope and objective of the paper and proposed methods are clear and targeted \n- The paper demonstrates an excellent knowledge of the surrounding relevant literature on MCMC for structure learning, and describes the relationship with the current work clearly \n- Well explained and solid experimental section\n\nWEAKNESSES:\n- [**Presentation of Section 3**] The main weakness of the paper is the presentation of the approximations in Section 3. Even as a reader well familiar with bridge sampling, Laplace approximations and MCMC, the part from lines 143 to 172 is unclear at first read and too condensed. I am guessing the final goal is to approximate p( \\Phi | D) of Eq (13). Rather than start from the various approximations needed to get there, I would suggest starting from there and write down the integrals that arise, then explain how to approximate each step. Otherwise, while I'm reading I keep thinking \"Why am I approximating the scores ? What is the final goal ?\". Some related questions/comments about clarity of this part below.\n    - Is the reason that you approximate the scores both **(a)** via Laplace  **and**  **(b)** via bridge sampling because  **(a)** is used to build the proposal while  **(b)** is needed to compute the (self-normalized) estimator in Eq. (13), specifically, the unnormalized target ? If so you should note that the estimator in Eq (13) has an **additional bias** ( in addition to the standard bias coming from self-normalized IS)  due to the fact that the bridge sampling estimator is not unbiased. You should also state that because you are using self-normalized IS, you do not need to know the normalizing constants of either q(G|D) or p(G|D), as the ratio of these two constants cancels out. \n- [**Difficulty introduced by the additional marginalization**]: while usually people assume there is inherent interest in a \"fully\" Bayesian treatment and marginalize hyperparameters as well when feasible (in this case, kernel hyperparameters), the authors should make it clearer why it is beneficial to do so specifically in the setting of GPNs with interest in p(G|D). There is a comment about this in lines 290-293 but it is quite vague.\n\nQUESTIONS:\n- How is the Gaussian proposal for the numerator of Eq. (11) chosen ? I would have expected a Laplace - IS here too ? Perhaps clarify what is the choice made here and possibly why.    \n - Could you comment theoretically/intuition-wise/experimentally on which of these approximations are the ones that affects accuracy most and how ? For example, should we focus on getting better proposals $q(G|D)$ or in approximating the scores with MC (bridge sampling, in your case) more efficiently to approximate the target (numerator of the IS weights, in IS terminology) better ? Further, could you provide a couple of example $\\Phi$ of interest already in the methodology (Section 3) for concreteness ? \n    - The comment in lines 290-293 is important in my view, but should be expanded upon. Why/how does marginalizing the hyperparameters mitigate the bias of the Laplace approximation? Why can we not \"just\" not marginalize them and simply use more MCMC samples ? \n- Regarding score equivalency, could you clarify why it sounds like it's not good if DAGs from the same MEC obtain different scores ? Does the property make it harder to approximate the true $p(G|D)$ somehow ?  \n- Does any Bernstein von Mises - type result hold for $p(G|D)$ ?\n\nLIMITATIONS:\nThe authors do not have an explicit limitations subsection, although limitations are not hard to infer from the text, including mainly computational cost of marginalizing the hyperparameters, and the unclear effects on the performance of the approximations performed by bridge sampling; it is also not clear whether the authors can use kernels appropriate for with non-stationary functions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an MCMC algorithm for Bayesian inference for Gaussian Process Networks where you model the distribution of a node as a function of its parents plus noise, with the function a Gaussian process.\n\nThe sampler uses a Laplace approximation to make informed moves between Networks.\n\nThe sampler is demonstrated on a simulated and real example.\n\nSTRENGTHS:\nThis is a fine paper -- which presents a sensible method for a general learning problem. It compares with some other methods and empirically gives some evidence of better performance.\n\nThe presentation is generally good.\n\nWEAKNESSES:\nThe ideas come across as rather incremental -- taking standard ideas and putting them together.\n\nThe question then becomes how important and useful is the resulting method. Here is where I think the paper is a bit lacking. The empirical study is somewhat limited. Also, with their method there a numerous tuning parameters (i.e. the specification of the prior) -- and it is unclear how robust their results are to different choices of the priors. It would also be good to see some investigation of when and why the method works well relative to other methods. I can see that the Bayesian paradigm potentially gives advantages in terms of averaging over model uncertainty and being able to quantify uncertainty -- but these come with certain disadvantages (potential dependence on prior; extra computational cost; how reliable are measures of uncertainty when the model is incorrect).\n\nThe results could be a bit clearer -- e.g. more information captions; make it clearer the E-SHD is smaller for better methods; perhaps clearer about which method is which.\n\nIs there any code available for the method?\n\nQUESTIONS:\nIt would be helpful to see the robustness of the results of your method as you vary the prior.\n\nIt would be good to see more thorough evaluation of the methods -- in particularly looking at understanding when the method works well/when it does not.\n\nIt would be helpful to have (easy to use) code available and details of how to reproduce the results.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWithin the broader task of causal network structure identification from observational data, i.e. finding the probabilistic graphical model which best explains (has highest likelihood) observations of a set of variables, this paper focus on a type of network known as GPN. GPNs are adapted to scenarios where all variables are continuous, and are built from a simple building block in which every child random variable's distribution, conditional on its parents, is a GP regression with independent Gaussian noise. For this task and this type of network, the paper proposes a fully Bayesian inference procedure based on MC and MCMC to sample PGM structure (i.e. edge presence and direction), and compute graph-related statistics of interest (such as the probability of presence of an edge between two given variables). This procedure holds the promise of more accurately representing the graph structure distribution than score-based methods.\n\nSTRENGTHS:\nOriginality: The paper's approach consists of \"pushing through\" with a strong MCMC sampling procedure for the stated problem. The problem statement is not original per se, but there is moderate inventivity in putting together the required technical steps to overcome each hurdle: speeding up eq 5 (graph posterior) thanks to eq 12 (Laplace approximation of per-variable score), importance sampling for feature posteriors in eq 13. Originality is not the paper's strong suit however.\nQuality: Derivations and implementations are all correct. Experimental validation on synthetic data is a necessity, especially where full enumeration of network structures is needed; the validation on the Sachs dataset is interesting, though one would wish for further real-data experiments. I appreciate that score equivalence/graph identifiability is addressed thoroughly. Evaluation methods and prior choices seem well-motivated and comparable to other literature.\nClarity: The paper is exceptionally clear. Citations are almost all accurately chosen (with the exception of a few in the introduction, cf below). \nSignificance: The paper tackles its stated problem heads-on. I believe we will see further Bayesian takes on network structure inference, maybe with smarter graph posterior approximations or even variational approaches for certain types of networks. The choice of GPN (and the particular further choices made for experimental evaluation) is rather flexible as it supports varied functional dependencies of child upon parent variables. The experimental evaluation places variants (order/partition) of the paper's methods at better or on par with BGe, and better than other methods. It is clearly superior in its sampling behaviour (fig 2). Its running time, unfortunately hidden in the supplementary material fig A8, is 30x that of BGe however, which might restrict applicability.\n\nWEAKNESSES:\nSignificance, as discusse above. \nMore real-world experiments would help assess the impact, in particular if they were analysed with the level of detail of synthetic experiments sec4.\n\nMinor errors and typos I found\n- line 162: perform, not make\n- line 208: data is generated\n- line 266: define CPDAG\n- line 289: increase, our approach\n- lines 24 and 29, citations seem quite cherry-picked and do not provide a variety of viewpoints as I would expect; I would hope there are no biased self-citations here.\n- supplementary line 479: do you mean TP instead of TN? eq 19, what does the lowercase p stand for in FPRp?\n- table 1: I would add the usual arrow-up, arrow-down to indicate \"higher/lower is better\" for easier reading\n\nQUESTIONS:\n- To make the point lines 168 ff., would it be worth counting the number of score evaluations?\n- Do some real-world scenarios require a different choice from the uniform prior over graph structures (line 141) ?\n- Table 1, rightmost column: can the poor performance of your method be explained by the fact that there is no reason to predict edge absence, when absence of correlation can be captured with a fairly wide conditional distribution, i.e. low covariance in GP?\n\nLIMITATIONS:\nThe limitations tied to algorithm choices are well discussed. It is worth motivating better why it is reasonable to focus on GPNs with respect to real-world needs, where all variable might not be continuous, where GP conditional distributions might be too loose, or where neural-network-parameterized conditional distributions might provide a more adequate model.\n\nI see no interesting limitations to discuss related to applications or societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-14"}