SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Enrico Giudice
Dep. of Mathematics and Computer Science
University of Basel, Basel, Switzerland
enrico.giudice@unibas.ch
Jack Kuipers
Dep. of Biosystems Science and Engineering
ETH Zurich, Basel, Switzerland
jack.kuipers@bsse.ethz.ch
Giusi Moffa
Dep. of Mathematics and Computer Science, University of Basel, Basel, Switzerland
and Division of Psychiatry, University College London, London, UK
giusi.moffa@unibas.ch
Gaussian Process Networks (GPNs) are a class of directed graphical models which employ Gaussian processes as priors for the conditional expectation of each variable given its parents in the network. The model allows the description of continuous joint distributions in a compact but flexible manner with minimal parametric assumptions on the dependencies between variables. Bayesian structure learning of GPNs requires computing the posterior over graphs of the network and is computationally infeasible even in low dimensions. This work implements Monte Carlo and Markov Chain Monte Carlo methods to sample from the posterior distribution of network structures. As such, the approach follows the Bayesian paradigm, comparing models via their marginal likelihood and computing the posterior probability of the GPN features. Simulation studies show that our method outperforms state-of-the-art algorithms in recovering the graphical structure of the network and provides an accurate approximation of its posterior distribution.
Bayesian networks (BNs) are a powerful tool for compactly representing joint distributions and the underlying relationships among a large set of variables [33]. These relationships are described via a directed acyclic graph (DAG), with each node in the graph representing a random variable. The joint distribution of a set of random variables  $\mathbf{X} = \{X_1, \dots, X_n\}$  factorizes into conditional distributions for each variable given its parents in the DAG:
$$
p (\mathbf {X}) = \prod_ {i = 1} ^ {n} p \left(X _ {i} \mid \operatorname {P a} _ {X _ {i}}\right). \tag {1}
$$
The DAG provides a visual description of the dependency structure among the variables, where missing edges encode conditional independence relations among pairs of variables. Thanks to their inherent flexibility and their ability to combine expert knowledge with data, Bayesian networks have been applied to a large range of domains [2].
In the absence of full prior knowledge of the underlying graph, inference on the structure is necessary. For this purpose, a plethora of structure learning algorithms have been proposed, which involve either learning the structure from the conditional independence relations in the data (i.e. constraint-based) or assigning a score to each DAG and searching the graph space for high-scoring networks. A recent overview of current algorithms appears in Kitson et al. [23] while a reproducible workflow for
benchmark studies is available in Rios et al. [37]. Hybrid methods combining both constraint- and score-based aspects generally offer current state-of-the-art performance.
Typical scoring functions rely on measures of goodness of fit of the graphical model for the given data, such as the Bayesian Information Criterion (BIC) and the Mutual Information Test (MIT) [24, 3]. Another common choice is to use the graph marginal likelihood that we can derive by integrating over the parameter prior on a graphical model. The vast majority of score-based methods implement searches aiming to maximise the scoring function and return one single optimal structure. By complementing the marginal likelihood with a graph prior, one can score graphs according to their posterior distribution and sample from it. By fully characterizing the posterior distribution over graphs it is possible to go beyond optimal point estimates and enable Bayesian model averaging. In this context, approaches have emerged to capture both network and parameter uncertainties across a variety of research areas [27, 1, 32].
Likewise, we focus on Bayesian methods which sample DAGs according to their posterior distribution. Unlike approaches based on maximizing the score function, which return a single graph estimate, Bayesian approaches provide a full posterior distribution over DAGs, which we can use to perform inference on the network's features of interest. However, Bayesian structure inference typically requires a parametric model for the conditional distributions of each variable given its parents in the network to compute the posterior. Variations of the BDe score [21] are the usual choice for discrete-variable networks. Other information theory-based score functions [41, 3] are not Bayesian since they do not estimate a marginal likelihood but rather provide a measure of fitness of a DAG to the data based on the minimum description length principle.
In the case of continuous-variable networks, most of the current research has focused on the linear-Gaussian case, due to the availability of the closed-form BGe score [12, 26]. When relaxing the linear-Gaussian assumption, current approaches fall outside of the Bayesian framework since they do not target a posterior probability for DAGs but rather search the DAG space for a high-scoring network by minimizing a generic loss function. For example, Elidan [6] employs rank correlation as a goodness-of-fit measure between a graph and the observed data; Sharma and van Beek [40] propose modeling the relations among variables via regression splines and scoring the networks via a cross-validated score. Using the formulation by Zheng et al. [52] of the structure learning problem as a constrained continuous optimization, Yu et al. [51] model the conditional distributions via a graph neural network.
In this work, we develop a sampling scheme to perform fully Bayesian structure inference on generic continuous-variable networks with potentially non-linear relations among variables. We follow the strategy of Friedman and Nachman [9] and model the functional dependencies between each variable and its parents via Gaussian process priors. We extend their approach to the Bayesian framework by making structural inference based on the posterior distribution, which also involves treating the hyperparameters of the model as random. After a short review in Section 2 of Bayesian structure inference, Gaussian processes and how they can be used to parameterize BNs, Section 3 proposes a sampling scheme to perform fully Bayesian inference on nonlinear, continuous networks. In Sections 4 and 5 we evaluate and compare our algorithm to existing approaches on simulated and real data.
2.1 Bayesian Structure Inference
When modeling a set of variables  $\mathbf{X}$  with a Bayesian network, a typical objective is to estimate the probability of a generic feature of interest  $\Psi$ . Examples of such a feature are the presence and direction of certain edges, the topological ordering of nodes or conditional independence relations among variables. Given a complete set  $D$  of observations of  $\mathbf{X}$ , we can obtain the posterior probability of  $\Psi$  by integrating the probability or presence of the feature over the posterior distribution of graphs:
$$
p (\Psi \mid D) = \sum_ {\mathcal {G}} p (\Psi \mid \mathcal {G}) p (\mathcal {G} \mid D), \quad p (\mathcal {G} \mid D) \propto p (D \mid \mathcal {G}) p (\mathcal {G}), \tag {2}
$$
wherein a key component is the likelihood of the data integrated over the prior distribution of the parameters  $\theta$  of the conditional distributions:
$$
p (D \mid \mathcal {G}) = \int p (D \mid \mathcal {G}, \theta) p (\theta \mid \mathcal {G}) d \theta . \tag {3}
$$
The marginal likelihood  $p(D \mid \mathcal{G})$  for a given DAG  $\mathcal{G}$  is available in closed form for specific combinations of priors on parameters and likelihood functions [11, 12].
The number of DAGs grows super-exponentially in the number of nodes [38], hence exact posterior inference is still exponentially complex, making it effectively intractable for large networks [47]. Approximate inference for  $p(\Psi \mid D)$  is possible if one can obtain samples  $\mathcal{G}_1, \ldots, \mathcal{G}_M$  from the posterior distribution over graphs  $p(\mathcal{G} \mid D)$ .
Markov Chain Monte Carlo (MCMC) methods have successfully tackled the problem of posterior inference in the large and complex space of DAGs. The rationale consists of defining a Markov chain whose stationary distribution is the posterior distribution of interest  $p(\mathcal{G} \mid D)$ . Madigan et al. [30] suggested a Metropolis-Hastings sampler: at each step, the algorithm proposes a new DAG  $\mathcal{G}'$  and it accepts or rejects it according to a probability  $\alpha$ . The value of  $\alpha$  depends on the relative posterior probability  $p(\mathcal{G}' \mid D)$  of the proposal graph compared to that of the current DAG:
$$
\alpha = \min  \left\{1, \frac {p \left(\mathcal {G} ^ {\prime} \mid D\right) Q \left(\mathcal {G} , \mathcal {G} ^ {\prime}\right)}{p \left(\mathcal {G} \mid D\right) Q \left(\mathcal {G} ^ {\prime} , \mathcal {G}\right)} \right\} \tag {4}
$$
where  $Q(\cdot, \cdot)$  are the transition probabilities from one DAG to another. In its original formulation, the algorithm proposes a new graph by adding or deleting one edge from the current DAG, with  $Q$  uniformly distributed in the set of neighbouring DAGs (including the current one).
Several refinements to the original method have been made over the years, providing for example more efficient proposal distributions [14, 20, 44]. The order MCMC approach by Friedman and Koller [8] introduced an important variation by sampling in the space of node orderings, which is smaller and smoother than the space of DAGs. Because of these attractive properties of the sample space, the order MCMC algorithm achieves superior mixing and convergence results compared to regular structure MCMC. Sampling orders however introduces a bias in the resulting posterior since node orders do not induce a uniform coverage in the DAG space [7]. The partition MCMC algorithm [25] corrects this bias in order MCMC by sampling in the larger space of ordered partitions of nodes to achieve unbiased samples from the posterior.
Current state-of-the-art methods rely on conditional independence tests to obtain a first coarse estimate of the structure to use as a starting point for a score-based method [48]. More sophisticated approaches involve iteratively expanding the search space to correct for errors in its initial estimate [28]. To achieve efficient posterior inference [49, 28] in the selected search space it helps to precompute all combinations of scores potentially needed in the chain so it can run at a very low computational cost.
Key to enabling efficient DAG sampling is decomposability, a property ensuring that we can express the posterior probability of a DAG  $\mathcal{G}$  as a product of local scores that depend only on each variable and its parents in  $\mathcal{G}$ :
$$
p (\mathcal {G} \mid D) \propto \prod_ {i} ^ {n} S \left(X _ {i}, \mathrm {P a} _ {X _ {i}} ^ {\mathcal {G}} \mid D\right). \tag {5}
$$
Decomposability guarantees that we need to recompute only those scores whose parent sets have changed, or we can precompute all needed combinations of scores efficiently [28].
2.2 Gaussian Processes
Gaussian processes are a flexible tool used in machine learning for regression and classification tasks. Formally, a Gaussian Process (GP) is a distribution over functions such that every finite collection of its function values  $\{f(x_1), f(x_2), \ldots, f(x_k)\}$  has a multivariate Gaussian distribution [36]. A GP is therefore fully specified by its mean function  $m(x) = \mathbb{E}[f(x)]$  and covariance function  $k(x, x') = \mathrm{Cov}[f(x), f(x')]$ . Due to their ability to model a large range of functional behaviours, GPs find common use as priors over regression functions  $f(X) = \mathbb{E}(Y \mid X)$ . A common GP regression model assumes independent Gaussian additive noise:
$$
Y = f (X) + \varepsilon , \quad f (X) \sim \operatorname {G P} \left(0, k \left(x, x ^ {\prime}\right)\right), \quad \varepsilon \sim \mathcal {N} \left(\mu , \sigma^ {2}\right). \tag {6}
$$
Notably, GP models admit a closed-form expression of the marginal likelihood, in this case, the likelihood of the  $N$  observations  $y$  marginalised over the prior distribution of  $f$ :
$$
p (y) = (2 \pi) ^ {- \frac {N}{2}} | K + \sigma^ {2} I | ^ {\frac {1}{2}} \exp \left(- \frac {1}{2} (y - \mu) ^ {\top} (K + \sigma^ {2} I) ^ {- 1} (y - \mu)\right) \tag {7}
$$
where  $K$  is the  $N\times N$  Gram matrix  $K_{ij} = k(x_i,x_j)$
2.3 Gaussian Process Networks
Gaussian process networks (GPNs) refer to Bayesian networks whose conditional distributions are modeled via Gaussian process priors [9]. The structural equation model defining the distribution of each variable  $X_{i}$  given its parents in a GPN is
$$
X _ {i} = f _ {i} \left(\mathrm {P a} _ {X _ {i}}\right) + \varepsilon_ {i} \tag {8}
$$
where  $\varepsilon_{i}$  has a normal distribution  $\mathcal{N}(\mu, \sigma^2)$  independent of the data and a Gaussian process prior is placed on the function  $f_{i}$ . Thanks to the nonparametric nature of GPs, the model in (8) can capture a wide range of functional dependencies between variables while maintaining the closed-form expression (7) for the marginal likelihood.
The covariance function  $k$  of the Gaussian process is typically parameterized by a set of hyperparameters  $\theta$  which determine the behaviour of samples of  $f_{i}$  such as smoothness, shape or periodicity. When these hyperparameters (together with the noise mean  $\mu$  and variance  $\sigma^2$ ) are unknown they are typically learned by maximizing the marginal likelihood. Since the marginal likelihood and its derivatives are available in closed form, the optimization can be performed efficiently via gradient ascent [36]. The GPN model enables structure learning of networks on continuous variables without the need to make strict parametric assumptions on the distributions of the variables. In scenarios even with low sample sizes, Friedman and Nachman [9] have shown that searching for the highest-scoring structure can accurately reconstruct the underlying DAG under different functional relationships.
When estimating the score of a GPN, the common approach to learning the hyperparameters by simple maximization can however lead to problematic estimates, since many local optima may exist [4]. In addition, the resulting plug-in estimate of the marginal likelihood would not correctly reflect the uncertainty in the data of the full posterior. Bayesian model averaging provides a natural solution by integrating over the prior distribution of the hyperparameters to obtain a true marginal likelihood  $p(D \mid \mathcal{G})$ , which we can then use to perform posterior inference on the structure.
3 Bayesian Structure Inference for GPNs
In this section, we describe a method to sample from the posterior distribution  $p(\Psi \mid D)$  of a generic feature  $\Psi$  of a GPN for continuous, non-linear data. To implement a fully Bayesian approach, we place priors over the hyperparameters  $\theta$  of the kernel function and the Gaussian noise.
$$
X = f \left(\operatorname {P a} _ {X}\right) + \varepsilon , \quad \varepsilon \sim \mathcal {N} (\mu , \sigma^ {2})
$$
$$
f \sim \operatorname {G P} (0, k _ {\boldsymbol {\theta}} (., .)) \tag {9}
$$
$$
\boldsymbol {\theta} \sim \pi (\boldsymbol {\theta}), \quad \mu , \sigma \sim \pi (\mu , \sigma).
$$
The priors ensure that the uncertainty in the functional relationships between each variable and its parents is fully accounted for. On the other hand, a maximum likelihood approach to estimating the hyperparameters could yield overly confident score functions and in turn misrepresent the posterior  $p(\mathcal{G} \mid D)$ . Under the GPN model, the score remains decomposable, a necessary condition for its efficient evaluation. As in Section 2.1, making inference about network features  $\Psi$  of interest is possible by sampling graphs from the posterior:
$$
p (\Psi \mid D) \approx \frac {1}{M} \sum_ {j = 1} ^ {M} p (\Psi \mid \mathcal {G} _ {j}), \quad \mathcal {G} _ {j} \sim p (\mathcal {G} _ {j} \mid D). \tag {10}
$$
Sampling graphs hinges on computing the scores of all variables given different combinations of their possible parent sets (see equation 5).
Let  $\Theta = \{\mu, \sigma, \theta\}$  be the  $d$ -dimensional set of hyperparameters for a given node  $X$  and its parents  $\mathrm{Pa}_X$ . Unless stated otherwise, throughout the rest of the text we assume a uniform prior over all structures  $p(\mathcal{G}) \propto 1$ . The score function is then the likelihood  $(\overline{7})$  of the observations  $x$  marginalized with respect to the hyperparameter priors:
$$
S (X, \mathrm {P a} _ {X}) = \int p (x \mid \mathrm {P a} _ {X}, \Theta) \pi (\Theta \mid \mathrm {P a} _ {X}) \mathrm {d} \Theta . \tag {11}
$$
If a variable  $X$  has no parents then the Gram matrix of the kernel is zero and the score function reduces to a Gaussian marginal likelihood. Since the above score function is generally intractable,
Algorithm 1 GP network sampling scheme
Input Data  $D$  of  $n$  variables, feature of interest  $\Psi$  Output Posterior probability of the feature  $p(\Psi \mid D)$    
1: for  $j\in \{1,\dots ,M\}$  do   
2: Sample DAG  $\mathcal{G}_j$  according to its Laplace approximate posterior  $q(\mathcal{G}_j\mid D)$  .  $\triangleright$  Eq. (13,5)   
3: for  $i\in \{1,\ldots ,n\}$  do   
4: Compute  $S(X_{i},\mathrm{Pa}_{X_{i}}^{\mathcal{G}_{j}})$  via MC estimation.  $\triangleright$  Eq. (12)   
5: Compute posterior  $p(\mathcal{G}_j\mid D)$  .  $\triangleright$  Eq. (5)   
6: Compute posterior probability of  $\Psi$  via importance sampling.  $\triangleright$  Eq. (14)
one option is to use Monte Carlo (MC) approaches such as bridge sampling [31] to approximate it. Bridge sampling employs a Gaussian proposal distribution  $g$  and a bridge function  $h$  chosen to minimize the MSE of the resulting estimator. The bridge sampling estimator of (11) is then defined as
$$
S (X, \mathrm {P a} _ {X}) \approx \frac {\frac {1}{N _ {1}} \sum_ {i = 1} ^ {N _ {1}} p \left(x \mid \mathrm {P a} _ {X} , \Theta_ {i}\right) \pi \left(\Theta_ {i} \mid \mathrm {P a} _ {X}\right) h \left(\Theta_ {i} \mid \mathrm {P a} _ {X}\right)}{\frac {1}{N _ {2}} \sum_ {j = 1} ^ {N _ {2}} g \left(\Theta_ {j} ^ {*} \mid \mathrm {P a} _ {X}\right) h \left(\Theta_ {j} ^ {*} \mid \mathrm {P a} _ {X}\right)}, \tag {12}
$$
$$
\Theta_ {i} \sim g (\Theta \mid \mathrm {P a} _ {X}), \quad \Theta_ {j} ^ {*} \sim p (\Theta \mid x, \mathrm {P a} _ {X}), \quad i = 1, \dots , N _ {1}, \quad j = 1, \dots , N _ {2}.
$$
The estimator is also a function of samples from the posterior of the hyperparameters  $p(\Theta \mid x, \mathrm{Pa}_X)$ , which can easily be obtained via MCMC sampling. One can show that other approaches such as importance sampling or harmonic mean are special cases of bridge sampling [18]. MC methods based on bridge sampling provide consistent estimators for the marginal likelihood but may be biased for finite samples [50]. Nonetheless, such methods can become computationally expensive in high dimensions (i.e. for large parent sets).
Since sampling DAGs via MCMC requires computing a large number of scores of potential parent sets, which may not all be represented in the final sample, we avoid computing these expendable scores by first running the MCMC algorithm using a Laplace approximation of the score (11) around the MAP value of  $\Theta$ :
$$
S _ {\mathrm {L}} (X, \mathrm {P a} _ {X}) = p (x | \mathrm {P a} _ {X}, \tilde {\Theta}) \pi (\tilde {\Theta} | \mathrm {P a} _ {X}) \frac {(2 \pi) ^ {d / 2}}{| H | ^ {1 / 2}} \tag {13}
$$
with  $\tilde{\Theta} = \operatorname*{argmax}_{\Theta}p(x|\mathrm{Pa}_X,\Theta)\pi (\Theta |\mathrm{Pa}_X)$ , and  $H_{ij} = -\frac{\partial^2p(x|\mathrm{Pa}_X,\Theta)\pi(\Theta|\mathrm{Pa}_X)}{\partial\Theta_i\partial\Theta_j}\bigg|_{\Theta = \tilde{\Theta}}$
We denote the resulting posterior probability of a DAG  $\mathcal{G}$  from this Laplace approximated score as  $q(\mathcal{G} \mid D)$  to distinguish it from the true posterior  $p(\mathcal{G} \mid D)$ .
The Laplace approximate score provides an approximation of the posterior at a lower computational cost, speeding up considerably the running time of the MCMC algorithm used to sample graphs. After sampling  $M$  graphs in the first step with the Laplace approximate score, we can make inference with respect to the true posterior by re-computing the scores and performing importance sampling. To estimate the posterior probability of a feature of interest  $\Psi$  via importance sampling we evaluate
$$
p (\Psi \mid D) \approx \frac {\sum_ {j = 1} ^ {M} p (\Psi \mid \mathcal {G} _ {j}) w _ {j}}{\sum_ {j = 1} ^ {M} w _ {j}}, w _ {j} = \frac {p (\mathcal {G} _ {j} \mid D)}{q (\mathcal {G} _ {j} \mid D)} \tag {14}
$$
where  $p(\mathcal{G}_j \mid D)$  and  $q(\mathcal{G}_j \mid D)$  are the posterior probabilities of DAG  $\mathcal{G}_j$  computed respectively with the bridge sampling MC estimate (12) and the Laplace approximation (13) for the score. The two posteriors do not need to be normalized for valid posterior inference on  $\Psi$ . This is because the normalizing constants in the importance sampling weights (14) cancel out. The procedure is summarized as pseudo-code in Algorithm 1.
Re-computing the score functions in a second step and implementing importance sampling is computationally advantageous compared to running the MCMC algorithm directly with the MC estimates of the scores. The advantage is simply due to the number of unique scores in the final chain being much lower than those evaluated or needed during the chain itself (see figure A.5 in the supplementary material). Regardless of the MCMC algorithm used, we expect a substantial improvement in run-time compared to using the MC scores directly.
3.1 Implementation Details
Bayesian inference and optimization of the hyperparameters were performed via the Stan interface RStan [43]. The library offers a highly efficient  $\mathrm{C + + }$  implementation of the No U-turn sampler [22], providing state-of-the-art posterior inference on the hyperparameters. We performed MC estimation of the marginal likelihood via the bridgesampling package [19], which can easily be combined with fitted Stan models.
In the implementation of Algorithm 1 we use order or partition MCMC to generate the network samples  $q(\mathcal{G}_j \mid D)$ ; the procedure can however accommodate a variety of sampling methods, as long as they result in samples from the posterior. Given its good performance in benchmarking studies [37], we use the recent BiDAG [45] hybrid implementation for MCMC inference on the graph. The hybrid sampler requires an initial search space which is then improved upon to correct for possible estimation errors [28]. As initial search space we take the output of the dual PC algorithm [13]. For the bridge sampling estimator [12] of the marginal likelihood we used  $N_1 = N_2 = 300$  particles from the proposal and posterior distribution over the hyperparameters. The proposal function  $g$  was set to a normal distribution, with its first two moments chosen to match those of the posterior distribution. R code to implement Algorithm 1 and reproduce the results in Sections 4 and 5 is available at https://github.com/enricogiudice/LearningGPNs
3.2 Score Equivalence