Title: Implicit Manifold Gaussian Process Regression
Venue: NeurIPS 2023
Accepted: yes
Keywords: Gaussian process

manifolds

manifold learning

uncertainty

regression

graph Laplacian
Abstract: Gaussian process regression is widely used because of its ability to provide well-calibrated uncertainty estimates and handle small or sparse datasets. However, it struggles with high-dimensional data. One possible way to scale this technique to higher dimensions is to leverage the implicit low-dimensional manifold upon which the data actually lies, as postulated by the manifold hypothesis. Prior work ordinarily requires the manifold structure to be explicitly provided though, i.e. given by a mesh or be known to be one of the well-known manifolds like the sphere. In contrast, in this paper we propose a Gaussian process regression technique capable of inferring implicit structure directly from data (labeled and unlabeled) in a fully differentiable way. For the resulting model, we discuss its convergence to the Matérn Gaussian process on the assumed manifold. Our technique scales up to hundreds of thousands of data points, and improves the predictive performance and calibration of the standard Gaussian process regression in some high-dimensional settings.

Implicit Manifold Gaussian Process Regression

Abstract

Gaussian process regression is widely used because of its ability to provide well-calibrated uncertainty estimates and handle small or sparse datasets. However, it struggles with high-dimensional data. One possible way to scale this technique to higher dimensions is to leverage the implicit low-dimensional manifold upon which the data actually lies, as postulated by the manifold hypothesis . Prior work ordinarily requires the manifold structure to be explicitly provided though, i.e. given by a mesh or be known to be one of the well-known manifolds like the sphere. In contrast, in this paper we propose a Gaussian process regression technique capable of inferring implicit structure directly from data (labeled and unlabeled) in a fully differentiable way. For the resulting model, we discuss its convergence to the Matérn Gaussian process on the assumed manifold. Our technique scales up to hundreds of thousands of data points, and improves the predictive performance and calibration of the standard Gaussian process regression in some high-dimensional settings.

1 Introduction

Gaussian processes are among the most adopted models for learning unknown functions within the Bayesian framework. Their data efficiency and aptitude for uncertainty quantification make them appealing for modeling and decision-making applications in the fields like robotics (Deisenroth and Rasmussen, 2011), geostatistics (Chilès and Delfiner, 2012), numerics (Hennig et al., 2015), etc.

The most widely used Gaussian process models, like squared exponential and Matérn (Rasmussen and Williams, 2006), impose the simple assumption of differentiability of the unknown function while also respecting the geometry of R d by virtue of being stationary or isotropic. Such simple assumptions make uncertainty estimates reliable , albeit too conservative at times. The same simplicity makes these models struggle from the curse of dimensionality . We hypothesize that it is still possible to leverage these simple priors for real world high-dimensional problems granted that they are adapted to the implicit low-dimensional submanifolds where the data actually lies, as illustrated by Figure 1.

Prediction

(a) Euclidean, prediction

(b) Ours, prediction

(c)Euclidean, uncertainty

Uncertainty

(d) Ours, uncertainty

Code available at https://github.com/nash169/manifold-gp .

Recent works in machine learning generalized Matérn Gaussian processes for modeling functions on non-Euclidean domains such as manifolds or graphs (Azangulov et al., 2022; 2023; Borovitskiy et al., 2021; 2023; 2020). Crucially, this line of work assumes known geometry (e.g. a manifold or a graph) beforehand. In this work we aim to widen the applicability of Gaussian processes for higher dimensional problems by automatically learning the implicit low-dimensional manifold upon which the data lies, the existence of which is suggested by the manifold hypothesis . We propose a new model which learns this structure and approximates the Matérn kernel on the implicit manifold.

Our approach can operate in both supervised and semi-supervised settings, with the emphasis on the latter: uncovering the implicit manifold may require a lot of samples from it, however these samples need not be labeled, and unlabeled data is usually more abundant. Taking inspiration in the manifold learning results of Coifman and Lafon (2006), Dunson et al. (2021) and others we approximate the unknown manifold by an appropriately weighted nearest neighbor graph. Then we use graph Matérn kernels thereon as approximations to the manifold Matérn kernels of Borovitskiy et al. (2020), extending them to the vicinity of the manifold in the ambient R d in an appropriate way.

1.1 Related Work and Contribution

High-dimensional Gaussian process regression is an area of active research, primarily motivated by decision making applications like Bayesian optimization. There are three main directions in this area: (1) selecting a small subset of input dimensions, (2) learning a small number of new features by linearly projecting the inputs and (3) learning non-linear features. Our technique belongs to the third direction. Further details on the area can be found in the recent review by Binois and Wycoff (2022).

A close relative of our technique in the literature is described in Dunson et al. (2022). It targets the low-dimensional setting where the inputs are densely sampled on the underlying surface. It is based on the heat (diffusion) kernels on graphs as in Kondor and Lafferty (2002) and uses the Nyström method to extend kernels to R d , both of which may incur a high computational cost. Another close relative is the concurrent work by Peach et al. (2023) on modeling vector fields on implicit manifolds.

We are targeting the high-dimensional setting. Here, larger datasets of partly labeled points are often needed to infer geometry. Because of this, we emphasize computational efficiency by leveraging sparse precision matrix structure of Matérn kernels (as opposed to the heat kernels) and use KNN for sparsifying the graph and accelerating the Nyström method. This results in linear computational complexity with respect to the number of data points. Furthermore, the model we propose is fully differentiable, which may be used to find both kernel and geometry hyperparameters by maximizing the marginal likelihood. Finally, to get reasonable predictions on the whole ambient space R d , we combine the prediction of the geometric model with the prediction of a classical Euclidean Gaussian process, weighting these by the relative distance to the manifold.

The geometric model is differentiable with respect to its kernel-, likelihood- and geometry-related hyperparameters, with gradient evaluation cost being linear with respect to the number of data points. After training, we can efficiently compute the predictive mean and kernel as well as sample the predictive model, providing the basic computational primitives needed for the downstream applications like Bayesian optimization. We evaluate our technique on a synthetic low-dimensional example and test it in a high-dimensional large dataset setting of predicting rotation angles of rotated MNIST images, improving over the standard Gaussian process regression.

2 Gaussian Processes

A Gaussian process f ∼ GP( m,k ) is a distribution over functions on a set X . It is determined by the mean function m ( x ) = E f ( x ) and the covariance function (kernel) k ( x , x ′ ) = Cov( f ( x ) , f ( x ′ )) .

Given data X , y , where X = ( x 1 , .., x n ) glyph[latticetop] and y = ( y 1 , .., y n ) glyph[latticetop] with x i ∈ X d , y i ∈ R , one usually assumes y i = f ( x i )+ ε i where ε i ∼ N(0 , σ 2 ε ) is IID noise and f ∼ GP(0 , k ) is some prior Gaussian process, whose mean is assumed to be zero in order to simplify notation. The posterior distribution f | y is then another Gaussian process f | y ∼ GP(ˆ m, ˆ k ) with (Rasmussen and Williams, 2006)

.

(a) Kernel, sphere S 2

(b) Kernel, graph P S 2

(c) Sample, sphere

S

2

(d) Sample, graph

P

S

2

Matérn Gaussian processes-including the limiting ν → ∞ case, squared exponential Gaussian processes-are the most popular family of models for X = R d . These have zero mean and kernels

where K ν is the modified Bessel function of the second kind (Gradshteyn and Ryzhik, 2014) and ν, κ, σ 2 are the hyperparameters responsible for smoothness, length scale and variance, respectively. We proceed to describe how Matérn processes can be generalized to inputs x lying on explicitly given manifolds or graphs instead of the Euclidean space R d .

2.1 Matérn Gaussian Processes on Explicit Manifolds and Graphs

For a domain which is a Riemannian manifold, an obvious and natural idea for generalizing Matérn Gaussian processes could be to substitute the Euclidean distances ‖ x -x ′ ‖ in Equation (2) with the geodesic distance. However, this approach results in ill-defined kernels that fail to be positive semi-definite (Feragen et al., 2015; Gneiting, 2013).

Another direction for generalization is based on the stochastic partial differential equation (SPDE) characterization of Matérn processes first described by Whittle (1963): f ∼ GP(0 , k ν,κ,σ 2 ) solves

where ∆ R d is the standard Laplacian operator and W is the Gaussian white noise with variance proportional to σ 2 . If taken to be the definition, this characterization can be easily extended to general Riemannian manifolds X = M by substituting ∆ R d with the Laplace-Beltrami operator ∆ M , taking d = dim M and substituting W with the appropriate generalization of the Gaussian white noise (Lindgren et al., 2011). Based on this idea, Borovitskiy et al. (2020) showed that on compact Riemannian manifolds, Matérn Gaussian processes are the zero-mean processes with kernels

where -λ l , f l are eigenvalues and eigenfunctions of the Laplace-Beltrami operator and C ν,κ is the normalizing constant ensuring that 1 X ∫ X k ν,κ,σ 2 ( x, x ) d x = σ 2 . This, alongside with considerations from Azangulov et al. (2022) allows one to practically compute k ν,κ,σ 2 for many compact manifolds.

If the domain X is a weighted undirected graph G , we can also use Equation (3) to define Matérn Gaussian processes on G (Borovitskiy et al., 2021). In this case, ∆ R d is substituted with the minus graph Laplacian -∆ G and W ∼ N(0 , σ 2 W I ) is the vector of IID Gaussians. Here, SPDE transforms into a stochastic linear system, whose solution is of the same form as Equation (4) but with a finite sum instead of the infinite series, with d = 0 because there is no canonical notion of dimension for graphs and with λ l , f l being the eigenvalues and eigenvectors-as functions on the node set-of the matrix ∆ G . These processes are illustrated on Figure 2.

3 Implicit Manifolds and Gaussian Processes on Them

Consider a dataset X = ( x 1 , .., x N ) glyph[latticetop] , x i ∈ R d partially labeled with labels y 1 , .., y n ∈ R , n ≤ N . Assume that x i are IID randomly sampled from a compact Riemannian submanifold M⊆ R d . As by Section 2.1, the manifold M is associated to a family of Matérn Gaussian processes tailored to its geometry. We do not assume to know M , only the fact that it exists, hence the question is: how can we recover the kernels of the aforementioned geometry-aware processes from the observed dataset?

It is clear from Equation (4) that to recover k ν,κ,σ 2 we need to get the eigenpairs -λ l , f l of the Laplace-Beltrami operator on M . Naturally, for a finite dataset this can only be done approximately. We proceed to discuss the relevant theory of Laplace-Beltrami eigenpair approximation.

3.1 Background on Approximating the Eigenpairs of the Laplace-Beltrami Operator

There exists a number of theoretical and empirical results on eigenpair approximation. Virtually all of them study approximating the implicit manifold by some kind of a weighted undirected graph 1 with node set { x 1 , .., x N } and weights that are somehow determined by the Euclidean distances ‖ x i -x j ‖ . The eigenvalues of the graph Laplacian on this graph are supposed to approximate the eigenvalues of the Laplace-Beltrami operator, while the eigenvectors-regarded as functions on the node set-approximate the values of the eigenfunctions of the Laplace-Beltrami operator at x i ∈ M . To approximate eigenfunctions elsewhere, any sort of continuous (smooth) interpolation suffices.

There are three popular notions of graph Laplacian. Let us denote the adjacency matrix of the weighted graph by A and define D to be the diagonal degree matrix with D ii = ∑ j A ij . Then

The first two of these are symmetric positive semi-definite matrices, the third is, generally speaking, non-symmetric. However, from the point of view of linear operators, all of them can be considered symmetric (self-adjoint) positive semi-definite: the first two with respect to the standard Euclidean inner product 〈· , ·〉 , and the third one with respect to the modified inner product 〈 v , u 〉 D = 〈 D v , u 〉 . Thus for each there exists an orthonormal basis of eigenvectors and eigenvalues are non-negative. 2

The most common way to define the graph is by setting A ij = exp ( -‖ x i -x j ‖ 2 / 4 α 2 ) for an α > 0 . If x i are IID samples from the uniform distribution on the manifold M , then all of the graph Laplacians, each multiplied by an appropriate power of α , converge to the Laplace-Beltrami operator, both pointwise (Hein et al., 2007) and spectrally (García Trillos et al., 2020), i.e. in the sense of eigenpair convergence, at least at the node set of the graph. 3 However, if the inputs x i are sampled non-uniformly, graph Laplacians, at best, converge to different continuous limits, none of which coincides with the Laplace-Beltrami operator (Hein et al., 2007).

Coifman and Lafon (2006) proposed a clever trick to handle non-uniformly sampled data x 1 , .., x N . Starting with ˜ A and ˜ D defined in the same way as A and D before, they define A = ˜ D -1 ˜ A ˜ D -1 . Intuitively, this corresponds to normalizing by the kernel density estimator to cancel out the unknown density. The corresponding ∆ rw then converges pointwise to the Laplace-Beltrami operator (Hein et al., 2007), though ∆ un and ∆ sym do not: they converge to different continuous limits. Dunson et al. (2021, Theorem 2) show that, under technical regularity assumptions, eigenvalues λ k and renormalized eigenvectors of ∆ rw converge to the respective eigenvalues and eigenfunctions of the Laplace-Beltrami operator, regardless of the sampling density of x i .

Both in the simple case and in the sampling density independent case, the graphs and their respective Laplacians turn out to be dense, requiring a lot of memory to store and being inefficient to operate with. To make computations efficient, sparse graphs such as KNN graphs are much more preferable over the dense graphs. Spectral convergence for KNN graphs is studied, for example, in Calder and Trillos (2022), for A ij = h ( ‖ x i -x j ‖ / α ) with a compactly supported regular function h , and with

limit depending on the sampling density. Unfortunately, we are unaware of any spectral convergence results in the literature that hold for KNN graphs and are independent of the data sampling density.

3.2 Approximating Matérn Kernels on Manifolds

Here we incorporate various convergence results, including but not limited to the ones described in Section 3.1, proving that all spectral convergence results imply the convergence of graph Matérn kernels to the respective manifold Matérn kernels.

Proposition 1. Denote the eigenpairs by λ l , f l for a graph Laplacian and by λ M l , f M l for the Laplace-Beltrami operator. Fix δ > 0 . Assume that, with probability at least 1 -δ , for all ε > 0 , for α small enough and for N large enough we have | λ l -λ M l | < ε and | f l ( x i ) -f M l ( x i ) | < ε . Then, with probability at least 1 -δ , we have k N,α,L ν,κ,σ 2 ( x i , x j ) → k ν,κ,σ 2 ( x i , x j ) as α → 0 , N,L →∞ , where

Proof. First prove that the tail of the series in Equation (4) converges uniformly to zero, then combine this with eigenpair bounds. See details in Appendix A.

Remark. The convergence in x i ∈ M can be lifted to pointwise convergence for all x ∈ M if eigenvectors are interpolated Lipschitz-continuously, simply because the eigenfunctions are smooth.

Inspired by this theory, we proceed to present the implicit manifold Gaussian process model.

3.3 Implicit Manifold Gaussian Process

Guided by the theory we described in the previous section we are now ready to formulate the implicit manifold Gaussian process model. Given the dataset x 1 , .., x N ∈ R d and y 1 , .., y n ∈ R , we put glyph[negationslash]

Here S K ( x i , x j ) = 1 if x i is one of the K nearest neighbors of x j or vice versa and S K ( x i , x j ) = 0 otherwise; all matrices are of size N × N and depend on α and K as hyperparameters. Thanks to the coefficient S K ( x i , x j ) that performs KNN sparsification, the matrix A is sparse when K glyph[lessmuch] N . 4

Then we consider the operator ∆ rw = I -D -1 A defined by Equation (5), whose matrix is also sparse. Denoting its eigenvalues-ordered from the smallest to the largest-by 0 = λ 0 ≤ λ 1 ≤ . . . ≤ λ N -1 , and its eigenvectors-orthonormal under the modified inner product 〈· , ·〉 D and regarded as functions on the node set of the graph-by f 0 , f 1 , . . . , f N -1 , we define Matérn kernel on graph nodes x i by

where L does not need to be equal to the actual number N of eigenpairs. Doing so means truncating the high frequency eigenvectors ( f l for l large), which always contribute less to the sum because they correspond to smaller values of Φ ν,κ ( λ l ) . This can massively reduce the computational costs.

By Proposition 1, Equation (8) approximates the manifold Matérn kernel with smoothness ν ′ = ν -dim( M ) / 2 . We adopt such a reparametrization because it does not require estimating the a priori unknown dim( M ) . This, however, makes the typical assumption of ν ∈ { 1 / 2 , 3 / 2 , 5 / 2 } inadequate. We chose a particular graph Laplacian normalization, namely the random walk normalized graph Laplacian ∆ rw , to approximate the true Laplace-Beltrami operator regardless of the potential non-uniform sampling of x 1 , .., x N , based on the theoretical insights described in Section 3.1.

The kernel in Equation (8) is only defined on the set of nodes x i , next step is to extend it to the whole space R d . Extending kernels is usually a difficult problem because one has to worry about positive

(a) Extended eigenvector

f

3

(

·

)

(b) Extended kernel

k

X

ν,κ,σ

2

(

•

,

·

)

(c) Function

γ

(dist(

·

,

M

))

semi-definiteness. To work around it, we extend the features f l . For this, we use Nyström method: we allow the first argument of S K to be an arbitrary vector from R d , defining S K ( x , x j ) = 1 if x j is one of the K nearest neighbors of x among x 1 , .., x N . This allows us to extend ˜ A , ˜ D , A and D as

where KNN( x ) is the set of the K nearest neighbors from x among x 1 , .., x N . With this, we define

It is easy to check that this extension perfectly reproduces the values f l ( x j ) , simply because A ( x , x j ) coincides with A ij when x = x i and because ( f l ( x 1 ) , .., f l ( x N )) glyph[latticetop] is the eigenvector of A corresponding to the eigenvalue 1 -λ l . It also allows us to extend the kernel k X ν,κ,σ 2 as well, such that k X ν,κ,σ 2 ( x , x ′ ) is defined for arbitrary x , x ′ ∈ R d and Equation (8) still holds for the nodes x 1 , .., x N . We visualize an extended eigenvector and an extended kernel in Figures 3a and 3b.

For x far away from the nodes x j the values of ˜ D ( x ) become very small, making the extension procedure numerically unstable. Furthermore, the geometric model is generally not so relevant far away from the manifold. Because of this, the final predictive model f ( p ) ∼ GP( m ( p ) , k ( p ) ) combines the geometric model f ( m ) ∼ GP( m ( m ) , k ( m ) ) -the posterior under the kernel k X ν,κ,σ 2 we defined just above-with the standard Euclidean model f ( e ) ∼ GP( m ( e ) , k ( e ) ) -the posterior under the standard Euclidean Gaussian process in R d , for instance with the squared exponential kernel:

where γ is a bump function that is zero outside the 3 α neighborhood of the manifold, illustrated in Figure 3c. Here dist( x , M ) can be computed as the distance from x to its nearest neighbor node x j . 5

4 Efficient Training of the Implicit Manifold Gaussian Processes

Here we describe how to perform the implicit manifold Gaussian process regression efficiently, being able to handle hundreds of thousands of points, in both supervised and semi-supervised regimes.

In all cases we need efficient (approximate) KNN to build a graph, extend the kernel beyond the nodes x i and combine the geometric model with the standard Euclidean one. For this we use FAISS (Johnson et al., 2019). The resulting sparse matrices, such as the Laplacian ∆ rw , we represent as black box functions capable of performing matrix-vector multiplications for any given input vector.

After hyperparameters are found-we will return to their search later-we need to compute the eigenpairs λ l , f l of ∆ rw . For this we run Lanczos algorithm (Meurant, 2006) to evaluate the eigenpairs λ sym l , f sym l of the symmetric matrix ∆ sym , putting λ l = λ sym l and f l = D -1 / 2 f sym l because the matrices ∆ rw and ∆ sym are similar, i.e. ∆ rw = D -1 / 2 ∆ sym D 1 / 2 . Importantly, Lanczos algorithm only relies on matrix-vector products with ∆ sym . We only compute a few hundred of eigenpairs, asking Lanczos to provide twice or trice as many and disregarding the rest.

When there is a lot of labeled data, we approximate the classical Euclidean (e.g. squared exponential) kernel using random Fourier features approximation (Rahimi and Recht, 2007), this allows linear computational complexity scaling with respect to the number of data points. The number of Fourier features is taken to be equal to L , with the same L as in Equation (8).

As it was already mentioned, we need to find hyperparameters ˆ θ = ( ˆ α, ˆ κ, ˆ σ 2 , ˆ σ 2 ε ) that determine the graph, Gaussian process prior and the noise variance that fit the observations y best. The ν parameter we assume manually fixed. To avoid nonsensical parameter values-a common difficulty often occurring when the data is scarce-one might want to assume a prior p ( θ ) on θ . Some specific choices of p ( θ ) are discussed in Appendix C. Then ˆ θ is a maximum a posteriori (MAP) estimate:

To simplify hyperparameter initialization and align with zero prior mean assumption it makes sense to preprocess y i to be centered and normalized.

To solve the optimization problem in Equation (13) we use restarted gradient descent. Repeatedly evaluating the gradient of log p ( y | θ , X ) is the main computational bottleneck. The key idea for doing this efficiently-viable for integer values of ν -is to reduce matrix-vector products with Matérn kernels' precision to iterated matrix-vector products with the Laplacian, which is sparse . First, we describe this in detail in the noiseless supervised setting, where the idea is most directly applicable.

4.1 Noiseless Supervised Learning

Here we assume that all inputs are labeled, i.e. N = n , and all observations are noiseless, i.e. σ 2 ε = 0 .

Denoting by P XX = K -1 XX the precision matrix, the log-likelihood log p ( y | θ , X ) , up to a multiplicative constant and an additive constant irrelevant for optimization, is given by

Its gradient may be given and then subsequently approximated (Gardner et al., 2018) by

where z is a random vector consisting of IID variables that are either 1 or -1 with probability 1 / 2 . The first term on the right-hand side is the stochastic estimate of the trace of Hutchinson (1989). Since the kernels from Section 3.3 coincide with graph Matérn kernels on the nodes x i , we have

However, as the graph bandwidth α is one of the hyperparameters we optimize over, using Equation (16) would entail repeated eigenpair computations and differentiating through this procedure. Because of this, we use an alternative way to compute matrix-vector products P XX u detailed below. 6

Proposition 2. Assuming ν ∈ N , the precision matrix P XX of k X ν,κ,σ 2 ( x i , x j ) can be given by

Using Proposition 2 to evaluate matrix-vector products P XX u and conjugate gradients (Meurant, 2006) to solve z glyph[latticetop] P -1 XX using only the matrix-vector products, we can efficiently evaluate the righthand side of Equation (15), with linear costs with respect to N , assuming that the graph is sparse. Preconditioning (Wenger et al., 2022) can be used to further improve the efficiency of the solve.

4.2 Noiseless Semi-Supervised Learning

Here we assume that inputs are partly unlabeled, i.e. N = n , while observations are still noiseless, i.e. σ 2 ε = 0 . Denote Z to be the labeled part of X . Then the matrix K XX in the log-likelihood given by Equation (14) should be substituted with K ZZ . However, while P XX can be represented using Equation (17), the precision P ZZ = K -1 ZZ cannot. We thus compute it as the Schur complement:

glyph[negationslash]

where partitioning of P XX corresponds to partitioning X into Z and the rest. Evaluating a matrixvector product P ZZ u requires a solve of D -1 ( C u ) . This solve can also be performed using conjugate gradients, keeping the computational complexity linear in N but increasing the constants.

4.3 Handling Noisy Observations

Finally, we assume noisy observations, i.e. σ 2 ε > 0 . The inputs can be partially unlabeled, i.e. N = n .

glyph[negationslash]

In this case, matrix K XX in the log-likelihood given by Equation (14) should be substituted with K ZZ + σ 2 ε I . To reduce this to the previously considered cases, we use the Taylor expansion

In practice, we only use the first two terms on the right-hand side as an approximation. This allows to retain linear computational complexity scaling with respect to N but increases the constants.

4.4 Resulting Algorithm

Here we provide a concise summary of the implicit manifold Gaussian process regression algorithm.

Step 1: KNN-index. Construct the KNN index on the points x 1 , .., x N . This allows linear time evaluation of any matrix-vector product with ˜ A , and thus also with A , ∆ rw , P XX for ν ∈ N , etc.

Step 2: hyperparameter optimization. Find the hyperparameters ˆ θ that solve Equation (13). Assuming ν = ˆ ν ∈ N is manually fixed, this relies only on matrix-vector products with ∆ rw .

Step 3: computing the eigenpairs. Fixing the graph bandwidth ˆ α found on Step 2, compute the eigenpairs λ l , f l corresponding to the L smallest eigenvalues λ l . For large N , use Lanczos algorithm.

After the steps above are finished, Equations (8) and (11) define the geometric kernel k X ˆ ν, ˆ κ, ˆ σ 2 ( x , x ′ ) for arbitrary x , x ′ ∈ R d . Then the respective prior GP(0 , k X ˆ ν, ˆ κ, ˆ σ 2 ) can be conditioned by the labeled data in the standard way, yielding the posterior f ( m ) ∼ GP( m ( m ) , k ( m ) ) . To get sensible predictions far away from the data, the geometric model f ( m ) is convexly combined with an independently trained classical Gaussian process model, as given by Equation (12). The resulting predictive model is still a Gaussian process, sum of two appropriately weighted independent Gaussian processes.

Remark. The number of neighbors K , the number of eigenpairs L and the smoothness ν are assumed to be manually fixed parameters. Higher values of K and L improve the quality of approximation of the manifold kernel, which is often linked to better predictive performance, but requires more computational resources. The parameter ν can be picked using cross validation or prior knowledge. Small integer values of ν reduce computational costs, but may be inadequate for higher dimensions of the assumed manifold due to the ν ′ = ν -dim( M ) / 2 link with the manifold kernel smoothness ν ′ .

5 Experiments

We start in Section 5.1 by examining a simple synthetic example to gain intuition on how noisesensitive the technique is. Then in Section 5.2 we consider real datasets, showing improvements in higher dimensions. More experiments, results, and additional discussion can be found in Appendix B

(a) Ground truth

(b) Noiseless, prediction

(c) Low noise, prediction

(d) High noise, prediction

5.1 Synthetic Examples

We consider a one dimensional manifold resembling the shape of a dumbbell which already appeared in Figures 1 and 3. The unknown function f ∗ is defined by fixing a point x ∗ in the top left part of the dumbbell, and computing sin(d( x ∗ , · )) where d( · , · ) denotes the geodesic (intrinsic) distance between a pair of points on the manifold. This function is illustrated in Figure 4a.

To measure performance we primarily rely on measuring negative log-likelihood (NLL) on the dense mesh of test locations. We do this because such metric is able to combine accuracy and calibration simultaneously. Additionally, we present the root mean square error (RMSE).

Weinvestigate a semi-supervised setting where the number of unlabeled points is large ( N -n = 1546 ) and the number of labeled points is small ( n = 10 ). We contaminate the inputs with noise, putting X = X noiseless +N(0 , σ 2 X I ) and do the same with the outputs, putting y = f ∗ ( X ) + N(0 , σ 2 y I ) for various values of σ X , σ y > 0 . Specifically, we consider σ X = σ y = β ∈ { 0 , 0 . 01 , 0 . 05 } to which we refer to as the noiseless setting, the low noise setting and the high noise setting, respectively.

The results for these are visualized in Figures 4b to 4d with performance metrics reported in Table 1. The implicit manifold Gaussian process regression is referred to as IMGP (we use ν = 1 ) and it is compared with the standard Euclidean Matérn5 / 2 Gaussian process. IMGP performs much better in the noiseless and the low noise settings. The high noise is enough to damage the calibration of IMGP, as it ties with the baseline model: NLL is slightly worse and RMSE is slightly better.

In Appendix B.1 we show how performance depends on the fraction n/N of labeled data points, the truncation level L and we discuss the choice of the K parameter in KNN. Additionally, in Appendix B.2 we consider noise-sensitivity for a 2D manifold.

5.2 High Dimensional Datasets

For the high-dimensional setting, we considered predicting rotation angles for MNIST-based datasets. Additionally, we examined a high-dimensional dataset from the UCI ML Repository, CT slices.

5.2.1 Setup

Datasets. We consider two MNIST-based datasets. The first one is created by extracting a single image per digit from the complete MNIST dataset. By randomly rotating these 10 images we obtained N = 10000 training samples and 1000 testing samples. We call it Single Rotated MNIST (SR-MNIST) . For the second dataset, we select 100 random samples from MNIST. By randomly rotating these, we generate N = 100000 training samples, most will be unlabeled, and 10 000 testing samples. We call it Multiple Rotated MNIST (MR-MNIST) . The last dataset, CT slices , has dimensionality of d = 385 , we split it to have N = 24075 training samples and 24075 testing samples. Dataset names can be complemented by the fraction of labeled samples, e.g. MR-MNIST-10% refers to n = 10% N .

Methods. We consider implicit manifold Gaussian processes in the supervised regime ( S-IMGP ) and in the semi-supervised regime ( SS-IMGP ). We compare them to the GPyTorch implementation of the Euclidean Matérn5 / 2 Gaussian Process. We refer to it as the Euclidean Gaussian Process (EGP) .

Additional details. We run 100 iterations of hyperparameter optimization using Adam with a fixed learning rate of 0 . 01 . For MNIST, with use IMGP with ν = 2 ; for CT slices-with ν = 3 .

5.2.2 Results

Table 2 shows the negative log-likelihood metric for different datasets and methods on the test set. The respective RMSEs are presented in Appendix B.3. On SR-MNIST, IMGP outperforms EGP in both supervised and semi-supervised scenarios. MR-MNIST is more challenging. In the supervised setting for n = 1% N , S-IMGP is incapable of inferring the underlying manifold structure, performing worse than EGP. However, SS-IMGP, with more data to infer manifold from, performs best. For n = 10% N , IMGP gets a better grip of the dataset's geometry, outperforming EGP in both regimes.

For CT slices, regardless of n , both S-IMGP and SS-IMGP performed poorly. Looking for an explanation, we considered two modifications. First, we fixed the graph bandwidth ˆ α found in the algorithm's Step 2 (cf. Section 4.4), and re-optimized the other hyperparameters κ, σ 2 , σ 2 ε by maximizing the likelihood of the eigenpair-based model (truncated to L = 2000 eigenpairs) computed in the algorithm's Step 3. This resulted in limited improvement but did not change the big picture-in fact, values for S-IMGP and SS-IMGP in Table 2 for CT slices already include this modification.

Second, on top of this hyperparameter re-optimization, we tried computing the eigenpairs using torch.linalg.eigh instead of the Lanczos implementation in GPyTorch , taking the same number L = 2000 of eigenpairs. The resulting methods S-IMGP (full) and SS-IMGP (full) showed considerable improvement over the baseline, as shown in Table 2. This indicated an issue with the quality of eigenpairs derived from the Lanczos method which requires further investigation. We discuss this in Appendix B.3, together with the aforementioned hyperparameter re-optimization procedure.

6 Conclusion

In this work, we propose the implicit manifold Gaussian process regression technique. It is able to use unlabeled data to improve predictions and uncertainty calibration by learning the implicit manifold upon which the data lies, being inspired by the convergence of graph Matérn Gaussian processes to their manifold counterparts. This helps building better probabilistic models in higher dimensional settings where the standard Euclidean Gaussian processes usually struggle. This is supported by our experiments in a synthetic low-dimensional setting and for high-dimensional datasets. Leveraging sparse structure of graph Matérn precision matrices and efficient approximate KNN, the technique is able to scale to large datasets of hundreds of thousands points, which is especially important in high dimension, where a large number of unlabeled points is often needed to learn the implicit manifold. The model is fully differentiable, making it possible to infer hyperparameters in the usual way.

Limitations. The quality of the constructed graph significantly influences the technique's performance. When dealing with data from complex manifolds or exhibiting highly non-uniform density, simplistic KNN strategies might fail to capture the manifold structure due to their reliance on a single graph bandwidth. In such scenarios, larger values of parameters K and L , or in high dimensions, of parameter ν , may be beneficial but could substantially increase computational costs. Furthermore, larger datasets coupled with high parameter values can lead to numerical stability issues, for instance, in the Lanczos algorithm, calling for further improvements and research. Despite these challenges, our method shows promise for advancing probabilistic modeling in higher dimensions.

References

Azangulov, A. Smolensky, A. Terenin, and V. Borovitskiy. Stationary Kernels and Gaussian Processes on Lie Groups and their Homogeneous Spaces I: the compact case. arXiv preprint arXiv:2208.14960 , 2022. Cited on pages 2, 3.

Azangulov, A. Smolensky, A. Terenin, and V. Borovitskiy. Stationary Kernels and Gaussian Processes on Lie Groups and their Homogeneous Spaces II: non-compact symmetric spaces. arXiv preprint arXiv:2301.13088 , 2023. Cited on page 2.

Binois and N. Wycoff. A survey on high-dimensional Gaussian process modeling with application to Bayesian optimization. ACM Transactions on Evolutionary Learning and Optimization , 2(2):126, 2022. Cited on page 2.

Borovitskiy, I. Azangulov, A. Terenin, P. Mostowsky, M. Deisenroth, and N. Durrande. Matérn Gaussian Processes on Graphs. In International Conference on Artificial Intelligence and Statistics , 2021. Cited on pages 2, 3.

Borovitskiy, M. R. Karimi, V. R. Somnath, and A. Krause. Isotropic Gaussian Processes on Finite Spaces of Graphs. In International Conference on Artificial Intelligence and Statistics , 2023. Cited on page 2.

Borovitskiy, A. Terenin, P. Mostowsky, and M. P. Deisenroth. Matérn Gaussian Processes on Riemannian Manifolds. In Advances in Neural Information Processing Systems , 2020. Cited on pages 2, 3, 14.

Calder and N. G. Trillos. Improved spectral convergence rates for graph Laplacians on ε -graphs and k-NN graphs. Applied and Computational Harmonic Analysis , 60:123-175, 2022. Cited on page 4.

J.-P. Chilès and P. Delfiner. Geostatistics: Modeling Spatial Uncertainty . John Wiley & Sons, 2012. Cited on page 1.

R. Coifman and S. Lafon. Diffusion Maps. Applied and Computational Harmonic Analysis . Special Issue: Diffusion Maps and Wavelets, 21(1):5-30, 2006. Cited on pages 2, 4.

De Vito, N. Mücke, and L. Rosasco. Reproducing kernel Hilbert spaces on manifolds: Sobolev and diffusion spaces. Analysis and Applications , 19(03):363-396, 2021. Cited on page 14.

Deisenroth and C. E. Rasmussen. PILCO: A model-based and data-efficient approach to policy search. In International Conference on Machine Learning , 2011. Cited on page 1.

Donnelly. Eigenfunctions of the Laplacian on compact Riemannian manifolds. Asian Journal of Mathematics , 10(1):115-126, 2006. Cited on page 14.

B. Dunson, H.-T. Wu, and N. Wu. Graph based Gaussian processes on restricted domains. Journal of the Royal Statistical Society Series B: Statistical Methodology , 84(2):414-439, 2022. Cited on page 2.

B. Dunson, H.-T. Wu, and N. Wu. Spectral convergence of graph Laplacian and heat kernel reconstruction in L ∞ from random samples. Applied and Computational Harmonic Analysis , 55:282-336, 2021. Cited on pages 2, 4.

Feragen, F. Lauze, and S. Hauberg. Geodesic exponential kernels: When curvature and linearity conflict. In Conference on Computer Vision and Pattern Recognition , 2015. Cited on page 3.

García Trillos, M. Gerlach, M. Hein, and D. Slepˇ cev. Error estimates for spectral convergence of the graph Laplacian on random geometric graphs toward the Laplace-Beltrami operator. Foundations of Computational Mathematics , 20(4):827-887, 2020. Cited on page 4.

Gardner, G. Pleiss, K. Q. Weinberger, D. Bindel, and A. G. Wilson. GPyTorch: Blackbox MatrixMatrix Gaussian Process Inference with GPU Acceleration. In Advances in Neural Information Processing Systems , 2018. Cited on pages 7, 18.

Gneiting. Strictly and non-strictly positive definite functions on spheres. Bernoulli , 19(4):13271349, 2013. Cited on page 3.

S. Gradshteyn and I. M. Ryzhik. Table of Integrals, Series, and Products . Academic Press, 7th edition, 2014. Cited on page 3.

Hein, J.-Y. Audibert, and U. von Luxburg. Graph Laplacians and their Convergence on Random Neighborhood Graphs. Journal of Machine Learning Research , 8(48):1325-1368, 2007. Cited on page 4.

Hennig, M. A. Osborne, and M. Girolami. Probabilistic numerics and uncertainty in computations. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences , 471(2179):20150142, 2015. Cited on page 1.

F. Hutchinson. A stochastic estimator of the trace of the influence matrix for Laplacian smoothing splines. Communications in Statistics-Simulation and Computation , 18(3):1059-1076, 1989. Cited on page 7.

Johnson, M. Douze, and H. Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data , 7(3):535-547, 2019. Cited on page 6.

I. Kondor and J. Lafferty. Diffusion kernels on graphs and other discrete structures. In International Conference on Machine Learning , 2002. Cited on page 2.

A. Lee and M. Verleysen. Nonlinear Dimensionality Reduction . Springer Science & Business Media, 2007. Cited on page 4.

Lindgren, H. Rue, and J. Lindström. An Explicit Link between Gaussian Fields and Gaussian Markov Random Fields: The Stochastic Partial Differential Equation Approach. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 73(4):423-498, 2011. Cited on page 3.

Ma and Y. Fu. Manifold Learning Theory and Applications . CRC press, 2011. Cited on page 4.

Meurant. The Lanczos and conjugate gradient algorithms: from theory to finite precision computations . SIAM, 2006. Cited on pages 7, 8.

Pavutnitskiy, S. O. Ivanov, E. Abramov, V. Borovitskiy, A. Klochkov, V. Vyalov, A. Zaikovskii, and A. Petiushko. Quadric hypersurface intersection for manifold learning in feature space. In International Conference on Artificial Intelligence and Statistics , 2022. Cited on page 4.

L. Peach, M. Vinao-Carl, N. Grossman, M. David, E. Mallas, D. Sharp, P. A. Malhotra, P. Vandergheynst, and A. Gosztolai. Implicit Gaussian process representation of vector fields over arbitrary latent manifolds. arXiv preprint arXiv:2309.16746 , 2023. Cited on page 2.

Rahimi and B. Recht. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems , 2007. Cited on page 7.

E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning . Adaptive Computation and Machine Learning. MIT Press, 2006. Cited on pages 1, 2.

Wenger, G. Pleiss, P. Hennig, J. Cunningham, and J. Gardner. Preconditioning for scalable Gaussian process hyperparameter optimization. In International Conference on Machine Learning , pages 23751-23780. PMLR, 2022. Cited on page 8.

Whittle. Stochastic Processes in Several Dimensions. Bulletin of the International Statistical Institute , 40:974-994, 1963. Cited on page 3.

T. Wilson, V. Borovitskiy, A. Terenin, P. Mostowsky, and M. P. Deisenroth. Efficiently Sampling Functions from Gaussian Process Posteriors. In International Conference on Machine Learning , 2020. Cited on page 2.

T. Wilson, V. Borovitskiy, A. Terenin, P. Mostowsky, and M. P. Deisenroth. Pathwise Conditioning of Gaussian Processes. Journal of Machine Learning Research , 22(105):1-47, 2021. Cited on page 2.

A Theory

Proposition 1. Denote the eigenpairs by λ l , f l for a graph Laplacian and by λ M l , f M l for the Laplace-Beltrami operator. Fix δ > 0 . Assume that, with probability at least 1 -δ , for all ε > 0 , for α small enough and for N large enough we have | λ l -λ M l | < ε and | f l ( x i ) -f M l ( x i ) | < ε . Then, with probability at least 1 -δ , we have k N,α,L ν,κ,σ 2 ( x i , x j ) → k ν,κ,σ 2 ( x i , x j ) as α → 0 , N,L →∞ , where

Proof. Fix small ε > 0 . We will prove that for α small enough and N,L large enough we have | k ν,κ,σ 2 f ( x i , x j ) -k N,α,L ν,κ,σ 2 f ( x i , x j ) | < Cε for some C > 0 with probability at least 1 -δ . Since the assumption holds on the same event of probability 1 -δ for all ε , this directly translates to the convergence on the same event. In fact, a probabilistic narrative is nonessential for what we actually prove, and we do not use it below. To simplify notation, we replace ∑ L -1 l =0 by ∑ L l =0 .

First, for any L ∈ Z > 0 define the truncated version k L ν,κ,σ 2 f of the manifold kernel k ν,κ,σ 2 f by

The manifold Matérn kernels are the reproducing kernels of Sobolev spaces, if the latter are defined appropriately (Borovitskiy et al., 2020). These are Mercer kernels (De Vito et al., 2021), hence, by Mercer's theorem, k L ν,κ,σ 2 f → k ν,κ,σ 2 f uniformly on M , i.e. for L large enough we have

Now suppose that α is small enough and N is large enough so that

for all l ∈ { 1 , .., L } and for all i ∈ { 1 , .., N } with probability at least 1 -δ .

Assuming the manifold is connected, by Donnelly (2006) we have | f M l | ≤ C λ ( λ M l ) d -1 4 for l > 0 , where C λ > 0 is a constant that depends on the geometry of the manifold. The case l = 0 is special because λ M 0 = 0 . Since f M 0 is a constant function, we have

for all l ≥ 0 , where C λ here is potentially different from the C λ before. Assuming, without loss of generality, ε < 1 , we have

The function Φ( λ ) = ( 2 ν κ 2 + λ ) -ν -dim( M ) / 2 is Lipschitz: | Φ( λ ) -Φ( λ ′ ) | ≤ C Φ | λ -λ ′ | where

Define an auxiliary kernel with manifold eigenvalues and graph eigenfunctions by

Then

Also, noting that | f l ( x i ) | ≤ ∣ ∣ f M l ( x i ) -f l ( x i ) ∣ ∣ + ∣ ∣ f M l ( x i ) ∣ ∣ ≤ ε ′ + C λ max(( λ M l ) d -1 4 , 1) , write

Finally,

This proves the claim.

Proposition 2. Assuming ν ∈ N , the precision matrix P XX of k X ν,κ,σ 2 ( x i , x j ) can be given by

Proof. The covariance matrix K XX is given by

where f l = D -1 / 2 f sym l and f sym l are the orthonormal eigenvectors of the symmetric normalized Laplacian ∆ sym . Denote

then ( K sym XX ) -1 = σ -2 C -1 ν,κ ∑ L -1 l =0 ( 2 ν κ 2 + λ l ) ν f sym l ( f sym l ) glyph[latticetop] = σ -2 C -1 ν,κ ( 2 ν κ 2 I + ∆ sym ) ν . We have

1.0

800

1.0

IMGP

2.0

IMGP

0.8

0.6

100

600

100

200

0.8

0.6

EGP

2.5

EGP

200

400

1000

0.4

1000

0.4

3.0

0.2

200

0.2

0.0

0

0.0

3.5

0

500

1000

1500

0

500

1000

1500

10

30

50

70

90

10

30

50

70

90

On the other hand,

B Additional Experimental Results and Details

Here we provide additional results and details for synthetic examples and high-dimensional datasets.

B.1 1D Dumbbell Manifold

In this section, we analyze our method's sensitivity to the number of labeled points, spectrum truncation, and neighbor count in graph construction, offering deeper insights into its behavior.

Eigenpairs Truncation. From a theoretical standpoint, utilizing the complete set of eigenpairs to construct the kernel should be beneficial. However, this assumption may not hold true in cases where the optimization problem is not fully converged. The trends of RMSE and NLL, as depicted in Figures 5a and 5b, illustrate the impact of increasing the number of eigenpairs for 100, 200, and 1000 iterations. In situations where hyperparameter optimization does not converge fully, the length scale parameter κ fails to reach sufficiently high values necessary to generate appropriate spectral density decay, which in turn would properly weigh higher frequency eigenpairs. In such scenarios, truncating the spectrum is similar to increasing the length scale, which might improve results in certain cases. In the 1D scenario, due to its simplicity, we opted for a modest number of eigenpairs, namely L = 50 . Exceeding this count did not yield any discernible improvements.

Dataset Size. We analyze the sensitivity to dataset size of our method (IMGP) against the Euclidean case (EGP) in the semi-supervised learning scenario. For fixed number of eigenpairs, L = 50 , Figures 5c and 5d show performance metrics (RMSE and NLL) depending on the percentage n % N of labeled points. In a typical scenario where we have at our disposal fewer labeled points compared to the number of unlabeled points, our method outperforms EGP in both accuracy (RMSE) and uncertainty quantification (NLL). As n increases EGP starts to match the performance of IMGP.

Number of neighbors. For the one dimensional case considered in this section, only three neighbors are necessary to capture the essential features of the manifold's geometry. Choosing a number less than three would hinder the algorithm's ability to capture the underlying manifold structure. On the other hand, increasing the number of neighbors beyond this threshold does not affect the solution, provided that sufficient time is allocated for hyperparameter optimization to converge and the graph bandwidth becomes small enough to 'correct' for all undesired edges in the graph structure (for instance in the central region of the dumbbell). In high-dimensional problems where the dimension of the underlying manifold is unknown, this suggests to incrementally increase the number of neighbors until the loss function stops improving, if it is computationally feasible to try multiple values of K .

(a) Ground truth

(b) IMGP pred.

β

= 0

(c)IMGP pred.

β

=0

.

01

(d)IMGP pred.

β

=0

.

05

(e) Euclidean, prediction

(f) Euclidean, uncertainty

(g) IMGP, uncertainty

1.80

1.20

0.600

0.00

-0.600

-1.20

-1.80

-2.40

1.80

0.536

0.459

1.20

0.484

0.409

0.600

0.432

0.358

0.00

0.380

0.307

-0.600

0.327

0.256

-1.20

0.275

0.205

-1.80

0.223

0.155

-2.40

0.171

0.104

B.2 2D Dragon Manifold

We consider another synthetic setting, a complex 2D manifold, depicted in Figure 6. The ground truth function is visualized in Figure 6a, it is the same as in the one-dimensional case, the sine of the geodesic distance to a point, which is located in the green area.

In the semi-supervised learning scenario, Figures 6e and 6b offer a comparison of the posterior mean between the Euclidean GP and IMGP, while Figures 6f and 6g illustrate the posterior standard deviation for both models-quite similar to each other, in this regime. Similar to what we did for the 1D compact manifold in Section 5.1, we evaluated the performance of IMGP under varying levels of sampling noise. Figures 6b to 6d display the IMGP predictions in the semi-supervised learning scenario as sampling noise increases. Similar to the 1D case, our approach consistently outperforms the standard Euclidean GP, as evidenced in Table 3.

Remark. We observed that for higher levels of sampling noise, the linear combination of posteriors, as described by Equation (12), significantly outperform the single geometric model.

B.3 High Dimensional Datasets

Rotated MNIST. For IMGP, we use ν = 2 . As discussed in Appendix B.1, the optimal number of eigenpairs L varies considerably depending on the convergence of the optimization problem. Given the limited number of iterations per run we opted to fix the number of eigenpairs at 20% N and 2% N , for SR-MNIST and MR-MNIST, respectively. Table 4 reports the complete results obtained for the rotated MNIST dataset, including both RMSE and NLL. Notably, these emphasize the importance of unlabeled points, as S-IMGP performs worse than both SS-IMGP and EGP on MR-MNIST-1%.

CT slices. For IMGP, we use ν = 3 . In Table 5 we report results for IMGP-S (full) and IMGP-SS (full). These are based on the 'exact' eigenpairs, computed by torch.linalg.eigh , as opposed to the standard Lanczos implementation we use by default. Additionally, these include the hyperparameter re-optimization step, as described in Section 5.2 and discussed below. Regarding RMSE, all three compared methods-IMGP-S (full) and IMGP-SS (full) and EGP-exhibit similar performance, with a slight advantage for SS-IMGP (full) as the number of training points increases. When considering NLL, as previously observed with MNIST, SS-IMGP performs best in all settings. However, NLL decreases for larger values of n/N , indicating a probable overfit in this regime.

Hyperparameter re-optimization. We observed this step to serve two important purposes: (1) fixing overly small values of the signal variance σ 2 , potentially caused by the absence of covariance normalization in the optimization process and poor convergence; (2) adjusting the length scale parameter to take into account the loss of high-frequency components due to truncation.

Implementation. Currently, the implementation faces two significant limitations that might restrict its usability to high-memory hardware setups. Firstly, due to the absence of rich enough sparse matrix routines in PyTorch, we had to develop our own custom implementation of differentiable sparse operators. We kept to high-level routines, which forced us to strike a balance between performance and memory efficiency. In particular, for matrix-vector sparse product operations, our approach relies on highly optimized vectorized code, delivering high performance on GPU at the expense of increased memory allocation. Secondly, we faced challenges with sparse eigen-solvers in the PyTorch ecosystem. Our attempts of using the PyTorch implementation of the LOBPCG algorithm, torch.lobpcg , yielded relatively poor results. We had similar experience with the Lanczos implementation from GPyTorch (Gardner et al., 2018). In light of this, when extracting higherfrequency components was needed, we resorted to two alternative solutions: PyTorch dense matrix eigen-decomposition torch.linalg.eigh and the SciPy Arpack wrapper scipy.linalg.eigsh . The first approach, while benefiting from GPU acceleration, can be infeasible because of limited GPU memory. The second approach, known for its efficiency in memory usage due to its Krylov subspace-based nature, is constrained to CPU utilization, significantly impacting the algorithm's overall performance.

C Hyperparameter Priors and Initialization

Here we describe hyperpameter priors which might be of help when using implicit manifold Gaussian process regression.

Graph Bandwidth α . Graph Laplacian converges to the Laplace-Beltrami operator when α tends to zero, motivating smaller α . However, in a non-asymptotic setting it is impractical to have α overly small as it will render the graph effectively disconnected and cause numerical instabilities. One appropriate prior could thus be gamma distribution , whose right tail discourages high values of α , and which, given an appropriate choice of the parameters, encourages α to align with the scale of pairwise distances between graph nodes. Specifically, we choose the parameters of the gamma distribution so as to (1) match its mode with the median ( Q 2 ) of pairwise average distance between K -th nearest neighbors because such an α would give reasonable weights in the KNN graph and (2) so as to place its standard 0 . 95 confidence interval to the right of a certain lower bound ¯ α we define further that ensures the graph is numerically not disconnected. Let

1000

Histogram

50th percentile

750

αLowerBound

b

Probability Density Curve

Frequency

500

250

0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Average DistanceValues

(a) Prior for the semi-supervised case

Probability

200

Histogram

50th percentile

150

αLowerBound

ProbabilityDensity Curve

Frequency/

100

50

0

0

2

4

8

Average DistanceValues

(b) Prior for the supervised case

The bandwidth's lower bound we compute as

where τ is a user-defined parameter.

Let η and β the shape and rate parameters of the gamma distribution. In order to achieve (1), we define

where ρ is used to achieve (2) and is given by

Considering the S-MNIST dataset, Figure 7 shows the bandwidth prior distribution for the semisupervised (labeled and unlabeled points) and the supervised (labeled points) scenarios.

Signal Variance σ 2 f . Assuming normalized y i , the signal variance σ 2 f should be close to 1 . Because of this a natural prior for σ 2 is the truncated normal (onto the set of positive reals σ 2 > 0 ), with mode at 1 . The pre-truncation variance can be chosen, for example, to have 0 at 3 standard deviations away from the mode, i.e. it can be chosen to be 1 / 9 . Note that setting a prior over the signal variance parameter requires evaluating the normalization constant C ν,κ 7 for the kernel at each hyperparameter optimization step, something that can be avoided otherwise.

Noise Variance σ 2 ε . Choosing a prior for the noise variance σ 2 ε is heavily problem-dependent. Truncated normal (onto the set σ 2 ε > 0 ) with mode at 0 could be a reasonable option. The pretruncation variance can be chosen, for example, to be 1 , 1 / 4 or 1 / 9 , placing the value 1 , which is the variance of the normalized observations y i , at 1 , 2 or 3 standard deviations away from the mode.

Length scale The interpretation and the scale of the length scale parameter is manifold-specific. This makes it very difficult to come up with any reasonable prior. Because of this, we suggest actually leaving the length scale parameter free.

Parameter initialization When doing MAP estimation, one can initialize parameters randomly, sampling them from respective priors.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe curse of dimensionality demonstrates in Gaussian process models in that they’re default kernel choices that often depend on the Euclidean distance between 2 points. Euclidean distance is a poor metric especially in high-dimensional settings, where we expect data points to lie on an, often unknown, manifold. This paper combines findings from multiple precursor works in assembling a tool set to for Gaussian process regression with a Metern kernel, while implicitly inferring the underlying manifold structure.\n\nSTRENGTHS:\nI believe that while the paper mostly combines existing theoretical results in manifold learning and numerical analysis, it presents a compelling, novel framework of tackling manifold learning with Gaussian process regression. \n\nThe paper is largely self-sufficient: it tackles the complex problem of learning manifolds in the context of Gaussian process by proposing solutions to all facets of the problem: characterizing manifolds with graph Laplacian, gradient-based learning and scalability.\n\nWEAKNESSES:\nI think the paper lacks a presentation of _how_ learning the manifold structure aids in the predictive performance of the rotated MNIST dataset. While one can clearly see _why_ the learning of an implicit manifold helps in predicting rotation angles for handwritten digit, as rotation expressly traverses around a nontrivial manifold, the benefits of this work go beyond simple improvements in prediction, but interpretability. While the authors did a very good job in demonstrating how the model correctly inferred the dumbbell-shaped manifold, I believe similar illustrations can also be done in the MNIST rotations.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\n- I believe the presentation of the paper could be improved by a clear separation of predecessor works and original contribution: for example, Section 3.1 is almost solely composed of existing work except for the theoretical contribution of the Matern kernel convergence — an overall marginal contribution in the paper. \n- Table 3: “RMSE” in the table seems confusing for prediction of rotating angles. Is it so that the author is referring to the shortest arc as a metric for distance in rotations? \n- I believe that the paper could benefit from a illustration of what implicit manifold is learned from the rotated MNIST data: for example, a display of what the kernel values matrix looks like along the trajectory of a rotation compared to the Euclidean kernel matrix, or something else showing the geodesics of the points on the rotation trajectory.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors propose a novel methodology for doing GP regression which is able to learn the implicit structure from the data. This is particularly useful in high-dimensional problems, where the data  lies on low-dimensional manifold. The proposed methods allows to learn the implicit manifold in a fully differentiable way, using nearest neighbour graph.\n\nThe method is based on Matern GPs on manifolds and graphs (Borovitskiy, 2020, 2021). To approximate the eigenvalues of the Laplace-Beltrami operator on the implicit data manifold, authors use random walk normalized graph Laplacian (on KNN graph) weighted to overcome possible non-uniform sampling density.\n\nThis work leverages efficient approximation to the KNN, sparse structure of the precision matrices, and RFF kernel approximation to be able to scale to large datasets.\n\nThe model is tested on a synthetic dumbbell-shape manifold and on rotated MNIST data.\n\nSTRENGTHS:\nThis is, to the best of my knowledge, a novel methodology for GP regression. This work is build on solid foundation and the method is derived naturally from Matern GPs on manifolds and graphs. Authors cleverly use a variety of techniques to keep good scalability. \n\nIn my opinion, the main contribution of this work is in the elegant assembly of the right methods, which gives a practically usable model for a well-known problem.\n\nThe manuscript is written clearly and is easy to follow.\n\nWEAKNESSES:\nDue to the incremental nature of the paper, it is not always clear what is a novel contribution of this work and what is a background from previous works (example, last paragraph of 2.1). \n\nThe experiments are somewhat limited. The method is tested on one synthetic dataset and a few versions of rotated MNIST. In the MNIST dataset, however, the added rotation must be inducing a lot of structure. It would be beneficial to showcase the method on a real dateset where an implicit manifold is already existing in the data.\n\nQUESTIONS:\n1. Could you please clarify on the following: There are no results on spectral convergence for KNN graphs independent of data sampling density. You use a weighing scheme to deal with non-uniform data sampling. You do not claim that it implies spectral convergence. How much of a problem is it, both theoretically and practically?\n\nSuggestions:\n1. In figure 2, perhaps use different colours foe kernel and sample plots.\n2. eq(11) I believe $\\lambda_i$ should be $\\lambda_l$?\n\nLIMITATIONS:\nThe limitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a methodology to extend Matern processes on implicit manifolds, which are modeled by $K$-NN graphs, and the kernel relies on the set of eigenvalues/vectors of the associated graph-Laplacian. An approximation of the eigenfunctions based on the eigenvectors is provided, which together with an additional trick allows to extend the process in the vicinity of the implicit manifold. The hyperparameters are learned in supervised and semi-supervised scenarios via a differentiable objective, and the performance is demonstrated in the experiments.\n\nSTRENGTHS:\n- The technical part of the paper seems correct and sensible, but I have not checked the derivations in details.\n- The idea is a direct and relatively simple extension of previous work.\n- The results mainly due to the synthetic experiment seem convincing.\n- The paper in general is well-written.\n\nWEAKNESSES:\n- The main drawback of graph-based methods is typically the construction of the graph and how well this recovers the actual structure of the underlying manifold. I suppose that the current method has the same limitation that is not clearly discussed/analyzed in the paper.\n- As the methodology applies to general manifolds, I would expect to see more empirical results. For example, some synthetic 3d surfaces and more complicated high-dimensional real-world datasets as the rotation of MNIST is closer to a synthetic experiment.\n\nQUESTIONS:\nQ1. Lines 98-100: Is there any reference that plugging in Eq. 3 the graph Laplacian and the Gaussian noise gives a Matern process on the graph?\n\nQ2. Why the dimension of the implicit manifold (graph) is 0? I suppose that the underlying manifold has some dimensionality. I think that there are approaches relying on graphs that try to estimate this dimensionality.\n\nQ3. How sensitive is the method with respect to the graph construction? In particular, how the number of neighbors $K$ influences the result? Can this be estimated somehow e.g. by cross-validation?\n\nQ4. How the method behaves if there are more than one connected components in the dataset? \n\nQ5. Lines 144-150: The graph Laplacian converges to the Laplace-Beltrami (in terms of eigenvalues/vectors/functions) for $K$-NN graphs as long as the data distribution is uniform on the manifold? While for non-uniform distributions such a convergence result does not exist?\n\nQ6. Lines 177-179: I think that here the gist is unclear.\n\nQ7. Which Nyström method is used?\n\nQ8. Fig 3. I think it is interesting to see the actual eigenvector on the data (e.g. color the points) and how this correlates to the extended version.\n\nQ9. Eq. 11 the eigenvalue should be probably $\\lambda_l$ and not $\\lambda_j$?\n\nLIMITATIONS:\nThe authors do not discuss the limitations of their work, and in particular, the construction of the graph which I think is critical for the performance of the model. The methodology that is propose does not have a direct negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work extends the reach of Matern Gaussian processes to additionally learning the implicit low-dimensional (unknown) manifold the data lives on - the existence of such a manifold is suggested by the manifold hypothesis. The theory draws from existing Laplacian Matern Gaussian processes (Borovitskiy et al., 2020). The model they propose is differentiable wrt all model and geometry hypers. It is able to scale to thousands of data points by leveraging the sparse structure of Mather precision matrices. They provide a way to extend predictions to the whole R^{d} space by reverting to a Euclidean GP away from the manifold.  The experimental evaluation shows support for the implicit manifold Gaussian process.\n\nSTRENGTHS:\n- The end-to-end differentiability where the kernel hyperparameters along with those that parameterise the underlying geometry can be learnt simultaneously using a single objective. \n- Scalability is achieved as the precision matrix corresponding to the KNN implied graph is sparse.  \n- The semi-supervsied case is interesting where a large amount of unlabelled data is leveraged to infer the underlying geometry of the manifold through a weighted nearest neighbour  graph.\nAll these points contribute to the quality and significance of the work.\n\nWEAKNESSES:\n- The exposition could benefit from an algorithm style pseudocode for the training and prediction steps, it should start with the interpretation of the data as a graph, computation of the elements needed for the Matern kernel etc - as I am a bit confused about the order of the steps. \n\n- While the details are all there in section 3.2, I think there need to be separate sections for construction of the matern kernel on graph nodes, computing the dependencies ie. the eigenpairs of the graph Laplacian, computing the kernel on arbitrary vectors in R^{d}, computing the predictive posterior. \n\n- I dont understand this line - pls explain or rewrite. Basically, why dont you just say how do you compute dim(M)?\n\nEssentially, from the theoretical results of Section 3.1 we borrow a particular weighting scheme, aiming to cancel out the possibly non-uniform density, and a specific choice of the graph Laplacian (the random walk normalized one).\n\nQUESTIONS:\n- The final predictive distribution is a mixture of Gaussians - one is the canonical Euclidean posterior predictive with SE-ARD kernel and the other is the geometry aware kernel underlying the same canonical posterior predictive equations? \n \n- What is the order of the steps in training - the log marginal likelihood depends entails the precision matrix which depends on the eigenpairs, but you say in line 206 that they are computed after the hyperparameters are found, these are mentioned in line 216 - is the \\hat{\\kappa} the number of neighbours of the KNN graph?\n\nLIMITATIONS:\nI think authors should devote a paragraph to this. I don't see limitations discussed anywhere.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}