Title: Posterior Contraction Rates for Matérn Gaussian Processes on Riemannian Manifolds
Venue: NeurIPS 2023
Accepted: yes
Keywords: Gaussian processes

posterior contraction

manifolds

kernels
Abstract: Gaussian processes are used in many machine learning applications that rely on uncertainty quantification. Recently, computational tools for working with these models in geometric settings, such as when inputs lie on a Riemannian manifold, have been developed. This raises the question: can these intrinsic models be shown theoretically to lead to better performance, compared to simply embedding all relevant quantities into $\mathbb{R}^d$ and using the restriction of an ordinary Euclidean Gaussian process? To study this, we prove optimal contraction rates for intrinsic Matérn Gaussian processes defined on compact Riemannian manifolds. We also prove analogous rates for extrinsic processes using trace and extension theorems between manifold and ambient Sobolev spaces: somewhat surprisingly, the rates obtained turn out to coincide with those of the intrinsic processes, provided that their smoothness parameters are matched appropriately. We illustrate these rates empirically on a number of examples, which, mirroring prior work, show that intrinsic processes can achieve better performance in practice. Therefore, our work shows that finer-grained analyses are needed to distinguish between different levels of data-efficiency of geometric Gaussian processes, particularly in settings which involve small data set sizes and non-asymptotic behavior.

Posterior Contraction Rates for Matérn Gaussian Processes on Riemannian Manifolds

Abstract

Gaussian processes are used in many machine learning applications that rely on uncertainty quantification. Recently, computational tools for working with these models in geometric settings, such as when inputs lie on a Riemannian manifold, have been developed. This raises the question: can these intrinsic models be shown theoretically to lead to better performance, compared to simply embedding all relevant quantities into R d and using the restriction of an ordinary Euclidean Gaussian process? To study this, we prove optimal contraction rates for intrinsic Matérn Gaussian processes defined on compact Riemannian manifolds. We also prove analogous rates for extrinsic processes using trace and extension theorems between manifold and ambient Sobolev spaces: somewhat surprisingly, the rates obtained turn out to coincide with those of the intrinsic processes, provided that their smoothness parameters are matched appropriately. We illustrate these rates empirically on a number of examples, which, mirroring prior work, show that intrinsic processes can achieve better performance in practice. Therefore, our work shows that finer-grained analyses are needed to distinguish between different levels of data-efficiency of geometric Gaussian processes, particularly in settings which involve small data set sizes and non-asymptotic behavior.

1 Introduction

Gaussian processes provide a powerful way to quantify uncertainty about unknown regression functions via the formulation of Bayesian learning. Motivated by applications in the physical and engineering sciences, a number of recent papers [11, 9, 10, 37] have studied how to extend this model class to spaces with geometric structure, in particular Riemannian manifolds including important special cases such as spheres and Grassmannians [4], hyperbolic spaces and spaces of positive definite matrices [5], as well as general manifolds approximated numerically by a mesh [11].

These Riemannian Gaussian process models are starting to be applied for statistical modeling, and decision-making settings such as Bayesian optimization. For example, in a robotics setting, Jaquier et al. [27] has shown that using Gaussian processes with the correct geometric structure allows one to learn quantities such as the orientation of a robotic arm with less data compared to baselines. The same model class has also been used by Coveney et al. [15] to perform Gaussian process regression on a manifold which models the geometry of a human heart for downstream applications in medicine.

Given these promising empirical results, it is important to understand whether these learning algorithms have good theoretical properties, as well as their limitations. Within the Bayesian framework, a natural way to quantify data-efficiency and generalization error is to posit a data-generating mech-

(a) Intrinsic

(b) Extrinsic

(c) Intrinsic

(d) Extrinsic

anism model and study if-and how fast-the posterior distribution concentrates around the true regression function as the number of observations goes to infinity.

Within the Riemannian setting, it is natural to compare intrinsic methods, which are formulated directly on the manifold of interest, with extrinsic ones, which require one to embed the manifold within a higher-dimensional Euclidean space. For example, the two-dimensional sphere can be embedded into the Euclidean space R 3 : intrinsic Gaussian processes model functions on the sphere while extrinsic ones model functions on R 3 , which are then restricted to the sphere. Are the former more efficient than the latter? Since embeddings-even isometric ones-at best only preserve distances locally, they can induce spurious dependencies, as points can be close in the ambient space but far away with respect to the intrinsic geodesic distance: this is illustrated in Figure 1. In cases where embeddings significantly alter distances, one can expect intrinsic models to perform better, and it is therefore interesting to quantify such differences.

In other settings, the manifold on which the data lies can be unknown, which makes using intrinsic methods directly no longer possible. There, one would like to understand how well extrinsic methods can be expected to perform. According to the manifold hypothesis [18], it is common for perceptual data such as text and images to concentrate on a lower-dimensional submanifold within, for instance, pixel space or sequence space. It is therefore also interesting to investigate how Gaussian process models-which, being kernel-based, are simpler than for instance deep neural networks-perform in such scenarios, at least in the asymptotic regime.

In this work, we develop geometric analogs of the Gaussian process posterior contraction theorems of van der Vaart and van Zanten [56]. More specifically, we derive posterior contraction rates for three main geometric model classes: (1) the intrinsic Riemannian Matérn Gaussian processes, (2) truncated versions of the intrinsic Riemannian Matérn Gaussian processes, which are used in practice to avoid infinite sums, and (3) the extrinsic Euclidean Matérn Gaussian processes under the assumption that the data lies on a compact Riemannian manifold. In all cases, we focus on IID randomly-sampled input points-commonly referred to as random design in the literature-and contraction in the sense of the L 2 ( p 0 ) distance, defined in Section 2. We focus on compact Riemannian manifolds: this allows one to define Matérn Gaussian processes through their Karhunen-Loève expansions, which requires a discrete spectrum for the Laplace-Beltrami operator-see for instance Borovitskiy et al. [11] and Chavel [13], Chapter 1-and is a common setting in statistics [39].

Contributions. We show that all three classes of Gaussian processes lead to optimal procedures, in the minimax sense, as long as the smoothness parameter of the kernel is aligned with the regularity of the unknown function. While this result is natural-though non trivial-in the case of intrinsic Matérn processes, it is rather remarkable that it also holds for extrinsic ones. This means that in order to understand their differences better, finite-sample considerations are necessary. We therefore present experiments that compute the worst case errors numerically. These experiments highlight that intrinsic models are capable of achieving better performance in the small-data regime. We conclude with a discussion of why these results-which might at first seem counterintuitive-are very natural

when viewed from an appropriate mathematical perspective: they suggest that optimality is perhaps best seen as a basic property or an important guarantee that any sensible model should satisfy.

2 Background

Gaussian process regression is a Bayesian approach to regression where the modeling assumptions are y i = f ( x i ) + ε i , with ε i ∼ N(0 , σ 2 ε ) , x i ∈ X , and f is assigned a Gaussian process prior. A Gaussian process is a random function f : X → R for which all finite-dimensional marginal distributions are multivariate Gaussian. The distribution of such a process is uniquely determined by its mean function m ( · ) = E ( f ( · )) and covariance kernel k ( · , · ′ ) = Cov( f ( · ) , f ( · ′ )) , hence we write f ∼ GP( m,k ) .

For Gaussian process regression, the posterior distribution given the data is also a Gaussian process with probability kernel Π( · | x , y ) = GP( m Π( ·| x , y ) , k Π( ·| x , y ) ) , see Rasmussen and Williams [41],

These quantities describe how incorporating data updates the information contained within the Gaussian process. We will be interested studying the case where X is a Riemannian manifold, but first review the existing theory on the asymptotic behaviour of the posterior when X = [0 , 1] d .

2.1 Posterior Contraction Rates

Posterior contraction results describe how the posterior distribution concentrates around the true data generating process, as the number of observations increases, so that it eventually uncovers the true data-generating mechanism. The area of posterior asymptotics is concerned with understanding conditions under which this does or does not occur, with questions of posterior contraction rates -how fast such convergence occurs-being of key interest. At present, there is a well-developed literature on posterior contraction rates, see Ghosal and van der Vaart [20] for a review.

In the context of Gaussian process regression with random design , which is the focus of this paper, the true data generating process is assumed to be of the form

where f 0 ∈ F ⊆ R X , a class of real-valued functions, and N( µ, σ 2 ) denotes the Gaussian with moments µ, σ 2 . Note that, in this particular variant, these equations exactly mirror those of the Gaussian process model's likelihood, including the use of the same noise variance σ 2 ε in both cases: in this paper, we focus on the particular case where σ ε is known in advance. This setting is restrictive, one can extend to an unknown σ ε > 0 using techniques that are not specific to our geometric setting: for instance, the approach of [55] allows to handle an unknown σ ε if one assumes an upper and lower bound on it and keep the same contraction rates. In practice, more general priors, including ones that do not assume an upper or lower bound on σ ε , can be used, such as a conjugate one like in Banerjee [6]-these can also be analyzed to obtain contraction rates, albeit with additional considerations. The generalization error for prediction in such models is strongly related to the weighted L 2 loss given by

which is arguably the natural way of measuring discrepancy between f and f 0 , given the fact that the covariates x i are sampled from p 0 . The posterior contraction rate is then defined as

where E f ∼ Π( ·| x , y ) ( · ) denotes expectation under the posterior distribution while E x , y ( · ) denotes expectation under the true data generating process. 1 In the case of covariates distributed on [0 , 1] d , posterior contraction rates have been derived under Matérn Gaussian process priors [47] in van der Vaart and van Zanten [56], who showed the following result.

Result 1 (Theorem 2 of van der Vaart and van Zanten [56]) . In the Bayesian regression model, let f be a mean-zero Matérn Gaussian process prior on R d with amplitude σ 2 f , length scale κ , and smoothness ν > d/ 2 . Assume that the true data generating process is given by (3) , where p 0 has a Lebesgue density on X = [0 , 1] d which is bounded from below and above by 0 < c p 0 < C p 0 < ∞ , respectively. Let f 0 ∈ H β ∩CH β with β > d/ 2 , where H β and CH β the Sobolev and Hölder spaces, respectively. Then there exists a constant C > 0 , which does not depend on n but does depend on d , σ 2 f , ν , κ , β , p 0 , σ 2 ε , ∥ f 0 ∥ H β ( M ) , and ∥ f 0 ∥ CH β ( M ) , such that

and, moreover, the posterior mean satisfies

Note that m Π( ·| x , y ) is the Bayes estimator [52] of f associated to the weighted L 2 loss and that the second inequality above is a direct consequence of the first. Therefore the posterior contraction rate implies the same convergence rate for m Π( ·| x , y ) . The best rate is attained when β = ν : that is, when true smoothness and prior smoothness match-which is known to be minimax optimal in the problem of estimating f 0 : see Tsybakov [52]. In this paper, we extend this result to the manifold setting.

2.2 Related Work and Current State of Affairs

The formalization of posterior contraction rates of Bayesian procedures dates back to the work of Schwartz [46] and Le Cam [29], but has been extensively developed since the seminal paper of Ghosal et al. [19] for various sampling and prior models, see for instance [20, 42] for reviews. This includes, in particular, work on Gaussian process priors [54, 56, 57, 43, 49]. Most of the results in the literature, however, assume Euclidean data: as a consequence, contraction properties of Bayesian models under manifold assumptions are still poorly understood, with exception of some recent developments in both density estimation [7, 8, 60] and regression [63, 60].

The results closest to ours are those of Yang and Dunson [63] and Castillo et al. [12]. In the former, the authors use an extrinsic length-scale-mixture of squared exponential Gaussian processes to achieve optimal contraction rates with respect to the weighted L 2 norm, using a completely different proof technique compared to us, and their results are restricted to f 0 having Hölder smoothness of order less than or equal to two. On the other hand Castillo et al. [12] consider, as an intrinsic process on the manifold, a hierarchical Gaussian process based on its heat kernel and provide posterior contraction rates. For the Matérn class, Li et al. [30] presents results which characterize the asymptotic behavior of kernel hyperparameters: our work complements these results by studying contraction of the Gaussian process itself toward the unknown ground-truth function. One can also study analogous discrete problems: Dunson et al. [17] and Sanz-Alonso and Yang [45] present posterior contraction rates for a specific graph Gaussian process model in a semi-supervised setting. In the next section, we present our results on Matérn processes, defined either by restriction of an ambient process or by an intrinsic construction, and discuss their implications.

3 Posterior Contraction Rates on Compact Riemannian Manifolds

We now study posterior contraction rates for Matérn Gaussian processes on manifolds, which are arguably the most-widely-used Gaussian process priors in both the Euclidean and Riemannian settings. We begin by more precisely describing our geometric setting before stating our key results and discussing their implications. From now on, we write X = M , to emphasize that the covariate space is a manifold.

Assumption 2. Assume that M⊂ R D is a smooth, compact submanifold (without boundary) of dimension d < D equipped with the standard Riemannian volume measure µ .

We denote |M| = ∫ M dµ ( x ) for volume of M . With this geometric setting defined, we will need to describe regularity assumptions in terms of functional spaces on the manifold M . We work with Hölder spaces CH γ ( M ) , defined using charts via the usual Euclidean Hölder spaces, the Sobolev spaces H s ( M ) , and Besov spaces B s ∞ , ∞ ( M ) which are one of the ways of generalizing

(a) Intrinsic

(b) Extrinsic

(c) Intrinsic

(d) Extrinsic

the Euclidean Hölder spaces of smooth functions to manifolds. We follow Coulhon et al. [14] and Castillo et al. [12], and define these spaces using the Laplace-Beltrami operator on M in Appendix A.

Recall that the data-generating process is given by (3), with f 0 as the true regression function and p 0 as the distribution of the covariates.

Assumption 3. Assume that p 0 is absolutely continuous with respect to µ , and that its density, denoted by p 0 , satisfies c ≤ p 0 ≤ C for 0 < c, C < ∞ . Assume the regression function f 0 satisfies f 0 ∈ H β ( M ) ∩ B β ∞ , ∞ ( M ) for some β > d/ 2 , and that σ 2 ε > 0 is fixed and known.

This setting can be extended to handle unknown variance σ ε by putting a prior on σ ε , following the strategy of Salomond [44] and Naulet and Barat [36]. Since we are focused primarily on the impact of the manifold, we do not pursue this here. With the setting fully defined, we proceed to develop posterior contraction results for different types of Matérn Gaussian process priors: intrinsic, intrinsic truncated and extrinsic.

3.1 Intrinsic Matérn Gaussian Processes

We now introduce the first geometric Gaussian process prior under study-the Riemannian Matérn kernel of Whittle [62], Lindgren et al. [33], and Borovitskiy et al. [11]. This process was originally defined using stochastic partial differential equations: here, we present it by its Karhunen-Loève expansion, to facilitate comparisons with its truncated analogs presented in Section 3.2.

Definition 4 (Intrinsic Matérn prior) . Let ν > 0 , and let ( λ j , f j ) j ≥ 0 be the eigenvalues and orthonormal eigenfunctions of the Laplace-Beltrami operator on M , in increasing order. Define the intrinsic Riemannian Matérn Gaussian process through its Karhunen-Loève expansion to be

where ν, κ, σ 2 f are positive parameters and C ν,κ is the normalization constant, chosen such that 1 |M| ∫ M Var( f ( x )) d µ ( x ) = σ 2 f , where Var denotes the variance.

The covariance kernels of these processes are visualized in Figure 2. With this prior, and the setting defined in Section 3, we are ready to present our first result: this model attains the desired optimal posterior contraction rate as soon as the regularity of the ground-truth function matches the regularity of the Gaussian process, as described by the parameter ν .

Theorem 5. Let f be a Riemannian Matérn Gaussian process prior of Definition 4 with smoothness parameter ν > d/ 2 and let f 0 satisfy Assumption 3. Then there is a C > 0 such that

All proofs are given in Appendix B. Our proof follows the general approach of van der Vaart and van Zanten [56], by first proving a contraction rate with respect to the distance n -1 / 2 ∥ f ( x ) -f 0 ( x ) ∥ R n at input locations x , and then extending the result to the true L 2 -distance by applying a suitable concentration inequality. The first part is obtained by studying the concentration function , which is known to be the key quantity to control in order to derive contraction rates of Gaussian process priors-see Ghosal and van der Vaart [20] and van der Vaart and van Zanten [57] for an overview.

Given our regularity assumptions on f 0 , the most difficult part lies in controlling the small-ball probabilities Π [ ∥ f ∥ C ( M ) < ε ] : we handle this by using results relating this quantity with the entropy of an RKHS unit ball with respect to the uniform norm. Since our process' RKHS is related to the Sobolev space H ν + d/ 2 ( M ) which admits a description in terms of charts, we apply results on the entropy of Sobolev balls in the Euclidean space to conclude the first part. Finally, to extend the rate to the true L 2 ( p 0 ) norm, following van der Vaart and van Zanten [56], we prove a Hölder-type property for manifold Matérn processes, and apply Bernstein's inequality. Together, this gives the claim.

This result is good news for the intrinsic Matérn model: it tells us that asymptotically it incorporates the data as efficiently as possible at least in terms of posterior contraction rates, given that its regularity matches the regularity of f 0 . An inspection of the proof shows that the constant C > 0 can be seen to depend on d , σ 2 f , ν , κ , β , p 0 , σ 2 ε , ∥ f 0 ∥ H β ( M ) , ∥ f 0 ∥ B β ∞∞ ( M ) , and ∥ f 0 ∥ CH β ( M ) . Theorem 5 extends to the case where the norm is raised to any power q > 1 rather than the second power, with the right-hand side raised to the same power: see Appendix B for details. We now consider variants of this prior that can be implemented in practice.

3.2 Truncated Matérn Gaussian Processes

The Riemannian Matérn prior's covariance kernel cannot in general be computed exactly, since Definition 4 involves an infinite sum. Arguably the simplest way to implement these processes numerically is to truncate the respective infinite series in the Karhunen-Loève expansion by taking the first J terms, which is also optimal in an L 2 ( M ) -sense.

Note that the truncated prior is a randomly-weighted finite sum of Laplace-Beltrami eigenfunctions, which have different smoothness properties compared to the original prior: the truncated prior takes its values in C ∞ ( M ) since the eigenfunctions of M are smooth-see for instance De Vito et al. [16]. Nevertheless, if the truncation level is allowed to grow as the sample size increases, then the regularity of the process degenerates and one gets a function with essentially-finite regularity in the limit.

Truncated random basis expansions have been studied extensively in the Bayesian literature in the Euclidean setting-see for instance Arbel et al. [2] and Yoo et al. [64] or Ghosal and van der Vaart [20], Chapter 11 for examples with priors based on wavelet expansions. It is known that truncating the expansion at a high enough level usually allows one to retain optimality. Instead of truncating deterministically, it is also possible to put a prior on the truncation level and resort to MCMC computations which would then select the optimal number of basis functions adaptively, at the expense of a more computationally intensive method-this is done, for instance, in van der Meulen et al. [53] in the context of drift estimation for diffusion processes. Random truncation has been proven to lead in many contexts to adaptive posterior contraction rates, meaning that although the prior does not depend on the smoothness β of f 0 , the posterior contraction rate-up to possible ln n terms-is of order n -β/ (2 β + d ) : see for instance Arbel et al. [2] and Rousseau and Szabo [43].

By analogy of the Euclidean case with its random Fourier feature approximations [40], we can call the truncated version of Definition 4 the manifold Fourier feature model, for which we now present our result.

Theorem 6. Let f be a Riemannian Matérn Gaussian process prior on M with smoothness parameter ν > d/ 2 , modified to truncate the infinite sum to at least J n ≥ cn d (min(1 ,ν/β )) 2 ν + d terms, and let f 0 satisfy Assumption 3. Then there is a C > 0 such that

The proof is essentially-the-same as the non-truncated Matérn, but involves tracking dependence of the inequalities on the truncation level J n , which implicitly defines a sequence of priors rather than a single fixed prior.

This result is excellent news for the intrinsic models: it means that they inherit the optimality properties of the limiting one, even in the absence of the infinite sum-in spite of the fact that the corresponding finite-truncation prior places its probability on C ∞ ( M ) . Again, the constant C > 0 can be seen to depend on d , σ 2 f , ν , κ , β , p 0 , σ 2 ε , ∥ f 0 ∥ H β ( M ) , ∥ f 0 ∥ B β ∞∞ ( M ) , and ∥ f 0 ∥ CH β ( M ) . This concludes our results for the intrinsic Riemannian Matérn priors. We now study what happens if, instead of working with a geometrically-formulated model, we simply embed everything into R d and formulate our models there.

3.3 Extrinsic Matérn Gaussian Processes

The results of the preceding sections provide good reason to be excited about the intrinsic Riemannian Matérn prior: the rates it obtains match the usual minimax rates seen for the Euclidean Matérn prior and Euclidean data, provided that we match the smoothness ν with the regularity of f 0 . Another possibility is to consider an extrinsic Gaussian process, that is, a Gaussian process defined over an ambient space. This has been considered by Yang and Dunson [63] for instance for the squareexponential process, in an adaptive setting where one does not assume that the regularity β of f 0 is explicitly known, but where β ≤ 2 . In this section we prove a non-adaptive analog of this result for the Matérn process.

Definition 7 (Extrinsic Matérn prior) . Assume that the manifold M is isometrically embedded in the Euclidean space R D , such that we can regard M as a subset of R D . Consider the Gaussian process with zero mean and kernel given by restricting onto M the standard Euclidean Matérn kernel

where σ f , κ, ν > 0 and K ν is the modified Bessel function of the second kind [22].

Since the extrinsic Matérn process is defined in a completely agnostic way with respect to the manifold geometry, we would expect it to be less performant when M is known. However, it turns out that the extrinsic Matérn process converges at the same rate as the intrinsic one, as given in the following claim.

Theorem 8. Let f be a mean-zero extrinsic Matérn Gaussian process prior with smoothness parameter ν > d/ 2 on M , and let f 0 satisfy Assumption 3. Then for some C > 0 we have

Theorem 8 is a surprising result because the optimal rates in this setting only require the knowledge of the regularity β , but not the knowledge of the manifold or the intrinsic dimension. More precisely, the prior is not designed to be an adaptive prior, since it is a fixed Gaussian process, but it surprisingly adapts to the dimension of the manifold, and thus to the manifold.

The proof is also based on control of concentration functions. The main difference is that, although the ambient process has a well known RKHS-the Sobolev space H s + D/ 2 ( R D ) -the restricted process has a non-explicit RKHS, which necessitates further analysis. We tackle this issue by using results from Große and Schneider [24] relating manifold and ambient Sobolev spaces by linear bounded trace and extension operators, and from Yang and Dunson [63] describing a general link between the RKHS of an ambient process and its restriction. This allows us to show that the restricted process has an RKHS that is actually norm-equivalent to the Sobolev space H ν + d/ 2 ( M ) , which allows us to conclude the result in the same manner as in the intrinsic case.

As consequence, our argument applies mutatis mutandis in any setting where suitable trace and extension theorems apply, with the Riemannian Matérn case corresponding to the usual Sobolev results. In particular, our arguments therefore apply directly to other processes possessing similar RKHSs, such as for instance various kernels defined on the sphere-see e.g. Wendland [61], Chapter 17 and Hubbert et al. [26]. The constant C > 0 can be seen to depend on d , D , σ 2 f , ν , κ , β , p 0 , σ 2 ε , ∥ f 0 ∥ H β ( M ) , ∥ f 0 ∥ B β ∞∞ ( M ) , ∥ f 0 ∥ CH β ( M ) -notice that here C depends implicitly on D because of the presence of trace and extension operator continuity constants. We now proceed to understand the significance of the overall results.

3.4 Summary of Results

As a consequence of our previous results, fixing a single common data generating distribution determined by p 0 , f 0 , under suitable conditions the intrinsic Matérn process, its truncated version, and the extrinsic Matérn process all possess the same posterior contraction rate with respect to the L 2 ( p 0 ) -norm, which depends on d , ν , and β , and is optimal if the regularities of f 0 and the prior match. These results imply the following immediate corollary, which follows by convexity of ∥·∥ 2 L 2 ( p 0 ) using Jensen's inequality.

Corollary 9. Under the assumptions of Theorems 5, 6 and 8, it follows that, for some C > 0

where m Π( ·| x , y ) is the posterior mean given a particular value of ( x i , y i ) n i =1 .

When ν = β , the optimality of the rates we present in the manifold setting can be easily inferred by lower bounding the L 2 -risk of the posterior mean by the L 2 -risk over a small subset of M and using charts, which translates the problem into the Euclidean framework for which the rate is known to be optimal-see for instance Tsybakov [52].

To contextualize this, observe that even in cases where the geometry of the manifold is non-flat, the asymptotic rates are unaffected by the choice of the prior's length scale κ -in either the intrinsic, or the extrinsic case-but only by the smoothness parameter ν . Indeed, the RKHS of the process is only determined-up to norm equivalence-by ν , which plays an important role in the proofs. This, and the fact that extrinsic processes attain the same rates, implies that the study of asymptotic posterior contraction rates cannot detect geometry in our setting, as was already hinted by Yang and Dunson [63]. Hence, in the geometric setting, optimal posterior contraction rates should be thought of more as a basic property that any reasonable model should satisfy. Differences in performance will be down to constant factors alone-but as we will see, these can be significant. To understand these differences, we turn to empirical analysis.

4 Experiments

From Theorems 5, 6 and 8, we know that intrinsic and extrinsic Gaussian processes exhibit the same posterior contraction rates in the asymptotic regime. Here, we study how these rates manifest themselves in practice, by examining how worst-case errors akin to those of Corollary 9 behave numerically. Specifically, we consider the pointwise worst-case error

where m ( τ ) Π( ·| x , y ) is the posterior mean corresponding to the zero-mean Matérn Gaussian process prior with smoothness ν , length scale κ , amplitude σ 2 f , which is intrinsic if τ = i or extrinsic if τ = e . We use a Gaussian likelihood with noise variance σ 2 ε and observations y i = f 0 ( x i ) + ε i , and examine this quantity as a function of the evaluation location t ∈ M . By allowing us to assess how error varies in different regions of the manifold, this provides us with a fine-grained picture of how posterior contraction behaves.

One can show that v ( τ ) may be computed without numerically solving an infinite-dimensional optimization problem. Specifically, (14) can be calculated, in the respective intrinsic and extrinsic cases, using

where, for the extrinsic case, α t = K (i) t X ( K (e) XX + σ 2 ε I ) -1 , and X ′ is a set of points sampled uniformly from the manifold M , the size of which determines approximation quality. The intrinsic expression is simply the posterior variance k (i) Π( ·| x , y ) ( t, t ) , and its connection with worst-case error is a well-known folklore result mentioned somewhat implicitly in, for instance, Mutny and Krause [35]. The extrinsic expression is very-closely-related, and arises by numerically approximating a certain RKHS norm. A derivation of both is given in Appendix F. To assess the approximation error of this formula, we also consider an analog of (16) but instead defined for the intrinsic model, and compare it to (15): in all cases, the difference between the exact and approximate expression was found to be smaller than differences between models. By computing these expressions, we therefore obtain, up to numerics, the pointwise worst-case expected error in our regression model.

For M we consider three settings: a dumbbell-shaped manifold, a sphere, and the dragon manifold from the Stanford 3D scanning repository. In all cases, we perform computations by approximating

10

-

4

10

-

2

10

0

Mean

Dumbbell

10

-

2

10

-

1

10

0

Sphere

10

-

4

10

-

2

10

0

Dragon

0

25

50

10

-

4

10

-

2

10

0

0

250

500

10

-

2

10

-

1

10

0

Data size

Intrinsic

Intrinsic (approx. formula)

Extrinsic

0

500

1000

10

-

4

10

-

2

10

0

the manifold using a mesh, and implementing the truncated Karhunen-Loève expansion with J = 500 eigenpairs obtained from the mesh. We fix smoothness ν = 5 2 , amplitude σ 2 f = 1 , and noise variance σ 2 ε = 0 . 0005 , for both the intrinsic and extrinsic Matérn Gaussian processes. Since the interpretation of the length scale parameter is manifold-specific, for the intrinsic Gaussian processes we set κ = 200 for the dumbbell, κ = 0 . 25 for the sphere, and κ = 0 . 05 for the dragon manifold. In all cases, this yielded functions that were neither close to being globally-constant, nor resembled noise. Each experiment was repeated 10 times to assess variability. Complete experimental details are given in Appendix G. 2

The length scales κ are defined differently for intrinsic and extrinsic Matérn kernels: in particular, using the same length scale in both models can result in kernels behaving very differently. To alleviate this, for the extrinsic process, we set the length scale by maximizing the extrinsic process' marginal likelihood using the full dataset generated by the intrinsic process, except in the dumbbell's case where the full dataset is relatively small, and therefore a larger set of 500 points was used instead. This allows us to numerically match intrinsic and extrinsic length scales to ensure a reasonably-fair comparison.

Figure 3 shows the mean, and spatial standard deviation of v τ ( t ) , where by spatial standard deviation we mean the sample standard deviation computed with respect to locations in space, rather than with respect to different randomly sampled datasets. From this, we see that on the dumbbell and dragon manifold-whose geometry differs significantly from the respective ambient Euclidean spacesintrinsic models obtain better mean performance. The standard deviation plot reveals that intrinsic models have errors that are less-variable across space. This means that extrinsic models exhibit higher errors in some regions rather than others-such as, for instance, regions where embedded Euclidean and Riemannian distances differ-whereas in intrinsic models the error decays in a more spatially-uniform manner.

In contrast, on the sphere, both models perform similarly. Moreover, both the mean and spatial standard deviation decrease at approximately the same rates, indicating that the extrinsic model's predictions are correct about-as-often as the intrinsic model's, as a function of space. This confirms the view that, since the sphere does not possess any bottleneck-like areas where embedded Euclidean distances are extremely different from their Riemannian analogs, it is significantly less affected by differences coming from embeddings.

In total, our experiments confirm that there are manifolds on which geometric models can perform significantly better than non-geometric models. This phenomenon was also noticed in Dunson et al. [17], where a prior based on the eigendecomposition of a random geometric graph, which can be thought as an approximation of our intrinsic Matérn processes, is compared to a standard extrinsic Gaussian process. In our experiments, we see this through expected errors, mirroring prior results on Bayesian optimization performance. From our theoretical results, such differences cannot be captured through posterior contraction rates, and therefore would require sharper technical tools, such as non-asymptotic analysis, to quantify theoretically.

5 Conclusion

In this work, we studied the asymptotic behavior of Gaussian process regression with different classes of Matérn processes on Riemannian manifolds. By using various results on Sobolev spaces on manifolds we derived posterior contraction rates for intrinsic Matérn process defined via their Karhunen-Loeve decomposition in the Laplace-Beltrami eigenbasis, including processes arising from truncation of the respective sum which can be implemented in practice. Next, using trace and extension theorems which relate manifold and Euclidean Sobolev spaces, we derived similar contraction rates for the restriction of an ambient Matérn process in the case where the manifold is embedded in Euclidean space. These theoretical asymptotic results were supplemented by experiments on several examples, showing significant differences in performance between intrinsic and extrincic methods in the small sample size regime when the manifold's geometric structure differs from the ambient Euclidean space. Our work therefore shows that capturing such differences cannot be done through asymptotic contraction rates, motivating and paving the way for further work on non-asymptotic error analysis to capture empirically-observed differences between extrinsic and intrinsic models.

Acknowledgments

The authors are grateful to Mojmír Mutný and Prof. Andreas Krause for fruitful discussions concerning this work. PR and JR were supported by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No. 834175). VB was supported by an ETH Zürich Postdoctoral Fellowship. AT was supported by Cornell University, jointly via the Center for Data Science for Enterprise and Society, the College of Engineering, and the Ann S. Bowers College of Computing and Information Science.

References

C. D. Aliprantis and K. C. Border. Infinite Dimensional Analysis: A Hitchhiker's Guide . Springer, 2006. Cited on page 29.

J. Arbel, G. Gayraud, and J. Rousseau. Bayesian Optimal Adaptive Estimation Using a Sieve Prior. Scandinavian Journal of Statistics , 2013. Cited on page 6.

T. Aubin. Some nonlinear problems in Riemannian geometry . Springer, 1998. Cited on page 14.

I. Azangulov, A. Smolensky, A. Terenin, and V. Borovitskiy. Stationary Kernels and Gaussian Processes on Lie Groups and their Homogeneous Spaces I: the compact case. arXiv:2208.14960 , 2022. Cited on page 1.

I. Azangulov, A. Smolensky, A. Terenin, and V. Borovitskiy. Stationary Kernels and Gaussian Processes on Lie Groups and their Homogeneous Spaces II: non-compact symmetric spaces. arXiv:2301.13088 , 2023. Cited on page 1.

S. Banerjee. Modeling massive spatial datasets using a conjugate Bayesian linear modeling framework. Spatial Statistics , 2020. Cited on page 3.

C. Berenfeld, P. Rosa, and J. Rousseau. Estimating a density near an unknown manifold: a Bayesian nonparametric approach. arXiv:2205.15717 , 2022. Cited on page 4.

A. Bhattacharya and D. B. Dunson. Nonparametric Bayesian density estimation on manifolds with applications to planar shapes. Biometrika , 2010. Cited on page 4.

V. Borovitskiy, I. Azangulov, A. Terenin, P. Mostowsky, M. P. Deisenroth, and N. Durrande. Matérn Gaussian processes on graphs. In International Conference on Artificial Intelligence and Statistics , 2021. Cited on page 1.

V. Borovitskiy, M. R. Karimi, V. R. Somnath, and A. Krause. Isotropic Gaussian Processes on Finite Spaces of Graphs. In Artificial Intelligence and Statistics , 2023. Cited on page 1.

V. Borovitskiy, A. Terenin, P. Mostowsky, and M. P. Deisenroth. Matérn Gaussian processes on Riemannian manifolds. Advances in Neural Information Processing Systems , 2020. Cited on pages 1, 2, 5, 15, 23, 25.

I. Castillo, G. Kerkyacharian, and D. Picard. Thomas Bayes' walk on manifolds. Probability Theory and Related Fields , 2013. Cited on pages 4, 5, 15, 32, 33.

I. Chavel. Eigenvalues in Riemannian Geometry . Elsevier, 1984. Cited on pages 2, 14.

T. Coulhon, G. Kerkyacharian, and P. Petrushev. Heat Kernel Generated frames in the setting of Dirichlet Spaces. Journal of Fourier Analysis and Applications , 2012. Cited on pages 5, 15, 32.

S. Coveney, C. Corrado, C. H. Roney, D. O'Hare, S. E. Williams, M. D. O'Neill, S. A. Niederer, R. H. Clayton, J. E. Oakley, and R. D. Wilkinson. Gaussian Process Manifold Interpolation for Probabilistic Atrial Activation Maps and Uncertain Conduction Velocity. Philosophical Transactions of the Royal Society A , 2020. Cited on page 1.

E. De Vito, N. Mücke, and L. Rosasco. Reproducing kernel Hilbert spaces on manifolds: Sobolev and Diffusion spaces. Analysis and Applications , 2021. Cited on pages 6, 14, 15, 17, 23, 25, 32.

D. B. Dunson, H.-T. Wu, and N. Wu. Graph Based Gaussian Processes on Restricted Domains. Journal of the Royal Statistical Society Series B: Statistical Methodology , 2021. Cited on pages 4, 10.

C. Fefferman, S. Mitter, and H. Narayanan. Testing the Manifold Hypothesis. Journal of the American Mathematical Society , 2016. Cited on page 2.

S. Ghosal, J. K. Ghosh, and A. van der Vaart. Convergence Rates of Posterior Distributions. The Annals of Statistics , 2000. Cited on page 4.

S. Ghosal and A. van der Vaart. Fundamentals of Nonparametric Bayesian Inference . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2017. Cited on pages 3-6, 17, 21, 28, 31.

E. Giné and R. Nickl. Mathematical Foundations of Infinite-Dimensional Statistical Models . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2015. Cited on pages 14, 15, 32.

I. S. Gradshteyn and I. M. Ryzhik. Table of Integrals, Series, and Products . Academic Press, 7th edition, 2014. Cited on page 7.

A. Grigoryan. Heat Kernel and Analysis on Manifolds . American Mathematical Society, 2009. Cited on page 14.

N. Große and C. Schneider. Sobolev spaces on Riemannian manifolds with bounded geometry: General coordinates and traces. Mathematische Nachrichten , 2013. Cited on pages 7, 14, 24, 32.

E. Hebey. Sobolev spaces on Riemannian manifolds , volume 1635. Springer, 1996. Cited on page 14.

S. Hubbert, E. Porcu, C. J. Oates, and M. Girolami. Sobolev Spaces, Kernels and Discrepancies over Hyperspheres. Transactions on Machine Learning Research , 2023. Cited on page 7.

N. Jaquier, V. Borovitskiy, A. Smolensky, A. Terenin, T. Asfour, and L. Rozo. Geometry-aware bayesian optimization in robotics using Riemannian Matérn kernels. In Conference on Robot Learning , 2022. Cited on page 1.

M. Kanagawa, P. Hennig, D. Sejdinovic, and B. K. Sriperumbudur. Gaussian processes and kernel methods: A review on connections and equivalences. arXiv:1807.02582 , 2018. Cited on pages 24, 25.

L. Le Cam. Convergence of Estimates Under Dimensionality Restrictions. The Annals of Statistics , 1973. Cited on page 4.

D. Li, W. Tang, and S. Banerjee. Inference for Gaussian Processes with Matern Covariogram on Compact Riemannian Manifolds. Journal of Machine Learning Research , 2023. Cited on page 4.

W. V. Li and W. Linde. Approximation, Metric Entropy and Small Ball Estimates for Gaussian Measures. The Annals of Probability , 1999. Cited on pages 32, 33.

M. Lifshits. Lectures on Gaussian processes . Springer, 2012. Cited on page 17.

F. Lindgren, H. Rue, and J. Lindström. An explicit link between Gaussian fields and Gaussian Markov random fields: the stochastic partial differential equation approach. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 2011. Cited on page 5.

S. V. Lototsky and B. L. Rozovsky. Stochastic partial differential equations . Springer, 2017. Cited on page 26.

M. Mutny and A. Krause. Experimental design for linear functionals in reproducing kernel hilbert spaces. Advances in Neural Information Processing Systems , 2022. Cited on page 8.

Z. Naulet and É. Barat. Some Aspects of Symmetric Gamma Process Mixtures. Bayesian Analysis , 2018. Cited on page 5.

M. Niu, P. Cheung, L. Lin, Z. Dai, N. D. Lawrence, and D. B. Dunson. Intrinsic Gaussian processes on complex constrained domains. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 2018. Cited on page 1.

T. Pinder and D. Dodd. GPJax: A Gaussian Process Framework in JAX. Journal of Open Source Software , 2022. Cited on page 35.

E. Porcu, M. Bevilacqua, R. Schaback, and C. J. Oates. The Matérn Model: A Journey through Statistics, Numerical Analysis and Machine Learning. arXiv:2303.02759 , 2023. Cited on page 2.

A. Rahimi and B. Recht. Random Features for Large-Scale Kernel Machines. Advances in Neural Information Processing Systems , 2007. Cited on page 6.

C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning . MIT Press, 2006. Cited on page 3.

J. Rousseau. On the frequentist properties of Bayesian nonparametric methods. Annual Review of Statistics and its Application , 2016. Cited on pages 3, 4.

J. Rousseau and B. Szabo. Asymptotic behaviour of the empirical Bayes posteriors associated to maximum marginal likelihood estimator. The Annals of Statistics , 2017. Cited on pages 4, 6.

J.-B. Salomond. Testing Un-Separated Hypotheses by Estimating a Distance. Bayesian Analysis , 2018. Cited on page 5.

D. Sanz-Alonso and R. Yang. Unlabeled Data Help in Graph-Based Semi-Supervised Learning: A Bayesian Nonparametrics Perspective. Journal of Machine Learning Research , 2022. Cited on page 4.

L. Schwartz. On Bayes procedures. Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete , 1965. Cited on page 4.

M. L. Stein. Interpolation of Spatial Data: Some Theory for Kriging . Springer, 1999. Cited on page 3.

R. S. Strichartz. Analysis of the Laplacian on the complete Riemannian manifold. Journal of functional analysis , 52(1):48-79, 1983. Cited on page 14.

W. Tang, L. Zhang, and S. Banerjee. On identifiability and consistency of the nugget in Gaussian spatial process models. Journal of the Royal Statistical Society Series B: Statistical Methodology , 2021. Cited on page 4.

H. Triebel. Theory of Function Spaces I . Birkhäuser, 1983. Cited on pages 15, 27.

H. Triebel. Theory of Function Spaces II . Birkhäuser, 1992. Cited on pages 14, 15.

A. B. Tsybakov. Introduction to Nonparametric Estimation . Springer, 2008. Cited on pages 4, 8.

F. van der Meulen, M. Schauer, and H. van Zanten. Reversible jump MCMC for nonparametric drift estimation for diffusion processes. Computational Statistics and Data Analysis , 2014. Cited on page 6.

A. van der Vaart and H. van Zanten. Adaptive Bayesian estimation using a Gaussian random field with inverse Gamma bandwidth. The Annals of Statistics , 2009. Cited on page 4.

A. van der Vaart and H. van Zanten. Bayesian inference with rescaled Gaussian process priors. Electronic Journal of Statistics , 2007. Cited on page 3.

A. van der Vaart and H. van Zanten. Information Rates of Nonparametric Gaussian Process Methods. Journal of Machine Learning Research , 2011. Cited on pages 2-6, 17-22, 30, 31.

A. van der Vaart and H. van Zanten. Rates of contraction of posterior distributions based on Gaussian process priors. The Annals of Statistics , 2008. Cited on pages 4, 5, 17.

A. van der Vaart and J. Wellner. Weak Convergence and Empirical Processes . Springer Series in Statistics. Springer Verlag, 1996. Cited on page 22.

H. van Zanten and A. van der Vaart. Reproducing kernel Hilbert spaces of Gaussian priors. Institute of Mathematical Statistics Collections , 2008. Cited on pages 17, 30, 32.

X. Wang and G. Lerman. Nonparametric Bayesian regression on manifolds via Brownian motion. arXiv:1507.06710 , 2015. Cited on page 4.

H. Wendland. Scattered Data Approximation . Cambridge University Press, 2004. Cited on pages 7, 25.

P. Whittle. Stochastic Processes in Several Dimensions. Bulletin of the International Statistical Institute , 1963. Cited on page 5.

Y. Yang and D. B. Dunson. Bayesian manifold regression. The Annals of Statistics , 2016. Cited on pages 4, 7, 8, 24.

W. W. Yoo, V. Rivoirard, and J. Rousseau. Adaptive supremum norm posterior contraction: Wavelet spike-and-slab and anisotropic Besov spaces. arXiv:1708.01909 , 2017. Cited on page 6.

A Preliminaries

Here we describe preliminaries necessary for Appendices B to E. This includes some basic properties of the Laplace-Beltrami operator on compact manifolds, partitions of unity subordinate to atlases, function spaces such as Hölder, Sobolev and Besov spaces, general Gaussian random elements on Banach spaces, and a certain technical lemma. Hereinafter, the expression a ≲ b means a ≤ Cb for some constant C > 0 whose value is irrelevant to our claims.

A.1 Laplace-Beltrami Operator and Subordinate Partitions of Unity

Let M denote a compact connected Riemannian manifold without boundary of dimension d ∈ Z > 0 . The Laplace-Beltrami operator ∆ on M is self-adjoint and positive semi-definite [48, Theorem 2.4]. Let ( L 2 ( M ) , ⟨· , ·⟩ L 2 ( M ) ) denote the Hilbert space of square integrable (equivalence classes of) functions on M with respect to the standard Riemannian volume measure.

By standard theory [13, 23], there exists an orthonormal basis { f j } ∞ j =0 of L 2 ( M ) consisting of the eigenfunctions of ∆ , namely ∆ f j = -λ j f j , with λ j ≥ 0 . We assume that the pairs ( λ j , f j ) are sorted such that 0 = λ 0 ≤ λ j ≤ λ j +1 . The growth of λ j can be characterized as follows.

Result 10 (Weyl's Law) . There exists a constant C > 0 such that for all j large enough we have

Following De Vito et al. [16] and Große and Schneider [24] we fix a family T = ( U l , ϕ l , χ l ) L l =1 , where L ∈ Z > 0 , the local coordinates ϕ l : U l ⊂ M → V l = ϕ l ( U l ) ⊂ R d are smooth diffeomorphisms, and the functions χ l form a partition of unity subordinate to {U l } L l =1 , that is χ l ∈ C ∞ ( M ) , supp( χ l ) ⊂ U l , 0 ≤ χ l ≤ 1 and ∑ l χ l = 1 -here, we can choose L finite by compactness of M . For convenience and without loss of generality, we assume that V l = (0 , 1) d . With this, we can start defining function spaces on M .

A.2 Hölder Spaces CH γ and the spaces of continuous functions C and smooth functions C ∞

Consider an arbitrary domain X ⊆ R d or X ⊆ M . We denote the class of infinitely differentiable functions on X by C ∞ ( X ) . Let C k ( X ) denote the Banach space of k ∈ Z > 0 times continuously differentiable functions on X with finite norm

where ∇ k is the k th covariant derivative, as in Hebey [25], Section 2.1 and Aubin [3], Definition 2.2. We also write C ( X ) = C 0 ( X ) for the space of continuous functions on X .

The Euclidean Hölder spaces CH γ ( R d ) , where γ = k + α with k ∈ Z ≥ 0 and 0 < α ≤ 1 , are defined by 3

̸

where

More information on these definitions may be found, for instance, in Giné and Nickl [21] and Triebel [51]. We now turn to the manifold versions of the Hölder spaces.

Definition 11 (Hölder spaces) . For all γ > 0 we define the Hölder space CH γ ( M ) on the manifold M to be the space of all f : M→ R satisfying

Since the charts ϕ l are smooth, Definition 11 can be easily seen to be independent of the chosen atlas, with equivalence of norms.

A.3 Sobolev and Besov Spaces

We now introduce the manifold versions of the Sobolev and Besov spaces, whose definitions in the standard Euclidean case may be found, for instance, in Triebel [51]. For Sobolev spaces we use the Bessel-potential-based definition, following De Vito et al. [16].

Definition 12 (Sobolev spaces) . For any s > 0 we define the Sobolev space H s ( M ) on the manifold M as the Hilbert space of functions f ∈ L 2 ( M ) such that ∥ f ∥ 2 H s ( M ) = ⟨ f, f ⟩ H s ( M ) < ∞ where

Remark 13. It is easy to see that substituting (1 + λ j ) s in (22) with β ( α + λ j ) s or with ( α + βλ s j ) for any α, β > 0 results in the same set of functions and an equivalent norm. The former follows from Borovitskiy et al. [11], eq. (109). The latter follows from the Binomial Theorem.

For Besov spaces we follow Coulhon et al. [14] and Castillo et al. [12] and define them in terms of approximations by low-frequency functions. We fix a function Φ ∈ C ∞ ( R ≥ 0 , R ≥ 0 ) such that K = supp(Φ) ⊂ [0 , 2] and Φ( x ) = 1 for x ∈ [0 , 1] . We also define the functions Φ j ( x ) = Φ ( 2 -j x ) .

Coulhon et al. [14], Corollary 3.6 shows that the operators Φ j ( √ ∆) defined by

are bounded in the space L p ( M ) for all 1 ≤ p ≤ ∞ . 4 Moreover, the same result also shows that we can express any f ∈ L p ( M ) as f = lim j →∞ Φ j ( √ ∆) f in L p ( M ) . Φ j ( √ ∆) f can intuitively be considered as a version of f filtered by a low-pass filter. The next definition introduces the Besov spaces B s p,q ( M ) , which are formulated in terms of quality-of-approximation by low-frequency functions.

Definition 14 (Besov spaces) . For any s > 0 and 1 ≤ p, q ≤ ∞ we define the Besov space B s p,q ( M ) on the manifold M as the space of functions f ∈ L p ( M ) such that ∥ f ∥ B s p,q ( M ) < ∞ where

The classical Besov spaces B s 2 , 2 coincide with the Sobolev spaces H s on R d , in the sense that they define the same set of functions and equivalent norms-see for instance Giné and Nickl [21] section 4.3.6-and even on manifolds if one follows the construction of Triebel [51], pages 7.3-7.4 for Besov spaces. Since our definition of the Besov spaces is somewhat non-standard, we present a proof.

Proposition 15. For all s > 0 , H s ( M ) = B s 2 , 2 ( M ) as sets and there exist two constants C 1 , C 2 > 0 such that for all f ∈ H s ( M ) = B s 2 , 2 ( M ) we have

Proof. It is enough to prove (25), the rest will follow automatically. The main technical tools used in the proof are Result 10 and summation by parts. First, we prove the upper bound. Denote the support set of Φ by K = supp(Φ) . Notice that

The last inequality results from the fact that [0 , 1] ⊂ K . By Weyl's Law (Result 10), there exists a constant C > 0 such that λ l ≤ Cl 2 /d . Without loss of generality, we can assume that C = 2 2 r , r ∈ Z > 0 . Hence √ λ l > 2 j implies l > 2 d ( j -r ) , thus we have

Now let R j = ∑ l> 2 dj ∣ ∣ ⟨ f l , f ⟩ L 2 ( ) ∣ ∣ 2 and S J = ∑ J j =1 2 2 js ≤ 2 2 s 2 2 s 1 2 2 Js , S 0 = 0 . Write

where we have used Result 10 to get existence of a c such that l 2 /d ≤ cλ l . This proves the upper bound. The proof for the lower bound is similar.

Proposition 15 provides a characterization of the Sobolev spaces H s ( M ) . There is yet another characterization of these spaces that will be useful later, in terms of charts. We present this characterization as part of the following result.

Theorem 16. On the Sobolev space H s ( M ) , the following norms are equivalent:

Proof. The equivalence between ∥·∥ H s ( M ) and ∥ f ∥ B s 2 , 2 ( M ) is given by Proposition 15. The proof of equivalence between ∥·∥ H s ( M ) and ∥ f ∥ H s T ( M ) can be found in De Vito et al. [16].

A.4 Gaussian Random Elements

Here we recall the definition of a Gaussian process as a Banach-space-valued random variable, following for instance van Zanten and van der Vaart [59].

Definition 17 (Gaussian random element) . Let ( B , ∥·∥ B ) be a Banach space, and f be a Borel random variable with values in B almost surely. We say that f is a Gaussian random element if b ∗ ( f ) is a univariate Gaussian random variable for every bounded linear functional b ∗ : B → R .

Random variables of this kind are also sometimes called Gaussian in the sense of duality . One should think of a Gaussian random element as a generalization of a Gaussian process, but which is better-behaved from a function-analytic point of view and in particular does not require the process to be an actual function-as opposed to, for instance, a measure or a distribution. Many connections between the usual Gaussian processes and Gaussian random elements exist, see Lifshits [32], Ghosal and van der Vaart [20], Appendix I, van der Vaart and van Zanten [57] for details. The following observation about Gaussian random elements will be useful later.

Lemma 18. A Gaussian process f on the manifold M with almost surely continuous sample paths is a Gaussian random element in the Banach space ( C ( M ) , ∥·∥ C ( M ) ) of continuous functions on M .

Proof. Since C ( M ) is separable, this follows from Lemma I.6 in Ghosal and van der Vaart [20].

A.5 A Technical Lemma

In order to apply Bernstein's inequality when going from the error at input locations to the L 2 ( p 0 ) error, we will use the following technical extrapolation lemma.

Lemma 19. For any function g : M→ R , a number γ ∈ R > 0 \ Z > 0 and a density p 0 : M→ R > 0 with 1 ≲ p 0 , we have

Proof. We use Lemma 15 from van der Vaart and van Zanten [56] and push it through charts. More precisely we have, using B γ ∞ , ∞ ( [0 , 1] D ) = CH γ ( [0 , 1] D ) for γ / ∈ Z > 0 , that

By definition of the the manifold Hölder spaces, this gives

Finally, since χ l are bounded, the charts are smooth, and p 0 is lower bounded, we have

which gives the result.

This concludes the necessary preliminaries. We now turn to the proofs of our main results.

B Proofs

Throughout this section we use the notation from Appendix A and results from Appendices C to E. We start by defining our notation for Gaussian likelihood and probability distribution of the sample.

Definition 20. For every x ∈ M n and f : M → R we define p ( f ) x , y to be the joint distribution corresponding to the marginal p x = p 0 and conditional p ( f ) y | x = N( f ( x ) , σ 2 ε I ) , where f ( x ) is the vector with entries f ( x i ) . Expectations with respect to p ( f ) x , y , to p x and to p ( f ) y | x we denote by E ( f ) x , y , by E x and by E ( f ) y | x , respectively. Sometimes, when f is clear from the context, we omit the subscript ( f ) .

Using van der Vaart and van Zanten [56], Theorem 1, which is valid for any compact metric space and thus also for M , we can deduce a posterior contraction rate at data input locactions, with respect to the empirical L 2 -norm 5

by studying the concentration functions with respect to the uniform norm. This is the purpose of the following lemma, whose proof mainly follows [56], but crucially relies on two new components: (i) the small ball asymptotics for manifolds we study in Appendix E and (ii) the characterization of the RKHS corresponding to both intrinsic and extrinsic priors as manifold Sobolev spaces we obtain in Appendix C. We recall that the prior Π n may depend on n if we consider a truncated intrinsic Matérn process.

Theorem 21. Let Π n denote the prior in either Theorem 5, Theorem 6 or Theorem 8 with smoothness parameter ν > d/ 2 . Let H n denote the corresponding RKHS. Define the concentration function for f 0 ∈ C ( M ) and ε > 0 by

Then if f 0 ∈ H β ( M ) ∩ B β ∞ , ∞ ( M ) , β > 0 we have φ f 0 ( ε n ) ≤ nε 2 n for ε n a multiple of n -min( ν,β ) 2 ν + d .

Proof. The first term on the right-hand side of Equation (49) is bounded by Cε -d/ν by Lemma 33. To bound the second term, we assume, without loss of generality, 6 that ν ≥ β . Consider an approximation f = Φ j ( √ ∆) f 0 of f 0 , where cε ≤ 2 -βj ≤ ε and c > 0 does not depend on j . Since we assume f 0 ∈ B β ∞ , ∞ ( M ) , by definition of B β ∞ , ∞ ( M ) we have

where in the last inequality the B β ∞ , ∞ ( M ) -norm is the constant implied by notation ≲ . We now show that

First notice that by Lemma 24 and Proposition 27, for any prior considered here we have H n ⊆ H ν + d/ 2 ( M ) and ∥·∥ H n ≲ ∥·∥ H ν + d/ 2 ( M ) with a constant that does not depend on n . Hence using

Result 10 (Weyl's Law) and properties of Φ , we have

Our assumption ν ≥ β implies that

Hence, we have ε -d/ν ≤ ε -2 β ( ν -β + d/ 2) which gives us φ f 0 ( ε ) ≲ ε -2 β ( ν -β + d/ 2) . It is then easy to check that ε n = Mn -β 2 ν + d satisfies φ f 0 ( ε n ) ≤ nε 2 n for M > 0 large enough.

From this, we deduce an upper bound on the error in the empirical L 2 norm ∥·∥ n , that is, on the Euclidean distance between the posterior Gaussian process f and the ground truth function f 0 evaluated at data locations x i .

Lemma 22. Let Π n denote the prior in either Theorem 5, Theorem 6 or Theorem 8 with smoothness parameter ν > d/ 2 . Fix f 0 ∈ H β ( M ) ∩ B β ∞ , ∞ ( M ) with β > 0 . Then

for ε n ∝ n -min( ν,β ) 2 ν + d with constant depending on f 0 , ν but not on x .

Proof. By Theorem 21 for ε n a multiple of n -min( β,ν ) 2 ν + d , we have φ f 0 ( ε n ) ≤ nε 2 n . By virtue of this, the proof of Theorem 1 and Proposition 11 of van der Vaart and van Zanten [56] imply the result. Indeed, the proof of Theorem 1 relies solely on the fact that φ f 0 ( ε n / 2) ≤ nε 2 n and an application of van der Vaart and van Zanten [56], Proposition 11. We have φ f 0 ( ε n ) ≤ nε 2 n ≤ n (2 ε n ) 2 and hence the condition is satisfied with ε n replaced by 2 ε n .

We now turn to the proofs of our main results, Theorems 5, 6 and 8, which for convenience we restate below. The idea of these proofs is to extend the result of Lemma 22 from input locations to the whole manifold M using an appropriate concentration inequality. To this end, the proof closely follows the one of Theorem 2 in van der Vaart and van Zanten [56], but relies on Lemma 22 proved above and on the concentration inequality we prove in Appendix D along with some important sample differentiablity properties of the prior processes.

Theorem 5. Let f be a Riemannian Matérn Gaussian process prior of Definition 4 with smoothness parameter ν > d/ 2 and let f 0 satisfy Assumption 3. Then there is a C > 0 such that

Theorem 6. Let f be a Riemannian Matérn Gaussian process prior on M with smoothness parameter ν > d/ 2 , modified to truncate the infinite sum to at least J n ≥ cn d (min(1 ,ν/β )) 2 ν + d terms, and let f 0 satisfy Assumption 3. Then there is a C > 0 such that

Theorem 8. Let f be a mean-zero extrinsic Matérn Gaussian process prior with smoothness parameter ν > d/ 2 on M , and let f 0 satisfy Assumption 3. Then for some C > 0 we have

Proof of Theorems 5, 6 and 8. To ease notation, we denote the expectations under the true data generating process by E x , y = E ( f 0 ) x , y and E y | x = E ( f 0 ) y | x , omitting the superscript ( · ) f 0 . Take ε n ∝ n -min( β,ν ) 2 ν + d satisfying φ f 0 ( ε n / 2) ≤ nε 2 n , noting that such a rate exists by Theorem 21. Then, for each n , there exists an element f n ∈ H n , where H n is the RKHS corresponding to Π n , satisfying

Write

Thus, we can work with f -f n instead of f -f 0 . Define B ( r ) = { ∥ f -f n ∥ L 2 ( p 0 ) > ε n r } . Then

Fix a γ such that d/ 2 < γ < ν, γ / ∈ Z > 0 and s > 0 , τ > 0 and define

Then B ( r ) ⊆ B (I) ( r ) ∪B (II) ( r ) ∪B (III) ( r ) , and thus for an indexed family of events A r to be chosen later, we have

For the first term, by Lemma 22 applied conditionally on x , for which we got a bound on the integrated empirical L 2 -norm uniformly on the design points, we have

Moreover, by Lemma 14 of van der Vaart and van Zanten [56] applied with r in the notation of the reference being equal to √ nε n r s , for each r > 0 , the event

is such that

It should be noted that the √ n factor disappears because of the discrepancy between the empirical L 2 norm ∥·∥ n and the Euclidean norm used in van der Vaart and van Zanten [56]. By Fubini's Theorem and since nε 2 n ≥ n d 2 ν + d ≥ 1 , the second term is bounded by

It remains to bound the last two terms. By Bayes' Rule, we have the equality

therefore for y ∈ A r ( x ) , we have

Hence taking expectations, using Tonelli's Theorem, and E x , y p ( f ) y | x ( y ) p ( f 0 ) y | x ( y ) = 1 gives

Therefore, the third term can be bounded as

Now, using Lemma 31, for a possibly small constant c > 0 independent of n , we have

Moreover, by using the bound on the concentration function in Theorem 21 and Ghosal and van der Vaart [20], Proposition 11.19, we can assume that

Therefore, the third term is bounded by

if τ 2 c > 2 + c -1 . It remains to bound the last term. We have, by the same arguments as above, that

As the squared empirical L 2 -norm is a Monte Carlo approximation of the true L 2 -norm, the probability in the integrand can be controlled via a concentration inequality. As in van der Vaart and van Zanten [56], we use Bernstein's inequality [58, Lemma 2.2.9]. For a collection Y 1 , . . . Y n of random variables such that E Y i = 0 and Y i ∈ [ -M,M ] almost surely for some constant M > 0 , this inequality asserts

where v ≥ Var( Y 1 + . . . + Y n ) . We put Y i = 1 n ( f ( x i ) -f n ( x i )) 2 -1 n ∥ f -f n ∥ 2 L 2 ( p 0 ) where x i are IID with x i ∼ p 0 . It is easy to check that E Y i = 0 and Y i ∈ [ -M,M ] for M = 1 n ∥ f -f n ∥ L ∞ ( M ) . Furthermore, v = 1 n ∥ f -f n ∥ 2 L ∞ ( M ) ∥ f -f n ∥ 2 L 2 ( M ) upper-bounds the variance of the respective sum, since

Using Bernstein's inequality with Y i , M , v as above and with x = 3 4 ∥ f -f n ∥ 2 L 2 ( p 0 ) , we have

Moreover, by Lemma 19, since γ / ∈ Z > 0 , we have

Using the Sobolev Embedding Theorem, given in De Vito et al. [16], Theorem 4, ∥ f -f n ∥ CH γ ( M ) ≲ ∥ f n ∥ H + ∥ f ∥ CH γ ( M ) ≲ τ √ nε n r s whenever ∥ f ∥ CH γ ( M ) ≤ τ √ nε n r s . Therefore, for a constant c > 0 , when ∥ f ∥ CH γ ( M ) ≤ τ √ nε n r s and ε n r ≤ ∥ f -f n ∥ L 2 ( p 0 ) , we have that

Hence, we can bound the last term as

We have n 2 γ 2 γ + d = n ( n -d/ 2 2 γ + d ) 2 . Since ε n ≲ n -min( ν,β ) 2 ν + d and min( ν, β ) > d/ 2 , we have nε 2 n ≲ n 2 γ 2 γ + d for some γ ∈ ( d/ 2 , ν ) . Moreover, for this choice of γ and s small enough we have 2 d 2 γ + d (1 -s ) ≥ 2 s , which proves that for some constants C, C ′ , C ′′ > 0 the fourth term is bounded by

This concludes the proof.

Remark 23. Following the proof, it is easy to see that ∥ f -f 0 ∥ 2 L 2 ( p 0 ) on the left-hand side of Equations (9) , (10) and (12) can be replaced with ∥ f -f 0 ∥ q L 2 ( p 0 ) for any q > 1 , changing the exponent -2 min( β,ν ) 2 ν + d on the right-hand side to -q min( β,ν ) 2 ν + d .

C Characterizing Reproducing Kernel Hilbert Spaces of Matérn Kernels

We start by describing the reproducing kernel Hilbert spaces (RKHSs) of the (truncated) intrinsic Matérn processes, proving that they coincide with (certain subspaces of) the manifold Sobolev spaces. We follow the ideas of Borovitskiy et al. [11], where the same was shown somewhat implicitly. We consider the more-involved case of the extrinsic Matérn processes immediately after.

The next lemma describes the RKHS of the intrinsic Matérn processes, including truncated variants. This result is easy to obtain since we have defined them in terms of the Karhunen-Loève expansions.

Lemma24. Let H J be the RKHS of the intrinsic Matérn Gaussian process with smoothness parameter ν truncated at the level J ∈ Z > 0 ∪ {∞} , and let { f j } ∞ j =0 be the orthonormal basis of LaplaceBeltrami eigenfunctions. The space H J is norm-equivalent-with constants depending only on ν, κ and σ 2 f -to the set of functions f = ∑ J j =1 b j f j with b j ∈ R , equipped with the inner product

In particular, H J ⊂ H ν + d/ 2 ( M ) for all J , and for every h ∈ H J we have ∥ h ∥ H J = ∥ h ∥ H ν + d/ 2 ( M ) .

Proof. By direct computation, the covariance k of the (truncated) intrinsic Gaussian process is

Hence, the covariance operator K : L 2 ( M ) → L 2 ( M ) defined by

is diagonal in the basis { f j } J j =1 , with Kf j = σ 2 f C ν,κ ( 2 ν κ 2 + λ j ) -( ν + d/ 2) f j . Then, Kanagawa et al. [28], Theorem 4.2 implies that H J consists of functions of form f = ∑ J j =1 a j f j satisfying

Using the elementary inequality min ( 2 ν κ 2 , 1 ) ≤ 2 ν κ 2 + λ 1+ λ ≤ max ( 2 ν κ 2 , 1 ) , we find that this space is norm-equivalent to the space H ν + d/ 2 J ( M ) of functions f = ∑ J j =1 a j f j satisfying

where the comparison constants √ σ 2 f C ν,κ min ( 1 , 2 ν κ 2 ) and √ σ 2 f C ν,κ max ( 1 , 2 ν κ 2 ) only depend on the parameters ν , κ , σ 2 f .

The next two theorems will be useful to characterize the RKHS of the extrinsic Matérn process on M . We start by a lemma relating the RKHS of the restriction of a Gaussian process to the original one.

Lemma 25. Assume that k is a kernel on R d , f ∼ GP(0 , k ) with almost surely continuous sample paths and ˜ H is the RKHS of k . If M⊆ R d is a submanifold, then the RKHS H corresponding to the restricted process f |M is the set of all restrictions g |M of functions g ∈ ˜ H equipped with the norm

Moreover there always exists an element g ∈ ˜ H such that g |M = f and ∥ g ∥ ˜ H = ∥ f ∥ H .

Proof. Lemma 5.1 in Yang and Dunson [63].

The last result will be used to characterize the RKHS of the extrinsic Matérn Gaussian processes using trace and extension operators. The second ingredient for this is the following.

Theorem 26. If s > D -d 2 then the restriction operator extends to a bounded linear map Tr : H s ( R D ) → H s -D -d 2 ( M ) . Moreover, for every u > 0 there exists a bounded right inverse Ex : H u ( M ) → H u + D -d 2 ( R D ) such that Tr ◦ Ex = id H u ( M ) where Tr corresponds to s = u + D -d 2 .

Proof. Theorem 4.10 in Große and Schneider [24].

The last two results allow us to characterize the RKHS of the extrinsic Matérn process on M .

Proposition 27. The RKHS H of a restricted extrinsic Matérn process f with smoothness parameter ν on M is norm-equivalent to the Sobolev space H ν + d/ 2 ( M ) .

Proof. Using Lemma 25, the RKHS H can be characterized as the set of functions f : M→ R that are the restrictions of some g ∈ ˜ H , where ˜ H is the RKHS of the ambient Matérn process ˜ f , with

Since ˜ H is norm-equivalent to the Sobolev space 7 H ν + D/ 2 ( R D ) -see for instance Kanagawa et al. [28]-by Theorem 26, for every f ∈ H we have

Similarly, for every g ∈ ˜ H with g |M = f , we have

Hence, taking the infimum, we obtain

The claim follows.

D Concentration and Sample Path Regularity

In this section, we prove that intrinsic, truncated intrinsic and extrinsic Matérn processes are subGaussian, uniformly with respect to the truncation parameter in the case of the truncated intrinsic Matérn process, and live in Hölder spaces with appropriate exponents. On our way to proving this, we characterize sample path regularity of (truncated) intrinsic Matérn processes, which is of independent interest. A simple way to do this would be to build on the results of Appendix C, using Driscoll's Theorem-given in Kanagawa et al. [28], Theorem 4.9-and the Sobolev Embedding Theorem-De Vito et al. [16], Theorem 4-but that would only give us that the sample paths are almost surely in CH γ ( M ) for every 0 < γ < ν -d/ 2 , γ / ∈ N , whereas here we improve the range of index to γ < ν .

Kolmogorov's continuity criterion is a standard tool to show that a given stochastic process has a Hölder continuous version: we re-prove it here because we will need a form of the result which gives explicit control of the Hölder norms, which is not usually included in the respective statement.

Lemma 28 (Kolmogorov's continuity criterion) . If g ∼ Π is a zero-mean Gaussian process on [0 , 1] d with

for some 0 < ρ ≤ 1 and C > 0 , then there exists a version of g with samples paths in CH α ( [0 , 1] d ) for every 0 < α < ρ . Moreover for every α < ρ this version satisfies E ∥ g ∥ 2 CH α ([0 , 1] d ) ≤ C ′ where C ′ < + ∞ depends only on C, ρ, α and d .

Proof. Take x, y ∈ [0 , 1] d , M > 0 and q ∈ N . Since the random variable g ( x ) -g ( y ) is Gaussian we have

where we have defined C q = C q (2 q )! 2 q q ! . The reason for considering the the 2 q th power will become clear later in the proof. By Markov's inequality, for every x, y ∈ [0 , 1] d we have

Now, take X = ∪ k ≥ 0 X k , X k = 2 -k Z d ∩ [0 , 1] d . Then, the previous inequality applied to any adjacent x, y ∈ X k , where we see X k as a graph where two vertices are connected if they differ by 2 -k , and u = M 2 -kα , implies

Denote K = sup x,y ∈ X adjacent | g ( x ) -g ( y ) | ∥ x -y ∥ α . Summing over k ≥ 1 and adjacent points in X -and there are at most C 2 k ( d +1) of them where C > 0 is an absolute constant-gives us for q > d +1 2( ρ -α ) that

In particular, for all q > max ( 1 , d +1 2( ρ -α ) ) , we have

for some constant C C,α,ρ,d < + ∞ . This means that K is finite almost surely. Since X is dense in [0 , 1] d and g is almost surely uniformly continuous on X by the classical version of the Kolmogorov's continuity criterion-see for instance Lototsky and Rozovsky [34]g admits a unique continuous extension to [0 , 1] d on a probability one event A . Let us define

for x ∈ [0 , 1] d . For any x, y ∈ [0 , 1] d and x n → x, y n → y, x n , y n ∈ X we have on A

On the complement of A , we have g = 0 . Hence, g is α -Hölder continuous on [0 , 1] d with the same (random) constant K , since

Since g is a version of g , the claim follows.

The next lemma applies our version of Kolmogorov's criterion, Lemma 28, to the intrinsic Matérn processes on M by considering charts. This allows us to show that the sample paths are almost surely in CH γ ( M ) for every γ < ν , which is both used in our arguments and also of independent interest. For the claims in Appendix B, we need to ensure that this property holds somewhat uniformly with respect to the truncation parameter, which is why we tracked the constants in our proof of Kolmogorov's criterion. As we will see, the main difficulty in the proof of the next result will be to tackle the case of regularity strictly larger than 1 .

Lemma 29. Let f ∼ Π n be an intrinsic Matérn process with smoothness parameter ν > 0 truncated at J n ∈ N ∪ {∞} . Then for every γ < ν we have

Proof. We start with the case ν ≤ 1 . Take 1 ≤ l ≤ L and define h l = ( χ l f ) ◦ ϕ -1 l . Then h l is a Gaussian process with covariance kernel ˜ K l given by

where x, y ∈ V l and K ( x, y ) = Cov( f ( x ) , f ( y )) is the covariance kernel of f . Let ˜ H l be the RKHS induced by ˜ K l . We seek to apply Lemma 28 to h l . For all x, y ∈ V l , where we recall that we can assume that V l = (0 , 1) d , , we have

where ∥ φ ∥ 2 CH ν ( V l ) can potentially be infinite or unbounded: we will show this is not the case. To do this it suffices to show that we have a continuous embedding ˜ H l ↪ →CH ν ( V l ) , that is ∥·∥ CH ν ( V l ) ≲ ∥·∥ ˜ H l . The RKHS ˜ H l is by definition the completion of

with respect to the topology induced by the RKHS norm

Denote the RKHS of K by H . By Theorem 16, and by the equality ∥·∥ H = ∥·∥ H ν + d/ 2 ( M ) on H which follows by Lemma 24, we have

Therefore, we have a continuous embedding ˜ H l ↪ → H ν + d/ 2 ( R d ) with ∥·∥ H ν + d/ 2 ( R d ) ≲ ∥·∥ ˜ H l on ˜ H l . By the Sobolev Embedding Theorem in R d -see for instance Triebel [50], Section 2.7.1,

Remark 2-we have B ν + d/ 2 2 , 2 ( R d ) = H ν + d/ 2 ( R d ) ↪ →CH ν ( R d ) , which implies ˜ H l ↪ →CH ν ( R d ) by composition. Thus, there exists a constant C = C ν such that

for x, y ∈ V l . Hence, by applying Lemma 28, there exists a version ˜ h l of h l with almost surely α -Hölder continuous sample paths for every α < ν . Now consider ˜ f = ∑ L l =1 ˜ h l ◦ ϕ l . Then ˜ f is a version of f . We proceed to bound E f ∼ Π n [ ∥ ˜ f ∥ 2 CH α ( M ) ] . For any 1 ≤ l, r ≤ L write

where C is the Lipshitz constant of ϕ r ◦ ϕ -1 l which is well defined and finite because this composition is a diffeomorphism and K is a random constant with E f ∼ Π n K 2 ≤ C α,ν,d . Hence

which gives the ν ≤ 1 case.

We now turn to the general case. The proof will be similar to the one of Ghosal and van der Vaart [20], Proposition I.3 although we need to control the Hölder norms and work through charts since our Gaussian processes are supported on manifolds. Assume for simplicity that d = 1 , 1 < ν ≤ 2 , otherwise it suffices to introduce coordinates and to proceed by induction on ⌊ ν ⌋ . Let l ∈ { 1 , . . . , L } , and as before define ˜ K l ( x, y ) = ( χ l ◦ ϕ -1 l ) ( x ) ( χ l ◦ ϕ -1 l ) ( y ) K ( ϕ -1 l ( x ) , ϕ -1 l ( y ) ) the covariance kernel of h l = ( χ l f ) ◦ ϕ -1 l as well as ˜ H l its RKHS.

First, let us construct an L 2 (Ω) -derivative ˙ h l of h l , where L 2 (Ω) is the space of random variables with finite variance with ⟨ a, b ⟩ L 2 (Ω) = E ( ab ) . This derivative is a square integrable process on V l such that

as h → 0 , for all x ∈ V l . For this, we will first show that ∂ ˜ K l ∂x ( x, · ) ∈ ˜ H l for every x ∈ V l -here ∂ ˜ K l ∂x denotes the derivative of the function ˜ K l ( · , · ′ ) with respect to the first argument-and that

We first show that ˜ K l ( x + δ, · ) -˜ K l ( x, · ) h is a Cauchy net 8 in ˜ H l . We have

where in (163) the derivative φ ′ exists because, exactly as in the case ν ≤ 1 , we can show show that ˜ H l ↪ →CH ν ( R d ) . This also implies that for a constant C = C ν we have

As | h -h ′ | ν -1 → 0 when h, h ′ → 0 , because ν > 1 , this proves that ˜ K l ( x + δ, · ) -˜ K l ( x, · ) h is a Cauchy net in ˜ H l : by completeness of ˜ H l it converges in ˜ H l to a limit. Since by general properties of RKHSs, convergence in ˜ H l implies pointwise convergence, the limit satisfies

Hence the partial derivative ∂ ˜ K l ∂x ( x, y ) exists for all y and ∂ ˜ K l ∂x ( x, · ) ∈ ˜ H l . Moreover, by the isometry L 2 (Ω) ∋ h l ( x ) ↦→ E f ∼ Π n h l ( x ) h l ( · ) = ˜ K l ( x, · ) ∈ ˜ H l , we deduce that h l is actually L 2 (Ω) -differentiable, with an L 2 (Ω) -derivative denoted as ˙ h l , and that the derivative process ˙ h l is Gaussian, as it is an L 2 (Ω) -limit of Gaussian random variables, satisfying E f ∼ Π n ˙ h l ( x ) ˙ h l ( y ) = 〈 ∂ ˜ K l ∂x ( x, · ) , ∂ ˜ K l ∂x ( y, · ) 〉 ˜ H l

Having established the existence of an L 2 (Ω) -derivative ˙ h l of the process h l , we would like now to show that ˙ h l possesses a ( γ -1) -regular version for every γ < ν . For this, we would like to apply Lemma 28 to ˙ h l . Notice that, by isometry, for all h > 0 we have

Therefore, by the same arguments as above, we have

where the transition from the second-to-last line to the last line is similar to (163)-(165).

Therefore, we can apply Lemma 28 to ˙ h l , and find a version ˜ h ′ l of ˙ h l with sample paths in CH α -1 ( V l ) almost surely for all α < ν , such that

This gives Hölder regularity of the respective derivatives: we now integrate these to obtain a Hölderregular version of the process itself. Take any c l ∈ (0 , 1) and consider ˜ h l = h l ( c l ) + ∫ · c l ˜ h ′ l ( t ) d t . Then since ˜ h ′ l is almost surely in CH α -1 ( V l ) , ˜ h l is has sample paths almost surely in CH α ( V l ) . Moreover, it is easy to check using our previous results that ˜ h l has an L 2 (Ω) -derivative given by ˜ h ′ l . This implies that ˜ h l is a version of h l .

To conclude the argument, we construct ˜ f from the obtained parts, by pulling ˜ h l back from the charts to the manifold. Consider ˜ f = ∑ L l =1 ˜ h l ◦ ϕ l . Arguing as in the case ν ≤ 1 , we see that ˜ f is a version of f with CH α ( M ) sample paths for every α < ν , and for every α < ν we have E f ∼ Π n ∥ ˜ f ∥ 2 CH α ( M ) ≤ C α,ν < + ∞ . This gives the claim.

With this, it is easy to prove that all Matérn Gaussian processes considered in this paper can be seen as Gaussian random elements in the Banach space ( C ( M ) , ∥·∥ C ( M ) ) of continuous functions on M . This allows us to use the same proof scheme as in van der Vaart and van Zanten [56] through the control of the concentration functions defined in Appendix B. It is also important that we work with Gaussian random elements in C ( M ) -and not only with the classical notion of Gaussian process, as the concentration functions are defined using the Gaussian random element RKHS defined in van Zanten and van der Vaart [59], which can potentially be different from the classical RKHS. Fortunately, when the process is a Gaussian random element in C ( M ) , van Zanten and van der Vaart [59], Theorem 2.1 implies that the two notions of RKHS coincide.

Corollary 30. The intrinsic Matérn Gaussian processes of Definition 4, their truncated versions as in Theorem 6 as well as the extrinsic Matérn Gaussian processes of Definition 7 are Gaussian random elements in ( C ( M ) , ∥·∥ C ( M ) ) .

Proof. By Lemma 18, it suffices to show that the processes have almost surely continuous sample paths. Since Euclidean Matérn Gaussian processes have continuous sample paths, this implies the same for their restrictions, the extrinsic Matérn Gaussian processes on M . For the intrinsic Matérn process, we apply Lemma 29.

Using Lemma 29 and known properties of Euclidean Matérn processes, we now show, in a sense, that all of the Matérn processes presented in this paper are sub-Gaussian, in a manner which holds uniformly with respect to the truncation parameter in the case of the truncated intrinsic Matérn process, and live in Hölder spaces with appropriate exponents. This result is used to control Hölder norms when going from the error at input locations to the L 2 ( p 0 ) -error. We use the notation Π n to emphasize that the prior depends on n when we consider a truncated intrinsic Matérn process.

Lemma 31. For f ∼ Π n with Π n the prior in either Definition 4, Theorem 6 or Definition 7, for every ν > 0 and γ < ν, γ / ∈ Z > 0 , there exists a constant σ ( f ) = σ γ ( f ) independent of n we have for x > 0 that

Proof. We start with the restriction f of an extrinsic Matérn process ˜ f to M , as in Definition 7. By van der Vaart and van Zanten [56], Section 3.1, for every γ < ν we have ˜ f ∈ CH γ ( [0 , 1] D ) almost surely. By Ghosal and van der Vaart [20], Lemma I.7, for every γ < ν , ˜ f is a Gaussian random element in the Banach space CH γ ( [0 , 1] D ) . In particular, by the Borell-TIS inequality [20, Proposition I.8] we have for x > 0 that

where σ ( ˜ f ) = ( E ∥ ∥ ˜ f ∥ ∥ 2 CH γ ([0 , 1] D ) ) 1 / 2 < ∞ . Since M is smooth, the restriction f also satisfies for x > 0 the expression

perhaps for a possibly larger constant σ ( f ) . Finally, the case of the intrinsic Matérn process f ∼ Π n truncated at J n ∈ Z > 0 ∪ {∞} follows in the same manner, as we have shown in Lemma 29 that sup n ≥ 1 E f ∼ Π n ∥ f ∥ 2 CH α ( M ) ≤ C α,ν .

E Small Ball Asymptotics

Here, we bound the probability that a Matérn Gaussian process lies in the ε -ball with respect to the L ∞ -norm for small ε . For a Banach space B , an element x ∈ B and a number r ∈ R > 0 , let us denote the closed r -ball around x by B B r ( x ) . We start by an upper bound on the metric entropy of Sobolev balls on M with respect to the uniform norm.

Lemma 32 (Entropy of Sobolev balls) . For any s > 0 define the ε -covering number of A ⊆ H s ( M ) with respect to the norm ∥·∥ L ∞ ( M ) by

Then for any s > d/ 2 , there exist C, ε 0 > 0 such that for every ε ≤ ε 0

where the left-hand side of the inequality above, as a function of ε , is called the metric entropy of the Sobolev ball B H s ( M ) 1 (0) with respect to the uniform norm ∥·∥ L ∞ ( M ) .

Proof. Using charts we will reduce the problem to the entropy of the unit ball of the Sobolev space H s ( [0 , 1] d ) for which the upper bound is known. Take f ∈ B H s ( M ) 1 (0) and consider approximations of f by g of the form

for some functions g l : V l → R where V l ⊆ R d . We have

This means that to approximate f by g uniformly on M , we need to choose the functions g l that approximate f ◦ ϕ -1 l well with respect to the uniform norm on [0 , 1] d .

Next, we show that the functions f ◦ ϕ -1 l are contained in an Euclidean Sobolev ball of radius R , with R depending only on ν and the atlas. To do this we use Große and Schneider [24], Lemma 2.1 to write

Note, importantly, that the remark just above Große and Schneider [24], Lemma 2.1 allows us to consider Besov spaces B s 2 , 2 coinciding with the Sobolev spaces H s instead of the Besov spaces B s 2 , ∞ -to get from the second line to the third. Note also that the constant hidden behind the notation ≲ in the last line where we use Theorem 16 is the radius R . Without loss of generality we assume R = 1 . By the Euclidean counterpart of the result we are proving [21, Theorem 4.3.36], we have

Let h 1 , .., h J ∈ B H s ([0 , 1] d ) 1 (0) be such that B H s ([0 , 1] d ) 1 (0) ⊂ ⋃ J j =1 B L ∞ ([0 , 1] d ) ε/L ( h k ) . Then for any f ∈ B H s ( M ) 1 (0) there exists a sequence { j l } L l =1 ⊆ { 1 , .., J } such that

This shows that N ( ε, B H s ( M ) 1 (0) , ∥·∥ L ∞ ( M ) ) ≤ LJ , where L is just the number of charts, thereby proving the claim.

For the related diffusion spaces [16], the RKHS corresponding to the heat (diffusion) kernels, Castillo et al. [12] uses the results of Coulhon et al. [14] to bound the entropy in terms of a wavelet frame instead of relying on charts. We believe this alternative proof scheme should work in our case as well. However, we could not, to the best of our effort, get a tight-enough bound for the Sobolev spaces by directly using the results of Coulhon et al. [14] and therefore we chose to rely on charts instead.

Having established regularity properties for our prior processes, we now turn to the small ball problem : we want to find sharp lower bounds on P ( ∥ f ∥ L ∞ ( M ) < ε ) where f ∼ Π is our prior process. This will be crucial in order to control the concentration functions used in Appendix B. In fact, it is well-known that this problem is closely related to the estimation of the metric entropy of the unit ball of the RKHS of f with respect to the uniform norm: see Li and Linde [31] for details. Since we have already characterized the RKHS of our processes in Proposition 27 and Lemma 24, we are able to lower bound the small-ball probabilities. The technicality here involves getting a bound uniform in the truncation parameter for the truncated intrinsic Matérn process, as the truncated Matérn process is a sequence of priors rather than a fixed prior.

Lemma 33. If f ∼ Π n where Π n is the prior in either Definition 4 and Theorem 6 or Definition 7 with smoothness parameter ν > d/ 2 , then there exist two constants C, ε 0 > 0 that do not depend on n such that for all ε ≤ ε 0 we have -ln P ( ∥ f ∥ L ∞ ( ) < ε ) ≤ Cε -d ν .

Proof. Because the processes are Gaussian random elements in C ( M ) by Lemma 18, their stochastic process RKHS given by Proposition 27 coincide with their Gaussian random element RKHS defined in van Zanten and van der Vaart [59]. Hence, for the non-truncated intrinsic and the extrinsic Matérn processes, the result follows by a direct application of Lemma 32 and Li and Linde [31], Theorem 1.2. For the intrinsic Matérn process truncated at J n it is not immediately clear that the constants C, ε 0 can be taken independent of n , so we go through the proof of Li and Linde [31], Proposition 3.1 to see that this is, in fact, the case. We first need a crude upper bound of the form

for some possibly large constant c > 0 . To get such a bound, we use Castillo et al. [12], Proposition 3 which shows the existence of a universal constant C > 0 such that for all ε ≤ min(1 , 4 σ ( f ))

where σ ( f ) = ( E f ∼ Π n ∥ f ∥ 2 L ∞ ( M ) ) 1 / 2 and n ( ε ) is defined in the following way in Castillo et al. [12], page 684 using auxiliary quantities l J that are defined in Li and Linde [31], page 1562, namely

with ( d ) = standing for the equality in distributions and the infimum being taken over all possible decompositions ∑ j ≥ 1 ε j h j with h j ∈ C ( M ) , ε j being a sequence of IID N(0 , 1) random variables, and the series being required to converge uniformly almost surely. 9

The function f = ∑ J n +1 j =1 ( 2 ν κ 2 + λ j -1 ) -ν + d/ 2 2 ε j f j -1 is a valid decomposition. Therefore

By the Sobolev Embedding Theorem and by Weyl's Law, given in Result 10, for every d/ 2 < γ < ν there exists a constant C = C γ, M such that for all J ∈ Z > 0 , allowing C to change from line to line, we have

By choosing J = 1 this gives us σ ( f ) ≤ C independent of n . Moreover, by choosing J ≥ Cε -d 2( ν -γ ) , again for a comparison constant C independent of n , this gives us n ( ε ) ≤ Cε -d 2( ν -γ ) for C independent of n . This implies using Castillo et al. [12], Proposition 3 that

for c > 0 independent of n .

With this crude bound, we can now continue the proof of Li and Linde [31], Proposition 3.1. For this, we need a metric entropy estimate. For this notice that for all J ∈ Z > 0 ∪ {∞} we have B H J 1 (0) ⊂ B H ∞ 1 (0) = B H ν + d/ 2 ( M ) 1 (0) , and therefore using Lemma 32, we have the metric entropy estimate for a constant C > 0 independent of J . Therefore following the proof of proposition 3.1 in Li and Linde [31] (with J ≡ 1 ) we find -ln P ( ∥ f ∥ L ∞ ( M ) < ε ) ≤ Cε -d ν for every ε ≤ ε 0 , where C, ε 0 > 0 are constants independent of n .

F Expressions for Pointwise Worst-case Errors

Let k be a kernel on some abstract input domain X , and let H k be the respective RKHS. Consider n input values X ⊆ X and let σ 2 ε > 0 be the noise variance. Define

Proposition 34. With notation above

Proof. To simplify notation, we shorten E ε ∼ N( 0 ,σ 2 ε I ) to E and denote α = K t X ( K XX + σ 2 ε I ) -1 . First of all, by direct computation,

As ∥ g ∥ H k = sup f ∈ H k , ∥ f ∥ H k ≤ 1 ⟨ g, f ⟩ H k , implying sup f ∈ H k , ∥ f ∥ H k ≤ 1 ⟨ g, f ⟩ 2 H k = ∥ g ∥ 2 H k , we have

We now move to the misspecified case. Consider the RKHS H c for some other kernel c : X ×X → R instead of H k . Then, continuing from (210), write

The next question is how to compute the norm on the right-hand side. There is not much hope of doing this exactly in the misspecified case: thus, we consider approximations. To this end, we take some large set of locations X ′ ⊆ X . Then we use ∥ g ∥ 2 H c ≈ g ( X ′ ) ⊤ C -1 X ′ X ′ g ( X ′ ) for g ( · ) = c ( t, · ) -∑ n j =1 α j c ( x j , · ) . As a result, we obtain the approximation

where v (e) ( t ) was first introduced in Section 4

Write

To compute spatial averages of this quantity, let g t ( · ) = c ( t, · ) -∑ n j =1 α j c ( x j , · ) , the same as g before, but now with explicit dependence on t . Similarly, put α t = K t X ( K XX + σ 2 ε I ) -1 . Then

From here we can also deduce that

where g X ′ ( X ′ ) = C X ′ X ′ -C X ′ X ( K XX + σ 2 ε I ) -1 K XX ′ .

G Full Experimental Details

All of our kernels were computed using GPJAX [38] and the GEOMETRIC KERNELS library. 10 We use three manifolds, each represented by a mesh: (i) a dumbbell-shaped manifold represented as a mesh with 1556 nodes, (ii) a sphere represented by an icosahedral mesh with 2562 nodes, and (iii) the Stanford dragon mesh, preprocessed to keep only its largest connected component, which has 100179 nodes. For the sphere, we also considered a finer icosahedral mesh with 10242 nodes, but this was found to have virtually no effect on the computed pointwise expected errors.

We use extrinsic Matérn and Riemannian Matérn kernels with the following hyperparameters: σ 2 f = 1 and σ 2 ε = 0 . 0005 . For the truncated Karhunen-Loève expansion, we used J = 500 eigenpairs obtained from the mesh. We selected smoothness values to ensure norm-equivalence of the intrinsic and extrinsic kernels' reproducing kernel Hilbert spaces, which was ν = 5 / 2 for the intrinsic model, and ν = 5 / 2 + d/ 2 for the extrinsic model, where d is the manifold's dimension. We used different length scales for each manifold: κ = 200 for the dumbbell, κ = 0 . 25 for the sphere, and κ = 0 . 05 for the dragon, selected to ensure that the Gaussian processes were neither approximately constant, nor white-noise-like. We considered data sizes of N = 50 , N = 500 , and N = 1000 , respectively, for the dumbbell, sphere, and dragon, sampled uniformly from the mesh's nodes, which in each case resulted in a reasonably-uniform distribution of points across the manifold. Finally, for the extrinsic pointwise error approximation, we used a subset X ′ uniformly sampled from each mesh's nodes, of size equal to the data size. For each respective test set, we used the full mesh. Each experiment was repeated for 10 different seeds.

To set the length scales for the extrinsic process, we used maximum marginal likelihood optimization on the full data, except for the dumbbell whose full data size is small and for which we instead generated a larger set consisting of 500 points. We optimzied only the length scale, leaving all other hyperparameters fixed. We used ADAM with a learning rate of 0 . 005 , and an initialization equal to the length scale κ of the intrinsic model, except for the dumbbell where this lead to divergence and we instead used an initial value of κ/ 4 . We ran the optimizer for a total of 1000 steps. With these settings, we found empirically that maximum marginal likelihood optimization always converged.

Peer reviews:
comments: SUMMARY:
The paper concerns Gaussian processes on manifolds. The authors present theorems for contraction rates for Matern Gaussian processes defined intrinsically and extrinsically through and embedding in a higher-dimensional Euclidean space. The authors shows that the rates are asymptotically equal in the two settings. Additionally, they treat the case of finitely truncated series expansion of the kernels to get a similar rate. Finally, it is shown experimentally that it can still be beneficial to work with the intrinsic geometry in the small sample domain.

STRENGTHS:
- extremely well-written paper. I found the exposition very clear
- novel theoretical results
- interesting findings (in line with what the authors state, I would not have expected the manifold dimension to show up in the extrinsic case)
- empirical study to cover the small sample case

WEAKNESSES:
- since the theorems are not in the main paper, one could perhaps consider if a longer format than NeurIPS (a journal paper) would be more suitable for the paper from a presentation point of view

QUESTIONS:
no questions

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the contraction rate(s) for both the intrinsic and extrinsic Mat\\'ern Gaussian process in compact Riemannian manifold. The authors proved that the (optimal) rate in both cases is $\\frac{2 \\min(\\beta, \\nu)}{2 \\nu +d}$, where $\\nu$ is the smooth parameter of the Mat\\'ern process, $\\beta$ is the regression function smoothness class, and $d$ is the dimension. The authors also showed with examples that the geometric models outperform the non-geometric ones through empirical error analysis.\n\nSTRENGTHS:\nThe results in this paper are novel and enlightening. Up to minor typos, I enjoy reading the paper. It is concerned with the fine topic of Gaussian processes on manifolds, and on why (and how) this kind of modeling is valuable through quantitative posterior contraction analysis. The main contribution is the (optimal) contraction rate of both intrinsic and ambient Mat\\'ern processes on compact manifolds in the nonparametric setting, the analysis of which is different from that in the Euclidean setting as the definition of the Mat\\'ern processes on manifolds is subtle (though the rates are the same). It is also shown how the underlying geometric analysis outperforms by numerical experiments.\n\nWEAKNESSES:\nAs for every (good) paper, there are always plenty of things remained to be done. For instance,\n\n(1)  In the current analysis it is assumed that the \"nugget\" $\\sigma_\\epsilon$ is given. In Bayesian setting, it would be interesting to know what happens if we put a prior on it (and more importantly, what prior to put). \n\n(2) The results, implicitly, compute the interpolation errors (as $L^2$ error based on $p_0$). What about extrapolation errors (that is, to predict an \"outside\" the domain)? This is also important in most geostatistics problems. \n\n(3) For the numerical experiments, the authors only consider the synthetic examples, e.g. dragon, sphere. It is interesting to see if the intrinsic and extrinsic modeling can bring a big difference for some real data. One such example is provided in [14].\n\nQUESTIONS:\nWhile the paper is generally well written, I feel the authors need to address the following points:\n\n(1) I catch the idea of the intrinsic vs extrinsic modeling quickly. But the authors may point out a reference for this terminology (is this inspired from [15])? \n\n(2) p.3, line 80: I guess the authors mean $f \\sim GP(m, k)$ instead of $GP(0,k)$.\n\n(3) p.3, line 115-116: the notation $C^{\\beta}$ for the H\\\"older class is nonstandard. Does this mean $\\mathcal{C}^{0, \\beta}$? I guess not since $\\beta > \\frac{d}{2}$ meaning that it can be larger than $1$. The authors may need to clarify this.\n\n(4) In Result 1, it is assumed that $f$ is mean zero. Is this also assumed in Theorems 5 and 6? It is known that for the Mat\\'ern processes in the Euclidean space, the mean function may raise the identifiability issue (see Stein's Interpolation of Spatial Data, or Tang, Zhang and Banerjee's paper On identifiability and consistency of the nugget in Gaussian spatial process models). There are also related discussions in [24]. The authors may need to clarify, and add a few sentences in the paper.\n\n(5) p.4, Assumption 3: the authors may \"indicate\" $\\sigma_\\epsilon$ is known earlier, as this is important for the Bayesian workers.\n\n(6) Regarding the use of Theorems 5 and 6: it is clear from the rates that one should take $\\nu = \\beta$ (i.e. if one can identify the function class, and set the same smoothness parameter in the Mat\\'ern process). However, we never know exactly $\\beta$. The question is whether there is an adaptive way to select $\\nu$. (Of course, this may be discussed in another paper. I only want to bring this question to the authors.)\n\n(7) Regarding Theorem 8: the rate is $\\frac{2 \\min(\\beta , \\nu)}{2 \\nu +d} = \\frac{2 \\nu}{2 \\nu +d}$ if $\\beta = \\nu$. This rate also appears in the prediction based on BLUE (best linear unbiased predictor) on the context of MLE. See e.g. Tang, Zhang and Banerjee's paper On identifiability and consistency of the nugget in Gaussian spatial process models, JRSS-B, 2021, page 1055. The authors may want to mention this as well.\n\n(8) There are a few more references that the authors may want to add.\n\n(a) Stein's book Interpolation of Spatial Data is one of the main references on the Mat\\'ern Gaussian processes. \n\n(b) Tang, Zhang and Banerjee's paper On identifiability and consistency of the nugget in Gaussian spatial process models studies the Mat\\'ern process (with nugget) in the Euclidean space, where the identifiability issue occurs as pointed out in (7). This is related to the setting of Theorem 8. Arafat, Porcu, Bevilacqua and Mateu's paper Equivalence and orthogonality of Gaussian measures on spheres, Journal of Multivariate Analysis, 2018 is also relevant. \n\n(c) Regarding the prior on $\\sigma^2_\\epsilon$, one such model is the conjugate Bayesian linear model in Banerjee's paper Modeling massive spatial datasets using a conjugate bayesian linear modeling framework, Spatial Statistics, 2020; this model was analyzed in Zhang, Tang and Banerjee's Exact Bayesian geostatistics using predictive stacking, arXiv:2304.12414, Section 2. I believe it should be possible to generalize the results in this paper to the conjugate bayesian linear model.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper establishes bounds on the contraction rate of matern processes on Riemannian manifolds. The authors study three variants:\n1) intrinsic matern process\n2) truncated intrinsic matern process\n3) extrinsic matern process\nand show that in each case, the optimal contraction rate can be achieved, which matches the Euclidean case.\n\nSTRENGTHS:\nThis is a fundamental problem, and it is remarkable that the authors are able to prove the same optimal contraction rates for both the intrinsic and extrinsic Riemannian matern processes. Furthermore, the manifold hypothesis has been receiving increasing attention recently, and so the analysis of Gaussian processes on manifolds is well motivated.\n\nI also appreciate the examples that the authors provided for illustrating the difference between intrinsic and extrinsic processes.\n\nWEAKNESSES:\nsee questions below\n\nQUESTIONS:\nRegarding the smoothness parameter nu:\n1) Can the authors provide intuition for why nu (as used in (7) and (10)) is the smoothness parameter (i.e. smoothness in what sense)?\n2) The Riemannian matern process (7) and the Euclidean matern process (10) appear quite different, can the authors explain the analogy between these two processes, and specifically, why is nu, as used in (7), comparable to nu as used in (10)? In particular, can the authors explain the comment from line 275 \"the intrinsic Matérn process, its truncated version and the extrinsic Matérn process all possess the same posterior contraction rates\", and how these two rates are comparable under the different assumptions?\n3) All theorems assume nu > d/2. Is this a standard assumption? How necessary is this assumption, and the theorems not work when nu is small? (and I have the a similar question for the beta > d/2 assumption in assumption 3 as well)\n\nRegarding the intrinsic matern process:\n4) expression (7) involves an sum over eigenfunctions of the laplace beltrami operator. The authors do mention that this sum can be truncated, but for a general manifold, it seems like even computing a single eigenfunction can be quite expensive. Can the authors comment on how this is done computationally (and what is the cost), when the manifold is an arbitrary one, e.g. the dragon?\n\n5) In figure 2, the authors give a dumbbell example which highlights the difference between intrinsic and extrinsic kernels -- one important difference seems to be that two points can be far away in manifold distance, but close in euclidean distance (under the embedding). Consequently, a function may have a small lipschitz constant wrt manifold distance, but a huge lipschitz constant wrt euclidean embedding distance. Intuitively, why is this not reflected in the contraction rates? Is it because the assumptions made do not care about things like lipschitz smoothness? (related to my earlier question of what is the meaning of nu?)\n\n\nRegarding the bound:\n\n6) What is contained in the constant C in the theorems, is this a universal constant? Or does it depend polynomially/exponentially on any problem parameters?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-31"}

comments: SUMMARY:
This paper investigates the theoretical properties and performance of Gaussian processes in machine learning, particularly when applied in geometric settings such as Riemannian manifolds. It compares intrinsic and extrinsic methods, with the former directly formulated on the manifold of interest and the latter requiring a higher-dimensional Euclidean space embedding. The research derives posterior contraction rates for three primary geometric model classes and shows that all three can lead to optimal procedures, given certain conditions. Empirical experiments support these theoretical findings, demonstrating better performance by intrinsic models in small-data regimes.

STRENGTHS:
NA

WEAKNESSES:
NA

QUESTIONS:
NA

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-08-01