SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Truncated random basis expansions have been studied extensively in the Bayesian literature in the Euclidean setting—see for instance Arbel et al. [2] and Yoo et al. [64] or Ghosal and van der Vaart [20], Chapter 11 for examples with priors based on wavelet expansions. It is known that truncating the expansion at a high enough level usually allows one to retain optimality. Instead of truncating deterministically, it is also possible to put a prior on the truncation level and resort to MCMC computations which would then select the optimal number of basis functions adaptively, at the expense of a more computationally intensive method—this is done, for instance, in van der Meulen et al. [53] in the context of drift estimation for diffusion processes. Random truncation has been proven to lead in many contexts to adaptive posterior contraction rates, meaning that although the prior does not depend on the smoothness  $\beta$  of  $f_0$ , the posterior contraction rate—up to possible  $\ln n$  terms—is of order  $n^{-\beta/(2\beta+d)}$ : see for instance Arbel et al. [2] and Rousseau and Szabo [43].
By analogy of the Euclidean case with its random Fourier feature approximations [40], we can call the truncated version of Definition 4 the manifold Fourier feature model, for which we now present our result.
Theorem 6. Let  $f$  be a Riemannian Matérn Gaussian process prior on  $\mathcal{M}$  with smoothness parameter  $\nu > d/2$ , modified to truncate the infinite sum to at least  $J_{n} \geq cn^{\frac{d(\min(1,\nu/\beta))}{2\nu+d}}$  terms, and let  $f_{0}$  satisfy Assumption 3. Then there is a  $C > 0$  such that
$$
\mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \mathbb {E} _ {f \sim \Pi (\cdot | \boldsymbol {x}, \boldsymbol {y})} \| f - f _ {0} \| _ {L ^ {2} (p _ {0})} ^ {2} \leq C n ^ {- \frac {2 \min  (\beta , \nu)}{2 \nu + d}}. \tag {10}
$$
The proof is essentially-the-same as the non-truncated Matérn, but involves tracking dependence of the inequalities on the truncation level  $J_{n}$ , which implicitly defines a sequence of priors rather than a single fixed prior.
This result is excellent news for the intrinsic models: it means that they inherit the optimality properties of the limiting one, even in the absence of the infinite sum—in spite of the fact that the corresponding finite-truncation prior places its probability on  $\mathcal{C}^{\infty}(\mathcal{M})$ . Again, the constant  $C > 0$  can be seen to depend on  $d,\sigma_f^2$ ,  $\nu ,\kappa ,\beta ,p_0,\sigma_\varepsilon^2$ ,  $\| f_0\|_{H^\beta (\mathcal{M})}$ ,  $\| f_0\|_{B_{\infty \infty}^\beta (\mathcal{M})}$ , and  $\| f_0\|_{\mathcal{CH}^\beta (\mathcal{M})}$ . This concludes our results for the intrinsic Riemannian Matérn priors. We now study what happens if, instead of working with a geometrically-formulated model, we simply embed everything into  $\mathbb{R}^d$  and formulate our models there.
3.3 Extrinsic Matérn Gaussian Processes
The results of the preceding sections provide good reason to be excited about the intrinsic Riemannian Matérn prior: the rates it obtains match the usual minimax rates seen for the Euclidean Matérn prior and Euclidean data, provided that we match the smoothness  $\nu$  with the regularity of  $f_0$ . Another possibility is to consider an extrinsic Gaussian process, that is, a Gaussian process defined over an ambient space. This has been considered by Yang and Dunson [63] for instance for the square-exponential process, in an adaptive setting where one does not assume that the regularity  $\beta$  of  $f_0$  is explicitly known, but where  $\beta \leq 2$ . In this section we prove a non-adaptive analog of this result for the Matérn process.
Definition 7 (Extrinsic Matérn prior). Assume that the manifold  $\mathcal{M}$  is isometrically embedded in the Euclidean space  $\mathbb{R}^D$ , such that we can regard  $\mathcal{M}$  as a subset of  $\mathbb{R}^D$ . Consider the Gaussian process with zero mean and kernel given by restricting onto  $\mathcal{M}$  the standard Euclidean Matérn kernel
$$
k _ {\nu , \kappa , \sigma_ {f} ^ {2}} (x, x ^ {\prime}) = \sigma_ {f} ^ {2} \frac {2 ^ {1 - \nu}}{\Gamma (\nu)} \left(\sqrt {2 \nu} \frac {\| x - x ^ {\prime} \| _ {\mathbb {R} ^ {D}}}{\kappa}\right) ^ {\nu} K _ {\nu} \left(\sqrt {2 \nu} \frac {\| x - x ^ {\prime} \| _ {\mathbb {R} ^ {D}}}{\kappa}\right) \tag {11}
$$
where  $\sigma_f, \kappa, \nu > 0$  and  $K_{\nu}$  is the modified Bessel function of the second kind [22].
Since the extrinsic Matérn process is defined in a completely agnostic way with respect to the manifold geometry, we would expect it to be less performant when  $\mathcal{M}$  is known. However, it turns out that the extrinsic Matérn process converges at the same rate as the intrinsic one, as given in the following claim.
Theorem 8. Let  $f$  be a mean-zero extrinsic Matérn Gaussian process prior with smoothness parameter  $\nu > d/2$  on  $\mathcal{M}$ , and let  $f_0$  satisfy Assumption 3. Then for some  $C > 0$  we have
$$
\mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \mathbb {E} _ {f \sim \Pi (\cdot | \boldsymbol {x}, \boldsymbol {y})} \| f - f _ {0} \| _ {L ^ {2} (p _ {0})} ^ {2} \leq C n ^ {- \frac {2 \min  (\beta , \nu)}{2 \nu + d}}. \tag {12}
$$
Theorem 8 is a surprising result because the optimal rates in this setting only require the knowledge of the regularity  $\beta$ , but not the knowledge of the manifold or the intrinsic dimension. More precisely, the prior is not designed to be an adaptive prior, since it is a fixed Gaussian process, but it surprisingly adapts to the dimension of the manifold, and thus to the manifold.
The proof is also based on control of concentration functions. The main difference is that, although the ambient process has a well known RKHS—the Sobolev space  $H^{s + D / 2}(\mathbb{R}^D)$ —the restricted process has a non-explicit RKHS, which necessitates further analysis. We tackle this issue by using results from Große and Schneider [24] relating manifold and ambient Sobolev spaces by linear bounded trace and extension operators, and from Yang and Dunson [63] describing a general link between the RKHS of an ambient process and its restriction. This allows us to show that the restricted process has an RKHS that is actually norm-equivalent to the Sobolev space  $H^{\nu +d / 2}(\mathcal{M})$ , which allows us to conclude the result in the same manner as in the intrinsic case.
As consequence, our argument applies mutatis mutandis in any setting where suitable trace and extension theorems apply, with the Riemannian Matérn case corresponding to the usual Sobolev results. In particular, our arguments therefore apply directly to other processes possessing similar RKHSs, such as for instance various kernels defined on the sphere—see e.g. Wendland [61], Chapter 17 and Hubbert et al. [26]. The constant  $C > 0$  can be seen to depend on  $d,D,\sigma_f^2$ ,  $\nu ,\kappa ,\beta ,p_0,\sigma_\varepsilon^2$ $\| f_0\|_{H^\beta (\mathcal{M})},\| f_0\|_{B_{\infty \infty}^\beta (\mathcal{M})},\| f_0\|_{\mathcal{CH}^\beta (\mathcal{M})}$  —notice that here  $C$  depends implicitly on  $D$  because of the presence of trace and extension operator continuity constants. We now proceed to understand the significance of the overall results.
3.4 Summary of Results
As a consequence of our previous results, fixing a single common data generating distribution determined by  $p_0$ ,  $f_0$ , under suitable conditions the intrinsic Matérn process, its truncated version, and the extrinsic Matérn process all possess the same posterior contraction rate with respect to the  $L^2(p_0)$ -norm, which depends on  $d$ ,  $\nu$ , and  $\beta$ , and is optimal if the regularities of  $f_0$  and the prior match. These results imply the following immediate corollary, which follows by convexity of  $\| \cdot \|_{L^2(p_0)}^2$  using Jensen's inequality.
Corollary 9. Under the assumptions of Theorems 5, 6 and 8, it follows that, for some  $C > 0$
$$
\mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \| m _ {\Pi (\cdot | \boldsymbol {x}, \boldsymbol {y})} - f _ {0} \| _ {L ^ {2} (p _ {0})} ^ {2} \leq C n ^ {- \frac {2 \min  (\beta , \nu)}{2 \nu + d}} \tag {13}
$$
where  $m_{\Pi(\cdot|\pmb{x}, \pmb{y})}$  is the posterior mean given a particular value of  $(x_i, y_i)_{i=1}^n$ .
When  $\nu = \beta$ , the optimality of the rates we present in the manifold setting can be easily inferred by lower bounding the  $L^2$ -risk of the posterior mean by the  $L^2$ -risk over a small subset of  $\mathcal{M}$  and using charts, which translates the problem into the Euclidean framework for which the rate is known to be optimal—see for instance Tsybakov [52].
To contextualize this, observe that even in cases where the geometry of the manifold is non-flat, the asymptotic rates are unaffected by the choice of the prior's length scale  $\kappa$  —in either the intrinsic, or the extrinsic case—but only by the smoothness parameter  $\nu$ . Indeed, the RKHS of the process is only determined—up to norm equivalence—by  $\nu$ , which plays an important role in the proofs. This, and the fact that extrinsic processes attain the same rates, implies that the study of asymptotic posterior contraction rates cannot detect geometry in our setting, as was already hinted by Yang and Dunson [63]. Hence, in the geometric setting, optimal posterior contraction rates should be thought of more as a basic property that any reasonable model should satisfy. Differences in performance will be down to constant factors alone—but as we will see, these can be significant. To understand these differences, we turn to empirical analysis.
From Theorems 5, 6 and 8, we know that intrinsic and extrinsic Gaussian processes exhibit the same posterior contraction rates in the asymptotic regime. Here, we study how these rates manifest themselves in practice, by examining how worst-case errors akin to those of Corollary 9 behave numerically. Specifically, we consider the pointwise worst-case error
$$
v ^ {(\tau)} (t) = \sup  _ {\| f _ {0} \| _ {H ^ {\nu + d / 2}} \leq 1} \mathbb {E} _ {\varepsilon_ {i} \sim \mathrm {N} \left(0, \sigma_ {\varepsilon} ^ {2}\right)} \left| m _ {\Pi (\cdot | \boldsymbol {x}, \boldsymbol {y})} ^ {(\tau)} (t) - f _ {0} (t) \right| ^ {2} \tag {14}
$$
where  $m_{\Pi(\cdot|\boldsymbol{x}, \boldsymbol{y})}^{(\tau)}$  is the posterior mean corresponding to the zero-mean Matérn Gaussian process prior with smoothness  $\nu$ , length scale  $\kappa$ , amplitude  $\sigma_f^2$ , which is intrinsic if  $\tau = \mathrm{i}$  or extrinsic if  $\tau = \mathrm{e}$ . We use a Gaussian likelihood with noise variance  $\sigma_\varepsilon^2$  and observations  $y_i = f_0(x_i) + \varepsilon_i$ , and examine this quantity as a function of the evaluation location  $t \in \mathcal{M}$ . By allowing us to assess how error varies in different regions of the manifold, this provides us with a fine-grained picture of how posterior contraction behaves.
One can show that  $v^{(\tau)}$  may be computed without numerically solving an infinite-dimensional optimization problem. Specifically, (14) can be calculated, in the respective intrinsic and extrinsic cases, using
$$
v ^ {(\mathrm {i})} (t) = k ^ {(\mathrm {i})} (t, t) - \mathbf {K} _ {t \mathbf {X}} ^ {(\mathrm {i})} \left(\mathbf {K} _ {\mathbf {X X}} ^ {(\mathrm {i})} + \sigma_ {\varepsilon} ^ {2} \mathbf {I}\right) ^ {- 1} \mathbf {K} _ {\mathbf {X} t} ^ {(\mathrm {i})} \tag {15}
$$
$$
v ^ {(\mathrm {e})} (t) \approx (\mathbf {K} _ {t \mathbf {X} ^ {\prime}} ^ {(\mathrm {i})} - \boldsymbol {\alpha} _ {t} \mathbf {K} _ {\mathbf {X X} ^ {\prime}} ^ {(\mathrm {i})}) (\mathbf {K} _ {\mathbf {X} ^ {\prime} \mathbf {X} ^ {\prime}} ^ {(\mathrm {i})}) ^ {- 1} (\mathbf {K} _ {\mathbf {X} ^ {\prime} t} ^ {(\mathrm {i})} - \mathbf {K} _ {\mathbf {X} ^ {\prime} \mathbf {X}} ^ {(\mathrm {i})} \boldsymbol {\alpha} _ {t} ^ {\top}) + \sigma_ {\varepsilon} ^ {2} \boldsymbol {\alpha} _ {t} \boldsymbol {\alpha} _ {t} ^ {\top} \tag {16}
$$
where, for the extrinsic case,  $\alpha_{t} = \mathbf{K}_{t\mathbf{X}}^{(\mathrm{i})}(\mathbf{K}_{\mathbf{XX}}^{(\mathrm{e})} + \sigma_{\varepsilon}^{2}\mathbf{I})^{-1}$ , and  $\mathbf{X}'$  is a set of points sampled uniformly from the manifold  $\mathcal{M}$ , the size of which determines approximation quality. The intrinsic expression is simply the posterior variance  $k_{\Pi (\cdot |x,y)}^{(\mathrm{i})}(t,t)$ , and its connection with worst-case error is a well-known folklore result mentioned somewhat implicitly in, for instance, Mutny and Krause [35]. The extrinsic expression is very-closely-related, and arises by numerically approximating a certain RKHS norm. A derivation of both is given in Appendix F. To assess the approximation error of this formula, we also consider an analog of (16) but instead defined for the intrinsic model, and compare it to (15): in all cases, the difference between the exact and approximate expression was found to be smaller than differences between models. By computing these expressions, we therefore obtain, up to numerics, the pointwise worst-case expected error in our regression model.
For  $\mathcal{M}$  we consider three settings: a dumbbell-shaped manifold, a sphere, and the dragon manifold from the Stanford 3D scanning repository. In all cases, we perform computations by approximating
Figure 3: Worst-case error estimates for the intrinsic and extrinsic processes, on the dumbbell, sphere, and dragon manifolds (lower is better,  $y$  axis is in the logarithmic scale). We see that, on the dumbbell and dragon manifold, intrinsic models achieve lower expected errors than extrinsic models for the ranges considered (top), and that their expected error consistently varies less as a function of space (bottom). In contrast, on the sphere, both models achieve similar performance, with differences between models falling within the range of variability caused by different random number seeds. We also see that the difference between computing the pointwise worst-case error exactly and approximately, in the intrinsic case where computing this difference is possible, is small in all cases.
the manifold using a mesh, and implementing the truncated Karhunen-Loève expansion with  $J = 500$  eigenpairs obtained from the mesh. We fix smoothness  $\nu = \frac{5}{2}$ , amplitude  $\sigma_f^2 = 1$ , and noise variance  $\sigma_{\varepsilon}^{2} = 0.0005$ , for both the intrinsic and extrinsic Matérn Gaussian processes. Since the interpretation of the length scale parameter is manifold-specific, for the intrinsic Gaussian processes we set  $\kappa = 200$  for the dumbbell,  $\kappa = 0.25$  for the sphere, and  $\kappa = 0.05$  for the dragon manifold. In all cases, this yielded functions that were neither close to being globally-constant, nor resembled noise. Each experiment was repeated 10 times to assess variability. Complete experimental details are given in Appendix G.
The length scales  $\kappa$  are defined differently for intrinsic and extrinsic Matérn kernels: in particular, using the same length scale in both models can result in kernels behaving very differently. To alleviate this, for the extrinsic process, we set the length scale by maximizing the extrinsic process' marginal likelihood using the full dataset generated by the intrinsic process, except in the dumbbell's case where the full dataset is relatively small, and therefore a larger set of 500 points was used instead. This allows us to numerically match intrinsic and extrinsic length scales to ensure a reasonably-fair comparison.
Figure 3 shows the mean, and spatial standard deviation of  $v_{\tau}(t)$ , where by spatial standard deviation we mean the sample standard deviation computed with respect to locations in space, rather than with respect to different randomly sampled datasets. From this, we see that on the dumbbell and dragon manifold—whose geometry differs significantly from the respective ambient Euclidean spaces—intrinsic models obtain better mean performance. The standard deviation plot reveals that intrinsic models have errors that are less-variable across space. This means that extrinsic models exhibit higher errors in some regions rather than others—such as, for instance, regions where embedded Euclidean and Riemannian distances differ—whereas in intrinsic models the error decays in a more spatially-uniform manner.
In contrast, on the sphere, both models perform similarly. Moreover, both the mean and spatial standard deviation decrease at approximately the same rates, indicating that the extrinsic model's predictions are correct about-as-often as the intrinsic model's, as a function of space. This confirms the view that, since the sphere does not possess any bottleneck-like areas where embedded Euclidean distances are extremely different from their Riemannian analogs, it is significantly less affected by differences coming from embeddings.
$^{2}$ Code available at: HTTPS://GITHUB.COM/ATERENIN/GEOMETRIC_ASYMPTOTICS.
In total, our experiments confirm that there are manifolds on which geometric models can perform significantly better than non-geometric models. This phenomenon was also noticed in Dunson et al. [17], where a prior based on the eigendecomposition of a random geometric graph, which can be thought as an approximation of our intrinsic Matérn processes, is compared to a standard extrinsic Gaussian process. In our experiments, we see this through expected errors, mirroring prior results on Bayesian optimization performance. From our theoretical results, such differences cannot be captured through posterior contraction rates, and therefore would require sharper technical tools, such as non-asymptotic analysis, to quantify theoretically.
In this work, we studied the asymptotic behavior of Gaussian process regression with different classes of Matérn processes on Riemannian manifolds. By using various results on Sobolev spaces on manifolds we derived posterior contraction rates for intrinsic Matérn process defined via their Karhunen-Loeve decomposition in the Laplace-Beltrami eigenbasis, including processes arising from truncation of the respective sum which can be implemented in practice. Next, using trace and extension theorems which relate manifold and Euclidean Sobolev spaces, we derived similar contraction rates for the restriction of an ambient Matérn process in the case where the manifold is embedded in Euclidean space. These theoretical asymptotic results were supplemented by experiments on several examples, showing significant differences in performance between intrinsic and extrinsic methods in the small sample size regime when the manifold's geometric structure differs from the ambient Euclidean space. Our work therefore shows that capturing such differences cannot be done through asymptotic contraction rates, motivating and paving the way for further work on non-asymptotic error analysis to capture empirically-observed differences between extrinsic and intrinsic models.
The authors are grateful to Mojmír Mutný and Prof. Andreas Krause for fruitful discussions concerning this work. PR and JR were supported by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No. 834175). VB was supported by an ETH Zürich Postdoctoral Fellowship. AT was supported by Cornell University, jointly via the Center for Data Science for Enterprise and Society, the College of Engineering, and the Ann S. Bowers College of Computing and Information Science.
[1] C. D. Aliprantis and K. C. Border. Infinite Dimensional Analysis: A Hitchhiker's Guide. Springer, 2006. Cited on page 29.
[2] J. Arbel, G. Gayraud, and J. Rousseau. Bayesian Optimal Adaptive Estimation Using a Sieve Prior. Scandinavian Journal of Statistics, 2013. Cited on page 6.
[3] T. Aubin. Some nonlinear problems in Riemannian geometry. Springer, 1998. Cited on page 14.
[4] I. Azangulov, A. Smolensky, A. Terenin, and V. Borovitskiy. Stationary Kernels and Gaussian Processes on Lie Groups and their Homogeneous Spaces I: the compact case. arXiv:2208.14960, 2022. Cited on page 1.
[5] I. Azangulov, A. Smolensky, A. Terenin, and V. Borovitskiy. Stationary Kernels and Gaussian Processes on Lie Groups and their Homogeneous Spaces II: non-compact symmetric spaces. arXiv:2301.13088, 2023. Cited on page 1.
[6] S. Banerjee. Modeling massive spatial datasets using a conjugate Bayesian linear modeling framework. Spatial Statistics, 2020. Cited on page 3.
[7] C. Berenfeld, P. Rosa, and J. Rousseau. Estimating a density near an unknown manifold: a Bayesian nonparametric approach. arXiv:2205.15717, 2022. Cited on page 4.
[8] A. Bhattacharya and D. B. Dunson. Nonparametric Bayesian density estimation on manifolds with applications to planar shapes. Biometrika, 2010. Cited on page 4.
[9] V. Borovitskiy, I. Azangulov, A. Terenin, P. Mostowsky, M. P. Deisenroth, and N. Durrande. Matérn Gaussian processes on graphs. In International Conference on Artificial Intelligence and Statistics, 2021. Cited on page 1.
[58] A. van der Vaart and J. Wellner. Weak Convergence and Empirical Processes. Springer Series in Statistics. Springer Verlag, 1996. Cited on page 22.
[59] H. van Zanten and A. van der Vaart. Reproducing kernel Hilbert spaces of Gaussian priors. Institute of Mathematical Statistics Collections, 2008. Cited on pages 17, 30, 32.
[60] X. Wang and G. Lerman. Nonparametric Bayesian regression on manifolds via Brownian motion. arXiv:1507.06710, 2015. Cited on page 4.
[61] H. Wendland. Scattered Data Approximation. Cambridge University Press, 2004. Cited on pages 7, 25.
[62] P. Whittle. Stochastic Processes in Several Dimensions. Bulletin of the International Statistical Institute, 1963. Cited on page 5.
[63] Y. Yang and D. B. Dunson. Bayesian manifold regression. The Annals of Statistics, 2016. Cited on pages 4, 7, 8, 24.
[64] W. W. Yoo, V. Rivoirard, and J. Rousseau. Adaptive supremum norm posterior contraction: Wavelet spike-and-slab and anisotropic Besov spaces. arXiv:1708.01909, 2017. Cited on page 6.
Here we describe preliminaries necessary for Appendices B to E. This includes some basic properties of the Laplace-Beltrami operator on compact manifolds, partitions of unity subordinate to atlases, function spaces such as Hölder, Sobolev and Besov spaces, general Gaussian random elements on Banach spaces, and a certain technical lemma. Hereinafter, the expression  $a \lesssim b$  means  $a \leq Cb$  for some constant  $C > 0$  whose value is irrelevant to our claims.
A.1 Laplace-Beltrami Operator and Subordinate Partitions of Unity
Let  $\mathcal{M}$  denote a compact connected Riemannian manifold without boundary of dimension  $d\in \mathbb{Z}_{>0}$ . The Laplace-Beltrami operator  $\Delta$  on  $\mathcal{M}$  is self-adjoint and positive semi-definite [48, Theorem 2.4]. Let  $(L^2 (\mathcal{M}),\langle \cdot ,\cdot \rangle_{L^2 (\mathcal{M})})$  denote the Hilbert space of square integrable (equivalence classes of) functions on  $\mathcal{M}$  with respect to the standard Riemannian volume measure.
By standard theory [13, 23], there exists an orthonormal basis  $\{f_j\}_{j=0}^{\infty}$  of  $L^2(\mathcal{M})$  consisting of the eigenfunctions of  $\Delta$ , namely  $\Delta f_j = -\lambda_j f_j$ , with  $\lambda_j \geq 0$ . We assume that the pairs  $(\lambda_j, f_j)$  are sorted such that  $0 = \lambda_0 \leq \lambda_j \leq \lambda_{j+1}$ . The growth of  $\lambda_j$  can be characterized as follows.
Result 10 (Weyl's Law). There exists a constant  $C > 0$  such that for all  $j$  large enough we have
$$
C ^ {- 1} j ^ {2 / d} \leq \lambda_ {j} \leq C j ^ {2 / d}. \tag {17}
$$
Proof. See Chavel [13], Chapter 1.