SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The first conclusion says that if the rank gap is smaller than the posterior bias (a 'small-bias' condition), SGPR credible sets will have asymptotic coverage at least at the level of the original full posterior, though perhaps larger, including possibly tending to one. The second conclusion gives further conditions on the rank gap under which the asymptotic coverages will be the same. Together with Proposition 3.2, the goal is to obtain a credible set  $C_m^\delta$  of smallest possible diameter subject to having sufficient coverage.
3.2 Fixed design with rescaled Brownian motion prior
We now apply our general results to a specific GP and find conditions on the number of inducing variables  $m$  needed to get good UQ. Consider the domain  $\mathcal{X} = [0,1]$  with regularly spaced design points  $x_{i} = \frac{i}{n + 1 / 2}$  for  $i = 1,\dots ,n$ . For  $B$  a standard Brownian motion, consider as prior  $f = \sqrt{c_n} B$ , where  $c_{n} = (n + 1 / 2)^{\frac{1 - 2\gamma}{1 + 2\gamma}}$  for  $\gamma >0$ . Thus  $f$  is a mean-zero GP with covariance kernel  $k(x,x^{\prime}) = c_{n}(x\wedge x^{\prime})$ , where  $x\wedge x^{\prime}\coloneqq \min (x,x^{\prime})$ . The scaling factor  $c_{n}$  controls the smoothness of the sample paths of the GP and plays the same role as the lengthscale parameter for stationary GPs. The present rescaled Brownian motion is a suitable prior to model Hölder functions of smoothness  $\gamma \in (0,1]$  (e.g. [33, 17, 29]). In particular, for a true  $f_0\in C^\alpha$ ,  $\alpha \in (0,1]$ , one obtains (full) posterior contraction rate  $n^{-\frac{\alpha\wedge\gamma}{2\gamma + 1}}$  in the global  $L^2$ -loss [33]. We sometimes write  $Q_{\gamma}^{*} = Q^{*}$  for the corresponding SGPR to make explicit that the underlying prior is  $\gamma$ -smooth.
Theorem 3.5. Let  $f_0 \in C^\alpha[0,1]$ ,  $\alpha \in (0,1]$  and  $x \in (0,1)$ . Consider the SGPR  $Q_\gamma^*$  with rescaled Brownian motion prior of regularity  $\gamma \in (0,1]$  and  $m = m_n \to \infty$  inducing variables. Then
$$
E _ {f _ {0}} Q _ {\gamma} ^ {*} (f: | f (x) - f _ {0} (x) | > M _ {n} (n ^ {- \frac {\alpha \wedge \gamma}{1 + 2 \gamma}} + n ^ {\frac {2}{1 + 2 \gamma}} m ^ {- 1 + \alpha} + n ^ {\frac {1 / 2 - \gamma}{1 + 2 \gamma}} m ^ {- 3 / 2}) | \mathcal {D} _ {n}) \rightarrow 0
$$
as  $n\to \infty$  , where  $M_{n}\rightarrow \infty$  is any (arbitrarily slowly growing) sequence. If  $m_{n}\gg n^{\frac{1}{1 + 2\gamma} (\frac{2 + \alpha}{1 + \alpha})}$  then the contraction rate of the SGPR matches that of the full posterior  $n^{-\frac{\alpha\wedge\gamma}{1 + 2\gamma}}$
Theorem 3.5 shows that for  $m \gg n^{\frac{1}{1 + 2\gamma} \left(\frac{2 + \alpha}{1 + \alpha}\right)}$  inducing points, the SGPR attains the same pointwise contraction rate as the full posterior. If  $\gamma = \alpha$ , we then recover the optimal (minimax) rate of
convergence for a  $C^\alpha$ -smooth function. For instance if  $\alpha = \gamma = 1$ , our result indicates that one can do minimax optimal pointwise estimation based on  $m_n \gg \sqrt{n}$  inducing variables, a substantial reduction over the full  $n$  observations. Note this convergence guarantee can still hold in cases when the SGPR diverges from the posterior as  $n \to \infty$ , see Remark 3.7.
The restriction  $\alpha \in (0,1]$  comes from rescaling the underlying Brownian motion [33]. One can extend this to  $\alpha > 1$  by considering a smoother baseline process [33], such as integrated Brownian motion, but the more complex form of the resulting eigenvectors and eigenvalues of  $\mathbf{K}_{nn}$  make our explicit proof approach difficult. Note also that the prior fixes the value  $f(0) = 0$ . One can avoid this by adding an independent normal random variable to  $f$ , but since we consider pointwise inference at a point  $x > 0$ , this will not affect our results and we thus keep  $f(0) = 0$  for simplicity. Nonetheless, the message here is clear: for sufficiently many inducing variables (but polynomially less than  $n$ ), an SGPR based on a  $\gamma$ -smooth prior can estimate an  $\alpha$ -smooth truth at rate  $n^{-\frac{\alpha \wedge \gamma}{1 + 2\gamma}}$ . We next turn to UQ.
Theorem 3.6. Let  $f_0 \in C^\alpha[0,1]$ ,  $\alpha \in (0,1]$  and  $x \in (0,1)$ . Consider the SGPR  $Q_{\gamma}^*$  with rescaled Brownian motion prior of regularity  $\gamma \in (0,1]$  and  $m = m_n \gg n^{\frac{1}{1 + 2\gamma}\left(\frac{2 + \alpha}{1 + \alpha}\right)}$  inducing variables. For  $\delta \in (0,1)$ , let  $q_m^\delta \coloneqq P_{f_0}(f_0 \in C_m^\delta(x))$  denote the frequentist coverage of the  $1 - \delta$  credible set  $C_m^\delta(x)$  given by (10). Then as  $n \to \infty$ :
(i) (Undersmoothing case) If  $\alpha >\gamma$  , then  $q_{m}^{\delta}\rightarrow P(|N(0,1 / 2)|\leq z_{1 - \delta})\eqqcolon p_{\delta} > 1 - \delta$  for all  $f\in C^{\alpha}[0,1]$  , where  $P(|N(0,1)|\leq z_{1 - \delta}) = 1 - \delta$  and  $N(0,1 / 2)$  is the normal distribution with mean zero and variance  $1 / 2$
(ii) (Correct smoothing case) If  $\alpha = \gamma$ , then for each  $p \in (0, p_{\delta}]$ , there exists  $f \in C^{\alpha}[0,1]$  such that  $q_{m}^{\delta} \to p$ .
(iii) (Oversmoothing case) If  $\alpha < \gamma$ , there exists  $f \in C^{\alpha}[0,1]$  such that  $q_m^\delta \to 0$ .
Theorem 3.6 provides exact expressions for the asymptotic coverage when the credible interval  $C_m^\delta$  has diameter  $O(n^{-\frac{\alpha \wedge \gamma}{1 + 2\gamma}})$ . In the undersmoothing case  $\alpha > \gamma$ , the  $1 - \delta$  SGPR credible sets are conservative from a frequentist perspective, i.e., their coverage converges to a value strictly between  $1 - \delta$  and 1. For instance if  $\delta = 0.05$  (0.1), the  $95\%$  (90%) credible set will have asymptotic coverage  $99.4\%$  (98.0%), indicating one does not want to enlarge such credible sets if using enough inducing variables. A desired asymptotic coverage can also be achieved by targeting the credibility according to the formula in Theorem 3.6(i). Reducing  $\gamma$  towards zero will generally ensure coverage for Hölder functions, but this will also increase the size of the credible intervals to size  $O(n^{-\frac{\gamma}{1 + 2\gamma}})$ , making them less informative. Note that the convergence in (i) is uniform over  $C^\alpha$ -balls of fixed radius.
In the oversmoothing case  $\alpha < \gamma$ , the posterior variance underestimates the actual uncertainty and so the credible interval is too narrow, giving overconfident (bad) UQ for many Hölder functions. In the correct smoothing case  $\alpha = \gamma$ , where we obtain the minimax optimal contraction rate  $n^{-\frac{\alpha}{1 + 2\alpha}}$ , coverage falls between these regimes - it does not fully tend to zero, but one can find a function whose coverage is arbitrarily bad, i.e. close to zero.
The best scenario thus occurs when the prior slightly undersmoothes the truth, in which case the SGPR credible interval will have slightly conservative coverage but its width  $O\left(n^{-\frac{\gamma}{1 + 2\gamma}}\right)$  is not too much larger than the minimax optimal size  $O\left(n^{-\frac{\alpha}{1 + 2\alpha}}\right)$ . Our results match the corresponding ones for the full computationally expensive posterior and the main messages are the same: undersmoothing leads to conservative coverage while oversmoothing leads to poor coverage [29].
Remark 3.7 (Kullback-Leibler). If the number of inducing variables grows like  $n^{\frac{1}{1 + 2\gamma} \left( \frac{2 + \alpha}{1 + \alpha} \right)} \ll m \ll n^{\frac{2}{1 + 2\gamma}}$ , then the conditions of Theorems 3.5 and 3.6 are met, but the KL-divergence between the SGPR and the true posterior tends to infinity as  $n \to \infty$  (Lemma A.6 in the supplement). Thus one does not need the SGPR to be an asymptotically exact approximation of the posterior to get similarly good frequentist pointwise estimation and UQ. In particular, one can take  $m$  polynomially smaller in  $n$  than is required for the KL-divergence to vanish, see [5] for related bounds and discussion.
While [5] consider approximation quality as measured by the KL-divergence between the SGPR and true posterior, we consider here the different question of whether the SGPR behaves well for pointwise inference, even when it is not a close KL-approximation. The data-generating setup in [5] is also not directly comparable to ours. They take expectations over both the data  $(x,y)$  and prior
on  $f$ , so their results are 'averaged' over the prior, which downweights the effect of sets of small prior probability. In contrast, the frequentist guarantees provided here hold assuming there is a true function  $f_0$  which generates the data according to (1) (Assumption 2.1), meaning that one seeks to understand the performance of the method for any fixed  $f_0$ , including worst-case scenarios. Such setups can lead to very different conclusions, see e.g. Chapters 6.2-6.3 in [11].
4 Connections with other Gaussian process priors
The rescaled Brownian motion (rBM) prior we consider here can be thought of as an approximation to the Matérn process of regularity  $\gamma \in (0,1]$  since they both model  $\gamma$ -smooth functions. The rescaling factor  $c_{n} = (n + 1 / 2)^{\frac{1 - 2\gamma}{1 + 2\gamma}}$  in the covariance function  $k(x,x^{\prime}) = c_{n}(x\wedge x^{\prime})$  plays the same role as the lengthscale parameter for the Matérn and calibrates the GP smoothness. We thus expect similar theoretical results and conclusions to also hold for the Matérn process, which seems to be the case in practice, as we verify numerically in Section 5. Indeed, similar UQ properties to the rBM posterior are conjectured to hold for the full Matérn posterior [39], in particular that the coverage of credible sets satisfies similar qualitative conclusions to the three cases in Theorem 3.6.
On a more technical level, the success of GPs in nonparametric estimation is known to depend on their sample smoothness, as measured through their small ball probability [33, 36]. The posteriors based on both GPs converge to a  $C^\alpha$ -truth at rate  $n^{-\frac{\alpha \wedge \gamma}{2\gamma + 1}}$  in  $L^2$ -loss ([33] for rBM, [34] for Matérn), indicating the closeness of their small-ball asymptotics and hence that both GPs distribute prior mass similarly over their supports. This gives a more quantitative notion of the similarities of these GPs.
Theoretical guarantees do exist for the Matérn SGPR. Theorem 3.5 of [32] nicely establishes that in a bandit-like setting and if the true  $f_0$  is in the prior reproducing kernel Hilbert space  $\mathcal{H}$  (roughly  $\alpha = \gamma + 1/2$ ), then inflating credible sets by a factor depending on the rank gap and an upper bound for  $\| f_0\|_{\mathcal{H}}$  guarantees coverage. Our parallel results suggest that this inflation is not necessary, since SGPR credible sets in this case will already be conservative. However, since our proofs rely on explicit expressions available for rBM but not for the Matérn, it seems difficult to extend our approach to the different setting considered in [32] (and vice-versa); other techniques are required. Our results can thus be viewed as a guide as to what to expect when using a Matérn SGPR and provide a different perspective reinforcing the main messages of [32], namely that SGPRs can provide reliable, if conservative, UQ.
On the other hand, the squared exponential kernel seems to behave qualitatively differently in numerical simulations, giving different coverage behaviour in the cases considered in Theorem 3.6, see Section 5. This is due to the difference in the smoothness of the sample paths, with squared exponential prior draws being far smoother, in particular analytic. This leads to somewhat different UQ behaviour for the true posterior [12]. Rescaled Brownian motion is a less good approximation for the squared exponential kernel than for the Matérn, and one must thus be cautious about transferring the messages derived here to the squared exponential.
We next investigate empirically the applicability of our theoretical results in finite sample sizes and whether the conclusions extend to related settings, such as different designs and other GP priors. We consider various nonparametric regression settings of the form (1) with  $C^\alpha$ -smooth truths and  $\gamma$ -smooth priors. We compute both the full posterior (columns marked GP) and SGPR (marked SGPR) with  $m \ll n$  eigenvector inducing variables for  $f(x_0)$  at a fixed point  $x_0 \in \mathcal{X}$ , and report the root-mean square error (RMSE), negative log predictive density (NLPD), and the length and coverage of the 90%-credible intervals of the form (10) (i.e.  $\delta = 0.1$ ), see Section B in the supplement for definitions. The true noise variance in (1) is taken to be  $\sigma^2 = 1$ , but is considered unknown and is estimated by maximizing the log-marginal likelihood as usual. We ran all simulations 500 times and report average values and their standard deviations when relevant. Simulations were run on a 2 GHz Quad-Core Intel Core i5 processor on a 2020 MacBook Pro with 16GB of RAM.
Settings of the theoretical results. We consider the setting of our theoretical results in Section 3 with  $\mathcal{X} = [0,1]$  and  $x_{i} = i / (n + 1 / 2)$  for  $i = 1,\dots ,n$ . To generate the data, we take  $f_0(x) = |x - 0.5|^{\alpha}$ , which is exactly  $\alpha$ -Hölder at  $x_0 = 0.5$ , and investigate pointwise inference at  $x_0 = 0.5$  for
different choices of  $\alpha$ . As Gaussian priors, we consider (i) rescaled Brownian motion of regularity  $\gamma$  described in Section 3, (ii) the Matérn kernel with parameter  $\gamma$  and (iii) the square exponential kernel  $k(x,x^{\prime}) = \exp \{-\frac{1}{2\ell_{n}^{2}} (x - x^{\prime})^{2}\}$ , where the lengthscale  $\ell_{n} = n^{-\frac{1}{1 + 2\gamma}}$  is chosen to be appropriate for estimating  $\gamma$ -Hölder smooth functions [33]. We take  $m^{*} := n^{\frac{1}{1 + 2\gamma}\frac{2 + \alpha}{1 + \alpha}}$  inducing variables for the SGPR. Results are given in Table 1: Fixed Design,  $n = 1000$ . We also consider the related case of uniform random design where  $x_{i} \sim U(0,1)$  (Table 1: Random Design,  $n = 500$ ).
Prior	Coverage	Length	RMSE	NLPD
GP	SGPR	GP	SGPR	GP	SGPR	GP	SGPR	GP
Fixed Design: n = 1000, (α,γ) = (1.0,0.5)
rBM	0.98	0.98	0.41	0.41	0.09	0.09	-0.90 (0.21)	-0.90 (0.21)
Matérn	0.98	0.98	0.49	0.49	0.10	0.10	-0.68 (0.16)	-0.68 (0.16)
SE	0.91	0.91	0.65	0.65	0.19	0.19	-0.28 (0.30)	-0.28 (0.30)
Fixed Design: n = 1000, (α,γ) = (0.5,0.5)
rBM	0.74	0.74	0.41	0.41	0.18	0.18	-0.09 (0.35)	-0.09 (0.35)
Matérn	0.84	0.84	0.49	0.49	0.17	0.17	-0.40 (0.32)	-0.40 (0.32)
SE	0.88	0.88	0.65	0.65	0.21	0.21	-0.21 (0.31)	-0.21 (0.31)
Random Design: n = 500, (α,γ) = (1.0,0.5)
rBM	0.98	0.98	0.49 (0.02)	0.49 (0.02)	0.11	0.11	-0.65 (0.15)	-0.65 (0.15)
Matérn	0.96	0.96	0.59 (0.03)	0.59 (0.03)	0.13	0.13	-0.55 (0.14)	-0.55 (0.14)
SE	0.92	0.92	0.76 (0.08)	0.76 (0.08)	0.20	0.20	-0.02 (0.25)	-0.02 (0.25)
Random Design: n = 500, (α,γ) = (0.3,0.5)
rBM	0.25	0.25	0.49 (0.02)	0.49 (0.02)	0.37	0.37	2.23 (0.86)	2.23 (0.86)
Matérn	0.47	0.47	0.59 (0.02)	0.59 (0.02)	0.34	0.34	0.91 (0.66)	0.91 (0.66)
SE	0.71	0.71	0.77 (0.08)	0.77 (0.08)	0.31	0.31	0.36 (0.52)	0.36 (0.52)
Table 1: Comparison of SGPR and full posterior (marked GP) for  $90\%$  pointwise credible intervals for different values of  $(\alpha, \gamma)$ . For the SGPR we use  $m^{*} \coloneqq n^{\frac{1}{1 + 2\gamma}\frac{2 + \alpha}{1 + \alpha}}$  inducing variables, corresponding to 178, 316, 106 and 244 in the four  $(n, \alpha, \gamma)$  regimes above (top-to-bottom).
Theorem 3.6 predicts that for  $90\%$  credible sets  $(\delta = 0.1)$ ,  $m \gg m^{*}$  and  $\alpha > \gamma$ , the coverage should converge to  $p_{0.1} = P(|N(0,1/2)| < z_{0.9}) = 0.98$ . We see that for  $n = 1000$ , the observed coverages for both the full posterior and SGPR based on the Brownian motion prior are very close to this predicted value, and are indeed conservative. In the oversmoothing case  $\alpha < \gamma$ , coverage is poor and far below the nominal level, while in the correct smoothing case  $(\alpha = \gamma)$ , coverage is moderately below the nominal level. We see that the asymptotic theory is applicable for reasonable samples sizes.
The Matérn process behaves qualitatively similarly to Brownian motion in these three cases, though the (still conservative) limiting coverage in the undersmoothing case is predicted to be slightly different [39], as reflected in Table 1. On the other hand, the square exponential behaves differently in all cases and generally has wider sets with this lengthscale choice. The wider credible intervals and larger RMSE suggest greater bias of the posterior mean. While rescaled Brownian motion seems a reasonable guide for the Matérn, it does not seem so for the squared exponential, see Section 4.
For all three GP priors, the reported metrics are practically indistinguishable between the SGPR and the full GP, confirming that  $m \gg n^{\frac{1}{1 + 2\gamma}\frac{2 + \alpha}{1 + \alpha}}$  inducing variables seems sufficient to obtain virtually the same pointwise credible intervals as the full posterior. However, since  $m^{*} \ll n^{\frac{2}{1 + 2\gamma}}$  for any  $\alpha > 0$ , the KL-divergence between the SGPR and the full posterior grows to infinity as  $n \to \infty$  in the Brownian motion case (Lemma A.6), and hence the SGPR will differ from the posteriors in some ways. This reflects that one does not necessarily need to fully approximate the posterior in order to match its pointwise inference properties. Note that for fixed design, the length of the credible sets is not random since it depends only on the features  $x_{1}, \ldots, x_{n}$ , and so no standard errors are reported.
Other settings. We turn to simulation settings not covered by our theory. Given the above discussion, we consider only the Matérn process in dimension  $d = 10$  and the corresponding SGPR with  $m = m_d^* \coloneqq n^{\frac{d}{d + 2\gamma}}$ . As features, consider random design with (i)  $x_i \sim \mathcal{U}([-1/2, 1/2]^d)$  and (ii)  $X_i \sim \mathcal{N}_d(0, \Sigma_\rho)$  for  $[\Sigma_\rho]_{ij} = 1$  for  $i = j$  and  $[\Sigma_\rho]_{ij} = \rho$  otherwise. To better reflect correlation in real-world data, we also consider a semi-synthetic dataset with real features but simulated responses.
As features, we took the first  $d = 10$  columns and randomly sampled  $n = 2000$  rows from a Korean meteorological dataset [8]. In all cases, we consider estimating the  $\alpha$ -Hölder function  $f_0(x) = \| x - x_0\|^{\alpha}$  at  $x_0 = 0$ , and the resulting pointwise  $90\%$  credible sets at  $x_0$  coming from the Matérn SGPR. Results are presented in Table 2, where we see that for  $m = m_d^*$  the performance of the pointwise credible sets from the SGPR matches that of the full posterior in each design. Due to the higher dimension, credible intervals are less confident (wider) and have larger coverage.
Multidimensional Random Design,  $n = 1000$
Design	Smoothness	Coverage	Length	NLPD
Type	ρ	α	γ	SGPR	GP	SGPR	GP	SGPR	GP
Uniform	0.5	0.5	0.96	0.96	1.46 (0.03)	1.46 (0.03)	0.96 (0.26)	0.96 (0.26)
Gaussian	0.0	0.7	0.5	0.99	0.99	2.17 (0.05)	2.17 (0.05)	1.34 (0.11)	1.34 (0.11)
Gaussian	0.2	0.9	0.5	1.00	1.00	2.13 (0.06)	2.13 (0.06)	1.90 (0.14)	1.90 (0.14)
Gaussian	0.5	1.1	0.5	1.00	1.00	2.03 (0.06)	2.03 (0.06)	1.36 (0.12)	1.36 (0.12)
Semi-synthetic Data,  $n = 2000$
Smoothness	Coverage	Length	RMSE	NLPD
α	γ	SGPR	GP	SGPR	GP	SGPR	GP	SGPR	GP
1.0	0.5	1.00	1.00	2.29	2.29	0.16	0.16	3.22 (0.47)	3.22 (0.47)
Table 2: Comparison of SGPR and full posterior (GP) with Matérn prior for  $90\%$  pointwise credible intervals for different values of  $(\alpha, \gamma)$ . The SGPR uses 534 and 1002 inducing variables in the first four rows and fifth row, respectively.
In this work, we established the frequentist asymptotic behaviour of pointwise credible intervals coming from sparse variational Gaussian process regression (SGPR) with eigenvector inducing variables based on a rescaled Brownian motion prior. We showed that if the prior undersmooths the truth and with enough inducing variables, SGPR credible sets can provide reliable, though conservative, uncertainty quantification, with the coverage converging to a value strictly between the nominal level and one. If the prior oversmooths the truth, UQ can be poor. We further showed that it is not necessary for the SGPR to converge to the true posterior in KL-divergence to have similar behaviour of the credible sets. Our results suggest that properly calibrated SGPRs can perform reliable UQ. We verified these conclusions in simulations and discussed connections with other GPs.
Despite the widespread use of SGPR, there are still relatively few theoretical guarantees for their use, particularly for UQ. Our work provides some new results in this direction, but most of this research area is still wide open. A key step would be to prove similar results for the most commonly used GP priors, notably the Matérn and squared exponential. Similarly, one would like to extend these results to other choices of inducing variables, for instance the eigenfunctions of the kernel operator. While our results give some intuition of what one can expect, they rely on a careful analysis of Brownian motion with fixed design, and new ideas and techniques will be needed for these other settings.
It is also unclear what minimal number of inducing variables is needed to get good pointwise UQ. When  $\alpha = \gamma$ , the threshold for minimax convergence rates for estimation in  $L^2$  is  $n^{\frac{1}{1 + 2\alpha}}$  [22], which is smaller than our bound  $n^{\frac{1}{1 + 2\alpha}\left(\frac{2 + \alpha}{1 + \alpha}\right)}$ , see Section B in the supplement for some related numerical simulations. Our results are also non-adaptive since they assume fixed prior smoothness  $\gamma$ , whereas one often selects  $\gamma$  in a data-driven way, for instance by maximizing the marginal likelihood or evidence lower bound (ELBO). However, adaptive UQ is a subtle topic which typically requires further assumptions (e.g. self-similarity) on the true generative function to even be possible [30], and extending our results to such settings will require significant technical and conceptual work.
Acknowledgements. We are grateful to four anonymous reviewers for helpful comments that improved the manuscript. The authors would like to thank the Imperial College London-CNRS PhD Joint Programme for funding which supported Luke Travis in his studies.
[35] A. W. van der Vaart. Asymptotic statistics, volume 3 of Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, 1998.
[36] A. W. van der Vaart and J. H. van Zanten. Rates of contraction of posterior distributions based on Gaussian process priors. Ann. Statist., 36(3):1435-1463, 2008. ISSN 0090-5364.
[37] Y. Wang and D. M. Blei. Frequentist consistency of variational Bayes. J. Amer. Statist. Assoc., 114(527):1147-1161, 2019.
[38] V. Wild, M. Kanagawa, and D. Sejdinovic. Connections and equivalences between the nyström method and sparse variational Gaussian processes. art. arXiv:2106.01121, June 2021.
[39] Y. Yang, A. Bhattacharya, and D. Pati. Frequentist coverage and sup-norm convergence rate in Gaussian process regression. art. 1708.04753, 2017.
[40] W. W. Yoo and S. Ghosal. Supremum norm posterior contraction and credible sets for nonparametric multivariate regression. Ann. Statist., 44(3):1069-1102, 2016.
[41] F. Zhang and C. Gao. Convergence rates of variational posterior distributions. Ann. Statist., 48 (4):2180-2207, 2020.
Supplementary material to "Pointwise uncertainty quantification for sparse variational Gaussian process regression with a Brownian motion prior"
Throughout this section, for sequences  $a_{n}$  and  $b_{n}$ , we will write  $a_{n} \lesssim b_{n}$  if there exists  $C > 0$  such that  $a_{n} \leq C b_{n}$  for  $n$  large enough, and we will write  $a_{n} \asymp b_{n}$  if both  $a_{n} \lesssim b_{n}$  and  $b_{n} \lesssim a_{n}$ . Recall that for simplicity, we take the noise variance  $\sigma^2 = 1$  in the regression model (1).
A.1 General results on pointwise inference