SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Luke Travis
luke.travis15@imperial.ac.uk
Kolyan Ray
kolyan.ray@imperial.ac.uk
We study pointwise estimation and uncertainty quantification for a sparse variational Gaussian process method with eigenvector inducing variables. For a rescaled Brownian motion prior, we derive theoretical guarantees and limitations for the frequentist size and coverage of pointwise credible sets. For sufficiently many inducing variables, we precisely characterize the asymptotic frequentist coverage, deducing when credible sets from this variational method are conservative and when overconfident/misleading. We numerically illustrate the applicability of our results and discuss connections with other common Gaussian process priors.
Consider the standard nonparametric regression model, where we observe  $n$  training samples  $\mathcal{D}_n = \{(x_1,y_1),\ldots ,(x_n,y_n)\}$  arising from the model
$$
y _ {i} = f \left(x _ {i}\right) + \varepsilon_ {i}, \quad \varepsilon_ {i} \sim^ {i i d} \mathcal {N} (0, \sigma^ {2}), \tag {1}
$$
where  $\sigma^2 > 0$  and the design points  $x_i \in \mathcal{X} \subset \mathbb{R}^d$  are either fixed or considered as i.i.d. random variables. Our goal is to predict outputs  $y^*$  based on new input features  $x^*$ , while accounting for the statistical uncertainty arising from the training data. A widely-used Bayesian approach is to endow  $f$  with a Gaussian process (GP) prior [25], which is especially popular due to its ability to provide uncertainty quantification via posterior credible sets.
While explicit expressions for the posterior distribution are available, a well-known drawback is that these require  $O(n^3)$  time and  $O(n^2)$  memory complexity, making computation infeasible for large data sizes  $n$ . To avoid this, there has been extensive research on low-rank GP approximations, where one chooses  $m \ll n$  inducing variables to summarize the posterior, thereby reducing computation to  $O(nm^2)$  time and  $O(nm)$  memory complexity, see the recent review [20].
We consider here the sparse Gaussian process regression (SGPR) approach introduced by Titsias [31], which is widely used in practice (see [1, 9] for implementations) and has been studied in many recent works [13, 21, 5, 6, 38, 28, 32, 22, 23]. While the computational dependence on the number of inducing variables  $m$  is relatively well-understood, fewer theoretical results are available on understanding how  $m$  affects the quality of statistical inference. In particular, it is crucial to understand how large  $m$  needs to be to achieve good statistical uncertainty quantification, ideally close to that of the true computationally expensive posterior. The present work contributes to this research direction by establishing precise theoretical guarantees for pointwise inference using a natural choice of SGPR with eigenvector inducing variables.
A main appeal of Bayesian methods is their ability to perform uncertainty quantification (UQ) via credible sets, often at the  $95\%$  level, see Figure 1 for a visualization involving a full GP and two
Figure 1: Plot of the (sparse) posterior means with the ribbons representing  $95\%$  posterior pointwise credible intervals for the i) SGPR with  $m = 5$  inducing variables (red); ii) SGPR with  $m = 106$  inducing variables (green); iii) full GP (blue) and iv) the true function (black). Here,  $n = 500$  and the covariance kernel is the rescaled Brownian motion with  $\gamma = 0.5$ .
SGPR approximations. We focus here on the canonical problem of pointwise estimation and UQ for  $f(x)$  at a single point  $x \in \mathcal{X}$ , which corresponds to the practically relevant problem of quantifying the uncertainty of a prediction  $f(x)$  based on a new input  $x$ . We study the frequentist behaviour of the resulting SGPR credible sets, with our main conclusion being that for well-calibrated priors and sufficiently many inducing points  $m$ , SGPRs can indeed provide reliable, though conservative, uncertainty quantification.
Concretely, we consider a rescaled Brownian motion prior that models Hölder smooth functions [33, 36, 29] and whose pointwise UQ for the corresponding full posterior was studied in [29]. We extend this analysis to the SGPR setting using the eigendecomposition of the sample covariance matrix as inducing variables, i.e. the optimal rank- $m$  posterior approximation. We provide a rigorous theoretical justification for pointwise estimation and UQ using this SGPR, provided  $m$  is large enough. Our main theoretical contributions are summarized that in dimension  $d = 1$ , for an  $\alpha$ -smooth truth  $f_0, \gamma$ -smooth prior,  $\alpha, \gamma \in (0,1]$ , and at least  $m \gg n^{\frac{1}{1 + 2\gamma} \left(\frac{2 + \alpha}{1 + \alpha}\right)}$  inducing variables:
1. The SGPR recovers the true value  $f_0(x)$  at a point  $x$  with convergence rate  $n^{-\frac{\min(\alpha, \gamma)}{2\gamma + 1}}$ . If  $\alpha = \gamma$ , this gives the minimax optimal rate  $n^{-\frac{\alpha}{2\alpha + 1}}$ .
2. For truths that are smoother than the prior  $(\alpha > \gamma)$ , SGPR credible sets are conservative with their frequentist coverage converging to a value strictly between their nominal level and one, which we characterize.
3. For truths that are rougher than the prior  $(\alpha \leq \gamma)$ , coverage can be arbitrarily bad.
4. It is not necessary for the KL divergence between the SGPR and true posterior to vanish as  $n \to \infty$  to get good (though conservative) coverage.
5. We provide a more general small-bias condition under which the coverage of SGPR credible sets is at least that of credible sets from the true posterior.
Our proofs exploit specific properties of Brownian motion to obtain explicit expressions, which allow us to obtain the above precise results. Such expressions seem difficult to obtain for other commons GPs (e.g. Matérn, squared exponential), meaning that existing theoretical guarantees for the Matérn SGPR apply to enlarged credible sets [32, 23]. Since our results indicate that pointwise SGPR credible sets are already conservative, such enlargements make the SGPR UQ procedure very
conservative, see Section 4. We provide a complimentary perspective to these works, studying true credible sets from a related GP, rather than modified credible sets from the Matérn.
The Brownian motion prior we consider here shares many similarities with the Matérn process, meaning our main qualitative conclusions also apply there, as we demonstrate in simulations. We thus verify the applicability of our limit theorems in numerical simulations, confirming they provide a good guide to the behaviour of SGPRs for finite sample sizes, including those based on Matérn priors. To the best of our knowledge, our work is the first establishing exact theoretical coverage guarantees for SGPR pointwise credible sets, and can be viewed as a guide of what to expect for the other Gaussian processes, see Section 4.
Related work. Rigorous theoretical guarantees for Bayesian UQ using the full posterior have received much recent attention. In parametric models, the classic Bernstein-von Mises theorem [35] ensures that regular credible sets are asymptotic confidence sets, justifying Bayesian UQ from a frequentist perspective. However, such a result fails to hold in high- and infinite-dimensional settings [10, 16], such as GPs, where the performance is sensitive to the choice of prior and model. Theoretical guarantees for UQ using full GPs have been established mainly involving  $L^2$ -type credible sets (e.g. [17, 7, 30]) which, while mathematically more tractable, are less reflective of actual practice than pointwise credible sets. Results for pointwise Bayesian UQ are more difficult to obtain and are correspondingly much rarer [29, 40, 39], see Section 3. From a technical perspective, we perform a careful and novel analysis of the gap between the pointwise behaviour of the true posterior and its SGPR approximation, after which we can invoke ideas from the proofs for the full posterior [29].
While the frequentist asymptotic properties of VB have increasingly been investigated, this is usually in the context of estimation, convergence rates or approximation quality (e.g. [24, 2, 41, 27, 26], including for SGPRs [5, 6, 22]) rather than UQ. Available results on mean-field VB often show that this factoriazable approach provides overconfident UQ by underestimating the posterior variance [3, 4], though there exist more positive results in certain low-dimensional models [37]. Regarding UQ for SGPRs, Nieman et al. [23] investigate the frequentist coverage of global  $L^2$ -credible sets from certain SGPRs, showing these can have coverage tending to 1, possibly after enlarging them. Vakili et al. [32] similarly show that enlarged pointwise or uniform credible sets can have good coverage in a different bandit-like setting.
Organization. In Section 2 we detail the problem setup, including notation and an overview of (sparse variational) Gaussian process regression. Main results on pointwise convergence and UQ for SGPRs based on a Brownian motion prior are found in Section 3, discussion on the connections with other GPs in Section 4, simulations in Section 5 and discussion in Section 6. In the supplement, we provide the proofs of our results (Section A) and additional simulations (Section B).
2 Problem setup and sparse variational Gaussian process regression
Recall that we observe  $n$  training examples  $\mathcal{D}_n = \{(x_1,y_1),\ldots ,(x_n,y_n)\}$  from model (1), and write  $\mathbf{y} = (y_{1},\dots ,y_{n})^{T}$  and  $\mathbf{f} = (f(x_1),\dots ,f(x_n))^T$ . We denote by  $P_{f}$  the probability distribution of  $\mathbf{y}$  from model (1) and by  $E_{f}$  the corresponding expectation. For  $\mathbf{u},\mathbf{v}\in \mathbb{R}^n$  we write  $\langle \mathbf{u},\mathbf{v}\rangle = \sum_{i = 1}^{n}\mathbf{u}_{i}\mathbf{v}_{i}$  for the inner-product on  $\mathbb{R}^n$  and  $\| \mathbf{u}\| = \langle \mathbf{u},\mathbf{u}\rangle^{1 / 2}$  for the usual Euclidean norm. Write  $C(\mathcal{X})$  for the space of continuous real-valued functions on  $\mathcal{X}$ . For  $\alpha \in (0,1]$ , we further define the class of Hölder smooth functions on [0,1] by  $C^\alpha = C^\alpha [0,1] = \{f:[0,1]\to \mathbb{R}:\sup_{x\neq y}\frac{|f(x) - f(y)|}{|x - y|^{\alpha}} < \infty \}$ . Throughout the paper we make the following frequentist assumption:
Assumption 2.1. There is a true  $f_0 \in L^2(\mathcal{X})$  generating the data  $\mathbf{y} \sim P_{f_0}$  according to (1).
We differentiate  $f$  coming from the Bayesian model with the 'true' generative  $f_0$ .
2.1 Sparse variational Gaussian process regression
In model (1), we assign to  $f \sim GP(\nu_0, k)$  a Gaussian process (GP) prior with mean function  $\nu_0: \mathcal{X} \to \mathbb{R}$  and covariance kernel  $k: \mathcal{X} \times \mathcal{X} \to \mathbb{R}$ . We henceforth take the prior mean  $\nu_0 = 0$  for simplicity since this will not alter the spirit of our results, see Remark 3.3 below. The resulting
posterior distribution is again a GP by conjugacy [25], with mean and covariance functions equal to
$$
\nu_ {n} (x) = \mathbf {k} _ {n} (x) ^ {T} (\mathbf {K} _ {n n} + \sigma^ {2} \mathbf {I} _ {n}) ^ {- 1} \mathbf {y},
$$
$$
k _ {n} (x, x ^ {\prime}) = k (x, x ^ {\prime}) - \mathbf {k} _ {n} (x) ^ {T} \left(\mathbf {K} _ {n n} + \sigma^ {2} \mathbf {I} _ {n}\right) ^ {- 1} \mathbf {k} _ {n} (x ^ {\prime}), \tag {2}
$$
where  $\mathbf{k}_n(x) = (k(x,x_1),\ldots ,k(x,x_n))^T$  and  $[\mathbf{K}_{nn}]_{ij} = k(x_i,x_j)$ . In particular, the posterior variance at a point  $x\in \mathcal{X}$  is  $\sigma_n^2 (x) = k_n(x,x)$ . In Section 3.2 below, we will specifically consider the choice  $\mathcal{X} = [0,1]$  and rescaled Brownian motion prior kernel  $k(x,x^{\prime}) = (n + 1 / 2)^{\frac{1 - 2\gamma}{1 + 2\gamma}}\min (x,x^{\prime})$  for  $\gamma >0$ .
We consider here the sparse variational Gaussian process (SGPR) approximation using inducing variables proposed by Titsias [31]. The idea is to summarize the posterior via a relatively small number of inducing variables  $\mathbf{u} = \{u_1,\dots ,u_m\}$ ,  $m\ll n$ , which are linear functionals of the prior process  $f$ . By assigning  $\mathbf{u}$  a multivariate normal distribution, one obtains a low-rank variational family. Titsias [31] explicitly computed the minimizer in Kullback Leibler sense between this family of GPs and the full posterior, which yields the SGPR  $Q^{*} = GP(\nu_{m},k_{m})$  with mean and covariance
$$
\nu_ {m} (x) = \mathbf {K} _ {x m} \left(\sigma^ {2} \mathbf {K} _ {m m} + \mathbf {K} _ {m n} \mathbf {K} _ {n m}\right) ^ {- 1} \mathbf {K} _ {m n} \mathbf {y}
$$
$$
k _ {m} \left(x, x ^ {\prime}\right) = k \left(x, x ^ {\prime}\right) - \mathbf {K} _ {x m} \mathbf {K} _ {m m} ^ {- 1} \mathbf {K} _ {m x ^ {\prime}} + \mathbf {K} _ {x m} \left(\mathbf {K} _ {m m} + \sigma^ {- 2} \mathbf {K} _ {m n} \mathbf {K} _ {n m}\right) ^ {- 1} \mathbf {K} _ {m x ^ {\prime}}, \tag {3}
$$
where  $\mathbf{K}_{xm} = (\mathrm{cov}(f(x),u_1),\ldots ,\mathrm{cov}(f(x),u_m)) = \mathbf{K}_{mx}^T$ $[\mathbf{K}_{mm}]_{ij} = \mathrm{cov}(u_i,u_j)$  and we write  $[\mathbf{K}_{nm}]_{ij} = \mathrm{cov}(f(x_i),u_j)$ . Recalling that  $\mathrm{cov}(f(x),f(x'))\coloneqq k(x,x')$  under the prior, for  $u_{j}$  of the form  $u_{j} = \sum_{i = 1}^{n}v_{j}^{i}f(x_{i})$  (see (4) below) we obtain  $\mathrm{cov}(f(x),u_j) = \sum_{j = 1}^{n}v_j^i k(x,x_i)$ . The SGPR variance at  $x\in \mathcal{X}$  is  $\sigma_m^2 (x) = k_m(x,x)$ . The computational complexity of obtaining  $\nu_{m}$  and  $k_{m}$  is  $O(nm^2)$ , which is much smaller than the  $O(n^{3})$  needed to compute the full posterior for  $m\ll n$ .
2.2 Eigenvector inducing variables
The approximation quality of the SGPR depends on the inducing variables being well chosen. We consider here arguably the conceptually simplest and most canonical sparse approximation based on the eigendecomposition of  $\mathbf{K}_{nn}$ . Let  $\mu_1 \geq \mu_2 \geq \dots \geq \mu_n \geq 0$  be the eigenvalues of the positive semi-definite matrix  $\mathbf{K}_{nn}$  with corresponding orthonormal eigenvectors  $\mathbf{v}_1, \ldots, \mathbf{v}_n$ . Writing  $\mathbf{v}_j = (v_j^1, \ldots, v_j^n)^T \in \mathbb{R}^n$ , we take as inducing variables
$$
u _ {j} = \mathbf {v} _ {j} ^ {T} \mathbf {f} = \sum_ {i = 1} ^ {n} v _ {j} ^ {i} f \left(x _ {i}\right), \quad j = 1, \dots , m, \tag {4}
$$
so that  $u_{j}$  is a linear combination of the inducing points placed at each data point with weights proportional to the entries of the  $j^{th}$ -largest eigenvector of  $\mathbf{K}_{nn}$ . Setting  $\mathbf{V}_m \coloneqq [\mathbf{v}_1 \dots \mathbf{v}_m] \in \mathbb{R}^{n \times m}$ , this choice of inducing points implies
$$
\mathbf {K} _ {x m} = \mathbf {k} _ {n} (x) ^ {T} \mathbf {V} _ {m}, \qquad \mathbf {K} _ {m m} = \operatorname {d i a g} (\mu_ {1}, \ldots , \mu_ {m}), \qquad \mathbf {K} _ {n m} = \mathbf {V} _ {m} \operatorname {d i a g} (\mu_ {1}, \ldots , \mu_ {m}) = \mathbf {K} _ {m n} ^ {T},
$$
see Section C.1 of [5]. Writing  $\mathbf{K}_{nn} = \sum_{i=1}^{n} \mu_i \mathbf{v}_i \mathbf{v}_i^T$  and substituting these expressions into (3), we obtain the SGPR with mean and covariance functions
$$
\nu_ {m} (x) = \mathbf {k} _ {n} (x) ^ {T} \left[ \sum_ {k = 1} ^ {m} \eta_ {k} \mathbf {v} _ {k} \mathbf {v} _ {k} ^ {T} \right] \mathbf {y} \tag {5}
$$
$$
k _ {m} (x, x ^ {\prime}) = k (x, x ^ {\prime}) - \mathbf {k} _ {n} (x) ^ {T} \left[ \sum_ {k = 1} ^ {m} \eta_ {k} \mathbf {v} _ {k} \mathbf {v} _ {k} ^ {T} \right] \mathbf {k} _ {n} (x ^ {\prime}), \tag {6}
$$
where  $\eta_{k} = \frac{1}{\sigma^{2} + \mu_{k}}$ . We study inferential properties of the SGPR with mean (5) and covariance (6).
Comparing these expressions with those for the full posterior (2), we see that this SGPR is the optimal rank- $m$  approximation in the sense that we have replaced  $(\mathbf{K}_{nn} + \sigma^2\mathbf{I}_n)^{-1} = \sum_{i=1}^{n}\eta_i\mathbf{v}_i\mathbf{v}_i^T$  in both the mean and covariance (2) with  $\sum_{i=1}^{m}\eta_i\mathbf{v}_i\mathbf{v}_i^T$ , corresponding to the  $m$  largest eigenvalues of  $\mathbf{K}_{nn}$ . These inducing variables are an example of interdomain inducing features [19], which can yield sparse representations in the spectral domain and computational benefits [14]. Computing this SGPR
involves numerically computing the first  $m$  eigenvalues and corresponding eigenvectors of  $\mathbf{K}_{nn}$ , which can be done in  $O(n^2 m)$  time using for instance Lanczos iteration [18].
This choice of SGPR using eigenvector inducing variables has been shown to minimize certain upper bounds for the Kullback-Leibler (KL) divergence between order  $m$  SGPRs and the true posterior [5], reflecting that it can provide a good posterior approximation. Indeed, one can show that for  $m = m_{n}$  growing fast enough (though sublinearly in  $n$ ), this SGPR converges to the posterior in the KL sense as  $n \to \infty$  [5]. Under weaker growth conditions on  $m$ , this SGPR still converges to the true generative  $f_{0}$  at the minimax optimal rate in the frequentist model [22], even when the SGPR does not converge to the true posterior as  $n \to \infty$ . Our results include both scenarios and will show that one can still obtain good frequentist UQ performance even for  $m$  small enough that the SGPR diverges from the full posterior (see Remark 3.7) unlike in [5].
3.1 General results on pointwise inference
We now present our main results concerning estimation and uncertainty quantification at a point  $x \in \mathcal{X}$  using the SGPR  $Q^{*}$  with eigenvector inducing variables. Since both the posterior and SGPR are GPs, their marginal distributions at  $x$  satisfy  $f(x)|\mathcal{D}_n \sim \mathcal{N}(\nu_n(x), \sigma_n^2(x))$  and  $f(x) \sim \mathcal{N}(\nu_m(x), \sigma_m(x)^2)$ , respectively. It thus suffices to study these quantities under the frequentist assumption that there is a 'true'  $f_0$  generating the data, i.e. Assumption 2.1. We further define the frequentist bias and variance of the (sparse) posterior mean:
$$
b _ {n} (x) = E _ {f _ {0}} \left(\nu_ {n} (x) - f _ {0} (x)\right), \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad b _ {m} (x) = E _ {f _ {0}} \left(\nu_ {m} (x) - f _ {0} (x)\right),
$$
$$
t _ {n} ^ {2} (x) = \operatorname {v a r} _ {f _ {0}} (\nu_ {n} (x)), \quad t _ {m} ^ {2} (x) = \operatorname {v a r} _ {f _ {0}} (\nu_ {m} (x)),
$$
where the expectation and variance are taken over  $\mathbf{y} \sim P_{f_0}$ . We have the following useful relationships between these quantities for the variational posterior and full posterior, which assumes fixed design, so that all statements are taken conditional on the design points  $x_1, \ldots, x_n \in \mathcal{X}$ . For simplicity, we take the noise variance to be  $\sigma^2 = 1$  in (1) for our theoretical results.
Lemma 3.1. For  $f_0 \in L^2(\mathcal{X})$ ,  $x \in \mathcal{X}$  and  $\mathbf{r}_m(x) := \left[ \sum_{j=m+1}^{n} \eta_j \mathbf{v}_j \mathbf{v}_j^T \right] \mathbf{k}_n(x)$ , we have:
$$
b _ {m} (x) = b _ {n} (x) - \left\langle \mathbf {r} _ {m} (x), \mathbf {f} _ {\mathbf {0}} \right\rangle \tag {7}
$$
$$
t _ {m} ^ {2} (x) = t _ {n} ^ {2} (x) - \left\| \mathbf {r} _ {m} (x) \right\| ^ {2} \leq t _ {n} ^ {2} (x) \tag {8}
$$
$$
\sigma_ {m} ^ {2} (x) = \sigma_ {n} ^ {2} (x) + \langle \mathbf {r} _ {m} (x), \mathbf {k} _ {n} (x) \rangle \geq \sigma_ {n} ^ {2} (x). \tag {9}
$$
The quantity  $\mathbf{r}_m(x)$  measures the rank gap due to ignoring the smallest  $n - m$  eigenvalues of  $\mathbf{K}_{mn}$ . The SGPR approximation increases the posterior variance  $\sigma_m^2 (x)\geq \sigma_n^2 (x)$  while reducing the frequentist variance of the posterior mean  $t_m^2 (x)\leq t_n^2 (x)$ . Thus if one can control the biases in (7), credible sets from the SGPR will have coverage at least as large as the full posterior. One can therefore ensure good coverage by taking  $m = m_{n}$  sufficiently small, but this can still lead to poor UQ by making the resulting credible intervals extremely wide, and therefore uninformative. We thus first study the effect of  $m$  on the quality of estimation, via the posterior convergence rate.
Proposition 3.2 (Pointwise contraction). For  $f_0 \in C(\mathcal{X})$ ,  $x \in \mathcal{X}$  and  $m = m_n \to \infty$ ,
$$
E _ {f _ {0}} Q ^ {*} (f: | f (x) - f _ {0} (x) | > M _ {n} \varepsilon_ {m} | \mathcal {D} _ {n}) \rightarrow 0,
$$
as  $n\to \infty$  , where  $M_{n}\rightarrow \infty$  is any (arbitrarily slowly growing) sequence and
$$
\varepsilon_ {m} ^ {2} = b _ {n} ^ {2} (x) + t _ {n} ^ {2} (x) + \sigma_ {n} ^ {2} (x) + | \langle \mathbf {r} _ {m} (x), \mathbf {f} _ {0} \rangle | ^ {2} + | \langle \mathbf {r} _ {m} (x), \mathbf {k} _ {n} (x) \rangle | ^ {2}.
$$
If  $\langle \mathbf{r}_m(x),\mathbf{f}_0\rangle = o(b_n(x))$  and  $\langle \mathbf{r}_m(x),\mathbf{k}_n(x)\rangle = o(\sigma_n^2 (x))$  as  $n\to \infty$  , then the rate matches that of the true posterior  $\varepsilon_{n}^{2} = b_{n}^{2}(x) + t_{n}^{2}(x) + \sigma_{n}^{2}(x)$
The last result says that the SGPR contracts about the true  $f_0$  at rate  $\varepsilon_m$ , i.e. it puts all but a vanishingly small amount of probability on functions for which  $|f(x) - f_0(x)| \leq M_n \varepsilon_m$ , where  $f_0$  is the true function generating the data. Such results not only quantify the typical distance between a point estimator  $\hat{f}(x)$  (e.g. SGPR mean/median) and the truth ([11], Theorem 8.7), but also the typical
spread of  $Q^{*}$  about the truth. Ideally, most of the  $Q^{*}$ -probability should be concentrated on such sets with radius  $\varepsilon_{m}$  proportional to the optimal (minimax) rate, whereupon they have smallest possible size from an information-theoretic perspective.
Contraction rates in global  $L^2$ -losses have been considered in the variational Bayes literature [24, 2, 41, 27, 26, 22]. Such results crucially rely on the true posterior concentrating exponentially fast about the truth (see e.g. Theorem 5 in [26]), which happens in losses which geometrize the statistical model [15]. This is known not to be true for pointwise or uniform loss [15], and hence these previous proof approaches cannot be applied to the present pointwise setting. We must thus use an alternative approach, exploiting the explicit Gaussian structure of the SGPR.
The SGPR approach in UQ is to consider a smallest pointwise credible set of probability  $1 - \delta$ :
$$
C _ {m} ^ {\delta} = C _ {m} ^ {\delta} \left(\mathcal {D} _ {n}\right) = \left[ \nu_ {m} (x) - z _ {1 - \delta} \sigma_ {m} (x), \nu_ {m} (x) + z _ {1 - \delta} \sigma_ {m} (x) \right], \tag {10}
$$
with  $P(|N(0,1)|\leq z_{1 - \delta}) = 1 - \delta$ . We next establish guarantees on the frequentist coverage of  $C_m^\delta$ , i.e. the probability  $1 - \delta'$  with  $\inf_{f_0\in \mathcal{F}}P_{f_0}(f_0(x)\in C_m^\delta)\geq 1 - \delta'$  over a target function class  $\mathcal{F}$ .
Remark 3.3 (Zero prior mean). Since  $f_0$  is a-priori unknown, such coverage guarantees must hold uniformly over a function class  $\mathcal{F}$  to be meaningful. For symmetric function classes, where  $f \in \mathcal{F}$  implies  $-f \in \mathcal{F}$ , taking a good prior mean  $\nu_0$  for  $f$  will make estimating  $-f$  correspondingly more difficult. Thus without loss of generality we may take zero mean  $\nu_0 = 0$ .
Proposition 3.4 (Pointwise UQ). Suppose that  $\langle \mathbf{r}_m(x),\mathbf{f}_0\rangle = o(b_n(x))$  as  $n\to \infty$ . Then for  $m = m_{n}\leq n$  and any  $\delta \in (0,1)$ , the frequentist coverage of the credible sets satisfies
$$
\liminf_{n\to \infty}P_{f_{0}}(f_{0}(x)\in C_{m}^{\delta})\geq \liminf_{n\to \infty}P_{f_{0}}(f_{0}(x)\in C_{n}^{\delta}).
$$
If in addition  $\| \mathbf{r}_m(x)\|^2 = o(t_n^2 (x))$  and  $\langle \mathbf{r}_m(x),\mathbf{k}_n(x)\rangle = o(\sigma_n^2 (x))$ , then
$$
\lim  _ {n \rightarrow \infty} P _ {f _ {0}} (f _ {0} (x) \in C _ {m} ^ {\delta}) = \lim  _ {n \rightarrow \infty} P _ {f _ {0}} (f _ {0} (x) \in C _ {n} ^ {\delta}).
$$