Title: Pointwise uncertainty quantification for sparse variational Gaussian process regression with a Brownian motion prior
Venue: NeurIPS 2023
Accepted: yes
Keywords: Gaussian process

sparse variational Bayes

uncertainty quantification

theoretical guarantees
Abstract: We study pointwise estimation and uncertainty quantification for a sparse variational Gaussian process method with eigenvector inducing variables. For a rescaled Brownian motion prior, we derive theoretical guarantees and limitations for the frequentist size and coverage of pointwise credible sets. For sufficiently many inducing variables, we precisely characterize the asymptotic frequentist coverage, deducing when credible sets from this variational method are conservative and when overconfident/misleading. We numerically illustrate the applicability of our results and discuss connections with other common Gaussian process priors.

Pointwise uncertainty quantification for sparse variational Gaussian process regression with a Brownian motion prior

Abstract

We study pointwise estimation and uncertainty quantification for a sparse variational Gaussian process method with eigenvector inducing variables. For a rescaled Brownian motion prior, we derive theoretical guarantees and limitations for the frequentist size and coverage of pointwise credible sets. For sufficiently many inducing variables, we precisely characterize the asymptotic frequentist coverage, deducing when credible sets from this variational method are conservative and when overconfident/misleading. We numerically illustrate the applicability of our results and discuss connections with other common Gaussian process priors.

1 Introduction

Consider the standard nonparametric regression model, where we observe n training samples D n = { ( x 1 , y 1 ) , . . . , ( x n , y n ) } arising from the model

where σ 2 > 0 and the design points x i ∈ X ⊂ R d are either fixed or considered as i.i.d. random variables. Our goal is to predict outputs y ∗ based on new input features x ∗ , while accounting for the statistical uncertainty arising from the training data. A widely-used Bayesian approach is to endow f with a Gaussian process (GP) prior [25], which is especially popular due to its ability to provide uncertainty quantification via posterior credible sets.

While explicit expressions for the posterior distribution are available, a well-known drawback is that these require O ( n 3 ) time and O ( n 2 ) memory complexity, making computation infeasible for large data sizes n . To avoid this, there has been extensive research on low-rank GP approximations, where one chooses m ≪ n inducing variables to summarize the posterior, thereby reducing computation to O ( nm 2 ) time and O ( nm ) memory complexity, see the recent review [20].

We consider here the sparse Gaussian process regression (SGPR) approach introduced by Titsias [31], which is widely used in practice (see [1, 9] for implementations) and has been studied in many recent works [13, 21, 5, 6, 38, 28, 32, 22, 23]. While the computational dependence on the number of inducing variables m is relatively well-understood, fewer theoretical results are available on understanding how m affects the quality of statistical inference. In particular, it is crucial to understand how large m needs to be to achieve good statistical uncertainty quantification, ideally close to that of the true computationally expensive posterior. The present work contributes to this research direction by establishing precise theoretical guarantees for pointwise inference using a natural choice of SGPR with eigenvector inducing variables.

A main appeal of Bayesian methods is their ability to perform uncertainty quantification (UQ) via credible sets, often at the 95% level, see Figure 1 for a visualization involving a full GP and two

Kolyan Ray

-0.1

0.0

0.1

0.2

0.3

0.4

0.2

0.4

0.6

0.8

x

Mean

Posterior

SGPR (m = 5)

SGPR (m = 106)

GP

Comparison of the Posteriors

SGPR approximations. We focus here on the canonical problem of pointwise estimation and UQ for f ( x ) at a single point x ∈ X , which corresponds to the practically relevant problem of quantifying the uncertainty of a prediction f ( x ) based on a new input x . We study the frequentist behaviour of the resulting SGPR credible sets, with our main conclusion being that for well-calibrated priors and sufficiently many inducing points m , SGPRs can indeed provide reliable, though conservative, uncertainty quantification.

Concretely, we consider a rescaled Brownian motion prior that models Hölder smooth functions [33, 36, 29] and whose pointwise UQ for the corresponding full posterior was studied in [29]. We extend this analysis to the SGPR setting using the eigendecomposition of the sample covariance matrix as inducing variables, i.e. the optimal rankm posterior approximation. We provide a rigorous theoretical justification for pointwise estimation and UQ using this SGPR, provided m is large enough. Our main theoretical contributions are summarized that in dimension d = 1 , for an α -smooth truth f 0 , γ -smooth prior, α, γ ∈ (0 , 1] , and at least m ≫ n 1 1+2 γ ( 2+ α 1+ α ) inducing variables:

The SGPR recovers the true value f 0 ( x ) at a point x with convergence rate n -min( α,γ ) 2 γ +1 . If α = γ , this gives the minimax optimal rate n -α 2 α +1 .

For truths that are smoother than the prior ( α > γ ) , SGPR credible sets are conservative with their frequentist coverage converging to a value strictly between their nominal level and one, which we characterize.

For truths that are rougher than the prior ( α ≤ γ ) , coverage can be arbitrarily bad.

It is not necessary for the KL divergence between the SGPR and true posterior to vanish as n →∞ to get good (though conservative) coverage.

We provide a more general small-bias condition under which the coverage of SGPR credible sets is at least that of credible sets from the true posterior.

Our proofs exploit specific properties of Brownian motion to obtain explicit expressions, which allow us to obtain the above precise results. Such expressions seem difficult to obtain for other commons GPs (e.g. Matérn, squared exponential), meaning that existing theoretical guarantees for the Matérn SGPR apply to enlarged credible sets [32, 23]. Since our results indicate that pointwise SGPR credible sets are already conservative, such enlargements make the SGPR UQ procedure very

conservative, see Section 4. We provide a complimentary perspective to these works, studying true credible sets from a related GP, rather than modified credible sets from the Matérn.

The Brownian motion prior we consider here shares many similarities with the Matérn process, meaning our main qualitative conclusions also apply there, as we demonstrate in simulations. We thus verify the applicability of our limit theorems in numerical simulations, confirming they provide a good guide to the behaviour of SGPRs for finite sample sizes, including those based on Matérn priors. To the best of our knowledge, our work is the first establishing exact theoretical coverage guarantees for SGPR pointwise credible sets, and can be viewed as a guide of what to expect for the other Gaussian processes, see Section 4.

Related work. Rigorous theoretical guarantees for Bayesian UQ using the full posterior have received much recent attention. In parametric models, the classic Bernstein-von Mises theorem [35] ensures that regular credible sets are asymptotic confidence sets, justifying Bayesian UQ from a frequentist perspective. However, such a result fails to hold in high- and infinite-dimensional settings [10, 16], such as GPs, where the performance is sensitive to the choice of prior and model. Theoretical guarantees for UQ using full GPs have been established mainly involving L 2 -type credible sets (e.g. [17, 7, 30]) which, while mathematically more tractable, are less reflective of actual practice than pointwise credible sets. Results for pointwise Bayesian UQ are more difficult to obtain and are correspondingly much rarer [29, 40, 39], see Section 3. From a technical perspective, we perform a careful and novel analysis of the gap between the pointwise behaviour of the true posterior and its SGPR approximation, after which we can invoke ideas from the proofs for the full posterior [29].

While the frequentist asymptotic properties of VB have increasingly been investigated, this is usually in the context of estimation, convergence rates or approximation quality (e.g. [24, 2, 41, 27, 26], including for SGPRs [5, 6, 22]) rather than UQ. Available results on mean-field VB often show that this factoriazable approach provides overconfident UQ by underestimating the posterior variance [3, 4], though there exist more positive results in certain low-dimensional models [37]. Regarding UQ for SGPRs, Nieman et al. [23] investigate the frequentist coverage of global L 2 -credible sets from certain SGPRs, showing these can have coverage tending to 1, possibly after enlarging them. Vakili et al. [32] similarly show that enlarged pointwise or uniform credible sets can have good coverage in a different bandit-like setting.

Organization. In Section 2 we detail the problem setup, including notation and an overview of (sparse variational) Gaussian process regression. Main results on pointwise convergence and UQ for SGPRs based on a Brownian motion prior are found in Section 3, discussion on the connections with other GPs in Section 4, simulations in Section 5 and discussion in Section 6. In the supplement, we provide the proofs of our results (Section A) and additional simulations (Section B).

2 Problem setup and sparse variational Gaussian process regression

Recall that we observe n training examples D n = { ( x 1 , y 1 ) , . . . , ( x n , y n ) } from model (1), and write y = ( y 1 , . . . , y n ) T and f = ( f ( x 1 ) , . . . , f ( x n )) T . We denote by P f the probability distribution of y from model (1) and by E f the corresponding expectation. For u , v ∈ R n we write ⟨ u , v ⟩ = ∑ n i =1 u i v i for the inner-product on R n and ∥ u ∥ = ⟨ u , u ⟩ 1 / 2 for the usual Euclidean norm. Write C ( X ) for the space of continuous real-valued functions on X . For α ∈ (0 , 1] , we further define the class of Hölder smooth functions on [0 , 1] by C α = C α [0 , 1] = { f : [0 , 1] → R : sup | f ( x ) -f ( y ) | α < ∞} . Throughout the paper we make the following frequentist assumption:

x = y | x -y |

̸

Assumption 2.1. There is a true f 0 ∈ L 2 ( X ) generating the data y ∼ P f 0 according to (1) .

We differentiate f coming from the Bayesian model with the 'true' generative f 0 .

2.1 Sparse variational Gaussian process regression

In model (1), we assign to f ∼ GP ( ν 0 , k ) a Gaussian process (GP) prior with mean function ν 0 : X → R and covariance kernel k : X × X → R . We henceforth take the prior mean ν 0 = 0 for simplicity since this will not alter the spirit of our results, see Remark 3.3 below. The resulting

posterior distribution is again a GP by conjugacy [25], with mean and covariance functions equal to

where k n ( x ) = ( k ( x, x 1 ) , . . . , k ( x, x n )) T and [ K nn ] ij = k ( x i , x j ) . In particular, the posterior variance at a point x ∈ X is σ 2 n ( x ) = k n ( x, x ) . In Section 3.2 below, we will specifically consider the choice X = [0 , 1] and rescaled Brownian motion prior kernel k ( x, x ′ ) = ( n +1 / 2) 1 -2 γ 1+2 γ min( x, x ′ ) for γ > 0 .

We consider here the sparse variational Gaussian process (SGPR) approximation using inducing variables proposed by Titsias [31]. The idea is to summarize the posterior via a relatively small number of inducing variables u = { u 1 , . . . , u m } , m ≪ n , which are linear functionals of the prior process f . By assigning u a multivariate normal distribution, one obtains a low-rank variational family. Titsias [31] explicitly computed the minimizer in Kullback Leibler sense between this family of GPs and the full posterior, which yields the SGPR Q ∗ = GP ( ν m , k m ) with mean and covariance

where K xm = ( cov ( f ( x ) , u 1 ) , . . . , cov ( f ( x ) , u m )) = K T mx , [ K mm ] ij = cov ( u i , u j ) and we write [ K nm ] ij = cov ( f ( x i ) , u j ) . Recalling that cov ( f ( x ) , f ( x ′ )) := k ( x, x ′ ) under the prior, for u j of the form u j = ∑ n i =1 v i j f ( x i ) (see (4) below) we obtain cov ( f ( x ) , u j ) = ∑ n j =1 v i j k ( x, x i ) . The SGPR variance at x ∈ X is σ 2 m ( x ) = k m ( x, x ) . The computational complexity of obtaining ν m and k m is O ( nm 2 ) , which is much smaller than the O ( n 3 ) needed to compute the full posterior for m ≪ n .

2.2 Eigenvector inducing variables

The approximation quality of the SGPR depends on the inducing variables being well chosen. We consider here arguably the conceptually simplest and most canonical sparse approximation based on the eigendecomposition of K nn . Let µ 1 ≥ µ 2 ≥ ·· · ≥ µ n ≥ 0 be the eigenvalues of the positive semi-definite matrix K nn with corresponding orthonormal eigenvectors v 1 , . . . , v n . Writing v j = ( v 1 j , . . . , v n j ) T ∈ R n , we take as inducing variables

so that u j is a linear combination of the inducing points placed at each data point with weights proportional to the entries of the j th -largest eigenvector of K nn . Setting V m := [ v 1 . . . v m ] ∈ R n × m , this choice of inducing points implies

see Section C.1 of [5]. Writing K nn = ∑ n i =1 µ i v i v T i and substituting these expressions into (3), we obtain the SGPR with mean and covariance functions

where η k = 1 σ 2 + µ k . We study inferential properties of the SGPR with mean (5) and covariance (6).

Comparing these expressions with those for the full posterior (2), we see that this SGPR is the optimal rankm approximation in the sense that we have replaced ( K nn + σ 2 I n ) -1 = ∑ n i =1 η i v i v T i in both the mean and covariance (2) with ∑ m i =1 η i v i v T i , corresponding to the m largest eigenvalues of K nn . These inducing variables are an example of interdomain inducing features [19], which can yield sparse representations in the spectral domain and computational benefits [14]. Computing this SGPR

involves numerically computing the first m eigenvalues and corresponding eigenvectors of K nn , which can be done in O ( n 2 m ) time using for instance Lanczos iteration [18].

This choice of SGPR using eigenvector inducing variables has been shown to minimize certain upper bounds for the Kullback-Leibler (KL) divergence between order m SGPRs and the true posterior [5], reflecting that it can provide a good posterior approximation. Indeed, one can show that for m = m n growing fast enough (though sublinearly in n ), this SGPR converges to the posterior in the KL sense as n →∞ [5]. Under weaker growth conditions on m , this SGPR still converges to the true generative f 0 at the minimax optimal rate in the frequentist model [22], even when the SGPR does not converge to the true posterior as n →∞ . Our results include both scenarios and will show that one can still obtain good frequentist UQ performance even for m small enough that the SGPR diverges from the full posterior (see Remark 3.7) unlike in [5].

3 Main results

3.1 General results on pointwise inference

We now present our main results concerning estimation and uncertainty quantification at a point x ∈ X using the SGPR Q ∗ with eigenvector inducing variables. Since both the posterior and SGPR are GPs, their marginal distributions at x satisfy f ( x ) |D n ∼ N ( ν n ( x ) , σ 2 n ( x )) and f ( x ) ∼ N ( ν m ( x ) , σ m ( x ) 2 ) , respectively. It thus suffices to study these quantities under the frequentist assumption that there is a 'true' f 0 generating the data, i.e. Assumption 2.1. We further define the frequentist bias and variance of the (sparse) posterior mean:

where the expectation and variance are taken over y ∼ P f 0 . Wehave the following useful relationships between these quantities for the variational posterior and full posterior, which assumes fixed design, so that all statements are taken conditional on the design points x 1 , . . . , x n ∈ X . For simplicity, we take the noise variance to be σ 2 = 1 in (1) for our theoretical results.

The quantity r m ( x ) measures the rank gap due to ignoring the smallest n -m eigenvalues of K nn . The SGPR approximation increases the posterior variance σ 2 m ( x ) ≥ σ 2 n ( x ) while reducing the frequentist variance of the posterior mean t 2 m ( x ) ≤ t 2 n ( x ) . Thus if one can control the biases in (7), credible sets from the SGPR will have coverage at least as large as the full posterior. One can therefore ensure good coverage by taking m = m n sufficiently small, but this can still lead to poor UQ by making the resulting credible intervals extremely wide, and therefore uninformative. We thus first study the effect of m on the quality of estimation, via the posterior convergence rate.

Proposition 3.2 (Pointwise contraction) . For f 0 ∈ C ( X ) , x ∈ X and m = m n →∞ ,

as n →∞ , where M n →∞ is any (arbitrarily slowly growing) sequence and

If ⟨ r m ( x ) , f 0 ⟩ = o ( b n ( x )) and ⟨ r m ( x ) , k n ( x ) ⟩ = o ( σ 2 n ( x )) as n →∞ , then the rate matches that of the true posterior ε 2 n = b 2 n ( x ) + t 2 n ( x ) + σ 2 n ( x ) .

The last result says that the SGPR contracts about the true f 0 at rate ε m , i.e. it puts all but a vanishingly small amount of probability on functions for which | f ( x ) -f 0 ( x ) | ≤ M n ε m , where f 0 is the true function generating the data. Such results not only quantify the typical distance between a point estimator ˆ f ( x ) (e.g. SGPR mean/median) and the truth ([11], Theorem 8.7), but also the typical

spread of Q ∗ about the truth. Ideally, most of the Q ∗ -probability should be concentrated on such sets with radius ε m proportional to the optimal (minimax) rate, whereupon they have smallest possible size from an information-theoretic perspective.

Contraction rates in global L 2 -losses have been considered in the variational Bayes literature [24, 2, 41, 27, 26, 22]. Such results crucially rely on the true posterior concentrating exponentially fast about the truth (see e.g. Theorem 5 in [26]), which happens in losses which geometrize the statistical model [15]. This is known not to be true for pointwise or uniform loss [15], and hence these previous proof approaches cannot be applied to the present pointwise setting. We must thus use an alternative approach, exploiting the explicit Gaussian structure of the SGPR.

The SGPR approach in UQ is to consider a smallest pointwise credible set of probability 1 -δ :

with P ( | N (0 , 1) | ≤ z 1 -δ ) = 1 -δ . We next establish guarantees on the frequentist coverage of C δ m , i.e. the probability 1 -δ ′ with inf f 0 ∈F P f 0 ( f 0 ( x ) ∈ C δ m ) ≥ 1 -δ ′ over a target function class F .

Remark 3.3 (Zero prior mean) . Since f 0 is a-priori unknown, such coverage guarantees must hold uniformly over a function class F to be meaningful. For symmetric function classes, where f ∈ F implies -f ∈ F , taking a good prior mean ν 0 for f will make estimating -f correspondingly more difficult. Thus without loss of generality we may take zero mean ν 0 = 0 .

Proposition 3.4 (Pointwise UQ) . Suppose that ⟨ r m ( x ) , f 0 ⟩ = o ( b n ( x )) as n → ∞ . Then for m = m n ≤ n and any δ ∈ (0 , 1) , the frequentist coverage of the credible sets satisfies

If in addition ∥ r m ( x ) ∥ 2 = o ( t 2 ( x )) and ⟨ r m ( x ) , k n ( x ) ⟩ = o ( σ 2 ( x )) , then

The first conclusion says that if the rank gap is smaller than the posterior bias (a 'small-bias' condition), SGPR credible sets will have asymptotic coverage at least at the level of the original full posterior, though perhaps larger, including possibly tending to one. The second conclusion gives further conditions on the rank gap under which the asymptotic coverages will be the same. Together with Proposition 3.2, the goal is to obtain a credible set C δ m of smallest possible diameter subject to having sufficient coverage.

3.2 Fixed design with rescaled Brownian motion prior

We now apply our general results to a specific GP and find conditions on the number of inducing variables m needed to get good UQ. Consider the domain X = [0 , 1] with regularly spaced design points x i = i n +1 / 2 for i = 1 , . . . , n . For B a standard Brownian motion, consider as prior f =

√ c n B , where c n = ( n +1 / 2) 1 -2 γ 1+2 γ for γ > 0 . Thus f is a mean-zero GP with covariance kernel k ( x, x ′ ) = c n ( x ∧ x ′ ) , where x ∧ x ′ := min( x, x ′ ) . The scaling factor c n controls the smoothness of the sample paths of the GP and plays the same role as the lengthscale parameter for stationary GPs. The present rescaled Brownian motion is a suitable prior to model Hölder functions of smoothness γ ∈ (0 , 1] (e.g. [33, 17, 29]). In particular, for a true f 0 ∈ C α , α ∈ (0 , 1] , one obtains (full) posterior contraction rate n -α ∧ γ 2 γ +1 in the global L 2 -loss [33]. We sometimes write Q ∗ γ = Q ∗ for the corresponding SGPR to make explicit that the underlying prior is γ -smooth.

Theorem 3.5. Let f 0 ∈ C α [0 , 1] , α ∈ (0 , 1] and x ∈ (0 , 1) . Consider the SGPR Q ∗ γ with rescaled Brownian motion prior of regularity γ ∈ (0 , 1] and m = m n →∞ inducing variables. Then

as n → ∞ , where M n → ∞ is any (arbitrarily slowly growing) sequence. If m n ≫ n 1 1+2 γ ( 2+ α 1+ α ) then the contraction rate of the SGPR matches that of the full posterior n -α ∧ γ 1+2 γ .

Theorem 3.5 shows that for m ≫ n 1 1+2 γ ( 2+ α 1+ α ) inducing points, the SGPR attains the same pointwise contraction rate as the full posterior. If γ = α , we then recover the optimal (minimax) rate of

convergence for a C α -smooth function. For instance if α = γ = 1 , our result indicates that one can do minimax optimal pointwise estimation based on m n ≫ √ n inducing variables, a substantial reduction over the full n observations. Note this convergence guarantee can still hold in cases when the SGPR diverges from the posterior as n →∞ , see Remark 3.7.

The restriction α ∈ (0 , 1] comes from rescaling the underlying Brownian motion [33]. One can extend this to α > 1 by considering a smoother baseline process [33], such as integrated Brownian motion, but the more complex form of the resulting eigenvectors and eigenvalues of K nn make our explicit proof approach difficult. Note also that the prior fixes the value f (0) = 0 . One can avoid this by adding an independent normal random variable to f , but since we consider pointwise inference at a point x > 0 , this will not affect our results and we thus keep f (0) = 0 for simplicity. Nonetheless, the message here is clear: for sufficiently many inducing variables (but polynomially less than n ), an SGPR based on a γ -smooth prior can estimate an α -smooth truth at rate n -α ∧ γ 1+2 γ . We next turn to UQ.

Theorem 3.6. Let f 0 ∈ C α [0 , 1] , α ∈ (0 , 1] and x ∈ (0 , 1) . Consider the SGPR Q ∗ γ with rescaled Brownian motion prior of regularity γ ∈ (0 , 1] and m = m n ≫ n 1 1+2 γ ( 2+ α 1+ α ) inducing variables. For δ ∈ (0 , 1) , let q δ m := P f 0 ( f 0 ∈ C δ m ( x )) denote the frequentist coverage of the 1 -δ credible set C δ m ( x ) given by (10) . Then as n →∞ :

(i) (Undersmoothing case) If α > γ , then q δ m → P ( | N (0 , 1 / 2) | ≤ z 1 -δ ) =: p δ > 1 -δ for all f ∈ C α [0 , 1] , where P ( | N (0 , 1) | ≤ z 1 -δ ) = 1 -δ and N (0 , 1 / 2) is the normal distribution with mean zero and variance 1 / 2 .

(ii) (Correct smoothing case) If α = γ , then for each p ∈ (0 , p δ ] , there exists f ∈ C α [0 , 1] such that q δ m → p .

(iii) (Oversmoothing case) If α < γ , there exists f ∈ C α [0 , 1] such that q δ m → 0 .

Theorem 3.6 provides exact expressions for the asymptotic coverage when the credible interval C δ m has diameter O ( n -α ∧ γ 1+2 γ ) . In the undersmoothing case α > γ , the 1 -δ SGPR credible sets are conservative from a frequentist perspective, i.e. their coverage converges to a value strictly between 1 -δ and 1. For instance if δ = 0 . 05 (0 . 1) , the 95% (90%) credible set will have asymptotic coverage 99.4% (98.0%), indicating one does not want to enlarge such credible sets if using enough inducing variables. A desired asymptotic coverage can also be achieved by targetting the credibility according to the formula in Theorem 3.6(i). Reducing γ towards zero will generally ensure coverage for Hölder functions, but this will also increase the size of the credible intervals to size O ( n -γ 1+2 γ ) , making them less informative. Note that the convergence in (i) is uniform over C α -balls of fixed radius.

In the oversmoothing case α < γ , the posterior variance underestimates the actual uncertainty and so the credible interval is too narrow, giving overconfident (bad) UQ for many Hölder functions. In the correct smoothing case α = γ , where we obtain the minimax optimal contraction rate n -α 1+2 α , coverage falls between these regimes - it does not fully tend to zero, but one can find a function whose coverage is arbitrarily bad, i.e. close to zero.

The best scenario thus occurs when the prior slightly undersmooths the truth, in which case the SGPR credible interval will have slightly conservative coverage but its width O ( n -γ 1+2 γ ) is not too much larger than the minimax optimal size O ( n -α 1+2 α ) . Our results match the corresponding ones for the full computationally expensive posterior and the main messages are the same: undersmoothing leads to conservative coverage while oversmoothing leads to poor coverage [29].

Remark 3.7 (Kullback-Leibler) . If the number of inducing variables grows like n 1 1+2 γ ( 2+ α 1+ α ) ≪ m ≪ n 2 1+2 γ , then the conditions of Theorems 3.5 and 3.6 are met, but the KL-divergence between the SGPR and the true posterior tends to infinity as n →∞ (Lemma A.6 in the supplement). Thus one does not need the SGPR to be an asymptotically exact approximation of the posterior to get similarly good frequentist pointwise estimation and UQ. In particular, one can take m polynomially smaller in n than is required for the KL-divergence to vanish, see [5] for related bounds and discussion.

While [5] consider approximation quality as measured by the KL-divergence between the SGPR and true posterior, we consider here the different question of whether the SGPR behaves well for pointwise inference, even when it is not a close KL-approximation. The data-generating setup in [5] is also not directly comparable to ours. They take expectations over both the data ( x, y ) and prior

on f , so their results are 'averaged' over the prior, which downweights the effect of sets of small prior probability. In contrast, the frequentist guarantees provided here hold assuming there is a true function f 0 which generates the data according to (1) (Assumption 2.1), meaning that one seeks to understand the performance of the method for any fixed f 0 , including worst-case scenarios. Such setups can lead to very different conclusions, see e.g. Chapters 6.2-6.3 in [11].

4 Connections with other Gaussian process priors

The rescaled Brownian motion (rBM) prior we consider here can be thought of as an approximation to the Matérn process of regularity γ ∈ (0 , 1] since they both model γ -smooth functions. The rescaling factor c n = ( n +1 / 2) 1 -2 γ 1+2 γ in the covariance function k ( x, x ′ ) = c n ( x ∧ x ′ ) plays the same role as the lengthscale parameter for the Matérn and calibrates the GP smoothness. We thus expect similar theoretical results and conclusions to also hold for the Matérn process, which seems to be the case in practice, as we verify numerically in Section 5. Indeed, similar UQ properties to the rBM posterior are conjectured to hold for the full Matérn posterior [39], in particular that the coverage of credible sets satisfies similar qualitative conclusions to the three cases in Theorem 3.6.

On a more technical level, the success of GPs in nonparametric estimation is known to depend on their sample smoothness, as measured through their small ball probability [33, 36]. The posteriors based on both GPs converge to a C α -truth at rate n -α ∧ γ 2 γ +1 in L 2 -loss ([33] for rBM, [34] for Matérn), indicating the closeness of their small-ball asymptotics and hence that both GPs distribute prior mass similarly over their supports. This gives a more quantitative notion of the similarities of these GPs.

Theoretical guarantees do exist for the Matérn SGPR. Theorem 3.5 of [32] nicely establishes that in a bandit-like setting and if the true f 0 is in the prior reproducing kernel Hilbert space H (roughly α = γ + 1 / 2 ), then inflating credible sets by a factor depending on the rank gap and an upper bound for ∥ f 0 ∥ H guarantees coverage. Our parallel results suggest that this inflation is not necessary, since SGPR credible sets in this case will already be conservative. However, since our proofs rely on explicit expressions available for rBM but not for the Matérn, it seems difficult to extend our approach to the different setting considered in [32] (and vice-versa); other techniques are required. Our results can thus be viewed as a guide as to what to expect when using a Matérn SGPR and provide a different perspective reinforcing the main messages of [32], namely that SGPRs can provide reliable, if conservative, UQ.

On the other hand, the squared exponential kernel seems to behave qualitatively differently in numerical simulations, giving different coverage behaviour in the cases considered in Theorem 3.6, see Section 5. This is due to the difference in the smoothness of the sample paths, with squared exponential prior draws being far smoother, in particular analytic. This leads to somewhat different UQ behaviour for the true posterior [12]. Rescaled Brownian motion is a less good approximation for the squared exponential kernel than for the Matérn, and one must thus be cautious about transferring the messages derived here to the squared exponential.

5 Numerical simulations

We next investigate empirically the applicability of our theoretical results in finite sample sizes and whether the conclusions extend to related settings, such as different designs and other GP priors. We consider various nonparametric regression settings of the form (1) with C α -smooth truths and γ -smooth priors. We compute both the full posterior (columns marked GP) and SGPR (marked SGPR) with m ≪ n eigenvector inducing variables for f ( x 0 ) at a fixed point x 0 ∈ X , and report the root-mean square error (RMSE), negative log predictive density (NLPD), and the length and coverage of the 90%-credible intervals of the form (10) (i.e. δ = 0 . 1 ), see Section B in the supplement for definitions. The true noise variance in (1) is taken to be σ 2 = 1 , but is considered unknown and is estimated by maximizing the log-marginal likelihood as usual. We ran all simulations 500 times and report average values and their standard deviations when relevant. Simulations were run on a 2 GHz Quad-Core Intel Core i5 processor on a 2020 Macbook Pro with 16GB of RAM.

Settings of the theoretical results. We consider the setting of our theoretical results in Section 3 with X = [0 , 1] and x i = i/ ( n + 1 / 2) for i = 1 , . . . , n . To generate the data, we take f 0 ( x ) = | x -0 . 5 | α , which is exactly α -Hölder at x 0 = 0 . 5 , and investigate pointwise inference at x 0 = 0 . 5 for

different choices of α . As Gaussian priors, we consider (i) rescaled Brownian motion of regularity γ described in Section 3, (ii) the Matérn kernel with parameter γ and (iii) the square exponential kernel k ( x, x ′ ) = exp {-1 2 ℓ 2 n ( x -x ′ ) 2 } , where the lengthscale ℓ n = n -1 1+2 γ is chosen to be appropriate ∗ 1 2+ α

for estimating γ -Hölder smooth functions [33]. We take m := n 1+2 γ 1+ α inducing variables for the SGPR. Results are given in Table 1: Fixed Design, n = 1000 . We also consider the related case of uniform random design where x i ∼ U (0 , 1) (Table 1: Random Design, n = 500 ).

Theorem 3.6 predicts that for 90% credible sets ( δ = 0 . 1 ), m ≫ m ∗ and α > γ , the coverage should converge to p 0 . 1 = P ( | N (0 , 1 / 2) | < z 0 . 9 ) = 0.98. We see that for n = 1000 , the observed coverages for both the full posterior and SGPR based on the Brownian motion prior are very close to this predicted value, and are indeed conservative. In the oversmoothing case α < γ , coverage is poor and far below the nominal level, while in the correct smoothing case ( α = γ ), coverage is moderately below the nominal level. We see that the asymptotic theory is applicable for reasonable samples sizes.

The Matérn process behaves qualitatively similarly to Brownian motion in these three cases, though the (still conservative) limiting coverage in the undersmoothing case is predicted to be slightly different [39], as reflected in Table 1. On the other hand, the square exponential behaves differently in all cases and generally has wider sets with this lengthscale choice. The wider credible intervals and larger RMSE suggest greater bias of the posterior mean. While rescaled Brownian motion seems a reasonable guide for the Matérn, it does not seem so for the squared exponential, see Section 4.

For all three GP priors, the reported metrics are practically indistinguishable between the SGPR and the full GP, confirming that m ≫ n 1 1+2 γ 2+ α 1+ α inducing variables seems sufficient to obtain virtually the same pointwise credible intervals as the full posterior. However, since m ∗ ≪ n 2 1+2 γ for any α > 0 , the KL-divergence between the SGPR and the full posterior grows to infinity as n →∞ in the Brownian motion case (Lemma A.6), and hence the SGPR will differ from the posteriors in some ways. This reflects that one does not necessarily need to fully approximate the posterior in order to match its pointwise inference properties. Note that for fixed design, the length of the credible sets is not random since it depends only on the features x 1 , . . . , x n , and so no standard errors are reported.

Other settings. We turn to simulation settings not covered by our theory. Given the above discussion, we consider only the Matérn process in dimension d = 10 and the corresponding SGPR with m = m ∗ d := n d d +2 γ . As features, consider random design with (i) x i ∼ U ([ -1 / 2 , 1 / 2] d ) and (ii) X i ∼ N d (0 , Σ ρ ) for [Σ ρ ] ij = 1 for i = j and [Σ ρ ] ij = ρ otherwise. To better reflect correlation in real-world data, we also consider a semi-synthetic dataset with real features but simulated responses.

As features, we took the first d = 10 columns and randomly sampled n = 2000 rows from a Korean meteorological dataset [8]. In all cases, we consider estimating the α -Hölder function f 0 ( x ) = ∥ x -x 0 ∥ α at x 0 = 0 , and the resulting pointwise 90%-credible sets at x 0 coming from the Matérn SGPR. Results are presented in Table 2, where we see that for m = m ∗ d the performance of the pointwise credible sets from the SGPR matches that of the full posterior in each design. Due to the higher dimension, credible intervals are less confident (wider) and have larger coverage.

Multidimensional Random Design, n = 1000

Table 2: Comparison of SGPR and full posterior (GP) with Matérn prior for 90% pointwise credible intervals for different values of ( α, γ ) . The SGPR uses 534 and 1002 inducing variables in the first four rows and fifth row, respectively.

6 Discussion

In this work, we established the frequentist asymptotic behaviour of pointwise credible intervals coming from sparse variational Gaussian process regression (SGPR) with eigenvector inducing variables based on a rescaled Brownian motion prior. We showed that if the prior undersmooths the truth and with enough inducing variables, SGPR credible sets can provide reliable, though conservative, uncertainty quantification, with the coverage converging to a value strictly between the nominal level and one. If the prior oversmooths the truth, UQ can be poor. We further showed that it is not necessary for the SGPR to converge to the true posterior in KL-divergence to have similar behaviour of the credible sets. Our results suggest that properly calibrated SGPRs can perform reliable UQ. We verified these conclusions in simulations and discussed connections with other GPs.

Despite the widespread use of SGPR, there are still relatively few theoretical guarantees for their use, particularly for UQ. Our work provides some new results in this direction, but most of this research area is still wide open. A key step would be to prove similar results for the most commonly used GP priors, notably the Matérn and squared exponential. Similarly, one would like to extend these results to other choices of inducing variables, for instance the eigenfunctions of the kernel operator. While our results give some intuition of what one can expect, they rely on a careful analysis of Brownian motion with fixed design, and new ideas and techniques will be needed for these other settings.

It is also unclear what minimal number of inducing variables is needed to get good pointwise UQ. When α = γ , the threshold for minimax convergence rates for estimation in L 2 is n 1 1+2 α [22], which is smaller than our bound n 1 1+2 α ( 2+ α 1+ α ) , see Section B in the supplement for some related numerical simulations. Our results are also non-adaptive since they assume fixed prior smoothness γ , whereas one often selects γ in a data-driven way, for instance by maximizing the marginal likelihood or evidence lower bound (ELBO). However, adaptive UQ is a subtle topic which typically requires further assumptions (e.g. self-similarity) on the true generative function to even be possible [30], and extending our results to such settings will require significant technical and conceptual work.

Acknowledgements. We are grateful to four anonymous reviewers for helpful comments that improved the manuscript. The authors would like to thank the Imperial College London-CNRS PhD Joint Programme for funding which supported Luke Travis in his studies.

References

M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mane, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viegas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems, 2016.

P. Alquier and J. Ridgway. Concentration of tempered posteriors and of their variational approximations. Ann. Statist. , 48(3):1475-1497, 2020.

C. M. Bishop. Pattern recognition and machine learning . Information Science and Statistics. Springer, New York, 2006.

D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: a review for statisticians. J. Amer. Statist. Assoc. , 112(518):859-877, 2017.

D. Burt, C. E. Rasmussen, and M. Van Der Wilk. Rates of convergence for sparse variational Gaussian process regression. In Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 862-871. PMLR, 09-15 Jun 2019.

D. R. Burt, C. E. Rasmussen, and M. van der Wilk. Convergence of sparse variational inference in Gaussian processes regression. J. Mach. Learn. Res. , 21(131):1-63, 2020.

I. Castillo and R. Nickl. Nonparametric Bernstein-von Mises theorems in Gaussian white noise. Ann. Statist. , 41(4):1999-2028, 2013.

D. Cho, C. Yoo, J. Im, and D.-H. Cha. Comparative assessment of various machine learningbased bias correction methods for numerical weather prediction model forecasts of extreme air temperatures in urban areas. Earth and Space Science , 7, April 2020. doi: 10.1029/ 2019EA000740.

A. G. de G. Matthews, M. van der Wilk, T. Nickson, K. Fujii, A. Boukouvalas, P. León-Villagrá, Z. Ghahramani, and J. Hensman. Gpflow: A Gaussian process library using TensorFlow. J. Mach. Learn. Res. , 18(40):1-6, 2017.

D. Freedman. On the Bernstein-von Mises theorem with infinite-dimensional parameters. Ann. Statist. , 27(4):1119-1140, 1999.

S. Ghosal and A. van der Vaart. Fundamentals of nonparametric Bayesian inference , volume 44 of Cambridge Series in Statistical and Probabilistic Mathematics . Cambridge University Press, Cambridge, 2017.

A. Hadji and B. Szabó. Can we trust Bayesian uncertainty quantification from Gaussian process priors with squared exponential covariance kernel? SIAM/ASA J. Uncertain. Quantif. , 9(1): 185-230, 2021.

J. Hensman, N. Fusi, and N. D. Lawrence. Gaussian processes for big data. In Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence, UAI 2013, Bellevue, WA, USA, August 11-15, 2013 . AUAI Press, 2013.

J. Hensman, N. Durrande, and A. Solin. Variational Fourier features for Gaussian processes. J. Mach. Learn. Res. , 18(151):1-52, 2018.

M. Hoffmann, J. Rousseau, and J. Schmidt-Hieber. On adaptive posterior concentration rates. Ann. Statist. , 43(5):2259-2295, 2015.

I. M. Johnstone. High dimensional Bernstein-von Mises: simple examples. In Borrowing strength: theory powering applications-a Festschrift for Lawrence D. Brown , volume 6 of Inst. Math. Stat. (IMS) Collect. , pages 87-98. Inst. Math. Statist., Beachwood, OH, 2010.

B. T. Knapik, A. W. van der Vaart, and J. H. van Zanten. Bayesian inverse problems with Gaussian priors. Ann. Statist. , 39(5):2626-2657, 2011.

C. Lanczos. An iteration method for the solution of the eigenvalue problem of linear differential and integral operators. J. Research Nat. Bur. Standards , 45:255-282, 1950.

M. Lázaro-Gredilla and A. Figueiras-Vidal. Inter-domain Gaussian processes for sparse inference using inducing features. In Advances in Neural Information Processing Systems , volume 22. Curran Associates, Inc., 2009.

H. Liu, Y.-S. Ong, X. Shen, and J. Cai. When Gaussian process meets big data: a review of scalable GPs. IEEE Trans. Neural Netw. Learn. Syst. , 31(11):4405-4423, 2020.

A. G. d. G. Matthews, J. Hensman, R. Turner, and Z. Ghahramani. On sparse variational methods and the Kullback-Leibler divergence between stochastic processes. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics , volume 51 of Proceedings of Machine Learning Research , pages 231-239, Cadiz, Spain, 09-11 May 2016. PMLR.

D. Nieman, B. Szabo, and H. van Zanten. Contraction rates for sparse variational approximations in Gaussian process regression. J. Mach. Learn. Res. , 23(205):1-26, 2022.

D. Nieman, B. Szabo, and H. van Zanten. Uncertainty quantification for sparse spectral variational approximations in Gaussian process regression. Electron. J. Stat. , 17(2):2250 - 2288, 2023.

D. Pati, A. Bhattacharya, and Y. Yang. On statistical optimality of variational Bayes. In Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics , volume 84 of Proceedings of Machine Learning Research , pages 1579-1588. PMLR, 09-11 Apr 2018.

C. E. Rasmussen and C. K. I. Williams. Gaussian processes for machine learning . Adaptive Computation and Machine Learning. MIT Press, Cambridge, MA, 2006.

K. Ray and B. Szabó. Variational Bayes for high-dimensional linear regression with sparse priors. J. Amer. Statist. Assoc. , 117(539):1270-1281, 2022.

K. Ray, B. Szabo, and G. Clara. Spike and slab variational Bayes for high dimensional logistic regression. In Advances in Neural Information Processing Systems , volume 33, pages 14423-14434. Curran Associates, Inc., 2020.

S. Rossi, M. Heinonen, E. Bonilla, Z. Shen, and M. Filippone. Sparse Gaussian processes revisited: Bayesian approaches to inducing-variable approximations. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , volume 130 of Proceedings of Machine Learning Research , pages 1837-1845. PMLR, 13-15 Apr 2021.

S. Sniekers and A. van der Vaart. Credible sets in the fixed design model with Brownian motion prior. J. Statist. Plann. Inference , 166:78-86, 2015.

B. Szabó, A. W. van der Vaart, and J. H. van Zanten. Frequentist coverage of adaptive nonparametric Bayesian credible sets. Ann. Statist. , 43(4):1391-1428, 2015.

M. Titsias. Variational learning of inducing variables in sparse Gaussian processes. In Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics , volume 5 of Proceedings of Machine Learning Research , pages 567-574. PMLR, 16-18 Apr 2009.

S. Vakili, J. Scarlett, D.-S. Shiu, and A. Bernacchia. Improved convergence rates for sparse approximation methods in kernel-based learning. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 21960-21983. PMLR, 17-23 Jul 2022.

A. van der Vaart and H. van Zanten. Bayesian inference with rescaled Gaussian process priors. Electron. J. Stat. , 1:433-448, 2007.

A. van der Vaart and H. van Zanten. Information rates of nonparametric Gaussian process methods. J. Mach. Learn. Res. , 12(60):2095-2119, 2011.

A. W. van der Vaart. Asymptotic statistics , volume 3 of Cambridge Series in Statistical and Probabilistic Mathematics . Cambridge University Press, Cambridge, 1998.

A. W. van der Vaart and J. H. van Zanten. Rates of contraction of posterior distributions based on Gaussian process priors. Ann. Statist. , 36(3):1435-1463, 2008. ISSN 0090-5364.

Y. Wang and D. M. Blei. Frequentist consistency of variational Bayes. J. Amer. Statist. Assoc. , 114(527):1147-1161, 2019.

V. Wild, M. Kanagawa, and D. Sejdinovic. Connections and equivalences between the nyström method and sparse variational Gaussian processes. art. arXiv:2106.01121, June 2021.

Y. Yang, A. Bhattacharya, and D. Pati. Frequentist coverage and sup-norm convergence rate in Gaussian process regression. art. 1708.04753, 2017.

W. W. Yoo and S. Ghosal. Supremum norm posterior contraction and credible sets for nonparametric multivariate regression. Ann. Statist. , 44(3):1069-1102, 2016.

F. Zhang and C. Gao. Convergence rates of variational posterior distributions. Ann. Statist. , 48 (4):2180-2207, 2020.

Supplementary material to 'Pointwise uncertainty quantification for sparse variational Gaussian process regression with a Brownian motion prior'

A Proofs

Throughout this section, for sequences a n and b n , we will write a n ≲ b n if there exists C > 0 such that a n ≤ Cb n for n large enough, and we will write a n ≍ b n if both a n ≲ b n and b n ≲ a n . Recall that for simplicity, we take the noise variance σ 2 = 1 in the regression model (1).

A.1 General results on pointwise inference

Proof of Lemma 3.1. Using the definitions (2) and (3) of the posterior and SGPR means, we have ν m ( x ) = ν n ( x ) -⟨ r m ( x ) , y ⟩ , from which (7) follows on taking the E f 0 -expectation. Next, recall that ν n ( x ) = k n ( x ) T [ ∑ n j =1 η j v j v T j ] y , and thus

from which (8) follows. Finally,

which gives (9).

Proof of Proposition 3.2. We will prove this for the full posterior, with the expressions for the SGPR following the same. Denoting by E Π | Y the expectation with respect to the posterior, Markov's inequality yields,

Moreover,

Now,

Proof of Proposition 3.4. We have,

for V m ∼ N ( b m ( x ) σ m ( x ) , t 2 m ( x ) σ 2 m ( x ) ) . Now, if ⟨ r m ( x ) , f 0 ⟩ = o ( b n ( x )) , we have b m ( x ) = b n ( x )(1 + o (1)) and further, by expressions (8) and (9) we have t 2 m ( x ) σ 2 ( x ) ≤ t 2 n ( x ) σ 2 ( x ) , and thus, m n

If in addition, we have ∥ r m ( x ) ∥ 2 = o ( t 2 n ( x )) and ⟨ r m ( x ) , k n ( x ) ⟩ = o ( σ 2 n ( x )) then by expressions (8) and (9) we have t 2 m ( x ) = t 2 n ( x )(1 + o (1)) and σ 2 m ( x ) = σ 2 n ( x )(1 + o (1)) and thus,

A.2 Rescaled Brownian motion prior

Recall that we take regularly spaced design points x i = i n +1 / 2 , i = 1 , . . . , n , and that we consider the rescaled Brownian motion prior, which is a mean-zero Gaussian process with covariance kernel k ( x, x ′ ) = c n ( x ∧ y ) = c n min( x, y ) with c n = ( n +1 / 2) 1 -2 γ 1+2 γ for γ > 0 . For notational convenience, we write N = ( n +1 / 2) /c n = ( n +1 / 2) 1 -1 -2 γ 1+2 γ . The kernel matrix K nn evaluated at the sample points then equals

For this specific choice of prior and design, we can compute fairly explicit expressions for the eigenvalues and eigenvectors of K nn , which will in turn allow us to obtain the precise pointwise asymptotics of the VB method needed for our results.

Lemma A.1 (Eigenvalues and eigenvectors) . The eigenvalues ( µ j ) n j =1 of the covariance matrix K nn are given by

for j = 1 , . . . , n. Moreover, their size satisfies

The corresponding orthonormal eigenvectors v 1 , . . . , v n , where v j = ( v 1 j , . . . , v n j ) T , are given by

Proof. Let v be an eigenvector of K nn with corresponding eigenvalue µ , so that K nn v = µv . Applying elementary row operations to this equation, we arrive at the following equivalent equations:

̸

From this, one can see that to obtain a non-trivial eigenvector v = 0 , we require v 1 = 0 . We thus consider the unnormalized eigenvector w = ( w 1 , . . . , w n ) T ∈ R n , for which we set w 1 = 1 for simplicity. Further define w 0 = 0 and w -1 = -1 , so that the first ( n -1) equations in the last display are equivalent (after rearranging) to the recurrence relation

̸

while the last equation gives the boundary condition

The linear recurrence relation (11) has solution

for with x = 1 -1 2 µN and y = 1 2 √ 4 µN -1 µ 2 N 2 (computation of the eigenvalues below reveals that 1 µ 2 N 2 -4 µN < 0 for all eigenvalues µ .) Note that (13) provides the solution to (11) for any arbitrary value of µ , but that (12) will later be needed to ensure that µ is an eigenvalue of K nn . Since x 2 + y 2 = 1 , we can conveniently rewrite this in complex exponential form as

and ψ is an alternative parametrization of µ . This form makes it easy to compute expressions for ϕ j ± , and hence evaluate (13). Indeed, we have

giving the form of the unnormalized eigenvector w = (1 , w 2 , . . . , w n ) T corresponding to an eigenvalue µ via (14). To normalize the eigenvector, using that cos(2 jψ ) = 1 -2 sin 2 ( jψ ) ,

̸

where in the last line we have used the Lagrange trigonometric identity that ∑ n j =0 cos(2 jψ ) = 1 2 + sin(2 n +1) ψ ) 2 sin ψ for any ψ = 0( mod π ) (else it equals n +1 ). On computing the values ψ corresponding to the eigenvalues in (17) below, one has sin((2 n +1) ψ ) = 2 sin(( n +1 / 2) ψ ) cos(( n +1 / 2) ψ ) = 0 , and hence ∥ w ∥ 2 2 = 2 n +1 4 sin 2 ψ . Thus the normalized eigenvector with v = w/ ∥ w ∥ 2 has the required coordinates,

Turning to the computation of the eigenvalues, the boundary condition (12) defines whether µ is an eigenvalue. Substituting the solution (15) into (12), any µ that satisfies

or equivalently

̸

is an eigenvalue of K nn . Indeed, if one approaches the problem in the traditional way, by computing the determinant of ( K nn -µ I n ) (see Lemma A.2 below), then one has

̸

̸

exactly as above. Recalling that ψ and µ are related by cos ψ = 1 -1 2 µN , this last expression is equivalent to

̸

where we have used the identity 2 cos A sin B = sin( A + B ) -sin( A -B ) twice. By (14), the eigenvalues µ are only defined by cos ψ , and hence we may restrict ψ to [ -π, π ) and look for solutions within this range. For ψ in this range, the condition sin ψ = 0 then implies sin( ψ/ 2) = 0 , and hence we need only look for solutions to the equation cos(( n +1 / 2) ψ ) = 0 . For ψ ∈ [ -π, π ) , these are given by

̸

̸

By symmetry, cos ψ j = cos ψ 1 -j for j = 1 , . . . , n and hence we may discard the roots j = -n +1 , . . . , 0 . Moreover, sin ψ -n = 0 violating the restriction sin ψ = 0 . This leaves us with the n eigenvalues

as desired.

It remains only to quantify the size of the eigenvalues µ j . Expanding cos x = 1 -x 2 / 2! + x 4 / 4! + o ( x 4 ) , we have x 2 -x 4 / 12 ≤ 2(1 -cos x ) ≤ x 2 for all x , and thus

Since ψ j = j -1 / 2 n +1 / 2 π, we have ψ 2 j / 12 < 5 / 6 (on bounding π 2 by 10 and the fraction by 1) and so 1 Nψ 2 j (1 -ψ 2 j / 12) < 6 Nψ 2 j . Hence we have 1 Nψ 2 j ≤ µ j ≤ 6 Nψ 2 j and µ j ≍ 1 Nψ 2 j .

Let | A | denote the determinant of a matrix A .

Lemma A.2. The solutions to the characteristic equation of K nn , | K nn -µ I n | = 0 , are given by the solutions to

where cos ψ = 1 -1 2 Nµ .

Proof. Using row operations, the desired determinant equals

which can be further be reduced to

the determinant of a symmetric tridiagonal n × n matrix. Using the Laplace expansion of the determinant in terms of the determinants of its ( n -1) × ( n -1) submatrices, we obtain the recurrence relation d n = ( 2 N -µ ) d n -1 -µ 2 d n -2 . The solution to this recurrence is given by

where ϕ ± satisfy ϕ 2 -( 2 N -µ ) ϕ + µ 2 = 0 . That is,

Similar to when we solved the recurrence in the proof of Lemma A.1, we end up with ϕ ± = µe ± iψ ′ , where cos( ψ ′ ) = 1 2 Nµ -1 . We use the notation ψ ′ to emphasise that these angles are different to the ψ ′ s found in the proof of Lemma A.1; indeed, it is easy to see that ψ ′ = ψ + π , so that sin( nψ ′ ) = ( -1) n sin( nψ ) . This yields

and the result is shown by seeing

Given the explicit expressions derived for the eigenvalues and eigenvectors of the covariance matrix K nn , we derive bounds for the size of the remainder terms presented in Lemma 3.1.

Lemma A.3. Suppose that f 0 ∈ C α [0 , 1] , α ∈ (0 , 1] , and consider the rescaled Brownian motion prior of regularity γ ∈ (0 , 1] . The corresponding SGPR Q ∗ based on the first m = m n → ∞ eigenvector inducing variables satisfies, as n →∞ ,

Proof. First, suppose that x = x i is a design point. Starting with the bias, we have,

In this case k n ( x i ) T v j = µ j v i j and the next step is to bound v T j f 0 for each j . Using the explicit form for v j given in Lemma A.1,

Now, writing π j := ( j -1 / 2) π , we have sin( x l π j ) = 1 2 i ( e ix l π j -e -ix l π j ) and thus

for f 0 ∈ C α ([0 , 1]) , where the first inequality follows from Lemma A.4 below. We have thus shown that | v T j f 0 | ≲ √ n ( j -1 / 2) -α . Using this, that µ j ≍ ( n +1 / 2) c n 1 ( j -1 / 2) 2 by Lemma A.1 and that η j ≲ 1 ,

as required. Next, we have,

Using again that x = x i and so k n ( x i ) T v j = µ j v i j , we have,

as required. Finally,

and again at a design point x = x i ,

This establishes the three relations at the design points x = x i .

Turning to the general case x ∈ [0 , 1] , let i be such that x i < x < x i +1 . Recalling that the covariance function equals k ( x, x ′ ) = c n ( x ∧ x ′ ) , we obtain

where 0 < x -x i < 1 n +1 / 2 . Consider first the case of the SGPR variance σ 2 m ( x ) . The last display implies that ( k n ( x ) -k n ( x i )) T v j = c n ( x -x i ) ∑ n l = i +1 v l j , and hence the quadratic term in (21) can be rewritten as

The first term is exactly the quadratic term evaluated at the design point x i , which we showed above has size O ( nc 2 n m -3 ) . Using Lemmas A.1 and A.4,

By Lemma A.1, v T j k n ( x i ) = µ j v i j = O ( c n √ nj -2 ) . Since also η j ≲ 1 , the second term in (22) is bounded by a multiple of

The third term in (22) can similarly be shown to be O ( c 2 n n -1 m -1 ) = O ( nc 2 n m -3 ) . We have thus shown that (22) is of size O ( nc 2 n m -3 ) exactly as in the case where x was a design point, which implies the desired bound for the SGPR variance σ 2 m ( x ) .

The case t 2 m ( x ) follows in a similar fashion, simply replacing η j by η 2 j , while the bias b m ( x ) uses additionally the bound | v T j f 0 | ≲ √ n ( j -1 / 2) -α derived above. In conclusion, the three quantities b m ( x ) , t 2 m ( x ) and σ 2 m ( x ) satisfy the same bounds as when x is a design point, which completes the proof.

The next standard bound controls the error when approximating a C α function by its Riemann sum. Lemma A.4. If f ∈ C α [0 , 1] with α ∈ (0 , 1] , then

with x i = i n +1 / 2 for i = 1 , . . . , n.

The following lemma shows that for m large enough, the bias, frequentist variance of the SGPR mean and SGPR variance can each be related to those of the full posterior derived in [29].

Lemma A.5. If m ≫ n 1 1+2 γ ( 2+ α 1+ α ) , then the SGPR satisfies

Proof. From [29], we have the following expressions for the full posterior:

Thus, if we can show that the O terms given in equations (18)-(20) are smaller than their respective terms above, we have the result. By Lemma A.3, we have b m ( x ) = b n ( x ) + O ( nc n m -(1+ α ) ) , and nc n m -(1+ α ) = n 2 1+2 γ m -(1+ α ) ≲ n -α 1+2 γ if and only if n 1 1+2 γ ( 2+ α 1+ α ) ≪ m . Thus, if m ≫ n 1 1+2 γ ( 2+ α 1+ α ) then nc n m -(1+ α ) = o ( b n ( x )) and b m ( x ) = b n ( x )(1 + o (1)) . Similarly,

Thus, t 2 m ( x ) = t 2 n ( x )(1 + o (1)) and σ 2 m ( x ) = σ 2 n ( x )(1 + o (1)) if m ≫ n 1 1+2 γ ( 2+ α 1+ α ) (in fact these are implied even at the smaller bound n 1 1+2 γ ).

Proof of Theorem 3.5. The proof follows from Proposition 3.2 combined with the two groups of expressions (7)-(9) and (18)-(20).

Proof of Theorem 3.6. The coverage equals

Lemma A.6. The Kullback-Leibler divergence between the SGPR Q ∗ and the full posterior satisfies

Suppose further that f 0 ∈ C α , α ∈ (0 , 1] , and consider the fixed design setting with the rescaled Brown motion prior of regularity γ ∈ (0 , 1] . If m ≫ n 1 1+2 γ , then

In particular, if n 1 1+2 γ ≪ m ≪ n 2 1+2 γ then E f 0 KL ( Q ∗ || Π( ·| Y )) →∞ .

Proof. We have from [31]

where K n = K nn + σ 2 I , Q n = Q nn + σ 2 I , K nn = ∑ n j =1 µ j v j v T j and Q nn = ∑ m j =1 µ j v j v T j . We thus have

| Q n | | K n | = ∏ n j = m +1 σ 2 σ 2 + µ j , and tr ( K n -Q n ) = ∑ n j = m +1 µ j , giving the results. We now consider the three terms given in the first part of the lemma. Defining M := ∑ n j = m +1 µ j µ j + σ 2 v j v T j , we have

Recalling from the proof of Lemma A.3 that | v T j f 0 | ≲ nj -α , we can control

Next, we have

E f 0 [ ε T M ε ] = E f 0 tr ( ε T M ε ) = E f 0 tr ( M εε T ) = tr ( ME f 0 [ εε T ]) = tr ( M ) ≍ n 2 1+2 γ m -1 . For the second term in the first part of the lemma,

Hence, adding these expressions to the third term in the first part of the lemma yields

The last term is of strictly smaller order than the first two if m ≫ n 1 1+2 γ .

B Additional numerical simulations

We provide here additional simulations to those in Section 5, investigating further the minimal number of inducing variables needed for good pointwise UQ and the sensitivity of the results to misspecification of the noise distribution. To illustrate the pointwise approach to UQ using full posterior inference and SGPRs, one can refer to Figure 1 in the paper.

Threshold for the number of inducing variables. We showed in Sections 3 and 5 that for m ≫ n 1 1+2 γ ( 2+ α 1+ α ) inducing variables, the SGPR behaves similarly to the true posterior for pointwise inference. When α = γ , the threshold for minimax convergence rates for estimation in L 2 is the smaller bound m ≫ n 1 1+2 α [22]. We next investigate how the SGPR behaves near this threshold.

Return to the fixed ( X i = i/ ( n +1 / 2) ) and uniform random ( X i ∼ iid U [0 , 1] ) design settings on [0 , 1] and consider the same Gaussian processes as before: rescaled Brownian motion, Matérn and rescaled square exponential. We consider three different values of m given by m ∗ -= n 1 1+2 γ / log n , m ∗ + = n 1 1+2 γ · log n , so that these values of m are just below and above the threshold n 1 1+2 γ , respectively, and the full posterior case m = n . Weconsider the 1 -Hölder function f 0 ( x ) = | x -1 / 2 | and 2 -Hölder function f 0 ( x ) = sign ( x -1 / 2) | x -1 / 2 | 2 .

Table 3: Comparison of SGPR and full posterior for 90% pointwise credible intervals for different values of ( n, α, γ ) in dimension d = 1 . The column headers describe how many inducing variables were used, with m ∗ -:= n 1 1+2 γ / log( n ) and m ∗ + := n 1 1+2 γ · log( n ) and the full posterior represented by n . The hypothesised bound n 1 1+2 γ equals 32, 6, 22 and 5 in the four different regimes (top-tobottom).

The results of our experiments for different values of ( n, α, γ ) are given in Table 3; we note here that we exclude estimates of the standard error in this Table due to space constraints, but they are comparable to those presented in Table 1 in the main text. In all cases, one can see that the coverage, length and RMSE are practically indistinguishable between the cases m = m ∗ + and the full posterior

1

m = n , suggesting that the choice of m ≫ n 1+2 γ inducing variables is sufficient to achieve the same performance as the full posterior for pointwise inference. Furthermore, for fixed design, we again see the Brownian motion with both choices m = m ∗ + and m = n achieves the expected coverage of 0.98 predicted by our theory.

Conversely, for m = m ∗ -inducing variables, the coverage and credible interval length was larger in almost all cases, sometimes significantly so. This suggests that below this threshold, the SGPR is no longer a good approximation for the true posterior, matching the findings in [22, 23] for L 2 -type inference. On the other hand, the RMSE is usually reduced, indicating that additional smoothing helps the sparse posterior mean for estimation, which is not surprising since in our simulation choices

the prior undersmooths the truth. The main message is that taking fewer inducing points than this threshold leads to a significant increase in the credible interval width, making the UQ less informative and overly conservative. It is interesting that this threshold shows so clearly, since m ∗ -and m ∗ + differ only by a subpolynomial factor.

The same story holds for random design, with some additional fluctuations due to this additional source of randomness. We recall that for fixed design, the credible intervals have deterministic length, whereas for random design these are also now random.

It would be interesting to extend our theoretical results to the threshold n 1 1+2 γ as suggested by these simulations. However, the mismatch between pointwise loss and the L 2 -distance induced by the likelihood makes using standard tools from Bayesian nonparametrics difficult, see [15] for further discussion.

Multidimensional setting. Consider again d = 10 dimensional covariates with the Matérn kernel of smoothness 1/2, and the corresponding threshold n d d +2 γ needed for the minimax contraction rate in L 2 [22]. We take uniform random design X i ∼ iid U ([ -1 / 2 , 1 / 2] d ) , true function f 0 ( x ) = ∥ x ∥ α for α = 0 . 9 and investigate the pointwise 90% -credible intervals at the point x 0 = (0 , . . . , 0) . We now take three values of m given by m ∗ d -= n d d +2 γ / log n , m ∗ d + = n d d +2 γ · log n and m = n . The results are presented in Table 4. Indeed, one sees that the variational posterior performs identically to the full posterior once at least m ∗ d + = n d/ ( d +2 γ ) · log n ≫ n d d +2 γ inducing variables have been used, while m ∗ d -inducing variables do not seem to be enough to get the same performance.

Misspecified noise. We now take the same fixed design setting as in Section 3 of the main paper, but sample Y i = f 0 ( X i ) + ε i , where ε i are independent standard Laplace random variables. Based on Table 5, the results are much the same as in the well-specified case, so that the credible sets demonstrate some robustness to noise misspecification. Note that while the coverage remains broadly similar, the credible intervals are slightly wider and have larger RMSE reflecting the larger noise.

B.1 Definitions of simulation statistics

For completeness, we define here the statistics reported in the simulation sections based on M Monte-Carlo samples. For Y ( j ) i = f 0 ( X i ) + ε ( j ) i , i = 1 , . . . , n and j = 1 , . . . , M , let C ( j ) m ( x 0 ) denote the credible interval for f ( x 0 ) , ¯ f ( j ) m ( x 0 ) and σ ( j ) m ( x 0 ) 2 the posterior mean and variance from the SGPR with m inducing variables computed from the j th realisation of the data. We then consider the Monte-Carlo estimates of the coverage , credible interval length , root-mean square error (RMSE)

and negative log-predictive density (NLPD) given by

respectively, where N ( f 0 ( x 0 ) | f ( j ) m ( x 0 ) , σ ( j ) m ( x 0 ) 2 ) denotes the density of the Gaussian distribution with mean f ( j ) m ( x 0 ) and variance σ ( j ) m ( x 0 ) 2 evaluated at the point f 0 ( x 0 ) .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBroadly, the paper addresses the issue of the quality of the approximate posterior of a sparse GP model in terms of uncertainty qualification (UQ). The paper chooses the setup with eigenvector-inducing variables and rescaled Brownian motion prior. However, as shown, the results also extend to other kernels (squared exponential (RBF) and Matérn kernels). Precisely, the paper's main conclusion is that with a well-calibrated prior and a sufficiently large number of inducing variables, the uncertainty quantification (UQ) obtained is reliable, though it can be conservative. The paper presents theoretical results on the rate and quality of SVGP posterior, nature of credible sets, and contraction rate of approximate posterior. The theoretical results are shown to match the empirical results based on experiments on simulation data and semi-simulation data.\n\nSTRENGTHS:\nThe topic of the paper is relevant as Gaussian processes are go-to models for many tasks, including Bayesian optimization, active learning, and various sequential tasks. Uncertainty quantification is one of the crucial properties of the Gaussian process models that make them popular for these tasks. Recently, a lot of research has been done on the quality of the approximate posterior of sparse GP models, and this paper rightly fits in there in terms of evaluating the quality of approximate posterior in terms of uncertainty quantification. The claims of the paper are sensible, and the detailed derivations in the appendix also look logical (I would like to see what other reviewers think about the proofs as that is not my core area). Apart from a few places, the paper's notations and flow are good and convenient to follow. The experiments are also chosen wisely and demonstrate the theoretical results.\n\nWEAKNESSES:\nA couple of weaknesses that I find in this paper are:\n\n* The paper talks about sparse variational Gaussian process models (SVGP) throughout, but actually, the model considered in the paper is sparse Gaussian process regression (SGPR). It can be clearly seen from Eq (3). It is better to make this clear in the paper. With a non-Gaussian likelihood, the equations for the posterior are different from those presented here.\n* Section 2.1 has issues with the notations. \n  - As pointed out earlier, the equations are of SGPR and not SVGP. \n  - L104 mean of the GP `m` is a typo?\n  - L117-L118 starts writing cov. Should it be `k`?\n  - $K$ is the gram matrix and should be bold as I believe the paper tries to follow that notation by making $k_n(x)$ bold.\n* A key takeaway message of the paper is missing. There are a bunch of theorems, derivations, and contributions, but I am still trying to figure out the main takeaway message from the paper that I should remember next time I plan to use a sparse GP model.\n* As we discuss uncertainty quantification, does it make sense to report negative log predictive density (NLPD) in the experiments along with the RMSE?\n* A plot of the true posterior, approximate posterior, and credible set would be good to demonstrate the setup and the output. \n* Theorem 3.5 derives a bound for the number of inducing variables `m`. How does it relate to Burt et al. (2019), as they also derive a bound for the number of inducing variables?\n\nQUESTIONS:\n* Is the metric for the quality of the approximate posterior different from the metric of UQ? Is having a “good” quality approximate posterior but not a well-calibrated UQ possible? As the authors also point out in the paper, there has been a lot of recent research evaluating the quality of approximate posterior and convergence rates. Is it much different?\n* What are the connections with Burt et al. (2019), and where do the authors agree and disagree with them? In general, I believe there should be more connections drawn with them.\n* A minor comment, references to the appendix need to be included. I do not see any reference to the appendix in the main paper, so connecting derivations and their corresponding sections becomes tricky.\n* More practical question: how to select the value of $\\alpha, \\gamma$ for a real-world dataset, let us say a UCI data set. I believe more evaluations should be done by extending section 4. It would make the work more applicable, in my opinion.\n* More questions are written in the weakness section.\n\nLIMITATIONS:\nThe authors do not discuss them, and I do not see a limitation directly of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper gives some theoretical results for frequentist coverage of the credible sets for sparce variational Gaussian processes using a class of Brownian motion based priors. It provides some conclusions relating the smoothness of the prior vs the smoothness of the true underlying function and the resulting ability of the Bayesian inference to produse reliable frequentist uncertainty estimates.\n\nSTRENGTHS:\nThe paper is very well written and easy to follow at a high level. The theoretical results are summarised well and give a good indication of the overall contribution of the paper. \n\nThe contribution seems to be an important starting point for providing guarantees for pointwise uncertainty estimates in SVGPs. It gives enough details to indicate the exact extent to which the results are valid. \n\nThe theoretical results are provided with a sufficient level of detail to be accessible by a more general (less statistically inclined) audience.\n\nThe discussion provides a number of directions for future work that can be taken up by the community.\n\nWEAKNESSES:\nThe main motivation for the work is the wide use of SVGPs in practice. However, the results in the paper consider only very rough (BM) priors which are rarely used in practice, and only touch on smoother, more commonly used options such as Matern kernels. Admittedly, it is still an important contribution that further work can build on.\n\nQUESTIONS:\nIt would be useful for my own understanding if the authors commented more on the statement that $L_2$-type credible sets are less reflective of actual practice than pointwise credible sets. \n\nI would also be curious why the authors chose this venue for publishing this work, given the theoretical nature of the results.\n\nLIMITATIONS:\nIt is possible that the paper will not be very accessible to the broader audience that it is addressing due to the theoretical nature of the main results. That said, I think the authors did a good job highlighting the exact contribution in the introduction and it should be accessible for practitioners in informing them of the possible consequences in the estimates of credible intervals based on the choice of priors. As the choice of prior is typically difficult in practice, these types of results are valuable in practice.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper gives algorithms for point wise estimation and uncertainty quantification (error bars) for \nsparse variational Gaussian processes with the Brownian motion as prior. The SGVP model though widely used in practice, lacks theoretical guarantees and this paper is the first to establish error guarantees for the model with Brownian priors. Experiments are presented to validate the results. The priors for SVGP models used in practice are not Brownian, extending the results in this paper to these priors remains an open question.\n\nSTRENGTHS:\n- This paper is among the first to address rigorously the statistical inference properties of the low rank approximations used in Gaussian processes. \n\nThe main result establishes rigorous bounds on the size of the confidence interval for the SGVP prediction given bounds on the smoothness of the prior and the ground truth and the number of eigenvalues retained in the approximation of the kernel matrix. This is a new and useful result for SGVPs. \n\n- The analysis is particular to the case of Brownian motion priors. BM priors have eigenvectors that can be expressed in closed form (they correspond to the trigonometric functions) and have strong self-similarity properties. However, in the experiments it is observed that the qualitative behavior that is observed for the BM priors is also observed for the Matern and squared exponential kernels used in practice.\n\nWEAKNESSES:\n- The result does not extend in a straightforward way to the Matern and squared exponential kernels used in practice with SGVPs. Brownian priors have many special properties that are not shared with other Gaussian processes used in ML. \n\n- The result although a useful addition to the literature on SGVPs do not seem not be of enough general interest for a clear accept \nand need further development for analyzing the instances of SGVPs used in practice. \n\n- The results are non adaptive as they require a pre-specified prior smoothness which is not a realistic assumption in practice.\n\nQUESTIONS:\nPg 2, the wedge notation for the minimum, the definitions of smoothness and rate should be introduced before stating main results. \n\nIt may be helpful to look at the stochastic process literature where the various properties of BM were originally developed, there may be other stochastic processes like the Ornstein-Uhlenbeck process considered in the stochastic process literature whose spectra are similar to the BM, however these do not seem to have not been used for SGVP.\n\nLIMITATIONS:\nLimitations are discussed clearly by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of uncertainty quantification for SVGP models with a Brownian motion prior for a non-parametric estimation problem. Theoretical results are presented that show\n\nSTRENGTHS:\nThe results are fundamental in nature and of broad interest, and clearly explained except in places where quantities are not clearly defined.\n\nWEAKNESSES:\nThe main contributions of the paper are a little unclear. It is a bit unclear which results are known and which results are new.\n\nIt is also unclear what is special about the rescaled Brownian motion prior. Does the paper recommend using the rBM prior over other priors because of some special properties. Otherwise the choice of rBM in the title is not well motivated.\n\nDoes the proof technique also extend to more general prior models beyond rBM.\n\nThe choice of rBM prior as the main focus is also not well motivated. If this is a standard choice, this must be explained clearly.\n\nIt would perhaps be useful to provide a brief sketch of the proof. For eg the discussion below Proposition 3.2 is somewhat of a proof sketch but isn't labeled as such.\n\nQUESTIONS:\nWhat precisely is Q*. Can you please define it formally in section 2. Perhaps Q* should include a gamma in the subscript. Is Q* the eigenvector SVGP or any SVGP as defined above (3).\n\nSimilarly please also provide the definition of rescaled BM in the preliminaries. \n\nA formula for the Mattern kernel in Section 5 would be useful.\n\nTable 1 columns currently labeled m* and n need to be renamed to something more informative and which doesnt require reading the paragraph above the table before reading the table. The caption can also be more informative (and better spaced from the table). The explicit value of m* would also be useful to know instead of the formula in the caption. A brief description of coverage, length and RMSE is necessary to read the table clearly. Alas, these are relegated to the appendix. (Similarly for Table2)\n\nn,2 subscript in Proposition 3.4 is unclear.\n\nIs Lemma 3.1 a noteworthy result or can it be part of section 2? Similarly the paragraph above Prop 3.2\n\nIn Section 2.1, is it GP(nu0, k) instead of m? please dont use m for mean function since m is the number of inducing points.\n\nThe space in which u_i belong is quite unclear. It appears they are actually part of the target space (ie same as space of y_i). Then the choice of the naming them Eigenvector inducing features is perhaps misleading, and Eigenvector inducing \"targets\" is perhaps more appropriate.\n\nThe term cov(f(x), u_i) is confusing. How is this covariance function defined? since the kernel requires two points in the domain as input. Please clarify this immediately following the definitions of Kxm etc., rather than point the reader to [38].\n\nThe bolding can be more consistent. I think r_m is a vector which is not bolded unlike other vector quantities.\n\nIn Section 3.1 end of line 150, shouldn't this also be f(x)|Dn since SVGP also is conditional on n samples?\n\nThe choice of naming tn as \"frequentist variance\" is unclear. Maybe a comment on that (at least in OpenReview) would be useful, but also to the reader of this paper. As suggested earlier, the discussion leading upto Prop 3.2 can be part of Section 2 since everything follows naturally from the preliminaries and does not require any special mention in the section on Main results.\n\nThe role of Mn in all results is a little unclear. An example Mn and its implications would be useful for the reader.\n\nLIMITATIONS:\nNo limitations envisioned.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}