Title: Sampling from Gaussian Process Posteriors using Stochastic Gradient Descent
Venue: NeurIPS 2023
Accepted: yes
Keywords: Gaussian processes

scalable learning

posterior sampling

Bayesian optimization
Abstract: Gaussian processes are a powerful framework for quantifying uncertainty and for sequential decision-making but are limited by the requirement of solving linear systems. In general, this has a cubic cost in dataset size and is sensitive to conditioning. We explore stochastic gradient algorithms as a computationally efficient method of approximately solving these linear systems: we develop low-variance optimization objectives for sampling from the posterior and extend these to inducing points. Counterintuitively, stochastic gradient descent often produces accurate predictions, even in cases where it does not converge quickly to the optimum. We explain this through a spectral characterization of the implicit bias from non-convergence. We show that stochastic gradient descent produces predictive distributions close to the true posterior both in regions with sufficient data coverage, and in regions sufficiently far away from the data. Experimentally, stochastic gradient descent achieves state-of-the-art performance on sufficiently large-scale or ill-conditioned regression tasks. Its uncertainty estimates match the performance of significantly more expensive baselines on a large-scale Bayesian~optimization~task.

Sampling from Gaussian Process Posteriors using Stochastic Gradient Descent

Abstract

Gaussian processes are a powerful framework for quantifying uncertainty and for sequential decision-making but are limited by the requirement of solving linear systems. In general, this has a cubic cost in dataset size and is sensitive to conditioning. We explore stochastic gradient algorithms as a computationally efficient method of approximately solving these linear systems: we develop low-variance optimization objectives for sampling from the posterior and extend these to inducing points. Counterintuitively, stochastic gradient descent often produces accurate predictions, even in cases where it does not converge quickly to the optimum. We explain this through a spectral characterization of the implicit bias from non-convergence. We show that stochastic gradient descent produces predictive distributions close to the true posterior both in regions with sufficient data coverage, and in regions sufficiently far away from the data. Experimentally, stochastic gradient descent achieves state-of-the-art performance on sufficiently large-scale or ill-conditioned regression tasks. Its uncertainty estimates match the performance of significantly more expensive baselines on a large-scale Bayesian optimization task.

1 Introduction

Gaussian processes (GPs) provide a comprehensive framework for learning unknown functions in an uncertainty-aware manner. This often makes Gaussian processes the model of choice for sequential decision-making, achieving state-of-the-art performance in tasks such as optimizing molecules in computational chemistry settings [21] and automated hyperparameter tuning [39, 24].

The main limitations of Gaussian processes is that their computational cost is cubic in the training dataset size. Significant research efforts have been directed at addressing this limitation resulting in two key classes of scalable inference methods: (i) inducing point methods [43, 23], which approximate the GP posterior, and (ii) conjugate gradient methods [20, 19, 4], which approximate the computation needed to obtain the GP posterior. Note that in structured settings, such as geospatial learning in low dimensions, specialized techniques are available [49, 48, 15]. Throughout this work, we focus on the generic setting, where scalability limitations are as of yet unresolved.

In recent years, stochastic gradient descent (SGD) has emerged as the leading technique for training machine learning models at scale [37, 42], in both deep learning and related settings like kernel methods [14] and Bayesian modeling [29]. While the principles behind the effectiveness of SGD are not yet fully understood, empirically, SGD often leads to good predictive performance-even when it does not fully converge. The latter is the default regime in deep learning, and has motivated researchers to study implicit biases and related properties of SGD [6, 52].

Code available at: HTTPS://GITHUB.COM/CAMBRIDGE-MLG/SGD-GP.

In the context of GPs, SGD is commonly used to learn kernel hyperparameters-by optimizing the marginal likelihood [18, 13, 19, 12, 11] or closely related variational objectives [43, 23]. In this work, we explore applying SGD to the complementary problem of approximating GP posterior samples given fixed kernel hyperparameters. In one of his seminal books on statistical learning theory, Vladimir Vapnik [45] famously said: 'When solving a given problem, try to avoid solving a more general problem as an intermediate step. ' Motivated by this viewpoint, as well as the aforementioned property of good performance often not requiring full convergence when using SGD, we ask: Do the linear systems arising in GP computations necessarily need to be solved to a small error tolerance? If not, can SGD help accelerate these computations?

We answer the latter question affirmatively, with specific contributions as follows: (i) We develop a scheme for drawing GP posterior samples by applying SGD to a quadratic problem. In particular, we re-cast the pathwise conditioning technique of Wilson et al. [50, 51] as an optimization problem to which we apply the low-variance SGD sampling estimator of Antorán et al. [3]. We extend the proposed method to inducing point Gaussian processes. (ii) We characterize the implicit bias in SGD-approximated GP posteriors showing that despite optimization not fully converging, these match the true posterior in regions both near and far away from the data. (iii) Experimentally, we show that SGD produces strong results-compared to variational and conjugate gradient methods-on both large-scale and poorly-conditioned regression tasks, and on a parallel Thompson sampling task, where error bar calibration is paramount.

2 Optimization and Pathwise Conditioning in Gaussian Processes

A random function f : X → R over some set X is a Gaussian process [35] if, for every finite set of points x ∈ X N , f ( x ) is multivariate Gaussian. A Gaussian process f ∼ GP( µ, k ) is uniquely determined by a mean function µ ( · ) = E ( f ( · )) and a covariance kernel k ( · , · ′ ) = Cov( f ( · ) , f ( · ′ )) . We denote by K xx the kernel matrix [ k ( x i , x j )] i,j =1 ,..,N . We consider the Bayesian model y = f ( x ) + ε , where ε ∼ N( 0 , Σ ) gives the likelihood, f ∼ GP(0 , k ) is the prior, and x , y are the training data. The posterior of this model is f | y ∼ GP( µ f | y , k f | y ) , with

Using pathwise conditioning [50, 51] one can also write the posterior directly as the random function

We assume throughout that Σ is diagonal. Due to the matrix inverses present, the cost to directly compute each of the above expressions is O ( N 3 ) .

2.1 Random Fourier Features and Efficient Sampling

Our techniques will rely on random Fourier features [34, 40]. Let X = R d and let k be stationarythat is, k ( x, x ′ ) = k ( x -x ′ ) . Assume, in this section only and without loss of generality, that k ( x, x ) = 1 . Random Fourier features are sets of L random functions Φ : X → R L whose components, indexed by ℓ ≤ L , are ϕ ℓ ( · ) = L -1 / 2 cos(2 π ⟨ ω ℓ , ·⟩ ) for ℓ odd, and ϕ ℓ ( · ) = L -1 / 2 sin(2 π ⟨ ω ℓ , ·⟩ ) for ℓ even. By taking ω ℓ to be samples distributed according to ρ , the normalized spectral measure of the kernel, for any x, x ′ ∈ X we have

which we will use to construct unbiased estimators of kernel matrices in the sequel. Fourier features can also be used to efficiently sample functions from the GP prior. Naively, evaluating the random function f at N ∗ points requires computing a matrix square root of K xx at O ( N 3 ∗ ) computational cost. However, given a precomputed set of Fourier features Φ , the Monte Carlo estimator

approximately samples from GP (0 , k ) at O ( N ∗ ) cost. The approximation error in (4) decays as L goes to infinity. Following Wilson et al. [50, 51], our proposed algorithms will approximately sample from the posterior by replacing f with ˜ f in the pathwise conditioning formula (2).

Infill asymptotics

Large-domain

Stochastic Gradient Descent

Conjugate Gradients

Exact GP

-

5

.

0

-

2

.

5

0

.

0

2

.

5

5

.

0

Infill asymptotics

Stochastic Gradient Descent

Conjugate Gradients

Variational Inference

-7

-3.5

0

3.5

7

-

4

-

2

0

2

4

Large-domain

asymptotics

-7

-3.5

0

3.5

7

-7

-3.5

0

3.5

-

5

.

0

-

2

.

5

0

.

0

2

.

5

5

.

0

Infill asymptotics

Stochastic Gradient Descent

Conjugate Gradients

Variational Inference

-7

-3.5

0

3.5

7

-

4

-

2

0

2

4

Large-domain

asymptotics

-7

-3.5

0

3.5

7

-7

-3.5

0

3.5

-

5

.

0

-

2

.

5

0

.

0

2

.

5

5

.

0

Stochastic Gradient Descent

Conjugate Gradients

Variational Inference

-7

-3.5

0

3.5

7

-

4

-

2

0

2

4

asymptotics

-7

-3.5

0

3.5

7

-7

-3.5

0

3.5

7

-

5

.

0

-

2

.

5

0

.

0

2

.

5

5

.

0

Infill asymptotics

-7

-3.5

0

3.5

7

-

4

-

2

0

2

4

Large-domain

asymptotics

-7

-3.5

0

3.5

7

-7

-3.5

0

3.5

Exact GP

Approximations

Approximations

Exact GP

Approximations

Exact GP

Approximations

2.2 Optimization-based Learning in Gaussian Processes

Following Matthews et al. [31] and Antorán et al. [3], both a Gaussian process' posterior mean and posterior samples can be expressed as solutions to quadratic optimization problems. Letting v ∗ = ( K xx + Σ ) -1 y in (1), we can express the GP posterior mean in terms of the quadratic problem

We say that k ( x i , · ) are the canonical basis functions , v i are the representer weights [38], and that ∥ v ∥ 2 K xx = v T K xx v is the regularizer . The respective optimization problem for obtaining posterior samples is similar, but involves a stochastic objective, which will be given and analyzed in Section 3.2.

Conjugate gradients (CG) are the most widely used algorithm to solve quadratic problems, both in the context of GPs [20, 19, 4] and more generally [33, 10]. CG reduces GP posterior inference to a series of matrix-vector products, each of O ( N 2 ) cost. Given the system A -1 b , the number of matrix-vector products needed to guarantee convergence of CG to within a tolerance of ε [41], is

where λ max ( A ) and λ min ( A ) are the maximum and minimum eigenvalues of A . CG performs well in many GP use cases, for instance Gardner et al. [19] and Wang et al. [46]. In general, however, the condition number cond( K xx + Σ ) need not be bounded, and conjugate gradients may fail to converge quickly [41]. Nonetheless, by exploiting the quadratic structure of the objective, CG obtains substantially better convergence rates than gradient descent [8, 52]. This presents a pessimistic outlook for gradient descent and related methods such as SGD which do not take explicit advantage of the quadratic nature of the objective. Nonetheless, the success of SGD in large-scale machine learning [9] motivates us to try it anyway.

Variational Inference

7

7

7

3 Gaussian Process Predictions via Stochastic Gradient Descent

We now develop and analyze techniques for drawing samples from GP posteriors using SGD. This is done by rewriting the pathwise conditioning formula (2) in terms of two stochastic optimization problems. As a preview of what this will produce, we showcase SGD's performance on a pair of toy problems, designed to capture complementary computational difficulties, in Figure 1.

3.1 A Stochastic Objective for Computing the Posterior Mean

We begin by deriving a stochastic objective for the posterior mean. The optimization problem (5), with optimal representer weight solution v ∗ , requires O ( N 2 ) operations to compute both its square error and regularizer terms exactly. The square error loss term is amenable to minibatching, which gives an unbiased estimate in O ( N ) operations. Assuming that k is stationary, following Section 2.1, we can stochastically estimate the regularizer with Fourier features using the identity ∥ v ∥ 2 K xx = E ω 1 ,...,ω L ∼ ρ v T Φ T ( x ) Φ ( x ) v . Combining both estimators gives our SGD objective

where D is the minibatch size and L the number of Fourier features. This regularizer estimate is unbiased even when drawing a single Fourier feature per step: the number of features controls the variance. Equation (7) presents O ( N ) complexity, in contrast with the O ( N 2 ) complexity of one CG step. We discuss sublinear inducing point techniques further on, but first turn to sampling.

3.2 Computing Posterior Samples

We now frame GP posterior samples in a manner amenable to SGD computation similarly to (7). First, we re-write the pathwise conditioning expression given in (2) as

We approximate the prior function sample using the Fourier feature approach of (4) and the posterior mean, defined in Section 2.2, with the minimizer v ∗ of (7) obtained by SGD. Each posterior sample's uncertainty reduction term is parametrized by a set of representer weights given by a linear solve against a noisy prior sample evaluated at the observed inputs, namely α ∗ = ( K xx + Σ ) -1 ( f ( x )+ ε ) . We construct an optimization objective targeting a sample's optimal representer weights as

Applying minibatch estimation to this objective results in high gradient variance, since the presence of ε i makes the targets noisy. To avoid this while targeting the same objective, we modify (9) as

moving the noise term into the regularizer. This modification preserves the optimal representer weights since objective (10) equals (9) up to a constant: a proof is given in Appendix D. This generalizes the variance reduction technique of Antorán et al. [3] to the GP setting. Figure 2 illustrates minibatch gradient variance for these objectives. Applying the minibatch and random feature estimators of (7), we obtain a per-step cost of O ( NS ) , for S the number of posterior samples.

3.3 Inducing Points

So far, our sampling objectives have presented linear cost in dataset size. In the large-scale setting, algorithms with costs independent of the dataset size are often preferable. For GPs, this can be achieved through inducing point posteriors [43, 23], to which we now extend SGD sampling.

10

0

10

1

10

2

Iterations

0

2k

4k

6k

Gradient Variance

Loss 1

Loss 2

10

1

10

2

10

3

10

4

Time in sec

0

.

00

0

.

25

0

.

50

0

.

75

RMSE

10

1

10

2

10

3

10

4

Time in sec

-

1

.

2

-

1

.

0

-

0

.

8

-

0

.

6

NLL

0.1h

1h

10h

Total Runtime

Standard SGD

M

= 1844352

Inducing SGD

M

= 1099206

Inducing SGD

M

= 431489

Inducing SGD

M

= 218782

Let z ∈ X M be a set of M ∈ N inducing points. Applying pathwise conditioning to the KullbackLeibler-optimal inducing point approximation of Titsias [43] gives the expression

Following Wild et al. [47], Theorem 5, the optimal inducing point mean µ ( z ) f | y can therefore be written

and we can again parameterize the uncertainty reduction term (8) as K ( · ) z α ∗ with

A full derivation is given in Appendix C. Exact implementation of (13) is precluded by the need to sample from a Gaussian with covariance K xz K -1 zz K zx . However, we identify this matrix as a Nyström approximation to K xx . Thus, we can approximate (13) by replacing f ( z ) with f , which can be sampled with Fourier features (4). The approximation error is small when M is large and the inducing points are close enough to the data. Our experiments in Appendix C show it to be negligible in practice. With this, we apply the stochastic estimator (7) to the inducing point sampling objective.

The inducing point objectives differ from those presented in previous sections in that there are O ( M ) and not O ( N ) learnable parameters, and we may choose the value of M and locations z freely. The cost of inducing point representer weight updates is thus O ( SM ) , where S is the number of samples. This contrasts with the O ( M 3 ) update cost of stochastic gradient variational Gaussian processes [23]. Figure 2 shows that SGD with M ≈ 100 k inducing points matches the performance of regular SGD on HOUSEELECTRIC ( N ≈ 2 M), but is an order of magnitude faster.

3.4 The Implicit Bias and Posterior Geometry of Stochastic Gradient Descent

We have detailed an SGD-based scheme for obtaining approximate samples from a posterior Gaussian process. Despite SGD's significantly lower cost per-iteration than CG, its convergence to the true optima, shown in Figure 3, is much slower with respect to both Euclidean representer weight space, and the reproducing kernel Hilbert space (RKHS) induced by the kernel. Despite this, the predictions obtained by SGD are very close to those of the exact GP, and effectively achieve the same test RMSE. Moreover, Figure 4 shows the SGD posterior on a 1D toy task exhibits error bars of the correct width close to the data, and which revert smoothly to the prior far away from the data. Empirically, differences between the SGD and exact posteriors concentrate at the borders of data-dense regions.

10

-

1

10

0

10

1

10

2

10

3

Time in sec

0

.

0

1

.

2

2

.

4

3

.

6

RMSE to Test Data

10

-

1

10

0

10

1

10

2

10

3

Time in sec

0

.

0

0

.

3

0

.

6

0

.

9

RMSE to Exact GP

0

.

0

3

.

5

Euclidean Error

10

-

1

10

0

10

1

10

2

10

3

Time in sec

30

160

0

.

0

0

.

6

RKHS Error

10

-

1

10

0

10

1

10

2

10

3

Time in sec

0

5

Opt.

noise

Low

noise

SGD (optimal noise)

SGD (low noise)

CG (optimal noise)

CG (low noise)

We now argue the behavior seen in Figure 4 is a general feature of SGD: one can expect it to obtain good performance even in situations where it does not converge to the exact solution. Consider posterior function samples in pathwise form, namely ( f | y )( · ) = f ( · )+ K ( · ) x v , where f ∼ GP(0 , k ) is a prior function sample and v are the learnable representer weights. We characterize the behavior of SGD-computed approximate posteriors by splitting the input space X into 3 regions, which we call the prior , interpolation , and extrapolation regions. This is done as follows.

(I) The Prior Region. This corresponds to points sufficiently distant from the observed data. Here, for kernels that decay over space, the canonical basis functions k ( x i , · ) go to zero. Thus, both the true posterior and any approximations formulated pathwise revert to the prior. More precisely, let X = R d , let k satisfy lim c →∞ k ( x ′ , c · x ) = 0 for all x ′ and x in X , and let ( f | y )( · ) be given by ( f | y )( · ) = f ( · ) + K ( · ) x v , with v ∈ R N . Then, by passing the limit through the sum, for any fixed v , it follows immediately that lim c →∞ ( f | y )( c · x ) = f ( c · x ) . Therefore, SGD cannot incur error in regions which are sufficiently far away from the data. This effect is depicted in Figure 4.

(II) The Interpolation Region. This includes points close to the training data. We characterize this region through subspaces of the RKHS, where we show SGD incurs small error.

Let K xx = UΛU T be the eigendecomposition of the kernel matrix. We index the eigenvalues Λ = diag( λ 1 , . . . , λ N ) in descending order. Define the spectral basis functions as eigenvectorweighed linear combinations of canonical basis functions

These functions-which also appear in kernel principal component analysis-are orthonormal with respect to the RKHS inner product. To characterize them further, we lift the Courant-Fischer characterization of eigenvalues and eigenvectors to the RKHS H k induced by k , obtaining the expression

This means in particular that the top spectral basis function, u (1) ( · ) , is a function of fixed RKHS norm-that is, of fixed degree of smoothness, as defined by the kernel k -which takes maximal values at the observations x 1 , .., x n . Thus, u (1) will be large near clusters of observations. The same will be true for the subsequent spectral basis functions, which also take maximal values at the observations, but are constrained to be RKHS-orthogonal to previous spectral basis functions. A proof of the preceding expression is given in Appendix E.3. Figure 4 confirms that the top spectral basis functions are indeed centered on the observed data.

Empirically, SGD matches the true posterior in this region. We formalize this observation by showing that SGD converges quickly in the directions spanned by spectral basis functions with large eigenvalues. Let proj u ( i ) ( · ) be the orthogonal projection onto the subspace spanned by u ( i ) .

·

-

10

-

5

0

5

10

-

5

.

0

-

2

.

5

0

.

0

2

.

5

5

.

0

SGD Posterior

Wasserstein-2 to Exact GP

10

0

10

1

10

2

10

3

10

4

Index

10

-

9

10

-

4

10

1

10

6

Kernel Matrix Eigenvalues

1

2

3

4

11

16

41

45

-

10

-

5

0

5

10

x

Spectral Basis Functions

0

1

2

3

4

-0.15

-0.07

0

0.07

0.15

u

(

i

)

(

)

Proposition 1. Let δ > 0 . Let Σ = σ 2 I for σ 2 > 0 . Let µ SGD be the predictive mean obtained by Polyak-averaged SGD after t steps, starting from an initial set of representer weights equal to zero, and using a sufficiently small learning rate of 0 < η < σ 2 λ 1 ( λ 1 + σ 2 ) . Assume the stochastic estimate of the gradient is G -sub-Gaussian. Then, with probability 1 -δ , we have for i = 1 , .., N that

The proof, as well as an additional pointwise convergence bound and a variant that handles projections onto general subspaces spanned by basis functions, are provided in Appendix E.1. In general, we expect G to be at most O ( λ 2 1 ∥ y ∥ ∞ ) with high probability.

The result extends immediately from the posterior mean to posterior samples. As consequence, SGD converges to the posterior GP quickly in the data-dense region , namely where the spectral basis functions corresponding to large eigenvalues are located. Since convergence speed on the span of each basis function is independent of the magnitude of the other basis functions' eigenvalues, SGD can perform well even when the kernel matrix is ill-conditioned. This is shown in Figure 3.

(III) The Extrapolation Region. This can be found by elimination from the input space of the prior and interpolation regions, in both of which SGD incurs low error. Consider the spectral basis functions u ( i ) ( · ) with small eigenvalues. By orthogonality of u (1) , .., u ( N ) , such functions cannot be large near the observations while retaining a prescribed norm. Their mass is therefore placed away from the observations. SGD converges slowly in this region, resulting in a large error in its solution in both a Euclidean and RKHS sense, as seen in Figure 3. Fortunately, due to the lack of data in the extrapolation region, the excess test error incurred due to SGD nonconvergence may be low, resulting in benign nonconvergence [52]. Similar phenomena have been observed in the inverse problems literature, where this is called iterative regularization [22, 27]. Figure 4 shows the Wasserstein distance to the exact GP predictions is high in this region, as when initialized at zero SGD tends to return small representer weights, thereby reverting to the prior.

0

.

0

0

.

6

1

.

2

RMSE

pol

N

= 15000

10

0

10

1

10

2

Time in sec

-

1

.

4

-

0

.

3

0

.

8

NLL

0

.

0

1

.

2

2

.

4

keggdirected

N

= 48827

10

0

10

1

10

2

10

3

Time in sec

-

1

.

4

0

.

0

1

.

4

0

.

0

0

.

6

1

.

2

3droad

N

= 434874

10

0

10

1

10

2

10

3

10

4

Time in sec

-

0

.

7

0

.

0

0

.

7

0

.

0

0

.

8

1

.

6

houseelectric

N

= 2049280

10

0

10

1

10

2

10

3

10

4

Time in sec

-

2

4

10

SGD

CG

4 Experiments

We now turn to empirical evaluation of SGD GPs, focusing on their predictive and decision-making properties. We compare SGD GPs with the two most popular scalable Gaussian process techniques: preconditioned conjugate gradient (CG) optimization [19, 46] and sparse stochastic variational inference (SVGP) [43, 23]. For CG, we use a pivoted Cholesky preconditioner of size 100, except in cases where this slows down convergence, where we instead report results without preconditioning. We employ the GPJax [32] SVGP implementation and use M = 4096 inducing points for all datasets, initializing their locations with the K -means algorithm. Full experimental details are in Appendix A.

4.1 Regression Baselines

We first compare SGD-based predictions with baselines in terms of predictive performance, scaling of computational cost with problem size, and robustness to the ill-conditioning of linear systems. Following Wang et al. [46], we consider 9 datasets from the UCI repository [16] ranging in size from N = 15 k to N ≈ 2 Mdatapoints and dimensionality from d = 3 to d = 90 . We report mean and standard deviation over five 90% -train 10% -test splits for the small and medium datasets, and three splits for the largest dataset. For all methods, we use a Matérn3 / 2 kernel with a fixed set of hyperparameters obtained using maximum marginal likelihood [2], as described in Appendix A.1.

We run SGD for 100 k steps, with a fixed batch size of 512 for both the mean function and samples. For CG, we run a maximum of 1000 steps for datasets with N ≤ 500 k, and a tolerance of 0 . 01 . On the four largest datasets, CG's per-step cost is too large to run 1000 steps: instead, we run 100 steps, which takes roughly 9 hours per function sample on a TPUv2 core. For SVGP, we learn the variational parameters for M = 4096 inducing points by maximizing the ELBO with Adam until convergence. For all methods, we estimate predictive variances for log-likelihood computations from 64 function samples drawn using pathwise conditioning (2).

Predictive performance and scalability with the number of inputs. Our complete set of results is provided in Table 1, including test RMSE, test negative log-likelihood (NLL) and compute time needed to obtain the predictive mean on a single core of a TPUv2 device. Drawing multiple samples requires repeating this computation, which we perform in parallel. In the small setting ( N ≤ 20 k), taking 100 k steps of SGD presents a compute cost comparable to running CG to tolerance, which usually takes around 500-800 steps. Here, CG converges to the exact solution, while SGD tends to present increased test error due to non-convergence. In the large setting ( N ≥ 100 k ) , where neither method converges within the provided compute budget, SGD achieves better RMSE and NLL and results. SVGP always converges faster than CG and SGD, but it only performs best on the BUZZ dataset, which can likely be summarised well by M = 4096 inducing points.

From Figure 5, we see that SGD makes the vast majority of its progress in prediction space in its first few iterations, improving roughly monotonically with the number of steps. Thus, early stopping after 100 k iterations incurs only moderate errors. In contrast, CG's initial steps actually increase test error, resulting in very poor performance if stopped too early. This interacts poorly with the number of CG steps needed, and the per-step cost, which generally grow with increased amounts of data [41].

Robustness to kernel matrix ill-conditioning. GP models are known to be sensitive to kernel matrix conditioning. We explore how this affects the algorithms under consideration by fixing the noise variance to a low value of σ 2 = 10 -6 and running them on our regression datasets. Table 1 shows the performance of CG severely degrades on all datasets and, for SVGP, optimization diverges for all datasets. SGD's results remain essentially-unchanged. This is because the noise only changes the smallest kernel matrix eigenvalues substantially and these do not affect convergence for the top spectral basis functions. This mirrors results presented previously for the ELEVATORS dataset in Figure 3.

Regression with large numbers of inducing points. We demonstrate the inducing point variant of our method, presented in Section 3.3, on HOUSEELECTRIC, our largest dataset ( N = 2 M). We select varying numbers of inducing points with a K -nearest-neighbor algorithm, described in Appendix A.2. Figure 2 shows the time required for 100 k SGD steps scales roughly linearly with inducing points. It takes 68m for full SGD and 50m, 25m, and 17m for M = 1099 k , 728 k, and 218 k, respectively. Performance in terms of RMSE and NLL degrades less than 10% even when using 218 k points.

4.2 Large-scale Parallel Thompson Sampling

A fundamental goal of scalable Gaussian processes is to produce uncertainty estimates useful for sequential decision making. Motivated by problems in large-scale recommender systems, where both the initial dataset and the total number of users queried are simultaneously large [36, 17], we benchmark SGD on a large-scale Bayesian optimization task.

We draw a target function from a GP prior g ∼ GP(0 , k ) and optimize it on X = [0 , 1] d using parallel Thompson sampling [25]. That is, we choose x new = arg max x ∈ X ( f | y )( · ) for a set of posterior function samples drawn in parallel. We compute these samples using pathwise conditioning with each respective scalable GP method. For each function sample maximum, we evaluate y new = g ( x new ) + ε with ε ∼ N(0 , 10 -6 ) and add the pair ( x new , y new ) to the training data. We use an acquisition batch size of 1000 samples, and maximize them with a multi-start gradient descent-based approach described in Appendix A.3. We set the search space dimensionality to d = 8 , the largest considered by Wilson et al. [50], and initialize all methods with a dataset of 50 k observations sampled uniformly at random from X . To eliminate model misspecification confounding, we use a Matérn3 / 2 kernel and consider length scales of (0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5) for both the target function and our models. For each length scale, we repeat the experiment for 10 seeds.

In large-scale Bayesian optimization, training and posterior function optimization costs can become significant, and predictions may be needed on demand. For this reason, we consider two variants of our experiment with different levels of compute. In the small-compute setting, SGD is run for 15 k steps, SVGP is given M = 1024 inducing points and 20 k steps to fit the variational parameters,

50k

60k

70k

80k

Acquisitions

4

.

0

4

.

5

5

.

0

5

.

5

6

.

0

50k

60k

70k

80k

Acquisitions

0h

1h

2h

3h

Time elapsed

0h

5h

10h

15h

Time elapsed

Small Compute Budget

Large Compute Budget

Random Search

SGD

CG

SVGP

and CG is run for 10 steps. In the large-compute setting, all methods are run for 5 times as many optimization steps.

Mean results and standard errors, across length scales and seeds, are presented in Figure 6. We plot the maximum function value achieved by each method. In the small-compute setting, the ∼ 1 . 5 (A100 GPU) hours required for 30 Thompson steps with SVGP and SGD are dominated by the algorithm used to maximise the models' posterior samples. In contrast, CG takes roughly twice the time, requiring almost 3h of wall-clock time. Despite this, SGD makes the largest progress per acquisition step, finding a target function value that improves upon the initial training set maximum twice as much as the other inference methods. VI and CG perform comparably, with the latter providing slightly more progress both per acquisition step and unit of time. Despite their limited compute budget, all methods outperform random search by a significant margin. In the large-compute setting, all inference methods achieve a similar maximum target value by the end of the experiment. CG and SGD make similar progress per acquisition step but SGD is faster per unit of time. SVGP is slightly slower than CG per both. In summary, our results suggest that SGD can be an appealing uncertainty quantification technique for large-scale GP-based sequential decision making.

5 Conclusion

In this work, we explored using stochastic gradient algorithms to approximately compute Gaussian process posterior means and function samples. We derived optimization objectives with linear and sublinear cost for both. We showed that SGD can produce accurate predictions-even in cases when it does not converge to an optimum. We developed a spectral characterization of the effects of non-convergence, showing that it manifests itself mainly through error in an extrapolation region located away-but not too far away-from the observations. We benchmarked SGD on regression tasks of various scales, achieving state-of-the-art performance for sufficiently large or ill-conditioned settings. On a Thompson sampling benchmark, where well-calibrated uncertainty is paramount, SGD matches the performance of more expensive baselines at a fraction of the computational cost.

Acknowledgments

Weare grateful to David R. Burt for suggesting a few tricks which helped us complete the development of our theory and thank Bruno Mlodozeniec for identifying and correcting a small mistake in the proof of Appendix E.1. JAL and SP were supported by the University of Cambridge Harding Distinguished Postgraduate Scholars Programme. JA acknowledges support from Microsoft Research, through its PhD Scholarship Programme, and from the EPSRC. JMHL acknowledges support from a Turing AI Fellowship under grant EP/V023756/1. AT was supported by Cornell University, jointly via the Center for Data Science for Enterprise and Society, the College of Engineering, and the Ann S. Bowers College of Computing and Information Science. This work has been performed using resources provided by the Cambridge Tier-2 system operated by the University of Cambridge Research Computing Service (http://www.hpc.cam.ac.uk) funded by an EPSRC Tier-2 capital grant. This work was also supported with Cloud TPUs from Google's TPU Research Cloud (TRC).

References

J. Antorán, R. Barbano, J. Leuschner, J. M. Hernández-Lobato, and B. Jin. Uncertainty Estimation for Computed Tomography with a Linearised Deep Image Prior. arXiv:2203.00479 , 2022. Cited on page 14.

J. Antorán, D. Janz, J. U. Allingham, E. A. Daxberger, R. Barbano, E. T. Nalisnick, and J. M. Hernández-Lobato. Adapting the Linearised Laplace Model Evidence for Modern Deep Learning. In International Conference on Machine Learning , 2022. Cited on page 8.

J. Antorán, S. Padhy, R. Barbano, E. T. Nalisnick, D. Janz, and J. M. Hernández-Lobato. Sampling-based inference for large linear models, with application to linearised Laplace. In International Conference on Learning Representations , 2023. Cited on pages 2-4, 14, 20, 21.

A. Artemev, D. R. Burt, and M. van der Wilk. Tighter Bounds on the Log Marginal Likelihood of Gaussian Process Regression Using Conjugate Gradients. In International Conference on Machine Learning , 2021. Cited on pages 1, 3.

R. Barbano, J. Leuschner, J. Antorán, B. Jin, and J. M. Hernández-Lobato. Bayesian Experimental Design for Computed Tomography with the Linearised Deep Image Prior. In ICML Workshop on Adaptive Experimental Design and Active Learning in the Real World , 2022. Cited on page 14.

M. Belkin, D. Hsu, S. Ma, and S. Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences , 2019. Cited on page 1.

C. M. Bishop. Pattern Recognition and Machine Learning . Springer, 2007. Cited on page 19.

G. Blanchard and N. Krämer. Optimal Learning Rates for Kernel Conjugate Gradient Regression. In Advances in Neural Information Processing Systems , 2010. Cited on page 3.

L. Bottou. Large-scale Machine Learning with Stochastic Gradient Descent. In International Conference on Computational Statistics , 2010. Cited on page 3.

S. P. Boyd and L. Vandenberghe. Convex Optimization . Cambridge University Press, 2014. Cited on page 3.

H. Chen, L. Zheng, R. Al Kontar, and G. Raskutti. Gaussian Process Parameter Estimation Using Mini-batch Stochastic Gradient Descent: Convergence Guarantees and Empirical Benefits. Journal of Machine Learning Research , 2022. Cited on page 2.

H. Chen, L. Zheng, R. Al Kontar, and G. Raskutti. Stochastic gradient descent in correlated settings: A study on Gaussian processes. Advances in Neural Information Processing Systems , 2020. Cited on page 2.

K. Cutajar, M. Osborne, J. Cunningham, and M. Filippone. Preconditioning Kernel Matrices. In International Conference on Machine Learning , pages 2529-2538, 2016. Cited on page 2.

B. Dai, B. Xie, N. He, Y. Liang, A. Raj, M.-F. F. Balcan, and L. Song. Scalable Kernel Methods via Doubly Stochastic Gradients. In Advances in Neural Information Processing Systems , 2014. Cited on page 1.

E. A. Daxberger, E. T. Nalisnick, J. U. Allingham, J. Antorán, and J. M. Hernández-Lobato. Bayesian Deep Learning via Subnetwork Inference. In International Conference on Machine Learning , 2021. Cited on page 1.

D. Dua and C. Graff. UCI Machine Learning Repository, 2017. Cited on page 8.

M. Elahi, F. Ricci, and N. Rubens. A survey of active learning in collaborative filtering recommender systems. Computer Science Review , 2016. Cited on page 9.

M. Filippone and R. Engler. Enabling scalable stochastic gradient-based inference for Gaussian processes by employing the Unbiased LInear System SolvEr (ULISSE). In International Conference on Machine Learning , 2015. Cited on page 2.

J. R. Gardner, G. Pleiss, K. Q. Weinberger, D. Bindel, and A. G. Wilson. GPyTorch: Blackbox Matrix-matrix Gaussian Process Inference with GPU Acceleration. In Advances in Neural Information Processing Systems , 2018. Cited on pages 1-3, 8.

M. N. Gibbs and D. J. C. MacKay. Efficient Implementation of Gaussian Processes for Interpolation. Technical report, University of Cambridge, 1996. Cited on pages 1, 3.

R. Gómez-Bombarelli, J. N. Wei, D. Duvenaud, J. M. Hernández-Lobato, B. SánchezLengeling, D. Sheberla, J. Aguilera-Iparraguirre, T. D. Hirzel, R. P. Adams, and A. AspuruGuzik. Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules. American Chemical Society Central Science , 2018. Cited on page 1.

P. C. Hansen. Rank-deficient and discrete ill-posed problems: numerical aspects of linear inversion . Society for Industrial and Applied Mathematics, 1998. Cited on page 7.

J. Hensman, N. Fusi, and N. D. Lawrence. Gaussian Processes for Big Data. In Uncertainty in Artificial Intelligence , 2013. Cited on pages 1-5, 8.

J. M. Hernández-Lobato, M. W. Hoffman, and Z. Ghahramani. Predictive Entropy Search for Efficient Global Optimization of Black-box Functions. In Advances in Neural Information Processing Systems , 2014. Cited on page 1.

J. M. Hernández-Lobato, J. Requeima, E. O. Pyzer-Knapp, and A. Aspuru-Guzik. Parallel and Distributed Thompson Sampling for Large-scale Accelerated Exploration of Chemical Space. In International Conference on Machine Learning , 2017. Cited on page 9.

R. A. Horn and C. R. Johnson. Matrix Analysis . Cambridge University Press, 2012. Cited on page 26.

B. Jin and Ž. Kereta. On the convergence of stochastic gradient descent for linear inverse problems in Banach spaces. SIAM Journal on Imaging Sciences , 2023. Cited on page 7.

S. Lang. Real and Functional Analysis . Springer, 2012. Cited on page 27.

S. Mandt, M. D. Hoffman, and D. M. Blei. Stochastic Gradient Descent as Approximate Bayesian Inference. Journal of Machine Learning Research , 2017. Cited on page 1.

A. G. d. G. Matthews, J. Hensman, R. Turner, and Z. Ghahramani. On Sparse Variational Methods and the Kullback-Leibler Divergence between Stochastic Processes. In Artificial Intelligence and Statistics , Cadiz, Spain, 2016. Cited on page 18.

A. G. d. G. Matthews, J. Hron, R. E. Turner, and Z. Ghahramani. Sample-then-optimize posterior sampling for Bayesian linear models. In NeurIPS Workshop on Advances in Approximate Bayesian Inference , 2017. Cited on page 3.

T. Pinder and D. Dodd. GPJax: A Gaussian Process Framework in JAX. Journal of Open Source Software , 2022. Cited on page 8.

W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery. Numerical Recipes: the Art of Scientific Computing . Cambridge University Press, 2007. Cited on page 3.

A. Rahimi and B. Recht. Random Features for Large-scale Kernel Machines. In Advances in Neural Information Processing Systems , 2008. Cited on page 2.

C. E. Rasmussen and C. K. Williams. Gaussian Processes for Machine Learning . MIT Press, 2006. Cited on page 2.

N. Rubens, M. Elahi, M. Sugiyama, and D. Kaplan. Active Learning in Recommender Systems. Recommender Systems Handbook , 2015. Cited on page 9.

S. Ruder. An overview of gradient descent optimization algorithms. arXiv:1609.04747 , 2016. Cited on page 1.

B. Schölkopf, R. Herbrich, and A. J. Smola. A Generalized Representer Theorem. In Computational Learning Theory , 2001. Cited on page 3.

J. Snoek, H. Larochelle, and R. P. Adams. Practical Bayesian Optimization of Machine Learning Algorithms. In Advances in Neural Information Processing Systems , 2012. Cited on page 1.

D. J. Sutherland and J. Schneider. On the error of random Fourier features. In Uncertainty in Artificial Intelligence , 2015. Cited on page 2.

A. Terenin, D. R. Burt, A. Artemev, S. Flaxman, M. van der Wilk, C. E. Rasmussen, and H. Ge. Numerically Stable Sparse Gaussian Processes via Minimum Separation using Cover Trees. Journal of Machine Learning Research , 2023. Cited on pages 3, 8.

Y. Tian, Y. Zhang, and H. Zhang. Recent Advances in Stochastic Gradient Descent in Deep Learning. Mathematics , 2023. Cited on page 1.

M. K. Titsias. Variational learning of inducing variables in sparse Gaussian processes. In Artificial Intelligence and Statistics , 2009. Cited on pages 1, 2, 4, 5, 8, 18.

M. K. Titsias. Variational model selection for sparse Gaussian process regression. Technical report, University of Manchester, 2009. Cited on page 18.

V. Vapnik. The Nature of Statistical Learning . Springer, 1995. Cited on page 2.

K. A. Wang, G. Pleiss, J. R. Gardner, S. Tyree, K. Q. Weinberger, and A. G. Wilson. Exact Gaussian Processes on a Million Data Points. In Advances in Neural Information Processing Systems , 2019. Cited on pages 3, 8, 14.

V. Wild, M. Kanagawa, and D. Sejdinovic. Connections and Equivalences between the Nyström Method and Sparse Variational Gaussian Processes. arXiv:2106.01121 , 2023. Cited on pages 5, 18.

W. J. Wilkinson, P. E. Chang, M. R. Andersen, and A. Solin. State Space Expectation Propagation: Efficient Inference Schemes for Temporal Gaussian Processes. In International Conference on Machine Learning , 2020. Cited on page 1.

A. G. Wilson and H. Nickisch. Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP). In International Conference on Machine Learning , 2015. Cited on page 1.

J. T. Wilson, V. Borovitskiy, A. Terenin, P. Mostowsky, and M. P. Deisenroth. Efficiently Sampling Functions from Gaussian Process Posteriors. In International Conference on Machine Learning , 2020. Cited on pages 2, 9.

J. T. Wilson, V. Borovitskiy, A. Terenin, P. Mostowsky, and M. P. Deisenroth. Pathwise Conditioning of Gaussian Processes. Journal of Machine Learning Research , 2021. Cited on page 2.

D. Zou, J. Wu, V. Braverman, Q. Gu, and S. M. Kakade. Benign Overfitting of ConstantStepsize SGD for Linear Regression. In Conference on Learning Theory , 2021. Cited on pages 1, 3, 7.

A Experimental Setup

We employ the CG implementation of the JAX SciPy module and follow Wang et al. [46] in using a pivoted Cholesky preconditioner of size 100 . Our preconditioner implementation resembles the implementation of the TensorFlow Probability library. For a small subset of datasets, we find the preconditioner to lead to slower convergence, and we report the results for conjugate gradients without preconditioning instead.

In addition to the experiment hyperparameters described in Section 4, for all methods, we use 2000 random Fourier features to draw each prior function used for computing posterior function samples via pathwise conditioning in (4).

For SGD, at each step, we draw 100 new Fourier features to estimate the regularizer term. In all SGD experiments, we use a Nesterov momentum value of 0 . 9 and Polyak averaging, following Antorán et al. [3]. For all regression experiments we use a learning rate of 0 . 5 to estimate the mean function representer weights, and a learning rate of 0 . 1 to draw samples. For Thompson sampling, we use a learning rate of 0 . 3 for the mean and 0 . 0003 for the samples. In both settings, we perform gradient clipping using optax.clip_by_global_norm with max_norm set to 0 . 1 .

Code available at: HTTPS://GITHUB.COM/CAMBRIDGE-MLG/SGD-GP.

A.1 Hyperparameter Selection for Regression

We use a zero prior mean function and the Matérn3 / 2 kernel, and share hyperparameters across all methods, including baselines. For each dataset, we choose a homoscedastic Gaussian noise variance, kernel variance and a separate length scale per input. For datasets with less than 50 k observations, we tune these hyperparameters to maximize the exact GP marginal likelihood. The cubic cost of this procedure makes it intractable at a larger scale: instead, for datasets with more than 50 k observations, we obtain hyperparameters using the following procedure:

From the training data, select a centroid data point uniformly at random.

Select a subset of 10 k data points with the smallest Euclidean distance to the centroid.

Find hyperparameters by maximizing the exact GP marginal likelihood using this subset.

Using 10 different centroids, repeat the preceding steps and average the hyperparameters.

This approach avoids aliasing bias [1, 5] due to data subsampling and is tractable for large datasets.

A.2 Inducing Point Selection

For the inducing point SGD experiment shown in Figure 2, we choose inducing points as a subset of the training points. Due to the large number of datapoints in our dataset, we found k -means to converge very slowly. Instead, we develop an ad-hoc point elimination algorithm based on k -nearest-neighbors (KNN). We use the KNN implementation ANNOY , which we first run on the HOUSEELECTRIC dataset with num_trees set to 50 . We then iterate through our dataset, retrieving 100 nearest neighbors for each original point. Of these 100 nearest neighbors, only the ones closer to the original point than some length scale parameter, in terms of ℓ 2 distance, are selected. If the number of points selected is larger than one, we eliminate the original point from our dataset and we also eliminate other points which are within the length scale neighborhood of the original and selected points simultaneously. The selected points are kept. We vary the number of points eliminated by our algorithm through the modification of the length scale parameter.

A.3 Maximizing Posterior Functions for Thompson Sampling

We maximize a random acquisition function sampled from the GP posterior in a three step process:

Evaluate the posterior function sample at a large number of nearby input locations. We find nearby locations using a combination of exploration and exploitation. For exploration, we sample locations uniformly at random from [0 , 1] d . For exploitation, we first subsample the training data with probabilities proportional to the observed objective function values and then add Gaussian noise ε nearby ∼ N(0 , σ 2 nearby ) , where σ nearby = l/ 2 and l is the kernel

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

1

.

0

x

-

2

-

1

0

1

2

y

Objective function

Acquisition functions

Observed values

Nearby locations

Top nearby locations

Acquisition maximizers

Optimization trace

length scale. Throughout experiments, we find 10% of nearby locations using the uniform exploration strategy and 90% using the exploitation strategy.

Select the nearby locations which have the highest acquisition function values. To find the top nearby locations, we first try 50 k nearby locations and then identify the location with the highest acquisition function value. We repeat this process 30 times, finding and evaluating a total of 1 . 5 mnearby locations and obtaining a total of 30 top nearby locations.

Maximize the acquisition function with gradient-based optimization, using the top nearby locations as initialization. After optimization, the best location becomes x new, the maximizer at which the true objective function will be evaluated in the next acquisition step. Initializing at the top nearby locations, we perform 100 steps of Adam on the sampled acquisition function, with a learning rate of 0 . 001 to find the maximizer.

In every Thompson step, we perform this three step process in parallel for 1000 random acquisition functions sampled from the GP posterior, resulting in a total of 1000 x new, which will be added to the training data and evaluated at the objective function. Note that, although we share the initial nearby locations between sampled acquisition functions, each acquisition function will, in general, produce distinct top nearby locations and maximizers. Figure 7 illustrates a single Thompson step on a 1D problem using 3 acquisition functions, 7 nearby locations and 3 top nearby locations.

B Additional Experimental Results

Figure 8 provides optimization traces for SGD, CG and their low noise variants on all datasets with under 50 k points. For these, we can compute the exact GP predictions, allowing us to evaluate error with respect to the exact GP in prediction space, representer weight space and in the RKHS. We observe the same trends reported in the main text. SGD decreases its prediction error monotonically, while CG does not. However, both take a similar amount of time to converge on smaller datasets and CG obtains lower error in the end. While SGD makes negligible progress in representer weight space, CG finds the optimal representer weights. CG's time to convergence is greatly increased in the low noise setting, but SGD is practically unaffected.

Table 2 provides quantitative results for inducing point SGD on the HOUSEELECTRIC dataset. SGD's time to convergence is shown to scale roughly linearly in the number of (inducing) points observed. However, for this dataset, keeping only 10% of observations and thus obtaining 10 × faster

0

.

0

0

.

4

0

.

8

1

.

2

pol

RMSE to Test Data

0

15

30

45

RMSE to Exact GP

0

20

Euclidean Error

400

600

0

1

RKHS Error

1

.

0

2

.

5

0

.

0

1

.

2

2

.

4

3

.

6

elevators

0

.

0

0

.

3

0

.

6

0

.

9

0

.

0

3

.

5

30

160

0

.

0

0

.

6

0

5

0

.

0

0

.

4

0

.

8

1

.

2

bike

0

.

0

0

.

6

1

.

2

1

.

8

0

100

30

160

0

.

0

1

.

5

0

.

0

1

.

5

10

-

1

10

0

10

1

10

2

10

3

Time in sec

0

.

4

1

.

0

1

.

6

2

.

2

protein

10

-

1

10

0

10

1

10

2

10

3

Time in sec

0

.

0

0

.

6

1

.

2

1

.

8

0

6

10

-

1

10

0

10

1

10

2

10

3

Time in sec

23k

25k

0

1

10

-

1

10

0

10

1

10

2

10

3

Time in sec

0

.

6

1

.

8

SGD (optimal noise)

SGD (low noise)

CG (optimal noise)

CG (low noise)

convergence leaves performance unaffected. This suggests the dataset can be summarized well by a small number of points. Indeed, SVGP obtains almost as strong performance as SGD in terms of RMSE with only 4096 inducing points. SVGP's NLL is weaker however, which is consistent with known issues of uncertainty overestimation when using a too small amount of inducing points. On the other hand, the large and potentially redundant nature of this dataset makes the corresponding optimization problem ill-conditioned, hurting CG's performance. Tables 3 and 4 provide additional baseline results for the subset of data approximation, which behaves similarly to SVGP.

Finally, Figure 9 shows all methods' Thompson sampling performance as a function of both compute time and acquisition steps for each individual true function length scale considered in our experiments. Differences among methods are more pronounced in the small compute budget setting. Here, SVGP performs well-on par with SGD-in the large length scale setting, where many observations can likely be summarized with 1024 inducing points. CG suffers from slow convergence due to illconditioning here. On the other hand, CG performs on par with SGD in the better-conditioned small length scale setting, while SVGP suffers. In the large compute setting, all methods perform similarly per acquisition step for all length scales except the small one, where SVGP suffers.

4

.

0

4

.

5

5

.

0

5

.

5

6

.

0

6

.

5

7

.

0

l

= 0

.

1

4

.

0

4

.

5

5

.

0

5

.

5

6

.

0

6

.

5

l

= 0

.

2

4

.

0

4

.

5

5

.

0

5

.

5

6

.

0

6

.

5

l

= 0

.

3

3

.

5

4

.

0

4

.

5

5

.

0

5

.

5

6

.

0

l

= 0

.

4

50k

60k

70k

80k

Acquisitions

3

.

5

4

.

0

4

.

5

5

.

0

5

.

5

l

= 0

.

5

50k

60k

70k

80k

Acquisitions

0h

1h

2h

3h

Time elapsed

0h

5h

10h

15h

Time elapsed

Small Compute Budget

Large Compute Budget

Random Search

SGD

CG

SVGP

C Inducing Points

Let z ∈ X M be a set of M > 0 inducing point locations. The variational inducing point framework of Titsias [43, 44] substitutes our observed targets y with the inducing targets u ∈ R M . Our Gaussian process, conditional on a set of inducing locations and targets, is given by the mean and covariance functions

Titsias [43] provides a closed-form expression for the variational distribution q = N( µ ( z ) , K ( z ) ) over inducing targets u which minimizes the associated Kullback-Leibler divergence, namely

In turn, Matthews et al. [30] shows that these expressions lift to a respective infinite-dimensional Kullback-Leibler minimization problem between the variational and true posterior Gaussian processes. Using this, we obtain a variational posterior with mean and covariance functions

These expressions will be our starting point. We now write this Gaussian process posterior in a pathwise manner and derive stochastic estimators for the corresponding representer weights.

C.1 Pathwise Representation of the Kullback-Leibler-optimal Variational Distribution

The pathwise expression of Section 3.3 can be derived by restricting the domain of the respective optimization problem which describes the exact posterior samples. This is done by replacing the reproducing kernel Hilbert space with a subset consisting of the span of a set of canonical basis functions centered at the inducing points, as described by Wild et al. [47], Theorem 5 for the posterior mean. Rather than deriving the respective result from scratch, we will verify its correctness by means of computing the relevant means and covariances. Define

We now calculate the moments of this Gaussian process' marginal distributions and show them to match those of the KL-optimal variational Gaussian process given in (20). Write

and

which recovers (21), as claimed.

C.2 Derivation of inducing point optimization objectives

We now derive the optimization objectives we use for inducing points.

The MAP objective (12) . To find the optimization objective for the inducing point mean function's representer weights, we apply (23) and begin from

Now, we recognize ( K zx Σ -1 K xz + K zz ) -1 K zx Σ -1 y = v ∗ as the expression for the optimizer of a ridge-regularized linear regression problem-see Bishop [7], Chapter 3-with parameters v , features K xz , Gaussian noise of covariance Σ , and regularizer K zz . This means that its respective optimization problem is

The sampling objective (13) . For the uncertainty reduction term's representer weights with inducing points, we also leverage (23), giving

This is again therefore the solution to a ridge-regularized quadratic problem, but now the targets are given by the random variable ( f ( z ) ( x ) + ε ) . We thus write the objective

C.3 On the Error in the Nyström Approximation K xz K -1 zz K zx ≈ K xx

Figure 10 compares the KL-optimal inducing point posterior GP with that obtained when applying the approximation presented in Section 3.3. That is, taking the prior function samples which we fit with the representer weighed canonical basis functions to be f ( x ) with f ∼ GP(0 , k ) instead of f ( z ) ( x ) = K xz K -1 zz f ( z ) . This amounts to approximating the Nyström-type matrix K xz K -1 zz K zx with its exact-posterior counterpart K xx . The difference between them is exactly equal to the posterior covariance: by posterior contraction, both of these matrices become very similar if there is an inducing point placed sufficiently close to every data point. In practice, this tends to occur when an inducing point is placed within roughly a half-length-scale of every observation. This is effectively what is needed for inducing point methods to provide a good approximation of the exact GP. This is reflected in Figure 10, where we see that our approximate inducing point posterior differs from the exact inducing point posterior only in situations where the latter fails to be a good approximation to the exact GP in the first place. This manifests as the approximate method providing larger error bars. When the number of inducing points increases, both methods become indistinguishable from each other and the exact GP. Fortunately, the linear cost of SGD in the number of inducing points allows us to use a very large number of these in practice.

D Low-variance Sampling Objectives

Here, we show that the modified sampling objective (10) matches the standard sample-then-optimize objective (9) up to a constant. We then compare both objectives' minibatch gradient variance.

D.1 Standard and Modified Sample-then-Optimize Objectives: Equivalent Gradients

Let LL T = Σ be the Cholesky factorization of the noise covariance, let f ( x ) ∼ N( 0 , K xx ) , and let s ∼ N( 0 , I ) . Consider the two objectives at hand, namely

-

6

-

3

0

3

6

Infill asymptotics

10 Inducing Points

15 Inducing Points

20 Inducing Points

-7

-3.5

0

3.5

7

-

4

-

2

0

2

4

Large-domain

asymptotics

1000 Inducing Points

-7

-3.5

0

3.5

7

2000 Inducing Points

-7

-3.5

0

3.5

7

3000 Inducing Points

Exact GP

Approx. Inducing GP

Inducing GP

and

We show these are equal up to a constant by showing they have the same derivatives. Taking derivative with respect to α , we have

respectively. These expressions match, giving the claim.

D.2 Minibatch Gradient Variance

Following Antorán et al. [3], we consider single sample minibatch gradient estimators for the data fit term in (7), that is D = 1 . For each of the two sampling objectives under consideration, the gradient estimators are

and

for (9) and (10), respectively. Since both objectives use the same Fourier feature estimator for the gradients of the regularizer, we omit these from our analysis. The variances for single sample gradient estimators of both expressions are given by

respectively. Expanding both expressions and subtracting them we arrive at

Whether our proposed sampling objective presents lower variance rests on the positivity of the above matrix. To analyze this, we must give concrete values to α . Following Antorán et al. [3], we consider two settings, initialization and convergence.

(I) Initialization. Here, α = 0 and therefore

is a positive definite matrix. Thus, our proposed objective has lower variance.

(II) Convergence. Here, α = ( K xx + Σ ) -1 ( f ( x ) + ε ) , so we have

Letting Σ = σ 2 I and UΛU T = K xx be the eigendecomposition of the kernel matrix, we have

Thus, at convergence, whether our proposed objective reduces variance rests on whether or not the matrix I -2 Λ ( Λ + σ 2 I ) -1 is positive definite.

Following Section 3.4, in practice, the representer weights only converge along the top few principal directions. For these, the corresponding eigenvalues will likely be much larger than the noise and thus 1 -2 λ λ + σ 2 < 0 . However, in the vast majority of directions, the opposite is true: this yields lower variance to the minibatch estimator of our proposed objective, which is shown in Figure 2.

E The Implicit Bias of SGD GPs

This section provides our main theoretical results.

E.1 Convergence of Stochastic Gradient Descent

Before studying what happens in Gaussian processes, we first prove a fairly standard result on the convergence of SGD for a quadratic objective appearing in kernel ridge regression, which represents the posterior mean. For simplicity, we do not analyze Nesterov momentum, nor aspects such as gradient clipping. We will take the Gaussian observation noise covariance to be a constant multiple of the identity in this subsection. Recall that a random variable z is called G -sub-Gaussian if E (exp( λ ( z -E ( z )))) ≤ exp( 1 2 Gλ 2 ) holds for all λ . Then, a random vector is called G -sub-Gaussian if its dot product with any unit vector is G -sub-Gaussian. One can show this condition is equivalent to having tails that are no heavier than those of a Gaussian random vector, formulated appropriately. Let UΛU T = K xx be the eigenvalue decomposition of the kernel matrix. The eigenvectors u i and eigenvalues λ i are given in descending order: λ 1 ≥ .. ≥ λ N > 0 .

Lemma 2. Let δ > 0 and Σ = σ 2 I for σ 2 > 0 . Let η be a sufficiently small learning rate of 0 < η < σ 2 λ 1 ( λ 1 + σ 2 ) . Let α ∗ ∈ R N be the solution of the respective linear system, and let α t be the Polyak-averaged SGD iterate after t steps, starting from an initial condition of α 0 = 0 . Assume that the stochastic estimate of the gradient is unbiased and G -sub-Gaussian. Then, with probability 1 -δ , we have for any I ⊆ { 1 , .., N } and all i ∈ I that

Proof. Consider the objective

and respective gradient

Let us first look at non-stochastic gradient optimization of L , without Polyak averaging. The iterate α t is given by

Writing M = ( I -η σ 2 K xx ( K xx + σ 2 I ) ) , and recalling that α 0 = 0 , we thus have that

where, for our choice of learning rate, (i) by direct calculation using simultaneous diagonalizability of M and K xx , the matrix M has eigenvalues whose absolute values are strictly less than 1, (ii) thus, the geometric series converges, (iii) and I -M is positive definite. Examining the error in direction u i , we use simultaneous diagonalizability to see that

which applies to ordinary gradient descent without stochastic gradients or Polyak averaging. Next, we consider stochastic gradient optimization. We will first consider the case where the gradient is independently perturbed by ζ t ∼ N( 0 , I ) for each step t > 0 , and then relax this to sub-Gaussian noise in the sequel. Specifically, we consider

with α ′ 0 = α 0 = 0 . For these iterates, consider the respective Polyak-averaged iterates denoted by α t = 1 t ∑ t j =1 α ′ j . We have

by expanding the Polyak averages and applying the triangle inequality, and where we have introduced the notation A i,t and B i,t . Using (63), the triangle inequality and another geometric series, we can bound the first sum as

For the second sum, note that we can re-index the order of summation to to count each ζ j only once, rather than once for each Polyak average. This gives

where u T i ζ j ∼ N(0 , 1) are IID. The variance of B i is bounded using another geometric series

Thus, from standard tail inequalities for Gaussian random variables, for any fixed δ ′ > 0 we have

We then take δ ′ = δ/ |I| , and apply the union bound for all indices in I . Finally, by comparing moment generating functions, we can relax the Gaussian assumption to G -sub-Gaussian random variables. To complete the claim, we combine the bounds for the two sums by writing

which gives the claim.

E.2 Convergence of SGD in RKHS Subspaces

The idea is to use the preceding result to show that SGD converges fast with respect to a certain seminorm. Let k be the kernel and let H k be its associated reproducing kernel Hilbert space. We now define the spectral basis functions from Section 3.4.

Definition 3. Let K xx = UΛU T be the respective eigendecomposition. Define the spectral basis functions

These are precisely the basis functions which arise in kernel principal component analysis. We also introduce two subspaces of the RKHS: the span of the representer weights, and the span of a subset of spectral basis functions. These are defined as follows.

Definition 4. Define the representer weight space

equipped with the subspace inner product.

Definition 5. Let I ⊆ { 1 , .., N } be an arbitrary set of indices. Define the interpolation subspace

We can use these subspaces to define a seminorm on the RKHS, which we will use to measure convergence rates of SGD.

Definition 6. Define the interpolation seminorm

Here, proj S f denotes orthogonal projection of a function f onto the subspace S of a Hilbert space: to ease notation, in cases where we project onto the span of a single vector, we write the vector in place of S . The first order of business is to understand the relationship between the spectral basis functions and representer weight space.

Lemma 7. The functions u ( i ) , for i = 1 , .., N , form an orthonormal basis of R k, x

̸

Proof. Let i = j . Using the reproducing property, definition of an eigenvector, the assumption that u is an eigenvector of K xx , and orthonormality of eigenvectors, we have

The fact that u ( i ) form a basis follows from the fact that they are linearly independent, that there are N of them in total, and that by definition this equals the dimension of R k, x . Finally, we calculate the norm:

Next, we compute a change-of-basis formula between the canonical basis functions and spectral basis functions.

Lemma 8. For some vector θ ∈ R N , let

Then we have

Proof. By expanding u ( j ) , we have

which since k ( x i , · ) is a basis means θ i = ∑ N j =1 w j U ij √ λ j , or in matrix-vector notation θ =

UΛ -1 / 2 w . The final claim about norms follows from orthonormality of u ( j ) by writing

This allows us to get an explicit expression for the previously-introduced seminorm.

Lemma 9. For θ ∈ R k, x , letting Λ I = diag( λ 1 1 1 ∈I , .., λ i 1 i ∈I , .., λ N 1 N ∈I ) , we have

Proof. Decompose θ in the above orthonormal basis, giving

By orthonormality, the definition of R I k, x , and properties of projections, we have

Therefore, again using orthonormality, we obtain

which gives the claim.

With this at hand, we claim that our variant of SGD converges quickly with respect to this seminorm. Proposition 10. Under the assumptions of Lemma 2, for any I we have

Proof. Using Lemma 9 and linearity, we have

where the final inequality comes from substituting in the result of Lemma 2, yielding the claim.

Our main claim follows directly from the established framework.

Proposition 1. Let δ > 0 . Let Σ = σ 2 I for σ 2 > 0 . Let µ SGD be the predictive mean obtained by Polyak-averaged SGD after t steps, starting from an initial set of representer weights equal to zero, and using a sufficiently small learning rate of 0 < η < σ 2 λ 1 ( λ 1 + σ 2 ) . Assume the stochastic estimate of the gradient is G -sub-Gaussian. Then, with probability 1 -δ , we have for i = 1 , .., N that

Proof. Choose I to be a singleton, and apply Proposition 10, where by replacing |I| with N in the final bound the claim can be made to hold with probability 1 -δ for all i = 1 , .., N simultaneously.

This concludes the first part of the argument for why SGD is a good idea: it converges fast with respect to this seminorm. In particular, taking I to be the full index set, a position-dependent pointwise convergence bound follows.

Corollary 11. Under the assumptions of Lemma 2, we have

Proof. Using Lemma 8, write

where w -w ∗ = Λ 1 / 2 U T ( v -v ∗ ) . Applying Lemma 2 with I = { 1 , .., N } , we conclude

The claim follows.

Examining the expression, the approximation error will be high at locations x ∈ X where u ( i ) ( x ) corresponding to tail eigenfunctions is large. We proceed to analyze when this occurs.

E.3 Courant-Fischer in the Reproducing Kernel Hilbert Space

Since the preceding seminorm is induced by a projection within the RKHS, the next step is to understand what kind of functions the corresponding subspace contains. To get a qualitative view, the basic idea is to lift the Courant-Fischer characterization of eigenvalues and eigenvectors to the RKHS.

We will need the following variant of the Min-Max Theorem, which is useful for describing multiple eigenvectors at once-as solutions to a sequence of Raleigh-quotient-type optimization problems, where the next optimization problem is performed on a subspace which ensures its solution is orthogonal to all previous solutions. Mirroring the rest of this work, we consider eigenvalues in decreasing order, namely λ 1 ≥ .. ≥ λ N ≥ 0 .

Result 12. Let A be a positive semi-definite matrix. Then

where the eigenvectors u i are the respective maximizers.

Proof. Horn and Johnson [26], Theorem 4.2.2, modified appropriately to handle eigenvalues in decreasing order, and where we choose i 1 , .., i k = 1 , .., N -i +1 .

We will prove the main claim below in two stages: first, on representer weight space R k, x , then on the full RKHS H k . We begin with the former.

Lemma 13. We have

Proof. Define w j = UΛ -1 / 2 u j , where we recall that u j are the respective eigenvectors of the kernel matrix. From the Result 12 form of the Courant-Fischer Min-Max Theorem, the reproducing property, and the explicit form of the RKHS inner product on R k, x in the basis defined by the canonical basis functions, we can conclude that

which gives the claim.

Next, we use a Representer-Theorem-type orthogonality argument to extend the optimization problem to all of H k . The argument will rely heavily on the Projection Theorem for Hilbert spaces: for an overview of the setting, see Lang [28], Chapter 5.

Proposition 14. We have

Proof. Let H ( i ) k be the orthogonal complement of the finite-dimensional subspace span { u ( j ) , j = 1 , .., i -1 } in H k and note by feasibility that u ( i ) ∈ H ( i ) k . Consider the decomposition of u ( i ) into its projection onto an orthogonal complement with respect to the finite-dimensional subspace H ( i ) k ∩ R k, x , namely

where this notation is defined as u ∥ = proj H ( i ) k ∩ R k, x u ( i ) and u ⊥ = u ( i ) -u ∥ . Observe that

where the first equality follows from the reproducing property, the second equality follows because u ⊥ ∈ H ( i ) k , the third equality follows because the projection is orthogonal and therefore self-adjoint, and last equality follows since u ⊥ by definition lives in the orthogonal complement of H ( i ) k ∩ R k, x within H ( i ) k , with the subspace inner product. Moreover, by the Projection Theorem and the fact that H ( i ) k inherits its norm from H k , note that

̸

We now argue by contradiction: suppose that u ⊥ = 0 . This means that ∥ u ⊥ ∥ H k > 0 . From the above, we have

Define the function

and note that ∥ u ′ ∥ = 1 , and ⟨ u ′ , u ( j ) ⟩ = 0 for all j < i since u ∥ ∈ H ( i ) k , which makes u ′ a feasible solution to the optimization problem considered. At the same time, it satisfies

which shows that u ( i ) , contrary to its definition, is not actually optimal-contradiction. We conclude that u ⊥ = 0 , which means that u ( i ) ∈ H ( i ) k ∩ R k, x , and the claim follows from Lemma 13.

This means that the top eigenvalues can be understood as the largest possible squared sums of canonical basis functions evaluated at the data, subject to RKHS-norm constraints. In situations where k ( x, · ) is continuous, non-negative, and decays as one moves away from the data, it is intuitively clear that such sums will be maximized by placing weight on canonical basis functions which cover as much of the data as possible. This mirrors the RKHS view of kernel principal component analysis.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nI thank the authors for supplying the additional experiments and making the comparisons. It appears the technique is a novel and competitive method that can perform very well even for difficult datasets in the large-scale regime. \n\nBelow is the initial review, unchanged.\n-------------------------------------------------------------------\n\nThe authors propose to sample from the posterior of a GP via optimisation using SGD. The authors rephrase sampling as a quadratic optimisation problem that allows an efficient approximation of the gradient using random kernel features. Then, SGD is used to find the solution. Theoretical derivations show tht the proposed algorithm gives good variance estimates in densely sampled areas as well as areas far outside the sampled regions. Further, the algorithm performs better than CG and SVGP in certain settings.\n\nSTRENGTHS:\nThe article provides a reltive complete package with an understandable derivation, good theoretical results and experimental evaluations.\n\nThe observations of the good performance of SGD in densely sampled regions is very interesting.\n\nWEAKNESSES:\nThe article omits subset of data (SoD) methods completely, both in the related work and the experimental section. [1] introduces them as as category, [2] establish them empirically as competitive method in sampling and [3] investigates the size of the subsampling dataset.\n\n[1] A Unifying View of Sparse Approximate Gaussian Process Regression, Qui~nonero-Candela and Rasmussen (2005) \n\n[2] A Framework for Evaluating Approximation Methods for Gaussian Process Regression, Chalupka and Murray and Williams (2013)\n\n[3] Adaptive Cholesky Gaussian Processes, Bartels et al. (2023)\n\nSubsampling methods perform similarly to the proposed method as they work well in densely sampled regions and also well in regions far away from the data distributions with only region of large error in the low sampled tails. This makes comparison mandatory.\n\nThe compairosn with SVGP is unfair due to the small number of inducing points (1024) and a bad optimisation algorithm for them (ADAM). It seems like the authors did not tune the competing methods well, making the resulting baselines weak. \n\nTheoretically, the authors omit in their complexity class evaluations the number of SGD steps, which might become very large. \nFurther, the authors omit the standard work on GP, while refering to standard notation introduced by it:\n\n[4] Gaussian Processes for Machine Learning, Carl Edward Rasmussen and Christopher K. I. Williams (2006)\n\nQUESTIONS:\nQuestion: Can you provide plots including a parameter study of SVGP including a stronger optimizer?\n\nSuggestion: I would suggest to introduce SoP method in the related work, but also compare them to the proposed method in the experimental section. I would suggest to compare them empirically to the proposed method by given them a dataset size with computation budget the same as the proposed method to allow a fair comparison on wallclock time.\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel approach for sampling from GP posteriors based on SGD, which bypasses the need to solve the typical linear system of equations that is prevalent in both the exact variant (cubic in the number of query points) or the pathwise approximation (cubic in the number of data points). The SGD approximation is provided for both exact GPs and for inducing point approximations. Moreover, the approximation quality is investigated for three regions of varying data density, and explanations for the (occasionally poor) approximation quality are given.\n\nSTRENGTHS:\n- __Novel, interesting idea:__ The use of SGD for atypical objectives is interesting, and the analysis of the approach is detailed from both a theoretical and empirical perspective.\n\n- __Clearly addressed limitations:__ The SGD approach is _not_ a silver bullet, and the authors make this clear by highlighting the approximation quality in data-dense regions (good), faraway regions (good), and interpolation regions (not as good). \n\n- __Informative, well-designed figures:__ The various figures are not only visually appealing, but provide . Figure 1 and 4 in particular highlight the strengths and shortcomings of the method nicely, and seamlessly add intuition as to why that is. \n\n- __Diverse Experiments:__ Experiments from both large-scale GP regression and BO are included, which demonstrates that the method is applicable and potent in both domains.\n\n- __Clear writing:__ The paper is consistently well-formulated, pedagogical and as far as I could tell, correct. Moreover, I beieve that there has been substantial effort to provide the reader with additional intuition for why the approach is effective.\n\nWEAKNESSES:\nI struggle to find weaknesses with the paper, but remain unconvinced on its potential impact due to the relatively small niche (sampling for GPs in large data regime) that is addressed. However, I am not overly confident in this assessment, and invite the authors to challenge my opinion on this topic. For example, do the authors see opportunities for impactful follow-up work which spans other areas of ML?\n\nQUESTIONS:\n- __Comparison to Pathwise Sampling:__ Does the proposed method hold any advantages to pathwise sampling in a low-to-moderate data regime in terms of accuracy or complexity? At which point (in #data points) does SGD start becoming beneficial?\n- __Computing the predicitve uncertainty:__ Perhaps a trivial question, but how is the predicitve uncertainty computed when one can only access the posterior mean and samples from the posterior (and not the posterior variance) in Section 4.1?\n\nLIMITATIONS:\nLimitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes SGD GP, a method based on stochastic gradient descent to efficiently compute the GP posterior samples given fixed hyperparameters. The method relies on the pathwise conditioning GP posterior formulation and random Fourier features (RFF) approximation. The key idea is to express the GP posterior quantities as solutions to quadratic optimization problems whose objective is a sum over data points and hence SGD can be applied. \n\nThe paper shows that SGD GP produce accurate predictions. However, SGD GP can converge slowly or converge to sub-optimum. However, non-convergence behaviors of SGD only occur in region closed to the data boundary. SGD GP performs comparably to SVGP and conjugate gradients (CG) in most settings, and can outperform them in large-scale systems or ill-conditioned problems.\n\nSTRENGTHS:\n- The paper is overall well-written and easy to follow. \n\n- The proposed method is novel and sound. It is shown to provide better predictive performance given the same inference time compared to SVGP and CG. What I found is most compelling is that SGD-GP seems to be a stronger alternative in large-scale or ill-conditioned systems.\n\n- I also find the spectral analysis of SGD convergence of three different regions to be very insightful.\n\nWEAKNESSES:\n- The major weakness is that the setting the paper considers is quite limited, which is the posterior inference given fixed learned hyperparameter. I would appreciate if the authors can elaborate on the significance of the setting and why the proposed methodology is in particular important. For example, how often would the ill-conditioned systems arise in practice and how common is the case that hyperparameters are known in advance. It also seems like the method is applied to a isotropic Gaussian likelihood (see the question section below). \n\n- The fact that SGD converges slowly in extrapolation region is a bit concerning. Especially in applications like Bayesian optimization, this is the region of high interests for exploration. Or in settings where there are distribution shifts, the under-calibrated uncertainty in this region can be an issue. In general, I found the \"benign non-convergence\" argument in the extrapolation region not convincing. Would appreciate if the authors can elaborate on this issue, and if if there is any potential way to alleviate the non-convergence property? \n\n- The SGD convergence analysis is insightful. But I would appreciate a more formal mathematical characterization of three regions.\n\nQUESTIONS:\n- Related to the first point above, can the method be adapted for broader settings where the hyperparameters need to be learned, or the likelihood is not Gaussian? If not, what is the limiting factor there?\n\n- Regarding the experiment evaluation, sec 4.1, is the predictive RMSE and NLL good metrics for evaluation? If I understand it correctly, SGD and CG are based on the same set of learned hyperpararmeters (and SVGP is based on a separately learned variational model). The core goal of the comparison is whether SGD recovers more faithful inference approximation compared to CG as a baseline. So I thought the valuable metric should be a distance against the \"exact\" inference result (e.g. CG with max. number of iterations and with high numeric precision). So I am not sure how to interpret Table 1. \n\n- In Fig 3 and Fig 5, do you have a sense why CG errors first go up and then go down? In Fig 5 last panel (houseelectric) would you expect CG error to match SGD performance eventually if running for more time? In both figures, I think it would be helpful to provide an exact baseline (e.g. CG ran to reach tolerance like1e-3). \n\nMinor comments. \n\n- A few notations and figures are not very clear. E.g. Fig 4. top left panel, indicate the size of error bands (blue shaded area) and dotted black line; top right corner, indicate the black dots (observations). In Proposition 1, define G-sub-gaussian.\n\nLIMITATIONS:\nThe limitation discussion is missing. I don't think the paper would have negative societal impact. My main concern on technical limitaiton is weakness point 1.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method for fast approximating a Gaussian process posterior when the data size is large. Exact computation complexity would be cubic in the data size, while this method is linear. It originates from the idea of pairwise conditioning of Gaussian process, where the law of Gaussian process posterior is expressed in terms of the law of Gaussian process prior. Decomposition into eigenfunctions gives a way of approximating the Gaussian process posterior, so the posterior inference transforms into appropriately choosing the coefficients of decomposition. Objectives are formed quadratic in the coefficients, which can be optimized with SGD for the sake of lower computational complexity compared to conjugate gradient methods. Ideas of inducing points are also discussed for reducing the data size needed. Intuitive discussion of errors in different regions are companied by figure illustrations as well as some theoretical results. Adequate numerical experiments are presented to support the method.\n\nSTRENGTHS:\n1. The paper has clear descriptions and is well written.\n2. The ideas are mostly original and combine the advantages of multiple methods.\n3. Many figure illustrations are present, making the ideas easily understood.\n4. Supportive numerical experiments are conducted.\n5. The problem of reducing the computational complexity of Gaussian process posteriors is itself very important and meaningful.\n\nWEAKNESSES:\nIt would be nice to discuss why the Fourier basis are used for eigen decomposition and how it performs compared to other basis such as wavelets.\n\nQUESTIONS:\nTo control the approximation error below a constant threshold, how does the number of basis L (number of components) needed scales with the data size and dimension? Either theoretical or numerical result could be interesting.\n\nLIMITATIONS:\nSee Weaknesses and Questions.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}