SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jihao Andreas Lin\*1,2 Javier Antoran\*1 Shreyas Padhy\*1 David Janz3 Jose Miguel Hernandez-Lobato Alexander Terenin1,4 1University of Cambridge 2Max Planck Institute for Intelligent Systems 3University of Alberta 4Cornell University
Gaussian processes are a powerful framework for quantifying uncertainty and for sequential decision-making but are limited by the requirement of solving linear systems. In general, this has a cubic cost in dataset size and is sensitive to conditioning. We explore stochastic gradient algorithms as a computationally efficient method of approximately solving these linear systems: we develop low-variance optimization objectives for sampling from the posterior and extend these to inducing points. Counterintuitively, stochastic gradient descent often produces accurate predictions, even in cases where it does not converge quickly to the optimum. We explain this through a spectral characterization of the implicit bias from non-convergence. We show that stochastic gradient descent produces predictive distributions close to the true posterior both in regions with sufficient data coverage, and in regions sufficiently far away from the data. Experimentally, stochastic gradient descent achieves state-of-the-art performance on sufficiently large-scale or ill-conditioned regression tasks. Its uncertainty estimates match the performance of significantly more expensive baselines on a large-scale Bayesian optimization task.
Gaussian processes (GPs) provide a comprehensive framework for learning unknown functions in an uncertainty-aware manner. This often makes Gaussian processes the model of choice for sequential decision-making, achieving state-of-the-art performance in tasks such as optimizing molecules in computational chemistry settings [21] and automated hyperparameter tuning [39, 24].
The main limitations of Gaussian processes is that their computational cost is cubic in the training dataset size. Significant research efforts have been directed at addressing this limitation resulting in two key classes of scalable inference methods: (i) inducing point methods [43, 23], which approximate the GP posterior, and (ii) conjugate gradient methods [20, 19, 4], which approximate the computation needed to obtain the GP posterior. Note that in structured settings, such as geospatial learning in low dimensions, specialized techniques are available [49, 48, 15]. Throughout this work, we focus on the generic setting, where scalability limitations are as of yet unresolved.
In recent years, stochastic gradient descent (SGD) has emerged as the leading technique for training machine learning models at scale [37, 42], in both deep learning and related settings like kernel methods [14] and Bayesian modeling [29]. While the principles behind the effectiveness of SGD are not yet fully understood, empirically, SGD often leads to good predictive performance—even when it does not fully converge. The latter is the default regime in deep learning, and has motivated researchers to study implicit biases and related properties of SGD [6, 52].
*Equal contribution, order chosen randomly. Code available at: HTTPS://GITHUB.COM/CAMBRIDGE-MLG/SGD-GP.
In the context of GPs, SGD is commonly used to learn kernel hyperparameters—by optimizing the marginal likelihood [18, 13, 19, 12, 11] or closely related variational objectives [43, 23]. In this work, we explore applying SGD to the complementary problem of approximating GP posterior samples given fixed kernel hyperparameters. In one of his seminal books on statistical learning theory, Vladimir Vapnik [45] famously said: "When solving a given problem, try to avoid solving a more general problem as an intermediate step." Motivated by this viewpoint, as well as the aforementioned property of good performance often not requiring full convergence when using SGD, we ask: Do the linear systems arising in  $GP$  computations necessarily need to be solved to a small error tolerance? If not, can SGD help accelerate these computations?
We answer the latter question affirmatively, with specific contributions as follows: (i) We develop a scheme for drawing GP posterior samples by applying SGD to a quadratic problem. In particular, we re-cast the pathwise conditioning technique of Wilson et al. [50, 51] as an optimization problem to which we apply the low-variance SGD sampling estimator of Antorán et al. [3]. We extend the proposed method to inducing point Gaussian processes. (ii) We characterize the implicit bias in SGD-approximated GP posteriors showing that despite optimization not fully converging, these match the true posterior in regions both near and far away from the data. (iii) Experimentally, we show that SGD produces strong results—compared to variational and conjugate gradient methods—on both large-scale and poorly-conditioned regression tasks, and on a parallel Thompson sampling task, where error bar calibration is paramount.
2 Optimization and Pathwise Conditioning in Gaussian Processes
A random function  $f: X \to \mathbb{R}$  over some set  $X$  is a Gaussian process [35] if, for every finite set of points  $\boldsymbol{x} \in X^{N}$ ,  $f(\boldsymbol{x})$  is multivariate Gaussian. A Gaussian process  $f \sim \mathrm{GP}(\mu, k)$  is uniquely determined by a mean function  $\mu(\cdot) = \mathbb{E}(f(\cdot))$  and a covariance kernel  $k(\cdot, \cdot') = \mathrm{Cov}(f(\cdot), f(\cdot'))$ . We denote by  $\mathbf{K}_{\boldsymbol{x}\boldsymbol{x}}$  the kernel matrix  $[k(x_i, x_j)]_{i,j=1,\dots,N}$ . We consider the Bayesian model  $\boldsymbol{y} = f(\boldsymbol{x}) + \varepsilon$ , where  $\varepsilon \sim \mathrm{N}(0, \Sigma)$  gives the likelihood,  $f \sim \mathrm{GP}(0, k)$  is the prior, and  $\boldsymbol{x}, \boldsymbol{y}$  are the training data. The posterior of this model is  $f \mid \boldsymbol{y} \sim \mathrm{GP}(\mu_{f|\boldsymbol{y}}, k_{f|\boldsymbol{y}})$ , with
$$
\mu_ {f | \boldsymbol {y}} (\cdot) = \mathbf {K} _ {(\cdot) \boldsymbol {x}} \left(\mathbf {K} _ {\boldsymbol {x x}} + \boldsymbol {\Sigma}\right) ^ {- 1} \boldsymbol {y} \quad k _ {f | \boldsymbol {y}} (\cdot , \cdot^ {\prime}) = \mathbf {K} _ {(\cdot , \cdot^ {\prime})} - \mathbf {K} _ {(\cdot) \boldsymbol {x}} \left(\mathbf {K} _ {\boldsymbol {x x}} + \boldsymbol {\Sigma}\right) ^ {- 1} \mathbf {K} _ {\boldsymbol {x} (\cdot^ {\prime})}. \tag {1}
$$
Using pathwise conditioning [50, 51] one can also write the posterior directly as the random function
$$
(f \mid \boldsymbol {y}) (\cdot) = f (\cdot) + \mathbf {K} _ {(\cdot) \boldsymbol {x}} \left(\mathbf {K} _ {\boldsymbol {x x}} + \boldsymbol {\Sigma}\right) ^ {- 1} (\boldsymbol {y} - f (\boldsymbol {x}) - \varepsilon) \quad \varepsilon \sim \mathrm {N} (\boldsymbol {0}, \boldsymbol {\Sigma}) \quad f \sim \mathrm {G P} (0, k). \tag {2}
$$
We assume throughout that  $\pmb{\Sigma}$  is diagonal. Due to the matrix inverses present, the cost to directly compute each of the above expressions is  $\mathcal{O}(N^3)$ .
2.1 Random Fourier Features and Efficient Sampling
Our techniques will rely on random Fourier features [34, 40]. Let  $X = \mathbb{R}^d$  and let  $k$  be stationary—that is,  $k(x,x') = k(x - x')$ . Assume, in this section only and without loss of generality, that  $k(x,x) = 1$ . Random Fourier features are sets of  $L$  random functions  $\Phi : X \to \mathbb{R}^L$  whose components, indexed by  $\ell \leq L$ , are  $\phi_{\ell}(\cdot) = L^{-1/2}\cos(2\pi \langle \omega_{\ell}, \cdot \rangle)$  for  $\ell$  odd, and  $\phi_{\ell}(\cdot) = L^{-1/2}\sin(2\pi \langle \omega_{\ell}, \cdot \rangle)$  for  $\ell$  even. By taking  $\omega_{\ell}$  to be samples distributed according to  $\rho$ , the normalized spectral measure of the kernel, for any  $x, x' \in X$  we have
$$
k \left(x, x ^ {\prime}\right) = \mathbb {E} _ {\omega_ {1}, \dots , \omega_ {L} \sim \rho} \langle \Phi (x), \Phi \left(x ^ {\prime}\right) \rangle , \tag {3}
$$
which we will use to construct unbiased estimators of kernel matrices in the sequel. Fourier features can also be used to efficiently sample functions from the GP prior. Naively, evaluating the random function  $f$  at  $N_{*}$  points requires computing a matrix square root of  $\mathbf{K}_{xx}$  at  $\mathcal{O}(N_{*}^{3})$  computational cost. However, given a precomputed set of Fourier features  $\Phi$ , the Monte Carlo estimator
$$
f (\cdot) \approx \tilde {f} (\cdot) = \boldsymbol {\theta} ^ {T} \boldsymbol {\Phi} (\cdot) = \sum_ {\ell = 1} ^ {L} \theta_ {\ell} \phi_ {\ell} (\cdot) \quad \boldsymbol {\theta} \sim \mathrm {N} (\mathbf {0}, \mathbf {I}) \tag {4}
$$
approximately samples from  $\mathrm{GP}(0,k)$  at  $\mathcal{O}(N_{*})$  cost. The approximation error in (4) decays as  $L$  goes to infinity. Following Wilson et al. [50, 51], our proposed algorithms will approximately sample from the posterior by replacing  $f$  with  $\tilde{f}$  in the pathwise conditioning formula (2).
Figure 1: Comparison of SGD, CG [46] and SVGP [23] for GP inference with a squared exponential kernel on 10k datapoints from  $\sin(2x) + \cos(5x)$  with observation noise  $\mathrm{N}(0,0.5)$ . We draw 2000 function samples with all methods by running them for 10 minutes on an RTX 2070 GPU. Infill asymptotics considers  $x_{i} \sim \mathrm{N}(0,1)$ . A large number of points near zero result in a very ill-conditioned kernel matrix, preventing CG from converging. SGD converges in all of input space except at the edges of the data. SVGP can summarise the data with only 20 inducing points. Note that CG converges to the exact solution if one uses more compute, but produces significant errors if stopped too early, as occurs under the given compute budget. Large domain asymptotics consider data on a regular grid with fixed spacing. This problem is better conditioned, allowing SGD and CG to recover the exact solution. However, 1024 inducing points are not enough for SVGP to summarize the data.
2.2 Optimization-based Learning in Gaussian Processes
Following Matthews et al. [31] and Antorán et al. [3], both a Gaussian process' posterior mean and posterior samples can be expressed as solutions to quadratic optimization problems. Letting  $\pmb{v}^{*} = (\mathbf{K}_{\pmb{x}\pmb{x}} + \pmb{\Sigma})^{-1}\pmb{y}$  in (1), we can express the GP posterior mean in terms of the quadratic problem
$$
\mu_ {f | \boldsymbol {y}} (\cdot) = \mathbf {K} _ {(\cdot) \boldsymbol {x}} \boldsymbol {v} ^ {*} = \sum_ {i = 1} ^ {N} v _ {i} ^ {*} k (x _ {i}, \cdot) \quad \boldsymbol {v} ^ {*} = \underset {\boldsymbol {v} \in \mathbb {R} ^ {N}} {\arg \min } \sum_ {i = 1} ^ {N} \frac {(y _ {i} - \mathbf {K} _ {x _ {i} \boldsymbol {x}} \boldsymbol {v}) ^ {2}}{\Sigma_ {i i}} + \| \boldsymbol {v} \| _ {\mathbf {K} _ {\boldsymbol {x x}}} ^ {2}. \tag {5}
$$
We say that  $k(x_{i},\cdot)$  are the canonical basis functions,  $v_{i}$  are the representer weights [38], and that  $\| \pmb {v}\|_{\mathbf{K}_{xx}}^2 = \pmb {v}^T\mathbf{K}_{xx}\pmb {v}$  is the regularizer. The respective optimization problem for obtaining posterior samples is similar, but involves a stochastic objective, which will be given and analyzed in Section 3.2.
Conjugate gradients (CG) are the most widely used algorithm to solve quadratic problems, both in the context of GPs [20, 19, 4] and more generally [33, 10]. CG reduces GP posterior inference to a series of matrix-vector products, each of  $\mathcal{O}(N^2)$  cost. Given the system  $\mathbf{A}^{-1}\mathbf{b}$ , the number of matrix-vector products needed to guarantee convergence of CG to within a tolerance of  $\varepsilon$  [41], is
$$
\mathcal {O} \left(\sqrt {\operatorname {c o n d} (\mathbf {A})} \log \frac {\operatorname {c o n d} (\mathbf {A}) \| \boldsymbol {b} \|}{\varepsilon}\right) \quad \operatorname {c o n d} (\mathbf {A}) = \frac {\lambda_ {\max } (\mathbf {A})}{\lambda_ {\min } (\mathbf {A})}, \tag {6}
$$
where  $\lambda_{\mathrm{max}}(\mathbf{A})$  and  $\lambda_{\mathrm{min}}(\mathbf{A})$  are the maximum and minimum eigenvalues of  $\mathbf{A}$ . CG performs well in many GP use cases, for instance Gardner et al. [19] and Wang et al. [46]. In general, however, the condition number  $\operatorname{cond}(\mathbf{K}_{xx} + \boldsymbol{\Sigma})$  need not be bounded, and conjugate gradients may fail to converge quickly [41]. Nonetheless, by exploiting the quadratic structure of the objective, CG obtains substantially better convergence rates than gradient descent [8, 52]. This presents a pessimistic outlook for gradient descent and related methods such as SGD which do not take explicit advantage of the quadratic nature of the objective. Nonetheless, the success of SGD in large-scale machine learning [9] motivates us to try it anyway.
3 Gaussian Process Predictions via Stochastic Gradient Descent
We now develop and analyze techniques for drawing samples from GP posteriors using SGD. This is done by rewriting the pathwise conditioning formula (2) in terms of two stochastic optimization problems. As a preview of what this will produce, we showcase SGD's performance on a pair of toy problems, designed to capture complementary computational difficulties, in Figure 1.
3.1 A Stochastic Objective for Computing the Posterior Mean
We begin by deriving a stochastic objective for the posterior mean. The optimization problem (5), with optimal representer weight solution  $\pmb{v}^{*}$ , requires  $\mathcal{O}(N^2)$  operations to compute both its square error and regularizer terms exactly. The square error loss term is amenable to minibatching, which gives an unbiased estimate in  $\mathcal{O}(N)$  operations. Assuming that  $k$  is stationary, following Section 2.1, we can stochastically estimate the regularizer with Fourier features using the identity  $\| \pmb{v} \|_{\mathbf{K}_{\pmb{x}\pmb{x}}}^2 = \mathbb{E}_{\omega_1,\dots,\omega_L \sim \rho} \pmb{v}^T \pmb{\Phi}^T(\pmb{x}) \pmb{\Phi}(\pmb{x}) \pmb{v}$ . Combining both estimators gives our SGD objective
$$
\frac {N}{D} \sum_ {i = 1} ^ {D} \frac {\left(y _ {i} - \mathbf {K} _ {x _ {i} \boldsymbol {x}} \boldsymbol {v}\right) ^ {2}}{\Sigma_ {i i}} + \sum_ {\ell = 1} ^ {L} \left(\boldsymbol {v} ^ {T} \phi_ {\ell} (\boldsymbol {x})\right) ^ {2} \tag {7}
$$
where  $D$  is the minibatch size and  $L$  the number of Fourier features. This regularizer estimate is unbiased even when drawing a single Fourier feature per step: the number of features controls the variance. Equation (7) presents  $\mathcal{O}(N)$  complexity, in contrast with the  $\mathcal{O}(N^2)$  complexity of one CG step. We discuss sublinear inducing point techniques further on, but first turn to sampling.
3.2 Computing Posterior Samples
We now frame GP posterior samples in a manner amenable to SGD computation similarly to (7). First, we re-write the pathwise conditioning expression given in (2) as
$$
(f \mid \boldsymbol {y}) (\cdot) = \underbrace {f (\cdot)} _ {\text {p r i o r}} + \underbrace {\mu_ {f \mid \boldsymbol {y}} (\cdot)} _ {\text {p o s t e r i o r m e a n}} - \underbrace {\mathbf {K} _ {(\cdot) \boldsymbol {x}} \left(\mathbf {K} _ {\boldsymbol {x x}} + \boldsymbol {\Sigma}\right) ^ {- 1} (f (\boldsymbol {x}) + \varepsilon)} _ {\text {u n c e r t a i n t y r e d u c t i o n t e r m}} \quad \varepsilon \sim \mathrm {N} (\boldsymbol {0}, \boldsymbol {\Sigma}) \quad f \sim \mathrm {G P} (0, k). \tag {8}
$$
We approximate the prior function sample using the Fourier feature approach of (4) and the posterior mean, defined in Section 2.2, with the minimizer  $\pmb{v}^*$  of (7) obtained by SGD. Each posterior sample's uncertainty reduction term is parametrized by a set of representer weights given by a linear solve against a noisy prior sample evaluated at the observed inputs, namely  $\pmb{\alpha}^{*} = (\mathbf{K}_{\pmb{x}\pmb{x}} + \pmb{\Sigma})^{-1}(f(\pmb{x}) + \pmb{\varepsilon})$ . We construct an optimization objective targeting a sample's optimal representer weights as
$$
\boldsymbol {\alpha} ^ {*} = \underset {\boldsymbol {\alpha} \in \mathbb {R} ^ {N}} {\arg \min } \sum_ {i = 1} ^ {N} \frac {(f (x _ {i}) + \varepsilon_ {i} - \mathbf {K} _ {x _ {i} \boldsymbol {x}} \boldsymbol {\alpha}) ^ {2}}{\Sigma_ {i i}} + \| \boldsymbol {\alpha} \| _ {\mathbf {K} _ {x x}} ^ {2} \quad \begin{array}{r l r} & & f (\boldsymbol {x}) \sim \mathrm {N} (\boldsymbol {0}, \mathbf {K} _ {x x}) \\ & & \varepsilon \sim \mathrm {N} (\boldsymbol {0}, \boldsymbol {\Sigma}). \end{array} \tag {9}
$$
Applying minibatch estimation to this objective results in high gradient variance, since the presence of  $\varepsilon_{i}$  makes the targets noisy. To avoid this while targeting the same objective, we modify (9) as
$$
\boldsymbol {\alpha} ^ {*} = \underset {\boldsymbol {\alpha} \in \mathbb {R} ^ {N}} {\arg \min } \sum_ {i = 1} ^ {N} \frac {(f (x _ {i}) - \mathbf {K} _ {x _ {i} \boldsymbol {x}} \boldsymbol {\alpha}) ^ {2}}{\Sigma_ {i i}} + \| \boldsymbol {\alpha} - \boldsymbol {\delta} \| _ {\mathbf {K} _ {x x}} ^ {2} \quad \begin{array}{r l} & f (\boldsymbol {x}) \sim \mathrm {N} (\boldsymbol {0}, \mathbf {K} _ {\boldsymbol {x x}}) \\ & \boldsymbol {\delta} \sim \mathrm {N} (\boldsymbol {0}, \boldsymbol {\Sigma} ^ {- 1}), \end{array} \tag {10}
$$
moving the noise term into the regularizer. This modification preserves the optimal representer weights since objective (10) equals (9) up to a constant: a proof is given in Appendix D. This generalizes the variance reduction technique of Antorán et al. [3] to the GP setting. Figure 2 illustrates minibatch gradient variance for these objectives. Applying the minibatch and random feature estimators of (7), we obtain a per-step cost of  $\mathcal{O}(NS)$ , for  $S$  the number of posterior samples.
3.3 Inducing Points
So far, our sampling objectives have presented linear cost in dataset size. In the large-scale setting, algorithms with costs independent of the dataset size are often preferable. For GPs, this can be achieved through inducing point posteriors [43, 23], to which we now extend SGD sampling.
Figure 2: Left: gradient variance throughout optimization for a single-sample minibatch estimator  $(D = 1)$  of (9), labeled Loss 1, and the proposed sampling objective (10), labeled Loss 2, on the ELEVATORS dataset  $(N \approx 16\mathrm{k})$ . Middle plots: test RMSE and negative log-likelihood (NLL) obtained by SGD and its inducing point variants, for decreasing numbers of inducing points, given in the rightmost plot, as a function of time on an A100 GPU, on the HOUSEELECTRIC dataset  $(N \approx 2\mathrm{M})$ .
Let  $z \in X^M$  be a set of  $M \in \mathbb{N}$  inducing points. Applying pathwise conditioning to the Kullback-Leibler-optimal inducing point approximation of Titsias [43] gives the expression
$$
\begin{array}{l} (f ^ {(z)} \mid \mathbf {y}) (\cdot) = f (\cdot) + \mu_ {f | \mathbf {y}} ^ {(z)} (\cdot) - \mathbf {K} _ {(\cdot) z} \mathbf {K} _ {z z} ^ {- 1} \mathbf {K} _ {z x} \left(\mathbf {K} _ {x z} \mathbf {K} _ {z z} ^ {- 1} \mathbf {K} _ {z x} + \boldsymbol {\Sigma}\right) ^ {- 1} (f ^ {(z)} (\boldsymbol {x}) + \varepsilon) \tag {11} \\ \varepsilon \sim \mathrm {N} (\mathbf {0}, \boldsymbol {\Sigma}) \quad f \sim \operatorname {G P} (0, k) \quad f ^ {(z)} (\cdot) = \mathbf {K} _ {(\cdot) z} \mathbf {K} _ {z z} ^ {- 1} f (z). \\ \end{array}
$$
Following Wild et al. [47], Theorem 5, the optimal inducing point mean  $\mu_{f|y}^{(z)}$  can therefore be written
$$
\mu_ {f | \boldsymbol {y}} ^ {(z)} (\cdot) = \mathbf {K} _ {(\cdot) z} \boldsymbol {v} ^ {*} = \sum_ {j = 1} ^ {M} v _ {i} ^ {*} k (z _ {j}, \cdot) \quad \boldsymbol {v} ^ {*} = \underset {\boldsymbol {v} \in \mathbb {R} ^ {M}} {\arg \min } \sum_ {i = 1} ^ {N} \frac {\left(y _ {i} - \mathbf {K} _ {x _ {i} z} \boldsymbol {v}\right) ^ {2}}{\sum_ {i i}} + \| \boldsymbol {v} \| _ {\mathbf {K} _ {z z}} ^ {2} \tag {12}
$$
and we can again parameterize the uncertainty reduction term (8) as  $\mathbf{K}_{(\cdot)\pmb{z}}\pmb{\alpha}^{*}$  with
$$
\boldsymbol {\alpha} ^ {*} = \underset {\boldsymbol {\alpha} \in \mathbb {R} ^ {M}} {\arg \min } \sum_ {i = 1} ^ {N} \frac {(f ^ {(z)} \left(x _ {i}\right) + \varepsilon_ {i} - \mathbf {K} _ {x _ {i} z} \boldsymbol {\alpha}) ^ {2}}{\Sigma_ {i i}} + \| \boldsymbol {\alpha} \| _ {\mathbf {K} _ {z z}} ^ {2} \quad \begin{array}{c} f ^ {(z)} (\boldsymbol {x}) \sim \mathrm {N} (\boldsymbol {0}, \mathbf {K} _ {x z} \mathbf {K} _ {z z} ^ {- 1} \mathbf {K} _ {z x}) \\ \varepsilon \sim \mathrm {N} (\boldsymbol {0}, \boldsymbol {\Sigma}). \end{array} \tag {13}
$$
A full derivation is given in Appendix C. Exact implementation of (13) is precluded by the need to sample from a Gaussian with covariance  $\mathbf{K}_{xz}\mathbf{K}_{zz}^{-1}\mathbf{K}_{zx}$ . However, we identify this matrix as a Nyström approximation to  $\mathbf{K}_{xx}$ . Thus, we can approximate (13) by replacing  $f^{(z)}$  with  $f$ , which can be sampled with Fourier features (4). The approximation error is small when  $M$  is large and the inducing points are close enough to the data. Our experiments in Appendix C show it to be negligible in practice. With this, we apply the stochastic estimator (7) to the inducing point sampling objective.
The inducing point objectives differ from those presented in previous sections in that there are  $\mathcal{O}(M)$  and not  $\mathcal{O}(N)$  learnable parameters, and we may choose the value of  $M$  and locations  $\mathbf{z}$  freely. The cost of inducing point representative weight updates is thus  $\mathcal{O}(SM)$ , where  $S$  is the number of samples. This contrasts with the  $\mathcal{O}(M^3)$  update cost of stochastic gradient variational Gaussian processes [23]. Figure 2 shows that SGD with  $M \approx 100\mathrm{k}$  inducing points matches the performance of regular SGD on HOUSEELECTRIC ( $N \approx 2\mathrm{M}$ ), but is an order of magnitude faster.
3.4 The Implicit Bias and Posterior Geometry of Stochastic Gradient Descent
We have detailed an SGD-based scheme for obtaining approximate samples from a posterior Gaussian process. Despite SGD's significantly lower cost per-iteration than CG, its convergence to the true optima, shown in Figure 3, is much slower with respect to both Euclidean representer weight space, and the reproducing kernel Hilbert space (RKHS) induced by the kernel. Despite this, the predictions obtained by SGD are very close to those of the exact GP, and effectively achieve the same test RMSE. Moreover, Figure 4 shows the SGD posterior on a 1D toy task exhibits error bars of the correct width close to the data, and which revert smoothly to the prior far away from the data. Empirically, differences between the SGD and exact posteriors concentrate at the borders of data-dense regions.
Figure 3: Convergence of GP posterior mean with SGD and CG as a function of time (on an A100 GPU) on the ELEVATORS dataset ( $N \approx 16\mathrm{k}$ ) while setting the noise scale to (i) maximize exact GP marginal likelihood and (ii) to  $10^{-3}$ , labeled low noise. We plot, in left-to-right order, test RMSE, RMSE to the exact GP mean at the test inputs, representer weight error  $\| \pmb{v} - \pmb{v}^{*}\|_{2}$ , and RKHS error  $\|\mu_{f|\pmb{y}} - \mu_{\mathrm{SGD}}\|_{H_k}$ . In the latter two plots, the low-noise setting is shown on the bottom.
We now argue the behavior seen in Figure 4 is a general feature of SGD: one can expect it to obtain good performance even in situations where it does not converge to the exact solution. Consider posterior function samples in pathwise form, namely  $(f\mid y)(\cdot) = f(\cdot) + \mathbf{K}_{(\cdot)x}v$ , where  $f\sim \mathrm{GP}(0,k)$  is a prior function sample and  $v$  are the learnable representative weights. We characterize the behavior of SGD-computed approximate posteriors by splitting the input space  $X$  into 3 regions, which we call the prior, interpolation, and extrapolation regions. This is done as follows.