SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Robert Allison
Bristol University
marfa@bristol.ac.uk
Anthony Stephenson
Samuel F
Alan Turing Institute
Edward Pyzer-Knapp
The accurate predictions and principled uncertainty measures provided by GP regression incur  $\mathcal{O}(n^3)$  cost which is prohibitive for modern-day large-scale applications. This has motivated extensive work on computationally efficient approximations. We introduce a new perspective by exploring robustness properties and limiting behaviour of GP nearest neighbour (GPnn) prediction. We demonstrate through theory and simulation that as the data-size  $n$  increases, accuracy of estimated parameters and GP model assumptions become increasingly irrelevant to GPnn predictive accuracy. Consequently, it is sufficient to spend small amounts of work on parameter estimation in order to achieve high MSE accuracy, even in the presence of gross misspecification. In contrast, as  $n\to \infty$ , uncertainty calibration and NLL are shown to remain sensitive to just one parameter, the additive noisevariance; but we show that this source of inaccuracy can be corrected for, thereby achieving both well-calibrated uncertainty measures and accurate predictions at remarkably low computational cost. We exhibit a very simple GPnn regression algorithm with stand-out performance compared to other state-of-the-art GP approximations as measured on large UCI datasets. It operates at a small fraction of those other methods' training costs, for example on a basic laptop taking about 30 seconds to train on a dataset of size  $n = 1.6\times 10^{6}$ .
We first briefly review the computational cost of exact GP regression and the motivation for this paper: Given  $n$  training samples  $X, \mathbf{y}$ , where  $X \in \mathbb{R}^{n \times d}$  has feature vector  $\mathbf{x}_i \in \mathbb{R}^d$  in its  $i$ 'th row and  $\mathbf{y} \in \mathbb{R}^n$ , exact GP regression [36] makes use of an  $n \times n$  gram matrix  $K = K_{X,\theta}$  constructed from a pre-specified positive definite covariance function  $c(\cdot, \cdot): \mathbb{R}^d \times \mathbb{R}^d \to \mathbb{R}_+$  together with length-scale, additive-noise variance and kernel-scale "hyperparameters"  $\pmb{\theta} = (l, \sigma_{\xi}^2, \sigma_f^2)$ . In the training phase estimates of the hyperparameters,  $\hat{\pmb{\theta}} = (\hat{l}, \hat{\sigma}_{\xi}^2, \hat{\sigma}_f^2)$ , are obtained by minimising the loss function
$$
\operatorname {l o s s} (\boldsymbol {\theta}) = - \log p (\boldsymbol {y} | X, \boldsymbol {\theta}) = \frac {1}{2} \left\{\boldsymbol {y} ^ {T} K _ {\boldsymbol {\theta}} ^ {- 1} \boldsymbol {y} + \log | K _ {\boldsymbol {\theta}} | + n \log (2 \pi) \right\}. \tag {1}
$$
Then for subsequent predictions the predictive distribution at a point  $\pmb{x}^{*} \in \mathbb{R}^{d}$  is defined by
$$
y ^ {*} \mid X, \mathbf {y} \sim \mathcal {N} \left(\mu^ {*}, \sigma^ {* 2}\right) \tag {2}
$$
$$
\mu^ {*} = \boldsymbol {k} ^ {* T} K ^ {- 1} \boldsymbol {y} \tag {3}
$$
$$
\sigma^ {* 2} = \hat {\sigma} _ {f} ^ {2} - \boldsymbol {k} ^ {* T} K ^ {- 1} \boldsymbol {k} ^ {*} + \hat {\sigma} _ {\xi} ^ {2} \tag {4}
$$
where  $K = K_{\hat{\theta}}$  with components  $[K]_{ij} = k_{\hat{\theta}}(\pmb{x}_i,\pmb{x}_j)$ ; the vector  $\pmb{k}^*$  has components  $k_i^* = k_{\hat{\theta}}(\pmb{x}_i,\pmb{x}^*)$ , and  $k_{\theta}(\pmb{x},\pmb{x}') = \sigma_f^2 c(\pmb{x} / l,\pmb{x}' / l) + \delta_{\pmb{x},\pmb{x}'}\sigma_\xi^2$  with a "normalised" covariance function  $c(\cdot ,\cdot)$  such that  $c(\pmb {x},\pmb {x}) = 1$ . The derivation of these steps is based on the assumption that the underlying random field is Gaussian, as is the additive noise with variance  $\sigma_{\xi}^{2}$ .
The single cost- $\mathcal{O}(n^3)$  step of inverting  $K$  is needed repeatedly to compute the loss. Sophisticated implementations reduce this toward  $\mathcal{O}(n^2)$  ([35]), but even that cost is generally impractical for  $n > 10^6$ . For a survey of numerous GP approximations and their reduced costs see [19].
Machine learning methods must tackle massive data problems to handle many modern day applications. Revolutionary developments in neural network methodologies achieve this, but Bayesian predictive methodologies, in particular GP regression with its major advantages of robustness and uncertainty measures, are somewhat behind the curve. This motivates development of fast implementations retaining the accuracy and well-principled uncertainty of exact GPs.
2 Background and Paper Outline
A feature common to all mainstream GP approximations is that training and prediction processes make joint use of the same underlying mathematical constructions. In the "subset-of-data" method the same subset of data is used both for parameter estimation and prediction. Similarly, in the various Bayesian committee methods ([18]) hyperparameters are estimated using a collection of subsets of data and then the same subsets are used in combination to make predictions. In the variational ([14, 31]) and other inducing point methods parameters are estimated using a low rank approximation to the kernel gram matrix and then the same low-rank matrix approximation is used to make predictions. Despite being almost universally adopted there is no obvious reason why constraining algorithms to use the same constructions for estimation and prediction will help rather than hinder the end goal of high performance at low cost. Whilst there are some passing mentions of decoupling prediction and estimation in the literature - e.g. [28, 1, 3] - it has not been adopted as a mainstream approach.
Our first observation is that allowing parameter-estimation and prediction processes to become decoupled may provide the flexibility to greatly improve cost-accuracy trade-off. As shown in Figure 1, GP approximations first obtain a point estimate of the kernel hyperparameters  $\hat{\theta}$  from training data and then feed  $\hat{\theta}$  into a predictive process. Our end-goal is only to obtain accurate and well-calibrated predictive distributions of  $y^{*}$  at each target point  $\boldsymbol{x}^{*}$ ; obtaining accurate parameter estimates is not a goal in itself. It follows that the computational budget devoted to parameter estimation need only be sufficient to provide parameters capable of delivering accurate and well-calibrated predictions.
Figure 1: Flowchart of the GP regression procedure. The dashed box indicates the usual approach of combining the parameter estimation and prediction tasks under one strategy.
This need not mean that  $\hat{\theta}$  is an accurate estimate of the parameters which leads us to the second observation and main theoretical component of this paper: In section 5, theory and simulations reveal that under widely applicable circumstances, as  $n$  increases the mean squared error (MSE) predictive accuracy obtained from GP nearest neighbour prediction becomes increasingly insensitive to model misspecification, i.e. insensitive to the wrong choice of covariance function, to the choice of  $\hat{l},\hat{\sigma}_f^2$  and  $\hat{\sigma}_{\xi}^{2}$  and even insensitive to departures from Gaussian model assumptions made for the underlying stochastic process and additive noise. Similarly the negative log likelihood (NLL) predictive-accuracy becomes insensitive to all of those factors apart from the variance of the additive noise  $\hat{\sigma}_{\xi}^{2}$ . In 6.1
we describe a simple calibration step that corrects for the latter inaccuracy thereby achieving near optimal limiting NLL values in addition to well-calibrated uncertainty measures whilst leaving the well-behaved MSE values completely unaltered. We apply these overall observations to construct a highly efficient, accurate and well calibrated regression algorithm in section 6.
Our key contributions: Demonstration of GPnn robustness against model and parameter misspecification through theory and simulation (5); derivation of explicit formulae for the limiting MSE, NLL and calibration performance of GPnn as  $n \to \infty$  (5.1); translation of this theory into a new GP approximation framework with stand-out performance relative to other state-of-the-art GP approximations (6,7.1); a simple generic method for re-calibrating uncertainty measures in GP regression with immediate applications to improving calibration of other GP approximations such as SVGP (6.1); achievement of massive scalability for GPs, for example a  $100 \times$  speed-up over state-of-the-art methods on a  $1.6 \times 10^{6}$  training set whilst also improving upon their performance (7.1); demonstrating that provably best possible MSE, NLL and calibration performance can be closely approached on data that is grossly misspecified relative to GP model assumptions (7.2).
3 Performance Measures, Weak and Strong Calibration
Along with many other GP publications we use mean squared error MSE (or its square root RMSE) and negative log likelihood (NLL), both computed from held-aside test data, to assess predictive performance. These are simply the mean values of  $e_i^* = (y_i^* - \mu_i^*)^2$  and  $l_i^* = 0.5 \cdot (\log \sigma_i^{*2} + (y_i^* - \mu_i^*)^2 / \sigma_i^{*2} + \log 2\pi)$  respectively. However, we find those measures alone inadequate for determining how well calibrated a predictive distribution is. We define "weakly calibrated" prediction to mean that  $\mathrm{E_{y^*|X,y}} \left\{ (y^* - \mu^*)^2 / \sigma^{*2} \right\} = 1$  and accordingly use "calibration" to be a measure of how well the average value of  $z_i^* = (y_i^* - \mu_i^*)^2 / \sigma_i^{*2}$  over test-data agrees with 1. This choice of metric (also made use of in [16]) can be motivated as follows: For a well-calibrated GP the expected squared deviation of  $y_i^*$  from  $\mu_i^*$  should match the corresponding predictive variance  $\sigma_i^{*2}$  (2), i.e.  $\mathrm{E_{y^*|X,y}} \left\{ z^* \right\} = 1$ . Hence, observing an average of  $z^*$  values close to 1 is consistent with a necessary condition for effective calibration. In practice, we find that GP approximation methods can fall well short of this condition (e.g. see the LHS plot in Figure 4, and Table 3 for those results in tabular form) whilst this is not evident from their MSE and NLL values alone. A better measure of calibration ("strong-calibration") would have been to see how well percentiles of the predictive distributions agree with those observed in test data, e.g. see [24], but we defer such a refinement to future work.
4 Prediction Method and Sources of Misspecification
4.1 GP Nearest Neighbour Prediction
We now describe what we mean by "GP nearest neighbour (GPnn) prediction". Assume that we are given parameters  $\hat{\pmb{\theta}} = (\hat{l},\hat{\sigma}_{\xi}^{2},\hat{\sigma}_{f}^{2})$  obtained from the parameter estimation phase of Figure 1. Then to compute the estimated pointwise-distribution of  $y^{*}$  at  $\pmb{x}^*$  indicated in Figure 1 we find the  $m$  nearest training-set neighbours  $N = N(\pmb{x}^{*})$  to  $\pmb{x}^{*}$  and apply exactly the same GP prediction formulae as in (2), (3) and (4) but with  $X\in \mathbb{R}^{n\times d}$  replaced by  $N\in \mathbb{R}^{m\times d}$  and  $\pmb{y}\in \mathbb{R}^n$  replaced by  $\pmb{y}_N\in \mathbb{R}^m$ . Note that in this setup conditioning on  $N(\pmb{x}^{*})$  is equivalent to conditioning on the full input matrix  $X$ . We obtain:
$$
y ^ {*} \mid N (\boldsymbol {x} ^ {*}), \boldsymbol {y} _ {N} \sim \mathcal {N} \left(\mu_ {N} ^ {*}, \sigma_ {N} ^ {* 2}\right) \tag {5}
$$
$$
\mu_ {N} ^ {*} = \hat {\boldsymbol {k}} _ {N} ^ {* T} \hat {\boldsymbol {K}} _ {N} ^ {- 1} \boldsymbol {y} _ {N} \tag {6}
$$
$$
\sigma_ {N} ^ {* 2} = \hat {\sigma} _ {f} ^ {2} - \hat {\boldsymbol {k}} _ {N} ^ {* T} \hat {K} _ {N} ^ {- 1} \hat {\boldsymbol {k}} _ {N} ^ {*} + \hat {\sigma} _ {\xi} ^ {2} \tag {7}
$$
where we have used hatted notation in a generic manner to cover all the potential sources of misspecification (4.2) that might arise when we carry out these predictions. The  $\mu_N^*$ ,  $\sigma_N^{*2}$  parameters are substituted for  $\mu^{*},\sigma^{*2}$  when computing the performance measures described in section 3.
Note: In this paper we use Euclidean distance for nearest neighbour assignment but more generally could employ a metric defined by the covariance function - see A. For the covariance functions used in this paper these metrics are "equivalent" because one is a monotonic function of the other.
In our algorithmic implementations we replace an exact  $m$  nearest neighbour algorithm with a much more efficient approximate nearest neighbour algorithm as discussed in section 6. However for the purpose of the theoretical analysis of robustness in section 5 this distinction can be ignored.
4.2 Sources of Misspecification
For the remainder of the paper we extend our theory and notation to encompass several (possibly simultaneous) sources of misspecification: standard GP theory assumes that data comes from a latent Gaussian random field  $\mathcal{GRF}[\sigma_f^2 c(\cdot /l,\cdot /l)]$  specified by covariance function  $c(\cdot ,\cdot)$  and parameters  $l,\sigma_f^2$ . The construction of the matrix  $K_{\theta}$  in section 1 assumes data to have arisen from this  $\mathcal{GRF}$  with i.i.d  $\mathcal{N}(0,\sigma_{\xi}^{2})$  additive noise. Henceforth, we limit covariance functions  $c(\pmb {x},\pmb{x}^{\prime})$  to be stationary, i.e. to vary only with  $(\pmb {x} - \pmb{x}^{\prime})$ . The forms of (possibly simultaneous) misspecifications to be accounted for in the theoretical treatment of 5.1 are: (a) parameter  $\sigma_{\xi}^{2}$  wrongly specified as  $\hat{\sigma}_{\xi}^{2}$ , (b) (normalised) covariance function  $c(\cdot ,\cdot)$  wrongly specified as  $\hat{c} (\cdot ,\cdot)$ , (c) parameters  $l,\sigma_f^2$  misspecified as  $\hat{l},\hat{\sigma}_f^2$  (relevant only if  $c(\cdot ,\cdot)$  not misspecified), (d) true additive noise is not Gaussian and (e) the data is generated by a non-Gaussian weakly stationary random field  $\mathcal{WSRF}$  rather than a  $\mathcal{GRF}$ .
5 GP nearest neighbour Limits and Robustness
In this section we investigate the behaviour of MSE, NLL and calibration for GPnn prediction as  $n\to \infty$  , showing how all of these performance measures become increasingly less sensitive to hyperparameter accuracy, kernel choice and the above departures from the GP model assumptions.
Assumptions: The true generative model from which the data arises is  $y_{i} = f(\pmb{x}_{i}) + \xi_{i}$  with  $\xi_{i} \stackrel{\mathrm{iid}}{\sim} P_{\xi}$ ,  $f(\pmb{x}) \sim \mathcal{WSRF}[\sigma_f^2 c(. / l, . / l)]$  and  $y_{i} \mid f(\pmb{x}_{i}) \sim P_{\xi}$  where the variance of the distribution  $P_{\xi}$  is  $\sigma_{\xi}^{2}$ . Neither the WSRF nor the additive noise distribution  $P_{\xi}$  need be Gaussian. The training  $\pmb{x}$  values are i.i.d. The MSE, NLL and calibration statistics on the test set are derived according to the nearest neighbour GP prediction process (4.1) and subject to any/all forms of misspecification in 4.2. Additionally, we assume that if and only if the  $m^{th}$  nearest neighbour converges to the test point under  $c$ , then it also converges under  $\hat{c}$  (A:Definition 10).
Result: Given a size- $n$  training set  $X$  and test point  $\pmb{x}^*$  in the support (A:Definition 9) of the measure of  $\pmb{x}$ , let  $f_{n}^{\mathrm{MSE}}(\hat{\pmb{\theta}}) = \operatorname{E}_{\mathbf{y},\mathbf{y}^{*}}\{e_{N}^{*}\}$ ,  $f_{n}^{\mathrm{NLL}}(\hat{\pmb{\theta}}) = \operatorname{E}_{\mathbf{y},\mathbf{y}^{*}}\{l_{N}^{*}\}$ ,  $f^{\mathrm{CAL}}(\hat{\pmb{\theta}}) = \operatorname{E}_{\mathbf{y},\mathbf{y}^{*}}\{z_{N}^{*}\}$ ; where expectations are w.r.t. the true generative process for  $\pmb{y}$  and  $y^{*}$  and the performance measures  $e_N^*, l_N^*, z_N^*$  (section 3) are for the nearest neighbour prediction process. Note that these are the expected (rather than mean) values of the performance measures described in section 3 and the dependence on  $n$  is implicit in the construction of the nearest neighbour sets  $N = N_{m}(\pmb{x}^{*})$  used for prediction. Then we have:
Theorem 1 (GPnn limits). As  $n \to \infty$ ,  $f_{n}^{\mathrm{MSE}}$ ,  $f_{n}^{\mathrm{CAL}}$ ,  $f_{n}^{\mathrm{NLL}} \to f_{\infty}^{\mathrm{MSE}}$ ,  $f_{\infty}^{\mathrm{CAL}}$ ,  $f_{\infty}^{\mathrm{NLL}}$  a.e w.r.t. the (i.i.d.) measure on  $\mathbf{x} \in X$  and  $\mathbf{x}^{*}$ , and pointwise as functions of  $\hat{\theta}$  where:
(i)  $f_{\infty}^{\mathrm{MSE}}(\hat{\pmb{\theta}}) = \sigma_{\xi}^{2}(1 + m^{-1})\pm \mathcal{O}\big(m^{-2}\big)$
(ii)  $f_{\infty}^{\mathrm{CAL}}(\hat{\pmb{\theta}}) = \frac{\sigma_{\xi}^{2}}{\hat{\sigma}_{\xi}^{2}}\pm \mathcal{O}\big(m^{-2}\big)$
(iii)  $f_{\infty}^{\mathrm{NLL}}(\hat{\pmb{\theta}}) = \frac{1}{2}\left\{\log \left(\hat{\sigma}_{\xi}^{2}(1 + m^{-1})\right) + \frac{\sigma_{\xi}^{2}}{\hat{\sigma}_{\xi}^{2}} + \log 2\pi \right\} \pm \mathcal{O}\big(m^{-2}\big).$
Setting  $\hat{\pmb{\theta}} = \pmb{\theta}$  (and in particular  $\hat{\sigma}_{\xi}^{2} = \sigma_{\xi}^{2}$ ) in the above provides matched-parameter limiting results.
Proof sketch. It is quite straightforward to derive expressions for each of the expectations  $f_{n}^{\mathrm{MSE}}$ ,  $f_{n}^{\mathrm{CAL}}$ ,  $f_{n}^{\mathrm{NLL}}$  since these only depend on the known marginal covariance matrices of the (misspecified)  $\mathcal{WSR}\mathcal{F}$ . We then use results concerning asymptotic convergence of Euclidean nearest neighbours, in combination with some standard linear algebra results and continuity properties, to
Figure 2: Behaviour of performance metrics as functions of kernel hyperparameters for increasing training set sizes  $n$ . The black dashed line denotes the true parameter value; the red dashed line shows the limiting behaviour as  $n \to \infty$  and the green dashed line shows the limiting behaviour when the hyperparameters are correct (the red and green dashed lines coincide for MSE). True  $l$  is shown in the title; additionally  $\sigma_{\xi}^{2} = 0.1$ ,  $\sigma_{f}^{2} = 0.9$ ,  $d = 20$ . When not varied, the assumed parameters are  $\hat{\sigma}_{\xi}^{2} = 0.2$ ,  $\hat{\sigma}_{f}^{2} = 0.8$ ,  $\hat{l} = l$ . Finally we generate the input data from the measure  $P_{\mathbf{x}} = \mathcal{N}\left(0,\frac{1}{d} I_d\right)$ .
obtain the stated limits. Note that in the expression for  $f_{\infty}^{\mathrm{MSE}}(\hat{\theta})$  above the right hand side must always exceed  $\sigma_{\xi}^{2}$  since this is an absolute lower bound on MSE performance; likewise  $f_{\infty}^{\mathrm{CAL}}(\hat{\theta})$  is constrained to be non-negative. See full proof (A).
Interpretation: The MSE results of Theorem 1 show that to within a small factor (e.g.  $m^{-1} = 0.0025$  when  $m = 400$  as for all reported runs of our algorithm) the best possible MSE will be achieved in the limit. The NLL results also tell us (by setting  $\sigma_{\xi}^{2} = \hat{\sigma}_{\xi}^{2}$ ) what the best possible limiting NLL value is, but only according to the possibly misspecified Gaussian model. The corrupting influence of an incorrect value of  $\hat{\sigma}_{\xi}^{2}$  on the limiting NLL value is clearly evident from the expression for  $f_{\infty}^{\mathrm{NLL}}$  and the picture is similar for calibration.
Remark 2. Theorem 1 shows that isotropic (e.g. RBF and Matérn) kernels converge to the best possible MSE as  $n \to \infty$  even on data generated with independent lengthscales on each  $x$  coordinate.
Note that 1 refers to pointwise convergence whereas we believe uniform convergence results should also be obtainable, e.g. perhaps of the form (or similar):
Conjecture 3.  $\operatorname{E}_{X,\mathbf{x}^*}\left\{f_n^{\mathrm{MSE}}(\hat{\pmb{\theta}})\right\} \to f_\infty^{\mathrm{MSE}} = \sigma_\xi^2 (1 + m^{-1})\pm \mathcal{O}\big(m^{-2}\big)$  uniformly as a function of  $\hat{\pmb{\theta}}$  as  $n\rightarrow \infty$
This particular conjecture would hold, for example, if the l.h.s. were shown to be a continuous function of  $\hat{\theta}$  reducing monotonically and pointwise to the limit with  $n$  (by Dini's theorem). We also have initial results on rate of convergence in Theorem 1 which we defer to a later publication once more fully extended.
5.2 Simulation of Limits and Robustness at Scale
At first sight it seems infeasible to demonstrate the above robustness and limit properties empirically on GP data-sets of size  $10^{6}$  or above. One major obstacle being the generation of GP synthetic datasets at this size which is computationally prohibitive even allowing for the speedups described in [29]. Fortunately we can avoid the need for large-scale data-generation, in addition to achieving other major efficiencies, by adopting the approach described in Algorithm 1.
The simulation algorithm gains its efficiency by exploiting the locality of the GPnn prediction process at  $\boldsymbol{x}^*$  whereby the predictive distribution only makes use of a size-  $(m + 1)$  marginal distribution of the full distribution of  $(\boldsymbol{y},\boldsymbol{y}^*)$  over  $\mathbb{R}^{n + 1}$ . (By the definition of a Gaussian process this marginal is a (low dimensional) multivariate Gaussian distribution from which samples can cheaply be generated). The following lemma is proved in Appendix B:
Algorithm 1 Simulation of GPnn Robustness and Limiting Behaviour
Input:  $n$  (training size),  $n^*$  (test size),  $m$  (number of nearest neighbours),  $d$  ( $x$ -dimension),  $c(\cdot, \cdot)$  (generative covariance function),  $\theta$  (generative kernel parameters),  $P(\cdot)$  ( $x$ -distribution)
Set-up phase A:
1. Produce the  $n$  training  $x$ -values and  $n^*$  test  $x^*$ -values by sampling independently from  $P(\cdot)$ .
2. Find the  $n^*$  size-  $m$  nearest neighbour sets  $N(\pmb{x}_i^*)$  for test points  $\pmb{x}_i^* (i = 1,\dots ,n^*)$ .
3. Generate  $n^*$  GP samples  $\pmb{y}_i\in \mathbb{R}^{m + 1}$  from  $\mathcal{N}(0,K_{U,\pmb{\theta}})$  where  $U = N(\pmb{x}_i^*)\cup \{\pmb{x}_i^*\}$ .
4. Store  $(N(\pmb{x}_i^*),\{y_{i,j}\}_{j = 1}^m)$  which will be used for predictions in Phase B together with  $y_{i}^{*} = y_{i,(m + 1)}\in \mathbb{R}$ , the true  $y$  value at  $\pmb{x}_i^*$ .
Robustness Evaluation phase: B
For several choices of assumed covariance function  $\hat{c} (\cdot ,\cdot)$  and parameters  $\hat{\theta}$
For  $i = 1,\dots ,n^{*}$  do:
1. Compute predictive distribution  $\mathcal{N}(\mu_i^*,\sigma_i^{*2})$  at  $\pmb{x}_i^*$  based on  $\hat{c},\hat{\theta}$  and  $(N(\pmb {x}_i^*),\{y_{i,j}\}_{j = 1}^m)$
2. Update NLL, MSE and calibration statistics using  $\mu_i^*$ ,  $\sigma_i^{*2}$  and  $y_{i}^{*}$ .
Output: NLL, MSE and calibration stats for range of covariance and parameter assumptions.
Lemma 4 (Algorithm 1 validity). The MSE, NLL and calibration estimates returned by Algorithm 1 are equally valid to those that would be obtained by applying the full GPnn predictive process (exactly as described in subsection 4.1) to synthetic data sets of size  $n$ .
Figure 2 shows how the observed performance metrics approach the limiting behaviour as  $n$  increases. In particular, from the RHS plot we see that as  $n$  increases, MSE becomes increasingly insensitive to departure of  $\hat{l}$  from the true value  $l = 1$ . This is a consequence of (what appears to be uniform) convergence of MSE toward constant value  $\sigma_{\xi}^{2} = 0.1$  (the best achievable MSE) as predicted by Theorem 1 (i). In practical terms: once a practitioner selects a particular kernel family, the accuracy of the hyperparameter  $\hat{l}$  becomes less and less critical to MSE predictive accuracy with increasing  $n$ , so that expenditure on estimating it accurately provides diminishing returns.