SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The interpretation of the leftmost two plots is similar albeit somewhat more involved: The dotted red lines show the asymptotic dependence on the misspecification of the noise-variance as predicted by Theorem 1 (ii) and (iii), i.e. plots of  $y = \frac{0.1}{\hat{\sigma}_{\xi}^{2}}$  and  $y = \frac{1}{2}\{\log \hat{\sigma}_{\xi}^{2} + \frac{0.1}{\hat{\sigma}_{\xi}^{2}} + \log 2\pi \}$  where  $0.1 = \sigma_{\xi}^{2}$  is the true noise value used to generate the synthetic GP data. We again see evidence of uniform convergence toward this limiting behaviour with increasing  $n$ . The green horizontal dotted lines show the limiting values of NLL and calibration ( $y = \frac{1}{2}\{\log \sigma_{\xi}^{2} + 1 + \log 2\pi \}$  and  $y = 1$  respectively) that can be achieved if the incorrect  $\hat{\sigma}_{\xi}^{2}$  value is replaced by the correct value  $\sigma_{\xi}^{2}$ . This underlines the importance of estimating this particular parameter more accurately in order to obtain improved NLL and uncertainty calibration for large  $n$ . Further plots from Algorithm 1, showing dependence of each metric on all of the parameters, are given in Figure 6.
6 A Highly Scalable GP Nearest Neighbour Regression Algorithm
6.1 Parameter Estimation
Parameter Estimation Phase 1 The first step of parameter estimation (Figure 1) involves randomly selecting a small subset  $E$  of the training data to obtain a first-pass estimate  $\hat{\theta} = (\hat{l},\hat{\sigma}_{\xi}^{2},\hat{\sigma}_{f}^{2})$ . Small subsets yield sub-optimal  $\hat{\theta}$  values, yet as shown in 7.1, these are capable of yielding strong MSE performance due to the robustness properties of section 5. We use the method in section 3.1 of [7] to estimate parameters from  $E$ , randomly partitioning  $E$  into  $w$  size-  $s$  subsets  $(ws = e)$  and using a block diagonal approximation (with  $w$  blocks of size  $s\times s$ ) to the full  $e\times e$  gram matrix. For Table 1 we set  $e = 3000$ ,  $s = 300$ ,  $m = 10$ . For strong computational efficiency we set  $e = |E|$  to a small constant value no matter the size of  $n$ . Thus as  $n$  grows, an increasingly small portion of the data is used for this phase of parameter estimation and the associated cost does not increase with  $n$ . Note that other choices of cheap parameter estimation could be substituted here.
Parameter Estimation Phase 2 (calibration) As shown in section 5, NLL and calibration performance derived from  $\hat{\pmb{\theta}} = (\hat{l},\hat{\sigma}_{\xi}^{2},\hat{\sigma}_{f}^{2})$  remain very sensitive to inaccuracies in  $\hat{\sigma}_{\xi}^{2}$ . An additional "calibration step" is used to refine those parameters: We randomly select a size  $c$  calibration set  $C$  (which is otherwise unused) from the training data and proceed according to Algorithm 2.
Algorithm 2 Calibration of Predictive Distribution
Input: A size  $c$  subset  $C$  of  $(\boldsymbol{x}^*, y^*)$  pairs from the training data, parameters  $\hat{\boldsymbol{\theta}} = (\hat{l}, \hat{\sigma}_{\xi}^{2}, \hat{\sigma}_{f}^{2})$ .
1. For each  $(\boldsymbol{x}_i^*, y_i^*) \in C$  use the efficient GPnn predictive algorithm of 6.2, with covariance function  $\hat{c}(.,.)$  and parameters  $\hat{\boldsymbol{\theta}} = (\hat{l}, \hat{\sigma}_{\xi}^{2}, \hat{\sigma}_{f}^{2})$ , to obtain an estimate of the mean and variance,  $\mu_i^*, \sigma_i^{*2}$  of the predictive distribution of  $y_i^*$  at  $\boldsymbol{x}_i^*$ .
2. Compute  $\alpha = \frac{1}{c}\sum_{i = 1}^{c}\frac{(y_i^* - \mu_i^*)^2}{\sigma_i^{*2}}.$
Output: Calibrated parameters  $\hat{\theta}^{\prime} = (\hat{l},\alpha \cdot \hat{\sigma}_{\xi}^{2},\alpha \cdot \hat{\sigma}_{f}^{2})$
Note that this process not only adjusts the noise variance estimate  $\sigma_{\xi}^{2}$  but also the kernel scale parameter  $\sigma_f^2$ . In so doing it simultaneously calibrates the predictive distribution and improves NLL performance whilst leaving unchanged the MSE performance obtained from the original parameter estimates  $\hat{\pmb{\theta}} = (\hat{l},\hat{\sigma}_{\xi}^{2},\hat{\sigma}_{f}^{2})$ . The lemma below is straightforward to prove (Appendix C):
Lemma 5 (Calibration). The parameters  $\hat{\theta}'$  output from Algorithm 2 produce GPnn predictions that (a) achieve perfect (weak) calibration on  $C$ , (b) minimise NLL on  $C$  over all choices of  $\alpha$  and (c) produce the same MSE as  $\hat{\theta}$  does on any choice of test set.
Remark 6. Algorithm 2 can be applied to other GP methods, such as SVGP, to improve calibration.
Table 1 uses  $c = 1000$ ; a simple refinement would be to select  $c$  automatically (e.g. using a bootstrap) with optional manual override. Where accurate uncertainty calibration is paramount practitioners could devote much larger CPU resources to this phase (which is also easily distributed); when no uncertainty measures are to be used this calibration step can be bypassed altogether.
6.2 Efficient Nearest Neighbour Prediction
In order to implement GPnn prediction described in 4.1 we use the scikit-learn NearestNeighbors package ([22]). This implements an efficient approximate nearest neighbour algorithm whereby one-time work is carried out to construct a table (at  $\mathcal{O}(dn\log n)$  (see e.g. [9, 21]) and counted within the total training times quoted) which subsequent calibration/test predictions then make use of. Query compute-costs are described in the associated documentation, e.g.  $\mathcal{O}(d\log n)$  for the Ball-tree algorithm which the default automated algorithm selection in SciKit-Learn should at least match. In contrast, exact kNN costs are listed in the documentation as  $\mathcal{O}(dn)$ . As is evident in Table 2, Figure 3 and the quoted query and table-setup complexities, the nearest neighbour work increases with  $x$ -dimension  $d$ . Alternative nearest neighbour algorithms and/or dimension-reduction techniques to help address this are yet to be investigated. We set the number of nearest neighbours to be  $m = 400$  for all usages in this paper having observed minimal sensitivity to this parameter on independent synthetic datasets. Although a simple cross-validation procedure could be followed if tuning of  $m$  is desired (at an increase in computational overhead), we wished to minimise such fine-tuning to emphasise the simplicity and robustness of the method we present, noting the strong performance we obtain despite this. At first glance, prediction complexity might appear restrictive, but some empirical tests on a laptop reveal comparable performance to SVGP prediction (Table 5).
7 Experimental Performance of GPnn Regression
7.1 Performance on Real World Datasets
Implementational Details<sup>1</sup>: Comparisons are made between our method and the state-of-the-art approaches of SVGP [14] and five distributed methods ([15, 2, 33, 7] and [18] following the recommendation in [4]). We have chosen not to include other highly-performant approximations
https://github.com/ant-stephenson/gpnn-experiments/
Table 1: RMSE and NLL results (mean and standard deviation over 3 runs) for the best distributed method (w.r.t. MSE), SVGP and our method.
Dataset	n	d	NLL Distributed	OURS	SVGP	RMSE Distributed	OURS	SVGP
Poletele	4.6e+03	19	0.0091 ± 0.015	-0.214 ± 0.019	-0.0667 ± 0.017	0.241 ± 0.0033	0.195 ± 0.0042	0.226 ± 0.0059
Bike	1.4e+04	13	0.977 ± 0.0057	0.953 ± 0.013	0.93 ± 0.0043	0.634 ± 0.004	0.624 ± 0.0079	0.606 ± 0.0033
Protein	3.6e+04	9	1.11 ± 0.0051	1.01 ± 0.0016	1.05 ± 0.0059	0.733 ± 0.0038	0.666 ± 0.0014	0.688 ± 0.0043
Ctslice	4.2e+04	378	-0.159 ± 0.052	-1.26 ± 0.01	0.467 ± 0.016	0.237 ± 0.012	0.132 ± 0.00062	0.384 ± 0.0064
Road3D	3.4e+05	2	0.685 ± 0.0041	0.371 ± 0.004	0.608 ± 0.018	0.478 ± 0.0023	0.351 ± 0.0014	0.443 ± 0.008
Song	4.6e+05	90	1.32 ± 0.0012	1.18 ± 0.0045	1.24 ± 0.0012	0.851 ± 6.7e-05	0.787 ± 0.0045	0.834 ± 0.0011
HouseE	1.6e+06	8	-1.34 ± 0.0013	-1.56 ± 0.0065	-1.46 ± 0.0046	0.0626 ± 5.2e-05	0.0506 ± 0.00072	0.0566 ± 0.00011
(e.g. structured kernel interpolation (SKI) methods and their extensions ([37, 39, 11])) since, to our knowledge, none have supplanted these methods in the community as ubiquitous benchmarks on general datasets. Parameters for our method are given in 6.1 and 6.2. SVGP used 1024 inducing points; the distributed methods all used randomly selected subsets of sizes as close as possible to 625. The learning rate for the Adam optimiser was 0.01 for SVGP and 0.1 for our method and distributed methods. All runs in Table 1 used the squared exponential ("RBF") covariance function. A "pre-whitening" process (E.1) was applied to x values for all methods and the y values normalised (using training data-derived means and sds) to have mean zero and variance 1. More complete details are given in E. SVGP was run on a single Tesla V100 GPU with 16GB memory; all distributed methods run on eight Intel Xeon Platinum 8000 CPUs sharing 32GB of memory. Our method was run on a MacBook Pro with  $2.4\mathrm{GHz}$  Intel core i5. See D for a full explanation of our selection and pre-processing of datasets which, apart from Protein, are taken from the UCI repository.
Results Runs were made on three randomly selected  $7/9, 2/9$  splits into training and test sets. Table 1 shows MSE and NLL results for our method alongside SVGP and distributed method (note:  $n = \text{training set size}$ ). The table shows only the best of the five distributed methods' results (w.r.t. MSE) but full results and details of all methods and all three performance measures are given in E. Complete calibration results are also plotted in Figure 4. With the exception of the Bike dataset our method is found to outperform all methods simultaneously for both MSE and NLL, and calibration likewise bar a narrow second place on the Song dataset. Table 2 and Figure 3 show that this is achieved whilst undercutting the training costs of the other methods, an effect that is very pronounced for large training sets (e.g. approximately  $100 \times$  faster than the other methods at  $n = 1.6 \times 10^6$  on House Electric). Figure 3 shows that a significant portion of time involves calibration; this can be parallelised (or eliminated if uncertainty is not required). Note also that larger timings observed for higher dimensional datasets are due to slower performance of the approximate nearest neighbour algorithm (6.2) in that regime, both for nn table construction and calibration. As discussed in 6.2, future improvements may reduce this effect. It is very interesting that "curse-of-dimensionality" has not impacted on the method's MSE, NLL or calibration competitiveness at large  $d$ . This was despite the fact that a PCA analysis of the training  $x$  values showed no concentration within a low dimensional space (as to be expected given the prewhitening that has been applied (subsection E.1).
Conjecture 7. Robustness to "curse-of-dimensionality" is at least partially explained by the increase in the intrinsic data-length-scale by a factor of order  $\sqrt{d}$  that must arise in order for GP methods to be effective.
The heuristic reasoning behind this conjecture is as follows: Unless length scale increases with  $d$  the kernel gram matrix will exhibit an abundance of exceptionally small off-diagonal entries and hence be unable able to gain significant predictive power. A  $\sqrt{d}$  increase serves to counterbalance this effect and seems consistent with length scales recovered from real data in practice.
7.2 Performance on Massive Synthetic Datasets
We generated size  $5 \times 10^{7}$  datasets using the 15-variable deterministic Oakley and O'Hagan function [20, 30] with i.i.d. variance- $\sigma_{\xi}^{2}$  additive noise sampled from a zero-mean Laplacian distribution (with much wider tails than  $\mathcal{N}(0, \sigma_{\xi}^{2})$ ). This function has 5 inputs contributing significantly to output
Table 2: Corresponding recorded training times (with mean and standard deviation from 3 runs) associated to the metrics in Table 1, i.e. recorded at the same time and with the time given for the "distributed" method relating to the best performing model in terms of MSE. Mean times are rounded to 3 s.f. and standard deviation to 2.
Dataset	n	d	Train time/s Distributed	OURS	SVGP
Poletele	4.6e+03	19	17.1 ± 0.66	28.8 ± 0.22	11.9 ± 0.081
Bike	1.4e+04	13	43.5 ± 0.64	28.4 ± 0.12	32.3 ± 0.15
Protein	3.6e+04	9	98.9 ± 1.7	27.7 ± 0.19	81.1 ± 1.1
Ctslice	4.2e+04	378	86.9 ± 1.7	76.1 ± 4.6	98.2 ± 1.8
Road3D	3.4e+05	2	1200.0 ± 110.0	27.9 ± 1.3	760.0 ± 8.0
Song	4.6e+05	90	1050.0 ± 110.0	138.0 ± 5.8	1080.0 ± 14.0
HouseE	1.6e+06	8	3110.0 ± 250.0	32.0 ± 0.34	3720.0 ± 17.0
Figure 3: Training times (s) for each model with "high" dimensional datasets highlighted.  $\dagger$  : without calibration.
Figure 4: Experiment results on a suite of UCI datasets. Optimal calibration performance is 1 (indicated by a black dashed line). Lower is better for NLL and RMSE. Y-axis truncated for readability for Calibration due to very large values on the "Ctslice" dataset. NLL is rescaled relative to the most extreme model performance. "symlog" refers to logarithmic axis rescaling applied to the  $y$ -axis on both positive and negative values ("symmetric").
variance, 5 with smaller effect, and 5 with almost no effect. These properties are poorly matched by the isotropic covariance functions being applied, resulting in gross misspecification of the assumed  $\mathcal{G}\mathcal{R}\mathcal{F}$  model and the additive noise. Figure 5 shows performance achieved with both the squared exponential ("RBF") covariance function and the exponential (Matérn 1/2) covariance function. It is very interesting to note the improvement in convergence rate achieved by the exponential kernel. (see Remark 2 for a potential explanation of why isotropic covariance functions are so effective at large  $n$ ).
Remark 8. We checked to see whether this strong exponential kernel performance extended to UCI datasets. Surprisingly, given that it is not recommended for use in  $GP$  regression (e.g. [36] page 85), it produced best RMSE performance across the board when compared with RBF and Matérn 3/2 kernels (Table 4), with Road3D RMSE reducing from 0.351 to 0.098. Exponential-kernel NLL performance was everywhere best apart from Ctslice and calibration was also better in most cases.
Related work: The basic "subset-of-data" approximation ([3]) also achieves training efficiency by using a small portion of training data and can achieve surprisingly good results ([3], [35] section 5.1, [13]). But it typically would need a much greater proportion of training data than we are using for large  $n$  due to its failure to leverage the power of large training sets for prediction; this explains why it is not consistently competitive with other methods. Passing references to the decoupling of prediction and estimation have been made, e.g. [28, 1, 3], but not shown to be as consistently powerful as we have found, nor justified in terms of robustness theory or explored as a mainstream
Figure 5: Behaviour of performance metrics on the Oakley and O'Hagan function-derived dataset ([20, 30]) as a function of data size,  $n$ . The horizontal green line indicates the limiting behaviour if the predictive and generative were to match. Shaded regions indicate  $95\%$  confidence intervals for the fitted curves.
approach. Various works (primarily from the geospatial community) make use of nearest neighbour (NN) techniques for GPs (e.g. [34, 25, 28, 6, 5, 8, 32, 38]). Vecchia ([34]) uses NNs to approximate likelihoods for parameter estimation, whilst Stein ([28]) adapts this work to REML using more distant points as well as NNs, again for parameter estimation purposes. In contrast, our focus is on using NNs for prediction, and whilst we have found passing references to its explicit use for this purpose (e.g. [27]) we have found little or no discussion of effectiveness in comparison to other methods on large datasets and no detailed accompanying analysis of its robustness properties and how these can achieve very high efficiency at scale. [5] gives a construction of a hierarchical fully Bayesian model ('NNGP') derived using collections of NN sets. This approach adds considerable computational overhead and code-complexity, e.g. use of Gibbs sampling. Whilst fully Bayesian treatment of hyperparameters is explored in the ML literature, e.g. [17], it has not been adopted by the ML community for use at scale due to its high computational cost relative to empirical Bayes methods ([17], section 5). Bearing this in mind, we consider the extension given in [8] for improved scalability (Algorithm 5 - 'conjugate NNGP') to be more relevant. In this hybrid method some hyperparameters are recovered as 'empirical-Bayes' point-estimates via grid-based search and the remainder treated in a fully Bayesian fashion using a conjugate prior with some choice of hyper-hyperparameters. This results in a Student- $t$  predictive distribution, rather than Gaussian, but with equal first moment to ours and variance differing only by a (hyper-hyperparameter dependent) multiplicative factor; an effect that our recalibration step would render redundant (see F.1). Recent work ([32, 38]) extends local geospatial GP methods into sparse variational ML applications. 'VNNGP' is shown to be competitive with other methods in [38] despite adding further approximations into the model. We note that when using all observations as the inducing points their predictive mechanism matches ours, up to choice of parameter estimation. We believe these pre-existing works, which differ significantly in approach and perspective, complement our own, which emphasizes the benefits of decoupling parameter estimation from prediction, the robustness properties that can be achieved at large scales, the efficacy of a simple recalibration step and can be run at high scale with a simple algorithm on an off-the-shelf laptop.
**Limitations and Future Research:** Our results exhibit a leap in speed and performance for GP regression at scale, but there remains more to be done to fully explain and extend performance (as evidenced by our remarks and conjectures). This is particularly so for high dimensional problems where (a) a faster nearest neighbour algorithm would have a particularly big pay-off and (b) there is a need to explain why "curse-of-dimensionality" appears not to have damaged the method's competitiveness (see Conjecture 7). Extensions of theory to broader aspects of GP robustness, rates of convergence and "strong calibration" (section 3) are current areas of some the authors' ongoing work.
We would like to thank IBM Research and EPSRC for supplying iCase funding for Anthony Stephenson and the UK National Cyber Security Centre for contributing toward Robert Allison's funding. We also thank the anonymous reviewers of this paper for their constructive comments and suggestions.
[37] A. G. Wilson and H. Nickisch. Kernel interpolation for scalable structured Gaussian processes (KISS-GP). 32nd International Conference on Machine Learning, ICML 2015, 3:1775-1784, 2015.
[38] L. Wu, G. Pleiss, and J. Cunningham. Variational Nearest Neighbor Gaussian Process. Proceedings of the 39th International Conference on Machine Learning, 2022.
[39] M. Yadav, D. Sheldon, and C. Musco. Faster kernel interpolation for gaussian processes. In A. Banerjee and K. Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 2971–2979. PMLR, 13–15 Apr 2021.
A Theoretical GPnn Results
A.1 Preliminary results
Let  $\rho (\pmb {x},\pmb{x}^{\prime}) = \sigma_{f}\sqrt{1 - c(\pmb{x} / l,\pmb{x}^{\prime} / l)}$  be the kernel-induced distance function over  $\mathbb{R}^d$  ([23]). We define  $\mathbf{x}_{(j,n)}(\pmb{x}^{*})$  as the  $j^{th}$  nearest neighbour random variable to a test point  $\pmb{x}^{*}$  under  $\rho$ , which we abbreviate to  $\mathbf{x}_{(j)}$  when the context is clear, and  $\pmb{x}_{(j)}(\pmb{x}^{*})\in N_{m}(\pmb{x}^{*})$  as the realised  $j^{th}$  nearest neighbour of the test point  $\pmb{x}^{*}$  from a training set  $X$ . From this we define  $\epsilon_{i} = \rho^{2}(\mathbf{x}_{(i)},\pmb{x}^{*})$  and  $\epsilon_{ij} = \rho^2 (\mathbf{x}_{(i)},\mathbf{x}_{(j)})$ .
Definition 9 (Support). Let  $P_{\mathbf{x}}$  be the probability measure of  $\mathbf{x}$  and  $S_{\boldsymbol{x},\epsilon}^{\rho}$  the closed ball of radius  $\epsilon > 0$  under the metric  $\rho$  centred at  $\boldsymbol{x}$ . Then we define support  $(P_{\mathbf{x}}) = \{ \boldsymbol{x} : P_{\mathbf{x}}(S_{\boldsymbol{x},\epsilon}^{\rho}) > 0 \forall \epsilon > 0 \}$ .
Definition 10 (Weakly-faithful). We define a pair of metrics  $\rho(\cdot, \cdot), \hat{\rho}(\cdot, \cdot)$  to be weakly-faithful w.r.t. each other if the following condition holds: The  $m^{th}$  nearest neighbour under  $\hat{\rho}$  converges to the test point as  $n \to \infty$  if and only if the  $m^{th}$  nearest neighbour under  $\rho$  converges to the test point in the limit.
(A1)  $\mathbf{x}^{\mathrm{ii}}\sim P_{\mathbf{x}}$  and  $\mathbf{x}^{*}\in \mathrm{support}(P_{\mathbf{x}})$  under the generative metric defined by  $c(\cdot ,\cdot)$
(A2)  $c(\cdot, \cdot), \hat{c}(\cdot, \cdot)$  are stationary kernels whose induced distance functions are weakly faithful metrics (Definition 10).
(A3)  $y_{i} = f(\pmb{x}_{i}) + \xi_{i}$  with  $\xi_{i} \stackrel{\mathrm{id}}{\sim} P_{\xi}$ ,  $f(\pmb{x}) \sim \mathcal{W}\mathcal{SRF}(\sigma_{f}^{2}c(. / l, . / l))$  and  $y_{i} \mid f(\pmb{x}_{i}) \sim P_{\xi}$  and  $\operatorname{E}[\xi] = 0$ ,  $\operatorname{E}[\xi^2] = \sigma_{\xi}^2$ .
Note: Assumption (A2) is not overly restrictive and encompasses commonly used kernels such as all those mentioned in this paper.
Lemma 11.  $\epsilon_{i}\to 0$  and  $\epsilon_{ij}\rightarrow 0$  as  $n\to \infty$  a.e. with respect to the measure over  $\mathbf{x}\in \mathbb{R}^d$ ,  $P_{\mathbf{x}}$ , for  $i,j\leq m$ ,  $\frac{m}{n}\rightarrow 0$  and under (A1-2).
Proof. Lemma 6.1 of [12] states that  $\left\| \mathbf{x}_{(m,n)}(\pmb {x}) - \pmb {x}\right\| \xrightarrow{n\to\infty}0$  with probability one (with respect to  $P_{\mathbf{x}}$ ). Their proof can be generalised immediately to state that  $\rho (\mathbf{x}_{(m,n)}(\pmb {x}),\pmb {x})\xrightarrow{n\to\infty}0$  by using our definition of support, 9, that directly invokes the metric  $\rho$ . Hence  $\epsilon_i\to 0$  for all  $i\leq m$  (since  $\pmb{x}^*$  is in support  $(P_{\mathbf{x}})$ ). Since  $\rho$  is a metric it satisfies the triangle inequality; hence  $\rho (\mathbf{x}_{(i)},\mathbf{x}_{(j)})\leq \rho (\mathbf{x}_{(i)},\pmb{x}^{*}) + \rho (\mathbf{x}_{(j)},\pmb{x}^{*})\xrightarrow{n\to\infty}0$  for all  $i,j\leq m$ .
Lemma 12. For an  $m$ -GPnn under the assumptions (A1-3),
$$
\lim _ {n \to \infty} \boldsymbol {k} _ {N} ^ {* T} K _ {N} ^ {- 1} \boldsymbol {k} _ {N} ^ {*} = \sigma_ {f} ^ {2} - \sigma_ {\xi} ^ {2} m ^ {- 1} + \mathcal {O} \big (m ^ {- 2} \big).
$$
Proof. From Lemma 11 we have that  $\lim_{n\to \infty}k(\mathbf{x}_{(j)}(\pmb{x}^{\ast}),\pmb{x}^{\ast}) = \lim_{n\to \infty}(\sigma_f^2 -\epsilon_i) = \sigma_f^2$  and  $\lim_{n\to \infty}k(\mathbf{x}_{(i)}(\pmb{x}^{\ast}),\mathbf{x}_{(j)}(\pmb{x}^{\ast})) = \lim_{n\to \infty}(\sigma_f^2 -\epsilon_{ij}) = \sigma_f^2$ . As a result,  $\pmb{k}_N^*\rightarrow \sigma_f^2\mathbf{1}$  and
$$
K ^ {\infty} := \lim  _ {n \rightarrow \infty} K _ {N} = \sigma_ {\xi} ^ {2} I + \sigma_ {f} ^ {2} \mathbf {1 1} ^ {T}. \tag {8}
$$
Now using Sherman-Morrison and the continuity of matrix inverse and matrix-matrix products:
$$
\left(A + \boldsymbol {b} \boldsymbol {c} ^ {T}\right) ^ {- 1} = A ^ {- 1} - \frac {A ^ {- 1} \boldsymbol {b} \boldsymbol {c} ^ {T} A ^ {- 1}}{1 + \boldsymbol {c} ^ {T} A ^ {- 1} \boldsymbol {b}} \tag {9}
$$
$$
\left. \left(K ^ {\infty}\right) ^ {- 1} = \left(\sigma_ {\xi} ^ {2} I + \sigma_ {f} ^ {2} \mathbf {1 1} ^ {T}\right) ^ {- 1} = \frac {1}{\sigma_ {\xi} ^ {2}} \left(I - \sigma_ {f} ^ {2} \frac {\mathbf {1 1} ^ {T}}{\sigma_ {\xi} ^ {2} + \sigma_ {f} ^ {2} \mathbf {1} ^ {T} \mathbf {1}}\right) \right. \tag {10}
$$