Title: Leveraging Locality and Robustness to Achieve Massively Scalable Gaussian Process Regression
Venue: NeurIPS 2023
Accepted: yes
Keywords: Gaussian Processes

Bayesian Inference

Regression

Bayesian Nonparametrics

Kernel Methods
Abstract: The accurate predictions and principled uncertainty measures provided by GP regression incur $O(n^3)$ cost which is prohibitive for modern-day large-scale applications. This has motivated extensive work on computationally efficient approximations. We introduce a new perspective by exploring robustness properties and limiting behaviour of GP nearest-neighbour (GPnn) prediction. We demonstrate through theory and simulation that as the data-size $n$ increases, accuracy of estimated parameters and GP model assumptions become increasingly irrelevant to GPnn predictive accuracy. Consequently, it is sufficient to spend small amounts of work on parameter estimation in order to achieve high MSE accuracy, even in the presence of gross misspecification. In contrast, as $n \rightarrow \infty$, uncertainty calibration and NLL are shown to remain sensitive to just one parameter, the additive noise-variance; but we show that this source of inaccuracy can be corrected for, thereby achieving both well-calibrated uncertainty measures and accurate predictions at remarkably low computational cost. We exhibit a very simple GPnn regression algorithm with stand-out performance compared to other state-of-the-art GP approximations as measured on large UCI datasets. It operates at a small fraction of those other methods' training costs, for example on a basic laptop taking about 30 seconds to train on a dataset of size $n = 1.6 \times 10^6$.

Leveraging Locality and Robustness to Achieve Massively Scalable Gaussian Process Regression

Abstract

The accurate predictions and principled uncertainty measures provided by GP regression incur O ( n 3 ) cost which is prohibitive for modern-day large-scale applications. This has motivated extensive work on computationally efficient approximations. We introduce a new perspective by exploring robustness properties and limiting behaviour of GP nearest neighbour (GPnn) prediction. We demonstrate through theory and simulation that as the data-size n increases, accuracy of estimated parameters and GP model assumptions become increasingly irrelevant to GPnn predictive accuracy. Consequently, it is sufficient to spend small amounts of work on parameter estimation in order to achieve high MSE accuracy, even in the presence of gross misspecification. In contrast, as n →∞ , uncertainty calibration and NLL are shown to remain sensitive to just one parameter, the additive noisevariance; but we show that this source of inaccuracy can be corrected for, thereby achieving both well-calibrated uncertainty measures and accurate predictions at remarkably low computational cost. We exhibit a very simple GPnn regression algorithm with stand-out performance compared to other state-of-the-art GP approximations as measured on large UCI datasets. It operates at a small fraction of those other methods' training costs, for example on a basic laptop taking about 30 seconds to train on a dataset of size n = 1 . 6 × 10 6 .

1 Introduction

We first briefly review the computational cost of exact GP regression and the motivation for this paper: Given n training samples X, y , where X ∈ R n × d has feature vector x i ∈ R d in its i 'th row and y ∈ R n , exact GP regression [36] makes use of an n × n gram matrix K = K X, θ constructed from a pre-specified positive definite covariance function c ( · , · ) : R d × R d → R + together with length-scale, additive-noise variance and kernel-scale 'hyperparameters' θ = ( l, σ 2 ξ , σ 2 f ) . In the training phase estimates of the hyperparameters, ˆ θ = ( ˆ l, ˆ σ 2 ξ , ˆ σ 2 f ) , are obtained by minimising the loss function

Then for subsequent predictions the predictive distribution at a point x ∗ ∈ R d is defined by

where K = K ˆ θ with components [ K ] ij = k ˆ θ ( x i , x j ) ; the vector k ∗ has components k ∗ i = k ˆ θ ( x i , x ∗ ) , and k θ ( x , x ′ ) = σ 2 f c ( x /l, x ′ /l ) + δ x , x ′ σ 2 ξ with a 'normalised' covariance function c ( · , · ) such that c ( x , x ) = 1 . The derivation of these steps is based on the assumption that the underlying random field is Gaussian, as is the additive noise with variance σ 2 ξ .

The single costO ( n 3 ) step of inverting K is needed repeatedly to compute the loss. Sophisticated implementations reduce this toward O ( n 2 ) ([35]), but even that cost is generally impractical for n > 10 6 . For a survey of numerous GP approximations and their reduced costs see [19].

Machine learning methods must tackle massive data problems to handle many modern day applications. Revolutionary developments in neural network methodologies achieve this, but Bayesian predictive methodologies, in particular GP regression with its major advantages of robustness and uncertainty measures, are somewhat behind the curve. This motivates development of fast implementations retaining the accuracy and well-principled uncertainty of exact GPs.

2 Background and Paper Outline

Afeature common to all mainstream GP approximations is that training and prediction processes make joint use of the same underlying mathematical constructions. In the 'subset-of-data' method the same subset of data is used both for parameter estimation and prediction. Similarly, in the various Bayesian committee methods ([18]) hyperparameters are estimated using a collection of subsets of data and then the same subsets are used in combination to make predictions. In the variational ([14, 31]) and other inducing point methods parameters are estimated using a low rank approximation to the kernel gram matrix and then the same low-rank matrix approximation is used to make predictions. Despite being almost universally adopted there is no obvious reason why constraining algorithms to use the same constructions for estimation and prediction will help rather than hinder the end goal of high performance at low cost. Whilst there are some passing mentions of decoupling prediction and estimation in the literature - e.g. [28, 1, 3] - it has not been adopted as a mainstream approach.

Our first observation is that allowing parameter-estimation and prediction processes to become decoupled may provide the flexibility to greatly improve cost-accuracy trade-off. As shown in Figure 1, GP approximations first obtain a point estimate of the kernel hyperparameters ˆ θ from training data and then feed ˆ θ into a predictive process. Our end-goal is only to obtain accurate and well-calibrated predictive distributions of y ∗ at each target point x ∗ ; obtaining accurate parameter estimates is not a goal in itself. It follows that the computational budget devoted to parameter estimation need only be sufficient to provide parameters capable of delivering accurate and wellcalibrated predictions.

Parameter estimation

Training data

(

X,

y

)

Prediction

Training data

(

X,

y

)

Target point

x

∗

Distribution of

y

∗

|

X,

y

at

x

∗

ˆ

θ

This need not mean that ˆ θ is an accurate estimate of the parameters which leads us to the second observation and main theoretical component of this paper: In section 5, theory and simulations reveal that under widely applicable circumstances, as n increases the mean squared error (MSE) predictive accuracy obtained from GP nearest neighbour prediction becomes increasingly insensitive to model misspecification, i.e. insensitive to the wrong choice of covariance function, to the choice of ˆ l , ˆ σ 2 f and ˆ σ 2 ξ and even insensitive to departures from Gaussian model assumptions made for the underlying stochastic process and additive noise. Similarly the negative log likelihood (NLL) predictive-accuracy becomes insensitive to all of those factors apart from the variance of the additive noise ˆ σ 2 ξ . In 6.1

we describe a simple calibration step that corrects for the latter inaccuracy thereby achieving near optimal limiting NLL values in addition to well-calibrated uncertainty measures whilst leaving the well-behaved MSE values completely unaltered. We apply these overall observations to construct a highly efficient, accurate and well calibrated regression algorithm in section 6.

Our key contributions: Demonstration of GPnn robustness against model and parameter misspecification through theory and simulation (5); derivation of explicit formulae for the limiting MSE, NLL and calibration performance of GPnn as n →∞ (5.1); translation of this theory into a new GP approximation framework with stand-out performance relative to other state-of-the-art GP approximations (6,7.1); a simple generic method for re-calibrating uncertainty measures in GP regression with immediate applications to improving calibration of other GP approximations such as SVGP (6.1); achievement of massive scalability for GPs, for example a 100 × speed-up over state-of-the-art methods on a 1 . 6 × 10 6 training set whilst also improving upon their performance (7.1); demonstrating that provably best possible MSE, NLL and calibration performance can be closely approached on data that is grossly misspecified relative to GP model assumptions (7.2).

3 Performance Measures, Weak and Strong Calibration

Along with many other GP publications we use mean squared error MSE (or its square root RMSE) and negative log likelihood (NLL), both computed from held-aside test data, to assess predictive performance. These are simply the mean values of e ∗ i = ( y ∗ i -µ ∗ i ) 2 and l ∗ i = 0 . 5 · (log σ ∗ 2 i +( y ∗ i -µ ∗ i ) 2 /σ ∗ 2 i +log2 π ) respectively. However, we find those measures alone inadequate for determining how well calibrated a predictive distribution is. We define 'weakly calibrated' prediction to mean that E y ∗ | X, y { ( y ∗ -µ ∗ ) 2 /σ ∗ 2 } = 1 and accordingly use 'calibration' to be a measure of how well the average value of z ∗ i = ( y ∗ i -µ ∗ i ) 2 /σ ∗ 2 i over test-data agrees with 1. This choice of metric (also made use of in [16]) can be motivated as follows: For a well-calibrated GP the expected squared deviation of y ∗ i from µ ∗ i should match the corresponding predictive variance σ ∗ i 2 (2), i.e. E y ∗ | X, y { z ∗ } = 1 . Hence, observing an average of z ∗ values close to 1 is consistent with a necessary condition for effective calibration. In practice, we find that GP approximation methods can fall well short of this condition (e.g. see the LHS plot in Figure 4, and Table 3 for those results in tabular form) whilst this is not evident from their MSE and NLL values alone. A better measure of calibration ('strong-calibration') would have been to see how well percentiles of the predictive distributions agree with those observed in test data, e.g. see [24], but we defer such a refinement to future work.

4 Prediction Method and Sources of Misspecification

4.1 GP Nearest Neighbour Prediction

We now describe what we mean by 'GP nearest neighbour (GPnn) prediction'. Assume that we are given parameters ˆ θ = ( ˆ l, ˆ σ 2 ξ , ˆ σ 2 f ) obtained from the parameter estimation phase of Figure 1. Then to compute the estimated pointwise-distribution of y ∗ at x ∗ indicated in Figure 1 we find the m nearest training-set neighbours N = N ( x ∗ ) to x ∗ and apply exactly the same GP prediction formulae as in (2), (3) and (4) but with X ∈ R n × d replaced by N ∈ R m × d and y ∈ R n replaced by y N ∈ R m . Note that in this setup conditioning on N ( x ∗ ) is equivalent to conditioning on the full input matrix X . We obtain:

where we have used hatted notation in a generic manner to cover all the potential sources of misspecification (4.2) that might arise when we carry out these predictions. The µ ∗ N , σ ∗ 2 N parameters are substituted for µ ∗ , σ ∗ 2 when computing the performance measures described in section 3.

Note: In this paper we use Euclidean distance for nearest neighbour assignment but more generally could employ a metric defined by the covariance function - see A. For the covariance functions used in this paper these metrics are 'equivalent' because one is a monotonic function of the other.

In our algorithmic implementations we replace an exact m nearest neighbour algorithm with a much more efficient approximate nearest neighbour algorithm as discussed in section 6. However for the purpose of the theoretical analysis of robustness in section 5 this distinction can be ignored.

4.2 Sources of Misspecification

For the remainder of the paper we extend our theory and notation to encompass several (possibly simultaneous) sources of misspecification: standard GP theory assumes that data comes from a latent Gaussian random field GRF [ σ 2 f c ( ./l, ./l )] specified by covariance function c ( · , · ) and parameters l, σ 2 f . The construction of the matrix K θ in section 1 assumes data to have arisen from this GRF with i.i.d N (0 , σ 2 ξ ) additive noise. Henceforth, we limit covariance functions c ( x , x ′ ) to be stationary, i.e. to vary only with ( x -x ′ ) . The forms of (possibly simultaneous) misspecifications to be accounted for in the theoretical treatment of 5.1 are: (a) parameter σ 2 ξ wrongly specified as ˆ σ 2 ξ , (b) (normalised) covariance function c ( · , · ) wrongly specified as ˆ c ( · , · ) , (c) parameters l, σ 2 f misspecified as ˆ l, ˆ σ 2 f (relevant only if c ( · , · ) not misspecified), (d) true additive noise is not Gaussian and (e) the data is generated by a non-Gaussian weakly stationary random field WSRF rather than a GRF .

5 GP nearest neighbour Limits and Robustness

In this section we investigate the behaviour of MSE, NLL and calibration for GPnn prediction as n → ∞ , showing how all of these performance measures become increasingly less sensitive to hyperparameter accuracy, kernel choice and the above departures from the GP model assumptions.

5.1 Theory

Assumptions: The true generative model from which the data arises is y i = f ( x i ) + ξ i with ξ i iid ∼ P ξ , f ( x ) ∼ WSRF [ σ 2 f c ( ./l, ./l ))] and y i | f ( x i ) ∼ P ξ where the variance of the distribution P ξ is σ 2 ξ . Neither the WSRF nor the additive noise distribution P ξ need be Gaussian. The training x values are i.i.d. The MSE, NLL and calibration statistics on the test set are derived according to the nearest neighbour GP prediction process (4.1) and subject to any/all forms of misspecification in 4.2. Additionally, we assume that if and only if the m th nearest neighbour converges to the test point under c , then it also converges under ˆ c (A:Definition 10).

Result: Given a sizen training set X and test point x ∗ in the support (A:Definition 9) of the measure of x , let f MSE n ( ˆ θ ) = E y , y ∗ { e ∗ N } , f NLL n ( ˆ θ ) = E y , y ∗ { l ∗ N } , f CAL ( ˆ θ ) = E y , y ∗ { z ∗ N } ; where expectations are w.r.t. the true generative process for y and y ∗ and the performance measures e ∗ N , l ∗ N , z ∗ N (section 3) are for the nearest neighbour prediction process. Note that these are the expected (rather than mean) values of the performance measures described in section 3 and the dependence on n is implicit in the construction of the nearest neighbour sets N = N m ( x ∗ ) used for prediction. Then we have:

Theorem 1 (GPnn limits) . As n → ∞ , f MSE n , f CAL n , f NLL n → f MSE ∞ , f CAL ∞ , f NLL ∞ a.e w.r.t. the (i.i.d.) measure on x ∈ X and x ∗ , and pointwise as functions of ˆ θ where:

Setting ˆ θ = θ (and in particular ˆ σ 2 ξ = σ 2 ξ ) in the above provides matched-parameter limiting results.

Proof sketch. It is quite straightfoward to derive expressions for each of the expectations f MSE n , f CAL n , f NLL n since these only depend on the known marginal covariance matrices of the (misspecified) WSRF . We then use results concerning asymptotic convergence of Euclidean nearest neighbours, in combination with some standard linear algebra results and continuity properties, to

0.0

0.5

1.0

2

0.00

0.25

0.50

0.75

1.00

1.25

NLL (

l

=0.5)

0.0

0.5

1.0

2

0.0

0.5

1.0

1.5

2.0

Calibration (

l

=0.5)

0

1

2

3

l

0.00

0.25

0.50

0.75

1.00

1.25

1.50

MSE (

l

=1.0)

n

1e+04

1e+05

1e+06

1e+07

f

(

)

f

(

)

obtain the stated limits. Note that in the expression for f MSE ∞ ( ˆ θ ) above the right hand side must always exceed σ 2 ξ since this is an absolute lower bound on MSE performance; likewise f CAL ∞ ( ˆ θ ) is constrained to be non-negative. See full proof (A). .

Interpretation: The MSE results of Theorem 1 show that to within a small factor (e.g. m -1 = 0 . 0025 when m = 400 as for all reported runs of our algorithm) the best possible MSE will be achieved in the limit. The NLL results also tell us (by setting σ 2 ξ = ˆ σ 2 ξ ) what the best possible limiting NLL value is, but only according to the possibly misspecified Gaussian model. The corrupting influence of an incorrect value of ˆ σ 2 ξ on the limiting NLL value is clearly evident from the expression for f NLL ∞ and the picture is similar for calibration.

Remark 2. Theorem 1 shows that isotropic (e.g. RBF and Matérn) kernels converge to the best possible MSE as n →∞ even on data generated with independent lengthscales on each x coordinate.

Note that 1 refers to pointwise convergence whereas we believe uniform convergence results should also be obtainable, e.g. perhaps of the form (or similar):

This particular conjecture would hold, for example, if the l.h.s. were shown to be a continuous function of ˆ θ reducing monotonically and pointwise to the limit with n (by Dini's theorem). We also have initial results on rate of convergence in Theorem 1 which we defer to a later publication once more fully extended.

5.2 Simulation of Limits and Robustness at Scale

At first sight it seems infeasible to demonstrate the above robustness and limit properties empirically on GP data-sets of size 10 6 or above. One major obstacle being the generation of GP synthetic datasets at this size which is computationally prohibitive even allowing for the speedups described in [29]. Fortunately we can avoid the need for large-scale data-generation, in addition to achieving other major efficiencies, by adopting the approach described in Algorithm 1.

The simulation algorithm gains its efficiency by exploiting the locality of the GPnn prediction process at x ∗ whereby the predictive distribution only makes use of a size-( m +1) marginal distribution of the full distribution of ( y , y ∗ ) over R n +1 . (By the definition of a Gaussian process this marginal is a (low dimensional) multivariate Gaussian distribution from which samples can cheaply be generated). The following lemma is proved in Appendix B:

Algorithm 1 Simulation of GPnn Robustness and Limiting Behaviour

Input: n (training size), n ∗ (test size), m (number of nearest neighbours), d ( x -dimension), c ( · , · ) (generative covariance function), θ (generative kernel parameters), P ( . ) ( x -distribution)

Set-up phase A:

Produce the n training x -values and n ∗ test x ∗ -values by sampling independently from P ( . ) .

Find the n ∗ sizem nearest neighbour sets N ( x ∗ i ) for test points x ∗ i ( i = 1 , . . . , n ∗ ).

Generate n ∗ GP samples y i ∈ R m +1 from N (0 , K U, θ ) where U = N ( x ∗ i ) ∪ { x ∗ i } .

Store ( N ( x ∗ i ) , { y i,j } j m =1 ) which will be used for predictions in Phase B together with y ∗ i = y i, ( m +1) ∈ R , the true y value at x ∗ i .

Robustness Evaluation phase: B

For several choices of assumed covariance function ˆ c ( · , · ) and parameters ˆ θ : For i = 1 , . . . , n ∗ do:

Compute predictive distribution N ( µ ∗ i , σ ∗ 2 i ) at x ∗ i based on ˆ c , ˆ θ and ( N ( x ∗ i ) , { y i,j } j m =1 ) .

Update NLL, MSE and calibration statistics using µ ∗ i , σ ∗ 2 i and y ∗ i .

Output: NLL, MSE and calibration stats for range of covariance and parameter assumptions.

Lemma 4 (Algorithm 1 validity) . The MSE, NLL and calibration estimates returned by Algorithm 1 are equally valid to those that would be obtained by applying the full GPnn predictive process (exactly as described in subsection 4.1) to synthetic data sets of size n .

Figure 2 shows how the observed performance metrics approach the limiting behaviour as n increases. In particular, from the RHS plot we see that as n increases, MSE becomes increasingly insensitive to departure of ˆ l from the true value l = 1 . This is a consequence of (what appears to be uniform) convergence of MSE toward constant value σ 2 ξ = 0 . 1 (the best achievable MSE) as predicted by Theorem 1 (i). In practical terms: once a practitioner selects a particular kernel family, the accuracy of the hyperparameter ˆ l becomes less and less critical to MSE predictive accuracy with increasing n , so that expenditure on estimating it accurately provides diminishing returns.

The interpretation of the leftmost two plots is similar albeit somewhat more involved: The dotted red lines show the asymptotic dependence on the misspecification of the noise-variance as predicted by Theorem 1 (ii) and (iii), i.e. plots of y = 0 . 1 ˆ σ 2 ξ and y = 1 2 { log ˆ σ 2 ξ + 0 . 1 ˆ σ 2 ξ +log2 π } where 0 . 1 = σ 2 ξ is the true noise value used to generate the synthetic GP data. We again see evidence of uniform convergence toward this limiting behaviour with increasing n . The green horizontal dotted lines show the limiting values of NLL and calibration ( y = 1 2 { log σ 2 ξ +1+log2 π } and y = 1 respectively) that can be achieved if the incorrect ˆ σ 2 ξ value is replaced by the correct value σ 2 ξ . This underlines the importance of estimating this particular parameter more accurately in order to obtain improved NLL and uncertainty calibration for large n . Further plots from Algorithm 1, showing dependence of each metric on all of the parameters, are given in Figure 6.

6 A Highly Scalable GP Nearest Neighbour Regression Algorithm

6.1 Parameter Estimation

Parameter Estimation Phase 1 The first step of parameter estimation (Figure 1) involves randomly selecting a small subset E of the training data to obtain a first-pass estimate ˆ θ = ( ˆ l, ˆ σ 2 ξ , ˆ σ 2 f ) . Small subsets yield sub-optimal ˆ θ values, yet as shown in 7.1, these are capable of yielding strong MSE performance due to the robustness properties of section 5. We use the method in section 3.1 of [7] to estimate parameters from E , randomly partitioning E into w sizes subsets ( ws = e ) and using a block diagonal approximation (with w blocks of size s × s ) to the full e × e gram matrix. For Table 1 we set e = 3000 , s = 300 , m = 10 . For strong computational efficiency we set e = | E | to a small constant value no matter the size of n . Thus as n grows, an increasingly small portion of the data is used for this phase of parameter estimation and the associated cost does not increase with n . Note that other choices of cheap parameter estimation could be substituted here.

Parameter Estimation Phase 2 (calibration) As shown in section 5, NLL and calibration performance derived from ˆ θ = ( ˆ l, ˆ σ 2 ξ , ˆ σ 2 f ) remain very sensitive to inaccuracies in ˆ σ 2 ξ . An additional 'calibration step' is used to refine those parameters: We randomly select a size c calibration set C (which is otherwise unused ) from the training data and proceed according to Algorithm 2.

Algorithm 2 Calibration of Predictive Distribution

Input: A size c subset C of ( x ∗ , y ∗ ) pairs from the training data, parameters ˆ θ = ( ˆ l, ˆ σ 2 ξ , ˆ σ 2 f ) .

For each ( x ∗ i , y ∗ i ) ∈ C use the efficient GPnn predictive algorithm of 6.2, with covariance function ˆ c ( ., . ) and parameters ˆ θ = ( ˆ l, ˆ σ 2 ξ , ˆ σ 2 f ) , to obtain an estimate of the mean and variance, µ ∗ i , σ ∗ 2 i of the predictive distribution of y ∗ i at x ∗ i .

Compute α = 1 c ∑ c i =1 ( y ∗ i -µ ∗ i ) 2 σ ∗ 2 i .

Output: Calibrated parameters ˆ θ ′ = ( ˆ l, α · ˆ σ 2 ξ , α · ˆ σ 2 f ) .

Note that this process not only adjusts the noise variance estimate σ 2 ξ but also the kernel scale parameter σ 2 f . In so doing it simultaneously calibrates the predictive distribution and improves NLL performance whilst leaving unchanged the MSE performance obtained from the original parameter estimates ˆ θ = ( ˆ l, ˆ σ 2 ξ , ˆ σ 2 f ) . The lemma below is straightforward to prove (Appendix C):

Lemma 5 (Calibration) . The parameters ˆ θ ′ output from Algorithm 2 produce GPnn predictions that (a) achieve perfect (weak) calibration on C , (b) minimise NLL on C over all choices of α and (c) produce the same MSE as ˆ θ does on any choice of test set.

Remark 6. Algorithm 2 can be applied to other GP methods, such as SVGP, to improve calibration.

Table 1 uses c = 1000 ; a simple refinement would be to select c automatically (e.g. using a bootstrap) with optional manual override. Where accurate uncertainty calibration is paramount practitioners could devote much larger CPU resources to this phase (which is also easily distributed); when no uncertainty measures are to be used this calibration step can be bypassed altogether.

6.2 Efficient Nearest Neighbour Prediction

In order to implement GPnn prediction described in 4.1 we use the scikit-learn NearestNeighbors package ([22]). This implements an efficient approximate nearest neighbour algorithm whereby one-time work is carried out to construct a table (at O ( dn log n ) (see e.g. [9, 21]) and counted within the total training times quoted) which subsequent calibration/test predictions then make use of. Query compute-costs are described in the associated documentation, e.g. O ( d log n ) for the Ball-tree algorithm which the default automated algorithm selection in SciKit-Learn should at least match. In contrast, exact kNN costs are listed in the documentation as O ( dn ) . As is evident in Table 2, Figure 3 and the quoted query and table-setup complexities, the nearest neighbour work increases with x -dimension d . Alternative nearest neighbour algorithms and/or dimension-reduction techniques to help address this are yet to be investigated. We set the number of nearest neighbours to be m = 400 for all usages in this paper having observed minimal sensitivity to this parameter on independent synthetic datasets. Although a simple cross-validation procedure could be followed if tuning of m is desired (at an increase in computational overhead), we wished to minimise such fine-tuning to emphasise the simplicity and robustness of the method we present, noting the strong performance we obtain despite this. At first glance, prediction complexity might appear restrictive, but some empirical tests on a laptop reveal comparable performance to SVGP prediction (Table 5).

7 Experimental Performance of GPnn Regression

7.1 Performance on Real World Datasets

Implementational Details 1 : Comparisons are made between our method and the state-of-theart approaches of SVGP [14] and five distributed methods ([15, 2, 33, 7] and [18] following the recommendation in [4]). We have chosen not to include other highly-performant approximations

(e.g. structured kernel interpolation (SKI) methods and their extensions ([37, 39, 11])), since, to our knowledge, none have supplanted these methods in the community as ubiquitous benchmarks on general datasets. Parameters for our method are given in 6.1 and 6.2. SVGP used 1024 inducing points; the distributed methods all used randomly selected subsets of sizes as close as possible to 625. The learning rate for the Adam optimiser was 0.01 for SVGP and 0.1 for our method and distributed methods. All runs in Table 1 used the the squared exponential ('RBF') covariance function. A 'pre-whitening' process (E.1) was applied to x values for all methods and the y values normalised (using training data-derived means and sds) to have mean zero and variance 1. More complete details are given in E. SVGP was run on a single Tesla V100 GPU with 16GB memory; all distributed methods run on eight Intel Xeon Platinum 8000 CPUs sharing 32GB of memory. Our method was run on a Macbook Pro with 2.4 GHz Intel core i5. See D for a full explanation of our selection and pre-processing of datasets which, apart from Protein, are taken from the UCI repository.

Results Runs were made on three randomly selected 7 / 9 , 2 / 9 splits into training and test sets. Table 1 shows MSE and NLL results for our method alongside SVGP and distributed method (note: n = training set size). The table shows only the best of the five distributed methods' results (w.r.t. MSE) but full results and details of all methods and all three performance measures are given in E. Complete calibration results are also plotted in Figure 4. With the exception of the Bike dataset our method is found to outperform all methods simultaneously for both MSE and NLL, and calibration likewise bar a narrow second place on the Song dataset. Table 2 and Figure 3 show that this is achieved whilst undercutting the training costs of the other methods, an effect that is very pronounced for large training sets (e.g. approximately 100 × faster than the other methods at n = 1 . 6 × 10 6 on House Electric). Figure 3 shows that a significant portion of time involves calibration; this can be parallelised (or eliminated if uncertainty is not required). Note also that larger timings observed for higher dimensional datasets are due to slower performance of the approximate nearest neighbour algorithm (6.2) in that regime, both for nn table construction and calibration. As discussed in 6.2, future improvements may reduce this effect. It is very interesting that 'curse-of-dimensionality' has not impacted on the method's MSE, NLL or calibration competitiveness at large d . This was despite the fact that a PCA analysis of the training x values showed no concentration within a low dimensional space (as to be expected given the prewhitening that has been applied (subsection E.1).

Conjecture 7. Robustness to 'curse-of-dimensionality' is at least partially explained by the increase in the intrinsic data-length-scale by a factor of order √ d that must arise in order for GP methods to be effective.

The heuristic reasoning behind this conjecture is as follows: Unless length scale increases with d the kernel gram matrix will exhibit an abundance of exceptionally small off-diagonal entries and hence be unable able to gain significant predictive power. A √ d increase serves to counterbalance this effect and seems consistent with length scales recovered from real data in practice.

7.2 Performance on Massive Synthetic Datasets

We generated size 5 × 10 7 datasets using the 15-variable deterministic Oakley and O'Hagan function [20, 30] with i.i.d. varianceσ 2 ξ additive noise sampled from a zero-mean Laplacian distribution (with much wider tails than N (0 , σ 2 ξ )) . This function has 5 inputs contributing significantly to output

10 4

10 5

10 6

n

10 1

10 2

10 3

Model

Dist.

SVGP

OURS

OURS

d

>20

<20

Bike

Ctslice

Protein

Road3D

Song

1

symlog(Calibration)

Bike

Ctslice

Protein

Road3D

Song

1

1

symlog(NLL) (Relative to extreme)

Bike

Ctslice

Protein

Road3D

Song

0.2

0.4

0.6

0.8

RMSE

BCM

GPOE

GRBCM

POE

RBCM

SVGP

OURS

variance, 5 with smaller effect, and 5 with almost no effect. These properties are poorly matched by the isotropic covariance functions being applied, resulting in gross misspecification of the assumed GRF model and the additive noise. Figure 5 shows performance achieved with both the squared exponential ('RBF') covariance function and the exponential (Matérn 1/2) covariance function. It is very interesting to note the improvement in convergence rate achieved by the exponential kernel. (see Remark 2 for a potential explanation of why isotropic covariance functions are so effective at large n ).

Remark 8. We checked to see whether this strong exponential kernel performance extended to UCI datasets. Surprisingly, given that it is not recommended for use in GP regression (e.g. [36] page 85), it produced best RMSE performance across the board when compared with RBF and Matérn 3/2 kernels (Table 4), with Road3D RMSE reducing from 0.351 to 0.098. Exponential-kernel NLL performance was everywhere best apart from Ctslice and calibration was also better in most cases.

8 Discussion

Related work : The basic 'subset-of-data' approximation ([3]) also achieves training efficiency by using a small portion of training data and can achieve surprisingly goods results ([3], [35] section 5.1, [13]). But it typically would need a much greater proportion of training data than we are using for large n due to its failure to leverage the power of large training sets for prediction; this explains why it is not consistently competitive with other methods. Passing references to the decoupling of prediction and estimation have been made, e.g. [28, 1, 3], but not shown to be as consistently powerful as we have found, nor justified in terms of robustness theory or explored as a mainstream

0

2

4

n

1e7

0.90

1.00

1.10

1.20

Calibration

0

2

4

n

1e7

0.05

0.10

0.15

MSE

0

2

4

n

1e7

-0.11

0.50

NLL

Exp,

2 =0.05

RBF,

2 =0.05

Exp,

2 =0.1

RBF,

2 =0.1

approach. Various works (primarily from the geospatial community) make use of nearest neighbour (NN) techniques for GPs (e.g. [34, 25, 28, 6, 5, 8, 32, 38]). Vecchia ([34]) uses NNs to approximate likelihoods for parameter estimation, whilst Stein ([28]) adapts this work to REML using more distant points as well as NNs, again for parameter estimation purposes. In contrast, our focus is on using NNs for prediction , and whilst we have found passing references to its explicit use for this purpose (e.g. [27]) we have found little or no discussion of effectiveness in comparison to other methods on large datasets and no detailed accompanying analysis of its robustness properties and how these can achieve very high efficiency at scale. [5] gives a construction of a hierarchical fully Bayesian model ('NNGP') derived using collections of NN sets. This approach adds considerable computational overhead and code-complexity, e.g. use of Gibbs sampling. Whilst fully Bayesian treatment of hyperparameters is explored in the ML literature, e.g. [17], it has not been adopted by the MLcommunity for use at scale due to its high computational cost relative to empirical Bayes methods ([17], section 5). Bearing this in mind, we consider the extension given in [8] for improved scalability (Algorithm 5 - 'conjugate NNGP') to be more relevant. In this hybrid method some hyperparameters are recovered as 'empirical-Bayes' point-estimates via grid-based search and the remainder treated in a fully Bayesian fashion using a conjugate prior with some choice of hyper-hyperparameters. This results in a Studentt predictive distribution, rather than Gaussian, but with equal first moment to ours and variance differing only by a (hyper-hyperparameter dependent) multiplicative factor; an effect that our recalibration step would render redundant (see F.1). Recent work ([32, 38]) extends local geospatial GP methods into sparse variational ML applications. 'VNNGP' is shown to be competitive with other methods in [38] despite adding further approximations into the model. We note that when using all observations as the inducing points their predictive mechanism matches ours, up to choice of parameter estimation. We believe these pre-existing works, which differ significantly in approach and perspective, complement our own, which emphasizes the benefits of decoupling parameter estimation from prediction, the robustness properties that can be achieved at large scales, the efficacy of a simple recalibration step and can be run at high scale with a simple algorithm on an off-the-shelf laptop.

Limitations and Future Research: Our results exhibit a leap in speed and performance for GP regression at scale, but there remains more to be done to fully explain and extend performance (as evidenced by our remarks and conjectures). This is particularly so for high dimensional problems where (a) a faster nearest neighbour algorithm would have a particularly big pay-off and (b) there is a need to explain why 'curse-of-dimensionality' appears not to have damaged the method's competitiveness (see Conjecture 7). Extensions of theory to broader aspects of GP robustness, rates of convergence and 'strong calibration' (section 3) are current areas of some the authors' ongoing work.

Acknowledgments

We would like to thank IBM Research and EPSRC for supplying iCase funding for Anthony Stephenson and the UK National Cyber Security Centre for contributing toward Robert Allison's funding. We also thank the anonymous reviewers of this paper for their constructive comments and suggestions.

References

F. Bachoc, N. Durrande, D. Rullière, and C. Chevalier. Properties and Comparison of Some Kriging Sub-model Aggregation Methods. Mathematical Geosciences , 2022.

Y. Cao and D. J. Fleet. Generalized product of experts for automatic and principled fusion of gaussian process predictions. arXiv preprint arXiv:1410.7827 , 2014.

K. Chalupka, C. K. Williams, and I. Murray. A framework for evaluating approximation methods for gaussian process regression. Journal of Machine Learning Research , 14:333-350, 2013.

S. Cohen, R. Mbuvha, T. Marwala, and M. Deisenroth. Healing products of Gaussian process experts. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 2068-2077. PMLR, 2020-13.

A. Datta, S. Banerjee, A. O. Finley, and A. E. Gelfand. Hierarchical Nearest-Neighbor Gaussian Process Models for Large Geostatistical Datasets. Journal of the American Statistical Association , 111(514):800-812, 2016.

A. Datta, S. Banerjee, A. O. Finley, and A. E. Gelfand. On nearest-neighbor Gaussian process models for massive spatial data. Wiley Interdisciplinary Reviews: Computational Statistics , 8(5):162-171, 2016.

M. Deisenroth and J. W. Ng. Distributed gaussian processes. In International Conference on Machine Learning , pages 1481-1490. PMLR, 2015.

A. O. Finley, A. Datta, B. D. Cook, D. C. Morton, H. E. Andersen, and S. Banerjee. Efficient algorithms for bayesian nearest neighbor gaussian processes. Journal of Computational and Graphical Statistics , 28(2):401-414, 2019. PMID: 31543693.

J. Friedman, J. Bentley, and R. Finkel. An Algorithm for Finding Best Matches in Logarithmic Expected Time. ACM Transactions on Mathematical Software (TOMS) , 3:209-226, 1977.

J. Gardner, G. Pleiss, K. Q. Weinberger, D. Bindel, and A. G. Wilson. Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. Advances in neural information processing systems , 31, 2018.

J. R. Gardner, G. Pleiss, R. Wu, K. Q. Weinberger, and A. G. Wilson. Product kernel interpolation for scalable gaussian processes. International Conference on Artificial Intelligence and Statistics, AISTATS 2018 , 84:1407-1416, 2018.

L. Györfi, M. Kohler, A. Krzy˙ zak, and H. Walk. A Distribution-Free Theory of Nonparametric Regression . Springer Series in Statistics, 2010.

K. Hayashi, M. Imaizumi, and Y. Yoshida. On random subsampling of gaussian process regression: A graphon-based analysis. In International Conference on Artificial Intelligence and Statistics , pages 2055-2065. PMLR, 2020.

J. Hensman, N. Fusi, and N. D. Lawrence. Gaussian processes for big data. arXiv preprint arXiv:1309.6835 , 2013.

G. E. Hinton. Training products of experts by minimizing contrastive divergence. Neural computation , 14(8):1771-1800, 2002.

M. Jankowiak, G. Pleiss, and J. Gardner. Parametric Gaussian process regressors. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 4702-4712. PMLR, 13-18 Jul 2020.

V. Lalchand and C. E. Rasmussen. Approximate Inference for Fully Bayesian Gaussian Process Regression, Apr. 2020. arXiv:1912.13440 [cs, stat].

H. Liu, J. Cai, Y. Wang, and Y. S. Ong. Generalized robust bayesian committee machine for large-scale gaussian process regression. In International Conference on Machine Learning , pages 3131-3140. PMLR, 2018.

H. Liu, Y.-S. Ong, X. Shen, and J. Cai. When gaussian process meets big data: A review of scalable gps. IEEE transactions on neural networks and learning systems , 31(11):4405-4423, 2020.

J. E. Oakley and A. O'Hagan. Probabilistic sensitivity analysis of complex models: A bayesian approach. Journal of the Royal Statistical Society. Series B (Statistical Methodology) , 66(3):751769, 2004.

S. Omohundro. Five Balltree Construction Algorithms. Technical report, International Computer Science Institute, 1989.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

B. Schölkopf. The Kernel Trick for Distances. In T. Leen, T. Dietterich, and V. Tresp, editors, Advances in Neural Information Processing Systems , volume 13. MIT Press, 2000.

H. Song, T. Diethe, M. Kull, and P. Flach. Distribution calibration for regression. 36th International Conference on Machine Learning, ICML 2019 , 2019-June:10347-10356, 2019.

M. L. Stein. The screening effect in kriging. Annals of Statistics , 30(1):298-323, 2002.

M. L. Stein. 2010 Rietz lecture when does the screening effect hold? Annals of Statistics , 39(6):2795-2819, 2011. arXiv: 1203.1801v1.

M. L. Stein. Limitations on low rank approximations for covariance matrices of spatial data. Spatial Statistics , 8:1-19, 2014. Spatial Statistics Miami.

M. L. Stein, Z. Chi, and L. J. Welty. Approximating likelihoods for large spatial data sets. Journal of the Royal Statistical Society. Series B: Statistical Methodology , 66(2):275-296, 2004.

A. Stephenson, R. Allison, and E. Pyzer-Knapp. Provably reliable large-scale sampling from gaussian processes. arXiv preprint arXiv:2211.08036 , 2022.

S. Surjanovic and D. Bingham. Virtual library of simulation experiments: Test functions and datasets. Retrieved May 5, 2023, from http://www.sfu.ca/~ssurjano .

M. Titsias. Variational learning of inducing variables in sparse gaussian processes. In Artificial intelligence and statistics , pages 567-574. PMLR, 2009.

G.-L. Tran, D. Milios, P. Michiardi, and M. Filippone. Sparse within sparse gaussian processes using neighbor information. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 10369-10378. PMLR, 2021-18.

V. Tresp. A bayesian committee machine. Neural computation , 12(11):2719-2741, 2000.

A. V. Vecchia. Estimation and Model Identification for Continuous Spatial Processes. Journal of the Royal Statistical Society. Series B (Methodological) , 50(2):297-312, 1988. Publisher: [Royal Statistical Society, Wiley].

K. Wang, G. Pleiss, J. Gardner, S. Tyree, K. Q. Weinberger, and A. G. Wilson. Exact gaussian processes on a million data points. Advances in Neural Information Processing Systems , 32, 2019.

C. K. Williams and C. E. Rasmussen. Gaussian processes for machine learning , volume 2. MIT press Cambridge, MA, 2006.

A. G. Wilson and H. Nickisch. Kernel interpolation for scalable structured Gaussian processes (KISS-GP). 32nd International Conference on Machine Learning, ICML 2015 , 3:1775-1784, 2015.

L. Wu, G. Pleiss, and J. Cunningham. Variational Nearest Neighbor Gaussian Process. Proceedings of the 39th International Conference on Machine Learning , 2022.

M. Yadav, D. Sheldon, and C. Musco. Faster kernel interpolation for gaussian processes. In A. Banerjee and K. Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , volume 130 of Proceedings of Machine Learning Research , pages 2971-2979. PMLR, 13-15 Apr 2021.

A Theoretical GPnn Results

A.1 Preliminary results

Let ρ ( x , x ′ ) = σ f √ 1 -c ( x /l, x ′ /l ) be the kernel-induced distance function over R d ([23]). We define x ( j,n ) ( x ∗ ) as the j th nearest neighbour random variable to a test point x ∗ under ρ , which we abbreviate to x ( j ) when the context is clear, and x ( j ) ( x ∗ ) ∈ N m ( x ∗ ) as the realised j th nearest neighbour of the test point x ∗ from a training set X . From this we define ϵ i = ρ 2 ( x ( i ) , x ∗ ) and ϵ ij = ρ 2 ( x ( i ) , x ( j ) ) .

Definition 9 (Support) . Let P x be the probability measure of x and S ρ x ,ϵ the closed ball of radius ϵ > 0 under the metric ρ centred at x . Then we define support( P x ) = { x : P x ( S ρ x ,ϵ ) > 0 ∀ ϵ > 0 } .

Definition 10 (Weakly-faithful) . We define a pair of metrics ρ ( · , · ) , ˆ ρ ( · , · ) to be weakly-faithful w.r.t. each other if the following condition holds: The m th nearest neighbour under ˆ ρ converges to the test point as n →∞ if and only if the m th nearest neighbour under ρ converges to the test point in the limit.

Assumptions

(A1) x iid ∼ P x and x ∗ ∈ support( P x ) under the generative metric defined by c ( · , · ) .

(A2) c ( · , · ) , ˆ c ( · , · ) are stationary kernels whose induced distance functions are weakly faithful metrics (Definition 10).

Note: Assumption (A2) is not overly restrictive and encompasses commonly used kernels such as all those mentioned in this paper.

Lemma 11. ϵ i → 0 and ϵ ij → 0 as n →∞ a.e. with respect to the measure over x ∈ R d , P x , for i, j ≤ m , m n → 0 and under (A1-2).

Proof. Lemma 6.1 of [12] states that ∥ ∥ x ( m,n ) ( x ) -x ∥ ∥ n →∞ - - - - → 0 with probability one (with respect to P x ). Their proof can be generalised immediately to state that ρ ( x ( m,n ) ( x ) , x ) n →∞ - - - - → 0 by using our definition of support, 9, that directly invokes the metric ρ . Hence ϵ i → 0 for all i ≤ m (since x ∗ is in support( P x ) ). Since ρ is a metric it satisfies the triangle inequality; hence ρ ( x ( i ) , x ( j ) ) ≤ ρ ( x ( i ) , x ∗ ) + ρ ( x ( j ) , x ∗ ) n →∞ - - - - → 0 for all i, j ≤ m .

Lemma 12. For an m -GPnn under the assumptions (A1-3),

Proof. From Lemma 11 we have that lim n →∞ k ( x ( j ) ( x ∗ ) , x ∗ ) = lim n →∞ ( σ 2 f -ϵ i ) = σ 2 f and lim n →∞ k ( x ( i ) ( x ∗ ) , x ( j ) ( x ∗ )) = lim n →∞ ( σ 2 f -ϵ ij ) = σ 2 f . As a result, k ∗ N → σ 2 f 1 and

Now using Sherman-Morrison and the continuity of matrix inverse and matrix-matrix products:

Thus,

Lemma 13 ( WSRF expectations) . Under (A3), E y , y ∗ { y y ∗ } = k ∗ and E y { yy T } = K .

Proof. By assumption on the covariance properties of y and the independence and zero-mean of the additive noise, E y { y i y j } = k ( x i , x j ) . Extending this to the joint distribution over y , y ∗ is straightforward and gives the results stated.

Lemma 13 is subsequently assumed to be in use throughout A.2.

A.2 Limit proofs

In the following statements only misspecification of type (d) and/or (e) (subsection 4.2) is considered to be at work.

Lemma 14 (MSE limit) . Under the assumptions (A1-3), for fixed m< ∞ , the predictive GPnn given in subsection 4.1 converges pointwise in the sense of MSE wrt P x -a.e. as

Proof. This follows from Lemma 12 by expanding the definition of MSE :

Since E y { µ ∗ N 2 } = E y { k ∗ N T K -1 N y N y T N K -1 N k ∗ N } = k ∗ N T K -1 N k ∗ N , and by assumption E y , y ∗ { y N y ∗ } = k ∗ N , even under a WSRF generative model (Lemma 13).

Corollary 15 (NLL limit) .

Proof. The proof follows straightforwardly from Lemma 12 and because σ ∗ 2 N = σ 2 f + σ 2 ξ -k ∗ N T K -1 N k ∗ N .

A.2.1 Full misspecification

For the remainder of A.2 we assume that the full range of possible misspecifications ((a)-(e)) outlined in subsection 4.2 are in action. We refer to this case as 'fully-misspecified' and introduce the notation ˆ µ ∗ N , ˆ σ ∗ 2 N to be understood to mean the predictive mean and variance under these misspecifications.

Lemma 16 (Fully misspecified MSE limit) . For a fully misspecified model, asymptotically

provided the misspecified kernel distance metric is weakly faithful in the sense that the m th nearest neighbour converges under both the true and misspecified metrics (Definition 10).

Proof.

We can use standard results to state that ( a ) + ( b ) = σ 2 f + σ 2 ξ . Then we define ˆ γ = ˆ σ 2 f ˆ σ 2 ξ + m ˆ σ 2 f and expand it in terms of m -1 :

In a manner similar to Lemma 12 we use this result to compute:

and

where we have used the expansion of 1 -m ˆ γ given earlier. Putting these results together gives

Lemma 17 (Calibration limit under full misspecification) .

Proof. We use continuity to write

By direct application of Lemma 12 ˆ σ ∗ 2 N n →∞ - - - - → ˆ σ 2 ξ (1 + m -1 ) -O ( m -2 ) and thus

Corollary 18 (NLL limit under full misspecification) .

Proof. We start with

For the second term we use Lemma 17 so that we have

Proof of Theorem 1. We construct the proof using all of the intermediate results given above. In particular item (i) follows from Lemma 16, item (ii) from Lemma 17 and item (iii) from Corollary 18.

B Validity of Algorithm 1 (Proof of Lemma 4)

Proof of Lemma 4. We prove the result for MSE, the proofs for NLL and calibration being essentially identical.

The proof involves showing that Algorithm 1 is exactly equivalent to Algorithm 1b, an alternative scheme which self-evidently provides valid estimates of MSE on sizen synthetic datasets:

Algorithm 1b Simulation of GPnn Robustness and Limiting Behaviour (Expensive)

Generate a sizen set of i.i.d. training x -values X = { x i } n i =1 with x i ∼ P x ( X is held constant henceforth).

Generate n ∗ i.i.d. test points x ∗ i ∼ P x .

Generate n ∗ separate independent synthetic GP samples y i ∈ R ( n +1) corresponding to each of the values ( x ∗ i , X ) .

From within X take the m nearest-neighbours N ( x ∗ i ) to the test point x ∗ i .

Corresponding to each test point x ∗ i evaluate the function e ∗ i ′ = f i ( y i ) = ( y ∗ i -µ ∗ N ( y ′ i )) 2 . Where, as defined in Equation 6, µ ∗ N = ˆ k ∗ N T ˆ K -1 N y ′ i with N = N ( x i ) and where y ′ i ∈ R m is formed from the components of y i corresponding to N = N ( x i ) .

Compute the average 1 n ∗ ∑ n ∗ i =1 e ∗ i ′ to obtain the MSE statistic.

Algorithm 1b clearly provides a valid evaluation of the MSE arising from the full GPnn prediction process described in section 3 (albeit at prohibitive expense for large n ). Hence it is sufficient to prove that Algorithm 1 is equivalent to Algorithm 1b. We do so by applying two minor alterations to Algorithm 1b in succession whose combined effect is to convert it to Algorithm 1. We also show that throughout this process equivalence is maintained with Algorithm 1b thus completing the proof:

Change 1: Let y i = ( y ∗ i , y ′ i , y ′′ i ) where ( y ∗ i , y ′ i ) ∈ R ( m +1) with y ′ i corresponding to the x -values in N ( x ∗ i ) . Since each function f i ( y i ) in line 5 above only depends on the components ( y ∗ i , y ′ i ) of the ( n + 1) -long vector y i we can equally write e ∗ i ′ = f i ( y ∗ i , y ′ i ) and hence truncate each of the vectors y i to ( y ∗ i , y ′ i ) before evaluating e ∗ i ′ in 5 and this clearly leaves the output of Algorithm 1b completely unchanged.

Change 2: Note (by a standard property of multivariate normal distributions) that each of the truncated vectors ( y ∗ i , y ′ i ) are i.i.d. from the multivariate normal distribution whose covariance Σ is the kernel gram matrix corresponding to just ( x ∗ i , N ( x ∗ i )) . Hence it is legitimate to change the (inefficient) way of sampling ( y ∗ i , y ′ i ) (via the large sample y ) to direct sampling from N ( 0 , Σ) whilst still maintaining equivalence with Algorithm 1b.

In this way we arrive at Algorithm 1.

C Parameter Calibration (Proof of Lemma 5)

Proof of Lemma 5. (a) Replacing parameters ˆ θ = ( ˆ l, ˆ σ 2 ξ , ˆ σ 2 f ) with ˆ θ ′ = ( ˆ l, α ˆ σ 2 ξ , α ˆ σ 2 f ) changes all of the σ ∗ i 2 values to ασ ∗ i 2 and therefore changes the calibration value on C from α = 1 c ∑ c i =1 ( y ∗ i -µ ∗ i ) 2 σ ∗ 2 i to α/α = 1 . (b) The NLL on C arising from parameters ( ˆ l, α ˆ σ 2 ξ , α ˆ σ 2 f ) is 1 2 c ∑ c i =1 { log ( α ˆ σ 2 ξ ) + ( y ∗ i -µ ∗ i ) 2 / ( ασ ∗ 2 i ) + log 2 π ) } which, on taking first and second derivatives w.r.t. α , is found to be uniquely minimised by α = 1 c ∑ c i =1 ( y ∗ i -µ ∗ i ) 2 σ ∗ 2 i . (c) It is easily shown that replacing parameters (ˆ σ 2 ξ , ˆ σ 2 f ) by ( k ˆ σ 2 ξ , k ˆ σ 2 f ) (for any k > 0 ) in the formula for µ ∗ (Equation 3 and Equation 6) does not

alter µ ∗ . Hence the value of MSE = 1 n ∗ ∑ n ∗ i =1 ( y ∗ i - µ ∗ i ) 2 on any size- n ∗ test set is unchanged when parameters ˆ θ ′ are used in place of ˆ θ .

D Real world datasets

We consider a variety of datasets from the standard UCI machine learning repository 2 . These datasets are commonly used in the GP literature (see [10] for instance) and are, in principle, easily available online. In practice, we encountered some difficulties: the dataset documentation is often limited; the dataset names commonly used in other published papers do not always match the UCI database naming and important details about data pre-processing, which features to use etc, are often omitted. There are numerous attempts on GitHub and elsewhere at cataloguing these datasets along with any pre-processing, however we had limited success using them, with many appearing unmaintained. Our focus in this work is on testing our methods on a variety of real world datasets and in a way that is, as far as possible, consistent with other papers. We therefore rejected datasets about which there is ambiguity over the correct features to use, or even which column to regress on or for which outlier rejection is required but undocumented elsewhere.

Referring to the datasets used in [10], we were able to locate the following:

Song ( https://archive.ics.uci.edu/ml/machine-learning-databases/00203/ YearPredictionMSD.txt.zip )

Bike ( https://archive.ics.uci.edu/ml/machine-learning-databases/00275/ Bike-Sharing-Dataset.zip )

Poletele ( https://archive.ics.uci.edu/ml/machine-learning-databases/ parkinsons/telemonitoring/parkinsons_updrs.data )

Keggdirected ( https://archive.ics.uci.edu/ml/machine-learning-databases/ 00220/Relation%20Network%20(Directed).data )

Keggundirected ( https://archive.ics.uci.edu/ml/ machine-learning-databases/00221/Reaction%20Network%20(Undirected) .data )

CTSlice ( https://archive.ics.uci.edu/ml/machine-learning-databases/ 00206/slice_localization_data.zip )

Road3d ( https://archive.ics.uci.edu/ml/machine-learning-databases/ 00246/3D_spatial_network.txt )

Protein ( https://archive.ics.uci.edu/ml/machine-learning-databases/ 00265/CASP.csv )

Buzz ( https://archive.ics.uci.edu/ml/machine-learning-databases/00248/ regression.tar.gz )

HouseElectric (HouseE) ( https://archive-beta.ics.uci.edu/dataset/235/ individual+household+electric+power+consumption )

We were unable to find any documentation on the Kegg datasets to indicate which of the columns should be used as the independent variable (the regressor) and neither is this mentioned in any literature of which we are aware. Initial runs of standard exact GP training and prediction produced RMSEs much higher that reported in [10]. Combining these two observations, we chose to exclude both Kegg datasets. Likewise we faced problems with Buzz. An analysis of the y values revealed a small proportion of extremely large outliers that we found could unduly distort performance results (e.g. depending on whether these outliers appeared in the test set for some of the random splits). With the lack of documentation we were unable to identify an outlier rejection scheme that we were confident would be consistent with results quoted in other papers. For this reason we have excluded Buzz.

The choice of ( x , y ) value that we applied for each of the used datasets is as follows:

Song. The first column is y , all remaining columns are x .

Bike. We use hour.csv . The y value is cnt . dteday (the date) is transformed to just be the integer representation of the day. instant is just an index so is dropped. registered and casual are dropped as registered + casual = cnt .

Poletele. The y value is total_UPDRS . The columns subject# and test_time are not relevant to the problem so are dropped.

CTSlice. y value is the final column. The first column is dropped as it is just an index. We additionally drop six columns which are constant over the majority of the dataset, namely columns 59, 69, 179, 189, 279 and 351.

Road3d. y value is the final column. The first column is dropped as it is just an index.

Protein. This dataset was processed as per https://github.com/hughsalimbeni/ bayesian_benchmarks , whereafter we used our own random (seeded) train/test split.

HouseElectric. y value is the column labelled 'Global active power', rescaled by 1000 / 60 and with 'Sub metering 1,2,3' columns subtracted. We convert the date column into day-ofyear/365 and the time column into time of day in minutes. Further, we remove any rows with null entries.

We note that although we are using a standard set of real-world datasets, it is not always clear exactly how others in the field have carried out their own preprocessing, limiting the ability to make direct comparisons to other results reported in the literature.

E Additional Implementation Details

E.1 Pre-whitening of Data

For all datasets covered in subsection 7.1 the following 'whitening' preprocessing step is adopted: Let y be the vector of all regressor values in the training dataset only, and X the matrix of all regressands in the training dataset only, where each row of X is a feature. Let µ y , σ 2 y be the sample mean and variance of y respectively in the training dataset, then the whitened y values used in both the training and test set are simply σ -1 y ( y -µ y ) . Let µ X , Σ X be the sample mean and covariance matrix of X respectively . Let Σ X = MM T , then the whitened x values in both the training and test data are 1 √ d M -1 ( x -µ X ) , where d is the feature dimension of X . Note : the performance metrics given in subsection 7.1 are expressed in terms of the whitened y values rather than the y values in their original form. This appears to be common practice in the literature and has no bearing on the comparative performance of the different methods within this paper.

E.2 Test-Set Batching

To prevent excessive memory consumption, we perform all predictions for the distributed and variational methods in batches of 1000 points at a time. Where this is not possible (e.g. for especially large datasets), we use smaller batches of 500 or 250 points, as appropriate.

E.3 Additional Implementation Details for SVGP

We use the sparse variational inducing point approach of [14], following the implementation provided by GPyTorch, which in particular uses a Choleksy decomposition to parameterise the covariance matrix of the variational prior. We broadly follow the SVGP implementation example provided by https://docs.gpytorch.ai/en/stable/examples/04_Variational_and_ Approximate_GPs/SVGP_Regression_CUDA.html . In particular, we follow their example in using the Adam optimiser to train our model over 100 epochs with a minibatch size of 1024 and a learning rate of 0.01. We opt to use 1024 inducing points. All experiments under this method are run on a SageMaker ml.p3.2xlarge instance, consisting of a single Tesla V100 GPU with 16GB of memory.

E.4 Additional Implementation Details for Distributed methods

A good introduction to distributed methods for Gaussian process inference is [7]. Here we run the product-of-experts (PoE) [15], generalised product-of-experts (gPoE) [2], Bayesian committee machine (BCM) [33], robust Bayesian committee machine (rBCM) [7] and generalised robust Bayesian committee machine (GrBCM) [18] following the recommendation in [4] to aggregate in f -space. There are three components to any distributed method: the hyperparameter inference, the partitioner and the aggregator . Hyperparameter estimation is the same for all of the methods: we use the method in section 3.1 of [7], randomly partitioning the entire training set into subsets of size 625 (or as close as possible with equal-sized experts given that in general n is not a multiple of 625). A block diagonal approximation (with n/ 625 blocks) is then used to approximate to the full n × n gram kernel matrix. To recover hyperparameters with this we use Gaussian Process models with a zero prior mean and a scaled square-exponential kernel. Training is conducted using the Adam optimiser with a learning rate of 0.1 over 100 optimiser iterations. Once the hyperparameters are trained, we run our distributed prediction mechanism to evaluate performance against the test-set. The 625-sized partitioned blocks are referred to as 'experts' and the shared hyperparameter values are distributed to each expert and held fixed thereafter. In the aggregator , or distributed prediction phase, each expert produces an individual predictive distribution and these are then aggregated to a final predictive mean and variance for each of our test points. GRBCM prediction is a little more complex than this as it makes use of an additional 'communications' expert as explained in [18], aggregating in f -space as recommended in [4]. We provide timing statistics for training these models.

We use our own GPyTorch-based implementation of distributed GP approximations. All exact GP calculations are performed using GPyTorch using the default settings (so 20 Lanczos iterations throughout and a CG tolerance of 1 for hyperparameter inference, and 10 -3 for posterior predictions). For all of our experiments, we utilise an AWS t3.2xlarge instance (consisting of 8 Intel Skylake Processors and 32 GB of RAM).

E.5 Reproducibility

All code used to generate tables and figures in this document can be found at https://github. com/ant-stephenson/gpnn-experiments/ .

F Related Work

F.1 NNGP

An hierarchical Bayesian approach to nearest neighbour GPs is derived in [5], who construct a full stochastic process allowing an end-to-end probabilistic approach they term 'NNGP' derived from a 'parent' GP using collections of nearest neighbour sets forming a 'reference' set. This work is modified in various ways in [8] including by adaption to a hybrid empirical-Bayes/fully-Bayesian approach to improve scalability. This is described in Algorithm 5 which presents an MCMC-free approach ('conjugate NNGP') where some parameters are estimated via K-fold cross-validation and the remainder are given conjugate priors to allow exact posterior inference. Under this model, the marginal predictive distribution is Studentt with mean and variance expressions that match that given by GPnn up to a scaling factor on the variance. Here we will go into more detail to make this connection explicit. First of all, note that they use an alternative parameterisation to us, specifying an inverse lengthscale ( ϕ ), a ratio ( α ) of noise variance ( τ 2 ) to kernelscale ( σ 2 ) and a vector of coefficients for their linear mean function ( β , which we can neglect since we focus exclusively on mean-zero GPs).

In Algorithm 5 they go on to give the following expressions for the predictive mean and variance:

We can see that the predictive mean m 0 matches our own in the mean-zero setting. In particular, in this case the only non-zero term is dot ( w,y [ N ( s, k )]) which can be re-written in our notation as k ∗ N T K -1 N y N . Note that this expression is, for fixed α , independent of choice of kernelscale. This can be seen using the notation given in section 1: k ∗ N T K -1 N y N = σ 2 f c ∗ N T ( σ 2 f C N + σ 2 ξ I ) -1 y N = c ∗ N T ( C N + αI ) -1 y N .

In addition to this, the predictive variance is equivalent to ours up to multiplicative scaling S = b ∗ σ ( k ) / [ σ 2 f ( a ∗ σ ( k ) -1)] ; i.e. ̂ Var( y ( s )) = Sσ ∗ 2 N , which be seen as follows: In the mean-zero case we have v 0 = 1+ α -dot ( w,z ) which can again be re-written in our notation as 1+ σ 2 ξ /σ 2 f -c ∗ N T C -1 α,N c ∗ N (where C α,N = C N + αI ). Hence, if we rescale this by a factor of σ 2 f we exactly obtain the GPnn predictive variance, so that ̂ Var( y ( s )) = b ∗ σ ( k ) v 0 / ( a ∗ σ ( k ) -1) = b ∗ σ ( k ) σ ∗ 2 N / [ σ 2 f ( a ∗ σ ( k ) -1)] as claimed. Since GPnn uses an additional recalibration step to rescale all predictive variances by a single shared multiplicative factor, the factor S between the two methods can be made effectively redundant: i.e. if our recalibration step (Algorithm 2) were applied to their method, the two methods would become equivalent with respect to RMSE and weak-calibration (setting aside differences in parameter estimation and their associated costs).

We reemphasise that whilst GPnn and Conjugate NNGP pointwise predictions can be related as above, the latter is given in the form of a t -distribution which, even with matched first two moments, will have a different shape to that of a Normal distribution. As such, the (Gaussian) NLL performance measure is no longer a valid choice to assess this model. RMSE and weak-calibration remain distribution-agnostic however.

G Further simulation results

0

2

0.0

0.2

0.4

0.6

0.8

MSE

0.0

0.5

1.0

0.0

0.2

0.4

0.6

0

2

0.00

0.25

0.50

0.75

1.00

0

2

0.0

0.5

1.0

1.5

NLL

0.0

0.5

1.0

0.0

0.5

1.0

0

2

0.0

0.5

1.0

1.5

2.0

0

2

2

f

0

1

2

Calibration

0.0

0.5

1.0

2

0.0

0.5

1.0

1.5

2.0

0

2

l

0

1

2

3

1000

10000

100000

1000000

10000000

f

(

)

f

(

)

H Further Results on UCI Datasets

H.1 Results for all distributed methods

H.2 Performance of different kernels

H.3 Prediction times

I Overall Computational Expenditure

Our distributed and variational method experiments were conducted using cloud computing resources. Experiments using our own method have been carried out on an author's laptop. SVGP experiments were run using a SageMaker virtual machine on a single Nvidia Tesla V100 GPU with 16GB memory. Distributed method experiments were run using eight Intel Xeon Platinum 8000 CPU cores (t3.2xlarge EC2 instances).

Below we will attempt to give reasonable indications of the amount of computational work expended to obtain the results in this paper, though note that we are neglecting the work expended in the development and research stages that did not directly contribute to the runs in the paper. As such, the costs presented are representative of the costs of replicating our paper, not repeating the research from scratch. Instead of reporting costs in dollars, we will report approximate computing hours for each instance type. The reader can then estimate their own costs using the current instance costs in the region of their choice, or under other cloud providers or even using on-premise compute.

This gives a total of around 5.4 hours of compute time on a 1 GPU VM and 302.2 hours on an 8 CPU VM.

Peer reviews:
comments: SUMMARY:
The paper proposes a Gaussian process regression technique that only uses nearest neighbors for prediction. The question of choosing hyperparameters is addressed and asymptotic behavior is analyzed. The method shows substantial speed up on UCI datasets while also providing improved predictive performance.

STRENGTHS:
The empirical evaluation shows very good results. Even so, the method is theoretically analyzed and discussed with interesting insights and hypotheses. The paper is well-written.

WEAKNESSES:
A comparison with other nearest neighbor based techniques might have been helpful (e.g. ones cited in the paper). Also, it might make sense to investigate more into the geostatistical work on the subject and cite something from the area. I know that nearest neighbor based predictions are quite often considered to be standard in the area.

QUESTIONS:
Due to time constraints and NeurIPS not accommodating my request for a reduced reviewing workload, I did not have the opportunity to thoroughly review the proofs or examine all the details.

LIMITATIONS:
The limitations are addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a change in perspective in how Gaussian process regression can be leveraged by conditioning predictive distribution on only the neighboring data points. The paper argues against the common practice of using one singular set of data points for model hyperparameter estimation and prediction, and presents a framework for running scalable approximation of GP regression using locality information.\n\nSTRENGTHS:\nI believe that the overall idea of leveraging locality in Gaussian process models has a lot of potential: while I am skeptical as the arguments in the paper seem unconvincing, I think there is merit in good empirical performance for such algorithms. However, I believe that a comparison with existing methods should be done in a more proper manner.\n\nWEAKNESSES:\nWhile using KNN in combination of GP regression might seem like a straightforward idea to massively speed up the inference of GP models, I believe that many key aspects of GPs are overlooked, constituting a major weakness from this paper. I list some of the specific examples below. \n- Combining KNN with GP is overall not a novel idea, the practice is not exactly new. A cursory search reveals that many research papers outside the machine learning community are already using it as a proxy for the less scalable fullscale GP regression. Some examples relevant to machine learning include: 1. Wu L, Pleiss G, Cunningham JP. Variational nearest neighbor Gaussian process. In: Proceedings of the 39th International Conference on Machine Learning, PMLR; 2022; 2. Chen H, Zheng L, Kontar RA, Raskutti G. Gaussian Process Parameter Estimation Using Mini-batch Stochastic Gradient Descent: Convergence Guarantees and Empirical Benefits. Journal of Machine Learning Research. 2022;23(227):1–59. \n- The paper focuses on _pointwise_ uncertainty prediction, while GP posteriors provide not only a pointwise uncertainty, but _pairwise_ covariance as well. The covariance kernel is such a crucial element that defines the hypothesis space of GP models that given a configuration of pointwise uncertainty, there exists countless GPs that takes identical pointwise variance. \n- While KNN is guaranteed to speed up massively in lower dimensions, it suffers from curse of dimensionality itself as it becomes harder to find nearest neighbors. \n- While it is not mandatory for parameter estimation and prediction to be conditioned on the same set of data, SVGP does not incur much computational expense at prediction level. Conventional GP regression and SVGP only need one matrix inversion operation to predict the uncertainty level on an arbitrary number of test points, while GPnn needs a matrix inversion operation for every separate data point -- essentially it becomes a balancing act of whether optimizing an SVGP or apply KNN to every test point is more expensive. \n- Theorem 1 discusses the asymptotic behavior as $n\\rightarrow\\infty$. While the result of insensitivity w.r.t. kernel parameter carries some value, the _rate_ of convergence matters in a finite data regime. For example, Ackermann function and inverse Ackermann function both converge to infinity but at drastically different rates. \n- While the common squared exponential kernel is designed to look at neighborhood information, many more expressive kernels can have longer-range connections. I think the paper tacitly admits that GPnn can only be used for interpolation tasks, but it is somewhat unfair to state that kernel hyperparameter does not matter as it is always a misspecified model. \n- The ease at obtaining easy uncertainty information given neighborhood information is conditioned on conjugate likelihood, with a Gaussian observational model. As some degree of variational inference is required for non-conjugate likelihood, GPnn struggles to apply beyond regression tasks. \n- Simulating synthetic data using Algorithm 11 seems flawed. Algorithm 1 generates test points as if every 2 test points are un-correlated, which naturally gives GPnn an advantage.\n\nQUESTIONS:\nNone.\n\nLIMITATIONS:\nI have listed the limitations in the \"weakness\" section.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper starts by proposing use of a simple nearest neighbors scheme for prediction, where instead of using the training set, one uses the $m$ nearest neighbors of the training set to the test set. They then, in Theorem 1, analyze the asymptotic expected MSE, calibration and negative log-likelihood of this prediction approach under a given estimator of the parameters. They find that there is robustness to poor estimation except in noise variance. They show empirically that their simulations match theory in algorithm 1. They then propose a scalable GP regression algorithm: noting that when using their nearest neighbor prediction approach, the MSE, calibration and negative log-likelihood are *somewhat* robust to wrong parameters (except noise variance), they use multiple approximation steps, including using a subset of the training data and using a structured covariance matrix. In order to improve noise variance estimation, they add a calibration step. They show that it outperforms several baselines on real world datasets.\n\nSTRENGTHS:\nThis is a very interesting paper: the resulting technique is very simple but pops out in a very non-obvious way. Further, it leads to a powerful consequence: you don't need the greatest parameter estimates for training if you do nearest neighbors prediction, as long as you add a calibration step to improve the estimate of noise variance. Despite my slight reservations about the baselines and the strange paper structure, I strongly recommend acceptance.\n\nWEAKNESSES:\nThis paper is written in a very non-standard way and completely ignores the standard ML paper format. It completely ignores the standard introduction, replacing it by what is often the 2nd or third section, which is some technical background. It then has the 2nd section, which is *similar* to what often comes at the end of an intro, but is somehow different. The related work section is very short. In general everything is somewhat terse. I don't completely hate it, but it's jarring to read this, and I'm not sure what the justification for doing it this way is, particularly since the authors seem aware of literature and the techniques used in the literature so presumably are aware of how standard ML papers are written to flow a certain way.\n\nThe main weakness is that I'm not sure how close to state of the art the baselines are.\n\nQUESTIONS:\nWhy did you use this particular, somewhat strange, paper structure?\n\nWhy is your calibration definition what it is?\n\nI'm a little confused: you said you replace $X$, but eqn. 5 still has it. Is this a typo?\n\nIs SVGP really state of the art? What about SKI [1] and its extensions?\n\n[1] Wilson, Andrew, and Hannes Nickisch. \"Kernel interpolation for scalable structured Gaussian processes (KISS-GP).\" International conference on machine learning. PMLR, 2015.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work at one level is about proposing a scalable GP approach called GPnn where the prediction step uses only the nearest neighbours of a test point in order to form the predictive distribution. This approach is well-motivated as the authors argue that there isn't a strong mathematical reason to couple the training step / estimation of hyperparameters and the prediction / generalisation over unseen inputs. At the end of the day, we care about the latter (prediction performance) and not really the former. At another level, there is a theoretical component where they show that predictive distributions obtained through GPnn are robust to a wide array of model misspecification, for instance, wrong kernel choice or Gaussian noise in the large data limit.\n\nSTRENGTHS:\nThe paper is well-written and clear. There is always a demand for new GP approximations which scale to millions of data points as SVGP has a known issue with over-estimating the aleatoric uncertainty. The algorithms clearly describe the simulation and evaluation step. There is a good summary of other distributed methodologies and clear delineation of how this work is different. Some components from this paper can be applied to other methods in order to refine estimates and calibration (Remark 5).\n\nWEAKNESSES:\nFigure 2 is hard to read - there is a lot going on and ideally should have been plotted with N on the x-axis.\n\nQUESTIONS:\n- What about a sensitivity analysis to the choice of M? or are GPnn predictions insensitive to this in the large data limit. \n- Can M be learnt instead of set? \n- Perhaps some areas of the input space require more M for good predictive performance than areas where the function isn't changing much.\n- It would be interesting to see the performance on non-stationary functions.  \n- There isn't sufficient insight on the selection of nearest neighbours  exact or approximate - at first blush one would dismiss the idea because of the added compute entailed in finding neighbours per test data point. What is the complexity of this as a function of N and d precisely?\n\nLIMITATIONS:\nThere is some discussion of limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}