Title: Computation-Aware Gaussian Processes: Model Selection And Linear-Time Inference
Venue: NeurIPS 2024
Accepted: yes
Keywords: Gaussian Processes

Model Selection

Approximate Inference

Variational Inference

Probabilistic Numerics
Abstract: Model selection in Gaussian processes scales prohibitively with the size of the training dataset, both in time and memory.
While many approximations exist, all incur inevitable approximation error.
Recent work accounts for this error in the form of computational uncertainty, which enables---at the cost of quadratic complexity---an explicit tradeoff between computational efficiency and precision.
Here we extend this development to model selection, which requires significant enhancements to the existing approach, including linear-time scaling in the size of the dataset.
We propose a novel training loss for hyperparameter optimization and demonstrate empirically that the resulting method can outperform SGPR, CGGP and SVGP, state-of-the-art methods for GP model selection, on medium to large-scale datasets.
Our experiments show that model selection for computation-aware GPs trained on 1.8 million data points can be done within a few hours on a single GPU.
As a result of this work, Gaussian processes can be trained on large-scale datasets without significantly compromising their ability to quantify uncertainty---a fundamental prerequisite for optimal decision-making.

Computation-Aware Gaussian Processes: Model Selection And Linear-Time Inference

Abstract

Model selection in Gaussian processes scales prohibitively with the size of the training dataset, both in time and memory. While many approximations exist, all incur inevitable approximation error. Recent work accounts for this error in the form of computational uncertainty, which enables-at the cost of quadratic complexity-an explicit tradeoff between computational efficiency and precision. Here we extend this development to model selection, which requires significant enhancements to the existing approach, including linear-time scaling in the size of the dataset. We propose a novel training loss for hyperparameter optimization and demonstrate empirically that the resulting method can outperform SGPR, CGGP and SVGP, state-of-the-art methods for GP model selection, on medium to largescale datasets. Our experiments show that model selection for computation-aware GPs trained on 1.8 million data points can be done within a few hours on a single GPU. As a result of this work, Gaussian processes can be trained on large-scale datasets without significantly compromising their ability to quantify uncertaintya fundamental prerequisite for optimal decision-making.

1 Introduction

Gaussian Processes (GPs) remain a popular probabilistic model class, despite the challenges in scaling them to large datasets. Since both computational and memory resources are limited in practice, approximations are necessary for both inference and model selection. Among the many approximation methods, perhaps the most common approach is to map the data to a lower-dimensional representation. The resulting posterior approximations typically have a functional form similar to the exact GP posterior, except where posterior mean and covariance feature low-rank updates . This strategy can be explicit-by either defining feature functions (e.g. Nystr¨ om [1], RFF [2])-or a lower-dimensional latent inducing point space (e.g. SoR, DTC, FITC [3], SGPR [4], SVGP [5]), or implicit-by using an iterative numerical method (e.g. CGGP [6-10]). All of these methods then compute coefficients for this lower-dimensional representation from the full set of observations by direct projection (e.g. CGGP) or via an optimization objective (e.g. SGPR, SVGP).

While effective and widely used in practice, the inevitable approximation error adversely impacts predictions, uncertainty quantification, and ultimately downstream decision-making. Many proposed methods come with theoretical error bounds [e.g. 2, 11-14], offering insights into the scaling and asymptotic properties of each method. However, theoretical bounds often require too many assumptions about the data-generating process to offer 'real-world' guarantees [15], and in practice, the fidelity of the approximation is ultimately determined by the available computational resources.

-

2

0

2

Predictive

CholeskyGP

SVGP

CaGP-CG

CaGP-Opt

0

.

0

0

.

5

1

.

0

0

.

0

0

.

2

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

GP mean

GP uncertainty (latent + noise)

Training data

Posterior for Data-Generating Hyperparameters

One central pathology is overconfidence, which has been shown to be detrimental in key applications of GPs such as Bayesian optimization [e.g. variance starvation of RFF, 16], and manifests itself even in state-of-the-art variational methods like SVGP. SVGP, because it treats inducing variables as 'virtual observations', can be overconfident at the locations of the inducing points if they are not in close proximity to training data, which becomes increasingly likely in higher dimensions. This phenomenon can be seen in a toy example in Figure 1, where SVGP has near zero posterior variance at the inducing point away from the data. See also Section S5.1 for a more detailed analysis.

These approximation errors are a central issue in inference, but they are exacerbated in model selection, where errors compound and result in biased selections of hyperparameters [12, 17, 18]. Continuing the example, SVGP has been observed to overestimate the observation noise [18], which can lead to oversmoothing. This issue can also be seen in Figure 1, where the SVGP model produces a smoother posterior mean than the exact (Cholesky)GP and attributes most variation from the posterior mean to observational noise (see also Figure S3(b)). There have been efforts to understand these biases [18] and to mitigate the impact of approximation error on model selection for certain approximations [e.g. CGGP, 12], but overcoming these issues for SVGP remains a challenge.

Recently, Wenger et al. [19] introduced computation-aware Gaussian processes (CaGP), a class of GP approximation methods which-for a fixed set of hyperparameters-provably does not suffer from overconfidence. Like SVGP and the other approximations mentioned above, CaGP also relies on low-rank posterior updates. Unlike these other methods, however, CaGP's posterior updates are constructed to guarantee that its posterior variance is always larger than the exact GP variance. This conservative estimate can be interpreted as additional uncertainty quantifying the approximation error due to limited computation; i.e. computational uncertainty . However, so far CaGP has fallen short in demonstrating wallclock time improvements for posterior inference over variational methods and model selection has so far remained an open problem.

Contributions In this work, we extend computation-aware Gaussian processes by demonstrating how to perform inference in linear time in the number of training data, while maintaining its theoretical guarantees. Second, we propose a novel objective that allows model selection without a significant bias that would arise from naively conducting model selection on the projected GP. In detail, we enforce a sparsity constraint on the 'actions' of the method, which unlocks linear-time inference, in a way that is amenable to hardware acceleration. We optimize these actions end-to-end alongside the hyperparameters with respect to a custom training loss, to optimally retain as much information from the data as possible given a limited computational budget. The resulting hyperparameters are less prone to oversmoothing and attributing variation to observational noise, as can be seen in Figure 1, when compared to SVGP. We demonstrate that our approach is strongly competitive on large-scale data with state-of-the-art variational methods, such as SVGP, without inheriting their pathologies. As a consequence of our work, one can train GPs on up to 1 . 8 million data points in a few hours on a single GPU without adversely impacting uncertainty quantification.

2 Background

We aim to learn a latent function mapping from X ⊆ R d to Y ⊆ R given a training dataset X = ( x 1 , . . . , x n ) ∈ R n × d of n inputs x j ∈ R d and corresponding targets y = ( y 1 , . . . , y n ) T ∈ R n .

Gaussian Processes A Gaussian process f ∼ GP ( µ, K θ ) is a stochastic process with mean function µ and kernel K θ such that f = f ( X ) = ( f ( x 1 ) , . . . , f ( x n )) T ∼ N ( µ , K θ ) is jointly Gaussian with mean µ i = µ ( x i ) and covariance K ij = K θ ( x i , x j ) . The kernel K θ depends on hyperparameters θ ∈ R p , which we omit in our notation. Assuming y | f ( X ) ∼ N ( f ( X ) , σ 2 I ) , the posterior is a Gaussian process GP ( µ ⋆ , K ⋆ ) where the mean and covariance functions evaluated at a test input x ⋄ ∈ R d are given by

where ˆ K = K + σ 2 I and the representer weights are defined as v ⋆ = ˆ K -1 ( y -µ ) .

In model selection, the computational bottleneck when optimizing kernel hyperparameters θ is the repeated evaluation of the negative log -marginal likelihood

and its gradient. Computing (2) and its gradient via a Cholesky decomposition has time complexity O ( n 3 ) and requires O ( n 2 ) memory, which is prohibitive for large n .

Sparse Gaussian Process Regression (SGPR) [4] Given a set of m ≪ n inducing points Z = ( z 1 , . . . , z m ) T and defining u := f ( Z ) = ( f ( z 1 ) , . . . , f ( z m )) T , SGPR defines a variational approximation to the posterior through the factorization p ⋆ ( f ( · ) | y ) ≈ q ( f ( · )) = ∫ p ( f ( · ) | u ) q ( u ) d u , where q ( u ) is an m -dimensional multivariate Gaussian. The mean and covariance of q ( u ) (denoted as m := E q ( u ) , Σ := Cov q ( u ) ) are jointly optimized alongside the kernel hyperparameters θ using the evidence lower bound (ELBO) as an objective:

The inducing point locations Z can be either optimized as additional parameters during training or chosen a-priori, typically in a data-dependent way [see e.g. Sec. 7.2 of 20]. Following Titsias [4], ELBOoptimization and posterior inference both require O ( nm 2 ) computation and O ( nm ) memory, a significant improvement over the costs of exact GPs.

Stochastic Variational Gaussian Processes (SVGP) [5] SVGP extends SGPR to reduce complexity further to O ( m 3 ) computation and O ( m 2 ) memory. It accomplishes this reduction by replacing the first term in Equation (4) with an unbiased approximation

Following Hensman et al. [5], we optimize m , Σ alongside θ , Z through joint gradient updates. Because the asymptotic complexities no longer depend on n , SVGP can scale to extremely large datasets that would not be able to fit into computer/GPU memory.

Computation-aware Gaussian Process Inference (CaGP) [19] CaGP 1 maps the data y into a lower dimensional subspace defined by its actions S i ∈ R n × i on the data, which defines an approximate posterior GP ( µ i , K i ) with

where C i = S i ( S T i ˆ KS i ) -1 S T i ≈ K -1 is a low-rank approximation of the precision matrix and v i = C i ( y -µ ) ≈ v ⋆ approximates the representer weights. Both converge to the corresponding exact quantity as the number of iterations, equivalently the downdate rank, i → n . Note that the CaGP posterior only depends on the space spanned by the columns of S i , not the actual matrix (Lemma S4). Finally, the CaGP posterior can be computed in O ( n 2 i ) time and O ( ni ) memory.

CaGP captures the approximation error incurred by limited computational resources as additional uncertainty about the latent function. More precisely, for any data-generating function y ∈ H K σ in the RKHS H K σ defined by the kernel, the worst-case squared error of the corresponding approximate posterior mean µ y i is equal to the approximate predictive variance (see Theorem S2):

This guarantee is identical to one for the exact GP posterior mean and variance (see Theorem S1), except with the approximate quantities instead. 2 Additionally, it holds that CaGP's marginal (predictive) variance is always larger or equal to the (predictive) variance of the exact GP and monotonically decreasing, i.e. K i ( x ⋄ , x ⋄ ) ≥ K j ( x ⋄ , x ⋄ ) ≥ K n ( x ⋄ , x ⋄ ) = K ⋆ ( x ⋄ , x ⋄ ) for i ≤ j ≤ n (see Proposition S1). Therefore, given fixed hyperparameters, CaGP is guaranteed to never be overconfident and as the computational budget increases, the precision of its estimate increases. Here we call such a posterior computation-aware , and we will extend the use of this object to model selection.

3 Model Selection in Computation-Aware Gaussian Processes

Model selection for GPs most commonly entails maximizing the evidence log p ( y | θ ) as a function of the kernel hyperparameters θ ∈ R p . As with posterior inference, evaluating this objective and its gradient is computationally prohibitive in the large-scale setting. Therefore, our central goal will be to perform model selection for computation-aware Gaussian processes in order to scale to a large number of training data while avoiding the introduction of pathologies via approximation.

We begin by viewing the computation-aware posterior as exact inference assuming we can only observe i linear projections ˜ y of the data defined by (linearly independent) actions S i ∈ R n × i , i.e.

is the action matrix with orthonormalized columns. The corresponding likelihood is given by

As we show in Lemma S1, the resulting Bayesian posterior is then given by Equation (5). Recall that the CaGP posterior only depends on the column space of the actions S i (see Lemma S4), which is why Equation (5) can be written in terms of S i directly rather than using S ′ i . 3

Projected-Data Log Marginal Likelihood The reinterpretation of the computation-aware posterior as exact Bayesian inference immediately suggests using evidence maximization for the projected data ˜ y = S ′ i T y ∈ R i as the model selection criterion, leading to the following loss (see Lemma S2):

Equation (9) involves a Gaussian random variable of dimension i ≪ n , and so all previously intractable quantities in Equation (2) (i.e. the inverse and determinant) are now cheap to compute. Analogous to the CaGP posterior, we can express the projected-data log marginal likelihood fully in terms of the actions S i without having to orthonormalize, which results in an additional term penalizing colinearity. Unfortunately, this training loss does not lead to good generalization performance, as there is only training signal in the i -dimensional space spanned by the actions S i the data are projected onto. Specifically, the quadratic loss term only promotes fitting the projected data ˜ y , not all observations y . See Figures S1 and S2 for experimental validation of this claim.

ELBO Training Loss Motivated by this observation, we desire a tractable training loss that encourages maximizing the evidence for the entire set of targets y . Importantly though, we need to accomplish this evidence maximization without incurring prohibitive O ( n 3 ) computational cost. We define a variational objective, using the computation-aware posterior q i ( f | y , θ ) in Equation (5) to define a variational family Q := { q i ( f ) = N ( f ; µ i ( X ) , K i ( X , X )) | S ∈ R n × i } parametrized by the action matrix S . We can then specify a (negative) evidence lower bound (ELBO) as follows:

This loss promotes learning the same hyperparameters as if we were to maximize the computationally intractable evidence log p ( y | θ ) while minimizing the error due to posterior approximation q i ( f ) ≈ p ⋆ ( f | y , θ ) . In the computation-aware setting, this translates to minimizing computational uncertainty, which captures this inevitable error.

Although both the evidence and KL terms of the ELBO involve computationally intractable terms, these problematic terms cancel out when combined. This results in an objective that costs the same as evaluating CaGP's predictive distribution, i.e.

where ˜ v i = ( S T ˆ KS ) -1 S T ( y -µ ) . For a derivation of this expression of the loss see Lemma S3. If we compare the training loss ℓ ELBO CaGP in Equation (10) with the projected-data NLL in Equation (9), there is an explicit squared loss penalty on the entire data y , rather than just for the projected data ˜ y , resulting in better generalization as Figures S1 and S2 show on synthetic data. In our experiments, this objective was critical to achieving state-of-the-art performance.

4 Choice of Actions

So far we have not yet specified the actions S ∈ R n × i mapping the data to a lower-dimensional space. Ideally, we would want to optimally compress the data both for inference and model selection.

Posterior Entropy Minimization We can interpret choosing actions S as a form of active learning, where instead of just observing individual datapoints, we allow ourselves to observe linear combinations of the data y . Taking an information-theoretic perspective [21], we would then aim to choose actions such that uncertainty about the latent function is minimized . In fact, we show in Lemma S5 that given a prior f ∼ GP ( µ, K ) for the latent function and a budget of i actions S , the actions that minimize the entropy of the posterior at the training data

are the topi eigenvectors s 1 , . . . , s i of ˆ K in descending order of the eigenvalue magnitude (see also Zhu et al. [22]). Unfortunately, computing these actions is just as prohibitive computationally as computing the intractable GP posterior.

(Conjugate) Gradient / Residual Actions Due to the intractability of choosing actions to minimize posterior entropy, we could try to do so approximately. The Lanczos algorithm [23] is an iterative method to approximate the eigenvalues and eigenvectors of symmetric positive-definite matrices. Given an appropriate seed vector, the space spanned by its approximate eigenvectors is equivalent to the span of the gradients / residuals r i = ( y -µ ) -ˆ Kv i -1 of the method of Conjugate Gradients (CG) [24] when used to iteratively compute an approximation v i ≈ v ⋆ = ˆ K -1 ( y -µ ) to the representer weights v ⋆ . We show in Lemma S4 that the CaGP posterior only depends on the span of its actions. Therefore choosing approximate eigenvectors computed via the Lanczos process as actions is equivalent to using CG residuals. This allows us to reinterpret CaGP-CG, as introduced by Wenger et al. [19], as approximately minimizing posterior entropy. 4 See Section S3.1 for details.

0

.

0

0

.

5

1

.

0

Eigenvectors

Action

S

1

Action

S

2

Action

S

3

Action

S

4

Action

S

5

0

.

0

0

.

5

1

.

0

CaGP-CG

0

1

0

.

0

0

.

5

1

.

0

CaGP-Opt

0

1

0

1

0

1

0

1

0

50

100

Epoch

3

4

5

6

Grassman distance

d

(

S

,

V

eigen

)

Method

Random

CaGP-CG

CaGP-Opt

As Figure 2 illustrates, CG actions are similar to the topi eigenspace all throughout hyperparameter optimization. However, this choice of actions focuses exclusively on posterior inference and incurs quadratic time complexity O ( n 2 i ) .

Learned Sparse Actions So far in our action choices we have entirely ignored model selection and tried to choose optimal actions assuming fixed kernel hyperparameters. The second contribution of this work, aside from demonstrating how to perform model selection, is recognizing that the actions should be informed by the outer optimization loop for the hyperparameters. We thus optimize the actions alongside the hyperparameters end-to-end , meaning the training loss for model selection defines what data projections are informative. This way the actions are adaptive to the hyperparameters without spending unnecessary budget on computing approximately optimal actions for the current choice of hyperparameters. Specifically, the actions are chosen by optimizing ℓ ELBO CaGP as a function of the hyperparameters θ and the actions S , s.t.

Naively this approach introduces an n × i dimensional optimization problem, which in general is computationally prohibitive. To keep the computational cost low and to optimally leverage hardware acceleration via GPUs, we impose a sparse block structure on the actions (see Eq. 14) where each block is a column vector s ′ j ∈ R k × 1 with k = n/i entries such that the total number of trainable action parameters, i.e. non-zero entries nnz( S ) = k · i = n , equals the number of training data. Due to the sparsity, these actions cannot perfectly match the maxi-

mum eigenvector actions. Nevertheless, we see in Figure 2 that optimizing these sparse actions in conjunction with hyperparameter optimization produces a nontrivial alignment with optimal action choice minimizing posterior entropy. Importantly, the sparsity constraint not only reduces the dimensionality of the optimization problem, but crucially also reduces the time complexity of posterior inference and model selection to linear in the number of training data points.

4.1 Algorithms and Computational Complexity

We give algorithms both for iteratively constructed dense actions (Algorithm S1), as used in CaGPCG, and for sparse batch actions (Algorithm S2), as used for CaGP-Opt, in the supplementary

material. 5 The time complexity is O ( n 2 i ) for dense actions and O ( ni max( i, k )) for sparse actions, where k is the maximum number of non-zero entries per column of S i . Both have the same linear memory requirement: O ( ni ) . Since the training loss ℓ ELBO CaGP only involves terms that are also present in the posterior predictive, both model selection and predictions incur the same complexity.

4.2 Related Work

Computational Uncertainty and Probabilistic Numerics All CaGP variants discussed in this paper fall into the category of probabilistic numerical methods [25-27], which aim to quantify approximation error arising from limited computational resources via additional uncertainty about the quantity of interest [e.g. 28-31]. Specifically, the iterative formulation of CaGP (i.e. Algorithm S1) originally proposed by Wenger et al. [19] employs a probabilistic linear solver [32-35].

Scalable GPs with Lower-Bounded Log Marginal Likelihoods Numerous scalable GP approximations beyond those in Sections 1 and 2 exist; see Liu et al. [36] for a comprehensive review. Many GP models [e.g., 4, 5, 37-39] learn hyperparameters through maximizing variational lower bounds in the same spirit as SGPR, SVGP and our method. Similar to our work, interdomain inducing point methods [40-42] learn a variational posterior approximation on a small set of linear functionals applied to the latent GP. However, unlike our method, their resulting approximate posterior is usually prone to underestimating uncertainty in the same manner as SGPR and SVGP. Finally, similar to our proposed training loss for CaGP-CG, Artemev et al. [43] demonstrate how one can use the method of conjugate gradients to obtain a tighter lower bound on the log marginal likelihood.

GP Approximation Biases and Computational Uncertainty Scalable GP methods inevitably introduce approximation error and thus yield biased hyperparameters and predictive distributions, with an exception of Potapczynski et al. [12] which trade bias for increased variance. Numerous works have studied pathologies associated with optimizing variational lower bounds, especially in the context of SVGP [12, 16-18], and various remedies have been proposed. In order to mitigate biases from approximation, several works alternatively propose replacing variational lower bounds with alternative model selection objectives, including leave-one-out cross-validation [44] and losses that directly target predictive performance [45, 46].

5 Experiments

We benchmark the generalization of computation-aware GPs with two different action choices, CaGP-Opt (ours) and CaGP-CG [19], using our proposed training objective in Equation (10) on a range of UCI datasets for regression [53]. We compare against SVGP [5], often considered to be state-of-the-art for large-scale GP regression. Per recommendations by Ober et al. [15], we also include SGPR [4] as a strong baseline for all datasets where this is computationally feasible. We also train Conjugate Gradient-based GPs (CGGP) [e.g. 7, 9, 10] using the training procedure proposed by Wenger et al. [10]. Note that CaGP-CG recovers CGGP in its posterior mean and produces nearly identical predictive error at half the computational cost for inference [Sec. 2.1 & 4 of 19], which is why the main difference between CaGP-CG and CGGP in our experiments is the training objective. Finally, we also train an exact CholeskyGP on the smallest datasets, where this is still feasible.

Experimental Details All datasets were randomly partitioned into train and test sets using a (0 . 9 , 0 . 1) split for five random seeds. We used a zero-mean GP prior and a Mat´ ern( 3 / 2 ) kernel with an outputscale o 2 and one lengthscale per input dimension l 2 j , as well as a scalar observation noise for the likelihood σ 2 , s.t. θ = ( o, l 1 , . . . , l d , σ ) ∈ R d +2 . We used the existing implementations of SGPR, SVGP and CGGP in GPyTorch [7] and also implemented CaGP in this framework (see Section S4.2 for our open-source implementation). For SGPR and SVGP we used m = 1024 inducing points and for CGGP, CaGP-CG and CaGP-Opt we chose i = 512 . We optimized the hyperparameters θ either with Adam [54] for a maximum of 1000 epochs in float32 or with LBFGS [55] for 100 epochs in float64 , depending on the method and problem scale. On the largest dataset 'Power', we used 400 epochs for SVGP and 200 for CaGP-Opt due to resource constraints. For SVGP we used a batch size of 1024 throughout. We scheduled the learning rate via PyTorch 's [56] LinearLR(end factor=0.1) scheduler for all methods and performed a hyperparameter sweep

for the (initial) learning rate. All experiments were run on an NVIDIA Tesla V100-PCIE-32GB GPU, except for 'Power', where we used an NVIDIA A100 80GB PCIe GPU to have sufficient memory for CaGP-Opt with i = 512 . Our exact experiment configuration can be found in Table S1.

Evaluation Metrics We evaluate the generalization performance once per epoch on the test set by computing the (average) negative log-likelihood (NLL) and the root mean squared error (RMSE), as well as recording the wallclock runtime. Runtime is measured at the epoch with the best average performance across random seeds.

CaGP-Opt Matches or Outperforms SVGP Table 1 and Figure 3 show generalization performance of all methods for the best choice of learning rate. In terms of both NLL and RMSE, CaGPOpt outperforms or matches the variational baselines SGPR and SVGP at comparable runtime (except on 'Bike'). SGPR remains competitive for smaller datasets; however, it does not scale to the largest datasets. There are some datasets and metrics in which specific methods dominate, for example on 'Bike' SGPR outperforms all other approximations, while on 'Protein' methods based on CG, i.e. CGGP and CaGP-CG, perform the best. However, CaGP-Opt consistently performs either best or second best and scales to over a million datapoints. These results are quite remarkable for numerous reasons. First, CaGP is comparable in runtime to SVGP on large datasets despite the fact that it incurs a linear-time computational complexity while SVGP is constant time. 6 Second, while all of the methods we compare approximate the GP posterior with low-rank updates, CaGP-Opt (with

-

2

.

5

0

.

0

2

.

5

Train Loss

Parkinsons

-

2

0

2

Bike

1

.

0

1

.

5

2

.

0

Protein

0

2

KEGGu

0

1

2

Road

-

2

0

2

Power

-

4

-

2

0

Test NLL

-

2

0

0

.

8

0

.

9

1

.

0

-

0

.

5

0

.

0

0

.

5

0

.

0

0

.

5

1

.

0

-

2

0

0

1min2min

Time

10

-

2

10

0

Test RMSE

0

4min

8min

Time

10

-

2

10

-

1

10

0

0

8min16min

Time

0

.

5

0

.

6

0

.

7

0

20min 40min

Time

0

.

2

0

.

4

0

1h

2h

Time

0

.

5

1

.

0

0

1.5h 3h 4.5h

Time

10

-

1

10

0

CholeskyGP

SGPR

SVGP

CGGP

CaGP-CG

CaGP-Opt

i = 512 ) here uses half the rank of SGPR/SVGP m = 1024 . Nevertheless, CaGP-Opt is able to substantially outperform SVGP even on spatial datasets like 3DRoad where low-rank posterior updates are often poor [57]. These results suggest that CaGP-Opt can be a more efficient approximation than inducing point methods, and that low-rank GP approximations may be more applicable than previously thought [58, 59]. Figure 3 shows the NLL and RMSE learning curves for the best choice of learning rate per method. CaGP-Opt often shows a 'ramp-up' phase, compared to SVGP, but then improves or matches its generalization performance. This gap is particularly large on 'Road', where CaGP-Opt is initially worse than SVGP but dominates in the second half of training.

SVGP Overestimates Observation Noise and (Often) Lengthscale In Figure S5 we show that SVGP typically learns larger observation noise than other methods as suggested by previous work [18, 45] and hinted at by observations on synthetic data in Figure 1 and Figure S3(b). Additionally on larger datasets SVGP also often learns large lengthscales, which in combination with a large observation noise can lead to an oversmoothing effect. In contrast, CaGP-Opt generally learns lower observational noise than SVGP. Of course, learning a small observation noise, in particular, is important for achieving low RMSE and thus also NLL, and points to why we should expect CaGPOpt to outperform SVGP. These hyperparameter results suggest that CaGP-Opt interprets more of the data as signal, while SVGP interprets more of the data as noise.

0

1.5h

3h

4.5h

Time

10

-

2

10

-

1

Coverage Error

SVGP

CaGP-Opt

CaGP Improves Uncertainty Quantification Over SVGP A key advantage of CaGP-Opt and CaGP-CG is that their posterior uncertainty estimates capture both the uncertainty due to limited data and due to limited computation. To that end, we assess the frequentist coverage of CaGP-Opt's uncertainty estimates. We report the absolute difference between a desired coverage percentage α

and the fraction of data that fall into the α credible interval of the CaGP-Opt posterior; i.e. ε α coverage = | α -1 n test ∑ n test i =1 1 ( y i ∈ I α q ( x i ) ) | . Figure 4 compares the 95% coverage error for both CaGP-Opt and SVGP on the largest dataset ('Power'). From this plot, we see that the CaGP credible intervals are more aligned with the desired coverage. We hypothesize that these results reflect the different uncertainty properties of the methods: CaGP-Opt overestimates posterior uncertainty while SVGP is prone towards overconfidence.

6 Conclusion

In this work, we introduced strategies for model selection and posterior inference for computationaware Gaussian processes, which scale linearly with the number of training data rather than quadratically. The key technical innovations being a sparse projection of the data, which balances minimizing posterior entropy and computational cost, and a scalable way to optimize kernel hyperparameters, both of which are amenable to GPU acceleration. All together, these advances enable competitive or improved performance over previous approximate inference methods on large-scale datasets, in terms of generalization and uncertainty quantification. Remarkably, our method outperforms SVGP-often considered the de-facto GP approximation standard- even when compressing the data into a space of half the dimension of the variational parameters. Finally, we also demonstrate that computation-aware GPs avoid many of the pathologies often observed in inducing point methods, such as overconfidence and oversmoothing.

Limitations While CaGP-Opt obtains the same linear time and memory costs as SGPR, it is not amenable to stochastic minibatching and thus cannot achieve the constant time/memory capabilities of SVGP. In practice, this asymptotic difference does not result in substantially different wall clock times, as SVGP requires many more optimizer steps than CaGP-Opt due to batching. (Indeed, on many datasets we find that CaGP-Opt is faster.) CaGP-Opt nevertheless requires larger GPUs than SVGP on datasets with more than a million data points. Moreover, tuning CaGP-Opt requires choosing the appropriate number of actions (i.e. the rank of the approximate posterior update), though we note that most scalable GP approximations have a similar tunable parameter (e.g. number of inducing points). Perhaps the most obvious limitation is that CaGP, unlike SVGP, is limited to GP regression with a conjugate observational noise model. We leave extensions to classification and other non-conjugate likelihoods as future work.

Outlook and Future Work An immediate consequence of this work is the ability to apply computation-aware Gaussian processes to 'real-world' problems, as our approach solves CaGP's open problems of model selection and scalability. Looking forward, an exciting future vision is a general framework for problems involving a Gaussian process model with a downstream task where the actions are chosen optimally, given resource constraints, to solve said task. Future work will pursue this direction beyond Gaussian likelihoods to non-conjugate models and downstream tasks such as Bayesian optimization.

Acknowledgments and Disclosure of Funding

JWand JPC are supported by the Gatsby Charitable Foundation (GAT3708), the Simons Foundation (542963), the NSF AI Institute for Artificial and Natural Intelligence (ARNI: NSF DBI 2229929) and the Kavli Foundation. JG and KW are supported by the NSF (IIS-2145644, DBI-2400135). PH gratefully acknowledges co-funding by the European Union (ERC, ANUBIS, 101123955). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. PH is a member of the Machine Learning Cluster of Excellence, funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy - EXC number 2064/1 - Project number 390727645; he also gratefully acknowledges the German Federal Ministry of Education and Research (BMBF) through the T¨ ubingen AI Center (FKZ: 01IS18039A); and funds from the Ministry of Science, Research and Arts of the State of Baden-W¨ urttemberg. GP acknowledges support from NSERC and the Canada CIFAR AI Chair program.

References

C. Williams and M. Seeger. 'Using the Nystr¨ om method to speed up kernel machines'. In: Advances in Neural Information Processing Systems (NeurIPS) . 2001 (cit. on p. 1).

A. Rahimi and B. Recht. 'Random Features for Large-Scale Kernel Machines'. In: Advances in Neural Information Processing Systems (NeurIPS) . 2007 (cit. on p. 1).

J. Qui˜ nonero-Candela and C. E. Rasmussen. 'A unifying view of sparse approximate Gaussian process regression'. In: Journal of Machine Learning Research 6 (2005) (cit. on p. 1).

M. Titsias. 'Variational learning of inducing variables in sparse Gaussian processes'. In: International Conference on Artificial Intelligence and Statistics (AISTATS) . PMLR. 2009, pp. 567-574 (cit. on pp. 1, 3, 7).

J. Hensman, N. Fusi, and N. D. Lawrence. 'Gaussian processes for big data'. In: Conference on Uncertainty in Artificial Intelligence (UAI) . 2013, pp. 282-290 (cit. on pp. 1, 3, 7).

M. Gibbs. 'Bayesian Gaussian processes for classification and regression'. PhD thesis. 1997 (cit. on p. 1).

J. R. Gardner, G. Pleiss, D. Bindel, K. Q. Weinberger, and A. G. Wilson. 'GPyTorch: Blackbox matrix-matrix Gaussian process inference with GPU acceleration'. In: Advances in Neural Information Processing Systems (NeurIPS) . 2018, pp. 7576-7586 (cit. on pp. 1, 5, 7).

G. Pleiss, J. Gardner, K. Weinberger, and A. G. Wilson. 'Constant-time predictive distributions for Gaussian processes'. In: International Conference on Machine Learning (ICML) . 2018, pp. 4114-4123 (cit. on p. 1).

K. A. Wang, G. Pleiss, J. R. Gardner, S. Tyree, K. Q. Weinberger, and A. G. Wilson. 'Exact Gaussian processes on a million data points'. In: Advances in Neural Information Processing Systems (NeurIPS) 32 (2019) (cit. on pp. 1, 5, 7).

J. Wenger, G. Pleiss, P. Hennig, J. P. Cunningham, and J. R. Gardner. 'Preconditioning for Scalable Gaussian Process Hyperparameter Optimization'. In: International Conference on Machine Learning (ICML) . 2022 (cit. on pp. 1, 5, 7).

F. Bach. 'On the equivalence between kernel quadrature rules and random feature expansions'. In: Journal of Machine Learning Research 18.21 (2017), pp. 1-38 (cit. on p. 1).

A. Potapczynski, L. Wu, D. Biderman, G. Pleiss, and J. P. Cunningham. 'Bias-Free Scalable Gaussian Processes via Randomized Truncations'. In: International Conference on Machine Learning (ICML) . 2021 (cit. on pp. 1, 2, 7).

D. R. Burt, C. E. Rasmussen, and M. van der Wilk. 'Rates of Convergence for Sparse Variational Gaussian Process Regression'. In: International Conference on Machine Learning (ICML) . 2019. URL: http://arxiv.org/abs/1903.03571 (cit. on p. 1).

M. Kang, F. Sch¨ afer, J. Guinness, and M. Katzfuss. Asymptotic properties of Vecchia approximation for Gaussian processes . 2024. DOI: 10.48550/arXiv.2401.15813 (cit. on p. 1).

S. W. Ober, D. R. Burt, A. Artemev, and M. van der Wilk. 'Recommendations for Baselines and Benchmarking Approximate Gaussian Processes'. In: NeurIPS Workshop on Gaussian Processes, Spatiotemporal Modeling, and Decision-making Systems . 2022. URL: https:// gp-seminar-series.github.io/assets/camera_ready/62.pdf (cit. on pp. 1, 7).

Z. Wang, C. Gehring, P. Kohli, and S. Jegelka. 'Batched Large-scale Bayesian Optimization in High-dimensional Spaces'. In: International Conference on Artificial Intelligence and Statistics (AISTATS) . 2018. DOI: 10.48550/arXiv.1706.01445 . URL: http://arxiv. org/abs/1706.01445 (cit. on pp. 2, 7).

R. E. Turner and M. Sahani. 'Two problems with variational expectation maximisation for time series models'. In: Bayesian Time Series Models . Cambridge University Press, 2011, pp. 104-124. DOI: 10.1017/CBO9780511984679.006 (cit. on pp. 2, 7).

M. Bauer, M. van der Wilk, and C. E. Rasmussen. 'Understanding probabilistic sparse Gaussian process approximations'. In: Advances in Neural Information Processing Systems (NeurIPS) . Vol. 29. 2016 (cit. on pp. 2, 7, 9).

J. Wenger, G. Pleiss, M. Pf¨ ortner, P. Hennig, and J. P. Cunningham. 'Posterior and Computational Uncertainty in Gaussian processes'. In: Advances in Neural Information Processing Systems (NeurIPS) . 2022. DOI: 10.48550/arXiv.2205.15449 (cit. on pp. 2, 3, 5, 7, 16, 17, 23).

D. R. Burt, C. E. Rasmussen, and M. v. d. Wilk. 'Convergence of Sparse Variational Inference in Gaussian Processes Regression'. In: Journal of Machine Learning Research (Aug. 2020). DOI: 10.48550/arXiv.2008.00323 (cit. on p. 3).

N. Houlsby, F. Husz´ ar, Z. Ghahramani, and M. Lengyel. 'Bayesian active learning for classification and preference learning'. In: arXiv (2011). URL: https://arxiv.org/abs/1112. 5745 (cit. on pp. 5, 21).

H. Zhu, C. K. Williams, R. Rohwer, and M. Morciniec. 'Gaussian regression and optimal finite dimensional linear models'. In: Neural Networks and Machine Learning . 1997 (cit. on p. 5).

C. Lanczos. An iteration method for the solution of the eigenvalue problem of linear differential and integral operators . United States Government Press Office Los Angeles, CA, 1950 (cit. on pp. 5, 21).

M. R. Hestenes and E. Stiefel. 'Methods of conjugate gradients for solving linear systems'. In: Journal of Research of the National Bureau of Standards 49 (1952) (cit. on pp. 5, 21).

P. Hennig, M. A. Osborne, and M. Girolami. 'Probabilistic numerics and uncertainty in computations'. In: Proceedings of the Royal Society of London A: Mathematical, Physical and Engineering Sciences 471.2179 (2015) (cit. on p. 7).

J. Cockayne, C. Oates, T. Sullivan, and M. Girolami. 'Bayesian probabilistic numerical methods'. In: SIAM Review 61.4 (2019), pp. 756-789 (cit. on p. 7).

P. Hennig, M. A. Osborne, and H. P. Kersting. Probabilistic Numerics: Computation as Machine Learning . Cambridge University Press, 2022. ISBN: 978-1-316-68141-1. DOI: 10 . 1017/9781316681411 (cit. on p. 7).

M. Pf¨ ortner, I. Steinwart, P. Hennig, and J. Wenger. Physics-Informed Gaussian Process Regression Generalizes Linear PDE Solvers . 2023. DOI: 10.48550/arXiv.2212.12474 (cit. on p. 7).

L. Tatzel, J. Wenger, F. Schneider, and P. Hennig. Accelerating Generalized Linear Models by Trading off Computation for Uncertainty . 2024. DOI: 10.48550/arXiv.2310.20285 (cit. on p. 7).

M. Pf¨ ortner, J. Wenger, J. Cockayne, and P. Hennig. Computation-Aware Kalman Filtering and Smoothing . 2024. DOI: 10.48550/arxiv.2405.08971 (cit. on p. 7).

D. Hegde, M. Adil, and J. Cockayne. Calibrated Computation-Aware Gaussian Processes . 2024. DOI: 10.48550/arXiv.2410.08796 (cit. on p. 7).

P. Hennig. 'Probabilistic Interpretation of Linear Solvers'. In: SIAM Journal on Optimization 25.1 (2015), pp. 234-260 (cit. on p. 7).

J. Cockayne, C. Oates, I. C. Ipsen, and M. Girolami. 'A Bayesian Conjugate Gradient Method'. In: Bayesian Analysis 14.3 (2019), pp. 937-1012 (cit. on p. 7).

S. Bartels, J. Cockayne, I. C. Ipsen, and P. Hennig. 'Probabilistic linear solvers: A unifying view'. In: Statistics and Computing 29.6 (2019), pp. 1249-1263 (cit. on p. 7).

J. Wenger and P. Hennig. 'Probabilistic Linear Solvers for Machine Learning'. In: Advances in Neural Information Processing Systems (NeurIPS) . 2020. DOI: 10.48550/arXiv.2010. 09691 . URL: http://arxiv.org/abs/2010.09691 (cit. on p. 7).

H. Liu, Y.-S. Ong, X. Shen, and J. Cai. 'When Gaussian process meets big data: A review of scalable GPs'. In: Transactions on Neural Networks and Learning Systems 31.11 (2020), pp. 4405-4423 (cit. on p. 7).

J. Hensman, A. Matthews, and Z. Ghahramani. 'Scalable variational Gaussian process classification'. In: International Conference on Artificial Intelligence and Statistics (AISTATS) . PMLR, 2015, pp. 351-360 (cit. on p. 7).

H. Salimbeni, C.-A. Cheng, B. Boots, and M. Deisenroth. 'Orthogonally Decoupled Variational Gaussian Processes'. In: Advances in Neural Information Processing Systems (NeurIPS) . Vol. 31. 2018 (cit. on p. 7).

L. Wu, G. Pleiss, and J. P. Cunningham. 'Variational nearest neighbor Gaussian process'. In: International Conference on Machine Learning (ICML) . PMLR, 2022, pp. 24114-24130 (cit. on p. 7).

J. Hensman, N. Durrande, and A. Solin. 'Variational Fourier Features for Gaussian Processes'. In: Journal of Machine Learning Research 18.151 (2018), pp. 1-52 (cit. on p. 7).

V. Dutordoir, N. Durrande, and J. Hensman. 'Sparse Gaussian Processes with Spherical Harmonic Features'. In: International Conference on Machine Learning (ICML) . Vol. 119. 2020, pp. 2793-2802 (cit. on p. 7).

M. Van der Wilk, V. Dutordoir, S. John, A. Artemev, V. Adam, and J. Hensman. A framework for interdomain and multioutput Gaussian processes . 2020. DOI: 10.48550/arXiv.2003. 01115 (cit. on p. 7).

A. Artemev, D. R. Burt, and M. van der Wilk. 'Tighter Bounds on the Log Marginal Likelihood of Gaussian Process Regression Using Conjugate Gradients'. In: International Conference on Machine Learning (ICML) . 2021 (cit. on p. 7).

M. Jankowiak and G. Pleiss. Scalable Cross Validation Losses for Gaussian Process Models . 2022. DOI: 10.48550/arXiv.2105.11535 (cit. on p. 7).

M. Jankowiak, G. Pleiss, and J. Gardner. 'Parametric Gaussian Process Regressors'. In: International Conference on Machine Learning (ICML) . Vol. 119. 2020, pp. 4702-4712 (cit. on pp. 7, 9).

Y. Wei, R. Sheth, and R. Khardon. 'Direct Loss Minimization for Sparse Gaussian Processes'. In: International Conference on Artificial Intelligence and Statistics (AISTATS) . Vol. 130. 2021, pp. 2566-2574 (cit. on p. 7).

A. Tsanas and M. Little. Parkinsons Telemonitoring . UCI Machine Learning Repository. 2009. DOI: 10.24432/C5ZS3N (cit. on p. 8).

H. Fanaee-T and J. Gama. Bike Sharing . UCI Machine Learning Repository. 2013. DOI: 10. 24432/C5W894 (cit. on p. 8).

P. Rana. Physicochemical Properties of Protein Tertiary Structure . UCI Machine Learning Repository. 2013. DOI: 10.24432/C5QW3H (cit. on p. 8).

M. Naeem and S. Asghar. KEGG Metabolic Reaction Network (Undirected) . UCI Machine Learning Repository. 2011. DOI: 10.24432/C5G609 (cit. on p. 8).

M. Kaul. 3D Road Network (North Jutland, Denmark) . UCI Machine Learning Repository. 2013. DOI: 10.24432/C5GP51 (cit. on p. 8).

G. Hebrail and A. Berard. Individual Household Electric Power Consumption . UCI Machine Learning Repository. 2006. DOI: 10.24432/C58K54 (cit. on p. 8).

M. Kelly, R. Longjohn, and K. Nottingham. The UCI Machine Learning Repository . 2017. URL: https://archive.ics.uci.edu (cit. on p. 7).

D. P. Kingma and J. Ba. 'Adam: A method for stochastic optimization'. In: International Conference on Learning Representations (ICLR) (2015) (cit. on p. 7).

J. Nocedal. 'Updating quasi-Newton matrices with limited storage'. In: Mathematics of Computation 35.151 (1980), pp. 773-782 (cit. on p. 7).

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. K¨ opf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. 'PyTorch: An Imperative Style, High-Performance Deep Learning Library'. In: Advances in Neural Information Processing Systems (NeurIPS) . 2019. DOI: 10.48550/arXiv.1912.01703 . URL: http: //arxiv.org/abs/1912.01703 (cit. on p. 7).

G. Pleiss, M. Jankowiak, D. Eriksson, A. Damle, and J. Gardner. 'Fast matrix square roots with applications to Gaussian processes and Bayesian optimization'. In: Advances in Neural Information Processing Systems (NeurIPS) . 2020, pp. 22268-22281 (cit. on p. 9).

A. Datta, S. Banerjee, A. O. Finley, and A. E. Gelfand. 'Hierarchical nearest-neighbor Gaussian process models for large geostatistical datasets'. In: Journal of the American Statistical Association 111.514 (2016), pp. 800-812 (cit. on p. 9).

M. Katzfuss and J. Guinness. 'A General Framework for Vecchia Approximations of Gaussian Processes'. In: Statistical Science 36.1 (2021). DOI: 10.1214/19-sts755 (cit. on p. 9).

M. Kanagawa, P. Hennig, D. Sejdinovic, and B. K. Sriperumbudur. Gaussian Processes and Kernel Methods: A Review on Connections and Equivalences . arXiv:1807.02582 [cs, stat]. 2018. DOI: 10.48550/arXiv.1807.02582 (cit. on p. 16).

V. Wild, M. Kanagawa, and D. Sejdinovic. 'Connections and Equivalences between the Nystr¨ om Method and Sparse Variational Gaussian Processes'. In: arXiv (2021). URL: http: //arxiv.org/abs/2106.01121 (cit. on pp. 16, 17).

G. H. Golub and C. F. Van Loan. Matrix computations . John Hopkins University Press, 2012 (cit. on pp. 21, 25).

G. Meurant and Z. Strakoˇ s. 'The Lanczos and conjugate gradient algorithms in finite precision arithmetic'. en. In: Acta Numerica 15 (May 2006), pp. 471-542. DOI: 10.1017/ S096249290626001X (cit. on p. 21).

Supplementary Material

This supplementary material contains additional results and in particular proofs for all theoretical statements. References referring to sections, equations or theorem-type environments within this document are prefixed with 'S', while references to, or results from, the main paper are stated as is.

S1 Theoretical Results

S1.1 Alternative Derivation of CaGP Posterior

Lemma S1 (CaGP Inference as Exact Inference Given a Modified Observation Model) Given a Gaussian process prior f ∼ GP ( µ, K ) and training data ( X , y ) the computation-aware GP posterior GP ( µ i , K i ) (see Equation (5) ) with linearly independent and fixed actions S is equivalent to an exact batch GP posterior ( f | ˜ y ) given data ˜ y = S ′ T y observed according to the likelihood ˜ y | f ( X ) ∼ N ( S ′ T f ( X ) , σ 2 I ) , where S ′ = S chol( S T S ) -T .

Proof. First note that by definition S ′ has orthonormal columns, since

Now by basic properties of Gaussian distributions, we have for arbitrary X ⋄ ∈ R n ⋄ × d that

is jointly Gaussian, where we used that I = S ′ T S ′ .

Therefore we have that ( f ( X ⋄ ) | ˜ y ) ∼ N ( µ i ( X ⋄ ) , K i ( X ⋄ , X ⋄ )) with

which is equivalent to the definition of the CaGP posterior in Equation (5). This proves the claim.

S1.2 Worst Case Error Interpretations of the Variance of Exact GPs, CaGPs and SVGPs

In order to understand the impact of approximation on the uncertainty quantification of both CaGP and SVGP, it is instructive to compare the theoretical guarantees they admit, when the latent function is assumed to be in the RKHS of the kernel. In the context of model selection, this corresponds to the ideal case where the optimization has converged to the ground truth hyperparameters.

Let f ∼ GP (0 , K ) be a Gaussian process with kernel K and define the observed process y ( · ) = f ( · ) + σε ( · ) where ε ∼ GP (0 , δ ) is white noise with noise level σ 2 > 0 , i.e. δ ( x , x ′ ) = 1 ( x = x ′ ) . Consequently, the covariance kernel of the data-generating process y ( · ) is given by K σ ( x , x ′ ) := K ( x , x ′ ) + σ 2 δ ( x , x ′ ) and we denote the corresponding RKHS as H K σ .

For exact Gaussian process inference, the pointwise (relative) worst-case squared error of the posterior mean is precisely given by the posterior predictive variance.

Theorem S1 (Worst Case Error Interpretation of GP Variance [60])

̸

Given a set of training inputs x 1 , . . . , x n ∈ X , the GP posterior GP ( µ ⋆ , K ⋆ ) satisfies for any x = x j that

If σ 2 = 0 , then the above also holds for x ⋄ = x j .

Proof. See Proposition 3.8 of Kanagawa et al. [60].

CaGP admits precisely the same guarantee just with the approximate posterior mean and covariance function. The fact that the impact of the approximation on the posterior mean is exactly captured by the approximate predictive variance function is what is meant by the method being computationaware .

Theorem S2 (Worst Case Error Interpretation of CaGP Variance [19])

̸

Given a set of training inputs x 1 , . . . , x n ∈ X , the CaGP posterior GP ( µ i , K i ) satisfies for any x = x j that

If σ 2 = 0 , then the above also holds for x ⋄ = x j .

Proof.

While SVGP also admits a decomposition of its approximate predictive variance into two (relative) worst-case errors, neither of these is the error we care about, namely the difference between the data-generating function y ∈ H K σ and the approximate posterior mean µ SVGP ( x ⋄ ) . It only involves a worst-case error term over the unit ball in the RKHS of the approximate kernel Q σ ≈ K σ .

Theorem S3 (Worst Case Error Interpretation of SVGP Variance [61])

Given a set of training inputs x 1 , . . . , x n ∈ X and (fixed) inducing points Z ∈ R m × d , the optimal variational posterior GP ( µ ⋆ SVGP , K ⋆ SVGP ) of SVGP is given by

where Q ( x , x ′ ) = K ( x , Z ) K ( Z , Z ) -1 K ( Z , x ′ ) is the Nystr¨ om approximation of the covariance function K ( x , x ′ ) (see Eqns. (25) and (26) of Wild et al. [61]). The optimized SVGP posterior

̸

satisfies for any x = x j that

If σ 2 = 0 , then the above also holds for x ⋄ = x j .

Proof. See Theorem 6 of Wild et al. [61].

S1.3 CaGP's Variance Decreases Monotonically as the Number of Iterations Increases

Proposition S1 (CaGP's Variance Decreases Monotonically with the Number of Iterations) Given a training dataset of size n , let GP ( µ i , K i ) be the corresponding CaGP posterior defined in Equation (5) where i ≤ n denotes the downdate rank / number of iterations and assume the CaGP actions S i ∈ R n × i are linearly independent. Then it holds for arbitrary x ⋄ ∈ X and i ≤ j ≤ n , that

where K ⋆ ( x ⋄ , x ⋄ ) is the variance of the exact GP posterior in Equation (1) .

Proof. Wenger et al. [19] originally defined the approximate precision matrix C i = ∑ i ℓ =1 1 η ℓ d ℓ d T ℓ = ∑ i ℓ =1 ˜ d ℓ ˜ d T ℓ as a sum of rank-1 matrices and show that this definition is equivalent to the batch form C i = S i ( S T i ˆ KS i ) -1 S T i we use in this work [see Lemma S1, Eqn. (S37) in 19]. Therefore we have that

where the last equality follows from the fact that S n ∈ R n × n has rank n and therefore

S2 Training Losses

S2.1 Projected-Data Log-Marginal Likelihood

Lemma S2 (Projected-Data Log-Marginal Likelihood) Under the assumptions of Lemma S1, the projected-data log-marginal likelihood is given by

Proof. By the same argument as in Lemma S1 we obtain that

Since S ′ i = S i L -T , where L -T = chol( S T i S i ) -T is the orthonormalizing matrix, L -T cancels in the quadratic loss term, giving

and finally we can decompose the log-determinant into a difference of log-determinants

which using LL T = S T i S i completes the proof.

S2.2 Evidence Lower Bound (ELBO)

Lemma S3 (Evidence Lower Bound Training Loss)

Define the variational family

then the evidence lower bound (ELBO) is given by

where ˜ v i = ( S T ˆ KS ) -1 S T ( y -µ ) are the 'projected' representer weights.

Proof. The ELBO is given by

We first compute the expected log-likelihood term.

Since both q ( f ) and the prior p ( f ) are Gaussian, the KL divergence term between them is given by

Next, we use the matrix determinant lemma det( A + UV T ) = det( I m + V T A -1 U ) det( A ) :

S2.3 Comparison of Training Losses

0

20

Train Loss

0

2

Test NLL

0

2

0

100

200

Epoch

10

-

1

Test std. MSE

0

2

4

Wallclock Time (s)

10

-

1

CaGP-CG + Projected-Data Log-Marginal Likelihood

CaGP-CG + ELBO Training Loss

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

-

5

0

5

CaGP-CG + Projected-Data Log-Marginal Likelihood

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

CaGP-CG + ELBO Training Loss

Test data

GP mean

GP uncertainty

S3 Choice of Actions

We begin by proving that the CaGP posterior in Equation (5) is uniquely defined by the space spanned by the columns of the actions colsp( S ) , rather than the specific choice of the matrix S .

Lemma S4 (The CaGP Posterior Is Uniquely Defined by the Column Space of the Actions) Let S , S ′ ∈ R n × i be two action matrices, each of which consists of non-zero and linearly independent action vectors, such that their column spaces are identical, i.e.

then the corresponding CaGP posteriors GP ( µ i , K i ) and GP ( µ ′ i , K ′ i ) are equivalent.

Proof. By assumption (S20) there exists W ∈ R i × i such that S ′ = SW . Since action vectors are assumed to be linearly independent and non-zero, it holds that i = rank( S ′ ) = rank( SW ) = rank( W ) , where the last equality follows from standard properties of the matrix rank. Therefore W is invertible, and we have that

Since the CaGP posterior in Equation (5) is fully defined via the approximate precision matrix C , the desired result follows.

Corollary S1 (Action Order and Magnitude Does Not Change CaGP Posterior)

The CaGP posterior in Equation (5) is invariant under permutation and rescaling of the actions.

Proof. This follows immediately by choosing a permutation or a diagonal matrix W , respectively, such that S ′ = SW in Lemma S4.

S3.1 (Conjugate) Gradient / Residual Policy

Consider the following linear system

with symmetric positive definite kernel matrix ˆ K = K + σ 2 I , observations y , prior mean evaluated at the data µ = µ ( X ) and representer weights v ⋆ .

Lanczos process [23] The Lanczos process is an iterative method, which computes approximate eigenvalues ˆ Λ = diag( ˆ λ 1 , . . . , ˆ λ i ) ∈ R i × i and approximate eigenvectors ˆ U = (ˆ u 1 · · · ˆ u i ) ∈ R n × i for a symmetric positive definite matrix ˆ K by repeated matrix-vector multiplication. Given an arbitrary starting vector q 1 ∈ R n , s.t. ∥ q 1 ∥ 2 = 1 , it returns i orthonormal vectors Q = ( q 1 · · · q i ) ∈ R n × i and a tridiagonal matrix T = Q T ˆ KQ ∈ R i × i . The eigenvalue approximations are given by an eigendecomposition of T = W ˆ Λ W T , where W ∈ R i × i orthonormal, and the eigenvector approximations are then given by ˆ U = QW ∈ R n × i [e.g. Sec. 10.1.4 of 62].

Conjugate Gradient Method [24] The conjugate gradient method is an iterative method to solve linear systems with symmetric positive definite system matrix by repeated matrix-vector multiplication. When applied to Equation (S21), it produces a sequence of representer weights approximations v i ≈ v ⋆ = ˆ K -1 ( y -µ ) . Its residuals r i = y -µ -ˆ Kv i are proportional to the Lanczos vectors for a Lanczos process initialized at q 1 = r 0 ∥ r 0 ∥ 2 , i.e. Q = RD where R ∈ R n × i is the matrix of residuals and D ∈ R i × i a diagonal matrix (e.g. [Alg. 11.3.2 in 62] or [Sec. 3 & Eqn. (3.4) of 63]).

Therefore choosing actions defined by the residuals of CG in CaGP-CG, i.e. S = R , is equivalent to choosing actions S ′ = ˆ U given by the eigenvector approximations computed by the Lanczos process initialized as above, since

and by Lemma S4 it holds that the corresponding CaGP posteriors with actions S and S ′ are equivalent.

S3.2 Information-theoretic Policy

In information-theoretic formulations of active learning, new data is selected to minimize uncertainty about a set of latent variables z . In other words, we would aim to minimize the entropy of the posterior H p ( z | X ) ( z ) = -∫ log p ( z | X ) p ( z | X ) d z as a function of the data X [21]. In analogy to active learning, in our setting we propose to perform computations y ↦→ S T i y to maximally reduce uncertainty about the latent function f ( X ) evaluated at the training data.

Lemma S5 (Information-theoretic Policy)

The actions S minimizing the entropy of the computation-aware posterior p ( f ( X ) | S T y ) at the training data, or equivalently the actions maximizing the mutual information between f ( X ) and the projected data S T y , are given by

where s 1 , . . . , s i are the topi eigenvectors of ˆ K in descending order of the eigenvalue magnitude.

Proof. Let ˜ y := S T y and f := f ( X ) . By assumption, we have that f ∼ N ( µ , K ) . Recall that the entropy of a Gaussian random vector f ∼ N ( m , S ) is given by H( f ) = 1 2 ( log det( S ) + n log(2 πe ) ) . Now since the covariance function of the computation-aware posterior in Equation (5) does not depend on the targets y , neither does its entropy H p ( f | S T y ) ( f ) .

Therefore, by definition of the conditional entropy and using the law of the unconscious statistician, it holds that

and since the covariance of a Gaussian conditioned on data doesn't depend on the data, we have

Therefore we can rewrite the mutual information in terms of prior and posterior entropy, such that

Via the matrix determinant lemma det( A + UWV T ) = det( W -1 + V T A -1 U ) det( W ) det( A ) , we obtain

for L a square root of S T S . Now we can upper bound the above as follows

where U are the orthonormal eigenvectors of ˆ K for the largest i eigenvalues. Now choosing S = U achieves the upper bound since U T U = I and therefore S i = U is a solution to the optimization problem.

Finally using the argument above and since H( f ) does not depend on S , we have that

S4 Algorithms

S4.2 Implementation

We provide an open-source implementation of CaGP-Opt as part of GPyTorch. To install the package via pip , execute the following in the command line:

S5 Additional Experimental Results and Details

S5.1 Inducing Points Placement and Uncertainty Quantification of SVGP

To better understand whether the overconfidence of SVGP at inducing points observed in the visualization in Figure 1 holds also in higher dimensions, we do the following experiment. For varying input dimension d ∈ { 1 , 2 , . . . , 25 } , we generate synthetic training data by sampling n = 500 inputs X uniformly at random with corresponding targets sampled from a zero-mean Gaussian process y ∼ GP (0 , K σ ) , where K σ ( · , · ) = K ( · , · ) + σ 2 δ ( · , · ) is given by the sum of a Mat´ ern( 3 / 2 ) and a white noise kernel with noise scale σ . We optimize the kernel hyperparameters, variational parameters and inducing points ( m = 64 ) jointly for 300 epochs using Adam with a linear learning rate scheduler. At convergence we measure the average distance between inducing points and the nearest datapoint measured in lengthscale units , i.e.

where l ∈ R d is the vector of lengthscales (one per input dimension). We also compute the average ratio of the posterior variance to the predictive variance at the inducing points , i.e.

The results of our experiments are shown in Figure S3. We find that as expected the inducing points are optimized to lie closer to datapoints than points sampled uniformly at random. However, the inducing points lie increasingly far away from the training data as the dimension increases relative to the lengthscale that SVGP learns. Therefore this experiment suggests that the phenomenon observed in Figure 1, that SVGP can be overconfident at inducing points if they are far away from training datapoints, to be increasingly present as the input dimension increases. This is further substantiated by Figure S3(b) since the proportion of posterior variance to predictive variance at the inducing points is very small already in d = 4 dimensions. This illustrates both SVGP's overconfidence at the inducing points (in particular in higher dimensions) and that its predictive variance is dominated by the learned observation noise, as we also saw in the illustrative Figure 1.

4

8

12

16

20

24

Input Dimension

d

0

.

0

0

.

2

0

.

4

0

.

6

Distance to data

¯

d

l

(

Z

,

X

)

Random Inducing Points

SVGP's Optimized Inducing Points

(a) Average distance of inducing points to the nearest datapoint measured in lengthscale units.

4

CholeskyGP w/ true hyperparameters

SVGP

8

12

16

20

24

Input Dimension

d

1

0

0

0

0

.

.

.

.

.

00

75

50

25

00

)

Z

(

ρ

¯

Variance ratio

S5.2 Grassman Distance Between Subspaces

In Figure 2 we compute the distance between the subspaces spanned by random vectors, the actions S of CaGP, and the space spanned by the topi eigenvectors. The notion of subspace distance we use is the Grassman distance, i.e. for two subspaces spanned by the columns of matrices A ∈ R n × p and B ∈ R n × p s.t. p ≥ q the Grassman subspace distance is defined by

where θ ∈ R q is the vector of principal angles between the two spaces, which can be computed via an SVD [e.g. Alg. 6.4.3 in 62].

S5.3 Generalization Experiment

S5.3.1 Impact of Learning Rate on Generalization

To show the impact of different choices of learning rate on the GP approximations we consider, we show the test metrics for the learning rate sweeps in our main experiment in Figure S4. Note that not all choices of learning rate appear since a small minority of runs fail outright, for example if the learning rate is too large.

0

1min

2min

3min

-

4

-

2

0

2

Parkinsons

Train Loss

0

1min

2min

3min

-

4

-

3

-

2

-

1

0

Test NLL

0

1min

2min

3min

10

-

3

10

-

2

10

-

1

10

0

Test RMSE

0

4min

8min

-

2

0

2

Bike

0

4min

8min

-

3

-

2

-

1

0

0

4min

8min

10

-

2

10

-

1

10

0

0

8min

16min

1

.

0

1

.

5

2

.

0

Protein

0

8min

16min

0

.

80

0

.

85

0

.

90

0

.

95

1

.

00

0

8min

16min

0

.

5

0

.

6

0

.

7

0

20min

40min

0

1

2

KEGGu

0

20min

40min

-

0

.

5

0

.

0

0

.

5

0

20min

40min

0

.

2

0

.

3

0

.

4

0

.

5

0

1h

2h

0

1

2

Road

0

1h

2h

0

.

0

0

.

5

1

.

0

0

1h

2h

0

.

25

0

.

50

0

.

75

1

.

00

0

1.5h

3h

4.5h

Time

-

2

0

2

Power

0

1.5h

3h

4.5h

Time

-

2

-

1

0

1

0

1.5h

3h

4.5h

Time

10

-

1

10

0

model

CholeskyGP

SGPR

SVGP

CGGP

CaGP-CG

CaGP-Opt

learning rate

0.0001

0.001

0.01

0.1

1.0

S5.3.2 Evolution Of Hyperparameters During Training

To better understand how the kernel hyperparameters of each method evolve during training, we show their trajectories in Figure S5 for each dataset. Note that we only show the first three lengthscales per dataset (rather than up to d = 26 ).

0

1.7min

10

-

1

10

0

10

1

Parkinsons

Outputscale

0

1.7min

10

-

3

10

-

1

Observation Noise

0

1.7min

10

-

1

10

0

10

1

Lengthscale: d=1

0

1.7min

10

0

10

1

Lengthscale: d=2

0

1.7min

10

0

10

1

Lengthscale: d=3

0

4min

8min

10

-

1

10

1

Bike

0

4min

8min

10

-

3

10

-

1

0

4min

8min

10

1

10

3

0

4min

8min

10

1

10

3

0

4min

8min

10

1

10

3

0

8min16min

10

-

1

Protein

0

8min16min

10

0

2

×

10

-

1

3

×

10

-

1

4

×

10

-

1

6

×

10

-

1

0

8min16min

10

0

4

×

10

-

1

6

×

10

-

1

2

×

10

0

0

8min16min

10

0

10

1

0

8min16min

10

0

2

×

10

0

3

×

10

0

4

×

10

0

0

20min 40min

10

-

2

10

-

1

10

0

KEGGu

0

20min 40min

10

-

2

10

-

1

0

20min 40min

10

0

10

1

10

2

0

20min 40min

10

0

10

1

10

2

0

20min 40min

10

0

10

1

10

2

0

1.4h

10

-

1

10

0

Road

0

1.4h

10

-

1

10

0

0

1.4h

10

-

1

0

1.4h

10

-

1

0

2.8h

Time

10

-

2

10

-

1

Power

0

2.8h

Time

10

-

3

10

-

2

10

-

1

0

2.8h

Time

10

0

10

1

0

2.8h

Time

10

0

10

1

10

2

0

2.8h

Time

10

0

CholeskyGP

SGPR

SVGP

CGGP

CaGP-CG

CaGP-Opt

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We give both illustrative and theoretical justification for the claims about our method in Section 3 and provide extensive empirical results in Section 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We explicitly discuss the limitations of our method in the conclusion in a dedicated paragraph.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide proofs to all theoretical statements in the supplementary material or cite the appropriate reference for the result.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We describe the experiments including datasets, number of repeats, hyperparameters, method implementations and hardware in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All data we use is publicly available in the UCI repository. We provide an open-source implementation of our method in Section S4.2.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We give a complete description of the choices we made for our benchmark experiments in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We repeat all experiments multiple times and report bootstrapped confidence intervals.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In Section 5, we describe the specific GPUs we use for each experiment and report wallclock time of all training runs.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Due to the methodological nature of this work, there are no potential harmful consequences of this work that we think need to be explicitly highlighted here.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work is foundational methodological research and does not have a specific negative societal impact that we feel must be explicitly highlighted here.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not release data or models with a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the appropriate source for all assets used in our work. All datasets for our experiments are licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) license.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: There are no new assets released with the paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors extend the work of Wenger et al. (2022) on \"computational uncertainty\" in Gaussian process regression models, which introduced the IterGP class of approximations. These approximations treat the limited computation in approximate GP methods as a source of uncertainty, resulting in uncertainty predictions that provably do not suffer from overconfidence. However, the original IterGP paper did not provide a straightforward method for model/hyperparameter selection: this paper addresses that by discussing two approaches for model selection, eventually settling on a new evidence lower bound using the variational family implied by the IterGP approximation, with different proposals for the \"action matrix.\" They compare against SVGP and SGPR, two popular baselines for approximate Gaussian process regression.\n\nSTRENGTHS:\nThe work presents a novel method for model selection and linear-time inference for the IterGP approximations originally proposed by Wenger et al. (2022). They discuss a number of viable options, and overall settle primarily on a newly derived ELBO with a sparse parameterization of an \"action matrix,\" although they do also provide experiments for conjugate gradient-based \"actions.\"\n\nIn my opinion, the quality of the work is largely high, is clearly written, and provides practical steps forward for the GP community. The paper is enjoyable to read and it was great to understand the various avenues that the authors explored to achieve practical model selection. Generally, the authors also seem relatively open about the limitations of the method and don't try to over-claim what they have achieved.\n\nWEAKNESSES:\nMy main concerns with the paper concern its in its evaluation, particularly with regards to the sparse variational methods. I do not recall seeing these problems on this dataset, although I have admittedly found Parkinson's more problematic, but not in the way shown (I usually find that it just fails with a bad Cholesky decomposition instead of showing divergent behavior). \n\nIn my experience, this is because while GPyTorch is a state-of-the-art package for conjugate gradient-based methods, it is problematic when it comes to its implementations of sparse variational methods. I believe these instabilities are rather due to GPyTorch's use of Lanczos and conjugate gradient algorithms when computing covariance matrix decompositions and log probabilities, in addition to the use of float32. In my experience, stochastic variational methods are very sensitive to numerical accuracy and so should only be computed using the full Cholesky decomposition and float64. Therefore, since I did not see mention of this (although apologies if I have missed it!) I would recommend changing the GPyTorch defaults to use Cholesky and float64, or using a package that does not rely on approximate matrix decompositions such as GPflow or GPJax. \n\nMoreover, one of the key attributes of SGPR is that because the loss is not stochastic, it can be optimized (for both hyperparameters and inducing locations) using second-order optimizers such as L-BFGS-B, as is commonly done. This is typically much faster than Adam, and essentially allows optimization to be parameter-free (as the usual defaults generally work well). It would have been good to compare to this, as is common in the literature, although I do commend the parameter search that the authors did for Adam optimization.\n\nFinally, it would be interesting to see how the resultant ELBOs compare, as well as comparisons of ELBOs and hyperparameters to exact GPs on smaller datasets where feasible (I think that about 15-20k datapoints should be feasible given the compute the authors had access to). Indeed, a lot of the claims throughout the paper center around comparing IterGP to SVGP/SGPR with respect to the exact GP, and so it seems a bit strange that there aren't really any experimental comparisons with respect to exact GPR!\n\nQUESTIONS:\nIn order of relative importance:\n1. Following from my main weaknesses, could you confirm whether you used the default GPyTorch settings in terms of faster computations? Apologies again if I missed this!\n2. Could you confirm why SGPR was not used for Road and Power? My understanding is that the memory cost, which would be the main limitation, should be roughly on the same order as for IterGP-Opt, or am I incorrect?\n3. In Fig. 1, could you confirm how you treated the inducing point locations? Although it shows your point nicely, I find it a bit misleading, as in my experience SVGP would never really optimize its inducing locations to be in a (relatively) large gap between datapoints, and so I was surprised to see an inducing location at about 0.6 on the x-axis.\n4. Out of interest, since the ELBO for IterGP-Opt is not stochastic, could L-BFGS-B be used there to speed up optimization?\n5. It's not clear to me how the statement in lines 111-113 would be implied by the equation above it - could you elaborate please?\n\nBelow I'll list some minor questions and typos:\n- Could you please change the colors used in the experiment plots and tables? I am unfortunately red-green colorblind, and I found it very difficult to disentangle the colors between SVGP and SGPR and again between IterGP-Opt and -CG. One example of colorblind-friendly colors in matplotlib can be `plt.style.use('tableau-colorblind10')`. Thanks!\n- Missing space in line 49 in \"introduced,a\"\n- Admittedly really nitpicky, but could you reframe Eq. 3 and so forth as a maximization problem, or be more clear that you're minimizing the negative ELBO? I think it makes more sense to keep the ELBO as a lower bound to the log marginal likelihood, and so maximizing makes more sense\n- line 99: \"we optimizer\" -> \"we optimize\"\n\nLIMITATIONS:\nIn my opinion the authors have fairly assessed the limitations of their work, namely the inability to do minibatching and the method's limitation to non-conjugate likelihoods.\n\n-----\nPost-discussion update\n-----\nI would like to thank the authors for an engaging discussion. Overall, I was pleased with the quality of the responses and the additional experiments provided. While the work is perhaps not the most ''exciting'' in terms of its empirical results and it has some limitations (e.g., not being able to be minibatched, as pointed out by another reviewer), I don't really think it's useful to judge a paper along these lines, and even so I think this paper represents a solid contribution and has the potential to pave the way for important developments further down the line. Therefore, I am raising my score accordingly.\n\nI would also like the opportunity to address the last comments left by the authors. As I was not able to respond to these before the author-reviewer discussion period ended, I have not taken these into account for my final score. However, I hope that the authors will take them into consideration when they update their paper.\n\n> As we write in the caption of Table 1 in the original submission, the reported number of epochs is determined as follows: \"[...] We report the epoch where each method obtained the lowest NLL, and all performance metrics (NLL, RMSE, and wall-clock time) are measured at this epoch. [...]\". The rebuttal PDF did not contain this full caption due to the limited space available, but the final version will, of course. The total number of epochs was determined by resource constraints (1000 epochs for Adam, 100 for LBFGS, except for Power; see also lines 235-238 and our rebuttal response). As Figure R1 (rebuttal pdf) shows this results in roughly similar overall training time between for example SGPR trained with LBFGS and IterGP-Opt trained with Adam. We would recommend interpreting Table (R)1 and Figure (R)1 together to understand the generalization performance.\n\nThanks for your clarification here. Perhaps I have misunderstood which NLL you are looking at, but I am concerned that this amounts to \"cheating\" - as I understand it you are looking at test metrics to determine when to stop the optimization process, which of course you wouldn't have access to in a real setting. It would be better to either fix the number of epochs or determine some form of early stopping that only relies on training or validation data.\n\n> We would like to politely disagree with the characterization that the choice of optimizer and its parameters doesn't determine the final result (even in theory). The log-marginal likelihood (also the ELBO) as a function of the kernel hyperparameters is generally non-convex and often has multiple (local) minima (for an example see Figure 5.5 of Rasmussen and Williams (2008)). Which modes an optimizer converges to depends on the initialization, the choice of optimizer and the learning rate (schedule / line search). \n\nYes, I am in agreement here - I had meant my comment in a highly idealized sense where we could assume that different optimizers would reach the same optimum, but it was sloppy at best and likely unrealistic in many settings.\n\n> The reason we would still recommend using Adam over L-BFGS for IterGP-Opt is the increased resource usage. While the performance is slightly better on both datasets as you correctly point out, the runtime is also larger (e.g. Parkinsons ~=1.3x, Bike 1.8x longer) and the memory usage is higher. For IterGP-Opt one optimizes the kernel hyperparameters and the action entries, which in our case amounts to parameters. As, for example, the PyTorch documentation on L-BFGS points out, L-BFGS requires param_bytes * (history_size+1) amount of memory, which for a typical history size (e.g. PyTorch's default of 100) adds significant overhead, as compared to a single gradient, which only requires param_bytes.\n\nI'm ok with this logic favoring Adam over L-BFGS for memory reasons, as long as it is clearly stated. But I do think it's a bit hard to say that Adam is necessarily faster for IterOpt-GP - it seems fairly close, and in my experience (admittedly, with SGPR) L-BFGS has a tendency to optimize much quicker initially but then spend a lot of time searching for the optimum. It might be instructive to show the training curves as well for both optimizers, particularly as the early stopping I've mentioned above might confound these determinations.\n\nFinally, I did want to comment on something I didn't pick up on from a previous response you gave but caught during the reviewer discussion phase:\n> As per the recommendation in Section 4.1 of Ober et al (2022), in our original experiments we used a (single precision) Cholesky decomposition both for SGPR and SVGP\n\nWhere in Ober et al (2022) does it say they used single precision for SGPR and SVGP? I could not find any mention of single vs. double precision, and I believe it's fairly common for SGPR and SVGP to still be done in double precision, despite the default single precision in GPyTorch.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors cast the IterGP method of Wenger et al, which is guaranteed to give conservative estimates of uncertainty relative to the posterior, as a variational procedure. This allows for hyperparameter selection and selection of parameters controlling the quality of the approximation ($\\mathbf{S}$) through maximization of an evidence lower bound. The authors compare this method to existing variational approximations on several benchmark datasets.\n\nSTRENGTHS:\n- The exposition is clear and well motivated.\n- Placing IterGP within a variational context both allows for a conceptual connection to SGPR/SVGP and (more importantly) practical progress in terms of selection of parameters.\n\nWEAKNESSES:\n- Datasets have not been properly cited. In particular, a general citation to UCI doesn’t give credit to the creators of each dataset. License information was also not stated.\n- I found the discussion of information gain not well-motivated. The definition of information gain was assumed to be known, and it wasn’t clear from the text why maximizing this quantity would lead to a good quality approximation. If the only goal is to suggest projection onto the largest $i$-eigenvalues, there are many results in the matrix approximation literature that also motivate this approach. Moreover, there is prior work in the Nystrom approximation literature considering the top $i$-eigenvalues as a subspace for performing inference, for example Ferrari-Trecate et al, https://proceedings.neurips.cc/paper/1998/hash/55c567fd4395ecef6d936cf77b8d5b2b-Abstract.html.\n- The phrasing in the caption of figure 4 conflates credible and confidence intervals. The accuracy of the credible interval should be with reference to the (exact) posterior and cannot be shown by frequentist coverage. To be clear, I think checking the empirical coverage of the methods is reasonable, but don’t think you can draw conclusions about the accuracy of the credible interval from this.\n\nQUESTIONS:\n- The authors claim SVGP has “no latent uncertainty” where data is absent. Do the authors mean that the posterior variance is actually 0 or extremely close to it away from the data? I don’t see how this can be true since SVGP recovers the posterior distribution in certain limits. If what is meant is that uncertainty is underestimated away from the data, this is quite a bit weaker as a statement and should be made clear.\n- “Unlike these other methods, the IterGP posterior updates are defined by a linear combination of kernel functions on training data, which guarantees that posterior variance estimates always over estimate the exact GP uncertainty” — Could you clarify this sentence? First, what is meant by “posterior updates”? Posterior mean updates? Posterior variance updates? Posterior sample path updates? Also, I don’t think this is the property that guarantees that posterior variances are overestimated. It is easy to write down a form of posterior variance that is both a linear combination of kernel function and underestimates the posterior variance, for example 0. \n- How does the expression for the approximate predictive standard deviation (section 2) in terms of a worst case quantity over an RKHS ball compare to the analogous statement for low-rank variational methods (given in Theorem 6 of Wild et al 2023, https://arxiv.org/pdf/2106.01121)? I think a brief comparison (possibly in the appendix if space does not allow) would be useful.\n- When stating what is meant by computation aware, should there be a convergence condition? I.e. not only should the precision increase, but it should converge to the posterior precision as the amount of compute increases under some sensible restrictions.\n- I don’t understand what is plotted on the right of figure 2, or what my takeaway from it should be. What is meant by plotting the magnitude of the eigenvalues of the matrix for each $x_j$? On the plot on the right, what notion of distance between subspaces is used?\n- Should I interpret equation 12 as stating the $\\mathbf{S}$ is block diagonal? If so, is this said explicitly somewhere?\n- Could you explain in more detail why SGPR diverges in these experiments? Have you checked if this is resolved by double precision?\n\n\n### Other\n\n- Line 49, missing space after comma before “a”\n- Line 61, the authors claim that IterGP has “calibrated” confidence. Is the claim that the confidence intervals are calibrated, or that they are conservative in the sense of overestimating the posterior credible intervals?\n- Line 73, h is introduced here as the latent function and never appears again. I’m a bit unclear why it is needed.\n- Line 82, before discussing computation in model selection, it should be stated that you assume maximum (marginal) likelihood will be used. There are alternative approaches (e.g. sampling)\n\nLIMITATIONS:\nThe authors point out limitations well; most importantly, the restriction to Gaussian likelihoods in the present work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a substantial extension to an existing class of models called IterGP. By introducing a novel training loss which combines both the hyperparameters and a sparse action matrix, the IterGP-Opt model offers linear-time scalability. Experiments over a range of UCI datasets demonstrate that IterGP-Opt matches or outperforms the benchmark SGPR and SVGP models.\n\nSTRENGTHS:\nThe paper is well written and clearly structured.\n\nThe proposed IterGP-Opt model displays a high degree of novelty.\n\nThe results and methodology are for the most part well presented.\n\nScalable inference is an area of high impact, so potential improvements over the SVGP model are of high interest.\n\nWEAKNESSES:\nI have a couple of concerns regarding the models used in the experiments:\n\nThe paper cites Ober et al \"Recommendations for Baselines and Benchmarking Approximate Gaussian Processes\" as inspiration for including SGPR as a strong benchmark, but does not follow the guidance for SGPR training outlined in that paper. Both in terms of the choice of optimiser, and the approach of not jointly optimising the inducing points with the kernel hyperparameters. This is concerning as it could be having a significantly detrimental impact on the performance of the SGPR baseline - as is further evidenced by it underperforming SVGP on three of the four tasks. In any case, given the popularity of the datasets used, it ought to be straightforward to verify the SGPR performance is consistent with other publications in the literature.\n\nOne of the other methods presented in Table 1 is \"IterGP-CG\", but its status should be clarified for the reader. The first reference to IterGP-CG in the paper is in the caption to Figure 1, where it is introduced as \"IterGP-CG (ours), ...\", implying this model is a novelty of the paper. Yet later in the Background section, and also in the footnote on page 7, it is pointed out that the IterGP-CG model is not new. Given that IterGP-CG is not mentioned at all in the Conclusion section, it seems likely the \"ours\" was just a typo, but this is of course a very important point to clarify! Table 1 itself should also be updated to clarify which method(s) are claimed to be novel to the paper.\n\nQUESTIONS:\nThe final sentence of the abstract claims \"As a result of this work, Gaussian processes can be trained on large-scale datasets without compromising their ability to quantify uncertainty\" - Doesn't this imply we are conducting exact inference on these large-scale datasets? I'd recommend the authors consider a suitable rephrasing such as \"significantly compromising\".\n\n\"SGPR typically requires more gradient iterations during training as it introduces new parameters (the inducing point locations Z).\"\nI was surprised by this comment, because in my experience SGPR typically requires significantly fewer training iterations than SVGP. In part because training is unbatched, allowing it to exploit standard LBFGS optimisation as opposed to using Adam. And also because it is often not desirable to train the inducing point locations jointly with the hyperparameters.\n\nLIMITATIONS:\nThe authors include a well written section on the limitations of IterGP-Opt.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}