Title: Improving Linear System Solvers for Hyperparameter Optimisation in Iterative Gaussian Processes
Venue: NeurIPS 2024
Accepted: yes
Keywords: Gaussian process

marginal likelihood

iterative linear system solver

pathwise

warm start

conjugate gradient

alternating projection

stochastic gradient descent
Abstract: Scaling hyperparameter optimisation to very large datasets remains an open problem in the Gaussian process community. This paper focuses on iterative methods, which use linear system solvers, like conjugate gradients, alternating projections or stochastic gradient descent, to construct an estimate of the marginal likelihood gradient. We discuss three key improvements which are applicable across solvers: (i) a pathwise gradient estimator, which reduces the required number of solver iterations and amortises the computational cost of making predictions, (ii) warm starting linear system solvers with the solution from the previous step, which leads to faster solver convergence at the cost of negligible bias, (iii) early stopping linear system solvers after a limited computational budget, which synergises with warm starting, allowing solver progress to accumulate over multiple marginal likelihood steps. These techniques provide speed-ups of up to $72\times$ when solving to tolerance, and decrease the average residual norm by up to $7\times$ when stopping early.

Improving Linear System Solvers for Hyperparameter Optimisation in Iterative Gaussian Processes

Abstract

Scaling hyperparameter optimisation to very large datasets remains an open problem in the Gaussian process community. This paper focuses on iterative methods, which use linear system solvers, like conjugate gradients, alternating projections or stochastic gradient descent, to construct an estimate of the marginal likelihood gradient. We discuss three key improvements which are applicable across solvers: (i) a pathwise gradient estimator, which reduces the required number of solver iterations and amortises the computational cost of making predictions, (ii) warm starting linear system solvers with the solution from the previous step, which leads to faster solver convergence at the cost of negligible bias, (iii) early stopping linear system solvers after a limited computational budget, which synergises with warm starting, allowing solver progress to accumulate over multiple marginal likelihood steps. These techniques provide speed-ups of up to 72 × when solving to tolerance, and decrease the average residual norm by up to 7 × when stopping early.

1 Introduction

Gaussian processes [22] (GPs) are a versatile class of probabilistic machine learning models which are used widely for Bayesian optimisation of black-box functions [24], climate and earth sciences [10, 26], and data-efficient learning in robotics and control [6]. However, their effectiveness depends on good estimates of hyperparameters, such as kernel length scales and observation noise. These quantities are typically learned by maximising the marginal likelihood, which balances model complexity with training data fit. In general, the marginal likelihood is a non-convex function of the hyperparameters and evaluating its gradient requires inverting the kernel matrix. Using direct methods, this requires compute and memory resources which are respectively cubic and quadratic in the number of training examples. This is intractable when dealing with large datasets of modern interest.

Methods to improve the scalability of Gaussian processes can roughly be grouped into two categories. Sparse methods [20, 27, 12] approximate the kernel matrix with a low-rank surrogate, which is cheaper to invert. This reduced flexibility may result in failure to properly fit increasingly large or sufficiently complex data [15]. On the other hand, iterative methods [11] express GP computations in terms of systems of linear equations. The solution to these linear systems is approximated up to a specified numerical precision with linear system solvers, such as conjugate gradients (CG) [11, 9, 30], alternating projections (AP) [23, 28, 33], or stochastic gradient descent (SGD) [15, 16]. These methods allow for a trade-off between compute time and accuracy. However, convergence can be slow in the large data regime, where system conditioning is often poor.

In this paper, we focus on iterative GPs and identify techniques, which were important to the success of previously proposed methods, but did not receive special attention in the literature. Many of these amount to amortisations which leverage previous computations to accelerate subsequent ones. We analyse and adapt these techniques, and show that they can be applied to accelerate different linear solvers, obtaining speed-ups of up to 72 × without sacrificing predictive performance (see Figure 1).

pol

elev

bike

0%

20%

40%

60%

80%

100%

pol

elev

bike

Dataset Name

0%

20%

40%

60%

80%

100%

Alternating Projections

pol

elev

bike

Dataset Name

0%

20%

40%

60%

80%

100%

Stochastic Gradient Descent

Hatched areas represent the fraction of the total runtime which is used by the linear system solver

Standard

Pathwise

Standard + warm start

Pathwise + warm start

In the following, we summarise our contributions:

We introduce a pathwise estimator of the marginal likelihood gradient and show that, under real-world conditions, the solutions to the linear systems required by this estimator are closer to the origin than those of the standard estimator, allowing our solvers to converge faster. Additionally, these solutions transform into samples from the GP posterior without further matrix inversions, amortising the computational costs of predictive posterior inference.

Wepropose to warm start linear system solvers throughout marginal likelihood optimisation by reusing linear system solutions to initialise the solver in the subsequent step. This results in faster convergence. Although this technically introduces bias into the optimisation, we show that, theoretically and empirically, the optimisation quality does not suffer.

We investigate the behaviour of linear system solvers on a limited compute budget , such that reaching the specified tolerance is not guaranteed. Here, warm starting allows the linear system solver to accumulate solver progress across marginal likelihood steps, progressively improving the solution quality of the linear system solver despite early stopping.

We demonstrate empirically that the methods above either reduce the required number of iterations until convergence without sacrificing performance or improve the performance if a limited compute budget hinders convergence. Across different UCI regression datasets and linear system solvers, we observe average speed-ups of up to 72 × when solving until the tolerance is reached, and increased performance when the compute budget is limited.

Source code available at: https://github.com/jandylin/iterative-gaussian-processes

2 Gaussian Process Regression and Marginal Likelihood Optimisation

Formally, a GP is a stochastic process f : X → R , such that, for any finite subset { x i } n i =1 ⊂ X , the set of random variables { f ( x i ) } n i =1 is jointly Gaussian. In particular, f is uniquely identified by a mean function µ ( · ) = E [ f ( · )] and a positive-definite kernel function k ( · , · ′ ; ϑ ) = Cov( f ( · ) , f ( · ′ )) with kernel hyperparameters ϑ . We use a Matérn3 / 2 kernel with length scales per dimension and a scalar signal scale and write f ∼ GP( µ, k ) to express that f is a GP with mean µ and kernel k .

For GP regression, let the training data consist of n inputs x ⊂ X and targets y ∈ R n . We consider the Bayesian model y i = f ( x i ) + ϵ i , where each ϵ i ∼ N (0 , σ 2 ) i.i.d. and f ∼ GP( µ, k ) . We assume µ = 0 without loss of generality. The posterior of this model is f | y ∼ GP( µ f | y , k f | y ) , with

where k ( · , x ; ϑ ) , k ( x , · ; ϑ ) and k ( x , x ; ϑ ) refer to pairwise evaluations, resulting in a 1 × n row vector, a n × 1 column vector and a n × n matrix respectively.

Pathwise Conditioning Wilson et al. [31, 32] express a GP posterior sample as a random function

where ϵ ∼ N ( 0 , σ 2 I ) is a random vector, f ∼ GP(0 , k ) is a zero-mean prior function sample, and f ( x ) is its evaluation at the training data. Following previous work [31, 32, 15, 16], we efficiently approximate the prior function sample using random features [21, 25] (see Appendix B for details). Using pathwise conditioning, a single linear solve suffices to evaluate a posterior function sample at arbitrary locations without further linear solves. In Section 3, we amortise the cost of this single linear solve during marginal likelihood optimisation to obtain posterior samples efficiently.

The Marginal Likelihood and Its Gradient With hyperparameters θ = { ϑ , σ } and regularised kernel matrix H θ = k ( x , x ; ϑ ) + σ 2 I ∈ R n × n , the marginal likelihood L as a function of θ and its gradient ∇ θ k L with respect to θ k can be expressed as

where the partial derivative of H θ with respect to θ k is a n × n matrix. We assume n is too large to compute the inverse or log-determinant of H θ and iterative methods are used instead.

2.1 Hierarchical View of Marginal Likelihood Optimisation for Iterative Gaussian Processes

Marginal likelihood optimisation for iterative GPs consists of bilevel optimisation, where the outer loop maximises the marginal likelihood (4) using stochastic estimates of its gradient (5). Computing these gradient estimates requires the solution to systems of linear equations. These solutions are obtained using an iterative solver in the inner loop. Figure 2 illustrates this three-level hierarchy.

Outer-Loop Optimiser The outer-loop optimiser maximises the marginal likelihood L (4) using its gradient (5). Common choices are L-BFGS [3], when exact gradients are available, and Adam [14] in the large-data setting, when stochastic approximation is required. We consider the case where gradients are stochastic and use Adam.

Outer-Loop Optimiser

e.g. L-BFGS, Adam

Gradient Estimator

e.g. Hutchinson trace estimator

Linear System Solver

e.g. conjugate gradients

Gradient Estimator The gradient (5) involves two computationally expensive components: linear solves against the targets H -1 θ y and the trace term tr ( H -1 θ ∂ H θ /∂θ k ) . An unbiased estimate of the latter can be obtained using s probe vectors and Hutchinson's trace estimator [13],

where the probe vectors z j ∈ R n satisfy ∀ j : E [ z j z T j ] = I , and z T j H -1 θ is obtained using a linear solve. We refer to this as the standard estimator and set s = 64 , unless otherwise specified.

Linear System Solver Substituting the trace estimator (6) back into the gradient (5), we obtain an unbiased gradient estimate in terms of the solution to a batch of systems of linear equations,

which share the same coefficient matrix H θ . Since H θ is positive-definite, the solution v = H -1 θ b to the system H θ v = b can be obtained by finding the unique minimiser of the quadratic objective

facilitating the use of iterative solvers. Most popular in the GP literature are conjugate gradients (CG) [9, 30-32], alternating projections (AP) [23, 28, 33] and stochastic gradient descent (SGD) [15, 16]. We consider these in our study and provide detailed descriptions of them in Appendix B. Solvers are often run until the relative residual norm ∥ b -H θ u ∥ / ∥ b ∥ reaches a certain tolerance τ [30, 18, 33]. We set τ = 0 . 01 , following Maddox et al. [18]. The linear system solver in the inner loop dominates the computational costs of marginal likelihood optimisation for iterative GPs, as shown in Figure 1. Therefore, improving linear system solvers is the main focus of our work.

0

2M

4M

6M

pol

Initial Distance

0

80k

160k

240k

Solver Iterations

0

500

1000

1500

Top Eigenvalue of

H

-

1

θ

0

500

1000

1500

σ

-

2

0

25

50

75

100

55k

110k

elev

0

25

50

75

100

0

8k

16k

24k

0

25

50

75

100

0

6

12

18

0

25

50

75

100

0

6

12

18

Marginal Likelihood Steps

Standard

Pathwise

Standard (theoretical)

Pathwise (theoretical)

3 Pathwise Estimation of Marginal Likelihood Gradients

We introduce the pathwise estimator , an alternative to the standard estimator (6) which reduces the required number of linear system solver iterations until convergence (see Figure 3). Additionally, the estimator simultaneously provides us with posterior function samples via pathwise conditioning, hence the name pathwise estimator. This facilitates predictions without further linear solves.

We modify the standard estimator (6) to absorb H -1 θ into the distribution of the probe vectors [2],

where ∀ j : E [ ˆ z j ˆ z T j ] = H -1 θ . Probe vectors ˆ z with the desired second moment can be obtained as

where ξ = f ( x ) + ϵ . Akin to the standard estimator in Section 2.1, we obtain v y and ˆ z j by solving

Initial Distance to the Linear System Solution Under realistic conditions, the pathwise estimator moves the solution of the linear system closer to the origin. To show this, we consider the generic linear system H θ u = b and measure the RKHS distance between the initialisation u init and the solution u = H -1 θ b as ∥ u init -u ∥ 2 H θ . With u init = 0 , which is standard [9, 30, 1, 15, 33, 16],

Since b is a random vector ( z in (7) and ξ in (11)), we analyse the expected squared distance

For the standard estimator (6), we substitute b := z with E [ zz T ] = I , yielding

For the pathwise estimator (9), we substitute b := ξ with E [ ξξ T ] = H θ , yielding

The initial distance for the standard estimator is equal to the trace of H -1 θ , whereas it is constant for the pathwise estimator. Figure 3 illustrates that this trace follows the top eigenvalue, which roughly matches the noise precision. As the model fits the data better, the noise precision increases, increasing the initial distance for the standard but not for the pathwise estimator. In practice, the latter leads to faster solver convergence, especially for problems with high noise precision (see Table 1).

1

4

16

64

1024

Number of Samples

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

Test Log-Likelihood

0

25

50

75

100

0

.

0

1

.

2

2

.

4

3

.

6

4

.

8

Time (min): CG

0

25

50

75

100

Marginal Likelihood Steps

0

12

24

36

48

Time (min): AP

0

25

50

75

100

0

20

40

60

80

Time (min): SGD

Number of probe vectors

for pathwise estimator

Absolute difference between pathwise

conditioning and exact predictions

64 (default)

16

Amortising Linear Solves for Optimisation and Prediction The name of the pathwise estimator comes from the fact that solving the linear systems (11) provides us with all of the terms we need to construct a set of s posterior samples via pathwise conditioning (3). Each of these is given by

We can use these to make predictions without requiring any additional linear system solves.

How Many Probe Vectors and Posterior Samples Do We Need? In the literature [9, 18, 2, 33], it is common to use s ≤ 16 probe vectors for marginal likelihood optimisation. However, a larger number of posterior samples, around s = 64 , is necessary to make accurate predictions [2, 15, 16] (see Figure 4). Thus, to amortise linear system solves across marginal likelihood optimisation and prediction, we must use the same number of probes for both. Interestingly, as shown in Figure 4, using 64 instead of 16 probe vectors only increases the runtime by around 10% because the computational costs are dominated by kernel function evaluations, which are shared among probe vectors.

Estimator Variance The standard estimator with Gaussian probe vectors and the pathwise estimator have the same variance if H θ and ∂ H θ /∂θ k commute with each other (see Appendix A.1). There has been work developing trace estimators with lower variance [19, 8], however, we did not pursue these as we find variance to be sufficiently low, even when relying on only s = 16 probe vectors.

Approximate Prior Function Samples Using Random Features In practice, the pathwise estimator requires samples from the prior f ∼ GP(0 , k ) , which are intractable for large datasets without the use of random features [31, 32]. In Figure 5, we show that, despite using random features, most of the time the marginal likelihood optimisation trajectory of the pathwise estimator matches the trajectory of exact optimisation using Cholesky factorisation and backpropagation. Further, we confirm that deviations of the pathwise estimator are indeed due to the use of random features by demonstrating that we can remove these deviations using exact samples from the prior instead.

-

0

.

5

0

0

.

5

Difference to Exact

0%

10%

20%

30%

40%

Pathwise Trajectory

0

25

50

75

100

1

.

0

1

.

4

1

.

8

2

.

2

2

.

6

0

25

50

75

100

Marginal Likelihood Steps

1

.

0

1

.

4

1

.

8

2

.

2

2

.

6

0

25

50

75

100

1

.

0

1

.

4

1

.

8

2

.

2

2

.

6

Pathwise estimator with random features

Pathwise estimator with exact prior

Exact

-

0

.

2

0

.

2

0

.

6

1

.

0

First Eigendirection

-

0

.

2

0

.

2

0

.

6

1

.

0

Second Eigendirection

Mean

First Eigendirection

Probe Vectors

-

4

0

4

-

4

0

4

-

4

0

4

×

10

-

2

-

4

0

4

Marginal Likelihood Steps

0

.

0

0

.

6

1

.

2

Mean

Initial Distance to Solution

0

25

50

75

100

0

10

20

Probe

Vectors

Initialise at zero (default)

Initialise at previous solution (warm start)

Solution

0

75

150

225

pol

CG

0

25

50

75

100

0

16

32

48

elev

0

75k

150k

225k

AP

0

25

50

75

100

Marginal Likelihood Steps

0

8k

16k

24k

0

70k

140k

SGD

0

25

50

75

100

0

2k

4k

6k

1

.

25

1

.

26

1

.

27

1

.

28

Test LLH

CG

AP SGD

-

0

.

40

-

0

.

39

-

0

.

38

-

0

.

37

Solver Iterations

Standard estimator

Standard estimator + warm start

4 Warm Starting Linear System Solvers

Linear system solvers are typically initialised at zero [9, 30, 1, 15, 33, 16]. 1 However, because the outer-loop marginal likelihood optimisation does not change the hyperparameters much between consecutive steps, we expect that the solution to inner-loop linear systems also does not change much between consecutive steps (see Appendix A.2 for a more formal argument). Therefore, we suggest to warm start linear system solvers by initialising them at the solution of the previous [17]. This requires that the targets of the linear systems, z j or ξ j , are not resampled throughout optimisation, which can introduce bias [5]. However, we find that warm starting consistently provides gains across all linear system solvers for both the standard and the pathwise estimator, and that the bias is negligible.

Visualising Warm Starts Figure 6 visualises the two top eigendirections of the inner-loop quadratic objective on POL. Throughout training, warm starting solvers at the solution to the previous linear system results in a substantially smaller initial distance to the current solution.

Effects on Linear System Solver Convergence Reducing the initial RKHS distance to the solution reduces the required number of solver iterations until the tolerance τ = 0 . 01 is reached for all solvers and all five datasets, as shown in Figure 7, Table 1 and Appendix C. However, the effectiveness depends on the solver type. CG is more sensitive to the direction of descent rather than the distance to the solution because it uses line searches to take big steps. It only obtains a 2 . 1 × speed-up on average. AP and SGD benefit more, with average speed-ups of 18 . 9 × and 5 . 1 × , respectively.

-

0

.

2

0

0

.

2

Difference to Exact

0%

15%

30%

45%

60%

Warm Start Trajectory

0

25

50

75

100

1

.

0

1

.

4

1

.

8

2

.

2

2

.

6

0

25

50

75

100

Marginal Likelihood Steps

1

.

0

1

.

4

1

.

8

2

.

2

2

.

6

0

25

50

75

100

1

.

0

1

.

4

1

.

8

2

.

2

2

.

6

Standard estimator + warm start

Exact

Does Warm Starting Introduce Bias? A potential concern when warm starting is that the latter introduces bias into the optimisation trajectory because the linear system targets are not resampled throughout optimisation. Although individual gradient estimates are unbiased, estimates are correlated along the optimisation trajectory. In fact, after fixing the targets, gradients become deterministic and it is unclear whether the induced optimum converges to the true optimum. 2 Fortunately, one can show that the marginal likelihood at the optimum implied by these gradients will converge in probability to the marginal likelihood of the true optimum.

Theorem 1. (informal) Under reasonable assumptions, the marginal likelihood L of the hyperparameters obtained by maximising the objective implied by the warm-started gradients ˜ θ ∗ will converge in probability to the marginal likelihood of a true maximum θ ∗ : L ( ˜ θ ∗ ) p →L ( θ ∗ ) as s →∞ .

See Appendices A.3 and A.4 for details. In practice, a small number of samples seems to be sufficient. In Appendix C, we illustrate that optimisation trajectories of warm-started solvers are almost identical to trajectories obtained by non-warm-started solvers across solver types and datasets.

WarmStarting the Pathwise Estimator One advantage of the pathwise estimator from Section 3 is the reduced RKHS distance between the origin and the solution. When warm starting, the inner-loop solver no longer initialises at the origin, and thus, one may be concerned that we lose this advantage. However, empirically, this is not the case. As shown in Table 1, combining both techniques further accelerates AP and SGD, reaching 72 . 1 × and 7 . 2 × average speed-ups across our datasets relative to the standard estimator without warm starting. Furthermore, since we run solvers until reaching the tolerance, the predictive performance is almost identical among all methods and solvers.

0

2

4

6

CG

0

.

0

0

.

2

0

.

4

0

.

6

AP

0

.

0

0

.

4

0

.

8

1

.

2

SGD

0

25

50

75

100

0

.

0

0

.

4

0

.

8

1

.

2

0

25

50

75

100

Marginal Likelihood Steps

0

.

00

0

.

04

0

.

08

0

.

12

0

25

50

75

100

0

.

0

0

.

2

0

.

4

0

.

6

-

2

.

3

-

0

.

5

1

.

3

3

.

1

Test LLH

CG

AP

SGD

0

.

9

1

.

1

1

.

3

1

.

5

Maximum solver epochs per marginal likelihood step:

Residual Norm

Standard

Pathwise

Standard + warm start

Pathwise + warm start

10

20

30

40

50

5 Solving Linear Systems on a Limited Compute Budget

Our experiments so far have only considered relatively small datasets with n< 50 k, such that innerloop solvers can reach the tolerance in a reasonable amount of time. However, on large datasets, where linear system conditioning may be poor, reaching a low relative residual norm can become computationally infeasible. Instead, linear system solvers are commonly given a limited compute budget. Gardner et al. [9] limit the number of CG iterations to 20, Wu et al. [33] use 11 epochs of AP, Antorán et al. [2] run SGD for 50k iterations, and Lin et al. [15, 16] run SGD for 100k iterations. While effective for managing computational costs, it is not well understood how early stopping before reaching the tolerance affects different solvers and marginal likelihood optimisation. Furthermore, it is unclear whether a certain tolerance is required to obtain good downstream predictive performance.

The Effects of Early Stopping We repeat the experiments from Table 1 but introduce limited compute budgets: 10, 20, 30, 40 or 50 solver epochs, where one epoch refers to computing each value in H θ once (see Appendix B for details). 3 In this setting, solvers terminate upon either reaching the relative residual norm tolerance or when the compute budget is exhausted, whichever occurs first.

In Figure 9, we illustrate the relative residual norms reached for each compute budget on the POL dataset (see Figures 14 to 17 in Appendix C for other datasets). In general, the residual norms increase as H θ becomes more ill-conditioned during optimisation, and as the compute budget is decreased. The increase in residual norms is much larger for CG than the other solvers, which is consistent with previous reports of CG not being amenable to early stopping [15]. AP seems to behave slightly better than SGD under a limited compute budget. Both the pathwise estimator and warm starting combine well with early stopping, reaching lower residual norms when using a budget of 10 solver epochs than the standard estimator without warm starting using a budget of 50 solver epochs.

In terms of predictive performance, we see that CG with the standard estimator and no warm starting suffers the most from early stopping. Changing to the pathwise estimator and warm starting recovers good performance most of the time. SGD also shows some sensitivity to early stopping, but there seems to be a stronger correlation between invested compute and final performance. Surprisingly, AP generally achieves good predictive performance even on the smallest compute budget, despite not reaching the tolerance of τ = 0 . 01 . Overall, the relationship between reaching a low residual norm and obtaining good predictive performance seems to be weak. This is an unexpected yet interesting observation, and future research should investigate the suitability of the relative residual norm as a metric to determine solver convergence.

Conjugate Gradients

0

5 10 15

1

.

0

1

.

4

1

.

8

house

Res. Norm

0

5 10 15

-

8

-

4

0

Test LLH

0 10 20 30

1

.

0

2

.

7

4

.

4

buzz

0 10 20 30

-

14

-

7

0

0 10 20 30

0

.

10

1

.

25

2

.

40

song

0 10 20 30

-

2

.

2

-

1

.

6

-

1

.

0

Alternating Projections

0

5 10 15

0

.

03

0

.

06

0

.

09

Res. Norm

0

5 10 15

0

.

0

1

.

3

2

.

6

Test LLH

0 10 20 30

0

.

01

0

.

06

0

.

11

0 10 20 30

-

0

.

13

-

0

.

08

-

0

.

03

0 10 20 30

0

.

00

0

.

04

0

.

08

0 10 20 30

-

1

.

13

-

1

.

12

-

1

.

11

Stochastic Gradient Descent

0

5 10 15

0

.

10

0

.

15

0

.

20

Res. Norm

0

5 10 15

0

.

2

1

.

0

1

.

8

Test LLH

0 10 20 30

0

.

0

0

.

3

0

.

6

0 10 20 30

-

0

.

2

-

0

.

1

0

.

0

0 10 20 30

0

.

00

0

.

05

0

.

10

0 10 20 30

-

1

.

13

-

1

.

12

-

1

.

11

Marginal Likelihood Steps

Pathwise estimator

Pathwise estimator + warm start

Demonstration on Large Datasets After analysing early stopping on small datasets, we now turn to larger UCI datasets 391 k <n< 1 . 8 M, where solving until reaching the tolerance becomes computationally infeasable. Thus, we introduce a compute budget of 10 solver epochs per marginal likelihood step. Hyperparameters are initialised with the heuristic of Lin et al. [15] and optimised using a learning rate of 0.03 for 30 Adam steps (15 for HOUSEELECTRIC due to high computational costs). We use the pathwise estimator because it accelerates solver convergence (see Section 3), and it enables efficient tracking of predictive performance during optimisation. See Appendix B for details.

Figure 10 visualises the evolution of the relative residual norm of the probe vector linear systems and the predictive test log-likelihood during marginal likelihood optimisation. A full set of results is in Appendix C. For all solvers, warm starting leads to lower residual norms throughout outer-loop steps. This suggests a synergistic behaviour between early stopping and warm starting: the latter allows solver progress to accumulate across marginal likelihood steps. This can be interpreted as amortising the inner-loop linear system solve over multiple outer-loop steps. Despite the lower residual norm, CG is brittle under early stopping, obtaining significantly worse performance than AP and SGD on BUZZ and HOUSEELECTRIC. AP and SGD seem to be more robust to early stopping. However, lower residual norms do not always translate to improved predictive performance. Furthermore, we find that SGD can suffer due to the optimal learning rate changing as the hyperparameters change.

6 Conclusion

Building upon a hierarchical view of marginal likelihood optimisation, this paper consolidates several iterative GP techniques into a common framework, analysing them and showing their applicability across different linear system solvers. Overall, these provide speed-ups of up to 72 × when solving until a specified tolerance is reached, and decrease the average relative residual norm by up to 7 × under a limited compute budget. Additionally, our analyses lead to the following findings: Firstly, the pathwise gradient estimater accelerates linear system solvers by moving solutions closer to the origin, and also provides amortised predictions as an added benefit by turning probe vectors into posterior samples via pathwise conditioning. Secondly, warm starting solvers at previous solutions during marginal likelihood optimisation reduces the number of solver iterations to tolerance at the cost of introducing negligible bias into the optimisation trajectory. Furthermore, warm starting combines well with pathwise gradient estimation. Finally, stopping linear system solvers after exhausting a limited compute budget generally increases the relative residual norm. However, when paired with warm starting, solver progress accumulates, amortising inner-loop linear system solves over multiple outer-loop steps. Nonetheless, we observe that low relative residual norms are not always necessary to obtain good predictive performance, which presents an interesting avenue for future research.

Acknowledgments

Jihao Andreas Lin and Shreyas Padhy were supported by the University of Cambridge Harding Distinguished Postgraduate Scholars Programme. José Miguel Hernández-Lobato acknowledges support from a Turing AI Fellowship under grant EP/V023756/1. We thank Runa Eschenhagen for helpful discussions. This work was performed using resources provided by the Cambridge Service for Data Driven Discovery (CSD3) operated by the University of Cambridge Research Computing Service (www.csd3.cam.ac.uk), provided by Dell EMC and Intel using Tier-2 funding from the Engineering and Physical Sciences Research Council (capital grant EP/T022159/1), and DiRAC funding from the Science and Technology Facilities Council (www.dirac.ac.uk). This work was also supported with Cloud TPUs from Google's TPU Research Cloud (TRC).

References

J. Antorán, R. Barbano, J. Leuschner, J. M. Hernández-Lobato, and B. Jin. Uncertainty Estimation for Computed Tomography with a Linearised Deep Image Prior. arXiv:2203.00479 , 2022. Cited on pages 4, 6.

J. Antorán, S. Padhy, R. Barbano, E. T. Nalisnick, D. Janz, and J. M. Hernández-Lobato. Sampling-based inference for large linear models, with application to linearised Laplace. In International Conference on Learning Representations , 2023. Cited on pages 4-6, 8.

A. Artemev, D. R. Burt, and M. van der Wilk. Tighter Bounds on the Log Marginal Likelihood of Gaussian Process Regression using Conjugate Gradients. In International Conference on Machine Learning , 2021. Cited on pages 3, 6.

J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018. Cited on page 17.

H. Chen, L. Zheng, R. Al Kontar, and G. Raskutti. Stochastic Gradient Descent in Correlated Settings: A Study on Gaussian Processes. In Advances in Neural Information Processing Systems , 2020. Cited on page 6.

M. P. Deisenroth, D. Fox, and C. E. Rasmussen. Gaussian Processes for Data-Efficient Learning in Robotics and Control. IEEE Trans. Pattern Anal. Mach. Intell. , 2015. Cited on page 1.

D. Dua and C. Graff. UCI Machine Learning Repository, 2017. Cited on pages 17, 33-36.

E. N. Epperly, J. A. Tropp, and R. J. Webber. XTrace: Making the Most of Every Sample in Stochastic Trace Estimation. Matrix Analysis and Applications , 45(1), 2024. Cited on page 5.

J. Gardner, G. Pleiss, K. Q. Weinberger, D. Bindel, and A. G. Wilson. GPyTorch: Blackbox Matrix-matrix Gaussian Process Inference with GPU Acceleration. In Advances in Neural Information Processing Systems , 2018. Cited on pages 1, 3-6, 8, 18, 19, 35.

P. Ghasemi, M. Karbasi, A. Z. Nouri, M. S. Tabrizi, and H. M. Azamathulla. Application of Gaussian process regression to forecast multi-step ahead SPEI drought index. Alexandria Engineering Journal , 60(6), 2021. Cited on page 1.

M. N. Gibbs and D. J. C. MacKay. Efficient Implementation of Gaussian Processes. Technical report, University of Cambridge, 1997. Cited on page 1.

J. Hensman, N. Fusi, and N. D. Lawrence. Gaussian Processes for Big Data. In Uncertainty in Artificial Intelligence , 2013. Cited on page 1.

M. Hutchinson. A stochastic estimator of the trace of the influence matrix for Laplacian smoothing splines. Communications in Statistics - Simulation and Computation , 19(2), 1990. Cited on page 3.

D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations , 2015. Cited on pages 3, 18.

J. A. Lin, J. Antorán, S. Padhy, D. Janz, J. M. Hernández-Lobato, and A. Terenin. Sampling from Gaussian Process Posteriors using Stochastic Gradient Descent. In Advances in Neural Information Processing Systems , 2023. Cited on pages 1, 3-6, 8, 9, 18, 20, 35.

J. A. Lin, S. Padhy, J. Antorán, A. Tripp, A. Terenin, C. Szepesvári, J. M. Hernández-Lobato, and D. Janz. Stochastic Gradient Descent for Gaussian Processes Done Right. In International Conference on Learning Representations , 2024. Cited on pages 1, 3-6, 8, 18, 20, 35.

J. A. Lin, S. Padhy, B. Mlodozeniec, and J. M. Hernández-Lobato. Warm Start Marginal Likelihood Optimisation for Iterative Gaussian Processes. In Advances in Approximate Bayesian Inference , 2024. Cited on page 6.

W. J. Maddox, S. Kapoor, and A. G. Wilson. When are Iterative Gaussian Processes Reliably Accurate? In ICML OPTML Workshop , 2021. Cited on pages 3, 5.

R. Meyer, C. Musco, C. Musco, and D. Woodruff. Hutch++: Optimal Stochastic Trace Estimation. Symposium on Simplicity in Algorithms , 2021, 2021. Cited on page 5.

J. Quiñonero-Candela and C. E. Rasmussen. A Unifying View of Sparse Approximate Gaussian Process Regression. Journal of Machine Learning Research , 6, 1, 2005. Cited on page 1.

A. Rahimi and B. Recht. Random Features for Large-scale Kernel Machines. In Advances in Neural Information Processing Systems , 2008. Cited on pages 3, 18.

C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning . MIT Press, 2006. Cited on page 1.

S. Shalev-Shwartz and T. Zhang. Stochastic Dual Coordinate Ascent Methods for Regularized Loss Minimization. Journal of Machine Learning Research , 14, 2012. Cited on pages 1, 3.

J. Snoek, H. Larochelle, and R. P. Adams. Practical Bayesian Optimization of Machine Learning Algorithms. In Advances in Neural Information Processing Systems , 2012. Cited on page 1.

D. J. Sutherland and J. Schneider. On the error of random Fourier features. In Uncertainty in Artificial Intelligence , 2015. Cited on pages 3, 18.

K. Tazi, J. A. Lin, R. Viljoen, A. Gardner, T. John, H. Ge, and R. E. Turner. Beyond Intuition, a Framework for Applying GPs to Real-World Data. In ICML Structured Probabilistic Inference &Generative Modeling Workshop , 2023. Cited on page 1.

M. K. Titsias. Variational learning of inducing variables in sparse Gaussian processes. In Artificial Intelligence and Statistics , 2009. Cited on page 1.

S. Tu, R. Roelofs, S. Venkataraman, and B. Recht. Large Scale Kernel Learning using Block Coordinate Descent. arXiv:1602.05310 , 2016. Cited on pages 1, 3.

R. Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science . Cambridge University Press, 2018. Cited on pages 13, 14.

K. A. Wang, G. Pleiss, J. R. Gardner, S. Tyree, K. Q. Weinberger, and A. G. Wilson. Exact Gaussian Processes on a Million Data Points. In Advances in Neural Information Processing Systems , 2019. Cited on pages 1, 3, 4, 6, 19, 35.

J. T. Wilson, V. Borovitskiy, A. Terenin, P. Mostowsky, and M. P. Deisenroth. Efficiently Sampling Functions from Gaussian Process Posteriors. In International Conference on Machine Learning , 2020. Cited on pages 2, 3, 5, 18.

J. T. Wilson, V. Borovitskiy, A. Terenin, P. Mostowsky, and M. P. Deisenroth. Pathwise Conditioning of Gaussian Processes. Journal of Machine Learning Research , 22, 1, 2021. Cited on pages 2, 3, 5, 18.

K. Wu, J. Wenger, H. Jones, G. Pleiss, and J. R. Gardner. Large-Scale Gaussian Processes via Alternating Projection. In International Conference on Artificial Intelligence and Statistics , 2024. Cited on pages 1, 3-6, 8, 19, 35.

A Mathematical Derivations

In this appendix, we provide mathematical derivations for claims in the main paper.

A.1 Variance of Standard and Pathwise Gradient Estimator

To compare the variances of the standard estimator (6) and the pathwise estimator (9), we calculate

The variance of the standard estimator is given by

The variance of the pathwise estimator is given by

Therefore, the variances of both estimators share the first trace term and only differ in the second trace term. Hence, their variances will be identical if

which is the case if H -1 θ and ∂ H θ /∂θ k commute with each other.

For example, consider the derivative with respect to the noise scale σ ,

In this case, H -1 θ and ∂ H θ /∂σ commute with each other, such that both estimators have the same variance. In general, a sufficient condition for matrix multiplication to be commutative is simultaneous diagonalisability of two matrices.

A.2 Taylor Approximation View of Warm Start

At iterations t and t +1 of the outer-loop marginal likelihood optimiser, associated with θ ( t ) and θ ( t +1) , the linear system solver must solve two batches of linear systems, namely

where H ( t ) θ and H ( t +1) θ are related through the change from θ ( t ) to θ ( t +1) and v ( t ) y and v ( t +1) y are further related through sharing the same right-hand side y in the linear system. In such a setting, where the coefficient matrix only changes slightly and the right-hand side remains fixed, we can approximate v ( t +1) using a first-order Taylor expansion of H ( t +1) θ ,

If ∆ = H ( t +1) θ -H ( t ) θ is small then v ( t ) will be close to v ( t +1) , such that we can reuse v ( t ) to initialise the linear system solver when solving for v ( t +1) . To satisfy the condition of fixed right-hand sides, we must set z ( t ) j = z j at the cost of introducing some bias throughout optimisation, which turns out to be negligible in practice (see Section 4 and Appendices A.3 and A.4 for details).

A.3 Convergence of Warm Starting Marginal Likelihood Optimisation

Recall the gradient of the marginal likelihood objective:

where H θ ∈ R n × n is a positive semi-definite symmetric matrix, y ∈ R n is a real vector, n is the number of 'data' examples, and d θ is the number of hyperparameters. Also, recall the the warm start estimator ˜ g k ( θ ) to the gradient ∂ L ( θ ) /∂θ k :

where the probe vectors z j are random variables with identity second moments: E [ z j z ⊺ j ] = I , and s is the number of probe vectors in the trace estimator.

Notation We will write S n -1 def = { x ∈ R n : ∥ x ∥ 2 = 1 } for a sphere in R n . For a real matrix A ∈ R m × n , we will denote the operator (spectral) norm sup x ∈S n -1 sup y ∈S m -1 y ⊺ A x with ∥ A ∥ op .

Definition 2 ( Sub-gaussian norm ) . The sub-gaussian norm of a sub-gaussian random variable X is defined as:

Definition 3 (Sub-exponential norm) . The sub-exponential norm of a sub-exponential random variable X is defined as:

We will denote the optimisation domain for the hyperparameters as Θ , where we assume Θ ⊆ R d θ .

Theorem 4. Assume that the probe vectors ( z 1 , . . . , z s ) are zero mean, coordinate-wise independent, and that elements of z j are sub-gaussian with norm ∥ z ji ∥ ψ 2 = σ ∀ i ∈ { 1 , . . . , n } , j ∈ { 1 , . . . , s } . Assume that the sum of the singular values of H -1 θ ∂ H θ ∂θ k is, for all k ∈ { 1 , . . . , d θ } , upper-bounded on the domain of θ by λ max . Then, for all δ > 0 :

The crux of the proof of Theorem 4 comes from bounding the spectral norm of the difference (( ∑ s j =1 z j z T j ) -I ) . To do so, it is useful to introduce the following definitions and lemmas.

Lemma 5 ( Computing the operator norm on a net [29, Exercise 4.4.3] ) . Let A be an n × n matrix and ε ∈ [0 , 1) . Then, for any ε -net Σ ε of the unit sphere S n -1 , we have:

Lemma 6 ( Size of ε -net on S n -1 [29, Corollary 4.2.13] ) . There exists an ε -net on S n -1 with cardinality at most ( 2 ε +1 ) n .

Lemma 7. Let M = 1 s ∑ s j =1 z j z ⊺ j -I be an n × n random matrix, where z ji are independent and identically distributed sub-gaussian random variables with sub-gaussian norm ∥ z ij ∥ ψ 2 = σ , and x ∈ S n -1 any unit vector in R n . Then:

where C 1 , C 2 are absolute constants.

Proof. We can rewrite:

Since ∑ n i =1 x i z ij is a weighted sum of independent sub-gaussian random variables, it is also subgaussian with squared norm:

where C is an absolute constant [29, Proposition 2.6.1]. Hence, since ( ∑ n i =1 x i z ij ) 2 is the square of a sub-gaussian random variable, it must be sub-exponential with the sub-exponential norm (see [29, Lemma 2.7.6]):

Lastly, since ( ∑ n i =1 x i z ij ) 2 is sub-exponential, so will the mean-centered counterpart ( ∑ n i =1 x i z ij ) 2 -E [( ∑ n i =1 x i z ij ) 2 ] = ( ∑ n i =1 x i z ij ) 2 -1 [29, Exercise 2.7.10] with sub-exponential norm:

where C 2 is another absolute constant. Hence, since ( ∑ n i =1 x i z ij ) 2 -1 for j ∈ { 1 , . . . , s } are sub-exponential, zero-mean and independent, we can apply Bernstein's inequality to bound the tail probability of x ⊺ M x :

holds for any β ≥ 0 , where C 1 , C 2 are absolute constants.

Lemma 8. Let M = 1 s ∑ s j =1 z j z ⊺ j -I be an n × n random matrix, where z ji are independent and identically distributed sub-Gaussian random variables with sub-gaussian norm ∥ z ij ∥ ψ 2 = σ . Then:

where C 1 , C 4 are absolute constants.

Proof. Pick an ε -net Σ ε on S n -1 of size at most ( 2 ε +1 ) n (Lemma 6). Then, we can bound the tail probability of the operator norm as:

Setting ε =

where C 4 is an absolute constant.

Lastly, to get the bound into the form P [ ∥ M ∥ op ≥ f ( δ, n, s ) ] ≤ δ , we can note that:

and so by setting β = max (√ n s C 1 log ( 9 2 δ ) , n s C 1 log ( 9 2 δ ) ) C 4 σ we get the desired result:

Proof of Theorem 4. Let ∑ n i =1 q i λ i p T i be a singular value decomposition (SVD) of A def = H -1 θ ∂ H θ ∂θ k , where { q i } n i =1 and { p i } n i =1 are two sets of orthonormal vectors. First, note that we can rewrite:

Therefore, we can bound the norm of the difference as

By Lemma 8, with probability at least 1 -δ we can bound the operator norm of M as:

with C 1 , C 2 absolute constants.

We can apply a union bound over all k ∈ { 1 , . . . , d θ } to bound the probability of the ℓ ∞ -norm of the gradient deviating by a certain amount:

Now, if ˜ g ( θ ) is a conservative field, and so is implicitly a gradient of some (approximate) objective ˜ L : Θ → R , the above result allows us to bound the error on the solution found when optimising using the approximate gradient ˜ g instead of the actual gradient g = ∇L . However, in general, ˜ g ( θ ) need not be strictly conservative. In practice, since ˜ g ( θ ) converges to a conservative field the more samples we take, we may assume that it is close enough to being conservative for the purposes of optimisation on hardware with finite numerical precision. Assuming that ˜ g ( θ ) is conservative allows us to show the following bound on the optimum found when optimising using ˜ g ( θ ) , which is a restatement of Theorem 1:

Theorem 9. Let ˜ g and L be defined as in Theorem 4. Assume ˜ g : Θ → R is a conservative field. Assume the optimisation domain Θ is convex, closed and bounded. Then, with probability at least 1 -δ :

where ∆Θ def = sup θ , θ ′ ∈ Θ ∥ θ ′ -θ ∥ is the maximum distance between two elements in Θ .

Proof. Let ˜ L : Θ → R be an approximate objective implied by the gradient field ˜ g , namely a scalar field such that ∇ ˜ L = ˜ g . Such a scalar field exists if ˜ g is a conservative field, and is unique up to a constant (which does not affect the optimum).

For any two points θ , θ ′ ∈ Θ , with ∆ θ def = θ ′ -θ , we have that

△ Difference of gradients bounded with probability at least (1 -δ ) by Theorem 4

The above inequality holds with probability at least (1 -δ ) . Hence,

Remark 10. Theorem 9 above implies that the objective of the optimum ˜ θ ∗ obtained with the approximate gradients converges to the objective of the true optimum θ ∗ in probability:

which trivially follows from the implication of Theorem 9 that for every α, δ > 0 , we can find an s ∈ N such that L ( ˜ θ ∗ ) ≥ L ( θ ∗ ) -α with probability at least 1 -δ .

Remark 11 (Convexity of Θ ) . We also note that the convexity of the hyperparameter domain Θ is a fairly mild assumption which is satisfied in the majority of practical settings. For example, optimising kernel length scales and noise scale on bounded intervals (10 -10 , 10 10 ) falls within the assumptions, but introducing a 'hole' into the domain (e.g. introducing a constraint like ∥ ϑ -1 ∥ ≥ 0 . 5 ) would break the assumption. In particular, this assumption is not a statement about the convexity of the objective L -in the proof, we allow for the objective to be arbitrarily non-convex, and only assume its differentiability.

A.4 Convergence of Warm Starting the Pathwise Estimator

The result in Appendix A.3 can be trivially extended for the pathwise estimator in (9) for any pairwise independent probe vectors ˆ z j (with second moment H -1 θ ) that upon rescaling by H 1 2 θ will be zero-mean with independent coordinates. This is true for probe vectors ˆ z j that are either i.i.d. N ( 0 , H -1 θ ) -distributed or obtained by transforming Radamacher random variables by H -1 2 θ .

B Implementation and Experiment Details

In this appendix, we provide details about our implementation and experiments.

General Our implementation uses the JAX library [4]. All reported experiments were conducted on internal NVIDIA A100-SXM4-80GB GPUs using double floating point precision. Some additional experiments and ablations were performed on Google Cloud TPUs (v4). The total compute time, including preliminary and failed experiments, and evaluation is around 4500 hours. The compute time of individual runs is reported in Tables 2 to 10. The source code is available here.

Datasets Our experiments are conducted using the datasets and data splits from the popular UCI regression benchmark [7]. They consist of various high-dimensional, multivariate regression tasks and are available under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. In particular, we used the POL ( n = 13500 , d = 26 ), ELEVATORS ( n = 14940 , d = 18 ), BIKE ( n = 15642 , d = 17 ), PROTEIN ( n = 41157 , d = 9 ), KEGGDIRECTED ( n = 43945 , d = 20 ), 3DROAD ( n = 391387 , d = 3 ), SONG ( n = 463811 , d = 90 ), BUZZ ( n = 524925 , d = 77 ), and HOUSEELECTRIC ( n = 1844352 , d = 11 ) datasets.

Kernel Function and Random Features In all experiments, we used the Matérn- 3 /2 kernel, parameterised by a scalar signal scale and a length scale per input dimension. For pathwise conditioning (3) and the pathwise gradient estimator (9), we used random Fourier features [21, 25] (1000 sin/cos pairs, 2000 features in total) to draw approximate samples from the Gaussian process prior. For an explanation about how to efficiently sample prior functions from a Gaussian process using random features, we refer to existing literature [31, 32, 15]. However, we want to discuss some details in terms of using this technique for the pathwise estimator from Section 3.

For pathwise gradient estimation, the linear system solver must solve linear systems of the form

with ξ = f ( x ) + ε , where f ( x ) ∼ N ( 0 , k ( x , x ; ϑ )) is a prior function f sample evaluated at the training data x , and ϵ ∼ N ( 0 , σ 2 I ) is a Gaussian random vector. Both quantities are resampled in each outer-loop marginal likelihood step if the pathwise estimator is used without warm starting. With warm starting enabled, the right-hand sides of the linear system must not be resampled. In this case, f and ϵ are sampled once and fixed afterwards. However, f depends on ϑ and ϵ depends on σ , and both ϑ and σ are hyperparameters which change in each outer-loop step. Therefore, what does it mean to keep f and ϵ fixed?

For ϵ , this amounts to the reparameterisation ϵ = σ w , where w ∼ N ( 0 , I ) is sampled once and fixed afterwards, such that ϵ becomes deterministic. For f , this refers to fixing the parameters of the random features, for example the frequencies in the case of random Fourier features. Intuitively, this corresponds to selecting a particular instance of a prior sample, although the distribution of the sample can change due to changes in the hyperparameters. In each outer-loop step, the random features are evaluated using the fixed random feature parameters and the updated kernel hyperparameters, and the prior function sample is then evaluated at the training data using the updated random features. Both of these operations are O ( n ) and efficient as long as the number of random features is reasonable.

Iterative Optimiser To optimise hyperparameters θ given an estimate of ∇L , we used the Adam optimiser [14] with default settings except for the learning rate. For all small datasets ( n< 50k ), we initialised the hyperparameters at 1.0 and used a learning rate of 0.1 to perform 100 steps of Adam. For all large datasets ( n> 50k ), we initialised the hyperparameters using a heuristic and used a learning rate of 0.03 to perform 30 steps of Adam (15 for HOUSEELECTRIC due to high computational costs). The heuristic to obtain initial hyperparameters for the large datasets consists of:

Select a centroid data example uniformly at random from the training data.

Find the 10k data examples with the smallest Euclidean distance to the centroid.

Obtain hyperparameters by maximising the exact marginal likelihood using this subset.

Repeat the procedure with 10 different centroids and average the hyperparameters.

This heuristic has previously been used by Lin et al. [15, 16] to avoid aliasing bias.

To enforce positive value constraints during hyperparameter optimisation, we used the softplus function. In particular, we reparameterise each hyperparameter θ k ∈ R > 0 as θ k = log(1 + exp( ν k )) and apply optimiser steps to ν k ∈ R instead, to facilitate unconstrained optimisation.

Gradient Estimator For all experiments, unless otherwise specified, the number of probe vectors was set to s = 64 for both the standard and the pathwise estimator. The distributions of the probe vectors are z ∼ N ( 0 , I ) for the standard estimator and ˆ z := H -1 θ ξ ∼ N ( 0 , H -1 θ ) for the pathwise estimator. See Section 3 for details about how to generate samples from N ( 0 , H -1 θ ) .

The probe vectors used by Gardner et al. [9] have conceptual similarities but the motivation is different. They used probe vectors z ∼ N ( 0 , P ) , where P is constructed using a low-rank pivoted Cholesky decomposition to implement the preconditioner. In contrast, our pathwise probe vectors are sampled using random features, and, for CG, we used the pivoted Cholesky preconditioner in addition to the pathwise probe vectors. Furthermore, we used the solution of the pathwise probe vector systems to construct posterior samples via pathwise conditioning, which has not been done by Gardner et al. [9].

The name of the pathwise estimator can be related to the reparameterisation trick by viewing a sample from the GP posterior as a deterministic transformation of a sample from the GP prior, and observing that the latter itself is an affine transformation of a standard normal random variable.

Linear System Solver We conducted two sets of experiments which only differ in the termination criterion of the linear system solver. In the first set, we stop linear system solvers once they reach a relative residual norm tolerance of τ = 0 . 01 . In the second set, we also restrict the maximum number of solver epochs to 10, 20, 30, 40 or 50, such that most of the time the residual norm does not reach τ .

For a generic system of linear equations H θ u = b , the residual is defined as r = b -H θ u and the relative residual norm is defined as ∥ r ∥ / ∥ b ∥ . In practice, to improve numerical stability, the relative residual norm tolerance is implemented by solving the system H θ ˜ u = ˜ b , where ˜ b := b / ( ∥ b ∥ + ϵ ) , until ∥ ˜ r ∥ := ∥ ˜ b -H θ ˜ u ∥ ≤ τ and then returning u := ( ∥ b ∥ + ϵ ) ˜ u , where epsilon is set to a small constant value to prevent division by zero. Since we are solving batches of systems of linear equations of the form H θ [ v y , v 1 , . . . , v s ] = [ y , z 1 , . . . , z s ] , we track the residuals of each individual system and calculate separate residual norms for the mean and for the probe vectors, where the residual norm for the mean ∥ r y ∥ corresponds to the system H θ v y = y and the residual norm for the probe vectors ∥ r z ∥ is defined as the arithmetic average over residual norms corresponding to the systems H θ [ v 1 , . . . , v s ] = [ z 1 , . . . , z s ] . Both relative residual norms must reach the tolerance τ to satisfy the termination criterion. We use separate residual norms because ∥ r y ∥ typically converges faster than ∥ r z ∥ , such that an average other all systems tends to dilute the latter (see Figures 14 to 17).

Conjugate Gradients The conjugate gradients algorithm [9, 30] computes necessary residuals as part of the algorithm. In terms of counting solver epochs, every conjugate gradient iteration counts as one solver epoch because in every iteration each value of H θ is computed once. Following previous work, we used a pivoted Cholesky preconditioner of rank 100 for all experiments [30]. We initialised conjugate gradients either at zero (no warm start) or at the previous solution (warm start). Otherwise, conjugate gradients does not have any other parameters. Pseudocode is provided in Algorithm 1.

Alternating Projections The alternating projections algorithm [33] also keeps track of the residuals as part of the algorithm. In terms of counting solver epochs, we convert the number of maximum solver epochs to a maximum number of solver iterations by multiplying with n/b , where b is the block size, because every iteration of alternating projections computes b/n of all entries of H θ . We used a block size of b = 1000 for all datasets, except PROTEIN and KEGGDIRECTED, where we used b = 2000 instead. We initialised alternating projections either at zero (no warm start) or at the previous solution (warm start). During each marginal likelihood step, the Cholesky factorisation of every block is computed once and cached afterwards (although, in practice, the Cholesky factorisation does not dominate the computational costs). In each iteration of alternating projections, the block with largest residual norm is selected to be processed. Pseudocode is provided in Algorithm 2.

Algorithm 2

Alternating projections for solving

H

θ

[

v

y

,

v

1

, . . . ,

v

s

] = [

y

,

z

1

, . . . ,

z

s

]

Stochastic Gradient Descent The stochastic gradient descent algorithm [15, 16] does not compute residuals as part of the algorithm. Therefore, we estimate the current residual by keeping a residual vector in memory and updating it sparsely whenever we compute the gradient on a batch of data, leveraging the property that the negative gradient is equal to the residual. In practice, we find that this estimates an approximate upper bound on the true residual, which becomes fairly accurate after a few iterations. In terms of counting solver epochs, we apply the same procedure as for alternating projections. The number of maximum solver epochs is converted to a maximum number of solver iterations by multiplying with n/b , where b is the batch size. We used a batch size of b = 500 , momentum of ρ = 0 . 9 , and no Polyak averaging, because averaging is not strictly necessary [16] and would interfere with our residual estimation heuristic. We use learning rates of 30, 20, 30, 20, and 20 respectively for the POL, ELEVATORS, BIKE, KEGGDIRECTED and PROTEIN datasets, picking the largest learning rate from a grid [5 , 10 , 20 , 30 , 50 , 60 , 70 , 80 , 90 , 100] that does not cause the inner linear system solver to diverge on the very first outer marginal likelihood loop. For the larger datasets, we use learning rates of 10, 10, 50, and 50 for 3DROAD, BUZZ, SONG and HOUSEELECTRIC, picking half of the largest learning rate as above. We find that the larger datasets are more sensitive to diverging when the hyperparameters change, and therefore we choose half of the largest learning rate possible at initialisation. Pseudocode is provided in Algorithm 3.

Algorithm 3

Stochastic gradient descent for solving

H

θ

[

v

y

,

v

1

, . . . ,

v

s

] = [

y

,

z

1

, . . . ,

z

s

]

C Additional Empirical Results

In this appendix, we provide additional result from our experiments.

For our first experiment (solving until reaching the relative residual norm tolerance), we present the predictive performance, time taken, and speed-ups in Tables 2 to 6. Hyperparameter trajectories are illustrated in Figures 11 to 13. Required number of solver iterations are shown in Figure 21.

For our second experiment (solving until reaching the tolerance or exhausting the compute budget), we tabulate the predictive performance, time taken, and average residual norms in Tables 7 to 10. The behaviour of residual norms is visualised in Figures 14 to 17. The evolution of residual norms and predictive performance via pathwise conditioning is depicted in Figures 18 to 20.

pol

Selected Length Scales

0

.

0

0

.

5

1

.

0

0

1

2

0

1

2

1

2

3

1

2

3

0

.

0

0

.

5

1

.

0

Signal Scale

1

3

5

1

3

5

1

3

5

1

3

5

elevators

Selected Length Scales

0

.

0

0

.

5

1

.

0

1

3

5

1

2

3

1

2

3

1

2

3

0

.

0

0

.

5

1

.

0

Signal Scale

1

2

3

1

2

3

1

2

3

1

2

3

bike

Selected Length Scales

0

.

0

0

.

5

1

.

0

0

2

4

0

5

10

0

5

10

0

5

10

0

.

0

0

.

5

1

.

0

Signal Scale

0

1

2

0

5

10

0

5

10

0

5

10

protein

Selected Length Scales

0

.

0

0

.

5

1

.

0

0

1

2

0

2

4

0

2

4

0

1

2

0

.

5

1

.

0

1

.

5

Signal Scale

0

1

2

0

1

2

0

1

2

0

1

2

keggdirected

Selected Length Scales

0

.

0

0

.

5

1

.

0

1

3

5

1

3

5

1

3

5

1

3

5

0

50

100

0

.

0

0

.

5

1

.

0

Signal Scale

0

50

100

1

3

5

0

50

100

1

3

5

0

50

100

1

4

7

0

50

100

1

4

7

Conjugate Gradients

Marginal Likelihood Steps

Exact

Standard

Pathwise

Standard + warm start

Pathwise + warm start

pol

Selected Length Scales

0

.

0

0

.

5

1

.

0

0

1

2

0

1

2

1

2

3

1

2

3

0

.

0

0

.

5

1

.

0

Signal Scale

1

3

5

1

3

5

1

3

5

1

3

5

elevators

Selected Length Scales

0

.

0

0

.

5

1

.

0

1

3

5

1

2

3

1

2

3

1

2

3

0

.

0

0

.

5

1

.

0

Signal Scale

1

2

3

1

2

3

1

2

3

1

2

3

bike

Selected Length Scales

0

.

0

0

.

5

1

.

0

0

2

4

0

5

10

0

5

10

0

5

10

0

.

0

0

.

5

1

.

0

Signal Scale

0

1

2

0

5

10

0

5

10

0

5

10

protein

Selected Length Scales

0

.

0

0

.

5

1

.

0

0

1

2

0

2

4

0

2

4

0

1

2

0

.

5

1

.

0

1

.

5

Signal Scale

0

1

2

0

1

2

0

1

2

0

1

2

keggdirected

Selected Length Scales

0

.

0

0

.

5

1

.

0

1

3

5

1

3

5

1

3

5

1

3

5

0

50

100

0

.

0

0

.

5

1

.

0

Signal Scale

0

50

100

1

3

5

0

50

100

1

3

5

0

50

100

1

4

7

0

50

100

1

4

7

Alternating Projections

Marginal Likelihood Steps

Exact

Standard

Pathwise

Standard + warm start

Pathwise + warm start

pol

Selected Length Scales

0

.

0

0

.

5

1

.

0

0

1

2

0

1

2

1

2

3

1

2

3

0

.

0

0

.

5

1

.

0

Signal Scale

1

3

5

1

3

5

1

3

5

1

3

5

elevators

Selected Length Scales

0

.

0

0

.

5

1

.

0

1

3

5

1

2

3

1

2

3

1

2

3

0

.

0

0

.

5

1

.

0

Signal Scale

1

2

3

1

2

3

1

2

3

1

2

3

bike

Selected Length Scales

0

.

0

0

.

5

1

.

0

0

2

4

0

5

10

0

5

10

0

5

10

0

.

0

0

.

5

1

.

0

Signal Scale

0

1

2

0

5

10

0

5

10

0

5

10

protein

Selected Length Scales

0

.

0

0

.

5

1

.

0

0

1

2

0

2

4

0

2

4

0

1

2

0

.

5

1

.

0

1

.

5

Signal Scale

0

1

2

0

1

2

0

1

2

0

1

2

keggdirected

Selected Length Scales

0

.

0

0

.

5

1

.

0

1

3

5

1

3

5

1

3

5

1

3

5

0

50

100

0

.

0

0

.

5

1

.

0

Signal Scale

0

50

100

1

3

5

0

50

100

1

3

5

0

50

100

1

4

7

0

50

100

1

4

7

Stochastic Gradient Descent

Marginal Likelihood Steps

Exact

Standard

Pathwise

Standard + warm start

Pathwise + warm start

0

.

0

0

.

4

0

.

8

1

.

2

pol

CG

0

.

0

0

.

1

0

.

2

0

.

3

elev

0

.

0

0

.

4

0

.

8

1

.

2

Residual Norm of Mean

bike

0

.

0

0

.

7

1

.

4

2

.

1

prot

0

25

50

75

100

0

.

0

0

.

7

1

.

4

2

.

1

0

.

00

0

.

01

0

.

02

0

.

03

AP

0

.

00

0

.

03

0

.

06

0

.

09

0

.

00

0

.

01

0

.

02

0

.

03

0

.

00

0

.

05

0

.

10

0

.

15

0

25

50

75

100

Marginal Likelihood Steps

0

.

00

0

.

01

0

.

02

0

.

03

0

.

00

0

.

04

0

.

08

0

.

12

SGD

0

.

00

0

.

13

0

.

26

0

.

39

0

.

00

0

.

04

0

.

08

0

.

12

0

.

00

0

.

12

0

.

24

0

.

36

0

25

50

75

100

0

.

00

0

.

04

0

.

08

0

.

12

0

.

06

0

.

07

0

.

08

0

.

09

Test RMSE

0

.

33

0

.

35

0

.

37

0

.

39

0

.

01

0

.

03

0

.

05

0

.

07

0

.

48

0

.

50

0

.

52

0

.

54

CG

AP

SGD

0

.

00

0

.

13

0

.

26

0

.

39

Maximum solver epochs per marginal likelihood step:

Standard estimator

Standard estimator + warm start

10

20

30

40

50

0

.

0

0

.

3

0

.

6

0

.

9

pol

CG

0

.

0

0

.

1

0

.

2

0

.

3

elev

0

.

0

0

.

3

0

.

6

0

.

9

Residual Norm of Mean

bike

0

.

0

0

.

6

1

.

2

1

.

8

prot

0

25

50

75

100

0

.

0

0

.

2

0

.

4

0

.

6

0

.

00

0

.

01

0

.

02

0

.

03

AP

0

.

00

0

.

02

0

.

04

0

.

06

0

.

00

0

.

01

0

.

02

0

.

03

0

.

00

0

.

04

0

.

08

0

.

12

0

25

50

75

100

Marginal Likelihood Steps

0

.

00

0

.

01

0

.

02

0

.

03

0

.

00

0

.

04

0

.

08

0

.

12

SGD

0

.

00

0

.

12

0

.

24

0

.

36

0

.

00

0

.

04

0

.

08

0

.

12

0

.

00

0

.

12

0

.

24

0

.

36

0

25

50

75

100

0

.

00

0

.

03

0

.

06

0

.

09

0

.

06

0

.

07

0

.

08

0

.

09

Test RMSE

0

.

34

0

.

35

0

.

36

0

.

37

0

.

00

0

.

03

0

.

06

0

.

09

0

.

48

0

.

49

0

.

50

0

.

51

CG

AP

SGD

0

.

03

0

.

06

0

.

09

0

.

12

Maximum solver epochs per marginal likelihood step:

Pathwise

Pathwise + warm start

10

20

30

40

50

0

2

4

6

pol

CG

0

.

0

0

.

1

0

.

2

0

.

3

elev

0

.

0

1

.

5

3

.

0

4

.

5

Residual Norm of Probe Vectors

bike

0

1

2

3

prot

0

25

50

75

100

0

2

4

6

0

.

0

0

.

2

0

.

4

0

.

6

AP

0

.

00

0

.

09

0

.

18

0

.

27

0

.

00

0

.

15

0

.

30

0

.

45

0

.

00

0

.

15

0

.

30

0

.

45

0

25

50

75

100

Marginal Likelihood Steps

0

.

00

0

.

11

0

.

22

0

.

33

0

.

0

0

.

4

0

.

8

1

.

2

SGD

0

.

0

0

.

4

0

.

8

1

.

2

0

.

0

0

.

4

0

.

8

1

.

2

0

.

0

0

.

3

0

.

6

0

.

9

0

25

50

75

100

0

.

0

0

.

4

0

.

8

1

.

2

-

2

.

3

-

1

.

0

0

.

3

1

.

6

Test LLH

-

4

.

8

-

3

.

2

-

1

.

6

0

.

0

1

.

0

1

.

5

2

.

0

2

.

5

-

1

.

0

-

0

.

8

-

0

.

6

-

0

.

4

CG

AP SGD

-

3

.

5

-

1

.

8

-

0

.

1

1

.

6

Maximum solver epochs per marginal likelihood step:

Standard estimator

Standard estimator + warm start

10

20

30

40

50

0

.

0

0

.

4

0

.

8

1

.

2

pol

CG

0

.

0

0

.

2

0

.

4

0

.

6

elev

0

.

0

0

.

4

0

.

8

1

.

2

Residual Norm of Probe Vectors

bike

0

.

0

0

.

6

1

.

2

1

.

8

prot

0

25

50

75 100

0

.

0

0

.

6

1

.

2

1

.

8

0

.

00

0

.

04

0

.

08

0

.

12

AP

0

.

00

0

.

04

0

.

08

0

.

12

0

.

00

0

.

04

0

.

08

0

.

12

0

.

00

0

.

04

0

.

08

0

.

12

0

25

50

75 100

Marginal Likelihood Steps

0

.

00

0

.

03

0

.

06

0

.

09

0

.

0

0

.

2

0

.

4

0

.

6

SGD

0

.

0

0

.

2

0

.

4

0

.

6

0

.

0

0

.

2

0

.

4

0

.

6

0

.

0

0

.

1

0

.

2

0

.

3

0

25

50

75 100

0

.

0

0

.

2

0

.

4

0

.

6

0

.

90

1

.

05

1

.

20

1

.

35

Test LLH

-

0

.

41

-

0

.

40

-

0

.

39

-

0

.

38

-

0

.

74

0

.

32

1

.

38

2

.

44

-

0

.

70

-

0

.

66

-

0

.

62

-

0

.

58

CG

AP SGD

-

2

.

35

-

1

.

04

0

.

27

1

.

58

Maximum solver epochs per marginal likelihood step:

Pathwise

Pathwise + warm start

10

20

30

40

50

0

5

10

15

0

.

4

0

.

7

1

.

0

house

Res. Norm Mean

0

5

10

15

1

.

0

1

.

4

1

.

8

0

5

10

15

0

.

4

0

.

7

1

.

0

Test RMSE

0

5

10

15

-

8

-

4

0

Test LLH

0

10

20

30

0

.

6

2

.

6

4

.

6

buzz

0

10

20

30

1

.

0

2

.

7

4

.

4

0

10

20

30

0

.

6

2

.

6

4

.

6

0

10

20

30

-

14

-

7

0

0

10

20

30

0

.

0

1

.

1

2

.

2

song

0

10

20

30

0

.

10

1

.

25

2

.

40

0

10

20

30

0

.

6

1

.

5

2

.

4

0

10

20

30

-

2

.

2

-

1

.

6

-

1

.

0

0

10

20

30

0

.

4

0

.

9

1

.

4

3droad

0

10

20

30

0

.

7

0

.

9

1

.

1

0

10

20

30

0

.

4

0

.

9

1

.

4

0

10

20

30

-

1

.

6

-

1

.

2

-

0

.

8

Conjugate Gradients

Marginal Likelihood Steps

Pathwise estimator

Pathwise estimator + warm start

Alternating Projections

0

5

10

15

0

.

00

0

.

02

0

.

04

house

Res. Norm Mean

0

5

10

15

0

.

03

0

.

06

0

.

09

0

5

10

15

0

.

02

0

.

03

0

.

04

Test RMSE

0

5

10

15

0

.

0

1

.

3

2

.

6

Test LLH

0

10

20

30

0

.

00

0

.

03

0

.

06

buzz

0

10

20

30

0

.

01

0

.

06

0

.

11

0

10

20

30

0

.

27

0

.

29

0

.

31

0

10

20

30

-

0

.

13

-

0

.

08

-

0

.

03

0

10

20

30

0

.

00

0

.

03

0

.

06

song

0

10

20

30

0

.

00

0

.

04

0

.

08

0

10

20

30

0

.

74

0

.

75

0

.

76

0

10

20

30

-

1

.

13

-

1

.

12

-

1

.

11

0

10

20

30

0

.

01

0

.

06

0

.

11

3droad

0

10

20

30

0

.

03

0

.

06

0

.

09

0

10

20

30

0

.

04

0

.

08

0

.

12

0

10

20

30

-

0

.

4

0

.

5

1

.

4

Marginal Likelihood Steps

Pathwise estimator

Pathwise estimator + warm start

Stochastic Gradient Descent

0

5

10

15

0

.

03

0

.

04

0

.

05

house

Res. Norm Mean

0

5

10

15

0

.

10

0

.

15

0

.

20

0

5

10

15

0

.

03

0

.

04

0

.

05

Test RMSE

0

5

10

15

0

.

2

1

.

0

1

.

8

Test LLH

0

10

20

30

0

.

00

0

.

15

0

.

30

buzz

0

10

20

30

0

.

0

0

.

3

0

.

6

0

10

20

30

0

.

26

0

.

29

0

.

32

0

10

20

30

-

0

.

2

-

0

.

1

0

.

0

0

10

20

30

0

.

00

0

.

05

0

.

10

song

0

10

20

30

0

.

00

0

.

05

0

.

10

0

10

20

30

0

.

74

0

.

75

0

.

76

0

10

20

30

-

1

.

13

-

1

.

12

-

1

.

11

0

10

20

30

0

.

03

0

.

09

0

.

15

3droad

0

10

20

30

0

.

04

0

.

07

0

.

10

0

10

20

30

0

.

05

0

.

10

0

.

15

0

10

20

30

-

2

.

5

-

0

.

5

1

.

5

Marginal Likelihood Steps

Pathwise estimator

Pathwise estimator + warm start

0

75

150

225

pol

CG

0

16

32

48

elev

0

75

150

225

Solver Iterations

bike

0

70

140

210

prot

0

25

50

75

100

0

50

100

150

0

75k

150k

225k

AP

0

8k

16k

24k

0

60k

120k

180k

0

12k

24k

36k

0

25

50

75

100

Marginal Likelihood Steps

0

30k

60k

90k

0

70k

140k

SGD

0

2k

4k

6k

0

200k

400k

600k

0

13k

26k

39k

0

25

50

75

100

0

100k

200k

300k

1

.

25

1

.

26

1

.

27

1

.

28

Test LLH

-

0

.

40

-

0

.

39

-

0

.

38

-

0

.

37

2

.

02

2

.

08

2

.

14

2

.

20

-

0

.

64

-

0

.

62

-

0

.

60

-

0

.

58

CG

AP SGD

1

.

06

1

.

07

1

.

08

1

.

09

Standard

Pathwise

Standard + warm start

Pathwise + warm start

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the contributions of Sections 3 to 5 and empirical results illustrated in Table 1 and Figure 10. Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes, the paper discusses the limitations of our work. In particular, the dependence on random features to sample prior functions is discussed in Section 3 and Figure 5, and the bias due to warm starting is discussed in Section 4 and Theorem 1.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All theoretical claims and results made in the paper are formally derived in Appendix A, including all assumptions and proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Methods and experiments are described in Sections 3 to 5, with implementation details, pseudocode, and link to source code being provided in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide open access to our source code required to reproduce our experiments by uploading it to an anonymous repository during the review phase and moving it to a public repository upon acceptance. The link to the repository is provided in Appendix B. In terms of data, we only use publicly available regression datasets, which are also discussed in Appendix B.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines (HTTPS://NIPS.CC/PUBL IC/GUIDES/CODESUBMISSIONPOLICY) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (HTTPS: //NIPS.CC/PUBLIC/GUIDES/CODESUBMISSIONPOLICY) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All datasets and train/test splits used in this paper are publicly available [7]. Experimental settings are explained in Sections 3 to 5 with details being provided in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All experiments are repeated 10 × using the train/test data splits provided by [7], except for the 3 largest datasets, where only 5 splits were used due to computational costs. Tables in the main paper only report the mean over data splits due to space constraints. Tables in the appendix report the mean ± standard errors over data splits. In figures, mean ± standard errors over data splits are reported via error bars when appropriate.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Details about compute resources are provided in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics HTTPS://NEURIPS.CC/PUBLIC/ETHICSGUIDELINES?

Answer: [Yes]

Justification: The research conducted in this paper does not involve any human subjects or participants and only uses curated, publicly available datasets [7]. Additionally, the research conducted in this paper considers generic regression tasks and does not pose a high risk in terms of safety, security, discrimination, surveillance, deception and harassment, environment, human rights, or bias and fairness. While specific applications of regression methods could potentially facilitate some of these harmful consequences, such applications are by no means a topic of the research conducted in this paper. Furthermore, implementation details and other information which is required to reproduce the research conducted in this paper is thoroughly provided in Appendix B.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: In this paper, we discuss and analyse techniques hyperparameter optimisation in Gaussian processes, which is foundational research and not tied to particular applications or deployment. There is no direct path to generating content such as fake speeches, images, or videos. Potential harm which could arise from intended use and correct functioning are thus minimal. Potential harm due to intended use but incorrect functioning is very unlikely because Gaussian processes are specifically designed to produce calibrated uncertainty estimates. While any general purpose regression algorithm, including Gaussian process regression, technically has the potential to be used in harmful applications or technology, it is by no means topic of this paper. Furthermore, our paper conducts empirical studies on publicly available datasets which do not have a harmful downstream impact and which are a standard benchmark in the literature [9, 30, 15, 16, 33].

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper uses curated, publicly available regression datasets [7] and the proposed methods do not pose a high risk for misuse or dual-use because they are for generic regression and, in particular, do not process or generate natural language, images, nor videos.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper uses datasets from the popular UCI regression benchmark made available through a GitHub repository [7] under the CC BY 4.0 license. This information is provided in Appendix B.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, PAPERSWITHCODE.COM/DATASETS has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets except the code required to reproduce our experiments, which has already been discussed earlier in this checklist.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on solving linear systems arising from large-scale Gaussian process hyperparameter optimisation.\n\nThe first technique proposed in this paper is a reformulation of the linear system for log determinant gradient estimation, which is called the \"pathwise estimator\".\nThe authors argue that this reformulation makes the all-zero initialization closer to the solution.\nThis reformulation also allows them to reuse the linear system solutions in the prediction time by pathwise conditioning (Wilson et al., 2021).\n\nThe second technique proposed in this paper is warm starting the linear solvers.\nThough, warm starts require using a set of **fixed** probe vectors, which necessarily introduces biases to the hyperparameter optimization.\nThe authors demonstrate empirically that the bias does not hurt in practice.\n\nSTRENGTHS:\n- The writing is very clear.\n- By combining several techniques, this paper achieves large speed-up factors over a wide range of datasets.\nBesides the speed-up, it appears that the pathwise estimator with warm start also stabilizes the linear systems solvers in some cases.\n- The empirical evaluations are extensive with datasets ranging from 10 thousand data points to 1.8 million data points.\nThe extensive experiments with recent scalable GP methods could provide tips and tricks to practitioners.\n\nWEAKNESSES:\nThe main weakness is that the techniques are not super novel per se.\n\nThere exists something similar to the pathwise estimator proposed in this paper.\nFor example, see Equation (7) in Gardner et al. (2018), where the probe vectors are sampled from $\\mathcal{N}(\\mathbf{0}, \\mathbf{P})$ with $\\mathbf{P} \\approx \\mathbf{H}_{\\mathbf{\\theta}}$.\nThough, the original motivation in Gardner et al. (2018) is different from this paper.\nNevertheless, they do appear to be similar at least on a conceptual level.\n\nIn addition, warming starting the linear solvers has been used in practice by Artemev et al. (2021) and Antoran et al. (2023).\n\nHowever, the main contribution of this paper is combining all techniques together and show their effectiveness on a wide range of datasets.\nFuture work could benefit from the the insights and empirical evaluations in this paper.\n\nMinor Weaknesses:\n- Theorem 1 is not very interesting.\nThis theorem is based on the fact that the gradient bias goes to zero as the number of probe vectors goes to infinity.\nMy impression is that the authors wrote this theorem to make the paper looks more \"theoretical\".\nFrankly, I think removing this theorem makes the paper look cleaner.\nA middle ground would be just presenting how fast the bias goes to zero as $s \\to \\infty$.\n\n- The name \"pathwise estimator\" might be somewhat confusing to some readers.\nThis name is usually reserved for the reparameterization trick gradient, which is evidently not the case for this paper.\nI suggest the authors think about alternative names for this estimator.\nAn possible option is to simply rephrase Section 3 as \"a reformulation of the linear systems\".\n\n- Line 172: footnote should be after the period.\n\nQUESTIONS:\n- The conjugate gradient method estimates both the log marginal likelihood (by tridiagonalization) and its gradient.\nI am wondering if using the pathwise estimator for conjugate gradient produces an estimate for the log marginal likelihood.\nThis is useful monitoring the progress of hyperparameter optimization.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of simultaneously fitting a Gaussian process (GP) to data along with determining the hyperparameters (kernel width, noise variance) for the GP.  The overall algorithm is standard, consisting of an outer loop, which is a simple gradient update to the hyperparameters, and an inner loop that fits the GP to the data.  Fitting the data of course requires a system solve with the kernel matrix.  Computing the gradient (or an approximation thereof) for the hyperparamters requires multiple system solves.\n\nThe paper gives some numerical experiements and some rough reasoning to argue that tweaking the trace estimator used for the gradient estimate and warm starting the linear solver saves a tangible amount of computation in this process.\n\nSTRENGTHS:\nHyperparameter optimization for fitting GPs is an important and prevalent problem in machine learning.  \n\nThe numerical experiments are thorough and appear to be carefully done, and lend good supporting evidence to the efficacy of the method.\n\nWEAKNESSES:\nIt is hard for me to see what the really new ideas are in this paper.  The the authors say, the most critical part of this whole process is the linear system solver.  They do not actually propose any new type of solver, but rather show that by modifying the way the gradient of the hyperparameters is estimated and using a warm start, the linear solver they happen to be using will take fewer iterations.  The idea of warm starting a linear solver using previous solutions is pervasive throughout numerical analysis, so it is no surprise that it gives you some gains.  The main novelty in the paper seems to be changing the trace estimator; that is, using equation (11) in place of (7).  \n\nSomething which is not really discussed, but seems important, is that the z in (7) are easier to generate than the $\\xi$ in (11).  The z can be random vectors with independent entries, the $\\xi$ need a prescribed covaraince structure that changes at every iteration.  Since this covariance matrix (the kernel matrix) is large, taking a random draw of the $\\xi$ seems nontrivial.  How this works, and why it does not have a significant effect on the number of computations needed, needs to be spelled out more clearly.\n\nThe numerical experiements are on interesting problems, but the paper would be stronger if the mathematical arguments were more precise.\n\nQUESTIONS:\nHow are the $\\xi$ in (11) generated and what is the computational cost relative to the system solves?\n\nLIMITATIONS:\nnone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents approaches that can speed up solving linear systems arising in the GP regression problem. The two basic ideas are a warm start and limiting the computation budget. Another novelty is the pathwise gradient estimator, which leads to fewer iterations needed for convergence. The detailed numerical experiments confirm the performance of the suggested approaches and raise questions on the connection between the accurate solving of linear systems and the performance of the GP regression method.\n\nSTRENGTHS:\n1. The novel approach for gradient estimation is proposed and demonstrates significant speed-up.\n2. Different techniques to accelerate solvers for linear systems are proposed, and their efficiency is confirmed via extensive experimental evaluation. This evaluation includes target solvers CG, AP, and SGD.\n3. The manuscript is well-prepared, the motivation and suggestions are clear, and the results are well-explained.\n4. The presented techniques can be easily incorporated into the existing packages for GP regression and used to solve practically important problems.\n\nWEAKNESSES:\nIn this paper, the authors use some heuristics to speed up linear solvers and suggest a new gradient estimator based on the transformation of the normal distribution. Although the suggested tricks demonstrate improvement, their theoretical interpretation and the possibility of obtaining even better results remain unclear. So, a more detailed discussion of further research directions related to the presented approaches would be interesting. The theoretical bounds for the potential gain obtained from linear solvers while preserving test error can make the work more solid and fundamental.\n\nQUESTIONS:\n1. The authors mentioned in the Appendix that they use precondition based on the Cholesky factorization. Please provide more details since, typically, such preconditioners, their structure, and properties significantly affect the performance of CG. \n2. Please add bold labeling the top timing and test log-likelihood values in Table 1 to simplify parsing such a large number of values\n3. Why are large datasets (lines 219-235) not included in Table 1 or a similar table where the runtime gain can be observed? Please add these results, too. The large datasets can illustrate the gain from the introduced approach even better.\n\nLIMITATIONS:\nThe authors do not provide explicit limitations of the suggested approach.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper investigates several iterative techniques for solving linear systems when applied to the problem of finding GP hyperparameters. Specifically, the following modifications to the method are suggested:
- A "pathwise" sampling estimator for the Hutchinson trace estimator
- Warm starting linear systems solvers

After this, a thorough investigation is made into the effect of these choices on training GP hyperparameters.

STRENGTHS:
The method follows an existing schema for training GP hyperparameters, and performs an in-depth investigation into how different choices affect the algorithm.

WEAKNESSES:
The main weakness of this paper is that it does not compare the results of the approximate methods to a reliable exact method. This makes it impossible to determine from the paper whether the predictive performances are actually good.

For datasets of these sizes (<50k), it is feasible on modern hardware to run a Cholesky on the full dataset. This should be run with BFGS optimisation. Reporting these results will give a ground-truth comparison for how good the resulted test metrics are.

If these are close, then readers will know how much to value these speedups. If they are distant, readers will be able to determine how much they value a faster approximation that does not get close to optimal predictor.

Without a comparison to any other method, the value of the overall method cannot be established. I'm happy to discuss this and significantly increase my score if this is addressed sufficiently.

QUESTIONS:
- Is there any information in the paper that gives a comparison to another method, or ideally a ground truth evaluation of an exact GP?

LIMITATIONS:
- No evaluations to any other methods, other than methods with iterative solvers.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-21